Skip to content

Replace EnsembleDistanceClassifier with a per-metric weighted ensemble - #44

Merged
sidchaini merged 6 commits into
mainfrom
ensemble
Nov 16, 2026
Merged

sidchaini merged 6 commits into
mainfrom
ensemble

Conversation

@sidchaini

@sidchaini sidchaini commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

This PR improves EnsembleDistanceClassifier, previously it picked one metric per quantile of a single feature and always used all features. It is replaced outright (feat_idx, n_quantiles, find_best_metrics, best_metrics_per_quantile_ and quantile_scores_df_ are removed).

The new EnsembleDistanceClassifier combines distance metrics so that no single metric has to be chosen:

  • every metric gets its own DistanceMetricClassifier on its own features, either passed in through feature_sets (column indices, or names with a DataFrame) or selected in fit with the paper's forward sequential feature selection and 1σ rule;
  • each member's class distances become class probabilities, softmax(-distance / temperature), with the temperature calibrated on internal cross-validation;
  • the ensemble averages the members' probabilities, weighted by each member's internal cross-validated F1 (weights="f1", or "uniform");
  • predict_proba and predict_proba_per_metric (how each metric voted) are available.
  • Members default to reference="knn", n_neighbors=15 over the 16 metrics in _UNIQUE_METRICS.
  • distclassipy/ensemble.py. The feature selection reproduces the paper's mlxtend-based SFS exactly (same subsets for all 18 metrics on the multiclass task). README, tutorial (section 4) and a new API page are updated.

On the DCPY paper's hidden sets (macro F1, same objects):

binary multiclass one-vs-rest
best single metric (chosen by CV) 70.7 96.5 94.7
ensemble, F1 weights 71.1 95.5 93.9
ensemble, weights learned from out-of-fold predictions 72.1 97.2 94.4

With F1 weights the ensemble is within about one point of the best single metric without choosing one, but the weights are nearly flat because all metrics score similar F1. Weights learned from the out-of-fold predictions beat the best single metric on two of three tasks; adding them as an option is a natural follow-up.

Bumps version to 0.4.1

_forward_sfs mirrors the paper's selection (mlxtend forward SFS: macro F1
over unshuffled stratified folds, ascending candidates, first-max
tie-break) and records the best subset of each size; _one_sigma_subset
applies the paper's rule (smallest subset within one std of the best).
…vote

The experimental per-quantile ensemble (feat_idx, n_quantiles,
find_best_metrics, best_metrics_per_quantile_, quantile_scores_df_) is
removed. The new EnsembleDistanceClassifier (distclassipy/ensemble.py)
gives every metric its own DistanceMetricClassifier on its own features:
passed in via feature_sets (indices, or names with a DataFrame), or else
selected in fit by the paper's forward SFS and 1-sigma rule. Each
member's out-of-fold class distances give its macro F1 (the weight,
weights="f1") and a softmax temperature (calibration); predict_proba is
the weighted mean of the members' probabilities, and
predict_proba_per_metric exposes each member's vote. Members default to
reference="knn", n_neighbors=15 over the 16 _UNIQUE_METRICS.

Undefined distances give probability 0 (uniform if none is finite);
classes with fewer than 2 objects raise a clear error.
- README: usage example and feature bullet for the new ensemble (all
  README code blocks run in sequence).
- Tutorial section 4 rewritten for the new API (fit with defaults, F1,
  per-metric weights/scores/features, per-metric votes for one object).
  The install cell now installs distclassipy>=0.4.0 only when it is
  missing (it pinned 0.2.1, which lacks the new ensemble); the notebook
  was re-executed top to bottom with the current code.
- New docs/ensemble.rst API page (0 Sphinx warnings with autodoc +
  napoleon); ensemble module docstring no longer parsed as a section.
cross_val_score's default error_score=np.nan turned a failing candidate
(e.g. a misspelt metric) into a NaN score, which np.argmax treats as the
maximum, so the broken candidate was 'selected' and fit later failed
with 'Encountered all NA values'. Use error_score='raise' so the clear
'metric not found' error reaches the user.
Base automatically changed from aiqr-cdf to main November 16, 2026 06:29
The new EnsembleDistanceClassifier ships in 0.4.1: the tutorial now asks
for distclassipy>=0.4.1, and the class docstring notes that it replaces
the experimental per-quantile ensemble.
@sidchaini
sidchaini merged commit 8eb6a51 into main Nov 16, 2026
20 checks passed
@sidchaini
sidchaini deleted the ensemble branch November 16, 2026 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant