Repository navigation
Conversation
_forward_sfs mirrors the paper's selection (mlxtend forward SFS: macro F1 over unshuffled stratified folds, ascending candidates, first-max tie-break) and records the best subset of each size; _one_sigma_subset applies the paper's rule (smallest subset within one std of the best).
…vote The experimental per-quantile ensemble (feat_idx, n_quantiles, find_best_metrics, best_metrics_per_quantile_, quantile_scores_df_) is removed. The new EnsembleDistanceClassifier (distclassipy/ensemble.py) gives every metric its own DistanceMetricClassifier on its own features: passed in via feature_sets (indices, or names with a DataFrame), or else selected in fit by the paper's forward SFS and 1-sigma rule. Each member's out-of-fold class distances give its macro F1 (the weight, weights="f1") and a softmax temperature (calibration); predict_proba is the weighted mean of the members' probabilities, and predict_proba_per_metric exposes each member's vote. Members default to reference="knn", n_neighbors=15 over the 16 _UNIQUE_METRICS. Undefined distances give probability 0 (uniform if none is finite); classes with fewer than 2 objects raise a clear error.
- README: usage example and feature bullet for the new ensemble (all README code blocks run in sequence). - Tutorial section 4 rewritten for the new API (fit with defaults, F1, per-metric weights/scores/features, per-metric votes for one object). The install cell now installs distclassipy>=0.4.0 only when it is missing (it pinned 0.2.1, which lacks the new ensemble); the notebook was re-executed top to bottom with the current code. - New docs/ensemble.rst API page (0 Sphinx warnings with autodoc + napoleon); ensemble module docstring no longer parsed as a section.
cross_val_score's default error_score=np.nan turned a failing candidate (e.g. a misspelt metric) into a NaN score, which np.argmax treats as the maximum, so the broken candidate was 'selected' and fit later failed with 'Encountered all NA values'. Use error_score='raise' so the clear 'metric not found' error reaches the user.
The new EnsembleDistanceClassifier ships in 0.4.1: the tutorial now asks for distclassipy>=0.4.1, and the class docstring notes that it replaces the experimental per-quantile ensemble.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR improves
EnsembleDistanceClassifier, previously it picked one metric per quantile of a single feature and always used all features. It is replaced outright (feat_idx,n_quantiles,find_best_metrics,best_metrics_per_quantile_andquantile_scores_df_are removed).The new
EnsembleDistanceClassifiercombines distance metrics so that no single metric has to be chosen:DistanceMetricClassifieron its own features, either passed in throughfeature_sets(column indices, or names with a DataFrame) or selected infitwith the paper's forward sequential feature selection and 1σ rule;softmax(-distance / temperature), with the temperature calibrated on internal cross-validation;weights="f1", or"uniform");predict_probaandpredict_proba_per_metric(how each metric voted) are available.reference="knn", n_neighbors=15over the 16 metrics in_UNIQUE_METRICS.distclassipy/ensemble.py. The feature selection reproduces the paper's mlxtend-based SFS exactly (same subsets for all 18 metrics on the multiclass task). README, tutorial (section 4) and a new API page are updated.On the DCPY paper's hidden sets (macro F1, same objects):
With F1 weights the ensemble is within about one point of the best single metric without choosing one, but the weights are nearly flat because all metrics score similar F1. Weights learned from the out-of-fold predictions beat the best single metric on two of three tasks; adding them as an option is a natural follow-up.
Bumps version to 0.4.1