Repository navigation
fix(trainer): report hard selection accuracy in the run summary - #432
Merged
Yifan Yang (Yif-Yang) merged 2 commits intoNov 21, 2026
Merged
Yifan Yang (Yif-Yang) merged 2 commits into
Yifan Yang (Yif-Yang) merged 2 commits into
Conversation
The summary set best_selection_hard to best_score, and when the final skill equalled the best skill it reused best_score as final_selection_hard. best_score is the gate-metric score, so under the soft or mixed gate metric (or with the semantic-density bonus) both fields reported a value that is not hard accuracy, while baseline_selection_hard for the same skill reported the real one. With hard 0.5 and soft 0.75, a soft-gate run summarized the initial skill as baseline_selection_hard 0.5 but best_selection_hard 0.75. Read the cached (hard, soft) selection scores instead, as baseline_selection_hard already does, and cache the final skill's selection scores so a final skill promoted to best is found too. best_score remains the fallback when a skill has no cached scores, as on a resume without history. Hard-gate runs report the same values.
Keep both CHANGELOG entries; no other change.
Contributor
|
Merged, thank you. The run summary now reports the cached hard and soft selection accuracy for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #430.
The run summary set
best_selection_hardtobest_score, and when the final skill equalled the best skill it reusedbest_scoreasfinal_selection_hard.best_scoreis the gate-metric score, not hard accuracy, so under thesoftormixedgate metric, or with the semantic-density bonus, both fields reported the wrong number. They now read the cached(hard, soft)selection scores, asbaseline_selection_hardalready does.User impact: in a soft-gate run where the initial skill stays best, the summary reported it as
baseline_selection_hard: 0.5butbest_selection_hard: 0.75andfinal_selection_hard: 0.75, two different "hard" accuracies for one skill. Anyone comparing runs or reading the summary got the gate score under a hard-accuracy label.Proof: the regression tests fail on upstream
22c1ed8and pass with this change.Root cause
ReflACTTrainer.train()inskillopt/engine/trainer.pywrote"best_selection_hard": best_scoreand, on thefinal skill == best skillpath,final_selection_hard, final_selection_soft = best_score, None. The selection cache already holds the real(hard, soft)pair per skill hash; it is whatbaseline_selection_harduses. This is the same mix-up of a gate score with a(hard, soft)pair that #422 fixed for the slow-update cache entry.Changes
best_selection_hardreadssel_cache[skill_hash(best_skill)][0].final skill == best skillpath,final_selection_hardandfinal_selection_softcome from the same cache entry, and the log line prints the hard score.best_scorestays the fallback when a skill has no cached scores, as on a resume without history, so the existing resume tests are unchanged.tests/test_trainer_summary_scores.py: a realReflACTTrainerrun with a mocked environment, as intests/test_trainer_resume.py, under thesoft,mixedand defaulthardgate metrics.Reproduction
Before:
After:
Testing
python -m pytest tests/test_trainer_summary_scores.py -q: on upstream22c1ed8, 2 failed (assert 0.75 == 0.5forsoft,assert 0.625 == 0.5formixed) and 1 passed (hard); with the fix, 3 passed.python -m pytest tests/test_trainer_summary_scores.py tests/test_trainer_resume.py tests/test_trainer_slow_update_score.py tests/test_trainer_failure_patterns.py -q: 15 passed.python -m pytest -q: 2358 passed, 12 skipped, 4 xfailed.python -m ruff checkon the changed files: clean.python -m ruff format --checkflagstrainer.pyonmainas well; the new lines follow the layout of the neighbouringbaseline_selection_hardentry, and the new test file is format-clean.python -m mkdocs build --strict: passed.Risk and compatibility
With the default
hardgate metric and no semantic-density bonus, the gate score equals hard accuracy, so those runs report the same values as before. Only the summary and its log line change; training decisions andbest_scoreitself are untouched. No backend, config key, file format or CLI change.