Skip to content

fix(trainer): report hard selection accuracy in the run summary - #432

Merged
Yifan Yang (Yif-Yang) merged 2 commits into
microsoft:mainfrom
Shizoqua:fix/trainer-summary-hard-scores
Nov 21, 2026
Merged

Yifan Yang (Yif-Yang) merged 2 commits into
microsoft:mainfrom
Shizoqua:fix/trainer-summary-hard-scores

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

Summary

Fixes #430.

The run summary set best_selection_hard to best_score, and when the final skill equalled the best skill it reused best_score as final_selection_hard. best_score is the gate-metric score, not hard accuracy, so under the soft or mixed gate metric, or with the semantic-density bonus, both fields reported the wrong number. They now read the cached (hard, soft) selection scores, as baseline_selection_hard already does.

User impact: in a soft-gate run where the initial skill stays best, the summary reported it as baseline_selection_hard: 0.5 but best_selection_hard: 0.75 and final_selection_hard: 0.75, two different "hard" accuracies for one skill. Anyone comparing runs or reading the summary got the gate score under a hard-accuracy label.

Proof: the regression tests fail on upstream 22c1ed8 and pass with this change.

Root cause

ReflACTTrainer.train() in skillopt/engine/trainer.py wrote "best_selection_hard": best_score and, on the final skill == best skill path, final_selection_hard, final_selection_soft = best_score, None. The selection cache already holds the real (hard, soft) pair per skill hash; it is what baseline_selection_hard uses. This is the same mix-up of a gate score with a (hard, soft) pair that #422 fixed for the slow-update cache entry.

Changes

  • best_selection_hard reads sel_cache[skill_hash(best_skill)][0].
  • On the final skill == best skill path, final_selection_hard and final_selection_soft come from the same cache entry, and the log line prints the hard score.
  • The final-skill evaluation is now cached, so a final skill promoted to best is found too.
  • best_score stays the fallback when a skill has no cached scores, as on a resume without history, so the existing resume tests are unchanged.
  • New tests/test_trainer_summary_scores.py: a real ReflACTTrainer run with a mocked environment, as in tests/test_trainer_resume.py, under the soft, mixed and default hard gate metrics.
  • CHANGELOG entry under Unreleased / Fixed.

Reproduction

import pathlib, tempfile
from unittest import mock

from skillopt.engine.trainer import ReflACTTrainer
from skillopt.envs.base import EnvAdapter

tmp = pathlib.Path(tempfile.mkdtemp())
(tmp / "skill.md").write_text("# Skill\n")
cfg = {
    "out_root": str(tmp / "out"), "skill_init": str(tmp / "skill.md"),
    "backend": "openai_chat", "optimizer_backend": "openai_chat", "target_backend": "openai_chat",
    "num_epochs": 0, "train_size": 1, "batch_size": 1, "accumulation": 1, "merge_batch_size": 2,
    "edit_budget": 2, "seed": 42, "sel_env_num": 2, "test_env_num": 2, "eval_test": True,
    "gate_metric": "soft",
}
adapter = mock.create_autospec(EnvAdapter, instance=True)
adapter.requires_ray.return_value = False
adapter.get_dataloader.return_value = None
adapter.get_task_types.return_value = []
adapter.build_eval_env.return_value = ["1", "2"]
# Selection hard accuracy 0.5, soft score 0.75.
adapter.rollout.return_value = [{"id": "1", "hard": 0, "soft": 0.5}, {"id": "2", "hard": 1, "soft": 1.0}]

summary = ReflACTTrainer(cfg, adapter).train()
print({key: summary[key] for key in ("baseline_selection_hard", "best_selection_hard", "final_selection_hard", "final_selection_soft")})

Before:

{'baseline_selection_hard': 0.5, 'best_selection_hard': 0.75, 'final_selection_hard': 0.75, 'final_selection_soft': None}

After:

{'baseline_selection_hard': 0.5, 'best_selection_hard': 0.5, 'final_selection_hard': 0.5, 'final_selection_soft': 0.75}

Testing

  • python -m pytest tests/test_trainer_summary_scores.py -q: on upstream 22c1ed8, 2 failed (assert 0.75 == 0.5 for soft, assert 0.625 == 0.5 for mixed) and 1 passed (hard); with the fix, 3 passed.
  • python -m pytest tests/test_trainer_summary_scores.py tests/test_trainer_resume.py tests/test_trainer_slow_update_score.py tests/test_trainer_failure_patterns.py -q: 15 passed.
  • python -m pytest -q: 2358 passed, 12 skipped, 4 xfailed.
  • python -m ruff check on the changed files: clean. python -m ruff format --check flags trainer.py on main as well; the new lines follow the layout of the neighbouring baseline_selection_hard entry, and the new test file is format-clean.
  • python -m mkdocs build --strict: passed.

Risk and compatibility

With the default hard gate metric and no semantic-density bonus, the gate score equals hard accuracy, so those runs report the same values as before. Only the summary and its log line change; training decisions and best_score itself are untouched. No backend, config key, file format or CLI change.

The summary set best_selection_hard to best_score, and when the final
skill equalled the best skill it reused best_score as
final_selection_hard. best_score is the gate-metric score, so under the
soft or mixed gate metric (or with the semantic-density bonus) both
fields reported a value that is not hard accuracy, while
baseline_selection_hard for the same skill reported the real one. With
hard 0.5 and soft 0.75, a soft-gate run summarized the initial skill as
baseline_selection_hard 0.5 but best_selection_hard 0.75.

Read the cached (hard, soft) selection scores instead, as
baseline_selection_hard already does, and cache the final skill's
selection scores so a final skill promoted to best is found too.
best_score remains the fallback when a skill has no cached scores, as
on a resume without history. Hard-gate runs report the same values.
Keep both CHANGELOG entries; no other change.
@Yif-Yang
Yifan Yang (Yif-Yang) merged commit 5e7df12 into microsoft:main Nov 21, 2026
@Yif-Yang

Copy link
Copy Markdown
Contributor

Merged, thank you. The run summary now reports the cached hard and soft selection accuracy for best_selection_hard and final_selection_hard instead of the gate-metric score, consistent with baseline_selection_hard. On main, the new tests fail (2 failures) and pass here. This only fixes reporting; training decisions are unaffected. Closes #430. Your branch and an earlier PR in this batch each added an entry at the same place in CHANGELOG.md, so I pushed one maintainer merge commit to your branch that keeps both entries and changes nothing else, then merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run summary reports the gate score as best_selection_hard and final_selection_hard

2 participants