Repository navigation
fix(envs): ignore an <answer> tag mentioned before the real answer - #433
Merged
Yifan Yang (Yif-Yang) merged 2 commits intoNov 21, 2026
Merged
Conversation
SearchQA, LiveMathematicianBench and OfficeQA extract the answer with
the lazy pattern <answer>(.*?)</answer>. A lazy match still starts at the
first "<answer>", so when the model mentions the tag in its reasoning
before answering ("I will wrap the final answer in <answer> tags.
<answer>Paris</answer>"), the captured text runs from that mention to the
real closing tag and a correct answer scores 0. The same happens with a
nested "<answer>draft<answer>Paris</answer>".
Use the pattern the DocVQA evaluator adopted: the body may not contain
another "<answer>", so the match starts at the last opening tag before
each closing tag. Responses with well-formed tags are extracted as
before, and the regex stays fast on long or repetitive input.
Keep both CHANGELOG entries; no other change.
Contributor
|
Merged, thank you. SearchQA, LiveMathematicianBench and OfficeQA now use the same extraction as DocVQA, so mentioning |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #428.
SearchQA, LiveMathematicianBench and OfficeQA extracted the answer with the lazy pattern
<answer>(.*?)</answer>. When the model mentioned the tag in its reasoning before answering, for exampleI will wrap the final answer in <answer> tags.followed by<answer>Paris</answer>, the captured text ran from that mention to the real closing tag, and a correct answer scored 0. All three now use the pattern the DocVQA evaluator already uses.User impact:
hardandsoftcome from the extracted answer, so a correct answer after a mention of the tag got zero training reward and counted as a miss in reported scores, on three of the six benchmarks.Proof: the regression tests fail on upstream
22c1ed8and pass with this change.Root cause
A lazy
(.*?)still starts its match at the first<answer>.extract_answer()inskillopt/envs/searchqa/evaluator.pyandskillopt/envs/livemathematicianbench/evaluator.py, and_FINAL_REinskillopt/envs/officeqa/rollout.py, all used it, so an earlier mention of the tag became the start of the answer. A nested<answer>draft<answer>Paris</answer>failed the same way.Changes
<answer>((?:(?!<answer>).)*?)</answer>: the body may not contain another<answer>, so each match starts at the last opening tag before its closing tag. This is the pattern DocVQA adopted.tests/test_searchqa_evaluator.py,tests/test_livemath_evaluator.pyandtests/test_officeqa_answer_extraction.py: the tag mentioned in prose, and the nested case.Reproduction
Before:
After: SearchQA extracts
Pariswith EM1.0, OfficeQA extracts'Paris', and LiveMath predicts'B'with EM1.0.Testing
python -m pytest tests/test_searchqa_evaluator.py tests/test_livemath_evaluator.py tests/test_officeqa_answer_extraction.py -q: on upstream22c1ed8, 6 failed and 26 passed; with the fix, 32 passed.tests/test_benchmark_task_output_paths.py: 155 passed.python -m pytest -q: 2361 passed, 12 skipped, 4 xfailed.python -m ruff checkon the changed files: clean.python -m ruff format --checkflags the three source files onmainas well, on lines this change does not touch; the test files are format-clean.python -m mkdocs build --strict: passed.Risk and compatibility
Only responses that contain an
<answer>before the opening tag of the final answer change outcome, and those were scored from the wrong text before. Scores can rise for runs whose target model mentions the tag in its reasoning, so results from before and after are not directly comparable. No backend, config key, file format or CLI change.