Skip to content

fix(envs): ignore an <answer> tag mentioned before the real answer - #433

Merged
Yifan Yang (Yif-Yang) merged 2 commits into
microsoft:mainfrom
Shizoqua:fix/answer-tag-stray-opening
Nov 21, 2026
Merged

Yifan Yang (Yif-Yang) merged 2 commits into
microsoft:mainfrom
Shizoqua:fix/answer-tag-stray-opening

Conversation

@Shizoqua

Copy link
Copy Markdown
Contributor

Summary

Fixes #428.

SearchQA, LiveMathematicianBench and OfficeQA extracted the answer with the lazy pattern <answer>(.*?)</answer>. When the model mentioned the tag in its reasoning before answering, for example I will wrap the final answer in <answer> tags. followed by <answer>Paris</answer>, the captured text ran from that mention to the real closing tag, and a correct answer scored 0. All three now use the pattern the DocVQA evaluator already uses.

User impact: hard and soft come from the extracted answer, so a correct answer after a mention of the tag got zero training reward and counted as a miss in reported scores, on three of the six benchmarks.

Proof: the regression tests fail on upstream 22c1ed8 and pass with this change.

Root cause

A lazy (.*?) still starts its match at the first <answer>. extract_answer() in skillopt/envs/searchqa/evaluator.py and skillopt/envs/livemathematicianbench/evaluator.py, and _FINAL_RE in skillopt/envs/officeqa/rollout.py, all used it, so an earlier mention of the tag became the start of the answer. A nested <answer>draft<answer>Paris</answer> failed the same way.

Changes

  • All three use <answer>((?:(?!<answer>).)*?)</answer>: the body may not contain another <answer>, so each match starts at the last opening tag before its closing tag. This is the pattern DocVQA adopted.
  • Responses with well-formed tags are extracted as before, including the last-tag-wins behavior and the no-tag fallback. The pattern stays fast on long or repetitive input: 200,000 repeated tags or a 2,000,000-character unclosed answer each finish in well under a second.
  • Regression tests added to tests/test_searchqa_evaluator.py, tests/test_livemath_evaluator.py and tests/test_officeqa_answer_extraction.py: the tag mentioned in prose, and the nested case.
  • CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.livemathematicianbench.evaluator import evaluate as livemath_evaluate
from skillopt.envs.officeqa.rollout import _extract_answer as officeqa_extract
from skillopt.envs.searchqa.evaluator import evaluate as searchqa_evaluate

text = "I will wrap the final answer in <answer> tags.\n<answer>Paris</answer>"
print("searchqa:", searchqa_evaluate(text, ["Paris"]))
print("officeqa:", repr(officeqa_extract(text)))

choices = [{"label": "A", "text": "2"}, {"label": "B", "text": "3"}]
result = livemath_evaluate("Use the <answer> tag format.\n<answer>B</answer>", choices[1], choices)
print("livemath:", repr(result["predicted_label"]), result["em"])

Before:

searchqa: {'em': 0.0, 'f1': 0.0, 'sub_em': 1.0, 'predicted_answer': 'tags.\n<answer>Paris', 'gold_answers': ['Paris']}
officeqa: 'tags.\n<answer>Paris'
livemath: 'TAG FORMAT.\n<ANSWER>B' 0.0

After: SearchQA extracts Paris with EM 1.0, OfficeQA extracts 'Paris', and LiveMath predicts 'B' with EM 1.0.

Testing

  • python -m pytest tests/test_searchqa_evaluator.py tests/test_livemath_evaluator.py tests/test_officeqa_answer_extraction.py -q: on upstream 22c1ed8, 6 failed and 26 passed; with the fix, 32 passed.
  • All SearchQA, LiveMathematicianBench and OfficeQA tests plus tests/test_benchmark_task_output_paths.py: 155 passed.
  • python -m pytest -q: 2361 passed, 12 skipped, 4 xfailed.
  • python -m ruff check on the changed files: clean. python -m ruff format --check flags the three source files on main as well, on lines this change does not touch; the test files are format-clean.
  • python -m mkdocs build --strict: passed.

Risk and compatibility

Only responses that contain an <answer> before the opening tag of the final answer change outcome, and those were scored from the wrong text before. Scores can rise for runs whose target model mentions the tag in its reasoning, so results from before and after are not directly comparable. No backend, config key, file format or CLI change.

SearchQA, LiveMathematicianBench and OfficeQA extract the answer with
the lazy pattern <answer>(.*?)</answer>. A lazy match still starts at the
first "<answer>", so when the model mentions the tag in its reasoning
before answering ("I will wrap the final answer in <answer> tags.
<answer>Paris</answer>"), the captured text runs from that mention to the
real closing tag and a correct answer scores 0. The same happens with a
nested "<answer>draft<answer>Paris</answer>".

Use the pattern the DocVQA evaluator adopted: the body may not contain
another "<answer>", so the match starts at the last opening tag before
each closing tag. Responses with well-formed tags are extracted as
before, and the regex stays fast on long or repetitive input.
Keep both CHANGELOG entries; no other change.
@Yif-Yang
Yifan Yang (Yif-Yang) merged commit 2d89db4 into microsoft:main Nov 21, 2026
1 check passed
@Yif-Yang

Copy link
Copy Markdown
Contributor

Merged, thank you. SearchQA, LiveMathematicianBench and OfficeQA now use the same extraction as DocVQA, so mentioning <answer> in the reasoning before the real answer no longer turns a correct answer into a miss. On main, the new tests fail (6 failures) and pass here. This changes scores for affected responses, from 0 to correct, so a note for the CHANGELOG reader is welcome in a later docs pass. Closes #428. Your branch and an earlier PR in this batch each added an entry at the same place in CHANGELOG.md, so I pushed one maintainer merge commit to your branch that keeps both entries and changes nothing else, then merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SearchQA, LiveMath and OfficeQA score the wrong text when <answer> is mentioned before the answer

2 participants