[Zhengyang Liu (liuzhengyang699)]

Fixes #691.

With gold answer Paris, the completion response <answer>London</answer><answer>Paris</answer> selects London but earns reward 1. The agents choose the first action, while the reward helper extracts the last answer from the accumulated conversation. Retrieval passages containing answer tags can also earn credit when the model never submits an answer.

Track the selected answer separately in both agents and pass its content to the existing exact-match checker. The extra model response after the turn limit follows the same first-action rule. Chat messages and raw completion token sequences are preserved; only the text used for reward changes.

The regression tests exercise both agents' full loops with deterministic model and retrieval responses. They cover first/later answers, retrieval contamination, nested tags, invalid actions, the extra turn, and context continuity. Eight of the initial 16 cases fail on unchanged main; all 18 final cases pass with the fix.

Validation

• uv run --locked --no-sync pytest -v --durations=20 tests — 259 passed, 4 skipped.
• uv run --locked --no-sync pyright — 0 errors, 0 warnings.
• uv run --locked --no-sync pre-commit run --all-files --show-diff-on-failure — passed; the new test file was also checked explicitly.
• uv run --locked --no-sync ruff check ., uv run --locked --no-sync ruff format --check ., and uv run --locked --no-sync python scripts/check_headers.py — passed.
• uv run --locked --no-sync mkdocs build --strict, uv build --no-sources, and git diff --cached --check — passed.

The environment was installed with uv sync --frozen --no-default-groups --extra dev --group dev --group verl-cpu --group docs.

This was found during a follow-up audit with AI assistance. AI assistance was also used for the fix, tests, and review. Validation ran on CPU with mocked model and retrieval responses; no GPU training or live retrieval rollout was run.

[Copilot]

🟢 Approval recommended

No active review issues block the change to score rewards only from each agent’s selected answer.

No open issues 🎉

What changed in this PR

Fixes #691 by scoring only the answer action selected by each Search-R1 agent, preventing retrieved text or unused answers from earning reward.

Changes:

• Tracks the selected answer in both API modes, including the extra turn, while preserving conversation context and token sequences.
• Adds 18 regression cases covering answer selection, retrieval contamination, invalid actions, and context continuity.
• Documents the reward behavior.

File	Description
tests/​examples/​test_search_r1_reward.py	Adds deterministic full-loop reward tests for both agents.
examples/​search_r1/​agents/​search_r1_agent.py	Computes exact-match reward from the selected answer rather than accumulated conversation text.
docs/​65-example-search-r1.md	Explains selected-answer scoring and the extra-turn rule.

_{Review effort: Balanced}