fix: keep the research loop's instructions consistent with what it allows - #28
Open
SongTonyLi wants to merge 2 commits into
Open
fix: keep the research loop's instructions consistent with what it allows#28SongTonyLi wants to merge 2 commits into
SongTonyLi wants to merge 2 commits into
Conversation
…lows Three places where what the model was told disagreed with what the harness would actually do, plus one where the UI lagged the model. The ledger was appended to every round's last tool message and left there, so by round three the model was reading three checklists that disagreed — the oldest still showing [ ] against items later rounds had ticked, each with its own copy of the stopping rule. Only the newest round now carries the one copy; the previous one is stripped before the fresh one lands. Once a cap tripped and the tool was withdrawn, the brief on that very request still said "search only to close a specific [ ] item" under a system prompt that still said "You have a web_search tool". A model taking that literally emitted another tool call and no prose, and needed a whole extra turn to be told in a tool reply that research was closed. The brief (and the transcript copy) now say so up front. On OpenRouter, assistant tool calls were given ids from their index (call_0) and tool replies from their position in the conversation (tool_3), so no reply ever answered its call — a 400 from strict OpenAI-compatible providers on every research turn after the first. Replies now consume the preceding assistant message's ids in order. The completeness gate opened its gaps in the ledger but published them to the panel only after the next search round — never, if the model answered the gap notice without searching. It publishes them at once. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AENaLjMDRDiM5gibbgueLM
The research loop had no way to find out what the user meant. A message naming something ambiguous — "the Mercury results", "the Jaguar lineup", a statistic with no year — was restated into a goal on one reading and researched to a confident, well-cited answer to a question nobody asked. The completeness gate can never catch that: the draft does cover the question as the run understood it. Goal derivation may now end with a CLARIFY block — one short question and two to four concrete readings — under a prompt biased hard against asking. When it does, the run pauses before its first search on a checkbox card in the bubble: pick any that apply and Continue, or Skip to let the run proceed on its own reading. The picks are folded into the question every later stage judges against (the ledger's instance detection and the completeness gate both read it) and stated in the brief on every turn, since the answering model's history holds only the ambiguous original. The card is the record afterwards: a saved message shows what was asked and chosen, with the form controls gone. Stop answers the card for this chat's run only; the prompt bar says what it is waiting for while paused. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AENaLjMDRDiM5gibbgueLM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three places where what the model was told disagreed with what the
harness would actually do, plus one where the UI lagged the model.
The ledger was appended to every round's last tool message and left
there, so by round three the model was reading three checklists that
disagreed — the oldest still showing [ ] against items later rounds had
ticked, each with its own copy of the stopping rule. Only the newest
round now carries the one copy; the previous one is stripped before the
fresh one lands.
Once a cap tripped and the tool was withdrawn, the brief on that very
request still said "search only to close a specific [ ] item" under a
system prompt that still said "You have a web_search tool". A model
taking that literally emitted another tool call and no prose, and
needed a whole extra turn to be told in a tool reply that research was
closed. The brief (and the transcript copy) now say so up front.
On OpenRouter, assistant tool calls were given ids from their index
(call_0) and tool replies from their position in the conversation
(tool_3), so no reply ever answered its call — a 400 from strict
OpenAI-compatible providers on every research turn after the first.
Replies now consume the preceding assistant message's ids in order.
The completeness gate opened its gaps in the ledger but published them
to the panel only after the next search round — never, if the model
answered the gap notice without searching. It publishes them at once.
Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01AENaLjMDRDiM5gibbgueLM