Repository navigation
feat: secondwind eval, and read tool results as the model saw them - #11
Merged
Merged
Conversation
The Claude Code reader wrote a tool result's content array out as raw JSON, so text results carried a wrapper the model never sees and images and PDFs carried their base64 as text. On real sessions that made tool output look seven times larger than it is, inflating cache-savings and the audit. Text blocks are now joined and an image or document is a short placeholder.
Per tool and command: blocks kept inline or offloaded, how often the model later needed what an offload hid, and net tokens. A hidden token counts only if the model used it before another result supplied it again. Every block is checked faithful independently of the optimizer's gate, and --blocks writes one line per result with its context size and remaining turns for policy studies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
secondwind evalreplays every recorded Claude Code session through the optimizer and reports, per tool and per command (Bash:cat,Bash:grep |,Bash:git log,Read, ...):unfaithful: blocks that did not come back intact--blocks <path>writes one line per result with its byte size, the context the model held, and the turns after it, so policies can be compared on held-out sessions.Fix in the trace reader
A tool result's content array was serialized as raw JSON: text results carried a wrapper the model never sees, and images and PDFs carried their base64 as text. On the author's 1,491 sessions that made tool output look like 739M tokens when the text the model read is 101M. The reader now joins text blocks and replaces an image or document with a short placeholder. This also corrects
cache-savingsand the audit, which read traces the same way.How a reopen is counted
A block counts as reopened when a token only it introduced, and its preview hid, appears in a later assistant message or tool call before any other result or user message supplies that token again. A grep followed by a read of the same file charges the read, not the grep. Net assumes a reopened block is fetched back whole, the costliest case.
Faithfulness
Checked per block, independently of the optimizer's own gate: an inline wire must decode to the certified content; an offload must resolve from the store to the original, byte-identical text or the same JSON value. On the author's corpus: 0 of 138,209 blocks unfaithful.
Tests
Unit tests for reopen attribution (including the re-supply rule), the command classifier on heredocs, loops, quoted programs,
git -Candtimeoutprefixes, faithfulness across JSON, log and code shapes, and the reader's text/image handling; an end-to-end test runssecondwind evalon a synthetic recorded session.cargo test --workspace: 354 passed. Clippy clean.