Skip to content

feat: secondwind eval, and read tool results as the model saw them - #11

Merged
hardy30894 merged 2 commits into
mainfrom
feat-eval-harness
Oct 23, 2026
Merged

hardy30894 merged 2 commits into
mainfrom
feat-eval-harness

Conversation

@hardy30894

Copy link
Copy Markdown
Member

What

secondwind eval replays every recorded Claude Code session through the optimizer and reports, per tool and per command (Bash:cat, Bash:grep |, Bash:git log, Read, ...):

  • results kept inline, offloaded, or left alone
  • how often the model later needed what an offload hid
  • raw, sent and handed-back tokens, and net savings
  • unfaithful: blocks that did not come back intact

--blocks <path> writes one line per result with its byte size, the context the model held, and the turns after it, so policies can be compared on held-out sessions.

Fix in the trace reader

A tool result's content array was serialized as raw JSON: text results carried a wrapper the model never sees, and images and PDFs carried their base64 as text. On the author's 1,491 sessions that made tool output look like 739M tokens when the text the model read is 101M. The reader now joins text blocks and replaces an image or document with a short placeholder. This also corrects cache-savings and the audit, which read traces the same way.

How a reopen is counted

A block counts as reopened when a token only it introduced, and its preview hid, appears in a later assistant message or tool call before any other result or user message supplies that token again. A grep followed by a read of the same file charges the read, not the grep. Net assumes a reopened block is fetched back whole, the costliest case.

Faithfulness

Checked per block, independently of the optimizer's own gate: an inline wire must decode to the certified content; an offload must resolve from the store to the original, byte-identical text or the same JSON value. On the author's corpus: 0 of 138,209 blocks unfaithful.

Tests

Unit tests for reopen attribution (including the re-supply rule), the command classifier on heredocs, loops, quoted programs, git -C and timeout prefixes, faithfulness across JSON, log and code shapes, and the reader's text/image handling; an end-to-end test runs secondwind eval on a synthetic recorded session. cargo test --workspace: 354 passed. Clippy clean.

The Claude Code reader wrote a tool result's content array out as raw JSON, so text results carried a wrapper the model never sees and images and PDFs carried their base64 as text. On real sessions that made tool output look seven times larger than it is, inflating cache-savings and the audit. Text blocks are now joined and an image or document is a short placeholder.
Per tool and command: blocks kept inline or offloaded, how often the model later needed what an offload hid, and net tokens. A hidden token counts only if the model used it before another result supplied it again. Every block is checked faithful independently of the optimizer's gate, and --blocks writes one line per result with its context size and remaining turns for policy studies.
@hardy30894
hardy30894 added this pull request to stack #15 October 23, 2026 02:36
@hardy30894
hardy30894 merged commit 2b747c1 into main Oct 23, 2026
3 checks passed
@hardy30894
hardy30894 deleted the feat-eval-harness branch October 23, 2026 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant