Repository navigation
test(bench): first honest correctness numbers, the tool-result cap measured, and the gate confirmed live - #657
Merged
Conversation
… bought none
Three arms on claude-haiku-4-5 against a stage carrying the stage_rowset_write
gate (stage mtime newer than the newest src commit, checked before believing
anything). 15 runs, $0.0644, summed from cost_micro_usd rather than estimated.
recursion set, prompts v2 completed correct
limit=25 3/4 2/4
limit=500 4/4 2/4
heavy set, limit=500 5/6 3/6
Raising the ceiling bought +1 completion and +0 correct answers. t3 was stopped
by the ceiling at 25 — §1 reproduces on a different model and a rewritten
prompt, so that finding is robust — and at 500 it COMPLETED with a serialized
todo-list update as its final answer. The published 3/4 -> 4/4 was a true
statement about termination and nothing else.
t4-long-chain has never measured a chain: 1 model call and 0 tool calls in every
arm on every model, failing the same way each time (answering the previous
task's data, or handling only the first number). Nothing in it requires a tool,
so the run ends at the first assistant message. Rebuild it on tool calls the way
h5-longchain is, or delete it.
The tool-result cap is measured for the first time, and it fires: h6-bigread,
rebased onto /memories/, offloaded 1 result before the model saw it, with the
63,793-byte fixture landing at its designed size and 2,665 tokens as the largest
result that got through. Reproduced twice. Four of §5's five dark claims are now
measured.
The gate is confirmed on a live request, from the ledger's own per-call record:
20 tools / 7,910 tok -> 19 tools / 7,010 tok, exactly -900, both siblings still
present. That is the seam the model reads — the distinction that killed
title-slimming.
A correction to §7. Three fresh stores with an identical config, minutes apart,
all OPENED WARM (7,621 of ~13,000 cached on the first call). So a process
boundary does not by itself cost a cold prompt; a prefix change does, and §7's
67.7% was measuring how often two heterogeneous journeys shared a config — the
blind spot it named for itself. The cached 7,621 is almost exactly the tools
segment, which suggests the system block diverges per store early on. Recorded
as a hypothesis: n=3, one afternoon, one model.
The v1 arms stay untouched under their published names, since §1-§6 cite them
and a gate test pins them as "unknown, never graded". The v2 arms are
arm-25-v2.json and arm-500-v2.json.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three paid arms on
claude-haiku-4-5(15 runs, $0.0644), written up asFINDINGS.md§10. Docs and run reports only — no source changes.t3is stopped by the ceiling at 25.t4-long-chainhas never measured a chain — 1 model call, 0 tool calls, every arm, every model.stage_rowset_writegate is confirmed live: 20 tools / 7,910 tok → 19 / 7,010, exactly −900, from the ledger's per-call record of a real request.The v1 arms are untouched under their published names; v2 arms are
arm-25-v2.json/arm-500-v2.json. 58 harness gates pass.Note:
build-and-auditis currently red for every PR — newly published npm advisories (Astro critical; xmldom, browserslist, fast-uri, js-yaml high). This PR changes no npm manifest. See #656.🤖 Generated with Claude Code