Repository navigation
critic: judge a long test file in batches of 20, so its schema fits - #381
Merged
Merged
Conversation
rc.17 put every listed test's name in one answer schema. A paid certify run
on flask showed what that does to a real file: tests/test_basic.py declares
about 90 tests, the critic seat made 9 calls on 295k input tokens, and it
had fallen to the loop. Probed directly, claude-haiku-4-5 refuses a keyed
schema of 40 names ("The compiled grammar is too large") and gemini-3.8-flash
refuses 90 with a bare 400; both accept 20. Size, not the pytest-style
names, was the cause.
A list longer than criticBatchSize (20) is now split: each batch carries the
whole code and test file but lists only its own tests, gets its own keyed
schema and its own retry, and the batches' findings are joined. A batch that
still will not answer sends the whole file to the loop, and the batches that
did answer are not kept beside it as a review of their own. On the same
flask files, through the production seat, both Haiku and Gemini judged all
90 tests in 5 calls (187k and 168k input).
VerdictGeneration moves 14 to 15. The bench doc records the probes and the
flask result as section 9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rc.17's keyed schema put every test's name in one schema. On flask's
tests/test_basic.py(about 90 tests), both providers refused it: Haiku past 20 names, with "compiled grammar is too large", and Gemini at 90. The critic then fell to the loop: 9 calls and 295k input.The fix: lists longer than 20 are judged in batches of at most 20 names. Each batch has its own schema and its own retry, and their findings are joined. If a batch still won't answer, the whole file goes to the loop.
Measured on the same flask files (
docs/design/critic-bench.md§9): Haiku and Gemini each judged all 90 tests in 5 calls, using 187k and 168k input.VerdictGeneration 14 → 15. The full suite, vet, gofmt, security, licensing, CLI-docs and publication gates pass.
🤖 Generated with Claude Code