Repository navigation
feat(build,verify): prove tests can fail — red first, then planted bugs (#111) - #122
Merged
Merged
Conversation
…gs (#111) build runs each criterion test against the pre-task code in a throwaway worktree (must fail) and the finished code (must pass), recording a <red-green> line. verify 3.7d counts them and plants 1-2 bugs in a throwaway copy to confirm a test goes red. New tested script templates/_scripts/expect-red.sh never touches the working tree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #111.
In plain words
An agent can write a test that passes no matter what the code does, so "all tests pass" can mean nothing. Now every test written for a task must fail before the code exists and pass after, and
verifydeliberately breaks 1–2 of the most important features in a throwaway copy to check that a test notices.What changed
templates/_scripts/expect-red.sh. Runs a test command in a throwaway copy of the repo (a git worktree), with new test files copied in (--copy) or a small bug swapped in (--swap FILE OLD NEW, or--patch). Answers RED / GREEN / NO-BUILD (didn't compile) / SETUP. Your working files are never touched; the copy is always removed.build, red first. Agents write criterion tests before the code. Thenbuilditself checks: the named tests pass on the finished code, and fail (or don't compile) against the pre-task code. A test that passes without the code goes back to the agent once, then fails the task. One<red-green>line per test goes in PLAN.md.verify3.7d "Tests Can Fail". Counts the red-green lines, then plants 1–2 bugs (flip a condition, skip a save) in the riskiest features' code in a throwaway copy. Caught = good; missed = an issue with a fix task. Tests that only a walkthrough or a person checks are listed as skipped.<red-green>), CLAUDE.md, CHANGELOG.Tests
14 new script tests (red first, a do-nothing test caught, planted bug caught, harmless change stays green, compile error vs failed assertion, "0 tests ran", ambiguous swap refused, working tree untouched, copy removed). Five planted bugs in the script itself were each caught, then reverted.
Cost
One extra test build per task that has criterion tests, in the throwaway copy. A shared build cache folder keeps later runs faster; the first is the slow one.
🤖 Generated with Claude Code