Skip to content

feat(build,verify): prove tests can fail — red first, then planted bugs (#111) - #122

Merged
rshankras merged 1 commit into
mainfrom
feat/111-tests-can-fail
Nov 6, 2026
Merged

rshankras merged 1 commit into
mainfrom
feat/111-tests-can-fail

Conversation

@rshankras

Copy link
Copy Markdown
Owner

Closes #111.

In plain words

An agent can write a test that passes no matter what the code does, so "all tests pass" can mean nothing. Now every test written for a task must fail before the code exists and pass after, and verify deliberately breaks 1–2 of the most important features in a throwaway copy to check that a test notices.

What changed

  • New script templates/_scripts/expect-red.sh. Runs a test command in a throwaway copy of the repo (a git worktree), with new test files copied in (--copy) or a small bug swapped in (--swap FILE OLD NEW, or --patch). Answers RED / GREEN / NO-BUILD (didn't compile) / SETUP. Your working files are never touched; the copy is always removed.
  • build, red first. Agents write criterion tests before the code. Then build itself checks: the named tests pass on the finished code, and fail (or don't compile) against the pre-task code. A test that passes without the code goes back to the agent once, then fails the task. One <red-green> line per test goes in PLAN.md.
  • verify 3.7d "Tests Can Fail". Counts the red-green lines, then plants 1–2 bugs (flip a condition, skip a save) in the riskiest features' code in a throwaway copy. Caught = good; missed = an issue with a fix task. Tests that only a walkthrough or a person checks are listed as skipped.
  • Reports: "12 of 12 tests failed before the code and pass now. Planted bugs: 2 of 2 caught." in VERIFICATION.md, its summary, and the completion message.
  • PLAN.md template (<red-green>), CLAUDE.md, CHANGELOG.

Tests

14 new script tests (red first, a do-nothing test caught, planted bug caught, harmless change stays green, compile error vs failed assertion, "0 tests ran", ambiguous swap refused, working tree untouched, copy removed). Five planted bugs in the script itself were each caught, then reverted.

Cost

One extra test build per task that has criterion tests, in the throwaway copy. A shared build cache folder keeps later runs faster; the first is the slow one.

🤖 Generated with Claude Code

…gs (#111)

build runs each criterion test against the pre-task code in a throwaway
worktree (must fail) and the finished code (must pass), recording a
<red-green> line. verify 3.7d counts them and plants 1-2 bugs in a
throwaway copy to confirm a test goes red. New tested script
templates/_scripts/expect-red.sh never touches the working tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rshankras
rshankras merged commit bfdd3b3 into main Nov 6, 2026
2 checks passed
@rshankras
rshankras deleted the feat/111-tests-can-fail branch November 6, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(build,verify): prove tests can fail — red first, then a planted bug

1 participant