guard: verify_test_suite_count --check becomes read-only fail-closed (A1 gate-fix) - #719
Conversation
--check auto-corrected drift: it WROTE tests/CLAUDE.md and exited 0. That made a check mode mutate the tree, made count drift structurally unable to fail CI (fail-open gate), and made tests/test_tests_claudemd_drift.py rewrite a tracked file on every test run. New contract: - --check (default) and --strict are READ-ONLY validation, never write. Drift is exit 1 with a "run --regenerate" hint. --strict is an alias reserved for future main-only CI wiring so the current CI contract stays stable. - --regenerate (deprecated alias --fix, still used by auto_merge.py) is the only writing mode, and now applies the same exactly-one assertion before writing so it can no longer launder a duplicated count line into a green tree. Count-line scanning hardened against the refine-r1 adversarial lens evasions: line-anchored and tolerant of spacing/colon-placement variants; labels restricted to exact ASCII Node/Shell/Python so a homoglyph label is MALFORMED instead of invisible; fenced blocks and HTML comments masked out (offset-preserving) so a format example is neither a duplicate nor a substitute for the real line; regeneration rewrites by match span in canonical form. Error text now shows the canonical format the matcher prefers instead of one it rejected. Preserved: exactly-one/dup detection (#679), vacuous zero-file exit 2, which now also guards the writing path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds five more verified defects into the read-only gate fix. Each reproduced as a failing test before the fix. Format-variant evasion (refine-r1 lens C/D): the exact-literal regex missed `**Python (N suites)** :` and `**Python (N suites):**`, so a document could carry two contradictory counts with the gate green. Matching is now line-anchored and tolerant of spacing and both colon placements. Homoglyph evasion (case E): a Cyrillic look-alike in the label was invisible. Labels are restricted to exact ASCII Node/Shell/Python and any other count-line-shaped label is reported MALFORMED instead of skipped. Fence/comment blindness (cases F/G/H): a fenced-only count line satisfied the exactly-one requirement (and auto-correct then rewrote the number inside the documentation example), while a legitimate format example or commented-out line was reported as a duplicate. Fenced blocks and HTML comments are masked out offset-preservingly, so they are neither duplicates nor substitutes for the real line, and regeneration never touches them. Duplicate laundering: the writing path used unbounded re.sub with no exactly-one assertion, rewriting BOTH duplicates and exiting 0. It now runs the same structural validation before writing and rewrites by match span. --repo was accepted and ignored: every git ls-files ran in the process CWD, so --repo at an empty tree reported "[OK] counts match". cwd is threaded into every git call, the target must be a git work tree, and a failed ls-files is exit 2 rather than a silent zero. Vacuous-zero guard was an AND over all three families, so a single-language wipeout was auto-blessed. It is now per family; a deliberate removal is declared by hand-editing that line to 0 suites. Error text now shows the canonical format the matcher prefers instead of one it rejected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Evicted from the merge queue: batch #737 (integrate/q-1785731578) red with every member individually green |
|
Re-admitting to the merge queue. This PR was evicted as collateral damage from a batch dissolution (see state/merge-queue/exceptions.jsonl, kind=batch_red_dissolved) whose stated cause was 'all members individually green but the batch was red'. The batch's actual failure is owned by #712, which wires |
|
Evicted from the merge queue: batch #741 (integrate/q-1785745378) red with every member individually green |
The defect
python tools/verify_test_suite_count.py --checkauto-corrected drift: itwrote
tests/CLAUDE.mdand exited 0.Three consequences:
checkmutated the working tree.ci.ymlruns--checkon every PR, so thegate was structurally fail-open — a PR adding test files was silently patched
rather than blocked.
tests/test_tests_claudemd_drift.pyrewrote a tracked file on every test run(test-pollution).
TDD proof
tests/test_verify_test_suite_count.py::test_check_mode_is_read_only_on_driftwrites a drifted fixture, snapshots the bytes, runs
--check, and asserts the fileis byte-identical afterwards with a non-zero exit. Against the pre-fix tool:
The new contract
--check(default)run --regeneratehint--strict--checktoday. Reserved so a future main-only strict CI step needs no further contract change--regenerate--fix--regenerate(kept sotools/auto_merge.py:105keeps working)--dry-run--regenerate: report, don't writeExit codes:
0clean/regenerated,1drift or broken invariant,2cannot evaluate.Consolidated defects (7 verified)
This PR is the consolidation point for every verified defect in this file. Each was
reproduced as a failing test before the fix.
1.
--checkmutated the tree and exited 0 on drift (plan finding F4) — above.2. Format-variant duplicates evaded the exactly-one assertion
(refine-r1 lens cases C/D). The
#679regex was exact-literal, so**Python (99 suites)** :and**Python (99 suites):**were invisible and adocument could carry two contradictory counts with the gate green. Matching is now
line-anchored and tolerant of spacing and both colon placements.
3. Unicode homoglyph labels were invisible (case E). A Cyrillic
oin the labelslipped past entirely. Labels are now restricted to exact ASCII
Node/Shell/Python,and a count-line-shaped line with any other label is reported MALFORMED (exit 1)
instead of being silently skipped.
4. Fence / HTML-comment blindness in both directions (cases F/G/H). A count line
that existed only inside a fenced block satisfied the requirement (and the old
auto-correct then rewrote the number inside the documentation example); conversely
a legitimate format example or a commented-out line was reported as a duplicate,
which made documenting the format forbidden by the gate. Fenced blocks and HTML
comments are now masked out offset-preservingly before matching, so they are neither
duplicates nor substitutes for the real line.
5.
--fixlaundered duplicates. The old writing path used unboundedre.subwith no exactly-one assertion: given two Python lines it rewrote both and exited
0, producing a file its own
--checkrejected.--regeneratenow runs the samestructural validation before writing and rewrites by match span in canonical form.
6.
--repowas accepted and ignored (regression lens P1).get_actual_counts()never used its
repo_rootargument — everygit ls-filesran in the process CWD.Pointing
--repoat an empty tree reported[OK] Test suite counts match.cwdisnow threaded into every git call, the target must be a git work tree, and a failed
git ls-filesis exit 2 rather than a silent count of zero.7. The vacuous-zero guard was an AND over all three families (regression lens P1).
Deleting every Python suite left
actual = (2, 3, 0), which was not(0,0,0), so thedoc was rewritten to
0and exited 0. The guard is now per family: any one familyderiving to zero while documented non-zero is exit 2. A deliberate removal is declared
by hand-editing that line to
0 suites, which makes documented and actual agree.Preserved throughout: the
#679exactly-one/duplicate detection and the#674vacuous zero-file fail-closed behavior — both were strengthened, never weakened, and
the latter now guards the writing path too.
Behavior change reviewers must know
A PR that adds or removes test files without regenerating counts now goes RED
instead of being silently auto-corrected. That is the intended stronger contract:
the gate's whole purpose is to catch this, and auto-correction made it impossible to
fail. Lane guidance is documented in
tests/CLAUDE.md:Tests that encoded the defect were rewritten to the new contract, deliberately:
test_check_mode_auto_corrects_drift->test_check_mode_reports_drift_and_hints_regeneratetest_verify_test_suite_count_prepush.py— five drift cases assertedexit 0+AUTO-CORRECT; they now assertexit 1+[DRIFT]+ non-mutation. That file's owndocstring describes it as reproducing the PR feat: CI gate-runability validator (guardrail) #605 drift escape, so restoring exit-1
restores its original intent.
test_tests_claudemd_drift.py— rewritten to snapshottests/CLAUDE.mdbytes andassert the gate did not mutate them.
test_check_mode_passes_when_counts_matchdocumented Shell as 1 suite while its ownfixture derived 2 (
test_a.test.shmatches bothtests/*.test.shandtests/test_*.shand is counted twice). Auto-correction had been masking the wrongexpectation; the fixture is renamed to match exactly one glob.
CI
.github/workflows/ci.ymlis untouched — it keeps invoking--check(lines 110, 202),which is now genuinely blocking. A follow-up can wire
--stricton main-only runs; thealias exists so that change needs no further tool contract change.
Also corrected
tools/CLAUDE.md, whose index entry claimed this gate was"wired into hooks/pre-push-policy.sh as pre-push gate (NOT in CI)" — it has been a
CI step at
ci.yml:110and:202the whole time (plan item A1.5).Not in scope
The lens also flagged positionless label binding (P2: swapping the
**Node/**Pythonlabels between sections still yields exactly one of each, so a count can be written into
the wrong section). Left for a follow-up — it needs section-aware parsing rather than the
line-level hardening this PR does.
🤖 Generated with Claude Code