Skip to content

guard: verify_test_suite_count --check becomes read-only fail-closed (A1 gate-fix) - #719

Merged
matt82198 merged 3 commits into
mainfrom
guard/count-check-no-write
Aug 3, 2026
Merged

matt82198 merged 3 commits into
mainfrom
guard/count-check-no-write

Conversation

@matt82198

Copy link
Copy Markdown
Owner

The defect

python tools/verify_test_suite_count.py --check auto-corrected drift: it
wrote tests/CLAUDE.md and exited 0.

Three consequences:

  1. A mode named check mutated the working tree.
  2. Count drift could never fail CI. ci.yml runs --check on every PR, so the
    gate was structurally fail-open — a PR adding test files was silently patched
    rather than blocked.
  3. tests/test_tests_claudemd_drift.py rewrote a tracked file on every test run
    (test-pollution).

TDD proof

tests/test_verify_test_suite_count.py::test_check_mode_is_read_only_on_drift
writes a drifted fixture, snapshots the bytes, runs --check, and asserts the file
is byte-identical afterwards with a non-zero exit. Against the pre-fix tool:

AssertionError: b'...ell (13 suites)**...' != b'...ell (2 suites)**...'
  : --check MUST be read-only; tests/CLAUDE.md was mutated by a check-mode run

The new contract

Mode Behavior
--check (default) READ-ONLY validation. Never writes. Drift = exit 1 + run --regenerate hint
--strict Exact alias for --check today. Reserved so a future main-only strict CI step needs no further contract change
--regenerate The only writing mode
--fix Deprecated alias for --regenerate (kept so tools/auto_merge.py:105 keeps working)
--dry-run With --regenerate: report, don't write

Exit codes: 0 clean/regenerated, 1 drift or broken invariant, 2 cannot evaluate.

Consolidated defects (7 verified)

This PR is the consolidation point for every verified defect in this file. Each was
reproduced as a failing test before the fix.

1. --check mutated the tree and exited 0 on drift (plan finding F4) — above.

2. Format-variant duplicates evaded the exactly-one assertion
(refine-r1 lens cases C/D). The #679 regex was exact-literal, so
**Python (99 suites)** : and **Python (99 suites):** were invisible and a
document could carry two contradictory counts with the gate green. Matching is now
line-anchored and tolerant of spacing and both colon placements.

3. Unicode homoglyph labels were invisible (case E). A Cyrillic o in the label
slipped past entirely. Labels are now restricted to exact ASCII Node/Shell/Python,
and a count-line-shaped line with any other label is reported MALFORMED (exit 1)
instead of being silently skipped.

4. Fence / HTML-comment blindness in both directions (cases F/G/H). A count line
that existed only inside a fenced block satisfied the requirement (and the old
auto-correct then rewrote the number inside the documentation example); conversely
a legitimate format example or a commented-out line was reported as a duplicate,
which made documenting the format forbidden by the gate. Fenced blocks and HTML
comments are now masked out offset-preservingly before matching, so they are neither
duplicates nor substitutes for the real line.

5. --fix laundered duplicates. The old writing path used unbounded re.sub
with no exactly-one assertion: given two Python lines it rewrote both and exited
0, producing a file its own --check rejected. --regenerate now runs the same
structural validation before writing and rewrites by match span in canonical form.

6. --repo was accepted and ignored (regression lens P1). get_actual_counts()
never used its repo_root argument — every git ls-files ran in the process CWD.
Pointing --repo at an empty tree reported [OK] Test suite counts match. cwd is
now threaded into every git call, the target must be a git work tree, and a failed
git ls-files is exit 2 rather than a silent count of zero.

7. The vacuous-zero guard was an AND over all three families (regression lens P1).
Deleting every Python suite left actual = (2, 3, 0), which was not (0,0,0), so the
doc was rewritten to 0 and exited 0. The guard is now per family: any one family
deriving to zero while documented non-zero is exit 2. A deliberate removal is declared
by hand-editing that line to 0 suites, which makes documented and actual agree.

Preserved throughout: the #679 exactly-one/duplicate detection and the #674
vacuous zero-file fail-closed behavior — both were strengthened, never weakened, and
the latter now guards the writing path too.

Behavior change reviewers must know

A PR that adds or removes test files without regenerating counts now goes RED
instead of being silently auto-corrected.
That is the intended stronger contract:
the gate's whole purpose is to catch this, and auto-correction made it impossible to
fail. Lane guidance is documented in tests/CLAUDE.md:

When your lane adds or removes a test suite, run
python tools/verify_test_suite_count.py --regenerate and commit the updated
counts in the same PR.

Tests that encoded the defect were rewritten to the new contract, deliberately:

  • test_check_mode_auto_corrects_drift -> test_check_mode_reports_drift_and_hints_regenerate
  • test_verify_test_suite_count_prepush.py — five drift cases asserted exit 0 +
    AUTO-CORRECT; they now assert exit 1 + [DRIFT] + non-mutation. That file's own
    docstring describes it as reproducing the PR feat: CI gate-runability validator (guardrail) #605 drift escape, so restoring exit-1
    restores its original intent.
  • test_tests_claudemd_drift.py — rewritten to snapshot tests/CLAUDE.md bytes and
    assert the gate did not mutate them.
  • test_check_mode_passes_when_counts_match documented Shell as 1 suite while its own
    fixture derived 2 (test_a.test.sh matches both tests/*.test.sh and
    tests/test_*.sh and is counted twice). Auto-correction had been masking the wrong
    expectation; the fixture is renamed to match exactly one glob.

CI

.github/workflows/ci.yml is untouched — it keeps invoking --check (lines 110, 202),
which is now genuinely blocking. A follow-up can wire --strict on main-only runs; the
alias exists so that change needs no further tool contract change.

Also corrected tools/CLAUDE.md, whose index entry claimed this gate was
"wired into hooks/pre-push-policy.sh as pre-push gate (NOT in CI)" — it has been a
CI step at ci.yml:110 and :202 the whole time (plan item A1.5).

Not in scope

The lens also flagged positionless label binding (P2: swapping the **Node/**Python
labels between sections still yields exactly one of each, so a count can be written into
the wrong section). Left for a follow-up — it needs section-aware parsing rather than the
line-level hardening this PR does.

🤖 Generated with Claude Code

matt82198 and others added 3 commits August 2, 2026 21:27
--check auto-corrected drift: it WROTE tests/CLAUDE.md and exited 0. That made a
check mode mutate the tree, made count drift structurally unable to fail CI
(fail-open gate), and made tests/test_tests_claudemd_drift.py rewrite a tracked
file on every test run.

New contract:
- --check (default) and --strict are READ-ONLY validation, never write. Drift is
  exit 1 with a "run --regenerate" hint. --strict is an alias reserved for future
  main-only CI wiring so the current CI contract stays stable.
- --regenerate (deprecated alias --fix, still used by auto_merge.py) is the only
  writing mode, and now applies the same exactly-one assertion before writing so
  it can no longer launder a duplicated count line into a green tree.

Count-line scanning hardened against the refine-r1 adversarial lens evasions:
line-anchored and tolerant of spacing/colon-placement variants; labels restricted
to exact ASCII Node/Shell/Python so a homoglyph label is MALFORMED instead of
invisible; fenced blocks and HTML comments masked out (offset-preserving) so a
format example is neither a duplicate nor a substitute for the real line;
regeneration rewrites by match span in canonical form. Error text now shows the
canonical format the matcher prefers instead of one it rejected.

Preserved: exactly-one/dup detection (#679), vacuous zero-file exit 2, which now
also guards the writing path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds five more verified defects into the read-only gate fix. Each reproduced as
a failing test before the fix.

Format-variant evasion (refine-r1 lens C/D): the exact-literal regex missed
`**Python (N suites)** :` and `**Python (N suites):**`, so a document could carry
two contradictory counts with the gate green. Matching is now line-anchored and
tolerant of spacing and both colon placements.

Homoglyph evasion (case E): a Cyrillic look-alike in the label was invisible.
Labels are restricted to exact ASCII Node/Shell/Python and any other
count-line-shaped label is reported MALFORMED instead of skipped.

Fence/comment blindness (cases F/G/H): a fenced-only count line satisfied the
exactly-one requirement (and auto-correct then rewrote the number inside the
documentation example), while a legitimate format example or commented-out line
was reported as a duplicate. Fenced blocks and HTML comments are masked out
offset-preservingly, so they are neither duplicates nor substitutes for the real
line, and regeneration never touches them.

Duplicate laundering: the writing path used unbounded re.sub with no exactly-one
assertion, rewriting BOTH duplicates and exiting 0. It now runs the same
structural validation before writing and rewrites by match span.

--repo was accepted and ignored: every git ls-files ran in the process CWD, so
--repo at an empty tree reported "[OK] counts match". cwd is threaded into every
git call, the target must be a git work tree, and a failed ls-files is exit 2
rather than a silent zero.

Vacuous-zero guard was an AND over all three families, so a single-language
wipeout was auto-blessed. It is now per family; a deliberate removal is declared
by hand-editing that line to 0 suites.

Error text now shows the canonical format the matcher prefers instead of one it
rejected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@matt82198 matt82198 added merge-queue Queued for the merge-queue advancer daemon merge-priority Jump the merge queue labels Aug 3, 2026
@matt82198 matt82198 added queue-rejected Evicted from the merge queue (red or culprit) and removed merge-queue Queued for the merge-queue advancer daemon labels Aug 3, 2026
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: batch #737 (integrate/q-1785731578) red with every member individually green

@matt82198

Copy link
Copy Markdown
Owner Author

Re-admitting to the merge queue. This PR was evicted as collateral damage from a batch dissolution (see state/merge-queue/exceptions.jsonl, kind=batch_red_dissolved) whose stated cause was 'all members individually green but the batch was red'. The batch's actual failure is owned by #712, which wires import_resolution_check into ci.yml as a hard 'Sibling import guard' step while the checker on main still flags the sanctioned sys.path idiom in tools/merge_queue.py:68-69 -- the fix for that is #724, which is not yet landed. #712 stays queue-rejected until #724 merges; this PR is individually green and goes back in the line.

@matt82198 matt82198 added merge-queue Queued for the merge-queue advancer daemon and removed queue-rejected Evicted from the merge queue (red or culprit) labels Aug 3, 2026
@matt82198 matt82198 added queue-rejected Evicted from the merge queue (red or culprit) and removed merge-queue Queued for the merge-queue advancer daemon labels Aug 3, 2026
@matt82198

Copy link
Copy Markdown
Owner Author

Evicted from the merge queue: batch #741 (integrate/q-1785745378) red with every member individually green

@matt82198

Copy link
Copy Markdown
Owner Author

Re-admitting: evicted by the premature dissolution of batch #741, whose only fault was being evaluated seconds after creation while GitHub had not yet created its windows check run. Fixed in #742 (merged) -- dissolution now requires positive evidence of failure, not mere absence.

@matt82198 matt82198 added merge-queue Queued for the merge-queue advancer daemon and removed queue-rejected Evicted from the merge queue (red or culprit) labels Aug 3, 2026
@matt82198
matt82198 merged commit fd41169 into main Aug 3, 2026
12 checks passed
@matt82198
matt82198 deleted the guard/count-check-no-write branch August 3, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-priority Jump the merge queue merge-queue Queued for the merge-queue advancer daemon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant