asvs: the engine-side half of the ASVS gate (#1202) + the tally (#1204) and residual-citation (#1205) lints - #302
Merged
Merged
Conversation
…ne-side (#1202) The ASVS gate lives in the vault and measures THIS tree. It makes three claims about the engine, and only one of them needs the private assessment record: 1. the recorded evidence anchors still resolve -- needs the record 2. scripts/asvs/scorecard.py exists at that literal path -- public 3. it RUNS, as a bare script, on a bare interpreter -- public Claims 2 and 3 were checked by neither repo. asvs-scorecard.yml states the invariant in as many words -- "No install step and no dependency: the verifier is stdlib-only ... precisely so this job cannot rot on a lockfile it does not own" -- and nothing enforced it. asvs-verifier-drift.yml's preflight job goes further and executes the INCOMING engine copy as `python ../engine/scripts/asvs/scorecard.py` from the vault checkout, so sys.path[0] is scripts/asvs and a first-party import has nothing to resolve against either. This repo's own tests could not see it: they import the module with the repo root on sys.path and the full extras installed, which is not the invocation that matters. Measured. With a USED `import httpx` appended to scripts/asvs/scorecard.py, every pre-existing gate returned a verdict IDENTICAL to its unmutated verdict -- ruff check PASS to PASS, ruff format unchanged, mypy unchanged (it types messagefoundry and messagefoundry_webconsole, not scripts/), tests/test_asvs_scorecard.py 54 tests PASS to PASS. `python -I -S -c "import httpx"` returns 1, so the mutation models the vault runner rather than a lint opinion. Breaking it would not red the gate. The mirror opens as a DRAFT when the incoming verifier does not run and a draft is never merged, so the vault would keep verifying its record with the previous copy -- the recurring condition (six hand-made mirror commits, one 326 lines behind) that splitting the drift workflow out was written to end. The guard: existence at the hardcoded path, a stdlib-only ast scan over every import including deferred ones, and a `python -I -S <script> --help` run from an unrelated working directory. Four injected violations were each confirmed to LAND by digest before their red was believed; a third-party import DEFERRED into a function body reds only the ast scan and passes the smoke run, which is why the scan is load-bearing. Both a positive and a negative control are permanent tests, and the module has its own CLI so it prints the inventory it scanned rather than only a verdict. Trigger checked rather than assumed, using the live noncode regex out of ci.yml under real grep -E: a PR touching only scripts/asvs/scorecard.py is code=true, so is one touching only the guard's own file (the gate is inside its own trigger), and a docs/SECURITY.md-only PR correctly stays code=false. No new workflow and no new required context: the trigger was already right, the check was what did not exist. Claim 1 is out of scope here and cannot be fixed here -- it needs the private record, and no credential-free cross-repo trigger exists. The vault's daily cron stays its only authority.
…he corpus (#1204) ADR 0156 made the ASVS count computed. It did not stop anyone writing one down: 44 documents assert a whole-corpus tally, roughly fifty distinct tallies exist, and approximately one is correct. FORWARD-ONLY, NOT A SWEEP. A one-time banner sweep was tried on the assessment corpus: about 850 net lines, its own repair commit, and the defect regenerated inside four days because eight new documents were written after it. Existing tallies are RECORDED in scripts/docs/asvs_tally_baseline.txt, not edited. The baseline stores an occurrence count per claim, so adding a copy of a grandfathered tally fails and removing one fails until the entry comes down with it. It can only shrink. THE IDIOM SET WAS REBUILT FROM THE CORPUS, and the previous attempt's recall was reproduced before anything replaced it. That attempt matched two shapes, an N / N / N / N tuple and "N of 345". Re-implementing exactly those two reds 2 of the 8 chapter reports -- matching the register's independent finding that it missed 6 of 8. The five idioms here red 8 of 8. Two are shapes the old set could not see at all: a Markdown TABLE ROW whose cells close to the corpus total, which is the shape the one CORRECT record is written in, and an ARITHMETIC assertion that closes to it. A third is markdown-emphasis tolerance: one chapter report writes its tally with the verdict words in backticks, and a bare \s* walks past it. THE DISCRIMINATOR is that a run of integers is a tally only when it SUMS TO THE PINNED CORPUS TOTAL. That is what keeps out HL7 field notation, X12 transaction sets, status-code lists and config defaults -- the previous attempt produced 73 false hits on those. Corpus STRUCTURE is deliberately not flagged: the requirement count and the level split are pinned constants that cannot go stale, and flagging them is what made the previous attempt red the method document's own worked example, which is a literal letter rather than a count. PROVED ON REAL FILES, BOTH DIRECTIONS. Eight mutations, one per idiom, appended to a real document here: all eight RED, each confirmed to have LANDED by digest first. Five controls through the identical path: all five stayed GREEN. Every one of those shapes is now a permanent test. TRIGGER. The defect is written into DOCUMENTS, so the PR carrying it is usually docs-only -- and a docs-only PR sets code=false and skips the whole pytest suite. A guard reachable only through pytest would not run on the shape it exists to police, which is why two guards in ci.yml are ungated. Measured with the live globs and the live regex: a docs-only PR FIRES the new workflow and ran no pytest at all before, and there was no tally lint on origin/main to run. The workflow carries its own file, the lint and the baseline in its own trigger. It is NOT a required check: it is paths-filtered, and a required check that cannot report wedges the PR forever. Also generalises the mirror contract added for #1202: the stdlib-only rule is stated once over a LIST of mirrored tools rather than for the verifier alone, and the tally lint is on that list before it acquires a dependency rather than after someone finds the mirror will not run. The four #1202 mutations were re-run against the refactored guard and still red. SCOPE. The engine can only lint documents it holds -- 4 files carry 23 tallies here, all baselined. The bulk of the corpus is in the assessment repo and can only be linted there; the tool is stdlib-only and mirrorable for that reason. Retiring the existing 44 documents is explicitly NOT part of this: that is a sweep, it was costed and refused, and it is the owner's decision.
wshallwshall
changed the title
docs/BACKLOG.md only, and both sides are pure appends to the tail -- verified rather than assumed, because a mechanical keep-both merge can restore a defect the branch removed: my diff vs merge-base +103 -0 their diff vs merge-base +53 -0 Zero deletions on either side, and each side's first 8,715 lines are the merge base verbatim, so keeping both cannot resurrect anything. The resolution was CONSTRUCTED as base + their append + mine rather than hand-edited, then checked: every line of both sides present in order, each of the four item headings occurring exactly once, and a sentinel that exists in neither side absent. 473 items, banner invariant OK. Ledger gate and the tally lint both green on the merged tree.
Half the ASVS record is prose nothing checks. Measured at the pinned record: 2,052 file:line citations inside `residual` across 247 cells, 1,086 of them bare basenames. A sample measured 44.9% stale. The demonstration case, verified against public engine code at this commit: cell 6.3.3's gated evidence anchor for the exposure check points at messagefoundry/__main__.py:1125, and that line reads `instance_exposed = not settings.api.is_loopback or ...` -- correct, and the only occurrence of that statement in the file. The prose in the SAME CELL cites __main__.py:1917, which is the middle of a warning string about the browser console. The gated half is right and the read half is wrong, about one control, in one cell. BULK PROMOTION INTO GATED ANCHORS IS NOT ATTEMPTED. Costed and refused: roughly a thousand hand-authored tokens, it doubles the gated surface, and it makes DELETE THE CITATION the cheapest compliant act on the basenames. THE KEY IS THE DESIGN. A citation is identified by cell id + field + FILE, never by its line number. The population of (cell, field, file) pairs cannot grow; the occurrence count in the frozen baseline stops an existing pair growing either, so the total can only shrink; and REPAIRING A STALE LINE IS FREE and needs no baseline edit, while adding a citation is refused and deleting one costs an edit. That ordering is deliberate -- it makes correction the cheapest compliant act, the exact inverse of the incentive the rejected approach created. IT REFUSES TO SUCCEED ON AN EMPTY SCAN. This tool runs where the data is, which is not this repo, so the dangerous failure is silent: pointed at a renamed field or a document with no cells, "no new citations" and "nothing was examined" would share an exit code. Both exit 2, and the scan inventory prints before the verdict. PROVED AT SCALE against the real record, locally, nothing committed: baseline of 1,233 claims over 2,052 occurrences; a new path-qualified citation REDs, a new bare basename REDs, deleting a citation without lowering the baseline REDs, a renamed field and a missing baseline both exit 2 -- and repairing 6.3.3's stale line stays GREEN with no baseline edit. Each mutation confirmed landed first. 43 fixture tests here, both directions on every shape. WHERE IT RUNS. The record exists only in the assessment repo, so unlike the tally lint this has NO engine-side subject. What ships here is the tool -- stdlib-only and mirrorable on the ADR 0156 section 7 footing, now on the mirrored-tools list -- plus fixture tests. The baseline and the wiring belong beside the data. NO ENGINE WORKFLOW IS ADDED, deliberately: a gate pointed at nothing is the failure this tool refuses to commit itself. Also: I expected a cell-id-keyed baseline to leak verdicts and MEASUREMENT REFUTED IT -- cited residuals occur across every verdict class including most passing ones. That argument is withdrawn. The baseline still belongs beside the data, on the stronger ground that a frozen list is unverifiable from a repo that cannot see what it grandfathers. Checked before committing, as for #1204: the new file under scripts/ matches 0 of 276 ASVS absence patterns, with a live control and an impossible-pattern control (scorecard.py::_python_sources scans scripts/**/*.py, so a new file there can flip an absence claim to FALSE).
wshallwshall
changed the title
tests/test_cutover_slug_rot.py caps present-tense mirror/private-repo prose at 54 hits. Two comment lines in this file pushed it to 56. Both meant "a copied tool", not "this repo is a mirror of a private source" -- but that guard's own docstring says a regex cannot separate those senses, and a coarse regex over prose is the right trade there. So the wording changes and the ceiling does not. Raising a ceiling so your own additions fit is the repair that makes the guard meaningless. Fix authored before the stream stood down; committing it now.
wshallwshall
enabled auto-merge (squash)
August 9, 2026 14:02
…04/#1205) docs/BACKLOG.md only -- the three-way append point across #302, #304 and #305. Both sides characterised BEFORE resolving, because "both sides append" is a claim to check and a keep-everything merge would restore anything the other side removed: mine vs base +160 -0 theirs vs base +63 -0 Zero deletions on either side, and each side's first 8,826 lines are the merge base verbatim, so concatenation is licensed. The resolution was CONSTRUCTED as base + theirs + mine, never hand-edited: 8,826 + 63 + 160 = 9,049 lines, re-read from disk and confirmed byte-identical to the candidate. PROVED POISONED-FIRST. A subsequence check cannot be broken by APPENDING, so passing subsequence checks is not evidence on its own. Five deliberately corrupted candidates were run through the same checks and every one was caught, naming which check fired: sentinel injected -> no-sentinel, exact-length a base line dropped -> both subsequences, exact-length, base-retained conflict marker left in -> no-conflict-markers, exact-length their append omitted -> theirs-subsequence, exact-length duplicated item heading -> exact-length, no-duplicate-item-numbers The exact-length check is the one that catches every append case, which is precisely the class the subsequence checks are blind to. Only then the real candidate: all seven checks pass, 281 item headings all distinct, each of #1199/#1201/#1202/#1204/#1205/#1206 occurring exactly once. Ledger gate rc=0, banner invariant OK at 476 items, and the #1204 tally lint green on the merged tree.
wshallwshall
disabled auto-merge
August 10, 2026 17:42
…l merge Session A's resolve and Session A's measurements. Committed by Session B only because of an ownership constraint, stated here so the next reader does not wonder why the sole writer did not write it: BACKLOG numbers 1202, 1204 and 1205 are allocated to THIS worktree (C:/Users/Scott/Code/MessageFoundry-asvs-ci-hygiene), and ledger_check's owns() compares the claim's `worktree` field casefolded against the repo path and never consults the branch -- so the pre-commit gate refuses this commit from any other path, including from A's own worktree. Nobody used --no-verify. WHY THIS IS NOT A LINE-LEVEL MERGE, which is the thing a future reader most needs so they do not "simplify" it back. Measured on this branch against origin/main 48f8712: live archive namespace in BOTH files origin/main 248 236 484 -- this worktree (42050cd) 281 195 476 0 MECHANICAL keep-both-sides 289 236 525 38 <-- the trap this commit 251 236 487 0 41 items differ between the branch and main. THIRTY-EIGHT OF THEM ARE ITEMS MAIN HAS SINCE ARCHIVED. A keep-both-sides merge resurrects every one into the live ledger while leaving it in the archive: duplicated across both files, the namespace inflated 484 -> 525, and each of those 38 reading OPEN again after being closed. GIT RAISES NO CONFLICT MARKER, because both sides added lines near the tail. Only THREE of the 41 are genuinely new: 1202, 1204, 1205. So both files are taken from main WHOLESALE and exactly those three are spliced back, before #1208 (item headings ascend even though the file as a whole does not). A's script asserts its own result: +3 headings exactly, none lost, no duplicates. THE ARCHIVE HALF IS DELIBERATE AND EXPLICIT, not incidental. The branch had touched BACKLOG-CLOSED.md zero times (main-only 41, branch-only 0), so taking main's copy loses nothing and is what a merge would do anyway -- but doing it as a stated step is what makes the branch self-consistent BEFORE it lands. Without it the tree reads 251/195/446 and a pre-merge namespace check fails against a resolve that is actually correct: a false red aimed at the right answer, which is the kind that gets a correct fix reverted. Re-derived from origin/main at 48f8712 (post-#326) rather than replaying a patch cut against ea75c37, because main moved underneath it. One number corrected in passing: the mechanical figure was first circulated as 286/522. It is 289/525 -- the smaller version credited the mechanical merge with telling the 3 good items from the 38 bad ones, which is precisely the one thing "mechanical" means it cannot do, and it erred in the direction that understates the damage. Verified here independently rather than quoted. Expect test_link_resolution to red branch-locally: main's ledger cites a test file main has and this older branch does not. Not in the spliced items; A simulated the merge and the file is in the merged tree, so it resolves on landing. Taking a newer ledger onto a stale tree necessarily imports references the tree cannot satisfy yet. Commit only -- deliberately not pushed. This worktree's HEAD is two commits ahead of PR #302's head and those two are on no remote, so a push from here would put them on the PR. That is the Coordinator's call, not a side effect of this commit. They are anchored at refs/rescue/unpushed/asvs-ci-hygiene-42050cd4 and 8391699's content is already on origin/main.
… the branch, provably git conflicts on docs/BACKLOG.md here and the conflict is BENIGN. The merge-base is 8391699 and BOTH sides rewrote that file wholesale relative to it -- main via the archive move, this branch via the #302 resolve -- so git cannot see that the branch rewrite ALREADY CONTAINS main's. It has no way to know; it is comparing text, not sets. Resolved `--ours` (the branch resolve). That is normally the mechanical move that goes wrong, and it is correct here for one specific reason, re-derived on the MERGE RESULT rather than assumed from the pre-merge state: THE BRANCH LEDGER IS A CLEAN SUPERSET OF MAIN'S. main items missing from the merge result : 0 shared items whose text differs : 0 extra items : exactly [1202, 1204, 1205] archive file byte-identical to main's : yes counts: live 251 | archive 236 | namespace 487 | in BOTH files 0 Checked item-by-item, not by count -- 251 is reachable by many wrong routes (drop one of main's and add four; reword one and keep the total), and only the per-item comparison rules those out. The count is the weakest of the four lines above. The archive file did NOT conflict and needed no resolution: the #302 resolve had already taken main's copy wholesale, so the two sides agreed. Deliberately NOT verified with `git merge-tree --write-tree`. It prints a VALID tree OID even when the merge conflicts, signalling only in the return code -- so an output-shape check reports CLEAN on a conflicted merge. Found by Session A and reproduced by the Coordinator today. The rc must also be read without a pipe in between, since a pipe clobbers $LASTEXITCODE. This commit was checked by measuring the resulting TREE instead, which sidesteps the instrument entirely.
…ding on quoted tallies
The tally lint red on docs/BACKLOG.md:4049, and it was a FALSE POSITIVE. The
flagged text is #1012's closing banner quoting the falsification transcript
that PROVES the bug that item fixed:
`scanned 3 cells (1 pass / 0 partial / 0 fail / 0 na / 1 unverified)`
Three synthetic cells, inside a code span, introduced as "reproduces the
defect in miniature", and the sentence's own point is that the components
DISAGREE with the stated total. A lint against stale hard-coded tallies fired
on a demonstration that a tally was WRONG, inside the item that fixed broken
tallies.
THE RULE IS EMPHASIS VERSUS ENCLOSURE, and it is narrow on purpose:
24 `pass`, 15 `partial`, 5 `na` numbers OUTSIDE the spans; backticks
decorate individual words. A claim.
COUNTED, unchanged.
`scanned 3 cells (1 pass / 0 partial)` number and word inside the SAME span.
The document is QUOTING. SKIPPED.
Only counted-verdict pairs are subject to it. The precedent is already house
style: CLAUDE.md section 11 permits quoting a glyph as a token in backticks
and calls that code rather than decoration.
THREE FIXES I REJECTED, because each would have been worse than the bug:
Rewording main's prose. That deletes the falsification transcript, which is
the thing that makes #1012's closure checkable -- damaging a correct record
to satisfy a broken instrument.
Grandfathering line 4049. The baseline header says the list is FROZEN and
may only SHRINK. Growing it would break the one invariant the file states
about itself, and a grandfathered entry ASSERTS "this is a real hard-coded
tally we tolerate", which is false. The baseline is untouched at 18 claims.
Stripping code spans wholesale before matching, which was my own first
instinct and is wrong twice over: it would blind ARITHMETIC, whose OWN
documented examples are backticked real tallies (`195 + 89 + 0 + 61 = 345`),
and it would delete the very words that make the emphasis case matchable.
WHY THIS IDIOM AND NOT THE OTHERS: LABELLED_VERDICTS is the only one of the
five with no corpus-total sum test. SLASH_RUN, TABLE_ROW and ARITHMETIC all
require the numbers to close to 345, which is what keeps a 3-cell transcript
out of them. I3 needs only three distinct counted verdict classes on a line,
so a quoted transcript satisfies it. The quotation boundary is the cheapest
correct discriminator that does not weaken the real detections.
VERIFIED IN BOTH DIRECTIONS, which is the whole risk -- a fix that stops the
false positive by blinding the lint is not a fix:
plain tally, abbreviated tally, backtick-EMPHASISED tally, backticked
ARITHMETIC tally -> all still fire
the wholly-quoted transcript -> no longer fires
line 4049 of main's ledger -> no idioms
full lint with the real baseline: 23 tallies, 23 grandfathered, 0 NEW, exit 0
44 tests pass (40 pre-existing + 4 new)
WATCHED FAIL: neutering the quotation predicate reds the new test AND
test_this_repo_is_clean_against_its_own_frozen_baseline -- so the repo-clean
gate, not just my own assertion, depends on it.
Found while resolving #302: taking main's ledger wholesale imported the quoted
transcript, and the lint met it on first contact. It IS proving something on
arrival, just not what it reported -- that its matcher had no quotation
boundary, learned before the baseline grew around it.
wshallwshall
added a commit
that referenced
this pull request
Aug 11, 2026
#1220 landed while #1221 was in flight, and both insert at the same
docs/BACKLOG.md line, so the tail conflicts. Resolved keep-both, #1220
then #1221, in numeric order.
KEEP-BOTH HAS BEEN THE WRONG ANSWER TWICE TONIGHT, so I checked the
property that makes it right here rather than reaching for it:
conflict blocks : 1
OURS carries : {1221} only
THEIRS carries : {1220} only
intersection : EMPTY
Two disjoint appends of different items, neither referencing the other's
text. That is a different situation from #302, where keep-both would have
resurrected 38 archived items, and from #1220's own subject, where it
ships seam 19 describing two contract changes. The phrase is not the
answer; the property is.
RE-DERIVED ON THE RESULT rather than trusting the resolution, as the
coordinator asked:
live 250 | archive 236 | namespace 486 | in BOTH files 0
duplicate headings in live: 0
#1220 and #1221 both live, neither archived
backlog_status_check: 486 items, each declaring exactly one status
ledger_check: clean
doc guards: 48 passed (link resolution, banner status, dast claims)
484 -> 486, nothing duplicated, which is what a correct two-append merge
must produce.
My own earlier advice to batch these was semantically right and
mechanically incomplete: I checked whether either item invalidated the
other's claims and never asked whether their HUNKS collided. They insert
at the same line. Two PRs bought the same parallelism at the cost of
exactly this one resolve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three engine-side pieces of the ASVS tracking rework: the half of the vault gate that can live here
(#1202), a forward-only lint refusing a new hard-coded ASVS tally (#1204), and a forward-only
lint refusing a new
file:linecitation in scorecard prose (#1205).A note on evidence throughout: the assessment corpus is a private security record and its
tallies are the posture, so no corpus text, no per-document tally and no verdict distribution
appears in this repo or in this description. Where the measurement is over that corpus, the
aggregate is reported and the line-level output is not.
Part 1 -- #1202: the engine-side half of the ASVS gate
asvs-scorecard.ymllives in the vault, fires on vault paths, and measures the engine tree.The gate makes three claims about this tree, and they split on whether they need the private
record:
scripts/asvs/scorecard.pyexists at that literal pathClaim 1 admits no engine-side trigger and none is added. Both inputs must be co-located, only
the vault holds the record, and GitHub's only credential-free cross-repo trigger is polling -- which
is already the cron.
repository_dispatchneeds a credential, and a credential in the public repothat reaches the private vault is a worse trade than the latency it buys, for a detector that is
post-merge either way (the gate checks out engine
main). That half is vault-side.Claims 2 and 3 belong here. The vault runs the verifier twice and neither job installs anything:
asvs-scorecard.ymlafter a baresetup-python, andasvs-verifier-drift.yml'spreflight, whichruns the incoming engine copy from the vault checkout so
sys.path[0]isscripts/asvsand afirst-party import has nothing to resolve against. The invariant is stated in
asvs-scorecard.yml'sown comment -- "No install step and no dependency: the verifier is stdlib-only ... precisely so this
job cannot rot on a lockfile it does not own" -- and was enforced nowhere. This repo's tests import
the module with the repo root on
sys.pathand full extras installed, so the invocation that matterswas never exercised.
Breaking it does not red the gate. The mirror opens as a draft when the incoming verifier
will not run, and a draft is never merged, so the vault keeps verifying its record with the previous
copy. That is the recurring condition -- six hand-made mirror commits, one found 326 lines behind --
that splitting
asvs-verifier-drift.ymlout was written to end.The old configuration was blind
A used
import httpxappended toscripts/asvs/scorecard.py(httpx is a real engine dependency,so not dead code a linter rejects anyway). Every pre-existing gate returned a verdict identical to
its unmutated one:
The new guard reds, and each injection landed
Applied, confirmed present by sha256 change, run, restored byte-exact:
B is why the
astscan is load-bearing: a--helpsmoke test is structurally blind to adeferred import. Re-run against the refactored guard in Part 2 and still red.
The first run of that harness reported
INJECTION LANDED: Falsefor A-C: the tree is CRLF undercore.autocrlf=trueand the\nanchors matched nothing. The confirm-it-landed step is the onlyreason that was not read as "the guard cannot see it."
Part 2 -- #1204: a forward-only lint refusing a NEW hard-coded ASVS tally
ADR 0156 made the count computed. It did not stop anyone writing one down: 44 documents assert a
whole-corpus tally, roughly fifty distinct tallies exist, and approximately one is correct.
The idiom set was rebuilt from the corpus, and the old recall reproduced first
The previous attempt matched two shapes: an
N / N / N / Ntuple andN of 345. Re-implementingexactly those two and running them over the corpus reds 2 of the 8 chapter reports -- which
matches the register's independent finding that it missed 6 of 8, so the reproduction is calibrated
against a number I did not choose. The five idioms here red 8 of 8:
SLASH_RUNLABELLED_VERDICTSAGAINST_TOTALTABLE_ROWARITHMETICTABLE_ROWis the shape the one correct record is written in, which is why a grep for thecorpus's own tuple idiom does not find it. Markdown-emphasis tolerance is worth naming separately:
one chapter report writes its tally with the verdict words in backticks, and a detector using a bare
\s*walks straight past it -- that alone was 1 of the 8.The discriminator is that a run of integers is a tally only when it sums to the pinned corpus
total. That is what keeps out HL7 field notation, X12 transaction sets, status-code lists and
config defaults; the previous attempt produced 73 false hits on exactly those. Corpus structure
is deliberately not flagged -- the requirement count and the level split are pinned constants that
cannot go stale, and flagging them is what made the previous attempt red the method document's own
worked example, which is a literal letter rather than a count.
Acceptance, measured: 8 of 8 chapter reports red (V1, V3, V4, V6, V10, V13, V15, V16); the old
two idioms red 2 of 8. Of the documents the old idioms missed, 14 assert a hard-coded verdict or
survey tally (the 6 chapter reports plus 8 others) under the definition above. The register's
figure was 13, measured at an earlier ref and against a looser definition that also counted
corpus-structure mentions; at today's ref that looser definition gives 20, of which 4 documents
postdate the register. Both numbers are reported rather than one being tuned to match -- the
checkable acceptance criterion is the 8 of 8 and the reproduced 2 of 8.
Proved on real files, both directions
Eight mutations, one per idiom shape, appended to a real document in this repo -- each confirmed to
have landed by digest, then reverted byte-exact:
Every one of those shapes is now a permanent test, positive and negative. 53 tests across the two
modules.
Forward-only, and the ratchet
Existing tallies are recorded, not edited, in
scripts/docs/asvs_tally_baseline.txt-- 18 claims,23 occurrences, across 4 files in this repo (
docs/adr/0019-*,docs/Secure_Build_Scorecard_MEFOR.md,docs/BACKLOG.md,docs/archive/backlog/BACKLOG-CLOSED.md; the design document calls these thepublic-repo leaks). A sweep was refused on the record: ~850 net lines, its own repair commit, and the
defect regenerated inside four days.
The baseline stores an occurrence count per claim, so it cannot idle:
The key is
path + idiom + the numbers, never the line number and never the surrounding prose --a baseline invalidated by an edit above it is a list that has to be re-typed, which is the
decaying-budget failure this programme is unpicking elsewhere.
Trigger: the gate must see a docs-only PR
The defect is written into documents, so the PR carrying it is usually docs-only -- and a
docs-only PR sets
code=falseand skips the entire pytest suite. A guard reachable only throughpytest would not run on the shape it exists to police, which is exactly why two guards in
ci.ymlare deliberately ungated. Measured with the live workflow globs and the live
noncoderegex:So a new workflow, not a pytest test alone. It is not a required check (paths-filtered; a
required check that cannot report wedges the PR forever) and it is not in
.github/required-contexts.txt.Why not ci.yml's ungated doc-guards step, which is the natural home:
ci.yml'schangesjob isunder open PR #299 and a second edit to those lines would conflict. The workflow header records the
follow-up -- fold this into the doc-guards list once #299 lands, so one place decides which guards run
on a docs-only PR rather than two.
Where the lint can actually run
Stated plainly, because it is the same shape as claim 1 above. The engine can only lint the
documents it holds. The bulk of the affected corpus is in the assessment repo and can only be
linted there. The tool is stdlib-only for exactly that reason -- it is mirrorable and runs on a bare
interpreter with no install, the same contract
scripts/asvs/scorecard.pyis held to -- but wiring itinto a workflow on that side is a change in that repo, not this one.
Part 1's guard was generalised rather than copied for this: the stdlib-only rule is now stated
once over a list of mirrored tools, and the tally lint is on that list before it acquires a
dependency rather than after someone discovers the mirror will not run. The fix that does not
generalise is the one that comes back.
Retiring the existing 44 documents is explicitly not in scope. That is a sweep; it was costed and
refused; it is the owner's decision.
One safety check worth recording
scorecard.py::_python_sourcesscansmessagefoundry,messagefoundry_webconsole,harnessandscriptsfor*.py, and every absence claim greps that corpus -- so a new file underscripts/can flip an absence claim to FALSE.
tests/test_docs_db_grants.pyrecords this trap in its ownheader. Run before committing each of the two new files under
scripts/: 0 of 276 absencepatterns match, with a control pattern proving the check was live and an impossible pattern proving
it was not matching everything.
Part 3 -- #1205: a forward-only lint refusing a NEW
file:linecitation in scorecard proseHalf the record is prose nothing checks. Measured at the pinned record: 2,052 citations inside
residualacross 247 cells, 1,086 of them bare basenames. A sample measured 44.9% stale.The demonstration case, verified against public engine code
Cell 6.3.3's gated evidence anchor for the exposure check points at
messagefoundry/__main__.py:1125:The prose in the same cell cites
__main__.py:1917-- the middle of a warning string. The gatedhalf is right and the read half is wrong, about one control, in one cell. A reviewer reads the
prose.
A round number made finer
On "1,064 bare basenames that cannot resolve even in principle", resolved against this worktree
(2,950 files, 2,630 distinct basenames): of 1,086 bare basenames, 129 match no file at all, 276
match two or more, and 677 match exactly one. So unresolvable in practice is 405; the other 677
resolve only by luck of there being a single candidate, which the citation format does not guarantee.
Both readings are reported rather than one being quoted.
The key is the design
A citation is identified by
cell id + field + FILE, never by its line number:total can only shrink;
deleting one costs an edit.
That ordering is chosen. The rejected bulk-promotion approach made delete the citation the cheapest
compliant act on ~1,000 basenames; this makes correction the cheapest, which is what you actually
want from someone who has just noticed a citation is wrong. Bulk promotion is not attempted --
~1,000 hand-authored tokens and it doubles the gated surface.
It refuses to succeed on an empty scan
This tool runs where the data is, which is not this repo, so the dangerous failure is the silent one:
pointed at a renamed field or a document with no cells, "no new citations" and "nothing was
examined" would otherwise share an exit code. Both exit 2, and the scan inventory prints before
the verdict.
Proved at scale against the real record (locally; nothing committed)
43 fixture tests, both directions on every shape (including
settings.pywith no line,06:17,127.0.0.1:8765, a version string, and an unknown extension).Where it can run
Unlike the tally lint, this one has no engine-side subject at all -- the record exists only in the
assessment repo. What ships here is the tool (stdlib-only, mirrorable on the ADR 0156 section 7
footing, now on the mirrored-tools list) plus fixture tests. The baseline and the wiring belong beside
the data. No engine workflow is added, deliberately: a gate pointed at nothing is precisely the
failure the tool refuses to commit itself.
A hypothesis I held, and measurement refuted
I expected a cell-id-keyed baseline to leak verdicts, on the reasoning that residuals are a property
of unresolved cells. Measured: cited residuals occur across every verdict class, including a large
majority of the passing ones, and cited-residual cells split close to evenly between pass and
not-pass. The verdict-leak argument does not hold and is withdrawn. The baseline still belongs
beside the data, on the stronger ground that a frozen list is unverifiable from a repo that cannot see
what it grandfathers.
Also measured, not actioned
Two other fields carry
file:line:absence[].mutation(32 citations across 24 cells) and one inevidence[].expect. The field set is a parameter (--field, supporting theabsence[].mutationlist-of-tables form, with a test), but the default stays
residual-- widening the default wouldsilently change what an existing baseline means.
Carried forward, not actioned here
Both routed to the session holding #299;
ci.ymlis untouched by this PR:ci.yml:836's allowlist comment still documents.gitignore/.gitattributesas docs-only..gitignorewas removed from the regex at line 847 by feat(#114): the outbound validate_startup hook, with its ledger banner (train) #327 and the comment was never updated, soit states the opposite of the code -- the third instance of that pattern in this file.
.gitattributesis still allowlisted, and fix(ci): an executable file is CODE wherever it lives, including under docs/ (#1200) #299 does not change that (itsalwayscoderule keys onfile extension, and
.gitattributeshas none).ci.ymlcomment near line 170, against CLAUDE.md section 11.Verification
ruff check .andruff format --check .clean over 1,063 files. 162 tests across the three new /extended modules plus the ASVS scorecard and backlog guards pass.
ledger_check.py --circ=0 againstfreshly fetched
origin/main. Backlog banner invariant OK at 474 items. The new workflow passesactionlint and its YAML anchor expands to the same four globs on both triggers. BACKLOG #1202, #1204
and #1205 allocated with
scripts/coord/alloc.ps1, never by grep. The #1204 and #1205 backlog entrieswere themselves run through the #1204 lint -- neither trips it.
docs/BACKLOG.mdconflicted once againstmain(#1201). Both sides were pure appends -- verifiedrather than assumed (
+103 -0mine,+53 -0theirs, each side's first 8,715 lines the baseverbatim). The resolution was constructed as base + theirs + mine rather than hand-edited, then
checked: every line of both sides present in order, each heading occurring exactly once, and a
neither-side sentinel absent.
Two pre-existing local failures reproduce with these changes removed and are unrelated:
mypyreports 21 errors in 4 files (optional-extra typing, e.g.
transports/dicom.py).