backlog: file #1244 -- a broken vault anchor changes no colour, so nobody sees it - #367
Merged
Merged
Conversation
…body sees it The scorecard lives in the vault and cites engine code: 1,998 evidence anchors name engine paths and 192 of them cite engine prose under docs/, the kind of line an ordinary documentation edit rewrites without anyone thinking about security evidence. Nothing engine-side references the scorecard and nothing can, because the data is what must stay private. Filed narrower than proposed, and the narrowing is the finding. The premise "nothing checks that the two repos agree" is FALSE: the vault gate's own trigger block documents this hazard, records a live instance where an engine PR moved eleven anchors while the gate stayed green, and a daily run was added to close it. The detector exists and works. What remains is that it cannot block pre-merge, cannot attribute a break to an engine change, and -- the real defect -- lands in a gate that is ALREADY RED at eleven failures and is not a required context. A twelfth failure moves no colour, no mergeability and no notification, so it is indistinguishable from yesterday's run. Detection that cannot change state is not detection anybody sees, which is why the live instance was caught by a person reading a diff. The fix has a constraint that makes the obvious build wrong: an engine-side check runs in a PUBLIC repository with public logs, so it must report the file and line that stopped matching and must not name the cell or the requirement id. A failing public log that enumerates those hands out coverage by subtraction. Report the line, never the cell. The item records its own framing error rather than presenting the narrow version as the original intent. It was first proposed as one item covering a whole class, generalised from three sightings in one evening. A class has no prove-a-fix so it could never be closed, and bundling limbs with different fixes is what splitting #1236 out of #1131 existed to prevent. Two of the three sightings did not survive: one had resolved itself, and the third was carried second-hand, never opened, and on re-reading was a different mechanism entirely. Counts asserted either side of the edit: 274 -> 275 items, open 200 -> 201, closed 74 -> 74. Exactly one new OPEN item and no status moved, which is the check that catches a glyph being read as a banner.
wshallwshall
enabled auto-merge (squash)
August 13, 2026 14:01
Pure tail collision: main gained #1246 and #1245's amendment while this branch appended #1244. Both sides only add. Resolved as a union, #1244 then #1246. The union is safe here and that was measured rather than assumed, because a union is exactly the resolution that silently reinstates a line one side deleted and the other still carries as context. Neither side deletes: main since the merge base is +29 -0, this branch is +31 -0. Verified after resolving: delta vs main items +1, open +1, closed +0 -- adds exactly one OPEN item, #1244 deletions vs main 0 duplicate headings none conflict markers 0 files differing docs/BACKLOG.md only Item numbers are left in merge order rather than re-sorted. The tail now reads 1242, 1245, 1244, 1246, which looks wrong and is not: this file has never been strictly ascending -- main carries eight descending steps of its own (344/343, 1011/1002, 1022/1019 among them) -- and nothing reads or enforces the order. Re-sorting would move lines this branch does not own to satisfy a rule that does not exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Files one backlog item. Docs-only:
docs/BACKLOG.md, +31 lines, 0 deletions, pure append.#1244 -- an engine change that breaks a vault ASVS anchor produces no state change and no
attribution.
The coupling
The scorecard lives in the vault (private) and cites code in the engine (public). All 1,998
evidence anchors name engine paths, and 192 cite engine prose under
docs/-- the kind of line anordinary documentation edit rewrites without anyone thinking about security evidence.
Nothing engine-side references the scorecard, and nothing can: the engine tree ships the verifier
(
scripts/asvs/scorecard.py) and zero scorecard data files, deliberately, because the data is whatstays private.
Filed narrower than proposed -- and the narrowing IS the finding
The premise I started from, "nothing checks that the two repos agree", is false. The vault gate's
own trigger block documents this hazard, records a live instance where an engine PR moved eleven
anchors while the gate stayed green, and a daily run was added to close it. The detector exists and
works, and a fix must start from it rather than rebuild it.
What remains:
verifyis not a required contextGap 3 is the real defect. A detector whose output is a count inside an already-failing,
non-blocking job is indistinguishable from yesterday's run. That is why the live instance -- PR #361
deleting a sentence an anchor pins verbatim -- was caught by a person reading a diff, not by the
instrument built to catch it.
A constraint that makes the obvious build wrong
An engine-side check runs in a public repository with public logs. It must report the engine file
and line that stopped matching and must not name the cell, the requirement id, or the count of
affected cells. A failing public log enumerating those hands out coverage by subtraction
(CLAUDE.md section 12). Report the line, never the cell. The natural implementation leaks, which is
what makes this non-trivial.
Prove-a-fix
Delete a line an anchor pins verbatim in an engine PR; assert the PR goes red before merge, naming
file and line and no cell id. Then the negative control: a PR touching an unanchored line stays
green, so the check discriminates rather than being always-red. A fix that only shortens the schedule
addresses gap 1 alone; making
verifyrequired without first clearing the standing 11 converts asilent problem into a permanently blocked repository.
Verification
+31 -0, exactly one new headingone new OPEN item, no status moved. That check is what catches a glyph being read as a status banner
rather than as emphasis; a sibling session hit exactly that today and an item silently flipped to
CLOSED.
alphabet
intent