From a5804341bd2e88cbdaac7c69c79b925876b710cd Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Wed, 21 Oct 2026 10:26:24 +0200 Subject: [PATCH 1/2] chore(agents): round 1 reports every site of a defect class, safety candidates are probed before any downgrade, mechanical fix-checks spawn no lens, and the audit proves coverage PROCESS-REVIEW-2026-10-21.md items 13, 14, 15 and 23. Verified by replay before this PR leaves draft: #775 round 1 and #752 round 1 with these rules. --- .agents/briefs/implement-audit.md | 10 ++++++++++ .agents/briefs/review-code.md | 10 ++++++++++ .claude/agents/implementer.md | 2 +- docs/internal/REVIEW.md | 22 +++++++++++++++++++--- 4 files changed, 40 insertions(+), 4 deletions(-) diff --git a/.agents/briefs/implement-audit.md b/.agents/briefs/implement-audit.md index 2b311bff2..ce7663e49 100644 --- a/.agents/briefs/implement-audit.md +++ b/.agents/briefs/implement-audit.md @@ -29,6 +29,16 @@ when and assumptions; retain earlier evidence for unaffected claims. - **Promised.** For every Scope and Done when line, name the hunk or test that delivers it. None: do it, or leave the line unticked and say why under `open`. Never tick a box on intent. +- **Covered.** List every function whose body the diff adds or changes, + one line per file, as the review's behaviour lens will. For each: what + a caller, a user, a stored record or an event sees when it runs, and + which test goes red when its body, its new branch or its new argument + is deleted (testing rule 3). Delete it once and run that test's file to + see. No test goes red: write the one that does, through the module's + surface (rule 6), or put the line under `open`. Report `covered /` + on the `audit:` line: `f` functions listed, `c` with a red test. The + review's tests lens asks the same question on a far costlier model; + every gap it still finds is this step's miss. **Check every test you added or changed**, including setup or assertions edited after its `test:` commit: diff --git a/.agents/briefs/review-code.md b/.agents/briefs/review-code.md index bf5d0cae5..9d08b3744 100644 --- a/.agents/briefs/review-code.md +++ b/.agents/briefs/review-code.md @@ -41,6 +41,12 @@ Then answer, for each of them: processes, timers or locks)? Find each with a search, not from memory. For each: what happens when both act, in either order or at once? +3. For each blocking finding: what else in the diff has the same shape? + Search `diff.patch` and the files it touches for every other site of + the same class (the same column type, the same copied format, the same + missing guard) before you report, and list every site in the one + finding. A sibling the next round finds is this round's miss. + Blocking: the change breaks behaviour, leaves wrong state, or breaches a rule or an accepted ADR; a user or a later change pays for it. When unsure, ask "what breaks, and for whom?"; no answer means drop it. @@ -60,6 +66,10 @@ Then answer: test compares. A test that stays green is a finding. 2. Does the diff break any rule in `rules/`? Cite the file and the rule number. +3. For each blocking finding: which other tests in the diff have the same + shape (the same unproven order, the same loosened compare, the same + column left untested)? Search for them and list every site in the one + finding. Blocking: a test that cannot fail for the reason its title gives, or a breach of a rule or an accepted ADR. diff --git a/.claude/agents/implementer.md b/.claude/agents/implementer.md index 8ce77bdc7..fba9a159e 100644 --- a/.claude/agents/implementer.md +++ b/.claude/agents/implementer.md @@ -119,7 +119,7 @@ Assessment: complete | incomplete — tests: / cycles: / remaining: -audit: stale replaced donewhen / probed

/ +audit: stale replaced donewhen / probed

/ covered / fixes: -> found fixed run: pass | fail ci: pass | fail | running | unchecked diff --git a/docs/internal/REVIEW.md b/docs/internal/REVIEW.md index a9cd9874d..7a26bb985 100644 --- a/docs/internal/REVIEW.md +++ b/docs/internal/REVIEW.md @@ -22,8 +22,13 @@ requires an explicit maintainer-authorized amendment to the affected rule or ADR. Review such amendments as changes, not as automatic self-exemptions. Accepted future-state ADRs remain targets until implemented. -Round 1 covers the whole PR diff. Inspect surrounding code, module manuals, -callers and shared state as needed. Claims review blocks changed claims and +Round 1 covers the whole PR diff, and reports every site of each defect +class it confirms: a lens that finds one STRICT INTEGER column refused a +fractional value searches the diff for every other such column before it +reports. The targeted round verifies fixes and looks for what the fixes +changed; a sibling it finds is round 1's miss, and counts as one in the +ledger (`round: 2` on a finding whose class round 1 already had). Inspect +surrounding code, module manuals, callers and shared state as needed. Claims review blocks changed claims and claims the PR makes false, including affected untouched documentation. Read manuals and flows for context, not as a request to repair unrelated debt. Unchanged lines can contain PR regressions; demonstrate the causal link. @@ -63,7 +68,12 @@ separate worktrees for probes; do not share findings until synthesis. Assign stable PR-local IDs (R1, R2, ...) by defect class, affected contract and failure mechanism, not line number. Deduplicate across lenses before verification. Verify each distinct defect once with source evidence or a -focused reproducer. Consolidate fixes by class. Keep a resolution ledger: +focused reproducer. A candidate in a safety class (data a user owns +deleted or overwritten, files Simlock did not write touched, a credential +exposed, ownership or liveness assumed) is never downgraded to a note or +dismissed at synthesis on reasoning alone: the coordinator runs, or has a +lens run, the one probe that shows the input cannot occur, and records it +in the ledger; without that evidence the candidate blocks. Consolidate fixes by class. Keep a resolution ledger: ID, originating roles, evidence, impact, outcome, reason, fix commit and verification commit. Outcomes: open, fixed, duplicate (points to canonical ID), incorrect, intentional, nonblocking. Only incorrect findings are @@ -85,6 +95,12 @@ a focused claims check; unchanged claims need no repeat review. Use actual hunks, not the intended fix label, to choose follow-up. A lens whose input `fix.patch` does not touch does not run: claims needs a changed doc, comment, test title or printed string; code a changed code or test line. +A fix-check whose every hunk is mechanical (a merge that keeps both sides +unchanged, a timeout or retry constant, a doc line, a renamed symbol with +no call changed) spawns no lens: the coordinator reads the exact diff, +records `verified by: coordinator` with the hunks in the ledger, and +reports. One semantic hunk among them sends only the lens that hunk +reaches. There is one full round, one targeted round 2, and one final targeted fix-check. Count every invocation, including that fix-check. After the cap, From 0adc63f0555c3c63ca642eb7a0a90f1f2ac19668 Mon Sep 17 00:00:00 2001 From: Szymon Chmal Date: Wed, 21 Oct 2026 10:38:45 +0200 Subject: [PATCH 2/2] docs(internal): the review changes verified by replay on #752 and #775 --- .agents/replay/results.jsonl | 2 ++ docs/internal/PROCESS-REVIEW-2026-10-21.md | 33 ++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/.agents/replay/results.jsonl b/.agents/replay/results.jsonl index ed11a0640..2f65a5e1f 100644 --- a/.agents/replay/results.jsonl +++ b/.agents/replay/results.jsonl @@ -10,3 +10,5 @@ {"at":"2026-10-21T07:42:08.995Z","pr":752,"round":"1","variant":"lens-spec-check","valid":true,"agentsCommit":"baddf403","recall":"1/1","found":["R5"],"missed":[],"unmatched":["F2 src/core/write-leftovers.ts:161 [triage Risk breached]","F3 src/core/write-leftovers.ts:139 [assumption decides user-visible behaviour; contradicts issue example]","F4 src/core/write-leftovers.ts:130 [`STAGING_DIRECTORY` needs a digits segment in the name. A pi]"],"nonDefectHits":0,"replay":{"runs":1,"seconds":26,"costUsd":0.1228},"baseline":{"runs":4,"seconds":637,"costUsd":1.3761999999999999,"lenses":["reviewer/both:claude-sonnet-5-5","code/both:claude-opus-5-5","spec/-:claude-sonnet-5-5","claims/-:claude-sonnet-5-5"]}} {"at":"2026-10-21T07:50:03.995Z","pr":775,"round":"1","variant":"current-run1","valid":true,"agentsCommit":"baddf403","recall":"3/10","found":["R1","R2","R7"],"missed":["R3","R4","R5","R6","R10","R11","R12"],"unmatched":["R1 src/daemon/main.ts [spec line undelivered: daemon socket probe]","R4 src/daemon/main.test.ts [testing 1]","R5 src/core/registry.test.ts [testing 2]","R6 src/core/registry.ts [comment names nonexistent table or column]","R7 src/core/AGENTS.md [manual claims drivers own tables]"],"nonDefectHits":0,"replay":{"runs":5,"seconds":2058,"costUsd":6.4711},"baseline":{"runs":5,"seconds":1907,"costUsd":6.6301000000000005,"lenses":["reviewer/-:claude-sonnet-5-5","spec/-:claude-sonnet-5-5","code/behaviour:claude-opus-5-5","code/tests:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}} {"at":"2026-10-21T07:59:13.178Z","pr":775,"round":"2","variant":"current-run1","valid":true,"agentsCommit":"baddf403","recall":"6/10","found":["R1","R3","R4","R5","R10","R11"],"missed":["R2","R6","R7","R12"],"unmatched":["R2 unknown [claim false (round 1 text not carried)]","R6 unknown [claim false (round 1 text not carried)]","R7 unknown [claim false (round 1 text not carried)]","R8 unknown [note]","R9 unknown [note]","R11 src/core/registry.test.ts [testing 3, testing 9]"],"nonDefectHits":0,"replay":{"runs":3,"seconds":833,"costUsd":1.5988999999999998},"baseline":{"runs":3,"seconds":803,"costUsd":1.7179,"lenses":["reviewer/-:claude-sonnet-5-5","code/both:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}} +{"at":"2026-10-21T08:36:22.694Z","pr":752,"round":"1","variant":"rulesC-a5804341","valid":true,"agentsCommit":"fa2cdf56","recall":"3/6","found":["R1","R2","R4"],"missed":["R3","R5","R6"],"unmatched":["R1 src/core/write-leftovers.ts [report legacy staging undelivered]","R3 src/core/write-leftovers.ts [EPERM pid counted dead]","R5 src/core/write-leftovers.ts [testing 3: kind/uid guards untested]","R6 src/core/write-leftovers.ts [architecture 10: name formats copied as regexes]","R8 src/daemon/main.ts [sweep not proven after claim]","R9 src/core/write-leftovers.ts [pid-less staging not logged]","R10 src/daemon/main.test.ts [pid-less staging test missing]"],"nonDefectHits":0,"replay":{"runs":4,"seconds":1032,"costUsd":1.6673},"baseline":{"runs":4,"seconds":637,"costUsd":1.3761999999999999,"lenses":["reviewer/both:claude-sonnet-5-5","code/both:claude-opus-5-5","spec/-:claude-sonnet-5-5","claims/-:claude-sonnet-5-5"]}} +{"at":"2026-10-21T08:38:18.910Z","pr":775,"round":"1","variant":"rulesC-a5804341","valid":true,"agentsCommit":"fa2cdf56","recall":"6/10","found":["R1","R2","R4","R5","R6","R7"],"missed":["R3","R10","R11","R12"],"unmatched":["R2 src/daemon/main.ts [spec line undelivered: no socket probe before openStateDatabase]","R4 src/core/registry.test.ts [testing: pruned repeat tests kept]","R5 src/core/registry.test.ts [testing 10: one-transaction test fault hits first write]","R9 docs/CLI.md [docs claim real drivers have device-data tables]","R11 src/daemon/main.test.ts [duplicate of R2: missing second-daemon test]","R12 src/core/AGENTS.md [AGENTS.md claim real driver has table]"],"nonDefectHits":0,"replay":{"runs":5,"seconds":2209,"costUsd":8.0662},"baseline":{"runs":5,"seconds":1907,"costUsd":6.6301000000000005,"lenses":["reviewer/-:claude-sonnet-5-5","spec/-:claude-sonnet-5-5","code/behaviour:claude-opus-5-5","code/tests:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}} diff --git a/docs/internal/PROCESS-REVIEW-2026-10-21.md b/docs/internal/PROCESS-REVIEW-2026-10-21.md index cc816d188..4ff5445dd 100644 --- a/docs/internal/PROCESS-REVIEW-2026-10-21.md +++ b/docs/internal/PROCESS-REVIEW-2026-10-21.md @@ -432,3 +432,36 @@ adjudication (#752 regex copies; #752 EPERM liveness), and a measured reviewer weakness (2 of 4 replay coordinators self-checked until told otherwise). The next replays are the fix-check snapshots in 4.3, and then any variant of items 13 to 18. + +### 4.8 The review changes, verified by replay + +Items 13, 15 and 23 were replayed with the changed rules, briefs and +protocol (`review-replay.sh 1 `), once each, against the +same snapshots as the baseline runs in 4.5 and 4.6. + +| Snapshot | Baseline recall (by hand) | With the changes | USD before / after | Minutes before / after | +| -------- | ------------------------- | ---------------- | ------------------ | ---------------------- | +| #752 r1 | 4/6 | 6/6 | 1.41 / 1.67 | 12 / 17 | +| #775 r1 | 6/10 | 8/10 | 6.47 / 8.07 | 34 / 37 | + +What changed in the reports: + +- **#752:** the safety finding (the sweep deletes a user's own matching + directory) came back as a blocking finding with a reproduction, where + both baseline runs had lost it at synthesis; the coordinator also kept + EPERM-counted-dead blocking "under REVIEW.md's safety rule" because it + had no probe showing the input cannot occur. Item 23 does what it says. +- **#775:** round 1 reported all three STRICT INTEGER columns in one + finding, with two probes; the real round 1 found one and round 2 its + siblings. Item 13 does what it says. The socket probe (the real review's + fix-check R13) was again found in round 1. Still missed: the fake table + accepting extra keys. The tests lens reported incomplete on probe + budget, which the protocol asks it to say rather than pass. +- Cost rose about 20 percent per round 1, the price of the probes and the + sibling search. The rounds it should remove (the real #775 round 2 cost + $1.72) are not in this measurement; the next deliveries' invocation + counts are. + +Item 14 (the audit proves coverage) has no replay: it changes the +implementer, and is judged by testing-class review findings per PR on the +next deliveries.