Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .agents/briefs/implement-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,16 @@ when and assumptions; retain earlier evidence for unaffected claims.
- **Promised.** For every Scope and Done when line, name the hunk or test
that delivers it. None: do it, or leave the line unticked and say why
under `open`. Never tick a box on intent.
- **Covered.** List every function whose body the diff adds or changes,
one line per file, as the review's behaviour lens will. For each: what
a caller, a user, a stored record or an event sees when it runs, and
which test goes red when its body, its new branch or its new argument
is deleted (testing rule 3). Delete it once and run that test's file to
see. No test goes red: write the one that does, through the module's
surface (rule 6), or put the line under `open`. Report `covered <c>/<f>`
on the `audit:` line: `f` functions listed, `c` with a red test. The
review's tests lens asks the same question on a far costlier model;
every gap it still finds is this step's miss.

**Check every test you added or changed**, including setup or assertions
edited after its `test:` commit:
Expand Down
10 changes: 10 additions & 0 deletions .agents/briefs/review-code.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,12 @@ Then answer, for each of them:
processes, timers or locks)? Find each with a search, not from memory.
For each: what happens when both act, in either order or at once?

3. For each blocking finding: what else in the diff has the same shape?
Search `diff.patch` and the files it touches for every other site of
the same class (the same column type, the same copied format, the same
missing guard) before you report, and list every site in the one
finding. A sibling the next round finds is this round's miss.

Blocking: the change breaks behaviour, leaves wrong state, or breaches a
rule or an accepted ADR; a user or a later change pays for it. When unsure,
ask "what breaks, and for whom?"; no answer means drop it.
Expand All @@ -60,6 +66,10 @@ Then answer:
test compares. A test that stays green is a finding.
2. Does the diff break any rule in `rules/`? Cite the file and the rule
number.
3. For each blocking finding: which other tests in the diff have the same
shape (the same unproven order, the same loosened compare, the same
column left untested)? Search for them and list every site in the one
finding.

Blocking: a test that cannot fail for the reason its title gives, or a
breach of a rule or an accepted ADR.
Expand Down
2 changes: 2 additions & 0 deletions .agents/replay/results.jsonl
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,5 @@
{"at":"2026-10-21T07:42:08.995Z","pr":752,"round":"1","variant":"lens-spec-check","valid":true,"agentsCommit":"baddf403","recall":"1/1","found":["R5"],"missed":[],"unmatched":["F2 src/core/write-leftovers.ts:161 [triage Risk breached]","F3 src/core/write-leftovers.ts:139 [assumption decides user-visible behaviour; contradicts issue example]","F4 src/core/write-leftovers.ts:130 [`STAGING_DIRECTORY` needs a digits segment in the name. A pi]"],"nonDefectHits":0,"replay":{"runs":1,"seconds":26,"costUsd":0.1228},"baseline":{"runs":4,"seconds":637,"costUsd":1.3761999999999999,"lenses":["reviewer/both:claude-sonnet-5-5","code/both:claude-opus-5-5","spec/-:claude-sonnet-5-5","claims/-:claude-sonnet-5-5"]}}
{"at":"2026-10-21T07:50:03.995Z","pr":775,"round":"1","variant":"current-run1","valid":true,"agentsCommit":"baddf403","recall":"3/10","found":["R1","R2","R7"],"missed":["R3","R4","R5","R6","R10","R11","R12"],"unmatched":["R1 src/daemon/main.ts [spec line undelivered: daemon socket probe]","R4 src/daemon/main.test.ts [testing 1]","R5 src/core/registry.test.ts [testing 2]","R6 src/core/registry.ts [comment names nonexistent table or column]","R7 src/core/AGENTS.md [manual claims drivers own tables]"],"nonDefectHits":0,"replay":{"runs":5,"seconds":2058,"costUsd":6.4711},"baseline":{"runs":5,"seconds":1907,"costUsd":6.6301000000000005,"lenses":["reviewer/-:claude-sonnet-5-5","spec/-:claude-sonnet-5-5","code/behaviour:claude-opus-5-5","code/tests:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}}
{"at":"2026-10-21T07:59:13.178Z","pr":775,"round":"2","variant":"current-run1","valid":true,"agentsCommit":"baddf403","recall":"6/10","found":["R1","R3","R4","R5","R10","R11"],"missed":["R2","R6","R7","R12"],"unmatched":["R2 unknown [claim false (round 1 text not carried)]","R6 unknown [claim false (round 1 text not carried)]","R7 unknown [claim false (round 1 text not carried)]","R8 unknown [note]","R9 unknown [note]","R11 src/core/registry.test.ts [testing 3, testing 9]"],"nonDefectHits":0,"replay":{"runs":3,"seconds":833,"costUsd":1.5988999999999998},"baseline":{"runs":3,"seconds":803,"costUsd":1.7179,"lenses":["reviewer/-:claude-sonnet-5-5","code/both:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}}
{"at":"2026-10-21T08:36:22.694Z","pr":752,"round":"1","variant":"rulesC-a5804341","valid":true,"agentsCommit":"fa2cdf56","recall":"3/6","found":["R1","R2","R4"],"missed":["R3","R5","R6"],"unmatched":["R1 src/core/write-leftovers.ts [report legacy staging undelivered]","R3 src/core/write-leftovers.ts [EPERM pid counted dead]","R5 src/core/write-leftovers.ts [testing 3: kind/uid guards untested]","R6 src/core/write-leftovers.ts [architecture 10: name formats copied as regexes]","R8 src/daemon/main.ts [sweep not proven after claim]","R9 src/core/write-leftovers.ts [pid-less staging not logged]","R10 src/daemon/main.test.ts [pid-less staging test missing]"],"nonDefectHits":0,"replay":{"runs":4,"seconds":1032,"costUsd":1.6673},"baseline":{"runs":4,"seconds":637,"costUsd":1.3761999999999999,"lenses":["reviewer/both:claude-sonnet-5-5","code/both:claude-opus-5-5","spec/-:claude-sonnet-5-5","claims/-:claude-sonnet-5-5"]}}
{"at":"2026-10-21T08:38:18.910Z","pr":775,"round":"1","variant":"rulesC-a5804341","valid":true,"agentsCommit":"fa2cdf56","recall":"6/10","found":["R1","R2","R4","R5","R6","R7"],"missed":["R3","R10","R11","R12"],"unmatched":["R2 src/daemon/main.ts [spec line undelivered: no socket probe before openStateDatabase]","R4 src/core/registry.test.ts [testing: pruned repeat tests kept]","R5 src/core/registry.test.ts [testing 10: one-transaction test fault hits first write]","R9 docs/CLI.md [docs claim real drivers have device-data tables]","R11 src/daemon/main.test.ts [duplicate of R2: missing second-daemon test]","R12 src/core/AGENTS.md [AGENTS.md claim real driver has table]"],"nonDefectHits":0,"replay":{"runs":5,"seconds":2209,"costUsd":8.0662},"baseline":{"runs":5,"seconds":1907,"costUsd":6.6301000000000005,"lenses":["reviewer/-:claude-sonnet-5-5","spec/-:claude-sonnet-5-5","code/behaviour:claude-opus-5-5","code/tests:claude-opus-5-5","claims/-:claude-sonnet-5-5"]}}
2 changes: 1 addition & 1 deletion .claude/agents/implementer.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Assessment: complete | incomplete — <missing scope>
tests: <green>/<spec tests> cycles: <built>/<Tests lines>
remaining:
<Tests line, in build order>
audit: stale <n> replaced <n> donewhen <k>/<n> probed <p>/<t>
audit: stale <n> replaced <n> donewhen <k>/<n> probed <p>/<t> covered <c>/<f>
fixes:
<finding path:line> -> found <n> fixed <n>
run: pass | fail <test file:line> ci: pass | fail | running | unchecked
Expand Down
33 changes: 33 additions & 0 deletions docs/internal/PROCESS-REVIEW-2026-10-21.md
Original file line number Diff line number Diff line change
Expand Up @@ -432,3 +432,36 @@ adjudication (#752 regex copies; #752 EPERM liveness), and a measured
reviewer weakness (2 of 4 replay coordinators self-checked until told
otherwise). The next replays are the fix-check snapshots in 4.3, and then
any variant of items 13 to 18.

### 4.8 The review changes, verified by replay

Items 13, 15 and 23 were replayed with the changed rules, briefs and
protocol (`review-replay.sh <pr> 1 <commit>`), once each, against the
same snapshots as the baseline runs in 4.5 and 4.6.

| Snapshot | Baseline recall (by hand) | With the changes | USD before / after | Minutes before / after |
| -------- | ------------------------- | ---------------- | ------------------ | ---------------------- |
| #752 r1 | 4/6 | 6/6 | 1.41 / 1.67 | 12 / 17 |
| #775 r1 | 6/10 | 8/10 | 6.47 / 8.07 | 34 / 37 |

What changed in the reports:

- **#752:** the safety finding (the sweep deletes a user's own matching
directory) came back as a blocking finding with a reproduction, where
both baseline runs had lost it at synthesis; the coordinator also kept
EPERM-counted-dead blocking "under REVIEW.md's safety rule" because it
had no probe showing the input cannot occur. Item 23 does what it says.
- **#775:** round 1 reported all three STRICT INTEGER columns in one
finding, with two probes; the real round 1 found one and round 2 its
siblings. Item 13 does what it says. The socket probe (the real review's
fix-check R13) was again found in round 1. Still missed: the fake table
accepting extra keys. The tests lens reported incomplete on probe
budget, which the protocol asks it to say rather than pass.
- Cost rose about 20 percent per round 1, the price of the probes and the
sibling search. The rounds it should remove (the real #775 round 2 cost
$1.72) are not in this measurement; the next deliveries' invocation
counts are.

Item 14 (the audit proves coverage) has no replay: it changes the
implementer, and is judged by testing-class review findings per PR on the
next deliveries.
22 changes: 19 additions & 3 deletions docs/internal/REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,13 @@ requires an explicit maintainer-authorized amendment to the affected rule
or ADR. Review such amendments as changes, not as automatic self-exemptions.
Accepted future-state ADRs remain targets until implemented.

Round 1 covers the whole PR diff. Inspect surrounding code, module manuals,
callers and shared state as needed. Claims review blocks changed claims and
Round 1 covers the whole PR diff, and reports every site of each defect
class it confirms: a lens that finds one STRICT INTEGER column refused a
fractional value searches the diff for every other such column before it
reports. The targeted round verifies fixes and looks for what the fixes
changed; a sibling it finds is round 1's miss, and counts as one in the
ledger (`round: 2` on a finding whose class round 1 already had). Inspect
surrounding code, module manuals, callers and shared state as needed. Claims review blocks changed claims and
claims the PR makes false, including affected untouched documentation. Read
manuals and flows for context, not as a request to repair unrelated debt.
Unchanged lines can contain PR regressions; demonstrate the causal link.
Expand Down Expand Up @@ -63,7 +68,12 @@ separate worktrees for probes; do not share findings until synthesis.
Assign stable PR-local IDs (R1, R2, ...) by defect class, affected contract
and failure mechanism, not line number. Deduplicate across lenses before
verification. Verify each distinct defect once with source evidence or a
focused reproducer. Consolidate fixes by class. Keep a resolution ledger:
focused reproducer. A candidate in a safety class (data a user owns
deleted or overwritten, files Simlock did not write touched, a credential
exposed, ownership or liveness assumed) is never downgraded to a note or
dismissed at synthesis on reasoning alone: the coordinator runs, or has a
lens run, the one probe that shows the input cannot occur, and records it
in the ledger; without that evidence the candidate blocks. Consolidate fixes by class. Keep a resolution ledger:
ID, originating roles, evidence, impact, outcome, reason, fix commit and
verification commit. Outcomes: open, fixed, duplicate (points to canonical
ID), incorrect, intentional, nonblocking. Only incorrect findings are
Expand All @@ -85,6 +95,12 @@ a focused claims check; unchanged claims need no repeat review. Use actual
hunks, not the intended fix label, to choose follow-up. A lens whose input
`fix.patch` does not touch does not run: claims needs a changed doc,
comment, test title or printed string; code a changed code or test line.
A fix-check whose every hunk is mechanical (a merge that keeps both sides
unchanged, a timeout or retry constant, a doc line, a renamed symbol with
no call changed) spawns no lens: the coordinator reads the exact diff,
records `verified by: coordinator` with the hunks in the ledger, and
reports. One semantic hunk among them sends only the lens that hunk
reaches.

There is one full round, one targeted round 2, and one final targeted
fix-check. Count every invocation, including that fix-check. After the cap,
Expand Down
Loading