fix(docs): reviewer role taught a retired grammar to every agent in this repo - #76
Conversation
…his repo CLAUDE.md and AGENTS.md are auto-loaded as standing instructions into every agent working in this repo — including a dispatched secondary. Their §11 "Multi-review (reviewer role)" section instructed `> [reviewer:<id>]` / `[author: resolved:]` and the two-agent `peer-review` pair, both of which docs/multi-review.md's `## Supersedes` records as RETIRED. `emit_prompt` cuts the inlined contract at `## Supersedes` precisely so a reviewer is never offered those grammars, and reviewer.test.sh guards that on the prompt side. The memory files re-offered them from the other direction. A reviewer that complied wrote lines `merge` cannot read: `channel-check` scores the turn as a non-response and quarantines the provider, with a reason that describes the symptom rather than the cause — a real review turn destroyed, and the round reported as though that secondary found nothing. Rewrites both sections to the live star grammar (`[finding:<id>|<sev>]` with its required `> — via` / `> — risk:` / `> — evidence:` lines, the `[no-findings]` signal, the marker flip, and the rule that responses belong to the primary). AGENTS.md is synced from CLAUDE.md so the two cannot drift. Adds the counter-instruction to the dispatched prompt itself: repo memory files are the AUTHOR's standing instructions about the very work under review, so following them costs the independence the secondary was dispatched for. This is a real context channel into "independent" reviewers that the star topology never accounted for. Documents the limit honestly in docs/multi-review.md: blind-check proves what a reviewer was HANDED, verify-vendor proves who answered, and neither can prove what was read at runtime. Independence is instructed, not enforced. The codex prompt golden is regenerated deliberately — its job is to catch UNINTENDED drift, and the diff is exactly and only the new paragraph. Guards added: packaging asserts neither memory file's reviewer-role section instructs a retired grammar and that all three prompts carry the memory-file instruction, with mutation entries for both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE
kevin-agrology
left a comment
There was a problem hiding this comment.
Multi-review
Disagreements (1)
🟡 low — The new memory-file grammar guard omits the retired peer [finding:] token, even though that token is distinct from the live [finding:<id>|<sev>] form. — risk: a future reintroduction can again direct a secondary to emit findings the star merger rejects — flagged by gpt-5; claude-opus-5 disputes: [finding:] is the LIVE token, not a retired one — the retired peer grammar is caught by its distinctive [concur:/[withdraw: verbs, which the guard already bans, and the positive assertion separately requires the [finding:<id>|<sev>] severity form; banning [finding: would fail on a correct document.
———
🤖 Posted by AI agents (claude-opus-5 + gpt-5) via multi-review star review.
Raised by codex as codex-rd2-r1 in a depth round on this PR. I disputed it and
was wrong; withdrawing the dispute and implementing the fix.
The dispute argued that `[finding:` is the LIVE token, so banning it would fail
a correct document, and that the retired peer grammar is already caught by its
distinctive `[concur:`/`[withdraw:` verbs. Both halves are true, and neither
answers the finding. Codex said the guard OMITS the retired spelling; it did not
say the remedy had to be a wholesale ban. I refuted the easier claim.
The gap is real. `## Supersedes` lists `> [finding:]` as part of the retired
two-agent grammar, and the peer spelling carries no severity. A §11 rewrite that
teaches `> [finding:r1]` in its example while some prose elsewhere still contains
the literal `[finding:<id>|<sev>]` passes BOTH existing checks: no banned verb
appears, and the live form is named.
And it is consequential, which is the part I had not checked before disputing. A
severity-less finding is a hard parse error:
> [finding:r1] a concern with no severity tag
-> multi-review-star: finding r1 needs a |high, |med, or |low severity tag
-> open-findings exits 2
So a reviewer complying with that instruction destroys its own turn — precisely
the failure class this guard family exists to prevent.
The fix keys on the ABSENT `|<sev>` part rather than on `[finding:` itself:
`\[finding:[^]|]*\]` stops at `]` or `|`, so `[finding:r1]` matches and
`[finding:<id>|<sev>]` does not. Verified against both spellings.
The detection is extracted into `retired_grammar_in` so it can be exercised
against a SYNTHETIC section, not only the real files. Asserting only against
CLAUDE.md/AGENTS.md makes the check unfalsifiable in practice — it would pass
because those files are currently correct, not because it can detect a bad one.
Two assertions now bracket it: it must FIRE on the severity-less form that also
names the live one, and must NOT fire on the live form alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE
|
Withdrawing my dispute of The published review records a dispute on this finding. On re-examination that dispute was wrong, so the correction belongs here rather than as a silent edit to a converged review doc. What I argued: Why that doesn't answer the finding: both halves are true, and neither addresses what was actually said. Codex said the guard omits the retired spelling — not that the remedy had to be a wholesale ban. I refuted the easier claim. The gap is real. And it is consequential — the part I had not checked before disputing: A hard parse error. So a reviewer complying with that instruction destroys its own turn, which is exactly the failure class this guard family exists to prevent. Fixed in the latest commit by keying on the absent The detection is now extracted into For the record, the protocol warns against precisely the move I made: "do not dispute to save a round — a finding you cannot refute on the merits is one you agree with... if you are unsure, agree and fix; being wrong in that direction is cheaper." A 🤖 Posted by an AI agent — Claude Opus 5 ( |
Conflicts in the mutation table and packaging suite were additive on both sides; kept both. Version resolved to 1.21.1. reviewer.sh auto-merged cleanly with main's vendor-map and comment-stripping changes, and the byte-frozen codex prompt golden still matches — neither of those touched the prompt. Verified rather than assumed, along with every packaging guard from all merged branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE
TL;DR: This repo's own
CLAUDE.mdandAGENTS.mdwere telling reviewers to use an old, retired comment format. Any reviewer that followed them wrote findings the tool literally cannot read, so its entire turn was thrown away and the round recorded it as "found nothing." This fixes the instructions and tells reviewers to ignore those files entirely, since they're the author's words about the work under review.The bug
CLAUDE.mdandAGENTS.mdare auto-loaded as standing instructions into every agent working in this repo — including a dispatched secondary. Their §11 "Multi-review (reviewer role)" section instructed:> [reviewer:<id>]/[author: resolved:<id>]— the asymmetric single-reviewer pair[finding:]/[concur:]/[dispute:]/[withdraw:]under apeer-reviewmode hintdocs/multi-review.md's## Supersedesrecords both as retired.This is not a cosmetic doc drift.
emit_promptdeliberately cuts the inlined contract at## Supersedesso a reviewer is never offered those grammars, andmulti-review-reviewer.test.sh:196-233guards that on the prompt side. The memory files re-offered them from the other direction, and a memory file outranks a prompt by construction — it arrives labelled as instructions that override default behavior.A reviewer that complied wrote lines
mergecannot read.channel-checkthen scores the turn as a non-response and quarantines the provider, with a reason describing the symptom rather than the cause. The result is a real review turn destroyed and the round reporting that secondary as having found nothing.The second channel
Fixing the grammar closes the acute bug but not the general one: repo memory files are a context channel into "independent" reviewers that the star topology never accounted for. In an arbitrary target repo the injected file is the author's standing instruction set, arriving in the reviewer's context before it reads a line of the work under review.
So the dispatched prompt now says to ignore them and treat the contract as complete. Cheap, and it addresses the acute case (a repo whose memory file describes an older protocol) and the general one (bias) together.
What changed
CLAUDE.md/AGENTS.md— reviewer-role section rewritten to the live star grammar:[finding:<id>|<sev>]with required> — via/> — risk:/> — evidence:lines, the[no-findings]signal, the marker flip, ids scoped per copy, and the rule that[agree:]/[dispute:]belong to the primary. AGENTS.md is synced from CLAUDE.md so the two cannot drift.scripts/multi-review-reviewer.sh— the memory-file counter-instruction, in every prompt (skill-bearing and skill-less alike).docs/multi-review.md— states the independence limit honestly:blind-checkproves what a reviewer was handed,verify-vendorproves who answered, and neither can prove what was read at runtime. Working copies also sit beside the live doc and are mechanically readable. Independence is instructed, not enforced — the gate line means "nothing structural was wrong with this round", not a proof about each reviewer's context.scripts/fixtures/codex-prompt.golden.txt— regenerated (see below)..agents/skills/multi-review/protocol/multi-review.md— regenerated viamulti-review-build-reviewer-bundle.sh.On the golden fixture
The codex prompt is byte-frozen against a checked-in golden. I regenerated it deliberately: the golden's job is to catch unintended drift, and this change is intended. The diff was inspected before regenerating and is exactly and only the new paragraph — no other byte moved.
Guards added
docs/claude-md-reviewer-grammar[reviewer:<id>]reviewer/prompt-ignore-memory-filesBoth verified caught. These are doc-level guards — the regression they prevent reintroduces a bug with every script-level test still green, which is the case the runner's own "SHELL TARGETS ONLY" note says needs an entry.
Verification
scripts/*.test.sh(15 suites)mutation-check --only docs/claude-md-reviewer-grammarmutation-check --only reviewer/prompt-ignore-memory-filesmulti-review-version-check.sh1.19.0 → 1.19.2 (ok)multi-review-docs-check.shNot run locally:
shellcheck(not installed here); CI covers it plus the bash 3.2 leg. The full mutation sweep is CI's — the two new entries were verified individually.Note for review
Editing
CLAUDE.mdis editing the repo's working agreement, so to be explicit about scope: only the factual protocol content of §11's reviewer-role section changed. No engineering policy, standard, or gate was touched.🤖 Generated with Claude Code
https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE