Skip to content

fix(docs): reviewer role taught a retired grammar to every agent in this repo - #76

Merged
kevin-agrology merged 3 commits into
mainfrom
fix/reviewer-role-star-grammar
Aug 11, 2026
Merged

fix(docs): reviewer role taught a retired grammar to every agent in this repo#76
kevin-agrology merged 3 commits into
mainfrom
fix/reviewer-role-star-grammar

Conversation

@kevin-agrology

Copy link
Copy Markdown
Contributor

TL;DR: This repo's own CLAUDE.md and AGENTS.md were telling reviewers to use an old, retired comment format. Any reviewer that followed them wrote findings the tool literally cannot read, so its entire turn was thrown away and the round recorded it as "found nothing." This fixes the instructions and tells reviewers to ignore those files entirely, since they're the author's words about the work under review.

The bug

CLAUDE.md and AGENTS.md are auto-loaded as standing instructions into every agent working in this repo — including a dispatched secondary. Their §11 "Multi-review (reviewer role)" section instructed:

  • > [reviewer:<id>] / [author: resolved:<id>] — the asymmetric single-reviewer pair
  • [finding:]/[concur:]/[dispute:]/[withdraw:] under a peer-review mode hint

docs/multi-review.md's ## Supersedes records both as retired.

This is not a cosmetic doc drift. emit_prompt deliberately cuts the inlined contract at ## Supersedes so a reviewer is never offered those grammars, and multi-review-reviewer.test.sh:196-233 guards that on the prompt side. The memory files re-offered them from the other direction, and a memory file outranks a prompt by construction — it arrives labelled as instructions that override default behavior.

A reviewer that complied wrote lines merge cannot read. channel-check then scores the turn as a non-response and quarantines the provider, with a reason describing the symptom rather than the cause. The result is a real review turn destroyed and the round reporting that secondary as having found nothing.

The second channel

Fixing the grammar closes the acute bug but not the general one: repo memory files are a context channel into "independent" reviewers that the star topology never accounted for. In an arbitrary target repo the injected file is the author's standing instruction set, arriving in the reviewer's context before it reads a line of the work under review.

So the dispatched prompt now says to ignore them and treat the contract as complete. Cheap, and it addresses the acute case (a repo whose memory file describes an older protocol) and the general one (bias) together.

What changed

  • CLAUDE.md / AGENTS.md — reviewer-role section rewritten to the live star grammar: [finding:<id>|<sev>] with required > — via / > — risk: / > — evidence: lines, the [no-findings] signal, the marker flip, ids scoped per copy, and the rule that [agree:]/[dispute:] belong to the primary. AGENTS.md is synced from CLAUDE.md so the two cannot drift.
  • scripts/multi-review-reviewer.sh — the memory-file counter-instruction, in every prompt (skill-bearing and skill-less alike).
  • docs/multi-review.md — states the independence limit honestly: blind-check proves what a reviewer was handed, verify-vendor proves who answered, and neither can prove what was read at runtime. Working copies also sit beside the live doc and are mechanically readable. Independence is instructed, not enforced — the gate line means "nothing structural was wrong with this round", not a proof about each reviewer's context.
  • scripts/fixtures/codex-prompt.golden.txt — regenerated (see below).
  • .agents/skills/multi-review/protocol/multi-review.md — regenerated via multi-review-build-reviewer-bundle.sh.
  • Version 1.19.0 → 1.19.2.

On the golden fixture

The codex prompt is byte-frozen against a checked-in golden. I regenerated it deliberately: the golden's job is to catch unintended drift, and this change is intended. The diff was inspected before regenerating and is exactly and only the new paragraph — no other byte moved.

Guards added

entry mutation what breaks
docs/claude-md-reviewer-grammar one bullet reverted to [reviewer:<id>] the retired grammar returns and turns are quarantined again
reviewer/prompt-ignore-memory-files the instruction reworded away an injected memory file is the reviewer's only standing instruction

Both verified caught. These are doc-level guards — the regression they prevent reintroduces a bug with every script-level test still green, which is the case the runner's own "SHELL TARGETS ONLY" note says needs an entry.

Verification

check result
scripts/*.test.sh (15 suites) all pass
mutation-check --only docs/claude-md-reviewer-grammar caught
mutation-check --only reviewer/prompt-ignore-memory-files caught
multi-review-version-check.sh 1.19.0 → 1.19.2 (ok)
multi-review-docs-check.sh pass

Not run locally: shellcheck (not installed here); CI covers it plus the bash 3.2 leg. The full mutation sweep is CI's — the two new entries were verified individually.

Note for review

Editing CLAUDE.md is editing the repo's working agreement, so to be explicit about scope: only the factual protocol content of §11's reviewer-role section changed. No engineering policy, standard, or gate was touched.


🤖 Generated with Claude Code

https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE

…his repo

CLAUDE.md and AGENTS.md are auto-loaded as standing instructions into every
agent working in this repo — including a dispatched secondary. Their §11
"Multi-review (reviewer role)" section instructed `> [reviewer:<id>]` /
`[author: resolved:]` and the two-agent `peer-review` pair, both of which
docs/multi-review.md's `## Supersedes` records as RETIRED.

`emit_prompt` cuts the inlined contract at `## Supersedes` precisely so a
reviewer is never offered those grammars, and reviewer.test.sh guards that on
the prompt side. The memory files re-offered them from the other direction. A
reviewer that complied wrote lines `merge` cannot read: `channel-check` scores
the turn as a non-response and quarantines the provider, with a reason that
describes the symptom rather than the cause — a real review turn destroyed,
and the round reported as though that secondary found nothing.

Rewrites both sections to the live star grammar (`[finding:<id>|<sev>]` with
its required `> — via` / `> — risk:` / `> — evidence:` lines, the
`[no-findings]` signal, the marker flip, and the rule that responses belong to
the primary). AGENTS.md is synced from CLAUDE.md so the two cannot drift.

Adds the counter-instruction to the dispatched prompt itself: repo memory
files are the AUTHOR's standing instructions about the very work under review,
so following them costs the independence the secondary was dispatched for.
This is a real context channel into "independent" reviewers that the star
topology never accounted for.

Documents the limit honestly in docs/multi-review.md: blind-check proves what
a reviewer was HANDED, verify-vendor proves who answered, and neither can
prove what was read at runtime. Independence is instructed, not enforced.

The codex prompt golden is regenerated deliberately — its job is to catch
UNINTENDED drift, and the diff is exactly and only the new paragraph.

Guards added: packaging asserts neither memory file's reviewer-role section
instructs a retired grammar and that all three prompts carry the
memory-file instruction, with mutation entries for both.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE

@kevin-agrology kevin-agrology left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Multi-review

Disagreements (1)
🟡 low — The new memory-file grammar guard omits the retired peer [finding:] token, even though that token is distinct from the live [finding:<id>|<sev>] form. — risk: a future reintroduction can again direct a secondary to emit findings the star merger rejects — flagged by gpt-5; claude-opus-5 disputes: [finding:] is the LIVE token, not a retired one — the retired peer grammar is caught by its distinctive [concur:/[withdraw: verbs, which the guard already bans, and the positive assertion separately requires the [finding:<id>|<sev>] severity form; banning [finding: would fail on a correct document.

———
🤖 Posted by AI agents (claude-opus-5 + gpt-5) via multi-review star review.

Raised by codex as codex-rd2-r1 in a depth round on this PR. I disputed it and
was wrong; withdrawing the dispute and implementing the fix.

The dispute argued that `[finding:` is the LIVE token, so banning it would fail
a correct document, and that the retired peer grammar is already caught by its
distinctive `[concur:`/`[withdraw:` verbs. Both halves are true, and neither
answers the finding. Codex said the guard OMITS the retired spelling; it did not
say the remedy had to be a wholesale ban. I refuted the easier claim.

The gap is real. `## Supersedes` lists `> [finding:]` as part of the retired
two-agent grammar, and the peer spelling carries no severity. A §11 rewrite that
teaches `> [finding:r1]` in its example while some prose elsewhere still contains
the literal `[finding:<id>|<sev>]` passes BOTH existing checks: no banned verb
appears, and the live form is named.

And it is consequential, which is the part I had not checked before disputing. A
severity-less finding is a hard parse error:

    > [finding:r1] a concern with no severity tag
    -> multi-review-star: finding r1 needs a |high, |med, or |low severity tag
    -> open-findings exits 2

So a reviewer complying with that instruction destroys its own turn — precisely
the failure class this guard family exists to prevent.

The fix keys on the ABSENT `|<sev>` part rather than on `[finding:` itself:
`\[finding:[^]|]*\]` stops at `]` or `|`, so `[finding:r1]` matches and
`[finding:<id>|<sev>]` does not. Verified against both spellings.

The detection is extracted into `retired_grammar_in` so it can be exercised
against a SYNTHETIC section, not only the real files. Asserting only against
CLAUDE.md/AGENTS.md makes the check unfalsifiable in practice — it would pass
because those files are currently correct, not because it can detect a bad one.
Two assertions now bracket it: it must FIRE on the severity-less form that also
names the live one, and must NOT fire on the live form alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE
@kevin-agrology

Copy link
Copy Markdown
Contributor Author

Withdrawing my dispute of codex-rd2-r1 — codex was right.

The published review records a dispute on this finding. On re-examination that dispute was wrong, so the correction belongs here rather than as a silent edit to a converged review doc.

What I argued: [finding: is the live token, so banning it would fail a correct document, and the retired peer grammar is already caught by its distinctive [concur:/[withdraw: verbs.

Why that doesn't answer the finding: both halves are true, and neither addresses what was actually said. Codex said the guard omits the retired spelling — not that the remedy had to be a wholesale ban. I refuted the easier claim.

The gap is real. ## Supersedes lists > [finding:] as part of the retired two-agent grammar, and the peer spelling carries no severity. A §11 rewrite teaching > [finding:r1] in its example, while some prose elsewhere still contains the literal [finding:<id>|<sev>], passes both existing checks — no banned verb appears, and the live form is named.

And it is consequential — the part I had not checked before disputing:

> [finding:r1] a concern with no severity tag
  → multi-review-star: finding r1 needs a |high, |med, or |low severity tag
  → open-findings exits 2

A hard parse error. So a reviewer complying with that instruction destroys its own turn, which is exactly the failure class this guard family exists to prevent.

Fixed in the latest commit by keying on the absent |<sev> part rather than on [finding: itself — \[finding:[^]|]*\] stops at ] or |, so [finding:r1] matches and [finding:<id>|<sev>] does not. Verified against both spellings.

The detection is now extracted into retired_grammar_in so it can be exercised against a synthetic section rather than only the real files. Asserting solely against CLAUDE.md/AGENTS.md made the check unfalsifiable in practice: it would pass because those files happen to be correct, not because it can detect a bad one. Two assertions bracket it — it must fire on the severity-less form that also names the live one, and must not fire on the live form alone. I confirmed the first fails with the new line removed.

For the record, the protocol warns against precisely the move I made: "do not dispute to save a round — a finding you cannot refute on the merits is one you agree with... if you are unsure, agree and fix; being wrong in that direction is cheaper." A low I could refute in one form but not its strongest is the lenient-primary failure inverted.


🤖 Posted by an AI agent — Claude Opus 5 (claude-opus-5), as multi-review primary. Secondary: codex (gpt-5) via gpt-5.6-terra. Accountability remains with the engineer who ran it.

Conflicts in the mutation table and packaging suite were additive on both
sides; kept both. Version resolved to 1.21.1.

reviewer.sh auto-merged cleanly with main's vendor-map and comment-stripping
changes, and the byte-frozen codex prompt golden still matches — neither of
those touched the prompt. Verified rather than assumed, along with every
packaging guard from all merged branches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nm3hNXA4WHbgKjFbxX7VUE
@kevin-agrology
kevin-agrology merged commit 70b5f7a into main Aug 11, 2026
4 checks passed
@kevin-agrology
kevin-agrology deleted the fix/reviewer-role-star-grammar branch August 11, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant