Skip to content

check-pr-fully-clean: skip agent-disclosed replies in the verdict scan - #4922

Merged
d-morrison merged 5 commits into
mainfrom
claude/project-thread-aj2ep4
Nov 3, 2026
Merged

d-morrison merged 5 commits into
mainfrom
claude/project-thread-aj2ep4

Conversation

@d-morrison

Copy link
Copy Markdown
Collaborator

Closes #4826.

Why

An agent replying to a review under the maintainer's login often quotes the reviewer's "Blocking" label.
check-pr-fully-clean.py admitted every non-bot not-clean comment, so that quote became a standing maintainer veto that no later clean review could clear.
Measured on Morrison-Lab/mds#599: two clean Claude reviews of the head could not pass the gate, and the only way out was editing the agent's own comment, which edits evidence.

What

check_review_comments now skips a comment only on positive evidence that it is a reply:

  • it ends with the Claude Code disclosure footer;
  • it has no review shape the scan admits from a human login and no review-data payload;
  • once its disposition lines are removed, the rest no longer reads as not-clean.

A disposition line is removed only when its tail is on an allowlist, so the skip fails closed:

  • after any disposition word, nothing, punctuation, or a commit reference;
  • after rebutted, declined or deferred only, one closed rebuttal phrase ("The line already breaks after the comma.", "The reviewer misread the diff.") holding no risky word.

Anything else keeps the whole line, label included.

Residual risk

The allowlist is closed, so the remaining way to drop a real veto is an agent reply that says "rebutted" or "declined" with a closed rebuttal phrase that is false.
That is accepted as the price of not editing the agent's own comments; say so if you would rather not accept it.

Verification

  • scripts/test_check_pr_fully_clean.py: 1271 passed, 0 failed, including about 40 negative controls built from nine rounds of adversarial probes.
  • Live run on mds#599 with the branch checker: the false maintainer veto is gone (per-reviewer: Claude=clean).

Generated by Claude Code

An agent replying to a review under the maintainer's login often quotes
the reviewer's "Blocking" label. The verdict scan admitted every non-bot
not-clean comment, so the quote became a standing maintainer veto that no
later clean review could clear, and the only way out was editing the
agent's own comment (measured on Morrison-Lab/mds#599, where two clean
Claude reviews of the head could not pass the gate).

Skip a comment only on positive evidence that it is a reply: it ends
with the Claude Code disclosure footer, carries no review shape the scan
admits from a human login (report structure, a named review head) and no
review-data payload, and once each disposition label is removed (a bold
finding label followed at once by rebutted, deferred, done, ..., or
"Addressing the blocking finding") the rest of the body no longer reads
as not-clean. The rest is classified as a whole, so a relayed "Needs
more work", a second finding after a disposition, or a verdict spread
over several lines still blocks.

What may follow a disposition is an allowlist, so the skip fails closed:
nothing, punctuation or a commit reference after any disposition, or,
after rebutted, declined or deferred only, an explanation holding no
negation, contrast, modal, open-work word, number or file path.
Anything else keeps the whole line, label included. The residual risk
is an explanation that states an open defect in none of those words,
which is then dropped; that is accepted as the price of not editing the
agent's own comments, and is stated here so a reviewer can refuse it.

Closes #4826.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019xfc9pMBq15Hq3DgNBx1ox
…lined/deferred

An explanation after a weighed disposition now drops its line only when
it reads as a rebuttal of the reviewer's reading ("The line already
breaks ...", "The reviewer misread ...") and holds no risky word, so
"Persists on the head." and "Agree with the reviewer." keep blocking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019xfc9pMBq15Hq3DgNBx1ox
Each rebuttal opener is now a closed phrase followed by at most six
plain words, and confirmations (valid, true, confirmed, persists, ...)
are risky words, so "The finding is already valid." keeps blocking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019xfc9pMBq15Hq3DgNBx1ox
Six free words after a closed phrase still admitted "The check handles
nothing."; only an optional "after/before/at the X" may follow now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019xfc9pMBq15Hq3DgNBx1ox
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Claude finished review — View run


That matches the saved diff file's length (no additional content I'm missing). I've completed a thorough review of this PR.

Review: #4922 — check-pr-fully-clean: skip agent-disclosed replies in the verdict scan

Summary of change. scripts/check-pr-fully-clean.py's check_review_comments previously read any non-bot comment under a maintainer login for a verdict, so an agent's ARD reply that quotes a reviewer's "Blocking" label (while disclosing itself and rebutting/deferring it) became a permanent maintainer veto no later clean review could clear (closes #4826). The fix adds _is_agent_disclosed_reply, which skips such a comment only when: it ends with the Claude Code disclosure footer; it carries no review shape/review-data payload the scan would otherwise admit from a human login; and after stripping only disposition lines whose tail matches a closed allowlist, the remainder still doesn't read as not-clean.

Verification performed.

  • Read the full diff (scripts/check-pr-fully-clean.py +139, scripts/test_check_pr_fully_clean.py +154).
  • Ran python3 scripts/test_check_pr_fully_clean.py: 1271 passed, 0 failed, including the ~40 new negative controls added for this feature (bare robot emoji, mid-body quoted footer, relayed reviews naming a head, open-claim tails after disposition words, multi-line review-data payloads, etc.).
  • Manually traced several regex paths (_REPLY_DISPOSITION_LINE_RE, _disposition_drops_line, _REPLY_BARE_TAIL_RE, _REPLY_REBUTTAL_TAIL_RE, _REPLY_RISKY_TAIL_RE) against specific test fixtures to confirm the match/skip logic behaves as commented.
  • Confirmed the new branch is correctly placed after the is_bot_author check and before the verdict == "not-clean" branch in check_review_comments (scripts/check-pr-fully-clean.py:4291-4298), so it only applies to non-bot, non-already-clean-superseding comments as intended.
  • Compared against the design the linked issue #4826 sketched (blanket-skip any agent-disclosed comment) — the shipped design is strictly more conservative: it still blocks if the remainder reads not-clean, which is the safer, fail-closed choice given this gate controls merges.
  • Checked for non-ASCII bytes / stray em-dashes in the diff (none; all em dashes are either \u2014 escapes inside regex literals or the repo's --- ASCII convention) — consistent with CLAUDE.md's shared/coding/ascii-punctuation-in-source.md.
  • Ran python3 -m pyflakes on both changed files and diffed against origin/main's versions: all four warnings reported are pre-existing and unrelated to this diff's lines.
  • py_compile succeeds on both files.
  • No prior claude[bot] review comments exist on this PR (gh pr view 4922 --comments returned none), so this is a first-round review, not a confirming one.

Findings. None. I found no bugs, no CLAUDE.md violations, and no hallucinated APIs/references. The function-length and tidyverse-style sections of the lab manual target R code; this is infrastructure Python in ai-config, where the applicable standard is this repo's own extensive regex-test-driven convention, which the PR follows (comments justify every regex choice, and the new logic is tested from both the clean and not-clean directions).

Verdict

Ready for merge

Structured Review Data (JSON)
{
  "schema_version": "1.1",
  "reviewer": "claude",
  "commit_sha": "56ec81b809d008b9791ce4fd5f92a97dcdafa4a4",
  "verdict": "CLEAN",
  "findings": [],
  "detailed_assessment": "Traced check-pr-fully-clean.py's new _is_agent_disclosed_reply and its disposition-line regexes against the full 1271-test suite (0 failures) and found no logic errors in the allowlist, tail-risk, or rebuttal-phrase matching paths.",
  "holistic_assessment": "The change is narrowly scoped to issue 4826, placed correctly relative to the existing bot-author and verdict-supersession branches, more conservative than the issue's own fix sketch, and verified live on mds#599 per the PR description."
}

Reviewed commit: 56ec81b

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

💰 Cost: $0.9724 (review) — run

Copy link
Copy Markdown
Collaborator Author

Merging under ai-config's standing mwc grant: CI is green, the Claude review on head 56ec81b is clean, and check-pr-fully-clean.py exits 0.

An independent adversarial review (a separate Claude subagent) ran nine rounds and returned CLEAN on d862ed5; the only later commit escapes one em dash for the ASCII check.
The cross-model review gate is waived because no qualifying reviewer could run: OpenCode Go (opencode.ai/zen/go/v1/chat/completions, deepseek-v4-pro) returned HTTP 429 at 2026-11-03 12:21 PST, and no other cross-harness reviewer is reachable from this cloud session.

Posted by Claude Code (AI agent) --- not written by a human.


Generated by Claude Code

@d-morrison
d-morrison merged commit 9329013 into main Nov 3, 2026
20 checks passed
@d-morrison
d-morrison deleted the claude/project-thread-aj2ep4 branch November 3, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

check-pr-fully-clean.py: the verdict scan reads agent-disclosed comments under the maintainer's login as human verdicts

2 participants