Skip to content

feat(ship-check): port output-honesty contracts from plan-review - #9

Merged
aliasunder merged 3 commits into
mainfrom
feat/output-honesty-contracts
Sep 4, 2026
Merged

feat(ship-check): port output-honesty contracts from plan-review#9
aliasunder merged 3 commits into
mainfrom
feat/output-honesty-contracts

Conversation

@aliasunder

Copy link
Copy Markdown
Owner

What

Ports two output-honesty contracts from the plan-check plugin's plan-review skill back into all four ship-check phase skills (pr-review, code-quality, test-audit, bug-check) and their agents:

  • Verification-basis line — every phase summary (default and comment mode, including the posted review body) names the PR head SHA actually reviewed. A review that doesn't say what it checked is indistinguishable from one that checked nothing. This is also the SHA pr-monitor's pre-merge delta review keys off, so phases now report it explicitly instead of it being reconstructed.
  • Proof-of-dismissal closing — each summary closes with one line per suspicion seriously considered and dropped, with the reason (or "none"). Extends bug-check's "No silent skipping" (confirmed findings must be reported) into the negative space: cleared suspicions must be reported too, so "no findings" is distinguishable from "didn't look."

What was deliberately NOT ported

plan-review's verification-state pinning (review against the repo as of the plan's date, not HEAD) doesn't transfer: ship-check reviews a PR head, which is already a pinned state. Only the output half — stating the SHA — carries over.

Changes

  • 4 phase skills: "Output honesty (both modes)" block in the report section; Reviewed at <HEAD_SHA> added to comment-mode review body templates; bug-check's inline summary template gains both lines
  • 4 agents: Reviewed at and Dismissed lines added to default- and comment-mode output templates

🤖 Generated with Claude Code

Every phase summary now names the PR head SHA it actually reviewed (a
review that doesn't say what it checked is indistinguishable from one
that checked nothing — and pr-monitor's delta review keys off this SHA),
and closes with proof-of-dismissal one-liners for suspicions considered
and dropped, so a clean bill is distinguishable from an unexamined diff.
Applied to all four phase skills (report sections + comment-mode review
bodies) and their agents' output templates. Verification-state pinning
was not ported — a PR head is already a pinned state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread plugins/ship-check/skills/bug-check/SKILL.md Outdated
@umm-actually

umm-actually Bot commented Sep 4, 2026

Copy link
Copy Markdown

umm-actually re-reviewed at 755eee9

1 new finding(s) posted (4 tracked finding(s) across all runs).

Context notes
  • Priority docs already in context: plugins/ship-check/agents/bug-checker.md, plugins/ship-check/agents/code-quality-reviewer.md, plugins/ship-check/agents/pr-reviewer.md, plugins/ship-check/agents/test-auditor.md

umm-actually · deepseek/deepseek-v4-flash-0731

…views

The dismissal list only reached the orchestrator summary — in comment
mode the PR-visible review body had no dismissal line, and a 0-findings
phase skipped posting entirely, so the artifact meant to distinguish a
clean diff from an unexamined one never landed on the PR. Added the
dismissal line to all four review-body templates and narrowed the skip
rule: skip only when there are neither findings nor dismissals.
(umm-actually finding on PR #9.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@umm-actually

umm-actually Bot commented Sep 4, 2026

Copy link
Copy Markdown

Add Reviewed-at and Dismissed to the pipeline's own status report
Low severity · correctness · low confidence

plugins/ship-check/skills/ship-check/SKILL.md:890 — beyond the diff's line ranges, in code the changes touch or depend on.

Pre-existing analog gap: the PR establishes a reviewed-SHA plus proof-of-dismissal contract for the phase skills, but the pipeline's final status snapshots (both default and comment mode) do not emit Reviewed at <SHA> or a Dismissed line. A consumer reading only the final report still cannot distinguish a reviewed PR from an unexamined one.

Failure scenario: A user reads only the final Ship check complete status snapshot as instructed. Even though each phase reported its SHA and dismissals to the chat stream, the snapshot itself shows neither, so the reader sees a concluding verdict while the underlying high-water and inspection claims are absent/incomplete at the point it's actually surfaced.

Suggested fix
Add `- Reviewed at: <SHA>` and `- Dismissed: …` to the ship-check output templates' Reporting section, mirroring the phase reports in this PR, so the pipeline conclusion carries the same two-line contract as the phases it summarizes.

umm-actually · deepseek/deepseek-v4-flash-0731

Comment thread plugins/ship-check/skills/bug-check/SKILL.md
…nism

The honesty block claimed the pre-merge delta review keys off the
phase-reported SHA; the orchestrator actually records the branch head
itself at Phase 4 close and nothing reads the phase's Reviewed-at field.
Reworded to what the reported SHA really provides: a cross-check of what
the phase saw against the orchestrator's own baseline.
(umm-actually finding on PR #9.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@umm-actually

umm-actually Bot commented Sep 4, 2026

Copy link
Copy Markdown

Remove spliced-in monitoring-loop content from the Phase 2 code-quality reviewer
Medium severity · correctness · medium confidence

plugins/ship-check/agents/code-quality-reviewer.md:100 — beyond the diff's line ranges, in code the changes touch or depend on.

The Phase 2 code-quality reviewer agent (code-quality-reviewer.md) has a large block of pr-monitor monitoring content spliced into its body, sitting between the orientation step (Load sequential thinking) and the Output Format section (new-file lines ~90–118, outside this PR's hunks). The agent's true job is a one-shot convention pass that loads vault standards, fixes findings, runs tests, commits, and summarizes — but the embedded block instructs it to run a continuous PR-monitoring loop: re-fetch threads and issue comments until all bot/Claude/human threads resolve, call ScheduleWakeup with delaySeconds: 240, “continue monitoring after reporting”, and “never auto-terminate — not after clean checks, not after silence.” That check-list of monitoring preconditions and the never-stop directive are pr-monitor content that directly contradicts the stated Core Responsibilities (“5. Run tests, commit, and push fixes”) and the default-mode report template. Any dispatch of this agent would load conflicting instructions about whether the task ends at the summary or continues indefinitely, and the monitoring checklist (issue-coverage counts, “If code was pushed during this run…”) references machinery the code-quality phase has no reason to invoke.

Failure scenario: A user dispatches the code-quality reviewer for ship-check Phase 2 (or standalone). The agent reads its own body, reaches the “Continuous monitoring” block after the orientation, and follows the pr-monitor directives — scheduling 240-second wakeups, re-fetching the PR, scrubbing threads, and refusing to terminate “for any other reason” than an explicit user “stop” or merge/close. Instead of completing a finite code-quality pass (fix findings, run tests, commit, report), it enters an indefinite monitoring loop on a PR it was only asked to review, posting or re-evaluating comments. Milder case: the contradictory instructions just garble the agent's behavior at the boundary between “summarize” and “keep monitoring,” so the phase either hangs until the user notices or produces PR-side activity the caller never asked for.

Suggested fix
Move the pr-monitor block (thread-resolution prerequisites, issue-coverage checks, ScheduleWakeup / never-auto-terminate loop, “pipeline context” notes about the delta review) out of the code-quality reviewer into the actual monitoring agent where it belongs, and restore the code-quality reviewer's body to end with its own monitor-free instruction — i.e., after the orientation section, jump straight to the Phase-2 procedure and Output Format, with no reference to ScheduleWakeup or the “never auto-terminate” rule.

umm-actually · deepseek/deepseek-v4-flash-0731

@aliasunder

Copy link
Copy Markdown
Owner Author

Re umm-actually finding on code-quality-reviewer.md (comment 5547436971, "spliced-in monitoring-loop content"): false positive — the described content does not exist in the file. Lines 90–118 of code-quality-reviewer.md contain the sequential-thinking triggers, Fixing and Committing, Attribution, and Comment-mode sections; a grep of the full 154-line file for ScheduleWakeup, monitoring, auto-terminate, wakeup, and issue-comment coverage returns zero matches on this branch. The quoted directives ("delaySeconds: 240", "never auto-terminate") are pr-monitor SKILL.md's own content — the finding appears to have conflated the two files.


🔍 ship-check · pr-monitor · claude-fable-5

@aliasunder
aliasunder merged commit 53f1fc3 into main Sep 4, 2026
7 checks passed
@aliasunder
aliasunder deleted the feat/output-honesty-contracts branch September 4, 2026 23:08
@aliasunder

Copy link
Copy Markdown
Owner Author

Re umm-actually finding 5547367668 (Reviewed-at/Dismissed missing from the pipeline's own status snapshots): valid pre-existing analog gap — it landed just before this PR merged, so it's resolved in follow-up #10: both orchestrator reporting templates now carry the two-line contract (default mode reports the delta-review high-water SHA, comment mode the reviewed head, plus a Dismissed roll-up of the phases' dismissal lines).


🔍 ship-check · pr-monitor · claude-fable-5

aliasunder added a commit that referenced this pull request Sep 4, 2026
…napshot (#10)

* feat(ship-check): carry Reviewed-at and Dismissed into the pipeline snapshot

The phase skills report their reviewed SHA and proof-of-dismissal (PR
\#9), but the orchestrator's Ship-check-complete snapshots — the report a
consumer actually reads — carried neither, so the concluding verdict
still couldn't distinguish a reviewed PR from an unexamined one. Both
reporting templates gain the two lines: Reviewed-at (the delta-review
high-water mark in default mode, the reviewed head in comment mode) and
a Dismissed roll-up of the phases' dismissal lines.
(umm-actually finding on PR \#9, post-merge.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(ship-check): comment-mode Reviewed-at must not assume one head SHA

Phases run sequentially in comment mode and the head can move between
dispatches, so a single reported SHA overstates what earlier phases saw.
Reworded to the Phase 4 completion head with per-phase SHAs when the
head moved — each phase already reports its own.
(umm-actually finding on PR \#10.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant