What happened
On PR #4417, the review agent produced two categories of false positives across 4 review rounds (~$28 total). First, in review round 2, the agent surfaced a [medium] scope-completeness finding claiming the PR could not close issue #4046 because npm publish and CHANGELOG tasks were incomplete. The human explained that npm publish is handled by the repo's automated Version Packages workflow, not individual PRs, and the changeset entry already existed. Second, across all 4 rounds, the agent repeatedly flagged stale references in documentation files (design.md, proposal.md, annotation-scheme/spec.md) that were outside the issue's authorized scope. The human documented a clear scoping rule: only files cited by issue #4046 should be updated in this PR.
What could go better
The review agent lacks two pieces of repo-specific context that caused repeated noise. (1) It does not know the repo uses changeset-based automated releases via Version Packages PRs, so it incorrectly flags missing npm publish steps as incomplete scope. (2) It does not understand that PRs may intentionally limit changes to files and components cited in the linked issue, so it flags stale references in uncited files as defects rather than informational observations. Confidence: high for the release workflow gap (clearly documented in the human's triage). Moderate for the scope convention — this may be a team preference rather than a hard rule, and different PRs may have different scope policies. These findings re-appeared across rounds, which is separately tracked by fullsend-ai/fullsend#3073 and #3515.
Proposed change
Add two context paragraphs to .fullsend/AGENTS.md. (1) Under a 'Release workflow' heading: "This repo uses changesets for versioning. Individual PRs include a .changeset/*.md entry declaring the bump level. npm publish is handled automatically by the Version Packages workflow — do not flag PRs for missing publish or CHANGELOG steps." (2) Under a 'Review scope' heading: "When a PR references a specific issue (Closes #XXXX), changes may be intentionally limited to files and components cited in that issue. Stale references or inconsistencies in uncited files should be noted as informational observations, not flagged as defects requiring fixes in the current PR."
Validation criteria
The next 5 review agent runs on PRs in this repo that reference specific issues should produce zero false positives about missing npm publish steps or Version Packages workflow, and should distinguish in-scope defects from out-of-scope informational observations in their findings.
Generated by retro agent from #4417
What happened
On PR #4417, the review agent produced two categories of false positives across 4 review rounds (~$28 total). First, in review round 2, the agent surfaced a [medium] scope-completeness finding claiming the PR could not close issue #4046 because npm publish and CHANGELOG tasks were incomplete. The human explained that npm publish is handled by the repo's automated Version Packages workflow, not individual PRs, and the changeset entry already existed. Second, across all 4 rounds, the agent repeatedly flagged stale references in documentation files (
design.md,proposal.md,annotation-scheme/spec.md) that were outside the issue's authorized scope. The human documented a clear scoping rule: only files cited by issue #4046 should be updated in this PR.What could go better
The review agent lacks two pieces of repo-specific context that caused repeated noise. (1) It does not know the repo uses changeset-based automated releases via Version Packages PRs, so it incorrectly flags missing npm publish steps as incomplete scope. (2) It does not understand that PRs may intentionally limit changes to files and components cited in the linked issue, so it flags stale references in uncited files as defects rather than informational observations. Confidence: high for the release workflow gap (clearly documented in the human's triage). Moderate for the scope convention — this may be a team preference rather than a hard rule, and different PRs may have different scope policies. These findings re-appeared across rounds, which is separately tracked by fullsend-ai/fullsend#3073 and #3515.
Proposed change
Add two context paragraphs to
.fullsend/AGENTS.md. (1) Under a 'Release workflow' heading: "This repo uses changesets for versioning. Individual PRs include a.changeset/*.mdentry declaring the bump level. npm publish is handled automatically by the Version Packages workflow — do not flag PRs for missing publish or CHANGELOG steps." (2) Under a 'Review scope' heading: "When a PR references a specific issue (Closes #XXXX), changes may be intentionally limited to files and components cited in that issue. Stale references or inconsistencies in uncited files should be noted as informational observations, not flagged as defects requiring fixes in the current PR."Validation criteria
The next 5 review agent runs on PRs in this repo that reference specific issues should produce zero false positives about missing npm publish steps or Version Packages workflow, and should distinguish in-scope defects from out-of-scope informational observations in their findings.
Generated by retro agent from #4417