Repository navigation
🐛 fix(review): A review a rate limit kept from reading the PR is no verdict and is asked again, not a human decision - #16994
Conversation
…-asked A reviewer that could not read the PR because of a rate limit (HTTP 429, GitHub's rate limit, or the hive proxy's agents.github_api_hourly_cap) returned requires_human, as the read instruction tells it to. The relay recorded that as a verdict: the security perspective's copy counted as "the security review withheld approval", the PR parked at 085 and the owner's delegate asked a person about a security finding that did not exist (projectbluefin/testsuite#1116). On a contributor PR without the security copy, the same verdict resolved hands-free to approve on an unread head. review.UnreadableReviewReason recognises such a report (requires_human, no in-scope finding, no high-risk marker, summary says both "not read" and a rate-limit cause). recordReviewVerdict skips it, so the dispatch entry stays open, expires after review.pending_ttl and the perspective is asked again. Perspectives that did read the PR are recorded as usual. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
The relay change does what it says: recordReviewVerdict now continues before writeOneVerdict (src/pkg/github/review_verdict_relay.go:114-125), so no report file lands for that perspective, the pending entry is never cleared, and it expires and re-dispatches. I read writeOneVerdict and review_request_watcher.go:700 to confirm nothing else closes the dispatch row. go build ./pkg/review/ ./pkg/github/ passes on the head.
One finding, and I think it needs a change before this lands:
UnreadableReviewReason drops explicit decision questions on PRs whose subject is rate limiting — src/pkg/review/unreadable.go:57-76, medium.
The classifier is keyword-only. It checks Verdict, DecisionRiskHigh, in-scope findings, and then requires a rate-limit term (rateLimitCauseRe, line 44) plus any one of unreadableReviewTerms (lines 32-37, which include the very general "not reviewed", "could not review", "unable to review") anywhere in Summary + DecisionQuestion. It never looks at r.DecisionRequested (src/pkg/review/review.go:163), even though the relay has already set that from the <!-- hive-verdict: needs-decision --> marker at review_verdict_relay.go:103-108 before this check runs.
Consequence: a reviewer who read a PR that changes agents.github_api_hourly_cap, the proxy's 429 handling, or GitHub rate-limit retries — this repo has many — and raises a genuine question such as "@owner should decide whether the hourly cap may be bypassed here; I could not review the retry budget against prod numbers" produces text matching rate[- ]?limit|hourly cap and could not review. That report is silently discarded, the PR is never parked at 085, and the perspective is re-asked until max_redispatch is exhausted. The same words on this PR's own review would trip it. The only escape hatch is hive-decision-risk: high, which the policy tells reviewers to add only for risk classes, not for every question.
Suggested fix, smallest form: return "", false when r.DecisionRequested is true (a reviewer who asked a specific question read enough to ask it). Optionally tighten the match so the rate-limit cause and the unreadable term must be in the same sentence, or require the summary to carry no @login mention. A test for a DecisionRequested report with rate-limit wording would fail without that guard; review_verdict_unreadable_test.go currently has no needs-decision case.
Open question, not a finding: the "could not read … 429" comment still posts to the PR (review_request_watcher.go:690-700 records the verdict after the comment lands). After this change that comment has no verdict behind it and a re-asked reviewer may post another. Probably acceptable, but worth a sentence in the doc paragraph.
Doc paragraph in src/docs/review-swarm.md:579 and the changelog fragment accurately describe the implemented behaviour.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
… as unreadable The hive reviewer on hivecommons#16994 pointed out that UnreadableReviewReason matched on words alone. A reviewer that read a PR about rate limiting and asked the owner a question ("I could not review the retry budget …; should the hourly cap be bypassed here?") would have had its question discarded and the perspective re-asked until max_redispatch ran out. A report carrying the needs-decision marker is now never treated as unreadable: a reviewer that asks the owner something read enough to ask it. The read instruction tells an unreadable reviewer to return a plain requires_human, so the testsuite#1116 shape is unaffected. The doc also notes that a "could not read" comment already posted stays on the PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Thanks, this one is right. Fixed in aa16b52.
|
|
🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
There was a problem hiding this comment.
Looks correct to me. Read the change against the v6 tree at the merge-base (8325a54), not just the diff.
What I verified
src/pkg/github/review_verdict_relay.go:114-125: thecontinueruns beforewriteOneVerdict, andwriteOneVerdict(line 421ff,verdictDispatchAuthorizedAt) is the only place a report binds to its dispatch record and lands in the collector's report dir. Nothing after the loop touches the pending list, so a skipped perspective genuinely leaves its dispatch row open for thepending_ttlexpiry path described inreview-swarm.md. The "staysincompleteuntil re-asked" claim follows from the same mechanism.- The filter sits after
decisionRequestedis folded into the report (lines 101-106), so aneeds-decisionmarker in the body correctly disqualifies the report from being dropped —UnreadableReviewReasonreturns early onDecisionRequested/DecisionRiskHigh(unreadable.go:65). Consequence: ther.DecisionQuestionconcatenation atunreadable.go:73can never contribute text, since a non-empty question impliesDecisionRequested. Harmless; noting it only so nobody later reads it as a live code path. - The
tighteningdiff fetch (line 95-98) still runs for an all-unreadable array becausereviewReportsHoldForHumanseesrequires_humanbefore the filter. One extra server-side REST call on the App token, not the agent's budget — not a defect. - Classifier is appropriately two-sided (an "unread" term and a rate-limit cause, both required), and the
TestUnreadableReviewReasonLeavesRealHoldsAlonecases cover the shapes that worried me (too-large diff, a review about 429 handling, in-scope finding present). Residual false-drop risk is bounded: a wrongly dropped hold is re-asked after 45m and still reaches a person viareview-dispatch-exhaustedaftermax_redispatch.
CI: the two red native-queue-policy runs fail in the workflow's own script — TypeError: isCosmeticLabelFlip is not a function in step "Replay last verdict for cosmetic label flips" (run 42386602825). This PR touches no workflow file, so that is infrastructure, not this diff. Everything else is green.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
…verdict Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
The hive's GitHub App lacks |
|
Needs a decision, not a merge: Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (6 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Conclusion: correct for the incident it targets; one bounded false-drop risk worth knowing about, not a blocker.
What I verified in the tree (not from the body):
pkg/github/review_verdict_relay.go— the new check runs after thedecisionRequestedoverride setsDecisionRequested=true, so a combined review that asked a real question is never dropped.continueskipswriteOneVerdict, so the pending row is untouched andwakestays false — matching the "expires afterpending_ttl" story.pkg/review/review.go:1298InScopeFinding,review.go:162-173DecisionRequested/DecisionQuestion/DecisionRiskHighall exist onmain; the guard's inputs are real.pkg/review/dispatch.go:35,39—DefaultPendingReviewTTL = 45m,DefaultMaxRedispatch = 3. That cap is what makes this safe: a report wrongly classified as unreadable costs at most ~3×45m before the head lands on "dispatch exhausted; waiting on a human" rather than being lost.- The two relay tests exercise the real
recordReviewVerdict→Collect→PlanDispatchchain, and the second one pins the important property: a read perspective in the same combined review is still recorded and the head readsincomplete, notapprove.
Open note (low) — pkg/review/unreadable.go, UnreadableReviewReason: this is a free-text classifier. A legitimate requires_human with no in-scope finding and no decision marker whose summary happens to contain both an unread-term and a rate-limit term is dropped — the test itself documents this ("test premise: without the decision marker this wording reads as unreadable"). Today's reviewer prompts (prompts_instructions.go:86) only tell the reviewer to say so, so the signal has to be inferred from prose. If this recurs, a structured field on the verdict (e.g. unreadable: rate_limit) emitted by the reviewer would make the relay decision exact instead of heuristic; the prompt text would then tell reviewers to set it. Not needed for this fix.
Docs: src/docs/review-swarm.md paragraph matches the implemented conditions one for one. Body and diff agree. I have not approved or changed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
…unread reports produced (#17082) #16994 made the relay skip a requires_human that only says a rate limit kept the reviewer from reading the PR. Report files written before it are still on disk and still aggregate to requires_human on every collect, so projectbluefin/testsuite#1116 stayed at 085 at head ae22434 after the upgrade. The verdict collect now applies UnreadableReviewReason too and sets those files aside as missing. The merge withdraws a stored requires_human whose every requires_human perspective, human summary and security hold comes from a set-aside report and that has no decision question, in-scope finding or exhausted fix cycle. The head reads incomplete (or has no verdict) and the planner re-asks the missing perspectives through the normal pending_ttl / max_redispatch path. A head with a set-aside perspective is never approve while it is owed, so the contributor rule's hands-free approve cannot mark an unread security review merge-eligible. Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…viewed head (#17087) contributorHandsFreeAggregate resolved a bare non-security requires_human (or decision question) on a contributor PR to approve + merge_eligible with no coverage check. Since #16994 the relay drops a rate-limited "could not read" review, so a head whose security review was dropped became merge-eligible on a correctness product question alone. A dead reviewer kick left the same hole. The approve branch now needs every perspective the head is owed (the configured set, capped by max_perspectives_per_pr), the same count the all-approve path uses. Short of that the head is incomplete with "review coverage incomplete: missing ...", and the planner re-asks only the missing perspective. The fixer lane, fix-cycle cap, decision questions and security findings are unchanged. Tests that approved hands-free on one report under the five-perspective default now add approving siblings or collect with max_perspectives_per_pr: 1. Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
… path needs full coverage (#17092) On a contributor PR, a perspective's reject next to another perspective's non-security decision question went to the hands-free rule, which made the question advisory and dropped the reject with it: approve and merge-eligible (or the fixer lane with a change request). The head now gets what the reject alone gives it, reject, and the question stays in reasons. On a hive-authored PR, the advisory approves for a non-risk decision question and for a bare non-risk requires_human never checked coverage, so a head whose security review never arrived (a dead kick, or a rate-limited review the relay no longer records, #16994) was merge-eligible. With a perspective missing the head now reads incomplete and the planner re-asks only that perspective. A fully reviewed head still resolves as advisory. Refs #16994 Refs #17082 Refs #17087 Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
…s no verdict and is asked again, whatever its wording (#17376) * fix(review): a review the GitHub proxy cut short is no verdict, whatever its wording When the reviewer reached its agents.github_api_hourly_cap, the proxy refused its reads and the session finished with requires_human in its own words: "GitHub reads stopped at the user's request after the hourly cap; ... could not be verified" (projectbluefin/utah-packages#959), "not assessed: GitHub API cap prevented reading" (projectbluefin/utah#1161). review.UnreadableReviewReason (#16994) only knows "could not read"-style wording, so these were recorded and parked the PRs at 085. The proxy now remembers, per agent, when it last refused a request for the hourly cap or the reserve floor (AgentGitHubRefusedSince). The review relay asks it: a requires_human with no in-scope finding, no high-risk marker and no already-landed PR, from an agent refused at or after the perspective's dispatch time, is not recorded. The dispatch row stays open and the perspective is re-asked after review.pending_ttl, as in the #16994 case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com> * fix(review): an expiry the agent GitHub API cap caused does not spend max_redispatch The previous commit sets aside a requires_human from a session the proxy refused at the agent cap, and the perspective is asked again after review.pending_ttl. But the expiry that follows still counted against review.max_redispatch (3), so a reviewer capped for about 2h15m on one head (three 45m TTLs) still parked it as review-dispatch-exhausted and a person was asked about a PR nobody had read. The proxy now also remembers when each refusal said the agent could read again (its Retry-After: the sliding hour freeing a slot, or the shared bucket's reset), and exposes it as GitHubProxy.AgentGitHubRefusal. The same process-wide hook feeds the relay and, through review.DispatchOptions.AgentGitHubRefusal, the planner. When every row of an expiry went to an agent the proxy refused after the row was sent, the planner waits for the refusal to clear (at most an hour past the TTL), then re-asks and counts the expiry in the new cap_redispatches[] instead of redispatches[]. That count is bounded separately (DefaultMaxCapRedispatch = 8) and ends in the same review-dispatch-exhausted hold, so a reviewer that can never read still reaches a person instead of looping. Any other expiry counts as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com> --------- Signed-off-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Danathar <Danathar@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
When a reviewer cannot read a PR because it hit a rate limit (an HTTP 429, GitHub's rate limit, or the hive's own per-agent
agents.github_api_hourly_cap), the hive no longer turns that into arequires_humanverdict. The relay drops the report. The perspective's dispatch entry stays open, expires afterreview.pending_ttl(45m by default), and the planner asks again. No person is asked, and no "security review withheld approval" is claimed for a review that never happened.What went wrong
On projectbluefin/testsuite#1116, the delta review of head
ae22434hit the agent API cap. All five perspectives came backrequires_humanwith summaries like "Could not read the delta d397ece..ae22434: every GitHub read returned HTTP 429 (hive agent API hourly cap, 181-211/400) … Not reviewed", and the security one said "Delta not readable (GitHub API 429 cap, no local checkout). Not reviewed at this head."The read instruction tells a reviewer that cannot read the diff to return
requires_human, and the relay recorded those reports as verdicts. After that:review.IsSecuritySensitiveFindingcounts anyrequires_humanfrom the security perspective as a security finding, whatever it says. So the aggregate parked the PR, and the adopted-PR stall moved it tostage/085-needs-decision(10:58Z, stall, needs-decision).adopted-security-findingrule and asked a person to "read the finding on testsuite PR 1116 (the security review withheld approval)" (11:05Z, comment).The lens mapping itself is right: the security perspective really did return
requires_human. What's wrong is that a "could not read" report was accepted as a judgement at all. There's a quieter version of the same bug: on a contributor PR, a lone unreadable correctnessrequires_humanwith no security finding resolves hands-free to approve (seeTestRecordReviewVerdictIsCollectable). That would mark a head nobody read as merge-eligible.The cap is the hive's own (
crux/proxy/agent_github_budget.go, a one-hour sliding window per agent), so the hive can tell what happened without relying on the model to change.What changes
review.UnreadableReviewReasonrecognises a report that isrequires_human, has no in-scope finding, nohive-decision-risk: highmarker and no explicit decision question (the needs-decision marker: a reviewer that asks the owner something read enough to ask it), and whose summary (or decision question) says both that the change was not read and that a rate limit was the cause (429, "rate limit", "too many requests", "hourly cap", "API cap" and similar).recordReviewVerdictskips such a report and logsunreadable review not recorded as a verdict. Other perspectives in the same combined review that did read the PR are recorded as usual, and the head reads asincompleteuntil the re-asked ones report.requires_humanfor any other reason, such as a diff too large to read, still parks the PR.Files
src/pkg/review/unreadable.go: the classifier.src/pkg/github/review_verdict_relay.go: the relay skips unreadable reports.src/docs/review-swarm.md: a paragraph under pending-review expiry.changelog.d/fixed-review-rate-limit-no-verdict.mdTests
TestRecordReviewVerdictRateLimitedReviewIsNoVerdictAndRetries(pkg/github) uses the five real testsuite PR 1116 summaries. It checks that nothing is recorded, that there is norequires_humanaggregate and no security finding, that there is no re-kick and no human hold inside the TTL, and that one combined retry for all five perspectives goes out after the TTL. It fails on v6 tip (5 verdicts recorded) and passes with the fix.TestRecordReviewVerdictKeepsReadPerspectivesAndDropsRateLimitedOnes(pkg/github): a real style approve is kept, and the rate-limited correctness report is dropped, givingincomplete, not approve. It fails on v6 tip and passes with the fix.TestUnreadableReviewReasonRecognisesRateLimitedReviews/TestUnreadableReviewReasonLeavesRealHoldsAlone(pkg/review) cover the classifier's edges: approve and changes_requested, too-large diffs, a review about rate-limit code, real security findings, in-scope findings, the high-risk marker, and an explicit decision question on a PR about rate limiting (added after the hive reviewer's finding; it fails without the guard).go test ./pkg/review/... ./pkg/github/ ./pkg/writeback/passes, as dogo vetand the docs citation and link checks.One thing is left as it was on purpose: after
review.max_redispatch(default 3) expiries with no verdict, the existingreview-dispatch-exhaustedhold still applies. A reviewer that stays rate-limited for more than about two hours ends up with a person, and the hold names the real reason.Refs projectbluefin/testsuite#1116
🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1