Repository navigation
π fix(escalation): a red PR triaged as infrastructure is re-run once instead of sent back to fixers until it escalates - #17002
Conversation
β¦atching fixers On projectbluefin/common#1647 the owning agent answered every fix kick with "infrastructure, not this change" (apt mirror dropped the connection in Setup runner), but only in prose. The fix loop counted six delivered re-dispatches on one head and escalated the PR, and the reviewer lane then pushed an empty "retrigger CI" commit because its DE-ESCALATE verdict said "infra flake: rebase, push". When every failing check run on a hive-authored PR's head carries a hive-ci-transient marker, the eval cycle now re-runs the failed jobs once for that head (rerun-failed-jobs, gated as writeback ci_rerun), and deliverReEngagement holds the reaper's and the merge watcher's fix kicks while that re-run is owed. The re-run is recorded per head in the escalation ledger; a head still red after it returns to the ordinary fix loop. The fix kick now tells the agent to stamp the marker and push nothing, and DE-ESCALATE pushes only when the branch is behind its base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
872eb48
βΉοΈ Sentinel notice β informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @Danathar, head
Hive did not add the |
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (5 waiting): fixes are handed out before new work as soon as you have capacity. |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether the hive-ci-transient marker may trigger an Actions re-run when stamped by any commenter, or must come from a hive-agent/App-authored comment now that it drives a write.
The code does what the body says and the state machine is sound; I have one trust-boundary question for a maintainer, not a defect.
Verified against the v6 tree at the merge-base
- Ordering:
recordRedStalenessβObserveRed(main_eval_cycle.go:678,main_sweeps.go:199) runs beforererunTransientTriagedPRs(main_eval_cycle.go:806) andreapStuckRedPRs(:808), soTransientTriagedSHAis current whenTransientRerunOwedis consulted. - The hold and the re-run cover the same PR set:
hivePRObservationsonly observesisHiveAgentAuthorPRs, andescalationObservationusespr.HasFailingRequiredCheck()forred(main_sweeps.go:111), the same gatererunTransientTriagedPRsapplies (transient_ci_rerun.go:52). So a head cannot be held bydeliverReEngagement(main_sweeps.go:462) without being eligible for the re-run that releases it. AfterRecordTransientRerun(settled on gate refusal,rerun-forbidden, orno-failed-runstoo βpkg/github/transient_ci_rerun.go:42-54),TransientRerunSHA == headand the fix loop resumes; new run ids stale the markers naturally. - Reset points:
SweepclearsTransientRerunSHAon human return and reviewer-lane return (escalation.go:619, :704);ObserveRedclearsTransientTriagedSHAon green and recomputes it per pass (:1110, :1136-1139). - Commit is signed off; the red
native-queue-policyruns are the workflow's ownTypeError: isCosmeticLabelFlip is not a function, unrelated to this diff.
The question
TransientCITriaged is computed from every comment body on the PR with no author check: redPRCommentTexts (pkg/github/shared_ci_incident.go:41-49) feeds all bodies to transientCITriaged (external_ci_deferral.go:43). On v6 that marker only defers a PR. With this PR it also causes an unsolicited Actions write (RerunFailedJobsByID, pkg/github/transient_ci_rerun.go:59) and suppresses fix kicks until that re-run settles. Exposure is bounded β hive-authored PRs only, one re-run round per head, write-back gate consulted first β so I'd call it medium, not blocking. But whether an arbitrary commenter on a public repo should be able to spend Actions minutes and pause the fix loop once per head is a policy choice. If the answer is "yes, bounded is fine", this looks correct to me as-is; if not, filtering redPRCommentTexts (or the marker match) to App/agent-authored comments is a small follow-up.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Check report (hive scanner, read-only β fork branch, nothing pushed) for head 872eb48: the red π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
|
Hive status β updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from Looks correct to me at
One open question, not a defect: if Checks were still Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. Evidence bundle `#17002f0f5113` sha256:f3478b5817cde4c63c20607e126b9a24a158775c978476a3c7f1f7449be6f0d0 β download from the dashboard or `hivectl review evidence #17002`. |
β¦nsient The hive reviewer on hivecommons#17002 pointed out that the transient-CI marker was read from every comment on the PR. Before this PR the marker only deferred a PR. Now it re-runs the failed jobs and holds the fix loop, so a comment from anyone on a public repo could spend Actions minutes and pause repairs once per head. redPRCommentTexts now also returns the bodies the hive's own accounts wrote, and TransientCITriaged is read from those alone. A hive that cannot name its own accounts falls back to every body, as the other hive-authorship checks do. The shared-CI and external-CI markers keep their existing any-author reading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Good catch. I took the narrow option rather than leave it to a policy call, because it only removes trust and adds none. It's in 6e2d823.
@clubanderson this answers the reviewer's question in the stricter direction. If you'd rather allow any commenter, it's a one-line revert. |
There was a problem hiding this comment.
docs-currency β delta re-review at 6e2d823 (previous head 872eb48 approved). Read the delta commit diff, src/pkg/github/shared_ci_incident.go, client.go:2749-2752, label_approval.go:117-155, and the two touched docs.
The ADR and glossary updates match the new hiveTexts path. One operator-facing caveat is missing: both docs state the restriction unconditionally, but redPRCommentTexts (src/pkg/github/shared_ci_incident.go:53-58) falls back to every author's comments when knowsOwnAccounts() is false (no configured bot login / ai_author / App login and /user unreachable, label_approval.go:143-155). An operator running such a PAT hive would read the ADR as a guarantee that outsiders cannot trigger a re-run, which is not true for them.
src/docs/adr/0010-escalation-circuit-breaker.md:60-62:
re-run is owed. Only markers in the hive's own comments count, because a
marker now causes a write: a comment from anyone else cannot spend Actions
minutes or pause the fix loop. (A hive that cannot name any of its own
accounts β no App login, `ai_author` or bot login configured and `/user`
unreachable β falls back to reading every comment, as the other
hive-authorship checks do.) The re-run gives the checks new run ids, so the markers go
Prior finding (correctness, low) at src/cmd/hive/transient_ci_rerun.go:79 β same-cycle fix kick after a transient re-run β is still present; the delta does not touch that file. It was advisory at the previous head and remains so.
β hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88
β¦fix kick The hive reviewer on hivecommons#17002 found a same-cycle gap. rerunTransientTriagedPRs records the re-run, and then reapStuckRedPRs runs on the same snapshot, where the head still reads red and stale and the re-run is no longer owed. When the agent stamped the marker in its own session there is no earlier kick, so no cooldown applies. The cycle that re-ran the jobs could then also kick a fixer and charge a re-engagement. rerunTransientTriagedPRs now returns the PRs whose failed jobs it re-ran, and the reaper skips those for the cycle (reaperSkipAfterRerun). A skipped re-run (no failed runs, re-run forbidden) is not returned, so the fix loop resumes on it as before. ADR-0010 and the glossary now note the fallback: a hive that can't name its own accounts reads every comment for the marker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Both points are fixed in 7404c8e. Same-cycle fix kick (the low finding from the 872eb48 review). It was a real gap; I missed it the first time because it was folded into the status card.
Docs caveat. ADR-0010 and the glossary now say that a hive which can't name any of its own accounts falls back to reading every comment for the marker. The ADR also notes that the cycle which re-runs the jobs sends no fix kick.
|
There was a problem hiding this comment.
Correctness delta re-review (872eb48 β 6e2d823; reviewed head 6e2d8233ad). Only the last commit is PR-authored; the rest of the compare is the v6 merge.
The hive-author gate on the transient marker looks right: redPRCommentTexts returns hiveTexts filtered by isHiveAccount with the documented knowsOwnAccounts() fallback (src/pkg/github/shared_ci_incident.go:52-56), and only TransientCITriaged consumes it while the shared-incident and external markers keep every-author texts (src/pkg/github/client.go:2749-2752). Test covers all three cases.
One open question (not a defect I verified): the marker is stamped by fix-lane agents on their triage comment. If those agents comment from an account that is not the App bot / ai_author / an extra hive bot login, the marker is now silently ignored and the #1647 re-run never fires β the PR falls back to the old kick-until-escalate loop with no log line. Worth confirming agent triage comments go through a hive-recognised login, or adding a Debug log in shared_ci_incident.go when a transient marker is dropped for author.
Prior note at src/cmd/hive/transient_ci_rerun.go:79 (same-cycle fix kick after re-run) is unchanged in this delta.
β hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
On the open question from the 6e2d823 review, about agent triage comments coming from a login the hive doesn't recognise:
So the marker is only dropped for a comment from someone outside the hive, which is the intent. I've left the code as it is to keep this PR moving. A Debug log for a dropped marker would be a cheap follow-up if anyone wants one. The same-cycle note at |
There was a problem hiding this comment.
docs-currency (read at head b46c8d9)
The ADR-0010, glossary, write-surface and sweeps tables all match the code (src/cmd/hive/transient_ci_rerun.go:62-117, src/pkg/github/transient_ci_rerun.go:34-77, src/pkg/github/shared_ci_incident.go:52-63), and the kick/lane text cites the real hive-baseline-check.sh <owner/repo> <check-name> [<pr-number>] --json signature (bin/hive-baseline-check.sh:31). One operator-facing gap:
src/docs/github-app-setup.md:104-113still names only the shared-CI rerun sweep as the consumer ofActions: Read and write. The transient re-run uses the samererun-failed-jobscall and the same forbidden backoff (src/pkg/github/transient_ci_rerun.go:61-66), but on a 403 it only logs a warning and returnsSkipped: "rerun-forbidden"; nounauthorized:*note is posted and the fix loop just resumes on that head (src/cmd/hive/transient_ci_rerun.go:96-102). An operator reading this section would conclude the permission is optional for them when the new behaviour silently degrades without it. Suggested addition after line 113:
The transient-CI re-run (a red hive PR whose failing runs all carry
`hive-ci-transient`, see ADR-0010) uses the same `rerun-failed-jobs` call and
the same per-repository refusal backoff. Without Actions write it logs
`transient CI rerun forbidden`, posts no note, and the head returns to the
ordinary fix loop; it is disabled together with the fix loop by
`escalation.disabled`.
β hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88
The hive reviewer on hivecommons#17002 noted that the setup page named only the shared-CI rerun sweep as a consumer of Actions: Read and write. The transient re-run uses the same rerun-failed-jobs call and backoff. Without the permission it logs and returns the head to the fix loop instead of posting a note, so an operator would not otherwise learn it is degraded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Docs gap fixed in f0f5113. |
There was a problem hiding this comment.
Read the diff at a7a8457 against the v6 tree (merge-base 2d80e41) and traced the callers. Looks correct to me; nothing I could turn into a defect.
What I verified
- Hold and re-run agree on scope.
TransientTriagedSHAis only ever written byObserveRed, whichrecordRedStalenessfeeds fromhivePRObservations(src/cmd/hive/main_sweeps.go:185-186, hive-authored only).reapStuckRedPRsiterates the same set (main_sweeps.go:518). So theisHiveAgentAuthorfilter inrerunTransientTriagedPRs(src/cmd/hive/transient_ci_rerun.go:62) cannot leave a contributor PR held with no re-run ever recorded βTransientRerunOwedis false for anything the ledger never saw. - No indefinite hold when the re-run step can't run. The only path where
deliverReEngagementholds (main_sweeps.go:449) butrerunTransientTriagedPRsis skipped isghClient == nil, andrunEvalCyclereturns before either atmain_eval_cycle.go:558. A skipped outcome (no-failed-runs, gate refusal,rerun-forbidden) still callsRecordTransientRerun(transient_ci_rerun.go:85), so the fix loop resumes next cycle as the comment says. - Marker trust.
transientCITriagednow readshiveTextsonly (src/pkg/github/client.go:2754), built fromisHiveAccount(shared_ci_incident.go:58-60), with the existingknowsOwnAccountsfallback. Markers are still pinned to head + run id (external_ci_deferral.go:36-43), so the re-run's new run ids expire them without any ledger write. - Same-cycle double dispatch is closed by
reaperSkipAfterRerun(transient_ci_rerun.go:108) and tested. - The merge-watcher hook passes
headSHA == ""(main_sweeps.go:430);TransientRerunOwedresolves that toCurRedSHA(src/pkg/escalation/transient_rerun.go:33), so the hold covers that path too.
One open question, not a finding: RerunTransientCIFailures marks the attempt settled before the loop (src/pkg/github/transient_ci_rerun.go:62), so if GitHub returns a non-403 error on every run (Skipped: "rerun-failed"), the head's single re-run is spent and the ordinary fix loop resumes with no re-run having happened. That is the documented contract ("once any re-run request reached GitHub the attempt is settled") and arguably the safe choice against spending Actions minutes in a loop, so I am only noting it for the maintainer, not asking for a change.
CI on a7a8457 was still running when I read it; the previous head was green.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Test-review of f0f5113. The ledger side is thorough (
Neither blocks the approve; both are a few lines in π Hive Agent: β hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
Summary
When an agent says a red check is infrastructure, not the change, the hive now re-runs the failed jobs once (
rerun-failed-jobs, through the write-back gate). It no longer sends the agent fix kick after fix kick until the PR escalates. The agent's verdict reaches the hive as the existinghive-ci-transientmarker. Until now only the dependency-bot hand-off read that marker. This PR also changes the fix-kick and reviewer-lane wording, so agents stamp the marker and never push an empty "retrigger CI" commit.What went wrong
On projectbluefin/common#1647,
Build and push image (x86_64)failed in the runner'sSetup runnerstep: apt got "Remote end closed connection" from azure.archive.ubuntu.com, exit 100.sec-check) answered the fix loop's kicks with "infrastructure, not this change" at 18:10, 20:12, 20:32 and 21:13Z on 11-06 (and twice before that). Each answer was prose only, with nohive-ci-transientmarker. The kick text told it to do exactly that: "if the failure is infrastructure β¦ leave one comment and end the turn."adjudicator(reviewer lane) pushed an empty commit, e8285f93b ("chore: retrigger CI after apt mirror outage", 0 files). This came from an agent prompt, not hive code: the lane's DE-ESCALATE verdict says "infra flake: rebase the branch on its base, push", and on a current branch the only push left is an empty one.On v6 tip nothing in the fix loop reads
TransientCITriaged.reapStuckRedPRsskips only fleet-wide shared failures, and the merge watcher's hook skips nothing. So a correctly stamped marker would not have stopped the kicks either.What changes
pkg/escalation/transient_rerun.go). That makes the re-run once per head, and it survives a restart. A person or the reviewer lane returning an escalated PR to the lane allows one more re-run.deliverReEngagementsends no fix kick and charges no budget while that re-run is owed. This covers both the reaper and the merge watcher's hook. After the re-run, the checks get new run ids, so the markers go stale on their own. A head that is still red then goes back to the ordinary fix loop.TransientCITriagedis read from comments by the hive's own accounts (isHiveAccount). A hive that can't name its accounts falls back to every comment, as other hive-authorship checks do. So nobody else can comment the marker onto a PR to spend Actions minutes or pause the fix loop. The shared-CI and external-CI markers still read every comment. This was added after the hive reviewer's question;TestEnrichCIStatus_TransientCITriagedOnlyFromHiveCommentsfails without it.reaperSkipAfterRerun), so one cycle never sends both a re-run and a fix kick for the same head. This was added after the hive reviewer's finding, andTestRerunTransientTriagedPRs_SameCycleDrawsNoFixKickcovers it.Client.RerunTransientCIFailuresreuses the shared-CI sweep'slatestFailedWorkflowRunsand its forbidden-rerun backoff. It asksWritebackAllowed(β¦, KindCIRerun, β¦)first. A hive-authored PR passes unless its repo is paused, off or issues-only.transient_markerfromhive-baseline-check.sh <repo> '<check>' <n> --json.gh run rerun --failed, or stamp the marker. The template copies and the compiled-in fallback are updated together.Files
src/pkg/escalation/escalation.go,transient_rerun.go: ledger fields and the owed/spent predicatessrc/pkg/github/transient_ci_rerun.go: the gated re-runsrc/cmd/hive/transient_ci_rerun.go,main_sweeps.go,main_eval_cycle.go: the eval-cycle step, the kick hold, the kick textsrc/pkg/policies/defaults/reviewer-lane.md,src/policies/reviewer-lane.md,src/pkg/scheduler/reviewer_lane.go: DE-ESCALATE wordinghive-ci-transientglossary entry, the GitHub write-surface tables, and a changelog fragmentTests
TestTransientTriagedHeadDrawsNoFixKick: on v6 tip it fails with "transient-triaged head got 6 fix kicks, want 0", which is the fix: gate all dashboard paths behind auth tokenΒ #1647 count. With this change it passes.TestRedPRFixKickInfraVerdictUsesTransientMarker: fails on v6 tip and passes with this change.TestRerunTransientTriagedPRs_OncePerHeadThenFixLoopResumesTestRerunTransientTriagedPRs_SkipsTestRerunTransientCIFailures(re-run, gate refusal with no write, 403, nothing to re-run)TestTransientRerunOwedOncePerHeadTestTransientRerunReArmedWhenUnparked./cmd/hive,./pkg/escalation,./pkg/github,./pkg/scheduler,./pkg/policies,./pkg/writeback.go vetis clean, andcheck-docs-citations.pyandcheck-docs-links.pypass.Not in this PR
ExternalCIDeferred(hive-ci-external) PR is still kicked by the reaper, even though kick builders defer it. The reaper reads only fleet-wideShared. Same class of bug, separate change.crux/pushbroker/pushbroker.go), and it is in the live build. The empty commit on fix: gate all dashboard paths behind auth tokenΒ #1647 still landed, so that push took some other path. That needs its own look.π€ Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1