Skip to content

πŸ› fix(escalation): a red PR triaged as infrastructure is re-run once instead of sent back to fixers until it escalates - #17002

Merged
hivecommons-hive[bot] merged 7 commits into
hivecommons:v6from
Danathar:fix/infra-failure-rerun
Nov 7, 2026
Merged

hivecommons-hive[bot] merged 7 commits into
hivecommons:v6from
Danathar:fix/infra-failure-rerun

Conversation

@Danathar

@Danathar Danathar commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When an agent says a red check is infrastructure, not the change, the hive now re-runs the failed jobs once (rerun-failed-jobs, through the write-back gate). It no longer sends the agent fix kick after fix kick until the PR escalates. The agent's verdict reaches the hive as the existing hive-ci-transient marker. Until now only the dependency-bot hand-off read that marker. This PR also changes the fix-kick and reviewer-lane wording, so agents stamp the marker and never push an empty "retrigger CI" commit.

What went wrong

On projectbluefin/common#1647, Build and push image (x86_64) failed in the runner's Setup runner step: apt got "Remote end closed connection" from azure.archive.ubuntu.com, exit 100.

  • The owning agent (sec-check) answered the fix loop's kicks with "infrastructure, not this change" at 18:10, 20:12, 20:32 and 21:13Z on 11-06 (and twice before that). Each answer was prose only, with no hive-ci-transient marker. The kick text told it to do exactly that: "if the failure is infrastructure … leave one comment and end the turn."
  • The fix loop cannot read prose. Each of those kicks was a delivered re-dispatch, and the sixth escalated the PR at 21:16Z ("6 delivered fix re-dispatches … 1 distinct red head").
  • At 21:55Z the adjudicator (reviewer lane) pushed an empty commit, e8285f93b ("chore: retrigger CI after apt mirror outage", 0 files). This came from an agent prompt, not hive code: the lane's DE-ESCALATE verdict says "infra flake: rebase the branch on its base, push", and on a current branch the only push left is an empty one.

On v6 tip nothing in the fix loop reads TransientCITriaged. reapStuckRedPRs skips only fleet-wide shared failures, and the merge watcher's hook skips nothing. So a correctly stamped marker would not have stopped the kicks either.

What changes

  • Ledger: the escalation ledger records the head a transient triage covers and the head the hive already re-ran (pkg/escalation/transient_rerun.go). That makes the re-run once per head, and it survives a restart. A person or the reviewer lane returning an escalated PR to the lane allows one more re-run.
  • Re-run step: a new step in the eval cycle runs just before the stuck-PR reaper. For a hive-authored red PR whose every failing run carries the marker, it re-runs the failed jobs once. It never comments and never pushes.
  • Fix kicks held: deliverReEngagement sends no fix kick and charges no budget while that re-run is owed. This covers both the reaper and the merge watcher's hook. After the re-run, the checks get new run ids, so the markers go stale on their own. A head that is still red then goes back to the ordinary fix loop.
  • Only the hive's own markers count: TransientCITriaged is read from comments by the hive's own accounts (isHiveAccount). A hive that can't name its accounts falls back to every comment, as other hive-authorship checks do. So nobody else can comment the marker onto a PR to spend Actions minutes or pause the fix loop. The shared-CI and external-CI markers still read every comment. This was added after the hive reviewer's question; TestEnrichCIStatus_TransientCITriagedOnlyFromHiveComments fails without it.
  • No kick in the re-run cycle: the reaper skips the PRs the re-run step just re-ran (reaperSkipAfterRerun), so one cycle never sends both a re-run and a fix kick for the same head. This was added after the hive reviewer's finding, and TestRerunTransientTriagedPRs_SameCycleDrawsNoFixKick covers it.
  • GitHub call: Client.RerunTransientCIFailures reuses the shared-CI sweep's latestFailedWorkflowRuns and its forbidden-rerun backoff. It asks WritebackAllowed(…, KindCIRerun, …) first. A hive-authored PR passes unless its repo is paused, off or issues-only.
  • Fix kick text: on an infrastructure failure, push nothing. End the one comment with the transient_marker from hive-baseline-check.sh <repo> '<check>' <n> --json.
  • Reviewer lane DE-ESCALATE: rebase and push only when the branch is behind its base. If the branch is current, push nothing; re-run with gh run rerun --failed, or stamp the marker. The template copies and the compiled-in fallback are updated together.

Files

  • src/pkg/escalation/escalation.go, transient_rerun.go: ledger fields and the owed/spent predicates
  • src/pkg/github/transient_ci_rerun.go: the gated re-run
  • src/cmd/hive/transient_ci_rerun.go, main_sweeps.go, main_eval_cycle.go: the eval-cycle step, the kick hold, the kick text
  • src/pkg/policies/defaults/reviewer-lane.md, src/policies/reviewer-lane.md, src/pkg/scheduler/reviewer_lane.go: DE-ESCALATE wording
  • Docs: ADR-0010, the hive-ci-transient glossary entry, the GitHub write-surface tables, and a changelog fragment

Tests

  • TestTransientTriagedHeadDrawsNoFixKick: on v6 tip it fails with "transient-triaged head got 6 fix kicks, want 0", which is the fix: gate all dashboard paths behind auth tokenΒ #1647 count. With this change it passes.
  • TestRedPRFixKickInfraVerdictUsesTransientMarker: fails on v6 tip and passes with this change.
  • New behaviour, so it has no tip run:
    • TestRerunTransientTriagedPRs_OncePerHeadThenFixLoopResumes
    • TestRerunTransientTriagedPRs_Skips
    • TestRerunTransientCIFailures (re-run, gate refusal with no write, 403, nothing to re-run)
    • TestTransientRerunOwedOncePerHead
    • TestTransientRerunReArmedWhenUnparked
  • Full packages pass: ./cmd/hive, ./pkg/escalation, ./pkg/github, ./pkg/scheduler, ./pkg/policies, ./pkg/writeback. go vet is clean, and check-docs-citations.py and check-docs-links.py pass.

Not in this PR

  • An ExternalCIDeferred (hive-ci-external) PR is still kicked by the reaper, even though kick builders defer it. The reaper reads only fleet-wide Shared. Same class of bug, separate change.
  • The pushbroker has an empty-commit guard (crux/pushbroker/pushbroker.go), and it is in the live build. The empty commit on fix: gate all dashboard paths behind auth tokenΒ #1647 still landed, so that push took some other path. That needs its own look.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…atching fixers

On projectbluefin/common#1647 the owning agent answered every fix kick
with "infrastructure, not this change" (apt mirror dropped the
connection in Setup runner), but only in prose. The fix loop counted six
delivered re-dispatches on one head and escalated the PR, and the
reviewer lane then pushed an empty "retrigger CI" commit because its
DE-ESCALATE verdict said "infra flake: rebase, push".

When every failing check run on a hive-authored PR's head carries a
hive-ci-transient marker, the eval cycle now re-runs the failed jobs
once for that head (rerun-failed-jobs, gated as writeback ci_rerun),
and deliverReEngagement holds the reaper's and the merge watcher's fix
kicks while that re-run is owed. The re-run is recorded per head in the
escalation ledger; a head still red after it returns to the ordinary
fix loop. The fix kick now tells the agent to stamp the marker and push
nothing, and DE-ESCALATE pushes only when the branch is behind its base.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 7, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 7, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
872eb48

ℹ️ Sentinel notice β€” informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @Danathar, head 872eb4886226) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation β€” a maintainer should confirm the change is intended before it merges.

  • sensitive_path β€” changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • src/pkg/policies/defaults/reviewer-lane.md
    • src/policies/reviewer-lane.md

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • check-failure

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (5 waiting): fixes are handed out before new work as soon as you have capacity.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether the hive-ci-transient marker may trigger an Actions re-run when stamped by any commenter, or must come from a hive-agent/App-authored comment now that it drives a write.

The code does what the body says and the state machine is sound; I have one trust-boundary question for a maintainer, not a defect.

Verified against the v6 tree at the merge-base

  • Ordering: recordRedStaleness β†’ ObserveRed (main_eval_cycle.go:678, main_sweeps.go:199) runs before rerunTransientTriagedPRs (main_eval_cycle.go:806) and reapStuckRedPRs (:808), so TransientTriagedSHA is current when TransientRerunOwed is consulted.
  • The hold and the re-run cover the same PR set: hivePRObservations only observes isHiveAgentAuthor PRs, and escalationObservation uses pr.HasFailingRequiredCheck() for red (main_sweeps.go:111), the same gate rerunTransientTriagedPRs applies (transient_ci_rerun.go:52). So a head cannot be held by deliverReEngagement (main_sweeps.go:462) without being eligible for the re-run that releases it. After RecordTransientRerun (settled on gate refusal, rerun-forbidden, or no-failed-runs too β€” pkg/github/transient_ci_rerun.go:42-54), TransientRerunSHA == head and the fix loop resumes; new run ids stale the markers naturally.
  • Reset points: Sweep clears TransientRerunSHA on human return and reviewer-lane return (escalation.go:619, :704); ObserveRed clears TransientTriagedSHA on green and recomputes it per pass (:1110, :1136-1139).
  • Commit is signed off; the red native-queue-policy runs are the workflow's own TypeError: isCosmeticLabelFlip is not a function, unrelated to this diff.

The question

TransientCITriaged is computed from every comment body on the PR with no author check: redPRCommentTexts (pkg/github/shared_ci_incident.go:41-49) feeds all bodies to transientCITriaged (external_ci_deferral.go:43). On v6 that marker only defers a PR. With this PR it also causes an unsolicited Actions write (RerunFailedJobsByID, pkg/github/transient_ci_rerun.go:59) and suppresses fix kicks until that re-run settles. Exposure is bounded β€” hive-authored PRs only, one re-run round per head, write-back gate consulted first β€” so I'd call it medium, not blocking. But whether an arbitrary commenter on a public repo should be able to spend Actions minutes and pause the fix loop once per head is a policy choice. If the answer is "yes, bounded is fine", this looks correct to me as-is; if not, filtering redPRCommentTexts (or the marker match) to App/agent-authored comments is a small follow-up.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Check report (hive scanner, read-only β€” fork branch, nothing pushed) for head 872eb48: the red native-queue-policy (TypeError: isCosmeticLabelFlip is not a function) is not caused by this diff. The workflow checks out the base branch's .github/scripts/native-queue-policy.cjs, and v6 lacked the isCosmeticLabelFlip/replayLastVerdict exports that the v5-authored replay step requires β€” shared incident #17003, now closed: #16997 merged into v6 at 15:40Z and v6's .cjs exports both symbols. Both red runs on this head (15:23Z) predate that merge. No change is needed here; the check re-runs on the next label/edit/push event on this PR.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer (folded here because this head already carries its hive review):

Looks correct to me at f0f5113. Read the new code and the callers, not just the diff:

  • Hold is real and budget-free. src/cmd/hive/main_sweeps.go:449-458 returns from deliverReEngagement before resolveRedPRFixer, so no kick and no re-engagement charge while TransientRerunOwed holds. Both the reaper and the merge-watcher hook go through this function, so the body's "covers both" claim checks out.
  • Once per head, survives restart, re-armed on unpark. src/pkg/escalation/transient_rerun.go:25-36 compares TransientTriagedSHA == head && TransientRerunSHA != head; escalation.go clears TransientRerunSHA on both the label-unpark and reviewer-return paths in Sweep, and clears TransientTriagedSHA when the PR goes green in ObserveRed. TestTransientRerunOwedOncePerHead / TestTransientRerunReArmedWhenUnparked exercise exactly those transitions.
  • No double-dispatch in one cycle. main_eval_cycle.go:806-816 feeds reaperSkipAfterRerun(escalatedPRs, reran) into reapStuckRedPRs; reran only contains PRs where len(outcome.RunIDs) > 0, so a skipped/refused re-run correctly falls back to the fix loop in the same pass.
  • Marker trust. src/pkg/github/shared_ci_incident.go:48-62 filters hiveTexts by isHiveAccount and client.go:2754 feeds only that to transientCITriaged; the external/shared markers keep the full list. Test covers hive-bot vs. outside author.
  • Settled vs. retry. RerunTransientCIFailures returns unsettled only on the latestFailedWorkflowRuns error; gate refusal, 403 and no-runs all settle and are recorded, so the hold releases.

One open question, not a defect: if latestFailedWorkflowRuns fails persistently (e.g. Actions API scope missing on an install), the head stays "owed" indefinitely and the fix loop stays held for it β€” transient_ci_rerun.go:84-89 just logs and retries every pass. A cap on consecutive list errors (then settle as "rerun-failed") would bound that; fine as a follow-up if you'd rather keep this diff tight.

Checks were still IN_PROGRESS on this head when I read it. I have not approved anything.

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Evidence bundle `#17002f0f5113` sha256:f3478b5817cde4c63c20607e126b9a24a158775c978476a3c7f1f7449be6f0d0 β€” download from the dashboard or `hivectl review evidence #17002`.

…nsient

The hive reviewer on hivecommons#17002 pointed out that the transient-CI marker was
read from every comment on the PR. Before this PR the marker only deferred
a PR. Now it re-runs the failed jobs and holds the fix loop, so a comment
from anyone on a public repo could spend Actions minutes and pause repairs
once per head.

redPRCommentTexts now also returns the bodies the hive's own accounts
wrote, and TransientCITriaged is read from those alone. A hive that cannot
name its own accounts falls back to every body, as the other
hive-authorship checks do. The shared-CI and external-CI markers keep
their existing any-author reading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Good catch. I took the narrow option rather than leave it to a policy call, because it only removes trust and adds none. It's in 6e2d823.

  • redPRCommentTexts now also returns the bodies written by the hive's own accounts (isHiveAccount: the App bot, ai_author or an additional hive login). TransientCITriaged is read from those alone.
  • A hive that can't name its own accounts (knowsOwnAccounts false) falls back to every comment, as the other hive-authorship checks do.
  • The shared-CI incident and external-CI markers keep reading every comment. They only defer a PR; they trigger no write.
  • Agents post their triage through the relay as the App, so their markers still count.
  • New test TestEnrichCIStatus_TransientCITriagedOnlyFromHiveComments: an App-authored marker triages the head, the same marker from mallory does not, and an external marker from mallory still defers. It fails without the filter.
  • ADR-0010 and the glossary entry say only hive-authored markers count.

./pkg/github/, ./cmd/hive/ and ./pkg/writeback/ pass, and go vet and the docs checks are clean.

@clubanderson this answers the reviewer's question in the stricter direction. If you'd rather allow any commenter, it's a one-line revert.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency β€” delta re-review at 6e2d823 (previous head 872eb48 approved). Read the delta commit diff, src/pkg/github/shared_ci_incident.go, client.go:2749-2752, label_approval.go:117-155, and the two touched docs.

The ADR and glossary updates match the new hiveTexts path. One operator-facing caveat is missing: both docs state the restriction unconditionally, but redPRCommentTexts (src/pkg/github/shared_ci_incident.go:53-58) falls back to every author's comments when knowsOwnAccounts() is false (no configured bot login / ai_author / App login and /user unreachable, label_approval.go:143-155). An operator running such a PAT hive would read the ADR as a guarantee that outsiders cannot trigger a re-run, which is not true for them.

src/docs/adr/0010-escalation-circuit-breaker.md:60-62:

re-run is owed. Only markers in the hive's own comments count, because a
marker now causes a write: a comment from anyone else cannot spend Actions
minutes or pause the fix loop. (A hive that cannot name any of its own
accounts β€” no App login, `ai_author` or bot login configured and `/user`
unreachable β€” falls back to reading every comment, as the other
hive-authorship checks do.) The re-run gives the checks new run ids, so the markers go

Prior finding (correctness, low) at src/cmd/hive/transient_ci_rerun.go:79 β€” same-cycle fix kick after a transient re-run β€” is still present; the delta does not touch that file. It was advisory at the previous head and remains so.

β€” hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88

…fix kick

The hive reviewer on hivecommons#17002 found a same-cycle gap. rerunTransientTriagedPRs
records the re-run, and then reapStuckRedPRs runs on the same snapshot,
where the head still reads red and stale and the re-run is no longer owed.
When the agent stamped the marker in its own session there is no earlier
kick, so no cooldown applies. The cycle that re-ran the jobs could then also
kick a fixer and charge a re-engagement.

rerunTransientTriagedPRs now returns the PRs whose failed jobs it re-ran,
and the reaper skips those for the cycle (reaperSkipAfterRerun). A skipped
re-run (no failed runs, re-run forbidden) is not returned, so the fix loop
resumes on it as before.

ADR-0010 and the glossary now note the fallback: a hive that can't name its
own accounts reads every comment for the marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Both points are fixed in 7404c8e.

Same-cycle fix kick (the low finding from the 872eb48 review). It was a real gap; I missed it the first time because it was folded into the status card.

  • rerunTransientTriagedPRs now returns the PRs whose failed jobs it re-ran, and runEvalCycle passes reaperSkipAfterRerun(escalatedPRs, reran) to the reaper. A head re-run this cycle draws no fix kick and charges no re-engagement.
  • A skipped re-run (no-failed-runs, rerun-forbidden) isn't returned, so the fix loop resumes on it as intended. That's the case you warned about holding on TransientTriaged alone.
  • escalatedPRs itself is not changed, so the work-list writers see the same set as before.
  • New test TestRerunTransientTriagedPRs_SameCycleDrawsNoFixKick builds your exact shape: the marker predates any kick, and the head is stale and red. It checks:
    • reaping with the escalated set alone sends 1 kick, which is the gap;
    • reaping with the skip set sends 0;
    • a skipped re-run is not reported.

Docs caveat. ADR-0010 and the glossary now say that a hive which can't name any of its own accounts falls back to reading every comment for the marker. The ADR also notes that the cycle which re-runs the jobs sends no fix kick.

./cmd/hive/ and ./pkg/escalation/ pass, and go vet and the docs checks are clean.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness delta re-review (872eb48 β†’ 6e2d823; reviewed head 6e2d8233ad). Only the last commit is PR-authored; the rest of the compare is the v6 merge.

The hive-author gate on the transient marker looks right: redPRCommentTexts returns hiveTexts filtered by isHiveAccount with the documented knowsOwnAccounts() fallback (src/pkg/github/shared_ci_incident.go:52-56), and only TransientCITriaged consumes it while the shared-incident and external markers keep every-author texts (src/pkg/github/client.go:2749-2752). Test covers all three cases.

One open question (not a defect I verified): the marker is stamped by fix-lane agents on their triage comment. If those agents comment from an account that is not the App bot / ai_author / an extra hive bot login, the marker is now silently ignored and the #1647 re-run never fires β€” the PR falls back to the old kick-until-escalate loop with no log line. Worth confirming agent triage comments go through a hive-recognised login, or adding a Debug log in shared_ci_incident.go when a transient marker is dropped for author.

Prior note at src/cmd/hive/transient_ci_rerun.go:79 (same-cycle fix kick after re-run) is unchanged in this delta.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@Danathar

Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

On the open question from the 6e2d823 review, about agent triage comments coming from a login the hive doesn't recognise:

So the marker is only dropped for a comment from someone outside the hive, which is the intent. I've left the code as it is to keep this PR moving. A Debug log for a dropped marker would be a cheap follow-up if anyone wants one.

The same-cycle note at transient_ci_rerun.go:79 is fixed in 7404c8e (reaperSkipAfterRerun, TestRerunTransientTriagedPRs_SameCycleDrawsNoFixKick); this review covers the head before it.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency (read at head b46c8d9)

The ADR-0010, glossary, write-surface and sweeps tables all match the code (src/cmd/hive/transient_ci_rerun.go:62-117, src/pkg/github/transient_ci_rerun.go:34-77, src/pkg/github/shared_ci_incident.go:52-63), and the kick/lane text cites the real hive-baseline-check.sh <owner/repo> <check-name> [<pr-number>] --json signature (bin/hive-baseline-check.sh:31). One operator-facing gap:

  • src/docs/github-app-setup.md:104-113 still names only the shared-CI rerun sweep as the consumer of Actions: Read and write. The transient re-run uses the same rerun-failed-jobs call and the same forbidden backoff (src/pkg/github/transient_ci_rerun.go:61-66), but on a 403 it only logs a warning and returns Skipped: "rerun-forbidden"; no unauthorized:* note is posted and the fix loop just resumes on that head (src/cmd/hive/transient_ci_rerun.go:96-102). An operator reading this section would conclude the permission is optional for them when the new behaviour silently degrades without it. Suggested addition after line 113:
The transient-CI re-run (a red hive PR whose failing runs all carry
`hive-ci-transient`, see ADR-0010) uses the same `rerun-failed-jobs` call and
the same per-repository refusal backoff. Without Actions write it logs
`transient CI rerun forbidden`, posts no note, and the head returns to the
ordinary fix loop; it is disabled together with the fix loop by
`escalation.disabled`.

β€” hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88

hivecommons-hive Bot and others added 2 commits November 7, 2026 16:58
The hive reviewer on hivecommons#17002 noted that the setup page named only the
shared-CI rerun sweep as a consumer of Actions: Read and write. The
transient re-run uses the same rerun-failed-jobs call and backoff. Without
the permission it logs and returns the head to the fix loop instead of
posting a note, so an operator would not otherwise learn it is degraded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Docs gap fixed in f0f5113. github-app-setup.md now says the transient-CI re-run uses the same rerun-failed-jobs call and the same backoff. Without Actions write it logs transient CI rerun forbidden (the log text in pkg/github/transient_ci_rerun.go:62), posts no note, and returns the head to the fix loop. It is turned off with escalation.disabled (cfg.Escalation.Disabled at the top of rerunTransientTriagedPRs). The docs citation and link checks pass.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff at a7a8457 against the v6 tree (merge-base 2d80e41) and traced the callers. Looks correct to me; nothing I could turn into a defect.

What I verified

  • Hold and re-run agree on scope. TransientTriagedSHA is only ever written by ObserveRed, which recordRedStaleness feeds from hivePRObservations (src/cmd/hive/main_sweeps.go:185-186, hive-authored only). reapStuckRedPRs iterates the same set (main_sweeps.go:518). So the isHiveAgentAuthor filter in rerunTransientTriagedPRs (src/cmd/hive/transient_ci_rerun.go:62) cannot leave a contributor PR held with no re-run ever recorded β€” TransientRerunOwed is false for anything the ledger never saw.
  • No indefinite hold when the re-run step can't run. The only path where deliverReEngagement holds (main_sweeps.go:449) but rerunTransientTriagedPRs is skipped is ghClient == nil, and runEvalCycle returns before either at main_eval_cycle.go:558. A skipped outcome (no-failed-runs, gate refusal, rerun-forbidden) still calls RecordTransientRerun (transient_ci_rerun.go:85), so the fix loop resumes next cycle as the comment says.
  • Marker trust. transientCITriaged now reads hiveTexts only (src/pkg/github/client.go:2754), built from isHiveAccount (shared_ci_incident.go:58-60), with the existing knowsOwnAccounts fallback. Markers are still pinned to head + run id (external_ci_deferral.go:36-43), so the re-run's new run ids expire them without any ledger write.
  • Same-cycle double dispatch is closed by reaperSkipAfterRerun (transient_ci_rerun.go:108) and tested.
  • The merge-watcher hook passes headSHA == "" (main_sweeps.go:430); TransientRerunOwed resolves that to CurRedSHA (src/pkg/escalation/transient_rerun.go:33), so the hold covers that path too.

One open question, not a finding: RerunTransientCIFailures marks the attempt settled before the loop (src/pkg/github/transient_ci_rerun.go:62), so if GitHub returns a non-403 error on every run (Skipped: "rerun-failed"), the head's single re-run is spent and the ordinary fix loop resumes with no re-run having happened. That is the documented contract ("once any re-run request reached GitHub the attempt is settled") and arguably the safe choice against spending Actions minutes in a loop, so I am only noting it for the maintainer, not asking for a change.

CI on a7a8457 was still running when I read it; the previous head was green.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit f53d91f into hivecommons:v6 Nov 7, 2026
68 of 69 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at f0f5113.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test-review of f0f5113. The ledger side is thorough (TestTransientRerunOwedOncePerHead covers by-sha / by-current-head / stale-triage / restart persistence; ReArmedWhenUnparked pins the Sweep reset). Two arms of RerunTransientCIFailures in pkg/github/transient_ci_rerun.go have no assertion:

  1. rerun-failed β€” a non-403 error on every run goes through continue, and the trailing else if out.Skipped == "" is the only thing that turns that into Skipped: "rerun-failed". Nothing drives it: sharedCIRerunFake can only answer rerun-failed-jobs with 201 or 403 (forbidRerun / forbidAfter), so a 5xx is not reproducible with the current fake. If that else if were dropped or reordered, the call would return {Settled:true, RunIDs:nil, Skipped:""} β€” indistinguishable from "nothing to do" to the sweep β€” and no test would notice. A failRerun bool on the fake that answers 500 and one subtest asserting Skipped == "rerun-failed" closes it.

  2. cached refusal β€” if c.sharedCIRerunRefused(repo) { return …"rerun-forbidden" } short-circuits before latestFailedWorkflowRuns. The "forbidden re-run settles the attempt" subtest only checks the first call's outcome; it never asserts that the refusal was recorded (markSharedCIRerunForbidden) or that a second call makes no workflow_runs request. A second RerunTransientCIFailures on the same client in that subtest, asserting the fake's run-listing count did not grow, covers both the mark and the short-circuit.

Neither blocks the approve; both are a few lines in transient_ci_rerun_test.go.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant