Skip to content

[scanner] stage reconciler oscillates 050↔070↔085 on a needs-decision hive PR, re-labelling every 1–5 min and firing ~140 workflow runs/hour (#16607: 2500 runs, 300+ label events) #16974

Closed

Description

@hivecommons-hive

What happens

The sweep's stage reconciler oscillates on #16607 (head 687f4cb, no new commits since 2026-11-05) between stage/050-review-pending, stage/070-fixing and stage/085-needs-decision, re-labelling the PR every 1–5 minutes, around the clock, with no change in any input (same head, same review verdict, same approvals).

Evidence (read from the GitHub API on 2026-11-07 ~07:30Z):

  • issues/16607/events: 153 labeled + 146 unlabeled events (the events API caps at 300 — the real count is higher); every one is hivecommons-hive[bot], every pair is an unlabeled of the current stage followed 3–4 s later by a labeled of a different stage. Sample from the capped window: 00:27:49 labeled 070-fixing, 00:29:31 unlabeled 070-fixing, 00:29:35 labeled 050-review-pending, 00:30:52 unlabeled 050-review-pending, 00:30:55 labeled 070-fixing, … 00:42:32 labeled 085-needs-decision; same shape at 18:30–18:37Z the day before (085 → 070 → 085 → 070). At 07:24Z today the PR wore 050-review-pending again, updated_at=07:24:01Z.
  • actions/runs?branch=scanner/fix-16265: 2500 workflow runs (API cap) on this one branch. Today alone: 02:00Z 84 runs, 03:00Z 120, 04:00Z 138, 05:00Z 84, 06:00Z 150, 07:00Z 54 (partial hour) — i.e. ~46 label-flip pairs per hour, each firing changelog-fragment-guard, DCO squash attribution (pull_request: labeled/unlabeled) and native-queue-policy (pull_request_target: labeled/unlabeled). gh pr view 16607 shows 692 cancelled Sign-off survives the squash and 491 cancelled native-queue-policy check-runs on the head.
  • No other open hive PR is churning: scanner/canary-egress-unreadable-body (🐛 fix(crux/proxy): canary scan refuses write bodies it could not read #16895) has 42 runs total, sync/v5-to-v6 and cli-pin-bump/codex had 0 runs today. The loop is specific to a hive-authored PR carrying a needs-decision review verdict (the hive-verdict: needs-decision marker, re-posted at 687f4cb as "HUMAN DECISION NEEDED").

Why (as far as the source shows)

#16942 fixed the workflow side (native-queue-policy no longer cancels in progress) but left the producer of the flips alone. src/pkg/github/automerge/automerge_stage_labels.go computes verdict.Stage fresh on every sweep from mutable signals — the review verdict (→ 085), the lane's "red / changes requested on a hive PR" path (→ 070 via fixingStage(obs)), and pass.Route.Checks.Pending > 0 (→ 050, see the act.WaitingOnHive block near line 1200) — and then writes the result through stagelabels.Enforce with no hysteresis. The label write itself fires the labeled/unlabeled workflow triggers, which puts checks back into pending, which the next sweep reads as 050, which completes, which the next sweep reads as 070/085, which fires the workflows again. The reconciler is feeding its own input: the only thing changing between sweeps is the check state the previous label write caused.

Why it matters

  • 3,000+ wasted Actions runs on a single PR in two days; the PR is parked on a human decision and cannot change state on its own.
  • The 085 label — the signal to @hanthor that a decision is wanted — is removed for minutes at a time, so a maintainer filtering on stage/085-needs-decision sees the PR appear and disappear.
  • Every flip rewrites the stage comment / dashboard row, so the "waiting since" clock and the decision question never settle.

Fix direction (decided by hive; additive, backward compatible)

  1. In the sweep's stage verdict, do not treat check runs that the hive's own label write caused as new pending evidence: once a PR has been placed on 085 for a head with a recorded decision question, keep 085 for that head until a human reply, a new push, or an explicit /hive command. Checks re-running on the same head are not new information.
  2. Add a per-(repo, number, head) transition dampener in stagelabels.Store: refuse to move between two stages more than 3 times within 30 minutes without a new head SHA or a human event, log the suppressed transition, and keep the current label.
  3. A regression test in pkg/github/automerge that feeds the same observation twice with only Checks.Pending toggled by the previous iteration's own label write and asserts the stage does not change.

Related: #16942 (workflow-side mitigation on the same PR), #16607 (the PR being churned).

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    agent/scannerCreated or modified by the scanner agent
    ciApproved by a Hive merger/owner for auto-merge on green CI
    on Nov 7, 2026
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 7, 2026
  3. hivecommons-hive commented on Nov 7, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by @thatguywob released (lease revoked).

    Claim history
    • 2026-11-07 07:32 UTC — claimed by thatguywob (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-07 07:56 UTC — released by thatguywob (contributor; lease revoked)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerCreated or modified by the scanner agentciApproved by a Hive merger/owner for auto-merge on green CIhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions