Skip to content

[ci-maintainer] stage reconciler flaps 070-fixing ↔ 085-needs-decision on #16607 every few minutes — ~120 workflow runs/hour on an unchanged head, native-queue-policy never settles #16880

Closed

Description

@hivecommons-hive

CI Issue

The stage reconciler is flapping PR #16607 (scanner/fix-16265, head 687f4cb) between stage/070-fixing and stage/085-needs-decision every 1–10 minutes, and has been since 2026-11-05 ~15:00Z — about 19 hours at the time of filing, with no push, review, comment or other external event in between. Each flip is a labeled + unlabeled pair, and each of those fires three workflows (changelog-fragment-guard, DCO squash attribution, Native queue policy), so one idle PR is generating ~120 workflow runs per hour on an unchanged head.

The visible CI damage is on Native queue policy: it is pull_request_target with cancel-in-progress: true keyed on the PR number and waits up to 22.5 min for evidence, so almost every run is cancelled by the next flip before it can report. The native-queue-policy check on #16607 shows pending and almost never settles.

Evidence

  • GET repos/hivecommons/hive/issues/16607/timeline (2026-11-06 ~10:00Z, truncated at 300 events by the API): 133 labeled + 126 unlabeled events, all by hivecommons-hive[bot]; 70× stage/070-fixing on/off, 64× stage/085-needs-decision on, 63× off, 13× stage/050-review-pending on/off. Per-hour label-event counts climb from 12 (15Z) to 44 (20Z, 21Z) and are still 30–40/hour at 09Z today.
  • Non-label events after the head was pushed (2026-11-05T15:00:48Z): one reviewed (bot adjudicator, 15:16:46Z, "HUMAN DECISION NEEDED — confirm the hive-merge relay may App-merge trusted-author pin-only PRs that touch auto_merge.human_merge_paths") and one bot comment (18:44:54Z). Nothing else. The flips are not reacting to new facts.
  • gh run list --branch scanner/fix-16265 --limit 1000: 1000 runs fit in 8.2 hours (01:44Z → 09:57Z); breakdown Native queue policy 332 cancelled / 1 running, changelog-fragment-guard 331 success, DCO squash attribution 167 success / 167 cancelled. All on head 687f4cb.
  • gh run list --workflow native-queue-policy.yml --limit 600 (00:12Z → 09:57Z): 423 runs on this branch — 402 cancelled, 20 success. The next-busiest branch in the same window has 17 runs. The flap is a single-PR outlier, not a fleet pattern.
  • gh run list --limit 300: the 300 most recent runs in the repository span only 90 minutes, and 144 of them are this one branch.
  • Where the two opinions come from (v5 @ 8b5a160): src/pkg/stageengine/derive_pr_classify.go — RequiresHumanStage returns 085 for a requires_human verdict with no trusted-human approval on the head (line ~577); PreserveNormalizedFixingStage (~585) converts that same verdict back to 070-fixing / LaneReasonChangesRequested when the previous record was already 070 on the same ActivityHead; PreserveStageOnStaleDecisionAnswer (~614) does the reverse for 085. The observed sequence — 070 → 085 → 070 with the same head and the same verdict — looks like the two preserve-paths taking turns depending on which record the reconciler saw last, rather than a stable fixed point.

Recommendation

Fix in the stage engine, not in the workflows — the three workflows run on labeled/unlabeled on purpose (changelog-fragment-guard.yml header, #14755/#14629: skipping or cancelling label runs left a required check scored by a stale sibling) and must keep doing so.

  1. Make the 070↔085 derivation for a requires_human verdict idempotent on the same HeadSHA: whichever of RequiresHumanStage / PreserveNormalizedFixingStage wins the first time for a given (head, verdict) must keep winning until the head, the verdict, a trusted-human review or a decision reply changes. A regression test that feeds the reconciler the same observation twice (prev = result of the first pass) and asserts no transition is the minimal fail-without-fix test.
  2. Add a generic anti-flap guard in the label writer (src/pkg/github/stage_writer.go / src/pkg/stageengine/writer.go): refuse to apply a stage transition that exactly reverses the previous transition on the same head within a short window (e.g. 15 min) and log it as stage flap suppressed instead. This bounds the blast radius of any future derivation bug: a flap would cost one label pair per window instead of one per sweep.
  3. Operationally, for ✨ feat(github): enforce pin-only exception in merge-request relay #16607 itself: the adjudicator's question is a merge/trust-policy decision for @hanthor ([scanner] human-merge pin-only exception: enforce in merge-request relay #16265 / feat: bounded trusted pin-only exception to per-repository human-merge paths #16242). Answering it stops the churn today; it does not fix the engine.

Decision taken by hive: filed as one issue for the single root cause (the reconciler oscillation); the per-workflow symptoms are not filed separately because the workflows' label-event behaviour is intentional and documented in-tree.

Done when: #16607 (or a reproduction PR) holds a single stage label across ≥3 consecutive reconciler sweeps with no external event, and gh run list --branch <head> shows no new Native queue policy runs between pushes.

Refs #16607 (affected PR; left open — it is the scanner's feature PR, not this bug).


Filed by ci-maintainer agent (ACMM L6 — full mode)


🐝 Hive Agent: ci-maintainer | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug.
    agent/ci-maintainerCreated or modified by the ci-maintainer agent
    ciApproved by a Hive merger/owner for auto-merge on green CI
    on Nov 6, 2026
  2. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 6, 2026
  3. hivecommons-hive commented on Nov 6, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by @mendezr released (lease revoked).

    Claim history
    • 2026-11-06 10:03 UTC — claimed by mendezr (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-06 10:13 UTC — released by mendezr (contributor; lease revoked)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/ci-maintainerCreated or modified by the ci-maintainer agentciApproved by a Hive merger/owner for auto-merge on green CIhelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions