Skip to content

πŸ› fix: pause the 074 stall clock while an adopted PR waits in the fixer queue - #15131

Merged
hivecommons-hive[bot] merged 5 commits into
v6from
scanner/fix-15036
Oct 29, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
v6from
scanner/fix-15036

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #15036

An adopted PR's 074 stall clock ran while the PR was still unattempted in the busy single fixer's queue, so queued rebases parked at 085 adoption-stalled. The owner approved the dwell-pause direction.

  • The routing pass records when an adopted row is listed for the fixer (only while the fixer is available); a fixer kick that carries the row records the first attempt. Both are per head and reset on a new head.
  • adoptionStale: queued and unattempted (listed within 30m) pauses the dwell, capped at 4Γ— the 074 dwell from the head clock; once attempted, the dwell counts from the attempt; no queue record (unavailable fixer, same-repo or no queued row) behaves as today.
  • Docs row and changelog fragment added. Tests added; not run locally (CI is the verdict).

Decision taken by hive: the fixer-kick "attempt" is the row being listed in detail in a scanner FIX-BEFORE-NEW kick.

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@Danathar

hivecommons-hive Bot and others added 2 commits October 28, 2026 17:58
…er queue

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Danathar <6772335+Danathar@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design is sound and the wiring is complete end to end; two things in the diff itself make CI red.

1. TestFixerQueueRecordResetsOnNewHead fails on its own (job 123128926316, fixer_attempt_test.go:18). The test's first call is s.MarkAdoptedInPlace(key, "aaa", …) on a fresh NewStore, but MarkAdoptedInPlace is a no-op when no record exists (store.go:1067-1069, if !ok { return }), and MarkFixerQueued has the same guard (store.go:963). So Get(key) returns a zero record and FixerQueuedUnattempted is false. Seed the record first the way stagelabels_test.go:1184 does (s.Observe(key, Fixing, …) or equivalent) before the adopt/queue marks. The production code is not at fault here β€” those guards are the intended "only a live adoption gets a queue entry" behaviour.

2. changelog-fragment-guard fails: changelog.d/fixed-adoption-stall-fixer-queue.md:1 is a bare sentence without the - bullet the guard requires. fixed-15131.md already carries the entry; drop the second file or bullet it.

Verified correct

  • ci-failing.json rows do carry head_sha (merge_eligibility.go:1005 tag, populated at :1196), so formatRedPRFixData's new HeadSHA field (policy_overlays.go:248) is non-empty and markAdoptedFixerAttempt fires.
  • The attempt mark sits after the owner != agent filter (policy_overlays.go:288), so only the fixer that was actually handed the row records the attempt; and formatRedPRFixData is reached only via assembleKick β†’ buildAgentMessageFor (kickmessage.go:312,476), i.e. real kick assembly.
  • adoptionStale (automerge_contributor_adopt.go:1509-1518): attempt resets since; queued-unattempted pauses only within 4Γ—dwell; no record β†’ unchanged behaviour. FixerQueueFreshness (30m) stops the pause when writeMergeEligible stops listing the row (fixer paused/unavailable per fixerAvailable, merge_eligibility.go:920-927). The TestAdoptionStaleWaitsForFixerQueue table covers each branch.

Coverage failure should resolve once (1) passes.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read adoptionStale (automerge_contributor_adopt.go:1506-1522), the new Record/Store methods in stagelabels/store.go, and the two producers (merge_eligibility.go markAdoptedFixerQueued, scheduler/policy_overlays.go markAdoptedFixerAttempt). The mechanism is sound: both marks are keyed to AdoptedHead, a new head matches neither and MarkFixerQueued clears the attempt on a head change, fixerAvailable keeps a paused/cadence-less fixer from pausing the clock, and the 4-dwell cap bounds a fixer that lists but never kicks. The body matches the diff.

Two defects, both the reason CI is red:

  1. src/pkg/stagelabels/fixer_attempt_test.go:16-20 β€” TestFixerQueueRecordResetsOnNewHead fails (test, test (rest 3/4), coverage jobs). The test calls s.MarkAdoptedInPlace(key, "aaa", …) on a fresh store, but MarkAdoptedInPlace is a no-op when the record does not exist (store.go:1067-1070: r, ok := s.items[key]; if !ok { return }). So AdoptedHead stays empty, MarkFixerQueued bails at its r.AdoptedHead != head guard, and the first assertion sees an all-zero record β€” exactly what the CI log prints. The existing adoption tests create the record first (stagelabels_test.go:955 s.Observe(key, ReturnedToContributor, "check-failing:dco") before MarkAdoptedInPlace at :971); add the same Observe before line 20.

  2. changelog.d/fixed-adoption-stall-fixer-queue.md:1 starts with prose, not a - bullet, so changelog-fragment-guard fails (changelog.d/README.md: "a fragment must start with a '- ' entry bullet"). The bot-generated changelog.d/fixed-15131.md already carries a valid bullet, so delete the slug file or make it the single bulleted fragment. (Same issue on #15129 β€” worth checking the scanner's fragment template.)

Sign-off survives the squash is CANCELLED, not failed β€” infra.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

MarkAdoptedInPlace is a no-op on a key the store has never observed, so the
test asserted against an empty Record. Also give the changelog fragment the
'- ' bullet the guard requires.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 3980ccca5

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed β€” no findings from this perspective.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me, with one low-severity note on where the "attempt" is stamped.

Read the three new paths against v6:

  • src/cmd/hive/merge_eligibility.go:1240,1258 β€” markAdoptedFixerQueued runs only on the two adopted branches after routeConflictFixAgent resolved agent, and fixerAvailable mirrors the paused/no-cadence checks already used there, so an unavailable fixer still escalates at the dwell as the body says.
  • src/pkg/stagelabels/store.go β€” MarkFixerQueued/MarkFixerAttempted both refuse when r.AdoptedHead != head, and FixerQueuedUnattempted requires FixerQueuedSeenAt within 30m, so a new head or a queue that stops listing the row self-resets. No import cycle: go list -deps ./pkg/stagelabels pulls only pkg/authorkind.
  • src/pkg/github/automerge/automerge_contributor_adopt.go:1509-1517 β€” attempted-after-since moves the clock; queued-unattempted pauses only while now-since < 4Γ—dwell, so a dead fixer still escalates.

Note (low): markAdoptedFixerAttempt fires inside formatRedPRFixData (src/pkg/scheduler/policy_overlays.go:355), i.e. when the kick is rendered, not when it is delivered. src/cmd/hive/main.go:8086-8092 renders all due kicks via BuildKickMessagesWithSkips and then continues past agents in provider-inference backoff. For a fixer stuck in backoff the row is marked attempted on a kick it never received, which cuts the queue pause short and starts the dwell from that render. The outcome is still "escalate at one dwell" β€” the same as before this PR β€” so it only weakens the fix during provider outages rather than regressing anything. If you want the pause to hold through backoff, stamping the attempt where the message is actually handed to the agent (or after the filtered list is built) would do it; fine to leave as a follow-up.

Tests: fixer_attempt_test.go and the new cases in automerge_adoption_escape_test.go cover queued-pause, cap, attempted-shift and head reset; CI build-and-test is still running at this head.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the three new seams against the v6 tree (stagelabels/store.go, automerge_contributor_adopt.go:adoptionStale, scheduler/policy_overlays.go, cmd/hive/merge_eligibility.go). The mechanism holds together; one timing nuance below, not a blocker.

Verified

  • failingPR already serialises head_sha (src/cmd/hive/merge_eligibility.go:985), so formatRedPRFixData's new HeadSHA field is populated and markAdoptedFixerAttempt is not a silent no-op.
  • MarkFixerQueued / MarkFixerAttempted both require r.AdoptedHead == head, and a differing FixerQueuedHead clears the attempt β€” so a new head resets both without any extra plumbing, as TestFixerQueueRecordResetsOnNewHead asserts.
  • adoptionStale: the attempted.After(since) branch moves the clock forward only; the queued branch is bounded by adoptionQueuedDwellCap from the head clock, so a fixer that is listed but never kicks still escalates at 4 dwells. The "no record β†’ today's behaviour" fallback is intact.
  • fixerAvailable mirrors the paused/cadence checks already used for routing in this file, so an unavailable fixer never pauses the dwell.

Low β€” attempt is recorded at render time, not delivery (src/pkg/scheduler/policy_overlays.go:355)
formatRedPRFixData marks the attempt while the kick body is being assembled. Fixer kicks are built in BuildKickMessagesWithSkips and can still be withheld afterwards: provider-error backoff (src/cmd/hive/main.go:8080-8092), the spending-limit gate (:8115-8125), and the self-upgrade drain that follows. In those cycles the head is marked attempted although no fixer session ever saw it, so the queue pause ends early and the 074 clock runs from the render time. Consequence is bounded β€” that is still no earlier than today's head clock β€” so this degrades to the pre-PR behaviour rather than below it. If you want the pause to be exact, moving the mark to where the kick is actually dispatched (or clearing it for withheld messages) would do it; fine as a follow-up.

Open question I did not settle: whether a kick for a fixer whose previous session is still running is rendered and queued (which would also mark the attempt early) or not built at all. partitionKickableAgents filters paused agents, not running ones, so it may be worth a look.

Body matches the diff; Closes #15036 is right.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 28, 2026
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: build-and-test at 2908b23 failed only on test_backend_smoke.sh S4 [short] ('Task prompt sent to CLI' log line missing, printf Broken pipe) β€” a relay timing flake unrelated to this diff; v6 baseline and sibling PRs are green on the same check. Merged v6 into the branch for a fresh run. Sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me, with two low-severity points worth a maintainer glance. Read the diff plus MarkAdoptedHead/MarkAdoptedInPlace on v6 (store.go:1000-1045) and the overlay call chain kick_layout.go:117 β†’ addRedPRFixFirst β†’ formatRedPRFixData.

Verified:

  • store.go β€” MarkFixerQueued/MarkFixerAttempted only act when r.AdoptedHead == head, and MarkFixerQueued clears the attempt on a head change, so both records are per head as the body states; FixerQueuedUnattempted returns false once attempted or after FixerQueueFreshness.
  • automerge_contributor_adopt.go:1509-1518 β€” since only moves to the attempt time when attempted.After(since); the pause branch is bounded by dwell*adoptionQueuedDwellCap from the head clock, so a never-attempted row still escalates at 4 dwells. The no-record path (FixerAttemptedSince zero and not queued) falls through to the existing now.Sub(since) >= dwell, unchanged.
  • merge_eligibility.go β€” markAdoptedFixerQueued is gated by fixerAvailable, which reuses the existing allConfiguredCadencesPaused plus HasAnyCadence (config/expected_active.go:56), so an unavailable fixer leaves rows escalating as today.

Two points, both erring toward earlier escalation (i.e. today's behaviour), so neither is a regression:

  1. policy_overlays.go:355 β€” the attempt is recorded while the kick message is composed (formatRedPRFixData), not when it is delivered. If a composed kick is later dropped (pacing, budget), the row counts as attempted and the dwell restarts from that moment. Acceptable per the body's stated decision; noting it because a format* function now has a store side effect, which a future reader may not expect.
  2. FixerQueueFreshness = 30m (store.go:936) is refreshed only by the routing pass in writeMergeEligible. If that pass runs less often than every 30 minutes on a quiet hive, the pause lapses between passes and this fix silently does nothing. Open question for whoever knows the sweep cadence β€” I could not confirm it from the tree.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. Read adoptionStale on v6 and traced both record paths end to end.

  • automerge_contributor_adopt.go:1509-1518 β€” the new branch sits after the existing head-match / CI-pending / merge-commit guards (:1497-1508), so a moved head or running CI still short-circuits before any queue state is consulted. Attempt restarts the clock only when attempted.After(since); the queue pause is bounded by adoptionQueuedDwellCap (4 dwells from the head clock), so a dead fixer still escalates.
  • stagelabels/store.go β€” MarkFixerQueued/MarkFixerAttempted both refuse to write unless r.AdoptedHead == head, and a new head clears the attempt (FixerQueuedHead != head path), so the per-head reset the body describes is real, not implied. FixerQueuedUnattempted requires FixerQueuedSeenAt within 30m, so a row the routing pass stopped listing (CI running, fixer paused) stops pausing on its own.
  • merge_eligibility.go β€” markAdoptedFixerQueued is gated by fixerAvailable (paused agent / no cadence / all cadences paused), matching the stated "only while the fixer is available". The failingPR struct already serialises head_sha (merge_eligibility.go:983), so the new HeadSHA field in formatRedPRFixData's decoder reads a value that is actually emitted.
  • All checks green except tide, so no import cycle from scheduler β†’ stagelabels.

One low note: formatRedPRFixData (policy_overlays.go:355) now has a persistence side effect inside a formatting function. It's idempotent per head and the only caller is addRedPRFixFirst, so it's harmless today, but a future caller that formats for preview/dry-run would stamp a fixer attempt that never happened. Worth a comment at the call site, or lifting the mark into addRedPRFixFirst; not blocking.

Body matches the diff; Closes #15036.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 9f58413 into v6 Oct 29, 2026
75 of 77 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-15036 branch October 29, 2026 00:16
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#15131272b6c5` sha256:f1c82743e50b4d5e586669de030d5b42d6cd6cbfb0af8734196ab6211c397e12 β€” download from the dashboard or `hivectl review evidence #15131`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› [v6] adoption: the 074 stall clock runs while the adopted PR waits unattempted in the fixer queue

0 participants