Skip to content

πŸ› Run the reconcile pass when wakes outpace the eval interval; age unblocked rows from the lift - #15787

Merged
clubanderson merged 4 commits into
v5from
fix/reconcile-starvation
Oct 30, 2026
Merged

clubanderson merged 4 commits into
v5from
fix/reconcile-starvation

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Closes #15779

Problem 1 β€” reconcile pass starved by wake-driven cached cycles

On hive's own spoke (webhook mode, 900s effective interval) the snapshot aged 50m–1h30m between reconcile passes while webhook / verdict / continuous re-kick wakes ran a cached cycle every ~5 minutes. Only the cadence ticker runs evalSourceEnumerate; every wake path runs evalSourceCached. On a busy spoke the ticker's tick lands while a long tick is running (a reconcile tick with sweeps ran 09:06β†’09:24, past the next 900s edge) and then competes in the select with wake channels that are ready again almost immediately β€” nothing enforced "one reconcile pass per interval", so issues sat at "New Β· waiting for the stage-label pass" for hours.

Fix in runLoopWith:

  • track lastEnumerateAt; reconcileOverdue(now) is true once a full effective interval has passed since the last reconcile pass started;
  • a wake tick (reviewWake, verdict, webhook, continuous re-kick) that is overdue enumerates instead β€” log cached snapshot older than eval interval; enumerating;
  • the loop top runs an owed reconcile pass whichever channel woke it (log eval interval elapsed without a reconcile pass; enumerating) and drains the then-redundant pending ticker tick, so a lost or outraced tick cannot starve enumeration.

Tests (cmd/hive/reconcile_starvation_test.go, fake loop deps, no sleeps): wakes every 5m against a 900s interval still produce an enumerate within the interval; a loop iteration with no ticker tick at all runs the overdue reconcile as a full tick. TestRunLoopHealthRecheckResetsTickerWhenWebhooksLapse now expects the one overdue reconcile pass that follows a cadence drop from 900s to 60s after 121s.

Problem 2 β€” tracker "since" keeps counting across an unblock

A blocked issue has no stage record (its stage is None), so after the blocked sweep lifted the block the row aged from issue creation ("New Β· since 17h" for an issue unblocked 20m ago). unblockIssue now records the lift via Store.NoteUnblocked (a stage-less record carrying UnblockedAt, kept by the next staging observation, dropped by a None observation); the tracker ages the row from Record.StageSinceAt() β€” the later of stage entry and the lift β€” and keeps the unstaged row 000-new. Caption text is unchanged.

Tests: pkg/stagelabels/unblocked_test.go, pkg/dashboard/stage_tracker_unblocked_since_test.go.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 30, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15804 ✨ Show the upgrade animation for every roll (announce + SIGTERM fallback) same files src/cmd/hive/main.go
#15865 ✨ feature: carry abandoned counts through the hub leaderboard same files src/cmd/hive/main.go
#15831 πŸ› Merge train seats only PRs green and reviewed on their current head; regressed members are ejected same files src/pkg/dashboard/stage_tracker_build.go
src/pkg/stagelabels/store.go
#15757 ✨ Auto-merge hive-authored superseding PRs without operator decision same files src/cmd/hive/main.go
#15748 πŸ› [reviewer] park PRs at 085 Needs decision when the fixer exhausted its attempts same files src/cmd/hive/main.go
src/pkg/dashboard/stage_tracker_build.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem 1 (reconcile starvation) reads correct to me; Problem 2 (tracker "since" after an unblock) only holds until the next issue stage-label pass, which deletes the record it depends on.

Problem 2 β€” the lift record does not survive the pass (medium)

The symptom in #15779 is a stage-less row ("New Β· since 17h"): the issue's stage decision is None, which is why it has no record and ages from creation. The fix stores the lift on a stage-less record (src/pkg/stagelabels/store.go:753-763, NoteUnblocked) and the tracker reads it via the new !hasRec && !rec.UnblockedAt.IsZero() branch (src/pkg/dashboard/stage_tracker_build.go:138-141).

But the issue stage-label pass drops every record whose decision is None:

// src/pkg/github/issue_stage_labels.go:478-479 (at 2951383)
if stage == stagelabels.None {
    store.Forget(key)

Forget deletes unconditionally (store.go:1899-1906). So for exactly the row the issue describes β€” one that stays unstaged after the unblock β€” the sequence is: blocked sweep lifts β†’ NoteUnblocked writes the record β†’ next SyncIssueStageLabels (every reconcile cycle) decides None β†’ Forget β†’ row is back to "since ". The reset is visible only for the window between the sweep and the following stage-label pass. The PR body's own sentence ("dropped by a None observation") describes this; it just isn't the behaviour the issue asks for.

Neither new test exercises that path: unblocked_test.go asserts that Observe(key, None, "") drops the record (the opposite expectation), and stage_tracker_unblocked_since_test.go never runs the stage-label pass between NoteUnblocked and buildStageTracker.

Suggested direction: in issue_stage_labels.go:478 (and ObserveProgress's None branch if you want symmetry), keep a record that carries only UnblockedAt instead of deleting it β€” e.g. clear the stage fields but retain the lift, or make Forget a no-op for stage-less lift-only records (only the merged/closed paths at :464 and Prune should truly drop it). Then add the stage-label pass to the tracker test.

Problem 1 β€” looks correct

Read runLoopWith at the PR head: reconcileOverdue is consulted in tick (main.go:5911-5919, cached β†’ enumerate and lastEnumerateAt reset on every enumerate), in the continuous re-kick path (:6121-6124), and at the loop top (:6147-6160), which also drains the pending ticker tick so the cadence tick does not immediately repeat the pass. lastEvalInterval is the same variable the healthCh case resets the ticker from (:6206-6210), so the threshold tracks interval changes. No issue found.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (rest 4/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 071-needs-author: optional-check-failing:test (rest 4/4)

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test (rest 4/4) red is this diff's fault, and it is confined to the two new tests β€” the production change looks correct to me.

Data race in both new tests (-race fails cmd/hive; coverage and test fail downstream of it):

  • src/cmd/hive/reconcile_starvation_test.go:32 and :74 reassign the local clock from the test goroutine after <-h.persisted, while the loop goroutine keeps reading it through the closure at :24/:68 (deps.now = func() time.Time { return clock }).
  • The read that races is the new loop-top check at src/cmd/hive/main.go:6147 (if now := loopNow(); … reconcileOverdue(now)), which runs on every iteration after persist has already signalled the test β€” so persisted does not establish happens-before for the next clock write. The race detector reports exactly this pair (write at reconcile_starvation_test.go:32, previous read via main.go:6147).
  • Fix: make the fake clock goroutine-safe β€” e.g. var clock atomic.Pointer[time.Time] / a mutex-guarded getter β€” and have deps.now read through it; the test logic itself is fine.

Production change β€” verified, no findings: lastEnumerateAt is only advanced inside tick when the source is (or is upgraded to) evalSourceEnumerate; reconcileOverdue is the single predicate used by the wake path (re-kick β†’ full tick instead of lean cycle), the cached-snapshot upgrade inside tick, and the loop top. Draining one pending ticker.Chan() after an overdue pass only drops a tick that would repeat the pass just run. stagelabels/store.go and stage_tracker_build.go additions are additive.

Not a defect, a suggestion: TestRunLoopWakeEnumeratesWhenSnapshotOlderThanInterval asserts the exact source sequence, which is good β€” once the clock is race-free it is a genuine regression test for #15779.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

hive/ci-report (scanner): test (rest 4/4) is red at 2951383 because of a data race in the new test, not infrastructure.

  • TestRunLoopWakeEnumeratesWhenSnapshotOlderThanInterval and TestRunLoopTopRunsOverdueReconcileWithoutTickerTick fail with testing.go:1712: race detected during execution of test (job 124121837017).
  • Race: write at src/cmd/hive/reconcile_starvation_test.go:32 (test goroutine) vs. previous read at reconcile_starvation_test.go:24 (run-loop goroutine). The fake/stub the test mutates after starting the loop is read concurrently by the loop; guard it with a mutex or atomic, or set it before the loop starts.

coverage is red only because it depends on that shard. No push from the hive on an author-owned branch; please push a new head and CI re-runs.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…blocked rows from the lift

Wake-driven cycles (review, verdict, webhook, continuous re-kick) plan
from the cached snapshot; only the cadence ticker enumerates, and on a
busy spoke its tick lands during long ticks and then competes with
always-ready wake channels, so the snapshot aged 50m-1h30m with no
reconcile pass. The loop now tracks the last reconcile start: a wake
cycle past the interval enumerates instead, and the loop top runs an
owed reconcile pass whichever channel woke it.

The blocked sweep records its lift on the stage store; an unblocked
issue's tracker row ages from the lift, not from issue creation.

Closes #15779

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…est clock

Review: the issue stage-label pass Forget()s every None decision, which
dropped the lift record right after the blocked sweep wrote it. Open
issues deciding None now go through ForgetStage, which keeps a lift-only
record; Forget (closed/merged) still drops everything. The tracker only
ages from the lift when the row is no longer blocked.

The new loop tests advanced a plain closure clock from the test
goroutine while the loop read it; a mutex-guarded clock fixes the race,
and the health re-check test now waits for the overdue pass it asserts.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Addressed both reviews: the stage-label pass now goes through Store.ForgetStage for open issues deciding None, keeping a lift-only record (test: unblocked_test.go, and the tracker test now runs the None pass between the lift and the build); the loop tests use a mutex-guarded clock and the health re-check test waits for the overdue pass before cancelling (go test -race ./cmd/hive/ green locally). Rebased onto v5.

@clubanderson
clubanderson force-pushed the fix/reconcile-starvation branch from 2951383 to cfeacca Compare October 30, 2026 17:20

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read both halves against the tree at the merge-base. Looks correct to me; one non-blocking observation.

Reconcile starvation (src/cmd/hive/main.go)

  • reconcileOverdue (main.go:5894-5896) is consulted from all three places a cached cycle can originate: tick() converts evalSourceCached β†’ evalSourceEnumerate when overdue (main.go:5911-5919), the continuous re-kick path calls tick(evalSourceCached) instead of the lean runEval when overdue (main.go:6121-6124), and the loop top runs tick(evalSourceEnumerate) before the select regardless of which channel fired (main.go:6147-6160). lastEnumerateAt is stamped inside tick whenever the resolved source is enumerate, so a wake that got promoted and the loop-top check cannot double-enumerate in the same pass.
  • The non-blocking ticker drain after the loop-top pass (main.go:6156-6159) is right: the pending cadence tick would only repeat the pass that just ran.
  • lastEvalInterval is declared at main.go:5771 and updated inside tick (main.go:6004-6008), so the closure sees mode changes. The reviewWake/verdictWake/webhookWake cases all go through tick(...), so they inherit the promotion without edits.

Unblocked-since aging (stagelabels, dashboard)

  • NoteUnblocked is only called from Client.unblockIssue (item_refresh_hook.go:54), i.e. issue keys. The dashboard Get callers that treat ok as "has a stage" (api_repo_item_decision_reroute.go:76, ensureStageRecord) are all PR-key paths, so a stage-less issue record does not reach them. buildStageTracker's lookup wrapper (stage_tracker_build.go:83-89) masks the stage-less record back to ok=false, so the rest of that builder keeps its existing contract.
  • SyncIssueStageLabelsWith switching Forget β†’ ForgetStage (issue_stage_labels.go:480) is what makes this work: without it the stage-label pass that runs right after an unblock would delete the lift. Forget (closed/merged) still drops everything.
  • ObserveProgress copies prev into next (store.go:663), so UnblockedAt survives staging; once staged, StageSinceAt() returns EnteredAt because it is later than the lift β€” so the row's "since" moves from lift-time to stage-entry-time on the first stage pass. That matches the doc comment on StageSinceAt and seems intended; noting it only so nobody is surprised that the lift time is a transient display state for issues the hive goes on to stage.

Tests: TestRunLoopWakeEnumeratesWhenSnapshotOlderThanInterval, TestRunLoopTopRunsOverdueReconcileWithoutTickerTick, TestNoteUnblockedAgesRecordFromLift, TestStageTrackerNewRowAgesFromUnblock cover each branch above. Nothing blocking from me.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission: no activity for 30m0s: merge-train:queued-behind#15748:3/9:0/0:main.go,main_sweeps.go,auth.md,dashboard-misc.md,+14

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 060-reviewed: check-pending

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 050-review-pending: required-check-pending:build-and-test

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 050-review-pending: required-check-pending:build-and-test

@clubanderson
clubanderson merged commit 7f3f140 into v5 Oct 30, 2026
75 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/reconcile-starvation branch October 30, 2026 19:28
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› Reconcile pass starved by wake-driven cached cycles; tracker age keeps counting across unblock

1 participant