Repository navigation
π Run the reconcile pass when wakes outpace the eval interval; age unblocked rows from the lift - #15787
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Problem 1 (reconcile starvation) reads correct to me; Problem 2 (tracker "since" after an unblock) only holds until the next issue stage-label pass, which deletes the record it depends on.
Problem 2 β the lift record does not survive the pass (medium)
The symptom in #15779 is a stage-less row ("New Β· since 17h"): the issue's stage decision is None, which is why it has no record and ages from creation. The fix stores the lift on a stage-less record (src/pkg/stagelabels/store.go:753-763, NoteUnblocked) and the tracker reads it via the new !hasRec && !rec.UnblockedAt.IsZero() branch (src/pkg/dashboard/stage_tracker_build.go:138-141).
But the issue stage-label pass drops every record whose decision is None:
// src/pkg/github/issue_stage_labels.go:478-479 (at 2951383)
if stage == stagelabels.None {
store.Forget(key)Forget deletes unconditionally (store.go:1899-1906). So for exactly the row the issue describes β one that stays unstaged after the unblock β the sequence is: blocked sweep lifts β NoteUnblocked writes the record β next SyncIssueStageLabels (every reconcile cycle) decides None β Forget β row is back to "since ". The reset is visible only for the window between the sweep and the following stage-label pass. The PR body's own sentence ("dropped by a None observation") describes this; it just isn't the behaviour the issue asks for.
Neither new test exercises that path: unblocked_test.go asserts that Observe(key, None, "") drops the record (the opposite expectation), and stage_tracker_unblocked_since_test.go never runs the stage-label pass between NoteUnblocked and buildStageTracker.
Suggested direction: in issue_stage_labels.go:478 (and ObserveProgress's None branch if you want symmetry), keep a record that carries only UnblockedAt instead of deleting it β e.g. clear the stage fields but retain the lift, or make Forget a no-op for stage-less lift-only records (only the merged/closed paths at :464 and Prune should truly drop it). Then add the stage-label pass to the tracker test.
Problem 1 β looks correct
Read runLoopWith at the PR head: reconcileOverdue is consulted in tick (main.go:5911-5919, cached β enumerate and lastEnumerateAt reset on every enumerate), in the continuous re-kick path (:6121-6124), and at the loop top (:6147-6160), which also drains the pending ticker tick so the cadence tick does not immediately repeat the pass. lastEvalInterval is the same variable the healthCh case resets the ticker from (:6206-6210), so the threshold tracks interval changes. No issue found.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
stage 080-awaiting-merge-admission β 071-needs-author: optional-check-failing:test (rest 4/4) |
There was a problem hiding this comment.
The test (rest 4/4) red is this diff's fault, and it is confined to the two new tests β the production change looks correct to me.
Data race in both new tests (-race fails cmd/hive; coverage and test fail downstream of it):
src/cmd/hive/reconcile_starvation_test.go:32and:74reassign the localclockfrom the test goroutine after<-h.persisted, while the loop goroutine keeps reading it through the closure at:24/:68(deps.now = func() time.Time { return clock }).- The read that races is the new loop-top check at
src/cmd/hive/main.go:6147(if now := loopNow(); β¦ reconcileOverdue(now)), which runs on every iteration afterpersisthas already signalled the test β sopersisteddoes not establish happens-before for the nextclockwrite. The race detector reports exactly this pair (write atreconcile_starvation_test.go:32, previous read viamain.go:6147). - Fix: make the fake clock goroutine-safe β e.g.
var clock atomic.Pointer[time.Time]/ a mutex-guarded getter β and havedeps.nowread through it; the test logic itself is fine.
Production change β verified, no findings: lastEnumerateAt is only advanced inside tick when the source is (or is upgraded to) evalSourceEnumerate; reconcileOverdue is the single predicate used by the wake path (re-kick β full tick instead of lean cycle), the cached-snapshot upgrade inside tick, and the loop top. Draining one pending ticker.Chan() after an overdue pass only drops a tick that would repeat the pass just run. stagelabels/store.go and stage_tracker_build.go additions are additive.
Not a defect, a suggestion: TestRunLoopWakeEnumeratesWhenSnapshotOlderThanInterval asserts the exact source sequence, which is good β once the clock is race-free it is a genuine regression test for #15779.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
hive/ci-report (scanner):
π Hive Agent: β hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
β¦blocked rows from the lift Wake-driven cycles (review, verdict, webhook, continuous re-kick) plan from the cached snapshot; only the cadence ticker enumerates, and on a busy spoke its tick lands during long ticks and then competes with always-ready wake channels, so the snapshot aged 50m-1h30m with no reconcile pass. The loop now tracks the last reconcile start: a wake cycle past the interval enumerates instead, and the loop top runs an owed reconcile pass whichever channel woke it. The blocked sweep records its lift on the stage store; an unblocked issue's tracker row ages from the lift, not from issue creation. Closes #15779 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
β¦est clock Review: the issue stage-label pass Forget()s every None decision, which dropped the lift record right after the blocked sweep wrote it. Open issues deciding None now go through ForgetStage, which keeps a lift-only record; Forget (closed/merged) still drops everything. The tracker only ages from the lift when the row is no longer blocked. The new loop tests advanced a plain closure clock from the test goroutine while the loop read it; a mutex-guarded clock fixes the race, and the health re-check test now waits for the overdue pass it asserts. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
Addressed both reviews: the stage-label pass now goes through |
2951383 to
cfeacca
Compare
There was a problem hiding this comment.
Read both halves against the tree at the merge-base. Looks correct to me; one non-blocking observation.
Reconcile starvation (src/cmd/hive/main.go)
reconcileOverdue(main.go:5894-5896) is consulted from all three places a cached cycle can originate:tick()convertsevalSourceCachedβevalSourceEnumeratewhen overdue (main.go:5911-5919), the continuous re-kick path callstick(evalSourceCached)instead of the leanrunEvalwhen overdue (main.go:6121-6124), and the loop top runstick(evalSourceEnumerate)before theselectregardless of which channel fired (main.go:6147-6160).lastEnumerateAtis stamped insidetickwhenever the resolved source is enumerate, so a wake that got promoted and the loop-top check cannot double-enumerate in the same pass.- The non-blocking ticker drain after the loop-top pass (main.go:6156-6159) is right: the pending cadence tick would only repeat the pass that just ran.
lastEvalIntervalis declared at main.go:5771 and updated insidetick(main.go:6004-6008), so the closure sees mode changes. ThereviewWake/verdictWake/webhookWakecases all go throughtick(...), so they inherit the promotion without edits.
Unblocked-since aging (stagelabels, dashboard)
NoteUnblockedis only called fromClient.unblockIssue(item_refresh_hook.go:54), i.e. issue keys. The dashboardGetcallers that treatokas "has a stage" (api_repo_item_decision_reroute.go:76,ensureStageRecord) are all PR-key paths, so a stage-less issue record does not reach them.buildStageTracker's lookup wrapper (stage_tracker_build.go:83-89) masks the stage-less record back took=false, so the rest of that builder keeps its existing contract.SyncIssueStageLabelsWithswitchingForgetβForgetStage(issue_stage_labels.go:480) is what makes this work: without it the stage-label pass that runs right after an unblock would delete the lift.Forget(closed/merged) still drops everything.ObserveProgresscopiesprevintonext(store.go:663), soUnblockedAtsurvives staging; once staged,StageSinceAt()returnsEnteredAtbecause it is later than the lift β so the row's "since" moves from lift-time to stage-entry-time on the first stage pass. That matches the doc comment onStageSinceAtand seems intended; noting it only so nobody is surprised that the lift time is a transient display state for issues the hive goes on to stage.
Tests: TestRunLoopWakeEnumeratesWhenSnapshotOlderThanInterval, TestRunLoopTopRunsOverdueReconcileWithoutTickerTick, TestNoteUnblockedAgesRecordFromLift, TestStageTrackerNewRowAgesFromUnblock cover each branch above. Nothing blocking from me.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
stage 080-awaiting-merge-admission: no activity for 30m0s: merge-train:queued-behind#15748:3/9:0/0:main.go,main_sweeps.go,auth.md,dashboard-misc.md,+14 |
|
stage 080-awaiting-merge-admission β 060-reviewed: check-pending |
|
stage 080-awaiting-merge-admission β 050-review-pending: required-check-pending:build-and-test |
|
stage 080-awaiting-merge-admission β 050-review-pending: required-check-pending:build-and-test |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Closes #15779
Problem 1 β reconcile pass starved by wake-driven cached cycles
On hive's own spoke (webhook mode, 900s effective interval) the snapshot aged 50mβ1h30m between reconcile passes while webhook / verdict / continuous re-kick wakes ran a cached cycle every ~5 minutes. Only the cadence ticker runs
evalSourceEnumerate; every wake path runsevalSourceCached. On a busy spoke the ticker's tick lands while a long tick is running (a reconcile tick with sweeps ran 09:06β09:24, past the next 900s edge) and then competes in theselectwith wake channels that are ready again almost immediately β nothing enforced "one reconcile pass per interval", so issues sat at "New Β· waiting for the stage-label pass" for hours.Fix in
runLoopWith:lastEnumerateAt;reconcileOverdue(now)is true once a full effective interval has passed since the last reconcile pass started;reviewWake, verdict, webhook, continuous re-kick) that is overdue enumerates instead β logcached snapshot older than eval interval; enumerating;eval interval elapsed without a reconcile pass; enumerating) and drains the then-redundant pending ticker tick, so a lost or outraced tick cannot starve enumeration.Tests (
cmd/hive/reconcile_starvation_test.go, fake loop deps, no sleeps): wakes every 5m against a 900s interval still produce an enumerate within the interval; a loop iteration with no ticker tick at all runs the overdue reconcile as a full tick.TestRunLoopHealthRecheckResetsTickerWhenWebhooksLapsenow expects the one overdue reconcile pass that follows a cadence drop from 900s to 60s after 121s.Problem 2 β tracker "since" keeps counting across an unblock
A blocked issue has no stage record (its stage is None), so after the blocked sweep lifted the block the row aged from issue creation ("New Β· since 17h" for an issue unblocked 20m ago).
unblockIssuenow records the lift viaStore.NoteUnblocked(a stage-less record carryingUnblockedAt, kept by the next staging observation, dropped by a None observation); the tracker ages the row fromRecord.StageSinceAt()β the later of stage entry and the lift β and keeps the unstaged row 000-new. Caption text is unchanged.Tests:
pkg/stagelabels/unblocked_test.go,pkg/dashboard/stage_tracker_unblocked_since_test.go.