Repository navigation
✨ Show the upgrade animation for every roll (announce + SIGTERM fallback) - #15804
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test |
|
stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test |
There was a problem hiding this comment.
Style review (head 9dfbfd4) — approve; two advisory notes, worst first.
src/pkg/dashboard/upgrade_announce.go:307— optional-body decode compares the error string (err.Error() != "EOF"). The package idiom for an optional JSON body iserrors.Is(err, io.EOF)(seepkg/dashboard/bob_key_probe.go:224); string-matching an error is brittle and is the only such comparison inpkg/.
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&req); err != nil && !errors.Is(err, io.EOF) {
(add "errors" and "io" to the imports.)
src/pkg/dashboard/upgrade_announce.go:362-383—startupPhaseLabelsrepeats everycmd/hiveboot-phase name as a string literal (boot_sequence.go:36-54), and nothing checks the two tables agree: a renamed or added phase silently falls back to the rawbootXxxidentifier in the banner.TestDefaultBootSequenceOrderalready importsdashboardat this head, so pinning the coupling is one loop:
--- a/src/cmd/hive/boot_sequence_test.go
+++ b/src/cmd/hive/boot_sequence_test.go
@@ -43,4 +43,9 @@ func TestDefaultBootSequenceOrder(t *testing.T) {
if got := defaultBootSequenceNames(); strings.Join(got, ",") != strings.Join(want, ",") {
t.Fatalf("names = %v, want %v", got, want)
}
+ for _, name := range append(defaultBootSequenceNames(), bootPhaseFirstEval) {
+ if dashboard.StartupPhaseLabel(name) == name {
+ t.Errorf("boot phase %q has no label in dashboard.startupPhaseLabels (#15788)", name)
+ }
+ }
}— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Correctness review (head 9dfbfd4)
Hub restart_spoke leaves a 15‑minute phantom "restart announced" banner on the freshly restarted pod
heartbeatUpgradeAnnounce returns a hub announce (Target:"", reason "hub restart") whenever the response carries RestartSpoke (src/pkg/hub/heartbeat_upgrade_announce.go:31-33). The hub re-delivers RestartSpoke on every beat for a multi-minute window — that is exactly why spokeRestartMinUptime exists (src/cmd/hive/main.go:464-467) and why RestartSpokeCallback ignores the instruction when uptime < 10 min (main.go:4719-4723). But the new UpgradeAnnounceCallback has no such guard (main.go:5332-5336): the restarted pod receives the same beats, calls AnnounceUpgrade, and
/api/status//api/versionreportpulling/announced("restart announced by the hub … ago") forupgradeAnnounceStaleAfter= 15 min (src/pkg/dashboard/upgrade_announce.go:36,:110-114); repeated beats keep the originalAtviasameAnnounceTarget("","")(:192-194), so it does not reset but does persist the full window;- each beat also broadcasts an SSE
upgradingframe (:196), whichversionRollRecordFromEventturns into a new roll record because the previous one iscompletedAt(static-src/js/version.jsdiff:if (prev && prev.source !== 'roll' && !prev.completedAt) return prev;), so right after "Restarted ✓" the bee animation starts again and a freshly opened tab adopts it viaversionAdoptServerRollProgress.
Net effect: every hub-triggered restart ends with a spurious in-flight upgrade animation on the new pod for up to 15 minutes. Mirror the restart guard in the announce callback:
spoke.UpgradeAnnounceCallback(func(a *spoke.UpgradeAnnounce) {
// The hub is about to roll (or has rolled) this pod itself
// (#15788): show the upgrade animation before the kill.
// A bare hub restart is re-delivered for a multi-minute
// window; RestartSpokeCallback ignores it inside
// spokeRestartMinUptime, so the announce must too or the
// restarted pod animates a roll that never comes.
if a.Target == "" && time.Since(b.processStartedAt) < spokeRestartMinUptime {
return
}
b.dashSrv.AnnounceUpgrade(*a, time.Now())
}),
(A test alongside TestBootHeartbeatWith_CallbacksReconcileRunningConfig in boot_heartbeat_deps_test.go asserting a targetless hub announce is dropped when processStartedAt is recent would pin it.)
Verified OK: addUrgent prepends (shutdown_hooks.go:68) so announce-restart runs before the websocket drain; broadcastFrame is non-blocking per client (dashboard/server.go:3909-3921) so the 100 ms settle is appropriate; entry.Upgrading/UpgradeTarget at hub/server.go:3278 is the post-reconcile value the existing upgrade-policy path already uses; Restarting outranks the drain in buildUpgradeProgress, which is correct for an exiting process.
— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test |
|
stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test |
There was a problem hiding this comment.
Correctness — delta re-review at 3d3c4db (previous head d231311 → this head is only Merge branch 'v5' carrying #15807/#15827/#15816; no PR-authored changes, no conflict-resolution edits — the delta diff is byte-identical to v5's apart from index.html hunk offsets).
Still present — hub restart_spoke re-delivery leaves a 15‑minute phantom "restart announced" banner on the restarted pod.
src/pkg/hub/spoke_restart.go:114-140—restartSpokeForHeartbeatdeliberately returnstrueon every beat forspokeRestartDeliveryWindow(5 min) after arming.src/pkg/hub/server.go:3269-3283— every such beat also setsresp.UpgradeAnnounceviaheartbeatUpgradeAnnounce(... Restart: true), which atsrc/pkg/hub/heartbeat_upgrade_announce.go:31-33unconditionally returns{Source: hub, Reason: "hub restart"}.src/cmd/hive/main.go:4718-4723—RestartSpokeCallbackguards against acting twice withtime.Since(b.processStartedAt) < spokeRestartMinUptime, but the newUpgradeAnnounceCallbackatsrc/cmd/hive/main.go:5332-5336has no such guard and callsb.dashSrv.AnnounceUpgrade(*a, time.Now())directly.src/pkg/dashboard/upgrade_announce.go:181-204—AnnounceUpgradealways stores a freshannouncedrecord;liveUpgradeAnnounce(:103-114) keeps it live forupgradeAnnounceStaleAfter= 15 min (:37).
Net effect: the freshly restarted pod's first heartbeats (inside the 5‑min window) re-announce the restart that already happened, so its dashboard shows pulling/announced "restart announced by the hub … hub restart" for up to 15 minutes with no kill coming. The SPA reads this from /api/status → upgradeProgress, so a page reload after the roll completes lands on a phantom banner.
Suggested fix — mirror the uptime guard for the restart-flavoured announce (a target-bearing announce is still honoured, since that case is filtered by sameCommit hub-side):
spoke.UpgradeAnnounceCallback(func(a *spoke.UpgradeAnnounce) {
// The hub is about to roll (or has rolled) this pod itself
// (#15788): show the upgrade animation before the kill.
// A bare restart announce riding the 5-minute restart_spoke
// re-delivery window after we already restarted is the roll
// that just finished, not a new one — same guard as
// RestartSpokeCallback above.
if a.Target == "" {
if up := time.Since(b.processStartedAt); up < spokeRestartMinUptime {
return
}
}
b.dashSrv.AnnounceUpgrade(*a, time.Now())
}),
Please add a test in src/cmd/hive (alongside the callback-wiring test) asserting a Target=="" announce is dropped when processStartedAt is recent.
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
stage 080-awaiting-merge-admission → 071-needs-author: conflicting |
5701bc0 to
132d7ef
Compare
|
@clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
stage 080-awaiting-merge-admission → 071-needs-author: conflicting |
…ack) The dashboard's bee/hive upgrade animation only played for self-upgrades: upgradeProgress derived from the drain latch, PVC marker and hub policy, so a hub kubectl roll, an operator rollout restart or a plain SIGTERM showed nothing until the SSE stream dropped and the page reconnected. Two layers, both rendering through the same SPA component: - Intent before the kill: POST /api/upgrade/announce (owner-only) and the heartbeat response's upgrade_announce (sent by the hub while its latch is open onto a build the hub-managed spoke is not running) mark upgradeProgress state=pulling/phase=announced with the target and publish an SSE 'upgrading' event. - Fallback at the kill: the first SIGTERM hook publishes phase=restarting (keeping an announced or hub-armed target) before the contributor drain, so the banner is up before the stream drops. The SPA keeps a roll record across the reconnect, polls /api/version every 2s while the pod is down, shows the new pod's boot phase (startupPhase from cmd/hive's boot sequence) and on reconnect compares build hash and process startedAt: 'Upgraded to <sha> ✓' or 'Restarted ✓' with the 15s toast countdown; the silent hash-change reload waits for that. ?demo=upgrade and Settings → Diagnostics → Preview upgrade animation walk the same state machine on demand. Closes #15788 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
132d7ef to
d261ce8
Compare
|
stage 080-awaiting-merge-admission → 050-review-pending: review-gate:no-review-bundle |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Closes #15788
Why
The operator asked: "when a roll is triggered is there a way to see the upgrading animation?" Today: no.
upgradeProgress(pkg/dashboard/upgrade_progress.go) derives only from self-upgrade inputs — drain latch, PVC marker, outcome, hub policy — and the SPA's bee/hive animation is driven by a localStorage record that onlyselfUpgrade()or an adoptedupgradeMarkerwrites. A hubkubectl rollout restart, an operator roll or a plain SIGTERM shows nothing until the SSE stream drops ("reconnecting...") and the page silently reloads on hash change.What
Two layers, both rendering through the same SPA component (chip, navbar indicator, version card) the self-upgrade uses, with the shared
UpgradeProgressvocabulary (statepulling/rolling + newphaseannounced/restarting,sourcehub/operator/signal):1. Intent before the kill
POST /api/upgrade/announce(owner-only){target|version, source, reason}→upgradeProgress=pulling/announcedwith the target, SSEevent: upgrading. Stale after 15 min without a kill.upgrade_announcewhile a hub-managed hive's latch is open onto a build it isn't running (or onrestart_spoke); spoke wiresUpgradeAnnounceCallback→AnnounceUpgrade.dashboard-standalone-upgrades.md.2. Fallback at the kill
announce-restart,addUrgentahead of the websocket drain) publishesrolling/restartingwith the announced or hub-armed target, 100 ms settle, then the rest of shutdown./api/versionand the initializing/api/statusexposestartedAtandstartupPhase/startupPhaseLabel(everycmd/hiveboot phase +firstEval).SPA (
version.js,sse-bootstrap.js):upgradinglistener → roll record (keyed on build hash + processstartedAt), survives reconnect, 2 s polling while down, showsstarting: opening stores…, then Upgraded to<sha>✓ / Restarted ✓ with the 15 s toast countdown; the hash-change reload waits for it. A self-upgrade already animating keeps its own record. Preview:?demo=upgradeor Settings → Diagnostics → Preview upgrade animation.Tests
upgrade_announce_test.go(403, announce → progress + SSE frame, restart keeps target / falls back to hub target, expiry, startup phase on initializing status and/api/version), hubheartbeat_upgrade_announce_test.go, spoke full-response dispatch,cmd/hiveboot-phase wrapper + callback wiring. No sleeps.roll_upgrade_state.test.js— event → banner → reconnect → upgraded/restarted/aborted, server-side adoption, self-upgrade precedence, demo query.check-api-reference-citations.sh --fixrun; style ratchet kept flat (tokenized card CSS).