Skip to content

✨ Show the upgrade animation for every roll (announce + SIGTERM fallback) - #15804

Merged
clubanderson merged 1 commit into
v5from
feat/roll-upgrade-animation
Oct 30, 2026
Merged

clubanderson merged 1 commit into
v5from
feat/roll-upgrade-animation

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Closes #15788

Why

The operator asked: "when a roll is triggered is there a way to see the upgrading animation?" Today: no. upgradeProgress (pkg/dashboard/upgrade_progress.go) derives only from self-upgrade inputs — drain latch, PVC marker, outcome, hub policy — and the SPA's bee/hive animation is driven by a localStorage record that only selfUpgrade() or an adopted upgradeMarker writes. A hub kubectl rollout restart, an operator roll or a plain SIGTERM shows nothing until the SSE stream drops ("reconnecting...") and the page silently reloads on hash change.

What

Two layers, both rendering through the same SPA component (chip, navbar indicator, version card) the self-upgrade uses, with the shared UpgradeProgress vocabulary (state pulling/rolling + new phase announced/restarting, source hub/operator/signal):

1. Intent before the kill

  • POST /api/upgrade/announce (owner-only) {target|version, source, reason} → upgradeProgress = pulling/announced with the target, SSE event: upgrading. Stale after 15 min without a kill.
  • Hub: heartbeat response carries upgrade_announce while a hub-managed hive's latch is open onto a build it isn't running (or on restart_spoke); spoke wires UpgradeAnnounceCallback → AnnounceUpgrade.
  • Operator one-liner documented in dashboard-standalone-upgrades.md.

2. Fallback at the kill

  • First SIGTERM hook (announce-restart, addUrgent ahead of the websocket drain) publishes rolling/restarting with the announced or hub-armed target, 100 ms settle, then the rest of shutdown.
  • /api/version and the initializing /api/status expose startedAt and startupPhase/startupPhaseLabel (every cmd/hive boot phase + firstEval).

SPA (version.js, sse-bootstrap.js): upgrading listener → roll record (keyed on build hash + process startedAt), survives reconnect, 2 s polling while down, shows starting: opening stores…, then Upgraded to <sha> ✓ / Restarted ✓ with the 15 s toast countdown; the hash-change reload waits for it. A self-upgrade already animating keeps its own record. Preview: ?demo=upgrade or Settings → Diagnostics → Preview upgrade animation.

Tests

  • Go: upgrade_announce_test.go (403, announce → progress + SSE frame, restart keeps target / falls back to hub target, expiry, startup phase on initializing status and /api/version), hub heartbeat_upgrade_announce_test.go, spoke full-response dispatch, cmd/hive boot-phase wrapper + callback wiring. No sleeps.
  • Node: roll_upgrade_state.test.js — event → banner → reconnect → upgraded/restarted/aborted, server-side adoption, self-upgrade precedence, demo query.
  • OpenAPI + api-reference rows added; check-api-reference-citations.sh --fix run; style ratchet kept flat (tokenized card CSS).

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 30, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15865 ✨ feature: carry abandoned counts through the hub leaderboard same files src/cmd/hive/main.go
src/pkg/hub/heartbeat.go
src/pkg/hub/server.go
src/pkg/hub/spoke/heartbeat.go
#15877 ✨ feature: show abandoned counts on contributor cards and status CONFLICT src/docs/api-reference/dashboard-misc.md
#15871 🐛 fix: stage tracker gives immediate Let-hive-decide feedback and says why a human is wanted same files src/pkg/dashboard/static/index.html
#15885 ✨ Add integration-aware orchestration above preparation and native queues same files src/cmd/hive/main.go
src/pkg/dashboard/api.go
src/pkg/dashboard/server.go
src/pkg/dashboard/static/index.html
#15880 🐛 dashboard: merge banners drain the toast countdown bar same files src/pkg/dashboard/static/index.html
#15853 ✨ Complete native GitHub queue CI and policy integration same files src/pkg/dashboard/static/index.html
#15748 🐛 [reviewer] park PRs at 085 Needs decision when the fixer exhausted its attempts same files src/cmd/hive/main.go
#15842 ✨ Add Apple MLX local inference and first-class ClankeR onboarding same files src/pkg/hub/server.go
#15849 ✨ Show online ClankeR IDs on the Contributors count pill CONFLICT src/docs/api-reference/dashboard-misc.md
#15854 🐛 fix: preserve merge notifications across reloads and show countdown same files src/pkg/dashboard/static/index.html
#15859 ✨ feature: add abandoned counts to contributor leaderboards same files src/pkg/dashboard/static/index.html

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 30, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style review (head 9dfbfd4) — approve; two advisory notes, worst first.

  1. src/pkg/dashboard/upgrade_announce.go:307 — optional-body decode compares the error string (err.Error() != "EOF"). The package idiom for an optional JSON body is errors.Is(err, io.EOF) (see pkg/dashboard/bob_key_probe.go:224); string-matching an error is brittle and is the only such comparison in pkg/.
		if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&req); err != nil && !errors.Is(err, io.EOF) {

(add "errors" and "io" to the imports.)

  1. src/pkg/dashboard/upgrade_announce.go:362-383 — startupPhaseLabels repeats every cmd/hive boot-phase name as a string literal (boot_sequence.go:36-54), and nothing checks the two tables agree: a renamed or added phase silently falls back to the raw bootXxx identifier in the banner. TestDefaultBootSequenceOrder already imports dashboard at this head, so pinning the coupling is one loop:
--- a/src/cmd/hive/boot_sequence_test.go
+++ b/src/cmd/hive/boot_sequence_test.go
@@ -43,4 +43,9 @@ func TestDefaultBootSequenceOrder(t *testing.T) {
 	if got := defaultBootSequenceNames(); strings.Join(got, ",") != strings.Join(want, ",") {
 		t.Fatalf("names = %v, want %v", got, want)
 	}
+	for _, name := range append(defaultBootSequenceNames(), bootPhaseFirstEval) {
+		if dashboard.StartupPhaseLabel(name) == name {
+			t.Errorf("boot phase %q has no label in dashboard.startupPhaseLabels (#15788)", name)
+		}
+	}
 }

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 9dfbfd4)

Hub restart_spoke leaves a 15‑minute phantom "restart announced" banner on the freshly restarted pod

heartbeatUpgradeAnnounce returns a hub announce (Target:"", reason "hub restart") whenever the response carries RestartSpoke (src/pkg/hub/heartbeat_upgrade_announce.go:31-33). The hub re-delivers RestartSpoke on every beat for a multi-minute window — that is exactly why spokeRestartMinUptime exists (src/cmd/hive/main.go:464-467) and why RestartSpokeCallback ignores the instruction when uptime < 10 min (main.go:4719-4723). But the new UpgradeAnnounceCallback has no such guard (main.go:5332-5336): the restarted pod receives the same beats, calls AnnounceUpgrade, and

  • /api/status//api/version report pulling/announced ("restart announced by the hub … ago") for upgradeAnnounceStaleAfter = 15 min (src/pkg/dashboard/upgrade_announce.go:36, :110-114); repeated beats keep the original At via sameAnnounceTarget("","") (:192-194), so it does not reset but does persist the full window;
  • each beat also broadcasts an SSE upgrading frame (:196), which versionRollRecordFromEvent turns into a new roll record because the previous one is completedAt (static-src/js/version.js diff: if (prev && prev.source !== 'roll' && !prev.completedAt) return prev;), so right after "Restarted ✓" the bee animation starts again and a freshly opened tab adopts it via versionAdoptServerRollProgress.

Net effect: every hub-triggered restart ends with a spurious in-flight upgrade animation on the new pod for up to 15 minutes. Mirror the restart guard in the announce callback:

			spoke.UpgradeAnnounceCallback(func(a *spoke.UpgradeAnnounce) {
				// The hub is about to roll (or has rolled) this pod itself
				// (#15788): show the upgrade animation before the kill.
				// A bare hub restart is re-delivered for a multi-minute
				// window; RestartSpokeCallback ignores it inside
				// spokeRestartMinUptime, so the announce must too or the
				// restarted pod animates a roll that never comes.
				if a.Target == "" && time.Since(b.processStartedAt) < spokeRestartMinUptime {
					return
				}
				b.dashSrv.AnnounceUpgrade(*a, time.Now())
			}),

(A test alongside TestBootHeartbeatWith_CallbacksReconcileRunningConfig in boot_heartbeat_deps_test.go asserting a targetless hub announce is dropped when processStartedAt is recent would pin it.)

Verified OK: addUrgent prepends (shutdown_hooks.go:68) so announce-restart runs before the websocket drain; broadcastFrame is non-blocking per client (dashboard/server.go:3909-3921) so the 100 ms settle is appropriate; entry.Upgrading/UpgradeTarget at hub/server.go:3278 is the post-reconcile value the existing upgrade-policy path already uses; Restarting outranks the drain in buildUpgradeProgress, which is correct for an exiting process.

— hive: agent=adjudicator-4 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 050-review-pending: required-check-pending:build-and-test

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=adjudicator-5 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness — delta re-review at 3d3c4db (previous head d231311 → this head is only Merge branch 'v5' carrying #15807/#15827/#15816; no PR-authored changes, no conflict-resolution edits — the delta diff is byte-identical to v5's apart from index.html hunk offsets).

Still present — hub restart_spoke re-delivery leaves a 15‑minute phantom "restart announced" banner on the restarted pod.

  • src/pkg/hub/spoke_restart.go:114-140 — restartSpokeForHeartbeat deliberately returns true on every beat for spokeRestartDeliveryWindow (5 min) after arming.
  • src/pkg/hub/server.go:3269-3283 — every such beat also sets resp.UpgradeAnnounce via heartbeatUpgradeAnnounce(... Restart: true), which at src/pkg/hub/heartbeat_upgrade_announce.go:31-33 unconditionally returns {Source: hub, Reason: "hub restart"}.
  • src/cmd/hive/main.go:4718-4723 — RestartSpokeCallback guards against acting twice with time.Since(b.processStartedAt) < spokeRestartMinUptime, but the new UpgradeAnnounceCallback at src/cmd/hive/main.go:5332-5336 has no such guard and calls b.dashSrv.AnnounceUpgrade(*a, time.Now()) directly.
  • src/pkg/dashboard/upgrade_announce.go:181-204 — AnnounceUpgrade always stores a fresh announced record; liveUpgradeAnnounce (:103-114) keeps it live for upgradeAnnounceStaleAfter = 15 min (:37).

Net effect: the freshly restarted pod's first heartbeats (inside the 5‑min window) re-announce the restart that already happened, so its dashboard shows pulling/announced "restart announced by the hub … hub restart" for up to 15 minutes with no kill coming. The SPA reads this from /api/status → upgradeProgress, so a page reload after the roll completes lands on a phantom banner.

Suggested fix — mirror the uptime guard for the restart-flavoured announce (a target-bearing announce is still honoured, since that case is filtered by sameCommit hub-side):

			spoke.UpgradeAnnounceCallback(func(a *spoke.UpgradeAnnounce) {
				// The hub is about to roll (or has rolled) this pod itself
				// (#15788): show the upgrade animation before the kill.
				// A bare restart announce riding the 5-minute restart_spoke
				// re-delivery window after we already restarted is the roll
				// that just finished, not a new one — same guard as
				// RestartSpokeCallback above.
				if a.Target == "" {
					if up := time.Since(b.processStartedAt); up < spokeRestartMinUptime {
						return
					}
				}
				b.dashSrv.AnnounceUpgrade(*a, time.Now())
			}),

Please add a test in src/cmd/hive (alongside the callback-wiring test) asserting a Target=="" announce is dropped when processStartedAt is recent.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 071-needs-author: conflicting

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 071-needs-author: conflicting

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 30, 2026
…ack)

The dashboard's bee/hive upgrade animation only played for self-upgrades:
upgradeProgress derived from the drain latch, PVC marker and hub policy,
so a hub kubectl roll, an operator rollout restart or a plain SIGTERM
showed nothing until the SSE stream dropped and the page reconnected.

Two layers, both rendering through the same SPA component:

- Intent before the kill: POST /api/upgrade/announce (owner-only) and
  the heartbeat response's upgrade_announce (sent by the hub while its
  latch is open onto a build the hub-managed spoke is not running) mark
  upgradeProgress state=pulling/phase=announced with the target and
  publish an SSE 'upgrading' event.
- Fallback at the kill: the first SIGTERM hook publishes
  phase=restarting (keeping an announced or hub-armed target) before the
  contributor drain, so the banner is up before the stream drops.

The SPA keeps a roll record across the reconnect, polls /api/version
every 2s while the pod is down, shows the new pod's boot phase
(startupPhase from cmd/hive's boot sequence) and on reconnect compares
build hash and process startedAt: 'Upgraded to <sha> ✓' or 'Restarted ✓'
with the 15s toast countdown; the silent hash-change reload waits for
that. ?demo=upgrade and Settings → Diagnostics → Preview upgrade
animation walk the same state machine on demand.

Closes #15788

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/roll-upgrade-animation branch from 132d7ef to d261ce8 Compare October 30, 2026 19:56
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 30, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission → 050-review-pending: review-gate:no-review-bundle

@clubanderson
clubanderson merged commit d442bab into v5 Oct 30, 2026
78 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/roll-upgrade-animation branch October 30, 2026 21:06
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

conflicts-with-open-pr Open PR has a merge conflict with another open PR dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

✨ Show the upgrade animation for every roll (hub roll, kubectl rollout restart, SIGTERM) — not only self-upgrades

1 participant