Skip to content

✨ dashboards: upgrade progress pill, history and Upgrade now (interrupt turns) - #14968

Merged
hivecommons-hive[bot] merged 3 commits into
v5from
feat/upgrade-visibility-ui
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
v5from
feat/upgrade-visibility-ui

Conversation

@clubanderson

@clubanderson clubanderson commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Refs #14950 — PR B of the scheduled-upgrade visibility feature. Motivation: #14856 (FMA hive silently missed two windows) and #14925 (drain latch).

Builds on #14953 (PR A — data model, history, payloads), now merged; this PR is rebased on v5.

What this adds

Hub hive card

  • Pill for draining / pulling / rolling / verifying / missed with the drain countdown and the agents holding the upgrade — Draining · 2/5 agents in turn: reviewer 38m, scanner 12m · 14m left, Missed window · drain lapsed. A quiet scheduled row renders exactly as before.
  • history badge; tooltip lists the hub-persisted attempts (red + miss count when any window was missed).
  • Upgrade now (interrupt turns) while draining or missed. Same POST /api/saas/hives/{id}/upgrade with ?interrupt=1, same owner/admin check, hiveConfirm (no native dialog), audit-logged (interrupt_turns=true) and on the hive timeline.

Hub → spoke: interrupt held per hive in memory (2 h TTL, cleared when the upgrade closes, dropped on hub restart → spoke falls back to draining); delivered beside upgrade_to as upgrade_interrupt on the heartbeat; the spoke's drain decision reads it and proceeds at once.

Spoke operator dashboard: /api/version now carries upgradeProgress + upgradeHistory; the version menu gains a Progress row and an Upgrade history list; the header chip shows Draining 14m / Missed window. .oc-version-* classes, no inline styles.

Docs: operator-reference "Scheduled-upgrade progress and history" — where it shows, and the interrupt action. Changelog added-upgrade-progress-ui.md.

Tests

  • upgradeInterrupts arm/pending/TTL/clear, appliesTo target matching, ?interrupt= parsing.
  • Heartbeat carries upgrade_interrupt only alongside upgrade_to, and only for the armed hive.
  • Spoke dispatch exposes the flag to the upgrade callback.
  • /api/version payload fields; markup tests for both dashboards (labels for every state, hiveConfirm, no inline styles in the history list).
  • No time.Sleep added. go test ./pkg/hub/ ./pkg/hub/spoke/ ./pkg/dashboard/ ./cmd/hive/ green locally.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14971 🐛 fix: claim release re-derives covered-by-pr from the claim ledger; close-on-merge on the webhook; ledger backfill same files src/cmd/hive/main.go
#14821 🐛 fix: preserve supersession decisions across stage reconciliation same files src/cmd/hive/main.go
src/pkg/dashboard/static/index.html
#14807 ✨ dashboard: stream Job pod logs in the terminal pane for sandbox.runtime: job agents same files src/docs/api-reference.md
src/pkg/dashboard/static/index.html
#14524 ✨ feat(hub,agent): Hub-coordinated live CLI session pool for multi-tenant runner capacity same files src/cmd/hive/main.go
src/docs/api-reference.md
src/pkg/hub/server.go
#14965 🐛 fix(dashboard): name the blocking PR on tracker rows and keep webhook unblocks from being overwritten same files src/cmd/hive/main.go
src/pkg/dashboard/static/index.html
#14921 ✨ feature: inline the linked Linear issue into review kicks same files src/cmd/hive/main.go
#14931 🐛 fix: preserve upgrade hold clock during target recovery same files src/pkg/hub/saas_upgrade.go

@clubanderson
clubanderson force-pushed the feat/upgrade-visibility-ui branch from 9785a94 to 783df80 Compare October 28, 2026 14:32
@kubestellar-prow kubestellar-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

clubanderson added a commit that referenced this pull request Oct 28, 2026
Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added the role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • changelog-fragment-guard

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 28, 2026
@hivecommons-hive

hivecommons-hive Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

  • Claimed issue: hive#14950
  • Closing PR: #14953

Decision needed (@hivecommons): #14953 already merged and closed the claimed issue, but this PR touches files the merged PR did not. Recommended: keep open until the extra changes are reviewed, or close as superseded if they are not wanted. Another option: merge anyway.

Files the merged PR did not touch: changelog.d/added-upgrade-progress-ui.md, src/pkg/dashboard/api_version.go, src/pkg/dashboard/sidebar_version_panel_test.go, src/pkg/dashboard/static/index.html, src/pkg/dashboard/upgrade_progress_ui_test.go and 7 more.

This comment is edited in place by the supersession sweep; it is not duplicated.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the second commit (783df80) against the tree. The interrupt path reads correctly: upgradeInterrupts is a leaf store with TTL eviction in pending, the flag is attached in both UpgradeTo branches that can reach a draining spoke (server.go:3234, :3245) and withheld on the pause/digest-pin branches, it is cleared when the latch closes (timeline.go:550) or on a plain re-arm (saas_upgrade.go:623), and the spoke resets the atomic on every heartbeat so a flag never outlives the instruction that carried it. cmd/hive/main.go only zeroes drainTimeout when there are in-flight turns, so a quiet spoke is unaffected.

Since this is stacked on #14953, it inherits the one defect I found there: pkg/hub/upgrade_progress.go:61 checks UpgradeFailed before Upgrading, and beginUpgrade does not clear UpgradeFailed, so a retry after a failed upgrade renders missed: <old error> through draining/pulling/rolling — which is exactly the moment the new Upgrade now (interrupt turns) button is gated to appear (draining or missed); the button will show for the wrong reason on that path. Fixing it in PR A fixes it here on rebase.

No other verified issues in the UI commit. The red changelog-fragment-guard is already flagged to you via the stage label; I did not dig into it.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the PR-B commit (783df80) plus the fragment commit on top, against the stacked #14953 base. The interrupt mechanism reads correctly: arm/clear under a leaf mutex, 2 h TTL in pending, the flag only rides when resp.UpgradeTo is set (server.go:3234, :3245), and the spoke stores it immediately before onUpgrade (spoke/heartbeat.go:1416) so a later instruction without it resets the atomic before the drain decision runs. handleUpgradeHive keeps the existing owner/admin check in front of ?interrupt=1.

Two concrete things:

1. changelog-fragment-guard is red because this PR now carries two fragments. 783df80 added changelog.d/added-upgrade-progress-ui.md and 73b496d added changelog.d/added-14968.md; changelog.d/README.md:1 is "one changelog fragment per PR". Drop one (the added-14968.md one already carries the (#14968) suffix the compiler wants).

2. Inherits the open-row gap I flagged on #14953, and it reaches the interrupt too. upgradeInterrupts.clear is called only from recordUpgradeHistoryTransition (timeline.go:550) on the old.Upgrading && !new.Upgrading edge. The six clearUpgradeLatch sites that mutate the registry outside that diff (server.go:3012, :3097, saas_upgrade.go:1113/1140/1191, digest_pin.go:344) never fire it, so an interrupt armed for an upgrade that lands through one of those paths stays pending for up to 2 h and rides along with the next UpgradeTo — appliesTo (upgrade_interrupt.go) matches when the new target differs only if r.Target is empty, so it is mostly bounded by the SHA check, but a re-arm to the same target within 2 h would cut turns the owner did not ask to cut. Low severity given the TTL + target match; fixing the gap at the clear sites in #14953 resolves it here for free.

I did not dig into the test (rest 3/4) / coverage failures — those may just be the merge-base skew of a stacked PR; worth a look once rebased onto v5.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@kubestellar-prow kubestellar-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

…pt turns)

Renders the scheduled-upgrade state machine from the previous change on
both surfaces and adds the owner action that resolves a stalled drain.

Hub hive card (dashboard.html):
- A pill for draining/pulling/rolling/verifying/missed with the drain
  countdown and the agents holding the upgrade ("Draining · 2/5 agents
  in turn: reviewer 38m, scanner 12m · 14m left"); a quiet scheduled row
  renders exactly as before.
- A "history" badge whose tooltip lists the hub's recent attempts; red
  with a miss count when any window was missed.
- "Upgrade now (interrupt turns)" while draining or missed. Same POST as
  Upgrade now with ?interrupt=1, same owner/admin check, hiveConfirm
  (no native dialog), audit-logged with interrupt_turns=true and
  recorded on the hive timeline.

Hub → spoke: the interrupt is held in memory per hive (2h TTL, cleared
when the upgrade closes, dropped on hub restart so the spoke falls back
to draining) and rides on the heartbeat beside upgrade_to as
upgrade_interrupt. The spoke's drain decision reads it and proceeds
immediately instead of waiting out the drain window.

Spoke operator dashboard: /api/version carries upgradeProgress and
upgradeHistory; the version menu gains a Progress row with the pill and
an Upgrade history list, and the header chip shows Draining/Missed
window while either state holds. Styled with .oc-version-* classes.

Tests: interrupt store arm/pending/TTL/clear and target matching,
request parsing, heartbeat response carries the flag only with
upgrade_to, spoke dispatch exposes it to the upgrade callback,
/api/version payload fields, and markup tests for both dashboards.

Refs #14950

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…nt bullet

The style ratchet counts inline style= attributes and raw px/rem values;
the new pill, history badge and interrupt action now use .upgrade-progress-*
(hub) and .oc-version-* (spoke) classes built on tokens.css, and the
progress line shares the existing stacked Version line. Operator raw-colour
baseline lowered to the measured 135. Changelog fragment starts with the
required '- ' bullet.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Keeps the Version cell's one-element-per-line assembly intact for
TestVersionCellPutsEachElementOnItsOwnLine.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the feat/upgrade-visibility-ui branch from 9078b25 to 814639d Compare October 28, 2026 15:07
@kubestellar-prow kubestellar-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scoped read: I traced the Upgrade now (interrupt turns) path end to end, since it is the only part of this PR that changes what the spoke does to running agents. I did not read the pill/history rendering in dashboard.html / index.html beyond the interrupt action.

What I verified (at 814639d)

  • Authorization: the interrupt is a flag on the existing owner/admin-gated handleUpgradeHive (src/pkg/hub/saas_upgrade.go:617-623); no new endpoint, no new trust surface. It is audit-logged with interrupt_turns and recorded on the timeline (:627-634).
  • Delivery: handleHeartbeat only sets UpgradeInterrupt when an unexpired request appliesTo the target actually being instructed (src/pkg/hub/server.go:3234, :3245), so a stale request cannot ride a different SHA.
  • Lifecycle: cleared when the Upgrading latch drops (src/pkg/hub/timeline.go:550), on a later non-interrupt request (saas_upgrade.go:623), or after the 2h TTL (upgrade_interrupt.go:14, :55-58). Hub restart drops it — safe direction (spoke drains). operator-reference.md's description of this matches the code.
  • Consumption: spoke.setUpgradeInterrupt is written on every heartbeat that carries UpgradeTo (spoke/heartbeat.go:1416), so the atomic reflects the latest instruction; src/cmd/hive/main.go:5181 zeroes drainTimeout only when the flag is set and turns are in flight, then goes through the normal selfUpgradeDrain.decide. Looks correct to me.

One nit (info): the comment on upgradeInterruptRequested (src/pkg/hub/upgrade_interrupt.go:87-88) says it accepts a JSON body {"interrupt":true}; the function only reads the ?interrupt query parameter. The dashboard sends ?interrupt=1 so nothing is broken — either drop the body clause from the comment or implement it.

The PR currently conflicts with v5 and will need a rebase before CI can judge it.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • overlayfs-exec-guard

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive
hivecommons-hive Bot merged commit cffbb05 into v5 Oct 28, 2026
83 of 89 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#14968814639d` sha256:9780c8ac157bf43575b8beaf1e4d7b967a9450948f2085b2fb5b6167db886f4e — download from the dashboard or `hivectl review evidence #14968`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 814639d.

@kubestellar-prow
kubestellar-prow Bot deleted the feat/upgrade-visibility-ui branch October 28, 2026 15:21
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant