Skip to content

🐛 fix: preserve upgrade hold clock during target recovery - #14931

Merged
hivecommons-hive[bot] merged 3 commits into
hivecommons:v5from
thatguywob:fix/14929-upgrade-hold
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
hivecommons:v5from
thatguywob:fix/14929-upgrade-hold

Conversation

@thatguywob

Copy link
Copy Markdown
Contributor

Fixes #14929

Stale recovery now keeps the original upgrade start time when a release-channel promotion advances the target. Repeated promotions therefore retain the original 65-minute silent-drain deadline and stuck-upgrade elapsed time. Newly requested upgrades still use the existing fresh-clock behavior, and recovery still stamps a missing clock.

Adds regression coverage for repeated stable promotions, hold expiry at the original deadline, and lost-clock recovery, plus a changelog fragment.

Validation: gofmt -l reports no changed files and git diff --check passes. Tests were not run locally per AGENTS.md; CI runs the test suite.

— hive: backend=codex


🐝 Hive Agent: contributor | SHA: 23f57c197

Keep stale recovery on the original attempt clock when a channel promotion advances its target. Cover repeated stable promotions, the silent-drain deadline, and lost-clock recovery.

Hive-Run: hivecommons#14929
Hive-Plan: preserve-stale-recovery-clock
Hive-Spec: hivecommons#14929#target-advancement-hold
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the hold-clock fix does what #14929 asks, but routing an advanced target around beginUpgrade's re-stamp also drops the upgrade mark the crash-loop rule depends on (#11824). One medium finding; the fix is one line.

Finding — promoted-target restart is now counted as a crash (medium)

  • src/pkg/hub/saas_upgrade.go:1258-1259 — pre-setting UpgradeTarget = recoverTarget makes beginUpgrade's condition at saas_upgrade.go:63 false, so neither the re-stamp nor s.alerts.noteUpgrade(...) at saas_upgrade.go:70 runs. On v5 the advanced-target path recorded a fresh mark here.
  • The only other mark source is alerts.go:1062-1064, which notes h.UpgradeStartedAt — i.e. the preserved original start.
  • upgradeAttributable (alerts.go:368-371) attributes a spoke start only inside [mark-2m, mark+alertUpgradeStartGrace], and alertUpgradeStartGrace = 10m (alerts.go:163) equals staleUpgradeTimeout = 10m (server.go:127). Recovery can only fire once the upgrade is ≥10m old, and the spoke restarts on its next heartbeat after heartbeatUpgrade is armed — so the restart the promotion causes always lands past the grace window of the only remaining mark.
  • Consequence: each promotion-driven recovery restart (plus the mutable-tag follow-up roll the grace comment describes) is counted by observeStart as an unattributed restart. Two of those inside alertCrashLoopWindow (2h) plus one ordinary start reaches alertCrashLoopMinRestarts = 3 and raises a spurious critical crash-loop alert on a hive that is simply being promoted. Previously this restart was excused.

Suggested fix: keep the clock, but still record the rollout — e.g. in the recoverTarget != upgradeTarget branch, call s.alerts.noteUpgrade(h.ID, time.Now()) alongside the pre-set, and assert in TestTriggerAutoUpgradesPromotionPreservesHoldDeadline that a start observed shortly after a promotion is not counted (observeStart stays below alertCrashLoopMinRestarts). noteUpgrade is nil-safe, so the existing test setup with no alerts still passes.

What I verified as correct

  • evaluateOrphanedUpgrade reads UpgradeStartedAt, so preserving it does hold the original 65m deadline as the new test asserts.
  • The zero-clock self-heal survives (saas_upgrade.go:63 still stamps on IsZero()), matching the last block of the new test.
  • begin_upgrade_test.go:62-79 ("new target gets a fresh clock") is unaffected because it calls beginUpgrade directly; the bypass is confined to the recovery loop and the comment says so.

Body claims match the diff; scope matches #14929. I have not run the suite here.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conclusion: the clock-preservation itself does what #14929 asks, but adopting the target before beginUpgrade has a side effect the test cannot see: an advanced target no longer records an upgrade mark for the crash-loop rule, so the spoke's restart onto the promoted image is counted as a crash.

Mechanism (read at merge-base and at 23f57c1):

  • src/pkg/hub/saas_upgrade.go:63-70 — beginUpgrade only calls s.alerts.noteUpgrade(h.ID, h.UpgradeStartedAt) inside the "genuinely new upgrade" branch (!h.Upgrading || h.UpgradeTarget != target || UpgradeStartedAt.IsZero()). This PR sets s.registry.Hives[i].UpgradeTarget = recoverTarget first (saas_upgrade.go:1260), so for a target advance that branch is now skipped entirely — no fresh clock and no new mark.
  • src/pkg/hub/alerts.go:368-375 — upgradeAttributable excuses a spoke start only if it falls within alertUpgradeStartGrace (10m, alerts.go:163) of a recorded mark. The only mark left is the original UpgradeStartedAt.
  • Stale recovery runs only once upgradeAge >= staleUpgradeTimeout (10m, server.go:127). So by construction the original mark is already ≥10m old when the promotion is delivered, and the restart the spoke performs to pull the new SHA cannot be attributed to an upgrade. observeStart (alerts.go:427-431) counts it; three such promotion restarts inside alertCrashLoopWindow (2h) with uptime under the ceiling fire a critical crash-loop alert (alerts.go:1066-1071) for a hive that was only following :stable.

This is exactly the case the base comment at saas_upgrade.go:1257-1259 ("a genuinely advanced target … DOES get a fresh clock") was protecting alongside #11824; the PR's rewritten comment drops that and the new test can't detect it because s.alerts is nil in the test server (noteUpgrade is nil-safe).

Severity: medium — false critical alert, no data impact; it needs ≥3 promotions during one stuck upgrade, which is plausible on a busy :stable channel but not common.

Suggested fix: keep the clock but still mark the delivery, e.g. in the recovery loop after s.beginUpgrade(i, recoverTarget):

if recoverTarget != upgradeTarget {
    s.alerts.noteUpgrade(h.ID, time.Now())
}

(noteUpgrade keeps a list of marks, so this does not disturb the original one.) A test with a non-nil alertState asserting observeStart returns 0 for a start a minute after the promotion would lock it in.

Everything else checks out: the lost-clock stamp still happens via the IsZero() arm, heartbeatUpgrade is armed with the advanced target, the same-target re-arm path is unchanged, and the silent-drain assertions in TestTriggerAutoUpgradesPromotionPreservesHoldDeadline match evaluateOrphanedUpgrade's use of UpgradeStartedAt.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the review swarm requested changes

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the change against src/pkg/hub/saas_upgrade.go on v5; the mechanism is sound.

beginUpgrade (saas_upgrade.go:58-73) re-stamps UpgradeStartedAt only when !h.Upgrading || h.UpgradeTarget != target || h.UpgradeStartedAt.IsZero(). Writing UpgradeTarget = recoverTarget first (saas_upgrade.go:1261 in this PR) makes the second clause false, so a channel promotion in the stale-recovery block (:1195-1276, which only runs with alreadyUpgrading) keeps the original clock while a zero clock is still stamped. The explicit s.alerts.noteUpgrade(h.ID, time.Now()) for an advanced target preserves the #11824 crash-loop exemption that beginUpgrade would otherwise skip. Other beginUpgrade callers are unaffected: they go through the unchanged helper and still get a fresh clock on a new target.

TestTriggerAutoUpgradesPromotionPreservesHoldDeadline asserts UpgradeStartedAt.Equal(started) after promotion, which the pre-change code (fresh clock on recoverTarget != old target) would fail, so the test does guard the fix.

One note, not a defect: when the clock is zero and the target advances, noteUpgrade is called twice (inside beginUpgrade and again at :1266) with two time.Now() values a few ns apart. noteUpgrade (alerts.go:333) dedupes only on Equal, so two marks are kept; both fall in the same grace window and the list is capped, so the effect is nil.

go build ./pkg/hub/ passes; I could not run the suite here (hive#9845), CI was still pending at the time of this read. Looks correct to me.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Quality review at 2c7340f (fork head — comment only)

  • TestTriggerAutoUpgradesPromotionPreservesHoldDeadline does fail without the fix: beginUpgrade re-stamps UpgradeStartedAt whenever h.UpgradeTarget != target, so the pre-fix recovery path (which called beginUpgrade before adopting the promoted target) resets started on the first promotion, and the promotion reset the original start assertion trips. Adopting the target first is what keeps the clock, and the test pins exactly that.
  • The second commit's s.alerts.noteUpgrade(h.ID, time.Now()) is covered by the promotion restart must not count as a crash assertion; without it a promoted-target restart is counted as a crash because beginUpgrade no longer notes the rollout (it only notes on a fresh clock).
  • Silent-drain assertions use a fixed started offset and evaluateOrphanedUpgrade at started ± silentDrainClearAfter, so they are deterministic; no time.Sleep added (sleep ratchet unaffected).
  • pkg/hub and its test files type-check against v5 locally; go test is not runnable in this pod, so CI (test (hub */4), in progress at time of writing) is the authoritative run.

No changes requested from the quality side.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the recovery branch and its callee at v5, not just the diff. This looks correct to me.

Mechanism. beginUpgrade (src/pkg/hub/saas_upgrade.go:63) re-stamps UpgradeStartedAt when !h.Upgrading || h.UpgradeTarget != target || h.UpgradeStartedAt.IsZero(). Writing UpgradeTarget = recoverTarget on the entry before the call (saas_upgrade.go:1260) makes the second clause false on a promotion, so only a lost/zero clock is still re-stamped — the ibm-alchemy self-heal is preserved. noteUpgrade(h.ID, time.Now()) at :1265 replaces the mark beginUpgrade used to record on the target-change path, so observeStart (alerts.go) still excuses the promoted rollout's restart; noteUpgrade dedupes by exact time and ignores zero, so calling it outside beginUpgrade is safe.

Consumers checked. evaluateOrphanedUpgrade (stale_upgrade.go:149-167) and the stuck-upgrade check (alerts.go:1113-1114) both key off UpgradeStartedAt; neither depends on the clock being reset when the target moves, and converged compares GitHash against the current UpgradeTarget, which this PR advances. No other non-test writer of UpgradeStartedAt is affected.

Test. TestTriggerAutoUpgradesPromotionPreservesHoldDeadline fails on v5 (the entry.UpgradeStartedAt.Equal(started) assertion) and passes with the change; the zero-clock tail case guards the self-heal. All helpers it uses already exist in the package.

Minor, non-blocking: after a promotion the entry is still past staleUpgradeTimeout, so every subsequent triggerAutoUpgrades sweep re-enters the stale branch and logs the re-arming heartbeat fallback Warn until the spoke collects — that is already how same-target re-arms behave today, so this PR does not introduce it, just extends it to the promoted case.

Closes #14929 as stated; scope matches the issue.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 095-pre-merge-soak → 060-reviewed: required-check-pending:Sign-off survives the squash

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • overlayfs-exec-guard

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
cd3a7b4

⚠️ Sentinel alert — maintainer review required

Hive flagged this PR (author @thatguywob, head cd3a7b42d9fe) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • src/Dockerfile

Hive added the sentinel-alert label. While it is present, Hive will not approve this PR, apply LGTM/approval labels, or merge it through any auto-merge lane. Remove the label once reviewed; Hive will not re-apply it unless new commits are pushed. Tune paths and behaviors under sentinel in hive.yaml or the dashboard Security tab.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge: the automerge lane will not merge this unattended.

Head: verdict approved @cd3a7b4, 58/60 checks green, 2 pending, DCO yes, soak 0s of 30m0s.

How to release it:

  • A maintainer must resolve the lane condition named above or review and merge manually; /hive approve does not override that condition.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 095-pre-merge-soak → 085-needs-decision: contributor-lane:sentinel-blocked

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

  • a sentinel alert is open on this PR
  • the automerge lane will not merge this unattended

Head: verdict approved @cd3a7b4, 58/60 checks green, 2 pending, DCO yes, sentinel alert, soak 0s of 30m0s.

How to release it:

  • Resolve the sentinel alert and have a maintainer clear its label.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

hive recommends: review the sentinel finding on #14931 and remove the sentinel label if intended; owner to confirm

Rationale: a sentinel alert on a PR from an author without write access is a human review. stage/085-needs-decision stays.

Decided by the hive at @clubanderson's request as the owner's delegate under the owner-delegate policy (rule sentinel-untrusted).

@hivecommons-hive

Copy link
Copy Markdown
Contributor

/lgtm

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@hivecommons-hive[bot]: changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

/approve

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hivecommons-hive[bot]
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Oct 28, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 583fe8d into hivecommons:v5 Oct 28, 2026
87 of 95 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#14931cd3a7b4` sha256:705a4d2ecef03b1faffaa0e4be65723a2bfd1972fdda7aa7e444d708709697d0 — download from the dashboard or `hivectl review evidence #14931`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @thatguywob holds the trusted tier in this hive's contributor registry.

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/095-pre-merge-soak Lane admitted; the pre-merge soak window is running before the merge is attempted labels Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review backlog (correctness): Target advancement during hold resets UpgradeStartedAt and restarts the 65m window

1 participant