Repository navigation
🐛 fix: preserve upgrade hold clock during target recovery - #14931
Conversation
Keep stale recovery on the original attempt clock when a channel promotion advances its target. Cover repeated stable promotions, the silent-drain deadline, and lost-clock recovery. Hive-Run: hivecommons#14929 Hive-Plan: preserve-stale-recovery-clock Hive-Spec: hivecommons#14929#target-advancement-hold Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of No conflicting open PRs or same-file overlaps were found. |
There was a problem hiding this comment.
Conclusion: the hold-clock fix does what #14929 asks, but routing an advanced target around beginUpgrade's re-stamp also drops the upgrade mark the crash-loop rule depends on (#11824). One medium finding; the fix is one line.
Finding — promoted-target restart is now counted as a crash (medium)
src/pkg/hub/saas_upgrade.go:1258-1259— pre-settingUpgradeTarget = recoverTargetmakesbeginUpgrade's condition atsaas_upgrade.go:63false, so neither the re-stamp nors.alerts.noteUpgrade(...)atsaas_upgrade.go:70runs. Onv5the advanced-target path recorded a fresh mark here.- The only other mark source is
alerts.go:1062-1064, which notesh.UpgradeStartedAt— i.e. the preserved original start. upgradeAttributable(alerts.go:368-371) attributes a spoke start only inside[mark-2m, mark+alertUpgradeStartGrace], andalertUpgradeStartGrace = 10m(alerts.go:163) equalsstaleUpgradeTimeout = 10m(server.go:127). Recovery can only fire once the upgrade is ≥10m old, and the spoke restarts on its next heartbeat afterheartbeatUpgradeis armed — so the restart the promotion causes always lands past the grace window of the only remaining mark.- Consequence: each promotion-driven recovery restart (plus the mutable-tag follow-up roll the grace comment describes) is counted by
observeStartas an unattributed restart. Two of those insidealertCrashLoopWindow(2h) plus one ordinary start reachesalertCrashLoopMinRestarts = 3and raises a spurious critical crash-loop alert on a hive that is simply being promoted. Previously this restart was excused.
Suggested fix: keep the clock, but still record the rollout — e.g. in the recoverTarget != upgradeTarget branch, call s.alerts.noteUpgrade(h.ID, time.Now()) alongside the pre-set, and assert in TestTriggerAutoUpgradesPromotionPreservesHoldDeadline that a start observed shortly after a promotion is not counted (observeStart stays below alertCrashLoopMinRestarts). noteUpgrade is nil-safe, so the existing test setup with no alerts still passes.
What I verified as correct
evaluateOrphanedUpgradereadsUpgradeStartedAt, so preserving it does hold the original 65m deadline as the new test asserts.- The zero-clock self-heal survives (
saas_upgrade.go:63still stamps onIsZero()), matching the last block of the new test. begin_upgrade_test.go:62-79("new target gets a fresh clock") is unaffected because it callsbeginUpgradedirectly; the bypass is confined to the recovery loop and the comment says so.
Body claims match the diff; scope matches #14929. I have not run the suite here.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Conclusion: the clock-preservation itself does what #14929 asks, but adopting the target before beginUpgrade has a side effect the test cannot see: an advanced target no longer records an upgrade mark for the crash-loop rule, so the spoke's restart onto the promoted image is counted as a crash.
Mechanism (read at merge-base and at 23f57c1):
src/pkg/hub/saas_upgrade.go:63-70—beginUpgradeonly callss.alerts.noteUpgrade(h.ID, h.UpgradeStartedAt)inside the "genuinely new upgrade" branch (!h.Upgrading || h.UpgradeTarget != target || UpgradeStartedAt.IsZero()). This PR setss.registry.Hives[i].UpgradeTarget = recoverTargetfirst (saas_upgrade.go:1260), so for a target advance that branch is now skipped entirely — no fresh clock and no new mark.src/pkg/hub/alerts.go:368-375—upgradeAttributableexcuses a spoke start only if it falls withinalertUpgradeStartGrace(10m,alerts.go:163) of a recorded mark. The only mark left is the originalUpgradeStartedAt.- Stale recovery runs only once
upgradeAge >= staleUpgradeTimeout(10m,server.go:127). So by construction the original mark is already ≥10m old when the promotion is delivered, and the restart the spoke performs to pull the new SHA cannot be attributed to an upgrade.observeStart(alerts.go:427-431) counts it; three such promotion restarts insidealertCrashLoopWindow(2h) with uptime under the ceiling fire a criticalcrash-loopalert (alerts.go:1066-1071) for a hive that was only following:stable.
This is exactly the case the base comment at saas_upgrade.go:1257-1259 ("a genuinely advanced target … DOES get a fresh clock") was protecting alongside #11824; the PR's rewritten comment drops that and the new test can't detect it because s.alerts is nil in the test server (noteUpgrade is nil-safe).
Severity: medium — false critical alert, no data impact; it needs ≥3 promotions during one stuck upgrade, which is plausible on a busy :stable channel but not common.
Suggested fix: keep the clock but still mark the delivery, e.g. in the recovery loop after s.beginUpgrade(i, recoverTarget):
if recoverTarget != upgradeTarget {
s.alerts.noteUpgrade(h.ID, time.Now())
}(noteUpgrade keeps a list of marks, so this does not disturb the original one.) A test with a non-nil alertState asserting observeStart returns 0 for a start a minute after the promotion would lock it in.
Everything else checks out: the lost-clock stamp still happens via the IsZero() arm, heartbeatUpgrade is armed with the advanced target, the same-target re-arm path is unchanged, and the silent-drain assertions in TestTriggerAutoUpgradesPromotionPreservesHoldDeadline match evaluateOrphanedUpgrade's use of UpgradeStartedAt.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@thatguywob this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Your relay session received this as a |
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
There was a problem hiding this comment.
Read the change against src/pkg/hub/saas_upgrade.go on v5; the mechanism is sound.
beginUpgrade (saas_upgrade.go:58-73) re-stamps UpgradeStartedAt only when !h.Upgrading || h.UpgradeTarget != target || h.UpgradeStartedAt.IsZero(). Writing UpgradeTarget = recoverTarget first (saas_upgrade.go:1261 in this PR) makes the second clause false, so a channel promotion in the stale-recovery block (:1195-1276, which only runs with alreadyUpgrading) keeps the original clock while a zero clock is still stamped. The explicit s.alerts.noteUpgrade(h.ID, time.Now()) for an advanced target preserves the #11824 crash-loop exemption that beginUpgrade would otherwise skip. Other beginUpgrade callers are unaffected: they go through the unchanged helper and still get a fresh clock on a new target.
TestTriggerAutoUpgradesPromotionPreservesHoldDeadline asserts UpgradeStartedAt.Equal(started) after promotion, which the pre-change code (fresh clock on recoverTarget != old target) would fail, so the test does guard the fix.
One note, not a defect: when the clock is zero and the target advances, noteUpgrade is called twice (inside beginUpgrade and again at :1266) with two time.Now() values a few ns apart. noteUpgrade (alerts.go:333) dedupes only on Equal, so two marks are kept; both fall in the same grace window and the list is capped, so the effect is nil.
go build ./pkg/hub/ passes; I could not run the suite here (hive#9845), CI was still pending at the time of this read. Looks correct to me.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Quality review at
No changes requested from the quality side. 🐝 Hive Agent: — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
There was a problem hiding this comment.
Read the recovery branch and its callee at v5, not just the diff. This looks correct to me.
Mechanism. beginUpgrade (src/pkg/hub/saas_upgrade.go:63) re-stamps UpgradeStartedAt when !h.Upgrading || h.UpgradeTarget != target || h.UpgradeStartedAt.IsZero(). Writing UpgradeTarget = recoverTarget on the entry before the call (saas_upgrade.go:1260) makes the second clause false on a promotion, so only a lost/zero clock is still re-stamped — the ibm-alchemy self-heal is preserved. noteUpgrade(h.ID, time.Now()) at :1265 replaces the mark beginUpgrade used to record on the target-change path, so observeStart (alerts.go) still excuses the promoted rollout's restart; noteUpgrade dedupes by exact time and ignores zero, so calling it outside beginUpgrade is safe.
Consumers checked. evaluateOrphanedUpgrade (stale_upgrade.go:149-167) and the stuck-upgrade check (alerts.go:1113-1114) both key off UpgradeStartedAt; neither depends on the clock being reset when the target moves, and converged compares GitHash against the current UpgradeTarget, which this PR advances. No other non-test writer of UpgradeStartedAt is affected.
Test. TestTriggerAutoUpgradesPromotionPreservesHoldDeadline fails on v5 (the entry.UpgradeStartedAt.Equal(started) assertion) and passes with the change; the zero-clock tail case guards the self-heal. All helpers it uses already exist in the package.
Minor, non-blocking: after a promotion the entry is still past staleUpgradeTimeout, so every subsequent triggerAutoUpgrades sweep re-enters the stale branch and logs the re-arming heartbeat fallback Warn until the spoke collects — that is already how same-target re-arms behave today, so this PR does not introduce it, just extends it to the promoted case.
Closes #14929 as stated; scope matches the issue.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
stage 095-pre-merge-soak → 060-reviewed: required-check-pending:Sign-off survives the squash |
|
@thatguywob this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
cd3a7b4
|
|
Needs a decision, not a merge: the automerge lane will not merge this unattended. Head: verdict approved @cd3a7b4, 58/60 checks green, 2 pending, DCO yes, soak 0s of 30m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
stage 095-pre-merge-soak → 085-needs-decision: contributor-lane:sentinel-blocked |
|
Needs a decision, not a merge:
Head: verdict approved @cd3a7b4, 58/60 checks green, 2 pending, DCO yes, sentinel alert, soak 0s of 30m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
hive recommends: review the sentinel finding on #14931 and remove the sentinel label if intended; owner to confirm Rationale: a sentinel alert on a PR from an author without write access is a human review. Decided by the hive at @clubanderson's request as the owner's delegate under the owner-delegate policy (rule |
|
/lgtm Posted by @clubanderson from the Hive dashboard; the merge gate decides from here. |
|
@hivecommons-hive[bot]: changing LGTM is restricted to collaborators DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/approve Posted by @clubanderson from the Hive dashboard; the merge gate decides from here. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: hivecommons-hive[bot] The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Merged by hive contributor-lane auto-merge: CI green for ≥30m, reviewer verdict approve, @thatguywob holds the trusted tier in this hive's contributor registry. |
Fixes #14929
Stale recovery now keeps the original upgrade start time when a release-channel promotion advances the target. Repeated promotions therefore retain the original 65-minute silent-drain deadline and stuck-upgrade elapsed time. Newly requested upgrades still use the existing fresh-clock behavior, and recovery still stamps a missing clock.
Adds regression coverage for repeated stable promotions, hold expiry at the original deadline, and lost-clock recovery, plus a changelog fragment.
Validation:
gofmt -lreports no changed files andgit diff --checkpasses. Tests were not run locally per AGENTS.md; CI runs the test suite.— hive: backend=codex
🐝 Hive Agent:
contributor| SHA:23f57c197