Skip to content

✨ hub: registration key rotation with a grace window - #19240

Open
hivecommons-hive[bot] wants to merge 1 commit into
v5from
scanner/fix-19235
Open

hivecommons-hive[bot] wants to merge 1 commit into
v5from
scanner/fix-19235

Conversation

@hivecommons-hive

@hivecommons-hive hivecommons-hive Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Step 2 of the hub registration protocol rollout (#19056). Before this, rotating a self-service registration key kept the old key alive for a fixed 10 minutes. That is too short for an owner who redeploys a spoke by hand. Rotating twice inside that window also caused an outage: the key the spoke was still running was retired at once, because the previous slot was given to the key minted a moment earlier.

With the new HIVE_HUB_SELFSERVICE_KEY_GRACE flag on:

  • the old key keeps authenticating for HIVE_HUB_SELFSERVICE_KEY_GRACE_WINDOW (default 1h, clamped to 168h; an invalid value falls back to the default);
  • rotating again while a window is open keeps the old (deployed) key and replaces the pending one, and the window restarts from that rotation;
  • overlap in both rotate responses and limits.key_rotation_overlap in discovery report the window in effect.

The flag is off by default, and rotation then behaves exactly as before. The record shape is unchanged, so existing registrations need no data change. Nothing writes to GitHub. The selfservice_hive_rotated audit line gains grace, replaced_pending and old_key_expires_at, and still carries no key and no hash. A caller-chosen grace in the request body is still refused, as before.

Open choice for review: the window restarts on a replacing rotation, so the owner always gets a full window to deploy the newest key. Repeated rotations can therefore keep the old key alive longer. Only the owner can rotate, and the window is capped.

Related issues

Closes #19235
Fixes #19056

Testing

  • Not run locally: CI runs validation for agent-authored PRs (AGENTS.md forbids local builds/tests in agent panes). Only gofmt -l was run on changed files.

New tests in selfservice_key_grace_test.go: both keys work inside the window and the old key is refused after it; a second rotation replaces the pending key, keeps the original and restarts the window; with the flag off, the legacy behaviour is unchanged; env parsing (default, custom, invalid, negative, ceiling); and an end-to-end v1 rotate with the audit line checked for grace fields and for no key or hash.

🤖 Generated with Claude Code

-- hive: agent=scanner backend=claude model=claude-sonnet-5-5 claude=2.1.328 requested_by=@clubanderson

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 19, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 19, 2026
@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#19253 ✨ customers: hub import disclosure, docs and consent views same files src/docs/env-vars.md
src/docs/hub-deployment.md
#19252 ✨ feature: let spokes sync their own registration repos CONFLICT src/docs/api-reference/hub-server.md (conflict)
src/pkg/hub/selfservice_registry.go
src/pkg/hub/selfservice_v1_api.go
#19202 ✨ ask: hub panel for optional Ask AI help same files src/docs/api-reference/hub-server.md
src/docs/env-vars.md
src/docs/hub-deployment.md
#19204 ✨ ask: hive backend and hub relay for optional Ask AI help same files src/docs/api-reference/hub-server.md
src/docs/env-vars.md
src/docs/hub-deployment.md
#19206 ✨ ask: hive panel and rollout verification for optional Ask AI help same files src/docs/api-reference/hub-server.md
src/docs/env-vars.md
src/docs/hub-deployment.md

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should a rotated-away registration key stay valid for a configurable window of up to 168h (restarting on every re-rotation) when HIVE_HUB_SELFSERVICE_KEY_GRACE is on?

Read selfservice_key_grace.go and the selfservice_registry.go change. The code does what the body says and I found no defect; this needs a human only because it changes credential-lifetime policy.

Verified: rotateWithPolicy (selfservice_key_grace.go, the replacePending branch) keeps PrevKeyHash and sets PrevKeyExpiresAt = now + window on every rotation. So an owner who rotates because a key leaked still has that key authenticating for 1h by default (up to 7d via HIVE_HUB_SELFSERVICE_KEY_GRACE_WINDOW), and repeated rotations keep pushing the window out. The PR body already names the restart behaviour as its open choice. It is off by default, the legacy rotateIfMatch path is preserved, and the audit line carries no key or hash.

Suggestion, not a defect: there is no way to rotate and revoke the old key immediately in the grace mode; an owner responding to a leak may want that.

-- hive: agent=reviewer-2 backend=claude model=claude-sonnet-5-5 claude=2.1.328

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

2 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/pkg/hub/selfservice_key_grace.go 41 ARITHMETIC_BASE
src/pkg/hub/selfservice_key_grace.go 41 ARITHMETIC_BASE

Verdict: uncovered (#17349).

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
@hivecommons-hive hivecommons-hive Bot removed the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Nov 19, 2026
@hivecommons-hive hivecommons-hive Bot added the stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue label Nov 19, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Needs a decision, not a merge:

  • the reviewer swarm returned requires_human @9cb87e2
  • the review found: the reviewer marked this change high-risk
  • reviewer question: @clubanderson should decide whether a rotation inside an open grace window may keep the old key alive and restart its window, given that this removes "rotate to evict a leaked key" as a fast path.
  • the automerge lane will not merge this unattended

Head: verdict requires_human @9cb87e2, 73/73 checks green, DCO yes, soak 0s of 10m0s.

What to do: approve this head (9cb87e2) to merge as-is, or leave a changes requested review (or reply /hive decision <what to change>) and the hive will push the fix.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

Updated 2026-11-19 16:56 UTC.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
@hivecommons-hive hivecommons-hive Bot removed the stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue label Nov 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

/approve

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

/hive decision approve #19240: green head, reviewer question answered with the reversible default — proceed as implemented

Rationale: no owner rule holds this back; a merge is reversible and keeps work moving.

Hive decided: Approved for merge.
Why: No special rule applies; a green branch is approved.
What happens next: The merge queue merges this PR once its checks are green — expected within 30 min.
If it doesn't happen: A maintainer can merge it by hand or send it back for a new commit; approving again does not help.

Decided by the hive at @clubanderson's request as the owner's delegate under the owner-delegate policy (rule default-release).

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Approved by @clubanderson for Hive auto-merge on green CI.

@hivecommons-hive hivecommons-hive Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Hive queued this PR but could not set the prow/lgtm commit status, so Prow may strip the lgtm label and the auto-merge queue may not hold.

@hivecommons-hive hivecommons-hive Bot added the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Nov 19, 2026
@kubestellar-prow kubestellar-prow Bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security delta re-review at ff39fea: the delta is a main merge, and none of this PR's selfservice_* files changed. The previous finding (old key stays valid; the window restarts on each rotation, selfservice_key_grace.go:113) is still present. It is a disclosed, owner-only, flag-off-by-default, capped (168h) design choice, and #19235 doesn't specify the restart. It is downgraded to a low out-of-scope advisory; a revoke-old-key path would be a good follow-up. The merged hub/ hunks (heartbeat_protocol.go) add no new auth surface. I did not review the crux/proxy, customers, dashboard or github hunks in depth.

Verdict: approve.

-- hive: agent=ci-maintainer backend=claude model=claude-sonnet-5-5 claude=2.1.328

@hivecommons-hive

hivecommons-hive Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review follow-ups: 1 open of 1 (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line, listed here instead of as issues. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low security — Replacing rotation restarts the old-key window, so a leaked old key cannot be cut off early (src/pkg/hub/selfservice_key_grace.go:113): Still present, unchanged since the previous head: h.PrevKeyExpiresAt = now.Add(policy.window) runs on every rotation, so repeated rotations keep the original (possibly leaked) key valid, each time fo…

Review findings from reviewer-2 (folded here because this head already carries its hive review):

The code reads correctly for its stated goal (a double rotation no longer cuts off the key the spoke is running), and the flag is off by default. One credentials-policy question is open.

Finding (security / credentials, medium): in rotateWithPolicy (src/pkg/hub/selfservice_key_grace.go, the policy.replacePending && h.PrevKeyHash != "" && now.Before(h.PrevKeyExpiresAt) branch) the old key is kept and PrevKeyExpiresAt is reset to now + window. Mechanism: an owner who rotates because key A leaked gets A live for up to the window (default 1h, ceiling 168h). If they rotate again inside that window, A is still kept and its expiry restarts, so repeated rotation can extend a leaked key's life and never retires it. Before this change a second rotation retired A immediately. The only remaining way to cut A off is DELETE of the registration, which drops the registry entry too.

Options for the decision: (a) accept it and document that revocation means delete, (b) do not restart the window when replacedPending is true (cap A's life at its first expiry), or (c) offer an explicit "revoke old key now" action. I lean to (b), which keeps the double-rotation protection without extending exposure.

Suggestion only: a test that pins the chosen behaviour (second rotation inside the window does or does not move PrevKeyExpiresAt) would make the decision explicit.

Review findings from reviewer (folded here because this head already carries its hive review):

Credential-rotation semantics, so this needs a person. The code reads correctly otherwise: rotateIfMatch delegating to rotateWithPolicy with the legacy overlap keeps the flag-off path unchanged, the store is restored on a failed save, and the audit line carries no key or hash.

Open question (medium), src/pkg/hub/selfservice_key_grace.go:106. With the flag on, a rotation inside an open window keeps PrevKeyHash and overwrites KeyHash. That is right when the spoke still runs the old key, but it is the mirror-image outage when the owner already deployed the newer key: rotate (key1 old, key2 new), redeploy the spoke with key2, rotate again 10 minutes later while the 1h window is open. key1 is kept, key2 is dropped, and the spoke now running key2 gets refused. The hub has no deployed/last-used signal on the record to tell the two cases apart (the PR keeps the record shape unchanged), so the PR trades one outage for another rather than removing it. The PR body names the window-restart choice but not this case. Options: accept and document it in hub-deployment.md, or record a first-successful-auth on the new key and only replace pending while it has not been used.

I did not verify behaviour of the auth path beyond the diff, so treat the consequence as following from the new rotation logic alone.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

/approve

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 19, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 19, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 19, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether a rotation inside an open grace window may keep the old key alive and restart its window, given that rotation is then not a way to revoke a leaked key.

Read selfservice_key_grace.go and the two rotate call sites in the diff. The flag-off path is unchanged and the store save/rollback looks right. One credential-policy choice needs an owner, not a code fix: see the thread on rotateWithPolicy.

-- hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.328

now := st.clock().UTC()
before := *h
var info selfServiceRotation
if policy.replacePending && h.PrevKeyHash != "" && now.Before(h.PrevKeyExpiresAt) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the flag on, a second rotation inside an open window keeps PrevKeyHash and sets PrevKeyExpiresAt = now.Add(policy.window) (the next lines). So an owner who rotates because the old key leaked cannot revoke it: it stays valid for another full window (up to 168h), and each further rotation restarts it. Conversely, if the owner already deployed the pending key, the second rotation drops the deployed key at once, the same outage this PR set out to remove. Both are consequences of the keep-old/replace-pending rule. Suggest either an explicit owner-initiated revoke of the old key, or not restarting the window (keep the original expiry) on a replace. Which behaviour is wanted is a decision for a maintainer.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 19, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

/approve

Posted by @clubanderson from the Hive dashboard; the merge gate decides from here.

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR conflicts-with-open-pr Open PR has a merge conflict with another open PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/L Denotes a PR that changes 100-499 lines, ignoring generated files. stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

0 participants