Repository navigation
ops(spire): durable x509pop node attestation (TODO 36) - #18
Merged
Merged
Conversation
join_token is single-use: an SVID expiry + agent restart crash-loops on
"token already used", dropping the orchestrator into SPIFFE-degraded mode.
Recurred 2026-07-09 and twice 2026-07-13, each needing manual token rotation
+ workload-entry re-registration.
x509pop derives the agent SPIFFE ID from a leaf-cert fingerprint: stable across
restarts, no consumable token, so the crash-loop cannot occur.
- deploy/spire/server.conf: add the x509pop NodeAttestor (ca_bundle_path);
join_token kept as a break-glass fallback attestor.
- deploy/spire/agent.conf: switch the agent to x509pop (leaf cert + key).
- docker-compose.yml: mount the CA bundle into spire-server and the leaf
cert/key into spire-agent (from ${SPIRE_X509POP_DIR:-./deploy/spire/x509pop});
drop the -joinToken command args.
- .gitignore: deploy/spire/x509pop/ (host-local trust material, like .env).
- deploy/spire/x509pop-migration.md: cert-gen + apply + re-register + rollback
runbook.
Both configs pass `spire-{server,agent} validate`; the generated CA/agent chain
verifies (openssl verify OK); docker compose config parses. The live cutover is
a deliberate host-side step per the runbook (certs are not committed); rollback
restores join_token. Blast radius of a bad apply is bounded to degraded mode.
Signed-off-by: Jodson Graves <info@ntari.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Durable SPIRE node attestation (#6 of the operator-readiness plan; SoHoLINK TODO 36).
Problem
join_tokenis single-use. When the agent SVID expires while the agent is down (a reboot, a multi-hour outage), the agent restarts, re-attests with the already-consumed token, and crash-loops on "join token does not exist or has already been used" — dropping the orchestrator into SPIFFE-degraded mode. This recurred 2026-07-09 and twice on 2026-07-13, each needing a manual token rotation + workload-entry re-registration.Fix — x509pop
x509popderives the agent's SPIFFE ID from a leaf-certificate fingerprint. The ID is stable across restarts and there is no consumable token, so the crash-loop cannot occur by construction.deploy/spire/server.conf— add thex509popNodeAttestor (ca_bundle_path);join_tokenkept as a break-glass fallback.deploy/spire/agent.conf— switch the agent tox509pop(leaf cert + key).docker-compose.yml— mount the CA bundle into spire-server and the leaf cert/key into spire-agent (from${SPIRE_X509POP_DIR:-./deploy/spire/x509pop}); drop the-joinTokencommand args..gitignore—deploy/spire/x509pop/(host-local trust material, treated like.env; certs never committed).deploy/spire/x509pop-migration.md— cert-gen + apply + re-register + rollback runbook.Verification
spire-server validate/spire-agent validate.openssl verify→ OK).docker compose configparses.Applying to prod (deliberate step — not in this PR)
The live cutover is host-side per the runbook: generate the certs into
deploy/spire/x509pop/, recreate the SPIRE pair, re-register the orchestrator workload entry under the new stable parent ID, recreate the orchestrator, then verifyidentity:readyand that an agent restart re-attests with no token. Rollback restoresjoin_token. Blast radius of a bad apply is bounded to SPIFFE-degraded mode (public site stays up; node routes 503) — recoverable by rollback, and there are no live SPIFFE node consumers today.Author does not self-merge — for review.