Skip to content

fix(homelab): skip unprovable join candidates and stop trusting a bare k0s token - #591

Merged
castrojo merged 3 commits into
projectbluefin:mainfrom
jeefy:fix/cluster-join-candidates
Nov 15, 2026
Merged

castrojo merged 3 commits into
projectbluefin:mainfrom
jeefy:fix/cluster-join-candidates

Conversation

@jeefy

@jeefy jeefy commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Join loop: tryCandidates (was attempt()) moves on to the next mDNS candidate when one cannot prove the passphrase (ErrNotProven). Previously it ended the whole round.
  • Joined check: joined() no longer treats /etc/k0s/token as proof that the node joined. It now requires one of:
    • the joined marker
    • the kubelet kubeconfig: /etc/kubernetes/kubelet.conf (kubeadm) or /var/lib/k0s/kubelet.conf (k0s worker)
  • New Go tests for cmd/bluefin-cluster:
    • control-plane and node passphrase handling
    • the exact DNS-SD record body, which must carry no secrets

Why

Join loop. mDNS answers are unauthenticated. A rogue advertiser, or another cluster's control plane on the same LAN, that answered first ended every round. The node then backed off and asked the same peer first again, so it never reached its own control plane.

The guess budget is unchanged. It is still shared by every candidate: charged before each PAKE and refunded only after the peer proves the passphrase. So ErrBudget still ends the round, and a rogue advertiser gets no extra guesses. Once a candidate fails to prove the passphrase, its other addresses are skipped.

Joined check. ops.Join writes the token before it starts k0s-first-boot.service. If that start failed, or the 15-minute token expired before k0sworker used it, a restarted bluefin-cluster join logged "already joined" and never asked for a fresh token.

Notes

  • cmd/bluefin-cluster statement coverage is 17.7%. The new tests cover the passphrase and DNS-SD logic; serve/runJoin are I/O glue and remain untested.
  • The DNS-SD text moved into dnssdBody(). The source-text check in test_homelab_cluster.py now looks there.

Testing

  • go vet ./... and go test -race ./... in files/homelab/cluster: pass. This covers tryCandidates with an injected join (a rogue candidate first, then budget exhaustion) and joined() under ops.Root.
  • pytest -n auto tests/unit: 1900 passed, 30 skipped.
  • ruff, mypy and docs-checks.py: clean.
  • just validate: passes.
  • Not exercised on a live mDNS LAN.

jeefy and others added 3 commits October 31, 2026 18:06
attempt() ended the whole round on the first ErrNotProven. mDNS answers
are unauthenticated, so a rogue advertiser, or a second cluster's control
plane on the same LAN, listed before the real control plane made every
round stop there: the node backed off and asked the same peer first
again, never reaching the control plane that knows its passphrase.

The loop (now tryCandidates, with the join call injected for tests) skips
the rest of a candidate that failed to prove the passphrase and moves on
to the next one. Its other addresses are not tried: they would only spend
guesses on the same answer. The guess budget is unchanged and still
shared by every candidate (charged before each PAKE, refunded only after
the peer proved the passphrase), so ErrBudget still ends the round and a
rogue advertiser gets no more guesses than before.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
joined() returned true as soon as /etc/k0s/token existed. ops.Join writes
that token before it starts k0s-first-boot.service, so when the start
failed, or k0sworker could not use the token before its 15-minute expiry,
a restarted bluefin-cluster join logged "already joined", wrote the
joined marker and never asked for a fresh token: the node stayed out of
the cluster for good.

A node now counts as joined only on the marker or on the kubelet
kubeconfig written after bootstrapping: /etc/kubernetes/kubelet.conf for
kubeadm (unchanged) and /var/lib/k0s/kubelet.conf for a k0s worker
(k0s's KubeletAuthConfigPath, which k0s itself checks first). joined()
honours ops.Root so it can be tested.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The cmd package had no Go tests; the only checks on its passphrase
handling and its mDNS advertisement were source-text assertions in
test_homelab_cluster.py. Add table-driven tests for:

- controlPlanePassphrase: operator-chosen passphrase normalised, stored
  0600 and preferred over a stored one; weak or non-ASCII ones refused
  without echoing them; a stored one reused unchanged; a 4-word one
  generated on first start and stored for the next.
- nodePassphrase: homelab.conf before the credential, the credential,
  neither, and non-ASCII input.
- the DNS-SD body, exactly, so the TXT record keeps carrying no secrets.

Minimal seams: the DNS-SD text moves into dnssdBody(), and the stored
passphrase is read under ops.Root like every other ops path. The Python
check now follows the body into dnssdBody() and still asserts the same
TXT format and that neither function mentions a passphrase or token.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy jeefy added the full-build Run the full image build and boot test on this pull request label Oct 31, 2026
@jeefy

jeefy commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jeefy added a commit that referenced this pull request Nov 3, 2026
…test.go

#591 adds cmd/bluefin-cluster/main_test.go with its own nodePassphrase
cases, so this file moves to join_test.go and drops the three cases
that duplicate them (conf beats credential, credential fallback,
non-ASCII). Kept: the missing-passphrase error naming its sources and
the manual-control-plane joins against a loopback join.Server.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy marked this pull request as ready for review November 3, 2026 02:56
@jeefy
jeefy added this pull request to the merge queue Nov 3, 2026
@jeefy
jeefy requested review from castrojo and hanthor November 3, 2026 02:56
@jeefy

jeefy commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jeefy
jeefy removed this pull request from the merge queue due to a manual request Nov 3, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy jeefy added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 3, 2026
@jeefy

jeefy commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo
castrojo added this pull request to the merge queue Nov 4, 2026
@castrojo
castrojo removed this pull request from the merge queue due to a manual request Nov 4, 2026
@castrojo
castrojo added this pull request to the merge queue Nov 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 9, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 9d93459: the join loop skips a candidate that cannot prove the passphrase (ErrNotProven moves to the next candidate, ErrBudget still ends the round) and joined() no longer trusts a bare /etc/k0s/token; Go tests cover both.

/approve

@hivecommons-hive

hivecommons-hive Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Ready for a human merge: verdict approved @9d93459, 25/25 checks green, soak 30m0s elapsed, labels: hold. Held: the hold label is set. The hive never merges this head while the gate stands.

Confirm the intent (resolve the sentinel alert or lift the hold), then /lgtm /approve (or merge) if you accept it.

Updated 2026-11-14 23:36 UTC.

@Danathar

Copy link
Copy Markdown
Contributor

/approve

@jeefy jeefy added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 13, 2026
@jeefy

jeefy commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danathar

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 13, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 14, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 14, 2026
@castrojo

Copy link
Copy Markdown
Contributor

/hold

Holding until the BuildStream cache root-cause fix #664 lands. Do not enqueue pre-fix builds.

@jeefy jeefy added the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
@jeefy
jeefy removed this pull request from the merge queue due to a manual request Nov 14, 2026
@castrojo

Copy link
Copy Markdown
Contributor

/hold

Cache repair order: #664 first; then #669 (payload-completeness correction, retargeted to main after #664 lands). Keep this PR out of the queue until both are merged, both main seed variants have actually succeeded, and the GCC payload can be pulled from the project CAS. Then resume #640 before #591, one at a time. #667 and #665 are closed as superseded; do not re-enqueue them.

@castrojo

Copy link
Copy Markdown
Contributor

Sequencing correction: #665, #667 and #669 are now closed because every correction is consolidated directly into #664 at 47ce75e, including the verified systemd clock-epoch fix. Keep this PR held until #664 passes the full queue and merges, both main seed variants succeed and a real project-CAS GCC payload pull succeeds. Then resume #640 before #591, one at a time; no separate #669 merge is required.

@castrojo

Copy link
Copy Markdown
Contributor

Keep held: final cache-repair sequence is #664, verified main seed payloads, then prepared #670 to remove advisory warming from the release critical path. #670 is draft behind #664 so the running full build is not restarted. Resume #640 before #591 only after those prerequisites are complete.

@castrojo castrojo added full-build Run the full image build and boot test on this pull request and removed full-build Run the full image build and boot test on this pull request labels Nov 14, 2026
@castrojo

Copy link
Copy Markdown
Contributor

/hold cancel

#664 is merged and passed its main build, all four boot buckets and publication. The additional dependency on #670 was unnecessary: its cache-producer completeness and advisory-warming improvements do not gate this approved consumer change. Cancelling my earlier sequencing hold.

Requested fresh pull-request checks using the current-main merge ref, rather than rerunning stale pre-#664 snapshots. The approved PR head is unchanged, so Danathar’s review is preserved. Use the normal merge queue and its required checks; no administrative bypass.

@castrojo castrojo removed the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
@castrojo
castrojo enabled auto-merge November 14, 2026 23:35
@castrojo castrojo added kind/bug Something is broken. and removed full-build Run the full image build and boot test on this pull request labels Nov 15, 2026
@castrojo
castrojo added this pull request to the merge queue Nov 15, 2026
Merged via the queue into projectbluefin:main with commit bb459ab Nov 15, 2026
143 of 162 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Something is broken. lgtm "Looks good to me", indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants