Skip to content

feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet, and fix the boot deadlocks behind it - #210

Closed
castrojo wants to merge 46 commits into
mainfrom
fix/boot-zfs-udev-and-depmod
Closed

castrojo wants to merge 46 commits into
mainfrom
fix/boot-zfs-udev-and-depmod

Conversation

@castrojo

@castrojo castrojo commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two things land together because the second is what made the first testable: the boot fixes that got a freshly installed host past local-fs.target, and the k0s → upstream Kubernetes cutover that sits on top of them.

Design: docs/superpowers/specs/2026-09-19-kubernetes-cutover-design.md.

Boot fixes (commits 1–17)

  • fix(kernel) — depmod -b pointed at /usr in the DDI and installer elements, so module indexes are generated against the real module root.
  • fix(zfs) — remove Flatcar's systemd-udevd.service.d/10-zfs.conf from the extracted ZFS rootfs and strip the tmpfiles.d rule that copies it back into /etc. Flatcar orders udevd After=systemd-sysext.service because it merges ZFS as a runtime sysext; we extract it into the rootfs at build time, so the drop-in deadlocks udev → var.mount → local-fs.target. This supersedes fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173, which removed the drop-in but left the tmpfiles.d rule that restores it.
  • fix(ddi) — mask systemd-homed-firstboot.service alongside systemd-firstboot.service. Both are ConditionFirstBoot=yes with StandardInput=tty ordered before first-boot-complete.target; either one left unmasked hangs a headless boot forever. Supersedes fix(boot): mask systemd-homed-firstboot on the target image #186.
  • test(installer) — the QEMU smoke gate now forwards SSH into the guest and requires HTTP 200 plus a ready status, instead of accepting any response.
  • fix(installer) — flash-installer searches dist/ with -maxdepth 1 so it cannot pick up a nested release image.
  • feat(installer) — a provenance sidecar: _record-provenance / _inputs-hash in the justfile write a plain-text <image>.provenance sidecar beside each exported artifact, recording the commit and input-tree tree hash it was built from, so a dist/ image can be traced back to a tree state and staleness is detectable rather than assumed.
  • test(installer) — just test-installer-boot, a firmware-boot recipe that boots the exported installer medium under QEMU and asserts it reaches the installer, covering the boot path the smoke gate starts after.
  • fix(flash) — flash-installer refuses to write to the disk the running system boots from or to a device with mounted partitions, and reads the image back after writing to prove the medium holds it.

Kubernetes cutover (commit 18)

BREAKING CHANGE: the k0s sysext and its release assets are gone. Hosts tracking k0s-@v.raw.zst must be reinstalled from a current DDI. There is no in-place migration: kubeadm-init.service blocks on containerd, which only reaches /var/lib/extensions via files/installer/repart.d/30-var.conf at install time. containerd-flatcar.raw is not a published release asset and has no sysupdate.containerd.d transfer, so updating to kubernetes-@v.raw.zst alone would leave a host with kubelet/kubeadm units and no container runtime. Publishing containerd on its own transfer is open item 3 in the cutover design doc.

  • elements/kubernetes/{kubernetes-bin,cni-plugins}.bst import the upstream release tarballs, pinned by include/kubernetes.yml (k8s v1.36.4, CNI plugins v1.9.1).
  • elements/oci/kubernetes-sysext.bst packs kubeadm, kubelet, kubectl and the CNI plugins into an EROFS sysext. The base DDI carries no Kubernetes — AGENTS.md hard rule 4.
  • files/os/sysupdate.kubernetes.d/70-kubernetes.transfer tracks the published kubernetes-@v.raw.zst asset.
  • The installer seeds /lib/extensions/kubernetes.raw straight onto /var, so first boot merges the sysext with no network fetch.
  • kubeadm-init.service initialises the control plane after the sysext merges; kubelet.service plus the kubeadm drop-in run the node.
  • bluefin-cluster-repo mirrors the read-only manifests from /usr into /var/lib/bluefin/cluster.git; bluefin-cluster-bootstrap seeds Cilium → Argo CD Core → git-daemon → root App, then hands over to Argo CD.

Security defects fixed by the new manifests

The k0s manifests carried four defects that open PRs were each patching individually. The replacement manifests do not have them:

Defect k0s behaviour Now
Console JWT secret (#72) jwt-secret: default-jwt-secret-… committed, identical on every install files/cluster/manifests/console/auth-secret.yaml mints a random secret in-cluster, once, idempotently
PostgreSQL superuser password committed in the manifest files/cluster/manifests/kubeflex/postgres-secret.yaml generates 32 random chars in-cluster
Kiosk TLS private key (#87) generated at sysext build time and shipped in public release assets issued by cert-manager at runtime (pki/issuers.yaml, kiosk/certificate.yaml)
Kiosk reachability (#198, #194) hostIP: 127.0.0.1 pinned the proxy to guest loopback files/cluster/manifests/kiosk/deployment.yaml publishes on every interface
Demo data on first boot (#193, #196) DEV_MODE=true pinned in the console deployment and in its tests DEV_MODE / ALLOW_DEV_MODE_IN_CLUSTER / DEV_USER_LOGIN removed; controlplanes/controlplanes.yaml deploys real ControlPlane instances

Argo CD also moves to Argo CD Core (files/cluster/seed/10-argocd-core/), which drops the argocd-server component entirely — so the --insecure stale-image exposure in #74 has no surface left.

Verification

  • just validate — DDI, installer, and oci/kubernetes-sysext.bst graphs all resolve.
  • pytest tests/unit — 273 passed, 1 xfailed.
  • bats tests/unit — 63 passed.
  • python3 .github/scripts/docs-checks.py — passed.
  • python3 .github/scripts/check-release-version.py — passed.
  • python3 .github/scripts/check-kubernetes-version.py — passed (v1.36.4 / CNI v1.9.1 consistent across include/kubernetes.yml, the transfer MatchPattern, and VERSION_ID).

Rebased onto main after #207 and #169; the rebase is content-identical to the previously pushed tip apart from those two merges.

Supersedes

Refs #72, #74, #87, #193, #196, #197, #198, #199. (#159 was already fixed on main and closed on 2026-09-20; the closing keyword was stale and has been dropped.)
Supersedes #173, #186, #200, #201, #203, #204, #205, #178, #81, #90, #91.

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

@castrojo
castrojo force-pushed the fix/boot-zfs-udev-and-depmod branch from 8189e09 to 2f041c6 Compare September 20, 2026 10:01
@castrojo

Copy link
Copy Markdown
Contributor Author

⚠️ Blocker found during review: this branch produces an inert cluster

Flagging before anyone flashes an artifact from this PR. The OS boots; the cluster does not come up.

files/os/systemd/system/kubeadm-init.service declares:

Requires=containerd.service
After=containerd.service

Nothing in this repository provides containerd.service. elements/oci/kubernetes-sysext.bst stages exactly kubectl, kubeadm, kubelet and the CNI plugins — no containerd, no runc, no crictl:

cp -a /usr/bin/kubectl sysext/usr/bin/kubectl
cp -a /usr/bin/kubeadm sysext/usr/bin/kubeadm
cp -a /usr/bin/kubelet sysext/usr/bin/kubelet
cp -a /usr/local/bin/cni/. sysext/usr/local/bin/cni/

A grep across elements/, include/ and files/os/ finds only references to containerd, never a provider.

It cascades: bluefin-cluster-bootstrap.service declares Requires=kubeadm-init.service. So a machine installed from this branch reaches multi-user.target with no container runtime, no control plane, no Argo CD, no console — the bootstrap chain never starts.

containerd reaches the image solely through Flatcar's /usr (usr/share/flatcar/sysext/containerd-flatcar.raw), which arrives with #140 (elements/flatcar/flatcar-usr.bst) and is consumed by #132 (os-stack.bst).

Consequence for the review queue

I had parked #140 and #132 under 2-discussing as part of the Flatcar-base architectural preference (see the anchor on #133). That classification was wrong for these two: they are hard prerequisites for this PR delivering a working appliance, not an open preference. Re-labelling accordingly.

The unit's header comment previously claimed "/usr/bin/kubeadm and containerd.service both arrive with the sysext merge", which is false for the second half and is what hid this. Corrected in df01abe, which now names the two PRs. tests/unit/test_unit_dependency_provenance.py already recorded the gap in its EXTERNAL_PROVIDERS note — that note is the thing that surfaced it.

Also fixed in this push

de09215 — os-cluster-bootstrap.bst and os-cluster-manifests.bst are kind: manual with no build-commands, so FSDK's default strip-commands shelled out to freedesktop-sdk-stripper, which base/base-stack.bst does not put in the sandbox:

sh: line 11: freedesktop-sdk-stripper: command not found
Command '...' failed with exitcode 127
FAILURE bluefin-server/os-cluster-manifests.bst: Running commands

That is what killed the first Build DDI artifacts run on this branch. Both now set variables: strip-binaries: "", matching os-countme.bst, os-image-info.bst and os-release-flatcar.bst.

just validate could not have caught it — bst show --deps all resolves the graph without executing a build command. Added tests/unit/test_manual_element_strip_contract.py, which asserts the invariant directly and is mutation-tested: removing either override fails the gate. Verified locally that just bst build bluefin-server/os-cluster-manifests.bst bluefin-server/os-cluster-bootstrap.bst now succeeds, having failed at exit 127 before.

Current state

  • pytest tests/unit — 324 passed, 1 xfailed
  • bats tests/unit — 92 passed
  • just validate — clean
  • Build DDI artifacts — re-running on df01abe

Do not merge on a green build alone. A green installer-test proves the installer writes a bootable disk; it does not prove the cluster starts, because the smoke gate's console probe is what would catch that and it is downstream of the same broken chain.

@castrojo

Copy link
Copy Markdown
Contributor Author

Containerd gap closed in 6753cc9 — the blocker above is resolved, and it did not need the payload-base decision.

Flatcar publishes containerd as a standalone systemd-sysext, which is the same delivery mechanism this PR already uses for Kubernetes. elements/flatcar/containerd-sysext.bst imports it; the installer stages it into the initrd; files/installer/repart.d/30-var.conf seeds it to /var/lib/extensions for an offline first boot, exactly as it already does for kubernetes.raw.

The image is self-contained — I unpacked the published raw to check rather than assuming:

/usr/bin/containerd, containerd-shim-runc-v2, ctr, runc
/usr/lib/systemd/system/containerd.service
/usr/lib/systemd/system/multi-user.target.wants/containerd.service
/usr/lib/extension-release.d/extension-release.containerd-flatcar
/usr/share/containerd/config.toml, config-cgroupfs.toml
/usr/share/SLSA/*.json.zst            (containerd 2.1.5, runc 1.3.3)

It carries its own multi-user.target.wants symlink, so the merge enables the service; no preset of ours is involved. sha256 78e38344… verified byte-for-byte against the pin.

Merge compatibility is not incidental: extension-release.containerd-flatcar declares ID=flatcar VERSION_ID=4593.2.5, and elements/bluefin-server/os-release-flatcar.bst writes ID=flatcar VERSION_ID=%{flatcar-version} from include/flatcar.yml, which is 4593.2.5. That element's own description says it exists "to support Flatcar system extensions" — this is the case it was built for.

Unlike our Kubernetes sysext (ID=_any), this one is version-locked to the Flatcar release, so bumping flatcar-version requires re-pinning the containerd image in the same change. Recorded in 30-var.conf.

One trap worth recording

The image must keep its upstream filename. systemd-sysext resolves metadata by stem: <name>.raw requires extension-release.<name>. My first cut installed it as containerd.raw, which builds, validates, passes unit tests, writes a bootable disk — and then sysext looks for extension-release.containerd, finds nothing, refuses the extension, and you are back to a missing containerd.service with no error anywhere in CI.

tests/unit/test_seeded_sysext_names.py now asserts the chain end to end and is mutation-tested against four regressions: renaming in CopyFiles=, staging a different name in the installer, shortening the element's install target, and dropping the seeding line altogether.

Also in this push

de09215 — os-cluster-bootstrap.bst and os-cluster-manifests.bst omitted variables: strip-binaries: "", so FSDK's default strip-commands shelled out to freedesktop-sdk-stripper, which is not in their sandbox. That killed the first Build DDI artifacts run at exit 127. tests/unit/test_manual_element_strip_contract.py now asserts the invariant.

df01abe — corrected the kubeadm-init.service header, which claimed containerd arrived with the Kubernetes sysext merge.

Two tests that pinned the broken state were removed rather than re-pinned: test_containerd_provider_is_recorded_as_pending recorded the gap as permanent, and test_var_seeds_the_sysext_into_the_systemd_sysext_scan_directory compared repeatable CopyFiles= for single-value equality. load_config() also drops configparser(strict=True), which raises DuplicateOptionError on valid systemd input.

Current state

  • pytest tests/unit — 328 passed, 1 xfailed
  • bats tests/unit — 92 passed
  • just validate — clean
  • just bst build flatcar/containerd-sysext.bst — clean
  • docs-checks.py, check-release-version.py, check-kubernetes-version.py — clean

#140 and #132 are no longer prerequisites for this PR. Re-labelling them back to the #133 decision cluster where they belong.

clubanderson pushed a commit to hivecommons/hive that referenced this pull request Sep 20, 2026
… claims them all, and a reference list yields every element

The reference tier (#3980) only ran when a PR had no closing keyword and
no issue in its branch name, so "Closes #159. Refs #72, #74, … #197"
claimed #159 alone. And neither parser read a list, so even a bare
"Refs #72, #74, #87" yielded #72. projectbluefin/server#210 carried
exactly that body while the contribute queue offered #197 to an agent.

claimsFromPR now reads references for every PR and unions them with the
closing/branch claims per issue — an issue both closed and referenced
keeps only its strong claim, so no existing claim changes target or
strength; this only adds weak ones, bounded by weakClaimDeferWindow as
before. referenceRefPattern takes a same-line list tail (comma,
semicolon, "and", "&", cross-repo elements; a newline ends it) and
parseRefs reads every #N out of the whole match rather than the
last-iteration capture of a repeated group. The closing parser stays
single-issue on purpose: GitHub closes only the first issue of
"Fixes #1, #2", and a strong claim must not outrun that.

Tests: seven new ParseReferencedIssues cases (lists, "and", oxford
comma, cross-repo element, line/prose termination, the real #210 body)
and FetchClaims cases for the union, the both-closed-and-referenced
case, and #210 end to end. pkg/github and pkg/dashboard green except
TestHiveOpenPRScript_MultilineBodyIsValidJSONWithoutPython3, which
fails identically on the untouched tree here (host lacks the script's
non-python fallback).

Fixes #7911

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016oaySY5QPzxR6wRKGC2oon
Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency

  • files/installer/repart.d/30-var.conf:29-33 claims the seeded containerd sysext "merge is additionally gated on os-release … version-locked to the Flatcar release". That is the pre-repack behaviour: elements/flatcar/containerd-sysext.bst (install-commands) rewrites the extension-release to ID=_any with no VERSION_ID, and tests/unit/test_seeded_sysext_names.py::test_seeded_sysexts_are_decoupled_from_the_host_os_release_version — added in this same PR — asserts exactly the opposite of what this comment says. A maintainer doing a flatcar-version bump who reads 30-var.conf first will conclude an OS update breaks the merge when the whole point of the repack is that it doesn't. (Re-pinning the source ref: on a bump is still required; the "gated on os-release" claim is what's wrong.)

correctness

  • Justfile (flash-installer, running-system guard): the guard resolves the disk behind / and /sysroot via lsblk -no PKNAME "${SRC}" and skips the check when that is empty. PKNAME is empty when the source is a whole-disk device (fs directly on /dev/sdX, no partition table), so on such a host the guard is silently inert and only the y/N prompt remains. Low likelihood, but the comment sells the guard as the real defence. Falling back to KNAME of SRC when PKNAME is empty closes it.

style

  • files/os/justfile:12-13: the role check accepts ROLE="server" but the error message says Supported: controller, and the step-0 comment says "Reject unsupported roles" while silently accepting an undocumented alias. Either document server or drop it.

No findings from: security, intent-alignment.

— hive: agent=reviewer backend=copilot model=claude-fable-5

`lsblk -no PKNAME` returns nothing when the filesystem sits directly on a
whole-disk device, so the running-system guard skipped its comparison and
left only the y/N prompt protecting the system disk. Fall back to KNAME
when PKNAME is empty, and cover both the refusal and the unrelated-disk
path in the bats suite.

Also correct the 30-var.conf rationale: containerd-sysext.bst repacks the
extension-release to ID=_any with no VERSION_ID, so the merge is not gated
on os-release and a flatcar-version bump only needs the source ref re-pinned.

And make the k8s role guard's message match its behaviour by documenting
the accepted "server" alias instead of naming only "controller".

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • bluefin-cluster-bootstrap.service keeps Requires=bluefin-cluster-repo.service (files/os/systemd/system/bluefin-cluster-bootstrap.service:30), but bluefin-cluster-repo.service is a Type=oneshot with no Restart= (files/os/systemd/system/bluefin-cluster-repo.service:10-14). A single transient failure of the repo publish (e.g. /var slow, git error) kills the bootstrap job permanently with result dependency — the exact stranding mode this PR's own comments document and fix for kubeadm-init.service (bluefin-cluster-bootstrap.service:11-18). Adding Restart=on-failure/RestartSec= to bluefin-cluster-repo.service closes the gap.
  • kubelet.service bridges CNI plugins with cp -an /usr/local/bin/cni/. /opt/cni/bin/ (files/os/systemd/system/kubelet.service:22), and tests/unit/test_unit_dependency_provenance.py pins the no-clobber flag. If /opt/cni/bin persists across reboots (nothing in the tree declares it volatile), a later Kubernetes sysext OTA never refreshes the plugins there — the node runs the old CNI binaries forever while /usr/local/share/kubernetes-cni-version advertises the new version. Consider overwriting everything except the Cilium-installed binary, or clearing stale upstream plugins before the copy.

docs-currency

  • docs/skills/kubernetes-sysext-ops.md troubleshooting is wrong in three places at head:
    • line 137: kubectl -n argocd logs deploy/gitd — the git-daemon is deploy/bluefin-git-daemon in namespace bluefin-system (files/cluster/seed/15-gitd/deployment.yaml:21-23).
    • line 104: kubectl -n argocd describe application root — the root Application is named bluefin-cluster (files/cluster/seed/20-root-app/application.yaml:15).
    • lines 140-141: "ordered After=/Requires= kubeadm-init.service" — the unit deliberately uses Wants=, and its comment block explains at length why Requires= was removed (bluefin-cluster-bootstrap.service:10-25). The runbook restates the design this PR reverted.

intent-alignment

  • The PR body's security table checks out against the diff (auth-secret.yaml Job, postgres-secret.yaml Job, cert-manager kiosk TLS, DEV_MODE absent from console.yaml, no hostIP pin). Two additions are not mentioned anywhere in the body: the provenance sidecar machinery (_record-provenance/_inputs-hash, Justfile:133-160) plus the new test-installer-boot recipe (Justfile:~305-407), and the flash-installer running-disk/mounted-partition guards and read-back verification (Justfile:~230-300). All look sound, but they are a sizable undeclared scope on top of an already 100-file PR.

style

  • files/cluster/manifests/console/console.yaml:120-124 — the console container omits runAsNonRoot, readOnlyRootFilesystem, and a pod seccompProfile, which every sibling workload in this tree sets (kiosk deployment.yaml:22-27, gitd deployment.yaml:49-54, both secret Jobs). If the image requires root, a comment saying so would stop the next hardening pass from breaking it; otherwise it looks like an oversight.
  • files/cluster/manifests/kiosk/assets/nginx.conf:41 hardcodes resolver 10.96.0.10, silently coupled to serviceSubnet: 10.96.0.0/12 in files/kubernetes/kubeadm.yaml:38. kubeadm.yaml cross-references its Cilium coupling but neither file mentions this one.

security — no exploitable findings; the cutover removes the shared JWT secret, committed postgres password, and baked-in TLS key as the body claims. One note: the kiosk certificate's SANs are only localhost/127.0.0.1/the Service name (files/cluster/manifests/kiosk/certificate.yaml:27-31), while the kiosk now publishes on every interface and the issue banner advertises https://\4:8080 — LAN clients get a name-mismatch on top of the untrusted CA. Informational; the runtime IP can't be known at manifest time.

No further findings from: security.

— hive: agent=reviewer backend=copilot model=claude-fable-5

…on update

bluefin-cluster-bootstrap.service declares Requires=bluefin-cluster-repo.service,
and a job killed with result 'dependency' never runs, so the dependent's own
Restart=on-failure never engages. The repo unit had no Restart=, so one
transient publish failure stranded the seed permanently — the same failure mode
the Wants= edge on kubeadm-init.service exists to prevent. Give the repo unit
Restart=on-failure/RestartSec=30s; the publish is idempotent.

/opt/cni/bin lives on /var and outlives a sysext update, so the unconditional
`cp -an` bridge pinned the node to the plugins it first booted with while
/usr/local/share/kubernetes-cni-version advertised newer ones. Key the copy on
that stamp: refresh when it changes, no-clobber when it does not. Cilium is
safe either way — cilium-cni is not a name the sysext ships.

Documentation and hardening, from the same review:

- The ops runbook named deploy/gitd in argocd; the deployment is
  bluefin-git-daemon in bluefin-system. It also named Application 'root', which
  is 'bluefin-cluster', and restated the reverted Requires= design for the
  bootstrap unit.
- console.yaml now takes the pod hardening baseline its siblings set
  (runAsNonRoot with the image's numeric uid 1001, RuntimeDefault seccomp) and
  documents why readOnlyRootFilesystem is a deliberate exception: the image
  writes /app/data/console.db on its own root filesystem.
- Record the kiosk resolver's coupling to serviceSubnet in both files, and the
  known LAN SAN gap on the kiosk certificate.

Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security

  • The containerd/runc sysext ships with no update channel (medium). elements/flatcar/containerd-sysext.bst pins Flatcar's runtime image (containerd 2.1.5, runc 1.3.3) and the installer seeds it to /var/lib/extensions/containerd-flatcar.raw (files/installer/repart.d/30-var.conf:39), but unlike kubernetes.raw there is no sysupdate transfer for it — the element's own comment concedes "containerd has no such transfer" (elements/flatcar/containerd-sysext.bst:57). The ID=_any repack deliberately makes the stale image keep merging across OS updates, so an installed fleet runs this exact runc/containerd forever unless reinstalled. Runtime CVEs (runc escapes) are the canonical reason a container runtime needs a patch path; the previous k0s bundle got its embedded runtime replaced by the k0s-@v OTA. A sysupdate.containerd.d transfer (or at minimum naming this as a tracked open item in docs/superpowers/specs/2026-09-19-kubernetes-cutover-design.md, whose open-items list omits it) would close this.

intent-alignment

  • PR body says the provenance sidecar is .provenance.json; the diff writes <image>.provenance, plain text (info). _record-provenance writes two commit/tree lines to "${IMG}.provenance" (Justfile, _record-provenance recipe), not JSON. Consumers in test-installer-boot parse the plain-text form, so nothing is broken — only the body's claim is wrong.

style

  • test-installer-boot still picks an installer image with head -n1 (low) while flash-installer in the same file now refuses to guess when dist/ holds more than one image. The boot gate can silently test the alphabetically-first of two exports; the provenance warning mitigates but the stricter guard already exists a few hundred lines up.

No findings from: correctness, docs-currency. (Correctness: units, ordering rationale, depmod -b /usr fix, ZFS drop-in + tmpfiles strip, and the flash read-back were checked against the tree at c523bbb and hold up; test coverage for the new units is extensive. Docs: k0s skills removed, kubernetes skills/ops added, README/SECURITY/AGENTS/NOTES all moved with the behavior.)

— hive: agent=reviewer backend=copilot model=claude-fable-5

… patch gap

test-installer-boot selected its image with `head -n1` while flash-installer
already refuses to guess between two exports. A boot result is evidence about
one named artifact, so picking the alphabetically-first of a stale and a fresh
export is the one thing this recipe must not do. It now fails the same way
flash-installer does, listing both candidates.

The containerd sysext has no sysupdate transfer, and the ID=_any repack means
the seeded image keeps merging across OS updates instead of being rejected, so
an installed host holds that containerd/runc until it is reinstalled. That is a
patch-path regression against k0s, whose embedded runtime rode the k0s-@v
transfer. Record it as an open item, including why it cannot be closed by
pointing a transfer at Flatcar's published image: that image declares
ID=flatcar with a concrete VERSION_ID, the exact metadata this element repacks
away, so fetching it unmodified would make systemd-sysext refuse the merge and
drop containerd.service on the next boot. The transfer needs a repacked image
on this repo's release feed first.

Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • (low) files/os/sysupdate.kubernetes.d/70-kubernetes.transfer targets Path=/var/lib/extensions with MatchPattern=kubernetes-@v.raw — so after the first OTA update the versioned instances (kubernetes-1.36.5.raw, up to InstancesMax) sit directly in systemd-sysext's scan directory alongside the kubernetes.raw CurrentSymlink. Each versioned file is discovered as its own image named kubernetes-1.36.5, which lacks extension-release.kubernetes-1.36.5, so sysext dissects and then ignores it on every merge (verified against systemd main: src/shared/discover-image.c image discovery keys on dirent name, src/basic/os-util.c path_extract_image_name truncates only on _/+, and the mis-named-release fallback requires a user.extension-release.strict=false xattr the EROFS image doesn't carry; the refused image is counted in n_ignored and skipped, not fatal — src/sysext/sysext.c merge_subprocess). Net effect today is only per-boot "ignored incompatible image" noise and stale instances accumulating in the scan dir, but it silently depends on that skip behavior — if the image ever gained the relax xattr, both names would merge as duplicates. Flatcar's bakery transfers avoid this exact situation by keeping versioned instances in /opt/extensions/<name>/ and pointing CurrentSymlink into /etc/extensions/ (the transfer's own comment block at files/os/sysupdate.kubernetes.d/70-kubernetes.transfer:1-11 doesn't mention this trade-off). Consider the split layout, or document why direct-to-scan-dir is preferred.

Verified against the diff + surrounding tree at 36dcf4b: the boot-fix claims in the body all match the code (depmod -b /usr at elements/oci/bluefin-server-ddi.bst:41 and bluefin-server-installer.bst:113/380; homed-firstboot mask at bluefin-server-ddi.bst:53; ZFS drop-in + tmpfiles strip at elements/flatcar/flatcar-zfs.bst:31-39; SSH-forwarded smoke gate, flash guards, provenance sidecar in Justfile), and the four k0s-era secret/exposure defects the body claims fixed are actually fixed in the new manifests (in-cluster JWT and postgres generation, cert-manager-issued kiosk TLS, no committed key material).

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar

Copy link
Copy Markdown
Contributor

My approval on this PR was at 4da8f9f. Since then there are eight new commits and 481 added lines, and main has moved underneath it, so I want to flag the structural change rather than let that old approval stand unqualified.

The boot-fix half of this PR is now on main. #220 merged as cdb1683 on 2026-09-21 and #227 as f9bdef4 on 2026-09-22. Between them they land the Flatcar ZFS udevd drop-in removal, the depmod -b fix, the systemd-homed-firstboot.service mask and the in-guest console readiness probe, which are commits 1 through 17 of this PR.

That is why this is now CONFLICTING. A local merge into main at dc48bfb leaves nine conflicted files, and they are the #220/#227 files:

Justfile
elements/flatcar/flatcar-zfs.bst
elements/oci/bluefin-server-ddi.bst
files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml
tests/unit/test_installer_contract.py
elements/bluefin-server/os-k0s-first-boot.bst
files/os/issue.d/40-kubestellar.issue
tests/unit/flash-installer_test.bats
tests/unit/test_console_issue.py

I think the rebase here is mostly deletion rather than conflict resolution: drop the boot-fix commits main now carries, and keep the Kubernetes cutover. That would also shrink this from 145 files to something reviewable as one change.

Main's installer-test is green at f9bdef4 (run 35686916034), so the gate this PR was carrying is already open.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes mainly on scope; the earlier approval predates eight new commits and the branch now conflicts with main.

  1. Please split and rebase. The boot fixes (commits 1-17) have effectively landed via #220, #227 and #243. What remains is the k0s -> kubeadm cutover, which is breaking for hosts tracking k0s-@v.raw.zst and deserves its own review.
  2. Security posture change. files/cluster/manifests/kiosk/deployment.yaml drops hostIP: 127.0.0.1 and publishes the console on hostPort: 8080 on every interface. The issue banner and docs still say "loopback, TLS only". If this is intended, it needs an explicit maintainer decision plus matching doc/banner updates. Cilium envoy metrics are also on 0.0.0.0:9964 (hostNetwork).
  3. files/cluster/seed/15-gitd/deployment.yaml runs git-daemon with --export-all --listen=0.0.0.0 and safe.directory=*. It's read-only and not exposed via a Service, but a NetworkPolicy restricting it to Argo's repo-server would be prudent.

On the plus side: images are digest-pinned, and the committed console JWT / postgres password are replaced with in-cluster generation. Nice.

@castrojo

Copy link
Copy Markdown
Contributor Author

Status: blocked on maintainer decisions, not a mechanical fix. hanthor's CR asks (1) to split out the k0s -> kubeadm cutover now that the boot fixes (commits 1-17) have landed via #220/#227/#243, and (2) for an explicit maintainer decision on the console moving from hostIP: 127.0.0.1 to hostPort: 8080 on every interface (plus Cilium envoy metrics on 0.0.0.0:9964), with banner/docs to match. The branch also conflicts with main in the #220/#227 files. The split and rebase follow once the exposure question is decided; the gitd NetworkPolicy (point 3) belongs with the cutover PR.

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 049e9a2 (diff plus surrounding tree).

correctness

  • files/os/systemd/system/kubeadm-init.service:7 uses ConditionPathExists=!/etc/kubernetes/admin.conf as the done-marker, but kubeadm writes admin.conf in the kubeconfig phase, before etcd/control-plane/wait-control-plane/upload-config/mark-control-plane/bootstrap-token. If kubeadm init (:99) fails after that point (the common "timed out waiting for the condition"), Restart=on-failure (:67) re-evaluates the condition, the unit is skipped, and init never completes: no node-role.kubernetes.io/control-plane label, so files/cluster/seed/15-gitd/deployment.yaml nodeSelector never matches and bluefin-cluster-bootstrap.service phase 3 times out with an API server that answers. The convergence the bootstrap unit relies on (bluefin-cluster-bootstrap.service:22-24) only holds for pre-kubeconfig failures. Suggest a success stamp written by ExecStartPost as the condition (and kubeadm reset -f before a retry when the stamp is absent but /etc/kubernetes/manifests exists).

intent-alignment

  • PR body: hosts on k0s-@v.raw.zst "must be reinstalled or re-seeded with kubernetes-@v.raw.zst". Re-seeding is not sufficient in this diff: kubeadm-init.service waits on containerd (:98), containerd only arrives via files/installer/repart.d/30-var.conf:41 at install time, .github/workflows/build.yml:105 publishes only kubernetes-*.raw.zst, and there is no sysupdate.containerd.d transfer (acknowledged as open item 3 in docs/superpowers/specs/2026-09-19-kubernetes-cutover-design.md). An upgraded k0s host that pulls the new DDI gets kubelet/kubeadm units with no runtime. Body/docs/skills/kubernetes-sysext-ops.md should say reinstall only, or ship containerd on a transfer.
  • Closes #159 is stale: #159 (E2E script swallowing failures) was already fixed on main (scripts/lima-e2e-kubestellar-test.sh:47 has no || fallback there) and closed 2026-09-20; this PR only flips that line to https.
  • Unrelated changes ride along: README.md:1-5 logo <picture> block; files/os/systemd/system-preset/zz-enable-sshd.preset → zz-disable-sshd.preset rename.

docs-currency

  • files/cluster/manifests/kubelet-csr-approver/deployment.yaml:47-51 says MAX_EXPIRATION_SEC=2592000 bounds serving certs to 30 days. The approver only denies CSRs whose spec.expirationSeconds exceeds the cap; kubelet's serving CSR sets none, so the signer default (kube-controller-manager --cluster-signing-duration, 1y) applies and the setting is inert. Comment overstates the control.

No findings from: security, style.

Confidence: 3/5 (needs attention) — 2 medium findings

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

…conf

kubeadm writes /etc/kubernetes/admin.conf in the `kubeconfig` phase, which
runs before etcd, control-plane, wait-control-plane, mark-control-plane and
bootstrap-token. Guarding kubeadm-init.service on that path meant that an
init which died after the kubeconfig phase left the node permanently
half-initialised: Restart=on-failure re-evaluated
ConditionPathExists=!/etc/kubernetes/admin.conf, the unit was marked skipped
instead of retried, and mark-control-plane never applied the control-plane
role label. The symptom surfaced far downstream — the nodeSelector in
files/cluster/seed/15-gitd/deployment.yaml never matched and
bluefin-cluster-bootstrap phase 3 timed out while the API server answered
normally.

Guard on /var/lib/bluefin/kubeadm-init.stamp instead, written by
ExecStartPost, which only runs when every ExecStart succeeded. A retry that
finds admin.conf without the stamp now runs `kubeadm reset -f` first, since
kubeadm refuses to init over a populated /etc/kubernetes/manifests and every
retry would otherwise fail identically.

Also:

- Correct the MAX_EXPIRATION_SEC comment in the kubelet-csr-approver
  manifest. The cap only denies CSRs that request a longer
  spec.expirationSeconds; the kubelet sets none on its serving CSR, so those
  certificates are issued for kube-controller-manager's
  --cluster-signing-duration regardless of this value.
- State in docs/skills/kubernetes-sysext-ops.md that the Kubernetes cutover
  is reinstall-only. containerd-flatcar.raw is seeded at install time by
  files/installer/repart.d/30-var.conf, is not a published release asset and
  has no sysupdate transfer, so an existing k0s host cannot be migrated in
  place with `systemd-sysupdate --component=kubernetes update`.
- Drop two unrelated cosmetic changes from the diff: the README logo block
  and the zz-enable-sshd.preset rename.

Hive-Fix-Attempt: 1/6

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Status note: #262 (merged) replaced the Flatcar base with an FSDK-composed OS and added an opt-in kubeadm sysext (kubelet, kubeadm, kubectl, crictl, containerd, runc, CNI; elements/kubeadm/, oci/kubeadm-sysext.bst) next to k0s. It has been tested joining a real kubeadm v1.34 cluster (Cilium VXLAN, iSCSI via democratic-csi). Much of this PR's cutover and the Flatcar boot fixes are now covered or moot; the k0s-removal part is still a separate decision. Rebasing this would mean starting over from main.

@castrojo

Copy link
Copy Markdown
Contributor Author

Superseded by PR #262. The boot deadlocks on the Flatcar base were retired with the migration to FSDK, and the opt-in kubeadm sysext along with Cilium/Kubernetes kernel networking options are now native in main.

@castrojo castrojo closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/scanner Filed or owned by the scanner agent.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants