Repository navigation
Conversation
hanthor
left a comment
There was a problem hiding this comment.
The vulnerability fix is right. The "persistent" half does not hold.
Removing the build-time openssl req is correct and worth doing — a private key baked into a published sysext is the same key on every install, and that is a real finding. No argument with the direction.
But the certificate this PR generates is regenerated on every boot, not persisted, because nothing removed the tmpfiles rule that owns the destination directory.
The collision
files/k0s/sysext/k0s-manifests.conf is unchanged by this PR and still contains:
C+ /var/lib/k0s/kiosk - - - - /usr/share/k0s/kiosk
The new unit writes into that exact path:
ExecStart=/bin/bash -eu -c 'test -s /var/lib/k0s/kiosk/key.pem && test -s /var/lib/k0s/kiosk/cert.pem && exit 0; mkdir -p /var/lib/k0s/kiosk; ... -keyout /var/lib/k0s/kiosk/key.pem -out /var/lib/k0s/kiosk/cert.pem ...'
In tmpfiles.d, the + suffix on C means remove the destination first, then copy. So on every boot:
systemd-tmpfiles-setup.serviceruns (ordered early, beforesysinit.target) and deletes/var/lib/k0s/kiosk/wholesale, re-copying it from/usr/share/k0s/kiosk— which, after this PR, no longer containscert.pemorkey.pem.k0s-kiosk-tls.serviceruns much later (After=network-online.target, well pastsysinit.target), finds no key, and itstest -s … && exit 0fast path falls through.- A brand-new self-signed cert is minted.
The unit is named "Generate persistent TLS certificate and key", the docs added here say the key "is stored persistently in /var/lib/k0s/kiosk/key.pem", and the idempotence guard in item 4 of your summary is written to skip regeneration — none of those are true as long as that C+ line points at the same directory. Operators get a new certificate fingerprint on every reboot, so any pinned or hand-trusted cert breaks and the browser warning returns each time.
Note this is new with this PR: before it, cert.pem/key.pem shipped inside /usr/share/k0s/kiosk, so the C+ re-copy reproduced the same (shared, baked-in) cert each boot. That was the vulnerability — but it was at least stable. This PR fixes the sharing and trades it for churn.
Suggested fix
Cleanest is to stop overlapping the two owners. Generate into a directory tmpfiles does not manage, e.g. /var/lib/kubestellar-kiosk/, and point the proxy's hostPath at it:
# files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml
path: /var/lib/kubestellar-kiosk(nginx.conf needs no change — it reads /etc/kubestellar-kiosk/cert.pem, which is the mount point, not the host path.) Dropping the + from the kiosk line would also stop the deletion, but then sysext upgrades stop propagating new kiosk assets, which is presumably why + is there.
Whichever way you go, please add ordering — After=systemd-tmpfiles-setup.service — so the two can never race, and a test asserting the generated path is not under a C+ destination.
Smaller points
ip/hostnamemay be absent. The SAN loop is guarded bycommand -v, so ifipis not in the OS stack the cert silently getsDNS:localhost,DNS:*.local,IP:127.0.0.1and no host IP — the console is then unreachable by address without a cert error, with nothing in the log saying why.opensslwas correctly added toos-stack.bst; consider assertingipis there too, or logging when the lookup yields nothing.After=network-online.targetdelaysk0scontroller.service(viaBefore=) purely to enumerate addresses. That is a real boot-time cost on a host with a slow link, for a cert that this PR intends to generate once.- First-boot SAN is frozen for 3650 days. On DHCP the address can move and the cert does not follow. Worth documenting in the runbook next to the rotation steps you added.
CI status
No checks have run on this PR at all:
mcp__github__pull_request_read(method="get_check_runs", pullNumber=91)
-> {"total_count":0,"check_runs":[]}
mergeable_state is unstable. It is a fork PR (Danathar/server) and appears to need maintainer approval to run workflows. The build job is the only thing that would exercise k0s-sysext.bst after the openssl build-depend was removed, so that removal is currently unverified by CI. I ran the local gates on main + this branch and they pass — 204 passed, 1 xfailed, 41/41 bats, all three checkers exit 0 — against a main baseline of 201 passed, 1 xfailed, so the three new tests are additive and nothing regressed. But the local suite cannot build the sysext.
Merge-order note
This and #90 both edit docs/skills/k0s-sysext-ops.md and tests/unit/test_kubestellar_kiosk.py. Each merges cleanly into main on its own, but they conflict with each other:
$ git merge pr90 # onto main+pr91
CONFLICT (content): Merge conflict in docs/skills/k0s-sysext-ops.md
Whichever lands second needs a rebase.
Generated by Claude Code
…t.pem/key.pem hanthor's review on projectbluefin#91 argued that the single whole-directory `C+ /var/lib/k0s/kiosk - - - - /usr/share/k0s/kiosk` tmpfiles rule deletes the directory wholesale on every boot and re-copies it from the sysext image (which never contains cert.pem/key.pem), so the persistent-cert fix this PR makes would trade a shared baked-in cert for a fresh one every reboot. I tried to verify the failure mechanism before just taking the suggested fix on faith: ran `systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev` (the exact flags systemd-tmpfiles-setup.service uses) twice against a sandboxed root with the original single-line config, planting fake cert.pem/key.pem between runs to simulate a reboot with an already-generated cert. On this host's systemd 259 (Fedora 44, matching the target OS), they survive both runs -- `C+` does not delete pre-existing destination content; per man(5) tmpfiles.d (Arch Wiki mirror, upstream docs returned 403): plain `C` skips the whole copy if the destination directory already exists and is non-empty, and `C+` relaxes only that skip -- it still won't overwrite/delete files that already exist at the destination. So the specific "deletes wholesale and re-copies" mechanism in the review does not reproduce here. That said, the suggested fix is still worth making regardless of the exact mechanism: relying on an undocumented-by-the-obvious-docs, version-dependent "C+ happens to leave extra files alone" behavior as the only thing standing between a private key and deletion is fragile for something this sensitive, and hanthor asked for the ordering guarantee either way ("whichever way you go, please add After=systemd-tmpfiles-setup.service"). Changed the rule from one whole-directory C+ to three per-file C+ lines (nginx.conf, kiosk-gate.js, kiosk-gate.css) so cert.pem/key.pem are structurally never named by any tmpfiles directive, added the After= ordering, and added logging when `hostname`/`ip` are unavailable (smaller point from the same review) plus a docs note on SAN staleness under DHCP. Added a regression test asserting no tmpfiles line targets the bare kiosk directory and that cert.pem/key.pem never appear in the config. Full suite: 205 passed, 1 xfailed (unchanged from main). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
|
@hanthor — pushed in I couldn't reproduce the "deletes wholesale every boot" claim. I ran Checked the actual semantics (upstream docs 403'd, used the Arch Wiki mirror): plain Made the fix anyway. Regardless of whether the exact mechanism reproduces, relying on an undocumented, version-dependent "
Full suite: 205 passed, 1 xfailed (unchanged). Left the |
The branch could not be refreshed onto current |
…sext build time Move KubeStellar kiosk TLS certificate and private key generation from build time in elements/oci/k0s-sysext.bst to a first-boot oneshot systemd service (files/k0s/sysext/k0s-kiosk-tls.service) ordered before k0scontroller.service. - Remove openssl req and build-depends from elements/oci/k0s-sysext.bst so no private key or static certificate is baked into the public sysext release asset. - Add files/k0s/sysext/k0s-kiosk-tls.service to generate key.pem (0600) and cert.pem (0644) under persistent /var/lib/k0s/kiosk/ on first boot. - Dynamically populate Subject Alternative Names (SAN) using localhost, *.local, loopback, hostname, and the host's actual global IP addresses. - Add Wants= and After= dependency on k0s-kiosk-tls.service to k0scontroller.service. - Add freedesktop-sdk.bst:components/openssl.bst to elements/bluefin-server/os-stack.bst. - Update documentation and unit test contracts in test_kubestellar_kiosk.py. Closes projectbluefin#87 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…t.pem/key.pem hanthor's review on projectbluefin#91 argued that the single whole-directory `C+ /var/lib/k0s/kiosk - - - - /usr/share/k0s/kiosk` tmpfiles rule deletes the directory wholesale on every boot and re-copies it from the sysext image (which never contains cert.pem/key.pem), so the persistent-cert fix this PR makes would trade a shared baked-in cert for a fresh one every reboot. I tried to verify the failure mechanism before just taking the suggested fix on faith: ran `systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev` (the exact flags systemd-tmpfiles-setup.service uses) twice against a sandboxed root with the original single-line config, planting fake cert.pem/key.pem between runs to simulate a reboot with an already-generated cert. On this host's systemd 259 (Fedora 44, matching the target OS), they survive both runs -- `C+` does not delete pre-existing destination content; per man(5) tmpfiles.d (Arch Wiki mirror, upstream docs returned 403): plain `C` skips the whole copy if the destination directory already exists and is non-empty, and `C+` relaxes only that skip -- it still won't overwrite/delete files that already exist at the destination. So the specific "deletes wholesale and re-copies" mechanism in the review does not reproduce here. That said, the suggested fix is still worth making regardless of the exact mechanism: relying on an undocumented-by-the-obvious-docs, version-dependent "C+ happens to leave extra files alone" behavior as the only thing standing between a private key and deletion is fragile for something this sensitive, and hanthor asked for the ordering guarantee either way ("whichever way you go, please add After=systemd-tmpfiles-setup.service"). Changed the rule from one whole-directory C+ to three per-file C+ lines (nginx.conf, kiosk-gate.js, kiosk-gate.css) so cert.pem/key.pem are structurally never named by any tmpfiles directive, added the After= ordering, and added logging when `hostname`/`ip` are unavailable (smaller point from the same review) plus a docs note on SAN staleness under DHCP. Added a regression test asserting no tmpfiles line targets the bare kiosk directory and that cert.pem/key.pem never appear in the config. Full suite: 205 passed, 1 xfailed (unchanged from main). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
6c2a436 to
814d878
Compare
Update: the second refresh onto |
castrojo
left a comment
There was a problem hiding this comment.
Reviewed. The blocking objection was answered in code two commits ago, so the CHANGES_REQUESTED here is stale. Holding my approval only until build reports, for a specific reason given below.
First: CI had never run on this PR at all. That is why it looked stalled. This is a fork PR from a contributor with read access, and build.yml triggers on plain pull_request while checking out the PR head, so every run sat at conclusion: action_required with zero jobs dispatched — indistinguishable from "no checks yet" in the UI. I have approved the runs. unit and docs now pass and build is in progress.
@hanthor's blocking objection is addressed. His concern was that the C+ /var/lib/k0s/kiosk whole-directory tmpfiles rule would wipe the generated material each boot, so the unit's test -s ... && exit 0 fast path would fall through and mint a fresh cert every boot — making the "persistent" claim in the unit description, the docs and the idempotence bullet false. The current head addresses all three things he asked for: the whole-directory rule is split into three per-file C+ lines for nginx.conf, kiosk-gate.js and kiosk-gate.css, so the generated TLS material is never named by tmpfiles; the unit gained After=systemd-tmpfiles-setup.service; and there is a test_tmpfiles_never_wipes_the_generated_tls_material test. I verified all three are present in the current diff. @hanthor, this is 2 commits newer than the head you reviewed — worth a fresh look.
The security direction is right: generating the cert and key at first boot rather than at sysext build time means machines stop sharing one baked-in key, which is the whole point.
Why I am waiting on build. This deletes the openssl build-depend from elements/oci/k0s-sysext.bst and moves generation into a first-boot unit. Removing a build-depend from a .bst element is exactly the class of change only the build job can validate — docs/skills/avoid-over-engineering.md documents the failure mode where a dropped dependency leaves the sandbox without a tool and a script/manual element fails in a confusing way. Since that job has literally never executed on this branch, approving now would be approving on no evidence for the one thing most likely to break. I will approve once it is green.
One live review item, not a stale one. files/k0s/sysext/k0s-kiosk-tls.service combines After=network-online.target with Before=k0scontroller.service, which delays Kubernetes startup behind network-online on every boot — including the overwhelmingly common case where the cert already exists and the unit does nothing but run two test -s. @hanthor raised it; the reply was that it is "a real tradeoff question... felt like a separate decision". I do not think it is separate, because the cost is paid on every boot forever while the benefit applies only to the first. That said, it is a judgement call about ordering semantics rather than a correctness bug, and it is the author's to make — I am flagging it, not blocking on it.
Two smaller things. [Install] WantedBy=k0scontroller.service multi-user.target in that unit is dead weight: the sysext ships no enablement symlink and k0scontroller.service already carries Wants=k0s-kiosk-tls.service. Harmless, but it makes a reader believe an enablement path exists. And the SAN is frozen for 3650 days at first boot, so on DHCP the address moves and the cert does not follow — correctly disclosed in the runbook rather than hidden, which I am fine with.
On the new tests. The added assertions are source-text greps (assert "keyout" not in sysext, exact tmpfiles line strings). That is the same pattern main recently deleted test_vm_dashboard_contract.py for. It is consistent with the rest of that file so I am not making it a blocker here, but it is worth not extending further — a test that greps the element text cannot catch the build-sandbox breakage that build is currently checking for.
No security defect found in the shell: $$(hostname) and awk "{print $$4}" are systemd's literal-dollar escape, so bash receives $(hostname) and awk receives {print $4} as intended.
Conflicts to sequence, not to fix now. This and #90 both touch docs/skills/k0s-sysext-ops.md and tests/unit/test_kubestellar_kiosk.py; each merges into main alone but not into the other, so whichever lands second rebases. This and #80 both edit elements/bluefin-server/os-stack.bst about three lines apart — probably auto-merges, worth eyeballing.
Not approving yet, purely pending build. Everything else here is ready.
The k0s sysext ships no enablement symlink for this unit and no preset enables it, so the [Install] section is never acted upon. The unit is pulled in by k0scontroller.service, which already carries Wants=k0s-kiosk-tls.service (plus a matching After= ordering). Keeping the section only misleads a reader into believing an enablement path exists. Removing it is a no-op at runtime. Assisted-by: Claude Opus 4.6 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
intent-alignment
- Host IPs never make it into the SAN —
files/k0s/sysext/k0s-kiosk-tls.service:11: after systemd unescapes$$→$, the ExecStart line containsawk "{print $4}". Because the awk program is in double quotes, bash (running-eu) expands$4itself; the positional parameter is unset, the command substitution subshell dies with$4: unbound variable, the error is silently swallowed, and thefor addr in ...loop iterates zero times. I reproduced this: with a stubbedipon PATH the script printsbash: $4: unbound variableand producesSAN=DNS:localhost,DNS:*.local,IP:127.0.0.1,DNS:<hostname>— no IPs. This contradicts PR body claim #3 ("all global IP addresses discovered on the host") and the new operator docs (docs/skills/k0s-sysext-ops.md:51, "configured with the host's actual IP addresses"). Fix: single-quote the awk program (awk '{print $$4}') or usecut/parameter expansion instead. Note the new contract test (tests/unit/test_kubestellar_kiosk.py:139-155) only substring-matches the unit text, so it passes despite the bug — worth a behavioral check of the script if feasible.
The core security move is sound and verified: no key material remains in the sysext build (elements/oci/k0s-sysext.bst build-time openssl req removed), the per-file tmpfiles C+ rules (files/k0s/sysext/k0s-manifests.conf:9-12) correctly avoid wiping generated pems, the kiosk proxy manifest already mounts /var/lib/k0s/kiosk via hostPath so consumption is unchanged, and OpenSSL 3 writes key.pem 0600 at creation so there is no pre-chmod exposure window.
No findings from: security, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
The unit's ExecStart script is a single-quoted `bash -eu -c` program, so the awk program had to be double-quoted, and after systemd unescaped `$$` to `$` bash expanded `$4` itself. Under `-u` that is an unbound variable, the command substitution died, and the address loop ran zero times: the certificate's SAN never contained a host IP even though the PR body and operator docs said it would. The contract test only substring-matched the unit text, so it passed. Use `tr -s " " | cut -d " " -f4` for the field instead, which needs no `$` at all, and add two behavioral tests that run the exact ExecStart program (with systemd's `$$` rewrite applied) against a fake state directory and stubbed `ip`, `hostname`, and `openssl`: one asserts the IPv4 and IPv6 addresses and the hostname land in `subjectAltName` with the expected modes, the other that an existing key and cert are left alone. The first test fails on the old unit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VXwvRu29c2MkwftzkgAHcC
|
Pushed What changed:
@castrojo this will have dismissed nothing of yours since you had not approved yet; the |
|
On |
There was a problem hiding this comment.
security
- Upgraded hosts keep the publicly-leaked key forever (medium). Hosts already deployed from an earlier release have the build-time
key.pem/cert.pemcopied into/var/lib/k0s/kiosk/by the old directory-levelC+rule. The new unit's idempotence guard (test -s .../key.pem && test -s .../cert.pem && exit 0, files/k0s/sysext/k0s-kiosk-tls.service:11) sees those files and skips generation, so the compromised, publicly-published private key stays in service indefinitely. The runbook (docs/skills/k0s-sysext-ops.md:50-53) frames rotation as an optional DHCP-change response, not required remediation. Consider having the unit detect and replace the known-bad material (e.g., cert withIP:10.0.2.15in the SAN, or a fingerprint check), or at minimum make the docs say upgrades from prior releases MUST rotate. - Brief world-readable key window (low). The script runs under systemd's default umask 0022, so
openssl ... -keyoutcreateskey.pemas 0644 in the 0755 dir before the trailingchmod 0600(files/k0s/sysext/k0s-kiosk-tls.service:11). Prefix the script withumask 077;to close the window.
style
- The entire generator is a single ~1000-char quoted
ExecStart=line requiring$$escaping that the tests must un-escape to exercise (tests/unit/test_kubestellar_kiosk.py:214). The sysext already ships an executable script (generate-postgres-secret.sh, elements/oci/k0s-sysext.bst:91); shipping this as a script file would match that idiom and be directly testable. (info)
No findings from: correctness, intent-alignment, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
Address review findings on projectbluefin#91: - Replace the inline ~1000-char ExecStart mega-line with a shipped helper script (files/k0s/kiosk-tls/generate-kiosk-tls.sh), matching the generate-postgres-secret.sh idiom and making the generator directly testable without $$ un-escaping. - Run the generator under umask 077 so key.pem is never world-readable, even between openssl -keyout and the trailing chmod 0600. - Detect the compromised build-time certificate that earlier releases published inside the sysext image (fixed SAN with IP:10.0.2.15, no OU=host-generated marker) and regenerate it on upgraded hosts instead of keeping it forever via the idempotence guard. Host-generated and operator-provided certificates are never touched. - Document the automatic remediation in docs/skills/k0s-sysext-ops.md and extend the unit tests to cover keep/replace paths. Assisted-by: Claude Fable 5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
correctness
- (low) If
generate-kiosk-tls.shfails (e.g.openssl reqrejects a SAN built from an unusual hostname — the value fromhostnameis interpolated unsanitized into-addextat files/k0s/kiosk-tls/generate-kiosk-tls.sh:56),k0scontroller.servicestill starts because it onlyWants=k0s-kiosk-tls.service(files/k0s/sysext/k0scontroller.service:5-6), and the kiosk proxy pod then crashloops on missing/var/lib/k0s/kiosk/{cert,key}.pem(hostPath mount in files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml:44-46). Fail-closed is defensible for a security fix, but considerRequires=(or a documented failure mode) so the degradation is visible at the unit level rather than as a crashlooping pod.
Verified, for the record: the leaked-cert detection string in generate-kiosk-tls.sh:25 matches real openssl x509 -ext subjectAltName output after the sed 1d | tr -d '[:space:]' normalization (OpenSSL 3.x prints IP Address: which normalizes to IPAddress:), so upgraded hosts do get the compromised pair replaced.
No findings from: security, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Closing as superseded by #210. `elements/oci/k0s-sysext.bst` and `files/k0s/` are deleted. #210 fixes #87 by removing the build-time key entirely rather than regenerating it at first boot: kiosk TLS is issued by cert-manager at runtime (`files/cluster/manifests/pki/issuers.yaml` + `files/cluster/manifests/kiosk/certificate.yaml`). That is strictly better than a first-boot generator script — the key material is managed, rotatable, and never passes through a release asset. Your diagnosis of the leak was correct and it is what drove the cert-manager dependency. |
Security Fix
Addresses architecture finding in #87: the k0s sysext build previously generated a self-signed TLS certificate and private key (
key.pem) at build time and baked them directly into the published sysext image.Changes
openssl reqandfreedesktop-sdk.bst:components/openssl.bstbuild-depends fromelements/oci/k0s-sysext.bstso no private key or static certificate is baked into the public sysext release asset.files/k0s/sysext/k0s-kiosk-tls.service, a oneshot systemd unit packaged into the sysext that generateskey.pem(mode0600) andcert.pem(mode0644) under persistent/var/lib/k0s/kiosk/on first boot beforek0scontroller.servicestarts.DNS:localhost,DNS:*.local,IP:127.0.0.1, hostname, and all global IP addresses discovered on the host (ip -o addr show scope global).key.pemandcert.pemexist. Operators can rotate the certificate at any time by removing them from/var/lib/k0s/kiosk/and restarting the unit.After=network-online.target k0s-kiosk-tls.serviceandWants=network-online.target k0s-kiosk-tls.servicetofiles/k0s/sysext/k0scontroller.service.freedesktop-sdk.bst:components/openssl.bsttoelements/bluefin-server/os-stack.bstto provide/usr/bin/opensslat runtime.docs/skills/k0s-sysext-ops.mdand repository layout indocs/skills/k0s-sysext.md. Added unit test contracts intests/unit/test_kubestellar_kiosk.py.Closes #87
— hive: backend=agy