Repository navigation
Conversation
The `installer-test / Install and boot in QEMU` gate has never passed on main. Two independent defects make it unreachable by construction. Transport: `41-kubestellar-kiosk-proxy.yaml` publishes hostPort 8080 on `hostIP: 127.0.0.1`, a deliberate hardening from #75, but a QEMU hostfwd with an unspecified guest address is delivered to the guest's DHCP address, not to guest loopback. Nothing listens there, so `hostfwd=tcp:127.0.0.1:8080-:8080` can never be served. Protocol: `files/k0s/kiosk/nginx.conf` has served the proxy as `listen 8080 ssl` since c752c48, while the recipe probed it over plain `http://` with `curl --fail`, which turns the 497 -> 302 redirect into an error and never yields the expected JSON. Keep the loopback binding and move the probe into the guest: inject a test-only readiness unit through the same SMBIOS systemd-credential channel the recipe already uses for `fstab.extra`, pull it in with `systemd.wants=`, and have it poll `https://127.0.0.1:8080` with `--insecure`, matching the self-signed TLS handling in tests/e2e, then announce the verdict on the system console that serial.log already captures. The dead hostfwd is removed. Necessary but not sufficient: the gate still needs #173, without which the target boots into emergency mode waiting for /dev/disk/by-partlabel/var and never applies any manifests. Assisted-by: Claude Opus 4.6 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The first CI run of this branch (35374112337) reached the new probe but added a boot-time symptom that no `main` run shows: "Timed out waiting for device /dev/ttyS0". Runs 34905862943, 34910341724, 34914418293 and 34920225526 only ever time out on /dev/disk/by-partlabel/var. `StandardOutput=journal+console` makes the unit console-touching, which pulls extra console ordering into a transaction that is already stuck in emergency mode. Nothing about the beacon needs journald forwarding, so echo the two lines to /dev/console directly and drop the setting. Same output in serial.log, no console machinery attached to the unit. Verified locally: the generated unit passes systemd-analyze verify, prints both lines when a self-signed TLS fixture answers, and keeps polling without emitting the marker when nothing is listening. Assisted-by: Claude Opus 4.6 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Follow-up in 87c381d after watching the first CI run (35374112337). That run confirmed the plumbing works — QEMU accepted both new
One thing that run did surface: an extra Re-verified locally: |
|
Second run in: 35380863738. Correcting myself: the
87c381d stays because it is simpler on its own terms, not because of the ttyS0 line. Net: this PR is necessary but not sufficient. #173 has to land before |
|
Runtime evidence: I built a throwaway integration of #173 + #181 + #186 and ran it through CI. Run 35388029502 is GREEN — Decisive serial console lines: and host-side: Three independent blockers, all required; any two still fail:
The marker is not a false positive. At t=38s and t=69s the loop was awake with k0s active and 10 pods present, but The deadline is not a problem. Readiness is reached at t=89s against a 900s budget — roughly 10x headroom. No change to Caveat, stated plainly: the run exercised the The integration branch was a throwaway: no temporary PR was opened, and the branch and worktree are deleted. Neither #173's nor #181's diff or head was touched by any of it. |
|
Thanks for the review. No findings were raised, so no code changes were needed. |
|
Closing as superseded by #210. Two reasons:
Your diagnosis of why |
Why
installer-test / Install and boot in QEMUhas never passed onmain. It wasintroduced in 50be1ed (#149); every
mainrun since has failed(34905862943, 34910341724, 34914418293, 34920225526). The last green
mainrun,34897477278, predates the job.
The job always dies in its last phase:
That is not flakiness. The readiness gate is unreachable by construction, for
two independent reasons.
Defect 1 — transport
Justfileforwarded the console with-nic user,...,hostfwd=tcp:127.0.0.1:8080-:8080. When the guest side of a libslirphostfwd is unspecified, libslirp substitutes the guest's DHCP address (10.0.2.15
by default), so the forward is delivered to
10.0.2.15:8080inside the guest.But
files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yamlpublishes theproxy with
hostIP: 127.0.0.1/hostPort: 8080— guest loopback only. Nothinglistens on
10.0.2.15:8080, so the forwarded connection can never be served.The loopback binding is deliberate: it came from e1ea72a "[sec-check] fix: bind
kubestellar-kiosk-proxy hostPort to 127.0.0.1 (#75)". This PR does not touch it.
The comment next to it, however, asserted "The VM flow reaches it via the existing
hostfwd of 127.0.0.1:8080", which is exactly the false premise — that comment is
corrected here.
Defect 2 — protocol
files/k0s/kiosk/nginx.confhas served the proxy aslisten 8080 sslwitherror_page 497 https://$http_host$request_uri;since c752c48 moved the kiosk toHTTPS with a self-signed certificate (baked into the k0s sysext by
elements/oci/k0s-sysext.bst). The recipe nevertheless probed it with plainhttp://127.0.0.1:8080/healthzundercurl --fail, so the 497 → 302 redirectbecame an error and the expected JSON never arrived.
Reproduced locally against a self-signed TLS listener:
What changed
Any host-side network probe of a loopback-bound guest service is impossible: slirp
can only deliver to an address the guest has actually bound, and the whole point of
#75 is that the console is bound to none of them. So the probe moves into the
guest rather than the binding moving out of loopback.
Justfile(test-installer-artifact)hostfwd=tcp:127.0.0.1:8080-:8080clause; it is provably dead.bluefin-kiosk-ready.servicethroughio.systemd.credential.binary:systemd.extra-unit.*— the same SMBIOSsystemd-credential channel the recipe already uses for
fstab.extra— and pullsit in with
systemd.wants=appended to the existingio.systemd.stub.kernel-cmdline-extrastring.https://127.0.0.1:8080/healthzandhttps://127.0.0.1:8080/with
--insecure, keeping the original assertions (.status == "ok", rootreturns 200), and echoes
KIOSK_CONSOLE_READYto the system console withStandardOutput=journal+console.serial.log, which the reusableworkflow already uploads as failure evidence.
41-kubestellar-kiosk-proxy.yaml— comment only.hostIP: 127.0.0.1andhostPort: 8080are unchanged; [sec-check] fix: bind kubestellar-kiosk-proxy hostPort to 127.0.0.1 (files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml) #75's hardening is fully preserved and the consoleis still published on no external interface at all.
tests/unit/test_installer_contract.py— contract test that fails onmain'sJustfile and passes here, so neither defect can silently return.
--insecurefollows the precedent set by b6a6881 "fix(e2e): support self-signed TLScertificates in readiness", which handles the same certificate in
tests/e2e/test_kubestellar_browser_login.pywithctx.check_hostname = False/ssl.CERT_NONEand--ignore-certificate-errors.Why not the alternatives
0.0.0.0/ the DHCP address — reverts [sec-check] fix: bind kubestellar-kiosk-proxy hostPort to 127.0.0.1 (files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml) #75 and publishesthe console on every LAN interface. Rejected.
hostfwd=...-127.0.0.1:8080— slirp would inject packets addressed to127.0.0.1on the guest's virtio NIC; Linux drops them as martian unlessroute_localnetis enabled, which re-opens the same exposure by a side door.files/os/systemd/system-preset/zz-enable-sshd.presetdeliberately ships
disable sshd.service/disable sshd.socket, and root has nopassword (
files/os/sysusers.d/10-root-creds.confsets none and the recipe maskssystemd-firstboot.service). Turning SSH on and forwarding it is the same class ofexposure [sec-check] fix: bind kubestellar-kiosk-proxy hostPort to 127.0.0.1 (files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml) #75 closed, for more moving parts.
consumer is CI. The credential injection keeps it entirely test-side.
Verification
Local (the gate itself needs the full DDI build, so it cannot run here):
just validate→ exit 0.python3 -m pytest tests/unit -q→208 passed, 1 xfailed.bats tests/unit→ 62 passed.systemd-analyze verifyon the generated unit → clean.KIOSK_CONSOLE_READY;%%{http_code}and the nested quoting survive systemd'sExecStart parser.
qemu-system-x86_64accepts both-smbios type=11strings (the base64 unit is 800chars; parsing is fine well past 8000).
Necessary but not sufficient
This PR alone will not turn
installer-testgreen. The same run log shows thebox never reaches userspace:
k0s therefore never starts and no manifest is ever applied. #173 (the
/varfix)must land as well. With both, the console beacon has a chance to run; until then
the gate will still time out, now with the probe correctly wired instead of aimed at
an address nothing serves.
Known cosmetic delta, and a correction
Both CI runs on this branch (35374112337, 35380863738) fail exactly where predicted,
with the probe correctly wired:
local-fs.targetfails,emergency.targetis isolated, and every job in the defaulttransaction — including
k0s-first-boot.serviceand therefore the readiness beacon —is cancelled. Nothing is ever applied to the cluster. That is #173's bug.
The serial log on this branch also carries one line
maindoes not:I first attributed this to
StandardOutput=journal+consoleand dropped it in 87c381din favour of writing to
/dev/consoledirectly. That diagnosis was wrong — theline survives the change. The accurate reading, from the full serial logs: both runs
are the same doomed boot, and
dev-ttyS0.deviceanddev-disk-by\x2dpartlabel-var.deviceare both 90s device waits racing to time outfirst. On
main/varwins, trips emergency, and cancels the getty job before itreports; here the getty job reports first and
/varfollows.serial-getty@ttyS0.serviceis the interactive login prompt on a serial port QEMU points at a write-only file — it
has no relationship to the beacon and no bearing on the gate. Any in-guest probe adds
a job to that transaction, so this delta is not specific to how the beacon is pulled in.
87c381d is kept on its own merits: writing to
/dev/consoleneeds no journaldforwarding and attaches no console semantics to the unit.