Skip to content

fix(ci): make the installer-test readiness gate reachable - #181

Closed
castrojo wants to merge 2 commits into
mainfrom
fix/installer-test-readiness-gate
Closed

castrojo wants to merge 2 commits into
mainfrom
fix/installer-test-readiness-gate

Conversation

@castrojo

@castrojo castrojo commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

installer-test / Install and boot in QEMU has never passed on main. It was
introduced in 50be1ed (#149); every main run since has failed
(34905862943, 34910341724, 34914418293, 34920225526). The last green main run,
34897477278, predates the job.

The job always dies in its last phase:

==> Polling KubeStellar Console readiness at http://127.0.0.1:8080 (deadline: 900s)...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!

That is not flakiness. The readiness gate is unreachable by construction, for
two independent reasons.

Defect 1 — transport

Justfile forwarded the console with
-nic user,...,hostfwd=tcp:127.0.0.1:8080-:8080. When the guest side of a libslirp
hostfwd is unspecified, libslirp substitutes the guest's DHCP address (10.0.2.15
by default), so the forward is delivered to 10.0.2.15:8080 inside the guest.

But files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml publishes the
proxy with hostIP: 127.0.0.1 / hostPort: 8080 — guest loopback only. Nothing
listens on 10.0.2.15:8080, so the forwarded connection can never be served.

The loopback binding is deliberate: it came from e1ea72a "[sec-check] fix: bind
kubestellar-kiosk-proxy hostPort to 127.0.0.1 (#75)". This PR does not touch it.
The comment next to it, however, asserted "The VM flow reaches it via the existing
hostfwd of 127.0.0.1:8080", which is exactly the false premise — that comment is
corrected here.

Defect 2 — protocol

files/k0s/kiosk/nginx.conf has served the proxy as listen 8080 ssl with
error_page 497 https://$http_host$request_uri; since c752c48 moved the kiosk to
HTTPS with a self-signed certificate (baked into the k0s sysext by
elements/oci/k0s-sysext.bst). The recipe nevertheless probed it with plain
http://127.0.0.1:8080/healthz under curl --fail, so the 497 → 302 redirect
became an error and the expected JSON never arrived.

Reproduced locally against a self-signed TLS listener:

=== OLD PROBE (plain HTTP against TLS listener, as on main) ===
healthz payload: []
OLD: NOT READY (gate would spin until deadline)
old curl exit=56
=== NEW PROBE (https + --insecure, e2e precedent) ===
status=[ok]
root_code=[200]
NEW: READY

What changed

Any host-side network probe of a loopback-bound guest service is impossible: slirp
can only deliver to an address the guest has actually bound, and the whole point of
#75 is that the console is bound to none of them. So the probe moves into the
guest rather than the binding moving out of loopback.

  • Justfile (test-installer-artifact)
    • Dropped the hostfwd=tcp:127.0.0.1:8080-:8080 clause; it is provably dead.
    • Injects a test-only bluefin-kiosk-ready.service through
      io.systemd.credential.binary:systemd.extra-unit.* — the same SMBIOS
      systemd-credential channel the recipe already uses for fstab.extra — and pulls
      it in with systemd.wants= appended to the existing
      io.systemd.stub.kernel-cmdline-extra string.
    • That unit polls https://127.0.0.1:8080/healthz and https://127.0.0.1:8080/
      with --insecure, keeping the original assertions (.status == "ok", root
      returns 200), and echoes KIOSK_CONSOLE_READY to the system console with
      StandardOutput=journal+console.
    • The host loop now waits for that marker in serial.log, which the reusable
      workflow already uploads as failure evidence.
  • 41-kubestellar-kiosk-proxy.yaml — comment only. hostIP: 127.0.0.1 and
    hostPort: 8080 are unchanged; [sec-check] fix: bind kubestellar-kiosk-proxy hostPort to 127.0.0.1 (files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml) #75's hardening is fully preserved and the console
    is still published on no external interface at all.
  • tests/unit/test_installer_contract.py — contract test that fails on main's
    Justfile and passes here, so neither defect can silently return.

--insecure follows the precedent set by b6a6881 "fix(e2e): support self-signed TLS
certificates in readiness", which handles the same certificate in
tests/e2e/test_kubestellar_browser_login.py with
ctx.check_hostname = False / ssl.CERT_NONE and --ignore-certificate-errors.

Why not the alternatives

Verification

Local (the gate itself needs the full DDI build, so it cannot run here):

  • just validate → exit 0.
  • python3 -m pytest tests/unit -q → 208 passed, 1 xfailed.
  • bats tests/unit → 62 passed.
  • systemd-analyze verify on the generated unit → clean.
  • The generated unit, run for real against a self-signed TLS fixture, prints
    KIOSK_CONSOLE_READY; %%{http_code} and the nested quoting survive systemd's
    ExecStart parser.
  • qemu-system-x86_64 accepts both -smbios type=11 strings (the base64 unit is 800
    chars; parsing is fine well past 8000).

Necessary but not sufficient

This PR alone will not turn installer-test green. The same run log shows the
box never reaches userspace:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.

k0s therefore never starts and no manifest is ever applied. #173 (the /var fix)
must land as well.
With both, the console beacon has a chance to run; until then
the gate will still time out, now with the probe correctly wired instead of aimed at
an address nothing serves.

Known cosmetic delta, and a correction

Both CI runs on this branch (35374112337, 35380863738) fail exactly where predicted,
with the probe correctly wired:

==> Polling KubeStellar Console readiness from inside the guest (deadline: 900s)...
[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for Activate k0s sysext on boot.
You are in emergency mode.

local-fs.target fails, emergency.target is isolated, and every job in the default
transaction — including k0s-first-boot.service and therefore the readiness beacon —
is cancelled. Nothing is ever applied to the cluster. That is #173's bug.

The serial log on this branch also carries one line main does not:

[ TIME ] Timed out waiting for device /dev/ttyS0.
[DEPEND] Dependency failed for Serial Getty on ttyS0.

I first attributed this to StandardOutput=journal+console and dropped it in 87c381d
in favour of writing to /dev/console directly. That diagnosis was wrong — the
line survives the change. The accurate reading, from the full serial logs: both runs
are the same doomed boot, and dev-ttyS0.device and
dev-disk-by\x2dpartlabel-var.device are both 90s device waits racing to time out
first. On main /var wins, trips emergency, and cancels the getty job before it
reports; here the getty job reports first and /var follows. serial-getty@ttyS0.service
is the interactive login prompt on a serial port QEMU points at a write-only file — it
has no relationship to the beacon and no bearing on the gate. Any in-guest probe adds
a job to that transaction, so this delta is not specific to how the beacon is pulled in.

87c381d is kept on its own merits: writing to /dev/console needs no journald
forwarding and attaches no console semantics to the unit.

The `installer-test / Install and boot in QEMU` gate has never passed on
main. Two independent defects make it unreachable by construction.

Transport: `41-kubestellar-kiosk-proxy.yaml` publishes hostPort 8080 on
`hostIP: 127.0.0.1`, a deliberate hardening from #75, but a QEMU hostfwd
with an unspecified guest address is delivered to the guest's DHCP
address, not to guest loopback. Nothing listens there, so
`hostfwd=tcp:127.0.0.1:8080-:8080` can never be served.

Protocol: `files/k0s/kiosk/nginx.conf` has served the proxy as
`listen 8080 ssl` since c752c48, while the recipe probed it over plain
`http://` with `curl --fail`, which turns the 497 -> 302 redirect into
an error and never yields the expected JSON.

Keep the loopback binding and move the probe into the guest: inject a
test-only readiness unit through the same SMBIOS systemd-credential
channel the recipe already uses for `fstab.extra`, pull it in with
`systemd.wants=`, and have it poll `https://127.0.0.1:8080` with
`--insecure`, matching the self-signed TLS handling in tests/e2e, then
announce the verdict on the system console that serial.log already
captures. The dead hostfwd is removed.

Necessary but not sufficient: the gate still needs #173, without which
the target boots into emergency mode waiting for
/dev/disk/by-partlabel/var and never applies any manifests.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The first CI run of this branch (35374112337) reached the new probe but
added a boot-time symptom that no `main` run shows: "Timed out waiting
for device /dev/ttyS0". Runs 34905862943, 34910341724, 34914418293 and
34920225526 only ever time out on /dev/disk/by-partlabel/var.

`StandardOutput=journal+console` makes the unit console-touching, which
pulls extra console ordering into a transaction that is already stuck in
emergency mode. Nothing about the beacon needs journald forwarding, so
echo the two lines to /dev/console directly and drop the setting. Same
output in serial.log, no console machinery attached to the unit.

Verified locally: the generated unit passes systemd-analyze verify,
prints both lines when a self-signed TLS fixture answers, and keeps
polling without emitting the marker when nothing is listening.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Contributor Author

Follow-up in 87c381d after watching the first CI run (35374112337).

That run confirmed the plumbing works — QEMU accepted both new -smbios strings, the guest booted, and the recipe logged ==> Polling KubeStellar Console readiness from inside the guest (deadline: 900s)... — and it failed exactly where expected:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
You are in emergency mode.

default.target is never reached, so the beacon never runs. That is #173's bug, not this one.

One thing that run did surface: an extra Timed out waiting for device /dev/ttyS0 that no main run shows (checked 34905862943, 34910341724, 34914418293, 34920225526 — all only time out on /var). StandardOutput=journal+console makes the unit console-touching, which drags extra console ordering into a transaction that is already wedged. The beacon does not need journald forwarding, so it now echoes straight to /dev/console and the setting is gone. Same two lines in serial.log, no console machinery bound to the unit.

Re-verified locally: systemd-analyze verify clean; the generated unit prints kiosk-ready: polling ... then KIOSK_CONSOLE_READY against a self-signed TLS fixture; with nothing listening it keeps polling and never emits the marker, so the gate cannot false-positive.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor Author

Second run in: 35380863738. build 34m39s pass, build-kernel 21m34s pass, unit 39s pass, installer-test fail at 17m42s on the /var bug. docs does not run — docs-checks.yml is path-filtered to **.md and this PR touches none.

Correcting myself: the /dev/ttyS0 line is not caused by StandardOutput=journal+console. It survived 87c381d. Full serial log from this run (evidence artifact, 24 lines):

[ TIME ] Timed out waiting for device /dev/ttyS0.
[DEPEND] Dependency failed for Serial Getty on ttyS0.
[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Activate k0s sysext on boot.
...
You are in emergency mode.

main's run 34920225526 printed its serial log in full too (it starts at the BdsDxe lines, so nothing was clipped by tail -n 100) and shows /var as the first message with no ttyS0 line. So both are 90s device waits racing: on main /var wins and trips emergency before the getty job reports; here the getty job reports first. serial-getty@ttyS0.service is the login prompt on a serial port QEMU points at a write-only file — unrelated to the beacon, and any in-guest probe adds a job to that transaction regardless of how it is pulled in. PR body updated with this correction.

87c381d stays because it is simpler on its own terms, not because of the ttyS0 line.

Net: this PR is necessary but not sufficient. #173 has to land before installer-test can get past emergency mode and actually exercise the beacon. I have not touched /var or argocd.

@castrojo

Copy link
Copy Markdown
Contributor Author

Runtime evidence: installer-test can pass, and this PR is one of the three things required.

I built a throwaway integration of #173 + #181 + #186 and ran it through CI. Run 35388029502 is GREEN — installer-test / Install and boot in QEMU conclusion success. That is the first time that job has ever passed in this repository. It was introduced at 50be1ed (#149) and every run on main since has failed: 34905862943, 34910341724, 34914418293, 34920225526.

Decisive serial console lines:

[    9.620696] bash[755]: kiosk-ready: polling https://127.0.0.1:8080
[   38.124609] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=2
[   69.115572] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=7
[   89.296275] bash[755]: KIOSK_CONSOLE_READY

and host-side: ==> KubeStellar Console is healthy: /healthz status ok, / returned HTTP 200.

Three independent blockers, all required; any two still fail:

  1. fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173 — the /var udev ordering deadlock. Without it the box sits in emergency mode and nothing downstream runs.
  2. fix(boot): mask systemd-homed-firstboot on the target image #186 — systemd-homed-firstboot.service masked. Without it the boot transaction blocks forever on homectl firstboot --prompt-new-user at a console nobody can answer. This is a product bug, not just a CI one: every headless first boot stalls.
  3. fix(ci): make the installer-test readiness gate reachable #181 — the readiness gate itself: probe from inside the guest over HTTPS (the proxy binds guest loopback, so no host-side probe can ever reach it), and emit the marker so it survives MaxLevelConsole=warning.

The marker is not a false positive. At t=38s and t=69s the loop was awake with k0s active and 10 pods present, but hz=000 root=000 — curl could not connect at all — and it refused to emit. It fired only at t=89s once both endpoints answered. The ExecStart body was also simulated offline with stubbed curl/jq: /healthz returns ok at round 3, / returns 200 at round 8, and the marker emits at round 8, never at 3. And it cannot be satisfied trivially by nginx alone — files/k0s/kiosk/nginx.conf defines no /healthz location, so both probes fall through location / and proxy_pass to the console service; nginx-up/console-down yields 502, which curl --fail suppresses. It went green because the console genuinely became ready.

The deadline is not a problem. Readiness is reached at t=89s against a 900s budget — roughly 10x headroom. No change to projectbluefin/actions is needed. (Note the 900 lives in that external reusable workflow, not this repo's Justfile, which defaults to 600.)

Caveat, stated plainly: the run exercised the <4> SyslogLevelPrefix variant of the marker on the throwaway branch, not #181's current head 87c381d, which writes >> /dev/console. Both defeat the same clamp, and #181's is if anything more robust since writing to the device bypasses journald unconditionally. Everything downstream of the marker plumbing — boot, k0s, CNI, the full KubeStellar stack, both HTTPS checks — is validated identically for either.

The integration branch was a throwaway: no temporary PR was opened, and the branch and worktree are deleted. Neither #173's nor #181's diff or head was touched by any of it.

@castrojo

Copy link
Copy Markdown
Contributor Author

Thanks for the review. No findings were raised, so no code changes were needed.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #210.

Two reasons:

  1. files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml is deleted there, and this PR re-commits hostIP: 127.0.0.1 with a comment defending it. feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet, and fix the boot deadlocks behind it #210 goes the other way and publishes the kiosk on every interface (files/cluster/manifests/kiosk/deployment.yaml), because bare-metal operators could not reach the console at all (bug(kiosk): kiosk-proxy hostPort binds 127.0.0.1, preventing bare-metal LAN access #198) and nginx terminates TLS with a cert-manager-issued certificate rather than a build-time self-signed one, so loopback pinning was never what protected it.

  2. The readiness probe is already fixed on that branch by a different mechanism: test-installer-artifact forwards SSH into the guest and polls the console through that tunnel, requiring /healthz status ok and HTTP 200 on /. Same guarantee as your systemd.extra-unit marker, without injecting a unit into the image under test.

Your diagnosis of why hostfwd can never reach a loopback hostPort — libslirp delivers to the guest DHCP address — is correct and is recorded in the branch comment history. Thanks.

@castrojo castrojo closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant