Skip to content

fix(kiosk): bridge guest loopback proxy via SSH port forwarding for QEMU - #202

Merged
castrojo merged 1 commit into
projectbluefin:mainfrom
mrbobbytables:fix/qemu-kiosk-loopback-tunnel
Sep 23, 2026
Merged

castrojo merged 1 commit into
projectbluefin:mainfrom
mrbobbytables:fix/qemu-kiosk-loopback-tunnel

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Problem

When launching QEMU with user-mode networking or running just install-vm, hostfwd to port 8080 fails to reach kubestellar-kiosk-proxy with a connection reset or empty reply. Because the kiosk proxy specifies hostIP: 127.0.0.1 for security hardening, incoming packets forwarded by libslirp to the guest's external virtio interface (10.0.2.15) cannot be delivered to the loopback-bound proxy.

Solution

  1. Justfile (install-vm):
    • Dropped the dead hostfwd=tcp::8080-:8080 clause from QEMU invocation.
    • Provisioned a dedicated VM ed25519 keypair via SMBIOS Type 11 tmpfiles.extra credentials and enabled sshd.service via kernel-cmdline-extra.
    • Established automated SSH port forwarding (localhost:8080 -> VM 127.0.0.1:8080), bridging host traffic directly into guest loopback where the proxy is listening.
    • Updated readiness probe to verify HTTPS/HTTP on the bridged port before opening browser.
  2. Justfile (tunnel-vm):
    • Added a standalone tunnel-vm recipe allowing operators and developers to forward local traffic to VM loopback on demand.
  3. Developer CLI (files/bin/bluefin-kubestellar):
    • Added tunnel subcommand (start, stop, status) and --tunnel option to manage SSH loopback bridging.
  4. Formula & Manifest Documentation:
    • Updated caveats in Formula/bluefin-kubestellar.rb.
    • Corrected comments in files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml.
  5. Contract Tests:
    • Added unit contract assertions in tests/unit/test_setup_kubestellar.py and bats tests in tests/unit/bluefin-kubestellar_test.bats.

Fixes #194

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 9c744c0

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  1. tmpfiles rule likely never installs the key — tunnel hangs forever. Justfile:454 injects only f~ /root/.ssh/authorized_keys .... systemd-tmpfiles f~ does not create parent directories, and this image's own design doc shows the required pattern with d lines first (d /var/home/core/.ssh 0700 ... before f~ .../authorized_keys, docs/superpowers/specs/2026-09-09-core-login-design.md:89-91). If /root/.ssh doesn't exist in the guest, authorized_keys is never written, BatchMode=yes pubkey auth at Justfile:490 never succeeds, and the while true wait loop at Justfile:482-494 spins indefinitely with no timeout or diagnostic. Add d /root/.ssh 0700 root root - (multi-line payload, as the design doc does) and consider a bounded retry.

  2. xdg-open https://localhost:8080/ opens a broken page. The kiosk proxy serves plaintext HTTP (41-kubestellar-kiosk-proxy.yaml:39-42, port named http; the comment itself says "plaintext HTTP"). xdg-open returns 0 once the browser is spawned, so the || http fallback at Justfile:526 never runs and users land on a TLS error. Open the http:// URL first.

security

  1. PR turns host-wide port 2222 into a live root SSH endpoint. hostfwd=tcp::2222-:22 (Justfile:472) binds 0.0.0.0 on the host. That forward pre-exists, but it was dead — sshd is disabled by preset (files/os/systemd/system-preset/zz-enable-sshd.preset). This PR enables sshd (Justfile:471) against a config with PermitRootLogin yes + PasswordAuthentication yes (files/os/ssh/sshd_config.d/bluefin-server.conf:4-6), so any LAN peer can now attempt root password login into the VM. Bind it loopback-only — hostfwd=tcp:127.0.0.1:2222-:22 — matching the repo's own idiom at Justfile:329. (6443 has the same binding but this PR doesn't change its exposure.)

docs-currency

  1. False LAN access message. Justfile:523 prints Access URL (LAN): https://${HOST_IP}:8080/, but the new ssh -L 8080:... tunnel (Justfile:497) binds the host's loopback only — the console is not reachable via HOST_IP, by design (the updated manifest comment at 41-kubestellar-kiosk-proxy.yaml:32-37 says LAN access requires an authenticated front end). Drop the LAN line; it also wrongly advertises https.

No findings from: intent-alignment, style.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor

Needs a rebase — see #210.

files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml is deleted there, and the loopback pinning you were tunnelling around is gone with it: files/cluster/manifests/kiosk/deployment.yaml now publishes on every interface, so plain QEMU hostfwd should reach the proxy directly.

That likely removes the need for the SSH bridge in the common case. What I would still take from this PR:

  • the tunnel subcommand in files/bin/bluefin-kubestellar (start/stop/status) and the Formula/bluefin-kubestellar.rb caveats — genuinely useful for reaching a remote appliance, independent of the loopback bug;
  • the install-vm SMBIOS Type 11 credential provisioning and sshd.service enablement.

Please drop the manifest hunk and rebase onto main once #210 lands. Refs #194.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

installer-test / Install and boot in QEMU is failing on this PR because it is failing on main itself, not because of anything in this diff.

Tracked as a single repository incident in #218. main has been red on this check since 2026-09-14; the same failure currently blocks 17 open PRs, including a docs-only PR (#208) and a Renovate digest bump (#49).

Failure mode is /dev/disk/by-partlabel/var never appearing, so /var and everything ordered after it depend-fail and the readiness probe times out.

Please don't retry this check per-PR — it will stay red until #218 is resolved.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @kubestellar-hive in commit 74b280a:

  1. Tmpfiles parent directory: Added d /root/.ssh 0700 root root - prior to writing authorized_keys so the directory is guaranteed to exist.
  2. Bounded retry: Added a 120-second timeout to the SSH readiness polling loop in install-vm.
  3. Loopback-only SSH forward: Bound host port 2222 forwarding strictly to 127.0.0.1:2222 (hostfwd=tcp:127.0.0.1:2222-:22) to prevent exposing root SSH on 0.0.0.0.
  4. Plaintext HTTP URL & dropped LAN advertisement: Updated xdg-open to launch http://localhost:8080/ directly and removed the LAN access line since the SSH tunnel binds loopback only.
  5. Contract test: Updated tests/unit/test_setup_kubestellar.py to assert the loopback-only forward binding.

Regarding @castrojo's note on #210: retaining the SSH bridge and bluefin-kubestellar tunnel tooling as requested; will rebase onto main and drop any redundant manifest hunks once #210 merges.

🐝 Hive Agent: contributor | SHA: 74b280a

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — confirm whether test-installer-artifact's hostfwd-based readiness poll is also broken by the loopback-bound proxy and belongs in the scope of #194 — @mrbobbytables

intent-alignment

  • (medium) The PR's root-cause analysis says a SLIRP hostfwd cannot reach the guest's loopback-bound kiosk proxy, but test-installer-artifact still uses hostfwd=tcp:127.0.0.1:8080-:8080 and polls readiness through it (Justfile:329, 340). If the analysis is right, that smoke recipe times out for the same reason #194 describes and is left unfixed; if it currently passes, the root-cause claim needs revisiting. Either way this contradiction needs a human/CI answer.

correctness

  • (medium) install-vm's cleanup trap only covers INT/TERM (Justfile:484), but the new failure paths exit 1 directly — SSH wait timeout (Justfile:500-502) and tunnel death (Justfile:520-522) — leaving the backgrounded QEMU (and tunnel) running orphaned. Previously the only explicit failure exit occurred after QEMU had already died. Trap EXIT (as test-installer-artifact does at Justfile:318) or call cleanup before those exits.
  • (low) start_tunnel in files/bin/bluefin-kubestellar backgrounds ssh via nohup without -o BatchMode=yes (files/bin/bluefin-kubestellar:180-182). With the fallback to a possibly passphrase-protected ~/.ssh/id_ed25519 (line 162), ssh can block prompting on /dev/tty, while the 0.5s liveness check still reports the tunnel "active".

security

  • (low) start_tunnel falls back to the user's personal ~/.ssh/id_ed25519 and connects to root@127.0.0.1:2222 with StrictHostKeyChecking=no/UserKnownHostsFile=/dev/null (files/bin/bluefin-kubestellar:162, 181). When the VM is not running, any local process can bind 2222 and impersonate the kiosk endpoint; forwarded console traffic (including login credentials) would then reach the impostor. Low, local-only, but pinning the VM host key (it's a known, self-generated VM) would close it.

style

  • (info) In the tunnel status path, status_tunnel; exit $? (files/bin/bluefin-kubestellar:233-234): under set -e the exit $? line is dead code when the function returns non-zero — errexit exits with status 3 first. It works (the bats test at tests/unit/bluefin-kubestellar_test.bats:35-39 passes coincidentally), but the propagation is subtle; status_tunnel || exit $? would make it explicit.

No findings from: docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed in commit 3437846:

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • Fix is incomplete against the stated root cause of #194. The PR removes hostfwd=tcp::8080-:8080 from install-vm because "a QEMU slirp hostfwd cannot reach guest loopback" — but test-installer-artifact (Justfile:328, reached via show-me-the-future) still boots the installed image with hostfwd=tcp:127.0.0.1:8080-:8080 and polls http://127.0.0.1:8080 for readiness (Justfile:340) with a 600s deadline. If the root-cause analysis in #194 is correct, that recipe times out for the identical reason and needs the same SSH bridge; if it actually passes, the premise of this PR deserves a second look. Either way the two recipes now contradict each other.

security

  • The rewritten -nic line (Justfile:472) tightens SSH to hostfwd=tcp:127.0.0.1:2222-:22 but leaves hostfwd=tcp::6443-:6443 bound to all host interfaces, exposing the guest kube-apiserver on the LAN — inconsistent with the loopback-hardening rationale this PR itself states (and its own test asserts loopback binding for 2222 but not 6443, tests/unit/test_setup_kubestellar.py:53).

correctness

  • files/bin/bluefin-kubestellar start_tunnel: when no key is found, ssh "${KEY_ARGS[@]}" expands an empty array under set -u, which is a fatal "unbound variable" on bash < 4.4 (macOS /bin/bash 3.2) — and this script ships via the Homebrew formula whose caveats now advertise tunnel (Formula/bluefin-kubestellar.rb:21). Same pattern in the tunnel-vm recipe is Linux-only and fine.
  • start_tunnel declares success and keeps the PID file after only sleep 0.5; auth or connect failures surfacing later leave a stale PID file and a false "tunnel active" message (low).

No findings from: style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review findings in commit 93ae9bd:

  1. Host loopback interface binding for 6443: Tightened hostfwd=tcp:127.0.0.1:6443-:6443 in Justfile to prevent exposing the guest kube-apiserver across external host LAN interfaces, aligning with the loopback isolation policy for port 2222.
  2. Safe array expansion under bash < 4.4: Updated files/bin/bluefin-kubestellar to expand ${KEY_ARGS[@]+"${KEY_ARGS[@]}"} so that running under set -u with no SSH key does not fail on older bash versions.
  3. Contract test coverage: Updated tests/unit/test_setup_kubestellar.py to assert loopback binding on both port 2222 and 6443.
  4. Sequencing with feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet, and fix the boot deadlocks behind it #210: Acknowledged the upstream refactoring in feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet, and fix the boot deadlocks behind it #210 where kiosk-proxy is modernized; this tunnel hardening remains ready to rebase cleanly once feat(kubernetes): replace the k0s sysext with upstream kubeadm/kubelet, and fix the boot deadlocks behind it #210 merges.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • tests/unit/bluefin-kubestellar_test.bats:35-39 — the new tunnel status test strictly asserts exit 3, but the script reads the real user state dir (~/.local/state/bluefin-server/ssh-tunnel.pid, files/bin/bluefin-kubestellar:17). A live tunnel or stale/reused PID on the machine running the tests makes it fail; the neighboring status test (lines 29-33) tolerates 0 or 3 for exactly this reason.
  • files/bin/bluefin-kubestellar:185-192 — start_tunnel declares success after only a 0.5s liveness check; with ExitOnForwardFailure an ssh that dies slightly later (e.g. local port 8080 already bound) leaves a stale PID file and a false "tunnel active" message. Low severity.

Verified the core mechanism holds at head: the installed image already boots with sd-stub SMBIOS credentials (Justfile:330-331 uses the same io.systemd.stub.kernel-cmdline-extra pattern), sshd ships in the image with PermitRootLogin yes + PubkeyAuthentication yes (files/os/ssh/sshd_config.d/bluefin-server.conf) and is preset-disabled (files/os/systemd/system-preset/zz-enable-sshd.preset), which systemd.wants=sshd.service overrides at boot. Loopback-binding the remaining hostfwds (Justfile:470-472) is a net security improvement.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar

Copy link
Copy Markdown
Contributor

#220 merged to main at 14:06 UTC today with the fix for the installer-test boot deadlock (udev sysext ordering, depmod path, homed-firstboot mask). This branch predates it, so its installer-test / Install and boot in QEMU failure is the old repo-wide one, not this change. A rebase onto current main is all it needs to get a real result from that check; I cannot update the branch from here.

@Danathar

Copy link
Copy Markdown
Contributor

The Justfile half of this has been overtaken. #227 merged as f9bdef4 on 2026-09-22 and solves the same problem in the same install-vm recipe, by the opposite route: instead of tunnelling host to guest over SSH, it injects an in-guest readiness unit as an SMBIOS credential and waits for KIOSK_CONSOLE_READY on the serial log. Main's comment at Justfile:595 says so explicitly. That is also why this conflicts on Justfile and 41-kubestellar-kiosk-proxy.yaml, the two files #227 touched.

The manifest change here is comment-only; hostIP: 127.0.0.1 and hostPort: 8080 are unchanged.

One design point worth raising before any rebase: to make the tunnel work, this injects /root/.ssh/authorized_keys via tmpfiles.extra and adds systemd.wants=sshd.service to the kernel cmdline. #80 is concurrently making PermitRootLogin no and a fail-closed core-only access gate the house rule. Those two pull in opposite directions even if this is only ever used in a dev VM.

What does look worth keeping is the bluefin-kubestellar tunnel subcommand in files/bin/bluefin-kubestellar (start / stop / status, --ssh-port, --local-port, --remote-port, -k). That is new developer surface and nothing on main duplicates it. Suggest splitting it out and dropping the Justfile and manifest hunks.

The red installer-test check is the repo-wide partlabel timeout from before #220 landed, not anything this PR does.

@castrojo
castrojo force-pushed the fix/qemu-kiosk-loopback-tunnel branch from 93ae9bd to 15ec11c Compare September 23, 2026 12:08
@castrojo
castrojo merged commit e4d9793 into projectbluefin:main Sep 23, 2026
1 of 2 checks passed

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • PR body claims changes the diff does not contain — #194 is not actually fixed. The body says the Justfile install-vm recipe was changed (dropped dead 8080 hostfwd, provisioned a VM ed25519 keypair via SMBIOS credentials, enabled sshd.service, automated SSH forwarding), that a tunnel-vm recipe was added, that comments in files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml were corrected, and that bats tests were added to tests/unit/bluefin-kubestellar_test.bats. None of these files are in the diff (only Formula/bluefin-kubestellar.rb, files/bin/bluefin-kubestellar, tests/unit/test_setup_kubestellar.py); the bats file at head 15ec11c contains zero mentions of tunnel, and the Justfile is untouched. Without the guest-side provisioning (key + sshd), the shipped tunnel subcommand cannot work out of the box, so the issue's core recommendation is unimplemented.

correctness

  • start_tunnel hardcodes root@127.0.0.1 (files/bin/bluefin-kubestellar:181), but the repo's own operator instructions say to tunnel as <your-user>@127.0.0.1 after manually starting sshd (Justfile:711-712). Nothing provisions a root-authorized key, and the key-discovery glob "$STATE_DIR"/vm-*/id_ed25519 (files/bin/bluefin-kubestellar:158) matches a path nothing in the repo creates — the provisioning the PR body describes is absent. tunnel start will fail against the VM as shipped.
  • start_tunnel declares success after a 0.5s kill -0 check (files/bin/bluefin-kubestellar:186-192). With BatchMode=yes, an auth or connection failure typically takes longer than 0.5s, so the command reports "SSH tunnel active" and leaves a PID file for a process about to die; a later tunnel status on a recycled PID can then report a false positive.
  • The new python test only asserts substrings like "tunnel" and "-L" exist in the script text (tests/unit/test_setup_kubestellar.py:36-41) — it cannot catch any of the above.

security

  • The tunnel uses -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null and silently falls back to the user's personal ~/.ssh/id_ed25519 (files/bin/bluefin-kubestellar:158-162, 182). Any local process that binds 127.0.0.1:2222 before the VM can impersonate the guest with no warning, and console traffic on localhost:8080 is then forwarded to it. Low severity given the localhost-only threat model, but pinning the host key or at least not falling back to the personal default key would remove it.

docs-currency

  • Formula caveat advertises bluefin-kubestellar tunnel (Formula/bluefin-kubestellar.rb:21-22) without noting the prerequisites the PR did not provision (guest sshd started, an authorized key for the login user); as written the caveat points users at a command that fails by default.

No findings from: style.

Confidence: 1/5 (needs attention) — 2 high findings

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(kiosk): QEMU hostfwd to 127.0.0.1:8080 cannot reach kiosk-proxy bound to guest loopback

3 participants