Skip to content

fix: sshd drop-in — prohibit root password login (files/os/ssh/sshd_config.d/bluefin-server.conf) - #169

Merged
castrojo merged 1 commit into
mainfrom
sec/fix-sshd-root-password
Sep 20, 2026
Merged

castrojo merged 1 commit into
mainfrom
sec/fix-sshd-root-password

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Security Fix

Restores the sshd hardening from #29, reverted by 37c8356:

  • files/os/ssh/sshd_config.d/bluefin-server.conf: PermitRootLogin yes → prohibit-password; PasswordAuthentication yes → no; KbdInteractiveAuthentication yes → no.

The same commit that reverted the hardening seeds the well-known default root password bluefin into every DDI image (elements/oci/bluefin-server-ddi.bst /etc/shadow). sshd is preset-disabled, but enabling it is the documented bring-up diagnostics path — at which point remote root with a published password works.

Preserved: console/serial bring-up (root / bluefin via /etc/securetty + issue banner) and key-based remote diagnostics.
Left open (deliberately): the seeded default password itself — Refs, not Closes. Forcing first-login rotation or per-install random passwords needs a maintainer decision; see the issue.
Spec note: docs/superpowers/specs/2026-09-09-core-login-design.md specifies the stricter PermitRootLogin no; this PR restores the previously reviewed baseline.

Refs #168


Filed by sec-check agent (ACMM L4/L5 — hold-gated mode). Hold-gated: human review required.

— hive: agent=sec-check backend=copilot model=kimi-k3

…twork

Commit 37c8356 reverted the #29 hardening (PermitRootLogin yes +
PasswordAuthentication yes) while the same change seeds the well-known
default root password 'bluefin' into every DDI image. Once an operator
enables sshd — the documented bring-up diagnostics path — remote root
login with a published password is possible.

Restore the hardened drop-in: PermitRootLogin prohibit-password,
PasswordAuthentication no, KbdInteractiveAuthentication no. Console and
serial bring-up (root / bluefin via /etc/securetty) is unaffected; remote
diagnostics remain possible with key auth. The design spec
(docs/superpowers/specs/2026-09-09-core-login-design.md) goes further
(PermitRootLogin no); this PR restores the previously reviewed baseline.

Refs #168 — leaves the seeded default password itself (shadow entry,
banner) in place pending maintainer decision on first-boot rotation.

Signed-off-by: sec-check agent
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added hold Indicates that a PR should not merge because someone has issued a /hold command. security agent/security Filed or owned by the security agent. labels Sep 16, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct and I'd like it in. I checked each claim in the description rather than taking it on faith, and they all hold up.

The revert is real, and it happened inside an unrelated commit:

$ git show 37c8356 --stat
37c8356 fix(boot): resolve first-boot login, /var mounting, and KubeStellar readiness
...
    Pre-generate SSH host keys and seed default root password 'bluefin' in /etc/shadow.
    Add tty0, ttyS0, console, and hvc0 to /etc/securetty for console root login.
    Permit root password login in sshd_config.d/bluefin-server.conf for bring-up diagnostics.

So one commit both seeded the credential and opened the door it fits. The seeded password is where you say it is:

$ grep -n "root:" elements/oci/bluefin-server-ddi.bst
72:      # Seed default root password 'bluefin' into /etc/shadow for console and ssh login
74:      printf 'root:$6$bluefin123$o2pQjPN1n3ZCnYUROlA01xAdnGO7mhVRAVw98x4xc8LAUIJv7b44bBNoER6fM4P.rrj4ePYFfkx9WvZICU8nt/:19700:0:99999:7:::\n' > /layer/etc/shadow
$ grep -n bluefin files/os/issue.d/40-kubestellar.issue
2:Default login: root / bluefin

and the preset-disabled claim checks out, which is what keeps this from being an active incident today rather than a latent one:

$ cat files/os/systemd/system-preset/zz-enable-sshd.preset
...
disable sshd.service
disable sshd.socket

The console path you say you preserve is preserved — /etc/securetty and the issue banner are untouched by this diff, and PermitRootLogin prohibit-password only governs the network path. One thing I specifically went looking for was whether anything in-tree logs in over SSH with a password and would break:

$ grep -rniE "sshpass|ssh .*root@|PreferredAuthentications|-o PasswordAuth" scripts/ tests/ workflows/ docs/
(no matches)

Nothing does. The gate is unchanged from the merge-base — base 5ebfeae and this head fd3a9cb both give 207 passed, 1 xfailed and 62 bats ok / 0 not ok — which is expected, since nothing covers this file at all.

Two follow-ups I'd ask you to consider, neither blocking.

Please add the regression test. This exact hardening has now been written three times (4a8fdf8, f78f7fe for #29, and this one) and silently removed once by a commit whose subject was about /var mounting. Nothing in tests/unit reads files/os/ssh/sshd_config.d/bluefin-server.conf — I checked, the only hits for these directives repo-wide are the file itself and the design spec:

$ grep -rn "PasswordAuthentication\|PermitRootLogin\|KbdInteractive" . --exclude-dir=.git
./files/os/ssh/sshd_config.d/bluefin-server.conf:8:PermitRootLogin prohibit-password
./files/os/ssh/sshd_config.d/bluefin-server.conf:10:PasswordAuthentication no
./files/os/ssh/sshd_config.d/bluefin-server.conf:11:KbdInteractiveAuthentication no
./docs/superpowers/specs/2026-09-09-core-login-design.md:41: ...

A handful of asserts in the style of the existing test_installer_contract.py would make the next accidental revert a red check instead of a quiet regression. That's the difference between this fix holding and it being reverted a fourth time.

Consider a numeric filename prefix. OpenSSH uses the first value obtained for each keyword, and Include /etc/ssh/sshd_config.d/*.conf expands in lexical order, so bluefin-server.conf loses to any drop-in named with a digit. Today that's harmless — the image ships its own minimal main config:

$ cat files/os/ssh/sshd_config
# Main OpenSSH daemon configuration
Include /etc/ssh/sshd_config.d/*.conf
$ find files/os/ssh -type f
files/os/ssh/sshd_config
files/os/ssh/sshd_config.d/bluefin-server.conf

so nothing currently sorts ahead of it. But a vendor drop-in landing as 50-*.conf later would silently win, and the failure mode is the hardening evaporating with no signal. Renaming to something like 10-bluefin-server.conf costs nothing and removes the hazard.

Last, a wording nit: the new comment says "for remote access inject an authorized_keys entry via the installer or systemd credentials", but grep -rn authorized_keys finds no such mechanism in this repo — only an illustrative tmpfiles line in the design spec. With PasswordAuthentication no and no key provisioning, enabling sshd for bring-up currently yields a daemon nobody can log into. That is the right secure default, but the comment reads as if the path exists; either point it at the issue that will build it or say plainly that key injection is operator-supplied.

Approving. The security direction is right, the claims are accurate, and the gate is clean — I'd just really like the contract test to come with it, here or immediately after.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot added the agent/scanner Filed or owned by the scanner agent. label Sep 17, 2026
@hivecommons-hive hivecommons-hive Bot changed the title [sec-check] fix: sshd drop-in — prohibit root password login (files/os/ssh/sshd_config.d/bluefin-server.conf) fix: sshd drop-in — prohibit root password login (files/os/ssh/sshd_config.d/bluefin-server.conf) Sep 17, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Deferring this PR's red installer-test / Install and boot in QEMU to the shared incident #175.

Baseline evidence — the same check is failing on the default branch main, so the signal is not PR-local:

This is one repository incident, not one failure per PR, so no PR-specific retry or diff repair is appropriate here. This PR stays deferred until installer-test is green on main or #175 closes.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI note: installer-test failure here is repo-wide, not from this diff. The QEMU boot dies at Timed out waiting for device /dev/disk/by-partlabel/var — every recent 'Build DDI artifacts' run across branches is failing the same way (last 8 runs all red). An sshd_config.d drop-in cannot remove a var partition; this PR will go green on re-run once the DDI /var partition regression is fixed on main.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff and verified the security claim against repository history rather than the PR description.

Verified:

  • 37c8356 ("fix(boot): resolve first-boot login, /var mounting, and KubeStellar readiness") is real and did both things this PR reacts to. It set PermitRootLogin yes / PasswordAuthentication yes / KbdInteractiveAuthentication yes in this drop-in, and the same commit seeds a default root password into the image: elements/oci/bluefin-server-ddi.bst writes a fixed SHA-512 crypt for the password bluefin into /layer/etc/shadow.
  • The current state is therefore a published, identical-on-every-image root password that is accepted over the network the moment sshd is enabled. That is a real exposure, not a theoretical one.

I checked the blast radius before agreeing to turn password auth off. Nothing in CI or repo tooling authenticates with a password: there is no sshpass, no PreferredAuthentications=password, and the QEMU smoke test drives the guest over the serial console plus an HTTP readiness poll rather than SSH. Console and serial bring-up remain available through /etc/securetty, and key-based remote diagnostics are unaffected.

The change is three directives in one drop-in, restoring a baseline that was already reviewed in #29. Scope is correct, and it deliberately does not try to solve the seeded-password problem itself, which is a maintainer decision rather than something to guess at here.

One thing the next reviewer should know: docs/superpowers/specs/2026-09-09-core-login-design.md specifies the stricter PermitRootLogin no, and open PR #80 implements that fuller hardening. This PR is intentionally the weaker, previously-reviewed baseline. I do not think that is a reason to hold it, since it strictly reduces exposure today and does not block #80 going further.

CI note: the only failing check is installer-test / Install and boot in QEMU, which fails identically on main at tip 5ebfeae (run 34920225526) with "Dependency failed for /var" into emergency mode. That is upstream of this PR. build and unit are green here.

Approving. Merge should wait for the /var regression on main to be fixed so this lands on a green base.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI status: installer-test / Install and boot in QEMU fails identically on main HEAD (run 34920225526, job 104235482419: Timed out waiting for device /dev/disk/by-partlabel/var → KubeStellar Console readiness timeout). This branch is rebased on main tip (5ebfeae) and touches only files/os/ssh/sshd_config.d/bluefin-server.conf, which cannot affect partition creation. Pre-existing break tracked in #158 / #161 / #166 / #175; this PR will go green once the installer /var fix lands. Holding as-is — no action possible from this diff.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage: the installer-test / Install and boot in QEMU failure is inherited from main, not caused by this PR.

Evidence:

Main breakage is tracked in #161, #166 and #175. Once that fix lands on main, a rebase/re-run should green this PR. Hold stays in place.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage: the failing installer-test / Install and boot in QEMU check is not caused by this PR's sshd drop-in. The branch is 0 commits behind main, and main's own installer-test has been failing since 2026-09-15 with a /var mount dependency deadlock (Dependency failed for /var → Basic System fails → 900s readiness timeout). The fixes live in the open integration work (PRs #173, #181, plus the fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock commit) — integration branch run 35388029502 is green with them. Will re-run CI here once those land on main. Hold stays.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage: the installer-test / Install and boot in QEMU failure is pre-existing on main, not caused by this PR's sshd drop-in.

Evidence:

  • This PR's run 35086132490: installed VM second boot fails with Timed out waiting for device /dev/ttyS0 / Dependency failed for /var → 900s KubeStellar Console readiness timeout.
  • main @ 5ebfeae (run 34920225526, includes attempted fix fix(test): preserve installer-created var partition #157): identical signature — Timed out waiting for device /dev/disk/by-partlabel/var → same 900s readiness timeout.
  • Same job fails on every recent branch (e.g. fix/installer-root-var-partition-sizing, fix/installer-test-readiness-gate), which are the in-flight fixes for the installer /var partition issue.

The sshd config change cannot affect /var partition device discovery. Will rebase/rerun once the installer fix lands on main. Hold stays.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage for the red installer-test / Install and boot in QEMU check:

Not cherry-picking #173 into this branch to avoid duplicating that open PR. Once #173 merges I'll rebase and re-run; expectation is green with no change to this diff.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI failure triage — not caused by this PR

installer-test / Install and boot in QEMU fails identically on main (e.g. run 34920225526, commit 5ebfeae) and on every branch since the gate was introduced in #149 — the job has never passed on main. This PR only adds files/os/ssh/sshd_config.d/bluefin-server.conf, which is not consumed until the installed system boots sshd; the boot never reaches that point.

Failure signature (this PR's run 35086132490 and main's run 34920225526):

  • [ TIME ] Timed out waiting for device /dev/ttyS0.
  • [ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
  • → emergency mode, KubeStellar console readiness times out after 900s.

Two unrelated device units timing out points at udevd, not at the partition layout: the partitions exist, but their .device units never appear.

Root cause is already identified and fixed in-flight: #173 (fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock) plus the gate repair in #181. The integration branch combining them (tmp/integration-173-181, commit bb7f25a) ran this exact job green: run 35388029502.

Action for this PR: rebase onto main after #173/#181 merge and re-run CI. Pulling those commits into this branch now would duplicate their open PRs, so leaving this PR as-is until they land. Hold label stays.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

CI triage (sec-check): the failing check installer-test / Install and boot in QEMU is a pre-existing main-branch breakage, not caused by this PR's sshd drop-in.

Evidence:

  • main itself fails the same job on every push since ci: run installer smoke through shared QEMU workflow #149 (2026-09-14): runs 34905862943, 34910341724, 34914418293, 34920225526 — all Build DDI artifacts failures on main.
  • Identical failure signature on unrelated branches (e.g. run 35461538285, feat-flatcar-runtime-parity): installed system soft-reboots during first boot (two systemd boot records, same bootid), after which device units never appear — /dev/ttyS0 and /dev/disk/by-partlabel/var both time out and the system drops to emergency mode.
  • This PR changes only files/os/ssh/sshd_config.d/bluefin-server.conf; the boot failure occurs before sshd is ever started.

Already tracked: #161, #166, #175, #158. This branch is up to date with main (5ebfeae); it will go green once the installer first-boot fix lands. Holding — no noise pushes to re-trigger a deterministic failure.

🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo
castrojo merged commit 1698c25 into main Sep 20, 2026
4 of 5 checks passed
castrojo added a commit that referenced this pull request Sep 23, 2026
…onents (#132)

* fix(renovate): match the freedesktop-sdk series the junction actually tracks

renovate.json filters freedesktop-sdk tags with

    "extractVersion": "^freedesktop-sdk-(?<version>25\\.08\\.[0-9]+)$"

while elements/freedesktop-sdk.bst has tracked 26.08 since the bump to
freedesktop-sdk-26.08.0-0-gdb97cce:

    track: freedesktop-sdk-26.08*
    ref: freedesktop-sdk-26.08.0-0-gdb97cce32cecadc7a3e98f06d557ebfa6ba9ad46

No tag upstream publishes can match that rule any more, so Renovate reports
no updates rather than an error. The junction silently stops receiving bumps
and the track-refs job in build.yml never has a ref to resolve. Nothing in
any log says so — the failure looks exactly like "already up to date".

Bumps the rule to 26.08, and adds check-renovate-series.py so the pair cannot
drift apart again: it reads the series out of each junction's `track:` glob
and asserts the matching extractVersion is anchored to the same one, failing
closed. Same shape as check-release-version.py and check-k0s-version.py —
wired into .pre-commit-config.yaml, and unit-tests.yml now also fires on
renovate.json and elements/*.bst so the guard runs when either side moves.

tests/unit/test_check_renovate_series.py covers the passing case, the exact
regression above, a junction with no packageRule, a junction with no
`track:`, the two series helpers, and the checked-in tree itself.

    $ python3 -m pytest tests/unit -q
    207 passed, 1 xfailed in 0.64s

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VwEniffvKGH7gHfqKneuaJ

* refactor: switch OS payload to Flatcar base, drop displaced FSDK components

Cut os-stack.bst over to the Flatcar base (projectbluefin/server#131), the
Phase 5 slice of the Flatcar base migration design doc. The installed OS DDI
payload now lands on a single-ABI Flatcar `/usr` that shares the vermagic,
glibc, and systemd of the Flatcar LTS kernel, instead of straddling FSDK 26.08
userspace against the Flatcar kernel.

- Add elements/flatcar/flatcar-usr.bst: import element mirroring the existing
  flatcar-zfs.bst pattern — a single pinned, digest-verified flatcar-container
  source, prebuilt binaries, FSDK stripper left alone. Extracts /usr, drops
  Flatcar's update/provisioning stack (update_engine, locksmithd, ignition,
  coreos-cloudinit, flatcar-update, download_sysext, ensure-sysext), flattens
  the nested module dir to single-level, stages the raw sysext images.
- Rewrite os-stack.bst: replace the FSDK base-runtime depends (runtime-minimal,
  systemd, dbus, dbus-broker, kmod, shadow, bootstrap/bash, uutils-coreutils,
  openssh-systemd, podman, xfsprogs, gnupg, ca-certificates, tzdata) with
  flatcar/flatcar-usr.bst. Keep all ~14 OS-policy elements and the kernel/zfs.
  The FSDK junction stays only for integration helpers (extra-fs, ldconfig);
  os-countme keeps its curl+jq via that junction.
- Delete elements/bluefin-server/linux-firmware-split.bst (only referenced by
  os-stack); Flatcar firmware now ships in /usr/lib/firmware. Keep the
  uutils-coreutils.bst file — installer-stack.bst still needs it (installer
  stays FSDK 26.08, hard rule 3).
- Update tests/unit/test_build_depends.py: the old tests locked in FSDK invariants
  (uutils-not-gnu, includes dbus-broker, includes bash) that the cutover breaks;
  replace them with invariants for the new base — os-stack depends on
  flatcar-usr.bst and excludes every displaced FSDK component.
- Update docs/skills/factory-integration.md and README.md: podman and the base
  userspace are now Flatcar/sysext, not FSDK base components.

No os-base option exists, so its removal is a no-op. The installer keeps its
FSDK 26.08 userspace; hard rules 2, 3, and 4 are preserved.

bluefin-server#131

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* fix(installer): ld.so.conf uses Flatcar /usr/lib64, not Debian multiarch

The installer's ld.so.conf hardcoded /usr/lib/x86_64-linux-gnu (the Debian/
Ubuntu multiarch path). On the Flatcar-based server image the FSDK libraries
it needs to index (libc.so.6, libgcc_s.so.1, libkmod.so) live under /usr/lib64,
so ldconfig cached the wrong path and the bad cache was copied over the build
sandbox's own. Point both ld.so.conf entries at /usr/lib64.

projectbluefin/server#132 (review item #6)

* fix(flatcar): drop duplicate flatcar-usr.bst, provided by #140

* fix(installer): keep host ld.so.conf on Debian multiarch path

The installer/dracut tooling is FSDK/Debian-based, so its libraries live
under /usr/lib/x86_64-linux-gnu. The host ldconfig must index that path;
/usr/lib64 is only correct for the Flatcar target-root cache (which the
-r /target-root ldconfig builds and copies over the host cache). Keeping
/usr/lib64 on the host risks FSDK libs not resolving during the build.

Addresses hanthor's review comment on projectbluefin/server#132
(points 1 and 4).

* ci(just): run the renovate series guard in validate

`check-renovate-series.py` was wired into pre-commit and the unit suite
but not into `just validate`, unlike its siblings
`check-release-version.py` and `check-k0s-version.py`. Since AGENTS.md
and docs/skills/ci-tooling.md point contributors at `just validate` as
the merge-contract check, the new invariant was effectively unenforced
there.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci: expand unit-tests workflow paths trigger

Trigger unit tests on changes to elements/**, include/**,
project.conf, .pre-commit-config.yaml, and .github/workflows/build.yml,
which are read and asserted by the unit test suite.

Fixes #187

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* fix(ci): pin just tool version in CI workflows

taiki-e/install-action resolves just@latest at run time when no version
is supplied, so the build tool that drives every job (just validate,
just test-unit, DDI/kernel export recipes) was unpinned while every
GitHub Action in the repo is SHA-pinned. Pin just to 1.58.0, the
version CI already installs, so this changes no runtime behavior.

Signed-off-by: hive-agent <hive-agent@users.noreply.github.com>

* fix(installer): align target ld.so.conf path + add linker-path contract test

hanthor review on #132:
- The target ld.so.conf was written to /target-root/tmp/ but read via
  'ldconfig -r /target-root -f /tmp/ld.so.conf'. That only resolves because
  of the -r chroot; make both agree on host /tmp explicitly.
- Add a contract test asserting the host indexes the Debian multiarch path
  and the target indexes Flatcar /usr/lib64, and that the two paths agree.

projectbluefin/server#132

* fix(installer): revert ld.so.conf write path — -f resolves post-chroot

My previous commit (8e96b1c) wrote the target conf to host /tmp/ld.so.conf,
but 'ldconfig -r /target-root -f /tmp/ld.so.conf' reads it post-chroot at
/target-root/tmp/ld.so.conf, so the build would fail to find the config.
Verified empirically that -r chroots the -f path. Revert the write to
/target-root/tmp/ld.so.conf and document the post-chroot resolve instead.
Fix the contract test, which had encoded the wrong paths.

* test(132): drop duplicate test_os_countme from #132

#122 adds test_os_countme_depends_on_curl_and_jq; #132 carried a verbatim
copy that would conflict on merge (Python keeps the second silently). Per
hanthor's coordination, land #122 first and let #132 drop its copy.

* docs(ci-tooling): pin the tool version in the canonical just snippet

The skill doc's install snippet still showed an unpinned 'tool: just', which is
the form issue #188 removed from the workflows. A workflow copied from the doc
would have reintroduced it, undoing this PR on the next new job.

Update the snippet to 'tool: just@1.58.0', matching all four call sites, and
explain that without a version install-action resolves just@latest at run time.
Note that the version is repeated per call site so a bump has to change all of
them in one commit.

Add the half-done pin to Red Flags, add a checklist item for it under
Verification, and record the 'tool: just always installs a working version'
rationalization.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): bound root-a partition to guarantee room for var (#207)

Caps `bluefin-server-root-a` at 8 GiB so the tail `/var` partition is always placeable on a 16 GiB target disk. Root cause of the `installer-test / Install and boot in QEMU` failure that has been red on main since 2026-09-14: serial evidence from run 35497423148 shows `Timed out waiting for device /dev/disk/by-partlabel/var` followed by emergency mode.

Verified safe: the DDI payload is 2.16 GiB uncompressed (`zstd -l dist/ddi/bluefin-server-ddi-26.08.0.raw.zst`), well under the new 8 GiB cap and above the 4 GiB `SizeMinBytes`.

Closes #161, closes #158, closes #175, closes #166.

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [sec-check] fix: harden sshd drop-in — no root password login over network (#169)

Restores the sshd hardening reverted by 37c8356. The same commit seeds the well-known default root password `bluefin` into every DDI `/etc/shadow`; with `PermitRootLogin yes` + `PasswordAuthentication yes`, enabling sshd (the documented bring-up diagnostics path) exposes remote root with published credentials.

Console/serial bring-up (root / bluefin via /etc/securetty + issue banner) and key-based remote diagnostics are preserved.

Refs #168 — the seeded default password itself stays open; forcing rotation or per-install randomisation is a separate decision.

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [scanner] fix: propagate lima-e2e browser-verification failures (#160)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: executed coverage for docs-checks.py main() — new tests/unit/test_docs_checks_main.py (#106)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: behavioral coverage for files/bin/bluefin-kubestellar — tests/unit/bluefin-kubestellar_test.bats (#171)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(build-depends): lock in os-countme curl+jq FSDK paths (#122)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test: files/ payload reachability gate — tests/unit/test_files_payload_reachability.py (#155)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(arch): produce Flatcar-to-FSDK version parity matrix (#142) (#184)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(flatcar): record container runtime parity (#192)

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(ci): move junction ref tracking out of the pull_request path (#180)

Moves junction tracking off the pull_request path onto a scheduled tracker that also syncs project.conf release-version from the tracked FSDK point release, so junction bumps arrive green instead of dying at check-release-version.py.

Assisted-by: Claude Opus 4.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(boot): resolve QEMU installer boot deadlock and console readiness probe (#220)

* fix(boot): resolve QEMU installer boot deadlock and console readiness probe

In `installer-test / Install and boot in QEMU`, target boot times out waiting for `/dev/disk/by-partlabel/var` and eventually fails KubeStellar Console readiness.

Root causes resolved:
1. `elements/flatcar/flatcar-zfs.bst`: The extracted Flatcar ZFS system extension ships `usr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf` (`After=systemd-sysext.service`) and a tmpfiles copy rule in `usr/lib/tmpfiles.d/10-zfs.conf`. Because ZFS is extracted directly into the OS rootfs, this ordering caused systemd-udevd to wait for systemd-sysext.service, which waited for local-fs.target, which waited for var.mount, creating a circular boot deadlock where udevd could never start and device nodes were never created. Remove the drop-in, strip the tmpfiles rule, and add `sed.bst` build dependency.
2. `elements/oci/bluefin-server-ddi.bst` & `elements/oci/bluefin-server-installer.bst`: Flatcar kernel module directories reside under `/usr/lib/modules`; point `depmod -b` to `/layer/usr` and `/target-root/usr` instead of root.
3. `elements/oci/bluefin-server-ddi.bst` & `Justfile`: Mask `systemd-homed-firstboot.service` alongside `systemd-firstboot.service` on the target image and kernel cmdline to prevent headless hangs on interactive user creation prompt.
4. `Justfile` & `41-kubestellar-kiosk-proxy.yaml`: Move console readiness probe inside the guest using an injected test unit over TLS to correctly reach loopback-bound kiosk proxy and announce readiness on serial console.
5. `tests/unit/test_installer_contract.py`: Add unit contract assertions for the fixes.

Fixes #218

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* fix(boot): drop dead host-side kiosk probe and rebuild cleaned zfs sysext

Address review findings on #220:

1. `Justfile`: remove the host-side poll of `http://127.0.0.1:8080/healthz` from
   the `installer-test` smoke recipe. The PR removed the QEMU hostfwd and the
   kiosk proxy terminates TLS (`files/k0s/kiosk/nginx.conf`, `listen 8080 ssl`),
   so this probe can never reach the guest. It could only ever hit an unrelated
   host process on port 8080 and false-pass the smoke test. The in-guest serial
   `KIOSK_CONSOLE_READY` marker is now the sole readiness check.

2. `elements/flatcar/flatcar-zfs.bst`: the preserved `/usr/lib/sysexts/zfs.raw`
   was the untouched upstream image and still carried the
   `After=systemd-sysext.service` udevd drop-in this PR removes from the rootfs.
   Inert today, but the deadlock ordering would return if that raw were ever
   merged for its stated runtime sysext usage. Rebuild the raw with mksquashfs
   from the cleaned install root after the drop-in and tmpfiles cleanup.

3. `tests/unit/test_installer_contract.py`: assert the host-side probe is gone
   from the smoke recipe and that the preserved raw is rebuilt post-cleanup.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): hard preflight abort when bluefin-installer-data is absent (#224)

Add a hard preflight check before any installation begins: if
/dev/disk/by-partlabel/bluefin-installer-data is absent or not a block
device, abort with diagnostic lsblk and /dev/disk/by-partlabel output.
This prevents unattended disk discovery from falling back to an empty
installer partition and potentially selecting and erasing its own
installation media on bare-metal USB drives.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot

Co-authored-by: Jorge Castro <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(justfile): probe install-vm kiosk readiness from inside the guest (rebased) (#227)

* fix(justfile): probe install-vm kiosk readiness from inside the guest

The kiosk proxy publishes hostPort 8080 on the guest's loopback only
(files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml) and
terminates TLS (files/k0s/kiosk/nginx.conf, `listen 8080 ssl`). This
broke install-vm's readiness wait and advertised access URLs the same
way #218/#220 diagnosed for the installer-test smoke recipe:

1. A QEMU/libslirp hostfwd with an unspecified guest address is
   delivered to the guest's DHCP address, never to 127.0.0.1, so
   hostfwd=tcp::8080-:8080 could never reach the loopback-bound proxy.
2. The host-side poll spoke plain HTTP against a TLS listener.
3. The printed LAN/local access URLs and xdg-open call were therefore
   unreachable.

Probe readiness from inside the guest instead: inject a oneshot
systemd unit via an smbios(11) credential that polls the proxy over
TLS from inside the guest and echoes a marker on the system console,
which is captured to a host-side serial log file the host polls
instead of curl. This is the same in-guest-probe approach proposed
for test-installer-artifact in #220 (not yet merged); test-installer-
artifact itself still probes from the host on this branch. Order
ttyS0 last on the kernel cmdline so /dev/console resolves to the
serial device the host captures, matching the ordering
test-installer-artifact already uses for the same reason.

The in-guest probe deliberately does not parse the /healthz response
body: curl --fail already fails on a non-2xx response, which is
exactly "healthz ok and / returns 200", and it avoids embedding quotes
inside a systemd ExecStart= line, whose escaping rules differ from a
plain shell.

Drop the non-working direct 8080 hostfwd and the LAN/local URLs and
xdg-open call that depended on it; print instructions for reaching the
console from the host over an SSH tunnel through the existing
hostfwd=tcp::2222-:22 forward instead, per the issue's suggested
direction. Preserve the unrelated hostfwd=tcp::6443-:6443 (Kubernetes
API) forward.

Correct the kiosk proxy manifest comment, which still claimed the VM
flow reaches the proxy via the existing hostfwd.

Fixes #222

Assisted-by: Claude (Anthropic) via Hive
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(os): correct login banner and de-brittle the jq test assertion

files/os/issue.d/40-kubestellar.issue still advertised
'KubeStellar Console: http://\4:8080/ (or http://127.0.0.1:8080 from
host)', which is doubly wrong after this PR: the proxy binds guest
loopback and terminates TLS, and the direct 8080 hostfwd is gone. Fix
in-scope rather than deferring, since the fix is one line and the PR
already changed the reachability story the banner describes: point at
https://127.0.0.1:8080/ and name the SSH tunnel the Justfile recipe
now prints instead.

Anchor test_install_vm_kiosk_readiness.py's jq-absence check to the
ExecStart= line instead of the whole recipe body, so a future comment
or variable elsewhere in the recipe containing 'jq' cannot fail it for
the wrong reason, and it stays anchored to the actual contract: the
ExecStart command line has no jq in it.

Verified test_install_vm_kiosk_readiness.py's four module-level tests
directly (no pytest in this environment): all pass, including the
updated assertion. test_console_issue.py needs PyYAML, unavailable
here; its one-line assertion update is a straight text match against
the corrected banner and needs a real run in CI before merge.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(os): reformat login banner for TTY legibility

The banner line grew from 71 to ~152 chars and would wrap badly on an
80-column console -- the entire point of /etc/issue.d is legibility
at a getty prompt. Split it across two lines and dropped the markdown
backticks, which render literally there:

  KubeStellar Console: https://127.0.0.1:8080/ (loopback, TLS only)
  Remote access: ssh -L 8080:127.0.0.1:8080 <user>@<host>

Added an assertion for the tunnel line to
tests/unit/test_console_issue.py.

Mutation-tested the ExecStart jq guard against the real Justfile
instead of an in-memory copy: piped the echo through 'jq -R .',
confirmed test_install_vm_probes_readiness_inside_the_guest_over_tls
fails, reverted. Inverse check: added a harmless '# jq' comment
elsewhere in the install-vm recipe, confirmed the test still passes --
proving the anchor to the ExecStart= line, not just narrowing the
match by luck.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(justfile): verify target partition layout before marking install complete

Assert that bluefin-server-root-a and var partitions exist on TARGET_RAW
before touching INSTALL_COMPLETE, preventing failed QEMU installer runs
from falsely marking installation as finished.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(justfile): use sfdisk --json to parse regular target disk partition table in install-vm

lsblk does not inspect partitions inside a regular file image; use sfdisk
--json and jq to parse partitiontable.partitions.name instead.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(justfile): mount /var and bound readiness wait in install-vm

The installed target carries no fstab entry for /var and gpt-auto cannot
mount it, so the install-vm boot needs the same SMBIOS fstab.extra
credential test-installer-artifact passes. Without it, k0s state (and the
kiosk proxy it serves) is missing and the readiness marker never appears.

Bound the readiness wait with INSTALL_VM_READY_DEADLINE (default 600s)
and dump the serial log tail on timeout or premature QEMU exit, so a
broken guest boot fails loudly instead of hanging forever.

Also document why the target partition layout is verified before marking
the install complete, and tell the operator to re-run the installer when
an interactive install was aborted.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Jorge Castro <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix: generate SSH host keys on device, not in the published DDI (#216)

Drops build-time ssh-keygen calls from elements/oci/bluefin-server-ddi.bst so the published DDI carries no private host keys. Adds ssh-host-keygen.service and sshd.service drop-in to generate host keys on first boot before sshd starts.

Reviewed-by: Danathar
Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci: include scripts/** in unit-tests workflow paths filter (#234)

Include scripts/** in pull_request and push paths filters for unit-tests workflow.

Reviewed-by: castrojo
Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: appliance identity, mDNS, and Bash prompt design (#208)

Appliance identity, mDNS, and Bash prompt design spec.

Reviewed-by: castrojo
Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(renovate): disable the homebrew manager for binary-asset formulae (#211)

Formula/kc-agent.rb and Formula/bluefin-kubestellar.rb pin per-architecture
release binaries with one sha256 each. Renovate's built-in homebrew manager
only models source-archive formulae: it rewrites the url to
https://github.com/<owner>/<repo>/archive/refs/tags/<tag>.tar.gz and carries
over a single checksum. The result leaves `bin.install "kc-agent"` with no
binary to install, and the other three architectures unresolvable.

That is not hypothetical — it is exactly what #209 proposed:

    - url ".../releases/download/v0.3.41/kc-agent_0.3.41_darwin_amd64.tar.gz"
    + url ".../archive/refs/tags/v0.3.42.tar.gz"

Bumping these formulae means fetching four assets and four checksums, so it
stays a manual change until a generator exists. Disabling the manager stops
the bot from filing PRs that cannot be merged.

Validated with renovate-config-validator 44.103.6, the version the repository
bot runs.

Assisted-by: Claude Opus 4.5 via GitHub Copilot

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix(installer): harden flash-installer with refusal guards, write verification, and GPT backup relocation (#225)

* fix(installer): constrain flash image lookup to top-level and verify PARTLABEL

Constrain flash-installer image lookup to dist/ maxdepth 1 to prevent
flashing stale nested checkout artifacts. After dd, trigger and settle
udev and verify the bluefin-installer-data PARTLABEL is present on the
target device. Add bats unit test coverage.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): make missing PARTLABEL fatal after flashing

Ensure flash-installer exits nonzero and suppresses the success message
when bluefin-installer-data PARTLABEL is not observed after writing.
Add positive and negative Bats contract tests.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): reread partition table via blockdev/partprobe before verifying partlabel

Invoke blockdev --rereadpt / partprobe after dd so kernel partition
tables are updated before udev triggers and the label verification runs.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): abort if partition table reread fails after flashing

Ensure flash-installer aborts with an error if both blockdev --rereadpt
and partprobe fail to update the kernel partition table, preventing stale
partition label checks. Add Bats unit test.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): harden flash-installer safety, write verification, and partition checks

- Reject disks backing the running system (/ and /sysroot).
- Reject disks with mounted partitions.
- Verify archive integrity with zstd -t before prompting.
- Run dd pipeline with pipefail.
- Flush buffers and verify payload bytes and sha256 read back from device.
- Relocate GPT backup header and verify with sfdisk --verify.
- Require exact match on bluefin-installer-data PARTLABEL with grep -Fxq.
- Add full unit test coverage in tests/unit/flash-installer_test.bats.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(installer): add pipefail, digest mismatch, and sfdisk verify Bats tests

Model dd of= by consuming stdin to EOF without masking SIGPIPE, test
failing zstd -dc pipeline abort under pipefail, test corrupt readback
digest mismatch abort, and distinguish sfdisk --relocate from --verify.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): fail closed on mount checks and stop interpolating image path

Address review findings on the flash-installer hardening:

- The mounted-partitions guard used lsblk's MOUNTPOINTS column, which needs
  util-linux >= 2.37. On older hosts lsblk failed, the failure was swallowed,
  and the guard passed silently. Fall back to the MOUNTPOINT column and refuse
  the device outright when neither column can be read.
- findmnt reports btrfs roots as /dev/sdXn[/subvol], which lsblk cannot
  resolve, so the running-system-disk refusal was skipped. Strip the
  subvolume suffix before resolving the parent disk.
- The image path was interpolated into a root shell command. Pass the image
  and the device as positional arguments instead.
- Refresh the bats header comment to describe the current guards and stubs,
  and document the added guards in docs/skills/ddi-installer-build.md.
- Add coverage for the btrfs subvolume source, the MOUNTPOINT fallback, the
  unreadable-mount-state refusal, the multi-image refusal, and the absence of
  image interpolation in the root shell.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): resolve flash target to its parent disk and hash in one pass

The running-system guard compared the target's KNAME against the parent
disk that / resolves to via PKNAME. Naming a partition node made the two
sides disagree by construction: targeting /dev/sda3 while / lives on
/dev/sda2 compared sda3 against sda, so the guard passed and only the
mount check stood in the way. Resolve the target through PKNAME first,
falling back to KNAME for whole-disk nodes, so both sides are parent
disks. Covered by a new Bats case that fails without the fix.

Also collapse the two pre-write decompressions into one: the digest and
the byte count now come from a single `zstd -dc` piped through `tee` into
a FIFO, with an explicit `wait` on the hashing subshell so the digest is
fully written before it is read. This roughly halves pre-write time on
multi-GB images.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Jorge Castro <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(kiosk): executed coverage for kiosk-gate.js — tests/unit/test_kiosk_gate_behavior.py (#232)

kiosk-gate.js gates the kiosk console while a session exists but the
kc-agent is unreachable. It had no executed coverage: the existing suite
only greps the source for literals, so inverting the session check,
dropping the dedupe guard, trapping keydown with no gate up, ignoring a
non-ok health response, or sending credentials to the agent all keep it
green.

Run the real script in node:vm against a stub DOM and assert the
behaviour instead.

Closes #231

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* revert: "docs: appliance identity, mDNS, and Bash prompt design (#208)" (#235)

This reverts commit 3f7959217ab396bb30576f4fd013f2202c166707.

The design specification contains an unviable mechanism: systemd-hostnamed
writes /etc/hostname using WRITE_STRING_FILE_ATOMIC (open temporary file
and renameat), which destroys the /etc/hostname symlink pointing to
/var/lib/bluefin-server/hostname upon static hostname mutation. Reverting
until an architecturally sound mechanism is reviewed independently.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot

Co-authored-by: Jorge Castro <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(creds): extend systemd-creds provisioning to SSH keys, network, firstboot, and TPM2 (#221)

Wire systemd credential provisioning for network configuration, first-boot
locale/timezone/hostname settings, and SSH key provisioning:
- bluefin-firstboot-credentials.service: runs systemd-firstboot non-interactively
  when firstboot.* credentials are present, setting the live hostname before networkd
  starts and stamping /etc/.bluefin-firstboot-credentials so it runs once.
- 10-bluefin-credentials.conf: drop-in for systemd-network-generator importing
  network.conf.*, network.link.*, network.netdev.*, and network.network.* credentials
  under /run/systemd/network/, falling back to DHCP when absent.
- Presets enabling bluefin-firstboot-credentials.service and
  systemd-network-generator.service.
- os-creds-prov.bst: converted to manual element to stage sysusers credentials and
  systemd credential units, with strip-binaries disabled.
- Document TPM2 sealing, ESP and hypervisor delivery for bare metal and VMs,
  and tmpfiles.extra payloads for SSH key provisioning.
- Contract unit tests in test_credentials_provisioning.py.

Fixes #138
Supersedes #144

Assisted-by: Gemini 3.8 Flash via GitHub Copilot

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* arch: make include/flatcar.yml the single source of truth for Flatcar pins (#143)

* arch: make include/flatcar.yml the single source of truth for Flatcar pins

include/flatcar.yml now declares every Flatcar upstream pin for the target
release - the version, kernel vermagic, and the vmlinuz / container / zfs
source sha256s - and every Flatcar element reads those values through
%{...} variable substitution instead of restating a literal.

- flatcar-kernel.bst and flatcar-zfs.bst reference
  %{flatcar-kernel-vmlinuz-sha256}, %{flatcar-kernel-container-sha256} and
  %{flatcar-zfs-sha256} instead of hard-coded refs.
- Add .github/scripts/check-flatcar-version.py, a grep-based gate (in the
  spirit of check-release-version.py) that fails the build when any pinned
  literal appears outside include/flatcar.yml in elements/ or files/.
- Wire the gate into 'just validate'.
- Add unit tests covering declared_pins(), the scan, and the live-tree
  invariant.

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* fix(flatcar): quote variable-substituted source refs

A bare `%` is a reserved YAML indicator and cannot start a plain scalar,
so `ref: %{flatcar-...-sha256}` made elements/flatcar/flatcar-kernel.bst
and flatcar-zfs.bst unparseable. Both bst and the pytest suite failed to
load the elements, breaking the unit, build and build-kernel jobs.

Quote the refs so they parse; BuildStream expands %{...} in source
configs, so the pins still resolve from include/flatcar.yml.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(kiosk): console shows live local cluster state on first boot (#214)

* fix(kiosk): console shows live local cluster state on first boot

On a freshly installed single-node k0s cluster no client kc-agent is
connected yet, so the KubeStellar console fell back to mock/demo cards
instead of the real cluster.

- Set NO_LOCAL_AGENT so the console reads the local embedded k0s API
  directly from this pod's ServiceAccount instead of waiting for a
  connected kc-agent.
- Set SKIP_ONBOARDING so the kiosk shows the dashboard immediately.
- Add a read-only ClusterRole bound to the console ServiceAccount so the
  in-cluster reads for nodes, namespaces and pods succeed.

Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* fix(kiosk): correct limitranges RBAC resource and SKIP_ONBOARDING doc

- 42-kubestellar-console-rbac.yaml: limitquotas is not a Kubernetes
  resource; the intended grant was limitranges. Misspelled resources are
  silently ignored in RBAC rules, so LimitRange reads were forbidden.
- SKIP_ONBOARDING comment: it skips the onboarding questionnaire, not
  sign-in (sign-in is bypassed by DEV_MODE/DEV_USER_LOGIN).
- NO_LOCAL_AGENT comment: the console forces this in-cluster regardless;
  the real fix for #193 is the ServiceAccount + read RBAC.

Addresses kubestellar-hive review findings on #214.

* fix(kiosk): scope console RBAC to real resources and correct comments

Follow-up to the #214 review findings.

- 42-kubestellar-console-rbac.yaml: replicationcontrollers is a core
  ("") resource, not an apps resource. Listed under apps it was silently
  ignored in the RBAC rule, exactly like the limitquotas typo, so the
  grant never applied. Move it to the core group.
- Document why the ClusterRole reads configmaps, events, limitranges and
  resourcequotas: the console dashboard renders them (namespace overview
  and ConfigMap cards). Record that verbs stay read-only, secrets are
  excluded, and the console is only reachable via the 127.0.0.1-bound
  kiosk proxy.
- 40-kubestellar-console.yaml: attribute the fix to the ServiceAccount
  plus read RBAC. The console backend already forces NO_LOCAL_AGENT on
  in-cluster, so the env var only records intent. Note that
  SKIP_ONBOARDING skips the questionnaire and DEV_MODE/DEV_USER_LOGIN
  bypasses sign-in.
- tests: pin every ClusterRole resource name to the real resource for
  its apiGroup and assert secrets are never granted, so a misspelled or
  misgrouped resource fails the suite instead of silently dropping the
  grant.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(k0s): pin argocd image by digest and upgrade past EOL v2.12.3 (#212)

* fix(k0s): pin argocd image by digest and upgrade past EOL v2.12.3

argocd-server ran quay.io/argoproj/argocd:v2.12.3 (the EOL 2.12 line with
unbackported CVEs) by mutable tag, so a re-pointed or compromised tag could
silently change what runs. Upgrade to the latest supported v2.14.9 and pin by
digest so the running image is integrity-checked.

--insecure serves plaintext HTTP; keep it only because the argocd-server
Service is ClusterIP-only, and document that it must stay internal (terminate
TLS at a proxy, then drop the flag) rather than silently disabling it and
breaking the deployment.

sec-check #74

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* fix(k0s): repin argocd to the live v2.14.9 manifest digest

The previously pinned digest sha256:af3dfddc… is a legacy schema-v1
manifest that returns 404 on digest lookup; kubelet pulls tag@digest
with manifest-list accept types, so argocd-server lands in
ImagePullBackOff. Verified against quay.io: af3dfddc… 404s on every
content type, while the v2.14.9 index sha256:f5885af0… resolves for all
types and carries the amd64 child 78e45540….

Note: the 2.14 line is itself past EOL (3.x is current); the digest fix
removes the hard regression but a maintainer may prefer a supported 3.x
release.

---------

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* test(installer): behavioural contract for the QEMU boot-proof readiness gate (#219)

* test(installer): behavioural contract for the QEMU boot-proof readiness gate

test-installer-artifact (the recipe show-me-the-future delegates to) polls
/healthz and / after booting the installed target in QEMU, and only breaks
out of the loop when the healthz body reports status "ok" AND the root
probe answers HTTP 200. Nothing exercised that decision logic before this
change: a prior attempt at issue #130 (PR #179) sliced the recipe body out
of the Justfile as text and asserted substrings, so every assertion stayed
green whether the underlying checks were correct, weakened, or deleted.

This suite instead runs the real recipe in a private sandbox copy of the
Justfile (the tests/unit/flash-installer_test.bats pattern), with
qemu-system-x86_64, curl and zstd replaced by logging stubs on PATH so
'just test-installer-artifact' actually executes. It proves, by running
the recipe rather than describing it:
- readiness requires both conditions together, not either alone
- a failing/malformed healthz body short-circuits before the root probe
  is ever called
- a QEMU process that dies before answering is detected and reported,
  independent of curl/jq state
- the serial log tail is dumped on both failure paths (timeout and death)

One substitution is applied to the sandboxed Justfile, asserted the same
way flash-installer_test.bats asserts its own: the OVMF_CODE lookup only
searches hardcoded absolute host paths, none of which exist on an
unprivileged CI runner, so the recipe would abort before ever reaching the
readiness loop. An env-gated candidate is added to the front of the
existing first_existing() call; unset, it defaults to a nonexistent path,
so production behaviour is unchanged.

Verified non-vacuous: dropping the root-code check (collapsing the two
conditions into one, as a plausible regression) fails 2 of the 10 cases;
restoring it is green again.

os-base: flatcar (#130's acceptance line) does not exist on main yet — no
such build option exists anywhere in the tree, and the whole Flatcar-base
cutover cluster (#132/#133/#136/#140/#163/#191) is parked 2-discussing
pending the hard-rule-1 ADR decision on #133. This does not close #130; it
pins the readiness contract that show-me-the-future/test-installer-artifact
already enforce today, independent of which payload eventually boots
under it, so the gate stays guarded while that decision is pending.

Assisted-by: Claude via Claude Code
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* test(installer): rewrite the readiness contract around the in-guest probe

#220 moved the readiness decision into the guest: a systemd unit injected
over an SMBIOS credential polls the kiosk proxy and prints
KIOSK_CONSOLE_READY on the console, and the host only greps the serial log
for that marker. Four tests still drove a host-side curl loop that no
longer exists and failed at the merged head.

Drive the host decision through the serial log instead: the marker present,
a progress line only, no marker at all, and a near-miss substring. Guard
the two conditions and the credential delivery as text in the unit, and
assert this recipe runs no host-side probe (scoped to its own body, since
install-vm legitimately forwards its own ports). Four recipe mutations
each fail between one and four tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK
Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>

* fix(test): scope the SMBIOS credential guard to test-installer-artifact

install-vm (#227) reuses the same kiosk-ready credential and cmdline
mechanism, so it now carries its own copy of both strings. A whole-file
grep -c counts both recipes' copies and fails against current main
(count 2, not 1). Scope the guard to this recipe's body, the same way
the host-probe guard just below it already does.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(test): describe what the readiness suite actually proves

The file header and curl-stub comments described an earlier revision of
this suite, from before the readiness probe moved into the guest (#220):
they claimed the two-condition check was exercised by routing requests
through the curl stub and parsing the body with jq. Nothing does that at
head, and the check is jq -r .status, not jq -e.

Rewrite the header to state the split the suite actually implements: the
host-side loop (marker, target death, deadline, log tail) is executed for
real against the sandboxed recipe, while the in-guest two-condition check
is pinned by fixed-string greps because it can only run inside the booted
guest. Correct the near-miss test's comment, which claimed exact-line
matching that the recipe's unanchored grep does not do, and the
refute_host_probe_attempted comment, which described a URL-shaped match it
no longer performs.

Drop the CURL_HEALTHZ_EXIT/CURL_HEALTHZ_BODY/CURL_ROOT_EXIT/CURL_ROOT_CODE
knobs: no test sets them, the host never invokes curl, and one test asserts
exactly that. The stub keeps its call logging, which refute_host_probe_
attempted needs, and now fails loudly if the host ever does probe.

No behavioural assertion changes; 12/12 still pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: hive-scanner <223556219+Copilot@users.noreply.github.com>

* feat(installer): add opt-in PXE network DDI pull (#229)

Extend the installer wrapper (bluefin-sysinstall) to honor inst.* kernel
cmdline parameters so PXE/netboot clients can download the OS DDI over the
LAN instead of reading the embedded bluefin-installer-data partition:

- inst.ddi_url=<url> fetches the zstd-compressed DDI via curl after
  systemd-networkd-wait-online brings up DHCP.
- inst.ddi_sha256=<hex> is mandatory when inst.ddi_url is set; the download
  is verified before the target disk is touched (fail closed).
- inst.target_disk=/dev/... overrides first-writable-disk auto-detection.

The downloaded DDI is stream-decompressed with zstd and fed into the native
systemd-sysinstall flow by staging a temporary /usr/lib/repart.sysinstall.d/
override whose 20-root-a.conf CopyBlocks= points at the downloaded image —
no custom installer logic. Without inst.ddi_url the embedded path is
byte-for-byte unchanged, and the hard preflight for the embedded partition is
skipped under PXE.

Add curl + zstd to installer-stack.bst. Part 1 (standalone PXE kernel/initrd
export) already landed in main; this completes issue #14 Part 2.

Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K via GitHub Copilot

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* build: os-base option to compare FSDK payload against Flatcar reference (#163)

* build: add os-base option to build-ddi for FSDK/Flatcar parity harness

projectbluefin/server#129 wants an os-base: fsdk | flatcar-reference
option so the FSDK-composed payload and the imported Flatcar reference
can be built from one tree and diffed.

build-ddi now takes an OS_BASE parameter (default fsdk) and routes to
the matching element:

  fsdk            -> oci/bluefin-server-ddi.bst (shipped payload)
  flatcar-reference -> oci/bluefin-server-ddi-flatcar-reference.bst,
                       added alongside the #126 imported Flatcar tree

The flatcar-reference route refuses with a pointer to #126 until that
element exists, and an unknown base is rejected. The default path is
unchanged, so CI's 'just build-ddi' keeps building the FSDK payload.

Adds a bats test covering the routing and guards.

Assisted-by: lemonade/Ornith-1.5-35B-A3B-GGUF via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

* fix: resolve flatcar-reference element against elements/ in build-ddi

The build-ddi os-base=flatcar-reference guard tested
`-f "oci/bluefin-server-ddi-flatcar-reference.bst"`, but project.conf
declares `element-path: elements`, so the element file lives at
`elements/oci/bluefin-server-ddi-flatcar-reference.bst`. The guard could
therefore never be satisfied and the flatcar-reference route was
permanently unreachable, matching the Justfile's existing convention of
referencing element files as `elements/...` on disk.

The bats test asserting the refusal passed for the wrong reason: it ran
against the repo root where no `oci/` directory exists at all. It now
runs against a sandbox that holds a decoy at the bare `oci/` path and
nothing under `elements/oci/`, so it fails if the guard resolves the
wrong path and stays a genuine refusal test once the #126 element lands.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(build): stop interpolating os-base into the build-ddi recipe body

just substitutes {{...}} textually into the recipe body before bash parses it,
so `case "{{OS_BASE}}"` made the argument executable:

    $ just build-ddi '$(echo PWNED >&2)'
    PWNED
    PWNED
    ERROR: unknown os-base '' (expected 'fsdk' or 'flatcar-reference')

It ran twice -- once at the case, once in the error echo -- so even the
rejection path executed it, and the message that was supposed to name the bad
value printed empty because the substitution had consumed it.

Exporting the parameter (`build-ddi $OS_BASE="fsdk"`) hands it to the recipe
as an environment variable, so bash only ever sees "$OS_BASE" and never expands
the caller's text. Same invocation now prints the payload verbatim and exits 1.

Only the local invoker could reach this, and it matches the existing {{ARGS}}
pattern in the bst recipe, but os-base is a fixed enum of two values and has no
reason to be interpolated at all.

Adds two regression tests: one asserts no {{OS_BASE}} remains in the recipe
body, the other passes $(touch <marker>) and asserts the marker is absent. The
marker matters -- the payload's own text necessarily appears in the rejection
message, so a string assertion could not distinguish echoed from executed.
Both fail against the previous Justfile, along with the parameter-shape guard.

Also documents os-base in docs/skills/ddi-installer-build.md, which still
showed a bare `just build-ddi` and is what operators drive the #129 parity
harness from: a Payload base section covering both values, the #126 refusal
before the reference element exists, and the fact that export-ddi does not
forward os-base, so the reference tree stays a comparison input and never
becomes a release artifact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

---------

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* feat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base (#140)

* feat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base

- Add elements/flatcar/flatcar-usr.bst to import Flatcar /usr as the OS payload base
- Mirror flatcar-zfs.bst with single pinned remote source flatcar-container.tar.gz
- Set strip-binaries to empty string to prevent stripping prebuilt binaries
- Flatten Flatcar's nested usr/lib/modules/<kver>/<kver>/ layout
- Explicitly remove Flatcar's update and provisioning stack with replacement comments
- Preserve usr/lib/flatcar/bootengine.img for stage-2 kernel boot
- Add contract tests in tests/unit/test_flatcar_usr.py

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* fix(flatcar): strip /usr/lib/modules to avoid collision with flatcar-kernel

Removes dead nested module flattening code and strips /usr/lib/modules
from flatcar-usr.bst completely so kernel modules are exclusively
managed by flatcar-kernel.bst.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(flatcar): remove update-engine and cloudinit systemd units and drop-ins

Address review feedback:
- Explicitly remove update-engine and update-engine-stub units, timers, and multi-user.target.wants symlinks
- Remove enable-oem-cloudinit and oem-cloudinit units and wants symlinks
- Remove locksmithctl, ignition-rmcfg, and update tmpfiles configuration
- Update unit tests to mock and assert absence of all removed units, binaries, and configs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* style(flatcar): drop unused findutils build-dependency from flatcar-usr.bst

The install-commands script only uses mkdir, tar, and rm, making findutils
unnecessary. Update unit contract tests accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

---------

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>

* fix(flatcar): use flatcar-kernel-container-sha256 variable in flatcar-usr.bst (#236)

Co-authored-by: castrojo <castrojo@users.noreply.github.com>

* refactor(arch): split release-version into installer and Flatcar axes (#136)

* refactor(arch): split release-version into installer and Flatcar axes

Split the single release-version axis in project.conf into two
independent axes:
  - installer-version: FSDK point release describing the installer
  - flatcar-version: Flatcar LTS release describing the OS payload

Update .github/scripts/check-release-version.py to independently
validate both axes against their respective pins (freedesktop-sdk.bst
junction ref for FSDK, include/flatcar.yml for Flatcar). Update unit
tests in tests/unit/test_release_version.py to cover the two-axis
contract, and update element asset naming and os-release metadata.

Closes #125

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* docs: update stale release-version references and clarify flatcar-version declaration

Updates avoid-over-engineering.md and ddi-installer-build.md to reference
the split axes (installer-version and flatcar-version), noting that release
tags track the installer axis while the DDI and UKI track Flatcar. Also adds
a comment in project.conf explaining why flatcar-version is deliberately declared
there in addition to include/flatcar.yml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(arch): declare flatcar-version only in include/flatcar.yml

project.conf restated flatcar-version alongside include/flatcar.yml, whose
own header declares it the single source of truth for the Flatcar axis.
Add include/flatcar.yml to project.conf's existing (@) include list instead,
so %{flatcar-version} and %{flatcar-kver} resolve project-wide from one
declaration, matching the k0s axis precedent.

check-release-version.py no longer compares two copies of the same fact: it
validates project.conf's installer-version against the freedesktop-sdk
junction ref and that include/flatcar.yml declares a well-formed
flatcar-version. Stale standing fact in docs/skills/index.md replaced with
the two-axis description this refactor introduces.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* refactor(arch): update check-k0s-version guard and k0s.yml comment for split axes

- Guard check-k0s-version against %{installer-version} and %{flatcar-version}
  in addition to %{release-version}
- Update header comment in include/k0s.yml referencing split axes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

* fix(test): update test_main_rejects_fname_on_the_os_release_axis assertion string

---------

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jorge Castro <jorge.castro@gmail.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>

---------

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Signed-off-by: hive-agent <hive-agent@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>
Co-authored-by: James Reilly <jreilly1821@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>
Co-authored-by: Jorge Castro <jorge.castro@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: hive-agent <hive-agent@users.noreply.github.com>
Co-authored-by: ci-maintainer <ci-maintainer@users.noreply.github.com>
Co-authored-by: Bob Killen <killen.bob@gmail.com>
Co-authored-by: kubestellar-hive[bot] <280983584+kubestellar-hive[bot]@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: mendezr <18744521+mendezr@users.noreply.github.com>
Co-authored-by: Danathar <doug.baggett@gmail.com>
castrojo added a commit that referenced this pull request Sep 25, 2026
…, and OVMF boot test (#243)

* test(installer): add test-installer-boot-usb recipe for bare-metal USB OVMF testing

Attach installer media via emulated xHCI USB drive in QEMU and boot via
OVMF firmware. Verify target GPT partition table using sfdisk --json and
jq for bluefin-server-root-a and var partitions. Add contract unit tests.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(installer): attach target disk as virtio-blk-pci device with bootindex=2 and enforce timeout

Enforce INSTALLER_TIMEOUT via timeout utility in test-installer-artifact
stage 1. Define target disk using -drive if=none and attach device
virtio-blk-pci with bootindex=2 so boot priorities are valid for QEMU.
Update contract tests.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): add 30s bounded wait for installer partition, mirror service output to serial console, and add edk2-i386-vars pairing

- In bluefin-sysinstall, wait up to 30s using an absolute SECONDS deadline with periodic udevadm trigger and settle for bluefin-installer-data to be enumerated on slower USB controllers before failing.
- Mirror systemd-sysinstall.service output to journal+console so installer execution progress and error diagnostics are captured on serial console.
- Deduplicate test-installer-boot-usb by having it delegate directly to test-installer-artifact.
- Add Homebrew and system edk2-i386-vars.fd to OVMF vars candidates for x86_64 firmware code pairing with fatal fallback.
- Strictly scope test-installer-boot-usb unit contract assertions and assert exact dynamic settle timeout.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): drop dead PXE copies, restore OVMF_VARS fallback

Three review findings from #228:

- test-installer-artifact still copied dist/bluefin-server-pxe-vmlinuz-*
  and -initrd-*.cpio.gz into $WORKDIR, but the recipe no longer passes
  -kernel/-initrd, so nothing consumed them. Under set -euo pipefail the
  copies made the test fail on hosts that have the installer image but no
  PXE artifacts. export-installer already asserts the PXE pair exists, so
  that coverage is not lost.

- Replacing the truncate-based blank variable store with exit 1 meant
  hosts shipping only OVMF_CODE could no longer run the test or install-vm.
  Restore the fallback in both recipes and log a warning, keeping the
  edk2-i386-vars.fd candidates this PR added.

- docs/skills/ddi-installer-build.md did not list test-installer-boot-usb
  and still implied the old direct-kernel behaviour of
  test-installer-artifact. Document the OVMF/xHCI USB boot path, the alias,
  and the OVMF_VARS fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): keep interactive installs on tty0, harden partition check

Four review findings from #228:

- StandardOutput/StandardError=journal+console routes installer output to
  /dev/console, which follows the last console= argument (ttyS0), so an
  interactive bare-metal install printed to serial instead of the attached
  display. TTYPath= is only honoured for Standard*=tty, so it did not
  counteract this. The wrapper now redirects its own output to /dev/tty0
  when 'unattended' is absent from the kernel command line and a local VT
  is writable, which restores the display for interactive installs while
  unattended/automated runs keep the serial capture that the drop-in was
  added for. Headless serial-only interactive installs fall through to
  journal+console unchanged.

- The target-disk partition check piped sfdisk into jq with 2>/dev/null and
  || true, so a missing or broken tool was reported as "Target disk did not
  receive expected partition layout". sfdisk and jq are now checked up front
  and their failures reported separately from a genuinely unpartitioned disk.
  jq was an undeclared host dependency of this recipe; it is now named in the
  error path and in docs/skills/ddi-installer-build.md.

- The comment added in the previous commit credited export-installer with
  asserting the PXE pair was exported; those assertions are in export-pxe.

- Restore the blank line before
  test_interactive_installer_uses_local_virtual_console (PEP8 E302) and
  separate the test-installer-boot-usb recipe from the install-vm comment
  block.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(installer): keep the PXE boot path under test, tighten the tty0 guard

test-installer-artifact now takes an INSTALLER_BOOT_MODE of usb (default,
firmware boots the raw image off an emulated xHCI USB drive) or pxe (QEMU boots
the exported vmlinuz/initrd pair with -kernel/-initrd/-append). Everything after
the install is shared. test-installer-boot-usb and test-installer-boot-pxe are
the named entry points, so the direct-kernel path the previous revision dropped
is exercised again rather than merely existence-checked by export-pxe.

Document next to the -smbios flag that systemd-stub ignores
io.systemd.stub.kernel-cmdline-extra under Secure Boot, so a secure-boot OVMF
build boots interactive and hangs until the install timeout, and point at pxe
mode as the way to test unattended installs on such firmware.

Restrict the interactive /dev/tty0 redirect to boots where the kernel actually
registered tty0 as a console. A writable VT device exists on serial-only bare
metal too, so the -w test alone sent the whole interactive install to a display
nobody was watching.

Correct the ci-tooling sentence that still claimed the uploaded PXE artifacts
feed test-installer-artifact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: generate console JWT secret at first boot, not in git

The kubestellar-console-github-oauth Secret committed a usable jwt-secret
(default-jwt-secret-123456789012345678901234). JWT_SECRET signs console
session tokens, so a value published in git lets anyone forge valid
sessions on every default install.

Mirror the kubeflex-postgres pattern (#98): the committed 01- manifest now
creates only the namespace, and a first-boot generator writes
02-kubestellar-console-secret.yaml with a random 256-bit jwt-secret and
empty OAuth credentials (consumed with optional: true). Idempotent, 0600,
runs from k0s-first-boot.service before k0scontroller starts.

Existing installs rotate on upgrade: tmpfiles C+ replaces the stale 01-
copy (dropping the published Secret) and the generator creates the 02-
file fresh, invalidating any tokens forged with the public value.

Closes #72

Assisted-by: Claude Fable 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix(boot): rename systemd presets to sort before Flatcar's 99-default 'disable *'

Flatcar's /usr ships /usr/lib/systemd/system-preset/99-default.preset
containing 'disable *'. systemd reads preset files in lexicographic
filename order and the first matching line wins, so every Bluefin
preset named zz-enable-*.preset sorted after it and was ignored at
first-boot preset-all. k0s-first-boot.service was therefore never
enabled: the QEMU smoke test sat at 'kiosk-ready: waiting k0s=inactive'
for the full 900s window on every run since the Flatcar payload switch.

Rename the presets to 20-bluefin-*.preset so they sort before
99-default.preset, and add a unit test asserting no staged preset can
regress behind the Flatcar catch-all again.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* test: update preset paths to 20-bluefin-*, add sort-order guard, log pod states in smoke probe

- Update the two unit tests that still referenced the removed
  zz-enable-*.preset filenames (the rename commit shipped without them).
- Add test_presets_sort_before_flatcar_disable_all so no future preset
  can sort behind Flatcar's 99-default.preset 'disable *' catch-all.
- Teach the installer smoke readiness unit to print the pod table and
  the last Warning events alongside the existing counters, so a run that
  reaches k0s=active but never gets a pod to Running names the blocker
  in the serial log instead of only counting it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>

* fix(security): force rotation of seeded root password at first login

The DDI seeds a well-known root password ('bluefin', hash public in this
repo) into every shipped image with shadow last-change 19700, so the
credential never expires and is identical fleet-wide. PR #169 already
restored the sshd drop-in hardening (PermitRootLogin prohibit-password,
PasswordAuthentication no), which closes the remote root path; the
remaining gap from #168 is that console bring-up never forces rotation.

Set the shadow last-change field to 0 (the `chage -d 0 root` equivalent):
every login as root — console included — is forced onto a per-install
password before any shell is granted. Update the console banner to
disclose the forced change, and add a unit contract locking in both the
rotation and the SSH no-password-auth policy as a regression guard.

The long-term removal of the baked password (core operator account,
systemd-credentials key provisioning) is tracked by #80; this is the
minimal interim hardening requested by the issue.

Fixes: #168

Hive-Run: #168
Hive-Plan: Security remediation — interim hardening of seeded root credential (issue #168, recommendation item 2)
Hive-Spec: docs/superpowers/specs/2026-09-09-core-login-design.md#interim-shadow-expiry
Assisted-by: GLM (z-ai/glm-5.3-flash) via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mendezr <mendezr@users.noreply.github.com>

* test(installer): surface node, pull, and k0s runtime errors in the kiosk waiter

The preset rename in this PR gets k0scontroller active, but the smoke gate
still times out: every pod sits Pending/ContainerCreating for the full 900s
window (kube-proxy ContainerCreating, kube-router Init:0/2, node tainted
not-ready). The current 30s debug line only shows Warning events, and the
FailedScheduling flood from ten Pending pods pushes the informative events
(image pulls, sandbox creation, CNI) out of the tail -n 5 window.

Extend the in-guest waiter's periodic dump so the serial log carries enough
to diagnose the next layer:
- node status line (kiosk-node) — shows Ready/NotReady and the CRI version
- all event kinds minus the FailedScheduling flood (kiosk-events)
- error/fail/taint lines from the k0scontroller journal (kiosk-k0s), where
  the embedded kubelet and containerd log

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>

* fix(boot): seed /etc/resolv.conf symlink via tmpfiles.d so kubelet can create pod sandboxes

The preset rename got k0s and kubelet running, but every pod sandbox then
failed with 'open /etc/resolv.conf: no such file or directory' and the
console readiness gate timed out (run 36014129412). Flatcar ships the
/etc/resolv.conf symlink inside the image /etc and strips /etc-populating
lines from /usr/lib/tmpfiles.d at image build; this DDI imports only
Flatcar's /usr and first-boots with an empty /etc, so the symlink never
exists.

Recreate Flatcar's stock symlink (/etc/resolv.conf ->
../run/systemd/resolve/resolv.conf, the full resolver list, matching
Flatcar >= 3510.3.0) with a tmpfiles.d rule installed from a new
os-resolv-conf.bst element, and guard it with a unit test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>

* fix(installer): bare-metal rootfs stability, OEM unit cleanup, and preflight hardening

- Set GrowFileSystem=no on bluefin-server-root-a in 20-root-a.conf to prevent
  interrupted first-boot metadata corruption on the immutable root partition.
- Disable experimental features (reflink, rmapbt, parent, exchange) and widen
  headroom in elements/oci/bluefin-server-ddi.bst mkfs.xfs to prevent ENOSPC
  metadata reservation failure during log replay.
- Remove Flatcar oem.mount, dev-disk-by-label-OEM.device, and fsck drop-ins in
  elements/flatcar/flatcar-usr.bst to prevent 90s device timeout on first boot.
- Default INSTALLER_PART="" before preflight in elements/oci/bluefin-server-installer.bst
  so PXE boots under `set -u` do not fail on unbound variable.
- Update contract tests in test_flatcar_usr.py and test_repart_layout.py.

Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): auto-pass target disk when singular and clarify numbered selection

- When exactly one non-installer target disk is present, pass it directly
  to systemd-sysinstall so the user is not prompted for the device path.
- When multiple target disks are present, print a hint that systemd-sysinstall's
  native interactive prompt accepts single-digit menu numbers (1, 2, ...).

Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(installer): replace hardlinked CA tree before DDI copy

Avoid copying BuildBarn CAS-hardlinked certificate files onto themselves when composing the OS DDI. Align installer documentation with the existing no-grow root recipe.

Assisted-by: github-copilot/gpt-6-sol via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>
Signed-off-by: mendezr <mendezr@users.noreply.github.com>
Co-authored-by: Jorge Castro <castrojo@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: ci-maintainer <ci-maintainer@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: sec-check <sec-check@hive.kubestellar.io>
Co-authored-by: mendezr <mendezr@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/scanner Filed or owned by the scanner agent. agent/security Filed or owned by the security agent. hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants