Skip to content

test: assert parsed properties instead of file text; no silent skips in CI - #278

Merged
jeefy merged 12 commits into
projectbluefin:mainfrom
jeefy:test/behavioral-checks
Sep 29, 2026
Merged

jeefy merged 12 commits into
projectbluefin:mainfrom
jeefy:test/behavioral-checks

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Most unit tests grepped config files for substrings or compared them byte for byte. This PR switches the highest-value ones to asserting what systemd, BuildStream or Kubernetes would actually do with those files. It also stops CI from silently skipping tests.

  • Shared parser in tests/unit/_systemd.py (SystemdFile, preset(), tmpfiles()). It reads files the way systemd does:

    • comments and backslash continuations;
    • repeated keys accumulate;
    • empty assignments reset the key, except that dependencies ignore them and one empty Condition*= clears all conditions;
    • drop-ins merge in order;
    • ExecStart= is split into argv with Environment= expanded;
    • the first matching preset rule wins, in filename order.

    test_systemd_parser.py pins these rules so a misread file cannot make a test pass vacuously. configparser can't do this: it keeps only the last of a repeated key.

  • Converted tests:

    • test_k0s_first_boot, test_credentials_provisioning, test_k0s_manifests, test_kubestellar_kiosk and test_repart_layout now assert parsed properties: ordering, conditions, the effective preset verdict, [Match]/[Network], the tmpfiles rule that wins, where an import element lands, and parsed Kubernetes manifests.
    • Two greps became executed checks:
      • The k0s first-boot role switch runs its own sh -c against a stub systemctl, with no token, an empty token and a real token.
      • The kiosk TLS seed runs against a stub openssl: the key must be created under umask 077 and never regenerated.
  • Stale Flatcar comments in test files now describe FSDK:

    • FSDK's own systemd-resolve.conf links /etc/resolv.conf to the 127.0.0.53 stub. The resolv test now requires the Bluefin rule to sort ahead of it.
    • FSDK ships systemd-firstboot. The Bluefin helper stays because it also sets the live kernel hostname before networkd starts.
    • The image contains only /usr.
  • Vacuous test fixed: test_sysext_export_contract parses the export-sysext recipe body. It now requires the recipe to exist and to export oci/k0s-sysext.bst before asserting that no k3s alias is emitted.

  • No silent skips in CI:

    • unit-tests.yml installs nodejs and shellcheck from the runner's Ubuntu archive.
    • Missing node or shellcheck still skips locally but fails when CI=true. This goes through conftest fixtures.
    • On every CI run on main, the 16 os-justfile bats tests were skipped ("unprivileged user+mount namespaces are unavailable"; see run 36510362559). The cause is Ubuntu's AppArmor userns restriction. The workflow now lifts it, and those tests fail instead of skipping when CI=true.
  • Duplicate preset coverage: the kubeadm preset check in test_kubeadm_sysext.py:93 compares three string literals, so it is always true. It doesn't cover this generically, so I left the k0s test in place and widened it to every files/**/*.preset, which now includes the kubeadm and ZFS sysext presets. See Follow-ups.

  • just test-unit without bats: it falls back to docker.io/bats/bats:1.14.0 pinned by index digest. The repo is mounted read-only and CI is forwarded. CONTRIBUTING.md has one line on this.

  • pytest config: pytest.ini adds testpaths = tests/unit and -ra, so skips are listed with their reasons.

Numbers

upstream/main this PR
pytest 371 passed 381 passed (+7 parser tests, +3 role cases)
bats in CI 58 run, 16 skipped 74 run, 0 skipped (the CI run on this PR shows it)
substring greps on raw file text 187 of 650 asserts (29%) 97 of 661 (15%)
exact file-text comparisons 15 9, all on files the firstboot helper writes (the behavior under test)
test_k0s_first_boot.py greps / exact 42 / 2 0 / 0

The remaining greps cover:

  • skill (docs) text;
  • nginx directives;
  • JS and CSS;
  • shell inside .bst install-commands;
  • secret-not-hardcoded negatives;
  • test files owned by open PRs.

Mutation proofs

Each mutation was applied to the payload. The table shows the upstream/main test and the rewritten test run against the mutated tree, which was then restored.

mutation upstream/main test this PR
k0scontroller.service: ExecStart= uses $K0S_ARGS (the controller args are no longer expanded) pass fail
k0s-first-boot.service: role branches swapped (a node with a token enables the controller) pass fail
kiosk proxy: hostIP: 127.0.0.1 removed (published on all interfaces) pass fail
kubestellar-seed: key generated under umask 022 (world-readable key) pass fail
new 80-bluefin-aaa.preset with enable k0s-first-boot.service, sorting before the opt-in file pass fail
console Deployment: DEV_MODE set to "false" pass fail
ZFS sysext preset renamed 50-zfs.preset → 10-zfs.preset (sorts before 20-ignition.preset) pass fail
export-sysext body replaced by a call to a script the test cannot see pass fail
20-wired.network: DHCP=no fail fail
20-bluefin-resolv.conf renamed zz-… (sorts after FSDK's stub rule) fail fail
20-wired.network: comments added and blank lines removed, no semantic change fail (false alarm) pass

Verification

  • python3 -m pytest: 381 passed. It also passes in a fresh detached worktree without files/boot-keys, with and without CI=true.
  • bats tests/unit: 74 ok, 0 skipped.
  • just test-unit passes with a local bats and through the container fallback. Without CI it reports 16 skips (the image has no just); with CI=true it fails, as intended.
  • python3 .github/scripts/docs-checks.py and the three version-check scripts behind just validate pass. No element changes, so bst show was not rerun.
  • actionlint's only finding is the existing ubuntu-26.04 runner label.
  • Every commit passes pytest and bats on its own.

Follow-ups

After the open PRs merge. I left their test files untouched:

Found while doing this, not fixed here:

  • 03-bluefin-countme.preset (installed from projectbluefin/common by os-countme.bst) sorts before 20-ignition.preset. Its enable bluefin-countme.timer therefore wins over an Ignition enabled: false; masking still works. It's the class of problem the preset-order test guards, but that test can't see presets from outside files/.
  • Flatcar narrative remains in payload text: files/os/tmpfiles.d/20-bluefin-resolv.conf, the os-resolv-conf.bst description, the header of files/os/libexec/bluefin-firstboot-credentials, and the os-creds-prov.bst description (touched by fix(security): ship root locked; provision access per node #275). I left these alone because editing them changes image content and cache keys.

castrojo
castrojo previously approved these changes Sep 29, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review: parsed property assertions in tests/unit/_systemd.py correctly replicate systemd file parsing, drop-in overrides, and environment expansion without relying on brittle text substring matching. Unit tests now fail closed when dependencies are missing rather than skipping silently. Full CI test suite and build verification pass.

jeefy and others added 12 commits September 29, 2026 04:01
tests/unit/_systemd.py reads unit files, drop-ins, .network files,
presets and tmpfiles.d the way systemd does: comments and backslash
continuations, repeated keys that accumulate, empty assignments that
reset (all conditions for Condition*=, never for dependencies), first
matching preset rule in filename order, and ExecStart= argv with
Environment= expanded. configparser cannot do this: it keeps only the
last of a repeated key and knows no continuation or reset rules.

test_systemd_parser.py pins those semantics so a test built on the
helper cannot pass vacuously because the parser misread a file.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…perties

Replace substring and byte-for-byte file comparisons with parsed
properties: unit ordering and conditions, the effective preset verdict,
the wired [Match]/[Network] settings, where each import element lands
in the image, and which tmpfiles.d rule wins for /etc/resolv.conf.

The join-token role switch now runs the unit's own sh -c script against
a stub systemctl (no token, empty token, token), so an inverted test or
swapped branch fails; string greps passed either way. The preset order
check covers every preset under files/, sysext presets included.

The resolv.conf comment described Flatcar's image build. On FSDK, /etc
starts empty and FSDK's systemd-resolve.conf links the 127.0.0.53 stub,
so the test now also requires the Bluefin rule to sort ahead of it.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…arrative

bluefin-firstboot-credentials.service, the network-generator drop-in,
the presets and 20-wired.network are now checked as parsed settings
instead of exact file text: the triggering ConditionCredential= set must
equal the imported set, the stamp is touched in ExecStartPost= and gates
the unit, and every shipped .network file sorts after the documented
10-static credential so it can still override the DHCP fallback.

The comments claimed Flatcar's /usr has no systemd-firstboot and that
/bin comes from Flatcar tmpfiles. FSDK ships systemd-firstboot; the
helper stays because it also sets the live kernel hostname before
networkd starts, and the image contains only /usr.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
k0scontroller.service is now checked on its effective command line,
with K0S_CONTROLLER_ARGS expanded from Environment=: the old grep still
passed if ExecStart= stopped referencing the variable. The worker's
join-token argv and condition, the manifest tmpfiles rules, the seed
unit's generators and ordering, and the Argo CD manifest's namespace
are parsed rather than grepped.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The console and proxy Deployments are parsed instead of grepped: each
login flag must be "true" (any 'value: "true"' used to satisfy all of
them), the proxy must publish on 127.0.0.1:8080 only (hostIP was never
checked), the console must not use hostNetwork, hostPort or a non
ClusterIP Service, and the read-only mount must be backed by the kiosk
hostPath.

kubestellar-seed's key script now runs against a stub openssl: the key
is created under umask 077, the certificate is chmod 0644, and an
existing key is never regenerated.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
configparser with strict=False kept only the last of a repeated key,
so splitting MakeDirectories= over several lines, which systemd-repart
accepts, would have dropped all but one directory from the check.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The negative check sliced the Justfile from the first "export-sysext:"
to the next comment line, so a renamed or reorganised recipe could
leave it checking an unrelated span. Parse the recipe body instead and
require it to exist and export oci/k0s-sysext.bst before asserting that
no k3s alias is emitted.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The kiosk-gate behaviour tests need node and the firstboot helper lint
needs shellcheck; both passed on main only because the runner image
happens to ship them. Install them from the runner's Ubuntu archive.

Every unit-tests run on main skips the 16 os-justfile bats tests
("unprivileged user+mount namespaces are unavailable"): Ubuntu's
AppArmor policy restricts unprivileged user namespaces. Lift the
restriction on the throwaway runner so the k8s recipe tests execute.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
node and shellcheck now come from session fixtures in conftest.py that
skip a local run without the tool but fail when CI=true, so a runner
image change can no longer turn the kiosk-gate and shellcheck tests
into silent skips.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A missing user namespace or just skipped all 16 os-justfile tests, and
on CI that happened on every run. Keep the skip for local runs but fail
when CI=true.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README promises podman and just are enough, but `just test-unit`
failed without a local bats. Fall back to bats/bats 1.14.0 pinned by
index digest, mounted read-only, and forward CI so the os-justfile
tests still refuse to skip in CI. CONTRIBUTING.md notes the pytest
prerequisites and the fallback.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
testpaths keeps a bare `pytest` on tests/unit (tests/e2e needs
selenium and a live console), and -ra lists every skip with its reason
in the summary instead of a bare "s".

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased branch onto current main to resolve the merge conflict in test_credentials_provisioning.py and test_k0s_first_boot.py created by PR #276. All 414 pytest unit tests, docs-checks, and 74 bats tests pass locally.

@jeefy
jeefy merged commit 787db84 into projectbluefin:main Sep 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants