Repository navigation
test: assert parsed properties instead of file text; no silent skips in CI - #278
Merged
Merged
Conversation
castrojo
previously approved these changes
Sep 29, 2026
castrojo
left a comment
Contributor
There was a problem hiding this comment.
Substantive review: parsed property assertions in tests/unit/_systemd.py correctly replicate systemd file parsing, drop-in overrides, and environment expansion without relying on brittle text substring matching. Unit tests now fail closed when dependencies are missing rather than skipping silently. Full CI test suite and build verification pass.
tests/unit/_systemd.py reads unit files, drop-ins, .network files, presets and tmpfiles.d the way systemd does: comments and backslash continuations, repeated keys that accumulate, empty assignments that reset (all conditions for Condition*=, never for dependencies), first matching preset rule in filename order, and ExecStart= argv with Environment= expanded. configparser cannot do this: it keeps only the last of a repeated key and knows no continuation or reset rules. test_systemd_parser.py pins those semantics so a test built on the helper cannot pass vacuously because the parser misread a file. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…perties Replace substring and byte-for-byte file comparisons with parsed properties: unit ordering and conditions, the effective preset verdict, the wired [Match]/[Network] settings, where each import element lands in the image, and which tmpfiles.d rule wins for /etc/resolv.conf. The join-token role switch now runs the unit's own sh -c script against a stub systemctl (no token, empty token, token), so an inverted test or swapped branch fails; string greps passed either way. The preset order check covers every preset under files/, sysext presets included. The resolv.conf comment described Flatcar's image build. On FSDK, /etc starts empty and FSDK's systemd-resolve.conf links the 127.0.0.53 stub, so the test now also requires the Bluefin rule to sort ahead of it. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…arrative bluefin-firstboot-credentials.service, the network-generator drop-in, the presets and 20-wired.network are now checked as parsed settings instead of exact file text: the triggering ConditionCredential= set must equal the imported set, the stamp is touched in ExecStartPost= and gates the unit, and every shipped .network file sorts after the documented 10-static credential so it can still override the DHCP fallback. The comments claimed Flatcar's /usr has no systemd-firstboot and that /bin comes from Flatcar tmpfiles. FSDK ships systemd-firstboot; the helper stays because it also sets the live kernel hostname before networkd starts, and the image contains only /usr. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
k0scontroller.service is now checked on its effective command line, with K0S_CONTROLLER_ARGS expanded from Environment=: the old grep still passed if ExecStart= stopped referencing the variable. The worker's join-token argv and condition, the manifest tmpfiles rules, the seed unit's generators and ordering, and the Argo CD manifest's namespace are parsed rather than grepped. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The console and proxy Deployments are parsed instead of grepped: each login flag must be "true" (any 'value: "true"' used to satisfy all of them), the proxy must publish on 127.0.0.1:8080 only (hostIP was never checked), the console must not use hostNetwork, hostPort or a non ClusterIP Service, and the read-only mount must be backed by the kiosk hostPath. kubestellar-seed's key script now runs against a stub openssl: the key is created under umask 077, the certificate is chmod 0644, and an existing key is never regenerated. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
configparser with strict=False kept only the last of a repeated key, so splitting MakeDirectories= over several lines, which systemd-repart accepts, would have dropped all but one directory from the check. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The negative check sliced the Justfile from the first "export-sysext:" to the next comment line, so a renamed or reorganised recipe could leave it checking an unrelated span. Parse the recipe body instead and require it to exist and export oci/k0s-sysext.bst before asserting that no k3s alias is emitted. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The kiosk-gate behaviour tests need node and the firstboot helper lint
needs shellcheck; both passed on main only because the runner image
happens to ship them. Install them from the runner's Ubuntu archive.
Every unit-tests run on main skips the 16 os-justfile bats tests
("unprivileged user+mount namespaces are unavailable"): Ubuntu's
AppArmor policy restricts unprivileged user namespaces. Lift the
restriction on the throwaway runner so the k8s recipe tests execute.
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
node and shellcheck now come from session fixtures in conftest.py that skip a local run without the tool but fail when CI=true, so a runner image change can no longer turn the kiosk-gate and shellcheck tests into silent skips. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A missing user namespace or just skipped all 16 os-justfile tests, and on CI that happened on every run. Keep the skip for local runs but fail when CI=true. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README promises podman and just are enough, but `just test-unit` failed without a local bats. Fall back to bats/bats 1.14.0 pinned by index digest, mounted read-only, and forward CI so the os-justfile tests still refuse to skip in CI. CONTRIBUTING.md notes the pytest prerequisites and the fallback. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
testpaths keeps a bare `pytest` on tests/unit (tests/e2e needs selenium and a live console), and -ra lists every skip with its reason in the summary instead of a bare "s". Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
force-pushed
the
test/behavioral-checks
branch
from
September 29, 2026 08:03
410c36e to
bf6f353
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Most unit tests grepped config files for substrings or compared them byte for byte. This PR switches the highest-value ones to asserting what systemd, BuildStream or Kubernetes would actually do with those files. It also stops CI from silently skipping tests.
Shared parser in
tests/unit/_systemd.py(SystemdFile,preset(),tmpfiles()). It reads files the way systemd does:Condition*=clears all conditions;ExecStart=is split into argv withEnvironment=expanded;test_systemd_parser.pypins these rules so a misread file cannot make a test pass vacuously. configparser can't do this: it keeps only the last of a repeated key.Converted tests:
test_k0s_first_boot,test_credentials_provisioning,test_k0s_manifests,test_kubestellar_kioskandtest_repart_layoutnow assert parsed properties: ordering, conditions, the effective preset verdict,[Match]/[Network], the tmpfiles rule that wins, where an import element lands, and parsed Kubernetes manifests.sh -cagainst a stubsystemctl, with no token, an empty token and a real token.openssl: the key must be created underumask 077and never regenerated.Stale Flatcar comments in test files now describe FSDK:
systemd-resolve.conflinks/etc/resolv.confto the 127.0.0.53 stub. The resolv test now requires the Bluefin rule to sort ahead of it.systemd-firstboot. The Bluefin helper stays because it also sets the live kernel hostname before networkd starts./usr.Vacuous test fixed:
test_sysext_export_contractparses theexport-sysextrecipe body. It now requires the recipe to exist and to exportoci/k0s-sysext.bstbefore asserting that nok3salias is emitted.No silent skips in CI:
unit-tests.ymlinstallsnodejsandshellcheckfrom the runner's Ubuntu archive.nodeorshellcheckstill skips locally but fails whenCI=true. This goes through conftest fixtures.os-justfilebats tests were skipped ("unprivileged user+mount namespaces are unavailable"; see run 36510362559). The cause is Ubuntu's AppArmor userns restriction. The workflow now lifts it, and those tests fail instead of skipping whenCI=true.Duplicate preset coverage: the
kubeadmpreset check intest_kubeadm_sysext.py:93compares three string literals, so it is always true. It doesn't cover this generically, so I left the k0s test in place and widened it to everyfiles/**/*.preset, which now includes the kubeadm and ZFS sysext presets. See Follow-ups.just test-unitwithout bats: it falls back todocker.io/bats/bats:1.14.0pinned by index digest. The repo is mounted read-only andCIis forwarded. CONTRIBUTING.md has one line on this.pytest config:
pytest.iniaddstestpaths = tests/unitand-ra, so skips are listed with their reasons.Numbers
upstream/maintest_k0s_first_boot.pygreps / exactThe remaining greps cover:
.bstinstall-commands;Mutation proofs
Each mutation was applied to the payload. The table shows the
upstream/maintest and the rewritten test run against the mutated tree, which was then restored.upstream/maintestk0scontroller.service:ExecStart=uses$K0S_ARGS(the controller args are no longer expanded)k0s-first-boot.service: role branches swapped (a node with a token enables the controller)hostIP: 127.0.0.1removed (published on all interfaces)kubestellar-seed: key generated underumask 022(world-readable key)80-bluefin-aaa.presetwithenable k0s-first-boot.service, sorting before the opt-in fileDEV_MODEset to"false"50-zfs.preset→10-zfs.preset(sorts before20-ignition.preset)export-sysextbody replaced by a call to a script the test cannot see20-wired.network:DHCP=no20-bluefin-resolv.confrenamedzz-…(sorts after FSDK's stub rule)20-wired.network: comments added and blank lines removed, no semantic changeVerification
python3 -m pytest: 381 passed. It also passes in a fresh detached worktree withoutfiles/boot-keys, with and withoutCI=true.bats tests/unit: 74 ok, 0 skipped.just test-unitpasses with a local bats and through the container fallback. WithoutCIit reports 16 skips (the image has nojust); withCI=trueit fails, as intended.python3 .github/scripts/docs-checks.pyand the three version-check scripts behindjust validatepass. No element changes, sobst showwas not rerun.ubuntu-26.04runner label.Follow-ups
After the open PRs merge. I left their test files untouched:
test_kubeadm_sysext.py):assert "20-ignition.preset" < "80-kubeadm.preset" < "90-systemd.preset"; the preset-order test now covers that file.unit()to_systemd.SystemdFile. The10-kubeadm.confdrop-in, with itsExecStart=reset, is exactly what the helper merges.needle in dropingreps.test_sysupdate_transfers.py), feat(update): automatic updates with a Kubernetes interlock, a boot health gate and a diskless update signal #271 (test_signed_lockstep.pyplus the new update units), feat(initrd): explain diskless boot failures and ignored Ignition kargs #273 (test_initrd_diagnostics.py) and fix(security): ship root locked; provision access per node #275 (test_root_credential_contract.py): adopt the parser where they grep unit or preset text.Found while doing this, not fixed here:
03-bluefin-countme.preset(installed from projectbluefin/common byos-countme.bst) sorts before20-ignition.preset. Itsenable bluefin-countme.timertherefore wins over an Ignitionenabled: false; masking still works. It's the class of problem the preset-order test guards, but that test can't see presets from outsidefiles/.files/os/tmpfiles.d/20-bluefin-resolv.conf, theos-resolv-conf.bstdescription, the header offiles/os/libexec/bluefin-firstboot-credentials, and theos-creds-prov.bstdescription (touched by fix(security): ship root locked; provision access per node #275). I left these alone because editing them changes image content and cache keys.