Skip to content

[quality] test-infrastructure: tests/test_image_repo.bats runs in no runner, and nothing detects the next unregistered suite #1165

Description

@hivecommons-hive

Finding

The test recipe in Justfile is the repository's developer-facing test runner, and it is maintained by hand. Nothing checks it against the contents of tests/, so a suite can be added and then silently never run.

That has already happened:

  • tests/test_image_repo.bats — 93 lines, 12 assertions over system_files/shared/usr/libexec/ublue-image-repo, the single source of truth for image-name/tag → upstream GitHub repository routing consumed by bonedigger-report. It is named in neither the Justfile test recipe nor .github/workflows/unit-tests.yml. No runner executes it.
  • tests/test_shared_just.bats — also absent from the test recipe.

tests/test_libvirt_helper.bats is the only suite whose omission is deliberate, and the Justfile says so in a comment — but that comment is prose, not something a runner can check, and a comment naming a deleted file would never be noticed either.

Evidence (executed at 44b4b9f, bats 1.14.0):

$ bats tests/test_image_repo.bats
1..12
ok 1 image-repo: bluefin latest routes to projectbluefin/bluefin
...
ok 12 image-repo: consumers do not hardcode upstream repo routing

The suite is green. It simply never runs.

$ bats tests/test_shared_just.bats
not ok 2 powerwash: delegates to bctl when bctl is available
not ok 3 powerwash: bctl delegation skips gum prompts and sudo entirely

That one is red, which is a separate repair (already open as #1106) — this issue does not claim it.

Recommendation

Make the recipe's coverage of tests/ a checked contract rather than a convention:

  • Register tests/test_image_repo.bats in the Justfile test recipe.
  • Add tests/test_suite_registration.bats, a drift gate asserting that every tests/test_*.bats and tests/test_*.py is either named in the test recipe or declared excluded in the comment block above it as # <file> is excluded — <reason>, that every registered path exists on disk, that every exclusion names a file that still exists, and that every exclusion states a reason.
  • Declare the two intentional exclusions (test_libvirt_helper.bats, test_shared_just.bats) in that machine-checkable form, each with its reason.
  • Register the gate itself in the recipe.

Scope note: this issue is about the Justfile recipe only. CI registration — .github/workflows/unit-tests.yml executes 27 of the 40 suites in tests/, and never invokes just test — is a separate, larger gap tracked on its own issue.

Priority

  • Impact: medium — one suite's worth of executed assertions is currently dead, and the mechanism that let it die is still in place
  • Effort: low

Filed by quality agent (hold-gated mode)

🐝 Hive Agent: quality | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: 44b4b9f

— hive: agent=quality backend=copilot model=claude-opus-5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/qualityFiled or owned by the quality agent.hive/hosted-projectbluefin-knuckle-gjvqRouted by the hosted Project Bluefin Hive deployment.qualityCode quality or test-coverage work.testingTest authoring or test infrastructure.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions