Skip to content

SELinux canary records no denial on utah:testing, so the e2e denial gate fails #1141

Description

@Danathar

Summary: on ghcr.io/projectbluefin/utah:testing, the e2e SELinux denial gate fails even though every test passes. The canary that is supposed to prove denials get recorded records nothing. dakota:testing passes the same gate. v1 now includes the SELinux-enforcing default (#1119), so utah's own e2e runs through e2e.yml@v1 will fail at this gate once utah's v1 pin is bumped.

What happens

Run 42923513204 (manual, utah:testing, suites=common, branch of #1138, now merged), jobs common-a and common-b:

  • The guest is enforcing before and after the tests (after-boot=Enforcing, after-run=Enforcing).
  • All scenarios that ran passed (common-a 52, common-b 36).
  • Step "SELinux denial gate" fails: SELinux canary: a deliberately denied httpd_t operation did not reach the AVC gate. Denial collection is broken, so a clean gate proves nothing.
  • In the run's e2e-results-…-common-a/selinux/ artifact, avc-scenario.log and avc-after-canary.log are both 0 bytes, and canary.json is {"canary_denials": [], "detected": false}.

The canary is at .github/workflows/e2e.yml:1674-1680 @ ef2024c9. It runs systemd-run … -p SELinuxContext=system_u:system_r:httpd_t:s0 /usr/bin/cat /etc/shadow, then collects with scripts/selinux_collect_avc.sh, which uses ausearch when /var/log/audit/audit.log is non-empty, otherwise journalctl -o cat | grep 'avc:'.

For comparison: run 42923526890 (dakota:testing, same branch) passes the gate.

Likely causes (unverified, check first)

  1. The canary never runs in httpd_t on utah, for example because the policy lacks httpd_t or systemd can't transition to it. systemd-run then fails without producing an AVC. Its output is discarded (StandardOutput/StandardError=null, || true), so nothing shows it.
  2. Denials don't reach the journal. The serial log shows audit: initializing netlink subsys (disabled) and audit_enabled=0 early in boot, then journald "Collecting audit messages is enabled" later.
  3. A dontaudit rule in utah's policy hides this particular access.

Ask

  • Make the canary say why it found nothing. Capture and log:

    • systemd-run's exit code and stderr;
    • cat /proc/cmdline;
    • seinfo -t httpd_t, if seinfo exists;
    • whether /var/log/audit/audit.log exists;
    • journalctl -k | grep -c avc.

    Add them to the step summary and the artifact.

  • Then fix the root cause in testsuite: a canary that produces a real denial on both utah and dakota (for example, a domain and access present in both policies), or collection that reads where utah's denials actually go. If utah truly can't produce a denial, the gate must fail with that specific reason, not "collection is broken".

Acceptance criteria

  • The SELinux denial gate passes on utah:testing and dakota:testing when no unexpected denial occurs, and still fails when one does.
  • If the canary can't run or can't be collected, the error names which of those it was.

Test plan

  • e2e.yml only runs when called, so a maintainer must start the test runs from the PR branch:
    • gh workflow run manual.yml --ref <branch> -f image=ghcr.io/projectbluefin/utah:testing -f suites=common
    • the same with image=ghcr.io/projectbluefin/dakota:testing.
  • Say in the PR which runs are needed.
  • Unit tests for any selinux_qualify.py change.

Out of scope

  • Turning the gate or the enforcing default off.
  • Changing the utah image (report it on the PR if the image itself is at fault).

e2e.yml is a human-merge release workflow here, so a maintainer merges the fix.

Activity

  1. added
    kind/bugSomething is broken.
    triage/acceptedScope accepted; ready for work. Use `/triage accepted`.
    on Nov 11, 2026
  2. hivecommons-hive commented on Nov 11, 2026

    @hivecommons-hive
    Contributor

    🛠️ Hive agent scanner (hive hosted-projectbluefin-knuckle-gjvq) started work on this issue at 2026-11-11T02:51:31Z. A pull request referencing this issue supersedes this mark; it lapses on its own at 2026-11-11T04:51:31Z if none appears.

  3. Danathar commented on Nov 11, 2026

    @Danathar
    ContributorAuthor

    Cause found: the diagnostics from #1148 show why the canary never records a denial on utah. The policy blocks the probe, but it's a dontaudit rule, so SELinux denies it silently.

    From results/selinux/canary-diagnostics.txt (manual run of #1148 on utah:testing, https://github.com/projectbluefin/testsuite/actions/runs/43010296886, both common-a and common-b):

    getenforce=Enforcing
    attempt_httpd_t=rc=203 status=203
    sesearch_dontaudit=dontaudit httpd_t exec_type:file { execute execute_no_trans }; dontaudit httpd_t file_type:file getattr; ...
    auditd=active
    audit_log=present (6 avc lines)
    journal_avc=6
    
    • The probe runs in httpd_t and fails at execve (203/EXEC).
    • The policy has dontaudit httpd_t exec_type:file { execute execute_no_trans }, so that denial is never logged.
    • Audit itself works: auditd is active, and other AVCs reach audit.log and the journal.
    • dakota:testing passed the same run (https://github.com/projectbluefin/testsuite/actions/runs/43010300745).

    Next step: pick a probe whose denial is audited.

    • Have the canary make an access that httpd_t (or another always-present domain) is denied without a matching dontaudit rule. Check with sesearch --dontaudit -s <domain> in the guest, as scripts/selinux_canary.sh already does.
    • Don't disable dontaudit rules (semodule -DB). That changes the policy under test and floods the AVC gate with unrelated denials.

    Acceptance as before: manual.yml with suites=common passes on both utah:testing and dakota:testing, with the canary's denial collected.

  4. 5 remaining items

  5. kylerankin commented on Nov 12, 2026

    @kylerankin
    Contributor

    Summary: The fix that makes the security-label check work on the utah test image was stuck, not because the fix was wrong, but because a test file added with it imported two things it never used, and a code-quality check refused to let the fix merge until that was clean. I removed those two unused imports and pushed the change to the fix's pull request, so the quality check should pass and the fix can move on.

    What I need from you: Let the pull request's checks finish, then run the manual test with suites=common on both utah:testing and dakota:testing and merge if both record the block.


    What was blocking the fix

    The only red check on the pull request was "Lint & syntax" (the ruff check). The new unit test added with the fix, tests/unit/test_selinux_canary.py, imported sys and pytest without using them, which ruff rejects, so the check failed and the hive would not merge.

    What I changed

    Removed the two unused imports from that test file. Verified locally: ruff check tests/ scripts/ --select E,F,W --ignore E501 reports no problems, the file compiles, and all four unit tests in it pass. Pushed the change as a new head on the pull request, so CI re-runs automatically.

    Where this leaves the issue

    The pull request (#1156, "fix(e2e): collect an audited SELinux denial in the canary") now carries the whole fix: the canary skips any label domain whose block the image hides behind a "don't-log" rule and stops at the first domain whose block is actually recorded, without disabling any rules. The first half of this issue — making the canary explain why it found nothing — is already merged (#1148).

    Your acceptance gate still stands: a manual run with suites=common that passes on both utah:testing and dakota:testing, with the block collected in each. That needs the test VMs, which a fork cannot use, so it has to run here. Until both pass, the pull request stays "Refs #1141", as you asked.

    — hive: backend=pi model=lemonade/ornith-1.5-35b-a3b-gguf-q6_k


    🐝 Hive Agent: contributor | SHA: 6a9240e5

  6. Danathar commented on Nov 12, 2026

    @Danathar
    ContributorAuthor

    Reopening: #1156's "Closes #1141" closed this when it merged (2026-11-12 8:27 AM EST), but Utah still fails the canary. Run of #1156's head on Utah: https://github.com/projectbluefin/testsuite/actions/runs/43220933663. All 88 common scenarios pass, and both jobs still fail the canary, now in sshd_t. #1156 is still useful: httpd_t's denial really is dontaudit on Utah, and it now skips it.

    The remaining cause is the collector, not the canary domain. On Utah, results/selinux/avc-scenario.log and avc-after-canary.log are both 0 bytes, while the canary's diagnostics report audit_log=present (7 avc lines) and journal_avc=7.

    scripts/selinux_collect_avc.sh:13 runs ausearch -m AVC,USER_AVC --raw first, whenever /var/log/audit/audit.log exists, which it does on Utah. The script reaches the guest as $SSH 'sudo bash -s' < scripts/selinux_collect_avc.sh (.github/workflows/e2e.yml:1682 and :1695, step "SELinux denial gate"), so its stdin is a pipe. When stdin is not a terminal, ausearch reads its input from stdin: man ausearch says it "can also take input from stdin", and that --input-logs "is needed if you are using ausearch from a cron job". So it swallows the rest of the script as audit data, finds nothing, and the journalctl and dmesg lines never run. Reproduced:

    $ printf 'echo FIRST\nausearch -m AVC,USER_AVC --raw 2>/dev/null || true\necho SECOND-LINE-RAN\n' | bash -s
    FIRST
    $ printf 'echo FIRST\nausearch --input-logs -m AVC,USER_AVC --raw 2>/dev/null || true\necho SECOND-LINE-RAN\n' | bash -s
    FIRST
    SECOND-LINE-RAN
    

    dakota's GNOME OS image has no ausearch, so it takes the grep branch, which is why it passes. On Utah the AVC gate collects nothing at all, so it would also miss real denials.

    Ask:

    1. scripts/selinux_collect_avc.sh:13: ausearch --input-logs -m AVC,USER_AVC --raw 2>/dev/null || true.
    2. A unit test that runs the script through a pipe (… | bash -s) with a stub ausearch that drains stdin, and asserts that the lines after it still run.
    3. In the PR, write Refs SELinux canary records no denial on utah:testing, so the e2e denial gate fails #1141, not Fixes or Closes. A maintainer closes this after manual.yml suites=common passes on utah:testing from the PR's head. The PR VM check boots only dakota:testing, so a green PR doesn't show Utah passing.

    Acceptance: common-a and common-b pass on utah:testing, including the SELinux gate and the canary, in a maintainer run of the PR's head.

  7. removed
    has-open-prAn open PR already links this issue with Fixes/Closes/Resolves; the hive will not dispatch it again
    on Nov 12, 2026
  8. added
    hive: churn-triagedMaintainer reviewed PR churn; remaining work is real, offer to contributors again
    on Nov 12, 2026
  9. Danathar commented on Nov 12, 2026

    @Danathar
    ContributorAuthor

    Closing: fixed by #1171, which is now merged.

    Evidence: maintainer run of #1171's head (21871f3f) on utah:testing with manual.yml and suites=common: https://github.com/projectbluefin/testsuite/actions/runs/43271798804

    • The collector now reads Utah's audit log, and the journal and kernel lines after it run. avc-scenario.log is no longer empty.
    • The canary's denial is collected in both httpd_t and sshd_t (canary denial detected: { entrypoint } comm=(cat) … -> bin_t:file), so the canary passes on Utah.
    • All 88 common scenarios pass.

    What's left is a different, real problem, now tracked in Utah: both jobs still fail the AVC gate on two { mac_admin } comm=chcon unconfined_service_t denials per boot. They come from bootc's capability probe when ublue-booted-image.service runs bootc status from a script. Before this fix the gate could not see them. They are tracked in projectbluefin/utah#1198. Utah's post-testing e2e picks up this fix only after its testsuite pin moves past this merge.

  10. added a commit that references this issue on Nov 12, 2026
    fca0aa4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    approved-directionhive: churn-triagedMaintainer reviewed PR churn; remaining work is real, offer to contributors againkind/bugSomething is broken.triage/acceptedScope accepted; ready for work. Use `/triage accepted`.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions