Repository navigation
Description
Summary: on ghcr.io/projectbluefin/utah:testing, the e2e SELinux denial gate fails even though every test passes. The canary that is supposed to prove denials get recorded records nothing. dakota:testing passes the same gate. v1 now includes the SELinux-enforcing default (#1119), so utah's own e2e runs through e2e.yml@v1 will fail at this gate once utah's v1 pin is bumped.
What happens
Run 42923513204 (manual, utah:testing, suites=common, branch of #1138, now merged), jobs common-a and common-b:
- The guest is enforcing before and after the tests (
after-boot=Enforcing,after-run=Enforcing). - All scenarios that ran passed (common-a 52, common-b 36).
- Step "SELinux denial gate" fails:
SELinux canary: a deliberately denied httpd_t operation did not reach the AVC gate. Denial collection is broken, so a clean gate proves nothing. - In the run's
e2e-results-…-common-a/selinux/artifact,avc-scenario.logandavc-after-canary.logare both 0 bytes, andcanary.jsonis{"canary_denials": [], "detected": false}.
The canary is at .github/workflows/e2e.yml:1674-1680 @ ef2024c9. It runs systemd-run … -p SELinuxContext=system_u:system_r:httpd_t:s0 /usr/bin/cat /etc/shadow, then collects with scripts/selinux_collect_avc.sh, which uses ausearch when /var/log/audit/audit.log is non-empty, otherwise journalctl -o cat | grep 'avc:'.
For comparison: run 42923526890 (dakota:testing, same branch) passes the gate.
Likely causes (unverified, check first)
- The canary never runs in
httpd_ton utah, for example because the policy lackshttpd_torsystemdcan't transition to it.systemd-runthen fails without producing an AVC. Its output is discarded (StandardOutput/StandardError=null,|| true), so nothing shows it. - Denials don't reach the journal. The serial log shows
audit: initializing netlink subsys (disabled)andaudit_enabled=0early in boot, then journald "Collecting audit messages is enabled" later. - A
dontauditrule in utah's policy hides this particular access.
Ask
-
Make the canary say why it found nothing. Capture and log:
systemd-run's exit code and stderr;cat /proc/cmdline;seinfo -t httpd_t, ifseinfoexists;- whether
/var/log/audit/audit.logexists; journalctl -k | grep -c avc.
Add them to the step summary and the artifact.
-
Then fix the root cause in testsuite: a canary that produces a real denial on both utah and dakota (for example, a domain and access present in both policies), or collection that reads where utah's denials actually go. If utah truly can't produce a denial, the gate must fail with that specific reason, not "collection is broken".
Acceptance criteria
- The SELinux denial gate passes on
utah:testinganddakota:testingwhen no unexpected denial occurs, and still fails when one does. - If the canary can't run or can't be collected, the error names which of those it was.
Test plan
e2e.ymlonly runs when called, so a maintainer must start the test runs from the PR branch:gh workflow run manual.yml --ref <branch> -f image=ghcr.io/projectbluefin/utah:testing -f suites=common- the same with
image=ghcr.io/projectbluefin/dakota:testing.
- Say in the PR which runs are needed.
- Unit tests for any
selinux_qualify.pychange.
Out of scope
- Turning the gate or the enforcing default off.
- Changing the utah image (report it on the PR if the image itself is at fault).
e2e.yml is a human-merge release workflow here, so a maintainer merges the fix.
Activity
hivecommons-hive commentedon Nov 11, 2026
🛠️ Hive agent
scanner(hive hosted-projectbluefin-knuckle-gjvq) started work on this issue at 2026-11-11T02:51:31Z. A pull request referencing this issue supersedes this mark; it lapses on its own at 2026-11-11T04:51:31Z if none appears.Danathar commentedon Nov 11, 2026
Cause found: the diagnostics from #1148 show why the canary never records a denial on utah. The policy blocks the probe, but it's a
dontauditrule, so SELinux denies it silently.From
results/selinux/canary-diagnostics.txt(manual run of #1148 onutah:testing, https://github.com/projectbluefin/testsuite/actions/runs/43010296886, bothcommon-aandcommon-b):httpd_tand fails at execve (203/EXEC).dontaudit httpd_t exec_type:file { execute execute_no_trans }, so that denial is never logged.audit.logand the journal.dakota:testingpassed the same run (https://github.com/projectbluefin/testsuite/actions/runs/43010300745).Next step: pick a probe whose denial is audited.
httpd_t(or another always-present domain) is denied without a matchingdontauditrule. Check withsesearch --dontaudit -s <domain>in the guest, asscripts/selinux_canary.shalready does.semodule -DB). That changes the policy under test and floods the AVC gate with unrelated denials.Acceptance as before:
manual.ymlwithsuites=commonpasses on bothutah:testinganddakota:testing, with the canary's denial collected.🐛 fix: make the SELinux canary say why it was not seen (#1148)
5 remaining items
lint(e2e): drop unused imports in the selinux canary unit test (proje…
kylerankin commentedon Nov 12, 2026
Summary: The fix that makes the security-label check work on the utah test image was stuck, not because the fix was wrong, but because a test file added with it imported two things it never used, and a code-quality check refused to let the fix merge until that was clean. I removed those two unused imports and pushed the change to the fix's pull request, so the quality check should pass and the fix can move on.
What I need from you: Let the pull request's checks finish, then run the manual test with
suites=commonon bothutah:testinganddakota:testingand merge if both record the block.What was blocking the fix
The only red check on the pull request was "Lint & syntax" (the ruff check). The new unit test added with the fix,
tests/unit/test_selinux_canary.py, importedsysandpytestwithout using them, which ruff rejects, so the check failed and the hive would not merge.What I changed
Removed the two unused imports from that test file. Verified locally:
ruff check tests/ scripts/ --select E,F,W --ignore E501reports no problems, the file compiles, and all four unit tests in it pass. Pushed the change as a new head on the pull request, so CI re-runs automatically.Where this leaves the issue
The pull request (#1156, "fix(e2e): collect an audited SELinux denial in the canary") now carries the whole fix: the canary skips any label domain whose block the image hides behind a "don't-log" rule and stops at the first domain whose block is actually recorded, without disabling any rules. The first half of this issue — making the canary explain why it found nothing — is already merged (#1148).
Your acceptance gate still stands: a manual run with
suites=commonthat passes on bothutah:testinganddakota:testing, with the block collected in each. That needs the test VMs, which a fork cannot use, so it has to run here. Until both pass, the pull request stays "Refs #1141", as you asked.— hive: backend=pi model=lemonade/ornith-1.5-35b-a3b-gguf-q6_k
🐝 Hive Agent:
contributor| SHA:6a9240e5fix(e2e): collect an audited SELinux denial in the canary (#1141) (#1156
Danathar commentedon Nov 12, 2026
Reopening: #1156's "Closes #1141" closed this when it merged (2026-11-12 8:27 AM EST), but Utah still fails the canary. Run of #1156's head on Utah: https://github.com/projectbluefin/testsuite/actions/runs/43220933663. All 88
commonscenarios pass, and both jobs still fail the canary, now insshd_t. #1156 is still useful:httpd_t's denial really isdontauditon Utah, and it now skips it.The remaining cause is the collector, not the canary domain. On Utah,
results/selinux/avc-scenario.logandavc-after-canary.logare both 0 bytes, while the canary's diagnostics reportaudit_log=present (7 avc lines)andjournal_avc=7.scripts/selinux_collect_avc.sh:13runsausearch -m AVC,USER_AVC --rawfirst, whenever/var/log/audit/audit.logexists, which it does on Utah. The script reaches the guest as$SSH 'sudo bash -s' < scripts/selinux_collect_avc.sh(.github/workflows/e2e.yml:1682and:1695, step "SELinux denial gate"), so its stdin is a pipe. When stdin is not a terminal,ausearchreads its input from stdin:man ausearchsays it "can also take input from stdin", and that--input-logs"is needed if you are using ausearch from a cron job". So it swallows the rest of the script as audit data, finds nothing, and thejournalctlanddmesglines never run. Reproduced:dakota's GNOME OS image has no
ausearch, so it takes thegrepbranch, which is why it passes. On Utah the AVC gate collects nothing at all, so it would also miss real denials.Ask:
scripts/selinux_collect_avc.sh:13:ausearch --input-logs -m AVC,USER_AVC --raw 2>/dev/null || true.… | bash -s) with a stubausearchthat drains stdin, and asserts that the lines after it still run.manual.ymlsuites=commonpasses onutah:testingfrom the PR's head. The PR VM check boots onlydakota:testing, so a green PR doesn't show Utah passing.Acceptance:
common-aandcommon-bpass onutah:testing, including the SELinux gate and the canary, in a maintainer run of the PR's head.fix: collect security denials from audit logs (#1171)
Danathar commentedon Nov 12, 2026
Closing: fixed by #1171, which is now merged.
Evidence: maintainer run of #1171's head (
21871f3f) onutah:testingwithmanual.ymlandsuites=common: https://github.com/projectbluefin/testsuite/actions/runs/43271798804avc-scenario.logis no longer empty.httpd_tandsshd_t(canary denial detected: { entrypoint } comm=(cat) … -> bin_t:file), so the canary passes on Utah.commonscenarios pass.What's left is a different, real problem, now tracked in Utah: both jobs still fail the AVC gate on two
{ mac_admin } comm=chcon unconfined_service_tdenials per boot. They come from bootc's capability probe whenublue-booted-image.servicerunsbootc statusfrom a script. Before this fix the gate could not see them. They are tracked in projectbluefin/utah#1198. Utah's post-testing e2e picks up this fix only after its testsuite pin moves past this merge.docs(agents): use Refs for fixes the PR's checks can't prove; parked …