Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions Containerfile
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ COPY scripts/install-packages.py \
scripts/mirror-shim.sh \
scripts/verify-efi-chain.sh \
scripts/fix-home-labels.sh \
scripts/relocate-selinux-store.sh \
scripts/regenerate-initramfs.sh \
iso/scripts/live-kernel.py \
scripts/install-v4l2loopback.sh \
Expand Down Expand Up @@ -215,6 +216,7 @@ RUN --mount=type=bind,from=v4l2loopback,source=/out,target=/tmp/utah-v4l2loopbac
mirror-shim.sh:utah-mirror-shim \
verify-efi-chain.sh:utah-verify-efi-chain \
fix-home-labels.sh:utah-fix-home-labels \
relocate-selinux-store.sh:utah-relocate-selinux-store \
regenerate-initramfs.sh:utah-regenerate-initramfs \
live-kernel.py:utah-live-kernel \
install-v4l2loopback.sh:utah-install-v4l2loopback; do \
Expand Down Expand Up @@ -284,6 +286,11 @@ RUN --mount=type=bind,from=packages,source=/repository,target=/etc/utah-packages
/usr/share/utah/base.toml /usr/share/utah/utah.toml && \
IMAGE_FLAVOR=main /usr/local/libexec/utah-verify-rpm-contract \
/usr/share/utah/base.toml /usr/share/utah/utah.toml && \
# The SELinux module store moves from Hummingbird's read-only
# /usr/lib/selinux to /etc/selinux so setsebool -P, semanage and semodule
# work on a booted system (#551). It runs before the home-label rebuild
# below so the rebuilt policy lands in the store's final location.
/usr/local/libexec/utah-relocate-selinux-store && \
/usr/local/libexec/utah-fix-home-labels && \
DNF="$(command -v dnf5 || command -v dnf)" && \
"$DNF" clean all && rm -rf /var/cache/libdnf5 /var/cache/dnf && \
Expand Down
5 changes: 5 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,11 @@ check:
grep -q 'serial-getty@ttyS0.service' contracts/bluefin-desktop.toml
test -f scripts/configure-services.sh
test -f scripts/configure-branding.sh
test -f scripts/relocate-selinux-store.sh
test -f iso/scripts/verify-selinux-store.sh
# The SELinux module store must leave Hummingbird's read-only
# /usr/lib/selinux before the home-label policy rebuild (#551).
grep -q '/usr/local/libexec/utah-relocate-selinux-store' Containerfile
test -f scripts/verify-desktop-contract.py
test -f scripts/verify-gnome-extensions.py
test -f scripts/mirror-shim.sh
Expand Down
44 changes: 44 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,50 @@ Why this is still pre-alpha:

See the [open issues](https://github.com/projectbluefin/utah/issues) to track progress or report bugs.

## Persistent SELinux changes

The update and recovery procedure below applies to OSTree installs.
Composefs installs remain experimental (#1129); SELinux policy persistence
and update verification on that backend are not established by this change.

SELinux stays enforcing on Utah, and lasting policy changes are made with the
same commands as on Bluefin: `sudo setsebool -P` for a boolean,
`sudo semanage fcontext` for a file-labelling rule, `sudo semanage port` for a
port label, and `sudo semodule -i` for a local module built with `audit2allow`.
Each of those survives a reboot.

Hummingbird's base image keeps the SELinux policy store under `/usr`, which is
read-only on a running system, so all of those commands used to fail — the
first of them silently. Utah moves the store to `/etc/selinux`, where Fedora
atomic and Bluefin keep it (links #551).

What happens on an update: the system rebuilds the policy in the new
deployment from the new image's policy plus your local changes, so both take
effect after the reboot. Nothing you changed is lost, and new policy shipped
in the image still reaches your machine.

One case needs attention: a local module the new image's policy can no longer
satisfy (for example, one that refers to a type the new policy removed) stops
that update from being applied. The machine simply keeps running the image it
had; nothing breaks. To recognise it, `bootc status` still shows the old image
after a reboot, and the log of the step that finalises updates names the
failed policy rebuild:

```bash
journalctl -b -1 -u ostree-finalize-staged.service
```

To recover, remove the module (`sudo semodule -r <name>`) and update again.
If you would rather discard every local SELinux change, restore the image's
own copy of the store and reboot:

```bash
sudo mv /etc/selinux /etc/selinux.local && sudo cp -a /usr/etc/selinux /etc/selinux
```

Changes carried in from another image (for example, a machine switched to
Utah from Bluefin) are kept, but are not promised to take effect on Utah.

## Contributing

See [docs/building.md](docs/building.md) to build locally, [docs/skills/](docs/skills/) for architecture docs, and [AGENTS.md](AGENTS.md) for agent rules.
48 changes: 48 additions & 0 deletions docs/skills/containerfile.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,54 @@ promotes the OGC kernel for gaming flavors, rebuilds that release's initramfs,
and verifies early microcode plus the installed TPM/passkey/smart-card unlock
modules before the image can commit.

## The SELinux module store lives under `/etc/selinux`

Hummingbird's base relocates the libsemanage store to `/usr/lib/selinux` and
appends `store-root=/usr/lib/selinux` to `/etc/selinux/semanage.conf`, so
the store is image-managed and immutable. On a booted bootc system `/usr` is
read-only, so every tool that writes -- or even locks -- the store fails:
`setsebool -P` exits 255 with no message, `semanage` and `semodule -l` cannot
take the read lock. Users could not persist a boolean, add a file context or
port label, or load a local module, all of which work on Bluefin (#551).

`utah-relocate-selinux-store` (`scripts/relocate-selinux-store.sh`) moves
`targeted/active` and `final` to `/etc/selinux` and rewrites `store-root`,
the layout Fedora atomic and Bluefin use. It runs inside the package
transaction RUN, immediately before `utah-fix-home-labels`, so that script's
`semodule -n -B` rebuilds into the store's final location. The compiled
policy the kernel loads was always at `/etc/selinux/targeted/policy`; only
the module store moves. The script fails the build rather than guess when
`semanage.conf` names a different root or anything other than those two
entries remains under `/usr/lib/selinux`, and is a no-op when the store is
already under `/etc` (`tests/test_selinux_store.py`).

Why updates still deliver policy: ostree rebuilds the policy in every new
deployment after the 3-way `/etc` merge (`semodule -N --refresh` in
`ostree-finalize-staged`, ostree >= 2022.3, ostreedev/ostree#2569). The merge
carries the machine's local modules, booleans, file contexts and port labels
forward and the refresh combines them with the new image's modules. The
trade-off accepted in #551: about 18 MB and some 1,300 files join the `/etc`
merge, and a local module the new policy cannot resolve fails the refresh,
so ostree leaves the machine on its current deployment -- `bootc status`
keeps showing the old image and `journalctl -b -1 -u
ostree-finalize-staged.service` names the failure. Recovery is
`semodule -r <module>` or restoring the pristine store from
`/usr/etc/selinux`; both are documented for users in `README.md`.

These update and recovery semantics apply to the OSTree backend. Composefs
remains opt-in (#1129); neither its policy refresh nor a pristine-store path
is established here. The lifecycle verifier refuses a guest without
`/run/ostree-booted` before performing its OSTree-specific checks. Keep the
hold until booted update/rollback evidence on the new head establishes the
supported backend; a passing image build or host fixture is insufficient.

The lifecycle harness checks this on a booted OSTree guest
(`iso/scripts/verify-selinux-store.sh`, see `lifecycle-e2e.md`). A
`switch` from a pre-#551 image, whose `/etc/selinux` has no `active/`,
takes the new image's store whole; a machine switched in from Bluefin
carries whatever store files it changed there, which become live again at
its first update -- they are kept, not promised to work.

## Clean and lint share a layer

Everything above writes build-time residue that bootc lint rejects: dnf logs
Expand Down
2 changes: 1 addition & 1 deletion docs/skills/index.json
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,7 @@
],
"description": "First-boot setup hooks stamp completion through libsetup.sh. Use when editing a hook, migrating one off the legacy `version-script` gate, or debugging a hook that never runs or runs every boot.",
"version": "1.0",
"last_updated": "2026-10-12",
"last_updated": "2026-10-13",
"doc_type": "procedure"
},
{
Expand Down
12 changes: 12 additions & 0 deletions docs/skills/lifecycle-e2e.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,18 @@ This file gate complements deployment-to-entry matching (issue #361 / PR #571);
it does not itself prove which deployment an entry represents. Fixture tests
run the same guest checker via `python3 -m unittest discover -s tests -p test_boot_files.py`.

Phases 1, 3 and 4 run `iso/scripts/verify-selinux-store.sh` (#551; evidence
`selinux-<mode>-<phase>.txt`): store location, sample local changes, the running
policy against the candidate's compiled policy rule by rule, then the rollback.
Modes are in the script header; the store design is in `containerfile.md`.
The SELinux verifier requires `/run/ostree-booted` before probing, mutating
or comparing policy. Composefs runs fail explicitly at that gate; they do not
establish SELinux acceptance. The `/usr/etc` pristine policy and previous-boot
`ostree-finalize-staged` log are OSTree-specific. Capture that log before
policy comparison so a mismatch retains its finalization diagnostics.
Keep #551 held until booted update/rollback evidence exists on the new head;
host fixtures do not prove either backend's update behavior.

The baseline image (default `ghcr.io/projectbluefin/utah:testing`, the target
ref `just iso testing` builds with) is the ref a live ISO installs its offline
payload under, and phase 1 fails unless the booted deployment tracks it. The
Expand Down
9 changes: 7 additions & 2 deletions docs/skills/setup-hooks.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: setup-hooks
version: "1.0"
last_updated: "2026-10-12"
last_updated: "2026-10-13"
id: setup-hooks
one_line_purpose: Change a first-boot setup hook without breaking the once-only contract.
entry_point: docs/skills/setup-hooks.md
Expand Down Expand Up @@ -132,7 +132,12 @@ Bluefin do (`scripts/fix-home-labels.sh`). ostree rebuilds the SELinux policy
the `/etc` merge, so `20-home-labels.sh` rewrites a lingering exact
`HOME=/var/home` to `HOME=/home` and warns about any other non-`/home`
value. `utah-fix-home-labels --check` at the image's last step repeats that
rebuild and fails the build if `file_contexts.homedirs` changes. Build-time
rebuild and fails the build if `file_contexts.homedirs` changes. Since #551
the module store that rebuild reads lives under `/etc/selinux` rather than
Hummingbird's `/usr/lib/selinux` (`utah-relocate-selinux-store`, see
`containerfile.md`), so a user's own `semodule`/`semanage` changes take part
in the same rebuild; the home-rule invariant above is what keeps those
rebuilds from re-keying `/var/home`. Build-time
`useradd -m` must name `/var/home/<user>` explicitly: `/var/home` does not
exist in a container, so the `/home` symlink dangles (#281).

Expand Down
85 changes: 83 additions & 2 deletions iso/scripts/lifecycle-e2e.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@
# Validate bootc upgrade and rollback lifecycle in QEMU.
#
# Lifecycle Phases:
# 1. Boot a known Utah deployment, verify desktop session and baseline digest
# 1. Boot a known Utah deployment, verify desktop session and baseline digest;
# when its SELinux store is writable, make four persistent SELinux changes
# 2. Stage upgrade to a candidate digest using bootc / uupd policy
# 3. Reboot into the upgraded deployment, verify graphical desktop and identity
# 3. Reboot into the upgraded deployment, verify graphical desktop and identity,
# and that persistent SELinux changes work and survived the update (#551)
# 4. Roll back to the previous deployment and reboot
# 5. Verify the restored deployment boots successfully to graphical desktop
#
Expand Down Expand Up @@ -164,6 +166,27 @@ verify_boot_files() {
|| diagnose_failure "Kernel/initrd file validation failed (${phase}); see boot-files-${phase}.txt"
}

# Persistent SELinux changes (#551): the candidate must keep its policy store
# under writable /etc/selinux, and ostree's policy rebuild in the new
# deployment must carry local changes forward while the image's own policy
# still takes effect. Modes: probe, apply, image-policy, verify, reject, remove; stdout and stderr
# land in evidence/selinux-<mode>-<phase>.txt. Returns the checker's status.
selinux_check() {
local phase="$1" mode="$2" checker
checker="$(< "${ROOT}/iso/scripts/verify-selinux-store.sh")"
ssh_target_sudo bash -c "${checker}" utah-selinux "${mode}" \
> "${EVIDENCE}/selinux-${mode}-${phase}.txt" 2>&1
}

# Prints `persistent` or `immutable`. Never calls diagnose_failure itself:
# callers run it in a command substitution, where the report would be
# swallowed, so they test its status instead.
selinux_probe() {
local phase="$1"
selinux_check "${phase}" probe || return 1
tail -n 1 "${EVIDENCE}/selinux-probe-${phase}.txt"
}

lifecycle_helper() {
python3 "${ROOT}/scripts/bootc_lifecycle.py" "$@"
}
Expand Down Expand Up @@ -584,6 +607,23 @@ verify_boot_files baseline

verify_bootmgr_entries "${WORK}/baseline-status.json" baseline booted

# A baseline that already keeps its store under /etc gets the four sample
# changes now, so Phase 3 proves they survive the update. A baseline from
# before the store moved cannot persist anything; Phase 3 then proves the
# update alone makes it possible (a machine installed earlier gains it by
# updating, no reinstall or migration step).
SELINUX_LOCAL_CHANGES=0
BASELINE_SELINUX="$(selinux_probe baseline)" \
|| diagnose_failure "Could not probe the SELinux store layout on the baseline deployment; see selinux-probe-baseline.txt"
if [[ "${BASELINE_SELINUX}" == "persistent" ]]; then
selinux_check baseline apply \
|| diagnose_failure "Persistent SELinux changes failed on the baseline deployment; see selinux-apply-baseline.txt"
SELINUX_LOCAL_CHANGES=1
echo " [Phase 1] SELinux: four local policy changes applied; they must survive the upgrade"
else
echo " [Phase 1] SELinux: baseline keeps its policy store under read-only /usr; the upgrade must make persistent changes possible"
fi

python3 "${ROOT}/scripts/bootc_lifecycle.py" record-diagnostics \
--output-dir "${EVIDENCE}" \
--phase baseline \
Expand Down Expand Up @@ -695,6 +735,37 @@ verify_boot_files upgraded
# necessarily exists since the guest booted from it.
verify_bootmgr_entries "${WORK}/upgraded-status.json" upgraded "booted,rollback"

# The candidate must make persistent SELinux changes work. With changes made
# in Phase 1 they must all be in effect after the reboot, together with the
# new image's policy; otherwise the tools must work for the first time here.
# Removal exercises the fourth tool path and leaves the deployment clean.
UPGRADED_SELINUX="$(selinux_probe upgraded)" \
|| diagnose_failure "Could not probe the SELinux store layout on the upgraded deployment; see selinux-probe-upgraded.txt"
[[ "${UPGRADED_SELINUX}" == "persistent" ]] \
|| diagnose_failure "Upgraded deployment keeps its SELinux policy store under read-only /usr; setsebool -P, semanage and semodule cannot work (#551)"
# Before apply, reject or remove rebuild the policy, the policy the upgraded
# deployment booted with must be the candidate's own, rule for rule; a later
# rebuild would otherwise hide an update that never took effect.
selinux_check upgraded image-policy \
|| diagnose_failure "The upgraded deployment is not running the candidate image's SELinux policy; see selinux-image-policy-upgraded.txt"
echo " [Phase 3] SELinux: running policy matches the candidate image's policy"
if [[ "${SELINUX_LOCAL_CHANGES}" == 1 ]]; then
selinux_check upgraded verify \
|| diagnose_failure "Local SELinux changes made before the upgrade are not in effect after it, or the image's policy is not; see selinux-verify-upgraded.txt"
echo " [Phase 3] SELinux: local changes survived the upgrade alongside the image's policy"
else
selinux_check upgraded apply \
|| diagnose_failure "Persistent SELinux changes failed on the upgraded deployment; see selinux-apply-upgraded.txt"
echo " [Phase 3] SELinux: persistent changes work on the upgraded deployment"
fi
# A module the policy compiler refuses must leave the store, the running
# policy, the local changes and the labels untouched.
selinux_check upgraded reject \
|| diagnose_failure "A rejected local SELinux module changed the policy state on the upgraded deployment; see selinux-reject-upgraded.txt"
echo " [Phase 3] SELinux: invalid local module rejected without disturbing the policy"
selinux_check upgraded remove \
|| diagnose_failure "Removing the local SELinux changes failed on the upgraded deployment; see selinux-remove-upgraded.txt"

python3 "${ROOT}/scripts/bootc_lifecycle.py" record-diagnostics \
--output-dir "${EVIDENCE}" \
--phase upgraded \
Expand Down Expand Up @@ -741,6 +812,16 @@ verify_boot_files rollback
# BLS entry went missing after `bootc rollback`.
verify_bootmgr_entries "${WORK}/rollback-status.json" rollback "booted,rollback"

# The restored baseline keeps its own store layout and runs its own image's
# policy again (with its own /etc's local changes), not the candidate's.
ROLLBACK_SELINUX="$(selinux_probe rollback)" \
|| diagnose_failure "Could not probe the SELinux store layout after rollback; see selinux-probe-rollback.txt"
[[ "${ROLLBACK_SELINUX}" == "${BASELINE_SELINUX}" ]] \
|| diagnose_failure "Rollback deployment reports a ${ROLLBACK_SELINUX} SELinux store; the baseline had ${BASELINE_SELINUX}"
selinux_check rollback image-policy \
|| diagnose_failure "The restored baseline is not running its own image's SELinux policy after rollback; see selinux-image-policy-rollback.txt"
echo " [Phase 4] SELinux: restored baseline runs its own image's policy"

python3 "${ROOT}/scripts/bootc_lifecycle.py" record-diagnostics \
--output-dir "${EVIDENCE}" \
--phase rollback \
Expand Down
Loading
Loading