Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ jobs:
with:
submodules: recursive
- name: Install just
uses: extractions/setup-just@f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b # v3
uses: extractions/setup-just@53165ef7e734c5c07cb06b3c8e7b647c5aa16db3 # v4
- name: Install zstandard
run: pip install --quiet zstandard
- name: Validate manifests
Expand Down
144 changes: 135 additions & 9 deletions .github/workflows/post-testing-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,31 +9,157 @@ on:
permissions: read-all

jobs:
promote-to-testing:
# Which images the run produced, and their digests, resolved once so the boot
# matrix and the promotion agree on exactly the same set.
resolve:
if: >-
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch == 'testing'
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
permissions:
packages: write
actions: read
contents: read
outputs:
images: ${{ steps.verify.outputs.images }}
steps:
- name: Checkout
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
ref: ${{ github.event.workflow_run.head_sha }}
- name: Download all flavor digests
env:
GH_TOKEN: ${{ github.token }}
run: gh run download "${{ github.event.workflow_run.id }}" --repo "${{ github.repository }}" --pattern 'image-digest-testing-*' --dir /tmp/digests
- name: Promote verified images to testing
# The whole set is checked before any of it is promoted. This used to
# copy each digest as it read it and assert the total afterwards, which
# is the wrong order twice over: a short or unexpected set had already
# moved :testing for the flavors it did reach by the time the assertion
# failed, and the total was a literal four, while config/flavors.json is
# supposed to be the only source for which images exist. `just check`
# forbids that literal everywhere else in this workflow directory.
- name: Verify the run produced exactly the promised images
id: verify
run: |
set -euo pipefail
find /tmp/digests -name '*.txt' -exec cat {} + > /tmp/digests.txt
python3 - <<'PY'
import json, os, pathlib, subprocess, sys

expected = {
item["image"]
for item in json.loads(
subprocess.run(
[sys.executable, "scripts/flavors.py", "images"],
capture_output=True, text=True, check=True,
).stdout
)
}
found = {}
for line in pathlib.Path("/tmp/digests.txt").read_text().splitlines():
name, _, digest = line.partition("=")
name, digest = name.strip(), digest.strip()
if not name or not digest:
continue
if not digest.startswith("sha256:"):
sys.exit(f"{name}: {digest!r} is not a digest")
if name in found and found[name] != digest:
sys.exit(f"{name}: the run reported two different digests")
found[name] = digest

missing = sorted(expected - set(found))
unexpected = sorted(set(found) - expected)
if missing or unexpected:
if missing:
print("The build did not produce these images:", ", ".join(missing))
if unexpected:
print("The build produced images no flavor asks for:", ", ".join(unexpected))
sys.exit(
"Refusing to advance :testing. config/flavors.json is the only "
"source for which images exist, and this run does not match it."
)
images = [{"image": name, "digest": found[name]} for name in sorted(expected)]
with open(os.environ["GITHUB_OUTPUT"], "a") as handle:
handle.write("images=" + json.dumps(images) + "\n")
print(f"verified all {len(expected)} promised images:", ", ".join(sorted(expected)))
PY

# Every promised image is booted before any of them is promoted. The build
# already asserts what an image contains -- the package contract, the desktop
# contract, the extensions, bootc lint -- and none of that can tell you it
# comes up, that the gaming flavors boot the OGC kernel rather than the stock
# one, or that the NVIDIA module matches the kernel that actually booted.
# Those are the claims the README calls unproven, so they are checked here
# rather than asserted in prose.
boot:
needs: resolve
runs-on: ubuntu-24.04
permissions:
contents: read
packages: read
strategy:
fail-fast: false
matrix:
image: ${{ fromJSON(needs.resolve.outputs.images) }}
# TCG emulation without KVM, which hosted runners do not offer. A desktop
# image takes minutes to reach gdm this way.
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
ref: ${{ github.event.workflow_run.head_sha }}
- name: Install just
uses: extractions/setup-just@f8a3cce218d9f83db3a2ecd90e41ac3de6cdfd9b # v3
- name: Free disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/share/boost "$AGENT_TOOLSDIRECTORY"
df -h /
- name: Install QEMU and UEFI firmware
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends qemu-system-x86 ovmf
- name: Log in to GHCR
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: echo "$GITHUB_TOKEN" | podman login ghcr.io -u "${{ github.actor }}" --password-stdin
- name: Boot ${{ matrix.image.image }} and verify it
env:
REGISTRY: ghcr.io/${{ github.repository_owner }}
run: |
set -euo pipefail
ref="${REGISTRY}/${{ matrix.image.image }}@${{ matrix.image.digest }}"
# By digest, so the image booted is the exact one that will be
# promoted rather than whatever the tag points at by then.
podman pull "$ref"
just e2e-boot "$ref"

promote-to-testing:
needs: [resolve, boot]
runs-on: ubuntu-24.04
permissions:
packages: write
actions: read
steps:
# No re-derivation here. resolve already checked the run against
# config/flavors.json and boot proved every one of those images starts,
# so this copies exactly that list and nothing else.
- name: Promote the booted images to testing
env:
GH_TOKEN: ${{ github.token }}
REGISTRY: ghcr.io/${{ github.repository_owner }}
IMAGES: ${{ needs.resolve.outputs.images }}
run: |
set -euo pipefail
echo "$GH_TOKEN" | skopeo login ghcr.io --username "${{ github.actor }}" --password-stdin
count=0
python3 - <<'PY' > /tmp/promote.txt
import json, os
for item in json.loads(os.environ["IMAGES"]):
print(item["image"], item["digest"], sep="=")
PY
test -s /tmp/promote.txt
while IFS='=' read -r name digest; do
[[ -n "$name" && "$digest" == sha256:* ]] || continue
[ -n "$name" ] || continue
echo "promoting ${name}@${digest}"
skopeo copy --all "docker://${REGISTRY}/${name}@${digest}" "docker://${REGISTRY}/${name}:testing"
count=$((count + 1))
done < <(find /tmp/digests -name '*.txt' -exec cat {} +)
test "$count" -eq 4
done < /tmp/promote.txt
41 changes: 41 additions & 0 deletions Containerfile.e2e
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# The boot test layer. Never published.
#
# A published image must contain what it promises and nothing else, so the unit
# that drives the boot check does not live in it. This derives from a finished
# flavor image, adds the verifier and a first-boot unit, and is installed to a
# throwaway disk by `just e2e-disk`. The image under test is unmodified in
# every other respect: same kernel, same packages, same configuration, so what
# boots here is what would boot on hardware.
ARG IMAGE
FROM ${IMAGE}

COPY scripts/verify-boot.sh /usr/local/libexec/utah-verify-boot

# Ordered after gdm rather than after graphical.target: reaching the target
# does not mean the display manager came up, and gdm is what a user meets
# first. The verifier polls for it anyway, so a slow start is not a failure.
#
# Output goes to the console because that is the only channel the harness has:
# it reads the serial port of a VM it cannot log in to.
RUN chmod 0755 /usr/local/libexec/utah-verify-boot && \
printf '%s\n' \
'[Unit]' \
'Description=Utah boot verification' \
'After=gdm.service systemd-user-sessions.service' \
'Wants=gdm.service' \
'' \
'[Service]' \
'Type=oneshot' \
'RemainAfterExit=yes' \
'ExecStart=/usr/local/libexec/utah-verify-boot' \
'StandardOutput=journal+console' \
'StandardError=journal+console' \
'# Whatever the verdict, the VM must not sit there until the job times' \
'# out. The harness decides pass or fail from the console text.' \
'ExecStopPost=/usr/bin/systemctl --no-block poweroff' \
'' \
'[Install]' \
'WantedBy=multi-user.target' \
> /usr/lib/systemd/system/utah-verify-boot.service && \
systemctl enable utah-verify-boot.service && \
test -x /usr/local/libexec/utah-verify-boot
82 changes: 82 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ check:
python3 -m py_compile scripts/verify-rpm-contract.py
python3 -m py_compile scripts/check-repo-availability.py
python3 scripts/install-packages.py --check packages/bluefin.toml
# Both scripts compose their package list from sections named one by
# one, so a new section in utah.toml can be live in one and invisible
# to the other. [services] was unverified that way, and [firmware]
# briefly was too.
python3 scripts/check-overlay-sections.py
python3 scripts/verify-rpm-contract.py --check packages/bluefin.toml
grep -qE 'reusable-build\.yml@(v1|[0-9a-f]{40} # v1)$' .github/workflows/build.yml
test -f Containerfile.kernel
Expand All @@ -75,6 +80,23 @@ check:
# config/flavors.json exists to stop: narrowing the build matrix while
# promote and release still name images nothing produces fails late.
! grep -rn 'utah-nvidia\|utah-gaming' .github/workflows/
# Nor its own count of them. Promotion asserted `count -eq 4` after it had
# already copied whatever it found, so a short set advanced :testing for
# the flavors it did reach and failed afterwards.
! grep -nE '\-eq [0-9]+' .github/workflows/post-testing-e2e.yml
grep -q 'scripts/flavors.py' .github/workflows/post-testing-e2e.yml
# Promotion must be gated on a real boot, not only on the build's own
# in-image contract checks. Those cannot see whether the thing starts.
test -f scripts/verify-boot.sh
test -f Containerfile.e2e
bash -n scripts/verify-boot.sh
grep -q 'e2e-boot' .github/workflows/post-testing-e2e.yml
grep -qE '^\s*needs: \[resolve, boot\]' .github/workflows/post-testing-e2e.yml
grep -q 'UTAH_BOOT_OK' scripts/verify-boot.sh
# The verification layer is a test artifact. If it ever appears in the
# image's own Containerfile it would ship a unit that powers the machine
# off shortly after gdm starts.
! grep -q 'verify-boot' Containerfile

# Verify branding, desktop defaults, first-boot Flatpak policy, and service
# enablement in an already-composed image. The same verifier runs in the
Expand Down Expand Up @@ -301,6 +323,66 @@ generate-bootable-image stream="testing":
sync
echo "Bootable disk ready: $disk"

# The runner has no KVM, so this is TCG emulation and slow -- minutes, not
# seconds. That is still the cheapest honest answer to "does it boot", and it
# is the one gate the build-time contract checks cannot supply.
#
# Boot an image headless and assert what only a boot can show.
e2e-boot image_ref timeout="1800":
#!/usr/bin/env bash
set -euo pipefail
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
# The verification layer is built here and never pushed; see
# Containerfile.e2e for why it is not part of the image under test.
test_ref="localhost/utah-e2e:$(date +%s)"
podman build --build-arg IMAGE="{{ image_ref }}" \
--tag "$test_ref" --file Containerfile.e2e .
disk="$work/boot.raw"
truncate -s 20G "$disk"
# console=ttyS0 is what makes the verifier readable: the harness has no way
# into this VM other than the serial port.
sudo podman run --rm --privileged --pid=host \
--security-opt label=type:unconfined_t \
-v "$work:/data:Z" \
"$test_ref" bootc install to-disk \
--via-loopback /data/boot.raw --filesystem btrfs --wipe \
--generic-image --karg console=ttyS0,115200 --karg utah.local
sudo chown "$(id -u):$(id -g)" "$disk"
# A fresh VM has no NVRAM entry, so install the removable-media fallback
# the same way generate-bootable-image does.
loop="$(sudo losetup --find --show --partscan "$disk")"
esp="$work/esp"; mkdir -p "$esp"
sudo mount "${loop}p2" "$esp"
sudo install -d "$esp/EFI/BOOT"
sudo install -m 0644 "$esp/EFI/fedora/shimx64.efi" "$esp/EFI/BOOT/BOOTX64.EFI"
sudo install -m 0644 "$esp/EFI/fedora/grubx64.efi" "$esp/EFI/BOOT/grubx64.efi"
sudo umount "$esp"
sudo losetup -d "$loop"
cp /usr/share/OVMF/OVMF_VARS_4M.fd "$work/vars.fd" 2>/dev/null \
|| cp /usr/share/OVMF/OVMF_VARS.fd "$work/vars.fd"
code=/usr/share/OVMF/OVMF_CODE_4M.fd
test -f "$code" || code=/usr/share/OVMF/OVMF_CODE.fd
log="$work/console.log"
set +e
timeout "{{ timeout }}" qemu-system-x86_64 \
-machine q35 -m 4096 -smp "$(nproc)" -no-reboot -nographic \
-drive "if=pflash,format=raw,unit=0,readonly=on,file=$code" \
-drive "if=pflash,format=raw,unit=1,file=$work/vars.fd" \
-drive "file=$disk,format=raw,if=virtio" \
-serial "file:$log" -display none
set -e
echo "--- serial console"
cat "$log" || true
echo "--- end serial console"
if grep -q '^UTAH_BOOT_OK ' "$log"; then
grep '^UTAH_BOOT_OK ' "$log"
exit 0
fi
echo "The VM never reported a successful boot." >&2
grep -E 'utah-boot:|Kernel panic|emergency|Failed to' "$log" >&2 || true
exit 1

# Build a single-architecture UEFI live ISO. This first slice proves the
# Utah live boot path; installer payload integration is intentionally the next
# ISO milestone.
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,10 @@ This is the honest list, and it is why the label above says pre-alpha.

- **Nothing is published.** No image, no ISO, no installer. An ISO is planned as
a fork of [dakota-iso](https://github.com/projectbluefin/dakota-iso).
- **The NVIDIA and gaming flavors are unproven.** The OGC kernel compiles with
- **The NVIDIA and gaming flavors are unproven.** Nothing asserted the claims
below across a boot until `post-testing-e2e.yml` gained a boot gate; it now
checks them on every flavor before `:testing` moves, so this entry should be
settled by the first green run rather than by argument. The OGC kernel compiles with
`sched_ext` and `binderfs` genuinely enabled, and the NVIDIA open module
compiles for the base kernel. The module against the OGC kernel, the driver
installer flags, and the flavored builds pulling the kernel cache image have
Expand Down
26 changes: 26 additions & 0 deletions docs/building.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,32 @@ The result is `output/utah-live.iso`, assembled with systemd-boot, a
intended to prove live desktop boot first; bootc-installer/offline payload
integration is the next ISO milestone.

### Proving an image boots

The Containerfile asserts what an image *contains* -- the package contract, the
desktop contract, the GNOME extensions, `bootc container lint`. None of that can
say whether it starts, and three of Utah's claims live entirely on the other
side of a boot: that a desktop appears at all, that the gaming flavors run the
OGC kernel rather than the stock one, and that the NVIDIA module matches the
kernel that actually booted rather than the one it was compiled against.

`just e2e-boot <image-ref>` answers those. It derives a throwaway layer from the
image (`Containerfile.e2e`), which adds a first-boot unit running
`scripts/verify-boot.sh`, installs that to a disk with `bootc install to-disk`,
and boots it under QEMU with the console on a serial port. The verifier prints
its findings and one marker line, `UTAH_BOOT_OK`, and powers the machine off;
the caller passes only if that marker appears. A VM that panics or hangs cannot
look like a pass, because a hang produces no marker either.

That layer is never published. A published image must contain what it promises
and nothing more, and it certainly must not carry a unit that powers the machine
off once gdm is up.

`post-testing-e2e.yml` runs it for every flavor between verifying the image set
and advancing `:testing`, so an image that does not boot cannot be promoted.
Hosted runners offer no KVM, so this is TCG emulation and takes minutes per
flavor; that is the cost of the only check that can answer the question.

### Image flavors

`config/flavors.json` decides which images exist. Everything derives from it --
Expand Down
Loading