Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 73 additions & 12 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -434,19 +434,62 @@ jobs:
# without an override ("bootupd is required for ostree-based
# installs", run 43525356388), and its lifecycle reboots stay as
# limited as before #1189.
# Two independent things drive how the installed disk boots:
# - COMPOSEFS: the image's install config (usr/lib/bootc/install/*.toml)
# sets composefs-backend, so bootc must install with --composefs-backend
# or the ostree backend aborts ("bootupd is required for ostree-based
# installs", #1215). Composefs deployments live under
# /sysroot/state/deploy/<id>, not /ostree/deploy/default/deploy.
# - FIRMWARE_BOOT: the lifecycle suite reboots into staged deployments,
# which a direct -kernel boot can't do (#1023). It boots through OVMF
# and the image's own loader. Firmware boot needs the image to install
# its own loader — bootc does that through bootupd (ostree images), but
# composefs images install systemd-boot via --bootloader systemd and
# still need firmware boot, so detect composefs too.
COMPOSEFS=false
FIRMWARE_BOOT=false
if [[ "${SUITE_DIR}" == "lifecycle" ]] \
&& sudo podman run --rm --quiet --entrypoint /bin/sh "${IMAGE}" -c 'command -v bootupctl' >/dev/null 2>&1; then
FIRMWARE_BOOT=true
elif [[ "${SUITE_DIR}" == "lifecycle" ]] \
&& sudo podman run --rm --quiet --entrypoint /bin/sh "${IMAGE}" -c \
'grep -lq composefs-backend /usr/lib/bootc/install/*.toml 2>/dev/null' >/dev/null 2>&1; then
# Composefs images (e.g. dakota) ship no bootupd, so they fall through
# the bootupctl probe above. Detect them from the install config and
# still force firmware boot (#1215).
COMPOSEFS=true
FIRMWARE_BOOT=true
elif [[ "${SUITE_DIR}" == "lifecycle" ]]; then
echo "::warning title=lifecycle without firmware boot::${IMAGE} has no bootupctl, so the lifecycle VM boots the first deployment's kernel directly and reboots cannot reach a staged deployment (#1023)"
echo "::warning title=lifecycle without firmware boot::${IMAGE} has no bootupctl and is not composefs-native, so the lifecycle VM boots the first deployment's kernel directly and reboots cannot reach a staged deployment (#1023)"
fi
if [[ "${FIRMWARE_BOOT}" != "true" ]] \
&& sudo podman run --rm --quiet "${IMAGE}" bootc install to-disk --help 2>&1 | grep -q "\-\-bootloader"; then
BOOTLOADER_FLAG="--bootloader systemd"
# Composefs images install with --composefs-backend (the ostree backend
# fails on them, #1215) plus --bootloader systemd to install systemd-boot,
# since they ship no bootupd. Probe both flags: older bootc lacks one.
INSTALL_EXTRA=""
INSTALL_HELP="$(sudo podman run --rm --quiet "${IMAGE}" bootc install to-disk --help 2>&1)"
if [[ "${COMPOSEFS}" == "true" ]] \
&& printf '%s\n' "${INSTALL_HELP}" | grep -q -- "--composefs-backend" \
&& printf '%s\n' "${INSTALL_HELP}" | grep -q "\-\-bootloader"; then
INSTALL_EXTRA="--composefs-backend --bootloader systemd"
elif [[ "${FIRMWARE_BOOT}" != "true" ]] \
&& printf '%s\n' "${INSTALL_HELP}" | grep -q "\-\-bootloader"; then
INSTALL_EXTRA="--bootloader systemd"
fi
BOOTLOADER_FLAG="${INSTALL_EXTRA}"
echo "COMPOSEFS=${COMPOSEFS}" >> "$GITHUB_ENV"
echo "FIRMWARE_BOOT=${FIRMWARE_BOOT}" >> "$GITHUB_ENV"

# Backend-aware deployment prefix. Ostree images store deployments
# under /ostree/deploy/default/deploy/<hash>.<serial>. Composefs images
# store them under /sysroot/state/deploy/<id> (#1215).
if [[ "${COMPOSEFS}" == "true" ]]; then
DEPLOY_DIR="sysroot/state/deploy"
STATE_DIR="sysroot/state"
Comment on lines +486 to +487

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness (high): /mnt/root is the physical root, i.e. bootc's /sysroot. bootc's composefs layout is state/deploy and state/os/default/var relative to /sysroot (bootc-dev/bootc crates/lib/src/composefs_consts.rs:9,19). With sysroot/ prefixed, the deploy find at :526 returns nothing and the step exits with "bootc install wrote no deployment". VAR also lands on a directory that does not exist. Note that STATEROOT/deploy must stay valid for the later loops.

Suggested change
DEPLOY_DIR="sysroot/state/deploy"
STATE_DIR="sysroot/state"
DEPLOY_DIR="state/deploy"
STATE_DIR="state"

Also set VAR per backend at :563: /mnt/root/state/os/default/var for composefs, keeping /mnt/root/${STATE_DIR}/var for ostree.

else
DEPLOY_DIR="ostree/deploy/default/deploy"
STATE_DIR="ostree/deploy/default"
fi

BOOTC_INSTALL_LOG="$(pwd)/bootc-install.log"
BOOTC_INSTALL_ATTEMPTS=3
sudo mkdir -p /mnt/root
Expand Down Expand Up @@ -480,7 +523,7 @@ jobs:
# instead of ending the step on a bare "mount: wrong fs type".
if sudo blkid -s TYPE -o value "${ROOT_PART}" >/dev/null 2>&1 \
&& sudo mount "${ROOT_PART}" /mnt/root; then
DEPLOY=$(sudo find /mnt/root/ostree/deploy/default/deploy/ -mindepth 1 -maxdepth 1 -type d -printf '%f\n' 2>/dev/null | head -1)
DEPLOY=$(sudo find "/mnt/root/${DEPLOY_DIR}" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' 2>/dev/null | head -1)
if [[ -n "${DEPLOY}" ]]; then
break
fi
Expand Down Expand Up @@ -516,8 +559,16 @@ jobs:
fi
ROOT_UUID=$(sudo blkid -s UUID -o value "${ROOT_PART}")
echo "root=${ROOT_PART} uuid=${ROOT_UUID}"
D="/mnt/root/ostree/deploy/default/deploy/${DEPLOY}"
VAR="/mnt/root/ostree/deploy/default/var"
D="/mnt/root/${DEPLOY_DIR}/${DEPLOY}"
VAR="/mnt/root/${STATE_DIR}/var"
# Direct kernel boot prep. For firmware boot (OVMF) systemd-boot reads
# the deployment's kernel + BLS entry straight from the disk, so none of
# this ostree-only setup runs. Guard it so composefs deployments are never
# given a bogus /ostree/ tree and a missing initramfs.img can't abort the
# step under set -e.
if [[ "${FIRMWARE_BOOT}" == "true" ]]; then
:
else
# el10/LTS images may have multiple entries in usr/lib/modules/ (e.g., a
# modules-only dir for the old kernel + a full dir with vmlinuz for the new
# one). Pick the version that actually contains a vmlinuz; fall back to the
Expand Down Expand Up @@ -608,6 +659,7 @@ jobs:
fi

echo "ostree boot path: ${OSTREE_PATH}"
fi
# Mask services not needed in CI to shorten first-boot time.
# NetworkManager-wait-online can block for up to 120s — most
# expensive to leave unmasked on a SLIRP QEMU network.
Expand Down Expand Up @@ -636,7 +688,16 @@ jobs:
if [[ "${FIRMWARE_BOOT}" == "true" ]]; then
# shellcheck disable=SC2086 # split KERNEL_ARGS into one argument per line
CI_KARGS=$(printf '%s\n' ${KERNEL_ARGS} | grep -vE '^(root=|rw$|ostree=)' | tr '\n' ' ')
ENTRIES=$(sudo find /mnt/root/boot/loader/entries -maxdepth 1 -name '*.conf' 2>/dev/null)
# systemd-boot reads BLS entries from the boot partition (xbootldr /
# p2). Some images also mirror them in the root's /boot (p3). Search
# both so composefs (p2) and ostree (p2 or p3) both get CI kargs.
mountpoint -q /mnt/boot || sudo mount "${LOOP}p2" /mnt/boot

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness (high): /mnt/boot is only created at :599, inside the direct-boot block that :569 now skips when FIRMWARE_BOOT=true. Because of that, this mount targets a directory that does not exist, and the step fails under bash -e for utah as well as dakota. The mount is also never undone before losetup -d at :966.

Suggested change
mountpoint -q /mnt/boot || sudo mount "${LOOP}p2" /mnt/boot
sudo mkdir -p /mnt/boot
mountpoint -q /mnt/boot || sudo mount "${LOOP}p2" /mnt/boot

Then add sudo umount /mnt/boot after the entries loop, before fi.

ENTRIES=""
for ENTRY_ROOT in /mnt/root/boot/loader/entries /mnt/boot/loader/entries; do
[[ -d "${ENTRY_ROOT}" ]] || continue
ENTRIES+="$(sudo find "${ENTRY_ROOT}" -maxdepth 1 -name '*.conf' 2>/dev/null)"
done
ENTRIES=$(printf '%s\n' "${ENTRIES}" | grep -v '^$' | sort -u)
if [[ -z "${ENTRIES}" ]]; then
echo "::error title=no loader entries::bootc wrote no /boot/loader/entries/*.conf; nothing to boot through the image's own loader"
exit 1
Expand Down Expand Up @@ -664,7 +725,7 @@ jobs:
echo "SELinux: expected=${SELINUX_EXPECTED} policy=${SELINUX_POLICY}"
echo "SELINUX_EXPECTED=${SELINUX_EXPECTED}" >> "$GITHUB_ENV"
SELINUX_DIR="$(pwd)/results/selinux"
STATEROOT="/mnt/root/ostree/deploy/default"
STATEROOT="/mnt/root/${STATE_DIR}"
mkdir -p "${SELINUX_DIR}"
if [[ "${SELINUX_EXPECTED}" == enforcing ]]; then
for DEP in $(sudo find "${STATEROOT}/deploy" -maxdepth 1 -mindepth 1 -type d 2>/dev/null); do
Expand All @@ -681,7 +742,7 @@ jobs:
# bootc may deploy with a different checksum than what `ls|head-1`
# returns, so we iterate to ensure the booted deployment gets set up.
BFT_UID=1001
for DEP in $(sudo find /mnt/root/ostree/deploy/default/deploy \
for DEP in $(sudo find "${STATEROOT}/deploy" \
-maxdepth 1 -mindepth 1 -type d 2>/dev/null); do
# CI VMs are not users: opt out of the daily countme ping
# (projectbluefin-countme.service honours this marker).
Expand Down Expand Up @@ -835,7 +896,7 @@ jobs:
sudo chmod 700 "${VAR}/home/bluefin-test/.ssh"
sudo chmod 600 "${VAR}/home/bluefin-test/.ssh/authorized_keys"
# Write key to AuthorizedKeysCommand target in every deployment
for DEP in $(sudo find /mnt/root/ostree/deploy/default/deploy \
for DEP in $(sudo find "${STATEROOT}/deploy" \
-maxdepth 1 -mindepth 1 -type d 2>/dev/null); do
sudo cp /tmp/vm_key.pub "${DEP}/etc/ssh/ci-authorized-keys"
sudo chown root:root "${DEP}/etc/ssh/ci-authorized-keys"
Expand Down Expand Up @@ -890,7 +951,7 @@ jobs:
fi
# Pre-boot diagnostics: confirm key files and sshd drop-in are correct.
echo "=== ci-authorized-keys ==="
for DEP in $(sudo find /mnt/root/ostree/deploy/default/deploy \
for DEP in $(sudo find "${STATEROOT}/deploy" \
-maxdepth 1 -mindepth 1 -type d 2>/dev/null); do
echo "--- ${DEP}/etc/ssh/ci-authorized-keys ---"
sudo cat "${DEP}/etc/ssh/ci-authorized-keys" || echo "(missing)"
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/test_e2e_lifecycle_firmware_boot.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def test_lifecycle_install_keeps_the_images_own_loader_and_needs_it():
lifecycle = install.index('if [[ "${SUITE_DIR}" == "lifecycle" ]]')
probe = install.index("command -v bootupctl", lifecycle)
firmware = install.index("FIRMWARE_BOOT=true", lifecycle)
override = install.index('BOOTLOADER_FLAG="--bootloader systemd"')
override = install.index('INSTALL_EXTRA="--bootloader systemd"')
# Firmware boot needs bootupd in the image; the systemd-boot override
# applies only when firmware boot is off (other suites, or an image
# without bootupd such as dakota, run 43525356388).
Expand Down
32 changes: 32 additions & 0 deletions tests/unit/test_e2e_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,38 @@ def test_manual_runs_checkout_the_selected_fork_ref_without_changing_callers():
assert "test_repository: ${{ github.repository }}" in manual


def test_e2e_composefs_firmware_boot_branch_for_dakota():
"""Dakota is composefs-native with no bootupd, so its lifecycle reboots
must go through the OVMF firmware path, not direct kernel boot (#1215,
#1023). The install uses --composefs-backend and a detected composefs
backend forces FIRMWARE_BOOT=true."""
workflow = _workflow("e2e.yml")

# Composefs is detected from the image's install config, and a detected
# composefs backend forces the firmware-boot path (no bootupd to use). The
# composefs branch is a fallback after the bootupctl probe, and it sets
# FIRMWARE_BOOT=true.
assert "grep -lq composefs-backend /usr/lib/bootc/install/*.toml" in workflow
composefs_probe = workflow.index("grep -lq composefs-backend /usr/lib/bootc/install/*.toml")
assert workflow.index("FIRMWARE_BOOT=true", composefs_probe) > composefs_probe

# Composefs images install with --composefs-backend (the ostree backend
# aborts without it, #1215) plus --bootloader systemd to install systemd-boot.
assert "--composefs-backend --bootloader systemd" in workflow

# Deployment discovery is backend-aware: composefs deployments live under
# /sysroot/state/deploy/<id>, not /ostree/deploy/default/deploy.
assert 'DEPLOY_DIR="sysroot/state/deploy"' in workflow
assert 'DEPLOY_DIR="ostree/deploy/default/deploy"' in workflow
assert 'D="/mnt/root/${DEPLOY_DIR}/${DEPLOY}"' in workflow

# The ostree-only direct-boot prep (kernel extraction, boot.N symlink,
# ostree= arg) must not run for firmware boot, or it would write a bogus
# /ostree/ tree onto the composefs deployment.
assert "Direct kernel boot prep" in workflow
assert 'if [[ "${FIRMWARE_BOOT}" == "true" ]]; then' in workflow


def test_e2e_ci_user_tmpfiles_has_no_recursive_home_relabel():
# A recursive `Z` on the test home walks the runner's Podman storage on
# every later boot and trips systemd-tmpfiles SELinux denials (#1205).
Expand Down
Loading