From 5d09ae7d1cc3691d538a10ae71fc15c51fb6ac18 Mon Sep 17 00:00:00 2001 From: ErenAri Date: Sat, 27 Jun 2026 15:29:48 +0300 Subject: [PATCH 1/2] =?UTF-8?q?feat(vm):=20close=20RHCOS=20coverage=20gaps?= =?UTF-8?q?=20=E2=80=94=20aarch64=20boot=20+=20broader=20artifact=20matrix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two real fixes unblock aarch64 VM validation (latent bugs for any aarch64 cloud-image profile, not just RHCOS): - aarch64 UEFI firmware: aarch64 `virt` has no built-in firmware, so the executor now supplies AAVMF as pflash (read-only CODE + a per-VM writable VARS copy), overridable via BPFCOMPAT_AARCH64_UEFI_CODE/_VARS. Without it an aarch64 guest never boots. - KVM only accelerates a same-arch guest: /dev/kvm on an x86_64 host cannot run an aarch64 guest, so qemuMachineArgs now falls back to TCG when guest arch != host arch instead of passing an invalid accel=kvm. Evidence (docs/evidence-rhcos.md) expanded to enterprise scope, all real boots: - x86_64: 6 artifacts × OpenShift 4.14/4.16/4.18. Adds perf-buffer, kprobe, and a BPF-LSM artifact (aegis). The LSM case is a real backport boundary — rejected on 4.14 (RHEL 9.2, EPERM/CAPABILITY_FAILURE) but load+attach all 4 hooks on 4.16/4.18 (RHEL 9.4). core-relocation-fail rejected everywhere (discriminator). - aarch64: real RHCOS 4.16 boot (5.14.0-427.50.1.el9_4.aarch64) under TCG, ring-buffer load+attach 1/1 — exercising the cross-compiled aarch64 validator, EDK II UEFI, Ignition, and SSH. Adds profile rhcos-4.16-arm64 and matrices/rhcos-arm64.yaml; env vars cataloged in envref + env-reference.md. RHCOS stays opt-in (BPFCOMPAT_ENABLE_RHCOS) and out of the README "Distributions covered" table. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 9 +++ README.md | 15 +++-- docs/env-reference.md | 2 + docs/evidence-rhcos.md | 105 +++++++++++++++++++++--------- internal/envref/envref.go | 10 +++ internal/vm/firmware_test.go | 57 ++++++++++++++++ internal/vm/qemu.go | 88 ++++++++++++++++++++++++- matrices/rhcos-arm64.yaml | 9 +++ vm/profiles/rhcos-4.16-arm64.yaml | 20 ++++++ 9 files changed, 278 insertions(+), 37 deletions(-) create mode 100644 internal/vm/firmware_test.go create mode 100644 matrices/rhcos-arm64.yaml create mode 100644 vm/profiles/rhcos-4.16-arm64.yaml diff --git a/CHANGELOG.md b/CHANGELOG.md index f4e8db3..32e84ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) once a ## [Unreleased] ### Added +- aarch64 VM support: the QEMU executor now supplies aarch64 UEFI firmware + (AAVMF pflash; `BPFCOMPAT_AARCH64_UEFI_CODE`/`_VARS`) and uses TCG software + emulation for a guest whose arch differs from the host (KVM only accelerates a + same-arch guest). This makes aarch64 cloud-image profiles actually boot. +- RHCOS evidence matrix expanded to 6 artifacts × 3 OpenShift releases on x86_64 + plus a real aarch64 boot (OpenShift 4.16, `5.14.0-427.50.1.el9_4.aarch64`). + New: `aegis` BPF-LSM shows a real backport boundary (rejected on RHEL 9.2, + load+attach all hooks on 9.4); profiles `rhcos-4.16-arm64`, matrices + `rhcos-arm64.yaml`; `make rhcos-image RHCOS_VERSION=` stages per-version/arch. - RHCOS evidence matrix: profiles for OpenShift 4.14 / 4.16 / 4.18 (`matrices/rhcos.yaml`) and a recorded multi-version, multi-artifact run in `docs/evidence-rhcos.md` — baseline load and ring-buffer load+attach pass on every release (RHEL 9.2 and diff --git a/README.md b/README.md index d1056b7..27fce49 100644 --- a/README.md +++ b/README.md @@ -114,11 +114,14 @@ different bootstrap. bpfcompat implements it (Ignition config over QEMU -matrix matrices/rhcos.yaml -runner vm -out report.json ``` - Recorded evidence matrix: **3 OpenShift releases (4.14 / 4.16 / 4.18)** × 3 - artifacts, real boots — baseline + ring-buffer load+attach **pass**, and a - CO-RE failure correctly **rejected** on every release — - [docs/evidence-rhcos.md](docs/evidence-rhcos.md). Without an image, the **RHEL / - AlmaLinux 9 (5.14)** profiles are the interim kernel approximation. Full guide: + Recorded evidence matrix: **3 OpenShift releases (4.14 / 4.16 / 4.18) × 6 + artifacts on x86_64, plus a real aarch64 boot** — + [docs/evidence-rhcos.md](docs/evidence-rhcos.md). Highlights: ring-buffer and + perf-buffer load+attach pass everywhere; a **BPF-LSM** program is rejected on + 4.14 (RHEL 9.2) but loads+attaches all hooks on 4.16/4.18 (RHEL 9.4) — a real + backport boundary; and a CO-RE failure is correctly rejected on every release. + Without an image, the **RHEL / AlmaLinux 9 (5.14)** profiles are the interim + kernel approximation. Full guide: [docs/rhcos-openshift.md](docs/rhcos-openshift.md). ## Try it in CI without your own KVM box @@ -583,7 +586,7 @@ Reference matrices (real, reproducible artifacts): - [`docs/case-study-falco-modern-bpf.md`](docs/case-study-falco-modern-bpf.md) — Falco `modern_bpf` across 5 kernels - [`docs/case-study-enterprise-kernels.md`](docs/case-study-enterprise-kernels.md) — RHEL/Oracle/Amazon/SUSE backported tier - [`docs/case-study-inspektor-gadget.md`](docs/case-study-inspektor-gadget.md) — published gadgets from OCI, zero config -- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.14·4.16·4.18 matrix, load + attach inside real RHCOS guests +- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.14·4.16·4.18 × 6 artifacts (x86_64) + a real aarch64 boot Internal evidence and program docs (acceptance records, runbooks, and planning notes — useful for contributors, not needed to use the tool): diff --git a/docs/env-reference.md b/docs/env-reference.md index 791ee22..b657bb2 100644 --- a/docs/env-reference.md +++ b/docs/env-reference.md @@ -123,6 +123,8 @@ Generated by `bpfcompat env --markdown`. Do not edit by hand. | Variable | Default | Description | |---|---|---| +| `BPFCOMPAT_AARCH64_UEFI_CODE` | /usr/share/AAVMF/AAVMF_CODE.fd | Path to the aarch64 UEFI firmware CODE image (pflash). aarch64 `virt` has no built-in firmware, so VM boots need it; install qemu-efi-aarch64 or point this at your distro's edk2 build. Ignored on x86_64. | +| `BPFCOMPAT_AARCH64_UEFI_VARS` | /usr/share/AAVMF/AAVMF_VARS.fd | Path to the aarch64 UEFI vars template (pflash). A per-VM writable copy is staged from it so guest NVRAM writes don't touch the shared template. Ignored on x86_64. | | `BPFCOMPAT_ENABLE_RHCOS` | false | Enable the RHEL CoreOS (rhcos) profile. RHCOS boots via the same Ignition path as Fedora CoreOS, but its image ships with an OpenShift release rather than a public URL. Stage the image with `make rhcos-image` and set this to 1/true once it is present; left off, rhcos stays unsupported so it is never claimed runnable without a real image. | ## Validator diff --git a/docs/evidence-rhcos.md b/docs/evidence-rhcos.md index 37a62ce..407726d 100644 --- a/docs/evidence-rhcos.md +++ b/docs/evidence-rhcos.md @@ -16,11 +16,12 @@ Ignition (`-fw_cfg name=opt/com.coreos/config`), SSH as `core`. The RHCOS version encodes the RHEL base (e.g. `416.94` = OpenShift 4.16 on RHEL 9.4), and the kernel column is the **in-guest `uname -r`** captured at run time. -| OpenShift | RHCOS bootimage | RHEL base | Kernel (`uname -r`) | Kernel BTF | -|---|---|---|---|---| -| 4.14 | `414.92.202407091253` | 9.2 | `5.14.0-284.73.1.el9_2.x86_64` | present | -| 4.16 | `416.94.202510081640` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present | -| 4.18 | `418.94.202510081222` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present | +| OpenShift | Arch | RHCOS bootimage | RHEL base | Kernel (`uname -r`) | Kernel BTF | +|---|---|---|---|---|---| +| 4.14 | x86_64 | `414.92.202407091253` | 9.2 | `5.14.0-284.73.1.el9_2.x86_64` | present | +| 4.16 | x86_64 | `416.94.202510081640` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present | +| 4.18 | x86_64 | `418.94.202510081222` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present | +| 4.16 | **aarch64** | `416.94.202501270445` | 9.4 | `5.14.0-427.50.1.el9_4.aarch64` | present | Note the OCP minor does **not** track the kernel linearly: 4.16 and 4.18 share the RHEL 9.4 `-427` kernel, while 4.14 is RHEL 9.2 `-284`. That is exactly the @@ -28,23 +29,47 @@ the RHEL 9.4 `-427` kernel, while 4.14 is RHEL 9.2 `-284`. That is exactly the vendor kernel. (The mirror's `4.18/latest` bootimage is RHEL-9.4-based; later 4.18 z-streams may move to 9.6 — the table records the bootimage actually run.) -## Matrix result (3 artifacts × 3 releases, real boots) +## x86_64 matrix (6 artifacts × 3 releases, real boots) -| Artifact | What it exercises | 4.14 | 4.16 | 4.18 | +| Artifact | What it exercises | 4.14 (9.2) | 4.16 (9.4) | 4.18 (9.4) | |---|---|---|---|---| -| `simple-pass` | baseline program load | ✅ load | ✅ load | ✅ load | -| `ringbuf-modern` | BPF ring buffer (upstream ≥ 5.8) + attach | ✅ load + attach 1/1 | ✅ load + attach 1/1 | ✅ load + attach 1/1 | -| `core-relocation-fail` | CO-RE relocation to a non-existent type | ❌ **rejected** | ❌ **rejected** | ❌ **rejected** | - -Two things this proves: - -1. **Backports work, tested not inferred.** The ring buffer lands upstream in - 5.8, yet `ringbuf-modern` loads *and attaches* on RHCOS's backported 5.14 - (both RHEL 9.2 and 9.4) — because the verdict comes from the real kernel. -2. **The verdict discriminates.** `core-relocation-fail` is **rejected on every - release** with `errno -22` and classification `CORE_RELOCATION_FAILURE` — so - the passes above are real acceptances, not a rubber stamp. (Its matrix targets - are non-blocking, so they record a per-target failure without failing the run.) +| `simple-pass` | baseline program load | ✅ load+attach | ✅ load+attach | ✅ load+attach | +| `ringbuf-modern` | tracepoint + ring buffer (upstream ≥ 5.8) | ✅ load+attach | ✅ load+attach | ✅ load+attach | +| `perfbuf-fallback` | tracepoint + perf-event buffer | ✅ load+attach | ✅ load+attach | ✅ load+attach | +| `attach-warn` | kprobe to a missing symbol | ✅ load / attach warn | ✅ load / attach warn | ✅ load / attach warn | +| `aegis` | **BPF-LSM** (4 hooks) + tracepoint | ❌ **rejected** (`CAPABILITY_FAILURE`, −13) | ✅ **load + attach 4/4** | ✅ **load + attach 4/4** | +| `core-relocation-fail` | CO-RE to a non-existent type (negative) | ❌ rejected (`CORE_RELOCATION_FAILURE`, −22) | ❌ rejected | ❌ rejected | + +Three things this proves: + +1. **Backports work, tested not inferred.** Ring buffer lands upstream in 5.8, + yet `ringbuf-modern` loads *and attaches* on RHCOS's backported 5.14 (RHEL 9.2 + and 9.4) — the verdict comes from the real kernel, and perf-buffer + kprobe + + tracepoint program types all behave too. +2. **A real capability boundary only a real boot finds.** The BPF-LSM artifact + `aegis` is **rejected on 4.14 (RHEL 9.2)** with `EPERM` / `CAPABILITY_FAILURE` + but **loads and attaches all 4 LSM hooks on 4.16 / 4.18 (RHEL 9.4)**. Same + nominal kernel line (5.14), different backport: BPF-LSM is active in 9.4 but + not 9.2. Version inference would miss this entirely. +3. **The verdict discriminates.** `core-relocation-fail` is rejected on every + release (`errno −22`, `CORE_RELOCATION_FAILURE`), so the passes above are real + acceptances, not a rubber stamp. (Negative-case targets are non-blocking, so + they record a per-target failure without failing the run.) + +## aarch64 matrix (cross-arch, real boot) + +OpenShift on ARM is real in enterprise, so the suite covers it too. RHCOS 4.16 +aarch64 was booted on an x86_64 host under **QEMU TCG** (software emulation — +slower, but a genuine aarch64 kernel and a real `bpf()` load): + +| Artifact | OpenShift 4.16 aarch64 (`5.14.0-427.50.1.el9_4.aarch64`) | +|---|---| +| `ringbuf-modern` | ✅ **load + attach 1/1** | + +This exercised the full aarch64 path: cross-compiled aarch64 validator, EDK II +(AAVMF) UEFI firmware via pflash, Ignition over `-fw_cfg`, SSH as `core`, and a +real ring-buffer load+attach inside the aarch64 guest. On a native ARM64 KVM +host the same run uses hardware acceleration automatically. ## In-guest validator output (representative — 4.16, ring buffer) @@ -84,20 +109,23 @@ Welcome to Red Hat Enterprise Linux CoreOS 416.94.202510081640-0 (Initramfs)! ## Provenance Images: public OpenShift mirror, -`mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos//latest/`. +`mirror.openshift.com/pub/openshift-v4//dependencies/rhcos//latest/`. The pull secret gates the container release payload, **not** these boot qcow2s. Decompressed qcow2 sha256 (as run): | OpenShift | image | sha256 (decompressed qcow2) | |---|---|---| -| 4.14 | `rhcos-4.14.34-x86_64-qemu.x86_64.qcow2` | `6d271daf23242570520891cc8013d7ab3e2fa5ab8ab9d37485b28b72ab61e99f` | -| 4.16 | `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2` | `d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399` | -| 4.18 | `rhcos-4.18.27-x86_64-qemu.x86_64.qcow2` | `a6f870c3fb8f5039962978980cf6a5a11cd2973a35fc2b2938106658983b18d6` | +| 4.14 x86_64 | `rhcos-4.14.34-x86_64-qemu.x86_64.qcow2` | `6d271daf23242570520891cc8013d7ab3e2fa5ab8ab9d37485b28b72ab61e99f` | +| 4.16 x86_64 | `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2` | `d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399` | +| 4.18 x86_64 | `rhcos-4.18.27-x86_64-qemu.x86_64.qcow2` | `a6f870c3fb8f5039962978980cf6a5a11cd2973a35fc2b2938106658983b18d6` | +| 4.16 aarch64 | `rhcos-4.16.36-aarch64-qemu.aarch64.qcow2` | `7af80164e48fee2ec60901e54494081837f896f909abdedf8a7d1bb7ce1488ac` | ## Honest limits -- **x86_64 only.** OpenShift on ARM (aarch64) is real but not covered here — it - needs an ARM64-capable KVM host and an aarch64 RHCOS bootimage. Not yet run. +- **aarch64 here ran under TCG** (software emulation) because the host is x86_64. + The kernel and `bpf()` load are genuine, just slow; a native ARM64 KVM host + runs it with hardware acceleration (the executor selects KVM automatically when + the guest arch matches the host). - **Not in public CI.** RHCOS is operator-supplied by design (no bundled image), so it does not run on every PR like the Ubuntu/FCOS lanes; this matrix is a recorded, reproducible operator run. @@ -106,6 +134,8 @@ Decompressed qcow2 sha256 (as run): ## Reproduce +x86_64 (6-artifact matrix across releases): + ```sh b=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos for v in 4.14 4.16 4.18; do @@ -113,13 +143,28 @@ for v in 4.14 4.16 4.18; do make rhcos-image RHCOS_VERSION="$v" RHCOS_IMAGE_URL="$url" # → vm/cache/rhcos-$v.qcow2 done -for art in simple-pass/simple_pass ringbuf-modern/ringbuf_modern core-relocation-fail/core_relocation_fail; do +for art in simple-pass/simple_pass ringbuf-modern/ringbuf_modern perfbuf-fallback/perfbuf_fallback \ + attach-warn/attach_warn aegis-live/aegis core-relocation-fail/core_relocation_fail; do BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \ -artifact examples/$art.bpf.o -matrix matrices/rhcos.yaml -runner vm \ -concurrency 3 -out report-$(basename $art).json done ``` -`RHCOS_VERSION` selects both the cache slot (`vm/cache/rhcos-.qcow2`) and the -matching profile in `matrices/rhcos.yaml`. `core-relocation-fail` is expected to -be rejected — that is the discriminator, not a regression. +aarch64 (needs an aarch64 validator + qemu-system-aarch64 + UEFI firmware +`qemu-efi-aarch64`; KVM on an ARM64 host, else TCG): + +```sh +b=https://mirror.openshift.com/pub/openshift-v4/aarch64/dependencies/rhcos +url=$(curl -fsSL "$b/4.16/latest/sha256sum.txt" | awk '/qemu.aarch64.qcow2.gz$/{print "'"$b"'/4.16/latest/"$2; exit}') +make rhcos-image RHCOS_VERSION=4.16-arm64 RHCOS_IMAGE_URL="$url" # → vm/cache/rhcos-4.16-arm64.qcow2 +# build the aarch64 validator (aarch64-linux-gnu-gcc + arm64 static libs) and point the run at it +BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \ + -artifact examples/ringbuf-modern/ringbuf_modern.bpf.o -matrix matrices/rhcos-arm64.yaml -runner vm +``` + +`RHCOS_VERSION` selects the cache slot (`vm/cache/rhcos-.qcow2`); the +matching profile lives in `matrices/rhcos.yaml` (x86_64) or +`matrices/rhcos-arm64.yaml` (aarch64). `aegis` rejected on 4.14 and +`core-relocation-fail` rejected everywhere are expected — they are the +discriminators, not regressions. diff --git a/internal/envref/envref.go b/internal/envref/envref.go index 5d1713a..b05f6ce 100644 --- a/internal/envref/envref.go +++ b/internal/envref/envref.go @@ -299,6 +299,16 @@ var catalog = []Var{ Category: "VM Runner", Description: "Enable the RHEL CoreOS (rhcos) profile. RHCOS boots via the same Ignition path as Fedora CoreOS, but its image ships with an OpenShift release rather than a public URL. Stage the image with `make rhcos-image` and set this to 1/true once it is present; left off, rhcos stays unsupported so it is never claimed runnable without a real image.", }, + { + Name: "BPFCOMPAT_AARCH64_UEFI_CODE", Default: "/usr/share/AAVMF/AAVMF_CODE.fd", + Category: "VM Runner", + Description: "Path to the aarch64 UEFI firmware CODE image (pflash). aarch64 `virt` has no built-in firmware, so VM boots need it; install qemu-efi-aarch64 or point this at your distro's edk2 build. Ignored on x86_64.", + }, + { + Name: "BPFCOMPAT_AARCH64_UEFI_VARS", Default: "/usr/share/AAVMF/AAVMF_VARS.fd", + Category: "VM Runner", + Description: "Path to the aarch64 UEFI vars template (pflash). A per-VM writable copy is staged from it so guest NVRAM writes don't touch the shared template. Ignored on x86_64.", + }, // ---------- HTTP server ---------- { diff --git a/internal/vm/firmware_test.go b/internal/vm/firmware_test.go new file mode 100644 index 0000000..df84adc --- /dev/null +++ b/internal/vm/firmware_test.go @@ -0,0 +1,57 @@ +package vm + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestAarch64PflashArgs(t *testing.T) { + args := aarch64PflashArgs("/fw/CODE.fd", "/run/VARS.fd") + joined := strings.Join(args, " ") + if !strings.Contains(joined, "if=pflash,format=raw,unit=0,readonly=on,file=/fw/CODE.fd") { + t.Errorf("missing read-only CODE pflash unit 0: %s", joined) + } + if !strings.Contains(joined, "if=pflash,format=raw,unit=1,file=/run/VARS.fd") { + t.Errorf("missing writable VARS pflash unit 1: %s", joined) + } +} + +func TestAarch64FirmwareArgsStagesWritableVars(t *testing.T) { + dir := t.TempDir() + code := filepath.Join(dir, "CODE.fd") + vars := filepath.Join(dir, "VARS.fd") + if err := os.WriteFile(code, []byte("code"), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(vars, []byte("vars-template"), 0o644); err != nil { + t.Fatal(err) + } + t.Setenv("BPFCOMPAT_AARCH64_UEFI_CODE", code) + t.Setenv("BPFCOMPAT_AARCH64_UEFI_VARS", vars) + + runDir := t.TempDir() + args, err := aarch64FirmwareArgs(runDir) + if err != nil { + t.Fatalf("aarch64FirmwareArgs: %v", err) + } + // A writable per-VM copy must exist under runDir (not the shared template). + copyPath := filepath.Join(runDir, "AAVMF_VARS.fd") + if _, err := os.Stat(copyPath); err != nil { + t.Fatalf("expected staged vars copy at %s: %v", copyPath, err) + } + if !strings.Contains(strings.Join(args, " "), copyPath) { + t.Errorf("pflash args should reference the staged vars copy, got %v", args) + } +} + +func TestAarch64FirmwareArgsMissing(t *testing.T) { + t.Setenv("BPFCOMPAT_AARCH64_UEFI_CODE", "/nonexistent/CODE.fd") + t.Setenv("BPFCOMPAT_AARCH64_UEFI_VARS", "/nonexistent/VARS.fd") + // With overrides pointing nowhere and (in CI) no system firmware, this errors + // clearly rather than silently producing an unbootable VM. + if _, err := aarch64FirmwareArgs(t.TempDir()); err == nil { + t.Skip("system firmware present; missing-firmware path not exercised here") + } +} diff --git a/internal/vm/qemu.go b/internal/vm/qemu.go index 71e0956..605203e 100644 --- a/internal/vm/qemu.go +++ b/internal/vm/qemu.go @@ -4,9 +4,11 @@ import ( "context" "encoding/json" "fmt" + "io" "os" "os/exec" "path/filepath" + "runtime" "strings" "syscall" "time" @@ -619,6 +621,17 @@ func startQEMU(ctx context.Context, profile Profile, overlayPath, serialLogPath, args := buildQEMUArgs(profile, overlayPath, serialLogPath, sshPort, seedMode, seedURL, seedDir, seedImagePath) + // aarch64 "virt" has no built-in firmware (unlike x86 SeaBIOS), so a UEFI + // build must be supplied as pflash or the guest never boots. Resolve and + // stage it next to the overlay (vmRunDir). + if normalizeArch(profile.Arch) == "aarch64" { + fwArgs, err := aarch64FirmwareArgs(filepath.Dir(overlayPath)) + if err != nil { + return nil, "", err + } + args = append(args, fwArgs...) + } + qemuBinary := qemuSystemBinary(profile) cmd := exec.CommandContext(ctx, qemuBinary, args...) cmd.Stdout = qemuLog @@ -633,6 +646,74 @@ func startQEMU(ctx context.Context, profile Profile, overlayPath, serialLogPath, return cmd, commandText, nil } +// aarch64FirmwareArgs resolves a UEFI firmware pair and stages a per-VM writable +// copy of the vars store under vmRunDir, returning the QEMU pflash args. Paths +// are overridable for non-Debian layouts via BPFCOMPAT_AARCH64_UEFI_CODE / +// BPFCOMPAT_AARCH64_UEFI_VARS. +func aarch64FirmwareArgs(vmRunDir string) ([]string, error) { + code := firstExistingPath( + os.Getenv("BPFCOMPAT_AARCH64_UEFI_CODE"), + "/usr/share/AAVMF/AAVMF_CODE.fd", + "/usr/share/qemu/edk2-aarch64-code.fd", + "/usr/share/edk2/aarch64/QEMU_EFI-silent-pflash.raw", + ) + if code == "" { + return nil, fmt.Errorf("aarch64 UEFI firmware not found; install qemu-efi-aarch64 or set BPFCOMPAT_AARCH64_UEFI_CODE") + } + varsTemplate := firstExistingPath( + os.Getenv("BPFCOMPAT_AARCH64_UEFI_VARS"), + "/usr/share/AAVMF/AAVMF_VARS.fd", + "/usr/share/qemu/edk2-arm-vars.fd", + ) + if varsTemplate == "" { + return nil, fmt.Errorf("aarch64 UEFI vars template not found; install qemu-efi-aarch64 or set BPFCOMPAT_AARCH64_UEFI_VARS") + } + varsCopy := filepath.Join(vmRunDir, "AAVMF_VARS.fd") + if err := copyRegularFile(varsTemplate, varsCopy); err != nil { + return nil, fmt.Errorf("stage UEFI vars: %w", err) + } + return aarch64PflashArgs(code, varsCopy), nil +} + +// aarch64PflashArgs is the pure pflash-arg shape: read-only CODE on unit 0 and a +// writable VARS store on unit 1. +func aarch64PflashArgs(codePath, varsPath string) []string { + return []string{ + "-drive", fmt.Sprintf("if=pflash,format=raw,unit=0,readonly=on,file=%s", codePath), + "-drive", fmt.Sprintf("if=pflash,format=raw,unit=1,file=%s", varsPath), + } +} + +func firstExistingPath(candidates ...string) string { + for _, c := range candidates { + c = strings.TrimSpace(c) + if c == "" { + continue + } + if _, err := os.Stat(c); err == nil { + return c + } + } + return "" +} + +func copyRegularFile(src, dst string) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.Create(dst) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() +} + func buildQEMUArgs(profile Profile, overlayPath, serialLogPath string, sshPort int, seedMode seedDeliveryMode, seedURL, seedDir, seedImagePath string) []string { memoryMB, cpus := boundedVMResources(profile.Boot) @@ -675,7 +756,12 @@ func qemuSystemBinary(profile Profile) string { } func qemuMachineArgs(profile Profile) []string { - return machineArgsFor(normalizeArch(profile.Arch), kvmAvailable()) + guest := normalizeArch(profile.Arch) + // KVM can only accelerate a guest whose arch matches the host: /dev/kvm on an + // x86_64 host cannot run an aarch64 guest. Use KVM only for a same-arch guest; + // otherwise fall back to TCG software emulation (slow but correct). + kvm := kvmAvailable() && guest == normalizeArch(runtime.GOARCH) + return machineArgsFor(guest, kvm) } // machineArgsFor is the pure acceleration decision: with KVM it pins -cpu host diff --git a/matrices/rhcos-arm64.yaml b/matrices/rhcos-arm64.yaml new file mode 100644 index 0000000..5abbcb9 --- /dev/null +++ b/matrices/rhcos-arm64.yaml @@ -0,0 +1,9 @@ +# RHEL CoreOS (OpenShift) aarch64 — opt-in, operator-supplied image. +# Separate from matrices/rhcos.yaml because an aarch64 run needs an aarch64 +# validator binary (and qemu-system-aarch64; KVM on an ARM64 host, else TCG). +# Stage: make rhcos-image RHCOS_VERSION=4.16-arm64 RHCOS_IMAGE_URL= +# Run with BPFCOMPAT_ENABLE_RHCOS=1. See docs/evidence-rhcos.md. +name: rhcos-arm64 +profiles: + - id: rhcos-4.16-arm64 + required: false diff --git a/vm/profiles/rhcos-4.16-arm64.yaml b/vm/profiles/rhcos-4.16-arm64.yaml new file mode 100644 index 0000000..1f969e7 --- /dev/null +++ b/vm/profiles/rhcos-4.16-arm64.yaml @@ -0,0 +1,20 @@ +# RHEL CoreOS (OpenShift 4.16), aarch64 — runnable with an operator-supplied image. +# +# Same Ignition boot path as x86_64 RHCOS (internal/vm/ignition.go); off by +# default, enable with BPFCOMPAT_ENABLE_RHCOS=1. Boots qemu-system-aarch64; uses +# KVM on an ARM64 host or TCG software emulation elsewhere (slow but correct). +# Stage the image: vm/cache/rhcos-4.16-arm64.qcow2 (from the aarch64 mirror). +id: rhcos-4.16-arm64 +distro: rhcos +version: "4.16" +kernel_family: "5.14" +arch: arm64 +image: + local_path: "vm/cache/rhcos-4.16-arm64.qcow2" +boot: + memory_mb: 2048 + cpus: 2 +validator: + path: "/usr/local/bin/bpfcompat-validator" +capabilities: + expected_btf: true From 8833814c759f887aa364a88179197fa0a6caceaa Mon Sep 17 00:00:00 2001 From: ErenAri Date: Sat, 27 Jun 2026 15:31:57 +0300 Subject: [PATCH 2/2] fix(vm): annotate operator-firmware file ops for gosec G703 Co-Authored-By: Claude Opus 4.8 --- internal/vm/qemu.go | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/internal/vm/qemu.go b/internal/vm/qemu.go index 605203e..0e489a2 100644 --- a/internal/vm/qemu.go +++ b/internal/vm/qemu.go @@ -690,20 +690,22 @@ func firstExistingPath(candidates ...string) string { if c == "" { continue } - if _, err := os.Stat(c); err == nil { + if _, err := os.Stat(c); err == nil { // #nosec G703 -- operator-supplied firmware path, not untrusted input return c } } return "" } +// copyRegularFile copies src to dst. src/dst are operator-controlled firmware +// paths (env-overridable defaults under /usr/share), not untrusted input. func copyRegularFile(src, dst string) error { - in, err := os.Open(src) + in, err := os.Open(src) // #nosec G703 -- operator-supplied firmware path if err != nil { return err } defer in.Close() - out, err := os.Create(dst) + out, err := os.Create(dst) // #nosec G703 -- staged under the run's private dir if err != nil { return err }