feat(vm): close RHCOS coverage gaps — aarch64 boot + broader artifact matrix

CHANGELOG.md

@@ -8,6 +8,15 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) once a
 ## [Unreleased]
 
 ### Added
+- aarch64 VM support: the QEMU executor now supplies aarch64 UEFI firmware
+  (AAVMF pflash; `BPFCOMPAT_AARCH64_UEFI_CODE`/`_VARS`) and uses TCG software
+  emulation for a guest whose arch differs from the host (KVM only accelerates a
+  same-arch guest). This makes aarch64 cloud-image profiles actually boot.
+- RHCOS evidence matrix expanded to 6 artifacts × 3 OpenShift releases on x86_64
+  plus a real aarch64 boot (OpenShift 4.16, `5.14.0-427.50.1.el9_4.aarch64`).
+  New: `aegis` BPF-LSM shows a real backport boundary (rejected on RHEL 9.2,
+  load+attach all hooks on 9.4); profiles `rhcos-4.16-arm64`, matrices
+  `rhcos-arm64.yaml`; `make rhcos-image RHCOS_VERSION=` stages per-version/arch.
 - RHCOS evidence matrix: profiles for OpenShift 4.14 / 4.16 / 4.18 (`matrices/rhcos.yaml`)
   and a recorded multi-version, multi-artifact run in `docs/evidence-rhcos.md` —
   baseline load and ring-buffer load+attach pass on every release (RHEL 9.2 and

README.md

@@ -114,11 +114,14 @@ different bootstrap. bpfcompat implements it (Ignition config over QEMU
     -matrix matrices/rhcos.yaml -runner vm -out report.json
   ```
 
-  Recorded evidence matrix: **3 OpenShift releases (4.14 / 4.16 / 4.18)** × 3
-  artifacts, real boots — baseline + ring-buffer load+attach **pass**, and a
-  CO-RE failure correctly **rejected** on every release —
-  [docs/evidence-rhcos.md](docs/evidence-rhcos.md). Without an image, the **RHEL /
-  AlmaLinux 9 (5.14)** profiles are the interim kernel approximation. Full guide:
+  Recorded evidence matrix: **3 OpenShift releases (4.14 / 4.16 / 4.18) × 6
+  artifacts on x86_64, plus a real aarch64 boot** —
+  [docs/evidence-rhcos.md](docs/evidence-rhcos.md). Highlights: ring-buffer and
+  perf-buffer load+attach pass everywhere; a **BPF-LSM** program is rejected on
+  4.14 (RHEL 9.2) but loads+attaches all hooks on 4.16/4.18 (RHEL 9.4) — a real
+  backport boundary; and a CO-RE failure is correctly rejected on every release.
+  Without an image, the **RHEL / AlmaLinux 9 (5.14)** profiles are the interim
+  kernel approximation. Full guide:
   [docs/rhcos-openshift.md](docs/rhcos-openshift.md).
 
 ## Try it in CI without your own KVM box
@@ -583,7 +586,7 @@ Reference matrices (real, reproducible artifacts):
 - [`docs/case-study-falco-modern-bpf.md`](docs/case-study-falco-modern-bpf.md) — Falco `modern_bpf` across 5 kernels
 - [`docs/case-study-enterprise-kernels.md`](docs/case-study-enterprise-kernels.md) — RHEL/Oracle/Amazon/SUSE backported tier
 - [`docs/case-study-inspektor-gadget.md`](docs/case-study-inspektor-gadget.md) — published gadgets from OCI, zero config
-- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.14·4.16·4.18 matrix, load + attach inside real RHCOS guests
+- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.14·4.16·4.18 × 6 artifacts (x86_64) + a real aarch64 boot
 
 Internal evidence and program docs (acceptance records, runbooks, and
 planning notes — useful for contributors, not needed to use the tool):

docs/env-reference.md

@@ -123,6 +123,8 @@ Generated by `bpfcompat env --markdown`. Do not edit by hand.
 
 | Variable | Default | Description |
 |---|---|---|
+| `BPFCOMPAT_AARCH64_UEFI_CODE` | /usr/share/AAVMF/AAVMF_CODE.fd | Path to the aarch64 UEFI firmware CODE image (pflash). aarch64 `virt` has no built-in firmware, so VM boots need it; install qemu-efi-aarch64 or point this at your distro's edk2 build. Ignored on x86_64. |
+| `BPFCOMPAT_AARCH64_UEFI_VARS` | /usr/share/AAVMF/AAVMF_VARS.fd | Path to the aarch64 UEFI vars template (pflash). A per-VM writable copy is staged from it so guest NVRAM writes don't touch the shared template. Ignored on x86_64. |
 | `BPFCOMPAT_ENABLE_RHCOS` | false | Enable the RHEL CoreOS (rhcos) profile. RHCOS boots via the same Ignition path as Fedora CoreOS, but its image ships with an OpenShift release rather than a public URL. Stage the image with `make rhcos-image` and set this to 1/true once it is present; left off, rhcos stays unsupported so it is never claimed runnable without a real image. |
 
 ## Validator

docs/evidence-rhcos.md

@@ -16,35 +16,60 @@ Ignition (`-fw_cfg name=opt/com.coreos/config`), SSH as `core`. The RHCOS
 version encodes the RHEL base (e.g. `416.94` = OpenShift 4.16 on RHEL 9.4), and
 the kernel column is the **in-guest `uname -r`** captured at run time.
 
-| OpenShift | RHCOS bootimage | RHEL base | Kernel (`uname -r`) | Kernel BTF |
-|---|---|---|---|---|
-| 4.14 | `414.92.202407091253` | 9.2 | `5.14.0-284.73.1.el9_2.x86_64` | present |
-| 4.16 | `416.94.202510081640` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present |
-| 4.18 | `418.94.202510081222` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present |
+| OpenShift | Arch | RHCOS bootimage | RHEL base | Kernel (`uname -r`) | Kernel BTF |
+|---|---|---|---|---|---|
+| 4.14 | x86_64 | `414.92.202407091253` | 9.2 | `5.14.0-284.73.1.el9_2.x86_64` | present |
+| 4.16 | x86_64 | `416.94.202510081640` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present |
+| 4.18 | x86_64 | `418.94.202510081222` | 9.4 | `5.14.0-427.93.1.el9_4.x86_64` | present |
+| 4.16 | **aarch64** | `416.94.202501270445` | 9.4 | `5.14.0-427.50.1.el9_4.aarch64` | present |
 
 Note the OCP minor does **not** track the kernel linearly: 4.16 and 4.18 share
 the RHEL 9.4 `-427` kernel, while 4.14 is RHEL 9.2 `-284`. That is exactly the
 "version number predicts nothing" property bpfcompat tests by booting the real
 vendor kernel. (The mirror's `4.18/latest` bootimage is RHEL-9.4-based; later
 4.18 z-streams may move to 9.6 — the table records the bootimage actually run.)
 
-## Matrix result (3 artifacts × 3 releases, real boots)
+## x86_64 matrix (6 artifacts × 3 releases, real boots)
 
-| Artifact | What it exercises | 4.14 | 4.16 | 4.18 |
+| Artifact | What it exercises | 4.14 (9.2) | 4.16 (9.4) | 4.18 (9.4) |
 |---|---|---|---|---|
-| `simple-pass` | baseline program load | ✅ load | ✅ load | ✅ load |
-| `ringbuf-modern` | BPF ring buffer (upstream ≥ 5.8) + attach | ✅ load + attach 1/1 | ✅ load + attach 1/1 | ✅ load + attach 1/1 |
-| `core-relocation-fail` | CO-RE relocation to a non-existent type | ❌ **rejected** | ❌ **rejected** | ❌ **rejected** |
-
-Two things this proves:
-
-1. **Backports work, tested not inferred.** The ring buffer lands upstream in
-   5.8, yet `ringbuf-modern` loads *and attaches* on RHCOS's backported 5.14
-   (both RHEL 9.2 and 9.4) — because the verdict comes from the real kernel.
-2. **The verdict discriminates.** `core-relocation-fail` is **rejected on every
-   release** with `errno -22` and classification `CORE_RELOCATION_FAILURE` — so
-   the passes above are real acceptances, not a rubber stamp. (Its matrix targets
-   are non-blocking, so they record a per-target failure without failing the run.)
+| `simple-pass` | baseline program load | ✅ load+attach | ✅ load+attach | ✅ load+attach |
+| `ringbuf-modern` | tracepoint + ring buffer (upstream ≥ 5.8) | ✅ load+attach | ✅ load+attach | ✅ load+attach |
+| `perfbuf-fallback` | tracepoint + perf-event buffer | ✅ load+attach | ✅ load+attach | ✅ load+attach |
+| `attach-warn` | kprobe to a missing symbol | ✅ load / attach warn | ✅ load / attach warn | ✅ load / attach warn |
+| `aegis` | **BPF-LSM** (4 hooks) + tracepoint | ❌ **rejected** (`CAPABILITY_FAILURE`, −13) | ✅ **load + attach 4/4** | ✅ **load + attach 4/4** |
+| `core-relocation-fail` | CO-RE to a non-existent type (negative) | ❌ rejected (`CORE_RELOCATION_FAILURE`, −22) | ❌ rejected | ❌ rejected |
+
+Three things this proves:
+
+1. **Backports work, tested not inferred.** Ring buffer lands upstream in 5.8,
+   yet `ringbuf-modern` loads *and attaches* on RHCOS's backported 5.14 (RHEL 9.2
+   and 9.4) — the verdict comes from the real kernel, and perf-buffer + kprobe +
+   tracepoint program types all behave too.
+2. **A real capability boundary only a real boot finds.** The BPF-LSM artifact
+   `aegis` is **rejected on 4.14 (RHEL 9.2)** with `EPERM` / `CAPABILITY_FAILURE`
+   but **loads and attaches all 4 LSM hooks on 4.16 / 4.18 (RHEL 9.4)**. Same
+   nominal kernel line (5.14), different backport: BPF-LSM is active in 9.4 but
+   not 9.2. Version inference would miss this entirely.
+3. **The verdict discriminates.** `core-relocation-fail` is rejected on every
+   release (`errno −22`, `CORE_RELOCATION_FAILURE`), so the passes above are real
+   acceptances, not a rubber stamp. (Negative-case targets are non-blocking, so
+   they record a per-target failure without failing the run.)
+
+## aarch64 matrix (cross-arch, real boot)
+
+OpenShift on ARM is real in enterprise, so the suite covers it too. RHCOS 4.16
+aarch64 was booted on an x86_64 host under **QEMU TCG** (software emulation —
+slower, but a genuine aarch64 kernel and a real `bpf()` load):
+
+| Artifact | OpenShift 4.16 aarch64 (`5.14.0-427.50.1.el9_4.aarch64`) |
+|---|---|
+| `ringbuf-modern` | ✅ **load + attach 1/1** |
+
+This exercised the full aarch64 path: cross-compiled aarch64 validator, EDK II
+(AAVMF) UEFI firmware via pflash, Ignition over `-fw_cfg`, SSH as `core`, and a
+real ring-buffer load+attach inside the aarch64 guest. On a native ARM64 KVM
+host the same run uses hardware acceleration automatically.
 
 ## In-guest validator output (representative — 4.16, ring buffer)
 
@@ -84,20 +109,23 @@ Welcome to Red Hat Enterprise Linux CoreOS 416.94.202510081640-0 (Initramfs)!
 ## Provenance
 
 Images: public OpenShift mirror,
-`mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/<ver>/latest/`.
+`mirror.openshift.com/pub/openshift-v4/<arch>/dependencies/rhcos/<ver>/latest/`.
 The pull secret gates the container release payload, **not** these boot qcow2s.
 Decompressed qcow2 sha256 (as run):
 
 | OpenShift | image | sha256 (decompressed qcow2) |
 |---|---|---|
-| 4.14 | `rhcos-4.14.34-x86_64-qemu.x86_64.qcow2` | `6d271daf23242570520891cc8013d7ab3e2fa5ab8ab9d37485b28b72ab61e99f` |
-| 4.16 | `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2` | `d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399` |
-| 4.18 | `rhcos-4.18.27-x86_64-qemu.x86_64.qcow2` | `a6f870c3fb8f5039962978980cf6a5a11cd2973a35fc2b2938106658983b18d6` |
+| 4.14 x86_64 | `rhcos-4.14.34-x86_64-qemu.x86_64.qcow2` | `6d271daf23242570520891cc8013d7ab3e2fa5ab8ab9d37485b28b72ab61e99f` |
+| 4.16 x86_64 | `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2` | `d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399` |
+| 4.18 x86_64 | `rhcos-4.18.27-x86_64-qemu.x86_64.qcow2` | `a6f870c3fb8f5039962978980cf6a5a11cd2973a35fc2b2938106658983b18d6` |
+| 4.16 aarch64 | `rhcos-4.16.36-aarch64-qemu.aarch64.qcow2` | `7af80164e48fee2ec60901e54494081837f896f909abdedf8a7d1bb7ce1488ac` |
 
 ## Honest limits
 
-- **x86_64 only.** OpenShift on ARM (aarch64) is real but not covered here — it
-  needs an ARM64-capable KVM host and an aarch64 RHCOS bootimage. Not yet run.
+- **aarch64 here ran under TCG** (software emulation) because the host is x86_64.
+  The kernel and `bpf()` load are genuine, just slow; a native ARM64 KVM host
+  runs it with hardware acceleration (the executor selects KVM automatically when
+  the guest arch matches the host).
 - **Not in public CI.** RHCOS is operator-supplied by design (no bundled image),
   so it does not run on every PR like the Ubuntu/FCOS lanes; this matrix is a
   recorded, reproducible operator run.
@@ -106,20 +134,37 @@ Decompressed qcow2 sha256 (as run):
 
 ## Reproduce
 
+x86_64 (6-artifact matrix across releases):
+
 ```sh
 b=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos
 for v in 4.14 4.16 4.18; do
   url=$(curl -fsSL "$b/$v/latest/sha256sum.txt" | awk '/qemu.x86_64.qcow2.gz$/{print "'"$b"'/'"$v"'/latest/"$2; exit}')
   make rhcos-image RHCOS_VERSION="$v" RHCOS_IMAGE_URL="$url"   # → vm/cache/rhcos-$v.qcow2
 done
 
-for art in simple-pass/simple_pass ringbuf-modern/ringbuf_modern core-relocation-fail/core_relocation_fail; do
+for art in simple-pass/simple_pass ringbuf-modern/ringbuf_modern perfbuf-fallback/perfbuf_fallback \
+           attach-warn/attach_warn aegis-live/aegis core-relocation-fail/core_relocation_fail; do
   BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \
     -artifact examples/$art.bpf.o -matrix matrices/rhcos.yaml -runner vm \
     -concurrency 3 -out report-$(basename $art).json
 done
 ```
 
-`RHCOS_VERSION` selects both the cache slot (`vm/cache/rhcos-<ver>.qcow2`) and the
-matching profile in `matrices/rhcos.yaml`. `core-relocation-fail` is expected to
-be rejected — that is the discriminator, not a regression.
+aarch64 (needs an aarch64 validator + qemu-system-aarch64 + UEFI firmware
+`qemu-efi-aarch64`; KVM on an ARM64 host, else TCG):
+
+```sh
+b=https://mirror.openshift.com/pub/openshift-v4/aarch64/dependencies/rhcos
+url=$(curl -fsSL "$b/4.16/latest/sha256sum.txt" | awk '/qemu.aarch64.qcow2.gz$/{print "'"$b"'/4.16/latest/"$2; exit}')
+make rhcos-image RHCOS_VERSION=4.16-arm64 RHCOS_IMAGE_URL="$url"   # → vm/cache/rhcos-4.16-arm64.qcow2
+# build the aarch64 validator (aarch64-linux-gnu-gcc + arm64 static libs) and point the run at it
+BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \
+  -artifact examples/ringbuf-modern/ringbuf_modern.bpf.o -matrix matrices/rhcos-arm64.yaml -runner vm
+```
+
+`RHCOS_VERSION` selects the cache slot (`vm/cache/rhcos-<ver>.qcow2`); the
+matching profile lives in `matrices/rhcos.yaml` (x86_64) or
+`matrices/rhcos-arm64.yaml` (aarch64). `aegis` rejected on 4.14 and
+`core-relocation-fail` rejected everywhere are expected — they are the
+discriminators, not regressions.

internal/envref/envref.go

@@ -299,6 +299,16 @@ var catalog = []Var{
 		Category:    "VM Runner",
 		Description: "Enable the RHEL CoreOS (rhcos) profile. RHCOS boots via the same Ignition path as Fedora CoreOS, but its image ships with an OpenShift release rather than a public URL. Stage the image with `make rhcos-image` and set this to 1/true once it is present; left off, rhcos stays unsupported so it is never claimed runnable without a real image.",
 	},
+	{
+		Name: "BPFCOMPAT_AARCH64_UEFI_CODE", Default: "/usr/share/AAVMF/AAVMF_CODE.fd",
+		Category:    "VM Runner",
+		Description: "Path to the aarch64 UEFI firmware CODE image (pflash). aarch64 `virt` has no built-in firmware, so VM boots need it; install qemu-efi-aarch64 or point this at your distro's edk2 build. Ignored on x86_64.",
+	},
+	{
+		Name: "BPFCOMPAT_AARCH64_UEFI_VARS", Default: "/usr/share/AAVMF/AAVMF_VARS.fd",
+		Category:    "VM Runner",
+		Description: "Path to the aarch64 UEFI vars template (pflash). A per-VM writable copy is staged from it so guest NVRAM writes don't touch the shared template. Ignored on x86_64.",
+	},
 
 	// ---------- HTTP server ----------
 	{

internal/vm/firmware_test.go

@@ -0,0 +1,57 @@
+package vm
+
+import (
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+)
+
+func TestAarch64PflashArgs(t *testing.T) {
+	args := aarch64PflashArgs("/fw/CODE.fd", "/run/VARS.fd")
+	joined := strings.Join(args, " ")
+	if !strings.Contains(joined, "if=pflash,format=raw,unit=0,readonly=on,file=/fw/CODE.fd") {
+		t.Errorf("missing read-only CODE pflash unit 0: %s", joined)
+	}
+	if !strings.Contains(joined, "if=pflash,format=raw,unit=1,file=/run/VARS.fd") {
+		t.Errorf("missing writable VARS pflash unit 1: %s", joined)
+	}
+}
+
+func TestAarch64FirmwareArgsStagesWritableVars(t *testing.T) {
+	dir := t.TempDir()
+	code := filepath.Join(dir, "CODE.fd")
+	vars := filepath.Join(dir, "VARS.fd")
+	if err := os.WriteFile(code, []byte("code"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(vars, []byte("vars-template"), 0o644); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("BPFCOMPAT_AARCH64_UEFI_CODE", code)
+	t.Setenv("BPFCOMPAT_AARCH64_UEFI_VARS", vars)
+
+	runDir := t.TempDir()
+	args, err := aarch64FirmwareArgs(runDir)
+	if err != nil {
+		t.Fatalf("aarch64FirmwareArgs: %v", err)
+	}
+	// A writable per-VM copy must exist under runDir (not the shared template).
+	copyPath := filepath.Join(runDir, "AAVMF_VARS.fd")
+	if _, err := os.Stat(copyPath); err != nil {
+		t.Fatalf("expected staged vars copy at %s: %v", copyPath, err)
+	}
+	if !strings.Contains(strings.Join(args, " "), copyPath) {
+		t.Errorf("pflash args should reference the staged vars copy, got %v", args)
+	}
+}
+
+func TestAarch64FirmwareArgsMissing(t *testing.T) {
+	t.Setenv("BPFCOMPAT_AARCH64_UEFI_CODE", "/nonexistent/CODE.fd")
+	t.Setenv("BPFCOMPAT_AARCH64_UEFI_VARS", "/nonexistent/VARS.fd")
+	// With overrides pointing nowhere and (in CI) no system firmware, this errors
+	// clearly rather than silently producing an unbootable VM.
+	if _, err := aarch64FirmwareArgs(t.TempDir()); err == nil {
+		t.Skip("system firmware present; missing-firmware path not exercised here")
+	}
+}