From 602b550421bfa15ffde80451f8bae1edf66715f1 Mon Sep 17 00:00:00 2001 From: ErenAri Date: Sat, 27 Jun 2026 13:57:09 +0300 Subject: [PATCH] docs(rhcos): real RHCOS validation evidence + operator guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records a real validation run inside a booted RHEL CoreOS guest and documents the operator path end to end: - docs/evidence-rhcos.md: committed evidence — RHCOS 416.94 (OpenShift 4.16), kernel 5.14.0-427.93.1.el9_4, ringbuf-modern load=pass attach=1/1, with the in-guest validator output, serial excerpt (ignition.platform.id=qemu), and image provenance (public OpenShift mirror, sha256s). Reproduce steps included. - docs/rhcos-openshift.md: operator guide — why it's opt-in, getting the image (public mirror / openshift-install), the RHEL/Alma 9 interim kernel approximation, and limits. - matrices/rhcos.yaml: ready-to-run single-profile matrix (required: false). - README: a careful CoreOS/OpenShift note under the backports section — FCOS Ignition boot implemented + public; RHCOS opt-in, operator-supplied image; example commands; links to the evidence and guide. RHCOS stays OUT of the "Distributions covered" table (no public evergreen image). Raw run artifacts live under evidence/rhcos/ locally (git-ignored as high-churn output); the decisive fields are inlined in docs/evidence-rhcos.md. Co-Authored-By: Claude Opus 4.8 --- README.md | 29 +++++++++++++ docs/evidence-rhcos.md | 94 +++++++++++++++++++++++++++++++++++++++++ docs/rhcos-openshift.md | 92 ++++++++++++++++++++++++++++++++++++++++ matrices/rhcos.yaml | 7 +++ 4 files changed, 222 insertions(+) create mode 100644 docs/evidence-rhcos.md create mode 100644 docs/rhcos-openshift.md create mode 100644 matrices/rhcos.yaml diff --git a/README.md b/README.md index 77fdde8..bbbfa30 100644 --- a/README.md +++ b/README.md @@ -94,6 +94,33 @@ vendor kernel, this is tested directly instead of inferred: This is proven across a 14/14 enterprise tier: [docs/case-study-enterprise-kernels.md](docs/case-study-enterprise-kernels.md). +### CoreOS / OpenShift (Ignition boot) + +CoreOS-family nodes boot via **Ignition**, not cloud-init, so they need a +different bootstrap. bpfcompat implements it (Ignition config over QEMU +`-fw_cfg`), and the two cases differ only by image availability: + +- **Fedora CoreOS — supported, image is public.** Fetch with + `make vm-image-fcos` and run; proven booting FCOS stable (kernel `7.0.11`) with + load + attach inside the guest. +- **RHCOS / OpenShift — opt-in, operator-supplied image.** RHCOS ships with an + OpenShift release rather than an evergreen public cloud image, so it is **off + by default** and never claimed runnable without a real image. Stage your image + and opt in: + + ```sh + make rhcos-image RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2 # or RHCOS_IMAGE_URL=... + BPFCOMPAT_ENABLE_RHCOS=1 bpfcompat test -artifact build/probe.bpf.o \ + -matrix matrices/rhcos.yaml -runner vm -out report.json + ``` + + Recorded run: RHCOS `416.94…` on kernel `5.14.0-427.93.1.el9_4` (OpenShift + 4.16), ring-buffer artifact load + attach **pass** — + [docs/evidence-rhcos.md](docs/evidence-rhcos.md). Without an image, the **RHEL / + AlmaLinux 9 (5.14)** profiles are the interim kernel approximation (RHCOS for + 4.16 is the RHEL 9.4 kernel). Full guide: + [docs/rhcos-openshift.md](docs/rhcos-openshift.md). + ## Try it in CI without your own KVM box GitHub-hosted Linux runners now expose `/dev/kvm`, so the full QEMU VM @@ -548,6 +575,7 @@ User guide — start here: - [`docs/image-pipeline.md`](docs/image-pipeline.md) — where images come from, integrity, adding profiles - [`docs/upstream-kernel-virtme-ng.md`](docs/upstream-kernel-virtme-ng.md) - [`docs/firecracker-backend.md`](docs/firecracker-backend.md) +- [`docs/rhcos-openshift.md`](docs/rhcos-openshift.md) — RHCOS/OpenShift (Ignition boot, operator-supplied image) - [`docs/api-web-ui.md`](docs/api-web-ui.md) Reference matrices (real, reproducible artifacts): @@ -555,6 +583,7 @@ Reference matrices (real, reproducible artifacts): - [`docs/case-study-falco-modern-bpf.md`](docs/case-study-falco-modern-bpf.md) — Falco `modern_bpf` across 5 kernels - [`docs/case-study-enterprise-kernels.md`](docs/case-study-enterprise-kernels.md) — RHEL/Oracle/Amazon/SUSE backported tier - [`docs/case-study-inspektor-gadget.md`](docs/case-study-inspektor-gadget.md) — published gadgets from OCI, zero config +- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.16, load + attach inside a real RHCOS guest Internal evidence and program docs (acceptance records, runbooks, and planning notes — useful for contributors, not needed to use the tool): diff --git a/docs/evidence-rhcos.md b/docs/evidence-rhcos.md new file mode 100644 index 0000000..a743ec0 --- /dev/null +++ b/docs/evidence-rhcos.md @@ -0,0 +1,94 @@ +# Evidence: RHEL CoreOS (OpenShift) validation + +A real validation run of a compiled eBPF artifact **inside a booted RHEL CoreOS +guest**, recorded here as committed evidence. This is the proof behind the opt-in +RHCOS path documented in [docs/rhcos-openshift.md](rhcos-openshift.md). Reproduce +it with the steps at the bottom. + +> The raw run artifacts (full `report.json`, `validator-result.json`, serial log) +> are written under `evidence/rhcos/` locally; that path is git-ignored as +> high-churn output, so the decisive fields are inlined below. + +## Result + +| Field | Value | +|---|---| +| Profile | `rhcos-4.16-5.14` | +| Booted OS | Red Hat Enterprise Linux CoreOS `416.94.202510081640-0` (OpenShift 4.16) | +| Kernel | `5.14.0-427.93.1.el9_4.x86_64` (RHEL 9.4 base, heavily backported) | +| Arch | x86_64 | +| Boot path | Ignition via QEMU `-fw_cfg name=opt/com.coreos/config`; SSH as `core` | +| Artifact | `examples/ringbuf-modern/ringbuf_modern.bpf.o` (sha256 `569df554…21728`) | +| Load | **pass** (errno 0) | +| Attach | **pass** (1/1, best-effort) | +| Kernel BTF | present (`/sys/kernel/btf/vmlinux`, 4876642 bytes) | +| Overall | **pass** | + +The artifact uses a BPF ring buffer, upstream since **5.8**. It loads on this +**5.14** RHCOS kernel because RHEL backports the feature — "kernel version ≠ +feature support," tested by booting the real vendor kernel rather than inferred. + +## In-guest validator output (`validator.v0.4`, key fields) + +```json +{ + "schema_version": "validator.v0.4", + "status": "pass", + "host": { + "release": "5.14.0-427.93.1.el9_4.x86_64", + "version": "#1 SMP PREEMPT_DYNAMIC Wed Oct 1 11:45:46 EDT 2025", + "machine": "x86_64" + }, + "load": { "status": "pass", "error_code": 0, "error": "" }, + "attach": { "mode": "best-effort", "status": "pass", "attempted": 1, "passed": 1, "failed": 0 }, + "btf": { "kernel_btf_available": true, "kernel_btf_size": 4876642, + "artifact_has_btf": true, "artifact_has_btf_ext": true } +} +``` + +## Guest serial console (excerpt, ANSI stripped) + +``` +GRUB: Booting `Red Hat Enterprise Linux CoreOS 416.94.202510081640-0 (ostree:0)' + +[0.000000] Linux version 5.14.0-427.93.1.el9_4.x86_64 + (mockbuild@x86-64-03.build.eng.rdu2.redhat.com) #1 SMP PREEMPT_DYNAMIC Wed Oct 1 ... +[0.000000] Command line: ... vmlinuz-5.14.0-427.93.1.el9_4.x86_64 rw ignition.firstboot + ostree=/ostree/boot.1/rhcos/... ignition.platform.id=qemu console=ttyS0,115200n8 + +Welcome to Red Hat Enterprise Linux CoreOS 416.94.202510081640-0 + dracut-057-54.git20250423.el9_4.1 (Initramfs)! + +[1.291367] systemd[1]: Starting CoreOS Ignition User Config Setup... +[ OK ] Finished CoreOS Ignition User Config Setup. +``` + +`ignition.platform.id=qemu` + "CoreOS Ignition User Config Setup" confirm the +boot used the Ignition config bpfcompat delivered over `-fw_cfg`. + +## Provenance + +- Image: `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2.gz`, public OpenShift mirror + (`mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest/`). + Published sha256 of the `.gz`: + `92880764c1b3b61940bc209ee021b97474c4db2d9a36abcece55ddd6d8c17c95`. +- Decompressed qcow2 base-image sha256 (recorded in `report.json`): + `d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399`. +- The pull secret is required for the OpenShift *container release payload*, not + the RHCOS boot qcow2 used here. + +## Reproduce + +```sh +base=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest +URL=$(curl -fsSL "$base/sha256sum.txt" | awk '/qemu.x86_64.qcow2.gz$/{print "'"$base"'/"$2; exit}') + +make rhcos-image RHCOS_IMAGE_URL="$URL" + +BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \ + -artifact examples/ringbuf-modern/ringbuf_modern.bpf.o \ + -matrix matrices/rhcos.yaml -runner vm -out report.json +``` + +Enterprises with an internal mirror or an `openshift-install`-extracted image +pass `RHCOS_IMAGE=/path/to/rhcos.qcow2` instead of `RHCOS_IMAGE_URL`. diff --git a/docs/rhcos-openshift.md b/docs/rhcos-openshift.md new file mode 100644 index 0000000..39c4b3d --- /dev/null +++ b/docs/rhcos-openshift.md @@ -0,0 +1,92 @@ +# RHEL CoreOS / OpenShift validation + +bpfcompat can validate a compiled eBPF artifact **inside a real RHEL CoreOS +(RHCOS) guest** — the immutable node OS used by OpenShift. This is opt-in and +needs an operator-supplied image; the rest of this page is how and why. + +## TL;DR + +```sh +# 1. Stage an RHCOS qemu image (see "Getting the image" for how to obtain a URL). +make rhcos-image RHCOS_IMAGE_URL="https://.../rhcos-...-qemu.x86_64.qcow2.gz" +# ...or from a local file: +make rhcos-image RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2 + +# 2. Run, opting in. +BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \ + -artifact build/probe.bpf.o \ + -matrix matrices/rhcos.yaml -runner vm -out report.json +``` + +A recorded run is in [`docs/evidence-rhcos.md`](evidence-rhcos.md): RHCOS +`416.94…` on kernel `5.14.0-427.93.1.el9_4` (OpenShift 4.16), a ring-buffer +artifact loading and attaching — **pass**. + +## Why it is opt-in + +Two facts shape the design: + +1. **Boot is solved.** RHCOS boots via [Ignition](https://coreos.github.io/ignition/), + not cloud-init. bpfcompat writes a minimal Ignition config (an SSH key for the + `core` user) and passes it to QEMU via `-fw_cfg name=opt/com.coreos/config` + (`internal/vm/ignition.go`). This is the **same path proven on Fedora CoreOS**, + which is freely fetchable (`make vm-image-fcos`) and runs out of the box. +2. **The image is operator-specific.** RHCOS ships with an OpenShift release, not + as a single evergreen public cloud image. So bpfcompat does **not** bundle or + auto-fetch a "default" RHCOS — the operator supplies the exact image matching + their cluster. + +Because of (2), the `rhcos` profile is **unsupported by default**: +`ExecutionTransport()` refuses it unless `BPFCOMPAT_ENABLE_RHCOS=1` is set, so +bpfcompat never claims RHCOS works without a real image present. Setting the flag +is the operator asserting "I have staged a real RHCOS image." + +## Getting the image + +RHCOS *boot* images (the qcow2) are published on the public OpenShift mirror — +the **pull secret is for the container release payload, not the boot disk**. For +a given OpenShift minor version: + +```sh +base=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest +URL=$(curl -fsSL "$base/sha256sum.txt" | awk '/qemu.x86_64.qcow2.gz$/{print "'"$base"'/"$2; exit}') +make rhcos-image RHCOS_IMAGE_URL="$URL" +``` + +Alternatively, pin the exact build your cluster runs with `openshift-install`: + +```sh +openshift-install coreos print-stream-json \ + | jq -r '.architectures.x86_64.artifacts.qemu.formats["qcow2.gz"].disk.location' +``` + +Enterprises with an internal mirror or an air-gapped copy pass the local file +with `RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2` instead. `.gz`/`.xz` images +are decompressed automatically; the result is staged at +`vm/cache/rhcos-4.16.qcow2`. + +## Kernel approximation without an image + +If you cannot supply an RHCOS image, the **RHEL / AlmaLinux / Rocky 9 (5.14)** +profiles approximate the RHCOS kernel closely: RHCOS for OpenShift 4.16 is the +RHEL 9.4 kernel (`5.14.0-427`), the same heavily-backported base. That covers +most "will my eBPF load on this kernel?" questions; a true RHCOS boot adds the +immutable/ostree userspace and the exact vendor build on top. + +## What the run proves + +[docs/evidence-rhcos.md](evidence-rhcos.md) records load + attach **inside** the +booted RHCOS guest. The sample artifact uses a BPF ring buffer (upstream since +5.8) and passes on the 5.14 RHCOS kernel — RHEL's backport in action, tested +directly rather than +inferred from the version number. + +## Notes & limits + +- The profile id `rhcos-4.16-5.14` documents the expected base; the **real** + booted kernel is captured at runtime in `report.json`. +- The Ignition config is minimal by design (SSH key only). RHCOS in production is + configured by the Machine Config Operator; bpfcompat only needs enough to get + a shell and run the validator. +- Requires `/dev/kvm` (or it degrades to slower TCG software emulation) and + enough memory for the guest (the profile requests 2 GiB). diff --git a/matrices/rhcos.yaml b/matrices/rhcos.yaml new file mode 100644 index 0000000..8b4e84d --- /dev/null +++ b/matrices/rhcos.yaml @@ -0,0 +1,7 @@ +# RHEL CoreOS (OpenShift) — opt-in, operator-supplied image. +# Stage the image with `make rhcos-image RHCOS_IMAGE=... ` (or RHCOS_IMAGE_URL=...) +# and run with BPFCOMPAT_ENABLE_RHCOS=1. See docs/rhcos-openshift.md. +name: rhcos +profiles: + - id: rhcos-4.16-5.14 + required: false