Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,33 @@ vendor kernel, this is tested directly instead of inferred:
This is proven across a 14/14 enterprise tier:
[docs/case-study-enterprise-kernels.md](docs/case-study-enterprise-kernels.md).

### CoreOS / OpenShift (Ignition boot)

CoreOS-family nodes boot via **Ignition**, not cloud-init, so they need a
different bootstrap. bpfcompat implements it (Ignition config over QEMU
`-fw_cfg`), and the two cases differ only by image availability:

- **Fedora CoreOS — supported, image is public.** Fetch with
`make vm-image-fcos` and run; proven booting FCOS stable (kernel `7.0.11`) with
load + attach inside the guest.
- **RHCOS / OpenShift — opt-in, operator-supplied image.** RHCOS ships with an
OpenShift release rather than an evergreen public cloud image, so it is **off
by default** and never claimed runnable without a real image. Stage your image
and opt in:

```sh
make rhcos-image RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2 # or RHCOS_IMAGE_URL=...
BPFCOMPAT_ENABLE_RHCOS=1 bpfcompat test -artifact build/probe.bpf.o \
-matrix matrices/rhcos.yaml -runner vm -out report.json
```

Recorded run: RHCOS `416.94…` on kernel `5.14.0-427.93.1.el9_4` (OpenShift
4.16), ring-buffer artifact load + attach **pass** —
[docs/evidence-rhcos.md](docs/evidence-rhcos.md). Without an image, the **RHEL /
AlmaLinux 9 (5.14)** profiles are the interim kernel approximation (RHCOS for
4.16 is the RHEL 9.4 kernel). Full guide:
[docs/rhcos-openshift.md](docs/rhcos-openshift.md).

## Try it in CI without your own KVM box

GitHub-hosted Linux runners now expose `/dev/kvm`, so the full QEMU VM
Expand Down Expand Up @@ -548,13 +575,15 @@ User guide — start here:
- [`docs/image-pipeline.md`](docs/image-pipeline.md) — where images come from, integrity, adding profiles
- [`docs/upstream-kernel-virtme-ng.md`](docs/upstream-kernel-virtme-ng.md)
- [`docs/firecracker-backend.md`](docs/firecracker-backend.md)
- [`docs/rhcos-openshift.md`](docs/rhcos-openshift.md) — RHCOS/OpenShift (Ignition boot, operator-supplied image)
- [`docs/api-web-ui.md`](docs/api-web-ui.md)

Reference matrices (real, reproducible artifacts):

- [`docs/case-study-falco-modern-bpf.md`](docs/case-study-falco-modern-bpf.md) — Falco `modern_bpf` across 5 kernels
- [`docs/case-study-enterprise-kernels.md`](docs/case-study-enterprise-kernels.md) — RHEL/Oracle/Amazon/SUSE backported tier
- [`docs/case-study-inspektor-gadget.md`](docs/case-study-inspektor-gadget.md) — published gadgets from OCI, zero config
- [`docs/evidence-rhcos.md`](docs/evidence-rhcos.md) — RHEL CoreOS / OpenShift 4.16, load + attach inside a real RHCOS guest

Internal evidence and program docs (acceptance records, runbooks, and
planning notes — useful for contributors, not needed to use the tool):
Expand Down
94 changes: 94 additions & 0 deletions docs/evidence-rhcos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Evidence: RHEL CoreOS (OpenShift) validation

A real validation run of a compiled eBPF artifact **inside a booted RHEL CoreOS
guest**, recorded here as committed evidence. This is the proof behind the opt-in
RHCOS path documented in [docs/rhcos-openshift.md](rhcos-openshift.md). Reproduce
it with the steps at the bottom.

> The raw run artifacts (full `report.json`, `validator-result.json`, serial log)
> are written under `evidence/rhcos/` locally; that path is git-ignored as
> high-churn output, so the decisive fields are inlined below.

## Result

| Field | Value |
|---|---|
| Profile | `rhcos-4.16-5.14` |
| Booted OS | Red Hat Enterprise Linux CoreOS `416.94.202510081640-0` (OpenShift 4.16) |
| Kernel | `5.14.0-427.93.1.el9_4.x86_64` (RHEL 9.4 base, heavily backported) |
| Arch | x86_64 |
| Boot path | Ignition via QEMU `-fw_cfg name=opt/com.coreos/config`; SSH as `core` |
| Artifact | `examples/ringbuf-modern/ringbuf_modern.bpf.o` (sha256 `569df554…21728`) |
| Load | **pass** (errno 0) |
| Attach | **pass** (1/1, best-effort) |
| Kernel BTF | present (`/sys/kernel/btf/vmlinux`, 4876642 bytes) |
| Overall | **pass** |

The artifact uses a BPF ring buffer, upstream since **5.8**. It loads on this
**5.14** RHCOS kernel because RHEL backports the feature — "kernel version ≠
feature support," tested by booting the real vendor kernel rather than inferred.

## In-guest validator output (`validator.v0.4`, key fields)

```json
{
"schema_version": "validator.v0.4",
"status": "pass",
"host": {
"release": "5.14.0-427.93.1.el9_4.x86_64",
"version": "#1 SMP PREEMPT_DYNAMIC Wed Oct 1 11:45:46 EDT 2025",
"machine": "x86_64"
},
"load": { "status": "pass", "error_code": 0, "error": "" },
"attach": { "mode": "best-effort", "status": "pass", "attempted": 1, "passed": 1, "failed": 0 },
"btf": { "kernel_btf_available": true, "kernel_btf_size": 4876642,
"artifact_has_btf": true, "artifact_has_btf_ext": true }
}
```

## Guest serial console (excerpt, ANSI stripped)

```
GRUB: Booting `Red Hat Enterprise Linux CoreOS 416.94.202510081640-0 (ostree:0)'

[0.000000] Linux version 5.14.0-427.93.1.el9_4.x86_64
(mockbuild@x86-64-03.build.eng.rdu2.redhat.com) #1 SMP PREEMPT_DYNAMIC Wed Oct 1 ...
[0.000000] Command line: ... vmlinuz-5.14.0-427.93.1.el9_4.x86_64 rw ignition.firstboot
ostree=/ostree/boot.1/rhcos/... ignition.platform.id=qemu console=ttyS0,115200n8

Welcome to Red Hat Enterprise Linux CoreOS 416.94.202510081640-0
dracut-057-54.git20250423.el9_4.1 (Initramfs)!

[1.291367] systemd[1]: Starting CoreOS Ignition User Config Setup...
[ OK ] Finished CoreOS Ignition User Config Setup.
```

`ignition.platform.id=qemu` + "CoreOS Ignition User Config Setup" confirm the
boot used the Ignition config bpfcompat delivered over `-fw_cfg`.

## Provenance

- Image: `rhcos-4.16.51-x86_64-qemu.x86_64.qcow2.gz`, public OpenShift mirror
(`mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest/`).
Published sha256 of the `.gz`:
`92880764c1b3b61940bc209ee021b97474c4db2d9a36abcece55ddd6d8c17c95`.
- Decompressed qcow2 base-image sha256 (recorded in `report.json`):
`d03128234c5dc6217bd37ee0caf6f192107d42d39a8a6b5c9b6148b0f4f92399`.
- The pull secret is required for the OpenShift *container release payload*, not
the RHCOS boot qcow2 used here.

## Reproduce

```sh
base=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest
URL=$(curl -fsSL "$base/sha256sum.txt" | awk '/qemu.x86_64.qcow2.gz$/{print "'"$base"'/"$2; exit}')

make rhcos-image RHCOS_IMAGE_URL="$URL"

BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \
-artifact examples/ringbuf-modern/ringbuf_modern.bpf.o \
-matrix matrices/rhcos.yaml -runner vm -out report.json
```

Enterprises with an internal mirror or an `openshift-install`-extracted image
pass `RHCOS_IMAGE=/path/to/rhcos.qcow2` instead of `RHCOS_IMAGE_URL`.
92 changes: 92 additions & 0 deletions docs/rhcos-openshift.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# RHEL CoreOS / OpenShift validation

bpfcompat can validate a compiled eBPF artifact **inside a real RHEL CoreOS
(RHCOS) guest** — the immutable node OS used by OpenShift. This is opt-in and
needs an operator-supplied image; the rest of this page is how and why.

## TL;DR

```sh
# 1. Stage an RHCOS qemu image (see "Getting the image" for how to obtain a URL).
make rhcos-image RHCOS_IMAGE_URL="https://.../rhcos-...-qemu.x86_64.qcow2.gz"
# ...or from a local file:
make rhcos-image RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2

# 2. Run, opting in.
BPFCOMPAT_ENABLE_RHCOS=1 ./bin/bpfcompat test \
-artifact build/probe.bpf.o \
-matrix matrices/rhcos.yaml -runner vm -out report.json
```

A recorded run is in [`docs/evidence-rhcos.md`](evidence-rhcos.md): RHCOS
`416.94…` on kernel `5.14.0-427.93.1.el9_4` (OpenShift 4.16), a ring-buffer
artifact loading and attaching — **pass**.

## Why it is opt-in

Two facts shape the design:

1. **Boot is solved.** RHCOS boots via [Ignition](https://coreos.github.io/ignition/),
not cloud-init. bpfcompat writes a minimal Ignition config (an SSH key for the
`core` user) and passes it to QEMU via `-fw_cfg name=opt/com.coreos/config`
(`internal/vm/ignition.go`). This is the **same path proven on Fedora CoreOS**,
which is freely fetchable (`make vm-image-fcos`) and runs out of the box.
2. **The image is operator-specific.** RHCOS ships with an OpenShift release, not
as a single evergreen public cloud image. So bpfcompat does **not** bundle or
auto-fetch a "default" RHCOS — the operator supplies the exact image matching
their cluster.

Because of (2), the `rhcos` profile is **unsupported by default**:
`ExecutionTransport()` refuses it unless `BPFCOMPAT_ENABLE_RHCOS=1` is set, so
bpfcompat never claims RHCOS works without a real image present. Setting the flag
is the operator asserting "I have staged a real RHCOS image."

## Getting the image

RHCOS *boot* images (the qcow2) are published on the public OpenShift mirror —
the **pull secret is for the container release payload, not the boot disk**. For
a given OpenShift minor version:

```sh
base=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/latest
URL=$(curl -fsSL "$base/sha256sum.txt" | awk '/qemu.x86_64.qcow2.gz$/{print "'"$base"'/"$2; exit}')
make rhcos-image RHCOS_IMAGE_URL="$URL"
```

Alternatively, pin the exact build your cluster runs with `openshift-install`:

```sh
openshift-install coreos print-stream-json \
| jq -r '.architectures.x86_64.artifacts.qemu.formats["qcow2.gz"].disk.location'
```

Enterprises with an internal mirror or an air-gapped copy pass the local file
with `RHCOS_IMAGE=/path/to/rhcos-qemu.x86_64.qcow2` instead. `.gz`/`.xz` images
are decompressed automatically; the result is staged at
`vm/cache/rhcos-4.16.qcow2`.

## Kernel approximation without an image

If you cannot supply an RHCOS image, the **RHEL / AlmaLinux / Rocky 9 (5.14)**
profiles approximate the RHCOS kernel closely: RHCOS for OpenShift 4.16 is the
RHEL 9.4 kernel (`5.14.0-427`), the same heavily-backported base. That covers
most "will my eBPF load on this kernel?" questions; a true RHCOS boot adds the
immutable/ostree userspace and the exact vendor build on top.

## What the run proves

[docs/evidence-rhcos.md](evidence-rhcos.md) records load + attach **inside** the
booted RHCOS guest. The sample artifact uses a BPF ring buffer (upstream since
5.8) and passes on the 5.14 RHCOS kernel — RHEL's backport in action, tested
directly rather than
inferred from the version number.

## Notes & limits

- The profile id `rhcos-4.16-5.14` documents the expected base; the **real**
booted kernel is captured at runtime in `report.json`.
- The Ignition config is minimal by design (SSH key only). RHCOS in production is
configured by the Machine Config Operator; bpfcompat only needs enough to get
a shell and run the validator.
- Requires `/dev/kvm` (or it degrades to slower TCG software emulation) and
enough memory for the guest (the profile requests 2 GiB).
7 changes: 7 additions & 0 deletions matrices/rhcos.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# RHEL CoreOS (OpenShift) — opt-in, operator-supplied image.
# Stage the image with `make rhcos-image RHCOS_IMAGE=... ` (or RHCOS_IMAGE_URL=...)
# and run with BPFCOMPAT_ENABLE_RHCOS=1. See docs/rhcos-openshift.md.
name: rhcos
profiles:
- id: rhcos-4.16-5.14
required: false
Loading