From 4ce47d13af431900eb4d1feee3066efc43f54885 Mon Sep 17 00:00:00 2001 From: AprilNEA Date: Fri, 7 Aug 2026 17:59:08 +0800 Subject: [PATCH 1/5] feat: add x86_64 microvm kernel flavor for bare-metal Linux hosts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The microvm flavor goes two-arch. x86_64 serves Firecracker sandboxes on bare-metal Linux KVM — ArcBox-on-Linux and the platform PaaS fleet (arcbox-bootkit currently pins the stock Firecracker CI kernel there). The config mirrors the arm64 flavor's cuts but encodes the x86_64 device model where it genuinely differs: ACPI stays ON (it is the Firecracker boot protocol and the virtio-mmio discovery path since 1.7 — the old VIRTIO_MMIO_CMDLINE_DEVICES note was stale), kvmclock replaces the missing RTC (no RTC_CLASS at all; ptp_kvm keeps the restore-resync path), the console is the legacy COM1 8250, and the artifact is the ELF vmlinux at the source root, not bzImage. The platform boot contract adds the one boot-path divergence from arm64: BLK_DEV_INITRD + RD_ZSTD for the bootkit initramfs, and squashfs with xz (backhand's default) + zlib + zstd decompressors for the run-env image. build-kernel.sh drops the arm64-only gate, splits the microvm assertion set per arch (PL031 + no-ACPI on arm64; ACPI + KVM_GUEST + initramfs + squashfs on x86_64), and learns that vmlinux lives at the source root. CI grows the fourth matrix leg and boot-smokes the freshly built x86_64 kernel under the pinned Firecracker on the KVM-capable amd64 runner — ACPI boot, virtio-mmio discovery, virtio-blk, ext4 root, ttyS0, and init exec proven on every build, since Apple Silicon dev machines cannot run it. The release ships microvm-kernel-x86_64 alongside the existing three artifacts. --- .github/workflows/build.yml | 60 +++++- CLAUDE.md | 16 +- README.md | 20 +- configs/arcbox-microvm-x86_64.config | 280 +++++++++++++++++++++++++++ scripts/build-kernel.sh | 45 +++-- 5 files changed, 392 insertions(+), 29 deletions(-) create mode 100644 configs/arcbox-microvm-x86_64.config diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 9f8a794..e2e8b7d 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -25,8 +25,9 @@ jobs: # ========================================================================== # Build kernel for each architecture x flavor (native runners, no # cross-compile). `system` boots the VZ/HV System VM; `microvm` boots - # Firecracker sandbox microVMs (arm64-only for now — Firecracker x86_64 - # needs an ELF vmlinux and its own config). + # Firecracker sandbox microVMs (arm64 nested on macOS, x86_64 bare-metal + # Linux hosts — the x86_64 artifact is the ELF vmlinux Firecracker boots + # directly, and it is KVM boot-smoked in CI below). # ========================================================================== build-kernel: name: Build Kernel (${{ matrix.arch }}, ${{ matrix.flavor }}) @@ -46,6 +47,10 @@ jobs: flavor: microvm artifact: microvm-kernel-arm64 runner: ubuntu-24.04-arm + - arch: x86_64 + flavor: microvm + artifact: microvm-kernel-x86_64 + runner: ubuntu-24.04 steps: - name: Checkout @@ -72,6 +77,52 @@ jobs: OUTPUT_DIR="$(pwd)/output" \ ./scripts/build-kernel.sh + # x86_64 hosted runners expose /dev/kvm, so the one kernel we cannot + # boot on developer hardware (Apple Silicon) is the one we CAN boot + # in CI: bring up the freshly built ELF vmlinux under the pinned + # Firecracker with a busybox init that prints a marker. This + # exercises the exact risk set of this flavor — ACPI boot + ACPI + # virtio-mmio discovery, virtio-blk, ext4 root, ttyS0 console, and + # init exec. arm64 runners have no KVM; that flavor is validated by + # the sandbox e2e suite on real hardware instead. + - name: Boot-smoke under Firecracker (KVM) + if: matrix.artifact == 'microvm-kernel-x86_64' + run: | + set -eux + test -e /dev/kvm + sudo chmod a+rw /dev/kvm + FC_VER=v1.16.1 + curl -sfL "https://github.com/firecracker-microvm/firecracker/releases/download/${FC_VER}/firecracker-${FC_VER}-x86_64.tgz" | tar xz + FC="release-${FC_VER}-x86_64/firecracker-${FC_VER}-x86_64" + sudo apt-get install -y busybox-static + dd if=/dev/zero of=rootfs.ext4 bs=1M count=16 + mkfs.ext4 -q rootfs.ext4 + mkdir mnt && sudo mount -o loop rootfs.ext4 mnt + sudo mkdir -p mnt/bin mnt/dev mnt/proc mnt/sys + sudo cp "$(command -v busybox)" mnt/bin/busybox + printf '#!/bin/busybox sh\n/bin/busybox echo ARCBOX-X86-MICROVM-BOOT-OK\n/bin/busybox reboot -f\n' | sudo tee mnt/init >/dev/null + sudo chmod +x mnt/init + sudo umount mnt + cat > fc.json <<'CFG' + { + "boot-source": { + "kernel_image_path": "output/microvm-kernel-x86_64", + "boot_args": "console=ttyS0 reboot=k panic=-1 pci=off root=/dev/vda rw init=/init" + }, + "drives": [ + { + "drive_id": "rootfs", + "path_on_host": "rootfs.ext4", + "is_root_device": true, + "is_read_only": false + } + ], + "machine-config": { "vcpu_count": 1, "mem_size_mib": 128 } + } + CFG + timeout 60 "$FC" --no-api --config-file fc.json | tee console.log || true + grep -q ARCBOX-X86-MICROVM-BOOT-OK console.log + - name: Generate checksum run: | cd output @@ -104,7 +155,7 @@ jobs: - name: Prepare release files run: | mkdir -p release - for NAME in kernel-arm64 kernel-x86_64 microvm-kernel-arm64; do + for NAME in kernel-arm64 kernel-x86_64 microvm-kernel-arm64 microvm-kernel-x86_64; do DIR="artifacts/$NAME" if [ -f "$DIR/$NAME" ]; then cp "$DIR/$NAME" "release/$NAME" @@ -126,7 +177,8 @@ jobs: ### Files - `kernel-arm64` — ARM64 Image, System VM (Apple Silicon / ARM VMs) - `kernel-x86_64` — x86_64 bzImage, System VM - - `microvm-kernel-arm64` — ARM64 Image, Firecracker sandbox microVMs + - `microvm-kernel-arm64` — ARM64 Image, Firecracker sandbox microVMs (nested, macOS hosts) + - `microvm-kernel-x86_64` — x86_64 ELF vmlinux, Firecracker sandbox microVMs (bare-metal Linux hosts / platform fleet) files: release/* draft: false prerelease: ${{ contains(github.ref_name, 'alpha') || contains(github.ref_name, 'beta') }} diff --git a/CLAUDE.md b/CLAUDE.md index 7749879..50b45c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,12 +12,16 @@ Two kernel **flavors** build from the same source — do not conflate them: artifact `kernel-{arch}`). Full container stack: netfilter, cgroups controllers, dm, overlayfs, NFS, HZ=1000/voluntary (ABX-498 tuning). - `microvm`: the Firecracker sandbox guest kernel - (`configs/arcbox-microvm-arm64.config`, artifact `microvm-kernel-arm64`, - arm64-only). Runs NESTED inside the System VM; optimized for kernel entry - → `/sbin/vm-agent` in the 200–300 ms class. virtio-mmio only — no - PCI/ACPI/EFI/netfilter/BPF; HZ=100/PREEMPT_NONE (nested ticks are - expensive). Consumed by boot-assets `upstream.toml` as the `vmlinux` - binary (`install_dir = "kernel"`). + (`configs/arcbox-microvm-{arch}.config`, artifact `microvm-kernel-{arch}`). + Optimized for kernel entry → PID 1 in the 200–300 ms class; virtio-mmio + only, no PCI/EFI/netfilter/BPF, HZ=100/PREEMPT_NONE. The arches differ + deliberately: arm64 runs NESTED inside the macOS System VM (DT + discovery, PL031 RTC, no ACPI; consumed by boot-assets `upstream.toml` + as the `vmlinux` binary, `install_dir = "kernel"`); x86_64 runs on + bare-metal Linux KVM (ACPI boot + discovery, kvmclock, ELF vmlinux + artifact, zstd initramfs + xz squashfs for the platform PaaS fleet; + KVM boot-smoked in CI). Keep the per-arch assertion sets in + build-kernel.sh in lockstep with any config edit. A flavor's load-bearing symbols are asserted post-`olddefconfig` in `scripts/build-kernel.sh` — extend the flavor's assertion list when adding a diff --git a/README.md b/README.md index 372354c..d50397e 100644 --- a/README.md +++ b/README.md @@ -16,13 +16,19 @@ Two independent guest kernels are built from the same kernel source: | Flavor | Config | Artifact | Boots | |--------|--------|----------|-------| | `system` (default) | `configs/arcbox-{arch}.config` | `kernel-{arch}` | ArcBox System VM (VZ/HV backends) | -| `microvm` | `configs/arcbox-microvm-arm64.config` | `microvm-kernel-arm64` | Firecracker sandbox microVMs, nested inside the System VM | - -The microvm flavor is arm64-only for now: Firecracker x86_64 boots an ELF -`vmlinux` (not a bzImage) and needs its own config -(`VIRTIO_MMIO_CMDLINE_DEVICES`, kvmclock). It targets kernel entry → -PID 1 in the 200–300 ms class under nested virtualization: virtio-mmio -device model only, no PCI/ACPI/EFI/netfilter, everything built in. +| `microvm` | `configs/arcbox-microvm-{arch}.config` | `microvm-kernel-{arch}` | Firecracker sandbox microVMs | + +The microvm flavor targets kernel entry → PID 1 in the 200–300 ms class: +virtio-mmio device model only, no PCI/EFI/netfilter, everything built in. +The two arches serve different hosts and differ deliberately: arm64 runs +NESTED inside the macOS System VM (DT device discovery, PL031 RTC, Image +artifact, no ACPI), while x86_64 runs on bare-metal Linux KVM hosts — +ArcBox-on-Linux sandboxes and the platform PaaS fleet — where ACPI is the +Firecracker boot protocol, kvmclock replaces the missing RTC, the +artifact is the ELF `vmlinux` Firecracker boots directly, and the +initramfs (zstd) + squashfs (xz) pair carries the platform boot contract. +CI boot-smokes the x86_64 kernel under Firecracker on a KVM-capable +runner. ```bash FLAVOR=microvm ./scripts/build-kernel.sh diff --git a/configs/arcbox-microvm-x86_64.config b/configs/arcbox-microvm-x86_64.config new file mode 100644 index 0000000..3304b7b --- /dev/null +++ b/configs/arcbox-microvm-x86_64.config @@ -0,0 +1,280 @@ +# ArcBox microVM (sandbox) kernel configuration for x86_64 +# +# Guest kernel for Firecracker sandbox microVMs on x86_64 Linux hosts — +# bare-metal KVM, not nested. Serves both consumers of the sandbox path: +# ArcBox-on-Linux sandboxes (vm-agent over vsock, ext4 template rootfs) +# and the platform PaaS fleet (arcbox-bootkit initramfs owning the mounts: +# zstd cpio rdinit, ext4 template on /dev/vda, xz squashfs run-env on +# /dev/vdb). +# +# The Firecracker x86_64 device model differs from aarch64 in ways this +# config encodes deliberately: +# - Boot protocol and device discovery are ACPI (Firecracker >= 1.7): +# virtio-mmio devices come from the DSDT, so ACPI stays ON here (the +# arm64 flavor cuts it — devices come from DT there) and +# VIRTIO_MMIO_CMDLINE_DEVICES is NOT needed. PCI and EFI stay off; the +# fleet boots with pci=off. +# - The bootable artifact is the uncompressed ELF vmlinux at the source +# root, not arch/x86/boot/bzImage (build-kernel.sh overrides +# KERNEL_IMAGE for this flavor). +# - There is no RTC device: cold-boot wall time comes from kvmclock +# (KVM_GUEST reads the host wall clock at boot), post-restore resync +# from ptp_kvm (/dev/ptp0, KVM_HC_CLOCK_PAIRING) — so no RTC_CLASS, +# where the arm64 flavor carries PL031. +# - The serial console is the legacy COM1 16550 at 0x3f8 (ttyS0), probed +# by the static 8250 table — no PNP/DT enumeration needed. +# +# Everything built in; CONFIG_MODULES stays off. Fragment style matches +# configs/arcbox-microvm-arm64.config: copied to .config, resolved by +# olddefconfig, then asserted by scripts/build-kernel.sh (microvm x86_64 +# assertion set). EXPERT is required to expose the VT/INPUT/... disables. + +# Basic setup +CONFIG_LOCALVERSION="-arcbox-microvm" +CONFIG_DEFAULT_HOSTNAME="arcbox-sandbox" +CONFIG_EXPERT=y +CONFIG_IKCONFIG=y +CONFIG_IKCONFIG_PROC=y +# CONFIG_MODULES is not set + +# Processor type and features +CONFIG_64BIT=y +CONFIG_SMP=y +CONFIG_NR_CPUS=32 +# (No HOTPLUG_CPU disable here, unlike arm64: x86 SMP def_bool-selects it.) +# 64-bit-only guests: templates are baked from amd64 OCI images +# (oci2rootfs), and dropping the compat syscall surface mirrors the arm64 +# flavor's COMPAT cut. +# CONFIG_IA32_EMULATION is not set +# CONFIG_X86_16BIT is not set +# The sandbox never hosts further guests; KVM init has no business here. +# CONFIG_VIRTUALIZATION is not set + +# KVM paravirt guest: kvmclock is the boot-time wall-clock source (there +# is no RTC device on Firecracker x86_64) and the clocksource; ptp_kvm +# rides the same interface for post-restore resync. Steal-time accounting +# is real on a shared bare-metal fleet node. +CONFIG_HYPERVISOR_GUEST=y +CONFIG_PARAVIRT=y +CONFIG_KVM_GUEST=y +CONFIG_PARAVIRT_TIME_ACCOUNTING=y + +# Core syscall surface for arbitrary workloads. Several of these are +# EXPERT-gated defaults; stated explicitly so a Kconfig default change can +# never silently drop them. +CONFIG_MULTIUSER=y +CONFIG_SYSVIPC=y +CONFIG_POSIX_MQUEUE=y +CONFIG_POSIX_TIMERS=y +CONFIG_FUTEX=y +CONFIG_EPOLL=y +CONFIG_SIGNALFD=y +CONFIG_TIMERFD=y +CONFIG_EVENTFD=y +CONFIG_AIO=y +CONFIG_IO_URING=y +CONFIG_ADVISE_SYSCALLS=y +CONFIG_MEMBARRIER=y +CONFIG_RSEQ=y +CONFIG_SHMEM=y +CONFIG_KALLSYMS=y +CONFIG_ELF_CORE=y +CONFIG_COREDUMP=y +CONFIG_BINFMT_ELF=y +CONFIG_BINFMT_SCRIPT=y + +# Timing / scheduler: HZ=100 + NO_HZ_IDLE + PREEMPT_NONE, same call as the +# arm64 flavor — a sandbox runs a single workload, so rare ticks and no +# preemption churn win even without the nested-virt tick tax. +CONFIG_HIGH_RES_TIMERS=y +CONFIG_NO_HZ_IDLE=y +CONFIG_HZ_100=y +CONFIG_HZ=100 +CONFIG_PREEMPT_NONE=y +# PSI taxes the scheduler wakeup fast path (ABX-498); nothing in the +# microVM consumes /proc/pressure. +# CONFIG_PSI is not set + +# Isolation primitives workloads actually use: Chromium/Puppeteer sandboxes +# need USER_NS + seccomp; language runtimes probe /proc/self/cgroup (core +# cgroups only, no controllers — a sandbox is a single workload, memory +# limiting is VM sizing). +CONFIG_NAMESPACES=y +CONFIG_USER_NS=y +CONFIG_PID_NS=y +CONFIG_NET_NS=y +CONFIG_IPC_NS=y +CONFIG_UTS_NS=y +CONFIG_TIME_NS=y +CONFIG_SECCOMP=y +CONFIG_SECCOMP_FILTER=y +CONFIG_CGROUPS=y + +# ACPI is load-bearing on x86 Firecracker (boot protocol, virtio-mmio +# discovery, VMGenID), but the DSDT is tiny — cut the default-on drivers +# for hardware Firecracker never presents. CPU-side mitigations for +# untrusted workloads stay at kernel defaults on purpose: do not add +# mitigations=off-style toggles to this config. +CONFIG_ACPI=y +# CONFIG_ACPI_AC is not set +# CONFIG_ACPI_BATTERY is not set +# CONFIG_ACPI_BUTTON is not set +# CONFIG_ACPI_FAN is not set +# CONFIG_ACPI_THERMAL is not set +# CONFIG_ACPI_TABLE_UPGRADE is not set + +# Boot-time fat from the Firecracker CI kernel, explicitly OFF +# CONFIG_AUDIT is not set +# CONFIG_PROFILING is not set +# CONFIG_BPF_SYSCALL is not set +# CONFIG_KEXEC is not set +# CONFIG_KEXEC_FILE is not set +# CONFIG_CRASH_DUMP is not set +# CONFIG_EFI is not set +# CONFIG_PCI is not set +# CONFIG_NUMA is not set +# CONFIG_SWAP is not set +# CONFIG_SUSPEND is not set +# CONFIG_HIBERNATION is not set +# CONFIG_CPU_FREQ is not set +# CONFIG_RANDOMIZE_BASE is not set +# CONFIG_IOMMU_SUPPORT is not set +# CONFIG_VT is not set +# CONFIG_INPUT is not set +# CONFIG_HID_SUPPORT is not set +# CONFIG_USB_SUPPORT is not set +# CONFIG_SOUND is not set +# CONFIG_WLAN is not set +# CONFIG_WIRELESS is not set +# CONFIG_RFKILL is not set +# CONFIG_DEBUG_FS is not set +# CONFIG_FTRACE is not set +# CONFIG_LOCKUP_DETECTOR is not set +# CONFIG_DETECT_HUNG_TASK is not set +# CONFIG_SCHED_DEBUG is not set +CONFIG_MAGIC_SYSRQ=y +CONFIG_JUMP_LABEL=y + +# Memory management. THP=madvise: opt-in for workloads that ask, without +# khugepaged churning snapshot dirty-page sets in the background. +CONFIG_TRANSPARENT_HUGEPAGE=y +CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y +# CONFIG_MEMORY_HOTPLUG is not set +# CONFIG_ZSWAP is not set + +# Networking. Static addressing comes from the kernel ip= parameter that +# arcbox-vm/sandbox/boot.rs appends (ip=:::::eth0:off), +# handled by IP_PNP; the autoconf protocols are never used. The platform +# fleet configures networking from its initramfs instead — both ride the +# same INET stack. NAT/firewalling happens on the host side; no iptables +# in the microVM. +CONFIG_NET=y +CONFIG_PACKET=y +CONFIG_UNIX=y +CONFIG_INET=y +CONFIG_IP_PNP=y +# CONFIG_IP_PNP_DHCP is not set +# CONFIG_IP_PNP_BOOTP is not set +# CONFIG_IP_PNP_RARP is not set +CONFIG_IPV6=y +# CONFIG_NETFILTER is not set +# CONFIG_BRIDGE is not set +CONFIG_VSOCKETS=y +CONFIG_VIRTIO_VSOCKETS=y + +# Firecracker device model. NETDEVICES/NET_CORE are NOT default-on: without +# them olddefconfig silently drops VIRTIO_NET (the exact silent-degrade +# class the build assertions exist for). ETHERNET is the vendor-NIC +# umbrella; virtio-net does not need it. +CONFIG_BLOCK=y +CONFIG_BLK_DEV=y +CONFIG_NETDEVICES=y +CONFIG_NET_CORE=y +# CONFIG_ETHERNET is not set +CONFIG_VIRTIO=y +CONFIG_VIRTIO_MENU=y +CONFIG_VIRTIO_MMIO=y +CONFIG_VIRTIO_BLK=y +CONFIG_VIRTIO_NET=y +# Balloon: the only memory-reclaim lever Firecracker offers; kept built-in +# so a future reclaim feature (and snapshots that include a balloon device) +# need no kernel change. Costs nothing at boot when the device is absent. +CONFIG_VIRTIO_BALLOON=y +CONFIG_HW_RANDOM=y +CONFIG_HW_RANDOM_VIRTIO=y +CONFIG_TTY=y +CONFIG_SERIAL_8250=y +CONFIG_SERIAL_8250_CONSOLE=y +CONFIG_SERIAL_8250_NR_UARTS=1 +CONFIG_SERIAL_8250_RUNTIME_UARTS=1 +CONFIG_SERIAL_EARLYCON=y +CONFIG_UNIX98_PTYS=y +CONFIG_DEVPTS_FS=y +# CONFIG_LEGACY_PTYS is not set +# The default sandbox rootfs ships an EMPTY /dev; without the devtmpfs +# automount PID 1 has no /dev/console for stdio (vm-agent mounts /dev +# itself, but only after its fds are already bound). +CONFIG_DEVTMPFS=y +CONFIG_DEVTMPFS_MOUNT=y + +# initramfs: the platform fleet boots rdinit=/init from a zstd cpio built +# by arcbox-bootkit; only the zstd decompressor is carried (the archive +# format is ours, not user input). ArcBox-on-Linux sandboxes boot +# root=/dev/vda directly and never load one — an absent initrd costs +# nothing. This is the one boot-path divergence from the arm64 flavor. +CONFIG_BLK_DEV_INITRD=y +CONFIG_RD_ZSTD=y +# CONFIG_RD_GZIP is not set +# CONFIG_RD_BZIP2 is not set +# CONFIG_RD_LZMA is not set +# CONFIG_RD_XZ is not set +# CONFIG_RD_LZO is not set +# CONFIG_RD_LZ4 is not set + +# Time: cold-boot wall clock + post-restore resync (CORE-75). Firecracker +# x86_64 exposes NO RTC device, so RTC_CLASS stays out entirely: kvmclock +# (KVM_GUEST above) sets CLOCK_REALTIME from the host wall clock at boot, +# and ptp_kvm exposes the host CLOCK_REALTIME at ns granularity via +# KVM_HC_CLOCK_PAIRING (/dev/ptp0) for vm-agent's accept-path resync. +# ptp_kvm being the only PTP driver also keeps /dev/ptp0 unambiguous. +CONFIG_PTP_1588_CLOCK=y +CONFIG_PTP_1588_CLOCK_KVM=y +# ACPI-based VMGenID: reseeds the guest RNG when a restored clone diverges +# from its snapshot origin. VIRT_DRIVERS is the (default-off) menu gate it +# lives under. +CONFIG_VIRT_DRIVERS=y +CONFIG_VMGENID=y + +# File systems. ext4 for template rootfs images (oci2rootfs); squashfs for +# the platform run-env image (backhand-written, xz by default — zlib and +# zstd decompressors kept so a compressor change in bootkit is not a +# kernel event). Still no overlayfs: the platform initramfs mounts the +# template and run-env side by side (verified against arcbox-bootkit's +# initramfs sources), and the ArcBox CoW layer lives host-side. +CONFIG_EXT4_FS=y +CONFIG_EXT4_FS_POSIX_ACL=y +CONFIG_EXT4_FS_SECURITY=y +CONFIG_SQUASHFS=y +CONFIG_SQUASHFS_XATTR=y +CONFIG_SQUASHFS_ZLIB=y +CONFIG_SQUASHFS_XZ=y +CONFIG_SQUASHFS_ZSTD=y +CONFIG_PROC_FS=y +CONFIG_PROC_SYSCTL=y +CONFIG_SYSFS=y +CONFIG_TMPFS=y +CONFIG_TMPFS_POSIX_ACL=y +CONFIG_TMPFS_XATTR=y +CONFIG_INOTIFY_USER=y +CONFIG_FANOTIFY=y +CONFIG_FILE_LOCKING=y +# CONFIG_OVERLAY_FS is not set +# CONFIG_FUSE_FS is not set +# CONFIG_BLK_DEV_LOOP is not set +# CONFIG_QUOTA is not set + +# Crypto (ext4 metadata checksums; squashfs decompressors pull their +# libraries via select) +CONFIG_CRYPTO=y +CONFIG_CRYPTO_CRC32C=y +CONFIG_CRC32=y diff --git a/scripts/build-kernel.sh b/scripts/build-kernel.sh index 73346dc..2ffbfad 100755 --- a/scripts/build-kernel.sh +++ b/scripts/build-kernel.sh @@ -10,9 +10,9 @@ # Flavors: # system (default) — System VM guest kernel (VZ/HV backends), # configs/arcbox-{arch}.config, output kernel-{arch} -# microvm — Firecracker sandbox guest kernel (arm64 only), +# microvm — Firecracker sandbox guest kernel, # configs/arcbox-microvm-{arch}.config, -# output microvm-kernel-{arch} +# output microvm-kernel-{arch} (x86_64: ELF vmlinux) set -e @@ -56,22 +56,34 @@ if [ "$FLAVOR" = "system" ]; then CONFIG_PREEMPT_VOLUNTARY" ASSERT_N="" elif [ "$FLAVOR" = "microvm" ]; then - if [ "$TARGET_ARCH" != "arm64" ]; then - echo "Error: microvm flavor is arm64-only (Firecracker x86_64 needs an ELF vmlinux and a separate config)" - exit 1 - fi CONFIG_FILE="$CONFIG_DIR/arcbox-microvm-$TARGET_ARCH.config" OUTPUT_NAME="microvm-kernel-$TARGET_ARCH" # The Firecracker sandbox contract: virtio-mmio devices, vsock exec/PTY, - # devtmpfs automount over the empty template /dev, static ip=, PL031 + - # ptp_kvm clocks, VMGenID RNG reseed. Also assert that the deliberately + # devtmpfs automount over the empty template /dev, static ip=, ptp_kvm + # restore resync, VMGenID RNG reseed. Also assert that the deliberately # cut subsystems stayed cut (a fragment typo re-enabling PCI/netfilter # would otherwise ship silently). ASSERT_Y="CONFIG_VIRTIO_MMIO CONFIG_VIRTIO_BLK CONFIG_VIRTIO_NET CONFIG_VIRTIO_VSOCKETS CONFIG_DEVTMPFS_MOUNT CONFIG_IP_PNP CONFIG_UNIX98_PTYS CONFIG_EXT4_FS CONFIG_SERIAL_8250_CONSOLE - CONFIG_RTC_DRV_PL031 CONFIG_PTP_1588_CLOCK_KVM CONFIG_VMGENID" - ASSERT_N="CONFIG_PCI CONFIG_NETFILTER CONFIG_MODULES CONFIG_ACPI CONFIG_EFI" + CONFIG_PTP_1588_CLOCK_KVM CONFIG_VMGENID" + ASSERT_N="CONFIG_PCI CONFIG_NETFILTER CONFIG_MODULES CONFIG_EFI" + if [ "$TARGET_ARCH" = "arm64" ]; then + # aarch64 Firecracker: DT device discovery, PL031 RTC; ACPI is cut + # and must stay cut. + ASSERT_Y="$ASSERT_Y CONFIG_RTC_DRV_PL031" + ASSERT_N="$ASSERT_N CONFIG_ACPI" + else + # x86_64 Firecracker: ACPI IS the boot protocol and virtio-mmio + # discovery path (>= 1.7), kvmclock replaces the missing RTC, and + # the bootable artifact is the ELF vmlinux at the source root — + # not bzImage. The initramfs/squashfs pair is the platform PaaS + # boot contract (arcbox-bootkit: zstd cpio + xz run-env image). + ASSERT_Y="$ASSERT_Y CONFIG_ACPI CONFIG_KVM_GUEST + CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD + CONFIG_SQUASHFS CONFIG_SQUASHFS_XZ" + KERNEL_IMAGE="vmlinux" + fi else echo "Error: Unsupported flavor: $FLAVOR (expected 'system' or 'microvm')" exit 1 @@ -83,6 +95,15 @@ fi ASSERT_Y=$(echo $ASSERT_Y) ASSERT_N=$(echo $ASSERT_N) +# Where the bootable artifact lands in the source tree. vmlinux is the ELF +# at the root (Firecracker x86_64 boots it directly); everything else is a +# packaged image under arch/*/boot. +if [ "$KERNEL_IMAGE" = "vmlinux" ]; then + KERNEL_IMAGE_SRC="vmlinux" +else + KERNEL_IMAGE_SRC="arch/$TARGET_ARCH/boot/$KERNEL_IMAGE" +fi + echo "========================================" echo " ArcBox Kernel Build" echo "========================================" @@ -140,7 +161,7 @@ do_build() { make ARCH=$TARGET_ARCH ${CROSS_COMPILE:+CROSS_COMPILE=$CROSS_COMPILE} -j"$(nproc)" $KERNEL_IMAGE # Copy output. - cp "arch/$TARGET_ARCH/boot/$KERNEL_IMAGE" "$OUTPUT_PATH" + cp "$KERNEL_IMAGE_SRC" "$OUTPUT_PATH" echo "" echo "Build complete!" ls -lh "$OUTPUT_PATH" @@ -191,7 +212,7 @@ for sym in $ASSERT_N; do done echo 'Building kernel...' make ARCH=$TARGET_ARCH -j\$(nproc) $KERNEL_IMAGE -cp arch/$TARGET_ARCH/boot/$KERNEL_IMAGE /output/$OUTPUT_NAME +cp $KERNEL_IMAGE_SRC /output/$OUTPUT_NAME echo 'Build complete!' ls -lh /output/$OUTPUT_NAME " From f94fb228df5d9e0b2c4e43566ab7eab5fa19c33a Mon Sep 17 00:00:00 2001 From: AprilNEA Date: Fri, 7 Aug 2026 18:10:06 +0800 Subject: [PATCH 2/5] fix: virtio-mmio devices come from the cmdline on x86_64, and pin the smoke's Firecracker digest MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI boot smoke caught the design error on its first run: the FC v1.16 x86_64 DSDT is 0x22E bytes with no virtio nodes — ACPI carries the boot protocol, MADT and VMGenID, but virtio-mmio devices arrive only via the virtio_mmio.device= entries Firecracker auto-appends to the cmdline. The smoke booted to 'VFS: unable to mount root' with the device announced on the cmdline and no driver listening. VIRTIO_MMIO_CMDLINE_DEVICES is now on and asserted; the config/README narrative tells the measured story. Review fixes: the smoke's Firecracker tarball is digest-pinned before execution, and the RANDOMIZE_BASE line now documents why it is inert here (x86 KASLR lives in the bzImage decompression stub, which direct ELF loading bypasses) rather than reading as a security trade. --- .github/workflows/build.yml | 5 ++++- CLAUDE.md | 6 +++--- README.md | 10 ++++++---- configs/arcbox-microvm-x86_64.config | 26 ++++++++++++++++++-------- scripts/build-kernel.sh | 16 +++++++++------- 5 files changed, 40 insertions(+), 23 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index e2e8b7d..eb034ea 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -92,7 +92,10 @@ jobs: test -e /dev/kvm sudo chmod a+rw /dev/kvm FC_VER=v1.16.1 - curl -sfL "https://github.com/firecracker-microvm/firecracker/releases/download/${FC_VER}/firecracker-${FC_VER}-x86_64.tgz" | tar xz + FC_SHA256=382a02a869e4d6d5cb14c40577f9545e8458021ea8b0b2d3fc10ec14d9c242e6 + curl -sfL "https://github.com/firecracker-microvm/firecracker/releases/download/${FC_VER}/firecracker-${FC_VER}-x86_64.tgz" -o fc.tgz + echo "${FC_SHA256} fc.tgz" | sha256sum -c - + tar xzf fc.tgz FC="release-${FC_VER}-x86_64/firecracker-${FC_VER}-x86_64" sudo apt-get install -y busybox-static dd if=/dev/zero of=rootfs.ext4 bs=1M count=16 diff --git a/CLAUDE.md b/CLAUDE.md index 50b45c4..9180aa6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,9 +18,9 @@ Two kernel **flavors** build from the same source — do not conflate them: deliberately: arm64 runs NESTED inside the macOS System VM (DT discovery, PL031 RTC, no ACPI; consumed by boot-assets `upstream.toml` as the `vmlinux` binary, `install_dir = "kernel"`); x86_64 runs on - bare-metal Linux KVM (ACPI boot + discovery, kvmclock, ELF vmlinux - artifact, zstd initramfs + xz squashfs for the platform PaaS fleet; - KVM boot-smoked in CI). Keep the per-arch assertion sets in + bare-metal Linux KVM (ACPI boot/MADT/VMGenID + cmdline virtio-mmio + discovery, kvmclock, ELF vmlinux artifact, zstd initramfs + xz + squashfs for the platform PaaS fleet; KVM boot-smoked in CI). Keep the per-arch assertion sets in build-kernel.sh in lockstep with any config edit. A flavor's load-bearing symbols are asserted post-`olddefconfig` in diff --git a/README.md b/README.md index d50397e..83b8562 100644 --- a/README.md +++ b/README.md @@ -23,10 +23,12 @@ virtio-mmio device model only, no PCI/EFI/netfilter, everything built in. The two arches serve different hosts and differ deliberately: arm64 runs NESTED inside the macOS System VM (DT device discovery, PL031 RTC, Image artifact, no ACPI), while x86_64 runs on bare-metal Linux KVM hosts — -ArcBox-on-Linux sandboxes and the platform PaaS fleet — where ACPI is the -Firecracker boot protocol, kvmclock replaces the missing RTC, the -artifact is the ELF `vmlinux` Firecracker boots directly, and the -initramfs (zstd) + squashfs (xz) pair carries the platform boot contract. +ArcBox-on-Linux sandboxes and the platform PaaS fleet — where ACPI +carries the boot protocol/MADT/VMGenID, virtio-mmio devices arrive via +Firecracker's auto-appended `virtio_mmio.device=` cmdline entries, +kvmclock replaces the missing RTC, the artifact is the ELF `vmlinux` +Firecracker boots directly, and the initramfs (zstd) + squashfs (xz) +pair carries the platform boot contract. CI boot-smokes the x86_64 kernel under Firecracker on a KVM-capable runner. diff --git a/configs/arcbox-microvm-x86_64.config b/configs/arcbox-microvm-x86_64.config index 3304b7b..76a5da6 100644 --- a/configs/arcbox-microvm-x86_64.config +++ b/configs/arcbox-microvm-x86_64.config @@ -8,12 +8,15 @@ # /dev/vdb). # # The Firecracker x86_64 device model differs from aarch64 in ways this -# config encodes deliberately: -# - Boot protocol and device discovery are ACPI (Firecracker >= 1.7): -# virtio-mmio devices come from the DSDT, so ACPI stays ON here (the -# arm64 flavor cuts it — devices come from DT there) and -# VIRTIO_MMIO_CMDLINE_DEVICES is NOT needed. PCI and EFI stay off; the -# fleet boots with pci=off. +# config encodes deliberately (each verified by the CI KVM boot smoke): +# - ACPI stays ON: Firecracker >= 1.7 boots x86_64 guests via ACPI — +# RSDP/FADT/MADT (SMP configuration) and the VMGenID device live there. +# virtio-mmio devices do NOT: the measured FC v1.16 DSDT is 0x22E bytes +# with no virtio nodes, and FC instead auto-appends +# virtio_mmio.device=... to the guest cmdline — so +# VIRTIO_MMIO_CMDLINE_DEVICES is load-bearing (without it the root +# device never appears and boot panics; the smoke caught exactly this). +# PCI and EFI stay off; the fleet boots with pci=off. # - The bootable artifact is the uncompressed ELF vmlinux at the source # root, not arch/x86/boot/bzImage (build-kernel.sh overrides # KERNEL_IMAGE for this flavor). @@ -110,8 +113,8 @@ CONFIG_SECCOMP=y CONFIG_SECCOMP_FILTER=y CONFIG_CGROUPS=y -# ACPI is load-bearing on x86 Firecracker (boot protocol, virtio-mmio -# discovery, VMGenID), but the DSDT is tiny — cut the default-on drivers +# ACPI is load-bearing on x86 Firecracker (boot protocol, MADT SMP +# tables, VMGenID), but the DSDT is tiny — cut the default-on drivers # for hardware Firecracker never presents. CPU-side mitigations for # untrusted workloads stay at kernel defaults on purpose: do not add # mitigations=off-style toggles to this config. @@ -137,6 +140,9 @@ CONFIG_ACPI=y # CONFIG_SUSPEND is not set # CONFIG_HIBERNATION is not set # CONFIG_CPU_FREQ is not set +# KASLR is inert under Firecracker's direct ELF load: x86 randomization +# lives in the bzImage decompression stub, which loading vmlinux bypasses +# — stating it off documents reality rather than trading security away. # CONFIG_RANDOMIZE_BASE is not set # CONFIG_IOMMU_SUPPORT is not set # CONFIG_VT is not set @@ -194,6 +200,10 @@ CONFIG_NET_CORE=y CONFIG_VIRTIO=y CONFIG_VIRTIO_MENU=y CONFIG_VIRTIO_MMIO=y +# Firecracker x86_64 announces virtio-mmio devices ONLY via auto-appended +# virtio_mmio.device= cmdline entries (see header) — this is the discovery +# path, not a fallback. +CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y CONFIG_VIRTIO_BLK=y CONFIG_VIRTIO_NET=y # Balloon: the only memory-reclaim lever Firecracker offers; kept built-in diff --git a/scripts/build-kernel.sh b/scripts/build-kernel.sh index 2ffbfad..662c44d 100755 --- a/scripts/build-kernel.sh +++ b/scripts/build-kernel.sh @@ -74,13 +74,15 @@ elif [ "$FLAVOR" = "microvm" ]; then ASSERT_Y="$ASSERT_Y CONFIG_RTC_DRV_PL031" ASSERT_N="$ASSERT_N CONFIG_ACPI" else - # x86_64 Firecracker: ACPI IS the boot protocol and virtio-mmio - # discovery path (>= 1.7), kvmclock replaces the missing RTC, and - # the bootable artifact is the ELF vmlinux at the source root — - # not bzImage. The initramfs/squashfs pair is the platform PaaS - # boot contract (arcbox-bootkit: zstd cpio + xz run-env image). - ASSERT_Y="$ASSERT_Y CONFIG_ACPI CONFIG_KVM_GUEST - CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD + # x86_64 Firecracker: ACPI carries the boot protocol, MADT (SMP) + # and VMGenID, while virtio-mmio devices arrive via auto-appended + # virtio_mmio.device= cmdline entries (the FC DSDT has no virtio + # nodes — CI-boot-smoke-verified), kvmclock replaces the missing + # RTC, and the bootable artifact is the ELF vmlinux at the source + # root — not bzImage. The initramfs/squashfs pair is the platform + # PaaS boot contract (arcbox-bootkit: zstd cpio + xz run-env). + ASSERT_Y="$ASSERT_Y CONFIG_ACPI CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES + CONFIG_KVM_GUEST CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD CONFIG_SQUASHFS CONFIG_SQUASHFS_XZ" KERNEL_IMAGE="vmlinux" fi From ac61beb7493aeced596e68a8bed4169bbe3c5cf3 Mon Sep 17 00:00:00 2001 From: AprilNEA Date: Fri, 7 Aug 2026 18:28:10 +0800 Subject: [PATCH 3/5] fix: preallocate legacy IRQ descriptors so Firecracker virtio-mmio IRQs exist MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The boot smoke's second finding, root-caused end to end: FC's MADT carries no PCAT_COMPAT (madt.rs header flags = 0) and its device model does not answer the i8259 data-port probe, so kernels >= 6.7 select the NULL PIC before early_irq_init() and preallocate zero legacy IRQ descriptors ('preallocated irqs: 0'). FC still hands ISA-range GSIs to its cmdline virtio-mmio devices, and request_irq() without a descriptor is -EINVAL — 'virtio_blk: probe failed with error -22', every device dead. FC's own CI kernels (6.1) predate the behavior change, which is why the incompatibility is invisible in their testing. One-hunk patch via the repo's patch mechanism: when the probe finds no PIC, still preallocate the 16-descriptor legacy range — the exact arrangement (NULL PIC + preallocated descriptors + IOAPIC delivery) the FC CI kernels demonstrably work in. Real-PIC and PCAT_COMPAT systems are unaffected; the system flavors pay 16 spare descriptors. --- ...6-preallocate-legacy-irq-descriptors.patch | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 patches/x86-preallocate-legacy-irq-descriptors.patch diff --git a/patches/x86-preallocate-legacy-irq-descriptors.patch b/patches/x86-preallocate-legacy-irq-descriptors.patch new file mode 100644 index 0000000..e1b9cc0 --- /dev/null +++ b/patches/x86-preallocate-legacy-irq-descriptors.patch @@ -0,0 +1,42 @@ +x86/apic: preallocate legacy IRQ descriptors when the PIC probe finds nothing + +Firecracker's MADT does not advertise PCAT_COMPAT (acpi-tables +madt.rs: header flags = 0) and its device model does not answer the +i8259 data-port probe, so on kernels >= 6.7 legacy_pic->probe() selects +the NULL PIC and returns 0 — and early_irq_init() then preallocates no +legacy IRQ descriptors at all ("preallocated irqs: 0" in the boot log). +Firecracker still assigns ISA-range GSIs (5..23) to virtio-mmio devices +announced on the kernel command line, and request_irq() on a +descriptor-less IRQ fails with -EINVAL, which surfaces as +"virtio_blk: probe with driver virtio_blk failed with error -22" and a +fully unusable device model. + +Kernels <= 6.6 preallocated the legacy descriptors *before* probing the +PIC, and that is exactly the environment Firecracker's own CI guest +kernels (5.10/6.1) demonstrably work in: NULL PIC, 16 preallocated +descriptors, interrupts delivered through the IOAPIC. Restore that +arrangement by preallocating the legacy range even when the probe finds +no PIC. On hardware (or VMMs) with a real PIC or a PCAT_COMPAT MADT +the probe result is unchanged; the cost elsewhere is 16 spare +descriptors. + +Verified by this repo's CI Firecracker KVM boot smoke. + +--- a/arch/x86/kernel/apic/vector.c ++++ b/arch/x86/kernel/apic/vector.c +@@ -737,8 +737,14 @@ + /* + * We don't know if PIC is present at this point so we need to do + * probe() to get the right number of legacy IRQs. ++ * ++ * ArcBox: a probe that finds no PIC must still preallocate the ++ * legacy range — Firecracker hands out ISA-range GSIs for ++ * virtio-mmio with no PIC and no PCAT_COMPAT advertised, and a ++ * descriptor-less request_irq() fails -EINVAL (see patch header). + */ +- return legacy_pic->probe(); ++ nr = legacy_pic->probe(); ++ return nr ? nr : NR_IRQS_LEGACY; + } + + void lapic_assign_legacy_vector(unsigned int irq, bool replace) From b2238917aec436f3422ddf93b0843e76aadb8237 Mon Sep 17 00:00:00 2001 From: AprilNEA Date: Fri, 7 Aug 2026 18:38:11 +0800 Subject: [PATCH 4/5] fix: assume the PIC present instead of only preallocating descriptors MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The smoke's round-3 verdict showed the descriptor-only patch fixed half the failure: request_irq moved from -EINVAL (no descriptor) to -ENOSYS (descriptor with no_irq_chip). The legacy bring-up is a chain — probe verdict -> descriptor prealloc -> mp_irqs identity mappings (mp_config_acpi_legacy_irqs loops over nr_legacy_irqs()) -> IOAPIC wiring — and patching stages individually is whack-a-mole. Supersede it at the root, the way probe_8259A's own PCAT_COMPAT comment prescribes: skip the port probe and pretend the PIC is there, so the whole chain comes up exactly as on a PCAT system. Every VMM this tree's kernels boot under either emulates the PIC (QEMU/KVM) or needs the full legacy setup despite lacking one (Firecracker). --- patches/x86-i8259-assume-pic-present.patch | 85 +++++++++++++++++++ ...6-preallocate-legacy-irq-descriptors.patch | 42 --------- 2 files changed, 85 insertions(+), 42 deletions(-) create mode 100644 patches/x86-i8259-assume-pic-present.patch delete mode 100644 patches/x86-preallocate-legacy-irq-descriptors.patch diff --git a/patches/x86-i8259-assume-pic-present.patch b/patches/x86-i8259-assume-pic-present.patch new file mode 100644 index 0000000..92c4a62 --- /dev/null +++ b/patches/x86-i8259-assume-pic-present.patch @@ -0,0 +1,85 @@ +x86/i8259: treat the PIC as present when PCAT_COMPAT is not advertised + +Firecracker's MADT does not advertise PCAT_COMPAT (acpi-tables madt.rs: +header flags = 0) and its device model does not answer the i8259 +data-port probe, so on kernels >= 6.7 — where the probe runs before +early_irq_init() — probe_8259A() selects the NULL PIC early enough to +suppress the entire legacy interrupt bring-up: no preallocated legacy +descriptors ("preallocated irqs: 0"), no mp_irqs identity mappings from +mp_config_acpi_legacy_irqs() (its loop is bounded by nr_legacy_irqs()), +and no IOAPIC wiring for the ISA range. Firecracker still assigns +ISA-range GSIs (5..23) to the virtio-mmio devices it announces on the +kernel command line, so request_irq() fails (-EINVAL with no +descriptor; -ENOSYS with a descriptor but no_irq_chip) and every virtio +device is dead: "virtio_blk: probe with driver virtio_blk failed". +Firecracker's own CI guest kernels (5.10/6.1) predate the probe +reordering, which is why the incompatibility is invisible upstream. + +Fix it the way probe_8259A()'s own PCAT_COMPAT comment prescribes: +"just pretend that the PIC is there and let legacy_pic->init() +initialize it for nothing" — skip the port probe in the +no-PCAT_COMPAT case too. Every VMM this tree's kernels boot under +either emulates the PIC (QEMU/KVM hosts) or needs the full legacy +setup despite lacking one (Firecracker), so pretending is correct in +all of them: init pokes a few nonexistent ports at boot and interrupts +are delivered through the IOAPIC exactly as on a PCAT system. + +Supersedes the narrower descriptor-preallocation approach: descriptors +alone leave the ISA range without mp_irqs mappings, and request_irq() +still fails with -ENOSYS (no_irq_chip) — both stages observed by the +CI Firecracker KVM boot smoke that verifies this patch. + +--- a/arch/x86/kernel/i8259.c ++++ b/arch/x86/kernel/i8259.c +@@ -301,9 +301,6 @@ + + static int probe_8259A(void) + { +- unsigned char new_val, probe_val = ~(1 << PIC_CASCADE_IR); +- unsigned long flags; +- + /* + * If MADT has the PCAT_COMPAT flag set, then do not bother probing + * for the PIC. Some BIOSes leave the PIC uninitialized and probing +@@ -316,29 +313,19 @@ + * must be routed through the PIC. So just pretend that the PIC is + * there and let legacy_pic->init() initialize it for nothing. + * +- * Alternatively this could just try to initialize the PIC and +- * repeat the probe, but for cases where there is no PIC that's +- * just pointless. +- */ +- if (pcat_compat) +- return nr_legacy_irqs(); +- +- /* +- * Check to see if we have a PIC. Mask all except the cascade and +- * read back the value we just wrote. If we don't have a PIC, we +- * will read 0xff as opposed to the value we wrote. ++ * ArcBox: extend that reasoning to the no-PCAT_COMPAT case and skip ++ * the port probe entirely. Firecracker advertises no PCAT_COMPAT ++ * (MADT flags = 0) and answers the data-port probe with all-ones, ++ * so the probe would select the NULL PIC *before* early_irq_init() ++ * — no legacy descriptors, no mp_irqs identity mappings, no IOAPIC ++ * wiring for the ISA range — yet Firecracker assigns ISA-range GSIs ++ * to its cmdline virtio-mmio devices, whose request_irq() then ++ * fails and every virtio device is dead. Every VMM this tree's ++ * kernels boot under either emulates the PIC (QEMU/KVM) or needs ++ * the full legacy setup despite lacking one (Firecracker), so ++ * pretending is correct in all of them; init pokes a few ++ * nonexistent ports at boot and the IOAPIC delivers as usual. + */ +- raw_spin_lock_irqsave(&i8259A_lock, flags); +- +- outb(0xff, PIC_SLAVE_IMR); /* mask all of 8259A-2 */ +- outb(probe_val, PIC_MASTER_IMR); +- new_val = inb(PIC_MASTER_IMR); +- if (new_val != probe_val) { +- printk(KERN_INFO "Using NULL legacy PIC\n"); +- legacy_pic = &null_legacy_pic; +- } +- +- raw_spin_unlock_irqrestore(&i8259A_lock, flags); + return nr_legacy_irqs(); + } + diff --git a/patches/x86-preallocate-legacy-irq-descriptors.patch b/patches/x86-preallocate-legacy-irq-descriptors.patch deleted file mode 100644 index e1b9cc0..0000000 --- a/patches/x86-preallocate-legacy-irq-descriptors.patch +++ /dev/null @@ -1,42 +0,0 @@ -x86/apic: preallocate legacy IRQ descriptors when the PIC probe finds nothing - -Firecracker's MADT does not advertise PCAT_COMPAT (acpi-tables -madt.rs: header flags = 0) and its device model does not answer the -i8259 data-port probe, so on kernels >= 6.7 legacy_pic->probe() selects -the NULL PIC and returns 0 — and early_irq_init() then preallocates no -legacy IRQ descriptors at all ("preallocated irqs: 0" in the boot log). -Firecracker still assigns ISA-range GSIs (5..23) to virtio-mmio devices -announced on the kernel command line, and request_irq() on a -descriptor-less IRQ fails with -EINVAL, which surfaces as -"virtio_blk: probe with driver virtio_blk failed with error -22" and a -fully unusable device model. - -Kernels <= 6.6 preallocated the legacy descriptors *before* probing the -PIC, and that is exactly the environment Firecracker's own CI guest -kernels (5.10/6.1) demonstrably work in: NULL PIC, 16 preallocated -descriptors, interrupts delivered through the IOAPIC. Restore that -arrangement by preallocating the legacy range even when the probe finds -no PIC. On hardware (or VMMs) with a real PIC or a PCAT_COMPAT MADT -the probe result is unchanged; the cost elsewhere is 16 spare -descriptors. - -Verified by this repo's CI Firecracker KVM boot smoke. - ---- a/arch/x86/kernel/apic/vector.c -+++ b/arch/x86/kernel/apic/vector.c -@@ -737,8 +737,14 @@ - /* - * We don't know if PIC is present at this point so we need to do - * probe() to get the right number of legacy IRQs. -+ * -+ * ArcBox: a probe that finds no PIC must still preallocate the -+ * legacy range — Firecracker hands out ISA-range GSIs for -+ * virtio-mmio with no PIC and no PCAT_COMPAT advertised, and a -+ * descriptor-less request_irq() fails -EINVAL (see patch header). - */ -- return legacy_pic->probe(); -+ nr = legacy_pic->probe(); -+ return nr ? nr : NR_IRQS_LEGACY; - } - - void lapic_assign_legacy_vector(unsigned int irq, bool replace) From ef58860c51c6e1e3ccc614d78e73838c189da339 Mon Sep 17 00:00:00 2001 From: AprilNEA Date: Fri, 7 Aug 2026 18:51:38 +0800 Subject: [PATCH 5/5] fix(x86-microvm): boot via hardware-reduced ACPI as Firecracker's kernel policy prescribes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three smoke rounds converged on the real model, confirmed by FC's own docs/kernel-policy.md. Firecracker x86_64 is a hardware-reduced ACPI platform: the FADT sets HW_REDUCED_ACPI (nulling the legacy PIC is by-design, so the i8259 patch is deleted, not needed), the DSDT enumerates virtio-mmio as LNRO0005 nodes, and — the piece this config was missing — ACPI initialization inside the guest requires CONFIG_PCI even though no PCI device ever appears. Cutting PCI produced 'AE_BAD_PARAMETER, During Region initialization' + 'Unable to load the System Description Tables', which silently killed all ACPI enumeration and left no virtio devices at all. The cmdline discovery enabled in the previous fix is reverted to off: FC deprecates it explicitly (kernel-policy.md suggests VIRTIO_MMIO_CMDLINE_DEVICES=n and X86_MPPARSE=n), and it is actively harmful here — FC still auto-appends the entries, and with the option on they spawn duplicate devices with raw ISA IRQs that cannot work on a hardware-reduced platform (the -22/-38 request_irq failures of rounds 2-4). Asserts follow: PCI moves to ASSERT_Y on x86_64 (stays ASSERT_N on arm64), VIRTIO_MMIO_CMDLINE_DEVICES and X86_MPPARSE join x86_64's ASSERT_N. --- CLAUDE.md | 7 +- README.md | 13 ++-- configs/arcbox-microvm-x86_64.config | 44 +++++++---- patches/x86-i8259-assume-pic-present.patch | 85 ---------------------- scripts/build-kernel.sh | 30 ++++---- 5 files changed, 56 insertions(+), 123 deletions(-) delete mode 100644 patches/x86-i8259-assume-pic-present.patch diff --git a/CLAUDE.md b/CLAUDE.md index 9180aa6..4b52a3d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -18,9 +18,10 @@ Two kernel **flavors** build from the same source — do not conflate them: deliberately: arm64 runs NESTED inside the macOS System VM (DT discovery, PL031 RTC, no ACPI; consumed by boot-assets `upstream.toml` as the `vmlinux` binary, `install_dir = "kernel"`); x86_64 runs on - bare-metal Linux KVM (ACPI boot/MADT/VMGenID + cmdline virtio-mmio - discovery, kvmclock, ELF vmlinux artifact, zstd initramfs + xz - squashfs for the platform PaaS fleet; KVM boot-smoked in CI). Keep the per-arch assertion sets in + bare-metal Linux KVM (hardware-reduced ACPI: DSDT-enumerated + virtio-mmio, PCI config option required by ACPI init, kvmclock, ELF + vmlinux artifact, zstd initramfs + xz squashfs for the platform PaaS + fleet; KVM boot-smoked in CI). Keep the per-arch assertion sets in build-kernel.sh in lockstep with any config edit. A flavor's load-bearing symbols are asserted post-`olddefconfig` in diff --git a/README.md b/README.md index 83b8562..e408527 100644 --- a/README.md +++ b/README.md @@ -23,12 +23,13 @@ virtio-mmio device model only, no PCI/EFI/netfilter, everything built in. The two arches serve different hosts and differ deliberately: arm64 runs NESTED inside the macOS System VM (DT device discovery, PL031 RTC, Image artifact, no ACPI), while x86_64 runs on bare-metal Linux KVM hosts — -ArcBox-on-Linux sandboxes and the platform PaaS fleet — where ACPI -carries the boot protocol/MADT/VMGenID, virtio-mmio devices arrive via -Firecracker's auto-appended `virtio_mmio.device=` cmdline entries, -kvmclock replaces the missing RTC, the artifact is the ELF `vmlinux` -Firecracker boots directly, and the initramfs (zstd) + squashfs (xz) -pair carries the platform boot contract. +ArcBox-on-Linux sandboxes and the platform PaaS fleet — where +Firecracker presents a hardware-reduced ACPI platform (DSDT-enumerated +virtio-mmio, `CONFIG_PCI` required by ACPI init per FC's kernel policy +despite zero PCI devices), kvmclock replaces the missing RTC, the +artifact is the ELF `vmlinux` Firecracker boots directly, and the +initramfs (zstd) + squashfs (xz) pair carries the platform boot +contract. CI boot-smokes the x86_64 kernel under Firecracker on a KVM-capable runner. diff --git a/configs/arcbox-microvm-x86_64.config b/configs/arcbox-microvm-x86_64.config index 76a5da6..2dc110a 100644 --- a/configs/arcbox-microvm-x86_64.config +++ b/configs/arcbox-microvm-x86_64.config @@ -8,15 +8,24 @@ # /dev/vdb). # # The Firecracker x86_64 device model differs from aarch64 in ways this -# config encodes deliberately (each verified by the CI KVM boot smoke): -# - ACPI stays ON: Firecracker >= 1.7 boots x86_64 guests via ACPI — -# RSDP/FADT/MADT (SMP configuration) and the VMGenID device live there. -# virtio-mmio devices do NOT: the measured FC v1.16 DSDT is 0x22E bytes -# with no virtio nodes, and FC instead auto-appends -# virtio_mmio.device=... to the guest cmdline — so -# VIRTIO_MMIO_CMDLINE_DEVICES is load-bearing (without it the root -# device never appears and boot panics; the smoke caught exactly this). -# PCI and EFI stay off; the fleet boots with pci=off. +# config encodes deliberately (each verified by the CI KVM boot smoke, +# and matching FC's own docs/kernel-policy.md): +# - Firecracker x86_64 is a HARDWARE-REDUCED ACPI platform: the FADT +# sets HW_REDUCED_ACPI (so the kernel nulls the legacy PIC by design +# — "preallocated irqs: 0" is normal), MADT carries SMP config, and +# the DSDT carries the virtio-mmio devices as LNRO0005 nodes, IRQs +# mapped through ACPI onto the IOAPIC. The deprecated fallbacks are +# explicitly off per FC's kernel policy: no MPTable parsing and no +# cmdline device discovery (FC still auto-appends virtio_mmio.device= +# entries; with the option off they are inert, and with it on they +# spawn duplicate devices with raw ISA IRQs that cannot work in +# hardware-reduced mode). +# - CONFIG_PCI stays ON despite zero PCI devices: FC's kernel policy is +# explicit that ACPI initialization inside the guest needs it — the +# smoke measured the failure shape as "AE_BAD_PARAMETER, During +# Region initialization" + "Unable to load the System Description +# Tables", killing all ACPI enumeration. The fleet boots with pci=off +# so nothing scans. EFI stays off. # - The bootable artifact is the uncompressed ELF vmlinux at the source # root, not arch/x86/boot/bzImage (build-kernel.sh overrides # KERNEL_IMAGE for this flavor). @@ -113,12 +122,16 @@ CONFIG_SECCOMP=y CONFIG_SECCOMP_FILTER=y CONFIG_CGROUPS=y -# ACPI is load-bearing on x86 Firecracker (boot protocol, MADT SMP -# tables, VMGenID), but the DSDT is tiny — cut the default-on drivers +# ACPI is load-bearing on x86 Firecracker (hardware-reduced boot, MADT +# SMP tables, LNRO0005 virtio-mmio enumeration, VMGenID), and PCI is +# load-bearing for ACPI itself (FC kernel policy; no PCI device ever +# appears — the fleet boots pci=off). Cut the default-on ACPI drivers # for hardware Firecracker never presents. CPU-side mitigations for # untrusted workloads stay at kernel defaults on purpose: do not add # mitigations=off-style toggles to this config. CONFIG_ACPI=y +CONFIG_PCI=y +# CONFIG_X86_MPPARSE is not set # CONFIG_ACPI_AC is not set # CONFIG_ACPI_BATTERY is not set # CONFIG_ACPI_BUTTON is not set @@ -134,7 +147,6 @@ CONFIG_ACPI=y # CONFIG_KEXEC_FILE is not set # CONFIG_CRASH_DUMP is not set # CONFIG_EFI is not set -# CONFIG_PCI is not set # CONFIG_NUMA is not set # CONFIG_SWAP is not set # CONFIG_SUSPEND is not set @@ -200,10 +212,10 @@ CONFIG_NET_CORE=y CONFIG_VIRTIO=y CONFIG_VIRTIO_MENU=y CONFIG_VIRTIO_MMIO=y -# Firecracker x86_64 announces virtio-mmio devices ONLY via auto-appended -# virtio_mmio.device= cmdline entries (see header) — this is the discovery -# path, not a fallback. -CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y +# Discovery is ACPI (LNRO0005 in the DSDT — see header); the cmdline +# mechanism is deprecated by FC and actively harmful here, so it stays +# off and the build asserts it stayed off. +# CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES is not set CONFIG_VIRTIO_BLK=y CONFIG_VIRTIO_NET=y # Balloon: the only memory-reclaim lever Firecracker offers; kept built-in diff --git a/patches/x86-i8259-assume-pic-present.patch b/patches/x86-i8259-assume-pic-present.patch deleted file mode 100644 index 92c4a62..0000000 --- a/patches/x86-i8259-assume-pic-present.patch +++ /dev/null @@ -1,85 +0,0 @@ -x86/i8259: treat the PIC as present when PCAT_COMPAT is not advertised - -Firecracker's MADT does not advertise PCAT_COMPAT (acpi-tables madt.rs: -header flags = 0) and its device model does not answer the i8259 -data-port probe, so on kernels >= 6.7 — where the probe runs before -early_irq_init() — probe_8259A() selects the NULL PIC early enough to -suppress the entire legacy interrupt bring-up: no preallocated legacy -descriptors ("preallocated irqs: 0"), no mp_irqs identity mappings from -mp_config_acpi_legacy_irqs() (its loop is bounded by nr_legacy_irqs()), -and no IOAPIC wiring for the ISA range. Firecracker still assigns -ISA-range GSIs (5..23) to the virtio-mmio devices it announces on the -kernel command line, so request_irq() fails (-EINVAL with no -descriptor; -ENOSYS with a descriptor but no_irq_chip) and every virtio -device is dead: "virtio_blk: probe with driver virtio_blk failed". -Firecracker's own CI guest kernels (5.10/6.1) predate the probe -reordering, which is why the incompatibility is invisible upstream. - -Fix it the way probe_8259A()'s own PCAT_COMPAT comment prescribes: -"just pretend that the PIC is there and let legacy_pic->init() -initialize it for nothing" — skip the port probe in the -no-PCAT_COMPAT case too. Every VMM this tree's kernels boot under -either emulates the PIC (QEMU/KVM hosts) or needs the full legacy -setup despite lacking one (Firecracker), so pretending is correct in -all of them: init pokes a few nonexistent ports at boot and interrupts -are delivered through the IOAPIC exactly as on a PCAT system. - -Supersedes the narrower descriptor-preallocation approach: descriptors -alone leave the ISA range without mp_irqs mappings, and request_irq() -still fails with -ENOSYS (no_irq_chip) — both stages observed by the -CI Firecracker KVM boot smoke that verifies this patch. - ---- a/arch/x86/kernel/i8259.c -+++ b/arch/x86/kernel/i8259.c -@@ -301,9 +301,6 @@ - - static int probe_8259A(void) - { -- unsigned char new_val, probe_val = ~(1 << PIC_CASCADE_IR); -- unsigned long flags; -- - /* - * If MADT has the PCAT_COMPAT flag set, then do not bother probing - * for the PIC. Some BIOSes leave the PIC uninitialized and probing -@@ -316,29 +313,19 @@ - * must be routed through the PIC. So just pretend that the PIC is - * there and let legacy_pic->init() initialize it for nothing. - * -- * Alternatively this could just try to initialize the PIC and -- * repeat the probe, but for cases where there is no PIC that's -- * just pointless. -- */ -- if (pcat_compat) -- return nr_legacy_irqs(); -- -- /* -- * Check to see if we have a PIC. Mask all except the cascade and -- * read back the value we just wrote. If we don't have a PIC, we -- * will read 0xff as opposed to the value we wrote. -+ * ArcBox: extend that reasoning to the no-PCAT_COMPAT case and skip -+ * the port probe entirely. Firecracker advertises no PCAT_COMPAT -+ * (MADT flags = 0) and answers the data-port probe with all-ones, -+ * so the probe would select the NULL PIC *before* early_irq_init() -+ * — no legacy descriptors, no mp_irqs identity mappings, no IOAPIC -+ * wiring for the ISA range — yet Firecracker assigns ISA-range GSIs -+ * to its cmdline virtio-mmio devices, whose request_irq() then -+ * fails and every virtio device is dead. Every VMM this tree's -+ * kernels boot under either emulates the PIC (QEMU/KVM) or needs -+ * the full legacy setup despite lacking one (Firecracker), so -+ * pretending is correct in all of them; init pokes a few -+ * nonexistent ports at boot and the IOAPIC delivers as usual. - */ -- raw_spin_lock_irqsave(&i8259A_lock, flags); -- -- outb(0xff, PIC_SLAVE_IMR); /* mask all of 8259A-2 */ -- outb(probe_val, PIC_MASTER_IMR); -- new_val = inb(PIC_MASTER_IMR); -- if (new_val != probe_val) { -- printk(KERN_INFO "Using NULL legacy PIC\n"); -- legacy_pic = &null_legacy_pic; -- } -- -- raw_spin_unlock_irqrestore(&i8259A_lock, flags); - return nr_legacy_irqs(); - } - diff --git a/scripts/build-kernel.sh b/scripts/build-kernel.sh index 662c44d..8d6f2a6 100755 --- a/scripts/build-kernel.sh +++ b/scripts/build-kernel.sh @@ -67,23 +67,27 @@ elif [ "$FLAVOR" = "microvm" ]; then CONFIG_VIRTIO_VSOCKETS CONFIG_DEVTMPFS_MOUNT CONFIG_IP_PNP CONFIG_UNIX98_PTYS CONFIG_EXT4_FS CONFIG_SERIAL_8250_CONSOLE CONFIG_PTP_1588_CLOCK_KVM CONFIG_VMGENID" - ASSERT_N="CONFIG_PCI CONFIG_NETFILTER CONFIG_MODULES CONFIG_EFI" + ASSERT_N="CONFIG_NETFILTER CONFIG_MODULES CONFIG_EFI" if [ "$TARGET_ARCH" = "arm64" ]; then - # aarch64 Firecracker: DT device discovery, PL031 RTC; ACPI is cut - # and must stay cut. + # aarch64 Firecracker: DT device discovery, PL031 RTC; ACPI and + # PCI are cut and must stay cut. ASSERT_Y="$ASSERT_Y CONFIG_RTC_DRV_PL031" - ASSERT_N="$ASSERT_N CONFIG_ACPI" + ASSERT_N="$ASSERT_N CONFIG_ACPI CONFIG_PCI" else - # x86_64 Firecracker: ACPI carries the boot protocol, MADT (SMP) - # and VMGenID, while virtio-mmio devices arrive via auto-appended - # virtio_mmio.device= cmdline entries (the FC DSDT has no virtio - # nodes — CI-boot-smoke-verified), kvmclock replaces the missing - # RTC, and the bootable artifact is the ELF vmlinux at the source - # root — not bzImage. The initramfs/squashfs pair is the platform - # PaaS boot contract (arcbox-bootkit: zstd cpio + xz run-env). - ASSERT_Y="$ASSERT_Y CONFIG_ACPI CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES - CONFIG_KVM_GUEST CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD + # x86_64 Firecracker is a hardware-reduced ACPI platform: the + # DSDT enumerates virtio-mmio (LNRO0005), and per FC's kernel + # policy ACPI initialization needs CONFIG_PCI even though no PCI + # device ever appears. The deprecated fallbacks (MPTable, cmdline + # virtio-mmio) must stay off — the cmdline path spawns duplicate + # devices with raw ISA IRQs that cannot work without a legacy + # PIC. kvmclock replaces the missing RTC; the bootable artifact + # is the ELF vmlinux at the source root — not bzImage. The + # initramfs/squashfs pair is the platform PaaS boot contract + # (arcbox-bootkit: zstd cpio + xz run-env). + ASSERT_Y="$ASSERT_Y CONFIG_ACPI CONFIG_PCI CONFIG_KVM_GUEST + CONFIG_BLK_DEV_INITRD CONFIG_RD_ZSTD CONFIG_SQUASHFS CONFIG_SQUASHFS_XZ" + ASSERT_N="$ASSERT_N CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES CONFIG_X86_MPPARSE" KERNEL_IMAGE="vmlinux" fi else