Repository navigation
feat(build): hermetic mock lane -- lock the build root, build offline from the lock - #260
Conversation
… from the lock Behind backend: hermetic (dispatch input, workflow_call input), the way Hummingbird's ci/build_rpms.sh --hermetic builds: 1. Lock (online): tools/hermetic_build.sh renders the factory's mock config (tools/mock_config.py -- Fedora + Hummingbird + earlier stages + the published factory, same exclusions as the container lane, libicu-77 excluded unless buildroot_icu77) with the pinned build root (config/buildroot-image) as mock's ready bootstrap image, and runs mock --calculate-build-dependencies. The resulting buildroot_lock.json records every package in the root, %generate_buildrequires included, with EVR, arch, URL and header digest, and the bootstrap image by digest. 2. The package cache key is computed from the lock's NEVRAs, salted "hermetic" so hermetic and container entries can never be confused. 3. Build (offline): the lock is materialized into a local repository (mock-hermetic-repo for the network half; file:// stage and factory RPMs, which it cannot read, are copied) and mock --hermetic-build runs under unshare --net. A failed %check is retried once, as in the other lanes. 4. The lock, the mock config and its sha256 are uploaded per package as lock-s<N>-<pkg>, even when the build fails. The canary gains pass6: the canary set through the hermetic lane, and verify-hermetic requires a lock with a pinned bootstrap image for every package, a compile or cache hit for each, and vulkan-loader's lock taking vulkan-headers from the stage-0 artifact through file:///work/prior. Proven locally first (podman, privileged, as build-stage.yml runs docker): vulkan-headers locked in 270 s and built offline in 236 s; vulkan-loader locked with vulkan-headers from a file:// stage repository and built offline, producing *.hum1.bfin RPMs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
The first canary locked vulkan-headers on the runner (217 packages, 2 m 19 s, nested podman working) and then published the cache before the offline build step, which sat after it, had produced anything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
…orkspace back
Invoked as root, mock built as root, and flac's %check refused it ("are you
running as root?"), twice, in the wider-subset canary. It now runs as a
mockbuilder user in the mock group, as Hummingbird's does; flac builds
offline locally (lock 432 s, build 375 s). The first cache hit also showed
the container leaves work/ owned by that user, so podman cp from the host
could not write work/result; both steps now chown it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
ProofCanary 36185810389: green. On the runner, pass6 built the canary set through the hermetic lane: nested podman pulled the pinned bootstrap image inside the privileged build container, the lock took 2 m 19 s for vulkan-headers (217 packages), and the offline build ran under Wider subset (dispatch 36182167832, 24 packages against the published factory, before the unprivileged-user fix):
Locally after the fix, flac builds offline: lock 432 s, build 375 s, Merging this as the opt-in switch. Every production run still uses 🤖 Generated with Claude Code |
Proven on a subset first, as agreed: the canary set (three passes, with stage handoff through the lock) and a 24-package dispatch against the published factory -- Rust and Python with %generate_buildrequires (rust-just, python-pydantic-core), meson and cmake C, pango, libadwaita, pipewire, gdm, ibus, xdg-desktop-portal, and the %check-heavy git, fish, flac and libratbag. All 24 build on the hermetic lane after two fixes this subset found: - mock must not run as root (merged in #260): flac and git refused a root %check. - a new network namespace starts with loopback down, so git's HTTP tests (t0410, t0611 serve on 127.0.0.1) failed. The offline build now brings lo up inside its namespace; nothing outside is reachable. Now: - backend defaults to hermetic in build-stage.yml, rebuild-rpms.yml (workflow_call, dispatch, and every wave). - A package can stay on the container lane with "build_lane": "container" and a required "build_lane_reason" in config/upstream-sources.json (validated by tools/package_inventory.py); prepare hands the list to the waves and build-stage.yml chooses the lane per package. None is needed yet. - The canary's passes 1-5 build hermetically; verify-hermetic reads pass1's locks; pass6 keeps the container lane proven, and a dispatch can point it at any subset and lane (lane_packages, lane_backend). - The hermetic lane gets the canary's flaky-%check injection through a define only the retry passes: each attempt has a fresh mock root, so a marker file cannot carry state. The state label does not include the lane, so this changes no package's input digest: packages move to the hermetic lane as they change, and the weekly full rebuild moves the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
Proven on a subset first, as agreed: the canary set (three passes, with stage handoff through the lock) and a 24-package dispatch against the published factory -- Rust and Python with %generate_buildrequires (rust-just, python-pydantic-core), meson and cmake C, pango, libadwaita, pipewire, gdm, ibus, xdg-desktop-portal, and the %check-heavy git, fish, flac and libratbag. All 24 build on the hermetic lane after two fixes this subset found: - mock must not run as root (merged in #260): flac and git refused a root %check. - a new network namespace starts with loopback down, so git's HTTP tests (t0410, t0611 serve on 127.0.0.1) failed. The offline build now brings lo up inside its namespace; nothing outside is reachable. Now: - backend defaults to hermetic in build-stage.yml, rebuild-rpms.yml (workflow_call, dispatch, and every wave). - A package can stay on the container lane with "build_lane": "container" and a required "build_lane_reason" in config/upstream-sources.json (validated by tools/package_inventory.py); prepare hands the list to the waves and build-stage.yml chooses the lane per package. None is needed yet. - The canary's passes 1-5 build hermetically; verify-hermetic reads pass1's locks; pass6 keeps the container lane proven, and a dispatch can point it at any subset and lane (lane_packages, lane_backend). - The hermetic lane gets the canary's flaky-%check injection through a define only the retry passes: each attempt has a fresh mock root, so a marker file cannot carry state. The state label does not include the lane, so this changes no package's input digest: packages move to the hermetic lane as they change, and the weekly full rebuild moves the rest. Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Item 4, first step: the hermetic lane behind a switch (
backend: hermetic), proven on a subset by the canary. Making it the default comes in a follow-up PR, after a full run on this lane.What it does
tools/hermetic_build.sh locktools/mock_config.py: Fedora + Hummingbird + earlier stages + published factory, the container lane's exclusions, libicu-77 excluded unlessbuildroot_icu77), with the pinned build root (config/buildroot-image) as mock's ready bootstrap image. Thenmock --calculate-build-dependencies, which writesbuildroot_lock.json: every package in the root,%generate_buildrequiresincluded, with EVR, arch, URL and header digest, plus the bootstrap image by digest.hermetic, so hermetic and container entries never meettools/hermetic_build.sh buildmock-hermetic-repo;file://stage and factory RPMs, which that tool cannot read, are copied. Thenmock --hermetic-buildruns underunshare --net(loopback only), and a failed%checkis retried once.lock-s<N>-<pkg>mock.cfg+ its sha256, uploaded even when the build failsThis is the design review's
rpmspec --buildrequires → DNF solve → buildroot_lock.json → build offline, done by mock itself, the same way Hummingbird'sci/build_rpms.sh --hermeticdoes it (--calculate-build-dependencies,mock-hermetic-repo,--hermetic-build,--network=none).Conflict with the approved spec (noted, as asked)
docs/superpowers/specs/2026-09-05-full-packit-rpm-factory-design.mdsays Packit invokes Mock (packit build --srpm … in-mock --root <cfg>) from the pinnedquay.io/packit/packitimage. It also requires a materialized, hashed mock root and "network isolation after dependencies are resolved". Where these conflict, I followed the spec as far as it can be followed:mock.cfg.sha256, uploaded), network isolation after resolution, and the earlier-stage artifact handoff withexcludepkgs.packit build in-mockis plainmock --root cfg srpm. It has no way to run the lock phase or--hermetic-build, so this lane calls mock directly, with the same config Packit would pass.ghcr.io/projectbluefin/utah-buildroot@sha256:…, ci(buildroot): factory-owned build-root mirror, and a cache key that survives a publish #252/fix(buildroot): keep the pin in config/ and move it to the first mirror image #255), notquay.io/packit/packit. quay prunes that image's digests (repeated mistake 7), which is exactly why the mirror exists. mock is installed at run time, as the existing mock lane already does.This leaves #43 (whether Packit drives the build) open rather than deciding it.
Proof so far
Local, podman
--privileged(how build-stage.yml runs docker), same scripts:vulkan-headers-1.4.350.0-1.hum1.bfin.noarch.rpmfile://stage repovulkan-loader{,-devel}-1.4.350.0-1.hum1.bfin.x86_64.rpm; the lock recordsfile:///work/prior/…/vulkan-headers-…hum1.bfin.noarch.rpmThe canary gains
pass6, the canary set through this lane.verify-hermeticrequires, for every package, a lock that pins the bootstrap image and a compile or cache hit. It also requires vulkan-loader's lock to take vulkan-headers from the stage-0 artifact. Canary numbers to follow.Not in this PR
hermeticthe default. The hand-built container root papers over gaps a clean mock root will show: a system bus for libratbag's%check, staged sysusers for openssh's regress tests,TESTS_OUTPUT_DIRfor librsvg evidence. A full dispatch on this lane will enumerate them, and the switch comes after that.sccache(mozjs140 only) cannot reach its cache from a network-less build. The design review's answer is mock's ccache plugin plusactions/cache.🤖 Generated with Claude Code