Skip to content

feat(build): hermetic mock lane -- lock the build root, build offline from the lock - #260

Merged
hanthor merged 4 commits into
mainfrom
feat/hermetic-builds
Sep 25, 2026
Merged

hanthor merged 4 commits into
mainfrom
feat/hermetic-builds

Conversation

@hanthor

@hanthor hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member

Item 4, first step: the hermetic lane behind a switch (backend: hermetic), proven on a subset by the canary. Making it the default comes in a follow-up PR, after a full run on this lane.

What it does

phase where what
lock (online) tools/hermetic_build.sh lock renders the factory's mock config (tools/mock_config.py: Fedora + Hummingbird + earlier stages + published factory, the container lane's exclusions, libicu-77 excluded unless buildroot_icu77), with the pinned build root (config/buildroot-image) as mock's ready bootstrap image. Then mock --calculate-build-dependencies, which writes buildroot_lock.json: every package in the root, %generate_buildrequires included, with EVR, arch, URL and header digest, plus the bootstrap image by digest.
key build-stage.yml cache key from the lock's NEVRAs, salted hermetic, so hermetic and container entries never meet
build (offline) tools/hermetic_build.sh build materializes the lock into a local repository. The network half goes through mock-hermetic-repo; file:// stage and factory RPMs, which that tool cannot read, are copied. Then mock --hermetic-build runs under unshare --net (loopback only), and a failed %check is retried once.
record artifact lock-s<N>-<pkg> lock + mock.cfg + its sha256, uploaded even when the build fails

This is the design review's rpmspec --buildrequires → DNF solve → buildroot_lock.json → build offline, done by mock itself, the same way Hummingbird's ci/build_rpms.sh --hermetic does it (--calculate-build-dependencies, mock-hermetic-repo, --hermetic-build, --network=none).

Conflict with the approved spec (noted, as asked)

docs/superpowers/specs/2026-09-05-full-packit-rpm-factory-design.md says Packit invokes Mock (packit build --srpm … in-mock --root <cfg>) from the pinned quay.io/packit/packit image. It also requires a materialized, hashed mock root and "network isolation after dependencies are resolved". Where these conflict, I followed the spec as far as it can be followed:

  • Followed: real Mock, one isolated root per package, the root config materialized and hashed (mock.cfg.sha256, uploaded), network isolation after resolution, and the earlier-stage artifact handoff with excludepkgs.
  • Could not follow as written: packit build in-mock is plain mock --root cfg srpm. It has no way to run the lock phase or --hermetic-build, so this lane calls mock directly, with the same config Packit would pass.
  • Container: it runs in the factory's pinned build-root mirror (ghcr.io/projectbluefin/utah-buildroot@sha256:…, ci(buildroot): factory-owned build-root mirror, and a cache key that survives a publish #252/fix(buildroot): keep the pin in config/ and move it to the first mirror image #255), not quay.io/packit/packit. quay prunes that image's digests (repeated mistake 7), which is exactly why the mirror exists. mock is installed at run time, as the existing mock lane already does.

This leaves #43 (whether Packit drives the build) open rather than deciding it.

Proof so far

Local, podman --privileged (how build-stage.yml runs docker), same scripts:

package lock offline build result
vulkan-headers 270 s, 217 locked packages 236 s vulkan-headers-1.4.350.0-1.hum1.bfin.noarch.rpm
vulkan-loader, with vulkan-headers in a file:// stage repo 238 s 213 s vulkan-loader{,-devel}-1.4.350.0-1.hum1.bfin.x86_64.rpm; the lock records file:///work/prior/…/vulkan-headers-…hum1.bfin.noarch.rpm

The canary gains pass6, the canary set through this lane. verify-hermetic requires, for every package, a lock that pins the bootstrap image and a compile or cache hit. It also requires vulkan-loader's lock to take vulkan-headers from the stage-0 artifact. Canary numbers to follow.

Not in this PR

  • Making hermetic the default. The hand-built container root papers over gaps a clean mock root will show: a system bus for libratbag's %check, staged sysusers for openssh's regress tests, TESTS_OUTPUT_DIR for librsvg evidence. A full dispatch on this lane will enumerate them, and the switch comes after that.
  • sccache (mozjs140 only) cannot reach its cache from a network-less build. The design review's answer is mock's ccache plugin plus actions/cache.

🤖 Generated with Claude Code

hanthor and others added 4 commits September 25, 2026 19:17
… from the lock

Behind backend: hermetic (dispatch input, workflow_call input), the way
Hummingbird's ci/build_rpms.sh --hermetic builds:

1. Lock (online): tools/hermetic_build.sh renders the factory's mock config
   (tools/mock_config.py -- Fedora + Hummingbird + earlier stages + the
   published factory, same exclusions as the container lane, libicu-77
   excluded unless buildroot_icu77) with the pinned build root
   (config/buildroot-image) as mock's ready bootstrap image, and runs
   mock --calculate-build-dependencies. The resulting buildroot_lock.json
   records every package in the root, %generate_buildrequires included,
   with EVR, arch, URL and header digest, and the bootstrap image by digest.
2. The package cache key is computed from the lock's NEVRAs, salted
   "hermetic" so hermetic and container entries can never be confused.
3. Build (offline): the lock is materialized into a local repository
   (mock-hermetic-repo for the network half; file:// stage and factory RPMs,
   which it cannot read, are copied) and mock --hermetic-build runs under
   unshare --net. A failed %check is retried once, as in the other lanes.
4. The lock, the mock config and its sha256 are uploaded per package as
   lock-s<N>-<pkg>, even when the build fails.

The canary gains pass6: the canary set through the hermetic lane, and
verify-hermetic requires a lock with a pinned bootstrap image for every
package, a compile or cache hit for each, and vulkan-loader's lock taking
vulkan-headers from the stage-0 artifact through file:///work/prior.

Proven locally first (podman, privileged, as build-stage.yml runs docker):
vulkan-headers locked in 270 s and built offline in 236 s; vulkan-loader
locked with vulkan-headers from a file:// stage repository and built
offline, producing *.hum1.bfin RPMs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
The first canary locked vulkan-headers on the runner (217 packages, 2 m 19 s,
nested podman working) and then published the cache before the offline
build step, which sat after it, had produced anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
…orkspace back

Invoked as root, mock built as root, and flac's %check refused it ("are you
running as root?"), twice, in the wider-subset canary. It now runs as a
mockbuilder user in the mock group, as Hummingbird's does; flac builds
offline locally (lock 432 s, build 375 s). The first cache hit also showed
the container leaves work/ owned by that user, so podman cp from the host
could not write work/result; both steps now chown it back.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
@hanthor

hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Proof

Canary 36185810389: green. On the runner, pass6 built the canary set through the hermetic lane: nested podman pulled the pinned bootstrap image inside the privileged build container, the lock took 2 m 19 s for vulkan-headers (217 packages), and the offline build ran under unshare --net. vulkan-loader's lock took vulkan-headers from the stage-0 artifact (file:///work/prior/…). verify-hermetic ✓. The previous canary also restored all three from the hermetic cache namespace, so keys from the lock are stable across runs.

Wider subset (dispatch 36182167832, 24 packages against the published factory, before the unprivileged-user fix):

result packages
✓ built offline pango, rust-just, python-pydantic-core (Rust + Python, %generate_buildrequires), python-typing-inspection, python-psutil, libadwaita, sassc, mesa-libGLU, distrobox, wl-clipboard, gnome-shell-extension-gsconnect, ibus, libei, libinput, libcupsfilters, gdm, libsndfile, gnome-tweaks
✗ ran as root (fixed in 530114f) flac (are you running as root?), git (Cannot run httpd tests as root)
✗ to check after the fix fish (cd/fish_config/path checks)
✗ known gap libratbag: %check needs a system bus, which the container lane starts by hand and a mock root does not have

Locally after the fix, flac builds offline: lock 432 s, build 375 s, %check passes as mockbuilder.

Merging this as the opt-in switch. Every production run still uses container. Making hermetic the default comes next, after the subset is re-run with the fix and the libratbag-class gaps have a per-package answer.

🤖 Generated with Claude Code

@hanthor
hanthor marked this pull request as ready for review September 25, 2026 20:58
@hanthor
hanthor merged commit 8c14d98 into main Sep 25, 2026
129 of 130 checks passed
hanthor added a commit that referenced this pull request Sep 25, 2026
Proven on a subset first, as agreed: the canary set (three passes, with
stage handoff through the lock) and a 24-package dispatch against the
published factory -- Rust and Python with %generate_buildrequires
(rust-just, python-pydantic-core), meson and cmake C, pango, libadwaita,
pipewire, gdm, ibus, xdg-desktop-portal, and the %check-heavy git, fish,
flac and libratbag. All 24 build on the hermetic lane after two fixes this
subset found:

- mock must not run as root (merged in #260): flac and git refused a root
  %check.
- a new network namespace starts with loopback down, so git's HTTP tests
  (t0410, t0611 serve on 127.0.0.1) failed. The offline build now brings
  lo up inside its namespace; nothing outside is reachable.

Now:
- backend defaults to hermetic in build-stage.yml, rebuild-rpms.yml
  (workflow_call, dispatch, and every wave).
- A package can stay on the container lane with "build_lane": "container"
  and a required "build_lane_reason" in config/upstream-sources.json
  (validated by tools/package_inventory.py); prepare hands the list to the
  waves and build-stage.yml chooses the lane per package. None is needed
  yet.
- The canary's passes 1-5 build hermetically; verify-hermetic reads pass1's
  locks; pass6 keeps the container lane proven, and a dispatch can point it
  at any subset and lane (lane_packages, lane_backend).
- The hermetic lane gets the canary's flaky-%check injection through a
  define only the retry passes: each attempt has a fresh mock root, so a
  marker file cannot carry state.

The state label does not include the lane, so this changes no package's
input digest: packages move to the hermetic lane as they change, and the
weekly full rebuild moves the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
hanthor added a commit that referenced this pull request Sep 25, 2026
Proven on a subset first, as agreed: the canary set (three passes, with
stage handoff through the lock) and a 24-package dispatch against the
published factory -- Rust and Python with %generate_buildrequires
(rust-just, python-pydantic-core), meson and cmake C, pango, libadwaita,
pipewire, gdm, ibus, xdg-desktop-portal, and the %check-heavy git, fish,
flac and libratbag. All 24 build on the hermetic lane after two fixes this
subset found:

- mock must not run as root (merged in #260): flac and git refused a root
  %check.
- a new network namespace starts with loopback down, so git's HTTP tests
  (t0410, t0611 serve on 127.0.0.1) failed. The offline build now brings
  lo up inside its namespace; nothing outside is reachable.

Now:
- backend defaults to hermetic in build-stage.yml, rebuild-rpms.yml
  (workflow_call, dispatch, and every wave).
- A package can stay on the container lane with "build_lane": "container"
  and a required "build_lane_reason" in config/upstream-sources.json
  (validated by tools/package_inventory.py); prepare hands the list to the
  waves and build-stage.yml chooses the lane per package. None is needed
  yet.
- The canary's passes 1-5 build hermetically; verify-hermetic reads pass1's
  locks; pass6 keeps the container lane proven, and a dispatch can point it
  at any subset and lane (lane_packages, lane_backend).
- The hermetic lane gets the canary's flaky-%check injection through a
  define only the retry passes: each attempt has a fresh mock root, so a
  marker file cannot carry state.

The state label does not include the lane, so this changes no package's
input digest: packages move to the hermetic lane as they change, and the
weekly full rebuild moves the rest.


Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant