Repository navigation
Conversation
Promotion copied each digest as it read it and asserted the total afterwards:
while IFS='=' read -r name digest; do
skopeo copy ... "docker://${REGISTRY}/${name}:testing"
done < <(find /tmp/digests -name '*.txt' -exec cat {} +)
test "$count" -eq 4
The order is wrong. A run that produced three of the four flavors advances
:testing for those three and then fails, so the stream is left carrying a set
nobody promised and no step undoes it. The same is true of a digest naming an
image no flavor asks for: it is copied first and counted second.
The literal four is the second fault. config/flavors.json is the single source
for which images exist -- the build matrix, the promote matrix and the release
matrix all read it, and `just check` already forbids naming a flavor directly
in these workflows. The count was the one place that escaped, so narrowing the
flavor set would have left promotion failing on a number rather than doing
less work.
Now the run is checked against `scripts/flavors.py images` before anything is
copied: every promised image present, exactly once, no image that is not
promised, and every value an actual digest. Only then does the promotion loop
run, over the verified list. `just check` gains a guard so the count cannot be
hardcoded here again.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
|
CI status on this head: the That is the same failure The three names are absent from the published factory repository, so unblocking them is a Generated by Claude Code |
…s a Flatpak Two things, both about a published image containing what it promises. Firefox. The package factory used to build the RPM and stopped: a full PGO and LTO Gecko compile was the longest build it had, it repeatedly hit the runner memory ceiling, and nothing else links it. There is no packages/firefox recipe any more, so `firefox` and `firefox-langpacks` can never appear in Utah's repositories and `just check-repos` was right to fail. What ships instead was already here: org.mozilla.firefox is in the desktop contract and 99-flatpaks.sh installs Bluefin's own policy into its systemconfig extension, and verify-desktop-contract asserts that on every build. Both names move to [unavailable] with that reasoning, so the parity arithmetic stays honest rather than passing silently. Booting. The Containerfile asserts what an image contains -- package contract, desktop contract, extensions, bootc lint -- and none of it can say the image starts. Three of this project's claims live entirely on the other side of a boot: that a desktop appears, that the gaming flavors run the OGC kernel and not the stock one, and that the NVIDIA module matches the kernel that actually booted rather than the one it was compiled against. The README calls those unproven, and they stayed unproven because nothing could prove them. scripts/verify-boot.sh checks them from inside a booted system and prints one marker line. Containerfile.e2e derives a throwaway layer that runs it on first boot and powers off; `just e2e-boot` installs that to a disk and boots it under QEMU with the console on a serial port. A hang or a panic produces no marker, so it cannot be mistaken for a pass. The layer is never published -- it carries a unit that powers the machine off shortly after gdm starts, which is exactly the sort of thing that must not reach anyone. post-testing-e2e now resolves the image set once, boots every image in it, and promotes only that list. Promotion previously copied each digest as it read it and counted afterwards; it now cannot advance :testing for an image that has not started. Hosted runners have no KVM so this is TCG and slow, which is the price of the only check that answers the question. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
|
The red Both names come from They are also not parity debt, so they do not belong in
Both are listed in That is fixed in projectbluefin/utah-packages#33 ( So the chain is: utah-packages publishes → Generated by Claude Code |
Utah reaches GDM and cannot bring up a wifi adapter. Established by checking
the whole chain rather than inferring it:
Hummingbird bootc-os base image 262 packages: kernel, kernel-core,
kernel-modules, kernel-modules-core, and
no linux-firmware
Hummingbird RPM repository 3,511 packages, and the only match for
firm/iwl/wifi/wireless is
NetworkManager-wifi
packages/bluefin.toml alsa-firmware, alsa-tools-firmware
packages/utah.toml nothing
Bluefin never names linux-firmware because it inherits it from its Fedora base
image, so it is absent from the parity contract this repository copies
byte-for-byte. Swapping the base to Hummingbird's bootc-os dropped it in
silence. Nothing Requires linux-firmware, so no dependency resolution pulls it
and the factory's consumer transaction is satisfied without it; the boot test
added in this branch runs under QEMU on virtio, which needs no firmware blobs.
Every gate passed on an image with no firmware at all. Wireless is the
headline, but GPUs and Bluetooth are in the same position.
Fedora splits one linux-firmware source into thirty subpackages by device
family. All thirty are listed here, which is what parity with Fedora means:
iwlwifi dvm/mvm/mld, atheros, brcmfmac, mt7xxx, realtek, mediatek,
nxpwireless, tiwilink and libertas for wireless; amd, intel and nvidia GPU;
cirrus and intel audio; amd-ucode; and the datacenter NIC families. Together
they are roughly 740 MiB. If image size becomes the binding constraint, the
datacenter NICs and the Qualcomm SoC firmware are the obvious trim for an
x86_64 desktop and would save about 350 MiB -- recorded in the section as a
reduction to decide on rather than a default to drift into.
install-packages.py needed the section wired in. contract() composes the
install set from named sections, so a new [firmware] section would have been
read by nobody and silently installed nothing -- the failure mode this whole
change exists to fix, repeated one level up. The install set goes from 71 to
101.
These are not published yet: utah-packages is building linux-firmware for the
first time, so check-repo-availability.py will report them missing until that
lands. That is the same position every other package in this branch is in.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
|
The Until 814422e it failed with two names: 814422e adds a The Why the firmware was missing at all: Bluefin inherits Generated by Claude Code |
Two corrections, one of them to my own last commit. FIRST, the set. I listed all thirty linux-firmware subpackages on the reasoning that parity with Fedora means taking what Fedora splits out. Bluefin is the actual parity target, and files/pkg-churn.json in projectbluefin/bluefin records what that image carries: twenty-one of the thirty, plus alsa-sof-firmware. It leaves behind the datacenter NIC families -- liquidio, mlxsw_spectrum, mrvlprestera, netronome, qed -- and the Qualcomm SoC ones. Matching Bluefin instead of Fedora drops about 350 MiB, most of it qcom-firmware at 148 MiB and mlxsw_spectrum-firmware at 102 MiB, neither of which an x86_64 desktop loads. Bluefin keeps qcom-wwan-firmware for modems and mt7xxx-firmware for MediaTek wireless while dropping the broader mediatek-firmware; that asymmetry is theirs and is copied rather than tidied. Dakota was read too. It curates at file level -- 6,810 files under components/linux-firmware.bst -- and its set is wider than Bluefin's, carrying qcom, netronome, mellanox and qed. Bluefin is the closer comparison and the contract this repository already tracks, so it is the one matched. Dakota does settle one thing: alsa-sof-firmware gets its own component there, components/sof-firmware.bst, which is a second vote for treating Sound Open Firmware as its own concern rather than a detail of the blob set. Without it a modern Intel laptop has no audio at all, and nothing in Utah's stack had it. SECOND, and worse: the check that should have caught this could not see it. check-repo-availability.py composes its list from sections named one at a time, and [firmware] was not among them -- so the twenty-two packages were installed and never preflighted. I predicted the contract check would report 32 missing names after the last commit; it reported 2, and that gap is the bug, not a happy surprise. The same enumeration was missing [services] entirely, which has been installed and unverified all along, and verify-rpm-contract.py's off-image --check path had the same shape. That is three copies of the failure this whole change exists to fix: a package set that is live in one place and invisible in another, failing quietly. So scripts/check-overlay-sections.py now asserts that every section of utah.toml is read by every consumer that must read it, with the one exemption written down rather than assumed -- verify-rpm-contract.py does not check [build], because the toolchain is installed, used and removed inside the build. It runs in `just check`, so the next section added to utah.toml fails there instead of on hardware. The install set goes to 93 and the contract verifier now reports firmware as its own count: 60 Bluefin, 10 GNOME, 2 services, 22 firmware. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
|
Correcting my previous comment: I predicted the
10d99b4 fixes both that and the set itself. The set. I had listed all thirty Dakota was read too. It curates at file level — 6,810 files under The guard. So the expected Generated by Claude Code |
|
25, not the 24 I said. The extra one is
All 25 are the same shape — built by the factory, not yet published:
Nothing here needs fixing on this side. They resolve together when the factory's publish gate passes, which is what Generated by Claude Code |
The mirror is gone on the factory side -- linux-firmware put two files over GitHub Pages' 100 MB per-file limit and the site over its 1 GB total, and a mirror that quietly sheds packages while this script treats it as authoritative is worse than no mirror at all. So this reads the same OCI image Utah's Containerfile already consumes, which is a better source anyway: the preflight and the build now ask the same thing rather than a copy of it. ghcr hands out a pull token for public images without credentials, so this stays a dependency-free urllib script. Layers are tried smallest first and streamed, stopping at the primary index. The factory now copies repository/repodata into its own layer ahead of the full tree, so that is a few megabytes; an image built as a single layer still works, it just reads until it reaches repodata. Streaming rather than reading the blob whole is load-bearing -- buffering the current 524 MiB single layer raised IncompleteRead. Verified against the live registry: 421 binary packages, the same count the Pages mirror reported, and the same 25 contract names outstanding. The image build path never touched Pages. packages/utah-packages.repo is already file:///etc/utah-packages, filled from the digest-pinned package image; this check was the only reader left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Approved by @castrojo for Hive auto-merge on green CI.
hanthor
left a comment
There was a problem hiding this comment.
mergeable_state: dirty — this needs a rebase before anything else
The branch is based on 3d1761e3; main is now 636b48e3. GitHub reports the PR as conflicted, so it cannot merge in its current state regardless of review outcome. It carries the lgtm label, which makes the stale state easy to miss.
The three faults it identifies are all real and the analysis is the most careful in the queue:
1. Copy-before-count. Confirmed on main today — .github/workflows/post-testing-e2e.yml runs skopeo copy in the loop and test "$count" -eq 4 after it, so a run producing three of four flavors moves :testing for those three and then fails, with nothing undoing the partial promotion. Verifying the full digest set against scripts/flavors.py images before any copy is the right restructuring, and it is strictly more complete than #64, which de-literalizes the count but keeps the ordering. Worth deciding explicitly which PR owns this file — #64 also edits it, and #44 adds the flavors.py count query both of them want.
2. Nothing had booted the image. Correct, and the most valuable part. A boot job between verification and promotion, with promote-to-testing gated on needs: [resolve, boot], is the missing link between "published" and "works". The per-flavor assertions in verify-boot.sh — systemctl is-system-running, gdm.service active, IMAGE_ID/VARIANT_ID, OGC kernel with sched_ext and binderfs, nvidia.ko present with nouveau blacklisted — are specific and positive, which is the property the other QEMU harnesses in this queue (#41, #59, #66, #67) need and do not all clearly have. Asserting on UTAH_BOOT_OK printed from inside the VM is the right shape.
3. Firefox recorded as present. Also correct, and main has since absorbed this — [unavailable] on 636b48e is now evolution-ews-core, firefox, fish, grub2-efi-x64-cdboot, pipewire-libs-extra, zsh. So that part of the diff is likely redundant after rebase, which should shrink the conflict.
On the "known red check"
The contract failure on grub2-efi-x64-cdboot and pipewire-libs-extra was indeed not from this PR — both are in [unavailable] on current main, so a rebase should clear it.
But the current blocker is different and worse: main@636b48e fails Build Utah at Containerfile:81 (install: cannot create regular file '/usr/local/libexec/utah-install-packages', run 34654001732). The new boot job installs the built image with bootc install to-disk — there is no successfully built image to install right now. #56 needs to land first, with its ${pair##:*} typo corrected, before the boot gate can run at all.
Coordination after rebase
Four PRs touch post-testing-e2e.yml or scripts/flavors.py: this one, #44 (adds flavors.py count and image), #64 (uses count in the promotion check), #40 (tests flavors.py). #44 and #64 already conflict with each other on Justfile:79. This PR's guard additions to just check land in the same region.
Suggested order: #56 (unblock) → #40 (tests, applies clean today) → #44 → #64 rebased to drop its duplicate count → this PR rebased, keeping the boot gate and the copy-before-count restructuring.
The boot gate is the single most valuable thing in the open queue after #56. Worth getting it rebased and moving.
Generated by Claude Code
|
Closing as superseded: Generated by Claude Code |
Utah could advance
:testingwhile carrying less than it promises. Three faults, in widening order.1. Promotion copied before it counted
A run that produced three of the four flavors moved
:testingfor those three, then failed on the count. The stream was left carrying a set nobody promised, and nothing undid it. A digest naming an image no flavor asks for was likewise copied before it was counted.The count was also a literal.
config/flavors.jsonis the single source for which images exist — build matrix, promote matrix, release matrix all read it, andjust checkalready forbids naming a flavor directly anywhere in.github/workflows/. The count was the one place that escaped it, so narrowing the flavor set would have left promotion failing on a number rather than doing less work.The run is now verified against
scripts/flavors.py imagesbefore anything is copied: every promised image present, each exactly once with a consistent value, no image that is not promised, and every value an actualsha256:digest. The promotion loop runs over that verified list.just checkgains a guard so the count cannot be hardcoded here again.2. Nothing had ever booted the image
The set was verified as published, never as working.
:testingcould advance to four images that do not reach a login screen.bootis a new job between verification and promotion, one runner per flavor:Containerfile.e2elayers a verifier onto the built image and is never published.just e2e-bootrunsbootc install to-disk, installs the removable-media EFI fallback, and boots the result under QEMU with a serial console.scripts/verify-boot.shruns inside the booted VM:systemctl is-system-running,gdm.serviceactive,IMAGE_ID/VARIANT_IDareutah, and per flavor the OGC kernel withsched_extand binderfs, ornvidia.kopresent for the booted kernel with nouveau blacklisted. It printsUTAH_BOOT_OKand powers off; the recipe greps for that line.promote-to-testingnowneeds: [resolve, boot], so a flavor that does not boot does not promote. TCG emulation, so no KVM is required on a free runner.3. Firefox was recorded as present when it is not an RPM
Bluefin's manifest asks for the
firefoxandfirefox-langpacksRPMs. The package factory used to build them and stopped: a full PGO and LTO Gecko compile was its longest build and it repeatedly hit the runner memory ceiling, for a leaf application no other package links. There is nopackages/firefoxrecipe any more, so the name can never appear in Utah's repositories.What ships instead is the Flatpak, which was already here:
org.mozilla.firefoxis incontracts/bluefin-desktop.toml, and99-flatpaks.shinstalls Bluefin's own policy into theorg.mozilla.firefox.systemconfigextension.verify-desktop-contract.pyasserts it on every build, which is a stronger check than the RPM name ever got.Both names move to
[unavailable]inpackages/utah.tomlwith that reasoning recorded, so the parity arithmetic stays honest: Bluefin asks for an RPM Utah does not install, and that is now written down rather than passing silently. It is the only entry there that is a deliberate substitution rather than parity debt, and it says so.Validation
yaml.safe_loadon the workflow,bash -non the new scripts,scripts/check_workflow_outputs.py, the flavor-literal guard and all four newjust checkguards pass. The resolve step was exercised locally against a full, a short, an extra-image and a malformed-digest set (rc 0/1/1/1, each with the right message), and the promotion payload rendered from a verified list.install-packages.py --checkreports 71 Bluefin parity packages validated and 6 documented as unavailable.just checkstill fails locally only on the GNOME extension submodules, which need network access this environment does not have; CI checks them out recursively.Known red check
contractfails ongrub2-efi-x64-cdbootandpipewire-libs-extra. Not from this PR — see the comment below.🤖 Generated with Claude Code
https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB