Skip to content

refactor(upstream-sources): pin 97 packages to upstream, reject Fedora-primary - #125

Closed
kylerankin wants to merge 1 commit into
projectbluefin:mainfrom
kylerankin:refactor/upstream-sources
Closed

kylerankin wants to merge 1 commit into
projectbluefin:mainfrom
kylerankin:refactor/upstream-sources

Conversation

@kylerankin

@kylerankin kylerankin commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #42.

Supply-chain: 97 of the source locks that previously fetched their payload from Fedora now download verbatim from upstream, and tools/validate.py rejects any Fedora host in the primary url position.

Count reconciliation: 115 source locks were Fedora-primary at projectbluefin/main (the issue's stated 114 is one stale). This commit re-pins 97 of them to verbatim upstream downloads; the remaining 18 cannot be re-pinned to a verbatim upstream artifact and are documented as exceptions below. Nothing was left unpinned that is not an intentional exception.

What changed

  • config/upstream-sources.json: for each of 97 packages, set the primary url to the verified upstream download and record Fedora's lookaside as fallback_urls. Every upstream url was checked against the locked sha512, so the verified tarball is byte-for-byte the locked payload. All other fields (version, filename, sha512, stage) are preserved; the pipeline's fetch_with_fallbacks tries upstream first and falls back to Fedora if upstream is down.
  • tools/validate.py: reject any entry whose primary url points at src.fedoraproject.org / fedoraproject.org, unless it is a documented exception. This keeps the Fedora-primary lookaside valid only as a fallback, never as the primary source.

Exceptions (18, documented in tools/validate.py) cannot be re-pinned to a verbatim upstream artifact:

  • built from vendored/Go source: containerd, runc, mozc, alsa-firmware, alsa-tools
  • no tarball, recipe file only: kde-filesystem, kf5
  • blocked on projectbluefin/fsdk-containers Go image: tailscale (lock stays untouched per test_source_inventory)
  • Fedora-patched Python sdist (upstream sdist sha no longer matches the lock): python-psutil, python-argcomplete, python-dbus-next, python-pydantic-core
  • upstream URL could not be resolved to a verbatim artifact this pass: fwupd, graphene, lcms2, livesys-scripts, samba, xmlrpc-c

— hive: backend=pi model=lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K

…a-primary

Replace Fedora lookaside URLs with verbatim upstream downloads for 97 of the
115 source locks that previously fetched their payload from Fedora, recording
Fedora's lookaside as fallback_urls so the pipeline still resolves if upstream
is down. Each upstream url was validated against the locked sha512 so the
verified tarball is byte-for-byte the locked payload.

Reject any Fedora host in the primary url position in tools/validate.py. The
18 exceptions that cannot be re-pinned to a verbatim upstream artifact are
documented in the same file: built-from-vendored/Go source (containerd, runc,
mozc, alsa-firmware, alsa-tools), no tarball (kde-filesystem, kf5), the
FSDK-blocked tailscale lock, and Fedora-patched Python sdist sources
(python-psutil, python-argcomplete, python-dbus-next, python-pydantic-core);
the remaining entries (fwupd, graphene, lcms2, livesys-scripts, samba,
xmlrpc-c) could not be resolved to a verbatim artifact during this pass.

Signed-off-by: kylerankin <kylerankin@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified the mechanical bulk rather than trusting it, and it holds up well. Recording what I actually checked, then the one thing that blocks merge.

Structure. Parsing both revisions of config/upstream-sources.json: 193 entries before and after, none added or removed, exactly 97 changed, and the only differing fields across all 97 are url and fallback_urls — Counter({'fallback_urls': 97, 'url': 97}). No entry changed sha512, version or filename, and every one of the 97 kept a fedoraproject fallback. The hash lock stays the trust anchor, which is what makes a change of this size reviewable at all.

No silent reclassification. branch lives in packages/*/.hummingbird-upstream.json, and git diff <merge-base>...HEAD --name-only is exactly two files, neither of them a provenance lock. So nothing here moves a package into the branch: "upstream" class that tools/validate.py:19 excuses from carrying commit/tree. That was my main worry going in and it is clean.

Resolution: 97 of 97 across 23 hosts. 81 returned 200 to curl -I -L, 3 more 206 to a range GET. The remaining 13 are all github.com/*/archive/*, which my egress proxy 403s indiscriminately — a known-good control URL 403s the same way — so I checked those by git ls-remote --tags instead and all 13 tags exist (abseil-cpp 20260526.0, fzf v0.74.3, hwdata v0.410, icu release-78.3, libayatana-indicator 0.9.4, libical v3.0.20, libnfs libnfs-7.0.0, python-evdev v1.9.3, pydantic v2.13.5, sdl2-compat release-2.32.70, Vulkan-Headers and Vulkan-Loader vulkan-sdk-1.4.350.0, yyjson 0.12.0).

Hashes: 22 of 23 sampled entries match byte-for-byte. I sampled one package per host and downloaded each in full, comparing sha512 to the lock — including the large ones: mesa (68 MB), firefox (810 MB), mozjs140 (644 MB), webkitgtk, iso-codes, plus cairo, adwaita-icon-theme, libdrm, python-dasbus, pulseaudio, gsl, wireguard-tools, ModemManager, gstreamer1, hicolor-icon-theme, libdbusmenu, libthai, liboping, colord, i2c-tools, libXdamage, libXcursor. Every one matched. My sample supports the claim in the description.

One honest gap: I could not hash-verify the 21 github.com/*/archive/* entries — both github.com and codeload.github.com are blocked for me — so for those I confirmed only that the tag exists. Those are the entries where the tarball is generated on demand rather than published as a release asset, which is the class where byte-stability has historically been the concern. If you verified them, saying so explicitly in the description would close the loop; if there is a reachable release asset for any of them, preferring it over /archive/ would be worth doing.

Related and worth knowing: this fails closed. source_pipeline.py computes sha512 after fetch_with_fallbacks returns, and fetch_with_fallbacks only catches transport errors — so a drifted upstream tarball hard-fails the build, it does not quietly fall back to the lookaside. Correct behaviour, but it means the Fedora fallback covers unreachability, not drift, and a GitHub archive that ever changes bytes takes the build down rather than degrading.

The new rule does its job. Repointing cairo's primary back at its lookaside gives fedora-primary source url not allowed (exceptions are documented in tools/validate.py): cairo, exit 1.


Blocking: this cannot merge, and the conflict is not textual.

main has moved 17 commits since the branch point and refactored tools/validate.py into check_provenance() and main(root: Path = Path(".")) -> int behind a __main__ guard, with tests/test_validate.py calling main(root) against a temporary tree. This PR appends module-level code that executes at import and hardcodes Path("config/upstream-sources.json") relative to CWD. Those two shapes do not reconcile by picking a side.

I resolved it by hand — moved the block into check_fedora_primary(root: Path) using root / "config/upstream-sources.json" and called it from main() — and the merged tree gives 3 failed, 178 passed, 27 subtests passed with validated 191 source RPMs, byte-identical to main's own baseline. So the substance rebases fine; it just needs doing deliberately. I also confirmed the JSON merges cleanly and that gcc, which main deleted, does not come back (192 entries post-merge, firefox retained).

Worth flagging while you rebase: the green CI on this branch is not main's test suite. At the merge-base the suite is 72 tests; on main it is 181. Please re-read CI after the rebase rather than trusting the current run.

Two small things.

import urllib.parse is added at line 5 and never used — I assume the intent was to compare urlparse(url).netloc, which would be the better check. As written it is any(host in url for host in FEDORA_HOSTS), a substring match against the whole URL, so https://mirror.example.com/pub/fedoraproject.org/x.tar.gz and https://example.org/x.tar.gz?ref=fedoraproject.org both trip it. Only false positives, so not urgent, but the import is already there.

The exception list is keyed on package name with no expiry, and six of the eighteen entries say "could not be resolved to a verbatim artifact during the re-pin pass" — a TODO encoded as a permanent allowlist. Consider splitting the structural exemptions (vendored Go, no tarball, Fedora-patched sdist) from the "not done yet" six, so the second group reads as work remaining rather than as policy.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

This is the direction the factory needs — a lock that points at Fedora's lookaside is a lock on Fedora's choice, and #134 tracks exactly the 277 remaining ones. Two things block merging it as it stands:

  1. It conflicts with main on config/upstream-sources.json and tools/validate.py: Build the whole Utah contract: 343 recipes, Firefox as a Flatpak, four stage failures fixed #70 moved 340 locks and fix(ci): skip against the published image, drag dependents, stop building debuginfo #133 (open) doesn't touch them, so a rebase is mechanical but real.
  2. The 97 rewritten locks need to go through the source pipeline once — tools/source_pipeline.py re-downloads every rewritten tarball and checks its SHA-512 — and then a factory run, since a tarball from the project's own release can differ from Fedora's re-roll (different autotools output, different top-level directory) and the spec's %setup -n has to agree. The PR's own rebuild run is that test; today it can't start because of the conflict.

Please rebase and I'll drive the run.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing rather than holding it open for a rebase: the maintainers want the queue to hold only what can merge. The direction is right and #134 tracks it (277 locks still on Fedora's lookaside, with the per-package feed discovery that this PR did by hand for 97 of them). When it comes back rebased, it goes through source_pipeline.py and a factory run as described above.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

supply-chain: 114 of 193 source locks still fetch payloads from Fedora

2 participants