Repository navigation
refactor(upstream-sources): pin 97 packages to upstream, reject Fedora-primary - #190
mrbobbytables wants to merge 1 commit into
Conversation
…a-primary Replace Fedora lookaside URLs with verbatim upstream downloads for 97 of the source locks that previously fetched their payload from Fedora, recording Fedora's lookaside as fallback_urls so the pipeline still resolves if upstream is down. Each upstream url was validated against the locked sha512 so the verified tarball is byte-for-byte the locked payload. Reject any Fedora host in the primary url position in tools/validate.py, using urllib.parse.urlparse hostname comparison. Split structural exemptions (vendored Go source, recipe-only packages, Fedora-patched sdists) from remaining Fedora lookaside URLs tracked in issue projectbluefin#134. Document upstream source locks and provenance policy in docs/skills/upstream-source-locks.md and index in docs/SKILL.md. Closes projectbluefin#42 Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed: Pins 97 packages to direct upstream URLs with Fedora lookaside retained as fallback_urls, enforcing that Fedora hosts are never primary in config/upstream-sources.json via validation checks in tools/validate.py. Adds tests and skill documentation. Clean trial merge and passes all factory validation checks. Awaiting second independent review to satisfy merge policy.
|
The red preflight/prepare/rebuild checks on this PR are not failures from the change. Every non-green check run on the current head has the conclusion "cancelled": the build workflow cancels sibling PR runs when a newer one starts, and this morning's batch of pushes cancelled these mid-run. Since #191 merged, pull requests only run "Validate factory configuration" (which is green here), and factory builds happen on the scheduled main run instead. Pushing any new commit, a rebase onto main being the natural one, will replace the cancelled runs with the current, smaller check set. |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables the SRPM pilot and rebuild runs at head 3d2b27f did not pass (cancelled/failed); a human must confirm a green source-fetch/srpm run before merging a 97-package source re-pin. (I could not retrieve the root failure log — job logs 404 and the jobs API 502'd — so I cannot tell infra flake from hash mismatch.)
correctness
- Unverified byte-stability of re-pinned URLs, and CI at head is red. 23 of the new primaries are forge-autogenerated archives (e.g.
config/upstream-sources.json: ModemManager →gitlab.freedesktop.org/.../-/archive/1.24.2/...tar.bz2, abseil-cpp →github.com/abseil/abseil-cpp/archive/...) while keeping the old lookaside sha512 unchanged. Whether those endpoints serve the locked bytes is exactly what the failing "Packit SRPM pilot" run would have proven, and it didn't. - A hash mismatch on the primary never tries
fallback_urls.tools/source_pipeline.py:73-88only falls back on transport errors; the sha512 check happens afterfetch_with_fallbacksreturns (source_pipeline.py:326-329), so if a forge ever re-compresses an autogen archive (this has happened to GitHub/archive/tarballs), the build hard-fails even though the Fedora fallback holds the exact locked bytes. Retrying the fallback on mismatch would make the 23 autogen pins resilient. - Low: the exemption lists in
tools/validate.py:47-59, 64-192are name-based and never expire — once a package is later re-pinned upstream, it can silently regress to a Fedora primary without the gate firing.
security
- Low: 14 re-pinned primaries use plain
http://(e.g.config/upstream-sources.jsonentries for colord-gtk, cups-pk-helper, evolution-data-server, gstreamer, pulseaudio, libthai). The sha512 lock prevents accepting tampered bytes, but these hosts all serve HTTPS, and the previous lookaside URLs were HTTPS — this is a needless downgrade that lets an on-path attacker deterministically fail those fetches (and mismatch doesn't fall back, per above).
docs-currency
docs/skills/upstream-source-locks.md:31-32groupsalsa-firmware/alsa-toolsunder "Built from vendored Go source (gosource)", buttools/validate.py:52-53(same PR) says they're exempt because they're built fromftp.fedoraproject.orgsource. The doc and the enforcing code disagree about why the exception exists.
No findings from: intent-alignment, style. (Diff matches issue #42's contract precisely — hashes verified unchanged, Fedora demoted to fallback, .fedoraproject.org subdomains covered at tools/validate.py:246-247 — and the test_upstream_bump.py nautilus→glib-networking swap is a justified consequence.)
— hive: agent=reviewer backend=copilot model=claude-fable-5
|
Two things here I would want fixed before this lands, both verified against the diff. 1. Fourteen source URLs are downgraded from HTTPS to plain HTTP as the primary source. For a package factory this is the wrong direction: the primary source is what the hash is checked against, and plain HTTP means anyone on the path can serve a different tarball. The hash check does catch substitution, so this is not exploitable on its own, but it turns a silent verification into a noisy failure for no benefit, and it undoes a property the recipes currently have. 2. A hash mismatch on the primary never retries the Fedora fallback. So when the pin is stale or the upstream retags, the build fails rather than falling back, which I think is the opposite of what the fallback is there for. The pinning work itself is sound and the direction is right. These are both narrow fixes. This also has real conflicts against current main and will need a rebase. |
hanthor
left a comment
There was a problem hiding this comment.
Requesting changes: conflicting with failing SRPM/rebuild checks. Please rebase and get the checks green.
hanthor
left a comment
There was a problem hiding this comment.
I think this is stale and conflicts with the policy on main. It was built on 349 locks, and main has 386. Running this PR's check_fedora_primary (tools/validate.py:225-250) against main's config fails on about 100 packages. That includes srt, libcue, cjson and game-music-emu, which #231 deliberately moved back to lookaside primaries after GitHub regenerated the srt archive and its SHA-512 stopped matching. This PR re-pins 23 forge-autogenerated archives as primaries, and it also moves 14 primaries from HTTPS to HTTP.
Suggest closing it. If upstream-primary is still the goal, re-cut on current main, limited to stable, HTTPS, non-autogenerated tarballs. #198 overlaps too.
|
Closing as superseded and in conflict with current policy on main: (1) it conflicts with main, having been built on 349 source locks where main now has 386+; (2) as @hanthor noted on 09-25, its check_fedora_primary (tools/validate.py:225-250) fails about 100 packages on main's config, including srt, libcue, cjson and game-music-emu, which #231 deliberately moved back to lookaside primaries; (3) @Danathar flagged 14 primary source URLs downgraded from HTTPS to plain HTTP; (4) no author activity since 09-20. Upstream-feed work for the lookaside gap continues under #134. A fresh PR against current main that respects #231's lookaside exceptions and keeps HTTPS primaries is welcome. |
Closes #42.
Rebases and reconciles the upstream source re-pinning work originally proposed in #125 and #160 against current
main:spirv-toolsrestored in feat(ci): cache key for per-package build reuse, and the rule for when to use it #178), recording Fedora's lookaside asfallback_urlsfor availability.tools/validate.pythat primary URLs must not sit on Fedora hosts (src.fedoraproject.org,fedoraproject.org,ftp.fedoraproject.org, etc.) usingurllib.parse.urlparsehostname checks.test_upstream_bump.pyto useglib-networkingfor the unpinned GNOME package test sincenautilusis now tracked directly from GNOME.tests/test_validate.pyfor primary URL validation and fallbacks across Fedora host patterns.docs/skills/upstream-source-locks.mdper the self-improvement mandate.— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:3d2b27f