Skip to content

feat(sources): close lookaside feed gap with forge and anitya polling - #166

Closed
mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:feat/lookaside-release-feeds
Closed

mrbobbytables wants to merge 3 commits into
projectbluefin:mainfrom
mrbobbytables:feat/lookaside-release-feeds

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Resolves #134

Problem

Previously, only 63 of 340 source locks in config/upstream-sources.json had pollable upstream URLs (GNOME cache.json or direct GitHub/GitLab archive URLs). The remaining 277 locks were on Fedora's lookaside cache or bare tarball directory listings without an automated release feed, leaving an unpolled "lookaside gap".

Solution

  1. Multi-feed resolution in tools/upstream_bump.py:

    • Added support for git forges (GitHub releases and tags with GITHUB_TOKEN rate-limit mitigation; GitLab tags API across multiple hosts).
    • Added support for Anitya / release-monitoring.org project release API.
    • Added support for explicit feed annotations in config/upstream-sources.json (supporting forge URLs, GNOME modules, Anitya IDs, and documented pinned categories).
    • Fail-closed lookaside safety: apply() refuses to rewrite lookaside URLs without an explicit upstream template.
    • Added --audit flag to report inventory feed coverage.
  2. Categorized all 277 gap packages in config/upstream-sources.json:

    • 197 packages annotated with explicit upstream feeds (108 GitHub, 45 GitLab, 44 GNOME), bringing total pollable inventory from 63 to 260 packages (76.5%).
    • 80 packages categorized into a documented pinned subset (e.g. xorg directory listings, bare ftp/http directory listings, sourceforge, kernel directory listings, vendored Go source, recipe-only).
    • Reduced unclassified lookaside gaps from 277 to 0 (0.0%).
  3. CI Workflow and Tests:

    • Renamed and modernized .github/workflows/bump-upstream-sources.yml with GITHUB_TOKEN rate-limit auth and inventory auditing.
    • Expanded tests/test_upstream_bump.py with 29 new unit tests covering forge detection, tag parsing, API polling, entry rewrites, feed resolution, Anitya integration, lookaside safety, and inventory auditing.
  4. Skill Documentation:

    • Created docs/skills/upstream-source-feeds.md and indexed it in docs/SKILL.md.

All tests and contract checks pass (just check, just test).

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 3036b5b

…polling

Expand upstream source lock polling and inventory management to bridge the
lookaside gap across 340 source locks:

- Add multi-feed resolution to tools/upstream_bump.py supporting GitHub,
  GitLab, GNOME, Anitya (release-monitoring.org), and documented pinned markers.
- Add inventory audit capabilities (--audit) to report feed coverage and
  ensure 0 unclassified lookaside gaps.
- Annotate the 277 gap packages in config/upstream-sources.json with explicit
  feed fields, bringing pollable inventory to 260 packages (76.5%) and
  classifying 80 packages into documented pinned categories.
- Ensure lookaside tarball safety by refusing automated URL rewrites on lookaside
  paths without explicit upstream templates.
- Rename and update bump-upstream-sources.yml workflow with GITHUB_TOKEN support
  and inventory auditing.
- Document upstream feed management in docs/skills/upstream-source-feeds.md.

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 3036b5b)

1. High — first outdated lookaside package crashes the entire --apply run; the scheduled workflow is effectively guaranteed to fail.
apply() raises ValueError for any entry whose url contains the lookaside prefix (tools/upstream_bump.py:635-639), and main() calls apply() for each finals proposal with no try/except (tools/upstream_bump.py:760-763). All 197 newly feed-annotated packages keep their lookaside URLs (e.g. SDL3, config/upstream-sources.json:13-21), yet plan() happily emits update/final proposals for them, so the first one with a newer upstream release aborts the whole batch — no later bumps applied, workflow step fails, no PR opened. The 44 GNOME-feed lookaside entries make this immediate: adwaita-icon-theme at 51~beta (config/upstream-sources.json:59-60) gets a final proposal for 51.0 via the gnome branch of plan() (tools/upstream_bump.py:582-592), then hits the lookaside raise on the very first scheduled run. "Fail-closed" per package becomes fail-total for the run — either skip lookaside-URL entries from finals, or catch and report per-package apply errors and continue.

2. Medium — GitHub tags feed can silently miss the newest release. forge_versions fetches a single page (per_page=100, no pagination, tools/upstream_bump.py:340-343); the GitHub tags endpoint is not ordered by recency or semver, so for repos with >100 tags the newest stable may be absent from the page, silently producing "nothing newer" for exactly the repos most likely to release often.

3. Test adequacy. LookasideSafetyTests (tests/test_upstream_bump.py) asserts apply() raises for lookaside URLs, but nothing tests the batch path in main() where that raise aborts every remaining bump — the exact interaction finding #1 lives in.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…apply and paginate forge tags

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @kubestellar-hive in commit f6a5f64:

  • Resilient batch apply: main() now catches ValueError during --apply on lookaside-locked packages, logging a skip message to stderr and continuing so one lookaside package does not abort the entire batch.
  • Forge tag pagination: forge_versions() now paginates up to 5 pages (500 tags) for GitHub and GitLab feeds instead of stopping at the first 100 entries.
  • Test coverage: Added unit tests covering GitHub tag pagination and multi-package batch apply handling of lookaside entries.

🐝 Hive Agent: contributor | SHA: unknown

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Blocker: Runtime exception in tools/upstream_bump.py during --apply. The newly feed-annotated packages keep lookaside URLs, but apply() raises ValueError whenever url matches lookaside prefix, aborting the automated bump workflow.
  • In-progress check: Commit f6a5f64 is still in progress across CI matrix jobs.
  • Outstanding: Author needs to address the lookaside URL migration/handling in apply(), followed by CI verification and independent review (+2).

…o upstream in apply()

Allow packages tracked via GNOME release feeds that currently point to Fedora lookaside
to migrate cleanly to download.gnome.org during apply().

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed blocker feedback from @castrojo in commits f6a5f64 and 105af05:

  • Lookaside migration in apply(): Added support for packages with GNOME feeds currently referencing Fedora lookaside URLs to cleanly migrate to download.gnome.org when applying bumps.
  • Resilient batch apply: main() catches lookaside ValueError on unmigrated lookaside packages during --apply, logging a skip notice and continuing so automated batch runs complete.
  • Tag pagination: Paginate up to 500 tags across GitHub and GitLab release feeds.
  • Tests: All 347 unit tests and validation checks pass cleanly.

🐝 Hive Agent: contributor | SHA: 105af05

🐝 Hive Agent: contributor | SHA: unknown

@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

@hanthor hanthor closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

277 of 340 source locks have no release feed to poll — the Fedora lookaside gap

3 participants