feat: nb-bottles — own GHCR bottle registry (mirror + repackage tooling, weekly CI publisher) - #320
Open
justrach wants to merge 6 commits into
Open
feat: nb-bottles — own GHCR bottle registry (mirror + repackage tooling, weekly CI publisher)#320justrach wants to merge 6 commits into
justrach wants to merge 6 commits into
Conversation
scripts/bottles/nb_bottles.py: mirror (Tier-1, digest-preserving OCI cross-repo mounts of pinned homebrew_bottle records), repackage (Tier-2, upstream github_release binaries -> bottle tarballs), pin (any Homebrew formula -> registry record from the live API), publish/verify/record. manage.md documents operations. bottles.yml publishes from CI so packages are created by GITHUB_TOKEN and born public (locally-pushed packages land private with no API to flip them); push to this branch runs the full bootstrap, weekly cron keeps new pins mirrored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
repackage now normalizes upstream artifact paths (bare 'rg', 'usr/bin/podman') into <name>/<version>/bin/<basename> so bottle kegs link correctly, and the bootstrap repackages all 21 eligible github_release formulae via the new `tier2` list command instead of a hardcoded pair. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…solation OCI repository paths forbid '@' — node@22 must publish to .../node/22 the way Homebrew does (which also keeps the NANOBREW_BOTTLE_DOMAIN rewrite path-compatible). The upstream source repo is now derived from each pinned asset URL instead of string concatenation. A failing package no longer aborts the fleet-wide mirror; failures are collected and reported at the end with a nonzero exit. This is what killed bootstrap run 1 at 42/231. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rom the registry Every top-150 install-on-request formula now has a homebrew_bottle record (bash, zsh, qemu, colima, mysql, postgresql@14/16/18, node@24, nginx, tailscale, rustup, btop, eza, poetry, ...), pinned from the live API via `nb_bottles.py pin --add`. 396 records / 268 pinned bottles total; the CI mirror picks them up automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 3 stalled at ~181/231: GHCR acked cross-repo mounts with 201 while the blob never landed. Mounts are now verified (retry, then real pull+push fallback), packages are paced 200ms apart, and the workflow's mirror step no longer skips the pin/repackage steps on partial failure — the mirror is idempotent and converges on re-run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The remote-fetched registry (nb update-registry) and the embedded default must carry the same records; the 36 analytics pins + hexyl only landed in the embedded copy. Union fallback masked the gap, but stale-remote + fresh-cache would have hidden new pins from update-registry users. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ddb3602c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if blob_exists(repo, digest, dst_token): | ||
| how, size = "exists", None | ||
| else: | ||
| _, how = push_blob(repo, b"", dst_token, mount_from=src_repo, digest=digest) |
There was a problem hiding this comment.
When GHCR cannot satisfy the cross-repo mount and returns a 202 upload session, push_blob immediately PUTs the data argument with the requested digest; here (and in the same pin --mirror path) that data is b"", so the fallback upload attempts to store an empty blob under the Homebrew bottle digest and fails before the later pull+push recovery can run. This breaks mirroring for any package/platform whose mount is denied, throttled, or otherwise not completed, despite the intended pull+push fallback.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Gives nanobrew its own GHCR bottle registry at
ghcr.io/justrach/nb-bottles/<name>, fed by two tiers:homebrew_bottleregistry entry, using OCI cross-repo mounts (instant, no re-upload; digests stay byte-identical to Homebrew's pins).github_releasebinary upstreams into proper keg-layout bottles so they ride the same install path.What's in the branch
scripts/bottles/nb_bottles.py— management CLI withmirror/repackage/verify/recordverbs; per-package failure isolation, versioned-formula (@) token handling, GHCR phantom-mount + throttling resilience for fleet runs..github/workflows/bottles.yml— weekly cron +workflow_dispatchpublisher (both only fire from the default branch, which is why this PR exists). Idempotent: re-runs skip already-published packages and converge.manage.md— operator runbook for the registry.registry/upstream.jsonwith the embedded registry.Verification
Verified end-to-end against the live registry (286 packages published): anonymous pulls work, Tier-1 digests match Homebrew's pinned digests, and a real
nb installfrom the mirror viaNANOBREW_BOTTLE_DOMAINsucceeds.🤖 Generated with Claude Code