Skip to content

Mirror container images with proxy mirror - #438

Draft
pinguinfuss wants to merge 3 commits into
git-pkgs:mainfrom
pinguinfuss:issue-411-mirror-images
Draft

pinguinfuss wants to merge 3 commits into
git-pkgs:mainfrom
pinguinfuss:issue-411-mirror-images

Conversation

@pinguinfuss

Copy link
Copy Markdown
Contributor

Closes #411.

proxy mirror now pre-fetches pkg:docker and pkg:oci images: index, platform manifests (with their attestations) and blobs, stored under the keys a pull reads. A later pull is served from cache even when the upstream is down.

  • Manifest cache keys no longer include the client's Accept header, so Docker, containerd and Podman share one entry. The old keys are still written and read as fallback.
  • proxy mirror uses the same upstream clients as serve. Before, Docker Hub answered every request with 401.
  • --platform (or platforms in the API) limits multi-arch images. repository_url only resolves to configured registries.

Tested offline against Docker Hub and ghcr.io with containerd and Docker.

Open question: Docker 29 also requests sha256-<digest> tags on every pull. Offline the proxy answers those with 502 and the pull fails, on main too. Returning 404 there fixes it. Should that go in here or in its own PR?

… header: a digest names one document, and a tag is shared by all clients that accept an index type, so Docker, containerd and Podman read the same entry and a pre-fetched image is found offline. Clients without index support keep their per-Accept entry, and the old keys are still written and read as fallback.
…o it follows OCI Bearer challenges, uses configured upstream credentials and the SSRF guard, and artifact downloads have no overall time limit. Before this the CLI used a plain client and Docker Hub answered every request with 401.
…kg:docker and pkg:oci PURLs: the mirror walks index, platform manifests (with their attestations) and blobs through the container handler, so a later pull is served from cache even when the upstream is down. A --platform filter (platforms in the API) limits multi-arch images, repository_url only resolves to configured registries, and names, tags and digests are validated before anything is fetched.
@pinguinfuss
pinguinfuss marked this pull request as draft October 11, 2026 22:39
@pinguinfuss

Copy link
Copy Markdown
Contributor Author

@andrew For #412 I'd change MirrorImage from #438 so it always revalidates tags with a HEAD and writes the tag only after all child manifests and blobs are cached. A failed or interrupted mirror then never leaves a tag pointing at an incomplete image, and the scheduled refresh can just reuse the normal mirror run. Would you rather have that change in #438, or in a separate PR for #412 on top of it?

@andrew andrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two reproduced failures need fixing before merge:

  • internal/handler/container_mirror.go:216: Mirroring updates the cached tag before fetching its layers. Starting with a fully mirrored tag, then moving that tag to an image with an unavailable layer, makes the mirror fail but leaves the tag pointing at the incomplete image. Publish the tag only after all selected manifests and blobs are cached, preserving the previous tag on failure. Please include that part of the proposed change in this PR and add a regression test.
  • internal/handler/container_manifest.go:194-198: An incompatible shared entry prevents lookup of compatible per-Accept entries. Caching a Docker manifest list, then an OCI index for the same tag, then taking the upstream offline makes a Docker-only Accept request return 503 despite its cached representation still being available. Continue to the fallback keys when the shared entry is incompatible, and add a regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

proxy mirror can't pre-fetch container images

2 participants