Mirror public images to internal registry#313
Conversation
- Add mirror_images.py for lock file generation, lint checking, and mirroring - Add mirror_images.lock.yaml with resolved digests for all images - Use MIRROR_REGISTRY ARG for flexible image sourcing in Dockerfiles - Add GitLab CI jobs for linting and mirroring images - Auto-cancel previous pipelines on new push - Add AGENTS.md with codebase overview for AI agents
aa7c235 to
07d3131
Compare
7defe6c to
00532b0
Compare
- Migrate all Docker image references to use registry.ddbuild.io mirrors - Factorize and simplify registry usage across CI and test configs - Fix OpenResty detection for mirrored base images - Ensure examples are usable outside Datadog internal infrastructure - Add build-step retries in run.py for transient failures - Increase npm fetch-retries in Dockerfiles - Remove CircleCI configuration
There was a problem hiding this comment.
Pull request overview
This PR migrates all public Docker image references (Docker Hub, ghcr.io, registry.k8s.io) in CI configs, Dockerfiles, and compose files to use an internal registry.ddbuild.io/ci/nginx-datadog/mirror/ registry. It also introduces tooling to manage the mirror lifecycle and adds CI automation for linting and mirroring.
Changes:
- Adds
bin/mirror_images.pyCLI tool withlint,add,lock,relock, andmirrorsubcommands;mirror_images.yamlsource config; andmirror_images.lock.yamllock file with digests for 101 images. - Replaces all public image references across CI configs (
.gitlab/build-and-test-*.yml,.gitlab/common.yml), Dockerfiles (build_env/,test/,example/,injection/), and compose files with mirror-prefixed equivalents. - Adds GitLab CI
mirror-imagesstage with lint and mirror jobs, auto-cancellation on new commits, and anAGENTS.mdcodebase overview.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
bin/mirror_images.py |
New CLI tool for managing mirrored image lifecycle |
mirror_images.yaml |
Source-of-truth list of all images to mirror |
mirror_images.lock.yaml |
Auto-generated lock file with resolved digests (missing alpine:3.20.3) |
.gitlab/mirror.yml |
New CI stage with lint and mirror jobs |
.gitlab-ci.yml |
Adds mirror-images stage, auto-cancel, and interruptible default |
.gitlab/build-and-test-fast.yml |
Replaces all public image refs with mirror refs |
.gitlab/build-and-test-all.yml |
Replaces all public image refs with mirror refs |
.gitlab/common.yml |
Replaces ingress-nginx and openresty image refs |
build_env/Dockerfile |
Introduces MIRROR_REGISTRY ARG; uses mirrored uv and alpine |
Makefile |
Passes MIRROR_REGISTRY build-arg conditionally |
.github/workflows/system-tests.yml |
Sets MIRROR_REGISTRY="" for GHA to use public images |
test/Dockerfile |
Replaces Python and uv image refs with mirrors |
test/services/client/Dockerfile |
Replaces alpine/curl-http3 with mirror ref |
injection/ingress-nginx/docker-compose.yaml |
Replaces testagent and nginx refs with mirrors |
example/tracing/docker-compose.yml |
Replaces datadog/agent with mirror ref |
example/openresty/docker-compose.yml |
Replaces datadog/agent with mirror ref |
example/tracing/services/client/Dockerfile |
Replaces alpine:3.19 with mirror ref |
example/openresty/services/client/Dockerfile |
Replaces alpine:3.19 with mirror ref |
example/ingress-nginx/test-application.yaml |
Replaces ealen/echo-server with mirror ref |
example/ingress-nginx/helm/values.yaml |
Updates image path but leaves conflicting registry: docker.io |
AGENTS.md |
New file with codebase overview for AI agents |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Wouldn't it be simpler to use our internal mirror that automatically proxies and caches the images? ( |
Having explicit mirrors also helps us to be a little bit more resilent in case image is pulled form upstream repository. than a pull-through cache Also its better to use the official registry.ddbuild.io rather than one of the rand ecs repositories |
…x-datadog into update_images # Conflicts: # .gitlab/common.yml
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #313 +/- ##
=======================================
Coverage 68.85% 68.85%
=======================================
Files 56 56
Lines 7471 7471
Branches 1058 1058
=======================================
Hits 5144 5144
Misses 1820 1820
Partials 507 507 🚀 New features to boost your workflow:
|
1310c43 to
2262c2b
Compare
- Add new images: nginx 1.29.6, openresty 1.29.2.1, ingress-nginx v1.13.8/v1.14.4/v1.15.0, alpine 3.20.3 - Extend mirror lint to scan GitLab CI matrix variables for undeclared images - Move datadog/agent entry to end of mirror_images.yaml - Update dd-trace-cpp submodule
Replace bare `except Exception: continue` with specific `(OSError, yaml.YAMLError)` catches and log warnings to stderr, preventing silent failures in the lint command.
- Lint now skips example/ directory (examples use public image references) - Remove example-only images from mirror_images.yaml - Use cwd as PROJECT_DIR so the script works from other repos - Fall back to script-relative mirror_images.yaml if not found in cwd - Add comment explaining GitLab CI auto_cancel workflow - Condense AGENTS.md
- Move print override after imports with explanatory comment - Wrap YAML path discovery in _find_mirror_yaml() to avoid leaked locals - Rename ambiguous variables: lock→progress_lock, _check→_resolve_tag/_check_if_mirrored, d→tag_digest, ref→digest_ref - Extract _current_lock_entries(), _extract_matrix_combos(), _expand_matrix_images() helpers - Consolidate _is_external() conditions and check_digest_exists() tool branches - Use collections.deque for BFS traversal, list comprehension for _version_sort_key - Move results dict mutation inside progress_lock for thread safety
- Revert dd-trace-cpp to origin/master commit (f8c3913) - Mark mirror_images.lock.yaml as linguist-generated
Pull all public Docker images from registry.ddbuild.io mirrors. Adds
bin/mirror_images.pyto manage the list, lock digests, lint, and sync. Removes CircleCI.example/keeps public refs (works outside DD infra)MIRROR_REGISTRY=""in GHA for public pulls