fix(ec2,codebuild): retry a throttled first image pull instead of failing the launch - #2519
Merged
Merged
Conversation
…ling the launch EC2 instances and CodeBuild builds started their containers with a bare `docker run`, whose implicit pull of a missing image gives up on the first `429 Too Many Requests`. On a host with an empty image cache -- a fresh CI runner -- every launch racing the first pull of an image failed: EC2 fell back to a metadata-only instance with a synthesized 10.0.0.x address, and CodeBuild failed the build in PROVISIONING. That is the cause of the ec2_instance_runtime, ec2_network_isolation and codebuild_real_execution e2e flakes (18 CodeBuild and 20 EC2 retries across the last 26 E2E runs). Reproduced with a container CLI that models an empty cache behind a throttling registry: before, the 2 EC2 tests and 4 of 5 CodeBuild tests fail, exactly the ones that flake in CI; after, all pass. New fakecloud_core::container_image::ensure_image keeps `docker run`'s semantics -- a cached image is used without contacting the registry, so these images, which stand in for an AMI or a curated build image, are not re-pulled per launch -- and pulls a missing one through pull_image, which retries transient failures with backoff and fails at once on a refusal.
This was referenced Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2516. That PR fixed the ECS and Lambda pull paths; EC2 instances and CodeBuild builds have a separate one with the same failure, and it is the cause of their e2e flakes.
Across the last 26 E2E runs, container-launching tests account for almost every retry. After #2516's ECS/Batch fix, the remaining image-pull ones are:
ec2_instance_runtime::run_instances_boots_real_container_with_user_dataec2_network_isolation::instances_isolate_by_subnet_networkcodebuild_real_execution::*(4 tests)Cause
Both runtimes start their container with a bare
docker run, whose implicit pull of a missing image gives up on the first429 Too Many Requests. On a host with an empty image cache (a fresh CI runner), the tests that race the first pull ofalpine:3(Docker Hub) orpublic.ecr.aws/docker/library/ubuntu:22.04fail: EC2 falls back to a metadata-only instance with a synthesized10.0.0.xaddress, and CodeBuild fails the build inPROVISIONING. A retry passes because the image is cached by then.Reproduced with a container CLI that models an empty cache behind a throttling registry (nothing cached until a pull succeeds, the first pull refused with a 429,
runthrottled while nothing is pulled):Fix
fakecloud_core::container_image::ensure_image, called beforedocker runin the EC2 Docker backend and in CodeBuild:docker rundoes today; these images stand in for an AMI or a curated build image, so they are not re-pulled per launch the way ECS images are)pull_image, which retries transient failures (throttling, 5xx, network timeouts) with backoff and fails at once on a refusalCodeBuild bounds the pull with the same 600s budget as the container start. The k8s EC2 backend is untouched.
Surfaces
website/content/docs/services/ec2.mdandcodebuild.mddescribe the pull behavior.Test plan
ensure_image(cached image skips the registry, throttled first pull retried, refused pull fails);fakecloud-corecontainer_image tests 17/17.cargo clippy -p fakecloud-core -p fakecloud-ec2 -p fakecloud-codebuild --all-targets -D warningsandcargo fmt --checkclean.Summary by cubic
Fixes EC2 and CodeBuild launches failing when a missing image's first pull is throttled. Both runtimes ran a bare
docker run, whose implicit pull gives up on the first429 Too Many Requests;ensure_imagenow pulls missing images with retries and backoff, and uses cached images without contacting the registry.PROVISIONING.Written for commit 7300768. Summary will update on new commits.