Skip to content

fix(ec2,codebuild): retry a throttled first image pull instead of failing the launch - #2519

Merged
vieiralucas merged 1 commit into
mainfrom
fix/ec2-codebuild-image-pull
Sep 13, 2026
Merged

vieiralucas merged 1 commit into
mainfrom
fix/ec2-codebuild-image-pull

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Sep 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Follow-up to #2516. That PR fixed the ECS and Lambda pull paths; EC2 instances and CodeBuild builds have a separate one with the same failure, and it is the cause of their e2e flakes.

Across the last 26 E2E runs, container-launching tests account for almost every retry. After #2516's ECS/Batch fix, the remaining image-pull ones are:

Test Retries
ec2_instance_runtime::run_instances_boots_real_container_with_user_data 12
ec2_network_isolation::instances_isolate_by_subnet_network 8
codebuild_real_execution::* (4 tests) 18

Cause

Both runtimes start their container with a bare docker run, whose implicit pull of a missing image gives up on the first 429 Too Many Requests. On a host with an empty image cache (a fresh CI runner), the tests that race the first pull of alpine:3 (Docker Hub) or public.ecr.aws/docker/library/ubuntu:22.04 fail: EC2 falls back to a metadata-only instance with a synthesized 10.0.0.x address, and CodeBuild fails the build in PROVISIONING. A retry passes because the image is cached by then.

Reproduced with a container CLI that models an empty cache behind a throttling registry (nothing cached until a pull succeeds, the first pull refused with a 429, run throttled while nothing is pulled):

  • before: both EC2 tests and 4 of 5 CodeBuild tests fail, exactly the ones that flake in CI
  • after: all 7 pass, with the throttled first pull retried

Fix

fakecloud_core::container_image::ensure_image, called before docker run in the EC2 Docker backend and in CodeBuild:

  • image already cached -> used as is, without contacting the registry (what docker run does today; these images stand in for an AMI or a curated build image, so they are not re-pulled per launch the way ECS images are)
  • missing -> pulled through pull_image, which retries transient failures (throttling, 5xx, network timeouts) with backoff and fails at once on a refusal

CodeBuild bounds the pull with the same 600s budget as the container start. The k8s EC2 backend is untouched.

Surfaces

  • Docs: website/content/docs/services/ec2.md and codebuild.md describe the pull behavior.
  • No API, SDK, conformance or count change.

Test plan

  • 3 new unit tests for ensure_image (cached image skips the registry, throttled first pull retried, refused pull fails); fakecloud-core container_image tests 17/17.
  • Cold-cache throttling repro above, before and after, with real Docker behind the fake CLI.
  • cargo clippy -p fakecloud-core -p fakecloud-ec2 -p fakecloud-codebuild --all-targets -D warnings and cargo fmt --check clean.

Summary by cubic

Fixes EC2 and CodeBuild launches failing when a missing image's first pull is throttled. Both runtimes ran a bare docker run, whose implicit pull gives up on the first 429 Too Many Requests; ensure_image now pulls missing images with retries and backoff, and uses cached images without contacting the registry.

  • EC2 previously fell back to metadata-only instances; CodeBuild failed at PROVISIONING.
  • CodeBuild bounds the pull with the same 600s timeout as the container start.
  • The k8s EC2 backend is untouched.
  • Docs for EC2 and CodeBuild describe the pull behavior.

Written for commit 7300768. Summary will update on new commits.

Review in cubic

…ling the launch

EC2 instances and CodeBuild builds started their containers with a bare
`docker run`, whose implicit pull of a missing image gives up on the first
`429 Too Many Requests`. On a host with an empty image cache -- a fresh CI
runner -- every launch racing the first pull of an image failed: EC2 fell
back to a metadata-only instance with a synthesized 10.0.0.x address, and
CodeBuild failed the build in PROVISIONING.

That is the cause of the ec2_instance_runtime, ec2_network_isolation and
codebuild_real_execution e2e flakes (18 CodeBuild and 20 EC2 retries across
the last 26 E2E runs). Reproduced with a container CLI that models an empty
cache behind a throttling registry: before, the 2 EC2 tests and 4 of 5
CodeBuild tests fail, exactly the ones that flake in CI; after, all pass.

New fakecloud_core::container_image::ensure_image keeps `docker run`'s
semantics -- a cached image is used without contacting the registry, so
these images, which stand in for an AMI or a curated build image, are not
re-pulled per launch -- and pulls a missing one through pull_image, which
retries transient failures with backoff and fails at once on a refusal.
@vieiralucas
vieiralucas merged commit 2137ed7 into main Sep 13, 2026
158 checks passed
@vieiralucas
vieiralucas deleted the fix/ec2-codebuild-image-pull branch September 13, 2026 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant