Serve app assets from S3#2192
Draft
theseanything wants to merge 4 commits into
Draft
Conversation
Rolling ECS deploys leave a window where old and new tasks serve traffic together. Requests for fingerprinted assets from a new release can be routed to an old task, which cannot serve them and returns 404. Add an S3 bucket to each environment to hold assets from every release, along with an origin access control so CloudFront can read from it. The bucket name is exported so the deploy pipelines can sync assets into it before deploying to ECS.
The deployer role needs to create the per-environment assets bucket and CloudFront origin access control, and the deploy pipelines use the same role to sync assets into the bucket.
Copy the built assets out of the container image and upload them to the environment assets bucket before the new task definition is deployed to ECS. This makes the assets for a release available in S3 before any task serves pages referencing them. Assets from previous releases are left in the bucket so tasks which have not yet been replaced keep working during a deployment.
When enabled, CloudFront serves /assets/* from the assets bucket instead of the applications. The bucket holds assets from every release, so requests can no longer fail when a deployment leaves tasks from two releases serving traffic at once. The option is off by default and should only be enabled for an environment once its deploy pipelines have populated the bucket for all three applications. Serving from S3 cannot fall back to the ALB via an origin group: CloudFront must not forward the viewer Host header to an S3 origin, and without it the ALB listener rules cannot route the request.
There was a problem hiding this comment.
Pull request overview
This PR changes the deployment and CloudFront setup so that /assets/* can be served from an S3 “assets bucket” (rather than from the Rails containers behind the ALB), preventing transient 404s and negatively cached misses during ECS rolling deployments. It introduces an opt-in toggle (serve_assets_from_s3) so environments can keep current behaviour until their assets bucket has been populated.
Changes:
- Add a per-environment assets S3 bucket plus CloudFront Origin Access Control (OAC), and optionally route
/assets/*to that S3 origin. - Add a new CodeBuild project and pipeline actions to extract assets from the container image and sync them to S3 during deployments.
- Extend deployer IAM permissions for CloudFront OAC management and access to the assets bucket.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| infra/modules/secure-bucket/outputs.tf | Exposes the S3 bucket regional domain name for use as a CloudFront S3 origin. |
| infra/modules/environment/variables.tf | Adds serve_assets_from_s3 environment-level toggle. |
| infra/modules/environment/outputs.tf | Exposes assets_bucket_name from the environment module. |
| infra/modules/environment/cloudfront.tf | Wires serve_assets_from_s3 into the CloudFront module instantiation. |
| infra/modules/deployer-access/policy.tf | Grants deployer role CloudFront OAC permissions and assets bucket access. |
| infra/modules/cloudfront/variables.tf | Adds serve_assets_from_s3 toggle to the CloudFront module. |
| infra/modules/cloudfront/s3.tf | Creates the secure assets bucket, its CloudFront-read policy, and the OAC resource. |
| infra/modules/cloudfront/outputs.tf | Outputs assets_bucket_name for consumption by deployment pipelines. |
| infra/modules/cloudfront/cloudfront.tf | Adds S3 origin and conditionally routes /assets/* to S3 instead of ALB. |
| infra/deployments/forms/pipelines/sync-assets.tf | Adds a shared CodeBuild project to sync assets from an image into the assets bucket. |
| infra/deployments/forms/pipelines/deploy-forms-runner-container.tf | Adds a pipeline action to sync assets before ECS deploy. |
| infra/deployments/forms/pipelines/deploy-forms-product-page-container.tf | Adds a pipeline action to sync assets before ECS deploy. |
| infra/deployments/forms/pipelines/deploy-forms-admin-container.tf | Adds a pipeline action to sync assets before ECS deploy. |
| infra/deployments/forms/pipelines/buildspecs/sync-assets/sync-assets.yml | Implements asset extraction from the image and aws s3 sync upload. |
| infra/deployments/forms/inputs.tf | Adds serve_assets_from_s3 to the environment settings input object. |
| infra/deployments/forms/environment/outputs.tf | Exposes assets_bucket_name from the environment deployment stack. |
| infra/deployments/forms/environment/main.tf | Passes serve_assets_from_s3 into the environment module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+78
to
+80
| output "assets_bucket_name" { | ||
| value = module.cloudfront[0].assets_bucket_name | ||
| } |
Comment on lines
+21
to
+41
|
|
||
| data "aws_iam_policy_document" "assets_bucket_cloudfront_read" { | ||
| statement { | ||
| sid = "AllowCloudFrontToReadAssets" | ||
| principals { | ||
| type = "Service" | ||
| identifiers = ["cloudfront.amazonaws.com"] | ||
| } | ||
| actions = [ | ||
| "s3:GetObject", | ||
| ] | ||
| resources = [ | ||
| "arn:aws:s3:::${local.assets_bucket_name}/*" | ||
| ] | ||
| condition { | ||
| test = "StringEquals" | ||
| variable = "AWS:SourceArn" | ||
| values = [aws_cloudfront_distribution.main.arn] | ||
| } | ||
| } | ||
| } |
Comment on lines
+17
to
+30
| - docker pull "${IMAGE_URI}" | ||
| - CONTAINER_ID=$(docker create "${IMAGE_URI}") | ||
| - docker cp "${CONTAINER_ID}:/app/public/assets" ./assets | ||
| - docker rm "${CONTAINER_ID}" | ||
|
|
||
| # Upload the assets so they are available before the new tasks | ||
| # serve any pages which reference them. Assets from previous | ||
| # releases are deliberately left in place so tasks which have not | ||
| # yet been replaced keep working during the deployment. | ||
| - | | ||
| aws s3 sync ./assets "s3://${ASSETS_BUCKET}/assets/" \ | ||
| --exclude ".vite/*" \ | ||
| --cache-control "public, max-age=300" \ | ||
| --no-progress |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this pull request solve?
During an ECS rolling deployment, old and new tasks serve traffic together for a few minutes. Each container image only holds the content-hashed assets for its own release, and CloudFront serves
/assets/*from the ALB — so on a cache miss a request for a new asset can hit an old task and 404 (which CloudFront then negatively caches for ~10s). Users could see unstyled pages or missing JS during deployment.The fix is to stop serving assets from the Rails containers altogether. Every deployment now uploads the release's assets to an S3 bucket before the new tasks start, and CloudFront serves
/assets/*from that bucket instead of theapps. Because the bucket keeps the assets from every release, any page a user is looking at — old or new — always references assets that exist, no matter which tasks are currently running. Serving from S3 is behind a setting that is
off by default, so nothing changes for an environment until its bucket has been populated.
This is also helpful as we look at custom branding. If we want to support user uploading assets (e.g. logo), this means this can be done without application deployments.
This is the same pattern used by GOV.UK.
Things to consider when reviewing
Hostheader to an S3 origin (it breaks OAC request signing), and without it a failed-over request wouldn't match any ALB listener rule.Reminders
If you've made changes to the deployer role (files in
modules/deployer-access):make <environment> forms/account applyon the relevant environments (dev,stagingand/orprod)apply-forms-terraform-<environment>pipelines have run successfullyThis PR does change the deployer role (S3 access to the assets buckets, CloudFront origin access control management), so
forms/accountmust be applied before the environment pipelines run.