Skip to content

ci: improved CI image hashing, multi-arch manifest, S3 snapshot uploads#36

Merged
pawelchcki merged 1 commit intomainfrom
add_aws_automations
Feb 26, 2026
Merged

ci: improved CI image hashing, multi-arch manifest, S3 snapshot uploads#36
pawelchcki merged 1 commit intomainfrom
add_aws_automations

Conversation

@pawelchcki
Copy link
Contributor

@pawelchcki pawelchcki commented Feb 2, 2026

Summary

  • Hash all CI image inputs (Dockerfile, build_env, setup scripts) instead of just the Dockerfile to correctly detect when rebuilds are needed
  • Build per-arch CI images natively, then combine into a multi-arch manifest with docker buildx imagetools create
  • Nydus conversion step that properly detects already-nydusified images by inspecting platform manifest layer media types (allow_failure)
  • Add integration test stage (test-rum) for RUM validation
  • Upload build artifacts (zipped mod_datadog.so) to S3 with --acl public-read:
    • httpd/branch-snapshots/pipeline-<id>/ for feature branches
    • httpd/snapshots/pipeline-<id>/ + httpd/snapshots/latest/ for main
  • Verify uploaded artifacts are publicly accessible via curl
  • Use upstream docker:27.3.1 and aws-cli:2.33.12 images directly from registry.ddbuild.io/images/
  • Remove allow_failure from build-ci-image
  • Rename DOCKERFILE_HASHCI_IMAGE_HASH

Test plan

  • Pipeline runs successfully on add_aws_automations branch
  • CI image build skipped when no hashed files changed
  • Nydus conversion correctly skips already-nydusified images
  • Artifacts uploaded to S3 with public-read ACL
  • Public download URLs printed and verified in after_script
  • Verify main branch pipeline uploads to httpd/snapshots/ and httpd/snapshots/latest/

@pawelchcki pawelchcki requested a review from a team as a code owner February 2, 2026 14:53
@pawelchcki pawelchcki requested review from dmehala and removed request for a team February 2, 2026 14:53
@codecov-commenter
Copy link

codecov-commenter commented Feb 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.12%. Comparing base (ee2228d) to head (fa2c2d7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #36   +/-   ##
=======================================
  Coverage   63.12%   63.12%           
=======================================
  Files           7        7           
  Lines         377      377           
  Branches       49       49           
=======================================
  Hits          238      238           
  Misses        101      101           
  Partials       38       38           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@xlamorlette-datadog xlamorlette-datadog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

The list-s3-artifacts job needs to be fixed:

$ aws s3 ls s3://rum-auto-instrumentation/httpd/ --recursive
Unable to locate credentials. You can configure credentials by running "aws configure".

@pawelchcki
Copy link
Contributor Author

/remove

@gh-worker-devflow-routing-ef8351
Copy link

gh-worker-devflow-routing-ef8351 bot commented Feb 5, 2026

View all feedbacks in Devflow UI.

2026-02-05 13:15:21 UTC ℹ️ Start processing command /remove


2026-02-05 13:15:24 UTC ℹ️ Devflow: /remove

@pawelchcki pawelchcki changed the title ci(aws): add job to list S3 bucket artifacts ci: improved CI image hashing, multi-arch manifest, S3 snapshot uploads Feb 26, 2026
@pawelchcki pawelchcki force-pushed the add_aws_automations branch 9 times, most recently from c4ddf47 to 1a9451a Compare February 26, 2026 14:10
- Improved CI image hashing (includes Dockerfile, build_env, setup script)
- Multi-arch manifest creation for CI images
- Nydus conversion with proper already-nydusified detection
- S3 snapshot uploads for branch and main pipelines
- Integration test stage (test-rum)
- Use upstream docker and aws-cli images directly
@pawelchcki pawelchcki merged commit 43b48ff into main Feb 26, 2026
10 checks passed
@pawelchcki pawelchcki deleted the add_aws_automations branch February 26, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants