Integrate one-pipeline for standardized OCI packaging#321
Draft
pawelchcki wants to merge 100 commits intomasterfrom
Draft
Integrate one-pipeline for standardized OCI packaging#321pawelchcki wants to merge 100 commits intomasterfrom
pawelchcki wants to merge 100 commits intomasterfrom
Conversation
…image The musl toolchain can now be built automatically from build_env/ as a bake dependency. This removes the need to pre-pull or maintain an external toolchain image. To use a pre-built image instead, set TOOLCHAIN_IMAGE. Also normalizes architecture naming (arm64 → aarch64) in Makefile for macOS compatibility. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…SI builds - Add inject-browser-sdk as submodule pinned to pawel-buildx-bake branch - Add docker bake targets for building inject-browser-sdk FFI artifacts - Add SSI targets (ssi-nginx, ssi-nginx-dev) that output OCI images with RUM enabled - Modify deps/CMakeLists.txt to support pre-built artifacts from docker bake - Update Dockerfile.nginx to copy pre-built artifacts instead of cloning repo - Fix compilation errors in src/rum/injection.cpp (pointer dereference, namespace) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Simplify docker-bake.hcl to only support standard nginx builds. Removes OPENRESTY_VERSIONS and INGRESS_NGINX_VERSIONS variables, their corresponding build targets, and group definitions. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove WAF_OPTIONS variable and waf_to_dir helper function - Remove WAF from nginx target matrix, args, and output paths - Delete unused packaging/Dockerfile.ingress and packaging/Dockerfile.openresty Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add a PUSH variable that switches SSI target outputs between type=docker (default) and type=registry when PUSH=true, enabling images to be pushed to ghcr.io. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Docker Bake targets and scripts for creating multi-arch OCI packages for the nginx SSI module using the datadog-packages tooling. New targets: - ssi-package-assemble[-dev]: Assembles nginx modules into package structure - ssi-package-create[-dev]: Creates per-arch OCI packages - ssi-package-merge[-dev]: Merges into multi-arch OCI index New scripts: - bin/generate-ssi-oci-package.sh: Full build (all nginx versions) - bin/generate-ssi-oci-package-dev.sh: Dev build (1.28.1 + 1.29.4 only) The packaging targets chain automatically via Docker Bake context dependencies, so running a single command builds nginx modules, assembles sources, and creates OCI packages. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use dockerfile-inline in bake to dynamically generate COPY statements from NGINX_VERSIONS variables, eliminating duplication between bake file and Dockerfiles. - Add ssi-nginx-modules[-dev] collector targets that combine all nginx build outputs into a single image - Simplify Dockerfile.ssi-sources to single COPY from collector - Remove Dockerfile.ssi-sources-dev (now unified) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add pipeline jobs that build multi-arch SSI OCI packages using docker buildx bake. Per-arch builds run in parallel on DinD runners, then merge into a single merged.tar artifact.
The CI Gatekeeper blocks third-party registry images. Use registry.ddbuild.io/images/mirror/docker:27 instead.
The docker:27 image is not available in the internal registry mirror. Use the existing test image (Alpine-based with docker-cli-compose) and install docker-cli-buildx at runtime.
Use CI_JOB_TOKEN to authenticate submodule clones via the gitlab.ddbuild.io mirror of DataDog GitHub repos.
The inject-browser-sdk and datadog-packages submodules use absolute GitHub URLs which the GitLab runners cannot access. Enable GIT_SUBMODULE_FORCE_HTTPS to rewrite them via CI_JOB_TOKEN.
Disable automatic submodule strategy and instead: 1. Rewrite github.com/DataDog URLs to gitlab.ddbuild.io via CI_JOB_TOKEN 2. Sync and update submodules manually in before_script
The test image doesn't include git. Install it alongside docker-cli-buildx before running submodule commands.
Let the runner handle submodule cloning via GIT_SUBMODULE_STRATEGY with GIT_CONFIG_COUNT/KEY/VALUE to rewrite GitHub URLs to the gitlab.ddbuild.io mirror using CI_JOB_TOKEN.
Replace COPY vendor + -mod=vendor with go mod download so the build doesn't require a pre-generated vendor directory.
ssi-package-create: 8 CPU / 16Gi (builds 18 nginx modules with 4-way parallelism via buildkitd.toml) ssi-package-merge: 4 CPU / 8Gi (lightweight merge of per-arch tars)
BuildKit: 4 -> 16 parallel build steps ssi-package-create: 32 CPU / 64Gi ssi-package-merge: 8 CPU / 16Gi
go mod download fails inside BuildKit due to private GitHub deps. Revert Dockerfile to use -mod=vendor and generate vendor dir via docker run golang:1.24-alpine before the bake.
The datadog-package tool depends on github.com/DataDog/go-service-authn (private). Configure git URL rewrite and GOPRIVATE inside the container so go mod vendor can fetch it via gitlab.ddbuild.io mirror.
Pushes the multi-arch OCI package to registry.ddbuild.io/ci/nginx-datadog/ssi-package:<commit-sha> using skopeo after the merge step completes.
Add pipeline jobs to build multi-arch SSI OCI packages for all supported nginx versions, merge them into a single multi-arch image index, and push to the internal registry. New jobs: - ssi-package-create: builds per-arch OCI packages using docker buildx bake (parallel matrix for amd64/arm64) - ssi-package-merge: combines per-arch packages into merged.tar - ssi-package-push: pushes the multi-arch OCI index to registry.ddbuild.io/ci/nginx-datadog/ssi-package:<commit_sha> New files: - .gitlab/ssi-package.yml: job definitions - buildkitd.toml: BuildKit max-parallelism config
…propagation - Extract case-insensitive parse_bool() for consistent boolean env var parsing across DD_RUM_ENABLED and trackResources/trackLongTasks/ trackUserInteractions (accepts true/false/1/0/yes/no/on/off) - Use ngx_pcalloc instead of ngx_palloc for error message buffers to ensure null-termination after ngx_snprintf - Guard against empty values vector in make_rum_json_config - Add null check on snippet_create_from_json in on_datadog_rum_config - Propagate return value of datadog_rum_merge_loc_config - Add unit tests for parse_bool, truthy/falsy variants, empty values
…' into pawel/packaging
Add support for configuring RUM via DD_RUM_* environment variables as fallback defaults. Nginx `datadog_rum_config` directive values always take precedence over env vars on a per-field basis. When no directive exists and valid env vars are present, RUM auto-enables. Environment variables: DD_RUM_ENABLED, DD_RUM_APPLICATION_ID, DD_RUM_CLIENT_TOKEN, DD_RUM_SITE, DD_RUM_SERVICE, DD_RUM_ENV, DD_RUM_VERSION, DD_RUM_SESSION_SAMPLE_RATE, DD_RUM_SESSION_REPLAY_SAMPLE_RATE, DD_RUM_TRACK_RESOURCES, DD_RUM_TRACK_LONG_TASKS, DD_RUM_TRACK_USER_INTERACTIONS, DD_RUM_REMOTE_CONFIGURATION_ID. Precedence rules: - `datadog_rum on/off` always wins over DD_RUM_ENABLED - `datadog_rum_config` fields always win over DD_RUM_* env vars - DD_RUM_ENABLED=false explicitly disables even with config env vars - Unrecognized DD_RUM_ENABLED values log a warning and skip auto-enable Also includes: - Case-insensitive parse_bool() for boolean env var parsing - Fix std::stod crash with std::strtod for sample rate parsing - Fix snippet memory leak on config error path - Guard against rum_enable=1 with null snippet (prevents segfault) - Null-terminated error messages via ngx_pcalloc - Empty values vector guard in make_rum_json_config - Use FetchContent for inject-browser-sdk dependency - RUM integration tests and C++ unit tests - RUM CI pipeline in GitLab
Resolve conflicts taking origin/master's refactored RUM code: - Use rum_config_map type alias and inline constexpr env mappings - Rename DD_RUM_ENV -> DD_RUM_ENVIRONMENT, DD_RUM_VERSION -> DD_RUM_MAJOR_VERSION - Use conf_err helper, unique_ptr for snippets, extracted helper functions - Use self.assertIn/assertIsNone in tests instead of bare assert - Remove duplicate CI job definitions
Build and test nginx-rum for all supported nginx versions (1.24.0 through 1.29.5) instead of just the subset of latest patch releases.
Replace custom ssi-package-merge and ssi-package-push jobs with the shared one-pipeline template from libdatadog-build, which handles datadog-package create, multi-arch merging, publishing, and promotion. The ssi-package-create job is split: ssi-build now runs ssi-package-assemble (stopping before datadog-package create), and prepare-oci-package.sh bridges the artifacts into one-pipeline's package-oci job.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #321 +/- ##
=======================================
Coverage 68.85% 68.85%
=======================================
Files 56 56
Lines 7471 7471
Branches 1058 1058
=======================================
Hits 5144 5144
Misses 1820 1820
Partials 507 507 🚀 New features to boost your workflow:
|
Without this, one-pipeline derives it from CI_PROJECT_NAME suffix
("datadog") giving "apm-library-datadog-package" instead of the
correct "apm-library-nginx-package".
Contributor
Author
|
Keeping in draft until improvements to CI reliability land |
# Conflicts: # .gitlab/build-and-test-all.yml
… tests - Update NGINX_VERSION in system-tests workflow from 1.29.5 to 1.29.6 to match the upstream system-tests Dockerfile - Add retry (max 2) to the GitLab coverage job for transient Vault failures - Apply formatting to test_injection.py
Allow ssi-build to start immediately without waiting for build-and-test-fast stage to complete.
The ssi-build job needs to clone the private inject-browser-sdk repo during Docker builds. Pass GIT_INSTEADOF_URL through bake to rewrite github.com/DataDog/ URLs to use the GitLab CI token.
Building 18 nginx versions with RUM in parallel fills the disk.
Instead of compiling all nginx modules from scratch inside Docker buildx (32 CPU, 64Gi RAM), collect pre-built RUM modules from build-nginx-rum-fast (fast pipeline) and build-nginx-rum-all (release tags) and restructure them for OCI packaging. - ssi-build: lightweight alpine job, needs build-nginx-rum-fast - ssi-build-all: same assembly, needs build-nginx-rum-all, release tags only - Revert buildkitd max-parallelism to 16 (disk issue no longer applies)
Gatekeeper policy blocks third-party registries on the CI runners. Use the existing nginx_musl_toolchain image from the internal registry.
Create block/allow test files for the SSI injection requirements test and set REQUIREMENTS_BLOCK_JSON_PATH / REQUIREMENTS_ALLOW_JSON_PATH variables so the one-pipeline shared job can find them.
The one-pipeline requirements_json_test expects packaging/sources/requirements.json to define native dependency support and deny rules. For nginx, we support glibc and musl on x64/arm64 with no deny rules (the injector matches nginx by binary path, no process-level filtering needed).
The nginx SSI package has no process-level deny rules in requirements.json (injection is controlled by package name matching, not binary path filtering), so there are no block test cases to validate.
The kubernetes-injection-test-ecr-publish job fails because ci.nginx-datadog.apm-ecosystems-reliability-profile doesn't exist in AWS SSM. This manual job copies it from ci.system-tests to nginx-datadog, httpd-datadog, and inject-browser-sdk. Remove this job after running it once.
Same image as kubernetes-injection-test-ecr-publish, which already has aws-cli and the right IAM credentials.
.pre stage gets dropped by one-pipeline shared config.
The ECR publish job requires an AWS SSM parameter that doesn't exist for nginx-datadog and can't be created from CI (IAM boundaries). Disable it until infra sets up the parameter.
Disabling with when:never breaks the pipeline because configure_system_tests needs this job. Use allow_failure instead until the SSM parameter is set up.
…rage - Add set -eo pipefail, shopt -s nullglob, empty-glob validation, and quoted dirname in ssi-package.yml assembly script - Sync docker-bake.hcl NGINX_VERSIONS with CI (add 1.25.x, 1.28.2, 1.29.5, 1.29.6) and update SSI_DEV to latest stable/mainline - Add arm64+musl test case in requirements-allow.json - Use set -euo pipefail in prepare-oci-package.sh - Add explanatory comment on package-oci optional needs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
libdatadog-buildto standardize OCI packaging, publishing, and promotionssi-package-mergeandssi-package-pushjobs with one-pipeline'spackage-oci→ publish → promote flowssi-package-createintossi-build(stops atssi-package-assemble) +prepare-oci-package.sh(bridges artifacts into one-pipeline)Changes
.gitlab/one-pipeline.locked.yml.gitlab/prepare-oci-package.shsources/fordatadog-package create.gitlab-ci.ymlbuild/shared-pipelinestages,package-ocineeds wiring.gitlab/ssi-package.ymlssi-package-create→ssi-build, remove merge/push jobsdocker-bake.hcloutputtossi-package-assembleto export to filesystemTest plan
docker buildx bake --print ssi-package-assemble-amd64shows the output directiveshared-pipelinestage jobs from one-pipelinessi-buildjob produces artifacts atartifacts/ssi-sources/package-ocijob runsprepare-oci-package.shand creates OCI package