ci: upload system-tests results to Test Optimization and add tracer-release nightly by nccatoni · Pull Request #3660 · DataDog/dd-trace-php

nccatoni · 2026-02-18T14:16:02Z

Description

Two changes to the system-tests CI integration:

Upload all system-tests results to Test Optimization — adds an after_script to the .system_tests base job that copies JUnit XML files and uploads them via the existing silent-upload-junit-to-datadog.sh script.
Add tracer-release nightly — create a new system-tests workflow that uses the official system-tests reusable workflow to run the tracer-release scenario group on master.

This is meant to replace the system-tests-dashboard nightly.

Reviewer checklist

This is a CI-only change, no code changes.
Appropriate labels assigned.

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

datadog-official · 2026-02-18T14:22:44Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1028 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integration\PHPInstallerTest::testSearchPhpBinaries
Test code or tested code printed unexpected output: Searching for available php binaries, this operation might take a while.

testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69b432b800000000c19f771aada6e8be
tid: 69b432b800000000
hexProcessTraceId: c19f771aada6e8be
hexProcessSpanId: a1af751fba58cc00
processTraceId: 13952001127083337918
processSpanId: 11650659540161580032

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106

testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 69b4333800000000d68d1862b64ed636
tid: 69b4333800000000
hexProcessTraceId: d68d1862b64ed636
hexProcessSpanId: 1439035ab8d68ced
processTraceId: 15460039908077262390
processSpanId: 1457199642614074605

View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 6b14c2f | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

Co-authored-by: Cursor <cursoragent@cursor.com>

codecov-commenter · 2026-02-18T14:35:08Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.30%. Comparing base (147f3c2) to head (6b14c2f).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3660      +/-   ##
==========================================
- Coverage   62.32%   62.30%   -0.02%     
==========================================
  Files         142      142              
  Lines       13586    13586              
  Branches     1775     1775              
==========================================
- Hits         8467     8465       -2     
- Misses       4311     4314       +3     
+ Partials      808      807       -1

see 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 147f3c2...6b14c2f. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Co-authored-by: Cursor <cursoragent@cursor.com>

- Add after_script to .system_tests base job to upload JUnit results to Datadog Test Optimization using the existing upload script - Add tracer-release scenario group to configure_system_tests on schedule and master runs - Remove separate GitHub Actions nightly workflow Made-with: Cursor

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

Replace the generic upload-junit-to-datadog.sh with a custom after_script that: - Uses --service system-tests (not dd-trace-php-tests) - References logs directly from system-tests/logs*/ - Uses the datadog-ci standalone binary (no npm needed) - Uses Python for JSON/zip (guaranteed available in the image) - Provides explicit error messages instead of silent exit 0 Made-with: Cursor

pr-commenter · 2026-02-26T17:25:52Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-03-12 18:55:43

Comparing candidate commit 85c13ba in PR branch add-nightly-system-tests with baseline commit a9caaa0 in branch master.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 190 metrics, 2 unstable metrics.

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

🟩 execution_time [-7.495µs; -5.665µs] or [-6.712%; -5.073%]

scenario:TraceSerializationBench/benchSerializeTrace-opcache

🟩 execution_time [-20.434µs; -9.266µs] or [-5.819%; -2.639%]

Made-with: Cursor

nccatoni · 2026-03-05T12:06:47Z

For this to work two things need to be done:
Adding docker/login-action and nick-fields/retry to the list of authorized actions for the repo
Adding an Datadog API key as a github repo secret

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a65519dc59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.github/workflows/system_tests.yml

.gitlab/generate-package.php

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b44c4ab3eb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.gitlab/generate-package.php

Leiyks · 2026-03-12T14:52:28Z

.gitlab/generate-package.php

+  after_script:
+    - |
+      set +e
+      echo "=== Uploading system-tests JUnit results to Test Optimization ==="
+
+      # Check that there are JUnit files to upload
+      ls system-tests/logs*/reportJunit.xml >/dev/null 2>&1
+      if [ $? -ne 0 ]; then echo "No JUnit XML files found, skipping upload"; exit 0; fi
+
+      # Download datadog-ci standalone binary
+      echo "Downloading datadog-ci..."
+      curl -L --fail "https://github.com/DataDog/datadog-ci/releases/latest/download/datadog-ci_linux-x64" -o /tmp/datadog-ci
+      if [ $? -ne 0 ]; then echo "Failed to download datadog-ci"; exit 0; fi
+      chmod +x /tmp/datadog-ci
+
+      # Download and extract Vault CLI
+      echo "Downloading Vault..."
+      VAULT_VERSION="1.20.0"
+      curl -L --fail "https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip" -o /tmp/vault.zip
+      if [ $? -ne 0 ]; then echo "Failed to download Vault"; exit 0; fi
+      python3 -c "import zipfile; zipfile.ZipFile('/tmp/vault.zip').extractall('/tmp/')"
+      chmod +x /tmp/vault
+
+      # Fetch API key from Vault
+      echo "Fetching API key from Vault..."
+      VAULT_JSON=$(/tmp/vault kv get --format=json "kv/k8s/gitlab-runner/dd-trace-php/datadoghq-api-key" 2>&1)
+      if [ $? -ne 0 ]; then echo "Failed to fetch API key from Vault: $VAULT_JSON"; exit 0; fi
+      export DATADOG_API_KEY=$(echo "$VAULT_JSON" | python3 -c "import sys,json; print(json.loads(sys.stdin.read())['data']['data']['key'])")
+      export DATADOG_SITE="datadoghq.com"
+
+      # Upload JUnit results
+      echo "Uploading JUnit results..."
+      /tmp/datadog-ci junit upload \
+        system-tests/logs*/reportJunit.xml \
+        --service system-tests \
+        --env ci \
+        --verbose \
+        --xpath-tag "test.codeowners=/testcase/properties/property[@name='test.codeowners']"
+      echo "=== Upload complete ==="


We have a script that basically do all of this in .gitlab/upload-junit-to-datadog.sh

You may need to tweak it a bit to add the xpath-tag thing tho

Add nightly system-tests workflow

17ff796

Run the official system-tests reusable workflow daily at 4 AM GMT using dev mode against the latest commit on the default branch. Co-authored-by: Cursor <cursoragent@cursor.com>

Enable push_to_test_optimization

10f7eab

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~Add nightly system-tests workflow~~ ci: add nightly system-tests workflow Feb 18, 2026

nccatoni and others added 6 commits February 18, 2026 15:40

Pass secrets explicitly to reusable workflow

2f73f4b

Co-authored-by: Cursor <cursoragent@cursor.com>

Only pass TEST_OPTIMIZATION_API_KEY secret

34d46d2

Co-authored-by: Cursor <cursoragent@cursor.com>

tmp: add pull_request trigger for testing

420b281

Co-authored-by: Cursor <cursoragent@cursor.com>

Set desired_execution_time to 300s

1185a36

Co-authored-by: Cursor <cursoragent@cursor.com>

Also pass DD_API_KEY secret

4af9b76

Co-authored-by: Cursor <cursoragent@cursor.com>

nccatoni changed the title ~~ci: add nightly system-tests workflow~~ ci: upload system-tests results to Test Optimization and add tracer-release nightly Feb 26, 2026

nccatoni added 2 commits February 26, 2026 16:54

ci: fix after_script by using upload script directly

42bcd97

The silent wrapper's grep returns exit code 1 when no summary lines are found, causing the after_script to fail. Use the non-silent version with || true to prevent after_script failures. Made-with: Cursor

nccatoni added 3 commits March 2, 2026 14:20

Run only tracer-release on schedule, it already includes other groups

34f79e3

Made-with: Cursor

Temporarily run tracer-release on all triggers for testing

bafec28

Made-with: Cursor

Tracer release scenario group on main

578e0b6

nccatoni force-pushed the add-nightly-system-tests branch from 9a812c9 to 578e0b6 Compare March 5, 2026 11:08

nccatoni added 2 commits March 5, 2026 12:11

Test

4db0655

Remove ref argument

a65519d

nccatoni marked this pull request as ready for review March 5, 2026 12:06

nccatoni requested a review from a team as a code owner March 5, 2026 12:06

chatgpt-codex-connector bot reviewed Mar 5, 2026

View reviewed changes

.github/workflows/system_tests.yml Outdated Show resolved Hide resolved

.gitlab/generate-package.php Outdated Show resolved Hide resolved

nccatoni marked this pull request as draft March 9, 2026 10:15

Every day schedule (include weekends)

b44c4ab

nccatoni marked this pull request as ready for review March 12, 2026 14:37

chatgpt-codex-connector bot reviewed Mar 12, 2026

View reviewed changes

.gitlab/generate-package.php Outdated Show resolved Hide resolved

Leiyks reviewed Mar 12, 2026

View reviewed changes

nccatoni added 6 commits March 12, 2026 17:52

Use upload-junit-to-datadog.sh

5159a71

Remove github workflow

85c13ba

Test

3c214dc

Add tracer-release system tests job on GitLab

be1373a

Merge branch 'master' into add-nightly-system-tests

a258de8

Fetch DD_API_KEY from Vault for tracer-release system tests

6b14c2f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660

ci: upload system-tests results to Test Optimization and add tracer-release nightly#3660
nccatoni wants to merge 22 commits intomasterfrom
add-nightly-system-tests

nccatoni commented Feb 18, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 18, 2026 •

edited by datadog-prod-us1-3 bot

Loading

Uh oh!

codecov-commenter commented Feb 18, 2026 •

edited

Loading

Uh oh!

pr-commenter bot commented Feb 26, 2026 •

edited

Loading

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

Leiyks Mar 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nccatoni commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer checklist

Uh oh!

datadog-official bot commented Feb 18, 2026 • edited by datadog-prod-us1-3 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

codecov-commenter commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pr-commenter bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ tracer ]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

scenario:TraceSerializationBench/benchSerializeTrace-opcache

Uh oh!

nccatoni commented Mar 5, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Leiyks Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nccatoni commented Feb 18, 2026 •

edited

Loading

datadog-official bot commented Feb 18, 2026 •

edited by datadog-prod-us1-3 bot

Loading

codecov-commenter commented Feb 18, 2026 •

edited

Loading

pr-commenter bot commented Feb 26, 2026 •

edited

Loading