[Tracing] Implement initial OTLP traces weblog tests by zacharycmontoya · Pull Request #6363 · DataDog/system-tests

zacharycmontoya · 2026-02-20T22:46:45Z

Motivation

We are seeing increased demand for exporting traces as OTLP from our DD SDKs (rather than Datadog-proprietary MessagePack), so we are prototyping and establishing requirements for generating OTLP traces payloads. This is only the first of a series of PRs to establish clear expectations for what the generated OTLP traces and trace stats will look like.

Changes

Scenario: Adds a new APM_TRACING_OTLP scenario to test the weblog application with the configuration needed for the DD SDK to export traces using OTLP. This also adds a include_opentelemetry property to the EndToEndScenario to set up the OpenTelemetry interface.
Tests: Adds tests/otel/test_tracing_otlp.py::Test_Otel_Tracing_OTLP::test_tracing to send a request to the weblog app using weblog.get("/") and asserts properties of the OTLP trace payload
Interfaces: Updates the OpenTelemetry interface with methods get_otel_spans and get_trace_stats to retrieve the OTLP payloads for test assertions

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

…ple. Notably, this creates a new scenario APM_TRACING_OTLP to enable the environment variables needed to configure the SDK to export traces as OTLP.

…is yet

…e only use the "/v1/metrics" subpath

github-actions · 2026-02-20T22:47:11Z

CODEOWNERS have been resolved as:

tests/otel/test_tracing_otlp.py                                         @DataDog/system-tests-core
.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/dd_constants.py                                                   @DataDog/system-tests-core
utils/interfaces/_open_telemetry.py                                     @DataDog/system-tests-core
utils/scripts/ci_orchestrators/workflow_data.py                         @DataDog/system-tests-core

datadog-official · 2026-02-20T23:15:31Z

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 2 Tests failed

tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP.test_single_server_trace[nginx] from system_tests_suite (Datadog) (Fix with Cursor)

assert 0 == 1
 +  where 0 = len([])

self = <tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP object at 0x7ff5fef4e540>

    def test_single_server_trace(self):
        """Validates the required elements of the OTLP payload for a single trace"""
        data = list(interfaces.open_telemetry.get_otel_spans(self.req))
    
        # Assert that there is only one OTLP request containing the desired server span
...

tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP.test_single_server_trace[nginx] from system_tests_suite (Datadog) (Fix with Cursor)

assert 0 == 1
 +  where 0 = len([])

self = <tests.otel.test_tracing_otlp.Test_Otel_Tracing_OTLP object at 0x7fe323f50a40>

    def test_single_server_trace(self):
        """Validates the required elements of the OTLP payload for a single trace"""
        data = list(interfaces.open_telemetry.get_otel_spans(self.req))
    
        # Assert that there is only one OTLP request containing the desired server span
...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a63042c | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

tests/otel/test_tracing_otlp.py

…v, since users do not need this feature for OTLP export to work

…e we're not immediately implementing that

…ing specifically: - At runtime determine if the request is JSON - If JSON, look up proto field names by their camelCase representation. Otherwise, look up field names by their snake_case representation - If JSON, assert that the 'traceId' and 'spanId' fields are case-insensitive hexadecimal strings, rather than base64-encoded strings - If JSON, assert that enums (e.g. span.kind and span.status.code) are encoded using an integer, not a string representation of the enum value name - Regardless of protocol, get the time before and after the test HTTP request is issued, and assert that the span's reported 'start_time_unix_nano' and 'end_time_unix_nano' fall in this range - Regardless of protocol, make the 'http.method' and 'http.status_code' span attribute assertions more flexible by also testing against their stable OpenTelemetry HTTP equivalents of 'http.request.method' and 'http.response.status_code', respectively

tests/otel/test_tracing_otlp.py

cbeauchesne · 2026-03-06T14:34:57Z

tests/otel/test_tracing_otlp.py

+    STATUS_CODE_ERROR = 2
+
+
+def get_keyvalue_generator(attributes: list[dict]) -> Iterator[tuple[str, Any]]:


I feel that data should be properly deserialized by the proxy, WDYT ?

I prefer this approach where we add workarounds in order to access the request properly, because then we can still store the original contents of the request. And if there are any invalid fields that we try to assert against, it should be easy for the developer to debug this (the original request is saved to disk via the interface). Do you feel strongly that we should change this at the moment?

…ision of 0 is respected for OTLP traces by default

zacharycmontoya added 5 commits February 20, 2026 14:29

Implement initial OTLP traces + trace stats with the .NET weblog exam…

dd04bd9

…ple. Notably, this creates a new scenario APM_TRACING_OTLP to enable the environment variables needed to configure the SDK to export traces as OTLP.

Skip assertions for the Trace Metrics, as we don't have a spec for th…

ebbbb5a

…is yet

Run formatter

7fee5a2

Remove "/api/v0.2/stats" subpath for the OpenTelemetry interface so w…

5f8b2a8

…e only use the "/v1/metrics" subpath

Add manifest entries for languages

221b031

Merge branch 'main' into zach.montoya/weblog-traces-otlp

86b284d

ida613 reviewed Feb 24, 2026

View reviewed changes

tests/otel/test_tracing_otlp.py Outdated Show resolved Hide resolved

ida613 reviewed Feb 24, 2026

View reviewed changes

tests/otel/test_tracing_otlp.py Outdated Show resolved Hide resolved

ida613 reviewed Feb 24, 2026

View reviewed changes

tests/otel/test_tracing_otlp.py Outdated Show resolved Hide resolved

zacharycmontoya added 5 commits March 5, 2026 14:10

Remove DD_TRACE_OTEL_ENABLED=true from the APM_TRACING_OTLP weblog en…

cacbab5

…v, since users do not need this feature for OTLP export to work

Remove all helper methods and setup related to OTLP trace stats, sinc…

14b1995

…e we're not immediately implementing that

Small refactoring of code and comments

6a1df07

Merge branch 'main' into zach.montoya/weblog-traces-otlp

f032d9d

cbeauchesne reviewed Mar 6, 2026

View reviewed changes

tests/otel/test_tracing_otlp.py Outdated Show resolved Hide resolved

cbeauchesne reviewed Mar 6, 2026

View reviewed changes

zacharycmontoya added 3 commits March 10, 2026 14:39

Add test case asserting that an incoming trace context's sampling dec…

746148b

…ision of 0 is respected for OTLP traces by default

Move enums to dd_constants.py

919b5ef

Merge branch 'main' into zach.montoya/weblog-traces-otlp

a63042c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tracing] Implement initial OTLP traces weblog tests#6363

[Tracing] Implement initial OTLP traces weblog tests#6363
zacharycmontoya wants to merge 14 commits intomainfrom
zach.montoya/weblog-traces-otlp

zacharycmontoya commented Feb 20, 2026

Uh oh!

github-actions bot commented Feb 20, 2026 •

edited

Loading

Uh oh!

datadog-official bot commented Feb 20, 2026 •

edited by datadog-prod-us1-6 bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne Mar 6, 2026

Uh oh!

zacharycmontoya Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		STATUS_CODE_ERROR = 2


		def get_keyvalue_generator(attributes: list[dict]) -> Iterator[tuple[str, Any]]:

Conversation

zacharycmontoya commented Feb 20, 2026

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions bot commented Feb 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official bot commented Feb 20, 2026 • edited by datadog-prod-us1-6 bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbeauchesne Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

zacharycmontoya Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Feb 20, 2026 •

edited

Loading

datadog-official bot commented Feb 20, 2026 •

edited by datadog-prod-us1-6 bot

Loading