Skip to content

fix(tracer): only set _dd.p.ksr after agent rates are received#4523

Open
bm1549 wants to merge 2 commits intomainfrom
brian.marks/fix-ksr-default
Open

fix(tracer): only set _dd.p.ksr after agent rates are received#4523
bm1549 wants to merge 2 commits intomainfrom
brian.marks/fix-ksr-default

Conversation

@bm1549
Copy link
Contributor

@bm1549 bm1549 commented Mar 11, 2026

What Does This Do

Fixes _dd.p.ksr (Knuth Sampling Rate) to only be set on spans when the agent has provided sampling rates via readRatesJSON(). Previously, ksr was unconditionally set in prioritySampler.apply(), including when the rate was the initial client-side default (1.0) before any agent response arrived.

Motivation

Cross-language consistency: Python, Java, PHP, and other tracers only set ksr when actual agent rates or sampling rules are applied, not for the default fallback. This aligns Go with that behavior.

See RFC: "Transmit Knuth sampling rate to backend"

Additional Notes

  • Added agentRatesLoaded bool field to prioritySampler, set to true in readRatesJSON()
  • apply() now gates ksr behind agentRatesLoaded check
  • Rule-based sampling path (applyTraceRuleSampling in span.go) unchanged — correctly always sets ksr
  • Tests added: ksr-not-set-without-agent-rates and ksr-set-after-agent-rates-received

Related PRs across tracers:

🤖 Generated with Claude Code

…ceived

The priority sampler was unconditionally setting the _dd.p.ksr tag on
every span, even when using the initial client-side default rate (1.0)
before any agent response was received. This is inconsistent with other
Datadog tracers, which only propagate ksr when actual agent rates or
sampling rules are applied.

Add an agentRatesLoaded flag to prioritySampler that is set to true the
first time readRatesJSON processes an agent response. The apply method
now only sets keyKnuthSamplingRate when this flag is true. Rule-based
sampling (applyTraceRuleSampling) is unaffected and continues to set
ksr unconditionally, which is the correct behavior.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bm1549 bm1549 added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Mar 11, 2026
@bm1549 bm1549 changed the title Fix _dd.p.ksr to only set after agent rates are received fix(tracer): only set _dd.p.ksr after agent rates are received Mar 11, 2026
@codecov
Copy link

codecov bot commented Mar 11, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.98%. Comparing base (c1389f4) to head (2b0e03a).

Additional details and impacted files
Files with missing lines Coverage Δ
ddtrace/tracer/sampler.go 94.04% <100.00%> (+0.37%) ⬆️

... and 259 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@datadog-prod-us1-6
Copy link

datadog-prod-us1-6 bot commented Mar 11, 2026

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 59.30% (+3.54%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ef998e2 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Mar 11, 2026

Benchmarks

Benchmark execution time: 2026-03-13 18:44:11

Comparing candidate commit ef998e2 in PR branch brian.marks/fix-ksr-default with baseline commit 2d3b985 in branch main.

Found 9 performance improvements and 0 performance regressions! Performance is the same for 139 metrics, 7 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:BenchmarkOTelApiWithCustomTags/datadog_otel_api-25

  • 🟩 allocated_mem [-440 bytes; -434 bytes] or [-11.565%; -11.416%]
  • 🟩 allocations [-1; -1] or [-3.704%; -3.704%]
  • 🟩 execution_time [-117.156ns; -86.244ns] or [-2.890%; -2.127%]

scenario:BenchmarkOTelApiWithCustomTags/otel_api-25

  • 🟩 allocated_mem [-442 bytes; -427 bytes] or [-8.626%; -8.327%]
  • 🟩 allocations [-1; -1] or [-2.381%; -2.381%]

scenario:BenchmarkStartSpanConfig/scenario_WithStartSpanConfig-25

  • 🟩 allocations [-1; -1] or [-5.263%; -5.263%]
  • 🟩 execution_time [-141.759ns; -130.641ns] or [-5.560%; -5.124%]

scenario:BenchmarkStartSpanConfig/scenario_none-25

  • 🟩 allocations [-1; -1] or [-4.762%; -4.762%]
  • 🟩 execution_time [-144.538ns; -128.862ns] or [-5.366%; -4.784%]

@bm1549 bm1549 marked this pull request as ready for review March 13, 2026 18:02
@bm1549 bm1549 requested review from a team as code owners March 13, 2026 18:02
@hannahkm
Copy link
Contributor

Can we resolve the conflicts with main? There's been some changes to the sampling rate functions since you last pulled.

Resolve conflicts in ddtrace/tracer/sampler.go: keep both
agentRatesLoaded field (from this branch) and lastCapped field
with rate capping logic (from main).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants