Skip to content

enhancement(antithesis): DogStatsD load generation#1782

Merged
blt merged 1 commit into
mainfrom
blt/dogstatsd-load-generation
Jun 3, 2026
Merged

enhancement(antithesis): DogStatsD load generation#1782
blt merged 1 commit into
mainfrom
blt/dogstatsd-load-generation

Conversation

@blt
Copy link
Copy Markdown
Contributor

@blt blt commented Jun 1, 2026

Summary

This PR introduces DogStatD load generation. If you're familiar with
my work in lading you might notice this is very different. The
implementation here is responsible for giving Antithesis choices between
'compliant' dogstatsd lines and 'feral' lines which are mechanically
allowable lines but are in practice not sensible. Also of interest
control of contexts and send rates are jettisoned entirely, that's an
antithesis scheduling concern.

Metrics, events and service-checks are all present. Depending on the
antithesis shots I may need to adjust the ratios of compliant to feral
outcomes.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

Antithesis shots confirmed load generation behaves like a feral animal.

References

@dd-octo-sts dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jun 1, 2026
@datadog-official

This comment has been minimized.

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Jun 1, 2026

Binary Size Analysis (Agent Data Plane)

Baseline: dc62bbd · Comparison: bc9e3b9 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.85 MiB (baseline) vs 37.85 MiB (comparison)
Size Change: +0 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
anon.85240eacea40817b540ad191ce7e90d0.1.llvm.7043237968540469967 +129 B 1
anon.85240eacea40817b540ad191ce7e90d0.1.llvm.3595781591922113741 -129 B 1
anon.85240eacea40817b540ad191ce7e90d0.4.llvm.7043237968540469967 +114 B 1
anon.85240eacea40817b540ad191ce7e90d0.4.llvm.3595781591922113741 -114 B 1
anon.85240eacea40817b540ad191ce7e90d0.3.llvm.7043237968540469967 +108 B 1
anon.85240eacea40817b540ad191ce7e90d0.3.llvm.3595781591922113741 -108 B 1
anon.85240eacea40817b540ad191ce7e90d0.0.llvm.7043237968540469967 +96 B 1
anon.85240eacea40817b540ad191ce7e90d0.0.llvm.3595781591922113741 -96 B 1
anon.85240eacea40817b540ad191ce7e90d0.2.llvm.7043237968540469967 +94 B 1
anon.85240eacea40817b540ad191ce7e90d0.2.llvm.3595781591922113741 -94 B 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW]    +129  [NEW]     +40    anon.85240eacea40817b540ad191ce7e90d0.1.llvm.7043237968540469967
  [NEW]    +114  [NEW]     +25    anon.85240eacea40817b540ad191ce7e90d0.4.llvm.7043237968540469967
  [NEW]    +108  [NEW]     +19    anon.85240eacea40817b540ad191ce7e90d0.3.llvm.7043237968540469967
  [NEW]     +96  [NEW]      +7    anon.85240eacea40817b540ad191ce7e90d0.0.llvm.7043237968540469967
  [NEW]     +94  [NEW]      +5    anon.85240eacea40817b540ad191ce7e90d0.2.llvm.7043237968540469967
  [DEL]     -94  [DEL]      -5    anon.85240eacea40817b540ad191ce7e90d0.2.llvm.3595781591922113741
  [DEL]     -96  [DEL]      -7    anon.85240eacea40817b540ad191ce7e90d0.0.llvm.3595781591922113741
  [DEL]    -108  [DEL]     -19    anon.85240eacea40817b540ad191ce7e90d0.3.llvm.3595781591922113741
  [DEL]    -114  [DEL]     -25    anon.85240eacea40817b540ad191ce7e90d0.4.llvm.3595781591922113741
  [DEL]    -129  [DEL]     -40    anon.85240eacea40817b540ad191ce7e90d0.1.llvm.3595781591922113741
  [ = ]       0  [ = ]       0    TOTAL

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Jun 1, 2026

Regression Detector (Agent Data Plane)

Run ID: b70e957d-bab1-4fea-ac61-d7f12708087c
Baseline: dc62bbd2 · Comparison: bc9e3b97 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
dsd_uds_1mb_3k_contexts_cpu (erratic) cpu ⚪ +7.61 metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic) cpu ⚪ +4.54 metrics profiles logs
otlp_ingest_metrics_5mb_memory memory ⚪ +3.64 metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored) memory ⚪ +3.03 metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored) cpu ⚪ +1.95 metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic) cpu ⚪ +1.15 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic) cpu ⚪ +0.94 metrics profiles logs
dsd_uds_1mb_3k_contexts_memory memory ⚪ +0.60 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.23 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ +0.20 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory memory ⚪ +0.14 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ +0.11 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput throughput ⚪ -0.08 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput throughput ⚪ -0.08 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ +0.06 metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput throughput ⚪ -0.02 metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored) throughput ⚪ -0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput throughput ⚪ -0.00 metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput throughput ⚪ +0.01 metrics profiles logs
dsd_uds_100mb_3k_contexts_memory memory ⚪ -0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_memory memory ⚪ -0.04 metrics profiles logs
otlp_ingest_traces_5mb_memory memory ⚪ -0.04 metrics profiles logs
otlp_ingest_metrics_5mb_throughput throughput ⚪ +0.04 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.09 metrics profiles logs
otlp_ingest_traces_5mb_throughput throughput ⚪ +0.15 metrics profiles logs
dsd_uds_500mb_3k_contexts_memory memory ⚪ -0.30 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory memory ⚪ -0.31 metrics profiles logs
dsd_uds_512kb_3k_contexts_memory memory ⚪ -0.33 metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput throughput ⚪ +0.35 metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic) cpu ⚪ -0.84 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic) cpu ⚪ -1.07 metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic) cpu ⚪ -1.15 metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic) cpu ⚪ -1.85 metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic) cpu 🟢 -6.02 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 127 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 39.8 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 61.4 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 184 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 26.6 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@blt blt changed the title DogStatsD load generation enhancement(antithesis): DogStatsD load generation Jun 1, 2026
@blt blt marked this pull request as ready for review June 1, 2026 14:26
Copilot AI review requested due to automatic review settings June 1, 2026 14:26
@blt blt requested review from a team as code owners June 1, 2026 14:26
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a31a487f1b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an Antithesis-oriented DogStatsD workload generator to the test harness, shifting the Antithesis setup away from the prior millstone-based load driver and toward generating a mix of “clean” (compliant) and “feral” (mechanically allowable but nonsensical) DogStatsD lines (metrics/events/service-checks). It also updates the Antithesis scratchbook documentation to reflect the new workload and its current bug-reproduction reach.

Changes:

  • Add a new harness::payload::dogstatsd module that generates metric, event, and service-check lines with clean/feral “vibes”.
  • Rework parallel_driver_send_dogstatsd to send randomly sized batches of generated DogStatsD messages and update associated SDK assertions.
  • Update Antithesis deployment and scratchbook docs for the new workload driver and topology.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 12 comments.

Show a summary per file
File Description
test/antithesis/scratchbook/property-catalog.md Updates catalog notes/status to reflect new driver behavior and workload reach.
test/antithesis/scratchbook/existing-assertions.md Updates assertion inventory and explains the new driver; needs minor wording alignment.
test/antithesis/scratchbook/deployment-topology.md Updates topology narrative for the new driver; contains an outdated UDS note vs actual deployment.
test/antithesis/scratchbook/bug-ledger.md Adds “workload reach” note and updates bug-hunt entry details.
test/antithesis/harness/src/rand.rs Extends randomness helpers with Probe for multiple numeric types and Boundary<T> samplers.
test/antithesis/harness/src/payload/dogstatsd/common.rs Adds shared “segment/number/tag” construction utilities and clean/feral vibe handling.
test/antithesis/harness/src/payload/dogstatsd/metrics.rs Adds feral metric line generator with randomized extensions/tags/value encodings.
test/antithesis/harness/src/payload/dogstatsd/events.rs Adds feral event generator (including skewed header lengths in feral mode).
test/antithesis/harness/src/payload/dogstatsd/service_checks.rs Adds feral service-check generator.
test/antithesis/harness/src/payload/dogstatsd.rs Adds top-level DogStatsD message selection and in-file format reference notes.
test/antithesis/harness/src/payload.rs Adds payload module root.
test/antithesis/harness/src/lib.rs Exposes the new payload module from the harness crate.
test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs Replaces old regime-based metric sender with batch-based clean/feral/mixed driver.
test/antithesis/harness/src/bin/first_sample_config/config.rs Changes sampled log_level values; currently introduces an unsupported level for ADP.
test/antithesis/harness/Cargo.toml Adds itoa/ryu dependencies for fast numeric formatting.
test/antithesis/deploy/Dockerfile Removes millstone from workload image build; builds/copies new test-command binaries.
test/antithesis/deploy/docker-compose.yaml Removes RUST_BACKTRACE env var; configures shared UDS volume for DogStatsD socket.
Cargo.lock Records new dependency edges for harness additions (itoa, ryu).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread test/antithesis/harness/src/bin/first_sample_config/config.rs Outdated
Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
Comment thread test/antithesis/scratchbook/deployment-topology.md Outdated
Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
Comment thread test/antithesis/harness/src/bin/first_sample_config/config.rs Outdated
Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
Comment thread test/antithesis/harness/src/bin/first_sample_config/config.rs Outdated
Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
Comment thread test/antithesis/scratchbook/deployment-topology.md Outdated
Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from a152801 to 4fc1007 Compare June 1, 2026 15:33
@blt blt force-pushed the blt/dogstatsd-load-generation branch from a31a487 to abbf74a Compare June 1, 2026 15:33
Copilot AI review requested due to automatic review settings June 1, 2026 18:17
@blt blt force-pushed the blt/dogstatsd-load-generation branch from abbf74a to bb12c83 Compare June 1, 2026 18:17
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from 4fc1007 to 0f6c10e Compare June 1, 2026 18:17
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from 0f6c10e to 5aaad6f Compare June 2, 2026 12:22
@blt blt force-pushed the blt/dogstatsd-load-generation branch from bb12c83 to 51aa65b Compare June 2, 2026 12:23
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch 2 times, most recently from 0239db1 to 29d780c Compare June 2, 2026 16:25
@blt blt force-pushed the blt/dogstatsd-load-generation branch from 51aa65b to e4dc275 Compare June 2, 2026 16:25
Copilot AI review requested due to automatic review settings June 2, 2026 20:16
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from 29d780c to f4d5a5a Compare June 2, 2026 20:16
@blt blt force-pushed the blt/dogstatsd-load-generation branch from e4dc275 to b9e6aa3 Compare June 2, 2026 20:16
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 4 comments.

Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
Comment thread test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs Outdated
Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
Comment thread test/antithesis/scratchbook/existing-assertions.md Outdated
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from f4d5a5a to 90f327d Compare June 2, 2026 20:33
@blt blt force-pushed the blt/dogstatsd-load-generation branch from b9e6aa3 to 4903cee Compare June 2, 2026 20:33
Copilot AI review requested due to automatic review settings June 2, 2026 20:48
@blt blt force-pushed the blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding branch from 90f327d to 6577832 Compare June 2, 2026 20:48
@blt blt force-pushed the blt/dogstatsd-load-generation branch from 4903cee to 6426b97 Compare June 2, 2026 20:48
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 6 comments.

Comment thread test/antithesis/harness/src/rand.rs
Comment thread test/antithesis/harness/src/rand.rs
Comment thread test/antithesis/harness/src/rand.rs
Comment thread test/antithesis/harness/src/bin/first_sample_config/config.rs
Comment thread test/antithesis/harness/src/payload/dogstatsd/common.rs
Comment thread test/antithesis/harness/src/bin/parallel_driver_send_dogstatsd.rs
@blt blt changed the base branch from blt/assert_adp_aliveness_on_bootup_add_sometimes_check_to_forwarding to graphite-base/1782 June 2, 2026 20:59
@blt blt force-pushed the blt/dogstatsd-load-generation branch from 6426b97 to ae74073 Compare June 2, 2026 20:59
@blt blt force-pushed the graphite-base/1782 branch from 6577832 to dc62bbd Compare June 2, 2026 20:59
@graphite-app graphite-app Bot changed the base branch from graphite-base/1782 to main June 2, 2026 21:00
@blt blt force-pushed the blt/dogstatsd-load-generation branch 2 times, most recently from 7e7e574 to c19a793 Compare June 2, 2026 21:27
This commit introduces DogStatD load generation. If you're familiar with
my work in lading you might notice this is very different. The
implementation here is responsible for giving Antithesis choices between
'compliant' dogstatsd lines and 'feral' lines which are _mechanically_
allowable lines but are in practice not sensible. Also of interest
control of contexts and send rates are jettisoned entirely, that's an
antithesis scheduling concern.

Metrics, events and service-checks are all present. Depending on the
antithesis shots I may need to adjust the ratios of compliant to feral
outcomes.
@blt blt force-pushed the blt/dogstatsd-load-generation branch from c19a793 to bc9e3b9 Compare June 2, 2026 21:37
@blt blt merged commit efe110d into main Jun 3, 2026
87 of 89 checks passed
@blt blt deleted the blt/dogstatsd-load-generation branch June 3, 2026 17:07
dd-octo-sts Bot pushed a commit that referenced this pull request Jun 3, 2026
## Summary
<!-- Please provide a brief summary about what this PR does.
This should help the reviewers give feedback faster and with higher quality. -->

This PR introduces DogStatD load generation. If you're familiar with
my work in lading you might notice this is very different. The
implementation here is responsible for giving Antithesis choices between
'compliant' dogstatsd lines and 'feral' lines which are _mechanically_
allowable lines but are in practice not sensible. Also of interest
control of contexts and send rates are jettisoned entirely, that's an
antithesis scheduling concern.

Metrics, events and service-checks are all present. Depending on the
antithesis shots I may need to adjust the ratios of compliant to feral
outcomes.

## Change Type
- [ ] Bug fix
- [ ] New feature
- [x] Non-functional (chore, refactoring, docs)
- [ ] Performance

## How did you test this PR?
<!-- Please how you tested these changes here -->

Antithesis shots confirmed load generation behaves like a feral animal.

## References

<!-- Please list any issues closed by this PR. -->

<!--
- Closes: <issue link>
-->

<!-- Any other issues or PRs relevant to this PR? Feel free to list them here. --> efe110d
@blt blt mentioned this pull request Jun 5, 2026
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test All things testing: unit/integration, correctness, SMP regression, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants