ci: make code coverage dynamic by hoolioh · Pull Request #1677 · DataDog/libdatadog

hoolioh · 2026-03-06T10:20:44Z

What does this PR do?

A brief description of the change being made with this pull request.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

How to test the change?

Describe here in detail how the change can be validated.

pr-commenter · 2026-03-06T10:50:38Z

Benchmarks

Comparison

Benchmark execution time: 2026-03-13 13:20:44

Comparing candidate commit 49ff857 in PR branch julio/rd-week-code-coverage with baseline commit 56bf633 in branch julio/rd-week-dynamic-pipeline.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 2 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Candidate

Candidate benchmark details

Group 1

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
ip_address/quantize_peer_ip_address_benchmark	execution_time	4.945µs	5.033µs ± 0.046µs	5.028µs ± 0.034µs	5.055µs	5.125µs	5.128µs	5.128µs	2.00%	0.479	-0.372	0.91%	0.003µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark	execution_time	[5.027µs; 5.039µs] or [-0.127%; +0.127%]	None	None	None

Group 2

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
concentrator/add_spans_to_concentrator	execution_time	13.063ms	13.088ms ± 0.015ms	13.087ms ± 0.009ms	13.097ms	13.116ms	13.128ms	13.145ms	0.45%	0.836	0.729	0.11%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
concentrator/add_spans_to_concentrator	execution_time	[13.086ms; 13.090ms] or [-0.016%; +0.016%]	None	None	None

Group 3

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
profile_add_sample2_frames_x1000	execution_time	737.492µs	740.731µs ± 1.210µs	740.802µs ± 0.808µs	741.564µs	742.433µs	744.022µs	744.586µs	0.51%	0.080	0.459	0.16%	0.086µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
profile_add_sample2_frames_x1000	execution_time	[740.564µs; 740.899µs] or [-0.023%; +0.023%]	None	None	None

Group 4

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
two way interface	execution_time	17.580µs	25.330µs ± 9.413µs	18.003µs ± 0.237µs	33.828µs	42.756µs	43.561µs	66.685µs	270.40%	0.906	0.310	37.07%	0.666µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
two way interface	execution_time	[24.025µs; 26.634µs] or [-5.150%; +5.150%]	None	None	None

Group 5

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching deserializing traces from msgpack to their internal representation	execution_time	48.081ms	48.307ms ± 0.896ms	48.176ms ± 0.037ms	48.218ms	48.314ms	51.737ms	58.387ms	21.20%	8.935	88.022	1.85%	0.063ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching deserializing traces from msgpack to their internal representation	execution_time	[48.183ms; 48.432ms] or [-0.257%; +0.257%]	None	None	None

Group 6

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_trace/test_trace	execution_time	245.347ns	256.005ns ± 11.987ns	250.807ns ± 3.268ns	257.064ns	281.192ns	288.095ns	289.590ns	15.46%	1.472	0.755	4.67%	0.848ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_trace/test_trace	execution_time	[254.344ns; 257.667ns] or [-0.649%; +0.649%]	None	None	None

Group 7

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
credit_card/is_card_number/	execution_time	3.895µs	3.915µs ± 0.003µs	3.915µs ± 0.001µs	3.916µs	3.919µs	3.924µs	3.925µs	0.28%	-0.538	9.426	0.08%	0.000µs	1	200
credit_card/is_card_number/	throughput	254749802.080op/s	255442645.236op/s ± 194819.398op/s	255453402.540op/s ± 91312.433op/s	255546626.299op/s	255685853.001op/s	255752245.787op/s	256716132.267op/s	0.49%	0.563	9.560	0.08%	13775.812op/s	1	200
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	79.520µs	79.998µs ± 0.095µs	79.994µs ± 0.044µs	80.038µs	80.101µs	80.171µs	80.901µs	1.13%	3.741	42.371	0.12%	0.007µs	1	200
credit_card/is_card_number/ 3782-8224-6310-005	throughput	12360834.311op/s	12500322.955op/s ± 14746.578op/s	12500945.697op/s ± 6842.289op/s	12507834.282op/s	12515457.717op/s	12523016.311op/s	12575517.239op/s	0.60%	-3.634	41.409	0.12%	1042.741op/s	1	200
credit_card/is_card_number/ 378282246310005	execution_time	67.827µs	67.915µs ± 0.067µs	67.900µs ± 0.028µs	67.940µs	68.010µs	68.089µs	68.544µs	0.95%	4.697	37.667	0.10%	0.005µs	1	200
credit_card/is_card_number/ 378282246310005	throughput	14589239.470op/s	14724383.494op/s ± 14533.807op/s	14727577.873op/s ± 6168.888op/s	14732633.193op/s	14737837.570op/s	14740145.620op/s	14743360.432op/s	0.11%	-4.645	36.997	0.10%	1027.695op/s	1	200
credit_card/is_card_number/37828224631	execution_time	3.895µs	3.915µs ± 0.003µs	3.914µs ± 0.001µs	3.916µs	3.918µs	3.921µs	3.927µs	0.31%	-1.253	17.651	0.07%	0.000µs	1	200
credit_card/is_card_number/37828224631	throughput	254678292.654op/s	255459737.589op/s ± 171883.392op/s	255476975.216op/s ± 84256.458op/s	255557544.847op/s	255633801.406op/s	255702564.922op/s	256772061.511op/s	0.51%	1.289	17.892	0.07%	12153.991op/s	1	200
credit_card/is_card_number/378282246310005	execution_time	64.615µs	64.689µs ± 0.040µs	64.687µs ± 0.025µs	64.709µs	64.758µs	64.804µs	64.848µs	0.25%	0.840	1.261	0.06%	0.003µs	1	200
credit_card/is_card_number/378282246310005	throughput	15420760.423op/s	15458646.956op/s ± 9546.979op/s	15459159.222op/s ± 5971.548op/s	15465439.768op/s	15472033.989op/s	15475768.661op/s	15476350.589op/s	0.11%	-0.835	1.247	0.06%	675.073op/s	1	200
credit_card/is_card_number/37828224631000521389798	execution_time	45.479µs	45.714µs ± 0.098µs	45.724µs ± 0.073µs	45.780µs	45.869µs	45.905µs	45.994µs	0.59%	-0.069	-0.517	0.21%	0.007µs	1	200
credit_card/is_card_number/37828224631000521389798	throughput	21742199.465op/s	21875163.601op/s ± 46902.947op/s	21870495.610op/s ± 34873.977op/s	21910948.355op/s	21950090.021op/s	21970108.352op/s	21988192.441op/s	0.54%	0.078	-0.520	0.21%	3316.539op/s	1	200
credit_card/is_card_number/x371413321323331	execution_time	6.592µs	6.629µs ± 0.012µs	6.634µs ± 0.005µs	6.638µs	6.642µs	6.647µs	6.652µs	0.27%	-1.077	0.474	0.18%	0.001µs	1	200
credit_card/is_card_number/x371413321323331	throughput	150337537.640op/s	150844710.490op/s ± 268660.428op/s	150742742.449op/s ± 115447.097op/s	151029265.700op/s	151425624.257op/s	151578212.068op/s	151707396.419op/s	0.64%	1.084	0.492	0.18%	18997.161op/s	1	200
credit_card/is_card_number_no_luhn/	execution_time	3.898µs	3.915µs ± 0.002µs	3.914µs ± 0.001µs	3.916µs	3.918µs	3.920µs	3.921µs	0.16%	-1.713	15.349	0.06%	0.000µs	1	200
credit_card/is_card_number_no_luhn/	throughput	255068908.039op/s	255458269.335op/s ± 145135.748op/s	255465780.360op/s ± 72627.062op/s	255538873.852op/s	255629397.046op/s	255728041.672op/s	256559591.111op/s	0.43%	1.737	15.553	0.06%	10262.647op/s	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	65.596µs	65.695µs ± 0.051µs	65.688µs ± 0.031µs	65.720µs	65.796µs	65.864µs	65.897µs	0.32%	1.112	1.898	0.08%	0.004µs	1	200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	15175184.111op/s	15221904.996op/s ± 11769.235op/s	15223536.742op/s ± 7167.362op/s	15230347.814op/s	15237397.230op/s	15240882.568op/s	15244936.813op/s	0.14%	-1.106	1.878	0.08%	832.211op/s	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	53.374µs	53.439µs ± 0.035µs	53.436µs ± 0.022µs	53.456µs	53.493µs	53.561µs	53.575µs	0.26%	1.134	2.091	0.07%	0.003µs	1	200
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	18665253.667op/s	18712943.409op/s ± 12387.596op/s	18714103.202op/s ± 7543.897op/s	18721646.633op/s	18729041.561op/s	18730770.977op/s	18735764.432op/s	0.12%	-1.129	2.071	0.07%	875.935op/s	1	200
credit_card/is_card_number_no_luhn/37828224631	execution_time	3.898µs	3.915µs ± 0.002µs	3.915µs ± 0.001µs	3.917µs	3.919µs	3.921µs	3.923µs	0.19%	-1.443	12.124	0.06%	0.000µs	1	200
credit_card/is_card_number_no_luhn/37828224631	throughput	254936869.948op/s	255414077.708op/s ± 159356.379op/s	255412699.978op/s ± 91003.883op/s	255506949.789op/s	255625628.514op/s	255678140.679op/s	256556602.713op/s	0.45%	1.466	12.295	0.06%	11268.198op/s	1	200
credit_card/is_card_number_no_luhn/378282246310005	execution_time	50.165µs	50.218µs ± 0.029µs	50.216µs ± 0.018µs	50.233µs	50.272µs	50.294µs	50.343µs	0.25%	0.943	1.890	0.06%	0.002µs	1	200
credit_card/is_card_number_no_luhn/378282246310005	throughput	19863904.445op/s	19913194.277op/s ± 11614.668op/s	19914065.828op/s ± 7049.067op/s	19921169.442op/s	19929675.152op/s	19933132.096op/s	19934205.162op/s	0.10%	-0.938	1.871	0.06%	821.281op/s	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	45.414µs	45.708µs ± 0.114µs	45.720µs ± 0.088µs	45.792µs	45.885µs	45.913µs	45.963µs	0.53%	-0.215	-0.449	0.25%	0.008µs	1	200
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	21756455.259op/s	21877984.802op/s ± 54578.915op/s	21872416.695op/s ± 41835.758op/s	21918674.715op/s	21966701.890op/s	22012353.550op/s	22019792.073op/s	0.67%	0.226	-0.440	0.25%	3859.312op/s	1	200
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	6.567µs	6.631µs ± 0.011µs	6.634µs ± 0.003µs	6.637µs	6.642µs	6.645µs	6.647µs	0.18%	-2.247	7.314	0.17%	0.001µs	1	200
credit_card/is_card_number_no_luhn/x371413321323331	throughput	150453522.715op/s	150809364.941op/s ± 251771.378op/s	150727755.528op/s ± 70493.267op/s	150844528.529op/s	151249093.494op/s	151733025.307op/s	152286193.966op/s	1.03%	2.268	7.470	0.17%	17802.925op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
credit_card/is_card_number/	execution_time	[3.914µs; 3.915µs] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number/	throughput	[255415645.141op/s; 255469645.331op/s] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	execution_time	[79.985µs; 80.011µs] or [-0.016%; +0.016%]	None	None	None
credit_card/is_card_number/ 3782-8224-6310-005	throughput	[12498279.221op/s; 12502366.688op/s] or [-0.016%; +0.016%]	None	None	None
credit_card/is_card_number/ 378282246310005	execution_time	[67.905µs; 67.924µs] or [-0.014%; +0.014%]	None	None	None
credit_card/is_card_number/ 378282246310005	throughput	[14722369.248op/s; 14726397.740op/s] or [-0.014%; +0.014%]	None	None	None
credit_card/is_card_number/37828224631	execution_time	[3.914µs; 3.915µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/37828224631	throughput	[255435916.204op/s; 255483558.974op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/378282246310005	execution_time	[64.683µs; 64.694µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/378282246310005	throughput	[15457323.836op/s; 15459970.075op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number/37828224631000521389798	execution_time	[45.701µs; 45.728µs] or [-0.030%; +0.030%]	None	None	None
credit_card/is_card_number/37828224631000521389798	throughput	[21868663.303op/s; 21881663.898op/s] or [-0.030%; +0.030%]	None	None	None
credit_card/is_card_number/x371413321323331	execution_time	[6.628µs; 6.631µs] or [-0.025%; +0.025%]	None	None	None
credit_card/is_card_number/x371413321323331	throughput	[150807476.739op/s; 150881944.242op/s] or [-0.025%; +0.025%]	None	None	None
credit_card/is_card_number_no_luhn/	execution_time	[3.914µs; 3.915µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number_no_luhn/	throughput	[255438154.916op/s; 255478383.754op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	execution_time	[65.688µs; 65.702µs] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005	throughput	[15220273.894op/s; 15223536.099op/s] or [-0.011%; +0.011%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	execution_time	[53.434µs; 53.444µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/ 378282246310005	throughput	[18711226.607op/s; 18714660.210op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	execution_time	[3.915µs; 3.916µs] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631	throughput	[255391992.447op/s; 255436162.970op/s] or [-0.009%; +0.009%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	execution_time	[50.214µs; 50.222µs] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number_no_luhn/378282246310005	throughput	[19911584.596op/s; 19914803.959op/s] or [-0.008%; +0.008%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	execution_time	[45.693µs; 45.724µs] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number_no_luhn/37828224631000521389798	throughput	[21870420.689op/s; 21885548.914op/s] or [-0.035%; +0.035%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	execution_time	[6.629µs; 6.632µs] or [-0.023%; +0.023%]	None	None	None
credit_card/is_card_number_no_luhn/x371413321323331	throughput	[150774471.849op/s; 150844258.033op/s] or [-0.023%; +0.023%]	None	None	None

Group 8

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
single_flag_killswitch/rules-based	execution_time	190.123ns	192.604ns ± 2.075ns	192.383ns ± 1.331ns	193.535ns	196.817ns	199.119ns	200.066ns	3.99%	1.167	1.532	1.07%	0.147ns	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
single_flag_killswitch/rules-based	execution_time	[192.317ns; 192.892ns] or [-0.149%; +0.149%]	None	None	None

Group 9

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sdk_test_data/rules-based	execution_time	145.120µs	147.084µs ± 1.573µs	146.911µs ± 0.535µs	147.408µs	148.566µs	154.430µs	161.370µs	9.84%	5.044	37.589	1.07%	0.111µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sdk_test_data/rules-based	execution_time	[146.866µs; 147.302µs] or [-0.148%; +0.148%]	None	None	None

Group 10

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
tags/replace_trace_tags	execution_time	2.303µs	2.379µs ± 0.019µs	2.380µs ± 0.006µs	2.387µs	2.407µs	2.412µs	2.418µs	1.59%	-1.780	4.670	0.80%	0.001µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
tags/replace_trace_tags	execution_time	[2.376µs; 2.381µs] or [-0.111%; +0.111%]	None	None	None

Group 11

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	205.991µs	206.742µs ± 0.429µs	206.671µs ± 0.291µs	207.030µs	207.393µs	208.073µs	208.292µs	0.78%	0.928	1.125	0.21%	0.030µs	1	200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	4800956.870op/s	4836958.392op/s ± 10027.758op/s	4838608.539op/s ± 6807.136op/s	4843704.458op/s	4850020.524op/s	4851982.890op/s	4854574.656op/s	0.33%	-0.914	1.081	0.21%	709.070op/s	1	200
normalization/normalize_name/normalize_name/bad-name	execution_time	18.583µs	18.711µs ± 0.124µs	18.648µs ± 0.029µs	18.783µs	18.993µs	19.074µs	19.194µs	2.93%	1.560	1.863	0.66%	0.009µs	1	200
normalization/normalize_name/normalize_name/bad-name	throughput	52099844.495op/s	53447255.110op/s ± 351215.226op/s	53625796.922op/s ± 82694.782op/s	53676236.542op/s	53752837.509op/s	53788724.894op/s	53813532.462op/s	0.35%	-1.532	1.737	0.66%	24834.667op/s	1	200
normalization/normalize_name/normalize_name/good	execution_time	10.686µs	10.779µs ± 0.048µs	10.771µs ± 0.032µs	10.808µs	10.874µs	10.909µs	10.918µs	1.37%	0.639	0.034	0.45%	0.003µs	1	200
normalization/normalize_name/normalize_name/good	throughput	91594748.475op/s	92776135.743op/s ± 413056.143op/s	92845904.166op/s ± 279685.649op/s	93079900.233op/s	93332846.518op/s	93551133.990op/s	93576374.751op/s	0.79%	-0.618	-0.000	0.44%	29207.480op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	execution_time	[206.683µs; 206.802µs] or [-0.029%; +0.029%]	None	None	None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...	throughput	[4835568.641op/s; 4838348.142op/s] or [-0.029%; +0.029%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	execution_time	[18.694µs; 18.728µs] or [-0.092%; +0.092%]	None	None	None
normalization/normalize_name/normalize_name/bad-name	throughput	[53398580.057op/s; 53495930.162op/s] or [-0.091%; +0.091%]	None	None	None
normalization/normalize_name/normalize_name/good	execution_time	[10.772µs; 10.786µs] or [-0.062%; +0.062%]	None	None	None
normalization/normalize_name/normalize_name/good	throughput	[92718890.134op/s; 92833381.352op/s] or [-0.062%; +0.062%]	None	None	None

Group 12

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
profile_add_sample_frames_x1000	execution_time	4.212ms	4.218ms ± 0.007ms	4.217ms ± 0.001ms	4.219ms	4.221ms	4.224ms	4.305ms	2.09%	11.695	151.933	0.16%	0.000ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
profile_add_sample_frames_x1000	execution_time	[4.217ms; 4.219ms] or [-0.022%; +0.022%]	None	None	None

Group 13

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching string interning on wordpress profile	execution_time	160.395µs	161.044µs ± 0.305µs	160.977µs ± 0.151µs	161.180µs	161.537µs	161.831µs	163.079µs	1.31%	2.043	9.411	0.19%	0.022µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching string interning on wordpress profile	execution_time	[161.002µs; 161.087µs] or [-0.026%; +0.026%]	None	None	None

Group 14

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
write only interface	execution_time	1.237µs	3.172µs ± 1.443µs	2.999µs ± 0.025µs	3.020µs	3.367µs	13.928µs	15.233µs	407.94%	7.488	56.682	45.37%	0.102µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
write only interface	execution_time	[2.972µs; 3.372µs] or [-6.304%; +6.304%]	None	None	None

Group 15

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
benching serializing traces from their internal representation to msgpack	execution_time	14.662ms	14.707ms ± 0.028ms	14.703ms ± 0.012ms	14.715ms	14.750ms	14.815ms	14.872ms	1.15%	2.363	8.515	0.19%	0.002ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
benching serializing traces from their internal representation to msgpack	execution_time	[14.703ms; 14.711ms] or [-0.027%; +0.027%]	None	None	None

Group 16

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
receiver_entry_point/report/2598	execution_time	3.385ms	3.411ms ± 0.014ms	3.408ms ± 0.008ms	3.418ms	3.437ms	3.448ms	3.455ms	1.39%	0.915	0.717	0.40%	0.001ms	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
receiver_entry_point/report/2598	execution_time	[3.409ms; 3.413ms] or [-0.055%; +0.055%]	None	None	None

Group 17

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
redis/obfuscate_redis_string	execution_time	32.473µs	33.556µs ± 1.455µs	32.623µs ± 0.070µs	35.232µs	36.171µs	36.509µs	37.012µs	13.45%	0.982	-0.876	4.32%	0.103µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
redis/obfuscate_redis_string	execution_time	[33.354µs; 33.757µs] or [-0.601%; +0.601%]	None	None	None

Group 18

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
sql/obfuscate_sql_string	execution_time	89.279µs	89.611µs ± 0.150µs	89.572µs ± 0.068µs	89.697µs	89.860µs	89.912µs	90.657µs	1.21%	1.999	10.599	0.17%	0.011µs	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
sql/obfuscate_sql_string	execution_time	[89.590µs; 89.632µs] or [-0.023%; +0.023%]	None	None	None

Group 19

cpu_model	git_commit_sha	git_commit_date	git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	`49ff857`	1773406973	julio/rd-week-code-coverage

scenario	metric	min	mean ± sd	median ± mad	p75	p95	p99	max	peak_to_median_ratio	skewness	kurtosis	cv	sem	runs	sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	493.273µs	494.117µs ± 0.670µs	494.061µs ± 0.290µs	494.358µs	494.737µs	495.096µs	501.803µs	1.57%	7.604	84.784	0.14%	0.047µs	1	200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	1992812.148op/s	2023817.240op/s ± 2716.403op/s	2024041.680op/s ± 1188.496op/s	2025159.684op/s	2026330.704op/s	2027117.143op/s	2027274.044op/s	0.16%	-7.487	83.005	0.13%	192.079op/s	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	370.026µs	370.661µs ± 0.301µs	370.626µs ± 0.194µs	370.839µs	371.194µs	371.528µs	371.584µs	0.26%	0.629	0.374	0.08%	0.021µs	1	200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	2691185.000op/s	2697881.679op/s ± 2186.824op/s	2698135.688op/s ± 1414.370op/s	2699469.842op/s	2700927.565op/s	2702198.205op/s	2702511.844op/s	0.16%	-0.624	0.366	0.08%	154.632op/s	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	167.326µs	167.565µs ± 0.123µs	167.560µs ± 0.086µs	167.643µs	167.788µs	167.939µs	167.957µs	0.24%	0.519	0.290	0.07%	0.009µs	1	200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	5953907.342op/s	5967824.826op/s ± 4371.967op/s	5968007.731op/s ± 3057.674op/s	5971113.420op/s	5974439.362op/s	5975776.897op/s	5976347.562op/s	0.14%	-0.515	0.282	0.07%	309.145op/s	1	200
normalization/normalize_service/normalize_service/[empty string]	execution_time	37.023µs	37.162µs ± 0.058µs	37.162µs ± 0.038µs	37.202µs	37.253µs	37.269µs	37.289µs	0.34%	-0.147	-0.498	0.16%	0.004µs	1	200
normalization/normalize_service/normalize_service/[empty string]	throughput	26817593.784op/s	26909560.238op/s ± 41909.654op/s	26908909.801op/s ± 27390.812op/s	26935628.102op/s	26982017.553op/s	27002223.733op/s	27010130.220op/s	0.38%	0.154	-0.496	0.16%	2963.460op/s	1	200
normalization/normalize_service/normalize_service/test_ASCII	execution_time	45.448µs	45.620µs ± 0.062µs	45.626µs ± 0.039µs	45.658µs	45.707µs	45.750µs	45.926µs	0.66%	0.305	2.343	0.13%	0.004µs	1	200
normalization/normalize_service/normalize_service/test_ASCII	throughput	21774021.463op/s	21920434.638op/s ± 29634.670op/s	21917530.664op/s ± 18748.044op/s	21939501.945op/s	21970732.470op/s	21985149.748op/s	22002974.802op/s	0.39%	-0.288	2.280	0.13%	2095.488op/s	1	200

scenario	metric	95% CI mean	Shapiro-Wilk pvalue	Ljung-Box pvalue (lag=1)	Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	execution_time	[494.024µs; 494.210µs] or [-0.019%; +0.019%]	None	None	None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...	throughput	[2023440.773op/s; 2024193.708op/s] or [-0.019%; +0.019%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	execution_time	[370.620µs; 370.703µs] or [-0.011%; +0.011%]	None	None	None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて	throughput	[2697578.606op/s; 2698184.752op/s] or [-0.011%; +0.011%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	execution_time	[167.548µs; 167.582µs] or [-0.010%; +0.010%]	None	None	None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters	throughput	[5967218.913op/s; 5968430.738op/s] or [-0.010%; +0.010%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	execution_time	[37.154µs; 37.170µs] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/[empty string]	throughput	[26903751.963op/s; 26915368.513op/s] or [-0.022%; +0.022%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	execution_time	[45.611µs; 45.628µs] or [-0.019%; +0.019%]	None	None	None
normalization/normalize_service/normalize_service/test_ASCII	throughput	[21916327.558op/s; 21924541.718op/s] or [-0.019%; +0.019%]	None	None	None

Baseline

Omitted due to size.

dd-octo-sts · 2026-03-06T11:04:57Z

Artifact Size Benchmark Report

aarch64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a	100.37 MB	100.37 MB	0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so	8.70 MB	8.70 MB	0% (0 B) 👌

aarch64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a	117.05 MB	117.05 MB	0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.22 MB	11.22 MB	0% (0 B) 👌

libdatadog-x64-windows

Artifact	Baseline	Commit	Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll	27.18 MB	27.18 MB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib	76.61 KB	76.61 KB	0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb	186.17 MB	186.13 MB	--.02% (-40.00 KB) 💪
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib	917.36 MB	917.36 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll	9.94 MB	9.94 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib	76.61 KB	76.61 KB	0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb	24.78 MB	24.78 MB	0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib	51.46 MB	51.46 MB	0% (0 B) 👌

libdatadog-x86-windows

Artifact	Baseline	Commit	Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll	22.98 MB	22.98 MB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib	77.80 KB	77.80 KB	0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb	190.34 MB	190.30 MB	--.02% (-40.00 KB) 💪
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib	901.01 MB	901.01 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll	7.54 MB	7.54 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib	77.80 KB	77.80 KB	0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb	26.53 MB	26.53 MB	0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib	47.08 MB	47.08 MB	0% (0 B) 👌

x86_64-alpine-linux-musl

Artifact	Baseline	Commit	Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a	87.58 MB	87.58 MB	0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so	10.22 MB	10.22 MB	0% (0 B) 👌

x86_64-unknown-linux-gnu

Artifact	Baseline	Commit	Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a	109.91 MB	109.91 MB	0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so	11.96 MB	11.96 MB	0% (0 B) 👌

hoolioh changed the base branch from main to julio/rd-week-dynamic-pipeline March 6, 2026 10:21

hoolioh changed the title ~~Julio/rd week code coverage~~ ci: make code coverage dynamic Mar 6, 2026

github-actions bot added the ci-build label Mar 6, 2026

hoolioh force-pushed the julio/rd-week-code-coverage branch from ed49658 to 9f46bc5 Compare March 13, 2026 12:18

hoolioh added 4 commits March 13, 2026 14:00

chore: clean up actions

56bf633

don't run coverage if crates haven't changed

2f32c18

ignore release branches

f8577fe

Modify coverage so it generates report for modified crates

49ff857

hoolioh force-pushed the julio/rd-week-code-coverage branch from 9f46bc5 to 49ff857 Compare March 13, 2026 13:03

hoolioh force-pushed the julio/rd-week-dynamic-pipeline branch 2 times, most recently from 91b03cc to a434e47 Compare March 13, 2026 13:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: make code coverage dynamic#1677

ci: make code coverage dynamic#1677
hoolioh wants to merge 4 commits intojulio/rd-week-dynamic-pipelinefrom
julio/rd-week-code-coverage

hoolioh commented Mar 6, 2026

Uh oh!

pr-commenter bot commented Mar 6, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

Group 14

Group 15

Group 16

Group 17

Group 18

Group 19

Uh oh!

dd-octo-sts bot commented Mar 6, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hoolioh commented Mar 6, 2026

What does this PR do?

Motivation

Additional Notes

How to test the change?

Uh oh!

pr-commenter bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Comparison

Explanation

More details about the CI and significant changes

Candidate

Group 1

Group 2

Group 3

Group 4

Group 5

Group 6

Group 7

Group 8

Group 9

Group 10

Group 11

Group 12

Group 13

Group 14

Group 15

Group 16

Group 17

Group 18

Group 19

Baseline

Uh oh!

dd-octo-sts bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Artifact Size Benchmark Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

pr-commenter bot commented Mar 6, 2026 •

edited

Loading

dd-octo-sts bot commented Mar 6, 2026 •

edited

Loading