Skip to content

Serialize X-Ray and EMF request bodies directly - #374

Merged
mheffner merged 1 commit into
rotel-dev:mainfrom
mheffner:perf2-json-serialize
Oct 30, 2026
Merged

mheffner merged 1 commit into
rotel-dev:mainfrom
mheffner:perf2-json-serialize

Conversation

@mheffner

Copy link
Copy Markdown
Member

Both exporters built their request bodies by formatting serde_json::Value trees through Display (which goes through a fmt::Write adapter), after first copying already-formatted strings into new Values:

  • X-Ray: each segment Value was formatted with to_string(), the strings were copied into json!({"TraceSegmentDocuments": [...]}) and that Value was formatted again.
  • EMF: each EMF log line Value was formatted with to_string(), then every event (timestamp + message) was copied into a json! object for the PutLogEvents body, which was formatted again.

In profiles at saturation Value's Display impl was ~20% of CPU in both.

Use serde_json::to_string / to_vec, and serialize the request bodies from small borrowing structs. Field order matches what the Value-based code produced (serde_json objects are key-sorted), so the bodies are byte for byte the same; tests compare against the previous construction.

Benchmark (rotel-bench, Intel N95, mock endpoints, 5+5 runs ABBA, mean +- stddev, vs main):
EMF, 20k points/s: CPU/point -16.4% (10.8 -> 9.0 %/10k/s)
EMF, saturation (1 core): export requests/s 34.7 -> 45.3 (+30.7%),
CPU/point -24.6%
X-Ray, 50k spans/s: CPU/span -7.8% (11.0 -> 10.1 %/10k/s)
X-Ray, saturation: export requests/s 2,454 -> 2,840 (+15.7%),
client p99 363 -> 314 ms
(at saturation both exporters export slower than rotel ingests, so the
export rate is measured at the endpoint)

Both exporters built their request bodies by formatting serde_json::Value
trees through Display (which goes through a fmt::Write adapter), after
first copying already-formatted strings into new Values:

- X-Ray: each segment Value was formatted with to_string(), the strings
  were copied into json!({"TraceSegmentDocuments": [...]}) and that Value
  was formatted again.
- EMF: each EMF log line Value was formatted with to_string(), then every
  event (timestamp + message) was copied into a json! object for the
  PutLogEvents body, which was formatted again.

In profiles at saturation Value's Display impl was ~20% of CPU in both.

Use serde_json::to_string / to_vec, and serialize the request bodies from
small borrowing structs. Field order matches what the Value-based code
produced (serde_json objects are key-sorted), so the bodies are byte for
byte the same; tests compare against the previous construction.

Benchmark (rotel-bench, Intel N95, mock endpoints, 5+5 runs ABBA,
mean +- stddev, vs main):
  EMF, 20k points/s:        CPU/point -16.4% (10.8 -> 9.0 %/10k/s)
  EMF, saturation (1 core):  export requests/s 34.7 -> 45.3 (+30.7%),
                             CPU/point -24.6%
  X-Ray, 50k spans/s:       CPU/span -7.8% (11.0 -> 10.1 %/10k/s)
  X-Ray, saturation:        export requests/s 2,454 -> 2,840 (+15.7%),
                            client p99 363 -> 314 ms
  (at saturation both exporters export slower than rotel ingests, so the
  export rate is measured at the endpoint)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mheffner
mheffner merged commit 0e3daef into rotel-dev:main Oct 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant