Skip to content

fix(distributed): carry which latency phases a worker actually measured - #294

Merged
kurok merged 1 commit into
mainfrom
fix/breakdown-available-wire
Sep 19, 2026
Merged

kurok merged 1 commit into
mainfrom
fix/breakdown-available-wire

Conversation

@kurok

@kurok kurok commented Sep 19, 2026 •

Copy link
Copy Markdown
Owner

Closes #242

The bug

LatencyBreakdown.available was not serialized, so the master rebuilt every breakdown with the default full phase set. aggregate_breakdowns intersects available to omit phases a backend cannot measure — so a --http2 --latency-breakdown distributed run reported:

dns:     {avg: 0.0, min: 0.0, max: 0.0, p50: 0.0, p95: 0.0, count: 5}
connect: {avg: 0.0, ...}
tls:     {avg: 0.0, ...}

Five samples of a measured-zero connection phase that never happened. That reads as an impossibly fast connection rather than an unknown one — which is exactly the failure mode the available mechanism was built to prevent.

Single-node --http2 got this right. Only the cluster path lied.

The fix

available travels with each sample and is restored on the master.

A worker from an older release does not send the key. The full phase set is what it would have meant, so a mixed-version cluster still merges rather than silently dropping every phase. A malformed available is rejected by name, consistent with the rest of _deserialize_stats after #237.

Tests

tests/test_distributed.py::TestBreakdownAvailableRoundTrip, all four verified against the unpatched source:

  • test_breakdown_available_round_trips — the issue's named case. Pre-fix: Tuples differ: ('dns', 'connect', 'tls', 'ttfb', 'transfer', 'total') != ('ttfb', 'transfer', 'total').
  • test_an_httpx_worker_does_not_contribute_phantom_connection_phases — runs the restored samples through aggregate_breakdowns and asserts dns/connect/tls are absent. This is the one that tests the behaviour rather than the field: pre-fix it prints the phantom {'avg': 0.0, 'count': 5} blocks in the failure message.
  • test_a_worker_that_does_not_send_available_means_the_full_set — the mixed-version path.
  • test_a_malformed_available_is_named.

Checks

ruff format/check clean, mypy clean, 1928 passed.

LatencyBreakdown.available was not serialized, so the master rebuilt
every breakdown with the default full phase set. aggregate_breakdowns
intersects available to omit phases a backend cannot measure -- so a
--http2 --latency-breakdown distributed run reported DNS, TCP and TLS as
0.0ms averages over every sample, which reads as an impossibly fast
connection rather than an unknown one. Single-node --http2 got this
right; only the cluster path lied.

available now travels with each sample and is restored on the master. A
worker from an older release does not send the key, and the full set is
what it would have meant, so a mixed-version cluster still merges.

Closes #242

Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@kurok
kurok merged commit b37d901 into main Sep 19, 2026
18 checks passed
@kurok
kurok deleted the fix/breakdown-available-wire branch September 19, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(distributed): wire format drops LatencyBreakdown.available, httpx workers' zero phases get averaged in

1 participant