Repository navigation
fix(distributed): carry which latency phases a worker actually measured - #294
Merged
Merged
Conversation
LatencyBreakdown.available was not serialized, so the master rebuilt every breakdown with the default full phase set. aggregate_breakdowns intersects available to omit phases a backend cannot measure -- so a --http2 --latency-breakdown distributed run reported DNS, TCP and TLS as 0.0ms averages over every sample, which reads as an impossibly fast connection rather than an unknown one. Single-node --http2 got this right; only the cluster path lied. available now travels with each sample and is restored on the master. A worker from an older release does not send the key, and the full set is what it would have meant, so a mixed-version cluster still merges. Closes #242 Signed-off-by: Yurii Riabikov <22548029+kurok@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #242
The bug
LatencyBreakdown.availablewas not serialized, so the master rebuilt every breakdown with the default full phase set.aggregate_breakdownsintersectsavailableto omit phases a backend cannot measure — so a--http2 --latency-breakdowndistributed run reported:Five samples of a measured-zero connection phase that never happened. That reads as an impossibly fast connection rather than an unknown one — which is exactly the failure mode the
availablemechanism was built to prevent.Single-node
--http2got this right. Only the cluster path lied.The fix
availabletravels with each sample and is restored on the master.A worker from an older release does not send the key. The full phase set is what it would have meant, so a mixed-version cluster still merges rather than silently dropping every phase. A malformed
availableis rejected by name, consistent with the rest of_deserialize_statsafter #237.Tests
tests/test_distributed.py::TestBreakdownAvailableRoundTrip, all four verified against the unpatched source:test_breakdown_available_round_trips— the issue's named case. Pre-fix:Tuples differ: ('dns', 'connect', 'tls', 'ttfb', 'transfer', 'total') != ('ttfb', 'transfer', 'total').test_an_httpx_worker_does_not_contribute_phantom_connection_phases— runs the restored samples throughaggregate_breakdownsand assertsdns/connect/tlsare absent. This is the one that tests the behaviour rather than the field: pre-fix it prints the phantom{'avg': 0.0, 'count': 5}blocks in the failure message.test_a_worker_that_does_not_send_available_means_the_full_set— the mixed-version path.test_a_malformed_available_is_named.Checks
ruff format/check clean, mypy clean,
1928 passed.