Fix latency increase when the application idles between RPCs - #123
Open
Stuart0l wants to merge 1 commit into
Open
Fix latency increase when the application idles between RPCs#123Stuart0l wants to merge 1 commit into
Stuart0l wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #116 (and #104). Root-cause analysis is in #116.
Changes
1. Bound the timing wheel's catch-up (
src/cc/timing_wheel.h)Added a
num_wheel_entries_counter toTimingWheel, maintained ininsert_into_wslot()andreap_wslot().reap()uses it to detect an empty wheel: when the wheel is empty and the lag exceeds one full revolution, it re-bases all slot timestamps in a single pass and returns, capping the cost atkWheelNumWslotsiterations. The existing loop is untouched, so sessions actually using the wheel take the same path as before.Note: This bounds rather than eliminates the catch-up. Since each slot stores its own deadline
tx_tsc_, skipped slots would go stale. We still need to updatetx_tsc_once for each slot.2. Fresh timestamp for
progress_tsc_(src/rpc_impl/rpc_req.cc)enqueue_request()now stampsprogress_tsc_withdpath_rdtsc()instead ofev_loop_tsc_, so a newly enqueued request is never seen as already timed out by the packet-loss scan. Costs onerdtsc(~8 ns) perenqueue_request.Evaluation
Two-node CloudLab xl170 cluster, RoCE 25 Gbps, 16-byte request/response, one outstanding RPC. Client enqueues, polls to completion, sleeps for
wait. Mean end-to-end latency; retransmissions went from one per RPC to zero at every wait above the RTO.Latency is now flat in the idle period rather than linear.
reap()in isolation (no NIC, empty wheel) drops from 149 ms to 24 µs after a 10 s idle, and is flat ~10–25 µs across the range. No change on the zero-wait path that §6.2's overhead numbers measure.Testing
All relevant unit tests pass (
PERF=OFF):timing_wheel_test(includingTimingWheelRateTest— pacing still achieves its 1/5/10/20/40 Gbps targets),packet_loss_test,rpc_kick/req/cr/rfr/resp_test,small_msg_test,large_msg_test.