Skip to content

Fix latency increase when the application idles between RPCs - #123

Open
Stuart0l wants to merge 1 commit into
erpc-io:masterfrom
Stuart0l:issue116-fix
Open

Fix latency increase when the application idles between RPCs#123
Stuart0l wants to merge 1 commit into
erpc-io:masterfrom
Stuart0l:issue116-fix

Conversation

@Stuart0l

@Stuart0l Stuart0l commented Aug 6, 2026

Copy link
Copy Markdown

Fixes #116 (and #104). Root-cause analysis is in #116.

Changes

1. Bound the timing wheel's catch-up (src/cc/timing_wheel.h)

Added a num_wheel_entries_ counter to TimingWheel, maintained in insert_into_wslot() and reap_wslot(). reap() uses it to detect an empty wheel: when the wheel is empty and the lag exceeds one full revolution, it re-bases all slot timestamps in a single pass and returns, capping the cost at kWheelNumWslots iterations. The existing loop is untouched, so sessions actually using the wheel take the same path as before.

Note: This bounds rather than eliminates the catch-up. Since each slot stores its own deadline tx_tsc_, skipped slots would go stale. We still need to update tx_tsc_ once for each slot.

2. Fresh timestamp for progress_tsc_ (src/rpc_impl/rpc_req.cc)

enqueue_request() now stamps progress_tsc_ with dpath_rdtsc() instead of ev_loop_tsc_, so a newly enqueued request is never seen as already timed out by the packet-loss scan. Costs one rdtsc (~8 ns) per enqueue_request.

Evaluation

Two-node CloudLab xl170 cluster, RoCE 25 Gbps, 16-byte request/response, one outstanding RPC. Client enqueues, polls to completion, sleeps for wait. Mean end-to-end latency; retransmissions went from one per RPC to zero at every wait above the RTO.

wait before after
0 4.68 µs 4.70 µs
10 ms 386 µs 41 µs
100 ms 3,690 µs 38 µs 97×
1 s 36,625 µs 56 µs 654×
5 s 75,790 µs 60 µs 1,270×

Latency is now flat in the idle period rather than linear. reap() in isolation (no NIC, empty wheel) drops from 149 ms to 24 µs after a 10 s idle, and is flat ~10–25 µs across the range. No change on the zero-wait path that §6.2's overhead numbers measure.

Testing

All relevant unit tests pass (PERF=OFF): timing_wheel_test (including TimingWheelRateTest — pacing still achieves its 1/5/10/20/40 Gbps targets), packet_loss_test, rpc_kick/req/cr/rfr/resp_test, small_msg_test, large_msg_test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Waiting in-between RPCs increase latency

1 participant