Skip to content

smite: re-arm TCP_QUICKACK after each send - #277

Merged
morehouse merged 1 commit into
lnfuzz:masterfrom
erickcestari:noise-quickack
Oct 8, 2026
Merged

smite: re-arm TCP_QUICKACK after each send#277
morehouse merged 1 commit into
lnfuzz:masterfrom
erickcestari:noise-quickack

Conversation

@erickcestari

@erickcestari erickcestari commented Oct 3, 2026

Copy link
Copy Markdown
Member

LDK doesn't set TCP_NODELAY, so it holds a small reply until we ACK its last segment, and Linux delays that ACK ~40 ms. This stalled the final pong of every LDK funding execution; fixing it takes LDK from 13.3 to 42.9 execs/s under Nyx.

TCP_QUICKACK is linux only. The others target kept the same exec/s.

For more info: https://catonmat.net/tcp-quickack

Bench: 
================================================================================
BEFORE (master 8d53f33)
================================================================================

$ smitebot bench-exec ldk.toml --no-build --cpu 3 -n 500 -r 5 --input funding-ldk.bin

Smite Nyx benchmark
  target:      ldk/ir
  sharedir:    /home/erick/smite-bench-quickack/sharedir/base/ldk
  input size:  406 bytes
  input buffer:1048576 bytes
  iterations:  500
  repeats:     5

  run 1/5:
  boot + snapshot creation: 3.866 s
  steady-state (snapshot restore + target run):
    execs/sec: 13.4
    exec time: 37.193 s
    latency:   min 65.64 ms  mean 74.39 ms  median 73.58 ms  p99 80.85 ms  max 89.42 ms
    input execution: mean 71.83 ms  median 70.98 ms   (guest runtime)
    nyx overhead:    mean 2.55 ms  median 2.55 ms   (restore + reset + ipc; 2295 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 82.9% stable (3137 / 18336 edges fluctuated)

  run 2/5:
  boot + snapshot creation: 3.827 s
  steady-state (snapshot restore + target run):
    execs/sec: 13.3
    exec time: 37.530 s
    latency:   min 64.68 ms  mean 75.06 ms  median 75.76 ms  p99 79.25 ms  max 96.18 ms
    input execution: mean 72.46 ms  median 73.11 ms   (guest runtime)
    nyx overhead:    mean 2.60 ms  median 2.61 ms   (restore + reset + ipc; 2350 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 89.8% stable (1863 / 18314 edges fluctuated)

  run 3/5:
  boot + snapshot creation: 3.795 s
  steady-state (snapshot restore + target run):
    execs/sec: 13.3
    exec time: 37.562 s
    latency:   min 66.04 ms  mean 75.12 ms  median 75.39 ms  p99 77.23 ms  max 90.05 ms
    input execution: mean 72.45 ms  median 72.70 ms   (guest runtime)
    nyx overhead:    mean 2.68 ms  median 2.63 ms   (restore + reset + ipc; 2335 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 96.7% stable (604 / 18522 edges fluctuated)

  run 4/5:
  boot + snapshot creation: 3.801 s
  steady-state (snapshot restore + target run):
    execs/sec: 13.4
    exec time: 37.321 s
    latency:   min 67.29 ms  mean 74.64 ms  median 73.39 ms  p99 77.65 ms  max 91.57 ms
    input execution: mean 72.03 ms  median 70.80 ms   (guest runtime)
    nyx overhead:    mean 2.61 ms  median 2.58 ms   (restore + reset + ipc; 2294 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 82.2% stable (3253 / 18316 edges fluctuated)

  run 5/5:
  boot + snapshot creation: 3.878 s
  steady-state (snapshot restore + target run):
    execs/sec: 13.3
    exec time: 37.707 s
    latency:   min 67.49 ms  mean 75.41 ms  median 76.13 ms  p99 77.71 ms  max 92.35 ms
    input execution: mean 72.82 ms  median 73.46 ms   (guest runtime)
    nyx overhead:    mean 2.59 ms  median 2.59 ms   (restore + reset + ipc; 2299 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 81.6% stable (3366 / 18342 edges fluctuated)

  aggregate over 5 runs:
    execs/sec: mean 13.3  stddev 0.1  min 13.3  max 13.4
    boot + snapshot: mean 3.833 s
    latency (mean across runs):
      min 66.23 ms  mean 74.92 ms  median 74.85 ms  p99 78.54 ms  max 91.92 ms
    input execution (mean across runs): mean 72.32 ms  median 72.21 ms
    nyx overhead (mean across runs):     mean 2.61 ms  median 2.59 ms
    failed iterations: 0 / 2500
    coverage determinism: 86.7% stable (12223 / 91830 edges fluctuated, summed over runs)

Per-step timing of one execution (local mode, RUST_LOG=debug):

INFO  [smite::runners] Reading input from "/input.bin"
DEBUG [smite_scenarios::scenarios::ir] [3.996µs] Executing IR program (40 instructions, 406 input bytes)
DEBUG [smite_scenarios::executor] [113.824µs] SendOpenChannel: 327 bytes
DEBUG [smite_scenarios::executor] [118.87µs] RecvAcceptChannel: waiting
DEBUG [smite_scenarios::executor] [1.241208ms] RecvAcceptChannel: received
DEBUG [smite_scenarios::executor] [6.388257ms] SendFundingCreated: 132 bytes
DEBUG [smite_scenarios::executor] [6.4026ms] RecvFundingSigned: waiting
DEBUG [smite_scenarios::executor] [7.842326ms] RecvFundingSigned: received
DEBUG [smite_scenarios::executor] [8.336964ms] BroadcastTransaction: txid=a12c138f7c1197af768f4280e79b2f3459949b754ca4ec7fe03e57f145a9b19c
DEBUG [smite_scenarios::executor] [24.688461ms] MineBlocks: mined 8 block(s)
DEBUG [smite_scenarios::executor] [28.120938ms] LookupShortChannelId: txid=a12c138f7c1197af768f4280e79b2f3459949b754ca4ec7fe03e57f145a9b19c scid=102x1x0
DEBUG [smite_scenarios::executor] [28.141658ms] SendChannelReady: 67 bytes
DEBUG [smite_scenarios::executor] [30.100752ms] RecvChannelReady: waiting
DEBUG [smite_scenarios::executor] [36.037821ms] RecvChannelReady: received
DEBUG [smite_scenarios::scenarios::ir] [36.054386ms] Program executed successfully
DEBUG [smite_scenarios::scenarios::ir] [76.476167ms] Target responded with pong
INFO  [smite::scenarios] Test case ran successfully!

================================================================================
AFTER (TCP_QUICKACK)
================================================================================

$ smitebot bench-exec ldk.toml --no-build --cpu 3 -n 500 -r 5 --input funding-ldk.bin

Smite Nyx benchmark
  target:      ldk/ir
  sharedir:    /home/erick/smite-bench-quickack/sharedir/quick/ldk
  input size:  406 bytes
  input buffer:1048576 bytes
  iterations:  500
  repeats:     5

  run 1/5:
  boot + snapshot creation: 3.836 s
  steady-state (snapshot restore + target run):
    execs/sec: 41.5
    exec time: 12.046 s
    latency:   min 23.06 ms  mean 24.09 ms  median 23.97 ms  p99 26.30 ms  max 28.43 ms
    input execution: mean 23.08 ms  median 22.89 ms   (guest runtime)
    nyx overhead:    mean 1.01 ms  median 988.5 µs   (restore + reset + ipc; 2307 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 95.6% stable (808 / 18564 edges fluctuated)

  run 2/5:
  boot + snapshot creation: 3.812 s
  steady-state (snapshot restore + target run):
    execs/sec: 42.6
    exec time: 11.735 s
    latency:   min 22.68 ms  mean 23.47 ms  median 23.20 ms  p99 25.41 ms  max 27.09 ms
    input execution: mean 22.47 ms  median 22.19 ms   (guest runtime)
    nyx overhead:    mean 1.00 ms  median 957.8 µs   (restore + reset + ipc; 2344 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 95.9% stable (738 / 18202 edges fluctuated)

  run 3/5:
  boot + snapshot creation: 3.811 s
  steady-state (snapshot restore + target run):
    execs/sec: 41.3
    exec time: 12.105 s
    latency:   min 23.07 ms  mean 24.21 ms  median 24.05 ms  p99 26.59 ms  max 28.36 ms
    input execution: mean 23.17 ms  median 22.99 ms   (guest runtime)
    nyx overhead:    mean 1.04 ms  median 1.02 ms   (restore + reset + ipc; 2366 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 96.5% stable (645 / 18328 edges fluctuated)

  run 4/5:
  boot + snapshot creation: 3.847 s
  steady-state (snapshot restore + target run):
    execs/sec: 41.6
    exec time: 12.005 s
    latency:   min 22.97 ms  mean 24.01 ms  median 23.89 ms  p99 27.12 ms  max 28.91 ms
    input execution: mean 23.00 ms  median 22.81 ms   (guest runtime)
    nyx overhead:    mean 1.02 ms  median 1.01 ms   (restore + reset + ipc; 2295 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 95.8% stable (777 / 18489 edges fluctuated)

  run 5/5:
  boot + snapshot creation: 3.803 s
  steady-state (snapshot restore + target run):
    execs/sec: 42.5
    exec time: 11.778 s
    latency:   min 23.13 ms  mean 23.56 ms  median 23.46 ms  p99 24.85 ms  max 26.88 ms
    input execution: mean 22.59 ms  median 22.51 ms   (guest runtime)
    nyx overhead:    mean 970.3 µs  median 953.8 µs   (restore + reset + ipc; 2364 dirty pages/exec)
    failed iterations: 0 / 500
    coverage determinism: 96.1% stable (731 / 18539 edges fluctuated)

  aggregate over 5 runs:
    execs/sec: mean 41.9  stddev 0.6  min 41.3  max 42.6
    boot + snapshot: mean 3.822 s
    latency (mean across runs):
      min 22.98 ms  mean 23.87 ms  median 23.71 ms  p99 26.05 ms  max 27.94 ms
    input execution (mean across runs): mean 22.86 ms  median 22.68 ms
    nyx overhead (mean across runs):     mean 1.01 ms  median 985.5 µs
    failed iterations: 0 / 2500
    coverage determinism: 96.0% stable (3699 / 92122 edges fluctuated, summed over runs)
    
Per-step timing of one execution (local mode, RUST_LOG=debug):

INFO  [smite::runners] Reading input from "/input.bin"
DEBUG [smite_scenarios::scenarios::ir] [4.395µs] Executing IR program (40 instructions, 406 input bytes)
DEBUG [smite_scenarios::executor] [141.657µs] SendOpenChannel: 327 bytes
DEBUG [smite_scenarios::executor] [151.992µs] RecvAcceptChannel: waiting
DEBUG [smite_scenarios::executor] [1.007632ms] RecvAcceptChannel: received
DEBUG [smite_scenarios::executor] [6.273395ms] SendFundingCreated: 132 bytes
DEBUG [smite_scenarios::executor] [6.31376ms] RecvFundingSigned: waiting
DEBUG [smite_scenarios::executor] [7.347959ms] RecvFundingSigned: received
DEBUG [smite_scenarios::executor] [7.671519ms] BroadcastTransaction: txid=643f0476a14aad978531c4d85ae3eea2f22475a143d9958566cf99ff7dd7d452
DEBUG [smite_scenarios::executor] [15.75045ms] MineBlocks: mined 8 block(s)
DEBUG [smite_scenarios::executor] [19.475696ms] LookupShortChannelId: txid=643f0476a14aad978531c4d85ae3eea2f22475a143d9958566cf99ff7dd7d452 scid=102x1x0
DEBUG [smite_scenarios::executor] [19.493837ms] SendChannelReady: 67 bytes
DEBUG [smite_scenarios::executor] [20.999596ms] RecvChannelReady: waiting
DEBUG [smite_scenarios::executor] [26.551994ms] RecvChannelReady: received
DEBUG [smite_scenarios::scenarios::ir] [26.56376ms] Program executed successfully
DEBUG [smite_scenarios::scenarios::ir] [26.7106ms] Target responded with pong
INFO  [smite::scenarios] Test case ran successfully!
Input 
v0 = LoadPrivateKey(0x2121212121212121212121212121212121212121212121212121212121212121)
v1 = DerivePoint(v0)
v2 = LoadPrivateKey(0x2222222222222222222222222222222222222222222222222222222222222222)
v3 = DerivePoint(v2)
v4 = LoadPrivateKey(0x2323232323232323232323232323232323232323232323232323232323232323)
v5 = DerivePoint(v4)
v6 = LoadPrivateKey(0x2424242424242424242424242424242424242424242424242424242424242424)
v7 = DerivePoint(v6)
v8 = LoadPrivateKey(0x2525252525252525252525252525252525252525252525252525252525252525)
v9 = DerivePoint(v8)
v10 = LoadPrivateKey(0x2626262626262626262626262626262626262626262626262626262626262626)
v11 = DerivePoint(v10)
v12 = LoadChainHashFromContext()
v13 = LoadChannelId(0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb)
v14 = LoadAmount(1000000)
v15 = LoadAmount(0)
v16 = LoadAmount(546)
v17 = LoadAmount(500000000)
v18 = LoadAmount(10000)
v19 = LoadAmount(1000)
v20 = LoadFeeratePerKw(2500)
v21 = LoadU16(144)
v22 = LoadU16(30)
v23 = LoadU8(0)
v24 = LoadShutdownScript(Empty)
v25 = LoadChannelType(StaticRemoteKey)
v26 = BuildOpenChannel(v12, v13, v14, v15, v16, v17, v18, v19, v20, v21, v22, v1, v3, v5, v7, v9, v11, v23, v24, v25)
v27 = SendOpenChannel(v26)
v28 = RecvAcceptChannel(v27)
v29 = ExtractFundingPubkey(v28)
v30 = CreateFundingTransaction(v1, v29, v14, v20)
v31 = SendFundingCreated(v30, v0, v13)
v32 = RecvFundingSigned(v31)
BroadcastTransaction(v30)
MineBlocks(8)
v35 = LoadPrivateKey(0x2727272727272727272727272727272727272727272727272727272727272727)
v36 = DerivePoint(v35)
v37 = LookupShortChannelId(v30)
SendChannelReady{include_alias=false}(v32, v36, v37)
RecvChannelReady()

@erickcestari

erickcestari commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

LDK is the unique implementation that doesn't use TCP_NODELAY. I'll try to upstream it, so in the future we can drop these changes.

I've opened this PR: https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/pulls/5065

morehouse
morehouse reviewed Oct 8, 2026
View reviewed changes
Comment thread smite/src/noise/connection.rs Outdated
Comment thread smite/src/noise/connection.rs Outdated
@erickcestari
erickcestari requested a review from morehouse October 8, 2026 17:10
morehouse
morehouse reviewed Oct 8, 2026
View reviewed changes
Comment thread smite/Cargo.toml Outdated
@erickcestari
smite: re-arm TCP_QUICKACK after each send
d775108 
LDK doesn't set TCP_NODELAY, so it holds a small reply until we ACK its
last segment, and Linux delays that ACK ~40 ms. This stalled the final
pong of every LDK funding execution; fixing it takes LDK from 13.3 to
42.9 execs/s under Nyx.
@erickcestari
erickcestari requested a review from morehouse October 8, 2026 17:33
morehouse
morehouse approved these changes Oct 8, 2026
View reviewed changes

@morehouse morehouse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this with a short LDK campaign. execs/sec went up by ~50%!

@morehouse
morehouse merged commit 4ded568 into lnfuzz:master Oct 8, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morehouse morehouse morehouse approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@erickcestari @morehouse