Summary
The Senders stage (execution/stagedsync/stage_senders.go) spins up a full
runtime.NumCPU() pool of ECDSA recovery workers, but in practice only ~5 of 12
hardware threads are busy during a large sender-recovery run. The
embarrassingly-parallel part (pubkey recovery) is starved by the serial work
around it, so adding recovery cores doesn't help — it's a pipeline/Amdahl
imbalance, not a worker-count cap.
Observed
During a mainnet backfill (~44k-block gap, --prune.mode=minimal) the senders
stage sat at ~477% CPU (≈5 cores) for an extended period on a 12-thread box,
repeatedly emitting [3/6 Senders] Flushed buffer file erigon-sortable-buf-*.
Why it isn't a worker-count cap
numOfGoroutines = secp256k1.NumOfContexts() (stage_senders.go:63), and
NumOfContexts() returns runtime.NumCPU() (one secp256k1 context per CPU). So
the recovery pool is 12; each worker has its own crypto context
(stage_senders.go:108-115). The recovery itself can use all cores.
Root cause — serial sections starve the parallel recovery
- Feed (main goroutine, serial): reads each canonical block body (snapshot
mmap → page-fault IO) and RLP-decodes every tx before pushing recovery jobs.
Body read + decode is single-threaded and dominates the cheap per-tx ECDSA
recovery, so workers idle waiting to be fed.
- Drain (single goroutine + lock): one goroutine drains
out, reassembles
per-block sender arrays under pendingMu, and collectorSenders.Collect()s
(stage_senders.go:134-174). Serial chokepoint with lock contention.
- ETL spill flushes: the collector uses
SmallSortableBuffers with
SortAndFlushInBackground(false) (stage_senders.go:117-119); the periodic
spill flushes stall the pipeline (the Flushed buffer file lines).
Suggested directions (not now)
- Parallelise / pipeline the body read + RLP-decode feed so the recovery pool
stays fed (e.g. a small reader/decoder pool, or prefetch+decode ahead).
- Reduce the single-drain bottleneck (shard the drain, or lock-free per-block
assembly).
- Larger ETL buffers and/or genuinely background flush so spill IO doesn't stall
the pipeline.
Priority
Looks fixable but is not urgent — filing to track. Same shape as the broader
"parallel compute gated by serial IO/marshalling around it" pattern.
Summary
The Senders stage (
execution/stagedsync/stage_senders.go) spins up a fullruntime.NumCPU()pool of ECDSA recovery workers, but in practice only ~5 of 12hardware threads are busy during a large sender-recovery run. The
embarrassingly-parallel part (pubkey recovery) is starved by the serial work
around it, so adding recovery cores doesn't help — it's a pipeline/Amdahl
imbalance, not a worker-count cap.
Observed
During a mainnet backfill (~44k-block gap,
--prune.mode=minimal) the sendersstage sat at ~477% CPU (≈5 cores) for an extended period on a 12-thread box,
repeatedly emitting
[3/6 Senders] Flushed buffer file erigon-sortable-buf-*.Why it isn't a worker-count cap
numOfGoroutines = secp256k1.NumOfContexts()(stage_senders.go:63), andNumOfContexts()returnsruntime.NumCPU()(one secp256k1 context per CPU). Sothe recovery pool is 12; each worker has its own crypto context
(stage_senders.go:108-115). The recovery itself can use all cores.
Root cause — serial sections starve the parallel recovery
mmap → page-fault IO) and RLP-decodes every tx before pushing recovery jobs.
Body read + decode is single-threaded and dominates the cheap per-tx ECDSA
recovery, so workers idle waiting to be fed.
out, reassemblesper-block sender arrays under
pendingMu, andcollectorSenders.Collect()s(stage_senders.go:134-174). Serial chokepoint with lock contention.
SmallSortableBufferswithSortAndFlushInBackground(false)(stage_senders.go:117-119); the periodicspill flushes stall the pipeline (the
Flushed buffer filelines).Suggested directions (not now)
stays fed (e.g. a small reader/decoder pool, or prefetch+decode ahead).
assembly).
the pipeline.
Priority
Looks fixable but is not urgent — filing to track. Same shape as the broader
"parallel compute gated by serial IO/marshalling around it" pattern.