Commitment trie bottleneck investigation: measurements + proposed solution for #20920

[mh0lt]

This issue presents two things — separable for review:

1. Measurements of where time is actually spent during the canonical SSTORE-bloated commitment workload. These are useful as input to the broader debate about whether and where to rewrite the commitment trie, independent of the proposed solution below.
2. A proposed solution path drawn from those measurements: cross-block branch caching with decoded payload, shipped as 10 incremental low-risk PRs.

Linked to #20920 (the user-facing perf gap). #20920 closes when measurements confirm the gap is materially closed, by whatever path. This issue closes when the proposed PR sequence is merged. If after that the gap isn't closed, reopen the architecture debate with fresh measurements.

TL;DR

For the canonical SSTORE-bloated workload from #20920, the bottleneck is data retrieval during unfold — bloom filter + decompression + shortened-key expansion + decode for ~39k branches per block. Fold parallelization would not buy much on this workload because fold is not on the critical path (does not appear in the top 30 CPU consumers).

The proposed fix is to reduce data retrieval, not parallelize compute: extend caching so the same hot-account branches don't get re-loaded every block.

Caveat: this is one extreme workload (single-account, dense storage, cold cache). The conclusions are scoped to "fix this use case." See Part 1 "Workload character" for what we did NOT measure.

---

Part 1: Measurements

Setup

• Workload: perf-devnet-3 block 24358306, 30M gas, 4,200 cold SSTOREs into one EOA's storage trie
• Hardware envelope: 6-core cgroup, 32 GB RAM, NVMe, 3.6 GHz pinned
• Base: PR parallel commitment calculations implemented #20805 (parallel commitment calculator) at HEAD 130a6e1b50
• Cold-cache wrapper between every run
• 7 runs over ~24 hours, varying touch-time warmer plumbing + worker count

Per-run results on block 24358306

Run	Variant	mgas/s	Cache stats (combined commitment-cache after Process)
I	Baseline (Stage A: EnableTrieWarmup removed)	7.16	branch 62.5% / acct 0% / stor 0/5758
M	+ Touch-time warmer (page-cache only, drain-cancel bug)	7.13	branch 62.3% / acct 0% / stor 0/5758
N	+ WaitForDrain fix (cancel→wait)	7.15	branch 62.7% / acct 0% / stor 0/5758
O	+ Cache handoff via WarmupConfig.PreWarmed	7.13	branch 72.3% / acct 99.8% / stor 0/5758
P	+ PreloadStorage direct leaf reads	7.16	branch 72.5% / acct 99.9% / stor 100% (trie view)
Q	TIP_TRIE_WARMUPERS=256	6.75	branch 72.3% / acct 99.9% / stor 100%
R	TIP_TRIE_WARMUPERS=8	7.08	branch 72.7% / acct 99.9% / stor 100%

Key invariant: total disk reads constant

/proc/PID/io read_bytes for block 24358306 across all 7 runs: 1.272 GB ± 200 KB. The warmer just shifts WHEN reads happen, not WHETHER.

CPU profile (Run P, bloat block dominates the 30s window)

Total samples = 4.22s (14% utilization across 6 cores)

Top by cum CPU (% of total samples):
85% Warmuper.warmupKey
84% → branchFromCacheOrDB
83% → SharedDomains.GetLatest → AggregatorRoTx.getLatest
67% → getLatestFromFile
59% → recsplit.Index.Lookup
39% → BinaryFuse.Contains  (xorfilter bloom)
20% → seg.Getter.Next  (decompression)
20% → seg.Getter.nextPos
14% → BranchData.ReplacePlainKeys  (decode)
14% → replaceShortenedKeysInBranch  (file-format expansion)

Fold work: NOT in top 30.

Worker scaling

TIP_TRIE_WARMUPERS	mgas/s	Δ vs default
8	7.08	-1%
~48 (default = NumCPU × 8)	7.16	baseline
256	6.75	-6%

Flat-then-regressing curve.

Per-block branch read count

Run P combined cache stats post-Process:

• branches: hit 102,606 / miss 39,124 — 39,124 distinct branches loaded per block
• accounts: hit 5,958 / miss 8 — 99.9% (touch-warmer pre-fills)
• storage: hit 5,759 / miss 5,910 — trie view 100%; the 5,910 misses are the touch-warmer's own initial reads

39k × ~30 KB per branch ≈ 1.17 GB → matches measured 1.27 GB read_bytes.

Workload character — important caveat

This benchmark is one extreme, chosen as canonical from EthPandaOps' benchmarkoor suite because it stresses cold-cache + dense-storage. It is NOT representative of all commitment workloads:

• Single-account, dense storage subtree. All 4,200 SSTOREs go into one EOA, so all keys share a 64-nibble account-hash prefix. The 16-way concurrent-commitment root fanout is unhelpful here (1 of 16 subtries does 100% of the work), but on cross-account workloads it matters.
• Cold cache. Multi-block hot-account workloads (Uniswap router, USDT, etc. on mainnet) have very different repeat-rates and cache dynamics.
• Per-block, not batch. Initial-sync batch commitment has a different cost profile (deferred writes, larger touched-key sets, different I/O amortization).
• No reorgs. Reorg-heavy periods stress invalidation paths that this bench doesn't touch.

The measurements above tell us about the bottleneck on THIS workload specifically. They are evidence about one corner of the design space, not a verdict on the trie overall.

Part 2: What the measurements rule out

These four hypotheses for the bottleneck are all falsified by the runs above:

• Fold work is NOT the bottleneck. Doesn't appear in the top 30 CPU consumers. Stage F (parallel tree-reduce fold) would deliver no measurable win on this workload.
• Read I/O queue depth is NOT the bottleneck. 8 → 48 → 256 warmer goroutines moves mgas/s by 7% in a flat-then-regressing curve. NVMe is not queue-saturated; adding workers doesn't help.
• In-memory cache misses are NOT the bottleneck. We took the storage cache hit rate from 0% to 100% (from the trie's perspective) without moving mgas/s. The work moved earlier in time but total cycles spent are the same.
• Unfold parallelism is NOT the bottleneck. Touch-time warmer runs unfold reads concurrent to EVM execution. Wall time unchanged. Stage E (parallel pre-unfold into trie cell state) would deliver no measurable win on this workload either.

Part 3: Conclusions

For this workload specifically:

• Fix data retrieval during unfold. The 39k branch reads × ~80 µs each per block (bloom + decompress + decode) are what we have to address. The smallest change that reduces this is keeping decoded branches cached across blocks so hot accounts don't re-load every time.
• Fold parallelization won't buy much. Fold work isn't even visible in the top 30 CPU consumers. Stage E (parallel pre-unfold) and Stage F (parallel tree-reduce fold) — both architectural options we'd considered — would not deliver a measurable win on this workload. The 3.1 CPU-seconds of unfold-side decode is the floor regardless of how compute is scheduled.
• More warmer goroutines won't help. Worker scaling is flat-then-regressing (8 → 7.08, 48 → 7.16, 256 → 6.75). Default already saturates what NVMe + MDBX can deliver in parallel.

Implications for the trie-rewrite debate (scoped)

These conclusions bear on the rewrite debate, but the scope of what they actually demonstrate is narrow:

• They DO rule out parallelism-of-existing-algorithm as a way to close the gap on THIS workload. Stage E/F would not deliver here.
• They DO NOT generalize to other workloads. A cross-account block where 16-way root fanout actually engages might benefit from parallelism in ways this bench can't show. Not measured.
• They DO NOT rule out a different file format (smaller branches, no shortened-key indirection, no bloom filter). A different format might lower the unfold floor. That's a legitimate rewrite motivation independent of these measurements.
• They DO NOT rule out a different trie shape (different account boundary, sub-fanout). The bloat block's single-account shape would specifically benefit from sub-fanout — that's a distinct angle this bench can't evaluate.
• They DO suggest that any rewrite motivated PURELY by parallelism would be ineffective on this workload. A rewrite motivated by format or shape change is not addressed here at all.

A complete rewrite-or-not decision would need measurements across more of the workload space — at minimum: a cross-account block, a multi-block hot-account window, a batch-commitment fixture, and a reorg-heavy fixture. This issue does not attempt to settle the rewrite question — it only justifies the conservative caching path for the immediate gap on this specific workload.

---

Part 4: Proposed solution

Given the conclusion that the lever is data-retrieval reduction during unfold, the proposed fix is cross-block branch caching with decoded payload — shipped as 10 incremental PRs.

A decoded cache directly addresses the measured bottleneck:

• Cache hit on hot accounts skips the bloom filter check, skips the seg decompression, skips the shortened-key expansion, skips the cell decode.
• Decoded payload (vs encoded bytes) saves the decode CPU on every hit.
• Cross-block persistence skips re-loading branches that haven't changed.

Caching is also the lowest-blast-radius option: cache miss = current behavior. No structural change to the trie algorithm, the file format, or the read path beyond a lookup-then-fallthrough.

PR sequence

#	PR	Type	Risk	Expected mgas/s	LOC
1	Extract canonical decodeBranchInto (refactor)	very low	0	<300
2	Extract canonical unfoldKeyPath orchestrator pattern	low	0	<400
3	Observability + WaitForDrain bug fix	very low	0	~80
4	PR #19954 carry-as-is invariants (dirty-flag, PutIfClean, clear-on-Reset, root-only restriction)	low-medium	0	~250
5	Switch WarmupCache to decoded payload	low	small	<500
6	Two-tier BranchCache (root-pinned + LRU) — still ephemeral	low-medium	small	~400
7	Cross-block persistence for BranchCache (feature-flagged)	medium-high	large on hot-account workloads	~300
8	Wire prefetcher + touch hooks (BAL + EVM-time)	medium	improves cache fill	~500
9	Centralize encode-side invalidation via BranchEncoder hook	low-medium	0	~150
10	Drop extractBranchCellAddresses (now redundant)	low	0	-200

PRs 1, 2, 9, 10 consolidate fragmented decode + traversal logic (4 partial decode paths, 2 traversals today) into single primitives — this pays back when caching lands on top because there's only one decoder to keep correct.

PR 7 is the perf-delivering PR. Lands behind COMMITMENT_BRANCH_CACHE_PERSIST=true for one release.

Validation discipline per PR

• Unit tests + lint pass
• Cold canonical-envelope bench (single 4s block, fast)
• For PRs 5-8: multi-block window to watch warm-cache hit rates evolve
• For PR 7: explicit reorg test (sync N → unwind 5 → resync) — MUST prove correct
• PR description includes expected mgas/s impact + actual measured

Sequencing

1. PR parallel commitment calculations implemented #20805 (parallel commitment calculator) merges — expected within days
2. M-Q prototype changes dropped from PR parallel commitment calculations implemented #20805 worktree (debug-quality only, not shippable). Worktree retained for measurement reference.
3. Branch commitment/decoded-warmup-cache off post-parallel commitment calculations implemented #20805 main, ship PRs 1-10 in order
4. Re-run cold canonical bench at each PR

Closing criteria for THIS issue

All 10 PRs above merged.

Closing criteria for #20920 (separate)

• Cold canonical bench shows mgas/s within 10% of geth/besu/nethermind on block 24358306 (target: ≥ 9.5 mgas/s)
• Multi-block hot-account window confirms cache delivers ≥ 90% hit rate

If after the 10 PRs land the gap isn't closed, reopen the architecture debate (rewrite, sub-fanout, format change) with fresh measurements.

Detailed design

agentspecs/commitment-decoded-cache-pr-next.md — full design with PR #19954 commits to carry vs replace, file:line refactor targets, decode boundary analysis.

[VBulikov]

yperbasis added Imp1 as Importance