[wip/DO NOT MERGE ] db/state: lock-free DomainMetrics + kv_get (atomics)

[sudeepdino008]

Not intended for merge. mh0lt's #21663 already removes this contention (process-level channel-fed collector) and is the PR to land. This branch is a performance-investigation record: what was measured, the read latencies of db and files in EVM-exec vs commitment-calc regimes

meant to be revisited/resumed once we get all cache optimizations and fixes from mh0lt's performance stack. I think we should get the same range of numbers or better, once the stack is merged.

What was tracked

KV_READ_METRICS=true emits the [Execution] domain reads log line. Setup: mainnet, --prune.mode=minimal, at chaintip, TIP_TRIE_WARMUPERS=256, --exec.state-cache=false, mutex profile via PERF_PROFILES=true.

Finding: the durations were measuring the instrumentation's own lock contention

The tip warmup pool (TIP_TRIE_WARMUPERS, default NumCPU*8) is 128 goroutines that all read the commitment domain concurrently during commitment calc. They serialize on the metric-recording locks (changeset.DomainMetrics mutex + the per-(domain,level) kv_get prometheus Summary mutex). The critical section is tiny, but at N-way fan-in the contended-mutex handoff (~1µs, futex park/unpark) dominates, and since the metric is wall-clock that wait is folded into the reported dbdur/fdur themselves.

Signature: fdur scales linearly with warmuper count — ≈ 0.9µs × N (16→26µs, 64→62µs, 128→115µs, 256→223µs) — i.e. N goroutines serializing on one lock, not storage cost. Throughput (~85–89 mgas/s) is identical across all configs.

This branch makes both recording paths lock-free (atomics + fixed [DomainLen] array; lock-free kv_get accumulator).

Aggregate (chaintip, W=256)

	dbdur	fdur	warmupKey mutex share	total mutex delay	mgas/s
main (baseline)	26µs	223µs	48% (summary.ObserveDuration)	151s	~85
this branch	3µs	28µs	1.6%	68s	~89

EVM-exec vs commitment-calc split (per-domain)

The contention was entirely in the commitment regime (the only place warmupers run). Removing the metric locks collapses it to parity with the execution domains:

domain (regime)	baseline (contended)	this branch
commitment (commit-calc)	47µs / 205µs	3µs / 40µs
storage (EVM-exec)	5µs / 4µs	4µs / 22µs
code (EVM-exec)	5µs / 0.2µs	4µs / 42µs
account (EVM-exec)	—	served from mem/cache

(dbdur / fdur, medians.) EVM-exec domains were never contended (no warmupers there); their dbdur is flat, fdur only wobbles with per-block cache warmth.

Relationship to #21663

#21663 removes the DomainMetrics collection lock (channel/actor) and adds source labels + RPC coverage — the superset there. It does not touch the per-level kv_get Summary lock in getLatestFromFile; this branch makes that lock-free too (atomics). The DomainMetrics part overlaps — #21663 is the one to merge.

Next contention source: BranchCache (mh0lt's perf stack)

The perf stack (#21380 / #21386) introduces commitment.BranchCache, whose LRU tail wraps hashicorp/golang-lru/v2 — a single exclusive mutex (even Get locks, to bump recency). Profiling #21663 at chaintip, the 256 warmupers serialize there: BranchCache.Get+Put ≈ 72% of mutex delay (320s total); bypassing that lock just relocates them back onto the kv_get Summary. So the warmup read path has three serialized locks — DomainMetrics, kv_get Summary, BranchCache — and the pool only runs free with all three lock-free/sharded.

BranchCache is not in main (so this branch can't address it) — flagging it for the stack: none of the stacked PRs (#21380/#21386 introduce it; #21414/#21416/#20893 don't touch it, including the parallel-commitment PR) shard the LRU tail.