LeiosDb: bound the WAL, remove the write hotspot, stop dying on lock contention

[ch1bo]

Several fixes to the Leios SQLite layer, stacked on the recent tx cache / fetch logic / eb validation work. The fixes were driven by multi-hour long stress tests of a twelve-node dozen-devnet at TPS=500 (effective ingest lower than that).

Important

Note the schema change has no migration: an existing database is not readable by this version and requires a wipe & resync.

What was wrong

1. page_size was a silent no-op. It was set after journal_mode = WAL, and SQLite cannot change the page size of a database already in WAL mode. Every run so far used the 4096 default. With 32768 actually in effect, WAL amplification measured 35x (34 GiB of log for 0.97 GiB of data) against ~18x at 4096. Reordered and pinned at 4096.

2. No busy_timeout was set, so SQLite refused a contended lock immediately and all waiting happened in an application loop whose Nth sleep was 100us * N — cumulative 100us * N(N+1)/2. The observed EB-body insert distribution matched that curve exactly: median 1 599 ms (N≈178), max 18 435 ms (N≈607), for 90 ms of actual work. busy_timeout = 1000 now does the waiting in C, and the application loop is a flat 20 ms backstop, making the ceiling linear rather than quadratic. Current ceiling is ~30 s after which a node would crash.

3. idx_ebTxs_txHashBytes was the dominant write cost. It existed for one query — decrementing missingTxCount for EBs referencing an arriving tx — and covered every (EB, tx) pair ever seen: 4.99M entries for 1.07M distinct hashes (4.7x on average, 43x worst), 887 MB of a 1 484 MB database. Each 13 568-row body insert scattered that many random-hash insertions, dirtying a page apiece.

An arriving tx can only complete EBs that were still waiting for it; EBs that already held it counted it as present when their body landed. The resolution between ebs and txs is now two tables: ebTxs keeps the ordered body, and ebsMissingTxs holds only pairs an EB still lacks, populated by one anti-join at body insert and retired as txs arrive. Both move in the same transaction. Serving is unchanged — PK range scan plus join.

Measured effect

Write amplification, one 13 568-row EB body insert against a 4M-row table:

	WAL generated	per row
with idx_ebTxs_txHashBytes	61.0 MB	4 494 B
two-table, no missing txs	1.8 MB	134 B
two-table, 2% missing	3.2 MB	235 B

EB body ingest latency at matched closure size (~6 500 txs):

	median	p90
before	179 ms	—
busy_timeout only	80 ms	284 ms
busy_timeout + two-table	36 ms	84 ms

Per instance, over a full run:

	before	after
WAL	9 304 MB against a 54 MB database	57–67 MB against a 3 070 MB database
first tooLate vote	~25 min	~1 h 15 min
certification	stopped at ~2 h	ran 5 h 20 min

The WAL figure is #2233: a writer waiting inside a deferred transaction pinned a read snapshot, freezing back-fill (framesCopied stuck at 546 while framesInLog reached 241 239), so no checkpoint could reset the log.

Instrumentation

TraceLeiosBlockIngest reports, per ingested body, the wait for the node-wide outstanding lock and then the database insert, tx-cache insert and mempool pull separately, plus total lock hold. This is what localised the cost; without it the DB insert and the lock wait are indistinguishable.

TraceLeiosDbBusyRetry fires when SQLite's own timeout was exhausted and the application backstop engaged, with attempt number and accumulated wait. A steady stream of these means the write path is saturated.

Remaining work

Database access is not yet congestion-free. In the same run the insert median still grew from 6 ms to 524 ms over seven hours as the instance database reached 3 070 MB, and the busiest instance logged 44 TraceLeiosDbBusyRetry events over 4.5 hours — each one a BEGIN IMMEDIATE that gave up after a full second in SQLite's handler. That is a floor on contention, not a measure of it: an attempt that waits and then succeeds is not traced. Neither ebTxs, ebsMissingTxs nor txs is pruned, so per-EB cost continues to rise with database size; pruning both resolution tables on the same age rule is the next step, and idx_ebsMissingTxs_ebHashBytes exists to support it.

[nfrisby]

The code itself looks great. Many of the comments were wrong/incomplete/(weirdly) harmfully misplaced, so I'm Requesting Changes just for that.