LeiosDemoDb: BEGIN IMMEDIATE for writing transactions - #2233
Merged
Conversation
A bare BEGIN is BEGIN DEFERRED, so a transaction that reads before it writes has to upgrade its lock. In WAL mode that upgrade fails with SQLITE_BUSY_SNAPSHOT whenever another connection committed in between, and that status is neither serviced by the busy handler nor retryable at the statement level: the transaction's snapshot is stale for good, so re-stepping the same statement can never succeed. The retry loops in withDie and dbStepInsert spin to exhaustion and then throw, which arrives as ExceptionInLinkedThread and takes the node down. Observed on a twelve-node devnet under load: 297 LeiosDbException ErrorBusy across all twelve nodes, every single one from the same statement, the stMarkNotifiedEbs UPDATE in sqlInsertTxs. That transaction inserts only txs not already persisted, so under backlog the inserts are frequently empty, the SELECT loop takes the snapshot, and that UPDATE is the first write, hence the upgrade. It never failed when inserts had happened, because the write lock was already held. Taking the lock at BEGIN makes that upgrade impossible, so the only place a writer can now be told BUSY is the BEGIN itself. That is already retried, since dbExec goes through withDie, and retrying there is sound in a way the old failure was not: no transaction is open and no work has been done, so re-attempting the BEGIN re-attempts the whole transaction. No separate transaction-level retry is needed; with IMMEDIATE the two coincide. Splits the helper so readers keep BEGIN DEFERRED and do not exclude each other, and only the three writers take the lock up front.
nfrisby
approved these changes
Aug 26, 2026
nfrisby
left a comment
nfrisby
left a comment
Contributor
There was a problem hiding this comment.
Looks to me like a reasonable fix to attempt
geo2a
approved these changes
Aug 27, 2026
Contributor
Author
|
More fixes following this in #2245 |
ch1bo
added a commit
that referenced
this pull request
Aug 29, 2026
…contention (#2245) Several fixes to the Leios SQLite layer, stacked on the recent tx cache / fetch logic / eb validation work. The fixes were driven by multi-hour long stress tests of a twelve-node `dozen-devnet` at `TPS=500` (effective ingest lower than that). > [!IMPORTANT] > > Note the schema change has no migration: an existing database is not readable by this version and requires a wipe & resync. ## What was wrong **1. `page_size` was a silent no-op.** It was set after `journal_mode = WAL`, and SQLite cannot change the page size of a database already in WAL mode. Every run so far used the 4096 default. With 32768 actually in effect, WAL amplification measured 35x (34 GiB of log for 0.97 GiB of data) against ~18x at 4096. Reordered and pinned at 4096. **2. No `busy_timeout` was set,** so SQLite refused a contended lock immediately and all waiting happened in an application loop whose Nth sleep was `100us * N` — cumulative `100us * N(N+1)/2`. The observed EB-body insert distribution matched that curve exactly: median 1 599 ms (N≈178), max 18 435 ms (N≈607), for 90 ms of actual work. `busy_timeout = 1000` now does the waiting in C, and the application loop is a flat 20 ms backstop, making the ceiling linear rather than quadratic. Current ceiling is ~30 s after which a node would crash. **3. `idx_ebTxs_txHashBytes` was the dominant write cost.** It existed for one query — decrementing `missingTxCount` for EBs referencing an arriving tx — and covered every `(EB, tx)` pair ever seen: 4.99M entries for 1.07M distinct hashes (4.7x on average, 43x worst), 887 MB of a 1 484 MB database. Each 13 568-row body insert scattered that many random-hash insertions, dirtying a page apiece. An arriving tx can only complete EBs that were still *waiting* for it; EBs that already held it counted it as present when their body landed. The resolution between `ebs` and `txs` is now two tables: `ebTxs` keeps the ordered body, and `ebsMissingTxs` holds only pairs an EB still lacks, populated by one anti-join at body insert and retired as txs arrive. Both move in the same transaction. Serving is unchanged — PK range scan plus join. ## Measured effect Write amplification, one 13 568-row EB body insert against a 4M-row table: | | WAL generated | per row | |------------------------------|---------------|---------| | with `idx_ebTxs_txHashBytes` | 61.0 MB | 4 494 B | | two-table, no missing txs | 1.8 MB | 134 B | | two-table, 2% missing | 3.2 MB | 235 B | EB body ingest latency at matched closure size (~6 500 txs): | | median | p90 | |----------------------------|--------|--------| | before | 179 ms | — | | `busy_timeout` only | 80 ms | 284 ms | | `busy_timeout` + two-table | 36 ms | 84 ms | Per instance, over a full run: | | before | after | |----------------------|-----------------------------------|--------------------------------------| | WAL | 9 304 MB against a 54 MB database | 57–67 MB against a 3 070 MB database | | first `tooLate` vote | ~25 min | ~1 h 15 min | | certification | stopped at ~2 h | ran 5 h 20 min | The WAL figure is #2233: a writer waiting inside a deferred transaction pinned a read snapshot, freezing back-fill (`framesCopied` stuck at 546 while `framesInLog` reached 241 239), so no checkpoint could reset the log. ## Instrumentation `TraceLeiosBlockIngest` reports, per ingested body, the wait for the node-wide outstanding lock and then the database insert, tx-cache insert and mempool pull separately, plus total lock hold. This is what localised the cost; without it the DB insert and the lock wait are indistinguishable. `TraceLeiosDbBusyRetry` fires when SQLite's own timeout was exhausted and the application backstop engaged, with attempt number and accumulated wait. A steady stream of these means the write path is saturated. ## Remaining work Database access is not yet congestion-free. In the same run the insert median still grew from 6 ms to 524 ms over seven hours as the instance database reached 3 070 MB, and the busiest instance logged 44 `TraceLeiosDbBusyRetry` events over 4.5 hours — each one a `BEGIN IMMEDIATE` that gave up after a full second in SQLite's handler. That is a floor on contention, not a measure of it: an attempt that waits and then succeeds is not traced. Neither `ebTxs`, `ebsMissingTxs` nor `txs` is pruned, so per-EB cost continues to rise with database size; pruning both resolution tables on the same age rule is the next step, and `idx_ebsMissingTxs_ebHashBytes` exists to support it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Should fix the last remaining occurrences of
LeiosDbException ErrorBusyA bare BEGIN is BEGIN DEFERRED, so a transaction that reads before it writes has to upgrade its lock. In WAL mode that upgrade fails with SQLITE_BUSY_SNAPSHOT whenever another connection committed in between, and that status is neither serviced by the busy handler nor retryable at the statement level: the transaction's snapshot is stale for good, so re-stepping the same statement can never succeed. The retry loops in withDie and dbStepInsert spin to exhaustion and then throw, which arrives as ExceptionInLinkedThread and takes the node down.
Observed on a twelve-node devnet under load: 297 LeiosDbException ErrorBusy across all twelve nodes, every single one from the same statement, the stMarkNotifiedEbs UPDATE in sqlInsertTxs. That transaction inserts only txs not already persisted, so under backlog the inserts are frequently empty, the SELECT loop takes the snapshot, and that UPDATE is the first write, hence the upgrade. It never failed when inserts had happened, because the write lock was already held.
Taking the lock at BEGIN makes that upgrade impossible, so the only place a writer can now be told BUSY is the BEGIN itself.
Splits the helper so readers keep BEGIN DEFERRED and do not exclude each other, and only the three writers take the lock up front.
Also reduces the maximum retries to
1000, leading to an error after exhausting retries within ~50s instead of~5000s.