Immutable/volatile split of LeiosDB + garbage collection - #2223
Conversation
e3544e6 to
dabc62d
Compare
- Add a column to the `ebs` SQLite table that distinguishes EBs as immutable.
- Mark EBs as immutable once their RB becomes immutable.
- Garbage Collect EBs and their transations if they are volatile and are older that
the immutable tip.
- Add trace events for Prometheus gauges:
* `TraceLeiosDbCopiedToImmutable` --- Rows moved into the immutable partition
by 'LeiosDbHandle.leiosDbCopyToImmutable'
* `TraceLeiosDbVolatileStats` --- volatile counts and file bytes, sampled on a
timer over a fresh read-only connection per tick, kept clear of startup.
* `TraceLeiosDbImmutableStats` --- the immutable counts maintained incrementally
in a one-row `immutableStats` table.
dabc62d to
7c1058f
Compare
| [ "pragma journal_mode = WAL;" | ||
| , "pragma synchronous = normal;" | ||
| , "pragma page_size = 32768;" | ||
| , "pragma mmap_size = 268435500;" |
There was a problem hiding this comment.
A claude instance mentioned that page size cannot be change after setting journal mode to WAL and it was all along at the default (4096). We should probably double check whether we really want the 32KiB
| [ "pragma journal_mode = WAL;" | |
| , "pragma synchronous = normal;" | |
| , "pragma page_size = 32768;" | |
| , "pragma mmap_size = 268435500;" | |
| [ "pragma synchronous = normal;" | |
| , "pragma page_size = 32768;" | |
| , "pragma mmap_size = 268435500;" | |
| , "pragma journal_mode = WAL;" |
| ( negate (fromIntegral nEbs) | ||
| , negate (fromIntegral nEbTxs) | ||
| , negate (fromIntegral nTxsTotal) | ||
| ) |
There was a problem hiding this comment.
Why are we writing metadata (counts of table rows) into the database itself?
| , " immutableEbTxs INTEGER NOT NULL," | ||
| , " immutableTxs INTEGER NOT NULL" | ||
| , ");" | ||
| , -- TODO(geo2a): why exactly is this needed? Initialize the stats with the existing data on node restart? |
There was a problem hiding this comment.
Didn't you put it? :D
Basically the same question as I had above.
| , -- NULL = body not downloaded, >0 = txs missing, 0 = just completed, <0 = notified | ||
| " missingTxCount INTEGER," | ||
| , -- 1 = the immutable chain references this EB | ||
| " immutable INTEGER NOT NULL DEFAULT 0," |
There was a problem hiding this comment.
Must: not re-use the same table for the immutable partition
The whole goal is to make queries scanning the table or recomputing indices not needing to process the immutable entries. A separate table is the absolute minimum requirement we need. A separate file is what we actually want, to allow SPOs to keep the immutable DB on a potentially slower disk.
2 participants
Implements input-output-hk/ouroboros-leios#969
cardano-nodePR: IntersectMBO/cardano-node#6656This PR splits the contents of LeiosDB into volatile and immutable partitions, and periodically marking EBs as immutable once their referencing RB becomes immutable and garbage collection orphaned volatile EBs.
An EB is marked as immutable via the
immutableflag column of theebstable. All transactions (rows of thetxstable) referenced by this EB via theebTxstable become immutable as well, but there are no explicit marker in those tables.The PR also adds observability of the volatile/immutable EB counts an the GC rates, which is wired into a Graphana dashboard which looks like this:
Benchmarking
The PR adds the
leios-gc-benchbenchmark, which works both synthetic data and with a realleios.db. I've generated some input data by running the devnet overnight to test the performance:The duration of the first immutable marking and GC are an outlier, which Claude attributes to cold OS page cache. The outliers are clearly visible on the following scatter plot (x axis is log scale):
When running the benchmark with synthetic data, the outliers don't show up, probably because the sqlite database is written into before the start of the measurement:
Implementation details
One particular peculiarity that I had to iron out was the use of the
unsafeFFI calls to the SQLite bindings. The immutalisation or GC operations need to run in asafeFFI call so that they are run on a separate GHC RTS capability and do not block the node from doing other things. This needs further investigation, together with auditing other FFI calls and making a weighted decision on which mechanism to better for every one of them.