Leios: add LeiosTxCache, but don't rely on it at all yet - #2188
Merged
Conversation
Contributor
Author
|
Most of the lines of code are: the big comments in |
Contributor
Author
|
The same
|
Contributor
Author
|
Benchmarks, on my laptop:
|
nfrisby
commented
Aug 6, 2026
| {-# LANGUAGE BangPatterns #-} | ||
| {-# OPTIONS_GHC -O2 #-} | ||
|
|
||
| -- | A mutable, salted open-addressing hash table for the Leios tx cache: linear |
Contributor
Author
There was a problem hiding this comment.
Issue input-output-hk/ouroboros-leios#1021 says:
- work with modest Leios parameters
- 100kB max EB body
- 1MB max EB closure
I didn't do that; I targeted the full 512 kB body and 12 MB closure case instead.
Thus:
- Maybe the
Referenceimpl is fine, no hash table needed? If the EB body is only 20% of the size this PR was expecting (so ≤~3000 txs instead of ≤~15000), then theMap TxHashworst-case will be 20% of what the benchmark reports. Maybe that's fine? (128 * ~3000 = ~384000, which is still a pretty bigMap, and every major GC has to walk the whole thing). - The 12 MB versus 1 MB closure doesn't relate to the hash table... except for the fact that if we don't need the hash table, then we could just have the
Mapcontain the txs' bytes directly, since that would only be at most ~128 MB (+ GHC heap overhead).- But that doesn't actually save much; merely side-steps having to read the txs from the LeiosDb.
- If we want that, it'd be easy: the
Referencemap is already parameterized bya=(); we'd simply change that toa=ShortByteString, I think? (This PR doesn't alter LeiosFetch at all yet, so I'm not sure what it'd actually need.)
$ cabal bench leios-txcache-bench --benchmark-options=pure
Configuration is affected by the following files:
- cabal.project
Build profile: -w ghc-9.12.2 -O1
In order, the following will be built (use -v for more details):
- ouroboros-consensus-3.0.1.0 (bench:leios-txcache-bench) (file ouroboros-consensus/bench/leios-txcache-bench/Main.hs changed)
Created semaphore called cabal_semaphore_b with 5 slots.
Preprocessing benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Building benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
[1 of 1] Compiling Main ( ouroboros-consensus/bench/leios-txcache-bench/Main.hs, /home/nfrisby/cardano-haskell/cardano-node/dist-newstyle/build/x86_64-linux/ghc-9.12.2/ouroboros-consensus-3.0.1.0/b/leios-txcache-bench/build/leios-txcache-bench/leios-txcache-bench-tmp/Main.o ) [Source file changed]
[2 of 2] Linking /home/nfrisby/cardano-haskell/cardano-node/dist-newstyle/build/x86_64-linux/ghc-9.12.2/ouroboros-consensus-3.0.1.0/b/leios-txcache-bench/build/leios-txcache-bench/leios-txcache-bench [Objects changed]
Running 1 benchmarks...
Benchmark leios-txcache-bench: RUNNING...
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 3011
total txs : 385408
generating data... done
populating index... done
== pure-wrapped index ==
population:
time : 1.62 s
allocated : 953.6 MiB
alloc rate : 589.8 MiB/s
residency (post major GC):
live : 73.6 MiB
max live : 73.6 MiB
peak mem in use : 186.0 MiB
lookup — EB batch of 3011 (x20 each):
0% hits (0/3011): avg 2.57 ms, per-tx 852.6 ns, alloc 602.8 KiB
20% hits (603/3011): avg 2.25 ms, per-tx 746.9 ns, alloc 601.9 KiB
40% hits (1205/3011): avg 2.13 ms, per-tx 706.1 ns, alloc 602.2 KiB
60% hits (1807/3011): avg 2.04 ms, per-tx 676.9 ns, alloc 603.0 KiB
80% hits (2409/3011): avg 1.97 ms, per-tx 655.3 ns, alloc 755.2 KiB
100% hits (3011/3011): avg 1.93 ms, per-tx 641.6 ns, alloc 805.8 KiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
1.87 ms 1.83 ms 1.64 ms 1.60 ms 1.51 ms 1.46 ms
1.92 ms 1.85 ms 1.66 ms 1.64 ms 1.52 ms 1.47 ms
1.95 ms 1.90 ms 1.66 ms 1.67 ms 1.60 ms 1.49 ms
1.99 ms 1.92 ms 1.74 ms 1.73 ms 1.63 ms 1.54 ms
2.09 ms 1.92 ms 1.87 ms 1.73 ms 1.71 ms 1.67 ms
2.18 ms 1.93 ms 1.91 ms 1.77 ms 1.76 ms 1.74 ms
2.31 ms 1.94 ms 1.92 ms 1.80 ms 1.78 ms 1.78 ms
2.35 ms 2.00 ms 1.96 ms 1.85 ms 1.84 ms 1.85 ms
2.38 ms 2.06 ms 2.02 ms 1.89 ms 1.85 ms 1.88 ms
2.51 ms 2.11 ms 2.13 ms 1.91 ms 1.88 ms 1.89 ms
2.54 ms 2.18 ms 2.14 ms 2.00 ms 1.89 ms 1.90 ms
2.57 ms 2.20 ms 2.15 ms 2.03 ms 1.96 ms 1.94 ms
2.59 ms 2.35 ms 2.16 ms 2.08 ms 1.97 ms 1.98 ms
2.72 ms 2.40 ms 2.16 ms 2.15 ms 2.02 ms 2.04 ms
2.78 ms 2.41 ms 2.20 ms 2.16 ms 2.05 ms 2.07 ms
2.79 ms 2.47 ms 2.34 ms 2.20 ms 2.07 ms 2.11 ms
3.05 ms 2.47 ms 2.35 ms 2.25 ms 2.07 ms 2.15 ms
3.15 ms 2.63 ms 2.37 ms 2.25 ms 2.11 ms 2.19 ms
3.18 ms 2.81 ms 2.67 ms 2.57 ms 2.41 ms 2.34 ms
4.43 ms 3.59 ms 3.47 ms 3.49 ms 3.83 ms 3.16 ms
Benchmark leios-txcache-bench: FINISH
Contributor
Author
There was a problem hiding this comment.
And:
$ (cd ../ouroboros-consensus/; git diff)
diff --git a/ouroboros-consensus/bench/leios-txcache-bench/Main.hs b/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
index 5abb36f7f..c87b81415 100644
--- a/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
+++ b/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
@@ -49,7 +49,7 @@ numEbs = maxAnnouncementCount
-- hash + 2-byte size) in a compact, non-CBOR layout — the encoding-independent
-- ceiling. (The current CBOR-precise 'LeiosDemoTypes.maxTxsPerEb' is 13888.)
txsPerEb :: Int
-txsPerEb = 15_058
+txsPerEb = 15_058 `div` 5
-- | Timed repetitions of the batch lookup (plus one warmup).
numLookupRuns :: Int
@@ -103,10 +103,10 @@ main = do
runs :: [(String, IO BenchCache)]
runs = case args of
["pure"] -> [("pure-wrapped index", newPureLeiosTxCache)]
- ["ht"] -> [("hash-table (shift 22)", newHashTableLeiosTxCache 22 salt0 salt1)]
+ ["ht"] -> [("hash-table (shift 20)", newHashTableLeiosTxCache 20 salt0 salt1)]
_ ->
[ ("pure-wrapped index", newPureLeiosTxCache)
- , ("hash-table (shift 22)", newHashTableLeiosTxCache 22 salt0 salt1)
+ , ("hash-table (shift 20)", newHashTableLeiosTxCache 20 salt0 salt1)
]
mapM_ (uncurry runBench) runs
$ cabal bench leios-txcache-bench --benchmark-options=ht
Configuration is affected by the following files:
- cabal.project
Build profile: -w ghc-9.12.2 -O1
In order, the following will be built (use -v for more details):
- ouroboros-consensus-3.0.1.0 (bench:leios-txcache-bench) (first run)
Created semaphore called cabal_semaphore_c with 5 slots.
Preprocessing benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Building benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Running 1 benchmarks...
Benchmark leios-txcache-bench: RUNNING...
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 3011
total txs : 385408
generating data... done
populating index... done
== hash-table (shift 20) ==
population:
time : 146.86 ms
allocated : 190.2 MiB
alloc rate : 1.26 GiB/s
residency (post major GC):
live : 52.5 MiB
max live : 81.1 MiB
peak mem in use : 111.0 MiB
lookup — EB batch of 3011 (x20 each):
0% hits (0/3011): avg 358.67 µs, per-tx 119.1 ns, alloc 602.8 KiB
20% hits (603/3011): avg 341.45 µs, per-tx 113.4 ns, alloc 602.3 KiB
40% hits (1205/3011): avg 361.91 µs, per-tx 120.2 ns, alloc 602.6 KiB
60% hits (1807/3011): avg 290.13 µs, per-tx 96.4 ns, alloc 602.9 KiB
80% hits (2409/3011): avg 401.07 µs, per-tx 133.2 ns, alloc 468.7 KiB
100% hits (3011/3011): avg 282.80 µs, per-tx 93.9 ns, alloc 448.8 KiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
297.48 µs 277.78 µs 273.22 µs 274.18 µs 317.12 µs 274.50 µs
297.75 µs 280.25 µs 273.76 µs 274.65 µs 318.08 µs 274.63 µs
298.18 µs 285.64 µs 309.87 µs 274.67 µs 319.24 µs 275.14 µs
298.24 µs 301.16 µs 314.82 µs 275.57 µs 319.61 µs 275.24 µs
301.19 µs 308.46 µs 315.06 µs 276.55 µs 328.44 µs 275.45 µs
306.88 µs 314.30 µs 315.61 µs 277.53 µs 329.10 µs 275.56 µs
310.56 µs 314.90 µs 315.66 µs 278.09 µs 341.58 µs 276.29 µs
312.27 µs 315.50 µs 315.93 µs 279.55 µs 346.31 µs 276.71 µs
322.71 µs 316.06 µs 317.01 µs 281.43 µs 351.39 µs 276.98 µs
331.28 µs 319.27 µs 317.02 µs 285.00 µs 351.81 µs 277.36 µs
336.50 µs 324.67 µs 325.87 µs 285.41 µs 358.78 µs 278.16 µs
340.59 µs 334.38 µs 331.20 µs 285.86 µs 361.42 µs 278.45 µs
346.26 µs 336.06 µs 339.37 µs 287.26 µs 368.83 µs 279.72 µs
356.32 µs 341.74 µs 345.32 µs 289.20 µs 381.43 µs 281.43 µs
370.36 µs 355.89 µs 392.86 µs 293.70 µs 418.50 µs 282.09 µs
399.16 µs 362.99 µs 424.78 µs 297.65 µs 482.84 µs 286.69 µs
424.08 µs 396.84 µs 462.74 µs 297.68 µs 497.48 µs 288.40 µs
451.03 µs 434.97 µs 507.09 µs 300.26 µs 596.73 µs 300.49 µs
475.05 µs 436.00 µs 507.70 µs 323.20 µs 603.24 µs 302.11 µs
597.61 µs 472.21 µs 533.34 µs 365.24 µs 629.54 µs 320.61 µs
Benchmark leios-txcache-bench: FINISH
Contributor
Author
There was a problem hiding this comment.
The Reference numbers at 20% cardinality are eerily close to the Optimized numbers at 100% cardinality.
nfrisby
commented
Aug 6, 2026
| action `catch` \e -> traceWith tracer (toTrace e) >> throwIO e | ||
|
|
||
| {------------------------------------------------------------------------------- | ||
| Shadow LeiosTxCache wiring |
Contributor
Author
There was a problem hiding this comment.
Issue input-output-hk/ouroboros-leios#1021 includes
Show what is better with the transaction cache
This PR patently can't do that by observing a running node, since it doesn't alter the LeiosFetch decision logic to use LeiosTxCache instead of scanning querying the LeiosDb. (I'd like to defer that to a separate PR... but maybe it'll just end up being a small commit eventually added to this PR.)
But: perhaps I can add yet another variant to leios-txcache-bench/Main.hs that runs via the SQLite-backed LeiosDb?
Contributor
Author
There was a problem hiding this comment.
commit e7771e77f163b3d4bb7334f8965518a553a7b9b7
Author: Nicolas Frisby <nicolas.frisby@moduscreate.com>
Date: Thu Aug 6 20:46:06 2026 -0400
LeiosTxCache: also benchmark a SQLite impl
On my machine
$ lsblk -d -o NAME,MODEL,SIZE,ROTA,TRAN && findmnt -no SOURCE,FSTYPE -T .
NAME MODEL SIZE ROTA TRAN
nvme0n1 WD_BLACK SN770 2TB 1.8T 0 nvme
/dev/nvme0n1p2 ext4
I see the following:
┌──────────────────────┬───────────┬────────┐
│ │ no reopen │ reopen │
├──────────────────────┼───────────┼────────┤
│ cache small (512 kB) │ 292 ms │ 294 ms │
├──────────────────────┼───────────┼────────┤
│ cache big (256 MB) │ 17 ms │ 339 ms │
└──────────────────────┴───────────┴────────┘
This is for the 1.9 million txs case and using sudo so fadvise(DONT_NEED) works.
If I reduce the tx count to 20%, the averages are are reduced to 20%.
Contributor
Author
There was a problem hiding this comment.
Claue claims that those numbers match that SSD's expectations and that no (reasonable?) SSD would be better for SQLite
Contributor
Author
There was a problem hiding this comment.
The high-level summary: SQLite relies on two caches, its own page cache and the operating system's page cache (ie Virtual Memory).
If we neuter either of them (and it's not at all clear that the adversary can't send messages that would spoil either/both caches), then the latency for checking a single max-tx-count EB against a full SQLite-backed LeiosTxCache is intolerable.
Even if we neuter neither cache, the 17 ms time isn't great.
nfrisby
commented
Aug 8, 2026
| -- | ||
| -- Solution: continue to rely on there being hours of slack, but moreover | ||
| -- actively ensure that slack. In particular, evict EBs as they get "too old", | ||
| -- regardless of whether new EBs have been arriving. For example, the |
Contributor
Author
There was a problem hiding this comment.
TODO reconsider this
For example, the LeiosTxCacheIndex should evict any EBs that are older than the youngest X RBs on the current selection, for X≥128.
The assumption here is that it always takes longer the immutable tip to advance by k-X blocks than it does for the LeiosFetch logic to finish reading txs from the LeiosDb once it successfully looked them up in the LeiosTxCacheIndex.
I'm wavering on whether that's true.
- For a caught-up and well-connected
mainnetnode, lengthening the chain by k-X takes a very long time. - But, if the node were eclipsed for a while and then manages to escape it, then it'll suddenly see many blocks. If they're all empty, then maybe it could acquire and select k-X blocks "very quickly". ... But faster than the LeiosFetch logic could finish its LeiosDb reads? Seems very unlikely, as long as k-X is great enough.
- And on
mainnet, k-X is at least 1500 blocks, so that does seem feasible.
But what about testnets, where k-X might be smaller? And also what about a syncing mainnet node; is that qualitatively different than the node-escaping-eclipse above?
These questions make it tempting to reconsider the other solutions: either LeiosFetch detecting and recovering from hit-prune faults or else explicit pinning.
- Maybe the recovery isn't that complicated: do a first pass on the EbBody to find the txs that are not in the LeiosTxCacheIndex, and issue jobs for those. Also spawn a read of the backing store for the txs that were in the index. Once that read finishes, issue more jobs for any txs whose read failed. Still monotonic, just two-step instead of all at once.
- How could the adversary weaponize the extra delay for that second batch of LeiosFetch jobs being issued?
- Each individual pin should be very brief; only for the duration of reading some of a single EB's txs from the LeiosDb.
- The pinning has to prevent the LeiosDb from deleting the txs---it's not even a restriction on the LeiosTxCache. That coupling seems undesirable, and the only way to avoid it would be to have the LeiosTxCache's backing store be independent. That's doable, but more engineering.
- If those reads are pipelined and overlap for a sustained period, that might extend the duration significantly. And while something is pinned, we either tolerate that the maximum size of the backing store being temporarily exceeded, or else we block insertions? Neither sounds very appealing.
- Perhaps relevant observation: EbBodies can arrive in sustained bursts, but their average over longer time frames (hours) will be close to f=0.05 arrival per slot. So, on average, there will often be nothing pinned.
Contributor
Author
There was a problem hiding this comment.
Maybe the recovery isn't that complicated
In fact... maybe recovery = do nothing?
- This PR reuses the LeiosDb as the backing store. So: if the txs aren't there, then no volatile EB references them.
- The LeiosFetch logic should only be fetching EBs that this node considers to be volatile.
- So: if the txs aren't there, then LeiosFetch doesn't need them; the EB that lead to those reads can't still be volatile.
Edit: ah. The above argument only works if the LeiosDb is deduping txs and promptly, which it currently does. The point: if we're fetching txs for some EB, then that body is already in the LeiosDb, which means it's keeping all of its txs we already have alive. In that case, LeiosFetch doesn't even need to read the txs that the index claims are present, so there's no read that can fail. So there is no possibility of hit-prune fault: neither LeiosFetch nor LeiosVote ever read the bytes of the txs that the LeiosTxCacheIndex claims are present. The hit-prune fault still exists, it just doesn't manifest as promptly as I think. If the index is stale such that it still contains some EB that was already GC'd out of the LeiosDb, then a hit might prevent some txs from being fetched that need to be fetched. That either causes the EB closure to never be complete or it causes it to falsely claim to be complete---both are hit-prune faults.
Only if the LeiosDb weren't deduping txs would LeiosFetch need to read the txs the LeiosTxCacheIndex claims to be present, so that it can copy (ie "dup") them to the EB it's currently fetching. In that case, the source EB could be older than the destination EB, in which case the hit-prune fault could happen and would need to be recovered from (by refetching the txs that we just GC'd from some peers).
Edit: A meta-level point: I'll have more clarity here once I've actually implemented the LeiosFetch logic---the reasoning depends on when exactly various things happens, and I can't see that all yet.
Contributor
Author
There was a problem hiding this comment.
OK, here's my summary of the last couple "thinking out loud comments" (I'm writing these so I'll think about them on Monday, but I'm failing to stop thinking about them now, on Sunday :/.)
- LeiosTxCacheIndex hits should prevent LeiosFetch from immediately sending a request to peers for those txs.
- But, unless we trust the "hours of slack" argument or we add explicit synchronization, then we need a detection and recovery mechanism for the case were the cache index hits were stale: we actually do need to fetch those txs from peers.
- Suppose X was a tx that had a cache hit.
- LeiosFetch will immediately sends requests for the txs that weren't cache hits ✔️
- LeiosFetch should now double-check that X is in the LeiosDb
- If it is, then suppressing the requests for X was correct ✔️
- If it isn't, then now we do need to send those requests after all. They're slightly delayed compared to if the cache hadn't been stale but this case is a rarity (TODO confirm adversary can't induce it).
- The LeiosFetch bookkeeping can still be monotonic, it just has to be slightly more complicated.
- Both cache hits and cache misses lead to jobs being created, but they start as separate jobs: jobs for peer versus jobs for the "double-check" logic. Key point: the total number of outstanding jobs can already be set to the greatest value it'll ever have.
- The double-check logic processes its jobs by converting them into jobs-for-peers whenever the cache hit turned out to be stale. There can be at most one converted job per original job, so the double-check logic can cleverly decrement the outstanding-job count to keep it accurate.
- A simple version of the rule: initialize the outstanding-jobs counter with one double-check job per tx with a cache hit.
- On each confirmed-present tx, decrement the outstanding job count (can be batched).
- On each confirmed-absent tx, add it to an accumulating for-peers job, batched/flushed on some practical heuristics.
- On each flush, issue that for-peers jobs and decrement the outstanding-job count by one less than the number of txs in that new for-peers job.
- A simple version of the rule: initialize the outstanding-jobs counter with one double-check job per tx with a cache hit.
Contributor
Author
There was a problem hiding this comment.
Hrmm...
- Assume the LeiosDb is deduping.
- In particular, we're assuming that once the EbBody is inserted into the LeiosDb, any of its txs that are already in the database cannot be GC'd before this EB is GC'd.
- Assume that the LeiosDb pruning function's first step is to prune the LeiosTxCacheIndex.
- Thus, the cache never falsely claims some tx that was pruned out is present.
- Now, suppose the EbBody arrival handler does these steps in the following order.
- Insert EbBody into LeiosDb.
- Insert EbBody into LeiosTxCacheIndex.
- Issue fetch requests to peers for the EbBody's txs that are in the TxNotYetInserted state in the LeiosTxCacheIndex.
I think that's enough. I'm not seeing what could go wrong.
The key point is: the EB itself is now keeping those txs alive. So the only way reads of those txs could fail is if the EB itself was GC'd... but that means those fails no longer matter; the code that was expecting the read results should also abort, since it's processing an orphaned EB.
There's one downside: a deduping insert like that is going to involve non-trivial on-disk latency, so the requests to peers would be delayed by at least that much.
However, it does seem like a reasonable place to start: no locking and no recovery.
MOREOVER, the latency might not be that bad: the general expectation is that EbBody's will arrive before peers offer the corresponding EbClosure. So as long as this work finishesd before the offers arrive, the "injected latency" is hidden. (The "Issue fetch requests to peers" above is misleading: the EbBody handler could only do that when those peers had already offer that EB's closure, which should be a rare case: EbBodys should usually diffuse faster than their closures... unless the closures are already present, in which case the truly slow part of diffusion already doesn't exist.)
Edit: it'd require some more complexity, but it should be safe to offer the EbBody to our downstream peers as soon as we've confirmed its hash&size match. It's possible their request might arrive before we've finished inserting it into the LeiosDb---that's the complex case we'd have to support. For example, we could insert it into an in-memory store before we start the LeiosDb write and then delete it from that store when the write is over. Our LeiosFetch servers would handle a request by checking the in-memory store first and then falling through to the LeiosDb. Something like that 🤷 would be able to prevent the LeiosDb latency from slowing down the diffusion of EbBodies.
nfrisby
force-pushed
the
nfrisby/leios-first-txcache-increment
branch
2 times, most recently
from
August 14, 2026 16:32
cbadcca to
9b740fe
Compare
nfrisby
force-pushed
the
nfrisby/leios-first-txcache-increment
branch
from
August 24, 2026 22:11
9b740fe to
871920e
Compare
ch1bo
marked this pull request as ready for review
August 28, 2026 08:19
ch1bo
self-requested a review
August 28, 2026 08:20
ch1bo
approved these changes
Aug 28, 2026
ch1bo
left a comment
ch1bo
left a comment
Contributor
There was a problem hiding this comment.
I have not read this code! However, I was using the API and running this a lot while working on #2235, including several night long runs and stress tests. I might be able to review this today, but would also be comfortable with merging to the leios-prototype as-is.
ch1bo
force-pushed
the
nfrisby/leios-first-txcache-increment
branch
from
August 28, 2026 08:59
871920e to
229034a
Compare
We switch the benchmark's body from Vector TxHash to ByteString in order to match the node's behavior. The Vector TxHash does improve the pure-wrapper's residency, since the Map's keys can be shared with the Vector's elements. Maybe, if the Map were preferred for the real node, then it might be worth the boxing overhead? But for now, we're simply storing the bytes of the body itself, and so the benchmark will do the same.
With these in place, the benchmark outputs are as follows; two separate runs.
In the worst-case scenario, the hash table eliminates ~80% of insertion
allocation, ~120 MiB heap footprint, 95% of ~20 ms per EB-sized batch of
lookups, and _all traversal_ during a major GC.
```
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 15058
total txs : 1927424
generating data... done
populating index... done
== pure-wrapped index ==
population:
time : 14.00 s
allocated : 4.55 GiB
alloc rate : 332.5 MiB/s
residency (post major GC):
live : 367.8 MiB
max live : 367.8 MiB
peak mem in use : 878.0 MiB
lookup — EB batch of 15058 (x20 each):
0% hits (0/15058): avg 26.46 ms, per-tx 1757.4 ns, alloc 2.1 MiB
20% hits (3012/15058): avg 26.03 ms, per-tx 1728.4 ns, alloc 2.3 MiB
40% hits (6024/15058): avg 25.87 ms, per-tx 1718.1 ns, alloc 2.5 MiB
60% hits (9036/15058): avg 24.49 ms, per-tx 1626.3 ns, alloc 3.0 MiB
80% hits (12047/15058): avg 23.48 ms, per-tx 1559.1 ns, alloc 3.1 MiB
100% hits (15058/15058): avg 21.99 ms, per-tx 1460.2 ns, alloc 3.3 MiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
25.55 ms 25.16 ms 24.36 ms 23.34 ms 22.58 ms 21.65 ms
25.70 ms 25.21 ms 24.37 ms 23.47 ms 22.59 ms 21.69 ms
25.71 ms 25.22 ms 24.70 ms 23.48 ms 22.60 ms 21.74 ms
25.72 ms 25.23 ms 24.71 ms 23.53 ms 22.61 ms 21.75 ms
25.78 ms 25.24 ms 24.78 ms 23.55 ms 22.64 ms 21.75 ms
25.78 ms 25.28 ms 24.78 ms 23.56 ms 22.83 ms 21.77 ms
25.82 ms 25.33 ms 25.08 ms 23.59 ms 22.92 ms 21.77 ms
25.88 ms 25.43 ms 25.12 ms 23.60 ms 23.05 ms 21.78 ms
25.93 ms 25.44 ms 25.19 ms 23.61 ms 23.20 ms 21.80 ms
26.02 ms 25.46 ms 25.30 ms 23.63 ms 23.32 ms 21.83 ms
26.16 ms 25.62 ms 25.34 ms 23.79 ms 23.36 ms 21.83 ms
26.20 ms 25.79 ms 25.42 ms 24.13 ms 23.45 ms 21.90 ms
26.29 ms 25.79 ms 25.89 ms 24.55 ms 23.56 ms 21.91 ms
26.38 ms 26.00 ms 26.15 ms 24.76 ms 23.64 ms 21.95 ms
26.40 ms 26.12 ms 27.42 ms 24.81 ms 23.67 ms 21.98 ms
26.47 ms 26.24 ms 27.48 ms 25.54 ms 23.68 ms 22.19 ms
27.15 ms 26.66 ms 27.63 ms 26.25 ms 23.73 ms 22.35 ms
28.58 ms 26.73 ms 27.79 ms 26.31 ms 23.76 ms 22.50 ms
28.64 ms 27.71 ms 27.84 ms 26.88 ms 25.82 ms 22.55 ms
29.09 ms 30.87 ms 28.06 ms 27.40 ms 26.52 ms 23.07 ms
```
```
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 15058
total txs : 1927424
generating data... done
populating index... done
== hash-table (shift 22) ==
population:
time : 846.36 ms
allocated : 840.5 MiB
alloc rate : 993.1 MiB/s
residency (post major GC):
live : 219.9 MiB
max live : 321.1 MiB
peak mem in use : 429.0 MiB
lookup — EB batch of 15058 (x20 each):
0% hits (0/15058): avg 1.75 ms, per-tx 116.5 ns, alloc 2.5 MiB
20% hits (3012/15058): avg 1.81 ms, per-tx 120.2 ns, alloc 2.5 MiB
40% hits (6024/15058): avg 1.68 ms, per-tx 111.5 ns, alloc 2.7 MiB
60% hits (9036/15058): avg 1.73 ms, per-tx 115.0 ns, alloc 2.7 MiB
80% hits (12047/15058): avg 1.82 ms, per-tx 120.7 ns, alloc 2.9 MiB
100% hits (15058/15058): avg 1.82 ms, per-tx 120.9 ns, alloc 2.9 MiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
1.44 ms 1.53 ms 1.54 ms 1.56 ms 1.62 ms 1.57 ms
1.46 ms 1.53 ms 1.55 ms 1.58 ms 1.63 ms 1.57 ms
1.46 ms 1.54 ms 1.55 ms 1.58 ms 1.64 ms 1.58 ms
1.46 ms 1.54 ms 1.56 ms 1.59 ms 1.65 ms 1.58 ms
1.47 ms 1.55 ms 1.57 ms 1.59 ms 1.65 ms 1.61 ms
1.48 ms 1.55 ms 1.57 ms 1.59 ms 1.67 ms 1.61 ms
1.49 ms 1.55 ms 1.58 ms 1.60 ms 1.68 ms 1.62 ms
1.50 ms 1.56 ms 1.59 ms 1.61 ms 1.69 ms 1.62 ms
1.51 ms 1.56 ms 1.59 ms 1.61 ms 1.70 ms 1.63 ms
1.51 ms 1.58 ms 1.60 ms 1.61 ms 1.71 ms 1.63 ms
1.53 ms 1.59 ms 1.60 ms 1.62 ms 1.72 ms 1.65 ms
1.57 ms 1.59 ms 1.60 ms 1.63 ms 1.75 ms 1.65 ms
1.57 ms 1.60 ms 1.61 ms 1.63 ms 1.78 ms 1.71 ms
1.63 ms 1.64 ms 1.62 ms 1.64 ms 1.80 ms 1.72 ms
1.65 ms 1.67 ms 1.62 ms 1.69 ms 1.81 ms 1.75 ms
2.06 ms 1.70 ms 1.65 ms 1.70 ms 1.86 ms 1.93 ms
2.12 ms 1.98 ms 1.68 ms 1.74 ms 1.88 ms 1.96 ms
2.41 ms 2.25 ms 1.68 ms 1.98 ms 1.89 ms 2.08 ms
2.86 ms 2.43 ms 2.00 ms 2.23 ms 2.23 ms 2.54 ms
2.91 ms 4.25 ms 2.82 ms 2.85 ms 3.00 ms 3.38 ms
```
The LeiosTxCache assumes the announcement has been inserted before its body is inserted. Similarly, the design of EB diffusion assumes upstream peers will send the announcement of a body before they offer that body. The implementation _currently_ doesn't enforce that. And so PR #2132 didn't notice that offers were being sent _before_ announcements. Part of the reason offers were being sent before announcements is because the forge loop itself was writing the EB body to disk. This commit instead has the forge loop innards return the EB so that the outer logic in one place can explicitly send the LeiosNotify announcement _before_ the body is written to the store, since writing the body to the store _triggers_ sending the offer via LeiosNotify. That's the order that the EB diffusion design requires. And it's _also_ the order that TxCache insertion requires, so that same code also does that now.
PR #2132 didn't do this, to keep its scope limited. However, for the same reasons as the preceding "Leios Forge: rearrange storing/announcing/caching" commit, the LeiosTxCache's assumptions about event ordering motivate adding this now. This commit refines the "peer" field of TraceLeiosAnnouncementAccepted to also clarify which mini protocol the announcement arrived via.
On my machine ``` $ lsblk -d -o NAME,MODEL,SIZE,ROTA,TRAN && findmnt -no SOURCE,FSTYPE -T . NAME MODEL SIZE ROTA TRAN nvme0n1 WD_BLACK SN770 2TB 1.8T 0 nvme /dev/nvme0n1p2 ext4 ``` I see the following: ┌──────────────────────┬───────────┬────────┐ │ │ no reopen │ reopen │ ├──────────────────────┼───────────┼────────┤ │ cache small (512 kB) │ 292 ms │ 294 ms │ ├──────────────────────┼───────────┼────────┤ │ cache big (256 MB) │ 17 ms │ 339 ms │ └──────────────────────┴───────────┴────────┘ This is for the 1.9 million txs case and using sudo so fadvise(DONT_NEED) works. If I reduce the tx count to 20%, the averages are are reduced to 20%.
Also, float the initialization of LeiosTxCache out to before ChainDB, so we can pass a callback into ChainDB.
Output of scripts/ci/run-fourmolu.sh and scripts/ci/run-cabal-gild.sh, which CI enforces. Mostly the layout of multi-line SPECIALISE pragmas, which fourmolu also spells SPECIALIZE, plus stanza spacing in the cabal file.
ch1bo
force-pushed
the
nfrisby/leios-first-txcache-increment
branch
from
August 28, 2026 19:06
5f4ed28 to
f4438c8
Compare
github-project-automation
Bot
moved this from 👀 In review
to ✅ Done
in Consensus Team Backlog
ch1bo
added a commit
that referenced
this pull request
Aug 29, 2026
…#2235, #2245) Merges the stacked series, bottom to top: * #2188 Leios: add LeiosTxCache, but don't rely on it at all yet * #2237 Rewrite the LeiosFetch decision logic * #2235 Leios prototype: EB validation in vote logic * #2245 LeiosDb: bound the WAL, remove the write hotspot, stop dying on lock contention IMPORTANT: the schema change in #2245 has no migration. An existing database is not readable by this version and requires a wipe & resync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is half of the implementation of Issue input-output-hk/ouroboros-leios#1021. The second half will be for LeiosFetch (and LeiosVoting, once it exists) to actually use the LeiosTxCache.
Suggestions:
LeiosTxCacheheader for a summary of the new component.cardano-crypto. But it still maybe deserves review by one of our crypto experts, since if it's attackable then our hash table's performance is unpredictable.