Repository navigation
zu2: memory_pages bounds nothing while an async write is going, and the overshoot grows with the load #775
Description
Options::memory_pages says it is "pages of anonymous memory the log may hold". Under Durability::Async it is not a bound on anything. It is a target the flusher works towards afterwards, and while the writing is going the log holds whatever the writer has appended.
Measured with a probe that samples resident_pages on a thread while a single session writes 8 KiB values, memory_pages: 2, mutable_pages: 1, compact_below: 0 so nothing but eviction moves the head:
| durability | records | written | bound | peak while writing |
|---|---|---|---|---|
| Async | 10000 | 80 MiB | 8 MiB | 48 MiB, 12 pages |
| Async | 40000 | 320 MiB | 8 MiB | 240 MiB, 60 pages |
| Durable | 40000 | 320 MiB | 8 MiB | 12 MiB, 3 pages |
Four times the load is five times the overshoot, so this is not a constant lag with a ceiling, it is the whole write staying in memory. At Durable the same load holds three pages, which is the bound plus the page being appended to and is exactly what the option promises.
The mechanism is not subtle once you look at it. Log::ensure_page allocates a page whenever an append reaches one, and it consults nothing:
let fresh = unsafe { alloc_zeroed(page_layout()) };
...
slot.store(fresh, Ordering::Release);
self.anonymous.fetch_add(1, Ordering::AcqRel);The only thing that ever gives a page back is evict_behind, which runs from evict_settled, which is the flusher's. And evict_behind cannot take a page until it is durable:
if page_start(victim + 1) > self.flushed() {
break;
}which is right, a page that left memory before its bytes were on the device would be a reader preading a hole. So at Durable the writer is its own back pressure, because it does not return until the bytes are down, and eviction always has something to take. At Async nothing connects the writer to the flusher at all, and the log grows in memory at whatever rate the CPU can append.
The existing test knows this and works around it. the_page_bound_holds_after_the_writing_stops in crates/zu2/tests/end_to_end.rs is named for it and asserts after drop(s) and db.sync(), with a comment that what is being waited for is a thread noticing. Nothing asserts anything about during.
Why this matters beyond the doc being wrong. #767 is heading towards making a bound the default, since the anonymous total at defaults is the whole log and memory_pages is usize::MAX. A bound that a bulk load walks straight through would give a default that looks right in the options and does nothing for the load phase of every benchmark we run, which is the phase where the memory figure is worst.
The fix is the one the log already uses for its other ceiling. A writer that fills the log waits for a compaction pass, a_writer_that_fills_the_log_waits_for_a_pass in db.rs covers it. The same shape applies here: an append that would take the anonymous count past memory_pages wakes the flusher and waits for it, rather than allocating and moving on. That turns the option into what it says, at the cost of turning a write burst that outruns the device into a wait, which is what a bound means.
- Make the append path respect the bound rather than the flusher chase it
- A test that samples during the write and not after it, with the numbers above as the thing it catches
- Say in
Options::memory_pageswhich durability modes it holds under, until it holds under both - Re-measure, because a writer that waits for the flusher is a throughput change and it has to be on a host
Numbers above are from the laptop, and they are a behaviour observation rather than a throughput one: what is being claimed is that the peak scales with the load, which it does by a factor of five across a factor of four. Nothing published comes from this.
Activity
zu2: a writer over the memory bound waits for the flusher
tamnd commentedon Aug 25, 2026
Fixed in 54c9359.
One correction to what I wrote above: this was not an oversight, it was a stated position.
Log::evict_settled's own doc says it, and has since #636:The numbers are the argument against that last sentence. It is fine if the overshoot has a ceiling. It does not: four times the load gave five times the overshoot, so what "while the burst is going on" costs is the burst, and a load phase is one long burst.
What went in.
Log::throttle, called fromallocatebefore the claim: an append that finds the anonymous count over the bound wakes the flusher, takes whatever is already durable withevict_behind, and then waits on thesyncedcondition variable that a landed device write signals. Same load, same options, peak goes from 60 pages to 3.Where it is called from matters more than what it does. It runs before
slot.appending_at, so the writer has no frontier published while it waits. Anywhere below that and a writer waiting for a flush would be waiting for the flusher to wait for it, sincesync_rangecallswait_for_writerson everything below its target. That rules out putting this inopened_page, which is where eviction already runs and which was the obvious place to reach for.Two other things it does not do. It never waits without a ceiling: twenty turns of half a millisecond each, then the writer takes the memory anyway. A database that holds too much for a moment is worse; one whose writers cannot be woken is not a database. And it does not throttle the engine's own sessions, since compaction copies a record to the tail before it can reclaim the page the record was in, so holding a pass because memory is tight holds the one thing that makes memory less tight.
the_page_bound_holds_while_the_writing_is_goingincrates/zu2/tests/end_to_end.rs, with a ceiling of eight pages against a load of eighty, which is set where it is because the writer's patience does run out and the claim being tested is that the overshoot does not scale with the loadOptions::memory_pagesneeded nothing after allThe cost, and it is the point rather than a regression: the same load went from 668 ms to 1.25 s on the laptop, an 8 MiB bound forcing a device write every 8 MiB. Laptop, so that is a shape and not a number. The four arm A/B queued on server3 for #757 is the run that says what it is worth, and it now measures this too.
tamnd commentedon Aug 25, 2026
Followed the counter through to the process, because everything above is
resident_pages, andresident_pagescounts slots rather than bytes. A page that eviction has taken out of its slot is not counted from the moment of the swap, but its memory does not go back until the epoch passes, so in principle the count could fall while the process holds the same memory.It does not. Same test, same options, run alone under
/usr/bin/time -lon the laptop:Peak footprint is the number to read here, since max rss on macOS carries the binary and the harness with it. 116.3 MiB to 22.1 is 5.3x, against 320 MiB of writes at an 8 MiB bound, and it says the retirement queue is not quietly holding what the counter says has gone. That is what one would expect during a write burst, since every session operation moves the epoch on and the deferred frees drain behind it, and it is worth having measured rather than assumed: a long running scan pins an epoch and would let the queue build, which is a property of the design that predates this and is not what this changes.
The run without the throttle also fails the test, which is the point of the test: