[wchwawa]

TLDR

A write that fails after it has already changed cached frames now latches the database closed, so the partial change can never reach a checkpoint. Before, a cross-tree atomic batch that failed part-way could become half durable, and NoKV's metadata store then refused to open. The 0.8 line gets the same fix in #83.

Problem

DB::atomic applies each tree's ops to cached frames under the commit gate and then submits one WAL record. When an op of a later tree fails (an allocation error, a blob read error) or the submit fails, the earlier ops stay applied in memory with no WAL record. The call reports AtomicErrorKind::OutcomeUnknown, but the next checkpoint round snapshots every dirty frame and writes it. Tree batches and single put, delete and rename whose WAL submit fails have the same shape, and so does a rename whose insert fails after its erase.

Consequences

After a reopen, one tree holds its half of a batch without the other. Main has the same apply-then-submit shape, and NoKV 0.12.x on Holt 0.8.7 hit it: an OutOfSpace ("need 520 bytes, 512 available", a Node256 allocation) in one keyspace of a metadata batch left the recovery tail one log sequence number ahead of the outbox on disk, and the metadata store refused to open until an operator intervened.

Example

A two-tree batch whose second tree fails part-way. Before: after a checkpoint and a reopen, the first tree has its keys and the second does not. After: the batch still fails with OutcomeUnknown; every later write, DB::checkpoint and background round fails with Error::FailedClosed; after a reopen neither tree has the batch.

Solution

BufferManager gains a failed_closed latch. A write that may have changed cached frames before failing sets it while it still holds the commit gate: an atomic DB batch whose apply or submit fails, a tree batch, a single put, delete or rename whose WAL submit fails, and a rename whose insert fails after its erase. Main's batches that carry an attached journal envelope latch the same way on the file WAL and on the memory stream, and never append their envelope after a failure. Every write checks the latch first, and every path that persists dirty state checks it: background checkpoint rounds, DB and tree checkpoint captures with and without a WAL (under the commit gate where one is held), inline flushes, and the final round on drop. The new Error::FailedClosed says to reopen. Reopening recovers the last durable state.

The checkpoint planner stops scheduling rounds once it sees the latch, drains the epochs in flight, and logs the refusal once. A write that queued behind the failing writer re-checks the latch once it holds the gates that serialize it, before it evaluates a condition or changes a frame: DB atomic batches including attached ones after their tree gates, tree batches and put_many_if_absent after the mutation gate, single put, delete and rename (conditional and deferred) after their endpoint locks, and create_tree, drop_tree, the dropped-tree finalizer and initialize_journal_stream after the maintenance gate.

Gain

A failed write can no longer become partially durable; the database refuses to continue instead, and a reopen recovers.

Tradeoffs

The latch stops writes and checkpoints of the whole database, not only the tree that failed, until a reopen: a failure that used to be silent corruption becomes an outage that needs a restart. Reads keep serving the in-memory state, which may include the partial change, until the reopen; an application that cannot tolerate that should stop when it sees FailedClosed or OutcomeUnknown. NoKV already does: its store poisons itself on OutcomeUnknown and refuses reads and writes until it is reopened. What it lacked was the guarantee that Holt's own background checkpoint would not persist the partial batch before that reopen. A latched store skips the final checkpoint on drop, so with asynchronous WAL acknowledgements the records acknowledged before the failure that still sit in the WAL writer's buffer are lost, as they already are when the checkpointer is off; handing that buffer to the OS on stop would change a trade-off three existing tests assert, and is left for a separate decision. The OutOfSpace itself is not fixed: it was not reproduced by targeted spillover probes, a spill fuzz, eleven NoKV workload runs, or 192 replays of operation logs captured from four NoKV workloads (10,244 to 31,235 batches each), so a future occurrence fails closed instead of corrupting. No crate is published by this PR.

Validation

Rust stable (the rustup toolchain first on PATH: the Homebrew 1.97 clippy does not know the clippy::assert_is_empty lint src/lib.rs allows), macOS, exact head f28d5ad.

• Red first: probes on unmodified 8e7d8d7 with the same scenarios as the new tests show the half batch durable after a reopen on a custom store and on a WAL store. On the final code, making fail_closed a no-op fails all three latch tests (a_batch_that_fails_part_way_fails_the_store_closed_and_is_never_persisted, a_partial_batch_on_a_wal_store_is_absent_after_reopen, a_partial_attached_batch_is_absent_after_reopen), and removing each of the four DB latch points fails exactly its test. failed_closed_store_stops_the_planner ran 15,285 refused rounds in 100 ms without the planner stop. Without the re-check, a queued DB batch, a put_if_absent and an attached batch returned Ok(true) behind the failing batch and were durable after the reopen; removing each re-check point fails its test.
• cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, and the soak tool's fmt and clippy pass.
• cargo test --workspace --all-features --lib --tests --examples --locked: 787 passed, 0 failed, 9 ignored (an earlier version of this description gave the first commit's 784); the doctests: 1 passed, 3 ignored. The four examples run; RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps passes; cargo publish --dry-run --locked passes.
• This PR's CI runs the Linux and macOS tests, the fuzz guard and the coverage job.
• Not run here: the io-uring backend tests and the benchmarks. No crate was published.