Skip to content

fix: ship the fail-closed latch for failed writes as 0.8.8 on top of 0.8.7 - #85

Merged
wchwawa merged 5 commits into
release/0.8.8from
fix/fail-closed-torn-batch-0.8.8
Oct 31, 2026
Merged

wchwawa merged 5 commits into
release/0.8.8from
fix/fail-closed-torn-batch-0.8.8

Conversation

@wchwawa

@wchwawa wchwawa commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

TLDR

Holt 0.8.8 is 0.8.7 plus the fail-closed fix of #83 and nothing else. A write that fails after it has already changed cached frames latches the database closed, so the partial change can never reach a checkpoint. NoKV pins Holt exactly and needs this fix now; the blob integrity change on release/0.8 (#81) stays out because it adds a one-way store format migration that NoKV has not qualified.

Problem

DB::atomic applies each tree's ops to cached frames under the commit gate and then submits one WAL record. When an op of a later tree fails (an allocation error, a blob read error) or the submit fails, the earlier ops stay applied in memory with no WAL record, and the next checkpoint round persists them. Tree batches and single put, delete and rename whose WAL submit fails have the same shape. #83 fixes this on release/0.8, but that branch already carries #81, which migrates version 6 stores to version 7 on their first writable open; older binaries cannot reopen a migrated store, and #81's performance acceptance is still open. Shipping #83 from release/0.8 would force that migration onto every 0.8 user who needs the fix.

Consequences

Without this release NoKV keeps a storage engine that can make its metadata store half durable: NoKV 0.12.x on Holt 0.8.7 hit an OutOfSpace in one keyspace of a metadata batch, the recovery tail ended one log sequence number ahead of the outbox on disk, and the store refused to open until an operator intervened. The alternative, adopting release/0.8 as it stands, puts an unqualified one-way format migration into a patch upgrade.

Example

A two-tree batch whose second tree fails part-way. On 0.8.7: after a checkpoint and a reopen, the first tree has its keys and the second does not. On 0.8.8: the batch still fails with OutcomeUnknown; every later write, DB::checkpoint and background round fails with Error::FailedClosed; after a reopen neither tree has the batch.

Solution

The new branch release/0.8.8 starts at the v0.8.7 commit. This pull request applies #83's four commits unchanged, except that the changelog keeps only the fail-closed entry, and a release commit sets the version to 0.8.8 in Cargo.toml and the three lockfiles and dates the changelog section. Error is #[non_exhaustive], so the new Error::FailedClosed variant is a compatible addition.

After this merges, v0.8.8 is tagged on the merge commit and the Release workflow publishes the crate. #83 still brings the same fix to release/0.8; it gains a commit that records 0.8.8 in that branch's changelog and renumbers the unreleased blob integrity release to 0.8.9, since 0.8.8 is taken.

Gain

NoKV and other 0.8 users get the fix as a drop-in patch. There is no format change: stores written by 0.8.7 and 0.8.8 open in either version, through a checkpoint and through WAL replay.

Tradeoffs

The latch stops writes and checkpoints of the whole database, not only the tree that failed, until a reopen: a failure that used to be silent corruption becomes an outage that needs a restart. Reads keep serving the in-memory state, which may include the partial change, until the reopen; NoKV already stops on OutcomeUnknown and refuses reads and writes until it reopens. A latched store skips the final checkpoint on drop, so with asynchronous WAL acknowledgements the records acknowledged before the failure that still sit in the WAL writer's buffer are lost, as they already are when the checkpointer is off. The OutOfSpace itself is not fixed, only contained; #83 lists the reproduction attempts. The two 0.8 branches carry the fix as different commits, and v0.8.8 is not an ancestor of release/0.8.

Validation

Rust stable (the rustup toolchain first on PATH), macOS, exact head 8e6c89d.

  • The four commits are fix: fail the database closed when a write fails after changing cached frames, and renumber the next 0.8 release 0.8.9 #83's, cherry-picked onto v0.8.7; the code applied without conflicts, and only the changelog needed resolving.
  • cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, and the soak tool's fmt and clippy pass.
  • cargo test --workspace --all-features --lib --tests --examples --locked: 712 passed, 0 failed, 8 ignored; the doctests: 1 passed, 3 ignored. The four examples run; RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps passes; cargo publish --dry-run --locked packages 0.8.8 and verifies it.
  • MSRV: cargo +1.82 build --lib --locked passes. The two soak smokes with the CI workflow's parameters (normal and db-normal) pass.
  • Cross-version: one program, built once against holt 0.8.7 from crates.io and once against this head, wrote and read the same store in both orders. Each order ran four rounds of 1,500 cross-tree atomic batches (every fifth value is 4,000 bytes, so blob spillover runs), alternating checkpointed rounds with WAL-only rounds that the other version had to replay. Both versions verified every round.
  • NoKV: through a local path patch, every NoKV workspace crate except the CLI crate passes against this head (1,218 passed, 0 failed, 15 ignored). The CLI crate's build script accepts Holt only from a registry, so it runs in NoKV's own pull request after publication.
  • Nightly Validation on this head (default 600-second soaks, 180-second fuzz campaign, Verus off): passed: the checkpoint and WAL fault matrix, the property tests, the crash, DB crash, DB normal and three normal soaks, and the atomic_model, db_model and read_index_model fuzz targets.
  • The CI workflow does not run on pull requests to release branches. Not run: the Linux io-uring backend tests, the benchmarks, and the coverage gate. Nothing is published until v0.8.8 is tagged.

…d frames

An atomic DB batch applies its ops to cached frames and only then submits
its WAL record. When an op of a later tree failed (an allocation error, a
blob read error) or the submit itself failed, the earlier ops stayed applied
in memory with no WAL record, and the next checkpoint round wrote those
frames: after a reopen, one tree held its half of the batch without the
other. NoKV hit exactly this: an allocation failure in one keyspace of a
metadata batch left the recovery tail one LSN ahead of the outbox on disk,
and the metadata store then refused to open.

Any write that may have changed cached frames before failing now latches
the shared buffer manager closed while it still holds the commit gate:
atomic DB batches (apply or submit), tree batches, and single put, delete
and rename whose WAL submit failed (rename also when its insert fails after
its erase). Once latched, every write is refused, and every path that
persists dirty state (background rounds, DB and tree checkpoints, the final
round on drop, inline flushes) checks the latch under the commit gate and
refuses with the new Error::FailedClosed. Reopening recovers the last
durable state; the failed call still reports OutcomeUnknown.

Tests fail a cross-tree batch part-way on a custom store (a cold child read
fails) and on a WAL store (a spillover failpoint), and check that later
writes and checkpoints are refused and that the half that applied is absent
after a reopen. Without the latch both reopen with the half batch durable.

Signed-off-by: wchwawa <wch19961116@gmail.com>
The test reopens a WAL store after a failed batch and expects every batch
before it to be durable. It ran with async acks and the checkpointer off,
so the last batches could still sit in the WAL writer's user-space buffer
when the DB dropped, and the reopen then missed them: the test failed in 8
of 8 local runs. Synced acks make every earlier batch durable before the
failure, which is what the assertions need.

Signed-off-by: wchwawa <wch19961116@gmail.com>
A store latched closed refuses every checkpoint round until it is reopened.
The planner still scheduled them: under write pressure (dirty blobs at the
threshold or a pending delete) it retried without parking and logged every
refusal, tens of thousands of lines a second until the database dropped;
otherwise it logged one refusal per idle interval.

The planner now stops scheduling rounds as soon as it sees the latch and
exits after draining the epochs already in flight. The refusal is logged
once, shared with the final round on drop, which is still attempted and
refused. A refused round now counts as attempted and failed in the
checkpointer stats, so the planner's behaviour is observable.

The test latches a store under write pressure and checks that at most one
round runs afterwards; without the stop, 16,929 refused rounds ran in
100 ms.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Writes checked the latch only on entry. A write that passed that check and
then queued behind a writer failing part-way acquired the gates after the
latch was set, evaluated its conditions against the partial change and
committed a WAL record, which a reopen replayed onto a state that never
held the partial change.

Every write that can queue behind another now re-checks the latch once it
holds the gates that serialize it, before it evaluates a condition or
changes a frame: DB atomic batches after their tree gates, tree batches
and put_many_if_absent after the mutation gate, single put, delete and
rename (conditional and deferred) after their endpoint locks, and
create_tree, drop_tree and the dropped-tree finalizer after the
maintenance gate. A failing writer latches before it releases those
gates, so a queued write sees the latch and is refused with FailedClosed;
DB batches report it as DefinitelyNotApplied.

The test holds a DB batch and a conditional tree put after their entry
checks, fails a cross-tree batch part-way on a cold child read, then
releases them and reopens. Without the re-check both returned Ok(true)
and were durable after the reopen.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Version 0.8.8 is 0.8.7 plus the fail-closed fix for writes that fail after changing cached frames. It carries no format change: stores written by 0.8.7 open unchanged, and 0.8.7 opens stores written by 0.8.8.

Signed-off-by: wchwawa <wch19961116@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant