Repository navigation
fix: ship the fail-closed latch for failed writes as 0.8.8 on top of 0.8.7 - #85
Merged
Merged
Conversation
…d frames An atomic DB batch applies its ops to cached frames and only then submits its WAL record. When an op of a later tree failed (an allocation error, a blob read error) or the submit itself failed, the earlier ops stayed applied in memory with no WAL record, and the next checkpoint round wrote those frames: after a reopen, one tree held its half of the batch without the other. NoKV hit exactly this: an allocation failure in one keyspace of a metadata batch left the recovery tail one LSN ahead of the outbox on disk, and the metadata store then refused to open. Any write that may have changed cached frames before failing now latches the shared buffer manager closed while it still holds the commit gate: atomic DB batches (apply or submit), tree batches, and single put, delete and rename whose WAL submit failed (rename also when its insert fails after its erase). Once latched, every write is refused, and every path that persists dirty state (background rounds, DB and tree checkpoints, the final round on drop, inline flushes) checks the latch under the commit gate and refuses with the new Error::FailedClosed. Reopening recovers the last durable state; the failed call still reports OutcomeUnknown. Tests fail a cross-tree batch part-way on a custom store (a cold child read fails) and on a WAL store (a spillover failpoint), and check that later writes and checkpoints are refused and that the half that applied is absent after a reopen. Without the latch both reopen with the half batch durable. Signed-off-by: wchwawa <wch19961116@gmail.com>
The test reopens a WAL store after a failed batch and expects every batch before it to be durable. It ran with async acks and the checkpointer off, so the last batches could still sit in the WAL writer's user-space buffer when the DB dropped, and the reopen then missed them: the test failed in 8 of 8 local runs. Synced acks make every earlier batch durable before the failure, which is what the assertions need. Signed-off-by: wchwawa <wch19961116@gmail.com>
A store latched closed refuses every checkpoint round until it is reopened. The planner still scheduled them: under write pressure (dirty blobs at the threshold or a pending delete) it retried without parking and logged every refusal, tens of thousands of lines a second until the database dropped; otherwise it logged one refusal per idle interval. The planner now stops scheduling rounds as soon as it sees the latch and exits after draining the epochs already in flight. The refusal is logged once, shared with the final round on drop, which is still attempted and refused. A refused round now counts as attempted and failed in the checkpointer stats, so the planner's behaviour is observable. The test latches a store under write pressure and checks that at most one round runs afterwards; without the stop, 16,929 refused rounds ran in 100 ms. Signed-off-by: wchwawa <wch19961116@gmail.com>
Writes checked the latch only on entry. A write that passed that check and then queued behind a writer failing part-way acquired the gates after the latch was set, evaluated its conditions against the partial change and committed a WAL record, which a reopen replayed onto a state that never held the partial change. Every write that can queue behind another now re-checks the latch once it holds the gates that serialize it, before it evaluates a condition or changes a frame: DB atomic batches after their tree gates, tree batches and put_many_if_absent after the mutation gate, single put, delete and rename (conditional and deferred) after their endpoint locks, and create_tree, drop_tree and the dropped-tree finalizer after the maintenance gate. A failing writer latches before it releases those gates, so a queued write sees the latch and is refused with FailedClosed; DB batches report it as DefinitelyNotApplied. The test holds a DB batch and a conditional tree put after their entry checks, fails a cross-tree batch part-way on a cold child read, then releases them and reopens. Without the re-check both returned Ok(true) and were durable after the reopen. Signed-off-by: wchwawa <wch19961116@gmail.com>
Version 0.8.8 is 0.8.7 plus the fail-closed fix for writes that fail after changing cached frames. It carries no format change: stores written by 0.8.7 open unchanged, and 0.8.7 opens stores written by 0.8.8. Signed-off-by: wchwawa <wch19961116@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TLDR
Holt 0.8.8 is 0.8.7 plus the fail-closed fix of #83 and nothing else. A write that fails after it has already changed cached frames latches the database closed, so the partial change can never reach a checkpoint. NoKV pins Holt exactly and needs this fix now; the blob integrity change on
release/0.8(#81) stays out because it adds a one-way store format migration that NoKV has not qualified.Problem
DB::atomicapplies each tree's ops to cached frames under the commit gate and then submits one WAL record. When an op of a later tree fails (an allocation error, a blob read error) or the submit fails, the earlier ops stay applied in memory with no WAL record, and the next checkpoint round persists them. Tree batches and single put, delete and rename whose WAL submit fails have the same shape. #83 fixes this onrelease/0.8, but that branch already carries #81, which migrates version 6 stores to version 7 on their first writable open; older binaries cannot reopen a migrated store, and #81's performance acceptance is still open. Shipping #83 fromrelease/0.8would force that migration onto every 0.8 user who needs the fix.Consequences
Without this release NoKV keeps a storage engine that can make its metadata store half durable: NoKV 0.12.x on Holt 0.8.7 hit an
OutOfSpacein one keyspace of a metadata batch, the recovery tail ended one log sequence number ahead of the outbox on disk, and the store refused to open until an operator intervened. The alternative, adoptingrelease/0.8as it stands, puts an unqualified one-way format migration into a patch upgrade.Example
A two-tree batch whose second tree fails part-way. On 0.8.7: after a checkpoint and a reopen, the first tree has its keys and the second does not. On 0.8.8: the batch still fails with
OutcomeUnknown; every later write,DB::checkpointand background round fails withError::FailedClosed; after a reopen neither tree has the batch.Solution
The new branch
release/0.8.8starts at thev0.8.7commit. This pull request applies #83's four commits unchanged, except that the changelog keeps only the fail-closed entry, and a release commit sets the version to 0.8.8 inCargo.tomland the three lockfiles and dates the changelog section.Erroris#[non_exhaustive], so the newError::FailedClosedvariant is a compatible addition.After this merges,
v0.8.8is tagged on the merge commit and the Release workflow publishes the crate. #83 still brings the same fix torelease/0.8; it gains a commit that records 0.8.8 in that branch's changelog and renumbers the unreleased blob integrity release to 0.8.9, since 0.8.8 is taken.Gain
NoKV and other 0.8 users get the fix as a drop-in patch. There is no format change: stores written by 0.8.7 and 0.8.8 open in either version, through a checkpoint and through WAL replay.
Tradeoffs
The latch stops writes and checkpoints of the whole database, not only the tree that failed, until a reopen: a failure that used to be silent corruption becomes an outage that needs a restart. Reads keep serving the in-memory state, which may include the partial change, until the reopen; NoKV already stops on
OutcomeUnknownand refuses reads and writes until it reopens. A latched store skips the final checkpoint on drop, so with asynchronous WAL acknowledgements the records acknowledged before the failure that still sit in the WAL writer's buffer are lost, as they already are when the checkpointer is off. TheOutOfSpaceitself is not fixed, only contained; #83 lists the reproduction attempts. The two 0.8 branches carry the fix as different commits, andv0.8.8is not an ancestor ofrelease/0.8.Validation
Rust stable (the rustup toolchain first on
PATH), macOS, exact head8e6c89d.v0.8.7; the code applied without conflicts, and only the changelog needed resolving.cargo fmt --all --check,cargo clippy --workspace --all-targets --all-features --locked -- -D warnings, and the soak tool's fmt and clippy pass.cargo test --workspace --all-features --lib --tests --examples --locked: 712 passed, 0 failed, 8 ignored; the doctests: 1 passed, 3 ignored. The four examples run;RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-depspasses;cargo publish --dry-run --lockedpackages 0.8.8 and verifies it.cargo +1.82 build --lib --lockedpasses. The two soak smokes with the CI workflow's parameters (normalanddb-normal) pass.atomic_model,db_modelandread_index_modelfuzz targets.v0.8.8is tagged.