fix(s3): stop a re-created bucket from adopting a previous one's objects - #2568
Merged
Merged
Conversation
A bucket absent from memory can still have a directory in the data path: after `/_fakecloud/reset` (which clears memory and deliberately leaves the store alone), or after a create or delete that stopped partway. CreateBucket cleared only the sidecars, so the stale `objects/` tree survived: the caller saw an empty bucket, and the previous incarnation's objects came back on the next load. CreateBucket now clears the whole stored directory for the name, before writing this bucket's own state (the clear removes `meta.toml`, so doing it afterwards would delete the bucket the create just wrote). That clear would otherwise turn re-creating a name into the thing that destroys recoverable data: the loader skips a bucket whose objects it cannot read, so such a bucket is absent from memory while its files sit intact on disk. The store now records which buckets `load` refused, and CreateBucket refuses those names with BucketAlreadyExists and a message naming the repair. DeleteBucket is the in-band escape -- it is the explicitly destructive verb -- so a refused name is never permanently stuck. - `S3Store::bucket_state_exists` and `bucket_load_refused`, recorded at load rather than probed per call, so "the loader could not read this" is never confused with "this is not in memory" - the refusal is cleared only once `remove_dir_all` actually succeeded - e2e: the re-created bucket is empty before AND after a restart; a load-refused name is declined without destroying its objects, and repairing the one bad file brings the whole bucket back; DeleteBucket frees a refused name - docs: Gotcha covering the clear, the refusal and the escape
Review of the commit before this one, three edges it got wrong: - The refusal is recorded at load and never re-probed, so an operator who followed the error text by deleting the bucket's directory -- without restarting -- found the name refused for the rest of the process lifetime, told to repair something that no longer existed. It now requires `bucket_state_exists` as well, which is also the caller that method was missing. - The whole-directory clear is a recursive remove running under the global S3 write lock. It is now skipped when there is no directory, so the ordinary create pays a `stat` rather than a walk, and only the create-after-reset that needs it does the work. - The DeleteBucket escape skipped the empty-bucket check, and with it the only thing that makes an Object Lock bucket undeletable. A reflexive `aws s3 rb`, or a retry of the tool that just got the 409, could destroy retained data that no bucket on real S3 lets go. The lock config is its own file and is usually readable when an object's is not, so the escape refuses while it is there and leaves that case to the operator. - `S3Store::bucket_subresource_exists`, readable without the bucket being loadable, which is what makes it usable on a refused bucket - e2e: removing the directory frees the name with no restart; a refused bucket under Object Lock is not discarded by DeleteBucket and its objects survive, and repairing the bad file brings it back still locked - docs: both edges, plus the asymmetry a caller sees (reads report the name missing while the create refuses it)
…me is reused Review of the two commits before this one found the pair combining into data loss, and the containment I had added for it aiming at the wrong file. Cutting the escape rather than patching it again. The refusal was recorded at load and never invalidated. Requiring the data to still be there let a create through once the directory was removed -- and that create no longer called `delete_bucket`, which had been the only thing dropping the refusal. So a healthy, live bucket ended up owning a name still in the refused set, and then: - any second account's DeleteBucket missed the per-account memory probe, found the stale refusal, and recursively deleted the live bucket's directory, with neither the emptiness check nor the ownership check the normal path runs; - a later `/_fakecloud/reset` refused the create for a bucket that read fine. A successful create now clears the refusal, which is the root cause. And the escape is gone: DeleteBucket could check neither emptiness (the objects are exactly what could not be read) nor ownership (the metadata carrying it is what failed), so it was a verb that discards retained data for any caller. The Object Lock guard added to contain that probed only the bucket-level `object_lock.toml`, so it missed per-object retention while wrongly blocking an empty lock-enabled bucket. Removing the directory frees the name with no restart, so the escape was never load-bearing -- an operator whose store the loader cannot read is already in the data path. - the refusal message carries `BucketName`, as AWS does with that code, and has its line continuations back (they had been mangled into runs of spaces, which no test caught because the assertion only checked the error code) - `bucket_subresource_exists` goes with its only caller - the two e2e tests written for the escape go with it; the surviving one now pins the exact 409 code, that the create leaves no refusal behind, and that a delete of a name absent from memory is a plain NoSuchBucket
…st its objects
Two layers can skip a bucket at load, and only one of them was recording a
refusal. The store skips a bucket whose OBJECTS it cannot read. The layer above
parses the bucket's own sidecars, and an unparseable `tags.toml`, `acl.toml` or
`inventory.toml` skips it there -- with the store reporting no trouble at all,
because the objects read fine.
That layer already hands each skipped bucket to a `refused` callback, and its doc
comment says the bucket is then "reported exactly like a store-level refusal".
`main.rs` passed `|_, _| {}`. So the name looked free, and this branch's
whole-directory clear deleted objects that were never unreadable -- where before
the branch only the sidecars were swept, leaving `objects/` to be recovered by
repairing the one bad file. Wiring the callback to a new
`S3Store::mark_bucket_load_refused` is what the hook was written for.
Also from the same review:
- The refusal clear moves ahead of the writes. A create that failed partway --
after `put_bucket_meta`, on a subresource -- left the refusal standing over a
directory that create had just made, and every later create for the name was
told to repair a directory holding nothing but that failed attempt's
`meta.toml`. It is only reached once the refusal is known not to apply, so
dropping it there cannot discard a live one. Gated on a read first, to keep a
write lock off the ordinary create path.
- The trailing assertion added last commit could not fail: nothing had left a
stale refusal by that point, so re-pasting the deleted DeleteBucket escape
verbatim kept it green. The property now sits on the genuinely refused bucket,
asserting both the `NoSuchBucket` and that its objects survive -- verified by
re-adding the escape, which fails it.
- Two doc comments the earlier cut falsified: the hydrate hook still advertised
"clearable by DeleteBucket", and the docs implied only object-level corruption
is protected.
(The previous commit message credited the mangled-literal fix to the wrong
string: it was the `BucketNotEmpty` Object Lock message, which that commit
deleted, not the `BucketAlreadyExists` one.)
…cloud into worktree-s3-stale-objects
…wrapper
Review of the commit before this one. No correctness bug, but the text an operator
reads was left describing the narrower behavior:
- The `CreateBucket` refusal named only "an unreadable object, or a delete that
stopped partway" -- while this branch is precisely what widened refusals to a
bucket's OWN files. Someone hitting it for a corrupt `tags.toml` was sent
hunting through `objects/`. It now names both kinds and points at the log for
which file it was.
- `bucket_load_refused`'s doc still said the set holds what `load` refused, eight
lines above the method that inserts entries `load` never saw.
- The loader's warning still offered "or the bucket is deleted" -- the in-band
escape removed two commits ago. It now says to remove the directory, and that no
API call discards it.
- The hydrate-layer warning did not mention that the name is now refused, unlike
the store-layer warning for the same outcome.
`hydrate_s3_state` is deleted. Its own sibling's doc says "wire this to the store's
`mark_bucket_load_refused`, not to a no-op: with the hook dropped on the floor, a
bucket skipped here is indistinguishable from a name nobody has used, and the next
create deletes its objects" -- and that wrapper WAS the no-op, exported from a
published crate as the convenient default. After this branch its only caller was a
unit test in the same file, which now passes `&mut |_, _| {}` explicitly, so the
choice to discard refusals is visible at the call site instead of hidden behind a
shorter name. This is the hole that shipped in #2558 and only became destructive
when the whole-directory clear landed; removing the shape stops it recurring.
Plus the unit test the wiring never had. Its correctness rests on
`escape_key_segment(bucket_name)` matching the on-disk directory name, because
`load` inserts the directory name while the sidecar layer inserts an escaped bucket
name -- and both e2e tests use names that escape to themselves, so a key-space
mismatch would pass them. `a_refusal_is_found_under_the_escaped_directory_name`
uses `odd:name`, which does not, and is verified failing: drop the escaping from
`mark_bucket_load_refused` and the mark is no longer found.
…cloud into worktree-s3-stale-objects
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Third of the three gaps #2558 documented as out of scope. A bucket absent from memory can still have a directory in the data path: after
/_fakecloud/reset(which clears memory and deliberately leaves the store alone), or after a create or delete that stopped partway.CreateBucketcleared only the sidecars, so a staleobjects/tree survived, and a re-created bucket adopted it: the caller saw an empty bucket, then the previous incarnation's objects came back on the next load.CreateBucketnow clears the whole stored directory for the name, before writing this bucket's own state. Ordering is load-bearing: the clear removesmeta.toml, so doing it afterwards would delete the bucket the create just wrote.That clear would otherwise turn re-creating a name into the thing that destroys recoverable data. The loader skips a bucket whose objects it cannot read (a corrupt object meta, a sidecar with no body) and logs a warning; such a bucket is absent from
ListBucketswhile its files sit intact on disk, so its name looks free. So the store now records which bucketsloadrefused, andCreateBucketdeclines those names withBucketAlreadyExistsplus a message naming the repair.DeleteBucketis the in-band escape, being the explicitly destructive verb, so a refused name is never permanently stuck.Deliberately minimal: a load-time refusal set and nothing else. No trash directory, no retention, no background reclaim -- that machinery is what made an earlier attempt at this unstable.
Changes
crates/fakecloud-persistence/src/s3.rs:S3Store::bucket_state_existsandS3Store::bucket_load_refused(both defaulting tofalse, so memory-only stores are unaffected).DiskS3Storerecords refusals duringload, keyed on the escaped directory name, starting from empty each load so a repaired bucket stops being refused.delete_bucketclears the refusal only onceremove_dir_allactually succeeded -- dropping it on a partial failure would leave the next create free to discard the remains.crates/fakecloud-s3/src/service/buckets.rs: the refusal check runs inside the write lock before anything is written; the directory clear precedes the meta write;DeleteBucketgains the escape branch for a refused name (no emptiness check is possible -- the objects are exactly what could not be read -- and no owner check either, since the metadata carrying ownership is what failed, so the warning log records that any account can reach it).Where the refusal stops
The refusal is deliberately bounded, and two review rounds cut it back:
bucket_state_exists), not just to have been refused at load. Otherwise an operator who removed the directory, without restarting, found the name refused for the rest of the process lifetime and was told to repair something that no longer existed.DeleteBucketrecursively deleted its directory (the memory probe is per-account, so it missed the bucket) and a later/_fakecloud/resetrefused the create for data that read fine.statin front of all the rest.There is deliberately no API call that discards a refused bucket. An earlier revision of this PR gave
DeleteBucketthat job, on the reasoning that it is the explicitly destructive verb. It could check neither emptiness (the objects are exactly what could not be read) nor ownership (the metadata carrying it is what failed), which made it a verb that destroys retained data for any caller -- and the Object Lock guard added to contain that probed only the bucket-levelobject_lock.toml, so it missed per-object retention while wrongly refusing to delete an empty lock-enabled bucket. Removing the directory frees the name with no restart, so the escape was never load-bearing: an operator whose store the loader cannot read is already in the data path.DeleteBucketanswersNoSuchBucketfor a refused name, agreeing with every read path.One review finding was declined: making
HeadBucket/ListBucketsreport a refused name as present, so they agree with the create's 409. There is no AWS analogue for "exists but unreadable", and inventing a status for the read paths is a bigger change than the bug warrants; a head-then-create script fails on the create rather than silently overwriting the data, which is the safe direction. The asymmetry is documented instead.Test plan
cargo nextest run -p fakecloud-s3 -p fakecloud-persistence-- 550 passcargo nextest run -p fakecloud-e2e --test s3_persistence --test s3-- 120 pass (plus--test s3_anonymous_access --test iam_enforcementgreen on the first revision)cargo nextest run -p fakecloud-conformance --test s3-- 51 passcargo clippy --workspace --all-targets -- -D warnings-- cleanEvery new e2e test was verified failing without its fix:
persistence_create_after_reset_reuses_the_name(the old objects reappear after the restart)persistence_recreating_a_load_refused_bucket_is_declined_not_destructive(the create succeeds andkeep.txt's body is gone)bucket_state_existshalf of the guard failspersistence_removing_a_refused_directory_frees_the_name_without_a_restartCloses the last of the three follow-ups from #2558 (the other two are #2565 and #2566).