Skip to content

fix(s3): stop a re-created bucket from adopting a previous one's objects - #2568

Merged
vieiralucas merged 9 commits into
mainfrom
worktree-s3-stale-objects
Sep 29, 2026
Merged

vieiralucas merged 9 commits into
mainfrom
worktree-s3-stale-objects

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

Summary

Third of the three gaps #2558 documented as out of scope. A bucket absent from memory can still have a directory in the data path: after /_fakecloud/reset (which clears memory and deliberately leaves the store alone), or after a create or delete that stopped partway. CreateBucket cleared only the sidecars, so a stale objects/ tree survived, and a re-created bucket adopted it: the caller saw an empty bucket, then the previous incarnation's objects came back on the next load.

CreateBucket now clears the whole stored directory for the name, before writing this bucket's own state. Ordering is load-bearing: the clear removes meta.toml, so doing it afterwards would delete the bucket the create just wrote.

That clear would otherwise turn re-creating a name into the thing that destroys recoverable data. The loader skips a bucket whose objects it cannot read (a corrupt object meta, a sidecar with no body) and logs a warning; such a bucket is absent from ListBuckets while its files sit intact on disk, so its name looks free. So the store now records which buckets load refused, and CreateBucket declines those names with BucketAlreadyExists plus a message naming the repair. DeleteBucket is the in-band escape, being the explicitly destructive verb, so a refused name is never permanently stuck.

Deliberately minimal: a load-time refusal set and nothing else. No trash directory, no retention, no background reclaim -- that machinery is what made an earlier attempt at this unstable.

Changes

  • crates/fakecloud-persistence/src/s3.rs: S3Store::bucket_state_exists and S3Store::bucket_load_refused (both defaulting to false, so memory-only stores are unaffected). DiskS3Store records refusals during load, keyed on the escaped directory name, starting from empty each load so a repaired bucket stops being refused. delete_bucket clears the refusal only once remove_dir_all actually succeeded -- dropping it on a partial failure would leave the next create free to discard the remains.
  • crates/fakecloud-s3/src/service/buckets.rs: the refusal check runs inside the write lock before anything is written; the directory clear precedes the meta write; DeleteBucket gains the escape branch for a refused name (no emptiness check is possible -- the objects are exactly what could not be read -- and no owner check either, since the metadata carrying ownership is what failed, so the warning log records that any account can reach it).

Where the refusal stops

The refusal is deliberately bounded, and two review rounds cut it back:

  • It requires the data to still BE there (bucket_state_exists), not just to have been refused at load. Otherwise an operator who removed the directory, without restarting, found the name refused for the rest of the process lifetime and was told to repair something that no longer existed.
  • A successful create clears the refusal. It is recorded at load and consulted long afterwards, so once a name holds a bucket that loaded, the refusal has stopped being true. Leaving it behind put a live bucket under a refused name, and from there a second account's DeleteBucket recursively deleted its directory (the memory probe is per-account, so it missed the bucket) and a later /_fakecloud/reset refused the create for data that read fine.
  • The clear is skipped when there is no directory. It is a recursive remove running under the global S3 write lock, so an unconditional call put a walk of a possibly huge tree in front of every other S3 request on the one create that needs it, and a stat in front of all the rest.

There is deliberately no API call that discards a refused bucket. An earlier revision of this PR gave DeleteBucket that job, on the reasoning that it is the explicitly destructive verb. It could check neither emptiness (the objects are exactly what could not be read) nor ownership (the metadata carrying it is what failed), which made it a verb that destroys retained data for any caller -- and the Object Lock guard added to contain that probed only the bucket-level object_lock.toml, so it missed per-object retention while wrongly refusing to delete an empty lock-enabled bucket. Removing the directory frees the name with no restart, so the escape was never load-bearing: an operator whose store the loader cannot read is already in the data path. DeleteBucket answers NoSuchBucket for a refused name, agreeing with every read path.

One review finding was declined: making HeadBucket / ListBuckets report a refused name as present, so they agree with the create's 409. There is no AWS analogue for "exists but unreadable", and inventing a status for the read paths is a bigger change than the bug warrants; a head-then-create script fails on the create rather than silently overwriting the data, which is the safe direction. The asymmetry is documented instead.

Test plan

  • cargo nextest run -p fakecloud-s3 -p fakecloud-persistence -- 550 pass
  • cargo nextest run -p fakecloud-e2e --test s3_persistence --test s3 -- 120 pass (plus --test s3_anonymous_access --test iam_enforcement green on the first revision)
  • cargo nextest run -p fakecloud-conformance --test s3 -- 51 pass
  • cargo clippy --workspace --all-targets -- -D warnings -- clean

Every new e2e test was verified failing without its fix:

  • removing the directory clear fails persistence_create_after_reset_reuses_the_name (the old objects reappear after the restart)
  • disabling the refusal fails persistence_recreating_a_load_refused_bucket_is_declined_not_destructive (the create succeeds and keep.txt's body is gone)
  • dropping the bucket_state_exists half of the guard fails persistence_removing_a_refused_directory_frees_the_name_without_a_restart
  • dropping the refusal clear fails that same test at its later half (the name is still refused after a reset, for a bucket that loaded)

Closes the last of the three follow-ups from #2558 (the other two are #2565 and #2566).

A bucket absent from memory can still have a directory in the data path:
after `/_fakecloud/reset` (which clears memory and deliberately leaves the
store alone), or after a create or delete that stopped partway. CreateBucket
cleared only the sidecars, so the stale `objects/` tree survived: the caller
saw an empty bucket, and the previous incarnation's objects came back on the
next load.

CreateBucket now clears the whole stored directory for the name, before
writing this bucket's own state (the clear removes `meta.toml`, so doing it
afterwards would delete the bucket the create just wrote).

That clear would otherwise turn re-creating a name into the thing that
destroys recoverable data: the loader skips a bucket whose objects it cannot
read, so such a bucket is absent from memory while its files sit intact on
disk. The store now records which buckets `load` refused, and CreateBucket
refuses those names with BucketAlreadyExists and a message naming the repair.
DeleteBucket is the in-band escape -- it is the explicitly destructive verb --
so a refused name is never permanently stuck.

- `S3Store::bucket_state_exists` and `bucket_load_refused`, recorded at load
  rather than probed per call, so "the loader could not read this" is never
  confused with "this is not in memory"
- the refusal is cleared only once `remove_dir_all` actually succeeded
- e2e: the re-created bucket is empty before AND after a restart; a
  load-refused name is declined without destroying its objects, and
  repairing the one bad file brings the whole bucket back; DeleteBucket
  frees a refused name
- docs: Gotcha covering the clear, the refusal and the escape
Review of the commit before this one, three edges it got wrong:

- The refusal is recorded at load and never re-probed, so an operator who
  followed the error text by deleting the bucket's directory -- without
  restarting -- found the name refused for the rest of the process lifetime,
  told to repair something that no longer existed. It now requires
  `bucket_state_exists` as well, which is also the caller that method was
  missing.
- The whole-directory clear is a recursive remove running under the global S3
  write lock. It is now skipped when there is no directory, so the ordinary
  create pays a `stat` rather than a walk, and only the create-after-reset that
  needs it does the work.
- The DeleteBucket escape skipped the empty-bucket check, and with it the only
  thing that makes an Object Lock bucket undeletable. A reflexive `aws s3 rb`,
  or a retry of the tool that just got the 409, could destroy retained data
  that no bucket on real S3 lets go. The lock config is its own file and is
  usually readable when an object's is not, so the escape refuses while it is
  there and leaves that case to the operator.

- `S3Store::bucket_subresource_exists`, readable without the bucket being
  loadable, which is what makes it usable on a refused bucket
- e2e: removing the directory frees the name with no restart; a refused bucket
  under Object Lock is not discarded by DeleteBucket and its objects survive,
  and repairing the bad file brings it back still locked
- docs: both edges, plus the asymmetry a caller sees (reads report the name
  missing while the create refuses it)
…me is reused

Review of the two commits before this one found the pair combining into data
loss, and the containment I had added for it aiming at the wrong file. Cutting
the escape rather than patching it again.

The refusal was recorded at load and never invalidated. Requiring the data to
still be there let a create through once the directory was removed -- and that
create no longer called `delete_bucket`, which had been the only thing dropping
the refusal. So a healthy, live bucket ended up owning a name still in the
refused set, and then:

- any second account's DeleteBucket missed the per-account memory probe, found
  the stale refusal, and recursively deleted the live bucket's directory, with
  neither the emptiness check nor the ownership check the normal path runs;
- a later `/_fakecloud/reset` refused the create for a bucket that read fine.

A successful create now clears the refusal, which is the root cause. And the
escape is gone: DeleteBucket could check neither emptiness (the objects are
exactly what could not be read) nor ownership (the metadata carrying it is what
failed), so it was a verb that discards retained data for any caller. The Object
Lock guard added to contain that probed only the bucket-level `object_lock.toml`,
so it missed per-object retention while wrongly blocking an empty lock-enabled
bucket. Removing the directory frees the name with no restart, so the escape was
never load-bearing -- an operator whose store the loader cannot read is already
in the data path.

- the refusal message carries `BucketName`, as AWS does with that code, and has
  its line continuations back (they had been mangled into runs of spaces, which
  no test caught because the assertion only checked the error code)
- `bucket_subresource_exists` goes with its only caller
- the two e2e tests written for the escape go with it; the surviving one now
  pins the exact 409 code, that the create leaves no refusal behind, and that a
  delete of a name absent from memory is a plain NoSuchBucket
…st its objects

Two layers can skip a bucket at load, and only one of them was recording a
refusal. The store skips a bucket whose OBJECTS it cannot read. The layer above
parses the bucket's own sidecars, and an unparseable `tags.toml`, `acl.toml` or
`inventory.toml` skips it there -- with the store reporting no trouble at all,
because the objects read fine.

That layer already hands each skipped bucket to a `refused` callback, and its doc
comment says the bucket is then "reported exactly like a store-level refusal".
`main.rs` passed `|_, _| {}`. So the name looked free, and this branch's
whole-directory clear deleted objects that were never unreadable -- where before
the branch only the sidecars were swept, leaving `objects/` to be recovered by
repairing the one bad file. Wiring the callback to a new
`S3Store::mark_bucket_load_refused` is what the hook was written for.

Also from the same review:

- The refusal clear moves ahead of the writes. A create that failed partway --
  after `put_bucket_meta`, on a subresource -- left the refusal standing over a
  directory that create had just made, and every later create for the name was
  told to repair a directory holding nothing but that failed attempt's
  `meta.toml`. It is only reached once the refusal is known not to apply, so
  dropping it there cannot discard a live one. Gated on a read first, to keep a
  write lock off the ordinary create path.
- The trailing assertion added last commit could not fail: nothing had left a
  stale refusal by that point, so re-pasting the deleted DeleteBucket escape
  verbatim kept it green. The property now sits on the genuinely refused bucket,
  asserting both the `NoSuchBucket` and that its objects survive -- verified by
  re-adding the escape, which fails it.
- Two doc comments the earlier cut falsified: the hydrate hook still advertised
  "clearable by DeleteBucket", and the docs implied only object-level corruption
  is protected.

(The previous commit message credited the mangled-literal fix to the wrong
string: it was the `BucketNotEmpty` Object Lock message, which that commit
deleted, not the `BucketAlreadyExists` one.)
…wrapper

Review of the commit before this one. No correctness bug, but the text an operator
reads was left describing the narrower behavior:

- The `CreateBucket` refusal named only "an unreadable object, or a delete that
  stopped partway" -- while this branch is precisely what widened refusals to a
  bucket's OWN files. Someone hitting it for a corrupt `tags.toml` was sent
  hunting through `objects/`. It now names both kinds and points at the log for
  which file it was.
- `bucket_load_refused`'s doc still said the set holds what `load` refused, eight
  lines above the method that inserts entries `load` never saw.
- The loader's warning still offered "or the bucket is deleted" -- the in-band
  escape removed two commits ago. It now says to remove the directory, and that no
  API call discards it.
- The hydrate-layer warning did not mention that the name is now refused, unlike
  the store-layer warning for the same outcome.

`hydrate_s3_state` is deleted. Its own sibling's doc says "wire this to the store's
`mark_bucket_load_refused`, not to a no-op: with the hook dropped on the floor, a
bucket skipped here is indistinguishable from a name nobody has used, and the next
create deletes its objects" -- and that wrapper WAS the no-op, exported from a
published crate as the convenient default. After this branch its only caller was a
unit test in the same file, which now passes `&mut |_, _| {}` explicitly, so the
choice to discard refusals is visible at the call site instead of hidden behind a
shorter name. This is the hole that shipped in #2558 and only became destructive
when the whole-directory clear landed; removing the shape stops it recurring.

Plus the unit test the wiring never had. Its correctness rests on
`escape_key_segment(bucket_name)` matching the on-disk directory name, because
`load` inserts the directory name while the sidecar layer inserts an escaped bucket
name -- and both e2e tests use names that escape to themselves, so a key-space
mismatch would pass them. `a_refusal_is_found_under_the_escaped_directory_name`
uses `odd:name`, which does not, and is verified failing: drop the escaping from
`mark_bucket_load_refused` and the mark is no longer found.
@vieiralucas
vieiralucas merged commit 2ec5cef into main Sep 29, 2026
157 checks passed
@vieiralucas
vieiralucas deleted the worktree-s3-stale-objects branch September 29, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant