Skip to content

fix(s3): honor Tags in CreateBucketConfiguration - #2557

Merged
vieiralucas merged 6 commits into
mainfrom
worktree-s3-createbucket-tags
Sep 25, 2026
Merged

vieiralucas merged 6 commits into
mainfrom
worktree-s3-createbucket-tags

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Summary

CreateBucket accepted the Tags tag set that was added to CreateBucketConfiguration in the 2025 S3 API, but dropped it: create_bucket only pulled LocationConstraint out of the body, so the bucket came out untagged and GetBucketTagging answered NoSuchTagSet.

That path matters for Terraform. The AWS provider (6.x) tags a bucket this way on create and skips PutBucketTagging when the create succeeds, so an aws_s3_bucket with tags applied cleanly and then read back tags = {}.

Verified against the vendored Smithy model before implementing: com.amazonaws.s3#CreateBucketConfiguration does carry a Tags member targeting TagSet (non-flattened, xmlName: "Tag"), and the Rust SDK serializes it as <Tags><Tag><Key/><Value/></Tag></Tags> — the same shape parse_tagging_xml already handles for PutBucketTagging.

The bug underneath it

Requiring s3:TagResource (below) did not work at first, and the reason turned out to be a separate defect one layer down. The AWS SDKs send CreateBucket as PUT /<bucket>/, and streaming_route in crates/fakecloud-core/src/dispatch.rs read the trailing slash as an object upload, so the body went to the handler unbuffered. Everything that runs before the handler therefore saw an empty body — IAM enforcement could not read CreateBucketConfiguration at all. The handler itself was unaffected, because it drains the stream into req.body for non-object PUTs, which is why the tags applied even though the authorization layer could not see them.

This was not limited to tags: with the old predicate, a trailing-slash CreateBucket also dropped LocationConstraint (so any non-us-east-1 create sent that way would have failed IllegalLocationConstraintException), and PUT /bucket/?tagging saw an empty body too.

The gate and the router had each derived "where does the bucket end and the key begin" separately, and drifted — twice (the trailing slash, then a doubled slash, then the virtual-hosted bucket-in-path shape). So the router's derivation is now extracted into s3_routing_path and the gate calls it: streaming requires the routed path to split into a bucket and a non-empty key, by the same split-and-drop-empty rule path_segments uses. They are the same code now and cannot drift again.

Changes

  • parse the <Tags> tag set out of CreateBucketConfiguration and store it on the new bucket
  • validate_tags runs before any state mutation, so duplicate keys and aws:-prefixed keys fail with InvalidTag and no bucket is created
  • persist the Tags subresource (bucket tags are not part of BucketMeta), written before meta.toml and before the in-memory insert, so a store failure leaves no half-created bucket; an untagged create deletes the subresource, so a previously half-written tags.toml can't be adopted by a later untagged create of the same name
  • authorize a tagged create as s3:CreateBucket and s3:TagResource, per the model text on CreateBucketConfiguration$Tags; an untagged create still needs only s3:CreateBucket
  • feed the create-time tag set to aws:RequestTag/<key> and aws:TagKeys
  • fix streaming_route as described above

Test plan

  • Unit (fakecloud-s3): tag set stored; no-tags case; InvalidTag on duplicate keys leaves no bucket; XML-entity round-trip; the CreateBucket + TagResource action pair (and that an untagged create yields only CreateBucket); request-tag extraction for both actions; helper gating on the CreateBucketConfiguration root; idempotent us-east-1 re-create does not re-apply the tag set.
  • Unit (fakecloud-core): streaming_route for /b/, /b//, a real key ending in /, and the virtual-hosted bucket-in-path shape.
  • E2E: create-with-tags (plus PutBucketTagging still replacing the set wholesale); duplicate-key rejection; tags surviving a restart; an untagged create clearing an orphan tags.toml; the s3:TagResource requirement; an aws:RequestTag-conditioned create.
  • Each of the three behavioral fixes was verified by reverting it and confirming the new test fails (tag persistence → NoSuchTagSet after restart; orphan clear → inherited ghost tag; virtual-hosted gate → still streams).
  • Suites: fakecloud-core 331 passed, fakecloud-s3 448+ passed, e2e s3 + s3_persistence + iam_enforcement + iam_enforcement_abac + s3_streaming_body + s3_aws_chunked + s3_access_points + s3_anonymous_access + s3_post_object + ecr all green, fakecloud-conformance --test s3 51 passed. cargo clippy --all-targets -- -D warnings and cargo fmt --check clean.

Surface sync

  • Reference docs — website/content/docs/services/s3.md gains a "Bucket tags at create time" bullet; website/content/docs/reference/security.md updates the aws:RequestTag/<key> row for the CreateBucket tag set and its s3:TagResource requirement.
  • SDKs / introspection — no change: this is AWS API behavior, not a /_fakecloud/* surface, so no first-party SDK touches it.
  • Counts / metadata — unchanged: no new operation, service, or conformance variant, so conformance-baseline.json, the ops index, the README headline, and the GitHub repo description all stay as they are.
  • README / AGENTS / CLAUDE / examples / changelog — checked, none of them assert CreateBucket tag behavior.

Known limitation (pre-existing, now documented)

Under virtual-hosted addressing real S3 treats the whole path as the key, so a key beginning with <bucket>/ is not representable — the router reads it as a client that put the bucket in the path. Nothing in a single request tells the two apart. Moved verbatim into s3_routing_path and now stated in its doc comment rather than implied.

Fixes #2553

🤖 Generated with Claude Code

https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf


Summary by cubic

CreateBucket now honors the Tags tag set in CreateBucketConfiguration, storing it on the new bucket so GetBucketTagging returns the tags immediately instead of NoSuchTagSet. This fixes the AWS Terraform provider (6.x) reading back tags = {} after a clean create.

A tagged create now also requires s3:TagResource alongside s3:CreateBucket and populates aws:RequestTag/* and aws:TagKeys, matching real S3. Fixes a dispatch bug where bucket-level PUTs like PUT /<bucket>/ were streamed unbuffered, so IAM enforcement saw an empty body; they now buffer through the same routing helper as object PUTs.

  • Tag validation runs before any state mutation; duplicate and aws:-prefixed keys fail with InvalidTag and no bucket is created.
  • Tags persist in their own subresource, written before bucket meta; an untagged create deletes any orphan tag file.

Written for commit c9c5308. Summary will update on new commits.

Review in cubic

vieiralucas and others added 6 commits September 24, 2026 17:32
CreateBucket accepted the `Tags` tag set that was added to
CreateBucketConfiguration in the 2025 S3 API, but dropped it: only
LocationConstraint was parsed out of the body, so the bucket was created
untagged and GetBucketTagging answered NoSuchTagSet.

That path matters for Terraform. The AWS provider (6.x) tags a bucket
this way on create and skips PutBucketTagging when the create succeeds,
so an `aws_s3_bucket` with `tags` applied cleanly and then read back
`tags = {}`.

- parse the `<Tags>` tag set out of CreateBucketConfiguration with the
  same parse_tagging_xml the PutBucketTagging path uses, and store it on
  the new bucket
- run validate_tags first, so duplicate keys and `aws:`-prefixed keys
  fail with InvalidTag before any bucket is created
- write the Tags subresource alongside the bucket meta, since bucket tags
  are not part of BucketMeta and would otherwise be lost on restart
- unit tests for the stored tag set, the no-tags case, InvalidTag
  rejection leaving no bucket behind, and XML-entity round-tripping
- e2e coverage for create-with-tags (plus PutBucketTagging still
  replacing the create-time set), the duplicate-key rejection, and tags
  surviving a restart in persistent mode
- document the behavior on the S3 service page

Fixes #2553

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
Tagging a bucket at create time was authorized as nothing but
s3:CreateBucket. The S3 model is explicit on CreateBucketConfiguration$Tags:
"You must have the s3:TagResource permission to create a general purpose
bucket with tags", so a principal granted s3:CreateBucket and denied
s3:TagResource could tag a bucket that real S3 would refuse to create.

Fixing that surfaced a second bug one layer down. The AWS SDKs send
CreateBucket as `PUT /<bucket>/`, and dispatch's streaming heuristic read
the trailing slash as an object upload, so the body was handed to the
handler unbuffered. Everything that runs before the handler therefore saw
an empty body: IAM enforcement could not read CreateBucketConfiguration at
all, which is why a policy conditioned on the create-time tag set never
matched. The handler itself was unaffected -- it drains the stream into
`req.body` for non-object PUTs, which is why the tags applied even though
the authorization layer could not see them.

- streaming_route now requires a non-empty key after the bucket on the
  path-style route, so `PUT /<bucket>/` is buffered like `PUT /<bucket>`
- S3 overrides iam_actions_for to append s3:TagResource when the create
  body carries a tag set; dispatch already requires every returned action
  to be allowed, so an untagged create still needs only s3:CreateBucket
- s3_request_tags covers CreateBucket and TagResource, so the create-time
  tag set populates aws:RequestTag/<key> and aws:TagKeys
- create_bucket_configuration_tags replaces the handler's ad-hoc
  `contains("<Tags>")` gate, which would have silently dropped the tag set
  from a client that wrote `<Tags xmlns=...>` or any attribute on the
  element
- unit tests for the action pair, the condition-key extraction, and the
  helper's gating; e2e tests for the TagResource requirement and for an
  aws:RequestTag-conditioned create; a dispatch regression test for the
  trailing-slash route

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
Write the Tags subresource ahead of meta.toml and the in-memory insert, so
a store error during CreateBucket leaves nothing usable behind: the loader
skips a bucket directory that has no meta.toml, so the failure mode is now
"no bucket" rather than "bucket exists, permanently untagged" -- the same
guarantee the InvalidTag path already gives.

Also document why the us-east-1 idempotent re-create does not re-apply the
create-time tag set (it re-applies no create-time setting -- not the canned
ACL, object lock, or object ownership either), with a test locking that in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
…te with routing

Two holes the previous commits left:

- An untagged CreateBucket now deletes the Tags subresource instead of
  leaving it alone. Writing tags.toml before meta.toml means a create that
  fails partway leaves an orphan tag file in a directory the loader skips
  (no meta.toml) -- but a later untagged create of that name would adopt
  the never-committed tag set on the next restart. Deleting is a no-op in
  the normal case, where no such file exists.
- streaming_route derived the path-style key with split_once, which calls
  the key of `PUT /<bucket>//` "/" and streams it, while routing filters
  empty segments and reads the same path as a bucket-level operation --
  the exact empty-body mismatch the trailing-slash fix was for. The key is
  now derived with the same split-and-drop-empty rule the request builder
  uses, so the two cannot disagree.

Adds a persistence e2e for the orphan tag file (verified failing without
the delete), plus dispatch tests for the doubled-slash path and for a real
object key that ends in '/', which must still stream.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
The gate was tightened for path-style addressing only. A virtual-hosted
request that also carries the bucket in the path (`Host: b.s3...` with
`PUT /b` or `PUT /b/`) still streamed, even though routing has an explicit
bucket-root branch that collapses exactly that shape to CreateBucket -- so
the handler and IAM saw an empty body, the CreateBucketConfiguration tag
set was dropped, and iam_actions_for omitted the s3:TagResource
requirement. Same failure mode as the path-style one, left open for the
one shape the router already special-cases.

Rather than tighten a second predicate, the router's path derivation moves
into s3_routing_path and the gate calls it: streaming now requires that the
routed path split into a bucket AND a non-empty key, by the same
split-and-drop-empty rule the request builder uses for path_segments. The
two cannot drift again -- they are the same code -- which is what produced
both of these bugs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
Under virtual-hosted addressing real S3 treats the whole path as the key, so
a key beginning with `<bucket>/` (or a key that is the bucket name) is not
representable here: the passthrough reads it as a client that put the bucket
in the path and drops the prefix. Nothing in a single request tells the two
apart, so the behavior resolves the ambiguity in favor of the mixed-addressing
client -- pre-existing, moved verbatim into the helper, now stated rather than
implied, including in the test that exercises the shape.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015paJopoDarw6WeFFoZMDMf
@vieiralucas
vieiralucas requested a lite review from Copilot September 24, 2026 21:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vieiralucas
vieiralucas merged commit c7727e4 into main Sep 25, 2026
157 checks passed
@vieiralucas
vieiralucas deleted the worktree-s3-createbucket-tags branch September 25, 2026 00:38
vieiralucas added a commit to Sorttech/fakecloud that referenced this pull request Sep 29, 2026
…ed with

CreateBucket set acl_grants, object_lock_config and ownership_controls in memory
and wrote none of them. None live in BucketMeta, and the loader restores the
latter two as None and falls back to the default owner grant when acl.toml is
absent, so after a restart a bucket created with `x-amz-acl: public-read` came
back owner-only and one created with `x-amz-bucket-object-lock-enabled: true`
came back with no lock configuration at all -- which silently stops retention
being enforced. Versioning survived, since that one IS in meta, which is what
made the gap easy to miss.

Each of the three is now written when the create sets it, and the bucket's
stored subresources are cleared first so a create cannot inherit a sidecar left
by a create or delete that stopped partway, or by a `/_fakecloud/reset` (which
clears memory and deliberately leaves the store alone). The meta is written
before that clear, so a create that fails has destroyed nothing; `objects/` is
never touched, because the loader skips a bucket whose objects it cannot read
and re-creating the name must not be what destroys data that is still
recoverable.

Persisting the ACL made a set of latent ACL bugs durable, so they are fixed
here too:

- A canned `x-amz-acl`, the `x-amz-grant-*` headers and an AccessControlPolicy
  body are mutually exclusive, as on S3. Letting one win silently discarded what
  the others asked for.
- Canned values are validated against the set the target accepts. An
  unrecognized one fell through to owner-only, so a typo wiped every public
  grant. The object-scoped `bucket-owner-*` values are accepted on a bucket and
  resolve to owner-only, which is what S3 ignoring them produces.
- A grant naming no resolvable grantee is refused rather than stored as an entry
  that can never match anyone, and grantee values are trimmed so a pretty-printed
  <ID> does not bake whitespace into a canonical id.
- `emailAddress=` grantees are kept as AmazonCustomerByEmail instead of being
  dropped, and logged as unenforced: nothing resolves them to a canonical user,
  so the grant conveys no access.
- PutBucketAcl honors the grant headers; without that arm a
  `put-bucket-acl --grant-read` fell through to an empty body and wiped the ACL.
- PutObjectAcl updates the versioned copy of the object too, so a versionId read
  no longer answers a different ACL than the unversioned one.
- An ACL on a bucket whose ownership disables them is refused by every path that
  can set one, CreateMultipartUpload included.
- PutBucketOwnershipControls validates every ObjectOwnership the document
  carries -- `bucket_owner_enforced` matches the stored text, so a second rule
  could turn ACLs off while the API reported otherwise -- and persists before
  mutating memory.

And three storage-integrity fixes in the same paths:

- Every writer of a persisted snapshot propagates a serialization failure rather
  than writing an empty document. An empty acl.toml is worse than none: the
  loader takes its presence to mean an explicit ACL and drops the owner grant.
  A sidecar whose grants are all unusable now falls back to that owner grant too.
- delete_object removes the sidecar before the body. `load` iterates `*.toml`
  and treats a sidecar with no body as fatal for the bucket, while an orphan
  body is never read, so the body is the residue an interrupted delete should
  leave. A body that cannot be unlinked warns rather than failing a delete that
  did happen.
- The analytics, intelligent-tiering and metrics sidecars neither default
  silently (losing the configurations, which the next Put then makes permanent)
  nor fail the bucket over a reporting config: they are skipped with a warning.

Fixes the create-time half of the gap found while reviewing faiscadev#2557.
fas89 added a commit to Agenticstiger/forge-cli that referenced this pull request Oct 5, 2026
…does

terraform-provider-aws 6.23 and later tag a bucket in its CreateBucket request
(CreateBucketConfiguration Tags, added to S3 in 2025) and skip PutBucketTagging
when the create succeeds. moto 5.2.3 reads only LocationConstraint from that body
and drops the tags, so every bucket the moto lanes applied was left untagged, and
the state-drift lane read a clean apply as tag drift (tags.fluid_contract,
tags.managed_by). On real S3 the tags are kept: eleven applied AWS products
planned no change on provider 6.67.

tests/iac/_moto_s3.py wraps moto's bucket PUT so a create also stores the body's
tags, the fix fakecloud made in its own S3 (faiscadev/fakecloud#2557), and
tests/iac/conftest.py applies it once per process. The drift lane passes
locally with it (5 of 5), and failed without it (2 of 5).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

S3 CreateBucket ignores Tags in CreateBucketConfiguration

2 participants