Skip to content

fix(s3): Garage/self-hosted S3 compatibility — path-style + SigV4, per-origin CORS rules#180

Open
nickgnat wants to merge 2 commits into
Techiebutler:mainfrom
nickgnat:fix/garage-s3-compat
Open

fix(s3): Garage/self-hosted S3 compatibility — path-style + SigV4, per-origin CORS rules#180
nickgnat wants to merge 2 commits into
Techiebutler:mainfrom
nickgnat:fix/garage-s3-compat

Conversation

@nickgnat

Copy link
Copy Markdown

Summary

Makes self-hosted S3 backends — specifically Garage, but also MinIO-behind-a-proxy setups — work out of the box, fixing two issues found running FreeFrame against Garage v2.3.0 in production:

  1. Presigned/client URLs: in non-AWS mode, boto3's defaults can produce virtual-host-style URLs (which need wildcard bucket DNS that self-hosted stores behind a reverse proxy usually don't have) and SigV2 presigned URLs, which Garage rejects with Received an unknown query parameter: 'AWSAccessKeyId'.
  2. Bucket CORS: the startup CORS config puts both allowed origins in a single rule. Garage responds to that rule by joining all AllowedOrigins into one comma-separated Access-Control-Allow-Origin header, which browsers hard-reject — so every HLS segment fetch failed with HLS error: networkError, and browser-direct presigned uploads failed the same way.

Changes

  • _build_s3_client() / _get_presign_client(): non-AWS mode (S3_STORAGEs3) now applies Config(s3={"addressing_style": "path"}, signature_version="s3v4"), merged with any caller-supplied config (the startup-timeout config keeps its timeouts). AWS mode is untouched — no endpoint, no forced config.
  • ensure_bucket_exists(): startup bucket CORS emits one rule per origin (deduped). AWS-style backends echo only the matching origin either way, so behavior there is identical; on Garage this is the difference between a valid single-origin ACAO header and an invalid joined one.
  • docs/deployment.md: Garage row in the provider endpoint table + a note that non-AWS mode is path-style + SigV4.
  • New tests (test_s3_non_aws_compat.py): client config forced/merged in non-AWS mode, AWS mode untouched, per-origin CORS rules incl. dedupe.

Testing

  • Backend tests pass (204 passed against a real Postgres 15 + alembic upgrade head, mirroring CI)
  • Frontend tests + build pass — no web changes
  • Tested manually in browser: verified against a production Garage v2.3.0 deployment — HLS playback (manifest proxy → presigned path-style SigV4 segment URLs), HeadBucket, and per-origin CORS echo (matching origin returned single-valued; unknown origins get no CORS grant)

Checklist

  • CHANGELOG.md entry added under ## [Unreleased] (for user-facing changes)

Screenshots

n/a — backend only


🤖 Generated with Claude Code

nickgnat and others added 2 commits July 17, 2026 09:17
…r-origin CORS rules

Non-AWS mode (S3_STORAGE != s3) now forces path-style addressing and SigV4
on both the server-side and presign clients. boto3 could otherwise emit
virtual-host-style URLs (broken behind reverse proxies without wildcard
bucket DNS) and SigV2 presigned URLs, which Garage rejects.

Startup bucket CORS is emitted as one rule per origin: Garage joins a
rule's AllowedOrigins into a single comma-separated
Access-Control-Allow-Origin header, which browsers reject — every HLS
segment fetch and presigned upload then fails CORS. Per-origin rules
behave identically on AWS-style backends.

AWS mode is unchanged in both cases.
botocore's Config.merge lets the *argument* win, so
_NON_AWS_COMPAT_CONFIG.merge(config) let a caller override the path-style
and SigV4 settings the surrounding comment describes as forced — and
replaced the `s3` sub-dict wholesale rather than deep-merging it. Merge
onto the caller's config instead, so the compat baseline holds while
non-conflicting caller options (the startup timeouts) still apply.

Cover it with a test that passes a deliberately conflicting config; it
fails on the previous merge direction.

Also document the one-rule-per-origin requirement in the manual bucket
CORS section, which still showed a single-rule example — the automatic
startup path already emits per-origin rules.

@ravirajsinh45 ravirajsinh45 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — well-diagnosed and unusually well-prepared. Thanks @nickgnat.

I ran this locally against Postgres 15 + MinIO rather than reviewing it on paper, and the SigV4 half turns out to matter more than the PR claims. On the real multipart part-upload path against MinIO:

no Content-Type Content-Type: video/mp4
main (SigV2) 200 403 SignatureDoesNotMatch
this PR (SigV4) 200 200

SigV2's string-to-sign includes Content-Type and boto3 presigns it empty, so browser uploads to MinIO survive today only because upload-store.ts:192 calls file.slice(start, end) without a contentType argument — that returns a typeless Blob, so the browser omits the header. Anyone who "fixes" that line to preserve file.type would break every part upload on MinIO with an opaque 403. Your change removes that landmine for MinIO as well as Garage, so it's worth more than the Garage-only framing in the description.

Rest of what I verified: full API suite at 205 passed against a real Postgres (your 204 + one test I added); presigned URLs are SigV4 + path-style with a real PUT/GET round-trip against MinIO; every boto3.client site in the repo goes through the two constructors you patched; the CORS change is still a single put_bucket_cors call carrying N rules, and ensure_bucket_exists() stays non-fatal when the backend rejects it — this MinIO build answers PutBucketCors with NotImplemented, so that path no-ops with a warning there and your Garage testing is the only real coverage it has. The payload itself is correct: two rules, one origin each, ETag exposed on both.

I pushed one commit to your branch rather than sending you round again for two small things — shout if you'd rather I hadn't:

  1. _NON_AWS_COMPAT_CONFIG.merge(config) gives the caller precedence, not the baseline — botocore's merge does config_options.update(other_config._user_provided_options), and the s3 dict is replaced wholesale rather than deep-merged, so a caller passing signature_version or its own s3={...} could silently downgrade the settings the comment calls forced. No live bug, since _STARTUP_S3_CONFIG only sets timeouts and retries. Flipped to config.merge(_NON_AWS_COMPAT_CONFIG) and added a test that passes a deliberately conflicting config — it fails on the old direction with assert 's3' == 's3v4'.
  2. Added a line to the manual bucket-CORS section of docs/deployment.md, which still showed a single-rule example — the same shape you just fixed in the automatic path.

One thing worth recording, not a change request: the addressing_style: "path" half is already botocore's default once endpoint_url is set (botocore/args.py:836force_path_style = s3_config.get('addressing_style') != 'virtual'). Still worth pinning, since an ~/.aws/config with addressing_style = virtual would flip it and an explicit Config wins over the shared config file — but SigV4 is doing the real work. I left your CHANGELOG wording alone since "could emit virtual-host-style URLs" stays true in that case.

Two adjacent things I found that are not yours to fix, noted so they're on record: AWS mode has the identical SigV2 presign behavior in us-east-1 (our default region), since it deliberately gets no config — same 403-on-Content-Type exposure, against a bucket policy that's stricter than MinIO's. And the startup bucket CORS ignores the CORS_ALLOW_ORIGINS setting from #173 while rewriting bucket CORS on every boot, so there's no manual workaround for an extra origin. We'll pick both up separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants