Skip to content

fix(auth): use default R2 endpoint for import-child s3 checks (0.27.1) - #192

Merged
johnyaku merged 1 commit into
mainfrom
fix/auth-s3-import-endpoint
Sep 16, 2026
Merged

johnyaku merged 1 commit into
mainfrom
fix/auth-s3-import-endpoint

Conversation

@johnyaku

Copy link
Copy Markdown
Contributor

Follow-up to #191. Re #190.

The remaining false failures

After #191 merged, dt auth check --type s3 still false-failed every import-source S3 endpoint. On the portal project:

✗ s3://chromium   endpoint unreachable / timed out
    Hint: Check network access to the S3 endpoint: AWS S3
✗ s3://visium     endpoint unreachable / timed out ...
✗ s3://wgs / wts / xenium  (same)

Root cause

An import child's remote config lives in the source repo, not the local .dvc/config. So _resolve_s3_remote_settings recovered the per-repo profile (from the source repo name, e.g. bcarc_chromium) but left endpoint_url = None. boto3 then defaulted to real AWS S3 instead of the Cloudflare R2 endpoint — hence "unreachable / timed out" against AWS S3 for every bucket. (The hint printing AWS S3 was the tell.)

Fix

Fall back to the org-wide secrets.default_endpointurl (dt config) when no per-remote endpoint URL is found. These per-repo R2 remotes all share one endpoint, so this is exactly right, and it only fills the gap — a remote whose own config supplies an endpoint keeps it.

Verified on portal (real R2)

s3://chromium  prof=bcarc_chromium  ep=https://…r2.cloudflarestorage.com/
     -> pass: credentials OK (profile 'bcarc_chromium'), bucket accessible
s3://visium    -> pass (profile 'bcarc_visium')
s3://wgs       -> pass (profile 'bcarc_wgs')
s3://wts       -> pass (profile 'bcarc_wts')
s3://xenium    -> pass (profile 'bcarc_xenium')

All six go from ✗ "unreachable (AWS S3)" to ✓ "credentials OK … bucket accessible" — matching the ground truth in #190 (all were actually healthy).

Tests

Added test_import_child_falls_back_to_default_endpoint and test_local_endpoint_not_overridden_by_default; hardened the existing resolve test. test_auth.py 337 passed.

Version 0.27.0 → 0.27.1.

🤖 Generated with Claude Code

#190)

The 0.27.0 S3 rewrite still false-failed every import-source S3 endpoint.
An import child's remote config lives in another repo, not the local
.dvc/config, so `_resolve_s3_remote_settings` recovered the per-repo
profile (from the source repo name) but left `endpoint_url` as None.
boto3 then talked to real AWS S3 instead of the R2 endpoint, and every
bucket reported "endpoint unreachable / timed out" against `AWS S3`.

Fall back to the org-wide `secrets.default_endpointurl` when no
per-remote endpoint is found — these per-repo remotes all share it.

Verified on the portal project: s3://chromium, visium, wgs, wts, xenium
all go from "unreachable (AWS S3)" to "credentials OK (profile
'bcarc_<x>'), bucket accessible" against R2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@johnyaku
johnyaku merged commit 97c3208 into main Sep 16, 2026
1 check passed
@johnyaku
johnyaku deleted the fix/auth-s3-import-endpoint branch September 16, 2026 05:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant