Skip to content

Separate virtual chunk credentials from repo storage#57

Open
charles-turner-bot wants to merge 2 commits into
ACCESS-NRI:mainfrom
charles-turner-bot:ct/separate-virtual-chunk-creds
Open

Separate virtual chunk credentials from repo storage#57
charles-turner-bot wants to merge 2 commits into
ACCESS-NRI:mainfrom
charles-turner-bot:ct/separate-virtual-chunk-creds

Conversation

@charles-turner-bot

Copy link
Copy Markdown
Contributor

Summary

This separates Icechunk repo storage auth from virtual chunk container auth.

Fixes #55.

What changed

  • add _resolve_vcc_credentials() to build authorize_virtual_chunk_access
    separately from repo storage_options
  • add virtual_chunk_credentials_options to IcechunkCatalog
  • thread the same runtime-only credential split through IcechunkCatalog.from_json()
    and derived child catalogs
  • add virtual_chunk_credentials_options to IcechunkStoreBuilder so builds can
    authorise virtual chunk access independently from the target Icechunk store
  • add regression tests for the new split on both the builder and catalog paths

Why

We already had a decent separation for non-secret virtual chunk store config
(endpoint/region/etc.) via the serialised virtual_chunk_model, but credential
handling still effectively assumed "whatever the repo store does".

That made cases like these awkward to reason about:

  • anonymous Icechunk store + authenticated virtual chunk container
  • different auth modes for the repo store and source-data container
  • re-opening a catalog from JSON while still needing explicit virtual chunk creds

This change keeps the existing sidecar behaviour for non-secret config, while
making credentials an explicit runtime concern.

Testing

  • pixi run pytest tests/test_utils.py tests/test_core.py tests/test_source_build.py -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Does the current config sharing force the virtual chunk container & the icechunk store to have the same config?

1 participant