You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat(storage): identity-only blob access, drop BLOB_CONNECTION_STRING - #252
Lets the apps reach blob storage with their managed identity only, so BLOB_CONNECTION_STRING (an account key) can be removed from deployed apps and shared-key access can later be disabled on the data account.
download_blob_to_tempfile / read_blob_range fall back to BLOB_ACCOUNT_URL + identity when no connection string is set.
Read links are per-blob, read-only, 60-minute SAS tokens (blob_read_sas), signed with a user-delegation key in Azure and the account key for the emulator. The container-wide stored policy is no longer created.
Stored image-layer and label-tile URLs stay unsigned. Fresh tokens are minted at response time for the label project, labeling workspace, project details (previews) and visualizer, including legacy stored URLs that carry an old policy token.
functions.bicep and deploy_apps.sh stop setting BLOB_CONNECTION_STRING; deploy_apps.sh deletes the leftover setting. infra/main.json regenerated.
Spec and ADR-0008 added.
Out of scope: Azure Files /data mount key, Batch account key, ACS connection string, and the allowSharedKeyAccess: false change (separate, gated follow-up).
Type of change
Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that changes existing behavior)
python -m pytest hastelib/tests -q --continue-on-collection-errors: 143 failed, 15 errors, 742 passed. The same 158 failures/errors occur on the base commit (local environment: missing osgeo, Windows fcntl); no new failures. hatch run test:pytest and npm run lint were not run.
black -l 79 and flake8 clean on touched Python files.
Not yet validated on a live environment: the rollout is dev1 with the setting removed, then dev2, prod, then deleting the policy and disabling shared-key access.
Additional context
Rollback for the config change: re-add BLOB_CONNECTION_STRING.
Review follow-up
The legacy setting is now deleted only after the app package is published, and the deploy script parses (bash -n).
Project-details URLs (previews, labels, training/inference zips) are signed before the ETag is computed.
Labeling responses return tileCredentialExpiresAt; the labeling tool, interactive labeler and building validation refresh tile URLs in place before expiry (ported from the community fork's helper, without its partner scoping). Unsaved labels and selections are untouched.
The visualizer does the same for the displayed prediction version: GetVisualizerResults returns ileCredentialExpiresAt; the page re-requests that version and swaps only the tile layer URLs (maps are not rebuilt; a version switch drops an in-flight refresh).
Tests: node --test src/util/tileCredentialRefresh.test.js passed (9); predictionResults.test.js passed (26); blob and loading-route tests pass; remaining API failures are the Windows-only prediction/footprint route tests. ESLint and the Vite build could not be run locally (no installed UI dependencies).
Second review follow-up: footprint and valid-area downloads and nested model artifact zips are re-signed per response; escaped blob names are decoded before signing; query strings are stripped only from HASTE-owned raster URLs; the visualizer refresh updates live layers atomically (failures retry) and applies fresh URLs on version switches.
Record the design and ADR for removing the storage account key from app settings and minting blob-scoped, short-lived read tokens signed by the app identity.
download_blob_to_tempfile and read_blob_range read only BLOB_CONNECTION_STRING and failed when it was empty. Route both through one credential-aware client that prefers the connection string (local emulator) and otherwise uses BLOB_ACCOUNT_URL with the app identity.
get_file_remote_path signed a container-wide token under a stored policy. Sign a one-blob, one-hour token instead, with the app identity on real Azure and the account key only for the emulator. The container read policy is no longer created, so deleting it revokes old tokens for good.
get_download_url signed a container-wide token under a stored policy. Sign a one-blob, one-hour token instead, using split_blob_url for artifact paths so Azurite hosts resolve. Remove the now-unused container read policy helper.
Blob and queue access is identity-based, so the account-key connection string is no longer set by Bicep or deploy_apps.sh. deploy_apps.sh also deletes the leftover setting from existing apps. Compiled main.json regenerated.
Adds sign_internal_blob_url, sign_titiler_tile_url and sign_label_project_urls so stored URLs can stay unsigned and be given a fresh per-blob read token when handed out.
Label tile URLs and visualizer layers no longer embed a token in persisted data; labels sign only around their own raster read. The label project, labeling workspace and project-details endpoints mint fresh per-blob tokens at response time, including for legacy stored URLs that carried a policy SAS.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
A deployment-script syntax error blocks deployment, credentials are removed before compatible code is published, and inadequate URL renewal breaks previews, long-running tile sessions, and persisted downloads.
The setting was removed before the package was published and a stray edit broke the script's syntax. Delete it after a successful publish so a failed deploy keeps the old code working.
Preview URLs are signed inside the cached load so the ETag covers the tokens the client holds and a 304 cannot revive an expired one. Label and zip download URLs in project details are re-signed the same way.
Label routes now include tileCredentialExpiresAt, matching the lifetime of
the per-blob read tokens signed into the tile URLs, so the UI can refresh
them before they expire.
Tile URLs carry read tokens that expire after 60 minutes. Add a controller
that refreshes them shortly before expiry (and on tab focus), merging only
the imagery URLs into existing state and TileLayers so unsaved labels and
selections are never replaced, with retry and rollback on failure.
Wire the refresh controller into the labeling tool, interactive labeler
(including the swipe pre-event layer) and building validation. Refreshed
imagery URLs are applied to existing TileLayers in place, so drawn labels,
selections and the loaded model stay untouched; a banner offers retry or
reload if a refresh fails.
Re-requests the displayed prediction version and swaps only the tile layer URLs on both maps. The scene key ignores the signed token so maps are not rebuilt, and a version switch drops any refresh in flight.
When artifact_path contains an escaped blob name such as a%20b.tif, split_blob_url passes that encoded name into the new blob-scoped signature. The actual blob is named a b.tif, so the resulting download token fails authentication. Decode the blob name before signing, as sign_internal_blob_url already does.
Preserve query credentials for external raster URLs
For a provider or other-account raster URL that requires query credentials, stripping the query here leaves TiTiler with an unauthenticated URL because sign_internal_blob_url cannot re-sign that source. Pass the original URL to the signer, which already replaces internal tokens while preserving external URLs. The same issue occurs in hastelib/src/hastegeo/core/processors/labels.py:80; only strip queries from HASTE-owned blobs when persisting tile templates.
…entials
Sign layer footprint and valid-area downloads and the nested model artifact zips at response time, decode escaped blob names before signing artifact downloads, and strip query strings only from HASTE-owned raster URLs so provider credentials survive.
…switch
The refresh now updates the existing layers through one applier that rolls back and throws on failure, so errors reach the controller's retry path and banner. A new results snapshot also updates the imagery layers from its own URLs instead of leaving the old ones in place.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
URL-signing and refresh-state bugs can strip required imagery credentials, retain expired overlays, and leave stale error banners after version or model changes.
A signed imagery URL in another container of the same account (for example, /provider/image.tif?sig=... when BLOB_CONTAINER=data) loses its credentials here because _internal_blob_client only checks the account endpoint. sign_internal_blob_url then refuses to sign that container, so the visualizer and generated label tiles receive an unsigned source that cannot be read. Check the configured container before stripping the query, matching the signing helper's scope.
Clear refresh errors when switching results or models
A failed refresh sets failed for the whole hook, but switching versions or loading a different model never clears it. Those operations install fresh credentials and replace the controller, yet this banner remains visible until the new controller's first successful refresh—normally about 55 minutes later. Associate the failure with the results snapshot that failed, as is already done for override, so a successful version/model load does not retain an obsolete error.
unsigned_internal_blob_url stripped the query for any blob in the app account, but signing only covers BLOB_CONTAINER, leaving other containers unreadable.
…pshot
A refreshed override with unavailable prediction rasters no longer falls back to stale signed URLs, and a refresh failure no longer outlives the results it failed for.
All branch deployment references use the same RC tag:
hastegeo_version: 1.0.58rc38
training_image_tag: 1.0.58rc38
imageprep_image_tag: 1.0.58rc38
wheel: hastegeo-1.0.58rc38-py3-none-any.whl
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lets the apps reach blob storage with their managed identity only, so
BLOB_CONNECTION_STRING(an account key) can be removed from deployed apps and shared-key access can later be disabled on the data account.download_blob_to_tempfile/read_blob_rangefall back toBLOB_ACCOUNT_URL+ identity when no connection string is set.blob_read_sas), signed with a user-delegation key in Azure and the account key for the emulator. The container-wide stored policy is no longer created.functions.bicepanddeploy_apps.shstop settingBLOB_CONNECTION_STRING;deploy_apps.shdeletes the leftover setting.infra/main.jsonregenerated.Out of scope: Azure Files
/datamount key, Batch account key, ACS connection string, and theallowSharedKeyAccess: falsechange (separate, gated follow-up).Type of change
Checklist
cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)Testing
python -m pytest hastelib/tests/core/utils/test_blob_signing.py hastelib/tests/core/utils/test_blob_sas.py hastelib/tests/core/utils/test_blob.py -q: passed.python -m pytest hastelib/tests -q --continue-on-collection-errors: 143 failed, 15 errors, 742 passed. The same 158 failures/errors occur on the base commit (local environment: missingosgeo, Windowsfcntl); no new failures.hatch run test:pytestandnpm run lintwere not run.black -l 79andflake8clean on touched Python files.Additional context
Rollback for the config change: re-add
BLOB_CONNECTION_STRING.Review follow-up
The legacy setting is now deleted only after the app package is published, and the deploy script parses (
bash -n).Project-details URLs (previews, labels, training/inference zips) are signed before the ETag is computed.
Labeling responses return
tileCredentialExpiresAt; the labeling tool, interactive labeler and building validation refresh tile URLs in place before expiry (ported from the community fork's helper, without its partner scoping). Unsaved labels and selections are untouched.The visualizer does the same for the displayed prediction version: GetVisualizerResults returns ileCredentialExpiresAt; the page re-requests that version and swaps only the tile layer URLs (maps are not rebuilt; a version switch drops an in-flight refresh).
Tests:
node --test src/util/tileCredentialRefresh.test.jspassed (9); predictionResults.test.js passed (26); blob and loading-route tests pass; remaining API failures are the Windows-only prediction/footprint route tests. ESLint and the Vite build could not be run locally (no installed UI dependencies).Second review follow-up: footprint and valid-area downloads and nested model artifact zips are re-signed per response; escaped blob names are decoded before signing; query strings are stripped only from HASTE-owned raster URLs; the visualizer refresh updates live layers atomically (failures retry) and applies fresh URLs on version switches.