Repository navigation
feat(gdal-runtime): integrate GDAL 3.13 runtime across consumers - #240
Merged
Merged
Conversation
Joaquín Rivero (jQuinRivero)
had a problem deploying
to
gdal-runtime-build
October 16, 2026 14:42 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Joaquín Rivero (jQuinRivero)
had a problem deploying
to
gdal-runtime-build
October 16, 2026 14:44 — with
GitHub Actions
Failure
GitHub Actions
Failure
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 16, 2026 14:48 — with
GitHub Actions
Active
GitHub Actions
Active
HASTE loads several GDAL builds per process today: the 3.9.2 bindings wheel plus the copies bundled inside rasterio, fiona and pyogrio. Driver restrictions only reach one of them, and no trusted GDAL 3.13 wheel exists for the Python 3.11 runtimes. The spec describes one GDAL 3.13.3 runtime built in CI from pinned sources: - every GDAL-linked binding links to it, so a process loads one libgdal; - it is published with a manifest, SBOM and attestation; - each service checks itself against the manifest at startup. The spec also covers opening untrusted inputs only with an explicit driver allowlist, and build-once Function App packages. ADR-0007 records the decision and amends ADR-0004.
Add sources.lock with the nine native sources of the HASTE GDAL runtime (GDAL 3.13.3, PROJ 9.8.1, sqlite, curl, OpenSSL, zlib, libdeflate, zstd, libjpeg-turbo) and their SHA-256 values taken from the upstream releases. fetch_sources.py downloads them and refuses any file whose hash does not match, so the build never compiles an unverified source.
haste_gdal_runtime ships inside the runtime wheel. configure() pins GDAL_DATA, PROJ_DATA and GDAL_CONFIG_FILE to the package, applies the operator settings (raw VRT bands and Python pixel functions off, plugin loading off, EMPTY_DIR) and lists disallowed drivers in GDAL_SKIP. harden() removes disallowed drivers and virtual file systems. report() and evaluate() compare the live process with the build manifest: GDAL version, a single libgdal, binding versions, driver and file-system inventory, configuration, and VRT probes run under both the configured and permissive settings. Work in progress: nothing has run against a real libgdal yet; the build that produces the manifest is the next step.
Mark the spec, ADR and source lock done, list the next steps in order, and record the build facts confirmed in the GDAL, PROJ and curl sources so the build scripts start from evidence.
Unit tests for sources.lock validation, mapped-library detection, configure() defaults and GDAL_SKIP merging, and evaluate() against synthetic runtime reports.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved defects affect deployment availability, local TiTiler operation, configuration enforcement, and GDAL error propagation.
6 open issues
What changed in this PR
Integrates a shared GDAL 3.13.3 runtime across HASTE’s APIs, tile server, imagery preparation, and training workflows.
Changes:
- Adds verified source builds, runtime checks, and pinned bindings.
- Hardens imagery ingestion and replaces training CLI calls with in-process operations.
- Adds prebuilt Function App deployment support, configuration guards, tests, and rollout documentation.
| File | Description |
|---|---|
| spec/features/gdal-3-13-runtime/user-stories.md | Defines acceptance criteria and ownership. |
| spec/features/gdal-3-13-runtime/test-plan.md | Defines runtime validation requirements. |
| spec/features/gdal-3-13-runtime/rollout.md | Documents rollout and rollback. |
| spec/features/gdal-3-13-runtime/README.md | Summarizes scope and success criteria. |
| spec/features/gdal-3-13-runtime/impact-analysis.md | Documents affected components and risks. |
| spec/architecture/decisions/0007-gdal-native-runtime.md | Records the shared-runtime architecture decision. |
| native/gdal-runtime/verify_build_manifest.py | Validates the native build baseline. |
| native/gdal-runtime/titiler_vrt_smoke.py | Checks reprojection with VRT disabled. |
| native/gdal-runtime/sources.lock | Pins native sources and checksums. |
| native/gdal-runtime/smoke_test.py | Orchestrates offline runtime checks. |
| native/gdal-runtime/requirements-smoke.txt | Locks smoke-test dependencies. |
| native/gdal-runtime/requirements-sdists.txt | Locks binding source archives. |
| native/gdal-runtime/requirements-fiona-build.txt | Pins Fiona’s build toolchain. |
| native/gdal-runtime/requirements-build.txt | Pins the common binding toolchain. |
| native/gdal-runtime/requirements-build-base.txt | Locks shared build dependencies. |
| native/gdal-runtime/patches/zlib-CVE-2026-85091.patch | Includes a zlib security patch. |
| native/gdal-runtime/install_build_tools.sh | Installs hash-verified build tools. |
| native/gdal-runtime/haste_gdal_runtime/gdalrc | Supplies controlled GDAL configuration. |
| native/gdal-runtime/haste_gdal_runtime/_native.py | Adds native inspection and security probes. |
| native/gdal-runtime/haste_gdal_runtime/__main__.py | Exposes manifest and verification commands. |
| native/gdal-runtime/fetch_sources.py | Downloads and verifies native sources. |
| native/gdal-runtime/fetch_binding_sdists.sh | Downloads hash-verified binding sources. |
| native/gdal-runtime/build_bindings.py | Builds bindings against shared libraries. |
| native/gdal-runtime/baseline_report.py | Captures runtime inventory and configuration. |
| infra/modules/functions.bicep | Applies Function App hardening settings. |
| hastelib/tests/core/utils/test_imagery.py | Tests restricted TIFF opening. |
| hastelib/tests/core/utils/test_imagery_runtime.py | Tests disguised-VRT mosaic rejection. |
| hastelib/tests/core/utils/test_gdal_security.py | Tests runtime activation and legacy hardening. |
| hastelib/tests/core/utils/test_footprints.py | Tests footprint content validation. |
| hastelib/tests/core/processors/test_uploader_format.py | Tests validation before chunk storage. |
| hastelib/tests/build/test_dependency_pins.py | Checks consumer runtime-pin consistency. |
| hastelib/src/hastegeo/workflows/prepare_imagery.py | Validates pre-event imagery downloads. |
| hastelib/src/hastegeo/core/utils/imagery.py | Restricts TIFF opens and mosaic inputs. |
| hastelib/src/hastegeo/core/utils/gdal_security.py | Adds runtime verification and header validation. |
| hastelib/src/hastegeo/core/utils/footprints.py | Restricts user footprints to verified GeoPackages. |
| hastelib/src/hastegeo/core/processors/uploader.py | Validates initial chunks before storage. |
| hastelib/src/hastegeo/__init__.py | Activates hardening on package import. |
| hastelib/pyproject.toml | Aligns geospatial dependency versions. |
| docs/security-configuration.md | Documents shared-runtime protections. |
| docker/training/scripts/entrypoint.sh | Propagates package-import failures. |
| docker/training/env/env.yml | Adopts Python 3.11 and shared bindings. |
| docker/training/Dockerfile | Applies runtime hardening settings. |
| docker/training/code/tests/test_run_workflow_predictions.py | Tests in-process conversion options. |
| docker/training/code/tests/test_gdal_runtime_integration.py | Tests mask rasterization and COG conversion. |
| docker/training/code/tests/test_create_masks_writes.py | Updates rasterization implementation checks. |
| docker/training/code/tests/test_create_masks_cleanup.py | Updates cleanup comments only. |
| docker/training/code/sitecustomize.py | Initializes hardening during Python startup. |
| docker/training/code/run_workflow.py | Replaces CLI COG conversion. |
| docker/training/code/output2visualizer.py | Stops swallowing hardening failures. |
| docker/training/code/merge_with_building_footprints.py | Stops swallowing hardening failures. |
| docker/training/code/inference.py | Stops swallowing hardening failures. |
| docker/training/code/fine_tune.py | Stops swallowing hardening failures. |
| docker/training/code/create_masks.py | Replaces CLI reprojection and rasterization. |
| docker/titiler/Dockerfile | Builds the local restricted TiTiler service. |
| docker/imageryprep/requirements.txt | Selects shared runtime wheels. |
| docker/imageryprep/Dockerfile | Requires runtime verification and hardening. |
| docker/docker-compose.yml | Configures local runtime protections. |
| api/titilerfuncapi/tests/test_gdal_runtime.py | Tests TiTiler runtime initialization. |
| api/titilerfuncapi/requirements.txt | Pins runtime and Rasterio wheels. |
| api/titilerfuncapi/gdal_runtime.py | Defines TiTiler startup verification. |
| api/titilerfuncapi/app/__init__.py | Verifies runtime before binding imports. |
| api/hastefuncqueues/requirements.txt | Selects shared runtime bindings. |
| api/hastefuncqueues/Dockerfile | Enables required runtime protections. |
| api/hastefuncapi/requirements.txt | Selects shared runtime bindings. |
| api/hastefuncapi/Dockerfile | Enables required runtime protections. |
| .grype.yaml | Exempts the patched zlib vulnerability. |
| .github/workflows/titiler-tests.yml | Separates route tests from runtime-wheel validation. |
| .github/workflows/deploy-apps.yml | Requires matching, attested deployment packages. |
| .github/workflows/dependency-validation.yml | Updates consumer dependency validation. |
| .github/scripts/tests/test_size_limit_workflows.py | Checks compiled hardening settings. |
| .github/scripts/tests/test_publish_gdal_runtime.py | Tests bundle validation and immutable publication. |
| .github/scripts/tests/test_prepare_training_runtime_env.py | Tests local runtime-wheel substitution. |
| .github/scripts/tests/test_gdal_hardening_guard.py | Tests configuration guard behavior. |
| .github/scripts/tests/test_filter_dependency_requirements.py | Tests dependency filtering. |
| .github/scripts/tests/test_deploy_network_access.py | Tests package deployment and enforcement restoration. |
| .github/scripts/tests/test_build_function_package.py | Tests package requirements and hash locks. |
| .github/scripts/prepare_training_runtime_env.py | Substitutes locally built training wheels. |
| .github/scripts/filter_dependency_requirements.py | Filters runtime-owned dependencies from public checks. |
| .github/scripts/deploy_apps.sh | Deploys verified packages and records hashes. |
| .github/scripts/check_gdal_hardening.py | Detects unsafe GDAL configuration assignments. |
| .github/scripts/check_env_drift.py | Recognizes native GDAL settings. |
| .github/scripts/build_function_package.py | Builds and verifies deployment archives. |
Review effort: Balanced
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 19, 2026 20:31 — with
GitHub Actions
Active
GitHub Actions
Active
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 21, 2026 15:34 — with
GitHub Actions
Active
GitHub Actions
Active
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 21, 2026 17:03 — with
GitHub Actions
Active
GitHub Actions
Active
rio-tiler opens an in-memory VRT in calculate_default_transform, so skipping the VRT driver made every reprojected tile fail with an unknown GDAL error. Allow VRT in the TiTiler runtime policy and drop it from GDAL_SKIP. Raw bands and Python pixel functions stay disabled, and user-supplied VRT content is still refused before GDAL opens it.
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 26, 2026 12:44 — with
GitHub Actions
Active
GitHub Actions
Active
The unpinned pandas resolved to 3.x in the Python 3.11 env, and geopandas 1.0.1 fails writing files with its string dtype (create_masks.py).
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 26, 2026 15:10 — with
GitHub Actions
Active
GitHub Actions
Active
Reprojection in rio-tiler needs the VRT driver, so the documented fallback applies. The training env pins pandas below 3. Dev rollout status updated.
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 26, 2026 19:33 — with
GitHub Actions
Active
GitHub Actions
Active
Joaquín Rivero (jQuinRivero)
marked this pull request as ready for review
October 27, 2026 12:07
Fail the training and imageryprep builds unless harden_gdal passes and every binding reports the runtime's GDAL release.
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 30, 2026 16:01 — with
GitHub Actions
Active
GitHub Actions
Active
…eywords ACR's dependency scanner reads a continuation line starting with 'from' as a FROM instruction and fails the build.
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
October 30, 2026 17:11 — with
GitHub Actions
Active
GitHub Actions
Active
The notices implied auditwheel adds system libraries such as the C++ runtime. The built runtime wheel bundles only the nine components listed in the table and the binding wheels bundle none, so say that and keep the future-build caveat.
Joaquín Rivero (jQuinRivero)
deployed
to
gdal-runtime-build
November 2, 2026 12:10 — with
GitHub Actions
Active
GitHub Actions
Active
Marcelo (madpsys)
approved these changes
Nov 2, 2026
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Implement the GDAL 3.13.3 shared runtime across HASTE consumers. The runtime is built from pinned, checksum-verified sources with rebuilt Python bindings, a build manifest, SBOM/notices and a fail-closed runtime check. Consumer pins, API/queue/TiTiler build-once packages, imageryprep/training integration, configuration guards and deployment verification are included. TiTiler startup applies the GDAL policy before importing bindings and preserves the upstream host allowlist and tile-only route restrictions.
Related work item: AB#577603.
Fixes #
Type of change
Checklist
cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint)Testing
python -m unittest discover -s native/gdal-runtime/tests -v— passed (47 tests).python -m unittest discover -s .github/scripts/tests -v— passed (79 tests).osgeois unavailable locally).az bicep build --file infra/main.bicep --outfile infra/main.json— passed; the compiler reports two existing hardcoded-environment-URL warnings ininfra/modules/apimApis.bicep.python .github/scripts/check_gdal_hardening.py— passed (106 files scanned).pre-commitblack, isort, flake8 and detect-secrets hooks — passed for all changed files.runtime.configure(policy)first. That fix is included. Upstream CI for this exact PR head is pending.hatch run test:pytest, UI lint, native binary build, target-image package smokes and full end-to-end workflow tests have not completed on this exact head.Additional context
mainand preserves the existing TiTiler URL/route restrictions andTITILER_ALLOWED_HOSTSsetting.