Skip to content

Fix Overture release id parsing and pin affine<3 (main fails since August) - #505

Closed
Zhonghao Zhang (Zhonghao1995) wants to merge 2 commits into
ImperialCollegeLondon:mainfrom
Zhonghao1995:fix/overture-release-id
Closed

Zhonghao Zhang (Zhonghao1995) wants to merge 2 commits into
ImperialCollegeLondon:mainfrom
Zhonghao1995:fix/overture-release-id

Conversation

@Zhonghao1995

@Zhonghao1995 Zhonghao Zhang (Zhonghao1995) commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #502 and #503.

Problem observed

On a fresh pip install -e . of main (f0c18be) on 2026-09-04, macOS 26.6.2, Python 3.11.15, every run failed twice over:

  1. The buildings download:
    FileNotFoundError: overturemaps-us-west-2/release/https:/stac.overturemaps.org/2026-08-19.0/theme=buildings/type=building/
    
    _get_latest_s3_url takes Path(href).parent of the newest child link in the Overture STAC catalog; those hrefs are absolute URLs now, so the release id came out as https:/stac.overturemaps.org/2026-08-19.0 and was then cached in .cache/overture_release.json for 72 hours.
  2. Once past that, clip_to_catchments:
    numba.core.errors.TypingError: ... Cannot determine Numba type of <class 'affine.Affine'>
    
    affine 3.0 (released 2026-08-08) turned Affine from a namedtuple into a plain class, and the @njit function pyflwdir.dem.slope that geospatial_utilities calls with the raster transform can no longer type it. pyflwdir 0.5.12 has not adapted yet.

Both come from upstream changes in August, so main currently fails the notebook runs in CI as well; the last green run on main predates them. The two fixes are in one PR because the notebooks need both to get through: with only one of them CI fails on the other error (that is what #505 and #506 showed separately).

Changes

  • Extract the Overture release id with a regex (\d{4}-\d{2}-\d{2}\.\d+) from the href instead of taking the parent path; the catalog ordering logic is unchanged. A cached entry whose release is not a well-formed id is ignored, so working directories that already hold the malformed cache recover on their own. Two mocked tests in tests/test_prepare_data.py, no network access.
  • Pin affine<3 in pyproject.toml. The alternative is to pass a plain tuple to pyflwdir.dem.slope in geospatial_utilities (the numba kernel only indexes transform[0], [4] and [5]); happy to switch to that if you prefer a code change over a pin.

Checks

pytest tests/test_prepare_data.py -k s3_url (2 passed), the repository's pre-commit hooks, and with both changes the documentation's Andorra bounding box builds end to end again with affine 2.4.0.

@Zhonghao1995 Zhonghao Zhang (Zhonghao1995) changed the title Fix Overture release id parsing for absolute STAC hrefs Fix Overture release id parsing and pin affine<3 (main fails since August) Sep 12, 2026
@Zhonghao1995

Copy link
Copy Markdown
Collaborator Author

Update: this PR now also carries the affine<3 pin that was #506, because the two problems block each other in CI. With only the Overture fix, the notebook runs fail on the numba typing error (#503); with only the pin, they fail on the Overture release id (#502). Both come from upstream changes in August (affine 3.0.0 was released on 2026-08-08, and the Overture STAC catalog switched to absolute hrefs with the 2026-08-19.0 release), so main itself currently fails the notebook runs; the last green run on main predates them. I closed #506 in favour of this one.

@Zhonghao1995

Copy link
Copy Markdown
Collaborator Author

Moved to #507: the same two commits, pushed as a branch in this repository now that I have write access, so CI gets the Codecov token and does not wait for approval. Closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Buildings download fails: Overture release id is derived from an absolute STAC href

1 participant