Skip to content

Fix Overture release id parsing and pin affine<3 (main fails since August) - #507

Open
Zhonghao Zhang (Zhonghao1995) wants to merge 2 commits into
mainfrom
fix/overture-release-id
Open

Zhonghao Zhang (Zhonghao1995) wants to merge 2 commits into
mainfrom
fix/overture-release-id

Conversation

@Zhonghao1995

Copy link
Copy Markdown
Collaborator

Replaces #505: the same two commits, now on a branch in this repository so that CI has access to the Codecov token and does not need approval for each run.

Closes #502 and #503.

Problem observed

On a fresh pip install -e . of main (f0c18be) on 2026-09-04, macOS 26.6.2, Python 3.11.15, every run failed twice over:

  1. The buildings download:
    FileNotFoundError: overturemaps-us-west-2/release/https:/stac.overturemaps.org/2026-08-19.0/theme=buildings/type=building/
    
    _get_latest_s3_url takes Path(href).parent of the newest child link in the Overture STAC catalog; those hrefs are absolute URLs now, so the release id came out as https:/stac.overturemaps.org/2026-08-19.0 and was then cached in .cache/overture_release.json for 72 hours.
  2. Once past that, clip_to_catchments:
    numba.core.errors.TypingError: ... Cannot determine Numba type of <class 'affine.Affine'>
    
    affine 3.0 (released 2026-08-08) turned Affine from a namedtuple into a plain class, and the @njit function pyflwdir.dem.slope that geospatial_utilities calls with the raster transform can no longer type it. pyflwdir 0.5.12 has not adapted yet.

Both come from upstream changes in August, so main currently fails the notebook runs in CI as well; the last green run on main predates them. The two fixes are in one PR because the notebooks need both to get through: with only one of them CI fails on the other error (that is what #505 and #506 showed separately).

Changes

  • Extract the Overture release id with a regex (\d{4}-\d{2}-\d{2}\.\d+) from the href instead of taking the parent path; the catalog ordering logic is unchanged. A cached entry whose release is not a well-formed id is ignored, so working directories that already hold the malformed cache recover on their own. Two mocked tests in tests/test_prepare_data.py, no network access.
  • Pin affine<3 in pyproject.toml. The alternative is to pass a plain tuple to pyflwdir.dem.slope in geospatial_utilities (the numba kernel only indexes transform[0], [4] and [5]); happy to switch to that if you prefer a code change over a pin.

Checks

pytest tests/test_prepare_data.py -k s3_url (2 passed), the repository's pre-commit hooks, and with both changes the documentation's Andorra bounding box builds end to end again with affine 2.4.0.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.28%. Comparing base (f0c18be) to head (658e365).

Files with missing lines Patch % Lines
src/swmmanywhere/prepare_data.py 75.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
- Coverage   87.34%   87.28%   -0.06%     
==========================================
  Files          23       23              
  Lines        2355     2360       +5     
  Branches      302      303       +1     
==========================================
+ Hits         2057     2060       +3     
- Misses        206      207       +1     
- Partials       92       93       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@barneydobson barneydobson left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy with that Zhonghao Zhang (@Zhonghao1995) go ahead and merge! and thanks for your contributions ✨

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Buildings download fails: Overture release id is derived from an absolute STAC href

3 participants