Port to geotessera 0.10.1 and remove workarounds - #4
Merged
Conversation
Reference rasters such as forest inventories often mark missing data with large sentinel values (for example 32766 and 32767). These were stripped from the aligned raster only after resampling, by exact comparison. With bilinear resampling a sentinel first blends with its real neighbours, producing large in-between values that no longer equal the sentinel and so pass through as apparently valid regression targets. align_raster_to_grid now converts the raster's declared nodata and any extra sentinel values to NaN before resampling, so they are excluded from interpolation entirely. Only the window of the source raster that covers the destination grid is read, keeping memory use bounded by the destination size rather than the source raster.
run_learning_curve works in percentages of the labelled data, but the evaluate() convenience wrapper still passed absolute training sizes straight through. A size of 10000 was therefore read as 10000 percent, so every requested size above roughly 100 collapsed to the same 80% training split and the learning curve's x-axis was meaningless. Results.summary() also read a 'size' key that progress events no longer carry, and raised KeyError. evaluate() now converts the requested sizes into percentages of the data, capped at 80% and deduplicated, and Results.summary() reports the actual number of training pixels used at each step.
The spatial MLP models train on neighbourhood features, and no such features exist for a fixed test set (one built from drawn train/test regions, or from a different test year). Instead of refusing, the learning curve quietly fell back to a random split of the spatial models' own training pixels -- so their scores appeared alongside the honestly held-out scores of the other models while measuring something easier, and looked optimistic as a result. These models are now skipped for such runs, with a status message saying so, and the server no longer extracts their neighbourhood features when they cannot be used. An unreachable and broken fallback branch (it trained on neighbourhood features but predicted plain pixel vectors, which always failed and scored 0.0) is removed.
Embeddings are produced on each tile's native UTM grid, and the geotessera guidance is to classify on that grid and reproject only the result. create_map's NPY fallback instead fetched each chunk already reprojected to plain latitude/longitude, resampling every 128-dimensional embedding vector before the model saw it, and producing a map on a different grid than the zarr path. The NPY fallback now reads one tile at a time on its native grid, crops it to the map area, and predicts there. The per-block prediction rasters -- not the embeddings -- are then reprojected onto a common coordinate system, with nearest-neighbour resampling so no class IDs or values are invented, before being merged into the output GeoTIFF. This also fixes the zarr path for map areas that span a UTM zone boundary: its chunks arrive in different zones' coordinate systems, and merging them previously failed with "CRS mismatch with source".
The map GeoTIFF was written with compress="lz4", which is not a compression method GeoTIFF supports. GDAL ignored the unknown value without raising an error, so every generated map was silently written uncompressed. Use DEFLATE, which every GeoTIFF reader understands.
load_embeddings_for_shapefile_vq splits the shapefile's bounding box into chunks using degree arithmetic, but a shapefile that was not already in lon/lat was first reprojected into target_crs. With a projected target CRS the chunk maths then ran over metre coordinates, producing nonsense chunk bounds and, in practice, an enormous number of tiny chunks. The shapefile is now always brought to lon/lat for chunking, and the polygons are reprojected onto each returned mosaic's own CRS just before rasterization. The docstring also now notes that mosaics come back resampled onto target_crs rather than the embeddings' native UTM grid, so accuracy measured through this loader includes that resampling as well as the VQ reconstruction.
The old floor of 0.9.0 admits releases that still download from the retired tessera-embeddings hosting, which is being shut down. geotessera 0.10 moved all data hosting to the Source Cooperative, so older releases will simply stop working. Note that geotessera has required Python 3.12 for some time, so installations using any of the extras that pull it in already need 3.12 even though the base package still supports 3.10.
The external tessera-zarr-utils package existed to work around two upstream problems -- a UTM-zone-boundary bug in zarr region reads and an incomplete rollout of years other than 2024 -- and its pinned release disabled zarr outright, so the compute server has been using NPY tiles for everything. geotessera 0.10.1 fixes both: its GeoTesseraZarr interface reads regions on their native UTM grids and serves every published year. The compute server now uses GeoTesseraZarr directly, with two small helpers alongside its existing cached GeoTessera handle: _get_zarr() opens the store once per process, caches chunk reads on disk next to the NPY tile cache, and remembers a failed open so callers fall back to NPY tiles as before; _probe_zarr_coverage() uses geotessera's single-pixel probe to accept only genuinely valid embeddings for the requested year and region. This re-enables the zarr fast path and removes the git-pinned dependency.
Summarise the reprojection and correctness fixes, the re-enabled zarr fast path via geotessera's own interface, and the new geotessera floor under an Unreleased heading.
run_learning_curve now removes spatial models from a run when the test set is a separate region or year, so their names no longer appear in progress events. The command-line printer still indexed every requested model unconditionally and crashed with KeyError -- after the embeddings had already been downloaded. Models absent from an event are now simply not printed; the engine already logs why they were skipped.
A destination grid lying just outside the raster, but within the sentinel path's two-pixel interpolation margin, passed the window-intersection check with a fraction of a pixel of overlap that then rounded down to a zero-width read. Reprojecting from an empty array crashes inside GDAL, where the same call without sentinel values quietly returns an all-NaN grid. A window that rounds to nothing now returns all-NaN like any other non-overlapping case.
A shapefile missing its projection file has no CRS, and the loader deliberately treats that as already lon/lat. The recent change that reprojects polygons onto each mosaic's coordinate system broke this: geopandas refuses to reproject naive geometries, so such a shapefile crashed the whole load. Polygons without a CRS are now rasterized as-is, restoring the previous behaviour.
The zarr store serves a bounding box that straddles a UTM zone edge from the centre zone alone, silently clipping at the edge. Map chunks started at the user's arbitrary western edge and stepped by 0.1 degrees, so a chunk could straddle a zone boundary and quietly lose the strip on its far side -- a nodata seam in the merged map, in exactly the multi-zone areas the native-grid rework is meant to handle. Chunk edges now also break at 6-degree multiples, so every request stays within a single zone.
Maps are now georeferenced in the embeddings' native UTM CRS rather than always EPSG:4326, and the GeoTIFF's own metadata was the only place that said so. The map_ready event now carries a crs field so consumers that assumed lon/lat can detect the change, and the changelog calls it out.
The zarr store handle cached its chunk reads on disk with no size limit, so a long-lived compute server generating maps over several large regions and years would fill the disk -- taking the co-located NPY and result caches down with it. The cache is now capped at 20 GB, using the eviction support the store interface already provides.
In the map endpoint's NPY fallback, the registry query that lists a map area's tiles ran outside any error handling, unlike every fetch and prediction step around it. An exception there escaped the response generator and truncated the event stream mid-response, leaving the frontend waiting with no error to show. A failed listing now yields an error event for that map area and moves on to the next one.
The list of models that train on neighbourhood features was spelled out as a literal tuple in six places across three modules, including every site that decides when such models must be skipped. Adding a variant or renaming one and missing a site would silently reintroduce the optimistic-scores bug the fixed-test-set skip exists to prevent. The names now live in a single SPATIAL_MODELS constant in classify.py, which every site imports.
sk818
approved these changes
Aug 29, 2026
sk818
added a commit
that referenced
this pull request
Aug 30, 2026
Release for PR #4. Highlights: - tessera-zarr-utils dependency removed; compute server uses GeoTesseraZarr directly with a 20 GiB bounded disk cache, zarr fast path re-enabled, geotessera floor raised to >=0.10.1. - Maps predicted on native UTM grids; only prediction rasters are reprojected, at merge time. Output GeoTIFF is georeferenced in the native UTM CRS (reported in the map_ready event), DEFLATE-compressed. - Sentinel nodata (e.g. 32766/32767) excluded from raster resampling. - Spatial MLP models skipped (not silently random-split) when the test set is a separate region or year. - evaluate() honours requested training sizes; Results.summary() fixed. - VQ loader chunks shapefile bounds in degrees regardless of CRS. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LxpfB5ug78vFswQ5tkW5Qw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from a code review (only limited testing), but the upgrade to geotessera>=0.10 is important so that the right source is used for the embeddings fetch: