Skip to content

Add seasonal FBFM40 support to griddle via LFPS #506

Description

@lindsaywiard

Griddle processes one Cloud Task per invocation, so a multi-step LFPS job (submit → poll → poll → ... → download, up to 2 hours) needs to submit, then re-enqueue itself to check again later rather than blocking.

This splits into two files: the orchestration (submit/poll/timeout/download) lives in a new griddle/handlers/lfps.py, and griddle/handlers/landfire.py grows a way to fetch a raster from a local path instead of only a gs:// COG, so LFPS output can flow through the exact same alignment/extent_buffer_cells/nodata pipeline as everything else in that file, unchanged.

Tasks

  • New griddle/handlers/lfps.py. Given a grid's Firestore doc, either submit a new LFPS job or poll an existing one:
    • No job yet → submit_job(), write job_id/submitted_at onto the doc (lfps_job: {...}), re-enqueue a delayed Cloud Task for the same grid_id.
    • In progress → poll_status().
    • Still running, under timeout → re-enqueue again.
    • Over timeout → raise ProcessingError (matching _validate_dem_has_data()'s pattern).
    • FailedProcessingError with LFPS's message.
    • Succeededdownload() and continue into the fetch path below.
      *Timeout: 20 min. Not LFPS's own ~2hr ceiling — our jobs are small and consistently fast (12-30s), so anything still running that long is more likely stuck than genuinely slow, and we'd rather fail fast and let the user retry than poll a dead job for two hours.
    • Re-enqueue delay/backoff: TBD, LFPS jobs typically finish in 12s–7min.
  • Griddle processes one Cloud Task per invocation, so a step that isn't done yet can't just wait — it re-enqueues itself and stops. That "stop without marking the grid complete or failed" signal is a new ProcessingDeferred exception (lib/errors.py), caught in main.py alongside the existing CancelledException/ProcessingError handling. The re-enqueue itself needs a synchronous Cloud-Tasks helper griddle doesn't have yet (api/tasks.py's is async, API-only) — new griddle/griddle/tasks.py.
  • _fetch_landfire_raster (in landfire.py) takes the raster location as a parameter instead of always building gs://{RASTERS_BUCKET}/LF{version}_{product}_CONUS.tif internally — RasterConnection opens a local path the same way it opens a gs:// one, so no separate fetch function or branch is needed there. fetch_fbfm40 gets a url= override so lfps.py can hand it an unzipped local file directly, reusing the existing remove_non_burnable/nodata/_to_dataset handling as-is.
  • Wire the new path into dispatch.py: source.product == "fbfm40" with an optional source.season field routes to lfps.py instead of the existing COG fetch; no season → unchanged behavior.
  • Nodata consolidation is already shared (_fetch_landfire_raster doesn't care where the path came from), so this falls out for free rather than needing separate handling.

Tests

  • Unit tests for lfps.py's orchestration, mocked lib.landfire_lfps client + mocked Firestore doc, no live calls:
    • no job → submits;
    • in progress/under timeout → re-enqueues;
    • in progress/over timeout → fails, no re-enqueue;
    • succeeded → downloads and continues;
    • failed → fails with LFPS's message.
  • Unit test for the unzip/locate-the-tif step in isolation (given zip bytes, finds the one GeoTIFF among the .tfw/.aux.xml/VAT sidecar files; errors cleanly if none found).
  • No new integration test needed for the fetch/align step itself — fetch_fbfm40's existing COG-path integration test already exercises that exact code, since the raster-fetch pipeline no longer branches on source at all once it has a path. The polling loop stays unit-test-only for the same reason as before: it doesn't resolve in a single process_grid_request call, so a griddle_runner-style test would either assert nothing or race a live multi-minute LFPS job.

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions