Repository navigation
Conversation
For zarr files, set up a .complete marker file to check when data is fully written. This replaces relying on zarr.json -- that doesn't work because zarr.json is the first to be written. This might happen from a failed or aborted calculation, or because multiple processes are trying to write the same file (e.g. running 2 countries which share a tile) A partially-written zarr might be not cause any warning signs later, since it might be techncially valid in spite of missing data. Now an empty .complete marker is written beside the zarr after its last chunk, and the entry counts as cached only once the marker exists. A zarr found without its marker is a write that was killed part-way or is still running: serialize raises FileExistsError and leaves it for someone to resolve by hand, usually by deleting the zarr, rather than writing over it and risking a race with a live writer. Entries written before this change have no marker, so each raises once until deleted.
An autouse fixture in jdluc/__tests__/conftest.py stubs Config.from_dot_env for every test not marked integration. Each field gets its type's empty value, with two dask workers, so a unit test that reaches for a storage root fails instead of touching a bucket. Integration tests still read the real .env. This replaces the fixture storage_test carried for itself, so the next unit test that writes a zarr needs no copy of it, and a new Config field needs no edit to the tests.
An optional `orchestration` package with one subcommand per workflow task (plan, ingest-tile, compute, trace), and an example Argo WorkflowTemplate that runs each tile's ingest -> compute as a DAG of its own, so a failed tile stops only itself. jdluc runs unmodified in the pods; every cache-key argument passes straight through, and an import-linter contract keeps jdluc from importing orchestration. trace writes the emissions-factor table to a new EXPORT_ROOT, through a small `export.write_table` utility. The package is built to be wrapped rather than edited: `build_parser` and `run` let a wrapper add subcommands, `plan.get_task_to_items` returns the lists for it to add to before `write_task_to_items`, and the template's tasks can be called by templateRef. See docs/orchestration.md. Also adds the Dockerfile the workflow's image is built from.
get_name no longer mentions mosaics, get_task_to_items describes the flat lists without a flat workflow to point at, and trace's --help is a whole sentence.
338c953 to
c6e3b0a
Compare
| # EXPORT_ROOT is where terminal deliverables land (the emissions-factor table): not a cache | ||
| # the pipeline reads back. Any fsspec URI, like the roots below. | ||
| # EXPORT_ROOT = '/absolute/path/to/export' | ||
| EXPORT_ROOT = 'gs://cornerstone-emissions-us-central1' |
There was a problem hiding this comment.
EXPORT_ROOT is the active line here and it points at the real bucket. Anyone copying this file and running trace would write straight into it. Could we make the placeholder the active line and keep the bucket commented out?
| resources: { requests: { storage: 150Gi } } | ||
|
|
||
| # harmonize -> emit -> attribute for one tile, from the warm ingest cache, without keys | ||
| - name: compute |
There was a problem hiding this comment.
No retryStrategy on any template, so a reclaimed spot node means a manual argo retry. Maybe retryStrategy: {limit: 2, retryPolicy: OnError} on ingest-tile and compute? The cache makes retries cheap.
| RUN uv sync --frozen --no-dev | ||
|
|
||
| # .git in its own last layer, so a commit that changes no source rebuilds only this layer. | ||
| COPY .git .git |
There was a problem hiding this comment.
This copies the whole .git into the image. If it's built from a clone with a token in the remote URL, the token ends up in the image (get_git_remote_url reads that URL). Could we pass the SHA and remote as build args instead, or strip the credentials? Also, the container runs as root, and the safe.directory line above suggests a non-root user was intended.
deliverable_name becomes product_name in write_table and get_table_output_uri, matching the cornerstone-product-name tag that get_metadata already writes from it.
orchestration: run the pipeline on Argo, one tile per pod (cornerstone-data/luc#19): write_table names its product # Conflicts: # jdluc/export.py
Brings in cornerstone-data/luc#19's write_table product_name rename.
|
@danohu, One more thing building on @tankoyarik comment about .git/credentials in the Dockerfile: export.get_metadata() stamps utils.get_git_remote_url() into cornerstone-remote-url on every exported table's attrs, and the new test confirms attrs survives the to_parquet/read_parquet round-trip. So if the image is ever built from a clone whose remote URL carries a token, that credential doesn't just sit in the image. It propagates into the metadata of every emissions-factors parquet handed downstream. Worth stripping credentials from the URL before it's used as provenance, not just before it's baked into the image. |
d033289 to
f9daffd
Compare
Stacked on #16 (zarr
.completemarker): the docs here rely on a killed pod leaving no zarr that passes for a result. Review #16 first; this PR's diff is only the orchestration work.What
An optional layer for running the pipeline on Kubernetes with Argo Workflows, one 10° tile per pod, so that a country or a global backfill takes hours rather than days.
jdlucruns unmodified inside the pods; Argo takes over only sequencing, fan-out, retries and visibility.docs/orchestration.mdhas the design and setup.orchestrationpackage, one subcommand per workflow task:plan,ingest-tile,compute,tracejdluc/export.py+trace.export_workflow: write the emissions-factor table, with provenance, to a newEXPORT_ROOTorchestration/workflows/cornerstone.example.yaml: an ArgoWorkflowTemplate, one DAG per tile, with cloud-neutral placeholdersDockerfile,.dockerignorejdlucnever importsorchestration), and mypy overorchestrationDesign notes
orchestrationmay not change a result. Every cache-key argument goes straight through tojdluc, withjdluc's own defaults.skip_ingest=Truefrom a warmINGEST_ROOT. That needs the list of datasets a run reads, whichorchestration/plan.pykeeps by hand.traceruns only once every tile has succeeded.argo retryresumes from the cache.__main__.build_parser()returns the parser and its subparsers,plan.get_task_to_items()returns the plan lists without writing them, and the template's tasks can be called withtemplateRef.Heads-up
EXPORT_ROOTis a new.envkey, andConfig.from_dot_env()requires every key, so an existing.envneeds it added (see.env.example).Verification
pytestpasses, including 18 new tests inorchestrationpre-commit run --all-filespasses (ruff, import-linter, mypy, mdformat)argo lint --offlineon the example templatedocker build