feat(querygen): add realization-batch checkpoint artifact#241
Conversation
…n planning-summary export (#237) **Stack (6 open PRs):** #237 → #238 → #239 → #240 → #241 → #242 **Chain goal:** Add per-batch checkpoint/resume to querygen so a re-run with the same `run_id` skips already-completed work. Nests resumable state under `<run_dir>/checkpoints/`, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (`--force` overrides) so checkpoints are never silently reused under changed settings. **This PR (1/6):** Extracts the tmpfile + atomic-rename idiom into a shared `core/atomic_io.py` helper and adopts it for querygen's planning-summary write — the atomic-write primitive every later checkpoint artifact (#239–242) writes through. --- ## What - `core/atomic_io.py` — `atomic_write_text` context manager: PID+uuid-uniquified tempfile, `Path.replace` for the atomic rename, no fsync (matches house style). - Querygen `export.py` adopts it for `export_planning_summary` (indented JSON for human-scannable persisted artifacts). - **Note:** annotation `record_builder`/`export_runner` should adopt `atomic_write_text` in a follow-up — kept out here to keep this PR's dependency surface to querygen only. ## Why Extracts the tmpfile + atomic-rename idiom (previously duplicated inline in two annotation sites — `record_builder.write_partition_manifest` and `export_runner.write_export_csv`) into one shared core helper, then adopts it for querygen's planning-summary write. Reaches the rule-of-3 with the upcoming querygen artifact writes in #239–242. ## Test plan - [x] `tests/unit/core/test_atomic_io.py` — round-trip + temp-cleanup-on-exception.
saschagobel
left a comment
There was a problem hiding this comment.
The realization-batch artifact boundary makes sense to me for crash recovery. Since Stage 2 batches do not carry chained state, checkpointing them independently is a good fit, and binding each artifact to its candidate_ids chunk is the right safeguard against accidentally applying a checkpoint to the wrong blueprint slice.
My concern is mainly with how this fits into the wider stack. I would avoid carrying the per-stage reuse model into this artifact contract. For v1, realization checkpoints should be validated against the same strict same-run/same-effective-settings compatibility model as the rest of the checkpoints. If settings that affect the run differ, I’d rather fail clearly and ask the user to start a new run or delete/recreate the existing one, rather than enabling partial reuse/recompute semantics through stage-specific drift handling.
Stack (6 open PRs): #236 → #237 → #238 → #239 → #240 → #241 → #242
Chain goal: Add per-batch checkpoint/resume to querygen so a re-run with the same
run_idskips already-completed work. Nests resumable state under<run_dir>/checkpoints/, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (--forceoverrides) so checkpoints are never silently reused under changed settings.This PR (5/6): Third and final vertical artifact slice. Adds per-batch checkpointing for Stage 2 realization. Stage 2 batches are independent (no chained state), so each batch resumes on its own merits — the
candidate_idsheader field binds each checkpoint to its exact frozen-result chunk.What
RealizationBatchArtifactschema withextra="forbid"+1:1 candidate_ids/queriesmodel_validator.assemble_realization_batch_artifact+export_realization_batch_artifact(atomic).read_realization_batch_artifact— thin typed wrapper over the feat(querygen): add planning-batch checkpoint artifact and read engine #239 engine.Why
Stage 2 has no chained state between batches, so each realization batch is checkpointed independently — a rerun replays only the batches that didn't complete. Binding the checkpoint to its
candidate_idschunk means a re-chunked frozen result can't silently reuse a stale realization.Test plan
tests/unit/core/querygen/test_realization_batches.py— round-trip, header-mismatch per field, version-mismatch, malformed JSON, length-validator.