feat(querygen): add frozen selected-blueprints checkpoint artifact#240
Conversation
…n planning-summary export (#237) **Stack (6 open PRs):** #237 → #238 → #239 → #240 → #241 → #242 **Chain goal:** Add per-batch checkpoint/resume to querygen so a re-run with the same `run_id` skips already-completed work. Nests resumable state under `<run_dir>/checkpoints/`, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (`--force` overrides) so checkpoints are never silently reused under changed settings. **This PR (1/6):** Extracts the tmpfile + atomic-rename idiom into a shared `core/atomic_io.py` helper and adopts it for querygen's planning-summary write — the atomic-write primitive every later checkpoint artifact (#239–242) writes through. --- ## What - `core/atomic_io.py` — `atomic_write_text` context manager: PID+uuid-uniquified tempfile, `Path.replace` for the atomic rename, no fsync (matches house style). - Querygen `export.py` adopts it for `export_planning_summary` (indented JSON for human-scannable persisted artifacts). - **Note:** annotation `record_builder`/`export_runner` should adopt `atomic_write_text` in a follow-up — kept out here to keep this PR's dependency surface to querygen only. ## Why Extracts the tmpfile + atomic-rename idiom (previously duplicated inline in two annotation sites — `record_builder.write_partition_manifest` and `export_runner.write_export_csv`) into one shared core helper, then adopts it for querygen's planning-summary write. Reaches the rule-of-3 with the upcoming querygen artifact writes in #239–242. ## Test plan - [x] `tests/unit/core/test_atomic_io.py` — round-trip + temp-cleanup-on-exception.
saschagobel
left a comment
There was a problem hiding this comment.
The selected-blueprints artifact makes sense to me as the Stage 1 handoff for crash recovery. Persisting the post-filter/post-dedup blueprint set gives Stage 2 a stable input and avoids having to repeat completed Stage 1 work after an abort.
My main concern here is placement/scope rather than the artifact itself. If SelectedBlueprintsArtifact is an internal resumability/checkpoint artifact, I’d prefer not to add it to the final querygen output schema module. Conceptually it is different from user-facing outputs like SyntheticQueryRow, SyntheticQueriesMeta, or the planning-summary artifact. A dedicated checkpoint/internal schema module, e.g. would make the boundary clearer and avoid expanding the final output contract with resumability internals.
I’d also keep this aligned with the narrower resumability scope: validate it against the same strict same-run compatibility model, rather than using it as part of broader reuse semantics under changed settings.
Stack (6 open PRs): #236 → #237 → #238 → #239 → #240 → #241 → #242
Chain goal: Add per-batch checkpoint/resume to querygen so a re-run with the same
run_idskips already-completed work. Nests resumable state under<run_dir>/checkpoints/, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (--forceoverrides) so checkpoints are never silently reused under changed settings.This PR (4/6): Second of three vertical artifact slices. Adds the frozen Stage 1 result — the immutable handoff written once after dedup. Its presence is the "Stage 1 done" marker that lets a rerun skip planning + dedup + the embedding-model load entirely, and lets Stage 2 chunk an immutable set.
What
SelectedBlueprintsArtifactschema — header includesnear_duplicate_tolerance(the dedup is a function of it) andenable_planning_memory(shapes the blueprints);embedding_modelrecorded for provenance but not validated.assemble_selected_blueprints_artifact+export_selected_blueprints(atomic, indented JSON).read_selected_blueprints_artifact— thin typed wrapper over the feat(querygen): add planning-batch checkpoint artifact and read engine #239 engine; documents thatembedding_modelis intentionally not part of the drift check.Why
The frozen result is the pivot of the whole resume design: writing it once after dedup means a same-
run_idrerun never has to re-plan, re-dedup, or reload the embedding model, and Stage 2 always chunks a stable, ordered set.embedding_modelstays out of the drift check because the artifact already captures the dedup output — re-validating the model would force a recompute that the frozen result exists precisely to avoid.Test plan
tests/unit/core/querygen/test_selected_blueprints.py— round-trip, header-mismatch per field, version-mismatch, malformed JSON, and theembedding_modelis-ignored-on-read invariant.