Skip to content

feat(querygen): add frozen selected-blueprints checkpoint artifact#240

Open
henrycgbaker wants to merge 1 commit into
feat/querygen-resumable/04-planning-batch-artifactfrom
feat/querygen-resumable/05-selected-blueprints-artifact
Open

feat(querygen): add frozen selected-blueprints checkpoint artifact#240
henrycgbaker wants to merge 1 commit into
feat/querygen-resumable/04-planning-batch-artifactfrom
feat/querygen-resumable/05-selected-blueprints-artifact

Conversation

@henrycgbaker

@henrycgbaker henrycgbaker commented May 29, 2026

Copy link
Copy Markdown
Collaborator

Stack (6 open PRs): #236#237#238#239#240#241#242

Chain goal: Add per-batch checkpoint/resume to querygen so a re-run with the same run_id skips already-completed work. Nests resumable state under <run_dir>/checkpoints/, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (--force overrides) so checkpoints are never silently reused under changed settings.

This PR (4/6): Second of three vertical artifact slices. Adds the frozen Stage 1 result — the immutable handoff written once after dedup. Its presence is the "Stage 1 done" marker that lets a rerun skip planning + dedup + the embedding-model load entirely, and lets Stage 2 chunk an immutable set.


What

  • SelectedBlueprintsArtifact schema — header includes near_duplicate_tolerance (the dedup is a function of it) and enable_planning_memory (shapes the blueprints); embedding_model recorded for provenance but not validated.
  • assemble_selected_blueprints_artifact + export_selected_blueprints (atomic, indented JSON).
  • read_selected_blueprints_artifact — thin typed wrapper over the feat(querygen): add planning-batch checkpoint artifact and read engine #239 engine; documents that embedding_model is intentionally not part of the drift check.

Why

The frozen result is the pivot of the whole resume design: writing it once after dedup means a same-run_id rerun never has to re-plan, re-dedup, or reload the embedding model, and Stage 2 always chunks a stable, ordered set. embedding_model stays out of the drift check because the artifact already captures the dedup output — re-validating the model would force a recompute that the frozen result exists precisely to avoid.

Test plan

  • tests/unit/core/querygen/test_selected_blueprints.py — round-trip, header-mismatch per field, version-mismatch, malformed JSON, and the embedding_model is-ignored-on-read invariant.

@github-actions github-actions Bot added feature Adds or expands user-facing functionality size: M 200-499 LOC labels May 29, 2026
@henrycgbaker henrycgbaker added contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts querygen Changes affecting the synthetic query generation tool and removed contracts Changes schemas, type contracts, or validation logic for inputs, outputs, and persisted artifacts labels May 29, 2026
saschagobel pushed a commit that referenced this pull request Jun 4, 2026
…n planning-summary export (#237)

**Stack (6 open PRs):** #237#238#239#240#241#242 

**Chain goal:** Add per-batch checkpoint/resume to querygen so a re-run
with the same `run_id` skips already-completed work. Nests resumable
state under `<run_dir>/checkpoints/`, writes a frozen Stage 1 result
(planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and
orchestrates resume behind a fail-fast config-drift gate (`--force`
overrides) so checkpoints are never silently reused under changed
settings.

**This PR (1/6):** Extracts the tmpfile + atomic-rename idiom into a
shared `core/atomic_io.py` helper and adopts it for querygen's
planning-summary write — the atomic-write primitive every later
checkpoint artifact (#239–242) writes through.

---

## What
- `core/atomic_io.py` — `atomic_write_text` context manager:
PID+uuid-uniquified tempfile, `Path.replace` for the atomic rename, no
fsync (matches house style).
- Querygen `export.py` adopts it for `export_planning_summary` (indented
JSON for human-scannable persisted artifacts).
- **Note:** annotation `record_builder`/`export_runner` should adopt
`atomic_write_text` in a follow-up — kept out here to keep this PR's
dependency surface to querygen only.

## Why
Extracts the tmpfile + atomic-rename idiom (previously duplicated inline
in two annotation sites — `record_builder.write_partition_manifest` and
`export_runner.write_export_csv`) into one shared core helper, then
adopts it for querygen's planning-summary write. Reaches the rule-of-3
with the upcoming querygen artifact writes in #239–242.

## Test plan
- [x] `tests/unit/core/test_atomic_io.py` — round-trip +
temp-cleanup-on-exception.
@saschagobel saschagobel self-requested a review June 5, 2026 13:24
@saschagobel saschagobel marked this pull request as ready for review June 5, 2026 13:24

@saschagobel saschagobel left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The selected-blueprints artifact makes sense to me as the Stage 1 handoff for crash recovery. Persisting the post-filter/post-dedup blueprint set gives Stage 2 a stable input and avoids having to repeat completed Stage 1 work after an abort.

My main concern here is placement/scope rather than the artifact itself. If SelectedBlueprintsArtifact is an internal resumability/checkpoint artifact, I’d prefer not to add it to the final querygen output schema module. Conceptually it is different from user-facing outputs like SyntheticQueryRow, SyntheticQueriesMeta, or the planning-summary artifact. A dedicated checkpoint/internal schema module, e.g. would make the boundary clearer and avoid expanding the final output contract with resumability internals.

I’d also keep this aligned with the narrower resumability scope: validate it against the same strict same-run compatibility model, rather than using it as part of broader reuse semantics under changed settings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature Adds or expands user-facing functionality querygen Changes affecting the synthetic query generation tool size: M 200-499 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants