refactor(core): add shared atomic_write_text helper, adopt in querygen planning-summary export#237
Conversation
saschagobel
left a comment
There was a problem hiding this comment.
Thanks, this is a helpful refactor and tests cover the core behavior well. I left two non-blocking comments: one on simplifying the dense atomic_write_text docstring, and one on avoiding a one-off local JSON helper / applying atomic JSON writes consistently.
There was a problem hiding this comment.
I’d avoid adding a private _atomic_write_json() helper in querygen/export.py. As currently used, it adds an extra layer for one call site; if this stays local, I think the write is clearer inline in export_planning_summary().
More importantly, this creates an inconsistency in the same module: export_planning_summary() uses atomic JSON writing, while export_queries() still writes meta_path directly with Path.write_text(). Since eval and later querygen artifacts will likely need the same behavior, I’d prefer to either:
- inline the
atomic_write_text()block here and keep the PR narrowly scoped, or - promote this to a shared
atomic_write_json()helper incore/atomic_io.pyand use it for bothexport_planning_summary()and the query metadata JSON.
My preference is option 2, with a small helper that wraps json.dump(..., indent=2) plus a trailing newline. That avoids a local one-off abstraction and gives the upcoming eval/querygen artifact writes one canonical JSON path.
…n export Extracts the tmpfile+rename atomic-write idiom into core/atomic_io.py and routes querygen's planning-summary export through it (indented JSON). Annotation (record_builder/export_runner) should adopt atomic_write_text later; left out here to keep this PR's dependency surface to querygen only.
7e1155f to
e973f6a
Compare
Stack (6 open PRs): #237 → #238 → #239 → #240 → #241 → #242
Chain goal: Add per-batch checkpoint/resume to querygen so a re-run with the same
run_idskips already-completed work. Nests resumable state under<run_dir>/checkpoints/, writes a frozen Stage 1 result (planning + dedup) plus per-batch Stage 1 and Stage 2 artifacts, and orchestrates resume behind a fail-fast config-drift gate (--forceoverrides) so checkpoints are never silently reused under changed settings.This PR (1/6): Extracts the tmpfile + atomic-rename idiom into a shared
core/atomic_io.pyhelper and adopts it for querygen's planning-summary write — the atomic-write primitive every later checkpoint artifact (#239–242) writes through.What
core/atomic_io.py—atomic_write_textcontext manager: PID+uuid-uniquified tempfile,Path.replacefor the atomic rename, no fsync (matches house style).export.pyadopts it forexport_planning_summary(indented JSON for human-scannable persisted artifacts).record_builder/export_runnershould adoptatomic_write_textin a follow-up — kept out here to keep this PR's dependency surface to querygen only.Why
Extracts the tmpfile + atomic-rename idiom (previously duplicated inline in two annotation sites —
record_builder.write_partition_manifestandexport_runner.write_export_csv) into one shared core helper, then adopts it for querygen's planning-summary write. Reaches the rule-of-3 with the upcoming querygen artifact writes in #239–242.Test plan
tests/unit/core/test_atomic_io.py— round-trip + temp-cleanup-on-exception.