fix(bulk-submit): key entry results by their manifest output file - #466
fix(bulk-submit): key entry results by their manifest output file#466angela-helios wants to merge 1 commit into
Conversation
Line numbers restart in every output file, but bulk_entry_results' key was (tenant, submitter, submission, manifest, line_number) — so in any multi-file manifest, every file after the first collided on its first entry and failed wholesale. The worker now names the file it is ingesting (BulkProcessingOptions::file_url), the stores key results by it, and both schemas carry it in the primary key: SQLite via a v15 table rebuild (pre-migration rows keep ''), Postgres via an in-place add-column + re-key guarded on the constraint's current shape. The insert is also an upsert now: the worker re-fetches a whole file after a transient failure, and the retry must overwrite its own earlier rows instead of colliding with them. Closes #457
|
Full-scale e2e against this build: the #448 Synthea submission — 20 NDJSON files, 11,286 resources, every file starting at line 1 — now ingests completely. The status manifest reports one output per file (AllergyIntolerance 26, CarePlan 54, Claim 1,073, Condition 491, DiagnosticReport 1,045, Observation 1,377, Procedure 1,906, …) with no error outcomes, and every resource type queries back with the expected counts. Before the fix the same submission died at the first line of the second file. |
|
The |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Closes #457
Line numbers restart in every manifest output file, but
bulk_entry_results's primary key had no file discriminator — so in any multi-file manifest, only the first file ingested and every other died wholesale on a UNIQUE violation at its first entry (single-file smoke tests never trip it; the #448 Synthea submission with 20 files did, instantly).BulkProcessingOptions::file_url, threaded per-file), both backends store it, and both primary keys include it.''.ADD COLUMN IF NOT EXISTS+ a re-key guarded on the constraint's current shape, so it is idempotent.Tests: new multi-file regression (two files, both starting at line 1, both must ingest and both results stored); persistence lib 759 green; rest
bulk_submit13 green. A full-scale e2e (17 Synthea patients, 20 NDJSON files, 11,286 resources through$bulk-submit) is running against this build — result to follow as a comment.