[tnunamak]

What

A parent checkpoint could be committed while records of a child stream mapped to it were not in the store. Two paths led there (Collection Profile 0.2.0 §5.3: STATE is staged only after all prior records are durably written):

1. Certified failure (§5.8). A connector emits RECORD for a child stream, then STATE for its parent, then a certified stream_collection_failed DONE. handleStateMessage flushed only the named state stream. The child buffers under its own name and a failed DONE does not flush it, so the parent cursor was staged and committed ahead of the child's records.
2. Transient manifest drift. A batch dropped as not_found for a manifest-declared stream was recorded as a gap, but collectReportedOmissions did not report it to decideCheckpointCommits. The parent that stream maps to saw no omission and committed.

Root cause

The flush before staging state was scoped to one stream, but the commit invariant is scoped to a state stream and every child that maps to it. The drift skip was tracked in a separate set that the omission collector never read.

Change

reference-implementation/runtime/index.ts, two edits:

• handleStateMessage: flushBatch(stateStream) becomes flushAll().
• collectReportedOmissions: streams in driftSkippedStreams are reported as retryable_skip.

Tests

• runtime-stream-collection-failed-commit.test.ts: the child's record is stored, and its batch event precedes run.state_advanced, before the parent commit of a certified failure.
• runtime-ingest-manifest-drift.test.ts: a drift-dropped child batch leaves the parent cursor unchanged and records a checkpoint_withheld gap.
• collection-profile.test.ts: the characterization test that pinned per-stream flushing now pins flush-all (records_flushed 2, buffered_records_dropped 0).

Guard tests in runtime-ingest-manifest-drift.test.ts:

• a child that drifts at DONE, after the parent's STATE, still withholds the parent;
• a child's drift withholds only the parent it maps to; an unrelated stream's cursor commits.

The three drift tests fail with the omission line removed and pass with it. The drift withhold is whole-run, like a retryable SKIP_RESULT: it does not compare a dropped record's position with the staged cursor, so a drop after STATE also withholds the parent. That is conservative (the window is re-collected, nothing is lost), and the second guard test pins it.

Each Postgres variant runs against a disposable database on the dedicated Postgres test listener and asserts the backend in use. Without PDPP_TEST_POSTGRES_URL those variants are reported as skipped.

cd reference-implementation
PDPP_TEST_POSTGRES_URL=<dedicated test listener> node --import tsx --test test/runtime-stream-collection-failed-commit.test.ts test/runtime-ingest-manifest-drift.test.ts   # 26 pass, 0 fail
node --import tsx --test test/runtime-stream-collection-failed-commit.test.ts test/runtime-ingest-manifest-drift.test.ts                                                   # 22 pass, 0 fail, 4 skipped (Postgres variants)
node --import tsx --test test/collection-profile.test.ts                                                                                                                   # 149 pass, 0 fail
npx tsc --noEmit                                                                                                                                                           # exit 0

With the per-stream flush restored, the certified-failure test fails on both SQLite and Postgres. collection-profile.test.ts asserts the runtime's flush counters, which do not depend on storage, so it stays on SQLite.

A wider run over 86 related test files showed one failure, connector-failure-diagnostics.test.ts ("stderr-bearing exit before DONE"). It fails the same way on unmodified main: the Node deprecation warning adds stderr bytes the test does not expect. ultracite check on the four touched files reports the same 5 errors and 79 warnings as the unmodified files.

Risks

• Cost of flushing every stream at each STATE: batches are per stream, so one STATE can now flush the partial batches of every buffered stream. With k buffered streams and STATE emitted under BATCH_SIZE (500) records, that is up to k ingest POSTs instead of 1. Streams that interleave records between STATEs pay this even when STATE is infrequent: for example, 2000 records spread evenly over four streams with a STATE for one stream every 1000 records make 5 POSTs before and 8 after (counting the terminal flush). DETAIL_COVERAGE and STREAM_EVIDENCE already flush every stream. Records are still written in arrival order per stream.
• A drift-dropped stream now holds back its parent's checkpoint, so the next run re-reads that window. That is the intended retry behavior.
• If the RS never learns a child stream, the parent is withheld on every run. It re-collects its window each run and the stall shows as checkpoint_stalled after 3 runs. Before this change the child's records were lost silently while the parent advanced.

Assisted-by: AI