The materialize-consistency suite's reference connector stores wrong sums in a merge
binding, intermittently, under the replayed-acknowledge scenario. Delivery is correct and
only the reduced value is wrong.
This is a defect in the suite's test subject — the reference connector — not in the harness
or in a shipping connector. It does not affect the suite's verdicts on real connectors, but
it is a real correctness bug and it will resurface.
Observed
One failing run of replayed-acknowledge (post-commit-apply class, Replay{times: 3} on
the 4th Acknowledge), on its clean half:
|
|
log binding |
610 of 610 rows, no losses, no duplicates |
| merged delta binding, summed per account |
correct for 40 of 40 |
| merged standard binding |
correct for 12 of 40 |
Errors ran in both directions and at very different magnitudes — one account off by 1
(155 stored against 156 expected), one short by 305 (72 against 377), one over by 292 (480
against 188). The standard row's seq was correct for every account; only the accumulated
sum was wrong.
Why this localises to the load-and-reduce path
Both merged bindings are written from the same Store requests in the same transaction, so
delivery cannot be the difference. A standard binding holds the runtime-reduced document,
which the runtime produces by reducing new documents onto whatever the connector returned
from Load. The delta binding never consults a base, and it was exact — so the fault is in
what Load returns, or in the boundary that decides what is safe to return.
Ruled out
- Staged rows surviving a non-idempotent apply. The
DELETE FROM _flow_staged sits
inside if idempotent, which looks like the cause, but both call sites pass
!has(NonIdempotentAcknowledge), so the clean path always deletes.
staged_txns ordering — it orders by transaction, so an upsert cannot be overwritten
by an older one.
write_rows upsert semantics — standard tables use
INSERT ... ON CONFLICT (key) DO UPDATE.
Load and Store disagreeing about a key — both use key_json.
- A reduction base drawn from uncommitted staging. This was a real defect and is fixed
(load now takes the committed transaction as a bound, with a regression test), but it did
not fix this: the failure occurred with that fix in place, confirmed by comparing the
run's first publish at 12:46:45 UTC against the commit at 12:39 UTC.
Reproduction
Not reproducible on demand. Fifteen consecutive passes after the single failure, seven of
them full-suite runs. It has never failed in isolation — eleven consecutive passes — and a
three-scenario subset intended to reproduce the concurrency cheaply produced four clean runs,
which disproves the theory that concurrency alone is the trigger.
Tooling for the next occurrence
Set FLOW_CONSISTENCY_TRACE_REDUCE=1 in the connector's environment (via the catalog's
local: endpoint env in harness/catalog.rs). It records, per merge-binding key, the base
each Load returned and the value each Store wrote, with the connector's pid and
transaction, into reduce.jsonl in the run directory.
The invariant to check: the base a load returns must equal the value the preceding
store wrote for that key. The first violation is the fault. A pid change at that point
would implicate an unplanned restart, which is the leading hypothesis — a wrong
committed-transaction boundary at recovery would discard committed staging (too low) or apply
uncommitted staging that is then re-delivered and counted twice (too high), which fits errors
in both directions.
The trace is off by default: it writes two lines per merge-binding document and one
scenario's trace reached 61 MB, which is not overhead a measured run should carry.
The
materialize-consistencysuite's reference connector stores wrong sums in a mergebinding, intermittently, under the
replayed-acknowledgescenario. Delivery is correct andonly the reduced value is wrong.
This is a defect in the suite's test subject — the reference connector — not in the harness
or in a shipping connector. It does not affect the suite's verdicts on real connectors, but
it is a real correctness bug and it will resurface.
Observed
One failing run of
replayed-acknowledge(post-commit-apply class,Replay{times: 3}onthe 4th
Acknowledge), on its clean half:logbindingErrors ran in both directions and at very different magnitudes — one account off by 1
(155 stored against 156 expected), one short by 305 (72 against 377), one over by 292 (480
against 188). The standard row's
seqwas correct for every account; only the accumulatedsum was wrong.
Why this localises to the load-and-reduce path
Both merged bindings are written from the same
Storerequests in the same transaction, sodelivery cannot be the difference. A standard binding holds the runtime-reduced document,
which the runtime produces by reducing new documents onto whatever the connector returned
from
Load. The delta binding never consults a base, and it was exact — so the fault is inwhat
Loadreturns, or in the boundary that decides what is safe to return.Ruled out
DELETE FROM _flow_stagedsitsinside
if idempotent, which looks like the cause, but both call sites pass!has(NonIdempotentAcknowledge), so the clean path always deletes.staged_txnsordering — it orders by transaction, so an upsert cannot be overwrittenby an older one.
write_rowsupsert semantics — standard tables useINSERT ... ON CONFLICT (key) DO UPDATE.LoadandStoredisagreeing about a key — both usekey_json.(
loadnow takes the committed transaction as a bound, with a regression test), but it didnot fix this: the failure occurred with that fix in place, confirmed by comparing the
run's first publish at 12:46:45 UTC against the commit at 12:39 UTC.
Reproduction
Not reproducible on demand. Fifteen consecutive passes after the single failure, seven of
them full-suite runs. It has never failed in isolation — eleven consecutive passes — and a
three-scenario subset intended to reproduce the concurrency cheaply produced four clean runs,
which disproves the theory that concurrency alone is the trigger.
Tooling for the next occurrence
Set
FLOW_CONSISTENCY_TRACE_REDUCE=1in the connector's environment (via the catalog'slocal:endpoint env inharness/catalog.rs). It records, per merge-binding key, the baseeach
Loadreturned and the value eachStorewrote, with the connector's pid andtransaction, into
reduce.jsonlin the run directory.The invariant to check: the base a
loadreturns must equal the value the precedingstorewrote for that key. The first violation is the fault. A pid change at that pointwould implicate an unplanned restart, which is the leading hypothesis — a wrong
committed-transaction boundary at recovery would discard committed staging (too low) or apply
uncommitted staging that is then re-delivered and counted twice (too high), which fits errors
in both directions.
The trace is off by default: it writes two lines per merge-binding document and one
scenario's trace reached 61 MB, which is not overhead a measured run should carry.