Skip to content

materialize-consistency: reference connector stores wrong sums in a merge binding #3296

Description

@mdibaiee

The materialize-consistency suite's reference connector stores wrong sums in a merge
binding, intermittently, under the replayed-acknowledge scenario. Delivery is correct and
only the reduced value is wrong.

This is a defect in the suite's test subject — the reference connector — not in the harness
or in a shipping connector. It does not affect the suite's verdicts on real connectors, but
it is a real correctness bug and it will resurface.

Observed

One failing run of replayed-acknowledge (post-commit-apply class, Replay{times: 3} on
the 4th Acknowledge), on its clean half:

log binding 610 of 610 rows, no losses, no duplicates
merged delta binding, summed per account correct for 40 of 40
merged standard binding correct for 12 of 40

Errors ran in both directions and at very different magnitudes — one account off by 1
(155 stored against 156 expected), one short by 305 (72 against 377), one over by 292 (480
against 188). The standard row's seq was correct for every account; only the accumulated
sum was wrong.

Why this localises to the load-and-reduce path

Both merged bindings are written from the same Store requests in the same transaction, so
delivery cannot be the difference. A standard binding holds the runtime-reduced document,
which the runtime produces by reducing new documents onto whatever the connector returned
from Load. The delta binding never consults a base, and it was exact — so the fault is in
what Load returns, or in the boundary that decides what is safe to return.

Ruled out

  • Staged rows surviving a non-idempotent apply. The DELETE FROM _flow_staged sits
    inside if idempotent, which looks like the cause, but both call sites pass
    !has(NonIdempotentAcknowledge), so the clean path always deletes.
  • staged_txns ordering — it orders by transaction, so an upsert cannot be overwritten
    by an older one.
  • write_rows upsert semantics — standard tables use
    INSERT ... ON CONFLICT (key) DO UPDATE.
  • Load and Store disagreeing about a key — both use key_json.
  • A reduction base drawn from uncommitted staging. This was a real defect and is fixed
    (load now takes the committed transaction as a bound, with a regression test), but it did
    not fix this: the failure occurred with that fix in place, confirmed by comparing the
    run's first publish at 12:46:45 UTC against the commit at 12:39 UTC.

Reproduction

Not reproducible on demand. Fifteen consecutive passes after the single failure, seven of
them full-suite runs. It has never failed in isolation — eleven consecutive passes — and a
three-scenario subset intended to reproduce the concurrency cheaply produced four clean runs,
which disproves the theory that concurrency alone is the trigger.

Tooling for the next occurrence

Set FLOW_CONSISTENCY_TRACE_REDUCE=1 in the connector's environment (via the catalog's
local: endpoint env in harness/catalog.rs). It records, per merge-binding key, the base
each Load returned and the value each Store wrote, with the connector's pid and
transaction, into reduce.jsonl in the run directory.

The invariant to check: the base a load returns must equal the value the preceding
store wrote for that key.
The first violation is the fault. A pid change at that point
would implicate an unplanned restart, which is the leading hypothesis — a wrong
committed-transaction boundary at recovery would discard committed staging (too low) or apply
uncommitted staging that is then re-delivered and counted twice (too high), which fits errors
in both directions.

The trace is off by default: it writes two lines per merge-binding document and one
scenario's trace reached 61 MB, which is not overhead a measured run should carry.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions