[Aming9303]

Summary

Implements the remaining dynamic WHERE ... IN (SELECT ...) path from #1659.
Static IN-list evaluation already exists on main and is left unchanged.

• Lower uncorrelated IN/NOT IN subqueries into separately connected streaming inputs.
• Retain outer-row multiplicity and lifetime metadata; reference-count inner values.
• Propagate outer/inner inserts, updates, deletes and batch inserts without multiplying output for inner duplicates.
• Preserve NULL/empty-set membership and three-valued boolean combinations; keep internal membership columns out of downstream schemas/records.
• Support multiple membership predicates, nested queries and child CTEs without leaking child names.
• Reject multi-column results. Tighten binding of known qualified source names so unsupported correlated references cannot silently bind to an inner column of the same name.
• Roll back retained state on recoverable operation errors, before forwarding output.
• Snapshot only mutable state needed for rollback; ordinary outer-row changes do not copy all retained rows. Inner changes rescan rows without cloning the complete outer collection.

Reproduction And Validation

On main 8b0b4cc021db089722cb3d48dbe915e8e9c532ec, the new real SQL-to-DAG regression failed with Factory(UnsupportedExpression(InSubquery ...)).

Current patch, Windows GNU Rust 1.83.0 / protoc 25.3, original lockfile:

cargo test -p dozer-sql --lib --locked
# 232 passed, 0 failed, 1 ignored
cargo test -p dozer-sql-expression --lib --locked
# 27 passed, 0 failed
cargo test -p dozer-sql --lib demo_streaming_membership --locked -- --nocapture

The DAG tests run parsed SQL through real sources, processors and a recording sink. They assert output multiplicity and projection, multiple predicates, nested queries, CTEs, and rejection of invalid/correlated subqueries. Processor regressions cover changes on both sides, inner/outer duplicates, NULL/empty sets, mixed numeric equality, independent inputs, and failure rollback. A property test compares randomized insert/delete/NULL event sequences against a multiset reference model for both IN and NOT IN.

rustfmt --check on the changed Rust files and git diff --check pass. This is not a claim that the entire workspace, external connectors, optional feature combinations or production deployment were tested.

Short Demo

Actual Rust membership-processor demo (WebM)

Screenshot

The video captures output from an actual local Rust regression run, demonstrating retained outer rows, duplicate inner references and final-reference retractions. It is a processor demonstration, not a cloud/UI deployment; the additional SQL-to-DAG coverage is in the tests.

Limits

• Correlated and projected subqueries are not added; unsupported scalar subquery forms remain unsupported.
• Existing streaming planner restrictions (e.g. ORDER BY/LIMIT) remain.
• Correctness-first in-memory state: inner changes rescan retained outer rows, and membership scans distinct inner values. No bounded-memory, indexed-performance or persistence guarantee is introduced.

Implementation notes are in dozer-sql/src/selection/membership.md.

Prepared and tested with Codex AI assistance; no competing contributor patch was copied. No credentials, real accounts or paid API operations were used.

/claim #1659

Closes #1659

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}