Skip to content

feat(sql): support streaming IN subqueries - #2540

Open
pythonyx135793 wants to merge 11 commits into
getdozer:mainfrom
pythonyx135793:codex/dozer1659-dynamic-in-r1367
Open

pythonyx135793 wants to merge 11 commits into
getdozer:mainfrom
pythonyx135793:codex/dozer1659-dynamic-in-r1367

Conversation

@pythonyx135793

@pythonyx135793 pythonyx135793 commented Oct 25, 2026 •

Copy link
Copy Markdown

/claim #1659

Adds live IN (SELECT ...) and NOT IN predicates to WHERE. Each inner query feeds a stateful selection processor alongside the outer stream. Changes on either side produce the matching inserts, updates and deletes; duplicate inner values are reference-counted, and duplicate outer rows keep their multiplicity.

This also handles NULL/empty results, compound predicates, multiple or nested subqueries, and the planner's projections, joins, grouping/HAVING, CTEs and UNION. The existing static-IN path stays intact. Unsupported correlation, row-valued membership and unsupported relational clauses return explicit errors.

Validation on Linux, Rust 1.77.2, source 80027f3d:

  • SQL, expression and core default-feature suites: 332 passed, zero failures, one pre-existing ignored test.
  • Changed-source formatting, scoped Clippy and the actual source → SQL → DAG → sink demo passed.
  • The same public-API regression fails against the original source because dynamic membership is unsupported there.
  • Real-DAG tests cover changing membership, all outer update transitions, duplicates, NULLs, batches, lexical CTE scope, nested DISTINCT, UNION update atomicity, and CTE-backed windows on either JOIN side and chained joins. TTL/nested-operator checks cover construction, not expiration behavior.

Validation run · Demo video

The demo shows actual engine output with its original timing. dozer-sql/tests/DYNAMIC_IN.md explains the support boundaries and how to run it. Retained state is in memory; general compound predicates can require a scan of retained rows when membership changes.

@CLAassistant

CLAassistant commented Oct 25, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

pythonyx135793 and others added 2 commits October 25, 2026 11:48
- Remove dozer-sql/examples/dynamic_in.rs
- Update dozer-sql/src/builder/mod.rs
- Remove dozer-sql/src/builder/parser.rs
- Update dozer-sql/src/selection/dynamic/mod.rs
- Update dozer-sql/src/selection/dynamic/processor.rs
- Remove dozer-sql/tests/DYNAMIC_IN.md
- Update dozer-sql/tests/dynamic_in.rs
@pythonyx135793

Copy link
Copy Markdown
Author

Pushed an update in 19fe5c8 that addresses the feedback; CI is re-running.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants