Skip to content

Keep reduced-term Spark join compilation for DuckDB 2.0 RIGHT joins - #156

Open
ila wants to merge 5 commits into
mainfrom
perf/duckdb2-spark-join-plans
Open

ila wants to merge 5 commits into
mainfrom
perf/duckdb2-spark-join-plans

Conversation

@ila

@ila ila commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Why

DuckDB 2.0 keeps some optimized one-sided outer joins as RIGHT joins. The regular N-term compiler accepted INNER and LEFT joins but rejected RIGHT, so Spark's gold.fact_market_history fell back to a 15-term inclusion-exclusion program during SF100 batch 2.

Change

Accept RIGHT joins in the same one-sided outer-join path as LEFT joins. The existing term builder and NULL-transition handling already orient themselves by the null-supplying side; no new delta rule is needed.

Evidence

  • An exact TPC-DI-shaped fact_market_history compile shrinks from 112,470 to 43,301 SQL bytes, with 14 to 9 UNION ALL occurrences while preserving the RIGHT join.
  • Extended the existing regular N-term SQL test with both match-transition directions and full EXCEPT ALL equality after a mixed INSERT/DELETE/UPDATE batch.
  • test/sql/left_join_regular_nterm.test: 60 assertions pass.
  • test/integration/test_regular_nterm_compiled.py: passes.
  • Spark/Delta integration with the new native binary: RIGHT-join MV is bag-equal before and after one mixed refresh.

This PR is stacked on #147 because the benchmark currently pins its head. It does not change DuckLake row-ID upsert logic.

@ila
ila changed the base branch from perf/ducklake-rowid-upsert-poc to main November 11, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant