Repository navigation
Upgrade OpenIVM to DuckDB 2.0 and pin LPTS 2.0 - #146
Merged
Merged
Conversation
This was referenced Nov 4, 2026
ila
marked this pull request as ready for review
November 11, 2026 20:50
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why this PR exists
OpenIVM was pinned to DuckDB 1.5.4. It intercepts materialized-view DDL, reads DuckDB logical plans, rewrites them into delta queries, and turns the result back into SQL through LPTS. DuckDB 2.0 changes interfaces at each of those points: extension build tooling, parsed-statement and catalog APIs, expression bindings, and the planner's CTE/correlation/scan shapes. A pin bump alone would fail to build or produce SQL with wrong bindings; some queries that were incremental under 1.5.4 would fall back to full refresh. This PR ports those boundaries to 2.0 while retaining the SQL-to-SQL refresh design.
What changed
a376261efrom cwida/lpts#20. Update CMake/CI for 2.0 extension and bundled DuckLake dependencies. The LPTS PR supplies the 2.0 logical-plan serialization this compiler requires; merge it before this PR.CREATE MATERIALIZED VIEW, replacement, and drop to those APIs instead of relying on text substitutions. Preserve quoted names, SQL string literals, the active schema, and cleanup whenDROP VIEWoccurs in a multi-statement submission. These changes keep an MV attached to the intended catalog/schema and prevent stale metadata after a drop.UNNEST, sparse scan bindings, joins, aggregates, and hidden output columns before LPTS serialization and delta rewriting. This matters because the delta rule must replace the right source scan and the generated SQL must refer to the same columns as the original query. A DuckLake filtered-CTE join stays incremental; correlated scalar aggregates use affected-group maintenance; reused aggregate CTEs preserve one shared producer.GROUP BY ... ORDER BY SUM(v) LIMIT n PERCENTkeeps unlimited incremental state and applies its order/percentage at the visible boundary. The tests assert classification where relevant and compare refreshed rows with the base query usingEXCEPT ALLin both directions.The source additions are concentrated in parser/plan normalization and the changed 2.0 bindings. They preserve incremental semantics across new plan shapes; the test additions exercise those shapes, lifecycle regressions, and the TPC-DI query corpus. This PR does not switch source capture to native triggers: DuckLake tables cannot host them, and DuckDB 2.0 rejects
MERGE INTOon triggered tables.Existing issues addressed
DROP VIEWperforms OpenIVM cleanup, allowing re-creation.USE <schema>.The issue reproducers were rerun on the code commit below with bidirectional bag checks where they refresh an MV. For #58, the catalog-qualified lookup and catalog-entry type path were verified in the diff. Other open correctness and transaction issues are not claimed as fixed here.
Validation
On code commit
7dd3ef49(the only later change,0dc16068, deletes an internal upgrade investigation note):GEN=ninja makeandGEN=ninja make test: 111 OpenIVM SQL test files passed, none skipped.GEN=ninja make test_release_internal: local integration cases passed; the PostgreSQL case was skipped becauseOPENIVM_POSTGRES_DSNwas unset.Performance status
The SF100 benchmark completed on this PR's executable code. Compared with a recent 1.5.4 OpenIVM artifact, the 2.0 run has the same 50 compiled dbt model queries, identical source-row counts, and the same 49 MV refresh classifications. Its recorded refresh time increased from 50.48 to 58.68 seconds in B2 and from 48.34 to 54.38 seconds in B3, although the total dbt time across all three batches is 1.3% lower because the older B2 run had slow schema-discovery calls. The largest specific slowdown is the
silver.watchesaggregate MERGE statement (214 → 2,940 ms in SF100 B3); its logged length and SQL prefix match, but the artifacts do not retain full SQL or EXPLAIN plans. The benchmark PR records the full breakdown. This is a performance regression signal to investigate, not an established correctness failure or an isolated DuckDB 2.0 cause.Diff size against main
git diff --numstat 19762672...0dc16068; additions and deletions are separate.src/andbenchmark/src/count as source;test/andbenchmark/queries/as tests/corpus;docs/and README as docs.