Skip to content

Upgrade OpenIVM to DuckDB 2.0 and pin LPTS 2.0 - #146

Merged
ila merged 10 commits into
mainfrom
upgrade/duckdb-2.0-parser-ctes-20261028
Nov 11, 2026
Merged

ila merged 10 commits into
mainfrom
upgrade/duckdb-2.0-parser-ctes-20261028

Conversation

@ila

@ila ila commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

Why this PR exists

OpenIVM was pinned to DuckDB 1.5.4. It intercepts materialized-view DDL, reads DuckDB logical plans, rewrites them into delta queries, and turns the result back into SQL through LPTS. DuckDB 2.0 changes interfaces at each of those points: extension build tooling, parsed-statement and catalog APIs, expression bindings, and the planner's CTE/correlation/scan shapes. A pin bump alone would fail to build or produce SQL with wrong bindings; some queries that were incremental under 1.5.4 would fall back to full refresh. This PR ports those boundaries to 2.0 while retaining the SQL-to-SQL refresh design.

What changed

  • Build and dependency pins. Pin DuckDB v2.0.0, the matching extension CI release line, and LPTS commit a376261e from cwida/lpts#20. Update CMake/CI for 2.0 extension and bundled DuckLake dependencies. The LPTS PR supplies the 2.0 logical-plan serialization this compiler requires; merge it before this PR.
  • Parsing and lifecycle. DuckDB 2.0 represents catalog names and planner output names as identifier objects rather than the old strings, and its DDL planning hooks expose different statement ownership and lookup paths. Adapt CREATE MATERIALIZED VIEW, replacement, and drop to those APIs instead of relying on text substitutions. Preserve quoted names, SQL string literals, the active schema, and cleanup when DROP VIEW occurs in a multi-statement submission. These changes keep an MV attached to the intended catalog/schema and prevent stale metadata after a drop.
  • Incremental compilation. Normalize 2.0 plan forms for CTE references, correlated table functions/UNNEST, sparse scan bindings, joins, aggregates, and hidden output columns before LPTS serialization and delta rewriting. This matters because the delta rule must replace the right source scan and the generated SQL must refer to the same columns as the original query. A DuckLake filtered-CTE join stays incremental; correlated scalar aggregates use affected-group maintenance; reused aggregate CTEs preserve one shared producer. GROUP BY ... ORDER BY SUM(v) LIMIT n PERCENT keeps unlimited incremental state and applies its order/percentage at the visible boundary. The tests assert classification where relevant and compare refreshed rows with the base query using EXCEPT ALL in both directions.
  • Semantic boundaries. Preserve hidden sort keys across the full-refresh fallback, reject side-effecting MV CTEs, and keep time-dependent expressions dynamic rather than freezing a value folded during CREATE. Adapt native transaction-local window deltas and DuckLake snapshot/scan handling to 2.0. Compiled SQL files now use DuckDB's filesystem API, including Windows long paths.

The source additions are concentrated in parser/plan normalization and the changed 2.0 bindings. They preserve incremental semantics across new plan shapes; the test additions exercise those shapes, lifecycle regressions, and the TPC-DI query corpus. This PR does not switch source capture to native triggers: DuckLake tables cannot host them, and DuckDB 2.0 rejects MERGE INTO on triggered tables.

Existing issues addressed

Issue Change
Fixes #30 Multi-statement DROP VIEW performs OpenIVM cleanup, allowing re-creation.
Fixes #58 Delta-column types come from the catalog; running-window and delta-column lookups are qualified.
Fixes #61 Affected-group filter relaxation edits the parsed query, preserving string literals.
Fixes #62 Delta INSERT target and columns are set before SQL serialization, without text search.
Fixes #63 Internal DDL payloads escape tabs and backslashes.
Fixes #67 MV parsing preserves literal contents and quoted identifiers.
Fixes #72 Unqualified MV creation respects USE <schema>.
Fixes #116 Refresh inside an explicit transaction stamps positional window deltas correctly.

The issue reproducers were rerun on the code commit below with bidirectional bag checks where they refresh an MV. For #58, the catalog-qualified lookup and catalog-entry type path were verified in the diff. Other open correctness and transaction issues are not claimed as fixed here.

Validation

On code commit 7dd3ef49 (the only later change, 0dc16068, deletes an internal upgrade investigation note):

  • GEN=ninja make and GEN=ninja make test: 111 OpenIVM SQL test files passed, none skipped.
  • GEN=ninja make test_release_internal: local integration cases passed; the PostgreSQL case was skipped because OPENIVM_POSTGRES_DSN was unset.
  • TPC-C rewriter corpus: 2,505/2,505 queries validated, created, refreshed, and checked; 2,346 incremental, 159 full-refresh, with no crash or timeout. The classification counts are not substituted for the refresh correctness checks.
  • Required LPTS #20: 47 SQL test files and the 4,755-file DuckDB corpus gate passed. SQLStorm matched its prior per-query states: 16,719 successes, 300 original DuckDB errors, 17 original DuckDB timeouts, and no LPTS-specific failures.
  • A separate TPC-DI SF10/SF100 benchmark PR is measuring this OpenIVM code against DuckDB 2.0; it is not used as proof of this PR's CI result.

Performance status

The SF100 benchmark completed on this PR's executable code. Compared with a recent 1.5.4 OpenIVM artifact, the 2.0 run has the same 50 compiled dbt model queries, identical source-row counts, and the same 49 MV refresh classifications. Its recorded refresh time increased from 50.48 to 58.68 seconds in B2 and from 48.34 to 54.38 seconds in B3, although the total dbt time across all three batches is 1.3% lower because the older B2 run had slow schema-discovery calls. The largest specific slowdown is the silver.watches aggregate MERGE statement (214 → 2,940 ms in SF100 B3); its logged length and SQL prefix match, but the artifacts do not retain full SQL or EXPLAIN plans. The benchmark PR records the full breakdown. This is a performance regression signal to investigate, not an established correctness failure or an isolated DuckDB 2.0 cause.

Diff size against main

git diff --numstat 19762672...0dc16068; additions and deletions are separate. src/ and benchmark/src/ count as source; test/ and benchmark/queries/ as tests/corpus; docs/ and README as docs.

Area Files Added LOC Deleted LOC
Source 75 3,549 2,281
Tests and query corpus 63 1,367 94
Docs 2 15 15
Build, CI, pins, other 7 124 87
Total 147 5,055 2,477

@ila
ila marked this pull request as ready for review November 11, 2026 20:50
@ila
ila merged commit dd357a1 into main Nov 11, 2026
20 checks passed
@ila
ila deleted the upgrade/duckdb-2.0-parser-ctes-20261028 branch November 11, 2026 20:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment