Skip to content

Upgrade LPTS to DuckDB 2.0 and stabilize recursive checking - #20

Merged
ila merged 6 commits into
mainfrom
upgrade/duckdb-2.0-20261028
Nov 5, 2026
Merged

ila merged 6 commits into
mainfrom
upgrade/duckdb-2.0-20261028

Conversation

@ila

@ila ila commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Why this PR exists

LPTS converts DuckDB logical plans back to SQL. DuckDB 2.0 changes the bound-expression and scan-filter APIs and the shapes of CTE, recursive, pruned set-operation, and scan plans. Updating only the DuckDB pin would leave LPTS unable to build or would produce SQL that reparses with different bindings or types. The port also exposed two practical problems: the round-trip checker could hang on recursive queries that DuckDB itself completes, and repeated inlined CTEs could generate hundreds of kilobytes of SQL.

DuckDB 2.0 port: what broke and how it works now

  • Pin DuckDB v2.0.0, matching DuckLake and extension CI tooling, and update CMake, static benchmark loading, and CI. DuckDB core is unchanged.
  • DuckDB 2.0 no longer exposes several bound-expression fields and scan filters in the form LPTS read under 1.5.4. LPTS now uses 2.0 getters and expression callbacks, renders the new ExpressionFilter form, distinguishes fixed-size ARRAY from LIST, and delegates optimizer-inserted comparison casts/collations to DuckDB's bound-expression SQL exporter. That last step makes the emitted SQL rebind to the same types rather than merely look similar.
  • DuckDB 2.0 also presents CTEs, recursion, scans, and set operations with different logical-plan bindings. LPTS now follows each CTE reference to its producer, maps recursive outputs and sparse scan column IDs to their visible positions, and preserves synthetic rowids, pruned UNION inputs, and hidden output columns. Without these repairs, SQL could parse successfully but read a different column or return a different number of duplicate rows.

The largest source changes are in lpts_ast_builder.cpp and lpts_expression_renderer.cpp because the 2.0 plan and expression APIs changed at many operator boundaries. The CTE serializer and checker changes below account for the other substantial additions. Tests cover round-trip results, not just successful parsing.

Recursive checking and SQL size

  • Recursive checker: the prior fused original/rewrite plan could eagerly evaluate an unbounded recursive branch that the original query skips, causing LPTS-only timeouts. Recursive checks now execute original and rewritten queries as separate MATERIALIZED CTEs within one SQL statement and snapshot, then compare both bags with EXCEPT ALL in both directions. This preserves the query result and avoids the fused recursive pipeline. Nonrecursive checks retain their existing fast path.
  • Repeated CTEs: this is the size of the generated SQL text, not a database file. If each CTE level refers to the previous level twice, repeated inlining makes its text and DuckDB's optimized plan grow roughly exponentially with depth. LPTS previously serialized equivalent producer subplans repeatedly; a depth-10 test emitted 464,799 bytes, which made parsing and planning needlessly expensive and could hit expression-depth limits. The serializer now emits a shared CTE for a repeatable producer, keeps each consumer's column positions correct, prunes unreachable CTEs, and materializes only producers still referenced more than once. The same test emits 8,052 bytes (about 58× smaller) and returns the same result. Sharing is disabled for volatile, sampling, recursive, or writing plans where changing evaluation count could change results. A bounded-depth CTE boundary also prevents excessive inlining in long plans.
  • Ten exact SQLStorm regressions now run through the native Unix make test_release gate; duplicated query text was removed from cte.test.

Validation at a376261e

  • GEN=ninja make format-fix and git diff --check passed.
  • GEN=ninja make test: 47 LPTS SQL test files, the ten pinned SQLStorm cases, and the 4,755-file DuckDB corpus gate passed with no new WRONG or FAIL.
  • Full SQLStorm TPC-H at SF 0.001 with a 10-second timeout: 16,719 successes, 300 original DuckDB errors, 17 original DuckDB timeouts, and zero LPTS-specific errors, timeouts, crashes, or incorrect results. All 17,036 per-query states match the prior run. The 25 longest source queries generated at most 15,672 bytes of SQL.
  • Extension CI builds and format/tidy checks passed on the PR head. The original DuckDB errors/timeouts are reported separately and are not LPTS failures.

CI annotation cleanup at fe2096d

  • Replaced deprecated DuckDB 2.0 DataChunk::SetValue/SetCardinality, Vector::Flatten(count), and KeywordHelper::WriteOptionallyQuoted calls with their supported equivalents. SQL identifier quoting is unchanged.
  • Targeted LPTS SQL tests and the 4,755-file DuckDB corpus gate passed with no new WRONG or FAIL. Full SQLStorm at SF 0.001 reproduced the previous 17,036-query result exactly: 16,719 successes, 300 source DuckDB errors, 17 source DuckDB timeouts, and zero LPTS errors, timeouts, crashes, or incorrect results. The local full make test run could not complete its two DuckLake tests because their INSTALL ducklake step requires network access; PR CI will rerun them.
  • The remaining CMake warnings about ENABLE_EXTENSION_AUTOLOADING and ENABLE_EXTENSION_AUTOINSTALL come from the pinned upstream extension-ci-tools Makefile, which still passes flags that DuckDB 2.0 ignores. LPTS itself does not set those flags.

OpenIVM's DuckDB 2.0 port, ila/openivm#146, pins this commit and should merge after this PR.

Diff size against main

git diff --numstat 29c606ee...a376261e; additions and deletions are separate. src/ and benchmark/ count as source; test/ as tests; docs/ and README as docs.

Area Files Added LOC Deleted LOC
Source 17 2,252 1,200
Tests 13 738 40
Docs 3 5 3
Build, CI, pins, other 6 57 28
Total 39 3,052 1,271

@ila
ila merged commit 1ce2284 into main Nov 5, 2026
18 checks passed
@ila
ila deleted the upgrade/duckdb-2.0-20261028 branch November 5, 2026 15:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant