[ila]

Why

DuckDB 2.0's DuckLake aggregate MERGE scans a large backing table even when the grouped delta is small. In the SF100 Figure 1 workload, silver.watches spent 2,547/2,479 ms in the MERGE statement for B2/B3 on the preceding candidate.

What changes

For insert-only DuckLake grouped aggregates, stage the grouped delta with each matching target row ID, then UPDATE matched rows and INSERT new groups. The existing MERGE remains for deletes, inline cascade maintenance, and any output column named rowid that hides DuckLake's virtual row ID. A standalone grouped DuckLake refresh executes the data program in one transaction so these writes commit together.

The review found a watermark regression from that transaction: it checked DuckLake sources before commit and could leave the watermark behind the MV's own new snapshot. OpenIVM now verifies the sources again after commit, before advancing the watermark. A concurrent source commit at that boundary leaves the refresh marked for recovery and triggers a retry; the existing no-op and concurrency regressions pass.

Performance evidence

SF100 Figure 1 run 41999990131 passed at POC pin 7dc2067a (one run, B1/B2/B3 inserts 100/1/0.01, OpenIVM only). Against the preceding SF100 run 41901172305, B2+B3 dbt time fell from 144.98 to 133.69 seconds (7.8%). Complete silver.watches refresh fell from 2,965/2,882 to 690/690 ms; its staged match, UPDATE and INSERT together took 291/290 ms versus 2,547/2,479 ms for the prior MERGE. All three batches and their 50 dbt nodes completed. SF100 post-batch validation was disabled, as in the preceding run. The runs are single observations, and other model timings also changed, so the dbt difference is not solely attributable to this path.

Current head f37e1ad2 adds the review fix, edge tests, and a guard requiring an explicit insert-only fact before staging row IDs in compile-only output. It also skips an unnecessary dependency lookup for split-protocol refreshes, which already require an atomic data transaction. silver.watches has a downstream view, so it does not enter the new standalone post-commit check. A local standalone refresh measured that additional source check at 9–19 ms; this is one small-case measurement, not an SF100 estimate.

Correctness checks

• test/sql/ducklake_aggregate.test: 229 assertions passed locally, including nullable two-column keys, a physical rowid column, both sides of a joined aggregate changing before one refresh, and a one-snapshot assertion for staged UPDATE/INSERT. The committed test passed using its normal install command.
• ducklake_deltas.test (244 assertions), ducklake_concurrent_source_commit.test (75), ducklake_chained.test (698), ducklake_left_join.test (168), and ducklake_chained_recovery.test (40) passed on the review fix.
• Local four-view watch pipeline with 60,000,420 initial events and subsequent batches matched the base queries with EXCEPT ALL in both directions. The larger review rerun also passed full bag checks for all four views.
• GEN=ninja make and git diff --check passed. make format-fix passed. The latest one-line lookup change rebuilt successfully; the watermark and concurrent-source tests passed again (244 and 75 assertions).

Stacked on DuckDB 2.0 PR #146. This remains a draft until its CI checks pass.