DuckLake: enforce partition predicates on temporal keys - #18
Open
fuziontech wants to merge 2 commits into
Open
Conversation
isEnforceableType excluded DATE and every timestamp type, so a DELETE covering whole partitions of a daily key read and rewrote every row of those partitions instead of ending their files. A pipeline that replaces day intervals paid for a full scan on every run. Identity partition keys of type DATE, TIMESTAMP(p) and TIMESTAMP(p) WITH TIME ZONE are now enforced, with a fail-open parse. A value the connector cannot read back is never guessed at in either direction: the file is not pruned, and it is not counted as matching. DuckDB writes four such values: infinity, -infinity, a BC date, and a year of five or more digits. The connector may not prune such a file, and it cannot apply a predicate to the rows of a file on its own. So one of these values in a table leaves the whole predicate with the engine, which keeps filtering. Enforcement is therefore decided per table: applyFilter enforces a temporal key only when the partition value of every visible data file decides the column. It checks that over the same files, and through the same code, that the split manager prunes by. The parse itself does not change. StatsValueParser already read these values as unknown. Its unit tests now pin that, along with the offset forms a writer outside UTC records. Transform partitions are unaffected. year, month, day and hour are not identity keys, so predicates on them stay unenforced and their pruning stays fail-open. Claude-Session: https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe
The check that every visible file's value parses listed the data files of the table. That is one row per file on a table Portola files by day and fills with thousands of files, paid at plan time, on every query with a predicate on the day. JdbcDuckLakeMetastore.distinctPartitionValues asks for the distinct values of one partition key instead: one row per day, not one per file. It returns each value in the shape DuckLakeDataFileEntry carries, so PartitionTransforms still reads it, and a file recording nothing under the key still comes back as the empty map that says so. It is a separate query from the one the split manager prunes by, because applyFilter decides enforcement before any file list is loaded. It runs only for a temporal key that a predicate constrains, and it is the third query enforcement already costs there. The parser tests now pin the day key a pipeline writes: midnight with no fractional seconds parses to the value TIMESTAMP '2026-01-01 00:00:00 UTC' carries at precision 6, and the same instant written with trailing zeros parses to the same value. A possible follow-up, deliberately not taken here: map infinity and -infinity to the exact sentinels DuckDB encodes them as, INT32_MAX days and INT64_MAX microseconds. Trino reads those same numbers back, so such a file would become prunable and a whole-partition DELETE beside one would stay on the metadata path. It would tie the connector to DuckDB's sentinel encoding, so it needs its own decision. Claude-Session: https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #14 (test-only). Retarget to
masteronce #14 merges; the diff here is the two top commits.Summary
isEnforceableTypeadmitted only integer, boolean and unbounded varchar partition keys, so a predicate on aDATE/TIMESTAMP/TIMESTAMP WITH TIME ZONEidentity-partition column was only fail-open pruned, never enforced — and a whole-partitionDELETE … WHERE day BETWEEN …on such a key read and rewrote every row of those partitions instead of ending the files. A customer's SQLMesh pipeline partitions 34 tables by aTIMESTAMPTZ daycolumn and replaces day intervals every hour; this is its hot path.applyFilterchecks that with oneSELECT DISTINCT partition_key_index, partition_valueper constrained temporal key (one row per distinct value, not per file; a LEFT JOIN keeps "no value recorded" distinct from SQL NULL). If any value is undecidable (infinity,-infinity, BC dates, 5-digit years — exactly whatStatsValueParseralready refuses), the column stays unenforced for that statement: rows are still filtered, and a DELETE takes the row-level path for the whole statement, because Trino's delete is all-or-nothing per table and a metadata delete over the other files could otherwise leave matching rows behind.year()/month()/day()/hour()transform partitions are unchanged: fail-open pruned, never enforced.2026-01-01,2026-01-01 12:34:56[.mmm|.uuuuuu][+00|+02|+05:30]; a value without fractional seconds compares equal to the precision-6 predicate (2026-01-01 00:00:00+00≡TIMESTAMP '2026-01-01 00:00:00 UTC').Tests
TestDuckLakeWriteRoundTrips: the DATE and TIMESTAMPTZ whole-day DELETE tests flip from "reads every row" togetProcessedInputPositions() == 0, files ended, no delete file, DuckDB round-trip kept;testDeletingWholeDaysBesideAnInfiniteDateLeavesItAlonepins the row-level fallback (exact files ended, no delete file, theinfinityrow intact in both engines, positions == rows of the deleted days);testReadingADayBesideAnInfiniteDatePrunesTheOtherDayscovers read-side pruning with stats pruning off.TestStatsValueParser: fail-open forms, signed vs bare 5-digit years,+05:30, and the precision pair above.-P ciclean.Follow-ups, not here
infinity/-infinity(DuckDB'sINT32_MAXdays /INT64_MAXµs) would make such files decidable; needs its own decision since it relies on the encoding.applyFilter's convergence call issues.TIMESTAMP(p>6)andTIMESTAMPTZ(p≠6)are never enforced (parser returns empty); the Performance paragraph describes the precision-6 case.https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe