DuckLake: test the writes a pipeline makes into DuckDB-created tables - #14
Open
fuziontech wants to merge 2 commits into
Open
DuckLake: test the writes a pipeline makes into DuckDB-created tables#14fuziontech wants to merge 2 commits into
fuziontech wants to merge 2 commits into
Conversation
A pipeline that writes from Trino into a DuckLake catalog DuckDB owns hits paths the existing write tests do not: every table there was created by the other engine, so its partitioning, its column types and its views are ones this connector has to accept rather than choose. TestDuckLakeWriteRoundTrips creates each table through DuckDB, writes it through Trino, and reads it back through both. It covers partitioned inserts and deletes, the types an event table carries, replacing a view DuckDB wrote, and inserting after DuckDB adds a column. Two behaviors the tests pin down are worth knowing before relying on them: - A DELETE covering whole partitions of a table filed by a DATE column reads every row of them. isEnforceableType excludes DATE, so the predicate is never enforced by the connector and the partition values cannot decide it. The files are still dropped, because nothing is left in them, but the statement pays for a full scan of the days it removes. The same delete over a VARCHAR or INTEGER key touches no row. - The query text stored in ducklake_view is the query as Trino formats it, and keeps whatever catalog name the view was written with. A view naming its tables through the Trino catalog does not bind in DuckDB, which has no database of that name. The DuckDB read helpers move from TestDuckLakeWrites onto the catalog itself, where both test classes reach them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe
A pipeline that materializes DATE_TRUNC('day', ts)::TIMESTAMPTZ files its
tables by a timestamp, not by a DATE. isEnforceableType admits only TINYINT,
SMALLINT, INTEGER, BIGINT, BOOLEAN and unbounded VARCHAR, so that key is
excluded for the same reason DATE is, and a DELETE covering whole days reads
every row of them.
The test pins the current cost down: the delete reports the rows of the two
days it removes as processed input positions. That number becomes 0 once a
predicate on a temporal partition column is enforced.
It also covers the write path the exclusion says nothing about. An identity
partition key of this type is writable, one file per day, and the value is
recorded in UTC whatever zone the row carried; a row DuckDB inserts afterwards
lands under the value Trino recorded for that day.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe
This was referenced Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TestDuckLakeWriteRoundTrips(9 tests): DuckDB creates the table throughduckdb_jdbc+ theducklakeextension against the testcontainers Postgres catalog, Trino writes, both engines read back.TestingDuckLakeCatalog.rows()/scalar()replace a private helper inTestDuckLakeWrites.What it establishes
year(ts)/day(ts)transformsducklake_file_partition_value; a DuckDB insert lands under the same valuesDATEkey, and on aTIMESTAMP WITH TIME ZONEkeyisEnforceableTypeadmits only integer, boolean and unbounded varchar keys, so temporal partition predicates are never enforced and the delete takes the row-level path. The assertions pin the current cost (getProcessedInputPositions() == 3) and are the ones to flip to0when temporal keys become enforceable.$-named fields), HUGEINT, UBIGINT, JSON, TIMESTAMPTZ with microseconds, DECIMAL(18,3), UUID, VARCHAR[], MAPCREATE OR REPLACE VIEWover a DuckDB view, thenDROP VIEWtrino-dialect row, nothing left after drop; storedsqlis Trino's formatted text with a literal catalog prefix, which DuckDB cannot bind (Catalog "ducklake" does not exist)ADD COLUMNWhy
The first real write workload on the cell is a customer's SQLMesh pipeline writing into tables DuckDB created and keeps reading. These tests pin what holds today and turn the two gaps (temporal partition enforcement, catalog-qualified view SQL) into concrete assertions for the follow-up changes.
Not in this PR
TestDuckLakeMetastoreConnectionPool.testIdleConnectionsAreReleasedflaked once in a full-module run (0L to be greater than 0L, timing onconnectorConnectionCount()); it is untouched here and passed on re-runs.Test plan
./mvnw -pl plugin/trino-ducklake test -Dtest='TestDuckLakeWriteRoundTrips*'→ 9 tests, 0 failures (JDK 25 container, testcontainers Postgres).-P cichecks (license, checkstyle, modernizer) pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01WCY5Jf2BQPCVKJTZU1TpEe