Nightly upstream merge - #23
Merged
Merged
Conversation
Add `query.max-output-data-size` configuration property Enforce a maximum limit on the output data size a query can produce during execution. If a query exceeds this limit, execution is terminated with the `EXCEEDED_OUTPUT_LIMIT` error code.
Use the input scale for the overflow contribution when reconstructing the decimal sum. Previously only the wrapped 128-bit value was scaled, leading to incorrect averages or spurious overflow failures.
Skip unread Parquet pages when column indexes are present. Pass the predicate into ParquetReader unconditionally. DELETE, UPDATE, MERGE, and $row_id skip pages and still match skip-off results. Tests extend Hive BaseTestParquetPageSkipping and append Hive fixtures as Iceberg DataFiles.
Pages were charged at getSizeInBytes, which excludes the unused capacity of the backing arrays left by block builders. Pages with excess retained bytes are now compacted once.
Filters finish before projections are evaluated, but keeping their row selection entirely in PageProcessor prevents capable lazy sources from avoiding work for rejected projected values. Applying every selection is also unsafe: when a source must decode the full block, remapping the page can replace compact projection output with a slower indirect representation. Offer the selection as an atomic operation. Sources accept only when they can avoid loading or decoding data; rejection preserves the existing projection path. Materialize known filter inputs before entering generated code so reader implementations do not interfere with optimizing the filter loop.
Masking a fully decoded block preserves semantics, but it does not provide late materialization because rejected values still incur decode and allocation costs. Allow flat primitive readers to consume ascending selected positions directly. Unsupported readers and selection shapes retain the existing full-read path.
Selected reads still pay decompression and decoding costs for pages that contain no selected values, leaving much of the sparse-read benefit unrealized. Allow flat readers to identify and bypass complete unselected data pages with bounded lookahead. Before advancing the chunk stream, copy only the buffered page that still aliases source-owned input. This keeps ownership local to the page reader without cross-layer reference tracking.
Selected decoding adds seek and run-processing overhead, so applying it to every filtered page can regress dense or fragmented selections. Masking a page after choosing a full read can also change downstream block representation without avoiding any reader work. Accept an engine selection only when at least one unloaded column will use row- or page-level skipping. Choose those modes only when encoding, physical type, null statistics, selectivity, run geometry, and confirmed skipped page bytes predict a meaningful gain. Otherwise leave the source page unchanged so projections retain the existing compact path.
The selection policy spans several cost regimes and cannot be justified by a single type or selectivity measurement. Provide a Zstandard benchmark covering encoding, value width, null layout, selection locality, page geometry, and projected column count. Compare full reads, forced selection, and the production decision while exposing page and policy counters.
Currently, `concat(a_char, a_char, a_char...)` works. It works by coercing `char` values to `varchar` (trimming trailing spaces), and so is not consistent with `concat(a_char, a_char)` which retains trailing spaces. If we were to act upon the TODO being removed, we would need to add implement `concat(a_char, a_char, a_char...)` consistently with the 2-arg version, so preserving the trailing spaces. This would be semantically breaking change though.
Bumps [com.azure:azure-core-tracing-opentelemetry](https://github.com/Azure/azure-sdk-for-java) from 1.0.0-beta.65 to 1.0.0-beta.66. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-experimental_1.0.0-beta.65...com.azure+azure-core-experimental_1.0.0-beta.66) --- updated-dependencies: - dependency-name: com.azure:azure-core-tracing-opentelemetry dependency-version: 1.0.0-beta.66 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
bill-ph
approved these changes
Sep 7, 2026
bill-ph
left a comment
bill-ph
left a comment
There was a problem hiding this comment.
Reviewed the current head and upstream merge scope. No P0 blockers requiring code changes found. CI is still in progress with no failures reported so far.
— Robo Bill
benben
approved these changes
Sep 7, 2026
benben
left a comment
benben
left a comment
Member
There was a problem hiding this comment.
Automated review generated on behalf of @benben.
Approve. Fast-forward style merge of 18 upstream trinodb/trino commits (Parquet selected-positions pushdown, page skipping, TopN retained-size accounting, query.max-output-data-size, dependency bumps) into the PostHog ducklake fork with no conflict resolutions. Fork-specific code (ducklake plugin, ColumnReaderFactory MILLIS decoder, ParquetPageSourceFactory ColumnsForFile) is untouched by upstream, and the ducklake positional-delete filter composes correctly with the new ParquetSourcePage.trySelectPositions path.
- minor
plugin/trino-ducklake/src/main/java/io/trino/plugin/ducklake/DuckLakePageSourceProvider.java:511Delete filter now composes with new engine position pushdown; untested in fork. PR only compiled the module; DELETE followed by filtered SELECT on the same file exercises the new ParquetSourcePage.trySelectPositions composition.
Model: fable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge current
trinodb/trinomaster into the PostHog ducklake fork.Sanity build:
Result: BUILD SUCCESS.