Forward-merge release/26.10 into main - #24106
Closed
rapids-bot[bot] wants to merge 5 commits into
Closed
Conversation
…ely (#23584) In #22996 and #22997 we added join filter pushdown optimisations for cudf-polars plans. These are represented as semi joins using the existing IR structure. As such, we are on the hook to execute them, even if that would not be beneficial for runtime execution. Examples are cases where the join that is being filtered will be performed via broadcast, or is already compatibly shuffled. In such cases carrying out the semi join merely adds extra work. To fix this, introduce a special `PushdownFilterHint` node that is, optionally, applied at runtime. To decide whether, and how, to apply these hints we now sample join inputs and estimate cardinality in addition to size. We use the cardinality estimate to decide whether or not a bloom filter would be effective: calculating the estimated false positive rate. During lowering, we build a join planning state that is updated at runtime with cardinality estimates and other relevant information such as partitioning. When we then come to execute a filter hint we can inspect this state and take an appropriate action. Filter hints are classified into two types: 1. Those that apply directly to a join input `Join(Hint(target, domain), domain)` 2. Those that apply indirectly: `Join1(Join2(Hint(target, j1_domain), j2_domain), j1_domain)` The former check at runtime whether or not the domain is already compatibly distributed (or will be broadcast) and then elide the hint. The latter cannot do so without inducing a cycle in the execution DAG, which is undesirable. We therefore always apply these "indirect" hints, but still only if the hint does not induce a shuffle. Better cost models for placement might elide some of these, or give us enough information to decide whether to reject them based on sampling. - Closes #23576 Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Tom Augspurger (https://github.com/TomAugspurger) - Peter Andreas Entschev (https://github.com/pentschev) URL: #23584
Contributor
Author
|
FAILURE - Unable to forward-merge due to an error, manual merge is necessary. Do not use the IMPORTANT: When merging this PR, do not use the auto-merger (i.e. the |
github-actions
Bot
added
Python
Now that hosting of cudf and related docs is on docs.nvidia.com, update the internal docs links to point there where possible. This is done in a few ways: - Explicit URLs are written out; - The intersphinx inventory is expanded and cross-project linking in the docs is now exclusively via intersphinx; - The libcudf C++ developer guide is included in the sphinx documentation. Along the way, I had to make a bunch of small changes to fix some minor issues in our internal cross-linking. Links to rapids.ai pages that are not yet migrated are left as is. Partially addresses #23917, but does not cull all RAPIDS occurrences yet. Authors: - Lawrence Mitchell (https://github.com/wence-) Approvers: - Bradley Dice (https://github.com/bdice) - Vyas Ramasubramani (https://github.com/vyasr) URL: #23971
rapids-bot
Bot
requested review from
GaryShen2008,
lamarrr,
msarahan and
wence-
September 10, 2026 10:14
github-actions
Bot
added
libcudf
Removing the `-Wno-error=deprecated-declarations` compile flag so all warnings are errors in C++ builds. Authors: - David Wendt (https://github.com/davidwendt) Approvers: - Nghia Truong (https://github.com/ttnghia) URL: #23985
Closes #23428 This PR makes `streaming_groupby::aggregate()` safe to call concurrently from multiple host threads on one instance, each with its own stream, without caller-side serialization. Newly discovered keys live in the hash set under a transient encoding, `max_distinct_keys + row_idx`, that carries no batch identifier, so overlapping insertions decode each other's values against the wrong batch table. The insertion phase also mutates shared host state that must stay in lockstep: the batch ID, the dense ID base, and the retained key batches. Rather than widen the encoding to carry a batch ID, a mutex serializes the insertion phase and a CUDA event orders it across calls on different streams. The aggregation that follows stays outside the lock, since it updates every group through `cudf::detail::atomic_add`/`atomic_min`/`atomic_max`. `_distinct_keys` becomes `std::atomic` because concurrent `aggregate()` otherwise races the read in `distinct_keys()`. `merge()` shares `probe_and_insert` and takes the same lock. Insertion kernels from different batches still do not overlap, which #23428 allows for a first pass, and `thrust::copy_if` inside insertion already forces a host-visible sync. Authors: - Yunsong Wang (https://github.com/PointKernel) Approvers: - Basit Ayantunde (https://github.com/lamarrr) - Nghia Truong (https://github.com/ttnghia) - Tianyu Liu (https://github.com/kingcrimsontianyu) URL: #23884
… SPMD engines (#23836) Turn on pinned host memory by default for cudf-polars' Ray, Dask, and SPMD engines, with the pinned memory pool starting at zero bytes. Authors: - Matthew Murray (https://github.com/Matt711) Approvers: - Peter Andreas Entschev (https://github.com/pentschev) URL: #23836
3 tasks
Contributor
|
I'm going to close this PR and let the bot recreate it so that it's less confused after we merged the manual #24107. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/26.10 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.