Skip to content

EXPERIMENT 2: Prefetching parquet byte ranges into pinned host memory and use cucascade to fill those buffers - #23450

Draft
Matt711 wants to merge 72 commits into
rapidsai:mainfrom
Matt711:fea/polars/hybrid-scan-cucascade-pinned
Draft

EXPERIMENT 2: Prefetching parquet byte ranges into pinned host memory and use cucascade to fill those buffers#23450
Matt711 wants to merge 72 commits into
rapidsai:mainfrom
Matt711:fea/polars/hybrid-scan-cucascade-pinned

Conversation

@Matt711

@Matt711 Matt711 commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Matt711 and others added 30 commits June 10, 2026 16:12
@Matt711
Add FusedScan and nxtx annotations for FusedScan and SplitScan
9db0749
@Matt711
Have both streaming scan class take paths
b6bc1c1
@Matt711
missed
c30e488
@Matt711
address review, add absolute path prefix to nvtx annotation
d3f4086
@Matt711
Dispatch SplitScan to HybridScanReader in the streaming engine
9f805ee
@Matt711
docs and cython boilerplate
cf44510
@Matt711
clean up from mixed branch
b7ecc32
@Matt711
remove stream sync
4e53ba5
@Matt711
remove newline
17682d2
@Matt711
add changes back from mixed branch
ff366ed
@Matt711
pre-commit check
9820545
@Matt711
mypy fix
9e9ead8
@Matt711
prepare hybrid-scan path for future async-prefetching pinned host mem…
d979010 
…ory datasource
@Matt711
use SplitScan for SINGLE_FILE even if too small so we can use hybrid …
64142d0 
…scan, see rapidsai#22857 (comment)
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
78b3ac9
@Matt711
prune more rows if page index bytes, release gil in hybrid scan cytho…
6627c55 
…n APIs
@Matt711
merge conflict
af66194
@Matt711
add C++ API read for cudf_polars to avoid repeated GIL acqusition/rel…
106d6b5 
…ease
@Matt711
remove page pruning and leave TODO
9b57816
@Matt711
share file metadata without copying when contructing hybrid scan readers
8d92fb0
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
53694ef
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
78e1428
@Matt711
pre-commit check
7f2cba1
@Matt711
merge conflict, code coverage, docs
74c3d7d
@Matt711
merge conflict
00e96c6
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
a027ad9
@Matt711
copyright
4bf25a8
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
258c008
@Matt711
Merge branch 'main' into fea/polars/hybrid-scan
b0be1d1
@Matt711
docs
2362e1d
@Matt711 Matt711 added the DO NOT MERGE Hold off on merging; see PR for details label Jul 27, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. CMake CMake build issue cudf-polars Issues specific to cudf-polars pylibcudf Issues specific to the pylibcudf package labels Jul 27, 2026
@Matt711 Matt711 mentioned this pull request Jul 27, 2026
Draft
@GPUtester GPUtester moved this to In Progress in cuDF Python Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@Matt711 Matt711

Labels

CMake CMake build issue cudf-polars Issues specific to cudf-polars DO NOT MERGE Hold off on merging; see PR for details libcudf Affects libcudf (C++/CUDA) code. pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.

Projects

cuDF Python
Status: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Matt711 @GPUtester