-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Use BooleanBufferBuilder rather than Vec<bool> in ArrowBytesViewMap #20064
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Use BooleanBufferBuilder rather than Vec<bool> in ArrowBytesViewMap #20064
Conversation
|
Thanks for the review! |
|
run benchmarks |
|
🤖 |
|
Thanks for the review and suggestions! |
Co-authored-by: Daniël Heres <danielheres@gmail.com>
|
🤖: Benchmark completed Details
|
alamb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @etk18 and @Dandandan -- this looks cool -- Using NullBufferBuilder will be even better. Thank you
|
Thanks for the reviews! I’ve updated the PR to use Happy to rebase or make further adjustments if needed. |
|
Thanks for the review! I’ve pushed updates to:
CI is rerunning now. Please let me know if anything else is needed. |
|
run benchmarks |
|
🤖 |
|
🤖: Benchmark completed Details
|
Which issue does this PR close?
Closes #20053
Rationale for this change
ArrowBytesViewMappreviously usedVec<bool>to track null values.This PR replaces it with
BooleanBufferBuilder, which is significantly morememory-efficient and faster, and aligns with Apache Arrow best practices for
building validity bitmaps.
This change improves performance and memory usage without changing behavior.
What changed
Vec<bool>withBooleanBufferBuilderfor null trackingTests
cargo test -p datafusion-physical-expr-commonScreen.Recording.2026-01-29.at.8.34.41.PM.mov