Skip to content

feat(parquet): sub-column projection pushdown for struct fields#22059

Open
EshwarCVS wants to merge 3 commits intoapache:mainfrom
EshwarCVS:feat/struct-projection-pushdown
Open

feat(parquet): sub-column projection pushdown for struct fields#22059
EshwarCVS wants to merge 3 commits intoapache:mainfrom
EshwarCVS:feat/struct-projection-pushdown

Conversation

@EshwarCVS
Copy link
Copy Markdown

Which issue does this PR close?

Closes #

Rationale for this change

When a query accesses only a single field of a wide struct column (e.g.
SELECT event['user_id'] FROM logs), DataFusion today reads all leaf
columns of event from Parquet. For structs with many fields this is
significant unnecessary I/O.

What changes are included in this PR?

Logical optimizer (2-pass pipeline)

  • ExtractLeafExpressions (pass 1): detects MoveTowardsLeafNodes
    sub-expressions (including get_field) inside Filter, Sort, Limit,
    Aggregate, and Join nodes and lifts them into named extraction
    projections (__datafusion_extracted_N) inserted below those nodes.
  • PushDownLeafProjections (pass 2): pushes those extraction
    projections further down toward leaf/datasource nodes, merging into
    existing projections where possible and routing each expression to the
    correct input side of multi-input nodes (Join, Union).

Physical layer — Parquet leaf-column projection

  • PushdownChecker / StructFieldAccess: extended the filter
    pushdown visitor to recognise get_field(Column, "field1", "field2", …)
    patterns and record a StructFieldAccess { root_index, field_path }
    instead of requiring the entire struct column.
  • resolve_struct_field_leaves: maps each StructFieldAccess to the
    exact Parquet leaf column indices by prefix-matching against
    SchemaDescriptor, enabling ProjectionMask::leaves() instead of
    ProjectionMask::roots().
  • build_filter_schema / prune_struct_type: constructs a narrowed
    Arrow schema that matches what the Parquet reader actually produces when
    projecting specific struct leaves, so reassign_expr_columns can
    correctly remap filter expressions.
  • build_projection_read_plan: unified entry point (used by
    opener.rs) that builds a leaf-level ProjectionMask from the physical
    projection expressions, with fast paths for all-plain-column and
    no-struct-column schemas.

Physical optimizer wiring

remove_unnecessary_projections / try_swapping_with_projection already
merges ProjectionExec nodes (including those containing get_field)
into DataSourceExec via ParquetSource::try_pushdown_projection
try_merge. The new physical-layer code ensures the merged expressions
are then used to build a narrow ProjectionMask.

How are these changes tested?

  • Unit tests in row_filter.rs: 10 new tests covering
    get_field pushdown allowance/denial, correct Parquet leaf index
    selection for simple and deeply-nested structs, end-to-end row
    filtering, and the projection-preserves-full-struct invariant.
  • Optimizer unit tests in extract_leaf_expressions.rs: 47 tests
    covering extraction from Filter/Sort/Limit/Aggregate/Join/Union/
    SubqueryAlias, deduplication, CSE interaction, and recovery projection
    correctness.
  • SQL logic tests in projection_pushdown.slt: end-to-end queries
    covering basic field access, filter pushdown, sort/TopK, multi-partition,
    joins, aggregation, nullable structs, SELECT * with struct field
    filters, and edge cases (Map columns, non-literal field names).

claude and others added 3 commits May 7, 2026 04:49
Spark's make_interval nullability rule mirrors `failOnError`:
- nullary call → not nullable (always returns zero interval)
- ANSI mode on (failOnError=true) → nullable only when any input is nullable
- ANSI mode off (default) → always nullable (overflow silently returns NULL)

Previous PRs (apache#19248, apache#19606) could not implement this correctly because
ReturnFieldArgs carries no ConfigOptions. The fix follows the SparkCast
pattern: store `ansi_mode` as a struct field, populate it via
`with_updated_config`, and use `self.ansi_mode` in `return_field_from_args`.

In ANSI mode, arithmetic overflow in make_interval_kernel now returns an
error instead of silently producing NULL, matching Spark's behavior.

Closes apache#19155
fix(spark): implement ANSI-aware custom nullability for make_interval
- Fix typo: "strinrg" → "string" in get_field arg resolution comment
- Remove duplicate comment block in resolve_struct_field_leaves
- Fix stale doc reference to removed PrimitiveOnly fast-path
- Fix minor typo: "all3" → "all 3" in test comment

https://claude.ai/code/session_01RskgheL63G1MpXnLxWZ71K
@github-actions github-actions Bot added datasource Changes to the datasource crate spark labels May 7, 2026
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't belong in this PR

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will update by removing those changes.

@kumarUjjawal
Copy link
Copy Markdown
Contributor

I don't see the point of this PR since it doesn't do any work which are mentioned in the PR body and the one change it has is unrelated to the body. Please open issue related to what you have mentioned in the body first.

@kumarUjjawal
Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate spark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants