Skip to content

feat: Support Spark expression str_to_map#3654

Merged
comphead merged 4 commits intoapache:mainfrom
unknowntpo:feat-str-to-map-on-main
Apr 29, 2026
Merged

feat: Support Spark expression str_to_map#3654
comphead merged 4 commits intoapache:mainfrom
unknowntpo:feat-str-to-map-on-main

Conversation

@unknowntpo
Copy link
Copy Markdown
Contributor

@unknowntpo unknowntpo commented Mar 10, 2026

Which issue does this PR close?

Closes #3168

Rationale for this change

Comet does not yet support Spark StringToMap, so this change wires it through serde and native execution.

What changes are included in this PR?

  • wire Spark StringToMap to the native SparkStrToMap UDF
  • register SparkStrToMap in the native function registry
  • add SQL file coverage for str_to_map

How are these changes tested?

make test-jvm \
  PROFILES="-Drat.skip=true -Dmaven.gitcommitid.skip=true -Dsuites=org.apache.comet.CometSqlFileTestSuite" \
  MAVEN_OPTS="-Dmaven.gitcommitid.skip=true -Drat.skip=true"

@unknowntpo unknowntpo changed the title feat Support Spark expression str_to_map feat: Support Spark expression str_to_map Mar 10, 2026
@unknowntpo unknowntpo changed the title feat: Support Spark expression str_to_map feat: Support Spark expression str_to_map Mar 10, 2026
@unknowntpo unknowntpo force-pushed the feat-str-to-map-on-main branch from d56a2f2 to 39e444b Compare April 18, 2026 23:40
@unknowntpo unknowntpo marked this pull request as ready for review April 19, 2026 04:43
@unknowntpo unknowntpo force-pushed the feat-str-to-map-on-main branch from 39e444b to 70f7686 Compare April 19, 2026 04:46
SELECT str_to_map(col) FROM (VALUES ('a:1,b:2'), ('x:9'), (NULL)) AS t(col)

-- Multi-row with custom delimiter
query spark_answer_only
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to check Comet enabled we need to remove spark_answer_only for all possible queries

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @unknowntpo for your contribution, I think to prove it works we need to run test queries with just query to let them run with Comet and also it would be nice to support all datatypes

@unknowntpo
Copy link
Copy Markdown
Contributor Author

@comphead Thanks for the review. I updated the SQL tests to use query instead of query spark_answer_only where applicable, so these cases run with Comet as well.

I also added table-column coverage for the string-like input types accepted by str_to_map: STRING, CHAR, and VARCHAR, including cases where the delimiters come from columns.

@comphead
Copy link
Copy Markdown
Contributor

Thanks @unknowntpo please add the function to docs/source/user-guide/latest/expressions.md

@unknowntpo unknowntpo force-pushed the feat-str-to-map-on-main branch from 5587788 to 21ecbde Compare April 29, 2026 01:14
@unknowntpo
Copy link
Copy Markdown
Contributor Author

Thanks @unknowntpo please add the function to docs/source/user-guide/latest/expressions.md

Okay, updated.

Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @unknowntpo for your contribution, I think it is good to go

@comphead comphead merged commit f10613b into apache:main Apr 29, 2026
196 of 199 checks passed
@unknowntpo unknowntpo deleted the feat-str-to-map-on-main branch April 30, 2026 00:56
@unknowntpo
Copy link
Copy Markdown
Contributor Author

Thanks @unknowntpo for your contribution, I think it is good to go

Thanks @comphead for reviewing my code !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support Spark expression: string_to_map

2 participants