Skip to content

docs: check off 53 implemented expressions in support doc#4147

Open
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:audit-expressions-doc
Open

docs: check off 53 implemented expressions in support doc#4147
andygrove wants to merge 1 commit intoapache:mainfrom
andygrove:audit-expressions-doc

Conversation

@andygrove
Copy link
Copy Markdown
Member

Summary

  • Audited the codebase against spark_expressions_support.md and marked 53 expressions that have full end-to-end implementations (JVM serde + protobuf + native execution) as supported
  • All verified against classOf[...] -> mappings in QueryPlanSerde.scala

Expressions checked off by category:

Category Count Expressions
datetime 24 date_add, date_diff, date_format, date_sub, date_trunc, dateadd, datediff, day, dayofmonth, dayofweek, dayofyear, hour, last_day, make_date, minute, month, next_day, quarter, second, trunc, unix_date, weekday, weekofyear, year
math 8 cosh, cot, expm1, hex, rand, randn, sinh, tanh
string 6 left, regexp_replace, right, split, substr, substring
hash 4 hash, sha1, sha2, xxhash64
array 2 array_min, flatten
bitwise 2 bit_count, bit_get
collection 2 concat, size
predicate 2 isnan, rlike
map 1 str_to_map
struct 1 named_struct
conversion 1 cast

Test plan

  • Doc-only change, no tests needed
  • Verified each expression has a classOf[...] -> Comet... mapping in QueryPlanSerde.scala

🤖 Generated with Claude Code

…rt.md

Audit the codebase against the expressions support doc and mark
expressions that have full end-to-end implementations (JVM serde +
protobuf + native execution) as supported. All verified against
classOf mappings in QueryPlanSerde.scala.

Expressions checked off by category:
- datetime (24): date_add, date_diff, date_format, date_sub, date_trunc,
  dateadd, datediff, day, dayofmonth, dayofweek, dayofyear, hour, last_day,
  make_date, minute, month, next_day, quarter, second, trunc, unix_date,
  weekday, weekofyear, year
- math (8): cosh, cot, expm1, hex, rand, randn, sinh, tanh
- string (6): left, regexp_replace, right, split, substr, substring
- hash (4): hash, sha1, sha2, xxhash64
- array (2): array_min, flatten
- bitwise (2): bit_count, bit_get
- collection (2): concat, size
- predicate (2): isnan, rlike
- map (1): str_to_map
- struct (1): named_struct
- conversion (1): cast
@andygrove andygrove requested a review from comphead April 29, 2026 22:04
Copy link
Copy Markdown
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove let me some time to prove:

  • we have sql test for each of them
  • test run with query mode

@comphead
Copy link
Copy Markdown
Contributor

# Function SQL file(s) #query #query spark_answer_only
1 date_add datetime/date_add.sql 4 0
2 date_diff datetime/date_diff.sql 4 0
3 date_format datetime/date_format.sql 0 0
4 date_sub datetime/date_sub.sql 4 0
5 date_trunc datetime/trunc_timestamp.sql (+ _dst, _dst_ambiguous) 5 0
6 dateadd ❌ missing
7 datediff datetime/date_diff.sql (shared) 4 0
8 day datetime/datetime.sql (shared) 3 0
9 dayofmonth ❌ missing
10 dayofweek datetime/datetime.sql (shared) 3 0
11 dayofyear datetime/datetime.sql (shared) 3 0
12 hour datetime/hour.sql 1 0
13 last_day datetime/last_day.sql 2 0
14 make_date datetime/make_date.sql 43 0
15 minute datetime/minute.sql 1 0
16 month datetime/datetime.sql (shared) 3 0
17 next_day datetime/next_day.sql 16 0
18 quarter datetime/datetime.sql (shared) 3 0
19 second datetime/second.sql 1 0
20 trunc datetime/trunc_date.sql 4 0
21 unix_date datetime/unix_date.sql 0 2
22 weekday datetime/datetime.sql (shared) 3 0
23 weekofyear datetime/datetime.sql (shared) 3 0
24 year datetime/datetime.sql (shared) 3 0
25 cosh math/cosh.sql 0 0
26 cot math/cot.sql 0 0
27 expm1 math/expm1.sql 0 0
28 hex string/hex.sql 2 0
29 rand ❌ missing
30 randn ❌ missing
31 sinh math/sinh.sql 0 0
32 tanh math/tanh.sql 0 0
33 left string/left.sql 8 0
34 regexp_replace string/regexp_replace.sql + regexp_replace_enabled.sql 3 0
35 right string/right.sql 0 15
36 split ❌ missing
37 substr string/substring.sql (shared) 47 0
38 substring string/substring.sql (shared) 47 0
39 hash hash/hash.sql (shared) 1 0
40 sha1 hash/hash.sql (shared) 1 0
41 sha2 hash/hash.sql (shared) 1 0
42 xxhash64 hash/hash.sql (shared) 1 0
43 array_min array/array_min.sql 1 1
44 flatten array/flatten.sql 0 2
45 bit_count bitwise/bitwise.sql (shared) 9 2
46 bit_get bitwise/bitwise.sql (shared) 9 2
47 concat string/concat.sql 8 0
48 size array/size.sql 1 1
49 isnan math/isnan.sql 2 0
50 rlike string/rlike.sql + rlike_enabled.sql 4 1
51 str_to_map map/str_to_map.sql 15 0
52 named_struct struct/create_named_struct.sql 4 0
53 cast cast/cast.sql 23 0

@comphead
Copy link
Copy Markdown
Contributor

I'll create another PR to modify where spark_answer_only is stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants