Skip to content

feat: Support Spark expression window_time #3732

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
0lai0 wants to merge 10 commits into
base: main
Choose a base branch
from
+85 −28
Draft

feat: Support Spark expression window_time #3732

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
0d2f579
feat: Support Spark expression window time
0lai0 Mar 19, 2026
9046b80
add SQL test and fix error
0lai0 Mar 20, 2026
1e4201a
Simplify annotations
0lai0 Mar 20, 2026
83f83ab
Apply suggestion from @andygrove
0lai0 Mar 20, 2026
9a618d8
Merge branch 'main' into expression_window_time
0lai0 Mar 20, 2026
4bd328b
add list of supported expressions to expressions.md and add test
0lai0 Mar 20, 2026
041008c
Merge branch 'main' into expression_window_time
0lai0 Mar 26, 2026
dfb3c7d
fix error
0lai0 Mar 26, 2026
d425ec4
Merge branch 'main' into expression_window_time
0lai0 Apr 14, 2026
56c39f2
Merge remote-tracking branch 'upstream/main' into expression_window_time
0lai0 May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 27 additions & 26 deletions docs/source/user-guide/latest/expressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,32 +94,33 @@ of expressions that be disabled.

## Date/Time Functions

| Expression | SQL |
| -------------- | ---------------------------- |
| DateAdd | `date_add` |
| DateDiff | `datediff` |
| DateFormat | `date_format` |
| DateSub | `date_sub` |
| DatePart | `date_part(field, source)` |
| Days | `days` |
| Extract | `extract(field FROM source)` |
| FromUnixTime | `from_unixtime` |
| Hour | `hour` |
| LastDay | `last_day` |
| Minute | `minute` |
| Second | `second` |
| TruncDate | `trunc` |
| TruncTimestamp | `date_trunc` |
| UnixDate | `unix_date` |
| UnixTimestamp | `unix_timestamp` |
| Year | `year` |
| Month | `month` |
| DayOfMonth | `day`/`dayofmonth` |
| DayOfWeek | `dayofweek` |
| WeekDay | `weekday` |
| DayOfYear | `dayofyear` |
| WeekOfYear | `weekofyear` |
| Quarter | `quarter` |
| Expression | SQL |
| -------------------------- | ---------------------------- |
| DateAdd | `date_add` |
| DateDiff | `datediff` |
| DateFormat | `date_format` |
| DateSub | `date_sub` |
| DatePart | `date_part(field, source)` |
| Days | `days` |
| Extract | `extract(field FROM source)` |
| FromUnixTime | `from_unixtime` |
| Hour | `hour` |
| LastDay | `last_day` |
| Minute | `minute` |
| Second | `second` |
| TruncDate | `trunc` |
| TruncTimestamp | `date_trunc` |
| UnixDate | `unix_date` |
| UnixTimestamp | `unix_timestamp` |
| Year | `year` |
| Month | `month` |
| DayOfMonth | `day`/`dayofmonth` |
| DayOfWeek | `dayofweek` |
| WeekDay | `weekday` |
| DayOfYear | `dayofyear` |
| WeekOfYear | `weekofyear` |
| Quarter | `quarter` |
| PreciseTimestampConversion | `window_time` |

## Math Expressions

Expand Down
3 changes: 2 additions & 1 deletion spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,8 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
classOf[WeekDay] -> CometWeekDay,
classOf[DayOfYear] -> CometDayOfYear,
classOf[WeekOfYear] -> CometWeekOfYear,
classOf[Quarter] -> CometQuarter)
classOf[Quarter] -> CometQuarter,
classOf[PreciseTimestampConversion] -> CometPreciseTimestampConversion)

private val conversionExpressions: Map[Class[_ <: Expression], CometExpressionSerde[_]] = Map(
classOf[Cast] -> CometCast)
Expand Down
24 changes: 23 additions & 1 deletion spark/src/main/scala/org/apache/comet/serde/datetime.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ package org.apache.comet.serde

import java.util.Locale

import org.apache.spark.sql.catalyst.expressions.{Attribute, DateAdd, DateDiff, DateFormatClass, DateFromUnixDate, DateSub, DayOfMonth, DayOfWeek, DayOfYear, Days, GetDateField, Hour, Hours, LastDay, Literal, MakeDate, Minute, Month, NextDay, Quarter, Second, SecondsToTimestamp, TruncDate, TruncTimestamp, UnixDate, UnixTimestamp, WeekDay, WeekOfYear, Year}
import org.apache.spark.sql.catalyst.expressions.{Attribute, DateAdd, DateDiff, DateFormatClass, DateFromUnixDate, DateSub, DayOfMonth, DayOfWeek, DayOfYear, Days, GetDateField, Hour, Hours, LastDay, Literal, MakeDate, Minute, Month, NextDay, PreciseTimestampConversion, Quarter, Second, SecondsToTimestamp, TruncDate, TruncTimestamp, UnixDate, UnixTimestamp, WeekDay, WeekOfYear, Year}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DateType, DoubleType, FloatType, IntegerType, LongType, StringType, TimestampNTZType, TimestampType}
import org.apache.spark.unsafe.types.UTF8String
Expand Down Expand Up @@ -635,6 +635,28 @@ object CometDateFormat extends CometExpressionSerde[DateFormatClass] {
}
}

object CometPreciseTimestampConversion extends CometExpressionSerde[PreciseTimestampConversion] {
override def getUnsupportedReasons(): Seq[String] =
Seq("Only conversions between TimestampType and LongType are supported")

override def getSupportLevel(expr: PreciseTimestampConversion): SupportLevel = {
(expr.fromType, expr.toType) match {
case (TimestampType, LongType) | (LongType, TimestampType) =>
Compatible()
case _ =>
Unsupported(Some(s"PreciseTimestampConversion from ${expr.fromType} to ${expr.toType}"))
}
}

override def convert(
expr: PreciseTimestampConversion,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
// Both types are i64 micros in Arrow, so no conversion is needed.
exprToProtoInternal(expr.child, inputs, binding)
}
}

/**
* Converts a timestamp to the number of hours since Unix epoch (1970-01-01 00:00:00 UTC). This is
* a V2 partition transform expression.
Expand Down
33 changes: 33 additions & 0 deletions spark/src/test/resources/sql-tests/expressions/datetime/window_time.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.


statement
CREATE TABLE test_window_time(time timestamp, value int) USING parquet

statement
INSERT INTO test_window_time VALUES (timestamp('2023-01-01 12:00:00'), 1), (timestamp('2023-01-01 12:05:00'), 2), (timestamp('2023-01-01 12:15:00'), 3), (NULL, 4)

-- spark_answer_only: window() is expanded by Spark with unsupported KnownNullable

-- basic window_time with tumbling window
query spark_answer_only
Copy link
Copy Markdown
Member

@andygrove andygrove Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both queries use spark_answer_only. Is there a reason these can't run with the default mode that verifies native execution? If there's another operator in the plan that causes fallback, could you note which one?

Copy link
Copy Markdown
Contributor Author

@0lai0 0lai0 Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove for review.
While this PR supports native execution, Spark's window() introduces unsupported ops that trigger a full fallback. Therefore, we temporarily use spark_answer_only for result verification. I'll add an inline comment to document this.
If I have misunderstood, please correct me.

Copy link
Copy Markdown
Member

@andygrove andygrove Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, windowed aggregates are marked as incompatible. Maybe you can enable in the test with spark.comet.operator.WindowExec.allowIncompatible=true? You may need to check exact config key but it should be something like this.

Copy link
Copy Markdown
Contributor Author

@0lai0 0lai0 Mar 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove . I dug into this and tried enabling the config spark.comet.operator.WindowExec.allowIncompatible=true, but the test still fails if I remove spark_answer_only.

The root cause might be at the expression level: Spark's TimeWindowing analyzer rule automatically expands window() into a CreateNamedStruct wrapped inside a KnownNullable expression. Since Comet doesn't currently support KnownNullable natively, the planner is forced to fall back to Spark anyway.
https://github.com/search?q=repo%3Aapache%2Fspark+KnownNullable&type=code

Fully supporting native struct creation and KnownNullable would be a broader effort outside the scope of this PR (which focuses on PreciseTimestampConversion issue for window_time).

Because of this, it might be better to keep spark_answer_only here and open a follow-up issue for those specific expressions. I'll make sure to update the inline comment to explicitly document the exact cause of the fallback for future reference.
Let me know your thoughts on this.

SELECT max(window_time(window)), sum(value) FROM (SELECT window(time, '10 minutes') AS window, value FROM test_window_time) GROUP BY window

-- window_time with sliding window
query spark_answer_only
SELECT max(window_time(window)), count(value) FROM (SELECT window(time, '10 minutes', '5 minutes') AS window, value FROM test_window_time) GROUP BY window
Loading