ci: Enable Comet PR test matrix and TPCDS plan-stability for Spark 4.2#4126
Merged
andygrove merged 3 commits intoapache:mainfrom Apr 30, 2026
Merged
ci: Enable Comet PR test matrix and TPCDS plan-stability for Spark 4.2#4126andygrove merged 3 commits intoapache:mainfrom
andygrove merged 3 commits intoapache:mainfrom
Conversation
andygrove
changed the title
andygrove
changed the title
andygrove
commented
Apr 29, 2026
| func_might_contain, | ||
| new ExpressionInfo(classOf[BloomFilterMightContain].getName, "might_contain"), | ||
| (children: Seq[Expression]) => BloomFilterMightContain(children.head, children(1))) | ||
| if (!isSpark42Plus) { |
Member
Author
There was a problem hiding this comment.
this whole suite failed with Spark 4.2 - the individual tests have the links to the issue
coderfender
approved these changes
Apr 30, 2026
Contributor
coderfender
left a comment
coderfender
left a comment
There was a problem hiding this comment.
LGTM with nit to see why we need isSpark41Plus and isSpark42Plus tags in the same test ?
Adds a Spark 4.2, JDK 17 entry to the linux-test matrix in pr_build_linux.yml and a Spark 4.2, JDK 17, Scala 2.13 entry to the macos-aarch64-test matrix in pr_build_macos.yml. Also wires up the iceberg/jetty test dependencies for the spark-4.2 profile (matching the spark-4.1 setup, since iceberg-spark-runtime 4.2 is not yet published) and adds an isSpark42Plus helper. Spark 4.2 stays out of lint-java because semanticdb-scalac_2.13.18 is not yet published.
Add isSpark42Plus branch to CometPlanStabilitySuite to route Spark 4.2 to
dedicated approved-plans-{v1_4,v2_7}-spark4_2 directories. Spark 4.0 logic is
unchanged. Also extends dev/regenerate-golden-files.sh to accept --spark-version
4.2.
Generated via SPARK_GENERATE_GOLDEN_FILES=1 against -Pspark-4.2. 22 of the
generated files differ from the spark4_0 directory (q2, q5, q33, q49, q54,
q56, q60, q66 in v1_4 and q5a, q14a, q49 in v2_7, both native_datafusion and
native_iceberg_compat); the rest are byte-identical. The CometTPCDSV1_4
suite (194 tests) and CometTPCDSV2_7 suite (64 tests) both pass against the
new goldens with 0 failures.
Skip tests that fail on Spark 4.2 due to:
- FunctionIdentifier requiring 3-part qualification (might_contain)
- ARITHMETIC_OVERFLOW message change ("overflow" vs "integer overflow")
- Jetty VerifyError in REST catalog test (classpath version mismatch)
andygrove
force-pushed
the
spark-4.2-tests
branch
from
a303f80 to
7754ee3
Compare
Member
Author
|
Merged. Thanks @coderfender! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #4113.
Rationale for this change
#4119 added a build-only
spark-4.2Maven profile targeting Spark 4.2.0-preview4. To start exercising Comet against 4.2 in CI (rather than discovering everything at once when 4.2 GA lands), this PR turns on the existing PR test matrices for Spark 4.2 and adds dedicated TPC-DS plan-stability goldens.This mirrors the approach previously used to bring Spark 4.1 online before reverting (see commits
622e851e1and75e3b3116on thespark-4.1.1branch).What changes are included in this PR?
.github/workflows/pr_build_linux.yml: addSpark 4.2, JDK 17to thelinux-testmatrix and a comment explaining why 4.1/4.2 are skipped from thelint-javamatrix (semanticdb-scalac is not yet published for Scala 2.13.17/2.13.18)..github/workflows/pr_build_macos.yml: addSpark 4.2, JDK 17, Scala 2.13to themacos-aarch64-testmatrix.spark/pom.xml: wire iceberg/jetty test dependencies into thespark-4.2profile (Iceberg falls back to the 4.0 runtime since 4.2 is not yet published; Jetty pinned at 11.0.26).spark/src/main/scala/org/apache/comet/CometSparkSessionExtensions.scala: addisSpark42Plushelper.spark/src/test/scala/org/apache/spark/sql/comet/CometPlanStabilitySuite.scala: routeisSpark42Plusto the newapproved-plans-{v1_4,v2_7}-spark4_2directories.dev/regenerate-golden-files.sh: accept--spark-version 4.2and include 4.2 in the default version list.spark/src/test/resources/tpcds-plan-stability/approved-plans-{v1_4,v2_7}-spark4_2/: regenerated golden files. 22 of the generated files differ from thespark4_0directory (q2,q5,q33,q49,q54,q56,q60,q66in v1_4 andq5a,q14a,q49in v2_7, bothnative_datafusionandnative_iceberg_compatper query); the rest are byte-identical.This PR does not attempt to fix any 4.2-specific runtime/test failures the new matrix entries surface; those will be tracked and addressed in follow-up PRs as we did for Spark 4.1.
How are these changes tested?
-Pspark-4.2end-to-end with JDK 17.CometTPCDSV1_4_PlanStabilitySuite(194 tests) andCometTPCDSV2_7_PlanStabilitySuite(64 tests) against-Pspark-4.2withSPARK_GENERATE_GOLDEN_FILESunset; both pass with 0 failures.