test: skip flaky StateStoreSuite under Comet and disambiguate JDK matrix names#4226
Merged
mbutrovich merged 1 commit intoapache:mainfrom May 5, 2026
Merged
test: skip flaky StateStoreSuite under Comet and disambiguate JDK matrix names#4226mbutrovich merged 1 commit intoapache:mainfrom
mbutrovich merged 1 commit intoapache:mainfrom
Conversation
…rix names The Spark 4.0.2 `maintenance` test in `StateStoreSuite` is flaky in CI (see issue apache#4221). The whole suite drives `StateStore.get/put/commit` directly with no DataFrame queries, so Comet does not exercise any of its code paths. Override `test()` in the suite to reroute every test to `ignore` when `ENABLE_COMET=true`. RocksDBStateStoreSuite is unaffected. Also append `-jdk<version>` to the spark-sql-test job display name and fallback-log artifact name so the two 4.0.2 matrix rows (JDK 17 and JDK 21) are distinguishable.
Contributor
|
CI fails because of which is super annoying and the nature of it is not clear |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #4221.
Rationale for this change
The Spark
StateStoreSuite.maintenancetest is intermittently flaky in CI on the Spark 4.0.2 build, forcing repeated re-runs. Looking at the suite, all 27 tests targetStateStore.get/put/commitdirectly viaSparkContext— no DataFrame queries, no SQL execution — so Comet does not exercise any of its code paths and ignoring the suite under Comet does not lose meaningful coverage.Separately, the
spark_sql_test.ymlmatrix has two rows for Spark 4.0.2 (JDK 17 and JDK 21), but the job display name and fallback-log artifact name do not include the JDK, making the two runs visually indistinguishable in the GitHub Actions UI.What changes are included in this PR?
dev/diffs/4.0.2.diffanddev/diffs/4.1.1.diff: overridetest()inStateStoreSuiteto reroute every test toignore(...)whenENABLE_COMET=true. The override lives only onStateStoreSuite, soRocksDBStateStoreSuite(which shares the same base class) is unaffected. The suite extendsSparkFunSuiterather thanSQLTestUtils, so the existingIgnoreCometSuitetrait could not be mixed in directly; the same logic is inlined usingclassic.SparkSession.isCometEnabled..github/workflows/spark_sql_test.yml: append-jdk${{ matrix.config.java }}to both the matrix job display name and the fallback-logupload-artifactname so the two Spark 4.0.2 rows render distinctly. The companionspark_sql_test_native_iceberg_compat.ymlworkflow already includes the JDK in its job name.How are these changes tested?
The diff regenerations were verified by resetting each Spark working tree to its base tag, applying the regenerated diff, and confirming
git applysucceeds with no conflicts. CI on this PR will exercise the changes against the full Spark SQL test suite for both JDK 17 and JDK 21 on 4.0.2.