Skip to content

ci: pin JDK per Spark version in Iceberg workflow matrix#4120

Open
dmefs wants to merge 2 commits intoapache:mainfrom
dmefs:fix/reduce-iceberg-ci-matrix
Open

ci: pin JDK per Spark version in Iceberg workflow matrix#4120
dmefs wants to merge 2 commits intoapache:mainfrom
dmefs:fix/reduce-iceberg-ci-matrix

Conversation

@dmefs
Copy link
Copy Markdown

@dmefs dmefs commented Apr 28, 2026

Which issue does this PR close?

Closes #4101 .

Rationale for this change

  • Iceberg's own CI does not exhaustively cross JDK with every Spark version; we are overcovering relative to upstream.
  • JDK 11 vs 17 differences that affect Comet are caught elsewhere: pr_build_linux.yml and spark_sql_test.yml both run JDK 11 + 17 against multiple Spark versions.
  • The Iceberg suites are about Iceberg/Comet integration, not JVM-level differences; the same JDK delta is unlikely to surface there but not in the broader matrix.
  • Frees CI capacity to add Spark 4.0.1 and 4.1.1 to the matrix without a net increase in PR runtime.

What changes are included in this PR?

The java-version: [11, 17] axis is removed from the base matrix of all three test jobs (iceberg-spark, iceberg-spark-extensions, iceberg-spark-runtime) and replaced with an include: block that injects the appropriate JDK per Spark version:

  • Spark 3.4.3 → JDK 11
  • Spark 3.5.8 → JDK 17

This reduces test jobs from 36 to 18 per PR with no changes to step logic or variable references.

How are these changes tested?

CI on this PR shows 18 jobs instead of 36.

@dmefs dmefs marked this pull request as ready for review April 28, 2026 06:26
@coderfender
Copy link
Copy Markdown
Contributor

coderfender commented Apr 28, 2026

Thank you for the PR @dmefs . I am assuming you verified the matrix built locally ?

@dmefs
Copy link
Copy Markdown
Author

dmefs commented Apr 28, 2026

Hi @coderfender, Thanks for the suggestion!
I'll verify the matrix build using my fork first. I'll convert this PR to draft while checking the action results.
Here is the run:
https://github.com/dmefs/datafusion-comet/actions/runs/25040787129.

@dmefs dmefs marked this pull request as draft April 28, 2026 08:05
@dmefs
Copy link
Copy Markdown
Author

dmefs commented Apr 28, 2026

Confirmed that there are 6 jobs per matrix and 3 matrices, for a total of 18 jobs per PR.
Screenshot 2026-04-28 at 4 32 06 PM
https://github.com/dmefs/datafusion-comet/actions/runs/25040787129

@dmefs dmefs marked this pull request as ready for review April 28, 2026 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reduce Iceberg CI matrix: pin JDK per Spark version

2 participants