docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP]#4244
Open
andygrove wants to merge 9 commits intoapache:mainfrom
Open
docs: refresh Spark version support, OS coverage, and version-pinned examples [WIP]#4244andygrove wants to merge 9 commits intoapache:mainfrom
andygrove wants to merge 9 commits intoapache:mainfrom
Conversation
…examples Update user-facing docs ahead of the 0.16 release: - Promote Spark 4.1 from experimental to fully supported across the installation page, compatibility guide, and Gluten comparison; keep 4.2 listed as experimental. Spark 4.1.1 now runs in CI under both JDK 17 and 21. - Restructure the Supported Operating Systems section in the installation guide to make clear that published Maven jars cover Linux only and that macOS users must build from source. Drop the Intel macOS claim since Apple Silicon is the only macOS variant exercised in CI. - Flip seven recently-added math expressions to supported in the contributor-guide tracking page: acosh, asinh, atanh, cbrt, degrees, pi, radians. - Move spark.comet.scan.enabled to the testing category and rewrite its description to reflect that it is intended for Comet's own test suites only. Remove the corresponding mention from the data sources page. - Replace hard-coded Comet versions in the Iceberg and Kubernetes guides with the \$COMET_VERSION placeholder used elsewhere, and drop the redundant spark.sql.extensions=...CometSparkSessionExtensions conf from the Iceberg examples (CometPlugin registers it automatically).
andygrove
changed the title
…t scans PR apache#4011 added non-AQE DPP and PR apache#4112 added AQE DPP with broadcast reuse for native Parquet scans, but the docs still claimed AQE DPP was unsupported. Update the scans compatibility page, the contributor-guide roadmap, and the Iceberg guide accordingly. AQE DPP for Iceberg native scans remains future work, tracked at apache#3510.
# Conflicts: # docs/source/user-guide/latest/iceberg.md # docs/source/user-guide/latest/kubernetes.md
andygrove
commented
May 7, 2026
Comment on lines
+97
to
+99
| .category(CATEGORY_TESTING) | ||
| .doc("Whether to enable native scans. Intended for use in Comet's own test suites to " + | ||
| "selectively disable native scans; not intended for production use.") |
Member
Author
There was a problem hiding this comment.
this is unrelated to 0.16 / spark 4 changes, but we don't want users using this config
andygrove
commented
May 7, 2026
Comment on lines
+206
to
+208
| # corresponding Spark release, so the scala-2.13 profile is not used here. | ||
| ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.0 -DskipTests install | ||
| ./mvnw "-Dmaven.repo.local=${LOCAL_REPO}" -P spark-4.1 -DskipTests install |
comphead
reviewed
May 7, 2026
| 10.042 | ||
| ] | ||
| } No newline at end of file | ||
| "1": [12.007], |
comphead
reviewed
May 7, 2026
| ```shell | ||
| $SPARK_HOME/bin/spark-shell \ | ||
| --packages org.apache.datafusion:comet-spark-spark4.1_2.13:0.14.0,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \ | ||
| --packages org.apache.datafusion:comet-spark-spark3.5_2.12:$COMET_VERSION,org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1,org.apache.iceberg:iceberg-core:1.8.1 \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #.
Rationale for this change
Several user-facing docs are out of date ahead of the 0.16 release:
[ ]in the contributor-guide tracking page.spark.comet.scan.enabledis documented as a user-facing scan toggle, but in practice it exists to let Comet's own tests selectively disable native scans.0.14.0and0.7.0) in the example commands and YAML, and the Iceberg example also redundantly setsspark.sql.extensions=org.apache.comet.CometSparkSessionExtensions, whichCometPluginalready registers.What changes are included in this PR?
compatibility/spark-versions.md,installation.md, andabout/gluten_comparison.md. The experimental table now lists only Spark 4.2.0-preview4. Spark 4.1.1 is shown as running on JDK 17/21.installation.mdinto a table that distinguishes published Maven jars (Linux amd64/arm64) from build-from-source coverage. Drop the Intel macOS row.math_funcsentries (acosh,asinh,atanh,cbrt,degrees,pi,radians) from[ ]to[x]indocs/source/contributor-guide/spark_expressions_support.mdto reflect commits a4f0229, 1b4b26f, e5351f4, and 356dd94.spark.comet.scan.enabledfromCATEGORY_SCANtoCATEGORY_TESTINGinCometConf.scalaand rewrite the doc string to make clear it is intended for Comet's own test suites. Remove the corresponding paragraph fromdatasources.md.0.14.0references iniceberg.mdand0.7.0references inkubernetes.mdwith the$COMET_VERSIONplaceholder used elsewhere in the docs. Drop the redundantspark.sql.extensions=org.apache.comet.CometSparkSessionExtensionsconf from both Iceberg spark-shell examples.How are these changes tested?
Documentation-only changes apart from the
CometConfdoc string and category move; theCONFIG_TABLE[testing]section inconfigs.mdis regenerated fromCometConfbyGenerateDocs.scalaon merge.