[AutoSparkUT] Migrate ten Spark SQL suites [fast-ut] [reduced-it]

[wjxiz1992]

JaCoCo sql-plugin line coverage: not measurable locally — Spark 3.3-only suites are not selected by the mandatory shim 350 coverage rig

Closes #15465

Summary

Migrate ten eligible Spark 3.3 SQL suites in one PR using minimal RAPIDS inheritance:

• WriteDistributionAndOrderingSuite
• OrcSourceV1Suite
• OrcSourceV2Suite
• KeyGroupedPartitioningSuite
• DataSourceV2DataFrameSuite
• BucketedWriteWithoutHiveSupportSuite
• CreateTableAsSelectSuite
• FileDataSourceV2FallBackSuite
• V1ReadFallbackWithDataFrameReaderSuite
• V1ReadFallbackWithCatalogSuite

All wrappers use RapidsSQLTestsTrait and are registered in RapidsTestSettings.

Spark source traceability

The current Spark master links were verified after syncing to 256d453a0074a8dc7b58e7d548653e52a87d98d4. The pinned compatibility source is Spark v3.3.0 commit f74867bddfbcdd4d08076db36851e88b15e66556.

RAPIDS suite	Original Spark suite	Spark master source	Spark 3.3.0 source	Original tests	Executed
RapidsWriteDistributionAndOrderingSuite	WriteDistributionAndOrderingSuite	L49-L1554	L39-L1052	92	92
RapidsOrcSourceV1Suite	OrcSourceV1Suite / OrcSourceSuite	L636-L1105	L585-L1030	34	26
RapidsOrcSourceV2Suite	OrcSourceV2Suite / OrcSourceSuite	L636-L1111	L585-L1036	34	26
RapidsKeyGroupedPartitioningSuite	KeyGroupedPartitioningSuite	L49-L4498	L40-L425	13	13
RapidsDataSourceV2DataFrameSuite	DataSourceV2DataFrameSuite	L48-L3966	L32-L256	21	21
RapidsBucketedWriteWithoutHiveSupportSuite	BucketedWriteWithoutHiveSupportSuite / BucketedWriteSuite	L35-L313	L33-L301	15	15
RapidsCreateTableAsSelectSuite	CreateTableAsSelectSuite	L30-L303	L31-L291	11	11
RapidsFileDataSourceV2FallBackSuite	FileDataSourceV2FallBackSuite	L83-L201	L84-L195	5	5
RapidsV1ReadFallbackWithDataFrameReaderSuite	V1ReadFallbackWithDataFrameReaderSuite / V1ReadFallbackSuite	L33-L83	L32-L82	4	4
RapidsV1ReadFallbackWithCatalogSuite	V1ReadFallbackWithCatalogSuite / V1ReadFallbackSuite	L33-L100	L32-L99	4	4

Total original tests: 233. Total effective tests executed: 217.

Inherited test names remain unchanged. The one replaced test maps as follows:

• RAPIDS Rapids - Fallback Parquet V2 to V1 → Spark Fallback Parquet V2 to V1: master L158-L200, Spark 3.3.0 L155-L194.

Remediation and deferred cases

The initial GPU run executed 231 tests and found 15 failures:

Tests: succeeded 216, failed 15, canceled 0, ignored 2, pending 0

• Replaced the CPU-only FileSourceScanExec assertion in Fallback Parquet V2 to V1 with GpuFileSourceScanExec, while preserving its listener, V1 command, ParquetFileFormat, and result assertions.
• Deferred seven ORC contracts by exact name in both V1 and V2. The P0 cases track legacy date/timestamp parity and numeric-only field names; P1 tracks Hadoop option propagation ([BUG] Spark UT framework: Propagate Hadoop configs from parquet options to underlying file system #11602) and Spark-version metadata; P2/WONT_FIX tracks the CPU-specific direct-encoding selection contract.
• Expected CPU results were not changed and GPU/CPU correctness differences were not relaxed.

The 17 ignored tests in the final summary are 14 ORC migration exclusions, one original test with a passing RAPIDS-aware replacement, and two inherited Spark ignores.

Each unresolved ORC contract now has a dedicated root-cause issue; the batch
migration issue is not used as exclusion evidence:

Excluded Spark test	Dedicated issue
Propagate Hadoop configs from orc options to underlying file system	#11602
Write Spark version into ORC file metadata	#15468
SPARK-31238: compatibility with Spark 2.4 in reading dates	#15470
SPARK-31284: compatibility with Spark 2.4 in reading timestamps	#15471
SPARK-31284, SPARK-31423: rebasing timestamps in write	#15473
Enforce direct encoding column-wise selectively	#15469
SPARK-36663: ... a column name which consists of only numbers	#15472

Validation

Focused remediation runs:

RapidsFileDataSourceV2FallBackSuite: succeeded 5, failed 0, ignored 1
RapidsOrcSourceV1Suite: succeeded 26, failed 0, ignored 8
RapidsOrcSourceV2Suite: succeeded 26, failed 0, ignored 8

Final combined Spark 3.3 GPU Maven package run:

Tests: succeeded 217, failed 0, canceled 0, ignored 17, pending 0
BUILD SUCCESS

Full repository ScalaStyle:

Processed 1720 file(s)
Found 0 errors
Found 0 warnings
BUILD SUCCESS

Coverage measurement

The mandatory KPI baseline is shim 350 nightly b22 at anchor 3421ec13231cc246ca2b4da5d0a2a391f0b87763. These wrappers are deliberately confined to the spark330 source set and produce Expected test count is: 0 in the matching shim 350 rig. A Spark 330 JaCoCo exec cannot be merged against shim 350 classfiles without invalid probe mappings, so no line delta is reported.

Scope markers

This PR changes only tests/src/test/spark330 wrappers and registration. It does not touch production, shared, or Databricks-specific code, and migration-UT PRs do not use the [databricks] title marker.

Documentation

• Updated for new or modified user-facing features or behaviors
• No user-facing change

Testing

• Added or modified tests to cover new code paths
• Covered by existing tests
  (Please provide the names of the existing tests in the PR description.)
• Not required

Performance

• Tests ran and results are added in the PR description
• Issue filed with a link in the PR description
• Not required

[greptile-apps]

Greptile Summary

Adds Spark 3.3 RAPIDS wrappers for ten upstream SQL suites.

• Registers the new suites in the Spark 3.3 test settings.
• Defers fourteen known ORC incompatibility cases through exact-name exclusions.
• Replaces the CPU-specific Parquet V2-to-V1 fallback assertion with a RAPIDS-aware test that verifies the V1 write command, result correctness, and GPU scan node.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsFileDataSourceV2FallBackSuite.scala	Adds a RAPIDS-aware replacement for the inherited Parquet V2-to-V1 fallback test.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsOrcSourceSuite.scala	Adds minimal RAPIDS wrappers for the Spark 3.3 ORC V1 and V2 suites.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Registers all ten wrappers and configures the documented ORC and fallback-test exclusions.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsWriteDistributionAndOrderingSuite.scala	Adds the Spark 3.3 RAPIDS wrapper for write distribution and ordering tests.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsDataSourceV2DataFrameSuite.scala	Adds the Spark 3.3 RAPIDS wrapper for DataSource V2 DataFrame tests.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsKeyGroupedPartitioningSuite.scala	Adds the Spark 3.3 RAPIDS wrapper for key-grouped partitioning tests.

_{Reviews (3): Last reviewed commit: "test: link ORC exclusions to dedicated i..." | Re-trigger Greptile}

[Copilot]

Pull request overview

This PR expands the Spark 3.3 migrated-unit-test matrix by adding RAPIDS wrapper suites (via RapidsSQLTestsTrait) for ten additional Spark SQL test suites and registering them in RapidsTestSettings, including targeted exclusions for known ORC gaps and a RAPIDS-aware replacement for one Parquet V2→V1 fallback plan assertion.

Changes:

• Register 10 additional Spark 3.3 SQL suites in RapidsTestSettings, including explicit known-issue exclusions for ORC V1/V2 and an adjusted UT exclusion for the original “Fallback Parquet V2 to V1”.
• Add minimal Spark 3.3 wrapper suites that inherit the upstream Spark suites while using RAPIDS test gating (RapidsSQLTestsTrait).
• Introduce a RAPIDS-specific replacement test for Parquet V2→V1 fallback that asserts GpuFileSourceScanExec while preserving the original command/result assertions.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Enables the migrated suites and wires in known-issue/adjusted-test exclusions.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsWriteDistributionAndOrderingSuite.scala	Adds Spark 3.3 wrapper for WriteDistributionAndOrderingSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsOrcSourceSuite.scala	Adds Spark 3.3 wrappers for ORC V1/V2 source suites.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsKeyGroupedPartitioningSuite.scala	Adds Spark 3.3 wrapper for KeyGroupedPartitioningSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsDataSourceV2DataFrameSuite.scala	Adds Spark 3.3 wrapper for DataSourceV2DataFrameSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsBucketedWriteSuite.scala	Adds Spark 3.3 wrapper for BucketedWriteWithoutHiveSupportSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsCreateTableAsSelectSuite.scala	Adds Spark 3.3 wrapper for CreateTableAsSelectSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsV1ReadFallbackSuite.scala	Adds Spark 3.3 wrappers for both V1ReadFallbackWithDataFrameReaderSuite and V1ReadFallbackWithCatalogSuite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsFileDataSourceV2FallBackSuite.scala	Adds RAPIDS replacement test for Parquet V2→V1 fallback that asserts GPU scan node.

[wjxiz1992]

build