[AutoSparkUT] Migrate ten Spark SQL suites [fast-ut] [reduced-it] by wjxiz1992 · Pull Request #15456 · NVIDIA/cudf-spark

wjxiz1992 · 2026-07-30T08:16:19Z

JaCoCo sql-plugin line coverage: +78 lines (+80 measured, excluding the recurring MemoryCheckerImpl +2 local GPU initialization artifact; clean Spark 330 exec vs shim 350 nightly b22 anchor classfiles)

Closes #15452

Summary

Migrate ten eligible Spark 3.3 SQL suites in one PR using minimal RAPIDS inheritance:

DataFrameReaderWriterSuite
InsertSuite
BucketedReadWithoutHiveSupportSuite
InMemoryColumnarQuerySuite
PathFilterSuite
DataSourceV2Suite
JsonParsingOptionsSuite
PartitionedWriteSuite
V1WriteFallbackSuite
GlobalTempViewSuite

All wrappers use RapidsSQLTestsTrait and are registered in RapidsTestSettings.

Spark v3.3.0 traceability

Spark source commit: f74867bddfbcdd4d08076db36851e88b15e66556 (v3.3.0). Inherited test names remain unchanged; replacement names use the Rapids - prefix at runtime.

RAPIDS suite	Original Spark suite	Spark 3.3.0 source	Original tests	Executed
`RapidsDataFrameReaderWriterSuite`	`DataFrameReaderWriterSuite`	DataFrameReaderWriterSuite.scala:144-1277	55	55
`RapidsInsertSuite`	`InsertSuite`	InsertSuite.scala:59-1128	31	31
`RapidsBucketedReadWithoutHiveSupportSuite`	`BucketedReadWithoutHiveSupportSuite` / `BucketedReadSuite`	BucketedReadSuite.scala:42-1074	32	20
`RapidsInMemoryColumnarQuerySuite`	`InMemoryColumnarQuerySuite`	InMemoryColumnarQuerySuite.scala:51-619	33	20
`RapidsPathFilterSuite`	`PathFilterSuite`	PathFilterSuite.scala:31-307	19	19
`RapidsDataSourceV2Suite`	`DataSourceV2Suite`	DataSourceV2Suite.scala:46-984	33	33
`RapidsJsonParsingOptionsSuite`	`JsonParsingOptionsSuite`	JsonParsingOptionsSuite.scala:27-188	16	16
`RapidsPartitionedWriteSuite`	`PartitionedWriteSuite`	PartitionedWriteSuite.scala:50-243	10	10
`RapidsV1WriteFallbackSuite`	`V1WriteFallbackSuite`	V1WriteFallbackSuite.scala:43-435	12	12
`RapidsGlobalTempViewSuite`	`GlobalTempViewSuite`	GlobalTempViewSuite.scala:28-195	9	9

Total original tests: 250. Total effective tests executed: 225.

Remediation and deferred cases

The initial run had 29 failures concentrated in CPU-private assertions.

Replaced 3 InsertSuite exception assertions with RAPIDS-aware tests that preserve the ANSI overflow/no-partial-write and FileAlreadyExistsException contracts.
Replaced DataSourceV2Suite partitioning reporting with a plan assertion against ShuffleExchangeLike, valid for CPU and GPU exchanges.
Deferred 12 bucket scan-plan assertions and 13 Spark cache serializer/statistics assertions. Every deferred exclusion links [AutoSparkUT] Migrate ten Spark SQL suites #15452 and includes an explicit P1 recovery trigger.

The 31 ignored tests in the final summary are 2 pre-existing upstream ignores plus 29 explicitly registered migration exclusions. Four excluded originals have passing RAPIDS-aware replacements; 25 require follow-up remediation.

Validation

Spark 3.3 combined GPU Maven package run:

Tests: succeeded 225, failed 0, canceled 0, ignored 31, pending 0
BUILD SUCCESS

The same clean 225-test run produced the JaCoCo exec used for the coverage delta.

Coverage measurement

Nightly baseline: shim 350 b22, anchor commit 3421ec13231cc246ca2b4da5d0a2a391f0b87763.

covered lines: +80

After excluding the recurring MemoryCheckerImpl +2 local GPU initialization artifact, the PR contribution is +78 lines.

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-07-30T08:18:26Z

build

greptile-apps · 2026-07-30T08:18:35Z

Greptile Summary

Adds ten Spark 3.3 RAPIDS SQL suite wrappers and registers them for execution.

Introduces RAPIDS-aware replacements for three insert exception tests and one data-source partitioning test.
Excludes inherited tests requiring GPU-specific cache, bucket-scan, and plan assertions.
Enables the remaining inherited tests through RapidsTestSettings.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Registers all ten migrated suites and records exclusions or RAPIDS-aware replacements for incompatible inherited assertions.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsInsertSuite.scala	Adds RAPIDS-aware ANSI overflow and file-existence exception tests, including no-partial-write verification.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsDataSourceV2Suite.scala	Replaces a CPU-specific partitioning test with plan checks using the exchange interface shared by Spark 3.3 CPU and GPU shuffle nodes.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsBucketedReadWithoutHiveSupportSuite.scala	Adds the minimal RAPIDS wrapper for Spark's non-Hive bucketed-read suite.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsInMemoryColumnarQuerySuite.scala	Adds the minimal RAPIDS wrapper for Spark's in-memory columnar query suite.

_{Reviews (3): Last reviewed commit: "test: include exception types in insert ..." | Re-trigger Greptile}

Copilot

🟡 Not ready to approve

The new RapidsInsertSuite uses exception-message aggregation that can omit exception type names, making the added FileAlreadyExistsException assertion brittle.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Migrates ten Apache Spark 3.3 SQL test suites into the cudf-spark RAPIDS test harness by adding minimal wrapper suites (mixing in RapidsSQLTestsTrait) and registering them in RapidsTestSettings, including a few RAPIDS-aware replacement tests and explicit exclusions for CPU-private assertions.

Changes:

Register 10 newly wrapped Spark SQL suites in RapidsTestSettings, with targeted exclusions and adjusted-test annotations.
Add wrapper suite classes for Spark 3.3 (spark330) that extend upstream Spark suites and apply RapidsSQLTestsTrait.
Add RAPIDS-aware replacement tests for select InsertSuite and DataSourceV2Suite cases where CPU-only assertions were previously failing.

File summaries

File	Description
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Registers the 10 migrated suites and configures exclusions/adjusted-test reasons.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsDataFrameReaderWriterSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `DataFrameReaderWriterSuite`.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsInsertSuite.scala	Wrapper + RAPIDS-aware replacement tests for ANSI overflow and file-exists failure cases.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsBucketedReadWithoutHiveSupportSuite.scala	Wrapper suite for Spark bucketing tests, with follow-up exclusions tracked in settings.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsInMemoryColumnarQuerySuite.scala	Wrapper suite for Spark cache/columnar query tests, with follow-up exclusions tracked in settings.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsPathFilterSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `PathFilterSuite`.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsDataSourceV2Suite.scala	Wrapper + RAPIDS-aware replacement test using `ShuffleExchangeLike` for plan assertions.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsJsonParsingOptionsSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `JsonParsingOptionsSuite`.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsPartitionedWriteSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `PartitionedWriteSuite`.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsV1WriteFallbackSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `V1WriteFallbackSuite`.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsGlobalTempViewSuite.scala	Wrapper suite enabling RAPIDS trait for Spark’s `GlobalTempViewSuite`.

Review details

Files reviewed: 11/11 changed files
Comments generated: 1
Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

wjxiz1992 · 2026-07-31T03:30:59Z

+  private def exceptionMessages(error: Throwable): String = {
+    val messages = new StringBuilder
+    var current = error
+    while (current != null) {
+      messages.append(Option(current.getMessage).getOrElse(""))
+      messages.append('\n')
+      current = current.getCause
+    }
+    messages.toString()
+  }


Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-07-30T09:11:05Z

build

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-07-31T03:12:09Z

build

test: migrate ten Spark SQL suites

fb5331c

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 requested review from Copilot and thirtiseven July 30, 2026 08:16

Copilot started reviewing on behalf of wjxiz1992 July 30, 2026 08:16 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

fix: correct DataSource V2 import grouping

b50fedb

Signed-off-by: Allen Xu <allxu@nvidia.com>

test: include exception types in insert failures

df3b543

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 requested review from firestarman and res-life July 31, 2026 04:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[AutoSparkUT] Migrate ten Spark SQL suites [fast-ut] [reduced-it] - #15456

[AutoSparkUT] Migrate ten Spark SQL suites [fast-ut] [reduced-it]#15456
wjxiz1992 wants to merge 3 commits into
NVIDIA:mainfrom
wjxiz1992:codex/migrate-ten-suites-20260730

wjxiz1992 commented Jul 30, 2026

Uh oh!

wjxiz1992 commented Jul 30, 2026

Uh oh!

greptile-apps Bot commented Jul 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

wjxiz1992 Jul 31, 2026

Uh oh!

wjxiz1992 commented Jul 30, 2026

Uh oh!

wjxiz1992 commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wjxiz1992 commented Jul 30, 2026

Summary

Spark v3.3.0 traceability

Remediation and deferred cases

Validation

Coverage measurement

Uh oh!

wjxiz1992 commented Jul 30, 2026

Uh oh!

greptile-apps Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Not ready to approve

Uh oh!

wjxiz1992 Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Jul 30, 2026

Uh oh!

wjxiz1992 commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

greptile-apps Bot commented Jul 30, 2026 •

edited

Loading