[SkipRecovery] Recover repeated-key JSON extractor tests [databricks] - #15446
Draft
wjxiz1992 wants to merge 1 commit into
Draft
[SkipRecovery] Recover repeated-key JSON extractor tests [databricks]#15446wjxiz1992 wants to merge 1 commit into
wjxiz1992 wants to merge 1 commit into
Conversation
Signed-off-by: Allen Xu <allxu@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JaCoCo sql-plugin line coverage: +2 lines (
GpuJsonTuple.scala; focused repeated-keyjson_tupleIT, shim 350)Contributes to NVIDIA/cudf-spark#11361.
Depends on NVIDIA/cudf-spark-jni#4915. This PR remains draft until the JNI change is merged and available to the build.
Summary
LAST_NON_NULLnamed-field matching from the JNI multi-path API used byGpuJsonTuple.test_get_json_object_formatstest_get_json_object_child_formatstest_json_tuple_formatsfrom_jsonand JSON scan paths. This is a partial recovery, not closure of the root issue.Validation
Validation used the exact JNI artifact built from the dependency PR (
sha256 cc47495dd18e769d96242738bcf8e3320290b53ec8f87163b42b93c5a514359a).dist,integration_testsMaven package, shim 350:BUILD SUCCESS(11/11 reactor modules,03:05)3 passed in 4.94srepeated_columnscluster:12 passed, 31 xfailed, 992 deselected, 6 warnings in 20.97s1 passed in 4.44s2 of 4python3 -m py_compileandgit diff --check: passedPerformance impact
The Spark-side change adds one enum argument to the existing batched JNI call. It does not add JNI/native calls, input or output columns, allocations, or copies.
LAST_NON_NULLscans duplicate named fields to match Spark semantics; no throughput benchmark was run.Documentation
Testing
(Please provide the names of the existing tests in the PR description.)
Performance