Skip to content

[SkipRecovery] Recover repeated-key JSON extractor tests [databricks] - #15446

Draft
wjxiz1992 wants to merge 1 commit into
NVIDIA:mainfrom
wjxiz1992:codex/skipped-11361-remaining
Draft

[SkipRecovery] Recover repeated-key JSON extractor tests [databricks]#15446
wjxiz1992 wants to merge 1 commit into
NVIDIA:mainfrom
wjxiz1992:codex/skipped-11361-remaining

Conversation

@wjxiz1992

Copy link
Copy Markdown
Collaborator

JaCoCo sql-plugin line coverage: +2 lines (GpuJsonTuple.scala; focused repeated-key json_tuple IT, shim 350)

Contributes to NVIDIA/cudf-spark#11361.

Depends on NVIDIA/cudf-spark-jni#4915. This PR remains draft until the JNI change is merged and available to the build.

Summary

  • Request LAST_NON_NULL named-field matching from the JNI multi-path API used by GpuJsonTuple.
  • Recover the repeated-key cases in:
    • test_get_json_object_formats
    • test_get_json_object_child_formats
    • test_json_tuple_formats
  • Keep the remaining 18 issue-11361 exemptions for from_json and JSON scan paths. This is a partial recovery, not closure of the root issue.

Validation

Validation used the exact JNI artifact built from the dependency PR (sha256 cc47495dd18e769d96242738bcf8e3320290b53ec8f87163b42b93c5a514359a).

  • Offline dist,integration_tests Maven package, shim 350: BUILD SUCCESS (11/11 reactor modules, 03:05)
  • Three recovered cases: 3 passed in 4.94s
  • Broader repeated_columns cluster: 12 passed, 31 xfailed, 992 deselected, 6 warnings in 20.97s
  • Focused JaCoCo run: 1 passed in 4.44s
  • Added production lines covered by the focused run: 2 of 4
  • python3 -m py_compile and git diff --check: passed

Performance impact

The Spark-side change adds one enum argument to the existing batched JNI call. It does not add JNI/native calls, input or output columns, allocations, or copies. LAST_NON_NULL scans duplicate named fields to match Spark semantics; no throughput benchmark was run.

Documentation

  • Updated for new or modified user-facing features or behaviors
  • No user-facing change

Testing

  • Added or modified tests to cover new code paths
  • Covered by existing tests
    (Please provide the names of the existing tests in the PR description.)
  • Not required

Performance

  • Tests ran and results are added in the PR description
  • Issue filed with a link in the PR description
  • Not required

Signed-off-by: Allen Xu <allxu@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants