fix(digester): preserve NullBuffer bit-offset when rebuilding sliced leaf arrays (PLT-1047)

[kurodo3]

Summary

Fixes PLT-1047 — ArrowDigester::hash_array panics on List<nullable T> when inner elements contain actual null values.

• Root cause: ArrayDataBuilder::null_bit_buffer() silently clears the builder's pre-existing nulls field (self.nulls = None). Arrow's PrimitiveArray::slice() produces arrays where the ScalarBuffer is trimmed to start at element 0 but the NullBuffer retains its original bit-offset (e.g. 3 for the second list element). When build() then reconstructs the NullBuffer from the raw bytes, it uses the array's data offset (0) rather than the bitmap's own bit-offset (3), so validate_nulls() counts nulls at bits 0..N instead of the correct range and panics with a null_count mismatch.

• Fix: Replace .null_count(n).null_bit_buffer(Some(raw_buf)) with .nulls(Some(effective.clone())). This stores the NullBuffer directly without discarding its internal bit-offset, so both validation and downstream hash operations see the correct null positions.

Test plan

• New test list_with_nullable_inner_elements_does_not_panic — reproduces the exact panic from the issue report
• New test list_with_nullable_inner_elements_is_deterministic — ensures hash is stable across calls
• New test large_list_with_nullable_inner_elements_does_not_panic — same fix applies to LargeList
• New test list_nullable_inner_null_placement_affects_hash — different null positions yield different hashes (correctness)
• All 49 existing tests continue to pass

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Fixes a panic in ArrowDigesterCore::traverse_leaf when hashing list arrays whose sliced child (leaf) arrays have a NullBuffer with a non-zero bit-offset (e.g., List<nullable T> where inner elements contain actual nulls). The change preserves the NullBuffer bit-offset when rebuilding ArrayData, and adds regression tests covering the reported scenarios.

Changes:

• Rebuild leaf ArrayData using ArrayDataBuilder::nulls(Some(NullBuffer)) to preserve NullBuffer bit-offset during hashing.
• Add regression tests ensuring hash_array does not panic for List<nullable> and LargeList<nullable> with inner nulls.
• Add correctness test asserting different inner-null placements produce different hashes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/arrow_digester_core.rs	Preserves NullBuffer bit-offset when rebuilding leaf arrays with combined nulls to prevent validation panics and incorrect null interpretation.
tests/arrow_digester.rs	Adds PLT-1047 regression + correctness tests for list arrays with nullable inner elements.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The doc comment for this test says it covers nulls at both the outer list level and inner element level, but the constructed array has no outer-level nulls (no None list entries). Either add an outer None entry to the input data or adjust the comment to match what the test actually exercises.

[kurodo3]

Good catch! The doc comment was indeed inaccurate — the array only contained inner-level nulls with no outer None list entries. I've updated the comment to say "nulls in the inner elements" so it accurately describes what the test exercises. No data changes were needed since the inner-null determinism scenario is the one we want to confirm.

[kurodo3]

Review round summary

One change made in response to Copilot's review:

• tests/arrow_digester.rs — corrected doc comment on list_with_nullable_inner_elements_is_deterministic: The original comment said the test covered "nulls at the outer level and inner level", but the array only contained inner nulls (no None list entries). Updated the comment to accurately say "nulls in the inner elements". No test data or assertions changed.

All 49 integration tests + 17 unit tests continue to pass.