feat: complete logical hashing implementation with platform-independent encoding and type normalization

[eywalker]

Summary

Complete rewrite of the Arrow logical hashing implementation, delivering all guarantees described in the design spec. Supersedes #6.

Logical hashing core (arrow_digester_core.rs)

• Struct field-order independence: Struct children are sorted alphabetically by name during both schema serialization (data_type_to_value) and data hashing (array_digest_update). Composite struct hashing via finalize_child_into_data — each child is independently hashed then finalized into the parent's data stream.
• Struct-level null propagation: When a struct is nullable, struct-level nulls are ANDed with child validity bitmaps so child digests reflect the combined nullability.
• Dictionary resolution: Dictionary-encoded arrays are cast to their plain value type before hashing, so DictionaryArray<Int32, Utf8> hashes identically to StringArray.
• List structural hashing: List element counts are written to a separate structural digest, separating structure from leaf data. This prevents collisions between differently-partitioned lists (e.g., [[1,2],[3]] vs [[1],[2,3]]).
• Recursive type normalization: normalize_data_type() / normalize_schema() recursively normalize Utf8→LargeUtf8, Binary→LargeBinary, List→LargeList, Dictionary→value_type at all nesting levels. Applied at schema storage (new()), schema serialization (data_type_to_value), and the data path (array_digest_update casts arrays to large equivalents).
• Platform-independent bit ordering: Validity bitmaps use BitVec<u8, Lsb0> (1-byte words) instead of BitVec<usize, Lsb0> (platform-dependent word size). Boolean packing uses Lsb0 to match Arrow's native layout. Bitmap lengths are cast to u64 before serialization.
• Logical schema comparison: update() compares schemas via serialized_schema() (canonical JSON), allowing batches with different column orders or equivalent type variants.
• Canonical schema JSON: element_type_to_value omits Arrow-internal field names (e.g., "item") for list/fixed-size-list elements. sort_json_value recursively sorts all JSON object keys.

API changes

• ArrowDigester::new() and ArrowDigesterCore::new() now take &Schema instead of Schema.

Documentation

• docs/design-spec.md: Rewritten to match actual implementation (Lsb0, structural digest, composite struct hashing, element_type_to_value, all known issues resolved).
• docs/byte-layout-spec.md: Comprehensive byte-level specification with 14 manually verified worked examples.
• CLAUDE.md: Project instructions for cargo formatting and TDD workflow.

Tests

• 85 total tests (32 unit + 39 integration + 14 byte-level verification)
• Type equivalence: Utf8/LargeUtf8, Binary/LargeBinary, List/LargeList at array, schema, and record-batch levels
• Deeply nested normalization: List(Struct({name: Utf8})) vs LargeList(Struct({name: LargeUtf8}))
• Streaming with type-equivalent and column-reordered schemas
• Dictionary equivalence with and without nulls
• Struct field-order independence at schema and record-batch levels
• Collision resistance for binary, string, and list partitioning

Test plan

• cargo test — 85 tests pass
• cargo clippy --tests — no errors
• cargo fmt --check — formatted

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR updates Starfix’s Arrow hashing to be platform-independent by standardizing validity bitmap storage/serialization and aligning boolean bit-packing with Arrow’s native LSB-first layout.

Changes:

• Switch validity bitmap storage from default BitVec (platform word-sized) to BitVec<u8, Lsb0> for deterministic cross-platform encoding.
• Change boolean value packing from Msb0 to Lsb0 to match Arrow’s boolean buffer bit layout.
• Encode validity bit-length as u64 before to_le_bytes() to ensure an 8-byte length prefix on all platforms.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 8 comments.

File	Description
src/arrow_digester_core.rs	Standardizes bitmap storage and length serialization; updates boolean packing order.
tests/arrow_digester.rs	Updates golden hash expectations to match the new canonical encoding.
tests/digest_bytes.rs	Updates expected outputs, but currently hardcodes digests in places that used to validate the manual byte-level construction.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

array_digest_update declares normalized_type/cast_array without initializing them on the _ => (data_type, array) path. In Rust, locals with Drop must be definitely initialized before the end of scope, so this pattern will fail to compile. Refactor the normalization block to avoid partially-initialized locals (e.g., have the match return an Option<ArrayRef> cast-guard plus a DataType, or scope each cast inside its branch and return (effective_type, effective_array, cast_guard) so all locals are initialized on every path).

Suggested change

	// These variables extend the lifetime of cast results. They are only
	// initialized (and read) in branches that perform a cast; the default
	// branch never touches them, which Rust's initialization analysis accepts.
	let (normalized_type, cast_array);
	let (effective_type, effective_array): (&DataType, &dyn Array) = match data_type {
	DataType::Utf8 => {
	normalized_type = DataType::LargeUtf8;
	cast_array =
	cast(array, &normalized_type).expect("Failed to cast Utf8 to LargeUtf8");
	(&normalized_type, cast_array.as_ref())
	}
	DataType::Binary => {
	normalized_type = DataType::LargeBinary;
	cast_array =
	cast(array, &normalized_type).expect("Failed to cast Binary to LargeBinary");
	(&normalized_type, cast_array.as_ref())
	}
	DataType::List(field) => {
	normalized_type = DataType::LargeList(Arc::clone(field));
	cast_array =
	cast(array, &normalized_type).expect("Failed to cast List to LargeList");
	(&normalized_type, cast_array.as_ref())
	}
	_ => (data_type, array),
	};
	// These variables extend the lifetime of cast results by keeping them
	// owned in a guard tuple returned from the match. This avoids partially
	// initialized locals while still providing a unified view for hashing.
	let (effective_type, effective_array, _cast_guard): (&DataType, &dyn Array, Option<(DataType, arrow::array::ArrayRef)>) =
	match data_type {
	DataType::Utf8 => {
	let normalized_type = DataType::LargeUtf8;
	let cast_array =
	cast(array, &normalized_type).expect("Failed to cast Utf8 to LargeUtf8");
	(&normalized_type, cast_array.as_ref(), Some((normalized_type, cast_array)))
	}
	DataType::Binary => {
	let normalized_type = DataType::LargeBinary;
	let cast_array = cast(array, &normalized_type)
	.expect("Failed to cast Binary to LargeBinary");
	(&normalized_type, cast_array.as_ref(), Some((normalized_type, cast_array)))
	}
	DataType::List(field) => {
	let normalized_type = DataType::LargeList(Arc::clone(field));
	let cast_array =
	cast(array, &normalized_type).expect("Failed to cast List to LargeList");
	(&normalized_type, cast_array.as_ref(), Some((normalized_type, cast_array)))
	}
	_ => (data_type, array, None),
	};

[eywalker]

This pattern compiles and works correctly — Rust's definite initialization analysis accepts it because normalized_type and cast_array are only read in branches that initialize them. The default branch returns (data_type, array) without touching either variable. A clarifying comment was added in 49c61c5 explaining the lifetime-extension intent. The code compiles cleanly under cargo clippy --tests -- -D warnings.

[Copilot]

The inline byte comment for validity_word.to_be_bytes() is incorrect: validity_word is a u8, so to_be_bytes() writes a single byte (0x01 here), not an 8-byte big-endian word. This comment is misleading for a byte-level spec test and should be updated to reflect the actual emitted bytes.

Suggested change

	final_digest.update(validity_word.to_be_bytes()); // 00 00 00 00 00 00 00 01
	final_digest.update(validity_word.to_be_bytes()); // 01

[eywalker]

Good catch. Fixed in c8272c9 — inline comment now correctly shows // 01 for u8 to_be_bytes().

[eywalker]

This PR is superseded by #10