feat(inference): normalize Ray tensor embeddings for Lance

[jiangxt2]

Description

Ray Data represents conventional two-dimensional NumPy embedding output as a fixed-shape Arrow tensor extension, while Lance vector columns require an Arrow FixedSizeList. This PR adds a narrow schema bridge to the generic Lance result sink so declared Ray V1/V2 and Arrow fixed-shape tensor columns can be written without requiring every custom Predictor to construct a FixedSizeList manually.

The sink now:

• validates the declared vector rank, dimension, and floating-point dtype on the driver;
• normalizes supported tensor columns to FixedSizeList in the existing distributed Arrow map_batches step;
• fails closed when a worker batch drifts from the driver schema, including non-vector field type, nullability, order, or metadata changes;
• fingerprints the normalized persisted schema in the result receipt; and
• reports sanitized cast details without exposing credential-bearing exception content.

Undeclared columns retain their previous behavior. The bridge does not infer model semantics, cast vector dtypes, flatten higher-rank tensors, mathematically normalize embedding values, or automatically create a vector index.

The distributed Lance vector integration gate now starts with a conventional two-dimensional NumPy embedding Predictor, writes through LanceResultSink, verifies the persisted values and schema, and then exercises the existing distributed index build, search, append, optimize, and compaction scenarios.

Related issues

None.

Additional information

• User-facing change: custom Predictors can return ordinary two-dimensional NumPy embeddings and write them to Lance when the vector column contract is declared explicitly.
• API compatibility: no request shape or dependency changes; direct Lance requests and undeclared vector columns keep their existing behavior.
• Documentation: the inference guide and support matrix describe the accepted representations, strict validation, normalization boundary, and limitations.
• Validation:
  • uv run --locked --no-sync python scripts/pr-precheck.py — passed with three reviewed inline-import warnings in the Docker integration test.
  • .venv/bin/python -m pytest tests/inference/test_lance_sink.py -q — 36 passed.
  • ./scripts/run_lance_vector_index_it.sh — passed against the Docker Ray cluster, writing 2,048 embeddings before distributed vector index and search validation.