[TheAli711]

Upgrades the engine to Zvec 0.8.1 and exposes its new storage-layer features. Intended for v0.3.0 (breaking changes are listed below). Spec: specs/SPEC-015-zvec-0-8-upgrade.md.

Upgrade

• Require zvec>=0.8.1. Picks up upstream fixes for crashes and missing rows when fetch/query race a flush, several filter bugs, and collection locks held by inherited file descriptors.
• On-disk compatibility: collections built with 0.7.0 (plain, int8, and int8 + rotation × hnsw/ivf/flat) open on 0.8.1, keep full recall, accept writes, and re-optimize correctly.

Features

• New quantizers: uniform_uint4/7/8 (hnsw, l2) and pq (hnsw, ivf; FP32), with num_chunk, num_bits, fast_scan (ivf), and OPQ via enable_rotate + opq_iter/opq_pq_iter. Every combination is validated server-side with a specific 422, because Zvec's own errors are generic, and some values crashed its constructors.
• Refiner search params: scale_factor on hnsw/hnsw_rabitq, and is_using_refiner + scale_factor on ivf.
• Document ids: any UTF-8 up to 1,024 bytes (no NUL or newline). GET /docs/{doc_id} routes ids that contain /. Field names can be up to 64 bytes.
• RaBitQ now works on macOS ARM64 / Intel Mac, as well as Linux.
• Benchmarks: the quant sweep covers the new variants and has a refiner axis.

Guard for an upstream regression

A new HNSW index with int8/int4 + enable_rotate returns near-random results after optimize on Zvec 0.8.0/0.8.1 (recall@10 = 0.000 on SIFT1M; 0.7.0 is fine). Reported as alibaba/zvec#886. The server rejects that combination with a 422 that links the issue. tests/unit/test_upstream_regressions.py reproduces it as a strict xfail, so CI goes red once an upstream release fixes it, which is the cue for ZS-076 to lift the guard. CI confirms the bug on Linux x86_64 too: the canary xfails there, so the guard applies to the production image.

Fixes (found while reviewing)

• is_using_refiner: true on an unquantized hnsw field returned 500 after optimize. It now returns 400 (this bug predates the upgrade).
• scale_factor that is negative, above 1000, or NaN/Infinity returned 500. It now returns 400.
• fp16 + enable_rotate created collections that rejected every insert (0.7.0 and 0.8.1). It now returns 422.
• Shutdown flushed and closed each collection under two separate exclusive-lock holds, so a queued optimize could slip in between (CI-only failure on 3.12). Both now happen under one hold (ZS-080).
• Backlogged rather than fixed here: ZS-078 (int params ≥ 2³¹ return 500) and ZS-079 (VECTOR_INT8 inserts return 500).

Measured (SIFT1M, hnsw, ef=200, c=16, Apple M5, engine tier)

quantize_type	disk vs fp32	recall@10	+ refiner	QPS vs fp32
fp32	687 MB	0.996	—	21.3k
int8	+20%	0.987	0.995	+25%
uniform_uint8	+20%	0.995	0.995	+20%
uniform_uint4	+11%	0.834	0.994	+16%
pq (32×8)	+7%	0.717	0.991	+20%

Disk never shrinks, because Zvec keeps the FP32 vectors. The refiner at scale_factor 0 halves throughput; scale_factor 5 gives pq 0.984 recall at +11% QPS. Full table in docs/API.md.

Breaking

• hnsw + int8/int4 + enable_rotate: true → 422 (until [Bug]: HNSW + INT8/INT4 + enable_rotate=True returns near-random results after optimize() (regression in 0.8.0) alibaba/zvec#886 is fixed). Existing collections are unaffected.
• enable_rotate: true with fp16 or uniform quantizers → 422 (those collections never worked).
• index_param.quantizer_param lists all six settings instead of only enable_rotate.

Test plan

• ruff check, ruff format --check, mypy, pytest (303 passed + 1 expected xfail), tracker --check, run locally on macOS arm64 / Python 3.13
• Benchmark tests (32) and SIFT1M sweeps on the engine and http tiers
• CI: Python 3.12 + 3.13 on Linux, and the canary still xfails there
• Deps: pillow 13.0.0 and certifi 2026.10.23 (bench extra only) clear the 42 Dependabot alerts

Closes ZS-068 – ZS-075, ZS-077 and ZS-080. Refs ZS-076, ZS-078, ZS-079.

🤖 Generated with Claude Code