Repository navigation
perf: NEON kernel for 5/6-bit widths — scalar path is ~4.6x slower than the 4-bit NEON path #40
Description
Found during #39 implementation (E2E bench, real EmbeddingGemma d=768, n=2000, 100 queries, release)
| mode | B/vec | r@1 | r@10 | ms/q |
|---|---|---|---|---|
| plain 4-bit (NEON) | 514 | 0.910 | 0.958 | 0.92 |
| plain 5-bit (scalar) | 642 | 0.970 | 0.979 | 4.27 |
| plain 6-bit (scalar) | 770 | 0.960 | 0.980 | 4.29 |
| 4-bit+residual (NEON) | 1028 | 0.990 | 0.984 | 1.80 |
Recall/size goals of #39 are met, but 5/6-bit scoring currently runs the portable scalar unpack loop instead of the batch-4 NEON kernel, costing ~4.6x query latency vs 4-bit.
Scope
-
score_neon_wideon aarch64: unpack 5/6-bit codes to bytes (or 16-bit indices), LUT gather from 32/64-entry centroid tables, widen+accumulate — keep bit-identity withscore_wide_scalar(existing bitwise parity test pattern). - Wire into the batch-4 slot loop next to
score_neon4whenbits != 4. - Optional AVX2 variant (
use_avx2 && bits != 4gate already exists). - Acceptance: 5/6-bit ms/q within ~1.5x of the 4-bit path on the same bench;
neon_matches_scalar_bitwise-style parity test extended to 5/6-bit.
Notes
- The 16-bit RMW pack/unpack helpers are correct but scalar; the kernel can read the same LSB-first layout directly.
- Until this lands, users needing both recall ≥0.98 and <2 ms/q should use residual (1.80 ms/q) or wait — documented in feat(core): parameterize quantizer bit-width (5/6-bit) — dominates residual mode on size AND speed #39.
Activity
perf(neon): batch-4 wide kernel for 5/6-bit scoring (#40)
ajianaz commentedon Aug 29, 2026
Progress: window-extraction kernel shipped (a8edd9c)
score_neon_wide4v2: one unaligned u64 window per 8 dims (replaces 8×unpack_code), L1 gather, NEON mul/add with per-lane bucket order bit-identical toscore_wide_scalar(bitwise-tested, 5/6-bit × dims {128,384}).E2E (real, d=768, n=2000): 5-bit 4.27 → 3.23 ms/q (−24%).
Honest ceiling analysis for the remaining gap (3.23 vs 0.89 ms/q)
Per-dim cost now: vectorized mul/add (~same as 4-bit path) + scalar extraction & gather (shift/and + L1 load). The 4-bit path extracts with 2 NEON ops per 16 dims and gathers via
vqtbl4(64-byte table = 16 f32 entries).The vector gather for 5/6-bit cannot use
vqtbldirectly: tables are 128/256 bytes andvqtbladdressing caps at 64 bytes. The standard workaround (range-split gather:vqtbl4on low nibble +vqtbl4on high bit as select +vbsl) costs ~2–3× the 4-bit gather per dim.Projected split-layout ceiling (separate low-4-bit + high-1/2-bit streams, vector extraction, split gather): ≈ 2.2–2.7 ms/q — i.e. 5/6-bit can likely never beat the 4-bit NEON path, and at best approaches the existing residual mode (1.76 ms/q @ r@10 0.984) which needs zero new machinery.
Recommendation
feat(core): configurable Lloyd-Max width (4/5/6-bit), default 5-bit s…
ajianaz commentedon Aug 29, 2026
v2 kernel landed in develop (merged via #41, eb26d67)
score_neon_wide4(u64-window extraction + L1 gather + NEON mul/add, bitwise-tested vs scalar) is in develop. 5-bit: 4.27 → 3.23 ms/q (−24%).Remaining scope for this issue, in priority order:
Keeping this open for the split-layout decision; no urgency — residual mode already covers the recall≥0.98 + <2 ms/q niche at 1.76 ms/q.
docs: benchmark width matrix (#39/#40) + changelog for Unreleased (#42)
ajianaz commentedon Aug 29, 2026
Closed: primary scope shipped, remaining scope declined with rationale
Shipped (merged via #41):
score_neon_wide4batch kernel for 5/6-bit — u64-window extraction + L1 gather + NEON mul/add, bitwise parity vs scalar (tested at 5/6-bit × dims {128,384}).Declined, with analysis (see ceiling comment above):
Final landscape is documented in docs/BENCHMARK.md (width matrix) — close as completed.
chore: release v0.3.0 to main (#46)
chore: release v0.3.1 to main (#55)