docs(adr): ADR-073 — KV-cache quantization priority (Metal)

[oceanwaves630]

PR description authored by Claude (Anthropic agent) on behalf of @oceanwaves630.

What

ADR-073 ranks the open KV-cache quantization levers (#154 f16 default-flip, #148 f16 paged-cache,
#118/#120/#122/#123 int8/int4 family) against what already ships. Docs only — one new file under
docs/adr/. Third sequencing ADR in the bundle set (siblings: ADR-071 GDN-state, ADR-072 weight-quant).

Why this is a split verdict, not a clean ranking

Source recon (verified against origin/main @ 454245434) found this lane is not a single yes/no
like weight-quant was:

• Rankable now on measured evidence: f16 KV already shipped and was measured decode-neutral
  (PR feat(inference): opt-in f16 KV cache for Metal GQA path (#154) #238, merged — 71.6 vs 72.8 tok/s @1k ctx, PPL Δ≈4e-6 on Qwen3.5-0.8B). It's a memory win, not
  a throughput one, because the model is GDN-dominated — only 6 of 24 layers carry a KV cache, so
  KV is <1% of decode bandwidth. PR feat(inference): opt-in f16 KV cache for Metal GQA path (#154) #238's own words: "The real decode lever is weight
  quantization, not KV dtype."
• Needs an experiment: the int8/int4 family (feat(kv_cache): quantized KV cache storage (int8/int4/fp8) #118/feat(kv_cache): fused quantized attention kernel for Metal #120/feat(kv_cache): pre-RoPE K quantization path #122/feat(kv_cache): WHT/SRFT rotation for Qwen int4 KV #123) has zero lattice-measured
  quality or throughput data. The central open question isn't just quality — it's whether KV bit-width
  moves decode throughput at all on a model whose KV is <1% of bandwidth.

Prior-art refutation (the W4-amendment pattern)

The external survey's central premise — "KV ≈ 16% of decode bytes," which drives every expected-speedup
number — is refuted: it assumes a generic Hkv=8, head_dim=128 shape. This engine is Hkv=2
(0.8B) / Hkv=4 (27B), head_dim=256, 6-of-24 / 16-of-64 KV layers, measured <1% at 1k. Causal
mechanism (the GDN-hybrid architecture structurally caps KV's share) recorded at priority level.

Decision

• P1 — f16 KV default-flip (feat(inference): wire f16 KV cache into default Metal decode path #154). Rankable now on the already-measured PR feat(inference): opt-in f16 KV cache for Metal GQA path (#154) #238 result; only
  blocker is the CI parity-test gap (test(inference): CI coverage for the f16 KV-cache Metal path (#238 follow-up) #252), an engineering completeness gate, not an empirical question.
• P-low — f16 for PagedKVCache (feat(kv_cache): f16 storage for PagedKVCache + Metal kernel updates #148). Mechanical port of the proven pattern to the second cache;
  roadmap-gated (that engine serves zero production traffic).
• P2 (gated) — int8/int4 family behind a minimal int8-K CPU experiment + bench(gdn-state): measure unified-memory bandwidth share of GDN-state vs KV vs weight traffic per decode step #491 (measure KV
  bandwidth share vs context). Quantize K-only/int8/full-attn-layers-only, CPU dequant-to-f32
  prototype, extend the existing ppl_metal/eval_perplexity harness. The experiment settles whether
  int8-K moves anything worth a fused Metal kernel before any kernel work starts.
• Deferred: int4 + WHT rotation (feat(kv_cache): WHT/SRFT rotation for Qwen int4 KV #123), fused kernel (feat(kv_cache): fused quantized attention kernel for Metal #120), pre-RoPE K (feat(kv_cache): pre-RoPE K quantization path #122) — all behind int8-K
  validation.

Structure

Measured/source-verified reality (KV1–KV12, each tagged + origin/main pointer) is kept separate from
the external prior-art survey, folded as [prior, unvalidated on our hardware]. Ordering fixed by the
measured decode profile, not the survey. Apple-GPU no-low-precision-matrix-unit constraint carried
through (KV int8/int4 is storage/bandwidth only, no compute payoff — same as weights in ADR-072).

[ohdearquant]

SPEC-GATE: SIGNED + approve press (codex r2 clean, content read end to end). The split-verdict structure is the honest shape for this evidence state: two levers ranked on measured data (P1 #154 default-flip riding KV3's measured neutrality+correctness; P-low #148 correctly roadmap-gated since the paged engine serves zero traffic), and the int8/int4 family gated behind a minimal experiment rather than ranked on data that does not exist. The refutation handling is the standard I asked repeated: the survey's 16%-KV premise refuted with the causal mechanism (GDN-hybrid caps KV to 6-of-24/16-of-64 layers, measured <1% at 1k) while honestly flagging the 8K-vs-1k mismatch and requiring re-derivation from our own long-context point rather than overclaiming. The P2 experiment scope (K-only, int8, full-attn layers only, CPU dequant shim, no rotation, no fused kernel) is exactly minimal, and naming #491 as prerequisite instrument ties the trilogy+1 together correctly. Riders inherited: GPU flock on all benches; the experiment's PPL number and the ≥1 long-context point are the public artifacts. Dependent work may proceed on merge, in P-order.