docs(adr): ADR-075 — MoE expert-offload priority (Metal)

[oceanwaves630]

PR description authored by Claude (Anthropic agent) on behalf of @oceanwaves630.

ADR-075 — MoE expert-offload priority (Metal)

Fifth sequencing ADR in the bundle set, after ADR-071 (GDN-state), ADR-072 (weight-quant), ADR-073 (KV-quant), ADR-074 (MTP). Docs-only.

Verdict: NEEDS-EXPERIMENT, reframed — gated, not ranked

The question as commissioned — rank the open MoE expert-offload levers — presupposes a working, measured, resident MoE serving path whose next optimization is offload. That precondition is false on the current head, and the reasons are the finding.

Load-bearing facts (all verified against origin/main @ c9ae21a9):

• Lattice ships exactly one MoE preset (qwen36_35b_a3b). The large model it actually serves and has measured (~4 tok/s) is qwen36_27b, which is dense — so no measured number this engine has ever produced touches MoE.
• The Metal MoE dispatch (ADR-053, merged PR feat(inference): MoE Metal dispatch with expert coalescing (ADR-053) #57) is wired into the live forward loop but has zero test/bench/measurement (encode_moe_ffn has one production call site, appears in no test), and runs only at f16 — Q4/QuaRot, Q8-CPU, and Q8-NEON all three explicitly refuse MoE.
• The engine constructor takes a fully-loaded &ModelWeights, so the CPU loader materializes the full model as f32 (~120 GiB) before any Metal buffer is built — a loader-sequencing ceiling that makes the model not loadable on any consumer Apple Silicon today, independent of offload.

Measured correction of a merged ADR (ADR-053)

ADR-053 quotes "~15 GiB" resident routed-expert weights and reasons its 32 GB feasibility from it — but it says "At Q4 with the current config dimensions," and Q4 was never implemented for MoE (the three refusal points above). The only precision that runs is f16, so the real resident footprint is 60 GiB — 4× the ADR-053 figure. This is recorded as an amendment here (ADR-053 stays an immutable record); its dispatch/routing design is unaffected, only its resident-memory arithmetic and the "borderline on 32 GB" claim are corrected (it's not borderline — it doesn't fit).

Structure

A Measured / source-verified reality table (E1–E12, each tagged source-read / unit-test-pinned / runtime-measured with a durable pointer) plus a derived memory-arithmetic table, held apart from an untrusted external prior-art survey that is confirmed repo-blind (targets a nonexistent crates/engine crate and a GGUF loader lattice doesn't have) and scoped to a different model family (DeepSeek-V4-Flash). The survey's arithmetic is unranked until re-derived on lattice's real 6.0 MiB/expert f16 numbers; its one convergent conclusion (build and measure the resident dispatch before adding disk streaming) matches the source-read verdict.

Decision

Gates the whole offload family (#682 the literal lever, #686 the epic, FP4 #683 / FP8 #684 adjacent) behind a minimal experiment:

• Step 0: confirm the loader-sequencing ceiling (instrument an actual MoE-preset load); if confirmed, fixing it (stream layer-by-layer) lowers the memory floor before any offload code.
• Step 1: drive a synthetic small MoE (4 layers, 16 experts) through the real MetalQwen35Engine for the first-ever measured MoE dispatch correctness + latency, under GPU flock, committed as an artifact.
• Step 2: only then re-derive the offload hit-rate arithmetic on real numbers, at which point MoE expert weight offloading: run models larger than RAM by streaming cold experts from disk #682's design becomes decidable.

Honest framing: this is not "REFUTED — wrong architecture" (MoE exists and is wired); it's "the lever is real, but the measurement floor beneath it does not exist yet." Includes a forward instruction (mirroring ADR-074's rider): when Step 1's bench lands, its numbers amend both this ADR's table and ADR-053's corrected figure with runtime-measured rows.

Testing

Docs-only. No executable behavior changes.

[ohdearquant]

SPEC-GATE: SIGNED (Leo). The reframe is the honest verdict: 'gated, not ranked' is correct when the resident path beneath every offload lever has never produced a token, and refusing to rank on a repo-blind survey scoped to a different model family is exactly right. The ADR-053 amendment is the house pattern done well — immutable record untouched, only the memory arithmetic and 32GB-feasibility corrected, with the causal mechanism (ADR-044 deferral enforced in the converter, no MoE quantizer ever landed) rather than a bare number. The codex-narrowed loadability claim (not loadable on 32-128 GB M-series/M-Max; Ultra-class unproven pending Step 0) survives my read at all three sites. Rider: Step 0's result lands as a new S-row on this ADR whichever way it goes — if from_safetensors turns out to stream internally, the 120 GiB peak claim gets corrected the same way this ADR corrected ADR-053. Proceed to G on merge.