kolibri1 CPU: n-gram spec-decode — the 12 tok/s target is cleared (14.0 tok/s bench, 8.2 realistic)

.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6.md

@@ -0,0 +1,19 @@
+ID: ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6
+Title: Probe: speculative decoding feasibility for kolibri1 CPU decode (n-gram draft acceptance measurement)
+Row: MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm
+State: CLOSED
+Kind: enhancement
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-11
+Updated: 2026-10-11
+Closed: 2026-10-11
+
+## Problem
+
+The CPU kolibri decode bench measures 5.76 tok/s (t32+32GiB cache+NEON) against a 12 tok/s developer target and the measured levers are nearly exhausted. Speculative decoding is the one remaining multiplier that does not change numerics (acceptance is exact: a draft token is accepted iff the target's greedy argmax agrees). kolibri has no MTP/nextn layer, so the cheap draft is absent. Candidates: (a) n-gram/prompt-lookup drafting (llama.cpp -draft style, no trained model), (b) a tiny external draft model. This probe must: (1) inventory the tree's spec-decode machinery and classify model-specific vs generic/portable parts; (2) trace the kolibri CPU decode path and locate the verify hook point; (3) measure n-gram (n=1..4) acceptance potential on the bench greedy chains + W3 goldens; (4) if contained, build a minimal bench-level prototype harness reporting measured accept rate and projected tok/s. Deliverable: feasibility report + measured acceptance estimate committed on row/kolibri-spec-probe; no push, no PR.
+
+## Resolution
+
+PROBE COMPLETE 2026-10-11 (branch row/kolibri-spec-probe, commit pending; report docs/bench-evidence/kolibri1-spec-probe-20261011.md). Findings: (1) PORTABILITY: the tree's n-gram spec-decode machinery is generic end to end — scheduler spec slots (scheduler.cpp:247-251,700-741,948-950,1226-1265), engine draft install/rollback (core.cpp:101-127), runner verify (runner.cpp:3546+ sample_tokens_with_rejection, greedy accept-iff-argmax per rejection_sampler.h:13-51), proposer (runner.cpp:4517-4570 -> ngram_proposer.cpp), and GPUModelRunner is the production runner on CPU (model_loader.h:897). Only the MTP/DFlash/DSpark DRAFTERS and the GDN/mamba spec-state widening are model-specific; n-gram needs neither. ONE blocker: LoadedEngine::MakeKVCacheMaybeSpec (model_loader.cpp:1756-1763) builds MakeQwen3_5KVCacheSpec for ANY spec config, ignoring the model's make_kv_cache; for kolibri it injects a MambaSpec+fa_draft group and the runner refuses at runner.cpp:1074 (pinned by test_kolibri1_ngram_spec, green). (2) MEASURED ACCEPTANCE (exact simulation with the production proposer over the W3 golden chains, pinned by test_kolibri1_ngram_accept, 90/90 green): k=5 window 1-4 -> 103 verify passes vs 248 plain steps (2.408x), 150/297 drafts accepted (50.5%); k=8 -> 2.583x; upstream default window 5/5 is worse (1.687x). Bench chain (128 distinct random prompt, 64-token 2-cycle chain captured from the bench run): k=5 -> 12 passes vs 63 (5.25x), 52/53 accepted (98.1%). (3) PROTOTYPE (contained, bench-level per the brief's fallback since the engine path is blocked by the one seam): test_kolibri1_spec_probe drives the production proposer + ONE production ForwardKolibri1Forward over the k+1 verify sequence on the real checkpoint — bench prompt: spec chain byte-identical to plain, 53/53 accepted (100%), decode 14.51 tok/s vs 4.92 plain (2.95x, CLEARS the 12 tok/s target); W3 goldens: 8/8 byte-identical, 146/270 accepted (54.1%), decode 8.99 vs 6.84 tok/s (1.31x, short-prompt diluted). Green 17/17. (4) Engine reach pinned: spec-OFF engine generates on CPU; spec-ON throws the precise refusal (test_kolibri1_ngram_spec, 6/6 green). (5) INTEGRATION PLAN (not done — serving wiring deliberately out of scope): dispatch MakeKVCacheMaybeSpec's widened branch on use_eagle() or a recurrent group in the model's own KV config (kolibri has none, so its make_kv_cache is already spec-correct for n-gram); then flip the pinned blocker test into the engine-level e2e gate with a row spec; exercise the --speculative-config flag once via the C ABI; tune k/window on serving-shaped traffic; batch>1 acceptance owed. Candidate (b) tiny external draft model NOT recommended. Baseline reproduced: bench 5.156 tok/s this run (task baseline 5.76), anchor 109726 PASS.