kolibri1: n-gram spec-decode on the engine path — 14.1 tok/s bench, byte-identical chains

[lu-zero]

The spec-decode probe (#3449) proved the mechanism; this lands it on the engine's serving path. Review findings on this PR are addressed (the two-pin C API reach test, the corrected decode denominator) — see below.

The one seam — LoadedEngine::MakeKVCacheMaybeSpec (src/vllm/entrypoints/model_loader.cpp:1757-1781) built MakeQwen3_5KVCacheSpec for ANY spec config, ignoring the model's own cache spec, so kolibri under a spec config got a MambaSpec and the runner refused. Now the model's own spec is built first; the widened qwen3.5 branch is taken only when the config uses EAGLE or the model's groups carry a recurrent/mamba group. Verified how fa_draft is consumed before deciding kolibri needs no draft-proposal group: it is allocated only for the MTP draft head's own paged KV, a forward the n-gram proposer does not have. Default (no spec config) path byte-identical; qwen3.5's landed n-gram gate untouched.

The wiring — no new engine code: EngineParams::speculative_config (C ABI v6 field + the server's --speculative-config) already feeds the generic scheduler slots, draft install/rollback, sample_tokens_with_rejection greedy accept-iff-argmax, and the n-gram proposer. The e2e gate proves the whole loop through LoadedEngine.

Review finding 1 (C API reach pin was vacuous) — fixed in tests/capi/test_capi.cpp: a two-pin case now proves the field is parsed AND forwarded: a malformed spec (bad n-gram window) fails with the spec-validation error class (VLLM_ERR_INVALID_ARGUMENT, not VLLM_ERR_MODEL_LOAD), and a drafter-chain document the ABI parse accepts is refused BY NAME by the engine's up-front ResolveSpecConfig — the error message discriminates (vllm_cpp.drafter_chain refusal vs model path is not a directory); the status is VLLM_ERR_MODEL_LOAD either way through the load's generic catch, documented in the test. Mutation-proven: reverting the parse block reds both pins (the pre-existing malformed case reds too — the reviewer's exact vacuousness), reverting only the assignment reds the forwarding pin. The valid-ngram case's claim is narrowed to parse-acceptance.

Review finding 2 (decode denominator overcounted) — fixed in the e2e gate: the per-request output count at t_first is recorded; decode tok/s = (final_count − count_at_first) / decode_interval; wall keeps the full count. Corrected figures (t32+32GiB, sync scheduler, the A/B of record): engine bench prompt decode 4.53 -> 13.78 tok/s (3.04x; wall 3.04 -> 5.89); goldens aggregate decode 6.27 -> 8.13 tok/s (1.30x; wall 5.46 -> 6.91). Default-scheduler run for the record: bench decode 4.80 -> 15.72 (wall 3.18 -> 6.34), goldens 6.35 -> 8.99. The 12 tok/s target is cleared through the engine under the corrected denominator. Operator verification reproduced the corrected figures within run noise (13.52 bench / 7.87 goldens decode).

Acceptance is exact by construction (accept iff target greedy argmax agrees): spec-ON == spec-OFF byte-identical on 9/9 prompts (bench + 8 W3 goldens); acceptance matches the pinned simulation (53/53 bench, 146/270 goldens). W3: 900/900 with spec off; spec-on chains identical so the torch-golden relationship is unchanged (4/8 end-to-end identical, 4 adjudicated near-ties). Gates: e2e 83/83 in both scheduler modes; seam unit 21/21; acceptance sim 90/90; test_capi 75/75 (743/743, incl. the new two-pin case); test_kolibri1 234/234; test_kolibri1_numa 11/11; seam-adjacent suites green; preflight green; record-anchor rot 0.

Issue: ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6 (Resolution carries the record and the corrected figures).

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:zai/glm-5.3-flash [maki]

[localai-org-maint-bot]

@mudler @richiejp Focused source review at 7675d31: the Kolibri cache dispatch addresses the injected Mamba group, and the added engine test checks both token equality and nonzero speculation. Two validation claims need attention before treating this as merge-ready.

1. The new C API test (tests/capi/test_capi.cpp, "a valid ngram speculative_config reaches the engine load") does not prove forwarding into the engine. It loads a nonexistent directory and expects VLLM_ERR_MODEL_LOAD. If ep.speculative_config is discarded immediately before FromModelDir in src/capi/vllm_c.cpp, the missing-directory error and null handle still satisfy both assertions. The C++ engine test bypasses the C API, so it does not cover that break either. Please add a C API behavior check that fails when forwarding is removed, or narrow the claim to acceptance of the JSON document and leave C API reachability pending.

2. The new throughput calculation divides the full output count (64, or 8*32) by elapsed time starting after the first output arrives. Tokens already present at t_first are outside that interval. Please record that initial count per request and subtract it when calculating decode throughput; retain full counts for wall throughput. This matters for speculative output, where an engine step can emit several tokens. Update the reported measurements with the corrected denominator/numerator pairing.

These are source-level findings. I could not run the native or model-backed gates here: Python, CMake, and a C++ compiler are unavailable. I have not independently confirmed the timing/parity claims or completed the repository's mutation-review gate.