diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index b1c23dd378..7b2d4bc9b3 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -281,7 +281,7 @@ claims it. | `ENG-LOAD-DIRECT-UPLOAD` | Cut the SECOND copy out of a safetensors load: a weight the device consumes VERBATIM VIEWS the read-only shard mmap (`OwnedBytes::Borrow`, keep-alive carried on `StTensor::mapping`) instead of being copied into an owned host buffer, so `ResidentWeight` uploads straight from the file mapping and the load moves those bytes ONCE. Backend- and arch-agnostic: it lives in the shared dense loader helpers, so every arch that routes through them inherits it. Additive leaf below `LOAD-SAFETENSORS`; the windowed source-page release is preserved and now also runs after the device upload. Non-verbatim paths (transpose, dtype conversion, dequant, concatenation, GGUF load-time repack) are unchanged and keep their copy. Also lands the measurement half of issue #150: per-phase load timing plus host-copy / borrowed / device-upload byte counters (`VT_LOAD_STATS`) | T1 | one move per byte: streamed shard iterator `vllm/model_executor/model_loader/weight_utils.py:905-954`; immediate copy into the destination parameter `vllm/model_executor/models/utils.py:252-279`; driver `vllm/model_executor/model_loader/base_loader.py:43-82` @ `555967922` | refcounted mapping `src/vllm/model_executor/model_loader/safetensors_reader.cpp:46-80,231-262`; byte counters `:337-400`; `BorrowStTensorBytes` + post-upload adoption/release `src/vllm/model_executor/models/qwen3_5_weights.cpp`; qualifying call sites `include/vllm/model_executor/models/dense_weight_loaders.h` (`LoadBf16Direct`, `LoadCtNvfp4W4A16`, `LoadCtMxfp4W4A16`), `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp` (`LoadModelBf16Direct`, `LoadCtNvfp4Raw`), `src/vllm/model_executor/models/qwen3_5_weights.cpp` (35B `LoadBf16Direct`); upload counter `include/vllm/model_executor/models/dense_attn_block.h`; fp4 resident upload counter + post-upload residency `include/vllm/model_executor/models/dense_nvfp4_gemm.h` (`ResidentNvfp4`) and `src/vllm/model_executor/models/qwen3_5.cpp` (private `ResidentNvfp4`); shared checker text normalization `scripts/checker_text.py` | [mechanism gate](../tests/vllm/test_load_direct_upload.cpp#L1) 14/14, 183 assertions (178 through round 4) — the 6 borrow-mechanism cases, 4 post-upload residency cases that pin the adopt branch, its RELEASE-BEFORE-REASSIGN ordering and the `VT_ADOPT_DEVICE_BYTES` decoupling, 3 fp4-resident cases that pin `ResidentNvfp4`'s upload accounting, its `d_dev` publication, its adoption and the previously unexercised NON-borrowed/host-addressable regime, and 1 case that pins the GENERAL adopt branch's `MADV_DONTNEED` by OBSERVING RESIDENCY (`mincore()` over the host mirror's interior pages, glibc pinned to the sbrk arena so `free()` cannot return them by itself) — the RSS half of this row, which every value assertion was blind to. RED under eleven mutations, each applied alone with the binary rebuilt (a failed build aborts rather than re-running a stale one), tree md5-verified restored and re-GREEN after each. Mutations are named by the TEST CASE they land on, never by a line ref, because carried-forward line refs are how the counts below were twice recorded wrong: drop the size-identity check 1 case / 5 assertions; skip the borrow in `LoadBf16Direct` 2 / 7; delete the adopt branch 5 / 19; move the release after the `bytes` reassignment 5 / 7 (incl. the ordering assertions read from inside the munmap); move the release after the host-addressable early return but BEFORE the env one 2 / 3; move it after the `VT_ADOPT_DEVICE_BYTES` early return, i.e. after BOTH, 3 / 4; delete the general branch's `MADV_DONTNEED` 1 / 1; `ResidentNvfp4` drop both `AddDeviceUpload` 3 / 3; drop both `d_dev` publications 3 / 20; drop both `AdoptDeviceBytesAsHost` 3 / 16; all six at once 3 / 23. FOUR of those rows had been recorded wrong (1/1, 2/2, 3/7 and 3/3) — every one a count measured against an older suite and copied forward rather than re-measured; all eleven are measured on this tree. The `qwen3_5.cpp` duplicate of `ResidentNvfp4` is in an anonymous namespace no test can call, so it is held to the same invariant by [`scripts/check-fp4-resident-consistency.py`](../scripts/check-fp4-resident-consistency.py) (42-case mutation suite), which checks PER BUFFER inside each buffer's own `if (!w.d_)` upload block — body-wide matching let one surviving `AddDeviceUpload` satisfy both buffers, so dropping exactly one counter passed (reproduced against the real `qwen3_5.cpp` text: old checker exit 0, new exit 1). Nine drop-one/substitute-one mutations of the LIVE duplicate now go RED. Round 5 closed two holes in it. It matched RAW source, so a statement left behind as a COMMENT, inside `#if 0`, or inside `if (false)` read as present while the compiler saw a deletion; the clause matchers now run on [`scripts/checker_text.py`](../scripts/checker_text.py)'s `normalize_source`, which blanks all three IN PLACE so byte offsets and line numbers survive — `strip_comments` had existed as two byte-identical private copies (check-runner-routing-consistency.py, check-surface-coverage.py) and both now import the shared helper instead of a third copy being written. And the ordering clause required PUBLISH-before-ADOPT but not COPY-before-ADOPT, so a body whose adoption runs BEFORE its copy passed although the upload then reads pages the adoption already released; new clause (f) READ-FIRST covers it. WHICH MOTION produces that order is part of the claim, because the two are different mutations: SINKING `d.b.Copy(...)` to below `AdoptDeviceBytesAsHost` leaves the `d_dev` publication in place so ONLY clause (f) bites (MEASURED on the live duplicate, old exit 0 / new exit 1 — the 0/1 row below), whereas HOISTING the adoption above the copy also lifts it above the publication and trips the PRE-EXISTING clause (e), which the old checker already caught (MEASURED 1/1) and which is therefore no evidence for (f). The equivalent mutations of the SHARED `dense_nvfp4_gemm.h` copy are red at run time, MEASURED one at a time with the binary DELETED before each rebuild and the header md5-verified (`4665255f7af6f52254367f9118ad92ee`) before and after each: sink `packed`'s `Copy` 2 cases / 2 assertions, sink `scale`'s 2 / 2, sink BOTH 2 / 4 — the failures are `w..bytes.data()[0] == kSrcPattern` and `AllBytesMatchPattern(...)` in `fp4 resident: ResidentNvfp4 COUNTS its upload…` and `fp4 resident: a host-addressable device adopts an OWNED fp4 mirror too`, TWO per mutated buffer, so an ODD count is not obtainable; hoist `packed`'s adoption 3 / 8 and hoist both 3 / 16, wider because a null `d_dev` makes the adoption return immediately and also takes the NON-host-addressable case. This clause had been recorded as **2 cases / 3 assertions**, which is the count of the *release after the host-addressable early return, BEFORE the env one* row carried onto a different mutation — the same failure mode as the four rows above, found by a fifth reviewer and re-measured here rather than copied. MEASURED on the LIVE `qwen3_5.cpp` on disk, one mutation at a time, md5-verified restored after each (35b5ea250f490105d579f4ffb573aa36 before and after all eleven) — old checker exit / new checker exit: delete the packed adoption 1/1, `//` it out 0/1, `/* */` it out 0/1, `#if 0` around it 0/1, `if (false)` around it 0/1, `//` the packed upload counter 0/1, `//` the packed `d_dev` publication 0/1, `#if 0` around that publication 0/1, `if (false)` around it 0/1, SINK the packed `Copy` to below its adoption 0/1. That gate is STRUCTURAL: it guards the duplicate against DELETION — including deletion disguised as a comment, an `#if 0` or a never-taken branch — against gross substitution and against mis-ordering, NOT against corruption, and it does not model the preprocessor (extending it to arbitrary `#ifdef` conditions is DECLINED: `#ifdef VT_CUTLASS_NVFP4` around the live adoption stays exit 0 in both checkers, because a build configuration is not a disguised deletion). Run-time proof exists only for the shared copy. The `mincore()` residency case now also carries a RUNTIME allocator guard — `#if __GLIBC__` proves the headers, not that glibc's allocator is running — which goes red (1 case / 1 assertion) when a purging allocator is simulated; the `mallopt` calls leave glibc's `no_dyn_threshold` set for the process either way, MEASURED as not specific to `M_TRIM_THRESHOLD` (glibc 2.39, `mallinfo2().hblks` probe: no mallopt LIVE, `M_TRIM_THRESHOLD` DISABLED, `M_MMAP_MAX` DISABLED), so it is recorded rather than removed. Round-4 re-run on the dev box, CLEAN Vulkan/llvmpipe Release build: `test_load_direct_upload` 14/14 (178), `test_safetensors` 34/34 (79), `test_qwen36_weights` 7/7 (45), `test_vulkan_backend` 35/35 (2107), `test_backend_cross_device` 11/11 (132), `test_opt_paged_engine` **6/6 token-exact (96/96), 0 declines, device type 3**. Round-5 re-run on the same box, branch rebased onto `a0fa12c7`, all identical except `test_load_direct_upload` **14/14 (183)** — the five added assertions are the RSS-reclaim case's runtime allocator guard, no case-count change — plus `test_checker_text` 33/33 and `scripts/gen-vulkan-spirv.py --check` `committed SPIR-V is up to date` run with the pinned `~/tools/glslang-16.5.0/bin/glslang` (without it the script exits 1 on this tree, so the green is a real compile-and-compare and not a skip). Round 6 changed NO compiled file — this row, the spec, the `(f)` clause docstring and one checker-test name — and re-ran the whole battery on the branch rebased onto `e3cc4f64`, CLEAN Release Vulkan build, 0 warnings: every count above reproduces byte-for-byte, `check-fp4-resident-consistency` rc 0, its suite 42, `test_checker_text` 33, `test_check_surface_coverage` 46, `test_check_runner_routing_consistency` 31, SPIR-V check up to date, preflight all green. GB10 Vulkan gates on the changed tree: `test_vulkan_backend` 35/35 (2650), `test_backend_cross_device` 11/11 (132), `test_opt_paged_engine` **6/6 prompts token-exact (96/96), 0 declines, device type 3**. GB10 CUDA (cutlass+triton) full `ctest` **383/393**, BOTH SACRED gates PASS (`test_qwen36_paged_engine`, `test_qwen27_paged_engine`); all 10 failures reproduce with the same signature on a clean `origin/main` CUDA build. MEASURED 27B bf16 load **1.54x warm / 1.61x cold**, bytes moved **100.196 -> 81.260 GiB** | [specs/load-direct-upload.md](specs/load-direct-upload.md) | `ACTIVE` | `CLAIM-ENG-LOAD-DIRECT-UPLOAD` | | `LOAD-SAFETENSORS-DIRECT-DENSE` | Layer-bounded target-device loading for ordinary plain-BF16 Qwen3.5 dense safetensors on discrete CUDA; additive leaf below `LOAD-SAFETENSORS`, preserving windowed source release and owned-shard lifetime. Plain weights, stacked raw-NK owners, tied logits, logical resident state and same-queue layer staging are implemented and locally CUDA-gated. H32 Triton AOT, plain-BF16 decode graphs and ratio-4 FA2 repair the transplanted hot path; the broader row remains speed-gating | T1 | target-device model construction `vllm/model_executor/model_loader/base_loader.py:43-82`; incremental safetensors yield `vllm/model_executor/model_loader/weight_utils.py:820-954`; stacked/tied parameters `vllm/model_executor/models/qwen3_5.py:276-303,483-492` | load/dispatch `src/vllm/model_executor/models/qwen3_5_dense_weights.cpp:52-133,187-246,334-472`; discrete/unified classifier `src/vt/cuda/cuda_backend.cu:251-265`; plain execution/residency and graph selection `src/vllm/model_executor/models/qwen3_5.cpp`, `src/vllm/model_executor/models/qwen3_5_dense.cpp`; H32 recurrence `src/vt/cuda/cuda_gdn.cu`; ratio-4 FA2 `src/vt/cuda/cuda_flash_attn_fa2.cu`, `src/vt/cuda/cuda_paged_attn.cu`; exact benchmark corpus/output capture `examples/bench/bench_core.h:61-68,357-386,409-589`; reference metrics/token collector `tools/bench/vllm_closed_loop_metrics.py`; guarded driver and summarizer `tools/bench/run_qwen35_4b_compare.sh`, `tools/bench/summarize_qwen35_4b_compare.py` | [Real 4B graph/direct ON/OFF/eager gate](../tests/vllm/models/test_qwen35_plain_weights.cpp#L80) passes **3/3, 1672/1672**; [H32/GDN tests](../tests/vt/test_ops_gdn.cpp#L1) **10/10** flag and **66/66, 4242/4242** full; [paged-attention tests](../tests/vt/test_ops_paged_attn.cpp#L1354) **25/25, 454474/454474**. Final 18-leg root `/tmp/qwen35-main-final-fa2-20260725` plus stable vLLM confirmation: direct ON/OFF/vLLM-0.25 total **5769.99/5660.70/5849.80 tok/s**, output **638.03/625.94/646.85**, TPOT/ITL **43.72/43.84/38.55 ms**, peak PSS **2.406/8.592/7.662 GiB**, stable PSS **0.759/8.589/4.029 GiB**, VRAM **12850.7/12843.3/12942.7 MiB**. ON=OFF output IDs 128/128 every pair; ON is **+1.93%** total and cuts peak/stable PSS **72.0%/91.2%**. H32 AOT/graph/FA2 A/B gains are **+4.59%/+0.39%/+1.60%**. Graph-node trace has 453 launches and 200972 child kernels; local FA2 is 180.28 us/call vs vLLM 178.40, so the remaining **0.9864x** throughput and TPOT gap is host/engine-side. Sanitizer availability and external 27B/35B remain open | [plain-BF16 direct-load spike](specs/qwen35-plain-bf16-direct-load.md); [2026-07-25 evidence](../docs/bench-evidence/qwen35-4b-main-repair-20260725.md) | `GATING` | - | | `ENG-MOE-HOSTFREE` | MoE Marlin resident host-weight release: after `BuildMoeMarlinResident` uploads+repacks the routed experts to the device Marlin resident, free the per-expert fp4 HOST mirror (`OwnedTensor` packed+scale bytes) and `madvise(MADV_DONTNEED)` the pages back to the OS — returns the ~16.9 GiB steady 35B host double-store (`LoadNvfp4Raw` `MakeOwned` copies kept resident forever). Guarded to the committed Marlin path (`MarlinMoeEnabled()`; retained for the `VT_NVFP4_MARLIN=0` wmma fallback that re-reads them); `VT_MOE_HOST_FREE=0` A/B rollback. Realizes `release_host_weights_after_upload` for the dominant host consumer. **Item-2 (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`): the host-free decision is now CONSUMED from `GetPlatform(d.q.device.type).residency_policy()` via `vllm::platforms::ShouldReleaseHostWeights(policy, MarlinMoeEnabled()/*kernel-path*/, VT_MOE_HOST_FREE/*env*/)` — `CudaPlatform` flag flipped false→true (reproduces today EXACTLY); `MarlinMoeEnabled()` stays the orthogonal KERNEL-PATH safety gate.** | T0 | streamed target-device weight construction `vllm/model_executor/model_loader/base_loader.py:43-82`; residency capability `vllm/platforms/interface.py:134-229`; Marlin MoE in-place repack (no host mirror) `vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py:375-434` | free region `src/vllm/model_executor/models/qwen3_5.cpp:3743-3781`; `OwnedTensor::ReleaseHost` decl `include/vllm/model_executor/models/qwen3_5_weights.h:65` + impl (madvise+swap) `src/vllm/model_executor/models/qwen3_5_weights.cpp:24`; public hook `Qwen3_5Model::PrepareMarlinResident` `src/vllm/model_executor/models/qwen3_5.cpp:9140` | `tests/vllm/test_qwen36_weights.cpp:283` (`ReleaseHost` frees buffer+capacity) + `:314` (`PrepareMarlinResident` release under Marlin / retention under `VT_NVFP4_MARLIN=0`, DGX release 27/27 + retention 15/15); DGX A/B (`VT_MOE_HOST_FREE`) 35B STEADY serving PSS **20.17→3.53 GiB** (root `dgx:~/work/mem35-hostfree`); token-neutral 315/315 + 235/235; c2 smoke clean; memcheck clean. Whole-window load-phase PEAK bounded by the `ENG-MOE-LOADSTREAM` follow-up. Ledger [parity-ledger.md#L521](parity-ledger.md#L521) | [moe-marlin-host-free.md](specs/moe-marlin-host-free.md) | `DONE` | `ac77bec` | -| `ENG-MOE-LOADSTREAM` | 35B load-phase PEAK-PSS interleave (follow-up to `ENG-MOE-HOSTFREE`): the steady free returns the routed-expert host mirror only AFTER the whole model loads, so whole-window `peak_pss`/`peak_rss` (~19.8 GiB) is still set by all N layers' ~256 experts host-coexisting at load. DEFER the routed-expert host copies (`LoadQwen3_5Moe` loads each layer WITHOUT experts + installs a per-layer `load_layer_experts` streaming closure that owns the mmap'd shards) and materialize ONE layer's experts inside `PrepareMarlinResident` immediately before that layer's device Marlin build + host free — so at most one layer's experts coexist on the host (peak ~ one layer, not all N). Device residents byte-identical (same source bytes, same per-layer build order). Non-CUDA/`VT_NVFP4_MARLIN=0`/no-Marlin build falls back to bulk host materialization (their forward reads the host bytes). 27B is a different loader (`LoadQwen3_5Dense`, true-W4A4) → unaffected; GGUF/synthetic/borrowed pass no shards owner → eager. **Item-2 (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`): the per-layer interleave gate is now CONSUMED from `GetPlatform(queue.device.type).residency_policy()` via `vllm::platforms::ShouldInterleaveLoadStream(policy, MarlinMoeEnabled())` — reproduces the old `queue.device.type != kCUDA` OR `!MarlinMoeEnabled()` gate EXACTLY (unified/CPU retain-host ⇒ policy false ⇒ materialize-all fallback); the ~4 GiB load-peak win is preserved.** | T0 | streamed target-device construction `vllm/model_executor/model_loader/base_loader.py:43-82`; in-place Marlin repack, no host mirror `vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py:375-434`; residency capability `vllm/platforms/interface.py:134-229` | deferred field `include/vllm/model_executor/models/qwen3_5_weights.h:1163` (`Qwen3_5MoeWeights::load_layer_experts`); loader defer + closure `src/vllm/model_executor/models/qwen3_5_weights.cpp:331,399` (`LoadMoeExpertsInto`/`LoadQwen3_5Moe`); shared shards owner `include/vllm/model_executor/models/model_registry.h:60` + `src/vllm/model_executor/models/model_registry.cpp:219` (`ModelSource::FromSafetensorsOwned`) + `src/vllm/entrypoints/model_loader.cpp:2556` (`LoadedEngine::FromModelDir`); per-layer interleave + `MaterializeAllDeferredExperts` `src/vllm/model_executor/models/qwen3_5.cpp:4498,4508` (`PrepareMarlinResident`) | CPU coexistence-bound contract `tests/vllm/test_qwen36_weights.cpp:324` ("deferred routed-expert load: move-safe closure + bounded coexistence", peak==1 across the per-layer materialize→free loop, move-safe closure); clean `-Werror` CPU build 0 warn, full ctest (3 HTTP/engine-proc parallel-port flakes pass isolated), tools 164/164. **DGX PROVEN** (`~/work/vllm.cpp-mem35-loadstream` new vs `-parent` 7a1a6d6 eager, production flags CUTLASS sm120a+Marlin+FA2 sm_121a, one flock): 35B load-to-ready **peak RSS (VmHWM) 21.43 GiB → 4.19 GiB (−17.24 GiB / −80%, below vLLM 13.3 GiB)**; token BYTE-IDENTICAL both binaries — 35B `test_qwen36_paged_engine` 315/315 + 27B `test_qwen27_paged_engine` 235/235; **27B UNAFFECTED** (peak RSS 24.8 GiB, matches its baseline — dense loader, no deferral); `compute-sanitizer memcheck` on the deferred load path **0 errors / 315 assertions** (no use-after-free of the freed host bytes); weights unit 127 assertions (CPU coexistence peak==1 + DGX residency). `benchmark_binding=false` — the orchestrator re-grids the binding `peak_pss`/`peak_rss` axes to confirm the FAIL→PASS flip | [moe-expert-load-stream.md](specs/moe-expert-load-stream.md) | `ANCHOR-BACKFILL` | CLAIM-MEM35-LOADSTREAM | +| `ENG-MOE-LOADSTREAM` | 35B load-phase PEAK-PSS interleave (follow-up to `ENG-MOE-HOSTFREE`): the steady free returns the routed-expert host mirror only AFTER the whole model loads, so whole-window `peak_pss`/`peak_rss` (~19.8 GiB) is still set by all N layers' ~256 experts host-coexisting at load. DEFER the routed-expert host copies (`LoadQwen3_5Moe` loads each layer WITHOUT experts + installs a per-layer `load_layer_experts` streaming closure that owns the mmap'd shards) and materialize ONE layer's experts inside `PrepareMarlinResident` immediately before that layer's device Marlin build + host free — so at most one layer's experts coexist on the host (peak ~ one layer, not all N). Device residents byte-identical (same source bytes, same per-layer build order). Non-CUDA/`VT_NVFP4_MARLIN=0`/no-Marlin build falls back to bulk host materialization (their forward reads the host bytes). 27B is a different loader (`LoadQwen3_5Dense`, true-W4A4) → unaffected; GGUF/synthetic/borrowed pass no shards owner → eager. **Item-2 (2026-07-19, `CLAIM-BACKEND-PLATFORM-2`): the per-layer interleave gate is now CONSUMED from `GetPlatform(queue.device.type).residency_policy()` via `vllm::platforms::ShouldInterleaveLoadStream(policy, MarlinMoeEnabled())` — reproduces the old `queue.device.type != kCUDA` OR `!MarlinMoeEnabled()` gate EXACTLY (unified/CPU retain-host ⇒ policy false ⇒ materialize-all fallback); the ~4 GiB load-peak win is preserved.** | T0 | streamed target-device construction `vllm/model_executor/model_loader/base_loader.py:43-82`; in-place Marlin repack, no host mirror `vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py:375-434`; residency capability `vllm/platforms/interface.py:134-229` | deferred field `include/vllm/model_executor/models/qwen3_5_weights.h:1163` (`Qwen3_5MoeWeights::load_layer_experts`); loader defer + closure `src/vllm/model_executor/models/qwen3_5_weights.cpp:331,399` (`LoadMoeExpertsInto`/`LoadQwen3_5Moe`); shared shards owner `include/vllm/model_executor/models/model_registry.h:60` + `src/vllm/model_executor/models/model_registry.cpp:219` (`ModelSource::FromSafetensorsOwned`) + `src/vllm/entrypoints/model_loader.cpp:2576` (`LoadedEngine::FromModelDir`); per-layer interleave + `MaterializeAllDeferredExperts` `src/vllm/model_executor/models/qwen3_5.cpp:4498,4508` (`PrepareMarlinResident`) | CPU coexistence-bound contract `tests/vllm/test_qwen36_weights.cpp:324` ("deferred routed-expert load: move-safe closure + bounded coexistence", peak==1 across the per-layer materialize→free loop, move-safe closure); clean `-Werror` CPU build 0 warn, full ctest (3 HTTP/engine-proc parallel-port flakes pass isolated), tools 164/164. **DGX PROVEN** (`~/work/vllm.cpp-mem35-loadstream` new vs `-parent` 7a1a6d6 eager, production flags CUTLASS sm120a+Marlin+FA2 sm_121a, one flock): 35B load-to-ready **peak RSS (VmHWM) 21.43 GiB → 4.19 GiB (−17.24 GiB / −80%, below vLLM 13.3 GiB)**; token BYTE-IDENTICAL both binaries — 35B `test_qwen36_paged_engine` 315/315 + 27B `test_qwen27_paged_engine` 235/235; **27B UNAFFECTED** (peak RSS 24.8 GiB, matches its baseline — dense loader, no deferral); `compute-sanitizer memcheck` on the deferred load path **0 errors / 315 assertions** (no use-after-free of the freed host bytes); weights unit 127 assertions (CPU coexistence peak==1 + DGX residency). `benchmark_binding=false` — the orchestrator re-grids the binding `peak_pss`/`peak_rss` axes to confirm the FAIL→PASS flip | [moe-expert-load-stream.md](specs/moe-expert-load-stream.md) | `ANCHOR-BACKFILL` | CLAIM-MEM35-LOADSTREAM | | `LOAD-GGUF` | GGUF reader, dequantization, Qwen name transforms, embedded vocabulary | T0 | Pinned vLLM has no GGUF loader: `vllm/model_executor/model_loader/__init__.py:31-65`; compatibility reference is llama.cpp | `src/vllm/model_executor/model_loader/gguf_reader.cpp:302`; `src/vllm/model_executor/model_loader/gguf_dequant.cpp:223`; `src/vllm/model_executor/models/qwen3_5_gguf_weights.cpp:432`; `src/vllm/entrypoints/model_loader.cpp:240` | `tests/vllm/test_gguf.cpp:53`; `tests/vllm/test_gguf_dequant.cpp:25`; `tests/vllm/test_gguf_qwen36_loader.cpp:153`; real 35B `tests/parity/test_qwen36_gguf_engine.cpp:145` | `planned: specs/gguf-loader.md` | `PARTIAL` | - | | `LOAD-GGUF-MMPROJ` | A SECOND, `clip`-architecture GGUF projector file beside the language file, and the Qwen3-VL vision tower loaded out of it. **W1 LANDED (#821):** `EngineParams::mmproj_path` / `vllm_model_params.mmproj_path` (C ABI v22) / the server `--mmproj` flag name the file, `clip_mmproj_gguf.h` reads its `clip.*` metadata and `v.*` / `mm.*` tensors into the SHARED `multimodal::Qwen3VLVisionWeights`, and `LoadedEngine::vision_tower()` holds the result. The two-tensor temporal patch embedding (`v.patch_embd.weight` + `v.patch_embd.weight.1`) is INTERLEAVED per channel into the `[out, C*T*p*p]` conv3d operand the tower reads, and a file carrying only the first half is refused by name — the MuseGlimmer condition enforced rather than assumed. MuseGlimmer's own `MuseGlimmerRefuseMmproj` now has a PRODUCTION caller (routed on `clip.projector_type == "muse-glimmer"`), which is a change to MuseGlimmer's behaviour and is stated as one. **NOT supported:** no forward consumes the loaded tower yet — there is no multimodal request path on the C ABI and no GGUF image/video driver — and the COMMITTED 334-name manifest with its CI accounting is owed by `QUANT-QWEN38-27B-GGUF-ARM`, because the live confirmation below is env-gated on a NAS file and CI reads only the synthetic fixture. Auto-discovery of a sibling `mmproj*.gguf` is deliberately out of scope: a directory holding two unrelated models must not silently fuse them. First consumer is `Qwen3.8-27B` ([#821](https://github.com/mudler/vllm.cpp/issues/821)), whose `mmproj-BF16.gguf` (334 tensors, `clip.projector_type = qwen3vl_merger`, header-verified 2026-08-18: data end == file size 931,146,432) ships BOTH halves; MuseGlimmer's lacks the second | T0 | Pinned vLLM has no GGUF loader at all (`555967922`, `model_loader/__init__.py:33-49`), so the compatibility reference is llama.cpp `b10451` = `10bf611e5` (`PROJECTOR_TYPE_QWEN3VL`; the previously recorded `tools/mtmd/clip-impl.h:330` was read at the SUPERSEDED local fork `237ad9b96` and is owed re-anchoring, [#1003](https://github.com/mudler/vllm.cpp/issues/1003)) | the flag `EngineParams::mmproj_path` ([model_loader.h:1](../include/vllm/entrypoints/model_loader.h#L1)), the reader [clip_mmproj_gguf.cpp:1](../src/vllm/model_executor/models/clip_mmproj_gguf.cpp#L1), the open + refuse + read site `src/vllm/entrypoints/model_loader.cpp::LoadedEngine::FromModelDir` (the `.gguf` branch, after the device-fit refusal and BEFORE the tokenizer), the reader `src/vllm/model_executor/models/clip_mmproj_gguf.cpp::LoadQwen3VLVisionFromClipMmproj` + `::ClipMmprojVisionConfig` + `::RefuseUnsupportedClipMmproj`, the holder `include/vllm/entrypoints/model_loader.h::vision_tower` on `LoadedEngine`, the C face `include/vllm.h` `vllm_model_params.mmproj_path` (ABI v22) wired in `src/capi/vllm_c.cpp`, the server flag `src/vllm/entrypoints/openai/server_main.cpp` `--mmproj`, and the now-reached refusal `src/vllm/model_executor/models/muse_glimmer_gguf_weights.cpp::MuseGlimmerRefuseMmproj` | [test_clip_mmproj_gguf.cpp:1](../tests/vllm/models/test_clip_mmproj_gguf.cpp#L1) (9 cases / 272 assertions hermetic: the `clip.*` config mapping, the DeepStack discovery, the per-position patch-embedding interleave, every block/merger slot, and four refusals; the ninth is the LIVE confirmation, which skips loudly unless `VLLM_CPP_QWEN38_27B_MMPROJ` names the real `mmproj-BF16.gguf` and adds 43 assertions when it does — 334 consumed names == 334 shipped with nothing unread, the `clip.*` geometry, and the join checked at all 1,769,472 positions against F32 bytes read straight from the mmap. Renaming a tensor in the reader AND the fixture together leaves the hermetic gate green at 9/9 and reds only the live case, which is what the live case buys) and [test_gguf_mmproj_reach.cpp:1](../tests/vllm/entrypoints/test_gguf_mmproj_reach.cpp#L1) (6 cases / 19 assertions hermetic, all through `LoadedEngine::FromModelDir`; the sixth is the LIVE confirmation that the ENGINE holds the tower, and it skips loudly unless BOTH `VLLM_CPP_QWEN38_27B_GGUF` and `VLLM_CPP_QWEN38_27B_MMPROJ` name the real files, so the 19 assertions belong to the five hermetic cases). The two are separate targets on purpose: deleting the `LoadQwen3VLVisionFromClipMmproj` call site in `model_loader.cpp` reds the SECOND and leaves the FIRST fully green, which is the measured difference between gating a class and gating a capability. The port target for the vision config mapping is `src/vllm/model_executor/models/minimax_h3_vision_gguf.cpp::MiniMaxH3EncoderVisionConfig`, which builds the same `Qwen3VLVisionConfig` from `visual.*` | [quantized arms of Qwen3.8-27B](specs/qwen38-27b-quant-arms.md) | `PARTIAL` | - | | `LOAD-HF-BPE` | HF tokenizer.json byte-level BPE and incremental detokenization | T0 | `vllm/tokenizers/registry.py:176`; `vllm/tokenizers/hf.py:163`; `tests/tokenizers_/test_hf.py:18`; `tests/tokenizers_/test_detokenize.py:148` | `src/vllm/tokenizer/tokenizer.cpp:250,494,529`; `src/vllm/v1/engine/detokenizer.cpp:409` | `tests/vllm/test_bpe.cpp:226`; `tests/vllm/test_tokenizer_parity.cpp:66,74,82,90`; `tests/vllm/test_pretokenizer.cpp:130` | `planned: specs/hf-tokenizer.md` | `ANCHOR-BACKFILL` | - | diff --git a/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6.md b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6.md index b74f9009f5..fb7d9e275a 100644 --- a/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6.md +++ b/.agents/issues/MODEL-TEXT-kolibri-1-kolibri1-for-causal-lm/ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6.md @@ -7,7 +7,7 @@ GitHub: - Mirror: PENDING Availability: FULL Created: 2026-10-11 -Updated: 2026-10-11 +Updated: 2026-10-12 Closed: 2026-10-11 ## Problem @@ -17,3 +17,104 @@ The CPU kolibri decode bench measures 5.76 tok/s (t32+32GiB cache+NEON) against ## Resolution PROBE COMPLETE 2026-10-11 (branch row/kolibri-spec-probe, commit pending; report docs/bench-evidence/kolibri1-spec-probe-20261011.md). Findings: (1) PORTABILITY: the tree's n-gram spec-decode machinery is generic end to end — scheduler spec slots (scheduler.cpp:247-251,700-741,948-950,1226-1265), engine draft install/rollback (core.cpp:101-127), runner verify (runner.cpp:3546+ sample_tokens_with_rejection, greedy accept-iff-argmax per rejection_sampler.h:13-51), proposer (runner.cpp:4517-4570 -> ngram_proposer.cpp), and GPUModelRunner is the production runner on CPU (model_loader.h:897). Only the MTP/DFlash/DSpark DRAFTERS and the GDN/mamba spec-state widening are model-specific; n-gram needs neither. ONE blocker: LoadedEngine::MakeKVCacheMaybeSpec (model_loader.cpp:1756-1763) builds MakeQwen3_5KVCacheSpec for ANY spec config, ignoring the model's make_kv_cache; for kolibri it injects a MambaSpec+fa_draft group and the runner refuses at runner.cpp:1074 (pinned by test_kolibri1_ngram_spec, green). (2) MEASURED ACCEPTANCE (exact simulation with the production proposer over the W3 golden chains, pinned by test_kolibri1_ngram_accept, 90/90 green): k=5 window 1-4 -> 103 verify passes vs 248 plain steps (2.408x), 150/297 drafts accepted (50.5%); k=8 -> 2.583x; upstream default window 5/5 is worse (1.687x). Bench chain (128 distinct random prompt, 64-token 2-cycle chain captured from the bench run): k=5 -> 12 passes vs 63 (5.25x), 52/53 accepted (98.1%). (3) PROTOTYPE (contained, bench-level per the brief's fallback since the engine path is blocked by the one seam): test_kolibri1_spec_probe drives the production proposer + ONE production ForwardKolibri1Forward over the k+1 verify sequence on the real checkpoint — bench prompt: spec chain byte-identical to plain, 53/53 accepted (100%), decode 14.51 tok/s vs 4.92 plain (2.95x, CLEARS the 12 tok/s target); W3 goldens: 8/8 byte-identical, 146/270 accepted (54.1%), decode 8.99 vs 6.84 tok/s (1.31x, short-prompt diluted). Green 17/17. (4) Engine reach pinned: spec-OFF engine generates on CPU; spec-ON throws the precise refusal (test_kolibri1_ngram_spec, 6/6 green). (5) INTEGRATION PLAN (not done — serving wiring deliberately out of scope): dispatch MakeKVCacheMaybeSpec's widened branch on use_eagle() or a recurrent group in the model's own KV config (kolibri has none, so its make_kv_cache is already spec-correct for n-gram); then flip the pinned blocker test into the engine-level e2e gate with a row spec; exercise the --speculative-config flag once via the C ABI; tune k/window on serving-shaped traffic; batch>1 acceptance owed. Candidate (b) tiny external draft model NOT recommended. Baseline reproduced: bench 5.156 tok/s this run (task baseline 5.76), anchor 109726 PASS. + +INTEGRATION LANDED 2026-10-12 (branch row/kolibri-spec-engine). The probe's +one blocker is fixed and the n-gram proposer now runs in the kolibri ENGINE +path behind the existing opt-in (EngineParams::speculative_config — the same +object the C ABI v6 field and the server's --speculative-config populate). + +THE SEAM FIX (the only product change): LoadedEngine::MakeKVCacheMaybeSpec +(src/vllm/entrypoints/model_loader.cpp) now builds the MODEL's own KV spec +(ModelRegistry::MakeKVCache) and takes the widened MakeQwen3_5KVCacheSpec +branch only when spec->use_eagle() OR the model's own groups carry a +recurrent/mamba group. A draft-free n-gram config on a model with no recurrent +group keeps the model's own spec: the drafts are ordinary scheduled tokens, so +the target's groups are the whole cache need and NO draft-proposal group is +added (the fa_draft group is consumed only by the MTP draft head's own paged +KV, runner.cpp:1865-1898, a forward n-gram does not have — and that allocation +is skipped outright on kolibri's multi-cache topology). The default (no spec +config) path is byte-identical: exactly ModelRegistry::MakeKVCache. qwen3.5's +ngram path keeps the widened spec (it HAS GDN layers), so the landed 27B ngram +gate is untouched. RED-FIRST: with the fix reverted, the new seam unit fails +(REQUIRE( 3 == 2 ) — the injected fa+gdn+fa_draft set) and the e2e fails at +construction with the pinned refusal "vt: runner: MambaSpec state shapes must +be positive at runner.cpp:1074" (the spec-OFF arm runs all 9 prompts first, +0 drafts). GREEN with the fix: unit 21/21; e2e 83/83. + +NO NEW ENGINE WIRING WAS NEEDED: the scheduler spec slots, the engine's draft +install/rollback, the runner's verify (sample_tokens_with_rejection) and the +draft-free proposer (propose_drafts_ngram) were already generic; the e2e gate +proves the whole loop executed (53/53 drafts accepted on the bench prompt). +tests/vllm/models/test_kolibri1_ngram_spec is flipped from the pinned blocker +into (1) a checkpoint-free seam unit (kolibri under an ngram spec config keeps +its own 2 groups — no MambaSpec, no fa_draft, byte-identical to spec-off — +while qwen3.5 still widens to fa+gdn+fa_draft with the conv row at +(K-1)+k taps and num_speculative_blocks=k) and (2) the engine e2e gate. +tests/capi/test_capi.cpp gains the ABI reach pin, in TWO halves (2026-10-12 +review fix — the first shape pinned only the PARSE: a missing checkpoint fails +with VLLM_ERR_MODEL_LOAD whether or not the parsed config is forwarded, so it +could not see a dropped assignment). The PARSE pin: a valid ngram +speculative_config is ACCEPTED at the ABI layer (MODEL_LOAD on a missing +checkpoint, not INVALID_ARGUMENT), and an invalid ngram window is a CALLER +error (VLLM_ERR_INVALID_ARGUMENT, never MODEL_LOAD). The FORWARDING pin: a +drafter-chain document the ABI layer accepts is REFUSED BY NAME by the +engine's up-front ResolveSpecConfig, before any model I/O — so against a +missing checkpoint the load fails with the spec-validation refusal, not the +missing-directory error. Mutation-verified both ways: reverting the whole +parse+assignment block fails the parse pin; dropping only the assignment into +EngineParams fails the forwarding pin (the reviewer's exact mutation). + +MEASURED THROUGH THE ENGINE (LoadedEngine, t32 + 32 GiB dequant cache, under +flock, VT_ASYNC_SCHED=0 so BOTH arms run the synchronous scheduler — ngram +forces sync via async_scheduling_compatible()==use_eagle(); with the default +async spec-OFF arm the goldens A/B is confounded by pipelined input prep and +is reported separately): +- EXACTNESS: spec-ON == spec-OFF BYTE-IDENTICAL on 9/9 prompts (the + deterministic bench prompt 128-in/64-out + the 8 W3 golden prompts). The + spec-OFF bench chain ends at the pinned anchor 109726. +- ACCEPTANCE (engine counters spec_drafts_accepted/proposed): bench prompt + 53/53 — the pinned number (exact sim over the torch chain: 52/53; the probe + harness over this tree's bf16 chain: 53/53). Goldens aggregate 146/270 — + EXACTLY the probe harness's 146/270 (sim over torch chains: 150/297). The + engine reproduces the harness's acceptance counts exactly, per prompt too. +- ENGINE tok/s (CORRECTED DENOMINATOR, 2026-10-12 review fix): decode tok/s + divides the tokens generated INSIDE the decode interval — the full output + count less the tokens already present when the first output arrived — by + the interval from that first output to finish; wall tok/s keeps the full + count over the whole request. (The first report divided the FULL count by + that interval, billing the first step's tokens with no generation time; + the corrected decode figures read modestly lower — the operator's + overcounted verification measured 14.06 bench / 8.60 goldens decode.) + Re-measured 2026-10-12 on this box (t32 + 32 GiB cache, under flock, + VT_ASYNC_SCHED=0): bench prompt decode 4.53 -> 13.78 tok/s (3.04x; wall 3.04 + -> 5.89, 1.93x) — CLEARS the 12 tok/s target through the engine (harness + reference: 4.92 -> 14.51, 2.95x). Goldens aggregate decode 6.27 -> 8.13 + tok/s (1.30x; wall 5.46 -> 6.91, 1.27x; harness reference: 6.84 -> 8.99, + 1.31x; short-prompt diluted, same shape). Default-scheduler run for the + record: bench decode 4.80 -> 15.72 (wall 3.18 -> 6.34), goldens decode 6.35 + -> 8.98 (wall 5.39 -> 7.60) — the async spec-OFF arm's pipelined input prep + is the confound; not the A/B of record. The denominator fix is + REPORTING-ONLY: acceptance (53/53 bench, 146/270 goldens) and byte-identity + (9/9) are unchanged. +- W3 VERDICT: unchanged with spec off — test_kolibri1_w3 900/900 PASS (the + default path is byte-identical; the KV-spec unit pins it too) and the decode + bench anchor 109726 PASS. With spec ON: byte-identical to spec-OFF (9/9), so + the torch-golden relationship is identical by transitivity — 4/8 prompts + token-identical end to end with the same per-prompt shared prefixes the + probe measured (141/145 argmax positions, 4 adjudicated near-tie flips + unchanged). No hard flip. +- REGRESSION SURFACE: test_kolibri1_ngram_accept 90/90 (pinned sim stays + green); test_kolibri1_spec_probe (the harness) green; seam-adjacent suites + test_loaded_engine_dense 157, test_model_registry 1229, + test_deepseek_v4_scaffold 684, test_kv_pool_min_fit 16, test_runner 1918, + test_kv_cache_fp8_wiring 487 — all green (4491 assertions). + +REMAINING DELTA (owed, none blocking): (a) the engine's reject-heavy path — on +the most reject-heavy golden (10/84 acceptance) the verify T=1+kr forward +makes spec-ON ~1.8x SLOWER wall than plain on that prompt (10.20 s vs 6.81 s); +the aggregate still wins 1.20x. A per-step cost trace of the reject path +(scheduler rollback + draft re-splice + rejected-slot re-schedule) is the +named follow-up. (b) Batch>1 acceptance (probe plan step 6). (c) Real-traffic +(non-degenerate-chain) acceptance (probe §9 caveat: the measured rates are an +upper bound). (d) Serving-shaped k/window tuning — config, not code +(k=5, window 1-4 measured on these chains; upstream default 5/5 is worse). diff --git a/.agents/specs/kolibri-1-cpu.md b/.agents/specs/kolibri-1-cpu.md index 0707d536cd..b64de08117 100644 --- a/.agents/specs/kolibri-1-cpu.md +++ b/.agents/specs/kolibri-1-cpu.md @@ -367,6 +367,44 @@ the lane ON — the default flip is blocked on (1) the idle-host A/B and (2) a near-tie adjudication of that anchor (gate-semantics change, own spec). Evidence: docs/bench-evidence/kolibri1-neon-paged-attn-20261008.md. +N-GRAM SPEC DECODE LANDED ON THE ENGINE PATH (2026-10-12, branch +`row/kolibri-spec-engine`, issue ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6): the +spec-probe's one seam is fixed — `LoadedEngine::MakeKVCacheMaybeSpec` +(src/vllm/entrypoints/model_loader.cpp) builds the model's own KV spec and +takes the widened qwen3.5 branch only when `spec->use_eagle()` or the model's +own groups carry a recurrent/mamba group. A draft-free n-gram config on +kolibri keeps kolibri's own two groups (full + sliding attention): the drafts +are ordinary scheduled tokens, so the target's groups are the whole cache need +and no draft-proposal group is added (the `fa_draft` group serves only the MTP +draft head's paged KV, a forward n-gram does not have). The default (no spec +config) path is byte-identical. No new engine wiring was needed: +`EngineParams::speculative_config` (the C ABI v6 field / the server's +--speculative-config) already fed the generic scheduler slots, engine draft +pull, runner verify and n-gram proposer. `test_kolibri1_ngram_spec` is flipped +from the pinned blocker into a checkpoint-free seam unit (kolibri keeps its +own groups under an ngram spec; qwen3.5 still widens) plus the engine e2e +gate: spec-ON == spec-OFF BYTE-IDENTICAL on 9/9 prompts (bench prompt + +8 W3 goldens), acceptance exactly 53/53 on the bench prompt and 146/270 on +the goldens (the probe harness's numbers), and engine-level decode tok/s +4.53 -> 13.78 on the bench prompt (3.04x, CLEARS the 12 tok/s target; +harness reference 4.92 -> 14.51) and 6.27 -> 8.13 on the goldens (1.30x), +measured with the synchronous scheduler on BOTH arms (ngram forces sync; +VT_ASYNC_SCHED=0, t32 + 32 GiB cache, under the lock). Decode tok/s uses the +CORRECTED denominator (2026-10-12 review fix): the tokens generated inside +the decode interval — the full count less the tokens already present when +the first output arrived — over that interval; wall tok/s keeps the full +count (bench wall 3.04 -> 5.89; goldens wall 5.46 -> 6.91). The first report +divided the full count by the decode interval, overcounting decode by the +first step's tokens (the operator's overcounted verification: 14.06 bench / +8.60 goldens decode); the correction is reporting-only — acceptance +(53/53, 146/270) and byte-identity (9/9) are unchanged. W3 unchanged: 900/900 +with spec off; with spec ON the chains are byte-identical to spec-OFF, so the +torch-golden relationship (4/8 prompts end-to-end identical, 4 adjudicated +near-tie flips) is unchanged. Owed: the engine's reject-heavy path costs more +than the harness's on the lowest-acceptance golden (verify T=1+kr for ~0.5 +emitted tokens/pass; aggregate still wins); batch>1 acceptance; real-traffic +acceptance; serving-shaped k/window tuning (config, not code). + ## R7 resolution — the tokenizer engine accepts the Kolibri-1 split regex ### Scope — what is actually in the file diff --git a/include/vllm.h b/include/vllm.h index 51cdc825fb..aaf329933e 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -481,8 +481,10 @@ typedef struct vllm_model_params { * the emitted tokens under greedy sampling, so it is a throughput knob. * NULL or "" => speculation DISABLED, the byte-identical default engine. * A malformed document or unsupported method fails vllm_engine_load with - * VLLM_ERR_INVALID_ARGUMENT. Only MTP is supported today, on the Qwen3.5/3.6 - * checkpoints that ship an mtp.* head (safetensors only). Borrowed for the + * VLLM_ERR_INVALID_ARGUMENT. "mtp" needs a Qwen3.5/3.6 checkpoint that + * ships an mtp.* head (safetensors only); "ngram" is draft-free and runs on + * any target (measured on the kolibri1 CPU checkpoint: the speculative + * chain is byte-identical to speculative-off). Borrowed for the * vllm_engine_load call only. */ const char* speculative_config; /* ── Automatic prefix caching (ABI v7) ───────────────────────────────────── diff --git a/include/vllm/entrypoints/model_loader.h b/include/vllm/entrypoints/model_loader.h index b323cd6ffd..b13b551889 100644 --- a/include/vllm/entrypoints/model_loader.h +++ b/include/vllm/entrypoints/model_loader.h @@ -748,10 +748,15 @@ class LoadedEngine { static std::pair ResolveSchedulerBlockSizes( const vllm::v1::KVCacheConfig& kv_cfg, int block_size, bool enable_prefix_caching); - // SPEC-MTP I5d: build the KV-cache spec, widened for speculation when a spec - // config is set (the extra GDN k+1 state slots + widened conv row + the - // `fa_draft` full-attn group, MakeQwen3_5KVCacheSpec num_spec>0). With no spec - // config it is exactly ModelRegistry::MakeKVCache (byte-identical). + // SPEC-MTP I5d: build the model's own KV-cache spec (ModelRegistry::MakeKVCache) + // and, when a spec config is set AND the method or the model needs spec state + // (an eagle method, or the model's own groups carry a recurrent/mamba group), + // rebuild it widened (the extra GDN k+1 state slots + widened conv row + the + // `fa_draft` full-attn group, MakeQwen3_5KVCacheSpec num_spec>0). A draft-free + // method (ngram) on a model with no recurrent group keeps the model's own spec: + // the drafts are ordinary scheduled tokens, so the target's groups are the + // whole cache need. With no spec config this is exactly + // ModelRegistry::MakeKVCache (byte-identical). public: // // PUBLIC because it is the only entry to the KV geometry the engine actually diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 4a0b54c8fc..a1b62db2ad 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -1614,10 +1614,14 @@ std::optional LoadedEngine::ResolveSpecConfig( // SPEC-NGRAM (ROAD-V1-D3): the draft-FREE n-gram proposer. num_speculative_tokens // (k) is REQUIRED (speculative.py:1224-1234); the prompt_lookup window defaults to // 5/5. There is NO draft checkpoint to load and NO n_predict-module constraint — - // the drafts come from matching the sequence's own suffix. The GDN spec verify - // machinery (widened conv + k+1 state slots) is still needed and is reused via - // MakeQwen3_5KVCacheSpec(num_spec>0) exactly like MTP; the fa_draft draft-KV group - // it allocates is simply unused (ngram has no draft-model forward). + // the drafts come from matching the sequence's own suffix. On a model WITH a + // recurrent (GDN) group the spec verify machinery (widened conv + k+1 state + // slots) is still needed and MakeKVCacheMaybeSpec reuses it via + // MakeQwen3_5KVCacheSpec(num_spec>0) exactly like MTP; the fa_draft draft-KV + // group it allocates is simply unused (ngram has no draft-model forward). On a + // model WITHOUT one (kolibri) the model's own KV spec is already spec-correct: + // the drafts are ordinary scheduled tokens and the cache need is the target's + // groups alone. if (cli.method == "ngram") { if (!cli.num_speculative_tokens.has_value()) { throw std::invalid_argument( @@ -1753,15 +1757,31 @@ vllm::v1::KVCacheConfig LoadedEngine::MakeKVCacheMaybeSpec( block_size, resolved_bs); block_size = resolved_bs; } - vllm::v1::KVCacheConfig kv; - if (spec.has_value()) { + // SPEC-NGRAM / kolibri (ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6): build the + // MODEL's own KV spec first, always. The widened qwen3.5 spec below exists + // for exactly two needs: (a) an eagle method (mtp/dflash/dspark), whose + // draft-model forward writes its own K/V into the extra `fa_draft` + // full-attn group, and (b) a model with a recurrent (GDN/mamba) group, + // whose conv row widens to (K-1)+k taps and whose SSM state needs the k+1 + // snapshot slots a k+1 verify step rewinds. A draft-FREE method (ngram) on + // a model with NO recurrent group — kolibri: sliding/full attention only — + // needs neither: the drafts are ordinary scheduled tokens verified by the + // target's own forward, so the target's own groups are the whole cache need + // and there is no draft-proposal group to add. Building + // MakeQwen3_5KVCacheSpec for such a model instead injected a MambaSpec + // group with empty shapes (the checkpoint carries no linear_* keys) plus an + // unused fa_draft group, and the runner refused at construction + // (runner.cpp:1074 "MambaSpec state shapes must be positive"). The default + // (no spec config) path is untouched: exactly ModelRegistry::MakeKVCache, + // byte-identical. + vllm::v1::KVCacheConfig kv = + ModelRegistry::MakeKVCache(model, config, block_size, num_blocks); + if (spec.has_value() && (spec->use_eagle() || kv.has_mamba_layers())) { // Speculation is Qwen3.5/3.6-only at this pin (both gate checkpoints); build // the widened spec KV directly (extra GDN k+1 state slots + widened conv row // + the `fa_draft` full-attn group). MakeQwen3_5KVCacheSpec(num_spec>0). kv = vllm::MakeQwen3_5KVCacheSpec(config, block_size, num_blocks, spec->ResolvedNumSpeculativeTokens()); - } else { - kv = ModelRegistry::MakeKVCache(model, config, block_size, num_blocks); } // FIX-KV-GROUP-LAYER-COUNT (#1963, #1966). THE single funnel: both branches // above return through here, so one call reaches every architecture and both diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 40a5605980..28564a39d3 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -176,12 +176,14 @@ target_compile_definitions(test_kolibri1_w3 PRIVATE KOLIBRI1_GOLDENS="${CMAKE_CURRENT_SOURCE_DIR}/vllm/models/kolibri1_goldens.json") vllm_cpp_add_test(test_kolibri1_decode_bench vllm/models/test_kolibri1_decode_bench.cpp) target_include_directories(test_kolibri1_decode_bench PRIVATE ${CMAKE_SOURCE_DIR}/src) -# Kolibri-1 n-gram SPEC-DECODE e2e probe (ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6, -# branch row/kolibri-spec-probe): drives the FULL paged engine on CPU with +# Kolibri-1 n-gram SPEC-DECODE engine gate (ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6, +# branch row/kolibri-spec-engine): pins the FIXED KV seam at unit level (an +# ngram spec config keeps kolibri's own KV groups — no MambaSpec, no fa_draft — +# while qwen3.5 still widens), then drives the FULL paged engine on CPU with # speculative_config method=ngram and gates spec-ON == spec-OFF byte-exactness # while measuring acceptance and wall tok/s on the W3 golden prompts + the -# deterministic bench prompt. Checkpoint-gated (skips when the real model is -# absent); needs no GPU. +# deterministic bench prompt. The e2e case is checkpoint-gated (skips when the +# real model is absent); the seam unit runs everywhere. Needs no GPU. vllm_cpp_add_test(test_kolibri1_ngram_spec vllm/models/test_kolibri1_ngram_spec.cpp) target_include_directories(test_kolibri1_ngram_spec PRIVATE ${CMAKE_SOURCE_DIR}/src) target_compile_definitions(test_kolibri1_ngram_spec PRIVATE diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index 5e257d6c8d..955af99007 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -1682,6 +1682,79 @@ TEST_CASE("capi: malformed speculative_config is INVALID_ARGUMENT") { CHECK(eng == nullptr); } +// A VALID draft-free ngram document is ACCEPTED at the ABI layer: the load of +// a nonexistent checkpoint fails with VLLM_ERR_MODEL_LOAD — the code every +// FromModelDir failure reports — not VLLM_ERR_INVALID_ARGUMENT, which is what +// the ABI layer returns for a document it rejects before any model I/O. This +// pins the PARSE only. A missing checkpoint fails the same way whether or not +// the parsed config is forwarded into EngineParams, so this case cannot see a +// dropped assignment (the review finding on this test); the FORWARDING pin is +// the next case, and the engine-level proof that the config drives kolibri's +// spec-decode loop lives in test_kolibri1_ngram_spec. +TEST_CASE("capi: a valid ngram speculative_config reaches the engine load") { + vllm_model_params p = vllm_model_params_default(); + p.model_path = "/nonexistent/vllm-cpp/model/dir"; + p.speculative_config = + "{\"method\":\"ngram\",\"num_speculative_tokens\":5," + "\"prompt_lookup_min\":1,\"prompt_lookup_max\":4}"; + vllm_engine* eng = nullptr; + CHECK(vllm_engine_load(&p, &eng) == VLLM_ERR_MODEL_LOAD); + CHECK(eng == nullptr); +} + +// THE FORWARDING PIN for the speculative_config field (the review finding on +// the valid-ngram case above): that case proves only that the document PARSES. +// This case proves the parsed document is FORWARDED into the engine, through +// two documents a VALID model dir is not needed for: +// +// (a) THE PARSE PIN — an invalid n-gram window is a CALLER error: the ABI +// layer rejects it with VLLM_ERR_INVALID_ARGUMENT (the spec-validation +// error class), never VLLM_ERR_MODEL_LOAD, and the handle is null. If the +// field's parse were dropped, this document would fall through to the +// missing-directory load error instead — this assertion goes red. +// +// (b) THE FORWARDING PIN — a drafter-chain document is ACCEPTED by the ABI +// parse (the chain validates) and REFUSED BY NAME by the ENGINE's +// up-front ResolveSpecConfig (FromModelDir's first act, ahead of every +// path, config, tokenizer and weight operation — the ordering +// tests/vllm/entrypoints/test_drafter_chain_reach.cpp gates at the C++ +// level). Against the same nonexistent checkpoint the load therefore +// fails with the SPEC-VALIDATION refusal naming "vllm_cpp.drafter_chain", +// NOT the "model path is not a directory" error the engine reports when +// it never sees the document. Engine-side spec refusals surface through +// the load's generic catch, so the STATUS is VLLM_ERR_MODEL_LOAD either +// way (pinned deliberately: a genuine std::invalid_argument out of +// FromModelDir keeps reporting as VLLM_ERR_MODEL_LOAD) — the MESSAGE is +// the discriminator, and it goes red the moment the assignment of the +// parsed config into EngineParams is dropped. +TEST_CASE("capi: speculative_config is parsed AND forwarded into the engine") { + vllm_model_params p = vllm_model_params_default(); + p.model_path = "/nonexistent/vllm-cpp/model/dir"; + + // (a) An invalid n-gram window: a spec-validation error at the ABI layer. + p.speculative_config = + "{\"method\":\"ngram\",\"num_speculative_tokens\":5," + "\"prompt_lookup_min\":0,\"prompt_lookup_max\":4}"; + vllm_engine* eng = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&p, &eng) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(eng == nullptr); + CHECK(std::string(vllm_last_error()).find("prompt_lookup_min") != + std::string::npos); + + // (b) A well-formed drafter chain: parsed here, refused by the engine + // before any model I/O — the refusal proves the document was forwarded. + p.speculative_config = + "{\"vllm_cpp\":{\"drafter_chain\":[{\"method\":\"ngram\"," + "\"num_speculative_tokens\":5}]}}"; + vllm_engine* eng2 = reinterpret_cast(0x1); + CHECK(vllm_engine_load(&p, &eng2) == VLLM_ERR_MODEL_LOAD); + CHECK(eng2 == nullptr); + const std::string err = vllm_last_error(); + CHECK(err.find("vllm_cpp.drafter_chain") != std::string::npos); + CHECK(err.find("speculative-config") != std::string::npos); + CHECK(err.find("model path is not a directory") == std::string::npos); +} + // ─── version / abi ─────────────────────────────────────────────────────────── TEST_CASE("capi: version and abi-version are exposed") { CHECK(std::string(vllm_version()).size() > 0); diff --git a/tests/vllm/models/test_kolibri1_ngram_spec.cpp b/tests/vllm/models/test_kolibri1_ngram_spec.cpp index 7c89430c1f..d3ad015c82 100644 --- a/tests/vllm/models/test_kolibri1_ngram_spec.cpp +++ b/tests/vllm/models/test_kolibri1_ngram_spec.cpp @@ -1,32 +1,62 @@ -// Kolibri-1 n-gram spec-decode ENGINE-LEVEL reach probe (ISSUE-LOCAL- -// 01M4P22PX6BPRQAY1CK20FZQF6, branch row/kolibri-spec-probe). +// Kolibri-1 n-gram SPEC-DECODE engine gate (ISSUE-LOCAL-01M4P22PX6BPRQAY1CK20FZQF6, +// branch row/kolibri-spec-engine). // -// THE PRECISE BLOCKER, pinned. The tree's n-gram spec-decode machinery is -// generic engine code end to end (see docs/bench-evidence/ -// kolibri1-spec-probe-20261011.md §1) — EXCEPT one qwen35-shaped seam in the -// loader: LoadedEngine::MakeKVCacheMaybeSpec -// (src/vllm/entrypoints/model_loader.cpp:1756-1763) builds -// vllm::MakeQwen3_5KVCacheSpec for ANY speculative config, ignoring the -// model's own make_kv_cache factory. For kolibri (FullAttentionSpec + -// SlidingWindowSpec, no recurrent group) that injects a GDN/MambaSpec group -// plus an fa_draft group the model does not have, and the runner refuses at -// src/vllm/v1/worker/gpu/runner.cpp:1074 ("runner: MambaSpec state shapes -// must be positive") during engine construction. +// HISTORY — this file pinned the BLOCKER. The probe +// (docs/bench-evidence/kolibri1-spec-probe-20261011.md §5c) found exactly one +// qwen35-shaped seam on the engine path: LoadedEngine::MakeKVCacheMaybeSpec +// (src/vllm/entrypoints/model_loader.cpp) built vllm::MakeQwen3_5KVCacheSpec +// for ANY speculative config, ignoring the model's own make_kv_cache. For +// kolibri (FullAttentionSpec + SlidingWindowSpec, no recurrent group) that +// injected a GDN/MambaSpec group with empty shapes (the checkpoint carries no +// linear_* keys) plus an unused fa_draft group, and the runner refused at +// construction (runner.cpp:1074 "MambaSpec state shapes must be positive"). +// The probe's pinned blocker test asserted that refusal. THE SEAM IS NOW +// FIXED: the widened spec is taken only when the method or the model needs +// spec state (spec->use_eagle(), or the model's own groups carry a +// recurrent/mamba group); a draft-free n-gram config on a model with no +// recurrent group keeps the model's own KV spec — the drafts are ordinary +// scheduled tokens, so the target's groups are the whole cache need and there +// is no draft-proposal group to add (the fa_draft group is consumed only by +// the MTP draft head's own paged KV, runner.cpp:1865-1898, a forward n-gram +// does not have — and that allocation is skipped outright on kolibri's +// multi-cache topology anyway). // -// This test pins the blocker exactly, so the integration that removes it -// flips this test deliberately rather than silently: +// THIS FILE NOW PINS THE FIX AT TWO LEVELS: // -// (a) the spec-OFF engine loads AND generates on CPU for kolibri (the -// production serving path, PR #3422 — the probe proves it still runs); -// (b) the spec-ON engine construction throws the precise error above — -// proving the seam is the KV-spec branch and nothing else (the throw -// fires inside FromModelDir, before any scheduling). +// (1) THE SEAM UNIT (always runs, no checkpoint): under an ngram spec config +// MakeKVCacheMaybeSpec returns kolibri's OWN KV spec — two groups, +// full-attention + sliding-window, no MambaSpec, no fa_draft — +// byte-identical to the spec-OFF build; and the qwen3.5 arm of the +// dispatch still widens (the landed 27B ngram gate's spec is untouched). +// The pre-fix tree produced a third, MambaSpec group with empty shapes +// for kolibri — this case is red on it (red-first). // -// The MEASURED acceptance + tok/s prototype lives in the bench-level harness -// tests/vllm/models/test_kolibri1_spec_probe.cpp (production proposer + -// production verify forward), which does not depend on this seam. +// (2) THE ENGINE e2e GATE (checkpoint-gated, CPU): the same decode-bench +// workload through the FULL engine — spec-OFF vs spec-ON +// (method=ngram, k=5, window 1-4) over the deterministic bench prompt +// (128 in, 64 greedy out) and the 8 W3 golden prompts (32 greedy out): +// (i) EXACTNESS: the spec-ON chain == the spec-OFF chain +// byte-for-byte on every prompt; +// (ii) the spec-OFF bench chain ends at the pinned anchor 109726; +// (iii) ALIVENESS + ACCEPTANCE: the engine's acceptance counters +// match the pinned numbers (bench prompt 53/53 — the exact +// simulation over the torch chain pins 52/53 in +// test_kolibri1_ngram_accept, the probe harness measured 53/53 +// over this tree's bf16 chain, and the engine runs that chain); +// (iv) the W3 relationship is UNCHANGED by spec decode: spec-ON == +// spec-OFF byte-identically, so the spec-ON chains stand exactly +// where the spec-OFF chains stand against the torch goldens +// (the per-prompt shared prefixes are reported); +// (v) engine-level decode tok/s of both arms, reported (the probe's +// 14.51 spec / 4.92 plain tok/s are the harness-level reference). // -// Checkpoint-GATED: skips loudly when the real model is absent. CPU only. +// The pinned ACCEPTANCE SIMULATION (no model run) lives in +// tests/vllm/v1/spec_decode/test_kolibri1_ngram_accept.cpp; the bench-level +// PROBE HARNESS (production proposer + production verify forward) lives in +// tests/vllm/models/test_kolibri1_spec_probe.cpp. Both stay green. +// +// Checkpoint-GATED: the e2e case skips loudly when the real model is absent; +// the seam unit runs everywhere. CPU only. #include #include @@ -42,7 +72,10 @@ #include "vllm/config/speculative.h" #include "vllm/entrypoints/model_loader.h" +#include "vllm/model_executor/models/model_registry.h" #include "vllm/sampling_params.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/kv_cache_interface.h" #ifndef KOLIBRI1_GOLDENS #define KOLIBRI1_GOLDENS "tests/vllm/models/kolibri1_goldens.json" @@ -51,9 +84,8 @@ namespace { constexpr const char* kRealModelDir = "/mnt/models/Aleph-Alpha/Kolibri-1"; -// The runner refusal the qwen3.5-only KV branch triggers for kolibri. -constexpr const char* kExpectedBlocker = - "runner: MambaSpec state shapes must be positive"; +// The bench's token-identity anchor (captured on main 323f81ca6). +constexpr int32_t kAnchorLastToken = 109726; vllm::SamplingParams Greedy(int max_tokens) { vllm::SamplingParams sp; @@ -63,9 +95,153 @@ vllm::SamplingParams Greedy(int max_tokens) { return sp; } +// A LoadedModel that carries a registration and nothing else. The engine +// funnel reads only registration() on the way to the KV geometry (the +// test_deepseek_v4_scaffold.cpp pattern), so this enters the seam without a +// checkpoint. +class ScaffoldModel final : public vllm::LoadedModel { + public: + explicit ScaffoldModel(const vllm::ModelRegistration& r) : LoadedModel(r) {} +}; + +// The REAL checkpoint's config shape, trimmed to the KV geometry (the +// test_kolibri1.cpp MakeKolibri1Config values): hybrid 4-sliding + 1-full +// over 50 layers, 4 KV heads, head_dim 128, window 513. +vllm::HfConfig MakeKolibri1Config() { + vllm::HfConfig config; + config.model_type = "kolibri1"; + config.architectures = {"Kolibri1ForCausalLM"}; + config.hidden_size = 2560; + config.num_hidden_layers = 50; + config.vocab_size = 128000; + config.num_attention_heads = 48; + config.num_key_value_heads = 4; + config.head_dim = 128; + + nlohmann::json j; + j["head_dim"] = 128; + j["sliding_window"] = 513; + j["use_sliding_window"] = true; + j["rope_theta"] = 10000.0; + j["num_experts"] = 384; + j["num_experts_per_tok"] = 6; + j["moe_intermediate_size"] = 512; + j["shared_expert_intermediate_size"] = 512; + j["norm_topk_prob"] = false; + j["rms_norm_eps"] = 1e-6; + j["hidden_act"] = "silu"; + j["head_dtype"] = "float32"; + j["tie_word_embeddings"] = false; + nlohmann::json kinds = nlohmann::json::array(); + for (int64_t i = 0; i < 50; ++i) + kinds.push_back(i % 5 == 4 ? "full_attention" : "sliding_attention"); + j["layer_types"] = kinds; + config.raw = j; + return config; +} + +// A Qwen3.5-shaped hybrid config (the test_loaded_engine_dense.cpp +// MakeDenseConfig geometry): the GDN keys the widened spec reads, with +// layer_types [LA, LA, LA, FA]. +vllm::HfConfig MakeQwen35Config() { + vllm::HfConfig c; + c.model_type = "qwen3_5_text"; + c.architectures = {"Qwen3_5ForConditionalGeneration"}; + c.hidden_size = 32; + c.num_hidden_layers = 4; + c.vocab_size = 128000; + c.num_attention_heads = 6; + c.num_key_value_heads = 2; + c.head_dim = 8; + c.layer_types = {"linear_attention", "linear_attention", "linear_attention", + "full_attention"}; + c.linear_num_key_heads = 2; + c.linear_num_value_heads = 6; + c.linear_key_head_dim = 8; + c.linear_value_head_dim = 8; + c.linear_conv_kernel_dim = 4; + c.raw = nlohmann::json::object(); + return c; +} + } // namespace -TEST_CASE("kolibri1 engine reach: spec-OFF generates on CPU; spec-ON is blocked at the qwen3.5 KV branch") { +// ── (1) THE SEAM UNIT ──────────────────────────────────────────────────────── +TEST_CASE("kolibri1 KV seam: ngram spec keeps the model's own groups; qwen3.5 still widens") { + using vllm::v1::KVCacheSpecKind; + const vllm::SpeculativeConfig ngram = + vllm::SpeculativeConfig::ResolveNgram(/*k=*/5, /*min=*/1, /*max=*/4); + + // ── The kolibri arm: the model's OWN spec, byte-identical with and + // without the ngram spec config. + const vllm::HfConfig kcfg = MakeKolibri1Config(); + const vllm::ModelRegistration& kreg = vllm::ModelRegistry::Resolve(kcfg); + ScaffoldModel kmodel(kreg); + + const vllm::v1::KVCacheConfig kv_off = + vllm::entrypoints::LoadedEngine::MakeKVCacheMaybeSpec( + kmodel, kcfg, /*block_size=*/16, /*num_blocks=*/8, std::nullopt); + const vllm::v1::KVCacheConfig kv_on = + vllm::entrypoints::LoadedEngine::MakeKVCacheMaybeSpec( + kmodel, kcfg, /*block_size=*/16, /*num_blocks=*/8, ngram); + + // kolibri's own shape: TWO groups — full attention + sliding window — + // split 10/40 over the 50 hybrid layers (i%5==4 is full). + REQUIRE(kv_off.kv_cache_groups.size() == 2); + CHECK(kv_off.kv_cache_groups[0].kv_cache_spec->kind() == + KVCacheSpecKind::kFullAttention); + CHECK(kv_off.kv_cache_groups[1].kv_cache_spec->kind() == + KVCacheSpecKind::kSlidingWindow); + CHECK(kv_off.kv_cache_groups[0].layer_names.size() == 10); + CHECK(kv_off.kv_cache_groups[1].layer_names.size() == 40); + CHECK_FALSE(kv_off.has_mamba_layers()); + + // THE SEAM, pinned: under a spec config the spec is the SAME two groups. + // No injected MambaSpec (the pre-fix tree built MakeQwen3_5KVCacheSpec for + // any spec config, whose gdn group carries EMPTY shapes for a checkpoint + // with no linear_* keys, and the runner then refuses at runner.cpp:1074), + // and no fa_draft group (the n-gram drafts are ordinary scheduled tokens; + // fa_draft serves only the MTP draft head's own paged KV, a forward the + // n-gram proposer does not have). + REQUIRE(kv_on.kv_cache_groups.size() == 2); + CHECK_FALSE(kv_on.has_mamba_layers()); + for (size_t g = 0; g < 2; ++g) { + CHECK(kv_on.kv_cache_groups[g].kv_cache_spec->kind() == + kv_off.kv_cache_groups[g].kv_cache_spec->kind()); + CHECK(kv_on.kv_cache_groups[g].layer_names == + kv_off.kv_cache_groups[g].layer_names); + CHECK(kv_on.kv_cache_groups[g].kv_cache_spec->page_size_bytes() == + kv_off.kv_cache_groups[g].kv_cache_spec->page_size_bytes()); + } + + // ── The qwen3.5 arm: the SAME ngram spec config still widens — the landed + // 27B ngram gate's KV spec is untouched by the dispatch change. + const vllm::HfConfig qcfg = MakeQwen35Config(); + const vllm::ModelRegistration& qreg = vllm::ModelRegistry::Resolve(qcfg); + ScaffoldModel qmodel(qreg); + const vllm::v1::KVCacheConfig qkv = + vllm::entrypoints::LoadedEngine::MakeKVCacheMaybeSpec( + qmodel, qcfg, /*block_size=*/32, /*num_blocks=*/8, ngram); + // fa + gdn + fa_draft: the widened spec. + REQUIRE(qkv.kv_cache_groups.size() == 3); + const auto* gdn = dynamic_cast( + qkv.kv_cache_groups[1].kv_cache_spec.get()); + REQUIRE(gdn != nullptr); + // conv_kernel_size - 1 + num_spec taps (mamba_utils.py:226): the widened + // conv row the k+1 verify step rewinds into. conv_dim = 2*(2*8) + 6*8. + CHECK(gdn->shapes[0] == std::vector{80, 3 + 5}); + CHECK(gdn->num_speculative_blocks == 5); + // The draft layer's own paged KV: exactly one buffer's worth of names, at + // index num_hidden_layers (qwen3_5_mtp.py:105-112). + CHECK(qkv.kv_cache_groups[2].kv_cache_spec->kind() == + KVCacheSpecKind::kFullAttention); + REQUIRE(qkv.kv_cache_groups[2].layer_names.size() == 1); + CHECK(qkv.kv_cache_groups[2].layer_names[0] == + "model.layers.4.self_attn.attn"); +} + +// ── (2) THE ENGINE e2e GATE ───────────────────────────────────────────────── +TEST_CASE("kolibri1 ngram spec-decode e2e gate: spec-ON == spec-OFF through the engine") { if (!std::filesystem::exists(std::string(kRealModelDir) + "/model.safetensors.index.json")) { MESSAGE("SKIP: " << kRealModelDir << " not mounted"); @@ -77,59 +253,218 @@ TEST_CASE("kolibri1 engine reach: spec-OFF generates on CPU; spec-ON is blocked return; } const nlohmann::json g = nlohmann::json::parse(gf); - const std::vector prompt = - g.at("prompts")[0].at("input_ids").get>(); - constexpr int kMaxTokens = 8; - - // (a) The spec-OFF engine: loads and generates on CPU for kolibri. - vllm::entrypoints::EngineParams params_off; - params_off.max_num_seqs = 8; - std::unique_ptr off; - REQUIRE_NOTHROW(off = vllm::entrypoints::LoadedEngine::FromModelDir( - kRealModelDir, params_off)); - const auto t0 = std::chrono::steady_clock::now(); - off->engine().add_request("off", prompt, Greedy(kMaxTokens)); - std::optional final; - while (off->engine().has_unfinished_requests()) { - for (const vllm::RequestOutput& item : off->engine().step()) { - if (item.finished) final = item; + + // The workload: the deterministic decode-bench prompt (128 in, 64 greedy + // out) plus the 8 W3 golden prompts (32 greedy out each). + struct Job { + std::string name; + std::vector prompt; + int max_tokens; + }; + std::vector jobs; + { + std::vector bench; + for (int64_t i = 0; i < 128; ++i) + bench.push_back(static_cast((i * 7919 + 13) % 128000)); + jobs.push_back({"bench", bench, 64}); + } + for (size_t i = 0; i < g.at("prompts").size(); ++i) { + jobs.push_back({"golden[" + std::to_string(i) + "] \"" + + g.at("prompts")[i].at("prompt").get() + + "\"", + g.at("prompts")[i].at("input_ids") + .get>(), + 32}); + } + + struct Arm { + std::vector> chains; + std::vector accepted, proposed; // per job + std::vector wall_s, decode_s; // per job + // The DECODE numerator per job: the output tokens generated INSIDE the + // decode interval [first output, finish] — the full count less the tokens + // already present when the first output arrived. A spec step can emit + // several tokens at once, and those were generated before the interval + // opened, so the full count over the interval overstates decode throughput. + std::vector decode_tokens; // per job + }; + + const auto run_arm = [&](bool spec_on, Arm& arm) { + vllm::entrypoints::EngineParams params; + params.max_num_seqs = 8; + if (spec_on) { + params.speculative_config = vllm::ParseSpeculativeConfigJson( + "{\"method\":\"ngram\",\"num_speculative_tokens\":5," + "\"prompt_lookup_min\":1,\"prompt_lookup_max\":4}"); + } + std::unique_ptr eng; + // THE SEAM PROOF: construction SUCCEEDS under the spec config. The + // pre-fix tree threw "runner: MambaSpec state shapes must be positive" + // here — the probe's pinned blocker (this file's earlier shape). + REQUIRE_NOTHROW(eng = vllm::entrypoints::LoadedEngine::FromModelDir( + kRealModelDir, params)); + arm.chains.resize(jobs.size()); + arm.accepted.assign(jobs.size(), 0); + arm.proposed.assign(jobs.size(), 0); + arm.wall_s.assign(jobs.size(), 0.0); + arm.decode_s.assign(jobs.size(), 0.0); + arm.decode_tokens.assign(jobs.size(), 0); + for (size_t j = 0; j < jobs.size(); ++j) { + const int64_t acc0 = eng->runner().spec_drafts_accepted(); + const int64_t prop0 = eng->runner().spec_drafts_proposed(); + const auto t0 = std::chrono::steady_clock::now(); + std::optional t_first; + int64_t tokens_at_first = 0; + eng->engine().add_request("req" + std::to_string(j), jobs[j].prompt, + Greedy(jobs[j].max_tokens)); + std::optional final; + while (eng->engine().has_unfinished_requests()) { + for (const vllm::RequestOutput& item : eng->engine().step()) { + if (!t_first.has_value() && !item.outputs.empty() && + !item.outputs[0].token_ids.empty()) { + t_first = std::chrono::steady_clock::now(); + tokens_at_first = + static_cast(item.outputs[0].token_ids.size()); + } + if (item.finished) final = item; + } + } + const auto t1 = std::chrono::steady_clock::now(); + REQUIRE(final.has_value()); + REQUIRE(final->outputs.size() == 1); + arm.chains[j] = final->outputs[0].token_ids; + REQUIRE(arm.chains[j].size() == + static_cast(jobs[j].max_tokens)); + arm.wall_s[j] = std::chrono::duration(t1 - t0).count(); + if (t_first.has_value()) { + arm.decode_s[j] = + std::chrono::duration(t1 - *t_first).count(); + } + // The corrected decode numerator: tokens generated inside the decode + // interval only. The wall numerator keeps the FULL count (below). + arm.decode_tokens[j] = + static_cast(arm.chains[j].size()) - tokens_at_first; + arm.accepted[j] = eng->runner().spec_drafts_accepted() - acc0; + arm.proposed[j] = eng->runner().spec_drafts_proposed() - prop0; + MESSAGE((spec_on ? "spec-ON " : "spec-OFF ") << jobs[j].name + << ": wall " << arm.wall_s[j] << " s, decode " + << arm.decode_s[j] << " s, drafts accepted " + << arm.accepted[j] << "/" << arm.proposed[j]); + } + }; + + Arm off, on; + run_arm(/*spec_on=*/false, off); + // The default engine never drafts. + for (size_t j = 0; j < jobs.size(); ++j) CHECK(off.proposed[j] == 0); + run_arm(/*spec_on=*/true, on); + + // (i) EXACTNESS: the spec-ON chain == the spec-OFF chain byte-for-byte on + // every prompt. This is the hard gate: greedy spec decode is + // exactness-preserving, and a divergence here would be a real defect in + // the verify wiring, not a tolerance question. + int exact = 0; + for (size_t j = 0; j < jobs.size(); ++j) { + if (on.chains[j] == off.chains[j]) { + ++exact; + } else { + size_t shared = 0; + while (shared < on.chains[j].size() && shared < off.chains[j].size() && + on.chains[j][shared] == off.chains[j][shared]) { + ++shared; + } + MESSAGE("DIVERGENCE on " << jobs[j].name << " at position " << shared); } + CHECK(on.chains[j] == off.chains[j]); } - const auto t1 = std::chrono::steady_clock::now(); - REQUIRE(final.has_value()); - REQUIRE(final->outputs.size() == 1); - const std::vector& chain = final->outputs[0].token_ids; - MESSAGE("spec-OFF engine generated " << chain.size() << " tokens in " - << std::chrono::duration(t1 - t0).count() - << " s on CPU; first tokens:"); - { - std::string s; - for (size_t i = 0; i < chain.size() && i < 8; ++i) - s += std::to_string(chain[i]) + ","; - MESSAGE(" " << s); + CHECK(exact == static_cast(jobs.size())); + + // (ii) The spec-OFF bench chain ends at the pinned anchor (the engine + // reproduces the bench's captured chain). + CHECK(off.chains[0].back() == kAnchorLastToken); + + // (iii) ACCEPTANCE. The bench prompt: the exact simulation over the torch + // chain pins 52/53 (test_kolibri1_ngram_accept, k=5 window 1-4); the probe + // harness over THIS TREE's bf16 chain measured 53/53 in 12 verify passes. + // The engine runs that same chain through the same proposer and the same + // greedy accept rule, so its counters must land on 53/53 — a materially + // different count means the engine's draft scheduling, not the drafter, + // is misbehaving. + CHECK(on.proposed[0] == 53); + CHECK(on.accepted[0] == 53); + // The W3 goldens: the probe harness measured 146/270 (54.1%) aggregate on + // these chains; the engine must land in the same neighborhood (a band, not + // a pin: the near-tie chains are the sensitivity, and the bench prompt + // carries the strict pin). + int64_t g_acc = 0, g_prop = 0; + for (size_t j = 1; j < jobs.size(); ++j) { + g_acc += on.accepted[j]; + g_prop += on.proposed[j]; + } + MESSAGE("goldens aggregate: accepted " << g_acc << "/" << g_prop + << " (probe harness: 146/270; simulation on torch chains: 150/297)"); + CHECK(g_acc >= 140); + CHECK(g_acc <= 152); + CHECK(g_prop >= 264); + CHECK(g_prop <= 276); + CHECK(g_acc > 0); + + // (iv) THE W3 RELATIONSHIP, unchanged by spec decode: spec-ON == spec-OFF + // byte-identically (gate i), so the spec-ON chains stand exactly where the + // spec-OFF chains stand against the torch goldens. The per-prompt shared + // prefixes are reported (the probe measured 4/8 prompts token-identical to + // the torch golden end to end, the other 4 flipping at a near-tie early). + int full_match = 0; + for (size_t j = 1; j < jobs.size(); ++j) { + const std::vector golden = + g.at("prompts")[j - 1].at("generated_ids").get>(); + size_t shared = 0; + while (shared < off.chains[j].size() && shared < golden.size() && + off.chains[j][shared] == golden[shared]) { + ++shared; + } + if (shared == golden.size()) ++full_match; + MESSAGE("W3 " << jobs[j].name << ": engine chain shares " << shared << "/" + << golden.size() << " tokens with the torch golden (spec-ON " + << "identical to spec-OFF by gate (i))"); + } + MESSAGE("W3 relationship: " << full_match << "/8 prompts token-identical to " + << "the torch golden end to end (probe: 4/8)"); + + // (v) ENGINE-LEVEL TOK/S, reported (never asserted: machine load may move + // it — the decode bench's own convention). Wall tok/s divides the FULL + // output count by the whole request. Decode tok/s divides the tokens + // generated INSIDE the decode interval — the full count LESS the tokens + // already present when the first output arrived — by the interval from + // that first output to finish. (An earlier pairing divided the full count + // by that interval, billing the first step's tokens — several at once under + // spec decode — with no generation time; the corrected denominator counts + // only tokens the interval actually produced, so the corrected decode + // figures read modestly LOWER than the overcounted ones.) + const auto tok_s = [](double seconds, int64_t tokens) { + return seconds > 0.0 ? static_cast(tokens) / seconds : 0.0; + }; + MESSAGE("ENGINE bench prompt: spec-OFF wall " << tok_s(off.wall_s[0], 64) + << " tok/s, decode " + << tok_s(off.decode_s[0], off.decode_tokens[0]) + << " tok/s | spec-ON wall " << tok_s(on.wall_s[0], 64) + << " tok/s, decode " << tok_s(on.decode_s[0], on.decode_tokens[0]) + << " tok/s (probe harness reference: 4.92 plain / 14.51 spec " + "decode tok/s)"); + double off_g_wall = 0.0, off_g_dec = 0.0, on_g_wall = 0.0, on_g_dec = 0.0; + int64_t off_g_dec_tok = 0, on_g_dec_tok = 0; + for (size_t j = 1; j < jobs.size(); ++j) { + off_g_wall += off.wall_s[j]; + off_g_dec += off.decode_s[j]; + on_g_wall += on.wall_s[j]; + on_g_dec += on.decode_s[j]; + off_g_dec_tok += off.decode_tokens[j]; + on_g_dec_tok += on.decode_tokens[j]; } - CHECK(chain.size() == static_cast(kMaxTokens)); - // The spec-OFF arm is the production default: no drafts exist. - CHECK(off->runner().spec_drafts_proposed() == 0); - off.reset(); - - // (b) The spec-ON engine: construction throws at the qwen3.5-only KV - // branch. The message is pinned so a change to the seam (the integration, - // or an accidental behavior change for qwen3.5) turns this red. - vllm::entrypoints::EngineParams params_on; - params_on.max_num_seqs = 8; - params_on.speculative_config = vllm::ParseSpeculativeConfigJson( - "{\"method\":\"ngram\",\"num_speculative_tokens\":5," - "\"prompt_lookup_min\":1,\"prompt_lookup_max\":4}"); - std::unique_ptr on; - REQUIRE_THROWS_WITH_AS( - on = vllm::entrypoints::LoadedEngine::FromModelDir(kRealModelDir, - params_on), - doctest::Contains(kExpectedBlocker), std::exception); - MESSAGE("spec-ON construction blocked at the pinned seam: \"" - << std::string(kExpectedBlocker) << "\""); - MESSAGE("integration owed: dispatch MakeKVCacheMaybeSpec's widened branch " - "on \"method needs spec state\" (use_eagle()) or \"the model's own " - "KV config has a recurrent group\" — kolibri has none, so its own " - "make_kv_cache is already spec-correct for n-gram"); + MESSAGE("ENGINE goldens aggregate: spec-OFF wall " + << tok_s(off_g_wall, 8 * 32) << " tok/s, decode " + << tok_s(off_g_dec, off_g_dec_tok) << " tok/s | spec-ON wall " + << tok_s(on_g_wall, 8 * 32) << " tok/s, decode " + << tok_s(on_g_dec, on_g_dec_tok) + << " tok/s (probe harness reference: 6.84 plain / 8.99 spec)"); }