localai-org-maint-bot · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026
diff --git a/.agents/issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md b/.agents/issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md
@@ -0,0 +1,22 @@
+ID: ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3
+Title: Align Kolibri-1 public documentation with current code
+Row: -
+State: OPEN
+Kind: docs
+GitHub: -
+Mirror: PENDING
+Availability: FULL
+Created: 2026-10-12
+Updated: 2026-10-12
+Closed: -
+
+## Problem
+
+The feature row still describes dequant-at-load and lists the TT arm as owed. README news and USAGE omit the reachable Kolibri-1 path and its validation limits.
+
+## Resolution
+
+2026-10-12: The documentation branch corrects the loader and device claims,
+adds CPU usage, and separates partial token agreement from full plugin parity.
+The five scoped checkers and their 117 mutation tests pass. The issue remains
+open pending independent review and upstream landing.
diff --git a/.agents/specs/docs-kolibri1-current-usage.md b/.agents/specs/docs-kolibri1-current-usage.md
@@ -0,0 +1,151 @@
+# Document current Kolibri-1 usage
+
+Work ID: `DOCS-KOLIBRI1-CURRENT-USAGE`.
+Base: `c6a971e53`.
+
+## Now
+
+Documentation patch implemented for review. No model, backend, or benchmark
+lifecycle changes. README news links the CPU recipe and validation limits.
+The feature row now reflects compressed loading and the Tenstorrent forward.
+The usage section preserves incomplete checkpoint provenance explicitly.
+
+## Scope and source inventory
+
+| Surface | Source | Correction |
+|---|---|---|
+| README news | `src/vllm/model_executor/models/kolibri1_registry.cpp` | Announce the reachable model with explicit validation limits |
+| Features model row | `src/vllm/model_executor/models/kolibri1_weights.cpp`, `kolibri1_forward.cpp`, `kolibri1_registry.cpp` in the same directory | Replace historical prose with the current loader, CPU and TT availability, and refused arms |
+| Usage recipe | `examples/cli/main.cpp`, `src/vllm/entrypoints/openai/server_main.cpp`, `include/vllm.h`, Kolibri registry | Add source-checked local-checkpoint instructions, defaults, and memory limits |
+| Existing correctness and performance evidence | `.agents/specs/kolibri1*.md`, `docs/gap-kolibri1-tt-captured-decode.md` | Distinguish measured comparisons from pending primary-oracle parity |
+
+Use a focused `docs/models/kolibri1.md` guide if the recipe needs more than a
+short usage section. Link the guide from `docs/USAGE.md` and the website model
+navigation. Preserve existing checkpoint provenance without inventing revisions
+or hashes. A missing artifact pin must remain explicit.
+
+## Design and exclusions
+
+Keep one compact feature row. Put commands and necessary limits in the usage
+surface. Put detailed historical evidence in its existing specs and link to it.
+Use plain sentences and avoid internal attempt names in the public overview.
+
+Do not edit runtime code, tests, checker semantics, measurements, capability
+states, unrelated models, ABI documentation, or benchmark index dispositions.
+Do not infer a speed result or full token parity from a registration, a fixture,
+a microbenchmark, or accepted near-ties. Do not run a GPU or download models.
+
+## Upstream and dependency chain
+
+This change ports no runtime behavior. The local public entry points define the
+commands. The retained Kolibri specs identify the Aleph Alpha plugin reference
+and its gateability. No new vLLM execution is applicable to this prose correction.
+The existing source and evidence define every public claim.
+
+## Tests and gates
+
+Run `scripts/check-readme-structure.py`, `scripts/check-supported-models.py`,
+`scripts/check-surface-coverage.py`, `scripts/check-benchmark-index.py`, and
+`scripts/check-quickstart-recipes.py`. Run their existing mutation suites.
+Check new relative links and every documented flag against source.
+Run `git diff --check` and the full `scripts/agent-preflight.sh`.
+Report baseline failures and missing tools separately from scoped results.
+No new test is necessary for this reversible prose change.
+
+A fresh implementer edits from this committed spec in a separate worktree.
+A fresh reviewer checks the immutable head against source and retained evidence.
+Mutation of product behavior is not applicable because no behavior changes.
+The reviewer reruns the existing documentation mutation suites. The operator
+reruns the scoped gate before pushing the exact commit to the fork.
+
+## Work breakdown
+
+1. Commit the canonical issue and this spec before public edits.
+2. Delegate the bounded documentation patch and source audit.
+3. Review the immutable patch and repeat its CPU-only checks.
+4. Open one fork pull request against upstream `main`.
+
+## Risks and stop conditions
+
+Missing source evidence requires narrower wording. Unrelated preflight failures
+must match the base before they can be reported as pre-existing. Never weaken a
+gate. No merge, remote compute, service change, or model download is authorized.
+
+## Owed
+
+- [ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3](../issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md): correct the Kolibri-1 public overview and usage.
+
+## Source audit and verification
+
+Checked against implementation base `23a85fa47` on 2026-10-12:
+
+| Claim | Source or retained evidence |
+|---|---|
+| Compressed FP8 and BF16 loading | `src/vllm/model_executor/models/kolibri1_weights.cpp:97-119` |
+| CPU dequantization and disabled-by-default cache | `src/vllm/model_executor/models/kolibri1_forward.cpp:175-205`, `kolibri1_dequant_cache.h:320-326` in the same directory |
+| Streamed TT experts, CPU and TT dispatch, refused arms | `src/vllm/model_executor/models/kolibri1_registry.cpp:256-275`, `:289-294`, `:319-348` |
+| CLI flags and executable | `examples/cli/main.cpp:109-151`, `examples/CMakeLists.txt:55-57` |
+| Server flags and executable | `src/vllm/entrypoints/openai/server_main.cpp:498-541`, `:677-681`, `examples/CMakeLists.txt:98-115` |
+| Thinking switch | `src/vllm/entrypoints/openai/reasoning_parsers/kolibri1.cpp:17-32` |
+| Parser aliases | `src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp:69-75`, `src/vllm/entrypoints/openai/tool_parsers/abstract.cpp:99-106` |
+| Checkpoint size and mixed storage | `.agents/specs/kolibri-1-cpu.md:17-30` |
+| Partial CPU token agreement | `.agents/specs/kolibri-1-cpu.md:226-241` |
+| Serving evidence and unmeasured plugin gate | `docs/bench-evidence/kolibri1-serve-20261008.md`, `.agents/oracles/aleph-alpha-inference.md` |
+
+The short checkpoint revision is preserved from the pre-change feature row.
+No full revision or shard hashes were found in that provenance. The usage
+section marks them missing. The old feature row's dequant-at-load, owed TT,
+and Transformers-generation descriptions are falsified by the anchors above.
+
+With `PATH=/tmp/vllm-doc-tools/bin:$PATH`, all commands below exited 0:
+
+```sh
+python3 scripts/check-readme-structure.py
+python3 scripts/check-supported-models.py
+python3 scripts/check-surface-coverage.py
+python3 scripts/check-benchmark-index.py
+python3 scripts/check-quickstart-recipes.py
+python3 tests/scripts/test_check_readme_structure.py
+python3 tests/scripts/test_check_supported_models.py
+python3 tests/scripts/test_check_surface_coverage.py
+python3 tests/scripts/test_check_benchmark_index.py
+python3 tests/scripts/test_check_quickstart_recipes.py
+git diff --check
+```
+
+The existing mutation suites ran 19, 14, 46, 6, and 32 tests respectively.
+A local path-and-heading check passed for every new link. Every new command
+flag was checked against the source above. No product test or product mutation
+was added because the patch changes prose only. No model download, model
+execution, GPU use, or new performance measurement was performed.
+
+The three new shell blocks pass `bash -n`. The curl body parses as JSON with
+`enable_thinking` set to the boolean `false`.
+
+Full preflight was invoked before edits. The required staged run was invoked
+before commit and remains pending at this checkpoint. The earlier duplicate
+run was stopped during mutation tests and is incomplete. Both runs report
+missing PyYAML for `check-release-workflow` and missing CMake for
+`check-test-registration`, matching the operator's base observations.
+These are environment failures, not successful gates. The final operator
+report must give the completed staged result before remote handoff.
+Logs: `/tmp/vllm-docs-preflight-impl.log` and
+`/tmp/vllm-docs-preflight-impl-staged.log`.
+
+## Review correction: checkpoint bytes
+
+The review found that the usage paragraph described tensor bytes as total shard
+file sizes. The checkpoint record at `.agents/specs/kolibri-1-cpu.md:17-22`
+identifies the value as index `metadata.total_size`. The paragraph now names
+tensor data and preserves the recorded shard count.
+
+On 2026-10-12, `/bin/bash /tmp/vllm-docs-verify.sh` exited 0 in the repair
+worktree. All seven document and record checks passed. The five existing
+mutation suites passed 19, 14, 46, 6, and 32 tests. The compile-scope check
+found no C++ sources, headers, or build files. Source review and
+`git diff --check` passed. Product mutation is not applicable to this prose
+correction. The canonical issue remains open pending landing.
+
+Pre-edit and staged preflight checks use `--role-only` for this repair. They
+check role metadata only and are not full preflight results. The operator
+retains the full baseline and implementation results described above.
diff --git a/README.md b/README.md
@@ -37,6 +37,8 @@
 
 ## News
 
+- **2026-10** **Kolibri-1 gains CPU chat serving.** Its reasoning and tool-call parsers are available.
+  Full parity against the author's vLLM plugin remains unverified. See the [recipe and limits](docs/USAGE.md#kolibri-1).
 - **2026-09** **Qwen3.8 gains an endpoint measurement tool.** The TensorFold comparison remains
   blocked by missing artifacts, with no speed result. See the
   [measurement instructions and limits](docs/benchmarks/qwen38-tensorfold-gap.md#measurement-tools).

diff --git a/docs/FEATURES.md b/docs/FEATURES.md
@@ -224,7 +224,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks.
 | `KevModel` | `jaredpalmer/kev-0.8b` (frozen Qwen3.5-0.8B-Base + rank-16 LoRA merged at convert time + PointerHead readout) | PointerHead golden-vector tests (25 cases, 126 assertions) + LoRA-merge + ForwardHidden goldens with perturbation gates vs the kev Python reference `jaredpalmer/kev` @`19dcae9b6e3e1a48200c5825aad9fc200d31e20a` (#3295); `/v1/systemone` dispatch reuses the Laya lane. E2E choice/score/noul through LocalAI `/v1/systemone` PASS. CUDA build verification + GGUF k-quant arm OWED per [spec](../.agents/specs/kev.md) | not measured |
 | `DeepseekV41ForCausalLM` | none loadable yet | **REGISTERED AND VALIDATING; NOT LOADABLE** (W1, 2026-09-13) per [model-matrix](../.agents/model-matrix.md) and [spec](../.agents/specs/deepseek-v4-1-flash.md): the architecture resolves through the registry while the load plan is validated; no token claim | not measured |
 | `MiMoV2ForCausalLM` | `vcruz305/MiMoV2-Flash-RL-EXL3` (EXL3-quantized, not yet loadable end-to-end) | **W1+W2 scaffold + W3 forward pass; NOT GATED.** Registry + config + hybrid KV-cache spec (full-attn 4 KV heads / SWA 8 KV heads, `v_head_dim=128 != head_dim=192`) + bf16/EXL3 dual-path weight loader gated; the device forward computes through all 48 layers (hybrid attention with sink bias, partial RoPE, attention_value_scale; layer 0 dense MLP, layers 1-47 MoE with sigmoid + noaux_tc routing) but no token has been emitted yet — the end-to-end gate (W5) is OWED | not measured |
-| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1` @`e52eb462` (78.8 GB: FP8 block-quantized linears with F32 `weight_scale_inv` grids, BF16 experts/shared/head/norms/router; 32 shards, manifest-verified 116,303 tensors) | **W1+W2+W3 landed; token-gated, and the real tokenizer loads (R7).** Registry + config + hybrid KV two-group spec (10 full-attention layers RNoPE — no positional encoding — / 40 sliding layers window 513 with RoPE, per-head qk-norm, head 128, GQA 48/4) + FP8 block-quant loader (dequant-at-load) gated; the CPU forward computes all 50 layers (sandwich norms, sigmoid-logit router top-6 with `expert_bias`, no renorm, ungated shared expert) and is gated against a scalar double-precision transcription of the author plugin's math (1,757 assertions). Token gate LANDED 2026-10-05 (transformers golden run, W3 900/900; HF ids fed directly then). The tokenizer engine refused the checkpoint's `\p{N}{1}` split-regex spelling until R7 (2026-10-08, row/kolibri-r7): it is now recognized as the classic Qwen2 pattern, and our Encode reproduces the HF reference ids on all 8 golden prompts (no-BOS contract gated on a real load). GGUF, CUDA, TT arms OWED; the author plugin ([aleph-alpha-inference](../../.agents/oracles/aleph-alpha-inference.md)) is the primary oracle, gateable=no until served | not measured |
+| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) |
 <!-- supported-arch-table:end -->
 
 ### Standalone and non-registered lanes

diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -844,6 +844,68 @@ directory. The [CLM page](models/clm.md) has the commands.
 GGUF k-quant arms are not implemented; `ClmModel` refuses a GGUF source by
 name.
 
+## Kolibri-1
+
+`Kolibri1ForCausalLM` loads the mixed FP8/BF16 safetensors checkpoint from
+`Aleph-Alpha/Kolibri-1`. The recorded checkpoint contains 32 shards.
+Its `model.safetensors.index.json` records 78,827,029,120 bytes of tensor data.
+The existing record identifies revision `e52eb462` but lacks the full revision
+and shard SHA-256 hashes. This is incomplete artifact provenance, not a
+reproducible download pin. See the [checkpoint record](../.agents/specs/kolibri-1-cpu.md#the-checkpoint).
+
+Use an existing local checkpoint directory containing the index, its shards,
+`config.json`, `tokenizer.json`, and `tokenizer_config.json`. Allow host memory
+for the weights, temporary dequantized projections, activations, and the KV cache.
+The checkpoint size is not a peak-memory estimate.
+
+After [building the CLI and server](BUILD.md), run a raw completion on CPU:
+
+```sh
+build/examples/vllm-cli \
+  --model /path/to/Kolibri-1 --device cpu \
+  --prompt "The capital of Germany is" --max-tokens 16 --temperature 0
+```
+
+For chat, select both Kolibri parsers explicitly:
+
+```sh
+build/examples/vllm-server \
+  --model /path/to/Kolibri-1 --device cpu --served-model-name kolibri1 \
+  --host 127.0.0.1 --port 8000 \
+  --max-model-len 2048 --kv-cache-dtype bfloat16 \
+  --reasoning-parser kolibri1 --tool-call-parser kolibri1
+```
+
+The server uses the checkpoint's chat template. In another terminal, send a
+chat request with thinking disabled:
+
+```sh
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"kolibri1","messages":[{"role":"user","content":"Name the capital of Germany."}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}'
+```
+
+This request omits `reasoning_effort`. When that template argument is non-null,
+it takes precedence over `enable_thinking`. Only the value `"none"` disables thinking.
+The explicit context limit keeps this example separate from the author's
+unverified million-token serving recipe. It does not limit weight memory.
+
+The loader retains FP8 blocks and BF16 tensors. The CPU forward dequantizes FP8
+projections when needed. Its optional dequantization cache defaults to disabled
+(`VT_KOLIBRI1_DEQUANT_CACHE_MB=0`), so enabling it adds a memory budget.
+The registry also dispatches to a Tenstorrent forward with streamed experts.
+See the [Tenstorrent measurements and remaining gaps](gap-kolibri1-tt-captured-decode.md).
+GGUF loading and other device forwards, including CUDA, are refused.
+
+The recorded CPU gate matched 141 of 145 argmax positions against a
+PyTorch transcription of the author's plugin. Four differences were accepted
+near-ties. Four of eight prompts matched token-for-token through the full
+continuation. This was not a native Transformers generation run.
+See the [CPU comparison](../.agents/specs/kolibri-1-cpu.md#now) and
+[serving validation](bench-evidence/kolibri1-serve-20261008.md).
+Full correctness and speed parity against the
+[author's vLLM plugin](../.agents/oracles/aleph-alpha-inference.md) remain unverified.
+
 ## Muse Glimmer 30B from a GGUF k-quant
 
 The text tower loads from a `muse-glimmer`-architecture GGUF, so the 30B model