diff --git a/.agents/issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md b/.agents/issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md new file mode 100644 index 0000000000..008697d67c --- /dev/null +++ b/.agents/issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md @@ -0,0 +1,22 @@ +ID: ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3 +Title: Align Kolibri-1 public documentation with current code +Row: - +State: OPEN +Kind: docs +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-12 +Updated: 2026-10-12 +Closed: - + +## Problem + +The feature row still describes dequant-at-load and lists the TT arm as owed. README news and USAGE omit the reachable Kolibri-1 path and its validation limits. + +## Resolution + +2026-10-12: The documentation branch corrects the loader and device claims, +adds CPU usage, and separates partial token agreement from full plugin parity. +The five scoped checkers and their 117 mutation tests pass. The issue remains +open pending independent review and upstream landing. diff --git a/.agents/specs/docs-kolibri1-current-usage.md b/.agents/specs/docs-kolibri1-current-usage.md new file mode 100644 index 0000000000..9f71f63b16 --- /dev/null +++ b/.agents/specs/docs-kolibri1-current-usage.md @@ -0,0 +1,151 @@ +# Document current Kolibri-1 usage + +Work ID: `DOCS-KOLIBRI1-CURRENT-USAGE`. +Base: `c6a971e53`. + +## Now + +Documentation patch implemented for review. No model, backend, or benchmark +lifecycle changes. README news links the CPU recipe and validation limits. +The feature row now reflects compressed loading and the Tenstorrent forward. +The usage section preserves incomplete checkpoint provenance explicitly. + +## Scope and source inventory + +| Surface | Source | Correction | +|---|---|---| +| README news | `src/vllm/model_executor/models/kolibri1_registry.cpp` | Announce the reachable model with explicit validation limits | +| Features model row | `src/vllm/model_executor/models/kolibri1_weights.cpp`, `kolibri1_forward.cpp`, `kolibri1_registry.cpp` in the same directory | Replace historical prose with the current loader, CPU and TT availability, and refused arms | +| Usage recipe | `examples/cli/main.cpp`, `src/vllm/entrypoints/openai/server_main.cpp`, `include/vllm.h`, Kolibri registry | Add source-checked local-checkpoint instructions, defaults, and memory limits | +| Existing correctness and performance evidence | `.agents/specs/kolibri1*.md`, `docs/gap-kolibri1-tt-captured-decode.md` | Distinguish measured comparisons from pending primary-oracle parity | + +Use a focused `docs/models/kolibri1.md` guide if the recipe needs more than a +short usage section. Link the guide from `docs/USAGE.md` and the website model +navigation. Preserve existing checkpoint provenance without inventing revisions +or hashes. A missing artifact pin must remain explicit. + +## Design and exclusions + +Keep one compact feature row. Put commands and necessary limits in the usage +surface. Put detailed historical evidence in its existing specs and link to it. +Use plain sentences and avoid internal attempt names in the public overview. + +Do not edit runtime code, tests, checker semantics, measurements, capability +states, unrelated models, ABI documentation, or benchmark index dispositions. +Do not infer a speed result or full token parity from a registration, a fixture, +a microbenchmark, or accepted near-ties. Do not run a GPU or download models. + +## Upstream and dependency chain + +This change ports no runtime behavior. The local public entry points define the +commands. The retained Kolibri specs identify the Aleph Alpha plugin reference +and its gateability. No new vLLM execution is applicable to this prose correction. +The existing source and evidence define every public claim. + +## Tests and gates + +Run `scripts/check-readme-structure.py`, `scripts/check-supported-models.py`, +`scripts/check-surface-coverage.py`, `scripts/check-benchmark-index.py`, and +`scripts/check-quickstart-recipes.py`. Run their existing mutation suites. +Check new relative links and every documented flag against source. +Run `git diff --check` and the full `scripts/agent-preflight.sh`. +Report baseline failures and missing tools separately from scoped results. +No new test is necessary for this reversible prose change. + +A fresh implementer edits from this committed spec in a separate worktree. +A fresh reviewer checks the immutable head against source and retained evidence. +Mutation of product behavior is not applicable because no behavior changes. +The reviewer reruns the existing documentation mutation suites. The operator +reruns the scoped gate before pushing the exact commit to the fork. + +## Work breakdown + +1. Commit the canonical issue and this spec before public edits. +2. Delegate the bounded documentation patch and source audit. +3. Review the immutable patch and repeat its CPU-only checks. +4. Open one fork pull request against upstream `main`. + +## Risks and stop conditions + +Missing source evidence requires narrower wording. Unrelated preflight failures +must match the base before they can be reported as pre-existing. Never weaken a +gate. No merge, remote compute, service change, or model download is authorized. + +## Owed + +- [ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3](../issues/_owed/ISSUE-LOCAL-01M4Q12F0EMYDYQV1P1PNJKEB3.md): correct the Kolibri-1 public overview and usage. + +## Source audit and verification + +Checked against implementation base `23a85fa47` on 2026-10-12: + +| Claim | Source or retained evidence | +|---|---| +| Compressed FP8 and BF16 loading | `src/vllm/model_executor/models/kolibri1_weights.cpp:97-119` | +| CPU dequantization and disabled-by-default cache | `src/vllm/model_executor/models/kolibri1_forward.cpp:175-205`, `kolibri1_dequant_cache.h:320-326` in the same directory | +| Streamed TT experts, CPU and TT dispatch, refused arms | `src/vllm/model_executor/models/kolibri1_registry.cpp:256-275`, `:289-294`, `:319-348` | +| CLI flags and executable | `examples/cli/main.cpp:109-151`, `examples/CMakeLists.txt:55-57` | +| Server flags and executable | `src/vllm/entrypoints/openai/server_main.cpp:498-541`, `:677-681`, `examples/CMakeLists.txt:98-115` | +| Thinking switch | `src/vllm/entrypoints/openai/reasoning_parsers/kolibri1.cpp:17-32` | +| Parser aliases | `src/vllm/entrypoints/openai/reasoning_parsers/abstract.cpp:69-75`, `src/vllm/entrypoints/openai/tool_parsers/abstract.cpp:99-106` | +| Checkpoint size and mixed storage | `.agents/specs/kolibri-1-cpu.md:17-30` | +| Partial CPU token agreement | `.agents/specs/kolibri-1-cpu.md:226-241` | +| Serving evidence and unmeasured plugin gate | `docs/bench-evidence/kolibri1-serve-20261008.md`, `.agents/oracles/aleph-alpha-inference.md` | + +The short checkpoint revision is preserved from the pre-change feature row. +No full revision or shard hashes were found in that provenance. The usage +section marks them missing. The old feature row's dequant-at-load, owed TT, +and Transformers-generation descriptions are falsified by the anchors above. + +With `PATH=/tmp/vllm-doc-tools/bin:$PATH`, all commands below exited 0: + +```sh +python3 scripts/check-readme-structure.py +python3 scripts/check-supported-models.py +python3 scripts/check-surface-coverage.py +python3 scripts/check-benchmark-index.py +python3 scripts/check-quickstart-recipes.py +python3 tests/scripts/test_check_readme_structure.py +python3 tests/scripts/test_check_supported_models.py +python3 tests/scripts/test_check_surface_coverage.py +python3 tests/scripts/test_check_benchmark_index.py +python3 tests/scripts/test_check_quickstart_recipes.py +git diff --check +``` + +The existing mutation suites ran 19, 14, 46, 6, and 32 tests respectively. +A local path-and-heading check passed for every new link. Every new command +flag was checked against the source above. No product test or product mutation +was added because the patch changes prose only. No model download, model +execution, GPU use, or new performance measurement was performed. + +The three new shell blocks pass `bash -n`. The curl body parses as JSON with +`enable_thinking` set to the boolean `false`. + +Full preflight was invoked before edits. The required staged run was invoked +before commit and remains pending at this checkpoint. The earlier duplicate +run was stopped during mutation tests and is incomplete. Both runs report +missing PyYAML for `check-release-workflow` and missing CMake for +`check-test-registration`, matching the operator's base observations. +These are environment failures, not successful gates. The final operator +report must give the completed staged result before remote handoff. +Logs: `/tmp/vllm-docs-preflight-impl.log` and +`/tmp/vllm-docs-preflight-impl-staged.log`. + +## Review correction: checkpoint bytes + +The review found that the usage paragraph described tensor bytes as total shard +file sizes. The checkpoint record at `.agents/specs/kolibri-1-cpu.md:17-22` +identifies the value as index `metadata.total_size`. The paragraph now names +tensor data and preserves the recorded shard count. + +On 2026-10-12, `/bin/bash /tmp/vllm-docs-verify.sh` exited 0 in the repair +worktree. All seven document and record checks passed. The five existing +mutation suites passed 19, 14, 46, 6, and 32 tests. The compile-scope check +found no C++ sources, headers, or build files. Source review and +`git diff --check` passed. Product mutation is not applicable to this prose +correction. The canonical issue remains open pending landing. + +Pre-edit and staged preflight checks use `--role-only` for this repair. They +check role metadata only and are not full preflight results. The operator +retains the full baseline and implementation results described above. diff --git a/README.md b/README.md index 7866986def..d893b05548 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,8 @@ ## News +- **2026-10** **Kolibri-1 gains CPU chat serving.** Its reasoning and tool-call parsers are available. + Full parity against the author's vLLM plugin remains unverified. See the [recipe and limits](docs/USAGE.md#kolibri-1). - **2026-09** **Qwen3.8 gains an endpoint measurement tool.** The TensorFold comparison remains blocked by missing artifacts, with no speed result. See the [measurement instructions and limits](docs/benchmarks/qwen38-tensorfold-gap.md#measurement-tools). diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5706cbda9d..38d75c76cf 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -224,7 +224,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `KevModel` | `jaredpalmer/kev-0.8b` (frozen Qwen3.5-0.8B-Base + rank-16 LoRA merged at convert time + PointerHead readout) | PointerHead golden-vector tests (25 cases, 126 assertions) + LoRA-merge + ForwardHidden goldens with perturbation gates vs the kev Python reference `jaredpalmer/kev` @`19dcae9b6e3e1a48200c5825aad9fc200d31e20a` (#3295); `/v1/systemone` dispatch reuses the Laya lane. E2E choice/score/noul through LocalAI `/v1/systemone` PASS. CUDA build verification + GGUF k-quant arm OWED per [spec](../.agents/specs/kev.md) | not measured | | `DeepseekV41ForCausalLM` | none loadable yet | **REGISTERED AND VALIDATING; NOT LOADABLE** (W1, 2026-09-13) per [model-matrix](../.agents/model-matrix.md) and [spec](../.agents/specs/deepseek-v4-1-flash.md): the architecture resolves through the registry while the load plan is validated; no token claim | not measured | | `MiMoV2ForCausalLM` | `vcruz305/MiMoV2-Flash-RL-EXL3` (EXL3-quantized, not yet loadable end-to-end) | **W1+W2 scaffold + W3 forward pass; NOT GATED.** Registry + config + hybrid KV-cache spec (full-attn 4 KV heads / SWA 8 KV heads, `v_head_dim=128 != head_dim=192`) + bf16/EXL3 dual-path weight loader gated; the device forward computes through all 48 layers (hybrid attention with sink bias, partial RoPE, attention_value_scale; layer 0 dense MLP, layers 1-47 MoE with sigmoid + noaux_tc routing) but no token has been emitted yet — the end-to-end gate (W5) is OWED | not measured | -| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1` @`e52eb462` (78.8 GB: FP8 block-quantized linears with F32 `weight_scale_inv` grids, BF16 experts/shared/head/norms/router; 32 shards, manifest-verified 116,303 tensors) | **W1+W2+W3 landed; token-gated, and the real tokenizer loads (R7).** Registry + config + hybrid KV two-group spec (10 full-attention layers RNoPE — no positional encoding — / 40 sliding layers window 513 with RoPE, per-head qk-norm, head 128, GQA 48/4) + FP8 block-quant loader (dequant-at-load) gated; the CPU forward computes all 50 layers (sandwich norms, sigmoid-logit router top-6 with `expert_bias`, no renorm, ungated shared expert) and is gated against a scalar double-precision transcription of the author plugin's math (1,757 assertions). Token gate LANDED 2026-10-05 (transformers golden run, W3 900/900; HF ids fed directly then). The tokenizer engine refused the checkpoint's `\p{N}{1}` split-regex spelling until R7 (2026-10-08, row/kolibri-r7): it is now recognized as the classic Qwen2 pattern, and our Encode reproduces the HF reference ids on all 8 golden prompts (no-BOS contract gated on a real load). GGUF, CUDA, TT arms OWED; the author plugin ([aleph-alpha-inference](../../.agents/oracles/aleph-alpha-inference.md)) is the primary oracle, gateable=no until served | not measured | +| `Kolibri1ForCausalLM` | `Aleph-Alpha/Kolibri-1`, mixed FP8/BF16 safetensors (78.8 GB) | CPU and Tenstorrent forwards. FP8 weights stay compressed at load. Kolibri chat template, reasoning, and tool-call parsers. CPU comparisons include accepted near-ties, not full token parity. GGUF and other device forwards refused. [Recipe and validation limits](USAGE.md#kolibri-1) | Full author-plugin comparison pending. [TT measurements and gaps](gap-kolibri1-tt-captured-decode.md) | ### Standalone and non-registered lanes diff --git a/docs/USAGE.md b/docs/USAGE.md index c11e9f1fcd..195a18df64 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -844,6 +844,68 @@ directory. The [CLM page](models/clm.md) has the commands. GGUF k-quant arms are not implemented; `ClmModel` refuses a GGUF source by name. +## Kolibri-1 + +`Kolibri1ForCausalLM` loads the mixed FP8/BF16 safetensors checkpoint from +`Aleph-Alpha/Kolibri-1`. The recorded checkpoint contains 32 shards. +Its `model.safetensors.index.json` records 78,827,029,120 bytes of tensor data. +The existing record identifies revision `e52eb462` but lacks the full revision +and shard SHA-256 hashes. This is incomplete artifact provenance, not a +reproducible download pin. See the [checkpoint record](../.agents/specs/kolibri-1-cpu.md#the-checkpoint). + +Use an existing local checkpoint directory containing the index, its shards, +`config.json`, `tokenizer.json`, and `tokenizer_config.json`. Allow host memory +for the weights, temporary dequantized projections, activations, and the KV cache. +The checkpoint size is not a peak-memory estimate. + +After [building the CLI and server](BUILD.md), run a raw completion on CPU: + +```sh +build/examples/vllm-cli \ + --model /path/to/Kolibri-1 --device cpu \ + --prompt "The capital of Germany is" --max-tokens 16 --temperature 0 +``` + +For chat, select both Kolibri parsers explicitly: + +```sh +build/examples/vllm-server \ + --model /path/to/Kolibri-1 --device cpu --served-model-name kolibri1 \ + --host 127.0.0.1 --port 8000 \ + --max-model-len 2048 --kv-cache-dtype bfloat16 \ + --reasoning-parser kolibri1 --tool-call-parser kolibri1 +``` + +The server uses the checkpoint's chat template. In another terminal, send a +chat request with thinking disabled: + +```sh +curl http://127.0.0.1:8000/v1/chat/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"kolibri1","messages":[{"role":"user","content":"Name the capital of Germany."}],"max_tokens":16,"temperature":0,"chat_template_kwargs":{"enable_thinking":false}}' +``` + +This request omits `reasoning_effort`. When that template argument is non-null, +it takes precedence over `enable_thinking`. Only the value `"none"` disables thinking. +The explicit context limit keeps this example separate from the author's +unverified million-token serving recipe. It does not limit weight memory. + +The loader retains FP8 blocks and BF16 tensors. The CPU forward dequantizes FP8 +projections when needed. Its optional dequantization cache defaults to disabled +(`VT_KOLIBRI1_DEQUANT_CACHE_MB=0`), so enabling it adds a memory budget. +The registry also dispatches to a Tenstorrent forward with streamed experts. +See the [Tenstorrent measurements and remaining gaps](gap-kolibri1-tt-captured-decode.md). +GGUF loading and other device forwards, including CUDA, are refused. + +The recorded CPU gate matched 141 of 145 argmax positions against a +PyTorch transcription of the author's plugin. Four differences were accepted +near-ties. Four of eight prompts matched token-for-token through the full +continuation. This was not a native Transformers generation run. +See the [CPU comparison](../.agents/specs/kolibri-1-cpu.md#now) and +[serving validation](bench-evidence/kolibri1-serve-20261008.md). +Full correctness and speed parity against the +[author's vLLM plugin](../.agents/oracles/aleph-alpha-inference.md) remain unverified. + ## Muse Glimmer 30B from a GGUF k-quant The text tower loads from a `muse-glimmer`-architecture GGUF, so the 30B model