From 377ff8b0e54f75c201a5393cfa142e865d7cec0e Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sat, 24 Oct 2026 08:39:49 -0300 Subject: [PATCH] docs: describe what the release does, not how the kernels are written MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The performance table and the two method paragraphs around it described our own kernel work in the vocabulary it is written in — cache types and head groupings per case, the plan shape picked per token count, the attention split and combine tree, tiled window sizes. Those are results pages and user documentation, so they now say what changed and what it bought, in the terms a reader needs to decide whether to use the release: - `docs/releases.md`: "speculative verify pass rework", "batched MoE decode", "decode attention for gated attention at long context" — every number kept as measured, with the thermal label untouched. - `docs/serve.md` (`--precise`): the reason a follow-up turn is not bit-identical stays (a different token count gets a different plan, so the last bits differ and a near-tie can flip); the plan's internals are gone. - `docs/hw-gate.md`: the long-context decode-attention check keeps what it covers and what it caught, without the per-case parameter list or the private reference. No reason given for the change, on purpose: it is housekeeping, and the numbers that sell the release are untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- docs/hw-gate.md | 2 +- docs/releases.md | 10 +++++----- docs/serve.md | 5 ++--- 3 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/hw-gate.md b/docs/hw-gate.md index 023ab353..318edae0 100644 --- a/docs/hw-gate.md +++ b/docs/hw-gate.md @@ -49,7 +49,7 @@ excerpts, since the log is public and the add-on's sources are not. It then runs | check | what | catches | |---|---|---| | ops | one `test-backend-ops -b HRX0 -j 4` run over MUL_MAT, MUL_MAT_ID, FLASH_ATTN_EXT, GATED_DELTA_NET, SSM_CONV, GET_ROWS, ROPE, RMS_NORM, SOFT_MAX | a kernel that disagrees with the CPU (#310, #313, #370, the IQ ports) | -| fa-decode-q8 | the add-on's `ggml-hrx-fa-decode-bench` on its q8_0 KV decode kernel against a double-precision reference: K q8_0 with V f16 (`-ctk q8_0` alone) and with V q8_0, Hq/Hkv 16/8 and 32/8, n_kv 1031, 2048 and 4103, six runs in one hold | a mixed K/V type bug test-backend-ops cannot see: it has no FLASH_ATTN_EXT case with K and V of different types (gpu-kernels #11 read f16 V dims 0..31 only) | +| fa-decode-q8 | the add-on's own long-context decode-attention check against a double-precision reference, quantised K with both cache value types and two head groupings, six runs in one hold | a mixed cache-type bug `test-backend-ops` cannot see: it has no `FLASH_ATTN_EXT` case with K and V of different types | | kat | `tools/kat.py check` on five models (below), against `tests/golden/hw-gate/kat-.json` | wrong answers, empty replies, NaN logprobs, nondeterminism, cached-prefix and multi-ubatch bugs, drift from the release pin (#308, #309, #315, #318, #319, #369) | | kld | `llama-perplexity --kl-divergence` on wiki.test.raw, 512 tokens x 8 chunks, against base logits stored at the release pin | small numeric drift that greedy answers do not show | | needle | `tests/hrx_long_prompt.py`: the 320-record vault-code needle (7.1K tokens on Qwen3, 7.4K on ZAYA1) on Qwen3-0.6B and ZAYA1-8B | #319, #369 | diff --git a/docs/releases.md b/docs/releases.md index 51789e8d..b9a6cbf3 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -72,14 +72,14 @@ before every run); **ours** = our sampler/batched standard (see the table legend the auto-opt database). **HRX speed (ours).** Qwen3.8-27B UD-Q4_K_XL with MTP, greedy 256-token chats: **36.0 / 26.8 / 29.9 -tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the verify kernels — 2–8-token -K-quant verify took the step from 121 ms to 88 ms (`#487`). Decode without MTP stays 12.2 tok/s. +tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the speculative verify pass was +reworked — the step it took is 121 ms to 88 ms per step (`#487`). Decode without MTP stays 12.2 tok/s. | change | PR | measured | |---|---|---| -| 2..8-token K-quant verify kernels | `#487` | 27B `--mtp` **+31–39 %** | -| batched MoE decode on our kernels | `#521`, `#526` | pin `744bd713d968`: the top-1 batched window from 5 tokens; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) | -| split-K decode attention for gated head-256 | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** | +| speculative verify pass rework | `#487` | 27B `--mtp` **+31–39 %** | +| batched MoE decode | `#521`, `#526` | pin `744bd713d968`: batching pays from five tokens per window; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) | +| decode attention for gated attention at long context | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** | | MoE formats for Qwen3.8-Flash-Next | `#494` | **5.1 → 26 tok/s**, 50 with MTP | | HIP prompt flash attention, causal block skipping | `#491` | Qwen3-4B pp4096 **+45 %** | | m128 prefill tiles + raster groups | `#490` | 27B prefill **+1.6…2.5 %** | diff --git a/docs/serve.md b/docs/serve.md index 4a03e204..bd16579c 100644 --- a/docs/serve.md +++ b/docs/serve.md @@ -801,9 +801,8 @@ SIGKILLed `serve`, and that a backend that exits on its own ends `serve` at once A follow-up turn in the same conversation is not bit-identical to a cold run of that conversation. The follow-up continues from the KV cache and prefills only its new user message (28 tokens in the test below), -where a cold run prefills the conversation in 512-token ubatches - and HRX picks a different kernel family for a -different token count (generic post-ops for 16 < T < 128, the WMMA prefill tiles for T >= 128), with a different -activation quantisation, split-K and combine tree, so the last bits differ and a near-tie flips. Measured on +where a cold run prefills the conversation in 512-token ubatches - and HRX serves a different token count with a +different plan, so the last bits of the logits differ and a near-tie can flip. Measured on 2026-10-22, 10 greedy two-turn chats per model, follow-up against a cold run of the same conversation. `--precise` (or `ONEBIT_PRECISE=1`) makes every request re-prefill its whole conversation, which is exactly the