diff --git a/docs/hw-gate.md b/docs/hw-gate.md index 023ab353..318edae0 100644 --- a/docs/hw-gate.md +++ b/docs/hw-gate.md @@ -49,7 +49,7 @@ excerpts, since the log is public and the add-on's sources are not. It then runs | check | what | catches | |---|---|---| | ops | one `test-backend-ops -b HRX0 -j 4` run over MUL_MAT, MUL_MAT_ID, FLASH_ATTN_EXT, GATED_DELTA_NET, SSM_CONV, GET_ROWS, ROPE, RMS_NORM, SOFT_MAX | a kernel that disagrees with the CPU (#310, #313, #370, the IQ ports) | -| fa-decode-q8 | the add-on's `ggml-hrx-fa-decode-bench` on its q8_0 KV decode kernel against a double-precision reference: K q8_0 with V f16 (`-ctk q8_0` alone) and with V q8_0, Hq/Hkv 16/8 and 32/8, n_kv 1031, 2048 and 4103, six runs in one hold | a mixed K/V type bug test-backend-ops cannot see: it has no FLASH_ATTN_EXT case with K and V of different types (gpu-kernels #11 read f16 V dims 0..31 only) | +| fa-decode-q8 | the add-on's own long-context decode-attention check against a double-precision reference, quantised K with both cache value types and two head groupings, six runs in one hold | a mixed cache-type bug `test-backend-ops` cannot see: it has no `FLASH_ATTN_EXT` case with K and V of different types | | kat | `tools/kat.py check` on five models (below), against `tests/golden/hw-gate/kat-.json` | wrong answers, empty replies, NaN logprobs, nondeterminism, cached-prefix and multi-ubatch bugs, drift from the release pin (#308, #309, #315, #318, #319, #369) | | kld | `llama-perplexity --kl-divergence` on wiki.test.raw, 512 tokens x 8 chunks, against base logits stored at the release pin | small numeric drift that greedy answers do not show | | needle | `tests/hrx_long_prompt.py`: the 320-record vault-code needle (7.1K tokens on Qwen3, 7.4K on ZAYA1) on Qwen3-0.6B and ZAYA1-8B | #319, #369 | diff --git a/docs/releases.md b/docs/releases.md index 51789e8d..b9a6cbf3 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -72,14 +72,14 @@ before every run); **ours** = our sampler/batched standard (see the table legend the auto-opt database). **HRX speed (ours).** Qwen3.8-27B UD-Q4_K_XL with MTP, greedy 256-token chats: **36.0 / 26.8 / 29.9 -tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the verify kernels — 2–8-token -K-quant verify took the step from 121 ms to 88 ms (`#487`). Decode without MTP stays 12.2 tok/s. +tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the speculative verify pass was +reworked — the step it took is 121 ms to 88 ms per step (`#487`). Decode without MTP stays 12.2 tok/s. | change | PR | measured | |---|---|---| -| 2..8-token K-quant verify kernels | `#487` | 27B `--mtp` **+31–39 %** | -| batched MoE decode on our kernels | `#521`, `#526` | pin `744bd713d968`: the top-1 batched window from 5 tokens; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) | -| split-K decode attention for gated head-256 | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** | +| speculative verify pass rework | `#487` | 27B `--mtp` **+31–39 %** | +| batched MoE decode | `#521`, `#526` | pin `744bd713d968`: batching pays from five tokens per window; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) | +| decode attention for gated attention at long context | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** | | MoE formats for Qwen3.8-Flash-Next | `#494` | **5.1 → 26 tok/s**, 50 with MTP | | HIP prompt flash attention, causal block skipping | `#491` | Qwen3-4B pp4096 **+45 %** | | m128 prefill tiles + raster groups | `#490` | 27B prefill **+1.6…2.5 %** | diff --git a/docs/serve.md b/docs/serve.md index 4a03e204..bd16579c 100644 --- a/docs/serve.md +++ b/docs/serve.md @@ -801,9 +801,8 @@ SIGKILLed `serve`, and that a backend that exits on its own ends `serve` at once A follow-up turn in the same conversation is not bit-identical to a cold run of that conversation. The follow-up continues from the KV cache and prefills only its new user message (28 tokens in the test below), -where a cold run prefills the conversation in 512-token ubatches - and HRX picks a different kernel family for a -different token count (generic post-ops for 16 < T < 128, the WMMA prefill tiles for T >= 128), with a different -activation quantisation, split-K and combine tree, so the last bits differ and a near-tie flips. Measured on +where a cold run prefills the conversation in 512-token ubatches - and HRX serves a different token count with a +different plan, so the last bits of the logits differ and a near-tie can flip. Measured on 2026-10-22, 10 greedy two-turn chats per model, follow-up against a cold run of the same conversation. `--precise` (or `ONEBIT_PRECISE=1`) makes every request re-prefill its whole conversation, which is exactly the