Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/hw-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ excerpts, since the log is public and the add-on's sources are not. It then runs
| check | what | catches |
|---|---|---|
| ops | one `test-backend-ops -b HRX0 -j 4` run over MUL_MAT, MUL_MAT_ID, FLASH_ATTN_EXT, GATED_DELTA_NET, SSM_CONV, GET_ROWS, ROPE, RMS_NORM, SOFT_MAX | a kernel that disagrees with the CPU (#310, #313, #370, the IQ ports) |
| fa-decode-q8 | the add-on's `ggml-hrx-fa-decode-bench` on its q8_0 KV decode kernel against a double-precision reference: K q8_0 with V f16 (`-ctk q8_0` alone) and with V q8_0, Hq/Hkv 16/8 and 32/8, n_kv 1031, 2048 and 4103, six runs in one hold | a mixed K/V type bug test-backend-ops cannot see: it has no FLASH_ATTN_EXT case with K and V of different types (gpu-kernels #11 read f16 V dims 0..31 only) |
| fa-decode-q8 | the add-on's own long-context decode-attention check against a double-precision reference, quantised K with both cache value types and two head groupings, six runs in one hold | a mixed cache-type bug `test-backend-ops` cannot see: it has no `FLASH_ATTN_EXT` case with K and V of different types |
| kat | `tools/kat.py check` on five models (below), against `tests/golden/hw-gate/kat-<model>.json` | wrong answers, empty replies, NaN logprobs, nondeterminism, cached-prefix and multi-ubatch bugs, drift from the release pin (#308, #309, #315, #318, #319, #369) |
| kld | `llama-perplexity --kl-divergence` on wiki.test.raw, 512 tokens x 8 chunks, against base logits stored at the release pin | small numeric drift that greedy answers do not show |
| needle | `tests/hrx_long_prompt.py`: the 320-record vault-code needle (7.1K tokens on Qwen3, 7.4K on ZAYA1) on Qwen3-0.6B and ZAYA1-8B | #319, #369 |
Expand Down
10 changes: 5 additions & 5 deletions docs/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,14 +72,14 @@ before every run); **ours** = our sampler/batched standard (see the table legend
the auto-opt database).

**HRX speed (ours).** Qwen3.8-27B UD-Q4_K_XL with MTP, greedy 256-token chats: **36.0 / 26.8 / 29.9
tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the verify kernels — 2–8-token
K-quant verify took the step from 121 ms to 88 ms (`#487`). Decode without MTP stays 12.2 tok/s.
tok/s** on code / prose / short, from 25.9 / 20.5 / 21.9 before the speculative verify pass was
reworked — the step it took is 121 ms to 88 ms per step (`#487`). Decode without MTP stays 12.2 tok/s.

| change | PR | measured |
|---|---|---|
| 2..8-token K-quant verify kernels | `#487` | 27B `--mtp` **+31–39 %** |
| batched MoE decode on our kernels | `#521`, `#526` | pin `744bd713d968`: the top-1 batched window from 5 tokens; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) |
| split-K decode attention for gated head-256 | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** |
| speculative verify pass rework | `#487` | 27B `--mtp` **+31–39 %** |
| batched MoE decode | `#521`, `#526` | pin `744bd713d968`: batching pays from five tokens per window; the batched cells below were measured one pin earlier (90 W cap, PCM pad (23 Oct)) |
| decode attention for gated attention at long context | `#500` | Qwen3.5-4B 32K decode **23.7 → 47.1 tok/s**; 27B **7.5 → 11.0** |
| MoE formats for Qwen3.8-Flash-Next | `#494` | **5.1 → 26 tok/s**, 50 with MTP |
| HIP prompt flash attention, causal block skipping | `#491` | Qwen3-4B pp4096 **+45 %** |
| m128 prefill tiles + raster groups | `#490` | 27B prefill **+1.6…2.5 %** |
Expand Down
5 changes: 2 additions & 3 deletions docs/serve.md
Original file line number Diff line number Diff line change
Expand Up @@ -801,9 +801,8 @@ SIGKILLed `serve`, and that a backend that exits on its own ends `serve` at once

A follow-up turn in the same conversation is not bit-identical to a cold run of that conversation. The
follow-up continues from the KV cache and prefills only its new user message (28 tokens in the test below),
where a cold run prefills the conversation in 512-token ubatches - and HRX picks a different kernel family for a
different token count (generic post-ops for 16 < T < 128, the WMMA prefill tiles for T >= 128), with a different
activation quantisation, split-K and combine tree, so the last bits differ and a near-tie flips. Measured on
where a cold run prefills the conversation in 512-token ubatches - and HRX serves a different token count with a
different plan, so the last bits of the logits differ and a near-tie can flip. Measured on
2026-10-22, 10 greedy two-turn chats per model, follow-up against a cold run of the same conversation.

`--precise` (or `ONEBIT_PRECISE=1`) makes every request re-prefill its whole conversation, which is exactly the
Expand Down
Loading