From e5e5468c8e8a44128693175d182672c0f47a8370 Mon Sep 17 00:00:00 2001 From: Giveen <1180939+giveen@users.noreply.github.com> Date: Thu, 19 Nov 2026 15:20:16 -0700 Subject: [PATCH] docs: what --kv fp8 costs Hy4-flash, and how to compare the formats The precision page said the fp8 rows had no score of their own against BF16. They do now: with the tier pinned at 4,800 slots, 100 chunks of the hy4flash BF16 base, T 1, 25,500 rows, mean KLD 0.2046 (default fp16: 0.2003), top-1 86.04% against 85.87%, PPL 17.4371 against 17.4934, and a heavier tail (p99 3.21 against 3.01). The default stays fp16: the difference is real and the memory win is a flag, not a default. The comparison needs the tier pinned, and this model declares no quality_flags for the gate, so the page now says to pass the same --expert-cache to both runs; the unpinned pair reads the opposite sign (0.2085 against 0.2030). --- docs/precision/hy4-flash.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/precision/hy4-flash.md b/docs/precision/hy4-flash.md index bbef12c81..8acb3ac54 100644 --- a/docs/precision/hy4-flash.md +++ b/docs/precision/hy4-flash.md @@ -53,7 +53,16 @@ A decode row's arithmetic does not depend on how many rows share its window. even, saturating. A code times its power-of-two scale is exact in BF16 and in fp16, so every reader decodes the row exactly: decode attention in fp32, prompt attention's tensor cores in fp16, the drafter's gather into its fp16 scratch. The window rings and the pooled keys stay fp16. -- `--kv fp8` has no score of its own against BF16 here. Needles in a 253,410-token prompt are found with both formats. +- **`--kv fp8` against BF16:** mean KLD 0.2046, top-1 86.04%, PPL 17.4371 over the 100 chunks (`--expert-cache 4800`, T 1, + 25,500 rows). The default's fp16 rows score mean KLD 0.2003, top-1 85.87% and PPL 17.4934 on the same tier size, so the + format costs 0.0043 KLD and a heavier tail (p99 3.21 against 3.01, max 23.3 against 20.8) while top-1 and PPL do not move + beyond their spread. It halves the cache: 8.6 KB a position against 14.6, and 8.4 GiB against 14.3 at 256K. The default stays + fp16, the format the BF16 reference kept; `--kv fp8` is the memory choice. The summary's 0.201 mean KLD for the pack was + measured before this comparison and without a pinned tier; the fp16 rows here land within 0.001 of it. +- Needles in a 253,410-token prompt are found with both formats. +- Pin the tier when comparing formats: this model declares no `quality_flags` for the gate, so pass `--expert-cache N` (and the + same count) to both runs. Without it the freer VRAM of fp8 moves experts onto the card and the two runs are not comparable: an + unpinned pair reads 0.2085 against 0.2030, the opposite sign to the pinned one. ## Experts and placement