Skip to content

docs: what --kv fp8 costs Hy4-flash, and how to compare the formats - #472

Open
giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:docs/hy4-fp8-quality
Open

giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:docs/hy4-fp8-quality

Conversation

@giveen

@giveen giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

The precision page said the fp8 rows have "no score of its own against BF16". Measured now, on this box (RTX 5090, 475 W), models/Hy4-flash-NVFP4.basalt, hy4flash-bf16-base.kld, 100 chunks, T 1, 25,500 rows, and the tier pinned at the same size for both runs:

--expert-cache 4800 --kv fp16 (default) --kv fp8
PPL 17.4934 ± 0.4416 17.4371 ± 0.4396
KLD mean 0.20033 0.20459
median 0.03168 0.03231
p99 3.0059 3.2121
max 20.7754 23.2828
top-1 85.87% 86.04%

The format costs 0.0043 mean KLD and a heavier tail, and halves the cache (14.6 → 8.6 KB a position, 14.3 → 8.4 GiB at 256K). The default stays fp16 — it is the format the BF16 reference kept, and the difference is real even though top-1 and PPL do not separate. --kv fp8 stays the memory choice with its published cost. If you would rather have fp8 by default, that is a one-line change plus this table; I did not make it.

Two method notes the page now carries, both found while measuring:

  1. The gate is unpinned for this model. --gpu-plan reports "quality_flags":"" for hy4flash, while qwen4exp pins --spec 5 --adapt-swaps 0 --peer-adapt-swaps 0 --resident-experts. The method page says to pin the placement first. Comparing the two formats without pinning flips the sign (unpinned: KLD 0.2085 fp16 against 0.2030 fp8), because fp8's freer VRAM moves experts onto the card. hy4flash rejects --adapt-swaps, so its pin has to be a fixed --expert-cache, which the page now says to pass. Giving the model a proper quality_flags is a separate change.
  2. The summary's 0.201 mean KLD for the pack was measured before this comparison and without a pinned tier; the fp16 rows above land within 0.001 of it.

No code changes; docs only. Reviewer's call whether the default should follow.

The precision page said the fp8 rows had no score of their own against BF16. They do now: with the tier pinned at 4,800 slots, 100 chunks of the hy4flash BF16 base, T 1, 25,500 rows, mean KLD 0.2046 (default fp16: 0.2003), top-1 86.04% against 85.87%, PPL 17.4371 against 17.4934, and a heavier tail (p99 3.21 against 3.01). The default stays fp16: the difference is real and the memory win is a flag, not a default.

The comparison needs the tier pinned, and this model declares no quality_flags for the gate, so the page now says to pass the same --expert-cache to both runs; the unpinned pair reads the opposite sign (0.2085 against 0.2030).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant