Repository navigation
Conversation
The precision page said the fp8 rows had no score of their own against BF16. They do now: with the tier pinned at 4,800 slots, 100 chunks of the hy4flash BF16 base, T 1, 25,500 rows, mean KLD 0.2046 (default fp16: 0.2003), top-1 86.04% against 85.87%, PPL 17.4371 against 17.4934, and a heavier tail (p99 3.21 against 3.01). The default stays fp16: the difference is real and the memory win is a flag, not a default. The comparison needs the tier pinned, and this model declares no quality_flags for the gate, so the page now says to pass the same --expert-cache to both runs; the unpinned pair reads the opposite sign (0.2085 against 0.2030).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The precision page said the fp8 rows have "no score of its own against BF16". Measured now, on this box (RTX 5090, 475 W),
models/Hy4-flash-NVFP4.basalt,hy4flash-bf16-base.kld, 100 chunks, T 1, 25,500 rows, and the tier pinned at the same size for both runs:--expert-cache 4800--kv fp16(default)--kv fp8The format costs 0.0043 mean KLD and a heavier tail, and halves the cache (14.6 → 8.6 KB a position, 14.3 → 8.4 GiB at 256K). The default stays fp16 — it is the format the BF16 reference kept, and the difference is real even though top-1 and PPL do not separate.
--kv fp8stays the memory choice with its published cost. If you would rather have fp8 by default, that is a one-line change plus this table; I did not make it.Two method notes the page now carries, both found while measuring:
--gpu-planreports"quality_flags":""forhy4flash, whileqwen4exppins--spec 5 --adapt-swaps 0 --peer-adapt-swaps 0 --resident-experts. The method page says to pin the placement first. Comparing the two formats without pinning flips the sign (unpinned: KLD 0.2085 fp16 against 0.2030 fp8), because fp8's freer VRAM moves experts onto the card.hy4flashrejects--adapt-swaps, so its pin has to be a fixed--expert-cache, which the page now says to pass. Giving the model a properquality_flagsis a separate change.No code changes; docs only. Reviewer's call whether the default should follow.