Repository navigation
Conversation
The engine's own header says the table is IQ4_NL, 90 B a row, and its reader calls that the production default - but the converter wrote Q8_0, 170 B a row. That is 50.66 GiB of a 122.6 GiB pack, 41 % of it, at twice the bytes the model has. `--ple-q iq4_nl` encodes the FP8 codes x scale as ggml block_iq4_nl (one fp16 scale a 32 values, the nearest codebook entry) in bounded row chunks; `q8_0` stays the default so an existing build is unchanged. Measured on Qwen3.8-Flash-Next from the same sources and the same --sections-from base: 122.6 -> 99.4 GiB, gguf 55.60 -> 32.39, the table 26.82 GiB. The rows decode to the FP8 source within the 4-bit block step (mean error 6.4e-4 on values averaging 7.4e-3), the engine opens the pack as IQ4_NL, and the struct fixture still produces its 7230 tokens. Also fixes the PLE test's np.uint32(x).view() under numpy 2, which failed before this change.
|
Rebased onto Tests: The pack numbers in the description are unchanged: 122.6 GiB -> 99.4 GiB (-23.2 GiB) with the table at 26.82 GiB instead of 50.66 GiB. |
What
The engine's own header says the PLE n-gram table is IQ4_NL, 90 B a row (
ngram.hpp:8; its reader test calls thatthe production default, with Q5_0/Q8_0/BF16 as the alternatives), but
tools/convertwrote Q8_0, 170 B a row. Thattable is 41 % of the pack.
--ple-q iq4_nlbuilds it as the model has it.q8_0stays the default, so an existing build is unchanged.Numbers
Same sources (BF16 + NVIDIA's NVFP4 checkpoint), same
--sections-frombase, one flag different:--ple-q iq4_nlggufsectionWhat that is worth to a low-RAM user
23 GiB is storage and RAM the model no longer needs. The PLE table is the part a small machine must hold or stream beside
the 63 GiB expert section, and halving its row bytes nearly halves that cost. It is also the gap to the NInfer artifact
(101 GiB, NF4 PLE) accounted for — their PLE choice is the whole difference in size.
It does not speed this machine up, and was never expected to: the gather is sixteen page faults a token either way,
because the cost is latency, not bandwidth (
ngram.cpp:473).Speed: unchanged, measured interleaved
A B B A on one RTX 5090, same binary:
Struct is flat (−0.5 %, spread ±2.5 %). Prose sits inside its own ±5 % spread and is confounded either way: the fixture
stops naturally, so it generated 981–1197 tokens across the four runs.
Verified
IQ4_NL, type 20, shape[160, 320001536], exactly 90 B a row.PleTableIQ4_NL, and the struct fixture still produces its 7230 tokens.pytest tools/convert/test_convert.py→ 28 passed, including a byte-exact check of the encoder against ggml'sblock_iq4_nlpacking.np.uint32(x).view()under numpy 2, which failed before this change.Not done: the KLD gate
No KLD run has been made. Scoring needs
flashnext-bf16-base.kld, which is neither on disk here nor on the modelcard;
quality.mdgenerates it from the BF16 GGUF withllama-perplexity --chunks 100. What we have instead is asensitivity signal, not a quality one: greedy output differs from the first token, so this is real numerics. The pack
to judge it against is the DQ8 baseline on the card — KLD 0.0561, top-1 92.25 %.
I would not make this the default until that delta is measured. As an option for machines where 23 GiB is the difference
between running and not, it is worth having.
Command