Skip to content

convert: build the PLE n-gram table as IQ4_NL, 23 GiB smaller - #311

Open
giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:qwen-iq4-ple
Open

giveen wants to merge 1 commit into
jesdga95:mainfrom
giveen:qwen-iq4-ple

Conversation

@giveen

@giveen giveen commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

What

The engine's own header says the PLE n-gram table is IQ4_NL, 90 B a row (ngram.hpp:8; its reader test calls that
the production default, with Q5_0/Q8_0/BF16 as the alternatives), but tools/convert wrote Q8_0, 170 B a row. That
table is 41 % of the pack.

--ple-q iq4_nl builds it as the model has it. q8_0 stays the default, so an existing build is unchanged.

Numbers

Same sources (BF16 + NVIDIA's NVFP4 checkpoint), same --sections-from base, one flag different:

shipped pack --ple-q iq4_nl
pack 122.6 GiB 99.4 GiB (−23.2 GiB, −19 %)
gguf section 55.60 GiB 32.39 GiB
PLE table 50.66 GiB (Q8_0, 170 B/row) 26.82 GiB (IQ4_NL, 90 B/row)

What that is worth to a low-RAM user

23 GiB is storage and RAM the model no longer needs. The PLE table is the part a small machine must hold or stream beside
the 63 GiB expert section, and halving its row bytes nearly halves that cost. It is also the gap to the NInfer artifact
(101 GiB, NF4 PLE) accounted for — their PLE choice is the whole difference in size.

It does not speed this machine up, and was never expected to: the gather is sixteen page faults a token either way,
because the cost is latency, not bandwidth (ngram.cpp:473).

Speed: unchanged, measured interleaved

A B B A on one RTX 5090, same binary:

struct prose
shipped 428.8, 439.6 → 434.2 229.4, 223.3 → 226.4
IQ4 PLE 436.1, 428.2 → 432.2 221.6, 209.6 → 215.6

Struct is flat (−0.5 %, spread ±2.5 %). Prose sits inside its own ±5 % spread and is confounded either way: the fixture
stops naturally, so it generated 981–1197 tokens across the four runs.

Verified

  • Rows decode to the FP8 source within the 4-bit block step (mean error 6.4e-4 on values averaging 7.4e-3).
  • The tensor is IQ4_NL, type 20, shape [160, 320001536], exactly 90 B a row.
  • The engine opens the pack as PleTable IQ4_NL, and the struct fixture still produces its 7230 tokens.
  • pytest tools/convert/test_convert.py → 28 passed, including a byte-exact check of the encoder against ggml's
    block_iq4_nl packing.
  • Also fixes the PLE test's np.uint32(x).view() under numpy 2, which failed before this change.

Not done: the KLD gate

No KLD run has been made. Scoring needs flashnext-bf16-base.kld, which is neither on disk here nor on the model
card; quality.md generates it from the BF16 GGUF with llama-perplexity --chunks 100. What we have instead is a
sensitivity signal, not a quality one: greedy output differs from the first token, so this is real numerics. The pack
to judge it against is the DQ8 baseline on the card — KLD 0.0561, top-1 92.25 %.

I would not make this the default until that delta is measured. As an option for machines where 23 GiB is the difference
between running and not, it is worth having.

Command

python3 tools/convert/convert.py --checkpoint DIR --sections-from Qwen3.8-FN-NVFP4.v2.basalt \
    --dense-q8-0 --ple-q iq4_nl --out Qwen3.8-FN-NVFP4-iq4ple.basalt

The engine's own header says the table is IQ4_NL, 90 B a row, and its
reader calls that the production default - but the converter wrote Q8_0,
170 B a row. That is 50.66 GiB of a 122.6 GiB pack, 41 % of it, at twice
the bytes the model has.

`--ple-q iq4_nl` encodes the FP8 codes x scale as ggml block_iq4_nl (one
fp16 scale a 32 values, the nearest codebook entry) in bounded row chunks;
`q8_0` stays the default so an existing build is unchanged. Measured on
Qwen3.8-Flash-Next from the same sources and the same --sections-from
base: 122.6 -> 99.4 GiB, gguf 55.60 -> 32.39, the table 26.82 GiB.

The rows decode to the FP8 source within the 4-bit block step (mean error
6.4e-4 on values averaging 7.4e-3), the engine opens the pack as IQ4_NL,
and the struct fixture still produces its 7230 tokens.

Also fixes the PLE test's np.uint32(x).view() under numpy 2, which failed
before this change.
@giveen

giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto 1771f9d86 (2437f9aaf), which cleared the conflicts with the converter's newer multi-file PLE shard handling: PleFp8 now takes the shard file list the checkpoint reader returns and keeps the quant parameter on top of it, jobs() carries the file per shard, and the writer still records type/type_id from the chosen row format. --ple-q reaches it as a.ple_q.

Tests: tools/convert/test_convert.py 29 passed, including a new assertion that the same table as --ple-q iq4_nl is 90 B a row with type IQ4_NL/20, its bytes equal fp8_rows_to_iq4_nl of the shards, its nbytes is rows * 90, and an unknown quant is refused. The wider tools/convert suite shows 7 failures (qwen35 projections, hy4flash BF16 engram/verify) that are identical on main in this checkout, so they are not from this change.

The pack numbers in the description are unchanged: 122.6 GiB -> 99.4 GiB (-23.2 GiB) with the table at 26.82 GiB instead of 50.66 GiB.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant