feat(gguf): implement Q8_0 dequantization - #366
Closed
Cyber-Marty wants to merge 2 commits into
Closed
Conversation
added 2 commits
September 3, 2026 10:56
parse_gguf_config unconditionally required gemma4.expert_count and hardcoded moe_enabled=True, so a dense Gemma-4 GGUF (gemma-4-12B-it, gemma-4-31B-it) could not load at all — the GGUF path structurally could not represent a non-MoE checkpoint, while docs/models.md lists dense family members and GGUF is the documented non-safetensors path. Mirror the HF-side parse_config: default the expert fields to 0 when absent from the metadata, derive moe_enabled = num_experts > 0, and gate expert_quant/moe_weight_format on it. MoE GGUFs parse exactly as before. Regression tests build a minimal GGUF header (magic + version + zero counts) so the parser runs against metadata dicts without any model download: MoE keys present -> moe path unchanged; keys absent or zero -> dense path with expert_quant none. Fixes FlashML-org#357
dequant.py declared Q8_0 in BLOCK_SHAPE, GGML_NAME and __all__ but had no dequant_q8_0, so dequantize() raised NotImplementedError on the format that the most widely distributed Gemma-4 GGUFs (unsloth's UD-*_XL dynamic quants) use on attention projections, the dense FFN and token_embd -- 237 of 658 tensors in the 26B-A4B UD-Q6_K_XL file. dequant_q8_0 follows the ggml block_q8_0 layout: fp16 scale d + 32 int8 quants, w = d*q with no offset. Tests round-trip through a reference quantizer mirroring quantize_row_q8_0 (d = max|w|/127, q = round(w/d)), bounding the error at half a quantization step, plus a hand-computed exact-values case and a dequantize() dispatch check. All pure torch / CPU. Fixes FlashML-org#358
Author
|
Closing — this branch accidentally carried the unmerged #359 commits (branched from the wrong base). Reopening from a clean main-based branch with only the Q8_0 changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dequant.pydeclared Q8_0 inBLOCK_SHAPE,GGML_NAMEand__all__, but there was nodequant_q8_0, sodequantize()raisedNotImplementedErroron the format that the mostwidely distributed Gemma-4 GGUFs use: unsloth's
UD-*_XLdynamic quants place Q8_0 on theattention projections, the dense FFN and
token_embd— 237 of 658 tensors ingemma-4-26B-A4B-it-UD-Q6_K_XL.gguf(see #358).Fix
dequant_q8_0follows the ggmlblock_q8_0layout: per 32-elem block, fp16 scaledq,w = d*qwith no offset (unlike Q4_0's(q-8)*d— ggml'squantize_row_q8_0storesq = round(w/d),d = max|w|/127).Registered in
_DEQUANTand__all__; module docstring updated to list Q8_0. Noexisting code path changes — Q4_0 / Q6_K / F32 / F16 / BF16 untouched.
Tests
New
tests/models/test_dequant_q8_0.py(4 cases, pure torch / CPU):quantize_row_q8_0) →dequant_q8_0→ reconstruction error bounded by half a quantization step;d=0.5, quants incl. -128/127/0) decodes exactly;dequantize(raw, GGML_Q8_0)routes instead of raising;BLOCK_SHAPE[Q8_0] == (32, 34)androw_bytesmath unchanged.tests/models/test_gemma4_gguf_config.py+test_dequant_q8_0.py+test_cpu_moe_q4_0.py:7 passed, 5 skipped (CUDA-only cases, by design).
Fixes #358