feat(models): FP8 E4M3 block-scale dequant + shared expert FP4 support (DSV4 Phase 1) - #66
Merged
Merged
Conversation
…SV4 Phase 1) Add FP8 E4M3 + UE8M0 per-128x128 block-scale dequantization for DeepSeek-V4-Flash attention weights (wq_a, wq_b, wkv, wo_a, wo_b). New module: quant/fp8_block.rs - dequantize(weights, scales, rows, cols) applies per-block E8M0 scale to E4M3 weight elements, supporting partial last blocks - 17 unit tests covering single/multi-block, partial blocks, edge cases, and real V4-Flash attention dimensions Safetensors loader changes: - dequantize_fp8_block_scaled: detects (.weight=F8_E4M3, .scale=F8_E8M0) companion pairs and dequantizes before the main loop sees them - Extended dequantize_per_expert_mxfp4 to also catch shared_experts.w[123] (was only catching per-routed-expert weights) - Main loop now skips tensors consumed by both dequantizers Expert FP4 E2M1 dequant was already working via dequantize_per_expert_mxfp4; this commit extends it to shared experts and adds the missing FP8 attention weight path.
…dequant Adds 3 integration tests in test_loading.rs that exercise the new safetensors loader code paths: - FP8 block-scaled attention weight dequant (wq_a, wkv) - Shared expert MXFP4 dequant (shared_experts.w[123]) - Multi-block FP8 with different per-block scales These cover the dequantize_fp8_block_scaled and extended shared expert pattern matching, bringing safetensors.rs back above the 89% coverage policy threshold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeepSeek-V4-Flash Phase 1: FP8/FP4 dequantization.
FP8 E4M3 + UE8M0 block-scale dequant (NEW)
V4-Flash attention weights (wq_a, wq_b, wkv, wo_a, wo_b) are stored as FP8 E4M3 with per-128x128 block UE8M0 scales — a format not previously handled by LARQL.
New module
quant/fp8_block.rs:dequantize(weights, scales, rows, cols)applies the per-block E8M0 scale to E4M3 weight elementsShared expert FP4 dequant (EXTENDED)
The existing
dequantize_per_expert_mxfp4only caught per-routed-expert weights (experts.<digit>.w[123].weight). V4-Flash also has a shared expert with the same I8+F8_E8M0 format:shared_experts.w[123].weight. Extended the pattern matcher to catch both.Safetensors loader wiring
dequantize_fp8_block_scaled: detects (.weight=F8_E4M3, .scale=F8_E8M0) companion pairs and dequantizes to f32 before the main tensor loopWeight format coverage after this PR
Test results (Valinor RTX 3060, Rust 1.97.0)
Phase context
Part of docs/dsv4-flash-implementation-plan.md: