[BUG] qwen3_5 dense: mixed-precision NVFP4 crashes in ct_bf16_fuse (Float8 × BFloat16 promotion)

[zhaochang22]

[BUG] qwen3_5 dense checkpoints: mixed-precision NVFP4 (FP8 attention + NVFP4 MLP) crashes in ct_bf16_fuse — Float8_e4m3fn × BFloat16 promotion

Environment

• FreeToken 0.1.2 (PyPI wheel freetoken-0.1.2-cp312, kernel-cache 0.1.2+cu130)
• Python 3.12.13 (uv-managed venv), Ubuntu 22.04, driver 580.119.02
• GPU: 1× NVIDIA RTX 4090 (also reproduced logic on a 2×4090 box; TP=1)
• Checkpoint: unsloth/Qwen3.8-27B-NVFP4 (llm-compressor export, quantization_config.format == "mixed-precision")

Summary

Serving a Qwen3.8-27B dense checkpoint (qwen3_5 family) quantized with the standard NVFP4 recipe — attention/GDN projections FP8, rest NVFP4 — crashes during weight loading. Both ft serve and ft checkpoint fail identically, which suggests the shared compressed-tensors walker lacks a dequant branch for Float8-quantized linears when they land in a bf16 fuse group.

Reproduction

ft serve --model unsloth/Qwen3.8-27B-NVFP4 --model-source modelscope
# same failure with:
ft serve --model unsloth/Qwen3.8-27B-NVFP4 --model-source modelscope --nvfp4-backend flashinfer
ft checkpoint --model <snapshot-dir> --out <out-dir>

Error

RuntimeError: Promotion for Float8 Types is not supported, attempted to promote Float8_e4m3fn and BFloat16

Key frames (identical for serve and checkpoint):

freetoken/engine/engine.py(459)        _load_weight_state_dict
freetoken/models/weight.py(251)        load_weight
freetoken/models/qwen3_5_moe/weight.py(648)  _iter_weights_compressed_tensors
freetoken/models/qwen3_5_moe/weight.py(589)  _emit_bf16_weight
freetoken/models/loader.py(251)        ct_bf16_fuse  →  torch.cat([...], dim=0)

Analysis

• The checkpoint's config.json reports quantization_config.format == "mixed-precision" with 303 ignored modules — the usual llm-compressor NVFP4 recipe (FP8 for attention/GDN linears, NVFP4 for the rest).
• The NVFP4 (weight_packed 4-bit) parts dequantize fine — loading progresses ~5s deep before failing.
• The crash happens when a fuse group for GDN in_proj is concatenated: one member tensor is still raw Float8_e4m3fn (an FP8-quantized linear that the compressed-tensors walker passed through unconverted), and torch.cat refuses to promote Float8 against BFloat16.
• --nvfp4-backend triton|flashinfer makes no difference (failure is upstream of kernel backends).

Expected behavior

Either (a) Float8-quantized linears in compressed-tensors checkpoints are dequantized to the compute dtype before fusion/emit, or (b) a clear early error naming the unsupported mixed format.

Notes

• A single-recipe export (Qwen/Qwen3.8-27B-FP8, pure FP8) presumably takes the mainline FP8 path and is unaffected — not yet tested.
• Side note: docs/models.md does not list Qwen3.8 yet, though the Desktop catalog does; might be worth a docs refresh alongside the fix.
• Happy to test a patch.

[zhaochang22]

Cross-referencing two related reports found while scanning the tracker:

• Loading nvidia/Qwen3.6-35B-A3B-NVFP4 fails: "Incomplete fp8 fusions: ['model.layers.37.linear_attn.in_proj_qkvz']" #206 (fixed 2026-08-26): the same GDN fuse group (linear_attn.in_proj_qkvz) failing on FP8 content, but for nvidia/Qwen3.6-35B-A3B-NVFP4. That fix appears not to cover the dense qwen3_5 path with an unsloth mixed-precision export (quantization_config.format == "mixed-precision": FP8 attention/GDN linears + NVFP4 rest), which is exactly where this issue crashes — ct_bf16_fuse → torch.cat hitting Float8_e4m3fn × BFloat16 promotion.
• Qwen3.8-27B-NVFP4 fails to load — RuntimeError: CUDA driver error: device not ready #139: the same checkpoint (unsloth/Qwen3.8-27B-NVFP4) failing with a different symptom (CUDA driver error: device not ready); possibly the same loader gap surfacing differently on that setup.

Happy to test a fix for the dense mixed-precision path.

[dcutugno]

Confirming this on a second platform and a second checkpoint variant, plus one data point the report leaves open.

Env: freetoken 0.1.2+g932e87e8a, Python 3.12, Windows 11, RTX 50-series (sm_120), TP=1.

Checkpoint: a third-party Qwen3.8-27B multimodal NVFP4 export — architectures: ["Qwen3_5ForConditionalGeneration"], model_type: qwen3_5, compressed-tensors 0.18.0, quantization_config.format == "mixed-precision". So this is not specific to the text-only variant.

Both ft serve (pointed straight at the source dir) and ft checkpoint fail identically, as described:

File "python/freetoken/models/qwen3_5_moe/weight.py", line 648, in _iter_weights_compressed_tensors
File "python/freetoken/models/qwen3_5_moe/weight.py", line 589, in _emit_bf16_weight
File "python/freetoken/models/loader.py", line 256, in freetoken.models.loader.ct_bf16_fuse
RuntimeError: Promotion for Float8 Types is not supported, attempted to promote Float8_e4m3fn and BFloat16

Note this lands at loader.py:256, whereas the report has loader.py:251 inside torch.cat. I'm on a dev build and the report used the plain PyPI 0.1.2 wheel, so that may just be line drift — but if they are genuinely two sites in ct_bf16_fuse (a torch.cat and a scale multiply), a fix covering only one would leave the other. Worth a check by someone with the source.

Storage layout of the two groups:

group	format	targets	tensors
group_0	float-quantized, 8-bit	232	weight F8_E4M3 + weight_scale BF16, per-channel [out, 1]
group_1	nvfp4-pack-quantized	168	weight_packed U8 + weight_scale F8_E4M3 + weight_global_scale / input_global_scale F32

transform_config and sparsity_config are both empty, so nothing else has to be reconstructed. group_0 covers linear_attn.in_proj_qkv / in_proj_z / out_proj, self_attn.{q,k,v,o}_proj, and some MLP projections — 9.35 B params in total.

Which tensor. Byte accounting in the convert log pins it: the first emit is 2,542,796,800 B = 248320 x 5120 x 2 (embed_tokens.weight), then +10240 (layers.0.input_layernorm), +96 (A_log), +81920 (conv1d.weight), +96 (dt_bias) — and the sixth emit is layer 0's first FP8 projection.

The scale dtype is a red herring. Torch has no promotion for float8 against any float type, so storing weight_scale as F32 would not help either — the FP8 weight has to be cast first:

w = torch.zeros(2, 2, dtype=torch.float8_e4m3fn)
for dt in (torch.float32, torch.float16, torch.bfloat16):
    w * torch.ones(2, 1, dtype=dt)   # RuntimeError: Promotion for Float8 Types is not supported
w.to(torch.bfloat16) * scale         # ok

Answering the open question in the report ("a single-recipe export (pure FP8) presumably takes the mainline FP8 path and is unaffected — not yet tested"): confirmed unaffected. The same model published as a single-recipe FP8 export (quantization_config.quant_method: "fp8") takes the mainline path — the log shows Loading fp8 weights rather than Loading compressed-tensors weights — and converts cleanly in 36 s, 915 weight tensors, 0 expert banks. Only the compressed-tensors walker is affected.

On #275: the diff routes the bare .weight branch through _load_maybe_quantized, which is exactly the tensor class failing here, so it should cover this case. Happy to test a build.

One design note for whoever picks this up: _emit_bf16_weight means the 232 group_0 tensors get dequantized 8.7 GiB -> 17.4 GiB, so a cast-only fix produces an FTW noticeably larger than the source checkpoint. Keeping them FP8 the way the quant_method: "fp8" path already does would be the better end state.

[yuyi2439]

#343 implements mixed-precision quantization support, but only for experts. The algorithm should be general, and then we can use it to load mixed-precision dense weight