[Bug]: qwen3_5_moe fails to convert/load compressed-tensors NVFP4 checkpoints with model.language_model prefix

[silvertakana]

Description

When attempting to convert or load a compressed-tensors (llm-compressor) NVFP4 export of a Qwen3.5 / Agents-A1 MoE model (e.g. Agents-A1-NVFP4 with model.language_model.* tensor hierarchy), ft checkpoint fails during expert bank conversion.

Depending on prefix handling, the conversion halts with:

1. ValueError: Missing MoE expert source layers: {'gate_up': [0..39], 'down': [0..39]} (when _NVFP4_EXPERT_KEY_RE does not match the compressed-tensors naming scheme)
2. KeyError: (0, 0, 'down_proj') or assertion failure in _iter_weights_compressed_tensors when encountering mixed precision (BF16 linear attention + NVFP4 full attention/experts).

Root Cause Analysis

1. Quant Detection (freetoken/models/qwen3_5_moe/config.py):
   _expert_quant() checks for quant_algo == "NVFP4" (ModelOpt style) or quant_algo == "MIXED_PRECISION". For checkpoints produced by llm-compressor, the top-level field is quant_method: "compressed-tensors", which results in expert_quant falling back to "none".

2. Regex & Suffix Mismatch (freetoken/models/qwen3_5_moe/weight.pyd):
   _NVFP4_EXPERT_KEY_RE expects keys matching (?P<kind>weight|weight_scale|weight_scale_2). In compressed-tensors checkpoints:

   • Packed weights are named *.weight_packed instead of *.weight.
   • Global scale scalars are named *.weight_global_scale / *.input_global_scale.
   • The tensor tree carries a model.language_model.layers... prefix rather than model.layers....

3. Hybrid Attention/GDN Quantization:
   In compressed-tensors MoE checkpoints for hybrid architectures (e.g. Qwen3.5 with GatedDeltaNet + full attention), only the full attention layers and routed experts are quantized to NVFP4, while the linear recurrent (GDN) projections (conv1d, A_log, in_proj_*, out_proj) remain in BF16. The compiled loader assumes all dense projections in a compressed-tensors checkpoint share a uniform quantization scheme.

Steps to Reproduce

1. Obtain a compressed-tensors NVFP4 checkpoint of Agents-A1 or Qwen3.5-MoE.
2. Run:

   ft checkpoint --model <checkpoint_dir> --out <ftw_dir> --dtype bfloat16 --moe-backend offload

3. Observe ValueError: Missing MoE expert source layers or failure during weight iteration.

Environment

• OS: Windows 11
• FreeToken Version: 0.1.2+g0ee42b72c
• GPU: NVIDIA GeForce RTX 4060