[Model] 512-experts NVFP4 MoE generates garbage at deep layers (L44/L46 decode outputs 400+, prefill hidden grows unbounded)

[johnbluer]

Environment

• 1Cat-vLLM 1.5.0 (docker image 1cat-vllm-sm70:1.5.0-gpuids)
• 8? Tesla V100-SXM3-32GB, TP8, --language-model-only, KV cache float16 (QSA requirement)
• Model: orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4 (Qwen4Exp, 512 experts, NVFP4 experts + FP8 attention/linear, ngram PLE, QSA)

Note: the marlin tile_n=64 issue (see #642) was worked around with zero-padding; this report is about what happens after the model loads.

Symptom

Model serves successfully, but output is garbage ("ervi acesaces acesaces..."), token logprobs finite (not NaN) in eager mode.

Layer-level tracing (custom probes in DecoderLayer.forward)

Prefill (real 32-token request), per-layer absmax:

• L0 input 0.037 ??L47 hidden 105??82 ??hidden grows unbounded across layers (~1.5??? net gain per 4-layer block; the linear_attention layers' hyper-connection gain 1.2??.2?, full_attention layers ??.0?)

Decode steps (dump via VLLM_SM70_DUMP_QWEN_LAYER_DIR):

• L44 MoE output amax = 407, L46 = 376, vs 20??3 for the other 46 layers
• The corresponding MoE inputs are normal (amax 6.1, std 0.535)
• Profile dummy with all-zero input ids triggers NaN starting at L36 and spreading +1 token per layer (causal), while alternating ids stay normal ??input-dependent

What was ruled out

• All embedding/ngram/PLE tables and 48?512 experts' scales: no NaN, global scales 2208??7360, no extreme outliers
• MoE kernel numerics: verified against fp16 reference, max_err = 0.0002 (with and without padding)
• HC (hyper-connection) weights: all 48 layers normal
• The same image + orcarouter/Qwen3.8-27B-Uncensored-NVFP4 (27B dense NVFP4, TP4) generates perfectly ("1+1=2" etc.) ??the platform itself is fine

Hypothesis / question

The deep-layer MoE blowup looks mathematically self-consistent for extreme experts: weight_scale saturated at 448 (e4m3 max) ? large fp4 values, one expert winning the router with near one-hot top-k, chained amplification at L44/L46.

Is this expected behavior for this model's NVFP4 compression (i.e., a checkpoint quality issue), or could there be an edge case in the 512-expert path? Has the team validated any 512-experts NVFP4 MoE model end-to-end? (1.5.0 release notes validated 35B-A3B hybrid FP8/NVFP4 ??much smaller expert count.)

Happy to provide the probe scripts, dump tensors, or any additional data.

---

??????:512-experts NVFP4 ???(??pad ?????TP8 ??????????????????????:prefill ??hidden ??????(L0 0.037??47 105-182);decode ??L44/L46 ??MoE ??? amax 407/376(??? 46 ??? 20-33),????????std 0.535)??????:??? NaN??oE kernel ??????? err 0.0002)??C ???????????27B dense NVFP4 ????e?)?????????a??NVFP4 ????????expert(scale ??? 448 + ??fp4)??????????checkpoint ?????????????????????????????? 512-experts NVFP4 MoE ?????