Question about outlier tokens of MoE, like DeepSeek-v2?

[L1aoXingyu]

Thank you for your great job!
I found the experiments are all conducted on Llama series. I'm particularly interested in whether you've extended your analysis of outlier tokens to Mixture-of-Experts (MoE) architectures. Could you please shed some light on it?

[ChenMnZ]

Hi,I haven't tried the Moe model yet.

In my guess, Moe models will show similar token-wise outliers like llama models.

[L1aoXingyu]

thanks for your kind reply.
from your experience, down_proj in moe is still the best aspect or candidate to observe for outlier token?

[ChenMnZ]

@L1aoXingyu
I recommend use the output of transformer block to observe outlier tokens. There are so many down_proj in one MoE transformer block, which leads some tedious situation if consider down_proj.

[L1aoXingyu]

thank you for your valuable suggestion!

currently, I’m using shared_moe in DeepSeek and Qwen, which involves a single down_proj for each transformer block. I agree that utilizing the output of the transformer block could be a more effective approach. I’ll look into implementing this improvement in my setup.