DeepSeek-R1 support

[loretoparisi]

Add support or a recipe to quantize DeepSeek-R1 and related distilled versions DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Qwen-32B, eventually including 7B, 14B Qwen, and Llama 8B

[YangWang92]

We are almost there, and I have prepared the hessian collection codes for Deepseek R1/V3. Please wait for us for one week.

[wejoncy]

Here is a way to quantize those related distilled versions

1. install vptq-algorithm by following https://github.com/microsoft/VPTQ/blob/algorithm/algorithm.md#environment-setting
2. install qllm by pip install qllm
3. create a example quant-config.json python -m qllm --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --quant_method=vptq --quant_config=help
4. copy and save the config, edit those values with your thought
5. do quatization python -m qllm --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --quant_method=vptq --quant_config=xx.json --save DeepSeek-R1-Distill-Qwen-32B-vptq

[YangWang92]

Some updates: we have already collected all hessian matrices and are trying to adapt the quantization algorithm.

[YangWang92]

Some updates: I have adapted the algorithm to deepseek models and stay tuned.

[YangWang92]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

[loretoparisi]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

💯 This is truly amazing.
Running a test on our 4 x H100 today!

[YangWang92]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

💯 This is truly amazing. Running a test on our 4 x H100 today!

The current inference speed is still a bit slow (based on deepseek repo's bare torch design). Please wait a moment while I prepare it—you can give it a try soon.

[YangWang92]

Hi @loretoparisi ,
Sorry about the late, we have supported deepseek model at here. Thanks!
https://github.com/VPTQ/DeepSeek-V3

[YangWang92]

Our next milestone will be supporting VPTQ in sglang and vllm. Thanks!

[ZanePoe]

Our next milestone will be supporting VPTQ in sglang and vllm. Thanks!我们的下一个里程碑将是在sglang和vllm中支持VPTQ。 谢谢!

Looking forward to it! I tried reasoning VPTQ models using the aphrodite-engine framework, but ran into a bit of a problem。
Fow now, quantized model requires prefix to imply which layer it is., just like vLLM/sglang did, https://github.com/vllm-project/vllm/blob/54a8804455a14234ba246f7cbaf29fb5e8587d64/vllm/model_executor/models/qwen2.py#L80C1-L81C1.Seems like we might need a PR to support str[prefix] for qwen/qwen2 or other models.
aphrodite-engine issue

[YangWang92]

Our next milestone will be supporting VPTQ in sglang and vllm. Thanks!我们的下一个里程碑将是在sglang和vllm中支持VPTQ。 谢谢!

Looking forward to it! I tried reasoning VPTQ models using the aphrodite-engine framework, but ran into a bit of a problem。 Fow now, quantized model requires prefix to imply which layer it is., just like vLLM/sglang did, https://github.com/vllm-project/vllm/blob/54a8804455a14234ba246f7cbaf29fb5e8587d64/vllm/model_executor/models/qwen2.py#L80C1-L81C1.Seems like we might need a PR to support str[prefix] for qwen/qwen2 or other models. aphrodite-engine issue

Let me see, I've been a bit busy lately.