Skip to content

Add test result in Ascend910B, change strategies for verified models #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Mario-LuX merged 1 commit into from
Nov 5, 2026
Merged

Add test result in Ascend910B, change strategies for verified models #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 20 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,17 @@ English | [简体中文](README.zh-CN.md)
> `CONTINUATION_ENGINE`. Within this project, each engine + model combination has its own
> continuation strategy. The combinations verified so far:
>
> **SGLang(tested on v0.5.19+)**
> Status: ✅ verified, ❌ not supported.
>
> **SGLang (NVIDIA GPU, tested on v0.5.19+)**
>
> | Model | Status | Notes |
> | --- | --- | --- |
> | Kimi-K3 | ✅ Verified | |
> | Qwen3.6-35B-A3B | ✅ Verified | |
> | GLM-5.3 | ✅ Verified | |
>
> **vLLM(tested on v0.30.0+)**
> **vLLM (NVIDIA GPU, tested on v0.30.0+)**
>
> | Model | Status | Notes |
> | --- | --- | --- |
Expand All @@ -24,10 +26,23 @@ English | [简体中文](README.zh-CN.md)
> | DeepSeek-V4-Flash-0731-nvfp4 | ✅ Verified | |
> | Kimi-K3 | ✅ Verified | vLLM must be started with `--enable-scale-out`, otherwise continuation does not work. Multimodal requests are not continued |
>
> **vLLM (Ascend 910B4, tested on vllm-ascend 0.30.0+)**
>
> | Model | Status | Notes |
> | --- | --- | --- |
> | Qwen3.6-35B-A3B | ✅ Verified | Tested with the W8A8 weights |
> | Qwen3.6-27B | ✅ Verified | Tested with the W8A8 weights |
> | Qwen3-30B-A3B | ✅ Verified | Tested with the W8A8 weights |
> | Qwen3-14B | ✅ Verified | |
> | Qwen3-8B | ✅ Verified | |
> | Gemma-4-31B-it | ✅ Verified | The model itself is occasionally unstable: it may emit its control tokens as plain text in `content` (for example `</think>`, `<\|tool_call>`) so that the tool_call is not parsed and is lost. This comes from the model and its parser, not from how the continuation prefix is built |
> | Gemma-4-26B-A4B-it | ✅ Verified | Same as Gemma-4-31B-it. Its reasoning may also occasionally end up in `content` |
> | Qwen3-Omni-30B-A3B-Instruct | ❌ Not supported | |
>
> The model names in the tables are exactly the strings `CONTINUATION_MODEL` should be set to
> (case-insensitive, exact match). A quantized or fine-tuned variant of a listed model, such as
> `GLM-5.3-FP8`, should use the listed name `GLM-5.3`; a different size or version
> (`Qwen3.6-27B`, `GLM-5.4`) is a different model and is not matched.
> (`Qwen3-32B`, `GLM-5.4`) is a different model and is not matched.
>
> Other models and engines have not been verified; they may work, but please verify them
> yourself before relying on them. We are continuing to adapt more engine + model
Expand All @@ -51,9 +66,8 @@ deployment actually serves; the `model` field of the request is not consulted),
`continuation_gateway/reconstruct.py`. Two formats are built in: Kimi-K3 (XTML) and a generic
`<think>...</think>` format. Supporting a model with another format only takes adding a
builder there. A model that matches no verified model is handled with the generic format, and
the gateway logs a warning at startup. On vLLM, Qwen3.6-35B-A3B additionally sends the
content-done partial message as separate `reasoning` / `content` fields, because its chat
template rewrites the string form and the engine then rejects it with a 400.
the gateway logs a warning at startup. On vLLM, some models (such as Qwen3.6-35B-A3B) additionally split
the content-done partial message into separate `reasoning` / `content` fields.

```
leg1: original request
Expand Down
26 changes: 20 additions & 6 deletions README.zh-CN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,17 @@
> 网关支持 **SGLang** 和 **vLLM** 两种推理引擎，通过 `CONTINUATION_ENGINE` 指定。本项目针对
> 不同的「引擎 + 模型」组合各有对应的续写策略，目前已验证的组合如下：
>
> **SGLang(在v0.5.19+版本上验证)**
> 状态：✅ 已验证，❌ 不支持。
>
> **SGLang（NVIDIA GPU，在 v0.5.19+ 版本上验证）**
>
> | 模型 | 状态 | 备注 |
> | --- | --- | --- |
> | Kimi-K3 | ✅ 已验证 | |
> | Qwen3.6-35B-A3B | ✅ 已验证 | |
> | GLM-5.3 | ✅ 已验证 | |
>
> **vLLM(在v0.30.0+版本上验证)**
> **vLLM（NVIDIA GPU，在 v0.30.0+ 版本上验证）**
>
> | 模型 | 状态 | 备注 |
> | --- | --- | --- |
Expand All @@ -23,9 +25,22 @@
> | DeepSeek-V4-Flash-0731-nvfp4 | ✅ 已验证 | |
> | Kimi-K3 | ✅ 已验证 | vLLM 必须开启 `--enable-scale-out`，否则续写无效；不支持多模态请求的续写 |
>
> **vLLM（昇腾 910B4，在 vllm-ascend 0.30.0+ 版本上验证）**
>
> | 模型 | 状态 | 备注 |
> | --- | --- | --- |
> | Qwen3.6-35B-A3B | ✅ 已验证 | 使用 W8A8 权重测试 |
> | Qwen3.6-27B | ✅ 已验证 | 使用 W8A8 权重测试 |
> | Qwen3-30B-A3B | ✅ 已验证 | 使用 W8A8 权重测试 |
> | Qwen3-14B | ✅ 已验证 | |
> | Qwen3-8B | ✅ 已验证 | |
> | Gemma-4-31B-it | ✅ 已验证 | 模型自身偶发不稳定：可能把控制 token 当作普通文本输出到 `content`（例如 `</think>`、`<\|tool_call>`），导致 tool_call 没有被解析出来而丢失。这来自模型及其解析器，不是续写前缀的构造问题 |
> | Gemma-4-26B-A4B-it | ✅ 已验证 | 同 Gemma-4-31B-it；另外它的思考内容偶发会落进 `content` |
> | Qwen3-Omni-30B-A3B-Instruct | ❌ 不支持 | |
>
> 表中的模型名就是 `CONTINUATION_MODEL` 应设置的字符串（大小写不敏感，精确匹配）。表中模型
> 的量化或微调版本，例如 `GLM-5.3-FP8`，应直接设置为表里的名字 `GLM-5.3`；不同的尺寸或版本
> （`Qwen3.6-27B`、`GLM-5.4`）是另一个模型，不会被匹配。
> （`Qwen3-32B`、`GLM-5.4`）是另一个模型，不会被匹配。
>
> 其它模型和引擎尚未验证，可能可用，使用前请自行验证。我们仍在持续适配更多的引擎 + 模型组合。

Expand All @@ -40,9 +55,8 @@ LLM 推理服务的崩溃续写网关：位于客户端和 OpenAI 兼容的推
前缀拼接方式由 `CONTINUATION_MODEL`（本部署实际 serve 的真实模型，不看请求里的 `model`
字段）决定（见 `continuation_gateway/reconstruct.py`），内置 Kimi-K3（XTML）和通用
`<think>...</think>` 两种格式，接入其它格式的模型只需要在那里加一个 builder。没有匹配到已验证
模型的，按通用格式处理，并在启动时打一条 warning。vLLM 上的 Qwen3.6-35B-A3B 在 content-done
时额外把 partial 消息拆成 `reasoning` / `content` 两个字段发送，因为它的 chat template 会重排
字符串形式，引擎随后以 400 拒绝。
模型的，按通用格式处理，并在启动时打一条 warning。vLLM 上部分模型（如 Qwen3.6-35B-A3B）在
content-done 时会额外把 partial 消息拆成 `reasoning` / `content` 两个字段发送。

```
leg1：原始请求
Expand Down
12 changes: 9 additions & 3 deletions continuation_gateway/reconstruct.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,9 @@ def _build_prefix_think_tag(reasoning: str, content: str) -> str:
# render-generate 路径（见 uses_render_generate()），要求 vLLM 开启 --enable-scale-out。
VERIFIED_MODELS = {
"sglang": ("kimi-k3", "qwen3.6-35b-a3b", "glm-5.3"),
"vllm": ("qwen3.6-35b-a3b", "glm-5.3", "deepseek-v4-flash-0731-nvfp4", "kimi-k3"),
"vllm": ("qwen3.6-35b-a3b", "qwen3.6-27b", "qwen3-30b-a3b", "qwen3-14b", "qwen3-8b",
"gemma-4-31b-it", "gemma-4-26b-a4b-it", "glm-5.3", "deepseek-v4-flash-0731-nvfp4",
"kimi-k3"),
}

# 关思考开关在 chat_template_kwargs 里的 key：默认 `thinking`，Qwen3 系的模板读的是
Expand All @@ -81,11 +83,15 @@ def _build_prefix_think_tag(reasoning: str, content: str) -> str:
_DEFAULT_THINKING_SWITCH_KEY = "thinking"
_THINKING_SWITCH_KEYS = {
"qwen3.6-35b-a3b": "enable_thinking",
"qwen3.6-27b": "enable_thinking",
"qwen3-30b-a3b": "enable_thinking",
"qwen3-14b": "enable_thinking",
"qwen3-8b": "enable_thinking",
}

# 引擎 -> 该引擎上 content-done 需要发结构化消息的模型型号。
# 引擎 -> 该引擎上 content-done 需要发结构化消息的模型型号。只登记测试出有问题确实需要的
_STRUCTURED_CONTENT_DONE = {
"vllm": ("qwen3.6-35b-a3b",),
"vllm": ("qwen3.6-35b-a3b", "qwen3.6-27b", "qwen3-8b"),
}


Expand Down
Loading