From 5cbb74b6b9c105c5a7c69341fbe7e18226b8140f Mon Sep 17 00:00:00 2001 From: Lu Xinlong Date: Wed, 4 Nov 2026 18:28:35 +0800 Subject: [PATCH] Add test result in Ascend910B, change some strategies for verified models --- README.md | 26 ++++++++++++++++++++------ README.zh-CN.md | 26 ++++++++++++++++++++------ continuation_gateway/reconstruct.py | 12 +++++++++--- 3 files changed, 49 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 43b1d34..f5d7542 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,9 @@ English | [简体中文](README.zh-CN.md) > `CONTINUATION_ENGINE`. Within this project, each engine + model combination has its own > continuation strategy. The combinations verified so far: > -> **SGLang(tested on v0.5.19+)** +> Status: ✅ verified, ❌ not supported. +> +> **SGLang (NVIDIA GPU, tested on v0.5.19+)** > > | Model | Status | Notes | > | --- | --- | --- | @@ -15,7 +17,7 @@ English | [简体中文](README.zh-CN.md) > | Qwen3.6-35B-A3B | ✅ Verified | | > | GLM-5.3 | ✅ Verified | | > -> **vLLM(tested on v0.30.0+)** +> **vLLM (NVIDIA GPU, tested on v0.30.0+)** > > | Model | Status | Notes | > | --- | --- | --- | @@ -24,10 +26,23 @@ English | [简体中文](README.zh-CN.md) > | DeepSeek-V4-Flash-0731-nvfp4 | ✅ Verified | | > | Kimi-K3 | ✅ Verified | vLLM must be started with `--enable-scale-out`, otherwise continuation does not work. Multimodal requests are not continued | > +> **vLLM (Ascend 910B4, tested on vllm-ascend 0.30.0+)** +> +> | Model | Status | Notes | +> | --- | --- | --- | +> | Qwen3.6-35B-A3B | ✅ Verified | Tested with the W8A8 weights | +> | Qwen3.6-27B | ✅ Verified | Tested with the W8A8 weights | +> | Qwen3-30B-A3B | ✅ Verified | Tested with the W8A8 weights | +> | Qwen3-14B | ✅ Verified | | +> | Qwen3-8B | ✅ Verified | | +> | Gemma-4-31B-it | ✅ Verified | The model itself is occasionally unstable: it may emit its control tokens as plain text in `content` (for example ``, `<\|tool_call>`) so that the tool_call is not parsed and is lost. This comes from the model and its parser, not from how the continuation prefix is built | +> | Gemma-4-26B-A4B-it | ✅ Verified | Same as Gemma-4-31B-it. Its reasoning may also occasionally end up in `content` | +> | Qwen3-Omni-30B-A3B-Instruct | ❌ Not supported | | +> > The model names in the tables are exactly the strings `CONTINUATION_MODEL` should be set to > (case-insensitive, exact match). A quantized or fine-tuned variant of a listed model, such as > `GLM-5.3-FP8`, should use the listed name `GLM-5.3`; a different size or version -> (`Qwen3.6-27B`, `GLM-5.4`) is a different model and is not matched. +> (`Qwen3-32B`, `GLM-5.4`) is a different model and is not matched. > > Other models and engines have not been verified; they may work, but please verify them > yourself before relying on them. We are continuing to adapt more engine + model @@ -51,9 +66,8 @@ deployment actually serves; the `model` field of the request is not consulted), `continuation_gateway/reconstruct.py`. Two formats are built in: Kimi-K3 (XTML) and a generic `...` format. Supporting a model with another format only takes adding a builder there. A model that matches no verified model is handled with the generic format, and -the gateway logs a warning at startup. On vLLM, Qwen3.6-35B-A3B additionally sends the -content-done partial message as separate `reasoning` / `content` fields, because its chat -template rewrites the string form and the engine then rejects it with a 400. +the gateway logs a warning at startup. On vLLM, some models (such as Qwen3.6-35B-A3B) additionally split +the content-done partial message into separate `reasoning` / `content` fields. ``` leg1: original request diff --git a/README.zh-CN.md b/README.zh-CN.md index 6177615..0823bf6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -6,7 +6,9 @@ > 网关支持 **SGLang** 和 **vLLM** 两种推理引擎，通过 `CONTINUATION_ENGINE` 指定。本项目针对 > 不同的「引擎 + 模型」组合各有对应的续写策略，目前已验证的组合如下： > -> **SGLang(在v0.5.19+版本上验证)** +> 状态：✅ 已验证，❌ 不支持。 +> +> **SGLang（NVIDIA GPU，在 v0.5.19+ 版本上验证）** > > | 模型 | 状态 | 备注 | > | --- | --- | --- | @@ -14,7 +16,7 @@ > | Qwen3.6-35B-A3B | ✅ 已验证 | | > | GLM-5.3 | ✅ 已验证 | | > -> **vLLM(在v0.30.0+版本上验证)** +> **vLLM（NVIDIA GPU，在 v0.30.0+ 版本上验证）** > > | 模型 | 状态 | 备注 | > | --- | --- | --- | @@ -23,9 +25,22 @@ > | DeepSeek-V4-Flash-0731-nvfp4 | ✅ 已验证 | | > | Kimi-K3 | ✅ 已验证 | vLLM 必须开启 `--enable-scale-out`，否则续写无效；不支持多模态请求的续写 | > +> **vLLM（昇腾 910B4，在 vllm-ascend 0.30.0+ 版本上验证）** +> +> | 模型 | 状态 | 备注 | +> | --- | --- | --- | +> | Qwen3.6-35B-A3B | ✅ 已验证 | 使用 W8A8 权重测试 | +> | Qwen3.6-27B | ✅ 已验证 | 使用 W8A8 权重测试 | +> | Qwen3-30B-A3B | ✅ 已验证 | 使用 W8A8 权重测试 | +> | Qwen3-14B | ✅ 已验证 | | +> | Qwen3-8B | ✅ 已验证 | | +> | Gemma-4-31B-it | ✅ 已验证 | 模型自身偶发不稳定：可能把控制 token 当作普通文本输出到 `content`（例如 ``、`<\|tool_call>`），导致 tool_call 没有被解析出来而丢失。这来自模型及其解析器，不是续写前缀的构造问题 | +> | Gemma-4-26B-A4B-it | ✅ 已验证 | 同 Gemma-4-31B-it；另外它的思考内容偶发会落进 `content` | +> | Qwen3-Omni-30B-A3B-Instruct | ❌ 不支持 | | +> > 表中的模型名就是 `CONTINUATION_MODEL` 应设置的字符串（大小写不敏感，精确匹配）。表中模型 > 的量化或微调版本，例如 `GLM-5.3-FP8`，应直接设置为表里的名字 `GLM-5.3`；不同的尺寸或版本 -> （`Qwen3.6-27B`、`GLM-5.4`）是另一个模型，不会被匹配。 +> （`Qwen3-32B`、`GLM-5.4`）是另一个模型，不会被匹配。 > > 其它模型和引擎尚未验证，可能可用，使用前请自行验证。我们仍在持续适配更多的引擎 + 模型组合。 @@ -40,9 +55,8 @@ LLM 推理服务的崩溃续写网关：位于客户端和 OpenAI 兼容的推 前缀拼接方式由 `CONTINUATION_MODEL`（本部署实际 serve 的真实模型，不看请求里的 `model` 字段）决定（见 `continuation_gateway/reconstruct.py`），内置 Kimi-K3（XTML）和通用 `...` 两种格式，接入其它格式的模型只需要在那里加一个 builder。没有匹配到已验证 -模型的，按通用格式处理，并在启动时打一条 warning。vLLM 上的 Qwen3.6-35B-A3B 在 content-done -时额外把 partial 消息拆成 `reasoning` / `content` 两个字段发送，因为它的 chat template 会重排 -字符串形式，引擎随后以 400 拒绝。 +模型的，按通用格式处理，并在启动时打一条 warning。vLLM 上部分模型（如 Qwen3.6-35B-A3B）在 +content-done 时会额外把 partial 消息拆成 `reasoning` / `content` 两个字段发送。 ``` leg1：原始请求 diff --git a/continuation_gateway/reconstruct.py b/continuation_gateway/reconstruct.py index 84d5efb..f4bcf1c 100644 --- a/continuation_gateway/reconstruct.py +++ b/continuation_gateway/reconstruct.py @@ -71,7 +71,9 @@ def _build_prefix_think_tag(reasoning: str, content: str) -> str: # render-generate 路径（见 uses_render_generate()），要求 vLLM 开启 --enable-scale-out。 VERIFIED_MODELS = { "sglang": ("kimi-k3", "qwen3.6-35b-a3b", "glm-5.3"), - "vllm": ("qwen3.6-35b-a3b", "glm-5.3", "deepseek-v4-flash-0731-nvfp4", "kimi-k3"), + "vllm": ("qwen3.6-35b-a3b", "qwen3.6-27b", "qwen3-30b-a3b", "qwen3-14b", "qwen3-8b", + "gemma-4-31b-it", "gemma-4-26b-a4b-it", "glm-5.3", "deepseek-v4-flash-0731-nvfp4", + "kimi-k3"), } # 关思考开关在 chat_template_kwargs 里的 key：默认 `thinking`，Qwen3 系的模板读的是 @@ -81,11 +83,15 @@ def _build_prefix_think_tag(reasoning: str, content: str) -> str: _DEFAULT_THINKING_SWITCH_KEY = "thinking" _THINKING_SWITCH_KEYS = { "qwen3.6-35b-a3b": "enable_thinking", + "qwen3.6-27b": "enable_thinking", + "qwen3-30b-a3b": "enable_thinking", + "qwen3-14b": "enable_thinking", + "qwen3-8b": "enable_thinking", } -# 引擎 -> 该引擎上 content-done 需要发结构化消息的模型型号。 +# 引擎 -> 该引擎上 content-done 需要发结构化消息的模型型号。只登记测试出有问题确实需要的 _STRUCTURED_CONTENT_DONE = { - "vllm": ("qwen3.6-35b-a3b",), + "vllm": ("qwen3.6-35b-a3b", "qwen3.6-27b", "qwen3-8b"), }