中文 · Issues · Serving recipes · Documentation
New model support images 🔥
vLLM
- Qwen3.8-Flash-Next —
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:qwen38-flash-next - GLM-5.3-Flash —
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:glm53-flash - Hy4-preview —
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:hy4-preview - DiffusionGemma-26B-A4B-it —
registry.mthreads.com/mcconline/inference/vllm/vllm-openai:jev
vLLM-Omni
- FLUX.3-Image-Klein —
registry.mthreads.com/mcconline/inference/vllm/vllm-omni:flux3 - MAGI-2 Preview —
registry.mthreads.com/mcconline/inference/vllm/vllm-omni:magi-2-preview
Read the MAGI-2 Preview on vLLM-Omni blog post for the S5000 measurements, the individual PR gains, and the reproduction steps.
These model-specific tags are maintained independently of the release line; we do not recommend production use without verifying the exact image and model revision.
vLLM-MUSA is the Moore Threads backend for vLLM, providing an OpenAI-compatible inference and serving engine for MUSA GPUs.
- v0.32.0-dev development line with the vLLM V1 engine.
- Uses the pinned MUSA stack: PyTorch/torch_musa 2.11.0.post2 (MUSA 5.2.0), MATE 0.2.8.post1, and torchada 0.1.105; this branch remains an upgrade candidate.
- MUSA-native attention, communication, custom ops, and compilation support.
- Per-checkpoint recipes for Qwen, DeepSeek-V4-Flash, Nemotron-3.5, and DiffusionGemma on S5000.
Use the published v0.32.0 release image:
export VLLM_MUSA_IMAGE=registry.mthreads.com/mcconline/inference/vllm/vllm-openai:v0.32.0
docker pull "${VLLM_MUSA_IMAGE}"The image is built independently of this branch. Verify its installed package
versions against third_party/PINS; build the local image from the
installation guide when exact branch parity is needed.
vllm serve /path/to/model \
--trust-remote-code \
--served-model-name my-modelThe command above assumes that vllm is installed and on the current
environment's PATH. For the release image, use the container launch shape
in the Docker guide.
The OpenAI-compatible endpoint is available at http://localhost:8000/v1.
For recommended tensor-parallel, scheduler, and speculative-decoding settings,
use the serving cookbook.
- Serving cookbook
- Installation
- Python and OpenAI-compatible usage
- Configuration
- Development
- MUSA developer guide
| vLLM-MUSA | PyTorch/MUSA | Engine | Status |
|---|---|---|---|
| v0.32.0-dev | 2.11.x | V1 engine; Runner V2 where supported | Upgrade candidate |
The V1 engine may automatically select Model Runner V2 for supported model
architectures; both runners are supported on MUSA.
This branch tracks the upstream v0.32.0 commit in third_party/PINS and
intentionally keeps the MUSA PyTorch 2.11.x stack; promote it to a supported
release only after build and model-smoke validation.
See the development guide for editable installs, tests, patch workflow, and issue reports. Contributions and bug reports are welcome through GitHub Issues.
vLLM-MUSA is released under the Apache License 2.0.