feat: add embedded-python model executor for qwen3 on cuda. by zhang-minchao · Pull Request #1891 · xLLM-AI/xllm

zhang-minchao · 2026-07-06T08:12:25Z

Description

Add an embedded-CPython model executor that runs Qwen3 inference on CUDA through the xLLM runtime, as an opt-in alternative to the native C++ model path. The C++ engine drives an in-process Python interpreter to execute the model forward pass, while all compute stays on the existing xllm_ops fused CUDA kernels, so the Python path is numerically validated against the native path (byte-identical greedy decode at both tp=1 and tp=2).

Enable it with --model_impl=python --python_model_path=<path-to-xllm>; the default native path is unchanged.

Main pieces:

C++ ↔ Python bridge — PyCausalLM + py_model_helper (pybind) let the C++ executor construct and call a Python model like any native CausalLM, sharing the KV cache, forward inputs, and attention metadata.
Python model package (xllm/python/) — Qwen3 model, layers (linear / layernorm / embedding / rotary), op dispatch, triton kernels, and a cudagraph-capable model_runner, all built on xllm_ops.
PROPERTY struct reflection (property_reflect.h + macros.h) — dependency-free field reflection behind an opt-in REFLECT_PROPERTIES, so the full ModelArgs / ParallelArgs cross the pybind boundary without hand-maintained field lists.
xllm_ops op registration — ops layer structured as compute/collectives modules; TP collectives registered as torch.library.custom_op for Dynamo capture; attention uses flashinfer Python API directly.
Graph-mode decode — decode uses manual full CUDA graph (bucketed by batch), with FlashInfer host planner metadata and device runtime metadata using the same empty-sequence representation for padded sequences; prefill runs eager for minimal TTFT.

服务启动

# Python 模型执行器 (eager)
./xllm --model /path/to/Qwen3-0.6B \
  --model_impl=python \
  --python_model_path=/path/to/xllm/xllm \
  --device_id=0 \
  --enable_graph=false

# Python 模型执行器 (decode full CUDA graph)
./xllm --model /path/to/Qwen3-0.6B \
  --model_impl=python \
  --python_model_path=/path/to/xllm/xllm \
  --device_id=0 \
  --enable_graph=false \
  --python_graph_backend=cudagraphs

Performance

Full regression on Qwen3-0.6B using two dedicated H100 GPUs (cuda:6 and
cuda:7), input length 1024, output length 256, and
CUDA_LAUNCH_BLOCKING=0. The run completed build, JSON-only Python cudagraph
startup, byte-identical C++/Python greedy parity at TP1 and TP2, and the same
cross-batch sweep for all four variants. The batch list deliberately mixes
bucket boundaries and odd values to exercise decode-graph padding.

Report generated on 2026-07-14:
log/regression/batch_sweep_20260714_175345.md.

TP=1 complete comparison (Median TTFT / TPOT, ms)

batch	bucket	pad?	cpp_eager TTFT	cpp_eager TPOT	cpp_graph TTFT	cpp_graph TPOT	py_eager TTFT	py_eager TPOT	py_graph TTFT	py_graph TPOT	py_graph vs cpp_graph TTFT	py_graph vs cpp_graph TPOT
1	1		7.302	2.902	6.205	1.955	8.566	5.249	7.499	1.928	+20.9%	-1.4%
2	2		8.721	2.954	8.360	1.892	9.573	5.314	8.531	1.882	+2.0%	-0.5%
3	4	y	11.770	2.958	11.637	2.125	11.734	5.386	12.135	2.094	+4.3%	-1.5%
5	8	y	14.027	3.013	16.983	2.029	13.989	5.458	15.157	2.042	-10.8%	+0.6%
7	8	y	21.141	3.079	17.556	2.163	14.936	5.515	14.670	2.207	-16.4%	+2.0%
8	8		24.314	3.165	25.143	2.155	25.114	5.451	25.197	2.146	+0.2%	-0.4%
13	16	y	34.719	3.360	23.631	2.538	34.988	5.762	30.559	2.489	+29.3%	-1.9%
16	16		28.872	3.297	31.570	2.810	33.539	5.755	31.001	2.565	-1.8%	-8.7%
31	32	y	45.784	3.709	45.402	3.482	44.744	6.060	45.499	3.525	+0.2%	+1.2%
47	48	y	70.691	4.528	68.503	4.214	76.332	6.443	63.107	4.283	-7.9%	+1.6%
100	112	y	124.055	7.470	124.159	7.224	127.821	7.570	131.613	7.246	+6.0%	+0.3%

TP=2 complete comparison (Median TTFT / TPOT, ms)

batch	bucket	pad?	cpp_eager TTFT	cpp_eager TPOT	cpp_graph TTFT	cpp_graph TPOT	py_eager TTFT	py_eager TPOT	py_graph TTFT	py_graph TPOT	py_graph vs cpp_graph TTFT	py_graph vs cpp_graph TPOT
1	1		8.953	4.713	9.301	4.518	15.072	11.429	15.204	4.536	+63.5%	+0.4%
2	2		10.810	4.775	11.266	4.531	14.350	11.417	15.313	4.467	+35.9%	-1.4%
3	4	y	12.701	4.802	13.210	4.306	15.089	11.421	15.709	4.432	+18.9%	+2.9%
5	8	y	20.564	4.861	15.261	4.404	15.801	11.471	16.405	4.631	+7.5%	+5.2%
7	8	y	18.141	4.897	15.632	4.485	18.196	11.454	22.172	4.724	+41.8%	+5.3%
8	8		20.633	4.935	26.476	4.437	29.357	11.485	29.045	4.737	+9.7%	+6.8%
13	16	y	28.271	5.086	37.933	5.067	35.116	11.334	49.283	5.170	+29.9%	+2.0%
16	16		32.237	5.063	34.987	5.074	34.056	11.691	35.159	5.256	+0.5%	+3.6%
31	32	y	48.867	5.464	60.212	5.564	51.107	12.052	68.112	5.889	+13.1%	+5.8%
47	48	y	63.373	5.813	75.758	6.308	70.360	12.514	71.868	6.604	-5.1%	+4.7%
100	112	y	128.817	7.661	132.780	9.150	137.056	13.685	140.100	9.421	+5.5%	+3.0%

Takeaways:

Python greedy parity is byte-identical to C++ at both TP1 and TP2.
py_graph TPOT tracks cpp_graph TPOT closely: TP1 ranges from -8.7% to
+2.0%, and TP2 from -1.4% to +6.8% across all tested batches.
py_graph substantially reduces Python decode overhead: its TPOT is much
lower than py_eager at both TP degrees because decode replays a captured CUDA
graph instead of dispatching the model from CPython every step.
Prefill remains eager: TTFT differences are workload and scheduling
sensitive and are not used as the decode-performance acceptance criterion.
Padding rows (pad?=y) validate empty-sequence padding: host planner and
device runtime metadata use repeated terminal indptr values, zero KV-length
deltas, and no dummy page allocation. All padded buckets passed at TP1 and
TP2.

Related Issues

Change Type

gemini-code-assist

Code Review

This pull request introduces a Python model executor framework that allows executing Python-defined model graphs, such as Qwen3, within xLLM's C++ worker using an embedded CPython interpreter. It adds the C++ bridge, a lightweight config reflection system, and the Python-side model, layers, op dispatching, and graph runner. Feedback on the changes focuses on strict adherence to the repository's style guide, including using project-root-relative paths for #include directives, preferring fixed-width integers and emplace_back in C++, using CHECK instead of TORCH_CHECK, ensuring complete Python type annotations, and routing Python diagnostic logs through the shared logger instead of print().

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

- cpp_framework_python_model_architecture.md: document the architecture decision to add python model execution on the c++ serving framework and why python owns both the model and its ModelExecutor. - refactor_python_model_impl.md: describe the implementable refactor plan covering cross-language interface, object lifecycle, migration order and acceptance conditions, baselined on PR xLLM-AI#1891.

The variable holds attention metadata, not generic metadata. Rename for clarity and update the design doc to match.

Add model_impl field to ModelContext so model_registry.cpp no longer reads the global FLAGS_model_impl directly. The caller (llm_worker_impl) sets it before calling create_llm_model.

- initialize PyTorch NCCL TP collectives and honor JSON executor settings. - prevent KV padding overflow and remove obsolete graph batch configuration. - add config, dump, and embedded Python regression coverage.

Update the Python executor design document to match the current CUDA, TP, JSON, and decode graph implementation.

Install FlashInfer 0.6.14 after the AOT build and make the embedded Python runtime share the xLLM TVM-FFI libraries.

- cpp_framework_python_model_architecture.md: document the architecture decision to add python model execution on the c++ serving framework and why python owns both the model and its ModelExecutor. - refactor_python_model_impl.md: describe the implementable refactor plan covering cross-language interface, object lifecycle, migration order and acceptance conditions, baselined on PR xLLM-AI#1891.

…d of hardware probe. - batch_prefill: determine FA2/FA3 call schema from the URI suffix (_sm90) instead of re-evaluating hardware and mask at runtime. - flashinfer_planinfo: select short (16-param) vs long (19-param) plan args from the explicit backend string, removing the platform.h dependency.

…h runner. - Remove max_seqs_per_batch from config dict; pass it as an explicit constructor argument through PyExecutorImpl → ModelExecutor → runner. - DecodeCudaGraphRunner now takes max_model_len and pre-allocates a single shared paged_kv_indices buffer sized for the worst case, reused across all bucket entries.

…da graph. - Add DecodeGraphSharesMaximumBlockTableBufferAcrossBuckets to verify the single shared paged_kv_indices buffer is reused across buckets. - Add ModelExecutorUsesExplicitRuntimeBatchLimit to confirm the runtime batch limit flows through to the decode runner. - Update DecodeMetadataFastPath* tests for max_seqs_per_batch=4 and assert the full padded shape including kv_cu_seq_lens and qo_indptr. - Add MakeFa2PrefillParams helper to pin FA2 for piecewise prefill eager references, isolating graph capture correctness from FA2/FA3 numerical differences.

Clarify at both call sites (batch_prefill.cpp ragged_run and flashinfer_planinfo.cpp prefill plan) why the FlashInfer ABI is chosen from the loaded module/backend (fa2 19-arg vs fa3 16-arg) rather than the SM90 hardware probe, which silently broke fa2-forced-on-SM90 graph piecewise prefill. Comment only; no logic change.

…model executor. - flashinfer: map sliding window W to window_left W-1 to match the native attention convention, so SWA models attend the correct span. - linear: add optional bias to Column/RowParallelLinear (column bias sharded on the output dim; row bias replicated and added once after all-reduce) and load qkv/o_proj bias when attention_bias is set; standard Qwen3 (attention_bias=False) is unchanged. - qwen3: cast position ids to int64 once in Qwen3Model.forward instead of once per layer inside fused_qk_norm_rope; the single cast stays inside the captured decode graph so replay re-casts static_positions correctly.

- importing the executor package as top-level 'python' required prepending the xllm/ source dir to sys.path, which exposed sibling dirs (proto/, models/, core/, ...) as importable top-level namespace packages and let 'import python' shadow globally inside the embedded interpreter. - the wheel already stages the package inside the installed 'xllm' package dir, so 'import xllm.python' resolves straight from site-packages with no sys.path manipulation; xllm/__init__.py stays lazy (stdlib-only at import). - --python_model_path / XLLM_PYTHON_MODEL_PATH now mean the directory containing the 'xllm' package (repo root for source-tree runs); launch_server.py defaults the env var to site-packages accordingly.

- anchor all cross-module imports at xllm.python now that the package identity is final; no behavior change. - addresses review feedback asking to avoid relative import paths in xllm/python, matching the absolute-import convention used across the package ecosystem (vLLM, SGLang).

zhang-minchao requested review from Clement-Wang26, DongheJin, DragonFive, JimHsiung, Kang-Meng, RobbieLeung, XuZhang99, liujinguang0125, liutongxuan, walsonyang, xiao-yu-chen, yingxudeng and yq33victor as code owners July 6, 2026 08:12

gemini-code-assist Bot reviewed Jul 6, 2026

View reviewed changes

zhang-minchao force-pushed the feat/python-model-executor branch from 7430c11 to 479ab8a Compare July 6, 2026 08:16

zhang-minchao changed the title ~~Feat/python model executor~~ feat: add embedded-python model executor for qwen3 on cuda. Jul 6, 2026

zhang-minchao force-pushed the feat/python-model-executor branch 2 times, most recently from c962bc8 to eb22c0b Compare July 6, 2026 11:25

zhang-minchao marked this pull request as draft July 6, 2026 12:22

zhang-minchao marked this pull request as ready for review July 7, 2026 19:18

zhang-minchao force-pushed the feat/python-model-executor branch from be4d1a1 to 357910e Compare July 7, 2026 19:19

XuZhang99 reviewed Jul 8, 2026

View reviewed changes

Comment thread xllm/core/framework/config/model_config.h

Comment thread xllm/core/layers/cuda/flashinfer_planinfo.cpp Outdated

Comment thread xllm/models/model_registry.cpp Outdated

XuZhang99 mentioned this pull request Jul 10, 2026

[Question]: models/llm/qwen3_5.h 必须编译 NPU 头文件，而不是 common 的 #1916

Open

XuZhang99 reviewed Jul 10, 2026

View reviewed changes

Comment thread xllm/core/kernels/cuda/cuda_ops_library.cpp

zhang-minchao force-pushed the feat/python-model-executor branch from 99a0f54 to 6f8e5e6 Compare July 10, 2026 07:23

XuZhang99 reviewed Jul 10, 2026

View reviewed changes

Comment thread docker/Dockerfile.cuda

XuZhang99 reviewed Jul 10, 2026

View reviewed changes

Comment thread docker/Dockerfile.cuda

zhang-minchao force-pushed the feat/python-model-executor branch 2 times, most recently from 06baa0a to 584611d Compare July 13, 2026 12:58

zhang-minchao added 25 commits July 15, 2026 23:30

refactor: rename meta -> attn_meta in PyCausalLM::forward

99d2d97

The variable holds attention metadata, not generic metadata. Rename for clarity and update the design doc to match.

refactor: use full name attention_metadata_dict, no abbreviation

bb82e14

refactor: pass model_impl via ModelContext instead of FLAGS

a8d1648

Add model_impl field to ModelContext so model_registry.cpp no longer reads the global FLAGS_model_impl directly. The caller (llm_worker_impl) sets it before calling create_llm_model.

bugfix: fix python executor tp and graph configuration.

2758095

- initialize PyTorch NCCL TP collectives and honor JSON executor settings. - prevent KV padding overflow and remove obsolete graph batch configuration. - add config, dump, and embedded Python regression coverage.

refactor: rename cuda torch ops library files.

e9ec3a8

Update the Python executor design document to match the current CUDA, TP, JSON, and decode graph implementation.

build: install flashinfer python runtime.

5335d52

Install FlashInfer 0.6.14 after the AOT build and make the embedded Python runtime share the xLLM TVM-FFI libraries.

docs: remove python executor design document.

84329fd

refactor: add dedicated python model executor.

1b15d80

test: cover decode graph metadata updates.

54d4779

docs: simplify python executor refactor design.

fa3318e

refactor: keep python executor changes scoped to python graph path

97aa83b

bugfix: release python config while holding gil.

8cb8101

refactor: narrow cuda graph metadata scope

7849cbe

refactor: use empty sequences for graph padding

9489e69

docs: explain empty sequence graph padding

f228654

docs: add python model architecture translation

2cc79ca

docs: document preferred python import paths.

b8bef42

zhang-minchao force-pushed the feat/python-model-executor branch from 45d57d0 to b8bef42 Compare July 15, 2026 15:47

perf: eliminate redundant tensor parallel all-reduce copies.

e15c726

JimHsiung approved these changes Jul 16, 2026

View reviewed changes

XuZhang99 approved these changes Jul 16, 2026

View reviewed changes

zhang-minchao merged commit b016a52 into xLLM-AI:main Jul 16, 2026
13 of 48 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add embedded-python model executor for qwen3 on cuda.#1891

feat: add embedded-python model executor for qwen3 on cuda.#1891
zhang-minchao merged 52 commits into
xLLM-AI:mainfrom
zhang-minchao:feat/python-model-executor

zhang-minchao commented Jul 6, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zhang-minchao commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

服务启动

Performance

TP=1 complete comparison (Median TTFT / TPOT, ms)

TP=2 complete comparison (Median TTFT / TPOT, ms)

Related Issues

Change Type

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zhang-minchao commented Jul 6, 2026 •

edited

Loading