diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md
index 9f9657d3de..e522a25e2f 100644
--- a/.github/CONTRIBUTING.md
+++ b/.github/CONTRIBUTING.md
@@ -33,7 +33,7 @@ For developers who want to contribute to our code, here is the guidance:
+ A function in plan
## 2. Install environment for development
-+ We strongly suggest you to read our **[Document](http://xxx/docs/)** before developing
++ We strongly suggest you to read our **[Document](../docs/src/content/docs/en/index.md)** before developing
+ For setting environment, please check our **[Readme file](/README.md)**
## 3. Build our project
diff --git a/.github/CONTRIBUTING_zh.md b/.github/CONTRIBUTING_zh.md
index 2e48f869dc..7c73dd1094 100644
--- a/.github/CONTRIBUTING_zh.md
+++ b/.github/CONTRIBUTING_zh.md
@@ -36,7 +36,7 @@ xLLM致力于为每一位用户和开发者提供开放的XX，因此无论您
+ 计划实现的功能
## 2. 配置开发环境
-+ 在开发之前，可以参考我们的 **[文档](http://xxx/docs/)**
++ 在开发之前，可以参考我们的 **[文档](../docs/src/content/docs/zh/index.md)**
+ 关于环境配置，参见 **[Readme file](/README.md)**
## 3. 项目构建和运行
@@ -45,4 +45,4 @@ xLLM致力于为每一位用户和开发者提供开放的XX，因此无论您
## 4. 测试
在pr提交之后，我们会对代码进行格式化及进一步测试。
-我们的测试目前还很不完善，因此欢迎开发者为测试作出贡献！
\ No newline at end of file
+我们的测试目前还很不完善，因此欢迎开发者为测试作出贡献！
diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml
new file mode 100644
index 0000000000..1156881003
--- /dev/null
+++ b/.github/workflows/deploy_docs.yml
@@ -0,0 +1,65 @@
+name: Deploy Docs
+
+on:
+ push:
+ branches: [main]
+ paths:
+ - 'docs/**'
+ - '.github/workflows/deploy_docs.yml'
+ pull_request:
+ branches: [main]
+ paths:
+ - 'docs/**'
+ - '.github/workflows/deploy_docs.yml'
+ workflow_dispatch:
+
+permissions:
+ contents: read
+ pages: write
+ id-token: write
+
+concurrency:
+ group: pages
+ cancel-in-progress: false
+
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout
+ uses: actions/checkout@v4
+
+ - name: Setup Node.js
+ uses: actions/setup-node@v4
+ with:
+ node-version: 22
+ cache: npm
+ cache-dependency-path: docs/package-lock.json
+
+ - name: Install dependencies
+ working-directory: docs
+ run: npm ci
+
+ - name: Build
+ working-directory: docs
+ env:
+ BASE_PATH: /xllm
+ SITE_URL: https://jd-opensource.github.io/xllm
+ run: npm run build
+
+ - name: Upload Pages artifact
+ uses: actions/upload-pages-artifact@v3
+ with:
+ path: docs/dist
+
+ deploy:
+ if: github.event_name != 'pull_request'
+ needs: build
+ runs-on: ubuntu-latest
+ environment:
+ name: github-pages
+ url: ${{ steps.deployment.outputs.page_url }}
+ steps:
+ - name: Deploy to GitHub Pages
+ id: deployment
+ uses: actions/deploy-pages@v4
diff --git a/README.md b/README.md
index 915223714a..aedb191d8e 100755
--- a/README.md
+++ b/README.md
@@ -12,19 +12,19 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. -->
-[English](./README.md) | [中文](./docs/project/README_zh.md)
+[English](./README.md) | [中文](./README.zh-CN.md)
-
+
-[](https://xllm.readthedocs.io/zh-cn/latest/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
+[](https://jd-opensource.github.io/xllm/en/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
-| Documentation Technical Report Documentation Technical Report
@@ -43,7 +43,7 @@ limitations under the License. -->
**xLLM** is an **efficient LLM inference framework**, specifically optimized for **Chinese AI accelerators**, enabling enterprise-grade deployment with enhanced efficiency and reduced cost. The framework adopts a **service-engine decoupled** inference architecture, achieving breakthrough efficiency through several technologies: at the service layer, including elastic scheduling of online/offline requests, dynamic PD disaggregation, a hybrid EPD mechanism for multimodal and high-availability fault tolerance; and at the engine layer, combined with technologies such as multi-stream parallel computing, graph fusion optimization, speculative inference, dynamic load balancing and global KV cache management. The overall architecture is shown below:
-
+
-
+
-
+
-[](https://xllm.readthedocs.io/zh-cn/latest/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
+[](https://jd-opensource.github.io/xllm/zh/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
-| Documentation Technical Report Documentation Technical Report
### 📢 新闻
@@ -41,7 +41,7 @@ limitations under the License. -->
**xLLM** 是一个高效的开源大模型推理框架，专为**国产芯片**优化设计，提供企业级的服务部署，使得性能更高、成本更低。该框架采用**服务-引擎分离的推理架构**，通过服务层的在离线请求弹性调度、动态PD分离、EPD混合机制及高可用容错设计，结合引擎层的多流并行计算、图融合优化、投机推理、动态负载均衡及全局KV缓存管理，实现推理效率突破性提升。xLLM整体架构和功能如下图所示：
-
+
-
+
diff --git a/docs/README.zh-CN.md b/docs/README.zh-CN.md
new file mode 100644
index 0000000000..e9d442f21a
--- /dev/null
+++ b/docs/README.zh-CN.md
@@ -0,0 +1,91 @@
+# xLLM 文档站
+
+[English](./README.md) | [简体中文](./README.zh-CN.md)
+
+本目录是 [xLLM](https://github.com/jd-opensource/xllm) 的 Astro + Starlight
+文档站。xLLM 是面向国产 AI 加速器的高性能大语言模型推理框架。
+
+站点基于 Starlight 和 `starlight-theme-rapide` 构建，包含自定义顶部导航、
+中英文切换，以及用于将文档正文复制为 Markdown 的 `Copy page` 页面操作。
+
+## 文档结构
+
+文档内容按语言维护在两个并行目录中：
+
+- 英文：`src/content/docs/en`
+- 简体中文：`src/content/docs/zh`
+
+当前站点路由规则如下。部署到 `/xllm` 等 base path 下时，下面的路由都会自动加上
+对应前缀。
+
+- `/` 重定向到 `/en/`
+- `/en/` 对应英文文档
+- `/zh/` 对应简体中文文档
+
+新增或移动页面时，请尽量在两个语言目录中保持相同的相对路径，方便 Starlight
+在同一主题的不同语言版本之间切换。
+
+## 项目结构
+
+```text
+.
+├── astro.config.mjs # Starlight、语言、侧边栏和组件配置
+├── package.json # npm 脚本和依赖
+├── src/
+│ ├── assets/ # 站点级资源，例如 logo
+│ ├── components/ # Starlight 组件覆盖
+│ ├── content/
+│ │ └── docs/
+│ │ ├── en/ # 英文文档
+│ │ ├── zh/ # 简体中文文档
+│ │ └── assets/ # 文档图片和架构图
+│ ├── pages/index.astro # 从 / 重定向到 /en/
+│ └── styles/theme.css # 项目主题样式
+└── public/ # 静态公共资源
+```
+
+## 本地开发
+
+从 xLLM 仓库根目录进入本目录：
+
+```sh
+cd docs
+```
+
+安装依赖：
+
+```sh
+npm install
+```
+
+启动本地开发服务：
+
+```sh
+npm run dev
+```
+
+构建生产站点：
+
+```sh
+npm run build
+```
+
+预览生产构建结果：
+
+```sh
+npm run preview
+```
+
+## 编辑文档
+
+- 面向用户的文档内容放在 `src/content/docs/en` 和 `src/content/docs/zh`。
+- 同一个页面同时存在中英文版本时，尽量保持两个文件的相对路径一致。
+- 共享的文档图片放在 `src/content/docs/assets`。
+- 新增需要出现在导航中的章节时，更新 `astro.config.mjs` 中的 `sidebar`
+ 配置。
+- 提交前运行 `npm run build`，用于检查路由、frontmatter 和 Starlight
+ 内容问题。
+
+## 相关仓库
+
+- xLLM 源码：
diff --git a/docs/assets/groupmatmul_performance.png b/docs/assets/groupmatmul_performance.png
deleted file mode 100644
index 65a23fe8c6..0000000000
Binary files a/docs/assets/groupmatmul_performance.png and /dev/null differ
diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs
new file mode 100644
index 0000000000..0e5c24033a
--- /dev/null
+++ b/docs/astro.config.mjs
@@ -0,0 +1,90 @@
+// @ts-check
+import { defineConfig } from 'astro/config';
+import starlight from '@astrojs/starlight';
+import starlightThemeRapide from 'starlight-theme-rapide';
+
+const base = process.env.BASE_PATH ?? '/';
+const site = process.env.SITE_URL ?? 'https://jd-opensource.github.io/xllm';
+
+// https://astro.build/config
+export default defineConfig({
+ site,
+ base,
+ integrations: [
+ starlight({
+ plugins: [starlightThemeRapide()],
+ customCss: ['./src/styles/theme.css'],
+ components: {
+ Header: './src/components/Header.astro',
+ PageTitle: './src/components/PageTitle.astro',
+ },
+ title: {
+ en: 'xLLM',
+ 'zh-CN': 'xLLM',
+ },
+ locales: {
+ en: {
+ label: 'EN',
+ lang: 'en',
+ },
+ zh: {
+ label: '中',
+ lang: 'zh-CN',
+ },
+ },
+ defaultLocale: 'en',
+ logo: {
+ src: './src/assets/logo_with_llm.png',
+ alt: 'xLLM',
+ replacesTitle: true,
+ },
+ social: [{ icon: 'github', label: 'GitHub', href: 'https://github.com/jd-opensource/xllm' }],
+ sidebar: [
+ { label: 'Home', translations: { 'zh-CN': '主页' }, slug: '' },
+ {
+ label: 'User Guide',
+ translations: { 'zh-CN': '用户指南' },
+ items: [
+ {
+ label: 'Getting Started',
+ translations: { 'zh-CN': '开始使用' },
+ autogenerate: { directory: 'getting_started' },
+ },
+ {
+ label: 'Supported Models',
+ translations: { 'zh-CN': '模型支持列表' },
+ slug: 'supported_models',
+ },
+ {
+ label: 'Popular Model Usage',
+ translations: { 'zh-CN': '热门模型使用' },
+ autogenerate: { directory: 'popular_model_usage' },
+ },
+ ],
+ },
+ {
+ label: 'Advanced Features',
+ translations: { 'zh-CN': '高级功能' },
+ autogenerate: { directory: 'features' },
+ },
+ {
+ label: 'Developer Guide',
+ translations: { 'zh-CN': '开发者指南' },
+ items: [
+ {
+ label: 'Development',
+ translations: { 'zh-CN': '开发' },
+ autogenerate: { directory: 'dev_guide' },
+ },
+ {
+ label: 'Design',
+ translations: { 'zh-CN': '设计文档' },
+ autogenerate: { directory: 'design' },
+ },
+ ],
+ },
+ { label: 'CLI Reference', translations: { 'zh-CN': 'CLI 参考' }, slug: 'cli_reference' },
+ ],
+ }),
+ ],
+});
diff --git a/docs/en/.readthedocs.yaml b/docs/en/.readthedocs.yaml
deleted file mode 100644
index ddc39e591a..0000000000
--- a/docs/en/.readthedocs.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-# Read the Docs configuration file
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-# Required
-version: 2
-
-# Set the OS, Python version, and other tools you might need
-build:
- os: ubuntu-24.04
- tools:
- python: "3.13"
- jobs:
- pre_build:
- # - cp -r docs/en/* docs/
- - mv docs/en/* docs/
- - rm -rf docs/zh/
- - find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
-
-# Build documentation with Mkdocs
-mkdocs:
- configuration: mkdocs_en.yml
-
-# Optionally, but recommended,
-# declare the Python requirements required to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
- install:
- - requirements: docs/requirements.txt
\ No newline at end of file
diff --git a/docs/en/accuracy_test.md b/docs/en/accuracy_test.md
deleted file mode 100644
index d761f2234e..0000000000
--- a/docs/en/accuracy_test.md
+++ /dev/null
@@ -1,59 +0,0 @@
-# 1. LLM Accuracy Test
-## 1.1 Setup ais_bench
-```bash
-# Create a virtual environment for ais_bench using conda or uv
-conda create --name ais_bench python=3.10 -y
-conda activate ais_bench
-
-# Clone ais_bench and install dependencies
-git clone https://gitee.com/aisbench/benchmark.git
-cd benchmark/
-pip3 install -e ./ --use-pep517
-
-# Download the dataset and copy it to the ais_bench directory
-cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
-```
-
-## 1.2 Modify Configuration
-Modify the accuracy test configuration file according to your actual situation: `/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`. It is recommended to set the sampling parameters as follows:
-```python
-models = [
- dict(
- attr="service",
- type=VLLMCustomAPIChat,
- abbr='vllm-api-general-chat',
- path="/path/to/model/Qwen3-8B", # Model path
- model="Qwen3-8B", # Model name
- request_rate = 0,
- retry = 2,
- host_ip = "127.0.0.1",
- host_port = 19000, # xllm server port
- max_out_len = 32768, # Limit maximum model length
- batch_size=32,
- trust_remote_code=False,
- generation_kwargs = dict(
- temperature = 0.6,
- # top_k = -1,
- top_p = 0.95,
- # seed = None,
- # repetition_penalty = 1,
- ),
- pred_postprocessor=dict(type=extract_non_reasoning_content)
- )
-]
-```
-
-## 1.3 Launch ais_bench
-Before using ais_bench, you need to start the xllm server first. Use `ais_bench -h` to get parameter descriptions. The launch commands for gsm8k and ceval datasets are as follows:
-```bash
-# Using gsm8k dataset
-ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
-
-# Using ceval dataset
-ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
-```
-
-We will integrate ais_bench and datasets (ceval and gsm8k) into the development image in the future. The ais_bench documentation and datasets are as follows:
-* [ais_bench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/index.html)
-* [Datasets](https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/all_params/datasets.html)
-
diff --git a/docs/en/features/basics.md b/docs/en/features/basics.md
deleted file mode 100644
index 7705216449..0000000000
--- a/docs/en/features/basics.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# Basics
-
-- xLLM uses a one-device-per-process architecture. Across multiple devices, RPC is used for function calls, and data communication during model computation uses device collective communication libraries.
-
-- HCCL/LCCL are high-performance collective communication frameworks that provide data-parallel and model-parallel collective communication for both single-node multi-device and multi-node multi-device scenarios.
diff --git a/docs/en/features/continuous_scheduler.md b/docs/en/features/continuous_scheduler.md
deleted file mode 100644
index bd96f9ab5e..0000000000
--- a/docs/en/features/continuous_scheduler.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# Continuous Scheduler
-
-## Feature Introduction
-xLLM implements a scheduling strategy that supports continuous batching. Continuous batching is a dynamic batching strategy that does not wait for a batch to be filled. Instead, it starts processing as soon as requests are available, while continuously accepting new requests and adding them to the currently executing batch. This approach significantly reduces latency while maintaining high throughput.
-
-## Usage
-xLLM implements the continuous batching scheduling strategy. And, the default scheduler is chunked prefill, `enable_chunked_prefill=true` by default.
diff --git a/docs/en/features/groupgemm.md b/docs/en/features/groupgemm.md
deleted file mode 100644
index fd8de8e466..0000000000
--- a/docs/en/features/groupgemm.md
+++ /dev/null
@@ -1,40 +0,0 @@
-# GroupGEMM Operator Optimization
-
-## Background
-The Mixture of Experts (MoE) architecture has become an important paradigm for scaling large language models. Its core idea is to dynamically route input tokens to different expert sub-networks for processing. During inference, the GroupGEMM operator is a key computational unit in the MoE architecture, responsible for efficiently executing parallel computations of multiple expert matrix multiplications, which dominate the overall inference time.
-
-## Function Introduction
-Considering the current performance bottleneck of GroupGEMM being I/O constrained, an optimization scheme is proposed that replaces data copying with index reordering, eliminating multiple copies of the token vector and instead maintaining an index table for expert allocation. Through this line number index, tokens are directly mapped to the corresponding expert computation units, and the allocation scheduling of tokens is fused with matrix multiplication into a single kernel.
-
-## User Interface
-
-### Operator Direct Call API
-```c++
-aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
- const aclTensorList *x,
- const aclTensorList *weight,
- const aclTensorList *scale,
- const aclTensorList *perTokenScale,
- const aclTensor *groupList,
- const aclTensorList *out,
- uint64_t *workspaceSize,
- aclOpExecutor **executor);
-
-aclnnStatus aclnnIndexGroupMatmul(
- void *workspace,
- uint64_t workspaceSize,
- aclOpExecutor *executor,
- aclrtStream stream);
-```
-
-- `x`: The input tensor list containing the data to be processed.
-- `weight`: The weight tensor containing the model parameters.
-- `scale`: The scaling factor used to adjust the values of the input tensors.
-- `perTokenScale`: The scaling factor for each token, used for dynamic adjustment.
-- `groupList`: The list of expert groups indicating which experts participate in the computation.
-- `out`: The output tensor list that stores the computation results.
-
-## Performance Effects
-
-
-* The optimized GroupMatmul operator shows significant advantages in computation time, especially when \( k = 128 \) and \( m = 64 \). As shown in the figure, the computation delay of the optimized operator is **reduced by 50%**.
\ No newline at end of file
diff --git a/docs/en/features/overview.md b/docs/en/features/overview.md
deleted file mode 100644
index d0a810def7..0000000000
--- a/docs/en/features/overview.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# Overall Architecture
-
-## Backgroud
-
-In recent years, with the groundbreaking progress of large language models (LLMs) ranging from tens of billions to trillions of parameters (such as GPT, Claude, DeepSeek, LLaMA, etc.) in the fields of natural language processing and multimodal interaction, the industry has an urgent need for efficient inference engines and service systems. How to reduce cluster inference costs and improve inference efficiency has become a key challenge for achieving large-scale commercial deployment.
-
-Although a number of optimization engines for large model inference have emerged, several technical bottlenecks remain in practical deployment:
-
-* **Hardware Adaptability Challenges:** Existing inference engines lack sufficient support for the architectural characteristics of specialized accelerators like domestic chips, making it difficult to fully exploit the performance potential of heterogeneous computing hardware, leading to low computational resource utilization.
-* **MoE Architecture Optimization Difficulties:** The token distribution process in expert parallelism mechanisms generates significant All-to-All communication overhead, while dynamic routing strategies cause expert load imbalance, severely constraining system scalability.
-* **Long Context Management Bottlenecks:** As model context windows continue to expand, the efficiency of optimizations in KV cache handling—such as memory fragmentation management and cross-node synchronization—directly impacts overall inference throughput performance.
-* **Hybrid Deployment Efficiency Limitations:** Existing inference clusters struggle to simultaneously guarantee service quality (SLO) and optimize resource utilization when handling both online services and offline tasks.
-* **Insufficient Dynamic PD Adaptation:** When input/output sequence lengths fluctuate drastically, static PD resource partitioning lacks the ability to adjust PD resource configurations in real-time. This can lead to idle GPU resources and poses a risk of SLO violations.
-
-To address these challenges, we present xLLM—an efficient and user-friendly open-source intelligent inference framework that provides enterprise-grade service guarantees and high-performance engine computing capabilities for model inference on domestic chips.
-
-## Feature Introduction
-
-xLLM provides intelligent computing capabilities, implementing joint inference acceleration across multiple computational system layers and algorithm-driven layers:
-
-### Computational System Layer
-
-#### Multi-Level Pipeline Execution Orchestration
-Asynchronizes CPU scheduling at the framework layer to pipeline it with chip inference computation, reducing computation bubbles; at the model graph layer, splits single batches to create pipelines between micro-batches, overlapping computation and communication; at the operator kernel layer, pipelines different computing units, overlapping computation and memory access.
-#### Dynamic Shape Graph Execution Optimization
-Addressing the static graph adaptation problem for large language models processing dynamic inputs (e.g., variable sequence lengths and batch sizes), xLLM achieves dynamic adaptation through parametric design capturing input dimensions. It combines a multi-graph caching scheme to reduce compilation overhead and uses a managed memory pool instead of absolute addresses to ensure safe reuse, ultimately achieving high execution efficiency while maintaining high flexibility.
-#### Operator Optimization
-xLLM implements specific optimizations for key operators in LLMs on domestic hardware chips, including GroupMatmul, Chunked Prefill, and others.
-#### xTensor Memory Management
-The xTensor memory management framework employs a method of *pre-allocated physical memory page pools + contiguous virtual address mapping*. It achieves efficient dynamic memory management through dynamic on-demand mapping of physical pages, reuse of reusable memory pages (Reusable), and optimized scheduling with asynchronous pre-mapping. Combined with NPU operator adaptation (such as virtual address FlashMLA), it results in improved memory utilization and reduced latency.
-
-### Algorithm-Driven Layer
-
-#### PD Separation
-xLLM fully supports PD separation scenarios, enabling efficient management of PD instances, communication between PD instances, and KV cache transfer.
-#### Global Scheduling
-xLLM provides intelligent, full-lifecycle resource scheduling management for requests and instances.
-##### Instance Scheduling
-We have implemented various instance scheduling strategies to select how to assign instances to more suitable ones. These include a simple Round Robin strategy, a prefix cache-aware strategy based on the prefix cache hit rate of requests on each instance, and a KV cache-aware strategy based on the free memory level of instances. Furthermore, for PD separation scenarios, where static PD ratios often struggle with traffic fluctuations and sudden changes in request input/output lengths, we implemented an adaptive PD dynamic scheduler responsible for global instance allocation for online requests and runtime PD dynamic adjustment.
-##### Request Scheduling
-We have implemented various request scheduling strategies supporting continuous batching, including chunked prefill, prefill priority, decode priority, and other batch strategies, all while fully supporting PD separation scenarios.
-#### Global KV Cache Management
-Utilizes ETCD as a metadata service middleware at the global level for cluster service registration, load information synchronization, and global cache state management. Each compute instance maintains a local multi-level cache pool. Regarding scheduling strategy, the system adopts a dynamic decision-making mechanism based on KV cache: it first performs prefix matching detection, calculates the KV cache reuse rate of candidate nodes, and finally selects the node with the optimal comprehensive performance for processing, achieving dynamic offloading and migration of KV cache.
-#### Speculative Inference
-xLLM incorporates an optimized speculative inference algorithm that generates multiple tokens at once to boost throughput. xLLM reduces communication costs by sinking the speculative module and optimizes speculative inference computation through methods like overlapping scheduling and computation timelines and reducing operator data movement in speculative scenarios.
-#### MoE Load Balancing
-xLLM implements expert weight updates based on historical expert load statistics for MoE models. During inference, it achieves effective dynamic load balancing through efficient expert load statistics and double-buffered, seamless expert weight updates.
-
-### Multimodal Support
-
-xLLM provides comprehensive support for various multimodal models, including Qwen2-VL and MiniCPMV.
diff --git a/docs/en/features/ppmatmul.md b/docs/en/features/ppmatmul.md
deleted file mode 100644
index 07ac17338f..0000000000
--- a/docs/en/features/ppmatmul.md
+++ /dev/null
@@ -1,36 +0,0 @@
-# PpMatmul Operator Optimization
-
-## Background
-
-In the inference of large models, matrix multiplication accounts for a high proportion and takes a long time. We have optimized the implementation of the matrix multiplication operator.
-
-## Feature Introduction
-
-The PpMatmul operator uses a Tiling strategy to decompose matrix multiplication into multiple smaller matrix multiplication tasks. However, when the number of tiles is small, tasks cannot be evenly distributed across all NPU cores, leading to the tail effect problem, which affects computational efficiency. We optimize the performance of the PpMatmul operator by prefetching memory or redistributing tasks.
-
-## User Interface
-
-### Operator Direct Call API
-
-```cpp
-aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
- const aclTensor *a,
- const aclTensor *b,
- const aclTensor *out,
- uint64_t *workspaceSize,
- aclOpExecutor **executor);
-
-aclnnStatus aclnnPpMatmulOpt(
- void *workspace,
- uint64_t workspaceSize,
- aclOpExecutor *executor,
- aclrtStream stream);
-```
-
-- `a`: Input matrix A.
-- `b`: Input matrix B.
-- `out`: Output matrix, storing the computation result.
-
-## Performance Effect
-
-For cases with a small number of tiles (e.g., when M is small, corresponding to a small batch size), there is an **18%** performance improvement of the operator compared to before optimization when (TP=4).
\ No newline at end of file
diff --git a/docs/en/features/topk_topp.md b/docs/en/features/topk_topp.md
deleted file mode 100644
index eb842f409f..0000000000
--- a/docs/en/features/topk_topp.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# Topk & Topp Operator Optimization
-
-## Background
-
-In natural language generation tasks, the topK and topP sampling strategies are widely used to control the diversity and quality of generated text. However, in small models, the computation time for these two strategies is relatively long. This is mainly due to the fewer parameters in small models, which leads to reduced efficiency in sorting and filtering when processing probability distributions, thereby affecting generation speed. Therefore, optimizing the implementation of topK and topP in small models can enhance their sampling efficiency.
-
-## Feature Introduction
-
-The implementation of the topKtopP operator merges multiple small operators, such as sorting, topK, softmax, and topP, into a single large operator, thereby improving computational efficiency and performance.
-
-## User Interface
-
-### Operator Call API
-
-```c++
-void top_k_top_p(torch::Tensor& logits,
- const torch::Tensor& topK,
- const torch::Tensor& topP);
-```
-
-- `logits`: The input logits tensor containing the model's output scores.
-- `topK`: The threshold tensor for selecting the top K probabilities.
-- `topP`: The threshold tensor for selecting the cumulative probabilities.
-
-## Performance Effect
-
-* After using the topKtopP fused operator, in the qwen2-0.5B model, TTOT **decreased by 37%**, and TTFT **increased by 10%**.
\ No newline at end of file
diff --git a/docs/en/getting_started/disagg_pd.md b/docs/en/getting_started/disagg_pd.md
deleted file mode 100644
index 564c0532ef..0000000000
--- a/docs/en/getting_started/disagg_pd.md
+++ /dev/null
@@ -1,99 +0,0 @@
-# PD disaggregation
-
-`xllm` supports PD disaggregation deployment, which requires integration with our other open-source library [xllm service](https://github.com/jd-opensource/xllm-service).
-
-## xLLM Service Dependencies
-
-First, download and install `xllm service`, similar to installing and compiling `xllm`:
-```bash
-git clone https://github.com/jd-opensource/xllm-service
-cd xllm_service
-git submodule init
-git submodule update
-```
-
-
-### etcd Installation
-
-`xllm_service` compilation and operation depend on [etcd](https://github.com/etcd-io/etcd).Use the [installation script](https://github.com/etcd-io/etcd/releases) provided by etcd for installation. The default installation path provided by the script is `/tmp/etcd-download-test/etcd`. You can either manually modify the installation path in the script or manually migrate after running the script:
-```bash
-mv /tmp/etcd-download-test/etcd /path/to/your/etcd
-```
-
-### xLLM Service Compilation
-Apply patch:
-```bash
-sh prepare.sh
-```
-Then execute the compilation:
-```bash
-mkdir -p build
-cd build
-cmake ..
-make -j 8
-cd ..
-```
-
-!!! warning "Potential Errors"
- You may encounter installation errors related to `boost-locale` and `boost-interprocess`: `vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`, `/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
- Reinstall these packages using `vcpkg`:
- ```bash
- /path/to/vcpkg remove boost-locale boost-interprocess
- /path/to/vcpkg install boost-locale:x64-linux
- /path/to/vcpkg install boost-interprocess:x64-linux
- ```
-
-## PD Disaggregation Execution
-
-Start etcd:
-```bash
-./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
-```
-
-Start xllm service:
-```bash
-ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
-```
-
-Taking Qwen2-7B as an example:
-
-- Start Prefill Instance
- ```bash
- /path/to/xllm --model=path/to/Qwen2-7B-Instruct \
- --port=8010 \
- --devices="npu:0" \
- --master_node_addr="127.0.0.1:18888" \
- --enable_prefix_cache=false \
- --enable_chunked_prefill=false \
- --enable_disagg_pd=true \
- --instance_role=PREFILL \
- --etcd_addr=127.0.0.1:12389 \
- --transfer_listen_port=26000 \
- --disagg_pd_port=7777 \
- --node_rank=0 \
- --nnodes=1
- ```
-- Start Decode Instance
- ```bash
- /path/to/xllm --model=path/to/Qwen2-7B-Instruct \
- --port=8020 \
- --devices="npu:1" \
- --master_node_addr="127.0.0.1:18898" \
- --enable_prefix_cache=false \
- --enable_chunked_prefill=false \
- --enable_disagg_pd=true \
- --instance_role=DECODE \
- --etcd_addr=127.0.0.1:12389 \
- --transfer_listen_port=26100 \
- --disagg_pd_port=7787 \
- --node_rank=0 \
- --nnodes=1
- ```
-
-Important notes:
-
-- PD disaggregation requires reading the `/etc/hccn.conf` file. Make sure this file on the physical machine is mapped into the container.
-
-- `etcd_addr` must match the `etcd_addr` of `xllm_service`
-
-The test command is similar to above. Note that the `PORT` in `curl http://localhost:{PORT}/v1/chat/completions ...` should be the `port` of the `http_server_port` of xLLM service.
\ No newline at end of file
diff --git a/docs/mkdocs/javascripts/mathjax.js b/docs/mkdocs/javascripts/mathjax.js
deleted file mode 100644
index f0d0ad52c9..0000000000
--- a/docs/mkdocs/javascripts/mathjax.js
+++ /dev/null
@@ -1,19 +0,0 @@
-window.MathJax = {
- tex: {
- inlineMath: [["\\(", "\\)"]],
- displayMath: [["\\[", "\\]"]],
- processEscapes: true,
- processEnvironments: true
- },
- options: {
- ignoreHtmlClass: ".*|",
- processHtmlClass: "arithmatex"
- }
-};
-
-document$.subscribe(() => {
- MathJax.startup.output.clearCache()
- MathJax.typesetClear()
- MathJax.texReset()
- MathJax.typesetPromise()
-})
\ No newline at end of file
diff --git a/docs/mkdocs/overrides/.icons/email-fill.svg b/docs/mkdocs/overrides/.icons/email-fill.svg
deleted file mode 100644
index f8623f2fcf..0000000000
--- a/docs/mkdocs/overrides/.icons/email-fill.svg
+++ /dev/null
@@ -1 +0,0 @@
-{title}
+
+
+
+
+ Copy page
+
+
+
[:simple-github: xLLM Service](https://github.com/jd-opensource/xllm-service)
-
## Project Overview
**xLLM-service** is a service-layer framework developed based on the **xLLM** inference engine, providing efficient, fault-tolerant, and flexible LLM inference services for clustered deployment.
@@ -51,4 +51,4 @@ It focuses on the full-lifecycle management of instances. All xllm instances mus
As the metrics and event hub, it receives Metrics data reported by various instances, uniformly collects and organizes statistical indicators, and provides data support for decisions such as service scheduling, fault tolerance, and scaling.
### Planner
-It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.
\ No newline at end of file
+It undertakes the functions of strategy analysis and decision-making. Based on the Metrics data reported by the Event Plane (including instance runtime indicators, machine load indicators, etc.), it analyzes the service scaling needs and the necessity of expanding hot instances, and outputs resource adjustment and instance optimization strategies.
diff --git a/docs/en/features/zero_evict_scheduler.md b/docs/src/content/docs/en/features/zero_evict_scheduler.md
similarity index 87%
rename from docs/en/features/zero_evict_scheduler.md
rename to docs/src/content/docs/en/features/zero_evict_scheduler.md
index fc3d16c7b8..90ccf30311 100644
--- a/docs/en/features/zero_evict_scheduler.md
+++ b/docs/src/content/docs/en/features/zero_evict_scheduler.md
@@ -1,5 +1,8 @@
-# Zero Evict Scheduler
-
+---
+title: "Zero Evict Scheduler"
+sidebar:
+ order: 22
+---
## Feature Introduction
xLLM supports the zero evict scheduling strategy. The zero evict scheduling strategy is an algorithm designed to minimize request eviction rates, reducing the need for prefill computation on evicted requests and consequently improving TPOT (Time Per Output Token).
This scheduling algorithm employs simulation rounds to detect whether a request can be scheduled without causing the eviction of other requests.
@@ -14,4 +17,4 @@ The aforementioned strategy has been implemented in xLLM and is exposed through
```
## Performance Impact
-After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.
\ No newline at end of file
+After enabling zero evict, on the Qwen3-8B model with an E2E latency constraint, the TPOT latency **decreased by 27%**.
diff --git a/docs/en/getting_started/launch_xllm.md b/docs/src/content/docs/en/getting_started/launch_xllm.md
similarity index 98%
rename from docs/en/getting_started/launch_xllm.md
rename to docs/src/content/docs/en/getting_started/launch_xllm.md
index d1e80d1aed..720cc79b38 100644
--- a/docs/en/getting_started/launch_xllm.md
+++ b/docs/src/content/docs/en/getting_started/launch_xllm.md
@@ -1,5 +1,8 @@
-# Launch xllm
-
+---
+title: "Launch xllm"
+sidebar:
+ order: 3
+---
Taking Qwen3 as an example, the script for launching xllm is as follows. The provided script is suitable for both single-node single-device and single-node multi-device scenarios. When using multiple devices on a single node, you need to modify `NNODES` (one device represents one node), as well as environment variables such as `ASCEND_RT_VISIBLE_DEVICES`, `CUDA_VISIBLE_DEVICES`, or `MLU_VISIBLE_DEVICES`.
## NPU
@@ -15,7 +18,6 @@ source /usr/local/Ascend/nnal/atb/set_env.sh
export ASCEND_RT_VISIBLE_DEVICES=0
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
-
MODEL_PATH="/path/to/model/Qwen3-8B" # Model path
MASTER_NODE_ADDR="127.0.0.1:9748" # Master node address (must be globally consistent)
START_PORT=18000 # Service starting port
@@ -88,7 +90,6 @@ do
done
```
-
## MLU
```bash
@@ -123,3 +124,4 @@ do
--node_rank=$i \ > $LOG_FILE 2>&1 &
done
```
+
diff --git a/docs/en/getting_started/multi_machine.md b/docs/src/content/docs/en/getting_started/multi_machine.md
similarity index 98%
rename from docs/en/getting_started/multi_machine.md
rename to docs/src/content/docs/en/getting_started/multi_machine.md
index 448f8e3b68..199742acef 100644
--- a/docs/en/getting_started/multi_machine.md
+++ b/docs/src/content/docs/en/getting_started/multi_machine.md
@@ -1,4 +1,8 @@
-# Multi-Node Deployment
+---
+title: "Multi-Node Deployment"
+sidebar:
+ order: 5
+---
This example demonstrates how to launch a 32-GPU (NPU) deployment across 2 machines.
Launching Services on the First Machine:
```shell
@@ -16,7 +20,6 @@ source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export HCCL_IF_BASE_PORT=43432 # HCCL communication base port
-
# 4. Start distributed service
MODEL_PATH="/path/to/your/DeepSeek-R1" # Model path
MASTER_NODE_ADDR="123.123.123.123:9748" # Master node address (must be globally consistent)
@@ -107,3 +110,4 @@ done
```
This example uses 2 machines. You can set the total number of GPUs/NPUs via `--nnodes`, where `--node_rank` specifies the global rank ID for each node.
The `--rank_tablefile=./ranktable_2s_32p.json`parameter points to the configuration file required for establishing the NPU communication domain. For instructions on generating this file, refer to [Ranktable Generation](https://gitee.com/mindspore/models/blob/master/utils/hccl_tools/README.md).
+
diff --git a/docs/en/getting_started/offline_service.md b/docs/src/content/docs/en/getting_started/offline_service.md
similarity index 85%
rename from docs/en/getting_started/offline_service.md
rename to docs/src/content/docs/en/getting_started/offline_service.md
index 5ccd51e565..90473fcacc 100644
--- a/docs/en/getting_started/offline_service.md
+++ b/docs/src/content/docs/en/getting_started/offline_service.md
@@ -1,5 +1,8 @@
-# Offline Inference
-
+---
+title: "Offline Inference"
+sidebar:
+ order: 7
+---
To facilitate users in quickly using xLLM for offline inference, we provide Python script examples for launching offline inference.
## LLM
@@ -12,4 +15,4 @@ Generate embedding example: [:simple-github: https://github.com/jd-opensource/xl
## VLM
-VLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)
\ No newline at end of file
+VLM inference example: [:simple-github: https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py](https://github.com/jd-opensource/xllm/blob/main/examples/generate_vlm.py)
diff --git a/docs/en/getting_started/online_service.md b/docs/src/content/docs/en/getting_started/online_service.md
similarity index 96%
rename from docs/en/getting_started/online_service.md
rename to docs/src/content/docs/en/getting_started/online_service.md
index e7aa397fda..4e4f1b85c1 100644
--- a/docs/en/getting_started/online_service.md
+++ b/docs/src/content/docs/en/getting_started/online_service.md
@@ -1,6 +1,9 @@
-# Online Service
-
-First, start the xllm service according to the [xllm launch documentation](launch_xllm.md). Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.
+---
+title: "Online Service"
+sidebar:
+ order: 6
+---
+First, start the xllm service according to the [xllm launch documentation](./launch_xllm.md). Below are examples of client calls for LLM and VLM. Please modify the parameters according to your actual situation.
## LLM Client Calls
### HTTP Call
@@ -135,7 +138,6 @@ else:
print(ans[0]['message'])
```
-
## VLM Client Calls
### HTTP API
@@ -179,7 +181,6 @@ response = requests.post(
print(response.json())
```
-
### OpenAI API
```python
from openai import OpenAI
@@ -223,3 +224,4 @@ chat_completion = client.chat.completions.create(
result = chat_completion.choices[0].message.content
print("Chat completion output:", result)
```
+
diff --git a/docs/en/getting_started/quick_start.md b/docs/src/content/docs/en/getting_started/quick_start.md
similarity index 87%
rename from docs/en/getting_started/quick_start.md
rename to docs/src/content/docs/en/getting_started/quick_start.md
index 98ca14d663..510a16545a 100644
--- a/docs/en/getting_started/quick_start.md
+++ b/docs/src/content/docs/en/getting_started/quick_start.md
@@ -1,5 +1,8 @@
-# Quick Start
-
+---
+title: "Quick Start"
+sidebar:
+ order: 1
+---
## Environment Setup
All images are stored [here](https://quay.io/repository/jd_xllm/xllm-ai?tab=tags). The docker startup command below uses the dev image as an example.
@@ -44,7 +47,7 @@ docker run -it \
### NVIDIA GPU
-We provide a [Dockerfile](../../../docker/Dockerfile.cuda) for NVIDIA GPU usage, which can be used to build custom image. Of course, you can also use dev image we built based on the default Dockerfile:
+We provide a [Dockerfile](https://github.com/jd-opensource/xllm/blob/main/docker/Dockerfile.cuda) for NVIDIA GPU usage, which can be used to build custom image. Of course, you can also use dev image we built based on the default Dockerfile:
```bash
docker pull quay.io/jd_xllm/xllm-ai:xllm-dev-cuda-x86
```
@@ -103,5 +106,6 @@ python setup.py build
```
## Launch xllm
-Please refer to [How to Launch xllm](launch_xllm.md).
+Please refer to [How to Launch xllm](./launch_xllm.md).
+
diff --git a/docs/en/index.md b/docs/src/content/docs/en/index.md
similarity index 93%
rename from docs/en/index.md
rename to docs/src/content/docs/en/index.md
index a291bb65c3..08221a0e52 100644
--- a/docs/en/index.md
+++ b/docs/src/content/docs/en/index.md
@@ -1,28 +1,17 @@
---
-hide:
- - navigation
+title: "xLLM"
+description: "xLLM documentation home"
---
-
-
-
-
-
-
-
-
Redirecting to English docs
+
+
+
+ Redirecting to English docs
+
+
diff --git a/docs/src/styles/theme.css b/docs/src/styles/theme.css
new file mode 100644
index 0000000000..7792f2d617
--- /dev/null
+++ b/docs/src/styles/theme.css
@@ -0,0 +1,191 @@
+:root,
+::backdrop,
+:root[data-theme='light'],
+[data-theme='light'] ::backdrop {
+ --docs-theme-color: rgba(220, 38, 38, 0.8);
+ --sl-color-accent: var(--docs-theme-color);
+ --sl-color-accent-high: var(--docs-theme-color);
+ --sl-color-bg-accent: var(--docs-theme-color);
+ --sl-color-text-accent: var(--docs-theme-color);
+ --sl-content-width: 60rem;
+}
+
+.sidebar-pane a > span:first-child,
+.sidebar-pane summary .group-label {
+ display: inline-block;
+ transform: scale(1);
+ transform-origin: left center;
+ transition:
+ color 160ms ease,
+ transform 220ms cubic-bezier(0.34, 1.56, 0.64, 1);
+}
+
+.sidebar-pane summary .group-label .large {
+ transition: color 160ms ease;
+}
+
+.sidebar-pane a:is(:hover, :focus-visible),
+.sidebar-pane summary:is(:hover, :focus-visible),
+.sidebar-pane a:is(:hover, :focus-visible) > span:first-child,
+.sidebar-pane summary:is(:hover, :focus-visible) .group-label,
+.sidebar-pane summary:is(:hover, :focus-visible) .group-label .large {
+ color: rgba(220, 38, 38, 1) !important;
+}
+
+.sidebar-pane a:is(:hover, :focus-visible) > span:first-child,
+.sidebar-pane summary:is(:hover, :focus-visible) .group-label {
+ transform: scale(1.035);
+}
+
+@media (prefers-reduced-motion: reduce) {
+ .sidebar-pane a > span:first-child,
+ .sidebar-pane summary .group-label,
+ .sidebar-pane summary .group-label .large {
+ transition: color 160ms ease;
+ }
+
+ .sidebar-pane a:is(:hover, :focus-visible) > span:first-child,
+ .sidebar-pane summary:is(:hover, :focus-visible) .group-label {
+ transform: none;
+ }
+}
+
+starlight-toc a > span {
+ display: inline-block;
+ transform: scale(1);
+ transform-origin: left center;
+ transition:
+ color 160ms ease,
+ transform 220ms cubic-bezier(0.34, 1.56, 0.64, 1);
+}
+
+starlight-toc a:is(:hover, :focus-visible) {
+ color: rgba(220, 38, 38, 0.8);
+}
+
+starlight-toc a[aria-current='true'],
+starlight-toc a[aria-current='true']:is(:hover, :focus-visible) {
+ color: rgba(220, 38, 38, 1) !important;
+}
+
+starlight-toc a:is(:hover, :focus-visible) > span {
+ transform: scale(1.035);
+}
+
+header.header .language-menu {
+ position: relative;
+}
+
+header.header .language-menu summary {
+ align-items: center;
+ background-clip: padding-box;
+ border: 1px solid transparent;
+ border-radius: 0.625rem;
+ color: var(--sl-color-gray-2);
+ cursor: pointer;
+ display: flex;
+ font-size: var(--sl-text-sm);
+ font-weight: 500;
+ gap: 0.35rem;
+ line-height: 1;
+ list-style: none;
+ padding: 0.55rem 0.75rem;
+ transform: scale(1);
+ transform-origin: center;
+ transition:
+ background-color 160ms ease,
+ border-color 160ms ease,
+ color 160ms ease,
+ transform 260ms cubic-bezier(0.34, 1.56, 0.64, 1);
+ user-select: none;
+ white-space: nowrap;
+ will-change: transform;
+}
+
+header.header .language-menu summary::-webkit-details-marker {
+ display: none;
+}
+
+header.header .language-menu summary:is(:hover, :focus-visible),
+header.header .language-menu[open] summary {
+ background-color: color-mix(in srgb, var(--sl-color-accent) 8%, transparent);
+ border-color: color-mix(in srgb, var(--sl-color-accent) 26%, transparent);
+ color: var(--sl-color-text-accent);
+ transform: scale(1.045);
+}
+
+header.header .language-menu svg {
+ fill: none;
+ flex: 0 0 auto;
+ height: 1rem;
+ stroke: currentColor;
+ stroke-linecap: round;
+ stroke-linejoin: round;
+ stroke-width: 2;
+ width: 1rem;
+}
+
+header.header .language-menu .chevron-icon {
+ transition: transform 160ms ease;
+}
+
+header.header .language-menu[open] .chevron-icon {
+ transform: rotate(180deg);
+}
+
+header.header .language-options {
+ background-color: color-mix(in srgb, var(--sl-color-bg-nav) 96%, transparent);
+ border: 1px solid var(--sl-color-gray-5);
+ border-radius: 0.625rem;
+ box-shadow: 0 0.85rem 2rem rgba(0, 0, 0, 0.12);
+ display: grid;
+ gap: 0.125rem;
+ inset-block-start: calc(100% + 0.4rem);
+ inset-inline-end: 0;
+ min-width: 100%;
+ padding: 0.25rem;
+ position: absolute;
+ z-index: calc(var(--sl-z-index-navbar) + 1);
+}
+
+header.header .language-options a {
+ border-radius: 0.5rem;
+ color: var(--sl-color-gray-2);
+ font-size: var(--sl-text-sm);
+ font-weight: 500;
+ line-height: 1;
+ padding: 0.55rem 0.75rem;
+ text-align: start;
+ text-decoration: none;
+ white-space: nowrap;
+}
+
+header.header .language-options a:is(:hover, :focus-visible),
+header.header .language-options a[aria-current='true'] {
+ background-color: color-mix(in srgb, var(--sl-color-accent) 10%, transparent);
+ color: var(--sl-color-text-accent);
+ outline: none;
+}
+
+@media (prefers-reduced-motion: reduce) {
+ starlight-toc a > span {
+ transition: color 160ms ease;
+ }
+
+ starlight-toc a:is(:hover, :focus-visible) > span,
+ header.header .language-menu summary:is(:hover, :focus-visible),
+ header.header .language-menu[open] summary {
+ transform: none;
+ }
+
+ header.header .language-menu summary {
+ transition:
+ background-color 160ms ease,
+ border-color 160ms ease,
+ color 160ms ease;
+ }
+
+ header.header .language-menu .chevron-icon {
+ transition: none;
+ }
+}
diff --git a/docs/tsconfig.json b/docs/tsconfig.json
new file mode 100644
index 0000000000..8bf91d3bb9
--- /dev/null
+++ b/docs/tsconfig.json
@@ -0,0 +1,5 @@
+{
+ "extends": "astro/tsconfigs/strict",
+ "include": [".astro/types.d.ts", "**/*"],
+ "exclude": ["dist"]
+}
diff --git a/docs/zh/.readthedocs.yaml b/docs/zh/.readthedocs.yaml
deleted file mode 100644
index 275f89f6f0..0000000000
--- a/docs/zh/.readthedocs.yaml
+++ /dev/null
@@ -1,28 +0,0 @@
-# Read the Docs configuration file
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-# Required
-version: 2
-
-# Set the OS, Python version, and other tools you might need
-build:
- os: ubuntu-24.04
- tools:
- python: "3.13"
- jobs:
- pre_build:
- # - cp -r docs/zh/* docs/
- - mv docs/zh/* docs/
- - rm -rf docs/en/
- - find docs/ -name "*.md" -exec sed -i 's#../assets/#assets/#g' {} \;
-
-# Build documentation with Mkdocs
-mkdocs:
- configuration: mkdocs_zh.yml
-
-# Optionally, but recommended,
-# declare the Python requirements required to build your documentation
-# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
- install:
- - requirements: docs/requirements.txt
\ No newline at end of file
diff --git a/docs/zh/accuracy_test.md b/docs/zh/accuracy_test.md
deleted file mode 100644
index c78e883067..0000000000
--- a/docs/zh/accuracy_test.md
+++ /dev/null
@@ -1,58 +0,0 @@
-# 1. LLM精度测试
-## 1.1 设置ais_bench
-```bash
-# 使用conda或uv为ais_bench创建虚拟环境
-conda create --name ais_bench python=3.10 -y
-conda activate ais_bench
-
-# 下载ais_bench并安装依赖
-git clone https://gitee.com/aisbench/benchmark.git
-cd benchmark/
-pip3 install -e ./ --use-pep517
-
-# 下载数据集并复制到ais_bench目录下
-cp -r /path/to/dataset /path/to/benchmark/ais_bench/datasets
-```
-
-## 1.2 修改配置
-根据实际情况修改精度测试配置文件：`/path/to/benchmark/ais_bench/benchmark/configs/models/vllm_api/vllm_api_general_chat.py`，采样参数建议按如下代码设置：
-```python
-models = [
- dict(
- attr="service",
- type=VLLMCustomAPIChat,
- abbr='vllm-api-general-chat',
- path="/path/to/model/Qwen3-8B", # 模型路径
- model="Qwen3-8B", # 模型名称
- request_rate = 0,
- retry = 2,
- host_ip = "127.0.0.1",
- host_port = 19000, # xllm服务端端口
- max_out_len = 32768, # 限制模型最大长度
- batch_size=32,
- trust_remote_code=False,
- generation_kwargs = dict(
- temperature = 0.6,
- # top_k = -1,
- top_p = 0.95,
- # seed = None,
- # repetition_penalty = 1,
- ),
- pred_postprocessor=dict(type=extract_non_reasoning_content)
- )
-]
-```
-
-## 1.3 启动ais_bench
-在使用ais_bench前需要先启动xllm服务。使用`ais_bench -h`能够获取参数含义，对于gsm8k和ceval数据集的启动命令如下：
-```bash
-# 使用gsm8k数据集
-ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_0_shot_cot_chat_prompt --dump-eval-details
-
-# 使用ceval数据集
-ais_bench --models vllm_api_general_chat --datasets ceval_gen_0_shot_cot_chat_prompt --merge-ds --dump-eval-details
-```
-
-我们会在未来将ais_bench和数据集（ceval和gsm8k）集成进开发镜像，ais_bench文档和数据集如下：
-* [ais_bench文档](https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/index.html)
-* [数据集](https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/base_tutorials/all_params/datasets.html)
diff --git a/docs/zh/features/basics.md b/docs/zh/features/basics.md
deleted file mode 100644
index 919757e4b7..0000000000
--- a/docs/zh/features/basics.md
+++ /dev/null
@@ -1,9 +0,0 @@
-# 基础知识
-
-- xLLM使用一卡一进程模式，多卡之间使用rpc进行函数调用，模型计算过程中的数据通信使用device集合通信库。
-
-- HCCL/LCCL是高性能集合通信，提供单机多卡以及多机多卡间的数据并行、模型并行集合通信方案。
-
-
-
-
diff --git a/docs/zh/features/continuous_scheduler.md b/docs/zh/features/continuous_scheduler.md
deleted file mode 100644
index d8461ff8cd..0000000000
--- a/docs/zh/features/continuous_scheduler.md
+++ /dev/null
@@ -1,7 +0,0 @@
-# Continuous调度器
-
-## 功能介绍
-xLLM实现了支持continuous batching的调度策略，continuous_batch是一种动态批处理策略，它不等待批次填满，而是在有请求时就开始处理，同时持续接收新请求并将其加入正在执行的批次中，从而在保持高吞吐量的同时显著降低延迟。
-
-## 使用方式
-xLLM提供了continuous batching调度策略。目前`enable_chunked_prefill`默认值为true，开箱即用时默认调度器是chunked prefill调度器。
diff --git a/docs/zh/features/groupgemm.md b/docs/zh/features/groupgemm.md
deleted file mode 100644
index 2993c3d9d5..0000000000
--- a/docs/zh/features/groupgemm.md
+++ /dev/null
@@ -1,41 +0,0 @@
-# GroupGEMM算子优化
-
-# 背景
-混合专家(Mixture of Experts, MoE)架构已成为扩展大规模语言模型的重要范式，其核心思想是将输入token动态路由至不同的专家子网络进行处理。在推理过程中，GroupGEMM算子是MoE架构的关键计算单元，负责高效执行多个专家矩阵乘法的并行计算，且在整个推理耗时中占据主导地位。
-
-## 功能介绍
-结合当前GroupGEMM的性能瓶颈为I/O受限，提出了一种优化方案，通过索引重排替代数据拷贝，取消了对token向量的多次复制，改为维护专家分配的索引表。通过该行号索引，直接将token映射到相应的专家计算单元，并将token的分配调度与矩阵乘法融合为一个单一的kernel。
-
-
-## 用户接口
-
-### 算子直调API
-```c++
-aclnnStatus aclnnIndexGroupMatmulGetWorkspaceSize(
- const aclTensorList *x,
- const aclTensorList *weight,
- const aclTensorList *scale,
- const aclTensorList *perTokenScale,
- const aclTensor *groupList,
- const aclTensorList *out,
- uint64_t *workspaceSize,
- aclOpExecutor **executor);
-
-aclnnStatus aclnnIndexGroupMatmul(
- void *workspace,
- uint64_t workspaceSize,
- aclOpExecutor *executor,
- aclrtStream stream);
-```
-
-- `x`: 输入的张量列表，包含待处理的数据。
-- `weight`: 权重张量，包含模型的参数。
-- `scale`: 缩放因子，用于调整输入张量的值。
-- `perTokenScale`:每个token的缩放因子，用于动态调整。
-- `groupList`: 专家组列表，指示哪些专家参与计算。
-- `out`: 输出张量列表，存储计算结果。
-
-## 性能效果
-
-
-* 优化后的GroupMatmul算子在计算时间上表现出明显的优势，尤其是在k为128，m为64情况下，如图所示，优化后算子计算延时 **减少50%**。
diff --git a/docs/zh/features/multimodal.md b/docs/zh/features/multimodal.md
deleted file mode 100755
index e786493260..0000000000
--- a/docs/zh/features/multimodal.md
+++ /dev/null
@@ -1,19 +0,0 @@
-# 多模态支持
-本文档主要介绍xLLM推理引擎中多模态的支持进展，包括支持模型及模态类型，以及离在线接口等。
-
-## 支持模型
-- Qwen2.5-VL: 包括7B/32B/72B。
-- Qwen3-VL: 包括2B/4B/8B/32B。
-- Qwen3-VL-MoE: 包括A3B/A22B。
-- MiniCPM-V-2_6: 7B。
-
-## 模态类型
-- 图片: 支持单图、多图的输入，以及图片+Prompt组合、纯文本Promot等输入方式。
-
-
-!!! warning "注意事项"
- - 目前多模态后端不支持prefix cache以及chunk prefill，正在支持中。
- - 目前，xLLM统一基于JinJa渲染ChatTemplate，部署MiniCPM-V-2_6，模型目录需提供ChatTemplate文件。
- - 图片支持Base64输入以及图片Url。
- - 目前多模态模型主要支持了图片模态，视频、音频等模态正在推进中。
-
diff --git a/docs/zh/features/overview.md b/docs/zh/features/overview.md
deleted file mode 100644
index 197c688991..0000000000
--- a/docs/zh/features/overview.md
+++ /dev/null
@@ -1,57 +0,0 @@
-# 整体架构
-
-## 背景
-
-近年来，随着百亿至万亿参数规模的大语言模型（如GPT、Claude、DeepSeek、LLaMA等）在自然语言处理和多模态交互领域取得突破性进展，产业界对高效推理引擎与服务体系的构建提出了迫切需求。如何降低集群推理成本、提升推理效率已成为实现规模化商业落地的关键挑战。
-
-尽管当前已涌现出一批面向大模型推理的优化引擎，但在实际部署过程中仍面临诸多技术瓶颈：
-
-- 硬件适配性挑战：现有推理引擎对国产芯片等专用加速器的架构特性支持不足，难以充分发挥异构计算硬件的性能潜力，导致计算资源利用率低下；
-- MoE架构优化难题：专家并行机制中的令牌分发过程产生显著的All-to-All通信开销，同时动态路由策略引发的专家负载不均衡问题严重制约了系统的可扩展性；
-- 长上下文管理瓶颈：随着模型上下文窗口持续扩展，KV缓存在内存碎片化处理、跨节点同步等方面的优化效率直接影响整体推理吞吐性能；
-- 混合部署效能局限：现有推理集群在同时处理在线服务和离线任务时，难以兼顾服务质量（SLO）保障与资源利用率优化。
-- 动态PD适配不足：当输入/输出序列长度出现剧烈波动时，静态PD资源划分缺乏实时调整PD资源配置的能力，既可能导致GPU资源闲置，又存在SLO违约风险。
-
-为此，我们提出了xLLM——高效且易用的开源智能推理框架，为模型在国产芯片上的推理提供企业级服务保障与高性能引擎计算能力。
-## 功能介绍
-
-xLLM提供智能计算能力，我们实现了多种计算系统层和算法驱动层的联合推理加速：
-
-### 计算系统层
-
-#### 多层流水线执行编排
-在框架层异步化CPU调度，使其与芯片推理计算形成流水线，减少计算空泡；在模型图层，切分单个batch，形成两个micro-batches之间的流水线，重叠计算通信；在算子内核层，不同计算单元间流水，重叠计算访存。
-#### 动态shape的图执行优化
-针对大语言模型处理动态输入（如可变序列长度和批大小）时面临的静态图适配问题，xLLM通过参数化设计捕获输入维度实现动态适配，并结合多图缓存方案减小编译开销，使用受管控的显存池替代绝对地址保障安全复用，最终在保持高灵活性的同时获得较高的执行效率。
-#### 算子优化
-xLLM实现了LLM中的关键算子在国产硬件芯片上的特定优化，包括GroupMatmul、Chunked Prefill等。
-#### xTensor显存管理
-xTensor 显存管理框架采用 物理内存页池预分配 + 虚拟地址连续性映射 的方法，通过动态按需映射物理页、复用可重用内存页（Reusable）及异步预映射优化调度，结合 NPU 算子适配（如虚拟地址化 FlashMLA），实现了高效动态内存管理，取得了内存利用率提升以及延迟降低。
-
-### 算法驱动层
-
-#### PD分离
-xLLM全面支持PD分离场景，实现了高效的PD实例的管理以及PD实例之间的通信以及kv cache传输。
-
-#### 全局调度
-xLLM对请求和实例做全周期的资源调度智能管理。
-##### 实例调度
-我们实现了多种实例调度策略来选择如何将实例分配到更适合的实例。包括简单的Round Robin策略，基于请求在各实例上的 prefix cache 命中率来选择的 prefix cache-aware 策略，基于实例上的显存空闲程度的 KV Cache-aware 策略。另外，针对PD分离场景，由于静态的PD比例往往无法很好应对流量以及请求输入输出长度突变的场景，我们实现了一种自适应的PD动态调度器，负责在线请求的全局实例分配与运行时PD动态调整。
-##### 请求调度
-我们实现了多种请求调度策略，支持continuous batching，包括chunked prefill，prefill优先和decode优先等batch策略，同时全面支持PD分离场景。
-#### 全局kv cache管理
-在全局层面采用ETCD作为元数据服务中间件，实现集群服务注册、负载信息同步及全局缓存状态管理。每个计算实例维护本地多级缓存池。在调度策略方面，系统采用基于 KV Cache 的动态决策机制：首先进行前缀匹配检测，计算各候选节点的 KV Cache 复用率，最终选择综合性能最优的节点进行处理，实现 KV Cache 的动态卸载与迁移。
-
-#### 投机推理
-xLLM内置优化后的投机推理算法，一次生成多个tokens提升吞吐。xLLM通过投机模块下沉减少通信成本，并使用调度和计算时序重叠优化、减少投机场景算子数据搬运等方式优化投机推理计算。
-#### MOE负载均衡
-xLLM针对MoE模型实现了基于历史专家负载统计的专家权重更新，在推理时通过高效专家负责统计和双缓冲无感知的专家权重更新实现有效的动态负载均衡。
-
-### 多模态支持
-
-xLLM对包括Qwen2-VL，MiniCPMV在内的多种多模态模型提供全面的支持。
-
-### 相关设计文档
-
-- [Graph Mode 设计文档](../design/graph_mode_design.md)
-- [生成式推荐设计文档](../design/generative_recommendation_design.md)
diff --git a/docs/zh/features/ppmatmul.md b/docs/zh/features/ppmatmul.md
deleted file mode 100644
index 038e931dad..0000000000
--- a/docs/zh/features/ppmatmul.md
+++ /dev/null
@@ -1,36 +0,0 @@
-# PpMatmul 算子优化
-
-## 背景
-
-针对大模型推理中矩阵乘法占比高、耗时长的问题，优化了矩阵乘法算子的实现。
-
-## 功能介绍
-
-PpMatmul 算子使用 Tiling 切分策略，将矩阵乘法分解为多个小的矩阵乘法任务。然而当 tile 数量较小时任务无法被均匀分配到所有 npu 核心上，导致 tail effect 问题，影响计算效率。我们通过预取内存或重新划分任务的方式，优化 PpMatmul 算子的性能。
-
-## 用户接口
-
-### 算子直调 API
-
-```cpp
-aclnnStatus aclnnPpMatmulOptGetWorkspaceSize(
- const aclTensor *a,
- const aclTensor *b,
- const aclTensor *out,
- uint64_t *workspaceSize,
- aclOpExecutor **executor);
-
-aclnnStatus aclnnPpMatmulOpt(
- void *workspace,
- uint64_t workspaceSize,
- aclOpExecutor *executor,
- aclrtStream stream);
-```
-
-- `a`: 输入矩阵 A。
-- `b`: 输入矩阵 B。
-- `out`: 输出矩阵，存储计算结果。
-
-## 性能效果
-
-对于 tile 数量较小的情况（例如 M 较小，对应于 batch size 较小的情况），在（TP=4）时，算子较优化前有 **18%** 的性能提升。
\ No newline at end of file
diff --git a/docs/zh/features/topk_topP.md b/docs/zh/features/topk_topP.md
deleted file mode 100644
index 4ba2a78108..0000000000
--- a/docs/zh/features/topk_topP.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# Topk&Topp算子优化
-
-## 背景
-在自然语言生成任务中，topK和topP采样策略被广泛应用于控制生成文本的多样性和质量。然而，在小模型中，这两种策略的计算耗时相对较长。这主要是由于小模型的参数较少，导致在处理概率分布时，排序和筛选的效率降低，从而影响了生成速度。因此，优化小模型中topK和topP的实现，可以提升其采样效率。
-
-
-## 功能介绍
-
-topKtopP算子的实现将排序、topK、softmax和topP等多个小算子融合为一个大算子，从而提高了计算效率和性能。
-
-
-## 用户接口
-### 算子调用API
-```c++
-void top_k_top_p(torch::Tensor& logits,
- const torch::Tensor& topK,
- const torch::Tensor& topP);
-```
-
-- `logits`: 输入的logits张量，包含模型的输出分数。
-- `topK`: 用于选择的前K个概率的阈值张量。
-- `topP`: 用于选择的累积概率的阈值张量。
-
-
-## 性能效果
-
-* 使用topKtopP融合算子后，在qwen2-0.5B模型中，TTOT **下降37%**,TTFT **提升10%**。
diff --git a/docs/zh/getting_started/disagg_pd.md b/docs/zh/getting_started/disagg_pd.md
deleted file mode 100644
index 1efe40cc98..0000000000
--- a/docs/zh/getting_started/disagg_pd.md
+++ /dev/null
@@ -1,88 +0,0 @@
-# PD分离部署
-
-`xllm`支持PD分离部署，这需要与我们的另一个开源库[xllm service](https://github.com/jd-opensource/xllm-service)配套使用。
-## xLLM Service依赖
-首先，我们下载安装`xllm service`，与安装编译`xllm`类似：
-```bash
-git clone https://github.com/jd-opensource/xllm-service
-cd xllm_service
-git submodule init
-git submodule update
-```
-### etcd安装
-`xllm_service`依赖[etcd](https://github.com/etcd-io/etcd)，使用etcd官方提供的[安装脚本](https://github.com/etcd-io/etcd/releases)进行安装，其脚本提供的默认安装路径是`/tmp/etcd-download-test/etcd`，我们可以手动修改其脚本中的安装路径，也可以运行完脚本之后手动迁移：
-```bash
-mv /tmp/etcd-download-test/etcd /path/to/your/etcd
-```
-### xLLM Service编译
-先应用patch:
-```bash
-sh prepare.sh
-```
-
-再执行编译:
-```bash
-mkdir -p build
-cd build
-cmake ..
-make -j 8
-cd ..
-```
-!!! warning "可能的错误"
- 这里能会遇到关于`boost-locale`和`boost-interprocess`的安装错误：`vcpkg-src/packages/boost-locale_x64-linux/include: No such file or directory`,`/vcpkg-src/packages/boost-interprocess_x64-linux/include: No such file or directory`
- 我们使用`vcpkg`重新安装这些包:
- ```bash
- /path/to/vcpkg remove boost-locale boost-interprocess
- /path/to/vcpkg install boost-locale:x64-linux
- /path/to/vcpkg install boost-interprocess:x64-linux
- ```
-## PD分离运行
-启动etcd:
-```bash
-./etcd-download-test/etcd --listen-peer-urls 'http://localhost:2390' --listen-client-urls 'http://localhost:2389' --advertise-client-urls 'http://localhost:2391'
-```
-启动xllm service:
-```bash
-ENABLE_DECODE_RESPONSE_TO_SERVICE=true ./xllm_master_serving --etcd_addr="127.0.0.1:12389" --http_server_port 28888 --rpc_server_port 28889 --tokenizer_path=/path/to/tokenizer_config_dir/
-```
-
-以Qwen2-7B为例
-
-- 启动Prefill实例
- ```bash
- /path/to/xllm --model=path/to/Qwen2-7B-Instruct \
- --port=8010 \
- --devices="npu:0" \
- --master_node_addr="127.0.0.1:18888" \
- --enable_prefix_cache=false \
- --enable_chunked_prefill=false \
- --enable_disagg_pd=true \
- --instance_role=PREFILL \
- --etcd_addr=127.0.0.1:12389 \
- --transfer_listen_port=26000 \
- --disagg_pd_port=7777 \
- --node_rank=0 \
- --nnodes=1
- ```
-- 启动Decode实例
- ```bash
- /path/to/xllm --model=path/to/Qwen2-7B-Instruct \
- --port=8020 \
- --devices="npu:1" \
- --master_node_addr="127.0.0.1:18898" \
- --enable_prefix_cache=false \
- --enable_chunked_prefill=false \
- --enable_disagg_pd=true \
- --instance_role=DECODE \
- --etcd_addr=127.0.0.1:12389 \
- --transfer_listen_port=26100 \
- --disagg_pd_port=7787 \
- --node_rank=0 \
- --nnodes=1
- ```
-需要注意：
-
-- PD分离需要读取`/etc/hccn.conf`文件，确保将物理机上的该文件映射到了容器中
-- `etcd_addr`需与`xllm_service`的`etcd_addr`相同
-
-测试命令和上面类似，注意`curl http://localhost:{PORT}/v1/chat/completions ...`的`PORT`选择为启动xLLM service的`http_server_port`。
\ No newline at end of file
diff --git a/docs/zh/xLLM_Technical_Report_zh.pdf b/docs/zh/xLLM_Technical_Report_zh.pdf
deleted file mode 100644
index afb3b25df0..0000000000
Binary files a/docs/zh/xLLM_Technical_Report_zh.pdf and /dev/null differ
diff --git a/mkdocs_en.yml b/mkdocs_en.yml
deleted file mode 100644
index 0a5bdb0547..0000000000
--- a/mkdocs_en.yml
+++ /dev/null
@@ -1,228 +0,0 @@
-site_name: xLLM
-site_url: !ENV READTHEDOCS_CANONICAL_URL
-
-# Repository
-repo_name: jd-opensource/xllm
-repo_url: https://github.com/jd-opensource/xllm
-
-edit_uri: edit/main/docs/en
-use_directory_urls: true
-# Copyright
-copyright: Copyright © 2025 xLLM Team
-# docs_dir: docs/en
-# Configuration
-theme:
- name: material
- language: en
- custom_dir: docs/mkdocs/overrides
- features:
- - announce.dismiss
- - content.action.edit
- - content.action.view
- - content.code.annotate
- - content.code.copy
- - content.code.select
- - content.footnote.tooltips
- # - content.tabs.link
- - content.tooltips
- # - header.autohide
- - navigation.expand
- - navigation.footer
- - navigation.indexes
- - navigation.path
- # - navigation.instant
- # - navigation.instant.prefetch
- # - navigation.instant.progress
- # - navigation.prune
- - navigation.sections
- - navigation.tabs
- # - navigation.tabs.sticky
- - navigation.top
- - navigation.tracking
- - search.highlight
- - search.share
- - search.suggest
- - toc.follow
- # - toc.integrate
- palette:
- - media: "(prefers-color-scheme)"
- toggle:
- icon: material/link
- name: Switch to light mode
- - media: "(prefers-color-scheme: light)"
- scheme: default
- primary: custom
- accent: custom
- toggle:
- # icon: material/toggle-switch
- icon: material/eye
- name: Switch to dark mode
- - media: "(prefers-color-scheme: dark)"
- scheme: slate
- primary: black
- accent: custom
- toggle:
- # icon: material/toggle-switch-off
- icon: material/eye-outline
- name: Switch to system preference
- # primary: custom
- # accent: custom
- font:
- text: Roboto
- code: Roboto Mono
- favicon: assets/logo.png
- logo: assets/logo.png
- icon:
- annotation: material/arrow-right-circle
- # icon:
- # logo: logo
-# Plugins
-plugins:
- # - blog
- # - static-i18n
- - git-revision-date-localized:
- exclude:
- - argparse/*
- - examples/*
- - search:
- separator: '[\s\u200b\-_,:!=\[\]()"`/]+|\.(?!\d)|&[lg]t;|(?!\b)(?=[A-Z][a-z])'
- - minify:
- minify_html: true
-extra_css:
- - mkdocs/stylesheets/extra.css
-# extra_dir: assets
-# Hooks
-# hooks:
-# - material/overrides/hooks/shortcodes.py
-# - material/overrides/hooks/translations.py
-
-# Additional configuration
-extra:
- source:
- path: https://github.com/jd-opensource/xllm/blob/main/docs/en/
- status:
- new: Recently added
- deprecated: Deprecated
- # analytics:
- # provider: google
- # property: !ENV GOOGLE_ANALYTICS_KEY
- social:
- - icon: fontawesome/brands/github
- link: https://github.com/jd-opensource/xllm
- - icon: gitcodeai
- link: https://gitcode.com/xLLM-AI/xllm
- - icon: fontawesome/brands/docker
- link: https://hub.docker.com/r/xllm/xllm-ai
- - icon: email-fill
- link: https://github.com/jd-opensource/xllm
- # - icon: fontawesome/brands/python
- # link: https://pypi.org/project/mkdocs-material/
- # - icon: fontawesome/brands/bluesky
- # link: https://bsky.app/profile/squidfunk.bsky.social
- # - icon: fontawesome/brands/mastodon
- # link: https://fosstodon.org/@squidfunk
- # - icon: fontawesome/brands/x-twitter
- # link: https://x.com/squidfunk
-
-# Extensions
-markdown_extensions:
- - tables
- - abbr
- - admonition
- - attr_list
- - def_list
- - footnotes
- - md_in_html
- - toc:
- permalink: true
- - pymdownx.arithmatex:
- generic: true
- - pymdownx.betterem:
- smart_enable: all
- - pymdownx.caret
- - pymdownx.details
- - pymdownx.emoji:
- emoji_generator: !!python/name:material.extensions.emoji.to_svg
- emoji_index: !!python/name:material.extensions.emoji.twemoji
- - pymdownx.highlight:
- anchor_linenums: true
- line_spans: __span
- pygments_lang_class: true
- - pymdownx.inlinehilite
- - pymdownx.keys
- - pymdownx.magiclink:
- normalize_issue_symbols: true
- repo_url_shorthand: true
- user: jd-opensource
- repo: xllm
- - pymdownx.mark
- - pymdownx.smartsymbols
- - pymdownx.snippets
- - pymdownx.superfences
- - pymdownx.tabbed
- # - pymdownx.superfences:
- # custom_fences:
- # - name: mermaid
- # class: mermaid
- # format: !!python/name:pymdownx.superfences.fence_code_format
- # - pymdownx.tabbed:
- # alternate_style: true
- # combine_header_slug: true
- # slugify: !!python/object/apply:pymdownx.slugs.slugify
- # kwds:
- # case: lower
- - pymdownx.tasklist:
- custom_checkbox: true
- - pymdownx.tilde
- - mdx_math:
- enable_dollar_delimiter: true
-extra_javascript:
- - mkdocs/javascripts/mathjax.js
- - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
-
-
-nav:
- - Home: index.md
- - User Guide:
- - getting_started/quick_start.md
- - supported_models.md
- - getting_started/launch_xllm.md
- - getting_started/disagg_pd.md
- - getting_started/multi_machine.md
- - Basic Usage:
- - getting_started/online_service.md
- - getting_started/offline_service.md
- - Features:
- - Overview:
- - features/overview.md
- - Feature List:
- - Runtime:
- - features/async_schedule.md
- - features/multi_streams.md
- - Scheduler:
- - features/continuous_scheduler.md
- - features/chunked_scheduler.md
- - features/zero_evict_scheduler.md
- - Distributed:
- - features/disagg_pd.md
- - Operators:
- - features/groupgemm.md
- - features/ppmatmul.md
- - features/topk_topP.md
- - Memory Management:
- - features/prefix_cache.md
- - features/global_kvcache.md
- - features/xtensor_memory.md
- - Multimodal:
- - features/multimodal.md
- - MoE Optimization:
- - features/moe_params.md
- - features/eplb.md
- - Advanced Guides:
- - Speculative Inference: features/mtp.md
- - GraphMode: features/graph_mode.md
- - Developer Guide:
- - dev_guide/code_arch.md
- - dev_guide/tilelang_ascend_kernel_dev.md
- - CLI Reference:
- - cli_reference.md
diff --git a/mkdocs_zh.yml b/mkdocs_zh.yml
deleted file mode 100644
index 13479521f9..0000000000
--- a/mkdocs_zh.yml
+++ /dev/null
@@ -1,281 +0,0 @@
-site_name: xLLM
-site_url: !ENV READTHEDOCS_CANONICAL_URL
-
-# Repository
-repo_name: jd-opensource/xllm
-repo_url: https://github.com/jd-opensource/xllm
-
-edit_uri: edit/main/docs/zh
-use_directory_urls: true
-# Copyright
-copyright: Copyright © 2025 xLLM Team
-# docs_dir: docs/zh
-# Configuration
-theme:
- name: material
- language: zh
- custom_dir: docs/mkdocs/overrides
- features:
- - announce.dismiss
- - content.action.edit
- - content.action.view
- - content.code.annotate
- - content.code.copy
- - content.code.select
- - content.footnote.tooltips
- # - content.tabs.link
- - content.tooltips
- # - header.autohide
- - navigation.expand
- - navigation.footer
- - navigation.indexes
- - navigation.path
- # - navigation.instant
- # - navigation.instant.prefetch
- # - navigation.instant.progress
- # - navigation.prune
- - navigation.sections
- - navigation.tabs
- # - navigation.tabs.sticky
- - navigation.top
- - navigation.tracking
- - search.highlight
- - search.share
- - search.suggest
- - toc.follow
- # - toc.integrate
- palette:
- - media: "(prefers-color-scheme)"
- toggle:
- icon: material/link
- name: 切换亮色模式
- - media: "(prefers-color-scheme: light)"
- scheme: default
- primary: custom
- accent: custom
- toggle:
- # icon: material/toggle-switch
- icon: material/eye
- name: 切换暗色模式
- - media: "(prefers-color-scheme: dark)"
- scheme: slate
- primary: black
- accent: custom
- toggle:
- # icon: material/toggle-switch-off
- icon: material/eye-outline
- name: 切换到系统偏好
- # primary: custom
- # accent: custom
- font:
- text: Roboto
- code: Roboto Mono
- favicon: assets/logo.png
- logo: assets/logo.png
- icon:
- annotation: material/arrow-right-circle
- # icon:
- # logo: logo
-# Plugins
-plugins:
- # - awesome-pages
- # - blog
- # - static-i18n
- - git-revision-date-localized:
- exclude:
- - argparse/*
- - examples/*
- - search:
- separator: '[\s\u200b\-_,:!=\[\]()"`/]+|\.(?!\d)|&[lg]t;|(?!\b)(?=[A-Z][a-z])'
- - minify:
- minify_html: true
-extra_css:
- - mkdocs/stylesheets/extra.css
-# extra_dir: assets
-# Hooks
-# hooks:
-# - material/overrides/hooks/shortcodes.py
-# - material/overrides/hooks/translations.py
-
-# Additional configuration
-extra:
- source:
- path: https://github.com/jd-opensource/xllm/blob/main/docs/zh/
-
- status:
- new: Recently added
- deprecated: Deprecated
- # analytics:
- # provider: google
- # property: !ENV GOOGLE_ANALYTICS_KEY
- social:
- - icon: fontawesome/brands/github
- link: https://github.com/jd-opensource/xllm
- - icon: gitcodeai
- link: https://gitcode.com/xLLM-AI/xllm
- - icon: fontawesome/brands/docker
- link: https://hub.docker.com/r/xllm/xllm-ai
- - icon: email-fill
- link: https://github.com/jd-opensource/xllm
- # - icon: fontawesome/brands/python
- # link: https://pypi.org/project/mkdocs-material/
- # - icon: fontawesome/brands/bluesky
- # link: https://bsky.app/profile/squidfunk.bsky.social
- # - icon: fontawesome/brands/mastodon
- # link: https://fosstodon.org/@squidfunk
- # - icon: fontawesome/brands/x-twitter
- # link: https://x.com/squidfunk
-
-# Extensions
-markdown_extensions:
- - tables
- - abbr
- - admonition
- - attr_list
- - def_list
- - footnotes
- - md_in_html
- - toc:
- permalink: true
- - pymdownx.arithmatex:
- generic: true
- - pymdownx.betterem:
- smart_enable: all
- - pymdownx.caret
- - pymdownx.details
- - pymdownx.emoji:
- emoji_generator: !!python/name:material.extensions.emoji.to_svg
- emoji_index: !!python/name:material.extensions.emoji.twemoji
- - pymdownx.highlight:
- anchor_linenums: true
- line_spans: __span
- pygments_lang_class: true
- - pymdownx.inlinehilite
- - pymdownx.keys
- - pymdownx.magiclink:
- normalize_issue_symbols: true
- repo_url_shorthand: true
- user: jd-opensource
- repo: xllm
- - pymdownx.mark
- - pymdownx.smartsymbols
- - pymdownx.snippets
- - pymdownx.superfences
- - pymdownx.tabbed
- # - pymdownx.superfences:
- # custom_fences:
- # - name: mermaid
- # class: mermaid
- # format: !!python/name:pymdownx.superfences.fence_code_format
- # - pymdownx.tabbed:
- # alternate_style: true
- # combine_header_slug: true
- # slugify: !!python/object/apply:pymdownx.slugs.slugify
- # kwds:
- # case: lower
- - pymdownx.tasklist:
- custom_checkbox: true
- - pymdownx.tilde
- - mdx_math:
- enable_dollar_delimiter: true
-extra_javascript:
- - mkdocs/javascripts/mathjax.js
- - https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
-
-
-nav:
- - 主页: index.md
- - 用户指南:
- - getting_started/quick_start.md
- - supported_models.md
- - getting_started/launch_xllm.md
- - getting_started/disagg_pd.md
- - getting_started/multi_machine.md
- - 基本用法:
- - getting_started/online_service.md
- - getting_started/offline_service.md
- - 功能介绍:
- - 概览:
- - features/overview.md
- - 功能列表:
- - 运行时:
- - features/async_schedule.md
- - features/multi_streams.md
- - 调度器:
- - features/continuous_scheduler.md
- - features/chunked_scheduler.md
- - features/zero_evict_scheduler.md
- - 分布式:
- - features/disagg_pd.md
- - 算子:
- - features/groupgemm.md
- - features/ppmatmul.md
- - features/topk_topP.md
- - KV存储管理:
- - features/prefix_cache.md
- - features/global_kvcache.md
- - features/xtensor_memory.md
- - 多模态:
- - features/multimodal.md
- - MoE优化:
- - features/moe_params.md
- - features/eplb.md
- - 进阶指南:
- - 投机推理: features/mtp.md
- - GraphMode: features/graph_mode.md
- - xLLM Service概览: features/xllm_service_overview.md
- - 开发者指南:
- - dev_guide/code_arch.md
- - dev_guide/tilelang_ascend_kernel_dev.md
- - CLI参考:
- - cli_reference.md
-
-# nav:
-# - 主页: index.md
-# - 开始使用: getting_started/
-# - 功能介绍: features/
-# - CLI参考: cli_reference.md
-# nav:
-# - 主页: zh/index.md
-# - 开始使用:
-# - zh/getting_started/compile.md
-# - 快速开始:
-# - 在线服务:
-# - zh/getting_started/single_node.md
-# - zh/getting_started/multi_node.md
-# - zh/getting_started/PD_disagg.md
-# - 多模态支持:
-# - zh/getting_started/start_vlm_service.md
-# - zh/getting_started/scripts.md
-# - 功能介绍:
-# - 概览:
-# - zh/features/overview.md
-# - 功能列表:
-# - 运行时:
-# - zh/features/async_schedule.md
-# - zh/features/multi_streams.md
-# - 调度算法:
-# - zh/features/continuous_scheduler.md
-# - zh/features/chunked_scheduler.md
-# - zh/features/zero_evict_scheduler.md
-# - zh/features/prefix_cache.md
-# - 分布式:
-# - zh/features/disagg_pd.md
-# - 算子:
-# - zh/features/groupgemm.md
-# - zh/features/ppmatmul.md
-# - zh/features/topk_topP.md
-# - KV存储管理:
-# - zh/features/global_kvcache.md
-# - zh/features/xtensor_memory.md
-# - 多模态:
-# - zh/features/multimodal.md
-# - MoE优化:
-# - zh/features/moe_params.md
-# - zh/features/eplb.md
-# - 进阶指南:
-# - 投机推理: zh/features/mtp.md
-# - ACLGraph: zh/features/acl_graph.md
-# - xLLM Service概览: zh/features/xllm_service_overview.md
-# - CLI参考:
-# - zh/cli_reference.md
diff --git a/pyproject.toml b/pyproject.toml
index 924ec7d448..b5b12713b8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,8 +24,8 @@ classifiers = [
]
[project.urls]
-Homepage = "https://xllm.readthedocs.io/zh-cn/latest/"
-Documentation = "https://xllm.readthedocs.io/zh-cn/latest/"
+Homepage = "https://github.com/jd-opensource/xllm"
+Documentation = "https://jd-opensource.github.io/xllm/en/"
Repository = "https://github.com/jd-opensource/xllm"
[project.scripts]
diff --git a/setup.py b/setup.py
index 35885b8537..5f30919f77 100644
--- a/setup.py
+++ b/setup.py
@@ -638,8 +638,8 @@ def parse_arguments() -> dict[str, Any]:
long_description_content_type="text/markdown",
url="https://github.com/jd-opensource/xllm",
project_urls={
- "Homepage": "https://xllm.readthedocs.io/zh-cn/latest/",
- "Documentation": "https://xllm.readthedocs.io/zh-cn/latest/",
+ "Homepage": "https://github.com/jd-opensource/xllm",
+ "Documentation": "https://jd-opensource.github.io/xllm/en/",
},
classifiers=[
"Intended Audience :: Developers",
+
-[](https://xllm.readthedocs.io/zh-cn/latest/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
+[](https://jd-opensource.github.io/xllm/en/) [](https://hub.docker.com/r/xllm/xllm-ai) [](https://opensource.org/licenses/Apache-2.0) [](https://arxiv.org/abs/2510.14686) [](https://deepwiki.com/jd-opensource/xllm)
+
+ 
+
+
+