diff --git a/README.md b/README.md
index 48f05047f..9470b1106 100644
--- a/README.md
+++ b/README.md
@@ -132,12 +132,15 @@ spec:
For a role whose replica spans multiple Pods, request one zone for each replica group:
```yaml
-roles:
- - name: server
- replicas: 2
- size: 4
- topology:
- requiredLevel: topology.kubernetes.io/zone
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+spec:
+ roles:
+ - name: server
+ replicas: 2
+ size: 4
+ topology:
+ requiredLevel: topology.kubernetes.io/zone
```
This is a role fragment; the engine still needs its own distributed execution settings. Use
@@ -191,7 +194,7 @@ spec:
The `ModelDeployment` below names the binding, which grants this namespace access to the cache.
Choose member nodes, quotas, block size and dtype for your engine before applying these resources.
-The [KV Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step.
+The [Shared Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step.
### Model Delivery
diff --git a/docs/README.md b/docs/README.md
index 74f9b755e..39966219d 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -13,7 +13,7 @@ The capability guides are [Heterogeneous Devices](modules/devices/_index.md),
For a first workload, follow [Usage](../README.md#usage), then use
[Accelerator Requests](modules/devices/requests.md) to choose its resource keys. The
-[Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster.
+[Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster.
Workloads that need RDMA also need [RDMA Operations](modules/rdma/operations.md).
For cluster operations, read [Architecture](getting-started/architecture.md) and [Installation
@@ -63,7 +63,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `rdma`
- Use for: requesting RDMA endpoints for a workload; how the NIC inventory, link states and labels are discovered; aligning requests with the kubelet TopologyManager policy.
- Aliases: `RDMA`, `NIC`, `RoCE`, `link state`, `rdma.capable`
-- Guides: [RDMA Networking](modules/rdma/_index.md); keys and policy in [RDMA Operations](modules/rdma/operations.md); inventory in [Network Topology](modules/rdma/network-topology.md).
+- Guides: [RDMA Networking](modules/rdma/_index.md); keys and policy in [RDMA Operations](modules/rdma/operations.md); inventory in [Network Topology](modules/rdma/network-topology.md); endpoints in [RDMA Network Endpoints Walkthrough](walkthroughs/rdma/network-endpoints.md).
- Specs: [NIC/RDMA topology](../specs/2026-09-02-devices-nic-rdma-topology.md) · [extended resource](../specs/2026-09-20-rdma-extended-resource.md) · [NVIDIA fabric domain](../specs/2026-09-20-nvidia-fabric-domain.md) · [fabric by protocol](../specs/2026-09-22-fabric-device-by-protocol.md) · [interface count](../specs/2026-09-22-fabric-interface-count.md)
- Code: NIC and link discovery in [pkg/devicemanager/detector](../pkg/devicemanager/detector/) (`network.go`, `link.go`) · request vocabulary in [pkg/nodefeature/rdma.go](../pkg/nodefeature/rdma.go) and [pkg/nodefeature/fabric.go](../pkg/nodefeature/fabric.go)
- Related: [Devices Manager](#devices-manager) (the interfaces are entries in the same per-node inventory)
@@ -74,7 +74,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `topology`
- Use for: describing domains such as zones and racks; letting Kueue place replicas within a domain; enabling a topology source; diagnosing a Pending group.
- Aliases: `TAS`, `Topograph`, `topology source`, `zone`, `placement`
-- Guides: [Topology Management](modules/topology/_index.md); semantics in [Topology-Aware Scheduling](modules/topology/scheduling.md); enablement in [Topology-Aware Scheduling Operations](modules/topology/operations.md).
+- Guides: [Topology Management](modules/topology/_index.md); semantics in [Topology-Aware Scheduling](modules/topology/scheduling.md); enablement in [Topology-Aware Scheduling Operations](modules/topology/operations.md); placement in [Hierarchical Placement Walkthrough](walkthroughs/topology/hierarchical-placement.md).
- Specs: [topology discovery](../specs/2026-09-22-topology-discovery.md) · [model placement preference](../specs/2026-09-26-model-placement-preference.md)
- Code: [pkg/worker/controllers/worker/topology_source.go](../pkg/worker/controllers/worker/topology_source.go) and [pkg/worker/controllers/worker/node_topology.go](../pkg/worker/controllers/worker/node_topology.go)
- Related: [Devices Manager](#devices-manager) (placement constrains which pool serves a replica) · [Model Deployment](#model-deployment) (replicas can prefer nodes holding their model)
@@ -85,7 +85,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `kv-cache`
- Use for: standing up a shared inference cache; pool grants and quotas; attaching a workload to a pool.
- Aliases: `KVCacheBackend`, `KVCachePool`, `Mooncake`, `prefix cache`, `injection`
-- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
+- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [Shared Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
- Specs: [backend](../specs/2026-08-28-kv-cache-backend.md) · [pool](../specs/2026-08-28-kv-cache-pool.md) · [injection](../specs/2026-08-28-kv-cache-injection.md) · [media and scaling](../specs/2026-09-05-kv-cache-media-and-scaling.md) · [high availability](../specs/2026-09-06-kv-cache-backend-high-availability.md)
- Code: store rendering and read-back in [pkg/worker/kvcache](../pkg/worker/kvcache/) · per-engine client config in [pkg/worker/kvcache/inject](../pkg/worker/kvcache/inject/) · `kv_cache_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/)
- Related: [Model Deployment](#model-deployment) (deployments and Pods consume a pool through injection)
@@ -107,7 +107,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `model-deployment`
- Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics.
- Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica`
-- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md).
+- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); disaggregated serving in [Prefill and Decode Walkthrough](walkthroughs/model-deployment/prefill-decode.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md); multi-node serving in [Multi-Host Serving Walkthrough](walkthroughs/model-deployment/multi-host.md).
- Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [role configuration mutability](../specs/2026-10-08-model-deployment-role-mutability.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md)
- Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/)
- Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas)
@@ -239,12 +239,16 @@ A skill link names a procedure; the entry's other fields carry shared facts.
| Page | Description |
|---|---|
| [Walkthroughs](walkthroughs/_index.md) | Choose a complete example by module. |
-| [Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. |
-| [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. |
+| [Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. |
+| [RDMA Network Endpoints Walkthrough](walkthroughs/rdma/network-endpoints.md) | Request and verify RDMA endpoints and NUMA alignment. |
+| [Hierarchical Placement Walkthrough](walkthroughs/topology/hierarchical-placement.md) | Place serving replicas across zones and physical racks. |
+| [Shared Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. |
| [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md) | Cache weights and verify captured model CR output. |
| [Model Deployment Walkthroughs](walkthroughs/model-deployment/_index.md) | Choose a serving shape and prepare GPUs. |
+| [Prefill and Decode Walkthrough](walkthroughs/model-deployment/prefill-decode.md) | Deploy prefill and decode roles fronted by router. |
| [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md) | Grow DP width with TP fixed and verify inference. |
| [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md) | Verify per-rank inference and Router coverage. |
+| [Multi-Host Serving Walkthrough](walkthroughs/model-deployment/multi-host.md) | Scale models across multiple GPU nodes. |
### Start here
diff --git a/docs/modules/devices/_index.md b/docs/modules/devices/_index.md
index 19b9737ac..8febf0e87 100644
--- a/docs/modules/devices/_index.md
+++ b/docs/modules/devices/_index.md
@@ -14,6 +14,9 @@ runtime isolation method depend on the manufacturer.
physically partitioned requests. [Device Discovery](discovery.md) explains
how the per-node `Devices` record connects discovery to allocation.
+Follow [Scheduling Walkthrough](../../walkthroughs/devices/scheduling.md) for
+end-to-end device allocation and Kueue queuing.
+
## Device operations
Check a node before installation with [Preflight Operations](preflight.md). If you
diff --git a/docs/modules/devices/nvidia-mig.md b/docs/modules/devices/nvidia-mig.md
index 900cf7bb3..f91680795 100644
--- a/docs/modules/devices/nvidia-mig.md
+++ b/docs/modules/devices/nvidia-mig.md
@@ -128,6 +128,7 @@ resources:
controller shapes it into the two keys above:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: mig-instance
diff --git a/docs/modules/devices/requests.md b/docs/modules/devices/requests.md
index d43037b8a..cf3705424 100644
--- a/docs/modules/devices/requests.md
+++ b/docs/modules/devices/requests.md
@@ -143,6 +143,10 @@ one container group.*
Accepted — both app containers claim the same family:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: trainer-pod
spec:
containers:
- name: trainer
@@ -158,6 +162,10 @@ spec:
Rejected — two families in one Pod:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: mixed-family-pod
spec:
containers:
- name: a
@@ -176,6 +184,10 @@ spec:
Rejected — the same family claimed in both container groups:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: split-group-pod
spec:
initContainers:
- name: warmup
@@ -295,6 +307,10 @@ resource sits deliberately outside the accelerator families.
Accepted:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: sliced-pod
spec:
containers:
- name: main
@@ -311,6 +327,10 @@ spec:
Rejected — two containers each holding a slice:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: multi-sliced-pod
spec:
containers:
- name: a
@@ -333,6 +353,10 @@ Accepted — the claim sits on an app container while the native sidecar carries
native sidecar carries the claim:
```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: accepted-sidecar-pod
spec: # accepted
initContainers:
- name: log-shipper
@@ -346,6 +370,10 @@ spec: # accepted
limits:
nvidia.com/gpu: "1"
---
+apiVersion: v1
+kind: Pod
+metadata:
+ name: rejected-sidecar-pod
spec: # rejected
initContainers:
- name: log-shipper
@@ -459,7 +487,10 @@ into the keys above:
| `accelerator: "1"`
`acceleratorPartitionedProfile: "3g.40gb"` | one hardware partition of that profile |
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
+metadata:
+ name: partitioned-instance
spec:
type: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
resources:
diff --git a/docs/modules/devices/thead-mig.md b/docs/modules/devices/thead-mig.md
index 2d4bfa83f..449beef3f 100644
--- a/docs/modules/devices/thead-mig.md
+++ b/docs/modules/devices/thead-mig.md
@@ -146,7 +146,10 @@ Or through the `Instance` API, as with [NVIDIA's
`acceleratorPartitionedProfile`](nvidia-mig.md#requesting-a-partition):
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
+metadata:
+ name: thead-instance
spec:
type:
resources:
diff --git a/docs/modules/kv-cache/_index.md b/docs/modules/kv-cache/_index.md
index 587f26032..4a1557154 100644
--- a/docs/modules/kv-cache/_index.md
+++ b/docs/modules/kv-cache/_index.md
@@ -10,7 +10,7 @@ capacity they receive.
## Quick start
-[KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and
+[Shared Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and
deployment in order. A plain Pod can also use a binding through [KV Cache
Injection](injection.md).
@@ -26,4 +26,4 @@ tier](local-disk-tier.md) have separate operating details.
**See also** — [Model Deployment](../model-deployment/_index.md) (cache consumers) ·
[RDMA Networking](../rdma/_index.md) (transport)
-**Next** → [KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache.
+**Next** → [Shared Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache.
diff --git a/docs/modules/kv-cache/backend.md b/docs/modules/kv-cache/backend.md
index f9d1250d4..fcd363fe6 100644
--- a/docs/modules/kv-cache/backend.md
+++ b/docs/modules/kv-cache/backend.md
@@ -735,6 +735,10 @@ other images may deregister later or wait differently. The operator controls the
the endpoint.
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCacheBackend
+metadata:
+ name: mooncake-dram
spec:
connection:
managed:
@@ -763,6 +767,11 @@ own success signal does not tell you that happened.
(no Deployment, no Service, no DaemonSet) and only observes.
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCacheBackend
+metadata:
+ name: external-mooncake
+spec:
connection:
external:
endpoints:
diff --git a/docs/modules/kv-cache/leader.md b/docs/modules/kv-cache/leader.md
index a5dfeeca7..469f5b46a 100644
--- a/docs/modules/kv-cache/leader.md
+++ b/docs/modules/kv-cache/leader.md
@@ -114,12 +114,17 @@ The default `leader.electionBackend: Kubernetes` elects through a **Kubernetes L
replica count. The Lease is named `-leader` in the operator's namespace. Scale an already
elected leader by changing only its count:
-```yaml
-spec:
- connection:
- managed:
- leader:
- replicas: 3
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: KVCacheBackend
+ metadata:
+ name: shared-mooncake
+ spec:
+ connection:
+ managed:
+ leader:
+- replicas: 1
++ replicas: 3
```
`leader.memberAddressing` is optional and only selects how members find the
diff --git a/docs/modules/kv-cache/local-disk-tier.md b/docs/modules/kv-cache/local-disk-tier.md
index c68789211..e23f14faf 100644
--- a/docs/modules/kv-cache/local-disk-tier.md
+++ b/docs/modules/kv-cache/local-disk-tier.md
@@ -7,6 +7,10 @@ its own and takes its offload flags from this list's presence. The list holds on
not be attributed).
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCacheBackend
+metadata:
+ name: shared-mooncake
spec:
connection:
managed:
@@ -244,14 +248,21 @@ Deleting a `KVCacheBackend` leaves the tier directory exactly as it was. Set
`members[].localDisks[].cleanAfterDelete: true` and the operator empties it as part of the deletion, on
every node that group's `nodeSelector` picks at the moment you delete it.
-```yaml
- members:
- - medium: DRAM
- capacityPerMember: 64Gi
- localDisks:
- - path: /var/lib/kvcache
- capacity: 512Gi
- cleanAfterDelete: true
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: KVCacheBackend
+ metadata:
+ name: shared-mooncake
+ spec:
+ connection:
+ managed:
+ members:
+ - medium: DRAM
+ capacityPerMember: 500Gi
+ localDisks:
+ - path: /var/lib/kvcache
+ capacity: 4Ti
++ cleanAfterDelete: true
```
**It defaults to `false`, which is how every release before it behaved.** What is on that disk is
diff --git a/docs/modules/model-delivery/artifact.md b/docs/modules/model-delivery/artifact.md
index c5a37d98d..52d75e850 100644
--- a/docs/modules/model-delivery/artifact.md
+++ b/docs/modules/model-delivery/artifact.md
@@ -219,12 +219,18 @@ Two consequences follow:
## Referencing it from a ModelDeployment
-```yaml
-spec:
- model:
- name: qwen-7b # the served name, unchanged
- artifactRef: # a ModelArtifact in this namespace
- name: qwen-7b
+Set the reference when you create the deployment. The webhook refuses to add it later.
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: qwen-7b-serving
+ spec:
+ model:
+ name: qwen-7b # the served name, unchanged
++ artifactRef: # a ModelArtifact in this namespace
++ name: qwen-7b
```
`artifactRef` is frozen with the rest of `spec.model`: other weights are another deployment. A
@@ -397,13 +403,21 @@ timeout evicts and requeues the replica.
## Instance model volumes
-```yaml
-spec:
- additionalVolumes:
- - mountPath: /models/qwen
- model:
- artifactRef:
- name: qwen-7b
+`additionalVolumes` is immutable while the Instance runs.
+Set it at creation, or set `spec.stop: true` first and edit it while the Instance is stopped.
+See [Instance admission](../devices/admission.md).
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: Instance
+ metadata:
+ name: workspace-demo
+ spec:
++ additionalVolumes:
++ - mountPath: /models/qwen
++ model:
++ artifactRef:
++ name: qwen-7b
```
The volume is always read-only and takes no `subPath`. A claim artifact's `path` is the sub-path and
diff --git a/docs/modules/model-delivery/image-source.md b/docs/modules/model-delivery/image-source.md
index b38004743..141123b0b 100644
--- a/docs/modules/model-delivery/image-source.md
+++ b/docs/modules/model-delivery/image-source.md
@@ -16,6 +16,10 @@ image read-only for the workload; packaging and pulling it adds disk and registr
## The source and its digest contract
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelArtifact
+metadata:
+ name: qwen
spec:
source:
image:
diff --git a/docs/modules/model-deployment/_index.md b/docs/modules/model-deployment/_index.md
index ba3d51761..1a39ef736 100644
--- a/docs/modules/model-deployment/_index.md
+++ b/docs/modules/model-deployment/_index.md
@@ -14,6 +14,10 @@ its own replicas and resources.
the role contract. [Prefill and Decode](prefill-decode.md) covers split roles.
[Engine Versions](engine-versions.md) records tested version floors.
+Follow [Prefill and Decode Walkthrough](../../walkthroughs/model-deployment/prefill-decode.md) for
+disaggregated serving, and [Multi-Host Serving Walkthrough](../../walkthroughs/model-deployment/multi-host.md)
+for distributed multi-node serving.
+
## Service operations
[Status](status.md), [Metrics](metrics.md) and
diff --git a/docs/modules/model-deployment/deployment.md b/docs/modules/model-deployment/deployment.md
index 054b96634..6ebcd3d06 100644
--- a/docs/modules/model-deployment/deployment.md
+++ b/docs/modules/model-deployment/deployment.md
@@ -308,6 +308,10 @@ Each Pod has its own volume. Only the main container mounts it.
Set it in the role configuration:
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: qwen-chat
spec:
roles:
- name: server
diff --git a/docs/modules/model-deployment/elastic-ep.md b/docs/modules/model-deployment/elastic-ep.md
index a94d7f3e6..4d205b27d 100644
--- a/docs/modules/model-deployment/elastic-ep.md
+++ b/docs/modules/model-deployment/elastic-ep.md
@@ -115,7 +115,7 @@ spec:
replicas: 1
size: 1
instanceType: elastic-gpu
- image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
+ image: gpustack/runner:cuda13.0-vllm0.29.0
elasticEp:
width: 2
ports:
diff --git a/docs/modules/model-deployment/prefill-decode.md b/docs/modules/model-deployment/prefill-decode.md
index cdcfe4194..116f57e10 100644
--- a/docs/modules/model-deployment/prefill-decode.md
+++ b/docs/modules/model-deployment/prefill-decode.md
@@ -141,14 +141,18 @@ has reserved quota, so `Prefill` still never starts without `Decode`.
Splitting `instanceType`s buys different hardware; the isolation it used to buy, every replica has
by default.
-**With a shared pool**, the same object plus one block:
-
-```yaml
-spec:
- # ... everything above, unchanged ...
- kvCache:
- poolRef:
- name: team-a-dram # a KVCachePoolBinding in THIS namespace
+**With a shared pool**, the same object plus one block. `spec.kvCache` is frozen after creation, so declare it in the first apply:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: qwen-pd
+ namespace: team-a
+ spec:
++ kvCache:
++ poolRef:
++ name: team-a-dram # a KVCachePoolBinding in THIS namespace
```
## The router block
@@ -231,10 +235,15 @@ there is no field for either, and neither is reachable through `extraArgs`.
The point-to-point leg renders Mooncake's `tcp` when the API field is unset:
-```yaml
-spec:
- kvTransfer:
- protocol: RDMA # unset renders "tcp"
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: qwen-pd
+ namespace: team-a
+ spec:
++ kvTransfer:
++ protocol: RDMA # unset renders "tcp"
```
The value is a property of **one link**, so it is deployment-wide: a per-role field could only
diff --git a/docs/modules/model-deployment/routing.md b/docs/modules/model-deployment/routing.md
index 61338e83f..127e1f4fe 100644
--- a/docs/modules/model-deployment/routing.md
+++ b/docs/modules/model-deployment/routing.md
@@ -83,13 +83,17 @@ How any of the three spreads concurrent traffic is not measured.
**Add `--policy round_robin` to `spec.router.extraArgs` on `vllm-router` or `sglang-gateway`** to
send each request to the next replica in turn, whatever it shares with earlier ones:
-```yaml
-spec:
- router:
- name: vllm-router # or sglang-gateway
- extraArgs:
- - --policy
- - round_robin
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: round-robin-serving
+ spec:
+ router:
+ name: vllm-router # or sglang-gateway
++ extraArgs:
++ - --policy
++ - round_robin
```
`--policy` is in neither router's [refused list](prefill-decode.md#the-router-block), so admission
diff --git a/docs/modules/rdma/_index.md b/docs/modules/rdma/_index.md
index 84c6f2fd8..f63efb1d7 100644
--- a/docs/modules/rdma/_index.md
+++ b/docs/modules/rdma/_index.md
@@ -13,6 +13,9 @@ count; kubelet decides which endpoints a container receives.
[RDMA Operations](operations.md) covers the resource keys, endpoint counts and kubelet
TopologyManager policy. It also explains why Kueue admission alone does not reserve an interface.
+Follow [RDMA Network Endpoints Walkthrough](../../walkthroughs/rdma/network-endpoints.md) to
+request and verify endpoints and NUMA alignment.
+
## Interface discovery
[Network Topology](network-topology.md) describes the interface inventory, link
diff --git a/docs/modules/rdma/operations.md b/docs/modules/rdma/operations.md
index cb35457cb..84cdcd808 100644
--- a/docs/modules/rdma/operations.md
+++ b/docs/modules/rdma/operations.md
@@ -271,8 +271,7 @@ that moves KV over EFA. A role naming no image gets the runner image, whatever i
Derive it from the runner image the deployment would otherwise run:
```dockerfile
-# Pin the base by digest.
-FROM gpustack/runner:cuda13.0-vllm0.29.0@sha256:
+FROM gpustack/runner:cuda13.0-vllm0.29.0
SHELL ["/bin/bash", "-eo", "pipefail", "-c"]
ARG AWS_EFA_VERSION=1.44.0
RUN curl --retry 3 --retry-connrefused -fL "https://efa-installer.amazonaws.com/aws-efa-installer-${AWS_EFA_VERSION}.tar.gz" | tar -zx -C /tmp \
diff --git a/docs/modules/topology/_index.md b/docs/modules/topology/_index.md
index 4d0960cb7..85b5771b9 100644
--- a/docs/modules/topology/_index.md
+++ b/docs/modules/topology/_index.md
@@ -13,6 +13,9 @@ those domains.
[Topology-Aware Scheduling](scheduling.md) explains topology
profiles, Kueue Topologies and how a replica asks to stay in one domain.
+Follow [Hierarchical Placement Walkthrough](../../walkthroughs/topology/hierarchical-placement.md) to
+place serving replicas across zones and physical racks.
+
## Topology sources
[Topology-Aware Scheduling Operations](operations.md) covers inventory
diff --git a/docs/modules/topology/operations.md b/docs/modules/topology/operations.md
index e3cb2d565..ba62f65ea 100644
--- a/docs/modules/topology/operations.md
+++ b/docs/modules/topology/operations.md
@@ -218,6 +218,10 @@ three durations must be positive.
Set `requiredLevel` on a role to require each replica's Pod group to fit in one domain at that level:
```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: zoned-deployment
spec:
roles:
- name: decode
diff --git a/docs/walkthroughs/_index.md b/docs/walkthroughs/_index.md
index 467cb2796..b6ea96f6f 100644
--- a/docs/walkthroughs/_index.md
+++ b/docs/walkthroughs/_index.md
@@ -12,12 +12,16 @@ Each module has its own directory. Configuration contracts remain in the capabil
| Module | Walkthrough | Task |
|---|---|---|
-| Devices | [Device Scheduling](devices/scheduling.md) | Follow detection, queues, admission, and accelerator allocation. |
+| Devices Manager | [Scheduling](devices/scheduling.md) | Follow detection, queues, admission, and accelerator allocation. |
+| RDMA Manager | [RDMA Network Endpoints](rdma/network-endpoints.md) | Request and verify shared and exclusive RDMA endpoints with NUMA alignment. |
+| Topology Aware | [Hierarchical Placement](topology/hierarchical-placement.md) | Publish zone and rack hierarchies and constrain replicas to a topology domain. |
| KV Cache | [Shared Cache](kv-cache/shared-cache.md) | Create a backend, pool, namespace grant, and serving workload. |
| Model Delivery | [Model Prefetch](model-delivery/prefetch.md) | Cache and verify model weights before inference. |
| Model Deployment | [Preparation and choices](model-deployment/_index.md) | Choose the serving shape and prepare GPU resources. |
+| Model Deployment | [Prefill and Decode](model-deployment/prefill-decode.md) | Disaggregate prefill and decode roles fronted by a router. |
| Model Deployment | [Elastic EP](model-deployment/elastic-ep.md) | Grow DP width from 2 to 4 with TP fixed at 2. |
| Model Deployment | [External DP](model-deployment/external-dp.md) | Serve fixed DP ranks and verify Router distribution. |
+| Model Deployment | [Multi-Host](model-deployment/multi-host.md) | Deploy distributed model instances across multiple GPU hosts. |
## Read the evidence
@@ -31,4 +35,4 @@ Their validation records state software identities and limits.
**See also** — [Architecture](../getting-started/architecture.md) ·
[Model Deployment](../modules/model-deployment/_index.md)
-**Next** → [Device Scheduling Walkthrough](devices/scheduling.md)
+**Next** → [Scheduling Walkthrough](devices/scheduling.md)
diff --git a/docs/walkthroughs/devices/scheduling.md b/docs/walkthroughs/devices/scheduling.md
index 9581afaef..f69a4f890 100644
--- a/docs/walkthroughs/devices/scheduling.md
+++ b/docs/walkthroughs/devices/scheduling.md
@@ -1,4 +1,4 @@
-# Device Scheduling Walkthrough
+# Scheduling Walkthrough
This page records a real session: every `kubectl` invocation and its real output, objects as YAML
trimmed to `metadata.labels` / `spec` / `status`, and a before / after `kubectl get instancetypes`
@@ -28,6 +28,7 @@ walkthrough](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configura
- [4. Managing a custom InstanceType](#4-managing-a-custom-instancetype)
- [5. Enabling CPU-manufacturer awareness](#5-enabling-cpu-manufacturer-awareness)
- [6. Pinning an Instance to a node and adding volumes](#6-pinning-an-instance-to-a-node-and-adding-volumes)
+- [Troubleshooting](#troubleshooting)
## The cluster
@@ -97,6 +98,7 @@ Labeled by two NodeFeatures: `node-a10g-gpustack-worker` (the `general.*` CPU ke
`node-a10g-gpustack-device-manager` (the `acceleratable.*` device keys):
```yaml
+apiVersion: v1
kind: Node
metadata:
name: node-a10g
@@ -125,6 +127,7 @@ Cluster-scoped, named after the node: the worker stamps `gpustack.ai/managed` +
Device Manager the accelerator key and the `.status` ledger:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Devices
metadata:
name: node-a10g
@@ -152,6 +155,7 @@ The finest, setting-independent grain, `gpustack--${gKey}[--${aKey}]-${os}-${arc
device (`acceleratable.`) keys, and pins nodes via `spec.nodeLabels`:
```yaml
+apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: gpustack--amd-epyc-7r32--nvidia-a10g-linux-amd64-1d
@@ -178,6 +182,7 @@ A **non-accelerated** (CPU) flavor carries `feature.gpustack.ai/acceleratable=fa
node's CPU-core count:
```yaml
+apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
name: gpustack--amd-epyc-7r13-linux-amd64-16c
@@ -205,6 +210,7 @@ per-accelerator `AdmissionCheck`. With awareness off its labels carry **no** `ge
aggregates the A10G across every CPU:
```yaml
+apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: gpustack--nvidia-a10g-linux-amd64
@@ -235,6 +241,7 @@ the fronting `queue-entrance`), `spec` enriched from the matching flavor, `statu
four-view:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: InstanceType
metadata:
name: gpustack--nvidia-a10g-linux-amd64
@@ -294,6 +301,7 @@ An os/arch-agnostic catalog view aggregated read-only from the flavors, with **n
`metadata.labels`** — its grouping identity lives in `spec`:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: InstanceTypeFlavor
metadata:
name: gpustack--nvidia-a10g
@@ -324,6 +332,27 @@ gpustack--nvidia-a10g-linux-amd64 gpustack-fnv64-c4680bb149644f1c 8/64Gi
gpustack--nvidia-tesla-t4-linux-amd64 gpustack-fnv64-6b371caa2da0b799 8/32Gi/100Gi 4/5 40/50 100/500 0/0 0/0 Active
```
+The worker writes one NodeFeature per node, in `gpustack-system`, named `-gpustack-worker`
+(see [Device Scheduling](../../modules/devices/scheduling.md)).
+This is the one for `node-a10g`, which carries the managed label and the CPU keys from the Node above:
+
+```yaml
+apiVersion: nfd.k8s-sigs.io/v1alpha1
+kind: NodeFeature
+metadata:
+ name: node-a10g-gpustack-worker
+ namespace: gpustack-system
+ labels:
+ nfd.node.kubernetes.io/node-name: node-a10g
+ app.kubernetes.io/part-of: gpustack-operator-worker
+spec:
+ labels:
+ gpustack.ai/managed: "true"
+ general.feature.gpustack.ai/amd: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32.count: "4"
+```
+
Flip that label off:
```console
@@ -332,6 +361,24 @@ $ kubectl -n gpustack-system patch nodefeature node-a10g-gpustack-worker \
nodefeature.nfd.k8s-sigs.io/node-a10g-gpustack-worker patched
```
+```diff
+ apiVersion: nfd.k8s-sigs.io/v1alpha1
+ kind: NodeFeature
+ metadata:
+ name: node-a10g-gpustack-worker
+ namespace: gpustack-system
+ labels:
+ nfd.node.kubernetes.io/node-name: node-a10g
+ app.kubernetes.io/part-of: gpustack-operator-worker
+ spec:
+ labels:
+- gpustack.ai/managed: "true"
++ gpustack.ai/managed: "false"
+ general.feature.gpustack.ai/amd: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32.count: "4"
+```
+
**After** — NFD propagates it; the operator retires the now-nodeless A10G flavor:
```console
@@ -358,6 +405,24 @@ gpustack--nvidia-a10g-linux-amd64 gpustack-fnv64-c4680bb149644f1c 8/64Gi
gpustack--nvidia-tesla-t4-linux-amd64 gpustack-fnv64-6b371caa2da0b799 8/32Gi/100Gi 4/5 40/50 100/500 0/0 0/0 Active
```
+```diff
+ apiVersion: nfd.k8s-sigs.io/v1alpha1
+ kind: NodeFeature
+ metadata:
+ name: node-a10g-gpustack-worker
+ namespace: gpustack-system
+ labels:
+ nfd.node.kubernetes.io/node-name: node-a10g
+ app.kubernetes.io/part-of: gpustack-operator-worker
+ spec:
+ labels:
+- gpustack.ai/managed: "false"
++ gpustack.ai/managed: "true"
+ general.feature.gpustack.ai/amd: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32: "true"
+ general.feature.gpustack.ai/amd-epyc-7r32.count: "4"
+```
+
---
## 3. Requesting a logical sliced GPU
@@ -367,6 +432,7 @@ fractional-accelerator workloads. Request 20 % of an accelerator's VRAM with
`acceleratorSlicedMemoryPercentage`:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: sliced-demo
@@ -430,6 +496,7 @@ another size, an admin authors an InstanceType referencing a catalog flavor by i
with a unit spec of its own:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: InstanceType
metadata:
name: a10g-12c128g
@@ -463,6 +530,7 @@ accelerator, two views of it.
Deploy an Instance onto the custom type, whole accelerator:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: custom-demo
@@ -557,7 +625,22 @@ Re-purpose the `custom-demo` Instance stopped in section 4: a drained Instance c
```console
$ kubectl -n default patch instance custom-demo --type=merge \
-p '{"spec":{"type":"gpustack--amd-epyc-7r32--nvidia-a10g-linux-amd64","stop":false}}'
+```
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: Instance
+ metadata:
+ name: custom-demo
+ namespace: default
+ spec:
+- type: a10g-12c128g
+- stop: true
++ type: gpustack--amd-epyc-7r32--nvidia-a10g-linux-amd64
++ stop: false
+```
+
+```console
$ kubectl get instancetypes
NAME ENTRANCE UNIT(CPU/RAM)/STORAGE ACCELERATOR(EX/SH/SL/PT) CPU PHASE
gpustack--amd-epyc-7r32--nvidia-a10g-linux-amd64 gpustack-fnv64-029fd9550e0c70bd 8/64Gi/100Gi 0/0 0/0 0/0 0/0 0/0 Active
@@ -567,9 +650,11 @@ gpustack--nvidia-a10g-linux-amd64 gpustack-fnv64-c4680bb149644f
`custom-demo` goes `Ready` on the aware pool, which drops to `0/0 0/0 0/0`, and the collapsed
`gpustack--nvidia-a10g-linux-amd64` also drops: one accelerator, two consistent views.
-The aware ClusterQueue's labels now **carry the CPU key**, the difference from section 1:
+The re-derive also creates an aware ClusterQueue. It is a second object beside the collapsed
+`gpustack--nvidia-a10g-linux-amd64` queue from section 1, and its labels **carry the CPU key**:
```yaml
+apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
name: gpustack--amd-epyc-7r32--nvidia-a10g-linux-amd64
@@ -614,6 +699,7 @@ one, and `spec.additionalVolumes` mounts paths beside the workspace — a shared
key, a directory on the node itself:
```yaml
+apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: pinned-demo
@@ -689,11 +775,21 @@ source names an object in the Instance's own namespace and may always be mounted
there has. Namespaces stay the tenancy boundary: put Instances whose authors should not read each
other's Secrets in their own.
+## Troubleshooting
+
+| Symptom | Root cause / Condition | Check & Mitigation |
+|---|---|---|
+| An Instance does not become Ready and its Workload stays pending | Quota, flavor labels or a node pin the scheduler cannot satisfy | Read `kubectl -n get instance -o jsonpath='{.status.phaseMessage}'`, then `kubectl -n get workloads`. Compare the type's remaining counts in `kubectl get instancetypes`. See [section 6](#6-pinning-an-instance-to-a-node-and-adding-volumes) for pins |
+| Pod rejected: `a Pod may request only one accelerator family, found [...]` | Two accelerator families in one Pod | Request one family. See [rule 1](../../modules/devices/requests.md#rule-1--one-family-in-exactly-one-container-group) |
+| Pod rejected: `at most one container may request a slicing family, found N` | More than one container requests a slice or a partition | Keep one claiming container. See [rule 6](../../modules/devices/requests.md#rule-6--at-most-one-container-may-request-a-slicing-family) |
+| Pod rejected: `a restartable init container (a native sidecar) may not request an accelerator` | A native sidecar carries the accelerator request | Move the request to an app container. See [rule 7](../../modules/devices/requests.md#rule-7--a-restartable-init-container-may-not-request-an-accelerator) |
+| A node's accelerators are missing from the `InstanceType` pool | The node's NodeFeature carries `gpustack.ai/managed: "false"`, or `node-management-manual` is `true` and no one labeled the node | Run `kubectl -n gpustack-system get nodefeature -gpustack-worker -o yaml` and read `spec.labels`. See [section 2](#2-removing-a-node-from-management) |
+| Instance rejected: `privileged mode is not allowed: enable the "instance-privileged-allowed" setting to allow it` | The `instance-privileged-allowed` setting is `false` | Allow it only if the node boundary is acceptable: `kubectl -n gpustack-system patch setting instance-privileged-allowed --type merge -p '{"spec":{"value":"true"}}'`. See [Settings](../../reference/settings.md) |
+
---
**See also** — [Accelerator Requests](../../modules/devices/requests.md) (the contract behind step 3) ·
[NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node)
(hardware partitioning) · [Settings](../../reference/settings.md#online-adjustable-settings) (the two gates)
-**Next** → [NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md) — hardware partitioning, from enabling
-the mode to reclaiming the instance.
+**Next** → [RDMA Network Endpoints Walkthrough](../rdma/network-endpoints.md) — request and verify RDMA endpoints with NUMA alignment.
diff --git a/docs/walkthroughs/kv-cache/shared-cache.md b/docs/walkthroughs/kv-cache/shared-cache.md
index 42fc72b94..8b3eb4311 100644
--- a/docs/walkthroughs/kv-cache/shared-cache.md
+++ b/docs/walkthroughs/kv-cache/shared-cache.md
@@ -1,4 +1,4 @@
-# KV Cache Walkthrough
+# Shared Cache Walkthrough
Create a store, a pool, a namespace grant and a workload in that order.
@@ -12,8 +12,10 @@ name in the manifests below, then apply them in that order.
- [Step 2: the pool and the grant](#step-2-the-pool-and-the-grant)
- [Step 3: the workload](#step-3-the-workload)
- [Step 4: high availability](#step-4-high-availability)
-- [Step 5: post-failover state](#step-5-post-failover-state)
-- [Failure modes](#failure-modes)
+- [Step 5: local disk tier](#step-5-local-disk-tier)
+- [Step 6: post-failover state](#step-6-post-failover-state)
+- [Step 7: a second team and a second domain](#step-7-a-second-team-and-a-second-domain)
+- [Troubleshooting](#troubleshooting)
## Creation order
@@ -34,6 +36,8 @@ same-namespace reference, so the name it accepts is a Binding's.
## Step 1: the store
+Replace the `kubernetes.io/os: linux` selector with the labels of the nodes that should contribute memory.
+
```yaml
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
@@ -46,7 +50,7 @@ spec:
leader: {}
members:
- nodeSelector:
- kubernetes.io/os: linux # replace with the nodes that should contribute
+ kubernetes.io/os: linux
medium: DRAM
capacityPerMember: 8Gi
```
@@ -150,7 +154,7 @@ spec:
replicas: 2
instanceType:
resources:
- accelerator: 1
+ accelerator: "1"
```
`kvCache` is optional. Omit it and the deployment runs with no shared cache at all, which is the
@@ -177,12 +181,23 @@ $ kubectl -n team-a get md qwen-chat -o jsonpath='{.status.conditions[?(@.type==
Everything so far runs one leader process that already holds a Kubernetes Lease. An update or a node
failure takes the store's metadata with it. Add standbys with one edit:
-```yaml
-spec:
- connection:
- managed:
- leader:
- replicas: 3
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: KVCacheBackend
+ metadata:
+ name: mooncake-dram
+ spec:
+ type: Mooncake
+ connection:
+ managed:
+- leader: {}
++ leader:
++ replicas: 3
+ members:
+ - nodeSelector:
+ kubernetes.io/os: linux
+ medium: DRAM
+ capacityPerMember: 8Gi
```
**The healthy steady state now reads `3 desired / 1 ready`, and that is not a broken Deployment.**
@@ -209,7 +224,47 @@ These conditions apply at one replica too; the phase does not summarize them:
`RolloutComplete` exists because this workload disables the deployment deadline that would normally
answer it: that deadline requires every replica to be available, and only one ever is here.
-## Step 5: post-failover state
+## Step 5: local disk tier
+
+A running member group cannot gain a disk tier. Admission refuses the edit, because the members
+would have to restart to mount the directory. Declare the tier when you create a backend. This diff
+turns Step 1's manifest into a second backend that selects the nodes carrying the disk:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: KVCacheBackend
+ metadata:
+- name: mooncake-dram
++ name: mooncake-disk
+ spec:
+ type: Mooncake
+ connection:
+ managed:
+ leader: {}
+ members:
+ - nodeSelector:
+- kubernetes.io/os: linux
++ kvcache: "true"
+ medium: DRAM
+ capacityPerMember: 8Gi
++ localDisks:
++ - path: /var/lib/kvcache
++ capacity: 100Gi
+```
+
+Create the directory on every selected node before you apply the backend. The published store image
+runs as uid 65532, so give the directory to that uid. The uid depends on the image; see
+[Directory requirements](../../modules/kv-cache/local-disk-tier.md#directory-requirements).
+
+```console
+$ install -d -o 65532 -g 0 -m 0750 /var/lib/kvcache
+```
+
+A pool names exactly one backend and cannot be re-pointed. Serving workloads from `mooncake-disk`
+therefore takes its own `KVCachePool` and Binding. The tier writes in buckets, so a small cache can
+show an empty tier while it works; see [Bucket writes](../../modules/kv-cache/local-disk-tier.md#bucket-writes).
+
+## Step 6: post-failover state
**A failover keeps nothing in memory.** A standby holds no data, so the replica that takes over knows
none of the objects held in member memory and rebuilds from member remounts alone; a single leader
@@ -220,26 +275,112 @@ misses afterwards. Plan for a cold cache after every failover and every leader r
restoring a snapshot can make the cache serve another key's bytes instead of a miss.
[High availability](../../modules/kv-cache/leader.md#high-availability) says why.
-## Failure modes
+## Step 7: a second team and a second domain
+
+Each Binding is one ceiling and one reuse domain. To give a second team cache capacity, create a
+Binding in its namespace:
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCachePoolBinding
+metadata:
+ name: team-b
+ namespace: team-b
+spec:
+ poolRef:
+ name: shared-dram
+ quota:
+ ceiling: 8Gi
+ domain:
+ name: llama-8b-v1
+ blockSize: 16
+ dtype: bfloat16
+```
+
+A namespace that runs a different dtype needs a second Binding, because a domain is immutable and
+holds one dtype. Mixing dtypes in one domain can return wrong blocks silently. The Binding below
+serves an `fp8_e4m3` deployment in `team-a`. Replace the instance type as in Step 3.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCachePoolBinding
+metadata:
+ name: qwen-fp8
+ namespace: team-a
+spec:
+ poolRef:
+ name: shared-dram
+ quota:
+ ceiling: 4Gi
+ domain:
+ name: qwen-fp8-v1
+ blockSize: 16
+ dtype: fp8_e4m3
+---
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: qwen-chat-fp8
+ namespace: team-a
+spec:
+ model:
+ name: Qwen/Qwen2.5-7B-Instruct
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ kvCache:
+ poolRef:
+ name: qwen-fp8
+ roles:
+ - name: server
+ replicas: 1
+ instanceType:
+ resources:
+ accelerator: "1"
+```
+
+The operator renders `--kv-cache-dtype fp8_e4m3` from the Binding onto every role. A role that sets
+that flag itself is refused. See [Engine dtype](../../modules/kv-cache/pool.md#engine-dtype).
+
+**The three ceilings ask for 20Gi against a pool `quota.total` of 16Gi.** The pool reports
+`QuotaWithinTotal=False` with reason `Oversubscribed`. That is not a fault: a ceiling is an ask, and
+the store recuts each grant against the capacity it can serve, in proportion to what each Binding
+asked. `status.effectiveQuota` on each Binding shows the grant. See
+[Ceiling and grant](../../modules/kv-cache/pool.md#ceiling-and-grant).
-**A published upstream store image, under high availability.** No published `kvcacheai/mooncake`
-image carries a leadership backend: the leader answers `UNAVAILABLE_IN_CURRENT_MODE` and runs as a
-permanent standby, and members answer `Invalid HA backend entry` and CrashLoopBackOff. `spec.image`
-and every `members[].image` need a build that has one. Leaving them unset is the simplest way.
+**A domain separates cached blocks. It does not wall off a tenant.** A Binding is not an enforcement
+boundary: a workload that knows another domain's name can still read and write that domain. Isolation
+of reuse needs a master that keeps a tenant ledger (the Step 1 default) and an engine that forwards
+the injected tenant. See [Limitations](../../modules/kv-cache/pool.md#limitations) and
+[Tenant compatibility](../../modules/kv-cache/injection.md#tenant-compatibility).
-**Expecting a failover or a restart to keep the cache.** Neither does: the new leader knows none of
-the objects held in member memory, so every lookup for one misses until an engine writes it again.
+## Troubleshooting
-**Reading `3/1` as a fault.** It is the designed steady state under high availability, and the one
-reading on this page most likely to be escalated as an outage.
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| `KVCacheBackend` stuck in `Provisioning` or `Error` | `LeaderAvailable=False` | Leader Deployment unready, image missing election backend, or external endpoint address unreachable |
+| `KVCacheBackend` shows `Degraded` | `MembersMounted` shortfall (`SegmentsShort`, `NoSegments`) | DaemonSet member Pods failing to mount or allocate memory/disk segments; check node capacity and daemon logs |
+| `KVCachePool` shows `Error` | `BackendResolved=False` / `CapacityAllocatable=False` | Referenced backend does not exist or has zero allocatable capacity |
+| `KVCachePool` `QuotaLedgerAvailable=False` | `MultiTenancyDisabled` | Backend explicitly set `multiTenancy: false`; ceiling quotas are intentionally not enforced |
+| `KVCachePoolBinding` `effectiveQuota: 0` | `QuotaGranted=False/ZeroGranted` | The master grants this domain nothing. Read the pool's `CapacityAllocatable`: `NothingToAllocate` means no member has mounted yet, and a restarted master can take roughly 30 s to remount its segments. Otherwise a proportional recut left this domain no share |
+| Pool reports oversubscribed | `QuotaWithinTotal=False/Oversubscribed` | The Bindings' ceilings sum past the pool's `quota.total`. This is not a fault: grants are recut in proportion and shown in each `status.effectiveQuota`. Lower a ceiling or raise `quota.total` for full grants |
+| Binding refused at apply with `Duplicate` on `spec.domain.name` | Domain already registered on the same master | Another Binding holds the name; an omitted name is `default`. Give each Binding a unique `domain.name` |
+| Two Bindings report one domain | `DomainExclusive=False/DomainClaimedByMultipleBindings` | Two creates raced past admission; delete one of them |
+| Member Pod stuck with `FailedMount ... hostPath type check failed` | Pod event | The tier directory does not exist on that node; create it as in Step 5 |
+| Member Pod runs but never becomes Ready | `MembersMounted=False` (`NoSegments`, or `MemberCrashLooping` if the container exits), or a `CAPACITY` below the expected figure | The directory exists but the image's user cannot write it; the container log says `no write permission on directory`. Run `install -d -o 65532 -g 0 -m 0750 ` for the default image. See [Directory requirements](../../modules/kv-cache/local-disk-tier.md#directory-requirements) |
+| Member init container restarts | `MembersMounted=False/MemberInitCrashLooping` | The directory-survey init container restarted; the condition message gives the restart count, exit code and termination message. The survey runs `sh -c`, so the member image needs a shell. An unwritable directory does not cause this |
+| Local disk tier reports preexisting content | `TierWasEmpty=False/PreexistingContent` | The directory held files from an earlier backend. Nothing was removed; empty it yourself. `cleanAfterDelete: true` empties it only when this backend is deleted |
+| HA Leader shows `3 desired / 1 ready` | Expected steady state | Only elected active leader is Ready; 2 standby leader replicas remain unready until failover |
+| Deletion of `KVCachePoolBinding` blocked | `Releasable=False/HeldByWorkloads` | Active ModelDeployments or Pods reference this binding; delete client workloads first |
---
**See also** — [KV Cache Backend](../../modules/kv-cache/backend.md) (every field of the store, and what status reports) ·
[KV Cache Leader](../../modules/kv-cache/leader.md) (the election, why there is no snapshot, and the member addressing choice in full) ·
[KV Cache Pool](../../modules/kv-cache/pool.md) (quota, domains and what a full quota does) ·
+[KV Cache Local Disk Tier](../../modules/kv-cache/local-disk-tier.md) (directory, bucket and cleanup rules) ·
[Model Deployment](../../modules/model-deployment/deployment.md) (roles, prefill/decode, rollout) ·
[Model Deployment Prefill and Decode](../../modules/model-deployment/prefill-decode.md) (router, transfer) ·
[KV Cache Injection](../../modules/kv-cache/injection.md) (what a Pod actually receives)
-**Next** → [KV Cache Pool](../../modules/kv-cache/pool.md) — the quota this walkthrough set once and did not explain.
+**Next** → [Model Prefetch Walkthrough](../model-delivery/prefetch.md) — cache weights on GPU nodes before serving.
diff --git a/docs/walkthroughs/model-delivery/prefetch.md b/docs/walkthroughs/model-delivery/prefetch.md
index e2c3a7db3..375b41c1e 100644
--- a/docs/walkthroughs/model-delivery/prefetch.md
+++ b/docs/walkthroughs/model-delivery/prefetch.md
@@ -10,7 +10,9 @@ The output below was captured with node model delivery enabled.
- [Step 1: create the artifact and prefetch](#step-1-create-the-artifact-and-prefetch)
- [Step 2: verify the cached model](#step-2-verify-the-cached-model)
- [Step 3: start a serving workload](#step-3-start-a-serving-workload)
-- [Step 4: release the cache](#step-4-release-the-cache)
+- [Step 4: pin weights against eviction](#step-4-pin-weights-against-eviction)
+- [Step 5: release the cache](#step-5-release-the-cache)
+- [Troubleshooting](#troubleshooting)
## Prerequisites
@@ -73,7 +75,6 @@ spec:
- name: rtx6000-models
quota:
bytes: 64Gi
- allowPinned: true
---
apiVersion: worker.gpustack.ai/v1
kind: ModelPrefetch
@@ -90,8 +91,6 @@ spec:
matchLabels:
kubernetes.io/hostname: gpu-node
minReady: 1
- retention:
- pinned: true
```
## Step 2: verify the cached model
@@ -141,7 +140,56 @@ Continue with [Elastic EP](../model-deployment/elastic-ep.md) to grow one collec
or [External DP](../model-deployment/external-dp.md) to route requests across fixed DP ranks.
Use the namespace and artifact created here in either manifest.
-## Step 4: release the cache
+## Step 4: pin weights against eviction
+
+The node cache removes unreferenced weights, oldest first, when usage passes the high watermark.
+Pinning keeps a digest out of that collection. Admission refuses `retention.pinned` on a prefetch
+whose `ModelStoreBinding` does not set `allowPinned`, so the administrator grants the right first.
+
+Edit `model-delivery.yaml`. The grant changes as follows:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelStoreBinding
+ metadata:
+ name: rtx6000-models
+ namespace: gpustack-elastic
+ spec:
+ storeRefs:
+ - name: rtx6000-models
+ quota:
+ bytes: 64Gi
++ allowPinned: true
+```
+
+The prefetch then asks for the pin:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelPrefetch
+ metadata:
+ name: deepseek-v2-lite-chat
+ namespace: gpustack-elastic
+ spec:
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ bindingRef:
+ name: rtx6000-models
+ placement:
+ nodeSelector:
+ matchLabels:
+ kubernetes.io/hostname: gpu-node
+ minReady: 1
++ retention:
++ pinned: true
+```
+
+Run `kubectl apply -f model-delivery.yaml` again. The file lists the grant before the prefetch, so
+the grant is in place when the prefetch is admitted. A pinned digest is never a collection candidate
+on the target nodes, though it still counts toward the store's usage. See
+[Pinning and expiry](../../modules/model-delivery/prefetch.md#pinning-and-expiry).
+
+## Step 5: release the cache
Keep the prefetch while a serving application needs the cached weights.
After deleting those consumers, remove the dedicated resources:
@@ -156,6 +204,23 @@ kubectl delete modelstore rtx6000-models
Cache collection follows the [retention rules](../../modules/model-delivery/prefetch.md).
Deleting these CRs does not delete the Kubernetes cluster or its disks.
+## Troubleshooting
+
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| `ModelArtifact` resolution failed | `Resolved=False/AccessDenied` | Private or gated repository; check namespace Secret token and inspect `InvalidToken` event |
+| `ModelArtifact` resolution failed | `Resolved=False/RevisionNotFound` | Branch, tag, or commit hash misspelled (ModelScope does not default silently to master) |
+| `ModelArtifact` resolution failed | `Resolved=False/SourceUnavailable` | Hub endpoint unreachable, large directory enumeration timeout, or 5xx network error |
+| `ModelArtifact` resolution failed | `Resolved=False/EmptyManifest` | Allow/ignore pattern filters excluded all files; review the glob patterns |
+| Model deployment blocked at `Starting` | `WeightsReady=False/NodeDeliveryUnavailable` | CSI driver `model.csi.gpustack.ai` missing or `modelManager.enabled: false` in Helm values |
+| Deployment blocked with pattern filters | `WeightsReady=False/FilterNeedsNodeDelivery` | Pattern filtering requires `Node` delivery mode; set `model-artifact-delivery-mode: Node` |
+| Pod stuck in `ContainerCreating` | `WeightsReady=False/Materializing` | Node plugin actively downloading; check Pod `FailedMount` event for downloaded byte count |
+| Weight download failed on node | `WeightsReady=False/MaterializationFailed` | Check `kubectl get nodemodelstore -o yaml` for digest failure `reason`, `message`, and `retryTime` |
+| `ModelPrefetch` fails to reach `Available` | `Degraded=True/NodeFailed` | One or more target nodes failed download; check node disk capacity and logs |
+| `ModelPrefetch` shows fewer ready nodes than `minReady` | `Lapsed=True/RetentionTTLExpired` | Node copies went unused past `retention.ttlAfterLastUse`, left `readyNodes` and released any pin, and nothing re-warms them; raise or remove the TTL |
+| New `ModelPrefetch` rejected at admission | Over quota or unauthorized | Prefetch exceeds `ModelStoreBinding` `quota.bytes`, sets `retention.pinned` without `allowPinned: true`, or targets undeclared store |
+| Peer sync downloads continuously from Hub | `modelManager.port: 0` or peer sync disabled | Verify setting `model-store-peer-sync: true` and node `spec.peerSyncEnabled: true` |
+
---
**See also** — [Model Delivery](../../modules/model-delivery/_index.md) ·
diff --git a/docs/walkthroughs/model-deployment/_index.md b/docs/walkthroughs/model-deployment/_index.md
index 494706680..5bbda1da8 100644
--- a/docs/walkthroughs/model-deployment/_index.md
+++ b/docs/walkthroughs/model-deployment/_index.md
@@ -1,7 +1,8 @@
# Model Deployment Walkthroughs
Choose a serving shape, prepare its model, then follow the matching walkthrough.
-Both examples use the official GPUStack runner and retain captured CR output and inference results.
+The Elastic EP and External DP examples use the official GPUStack runner and retain captured CR output and inference results.
+The Prefill and Decode and the Multi-Host pages are configuration examples. They record no captured run.
## Contents
@@ -12,10 +13,12 @@ Both examples use the official GPUStack runner and retain captured CR output and
## Choose an example
-| Walkthrough | Serving shape | What was measured |
+| Walkthrough | Serving shape | What it covers |
|---|---|---|
+| [Prefill and Decode](prefill-decode.md) | Disaggregated roles fronted by router | Not measured here. Separate Prefill and Decode roles, `llm-d-router` routing, joint admission, and KV cache transfer. |
| [Elastic EP](elastic-ep.md) | One Internal LB instance on Ray | TP2, DP width 2→4, EP world 4→8; ordinary and streaming requests before and after expansion. |
| [External DP](external-dp.md) | Two fixed DP ranks using multiprocessing | TP2, DP2, EP world4; direct requests to each rank and Router distribution across both ranks. |
+| [Multi-Host](multi-host.md) | Coordinated instance spanning multiple hosts | Not measured here. `size: 2`, sixteen GPUs across two physical nodes, leader `m0` routing, and gang admission. |
A Router can front either shape. The engine's LB mode determines which Pods accept inference requests.
Increasing `roles[].replicas` adds complete groups; see [role scaling](../../modules/model-deployment/deployment.md#rollout-behavior).
@@ -68,7 +71,7 @@ kubectl apply -f gpu-instance-type.yaml
kubectl get instancetype elastic-cpu-gpu
```
-Both examples refer to `elastic-cpu-gpu`. Replace that name if you use an existing InstanceType.
+All four examples refer to `elastic-cpu-gpu`. Replace that name if you use an existing InstanceType.
## Check the software identity
@@ -84,11 +87,12 @@ kubectl -n "$OPERATOR_NAMESPACE" exec "$OPERATOR_POD" -c "$OPERATOR_CONTAINER" -
```
The `dev` tag can move. Each walkthrough's validation record identifies the binary and image used.
-The serving manifests pin the official runner digest; no engine packages were replaced during validation.
+The serving manifests name the runner by tag. Validation records list the observed image tags or digests.
+No engine packages were replaced during validation.
---
**See also** — [Model Deployment](../../modules/model-deployment/_index.md) (configuration guides) ·
[Model Prefetch Walkthrough](../model-delivery/prefetch.md)
-**Next** → [Elastic EP Walkthrough](elastic-ep.md) or [External DP Walkthrough](external-dp.md)
+**Next** → [Prefill and Decode Walkthrough](prefill-decode.md) — disaggregated serving with router.
diff --git a/docs/walkthroughs/model-deployment/elastic-ep.md b/docs/walkthroughs/model-deployment/elastic-ep.md
index 6d6a81ea1..4e7b86ab0 100644
--- a/docs/walkthroughs/model-deployment/elastic-ep.md
+++ b/docs/walkthroughs/model-deployment/elastic-ep.md
@@ -13,6 +13,7 @@ Captured CR output appears beside each check.
- [Step 4: release the application](#step-4-release-the-application)
- [Diagnose a stalled resize](#diagnose-a-stalled-resize)
- [Validation record](#validation-record)
+- [Troubleshooting](#troubleshooting)
## Prerequisites
@@ -49,7 +50,7 @@ spec:
replicas: 1
size: 1
instanceType: elastic-cpu-gpu
- image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
+ image: gpustack/runner:cuda13.0-vllm0.29.0
imagePullPolicy: IfNotPresent
resources:
accelerator: "2"
@@ -99,13 +100,19 @@ This manifest has one serving instance and no router. Only its master exposes th
the other GPU members participate through Ray.
A separate recorded run added this `spec.router` block before creating the deployment:
-```yaml
-router:
- name: vllm-router
- image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db
- extraArgs:
- - --policy
- - round_robin
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: tp2-elastic
+ namespace: gpustack-elastic
+ spec:
++ router:
++ name: vllm-router
++ image: gpustack/llm-router:v0.2.0
++ extraArgs:
++ - --policy
++ - round_robin
```
The engine still uses Internal LB. This Router forwards to the master, which distributes work to the DP engines.
@@ -203,6 +210,20 @@ kubectl -n gpustack-elastic patch modeldeployment tp2-elastic --type=json \
-p '[{"op":"test","path":"/spec/roles/0/elasticEp/width","value":2},{"op":"replace","path":"/spec/roles/0/elasticEp/width","value":4}]'
```
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: tp2-elastic
+ namespace: gpustack-elastic
+ spec:
+ roles:
+ - name: server
+ elasticEp:
+- width: 2
++ width: 4
+```
+
Wait for the operator's observation of the current generation to reach width 4 at every layer.
Read the diagnostic ConfigMap belonging to this deployment UID:
@@ -279,7 +300,9 @@ That release check also verified deletion of the owned head, GPU Pods, Router, S
For a shared cluster, compare the application's captured device claims with the ledger;
other workloads may keep cards occupied.
-ModelPrefetch keeps the model pinned independently of the serving application.
+A ModelPrefetch keeps the weights on the node independently of the serving application.
+Without a pin, the node cache can still collect them once they lapse. To keep them, see
+[Step 4 of the prefetch walkthrough](../model-delivery/prefetch.md#step-4-pin-weights-against-eviction).
When no consumer needs the weights, remove the prefetch and its dedicated binding, artifact, and store.
See [Model Prefetch](../../modules/model-delivery/prefetch.md) for cache retention and collection behavior.
@@ -367,6 +390,17 @@ An earlier run reached `CREATED` placement groups and eight live GPU workers, bu
remained pending and requests timed out. This recorded run completed expansion and inference.
The earlier root cause remains unconfirmed; the result does not isolate shared memory or head placement as its cause.
+## Troubleshooting
+
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| Deployment stuck in `Starting` | `QuotaReserved=False/Pending` | Replicas wait for quota in the role's ClusterQueue, which the condition message names. Width times TP is the GPU demand; compare it with the queue's quota |
+| Pods stay gated and `kubectl get workloads` lists none for the role | `QuotaReserved=False/PodGroupIncomplete` | Fewer replicas exist than the role declares, so Kueue composes no Workload for the missing ones. The message gives the counts. Read `kubectl get pods` and the events to see which replica is missing |
+| Expansion from width 2 to 4 does not complete | `ElasticResize=False/ScaleRefused` or `ElasticResize=Unknown/WorldUnprovable` | `ScaleRefused`: the engine refused the resize until the retry bound was spent, and the message carries its last refusal. `WorldUnprovable`: the live width cannot be proved. An unanswered request is never resent; the outcome is decided by observation. See [When a resize holds](../../modules/model-deployment/elastic-ep.md#when-a-resize-holds) and [Diagnosis](../../modules/model-deployment/elastic-ep.md#diagnosis) |
+| Role update changed image or args but pods didn't roll | `ReplicasUpToDate=False/RolloutHeldByWeights` | The weights the replacements need are blocked, so the operator keeps the serving replicas. `WeightsReady` says what blocks them |
+| Role update stalled with old pods retained | `ReplicasUpToDate=False/RolloutHeldByCache` | The KV cache connection could not be resolved. The operator keeps the old replicas and rolls the edit out once the connection returns |
+| Router requests fail during or after a resize | None; the engine answers 503 and withdraws readiness while it reconfigures | Wait for the master Pod to be Ready, then repeat the request. Read `status.roles[].endpoints.serving.state` and the Router's `/observer/endpoints` with the [Router observation steps](external-dp.md#step-4-check-router-coverage). Expect the master only |
+
---
**See also** — [Elastic EP](../../modules/model-deployment/elastic-ep.md) (profile contract) ·
diff --git a/docs/walkthroughs/model-deployment/external-dp.md b/docs/walkthroughs/model-deployment/external-dp.md
index 27e6f53ee..3ea4499b9 100644
--- a/docs/walkthroughs/model-deployment/external-dp.md
+++ b/docs/walkthroughs/model-deployment/external-dp.md
@@ -16,6 +16,7 @@ Follow the steps to verify both engine inference and Router coverage.
- [Understand the scaling limits](#understand-the-scaling-limits)
- [Step 5: inspect and release the application](#step-5-inspect-and-release-the-application)
- [Validation record](#validation-record)
+- [Troubleshooting](#troubleshooting)
## Deployment shape
@@ -69,7 +70,7 @@ spec:
replicas: 1
size: 2
instanceType: elastic-cpu-gpu
- image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
+ image: gpustack/runner:cuda13.0-vllm0.29.0
imagePullPolicy: IfNotPresent
resources:
accelerator: "2"
@@ -92,7 +93,7 @@ spec:
- allgather_reducescatter
router:
name: vllm-router
- image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db
+ image: gpustack/llm-router:v0.2.0
extraArgs:
- --policy
- round_robin
@@ -278,7 +279,8 @@ A Router in front of that Internal endpoint does not turn the engine into Extern
Increasing `roles[].replicas` creates additional complete groups with their existing TP/DP/EP shape.
That operation does not enlarge an existing group's EP world and was not measured in this External run.
-The ordinary role's `size` cannot be changed after creation.
+Changing `size` replaces whole replicas. It does not resize a running DP group.
+See [Rollout behavior](../../modules/model-deployment/deployment.md#rollout-behavior).
Hybrid and multiple HTTP ports per Pod are outside this walkthrough.
## Step 5: inspect and release the application
@@ -323,7 +325,9 @@ The corrected run used binary `20d674b7f8c1e1d4c22b91a09fd178b9738d70be` from
`thxcode/gpustack-operator:dev-20d674b7@sha256:43c14087fd3391e4bf4a3327f246053d9e59b2feb684cb792f63a2f0db659fbd`.
That test image replaced the operator binary on the pinned official base. It retained the packaged vendor assets.
-The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements.
+The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements:
+`gpustack/runner:cuda13.0-vllm0.29.0`.
+The router image was `gpustack/llm-router:v0.2.0`.
Its CuPy installation contained only `cupy-cuda13x` 14.2.0.
The initial operator retained `member-index=0` and registered only member 0.
@@ -340,6 +344,16 @@ Each returned eight completion tokens; both streams ended with `data: [DONE]`.
The correction removed the leader restriction only from eligible External Server endpoints.
Both engine Pods remained running through the Operator upgrade. The Router Pod restarted with its corrected selector.
+## Troubleshooting
+
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| No router Pod exists | `RouterReady=False/RenderFailed` | The operator refused to render the router from the spec, so it creates no router Pod for it. The condition message carries the refusal, such as roles that serve on different ports. Fix `spec.router` or the roles. See [Routing](../../modules/model-deployment/routing.md#which-pods-receive-traffic) and [The router block](../../modules/model-deployment/prefill-decode.md#the-router-block) |
+| Router sends all traffic to rank 0 only | None; the leader-only endpoint selector still applies | Check that the operator build includes the External selector fix: the initial build in the validation record registered only member 0. Check that `parallelism.loadBalance` reads `External` in the role status. Check that the Router's `/observer/endpoints` lists both ranks. See [Routing](../../modules/model-deployment/routing.md#which-pods-receive-traffic) |
+| DP ranks fail distributed initialization | None; read the engine logs of both members | Each member reaches the leader at `--r-m0.--r`. Check `$(GPUSTACK_REPLICA_LEADER_ADDRESS)` in `--data-parallel-address`, distinct rank values, matching DP sizes, and the DP RPC port between the Pods |
+| Direct HTTP calls succeed but Router calls fail | `endpoints.eligible` is `null` or `0`, or `endpoints.serving.state` is not `Confirmed` | The Router selects only endpoints the operator qualified. Read `status.roles[].endpoints` and the Router's `/observer/endpoints`. See [Model Deployment Status](../../modules/model-deployment/status.md#status) |
+| Router and Services remain after deployment deletion | None; the cause is unconfirmed | This run left them for about two minutes and needed explicit cleanup. [Issue #601](https://github.com/gpustack/gpustack-operator/issues/601) tracks related delays, but this run did not establish that cause. Confirm the leftovers' owner UID, then delete them with `kubectl -n gpustack-elastic delete deployments,services -l app.kubernetes.io/instance=external-fixed` |
+
---
**See also** — [Elastic EP Walkthrough](elastic-ep.md) (collective expansion) ·
diff --git a/docs/walkthroughs/model-deployment/multi-host.md b/docs/walkthroughs/model-deployment/multi-host.md
new file mode 100644
index 000000000..dd4365a89
--- /dev/null
+++ b/docs/walkthroughs/model-deployment/multi-host.md
@@ -0,0 +1,213 @@
+# Multi-Host Serving Walkthrough
+
+Deploy large language models across multiple physical GPU nodes in a single coordinated serving group.
+When a model exceeds the memory of a single machine, an instance spans multiple Pods with gang admission.
+The operator coordinates leader discovery and environment variable injection across all group members.
+
+## Contents
+
+- [Deployment shape](#deployment-shape)
+- [Prerequisites](#prerequisites)
+- [Step 1: define the multi-host deployment](#step-1-define-the-multi-host-deployment)
+- [Step 2: verify gang admission and leader injection](#step-2-verify-gang-admission-and-leader-injection)
+- [Step 3: verify distributed inference](#step-3-verify-distributed-inference)
+- [Step 4: scale replicas and update member size](#step-4-scale-replicas-and-update-member-size)
+- [Step 5: release the application](#step-5-release-the-application)
+- [Troubleshooting](#troubleshooting)
+
+## Deployment shape
+
+A multi-host serving role configures `spec.roles[].size` greater than one.
+Each serving replica forms a collective unit composed of *size* distinct member Pods.
+All member Pods in a group live and die together under gang scheduling.
+
+Member `m0` acts as the primary master replica and receives all incoming Service traffic.
+Subordinate members (`m1`, `m2`, etc.) run auxiliary parallel workers and synchronize with `m0`.
+The operator injects cluster discovery environment variables into each container automatically.
+
+## Prerequisites
+
+Prepare a cluster with at least two physical GPU nodes containing identical accelerator counts.
+Complete [cluster preparation](_index.md#prepare-the-cluster) and [model prefetch](../model-delivery/prefetch.md).
+Verify inter-node network connectivity and high-speed fabric (such as RoCE, InfiniBand, or AWS EFA).
+Each node in this walkthrough provides eight GPUs.
+Tensor parallelism spans the GPUs inside each host. Pipeline parallelism spans the two hosts.
+
+This layout is a configuration example. The walkthrough does not record a measured run on sixteen GPUs.
+
+## Step 1: define the multi-host deployment
+
+Save the following definition as `multi-host.yaml`.
+The specification declares `replicas: 1` and `size: 2`, allocating sixteen whole GPUs across two Pods.
+The launch script configures tensor parallelism across eight local GPUs and pipeline parallelism across the two hosts.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: deepseek-multi-host
+ namespace: gpustack-elastic
+spec:
+ model:
+ name: /models
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ roles:
+ - name: server
+ kind: Server
+ replicas: 1
+ size: 2
+ instanceType: elastic-cpu-gpu
+ image: gpustack/runner:cuda13.0-vllm0.29.0
+ resources:
+ accelerator: "8"
+ command:
+ - /bin/bash
+ - -c
+ - |
+ set -euo pipefail
+ if [ "${GPUSTACK_MEMBER_INDEX}" -eq 0 ]; then
+ ray start --head --port=6379 --disable-usage-stats
+ exec vllm serve /models --served-model-name deepseek-v2-lite-chat \
+ --host 0.0.0.0 --port 8000 \
+ --tensor-parallel-size 8 \
+ --pipeline-parallel-size "${GPUSTACK_REPLICA_SIZE}" \
+ --distributed-executor-backend ray
+ else
+ exec ray start --address="${GPUSTACK_REPLICA_LEADER_ADDRESS}:6379" --block
+ fi
+```
+
+The operator does not compose multi-node arguments. A multi-host role takes over the command line with
+`command`, and the script reads the three variables the operator injects into every member.
+This snippet illustrates how the operator wires member coordinates to the entrypoint script.
+
+Production multi-host engines typically use Ray (`--distributed-executor-backend ray`).
+Member `m0` initializes the Ray head cluster and serves the API; subordinate members join the Ray cluster
+with `ray start --address=... --block`. A role with `command` reports `status.roles[].unmanaged: true`,
+because the operator renders no engine argument for it. See
+[the override tiers](../../modules/model-deployment/deployment.md#the-three-override-tiers).
+
+Apply the deployment manifest:
+
+```bash
+kubectl apply -f multi-host.yaml
+kubectl -n gpustack-elastic get modeldeployment deepseek-multi-host
+```
+
+## Step 2: verify gang admission and leader injection
+
+Check the member Pods:
+
+```bash
+kubectl -n gpustack-elastic get pods -l app.kubernetes.io/instance=deepseek-multi-host
+```
+
+You will observe two member Pods named with their member index suffixes:
+`deepseek-multi-host-server-r0-m0` and `deepseek-multi-host-server-r0-m1`.
+Inspect the injected environment variables in the secondary worker `m1`:
+
+```bash
+kubectl -n gpustack-elastic exec deepseek-multi-host-server-r0-m1 -- env | grep GPUSTACK_
+```
+
+The operator automatically populates:
+- `GPUSTACK_REPLICA_LEADER_ADDRESS`: the DNS name of member `m0`, `deepseek-multi-host-server-r0-m0.deepseek-multi-host-server-r0`.
+- `GPUSTACK_REPLICA_SIZE`: total member count (`2`).
+- `GPUSTACK_MEMBER_INDEX`: the zero-based index of this specific member (`1`).
+
+Inspect the Kubernetes Service created for the role:
+
+```bash
+kubectl -n gpustack-elastic get svc deepseek-multi-host-server -o yaml
+```
+
+The Service selector targets only member `m0`. External traffic lands on the primary replica.
+
+## Step 3: verify distributed inference
+
+Once both member Pods reach the `Ready` condition, query the server Service:
+
+```bash
+kubectl -n gpustack-elastic run test-client --rm -i --restart=Never \
+ --image=curlimages/curl -- \
+ curl -s http://deepseek-multi-host-server:8000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "deepseek-v2-lite-chat",
+ "messages": [{"role": "user", "content": "Hello distributed inference!"}],
+ "max_tokens": 16
+ }' | jq .
+```
+
+The master rank coordinates distributed collective operations with the secondary worker and returns tokens.
+
+## Step 4: scale replicas and update member size
+
+To scale serving throughput, increase `replicas` to launch additional multi-host groups:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: deepseek-multi-host
+ namespace: gpustack-elastic
+ spec:
+ roles:
+ - name: server
+- replicas: 1
++ replicas: 2
+ size: 2
+```
+
+Scaling to `replicas: 2` creates another complete two-host group (`m0` and `m1`), reserving sixteen additional GPUs.
+
+To change the member topology of each instance, you can also update `size`, `resources` or `instanceType`.
+These edits do not resize a running collective engine in place.
+
+The operator deletes each old instance and its Workload first.
+It then creates the replacement in that slot and waits for fresh Kueue gang admission.
+The next old instance is deleted only after the replacement is admitted.
+A role with one replica therefore stops serving during the replacement.
+
+The script reads `GPUSTACK_REPLICA_SIZE`, so the pipeline degree follows `size`.
+The tensor degree is fixed at 8, so it must still match the accelerators per member.
+See [Rollout behavior](../../modules/model-deployment/deployment.md#rollout-behavior).
+
+## Step 5: release the application
+
+Remove the deployment and verify that all GPU allocations are freed:
+
+```bash
+kubectl -n gpustack-elastic delete modeldeployment deepseek-multi-host
+kubectl -n gpustack-elastic get pods -l app.kubernetes.io/instance=deepseek-multi-host
+```
+
+Confirm that the accelerators on both host nodes are free again. Each card reports `remaining`:
+
+```bash
+kubectl get devices -o json | jq '[.items[].status.groups[].accelerators[] |
+ {mode, remaining, allocatedSlices: (.allocatedSlices // 0)}]'
+```
+
+## Troubleshooting
+
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| Deployment stuck in `Starting`, Kueue composes no Workload | `QuotaReserved=False/PodGroupIncomplete` | Fewer instances exist than the role declares. Read the message for the counts, then check the Pods that are missing or unschedulable |
+| An instance was lost and is being rebuilt | `ReplicasUpToDate=False/ReplacementInProgress` | A replica disappeared for a reason outside the spec: a drained node, Kueue reclaiming quota, an eviction or a manual delete. The Pods of an instance share one lifecycle, so the whole instance is rebuilt. Check the node and Kueue events |
+| Replicas wait for quota | `QuotaReserved=False/Pending` | The ClusterQueue has insufficient capacity to admit the replica. Free capacity, raise the pool quota, or reduce per-member accelerator requests |
+| Secondary worker cannot reach the leader | Engine logs on `m1` | Check that the headless Service resolves `GPUSTACK_REPLICA_LEADER_ADDRESS` and that no NetworkPolicy blocks the engine's distributed port |
+| Engine errors on parallel size | Engine startup log | Check that `--tensor-parallel-size` equals the accelerators per member and that the pipeline degree equals `size` |
+
+---
+
+**See also** — [Model Deployment Configuration](../../modules/model-deployment/deployment.md) (role fields and scaling) ·
+[Prefill and Decode Walkthrough](prefill-decode.md) (disaggregated serving) ·
+[Elastic EP Walkthrough](elastic-ep.md) (collective expansion) ·
+[External DP Walkthrough](external-dp.md) (fixed per-rank serving)
+
+**Next** → [Model Deployment Status](../../modules/model-deployment/status.md) — observe status conditions and phases.
diff --git a/docs/walkthroughs/model-deployment/prefill-decode.md b/docs/walkthroughs/model-deployment/prefill-decode.md
new file mode 100644
index 000000000..7d460f6fd
--- /dev/null
+++ b/docs/walkthroughs/model-deployment/prefill-decode.md
@@ -0,0 +1,208 @@
+# Prefill and Decode Walkthrough
+
+Deploy a disaggregated LLM serving topology with dedicated prefill and decode roles fronted by a router.
+The operator coordinates joint admission across both roles and routes requests through a unified endpoint.
+Configuration diffs show how the roles cooperate. This page records no captured run.
+
+## Contents
+
+- [Deployment shape](#deployment-shape)
+- [Prerequisites](#prerequisites)
+- [Step 1: start disaggregated serving](#step-1-start-disaggregated-serving)
+- [Step 2: pin the KV cache transfer protocol](#step-2-pin-the-kv-cache-transfer-protocol)
+- [Step 3: verify router endpoints and inference](#step-3-verify-router-endpoints-and-inference)
+- [Step 4: inspect joint admission and scaling](#step-4-inspect-joint-admission-and-scaling)
+- [Step 5: release the application](#step-5-release-the-application)
+- [Troubleshooting](#troubleshooting)
+
+## Deployment shape
+
+Prefill and decode disaggregation separates prompt processing from token generation.
+A compute-heavy Prefill role processes initial prompt context with high parallel throughput.
+A memory-bandwidth-bound Decode role generates subsequent tokens sequentially.
+
+A managed Router fronts both roles and exposes a single external HTTP endpoint.
+The Router parses prompt requests, forwards prefill work, and coordinates KV cache transfer to decoders.
+Kueue schedules both roles as a joint unit, preventing half-admitted workloads.
+
+## Prerequisites
+
+Complete [cluster and GPU preparation](_index.md#prepare-the-cluster) and
+[model prefetch](../model-delivery/prefetch.md).
+Ensure sufficient GPUs exist for both roles simultaneously (at least six GPUs for this example).
+Use the shared namespace, cached artifact, and GPU InstanceType.
+
+## Step 1: start disaggregated serving
+
+Save the following manifest as `pd-deployment.yaml`.
+The definition specifies one Prefill replica with two GPUs and two Decode replicas with two GPUs each.
+The `spec.router` stanza deploys `llm-d-router` as the unified front door.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: deepseek-pd
+ namespace: gpustack-elastic
+spec:
+ model:
+ name: /models
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ router:
+ name: llm-d-router
+ roles:
+ - name: prefill
+ kind: Prefill
+ replicas: 1
+ instanceType: elastic-cpu-gpu
+ image: gpustack/runner:cuda13.0-vllm0.29.0
+ resources:
+ accelerator: "2"
+ - name: decode
+ kind: Decode
+ replicas: 2
+ instanceType: elastic-cpu-gpu
+ image: gpustack/runner:cuda13.0-vllm0.29.0
+ resources:
+ accelerator: "2"
+```
+
+Apply the manifest to the cluster:
+
+```bash
+kubectl apply -f pd-deployment.yaml
+kubectl -n gpustack-elastic get modeldeployment deepseek-pd
+```
+
+## Step 2: pin the KV cache transfer protocol
+
+Disaggregated serving requires prompt KV tensors to flow from prefill workers to decode workers.
+The pair in Step 1 already hands blocks over directly. Unset `spec.kvTransfer.protocol` renders `tcp`.
+Pin the protocol explicitly. Add the lines marked `+` to `pd-deployment.yaml`:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: deepseek-pd
+ namespace: gpustack-elastic
+ spec:
++ kvTransfer:
++ protocol: TCP
+ router:
+ name: llm-d-router
+ roles:
+ - name: prefill
+ kind: Prefill
+```
+
+Apply the update:
+
+```bash
+kubectl apply -f pd-deployment.yaml
+```
+
+The protocol is deployment-wide, because it describes one link between the two roles.
+It accepts `Auto`, `TCP`, `RDMA`, `EFA`, `CANN` and `ROCM`. `Auto` renders `tcp` too.
+
+An edit restarts every role, because the value renders into both ends' arguments.
+The roles can run different protocols until both converge.
+`RDMA` and `EFA` also need a matching device request on each role.
+See the [RDMA walkthrough](../rdma/network-endpoints.md#step-3-prefill-and-decode-interface-requests).
+
+A shared store through `spec.kvCache.poolRef` is part of the deployment identity.
+Declare it in the first apply. Adding it later is refused, because a different cache describes a different deployment.
+See [deployment identity fields](../../modules/model-deployment/deployment.md#deployment-identity-fields).
+
+## Step 3: verify router endpoints and inference
+
+Monitor status conditions until all roles and the router reach ready states:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment deepseek-pd -o yaml
+```
+
+Read the reported endpoint. It carries the scheme and a cluster-internal `.svc` host name:
+
+```bash
+ROUTER_ENDPOINT=$(kubectl -n gpustack-elastic get modeldeployment deepseek-pd \
+ -o jsonpath='{.status.endpoint}')
+echo "Router endpoint: $ROUTER_ENDPOINT"
+```
+
+Send a test completion request through the Router from inside the cluster:
+
+```bash
+kubectl -n gpustack-elastic run test-client --rm -i --restart=Never \
+ --image=curlimages/curl -- \
+ curl -s "${ROUTER_ENDPOINT}/v1/chat/completions" \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "deepseek-v2-lite-chat",
+ "messages": [{"role": "user", "content": "Explain prefill and decode disaggregation."}],
+ "max_tokens": 16
+ }'
+```
+
+The Router receives the request, delegates the prompt phase, transfers KV tensors, and streams tokens.
+`status.endpoint` remains empty until the router Pod passes its readiness check.
+
+## Step 4: inspect joint admission and scaling
+
+The operator manages Kueue workloads across all declared roles simultaneously.
+Inspect the Kueue Workloads that the deployment's Pods own:
+
+```bash
+kubectl -n gpustack-elastic get workloads
+```
+
+Joint admission guarantees that neither role starts execution without capacity for the other.
+If the cluster lacks capacity for decode workers, prefill workers remain queued rather than idling.
+You can adjust `replicas` independently on either role to balance prompt versus token throughput.
+
+Role kinds (`kind: Prefill`, `kind: Decode`), role names and the declared role set are frozen.
+The `size`, `resources`, `instanceType` and `command` of an existing role are editable.
+An edit replaces whole replicas, one at a time, with fresh Kueue admission.
+See [rollout behavior](../../modules/model-deployment/deployment.md#rollout-behavior).
+
+## Step 5: release the application
+
+After testing disaggregated serving, delete the deployment to release GPUs and router services:
+
+```bash
+kubectl -n gpustack-elastic delete modeldeployment deepseek-pd
+kubectl -n gpustack-elastic get pods,services -l app.kubernetes.io/instance=deepseek-pd
+```
+
+Verify that the accelerators are free again. Each card reports `remaining`:
+
+```bash
+kubectl get devices -o json | jq '[.items[].status.groups[].accelerators[] |
+ {mode, remaining, allocatedSlices: (.allocatedSlices // 0)}]'
+```
+
+## Troubleshooting
+
+| Symptom | Condition / Reason | Check & Mitigation |
+|---|---|---|
+| Deployment stuck in `Starting`, Kueue composes no Workload | `QuotaReserved=False/PodGroupIncomplete` | Fewer replicas exist than a role declares. Read the message for the counts, then check the missing or unschedulable Pods |
+| No Workload asks for quota any more | `QuotaReserved=False/Parked` | The groups could not all be placed, so their Workloads were deactivated. An identical re-apply does not clear this. Free the capacity and reactivate the Workloads, or recreate the deployment |
+| Part of the deployment was taken by another workload | `QuotaReserved=False/PreemptedInPart` | A higher-priority workload reclaimed some replicas. The replicas that kept their quota hold accelerators until the reclaimed ones are admitted again or the deployment is deleted. Check what took the quota |
+| `status.endpoint` is empty | `RouterReady=False/NoReadyReplicas` | The router has no ready replica. It may still pull its image or fail readiness. Inspect the router Pod |
+| Router is not created | `RouterReady=False/RenderFailed` | The router render was refused as a spec problem, so no router Pod exists. Read the condition message and check `spec.router` |
+| Disaggregated roles fail KV transfer | Engine logs on both roles | The protocol is deployment-wide, so the two ends cannot name different values. Check that the image carries the matching Mooncake build, that the device request fits the protocol, and that the nodes can reach each other |
+| Router sees no cache events | `KVEventsPublishing=False/PublisherDisabled` | A role that produces cache blocks renders no publisher. A routed SGLang deployment always reports this, because only vLLM renders one. The condition reports configuration, not traffic. See [KV events](../../modules/model-deployment/deployment.md#operator-owned-keys) |
+| Missing role readiness | `RoleKindsReady=False` | One of the role kinds (Prefill or Decode) has zero ready replicas; check failed engine pods |
+
+---
+
+**See also** — [Model Deployment Prefill and Decode](../../modules/model-deployment/prefill-decode.md) (configuration and transfer) ·
+[Routing](../../modules/model-deployment/routing.md) (router engine combinations) ·
+[Elastic EP Walkthrough](elastic-ep.md) (collective expansion) ·
+[External DP Walkthrough](external-dp.md) (fixed per-rank serving)
+
+**Next** → [Multi-Host Serving Walkthrough](multi-host.md) — scale models across multiple GPU nodes.
diff --git a/docs/walkthroughs/rdma/network-endpoints.md b/docs/walkthroughs/rdma/network-endpoints.md
new file mode 100644
index 000000000..7aad42cfb
--- /dev/null
+++ b/docs/walkthroughs/rdma/network-endpoints.md
@@ -0,0 +1,224 @@
+# RDMA Network Endpoints Walkthrough
+
+Request and verify RDMA endpoints for workloads, align network adapters with GPUs on NUMA nodes,
+and configure multi-tenant and cache fabric interconnects.
+
+On a cluster with RDMA-capable hardware and GPUStack installed, follow these steps to verify
+interface discovery, container injection, NUMA alignment, and deployment integration.
+
+## Contents
+
+- [Prerequisites](#prerequisites)
+- [Step 1: request RDMA from a Pod](#step-1-request-rdma-from-a-pod)
+- [Step 2: NUMA alignment with kubelet](#step-2-numa-alignment-with-kubelet)
+- [Step 3: prefill and decode interface requests](#step-3-prefill-and-decode-interface-requests)
+- [Step 4: KV cache fabric over RDMA](#step-4-kv-cache-fabric-over-rdma)
+- [Step 5: AWS EFA differences](#step-5-aws-efa-differences)
+- [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+Device Manager discovers network interfaces from `/sys/class/net`, resolves bound RDMA devices, and
+verifies link status before publishing endpoints.
+
+When at least one endpoint is active, the node receives the `feature.gpustack.ai/rdma.capable=true`
+label (see [Network Topology](../../modules/rdma/network-topology.md)).
+Device Manager registers three extended resource keys with the kubelet — shared, exclusive, and
+SR-IOV partitioned — with capacities and semantics defined in
+[The RDMA resource keys](../../modules/rdma/network-topology.md#the-rdma-resource-keys).
+
+Inspect discovered interfaces and allocatable capacities:
+
+```bash
+kubectl get devices -o json |
+ jq '[.spec.interfaces[] | (., (.virtualFunctions // [])[])]
+ | map(select(.rdma or .link)) | map({name, pciBusId, rdmaDevice, link})'
+
+kubectl get node -o json |
+ jq '.status.allocatable | with_entries(select(.key | contains("gpustack.ai/rdma")))'
+```
+
+## Step 1: request RDMA from a Pod
+
+A workload requests whole accelerators and RDMA endpoints within the same container.
+Pods must carry the `kueue.x-k8s.io/queue-name` label to participate in admission.
+Name the entrance `LocalQueue` of the pool that holds the accelerators.
+LocalQueue names look like `gpustack-fnv64-`. List them with `kubectl -n team-a get localqueues`:
+
+```yaml
+apiVersion: v1
+kind: Pod
+metadata:
+ name: trainer-0
+ namespace: team-a
+ labels:
+ kueue.x-k8s.io/queue-name: gpustack-fnv64-
+spec:
+ restartPolicy: Never
+ containers:
+ - name: trainer
+ image: ubuntu:24.04
+ command:
+ - "sleep"
+ - "3600"
+ resources:
+ limits:
+ nvidia.com/gpu: "1"
+ device.gpustack.ai/rdma.shared: "1"
+```
+
+The allocated container receives character devices and `NCCL_IB_HCA` configuration
+(see [Injected devices](../../modules/rdma/operations.md#injected-devices)).
+
+Prefer `device.gpustack.ai/rdma.shared` when a container needs a single endpoint. Use the exclusive
+key `device.gpustack.ai/rdma` when requesting multiple distinct network endpoints.
+
+## Step 2: NUMA alignment with kubelet
+
+Kubernetes does not pair specific accelerators with adjacent network adapters by default.
+Enable NUMA alignment by setting the TopologyManager policy in `/var/lib/kubelet/config.yaml`
+(see [Enabling NUMA alignment on the kubelet](../../modules/rdma/operations.md#enabling-numa-alignment-on-the-kubelet)):
+
+```diff
+ apiVersion: kubelet.config.k8s.io/v1beta1
+ kind: KubeletConfiguration
++topologyManagerPolicy: restricted
++topologyManagerScope: container
+```
+
+Restart the kubelet after saving the configuration:
+
+```bash
+systemctl restart kubelet
+kubectl get --raw /api/v1/nodes//proxy/configz |
+ jq -r '.kubeletconfig | {topologyManagerPolicy, topologyManagerScope}'
+```
+
+`restricted` ensures that single-socket requests land on one NUMA node, while multi-GPU workloads
+spanning multiple sockets are packed into the minimum number of NUMA nodes.
+
+## Step 3: prefill and decode interface requests
+
+In a `ModelDeployment`, roles do not declare raw resource keys. Specify the number of network
+interfaces under `roles[].resources.interface`.
+The operator refuses an interface request when no RDMA or EFA transfer leg uses it.
+A prefill and decode pair with `spec.kvTransfer.protocol: RDMA` is such a leg:
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: qwen-pd
+ namespace: team-a
+spec:
+ model:
+ name: Qwen/Qwen2.5-7B-Instruct
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ router:
+ name: llm-d-router
+ kvTransfer:
+ protocol: RDMA
+ roles:
+ - name: prefill
+ kind: Prefill
+ replicas: 1
+ instanceType: gpustack--nvidia-a10g-linux-amd64
+ resources:
+ accelerator: "2"
+ interface: "2"
+ - name: decode
+ kind: Decode
+ replicas: 1
+ instanceType: gpustack--nvidia-a10g-linux-amd64
+ resources:
+ accelerator: "2"
+ interface: "2"
+```
+
+For the `RDMA` protocol, the operator translates `interface: 1` into `device.gpustack.ai/rdma.shared: 1`.
+When `interface` is greater than 1, it renders `device.gpustack.ai/rdma: N`.
+Each allocated endpoint then corresponds to an independent physical adapter
+(see [A minimal deployment](../../modules/model-deployment/deployment.md#a-minimal-deployment)).
+
+The same request without `router`, `kvTransfer` or a cache backend that uses RDMA fails at admission with
+"no effective RDMA or EFA transfer leg uses the requested interfaces".
+A cache backend on RDMA is the other leg, and Step 4 sets it up.
+
+## Step 4: KV cache fabric over RDMA
+
+To route KV cache transport over an RDMA fabric instead of TCP, edit the `mooncake-dram` backend
+from [Step 1 of the Shared Cache walkthrough](../kv-cache/shared-cache.md#step-1-the-store).
+The transport is editable on a running backend:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: KVCacheBackend
+ metadata:
+ name: mooncake-dram
+ spec:
+ type: Mooncake
++ transport:
++ protocol: RDMA
+ connection:
+ managed:
+ leader: {}
+ members:
+ - nodeSelector:
+ kubernetes.io/os: linux
+ medium: DRAM
+ capacityPerMember: 8Gi
++ fabricInterfaceCount: 2
+```
+
+Member DaemonSet Pods automatically receive the network devices, host privileges, and limits
+required for high-performance transport (see [The members](../../modules/kv-cache/backend.md#the-members)).
+
+## Step 5: AWS EFA differences
+
+On AWS Elastic Fabric Adapter (EFA) instances, RDMA devices bypass `/sys/class/net` and do not expose
+standard verbs completion queues. Declare the `EFA` protocol explicitly on a backend for those nodes:
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: KVCacheBackend
+metadata:
+ name: mooncake-efa
+spec:
+ type: Mooncake
+ transport:
+ protocol: EFA
+ connection:
+ managed:
+ leader: {}
+ members:
+ - nodeSelector:
+ node.kubernetes.io/instance-type: c6in.32xlarge
+ medium: DRAM
+ capacityPerMember: 64Gi
+ fabricInterfaceCount: 2
+```
+
+In `ModelDeployment`, set `kvTransfer.protocol: EFA` and specify custom runner images containing
+`libfabric` and EFA-enabled Mooncake builds.
+
+## Troubleshooting
+
+| Symptom | Root cause | Check & Mitigation |
+|---|---|---|
+| RDMA resource keys present but all `Unhealthy` | All port links failed or unreadable | Inspect `kubectl get devices -o json` for `.spec.interfaces[].link`; verify physical cable and subnet manager |
+| Workload admitted by Kueue but Pod remains `Pending` | Over-admission: Kueue does not meter network resources | Inspect cluster allocatable RDMA keys (`device.gpustack.ai/rdma*`); add nodes or wait for active pods to finish |
+| Pod rejected with `TopologyAffinityError` | Free GPUs and NICs reside on mismatched NUMA nodes | Ensure kubelet has `topologyManagerPolicy: restricted`; verify GPU and NIC requests are declared in the same container |
+| Fewer `uverbs` devices inside container than expected | Shared tokens coalesced into duplicate endpoints | Requesting `rdma.shared: N` (N>1) may merge tokens from the same NIC; use `device.gpustack.ai/rdma: N` for distinct adapters |
+| Node unexpectedly loses `feature.gpustack.ai/rdma.capable` | Every endpoint on the node is unusable | Check kernel dmesg and interface carrier status; operator removes the label when every endpoint reports `Failed` or lacks a bound RDMA device. See [Network Topology](../../modules/rdma/network-topology.md#the-three-node-labels) |
+| AWS EFA workload remains `Pending` indefinitely | Workload requested `device.gpustack.ai/rdma` instead of EFA | EFA nodes expose 0 allocatable for standard RDMA keys; configure `transport.protocol: EFA` or `TCP` |
+| `ModelDeployment` rejected with "no effective RDMA or EFA transfer leg" | Interface requested with no RDMA or EFA leg | Pair the request with `kvTransfer.protocol: RDMA` or `EFA` on a prefill and decode pair, or with a `KVCacheBackend` on that transport. A role with its own `command` has no managed direct leg |
+
+---
+
+**See also** — [Network Topology](../../modules/rdma/network-topology.md) (discovery and link states) ·
+[RDMA Operations](../../modules/rdma/operations.md) (resource keys, limits, and EFA configuration) ·
+[Scheduling Walkthrough](../devices/scheduling.md)
+
+**Next** → [Hierarchical Placement Walkthrough](../topology/hierarchical-placement.md)
diff --git a/docs/walkthroughs/topology/hierarchical-placement.md b/docs/walkthroughs/topology/hierarchical-placement.md
new file mode 100644
index 000000000..5120bbc36
--- /dev/null
+++ b/docs/walkthroughs/topology/hierarchical-placement.md
@@ -0,0 +1,220 @@
+# Hierarchical Placement Walkthrough
+
+Configure topology levels across cloud zones and physical racks, request domain placement for
+multi-Pod replicas, and see what a node move and a hierarchy change do.
+
+Follow these steps to discover infrastructure hierarchies, apply placement constraints to serving
+workloads, and manage topology updates.
+
+## Contents
+
+- [Prerequisites](#prerequisites)
+- [Step 1: read-only cloud zone discovery](#step-1-read-only-cloud-zone-discovery)
+- [Step 2: request zone placement from ModelDeployment](#step-2-request-zone-placement-from-modeldeployment)
+- [Step 3: static inventory snapshot with ConfigMap](#step-3-static-inventory-snapshot-with-configmap)
+- [Step 4: request rack placement](#step-4-request-rack-placement)
+- [Step 5: move a node, then change the hierarchy](#step-5-move-a-node-then-change-the-hierarchy)
+- [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+Topology-Aware Scheduling (TAS) operates continuously through Kueue. When no custom `TopologySource` is
+declared, nodes use a single-level hierarchy consisting solely of `kubernetes.io/hostname`.
+
+Each node must match exactly one Ready `TopologySource`. If a node matches zero or multiple sources,
+the operator falls back to the hostname profile to prevent ambiguous hierarchy trees.
+
+Verify existing node labels and cluster status before configuring topology sources:
+
+```bash
+kubectl get nodes --show-labels
+kubectl get topologysources
+kubectl get topologies.kueue.x-k8s.io
+```
+
+## Step 1: read-only cloud zone discovery
+
+When nodes already carry cloud provider labels for regions and zones, define a read-only
+`TopologySource` using `nodeLabels`:
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: TopologySource
+metadata:
+ name: cloud-zones
+spec:
+ nodeSelector:
+ matchLabels:
+ node-role.kubernetes.io/worker: ""
+ levels:
+ - topology.kubernetes.io/region
+ - topology.kubernetes.io/zone
+ nodeLabels: {}
+```
+
+The operator validates that labels form a strict tree (each zone belongs to exactly one region) and
+computes a profile hash. Nodes receive `topology.gpustack.ai/profile=fnv64-`, and Kueue
+materializes the matching `Topology` resource.
+
+Check source readiness:
+
+```bash
+kubectl get topologysources cloud-zones -o yaml
+```
+
+## Step 2: request zone placement from ModelDeployment
+
+When a serving replica consists of multiple Pods (such as tensor-parallel groups spanning multiple nodes),
+request placement within a single availability zone:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: qwen-distributed
+ namespace: team-a
+ spec:
+ model:
+ name: Qwen/Qwen2.5-7B-Instruct
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ roles:
+ - name: server
+ replicas: 1
+ size: 2
+ instanceType: gpustack--nvidia-a10g-linux-amd64
+ resources:
+ accelerator: "1"
++ topology:
++ requiredLevel: topology.kubernetes.io/zone
+```
+
+Kueue TAS admits the replica only when a single zone contains enough unallocated capacity to host all
+2 Pods. If capacity is scattered across zones without filling any single zone, the workload waits in
+`Pending` rather than stranding split Pods.
+
+## Step 3: static inventory snapshot with ConfigMap
+
+In private data centers without cloud labels, publish rack hierarchies using a versioned snapshot
+stored in a `ConfigMap`:
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: topology-inventory
+ namespace: gpustack-system
+data:
+ snapshot.yaml: |
+ apiVersion: topology.gpustack.ai/v1alpha1
+ revision: inventory-42
+ nodes:
+ worker-a:
+ topology.gpustack.ai/region: region-a
+ topology.gpustack.ai/zone: zone-a
+ topology.gpustack.ai/rack: rack-01
+ worker-b:
+ topology.gpustack.ai/region: region-a
+ topology.gpustack.ai/zone: zone-a
+ topology.gpustack.ai/rack: rack-02
+---
+apiVersion: worker.gpustack.ai/v1
+kind: TopologySource
+metadata:
+ name: data-center
+spec:
+ nodeSelector: {}
+ levels:
+ - topology.gpustack.ai/region
+ - topology.gpustack.ai/zone
+ - topology.gpustack.ai/rack
+ configMap:
+ configMapRef:
+ namespace: gpustack-system
+ name: topology-inventory
+ key: snapshot.yaml
+ maxStaleness: 10m
+```
+
+The operator validates the snapshot atomically, creates NFD `NodeFeature` resources, and writes the
+custom labels to each node.
+
+## Step 4: request rack placement
+
+With rack labels published, narrow the workload constraint from zone-level to rack-level:
+
+```diff
+ apiVersion: worker.gpustack.ai/v1
+ kind: ModelDeployment
+ metadata:
+ name: qwen-distributed
+ namespace: team-a
+ spec:
+ roles:
+ - name: server
+ topology:
+- requiredLevel: topology.kubernetes.io/zone
++ requiredLevel: topology.gpustack.ai/rack
+```
+
+All Pods within each replica group are placed within the same physical rack, minimizing inter-switch
+network latency.
+
+## Step 5: move a node, then change the hierarchy
+
+When a node is cabled into a new rack, update the ConfigMap snapshot with an incremented revision:
+
+```diff
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+ name: topology-inventory
+ namespace: gpustack-system
+ data:
+ snapshot.yaml: |
+ apiVersion: topology.gpustack.ai/v1alpha1
+- revision: inventory-42
++ revision: inventory-43
+ nodes:
+ worker-a:
+ topology.gpustack.ai/region: region-a
+ topology.gpustack.ai/zone: zone-a
+ topology.gpustack.ai/rack: rack-01
+ worker-b:
+ topology.gpustack.ai/region: region-a
+ topology.gpustack.ai/zone: zone-a
+- topology.gpustack.ai/rack: rack-02
++ topology.gpustack.ai/rack: rack-03
+```
+
+The profile is a hash of the ordered level keys, and the keys did not change.
+The node keeps its profile, so the worker creates no flavor and applies no hold.
+The node carries the new rack value after NFD publishes it.
+
+A change to the ordered level keys is a different operation that can evict and readmit workloads,
+unlike the revision-only change in Step 5. See
+[Change a live hierarchy](../../modules/topology/operations.md#change-a-live-hierarchy) for the
+`HoldAndDrain` migration sequence and its interruption impact before editing `levels`.
+
+## Troubleshooting
+
+For general failure surfaces (including `AwaitingQueueStatus`, `HoldAndDrain` flavor migration, and
+domain capacity fragmentation), see [Topology-Aware Scheduling Failure Surfaces](../../modules/topology/scheduling.md#failure-surfaces)
+and [Change a live hierarchy](../../modules/topology/operations.md#change-a-live-hierarchy).
+
+| Condition / Symptom | Status / Reason | Root cause & Mitigation |
+|---|---|---|
+| `TopologySource` `Ready=False` | `InvalidHierarchy` / `SnapshotInvalid` | Label hierarchy invalid (child key must have unique parent value); fix snapshot syntax or missing node entries |
+| `TopologySource` `Ready=False` | `OwnershipConflict` | Multiple writable sources target the same node; adjust node selectors to ensure disjoint sets |
+| `TopologySource` `Valid=False` | `ConfigMapReadFailed` | Verify ConfigMap name and namespace match the worker installation |
+| Node Topology Profile falls back to single-level `hostname` | Ambiguous source ownership | Node selected by 0 or multiple Ready sources; check node selectors across all sources |
+| Multi-role deployment has 1 Workload reserved but Pod not bound | Joint admission gang hold | First role fits in domain, but held until secondary role (prefill/decode) secures domain capacity; check other roles |
+
+---
+
+**See also** — [Topology-Aware Scheduling](../../modules/topology/scheduling.md) (levels and profiles) ·
+[Topology-Aware Scheduling Operations](../../modules/topology/operations.md) (ConfigMap and Webhook sources) ·
+[Model Deployment Configuration](../../modules/model-deployment/deployment.md)
+
+**Next** → [Shared Cache Walkthrough](../kv-cache/shared-cache.md)
diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml
index 6e1e4ef2d..d7fb7ca35 100644
--- a/site/data/navigation.yaml
+++ b/site/data/navigation.yaml
@@ -27,18 +27,26 @@ order:
- vendor-prerequisites
/docs/walkthroughs/:
- devices
+ - rdma
+ - topology
- kv-cache
- model-delivery
- model-deployment
/docs/walkthroughs/devices/:
- scheduling
+ /docs/walkthroughs/rdma/:
+ - network-endpoints
+ /docs/walkthroughs/topology/:
+ - hierarchical-placement
/docs/walkthroughs/kv-cache/:
- shared-cache
/docs/walkthroughs/model-delivery/:
- prefetch
/docs/walkthroughs/model-deployment/:
+ - prefill-decode
- elastic-ep
- external-dp
+ - multi-host
/docs/modules/devices/:
- preflight
- discovery
@@ -99,24 +107,30 @@ labels:
/docs/walkthroughs/: Walkthroughs
/docs/walkthroughs/devices/: Devices
/docs/walkthroughs/devices/scheduling/: Scheduling
+ /docs/walkthroughs/rdma/: RDMA
+ /docs/walkthroughs/rdma/network-endpoints/: RDMA network endpoints
+ /docs/walkthroughs/topology/: Topology
+ /docs/walkthroughs/topology/hierarchical-placement/: Hierarchical placement
/docs/walkthroughs/kv-cache/: KV Cache
/docs/walkthroughs/kv-cache/shared-cache/: Shared cache
/docs/walkthroughs/model-delivery/: Model Delivery
/docs/walkthroughs/model-delivery/prefetch/: Model prefetch
- /docs/walkthroughs/model-deployment/: Model Deployment
+ /docs/walkthroughs/model-deployment/: Model deployment
+ /docs/walkthroughs/model-deployment/prefill-decode/: Prefill and decode
/docs/walkthroughs/model-deployment/elastic-ep/: Elastic EP
/docs/walkthroughs/model-deployment/external-dp/: External DP
+ /docs/walkthroughs/model-deployment/multi-host/: Multi-host serving
/docs/getting-started/architecture/: Architecture
- /docs/getting-started/vendor-prerequisites/: Prerequisites
- /docs/modules/devices/preflight/: Preflight
- /docs/modules/devices/discovery/: Discovery
- /docs/modules/devices/requests/: Resource requests
- /docs/modules/devices/scheduling/: Scheduling
+ /docs/getting-started/vendor-prerequisites/: Vendor prerequisites
+ /docs/modules/devices/preflight/: Preflight operations
+ /docs/modules/devices/discovery/: Device discovery
+ /docs/modules/devices/requests/: Accelerator requests
+ /docs/modules/devices/scheduling/: Scheduling chain
/docs/modules/devices/admission/: Admission
/docs/modules/devices/nvidia-mig/: NVIDIA MIG
/docs/modules/devices/hygon-mig/: Hygon MIG
/docs/modules/devices/thead-mig/: T-Head MIG
- /docs/modules/rdma/network-topology/: Interface discovery
+ /docs/modules/rdma/network-topology/: Network topology
/docs/modules/rdma/operations/: Operations
/docs/modules/topology/scheduling/: Scheduling
/docs/modules/topology/operations/: Operations
@@ -126,14 +140,14 @@ labels:
/docs/modules/kv-cache/leader/: Leader
/docs/modules/kv-cache/local-disk-tier/: Local disk tier
/docs/modules/kv-cache/disk-heavy-nodes/: Disk-heavy nodes
- /docs/modules/model-delivery/artifact/: Artifacts
- /docs/modules/model-delivery/image-source/: Image sources
- /docs/modules/model-delivery/prefetch/: Prefetch and quotas
- /docs/modules/model-delivery/node-store/: Node cache
- /docs/modules/model-delivery/peer-sync/: Peer sync
+ /docs/modules/model-delivery/artifact/: Model artifact
+ /docs/modules/model-delivery/image-source/: Model image source
+ /docs/modules/model-delivery/prefetch/: Model prefetch
+ /docs/modules/model-delivery/node-store/: Node model store
+ /docs/modules/model-delivery/peer-sync/: Node-to-node sync
/docs/modules/model-delivery/operations/: Operations
- /docs/modules/model-delivery/views/: API reference
- /docs/modules/model-deployment/deployment/: Configuration
+ /docs/modules/model-delivery/views/: Model artifact API
+ /docs/modules/model-deployment/deployment/: Deployment configuration
/docs/modules/model-deployment/elastic-ep/: Elastic EP
/docs/modules/model-deployment/engine-versions/: Engine versions
/docs/modules/model-deployment/prefill-decode/: Prefill and decode
@@ -141,15 +155,15 @@ labels:
/docs/modules/model-deployment/metrics/: Metrics
/docs/modules/model-deployment/status/: Status
/docs/modules/model-deployment/shutdown/: Shutdown
- /docs/operate/installation-modes/: Installation
+ /docs/operate/installation-modes/: Installation modes
/docs/operate/high-availability/: High availability
/docs/operate/migration/: Upgrades
- /docs/operate/migration/from-v0.5/: From v0.5
- /docs/operate/migration/to-subcharts/: Subchart migration
- /docs/operate/migration/kv-cache-dtype/: Cache data types
+ /docs/operate/migration/from-v0.5/: From v0.5.x
+ /docs/operate/migration/to-subcharts/: Bundled subcharts
+ /docs/operate/migration/kv-cache-dtype/: Enforced binding dtype
/docs/operate/migration/troubleshooting/: Troubleshooting
/docs/reference/commands/: Commands
- /docs/reference/settings/: Settings
+ /docs/reference/settings/: Settings & environment variables
/docs/reference/instance-type-unit-resources/: Resource presets
/docs/reference/instance-metrics/: Instance metrics
/docs/contribute/development/: Development