Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,12 +132,15 @@ spec:
For a role whose replica spans multiple Pods, request one zone for each replica group:

```yaml
roles:
- name: server
replicas: 2
size: 4
topology:
requiredLevel: topology.kubernetes.io/zone
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
spec:
roles:
- name: server
replicas: 2
size: 4
topology:
requiredLevel: topology.kubernetes.io/zone
```

This is a role fragment; the engine still needs its own distributed execution settings. Use
Expand Down Expand Up @@ -191,7 +194,7 @@ spec:

The `ModelDeployment` below names the binding, which grants this namespace access to the cache.
Choose member nodes, quotas, block size and dtype for your engine before applying these resources.
The [KV Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step.
The [Shared Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step.

### Model Delivery

Expand Down
18 changes: 11 additions & 7 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ The capability guides are [Heterogeneous Devices](modules/devices/_index.md),

For a first workload, follow [Usage](../README.md#usage), then use
[Accelerator Requests](modules/devices/requests.md) to choose its resource keys. The
[Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster.
[Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster.
Workloads that need RDMA also need [RDMA Operations](modules/rdma/operations.md).

For cluster operations, read [Architecture](getting-started/architecture.md) and [Installation
Expand Down Expand Up @@ -63,7 +63,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `rdma`
- Use for: requesting RDMA endpoints for a workload; how the NIC inventory, link states and labels are discovered; aligning requests with the kubelet TopologyManager policy.
- Aliases: `RDMA`, `NIC`, `RoCE`, `link state`, `rdma.capable`
- Guides: [RDMA Networking](modules/rdma/_index.md); keys and policy in [RDMA Operations](modules/rdma/operations.md); inventory in [Network Topology](modules/rdma/network-topology.md).
- Guides: [RDMA Networking](modules/rdma/_index.md); keys and policy in [RDMA Operations](modules/rdma/operations.md); inventory in [Network Topology](modules/rdma/network-topology.md); endpoints in [RDMA Network Endpoints Walkthrough](walkthroughs/rdma/network-endpoints.md).
- Specs: [NIC/RDMA topology](../specs/2026-09-02-devices-nic-rdma-topology.md) · [extended resource](../specs/2026-09-20-rdma-extended-resource.md) · [NVIDIA fabric domain](../specs/2026-09-20-nvidia-fabric-domain.md) · [fabric by protocol](../specs/2026-09-22-fabric-device-by-protocol.md) · [interface count](../specs/2026-09-22-fabric-interface-count.md)
- Code: NIC and link discovery in [pkg/devicemanager/detector](../pkg/devicemanager/detector/) (`network.go`, `link.go`) · request vocabulary in [pkg/nodefeature/rdma.go](../pkg/nodefeature/rdma.go) and [pkg/nodefeature/fabric.go](../pkg/nodefeature/fabric.go)
- Related: [Devices Manager](#devices-manager) (the interfaces are entries in the same per-node inventory)
Expand All @@ -74,7 +74,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `topology`
- Use for: describing domains such as zones and racks; letting Kueue place replicas within a domain; enabling a topology source; diagnosing a Pending group.
- Aliases: `TAS`, `Topograph`, `topology source`, `zone`, `placement`
- Guides: [Topology Management](modules/topology/_index.md); semantics in [Topology-Aware Scheduling](modules/topology/scheduling.md); enablement in [Topology-Aware Scheduling Operations](modules/topology/operations.md).
- Guides: [Topology Management](modules/topology/_index.md); semantics in [Topology-Aware Scheduling](modules/topology/scheduling.md); enablement in [Topology-Aware Scheduling Operations](modules/topology/operations.md); placement in [Hierarchical Placement Walkthrough](walkthroughs/topology/hierarchical-placement.md).
- Specs: [topology discovery](../specs/2026-09-22-topology-discovery.md) · [model placement preference](../specs/2026-09-26-model-placement-preference.md)
- Code: [pkg/worker/controllers/worker/topology_source.go](../pkg/worker/controllers/worker/topology_source.go) and [pkg/worker/controllers/worker/node_topology.go](../pkg/worker/controllers/worker/node_topology.go)
- Related: [Devices Manager](#devices-manager) (placement constrains which pool serves a replica) · [Model Deployment](#model-deployment) (replicas can prefer nodes holding their model)
Expand All @@ -85,7 +85,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `kv-cache`
- Use for: standing up a shared inference cache; pool grants and quotas; attaching a workload to a pool.
- Aliases: `KVCacheBackend`, `KVCachePool`, `Mooncake`, `prefix cache`, `injection`
- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [Shared Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
- Specs: [backend](../specs/2026-08-28-kv-cache-backend.md) · [pool](../specs/2026-08-28-kv-cache-pool.md) · [injection](../specs/2026-08-28-kv-cache-injection.md) · [media and scaling](../specs/2026-09-05-kv-cache-media-and-scaling.md) · [high availability](../specs/2026-09-06-kv-cache-backend-high-availability.md)
- Code: store rendering and read-back in [pkg/worker/kvcache](../pkg/worker/kvcache/) · per-engine client config in [pkg/worker/kvcache/inject](../pkg/worker/kvcache/inject/) · `kv_cache_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/)
- Related: [Model Deployment](#model-deployment) (deployments and Pods consume a pool through injection)
Expand All @@ -107,7 +107,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `model-deployment`
- Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics.
- Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica`
- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md).
- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); disaggregated serving in [Prefill and Decode Walkthrough](walkthroughs/model-deployment/prefill-decode.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md); multi-node serving in [Multi-Host Serving Walkthrough](walkthroughs/model-deployment/multi-host.md).
- Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [role configuration mutability](../specs/2026-10-08-model-deployment-role-mutability.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md)
- Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/)
- Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas)
Expand Down Expand Up @@ -239,12 +239,16 @@ A skill link names a procedure; the entry's other fields carry shared facts.
| Page | Description |
|---|---|
| [Walkthroughs](walkthroughs/_index.md) | Choose a complete example by module. |
| [Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. |
| [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. |
| [Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. |
| [RDMA Network Endpoints Walkthrough](walkthroughs/rdma/network-endpoints.md) | Request and verify RDMA endpoints and NUMA alignment. |
| [Hierarchical Placement Walkthrough](walkthroughs/topology/hierarchical-placement.md) | Place serving replicas across zones and physical racks. |
| [Shared Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. |
| [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md) | Cache weights and verify captured model CR output. |
| [Model Deployment Walkthroughs](walkthroughs/model-deployment/_index.md) | Choose a serving shape and prepare GPUs. |
| [Prefill and Decode Walkthrough](walkthroughs/model-deployment/prefill-decode.md) | Deploy prefill and decode roles fronted by router. |
| [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md) | Grow DP width with TP fixed and verify inference. |
| [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md) | Verify per-rank inference and Router coverage. |
| [Multi-Host Serving Walkthrough](walkthroughs/model-deployment/multi-host.md) | Scale models across multiple GPU nodes. |

### Start here

Expand Down
3 changes: 3 additions & 0 deletions docs/modules/devices/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,9 @@ runtime isolation method depend on the manufacturer.
physically partitioned requests. [Device Discovery](discovery.md) explains
how the per-node `Devices` record connects discovery to allocation.

Follow [Scheduling Walkthrough](../../walkthroughs/devices/scheduling.md) for
end-to-end device allocation and Kueue queuing.

## Device operations

Check a node before installation with [Preflight Operations](preflight.md). If you
Expand Down
1 change: 1 addition & 0 deletions docs/modules/devices/nvidia-mig.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ resources:
controller shapes it into the two keys above:

```yaml
apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: mig-instance
Expand Down
31 changes: 31 additions & 0 deletions docs/modules/devices/requests.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,10 @@ one container group.*
Accepted — both app containers claim the same family:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: trainer-pod
spec:
containers:
- name: trainer
Expand All @@ -158,6 +162,10 @@ spec:
Rejected — two families in one Pod:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: mixed-family-pod
spec:
containers:
- name: a
Expand All @@ -176,6 +184,10 @@ spec:
Rejected — the same family claimed in both container groups:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: split-group-pod
spec:
initContainers:
- name: warmup
Expand Down Expand Up @@ -295,6 +307,10 @@ resource sits deliberately outside the accelerator families.
Accepted:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: sliced-pod
spec:
containers:
- name: main
Expand All @@ -311,6 +327,10 @@ spec:
Rejected — two containers each holding a slice:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: multi-sliced-pod
spec:
containers:
- name: a
Expand All @@ -333,6 +353,10 @@ Accepted — the claim sits on an app container while the native sidecar carries
native sidecar carries the claim:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: accepted-sidecar-pod
spec: # accepted
initContainers:
- name: log-shipper
Expand All @@ -346,6 +370,10 @@ spec: # accepted
limits:
nvidia.com/gpu: "1"
---
apiVersion: v1
kind: Pod
metadata:
name: rejected-sidecar-pod
spec: # rejected
initContainers:
- name: log-shipper
Expand Down Expand Up @@ -459,7 +487,10 @@ into the keys above:
| `accelerator: "1"`<br/>`acceleratorPartitionedProfile: "3g.40gb"` | one hardware partition of that profile |

```yaml
apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: partitioned-instance
spec:
type: gpustack--nvidia-h100-80gb-hbm3-linux-amd64
resources:
Expand Down
3 changes: 3 additions & 0 deletions docs/modules/devices/thead-mig.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,10 @@ Or through the `Instance` API, as with [NVIDIA's
`acceleratorPartitionedProfile`](nvidia-mig.md#requesting-a-partition):

```yaml
apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: thead-instance
spec:
type: <a T-Head InstanceType from your pool>
resources:
Expand Down
4 changes: 2 additions & 2 deletions docs/modules/kv-cache/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ capacity they receive.

## Quick start

[KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and
[Shared Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and
deployment in order. A plain Pod can also use a binding through [KV Cache
Injection](injection.md).

Expand All @@ -26,4 +26,4 @@ tier](local-disk-tier.md) have separate operating details.
**See also** — [Model Deployment](../model-deployment/_index.md) (cache consumers) ·
[RDMA Networking](../rdma/_index.md) (transport)

**Next** → [KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache.
**Next** → [Shared Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache.
9 changes: 9 additions & 0 deletions docs/modules/kv-cache/backend.md
Original file line number Diff line number Diff line change
Expand Up @@ -735,6 +735,10 @@ other images may deregister later or wait differently. The operator controls the
the endpoint.

```yaml
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: mooncake-dram
spec:
connection:
managed:
Expand Down Expand Up @@ -763,6 +767,11 @@ own success signal does not tell you that happened.
(no Deployment, no Service, no DaemonSet) and only observes.

```yaml
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: external-mooncake
spec:
connection:
external:
endpoints:
Expand Down
17 changes: 11 additions & 6 deletions docs/modules/kv-cache/leader.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,12 +114,17 @@ The default `leader.electionBackend: Kubernetes` elects through a **Kubernetes L
replica count. The Lease is named `<backend>-leader` in the operator's namespace. Scale an already
elected leader by changing only its count:

```yaml
spec:
connection:
managed:
leader:
replicas: 3
```diff
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: shared-mooncake
spec:
connection:
managed:
leader:
- replicas: 1
+ replicas: 3
```

`leader.memberAddressing` is optional and only selects how members find the
Expand Down
27 changes: 19 additions & 8 deletions docs/modules/kv-cache/local-disk-tier.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ its own and takes its offload flags from this list's presence. The list holds on
not be attributed).

```yaml
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: shared-mooncake
spec:
connection:
managed:
Expand Down Expand Up @@ -244,14 +248,21 @@ Deleting a `KVCacheBackend` leaves the tier directory exactly as it was. Set
`members[].localDisks[].cleanAfterDelete: true` and the operator empties it as part of the deletion, on
every node that group's `nodeSelector` picks at the moment you delete it.

```yaml
members:
- medium: DRAM
capacityPerMember: 64Gi
localDisks:
- path: /var/lib/kvcache
capacity: 512Gi
cleanAfterDelete: true
```diff
apiVersion: worker.gpustack.ai/v1
kind: KVCacheBackend
metadata:
name: shared-mooncake
spec:
connection:
managed:
members:
- medium: DRAM
capacityPerMember: 500Gi
localDisks:
- path: /var/lib/kvcache
capacity: 4Ti
+ cleanAfterDelete: true
```

**It defaults to `false`, which is how every release before it behaved.** What is on that disk is
Expand Down
40 changes: 27 additions & 13 deletions docs/modules/model-delivery/artifact.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,12 +219,18 @@ Two consequences follow:

## Referencing it from a ModelDeployment

```yaml
spec:
model:
name: qwen-7b # the served name, unchanged
artifactRef: # a ModelArtifact in this namespace
name: qwen-7b
Set the reference when you create the deployment. The webhook refuses to add it later.

```diff
apiVersion: worker.gpustack.ai/v1
kind: ModelDeployment
metadata:
name: qwen-7b-serving
spec:
model:
name: qwen-7b # the served name, unchanged
+ artifactRef: # a ModelArtifact in this namespace
+ name: qwen-7b
```

`artifactRef` is frozen with the rest of `spec.model`: other weights are another deployment. A
Expand Down Expand Up @@ -397,13 +403,21 @@ timeout evicts and requeues the replica.

## Instance model volumes

```yaml
spec:
additionalVolumes:
- mountPath: /models/qwen
model:
artifactRef:
name: qwen-7b
`additionalVolumes` is immutable while the Instance runs.
Set it at creation, or set `spec.stop: true` first and edit it while the Instance is stopped.
See [Instance admission](../devices/admission.md).

```diff
apiVersion: worker.gpustack.ai/v1
kind: Instance
metadata:
name: workspace-demo
spec:
+ additionalVolumes:
+ - mountPath: /models/qwen
+ model:
+ artifactRef:
+ name: qwen-7b
```

The volume is always read-only and takes no `subPath`. A claim artifact's `path` is the sub-path and
Expand Down
4 changes: 4 additions & 0 deletions docs/modules/model-delivery/image-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ image read-only for the workload; packaging and pulling it adds disk and registr
## The source and its digest contract

```yaml
apiVersion: worker.gpustack.ai/v1
kind: ModelArtifact
metadata:
name: qwen
spec:
source:
image:
Expand Down
4 changes: 4 additions & 0 deletions docs/modules/model-deployment/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ its own replicas and resources.
the role contract. [Prefill and Decode](prefill-decode.md) covers split roles.
[Engine Versions](engine-versions.md) records tested version floors.

Follow [Prefill and Decode Walkthrough](../../walkthroughs/model-deployment/prefill-decode.md) for
disaggregated serving, and [Multi-Host Serving Walkthrough](../../walkthroughs/model-deployment/multi-host.md)
for distributed multi-node serving.

## Service operations

[Status](status.md), [Metrics](metrics.md) and
Expand Down
Loading
Loading