diff --git a/.agents/skills/gpustack-operator-docs/SKILL.md b/.agents/skills/gpustack-operator-docs/SKILL.md
index 66760a51f..cad6c3a83 100644
--- a/.agents/skills/gpustack-operator-docs/SKILL.md
+++ b/.agents/skills/gpustack-operator-docs/SKILL.md
@@ -61,7 +61,7 @@ one, not to widen the overview.
| The local disk tier itself: what it renders, its bucket, its eviction, its host directory | `docs/modules/kv-cache/local-disk-tier.md` |
| Configuring a node that is mostly disk: why a group is never disk alone, the thin-segment shape, the memory floor | `docs/modules/kv-cache/disk-heavy-nodes.md` |
| A `KVCachePool` or `KVCachePoolBinding`: the grant, the reuse domain, a quota ceiling or grant, what a full quota does | `docs/modules/kv-cache/pool.md` |
-| Standing a cache up end to end, or which object comes first: the pasteable four-object sequence | `docs/modules/kv-cache/walkthrough.md` |
+| Standing a cache up end to end, or which object comes first: the pasteable four-object sequence | `docs/walkthroughs/kv-cache/shared-cache.md` |
| How a **Pod** consumes a pool: the inject label and annotations, the injected keys per engine, a refusal, the isolation record | `docs/modules/kv-cache/injection.md` |
| A `ModelArtifact`: its sources, resolution and revalidation, the manifest digest, how a `ModelDeployment` or an `Instance` mounts or downloads it, claim placement, the weight identity in KV keys | `docs/modules/model-delivery/artifact.md` |
| The `image` source of a `ModelArtifact`: the digest contract, building weights into an image, image-volume delivery, the version floors, double storage, kubelet image GC, registry mirrors | `docs/modules/model-delivery/image-source.md` |
@@ -70,6 +70,9 @@ one, not to widen the overview.
| A `NodeModelStore` or the `model-manager` plugin: a field and its writer, the status guard, mount authorization, materialization, a failure reason, collection, a metric | `docs/modules/model-delivery/node-store.md` |
| Running node delivery: the chart values, where the node's configuration comes from, reading a node, the watermark cap, switching delivery, where replicas land and turning the preference off, upgrading, removing the cache | `docs/modules/model-delivery/operations.md` |
| The `ModelDeployment` contract: the inherited reuse domain, the three override tiers, the owned-key table, the runner-image formula, prefill/decode pairing, the topology-placement field contract | `docs/modules/model-deployment/deployment.md` |
+| A recorded Elastic EP expansion, its captured CR output and inference checks | `docs/walkthroughs/model-deployment/elastic-ep.md` |
+| A recorded model prefetch sequence and captured artifact, prefetch and node-store output | `docs/walkthroughs/model-delivery/prefetch.md` |
+| A fixed External DP group, its minimum arguments, per-rank requests, or measured Router coverage | `docs/walkthroughs/model-deployment/external-dp.md` |
| A `ModelDeployment` metrics snapshot, which series each field reads per engine, role and router, cache-hit scope or Pod scrape annotation | `docs/modules/model-deployment/metrics.md` |
| What a `ModelDeployment` status condition or published field means, and how to read them when a deployment misbehaves | `docs/modules/model-deployment/status.md` |
| How a prefill role and a decode role are paired: the connector each engine and router renders, `spec.router` and its fields, `spec.kvTransfer`, roles on different hardware, a role's own Service | `docs/modules/model-deployment/prefill-decode.md` |
@@ -89,7 +92,7 @@ one, not to widen the overview.
| A subcommand, one of its flags, its exit codes, its invocation | `docs/reference/commands.md` |
| A user-visible capability, a vendor's slicing support, the install flow | `README.md` |
| Vendor prerequisites, vendor GPU Operator coexistence | `docs/getting-started/vendor-prerequisites.md` |
-| A recorded run with real output | `docs/getting-started/walkthrough.md`, or the walkthrough section of `docs/modules/devices/nvidia-mig.md` |
+| A recorded run with real output | `docs/walkthroughs/devices/scheduling.md`, or the walkthrough section of `docs/modules/devices/nvidia-mig.md` |
A decision *record* — why an approach was chosen over another — belongs in `specs/`, not in `docs/`.
The docs state the rule that resulted; a `> **Why**` note carries only as much rationale as a reader
diff --git a/.agents/skills/gpustack-operator-docs/references/conventions.md b/.agents/skills/gpustack-operator-docs/references/conventions.md
index 0f7709c66..7421c2ff1 100644
--- a/.agents/skills/gpustack-operator-docs/references/conventions.md
+++ b/.agents/skills/gpustack-operator-docs/references/conventions.md
@@ -21,7 +21,7 @@ resources. [Architecture](../../getting-started/architecture.md) introduces the
---
-**See also** — [Device Discovery](discovery.md) · [Walkthrough](../../getting-started/walkthrough.md)
+**See also** — [Device Discovery](discovery.md) · [Walkthrough](../../walkthroughs/devices/scheduling.md)
**Next** → [Admission](admission.md) — the admission gates a request passes.
```
@@ -81,7 +81,7 @@ something that belongs in a list, a table, or another page.
| Cap | Limit | Exempt |
|---|---|---|
| prose paragraph | 5 lines / 500 rendered characters | fenced blocks, table rows, list *markers*, `> **Why**` notes, the footer |
-| page length | 1000 lines | `docs/getting-started/walkthrough.md`, `docs/operate/*`, `docs/modules/devices/*-mig.md` — recordings and runbooks |
+| page length | 1000 lines | `docs/walkthroughs/devices/scheduling.md`, `docs/operate/*`, `docs/modules/devices/*-mig.md` — recordings and runbooks |
| `##` sections | 10 per page, `## Contents` not counted | `docs/README.md`, `docs/reference/*` — both are lookup tables |
A paragraph nested under a list item is prose a reader still has to get through, so it is measured on
@@ -106,7 +106,7 @@ it as follows — when a page starts serving two modes at once, that is the mome
| Mode | Reader is… | Our pages |
|---|---|---|
-| Tutorial | learning by doing | `README.md` Usage examples, `docs/getting-started/walkthrough.md`, the MIG walkthrough, `docs/modules/kv-cache/walkthrough.md` |
+| Tutorial | learning by doing | `README.md` Usage examples, `docs/walkthroughs/devices/scheduling.md`, the MIG walkthrough, `docs/walkthroughs/kv-cache/shared-cache.md` |
| How-to | achieving a goal | `docs/operate/*`, `docs/modules/devices/*-mig.md`, `docs/operate/migration/*`, `docs/contribute/development.md`, a domain runbook (`docs/modules/model-delivery/operations.md`) |
| Reference | looking something up | `docs/modules/devices/requests.md`, `docs/reference/settings.md`, `docs/reference/*` |
| Explanation | building understanding | `docs/getting-started/architecture.md` and the deep pages under `docs/modules/` (contract pages read as reference, mechanism pages as explanation) |
diff --git a/.agents/skills/gpustack-operator-docs/references/page-map.md b/.agents/skills/gpustack-operator-docs/references/page-map.md
index f0f2e7176..6320a04bc 100644
--- a/.agents/skills/gpustack-operator-docs/references/page-map.md
+++ b/.agents/skills/gpustack-operator-docs/references/page-map.md
@@ -176,7 +176,7 @@ backend and links to it. And never the *enforcement* the Binding does not do: th
it is a grant and not an isolation boundary belongs here, but the mechanism a real boundary would need
is not this page's to design.
-## `docs/modules/kv-cache/walkthrough.md`
+## `docs/walkthroughs/kv-cache/shared-cache.md`
**Owns** — the ORDER, and nothing else: the four objects a working shared cache takes, one pasteable
manifest each, the check to run after each one, and the three configurations that read as working and
@@ -332,7 +332,7 @@ contract and does not teach.
**Never** — implementation. A rule's *enforcement point* is a link to `admission.md`.
-## `docs/getting-started/walkthrough.md`
+## `docs/walkthroughs/devices/scheduling.md`
**Owns** — recorded runs with real `kubectl` output and before/after comparisons.
@@ -428,3 +428,30 @@ supersession is recorded that way because it has no new spec to be recorded in.
`**Corrected after shipping.**`. A spec naming another spec by file name is the case that forces one:
when that file leaves the tree the name resolves to nothing, and leaving it alone preserves a pointer
rather than a record.
+
+## `docs/walkthroughs/model-deployment/elastic-ep.md`
+
+**Owns** — the user sequence for elastic TP2/DP2→4 expansion, captured CR output,
+short inference checks, log inspection, and application release. Distinguishes independent replicas
+from an elastic collective and states which path was measured.
+
+**Never** — private engine code, raw infrastructure identities, or broad support claims inferred from
+one runtime configuration. The Elastic EP guide owns the profile contract; the resize spec owns design analysis.
+
+## `docs/walkthroughs/model-deployment/external-dp.md`
+
+**Owns** — the fixed External DP serving sequence, minimum argument dependencies, captured CR output,
+per-rank inference checks, Router coverage limits, and application cleanup. Links shared model prefetch steps.
+
+**Never** — imply online collective resize from fixed-group inference, or claim all ranks receive Router
+traffic from HTTP success alone. P/D combinations remain with their owning guide and follow-up issue.
+
+## `docs/walkthroughs/model-delivery/prefetch.md`
+
+**Owns** — the shared model preparation example, its fixed model revision and captured artifact,
+prefetch, and node-store output. Model-serving walkthroughs link here before creating workloads.
+
+## `docs/walkthroughs/_index.md`
+
+**Owns** — the walkthrough entry point and module choices. Each module keeps its examples under
+its own directory; capability guides link to the relevant example.
diff --git a/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh b/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh
index b6b4a4f48..7380b707e 100755
--- a/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh
+++ b/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh
@@ -78,7 +78,7 @@ H2_CAP=10 # "##" sections in one page
# Cap exemptions, declared here rather than escaped inline. Each is a space-separated glob list.
# A recording and a runbook are as long as the hardware makes them.
-LINE_CAP_EXEMPT='docs/getting-started/walkthrough.md docs/operate/* docs/modules/devices/*-mig.md'
+LINE_CAP_EXEMPT='docs/walkthroughs/devices/scheduling.md docs/operate/* docs/modules/devices/*-mig.md'
# The index is a table of contents and a reference page is a lookup table: both are meant to be flat.
H2_CAP_EXEMPT='docs/README.md docs/reference/*'
diff --git a/README.md b/README.md
index dd8aef8d5..48f05047f 100644
--- a/README.md
+++ b/README.md
@@ -191,7 +191,7 @@ spec:
The `ModelDeployment` below names the binding, which grants this namespace access to the cache.
Choose member nodes, quotas, block size and dtype for your engine before applying these resources.
-The [KV Cache Walkthrough](./docs/modules/kv-cache/walkthrough.md) explains what to check at each step.
+The [KV Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step.
### Model Delivery
@@ -299,7 +299,7 @@ ssh -i ~/.ssh/id_ed25519 -p 2222 root@127.0.0.1
```
The SSH session enters the workload container, where the accelerator and workspace are available.
-The [walkthrough](./docs/getting-started/walkthrough.md) shows the resulting resources. Pods can
+The [walkthrough](./docs/walkthroughs/devices/scheduling.md) shows the resulting resources. Pods can
request accelerators directly; see
[Accelerator Requests](./docs/modules/devices/requests.md) for complete manifests.
diff --git a/docs/README.md b/docs/README.md
index f7ce9b0a8..e3aa4392e 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -13,8 +13,8 @@ The capability guides are [Heterogeneous Devices](modules/devices/_index.md),
For a first workload, follow [Usage](../README.md#usage), then use
[Accelerator Requests](modules/devices/requests.md) to choose its resource keys. The
-[Walkthrough](getting-started/walkthrough.md) records a run on a four-node cluster. Workloads that
-need RDMA also need [RDMA Operations](modules/rdma/operations.md).
+[Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster.
+Workloads that need RDMA also need [RDMA Operations](modules/rdma/operations.md).
For cluster operations, read [Architecture](getting-started/architecture.md) and [Installation
Modes](operate/installation-modes.md) first. The [settings](reference/settings.md), [vendor
@@ -27,6 +27,8 @@ and [RDMA](modules/rdma/operations.md) runbooks as those features are enabled. H
has separate runbooks for [NVIDIA](modules/devices/nvidia-mig.md),
[T-Head](modules/devices/thead-mig.md) and [Hygon](modules/devices/hygon-mig.md).
+For complete examples, use the [Walkthroughs](walkthroughs/_index.md) entry point.
+
Start an upgrade with the relevant page under [migration](operate/migration/to-subcharts.md).
For code changes, read [Architecture](getting-started/architecture.md), then the relevant deep page in the
@@ -83,7 +85,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `kv-cache`
- Use for: standing up a shared inference cache; pool grants and quotas; attaching a workload to a pool.
- Aliases: `KVCacheBackend`, `KVCachePool`, `Mooncake`, `prefix cache`, `injection`
-- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](modules/kv-cache/walkthrough.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
+- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md).
- Specs: [backend](../specs/2026-08-28-kv-cache-backend.md) · [pool](../specs/2026-08-28-kv-cache-pool.md) · [injection](../specs/2026-08-28-kv-cache-injection.md) · [media and scaling](../specs/2026-09-05-kv-cache-media-and-scaling.md) · [high availability](../specs/2026-09-06-kv-cache-backend-high-availability.md)
- Code: store rendering and read-back in [pkg/worker/kvcache](../pkg/worker/kvcache/) · per-engine client config in [pkg/worker/kvcache/inject](../pkg/worker/kvcache/inject/) · `kv_cache_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/)
- Related: [Model Deployment](#model-deployment) (deployments and Pods consume a pool through injection)
@@ -94,7 +96,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `model-delivery`
- Use for: giving model weights a stable identity; node caching and delivery; prefetch, peer sync, progress views and cache operations.
- Aliases: `ModelArtifact`, `NodeModelStore`, `ModelStore`, `ModelPrefetch`, `model cache`, `Hugging Face`
-- Guides: [Model Delivery](modules/model-delivery/_index.md); the artifact contract in [Model Artifact](modules/model-delivery/artifact.md); the node cache in [Node Model Store](modules/model-delivery/node-store.md); the runbook in [Model Store Operations](modules/model-delivery/operations.md).
+- Guides: [Model Delivery](modules/model-delivery/_index.md); the artifact contract in [Model Artifact](modules/model-delivery/artifact.md); the node cache in [Node Model Store](modules/model-delivery/node-store.md); the runbook in [Model Store Operations](modules/model-delivery/operations.md); a measured example in [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md).
- Specs: [model artifact](../specs/2026-09-25-model-artifact.md) · [node model store](../specs/2026-09-25-node-model-store.md) · [prefetch](../specs/2026-09-27-model-prefetch.md) · [peer sync](../specs/2026-09-27-model-peer-sync.md) · [image source](../specs/2026-09-27-model-image-source.md)
- Code: Hub resolution in [pkg/modelartifact](../pkg/modelartifact/) · the per-node plugin and cache in [pkg/modelmanager](../pkg/modelmanager/) · cache configuration in [pkg/modelstore](../pkg/modelstore/)
- Related: [Model Deployment](#model-deployment) (deployments mount or download artifacts) · [KV Cache](#kv-cache) (the weight identity enters KV keys) · [GPU Instances](#gpu-instances) (an Instance can mount weights)
@@ -105,7 +107,7 @@ A skill link names a procedure; the entry's other fields carry shared facts.
- ID: `model-deployment`
- Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics.
- Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica`
-- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md).
+- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md).
- Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md)
- Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/)
- Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas)
@@ -196,7 +198,6 @@ A skill link names a procedure; the entry's other fields carry shared facts.
| [KV Cache Local Disk Tier](modules/kv-cache/local-disk-tier.md) | Configure local disk storage. |
| [KV Cache on Disk-Heavy Nodes](modules/kv-cache/disk-heavy-nodes.md) | Size memory on disk-heavy nodes. |
| [KV Cache Pool](modules/kv-cache/pool.md) | Grant and limit cache use. |
-| [KV Cache Walkthrough](modules/kv-cache/walkthrough.md) | Create a working cache from backend to workload. |
| [KV Cache Injection Reference](modules/kv-cache/injection.md) | Attach a Pod to a pool. |
### Model delivery
@@ -233,12 +234,23 @@ A skill link names a procedure; the entry's other fields carry shared facts.
| [Accelerated Instances](modules/instances/_index.md) | Start an accelerator-backed workspace. |
| [Instance Metrics Reference](reference/instance-metrics.md) | Read an Instance’s resource use. |
+### Walkthroughs
+
+| Page | Description |
+|---|---|
+| [Walkthroughs](walkthroughs/_index.md) | Choose a complete example by module. |
+| [Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. |
+| [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. |
+| [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md) | Cache weights and verify captured model CR output. |
+| [Model Deployment Walkthroughs](walkthroughs/model-deployment/_index.md) | Choose a serving shape and prepare GPUs. |
+| [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md) | Grow DP width with TP fixed and verify inference. |
+| [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md) | Verify per-rank inference and Router coverage. |
+
### Start here
| Page | Description |
|---|---|
| [Architecture](getting-started/architecture.md) | See the operator’s four-stage path. |
-| [Walkthrough](getting-started/walkthrough.md) | Follow a recorded cluster run. |
| [Vendor Prerequisites](getting-started/vendor-prerequisites.md) | Prepare each manufacturer’s driver. |
### Cluster operations and upgrades
diff --git a/docs/getting-started/architecture.md b/docs/getting-started/architecture.md
index ee6116da9..727bd5a2d 100644
--- a/docs/getting-started/architecture.md
+++ b/docs/getting-started/architecture.md
@@ -106,7 +106,7 @@ allocation. [Admission](../modules/devices/admission.md#the-five-gates) owns the
its failure behavior.
InstanceType status reports the remaining capacity as workloads allocate and release devices. Use
-`kubectl get instancetype -w` to watch those changes. The [Walkthrough](walkthrough.md) shows the
+`kubectl get instancetype -w` to watch those changes. The [Walkthrough](../walkthroughs/devices/scheduling.md) shows the
request and its resulting Kubernetes resources.
## Vocabulary
@@ -126,7 +126,7 @@ The hardware terms Device, Accelerator and Resource are defined in
## Related documentation
-- [Walkthrough](walkthrough.md) — submit a workload and inspect the scheduling chain.
+- [Walkthrough](../walkthroughs/devices/scheduling.md) — submit a workload and inspect the scheduling chain.
- [GPU Instances](../modules/instances/_index.md) — configure an interactive container and SSH access.
- [Model Deployment](../modules/model-deployment/_index.md) — configure inference engines and replicas.
- [Model Delivery](../modules/model-delivery/_index.md) — resolve, cache and mount model weights.
@@ -138,4 +138,4 @@ The hardware terms Device, Accelerator and Resource are defined in
**See also** — [Accelerator Requests](../modules/devices/requests.md) ·
[Settings](../reference/settings.md) · [All documentation](../README.md)
-**Next** → [Walkthrough](walkthrough.md) — follow a workload from submission to allocation.
+**Next** → [Walkthrough](../walkthroughs/devices/scheduling.md) — follow a workload from submission to allocation.
diff --git a/docs/modules/devices/admission.md b/docs/modules/devices/admission.md
index 91fd3ddca..49b8be593 100644
--- a/docs/modules/devices/admission.md
+++ b/docs/modules/devices/admission.md
@@ -416,6 +416,6 @@ AdmissionCheck. The operator never leaves such a queue admitting — see
**See also** — [Accelerator Requests](requests.md) (the normative request contract) ·
[Device Discovery](discovery.md#the-device-plugin-allocator) (gate 5 in detail) ·
-[Walkthrough](../../getting-started/walkthrough.md) (the four views moving on a live cluster)
+[Walkthrough](../../walkthroughs/devices/scheduling.md) (the four views moving on a live cluster)
**Next** → [Installation Modes](../../operate/installation-modes.md) — how the chain gets deployed.
diff --git a/docs/modules/devices/nvidia-mig.md b/docs/modules/devices/nvidia-mig.md
index cb052cb94..900cf7bb3 100644
--- a/docs/modules/devices/nvidia-mig.md
+++ b/docs/modules/devices/nvidia-mig.md
@@ -344,7 +344,7 @@ It does create and destroy the instances backing scheduled workloads
## Walkthrough: three MIG configurations on one node
A recorded run on a live Kubernetes cluster, in the style of the
-[scheduling-chain walkthrough](../../getting-started/walkthrough.md): every command is the real `kubectl` (or on-node
+[scheduling-chain walkthrough](../../walkthroughs/devices/scheduling.md): every command is the real `kubectl` (or on-node
`nvidia-smi`) invocation and its real output, on `node-h100`, a genericized node of **eight** H100-80GB GPUs
running operator defaults. Eight is the point: the families are served by **disjoint GPU populations**, so
partitioning *some* GPUs moves exactly those.
@@ -782,4 +782,4 @@ gpustack--nvidia-h100-80gb-hbm3-linux-amd64 gpustack-fnv64-e4768a65ca0ce96b
"what can I still get") · [Device Discovery](discovery.md#the-partitioned-family-fungible-tokens)
(how a partition is placed and reclaimed)
-**Next** → [Walkthrough](../../getting-started/walkthrough.md) — the logical-slicing counterpart on a live cluster.
+**Next** → [Walkthrough](../../walkthroughs/devices/scheduling.md) — the logical-slicing counterpart on a live cluster.
diff --git a/docs/modules/devices/requests.md b/docs/modules/devices/requests.md
index 3eab020ae..d43037b8a 100644
--- a/docs/modules/devices/requests.md
+++ b/docs/modules/devices/requests.md
@@ -529,7 +529,7 @@ A manifest asking `"3"` for a heavier share of one accelerator now needs a node
`InstanceType.status.acceleratorShared.onceMaxRequest` moved with it: it counts a node's accelerators
with a free share, where it used to sum their shares. Output recorded before the change, in the
-[Walkthrough](../../getting-started/walkthrough.md) and [NVIDIA MIG Operations](./nvidia-mig.md), still shows the
+[Walkthrough](../../walkthroughs/devices/scheduling.md) and [NVIDIA MIG Operations](./nvidia-mig.md), still shows the
old reading under `SH` — `10/10` on one free accelerator now reads `1/10`.
## Limitations
@@ -596,4 +596,4 @@ partitioning) · [Admission](./admission.md) (where these keys are checked) ·
served) · [RDMA Operations](../rdma/operations.md) (how many RDMA endpoints to ask for beside N
accelerators, and the kubelet policy that aligns the two sides)
-**Next** → [Walkthrough](../../getting-started/walkthrough.md) — the same requests on a live cluster.
+**Next** → [Walkthrough](../../walkthroughs/devices/scheduling.md) — the same requests on a live cluster.
diff --git a/docs/modules/devices/scheduling.md b/docs/modules/devices/scheduling.md
index f13ad3f5c..fb852c433 100644
--- a/docs/modules/devices/scheduling.md
+++ b/docs/modules/devices/scheduling.md
@@ -365,7 +365,7 @@ The per-accelerator **AdmissionCheck**, third of the five gates; its behavior is
**See also** — [Device Discovery](discovery.md) (where the capacity signals come from) ·
[Topology-Aware Scheduling](../topology/scheduling.md) (how topology profiles enter this chain) ·
-[Walkthrough](../../getting-started/walkthrough.md) (the same objects on a live cluster) ·
+[Walkthrough](../../walkthroughs/devices/scheduling.md) (the same objects on a live cluster) ·
[Settings](../../reference/settings.md#online-adjustable-settings)
**Next** → [Admission](admission.md) — the five gates a request passes.
diff --git a/docs/modules/kv-cache/_index.md b/docs/modules/kv-cache/_index.md
index 0a4fb700d..587f26032 100644
--- a/docs/modules/kv-cache/_index.md
+++ b/docs/modules/kv-cache/_index.md
@@ -10,7 +10,7 @@ capacity they receive.
## Quick start
-[KV Cache Walkthrough](walkthrough.md) creates the backend, pool, binding and
+[KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and
deployment in order. A plain Pod can also use a binding through [KV Cache
Injection](injection.md).
@@ -26,4 +26,4 @@ tier](local-disk-tier.md) have separate operating details.
**See also** — [Model Deployment](../model-deployment/_index.md) (cache consumers) ·
[RDMA Networking](../rdma/_index.md) (transport)
-**Next** → [KV Cache Walkthrough](walkthrough.md) — create a shared cache.
+**Next** → [KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache.
diff --git a/docs/modules/model-delivery/_index.md b/docs/modules/model-delivery/_index.md
index e9a8e6015..3498c6b9e 100644
--- a/docs/modules/model-delivery/_index.md
+++ b/docs/modules/model-delivery/_index.md
@@ -3,6 +3,8 @@
`ModelArtifact` gives a model version a stable identity. Its delivery can be handled by the engine,
a node cache, a PVC or an image.
+Follow the [Model Prefetch Walkthrough](../../walkthroughs/model-delivery/prefetch.md) for a complete example with captured CR output.
+
## Contents
- [Sources and delivery](#sources-and-delivery)
diff --git a/docs/modules/model-deployment/_index.md b/docs/modules/model-deployment/_index.md
index e92c76ea7..ba3d51761 100644
--- a/docs/modules/model-deployment/_index.md
+++ b/docs/modules/model-deployment/_index.md
@@ -20,8 +20,11 @@ the role contract. [Prefill and Decode](prefill-decode.md) covers split roles.
[Routing](routing.md) explain what the running service reports and how requests
are directed.
-`Elastic EP` is its own mode rather than another role shape: it keeps one serving instance and
-changes the size of its collective while it runs.
+[Elastic EP](elastic-ep.md) keeps one serving instance and changes its collective while it runs.
+[Elastic EP Walkthrough](../../walkthroughs/model-deployment/elastic-ep.md) follows prefetch, TP2/DP2 startup, and DP2→4 expansion
+with captured resource output and completed inference requests.
+[External DP Walkthrough](../../walkthroughs/model-deployment/external-dp.md) runs a fixed group with one HTTP endpoint per rank
+and checks managed Router coverage.
---
diff --git a/docs/modules/model-deployment/elastic-ep.md b/docs/modules/model-deployment/elastic-ep.md
index be454326c..f2a13f79f 100644
--- a/docs/modules/model-deployment/elastic-ep.md
+++ b/docs/modules/model-deployment/elastic-ep.md
@@ -1,7 +1,7 @@
# Elastic EP
An elastic expert-parallel deployment keeps one vLLM serving instance and changes its GPU collective
-while it runs. The operator adds or removes GPU members inside one Ray cluster.
+while it runs. The operator adds GPU members inside one Ray cluster.
The engine reconfigures while Kubernetes, Kueue, Ray and accelerator accounting converge.
## Contents
@@ -55,13 +55,15 @@ is why the master is counted inside `width` and is never retired.
## Requirements
-The recorded runtime baseline uses a vLLM 0.29.0 image with a full Ray installation. The
-official vLLM image of that version ships without Ray, and the profile will not start on it. The
-pinned requirement in the resize spec is Ray 2.56.1 with the full default closure.
+Use a vLLM image with a full Ray installation. Check the installed engine and Ray versions
+before starting the profile.
The TP size must fit the selected MoE model, engine kernels, and GPUs available on one node.
The operator does not limit TP to the size used by a particular test.
-Static TP=2 inference has passed; online TP>1 resizing remains under investigation.
+
+The [Elastic EP Walkthrough](../../walkthroughs/model-deployment/elastic-ep.md) records successful TP2/DP2 startup and DP2→4
+expansion using the official GPUStack runner and DeepSeek-V2-Lite-Chat on an eight-card RTX PRO 6000 node.
+That result is limited to its pinned software, model, configuration, and short inference checks.
Admission of TP=2, 4 or 8 does not establish dynamic inference support.
The operator directly manages a separate CPU Ray head Pod and its Service.
@@ -110,7 +112,7 @@ spec:
replicas: 1
size: 1
instanceType: elastic-gpu
- image: your-registry/vllm-ray:0.29.0
+ image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
elasticEp:
width: 2
ports:
diff --git a/docs/modules/model-deployment/prefill-decode.md b/docs/modules/model-deployment/prefill-decode.md
index 38da40cb4..2c335d0b8 100644
--- a/docs/modules/model-deployment/prefill-decode.md
+++ b/docs/modules/model-deployment/prefill-decode.md
@@ -19,6 +19,11 @@ decode](deployment.md#prefill-and-decode) describes the role fields and admissio
Without `spec.router`, `kind` adds the role discriminator to the engine's shared-store connector and
nothing pairs the two roles.
+For a multi-Pod P/D replica, managed HTTP routing selects its leader (`member-index=0`).
+A router policy such as `round_robin` chooses among those leaders; it does not add non-HTTP members.
+External DP inside a Prefill or Decode role has not been validated.
+[Issue #754](https://github.com/gpustack/gpustack-operator/issues/754) tracks that deferred combination.
+
With the managed `llm-d-router`, the connector a native vLLM role runs depends on whether
`spec.kvCache` is set; the two settings render two different connector documents, not one document
with a field toggled:
diff --git a/docs/modules/model-deployment/routing.md b/docs/modules/model-deployment/routing.md
index 84d1e3097..61338e83f 100644
--- a/docs/modules/model-deployment/routing.md
+++ b/docs/modules/model-deployment/routing.md
@@ -1,10 +1,13 @@
# Model Deployment Routing
-A router only chooses when a role has more than one replica. Every routing choice measured here ran
+A managed router selects among the HTTP endpoints of serving replicas. Every routing choice measured here ran
on a server role; on a prefill/decode pair, where each half is chosen separately, none has been run.
+See [Prefill and Decode](prefill-decode.md) for the External DP boundary within P/D roles.
+
## Contents
+- [Which Pods receive traffic](#which-pods-receive-traffic)
- [Each router's default](#each-routers-default)
- [Prefix affinity](#prefix-affinity)
- [Switching to round robin](#switching-to-round-robin)
@@ -12,6 +15,31 @@ on a server role; on a prefill/decode pair, where each half is chosen separately
- [Confirming that a Router serves](#confirming-that-a-router-serves)
- [Per-worker routing metrics](#per-worker-routing-metrics)
+## Which Pods receive traffic
+
+The engine's LB mode determines the candidate endpoints. The Router policy chooses among those candidates.
+`round_robin` does not change the endpoint selector.
+
+| Deployment shape | HTTP candidates |
+|---|---|
+| Internal Server group | Member 0 of each replica. |
+| Qualified External Server group | Every qualified member; each member serves one DP rank. |
+| Prefill or Decode group | Member 0 of each replica, including groups with `size > 1`. |
+| Unknown mode or unverified group | Keep the existing leader restriction; do not expand discovery. |
+
+For an eligible External Server role, Router discovery and ordinary Services omit `member-index=0`.
+They retain the endpoint eligibility label, so an unhealthy group cannot receive new traffic.
+The replica's headless Service always publishes every member for peer communication.
+
+Known non-Internal modes follow the External selector rule. Status retains the engine's distinct mode name.
+Hybrid GPU behavior remains untested. See [Prefill and Decode](prefill-decode.md) for P/D limitations.
+
+Managed roles expose one fixed HTTP port per Pod, defaulting to 8000.
+All routed roles must use the same serving port. A shared custom port remains supported.
+Admission rejects vLLM's MultiPort mode because it starts endpoints on additional, incremented ports.
+A decoder proxy's internal backend port remains separate from its published HTTP port.
+See the [External DP walkthrough](../../walkthroughs/model-deployment/external-dp.md) for a minimal per-rank configuration.
+
## Each router's default
**`vllm-router` and `sglang-gateway` route by `cache_aware`, and `llm-d-router` scores every
diff --git a/docs/reference/instance-type-unit-resources.md b/docs/reference/instance-type-unit-resources.md
index 3ebfff88d..c54123016 100644
--- a/docs/reference/instance-type-unit-resources.md
+++ b/docs/reference/instance-type-unit-resources.md
@@ -255,6 +255,6 @@ at the top of that file; when either changes, compare the rows here with the pre
**See also** — [Scheduling Chain](../modules/devices/scheduling.md#unit-spec-defaults)
(where the unit spec is stamped) · [Admission](../modules/devices/admission.md#the-instancetype-and-instance-webhooks)
-(what the webhooks enforce on it) · [Walkthrough](../getting-started/walkthrough.md#4-managing-a-custom-instancetype)
+(what the webhooks enforce on it) · [Walkthrough](../walkthroughs/devices/scheduling.md#4-managing-a-custom-instancetype)
**Next** → [Settings](settings.md#online-adjustable-settings) — the switches named on this page.
diff --git a/docs/reference/settings.md b/docs/reference/settings.md
index 78058dcc8..f73ebb9cd 100644
--- a/docs/reference/settings.md
+++ b/docs/reference/settings.md
@@ -397,4 +397,4 @@ user-facing knobs, listed for completeness.
[Installation Modes](../operate/installation-modes.md) (which flags a mode sets for you) ·
[High Availability Operations](../operate/high-availability.md)
-**Next** → [Walkthrough](../getting-started/walkthrough.md) — a setting flipped on a live cluster, before and after.
+**Next** → [Walkthrough](../walkthroughs/devices/scheduling.md) — a setting flipped on a live cluster, before and after.
diff --git a/docs/walkthroughs/_index.md b/docs/walkthroughs/_index.md
new file mode 100644
index 000000000..467cb2796
--- /dev/null
+++ b/docs/walkthroughs/_index.md
@@ -0,0 +1,34 @@
+# Walkthroughs
+
+Follow a complete example, then compare the resource output with the observed result.
+Each module has its own directory. Configuration contracts remain in the capability guides.
+
+## Contents
+
+- [Choose a walkthrough](#choose-a-walkthrough)
+- [Read the evidence](#read-the-evidence)
+
+## Choose a walkthrough
+
+| Module | Walkthrough | Task |
+|---|---|---|
+| Devices | [Device Scheduling](devices/scheduling.md) | Follow detection, queues, admission, and accelerator allocation. |
+| KV Cache | [Shared Cache](kv-cache/shared-cache.md) | Create a backend, pool, namespace grant, and serving workload. |
+| Model Delivery | [Model Prefetch](model-delivery/prefetch.md) | Cache and verify model weights before inference. |
+| Model Deployment | [Preparation and choices](model-deployment/_index.md) | Choose the serving shape and prepare GPU resources. |
+| Model Deployment | [Elastic EP](model-deployment/elastic-ep.md) | Grow DP width from 2 to 4 with TP fixed at 2. |
+| Model Deployment | [External DP](model-deployment/external-dp.md) | Serve fixed DP ranks and verify Router distribution. |
+
+## Read the evidence
+
+Commands sit beside captured output or the checks needed to interpret it.
+A CR's declared configuration does not prove runtime parallelism or successful inference.
+The model-serving walkthroughs pair CR snapshots with completed requests and native runtime measurements.
+Their validation records state software identities and limits.
+
+---
+
+**See also** — [Architecture](../getting-started/architecture.md) ·
+[Model Deployment](../modules/model-deployment/_index.md)
+
+**Next** → [Device Scheduling Walkthrough](devices/scheduling.md)
diff --git a/docs/getting-started/walkthrough.md b/docs/walkthroughs/devices/scheduling.md
similarity index 95%
rename from docs/getting-started/walkthrough.md
rename to docs/walkthroughs/devices/scheduling.md
index 3e0131e84..9581afaef 100644
--- a/docs/getting-started/walkthrough.md
+++ b/docs/walkthroughs/devices/scheduling.md
@@ -1,4 +1,4 @@
-# Walkthrough
+# Device Scheduling Walkthrough
This page records a real session: every `kubectl` invocation and its real output, objects as YAML
trimmed to `metadata.labels` / `spec` / `status`, and a before / after `kubectl get instancetypes`
@@ -10,14 +10,14 @@ until section 5 flips it on.
Watch three columns: **UNIT(CPU/RAM)/STORAGE**, the per-unit request the InstanceType charges; **CPU**,
the collapsed CPU pool's `remaining/capacity` cores; **ACCELERATOR(EX/SH/SL/PT)**, the
-`onceMaxRequest/remaining` of each [four-view](../modules/devices/admission.md#four-view-status)
+`onceMaxRequest/remaining` of each [four-view](../../modules/devices/admission.md#four-view-status)
projection. This run predates the shared card-count reading, so its `SH` once-max figures are
-shares, not accelerators ([Pre-release breaks](../modules/devices/requests.md#pre-release-breaks)).
+shares, not accelerators ([Pre-release breaks](../../modules/devices/requests.md#pre-release-breaks)).
Each accelerator counts in exactly one of `EX`/`SH`/`SL` (unpartitioned) or `PT` (partitioned), so
`0/0` under `PT` throughout means none is in a partitioning mode. For the all-partitioned and **mixed**
configurations see the [three-configuration
-walkthrough](../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node).
+walkthrough](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node).
## Contents
@@ -415,17 +415,17 @@ Deleting the Instance releases the slice: the row returns to `1/1 10/10 100/100
> instead hard-partitions into fixed hardware instances the operator materializes on demand:
>
> - a different resource family (`.partitioned*`, reported under `PT`), with a different request shape —
-> keys and rules in [Accelerator Requests](../modules/devices/requests.md);
+> keys and rules in [Accelerator Requests](../../modules/devices/requests.md);
> - MIG *mode* is the administrator's, driven with `nvidia-smi`, so it has its own runbook and a recorded
> enable → request → reclaim → disable walkthrough in
-> [NVIDIA MIG Operations](../modules/devices/nvidia-mig.md).
+> [NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md).
---
## 4. Managing a custom InstanceType
A derived pool is sized from a **per-product preset**: 8 CPU / 64 GiB for this A10G, 4 CPU / 16 GiB for
-an unrecognised accelerator ([preset reference](../reference/instance-type-unit-resources.md)). For
+an unrecognised accelerator ([preset reference](../../reference/instance-type-unit-resources.md)). For
another size, an admin authors an InstanceType referencing a catalog flavor by its `acceleratorGroup`,
with a unit spec of its own:
@@ -666,10 +666,10 @@ spec:
- **They mount into the workload container only.** The SSH sidecar needs no change: it enters that
container's mount namespace per session, so every mount is visible over SSH too.
- **A persistent claim places the Instance where it can attach.** The workspace claim and every
- `persistent` entry follow the [claim placement rules](../modules/model-delivery/artifact.md#claim-delivery-and-placement):
+ `persistent` entry follow the [claim placement rules](../../modules/model-delivery/artifact.md#claim-delivery-and-placement):
a node-local volume pins the Instance to its node and that node's capacity; network or `ReadWriteMany`
storage avoids the pin. A claim nothing binds creates no Pod and says why in `status.phaseMessage`;
- [`instance-persistent-volume-placement`](../reference/settings.md#online-adjustable-settings) turns this off.
+ [`instance-persistent-volume-placement`](../../reference/settings.md#online-adjustable-settings) turns this off.
- **Both new fields are immutable while the Instance runs**, editable while stopped — the rule the rest
of `spec` follows.
@@ -682,7 +682,7 @@ kept separate so node-path mounts can be allowed without a container escape:
| `instance-host-path-volume-allowed` | `spec.additionalVolumes[*].hostPath` — reaches the node's filesystem, but not its devices or kernel. |
Each gates taking its escape, so turning one off stops new grants without stranding an Instance
-that already holds one; [Settings](../reference/settings.md#online-adjustable-settings) has the exact terms.
+that already holds one; [Settings](../../reference/settings.md#online-adjustable-settings) has the exact terms.
Both govern the **node** boundary, not the namespace one: a `persistent`, `configMap` or `secret`
source names an object in the Instance's own namespace and may always be mounted, the same reach a Pod
@@ -691,9 +691,9 @@ other's Secrets in their own.
---
-**See also** — [Accelerator Requests](../modules/devices/requests.md) (the contract behind step 3) ·
-[NVIDIA MIG Operations](../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node)
-(hardware partitioning) · [Settings](../reference/settings.md#online-adjustable-settings) (the two gates)
+**See also** — [Accelerator Requests](../../modules/devices/requests.md) (the contract behind step 3) ·
+[NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node)
+(hardware partitioning) · [Settings](../../reference/settings.md#online-adjustable-settings) (the two gates)
-**Next** → [NVIDIA MIG Operations](../modules/devices/nvidia-mig.md) — hardware partitioning, from enabling
+**Next** → [NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md) — hardware partitioning, from enabling
the mode to reclaiming the instance.
diff --git a/docs/modules/kv-cache/walkthrough.md b/docs/walkthroughs/kv-cache/shared-cache.md
similarity index 85%
rename from docs/modules/kv-cache/walkthrough.md
rename to docs/walkthroughs/kv-cache/shared-cache.md
index dae9b7653..42fc72b94 100644
--- a/docs/modules/kv-cache/walkthrough.md
+++ b/docs/walkthroughs/kv-cache/shared-cache.md
@@ -26,7 +26,7 @@ name in the manifests below, then apply them in that order.
The order is fixed because each object names the one above it, so creating them the other way round
leaves references that resolve to nothing. Why the two scopes split this way is on
-[KV Cache Pool](pool.md#two-kinds-split-by-scope).
+[KV Cache Pool](../../modules/kv-cache/pool.md#two-kinds-split-by-scope).
**The Binding is the authorization point.** A namespace gets access to a store when an administrator
creates a Binding in it. A user naming a pool is not a path this API has: `poolRef` is a
@@ -54,7 +54,7 @@ spec:
`spec.image` is left unset on purpose: the cluster-wide `kv-cache-backend-image`
[Setting](../../reference/settings.md) supplies this project's own build, which is the one that can elect a leader
later. Naming a published upstream image here works until Step 4 and then does not. See
-[High availability](leader.md#high-availability).
+[High availability](../../modules/kv-cache/leader.md#high-availability).
`capacityPerMember` is charged to each member Pod's host memory request, so it is a claim on the node
and not a hint. One member Pod runs per node the selector matches.
@@ -62,7 +62,7 @@ and not a hint. One member Pod runs per node the selector matches.
`leader: {}` takes the field defaults, `multiTenancy` included (it defaults on), so this master keeps
a per-tenant quota ledger and Steps 2 and 3 read against it. A backend pinned to a store image from
before Mooncake 0.3.12 is the one exception and declares `multiTenancy: false` out loud. See
-[The project's own build variants](backend.md#the-projects-own-build-variants).
+[The project's own build variants](../../modules/kv-cache/backend.md#the-projects-own-build-variants).
Wait for it, then read what it reports:
@@ -117,7 +117,7 @@ the master; the name is what keeps the two apart.
**A quota ceiling is not a reservation.** It is the most this namespace may hold at once, and going
over it does not fail a write. See
-[Full-quota behavior](pool.md#full-quota-behavior).
+[Full-quota behavior](../../modules/kv-cache/pool.md#full-quota-behavior).
**The ceiling is enforced because the Step-1 leader carries its tenant ledger, which the default
gave it.** A backend declared `leader.multiTenancy: false` holds no ledger instead: its pool is
@@ -126,7 +126,7 @@ admitted with a warning, the Binding reports `QuotaGranted=True` with reason `Un
**A multi-tenant master refuses a tenant name absent from its ledger.** An engine that ignores the
injected tenant then needs a second Binding whose domain is `default`, or that leaves `name` out.
-See [Tenant compatibility](injection.md#tenant-compatibility).
+See [Tenant compatibility](../../modules/kv-cache/injection.md#tenant-compatibility).
## Step 3: the workload
@@ -159,12 +159,12 @@ useful comparison to have run once before attributing anything to the pool.
**The engine is configured by injection, not by anything you write here.** The operator resolves the
Binding, reads the backend's endpoint and the domain, and injects the store's environment into every
role's Pod. What is injected per engine, and every refusal, is on
-[KV Cache Injection](injection.md).
+[KV Cache Injection](../../modules/kv-cache/injection.md).
**The store version must match the engine's client.** The engine version above decides the client,
and Step 1's unset `spec.image` leaves the store on the Settings default. A pair on different lines
starts healthy and then fails every write, so the version above is a choice. See
-[The store version must match the engine's client](backend.md#the-store-version-must-match-the-engines-client).
+[The store version must match the engine's client](../../modules/kv-cache/backend.md#the-store-version-must-match-the-engines-client).
Check the attachment from the deployment's own status rather than from the Pods:
@@ -192,12 +192,12 @@ as the old leader steps down. Both readings are normal.
**Scaling from one to three keeps the first leader Pod and the member templates** because the Lease
election and accounts were already present at one replica. See
-[the leader Deployment](leader.md#the-deployment-and-the-two-probes) for the
+[the leader Deployment](../../modules/kv-cache/leader.md#the-deployment-and-the-two-probes) for the
cost of changing `leader.electionBackend`.
**`leader.memberAddressing` chooses how members find the master**, and defaults to
`Service`. An explicit `Lease` value uses the member's API access to read the current holder. See
-[High availability](leader.md#high-availability) for the measured failover limits.
+[High availability](../../modules/kv-cache/leader.md#high-availability) for the measured failover limits.
These conditions apply at one replica too; the phase does not summarize them:
@@ -218,7 +218,7 @@ misses afterwards. Plan for a cold cache after every failover and every leader r
**The store's snapshot is not offered, and its flags are refused in `leader.extraArgs`**, because
restoring a snapshot can make the cache serve another key's bytes instead of a miss.
-[High availability](leader.md#high-availability) says why.
+[High availability](../../modules/kv-cache/leader.md#high-availability) says why.
## Failure modes
@@ -235,11 +235,11 @@ reading on this page most likely to be escalated as an outage.
---
-**See also** — [KV Cache Backend](backend.md) (every field of the store, and what status reports) ·
-[KV Cache Leader](leader.md) (the election, why there is no snapshot, and the member addressing choice in full) ·
-[KV Cache Pool](pool.md) (quota, domains and what a full quota does) ·
-[Model Deployment](../model-deployment/deployment.md) (roles, prefill/decode, rollout) ·
-[Model Deployment Prefill and Decode](../model-deployment/prefill-decode.md) (router, transfer) ·
-[KV Cache Injection](injection.md) (what a Pod actually receives)
+**See also** — [KV Cache Backend](../../modules/kv-cache/backend.md) (every field of the store, and what status reports) ·
+[KV Cache Leader](../../modules/kv-cache/leader.md) (the election, why there is no snapshot, and the member addressing choice in full) ·
+[KV Cache Pool](../../modules/kv-cache/pool.md) (quota, domains and what a full quota does) ·
+[Model Deployment](../../modules/model-deployment/deployment.md) (roles, prefill/decode, rollout) ·
+[Model Deployment Prefill and Decode](../../modules/model-deployment/prefill-decode.md) (router, transfer) ·
+[KV Cache Injection](../../modules/kv-cache/injection.md) (what a Pod actually receives)
-**Next** → [KV Cache Pool](pool.md) — the quota this walkthrough set once and did not explain.
+**Next** → [KV Cache Pool](../../modules/kv-cache/pool.md) — the quota this walkthrough set once and did not explain.
diff --git a/docs/walkthroughs/model-delivery/prefetch.md b/docs/walkthroughs/model-delivery/prefetch.md
new file mode 100644
index 000000000..e2c3a7db3
--- /dev/null
+++ b/docs/walkthroughs/model-delivery/prefetch.md
@@ -0,0 +1,164 @@
+# Model Prefetch Walkthrough
+
+Cache DeepSeek-V2-Lite-Chat on one GPU node before starting inference.
+This shared preparation serves both the Elastic EP and External DP walkthroughs.
+The output below was captured with node model delivery enabled.
+
+## Contents
+
+- [Prerequisites](#prerequisites)
+- [Step 1: create the artifact and prefetch](#step-1-create-the-artifact-and-prefetch)
+- [Step 2: verify the cached model](#step-2-verify-the-cached-model)
+- [Step 3: start a serving workload](#step-3-start-a-serving-workload)
+- [Step 4: release the cache](#step-4-release-the-cache)
+
+## Prerequisites
+
+Enable [node model delivery](../../modules/model-delivery/operations.md) and select the GPU node.
+You need `kubectl`, `jq`, access to cluster-scoped model stores, and enough disk for the weights.
+The example uses a dedicated namespace and store grant.
+
+## Step 1: create the artifact and prefetch
+
+Save the following as `model-delivery.yaml`. Replace both `gpu-node` values with your GPU node's
+`kubernetes.io/hostname` label. The store grant and the prefetch placement must select that same node.
+The fixed file list resolves to 12 files and 31,418,807,311 bytes.
+
+```yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+ name: gpustack-elastic
+---
+apiVersion: worker.gpustack.ai/v1
+kind: ModelArtifact
+metadata:
+ name: deepseek-v2-lite-chat
+ namespace: gpustack-elastic
+spec:
+ source:
+ huggingFace:
+ repository: deepseek-ai/DeepSeek-V2-Lite-Chat
+ revision: 85864749cd611b4353ce1decdb286193298f64c7
+ allowPatterns:
+ - model-00001-of-000004.safetensors
+ - model-00002-of-000004.safetensors
+ - model-00003-of-000004.safetensors
+ - model-00004-of-000004.safetensors
+ - config.json
+ - configuration_deepseek.py
+ - generation_config.json
+ - model.safetensors.index.json
+ - modeling_deepseek.py
+ - tokenization_deepseek_fast.py
+ - tokenizer.json
+ - tokenizer_config.json
+---
+apiVersion: worker.gpustack.ai/v1
+kind: ModelStore
+metadata:
+ name: rtx6000-models
+spec:
+ nodeSelector:
+ matchLabels:
+ kubernetes.io/hostname: gpu-node
+---
+apiVersion: worker.gpustack.ai/v1
+kind: ModelStoreBinding
+metadata:
+ name: rtx6000-models
+ namespace: gpustack-elastic
+spec:
+ storeRefs:
+ - name: rtx6000-models
+ quota:
+ bytes: 64Gi
+ allowPinned: true
+---
+apiVersion: worker.gpustack.ai/v1
+kind: ModelPrefetch
+metadata:
+ name: deepseek-v2-lite-chat
+ namespace: gpustack-elastic
+spec:
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ bindingRef:
+ name: rtx6000-models
+ placement:
+ nodeSelector:
+ matchLabels:
+ kubernetes.io/hostname: gpu-node
+ minReady: 1
+ retention:
+ pinned: true
+```
+
+## Step 2: verify the cached model
+
+```bash
+kubectl apply -f model-delivery.yaml
+kubectl -n gpustack-elastic get modelartifacts,modelprefetches
+kubectl -n gpustack-elastic get modelprefetch deepseek-v2-lite-chat -o yaml
+kubectl get nodemodelstores -o json
+```
+
+Captured `kubectl get modelartifacts,modelprefetches` output:
+
+```text
+NAME SOURCE REVISION SIZE READY DOWNLOADING RESOLVED
+modelartifact.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-ai/DeepSeek-V2-Lite-Chat 85864749cd61 31418807311 1 0 True
+
+NAME ARTIFACT BINDING READY DESIRED AVAILABLE
+modelprefetch.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-v2-lite-chat rtx6000-models 1 1 True
+```
+
+Check the matching node's model entry as well:
+
+```bash
+kubectl get nodemodelstores -o json |
+ jq '.items[] | .status.models[]? | {digest, sizeBytes, state}'
+```
+
+Captured projection:
+
+```json
+{
+ "digest": "sha256:1369aae0b5cc2d0e260557bcea04b0d6aad40f8bb6accdac60f1ee543887d9f7",
+ "sizeBytes": 31418807311,
+ "state": "Ready"
+}
+```
+
+The validation also retained the successful warm-up Pod and compared its 12 SHA-256 results with
+the resolved manifest. Warm-up Pods are transient; collect their logs before completion if you need
+that evidence. See [Model Prefetch](../../modules/model-delivery/prefetch.md).
+Wait for the required content before creating the serving workload.
+
+## Step 3: start a serving workload
+
+Continue with [Elastic EP](../model-deployment/elastic-ep.md) to grow one collective,
+or [External DP](../model-deployment/external-dp.md) to route requests across fixed DP ranks.
+Use the namespace and artifact created here in either manifest.
+
+## Step 4: release the cache
+
+Keep the prefetch while a serving application needs the cached weights.
+After deleting those consumers, remove the dedicated resources:
+
+```bash
+kubectl -n gpustack-elastic delete modelprefetch deepseek-v2-lite-chat
+kubectl -n gpustack-elastic delete modelstorebinding rtx6000-models
+kubectl -n gpustack-elastic delete modelartifact deepseek-v2-lite-chat
+kubectl delete modelstore rtx6000-models
+```
+
+Cache collection follows the [retention rules](../../modules/model-delivery/prefetch.md).
+Deleting these CRs does not delete the Kubernetes cluster or its disks.
+
+---
+
+**See also** — [Model Delivery](../../modules/model-delivery/_index.md) ·
+[Model Prefetch](../../modules/model-delivery/prefetch.md) (configuration and retention)
+
+**Next** → [Model Deployment Walkthroughs](../model-deployment/_index.md)
diff --git a/docs/walkthroughs/model-deployment/_index.md b/docs/walkthroughs/model-deployment/_index.md
new file mode 100644
index 000000000..494706680
--- /dev/null
+++ b/docs/walkthroughs/model-deployment/_index.md
@@ -0,0 +1,94 @@
+# Model Deployment Walkthroughs
+
+Choose a serving shape, prepare its model, then follow the matching walkthrough.
+Both examples use the official GPUStack runner and retain captured CR output and inference results.
+
+## Contents
+
+- [Choose an example](#choose-an-example)
+- [Prepare the cluster](#prepare-the-cluster)
+- [Prepare the GPU InstanceType](#prepare-the-gpu-instancetype)
+- [Check the software identity](#check-the-software-identity)
+
+## Choose an example
+
+| Walkthrough | Serving shape | What was measured |
+|---|---|---|
+| [Elastic EP](elastic-ep.md) | One Internal LB instance on Ray | TP2, DP width 2→4, EP world 4→8; ordinary and streaming requests before and after expansion. |
+| [External DP](external-dp.md) | Two fixed DP ranks using multiprocessing | TP2, DP2, EP world4; direct requests to each rank and Router distribution across both ranks. |
+
+A Router can front either shape. The engine's LB mode determines which Pods accept inference requests.
+Increasing `roles[].replicas` adds complete groups; see [role scaling](../../modules/model-deployment/deployment.md#rollout-behavior).
+That operation was not measured in these walkthroughs.
+
+## Prepare the cluster
+
+Install the operator and enable node model delivery using the
+[installation guide](../../operate/installation-modes.md) and
+[model-store operations](../../modules/model-delivery/operations.md).
+Use `kubectl`, `jq`, `curl`, and a kubeconfig for the intended cluster.
+The recorded cluster had one CPU node and one node with eight NVIDIA RTX PRO 6000 Blackwell GPUs.
+
+```bash
+kubectl get nodes
+kubectl get devices -o json
+kubectl get instancetypes
+kubectl get nodemodelstores -o json
+```
+
+Confirm model-manager readiness and free GPUs before creating a serving workload.
+External DP needs four free cards. Elastic EP needs four initially and eight after expansion.
+Complete the [model prefetch walkthrough](../model-delivery/prefetch.md) before starting either example.
+
+## Prepare the GPU InstanceType
+
+Save the following as `gpu-instance-type.yaml` and apply it once.
+For another GPU family, select an appropriate whole-GPU InstanceType and verify model compatibility.
+Each serving member requests two GPUs; CPU and memory budgets scale with that count.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: InstanceType
+metadata:
+ name: elastic-cpu-gpu
+spec:
+ generalGroup: generic
+ acceleratable: true
+ acceleratorGroup: nvidia-rtx-pro-6000-blackwell-server-edition
+ os: linux
+ arch: amd64
+ localStorage: 10Gi
+ unitResources:
+ cpu: "4"
+ ram: 16Gi
+```
+
+```bash
+kubectl apply -f gpu-instance-type.yaml
+kubectl get instancetype elastic-cpu-gpu
+```
+
+Both examples refer to `elastic-cpu-gpu`. Replace that name if you use an existing InstanceType.
+
+## Check the software identity
+
+Read the running images and binary versions. Repeat for the worker, device-manager, and model-manager Pods.
+Set the namespace, Pod, and container names from your installation:
+
+```bash
+kubectl get pods -A
+kubectl -n "$OPERATOR_NAMESPACE" get pod "$OPERATOR_POD" -o json |
+ jq '.status.containerStatuses[] | {name, image, imageID}'
+kubectl -n "$OPERATOR_NAMESPACE" exec "$OPERATOR_POD" -c "$OPERATOR_CONTAINER" -- \
+ gpustack-operator --version
+```
+
+The `dev` tag can move. Each walkthrough's validation record identifies the binary and image used.
+The serving manifests pin the official runner digest; no engine packages were replaced during validation.
+
+---
+
+**See also** — [Model Deployment](../../modules/model-deployment/_index.md) (configuration guides) ·
+[Model Prefetch Walkthrough](../model-delivery/prefetch.md)
+
+**Next** → [Elastic EP Walkthrough](elastic-ep.md) or [External DP Walkthrough](external-dp.md)
diff --git a/docs/walkthroughs/model-deployment/elastic-ep.md b/docs/walkthroughs/model-deployment/elastic-ep.md
new file mode 100644
index 000000000..6d6a81ea1
--- /dev/null
+++ b/docs/walkthroughs/model-deployment/elastic-ep.md
@@ -0,0 +1,375 @@
+# Elastic EP Walkthrough
+
+Start one elastic vLLM instance and grow its expert-parallel collective on the same eight-GPU node.
+This example holds TP=2 and expands DP width from 2 to 4, so EP world grows from 4 to 8.
+Captured CR output appears beside each check.
+
+## Contents
+
+- [Prerequisites](#prerequisites)
+- [Step 1: start the elastic deployment](#step-1-start-the-elastic-deployment)
+- [Step 2: verify baseline inference](#step-2-verify-baseline-inference)
+- [Step 3: expand and verify inference](#step-3-expand-and-verify-inference)
+- [Step 4: release the application](#step-4-release-the-application)
+- [Diagnose a stalled resize](#diagnose-a-stalled-resize)
+- [Validation record](#validation-record)
+
+## Prerequisites
+
+Complete [cluster and GPU preparation](_index.md#prepare-the-cluster) and
+[model prefetch](../model-delivery/prefetch.md).
+The [Elastic EP guide](../../modules/model-deployment/elastic-ep.md) owns the profile's configuration and limits.
+
+Keep `replicas: 1`, `size: 1`, and TP=2 throughout this walkthrough.
+Width 2 consumes four whole GPUs. Width 4 consumes eight.
+This release supports width increases; width decreases are refused.
+The engine uses Internal LB, and only its master accepts inference requests.
+
+## Step 1: start the elastic deployment
+
+Save this as `elastic.yaml`. Use the artifact and InstanceType from the preparation steps.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: tp2-elastic
+ namespace: gpustack-elastic
+spec:
+ model:
+ name: /models
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ engine:
+ name: vLLM
+ version: 0.29.0
+ roles:
+ - name: server
+ kind: Server
+ replicas: 1
+ size: 1
+ instanceType: elastic-cpu-gpu
+ image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
+ imagePullPolicy: IfNotPresent
+ resources:
+ accelerator: "2"
+ elasticEp:
+ width: 2
+ extraArgs:
+ - --tensor-parallel-size
+ - "2"
+ - --max-model-len
+ - "4096"
+ - --max-num-seqs
+ - "32"
+ - --enforce-eager
+ - --all2all-backend
+ - allgather_reducescatter
+ - --api-server-count
+ - "1"
+ env:
+ - name: VLLM_USE_RAY_V2_EXECUTOR_BACKEND
+ value: "1"
+ - name: RAY_LOGGER_LEVEL
+ value: debug
+ - name: RAY_BACKEND_LOG_LEVEL
+ value: debug
+ - name: RAY_DEDUP_LOGS
+ value: "0"
+```
+
+```bash
+kubectl apply -f elastic.yaml
+kubectl -n gpustack-elastic get modeldeployments,pods,services
+```
+
+The operator creates one CPU-only Ray head without resource requests or limits.
+Normal Kubernetes admission and scheduling still apply.
+In the recorded run, this head landed on the GPU node but claimed no GPU. The two GPU Pods each exposed two GPUs to Ray autodiscovery.
+
+With `shmSize` omitted, the recorded `/dev/shm` capacity was 16 GiB on every elastic Pod.
+Ray's object store used that mount. A separate CPU-only probe verified explicit `shmSize: 32Gi`.
+Kubelet can cap the mount by node allocatable memory or Pod memory limits; read the actual mount.
+See [Shared memory](../../modules/model-deployment/deployment.md#shared-memory) before changing the budget.
+
+The recorded test also installed a read-only diagnostic extension to measure native parallel groups.
+That test instrumentation is omitted from this application manifest.
+
+This manifest has one serving instance and no router. Only its master exposes the inference API;
+the other GPU members participate through Ray.
+A separate recorded run added this `spec.router` block before creating the deployment:
+
+```yaml
+router:
+ name: vllm-router
+ image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db
+ extraArgs:
+ - --policy
+ - round_robin
+```
+
+The engine still uses Internal LB. This Router forwards to the master, which distributes work to the DP engines.
+Use the [Router observation steps](external-dp.md#step-4-check-router-coverage) with deployment name `tp2-elastic`.
+For this Elastic shape, expect one master endpoint; exclude the CPU head and other GPU members.
+
+## Step 2: verify baseline inference
+
+Save each CR snapshot before changing the width:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > baseline-cr.json
+jq '{name: .metadata.name, width: .spec.roles[0].elasticEp.width,
+ phase: .status.phase, roles: [.status.roles[] |
+ {name, desired, quotaReserved, ready, parallelism}]}' baseline-cr.json
+```
+
+Projection of the captured baseline CR:
+
+```json
+{
+ "name": "tp2-elastic",
+ "width": 2,
+ "phase": "Ready",
+ "roles": [
+ {
+ "name": "server",
+ "desired": 1,
+ "quotaReserved": 1,
+ "ready": 1,
+ "parallelism": {
+ "declared": {
+ "tensorParallel": 2
+ },
+ "loadBalance": "Internal",
+ "source": {
+ "complete": true,
+ "kind": "ExtraArgs"
+ }
+ }
+ }
+ ]
+}
+```
+
+The role still has one serving instance. `width` is desired state, and `parallelism.declared` describes
+arguments. Neither field proves the runtime's effective DP width.
+The captured conditions included `ReplicasUpToDate=Unknown` with reason `RolloutNotObserved`;
+`phase: Ready` did not turn every condition true.
+
+In a separate terminal, forward the serving Service:
+
+```bash
+kubectl -n gpustack-elastic port-forward service/tp2-elastic 8000:8000
+```
+
+Send both request types. Keep the headers and response bodies. Each command makes one attempt.
+The recorded test sent equivalent requests directly to the master inside the cluster;
+the port-forward is a workstation access option.
+
+```bash
+curl --fail-with-body --max-time 30 -D ordinary.headers \
+ http://127.0.0.1:8000/v1/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"/models","prompt":"Complete the sentence: The sky is","max_tokens":8,"temperature":0,"stream":false}' \
+ -o ordinary.json
+jq '{text: .choices[0].text, finish_reason: .choices[0].finish_reason,
+ completion_tokens: .usage.completion_tokens}' ordinary.json
+
+curl --fail-with-body --no-buffer --max-time 30 -D stream.headers \
+ http://127.0.0.1:8000/v1/completions \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"/models","prompt":"Complete the sentence: The sky is","max_tokens":8,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}' \
+ -o stream.txt
+cat stream.txt
+```
+
+Require HTTP 200, nonempty generated text, positive completion tokens, and a final generation reason.
+For SSE, require the usage event and `data: [DONE]`. A health response does not exercise generation.
+The recorded requests used unique request IDs and checked their response IDs as well.
+
+## Step 3: expand and verify inference
+
+Record the master Pod's UID and restart count before the patch. Select ordinal zero within this deployment:
+
+```bash
+kubectl -n gpustack-elastic get pods -o json |
+ jq '.items[] | select(.metadata.labels["app.kubernetes.io/instance"] == "tp2-elastic") |
+ select(.metadata.labels["app.kubernetes.io/component"] == "server") |
+ select(.metadata.labels["modeldeployment.gpustack.ai/pod-ordinal"] == "0") |
+ {name: .metadata.name, uid: .metadata.uid,
+ containers: [.status.containerStatuses[] | {name, containerID, restartCount}]}'
+
+kubectl -n gpustack-elastic patch modeldeployment tp2-elastic --type=json \
+ -p '[{"op":"test","path":"/spec/roles/0/elasticEp/width","value":2},{"op":"replace","path":"/spec/roles/0/elasticEp/width","value":4}]'
+```
+
+Wait for the operator's observation of the current generation to reach width 4 at every layer.
+Read the diagnostic ConfigMap belonging to this deployment UID:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > current-cr.json
+MD_UID=$(jq -r '.metadata.uid' current-cr.json)
+MD_GENERATION=$(jq -r '.metadata.generation' current-cr.json)
+kubectl -n gpustack-elastic get configmaps -o json |
+ jq --arg uid "$MD_UID" '[.items[] |
+ select(any(.metadata.ownerReferences[]?; .uid == $uid)) |
+ select(.data["observation.json"] != null) |
+ .data["observation.json"] | fromjson] |
+ if length == 1 then .[0] else error("expected one observation") end' > observation.json
+jq -e --argjson generation "$MD_GENERATION" '
+ .observedGeneration == $generation and
+ ([.desired, .admitted, .allocated, .ray, .effective] |
+ all(.Known == true and .Value == 4))' observation.json
+```
+
+If this check fails, continue observing or inspect the hold reason before declaring completion.
+The ConfigMap is internal diagnostic state, not a public CR status API.
+Its effective-width observation includes rank-directed inference probes.
+Pair it with the retained master identity and both client request checks below.
+
+After the observation passes, save the final CR:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > scaled-cr.json
+jq '{name: .metadata.name, width: .spec.roles[0].elasticEp.width,
+ phase: .status.phase, desired: .status.roles[0].desired,
+ ready: .status.roles[0].ready}' scaled-cr.json
+```
+
+Projection captured after the resize completed:
+
+```json
+{
+ "name": "tp2-elastic",
+ "width": 4,
+ "phase": "Ready",
+ "desired": 1,
+ "ready": 1
+}
+```
+
+Confirm four GPU members and one head. Read the master and Ray logs, then repeat both inference requests.
+The engine may withdraw readiness and return 503 during reconfiguration.
+Do not resend the width change while the current operation is unresolved.
+
+## Step 4: release the application
+
+When you have finished inspecting the elastic application, delete only its ModelDeployment:
+
+```bash
+kubectl -n gpustack-elastic delete modeldeployment tp2-elastic --wait=true
+kubectl -n gpustack-elastic get modeldeployments,pods,services
+kubectl get devices -o json > released-devices.json
+jq '[.items[].status.groups[].accelerators[] |
+ {mode: (.mode // 0), remaining, allocatedSlices: (.allocatedSlices // 0)}] |
+ {cards: length, free: map(select(.mode == 0 and .remaining == 1600000 and .allocatedSlices == 0)) | length}' \
+ released-devices.json
+```
+
+Captured projection after deleting the official-runner Elastic application with its Router:
+
+```json
+{
+ "cards": 8,
+ "free": 8
+}
+```
+
+That release check also verified deletion of the owned head, GPU Pods, Router, Services, and Workload.
+For a shared cluster, compare the application's captured device claims with the ledger;
+other workloads may keep cards occupied.
+
+ModelPrefetch keeps the model pinned independently of the serving application.
+When no consumer needs the weights, remove the prefetch and its dedicated binding, artifact, and store.
+See [Model Prefetch](../../modules/model-delivery/prefetch.md) for cache retention and collection behavior.
+
+Application deletion leaves the Kubernetes nodes, disks, and cloud network in place.
+Retain them while the owner is inspecting logs. Delete infrastructure only after the owner approves,
+using the provisioner's recorded resource ownership, and verify deletion in the provider and state file.
+
+## Diagnose a stalled resize
+
+Read the CR conditions and the owned Pods first. Select the head, master, or worker from the Pod list:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o yaml
+kubectl -n gpustack-elastic get pods -o wide
+kubectl -n gpustack-elastic get events --sort-by=.metadata.creationTimestamp
+kubectl -n gpustack-elastic logs "$POD" -c main --timestamps --tail=200
+kubectl -n gpustack-elastic exec "$POD" -c main -- \
+ sh -c 'ls /tmp/ray/session_latest/logs'
+kubectl -n gpustack-elastic exec "$POD" -c main -- \
+ sh -c 'tail -200 /tmp/ray/session_latest/logs/raylet.out /tmp/ray/session_latest/logs/raylet.err'
+```
+
+| Symptom | Read next |
+|---|---|
+| GPU Pods remain Pending | Kueue Workloads, queue quota, Pod events, InstanceType capacity, and the per-card `Devices` ledger. |
+| Placement groups exist but inference times out | Master logs, EngineCore actor state, GPU worker logs, and Ray session logs. `CREATED` placement groups alone do not establish inference readiness. |
+| Ray sees the wrong GPU count | Actual allocation annotations, container-visible GPUs, and live Ray node capacity. Each GPU member must expose exactly TP GPUs. |
+| Unexpected engine arguments or missing Ray | Running image digest and installed vLLM/Ray versions. `spec.engine.version` does not inspect the image. |
+| Shared-memory pressure | Actual `/dev/shm` capacity, Pod memory limits, node allocatable memory, and the object store's location. |
+
+`RAY_LOGGER_LEVEL` controls Python Ray logging; `RAY_BACKEND_LOG_LEVEL` controls backend logging.
+The operator defaults both levels to `info`.
+The manifest sets both to `debug` and disables deduplication for inspection.
+Environment changes affect newly created Pods. Preserve current logs before replacing a failing deployment.
+
+## Validation record
+
+The recorded software and model identities were:
+
+| Component | Recorded identity |
+|---|---|
+| Operator | `gpustack/gpustack-operator:dev`, binary revision `d00f8fa8c98334a71af4c6eb302ee80d08289309` |
+| Operator image digest | `sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36` |
+| Inference engine | vLLM 0.29.0 with full Ray 2.54.0 |
+| Engine image | `gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91` |
+| Model | `deepseek-ai/DeepSeek-V2-Lite-Chat` |
+| Model revision | `85864749cd611b4353ce1decdb286193298f64c7` |
+
+The official GPUStack runner includes Ray. The run used its installed packages without replacement.
+Container reads found `cupy-cuda13x` 14.2.0 and no `cupy-cuda12x` installation.
+Verify the image digest and installed vLLM/Ray versions when selecting another runner.
+Do not add `--elastic-ep-max-dp-size` to this example; that argument starts with vLLM 0.30.0.
+
+Recorded operator binary output:
+
+```text
+gpustack-operator version dev (d00f8fa8c98334a71af4c6eb302ee80d08289309)
+```
+
+The recorded result combined native group measurements, actor state, GPU allocation, and completed requests:
+
+| Check | Width 2 | Width 4 |
+|---|---|---|
+| Actual TP / DP / EP world | 2 / 2 / 4 | 2 / 4 / 8 |
+| Distinct allocated GPU UUIDs | 4 | 8 |
+| Live EngineCore actors | 2 | 4 |
+| Live GPU worker actors | 4 | 8 |
+| Ordinary request | HTTP 200, 8 generated tokens | HTTP 200, 8 generated tokens |
+| SSE request | HTTP 200, 8 generated tokens, `[DONE]` | HTTP 200, 8 generated tokens, `[DONE]` |
+
+All four recorded requests succeeded on their first attempt. The original master UID and HTTP process
+identity remained unchanged. The measurements joined native workers to Ray actors, Pod identities,
+and host GPU UUIDs. Placement groups used `STRICT_PACK`, with two GPU bundles per DP engine.
+
+The additional Router run also completed width 2→4 with the same official runner.
+Before and after expansion, its selector and registry contained only the original master.
+Each stage completed one ordinary request and one SSE request through the Router, with HTTP 200 and eight tokens.
+The master's Router counter increased by two at each stage. Worker and CPU-head endpoints were absent.
+
+These checks do not establish sustained concurrency, uninterrupted requests throughout resizing,
+per-rank request coverage, or inference quality. The response's `system_fingerprint` retained its startup DP2 text after
+expansion, so it was not used to establish the effective width.
+
+An earlier run reached `CREATED` placement groups and eight live GPU workers, but new EngineCore actors
+remained pending and requests timed out. This recorded run completed expansion and inference.
+The earlier root cause remains unconfirmed; the result does not isolate shared memory or head placement as its cause.
+
+---
+
+**See also** — [Elastic EP](../../modules/model-deployment/elastic-ep.md) (profile contract) ·
+[External DP Walkthrough](external-dp.md) (fixed per-rank serving)
+
+**Next** → [Model Deployment Status](../../modules/model-deployment/status.md)
diff --git a/docs/walkthroughs/model-deployment/external-dp.md b/docs/walkthroughs/model-deployment/external-dp.md
new file mode 100644
index 000000000..27e6f53ee
--- /dev/null
+++ b/docs/walkthroughs/model-deployment/external-dp.md
@@ -0,0 +1,348 @@
+# External DP Walkthrough
+
+Run one fixed vLLM data-parallel (DP) group with an HTTP endpoint on each rank.
+This walkthrough records two ranks using the official GPUStack runner.
+Direct requests and managed Router requests passed on both ranks after the selector fix.
+Follow the steps to verify both engine inference and Router coverage.
+
+## Contents
+
+- [Deployment shape](#deployment-shape)
+- [Prerequisites](#prerequisites)
+- [Step 1: start the fixed group](#step-1-start-the-fixed-group)
+- [Step 2: read the running resources](#step-2-read-the-running-resources)
+- [Step 3: send requests to each rank](#step-3-send-requests-to-each-rank)
+- [Step 4: check Router coverage](#step-4-check-router-coverage)
+- [Understand the scaling limits](#understand-the-scaling-limits)
+- [Step 5: inspect and release the application](#step-5-inspect-and-release-the-application)
+- [Validation record](#validation-record)
+
+## Deployment shape
+
+This example uses one Server role with `replicas: 1` and `size: 2`.
+Each Pod has two GPUs and runs one DP rank with tensor parallelism (TP)=2.
+With expert parallelism (EP) enabled, the collective has four GPU workers: TP2, DP2, EP world4.
+The deployment consumes four whole GPUs.
+
+External LB means each DP rank accepts its own requests.
+A load balancer must distribute requests across those HTTP endpoints.
+The `round_robin` policy chooses among discovered endpoints; it cannot repair a missing endpoint.
+
+This example uses multiprocessing (`mp`) for both DP and local TP execution.
+The runner contains Ray, but this fixed group does not start Ray.
+See [vLLM's DP deployment guide](https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment/)
+for the engine's Internal and External LB distinction.
+
+## Prerequisites
+
+Complete [cluster and GPU preparation](_index.md#prepare-the-cluster) and
+[model prefetch](../model-delivery/prefetch.md).
+Use the shared namespace, cached artifact, model revision, and GPU InstanceType.
+The GPU node needs four free cards and the verified model cache.
+
+Each Pod needs a separate Pod IP. The ranks must reach each other over the Pod network.
+The DP RPC port defaults to 29550; collective communication also uses other ports.
+Setting an engine RPC port does not create a Kubernetes Service or a firewall rule.
+
+## Step 1: start the fixed group
+
+Save this as `external-dp.yaml`. Replace `elastic-cpu-gpu` with your GPU InstanceType if needed.
+The model delivery steps already created `deepseek-v2-lite-chat` in `gpustack-elastic`.
+
+```yaml
+apiVersion: worker.gpustack.ai/v1
+kind: ModelDeployment
+metadata:
+ name: external-fixed
+ namespace: gpustack-elastic
+spec:
+ model:
+ name: /models
+ artifactRef:
+ name: deepseek-v2-lite-chat
+ engine:
+ name: vLLM
+ version: "0.29.0"
+ roles:
+ - name: server
+ kind: Server
+ replicas: 1
+ size: 2
+ instanceType: elastic-cpu-gpu
+ image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91
+ imagePullPolicy: IfNotPresent
+ resources:
+ accelerator: "2"
+ extraArgs:
+ - --tensor-parallel-size
+ - "2"
+ - --data-parallel-size
+ - "2"
+ - --data-parallel-rank
+ - $(GPUSTACK_MEMBER_INDEX)
+ - --data-parallel-address
+ - $(GPUSTACK_REPLICA_LEADER_ADDRESS)
+ - --enable-expert-parallel
+ - --max-model-len
+ - "4096"
+ - --max-num-seqs
+ - "32"
+ - --enforce-eager
+ - --all2all-backend
+ - allgather_reducescatter
+ router:
+ name: vllm-router
+ image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db
+ extraArgs:
+ - --policy
+ - round_robin
+```
+
+The operator supplies the member index and replica leader address as environment variables.
+Kubernetes expands their references in the container arguments.
+Rank 0 and rank 1 therefore share the leader address and receive distinct rank values.
+The ordinary Server role still requires explicit TP, DP, rank, and address arguments.
+It does not infer those arguments from `size` or GPU requests.
+
+The length, sequence, eager-execution, and all-to-all arguments preserve the recorded model settings.
+They do not select External LB.
+The measured run also mounted a diagnostic worker extension to read native groups and GPU identities.
+That test instrumentation is omitted from this serving manifest.
+
+```bash
+kubectl apply --dry-run=server -f external-dp.yaml
+kubectl apply -f external-dp.yaml
+kubectl -n gpustack-elastic get modeldeployment external-fixed
+kubectl -n gpustack-elastic get pods \
+ -l app.kubernetes.io/instance=external-fixed
+```
+
+### Arguments you can omit
+
+The minimal run omitted all six arguments below and still completed ordinary and streaming requests on both ranks.
+These defaults apply to this vLLM version and deployment shape.
+
+| Omitted argument | Why it is unnecessary here |
+|---|---|
+| `--data-parallel-external-lb` | An explicit `--data-parallel-rank` selects External LB. |
+| `--data-parallel-size-local 1` | External LB runs one local DP rank. |
+| `--api-server-count 1` | External LB defaults to one API server per rank. |
+| `--data-parallel-backend mp` | `mp` is the default DP backend. |
+| `--distributed-executor-backend mp` | The local TP group fits the Pod's two GPUs and runs outside Ray. |
+| `--data-parallel-rpc-port 13345` | Both ranks can use the default RPC port, 29550. |
+
+The operator generates the ordinary HTTP serving port, model mount, Pod wiring, and Services.
+This example serves each rank on the same fixed port, 8000.
+
+A single API server process and a single HTTP port are different constraints.
+Internal LB may use several API server processes behind one port.
+Keep the engine default for Internal LB; do not force `--api-server-count 1` on every configuration.
+Managed roles reject MultiPort mode because its additional listeners are outside this endpoint contract.
+
+`--data-parallel-backend` selects execution placement; it does not select the LB mode.
+`--data-parallel-start-rank` alone is not an External LB declaration.
+
+## Step 2: read the running resources
+
+Retain the CR output beside the request results:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment external-fixed -o json |
+ jq '{name: .metadata.name, phase: .status.phase,
+ size: .spec.roles[0].size, replicas: .spec.roles[0].replicas,
+ accelerator: .spec.roles[0].resources.accelerator}'
+```
+
+Recorded values from the captured CR:
+
+```json
+{
+ "name": "external-fixed",
+ "phase": "Ready",
+ "size": 2,
+ "replicas": 1,
+ "accelerator": "2"
+}
+```
+
+The captured status also reported one ready replica group and two serving HTTP endpoints:
+
+```bash
+kubectl -n gpustack-elastic get modeldeployment external-fixed -o json |
+ jq '.status.roles[] | {name, ready, loadBalance: .parallelism.loadBalance, endpoints}'
+```
+
+```json
+{
+ "name": "server",
+ "ready": 1,
+ "loadBalance": "External",
+ "endpoints": {
+ "eligible": 1,
+ "serving": {"state": "Confirmed", "value": 2}
+ }
+}
+```
+
+Here `eligible` counts replica groups; the serving observation counts HTTP endpoints.
+Keep the full CR with `kubectl -n gpustack-elastic get modeldeployment external-fixed -o yaml`.
+Read Pod UIDs, images, addresses, and endpoint labels separately:
+
+```bash
+kubectl -n gpustack-elastic get pods \
+ -l app.kubernetes.io/instance=external-fixed -o json |
+ jq '.items[] | {name: .metadata.name, uid: .metadata.uid,
+ labels: .metadata.labels, podIP: .status.podIP,
+ containers: .status.containerStatuses}'
+kubectl get devices -o json
+```
+
+The recorded native checks found:
+
+| Pod member | DP rank | TP size | DP size | EP world | Distinct GPUs |
+|---|---|---|---|---|---|
+| 0 | 0 | 2 | 2 | 4 | 2 |
+| 1 | 1 | 2 | 2 | 4 | 2 |
+
+All four worker GPU identities were distinct. Both ranks reported External LB enabled and Elastic EP disabled.
+Declared arguments and `Ready` status alone do not establish these native results.
+
+## Step 3: send requests to each rank
+
+Open one port-forward per rank in separate terminals:
+
+```bash
+kubectl -n gpustack-elastic port-forward pod/external-fixed-server-r0-m0 18000:8000
+```
+
+```bash
+kubectl -n gpustack-elastic port-forward pod/external-fixed-server-r0-m1 18001:8000
+```
+
+Send an ordinary request and a streaming request to each endpoint:
+
+```bash
+for port in 18000 18001; do
+ curl --fail-with-body --max-time 30 "http://127.0.0.1:$port/v1/completions" \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"/models","prompt":"The sky is","max_tokens":8,"temperature":0}'
+ curl --fail-with-body --no-buffer --max-time 30 "http://127.0.0.1:$port/v1/completions" \
+ -H 'Content-Type: application/json' \
+ -d '{"model":"/models","prompt":"A GPU is","max_tokens":8,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}'
+done
+```
+
+The measured requests used direct Pod addresses inside the cluster. Each request had one attempt.
+Both ranks returned HTTP 200, nonempty text, eight completion tokens, and a valid finish reason.
+Both streams ended with `data: [DONE]`.
+These short checks establish functional inference, not throughput or sustained-load behavior.
+
+## Step 4: check Router coverage
+
+Discover the Router Pod and forward its API and metrics ports:
+
+```bash
+ROUTER_POD=$(kubectl -n gpustack-elastic get pods \
+ -l app.kubernetes.io/instance=external-fixed -o json |
+ jq -r '.items[] | select(any(.spec.containers[]; .name == "router")) | .metadata.name')
+kubectl -n gpustack-elastic port-forward "pod/$ROUTER_POD" 18081:8081 19090:9090
+```
+
+In another terminal, read the registered workers and per-worker request counters:
+
+```bash
+curl --fail-with-body http://127.0.0.1:18081/observer/endpoints |
+ jq '.workers[] | {url, selection}'
+curl --fail-with-body http://127.0.0.1:19090/metrics |
+ grep '^vllm_router_processed_requests_total'
+```
+
+Repeat the completion requests through port 18081. Compare the counters before and after.
+For complete External LB coverage, the registry must contain both rank endpoints.
+Requests must increase each rank's counter. A successful response through the Router is insufficient.
+
+The corrected run registered both healthy endpoints and included both in selection.
+Four Router requests passed on their first attempt: two ordinary requests and two streams.
+Each rank's request counter increased by two. The [validation record](#validation-record) compares the original and corrected runs.
+
+For ordinary Internal groups and [P/D deployments](../../modules/model-deployment/prefill-decode.md), the leader restriction remains necessary.
+See [Prefill and Decode](../../modules/model-deployment/prefill-decode.md) for the deferred External DP combination and its tracking issue.
+This Server example does not validate that combination.
+
+## Understand the scaling limits
+
+This fixed External DP group has no online DP/EP resize contract.
+vLLM 0.29.0 rejects combining its native Elastic EP mode with External or Hybrid LB.
+Use the [Ray Internal Elastic walkthrough](elastic-ep.md) for `elasticEp.width` growth.
+A Router in front of that Internal endpoint does not turn the engine into External LB.
+
+Increasing `roles[].replicas` creates additional complete groups with their existing TP/DP/EP shape.
+That operation does not enlarge an existing group's EP world and was not measured in this External run.
+The ordinary role's `size` cannot be changed after creation.
+Hybrid and multiple HTTP ports per Pod are outside this walkthrough.
+
+## Step 5: inspect and release the application
+
+Read both ranks' logs when startup or collective requests stall:
+
+```bash
+kubectl -n gpustack-elastic logs external-fixed-server-r0-m0 -c main
+kubectl -n gpustack-elastic logs external-fixed-server-r0-m1 -c main
+kubectl -n gpustack-elastic get modeldeployment external-fixed -o yaml
+kubectl -n gpustack-elastic get services,endpointslices
+```
+
+Check matching DP sizes, distinct rank values, a shared leader address, network access, and free GPU resources.
+Keep Router discovery evidence separate from engine startup and inference evidence.
+The replica's headless Service supplies peer DNS; narrowing it to the HTTP leader can prevent group startup.
+
+After saving the results, stop the port-forward processes and delete this application:
+
+```bash
+kubectl -n gpustack-elastic delete modeldeployment external-fixed
+kubectl -n gpustack-elastic get pods,services,deployments \
+ -l app.kubernetes.io/instance=external-fixed
+kubectl get devices -o json
+```
+
+Verify that its owned workloads are gone and all four GPU claims are released.
+In the final corrected run, engine Pods disappeared and all eight cards returned to their free state.
+The Router and Services remained after about two minutes of observation.
+Their deletion required explicit cleanup after confirming their owner UID.
+
+Automatic dependent cleanup was not established by that run; its cause remains unconfirmed.
+[Issue #601](https://github.com/gpustack/gpustack-operator/issues/601) tracks related cleanup delays after worker outages; this run did not establish that cause.
+
+Deleting the application retains the model cache and cloud cluster.
+
+## Validation record
+
+The initial operator binary was `d00f8fa8c98334a71af4c6eb302ee80d08289309` from
+`gpustack/gpustack-operator:dev@sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36`.
+The corrected run used binary `20d674b7f8c1e1d4c22b91a09fd178b9738d70be` from
+`thxcode/gpustack-operator:dev-20d674b7@sha256:43c14087fd3391e4bf4a3327f246053d9e59b2feb684cb792f63a2f0db659fbd`.
+That test image replaced the operator binary on the pinned official base. It retained the packaged vendor assets.
+
+The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements.
+Its CuPy installation contained only `cupy-cuda13x` 14.2.0.
+
+The initial operator retained `member-index=0` and registered only member 0.
+All four Router requests completed, but all four increased only member 0's counter.
+After the fix, discovery registered both healthy endpoints and included both in selection.
+Four Router requests completed on their first attempt: two ordinary and two streaming requests.
+Each returned eight completion tokens; both streams ended with `data: [DONE]`.
+
+| Operator run | Registered ranks | Rank 0 counter increase | Rank 1 counter increase | Coverage |
+|---|---|---|---|---|
+| Initial official dev | 0 | 4 | 0 | Failed |
+| Selector fix | 0, 1 | 2 | 2 | Passed |
+
+The correction removed the leader restriction only from eligible External Server endpoints.
+Both engine Pods remained running through the Operator upgrade. The Router Pod restarted with its corrected selector.
+
+---
+
+**See also** — [Elastic EP Walkthrough](elastic-ep.md) (collective expansion) ·
+[Routing](../../modules/model-deployment/routing.md) (backend selection) · [Model Deployment Configuration](../../modules/model-deployment/deployment.md) (role fields)
+
+**Next** → [Model Deployment Status](../../modules/model-deployment/status.md)
diff --git a/pkg/worker/controllers/worker/model_deployment.go b/pkg/worker/controllers/worker/model_deployment.go
index e55ca03e8..eb9cfa285 100644
--- a/pkg/worker/controllers/worker/model_deployment.go
+++ b/pkg/worker/controllers/worker/model_deployment.go
@@ -1418,11 +1418,10 @@ func modelDeploymentRoleExternalDP(
// an accepted error-path semantics change, deliberately kept for a reviewed change.
return false
}
- shape, _ := modelDeploymentLoadBalance(reading.declared.Wiring)
+ shape, _ := modelDeploymentLoadBalance(reading.declared)
- return shape == workercore.ModelDeploymentLoadBalanceExternal ||
- shape == workercore.ModelDeploymentLoadBalanceHybrid ||
- shape == workercore.ModelDeploymentLoadBalanceMultiPort
+ return shape != workercore.ModelDeploymentLoadBalanceInternal &&
+ shape != workercore.ModelDeploymentLoadBalanceUnknown
}
// syncModelDeploymentService converges every Service the deployment owns: the one it is reached
@@ -1469,7 +1468,7 @@ func (r *ModelDeploymentReconciler) syncModelDeploymentService(
continue
}
if modelDeploymentEligibilitySelectorActive(md, role, live.Spec.Selector, eligibilityDecided) {
- modelDeploymentSelectEligibleEndpoints(rendered[i])
+ modelDeploymentSelectEligibleEndpoints(rendered[i], md, role)
}
}
expected := make([]ctrlcli.Object, len(rendered))
diff --git a/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go b/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go
new file mode 100644
index 000000000..867826fb1
--- /dev/null
+++ b/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go
@@ -0,0 +1,50 @@
+package worker
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+
+ workercore "gpustack.ai/gpustack/api/worker/v1alpha1"
+)
+
+func TestModelDeploymentBalanceRouting(t *testing.T) {
+ cases := []struct {
+ name string
+ args []string
+ kind workercore.ModelDeploymentRoleKind
+ balance workercore.ModelDeploymentLoadBalance
+ external bool
+ reason string
+ }{
+ {name: "normalized hybrid internal", args: []string{"--data-parallel-hybrid-lb", "--data-parallel-size=4", "--data-parallel-size-local=4"}, balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "explicit single node internal", args: []string{"--data-parallel-size=4", "--data-parallel-start-rank=0", "--nnodes=1"}, balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "unreadable node count", args: []string{"--data-parallel-start-rank=0", "--nnodes=2:3"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"},
+ {name: "missing node count", args: []string{"--data-parallel-start-rank=0", "--nnodes"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"},
+ {name: "nonpositive node count", args: []string{"--data-parallel-start-rank=0", "--nnodes=0"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"},
+ {name: "multiple nodes need local size", args: []string{"--data-parallel-start-rank=0", "--nnodes=2"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "requires an explicit data-parallel-size-local"},
+ {name: "internal", balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "rank", args: []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}, balance: workercore.ModelDeploymentLoadBalanceExternal, external: true},
+ {name: "start rank alone", args: []string{"--data-parallel-start-rank=1"}, balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "implicit external", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1"}, balance: workercore.ModelDeploymentLoadBalanceExternal, external: true},
+ {name: "implicit hybrid", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=2", "--data-parallel-start-rank=2"}, balance: workercore.ModelDeploymentLoadBalanceHybrid, external: true},
+ {name: "all local internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=4", "--data-parallel-start-rank=0"}, balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "headless internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1", "--headless"}, balance: workercore.ModelDeploymentLoadBalanceInternal},
+ {name: "hybrid", args: []string{"--data-parallel-hybrid-lb"}, balance: workercore.ModelDeploymentLoadBalanceHybrid, external: true},
+ {name: "multi port", args: []string{"--data-parallel-multi-port-external-lb"}, balance: workercore.ModelDeploymentLoadBalanceMultiPort, external: true},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ md := newRenderDeployment()
+ role := &md.Spec.Roles[0]
+ role.ExtraArgs = tc.args
+ role.Kind = tc.kind
+ status := ReadModelDeploymentRoleParallelism(md.Spec.Engine.Name, role)
+ assert.Equal(t, tc.balance, status.LoadBalance)
+ assert.Equal(t, tc.external, modelDeploymentRoleExternalDP(md, role))
+ if tc.reason != "" {
+ assert.Contains(t, status.Source.UnreadableReason, tc.reason)
+ }
+ })
+ }
+}
diff --git a/pkg/worker/controllers/worker/model_deployment_drain_collector.go b/pkg/worker/controllers/worker/model_deployment_drain_collector.go
index 1a1884044..953fddc00 100644
--- a/pkg/worker/controllers/worker/model_deployment_drain_collector.go
+++ b/pkg/worker/controllers/worker/model_deployment_drain_collector.go
@@ -893,7 +893,10 @@ func modelDeploymentDrainDisaggregated(engine string, container *corev1.Containe
return fmt.Sprintf("the member's parallelism could not be read: %v", err), true
}
- shape, _ := modelDeploymentLoadBalance(reading.declared.Wiring)
+ shape, reason := modelDeploymentLoadBalance(reading.declared)
+ if shape == workercore.ModelDeploymentLoadBalanceUnknown {
+ return reason, true
+ }
switch shape {
case workercore.ModelDeploymentLoadBalanceExternal,
workercore.ModelDeploymentLoadBalanceHybrid,
diff --git a/pkg/worker/controllers/worker/model_deployment_external_routing_test.go b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go
new file mode 100644
index 000000000..45ca77879
--- /dev/null
+++ b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go
@@ -0,0 +1,184 @@
+package worker
+
+import (
+ "context"
+ "fmt"
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+ core "k8s.io/api/core/v1"
+ meta "k8s.io/apimachinery/pkg/apis/meta/v1"
+ "k8s.io/apimachinery/pkg/labels"
+ ctrlrecord "k8s.io/client-go/tools/record"
+ ctrlcli "sigs.k8s.io/controller-runtime/pkg/client"
+
+ gpustackcore "gpustack.ai/gpustack/api/v1"
+ workercore "gpustack.ai/gpustack/api/worker/v1alpha1"
+)
+
+func TestExternalDPRoutingSelectsQualifiedMembers(t *testing.T) {
+ for _, routerName := range []string{
+ workercore.ModelDeploymentRouterLLMD,
+ workercore.ModelDeploymentRouterVLLM,
+ workercore.ModelDeploymentRouterSGLang,
+ } {
+ for _, decided := range []bool{false, true} {
+ t.Run(fmt.Sprintf("%s/decided=%t", routerName, decided), func(t *testing.T) {
+ md := newRenderDeployment()
+ md.Spec.KVCache = nil
+ md.Spec.Engine.Version = "0.29.0"
+ md.Spec.Router = &workercore.ModelDeploymentRouter{Name: routerName}
+ md.Spec.Roles[0].ReplicaSize = 2
+ md.Spec.Roles[0].ExtraArgs = []string{"--data-parallel-size", "2", "--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}
+ internal := *md.Spec.Roles[0].DeepCopy()
+ internal.Name = "internal"
+ internal.ExtraArgs = nil
+ normalized := *internal.DeepCopy()
+ normalized.Name = "normalized"
+ normalized.ExtraArgs = []string{"--data-parallel-hybrid-lb", "--data-parallel-size=4", "--data-parallel-size-local=4"}
+ md.Spec.Roles = append(md.Spec.Roles, internal, normalized)
+ if decided {
+ ModelDeploymentConditionEndpointEligibility.True(md, modelDeploymentReasonEndpointsQualified, "all groups qualified")
+ }
+ objects, err := renderModelDeploymentRouterObjects(context.Background(), md, nil, false)
+ require.NoError(t, err)
+ discovery := routingDiscoverySelector(t, objects.Deployment.Spec.Template.Spec.Containers[0].Args, objects.ConfigMap)
+ for i := range md.Spec.Roles {
+ role := &md.Spec.Roles[i]
+ published := labels.SelectorFromSet(objects.Contract.Roles[i].Selector)
+ service := labels.SelectorFromSet(renderModelDeploymentRoleService(md, role, nil).Spec.Selector)
+ for member := range 2 {
+ pod, renderErr := renderModelDeploymentPodTemplate(context.Background(), ModelDeploymentRenderInput{
+ Deployment: md, Role: role, InstanceType: newRenderInstanceType(),
+ })
+ require.NoError(t, renderErr)
+ stampModelDeploymentPod(pod, md, role, 0, member)
+ answers := i == 0 || member == 0
+ if answers {
+ pod.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue
+ }
+ want := member == 0 || (decided && i == 0)
+ for name, selector := range map[string]labels.Selector{"discovery": discovery, "published": published, "service": service} {
+ assert.Equal(t, want, selector.Matches(labels.Set(pod.Labels)), "%s/%s/member-%d", name, role.Name, member)
+ if decided {
+ withdrawn := pod.DeepCopy()
+ delete(withdrawn.Labels, modelDeploymentLabelKeyEndpointEligible)
+ assert.False(t, selector.Matches(labels.Set(withdrawn.Labels)), "%s must exclude withdrawn endpoints", name)
+ }
+ }
+ }
+ headless := renderModelDeploymentReplicaService(md, role, 0)
+ assert.Equal(t, core.ClusterIPNone, headless.Spec.ClusterIP)
+ assert.True(t, headless.Spec.PublishNotReadyAddresses)
+ assert.NotContains(t, headless.Spec.Selector, modelDeploymentLabelKeyEndpointEligible)
+ }
+ assert.False(t, discovery.Matches(labels.Set(objects.Deployment.Spec.Template.Labels)), "the router is not an engine endpoint")
+ })
+ }
+ }
+}
+
+func routingDiscoverySelector(t *testing.T, args []string, config *core.ConfigMap, flags ...string) labels.Selector {
+ t.Helper()
+ if config != nil {
+ selector, err := labels.Parse(config.Data[modelDeploymentRouterSelectorKey])
+ require.NoError(t, err)
+ return selector
+ }
+ flag := "--selector"
+ if len(flags) > 0 {
+ flag = flags[0]
+ }
+ terms := map[string]string{}
+ for i, arg := range args {
+ if arg == flag {
+ for _, term := range args[i+1:] {
+ if len(term) > 1 && term[:2] == "--" {
+ break
+ }
+ parsed, err := labels.ConvertSelectorToLabelsMap(term)
+ require.NoError(t, err)
+ for key, value := range parsed {
+ terms[key] = value
+ }
+ }
+ break
+ }
+ }
+ require.NotEmpty(t, terms)
+ return labels.SelectorFromSet(terms)
+}
+
+func TestPDRoutingKeepsReplicaLeaders(t *testing.T) {
+ for _, routerName := range []string{workercore.ModelDeploymentRouterLLMD, workercore.ModelDeploymentRouterVLLM, workercore.ModelDeploymentRouterSGLang} {
+ for _, sizes := range [][2]int{{1, 1}, {2, 1}, {1, 2}, {2, 2}} {
+ t.Run(fmt.Sprintf("%s/%d-%d", routerName, sizes[0], sizes[1]), func(t *testing.T) {
+ md := twoRoleDeployment()
+ md.Spec.Roles[0].Kind = workercore.ModelDeploymentRoleKindPrefill
+ md.Spec.Roles[1].Kind = workercore.ModelDeploymentRoleKindDecode
+ md.Spec.Router = &workercore.ModelDeploymentRouter{Name: routerName}
+ if routerName == workercore.ModelDeploymentRouterSGLang {
+ md.Spec.Engine.Name = workercore.ModelDeploymentEngineSGLang
+ }
+ if routerName != workercore.ModelDeploymentRouterLLMD {
+ md.Spec.Router.ExtraArgs = []string{"--policy", "round_robin"}
+ }
+ for i := range md.Spec.Roles {
+ md.Spec.Roles[i].ReplicaSize = int32(sizes[i])
+ md.Spec.Roles[i].Replicas = 2
+ }
+ ModelDeploymentConditionEndpointEligibility.True(md, modelDeploymentReasonEndpointsQualified, "qualified")
+ objects, err := renderModelDeploymentRouterObjects(context.Background(), md, nil, true)
+ require.NoError(t, err)
+
+ for i := range md.Spec.Roles {
+ role := &md.Spec.Roles[i]
+ discovery := routingDiscoverySelector(t, objects.Deployment.Spec.Template.Spec.Containers[0].Args, objects.ConfigMap, "--"+role.Name+"-selector")
+ service := renderModelDeploymentRoleService(md, role, nil)
+ selectors := []labels.Selector{labels.SelectorFromSet(service.Spec.Selector), labels.SelectorFromSet(objects.Contract.Roles[i].Selector)}
+ if i == 0 {
+ selectors = append(selectors, labels.SelectorFromSet(renderModelDeploymentService(md).Spec.Selector))
+ }
+ for ordinal := range 2 {
+ for member := range sizes[i] {
+ pod, err := renderModelDeploymentPodTemplate(context.Background(), ModelDeploymentRenderInput{Deployment: md, Role: role, InstanceType: newRenderInstanceType()})
+ require.NoError(t, err)
+ stampModelDeploymentPod(pod, md, role, ordinal, member)
+ assert.Equal(t, member == 0, discovery.Matches(labels.Set(pod.Labels)))
+ for _, selector := range selectors {
+ assert.Equal(t, member == 0, selector.Matches(labels.Set(pod.Labels)))
+ }
+ other := labels.SelectorFromSet(objects.Contract.Roles[1-i].Selector)
+ assert.False(t, other.Matches(labels.Set(pod.Labels)))
+ peer := renderModelDeploymentReplicaService(md, role, ordinal)
+ assert.True(t, peer.Spec.PublishNotReadyAddresses)
+ assert.True(t, labels.SelectorFromSet(peer.Spec.Selector).Matches(labels.Set(pod.Labels)))
+ }
+ }
+ }
+ })
+ }
+ }
+}
+
+func TestExternalDPServiceEligibilityRetention(t *testing.T) {
+ md := newRenderDeployment()
+ md.Spec.Engine.Version = "0.29.0"
+ md.Spec.KVCache = nil
+ role := &md.Spec.Roles[0]
+ role.ReplicaSize = 2
+ role.ExtraArgs = []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}
+ cli := newModelDeploymentClient(md, newRenderInstanceType())
+ reconciler := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(16)}
+ for _, status := range []meta.ConditionStatus{meta.ConditionTrue, meta.ConditionFalse, meta.ConditionUnknown} {
+ md.Status.Conditions = []gpustackcore.Condition{{Type: string(ModelDeploymentConditionEndpointEligibility), Status: status}}
+ require.NoError(t, reconciler.syncModelDeploymentService(context.Background(), md, false))
+ for _, name := range []string{md.Name, md.Name + "-" + role.Name} {
+ service := new(core.Service)
+ require.NoError(t, cli.Get(context.Background(), ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, service))
+ assert.NotContains(t, service.Spec.Selector, modelDeploymentMemberIndexLabel)
+ assert.Equal(t, modelDeploymentEndpointEligibleValue, service.Spec.Selector[modelDeploymentLabelKeyEndpointEligible])
+ }
+ }
+}
diff --git a/pkg/worker/controllers/worker/model_deployment_parallelism.go b/pkg/worker/controllers/worker/model_deployment_parallelism.go
index 401c35ccd..36b7bc442 100644
--- a/pkg/worker/controllers/worker/model_deployment_parallelism.go
+++ b/pkg/worker/controllers/worker/model_deployment_parallelism.go
@@ -18,6 +18,8 @@ import (
// engine's own default of one -- the operator composes no degree of its own, because the
// transfer document must agree with the engine, and only the author's numbers can be right.
type ModelDeploymentDeclaredParallelism struct {
+ // NodeCount retains an explicit --nnodes value; zero means absent or unreadable.
+ NodeCount int
TensorParallel int
PipelineParallel int
DataParallel int
@@ -112,6 +114,7 @@ var modelDeploymentParallelFlags = map[string][]modelDeploymentParallelFlag{
{names: []string{"--data-parallel-address", "-dpa"}, wiring: true, takesValue: true},
{names: []string{"--data-parallel-rpc-port", "-dpp"}, wiring: true, takesValue: true},
{names: []string{"--data-parallel-backend", "-dpb"}, wiring: true, takesValue: true},
+ {names: []string{"--headless"}, wiring: true},
{names: []string{"--data-parallel-hybrid-lb", "-dph"}, wiring: true},
{names: []string{"--data-parallel-external-lb", "-dpe"}, wiring: true},
{names: []string{"--data-parallel-multi-port-external-lb", "-dpm"}, wiring: true},
@@ -242,6 +245,10 @@ func scanModelDeploymentParallelism(
if flag.takesValue && !hasValue && i+1 < len(extraArgs) &&
modelDeploymentConsumableValue(extraArgs[i+1]) {
i++
+ value = extraArgs[i]
+ }
+ if canonical == "--nnodes" {
+ declared.NodeCount, _ = modelDeploymentEngineInt(value)
}
case flag.degree:
if !hasValue {
@@ -439,7 +446,7 @@ func ReadModelDeploymentRoleParallelism(
return modelDeploymentParallelismUnreadable(overflow)
}
- loadBalance, balanceReason := modelDeploymentLoadBalance(reading.declared.Wiring)
+ loadBalance, balanceReason := modelDeploymentLoadBalance(reading.declared)
return workercore.ModelDeploymentRoleParallelismStatus{
Declared: declared,
@@ -472,7 +479,6 @@ func modelDeploymentParallelismUnreadable(reason string) workercore.ModelDeploym
var modelDeploymentBalanceShapes = map[string]workercore.ModelDeploymentLoadBalance{
"--data-parallel-external-lb": workercore.ModelDeploymentLoadBalanceExternal,
"--data-parallel-rank": workercore.ModelDeploymentLoadBalanceExternal,
- "--data-parallel-start-rank": workercore.ModelDeploymentLoadBalanceExternal,
"--data-parallel-hybrid-lb": workercore.ModelDeploymentLoadBalanceHybrid,
"--data-parallel-multi-port-external-lb": workercore.ModelDeploymentLoadBalanceMultiPort,
}
@@ -481,19 +487,52 @@ var modelDeploymentBalanceShapes = map[string]workercore.ModelDeploymentLoadBala
// voting one shape agree (an assigned rank alongside an external balancer is still External);
// flags voting different shapes are not derivable and name their disagreement.
func modelDeploymentLoadBalance(
- wiring []string,
+ declared ModelDeploymentDeclaredParallelism,
) (workercore.ModelDeploymentLoadBalance, string) {
+ wiring := declared.Wiring
shapes := map[workercore.ModelDeploymentLoadBalance][]string{}
for _, flag := range wiring {
if shape, ok := modelDeploymentBalanceShapes[flag]; ok {
shapes[shape] = append(shapes[shape], flag)
}
}
+ startRank := slices.Contains(wiring, "--data-parallel-start-rank")
+ if startRank && len(shapes[workercore.ModelDeploymentLoadBalanceExternal]) > 0 {
+ shapes[workercore.ModelDeploymentLoadBalanceHybrid] = append(
+ shapes[workercore.ModelDeploymentLoadBalanceHybrid], "--data-parallel-start-rank")
+ }
if len(shapes) == 0 {
+ // Headless ranks belong to an Internal group. A start rank only selects
+ // external routing when the local group is smaller than the global group.
+ if startRank && !slices.Contains(wiring, "--headless") {
+ local := declared.DataParallelLocal
+ if local > 0 && local < declared.DataParallel {
+ if local == 1 {
+ return workercore.ModelDeploymentLoadBalanceExternal, ""
+ }
+ return workercore.ModelDeploymentLoadBalanceHybrid, ""
+ }
+ if local == 0 && slices.Contains(wiring, "--nnodes") && declared.NodeCount != 1 {
+ if declared.NodeCount <= 0 {
+ return workercore.ModelDeploymentLoadBalanceUnknown,
+ "nnodes must be a readable positive integer to derive start-rank routing"
+ }
+ return workercore.ModelDeploymentLoadBalanceUnknown,
+ "start-rank routing requires an explicit data-parallel-size-local when nnodes exceeds 1"
+ }
+ }
return workercore.ModelDeploymentLoadBalanceInternal, ""
}
if len(shapes) == 1 {
for shape := range shapes {
+ if shape == workercore.ModelDeploymentLoadBalanceHybrid {
+ if declared.DataParallelLocal == 1 {
+ return workercore.ModelDeploymentLoadBalanceExternal, ""
+ }
+ if declared.DataParallelLocal == declared.DataParallel {
+ return workercore.ModelDeploymentLoadBalanceInternal, ""
+ }
+ }
return shape, ""
}
}
diff --git a/pkg/worker/controllers/worker/model_deployment_parallelism_test.go b/pkg/worker/controllers/worker/model_deployment_parallelism_test.go
index 06d73a24c..9dc8a9576 100644
--- a/pkg/worker/controllers/worker/model_deployment_parallelism_test.go
+++ b/pkg/worker/controllers/worker/model_deployment_parallelism_test.go
@@ -157,7 +157,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) {
name: "vllm: single-letter alias concatenated value records",
engine: vllm,
extraArgs: []string{"-n2"},
- set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"} },
+ set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"}; p.NodeCount = 2 },
},
{
name: "vllm: node-rank concatenated value records",
@@ -169,7 +169,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) {
name: "vllm: single-letter alias with = value records",
engine: vllm,
extraArgs: []string{"-n=2"},
- set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"} },
+ set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"}; p.NodeCount = 2 },
},
{
name: "vllm: underscore unique prefix resolves",
@@ -268,6 +268,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) {
extraArgs: []string{"--nnodes", "2", "--tensor-parallel-size", "2"},
set: func(p *ModelDeploymentDeclaredParallelism) {
p.TensorParallel = 2
+ p.NodeCount = 2
p.Wiring = []string{"--nnodes"}
},
},
@@ -275,7 +276,10 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) {
name: "vllm: wiring short aliases record",
engine: vllm,
extraArgs: []string{"-n", "2", "-r", "0"},
- set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes", "--node-rank"} },
+ set: func(p *ModelDeploymentDeclaredParallelism) {
+ p.Wiring = []string{"--nnodes", "--node-rank"}
+ p.NodeCount = 2
+ },
},
{
name: "vllm: boolean wiring consumes nothing",
@@ -643,6 +647,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) {
extraArgs: []string{"--nnodes", "2", "--dist-init-addr", "a:1", "--nccl-init-addr", "b:2"},
set: func(p *ModelDeploymentDeclaredParallelism) {
p.Wiring = []string{"--nnodes", "--dist-init-addr"}
+ p.NodeCount = 2
},
},
{
@@ -999,11 +1004,11 @@ func TestReadModelDeploymentRoleParallelism(t *testing.T) {
},
},
{
- name: "an assigned start rank alone derives External",
+ name: "an assigned start rank alone does not select external routing",
engine: vllm,
role: workercore.ModelDeploymentRole{ExtraArgs: []string{"--data-parallel-start-rank=1"}},
want: workercore.ModelDeploymentRoleParallelismStatus{
- LoadBalance: workercore.ModelDeploymentLoadBalanceExternal,
+ LoadBalance: workercore.ModelDeploymentLoadBalanceInternal,
Source: workercore.ModelDeploymentParallelismSourceStatus{
Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs,
Complete: true,
@@ -1011,16 +1016,17 @@ func TestReadModelDeploymentRoleParallelism(t *testing.T) {
},
},
{
- name: "start rank and external balancer agree on External",
+ name: "start rank and external balancer conflict",
engine: vllm,
role: workercore.ModelDeploymentRole{
ExtraArgs: []string{"--data-parallel-start-rank=1", "--data-parallel-external-lb"},
},
want: workercore.ModelDeploymentRoleParallelismStatus{
- LoadBalance: workercore.ModelDeploymentLoadBalanceExternal,
+ LoadBalance: workercore.ModelDeploymentLoadBalanceUnknown,
Source: workercore.ModelDeploymentParallelismSourceStatus{
- Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs,
- Complete: true,
+ Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs,
+ Complete: true,
+ UnreadableReason: "the balance flags --data-parallel-external-lb and --data-parallel-start-rank derive to different shapes",
},
},
},
diff --git a/pkg/worker/controllers/worker/model_deployment_router.go b/pkg/worker/controllers/worker/model_deployment_router.go
index 0e3c4214e..e21f51b27 100644
--- a/pkg/worker/controllers/worker/model_deployment_router.go
+++ b/pkg/worker/controllers/worker/model_deployment_router.go
@@ -124,22 +124,17 @@ func renderModelDeploymentRouterObjects(
for i := range md.Spec.Roles {
role := &md.Spec.Roles[i]
kind := ModelDeploymentEffectiveRoleKind(role)
- // THE PUBLISHED SELECTOR NAMES THE PODS A REQUEST MAY REACH, which is one member per
- // replica -- the leader -- rather than every Pod the role runs. The field is published
- // VERBATIM for a router to be configured from, so a selector that also named the members
- // serving no API would hand that router the very defect the discovery selector below
- // exists to remove. The leader term is what keeps the two selectors in one semantics; they
- // differ in scope only, this one naming a role and that one the deployment.
+ // Keep legacy leader-only routing until the eligibility gate is active.
+ // Once qualified, External-DP members all serve HTTP and must remain selectable.
selector := modelDeploymentSelectorLabels(md, role)
selector[modelDeploymentLabelKeyRoleKind] = roles[i].Kind
selector[modelDeploymentMemberIndexLabel] = strconvx.Itoa(modelDeploymentLeaderMemberIndex)
- // The eligibility term rides on the published selector beside the leader term, so a router
- // configured from it can only ever reach endpoints the reconciler has marked eligible. It
- // narrows the same set the terms above already name; it never widens one. A shape whose
- // group-forward capability can never activate renders WITHOUT the term: a selector no pass
- // would ever satisfy would hand the router zero endpoints and call it discovery.
+ // Unsupported shapes never gain an eligibility selector that no pass can satisfy.
if modelDeploymentEligibilitySelectorActive(md, role, nil, retainedEligibility) {
selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue
+ if modelDeploymentRoleExternalDP(md, role) {
+ delete(selector, modelDeploymentMemberIndexLabel)
+ }
}
// The renderer is handed the SAME map that is published, rather than one built beside it,
// because a router configured by argv discovers each role by these labels: two derivations
@@ -156,19 +151,9 @@ func renderModelDeploymentRouterObjects(
}
contract.Roles = append(contract.Roles, roleStatus)
}
- // DISCOVERY NAMES ONLY THE PODS THAT ANSWER THE API: one member per replica, the leader, which
- // at size one is the replica's only Pod. Without the index term the selector would name every
- // member, a router would spread requests across the ones serving nothing, and the failures
- // would interleave with successes rather than the router being visibly broken. The term is a
- // plain equality rather than something sized per role, because a label selector is a
- // conjunction and the pickers filter by kind: two server roles of different sizes share this
- // one selector, and only a member index written on every member lets a single term hold for
- // both.
- //
- // THE EQUALITIES ARE THE SOURCE AND THE EXPRESSION IS DERIVED FROM THEM, because the two
- // routers configured by argv match on equality alone and cannot be handed an expression: one of
- // them splits each entry on its first "=" and drops an entry carrying none without a word. A
- // second literal here would let the string and the map name different sets.
+ // Start with legacy leader-only discovery. Qualified External-DP roles use the
+ // eligibility label instead, which also excludes internal followers in mixed roles.
+ // Derive every selector form from these equalities so all routers discover the same set.
endpointLabels := map[string]string{
modelDeploymentLabelKeyName: modelDeploymentLabelValueName,
modelDeploymentLabelKeyInstance: md.Name,
@@ -191,10 +176,17 @@ func renderModelDeploymentRouterObjects(
}
if endpointActivatable && (modelDeploymentEligibilityDecided(md) || retainedEligibility) {
endpointLabels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue
+ for i := range md.Spec.Roles {
+ if modelDeploymentRoleExternalDP(md, &md.Spec.Roles[i]) {
+ // Eligibility selects every answering External-DP member and only the
+ // leaders of internal roles, including deployments with both shapes.
+ delete(endpointLabels, modelDeploymentMemberIndexLabel)
+ break
+ }
+ }
}
- // The router's own Pod carries the first two labels and NOT the member index, so the equalities
- // already exclude it. The negation is kept in the expression, where it costs nothing and says
- // out loud what the index term achieves by arithmetic.
+ // The router Pod has neither a member index nor endpoint eligibility.
+ // Keep the explicit exclusion in the expression as well.
//
// The terms are listed rather than sorted out of the map because their ORDER IS RENDERED: this
// string is a ConfigMap value and the Pod's configuration hash is taken over it, so reordering
@@ -202,7 +194,9 @@ func renderModelDeploymentRouterObjects(
endpointSelectorTerms := []string{
modelDeploymentLabelKeyName + "=" + endpointLabels[modelDeploymentLabelKeyName],
modelDeploymentLabelKeyInstance + "=" + endpointLabels[modelDeploymentLabelKeyInstance],
- modelDeploymentMemberIndexLabel + "=" + endpointLabels[modelDeploymentMemberIndexLabel],
+ }
+ if member, present := endpointLabels[modelDeploymentMemberIndexLabel]; present {
+ endpointSelectorTerms = append(endpointSelectorTerms, modelDeploymentMemberIndexLabel+"="+member)
}
if len(endpointLabels[modelDeploymentLabelKeyEndpointEligible]) > 0 {
endpointSelectorTerms = append(endpointSelectorTerms,
diff --git a/pkg/worker/controllers/worker/model_deployment_service.go b/pkg/worker/controllers/worker/model_deployment_service.go
index f419f58f3..52a2c339b 100644
--- a/pkg/worker/controllers/worker/model_deployment_service.go
+++ b/pkg/worker/controllers/worker/model_deployment_service.go
@@ -102,7 +102,7 @@ func renderModelDeploymentService(md *workercore.ModelDeployment) *core.Service
svc := renderModelDeploymentServiceFor(md, &md.Spec.Roles[0], md.Name)
modelDeploymentFrontLeadersOnly(svc, &md.Spec.Roles[0])
if modelDeploymentEligibilitySelectorActive(md, &md.Spec.Roles[0], nil, modelDeploymentEligibilityDecided(md)) {
- modelDeploymentSelectEligibleEndpoints(svc)
+ modelDeploymentSelectEligibleEndpoints(svc, md, &md.Spec.Roles[0])
}
return svc
@@ -153,20 +153,6 @@ func modelDeploymentGroupForwardActivatable(
kind != workercore.ModelDeploymentRoleKindDecode
}
-// modelDeploymentSelectEligibleEndpoints narrows an ordinary Service to the endpoints the
-// reconciler has marked eligible, by the same equality every selector that names endpoints
-// carries. The term rides ON TOP of the identity selector and the leader narrowing -- it gates
-// which of the members a Service already fronts are answerable, never which members exist.
-//
-// THE TERM IS A LABEL EQUALITY BECAUSE ONLY A LABEL CAN SATISFY A SERVICE SELECTOR. The key is
-// written by the reconciler's runtime convergence, outside every rendered hash, so gaining or
-// losing it moves no replica; what it moves is the Service's endpoint set, which is exactly the
-// lever a disqualification pulls.
-//
-// A REPLICA'S HEADLESS SERVICE NEVER TAKES THE TERM. That Service exists to publish every member
-// to its peers with unready addresses included; eligibility is about who may ANSWER, and the
-// peers' collective needs every member's record regardless of who answers.
-
// modelDeploymentEligibilityDecided reports whether a pass has actually answered the whole-group
// predicate: the EndpointEligibility condition is True (qualified -- narrow and let the labels
// admit) or False (revoked -- narrow and let the labels withdraw). A decided condition is the
@@ -216,8 +202,18 @@ func modelDeploymentEligibilitySelectorActive(
liveSelector[modelDeploymentLabelKeyEndpointEligible] == modelDeploymentEndpointEligibleValue
}
-func modelDeploymentSelectEligibleEndpoints(svc *core.Service) {
+// modelDeploymentSelectEligibleEndpoints selects qualified HTTP endpoints.
+// External roles serve every rank; Internal roles retain their leader restriction.
+// Headless peer Services never use endpoint eligibility.
+func modelDeploymentSelectEligibleEndpoints(
+ svc *core.Service, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole,
+) {
svc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue
+ // Every qualified External-DP member serves HTTP. The eligibility label already
+ // excludes unhealthy groups, so the legacy leader restriction must not hide ranks.
+ if modelDeploymentRoleExternalDP(md, role) {
+ delete(svc.Spec.Selector, modelDeploymentMemberIndexLabel)
+ }
}
// renderModelDeploymentRoleService renders the Service that fronts ONE role.
@@ -248,7 +244,7 @@ func renderModelDeploymentRoleService(
svc := renderModelDeploymentServiceFor(md, role, md.Name+"-"+role.Name)
modelDeploymentFrontLeadersOnly(svc, role)
if modelDeploymentEligibilitySelectorActive(md, role, nil, modelDeploymentEligibilityDecided(md)) {
- modelDeploymentSelectEligibleEndpoints(svc)
+ modelDeploymentSelectEligibleEndpoints(svc, md, role)
}
if modelDeploymentPublishesKVEvents(md, role, manufacturers[role.Name]) {
for _, port := range inject.KVEventsPorts() {
diff --git a/pkg/worker/webhooks/worker/model_deployment.go b/pkg/worker/webhooks/worker/model_deployment.go
index 75cf29c08..079398e52 100644
--- a/pkg/worker/webhooks/worker/model_deployment.go
+++ b/pkg/worker/webhooks/worker/model_deployment.go
@@ -1645,6 +1645,7 @@ func validateModelDeploymentRoles(md *workercore.ModelDeployment) field.ErrorLis
errs = append(errs, validateModelDeploymentRoleExtraArgs(md.Spec.Engine.Name, role, rolePath)...)
errs = append(errs, validateModelDeploymentRoleParallelWidth(md.Spec.Engine.Name, role, rolePath)...)
+ errs = append(errs, validateModelDeploymentRoleLoadBalance(md.Spec.Engine.Name, role, rolePath)...)
errs = append(errs, validateModelDeploymentRoleEnv(md.Spec.Engine.Name, role, rolePath)...)
errs = append(errs, validateModelDeploymentRoleResources(role, rolePath)...)
errs = append(errs, validateModelDeploymentRoleAdditionalVolumes(role, rolePath)...)
@@ -1943,6 +1944,28 @@ func validateModelDeploymentRoleExtraArgs(
return errs
}
+// validateModelDeploymentRoleLoadBalance keeps managed roles on one HTTP endpoint per Pod.
+func validateModelDeploymentRoleLoadBalance(
+ engine string, role *workercore.ModelDeploymentRole, rolePath *field.Path,
+) field.ErrorList {
+ if len(role.Command) > 0 {
+ return nil
+ }
+ status := workerctrl.ReadModelDeploymentRoleParallelism(engine, role)
+ if status.Source.Kind != workercore.ModelDeploymentParallelismSourceKindExtraArgs {
+ return nil // The degree parser reports unreadable arguments separately.
+ }
+ if status.LoadBalance == workercore.ModelDeploymentLoadBalanceUnknown {
+ return field.ErrorList{field.Invalid(rolePath.Child("extraArgs"), role.ExtraArgs,
+ status.Source.UnreadableReason)}
+ }
+ if status.LoadBalance == workercore.ModelDeploymentLoadBalanceMultiPort {
+ return field.ErrorList{field.Invalid(rolePath.Child("extraArgs"), role.ExtraArgs,
+ "MultiPort load balancing is unsupported; managed roles require one HTTP endpoint per Pod")}
+ }
+ return nil
+}
+
// validateModelDeploymentRoleParallelWidth is the first check that ever ties a role's declared
// parallelism to its request: the per-member engine width must fit the per-member card count.
//
diff --git a/pkg/worker/webhooks/worker/model_deployment_balance_test.go b/pkg/worker/webhooks/worker/model_deployment_balance_test.go
new file mode 100644
index 000000000..79b9524ce
--- /dev/null
+++ b/pkg/worker/webhooks/worker/model_deployment_balance_test.go
@@ -0,0 +1,41 @@
+package worker
+
+import (
+ "testing"
+
+ "github.com/stretchr/testify/assert"
+
+ workercore "gpustack.ai/gpustack/api/worker/v1alpha1"
+)
+
+func TestModelDeploymentManagedBalance(t *testing.T) {
+ cases := []struct {
+ name string
+ args, command []string
+ kind workercore.ModelDeploymentRoleKind
+ want string
+ }{
+ {name: "internal"},
+ {name: "single node internal", args: []string{"--data-parallel-size=4", "--data-parallel-start-rank=0", "--nnodes=1"}},
+ {name: "external rank", args: []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}},
+ {name: "multi port", args: []string{"-dpm"}, want: "MultiPort"},
+ {name: "conflicting modes", args: []string{"-dpe", "-dph"}, want: "different shapes"},
+ {name: "headless internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1", "--headless"}},
+ {name: "custom command", command: []string{"custom-engine"}, args: []string{"-dpm"}},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ md := modelDeployment(workercore.ModelDeploymentEngineVLLM)
+ role := &md.Spec.Roles[0]
+ role.ExtraArgs = tc.args
+ role.Command = tc.command
+ role.Kind = tc.kind
+ errs := validateModelDeploymentRoles(md)
+ if tc.want == "" {
+ assert.Empty(t, errs)
+ } else if assert.NotEmpty(t, errs) {
+ assert.Contains(t, errs.ToAggregate().Error(), tc.want)
+ }
+ })
+ }
+}
diff --git a/site/content/docs/_index.md b/site/content/docs/_index.md
index d6fc2db8d..80d976a94 100644
--- a/site/content/docs/_index.md
+++ b/site/content/docs/_index.md
@@ -10,7 +10,7 @@ Install the operator, configure a capability, or find the reference for a runnin
- [Architecture](getting-started/architecture/) explains the components and scheduling chain.
- [Vendor Prerequisites](getting-started/vendor-prerequisites/) lists the drivers and runtime setup.
-- [Walkthrough](getting-started/walkthrough/) follows a request through the operator.
+- [Walkthroughs](walkthroughs/) groups complete examples by module.
## Capabilities
diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml
index ca2a686eb..6e1e4ef2d 100644
--- a/site/data/navigation.yaml
+++ b/site/data/navigation.yaml
@@ -1,6 +1,8 @@
groups:
- name: Getting started
path: /docs/getting-started/
+ - name: Walkthroughs
+ path: /docs/walkthroughs/
- name: Devices Manager
path: /docs/modules/devices/
- name: RDMA Manager
@@ -23,7 +25,20 @@ order:
/docs/getting-started/:
- architecture
- vendor-prerequisites
- - walkthrough
+ /docs/walkthroughs/:
+ - devices
+ - kv-cache
+ - model-delivery
+ - model-deployment
+ /docs/walkthroughs/devices/:
+ - scheduling
+ /docs/walkthroughs/kv-cache/:
+ - shared-cache
+ /docs/walkthroughs/model-delivery/:
+ - prefetch
+ /docs/walkthroughs/model-deployment/:
+ - elastic-ep
+ - external-dp
/docs/modules/devices/:
- preflight
- discovery
@@ -40,7 +55,6 @@ order:
- scheduling
- operations
/docs/modules/kv-cache/:
- - walkthrough
- backend
- pool
- injection
@@ -82,9 +96,18 @@ order:
- development
- internals
labels:
+ /docs/walkthroughs/: Walkthroughs
+ /docs/walkthroughs/devices/: Devices
+ /docs/walkthroughs/devices/scheduling/: Scheduling
+ /docs/walkthroughs/kv-cache/: KV Cache
+ /docs/walkthroughs/kv-cache/shared-cache/: Shared cache
+ /docs/walkthroughs/model-delivery/: Model Delivery
+ /docs/walkthroughs/model-delivery/prefetch/: Model prefetch
+ /docs/walkthroughs/model-deployment/: Model Deployment
+ /docs/walkthroughs/model-deployment/elastic-ep/: Elastic EP
+ /docs/walkthroughs/model-deployment/external-dp/: External DP
/docs/getting-started/architecture/: Architecture
/docs/getting-started/vendor-prerequisites/: Prerequisites
- /docs/getting-started/walkthrough/: Walkthrough
/docs/modules/devices/preflight/: Preflight
/docs/modules/devices/discovery/: Discovery
/docs/modules/devices/requests/: Resource requests
@@ -97,7 +120,6 @@ labels:
/docs/modules/rdma/operations/: Operations
/docs/modules/topology/scheduling/: Scheduling
/docs/modules/topology/operations/: Operations
- /docs/modules/kv-cache/walkthrough/: Walkthrough
/docs/modules/kv-cache/backend/: Backends
/docs/modules/kv-cache/pool/: Pools and bindings
/docs/modules/kv-cache/injection/: Pod injection
diff --git a/site/layouts/index.html b/site/layouts/index.html
index 9253cd98a..6a9633ba7 100644
--- a/site/layouts/index.html
+++ b/site/layouts/index.html
@@ -5,7 +5,7 @@
Choose how the operator and its dependencies are deployed.
Capabilities
diff --git a/specs/2026-10-01-s1-router-qualification-and-drain.md b/specs/2026-10-01-s1-router-qualification-and-drain.md
index 5eb25c353..a60fb9998 100644
--- a/specs/2026-10-01-s1-router-qualification-and-drain.md
+++ b/specs/2026-10-01-s1-router-qualification-and-drain.md
@@ -117,6 +117,13 @@ map; internal headless replica Services keep selecting every member with
`publishNotReadyAddresses` untouched. The existing ready-instance status fields are preserved
unchanged.
+**Corrected after shipping.** For qualified External Server roles, ordinary Services and Router discovery remove the legacy
+`member-index=0` equality and retain endpoint eligibility. Internal and P/D roles keep the leader
+restriction. Known non-Internal modes use the External selection rule; unknown modes do not expand
+selection. Hybrid runtime validation and External DP inside P/D remain separate follow-up work.
+Managed roles retain one HTTP port per Pod. MultiPort mode is refused because discovery cannot
+represent its additional listeners.
+
- **Backfill before narrowing.** Before any Service or Router selector gains the key, the
reconciler backfills it onto every existing healthy qualifying endpoint — no window in which a
new selector term matches nothing and healthy endpoints disappear from selection. A static