From 2783b1dd1f6e6fb098b98aa1651d8b5519c0a15c Mon Sep 17 00:00:00 2001 From: thxCode Date: Thu, 8 Oct 2026 17:30:22 +0800 Subject: [PATCH 1/4] docs(model-deployment): add measured scaling walkthrough Signed-off-by: thxCode --- .../skills/gpustack-operator-docs/SKILL.md | 1 + .../references/page-map.md | 9 + docs/README.md | 3 +- docs/modules/model-deployment/_index.md | 5 +- docs/modules/model-deployment/elastic-ep.md | 10 +- .../model-deployment/scaling-walkthrough.md | 579 ++++++++++++++++++ site/data/navigation.yaml | 2 + 7 files changed, 602 insertions(+), 7 deletions(-) create mode 100644 docs/modules/model-deployment/scaling-walkthrough.md diff --git a/.agents/skills/gpustack-operator-docs/SKILL.md b/.agents/skills/gpustack-operator-docs/SKILL.md index 66760a51f..b70895ea3 100644 --- a/.agents/skills/gpustack-operator-docs/SKILL.md +++ b/.agents/skills/gpustack-operator-docs/SKILL.md @@ -70,6 +70,7 @@ one, not to widen the overview. | A `NodeModelStore` or the `model-manager` plugin: a field and its writer, the status guard, mount authorization, materialization, a failure reason, collection, a metric | `docs/modules/model-delivery/node-store.md` | | Running node delivery: the chart values, where the node's configuration comes from, reading a node, the watermark cap, switching delivery, where replicas land and turning the preference off, upgrading, removing the cache | `docs/modules/model-delivery/operations.md` | | The `ModelDeployment` contract: the inherited reuse domain, the three override tiers, the owned-key table, the runner-image formula, prefill/decode pairing, the topology-placement field contract | `docs/modules/model-deployment/deployment.md` | +| A recorded model-serving sequence from prefetch through elastic expansion, its captured CR output, or the distinction from independent replica scaling | `docs/modules/model-deployment/scaling-walkthrough.md` | | A `ModelDeployment` metrics snapshot, which series each field reads per engine, role and router, cache-hit scope or Pod scrape annotation | `docs/modules/model-deployment/metrics.md` | | What a `ModelDeployment` status condition or published field means, and how to read them when a deployment misbehaves | `docs/modules/model-deployment/status.md` | | How a prefill role and a decode role are paired: the connector each engine and router renders, `spec.router` and its fields, `spec.kvTransfer`, roles on different hardware, a role's own Service | `docs/modules/model-deployment/prefill-decode.md` | diff --git a/.agents/skills/gpustack-operator-docs/references/page-map.md b/.agents/skills/gpustack-operator-docs/references/page-map.md index f0f2e7176..616d92b3c 100644 --- a/.agents/skills/gpustack-operator-docs/references/page-map.md +++ b/.agents/skills/gpustack-operator-docs/references/page-map.md @@ -428,3 +428,12 @@ supersession is recorded that way because it has no new spec to be recorded in. `**Corrected after shipping.**`. A spec naming another spec by file name is the case that forces one: when that file leaves the tree the name resolves to nothing, and leaving it alone preserves a pointer rather than a record. + +## `docs/modules/model-deployment/scaling-walkthrough.md` + +**Owns** — the user sequence from model prefetch to elastic TP2/DP2→4 expansion, captured CR output, +short inference checks, log inspection, and application release. Distinguishes independent replicas +from an elastic collective and states which path was measured. + +**Never** — private engine code, raw infrastructure identities, or broad support claims inferred from +one runtime configuration. The Elastic EP guide owns the profile contract; the resize spec owns design analysis. diff --git a/docs/README.md b/docs/README.md index f7ce9b0a8..66a0350e2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -105,7 +105,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `model-deployment` - Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics. - Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica` -- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md). +- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md). - Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md) - Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/) - Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas) @@ -220,6 +220,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. | [Model Deployment Configuration](modules/model-deployment/deployment.md) | Configure serving roles and overrides. | | [Model Deployment Prefill and Decode](modules/model-deployment/prefill-decode.md) | Pair serving roles. | | [Elastic EP](modules/model-deployment/elastic-ep.md) | Resize one serving instance's collective in place. | +| [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md) | Prefetch weights, expand an elastic collective, and read captured CR output. | | [Engine Versions](modules/model-deployment/engine-versions.md) | Check supported engine versions. | | [Model Deployment Routing](modules/model-deployment/routing.md) | Choose a routing policy, confirm a Router serves. | | [Model Deployment Metrics](modules/model-deployment/metrics.md) | Read serving metrics. | diff --git a/docs/modules/model-deployment/_index.md b/docs/modules/model-deployment/_index.md index e92c76ea7..3bd9e6499 100644 --- a/docs/modules/model-deployment/_index.md +++ b/docs/modules/model-deployment/_index.md @@ -20,8 +20,9 @@ the role contract. [Prefill and Decode](prefill-decode.md) covers split roles. [Routing](routing.md) explain what the running service reports and how requests are directed. -`Elastic EP` is its own mode rather than another role shape: it keeps one serving instance and -changes the size of its collective while it runs. +[Elastic EP](elastic-ep.md) keeps one serving instance and changes its collective while it runs. +[Model Scaling Walkthrough](scaling-walkthrough.md) follows prefetch, TP2/DP2 startup, and DP2→4 expansion +with captured resource output and completed inference requests. --- diff --git a/docs/modules/model-deployment/elastic-ep.md b/docs/modules/model-deployment/elastic-ep.md index be454326c..e14bbee43 100644 --- a/docs/modules/model-deployment/elastic-ep.md +++ b/docs/modules/model-deployment/elastic-ep.md @@ -55,13 +55,15 @@ is why the master is counted inside `width` and is never retired. ## Requirements -The recorded runtime baseline uses a vLLM 0.29.0 image with a full Ray installation. The -official vLLM image of that version ships without Ray, and the profile will not start on it. The -pinned requirement in the resize spec is Ray 2.56.1 with the full default closure. +Use a vLLM image with a full Ray installation. Check the installed engine and Ray versions +before starting the profile. The TP size must fit the selected MoE model, engine kernels, and GPUs available on one node. The operator does not limit TP to the size used by a particular test. -Static TP=2 inference has passed; online TP>1 resizing remains under investigation. + +The [Model Scaling Walkthrough](scaling-walkthrough.md) records successful TP2/DP2 startup and DP2→4 +expansion using the official GPUStack runner and DeepSeek-V2-Lite-Chat on an eight-card RTX PRO 6000 node. +That result is limited to its pinned software, model, configuration, and short inference checks. Admission of TP=2, 4 or 8 does not establish dynamic inference support. The operator directly manages a separate CPU Ray head Pod and its Service. diff --git a/docs/modules/model-deployment/scaling-walkthrough.md b/docs/modules/model-deployment/scaling-walkthrough.md new file mode 100644 index 000000000..4490a2cc5 --- /dev/null +++ b/docs/modules/model-deployment/scaling-walkthrough.md @@ -0,0 +1,579 @@ +# Model Scaling Walkthrough + +Prefetch a model, start one elastic vLLM instance, and grow its expert-parallel collective. +The recorded run used one CPU node and one node with eight NVIDIA RTX PRO 6000 Blackwell GPUs. +Captured custom-resource output appears beside the checks it supports. + +## Contents + +- [Choose the scaling operation](#choose-the-scaling-operation) +- [Prepare the cluster](#prepare-the-cluster) +- [Prefetch the weights](#prefetch-the-weights) +- [Start the elastic deployment](#start-the-elastic-deployment) +- [Check the baseline](#check-the-baseline) +- [Expand the collective](#expand-the-collective) +- [Inspect a stalled resize](#inspect-a-stalled-resize) +- [Scale independent replicas](#scale-independent-replicas) +- [Release the application](#release-the-application) + +## Choose the scaling operation + +| Operation | Field | Effect | Engine scope | +|---|---|---|---| +| Independent serving replicas | `roles[].replicas` | Starts or retires complete instances, each with its own model and engine. | vLLM or SGLang; use the [engine version requirements](engine-versions.md). | +| Elastic DP/EP width | `roles[].elasticEp.width` | Changes the DP engines and EP collective inside one serving instance. | vLLM with Elastic EP, a compatible MoE model, and a full Ray installation. | + +The elastic path keeps `replicas: 1`, `size: 1`, and TP=2 throughout. +Here, `width` is the DP size. Increasing it from 2 to 4 grows the EP world from 4 to 8. +Width 2 consumes four whole GPUs. Width 4 consumes eight. +TP is fixed after creation. This release supports width increases; width decreases are refused. +See [Elastic EP](elastic-ep.md) for the complete contract. + +The recorded validation covers elastic TP2/DP2 startup and DP2→4 expansion with short inference requests. +The independent-replica example is a configuration alternative and was not executed in this run. +Neither path adds Kubernetes nodes or cloud capacity. + +## Prepare the cluster + +Install the operator and enable node model delivery using the +[installation guide](../../operate/installation-modes.md) and +[model-store operations](../model-delivery/operations.md). +The commands below require `kubectl`, `jq`, `curl`, and access to cluster-scoped model-store resources. +Use an isolated kubeconfig for the test cluster. + +The recorded software and model identities were: + +| Component | Recorded identity | +|---|---| +| Operator | `gpustack/gpustack-operator:dev`, binary revision `d00f8fa8c98334a71af4c6eb302ee80d08289309` | +| Operator image digest | `sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36` | +| Inference engine | vLLM 0.29.0 with full Ray 2.54.0 | +| Engine image | `gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91` | +| Model | `deepseek-ai/DeepSeek-V2-Lite-Chat` | +| Model revision | `85864749cd611b4353ce1decdb286193298f64c7` | + +The official GPUStack runner includes Ray. The run used its installed packages without replacement. +Container reads found `cupy-cuda13x` 14.2.0 and no `cupy-cuda12x` installation. +Verify the image digest and installed vLLM/Ray versions when selecting another runner. +Do not add `--elastic-ep-max-dp-size` to this example; that argument starts with vLLM 0.30.0. + +Read the actual operator Pod's image identity and binary version. Repeat for worker, device-manager, +and model-manager Pods. Set the namespace, Pod, and container names from your installation: + +```bash +kubectl get pods -A +kubectl -n "$OPERATOR_NAMESPACE" get pod "$OPERATOR_POD" -o json | + jq '.status.containerStatuses[] | {name, image, imageID}' +kubectl -n "$OPERATOR_NAMESPACE" exec "$OPERATOR_POD" -c "$OPERATOR_CONTAINER" -- \ + gpustack-operator --version +``` + +Recorded binary output: + +```text +gpustack-operator version dev (d00f8fa8c98334a71af4c6eb302ee80d08289309) +``` + +The `dev` tag can move. A matching tag string alone does not identify the binary under test. +Confirm eight free GPUs, model-manager readiness, and enough disk for the weights before starting. + +```bash +kubectl get nodes +kubectl get devices -o json +kubectl get instancetypes +kubectl get nodemodelstores -o json +``` + +## Prefetch the weights + +Save the following as `model-delivery.yaml`. Replace both `gpu-node` values with your GPU node's +`kubernetes.io/hostname` label. The store grant and the prefetch placement must select that same node. +The fixed file list resolves to 12 files and 31,418,807,311 bytes. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: gpustack-elastic +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelArtifact +metadata: + name: deepseek-v2-lite-chat + namespace: gpustack-elastic +spec: + source: + huggingFace: + repository: deepseek-ai/DeepSeek-V2-Lite-Chat + revision: 85864749cd611b4353ce1decdb286193298f64c7 + allowPatterns: + - model-00001-of-000004.safetensors + - model-00002-of-000004.safetensors + - model-00003-of-000004.safetensors + - model-00004-of-000004.safetensors + - config.json + - configuration_deepseek.py + - generation_config.json + - model.safetensors.index.json + - modeling_deepseek.py + - tokenization_deepseek_fast.py + - tokenizer.json + - tokenizer_config.json +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelStore +metadata: + name: rtx6000-models +spec: + nodeSelector: + matchLabels: + kubernetes.io/hostname: gpu-node +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelStoreBinding +metadata: + name: rtx6000-models + namespace: gpustack-elastic +spec: + storeRefs: + - name: rtx6000-models + quota: + bytes: 64Gi + allowPinned: true +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelPrefetch +metadata: + name: deepseek-v2-lite-chat + namespace: gpustack-elastic +spec: + artifactRef: + name: deepseek-v2-lite-chat + bindingRef: + name: rtx6000-models + placement: + nodeSelector: + matchLabels: + kubernetes.io/hostname: gpu-node + minReady: 1 + retention: + pinned: true +``` + +```bash +kubectl apply -f model-delivery.yaml +kubectl -n gpustack-elastic get modelartifacts,modelprefetches +kubectl -n gpustack-elastic get modelprefetch deepseek-v2-lite-chat -o yaml +kubectl get nodemodelstores -o json +``` + +Captured `kubectl get modelartifacts,modelprefetches` output: + +```text +NAME SOURCE REVISION SIZE READY DOWNLOADING RESOLVED +modelartifact.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-ai/DeepSeek-V2-Lite-Chat 85864749cd61 31418807311 1 0 True + +NAME ARTIFACT BINDING READY DESIRED AVAILABLE +modelprefetch.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-v2-lite-chat rtx6000-models 1 1 True +``` + +Check the matching node's model entry as well: + +```bash +kubectl get nodemodelstores -o json | + jq '.items[] | .status.models[]? | {digest, sizeBytes, state}' +``` + +Captured projection: + +```json +{ + "digest": "sha256:1369aae0b5cc2d0e260557bcea04b0d6aad40f8bb6accdac60f1ee543887d9f7", + "sizeBytes": 31418807311, + "state": "Ready" +} +``` + +The validation also retained the successful warm-up Pod and compared its 12 SHA-256 results with +the resolved manifest. Warm-up Pods are transient; collect their logs before completion if you need +that evidence. See [Model Prefetch](../model-delivery/prefetch.md). +Wait for the required content before creating the serving workload. + +## Start the elastic deployment + +Save this as `elastic.yaml`. The InstanceType matches the recorded RTX PRO 6000 hardware. +For another GPU family, use an appropriate existing whole-GPU InstanceType and verify model compatibility. +Each GPU member requests two GPUs; its CPU and memory budgets scale with that count. + +```yaml +apiVersion: worker.gpustack.ai/v1 +kind: InstanceType +metadata: + name: elastic-cpu-gpu +spec: + generalGroup: generic + acceleratable: true + acceleratorGroup: nvidia-rtx-pro-6000-blackwell-server-edition + os: linux + arch: amd64 + localStorage: 10Gi + unitResources: + cpu: "4" + ram: 16Gi +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelDeployment +metadata: + name: tp2-elastic + namespace: gpustack-elastic +spec: + model: + name: /models + artifactRef: + name: deepseek-v2-lite-chat + engine: + name: vLLM + version: 0.29.0 + roles: + - name: server + kind: Server + replicas: 1 + size: 1 + instanceType: elastic-cpu-gpu + image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91 + imagePullPolicy: IfNotPresent + resources: + accelerator: "2" + elasticEp: + width: 2 + extraArgs: + - --tensor-parallel-size + - "2" + - --max-model-len + - "4096" + - --max-num-seqs + - "32" + - --enforce-eager + - --all2all-backend + - allgather_reducescatter + - --api-server-count + - "1" + env: + - name: VLLM_USE_RAY_V2_EXECUTOR_BACKEND + value: "1" + - name: RAY_LOGGER_LEVEL + value: debug + - name: RAY_BACKEND_LOG_LEVEL + value: debug + - name: RAY_DEDUP_LOGS + value: "0" +``` + +```bash +kubectl apply -f elastic.yaml +kubectl -n gpustack-elastic get modeldeployments,pods,services +``` + +The operator creates one CPU-only Ray head without resource requests or limits. +Normal Kubernetes admission and scheduling still apply. +In the recorded run, this head landed on the GPU node but claimed no GPU. The two GPU Pods each exposed two GPUs to Ray autodiscovery. + +With `shmSize` omitted, the recorded `/dev/shm` capacity was 16 GiB on every elastic Pod. +Ray's object store used that mount. A separate CPU-only probe verified explicit `shmSize: 32Gi`. +Kubelet can cap the mount by node allocatable memory or Pod memory limits; read the actual mount. +See [Shared memory](deployment.md#shared-memory) before changing the budget. + +The recorded test also installed a read-only diagnostic extension to measure native parallel groups. +That test instrumentation is omitted from this application manifest. + +This deployment has one serving instance and no router. Only its master exposes the inference API; +the other GPU members participate through Ray. The run tests collective expansion and inference recovery. +Router forwarding, replica load balancing, and failover are outside this recorded test. + +## Check the baseline + +Save each CR snapshot before changing the width: + +```bash +kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > baseline-cr.json +jq '{name: .metadata.name, width: .spec.roles[0].elasticEp.width, + phase: .status.phase, roles: [.status.roles[] | + {name, desired, quotaReserved, ready, parallelism}]}' baseline-cr.json +``` + +Projection of the captured baseline CR: + +```json +{ + "name": "tp2-elastic", + "width": 2, + "phase": "Ready", + "roles": [ + { + "name": "server", + "desired": 1, + "quotaReserved": 1, + "ready": 1, + "parallelism": { + "declared": { + "tensorParallel": 2 + }, + "loadBalance": "Internal", + "source": { + "complete": true, + "kind": "ExtraArgs" + } + } + } + ] +} +``` + +The role still has one serving instance. `width` is desired state, and `parallelism.declared` describes +arguments. Neither field proves the runtime's effective DP width. +The captured conditions included `ReplicasUpToDate=Unknown` with reason `RolloutNotObserved`; +`phase: Ready` did not turn every condition true. + +In a separate terminal, forward the serving Service: + +```bash +kubectl -n gpustack-elastic port-forward service/tp2-elastic 8000:8000 +``` + +Send both request types. Keep the headers and response bodies. Each command makes one attempt. +The recorded test sent equivalent requests directly to the master inside the cluster; +the port-forward is a workstation access option. + +```bash +curl --fail-with-body --max-time 30 -D ordinary.headers \ + http://127.0.0.1:8000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"/models","prompt":"Complete the sentence: The sky is","max_tokens":8,"temperature":0,"stream":false}' \ + -o ordinary.json +jq '{text: .choices[0].text, finish_reason: .choices[0].finish_reason, + completion_tokens: .usage.completion_tokens}' ordinary.json + +curl --fail-with-body --no-buffer --max-time 30 -D stream.headers \ + http://127.0.0.1:8000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"/models","prompt":"Complete the sentence: The sky is","max_tokens":8,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}' \ + -o stream.txt +cat stream.txt +``` + +Require HTTP 200, nonempty generated text, positive completion tokens, and a final generation reason. +For SSE, require the usage event and `data: [DONE]`. A health response does not exercise generation. +The recorded requests used unique request IDs and checked their response IDs as well. + +## Expand the collective + +Record the master Pod's UID and restart count before the patch. Select ordinal zero within this deployment: + +```bash +kubectl -n gpustack-elastic get pods -o json | + jq '.items[] | select(.metadata.labels["app.kubernetes.io/instance"] == "tp2-elastic") | + select(.metadata.labels["app.kubernetes.io/component"] == "server") | + select(.metadata.labels["modeldeployment.gpustack.ai/pod-ordinal"] == "0") | + {name: .metadata.name, uid: .metadata.uid, + containers: [.status.containerStatuses[] | {name, containerID, restartCount}]}' + +kubectl -n gpustack-elastic patch modeldeployment tp2-elastic --type=json \ + -p '[{"op":"test","path":"/spec/roles/0/elasticEp/width","value":2},{"op":"replace","path":"/spec/roles/0/elasticEp/width","value":4}]' +``` + +Wait for the operator's observation of the current generation to reach width 4 at every layer. +Read the diagnostic ConfigMap belonging to this deployment UID: + +```bash +kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > current-cr.json +MD_UID=$(jq -r '.metadata.uid' current-cr.json) +MD_GENERATION=$(jq -r '.metadata.generation' current-cr.json) +kubectl -n gpustack-elastic get configmaps -o json | + jq --arg uid "$MD_UID" '[.items[] | + select(any(.metadata.ownerReferences[]?; .uid == $uid)) | + select(.data["observation.json"] != null) | + .data["observation.json"] | fromjson] | + if length == 1 then .[0] else error("expected one observation") end' > observation.json +jq -e --argjson generation "$MD_GENERATION" ' + .observedGeneration == $generation and + ([.desired, .admitted, .allocated, .ray, .effective] | + all(.Known == true and .Value == 4))' observation.json +``` + +If this check fails, continue observing or inspect the hold reason before declaring completion. +The ConfigMap is internal diagnostic state, not a public CR status API. +Its effective-width observation includes rank-directed inference probes. +Pair it with the retained master identity and both client request checks below. + +After the observation passes, save the final CR: + +```bash +kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o json > scaled-cr.json +jq '{name: .metadata.name, width: .spec.roles[0].elasticEp.width, + phase: .status.phase, desired: .status.roles[0].desired, + ready: .status.roles[0].ready}' scaled-cr.json +``` + +Projection captured after the resize completed: + +```json +{ + "name": "tp2-elastic", + "width": 4, + "phase": "Ready", + "desired": 1, + "ready": 1 +} +``` + +Confirm four GPU members and one head. Read the master and Ray logs, then repeat both inference requests. +The engine may withdraw readiness and return 503 during reconfiguration. +Do not resend the width change while the current operation is unresolved. + +The recorded result combined native group measurements, actor state, GPU allocation, and completed requests: + +| Check | Width 2 | Width 4 | +|---|---|---| +| Actual TP / DP / EP world | 2 / 2 / 4 | 2 / 4 / 8 | +| Distinct allocated GPU UUIDs | 4 | 8 | +| Live EngineCore actors | 2 | 4 | +| Live GPU worker actors | 4 | 8 | +| Ordinary request | HTTP 200, 8 generated tokens | HTTP 200, 8 generated tokens | +| SSE request | HTTP 200, 8 generated tokens, `[DONE]` | HTTP 200, 8 generated tokens, `[DONE]` | + +All four recorded requests succeeded on their first attempt. The original master UID and HTTP process +identity remained unchanged. The measurements joined native workers to Ray actors, Pod identities, +and host GPU UUIDs. Placement groups used `STRICT_PACK`, with two GPU bundles per DP engine. + +These checks do not establish sustained concurrency, uninterrupted requests throughout resizing, +per-rank request coverage, or inference quality. The response's `system_fingerprint` retained its startup DP2 text after +expansion, so it was not used to establish the effective width. + +## Inspect a stalled resize + +Read the CR conditions and the owned Pods first. Select the head, master, or worker from the Pod list: + +```bash +kubectl -n gpustack-elastic get modeldeployment tp2-elastic -o yaml +kubectl -n gpustack-elastic get pods -o wide +kubectl -n gpustack-elastic get events --sort-by=.metadata.creationTimestamp +kubectl -n gpustack-elastic logs "$POD" -c main --timestamps --tail=200 +kubectl -n gpustack-elastic exec "$POD" -c main -- \ + sh -c 'ls /tmp/ray/session_latest/logs' +kubectl -n gpustack-elastic exec "$POD" -c main -- \ + sh -c 'tail -200 /tmp/ray/session_latest/logs/raylet.out /tmp/ray/session_latest/logs/raylet.err' +``` + +| Symptom | Read next | +|---|---| +| GPU Pods remain Pending | Kueue Workloads, queue quota, Pod events, InstanceType capacity, and the per-card `Devices` ledger. | +| Placement groups exist but inference times out | Master logs, EngineCore actor state, GPU worker logs, and Ray session logs. `CREATED` placement groups alone do not establish inference readiness. | +| Ray sees the wrong GPU count | Actual allocation annotations, container-visible GPUs, and live Ray node capacity. Each GPU member must expose exactly TP GPUs. | +| Unexpected engine arguments or missing Ray | Running image digest and installed vLLM/Ray versions. `spec.engine.version` does not inspect the image. | +| Shared-memory pressure | Actual `/dev/shm` capacity, Pod memory limits, node allocatable memory, and the object store's location. | + +`RAY_LOGGER_LEVEL` controls Python Ray logging; `RAY_BACKEND_LOG_LEVEL` controls backend logging. +The operator defaults both levels to `info`. +The manifest sets both to `debug` and disables deduplication for inspection. +Environment changes affect newly created Pods. Preserve current logs before replacing a failing deployment. + +An earlier run reached `CREATED` placement groups and eight live GPU workers, but new EngineCore actors +remained pending and requests timed out. This recorded run completed expansion and inference. +The earlier root cause remains unconfirmed; the result does not isolate shared memory or head placement as its cause. + +## Scale independent replicas + +Use a separate deployment without `elasticEp` when you need independent serving instances. +This alternative was not runtime-tested in the recorded elastic run. +First [release the elastic application](#release-the-application) so it does not consume all eight GPUs. +Keep the shared artifact, cache, and InstanceType. + +Save this as `replicas.yaml`: + +```yaml +apiVersion: worker.gpustack.ai/v1 +kind: ModelDeployment +metadata: + name: replica-demo + namespace: gpustack-elastic +spec: + model: + name: /models + artifactRef: + name: deepseek-v2-lite-chat + engine: + name: vLLM + version: 0.29.0 + roles: + - name: server + kind: Server + replicas: 1 + size: 1 + instanceType: elastic-cpu-gpu + image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91 + resources: + accelerator: "2" + extraArgs: + - --tensor-parallel-size + - "2" + - --max-model-len + - "4096" + - --enforce-eager +``` + +```bash +kubectl apply -f replicas.yaml +kubectl -n gpustack-elastic patch modeldeployment replica-demo --type=json \ + -p '[{"op":"replace","path":"/spec/roles/0/replicas","value":2}]' +kubectl -n gpustack-elastic get modeldeployment replica-demo -o yaml +kubectl -n gpustack-elastic get pods +``` + +Two replicas each request two GPUs, for four GPUs total. Each replica has its own TP2 engine. +Read role readiness and endpoint eligibility, and send inference to each instance before declaring both usable. +Managed routing is a separate configuration; see [Routing](routing.md). +SGLang also supports independent replicas, with its own runner image and engine arguments. + +## Release the application + +When you have finished inspecting the elastic application, delete only its ModelDeployment: + +```bash +kubectl -n gpustack-elastic delete modeldeployment tp2-elastic --wait=true +kubectl -n gpustack-elastic get modeldeployments,pods,services +kubectl get devices -o json > released-devices.json +jq '[.items[].status.groups[].accelerators[] | + {mode: (.mode // 0), remaining, allocatedSlices: (.allocatedSlices // 0)}] | + {cards: length, free: map(select(.mode == 0 and .remaining == 1600000 and .allocatedSlices == 0)) | length}' \ + released-devices.json +``` + +The release check used an earlier test-runner deployment on the same otherwise idle node. +Captured projection after deleting the earlier application: + +```json +{ + "cards": 8, + "free": 8 +} +``` + +That release check also verified deletion of the owned head, GPU Pods, Services, and Workload. +For a shared cluster, compare the application's captured device claims with the ledger; +other workloads may keep cards occupied. If you ran the replica alternative, delete `replica-demo` as well. + +ModelPrefetch keeps the model pinned independently of the serving application. +When no consumer needs the weights, remove the prefetch and its dedicated binding, artifact, and store. +See [Model Prefetch](../model-delivery/prefetch.md) for cache retention and collection behavior. + +Application deletion leaves the Kubernetes nodes, disks, and cloud network in place. +Retain them while the owner is inspecting logs. Delete infrastructure only after the owner approves, +using the provisioner's recorded resource ownership, and verify deletion in the provider and state file. + +--- + +**See also** — [Elastic EP](elastic-ep.md) (profile contract) · +[Model Prefetch](../model-delivery/prefetch.md) (weight delivery) · +[GPU Instances](../instances/_index.md) (direct GPU requests) + +**Next** → [Model Deployment Status](status.md) — interpret conditions and serving endpoints. diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml index ca2a686eb..a719b82ca 100644 --- a/site/data/navigation.yaml +++ b/site/data/navigation.yaml @@ -58,6 +58,7 @@ order: /docs/modules/model-deployment/: - deployment - elastic-ep + - scaling-walkthrough - engine-versions - prefill-decode - routing @@ -113,6 +114,7 @@ labels: /docs/modules/model-delivery/views/: API reference /docs/modules/model-deployment/deployment/: Configuration /docs/modules/model-deployment/elastic-ep/: Elastic EP + /docs/modules/model-deployment/scaling-walkthrough/: Scaling walkthrough /docs/modules/model-deployment/engine-versions/: Engine versions /docs/modules/model-deployment/prefill-decode/: Prefill and decode /docs/modules/model-deployment/routing/: Routing From 21cf4fac53093d0a272bc31ca27bf4ae72c3c831 Mon Sep 17 00:00:00 2001 From: thxCode Date: Thu, 8 Oct 2026 19:01:21 +0800 Subject: [PATCH 2/4] docs(model-deployment): record external DP and router validation Signed-off-by: thxCode --- .../skills/gpustack-operator-docs/SKILL.md | 1 + .../references/page-map.md | 8 + docs/README.md | 3 +- docs/modules/model-deployment/_index.md | 2 + .../external-dp-walkthrough.md | 306 ++++++++++++++++++ .../model-deployment/prefill-decode.md | 5 + docs/modules/model-deployment/routing.md | 6 +- .../model-deployment/scaling-walkthrough.md | 26 +- site/data/navigation.yaml | 2 + 9 files changed, 354 insertions(+), 5 deletions(-) create mode 100644 docs/modules/model-deployment/external-dp-walkthrough.md diff --git a/.agents/skills/gpustack-operator-docs/SKILL.md b/.agents/skills/gpustack-operator-docs/SKILL.md index b70895ea3..fea73f99b 100644 --- a/.agents/skills/gpustack-operator-docs/SKILL.md +++ b/.agents/skills/gpustack-operator-docs/SKILL.md @@ -71,6 +71,7 @@ one, not to widen the overview. | Running node delivery: the chart values, where the node's configuration comes from, reading a node, the watermark cap, switching delivery, where replicas land and turning the preference off, upgrading, removing the cache | `docs/modules/model-delivery/operations.md` | | The `ModelDeployment` contract: the inherited reuse domain, the three override tiers, the owned-key table, the runner-image formula, prefill/decode pairing, the topology-placement field contract | `docs/modules/model-deployment/deployment.md` | | A recorded model-serving sequence from prefetch through elastic expansion, its captured CR output, or the distinction from independent replica scaling | `docs/modules/model-deployment/scaling-walkthrough.md` | +| A fixed External DP group, its minimum arguments, per-rank requests, or measured Router coverage | `docs/modules/model-deployment/external-dp-walkthrough.md` | | A `ModelDeployment` metrics snapshot, which series each field reads per engine, role and router, cache-hit scope or Pod scrape annotation | `docs/modules/model-deployment/metrics.md` | | What a `ModelDeployment` status condition or published field means, and how to read them when a deployment misbehaves | `docs/modules/model-deployment/status.md` | | How a prefill role and a decode role are paired: the connector each engine and router renders, `spec.router` and its fields, `spec.kvTransfer`, roles on different hardware, a role's own Service | `docs/modules/model-deployment/prefill-decode.md` | diff --git a/.agents/skills/gpustack-operator-docs/references/page-map.md b/.agents/skills/gpustack-operator-docs/references/page-map.md index 616d92b3c..003cbc211 100644 --- a/.agents/skills/gpustack-operator-docs/references/page-map.md +++ b/.agents/skills/gpustack-operator-docs/references/page-map.md @@ -437,3 +437,11 @@ from an elastic collective and states which path was measured. **Never** — private engine code, raw infrastructure identities, or broad support claims inferred from one runtime configuration. The Elastic EP guide owns the profile contract; the resize spec owns design analysis. + +## `docs/modules/model-deployment/external-dp-walkthrough.md` + +**Owns** — the fixed External DP serving sequence, minimum argument dependencies, captured CR output, +per-rank inference checks, Router coverage limits, and application cleanup. Links shared model prefetch steps. + +**Never** — imply online collective resize from fixed-group inference, or claim all ranks receive Router +traffic from HTTP success alone. P/D combinations remain with their owning guide and follow-up issue. diff --git a/docs/README.md b/docs/README.md index 66a0350e2..259a174e6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -105,7 +105,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `model-deployment` - Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics. - Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica` -- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md). +- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md); fixed per-rank HTTP serving in [External DP Walkthrough](modules/model-deployment/external-dp-walkthrough.md). - Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md) - Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/) - Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas) @@ -221,6 +221,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. | [Model Deployment Prefill and Decode](modules/model-deployment/prefill-decode.md) | Pair serving roles. | | [Elastic EP](modules/model-deployment/elastic-ep.md) | Resize one serving instance's collective in place. | | [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md) | Prefetch weights, expand an elastic collective, and read captured CR output. | +| [External DP Walkthrough](modules/model-deployment/external-dp-walkthrough.md) | Run fixed External DP ranks and verify Router coverage. | | [Engine Versions](modules/model-deployment/engine-versions.md) | Check supported engine versions. | | [Model Deployment Routing](modules/model-deployment/routing.md) | Choose a routing policy, confirm a Router serves. | | [Model Deployment Metrics](modules/model-deployment/metrics.md) | Read serving metrics. | diff --git a/docs/modules/model-deployment/_index.md b/docs/modules/model-deployment/_index.md index 3bd9e6499..9b3566e2a 100644 --- a/docs/modules/model-deployment/_index.md +++ b/docs/modules/model-deployment/_index.md @@ -23,6 +23,8 @@ are directed. [Elastic EP](elastic-ep.md) keeps one serving instance and changes its collective while it runs. [Model Scaling Walkthrough](scaling-walkthrough.md) follows prefetch, TP2/DP2 startup, and DP2→4 expansion with captured resource output and completed inference requests. +[External DP Walkthrough](external-dp-walkthrough.md) runs a fixed group with one HTTP endpoint per rank +and checks managed Router coverage. --- diff --git a/docs/modules/model-deployment/external-dp-walkthrough.md b/docs/modules/model-deployment/external-dp-walkthrough.md new file mode 100644 index 000000000..5090d308b --- /dev/null +++ b/docs/modules/model-deployment/external-dp-walkthrough.md @@ -0,0 +1,306 @@ +# External DP Walkthrough + +Run one fixed vLLM data-parallel (DP) group with an HTTP endpoint on each rank. +This walkthrough records two ranks using the official GPUStack runner. +Direct requests passed on both ranks. The recorded operator's managed Router discovered only rank 0. +That routing limitation remains visible in the checks below. + +## Contents + +- [Choose the deployment shape](#choose-the-deployment-shape) +- [Prepare the model and GPUs](#prepare-the-model-and-gpus) +- [Start the fixed group](#start-the-fixed-group) +- [Omit redundant arguments](#omit-redundant-arguments) +- [Read the running resources](#read-the-running-resources) +- [Send requests to each rank](#send-requests-to-each-rank) +- [Check Router coverage](#check-router-coverage) +- [Understand the scaling limits](#understand-the-scaling-limits) +- [Inspect and release the application](#inspect-and-release-the-application) + +## Choose the deployment shape + +This example uses one Server role with `replicas: 1` and `size: 2`. +Each Pod has two GPUs and runs one DP rank with tensor parallelism (TP)=2. +With expert parallelism (EP) enabled, the collective has four GPU workers: TP2, DP2, EP world4. +The deployment consumes four whole GPUs. + +External LB means each DP rank accepts its own requests. +A load balancer must distribute requests across those HTTP endpoints. +The `round_robin` policy chooses among discovered endpoints; it cannot repair a missing endpoint. + +This example uses multiprocessing (`mp`) for both DP and local TP execution. +The runner contains Ray, but this fixed group does not start Ray. +See [vLLM's DP deployment guide](https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment/) +for the engine's Internal and External LB distinction. + +## Prepare the model and GPUs + +Complete [cluster preparation](scaling-walkthrough.md#prepare-the-cluster) and +[model prefetch](scaling-walkthrough.md#prefetch-the-weights) first. +Use the same namespace, cached artifact, model revision, and GPU InstanceType. +The GPU node needs four free cards and the verified model cache. + +For the recorded hardware, copy the `InstanceType` document from +[the elastic deployment example](scaling-walkthrough.md#start-the-elastic-deployment) +into `external-instance-type.yaml`. Apply that single resource before creating this deployment: + +```bash +kubectl apply -f external-instance-type.yaml +kubectl get instancetype elastic-cpu-gpu +``` + +For another GPU family, select an existing whole-GPU InstanceType and replace the name in the manifest below. + +The recorded operator binary was `d00f8fa8c98334a71af4c6eb302ee80d08289309` from +`gpustack/gpustack-operator:dev@sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36`. +The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements. +Its CuPy installation contained only `cupy-cuda13x` 14.2.0. + +Each Pod needs a separate Pod IP. The ranks must reach each other over the Pod network. +The DP RPC port defaults to 29550; collective communication also uses other ports. +Setting an engine RPC port does not create a Kubernetes Service or a firewall rule. + +## Start the fixed group + +Save this as `external-dp.yaml`. Replace `elastic-cpu-gpu` with your GPU InstanceType if needed. +The model delivery steps already created `deepseek-v2-lite-chat` in `gpustack-elastic`. + +```yaml +apiVersion: worker.gpustack.ai/v1 +kind: ModelDeployment +metadata: + name: external-minimal + namespace: gpustack-elastic +spec: + model: + name: /models + artifactRef: + name: deepseek-v2-lite-chat + engine: + name: vLLM + version: "0.29.0" + roles: + - name: server + kind: Server + replicas: 1 + size: 2 + instanceType: elastic-cpu-gpu + image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91 + imagePullPolicy: IfNotPresent + resources: + accelerator: "2" + extraArgs: + - --tensor-parallel-size + - "2" + - --data-parallel-size + - "2" + - --data-parallel-rank + - $(GPUSTACK_MEMBER_INDEX) + - --data-parallel-address + - $(GPUSTACK_REPLICA_LEADER_ADDRESS) + - --enable-expert-parallel + - --max-model-len + - "4096" + - --max-num-seqs + - "32" + - --enforce-eager + - --all2all-backend + - allgather_reducescatter + router: + name: vllm-router + image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db + extraArgs: + - --policy + - round_robin +``` + +The operator supplies the member index and replica leader address as environment variables. +Kubernetes expands their references in the container arguments. +Rank 0 and rank 1 therefore share the leader address and receive distinct rank values. +The ordinary Server role still requires explicit TP, DP, rank, and address arguments. +It does not infer those arguments from `size` or GPU requests. + +The length, sequence, eager-execution, and all-to-all arguments preserve the recorded model settings. +They do not select External LB. +The measured run also mounted a diagnostic worker extension to read native groups and GPU identities. +That test instrumentation is omitted from this serving manifest. + +```bash +kubectl apply --dry-run=server -f external-dp.yaml +kubectl apply -f external-dp.yaml +kubectl -n gpustack-elastic get modeldeployment external-minimal +kubectl -n gpustack-elastic get pods \ + -l app.kubernetes.io/instance=external-minimal +``` + +## Omit redundant arguments + +The minimal run omitted all six arguments below and still completed ordinary and streaming requests on both ranks. +These defaults apply to this vLLM version and deployment shape. + +| Omitted argument | Why it is unnecessary here | +|---|---| +| `--data-parallel-external-lb` | An explicit `--data-parallel-rank` selects External LB. | +| `--data-parallel-size-local 1` | External LB runs one local DP rank. | +| `--api-server-count 1` | External LB defaults to one API server per rank. | +| `--data-parallel-backend mp` | `mp` is the default DP backend. | +| `--distributed-executor-backend mp` | The local TP group fits the Pod's two GPUs and runs outside Ray. | +| `--data-parallel-rpc-port 13345` | Both ranks can use the default RPC port, 29550. | + +The operator generates the ordinary HTTP serving port, model mount, Pod wiring, and Services. +This example serves each rank on port 8000. +`--data-parallel-backend` selects execution placement; it does not select the LB mode. +`--data-parallel-start-rank` alone is not an External LB declaration. + +## Read the running resources + +Retain the CR output beside the request results: + +```bash +kubectl -n gpustack-elastic get modeldeployment external-minimal -o json | + jq '{name: .metadata.name, phase: .status.phase, + size: .spec.roles[0].size, replicas: .spec.roles[0].replicas, + accelerator: .spec.roles[0].resources.accelerator}' +``` + +Recorded values from the captured CR: + +```json +{ + "name": "external-minimal", + "phase": "Ready", + "size": 2, + "replicas": 1, + "accelerator": "2" +} +``` + +Read Pod UIDs, images, addresses, and endpoint labels separately: + +```bash +kubectl -n gpustack-elastic get pods \ + -l app.kubernetes.io/instance=external-minimal -o json | + jq '.items[] | {name: .metadata.name, uid: .metadata.uid, + labels: .metadata.labels, podIP: .status.podIP, + containers: .status.containerStatuses}' +kubectl get devices -o json +``` + +The recorded native checks found: + +| Pod member | DP rank | TP size | DP size | EP world | Distinct GPUs | +|---|---|---|---|---|---| +| 0 | 0 | 2 | 2 | 4 | 2 | +| 1 | 1 | 2 | 2 | 4 | 2 | + +All four worker GPU identities were distinct. Both ranks reported External LB enabled and Elastic EP disabled. +Declared arguments and `Ready` status alone do not establish these native results. + +## Send requests to each rank + +Open one port-forward per rank in separate terminals: + +```bash +kubectl -n gpustack-elastic port-forward pod/external-minimal-server-r0-m0 18000:8000 +``` + +```bash +kubectl -n gpustack-elastic port-forward pod/external-minimal-server-r0-m1 18001:8000 +``` + +Send an ordinary request and a streaming request to each endpoint: + +```bash +for port in 18000 18001; do + curl --fail-with-body --max-time 30 "http://127.0.0.1:$port/v1/completions" \ + -H 'Content-Type: application/json' \ + -d '{"model":"/models","prompt":"The sky is","max_tokens":8,"temperature":0}' + curl --fail-with-body --no-buffer --max-time 30 "http://127.0.0.1:$port/v1/completions" \ + -H 'Content-Type: application/json' \ + -d '{"model":"/models","prompt":"A GPU is","max_tokens":8,"temperature":0,"stream":true,"stream_options":{"include_usage":true}}' +done +``` + +The measured requests used direct Pod addresses inside the cluster. Each request had one attempt. +Both ranks returned HTTP 200, nonempty text, eight completion tokens, and a valid finish reason. +Both streams ended with `data: [DONE]`. +These short checks establish functional inference, not throughput or sustained-load behavior. + +## Check Router coverage + +Discover the Router Pod and forward its API and metrics ports: + +```bash +ROUTER_POD=$(kubectl -n gpustack-elastic get pods \ + -l app.kubernetes.io/instance=external-minimal -o json | + jq -r '.items[] | select(any(.spec.containers[]; .name == "router")) | .metadata.name') +kubectl -n gpustack-elastic port-forward "pod/$ROUTER_POD" 18081:8081 19090:9090 +``` + +In another terminal, read the registered workers and per-worker request counters: + +```bash +curl --fail-with-body http://127.0.0.1:18081/observer/endpoints | + jq '.workers[] | {url, selection}' +curl --fail-with-body http://127.0.0.1:19090/metrics | + grep '^vllm_router_processed_requests_total' +``` + +Repeat the completion requests through port 18081. Compare the counters before and after. +For complete External LB coverage, the registry must contain both rank endpoints. +Requests must increase each rank's counter. A successful response through the Router is insufficient. + +The recorded Router registered only member 0 because its selector retained `member-index=0`. +All four Router requests completed, but all four increased only member 0's counter. +Member 1 received no Router traffic. Direct rank inference passed; managed Router coverage failed. +Keep this distinction when evaluating the recorded operator revision. + +For ordinary Internal groups and [P/D deployments](prefill-decode.md), the leader restriction remains necessary. +External DP inside a Prefill or Decode role is deferred to +[issue #754](https://github.com/gpustack/gpustack-operator/issues/754). +This Server example does not validate that combination. + +## Understand the scaling limits + +This fixed External DP group has no online DP/EP resize contract. +vLLM 0.29.0 rejects combining its native Elastic EP mode with External or Hybrid LB. +Use the [Ray Internal Elastic walkthrough](scaling-walkthrough.md) for `elasticEp.width` growth. +A Router in front of that Internal endpoint does not turn the engine into External LB. + +Increasing `roles[].replicas` creates additional complete groups with their existing TP/DP/EP shape. +That operation does not enlarge an existing group's EP world and was not measured in this External run. +The ordinary role's `size` cannot be changed after creation. +Hybrid and multiple HTTP ports per Pod are outside this walkthrough. + +## Inspect and release the application + +Read both ranks' logs when startup or collective requests stall: + +```bash +kubectl -n gpustack-elastic logs external-minimal-server-r0-m0 -c main +kubectl -n gpustack-elastic logs external-minimal-server-r0-m1 -c main +kubectl -n gpustack-elastic get modeldeployment external-minimal -o yaml +kubectl -n gpustack-elastic get services,endpointslices +``` + +Check matching DP sizes, distinct rank values, a shared leader address, network access, and free GPU resources. +Keep Router discovery evidence separate from engine startup and inference evidence. +The replica's headless Service supplies peer DNS; narrowing it to the HTTP leader can prevent group startup. + +After saving the results, stop the port-forward processes and delete this application: + +```bash +kubectl -n gpustack-elastic delete modeldeployment external-minimal +kubectl -n gpustack-elastic get pods,services,deployments \ + -l app.kubernetes.io/instance=external-minimal +kubectl get devices -o json +``` + +Verify that its owned workloads are gone and all four GPU claims are released. +The recorded cleanup deleted both External test applications and returned all eight cards to their free state. +Deleting the application retains the model cache and cloud cluster. + +--- + +**See also** — [Model Scaling Walkthrough](scaling-walkthrough.md) (prefetch and Elastic EP) · +[Routing](routing.md) (backend selection) · [Model Deployment Configuration](deployment.md) (role fields) diff --git a/docs/modules/model-deployment/prefill-decode.md b/docs/modules/model-deployment/prefill-decode.md index 38da40cb4..2c335d0b8 100644 --- a/docs/modules/model-deployment/prefill-decode.md +++ b/docs/modules/model-deployment/prefill-decode.md @@ -19,6 +19,11 @@ decode](deployment.md#prefill-and-decode) describes the role fields and admissio Without `spec.router`, `kind` adds the role discriminator to the engine's shared-store connector and nothing pairs the two roles. +For a multi-Pod P/D replica, managed HTTP routing selects its leader (`member-index=0`). +A router policy such as `round_robin` chooses among those leaders; it does not add non-HTTP members. +External DP inside a Prefill or Decode role has not been validated. +[Issue #754](https://github.com/gpustack/gpustack-operator/issues/754) tracks that deferred combination. + With the managed `llm-d-router`, the connector a native vLLM role runs depends on whether `spec.kvCache` is set; the two settings render two different connector documents, not one document with a field toggled: diff --git a/docs/modules/model-deployment/routing.md b/docs/modules/model-deployment/routing.md index 84d1e3097..dbf11d44b 100644 --- a/docs/modules/model-deployment/routing.md +++ b/docs/modules/model-deployment/routing.md @@ -1,8 +1,12 @@ # Model Deployment Routing -A router only chooses when a role has more than one replica. Every routing choice measured here ran +A managed router selects among the HTTP endpoints of serving replicas. Every routing choice measured here ran on a server role; on a prefill/decode pair, where each half is chosen separately, none has been run. +External DP within a Prefill or Decode role remains unverified; see +[issue #754](https://github.com/gpustack/gpustack-operator/issues/754). +Keep each P/D replica's leader as its HTTP endpoint until that combination is validated. + ## Contents - [Each router's default](#each-routers-default) diff --git a/docs/modules/model-deployment/scaling-walkthrough.md b/docs/modules/model-deployment/scaling-walkthrough.md index 4490a2cc5..394070097 100644 --- a/docs/modules/model-deployment/scaling-walkthrough.md +++ b/docs/modules/model-deployment/scaling-walkthrough.md @@ -32,6 +32,8 @@ See [Elastic EP](elastic-ep.md) for the complete contract. The recorded validation covers elastic TP2/DP2 startup and DP2→4 expansion with short inference requests. The independent-replica example is a configuration alternative and was not executed in this run. Neither path adds Kubernetes nodes or cloud capacity. +For a fixed group with a separate HTTP endpoint per DP rank, use the +[External DP Walkthrough](external-dp-walkthrough.md). ## Prepare the cluster @@ -286,9 +288,22 @@ See [Shared memory](deployment.md#shared-memory) before changing the budget. The recorded test also installed a read-only diagnostic extension to measure native parallel groups. That test instrumentation is omitted from this application manifest. -This deployment has one serving instance and no router. Only its master exposes the inference API; -the other GPU members participate through Ray. The run tests collective expansion and inference recovery. -Router forwarding, replica load balancing, and failover are outside this recorded test. +This manifest has one serving instance and no router. Only its master exposes the inference API; +the other GPU members participate through Ray. +A separate recorded run added this `spec.router` block before creating the deployment: + +```yaml +router: + name: vllm-router + image: gpustack/llm-router:v0.2.0@sha256:98e70d94351baa9dc13897a545aff5c83ffac5e89ab810e204bc894b519d96db + extraArgs: + - --policy + - round_robin +``` + +The engine still uses Internal LB. This Router forwards to the master, which distributes work to the DP engines. +Use the [Router observation steps](external-dp-walkthrough.md#check-router-coverage) with deployment name `tp2-elastic`. +For this Elastic shape, expect one master endpoint; exclude the CPU head and other GPU members. ## Check the baseline @@ -445,6 +460,11 @@ All four recorded requests succeeded on their first attempt. The original master identity remained unchanged. The measurements joined native workers to Ray actors, Pod identities, and host GPU UUIDs. Placement groups used `STRICT_PACK`, with two GPU bundles per DP engine. +The additional Router run also completed width 2→4 with the same official runner. +Before and after expansion, its selector and registry contained only the original master. +Each stage completed one ordinary request and one SSE request through the Router, with HTTP 200 and eight tokens. +The master's Router counter increased by two at each stage. Worker and CPU-head endpoints were absent. + These checks do not establish sustained concurrency, uninterrupted requests throughout resizing, per-rank request coverage, or inference quality. The response's `system_fingerprint` retained its startup DP2 text after expansion, so it was not used to establish the effective width. diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml index a719b82ca..f0c1c1d6f 100644 --- a/site/data/navigation.yaml +++ b/site/data/navigation.yaml @@ -59,6 +59,7 @@ order: - deployment - elastic-ep - scaling-walkthrough + - external-dp-walkthrough - engine-versions - prefill-decode - routing @@ -115,6 +116,7 @@ labels: /docs/modules/model-deployment/deployment/: Configuration /docs/modules/model-deployment/elastic-ep/: Elastic EP /docs/modules/model-deployment/scaling-walkthrough/: Scaling walkthrough + /docs/modules/model-deployment/external-dp-walkthrough/: External DP walkthrough /docs/modules/model-deployment/engine-versions/: Engine versions /docs/modules/model-deployment/prefill-decode/: Prefill and decode /docs/modules/model-deployment/routing/: Routing From 36d990eb72edd26a49ac7fd5d52aefaadde99513 Mon Sep 17 00:00:00 2001 From: thxCode Date: Thu, 8 Oct 2026 19:37:05 +0800 Subject: [PATCH 3/4] fix(model-deployment): route qualified external DP members Signed-off-by: thxCode --- .../controllers/worker/model_deployment.go | 9 +- .../model_deployment_balance_routing_test.go | 50 +++++ .../model_deployment_drain_collector.go | 5 +- .../model_deployment_external_routing_test.go | 184 ++++++++++++++++++ .../worker/model_deployment_parallelism.go | 45 ++++- .../model_deployment_parallelism_test.go | 24 ++- .../worker/model_deployment_router.go | 50 +++-- .../worker/model_deployment_service.go | 30 ++- .../webhooks/worker/model_deployment.go | 23 +++ .../worker/model_deployment_balance_test.go | 41 ++++ 10 files changed, 398 insertions(+), 63 deletions(-) create mode 100644 pkg/worker/controllers/worker/model_deployment_balance_routing_test.go create mode 100644 pkg/worker/controllers/worker/model_deployment_external_routing_test.go create mode 100644 pkg/worker/webhooks/worker/model_deployment_balance_test.go diff --git a/pkg/worker/controllers/worker/model_deployment.go b/pkg/worker/controllers/worker/model_deployment.go index e55ca03e8..eb9cfa285 100644 --- a/pkg/worker/controllers/worker/model_deployment.go +++ b/pkg/worker/controllers/worker/model_deployment.go @@ -1418,11 +1418,10 @@ func modelDeploymentRoleExternalDP( // an accepted error-path semantics change, deliberately kept for a reviewed change. return false } - shape, _ := modelDeploymentLoadBalance(reading.declared.Wiring) + shape, _ := modelDeploymentLoadBalance(reading.declared) - return shape == workercore.ModelDeploymentLoadBalanceExternal || - shape == workercore.ModelDeploymentLoadBalanceHybrid || - shape == workercore.ModelDeploymentLoadBalanceMultiPort + return shape != workercore.ModelDeploymentLoadBalanceInternal && + shape != workercore.ModelDeploymentLoadBalanceUnknown } // syncModelDeploymentService converges every Service the deployment owns: the one it is reached @@ -1469,7 +1468,7 @@ func (r *ModelDeploymentReconciler) syncModelDeploymentService( continue } if modelDeploymentEligibilitySelectorActive(md, role, live.Spec.Selector, eligibilityDecided) { - modelDeploymentSelectEligibleEndpoints(rendered[i]) + modelDeploymentSelectEligibleEndpoints(rendered[i], md, role) } } expected := make([]ctrlcli.Object, len(rendered)) diff --git a/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go b/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go new file mode 100644 index 000000000..867826fb1 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_balance_routing_test.go @@ -0,0 +1,50 @@ +package worker + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" +) + +func TestModelDeploymentBalanceRouting(t *testing.T) { + cases := []struct { + name string + args []string + kind workercore.ModelDeploymentRoleKind + balance workercore.ModelDeploymentLoadBalance + external bool + reason string + }{ + {name: "normalized hybrid internal", args: []string{"--data-parallel-hybrid-lb", "--data-parallel-size=4", "--data-parallel-size-local=4"}, balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "explicit single node internal", args: []string{"--data-parallel-size=4", "--data-parallel-start-rank=0", "--nnodes=1"}, balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "unreadable node count", args: []string{"--data-parallel-start-rank=0", "--nnodes=2:3"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"}, + {name: "missing node count", args: []string{"--data-parallel-start-rank=0", "--nnodes"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"}, + {name: "nonpositive node count", args: []string{"--data-parallel-start-rank=0", "--nnodes=0"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "nnodes must be a readable positive integer"}, + {name: "multiple nodes need local size", args: []string{"--data-parallel-start-rank=0", "--nnodes=2"}, balance: workercore.ModelDeploymentLoadBalanceUnknown, reason: "requires an explicit data-parallel-size-local"}, + {name: "internal", balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "rank", args: []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}, balance: workercore.ModelDeploymentLoadBalanceExternal, external: true}, + {name: "start rank alone", args: []string{"--data-parallel-start-rank=1"}, balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "implicit external", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1"}, balance: workercore.ModelDeploymentLoadBalanceExternal, external: true}, + {name: "implicit hybrid", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=2", "--data-parallel-start-rank=2"}, balance: workercore.ModelDeploymentLoadBalanceHybrid, external: true}, + {name: "all local internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=4", "--data-parallel-start-rank=0"}, balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "headless internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1", "--headless"}, balance: workercore.ModelDeploymentLoadBalanceInternal}, + {name: "hybrid", args: []string{"--data-parallel-hybrid-lb"}, balance: workercore.ModelDeploymentLoadBalanceHybrid, external: true}, + {name: "multi port", args: []string{"--data-parallel-multi-port-external-lb"}, balance: workercore.ModelDeploymentLoadBalanceMultiPort, external: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + md := newRenderDeployment() + role := &md.Spec.Roles[0] + role.ExtraArgs = tc.args + role.Kind = tc.kind + status := ReadModelDeploymentRoleParallelism(md.Spec.Engine.Name, role) + assert.Equal(t, tc.balance, status.LoadBalance) + assert.Equal(t, tc.external, modelDeploymentRoleExternalDP(md, role)) + if tc.reason != "" { + assert.Contains(t, status.Source.UnreadableReason, tc.reason) + } + }) + } +} diff --git a/pkg/worker/controllers/worker/model_deployment_drain_collector.go b/pkg/worker/controllers/worker/model_deployment_drain_collector.go index 1a1884044..953fddc00 100644 --- a/pkg/worker/controllers/worker/model_deployment_drain_collector.go +++ b/pkg/worker/controllers/worker/model_deployment_drain_collector.go @@ -893,7 +893,10 @@ func modelDeploymentDrainDisaggregated(engine string, container *corev1.Containe return fmt.Sprintf("the member's parallelism could not be read: %v", err), true } - shape, _ := modelDeploymentLoadBalance(reading.declared.Wiring) + shape, reason := modelDeploymentLoadBalance(reading.declared) + if shape == workercore.ModelDeploymentLoadBalanceUnknown { + return reason, true + } switch shape { case workercore.ModelDeploymentLoadBalanceExternal, workercore.ModelDeploymentLoadBalanceHybrid, diff --git a/pkg/worker/controllers/worker/model_deployment_external_routing_test.go b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go new file mode 100644 index 000000000..45ca77879 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go @@ -0,0 +1,184 @@ +package worker + +import ( + "context" + "fmt" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + ctrlrecord "k8s.io/client-go/tools/record" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + + gpustackcore "gpustack.ai/gpustack/api/v1" + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" +) + +func TestExternalDPRoutingSelectsQualifiedMembers(t *testing.T) { + for _, routerName := range []string{ + workercore.ModelDeploymentRouterLLMD, + workercore.ModelDeploymentRouterVLLM, + workercore.ModelDeploymentRouterSGLang, + } { + for _, decided := range []bool{false, true} { + t.Run(fmt.Sprintf("%s/decided=%t", routerName, decided), func(t *testing.T) { + md := newRenderDeployment() + md.Spec.KVCache = nil + md.Spec.Engine.Version = "0.29.0" + md.Spec.Router = &workercore.ModelDeploymentRouter{Name: routerName} + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].ExtraArgs = []string{"--data-parallel-size", "2", "--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"} + internal := *md.Spec.Roles[0].DeepCopy() + internal.Name = "internal" + internal.ExtraArgs = nil + normalized := *internal.DeepCopy() + normalized.Name = "normalized" + normalized.ExtraArgs = []string{"--data-parallel-hybrid-lb", "--data-parallel-size=4", "--data-parallel-size-local=4"} + md.Spec.Roles = append(md.Spec.Roles, internal, normalized) + if decided { + ModelDeploymentConditionEndpointEligibility.True(md, modelDeploymentReasonEndpointsQualified, "all groups qualified") + } + objects, err := renderModelDeploymentRouterObjects(context.Background(), md, nil, false) + require.NoError(t, err) + discovery := routingDiscoverySelector(t, objects.Deployment.Spec.Template.Spec.Containers[0].Args, objects.ConfigMap) + for i := range md.Spec.Roles { + role := &md.Spec.Roles[i] + published := labels.SelectorFromSet(objects.Contract.Roles[i].Selector) + service := labels.SelectorFromSet(renderModelDeploymentRoleService(md, role, nil).Spec.Selector) + for member := range 2 { + pod, renderErr := renderModelDeploymentPodTemplate(context.Background(), ModelDeploymentRenderInput{ + Deployment: md, Role: role, InstanceType: newRenderInstanceType(), + }) + require.NoError(t, renderErr) + stampModelDeploymentPod(pod, md, role, 0, member) + answers := i == 0 || member == 0 + if answers { + pod.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + } + want := member == 0 || (decided && i == 0) + for name, selector := range map[string]labels.Selector{"discovery": discovery, "published": published, "service": service} { + assert.Equal(t, want, selector.Matches(labels.Set(pod.Labels)), "%s/%s/member-%d", name, role.Name, member) + if decided { + withdrawn := pod.DeepCopy() + delete(withdrawn.Labels, modelDeploymentLabelKeyEndpointEligible) + assert.False(t, selector.Matches(labels.Set(withdrawn.Labels)), "%s must exclude withdrawn endpoints", name) + } + } + } + headless := renderModelDeploymentReplicaService(md, role, 0) + assert.Equal(t, core.ClusterIPNone, headless.Spec.ClusterIP) + assert.True(t, headless.Spec.PublishNotReadyAddresses) + assert.NotContains(t, headless.Spec.Selector, modelDeploymentLabelKeyEndpointEligible) + } + assert.False(t, discovery.Matches(labels.Set(objects.Deployment.Spec.Template.Labels)), "the router is not an engine endpoint") + }) + } + } +} + +func routingDiscoverySelector(t *testing.T, args []string, config *core.ConfigMap, flags ...string) labels.Selector { + t.Helper() + if config != nil { + selector, err := labels.Parse(config.Data[modelDeploymentRouterSelectorKey]) + require.NoError(t, err) + return selector + } + flag := "--selector" + if len(flags) > 0 { + flag = flags[0] + } + terms := map[string]string{} + for i, arg := range args { + if arg == flag { + for _, term := range args[i+1:] { + if len(term) > 1 && term[:2] == "--" { + break + } + parsed, err := labels.ConvertSelectorToLabelsMap(term) + require.NoError(t, err) + for key, value := range parsed { + terms[key] = value + } + } + break + } + } + require.NotEmpty(t, terms) + return labels.SelectorFromSet(terms) +} + +func TestPDRoutingKeepsReplicaLeaders(t *testing.T) { + for _, routerName := range []string{workercore.ModelDeploymentRouterLLMD, workercore.ModelDeploymentRouterVLLM, workercore.ModelDeploymentRouterSGLang} { + for _, sizes := range [][2]int{{1, 1}, {2, 1}, {1, 2}, {2, 2}} { + t.Run(fmt.Sprintf("%s/%d-%d", routerName, sizes[0], sizes[1]), func(t *testing.T) { + md := twoRoleDeployment() + md.Spec.Roles[0].Kind = workercore.ModelDeploymentRoleKindPrefill + md.Spec.Roles[1].Kind = workercore.ModelDeploymentRoleKindDecode + md.Spec.Router = &workercore.ModelDeploymentRouter{Name: routerName} + if routerName == workercore.ModelDeploymentRouterSGLang { + md.Spec.Engine.Name = workercore.ModelDeploymentEngineSGLang + } + if routerName != workercore.ModelDeploymentRouterLLMD { + md.Spec.Router.ExtraArgs = []string{"--policy", "round_robin"} + } + for i := range md.Spec.Roles { + md.Spec.Roles[i].ReplicaSize = int32(sizes[i]) + md.Spec.Roles[i].Replicas = 2 + } + ModelDeploymentConditionEndpointEligibility.True(md, modelDeploymentReasonEndpointsQualified, "qualified") + objects, err := renderModelDeploymentRouterObjects(context.Background(), md, nil, true) + require.NoError(t, err) + + for i := range md.Spec.Roles { + role := &md.Spec.Roles[i] + discovery := routingDiscoverySelector(t, objects.Deployment.Spec.Template.Spec.Containers[0].Args, objects.ConfigMap, "--"+role.Name+"-selector") + service := renderModelDeploymentRoleService(md, role, nil) + selectors := []labels.Selector{labels.SelectorFromSet(service.Spec.Selector), labels.SelectorFromSet(objects.Contract.Roles[i].Selector)} + if i == 0 { + selectors = append(selectors, labels.SelectorFromSet(renderModelDeploymentService(md).Spec.Selector)) + } + for ordinal := range 2 { + for member := range sizes[i] { + pod, err := renderModelDeploymentPodTemplate(context.Background(), ModelDeploymentRenderInput{Deployment: md, Role: role, InstanceType: newRenderInstanceType()}) + require.NoError(t, err) + stampModelDeploymentPod(pod, md, role, ordinal, member) + assert.Equal(t, member == 0, discovery.Matches(labels.Set(pod.Labels))) + for _, selector := range selectors { + assert.Equal(t, member == 0, selector.Matches(labels.Set(pod.Labels))) + } + other := labels.SelectorFromSet(objects.Contract.Roles[1-i].Selector) + assert.False(t, other.Matches(labels.Set(pod.Labels))) + peer := renderModelDeploymentReplicaService(md, role, ordinal) + assert.True(t, peer.Spec.PublishNotReadyAddresses) + assert.True(t, labels.SelectorFromSet(peer.Spec.Selector).Matches(labels.Set(pod.Labels))) + } + } + } + }) + } + } +} + +func TestExternalDPServiceEligibilityRetention(t *testing.T) { + md := newRenderDeployment() + md.Spec.Engine.Version = "0.29.0" + md.Spec.KVCache = nil + role := &md.Spec.Roles[0] + role.ReplicaSize = 2 + role.ExtraArgs = []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"} + cli := newModelDeploymentClient(md, newRenderInstanceType()) + reconciler := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(16)} + for _, status := range []meta.ConditionStatus{meta.ConditionTrue, meta.ConditionFalse, meta.ConditionUnknown} { + md.Status.Conditions = []gpustackcore.Condition{{Type: string(ModelDeploymentConditionEndpointEligibility), Status: status}} + require.NoError(t, reconciler.syncModelDeploymentService(context.Background(), md, false)) + for _, name := range []string{md.Name, md.Name + "-" + role.Name} { + service := new(core.Service) + require.NoError(t, cli.Get(context.Background(), ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, service)) + assert.NotContains(t, service.Spec.Selector, modelDeploymentMemberIndexLabel) + assert.Equal(t, modelDeploymentEndpointEligibleValue, service.Spec.Selector[modelDeploymentLabelKeyEndpointEligible]) + } + } +} diff --git a/pkg/worker/controllers/worker/model_deployment_parallelism.go b/pkg/worker/controllers/worker/model_deployment_parallelism.go index 401c35ccd..36b7bc442 100644 --- a/pkg/worker/controllers/worker/model_deployment_parallelism.go +++ b/pkg/worker/controllers/worker/model_deployment_parallelism.go @@ -18,6 +18,8 @@ import ( // engine's own default of one -- the operator composes no degree of its own, because the // transfer document must agree with the engine, and only the author's numbers can be right. type ModelDeploymentDeclaredParallelism struct { + // NodeCount retains an explicit --nnodes value; zero means absent or unreadable. + NodeCount int TensorParallel int PipelineParallel int DataParallel int @@ -112,6 +114,7 @@ var modelDeploymentParallelFlags = map[string][]modelDeploymentParallelFlag{ {names: []string{"--data-parallel-address", "-dpa"}, wiring: true, takesValue: true}, {names: []string{"--data-parallel-rpc-port", "-dpp"}, wiring: true, takesValue: true}, {names: []string{"--data-parallel-backend", "-dpb"}, wiring: true, takesValue: true}, + {names: []string{"--headless"}, wiring: true}, {names: []string{"--data-parallel-hybrid-lb", "-dph"}, wiring: true}, {names: []string{"--data-parallel-external-lb", "-dpe"}, wiring: true}, {names: []string{"--data-parallel-multi-port-external-lb", "-dpm"}, wiring: true}, @@ -242,6 +245,10 @@ func scanModelDeploymentParallelism( if flag.takesValue && !hasValue && i+1 < len(extraArgs) && modelDeploymentConsumableValue(extraArgs[i+1]) { i++ + value = extraArgs[i] + } + if canonical == "--nnodes" { + declared.NodeCount, _ = modelDeploymentEngineInt(value) } case flag.degree: if !hasValue { @@ -439,7 +446,7 @@ func ReadModelDeploymentRoleParallelism( return modelDeploymentParallelismUnreadable(overflow) } - loadBalance, balanceReason := modelDeploymentLoadBalance(reading.declared.Wiring) + loadBalance, balanceReason := modelDeploymentLoadBalance(reading.declared) return workercore.ModelDeploymentRoleParallelismStatus{ Declared: declared, @@ -472,7 +479,6 @@ func modelDeploymentParallelismUnreadable(reason string) workercore.ModelDeploym var modelDeploymentBalanceShapes = map[string]workercore.ModelDeploymentLoadBalance{ "--data-parallel-external-lb": workercore.ModelDeploymentLoadBalanceExternal, "--data-parallel-rank": workercore.ModelDeploymentLoadBalanceExternal, - "--data-parallel-start-rank": workercore.ModelDeploymentLoadBalanceExternal, "--data-parallel-hybrid-lb": workercore.ModelDeploymentLoadBalanceHybrid, "--data-parallel-multi-port-external-lb": workercore.ModelDeploymentLoadBalanceMultiPort, } @@ -481,19 +487,52 @@ var modelDeploymentBalanceShapes = map[string]workercore.ModelDeploymentLoadBala // voting one shape agree (an assigned rank alongside an external balancer is still External); // flags voting different shapes are not derivable and name their disagreement. func modelDeploymentLoadBalance( - wiring []string, + declared ModelDeploymentDeclaredParallelism, ) (workercore.ModelDeploymentLoadBalance, string) { + wiring := declared.Wiring shapes := map[workercore.ModelDeploymentLoadBalance][]string{} for _, flag := range wiring { if shape, ok := modelDeploymentBalanceShapes[flag]; ok { shapes[shape] = append(shapes[shape], flag) } } + startRank := slices.Contains(wiring, "--data-parallel-start-rank") + if startRank && len(shapes[workercore.ModelDeploymentLoadBalanceExternal]) > 0 { + shapes[workercore.ModelDeploymentLoadBalanceHybrid] = append( + shapes[workercore.ModelDeploymentLoadBalanceHybrid], "--data-parallel-start-rank") + } if len(shapes) == 0 { + // Headless ranks belong to an Internal group. A start rank only selects + // external routing when the local group is smaller than the global group. + if startRank && !slices.Contains(wiring, "--headless") { + local := declared.DataParallelLocal + if local > 0 && local < declared.DataParallel { + if local == 1 { + return workercore.ModelDeploymentLoadBalanceExternal, "" + } + return workercore.ModelDeploymentLoadBalanceHybrid, "" + } + if local == 0 && slices.Contains(wiring, "--nnodes") && declared.NodeCount != 1 { + if declared.NodeCount <= 0 { + return workercore.ModelDeploymentLoadBalanceUnknown, + "nnodes must be a readable positive integer to derive start-rank routing" + } + return workercore.ModelDeploymentLoadBalanceUnknown, + "start-rank routing requires an explicit data-parallel-size-local when nnodes exceeds 1" + } + } return workercore.ModelDeploymentLoadBalanceInternal, "" } if len(shapes) == 1 { for shape := range shapes { + if shape == workercore.ModelDeploymentLoadBalanceHybrid { + if declared.DataParallelLocal == 1 { + return workercore.ModelDeploymentLoadBalanceExternal, "" + } + if declared.DataParallelLocal == declared.DataParallel { + return workercore.ModelDeploymentLoadBalanceInternal, "" + } + } return shape, "" } } diff --git a/pkg/worker/controllers/worker/model_deployment_parallelism_test.go b/pkg/worker/controllers/worker/model_deployment_parallelism_test.go index 06d73a24c..9dc8a9576 100644 --- a/pkg/worker/controllers/worker/model_deployment_parallelism_test.go +++ b/pkg/worker/controllers/worker/model_deployment_parallelism_test.go @@ -157,7 +157,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) { name: "vllm: single-letter alias concatenated value records", engine: vllm, extraArgs: []string{"-n2"}, - set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"} }, + set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"}; p.NodeCount = 2 }, }, { name: "vllm: node-rank concatenated value records", @@ -169,7 +169,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) { name: "vllm: single-letter alias with = value records", engine: vllm, extraArgs: []string{"-n=2"}, - set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"} }, + set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes"}; p.NodeCount = 2 }, }, { name: "vllm: underscore unique prefix resolves", @@ -268,6 +268,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) { extraArgs: []string{"--nnodes", "2", "--tensor-parallel-size", "2"}, set: func(p *ModelDeploymentDeclaredParallelism) { p.TensorParallel = 2 + p.NodeCount = 2 p.Wiring = []string{"--nnodes"} }, }, @@ -275,7 +276,10 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) { name: "vllm: wiring short aliases record", engine: vllm, extraArgs: []string{"-n", "2", "-r", "0"}, - set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes", "--node-rank"} }, + set: func(p *ModelDeploymentDeclaredParallelism) { + p.Wiring = []string{"--nnodes", "--node-rank"} + p.NodeCount = 2 + }, }, { name: "vllm: boolean wiring consumes nothing", @@ -643,6 +647,7 @@ func TestParseModelDeploymentDeclaredParallelism(t *testing.T) { extraArgs: []string{"--nnodes", "2", "--dist-init-addr", "a:1", "--nccl-init-addr", "b:2"}, set: func(p *ModelDeploymentDeclaredParallelism) { p.Wiring = []string{"--nnodes", "--dist-init-addr"} + p.NodeCount = 2 }, }, { @@ -999,11 +1004,11 @@ func TestReadModelDeploymentRoleParallelism(t *testing.T) { }, }, { - name: "an assigned start rank alone derives External", + name: "an assigned start rank alone does not select external routing", engine: vllm, role: workercore.ModelDeploymentRole{ExtraArgs: []string{"--data-parallel-start-rank=1"}}, want: workercore.ModelDeploymentRoleParallelismStatus{ - LoadBalance: workercore.ModelDeploymentLoadBalanceExternal, + LoadBalance: workercore.ModelDeploymentLoadBalanceInternal, Source: workercore.ModelDeploymentParallelismSourceStatus{ Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs, Complete: true, @@ -1011,16 +1016,17 @@ func TestReadModelDeploymentRoleParallelism(t *testing.T) { }, }, { - name: "start rank and external balancer agree on External", + name: "start rank and external balancer conflict", engine: vllm, role: workercore.ModelDeploymentRole{ ExtraArgs: []string{"--data-parallel-start-rank=1", "--data-parallel-external-lb"}, }, want: workercore.ModelDeploymentRoleParallelismStatus{ - LoadBalance: workercore.ModelDeploymentLoadBalanceExternal, + LoadBalance: workercore.ModelDeploymentLoadBalanceUnknown, Source: workercore.ModelDeploymentParallelismSourceStatus{ - Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs, - Complete: true, + Kind: workercore.ModelDeploymentParallelismSourceKindExtraArgs, + Complete: true, + UnreadableReason: "the balance flags --data-parallel-external-lb and --data-parallel-start-rank derive to different shapes", }, }, }, diff --git a/pkg/worker/controllers/worker/model_deployment_router.go b/pkg/worker/controllers/worker/model_deployment_router.go index 0e3c4214e..e21f51b27 100644 --- a/pkg/worker/controllers/worker/model_deployment_router.go +++ b/pkg/worker/controllers/worker/model_deployment_router.go @@ -124,22 +124,17 @@ func renderModelDeploymentRouterObjects( for i := range md.Spec.Roles { role := &md.Spec.Roles[i] kind := ModelDeploymentEffectiveRoleKind(role) - // THE PUBLISHED SELECTOR NAMES THE PODS A REQUEST MAY REACH, which is one member per - // replica -- the leader -- rather than every Pod the role runs. The field is published - // VERBATIM for a router to be configured from, so a selector that also named the members - // serving no API would hand that router the very defect the discovery selector below - // exists to remove. The leader term is what keeps the two selectors in one semantics; they - // differ in scope only, this one naming a role and that one the deployment. + // Keep legacy leader-only routing until the eligibility gate is active. + // Once qualified, External-DP members all serve HTTP and must remain selectable. selector := modelDeploymentSelectorLabels(md, role) selector[modelDeploymentLabelKeyRoleKind] = roles[i].Kind selector[modelDeploymentMemberIndexLabel] = strconvx.Itoa(modelDeploymentLeaderMemberIndex) - // The eligibility term rides on the published selector beside the leader term, so a router - // configured from it can only ever reach endpoints the reconciler has marked eligible. It - // narrows the same set the terms above already name; it never widens one. A shape whose - // group-forward capability can never activate renders WITHOUT the term: a selector no pass - // would ever satisfy would hand the router zero endpoints and call it discovery. + // Unsupported shapes never gain an eligibility selector that no pass can satisfy. if modelDeploymentEligibilitySelectorActive(md, role, nil, retainedEligibility) { selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + if modelDeploymentRoleExternalDP(md, role) { + delete(selector, modelDeploymentMemberIndexLabel) + } } // The renderer is handed the SAME map that is published, rather than one built beside it, // because a router configured by argv discovers each role by these labels: two derivations @@ -156,19 +151,9 @@ func renderModelDeploymentRouterObjects( } contract.Roles = append(contract.Roles, roleStatus) } - // DISCOVERY NAMES ONLY THE PODS THAT ANSWER THE API: one member per replica, the leader, which - // at size one is the replica's only Pod. Without the index term the selector would name every - // member, a router would spread requests across the ones serving nothing, and the failures - // would interleave with successes rather than the router being visibly broken. The term is a - // plain equality rather than something sized per role, because a label selector is a - // conjunction and the pickers filter by kind: two server roles of different sizes share this - // one selector, and only a member index written on every member lets a single term hold for - // both. - // - // THE EQUALITIES ARE THE SOURCE AND THE EXPRESSION IS DERIVED FROM THEM, because the two - // routers configured by argv match on equality alone and cannot be handed an expression: one of - // them splits each entry on its first "=" and drops an entry carrying none without a word. A - // second literal here would let the string and the map name different sets. + // Start with legacy leader-only discovery. Qualified External-DP roles use the + // eligibility label instead, which also excludes internal followers in mixed roles. + // Derive every selector form from these equalities so all routers discover the same set. endpointLabels := map[string]string{ modelDeploymentLabelKeyName: modelDeploymentLabelValueName, modelDeploymentLabelKeyInstance: md.Name, @@ -191,10 +176,17 @@ func renderModelDeploymentRouterObjects( } if endpointActivatable && (modelDeploymentEligibilityDecided(md) || retainedEligibility) { endpointLabels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + for i := range md.Spec.Roles { + if modelDeploymentRoleExternalDP(md, &md.Spec.Roles[i]) { + // Eligibility selects every answering External-DP member and only the + // leaders of internal roles, including deployments with both shapes. + delete(endpointLabels, modelDeploymentMemberIndexLabel) + break + } + } } - // The router's own Pod carries the first two labels and NOT the member index, so the equalities - // already exclude it. The negation is kept in the expression, where it costs nothing and says - // out loud what the index term achieves by arithmetic. + // The router Pod has neither a member index nor endpoint eligibility. + // Keep the explicit exclusion in the expression as well. // // The terms are listed rather than sorted out of the map because their ORDER IS RENDERED: this // string is a ConfigMap value and the Pod's configuration hash is taken over it, so reordering @@ -202,7 +194,9 @@ func renderModelDeploymentRouterObjects( endpointSelectorTerms := []string{ modelDeploymentLabelKeyName + "=" + endpointLabels[modelDeploymentLabelKeyName], modelDeploymentLabelKeyInstance + "=" + endpointLabels[modelDeploymentLabelKeyInstance], - modelDeploymentMemberIndexLabel + "=" + endpointLabels[modelDeploymentMemberIndexLabel], + } + if member, present := endpointLabels[modelDeploymentMemberIndexLabel]; present { + endpointSelectorTerms = append(endpointSelectorTerms, modelDeploymentMemberIndexLabel+"="+member) } if len(endpointLabels[modelDeploymentLabelKeyEndpointEligible]) > 0 { endpointSelectorTerms = append(endpointSelectorTerms, diff --git a/pkg/worker/controllers/worker/model_deployment_service.go b/pkg/worker/controllers/worker/model_deployment_service.go index f419f58f3..52a2c339b 100644 --- a/pkg/worker/controllers/worker/model_deployment_service.go +++ b/pkg/worker/controllers/worker/model_deployment_service.go @@ -102,7 +102,7 @@ func renderModelDeploymentService(md *workercore.ModelDeployment) *core.Service svc := renderModelDeploymentServiceFor(md, &md.Spec.Roles[0], md.Name) modelDeploymentFrontLeadersOnly(svc, &md.Spec.Roles[0]) if modelDeploymentEligibilitySelectorActive(md, &md.Spec.Roles[0], nil, modelDeploymentEligibilityDecided(md)) { - modelDeploymentSelectEligibleEndpoints(svc) + modelDeploymentSelectEligibleEndpoints(svc, md, &md.Spec.Roles[0]) } return svc @@ -153,20 +153,6 @@ func modelDeploymentGroupForwardActivatable( kind != workercore.ModelDeploymentRoleKindDecode } -// modelDeploymentSelectEligibleEndpoints narrows an ordinary Service to the endpoints the -// reconciler has marked eligible, by the same equality every selector that names endpoints -// carries. The term rides ON TOP of the identity selector and the leader narrowing -- it gates -// which of the members a Service already fronts are answerable, never which members exist. -// -// THE TERM IS A LABEL EQUALITY BECAUSE ONLY A LABEL CAN SATISFY A SERVICE SELECTOR. The key is -// written by the reconciler's runtime convergence, outside every rendered hash, so gaining or -// losing it moves no replica; what it moves is the Service's endpoint set, which is exactly the -// lever a disqualification pulls. -// -// A REPLICA'S HEADLESS SERVICE NEVER TAKES THE TERM. That Service exists to publish every member -// to its peers with unready addresses included; eligibility is about who may ANSWER, and the -// peers' collective needs every member's record regardless of who answers. - // modelDeploymentEligibilityDecided reports whether a pass has actually answered the whole-group // predicate: the EndpointEligibility condition is True (qualified -- narrow and let the labels // admit) or False (revoked -- narrow and let the labels withdraw). A decided condition is the @@ -216,8 +202,18 @@ func modelDeploymentEligibilitySelectorActive( liveSelector[modelDeploymentLabelKeyEndpointEligible] == modelDeploymentEndpointEligibleValue } -func modelDeploymentSelectEligibleEndpoints(svc *core.Service) { +// modelDeploymentSelectEligibleEndpoints selects qualified HTTP endpoints. +// External roles serve every rank; Internal roles retain their leader restriction. +// Headless peer Services never use endpoint eligibility. +func modelDeploymentSelectEligibleEndpoints( + svc *core.Service, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, +) { svc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + // Every qualified External-DP member serves HTTP. The eligibility label already + // excludes unhealthy groups, so the legacy leader restriction must not hide ranks. + if modelDeploymentRoleExternalDP(md, role) { + delete(svc.Spec.Selector, modelDeploymentMemberIndexLabel) + } } // renderModelDeploymentRoleService renders the Service that fronts ONE role. @@ -248,7 +244,7 @@ func renderModelDeploymentRoleService( svc := renderModelDeploymentServiceFor(md, role, md.Name+"-"+role.Name) modelDeploymentFrontLeadersOnly(svc, role) if modelDeploymentEligibilitySelectorActive(md, role, nil, modelDeploymentEligibilityDecided(md)) { - modelDeploymentSelectEligibleEndpoints(svc) + modelDeploymentSelectEligibleEndpoints(svc, md, role) } if modelDeploymentPublishesKVEvents(md, role, manufacturers[role.Name]) { for _, port := range inject.KVEventsPorts() { diff --git a/pkg/worker/webhooks/worker/model_deployment.go b/pkg/worker/webhooks/worker/model_deployment.go index 75cf29c08..079398e52 100644 --- a/pkg/worker/webhooks/worker/model_deployment.go +++ b/pkg/worker/webhooks/worker/model_deployment.go @@ -1645,6 +1645,7 @@ func validateModelDeploymentRoles(md *workercore.ModelDeployment) field.ErrorLis errs = append(errs, validateModelDeploymentRoleExtraArgs(md.Spec.Engine.Name, role, rolePath)...) errs = append(errs, validateModelDeploymentRoleParallelWidth(md.Spec.Engine.Name, role, rolePath)...) + errs = append(errs, validateModelDeploymentRoleLoadBalance(md.Spec.Engine.Name, role, rolePath)...) errs = append(errs, validateModelDeploymentRoleEnv(md.Spec.Engine.Name, role, rolePath)...) errs = append(errs, validateModelDeploymentRoleResources(role, rolePath)...) errs = append(errs, validateModelDeploymentRoleAdditionalVolumes(role, rolePath)...) @@ -1943,6 +1944,28 @@ func validateModelDeploymentRoleExtraArgs( return errs } +// validateModelDeploymentRoleLoadBalance keeps managed roles on one HTTP endpoint per Pod. +func validateModelDeploymentRoleLoadBalance( + engine string, role *workercore.ModelDeploymentRole, rolePath *field.Path, +) field.ErrorList { + if len(role.Command) > 0 { + return nil + } + status := workerctrl.ReadModelDeploymentRoleParallelism(engine, role) + if status.Source.Kind != workercore.ModelDeploymentParallelismSourceKindExtraArgs { + return nil // The degree parser reports unreadable arguments separately. + } + if status.LoadBalance == workercore.ModelDeploymentLoadBalanceUnknown { + return field.ErrorList{field.Invalid(rolePath.Child("extraArgs"), role.ExtraArgs, + status.Source.UnreadableReason)} + } + if status.LoadBalance == workercore.ModelDeploymentLoadBalanceMultiPort { + return field.ErrorList{field.Invalid(rolePath.Child("extraArgs"), role.ExtraArgs, + "MultiPort load balancing is unsupported; managed roles require one HTTP endpoint per Pod")} + } + return nil +} + // validateModelDeploymentRoleParallelWidth is the first check that ever ties a role's declared // parallelism to its request: the per-member engine width must fit the per-member card count. // diff --git a/pkg/worker/webhooks/worker/model_deployment_balance_test.go b/pkg/worker/webhooks/worker/model_deployment_balance_test.go new file mode 100644 index 000000000..79b9524ce --- /dev/null +++ b/pkg/worker/webhooks/worker/model_deployment_balance_test.go @@ -0,0 +1,41 @@ +package worker + +import ( + "testing" + + "github.com/stretchr/testify/assert" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" +) + +func TestModelDeploymentManagedBalance(t *testing.T) { + cases := []struct { + name string + args, command []string + kind workercore.ModelDeploymentRoleKind + want string + }{ + {name: "internal"}, + {name: "single node internal", args: []string{"--data-parallel-size=4", "--data-parallel-start-rank=0", "--nnodes=1"}}, + {name: "external rank", args: []string{"--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"}}, + {name: "multi port", args: []string{"-dpm"}, want: "MultiPort"}, + {name: "conflicting modes", args: []string{"-dpe", "-dph"}, want: "different shapes"}, + {name: "headless internal", args: []string{"--data-parallel-size=4", "--data-parallel-size-local=1", "--data-parallel-start-rank=1", "--headless"}}, + {name: "custom command", command: []string{"custom-engine"}, args: []string{"-dpm"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + md := modelDeployment(workercore.ModelDeploymentEngineVLLM) + role := &md.Spec.Roles[0] + role.ExtraArgs = tc.args + role.Command = tc.command + role.Kind = tc.kind + errs := validateModelDeploymentRoles(md) + if tc.want == "" { + assert.Empty(t, errs) + } else if assert.NotEmpty(t, errs) { + assert.Contains(t, errs.ToAggregate().Error(), tc.want) + } + }) + } +} From a76328adb077e699b7be1dd6aa31027ebc3fe22c Mon Sep 17 00:00:00 2001 From: thxCode Date: Thu, 8 Oct 2026 20:20:05 +0800 Subject: [PATCH 4/4] docs: group walkthroughs by module and preserve runtime evidence Signed-off-by: thxCode --- .../skills/gpustack-operator-docs/SKILL.md | 9 +- .../references/conventions.md | 6 +- .../references/page-map.md | 20 +- .../scripts/check-docs.sh | 2 +- README.md | 4 +- docs/README.md | 28 +- docs/getting-started/architecture.md | 6 +- docs/modules/devices/admission.md | 2 +- docs/modules/devices/nvidia-mig.md | 4 +- docs/modules/devices/requests.md | 4 +- docs/modules/devices/scheduling.md | 2 +- docs/modules/kv-cache/_index.md | 4 +- docs/modules/model-delivery/_index.md | 2 + docs/modules/model-deployment/_index.md | 4 +- docs/modules/model-deployment/elastic-ep.md | 6 +- docs/modules/model-deployment/routing.md | 30 +- .../reference/instance-type-unit-resources.md | 2 +- docs/reference/settings.md | 2 +- docs/walkthroughs/_index.md | 34 ++ .../devices/scheduling.md} | 28 +- .../kv-cache/shared-cache.md} | 34 +- docs/walkthroughs/model-delivery/prefetch.md | 164 +++++++ docs/walkthroughs/model-deployment/_index.md | 94 ++++ .../model-deployment/elastic-ep.md} | 420 ++++-------------- .../model-deployment/external-dp.md} | 168 ++++--- site/content/docs/_index.md | 2 +- site/data/navigation.yaml | 34 +- site/layouts/index.html | 2 +- ...10-01-s1-router-qualification-and-drain.md | 7 + 29 files changed, 653 insertions(+), 471 deletions(-) create mode 100644 docs/walkthroughs/_index.md rename docs/{getting-started/walkthrough.md => walkthroughs/devices/scheduling.md} (95%) rename docs/{modules/kv-cache/walkthrough.md => walkthroughs/kv-cache/shared-cache.md} (85%) create mode 100644 docs/walkthroughs/model-delivery/prefetch.md create mode 100644 docs/walkthroughs/model-deployment/_index.md rename docs/{modules/model-deployment/scaling-walkthrough.md => walkthroughs/model-deployment/elastic-ep.md} (58%) rename docs/{modules/model-deployment/external-dp-walkthrough.md => walkthroughs/model-deployment/external-dp.md} (62%) diff --git a/.agents/skills/gpustack-operator-docs/SKILL.md b/.agents/skills/gpustack-operator-docs/SKILL.md index fea73f99b..cad6c3a83 100644 --- a/.agents/skills/gpustack-operator-docs/SKILL.md +++ b/.agents/skills/gpustack-operator-docs/SKILL.md @@ -61,7 +61,7 @@ one, not to widen the overview. | The local disk tier itself: what it renders, its bucket, its eviction, its host directory | `docs/modules/kv-cache/local-disk-tier.md` | | Configuring a node that is mostly disk: why a group is never disk alone, the thin-segment shape, the memory floor | `docs/modules/kv-cache/disk-heavy-nodes.md` | | A `KVCachePool` or `KVCachePoolBinding`: the grant, the reuse domain, a quota ceiling or grant, what a full quota does | `docs/modules/kv-cache/pool.md` | -| Standing a cache up end to end, or which object comes first: the pasteable four-object sequence | `docs/modules/kv-cache/walkthrough.md` | +| Standing a cache up end to end, or which object comes first: the pasteable four-object sequence | `docs/walkthroughs/kv-cache/shared-cache.md` | | How a **Pod** consumes a pool: the inject label and annotations, the injected keys per engine, a refusal, the isolation record | `docs/modules/kv-cache/injection.md` | | A `ModelArtifact`: its sources, resolution and revalidation, the manifest digest, how a `ModelDeployment` or an `Instance` mounts or downloads it, claim placement, the weight identity in KV keys | `docs/modules/model-delivery/artifact.md` | | The `image` source of a `ModelArtifact`: the digest contract, building weights into an image, image-volume delivery, the version floors, double storage, kubelet image GC, registry mirrors | `docs/modules/model-delivery/image-source.md` | @@ -70,8 +70,9 @@ one, not to widen the overview. | A `NodeModelStore` or the `model-manager` plugin: a field and its writer, the status guard, mount authorization, materialization, a failure reason, collection, a metric | `docs/modules/model-delivery/node-store.md` | | Running node delivery: the chart values, where the node's configuration comes from, reading a node, the watermark cap, switching delivery, where replicas land and turning the preference off, upgrading, removing the cache | `docs/modules/model-delivery/operations.md` | | The `ModelDeployment` contract: the inherited reuse domain, the three override tiers, the owned-key table, the runner-image formula, prefill/decode pairing, the topology-placement field contract | `docs/modules/model-deployment/deployment.md` | -| A recorded model-serving sequence from prefetch through elastic expansion, its captured CR output, or the distinction from independent replica scaling | `docs/modules/model-deployment/scaling-walkthrough.md` | -| A fixed External DP group, its minimum arguments, per-rank requests, or measured Router coverage | `docs/modules/model-deployment/external-dp-walkthrough.md` | +| A recorded Elastic EP expansion, its captured CR output and inference checks | `docs/walkthroughs/model-deployment/elastic-ep.md` | +| A recorded model prefetch sequence and captured artifact, prefetch and node-store output | `docs/walkthroughs/model-delivery/prefetch.md` | +| A fixed External DP group, its minimum arguments, per-rank requests, or measured Router coverage | `docs/walkthroughs/model-deployment/external-dp.md` | | A `ModelDeployment` metrics snapshot, which series each field reads per engine, role and router, cache-hit scope or Pod scrape annotation | `docs/modules/model-deployment/metrics.md` | | What a `ModelDeployment` status condition or published field means, and how to read them when a deployment misbehaves | `docs/modules/model-deployment/status.md` | | How a prefill role and a decode role are paired: the connector each engine and router renders, `spec.router` and its fields, `spec.kvTransfer`, roles on different hardware, a role's own Service | `docs/modules/model-deployment/prefill-decode.md` | @@ -91,7 +92,7 @@ one, not to widen the overview. | A subcommand, one of its flags, its exit codes, its invocation | `docs/reference/commands.md` | | A user-visible capability, a vendor's slicing support, the install flow | `README.md` | | Vendor prerequisites, vendor GPU Operator coexistence | `docs/getting-started/vendor-prerequisites.md` | -| A recorded run with real output | `docs/getting-started/walkthrough.md`, or the walkthrough section of `docs/modules/devices/nvidia-mig.md` | +| A recorded run with real output | `docs/walkthroughs/devices/scheduling.md`, or the walkthrough section of `docs/modules/devices/nvidia-mig.md` | A decision *record* — why an approach was chosen over another — belongs in `specs/`, not in `docs/`. The docs state the rule that resulted; a `> **Why**` note carries only as much rationale as a reader diff --git a/.agents/skills/gpustack-operator-docs/references/conventions.md b/.agents/skills/gpustack-operator-docs/references/conventions.md index 0f7709c66..7421c2ff1 100644 --- a/.agents/skills/gpustack-operator-docs/references/conventions.md +++ b/.agents/skills/gpustack-operator-docs/references/conventions.md @@ -21,7 +21,7 @@ resources. [Architecture](../../getting-started/architecture.md) introduces the --- -**See also** — [Device Discovery](discovery.md) · [Walkthrough](../../getting-started/walkthrough.md) +**See also** — [Device Discovery](discovery.md) · [Walkthrough](../../walkthroughs/devices/scheduling.md) **Next** → [Admission](admission.md) — the admission gates a request passes. ``` @@ -81,7 +81,7 @@ something that belongs in a list, a table, or another page. | Cap | Limit | Exempt | |---|---|---| | prose paragraph | 5 lines / 500 rendered characters | fenced blocks, table rows, list *markers*, `> **Why**` notes, the footer | -| page length | 1000 lines | `docs/getting-started/walkthrough.md`, `docs/operate/*`, `docs/modules/devices/*-mig.md` — recordings and runbooks | +| page length | 1000 lines | `docs/walkthroughs/devices/scheduling.md`, `docs/operate/*`, `docs/modules/devices/*-mig.md` — recordings and runbooks | | `##` sections | 10 per page, `## Contents` not counted | `docs/README.md`, `docs/reference/*` — both are lookup tables | A paragraph nested under a list item is prose a reader still has to get through, so it is measured on @@ -106,7 +106,7 @@ it as follows — when a page starts serving two modes at once, that is the mome | Mode | Reader is… | Our pages | |---|---|---| -| Tutorial | learning by doing | `README.md` Usage examples, `docs/getting-started/walkthrough.md`, the MIG walkthrough, `docs/modules/kv-cache/walkthrough.md` | +| Tutorial | learning by doing | `README.md` Usage examples, `docs/walkthroughs/devices/scheduling.md`, the MIG walkthrough, `docs/walkthroughs/kv-cache/shared-cache.md` | | How-to | achieving a goal | `docs/operate/*`, `docs/modules/devices/*-mig.md`, `docs/operate/migration/*`, `docs/contribute/development.md`, a domain runbook (`docs/modules/model-delivery/operations.md`) | | Reference | looking something up | `docs/modules/devices/requests.md`, `docs/reference/settings.md`, `docs/reference/*` | | Explanation | building understanding | `docs/getting-started/architecture.md` and the deep pages under `docs/modules/` (contract pages read as reference, mechanism pages as explanation) | diff --git a/.agents/skills/gpustack-operator-docs/references/page-map.md b/.agents/skills/gpustack-operator-docs/references/page-map.md index 003cbc211..6320a04bc 100644 --- a/.agents/skills/gpustack-operator-docs/references/page-map.md +++ b/.agents/skills/gpustack-operator-docs/references/page-map.md @@ -176,7 +176,7 @@ backend and links to it. And never the *enforcement* the Binding does not do: th it is a grant and not an isolation boundary belongs here, but the mechanism a real boundary would need is not this page's to design. -## `docs/modules/kv-cache/walkthrough.md` +## `docs/walkthroughs/kv-cache/shared-cache.md` **Owns** — the ORDER, and nothing else: the four objects a working shared cache takes, one pasteable manifest each, the check to run after each one, and the three configurations that read as working and @@ -332,7 +332,7 @@ contract and does not teach. **Never** — implementation. A rule's *enforcement point* is a link to `admission.md`. -## `docs/getting-started/walkthrough.md` +## `docs/walkthroughs/devices/scheduling.md` **Owns** — recorded runs with real `kubectl` output and before/after comparisons. @@ -429,19 +429,29 @@ supersession is recorded that way because it has no new spec to be recorded in. when that file leaves the tree the name resolves to nothing, and leaving it alone preserves a pointer rather than a record. -## `docs/modules/model-deployment/scaling-walkthrough.md` +## `docs/walkthroughs/model-deployment/elastic-ep.md` -**Owns** — the user sequence from model prefetch to elastic TP2/DP2→4 expansion, captured CR output, +**Owns** — the user sequence for elastic TP2/DP2→4 expansion, captured CR output, short inference checks, log inspection, and application release. Distinguishes independent replicas from an elastic collective and states which path was measured. **Never** — private engine code, raw infrastructure identities, or broad support claims inferred from one runtime configuration. The Elastic EP guide owns the profile contract; the resize spec owns design analysis. -## `docs/modules/model-deployment/external-dp-walkthrough.md` +## `docs/walkthroughs/model-deployment/external-dp.md` **Owns** — the fixed External DP serving sequence, minimum argument dependencies, captured CR output, per-rank inference checks, Router coverage limits, and application cleanup. Links shared model prefetch steps. **Never** — imply online collective resize from fixed-group inference, or claim all ranks receive Router traffic from HTTP success alone. P/D combinations remain with their owning guide and follow-up issue. + +## `docs/walkthroughs/model-delivery/prefetch.md` + +**Owns** — the shared model preparation example, its fixed model revision and captured artifact, +prefetch, and node-store output. Model-serving walkthroughs link here before creating workloads. + +## `docs/walkthroughs/_index.md` + +**Owns** — the walkthrough entry point and module choices. Each module keeps its examples under +its own directory; capability guides link to the relevant example. diff --git a/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh b/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh index b6b4a4f48..7380b707e 100755 --- a/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh +++ b/.agents/skills/gpustack-operator-docs/scripts/check-docs.sh @@ -78,7 +78,7 @@ H2_CAP=10 # "##" sections in one page # Cap exemptions, declared here rather than escaped inline. Each is a space-separated glob list. # A recording and a runbook are as long as the hardware makes them. -LINE_CAP_EXEMPT='docs/getting-started/walkthrough.md docs/operate/* docs/modules/devices/*-mig.md' +LINE_CAP_EXEMPT='docs/walkthroughs/devices/scheduling.md docs/operate/* docs/modules/devices/*-mig.md' # The index is a table of contents and a reference page is a lookup table: both are meant to be flat. H2_CAP_EXEMPT='docs/README.md docs/reference/*' diff --git a/README.md b/README.md index dd8aef8d5..48f05047f 100644 --- a/README.md +++ b/README.md @@ -191,7 +191,7 @@ spec: The `ModelDeployment` below names the binding, which grants this namespace access to the cache. Choose member nodes, quotas, block size and dtype for your engine before applying these resources. -The [KV Cache Walkthrough](./docs/modules/kv-cache/walkthrough.md) explains what to check at each step. +The [KV Cache Walkthrough](./docs/walkthroughs/kv-cache/shared-cache.md) explains what to check at each step. ### Model Delivery @@ -299,7 +299,7 @@ ssh -i ~/.ssh/id_ed25519 -p 2222 root@127.0.0.1 ``` The SSH session enters the workload container, where the accelerator and workspace are available. -The [walkthrough](./docs/getting-started/walkthrough.md) shows the resulting resources. Pods can +The [walkthrough](./docs/walkthroughs/devices/scheduling.md) shows the resulting resources. Pods can request accelerators directly; see [Accelerator Requests](./docs/modules/devices/requests.md) for complete manifests. diff --git a/docs/README.md b/docs/README.md index 259a174e6..e3aa4392e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -13,8 +13,8 @@ The capability guides are [Heterogeneous Devices](modules/devices/_index.md), For a first workload, follow [Usage](../README.md#usage), then use [Accelerator Requests](modules/devices/requests.md) to choose its resource keys. The -[Walkthrough](getting-started/walkthrough.md) records a run on a four-node cluster. Workloads that -need RDMA also need [RDMA Operations](modules/rdma/operations.md). +[Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) records a run on a four-node cluster. +Workloads that need RDMA also need [RDMA Operations](modules/rdma/operations.md). For cluster operations, read [Architecture](getting-started/architecture.md) and [Installation Modes](operate/installation-modes.md) first. The [settings](reference/settings.md), [vendor @@ -27,6 +27,8 @@ and [RDMA](modules/rdma/operations.md) runbooks as those features are enabled. H has separate runbooks for [NVIDIA](modules/devices/nvidia-mig.md), [T-Head](modules/devices/thead-mig.md) and [Hygon](modules/devices/hygon-mig.md). +For complete examples, use the [Walkthroughs](walkthroughs/_index.md) entry point. + Start an upgrade with the relevant page under [migration](operate/migration/to-subcharts.md). For code changes, read [Architecture](getting-started/architecture.md), then the relevant deep page in the @@ -83,7 +85,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `kv-cache` - Use for: standing up a shared inference cache; pool grants and quotas; attaching a workload to a pool. - Aliases: `KVCacheBackend`, `KVCachePool`, `Mooncake`, `prefix cache`, `injection` -- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](modules/kv-cache/walkthrough.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md). +- Guides: [KV Cache](modules/kv-cache/_index.md); setup order in [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md); store and grants in [KV Cache Backend](modules/kv-cache/backend.md) and [KV Cache Pool](modules/kv-cache/pool.md). - Specs: [backend](../specs/2026-08-28-kv-cache-backend.md) · [pool](../specs/2026-08-28-kv-cache-pool.md) · [injection](../specs/2026-08-28-kv-cache-injection.md) · [media and scaling](../specs/2026-09-05-kv-cache-media-and-scaling.md) · [high availability](../specs/2026-09-06-kv-cache-backend-high-availability.md) - Code: store rendering and read-back in [pkg/worker/kvcache](../pkg/worker/kvcache/) · per-engine client config in [pkg/worker/kvcache/inject](../pkg/worker/kvcache/inject/) · `kv_cache_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) - Related: [Model Deployment](#model-deployment) (deployments and Pods consume a pool through injection) @@ -94,7 +96,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `model-delivery` - Use for: giving model weights a stable identity; node caching and delivery; prefetch, peer sync, progress views and cache operations. - Aliases: `ModelArtifact`, `NodeModelStore`, `ModelStore`, `ModelPrefetch`, `model cache`, `Hugging Face` -- Guides: [Model Delivery](modules/model-delivery/_index.md); the artifact contract in [Model Artifact](modules/model-delivery/artifact.md); the node cache in [Node Model Store](modules/model-delivery/node-store.md); the runbook in [Model Store Operations](modules/model-delivery/operations.md). +- Guides: [Model Delivery](modules/model-delivery/_index.md); the artifact contract in [Model Artifact](modules/model-delivery/artifact.md); the node cache in [Node Model Store](modules/model-delivery/node-store.md); the runbook in [Model Store Operations](modules/model-delivery/operations.md); a measured example in [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md). - Specs: [model artifact](../specs/2026-09-25-model-artifact.md) · [node model store](../specs/2026-09-25-node-model-store.md) · [prefetch](../specs/2026-09-27-model-prefetch.md) · [peer sync](../specs/2026-09-27-model-peer-sync.md) · [image source](../specs/2026-09-27-model-image-source.md) - Code: Hub resolution in [pkg/modelartifact](../pkg/modelartifact/) · the per-node plugin and cache in [pkg/modelmanager](../pkg/modelmanager/) · cache configuration in [pkg/modelstore](../pkg/modelstore/) - Related: [Model Deployment](#model-deployment) (deployments mount or download artifacts) · [KV Cache](#kv-cache) (the weight identity enters KV keys) · [GPU Instances](#gpu-instances) (an Instance can mount weights) @@ -105,7 +107,7 @@ A skill link names a procedure; the entry's other fields carry shared facts. - ID: `model-deployment` - Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics. - Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica` -- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md); fixed per-rank HTTP serving in [External DP Walkthrough](modules/model-deployment/external-dp-walkthrough.md). +- Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md). - Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md) - Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/) - Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas) @@ -196,7 +198,6 @@ A skill link names a procedure; the entry's other fields carry shared facts. | [KV Cache Local Disk Tier](modules/kv-cache/local-disk-tier.md) | Configure local disk storage. | | [KV Cache on Disk-Heavy Nodes](modules/kv-cache/disk-heavy-nodes.md) | Size memory on disk-heavy nodes. | | [KV Cache Pool](modules/kv-cache/pool.md) | Grant and limit cache use. | -| [KV Cache Walkthrough](modules/kv-cache/walkthrough.md) | Create a working cache from backend to workload. | | [KV Cache Injection Reference](modules/kv-cache/injection.md) | Attach a Pod to a pool. | ### Model delivery @@ -220,8 +221,6 @@ A skill link names a procedure; the entry's other fields carry shared facts. | [Model Deployment Configuration](modules/model-deployment/deployment.md) | Configure serving roles and overrides. | | [Model Deployment Prefill and Decode](modules/model-deployment/prefill-decode.md) | Pair serving roles. | | [Elastic EP](modules/model-deployment/elastic-ep.md) | Resize one serving instance's collective in place. | -| [Model Scaling Walkthrough](modules/model-deployment/scaling-walkthrough.md) | Prefetch weights, expand an elastic collective, and read captured CR output. | -| [External DP Walkthrough](modules/model-deployment/external-dp-walkthrough.md) | Run fixed External DP ranks and verify Router coverage. | | [Engine Versions](modules/model-deployment/engine-versions.md) | Check supported engine versions. | | [Model Deployment Routing](modules/model-deployment/routing.md) | Choose a routing policy, confirm a Router serves. | | [Model Deployment Metrics](modules/model-deployment/metrics.md) | Read serving metrics. | @@ -235,12 +234,23 @@ A skill link names a procedure; the entry's other fields carry shared facts. | [Accelerated Instances](modules/instances/_index.md) | Start an accelerator-backed workspace. | | [Instance Metrics Reference](reference/instance-metrics.md) | Read an Instance’s resource use. | +### Walkthroughs + +| Page | Description | +|---|---| +| [Walkthroughs](walkthroughs/_index.md) | Choose a complete example by module. | +| [Device Scheduling Walkthrough](walkthroughs/devices/scheduling.md) | Follow a recorded accelerator scheduling run. | +| [KV Cache Walkthrough](walkthroughs/kv-cache/shared-cache.md) | Create a shared cache from backend to workload. | +| [Model Prefetch Walkthrough](walkthroughs/model-delivery/prefetch.md) | Cache weights and verify captured model CR output. | +| [Model Deployment Walkthroughs](walkthroughs/model-deployment/_index.md) | Choose a serving shape and prepare GPUs. | +| [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md) | Grow DP width with TP fixed and verify inference. | +| [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md) | Verify per-rank inference and Router coverage. | + ### Start here | Page | Description | |---|---| | [Architecture](getting-started/architecture.md) | See the operator’s four-stage path. | -| [Walkthrough](getting-started/walkthrough.md) | Follow a recorded cluster run. | | [Vendor Prerequisites](getting-started/vendor-prerequisites.md) | Prepare each manufacturer’s driver. | ### Cluster operations and upgrades diff --git a/docs/getting-started/architecture.md b/docs/getting-started/architecture.md index ee6116da9..727bd5a2d 100644 --- a/docs/getting-started/architecture.md +++ b/docs/getting-started/architecture.md @@ -106,7 +106,7 @@ allocation. [Admission](../modules/devices/admission.md#the-five-gates) owns the its failure behavior. InstanceType status reports the remaining capacity as workloads allocate and release devices. Use -`kubectl get instancetype -w` to watch those changes. The [Walkthrough](walkthrough.md) shows the +`kubectl get instancetype -w` to watch those changes. The [Walkthrough](../walkthroughs/devices/scheduling.md) shows the request and its resulting Kubernetes resources. ## Vocabulary @@ -126,7 +126,7 @@ The hardware terms Device, Accelerator and Resource are defined in ## Related documentation -- [Walkthrough](walkthrough.md) — submit a workload and inspect the scheduling chain. +- [Walkthrough](../walkthroughs/devices/scheduling.md) — submit a workload and inspect the scheduling chain. - [GPU Instances](../modules/instances/_index.md) — configure an interactive container and SSH access. - [Model Deployment](../modules/model-deployment/_index.md) — configure inference engines and replicas. - [Model Delivery](../modules/model-delivery/_index.md) — resolve, cache and mount model weights. @@ -138,4 +138,4 @@ The hardware terms Device, Accelerator and Resource are defined in **See also** — [Accelerator Requests](../modules/devices/requests.md) · [Settings](../reference/settings.md) · [All documentation](../README.md) -**Next** → [Walkthrough](walkthrough.md) — follow a workload from submission to allocation. +**Next** → [Walkthrough](../walkthroughs/devices/scheduling.md) — follow a workload from submission to allocation. diff --git a/docs/modules/devices/admission.md b/docs/modules/devices/admission.md index 91fd3ddca..49b8be593 100644 --- a/docs/modules/devices/admission.md +++ b/docs/modules/devices/admission.md @@ -416,6 +416,6 @@ AdmissionCheck. The operator never leaves such a queue admitting — see **See also** — [Accelerator Requests](requests.md) (the normative request contract) · [Device Discovery](discovery.md#the-device-plugin-allocator) (gate 5 in detail) · -[Walkthrough](../../getting-started/walkthrough.md) (the four views moving on a live cluster) +[Walkthrough](../../walkthroughs/devices/scheduling.md) (the four views moving on a live cluster) **Next** → [Installation Modes](../../operate/installation-modes.md) — how the chain gets deployed. diff --git a/docs/modules/devices/nvidia-mig.md b/docs/modules/devices/nvidia-mig.md index cb052cb94..900cf7bb3 100644 --- a/docs/modules/devices/nvidia-mig.md +++ b/docs/modules/devices/nvidia-mig.md @@ -344,7 +344,7 @@ It does create and destroy the instances backing scheduled workloads ## Walkthrough: three MIG configurations on one node A recorded run on a live Kubernetes cluster, in the style of the -[scheduling-chain walkthrough](../../getting-started/walkthrough.md): every command is the real `kubectl` (or on-node +[scheduling-chain walkthrough](../../walkthroughs/devices/scheduling.md): every command is the real `kubectl` (or on-node `nvidia-smi`) invocation and its real output, on `node-h100`, a genericized node of **eight** H100-80GB GPUs running operator defaults. Eight is the point: the families are served by **disjoint GPU populations**, so partitioning *some* GPUs moves exactly those. @@ -782,4 +782,4 @@ gpustack--nvidia-h100-80gb-hbm3-linux-amd64 gpustack-fnv64-e4768a65ca0ce96b "what can I still get") · [Device Discovery](discovery.md#the-partitioned-family-fungible-tokens) (how a partition is placed and reclaimed) -**Next** → [Walkthrough](../../getting-started/walkthrough.md) — the logical-slicing counterpart on a live cluster. +**Next** → [Walkthrough](../../walkthroughs/devices/scheduling.md) — the logical-slicing counterpart on a live cluster. diff --git a/docs/modules/devices/requests.md b/docs/modules/devices/requests.md index 3eab020ae..d43037b8a 100644 --- a/docs/modules/devices/requests.md +++ b/docs/modules/devices/requests.md @@ -529,7 +529,7 @@ A manifest asking `"3"` for a heavier share of one accelerator now needs a node `InstanceType.status.acceleratorShared.onceMaxRequest` moved with it: it counts a node's accelerators with a free share, where it used to sum their shares. Output recorded before the change, in the -[Walkthrough](../../getting-started/walkthrough.md) and [NVIDIA MIG Operations](./nvidia-mig.md), still shows the +[Walkthrough](../../walkthroughs/devices/scheduling.md) and [NVIDIA MIG Operations](./nvidia-mig.md), still shows the old reading under `SH` — `10/10` on one free accelerator now reads `1/10`. ## Limitations @@ -596,4 +596,4 @@ partitioning) · [Admission](./admission.md) (where these keys are checked) · served) · [RDMA Operations](../rdma/operations.md) (how many RDMA endpoints to ask for beside N accelerators, and the kubelet policy that aligns the two sides) -**Next** → [Walkthrough](../../getting-started/walkthrough.md) — the same requests on a live cluster. +**Next** → [Walkthrough](../../walkthroughs/devices/scheduling.md) — the same requests on a live cluster. diff --git a/docs/modules/devices/scheduling.md b/docs/modules/devices/scheduling.md index f13ad3f5c..fb852c433 100644 --- a/docs/modules/devices/scheduling.md +++ b/docs/modules/devices/scheduling.md @@ -365,7 +365,7 @@ The per-accelerator **AdmissionCheck**, third of the five gates; its behavior is **See also** — [Device Discovery](discovery.md) (where the capacity signals come from) · [Topology-Aware Scheduling](../topology/scheduling.md) (how topology profiles enter this chain) · -[Walkthrough](../../getting-started/walkthrough.md) (the same objects on a live cluster) · +[Walkthrough](../../walkthroughs/devices/scheduling.md) (the same objects on a live cluster) · [Settings](../../reference/settings.md#online-adjustable-settings) **Next** → [Admission](admission.md) — the five gates a request passes. diff --git a/docs/modules/kv-cache/_index.md b/docs/modules/kv-cache/_index.md index 0a4fb700d..587f26032 100644 --- a/docs/modules/kv-cache/_index.md +++ b/docs/modules/kv-cache/_index.md @@ -10,7 +10,7 @@ capacity they receive. ## Quick start -[KV Cache Walkthrough](walkthrough.md) creates the backend, pool, binding and +[KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) creates the backend, pool, binding and deployment in order. A plain Pod can also use a binding through [KV Cache Injection](injection.md). @@ -26,4 +26,4 @@ tier](local-disk-tier.md) have separate operating details. **See also** — [Model Deployment](../model-deployment/_index.md) (cache consumers) · [RDMA Networking](../rdma/_index.md) (transport) -**Next** → [KV Cache Walkthrough](walkthrough.md) — create a shared cache. +**Next** → [KV Cache Walkthrough](../../walkthroughs/kv-cache/shared-cache.md) — create a shared cache. diff --git a/docs/modules/model-delivery/_index.md b/docs/modules/model-delivery/_index.md index e9a8e6015..3498c6b9e 100644 --- a/docs/modules/model-delivery/_index.md +++ b/docs/modules/model-delivery/_index.md @@ -3,6 +3,8 @@ `ModelArtifact` gives a model version a stable identity. Its delivery can be handled by the engine, a node cache, a PVC or an image. +Follow the [Model Prefetch Walkthrough](../../walkthroughs/model-delivery/prefetch.md) for a complete example with captured CR output. + ## Contents - [Sources and delivery](#sources-and-delivery) diff --git a/docs/modules/model-deployment/_index.md b/docs/modules/model-deployment/_index.md index 9b3566e2a..ba3d51761 100644 --- a/docs/modules/model-deployment/_index.md +++ b/docs/modules/model-deployment/_index.md @@ -21,9 +21,9 @@ the role contract. [Prefill and Decode](prefill-decode.md) covers split roles. are directed. [Elastic EP](elastic-ep.md) keeps one serving instance and changes its collective while it runs. -[Model Scaling Walkthrough](scaling-walkthrough.md) follows prefetch, TP2/DP2 startup, and DP2→4 expansion +[Elastic EP Walkthrough](../../walkthroughs/model-deployment/elastic-ep.md) follows prefetch, TP2/DP2 startup, and DP2→4 expansion with captured resource output and completed inference requests. -[External DP Walkthrough](external-dp-walkthrough.md) runs a fixed group with one HTTP endpoint per rank +[External DP Walkthrough](../../walkthroughs/model-deployment/external-dp.md) runs a fixed group with one HTTP endpoint per rank and checks managed Router coverage. --- diff --git a/docs/modules/model-deployment/elastic-ep.md b/docs/modules/model-deployment/elastic-ep.md index e14bbee43..f2a13f79f 100644 --- a/docs/modules/model-deployment/elastic-ep.md +++ b/docs/modules/model-deployment/elastic-ep.md @@ -1,7 +1,7 @@ # Elastic EP An elastic expert-parallel deployment keeps one vLLM serving instance and changes its GPU collective -while it runs. The operator adds or removes GPU members inside one Ray cluster. +while it runs. The operator adds GPU members inside one Ray cluster. The engine reconfigures while Kubernetes, Kueue, Ray and accelerator accounting converge. ## Contents @@ -61,7 +61,7 @@ before starting the profile. The TP size must fit the selected MoE model, engine kernels, and GPUs available on one node. The operator does not limit TP to the size used by a particular test. -The [Model Scaling Walkthrough](scaling-walkthrough.md) records successful TP2/DP2 startup and DP2→4 +The [Elastic EP Walkthrough](../../walkthroughs/model-deployment/elastic-ep.md) records successful TP2/DP2 startup and DP2→4 expansion using the official GPUStack runner and DeepSeek-V2-Lite-Chat on an eight-card RTX PRO 6000 node. That result is limited to its pinned software, model, configuration, and short inference checks. Admission of TP=2, 4 or 8 does not establish dynamic inference support. @@ -112,7 +112,7 @@ spec: replicas: 1 size: 1 instanceType: elastic-gpu - image: your-registry/vllm-ray:0.29.0 + image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91 elasticEp: width: 2 ports: diff --git a/docs/modules/model-deployment/routing.md b/docs/modules/model-deployment/routing.md index dbf11d44b..61338e83f 100644 --- a/docs/modules/model-deployment/routing.md +++ b/docs/modules/model-deployment/routing.md @@ -3,12 +3,11 @@ A managed router selects among the HTTP endpoints of serving replicas. Every routing choice measured here ran on a server role; on a prefill/decode pair, where each half is chosen separately, none has been run. -External DP within a Prefill or Decode role remains unverified; see -[issue #754](https://github.com/gpustack/gpustack-operator/issues/754). -Keep each P/D replica's leader as its HTTP endpoint until that combination is validated. +See [Prefill and Decode](prefill-decode.md) for the External DP boundary within P/D roles. ## Contents +- [Which Pods receive traffic](#which-pods-receive-traffic) - [Each router's default](#each-routers-default) - [Prefix affinity](#prefix-affinity) - [Switching to round robin](#switching-to-round-robin) @@ -16,6 +15,31 @@ Keep each P/D replica's leader as its HTTP endpoint until that combination is va - [Confirming that a Router serves](#confirming-that-a-router-serves) - [Per-worker routing metrics](#per-worker-routing-metrics) +## Which Pods receive traffic + +The engine's LB mode determines the candidate endpoints. The Router policy chooses among those candidates. +`round_robin` does not change the endpoint selector. + +| Deployment shape | HTTP candidates | +|---|---| +| Internal Server group | Member 0 of each replica. | +| Qualified External Server group | Every qualified member; each member serves one DP rank. | +| Prefill or Decode group | Member 0 of each replica, including groups with `size > 1`. | +| Unknown mode or unverified group | Keep the existing leader restriction; do not expand discovery. | + +For an eligible External Server role, Router discovery and ordinary Services omit `member-index=0`. +They retain the endpoint eligibility label, so an unhealthy group cannot receive new traffic. +The replica's headless Service always publishes every member for peer communication. + +Known non-Internal modes follow the External selector rule. Status retains the engine's distinct mode name. +Hybrid GPU behavior remains untested. See [Prefill and Decode](prefill-decode.md) for P/D limitations. + +Managed roles expose one fixed HTTP port per Pod, defaulting to 8000. +All routed roles must use the same serving port. A shared custom port remains supported. +Admission rejects vLLM's MultiPort mode because it starts endpoints on additional, incremented ports. +A decoder proxy's internal backend port remains separate from its published HTTP port. +See the [External DP walkthrough](../../walkthroughs/model-deployment/external-dp.md) for a minimal per-rank configuration. + ## Each router's default **`vllm-router` and `sglang-gateway` route by `cache_aware`, and `llm-d-router` scores every diff --git a/docs/reference/instance-type-unit-resources.md b/docs/reference/instance-type-unit-resources.md index 3ebfff88d..c54123016 100644 --- a/docs/reference/instance-type-unit-resources.md +++ b/docs/reference/instance-type-unit-resources.md @@ -255,6 +255,6 @@ at the top of that file; when either changes, compare the rows here with the pre **See also** — [Scheduling Chain](../modules/devices/scheduling.md#unit-spec-defaults) (where the unit spec is stamped) · [Admission](../modules/devices/admission.md#the-instancetype-and-instance-webhooks) -(what the webhooks enforce on it) · [Walkthrough](../getting-started/walkthrough.md#4-managing-a-custom-instancetype) +(what the webhooks enforce on it) · [Walkthrough](../walkthroughs/devices/scheduling.md#4-managing-a-custom-instancetype) **Next** → [Settings](settings.md#online-adjustable-settings) — the switches named on this page. diff --git a/docs/reference/settings.md b/docs/reference/settings.md index 78058dcc8..f73ebb9cd 100644 --- a/docs/reference/settings.md +++ b/docs/reference/settings.md @@ -397,4 +397,4 @@ user-facing knobs, listed for completeness. [Installation Modes](../operate/installation-modes.md) (which flags a mode sets for you) · [High Availability Operations](../operate/high-availability.md) -**Next** → [Walkthrough](../getting-started/walkthrough.md) — a setting flipped on a live cluster, before and after. +**Next** → [Walkthrough](../walkthroughs/devices/scheduling.md) — a setting flipped on a live cluster, before and after. diff --git a/docs/walkthroughs/_index.md b/docs/walkthroughs/_index.md new file mode 100644 index 000000000..467cb2796 --- /dev/null +++ b/docs/walkthroughs/_index.md @@ -0,0 +1,34 @@ +# Walkthroughs + +Follow a complete example, then compare the resource output with the observed result. +Each module has its own directory. Configuration contracts remain in the capability guides. + +## Contents + +- [Choose a walkthrough](#choose-a-walkthrough) +- [Read the evidence](#read-the-evidence) + +## Choose a walkthrough + +| Module | Walkthrough | Task | +|---|---|---| +| Devices | [Device Scheduling](devices/scheduling.md) | Follow detection, queues, admission, and accelerator allocation. | +| KV Cache | [Shared Cache](kv-cache/shared-cache.md) | Create a backend, pool, namespace grant, and serving workload. | +| Model Delivery | [Model Prefetch](model-delivery/prefetch.md) | Cache and verify model weights before inference. | +| Model Deployment | [Preparation and choices](model-deployment/_index.md) | Choose the serving shape and prepare GPU resources. | +| Model Deployment | [Elastic EP](model-deployment/elastic-ep.md) | Grow DP width from 2 to 4 with TP fixed at 2. | +| Model Deployment | [External DP](model-deployment/external-dp.md) | Serve fixed DP ranks and verify Router distribution. | + +## Read the evidence + +Commands sit beside captured output or the checks needed to interpret it. +A CR's declared configuration does not prove runtime parallelism or successful inference. +The model-serving walkthroughs pair CR snapshots with completed requests and native runtime measurements. +Their validation records state software identities and limits. + +--- + +**See also** — [Architecture](../getting-started/architecture.md) · +[Model Deployment](../modules/model-deployment/_index.md) + +**Next** → [Device Scheduling Walkthrough](devices/scheduling.md) diff --git a/docs/getting-started/walkthrough.md b/docs/walkthroughs/devices/scheduling.md similarity index 95% rename from docs/getting-started/walkthrough.md rename to docs/walkthroughs/devices/scheduling.md index 3e0131e84..9581afaef 100644 --- a/docs/getting-started/walkthrough.md +++ b/docs/walkthroughs/devices/scheduling.md @@ -1,4 +1,4 @@ -# Walkthrough +# Device Scheduling Walkthrough This page records a real session: every `kubectl` invocation and its real output, objects as YAML trimmed to `metadata.labels` / `spec` / `status`, and a before / after `kubectl get instancetypes` @@ -10,14 +10,14 @@ until section 5 flips it on. Watch three columns: **UNIT(CPU/RAM)/STORAGE**, the per-unit request the InstanceType charges; **CPU**, the collapsed CPU pool's `remaining/capacity` cores; **ACCELERATOR(EX/SH/SL/PT)**, the -`onceMaxRequest/remaining` of each [four-view](../modules/devices/admission.md#four-view-status) +`onceMaxRequest/remaining` of each [four-view](../../modules/devices/admission.md#four-view-status) projection. This run predates the shared card-count reading, so its `SH` once-max figures are -shares, not accelerators ([Pre-release breaks](../modules/devices/requests.md#pre-release-breaks)). +shares, not accelerators ([Pre-release breaks](../../modules/devices/requests.md#pre-release-breaks)). Each accelerator counts in exactly one of `EX`/`SH`/`SL` (unpartitioned) or `PT` (partitioned), so `0/0` under `PT` throughout means none is in a partitioning mode. For the all-partitioned and **mixed** configurations see the [three-configuration -walkthrough](../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node). +walkthrough](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node). ## Contents @@ -415,17 +415,17 @@ Deleting the Instance releases the slice: the row returns to `1/1 10/10 100/100 > instead hard-partitions into fixed hardware instances the operator materializes on demand: > > - a different resource family (`.partitioned*`, reported under `PT`), with a different request shape — -> keys and rules in [Accelerator Requests](../modules/devices/requests.md); +> keys and rules in [Accelerator Requests](../../modules/devices/requests.md); > - MIG *mode* is the administrator's, driven with `nvidia-smi`, so it has its own runbook and a recorded > enable → request → reclaim → disable walkthrough in -> [NVIDIA MIG Operations](../modules/devices/nvidia-mig.md). +> [NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md). --- ## 4. Managing a custom InstanceType A derived pool is sized from a **per-product preset**: 8 CPU / 64 GiB for this A10G, 4 CPU / 16 GiB for -an unrecognised accelerator ([preset reference](../reference/instance-type-unit-resources.md)). For +an unrecognised accelerator ([preset reference](../../reference/instance-type-unit-resources.md)). For another size, an admin authors an InstanceType referencing a catalog flavor by its `acceleratorGroup`, with a unit spec of its own: @@ -666,10 +666,10 @@ spec: - **They mount into the workload container only.** The SSH sidecar needs no change: it enters that container's mount namespace per session, so every mount is visible over SSH too. - **A persistent claim places the Instance where it can attach.** The workspace claim and every - `persistent` entry follow the [claim placement rules](../modules/model-delivery/artifact.md#claim-delivery-and-placement): + `persistent` entry follow the [claim placement rules](../../modules/model-delivery/artifact.md#claim-delivery-and-placement): a node-local volume pins the Instance to its node and that node's capacity; network or `ReadWriteMany` storage avoids the pin. A claim nothing binds creates no Pod and says why in `status.phaseMessage`; - [`instance-persistent-volume-placement`](../reference/settings.md#online-adjustable-settings) turns this off. + [`instance-persistent-volume-placement`](../../reference/settings.md#online-adjustable-settings) turns this off. - **Both new fields are immutable while the Instance runs**, editable while stopped — the rule the rest of `spec` follows. @@ -682,7 +682,7 @@ kept separate so node-path mounts can be allowed without a container escape: | `instance-host-path-volume-allowed` | `spec.additionalVolumes[*].hostPath` — reaches the node's filesystem, but not its devices or kernel. | Each gates taking its escape, so turning one off stops new grants without stranding an Instance -that already holds one; [Settings](../reference/settings.md#online-adjustable-settings) has the exact terms. +that already holds one; [Settings](../../reference/settings.md#online-adjustable-settings) has the exact terms. Both govern the **node** boundary, not the namespace one: a `persistent`, `configMap` or `secret` source names an object in the Instance's own namespace and may always be mounted, the same reach a Pod @@ -691,9 +691,9 @@ other's Secrets in their own. --- -**See also** — [Accelerator Requests](../modules/devices/requests.md) (the contract behind step 3) · -[NVIDIA MIG Operations](../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node) -(hardware partitioning) · [Settings](../reference/settings.md#online-adjustable-settings) (the two gates) +**See also** — [Accelerator Requests](../../modules/devices/requests.md) (the contract behind step 3) · +[NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md#walkthrough-three-mig-configurations-on-one-node) +(hardware partitioning) · [Settings](../../reference/settings.md#online-adjustable-settings) (the two gates) -**Next** → [NVIDIA MIG Operations](../modules/devices/nvidia-mig.md) — hardware partitioning, from enabling +**Next** → [NVIDIA MIG Operations](../../modules/devices/nvidia-mig.md) — hardware partitioning, from enabling the mode to reclaiming the instance. diff --git a/docs/modules/kv-cache/walkthrough.md b/docs/walkthroughs/kv-cache/shared-cache.md similarity index 85% rename from docs/modules/kv-cache/walkthrough.md rename to docs/walkthroughs/kv-cache/shared-cache.md index dae9b7653..42fc72b94 100644 --- a/docs/modules/kv-cache/walkthrough.md +++ b/docs/walkthroughs/kv-cache/shared-cache.md @@ -26,7 +26,7 @@ name in the manifests below, then apply them in that order. The order is fixed because each object names the one above it, so creating them the other way round leaves references that resolve to nothing. Why the two scopes split this way is on -[KV Cache Pool](pool.md#two-kinds-split-by-scope). +[KV Cache Pool](../../modules/kv-cache/pool.md#two-kinds-split-by-scope). **The Binding is the authorization point.** A namespace gets access to a store when an administrator creates a Binding in it. A user naming a pool is not a path this API has: `poolRef` is a @@ -54,7 +54,7 @@ spec: `spec.image` is left unset on purpose: the cluster-wide `kv-cache-backend-image` [Setting](../../reference/settings.md) supplies this project's own build, which is the one that can elect a leader later. Naming a published upstream image here works until Step 4 and then does not. See -[High availability](leader.md#high-availability). +[High availability](../../modules/kv-cache/leader.md#high-availability). `capacityPerMember` is charged to each member Pod's host memory request, so it is a claim on the node and not a hint. One member Pod runs per node the selector matches. @@ -62,7 +62,7 @@ and not a hint. One member Pod runs per node the selector matches. `leader: {}` takes the field defaults, `multiTenancy` included (it defaults on), so this master keeps a per-tenant quota ledger and Steps 2 and 3 read against it. A backend pinned to a store image from before Mooncake 0.3.12 is the one exception and declares `multiTenancy: false` out loud. See -[The project's own build variants](backend.md#the-projects-own-build-variants). +[The project's own build variants](../../modules/kv-cache/backend.md#the-projects-own-build-variants). Wait for it, then read what it reports: @@ -117,7 +117,7 @@ the master; the name is what keeps the two apart. **A quota ceiling is not a reservation.** It is the most this namespace may hold at once, and going over it does not fail a write. See -[Full-quota behavior](pool.md#full-quota-behavior). +[Full-quota behavior](../../modules/kv-cache/pool.md#full-quota-behavior). **The ceiling is enforced because the Step-1 leader carries its tenant ledger, which the default gave it.** A backend declared `leader.multiTenancy: false` holds no ledger instead: its pool is @@ -126,7 +126,7 @@ admitted with a warning, the Binding reports `QuotaGranted=True` with reason `Un **A multi-tenant master refuses a tenant name absent from its ledger.** An engine that ignores the injected tenant then needs a second Binding whose domain is `default`, or that leaves `name` out. -See [Tenant compatibility](injection.md#tenant-compatibility). +See [Tenant compatibility](../../modules/kv-cache/injection.md#tenant-compatibility). ## Step 3: the workload @@ -159,12 +159,12 @@ useful comparison to have run once before attributing anything to the pool. **The engine is configured by injection, not by anything you write here.** The operator resolves the Binding, reads the backend's endpoint and the domain, and injects the store's environment into every role's Pod. What is injected per engine, and every refusal, is on -[KV Cache Injection](injection.md). +[KV Cache Injection](../../modules/kv-cache/injection.md). **The store version must match the engine's client.** The engine version above decides the client, and Step 1's unset `spec.image` leaves the store on the Settings default. A pair on different lines starts healthy and then fails every write, so the version above is a choice. See -[The store version must match the engine's client](backend.md#the-store-version-must-match-the-engines-client). +[The store version must match the engine's client](../../modules/kv-cache/backend.md#the-store-version-must-match-the-engines-client). Check the attachment from the deployment's own status rather than from the Pods: @@ -192,12 +192,12 @@ as the old leader steps down. Both readings are normal. **Scaling from one to three keeps the first leader Pod and the member templates** because the Lease election and accounts were already present at one replica. See -[the leader Deployment](leader.md#the-deployment-and-the-two-probes) for the +[the leader Deployment](../../modules/kv-cache/leader.md#the-deployment-and-the-two-probes) for the cost of changing `leader.electionBackend`. **`leader.memberAddressing` chooses how members find the master**, and defaults to `Service`. An explicit `Lease` value uses the member's API access to read the current holder. See -[High availability](leader.md#high-availability) for the measured failover limits. +[High availability](../../modules/kv-cache/leader.md#high-availability) for the measured failover limits. These conditions apply at one replica too; the phase does not summarize them: @@ -218,7 +218,7 @@ misses afterwards. Plan for a cold cache after every failover and every leader r **The store's snapshot is not offered, and its flags are refused in `leader.extraArgs`**, because restoring a snapshot can make the cache serve another key's bytes instead of a miss. -[High availability](leader.md#high-availability) says why. +[High availability](../../modules/kv-cache/leader.md#high-availability) says why. ## Failure modes @@ -235,11 +235,11 @@ reading on this page most likely to be escalated as an outage. --- -**See also** — [KV Cache Backend](backend.md) (every field of the store, and what status reports) · -[KV Cache Leader](leader.md) (the election, why there is no snapshot, and the member addressing choice in full) · -[KV Cache Pool](pool.md) (quota, domains and what a full quota does) · -[Model Deployment](../model-deployment/deployment.md) (roles, prefill/decode, rollout) · -[Model Deployment Prefill and Decode](../model-deployment/prefill-decode.md) (router, transfer) · -[KV Cache Injection](injection.md) (what a Pod actually receives) +**See also** — [KV Cache Backend](../../modules/kv-cache/backend.md) (every field of the store, and what status reports) · +[KV Cache Leader](../../modules/kv-cache/leader.md) (the election, why there is no snapshot, and the member addressing choice in full) · +[KV Cache Pool](../../modules/kv-cache/pool.md) (quota, domains and what a full quota does) · +[Model Deployment](../../modules/model-deployment/deployment.md) (roles, prefill/decode, rollout) · +[Model Deployment Prefill and Decode](../../modules/model-deployment/prefill-decode.md) (router, transfer) · +[KV Cache Injection](../../modules/kv-cache/injection.md) (what a Pod actually receives) -**Next** → [KV Cache Pool](pool.md) — the quota this walkthrough set once and did not explain. +**Next** → [KV Cache Pool](../../modules/kv-cache/pool.md) — the quota this walkthrough set once and did not explain. diff --git a/docs/walkthroughs/model-delivery/prefetch.md b/docs/walkthroughs/model-delivery/prefetch.md new file mode 100644 index 000000000..e2c3a7db3 --- /dev/null +++ b/docs/walkthroughs/model-delivery/prefetch.md @@ -0,0 +1,164 @@ +# Model Prefetch Walkthrough + +Cache DeepSeek-V2-Lite-Chat on one GPU node before starting inference. +This shared preparation serves both the Elastic EP and External DP walkthroughs. +The output below was captured with node model delivery enabled. + +## Contents + +- [Prerequisites](#prerequisites) +- [Step 1: create the artifact and prefetch](#step-1-create-the-artifact-and-prefetch) +- [Step 2: verify the cached model](#step-2-verify-the-cached-model) +- [Step 3: start a serving workload](#step-3-start-a-serving-workload) +- [Step 4: release the cache](#step-4-release-the-cache) + +## Prerequisites + +Enable [node model delivery](../../modules/model-delivery/operations.md) and select the GPU node. +You need `kubectl`, `jq`, access to cluster-scoped model stores, and enough disk for the weights. +The example uses a dedicated namespace and store grant. + +## Step 1: create the artifact and prefetch + +Save the following as `model-delivery.yaml`. Replace both `gpu-node` values with your GPU node's +`kubernetes.io/hostname` label. The store grant and the prefetch placement must select that same node. +The fixed file list resolves to 12 files and 31,418,807,311 bytes. + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: gpustack-elastic +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelArtifact +metadata: + name: deepseek-v2-lite-chat + namespace: gpustack-elastic +spec: + source: + huggingFace: + repository: deepseek-ai/DeepSeek-V2-Lite-Chat + revision: 85864749cd611b4353ce1decdb286193298f64c7 + allowPatterns: + - model-00001-of-000004.safetensors + - model-00002-of-000004.safetensors + - model-00003-of-000004.safetensors + - model-00004-of-000004.safetensors + - config.json + - configuration_deepseek.py + - generation_config.json + - model.safetensors.index.json + - modeling_deepseek.py + - tokenization_deepseek_fast.py + - tokenizer.json + - tokenizer_config.json +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelStore +metadata: + name: rtx6000-models +spec: + nodeSelector: + matchLabels: + kubernetes.io/hostname: gpu-node +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelStoreBinding +metadata: + name: rtx6000-models + namespace: gpustack-elastic +spec: + storeRefs: + - name: rtx6000-models + quota: + bytes: 64Gi + allowPinned: true +--- +apiVersion: worker.gpustack.ai/v1 +kind: ModelPrefetch +metadata: + name: deepseek-v2-lite-chat + namespace: gpustack-elastic +spec: + artifactRef: + name: deepseek-v2-lite-chat + bindingRef: + name: rtx6000-models + placement: + nodeSelector: + matchLabels: + kubernetes.io/hostname: gpu-node + minReady: 1 + retention: + pinned: true +``` + +## Step 2: verify the cached model + +```bash +kubectl apply -f model-delivery.yaml +kubectl -n gpustack-elastic get modelartifacts,modelprefetches +kubectl -n gpustack-elastic get modelprefetch deepseek-v2-lite-chat -o yaml +kubectl get nodemodelstores -o json +``` + +Captured `kubectl get modelartifacts,modelprefetches` output: + +```text +NAME SOURCE REVISION SIZE READY DOWNLOADING RESOLVED +modelartifact.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-ai/DeepSeek-V2-Lite-Chat 85864749cd61 31418807311 1 0 True + +NAME ARTIFACT BINDING READY DESIRED AVAILABLE +modelprefetch.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-v2-lite-chat rtx6000-models 1 1 True +``` + +Check the matching node's model entry as well: + +```bash +kubectl get nodemodelstores -o json | + jq '.items[] | .status.models[]? | {digest, sizeBytes, state}' +``` + +Captured projection: + +```json +{ + "digest": "sha256:1369aae0b5cc2d0e260557bcea04b0d6aad40f8bb6accdac60f1ee543887d9f7", + "sizeBytes": 31418807311, + "state": "Ready" +} +``` + +The validation also retained the successful warm-up Pod and compared its 12 SHA-256 results with +the resolved manifest. Warm-up Pods are transient; collect their logs before completion if you need +that evidence. See [Model Prefetch](../../modules/model-delivery/prefetch.md). +Wait for the required content before creating the serving workload. + +## Step 3: start a serving workload + +Continue with [Elastic EP](../model-deployment/elastic-ep.md) to grow one collective, +or [External DP](../model-deployment/external-dp.md) to route requests across fixed DP ranks. +Use the namespace and artifact created here in either manifest. + +## Step 4: release the cache + +Keep the prefetch while a serving application needs the cached weights. +After deleting those consumers, remove the dedicated resources: + +```bash +kubectl -n gpustack-elastic delete modelprefetch deepseek-v2-lite-chat +kubectl -n gpustack-elastic delete modelstorebinding rtx6000-models +kubectl -n gpustack-elastic delete modelartifact deepseek-v2-lite-chat +kubectl delete modelstore rtx6000-models +``` + +Cache collection follows the [retention rules](../../modules/model-delivery/prefetch.md). +Deleting these CRs does not delete the Kubernetes cluster or its disks. + +--- + +**See also** — [Model Delivery](../../modules/model-delivery/_index.md) · +[Model Prefetch](../../modules/model-delivery/prefetch.md) (configuration and retention) + +**Next** → [Model Deployment Walkthroughs](../model-deployment/_index.md) diff --git a/docs/walkthroughs/model-deployment/_index.md b/docs/walkthroughs/model-deployment/_index.md new file mode 100644 index 000000000..494706680 --- /dev/null +++ b/docs/walkthroughs/model-deployment/_index.md @@ -0,0 +1,94 @@ +# Model Deployment Walkthroughs + +Choose a serving shape, prepare its model, then follow the matching walkthrough. +Both examples use the official GPUStack runner and retain captured CR output and inference results. + +## Contents + +- [Choose an example](#choose-an-example) +- [Prepare the cluster](#prepare-the-cluster) +- [Prepare the GPU InstanceType](#prepare-the-gpu-instancetype) +- [Check the software identity](#check-the-software-identity) + +## Choose an example + +| Walkthrough | Serving shape | What was measured | +|---|---|---| +| [Elastic EP](elastic-ep.md) | One Internal LB instance on Ray | TP2, DP width 2→4, EP world 4→8; ordinary and streaming requests before and after expansion. | +| [External DP](external-dp.md) | Two fixed DP ranks using multiprocessing | TP2, DP2, EP world4; direct requests to each rank and Router distribution across both ranks. | + +A Router can front either shape. The engine's LB mode determines which Pods accept inference requests. +Increasing `roles[].replicas` adds complete groups; see [role scaling](../../modules/model-deployment/deployment.md#rollout-behavior). +That operation was not measured in these walkthroughs. + +## Prepare the cluster + +Install the operator and enable node model delivery using the +[installation guide](../../operate/installation-modes.md) and +[model-store operations](../../modules/model-delivery/operations.md). +Use `kubectl`, `jq`, `curl`, and a kubeconfig for the intended cluster. +The recorded cluster had one CPU node and one node with eight NVIDIA RTX PRO 6000 Blackwell GPUs. + +```bash +kubectl get nodes +kubectl get devices -o json +kubectl get instancetypes +kubectl get nodemodelstores -o json +``` + +Confirm model-manager readiness and free GPUs before creating a serving workload. +External DP needs four free cards. Elastic EP needs four initially and eight after expansion. +Complete the [model prefetch walkthrough](../model-delivery/prefetch.md) before starting either example. + +## Prepare the GPU InstanceType + +Save the following as `gpu-instance-type.yaml` and apply it once. +For another GPU family, select an appropriate whole-GPU InstanceType and verify model compatibility. +Each serving member requests two GPUs; CPU and memory budgets scale with that count. + +```yaml +apiVersion: worker.gpustack.ai/v1 +kind: InstanceType +metadata: + name: elastic-cpu-gpu +spec: + generalGroup: generic + acceleratable: true + acceleratorGroup: nvidia-rtx-pro-6000-blackwell-server-edition + os: linux + arch: amd64 + localStorage: 10Gi + unitResources: + cpu: "4" + ram: 16Gi +``` + +```bash +kubectl apply -f gpu-instance-type.yaml +kubectl get instancetype elastic-cpu-gpu +``` + +Both examples refer to `elastic-cpu-gpu`. Replace that name if you use an existing InstanceType. + +## Check the software identity + +Read the running images and binary versions. Repeat for the worker, device-manager, and model-manager Pods. +Set the namespace, Pod, and container names from your installation: + +```bash +kubectl get pods -A +kubectl -n "$OPERATOR_NAMESPACE" get pod "$OPERATOR_POD" -o json | + jq '.status.containerStatuses[] | {name, image, imageID}' +kubectl -n "$OPERATOR_NAMESPACE" exec "$OPERATOR_POD" -c "$OPERATOR_CONTAINER" -- \ + gpustack-operator --version +``` + +The `dev` tag can move. Each walkthrough's validation record identifies the binary and image used. +The serving manifests pin the official runner digest; no engine packages were replaced during validation. + +--- + +**See also** — [Model Deployment](../../modules/model-deployment/_index.md) (configuration guides) · +[Model Prefetch Walkthrough](../model-delivery/prefetch.md) + +**Next** → [Elastic EP Walkthrough](elastic-ep.md) or [External DP Walkthrough](external-dp.md) diff --git a/docs/modules/model-deployment/scaling-walkthrough.md b/docs/walkthroughs/model-deployment/elastic-ep.md similarity index 58% rename from docs/modules/model-deployment/scaling-walkthrough.md rename to docs/walkthroughs/model-deployment/elastic-ep.md index 394070097..6d6a81ea1 100644 --- a/docs/modules/model-deployment/scaling-walkthrough.md +++ b/docs/walkthroughs/model-deployment/elastic-ep.md @@ -1,229 +1,36 @@ -# Model Scaling Walkthrough +# Elastic EP Walkthrough -Prefetch a model, start one elastic vLLM instance, and grow its expert-parallel collective. -The recorded run used one CPU node and one node with eight NVIDIA RTX PRO 6000 Blackwell GPUs. -Captured custom-resource output appears beside the checks it supports. +Start one elastic vLLM instance and grow its expert-parallel collective on the same eight-GPU node. +This example holds TP=2 and expands DP width from 2 to 4, so EP world grows from 4 to 8. +Captured CR output appears beside each check. ## Contents -- [Choose the scaling operation](#choose-the-scaling-operation) -- [Prepare the cluster](#prepare-the-cluster) -- [Prefetch the weights](#prefetch-the-weights) -- [Start the elastic deployment](#start-the-elastic-deployment) -- [Check the baseline](#check-the-baseline) -- [Expand the collective](#expand-the-collective) -- [Inspect a stalled resize](#inspect-a-stalled-resize) -- [Scale independent replicas](#scale-independent-replicas) -- [Release the application](#release-the-application) - -## Choose the scaling operation - -| Operation | Field | Effect | Engine scope | -|---|---|---|---| -| Independent serving replicas | `roles[].replicas` | Starts or retires complete instances, each with its own model and engine. | vLLM or SGLang; use the [engine version requirements](engine-versions.md). | -| Elastic DP/EP width | `roles[].elasticEp.width` | Changes the DP engines and EP collective inside one serving instance. | vLLM with Elastic EP, a compatible MoE model, and a full Ray installation. | - -The elastic path keeps `replicas: 1`, `size: 1`, and TP=2 throughout. -Here, `width` is the DP size. Increasing it from 2 to 4 grows the EP world from 4 to 8. -Width 2 consumes four whole GPUs. Width 4 consumes eight. -TP is fixed after creation. This release supports width increases; width decreases are refused. -See [Elastic EP](elastic-ep.md) for the complete contract. - -The recorded validation covers elastic TP2/DP2 startup and DP2→4 expansion with short inference requests. -The independent-replica example is a configuration alternative and was not executed in this run. -Neither path adds Kubernetes nodes or cloud capacity. -For a fixed group with a separate HTTP endpoint per DP rank, use the -[External DP Walkthrough](external-dp-walkthrough.md). - -## Prepare the cluster - -Install the operator and enable node model delivery using the -[installation guide](../../operate/installation-modes.md) and -[model-store operations](../model-delivery/operations.md). -The commands below require `kubectl`, `jq`, `curl`, and access to cluster-scoped model-store resources. -Use an isolated kubeconfig for the test cluster. - -The recorded software and model identities were: - -| Component | Recorded identity | -|---|---| -| Operator | `gpustack/gpustack-operator:dev`, binary revision `d00f8fa8c98334a71af4c6eb302ee80d08289309` | -| Operator image digest | `sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36` | -| Inference engine | vLLM 0.29.0 with full Ray 2.54.0 | -| Engine image | `gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91` | -| Model | `deepseek-ai/DeepSeek-V2-Lite-Chat` | -| Model revision | `85864749cd611b4353ce1decdb286193298f64c7` | - -The official GPUStack runner includes Ray. The run used its installed packages without replacement. -Container reads found `cupy-cuda13x` 14.2.0 and no `cupy-cuda12x` installation. -Verify the image digest and installed vLLM/Ray versions when selecting another runner. -Do not add `--elastic-ep-max-dp-size` to this example; that argument starts with vLLM 0.30.0. - -Read the actual operator Pod's image identity and binary version. Repeat for worker, device-manager, -and model-manager Pods. Set the namespace, Pod, and container names from your installation: +- [Prerequisites](#prerequisites) +- [Step 1: start the elastic deployment](#step-1-start-the-elastic-deployment) +- [Step 2: verify baseline inference](#step-2-verify-baseline-inference) +- [Step 3: expand and verify inference](#step-3-expand-and-verify-inference) +- [Step 4: release the application](#step-4-release-the-application) +- [Diagnose a stalled resize](#diagnose-a-stalled-resize) +- [Validation record](#validation-record) -```bash -kubectl get pods -A -kubectl -n "$OPERATOR_NAMESPACE" get pod "$OPERATOR_POD" -o json | - jq '.status.containerStatuses[] | {name, image, imageID}' -kubectl -n "$OPERATOR_NAMESPACE" exec "$OPERATOR_POD" -c "$OPERATOR_CONTAINER" -- \ - gpustack-operator --version -``` - -Recorded binary output: - -```text -gpustack-operator version dev (d00f8fa8c98334a71af4c6eb302ee80d08289309) -``` +## Prerequisites -The `dev` tag can move. A matching tag string alone does not identify the binary under test. -Confirm eight free GPUs, model-manager readiness, and enough disk for the weights before starting. +Complete [cluster and GPU preparation](_index.md#prepare-the-cluster) and +[model prefetch](../model-delivery/prefetch.md). +The [Elastic EP guide](../../modules/model-deployment/elastic-ep.md) owns the profile's configuration and limits. -```bash -kubectl get nodes -kubectl get devices -o json -kubectl get instancetypes -kubectl get nodemodelstores -o json -``` - -## Prefetch the weights - -Save the following as `model-delivery.yaml`. Replace both `gpu-node` values with your GPU node's -`kubernetes.io/hostname` label. The store grant and the prefetch placement must select that same node. -The fixed file list resolves to 12 files and 31,418,807,311 bytes. - -```yaml -apiVersion: v1 -kind: Namespace -metadata: - name: gpustack-elastic ---- -apiVersion: worker.gpustack.ai/v1 -kind: ModelArtifact -metadata: - name: deepseek-v2-lite-chat - namespace: gpustack-elastic -spec: - source: - huggingFace: - repository: deepseek-ai/DeepSeek-V2-Lite-Chat - revision: 85864749cd611b4353ce1decdb286193298f64c7 - allowPatterns: - - model-00001-of-000004.safetensors - - model-00002-of-000004.safetensors - - model-00003-of-000004.safetensors - - model-00004-of-000004.safetensors - - config.json - - configuration_deepseek.py - - generation_config.json - - model.safetensors.index.json - - modeling_deepseek.py - - tokenization_deepseek_fast.py - - tokenizer.json - - tokenizer_config.json ---- -apiVersion: worker.gpustack.ai/v1 -kind: ModelStore -metadata: - name: rtx6000-models -spec: - nodeSelector: - matchLabels: - kubernetes.io/hostname: gpu-node ---- -apiVersion: worker.gpustack.ai/v1 -kind: ModelStoreBinding -metadata: - name: rtx6000-models - namespace: gpustack-elastic -spec: - storeRefs: - - name: rtx6000-models - quota: - bytes: 64Gi - allowPinned: true ---- -apiVersion: worker.gpustack.ai/v1 -kind: ModelPrefetch -metadata: - name: deepseek-v2-lite-chat - namespace: gpustack-elastic -spec: - artifactRef: - name: deepseek-v2-lite-chat - bindingRef: - name: rtx6000-models - placement: - nodeSelector: - matchLabels: - kubernetes.io/hostname: gpu-node - minReady: 1 - retention: - pinned: true -``` - -```bash -kubectl apply -f model-delivery.yaml -kubectl -n gpustack-elastic get modelartifacts,modelprefetches -kubectl -n gpustack-elastic get modelprefetch deepseek-v2-lite-chat -o yaml -kubectl get nodemodelstores -o json -``` - -Captured `kubectl get modelartifacts,modelprefetches` output: - -```text -NAME SOURCE REVISION SIZE READY DOWNLOADING RESOLVED -modelartifact.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-ai/DeepSeek-V2-Lite-Chat 85864749cd61 31418807311 1 0 True - -NAME ARTIFACT BINDING READY DESIRED AVAILABLE -modelprefetch.worker.gpustack.ai/deepseek-v2-lite-chat deepseek-v2-lite-chat rtx6000-models 1 1 True -``` - -Check the matching node's model entry as well: - -```bash -kubectl get nodemodelstores -o json | - jq '.items[] | .status.models[]? | {digest, sizeBytes, state}' -``` - -Captured projection: - -```json -{ - "digest": "sha256:1369aae0b5cc2d0e260557bcea04b0d6aad40f8bb6accdac60f1ee543887d9f7", - "sizeBytes": 31418807311, - "state": "Ready" -} -``` - -The validation also retained the successful warm-up Pod and compared its 12 SHA-256 results with -the resolved manifest. Warm-up Pods are transient; collect their logs before completion if you need -that evidence. See [Model Prefetch](../model-delivery/prefetch.md). -Wait for the required content before creating the serving workload. +Keep `replicas: 1`, `size: 1`, and TP=2 throughout this walkthrough. +Width 2 consumes four whole GPUs. Width 4 consumes eight. +This release supports width increases; width decreases are refused. +The engine uses Internal LB, and only its master accepts inference requests. -## Start the elastic deployment +## Step 1: start the elastic deployment -Save this as `elastic.yaml`. The InstanceType matches the recorded RTX PRO 6000 hardware. -For another GPU family, use an appropriate existing whole-GPU InstanceType and verify model compatibility. -Each GPU member requests two GPUs; its CPU and memory budgets scale with that count. +Save this as `elastic.yaml`. Use the artifact and InstanceType from the preparation steps. ```yaml apiVersion: worker.gpustack.ai/v1 -kind: InstanceType -metadata: - name: elastic-cpu-gpu -spec: - generalGroup: generic - acceleratable: true - acceleratorGroup: nvidia-rtx-pro-6000-blackwell-server-edition - os: linux - arch: amd64 - localStorage: 10Gi - unitResources: - cpu: "4" - ram: 16Gi ---- -apiVersion: worker.gpustack.ai/v1 kind: ModelDeployment metadata: name: tp2-elastic @@ -283,7 +90,7 @@ In the recorded run, this head landed on the GPU node but claimed no GPU. The tw With `shmSize` omitted, the recorded `/dev/shm` capacity was 16 GiB on every elastic Pod. Ray's object store used that mount. A separate CPU-only probe verified explicit `shmSize: 32Gi`. Kubelet can cap the mount by node allocatable memory or Pod memory limits; read the actual mount. -See [Shared memory](deployment.md#shared-memory) before changing the budget. +See [Shared memory](../../modules/model-deployment/deployment.md#shared-memory) before changing the budget. The recorded test also installed a read-only diagnostic extension to measure native parallel groups. That test instrumentation is omitted from this application manifest. @@ -302,10 +109,10 @@ router: ``` The engine still uses Internal LB. This Router forwards to the master, which distributes work to the DP engines. -Use the [Router observation steps](external-dp-walkthrough.md#check-router-coverage) with deployment name `tp2-elastic`. +Use the [Router observation steps](external-dp.md#step-4-check-router-coverage) with deployment name `tp2-elastic`. For this Elastic shape, expect one master endpoint; exclude the CPU head and other GPU members. -## Check the baseline +## Step 2: verify baseline inference Save each CR snapshot before changing the width: @@ -380,7 +187,7 @@ Require HTTP 200, nonempty generated text, positive completion tokens, and a fin For SSE, require the usage event and `data: [DONE]`. A health response does not exercise generation. The recorded requests used unique request IDs and checked their response IDs as well. -## Expand the collective +## Step 3: expand and verify inference Record the master Pod's UID and restart count before the patch. Select ordinal zero within this deployment: @@ -445,31 +252,42 @@ Confirm four GPU members and one head. Read the master and Ray logs, then repeat The engine may withdraw readiness and return 503 during reconfiguration. Do not resend the width change while the current operation is unresolved. -The recorded result combined native group measurements, actor state, GPU allocation, and completed requests: +## Step 4: release the application -| Check | Width 2 | Width 4 | -|---|---|---| -| Actual TP / DP / EP world | 2 / 2 / 4 | 2 / 4 / 8 | -| Distinct allocated GPU UUIDs | 4 | 8 | -| Live EngineCore actors | 2 | 4 | -| Live GPU worker actors | 4 | 8 | -| Ordinary request | HTTP 200, 8 generated tokens | HTTP 200, 8 generated tokens | -| SSE request | HTTP 200, 8 generated tokens, `[DONE]` | HTTP 200, 8 generated tokens, `[DONE]` | +When you have finished inspecting the elastic application, delete only its ModelDeployment: -All four recorded requests succeeded on their first attempt. The original master UID and HTTP process -identity remained unchanged. The measurements joined native workers to Ray actors, Pod identities, -and host GPU UUIDs. Placement groups used `STRICT_PACK`, with two GPU bundles per DP engine. +```bash +kubectl -n gpustack-elastic delete modeldeployment tp2-elastic --wait=true +kubectl -n gpustack-elastic get modeldeployments,pods,services +kubectl get devices -o json > released-devices.json +jq '[.items[].status.groups[].accelerators[] | + {mode: (.mode // 0), remaining, allocatedSlices: (.allocatedSlices // 0)}] | + {cards: length, free: map(select(.mode == 0 and .remaining == 1600000 and .allocatedSlices == 0)) | length}' \ + released-devices.json +``` -The additional Router run also completed width 2→4 with the same official runner. -Before and after expansion, its selector and registry contained only the original master. -Each stage completed one ordinary request and one SSE request through the Router, with HTTP 200 and eight tokens. -The master's Router counter increased by two at each stage. Worker and CPU-head endpoints were absent. +Captured projection after deleting the official-runner Elastic application with its Router: -These checks do not establish sustained concurrency, uninterrupted requests throughout resizing, -per-rank request coverage, or inference quality. The response's `system_fingerprint` retained its startup DP2 text after -expansion, so it was not used to establish the effective width. +```json +{ + "cards": 8, + "free": 8 +} +``` + +That release check also verified deletion of the owned head, GPU Pods, Router, Services, and Workload. +For a shared cluster, compare the application's captured device claims with the ledger; +other workloads may keep cards occupied. + +ModelPrefetch keeps the model pinned independently of the serving application. +When no consumer needs the weights, remove the prefetch and its dedicated binding, artifact, and store. +See [Model Prefetch](../../modules/model-delivery/prefetch.md) for cache retention and collection behavior. + +Application deletion leaves the Kubernetes nodes, disks, and cloud network in place. +Retain them while the owner is inspecting logs. Delete infrastructure only after the owner approves, +using the provisioner's recorded resource ownership, and verify deletion in the provider and state file. -## Inspect a stalled resize +## Diagnose a stalled resize Read the CR conditions and the owned Pods first. Select the head, master, or worker from the Pod list: @@ -497,103 +315,61 @@ The operator defaults both levels to `info`. The manifest sets both to `debug` and disables deduplication for inspection. Environment changes affect newly created Pods. Preserve current logs before replacing a failing deployment. -An earlier run reached `CREATED` placement groups and eight live GPU workers, but new EngineCore actors -remained pending and requests timed out. This recorded run completed expansion and inference. -The earlier root cause remains unconfirmed; the result does not isolate shared memory or head placement as its cause. +## Validation record -## Scale independent replicas +The recorded software and model identities were: -Use a separate deployment without `elasticEp` when you need independent serving instances. -This alternative was not runtime-tested in the recorded elastic run. -First [release the elastic application](#release-the-application) so it does not consume all eight GPUs. -Keep the shared artifact, cache, and InstanceType. +| Component | Recorded identity | +|---|---| +| Operator | `gpustack/gpustack-operator:dev`, binary revision `d00f8fa8c98334a71af4c6eb302ee80d08289309` | +| Operator image digest | `sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36` | +| Inference engine | vLLM 0.29.0 with full Ray 2.54.0 | +| Engine image | `gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91` | +| Model | `deepseek-ai/DeepSeek-V2-Lite-Chat` | +| Model revision | `85864749cd611b4353ce1decdb286193298f64c7` | -Save this as `replicas.yaml`: +The official GPUStack runner includes Ray. The run used its installed packages without replacement. +Container reads found `cupy-cuda13x` 14.2.0 and no `cupy-cuda12x` installation. +Verify the image digest and installed vLLM/Ray versions when selecting another runner. +Do not add `--elastic-ep-max-dp-size` to this example; that argument starts with vLLM 0.30.0. -```yaml -apiVersion: worker.gpustack.ai/v1 -kind: ModelDeployment -metadata: - name: replica-demo - namespace: gpustack-elastic -spec: - model: - name: /models - artifactRef: - name: deepseek-v2-lite-chat - engine: - name: vLLM - version: 0.29.0 - roles: - - name: server - kind: Server - replicas: 1 - size: 1 - instanceType: elastic-cpu-gpu - image: gpustack/runner:cuda13.0-vllm0.29.0@sha256:1c826749ed16fbd9f9594d7a49f774904662d9c46231e08e32b494e8bac4ee91 - resources: - accelerator: "2" - extraArgs: - - --tensor-parallel-size - - "2" - - --max-model-len - - "4096" - - --enforce-eager -``` +Recorded operator binary output: -```bash -kubectl apply -f replicas.yaml -kubectl -n gpustack-elastic patch modeldeployment replica-demo --type=json \ - -p '[{"op":"replace","path":"/spec/roles/0/replicas","value":2}]' -kubectl -n gpustack-elastic get modeldeployment replica-demo -o yaml -kubectl -n gpustack-elastic get pods +```text +gpustack-operator version dev (d00f8fa8c98334a71af4c6eb302ee80d08289309) ``` -Two replicas each request two GPUs, for four GPUs total. Each replica has its own TP2 engine. -Read role readiness and endpoint eligibility, and send inference to each instance before declaring both usable. -Managed routing is a separate configuration; see [Routing](routing.md). -SGLang also supports independent replicas, with its own runner image and engine arguments. - -## Release the application - -When you have finished inspecting the elastic application, delete only its ModelDeployment: - -```bash -kubectl -n gpustack-elastic delete modeldeployment tp2-elastic --wait=true -kubectl -n gpustack-elastic get modeldeployments,pods,services -kubectl get devices -o json > released-devices.json -jq '[.items[].status.groups[].accelerators[] | - {mode: (.mode // 0), remaining, allocatedSlices: (.allocatedSlices // 0)}] | - {cards: length, free: map(select(.mode == 0 and .remaining == 1600000 and .allocatedSlices == 0)) | length}' \ - released-devices.json -``` +The recorded result combined native group measurements, actor state, GPU allocation, and completed requests: -The release check used an earlier test-runner deployment on the same otherwise idle node. -Captured projection after deleting the earlier application: +| Check | Width 2 | Width 4 | +|---|---|---| +| Actual TP / DP / EP world | 2 / 2 / 4 | 2 / 4 / 8 | +| Distinct allocated GPU UUIDs | 4 | 8 | +| Live EngineCore actors | 2 | 4 | +| Live GPU worker actors | 4 | 8 | +| Ordinary request | HTTP 200, 8 generated tokens | HTTP 200, 8 generated tokens | +| SSE request | HTTP 200, 8 generated tokens, `[DONE]` | HTTP 200, 8 generated tokens, `[DONE]` | -```json -{ - "cards": 8, - "free": 8 -} -``` +All four recorded requests succeeded on their first attempt. The original master UID and HTTP process +identity remained unchanged. The measurements joined native workers to Ray actors, Pod identities, +and host GPU UUIDs. Placement groups used `STRICT_PACK`, with two GPU bundles per DP engine. -That release check also verified deletion of the owned head, GPU Pods, Services, and Workload. -For a shared cluster, compare the application's captured device claims with the ledger; -other workloads may keep cards occupied. If you ran the replica alternative, delete `replica-demo` as well. +The additional Router run also completed width 2→4 with the same official runner. +Before and after expansion, its selector and registry contained only the original master. +Each stage completed one ordinary request and one SSE request through the Router, with HTTP 200 and eight tokens. +The master's Router counter increased by two at each stage. Worker and CPU-head endpoints were absent. -ModelPrefetch keeps the model pinned independently of the serving application. -When no consumer needs the weights, remove the prefetch and its dedicated binding, artifact, and store. -See [Model Prefetch](../model-delivery/prefetch.md) for cache retention and collection behavior. +These checks do not establish sustained concurrency, uninterrupted requests throughout resizing, +per-rank request coverage, or inference quality. The response's `system_fingerprint` retained its startup DP2 text after +expansion, so it was not used to establish the effective width. -Application deletion leaves the Kubernetes nodes, disks, and cloud network in place. -Retain them while the owner is inspecting logs. Delete infrastructure only after the owner approves, -using the provisioner's recorded resource ownership, and verify deletion in the provider and state file. +An earlier run reached `CREATED` placement groups and eight live GPU workers, but new EngineCore actors +remained pending and requests timed out. This recorded run completed expansion and inference. +The earlier root cause remains unconfirmed; the result does not isolate shared memory or head placement as its cause. --- -**See also** — [Elastic EP](elastic-ep.md) (profile contract) · -[Model Prefetch](../model-delivery/prefetch.md) (weight delivery) · -[GPU Instances](../instances/_index.md) (direct GPU requests) +**See also** — [Elastic EP](../../modules/model-deployment/elastic-ep.md) (profile contract) · +[External DP Walkthrough](external-dp.md) (fixed per-rank serving) -**Next** → [Model Deployment Status](status.md) — interpret conditions and serving endpoints. +**Next** → [Model Deployment Status](../../modules/model-deployment/status.md) diff --git a/docs/modules/model-deployment/external-dp-walkthrough.md b/docs/walkthroughs/model-deployment/external-dp.md similarity index 62% rename from docs/modules/model-deployment/external-dp-walkthrough.md rename to docs/walkthroughs/model-deployment/external-dp.md index 5090d308b..27e6f53ee 100644 --- a/docs/modules/model-deployment/external-dp-walkthrough.md +++ b/docs/walkthroughs/model-deployment/external-dp.md @@ -2,22 +2,22 @@ Run one fixed vLLM data-parallel (DP) group with an HTTP endpoint on each rank. This walkthrough records two ranks using the official GPUStack runner. -Direct requests passed on both ranks. The recorded operator's managed Router discovered only rank 0. -That routing limitation remains visible in the checks below. +Direct requests and managed Router requests passed on both ranks after the selector fix. +Follow the steps to verify both engine inference and Router coverage. ## Contents -- [Choose the deployment shape](#choose-the-deployment-shape) -- [Prepare the model and GPUs](#prepare-the-model-and-gpus) -- [Start the fixed group](#start-the-fixed-group) -- [Omit redundant arguments](#omit-redundant-arguments) -- [Read the running resources](#read-the-running-resources) -- [Send requests to each rank](#send-requests-to-each-rank) -- [Check Router coverage](#check-router-coverage) +- [Deployment shape](#deployment-shape) +- [Prerequisites](#prerequisites) +- [Step 1: start the fixed group](#step-1-start-the-fixed-group) +- [Step 2: read the running resources](#step-2-read-the-running-resources) +- [Step 3: send requests to each rank](#step-3-send-requests-to-each-rank) +- [Step 4: check Router coverage](#step-4-check-router-coverage) - [Understand the scaling limits](#understand-the-scaling-limits) -- [Inspect and release the application](#inspect-and-release-the-application) +- [Step 5: inspect and release the application](#step-5-inspect-and-release-the-application) +- [Validation record](#validation-record) -## Choose the deployment shape +## Deployment shape This example uses one Server role with `replicas: 1` and `size: 2`. Each Pod has two GPUs and runs one DP rank with tensor parallelism (TP)=2. @@ -33,34 +33,18 @@ The runner contains Ray, but this fixed group does not start Ray. See [vLLM's DP deployment guide](https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment/) for the engine's Internal and External LB distinction. -## Prepare the model and GPUs +## Prerequisites -Complete [cluster preparation](scaling-walkthrough.md#prepare-the-cluster) and -[model prefetch](scaling-walkthrough.md#prefetch-the-weights) first. -Use the same namespace, cached artifact, model revision, and GPU InstanceType. +Complete [cluster and GPU preparation](_index.md#prepare-the-cluster) and +[model prefetch](../model-delivery/prefetch.md). +Use the shared namespace, cached artifact, model revision, and GPU InstanceType. The GPU node needs four free cards and the verified model cache. -For the recorded hardware, copy the `InstanceType` document from -[the elastic deployment example](scaling-walkthrough.md#start-the-elastic-deployment) -into `external-instance-type.yaml`. Apply that single resource before creating this deployment: - -```bash -kubectl apply -f external-instance-type.yaml -kubectl get instancetype elastic-cpu-gpu -``` - -For another GPU family, select an existing whole-GPU InstanceType and replace the name in the manifest below. - -The recorded operator binary was `d00f8fa8c98334a71af4c6eb302ee80d08289309` from -`gpustack/gpustack-operator:dev@sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36`. -The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements. -Its CuPy installation contained only `cupy-cuda13x` 14.2.0. - Each Pod needs a separate Pod IP. The ranks must reach each other over the Pod network. The DP RPC port defaults to 29550; collective communication also uses other ports. Setting an engine RPC port does not create a Kubernetes Service or a firewall rule. -## Start the fixed group +## Step 1: start the fixed group Save this as `external-dp.yaml`. Replace `elastic-cpu-gpu` with your GPU InstanceType if needed. The model delivery steps already created `deepseek-v2-lite-chat` in `gpustack-elastic`. @@ -69,7 +53,7 @@ The model delivery steps already created `deepseek-v2-lite-chat` in `gpustack-el apiVersion: worker.gpustack.ai/v1 kind: ModelDeployment metadata: - name: external-minimal + name: external-fixed namespace: gpustack-elastic spec: model: @@ -128,12 +112,12 @@ That test instrumentation is omitted from this serving manifest. ```bash kubectl apply --dry-run=server -f external-dp.yaml kubectl apply -f external-dp.yaml -kubectl -n gpustack-elastic get modeldeployment external-minimal +kubectl -n gpustack-elastic get modeldeployment external-fixed kubectl -n gpustack-elastic get pods \ - -l app.kubernetes.io/instance=external-minimal + -l app.kubernetes.io/instance=external-fixed ``` -## Omit redundant arguments +### Arguments you can omit The minimal run omitted all six arguments below and still completed ordinary and streaming requests on both ranks. These defaults apply to this vLLM version and deployment shape. @@ -148,16 +132,22 @@ These defaults apply to this vLLM version and deployment shape. | `--data-parallel-rpc-port 13345` | Both ranks can use the default RPC port, 29550. | The operator generates the ordinary HTTP serving port, model mount, Pod wiring, and Services. -This example serves each rank on port 8000. +This example serves each rank on the same fixed port, 8000. + +A single API server process and a single HTTP port are different constraints. +Internal LB may use several API server processes behind one port. +Keep the engine default for Internal LB; do not force `--api-server-count 1` on every configuration. +Managed roles reject MultiPort mode because its additional listeners are outside this endpoint contract. + `--data-parallel-backend` selects execution placement; it does not select the LB mode. `--data-parallel-start-rank` alone is not an External LB declaration. -## Read the running resources +## Step 2: read the running resources Retain the CR output beside the request results: ```bash -kubectl -n gpustack-elastic get modeldeployment external-minimal -o json | +kubectl -n gpustack-elastic get modeldeployment external-fixed -o json | jq '{name: .metadata.name, phase: .status.phase, size: .spec.roles[0].size, replicas: .spec.roles[0].replicas, accelerator: .spec.roles[0].resources.accelerator}' @@ -167,7 +157,7 @@ Recorded values from the captured CR: ```json { - "name": "external-minimal", + "name": "external-fixed", "phase": "Ready", "size": 2, "replicas": 1, @@ -175,11 +165,32 @@ Recorded values from the captured CR: } ``` +The captured status also reported one ready replica group and two serving HTTP endpoints: + +```bash +kubectl -n gpustack-elastic get modeldeployment external-fixed -o json | + jq '.status.roles[] | {name, ready, loadBalance: .parallelism.loadBalance, endpoints}' +``` + +```json +{ + "name": "server", + "ready": 1, + "loadBalance": "External", + "endpoints": { + "eligible": 1, + "serving": {"state": "Confirmed", "value": 2} + } +} +``` + +Here `eligible` counts replica groups; the serving observation counts HTTP endpoints. +Keep the full CR with `kubectl -n gpustack-elastic get modeldeployment external-fixed -o yaml`. Read Pod UIDs, images, addresses, and endpoint labels separately: ```bash kubectl -n gpustack-elastic get pods \ - -l app.kubernetes.io/instance=external-minimal -o json | + -l app.kubernetes.io/instance=external-fixed -o json | jq '.items[] | {name: .metadata.name, uid: .metadata.uid, labels: .metadata.labels, podIP: .status.podIP, containers: .status.containerStatuses}' @@ -196,16 +207,16 @@ The recorded native checks found: All four worker GPU identities were distinct. Both ranks reported External LB enabled and Elastic EP disabled. Declared arguments and `Ready` status alone do not establish these native results. -## Send requests to each rank +## Step 3: send requests to each rank Open one port-forward per rank in separate terminals: ```bash -kubectl -n gpustack-elastic port-forward pod/external-minimal-server-r0-m0 18000:8000 +kubectl -n gpustack-elastic port-forward pod/external-fixed-server-r0-m0 18000:8000 ``` ```bash -kubectl -n gpustack-elastic port-forward pod/external-minimal-server-r0-m1 18001:8000 +kubectl -n gpustack-elastic port-forward pod/external-fixed-server-r0-m1 18001:8000 ``` Send an ordinary request and a streaming request to each endpoint: @@ -226,13 +237,13 @@ Both ranks returned HTTP 200, nonempty text, eight completion tokens, and a vali Both streams ended with `data: [DONE]`. These short checks establish functional inference, not throughput or sustained-load behavior. -## Check Router coverage +## Step 4: check Router coverage Discover the Router Pod and forward its API and metrics ports: ```bash ROUTER_POD=$(kubectl -n gpustack-elastic get pods \ - -l app.kubernetes.io/instance=external-minimal -o json | + -l app.kubernetes.io/instance=external-fixed -o json | jq -r '.items[] | select(any(.spec.containers[]; .name == "router")) | .metadata.name') kubectl -n gpustack-elastic port-forward "pod/$ROUTER_POD" 18081:8081 19090:9090 ``` @@ -250,21 +261,19 @@ Repeat the completion requests through port 18081. Compare the counters before a For complete External LB coverage, the registry must contain both rank endpoints. Requests must increase each rank's counter. A successful response through the Router is insufficient. -The recorded Router registered only member 0 because its selector retained `member-index=0`. -All four Router requests completed, but all four increased only member 0's counter. -Member 1 received no Router traffic. Direct rank inference passed; managed Router coverage failed. -Keep this distinction when evaluating the recorded operator revision. +The corrected run registered both healthy endpoints and included both in selection. +Four Router requests passed on their first attempt: two ordinary requests and two streams. +Each rank's request counter increased by two. The [validation record](#validation-record) compares the original and corrected runs. -For ordinary Internal groups and [P/D deployments](prefill-decode.md), the leader restriction remains necessary. -External DP inside a Prefill or Decode role is deferred to -[issue #754](https://github.com/gpustack/gpustack-operator/issues/754). +For ordinary Internal groups and [P/D deployments](../../modules/model-deployment/prefill-decode.md), the leader restriction remains necessary. +See [Prefill and Decode](../../modules/model-deployment/prefill-decode.md) for the deferred External DP combination and its tracking issue. This Server example does not validate that combination. ## Understand the scaling limits This fixed External DP group has no online DP/EP resize contract. vLLM 0.29.0 rejects combining its native Elastic EP mode with External or Hybrid LB. -Use the [Ray Internal Elastic walkthrough](scaling-walkthrough.md) for `elasticEp.width` growth. +Use the [Ray Internal Elastic walkthrough](elastic-ep.md) for `elasticEp.width` growth. A Router in front of that Internal endpoint does not turn the engine into External LB. Increasing `roles[].replicas` creates additional complete groups with their existing TP/DP/EP shape. @@ -272,14 +281,14 @@ That operation does not enlarge an existing group's EP world and was not measure The ordinary role's `size` cannot be changed after creation. Hybrid and multiple HTTP ports per Pod are outside this walkthrough. -## Inspect and release the application +## Step 5: inspect and release the application Read both ranks' logs when startup or collective requests stall: ```bash -kubectl -n gpustack-elastic logs external-minimal-server-r0-m0 -c main -kubectl -n gpustack-elastic logs external-minimal-server-r0-m1 -c main -kubectl -n gpustack-elastic get modeldeployment external-minimal -o yaml +kubectl -n gpustack-elastic logs external-fixed-server-r0-m0 -c main +kubectl -n gpustack-elastic logs external-fixed-server-r0-m1 -c main +kubectl -n gpustack-elastic get modeldeployment external-fixed -o yaml kubectl -n gpustack-elastic get services,endpointslices ``` @@ -290,17 +299,50 @@ The replica's headless Service supplies peer DNS; narrowing it to the HTTP leade After saving the results, stop the port-forward processes and delete this application: ```bash -kubectl -n gpustack-elastic delete modeldeployment external-minimal +kubectl -n gpustack-elastic delete modeldeployment external-fixed kubectl -n gpustack-elastic get pods,services,deployments \ - -l app.kubernetes.io/instance=external-minimal + -l app.kubernetes.io/instance=external-fixed kubectl get devices -o json ``` Verify that its owned workloads are gone and all four GPU claims are released. -The recorded cleanup deleted both External test applications and returned all eight cards to their free state. +In the final corrected run, engine Pods disappeared and all eight cards returned to their free state. +The Router and Services remained after about two minutes of observation. +Their deletion required explicit cleanup after confirming their owner UID. + +Automatic dependent cleanup was not established by that run; its cause remains unconfirmed. +[Issue #601](https://github.com/gpustack/gpustack-operator/issues/601) tracks related cleanup delays after worker outages; this run did not establish that cause. + Deleting the application retains the model cache and cloud cluster. +## Validation record + +The initial operator binary was `d00f8fa8c98334a71af4c6eb302ee80d08289309` from +`gpustack/gpustack-operator:dev@sha256:84f85914bdcb398b2f662dfdcb0845ee853597ec5661a5cf41828acae6a11e36`. +The corrected run used binary `20d674b7f8c1e1d4c22b91a09fd178b9738d70be` from +`thxcode/gpustack-operator:dev-20d674b7@sha256:43c14087fd3391e4bf4a3327f246053d9e59b2feb684cb792f63a2f0db659fbd`. +That test image replaced the operator binary on the pinned official base. It retained the packaged vendor assets. + +The official runner below contained vLLM 0.29.0 and Ray 2.54.0, with no package replacements. +Its CuPy installation contained only `cupy-cuda13x` 14.2.0. + +The initial operator retained `member-index=0` and registered only member 0. +All four Router requests completed, but all four increased only member 0's counter. +After the fix, discovery registered both healthy endpoints and included both in selection. +Four Router requests completed on their first attempt: two ordinary and two streaming requests. +Each returned eight completion tokens; both streams ended with `data: [DONE]`. + +| Operator run | Registered ranks | Rank 0 counter increase | Rank 1 counter increase | Coverage | +|---|---|---|---|---| +| Initial official dev | 0 | 4 | 0 | Failed | +| Selector fix | 0, 1 | 2 | 2 | Passed | + +The correction removed the leader restriction only from eligible External Server endpoints. +Both engine Pods remained running through the Operator upgrade. The Router Pod restarted with its corrected selector. + --- -**See also** — [Model Scaling Walkthrough](scaling-walkthrough.md) (prefetch and Elastic EP) · -[Routing](routing.md) (backend selection) · [Model Deployment Configuration](deployment.md) (role fields) +**See also** — [Elastic EP Walkthrough](elastic-ep.md) (collective expansion) · +[Routing](../../modules/model-deployment/routing.md) (backend selection) · [Model Deployment Configuration](../../modules/model-deployment/deployment.md) (role fields) + +**Next** → [Model Deployment Status](../../modules/model-deployment/status.md) diff --git a/site/content/docs/_index.md b/site/content/docs/_index.md index d6fc2db8d..80d976a94 100644 --- a/site/content/docs/_index.md +++ b/site/content/docs/_index.md @@ -10,7 +10,7 @@ Install the operator, configure a capability, or find the reference for a runnin - [Architecture](getting-started/architecture/) explains the components and scheduling chain. - [Vendor Prerequisites](getting-started/vendor-prerequisites/) lists the drivers and runtime setup. -- [Walkthrough](getting-started/walkthrough/) follows a request through the operator. +- [Walkthroughs](walkthroughs/) groups complete examples by module. ## Capabilities diff --git a/site/data/navigation.yaml b/site/data/navigation.yaml index f0c1c1d6f..6e1e4ef2d 100644 --- a/site/data/navigation.yaml +++ b/site/data/navigation.yaml @@ -1,6 +1,8 @@ groups: - name: Getting started path: /docs/getting-started/ + - name: Walkthroughs + path: /docs/walkthroughs/ - name: Devices Manager path: /docs/modules/devices/ - name: RDMA Manager @@ -23,7 +25,20 @@ order: /docs/getting-started/: - architecture - vendor-prerequisites - - walkthrough + /docs/walkthroughs/: + - devices + - kv-cache + - model-delivery + - model-deployment + /docs/walkthroughs/devices/: + - scheduling + /docs/walkthroughs/kv-cache/: + - shared-cache + /docs/walkthroughs/model-delivery/: + - prefetch + /docs/walkthroughs/model-deployment/: + - elastic-ep + - external-dp /docs/modules/devices/: - preflight - discovery @@ -40,7 +55,6 @@ order: - scheduling - operations /docs/modules/kv-cache/: - - walkthrough - backend - pool - injection @@ -58,8 +72,6 @@ order: /docs/modules/model-deployment/: - deployment - elastic-ep - - scaling-walkthrough - - external-dp-walkthrough - engine-versions - prefill-decode - routing @@ -84,9 +96,18 @@ order: - development - internals labels: + /docs/walkthroughs/: Walkthroughs + /docs/walkthroughs/devices/: Devices + /docs/walkthroughs/devices/scheduling/: Scheduling + /docs/walkthroughs/kv-cache/: KV Cache + /docs/walkthroughs/kv-cache/shared-cache/: Shared cache + /docs/walkthroughs/model-delivery/: Model Delivery + /docs/walkthroughs/model-delivery/prefetch/: Model prefetch + /docs/walkthroughs/model-deployment/: Model Deployment + /docs/walkthroughs/model-deployment/elastic-ep/: Elastic EP + /docs/walkthroughs/model-deployment/external-dp/: External DP /docs/getting-started/architecture/: Architecture /docs/getting-started/vendor-prerequisites/: Prerequisites - /docs/getting-started/walkthrough/: Walkthrough /docs/modules/devices/preflight/: Preflight /docs/modules/devices/discovery/: Discovery /docs/modules/devices/requests/: Resource requests @@ -99,7 +120,6 @@ labels: /docs/modules/rdma/operations/: Operations /docs/modules/topology/scheduling/: Scheduling /docs/modules/topology/operations/: Operations - /docs/modules/kv-cache/walkthrough/: Walkthrough /docs/modules/kv-cache/backend/: Backends /docs/modules/kv-cache/pool/: Pools and bindings /docs/modules/kv-cache/injection/: Pod injection @@ -115,8 +135,6 @@ labels: /docs/modules/model-delivery/views/: API reference /docs/modules/model-deployment/deployment/: Configuration /docs/modules/model-deployment/elastic-ep/: Elastic EP - /docs/modules/model-deployment/scaling-walkthrough/: Scaling walkthrough - /docs/modules/model-deployment/external-dp-walkthrough/: External DP walkthrough /docs/modules/model-deployment/engine-versions/: Engine versions /docs/modules/model-deployment/prefill-decode/: Prefill and decode /docs/modules/model-deployment/routing/: Routing diff --git a/site/layouts/index.html b/site/layouts/index.html index 9253cd98a..6a9633ba7 100644 --- a/site/layouts/index.html +++ b/site/layouts/index.html @@ -5,7 +5,7 @@

GPUStack Operator

Start here

Architecture

See how discovery, capacity, and admission fit together.

-
Walkthrough

Follow an accelerator request through a working cluster.

+
Walkthroughs

Run a complete example and inspect its results.

Installation modes

Choose how the operator and its dependencies are deployed.

Capabilities

diff --git a/specs/2026-10-01-s1-router-qualification-and-drain.md b/specs/2026-10-01-s1-router-qualification-and-drain.md index 5eb25c353..a60fb9998 100644 --- a/specs/2026-10-01-s1-router-qualification-and-drain.md +++ b/specs/2026-10-01-s1-router-qualification-and-drain.md @@ -117,6 +117,13 @@ map; internal headless replica Services keep selecting every member with `publishNotReadyAddresses` untouched. The existing ready-instance status fields are preserved unchanged. +**Corrected after shipping.** For qualified External Server roles, ordinary Services and Router discovery remove the legacy +`member-index=0` equality and retain endpoint eligibility. Internal and P/D roles keep the leader +restriction. Known non-Internal modes use the External selection rule; unknown modes do not expand +selection. Hybrid runtime validation and External DP inside P/D remain separate follow-up work. +Managed roles retain one HTTP port per Pod. MultiPort mode is refused because discovery cannot +represent its additional listeners. + - **Backfill before narrowing.** Before any Service or Router selector gains the key, the reconciler backfills it onto every existing healthy qualifying endpoint — no window in which a new selector term matches nothing and healthy endpoints disappear from selection. A static