Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .agents/skills/gpustack-operator-benchmark/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
---
name: gpustack-operator-benchmark
description: "Benchmark ModelDeployment serving topologies on a GPU cluster: warm NodeModelStore cache, drive in-cluster load across EPD, P/D, Elastic EP, and External DP, and collect stage and transfer evidence."
description: "Benchmark the operator on a GPU cluster — today the ModelDeployment serving topologies: warm NodeModelStore cache, drive in-cluster load across EPD, P/D, Elastic EP, and External DP, and collect stage and transfer evidence."
disable-model-invocation: true
allowed-tools: "Read, Agent, Bash(bash .claude/skills/_e2e-lib/scripts/preflight.sh*), Bash(bash .claude/skills/_e2e-lib/scripts/assert-core.sh*), Bash(bash .claude/skills/_e2e-lib/scripts/cluster-auth.sh*), Bash(bash .claude/skills/_e2e-lib/scripts/kube-context.sh*), Bash(bash .claude/skills/gpustack-operator-benchmark/cases/*), Bash(kubectl get*), Bash(kubectl describe*), Bash(kubectl logs*), Bash(kubectl cluster-info*), Bash(kubectl version*), Bash(kubectl config current-context), Bash(kubectl config get-contexts*), Bash(git diff*), Bash(git rev-parse*), Bash(command -v*), Bash(date*), Bash(mkdir -p .claude/reports/*), Bash(tee .claude/reports/*)"
---

# GPUStack Operator — ModelDeployment Serving Benchmark
# GPUStack Operator — Benchmark

Use [the shared module map](../../../docs/README.md#module-map) to locate the relevant guides,
owning specs, code entry points and related modules. [AGENTS.md](../../../AGENTS.md#finding-sources)
owns the retrieval and evidence rules; this skill owns its execution procedure.

Measure and compare the four `ModelDeployment` serving topologies on a real multi-GPU cluster (8 × GPU baseline) and collect both end-to-end latency/throughput distributions (`TTFT`, `ITL`, `TPOT`, `E2E`, `output_tok/s`, `req/s`) and per-stage execution / KV-transfer / EC-transfer evidence:
Measure and compare the `ModelDeployment` serving scenarios on a real multi-GPU cluster (8 × GPU baseline) and collect both end-to-end latency/throughput distributions (`TTFT`, `ITL`, `TPOT`, `E2E`, `output_tok/s`, `req/s`) and per-stage execution / KV-transfer / EC-transfer evidence. The scenarios below are the suite's first five; a scenario outside serving joins this suite, not a new skill:

- **Scenario 1 (`cases/case-1-epd.sh`)** — [Encode, Prefill and Decode](../../../docs/walkthroughs/model-deployment/encode-prefill-decode.md) ([benchmark](../../../docs/modules/model-deployment/encode-prefill-decode-benchmark.md)): 3-arm comparison (`Aggregated` vs `P/D` vs `E/P/D`) on a 3-GPU budget with `Qwen/Qwen2.5-VL-7B-Instruct` (`NIXL` encoder cache + `Mooncake` KV cache).
- **Scenario 2 (`cases/case-2-prefill-decode.sh`)** — [Prefill and Decode](../../../docs/walkthroughs/model-deployment/prefill-decode.md) ([benchmark](../../../docs/modules/model-deployment/prefill-decode-benchmark.md)): 2-arm comparison (`Aggregated` 3 × `Server(TP=2)` vs `P/D` 1 × `Prefill(TP=2)` + 2 × `Decode(TP=2)`) on a 6-GPU budget with `Qwen/Qwen3-32B`.
Expand Down
43 changes: 22 additions & 21 deletions .agents/skills/gpustack-operator-e2e/SKILL.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .agents/skills/gpustack-operator-e2e/cases/case-100.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# CASE 100 — The weight identity in the KV store key: one artifact's deployments share blocks
# through the store, and a deployment of another commit on the same Binding reads none
# (MUTATING, cluster-scoped store objects; needs real accelerators)
# (MUTATING, self-cleaning: cluster-scoped store objects; needs real accelerators)
#
# case-100.sh <NS>
#
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/gpustack-operator-e2e/cases/case-116.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
#
# CASE 116 — Every encoder-role admission refusal fires, each naming what its rule refuses
# (MUTATING for the fixture only: one settings key and one injected node label, both restored by
# (MUTATING, self-recovering: one settings key and one injected node label, both restored by
# the trap; every verdict below is a server-side dry run that persists nothing)
# AUTO-SKIPS its matrix rows without an InstanceType that reports an admitted accelerator
# platform, which this case builds by injecting a CPU vendor label.
Expand Down
3 changes: 2 additions & 1 deletion .agents/skills/gpustack-operator-e2e/cases/case-123.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@
#
# CASE 123 — vllm-router admits vLLM encoder deployments on NVIDIA and AMD, refuses unadmitted
# engines and user-supplied --epd-config
# (NON-MUTATING on GPU clusters; fixture-mutating on CPU-only clusters, restored by trap)
# (MUTATING, self-recovering: the platform fixture on CPU-only clusters, restored by
# the trap — GPU clusters need no fixture, and every verdict is a server-side dry run)
# AUTO-SKIPS the platform rows without an InstanceType reporting nvidia or amd
#
# case-123.sh <NS>
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/gpustack-operator-e2e/cases/case-18.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# CASE 18 — CPU-manufacturer awareness reshapes the catalog (MUTATING: flips an editable setting, self-restoring)
# CASE 18 — CPU-manufacturer awareness reshapes the catalog (MUTATING, self-recovering: flips an editable setting)
#
# case-18.sh <NS>
#
Expand Down
3 changes: 3 additions & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-42.sh
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@
# one-slice partitions requested together, then one two-accelerator request expected to
# be refused.
# Verdicts: One row per contract above, plus the setup gates. Any FAIL fails the case.
# Expected: Every contract in the Goal holds from both sides — the operator's records and the
# container or the vendor's own registry — and the table names both sides per row.
# Cleanup: An EXIT trap deletes the case's Pods and waits for their release.
set -uo pipefail

CASE_ID=42
Expand Down
7 changes: 5 additions & 2 deletions .agents/skills/gpustack-operator-e2e/cases/case-45.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
#
# CASE 45 — The ModelDeployment admission surface: every refusal fires, and each one fires from the
# layer that is supposed to own it
# (NON-MUTATING for the refusals, which are server-side dry-runs; one short-lived object for the
# controller-level check)
# (MUTATING, self-cleaning: the refusals are server-side dry-runs; one short-lived object
# serves the controller-level check)
#
# case-45.sh <NS>
#
Expand Down Expand Up @@ -74,6 +74,9 @@
# the behaviour is missing, but that asserting it would not DISCRIMINATE. See the row
# about what the unregistered domain costs, near the end.
#
# Expected: Every refusal fires from the layer the Goal names, the field and message asserted
# per row; the created-object row observes creation before it judges.
# Cleanup: An EXIT trap deletes the one short-lived deployment (case45-nobind).
set -o pipefail

NS="${1:-}"
Expand Down
10 changes: 8 additions & 2 deletions .agents/skills/gpustack-operator-e2e/cases/case-47.sh
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
#
# CASE 47 — Two deployments on one Binding share a reuse domain; two on different Bindings do not
# (MUTATING: three ModelDeployments, one extra Binding and one probe Pod, all in the fixture's
# namespace, all removed by the shared teardown)
# (MUTATING, self-cleaning: three ModelDeployments, one extra Binding and one probe Pod, all
# removed by the shared teardown)
#
# case-47.sh <NS>
#
Expand Down Expand Up @@ -43,6 +43,12 @@
# image from. No GPU, no engine image.
# Needs: a live Mooncake store (the shared fixture brings up backend, pool and Bindings)
#
# Inputs: All real, nothing mocked: the fixture's pool and Bindings, one additional Binding in
# the same namespace, and the engine image the deployments render.
# Expected: Deployments on the same Binding render the same reuse domain and the third
# deployment a different one; the probe writes land under the rendered values.
# Cleanup: The EXIT teardown deletes the three deployments, the extra Binding and the probe
# Pod; the shared fixture's teardown covers the rest.
set -uo pipefail

NS="${1:?usage: case-47.sh <NS>}"
Expand Down
11 changes: 9 additions & 2 deletions .agents/skills/gpustack-operator-e2e/cases/case-48.sh
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
#
# CASE 48 — The deliberate break: what a deployment looks like when the cache cannot work
# (MUTATING: one external KVCacheBackend and one KVCachePool, both CLUSTER-SCOPED, plus one
# KVCachePoolBinding and two ModelDeployments in the fixture's namespace — all removed here)
# (MUTATING, self-cleaning: one external KVCacheBackend and one KVCachePool, both
# cluster-scoped, plus one KVCachePoolBinding and two ModelDeployments — all removed here)
#
# case-48.sh <NS>
#
Expand Down Expand Up @@ -47,6 +47,13 @@
# Needs: a live Mooncake store (the shared fixture brings up backend, pool and Bindings) and an
# InstanceType, for the role to reference
#
# Inputs: All real, nothing mocked: the fixture's pool and Bindings beside the case's own
# unreachable pool, and the engine image the deployments render.
# Expected: The unreachable pool renders no connector onto its deployment while the reachable
# pool's image still gets its render; every CacheAttached row pairs the condition
# with a fact not true of everything.
# Cleanup: The EXIT teardown deletes the namespace objects and both cluster-scoped objects
# the case created.
set -uo pipefail

NS="${1:?usage: case-48.sh <NS>}"
Expand Down
7 changes: 5 additions & 2 deletions .agents/skills/gpustack-operator-e2e/cases/case-51.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
#
# CASE 51 — Every P/D refusal fires, from the layer that owns it, and a replicas change replaces
# nobody
# (NON-MUTATING for the refusals, which are server-side dry-runs; one short-lived deployment for
# the convergence rows)
# (MUTATING, self-cleaning: the refusals are server-side dry-runs; one short-lived deployment
# serves the convergence rows)
#
# case-51.sh <NS>
#
Expand Down Expand Up @@ -43,6 +43,9 @@
# and whether a short pool leaves both roles queued is case-50's. This file stops at
# the operator's own writes, which is what it can assert without an engine.
#
# Expected: Each refusal names its owning layer and message; the replicas change converges
# without replacing the replica already running.
# Cleanup: An EXIT trap deletes the short-lived deployment and its groups' Workloads.
set -o pipefail

NS="${1:-}"
Expand Down
2 changes: 2 additions & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-68.sh
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@
# against one object would be asserting one of them against a state that cannot produce it. Phase E
# needs a THIRD object for the same reason: Phase C deactivates the deployment it runs on, and a
# deactivated workload is never admitted again whatever the cluster does.
# Cleanup: An EXIT trap deletes the three ModelDeployments and the held InstanceType the case
# created.
set -uo pipefail

# Route every kubectl through the retrying shim. Against a remote API endpoint a read can fail on
Expand Down
8 changes: 8 additions & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-70.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
#
# case-70.sh <NS>
#
# Goal: A routed deployment owns exactly one complete router object set through both
# router transitions, and its status reports every cache-event publication state
# observable from the cluster.
# Environment: A single-node cluster with no accelerator, the operator deployed, the general
# InstanceType materialized by CASE 1, and pull access to the Mooncake fixture image.
# No serving engine image is required: the assertions read rendered objects, Pod
Expand All @@ -18,6 +21,11 @@
# garbage collection remove all six. The deployment remains Starting with no ready
# role replicas. KVEventsPublishing reaches NotApplicable, NoRouter,
# PublisherDisabled, RoleUnmanaged and Publishing, with status and reason asserted.
# Inputs: All real: the operator's charts and images, the Mooncake fixture image and a Ready
# Binding. No serving engine image — the assertions read rendered objects, Pod
# specifications and status.
# Cleanup: The teardown releases any finalizer-held Pods it created and deletes the lifecycle
# objects.
set -uo pipefail

E2E_SHIM_DIR="$(cd "$(dirname "$0")/../../_e2e-lib/scripts/kubectl-shim" 2>/dev/null && pwd)"
Expand Down
10 changes: 10 additions & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-71.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@
# This case is separate from CASE 70 because the upstream router images may be unavailable. An image
# pull failure is a stated SKIP. Any other failure to become Ready is a FAIL. No accelerator or
# serving engine is needed: the router runs outside Kueue and discovers the role Pods dynamically.
# Goal: The router endpoint publishes only once the router Deployment has a ready replica —
# the readiness contract status.endpoint carries.
# Environment: A single-node cluster with no accelerator, the operator deployed, the general
# InstanceType from CASE 1, and pull access to a router image.
# Inputs: All real: the router image is pulled from its registry; no accelerator and no
# serving engine image is needed.
# Expected: With the image pullable, the deployment publishes its endpoint once the router
# replica is Ready and RouterReady reads True/Ready. An image pull failure is the
# stated SKIP; any other failure to become Ready is a FAIL.
# Cleanup: An EXIT trap deletes the deployment and any Pods left under it.
set -uo pipefail

E2E_SHIM_DIR="$(cd "$(dirname "$0")/../../_e2e-lib/scripts/kubectl-shim" 2>/dev/null && pwd)"
Expand Down
1 change: 1 addition & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-89.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
#
# CASE 89 — A ModelDeployment role's rendered Pod requests the selected fabric device.
# (NON-MUTATING; E2E_REJECT_MANIFEST, when set, is a server-side dry run)
#
# case-89.sh <NS> <MODEL_DEPLOYMENT> <ROLE> <COUNT> <KEY|none>
#
Expand Down
1 change: 1 addition & 0 deletions .agents/skills/gpustack-operator-e2e/cases/case-90.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env bash
#
# CASE 90 — A ModelDeployment metrics snapshot reflects served traffic and Pod endpoints.
# (MUTATING, self-cleaning: one temporary probe Pod, deleted by the trap)
#
# case-90.sh <NS> <MODEL_DEPLOYMENT>
#
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/gpustack-operator-e2e/cases/case-91.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash
#
# CASE 91 — A routed prefill/decode pair moves real KV bytes over its direct leg, and writes the
# store when it has one.
# store when it has one. (NON-MUTATING)
#
# case-91.sh <NS> <MODEL_DEPLOYMENT>
#
Expand Down
10 changes: 5 additions & 5 deletions docs/modules/model-deployment/routing.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ pairing:
| Mode | `llm-d-router` | `vllm-router` | `sglang-gateway` |
|---|---|---|---|
| Multi-replica *Server* | One fixed scoring profile — prefix match, queue depth, KV utilization — with no policy switch; measured 187/170 on vLLM and 375/0, every request on one replica, on SGLang | `cache_aware` by default; `--policy round_robin` spreads evenly, measured 60 per replica | The same policies as `vllm-router`; `round_robin` measured 60 per replica |
| *PD* pair | The disaggregated pairing's own router: the token-counted prefill/decode split, per-role connector documents, the decode sidecar | Its own *PD* mode over the pair | Native `--pd-disaggregation` |
| *PD* pair | The disaggregated pairing's own router: the token-counted prefill/decode split, per-role connector documents, the decode sidecar | Its own *PD* mode over the pair | Native `--pd-disaggregation`; combining it with `--dp-aware` produced an invalid *decode* destination in the recorded container run, so the operator never renders the two together |
| *EPD* | An `encode` scheduling profile beside the pair; the decode sidecar coordinates the encoder transfer | Drives encode, prefill and decode itself through operator-owned `--epd-config`, no sidecar | Routes the pair natively; the prefiller reaches the encoder through `--encoder-urls` |
| *Elastic EP* | The engine's own internal load balancing spreads work inside the collective, and the deployment publishes one endpoint, so a router is not how this mode balances | Same | Same |
| *External DP* | Admitted, but no `--policy` to spread the ranks evenly and no recorded run — use the engine's own router | Every rank endpoint behind `--policy round_robin` (recorded) | Per-member endpoints; the registration boundary is the paragraph below |
Expand Down Expand Up @@ -63,10 +63,10 @@ carries a rank suffix (`http://host:port@0`, `http://host:port@1`), and every su
member's one physical serving listener — the suffix is a rank, not a port.

That registration is the gateway patch this project ships: an image carrying it is
`gpustack/llm-router:v0.4.0` or later, while the default
[`model-deployment-router-image`](../../reference/settings.md) still names `v0.3.1`, which
predates the patch. The registration activates only where *`spec.router.image`* or the setting
names an image that carries the patch.
`gpustack/llm-router:v0.4.0` or later, which is also the default
[`model-deployment-router-image`](../../reference/settings.md), so a deployment that names no
image of its own registers per rank out of the box. An older default or a named image that
predates the patch runs the gateway without the fan-out.

No runtime run has exercised the flag: the render gate and the patched fan-out are verified in
source and unit tests on the recorded shape.
Expand Down
Loading
Loading