diff --git a/.agents/skills/_e2e-lib/scripts/kubectl-shim/kubectl b/.agents/skills/_e2e-lib/scripts/kubectl-shim/kubectl index 40d7da1e8..22344c9b3 100755 --- a/.agents/skills/_e2e-lib/scripts/kubectl-shim/kubectl +++ b/.agents/skills/_e2e-lib/scripts/kubectl-shim/kubectl @@ -119,7 +119,7 @@ transient() { } errf="$(mktemp)" -trap 'rm -f "$errf" "${SPOOL:-}"' EXIT +trap 'rm -f "$errf" ${SPOOL:+"$SPOOL"}' EXIT attempt=0 while :; do if [ -n "$SPOOL" ]; then diff --git a/.agents/skills/gpustack-operator-e2e/SKILL.md b/.agents/skills/gpustack-operator-e2e/SKILL.md index 814f312cc..78a631e44 100644 --- a/.agents/skills/gpustack-operator-e2e/SKILL.md +++ b/.agents/skills/gpustack-operator-e2e/SKILL.md @@ -163,6 +163,7 @@ Each case is self-contained; its header (see **Case header contract**) states go | 112 | Image source: a tag-only reference is refused naming the digest contract; an image artifact resolves claim-shaped (no manifest digest, no revision, no `status.nodes`); an Instance pinned to the node mounts the pinned image read-only through an image volume and reads the fixture byte for byte; a `ModelPrefetch` naming the artifact is refused | `pkg/worker/webhooks/worker/{model_artifact,model_prefetch}.go`, `pkg/worker/controllers/worker/{model_artifact_placement,model_deployment_artifact,instance,model_placement_preference}.go`, `pkg/kubediscovery/feature.go`, `cases/case-112.sh` | yes (confirm) | one worker whose kubelet/containerd serve image volumes (kubelet 1.35+, containerd 2.1+), docker on the runner, the stock `registry:2`, `python:3.12-slim` and `busybox:1.36` images pullable | | 113 | ModelScope source: a `modelScope` artifact resolves to the commit `git ls-remote` also names and materializes on a cold node hashing to the hub's own sha256; a second node pulls from the peer with its hub byte counter flat; a `modelscope` SDK at the floor downloads at the resolved commit; a well-formed `modelScope` source is admitted and a three-part repository refused | `pkg/modelartifact/modelscope.go`, `pkg/worker/controllers/worker/model_artifact.go`, `pkg/modelmanager/{driver/authorize.go,materialize/materialize.go,report/report.go}`, `pkg/worker/webhooks/worker/model_artifact.go`, `cases/case-113.sh` | yes (confirm) | two schedulable workers, the chart with `modelManager.enabled`, the stock python image pullable, and an internet path to www.modelscope.cn and pypi.org from the nodes and the runner (`E2E_C113_OFFLINE=1` skips) | | 114 | Expected digest: a match resolves with `digestSource Hub`, a foreign anchor is refused `DigestMismatch` naming both digests, a twice-unreachable hub writes the anchor as the identity (`digestSource Expected`, no revision) with the hub's request log flat, `Engine` delivery holds both anchored artifacts with `AnchorNeedsNodeDelivery` and no Pod, and an `Expected` identity mounts on a second node from the peer, hub log still flat | `api/worker/v1alpha1/model_artifact.go`, `pkg/worker/webhooks/worker/model_artifact.go`, `pkg/worker/controllers/worker/{model_artifact,model_artifact_placement,model_deployment}.go`, `pkg/modelmanager/materialize/materialize.go`, `pkg/modelmanager/peer/client.go`, `cases/case-114.sh` | yes (confirm) | one schedulable worker for the anchor legs, two for the peer leg (`E2E_C114_OFFLINE=1` skips that leg), the chart with `modelManager.enabled`, the stock python image pullable, one InstanceType for the engine-block rows; no internet — the hub is the case's own test hub | +| 115 | Ordinary role updates replace whole groups: initially queued resources without a replacement slot, size growth and shrink, whole-card/slice resources, takeover command, optional cross-pool InstanceType, and P/D sibling identity | `pkg/worker/controllers/worker/model_deployment{,_initial_queue,_replacement_slot,_pod_group,_render,_service}.go`, `pkg/worker/webhooks/worker/model_deployment.go`, `cases/case-115.sh` | yes (confirm) | an idle NVIDIA pool with onceMaxRequest=1; two free whole cards for the size leg; logical slice capacity for slice edits; a different accelerator group for the cross-pool leg; a CPU type for P/D size and command checks; no engine inference | Each note below is something the **lead** must act on before or around a run. What a case *does* — its goal, environment, inputs, assertions and cleanup — lives in its own header, which the **Case header contract** below requires to be readable on its own; the index never restates it. diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-115.sh b/.agents/skills/gpustack-operator-e2e/cases/case-115.sh new file mode 100755 index 000000000..b2d9b671f --- /dev/null +++ b/.agents/skills/gpustack-operator-e2e/cases/case-115.sh @@ -0,0 +1,417 @@ +#!/usr/bin/env bash +# +# CASE 115 — Role shape edits replace complete replicas, including an initially queued group +# (MUTATING, self-recovering; AUTO-SKIPS without a suitable accelerator pool) +# +# E2E_C115_IT= [E2E_C115_OTHER_IT=] case-115.sh +# +# Goal: Observe size, resources, command and InstanceType edits on the same deployment. +# A first queued group has no replacement slot. Reducing its accelerator request +# must replace its obsolete members and Workload rather than leave it queued forever. +# Environment: A namespace watched by Kueue, an Active accelerator pool with at least one +# whole-card unit, and onceMaxRequest=1. Other shapes SKIP the case explicitly. +# The initially queued leg needs total capacity of at least two units. A one-card +# pool refuses a two-card request at admission, so that leg SKIPs on such a pool. +# The size leg needs two free units at the start; without them only that leg SKIPs. +# The optional second type must identify a different accelerator group. Without it, +# only the cross-pool leg SKIPs. Run with the selected pools otherwise idle. +# A logical slice capacity enables whole-card to slice and slice percentage edits. +# E2E_C115_CPU_IT enables a CPU P/D size and command leg; otherwise that leg SKIPs. +# Inputs: Real ModelDeployments, Pods, Workloads and InstanceTypes. No mocked +# hardware or edited quotas. Shell commands are placeholders; no inference is proved. +# E2E_C115_IMAGE overrides ubuntu. E2E_C115_TIMEOUT sets each convergence bound. +# E2E_C115_RAW_DIR optionally retains every observation, including failure evidence. +# Expected: The initial queued Pod has no node, no admission and no replacement slot. An edit +# gives the same deployment fresh Pod and Workload UIDs, matching template resources, +# PodSet count and queue, and current Admitted=True. Size grows and shrinks as whole +# groups. Command and cross-pool edits replace the members without mixing generations. +# A P/D edit preserves the untouched role's Pod and Workload UIDs. +# Cleanup: The finally block deletes only this case's deployment and its captured Workloads. +# It waits for their Pods and Workloads to disappear on pass and fail. No baseline +# setting, node, pool or device-manager configuration is changed. + +set -uo pipefail +NS="${1:?usage: E2E_C115_IT= case-115.sh }" +export E2E_C115_NAMESPACE="$NS" +E2E_C115_SOURCE_ROOT="$(cd "$(dirname "$0")/../../../.." && pwd)" +export E2E_C115_SOURCE_ROOT +python3 - <<'PY' +import json +import os +import pathlib +import re +import subprocess +import time +import uuid +from decimal import Decimal + +NS = os.environ['E2E_C115_NAMESPACE'] +IT = os.environ.get('E2E_C115_IT', '') +OTHER = os.environ.get('E2E_C115_OTHER_IT', '') +CPU_IT = os.environ.get('E2E_C115_CPU_IT', '') +NAME = 'case115-' + uuid.uuid4().hex[:8] +TIMEOUT = int(os.environ.get('E2E_C115_TIMEOUT', '180')) +RAW = pathlib.Path(os.environ['E2E_C115_RAW_DIR']) if os.environ.get('E2E_C115_RAW_DIR') else None +COMMAND = ['sh', '-c', 'sleep 86400'] +ROWS = [] +WORKLOADS = set() +MD_UID = None +OWNERSHIP_LABEL = 'gpustack.ai/e2e-case-115-token' +OWNERSHIP_TOKEN = 'case115-' + uuid.uuid4().hex +SOURCE = pathlib.Path(os.environ['E2E_C115_SOURCE_ROOT']) / 'pkg/nodefeature/knowns.go' +MAX_UNITS = int(re.search(r'ResourceMaxUnits\s*=\s*([0-9_]+)', SOURCE.read_text()).group(1).replace('_', '')) + + +def kubectl(*args, obj=None, allow_missing=False): + result = subprocess.run( + ['kubectl', '--request-timeout=30s', *args], + input=json.dumps(obj) if obj is not None else None, + capture_output=True, text=True, timeout=45, + ) + if result.returncode: + if allow_missing and 'NotFound' in result.stderr: + return None + raise RuntimeError(result.stderr.strip() or result.stdout.strip()) + return json.loads(result.stdout) if result.stdout.lstrip().startswith('{') else result.stdout + + +def record(status, check, detail): + ROWS.append((status, check, detail)) + print(' | '.join(ROWS[-1]), flush=True) + + +def discover_owned_deployment(): + global MD_UID + current = kubectl('-n', NS, 'get', 'modeldeployments.worker.gpustack.ai', NAME, '-o', 'json', + allow_missing=True) + if current is None: + return None + labels = current.get('metadata', {}).get('labels', {}) + if labels.get(OWNERSHIP_LABEL) != OWNERSHIP_TOKEN: + return None + MD_UID = current['metadata']['uid'] + return current + + +def create_owned_deployment(obj): + global MD_UID + try: + created = kubectl('create', '-f', '-', '-o', 'json', obj=obj) + except Exception: + # The API may have persisted the object before the client lost its response. + # Recover its UID only after checking the case-specific ownership token. + discover_owned_deployment() + raise + if created.get('metadata', {}).get('labels', {}).get(OWNERSHIP_LABEL) != OWNERSHIP_TOKEN: + raise RuntimeError('created deployment is missing the case ownership token') + MD_UID = created['metadata']['uid'] + return created + + +def delete_owned_deployment(uid): + options = {'apiVersion': 'v1', 'kind': 'DeleteOptions', + 'preconditions': {'uid': uid}} + path = ('/apis/worker.gpustack.ai/v1/namespaces/' + NS + + '/modeldeployments/' + NAME) + kubectl('delete', '--raw', path, '-f', '-', obj=options) + + +def has_condition(obj, name): + return any(c['type'] == name and c['status'] == 'True' + for c in obj.get('status', {}).get('conditions', [])) + + +def owns_workload(workload): + return any(o.get('kind') == 'Pod' and o.get('name', '').startswith(NAME + '-') + for o in workload['metadata'].get('ownerReferences', [])) + + +def snapshot(label): + md = kubectl('-n', NS, 'get', 'modeldeployments.worker.gpustack.ai', NAME, '-o', 'json') + pods = kubectl('-n', NS, 'get', 'pods', '-l', 'app.kubernetes.io/instance=' + NAME, '-o', 'json')['items'] + pods = [p for p in pods if any(o['uid'] == md['metadata']['uid'] + for o in p['metadata'].get('ownerReferences', []))] + uids = {p['metadata']['uid'] for p in pods} + workloads = kubectl('-n', NS, 'get', 'workloads.kueue.x-k8s.io', '-o', 'json')['items'] + workloads = [w for w in workloads if owns_workload(w) or any(o['uid'] in uids + for o in w['metadata'].get('ownerReferences', []))] + WORKLOADS.update(w['metadata']['name'] for w in workloads) + result = {'deployment': md, 'pods': pods, 'workloads': workloads} + if RAW: + RAW.mkdir(parents=True, exist_ok=True) + with (RAW / (NAME + '-' + label.replace('/', '-') + '.jsonl')).open('a') as stream: + stream.write(json.dumps(result) + '\n') + return result + + +def wait(label, predicate): + deadline = time.monotonic() + TIMEOUT + while True: + state = snapshot(label) + if predicate(state): + return state + if time.monotonic() >= deadline: + raise RuntimeError(label + ' did not converge; observation retained in E2E_C115_RAW_DIR') + time.sleep(2) + + +def main_container(pod): + return next(c for c in pod['spec']['containers'] if c['name'] == 'main') + + +def accelerator_matches(container, role, inst_type): + accelerated = inst_type['spec'].get('acceleratable', False) + resources = role['resources'] + memory = resources.get('acceleratorSlicedMemoryPercentage', 0) + cores = resources.get('acceleratorSlicedCoresPercentage', 0) + for kind in ('requests', 'limits'): + actual = container['resources'][kind] + if memory or cores: + expected_memory, expected_cores = memory or cores, cores or memory + expected = {'nvidia.com/gpu.sliced': str(resources['accelerator']), + 'nvidia.com/gpu.sliced.memory-percentage': str(expected_memory), + 'nvidia.com/gpu.sliced.cores-percentage': str(expected_cores)} + if not accelerated or any(actual.get(key) != value for key, value in expected.items()): + return False + units = actual.get('nvidia.com/gpu.sliced.units', '') + if not units: + return False + scale = {'k': 1000, 'M': 1000000, 'G': 1000000000}.get(units[-1], 1) + value = Decimal(units[:-1] if scale != 1 else units) * scale + if value != expected_memory * MAX_UNITS // 100 or 'nvidia.com/gpu' in actual: + return False + else: + expected = str(resources['accelerator']) if accelerated else None + if actual.get('nvidia.com/gpu') != expected or 'nvidia.com/gpu.sliced' in actual: + return False + return True + + +def converged(state, old=None, changed='server'): + md = state['deployment'] + roles = md['spec']['roles'] + pods, workloads = state['pods'], state['workloads'] + if md['metadata']['uid'] != MD_UID or len(pods) != sum(r['size'] for r in roles) or len(workloads) != len(roles): + return False + if md['metadata'].get('annotations', {}).get('modeldeployment.gpustack.ai/replacement-slots'): + return False + if not all(has_condition(p, 'Ready') and not p['metadata'].get('deletionTimestamp') for p in pods): + return False + for role in roles: + members = [p for p in pods if p['metadata']['labels']['app.kubernetes.io/component'] == role['name']] + if len(members) != role['size']: + return False + member_uids = {p['metadata']['uid'] for p in members} + owned = [w for w in workloads if any(o['uid'] in member_uids for o in w['metadata'].get('ownerReferences', []))] + if len(owned) != 1: + return False + workload = owned[0] + if not has_condition(workload, 'Admitted') or not workload.get('status', {}).get('admission'): + return False + pod_set = workload['spec']['podSets'] + if len(pod_set) != 1 or pod_set[0]['name'] != role['name'] or pod_set[0]['count'] != role['size']: + return False + owners = {o['uid'] for o in workload['metadata'].get('ownerReferences', []) if o['kind'] == 'Pod'} + if owners != member_uids: + return False + inst_type = kubectl('get', 'instancetypes.worker.gpustack.ai', role['instanceType'], '-o', 'json') + entrance = inst_type['status']['entrance'] + if workload['spec']['queueName'] != entrance: + return False + for pod in members: + container = main_container(pod) + if container.get('command') != role['command']: + return False + if pod['metadata']['labels'].get('kueue.x-k8s.io/queue-name') != entrance: + return False + if inst_type['spec'].get('acceleratable', False): + node = pod['spec'].get('nodeName') + if not node: + return False + devices = kubectl('get', 'devices.worker.gpustack.ai', node, '-o', 'json') + offered = {a['id'] for group in devices['spec']['groups'] + if group['manufacturer'] + '-' + group['id'] == inst_type['spec']['acceleratorGroup'] + for a in group.get('accelerators', [])} + allocation = json.loads(pod['metadata'].get('annotations', {}).get( + 'device.gpustack.ai/accelerator.allocated', '{}')) + groups = allocation.get('main', {}).get('devices', {}).get('groups', []) + assigned = {a['id'] for group in groups for a in group.get('accelerators', [])} + if not groups or not assigned or not assigned.issubset(offered): + return False + if any(group['manufacturer'] + '-' + group['id'] != inst_type['spec']['acceleratorGroup'] + for group in groups): + return False + if not accelerator_matches(container, role, inst_type): + return False + if main_container(pod_set[0]['template']).get('resources') != container.get('resources'): + return False + if main_container(pod_set[0]['template']).get('command') != container.get('command'): + return False + if old: + changed_pods = [p for p in old['pods'] if p['metadata']['labels']['app.kubernetes.io/component'] == changed] + old_pods = {p['metadata']['uid'] for p in changed_pods} + if old_pods.intersection(p['metadata']['uid'] for p in pods): + return False + for pod_before in changed_pods: + current = kubectl('-n', NS, 'get', 'pods', pod_before['metadata']['name'], + '-o', 'json', allow_missing=True) + if current and current['metadata']['uid'] == pod_before['metadata']['uid']: + return False + for workload_before in old['workloads']: + if not any(o['uid'] in old_pods for o in workload_before['metadata'].get('ownerReferences', [])): + continue + current = kubectl('-n', NS, 'get', 'workloads.kueue.x-k8s.io', + workload_before['metadata']['name'], '-o', 'json', allow_missing=True) + if current and current['metadata']['uid'] == workload_before['metadata']['uid']: + return False + for role in roles: + if role['name'] == changed: + continue + def identity(view): + uids = {p['metadata']['uid'] for p in view['pods'] + if p['metadata']['labels']['app.kubernetes.io/component'] == role['name']} + work = {w['metadata']['uid'] for w in view['workloads'] + if any(o['uid'] in uids for o in w['metadata'].get('ownerReferences', []))} + return uids, work + if identity(state) != identity(old): + return False + return True + + +def edit(label, fields): + old = snapshot(label + '-before') + patch = [{'op': 'add', 'path': '/spec/roles/0/' + key, 'value': value} + for key, value in fields.items()] + kubectl('-n', NS, 'patch', 'modeldeployments.worker.gpustack.ai', NAME, + '--type=json', '-p', json.dumps(patch)) + changed = old['deployment']['spec']['roles'][0]['name'] + wait(label, lambda state: converged(state, old, changed)) + record('PASS', label, 'same deployment; fresh complete group and Admitted Workload') + + +def cleanup(): + if MD_UID is None: + discover_owned_deployment() + if MD_UID is None: + return + current = kubectl('-n', NS, 'get', 'modeldeployments.worker.gpustack.ai', NAME, '-o', 'json', + allow_missing=True) + if current is None: + return + labels = current.get('metadata', {}).get('labels', {}) + if (labels.get(OWNERSHIP_LABEL) != OWNERSHIP_TOKEN + or current.get('metadata', {}).get('uid') != MD_UID): + raise RuntimeError('refusing to delete a deployment without matching case ownership') + delete_owned_deployment(MD_UID) + for name in sorted(WORKLOADS): + kubectl('-n', NS, 'delete', 'workloads.kueue.x-k8s.io', name, + '--ignore-not-found', '--wait=false') + deadline = time.monotonic() + TIMEOUT + while time.monotonic() < deadline: + pods = kubectl('-n', NS, 'get', 'pods', '-l', 'app.kubernetes.io/instance=' + NAME, '-o', 'json')['items'] + md = kubectl('-n', NS, 'get', 'modeldeployments.worker.gpustack.ai', NAME, '-o', 'json', allow_missing=True) + workloads = kubectl('-n', NS, 'get', 'workloads.kueue.x-k8s.io', '-o', 'json')['items'] + discovered = {w['metadata']['name'] for w in workloads if owns_workload(w)} + for name in discovered - WORKLOADS: + kubectl('-n', NS, 'delete', 'workloads.kueue.x-k8s.io', name, + '--ignore-not-found', '--wait=false') + WORKLOADS.update(discovered) + remaining = [w for w in workloads if w['metadata']['name'] in WORKLOADS] + if not pods and md is None and not remaining: + record('PASS', 'cleanup', 'owned deployment, Pods and Workloads absent') + return + time.sleep(2) + raise RuntimeError('owned deployment, Pods or Workloads remain after cleanup') + + +try: + if not IT: + record('SKIP', 'role shape updates', 'E2E_C115_IT is unset; NOTHING WAS VERIFIED') + else: + inst_type = kubectl('get', 'instancetypes.worker.gpustack.ai', IT, '-o', 'json') + resource = inst_type['status']['accelerator'] + suitable = (inst_type['status']['phase'] == 'Active' + and inst_type['status']['detail']['manufacturer'] == 'nvidia' + and resource.get('onceMaxRequest') == '1' + and int(resource['remaining']) >= 1) + if not suitable: + record('SKIP', 'role shape updates', 'needs a free NVIDIA whole-card unit and onceMaxRequest=1; NOTHING WAS VERIFIED') + else: + obj = {'apiVersion': 'worker.gpustack.ai/v1', 'kind': 'ModelDeployment', + 'metadata': {'name': NAME, 'namespace': NS}, 'spec': { + 'model': {'name': 'Qwen/Qwen2.5-0.5B-Instruct'}, + 'engine': {'name': 'vLLM', 'version': '0.29.0'}, 'roles': [{ + 'name': 'server', 'kind': 'Server', 'replicas': 1, 'size': 1, + 'instanceType': IT, 'image': os.environ.get('E2E_C115_IMAGE', 'ubuntu:24.04'), + 'command': COMMAND, 'resources': {'accelerator': '2' if int(resource['capacity']) >= 2 else '1'}, + 'terminationGracePeriodSeconds': 15}]}} + obj['metadata']['labels'] = {OWNERSHIP_LABEL: OWNERSHIP_TOKEN} + created = create_owned_deployment(obj) + def initially_queued(state): + if len(state['pods']) != 1 or len(state['workloads']) != 1: + return False + pod, workload = state['pods'][0], state['workloads'][0] + return (not pod['spec'].get('nodeName') and not has_condition(workload, 'Admitted') + and any(g['name'] == 'kueue.x-k8s.io/admission' for g in pod['spec'].get('schedulingGates', [])) + and not state['deployment']['metadata'].get('annotations', {}).get('modeldeployment.gpustack.ai/replacement-slots')) + if int(resource['capacity']) >= 2: + wait('initially queued without a slot', initially_queued) + record('PASS', 'initially queued without a slot', 'real Pod and owning Workload; no admission or node') + edit('queued resources two to one', {'resources': {'accelerator': '1'}}) + else: + record('SKIP', 'initially queued resources two to one', 'one-card pool refuses the initial request; queued resource behavior remains UNVERIFIED') + wait('one-card baseline', converged) + if int(resource['remaining']) >= 2: + edit('size one to two', {'size': 2}) + edit('size two to one', {'size': 1}) + else: + record('SKIP', 'size one to two and back', 'fewer than two free units at the start; multi-member replacement remains UNVERIFIED') + edit('takeover command replacement', {'command': ['sh', '-c', 'true; sleep 86400']}) + if int(inst_type['status'].get('acceleratorSliced', {}).get('capacity') or 0) > 0: + edit('whole card to half slice', {'resources': {'accelerator': '1', + 'acceleratorSlicedMemoryPercentage': 50, 'acceleratorSlicedCoresPercentage': 50}}) + edit('slice fifty to forty percent', {'resources': {'accelerator': '1', + 'acceleratorSlicedMemoryPercentage': 40, 'acceleratorSlicedCoresPercentage': 40}}) + edit('slice to whole card', {'resources': {'accelerator': '1'}}) + else: + record('SKIP', 'logical slice resource updates', 'no logical slice capacity; slice replacement remains UNVERIFIED') + if OTHER: + second = kubectl('get', 'instancetypes.worker.gpustack.ai', OTHER, '-o', 'json') + if second['spec']['acceleratorGroup'] == inst_type['spec']['acceleratorGroup']: + raise RuntimeError('E2E_C115_OTHER_IT names the same accelerator group; it cannot prove a cross-pool update') + edit('InstanceType cross-pool replacement', {'instanceType': OTHER}) + else: + record('SKIP', 'InstanceType cross-pool replacement', 'E2E_C115_OTHER_IT is unset; cross-pool behavior remains UNVERIFIED') + if CPU_IT: + cleanup() + MD_UID = None + WORKLOADS.clear() + NAME += '-pd' + obj['metadata']['name'] = NAME + obj['spec']['kvTransfer'] = {'protocol': 'TCP'} + obj['spec']['roles'] = [{ + 'name': kind.lower(), 'kind': kind, 'replicas': 1, 'size': 1, + 'instanceType': CPU_IT, 'image': os.environ.get('E2E_C115_IMAGE', 'ubuntu:24.04'), + 'command': COMMAND, 'resources': {'accelerator': '0'}, + 'terminationGracePeriodSeconds': 15, + } for kind in ('Prefill', 'Decode')] + created = create_owned_deployment(obj) + wait('P/D baseline', converged) + edit('prefill size one to two preserves decode', {'size': 2}) + edit('prefill size two to one preserves decode', {'size': 1}) + edit('prefill command replacement preserves decode', {'command': ['sh', '-c', 'true; sleep 86400']}) + else: + record('SKIP', 'P/D sibling preservation', 'E2E_C115_CPU_IT is unset; P/D size and command behavior remains UNVERIFIED') +except Exception as error: + record('FAIL', 'role shape updates', str(error)) +finally: + try: + cleanup() + except Exception as error: + record('FAIL', 'cleanup', str(error)) + +print('\nSTATUS | CHECK | OBJECT') +for row in ROWS: + print(' | '.join(row)) +raise SystemExit(1 if any(row[0] == 'FAIL' for row in ROWS) else 0) +PY diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-22.sh b/.agents/skills/gpustack-operator-e2e/cases/case-22.sh index 85a8a8b15..1defa9cf6 100755 --- a/.agents/skills/gpustack-operator-e2e/cases/case-22.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-22.sh @@ -85,6 +85,8 @@ best=None for n in json.load(sys.stdin).get('items',[]): a=n.get('status',{}).get('allocatable',{}) for k,v in a.items(): + # Allocation checks below read accelerator grants, so exclude RDMA interfaces. + if k == 'device.gpustack.ai/rdma': continue if not re.match(r'^[^/]+/[^.]+$', k): continue # the bare exclusive whole-card resource try: cnt=int(v) except: continue @@ -327,7 +329,7 @@ fill_exclusive() { local tag="$1" check="$2" count="$3" lq="$4" forbidden="$5" local tagl="${tag,,}" i p c cc seen=" " FILL_CARDS="" - for i in $(seq 1 "$count"); do + for ((i = 1; i <= count; i++)); do p="${PODPFX}-${tagl}-excl-${i}"; TESTPODS+=("$p") mkpod "$p" "$EXCL" "$lq" if [ "$(wait_settled "$p")" != "Running" ]; then diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-43.sh b/.agents/skills/gpustack-operator-e2e/cases/case-43.sh index 987ad5668..61642ca27 100755 --- a/.agents/skills/gpustack-operator-e2e/cases/case-43.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-43.sh @@ -149,7 +149,7 @@ restore() { for ns in "$NS_A" "$NS_B"; do kubectl -n "$ns" get kvcachepoolbindings.worker.gpustack.ai -o name 2>/dev/null \ | while read -r b; do - kubectl -n "$ns" patch "$b" --subresource=status --type=merge \ + kubectl -n "$ns" patch kvcachepoolbindings.v1alpha1.worker.gpustack.ai "${b##*/}" --subresource=status --type=merge \ -p '{"status":{"usedBy":null}}' >/dev/null 2>&1 || true done done @@ -630,8 +630,20 @@ echo "== 6. a binding a workload holds cannot be deleted ==" # WRITTEN BY THIS CASE. The ModelDeployment controller fills a Binding's usedBy in production, and # this case runs no ModelDeployment. A case that waited for a writer would wait forever, and one # that skipped the write would assert a release nothing was holding. -kubectl -n "$NS_A" patch kvcachepoolbindings.worker.gpustack.ai bind-a --subresource=status --type=merge \ - -p '{"status":{"usedBy":[{"kind":"ModelDeployment","namespace":"","name":"qwen-e2e"}]}}' >/dev/null 2>&1 +# The public v1 proxy exposes no status writer. Use the storage version for this fixture write. +if ! kubectl -n "$NS_A" patch kvcachepoolbindings.v1alpha1.worker.gpustack.ai bind-a --subresource=status --type=merge \ + -p '{"status":{"usedBy":[{"kind":"ModelDeployment","namespace":"","name":"qwen-e2e"}]}}' >/dev/null; then + record FAIL "the holder fixture is written" "the storage-version status patch failed; no held deletion can be asserted" + print_rows + exit 1 +fi +holder_kind="$(kubectl -n "$NS_A" get kvcachepoolbindings.v1alpha1.worker.gpustack.ai bind-a \ + -o 'jsonpath={.status.usedBy[?(@.name=="qwen-e2e")].kind}')" +if [ "$holder_kind" != ModelDeployment ]; then + record FAIL "the holder fixture is read back" "usedBy does not contain ModelDeployment/qwen-e2e" + print_rows + exit 1 +fi kubectl -n "$NS_A" delete kvcachepoolbindings.worker.gpustack.ai bind-a --wait=false >/dev/null 2>&1 || true sleep 20 @@ -650,8 +662,12 @@ fi # Released by hand, since nothing else will: this is the same patch a real consumer's controller # would make when it stops using the pool. -kubectl -n "$NS_A" patch kvcachepoolbindings.worker.gpustack.ai bind-a --subresource=status --type=merge \ - -p '{"status":{"usedBy":null}}' >/dev/null 2>&1 +if ! kubectl -n "$NS_A" patch kvcachepoolbindings.v1alpha1.worker.gpustack.ai bind-a --subresource=status --type=merge \ + -p '{"status":{"usedBy":null}}' >/dev/null; then + record FAIL "the holder fixture is released" "the storage-version status patch failed" + print_rows + exit 1 +fi # Deliberately NOT wait_for with an empty expected value. That helper reads kubectl under # 2>/dev/null and compares the output, so a transport error, an RBAC denial or a webhook intercepting # the GET yields the same empty string a deleted object does — and with an empty `want` the FIRST diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-45.sh b/.agents/skills/gpustack-operator-e2e/cases/case-45.sh index b1608776d..c5b7a6db0 100644 --- a/.agents/skills/gpustack-operator-e2e/cases/case-45.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-45.sh @@ -61,11 +61,11 @@ # Inputs: All real, nothing mocked. Server-side dry-run (`--dry-run=server`) runs the schema # and the webhook and persists nothing; the one row that needs a controller verdict # creates a ModelDeployment naming a Binding that does not exist and deletes it again. -# Before it is deleted that stored object is edited three times, because a frozen field -# is a rule about an UPDATE and no dry-run create reaches it: `size` and `model` must be -# refused (the model refusal must name spec.model, state that the value describes a -# different deployment, and say nothing of conflict, concurrency, locks or races), and -# `replicas` must be accepted and read back from storage. +# Before it is deleted that stored object is edited three times, because an UPDATE +# rule is not exercised by a dry-run create: `size` and `replicas` must be accepted, +# and the accepted values must be read back from storage. The `model` refusal must +# name spec.model and state that the value describes a different deployment. It must +# say nothing of conflict, concurrency, locks or races. # # Deferred: The serving half of this case — `replicas: 2` reaching `status.roles[0].ready == 2` # and `status.endpoint` answering an inference request — is NOT here. It needs the @@ -460,19 +460,17 @@ record SKIP "the unregistered domain costs the replicas their connector and noth # cannot fire on a create at all and no dry run reaches it. These two patch the deployment that is # still standing. # -# THE PAIR IS THE ROW. `replicas` and `size` sit side by side in the same struct and are edited -# through the same patch shape, so a webhook that refused both -- or a patch that failed for a -# reason having nothing to do with either -- reads exactly like a correct one through the refusal -# alone. The acceptance is what says the instrument can still say yes. +# Ordinary size and replica-count edits are accepted. Model and engine identity remain frozen. if [ "$nobind_ready" = yes ]; then size_out="$(kubectl -n "$NS" patch modeldeployments.worker.gpustack.ai case45-nobind --type=json \ -p '[{"op":"replace","path":"/spec/roles/0/size","value":2}]' 2>&1 | tr '\n' ' ')" - if [ -n "$size_out" ] && [ -z "${size_out##*roles\[0\].size*}" ]; then - record PASS "an edit to a role's size is refused, naming the field" \ - "refused at spec.roles[0].size: every member of a running instance was built for the rank layout the old value described" + stored_size="$(kubectl -n "$NS" get modeldeployments.worker.gpustack.ai case45-nobind \ + -o jsonpath='{.spec.roles[0].size}' 2>/dev/null)" + if [ -n "$size_out" ] && [ -z "${size_out##*patched*}" ] && [ "$stored_size" = 2 ]; then + record PASS "an ordinary role size edit is accepted and stored" "size=${stored_size}" else - record FAIL "an edit to a role's size is refused, naming the field" \ - "wanted a refusal naming roles[0].size, got: $(echo "$size_out" | cut -c1-160)" + record FAIL "an ordinary role size edit is accepted and stored" \ + "wanted size=2, stored=[${stored_size:-none}], patch: $(echo "$size_out" | cut -c1-160)" fi # THE MODEL SAYS WHICH DEPLOYMENT THIS IS, so it is frozen too, and its refusal is read for its @@ -513,7 +511,7 @@ if [ "$nobind_ready" = yes ]; then -p '[{"op":"replace","path":"/spec/roles/0/replicas","value":2}]' 2>&1 | tr '\n' ' ')" if [ -n "$replicas_out" ] && [ -z "${replicas_out##*patched*}" ]; then record PASS "the control: an edit to a role's replicas is accepted" \ - "patched, so the refusal above is this rule answering rather than the whole struct being frozen" + "patched alongside the accepted size edit" else record FAIL "the control: an edit to a role's replicas is accepted" \ "wanted the patch to be accepted, got: $(echo "$replicas_out" | cut -c1-160)" @@ -526,7 +524,7 @@ if [ "$nobind_ready" = yes ]; then record FAIL "the accepted replicas edit reached storage" "replicas=[${stored_replicas:-none}]" fi else - record SKIP "an edit to a role's size is refused, naming the field" \ + record SKIP "an ordinary role size edit is accepted and stored" \ "the subject deployment was never created, so there is no stored object to edit" record SKIP "an edit to the model is refused on a live object, naming the field" \ "the subject deployment was never created, so there is no stored object to edit" diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-63.sh b/.agents/skills/gpustack-operator-e2e/cases/case-63.sh index d4c08264d..db2cdc6b2 100644 --- a/.agents/skills/gpustack-operator-e2e/cases/case-63.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-63.sh @@ -276,6 +276,7 @@ spec: managed: leader: replicas: 3 + multiTenancy: false members: - nodeSelector: {kubernetes.io/os: linux} medium: DRAM diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-65.sh b/.agents/skills/gpustack-operator-e2e/cases/case-65.sh index f07d9733e..7061def12 100755 --- a/.agents/skills/gpustack-operator-e2e/cases/case-65.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-65.sh @@ -275,7 +275,7 @@ spec: # Omitting the key is refused at apply, which the host-directory gate above used to # hide -- that gate exits 0, so on any cluster without the directory this case reported # nothing rather than reporting that it could not build its own fixture. - leader: {electionBackend: None} + leader: {electionBackend: None, multiTenancy: false} members: - nodeSelector: {kubernetes.io/os: linux} medium: DRAM diff --git a/.agents/skills/gpustack-operator-e2e/cases/case-68.sh b/.agents/skills/gpustack-operator-e2e/cases/case-68.sh index ad93cd99a..ed6505c1d 100755 --- a/.agents/skills/gpustack-operator-e2e/cases/case-68.sh +++ b/.agents/skills/gpustack-operator-e2e/cases/case-68.sh @@ -190,7 +190,7 @@ group_names() { # value on every round -- a wait that can only succeed immediately or run out the clock. It reads # like a wait and behaves like a single sample. group_count_is() { [ "$(group_names "$1" | wc -l | tr -d ' ')" = "$2" ]; } -admitted_count_is_not() { [ "$(admitted_count)" != "$1" ]; } +admitted_count_is_not() { [ "$(admitted_count "$MD")" != "$1" ]; } role_count_is() { # role_count_is [ "$(k -n "$NS" get pods -l "app.kubernetes.io/instance=$1,app.kubernetes.io/component=$2" \ -o name 2>/dev/null | grep -c .)" = "$3" ] @@ -227,14 +227,12 @@ deployment_workloads() { } admitted_count() { - k -n "$NS" get workloads.kueue.x-k8s.io \ - -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Admitted")].status}{"\n"}{end}' 2>/dev/null \ - | grep -c '^True$' + local counts + counts="$(deployment_admitted "$1")" + printf '%s\n' "${counts%%/*}" } -# How many of ONE deployment's workloads report Admitted, as "/". The whole-cluster -# count above cannot answer Phase E: another deployment being admitted would satisfy it, and the -# claim there is about every group of one deployment. +# How many of one deployment's workloads report Admitted, as "/". deployment_admitted() { # deployment_admitted local wl total=0 ok=0 for wl in $(deployment_workloads "$1"); do @@ -513,10 +511,10 @@ else # NO ROLE ADMITTED -- and the control below is what makes this mean anything. sleep 30 - if [ "$(admitted_count)" = 0 ]; then + if [ "$(admitted_count "$MD")" = 0 ]; then record PASS "no role admitted while one group cannot be placed" "0 admitted workloads" else - record FAIL "no role admitted while one group cannot be placed" "$(admitted_count) admitted" + record FAIL "no role admitted while one group cannot be placed" "$(admitted_count "$MD") admitted" fi # AND THE DEPLOYMENT MUST NOT CLAIM IT HOLDS QUOTA. This row exists because this cluster case diff --git a/api/worker/v1alpha1/generated.proto b/api/worker/v1alpha1/generated.proto index 5773d1d8f..cece22013 100644 --- a/api/worker/v1alpha1/generated.proto +++ b/api/worker/v1alpha1/generated.proto @@ -3647,9 +3647,12 @@ message ModelDeploymentRetirementStatus { // nothing at all: it adds or removes instances, and every instance that stays keeps running, // keeps the accelerators it was admitted with and keeps whatever cache it holds. // -// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role. -// A replica's group is named from the deployment, its role and its ordinal, so edits to one role's -// running configuration do not rename a sibling role's groups. +// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role, +// and it refuses a kind change on a role that exists: the name and the kind are what the role IS, +// while size, instanceType, resources and command are how it is currently running and are +// editable beside every other container field. A replica's group is named from the deployment, +// its role and its ordinal, so edits to one role's running configuration do not rename a sibling +// role's groups. // // A DEPARTURE THIS OPERATOR DID NOT INITIATE IS NOT A ROLLOUT. The replica that left is replaced on // its own, under a new name, while its siblings keep serving — see @@ -3709,10 +3712,14 @@ message ModelDeploymentRole { // start together, they are replaced together, and none of them serves alone — the instance, // not the Pod, is the unit that appears and disappears. // - // THIS NUMBER IS FIXED AT CREATION AND CANNOT BE CHANGED. An instance's size is the shape of the - // instance, not a dial on it: the Pods a running instance is made of are not the Pods a different - // size asks for. Scaling is what replicas is for, and it leaves every running instance alone. To - // serve at a different size, create a deployment that declares it. + // IT IS EDITABLE. The edit replaces the role's replicas one at a time: an old replica and its + // Workload are deleted first, its replacement is created only after that slot is free, and the + // replacement's fresh admission gates deleting the next old replica. Surviving replicas keep + // their deployed configuration until each is selected for replacement; continuous serving is + // not guaranteed -- a one-replica role stops for the length of its swap. Scaling WITHOUT + // replacement is what replicas is for. + // + // THE ELASTIC-EP PROFILE PINS THIS NUMBER AT ONE; see ElasticEP. // // ABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods // is rendered with stable names and publishes the first Pod's address, this instance's size and @@ -3735,6 +3742,12 @@ message ModelDeploymentRole { // InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against. // It is what the queue-name entrance label is derived from. // + // IT IS EDITABLE. Moving a role to another type runs the same whole-group replacement: each + // old replica leaves before its replacement is created, and that replacement is admitted + // against the queue the new type publishes before the next old replica may leave. Existing + // Pods keep the type they were admitted with until they are deleted. The elastic-EP profile + // freezes this field after creation. + // // +required // +k8s:validation:minLength=1 // +k8s:validation:maxLength=253 @@ -3752,6 +3765,11 @@ message ModelDeploymentRole { // InstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards // a Pod wants is a property of the model being served, so two deployments on one InstanceType // routinely want different counts. + // + // IT IS EDITABLE. A change runs the same whole-group replacement, and the new request is + // admitted against the pool as each replacement is created. Omission behaves as on create: + // the accelerator half is defaulted, to one whole card of an acceleratable type. The + // elastic-EP profile freezes this field at the whole-card group it bound to TP at creation. optional ModelDeploymentRoleResources resources = 4; // Image is the container image to run. Leaving it empty is the ordinary case: the operator then @@ -3810,10 +3828,11 @@ message ModelDeploymentRole { // there is deliberately no Args, because a second append tier beside ExtraArgs would have no // defined precedence. // - // IT IS FROZEN AFTER CREATION, because it decides whether the operator configures this role at - // all: a role that supplies one is taken over by its author, which changes cache injection and - // what status can claim. The rest of the container fields are how the build is fetched, shaped - // and tuned, and stay editable. + // IT IS EDITABLE IN EVERY DIRECTION: managed to take-over, take-over back to managed, and + // take-over to take-over. Each edit runs the whole-group replacement, and the deployed Pods + // keep the mode they were created with until each is deleted for replacement, so during a + // rollout the role can hold both modes at once; cache injection and status read what is + // deployed. The elastic-EP profile freezes this field: the elastic renderer owns the argv. // // +optional // +listType=atomic @@ -3906,6 +3925,8 @@ message ModelDeploymentRole { // admission refuses a decrease until Elastic EP scale-down support is complete. // The role cannot enable or disable this profile after creation. // Its tensor parallel size comes from ExtraArgs and cannot change after creation. +// The ordinary role fields' editability does not reach this role: instanceType, resources and +// command are frozen with the profile, and replicas and size stay at one. message ModelDeploymentRoleElasticEP { // Width is the total number of GPU engines in the elastic collective, INCLUDING the // reserved API/DP-master member; width-1 of them are Ray-only workers. It is the total diff --git a/api/worker/v1alpha1/model_deployment.go b/api/worker/v1alpha1/model_deployment.go index 36fcea75b..338bef908 100644 --- a/api/worker/v1alpha1/model_deployment.go +++ b/api/worker/v1alpha1/model_deployment.go @@ -319,9 +319,12 @@ type ModelDeploymentKVTransfer struct { // nothing at all: it adds or removes instances, and every instance that stays keeps running, // keeps the accelerators it was admitted with and keeps whatever cache it holds. // -// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role. -// A replica's group is named from the deployment, its role and its ordinal, so edits to one role's -// running configuration do not rename a sibling role's groups. +// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role, +// and it refuses a kind change on a role that exists: the name and the kind are what the role IS, +// while size, instanceType, resources and command are how it is currently running and are +// editable beside every other container field. A replica's group is named from the deployment, +// its role and its ordinal, so edits to one role's running configuration do not rename a sibling +// role's groups. // // A DEPARTURE THIS OPERATOR DID NOT INITIATE IS NOT A ROLLOUT. The replica that left is replaced on // its own, under a new name, while its siblings keep serving — see @@ -381,10 +384,14 @@ type ModelDeploymentRole struct { // start together, they are replaced together, and none of them serves alone — the instance, // not the Pod, is the unit that appears and disappears. // - // THIS NUMBER IS FIXED AT CREATION AND CANNOT BE CHANGED. An instance's size is the shape of the - // instance, not a dial on it: the Pods a running instance is made of are not the Pods a different - // size asks for. Scaling is what replicas is for, and it leaves every running instance alone. To - // serve at a different size, create a deployment that declares it. + // IT IS EDITABLE. The edit replaces the role's replicas one at a time: an old replica and its + // Workload are deleted first, its replacement is created only after that slot is free, and the + // replacement's fresh admission gates deleting the next old replica. Surviving replicas keep + // their deployed configuration until each is selected for replacement; continuous serving is + // not guaranteed -- a one-replica role stops for the length of its swap. Scaling WITHOUT + // replacement is what replicas is for. + // + // THE ELASTIC-EP PROFILE PINS THIS NUMBER AT ONE; see ElasticEP. // // ABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods // is rendered with stable names and publishes the first Pod's address, this instance's size and @@ -407,6 +414,12 @@ type ModelDeploymentRole struct { // InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against. // It is what the queue-name entrance label is derived from. // + // IT IS EDITABLE. Moving a role to another type runs the same whole-group replacement: each + // old replica leaves before its replacement is created, and that replacement is admitted + // against the queue the new type publishes before the next old replica may leave. Existing + // Pods keep the type they were admitted with until they are deleted. The elastic-EP profile + // freezes this field after creation. + // // +required // +k8s:validation:minLength=1 // +k8s:validation:maxLength=253 @@ -424,6 +437,11 @@ type ModelDeploymentRole struct { // InstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards // a Pod wants is a property of the model being served, so two deployments on one InstanceType // routinely want different counts. + // + // IT IS EDITABLE. A change runs the same whole-group replacement, and the new request is + // admitted against the pool as each replacement is created. Omission behaves as on create: + // the accelerator half is defaulted, to one whole card of an acceleratable type. The + // elastic-EP profile freezes this field at the whole-card group it bound to TP at creation. Resources *ModelDeploymentRoleResources `json:"resources,omitempty" protobuf:"bytes,4,opt,name=resources"` // Image is the container image to run. Leaving it empty is the ordinary case: the operator then @@ -482,10 +500,11 @@ type ModelDeploymentRole struct { // there is deliberately no Args, because a second append tier beside ExtraArgs would have no // defined precedence. // - // IT IS FROZEN AFTER CREATION, because it decides whether the operator configures this role at - // all: a role that supplies one is taken over by its author, which changes cache injection and - // what status can claim. The rest of the container fields are how the build is fetched, shaped - // and tuned, and stay editable. + // IT IS EDITABLE IN EVERY DIRECTION: managed to take-over, take-over back to managed, and + // take-over to take-over. Each edit runs the whole-group replacement, and the deployed Pods + // keep the mode they were created with until each is deleted for replacement, so during a + // rollout the role can hold both modes at once; cache injection and status read what is + // deployed. The elastic-EP profile freezes this field: the elastic renderer owns the argv. // // +optional // +listType=atomic @@ -578,6 +597,8 @@ type ModelDeploymentRole struct { // admission refuses a decrease until Elastic EP scale-down support is complete. // The role cannot enable or disable this profile after creation. // Its tensor parallel size comes from ExtraArgs and cannot change after creation. +// The ordinary role fields' editability does not reach this role: instanceType, resources and +// command are frozen with the profile, and replicas and size stay at one. type ModelDeploymentRoleElasticEP struct { // Width is the total number of GPU engines in the elastic collective, INCLUDING the // reserved API/DP-master member; width-1 of them are Ray-only workers. It is the total diff --git a/api/worker/v1alpha1/model_deployment_test.go b/api/worker/v1alpha1/model_deployment_test.go index 05612c7cd..1c814827b 100644 --- a/api/worker/v1alpha1/model_deployment_test.go +++ b/api/worker/v1alpha1/model_deployment_test.go @@ -25,6 +25,52 @@ func TestModelDeploymentKVCacheIsOptional(t *testing.T) { assert.Equal(t, "object", spec.Properties["kvCache"].Type) } +// TestModelDeploymentRoleSizeBoundsSurviveMutability pins the schema bounds the mutable role +// contract keeps. size is editable, and every admitted value stays inside 1..64 with a default +// of one; replicas keeps its own ceiling. These are schema facts, so they are pinned where the +// schema is generated rather than assumed to have survived an annotation edit. +func TestModelDeploymentRoleSizeBoundsSurviveMutability(t *testing.T) { + role := roleSchema(t, "spec") + + size, ok := role.Properties["size"] + require.True(t, ok, "the schema has no spec.roles[].size") + require.NotNil(t, size.Default, "size carries a default") + assert.Equal(t, "1", string(size.Default.Raw), "an omitted size lands on one") + require.NotNil(t, size.Minimum, "size carries a lower bound") + assert.InDelta(t, 1, *size.Minimum, 0, "one Pod is the smallest instance") + require.NotNil(t, size.Maximum, "size carries an upper bound") + assert.InDelta(t, 64, *size.Maximum, 0, "the upper bound is the operator's own") + + replicas, ok := role.Properties["replicas"] + require.True(t, ok, "the schema has no spec.roles[].replicas") + require.NotNil(t, replicas.Default, "replicas carries a default") + assert.Equal(t, "1", string(replicas.Default.Raw)) + require.NotNil(t, replicas.Maximum, "replicas carries an upper bound") + assert.InDelta(t, 1024, *replicas.Maximum, 0) +} + +// TestModelDeploymentRolesRemainAMapOnName pins the role list's merge identity. Roles are +// listType=map keyed on name: reordering the list is not a change, and a name is what matches +// an update to the role it edits. The mutable role contract depends on exactly this identity. +func TestModelDeploymentRolesRemainAMapOnName(t *testing.T) { + roles := func(t *testing.T) extension.JSONSchemaProps { + t.Helper() + + crd := GetCustomResourceDefinitions()["ModelDeployment"] + require.NotNil(t, crd, "ModelDeployment is not registered") + require.Len(t, crd.Spec.Versions, 1) + + prop := crd.Spec.Versions[0].Schema.OpenAPIV3Schema.Properties["spec"].Properties["roles"] + require.NotNil(t, prop.Items, "roles is a list") + return prop + }(t) + + require.NotNil(t, roles.XListType, "roles carries a list type") + assert.Equal(t, "map", *roles.XListType, "reordering roles must not itself be a change") + assert.Equal(t, []string{"name"}, roles.XListMapKeys, + "a role's name is what matches an update to the role it edits") +} + // roleSchema returns the schema of one entry under the given top-level section, as the generated // CRD carries it. // diff --git a/api/worker/v1alpha1/zz_generated.crds.go b/api/worker/v1alpha1/zz_generated.crds.go index 725984a54..4990d2387 100644 --- a/api/worker/v1alpha1/zz_generated.crds.go +++ b/api/worker/v1alpha1/zz_generated.crds.go @@ -4813,7 +4813,7 @@ func crd_gpustack_api_worker_v1alpha1_ModelDeployment() *v1.CustomResourceDefini XListType: ptr.To[string]("atomic"), }, "command": { - Description: "Command replaces the whole argv, which is the TAKE-OVER tier: the user owns the whole\ncommand line, the operator synthesizes no engine argument and no client environment, the\nrole is marked unmanaged and CacheAttached goes to Unknown. Arguments fold into Command;\nthere is deliberately no Args, because a second append tier beside ExtraArgs would have no\ndefined precedence.\nIT IS FROZEN AFTER CREATION, because it decides whether the operator configures this role at\nall: a role that supplies one is taken over by its author, which changes cache injection and\nwhat status can claim. The rest of the container fields are how the build is fetched, shaped\nand tuned, and stay editable.", + Description: "Command replaces the whole argv, which is the TAKE-OVER tier: the user owns the whole\ncommand line, the operator synthesizes no engine argument and no client environment, the\nrole is marked unmanaged and CacheAttached goes to Unknown. Arguments fold into Command;\nthere is deliberately no Args, because a second append tier beside ExtraArgs would have no\ndefined precedence.\nIT IS EDITABLE IN EVERY DIRECTION: managed to take-over, take-over back to managed, and\ntake-over to take-over. Each edit runs the whole-group replacement, and the deployed Pods\nkeep the mode they were created with until each is deleted for replacement, so during a\nrollout the role can hold both modes at once; cache injection and status read what is\ndeployed. The elastic-EP profile freezes this field: the elastic renderer owns the argv.", Type: "array", Items: &v1.JSONSchemaPropsOrArray{ Schema: &v1.JSONSchemaProps{ @@ -4910,7 +4910,7 @@ func crd_gpustack_api_worker_v1alpha1_ModelDeployment() *v1.CustomResourceDefini XListType: ptr.To[string]("atomic"), }, "instanceType": { - Description: "InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against.\nIt is what the queue-name entrance label is derived from.", + Description: "InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against.\nIt is what the queue-name entrance label is derived from.\nIT IS EDITABLE. Moving a role to another type runs the same whole-group replacement: each\nold replica leaves before its replacement is created, and that replacement is admitted\nagainst the queue the new type publishes before the next old replica may leave. Existing\nPods keep the type they were admitted with until they are deleted. The elastic-EP profile\nfreezes this field after creation.", Type: "string", MaxLength: ptr.To[int64](253), MinLength: ptr.To[int64](1), @@ -5000,7 +5000,7 @@ func crd_gpustack_api_worker_v1alpha1_ModelDeployment() *v1.CustomResourceDefini Minimum: ptr.To[float64](1), }, "resources": { - Description: "Resources is what one Pod of this role asks of an accelerator, and it is a STRUCTURED\nFIELD FOR THE SAME REASON Replicas and InstanceType are: admission and scheduling read it.\nIt carries only the ACCELERATOR half of a request, because that is the only half a workload\ndecides. CPU, memory and ephemeral storage are DERIVED from the InstanceType's per-unit\nresources scaled by the requested card count, so they are not expressible here at all — a\nstronger guarantee than refusing them, since a field that does not exist cannot be shadowed by\nthe container fields below either.\nInstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards\na Pod wants is a property of the model being served, so two deployments on one InstanceType\nroutinely want different counts.", + Description: "Resources is what one Pod of this role asks of an accelerator, and it is a STRUCTURED\nFIELD FOR THE SAME REASON Replicas and InstanceType are: admission and scheduling read it.\nIt carries only the ACCELERATOR half of a request, because that is the only half a workload\ndecides. CPU, memory and ephemeral storage are DERIVED from the InstanceType's per-unit\nresources scaled by the requested card count, so they are not expressible here at all — a\nstronger guarantee than refusing them, since a field that does not exist cannot be shadowed by\nthe container fields below either.\nInstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards\na Pod wants is a property of the model being served, so two deployments on one InstanceType\nroutinely want different counts.\nIT IS EDITABLE. A change runs the same whole-group replacement, and the new request is\nadmitted against the pool as each replacement is created. Omission behaves as on create:\nthe accelerator half is defaulted, to one whole card of an acceleratable type. The\nelastic-EP profile freezes this field at the whole-card group it bound to TP at creation.", Type: "object", Properties: map[string]v1.JSONSchemaProps{ "accelerator": { @@ -5068,7 +5068,7 @@ func crd_gpustack_api_worker_v1alpha1_ModelDeployment() *v1.CustomResourceDefini XIntOrString: true, }, "size": { - Description: "ReplicaSize is how many Pods form ONE serving instance. Those Pods are fate-sharing: they\nstart together, they are replaced together, and none of them serves alone — the instance,\nnot the Pod, is the unit that appears and disappears.\nTHIS NUMBER IS FIXED AT CREATION AND CANNOT BE CHANGED. An instance's size is the shape of the\ninstance, not a dial on it: the Pods a running instance is made of are not the Pods a different\nsize asks for. Scaling is what replicas is for, and it leaves every running instance alone. To\nserve at a different size, create a deployment that declares it.\nABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods\nis rendered with stable names and publishes the first Pod's address, this instance's size and\neach Pod's own rank to every container. What an engine does with those facts -- which\nparallelism it turns on, and over how many ranks -- stays the author's to say.\nTHE GO IDENTIFIER IS NOT Size BECAUSE gogo protobuf generates a Size() method on this type and\nGo forbids a field and a method sharing a name; the API field is size.\nTHE UPPER BOUND IS THE SAME LIMIT REPLICAS CARRIES, AND IT MULTIPLIES WITH IT: this number is\nhow many Pods one instance is rendered as, so a pass renders replicas times this many before it\nwrites any of them. It is set far above the sizes an accelerator topology makes sense at, and\nfar below the range that turns one accepted field value into an out-of-memory worker.", + Description: "ReplicaSize is how many Pods form ONE serving instance. Those Pods are fate-sharing: they\nstart together, they are replaced together, and none of them serves alone — the instance,\nnot the Pod, is the unit that appears and disappears.\nIT IS EDITABLE. The edit replaces the role's replicas one at a time: an old replica and its\nWorkload are deleted first, its replacement is created only after that slot is free, and the\nreplacement's fresh admission gates deleting the next old replica. Surviving replicas keep\ntheir deployed configuration until each is selected for replacement; continuous serving is\nnot guaranteed -- a one-replica role stops for the length of its swap. Scaling WITHOUT\nreplacement is what replicas is for.\nTHE ELASTIC-EP PROFILE PINS THIS NUMBER AT ONE; see ElasticEP.\nABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods\nis rendered with stable names and publishes the first Pod's address, this instance's size and\neach Pod's own rank to every container. What an engine does with those facts -- which\nparallelism it turns on, and over how many ranks -- stays the author's to say.\nTHE GO IDENTIFIER IS NOT Size BECAUSE gogo protobuf generates a Size() method on this type and\nGo forbids a field and a method sharing a name; the API field is size.\nTHE UPPER BOUND IS THE SAME LIMIT REPLICAS CARRIES, AND IT MULTIPLIES WITH IT: this number is\nhow many Pods one instance is rendered as, so a pass renders replicas times this many before it\nwrites any of them. It is set far above the sizes an accelerator topology makes sense at, and\nfar below the range that turns one accepted field value into an out-of-memory worker.", Type: "integer", Format: "int32", Default: &v1.JSON{ diff --git a/api/worker/zz_generated.openapi.go b/api/worker/zz_generated.openapi.go index 1a59f8bea..1d22eae86 100644 --- a/api/worker/zz_generated.openapi.go +++ b/api/worker/zz_generated.openapi.go @@ -9821,7 +9821,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRole(ref common.Referenc return common.OpenAPIDefinition{ Schema: spec.Schema{ SchemaProps: spec.SchemaProps{ - Description: "ModelDeploymentRole is one engine role and its replicas.\n\nReplicas, InstanceType and Resources are STRUCTURED FIELDS AND MUST STAY SO. They are inputs to admission and scheduling — Kueue PodSet counts, flavor selection and the request the queue accounts — so a container field able to shadow any of them would make the admission feasibility check read a ledger that does not match reality. That is why the container fields below carry no resource request at all: the accelerator half belongs in Resources and the rest is derived from the InstanceType, and neither can be overridden here.\n\nEDITING A CONTAINER FIELD ROLLS THIS ROLE'S REPLICAS, and only this role's -- with one exception. The declared parallel degrees of a prefill or decode role -- the degree flags in ExtraArgs, and vLLM's VLLM_DP_SIZE environment entry -- are the one container field that can render into a document BOTH roles carry: the vLLM-Ascend transfer leg writes the same parallel blocks into both roles' Pods, so on that leg editing one role's degrees rewrites the other role's Pods too, and the edit rolls the pair. Where no shared document renders them, a degree edit stays this role's own like every other container-field edit: every replica is a Kueue pod group of its own, so they are replaced one at a time -- one per role per pass -- and every sibling role keeps serving throughout. A `replicas` change rolls nothing at all: it adds or removes instances, and every instance that stays keeps running, keeps the accelerators it was admitted with and keeps whatever cache it holds.\n\nTHE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role. A replica's group is named from the deployment, its role and its ordinal, so edits to one role's running configuration do not rename a sibling role's groups.\n\nA DEPARTURE THIS OPERATOR DID NOT INITIATE IS NOT A ROLLOUT. The replica that left is replaced on its own, under a new name, while its siblings keep serving — see docs/modules/model-deployment/deployment.md under \"One group per replica\" and \"Rollout is a rolling replacement\".", + Description: "ModelDeploymentRole is one engine role and its replicas.\n\nReplicas, InstanceType and Resources are STRUCTURED FIELDS AND MUST STAY SO. They are inputs to admission and scheduling — Kueue PodSet counts, flavor selection and the request the queue accounts — so a container field able to shadow any of them would make the admission feasibility check read a ledger that does not match reality. That is why the container fields below carry no resource request at all: the accelerator half belongs in Resources and the rest is derived from the InstanceType, and neither can be overridden here.\n\nEDITING A CONTAINER FIELD ROLLS THIS ROLE'S REPLICAS, and only this role's -- with one exception. The declared parallel degrees of a prefill or decode role -- the degree flags in ExtraArgs, and vLLM's VLLM_DP_SIZE environment entry -- are the one container field that can render into a document BOTH roles carry: the vLLM-Ascend transfer leg writes the same parallel blocks into both roles' Pods, so on that leg editing one role's degrees rewrites the other role's Pods too, and the edit rolls the pair. Where no shared document renders them, a degree edit stays this role's own like every other container-field edit: every replica is a Kueue pod group of its own, so they are replaced one at a time -- one per role per pass -- and every sibling role keeps serving throughout. A `replicas` change rolls nothing at all: it adds or removes instances, and every instance that stays keeps running, keeps the accelerators it was admitted with and keeps whatever cache it holds.\n\nTHE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role, and it refuses a kind change on a role that exists: the name and the kind are what the role IS, while size, instanceType, resources and command are how it is currently running and are editable beside every other container field. A replica's group is named from the deployment, its role and its ordinal, so edits to one role's running configuration do not rename a sibling role's groups.\n\nA DEPARTURE THIS OPERATOR DID NOT INITIATE IS NOT A ROLLOUT. The replica that left is replaced on its own, under a new name, while its siblings keep serving — see docs/modules/model-deployment/deployment.md under \"One group per replica\" and \"Rollout is a rolling replacement\".", Type: []string{"object"}, Properties: map[string]spec.Schema{ "name": { @@ -9854,7 +9854,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRole(ref common.Referenc }, "size": { SchemaProps: spec.SchemaProps{ - Description: "ReplicaSize is how many Pods form ONE serving instance. Those Pods are fate-sharing: they start together, they are replaced together, and none of them serves alone — the instance, not the Pod, is the unit that appears and disappears.\n\nTHIS NUMBER IS FIXED AT CREATION AND CANNOT BE CHANGED. An instance's size is the shape of the instance, not a dial on it: the Pods a running instance is made of are not the Pods a different size asks for. Scaling is what replicas is for, and it leaves every running instance alone. To serve at a different size, create a deployment that declares it.\n\nABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods is rendered with stable names and publishes the first Pod's address, this instance's size and each Pod's own rank to every container. What an engine does with those facts -- which parallelism it turns on, and over how many ranks -- stays the author's to say.\n\nTHE GO IDENTIFIER IS NOT Size BECAUSE gogo protobuf generates a Size() method on this type and Go forbids a field and a method sharing a name; the API field is size.\n\nTHE UPPER BOUND IS THE SAME LIMIT REPLICAS CARRIES, AND IT MULTIPLIES WITH IT: this number is how many Pods one instance is rendered as, so a pass renders replicas times this many before it writes any of them. It is set far above the sizes an accelerator topology makes sense at, and far below the range that turns one accepted field value into an out-of-memory worker.", + Description: "ReplicaSize is how many Pods form ONE serving instance. Those Pods are fate-sharing: they start together, they are replaced together, and none of them serves alone — the instance, not the Pod, is the unit that appears and disappears.\n\nIT IS EDITABLE. The edit replaces the role's replicas one at a time: an old replica and its Workload are deleted first, its replacement is created only after that slot is free, and the replacement's fresh admission gates deleting the next old replica. Surviving replicas keep their deployed configuration until each is selected for replacement; continuous serving is not guaranteed -- a one-replica role stops for the length of its swap. Scaling WITHOUT replacement is what replicas is for.\n\nTHE ELASTIC-EP PROFILE PINS THIS NUMBER AT ONE; see ElasticEP.\n\nABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods is rendered with stable names and publishes the first Pod's address, this instance's size and each Pod's own rank to every container. What an engine does with those facts -- which parallelism it turns on, and over how many ranks -- stays the author's to say.\n\nTHE GO IDENTIFIER IS NOT Size BECAUSE gogo protobuf generates a Size() method on this type and Go forbids a field and a method sharing a name; the API field is size.\n\nTHE UPPER BOUND IS THE SAME LIMIT REPLICAS CARRIES, AND IT MULTIPLIES WITH IT: this number is how many Pods one instance is rendered as, so a pass renders replicas times this many before it writes any of them. It is set far above the sizes an accelerator topology makes sense at, and far below the range that turns one accepted field value into an out-of-memory worker.", Minimum: ptr.To[float64](1), Maximum: ptr.To[float64](64), Type: []string{"integer"}, @@ -9863,7 +9863,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRole(ref common.Referenc }, "instanceType": { SchemaProps: spec.SchemaProps{ - Description: "InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against. It is what the queue-name entrance label is derived from.", + Description: "InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against. It is what the queue-name entrance label is derived from.\n\nIT IS EDITABLE. Moving a role to another type runs the same whole-group replacement: each old replica leaves before its replacement is created, and that replacement is admitted against the queue the new type publishes before the next old replica may leave. Existing Pods keep the type they were admitted with until they are deleted. The elastic-EP profile freezes this field after creation.", Default: "", MinLength: ptr.To[int64](1), MaxLength: ptr.To[int64](253), @@ -9873,7 +9873,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRole(ref common.Referenc }, "resources": { SchemaProps: spec.SchemaProps{ - Description: "Resources is what one Pod of this role asks of an accelerator, and it is a STRUCTURED FIELD FOR THE SAME REASON Replicas and InstanceType are: admission and scheduling read it.\n\nIt carries only the ACCELERATOR half of a request, because that is the only half a workload decides. CPU, memory and ephemeral storage are DERIVED from the InstanceType's per-unit resources scaled by the requested card count, so they are not expressible here at all — a stronger guarantee than refusing them, since a field that does not exist cannot be shadowed by the container fields below either.\n\nInstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards a Pod wants is a property of the model being served, so two deployments on one InstanceType routinely want different counts.", + Description: "Resources is what one Pod of this role asks of an accelerator, and it is a STRUCTURED FIELD FOR THE SAME REASON Replicas and InstanceType are: admission and scheduling read it.\n\nIt carries only the ACCELERATOR half of a request, because that is the only half a workload decides. CPU, memory and ephemeral storage are DERIVED from the InstanceType's per-unit resources scaled by the requested card count, so they are not expressible here at all — a stronger guarantee than refusing them, since a field that does not exist cannot be shadowed by the container fields below either.\n\nInstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards a Pod wants is a property of the model being served, so two deployments on one InstanceType routinely want different counts.\n\nIT IS EDITABLE. A change runs the same whole-group replacement, and the new request is admitted against the pool as each replacement is created. Omission behaves as on create: the accelerator half is defaulted, to one whole card of an acceleratable type. The elastic-EP profile freezes this field at the whole-card group it bound to TP at creation.", Ref: ref(v1alpha1.ModelDeploymentRoleResources{}.OpenAPIModelName()), }, }, @@ -9977,7 +9977,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRole(ref common.Referenc }, }, SchemaProps: spec.SchemaProps{ - Description: "Command replaces the whole argv, which is the TAKE-OVER tier: the user owns the whole command line, the operator synthesizes no engine argument and no client environment, the role is marked unmanaged and CacheAttached goes to Unknown. Arguments fold into Command; there is deliberately no Args, because a second append tier beside ExtraArgs would have no defined precedence.\n\nIT IS FROZEN AFTER CREATION, because it decides whether the operator configures this role at all: a role that supplies one is taken over by its author, which changes cache injection and what status can claim. The rest of the container fields are how the build is fetched, shaped and tuned, and stay editable.", + Description: "Command replaces the whole argv, which is the TAKE-OVER tier: the user owns the whole command line, the operator synthesizes no engine argument and no client environment, the role is marked unmanaged and CacheAttached goes to Unknown. Arguments fold into Command; there is deliberately no Args, because a second append tier beside ExtraArgs would have no defined precedence.\n\nIT IS EDITABLE IN EVERY DIRECTION: managed to take-over, take-over back to managed, and take-over to take-over. Each edit runs the whole-group replacement, and the deployed Pods keep the mode they were created with until each is deleted for replacement, so during a rollout the role can hold both modes at once; cache injection and status read what is deployed. The elastic-EP profile freezes this field: the elastic renderer owns the argv.", Type: []string{"array"}, Items: &spec.SchemaOrArray{ Schema: &spec.Schema{ @@ -10068,7 +10068,7 @@ func schema_gpustack_api_worker_v1alpha1_ModelDeploymentRoleElasticEP(ref common return common.OpenAPIDefinition{ Schema: spec.Schema{ SchemaProps: spec.SchemaProps{ - Description: "ModelDeploymentRoleElasticEP is the elastic-EP profile of one role.\n\nWidth is the only mutable field: it answers how large the collective currently should be, which is a running-state question. This release accepts increases and unchanged values; admission refuses a decrease until Elastic EP scale-down support is complete. The role cannot enable or disable this profile after creation. Its tensor parallel size comes from ExtraArgs and cannot change after creation.", + Description: "ModelDeploymentRoleElasticEP is the elastic-EP profile of one role.\n\nWidth is the only mutable field: it answers how large the collective currently should be, which is a running-state question. This release accepts increases and unchanged values; admission refuses a decrease until Elastic EP scale-down support is complete. The role cannot enable or disable this profile after creation. Its tensor parallel size comes from ExtraArgs and cannot change after creation. The ordinary role fields' editability does not reach this role: instanceType, resources and command are frozen with the profile, and replicas and size stay at one.", Type: []string{"object"}, Properties: map[string]spec.Schema{ "width": { diff --git a/docs/README.md b/docs/README.md index e3aa4392e..74f9b755e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -108,10 +108,10 @@ A skill link names a procedure; the entry's other fields carry shared facts. - Use for: describing a serving workload's model, engine and roles; prefill/decode pairing; replica routing, status and metrics. - Aliases: `ModelDeployment`, `prefill`, `decode`, `router`, `vLLM`, `SGLang`, `replica` - Guides: [Model Deployment](modules/model-deployment/_index.md); the role contract in [Model Deployment Configuration](modules/model-deployment/deployment.md); pairing in [Prefill and Decode](modules/model-deployment/prefill-decode.md); replica selection and Router observation in [Routing](modules/model-deployment/routing.md); what a status field means in [Status](modules/model-deployment/status.md); retirement and drain in [Shutdown](modules/model-deployment/shutdown.md); in-place collective resize in [Elastic EP](modules/model-deployment/elastic-ep.md); a recorded run in [Elastic EP Walkthrough](walkthroughs/model-deployment/elastic-ep.md); fixed per-rank HTTP serving in [External DP Walkthrough](walkthroughs/model-deployment/external-dp.md). -- Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md) +- Specs: [consolidation](../specs/2026-09-18-kvcache-and-model-deployment-consolidation.md) · [role replica admission](../specs/2026-09-19-role-replica-admission-unit.md) · [role configuration mutability](../specs/2026-10-08-model-deployment-role-mutability.md) · [P/D pairing and router](../specs/2026-09-12-model-deployment-pd-pairing-and-router.md) · [router implementations](../specs/2026-09-19-model-deployment-router-implementations.md) · [router qualification and drain](../specs/2026-10-01-s1-router-qualification-and-drain.md) · [elastic EP resize](../specs/2026-10-01-s2-elastic-ep-resize.md) - Code: `model_deployment_*.go` reconcilers under [pkg/worker/controllers/worker](../pkg/worker/controllers/worker/) · native resize client in [pkg/worker/elasticengine](../pkg/worker/elasticengine/) · webhooks in [pkg/worker/webhooks/worker](../pkg/worker/webhooks/worker/) · router image under [pack/llm-router](../pack/llm-router/) - Related: [KV Cache](#kv-cache) (a deployment can attach a shared cache) · [Model Delivery](#model-delivery) (weights reach replicas through an artifact) · [Topology Aware](#topology-aware) (the placement field constrains replicas) -- Skills: [gpustack-operator-e2e](../.agents/skills/gpustack-operator-e2e/SKILL.md) (serving cases) +- Skills: [gpustack-operator-e2e](../.agents/skills/gpustack-operator-e2e/SKILL.md) (serving cases and role update scheduling checks, case 115) ### GPU Instances diff --git a/docs/modules/devices/admission.md b/docs/modules/devices/admission.md index 49b8be593..7a897075b 100644 --- a/docs/modules/devices/admission.md +++ b/docs/modules/devices/admission.md @@ -324,6 +324,10 @@ edit touches only the InstanceType, never a Node or the ClusterQueue notes. `spec.cpu` (generic) or `spec.accelerator.cpu` (accelerated). - **Instance validating** — enforces the unit spec on **Create and Update**: a submission's RAM must not exceed `unitRAM × count`, its local storage not the InstanceType's `LocalStorage`. + - Changing `spec.type`, `spec.resources` or any other template field requires an already stored + `spec.stop: true`. Only `spec.volume` is immutable regardless of `spec.stop`. + Stop the Instance before editing its configuration. + Starting requires `status.phase: Stopped` and revalidates the updated resource request. - On a non-accelerated type, create and start also bound the CPU request by the pool's CPU **capacity** (`status.cpu.capacity`), never by what is unrequested right now: an Instance submitted while every core is requested is admitted and waits in its queue. diff --git a/docs/modules/kv-cache/pool.md b/docs/modules/kv-cache/pool.md index a21af8b67..8aaaf88d9 100644 --- a/docs/modules/kv-cache/pool.md +++ b/docs/modules/kv-cache/pool.md @@ -339,6 +339,8 @@ reason on `Releasable=False` before acting. They need different remedies: They are separate because the action differs: removing workloads, draining a domain and restoring an unanswering master are three different operations. +The API accepts the deletion request. The Binding remains in `Deleting` until the hold clears. + **A master that holds no tenant ledger releases the Binding rather than holding it.** With multi-tenancy declared off there is no ledger for a quota entry to be in, so the deletion strands nothing and completes, the same answer the pool's own teardown takes. Every other failed ledger request leaves diff --git a/docs/modules/model-deployment/deployment.md b/docs/modules/model-deployment/deployment.md index 828c71c8d..054b96634 100644 --- a/docs/modules/model-deployment/deployment.md +++ b/docs/modules/model-deployment/deployment.md @@ -70,10 +70,11 @@ they keep serving without interruption and keep whatever cache they hold. fate-sharing: they start together, they are admitted together, and they are replaced together. Use it when one instance genuinely spans hosts (tensor, pipeline, expert or sequence parallelism). -**`size` cannot be changed after creation.** The Pods a running instance is made of are not the Pods -a different size asks for, so no edit exists that does not replace every instance of the role at -once. To serve at a different size, create a deployment that declares it. Scaling is what `replicas` -is for, and it disturbs nothing already running. +**`size` is editable, and the edit replaces the role's instances.** An instance is never reshaped in +place: each old instance and its Workload are deleted first, and the replacement created in that +slot is admitted fresh before the next old instance may be deleted. See +[Rollout behavior](#rollout-behavior) for the cadence and its costs. Scaling without replacement is +what `replicas` is for, and it disturbs nothing already running. Above 1, the operator names each Pod of an instance `--r-m` and publishes them behind a headless Service per instance, so every Pod can address the others by a name @@ -124,8 +125,9 @@ The key follows the bound cache backend's effective group protocol and any direc transport the engine actually renders. Different backend-group protocols or an RDMA and EFA mix are refused for a positive count. -A count with no managed RDMA or EFA transfer leg is also refused. The count is frozen with the other -role resources. See [RDMA Operations](../rdma/operations.md) for the allocation and topology limits. +A count with no managed RDMA or EFA transfer leg is also refused, whenever the request, the +transfer settings or the pool the roles name change. See [RDMA Operations](../rdma/operations.md) +for the allocation and topology limits. ## Prefill and decode @@ -174,8 +176,8 @@ a role's own address. |---|---|---| | label `kueue.x-k8s.io/pod-group-name` | `gpustack-fnv64-` over the namespace, deployment, role and ordinal — always the hashed form, on every shape | membership: it is what makes a replica its own group. The name is unique, not parseable; the ordinal travels in its own label | | label `modeldeployment.gpustack.ai/pod-ordinal` | the replica's slot within its role, from 0 up | the per-replica identity: the group name derives from it, and a scale-down sheds the highest ordinals first | -| label `modeldeployment.gpustack.ai/member-index` | which Pod of its instance this is, from 0 up — **present only above `size: 1`** | what tells two Pods of one instance apart, and what a `Service` selector matches to front only the leader. A container reads its own index through this label rather than from a rendered value, which is what keeps one Pod template per instance. A Pod without the label is member 0 | -| annotation `kueue.x-k8s.io/pod-group-total-count` | the role's `size` | how many Pods Kueue waits for before composing anything. A replica-count change moves no total any member carries — a resize is a trim, not a rebuild | +| label `modeldeployment.gpustack.ai/member-index` | which Pod of its instance this is, from 0 up — rendered at **every** size, one Pod included | what tells two Pods of one instance apart, and what a `Service` selector matches to front only the leader above `size: 1`. A container reads its own index through this label rather than from a rendered value, which is what keeps one Pod template per instance | +| annotation `kueue.x-k8s.io/pod-group-total-count` | the role's `size` | how many Pods Kueue waits for before composing anything. A replica-count change moves no total any member carries — a resize is a trim, not a rebuild. A `size` edit never edits a running group's total either: it replaces whole groups, and each new group carries its own | | annotation `kueue.x-k8s.io/role-hash` | the role's `name` | names the single PodSet the replica's group composes, which is what lets status attribute a Workload back to the role that asked for it. Every replica of a role names the same PodSet | | annotation `kueue.x-k8s.io/pod-group-serving` | `"true"` | an inference deployment never finishes; without it Kueue reclaims the quota of a replica that exited | | label `kueue.x-k8s.io/queue-name` | the `status.entrance` **published by** the role's InstanceType | unchanged; Kueue refuses a group whose Pods disagree on it. Read from the type so this operator and the LocalQueue it creates cannot disagree about the queue | @@ -462,22 +464,103 @@ appearing as an unattributable `ImagePullBackOff`. Changing `replicas` adds or removes instances and nothing more: the survivors are not restarted, do not reload their weights and keep their cached blocks. What still replaces **every** instance of the -role is an edit that changes what a replica's Pod renders: `image`, `extraArgs`, `env`, `ports`, -`additionalVolumes`, `shmSize`, `terminationGracePeriodSeconds`. A change to `size` is not on that list because -it cannot be made: see `roles[].size` above. +role is an edit that changes what a replica's Pod renders: `size`, `instanceType`, `resources`, +`command`, `image`, `extraArgs`, `env`, `ports`, `additionalVolumes`, `shmSize`, +`terminationGracePeriodSeconds`. Such an edit **deletes and recreates** the role's replicas, one replica per role per pass, waited -out. The role set itself cannot be edited at all; admission refuses it, so there is no role rename or -addition to roll. There are no surge or unavailable knobs. +out. The role set itself cannot be edited at all; admission refuses a role added, removed, renamed +or moved to another kind, so there is none of that to roll. There are no surge or unavailable knobs. The one-at-a-time cadence is a constraint: each replica's group declares a total — the role's `size` — and a replacement created beside its still-counted member reads as excess, which Kueue answers by deleting the newer Pod, the replacement itself. A replacement is a **fresh admission**, not a rider on the reservation the departed replica held: -freeing the slot deletes that replica's Workload, and the reservation goes with it. The cadence guard -therefore turns a replica over only when every replica the role declares holds an admitted Workload; -on a full pool a rollout waits for capacity rather than shedding replicas it cannot re-reserve. +freeing the slot deletes that replica's Workload, and the reservation goes with it. + +The cadence guard therefore turns a healthy replica over only when every replica the role declares holds an +admitted Workload; on a full pool a rollout waits for capacity rather than shedding replicas it +cannot re-reserve. That gate holds back deleting the next healthy replica, not the configuration +already waiting in a replacement slot: a queued replacement is superseded by a newer edit and +admitted once, as the newer shape — the obsolete configuration need never be admitted. + +The slot is reused, but its obsolete Pods and Workload are deleted and replaced with fresh objects. + +A complete group still waiting for its first admission can also be replaced after a configuration edit. +It uses the same single replacement slot and fresh admission rules. +The operator selects this queued group before deleting an admitted sibling. +This exception requires every member to remain behind Kueue's admission gate. +An eviction, preemption, or earlier execution does not qualify as first admission. + +### Replacing a role whose shape moved + +`size`, `instanceType`, `resources` and `command` roll the same way as any Pod field, and each +carries consequences the Pod fields do not. + +**One unresolved replacement per role.** A replacement's fresh admission gates deleting the next +old replica, so a role turns over one replica at a time no matter how many of its fields one edit +moved: a patch changing `size`, `resources` and `command` together is one rollout, not three. The +role remembers which replica it is replacing until that replacement is admitted, so an operator +restart resumes the same rollout instead of selecting a second replica. + +**A broken replica does not jump the queue.** An old replica that loses a member while its role's +replacement is unresolved waits behind that replacement, then is rebuilt whole before another +healthy old replica is rolled. A replacement that itself loses a member is recovered in its own +slot. Recovery never mixes members: an interrupted replacement is completed with its own +configuration, and an old survivor is never filled with new-configuration members. + +**The old replicas run the configuration they were created with until each is deleted for +replacement.** Nothing reshapes a running instance: an instance of two Pods keeps its two members, +its leader address and its rank layout until it leaves. Cache injection, the unmanaged marker and +retirement read the deployed Pods, so a role part-way through a `command` edit serves through +replicas in both modes, and what status claims is what is actually deployed. + +A role moving between a managed command and a takeover keeps each running Pod's own mode. A managed +replica keeps its qualification rules. Once qualification is active, an Unknown observation keeps +existing eligibility without granting new eligibility; a definite fault still withdraws it. A role +whose qualification was never activated keeps its legacy readiness rule. + +A takeover replica keeps its legacy Service membership and stays `unmanaged: true`. Routing +membership does not establish engine health or cache qualification. + +**Mixed shapes can appear across instances, never within one.** A replacement is created only +after the instance it replaces has left — members and Workload — so no ordinal holds old and +new members at once. Replicas not yet selected keep serving their old shape beside replicas +already replaced onto the new one; above `size: 1` each shape's leader is picked by the +member-index label, so a leader of each shape can answer the same Service during that window. + +Peer DNS is per instance — one headless Service per replica — so the old and the new members +resolve their own shape's peers and never each other's. Each instance keeps its peer Service until +its members have left, so a terminating replica's members can still resolve one another. Each +deployed shape also answers its own way while both serve: a leader-served replica through its +leader, an External DP replica through its rank-carrying members. + +**A one-replica role has a gap.** Its only instance must leave before its replacement is admitted, +so serving stops until the replacement is ready. The replacement's Kueue admission gates the next +old replica's departure. Admission does not wait for engine readiness, so the next old replica may +leave while the previous replacement is still starting. + +A multi-replica role can temporarily have no serving capacity. Replacement does not guarantee +uninterrupted inference. + +**Siblings are untouched unless the edit reaches them.** An edit to one role's shape never rewrites +another role's groups: their Workloads, admission, member names and DNS stay as they are. + +The exception is the shared transfer document: on the vLLM-Ascend leg the parallel degrees one role +declares render into the document both roles carry, so a degree edit — through `extraArgs` or +through a replaced `command` — rolls the pair, and until both halves converge the pair can fail to +complete a transfer. See [Prefill and Decode](prefill-decode.md#direct-transfer-transport) for what +that window means. + +**Admission rejudges a moved dependency, not the world.** The interface and pool-transport checks +re-run for a role whose own interface request, type or command presence moved, for the roles whose +rendered direct-transfer leg a router or transfer edit actually changes — a router fronting no +prefill/decode pair renders none — and on a type move for the moved role. Everything else keeps its +admitted verdict, so a drifted store is repaired at the pool, not refused at the next edit. + +A role carrying `elasticEp` is excluded from all of this: the profile pins that role's shape, and +width is the only thing that moves ([Elastic EP](elastic-ep.md#changing-the-width)). The cost rides on the block lease described under [Workload impact](../kv-cache/injection.md#workload-impact): a lease survives a long queue and does **not** @@ -541,22 +624,26 @@ role set cannot be edited at all; admission refuses it. ### Deployment identity fields -Fields that identify the deployment are frozen. Fields that control how it runs are generally -editable; the table below lists both groups and the scheduling exceptions. +Fields that identify the deployment are frozen. Fields that control how it runs are editable; the +table below lists both groups. | Frozen | Editable | |---|---| | `model`, `engine.name`, `kvCache` | `engine.version`, `kvTransfer` | | `router.name` in place — the router block itself may be added or removed | `router.replicas`, `router.extraArgs` | -| the set of roles, and each role's `name` and `kind` | `roles[].replicas` | -| `roles[].size` | | -| `roles[].instanceType` | `roles[].extraArgs`, `roles[].env` | -| `roles[].resources` | the role's own Pod fields — `image`, `imagePullPolicy`, `imagePullSecrets`, `privileged`, `ports`, `additionalVolumes`, `shmSize`, `terminationGracePeriodSeconds` | -| `roles[].command` | labels and annotations | +| the set of roles, and each role's `name` and `kind` | `roles[].replicas`, `roles[].size` | +| on a role carrying `elasticEp`: its `size`, `instanceType`, `resources` and `command` | `roles[].instanceType`, `roles[].resources`, `roles[].command` | +| | `roles[].extraArgs`, `roles[].env` | +| | the role's own Pod fields — `image`, `imagePullPolicy`, `imagePullSecrets`, `privileged`, `ports`, `additionalVolumes`, `shmSize`, `terminationGracePeriodSeconds` | +| | labels and annotations | -`roles[].resources` is frozen because changing it renegotiates scheduling, much like deleting and -recreating the deployment. `roles[].privileged` stays editable because it controls how the role -runs. +The role's shape fields — `size`, `instanceType`, `resources`, `command` — are editable on the same +terms as the Pod fields: each edit replaces that role's replicas one at a time, deleting each old +instance before creating its replacement, whose fresh admission gates the next deletion. +`roles[].privileged` stays editable because it controls how the role runs. + +The elastic-EP row is the exception: the profile admits one fixed member shape, and +[width](elastic-ep.md#changing-the-width) is the only shape knob it offers. **To change a frozen field, create another deployment.** A frozen field is not a lock protecting a concurrent writer, and the refusal says so: what you are describing is a different deployment, so it @@ -566,14 +653,37 @@ cache-pool registration; a frozen field is none of those things. When adding a field, decide whether it identifies the deployment or controls how it runs before choosing its update rule. -> **A merge patch that omits a frozen field is an edit to that frozen field.** `roles` is a list, and -> `kubectl patch --type=merge` replaces a list wholesale rather than merging into it, so a role -> restated without its `command` sets `command` to null, and the edit is refused naming -> that field rather than the one you meant to change. +> **A merge patch replaces the `roles` list wholesale.** `kubectl patch --type=merge` follows +> RFC 7386, which replaces an array rather than merging into it: a role restated with only some of +> its fields loses the ones it did not carry. An omitted `size` lands on the default of 1 and an +> omitted `resources` lands on the accelerator default, exactly as a full-list rewrite says; a role +> restated without its `name` is refused for the missing name, and one restated without `instanceType` +> is refused for the empty value — none of these fields has a default that can stand in. +> `listType=map` does not soften this: it is what keeps role names unique and drives `kubectl apply`'s +> merge, not what a client-side merge patch does to an array. +> +> `kind` still binds too: a role restated without its `kind` picks up the default `Server`, so +> restating a prefiller that way is refused as a kind change rather than silently converting it. +> +> Change one field with a JSON patch, or restate the role in full: +> +> ```bash +> kubectl patch modeldeployment qwen-chat -n team-a --type=json \ +> -p '[{"op":"replace","path":"/spec/roles/0/size","value":2}]' +> ``` > -> Change one field with a JSON patch (`--type=json`, `/spec/roles/0/replicas`), or send the whole -> object with `kubectl apply` or `kubectl edit`. Omitting a value in a merge patch sets it to null, -> and the rule reads what you actually sent. +> ```yaml +> # --type=merge --patch-file=role.yaml: the whole role, not the field +> spec: +> roles: +> - name: server +> kind: Server +> replicas: 4 +> size: 2 +> instanceType: h20-8x +> resources: +> accelerator: "2" +> ``` ### One group per replica diff --git a/docs/modules/model-deployment/elastic-ep.md b/docs/modules/model-deployment/elastic-ep.md index f2a13f79f..a94d7f3e6 100644 --- a/docs/modules/model-deployment/elastic-ep.md +++ b/docs/modules/model-deployment/elastic-ep.md @@ -45,6 +45,9 @@ enforces the rest: - you cannot supply `command`, because the renderer owns the command line of the managed role; - there is no initial fabric to declare. +The role's `size`, `instanceType` and `resources` are frozen with the profile: the ordinary role +fields' editability does not reach it, and width is the only shape knob it offers. + Adding or removing the profile on an existing role is refused. The effective TP value is fixed after creation. Equivalent TP argument spellings are accepted. Omitting TP and explicitly setting TP=1 are equivalent. @@ -289,6 +292,8 @@ The operator retains existing elastic Pods, so an env edit does not change their - There is one serving instance per deployment. `replicas` and `size` are not a width mechanism. - Only the master serves HTTP. - No RDMA fabric is requested for this profile. +- The ordinary role fields' editability does not apply: the shape fields are frozen with the + profile, as described above. - Scale-down is not supported. A width decrease is rejected by admission; see [issue #741](https://github.com/gpustack/gpustack-operator/issues/741). **See also** — [Model Deployment Configuration](deployment.md) for the role contract · diff --git a/docs/modules/model-deployment/prefill-decode.md b/docs/modules/model-deployment/prefill-decode.md index 2c335d0b8..cdcfe4194 100644 --- a/docs/modules/model-deployment/prefill-decode.md +++ b/docs/modules/model-deployment/prefill-decode.md @@ -273,6 +273,10 @@ Editing it [turns over every role](deployment.md#rollout-behavior): the value re into both ends' arguments, so every role's replicas turn over one at a time. A prefiller and a decoder can disagree on the protocol until both converge, the same window an `engine.version` edit opens. +The same shared-document rule reaches the parallel degrees on the Ascend leg. The pair turns over +together; see [Rollout behavior](deployment.md#rollout-behavior) for the mechanism. Until both roles +converge, the pair can fail to complete a transfer. + ## Different hardware per role A Kueue Workload carries one `queueName`, and that name is the one the role's `instanceType` diff --git a/docs/modules/model-deployment/shutdown.md b/docs/modules/model-deployment/shutdown.md index efe2cec2b..ffc7651ae 100644 --- a/docs/modules/model-deployment/shutdown.md +++ b/docs/modules/model-deployment/shutdown.md @@ -17,6 +17,11 @@ a `replicas` reduction, a surplus shed, a dropped role — goes through [the retirement protocol](#the-retirement-protocol) first, and its Pod is deleted when that finishes. +An edit that replaces a role's configuration deletes that role's old replicas one at a time, and +each deleted replica drains exactly as below. The protocol reads a leaving replica as it is +deployed — the size and command it was created with — so an edit elsewhere does not change how it +is classified: a take-over replica is still held, a prefill or decode replica still refused. + `G` below is the role's grace: `roles[].terminationGracePeriodSeconds`, 30 when unset, written to the Pod's field of the same name. @@ -86,6 +91,12 @@ For `llm-d-router`, withdrawal also waits for recent dispatches reported for a t This applies after the Router removes that member from selection. These records do not increase the serving count. A dispatch to an unrelated survivor does not hold the target. +Without `spec.router`, withdrawal cannot observe established connections made directly through the +role's Service. An admitted, running replica stays held during scale-down or surplus removal. +The withdrawal budget expires after 30 seconds and retirement ends as `Aborted`, retaining its Pods, +Workload and capacity. Removing the Service endpoint does not prove those connections have ended. +Groups that never executed or received admission follow their separate cleanup path. + Two mechanisms are involved and they answer different questions. The protocol decides **whether the Pod may be deleted**; the [drain window](#the-drain-window) above decides **how long the engine takes to exit once it is**, on the kubelet's grace clock after the delete. A replica passes the diff --git a/docs/modules/model-deployment/status.md b/docs/modules/model-deployment/status.md index db58966cf..5137c5d98 100644 --- a/docs/modules/model-deployment/status.md +++ b/docs/modules/model-deployment/status.md @@ -20,6 +20,11 @@ Ready instances by kind: `R` is managed router Pods, `S` is ordinary servers, `P and `D` is decoders. For example, `1R2S` means one Ready router and two Ready servers; `1R3P4D` means one Ready router, three Ready prefillers and four Ready decoders. +**During a replacement, status combines desired and observed fields.** See +[Rollout behavior](deployment.md#rollout-behavior). `desired` and `parallelism` describe the current +spec; they can differ from running replicas while replacement waits. `ready` counts replicas still +serving their deployed configuration. `unmanaged` and cache conditions read deployed command mode. + Without a router, the `R` part is absent; a declared kind with nothing Ready shows zero. Several roles of the same kind are summed. A serving instance made of several Pods still counts once; use `status.roles[].desired` to see the requested count. diff --git a/go.mod b/go.mod index b9ae60303..f08b2804e 100644 --- a/go.mod +++ b/go.mod @@ -116,7 +116,7 @@ require ( github.com/evanphx/json-patch/v5 v5.9.11 // indirect github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect github.com/fatih/camelcase v1.0.0 // indirect - github.com/fatih/color v1.13.0 // indirect + github.com/fatih/color v1.18.0 // indirect github.com/felixge/httpsnoop v1.0.4 // indirect github.com/fsnotify/fsnotify v1.9.0 // indirect github.com/fxamacker/cbor/v2 v2.9.0 // indirect @@ -156,12 +156,15 @@ require ( github.com/jmoiron/sqlx v1.4.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.18.1 // indirect + github.com/kubeflow/mpi-operator v0.8.0 // indirect + github.com/kubeflow/trainer/v2 v2.2.0 // indirect + github.com/kubeflow/training-operator v1.9.3 // indirect github.com/kylelemons/godebug v1.1.0 // indirect github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 // indirect github.com/lann/ps v0.0.0-20150810152359-62de8c46ede0 // indirect github.com/lib/pq v1.11.2 // indirect github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect - github.com/mattn/go-colorable v0.1.13 // indirect + github.com/mattn/go-colorable v0.1.14 // indirect github.com/mattn/go-isatty v0.0.20 // indirect github.com/miekg/dns v1.1.72 // indirect github.com/mitchellh/copystructure v1.2.0 // indirect @@ -179,7 +182,9 @@ require ( github.com/peterbourgon/diskv v2.0.1+incompatible // indirect github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect + github.com/project-codeflare/appwrapper v1.2.0 // indirect github.com/prometheus/procfs v0.20.1 // indirect + github.com/ray-project/kuberay/ray-operator v1.6.0 // indirect github.com/rubenv/sql-migrate v1.8.1 // indirect github.com/russross/blackfriday/v2 v2.1.0 // indirect github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect @@ -221,8 +226,10 @@ require ( k8s.io/kms v0.35.3 // indirect oras.land/oras-go/v2 v2.6.0 // indirect sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.33.0 // indirect + sigs.k8s.io/jobset v0.11.1 // indirect sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect sigs.k8s.io/kustomize/api v0.20.1 // indirect sigs.k8s.io/kustomize/kyaml v0.20.1 // indirect sigs.k8s.io/randfill v1.0.0 // indirect + volcano.sh/apis v1.14.0 // indirect ) diff --git a/go.sum b/go.sum index fbe4ae66f..b57b67ade 100644 --- a/go.sum +++ b/go.sum @@ -101,6 +101,8 @@ github.com/fatih/camelcase v1.0.0 h1:hxNvNX/xYBp0ovncs8WyWZrOrpBNub/JfaMvbURyft8 github.com/fatih/camelcase v1.0.0/go.mod h1:yN2Sb0lFhZJUdVvtELVWefmrXpuZESvPmqwoZc+/fpc= github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w= github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk= +github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= +github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= github.com/foxcpp/go-mockdns v1.2.0 h1:omK3OrHRD1IWJz1FuFBCFquhXslXoF17OvBS6JPzZF0= @@ -241,6 +243,12 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/kubeflow/mpi-operator v0.8.0 h1:22EccUiO4VDooG4TooesdE8o8euGVj89PIRhiP24SRE= +github.com/kubeflow/mpi-operator v0.8.0/go.mod h1:1/YGVaFrbQd1a1tFsfE3qBcd4qU30numDnZmRnCy/jI= +github.com/kubeflow/trainer/v2 v2.2.0 h1:whChZckEtycLo1UKH/M0hiNaCUozbLvf9JrgmMv+TAg= +github.com/kubeflow/trainer/v2 v2.2.0/go.mod h1:i+FsPxb3zMYJ9LRf6ARMy2J12FGEU7Zvc+DgzHhtIE4= +github.com/kubeflow/training-operator v1.9.3 h1:aaSHqskOtCY8Dn8sVd+l9p5XAczW6O0nYMPjVLAw/lc= +github.com/kubeflow/training-operator v1.9.3/go.mod h1:6zI0hgeCOheiW5Z12IcVkKBuSWm714fz8mUvYu4Fid4= github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc= github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw= github.com/lann/builder v0.0.0-20180802200727-47ae307949d0 h1:SOEGU9fKiNWd/HOJuq6+3iTQz8KNCLtVX6idSoTLdUw= @@ -257,6 +265,8 @@ github.com/lithammer/dedent v1.1.0/go.mod h1:jrXYCQtgg0nJiN+StA2KgR7w6CiQNv9Fd/Z github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= +github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= +github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= @@ -313,6 +323,8 @@ github.com/poy/onpar v1.1.2 h1:QaNrNiZx0+Nar5dLgTVp5mXkyoVFIbepjyEoGSnhbAY= github.com/poy/onpar v1.1.2/go.mod h1:6X8FLNoxyr9kkmnlqpK6LSoiOtrO6MICtWwEuWkLjzg= github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g= github.com/prashantv/gostub v1.1.0/go.mod h1:A5zLQHz7ieHGG7is6LLXLz7I8+3LZzsrV0P1IAHhP5U= +github.com/project-codeflare/appwrapper v1.2.0 h1:MvenqEV/NuNOoq9SmttRABw0Yueiu8p3S0kFOybYeZo= +github.com/project-codeflare/appwrapper v1.2.0/go.mod h1:8e4sj4GJQTx88w19ZnvCBKbYwr6Rx6IN2MpvvMYIXWs= github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o= github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg= github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk= @@ -323,6 +335,8 @@ github.com/prometheus/otlptranslator v1.0.0 h1:s0LJW/iN9dkIH+EnhiD3BlkkP5QVIUVEo github.com/prometheus/otlptranslator v1.0.0/go.mod h1:vRYWnXvI6aWGpsdY/mOT/cbeVRBlPWtBNDb7kGR3uKM= github.com/prometheus/procfs v0.20.1 h1:XwbrGOIplXW/AU3YhIhLODXMJYyC1isLFfYCsTEycfc= github.com/prometheus/procfs v0.20.1/go.mod h1:o9EMBZGRyvDrSPH1RqdxhojkuXstoe4UlK79eF5TGGo= +github.com/ray-project/kuberay/ray-operator v1.6.0 h1:1hs7Di2U/vm09ot7MIU8jUqEo7gHDc4gkbv1eFvqCsQ= +github.com/ray-project/kuberay/ray-operator v1.6.0/go.mod h1:7vBLj6LNKuqbgERKugX3tsYm7uYRFwoE5BpQBUADeK0= github.com/redis/go-redis/extra/rediscmd/v9 v9.0.5 h1:EaDatTxkdHG+U3Bk4EUr+DZ7fOGwTfezUiUJMaIcaho= github.com/redis/go-redis/extra/rediscmd/v9 v9.0.5/go.mod h1:fyalQWdtzDBECAQFBJuQe5bzQ02jGd5Qcbgb97Flm7U= github.com/redis/go-redis/extra/redisotel/v9 v9.0.5 h1:EfpWLLCyXw8PSM2/XNJLjI3Pb27yVE+gIAfeqp8LUCc= @@ -588,6 +602,8 @@ sigs.k8s.io/cluster-inventory-api v0.0.0-20251028164203-2e3fabb46733 h1:l90ANqbl sigs.k8s.io/cluster-inventory-api v0.0.0-20251028164203-2e3fabb46733/go.mod h1:guwenlZ9iIfYlNxn7ExCfugOLTh6wjjRX3adC36YCmQ= sigs.k8s.io/controller-runtime v0.23.3 h1:VjB/vhoPoA9l1kEKZHBMnQF33tdCLQKJtydy4iqwZ80= sigs.k8s.io/controller-runtime v0.23.3/go.mod h1:B6COOxKptp+YaUT5q4l6LqUJTRpizbgf9KSRNdQGns0= +sigs.k8s.io/jobset v0.11.1 h1:WgLEBnXTeX+4MxfVkIbfm4jIVgjvIY6Jfaei2okxKNk= +sigs.k8s.io/jobset v0.11.1/go.mod h1:aPKgaoGXcxUVUQNB8RbuE7Q9pET9A9MDRsjvHO2lsH0= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg= sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg= sigs.k8s.io/kueue v0.17.1 h1:r4cwg3ZFF75l7RjVbdCJgM+0RpU0MJGp7t3rLH8Zyh4= @@ -605,3 +621,5 @@ sigs.k8s.io/structured-merge-diff/v6 v6.3.2/go.mod h1:M3W8sfWvn2HhQDIbGWj3S099Yo sigs.k8s.io/yaml v1.2.0/go.mod h1:yfXDCHCao9+ENCvLSE62v9VSji2MKu5jeNfTrofGhJc= sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs= sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4= +volcano.sh/apis v1.14.0 h1:Rf8wzGFmgzbyg85GO0CxapDDEk/VM3D7QbjbI/sH+OQ= +volcano.sh/apis v1.14.0/go.mod h1:CKQbxVt0o4lTKisC0MonoXWruGFC0S8KU+UuzaZ5E7k= diff --git a/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentrole.go b/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentrole.go index 48bd75510..4d349e5c3 100644 --- a/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentrole.go +++ b/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentrole.go @@ -33,9 +33,12 @@ import ( // nothing at all: it adds or removes instances, and every instance that stays keeps running, // keeps the accelerators it was admitted with and keeps whatever cache it holds. // -// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role. -// A replica's group is named from the deployment, its role and its ordinal, so edits to one role's -// running configuration do not rename a sibling role's groups. +// THE SET OF ROLES IS FIXED AFTER CREATION. Admission refuses adding, removing or renaming a role, +// and it refuses a kind change on a role that exists: the name and the kind are what the role IS, +// while size, instanceType, resources and command are how it is currently running and are +// editable beside every other container field. A replica's group is named from the deployment, +// its role and its ordinal, so edits to one role's running configuration do not rename a sibling +// role's groups. // // A DEPARTURE THIS OPERATOR DID NOT INITIATE IS NOT A ROLLOUT. The replica that left is replaced on // its own, under a new name, while its siblings keep serving — see @@ -80,10 +83,14 @@ type ModelDeploymentRoleApplyConfiguration struct { // start together, they are replaced together, and none of them serves alone — the instance, // not the Pod, is the unit that appears and disappears. // - // THIS NUMBER IS FIXED AT CREATION AND CANNOT BE CHANGED. An instance's size is the shape of the - // instance, not a dial on it: the Pods a running instance is made of are not the Pods a different - // size asks for. Scaling is what replicas is for, and it leaves every running instance alone. To - // serve at a different size, create a deployment that declares it. + // IT IS EDITABLE. The edit replaces the role's replicas one at a time: an old replica and its + // Workload are deleted first, its replacement is created only after that slot is free, and the + // replacement's fresh admission gates deleting the next old replica. Surviving replicas keep + // their deployed configuration until each is selected for replacement; continuous serving is + // not guaranteed -- a one-replica role stops for the length of its swap. Scaling WITHOUT + // replacement is what replicas is for. + // + // THE ELASTIC-EP PROFILE PINS THIS NUMBER AT ONE; see ElasticEP. // // ABOVE ONE, THE PODS OF AN INSTANCE NEED EACH OTHER'S ADDRESSES, so an instance of several Pods // is rendered with stable names and publishes the first Pod's address, this instance's size and @@ -100,6 +107,12 @@ type ModelDeploymentRoleApplyConfiguration struct { ReplicaSize *int32 `json:"size,omitempty"` // InstanceType is the name of the InstanceType whose pool this role's Pods are admitted against. // It is what the queue-name entrance label is derived from. + // + // IT IS EDITABLE. Moving a role to another type runs the same whole-group replacement: each + // old replica leaves before its replacement is created, and that replacement is admitted + // against the queue the new type publishes before the next old replica may leave. Existing + // Pods keep the type they were admitted with until they are deleted. The elastic-EP profile + // freezes this field after creation. InstanceType *string `json:"instanceType,omitempty"` // Resources is what one Pod of this role asks of an accelerator, and it is a STRUCTURED // FIELD FOR THE SAME REASON Replicas and InstanceType are: admission and scheduling read it. @@ -113,6 +126,11 @@ type ModelDeploymentRoleApplyConfiguration struct { // InstanceType alone cannot supply this half: its UnitResources size ONE card, and how many cards // a Pod wants is a property of the model being served, so two deployments on one InstanceType // routinely want different counts. + // + // IT IS EDITABLE. A change runs the same whole-group replacement, and the new request is + // admitted against the pool as each replacement is created. Omission behaves as on create: + // the accelerator half is defaulted, to one whole card of an acceleratable type. The + // elastic-EP profile freezes this field at the whole-card group it bound to TP at creation. Resources *ModelDeploymentRoleResourcesApplyConfiguration `json:"resources,omitempty"` // Image is the container image to run. Leaving it empty is the ordinary case: the operator then // synthesizes one from the pool's accelerator backend, the observed runtime version and the @@ -140,10 +158,11 @@ type ModelDeploymentRoleApplyConfiguration struct { // there is deliberately no Args, because a second append tier beside ExtraArgs would have no // defined precedence. // - // IT IS FROZEN AFTER CREATION, because it decides whether the operator configures this role at - // all: a role that supplies one is taken over by its author, which changes cache injection and - // what status can claim. The rest of the container fields are how the build is fetched, shaped - // and tuned, and stay editable. + // IT IS EDITABLE IN EVERY DIRECTION: managed to take-over, take-over back to managed, and + // take-over to take-over. Each edit runs the whole-group replacement, and the deployed Pods + // keep the mode they were created with until each is deleted for replacement, so during a + // rollout the role can hold both modes at once; cache injection and status read what is + // deployed. The elastic-EP profile freezes this field: the elastic renderer owns the argv. Command []string `json:"command,omitempty"` // ExtraArgs is appended AFTER the operator-synthesized arguments. An entry naming a key the // operator owns is REJECTED rather than merged: a silent merge produces two values for one diff --git a/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentroleelasticep.go b/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentroleelasticep.go index 498e44fed..0330dc63a 100644 --- a/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentroleelasticep.go +++ b/pkg/kubeclients/applyconfiguration/worker/v1alpha1/modeldeploymentroleelasticep.go @@ -12,6 +12,8 @@ package v1alpha1 // admission refuses a decrease until Elastic EP scale-down support is complete. // The role cannot enable or disable this profile after creation. // Its tensor parallel size comes from ExtraArgs and cannot change after creation. +// The ordinary role fields' editability does not reach this role: instanceType, resources and +// command are frozen with the profile, and replicas and size stay at one. type ModelDeploymentRoleElasticEPApplyConfiguration struct { // Width is the total number of GPU engines in the elastic collective, INCLUDING the // reserved API/DP-master member; width-1 of them are Ray-only workers. It is the total diff --git a/pkg/worker/controllers/worker/model_deployment.go b/pkg/worker/controllers/worker/model_deployment.go index eb9cfa285..73653fe8b 100644 --- a/pkg/worker/controllers/worker/model_deployment.go +++ b/pkg/worker/controllers/worker/model_deployment.go @@ -3,6 +3,7 @@ package worker import ( "context" "fmt" + "maps" "slices" "strings" "sync" @@ -560,16 +561,54 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( }) createOrdinals := make(map[string][]int, len(md.Spec.Roles)) + // partialCreates are the members filled into the group a replacement slot is building, gathered + // from the same decision below that admits a partial create as fillable at all. + partialCreates := make(map[string][]*core.Pod, len(md.Spec.Roles)) // The ordinals this pass condemns for replacement, gathered here so the whole-group health // predicate can ask "is this replica on its way out" from the same decision the rollout makes // rather than from a second comparison that could reach a different answer. + // + // Only a replica this pass selected is recorded. An outdated replica nobody has chosen is + // still serving, and failing its qualification would withdraw its endpoint before any + // replacement exists. replacedByRole := make(map[string]map[int]bool, len(md.Spec.Roles)) + // holdingRoles are the roles whose replacement has not been admitted yet. They take part in + // every decision below except selecting a further healthy replica. + holdingRoles := make(map[string]modelDeploymentReplacementSlot, len(md.Spec.Roles)) + slots := modelDeploymentReplacementSlotsOf(md) for i := range md.Spec.Roles { role := &md.Spec.Roles[i] live := liveByRole[role.Name] declared := int(role.Replicas) want := desired[role.Name] + // A ROLE ALREADY REPLACING ONE REPLICA DOES NOT SELECT ANOTHER until that replacement + // holds an admission. A queued one holds nothing, so spending a second healthy replica + // behind it would leave the role with less admitted capacity every pass it runs. + if slot, held := slots.slotFor(role.Name); held { + resolved, resolveErr := r.resolveReplacementSlot(ctx, md, slot, role, want[slot.Ordinal]) + if resolveErr != nil { + return objectWriteResult( + logger, resolveErr, "resolve the replacement slot", _requeueAfterConflict) + } + if resolved.cleared { + // THE CLEARING IS PERSISTED, not just dropped from this pass's view: a slot left on + // the object would keep holding the role forever, because the next pass would read + // the same unresolved replacement this one just resolved. + if err = r.endReplacementSlot(ctx, md, role.Name); err != nil { + return objectWriteResult( + logger, err, "clear the replacement slot", _requeueAfterConflict) + } + slots = slots.clearRole(role.Name) + } else { + slots = slots.setSlot(resolved.slot) + holdingRoles[role.Name] = resolved.slot + } + if resolved.requeue { + requeue = true + } + } + // THE ORDINALS THE SPEC STILL NAMES KEEP THEIR PODS, and the ones it no longer names go -- // highest first, so a scale-down sheds the youngest slots and the ordinals stay dense from // zero. Each departing replica takes its own Workload with it: its group is its own, so no @@ -582,12 +621,9 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // THE COUNTS BELOW ARE IN TWO DIFFERENT UNITS, and keeping them apart is what makes a role // of multi-Member replicas arithmetically the same as one of single-Member replicas. // `declared` counts REPLICAS, which is what the spec states and what a scale moves. - // `declaredPods` counts the Pods those replicas are made of, which is what a list of live - // Pods can be compared against. At size one the two are equal, which is why every decision - // here reads identically for the shape that existed before this one. - size := modelDeploymentRoleSize(role) - declaredPods := declared * size - + // The desired member size is not used to measure the live set at all: it describes what a + // replacement will be built at, while the running replicas were built at whatever the spec + // said when they were created. removed := make([]*core.Pod, 0) kept := make([]*core.Pod, 0, len(live)) occupied := make(map[int]bool, declared) @@ -649,7 +685,7 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // nothing separates, which is determinism rather than meaning. Each departing replica takes // its Workload with it: leaving the excess standing would have Kueue evict a member of its // own choosing on every pass. - if surplus := len(kept) - declaredPods; surplus > 0 { + if surplus := modelDeploymentSurplusCount(kept); surplus > 0 { shedSurplus := modelDeploymentSurplusReplicas(kept, want, surplus) // THE SHED IS ONE DECISION AND IT IS NOT PARTLY TAKEN. The protocol is handed the // whole set rather than the first member of it, because a replica is removed as a @@ -758,36 +794,63 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( outdatedByOrdinal[ordinal] = true } - // A REPLICA SHORT OF ITS MEMBERS IS OUTDATED TOO, and nothing above notices it: every Pod - // that IS there can carry the current fingerprint while the replica is still missing a - // member, and a replica Kueue cannot compose a Workload for is not one the deployment is - // serving from. It is condemned on the same terms as one carrying a stale render, which - // keeps the repair path single. - seen := make(map[int]int, len(occupied)) - for _, pod := range kept { - if ordinal, ok := modelDeploymentPodOrdinal(pod); ok { - if retirement.names(role.Name, ordinal) { - continue - } - seen[ordinal]++ - } - } + // A REPLICA SHORT OF THE MEMBERS IT DECLARES IS OUTDATED TOO, and nothing above notices it: + // every Pod that IS there can carry the current fingerprint while the replica is still + // missing a member, and a replica Kueue cannot compose a Workload for is not one the + // deployment is serving from. It is condemned on the same terms as one carrying a stale + // render, which keeps the repair path single. + // + // The total compared is the one the members carry, never the role's desired size: a role + // edited from four members to two leaves every running replica whole at four, and reading + // that as loss would condemn them all at once. incompleteByOrdinal := make(map[int]bool, len(occupied)) - for ordinal := range occupied { + for _, view := range modelDeploymentGroupPods(modelDeploymentPodValues(kept), true) { + if !view.Seated || !occupied[view.Ordinal] { + continue + } // AN ORDINAL THE RETIREMENT PROTOCOL OWNS IS NOT THIS RULE'S BUSINESS. The rollout // condemns a replica whose members do not agree with the spec, and while a reservation - // is in flight the agreement is not yet decided -- the protocol is removing one member of - // a doubled ordinal, which leaves the ordinal looking over-complete to this rule for as - // long as the operation takes. Condemning it here would delete the whole ordinal, and + // is in flight the agreement is not yet decided -- the protocol is removing one member + // of a doubled ordinal, which leaves the ordinal looking over-complete to this rule for + // as long as the operation takes. Condemning it here would delete the whole ordinal, and // with it the member the surplus rule existed to keep: the rollout would undo the // surplus rule's careful choice with a blunter one, on a state the surplus rule had // already resolved. - if retirement.names(role.Name, ordinal) { + if retirement.names(role.Name, view.Ordinal) { continue } - if seen[ordinal] != size { - outdatedByOrdinal[ordinal] = true - incompleteByOrdinal[ordinal] = true + if modelDeploymentDeployedReplicaShape(view).State == modelDeploymentReplicaShapeIncomplete { + outdatedByOrdinal[view.Ordinal] = true + incompleteByOrdinal[view.Ordinal] = true + } + } + + // A PARTIAL CREATE IS FILLED ONLY IN THE GROUP A SLOT IS BUILDING, and only while that + // group is the current render and holds nothing admitted. A create that failed leaves the + // replacement one member short of its total, and Kueue composes no Workload for an + // incomplete group, so the slot would otherwise wait on a group that can never be granted + // anything. + // + // IT IS NEVER FILLED BESIDE A GROUP RUNNING AN EARLIER CONFIGURATION. Equal size is not + // compatibility: a member rendered from the current spec beside one rendered from the + // previous one is a group whose members disagree about what they are, which is not a state + // this operator can render, admit or account for. So a replica that lost a member under an + // older render is not filled here -- it waits behind the slot below and is replaced whole. + // + // The members are those of the slot's own ordinal and no other, and the group must hold no + // Workload: a group with one standing has proved no admission, and a group with a Workload + // on it is a group Kueue has already composed, where topping it up is Kueue's business and + // not this operator's. + if slot, holding := holdingRoles[role.Name]; holding { + partial, fillable, err := r.modelDeploymentPartialGroupMembers(ctx, md, slot, want[slot.Ordinal]) + if err != nil { + logger.Error(err, "read the replacing replica's group", "role", role.Name, + "ordinal", slot.Ordinal) + + return ctrl.Result{}, err + } + if fillable { + partialCreates[role.Name] = append(partialCreates[role.Name], partial...) } } @@ -797,10 +860,6 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( if outdatedByOrdinal[ordinal] { rollout.outdated++ outdated = append(outdated, modelDeploymentReplicaMembers(kept, ordinal)...) - if replacedByRole[role.Name] == nil { - replacedByRole[role.Name] = make(map[int]bool) - } - replacedByRole[role.Name][ordinal] = true } } @@ -885,9 +944,9 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // no declared count covers and the barrier does not catch. The guard therefore turns a // replica over only when every replica the role declares holds an admitted Workload: one // admission is in flight at a time, and the pass comes back for the next departure once - // the replacement has actually been admitted. A deployment whose replicas hold no - // admitted Workloads at all never rolls -- without admission there is no capacity to - // trade, and an edit waits for it rather than stripping what serves. + // the replacement has actually been admitted. A complete group still waiting for its first + // admission can be replaced without spending admitted capacity. Previously admitted groups + // keep this guard after their admission is revoked. // // THE DEPARTING REPLICA'S WORKLOAD IS DELETED WITH IT, and that is not bookkeeping. The // group is annotated serving, so Kueue never releases the finalizer it holds on the Pod @@ -930,11 +989,27 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // of the role, ordinal-less ones included, which would be the wrong denominator -- but every // such Pod is outdated by the rule above, so reaching the comparison at all means there are // none, and the count is over seated replicas exactly when it is read. + // TWO KINDS OF REPLICA TURN OVER WITHOUT THE ADMISSION A HEALTHY ONE NEEDS, and they are kept + // apart because they are not waited for the same way. + // + // A POD CLAIMING NO ORDINAL holds no seat and no admission, and no ordinal a slot could name + // describes it, so there is nothing for it to wait behind: it rolls on its own path here. + // + // A REPLICA SHORT OF ITS MEMBERS is broken, and a broken replica BEHIND A SLOT WAITS FOR IT. + // The slot's replacement has to be admitted first, because that is the one group the role is + // committed to; a broken group behind it is not admitted and will not be, and taking it now + // would delete the member it still holds to rebuild a group whose remaining members are + // already obsolete. Once the slot clears it is taken whole, ahead of any healthy outdated + // replica -- see the pick below. ungated := make([]*core.Pod, 0, len(outdated)) + broken := make([]*core.Pod, 0, len(outdated)) for _, pod := range outdated { ordinal, ok := modelDeploymentPodOrdinal(pod) - if !ok || incompleteByOrdinal[ordinal] { + switch { + case !ok: ungated = append(ungated, pod) + case incompleteByOrdinal[ordinal]: + broken = append(broken, pod) } } @@ -949,41 +1024,121 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // is what leaves. Same pick as below -- highest ordinal -- so two passes over one state // still choose the same Pod, and a seated replica goes before one holding no ordinal // because the pick reads a missing ordinal as below every seat. + _, holding := holdingRoles[role.Name] + admitted := modelDeploymentAdmittedReplicas(workloads, live) + // A BROKEN REPLICA IS TAKEN WHOLE AND TAKEN FIRST, AHEAD OF ANY HEALTHY OLD REPLICA. + // It is the only one of the two that is already down, so leaving it for a healthy + // replica's turn is how a role ends up replacing capacity that was serving while the + // capacity that was lost waits. The ordinal it sits at does not enter into it: a broken + // replica below a healthy one is the more urgent of the two. + brokenTurnsOver := len(broken) > 0 && !holding unguarded := len(ungated) > 0 + var initiallyQueued []*core.Pod + if !holding && !brokenTurnsOver && !unguarded && admitted != declared { + for _, view := range modelDeploymentGroupPodsByReplica(modelDeploymentPodValues(outdated)) { + queued, queueErr := r.initialQueuedReplica(ctx, view, workloads) + if queueErr != nil { + return ctrl.Result{}, queueErr + } + if queued { + initiallyQueued = append(initiallyQueued, view.Members...) + } + } + } pick := outdated - if unguarded { + switch { + case brokenTurnsOver: + pick = broken + case unguarded: pick = ungated + case len(initiallyQueued) > 0: + pick = initiallyQueued } - if admitted := modelDeploymentAdmittedReplicas(workloads, live); unguarded || - admitted == declared { + // A ROLE WHOSE REPLACEMENT IS NOT ADMITTED YET SPENDS NO FURTHER HEALTHY CAPACITY. The + // queued replacement holds nothing, so a second deletion behind it would remove another + // admitted replica on every pass until the role held none. + // + // The hold covers the broken replicas too: the slot's own replacement is the group the + // role is committed to, and a broken group behind it is repaired whole once that one is + // admitted rather than being dismantled to save a pass. + turnsOver := brokenTurnsOver || unguarded || len(initiallyQueued) > 0 || (!holding && admitted == declared) + if turnsOver { // THE WHOLE REPLICA TURNS OVER, not the member that was picked. The pick names one // Pod, and that Pod's replica is what departs: its members hold one Kueue group, // so a delete of some of them leaves a group Kueue will not admit and will not // release, and the Workload delete below would stop the rest regardless. pod := modelDeploymentHighestOrdinalReplica(pick) + ordinal, seated := modelDeploymentPodOrdinal(pod) departing := []*core.Pod{pod} - if ordinal, ok := modelDeploymentPodOrdinal(pod); ok { + if seated { departing = modelDeploymentReplicaMembers(pick, ordinal) } - logger.Info("recreating replica", - "pod", pod.Name, "members", len(departing), - "why", map[bool]string{true: "short of its members", false: "built from an earlier spec"}[unguarded]) - departed := make([]core.Pod, 0, len(departing)) - for _, member := range departing { - if err = r.Client.Delete(ctx, member); err != nil && !kerrors.IsNotFound(err) { - logger.Error(err, "delete outdated replica member", "pod", member.Name) + // A POD CLAIMING NO ORDINAL IS NOT ANY SLOT'S, and a slot keyed on one cannot + // describe it: its vacancy question is about an ordinal its members do not sit + // on, so it would wait on a group the Pod never belonged to. Such a Pod rolls on + // the ungated repair path alone, where the group name it does carry is the evidence. + held, hasSlot := slots.slotFor(role.Name) + switch { + case hasSlot && seated && held.Ordinal == ordinal: + if err = r.cleanReplacementSlot(ctx, md, held, departing); err != nil { + logger.Error(err, "clean the replacing replica", + "role", role.Name, "ordinal", ordinal) + return ctrl.Result{}, err + } + case seated: + logger.Info("recreating replica", + "pod", pod.Name, "members", len(departing), + "why", map[bool]string{true: "short of its members", false: "built from an earlier spec"}[brokenTurnsOver]) + + // The slot is written before any delete, and it is what a restart between + // the delete and the create recovers. + slot := replacementCapturedIdentities(modelDeploymentReplicaView{ + Role: role.Name, + Ordinal: ordinal, + Members: departing, + }, workloads) + if err = r.beginReplacementSlot(ctx, md, slot); err != nil { + return objectWriteResult( + logger, err, "record the replacement slot", _requeueAfterConflict) + } + slots = slots.setSlot(slot) + + if err = r.cleanReplacementSlot(ctx, md, slot, departing); err != nil { + logger.Error(err, "clean the replaced replica", + "role", role.Name, "ordinal", ordinal) + return ctrl.Result{}, err + } + + default: + logger.Info("recreating replica", + "pod", pod.Name, + "why", map[bool]string{ + true: "holds no ordinal, so no slot can name it", + false: "short of its members", + }[unguarded]) + departed := make([]core.Pod, 0, 1) + uid := pod.UID + if err = r.Client.Delete(ctx, pod, ctrlcli.Preconditions{UID: &uid}); err != nil && + !kerrors.IsNotFound(err) { + logger.Error(err, "delete the ordinal-less replica member", "pod", pod.Name) + return ctrl.Result{}, err + } + departed = append(departed, *pod) + if err = r.deleteModelDeploymentGroupWorkload(ctx, md, departed); err != nil { + logger.Error(err, "delete the departing replica's workload", "pod", pod.Name) return ctrl.Result{}, err } - departed = append(departed, *member) } - // One call for the whole replica: its members share a single Workload, so this is - // one delete however many of them there were. - if err = r.deleteModelDeploymentGroupWorkload(ctx, md, departed); err != nil { - logger.Error(err, "delete the departing replica's workload", "pod", pod.Name) - return ctrl.Result{}, err + + if seated { + if replacedByRole[role.Name] == nil { + replacedByRole[role.Name] = make(map[int]bool) + } + replacedByRole[role.Name][ordinal] = true } + requeue = true } // Otherwise the rollout holds rather than proceeding short, and it asks for no requeue. @@ -992,23 +1147,8 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // API server, which is at least as fresh as the cache the event came from. } - // THE CREATE GATE, PER MISSING ORDINAL: an ordinal with no live Pod in the cached list is - // created for only once the API server holds no Pod for it either. The same read answers - // the two questions a missing ordinal carries -- whether a create this deployment already - // issued is standing on the server unseen by the cache, and whether a departing holder has - // finished leaving -- and it is read ON THE API SERVER for both, because the cache lags the - // server in exactly the window each question lives in. - // - // AN EARLIER GATE HERE ASKED THE ORDINAL'S WORKLOAD WHETHER KUEE WANTED A REPLACEMENT, AND - // NO REPLACEMENT PATH CONSULTS THAT ASK ANY MORE, because on this design the ask has no - // object to be asked of. Freeing an ordinal deletes that replica's Workload with the Pod, - // so by the time the Pod is gone there is no Workload left to read a verdict from -- an ask - // gated on it would be a condition that is never true, a wait with no exit. Waiting for the - // vacancy itself replaces the ask and covers it strictly: while the departing Pod is still - // on the server the ask could already read True, yet creating then seats two members in a - // one-member group and Kueue's answer to that excess is to delete the newest gated Pod -- - // the replacement itself. - // + // A reused ordinal requires both Pod and Workload vacancy on the API server. + // The cache can miss a successful create or a holder still leaving behind its finalizer. // THE MISSING ORDINALS ARE FILLED LOWEST FIRST, and only as many as the count is short: a // role short by one with two ordinals free -- a pre-per-replica pod still serving among // them -- creates the lower slot and lets the surplus pod's own departure settle the other. @@ -1018,17 +1158,42 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // as present here even if it is short a member; the rollout above is what repairs that, // because filling the gap in place would seat a member in a group Kueue already refused. if missing := declared - len(occupied) - orphans; missing > 0 { - // ONE UNCACHED READ FOR THE ROLE, taken before the loop rather than inside it: every - // free ordinal asks the same question of the same objects, and asking it per ordinal - // cost one list each while a departure drained. + // Two uncached lists answer vacancy for every ordinal of this role. taken, takenErr := r.modelDeploymentTakenGroups(ctx, md, role.Name, true) if takenErr != nil { - logger.Error(takenErr, "read the role's groups on the api server", "role", role.Name) return ctrl.Result{}, takenErr } + workloads := new(kueue.WorkloadList) + if err := r.APIReader.List(ctx, workloads, ctrlcli.InNamespace(md.Namespace)); err != nil { + return ctrl.Result{}, err + } + for ordinal := range declared { + group := modelDeploymentReplicaGroupName(md, role.Name, ordinal) + for i := range workloads.Items { + if workloadOwnsGroupName(&workloads.Items[i], group) { + taken.Insert(group) + break + } + } + } + + // The slot's ordinal is filled first: it is the one the role is committed to, and a role + // short for another reason would otherwise spend its creates while this one waits. + // + // IT ASKS THE SAME UNCACHED QUESTION AS EVERY OTHER ORDINAL. The member this slot + // deleted is still standing on the server behind its finalizer, and a replacement is + // rendered for the very name that member holds -- so filling this ordinal ahead of the + // gate would seat two members in one group and collide on the name. + if slot, held := holdingRoles[role.Name]; held && !occupied[slot.Ordinal] { + if !taken.Has(modelDeploymentReplicaGroupName(md, role.Name, slot.Ordinal)) { + createOrdinals[role.Name] = append(createOrdinals[role.Name], slot.Ordinal) + } else { + requeue = true + } + } for ordinal := 0; ordinal < declared && len(createOrdinals[role.Name]) < missing; ordinal++ { - if occupied[ordinal] { + if occupied[ordinal] || slices.Contains(createOrdinals[role.Name], ordinal) { continue } @@ -1075,6 +1240,7 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // would only produce a Pod that sits Pending while its Workload holds quota. if weights != nil && weights.Blocked { clear(createOrdinals) + clear(partialCreates) } // The placement preference is read once for every member this pass creates, and not at all by a @@ -1086,6 +1252,14 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( break } } + if preference == nil { + for _, created := range partialCreates { + if len(created) > 0 { + preference = weights.placementPreference(ctx, r.Client) + break + } + } + } // A FAILED CREATE DOES NOT END THE PASS EITHER, and that is what makes the incomplete group a // REPORTED state rather than a silent one. Returning here would skip the status write below, so @@ -1101,6 +1275,18 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( for i := range md.Spec.Roles { role := &md.Spec.Roles[i] for _, ordinal := range createOrdinals[role.Name] { + if slot, held := slots.slotFor(role.Name); held && slot.Ordinal == ordinal && + slot.Phase == modelDeploymentReplacementCleanup { + slot = modelDeploymentReplacementSlot{ + Role: role.Name, Ordinal: ordinal, Phase: modelDeploymentReplacementWaitingForAdmission, + MemberHashes: replacementRenderHashes(desired[role.Name][ordinal]), + } + if err := r.advanceReplacementSlot(ctx, md, slot); err != nil { + return objectWriteResult(logger, err, "record replacement creation", _requeueAfterConflict) + } + slots = slots.setSlot(slot) + } + // EVERY MEMBER OF THE REPLICA IS ISSUED, AND A FAILURE ON ONE DOES NOT ABANDON THE REST. // The members share one Kueue group whose declared total is the replica's size, and // Kueue composes no Workload at all until that many exist -- so a replica left short is @@ -1142,6 +1328,48 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( // outdated, not eight. rollout.accounted++ } + + // THE MISSING MEMBER OF A PARTIAL REPLICA IS ISSUED THE WAY A WHOLE REPLICA'S MEMBERS ARE, + // into the group its own ordinal names and with the same placement terms, because it is that + // group's member and Kueue composes the group from all of them alike. + for _, rendered := range partialCreates[role.Name] { + pod := rendered.DeepCopy() + if weights != nil { + injectModelArtifactAffinity(pod, weights.Affinity) + } + injectModelPlacementPreference(pod, preference) + if err = r.Client.Create(ctx, pod); err != nil { + logger.Error(err, "create the missing member of a partial replica", + "role", role.Name, + "ordinal", pod.Labels[modelDeploymentReplicaOrdinalLabel], + "member", pod.Labels[modelDeploymentMemberIndexLabel]) + if createErr == nil { + createErr = err + } + + continue + } + logger.Info("created the missing member of a partial replica", "pod", pod.Name) + } + } + + for name, slot := range slots { + if slot.Phase != modelDeploymentReplacementWaitingForAdmission || + (len(createOrdinals[name]) == 0 && len(partialCreates[name]) == 0) { + continue + } + members, err := r.replacementOccupancyOnServer(ctx, md, name, slot.Ordinal) + if err != nil { + return ctrl.Result{}, err + } + uids := sets.New[types.UID]() + for _, member := range members { + uids.Insert(member.UID) + } + slot.CurrentMemberUIDs = sets.List(uids) + if err := r.advanceReplacementSlot(ctx, md, slot); err != nil { + return objectWriteResult(logger, err, "record replacement members", _requeueAfterConflict) + } } // The whole-group health predicate is evaluated ONCE here, before any Service or selector @@ -1161,11 +1389,16 @@ func (r *ModelDeploymentReconciler) convergeModelDeployment( eligibilityDecided := modelDeploymentQualificationsDecided(qualifications) - if err = r.convergeModelDeploymentEndpointEligibility(ctx, md, actual, qualificationByMember); err != nil { + if err = r.convergeModelDeploymentEndpointEligibility( + ctx, md, actual, qualificationByMember, replacedByRole); err != nil { logger.Error(err, "converge endpoint eligibility") return ctrl.Result{}, err } - if err = r.syncModelDeploymentService(ctx, md, eligibilityDecided); err != nil { + servicePods, err := readModelDeploymentPods(ctx, md, r.APIReader) + if err != nil { + return ctrl.Result{}, err + } + if err = r.syncModelDeploymentService(ctx, md, eligibilityDecided, servicePods); err != nil { logger.Error(err, "sync service") return ctrl.Result{}, err } @@ -1288,33 +1521,98 @@ func (r *ModelDeploymentReconciler) recordModelDeploymentRuntimeVersionSkew( func (r *ModelDeploymentReconciler) convergeModelDeploymentEndpointEligibility( ctx context.Context, md *workercore.ModelDeployment, pods []core.Pod, qualificationByMember map[types.UID]modelDeploymentInstanceQualification, + replacedByRole map[string]map[int]bool, ) error { + logger := ctrllog.FromContext(ctx) + roles := make(map[string]*workercore.ModelDeploymentRole, len(md.Spec.Roles)) for i := range md.Spec.Roles { roles[md.Spec.Roles[i].Name] = &md.Spec.Roles[i] } + // THE REPLICA EACH POD BELONGS TO, grouped once. Which members of a replica answer is a question + // about the replica, so it is answered from the replica's own members rather than from each Pod + // in isolation and the role the spec states now. + replicaOf := make(map[types.UID]modelDeploymentReplicaView, len(pods)) + for _, view := range modelDeploymentGroupPods(pods, true) { + for _, member := range view.Members { + replicaOf[member.UID] = view + } + } + + // The selector already published decides whether an undecided deployment keeps its existing + // routing. Reading it rather than passing nothing is what keeps an enabled selector enabled + // across the passes that did not decide it. + liveSelectors, err := r.modelDeploymentLiveRoleSelectors(ctx, md) + if err != nil { + return err + } + for i := range pods { pod := &pods[i] role := roles[modelDeploymentPodRole(pod)] if role == nil { continue } + view, seated := replicaOf[pod.UID] + if !seated { + // A Pod holding no seat of its own answers under readiness alone, which is the rule a + // pre-replica deployment ran on. + view = modelDeploymentReplicaView{Members: []*core.Pod{pod}, Seated: false} + } qualification, evaluated := qualificationByMember[pod.UID] - eligible := modelDeploymentPodEligible(md, role, pod, qualification, evaluated) - carried := pod.Labels[modelDeploymentLabelKeyEndpointEligible] == modelDeploymentEndpointEligibleValue - if eligible == carried { + eligible := modelDeploymentPodEligible(md, role, pod, view, qualification, evaluated) + // The gate is asked of the mode this replica is running, not of the mode the spec states + // now. A desired command line would take the gate away from an old managed replica that has + // activated revocation, which is the replica the gate exists to withdraw. + effective := *role + if modelDeploymentReplicaRunsCommand(view.Members) { + effective.Command = nil + } + gated := modelDeploymentEligibilitySelectorActive(md, &effective, + liveSelectors[role.Name], modelDeploymentEligibilityDecided(md)) + answering := modelDeploymentPodAnswersAPI(pod, view, md.Spec.Engine.Name, + eligible, gated, replacedByRole[role.Name]) + patched := pod.DeepCopy() + changed := modelDeploymentApplyLabel( + patched, modelDeploymentLabelKeyEndpointEligible, eligible) + changed = modelDeploymentApplyLabel( + patched, modelDeploymentLabelKeyAPIAnswering, answering) || changed + if !changed { continue } - patched := pod.DeepCopy() - if eligible { - patched.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue - } else { - delete(patched.Labels, modelDeploymentLabelKeyEndpointEligible) + // The write is bound to the Pod this pass judged. A name is not an identity, so a take-over + // replica's legacy membership must not land on a managed replica that has qualified nothing + // yet. The object is read back, a different incarnation under that name is skipped, and the + // patch carries the version that read returned. + fresh := new(core.Pod) + if err := r.Client.Get(ctx, ctrlcli.ObjectKeyFromObject(pod), fresh); err != nil { + if kerrors.IsNotFound(err) { + // A Pod this pass deleted has no labels left to write, and reporting a whole + // deployment unhealthy over a replica on its way out would be worse than the label. + continue + } + + return err } - if err := r.Client.Patch(ctx, patched, ctrlcli.MergeFrom(pod)); err != nil { + if fresh.UID != pod.UID { + // A different object now holds this name. The answer computed above describes the one + // that is gone, and the next pass judges whatever stands here now. + logger.V(3).Info("skipping a routing write for a member that was replaced", + "role", role.Name, "pod", pod.Name, + "observed", pod.UID, "standing", fresh.UID) + + continue + } + rewritten := fresh.DeepCopy() + modelDeploymentApplyLabel( + rewritten, modelDeploymentLabelKeyEndpointEligible, eligible) + modelDeploymentApplyLabel( + rewritten, modelDeploymentLabelKeyAPIAnswering, answering) + if err := r.Client.Patch(ctx, rewritten, ctrlcli.MergeFromWithOptions(fresh, + ctrlcli.MergeFromWithOptimisticLock{})); err != nil && !kerrors.IsNotFound(err) { return err } } @@ -1322,6 +1620,80 @@ func (r *ModelDeploymentReconciler) convergeModelDeploymentEndpointEligibility( return nil } +// modelDeploymentApplyLabel writes or removes one membership label and reports whether the object +// moved. Absence is the disqualification, so a false never writes "false". +func modelDeploymentApplyLabel(pod *core.Pod, key string, present bool) bool { + carried := pod.Labels[key] == modelDeploymentEndpointEligibleValue + if present == carried { + return false + } + if present { + pod.Labels[key] = modelDeploymentEndpointEligibleValue + } else { + delete(pod.Labels, key) + } + + return true +} + +// modelDeploymentPodAnswersAPI reports whether a member is behind the deployment's HTTP Service +// under the routing rules of the mode and the shape it was built with. +// +// THE MODE AND THE SHAPE ARE READ FROM THE POD AND ITS REPLICA, never from the spec. A role mid-edit +// carries replicas of both modes at once, and each keeps the routing it had: a take-over replica +// answers under the legacy membership rules, which this operator cannot verify and does not pretend +// to, and a managed replica answers only once it qualifies, which is the eligibility key beside it. +// A replica whose followers are still serving its previous four-member shape answers under that shape +// until its own turn comes. +// +// A MEMBER THIS PASS SELECTED FOR REPLACEMENT ANSWERS NOTHING, however its mode reads. That is the +// one rule that removes capacity, and it names one replica rather than the whole role: its outdated +// siblings keep serving until they are selected in their own turn. +func modelDeploymentPodAnswersAPI( + pod *core.Pod, view modelDeploymentReplicaView, engine string, + eligible, gated bool, replacingByOrdinal map[int]bool, +) bool { + if !podIsReady(pod) { + return false + } + ordinal, seated := modelDeploymentPodOrdinal(pod) + if !seated || replacingByOrdinal[ordinal] { + return false + } + + switch modelDeploymentDeployedAnsweringShape(view, engine) { + case modelDeploymentAnsweringAll, modelDeploymentAnsweringSole: + case modelDeploymentAnsweringLeader: + if !modelDeploymentAnsweringMemberLeader(pod) { + return false + } + default: + // Unknown shape retains carried routing, subject to readiness and qualification. + if pod.Labels[modelDeploymentLabelKeyAPIAnswering] != modelDeploymentAPIAnsweringValue { + return false + } + } + + managed, readable := modelDeploymentMemberExecution(pod) + if !readable { + // A member whose mode cannot be read keeps what readiness alone allows, which is the legacy + // behavior rather than a new judgement about it. + return true + } + if !managed { + return true + } + if !gated { + // A ROLE WHOSE ELIGIBILITY SELECTOR WAS NEVER ACTIVATED IS STILL ROUTED THE WAY IT ALWAYS + // WAS, and readiness was always the whole rule there. Reading "managed means qualified" for + // such a role would withdraw every managed endpoint in the deployment on the first pass that + // touched it -- which on a mode edit is the pass that rewrites the selector. + return true + } + + return eligible +} + // modelDeploymentPodEligible is the member rule: a Pod's endpoints may be selected exactly when // the whole group it belongs to qualifies and the member itself is one the shape lets answer. // @@ -1341,6 +1713,7 @@ func (r *ModelDeploymentReconciler) convergeModelDeploymentEndpointEligibility( // the whole replica at size one. Nothing here widens or narrows that. func modelDeploymentPodEligible( md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, pod *core.Pod, + view modelDeploymentReplicaView, qualification modelDeploymentInstanceQualification, evaluated bool, ) bool { if role.ElasticEP != nil { @@ -1388,12 +1761,11 @@ func modelDeploymentPodEligible( return false } - // WHICH MEMBERS ANSWER IS THE SHAPE'S DECISION, and it is the one derivation the retirement drain - // also uses. This reader happens to hold the role, so it reads the facts from there; the drain - // holds the members instead and reads the same facts from the Pods the role rendered. Sharing - // the derivation is what keeps the two from answering "which members serve" differently. - shape := modelDeploymentAnsweringShapeOf( - modelDeploymentRoleExternalDP(md, role), modelDeploymentRoleSize(role)) + // Which members answer is the replica's own shape, read from the members it is running rather + // than from the role the spec states now. An activated External-DP replica whose argv has since + // been edited to an internal one keeps its ranks; reading the role would revoke every follower + // of a group that is still serving. + shape := modelDeploymentDeployedAnsweringShape(view, md.Spec.Engine.Name) if shape == modelDeploymentAnsweringAll || shape == modelDeploymentAnsweringSole { return true } @@ -1432,9 +1804,11 @@ func modelDeploymentRoleExternalDP( // a Service is NOT rebuilt when the group is: the group's shape decides which Pods exist, and the // address they answer on must survive that. func (r *ModelDeploymentReconciler) syncModelDeploymentService( - ctx context.Context, md *workercore.ModelDeployment, eligibilityDecided bool, + ctx context.Context, md *workercore.ModelDeployment, eligibilityDecided bool, pods []core.Pod, ) error { - rendered := renderModelDeploymentServices(md, r.modelDeploymentRoleManufacturers(ctx, md)) + // The standing members come from the API server: a cache that has not caught up would retire + // the peer records of a collective whose members are still running. + rendered := renderModelDeploymentServices(md, r.modelDeploymentRoleManufacturers(ctx, md), pods) if ModelDeploymentElasticRole(md) != nil { for _, service := range rendered { service.Spec.Selector[modelDeploymentReplicaOrdinalLabel] = "0" @@ -1467,7 +1841,20 @@ func (r *ModelDeploymentReconciler) syncModelDeploymentService( if role == nil { continue } - if modelDeploymentEligibilitySelectorActive(md, role, live.Spec.Selector, eligibilityDecided) { + members := make([]*core.Pod, 0, len(pods)) + for i := range pods { + if modelDeploymentPodRole(&pods[i]) == role.Name && modelDeploymentOwns(&pods[i], md) && + pods[i].DeletionTimestamp == nil { + members = append(members, &pods[i]) + } + } + role = modelDeploymentEffectiveRoleForEligibility(role, members) + + selector := maps.Clone(live.Spec.Selector) + if live.Annotations[modelDeploymentEligibilityActiveAnnotation] == modelDeploymentEndpointEligibleValue { + selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + } + if modelDeploymentEligibilitySelectorActive(md, role, selector, eligibilityDecided) { modelDeploymentSelectEligibleEndpoints(rendered[i], md, role) } } @@ -1912,6 +2299,17 @@ func modelDeploymentAdmittedReplicas(workloads []kueue.Workload, pods []*core.Po return admitted } +// modelDeploymentPodValues copies a list of live Pods into the value form the grouping helpers +// take, so a caller holding pointers does not have to hold a second copy of the same list. +func modelDeploymentPodValues(pods []*core.Pod) []core.Pod { + values := make([]core.Pod, 0, len(pods)) + for _, pod := range pods { + values = append(values, *pod) + } + + return values +} + // modelDeploymentReplicaMembers is every live Pod seated on one ordinal, which is what a replica is // once it may have more than one member. The order is the caller's list order and carries no // meaning: a replica leaves as a whole, so nothing downstream picks between its members. @@ -1926,6 +2324,39 @@ func modelDeploymentReplicaMembers(pods []*core.Pod, ordinal int) []*core.Pod { return members } +// seat is one position inside one replica: an ordinal and a member index within it. +type seat struct{ ordinal, member int } + +// modelDeploymentSurplusCount measures the excess this rule exists for: members beyond the seats +// they can occupy. +// +// It counts seats, not desired members: a replica edited from four members to two still fills the +// four seats it was built at, and comparing that against the desired count would trim a healthy +// admitted group until it matched a size the spec only wants for its replacement. +// +// A POD CLAIMING NO ORDINAL IS NOT SURPLUS. It cannot be excess on a seat, and it is judged on the +// ungated rollout instead: it holds a Workload of its own, so it is a replica the rollout turns +// over one per pass rather than a duplicate to trim here. +func modelDeploymentSurplusCount(kept []*core.Pod) int { + seats := make(map[seat]bool, len(kept)) + excess := 0 + for _, pod := range kept { + ordinal, ok := modelDeploymentPodOrdinal(pod) + if !ok { + continue + } + key := seat{ordinal: ordinal, member: modelDeploymentPodMemberIndex(pod)} + if seats[key] { + excess++ + + continue + } + seats[key] = true + } + + return excess +} + // modelDeploymentSurplusReplicas picks which Pods of an over-counted role leave: the Pods that // claim no ordinal first, then the Pods a seat holds beyond its one occupant, at most `surplus` of // them. @@ -2143,15 +2574,21 @@ func (r *ModelDeploymentReconciler) getModelDeploymentRuntimeClassName( // not adopted by the new one. func (r *ModelDeploymentReconciler) listModelDeploymentPods( ctx context.Context, md *workercore.ModelDeployment, +) ([]core.Pod, error) { + return readModelDeploymentPods(ctx, md, r.Client) +} + +// readModelDeploymentPods filters by identity labels and requires the deployment's owner UID. +func readModelDeploymentPods( + ctx context.Context, md *workercore.ModelDeployment, reader ctrlcli.Reader, ) ([]core.Pod, error) { podList := new(core.PodList) - err := r.Client.List(ctx, podList, + err := reader.List(ctx, podList, ctrlcli.InNamespace(md.Namespace), ctrlcli.MatchingLabels{ modelDeploymentLabelKeyName: modelDeploymentLabelValueName, modelDeploymentLabelKeyInstance: md.Name, - }, - ctrlclix.WithoutQuorum) + }) if err != nil { return nil, err } diff --git a/pkg/worker/controllers/worker/model_deployment_answering_shape_test.go b/pkg/worker/controllers/worker/model_deployment_answering_shape_test.go index 69feb3a69..adbad3ca9 100644 --- a/pkg/worker/controllers/worker/model_deployment_answering_shape_test.go +++ b/pkg/worker/controllers/worker/model_deployment_answering_shape_test.go @@ -2,6 +2,7 @@ package worker import ( "context" + "strconv" "strings" "testing" "time" @@ -11,6 +12,7 @@ import ( core "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" ) @@ -33,6 +35,27 @@ func shapeMember(ordinal, member int, uid string, command ...string) core.Pod { return pod } +// shapeSizedMember is a member whose group total is written explicitly, for the cases where the +// replica was NOT rendered at the size the role now declares. planFor leaves a total it has already +// stamped alone, so this is how a case says what the replica was actually built at. +func shapeSizedMember(ordinal, member int, uid string, total int, command ...string) core.Pod { + pod := shapeMember(ordinal, member, uid, command...) + pod.Annotations[kueuepodconst.GroupTotalCountAnnotation] = strconv.Itoa(total) + + return pod +} + +// shapeNoEngineMember is a member with no engine container at all, which is what a Pod this operator +// never rendered looks like. The retirement guards hold on one rather than classify the replica from +// its siblings. +func shapeNoEngineMember(ordinal int, uid string, total int) core.Pod { + pod := healthPod("server", ordinal, 0, healthBool(true), uid) + pod.Spec.Containers = nil + pod.Annotations = map[string]string{kueuepodconst.GroupTotalCountAnnotation: strconv.Itoa(total)} + + return pod +} + // shapeUnlabelledMember is a member carrying no member-index label at all, which is what a replica // rendered before the label existed looks like. It is the case the leader search has to get right // for the wrong reason not to be right by accident: read through a helper that defaults a missing @@ -109,6 +132,26 @@ func planFor( ) *modelDeploymentRetirementPlan { t.Helper() + // THE GROUP TOTAL IS STAMPED FROM THE ROLE'S OWN SIZE, which is what every case below means by + // its members: these replicas were rendered from this deployment at this size. A member carrying + // seat one of a group declaring one is a shape the renderer never emits, and a fixture that + // produced one would be testing a group this operator could not classify rather than the drain. + for i := range pods { + if _, ok := pods[i].Annotations[kueuepodconst.GroupTotalCountAnnotation]; ok { + continue + } + total := 1 + for r := range md.Spec.Roles { + if pods[i].Labels[modelDeploymentLabelKeyComponent] == md.Spec.Roles[r].Name { + total = modelDeploymentRoleSize(&md.Spec.Roles[r]) + } + } + if pods[i].Annotations == nil { + pods[i].Annotations = map[string]string{} + } + pods[i].Annotations[kueuepodconst.GroupTotalCountAnnotation] = strconv.Itoa(total) + } + uids := make([]string, 0, len(pods)) for i := range pods { uids = append(uids, string(pods[i].UID)) @@ -206,8 +249,8 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { name: "a leader-served replica whose role is gone drains through the leader", md: shapeAbsentRole(2), pods: []core.Pod{ - shapeMember(1, 0, "m-0", shapePlainCommand...), - shapeMember(1, 1, "m-1", shapePlainCommand...), + shapeSizedMember(1, 0, "m-0", 2, shapePlainCommand...), + shapeSizedMember(1, 1, "m-1", 2, shapePlainCommand...), }, wantState: workercore.ModelDeploymentRetirementStateDeleting, wantReads: map[string]int{"m-0": 2}, @@ -220,20 +263,23 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { wantReads: map[string]int{"m-0": 2}, }, { - // THE EXTERNAL-DP SHAPE IS REACHABLE FROM THE DRAIN ONLY WITHOUT A ROLE. With a role - // declared, the disaggregation refusal above the selection is the earlier and the - // stronger answer, and it is not this task's to move. - name: "an External-DP replica whose role is gone reads every member", + // THE MEMBERS ANSWER EVEN WITH NO ROLE TO ASK. The role is the one thing this protocol + // tolerates missing, and letting its absence decide the guard would have drained a + // disaggregated replica on the strength of a role that no longer exists. + name: "an External-DP replica whose role is gone is still refused", md: shapeAbsentRole(2), pods: []core.Pod{ shapeMember(1, 0, "m-0", shapeExternalDPCommand...), shapeMember(1, 1, "m-1", shapeExternalDPCommand...), }, - wantState: workercore.ModelDeploymentRetirementStateDeleting, - wantReads: map[string]int{"m-0": 2, "m-1": 2}, + wantState: workercore.ModelDeploymentRetirementStateDraining, + wantReads: map[string]int{}, + wantReason: "a disaggregated replica has no verifiable decoder-side release", }, { - name: "an External-DP replica without a role holds when one of its members is busy", + // The refusal comes before any read, so the busy member never gets its turn. The busy + // member path stays covered above on leader-served replicas, where it is reachable. + name: "an External-DP replica holds before reading a busy member", md: shapeAbsentRole(2), pods: []core.Pod{ shapeMember(1, 0, "m-0", shapeExternalDPCommand...), @@ -241,8 +287,8 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { }, busy: map[types.UID]bool{"m-1": true}, wantState: workercore.ModelDeploymentRetirementStateDraining, - wantReads: map[string]int{"m-0": 2, "m-1": 1}, - wantReason: "still holds", + wantReads: map[string]int{}, + wantReason: "a disaggregated replica has no verifiable decoder-side release", }, { name: "a role declaring External-DP is refused before any member is read", @@ -256,7 +302,12 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { wantReason: "no verifiable decoder-side release", }, { - name: "a leader-served replica with no member carrying the leader index holds", + // A GROUP WITH NO MEMBER ON SEAT ZERO HAS ITS SEATS OUT OF ITS OWN RANGE, and the + // deployed-shape reader says so. It also means the leader search's own "no member carries + // the leader index" branch is unreachable from a readable shape: seats are unique and in + // range, so a group of N always has a seat zero. That branch stays as defense behind the + // shape check rather than as a route a case can reach. + name: "a replica whose seats are all outside its declared total holds", md: shapeDeployment(2), pods: []core.Pod{ shapeMember(1, 1, "m-1", shapePlainCommand...), @@ -264,13 +315,13 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { }, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "no member carrying the leader index", + wantReason: "claims seat 2 of a replica declaring only 2 members", }, { - // THE CASE THE LABEL READ EXISTS FOR. Neither member says which rank it is, so neither - // can be shown to be the leader, and a search that treated "unknown rank" as "rank zero" - // would return this whole replica as its own answer and drain it on a guess. - name: "a leader-served replica whose members carry no member index holds", + // A MULTI-MEMBER GROUP WITH NO SEATS IS NOT A LEGACY REPLICA. Replicas written before the + // member-index label existed were exactly the single-member ones, so this is a shape the + // renderer never emits and the reader refuses it rather than counting its members. + name: "a multi-member replica whose members carry no member index holds", md: shapeDeployment(2), pods: []core.Pod{ shapeUnlabelledMember(1, "m-0", shapePlainCommand...), @@ -278,10 +329,13 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { }, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "no member carrying the leader index", + wantReason: "no member declares which seat it holds", }, { - name: "a leader-served replica with two leaders holds and says how many", + // TWO MEMBERS ON ONE SEAT IS A DUPLICATE, so the reader that validates seats answers before + // the leader search that counts them. Both refuse the group; the seat reader names the + // smaller fact, and two members on seat zero cannot happen in a readable group at all. + name: "a replica with two members on the leader seat holds", md: shapeDeployment(2), pods: []core.Pod{ shapeMember(1, 0, "m-0", shapePlainCommand...), @@ -289,7 +343,39 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { }, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "2 members carrying the leader index", + wantReason: "both claim seat 0", + }, + { + // THE COUNTEREXAMPLE FOR THE DEPLOYED-SHAPE CHECK. Three members declaring a group of + // three, with two of them claiming seat one. There is exactly one member carrying the + // leader index, so the leader search finds a leader, names it and lets the drain run -- + // over a group that cannot be told apart from itself, which is the shape every other + // reader here refuses to classify. + name: "a three-member group whose seats duplicate holds", + md: shapeDeployment(3), + pods: []core.Pod{ + shapeMember(1, 0, "m-0", shapePlainCommand...), + shapeMember(1, 1, "m-1", shapePlainCommand...), + shapeMember(1, 1, "m-1b", shapePlainCommand...), + }, + wantState: workercore.ModelDeploymentRetirementStateDraining, + wantReads: map[string]int{}, + wantReason: "both claim seat 1", + }, + { + // THE SAME HOLE FROM THE OTHER SIDE: every seat is distinct, so nothing duplicates, and + // the member holding seat ninety-nine is outside a group of three. The leader search + // still finds seat zero and never looks at the other two. + name: "a three-member group with a seat outside its declared total holds", + md: shapeDeployment(3), + pods: []core.Pod{ + shapeMember(1, 0, "m-0", shapePlainCommand...), + shapeMember(1, 1, "m-1", shapePlainCommand...), + shapeMember(1, 99, "m-99", shapePlainCommand...), + }, + wantState: workercore.ModelDeploymentRetirementStateDraining, + wantReads: map[string]int{}, + wantReason: "claims seat 99", }, { name: "a replica whose members disagree on the shape holds", @@ -303,18 +389,32 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { wantReason: "the replica's members disagree on whether it is disaggregated", }, { - name: "a role that declares more members than the replica holds", + // THE SIZE FREEZE THIS TASK REMOVES. The role now asks for three members; the replica + // standing was rendered at one and holds its one. Comparing against the role made every + // old replica of an edited role undeclassifiable, and an undeclassifiable replica is one + // this operator never drains -- so a size edit would have pinned the old shape in place + // for as long as the edit existed. + name: "a replica built smaller than the role now declares still drains", + md: shapeDeployment(3), + pods: []core.Pod{shapeSizedMember(1, 0, "m-0", 1, shapePlainCommand...)}, + wantState: workercore.ModelDeploymentRetirementStateDeleting, + wantReads: map[string]int{"m-0": 2}, + }, + { + // The genuine fault, still a refusal: the group says it was built at three and is not. + name: "a replica short of the total its own group declared holds", md: shapeDeployment(3), - pods: []core.Pod{shapeMember(1, 0, "m-0", shapePlainCommand...)}, + pods: []core.Pod{shapeSizedMember(1, 0, "m-0", 3, shapePlainCommand...)}, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "declares 3 members and the replica holds 1", + wantReason: "short of the group it was built at", }, { - // THE DIRECTION THAT REACHES THE CHECK. A role declaring External-DP is refused one - // step earlier, so the only disagreement this can observe is a leader-served role - // above members whose own rendered command says otherwise. - name: "a role and a rendered command that disagree on the shape", + // THE UNSUPPORTED OLD SHAPE, REACHED WITH THE ROLE DISAGREEING. A leader-served role + // above members whose own command is disaggregated is a group the prefiller queues do not + // measure. Reading the role instead of the members drained it on the replacement's + // description. + name: "a leader-served role over disaggregated members is refused on their command", md: shapeDeployment(2), pods: []core.Pod{ shapeMember(1, 0, "m-0", shapeExternalDPCommand...), @@ -322,18 +422,33 @@ func TestDrainReadsTheMembersTheShapeMakesAnswerable(t *testing.T) { }, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "the role and the replica's own rendered command disagree", + wantReason: "a disaggregated replica has no verifiable decoder-side release", + }, + { + // THE MISSING-ENGINE CONTROL WITH A CHANGED DESIRED SPEC. The role now declares the + // disaggregated flag, so every reading about the replacement says the group is + // disaggregated and the guard has a tempting answer available. The members still carry no + // engine container, so this operator has not read what is running and must not delete on + // the replacement's description. + name: "a member with no engine container holds even when the role declares the shape", + md: shapeDeployment(1, "--data-parallel-external-lb"), + pods: []core.Pod{ + healthPod("server", 1, 0, healthBool(true), "m-0"), + }, + wantState: workercore.ModelDeploymentRetirementStateDraining, + wantReads: map[string]int{}, + wantReason: "could not be established", }, { name: "a replica with no role and no readable command cannot be classified", md: shapeAbsentRole(2), pods: []core.Pod{ - healthPod("server", 1, 0, healthBool(true), "m-0"), - shapeMember(1, 1, "m-1", shapePlainCommand...), + shapeNoEngineMember(1, "m-0", 2), + shapeSizedMember(1, 1, "m-1", 2, shapePlainCommand...), }, wantState: workercore.ModelDeploymentRetirementStateDraining, wantReads: map[string]int{}, - wantReason: "no engine container to read", + wantReason: "carries no engine container to read", }, } @@ -547,3 +662,57 @@ func TestDrainReasonsNameTheShape(t *testing.T) { assert.True(t, strings.Contains(reason, "leader-served"), "the reason names the shape, not just the symptom: %q", reason) } + +// A managed replica whose parallelism cannot be read has no established shape. It must hold, not +// be read as leader-served, because that reading withdraws the followers of an External-DP replica +// on a failed read. +func TestDeployedAnsweringShape_UnreadableParallelismHolds(t *testing.T) { + testCases := []struct { + name string + command []string + want modelDeploymentAnsweringShape + }{ + { + name: "a readable External-DP command answers through every member", + command: shapeExternalDPCommand, + want: modelDeploymentAnsweringAll, + }, + { + name: "a readable plain command answers through its leader", + command: shapePlainCommand, + want: modelDeploymentAnsweringLeader, + }, + { + name: "an unreadable parallelism value holds", + command: []string{"vllm", "serve", "model", "--tensor-parallel-size", "many"}, + want: modelDeploymentAnsweringUnknown, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + leader := shapeSizedMember(0, 0, "uid-leader", 2, tc.command...) + follower := shapeSizedMember(0, 1, "uid-follower", 2, tc.command...) + view := modelDeploymentReplicaView{ + Role: "server", Ordinal: 0, Seated: true, + Members: []*core.Pod{&leader, &follower}, + } + + assert.Equal(t, tc.want, modelDeploymentDeployedAnsweringShape(view, workercore.ModelDeploymentEngineVLLM)) + }) + } +} + +// TestDrain_AReplicaWithNoMembersNamesThatAsTheHold keeps the hold reason honest for an already +// unusual state: with no member there is no engine container missing, there is no member at all. +func TestDrain_AReplicaWithNoMembersNamesThatAsTheHold(t *testing.T) { + r := &ModelDeploymentReconciler{} + plan := &modelDeploymentRetirementPlan{ + Reservation: &workercore.ModelDeploymentRetirementStatus{RoleName: "server"}, + } + + reason, drained := r.observeModelDeploymentRetirementDrained(context.Background(), shapeDeployment(1), plan) + + assert.False(t, drained) + assert.Equal(t, "the replica has no members to read", reason) +} diff --git a/pkg/worker/controllers/worker/model_deployment_cache_attached.go b/pkg/worker/controllers/worker/model_deployment_cache_attached.go index 549467184..53bb0225b 100644 --- a/pkg/worker/controllers/worker/model_deployment_cache_attached.go +++ b/pkg/worker/controllers/worker/model_deployment_cache_attached.go @@ -25,6 +25,7 @@ const ( modelDeploymentReasonCacheOperationsFailing = "CacheOperationsFailing" modelDeploymentReasonCacheNotApplicable = "NotApplicable" modelDeploymentReasonUnmanaged = "Unmanaged" + modelDeploymentReasonCacheNotAttributable = "CacheNotAttributable" modelDeploymentReasonNoReplicaReady = "NoReplicaReady" modelDeploymentReasonNoObservationAvailable = "NoObservationAvailable" ) @@ -98,12 +99,13 @@ func (r *ModelDeploymentReconciler) observeModelDeploymentCache( return } - if role := modelDeploymentUnmanagedRole(md); role != "" { - // The operator synthesized no argument and no client environment for that role, so it does - // not claim the observation is about its own doing. Whatever the pool reports. - ModelDeploymentConditionCacheAttached.Unknown(holder, modelDeploymentReasonUnmanaged, - fmt.Sprintf("role %q replaced the whole command line, so the operator configured no "+ - "cache client for it and does not report on one it did not render", role)) + if replica, reason, why := modelDeploymentUnclaimableReplica(md, pods); replica != "" { + // THE OPERATOR CONTRIBUTED NOTHING THIS REPLICA COULD ANSWER FOR, so it does not claim the + // observation is about its own doing. Whatever the pool reports, and whatever the shared + // domain corroborates: a domain-wide fact cannot stand in for a replica whose command line + // could not be attributed. + ModelDeploymentConditionCacheAttached.Unknown(holder, reason, + fmt.Sprintf("%s does not report a cache client it was not shown to have: %s", replica, why)) return } @@ -193,16 +195,39 @@ func (r *ModelDeploymentReconciler) readModelDeploymentCache( return active, failing, unreadable } -// modelDeploymentUnmanagedRole names the first role that took over its command line, or "". -func modelDeploymentUnmanagedRole(md *workercore.ModelDeployment) string { +// modelDeploymentUnclaimableReplica names a replica of a declared role whose running command line +// this operator cannot speak for, with the machine-readable reason and a message. It refuses both a +// replica running a command line the operator did not write and one whose command line cannot be +// attributed, because a shared domain may not corroborate either. +func modelDeploymentUnclaimableReplica( + md *workercore.ModelDeployment, pods []core.Pod, +) (string, string, string) { + declared := make(map[string]bool, len(md.Spec.Roles)) for i := range md.Spec.Roles { - role := &md.Spec.Roles[i] - if len(role.Command) > 0 { - return role.Name + declared[md.Spec.Roles[i].Name] = true + } + + for _, replica := range modelDeploymentGroupPodsByReplica(pods) { + // Auxiliary heads some engines render are built by this operator and carry none of its + // annotations, so only a role the spec declares is asked. + if !declared[replica.Role] { + continue + } + + execution, why := modelDeploymentDeployedReplicaExecution(replica.Members) + switch execution { + case modelDeploymentExecutionTakeover: + return modelDeploymentPodDescription(replica.Members[0]), + modelDeploymentReasonUnmanaged, + "it runs a command line this operator contributed no arguments to, so it was " + + "configured with no cache client and is not reported on one" + case modelDeploymentExecutionUnreadable: + return modelDeploymentPodDescription(replica.Members[0]), + modelDeploymentReasonCacheNotAttributable, why } } - return "" + return "", "", "" } // modelDeploymentReadyReplicas selects the replicas an engine can be asked about: whole, ready, not @@ -233,17 +258,29 @@ func modelDeploymentUnmanagedRole(md *workercore.ModelDeployment) string { // replica of one member is whole as soon as that Pod is there, so exactly the same Pods are asked as // were asked before. func modelDeploymentReadyReplicas(md *workercore.ModelDeployment, pods []core.Pod) []*core.Pod { - sizes := make(map[string]int, len(md.Spec.Roles)) + declared := make(map[string]bool, len(md.Spec.Roles)) for i := range md.Spec.Roles { - sizes[md.Spec.Roles[i].Name] = modelDeploymentRoleSize(&md.Spec.Roles[i]) + declared[md.Spec.Roles[i].Name] = true } // The grouping drops the members already on their way out, so a replica losing one is short of // its declared total here and falls out with the incomplete ones. + // + // COMPLETENESS IS READ FROM THE REPLICA'S OWN GROUP TOTAL. The role's declared size describes + // the replica about to be built, and a role whose size has been edited would otherwise drop + // every healthy old replica of it out of the set of engines this operator is willing to ask + // -- turning a configuration change into a silent cache observation gap. ready := make([]*core.Pod, 0, len(pods)) for _, replica := range modelDeploymentGroupPodsByReplica(pods) { - size, declared := sizes[replica.Role] - if !declared || len(replica.Members) != size { + if !declared[replica.Role] { + continue + } + if !modelDeploymentDeployedReplicaShape(replica).modelDeploymentReplicaIsWhole() { + continue + } + // Only a replica proven managed is scraped: the gate above already refused the rest. + if execution, _ := modelDeploymentDeployedReplicaExecution(replica.Members); execution != + modelDeploymentExecutionManaged { continue } diff --git a/pkg/worker/controllers/worker/model_deployment_cache_attached_test.go b/pkg/worker/controllers/worker/model_deployment_cache_attached_test.go index 5c4a99ebe..b166126cc 100644 --- a/pkg/worker/controllers/worker/model_deployment_cache_attached_test.go +++ b/pkg/worker/controllers/worker/model_deployment_cache_attached_test.go @@ -12,6 +12,7 @@ import ( "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" "k8s.io/utils/ptr" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" ) @@ -106,15 +107,35 @@ func TestModelDeploymentCacheAttached_Table(t *testing.T) { // The operator rendered no cache client for it, so it does not report on one it did not // render — whatever the pool says. The domain here deliberately HOLDS data, which would // otherwise corroborate a True. - name: "a role that took over the command line", + // + // THE REPLICAS CARRY NO DRAIN HOOK, which is what makes them takeover replicas: a role + // supplying its own command line gets no hook, no synthesized arguments and no client + // environment. Declaring Command on the role is no longer what opens this gate -- the + // running Pods are what do -- so a fixture that only edited the spec would now describe + // replicas the operator did build and would be asking the wrong question. + name: "a replica running a command line the operator contributed nothing to", mutateMD: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Command = []string{"/bin/my-server"} }, - pods: func(md *workercore.ModelDeployment) []core.Pod { return readyPods(md, 2) }, + pods: func(md *workercore.ModelDeployment) []core.Pod { return takeoverPods(md, 2) }, domain: readyDomain(func(d *modelDeploymentDomain) { d.Blocks = ptr.To[int64](512) }), wantStatus: "Unknown", wantReason: modelDeploymentReasonUnmanaged, }, + { + // The reverse transition is the case that made this observable: the role has gone back + // to a managed command line, so the DESIRED spec claims the operator built these + // arguments -- while the Pods still standing are running a command line it never wrote. + // Reading the spec here would report a cache for an engine configured with no client. + name: "a role declared managed while its running replicas were built as a takeover", + mutateMD: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"/bin/my-server"} + }, + pods: func(md *workercore.ModelDeployment) []core.Pod { return takeoverPods(md, 2) }, + domain: readyDomain(), + wantStatus: "Unknown", + wantReason: modelDeploymentReasonUnmanaged, + }, { name: "no replica is ready yet", pods: func(md *workercore.ModelDeployment) []core.Pod { return nil }, @@ -291,6 +312,19 @@ func readyPods(md *workercore.ModelDeployment, n int32) []core.Pod { return pods } +// takeoverPods are replicas whose command line the operator did not write: the same Pods +// readyPods builds with the serving-port annotation removed, which is exactly what the render emits +// for a role supplying its own command line -- it stamps nothing, because it cannot know what the +// replacement serves. +func takeoverPods(md *workercore.ModelDeployment, n int32) []core.Pod { + pods := readyPods(md, n) + for i := range pods { + delete(pods[i].Annotations, "prometheus.io/port") + } + + return pods +} + // TestModelDeploymentCacheAttached_IsNeverTrueFromARender is G5 and F8's whole premise, asserted // where it can actually be caught. // @@ -436,3 +470,56 @@ func TestModelDeploymentCacheAttached_AnInstanceShortAMemberIsNotAsked(t *testin "a rank that is not Ready withholds its whole instance, leader included") }) } + +// TestModelDeploymentCacheAttached_UnreadableExecutionIsNeverTrue is the control the review named. +// +// THE MEMBERS OF ONE GROUP DISAGREE on whether the operator wrote their command line, so this +// operator cannot say what either container was configured with. The shared domain deliberately +// HOLDS data here, which is the signal that would otherwise corroborate a True: it is a domain-wide +// fact, and letting it stand in for a replica whose client was never shown would attribute a +// domain observation to containers that may carry no client at all. +func TestModelDeploymentCacheAttached_UnreadableExecutionIsNeverTrue(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 1 + }) + domain := readyDomain(func(d *modelDeploymentDomain) { d.Blocks = ptr.To[int64](512) }) + + // A MANAGED REPLICA DOES report here, which is what shows the case below is refused for the + // disagreement rather than because the domain is unusable: same domain, same replicas. + assert.Equal(t, "True", + ModelDeploymentConditionCacheAttached.GetStatus( + cacheAttachedOf(t, md, readyPods(md, 1), domain, nil)), + "the corroborating signal is usable, so a refusal below is about the replicas") + + // ONE GROUP OF TWO, hand-built because readyReplica names each Pod for itself and two members + // of one replica have to name the SAME group. The disagreement is the defect under test, so the + // fixture states it outright rather than arriving at it by accident. + group := modelDeploymentReplicaGroupName(md, "server", 0) + member := func(name, seat string, managed bool) core.Pod { + pod := *readyReplica(md, 0, true) + pod.Name, pod.UID = name, types.UID("uid-"+name) + pod.Annotations = map[string]string{ + kueuepodconst.GroupTotalCountAnnotation: "2", + } + pod.Labels = map[string]string{ + modelDeploymentLabelKeyComponent: "server", + modelDeploymentReplicaOrdinalLabel: "0", + modelDeploymentMemberIndexLabel: seat, + kueuepodconst.GroupNameLabel: group, + } + if managed { + pod.Annotations["prometheus.io/port"] = "8000" + } + + return pod + } + half := []core.Pod{member("s-half-0", "0", true), member("s-half-1", "1", false)} + + got := cacheAttachedOf(t, md, half, domain, nil) + assert.NotEqual(t, "True", ModelDeploymentConditionCacheAttached.GetStatus(got), + "an unattributable command line may never be corroborated by a shared domain") + assert.Equal(t, "Unknown", ModelDeploymentConditionCacheAttached.GetStatus(got)) + assert.Equal(t, modelDeploymentReasonCacheNotAttributable, + ModelDeploymentConditionCacheAttached.GetReason(got), + "the reason names the refusal rather than the missing observation it would otherwise report") +} diff --git a/pkg/worker/controllers/worker/model_deployment_composition_test.go b/pkg/worker/controllers/worker/model_deployment_composition_test.go new file mode 100644 index 000000000..5f2e3cd1c --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_composition_test.go @@ -0,0 +1,322 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" + ctrlrecord "k8s.io/client-go/tools/record" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" + kueuepod "sigs.k8s.io/kueue/pkg/controller/jobs/pod" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + utilexpectations "sigs.k8s.io/kueue/pkg/util/expectations" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" +) + +// This file builds the Workload a replica's group composes to, by running Kueue's own pod job rather +// than writing one by hand. A composed one names its PodSet after the role hash, counts the members +// sharing that hash, owns each member by a real owner reference, carries Kueue's finalizer, and +// refuses to compose when the group's members do not agree on their total. +// +// The constructor does not refuse a stale adoption by itself: it composes for the members standing +// under a group name, and the lifecycle fixture refuses to write over a Workload that name already +// holds. TestComposition_AStaleWorkloadCannotAbsorbANewMember states the constructor's half and +// TestLifecycle_TwoIncarnationsOfOneReplicaGetDistinctIdentities states the property through real +// creates. + +// newCompositionClient builds a client that answers Kueue's own group lookup, which is a List of +// Pods by the PodGroupNameCacheKey index. Without that index the lookup cannot be performed at all. +// +// A seeded ModelDeployment is given a resource version first: an object read from a server always +// carries one, and the lifecycle records its replacement intent through a real optimistic lock that +// needs one to compare against. +func newCompositionClient(objs ...ctrlcli.Object) ctrlcli.WithWatch { + for _, obj := range objs { + if md, ok := obj.(*workercore.ModelDeployment); ok && md.ResourceVersion == "" { + md.ResourceVersion = "1" + } + } + + return ctrlfake.NewClientBuilder(). + WithScheme(scheme.Scheme). + WithIndex(&core.Pod{}, kueuepod.PodGroupNameCacheKey, func(obj ctrlcli.Object) []string { + pod, ok := obj.(*core.Pod) + if !ok { + return nil + } + group := pod.Labels[kueuepodconst.GroupNameLabel] + if group == "" { + return nil + } + + return []string{group} + }). + WithIndex(&core.Pod{}, "spec.nodeName", func(obj ctrlcli.Object) []string { + pod, ok := obj.(*core.Pod) + if !ok || pod.Spec.NodeName == "" { + return nil + } + + return []string{pod.Spec.NodeName} + }). + WithStatusSubresource(&workercore.ModelDeployment{}, &workercore.KVCachePoolBinding{}, + &workercore.Devices{}, &kueue.Workload{}). + WithObjects(objs...). + Build() +} + +// kueueComposeGroup runs Kueue's composition for a group and returns the Workload it decided on. +func kueueComposeGroup(ctx context.Context, cli ctrlcli.Client, group string) (*kueue.Workload, error) { + podJob := kueuepod.NewPod( + kueuepod.WithExcessPodExpectations(utilexpectations.NewStore("composition-test"))) + key := types.NamespacedName{Namespace: "group/team-a", Name: group} + if _, err := podJob.Load(ctx, cli, &key); err != nil { + return nil, err + } + + return podJob.ConstructComposableWorkload(ctx, cli, ctrlrecord.NewFakeRecorder(64), nil) +} + +// createMember creates one group member with a stamped UID and returns it. +// +// The UID is stamped before anything owns it. The fake client assigns none to a GenerateName +// create, and an empty UID on both sides of an ownership match matches everything, so an unstamped +// Pod would let a Workload claim a group it never composed for. +func createMember(t *testing.T, ctx context.Context, cli ctrlcli.Client, pod *core.Pod, uid string) *core.Pod { + t.Helper() + + live := pod.DeepCopy() + // The member came off another client, so it carries a version and an owner the target server + // has never seen. Both belong to the object being created, not to the Pod spec. + live.UID = types.UID(uid) + live.ResourceVersion = "" + live.OwnerReferences = nil + live.Status = core.PodStatus{} + require.NoError(t, cli.Create(ctx, live)) + + // The name is generated, so the created object rather than the rendered one is what callers + // address by. + stored := new(core.Pod) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{ + Namespace: live.Namespace, Name: live.Name, + }, stored)) + + return stored +} + +// TestComposition_AnIncompleteGroupComposesNoWorkload is the control everything else in this file +// is read against, and it has to fail for the fixture to be worth anything: without it, a client +// that answered every group would satisfy every later assertion, since the admission proof asks +// whether a Workload exists and admits. +func TestComposition_AnIncompleteGroupComposesNoWorkload(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2, "size two renders two members") + + cli := newCompositionClient() + createMember(t, ctx, cli, members[0], "pod-first") + + group := modelDeploymentReplicaGroupName(md, "server", 0) + wl, err := kueueComposeGroup(ctx, cli, group) + require.Error(t, err, "a group one member short composes no Workload") + assert.Nil(t, wl) + + // The second member completes the group, so the refusal was the count and nothing else. + createMember(t, ctx, cli, members[1], "pod-second") + + wl, err = kueueComposeGroup(ctx, cli, group) + require.NoError(t, err, "the completed group composes") + require.NotNil(t, wl) + assert.Len(t, wl.Spec.PodSets, 1, "members sharing one role hash compose one podset") + assert.Equal(t, int32(2), wl.Spec.PodSets[0].Count, + "and the count is read from the members rather than declared anywhere") +} + +// TestComposition_MembersDisagreeingOnTheirTotalComposeNoWorkload covers the other refusal: two +// members claiming different group sizes are not one group, which is the state a size edit would +// create if the running member's annotation were rewritten. +func TestComposition_MembersDisagreeingOnTheirTotalComposeNoWorkload(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + + cli := newCompositionClient() + createMember(t, ctx, cli, members[0], "pod-first") + + disagreeing := members[1].DeepCopy() + disagreeing.Annotations[kueuepodconst.GroupTotalCountAnnotation] = "3" + createMember(t, ctx, cli, disagreeing, "pod-second") + + _, err := kueueComposeGroup(ctx, cli, modelDeploymentReplicaGroupName(md, "server", 0)) + require.Error(t, err, "members disagreeing on their total are not one group") +} + +// TestComposition_AStaleWorkloadCannotAbsorbANewMember proves the property the vacancy proof leans +// on: a Workload composed for one incarnation does not adopt the members of the next, even though +// the new member reuses the old one's name and group. +func TestComposition_AStaleWorkloadCannotAbsorbANewMember(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + group := modelDeploymentReplicaGroupName(md, "server", 0) + + cli := newCompositionClient() + old := createMember(t, ctx, cli, members[0], "pod-old-generation") + + stale, err := kueueComposeGroup(ctx, cli, group) + require.NoError(t, err) + stale.UID = types.UID("wl-old-generation") + require.NoError(t, cli.Create(ctx, stale)) + + require.NoError(t, cli.Delete(ctx, old.DeepCopy())) + fresh := createMember(t, ctx, cli, members[0], "pod-new-generation") + + current := new(kueue.Workload) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: "team-a", Name: group}, current)) + assert.False(t, modelDeploymentWorkloadOwnsAny(current, setsOf(string(fresh.UID))), + "a Workload composed for the old member does not own the new one") +} + +// TestComposition_RepeatIncarnationsGetDistinctUIDs pins the premise the lifecycle rests on: two +// Pods of one ordinal, and two Workloads of one group name, are told apart only by the identity +// the API server assigns each. +func TestComposition_RepeatIncarnationsGetDistinctUIDs(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + group := modelDeploymentReplicaGroupName(md, "server", 0) + + cli := newCompositionClient() + first := createMember(t, ctx, cli, members[0], "pod-generation-one") + firstWL, err := kueueComposeGroup(ctx, cli, group) + require.NoError(t, err) + firstWL.UID = types.UID("wl-generation-one") + require.NoError(t, cli.Create(ctx, firstWL)) + + require.NoError(t, cli.Delete(ctx, first.DeepCopy())) + second := createMember(t, ctx, cli, members[0], "pod-generation-two") + secondWL, err := kueueComposeGroup(ctx, cli, group) + require.NoError(t, err) + secondWL.UID = types.UID("wl-generation-two") + + assert.NotEqual(t, first.UID, second.UID, + "two incarnations of one ordinal are distinct objects, which is what lets a cleanup delete "+ + "one without touching the other") + assert.NotEqual(t, firstWL.UID, secondWL.UID, + "and so are the Workloads the two groups composed") +} + +// TestComposition_ADepartingMemberHoldsAFinalizer shows why a departed member keeps the group +// occupied: the delete is accepted and the Pod stays, because the finalizer is on the Pod. +func TestComposition_ADepartingMemberHoldsAFinalizer(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + + cli := newCompositionClient() + held := createMember(t, ctx, cli, members[0], "pod-held") + held.Finalizers = []string{kueuepodconst.PodFinalizer} + require.NoError(t, cli.Update(ctx, held)) + require.NoError(t, cli.Delete(ctx, held.DeepCopy())) + + after := new(core.Pod) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: "team-a", Name: held.Name}, after)) + assert.NotEmpty(t, after.Finalizers, "the finalizer is what the vacancy proof waits on") + require.NotNil(t, after.DeletionTimestamp) + + // Removing the last finalizer of a Pod already marked for deletion removes it, so there is no + // second delete to issue: the server completes the one that was accepted earlier. + released := after.DeepCopy() + released.Finalizers = nil + require.NoError(t, cli.Update(ctx, released)) + + gone := new(core.Pod) + assert.Error(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: "team-a", Name: held.Name}, gone), + "and once it releases, the member is gone for good") +} + +// TestComposition_AnExcessMemberIsDeletedByKueue states the excess rule from the other side: a group +// carrying more members than it declares reserves for the declared count and has Kueue delete the +// rest, which is what makes a create beside a still-departing member a fault rather than a delay. +func TestComposition_AnExcessMemberIsDeletedByKueue(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + + cli := newCompositionClient() + seated := createMember(t, ctx, cli, members[0], "pod-seat-one") + seated.Finalizers = []string{kueuepodconst.PodFinalizer} + require.NoError(t, cli.Update(ctx, seated)) + + duplicate := seated.DeepCopy() + duplicate.Name = seated.Name + "-duplicate" + createMember(t, ctx, cli, duplicate, "pod-seat-two") + + wl, err := kueueComposeGroup(ctx, cli, modelDeploymentReplicaGroupName(md, "server", 0)) + require.NoError(t, err) + require.NotNil(t, wl) + assert.Equal(t, int32(1), wl.Spec.PodSets[0].Count, + "the composed Workload reserves for the declared count, not the excess") + + after := new(core.PodList) + require.NoError(t, cli.List(ctx, after, ctrlcli.InNamespace("team-a"))) + byUID := map[types.UID]*core.Pod{} + for i := range after.Items { + byUID[after.Items[i].UID] = &after.Items[i] + } + assert.NotNil(t, byUID[types.UID("pod-seat-one")], "one member of the seat stands") + assert.Nil(t, byUID[types.UID("pod-seat-two")], + "and the excess member is gone: Kueue strips its finalizer before deleting it, so a create "+ + "beside a still-present member ends with the member that is newest removed, which is "+ + "why the create gate waits for the vacancy instead of racing it") +} + +// TestComposition_CarriesTheRoleHashAsThePodSetName pins that our PodSet name is the one we wrote, +// because the admission proof rebuilds the composition from that same annotation. +func TestComposition_CarriesTheRoleHashAsThePodSetName(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + + cli := newCompositionClient() + created := createMember(t, ctx, cli, members[0], "pod-first") + + wl, err := kueueComposeGroup(ctx, cli, modelDeploymentReplicaGroupName(md, "server", 0)) + require.NoError(t, err) + require.Len(t, wl.Spec.PodSets, 1) + + assert.NotEmpty(t, created.Annotations[kueuepodconst.RoleHashAnnotation], + "this operator writes the role hash, so the grouping is Kueue's verbatim read rather than "+ + "its fallback digest") + assert.Equal(t, created.Annotations[kueuepodconst.RoleHashAnnotation], string(wl.Spec.PodSets[0].Name), + "and Kueue names the podset after it, which keeps one role's pods in one podset whatever "+ + "their index") +} + +// TestComposition_ComposesTheEntranceQueue pins the field the admission proof compares: Kueue +// copies the member's queue label onto the Workload, so the live queue is readable evidence and a +// Workload naming none has not proved where it submitted. +func TestComposition_ComposesTheEntranceQueue(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + queue := members[0].Labels[kueuectrlconst.QueueLabel] + require.NotEmpty(t, queue, "the render names an InstanceType entrance queue") + + cli := newCompositionClient() + createMember(t, ctx, cli, members[0], "pod-first") + + wl, err := kueueComposeGroup(ctx, cli, modelDeploymentReplicaGroupName(md, "server", 0)) + require.NoError(t, err) + assert.Equal(t, queue, string(wl.Spec.QueueName), + "the composed Workload submits where its members were sent") +} diff --git a/pkg/worker/controllers/worker/model_deployment_correction_test.go b/pkg/worker/controllers/worker/model_deployment_correction_test.go new file mode 100644 index 000000000..e48832fa5 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_correction_test.go @@ -0,0 +1,486 @@ +package worker + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + ctrlrecord "k8s.io/client-go/tools/record" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" +) + +func releaseLifecyclePod(t *testing.T, cli ctrlcli.Client, pod *core.Pod) { + t.Helper() + stored := new(core.Pod) + require.NoError(t, cli.Get(context.Background(), ctrlcli.ObjectKeyFromObject(pod), stored)) + stored.Finalizers = nil + require.NoError(t, cli.Update(context.Background(), stored)) + if stored.DeletionTimestamp == nil { + require.NoError(t, cli.Delete(context.Background(), stored)) + } +} + +func releaseLifecycleWorkload(t *testing.T, cli ctrlcli.Client, wl *kueue.Workload) { + t.Helper() + stored := new(kueue.Workload) + require.NoError(t, cli.Get(context.Background(), ctrlcli.ObjectKeyFromObject(wl), stored)) + stored.Finalizers = nil + require.NoError(t, cli.Update(context.Background(), stored)) + if stored.DeletionTimestamp == nil { + require.NoError(t, cli.Delete(context.Background(), stored)) + } +} + +func TestCorrection_WorkloadVacancyBeforeReusingOrdinal(t *testing.T) { + cases := []struct { + name string + active bool + foreign bool + }{ + {name: "active_terminating", active: true}, + {name: "ordinary_terminating"}, + {name: "active_foreign", active: true, foreign: true}, + {name: "ordinary_foreign", foreign: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 2 }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + before := f.live("server", 1) + require.Len(t, before, 1) + wl := f.workloadNamed(modelDeploymentReplicaGroupName(md, "server", 1)) + require.NotNil(t, wl) + sibling := f.live("server", 0) + require.Len(t, sibling, 1) + siblingWL := f.workloadNamed(modelDeploymentReplicaGroupName(md, "server", 0)) + require.NotNil(t, siblingWL) + if tc.active { + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(ctx, edited)) + f.pass(false) + require.Equal(t, 1, f.slotMust("server").Ordinal) + } else { + require.NoError(t, f.cli.Delete(ctx, before[0])) + require.NoError(t, f.cli.Delete(ctx, wl)) + } + releaseLifecyclePod(t, f.cli, before[0]) + if tc.foreign { + releaseLifecycleWorkload(t, f.cli, wl) + wl = &kueue.Workload{ObjectMeta: meta.ObjectMeta{ + Namespace: md.Namespace, Name: wl.Name, UID: types.UID("foreign-workload"), + Finalizers: []string{"example.test/held"}, + }} + require.NoError(t, f.cli.Create(ctx, wl)) + } + for range 3 { + f.pass(true) + assert.Empty(t, f.live("server", 1), "a retained Workload forbids creation") + held := f.workloadNamed(wl.Name) + require.NotNil(t, held) + assert.Equal(t, wl.UID, held.UID) + if tc.foreign { + assert.Nil(t, held.DeletionTimestamp, "foreign objects must not be deleted") + } + assert.Equal(t, sibling[0].UID, f.live("server", 0)[0].UID) + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) + } + releaseLifecycleWorkload(t, f.cli, wl) + f.pass(true) + after := f.live("server", 1) + require.Len(t, after, 1, "authorized disappearance permits recreation") + assert.NotEmpty(t, after[0].UID) + assert.NotEqual(t, before[0].UID, after[0].UID) + f.admit() + fresh := f.workloadNamed(wl.Name) + require.NotNil(t, fresh) + assert.NotEqual(t, wl.UID, fresh.UID) + assert.Equal(t, sibling[0].UID, f.live("server", 0)[0].UID) + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) + }) + } +} + +func queuedCorrectionFixture(t *testing.T, observed ...bool) (*lifecycleFixture, []*core.Pod, *kueue.Workload) { + t.Helper() + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(ctx, edited)) + f.pass(false) + f.admit() + f.pass(false) + members := f.live("server", 1) + require.Len(t, members, 2) + wl, err := kueueComposeGroup(ctx, f.cli, modelDeploymentReplicaGroupName(md, "server", 1)) + require.NoError(t, err) + require.NotNil(t, wl) + require.NoError(t, f.cli.Create(ctx, wl)) + for _, member := range members { + stored := new(core.Pod) + require.NoError(t, f.cli.Get(ctx, ctrlcli.ObjectKeyFromObject(member), stored)) + stored.Finalizers = []string{kueuepodconst.PodFinalizer} + require.NoError(t, f.cli.Update(ctx, stored)) + } + if len(observed) == 0 || observed[0] { + f.pass(true) + } + require.Equal(t, 1, f.slotMust("server").Ordinal) + require.Empty(t, wl.Status.Conditions, "the composed group remains queued") + return f, f.live("server", 1), f.workloadNamed(wl.Name) +} + +func TestCorrection_CurrentReplacementMemberLoss(t *testing.T) { + cases := []struct { + name string + all bool + disappears bool + unobserved bool + }{ + {name: "terminating_gated_member"}, + {name: "member_gone", disappears: true}, + {name: "all_members_gone", all: true, disappears: true}, + {name: "all_members_gone_before_next_pass", all: true, disappears: true, unobserved: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + f, members, wl := queuedCorrectionFixture(t, !tc.unobserved) + require.NotNil(t, wl) + require.NotEmpty(t, wl.UID) + sibling := f.live("server", 0) + require.Len(t, sibling, 2) + siblingWL := f.workloadNamed(modelDeploymentReplicaGroupName(f.md, "server", 0)) + require.NotNil(t, siblingWL) + lost := members[:1] + if tc.all { + lost = members + } + for _, member := range lost { + require.NotEmpty(t, member.UID) + require.NoError(t, f.cli.Delete(ctx, member)) + if tc.disappears { + releaseLifecyclePod(t, f.cli, member) + } + } + f.pass(true) + held := f.workloadNamed(wl.Name) + require.NotNil(t, held) + assert.Equal(t, wl.UID, held.UID) + require.NotNil(t, held.DeletionTimestamp, "the broken active group's reservation must leave") + slot := f.slotMust("server") + assert.Equal(t, 1, slot.Ordinal, "repair uses the current slot") + assert.Empty(t, f.live("server", 1), "all surviving members leave before any replacement") + for range 3 { + f.pass(true) + assert.Empty(t, f.live("server", 1), "held old identities cannot be topped up") + for i, pod := range f.live("server", 0) { + assert.Equal(t, sibling[i].UID, pod.UID) + } + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) + } + f.admit() + f.pass(true) + after := f.live("server", 1) + require.Len(t, after, 2) + for _, pod := range after { + require.NotEmpty(t, pod.UID) + for _, old := range members { + assert.NotEqual(t, old.UID, pod.UID) + } + assert.Equal(t, "vllm/vllm-openai:v0.26.0", pod.Spec.Containers[0].Image) + } + f.admit() + fresh := f.workloadNamed(wl.Name) + require.NotNil(t, fresh) + assert.NotEqual(t, wl.UID, fresh.UID) + for _, pod := range after { + assert.True(t, modelDeploymentWorkloadOwnsAny(fresh, sets.New(pod.UID))) + } + for i, pod := range f.live("server", 0) { + assert.Equal(t, sibling[i].UID, pod.UID) + } + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) + }) + } +} + +func TestCorrection_MixedServicesSelectRoutingUnion(t *testing.T) { + cases := []struct { + name string + active bool + qualified bool + external bool + }{ + {name: "revoked_managed", active: true}, + {name: "unknown_retains_active", active: true, qualified: true}, + {name: "never_activated_legacy"}, + {name: "external_ranks", active: true, qualified: true, external: true}, + } + for _, tc := range cases { + for _, desiredTakeover := range []bool{false, true} { + name := tc.name + map[bool]string{false: "_to_managed", true: "_to_takeover"}[desiredTakeover] + t.Run(name, func(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + if tc.external { + md.Spec.Roles[0].ExtraArgs = []string{"--data-parallel-size", "2", "--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"} + } + }) + managed := replicaMembersAt(t, md, "server", 0) + takeoverMD := md.DeepCopy() + takeoverMD.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + takeoverMD.Spec.Roles[0].ExtraArgs = nil + takeover := replicaMembersAt(t, takeoverMD, "server", 1) + require.Len(t, managed, 2) + require.Len(t, takeover, 2) + if tc.active { + ModelDeploymentConditionEndpointEligibility.True(md, "EndpointsQualified", "activate engine qualification") + } + pods := make([]core.Pod, 0, 4) + for i, pod := range append(managed, takeover...) { + pod.UID = types.UID(string(rune('a' + i))) + pod.Name = "member-" + string(pod.UID) + pod.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} + if i < 2 && tc.qualified { + pod.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + } + pods = append(pods, *pod) + } + cli := newCompositionClient(md, &pods[0], &pods[1], &pods[2], &pods[3]) + if tc.active { + for _, svc := range renderModelDeploymentServices(md, nil, pods[:2]) { + require.NoError(t, cli.Create(context.Background(), svc)) + } + } + if desiredTakeover { + md.Spec.Roles[0].Command = takeoverMD.Spec.Roles[0].Command + md.Spec.Roles[0].ExtraArgs = nil + } + if tc.name == "unknown_retains_active" { + ModelDeploymentConditionEndpointEligibility.Unknown(md, "ObservationHeld", "retain the activated selector") + } + r := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(context.Background(), md, pods, nil, nil)) + actual := replicaPods(t, cli) + require.Len(t, actual, 4) + require.NoError(t, r.syncModelDeploymentService(context.Background(), md, false, actual)) + published := new(core.ServiceList) + require.NoError(t, cli.List(context.Background(), published, ctrlcli.InNamespace(md.Namespace))) + svcs := published.Items + fronts, peers := 0, 0 + for _, svc := range svcs { + selector := labels.SelectorFromSet(svc.Spec.Selector) + if svc.Spec.ClusterIP == core.ClusterIPNone { + peers++ + assert.NotContains(t, svc.Spec.Selector, modelDeploymentLabelKeyEndpointEligible) + assert.NotContains(t, svc.Spec.Selector, modelDeploymentLabelKeyAPIAnswering) + continue + } + fronts++ + for _, pod := range actual { + ordinal, ok := modelDeploymentPodOrdinal(&pod) + require.True(t, ok) + leader := modelDeploymentAnsweringMemberLeader(&pod) + want := leader + if ordinal == 0 { + want = (leader || tc.external) && (!tc.active || tc.qualified) + } + assert.Equal(t, want, selector.Matches(labels.Set(pod.Labels)), "%s full selector for %s", svc.Name, pod.Name) + } + } + require.Equal(t, 2, fronts, "deployment and role HTTP selectors are both checked") + require.Equal(t, 2, peers) + // A later Unknown observation still honors the original activation decision. + if tc.active { + ModelDeploymentConditionEndpointEligibility.Unknown(md, "ObservationHeld", "retain activation") + for i := range actual { + delete(actual[i].Labels, modelDeploymentLabelKeyEndpointEligible) + require.NoError(t, cli.Update(context.Background(), &actual[i])) + } + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(context.Background(), md, actual, nil, nil)) + actual = replicaPods(t, cli) + require.NoError(t, r.syncModelDeploymentService(context.Background(), md, false, actual)) + require.NoError(t, cli.List(context.Background(), published, ctrlcli.InNamespace(md.Namespace))) + for _, svc := range published.Items { + if svc.Spec.ClusterIP == core.ClusterIPNone { + continue + } + for _, pod := range actual { + ordinal, _ := modelDeploymentPodOrdinal(&pod) + want := ordinal == 1 && modelDeploymentAnsweringMemberLeader(&pod) + assert.Equal(t, want, labels.SelectorFromSet(svc.Spec.Selector).Matches(labels.Set(pod.Labels)), + "%s retains managed revocation after Unknown for %s", svc.Name, pod.Name) + } + } + } + }) + } + } +} + +func TestCorrection_CurrentCleanupLostDeleteResponse(t *testing.T) { + ctx := context.Background() + f, members, wl := queuedCorrectionFixture(t) + sibling := f.live("server", 0) + require.Len(t, sibling, 2) + siblingWL := f.workloadNamed(modelDeploymentReplicaGroupName(f.md, "server", 0)) + require.NotNil(t, siblingWL) + require.NoError(t, f.cli.Delete(ctx, members[0])) + lost := false + writer := ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + Delete: func(ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.DeleteOption, + ) error { + if err := next.Delete(ctx, obj, opts...); err != nil { + return err + } + if pod, ok := obj.(*core.Pod); ok && pod.UID == members[1].UID && !lost { + lost = true + return errors.New("connection lost after successful delete") + } + return nil + }, + }) + r := &ModelDeploymentReconciler{Client: writer, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + _, err := reconcileModelDeploymentWith(t, r) + require.ErrorContains(t, err, "connection lost") + require.True(t, lost) + slot := f.slotMust("server") + assert.Equal(t, wl.UID, slot.WorkloadUID, "identity was persisted before cleanup") + assert.Equal(t, 1, slot.Ordinal) + // All Pod finalizers complete before the retry. Only the persisted Workload identity remains. + f.admit() + require.Empty(t, f.live("server", 1)) + require.NotNil(t, f.workloadNamed(wl.Name)) + f.pass(true) + held := f.workloadNamed(wl.Name) + require.NotNil(t, held) + require.NotNil(t, held.DeletionTimestamp, "retry deletes the captured reservation without live Pods") + assert.Empty(t, f.live("server", 1)) + f.admit() + f.pass(true) + after := f.live("server", 1) + require.Len(t, after, 2) + for _, pod := range after { + for _, old := range members { + assert.NotEqual(t, old.UID, pod.UID) + } + } + for i, pod := range f.live("server", 0) { + assert.Equal(t, sibling[i].UID, pod.UID) + } + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) +} + +func TestCorrection_UnknownShapeRetainsOnlyCarriedRouting(t *testing.T) { + cases := []struct { + name string + absent bool + notReady bool + peerFailed bool + replacing bool + want bool + }{ + {name: "retain_selected", want: true}, + {name: "never_grant_absent", absent: true}, + {name: "not_ready_withdraws", notReady: true}, + {name: "positive_peer_failure_withdraws", peerFailed: true}, + {name: "selected_replacement_withdraws", replacing: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].Replicas = 2 + }) + managed := replicaMembersAt(t, md, "server", 0) + takeoverMD := md.DeepCopy() + takeoverMD.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + takeover := replicaMembersAt(t, takeoverMD, "server", 1) + require.Len(t, managed, 2) + require.Len(t, takeover, 2) + pods := make([]core.Pod, 0, 4) + objects := make([]ctrlcli.Object, 0, 1+len(managed)+len(takeover)) + objects = append(objects, md) + for i, member := range append(managed, takeover...) { + member.UID = types.UID("unknown-member-" + string(rune('a'+i))) + member.Name = string(member.UID) + member.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} + pods = append(pods, *member) + } + leader := &pods[0] + require.True(t, modelDeploymentAnsweringMemberLeader(leader)) + leader.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + if !tc.absent { + leader.Labels[modelDeploymentLabelKeyAPIAnswering] = modelDeploymentAPIAnsweringValue + } + pods[1].Annotations[kueuepodconst.GroupTotalCountAnnotation] = "3" + if tc.notReady { + leader.Status.Conditions[0].Status = core.ConditionFalse + } + if tc.peerFailed { + pods[1].Status.Conditions[0].Status = core.ConditionFalse + } + for i := range pods { + objects = append(objects, &pods[i]) + } + cli := newCompositionClient(objects...) + ModelDeploymentConditionEndpointEligibility.True(md, "EndpointsQualified", "already activated") + for _, svc := range renderModelDeploymentServices(md, nil, pods) { + require.NoError(t, cli.Create(ctx, svc)) + } + ModelDeploymentConditionEndpointEligibility.Unknown(md, "ObservationHeld", "unreadable shape") + var replacing map[string]map[int]bool + if tc.replacing { + replacing = map[string]map[int]bool{"server": {0: true}} + } + qualifications := qualifyModelDeploymentInstances(ctx, md, pods, + modelDeploymentPendingReplacement{ordinals: replacing}, defaultGroupForwardFetch) + byMember := modelDeploymentQualificationsByMember(qualifications) + q, found := byMember[leader.UID] + require.True(t, found) + require.Equal(t, modelDeploymentLegUnknown, q.Leg(modelDeploymentLegMemberSetComplete).Verdict) + require.Equal(t, tc.notReady || tc.peerFailed || tc.replacing, q.HasFailure(), + "unknown shape alone supplies no positive health failure") + r := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(ctx, md, pods, byMember, replacing)) + actual := replicaPods(t, cli) + require.Len(t, actual, 4) + require.NoError(t, r.syncModelDeploymentService(ctx, md, false, actual)) + stored := new(core.Pod) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKeyFromObject(leader), stored)) + assert.Equal(t, tc.want, stored.Labels[modelDeploymentLabelKeyAPIAnswering] == modelDeploymentAPIAnsweringValue) + for _, name := range []string{md.Name, md.Name + "-server"} { + svc := new(core.Service) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, svc)) + assert.Equal(t, tc.want, labels.SelectorFromSet(svc.Spec.Selector).Matches(labels.Set(stored.Labels)), + "%s uses carried membership through Unknown", name) + } + }) + } +} diff --git a/pkg/worker/controllers/worker/model_deployment_elastic_realization.go b/pkg/worker/controllers/worker/model_deployment_elastic_realization.go index 7c03a94bb..fe2f5ddda 100644 --- a/pkg/worker/controllers/worker/model_deployment_elastic_realization.go +++ b/pkg/worker/controllers/worker/model_deployment_elastic_realization.go @@ -351,11 +351,11 @@ func (r *ModelDeploymentReconciler) convergeModelDeploymentElastic(ctx context.C masters := modelDeploymentElasticMasters(md, actual) qualifications := qualifyModelDeploymentInstances(ctx, md, masters, modelDeploymentPendingReplacement{}, fetch) byMember := modelDeploymentQualificationsByMember(qualifications) - if err := r.convergeModelDeploymentEndpointEligibility(ctx, md, actual, byMember); err != nil { + if err := r.convergeModelDeploymentEndpointEligibility(ctx, md, actual, byMember, nil); err != nil { return true, err } decided := modelDeploymentQualificationsDecided(qualifications) - if err := r.syncModelDeploymentService(ctx, md, decided); err != nil { + if err := r.syncModelDeploymentService(ctx, md, decided, actual); err != nil { return true, err } return true, r.syncModelDeploymentRouter(ctx, md, decided) diff --git a/pkg/worker/controllers/worker/model_deployment_elastic_reconcile.go b/pkg/worker/controllers/worker/model_deployment_elastic_reconcile.go index fe6e8d5b7..d1ef6176d 100644 --- a/pkg/worker/controllers/worker/model_deployment_elastic_reconcile.go +++ b/pkg/worker/controllers/worker/model_deployment_elastic_reconcile.go @@ -347,7 +347,11 @@ func (r *ModelDeploymentReconciler) elasticSetWithdrawn(ctx context.Context, md } } if withdraw { - if err := r.syncModelDeploymentService(ctx, md, true); err != nil { + pods, err := r.elasticLiveMembers(ctx, md) + if err != nil { + return err + } + if err = r.syncModelDeploymentService(ctx, md, true, pods); err != nil { return err } return r.syncModelDeploymentRouter(ctx, md, true) diff --git a/pkg/worker/controllers/worker/model_deployment_external_routing_test.go b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go index 45ca77879..3df278216 100644 --- a/pkg/worker/controllers/worker/model_deployment_external_routing_test.go +++ b/pkg/worker/controllers/worker/model_deployment_external_routing_test.go @@ -47,7 +47,7 @@ func TestExternalDPRoutingSelectsQualifiedMembers(t *testing.T) { for i := range md.Spec.Roles { role := &md.Spec.Roles[i] published := labels.SelectorFromSet(objects.Contract.Roles[i].Selector) - service := labels.SelectorFromSet(renderModelDeploymentRoleService(md, role, nil).Spec.Selector) + service := labels.SelectorFromSet(renderModelDeploymentRoleService(md, role, nil, nil, false).Spec.Selector) for member := range 2 { pod, renderErr := renderModelDeploymentPodTemplate(context.Background(), ModelDeploymentRenderInput{ Deployment: md, Role: role, InstanceType: newRenderInstanceType(), @@ -135,10 +135,10 @@ func TestPDRoutingKeepsReplicaLeaders(t *testing.T) { for i := range md.Spec.Roles { role := &md.Spec.Roles[i] discovery := routingDiscoverySelector(t, objects.Deployment.Spec.Template.Spec.Containers[0].Args, objects.ConfigMap, "--"+role.Name+"-selector") - service := renderModelDeploymentRoleService(md, role, nil) + service := renderModelDeploymentRoleService(md, role, nil, nil, false) selectors := []labels.Selector{labels.SelectorFromSet(service.Spec.Selector), labels.SelectorFromSet(objects.Contract.Roles[i].Selector)} if i == 0 { - selectors = append(selectors, labels.SelectorFromSet(renderModelDeploymentService(md).Spec.Selector)) + selectors = append(selectors, labels.SelectorFromSet(renderModelDeploymentService(md, nil).Spec.Selector)) } for ordinal := range 2 { for member := range sizes[i] { @@ -173,7 +173,7 @@ func TestExternalDPServiceEligibilityRetention(t *testing.T) { reconciler := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(16)} for _, status := range []meta.ConditionStatus{meta.ConditionTrue, meta.ConditionFalse, meta.ConditionUnknown} { md.Status.Conditions = []gpustackcore.Condition{{Type: string(ModelDeploymentConditionEndpointEligibility), Status: status}} - require.NoError(t, reconciler.syncModelDeploymentService(context.Background(), md, false)) + require.NoError(t, reconciler.syncModelDeploymentService(context.Background(), md, false, nil)) for _, name := range []string{md.Name, md.Name + "-" + role.Name} { service := new(core.Service) require.NoError(t, cli.Get(context.Background(), ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, service)) diff --git a/pkg/worker/controllers/worker/model_deployment_final_review_test.go b/pkg/worker/controllers/worker/model_deployment_final_review_test.go new file mode 100644 index 000000000..ee0001be2 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_final_review_test.go @@ -0,0 +1,217 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + kerrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/types" + ctrlrecord "k8s.io/client-go/tools/record" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/utils/strconvx" +) + +func TestFinalReview_OrdinalLessPodDoesNotInterruptAnActiveReplacement(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 2 }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + sibling := f.live("server", 0) + require.Len(t, sibling, 1) + siblingWL := f.workloadNamed(modelDeploymentReplicaGroupName(md, "server", 0)) + require.NotNil(t, siblingWL) + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(ctx, edited)) + f.pass(false) + slot := f.slotMust("server") + require.Equal(t, modelDeploymentReplacementCleanup, slot.Phase) + require.NotEmpty(t, slot.MemberUIDs) + + legacy := sibling[0].DeepCopy() + legacy.Name = "qwen-server-legacy" + legacy.UID = "" + legacy.ResourceVersion = "" + delete(legacy.Labels, modelDeploymentReplicaOrdinalLabel) + legacy.Labels[kueuepodconst.GroupNameLabel] = legacy.Name + require.NoError(t, f.cli.Create(ctx, legacy)) + require.NotEmpty(t, legacy.UID) + view := ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + List: func(ctx context.Context, next ctrlcli.WithWatch, list ctrlcli.ObjectList, opts ...ctrlcli.ListOption) error { + if err := next.List(ctx, list, opts...); err != nil { + return err + } + if pods, ok := list.(*core.PodList); ok { + for i := range pods.Items { + if replacementCapturesMember(slot, pods.Items[i].UID) { + pods.Items[i].DeletionTimestamp = nil + } + } + } + return nil + }, + }) + r := &ModelDeploymentReconciler{Client: view, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + require.NotPanics(t, func() { + _, err := reconcileModelDeploymentWith(t, r) + require.NoError(t, err) + }) + removed := new(core.Pod) + require.NoError(t, f.cli.Get(ctx, ctrlcli.ObjectKeyFromObject(legacy), removed)) + assert.NotNil(t, removed.DeletionTimestamp, "the ordinal-less Pod is repaired independently") + assert.Equal(t, slot.Ordinal, f.slotMust("server").Ordinal) + assert.Equal(t, sibling[0].UID, f.live("server", 0)[0].UID) + assert.Equal(t, siblingWL.UID, f.workloadNamed(siblingWL.Name).UID) + for range 8 { + f.admit() + f.pass(true) + } + assert.True(t, allAtImage(f, "vllm/vllm-openai:v0.26.0", 2)) +} + +func TestFinalReview_ServicesKeepDeployedRanksBeforeNewConfigurationAppears(t *testing.T) { + cases := []struct { + name string + takeover bool + }{ + {name: "managed_to_takeover", takeover: true}, + {name: "external_to_internal"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].ExtraArgs = []string{"--data-parallel-size", "2", "--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)"} + }) + ModelDeploymentConditionEndpointEligibility.True(md, "EndpointsQualified", "activate qualification") + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2) + pods := make([]core.Pod, 0, len(members)) + objects := make([]ctrlcli.Object, 0, 1+len(members)) + objects = append(objects, md) + for _, member := range members { + member.UID = types.UID("rank-" + member.Name) + member.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} + member.Labels[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + pods = append(pods, *member) + objects = append(objects, member) + } + cli := newCompositionClient(objects...) + r := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(ctx, md, pods, nil, nil)) + pods = replicaPods(t, cli) + require.Len(t, pods, 2) + require.NoError(t, r.syncModelDeploymentService(ctx, md, false, pods)) + for _, pod := range pods { + for _, name := range []string{md.Name, md.Name + "-server"} { + svc := new(core.Service) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, svc)) + require.True(t, labels.SelectorFromSet(svc.Spec.Selector).Matches(labels.Set(pod.Labels)), "the starting selector serves each rank") + } + } + + md.Spec.Roles[0].ExtraArgs = []string{"--tensor-parallel-size", "2"} + if tc.takeover { + md.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + md.Spec.Roles[0].ExtraArgs = nil + } + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(ctx, md, pods, nil, nil)) + pods = replicaPods(t, cli) + require.Len(t, pods, 2, "no new mode or shape has appeared") + require.NoError(t, r.syncModelDeploymentService(ctx, md, false, pods)) + for _, name := range []string{md.Name, md.Name + "-server"} { + svc := new(core.Service) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, svc)) + for _, pod := range pods { + assert.True(t, labels.SelectorFromSet(svc.Spec.Selector).Matches(labels.Set(pod.Labels)), "%s still serves deployed rank %s", name, pod.Name) + } + } + }) + } +} + +func TestFinalReview_CacheLagDoesNotRemovePeerDNS(t *testing.T) { + cases := []struct { + name string + replicas int32 + ordinal int + desiredSize int32 + terminating bool + }{ + {name: "standing", replicas: 1, desiredSize: 1}, + {name: "terminating", replicas: 1, desiredSize: 1, terminating: true}, + {name: "standing_surplus_and_size_change", replicas: 2, ordinal: 1, desiredSize: 1}, + {name: "terminating_surplus_and_size_change", replicas: 2, ordinal: 1, desiredSize: 1, terminating: true}, + {name: "standing_surplus", replicas: 2, ordinal: 1, desiredSize: 2}, + {name: "terminating_surplus", replicas: 2, ordinal: 1, desiredSize: 2, terminating: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = tc.replicas + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + members := f.live("server", tc.ordinal) + require.Len(t, members, 2) + published := new(core.ServiceList) + require.NoError(t, f.cli.List(ctx, published, ctrlcli.InNamespace(md.Namespace))) + peers := make([]core.Service, 0, 1) + for _, svc := range published.Items { + if svc.Spec.ClusterIP == core.ClusterIPNone && + svc.Spec.Selector[modelDeploymentReplicaOrdinalLabel] == strconvx.Itoa(tc.ordinal) { + peers = append(peers, svc) + } + } + require.Len(t, peers, 1) + peerName := peers[0].Name + peer := peers[0].DeepCopy() + if tc.terminating { + for _, member := range members { + require.NoError(t, f.cli.Delete(ctx, member)) + } + } + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Replicas = 1 + edited.Spec.Roles[0].ReplicaSize = tc.desiredSize + require.NoError(t, f.cli.Update(ctx, edited)) + view := ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + List: func(ctx context.Context, next ctrlcli.WithWatch, list ctrlcli.ObjectList, opts ...ctrlcli.ListOption) error { + if err := next.List(ctx, list, opts...); err != nil { + return err + } + if pods, ok := list.(*core.PodList); ok { + pods.Items = nil + } + return nil + }, + }) + r := &ModelDeploymentReconciler{Client: view, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + _, err := reconcileModelDeploymentWith(t, r) + require.NoError(t, err) + require.Len(t, replicaPods(t, f.cli), int(tc.replicas)*2, "the server still holds the old members") + require.Empty(t, replicaPods(t, view), "the cache has not observed them") + err = f.cli.Get(ctx, ctrlcli.ObjectKey{Namespace: md.Namespace, Name: peerName}, peer) + assert.NoError(t, err, "old members retain peer DNS through cache lag") + for _, member := range members { + releaseLifecyclePod(t, f.cli, member) + } + f.pass(true) + err = f.cli.Get(ctx, ctrlcli.ObjectKey{Namespace: md.Namespace, Name: peerName}, peer) + assert.True(t, kerrors.IsNotFound(err), "peer DNS leaves once every old member is gone") + }) + } +} diff --git a/pkg/worker/controllers/worker/model_deployment_group_shape.go b/pkg/worker/controllers/worker/model_deployment_group_shape.go new file mode 100644 index 000000000..beb57cb4a --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_group_shape.go @@ -0,0 +1,304 @@ +package worker + +import ( + core "k8s.io/api/core/v1" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + "gpustack.ai/gpustack/pkg/utils/strconvx" +) + +// A replica's deployed shape is read from the group total and member indexes its members carry, +// never from the role's declared size. + +// modelDeploymentReplicaShapeState is how far a replica's members establish its shape. +type modelDeploymentReplicaShapeState int + +const ( + // modelDeploymentReplicaShapeUnreadable is a shape the members do not establish: totals that + // disagree, seats that are duplicated, out of range or unparsable, or a multi-member group with + // no seat evidence. + modelDeploymentReplicaShapeUnreadable modelDeploymentReplicaShapeState = iota + // modelDeploymentReplicaShapeIncomplete is a readable shape whose members are short of the + // total they declare. + modelDeploymentReplicaShapeIncomplete + // modelDeploymentReplicaShapeComplete is a readable shape the replica fills. + modelDeploymentReplicaShapeComplete +) + +// modelDeploymentReplicaShape is one replica's deployed size and how well its members establish it. +type modelDeploymentReplicaShape struct { + // Total is the group size the members declare. It is zero only when the shape is unreadable. + Total int + // State is what the members establish. + State modelDeploymentReplicaShapeState + // Reason names the contradiction or the shortfall. It is the text a hold carries, so it says + // which fact was read and which was expected. + Reason string +} + +// modelDeploymentReplicaIsWhole reports whether the members fill the total they declare. +func (s modelDeploymentReplicaShape) modelDeploymentReplicaIsWhole() bool { + return s.State == modelDeploymentReplicaShapeComplete +} + +// modelDeploymentReplicaLegVerdict is the leg verdict this shape implies. +// +// A replica short of its total lost members and is a fault. Members that contradict each other +// describe nothing, which is an unknown rather than a fault: this operator has established nothing +// about them, including that they are broken. +func (s modelDeploymentReplicaShape) modelDeploymentReplicaLegVerdict() modelDeploymentLegVerdict { + switch s.State { + case modelDeploymentReplicaShapeComplete: + return modelDeploymentLegVerified + case modelDeploymentReplicaShapeIncomplete: + return modelDeploymentLegFailed + default: + return modelDeploymentLegUnknown + } +} + +// modelDeploymentDeployedReplicaShape reads one replica's shape from its own members. It is PURE. +func modelDeploymentDeployedReplicaShape(view modelDeploymentReplicaView) modelDeploymentReplicaShape { + if len(view.Members) == 0 { + return modelDeploymentReplicaShape{ + State: modelDeploymentReplicaShapeUnreadable, + Reason: "the replica holds no members, so it has no deployed shape to read", + } + } + + total, reason := modelDeploymentReplicaDeclaredTotal(view.Members) + if reason != "" { + return modelDeploymentReplicaShape{State: modelDeploymentReplicaShapeUnreadable, Reason: reason} + } + + if reason = modelDeploymentReplicaSeatReason(view.Members, total); reason != "" { + return modelDeploymentReplicaShape{State: modelDeploymentReplicaShapeUnreadable, Reason: reason} + } + + if held := len(view.Members); held != total { + return modelDeploymentReplicaShape{ + Total: total, + State: modelDeploymentReplicaShapeIncomplete, + Reason: "the replica holds " + strconvx.Itoa(held) + " of " + strconvx.Itoa(total) + + " members it declared, so it is short of the group it was built at", + } + } + + return modelDeploymentReplicaShape{Total: total, State: modelDeploymentReplicaShapeComplete} +} + +// modelDeploymentReplicaHoldsReason says how a replica's membership stands against its deployed +// total. An unreadable shape speaks for itself, because a count would be a claim the members do +// not support. +func modelDeploymentReplicaHoldsReason( + view modelDeploymentReplicaView, shape modelDeploymentReplicaShape, +) string { + if shape.State == modelDeploymentReplicaShapeUnreadable { + return shape.Reason + } + + return "the replica holds " + strconvx.Itoa(len(view.Members)) + " of " + + strconvx.Itoa(shape.Total) + " declared members" +} + +// modelDeploymentReplicaDeclaredTotal is the group size every member agrees on, or why they do not. +// +// A member declaring no total is a replica of one: that is what a Pod from before the annotation +// existed is, and modelDeploymentPodDescription has always read it so. A member declaring one is +// held to it, so a group mixing the two contradicts itself. +func modelDeploymentReplicaDeclaredTotal(members []*core.Pod) (int, string) { + total := 0 + for _, member := range members { + declared, ok := modelDeploymentMemberDeclaredTotal(member) + if !ok { + return 0, "member " + member.Name + " declares a group total that is not a replica size, " + + "so the replica's shape could not be established" + } + + if total == 0 { + total = declared + + continue + } + if total != declared { + return 0, "the replica's members declare group totals of " + strconvx.Itoa(total) + + " and " + strconvx.Itoa(declared) + ", so its shape could not be established" + } + } + + return total, "" +} + +// modelDeploymentMemberDeclaredTotal reads one member's own group total, and reports false for a +// value no replica could have been built at. +// +// ABSENCE IS NOT A FAILURE and reads as a replica of one; a value that is PRESENT and is not a size +// is. One is a Pod from before the annotation existed, the other is a claim this operator cannot +// act on. +func modelDeploymentMemberDeclaredTotal(member *core.Pod) (int, bool) { + raw, present := member.Annotations[kueuepodconst.GroupTotalCountAnnotation] + if !present { + return 1, true + } + + total, err := strconvx.Atoi[int](raw) + if err != nil || total < 1 { + return 0, false + } + + return total, true +} + +// modelDeploymentReplicaSeatReason reports why a replica's members cannot be told apart as seats, +// or "" when they can. +// +// No seat evidence is accepted except for a provable legacy single-member replica: the render has +// always written a seat onto every member, so a larger group with none was never produced here. A +// seat label that is present and unparsable is a defect, not that legacy case. +func modelDeploymentReplicaSeatReason(members []*core.Pod, total int) string { + seats := make(map[int]string, len(members)) + claimed, invalid := 0, 0 + for _, member := range members { + index, state := modelDeploymentMemberSeat(member) + switch state { + case modelDeploymentSeatInvalid: + invalid++ + + continue + case modelDeploymentSeatAbsent: + + continue + } + + claimed++ + if holder, taken := seats[index]; taken { + return "members " + holder + " and " + member.Name + " both claim seat " + + strconvx.Itoa(index) + " of a replica declaring " + strconvx.Itoa(total) + + " members, so the replica cannot be told apart from itself" + } + seats[index] = member.Name + + if index >= total { + return "member " + member.Name + " claims seat " + strconvx.Itoa(index) + + " of a replica declaring only " + strconvx.Itoa(total) + + " members, so the replica cannot be told apart from itself" + } + } + + if invalid > 0 { + return strconvx.Itoa(invalid) + " of " + strconvx.Itoa(len(members)) + + " members declare a member index that is not one, so the replica cannot be told apart " + + "from itself" + } + if claimed == 0 { + if total == 1 && len(members) == 1 { + return "" + } + + return "no member declares which seat it holds, and a replica of " + strconvx.Itoa(total) + + " members is a shape whose seats were always written, so the replica cannot be told " + + "apart from itself" + } + if claimed != len(members) { + return strconvx.Itoa(claimed) + " of " + strconvx.Itoa(len(members)) + + " members declare which seat they hold and the rest do not, so the replica cannot be " + + "told apart from itself" + } + + return "" +} + +// modelDeploymentSeatState is what a member's seat label carries: a usable seat, none at all, or a +// claim that is not a seat. +// +// IT IS NOT A BOOL because absent and invalid are different facts that must not collapse. Absent +// marks a Pod from before the label existed; invalid marks a claim this operator cannot use. +type modelDeploymentSeatState int + +const ( + modelDeploymentSeatValid modelDeploymentSeatState = iota + modelDeploymentSeatAbsent + modelDeploymentSeatInvalid +) + +// modelDeploymentMemberSeat reads which seat a member claims. +// +// IT IS NOT modelDeploymentPodMemberIndex, which answers "which member does the engine call me" and +// defaults an absent label to the leader index. That default is right there and wrong here: it would +// make every member of a legacy replica a duplicate of the leader. +func modelDeploymentMemberSeat(member *core.Pod) (int, modelDeploymentSeatState) { + raw, present := member.Labels[modelDeploymentMemberIndexLabel] + if !present { + return 0, modelDeploymentSeatAbsent + } + + index, err := strconvx.Atoi[int](raw) + if err != nil || index < 0 { + return 0, modelDeploymentSeatInvalid + } + + return index, modelDeploymentSeatValid +} + +// modelDeploymentReplicaExecution is who built the command line a replica is running. +// +// It is a question about the running Pod. The role as declared now describes the replica about to +// be built and says nothing about the one still serving. +type modelDeploymentReplicaExecution int + +const ( + // modelDeploymentExecutionUnreadable is a replica whose command line could not be attributed, + // because a member carries no engine container to read. + modelDeploymentExecutionUnreadable modelDeploymentReplicaExecution = iota + // modelDeploymentExecutionManaged is a replica running a command line this operator built. + modelDeploymentExecutionManaged + // modelDeploymentExecutionTakeover is a replica running a command line the role supplied in + // full, which this operator contributed no arguments, no environment and no connector to. + modelDeploymentExecutionTakeover +) + +// modelDeploymentDeployedReplicaExecution reports who built the command line this replica is +// running, read from its own members. It is PURE. +// +// The evidence is the serving-port annotation, the same one the group-forward probe relies on: the +// render stamps it only for roles running the operator's own command line. It is not the drain +// hook, which the Elastic realization clears from every Ray worker it renders. +func modelDeploymentDeployedReplicaExecution(members []*core.Pod) (modelDeploymentReplicaExecution, string) { + if len(members) == 0 { + return modelDeploymentExecutionUnreadable, + "the replica holds no members, so it has no running command line to read" + } + + // EVERY MEMBER IS ASKED. One with no engine container has not been read, and skipping it would + // attribute a replica from the members that happened to be legible. + built := false + for _, member := range members { + managed, ok := modelDeploymentMemberExecution(member) + if !ok { + return modelDeploymentExecutionUnreadable, "member " + member.Name + " carries no engine " + + "container to read, so the replica's execution could not be established" + } + if member != members[0] && managed != built { + return modelDeploymentExecutionUnreadable, "the replica's members disagree on who built " + + "their command line, so its execution could not be established" + } + + built = managed + } + + if built { + return modelDeploymentExecutionManaged, "" + } + + return modelDeploymentExecutionTakeover, "" +} + +// modelDeploymentMemberExecution reads whether this operator built one member's command line, and +// reports false for a member carrying no engine container to read. +func modelDeploymentMemberExecution(member *core.Pod) (managed, ok bool) { + if _, found := modelDeploymentDrainContainerOf(member, modelDeploymentMainContainerName); !found { + return false, false + } + + return member.Annotations["prometheus.io/port"] != "", true +} diff --git a/pkg/worker/controllers/worker/model_deployment_group_shape_test.go b/pkg/worker/controllers/worker/model_deployment_group_shape_test.go new file mode 100644 index 000000000..6a669efb4 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_group_shape_test.go @@ -0,0 +1,559 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + "k8s.io/apimachinery/pkg/types" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" + "gpustack.ai/gpustack/pkg/utils/strconvx" +) + +// deployedMember builds one member of a replica, carrying only what the deployed reader reads: the +// Kueue group total it was rendered with, the seat it claims, and the serving-port annotation the +// render stamps on a command line it built. +// +// THE PORT ANNOTATION IS THE EXECUTION MARKER BECAUSE THAT IS WHAT THE RENDER DOES: it stamps +// prometheus.io/port only when the operator wrote the command. A fixture that invented a separate +// label would test the fixture rather than the contract. +func deployedMember(name string, total, seat int, managed, labelled bool) *core.Pod { + pod := &core.Pod{} + pod.Name = name + pod.Annotations = map[string]string{} + if total > 0 { + pod.Annotations[kueuepodconst.GroupTotalCountAnnotation] = strconvx.Itoa(total) + } + if managed { + pod.Annotations["prometheus.io/port"] = "8000" + } + if labelled { + pod.Labels = map[string]string{modelDeploymentMemberIndexLabel: strconvx.Itoa(seat)} + } + pod.Spec.Containers = []core.Container{{Name: modelDeploymentMainContainerName}} + + return pod +} + +// seatedMember is a member whose seat label is present but carries something that is not a seat, +// which is the corrupt case the reader must not confuse with a Pod written before labels existed. +func seatedMember(name string, total int, seat string) *core.Pod { + pod := deployedMember(name, total, 0, true, true) + pod.Labels[modelDeploymentMemberIndexLabel] = seat + + return pod +} + +// deployedReplica is the same fixture for a replica, so the cases below read as the group they +// describe rather than as a slice assembled three times. +func deployedReplica(role string, ordinal int, members ...*core.Pod) modelDeploymentReplicaView { + view := modelDeploymentReplicaView{Role: role, Ordinal: ordinal, Seated: true} + view.Members = append(view.Members, members...) + + return view +} + +func TestModelDeploymentDeployedReplicaShapeReadsThePodOwnTotal(t *testing.T) { + cases := []struct { + name string + view modelDeploymentReplicaView + total int + state modelDeploymentReplicaShapeState + whole bool + reasonHas string + }{ + { + name: "a single-member replica is whole at the total it was rendered with", + view: deployedReplica("server", 0, deployedMember("a", 1, 0, true, true)), + total: 1, state: modelDeploymentReplicaShapeComplete, whole: true, + }, + { + name: "a multi-member replica holding every seat is whole", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), deployedMember("b", 2, 1, true, true)), + total: 2, state: modelDeploymentReplicaShapeComplete, whole: true, + }, + { + // The fact this whole task exists: a replica the spec no longer counts is still whole. + name: "a replica short of the DESIRED count is whole at the count it was built at", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), deployedMember("b", 2, 1, true, true)), + total: 2, state: modelDeploymentReplicaShapeComplete, whole: true, + }, + { + name: "a replica that lost a member is incomplete at the total it declared", + view: deployedReplica("server", 0, deployedMember("a", 2, 0, true, true)), + total: 2, state: modelDeploymentReplicaShapeIncomplete, + reasonHas: "1 of 2 members", + }, + { + // Every deployment predating multi-member replicas is full of these, so a Pod with no + // total at all is a replica of one rather than an unreadable object. + name: "a member declaring no total is a replica of one", + view: deployedReplica("server", 0, deployedMember("a", 0, 0, true, true)), + total: 1, state: modelDeploymentReplicaShapeComplete, whole: true, + }, + { + // The legacy fallback is for ONE member and nothing else. Replicas written before the + // seat label existed were exactly the single-member ones, so a group of several with no + // seats is a shape this operator never produced and cannot be read as whole. + name: "a multi-member replica whose members claim no seats cannot be classified", + view: deployedReplica("server", 0, deployedMember("a", 2, 0, true, false), deployedMember("b", 2, 1, true, false)), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "no member declares which seat it holds", + }, + { + // THE CONTROL THE REVIEW NAMED. Two members, one seat label each, both unparsable. The + // seats are present but useless, which is not the same as absent, and a reader that + // collapsed the two counted two members against a total of two and called it whole. + name: "members whose seat labels are present but unparsable cannot be classified", + view: deployedReplica("server", 0, + seatedMember("a", 2, "bad"), seatedMember("b", 2, "bad")), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "declare a member index that is not one", + }, + { + name: "a member whose seat is a negative number cannot be classified", + view: deployedReplica("server", 0, deployedMember("a", 2, 0, true, true), seatedMember("b", 2, "-1")), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "declare a member index that is not one", + }, + { + // One member absent both its seat and its total is the Pod from before either existed, + // and the single shape the reader may still call whole with no seat evidence at all. + name: "a lone member carrying neither total nor seat is the legacy single-member replica", + view: deployedReplica("server", 0, deployedMember("a", 0, 0, true, false)), + total: 1, state: modelDeploymentReplicaShapeComplete, whole: true, + }, + { + name: "members declaring different totals cannot be classified", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), deployedMember("b", 1, 0, true, true)), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "group totals of 2 and 1", + }, + { + name: "two members claiming one seat cannot be classified", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), deployedMember("b", 2, 0, true, true)), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "both claim seat 0", + }, + { + name: "a member holding a seat outside the total cannot be classified", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), + deployedMember("b", 2, 1, true, true), deployedMember("c", 2, 2, true, true)), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "claims seat 2", + }, + { + name: "a replica where only some members claim a seat cannot be classified", + view: deployedReplica("server", 0, + deployedMember("a", 2, 0, true, true), deployedMember("b", 2, 1, true, false)), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "declare which seat they hold", + }, + { + name: "a replica holding no members has no shape to read", + view: deployedReplica("server", 0), + state: modelDeploymentReplicaShapeUnreadable, + reasonHas: "no members", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + shape := modelDeploymentDeployedReplicaShape(tc.view) + + assert.Equal(t, tc.state, shape.State) + assert.Equal(t, tc.total, shape.Total) + assert.Equal(t, tc.whole, shape.modelDeploymentReplicaIsWhole()) + if tc.reasonHas != "" { + assert.Contains(t, shape.Reason, tc.reasonHas) + } else if shape.State == modelDeploymentReplicaShapeComplete { + assert.Empty(t, shape.Reason, "a whole replica has nothing to explain") + } + }) + } +} + +func TestModelDeploymentDeployedReplicaShapeRejectsATotalNoReplicaWasBuiltAt(t *testing.T) { + for _, total := range []string{"0", "-1", "many", "1.5"} { + t.Run(total, func(t *testing.T) { + member := deployedMember("a", 1, 0, true, true) + member.Annotations[kueuepodconst.GroupTotalCountAnnotation] = total + + shape := modelDeploymentDeployedReplicaShape(deployedReplica("server", 0, member)) + + assert.Equal(t, modelDeploymentReplicaShapeUnreadable, shape.State) + assert.Zero(t, shape.Total, "an unreadable shape carries no total to read figures against") + assert.Contains(t, shape.Reason, "not a replica size") + }) + } +} + +func TestModelDeploymentDeployedReplicaExecutionReadsTheRenderedCommandLine(t *testing.T) { + cases := []struct { + name string + members []*core.Pod + execution modelDeploymentReplicaExecution + reasonHas string + }{ + { + name: "a replica whose hook the operator wrote is managed", + members: []*core.Pod{deployedMember("a", 1, 0, true, true)}, + execution: modelDeploymentExecutionManaged, + }, + { + name: "a replica running a command line the operator contributed nothing to is a takeover", + members: []*core.Pod{deployedMember("a", 1, 0, false, true)}, + execution: modelDeploymentExecutionTakeover, + }, + { + name: "every member of a takeover replica agrees on it", + members: []*core.Pod{ + deployedMember("a", 2, 0, false, true), deployedMember("b", 2, 1, false, true), + }, + execution: modelDeploymentExecutionTakeover, + }, + { + name: "members disagreeing on who wrote their command line cannot be classified", + members: []*core.Pod{ + deployedMember("a", 2, 0, true, true), deployedMember("b", 2, 1, false, true), + }, + execution: modelDeploymentExecutionUnreadable, + reasonHas: "disagree on who built", + }, + { + // A member this operator has not read is not a member to skip: the replica is not + // classifiable from the members that happened to be legible. + name: "a member with no engine container makes the replica unreadable", + members: []*core.Pod{ + deployedMember("a", 2, 0, true, true), deployedNoEngineMember("b"), + }, + execution: modelDeploymentExecutionUnreadable, + reasonHas: "member b carries no engine container", + }, + { + name: "a replica with nothing to read is unreadable", + members: []*core.Pod{deployedNoEngineMember("a")}, + execution: modelDeploymentExecutionUnreadable, + reasonHas: "carries no engine container", + }, + { + name: "a replica holding no members has no command line to read", + execution: modelDeploymentExecutionUnreadable, + reasonHas: "no members", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + execution, reason := modelDeploymentDeployedReplicaExecution(tc.members) + + require.Equal(t, tc.execution, execution) + if tc.reasonHas != "" { + assert.Contains(t, reason, tc.reasonHas) + } else { + assert.Empty(t, reason) + } + }) + } +} + +func deployedNoEngineMember(name string) *core.Pod { + pod := &core.Pod{} + pod.Name = name + pod.Spec.Containers = []core.Container{{Name: "sidecar"}} + + return pod +} + +// oldReplicaPods are the members a role rendered BEFORE an edit: the given group size, the given +// command ownership, and nothing else about them tied to what the role says now. +// +// IT IS WRITTEN THE WAY THE RENDER WRITES A MEMBER -- the group total, the seat, and the +// serving-port annotation the render stamps only on a command line it built. A takeover replica is +// the same member with the port annotation absent, because that is precisely what the render emits +// for a role that supplies its own command. +func oldReplicaPods(md *workercore.ModelDeployment, ordinal, size int, managed bool) []core.Pod { + pods := make([]core.Pod, 0, size) + for member := range size { + pod := healthPod("server", ordinal, member, healthBool(true), + "uid-old-"+strconvx.Itoa(ordinal)+"-"+strconvx.Itoa(member)) + pod.Annotations = map[string]string{ + kueuepodconst.GroupTotalCountAnnotation: strconvx.Itoa(size), + } + pod.Spec.Containers = []core.Container{{ + Name: modelDeploymentMainContainerName, Image: "vllm/vllm-openai:v0.29.0", + }} + if managed { + pod.Annotations["prometheus.io/port"] = "8000" + } + pods = append(pods, pod) + } + + return pods +} + +// TestDeployedFactsSurviveADesiredSizeOrCommandEdit is the acceptance this whole task is written +// against, asserted through every observation T1 owns rather than through the reader alone. +// +// THE DEPLOYMENT HAS BEEN EDITED TWICE: the role now asks for four members where these replicas +// were built at two, and it now declares no command of its own where these replicas were built as a +// takeover. Nothing about the running Pods changed. What each reader must still say about them: +// +// - the replica is WHOLE, because it holds every member of the group it declared; +// - it QUALIFIES, so its endpoints stay selected and the deployment keeps serving; +// - status counts it READY, because it is serving exactly what it always served; +// - the cache condition can still ASK it, and does not claim a client for one whose command line +// the operator did not write; +// - its departure is CLASSIFIABLE, so an edit can actually retire it rather than freezing it. +// +// EVERY FIGURE HERE WOULD READ THE OTHER WAY IF ANY READER STILL TOOK THE ROLE'S SIZE, and the +// point of asserting them together is that they are one fact read in five places rather than five +// answers that can drift apart. +func TestDeployedFactsSurviveADesiredSizeOrCommandEdit(t *testing.T) { + md := healthDeployment(4) + pods := oldReplicaPods(md, 0, 2, true) + + view := modelDeploymentGroupPodsByReplica(pods) + require.Len(t, view, 1) + + t.Run("the replica is whole at the size it was built at", func(t *testing.T) { + shape := modelDeploymentDeployedReplicaShape(view[0]) + + assert.Equal(t, 2, shape.Total, "the group declares two, whatever the role now asks for") + assert.True(t, shape.modelDeploymentReplicaIsWhole()) + }) + + t.Run("it qualifies, so its endpoints are not withdrawn", func(t *testing.T) { + qualifications := qualifyModelDeploymentInstances( + context.Background(), md, pods, modelDeploymentPendingReplacement{}, boundProbeFetch, + ) + require.Len(t, qualifications, 1) + assert.True(t, qualifications[0].Eligible(), + "a healthy replica the role has been edited away from is still serving") + assert.False(t, qualifications[0].HasFailure()) + }) + + t.Run("status counts it ready", func(t *testing.T) { + qualifications := qualifyModelDeploymentInstances( + context.Background(), md, pods, modelDeploymentPendingReplacement{}, boundProbeFetch, + ) + statuses := modelDeploymentRoleStatuses( + md, pods, nil, modelDeploymentRoleQualified(qualifications), + ) + require.Len(t, statuses, 1) + assert.Equal(t, int32(1), statuses[0].Ready, + "the replica is serving; a desired-size edit does not unserve it") + assert.False(t, statuses[0].Unmanaged) + }) + + t.Run("the cache can ask it", func(t *testing.T) { + assert.Len(t, modelDeploymentReadyReplicas(md, pods), 1) + + unclaimable, _, _ := modelDeploymentUnclaimableReplica(md, pods) + assert.Empty(t, unclaimable, "these replicas were built from a managed command line") + }) + + t.Run("its departure is classifiable", func(t *testing.T) { + shape, reason := modelDeploymentRetirementAnsweringShape(md, view[0].Members) + + assert.Empty(t, reason, "an edit must not freeze the replica it means to retire") + assert.Equal(t, modelDeploymentAnsweringLeader, shape) + }) +} + +// TestDeployedFactsFollowTheReplicasNotTheRole is the reverse transition for command ownership. +// +// THE ROLE IS DECLARED MANAGED AND THE RUNNING PODS ARE NOT, which is the state an operator creates +// by clearing Command while the old takeover replicas are still up. Reading the role would report a +// cache client for containers that never received one. +func TestDeployedFactsFollowTheReplicasNotTheRole(t *testing.T) { + md := healthDeployment(2) + pods := oldReplicaPods(md, 0, 2, false) + + execution, reason := modelDeploymentDeployedReplicaExecution( + modelDeploymentGroupPodsByReplica(pods)[0].Members) + assert.Empty(t, reason) + assert.Equal(t, modelDeploymentExecutionTakeover, execution) + + replica, reason, why := modelDeploymentUnclaimableReplica(md, pods) + assert.NotEmpty(t, replica, "a replica running a command line we did not build is named") + assert.Equal(t, modelDeploymentReasonUnmanaged, reason) + assert.NotEmpty(t, why) + + qualifications := qualifyModelDeploymentInstances( + context.Background(), md, pods, modelDeploymentPendingReplacement{}, boundProbeFetch, + ) + statuses := modelDeploymentRoleStatuses( + md, pods, nil, modelDeploymentRoleQualified(qualifications), + ) + require.Len(t, statuses, 1) + assert.True(t, statuses[0].Unmanaged, + "the field reports what is running, which is not what the role now declares") +} + +// renderedReplica renders one replica member, so the execution facts are read off what this +// operator actually emits. +func renderedReplica( + t *testing.T, mutate ...func(*workercore.ModelDeployment), +) []*core.Pod { + t.Helper() + + md := newRenderDeployment(mutate...) + pod, err := renderModelDeploymentPod(context.Background(), ModelDeploymentRenderInput{ + Deployment: md, Role: &md.Spec.Roles[0], InstanceType: newRenderInstanceType(), + }) + require.NoError(t, err) + + return []*core.Pod{pod} +} + +// TestRenderedPodsCarryTheirExecution reads execution facts off rendered Pods. +func TestRenderedPodsCarryTheirExecution(t *testing.T) { + t.Run("a managed render reads as managed", func(t *testing.T) { + execution, why := modelDeploymentDeployedReplicaExecution(renderedReplica(t)) + + assert.Empty(t, why) + assert.Equal(t, modelDeploymentExecutionManaged, execution) + }) + + t.Run("a role supplying its own command line reads as a takeover", func(t *testing.T) { + execution, why := modelDeploymentDeployedReplicaExecution(renderedReplica(t, + func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"custom-server"} + })) + + assert.Empty(t, why) + assert.Equal(t, modelDeploymentExecutionTakeover, execution) + }) + + t.Run("a managed render stripped of its drain hook is still managed", func(t *testing.T) { + // The Elastic realization clears every Ray worker's Lifecycle, so a reader asking for the + // hook calls a managed worker a takeover replica. + members := renderedReplica(t) + for _, member := range members { + for i := range member.Spec.Containers { + member.Spec.Containers[i].Lifecycle = nil + } + } + + execution, why := modelDeploymentDeployedReplicaExecution(members) + + assert.Empty(t, why) + assert.Equal(t, modelDeploymentExecutionManaged, execution, + "the Elastic worker transformation must not change what the reader concludes") + }) +} + +// TestRenderedPodsCarryTheirGroupTotal is the other half of the same argument: the size the replica +// is measured against is one the renderer writes, so a reader that wants it does not need the spec. +func TestRenderedPodsCarryTheirGroupTotal(t *testing.T) { + members := renderedReplica(t) + shape := modelDeploymentDeployedReplicaShape(deployedReplica("server", 0, + members[0].DeepCopy())) + + assert.Equal(t, modelDeploymentReplicaShapeComplete, shape.State) + assert.Equal(t, 1, shape.Total, "a single-member role renders a group of one") +} + +// TestActualElasticRenderCarriesExecutionFacts drives renderModelDeploymentPods on an Elastic +// deployment and asserts the execution facts of the Pods it emits. +func TestActualElasticRenderCarriesExecutionFacts(t *testing.T) { + md := newRenderDeployment() + md.Spec.KVCache = nil + md.Spec.Engine.Version = "0.29.0" + md.Spec.Roles[0].Replicas = 1 + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + md.Spec.Roles[0].ElasticEP = &workercore.ModelDeploymentRoleElasticEP{Width: 2} + md.ResourceVersion = "1" + require.NotNil(t, ModelDeploymentElasticRole(md), "the dispatcher takes the elastic path") + + cli := ctrlfake.NewClientBuilder().WithScheme(scheme.Scheme). + WithObjects(md, newRenderInstanceType()). + WithStatusSubresource(&workercore.ModelDeployment{}). + WithIndex(&core.Pod{}, "spec.nodeName", func(obj ctrlcli.Object) []string { + return []string{obj.(*core.Pod).Spec.NodeName} + }). + WithIndex(&core.Pod{}, modelDeploymentDrainIndexPodUID, func(obj ctrlcli.Object) []string { + return []string{string(obj.GetUID())} + }).Build() + + r := &ModelDeploymentReconciler{Client: cli, APIReader: cli} + rendered, err := r.renderModelDeploymentPods(context.Background(), md, nil, nil, nil) + require.NoError(t, err) + + var members []core.Pod + for _, ordinals := range rendered { + for _, pods := range ordinals { + for _, pod := range pods { + members = append(members, *pod) + } + } + } + require.NotEmpty(t, members, "the elastic render produced no members") + + // Master and Ray worker both carry the port, and the render strips the worker's drain hook. + head := modelDeploymentElasticHeadName(md) + workers := 0 + for _, member := range members { + if member.Labels[modelDeploymentLabelKeyComponent] == head { + continue + } + workers++ + view := deployedReplica("server", 0, member.DeepCopy()) + execution, why := modelDeploymentDeployedReplicaExecution(view.Members) + + assert.Empty(t, why, "member %s", member.Name) + assert.Equal(t, modelDeploymentExecutionManaged, execution, + "member %s is an operator-built replica and must not read as a takeover", member.Name) + assert.Equal(t, modelDeploymentReplicaShapeComplete, + modelDeploymentDeployedReplicaShape(view).State, "member %s", member.Name) + } + assert.GreaterOrEqual(t, workers, 2, "the elastic render produced a master and at least one Ray worker") + + // The head is part of the render and is not a declared role. + var sawHead bool + for _, member := range members { + if member.Labels[modelDeploymentLabelKeyComponent] == head { + sawHead = true + } + } + require.True(t, sawHead, "the elastic render emitted its auxiliary head") +} + +// TestAuxiliaryHeadIsNeverClaimed covers the auxiliary head, which is built by this operator and +// carries none of its annotations. It is not a declared role, so neither the cache condition nor +// status may name it. +func TestAuxiliaryHeadIsNeverClaimed(t *testing.T) { + md := newRenderDeployment() + md.Spec.KVCache = nil + md.Spec.Roles[0].Replicas = 1 + head := renderModelDeploymentElasticHeadPod(md, &md.Spec.Roles[0], core.Container{ + Name: modelDeploymentMainContainerName, Image: "vllm/vllm-openai:v0.29.0", + }) + head.Name, head.UID = modelDeploymentElasticHeadName(md), types.UID("elastic-head-uid") + head.Annotations = map[string]string{} + + replica, reason, why := modelDeploymentUnclaimableReplica(md, []core.Pod{*head}) + assert.Empty(t, replica, + "a role the spec does not declare is not asked about; reason %q why %q", reason, why) + + statuses := modelDeploymentRoleStatuses(md, []core.Pod{*head}, nil, nil) + require.Len(t, statuses, 1) + assert.False(t, statuses[0].Unmanaged, + "the declared role reports on its own members, not on an auxiliary head") +} diff --git a/pkg/worker/controllers/worker/model_deployment_health.go b/pkg/worker/controllers/worker/model_deployment_health.go index a4d16c952..0f250e1da 100644 --- a/pkg/worker/controllers/worker/model_deployment_health.go +++ b/pkg/worker/controllers/worker/model_deployment_health.go @@ -297,16 +297,19 @@ func qualifyModelDeploymentInstance( pending modelDeploymentPendingReplacement, fetch modelDeploymentGroupForwardFetch, ) modelDeploymentInstanceQualification { - size := modelDeploymentRoleSize(role) - // THE CHEAP LEGS ARE DECIDED BEFORE THE OBSERVATION IS ASKED FOR. A replica that is short a // member or holds an unready one is already condemned by facts the informer cache carries, so // it never pays for a live observation, and more importantly its definite fault is never // softened into a hold by an observation that happens to be missing. + // + // Completeness is measured against the group total the members declare, never the role's + // declared size, so a replica the role has been edited away from stays whole. + // A shape the members cannot establish holds; a replica short of its total still fails. + shape := modelDeploymentDeployedReplicaShape(view) membersComplete := modelDeploymentQualificationLeg{ Name: modelDeploymentLegMemberSetComplete, - Verdict: legVerdict(modelDeploymentReplicaIsComplete(view, size)), - Reason: "the replica holds " + strconvx.Itoa(len(view.Members)) + " of " + strconvx.Itoa(size) + " declared members", + Verdict: shape.modelDeploymentReplicaLegVerdict(), + Reason: modelDeploymentReplicaHoldsReason(view, shape), } unready, unobserved := 0, 0 @@ -397,15 +400,6 @@ func qualifyModelDeploymentInstance( return qualification } -// legVerdict maps a boolean onto a verdict, for the legs whose only question is yes or no. -func legVerdict(ok bool) modelDeploymentLegVerdict { - if ok { - return modelDeploymentLegVerified - } - - return modelDeploymentLegFailed -} - // modelDeploymentMemberReadiness reads one member's own readiness as three answers rather than // the two podIsReady returns. // diff --git a/pkg/worker/controllers/worker/model_deployment_health_test.go b/pkg/worker/controllers/worker/model_deployment_health_test.go index 5b4e0a637..48cd50767 100644 --- a/pkg/worker/controllers/worker/model_deployment_health_test.go +++ b/pkg/worker/controllers/worker/model_deployment_health_test.go @@ -16,6 +16,7 @@ import ( core "k8s.io/api/core/v1" meta "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" "gpustack.ai/gpustack/pkg/utils/strconvx" @@ -62,6 +63,38 @@ func healthPod(role string, ordinal, member int, ready *bool, uid string) core.P func healthBool(v bool) *bool { return &v } +// stampHealthGroupTotals writes onto every member the Kueue group total a rendered replica always +// carries, taken from the size of the role it belongs to. +// +// THE STAMP IS WHAT MAKES THESE FIXTURES HONEST RATHER THAN MERELY PASSING. The completeness leg +// reads a replica's deployed size from its members rather than from the role, so a Pod without the +// annotation reads as a replica of one -- and every case here below would then be judging a +// multi-member replica against a total of one. healthPod is given the total it was rendered with by +// this call rather than by an argument, because the size belongs to the deployment and a case that +// wants a replica built at a DIFFERENT size than the role declares -- which is the state this whole +// change exists for -- stamps it by hand and is explicit about doing so. +func stampHealthGroupTotals(md *workercore.ModelDeployment, pods []core.Pod) []core.Pod { + sizes := make(map[string]int, len(md.Spec.Roles)) + for i := range md.Spec.Roles { + sizes[md.Spec.Roles[i].Name] = modelDeploymentRoleSize(&md.Spec.Roles[i]) + } + + stamped := make([]core.Pod, len(pods)) + for i := range pods { + stamped[i] = *pods[i].DeepCopy() + total := sizes[modelDeploymentPodRole(&stamped[i])] + if total < 1 { + total = 1 + } + if stamped[i].Annotations == nil { + stamped[i].Annotations = map[string]string{} + } + stamped[i].Annotations[kueuepodconst.GroupTotalCountAnnotation] = strconvx.Itoa(total) + } + + return stamped +} + // healthDeployment builds a deployment of one engine-shaped role of the given replica size, and // optionally a second role, which is what the P/D shape needs. func healthDeployment(replicaSize int32, extraRoles ...string) *workercore.ModelDeployment { @@ -352,6 +385,7 @@ func TestWholeGroupHealthPredicate(t *testing.T) { if pods == nil { pods = []core.Pod{healthPod("server", 0, 0, healthBool(true), "uid-a")} } + pods = stampHealthGroupTotals(md, pods) qualifications := qualifyModelDeploymentInstances( context.Background(), md, pods, tc.pending, boundProbeFetch, @@ -529,8 +563,9 @@ func TestEndpointEligibilityStatusMapping(t *testing.T) { } } + pods := stampHealthGroupTotals(md, tc.pods) qualifications := qualifyModelDeploymentInstances( - context.Background(), md, tc.pods, tc.pending, boundProbeFetch, + context.Background(), md, pods, tc.pending, boundProbeFetch, ) holder := &workercore.ModelDeployment{ObjectMeta: meta.ObjectMeta{Name: md.Name}} observeModelDeploymentEndpointEligibility(holder, qualifications) @@ -543,7 +578,7 @@ func TestEndpointEligibilityStatusMapping(t *testing.T) { "the machine-readable class of that verdict") roles := modelDeploymentRoleStatuses( - md, tc.pods, nil, modelDeploymentRoleQualified(qualifications), + md, pods, nil, modelDeploymentRoleQualified(qualifications), ) require.Len(t, roles, 1) assert.Equal(t, tc.wantEligible, roles[0].Endpoints.Eligible, @@ -627,15 +662,12 @@ func TestACondemnedReplicaIsNeverProbed(t *testing.T) { t.Run("an incomplete replica condemns it before any probe", func(t *testing.T) { calls = 0 md := healthDeployment(2) - pods := []core.Pod{ + // One member of a replica declared at two. The group total is stamped from the role, + // because these members were rendered from it; a replica short of a DIFFERENT size than + // the role declares is the edit case and is asserted in the table above instead. + pods := stampHealthGroupTotals(md, []core.Pod{ healthPod("server", 0, 0, healthBool(true), "uid-a"), - healthPod("server", 0, 1, healthBool(true), "uid-b"), - healthPod("server", 1, 0, healthBool(true), "uid-c"), - } - _ = pods - pods = []core.Pod{ - healthPod("server", 0, 0, healthBool(true), "uid-a"), - } + }) qualifications := qualifyModelDeploymentInstances( context.Background(), md, pods, modelDeploymentPendingReplacement{}, countingFetch, diff --git a/pkg/worker/controllers/worker/model_deployment_initial_queue.go b/pkg/worker/controllers/worker/model_deployment_initial_queue.go new file mode 100644 index 000000000..f48e4ac9e --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_initial_queue.go @@ -0,0 +1,92 @@ +package worker + +import ( + "context" + + core "k8s.io/api/core/v1" + kerrors "k8s.io/apimachinery/pkg/api/errors" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/sets" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + utilpod "sigs.k8s.io/kueue/pkg/util/pod" +) + +// initialQueuedReplica verifies that every current member still waits for its first admission. +// Kueue removes the admission gate before starting a Pod. Kubernetes does not permit adding it back. +func (r *ModelDeploymentReconciler) initialQueuedReplica( + ctx context.Context, view modelDeploymentReplicaView, workloads []kueue.Workload, +) (bool, error) { + if !view.Seated || !modelDeploymentDeployedReplicaShape(view).modelDeploymentReplicaIsWhole() { + return false, nil + } + + // A member whose cached status already shows progress cannot be in the initial queue: the gate + // is never added back and a node assignment or container status never goes away. Only the + // survivors are confirmed on the API server. + for _, member := range view.Members { + if member.Spec.NodeName != "" || len(member.Status.ContainerStatuses) != 0 || + len(member.Status.InitContainerStatuses) != 0 || + len(member.Status.EphemeralContainerStatuses) != 0 { + return false, nil + } + } + + for i, member := range view.Members { + standing := new(core.Pod) + if err := r.APIReader.Get(ctx, ctrlcli.ObjectKeyFromObject(member), standing); err != nil { + if kerrors.IsNotFound(err) { + return false, nil + } + return false, err + } + if standing.UID != member.UID || standing.DeletionTimestamp != nil || + standing.Status.Phase != core.PodPending || standing.Spec.NodeName != "" || + !utilpod.HasGate(standing, kueuepodconst.SchedulingGateName) || + len(standing.Status.ContainerStatuses) != 0 || + len(standing.Status.InitContainerStatuses) != 0 || + len(standing.Status.EphemeralContainerStatuses) != 0 { + return false, nil + } + view.Members[i] = standing + } + + wl := replicaOwningWorkload(view.Members, workloadPointers(workloads)) + if wl == nil || wl.UID == "" || wl.DeletionTimestamp != nil || !workloadInInitialQueue(wl) { + return false, nil + } + for _, member := range view.Members { + if !modelDeploymentWorkloadOwnsAny(wl, sets.New(member.UID)) { + return false, nil + } + } + + return true, nil +} + +// workloadInInitialQueue rejects admission and evidence of an earlier execution or eviction. +// A false eviction condition can retain history after Kueue grants quota again. +func workloadInInitialQueue(wl *kueue.Workload) bool { + if wl.Status.AccumulatedPastExecutionTimeSeconds != nil || wl.Status.RequeueState != nil || + len(wl.Status.ReclaimablePods) != 0 || + (wl.Status.SchedulingStats != nil && len(wl.Status.SchedulingStats.Evictions) != 0) { + return false + } + for _, condition := range wl.Status.Conditions { + switch condition.Type { + case kueue.WorkloadAdmitted: + if condition.Status != meta.ConditionFalse { + return false + } + case kueue.WorkloadEvicted, kueue.WorkloadPreempted, kueue.WorkloadRequeued, kueue.WorkloadFinished: + return false + case kueue.WorkloadPodsReady: + if condition.Status != meta.ConditionFalse || condition.Reason == kueue.WorkloadWaitForRecovery { + return false + } + } + } + + return true +} diff --git a/pkg/worker/controllers/worker/model_deployment_initial_queue_test.go b/pkg/worker/controllers/worker/model_deployment_initial_queue_test.go new file mode 100644 index 000000000..39dfdb173 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_initial_queue_test.go @@ -0,0 +1,378 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrlrecord "k8s.io/client-go/tools/record" + "k8s.io/utils/ptr" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/nodefeature" +) + +func TestLifecycle_AnInitialQueuedRoleAcceptsUpdatedSchedulingInputs(t *testing.T) { + cases := []struct { + name string + replicas, size int32 + queuedOrdinal int + admitSibling, changePool, reserved bool + }{ + {name: "resource reduction", replicas: 1, size: 1}, + {name: "pool change", replicas: 1, size: 1, changePool: true}, + {name: "complete two-member group", replicas: 1, size: 2}, + {name: "queued group before healthy higher ordinal", replicas: 2, size: 1, admitSibling: true}, + {name: "one slot for multiple initial queues", replicas: 2, size: 1, queuedOrdinal: 1}, + {name: "first quota reservation before admission", replicas: 1, size: 1, reserved: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = tc.replicas + md.Spec.Roles[0].ReplicaSize = tc.size + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{Accelerator: ptr.To(resource.MustParse("2"))} + }) + f := newLifecycleFixture(t, md, newRenderInstanceTypeB()) + f.pass(false) + for ordinal := range int(tc.replicas) { + setInitialReplicaQueue(t, f, ordinal) + } + standInForKueue(t, f.cli, false) + if tc.admitSibling { + setInitialReplicaAdmitted(t, f, 1) + } + before := f.live("server", tc.queuedOrdinal) + require.Len(t, before, int(tc.size)) + accName := nodefeature.GetAcceleratableResourceName( + nodefeature.ManufacturerNVIDIA, workercore.DeviceAllocationModeExclusive) + qtyEqual(t, qty("2"), before[0].Spec.Containers[0].Resources.Requests[accName], "the initial request") + oldWorkload := f.workloadNamed(before[0].Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, oldWorkload) + if tc.reserved { + oldWorkload.Status.Admission = &kueue.Admission{ClusterQueue: "first-reservation"} + oldWorkload.Status.Conditions = []meta.Condition{ + initialQueueCondition(kueue.WorkloadQuotaReserved, meta.ConditionTrue, "QuotaReserved"), + initialQueueCondition(kueue.WorkloadAdmitted, meta.ConditionFalse, "NoReservation"), + initialQueueCondition(kueue.WorkloadPodsReady, meta.ConditionFalse, kueue.WorkloadWaitForStart), + } + require.NoError(t, f.cli.Status().Update(context.Background(), oldWorkload)) + } + _, holding := f.slot("server") + require.False(t, holding) + untouched := initialQueueSiblingUIDs(t, f, tc.queuedOrdinal) + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Resources.Accelerator = ptr.To(resource.MustParse("1")) + queue := "queue-for-h20-8x" + if tc.changePool { + edited.Spec.Roles[0].InstanceType = "a100-8x" + queue = "queue-for-a100-8x" + } + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + slot, holding := f.slot("server") + require.True(t, holding, "an initial queue starts replacement before any admission") + require.Equal(t, tc.queuedOrdinal, slot.Ordinal) + require.Len(t, slot.MemberUIDs, int(tc.size)) + require.Empty(t, f.live("server", tc.queuedOrdinal)) + f.admit() + f.pass(false) + setInitialReplicaQueue(t, f, tc.queuedOrdinal) + standInForKueue(t, f.cli, false) + for range 6 { + f.pass(false) + } + members := f.live("server", tc.queuedOrdinal) + require.Len(t, members, int(tc.size)) + for _, member := range members { + for _, old := range before { + require.NotEqual(t, old.UID, member.UID) + } + qtyEqual(t, qty("1"), member.Spec.Containers[0].Resources.Requests[accName], "the replacement request") + qtyEqual(t, qty("16"), member.Spec.Containers[0].Resources.Limits[core.ResourceCPU], "derived CPU capacity") + require.Equal(t, queue, member.Labels[kueuectrlconst.QueueLabel]) + } + fresh := f.workloadNamed(members[0].Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, fresh) + require.NotEqual(t, oldWorkload.UID, fresh.UID) + require.Equal(t, kueue.LocalQueueName(queue), fresh.Spec.QueueName) + var count int32 + for _, podSet := range fresh.Spec.PodSets { + count += podSet.Count + qtyEqual(t, qty("1"), podSet.Template.Spec.Containers[0].Resources.Requests[accName], "the composed request") + } + require.Equal(t, tc.size, count) + require.Equal(t, untouched, initialQueueSiblingUIDs(t, f, tc.queuedOrdinal), "queued replacement preserves sibling Pod and Workload identities") + slot, holding = f.slot("server") + require.True(t, holding) + require.Equal(t, tc.queuedOrdinal, slot.Ordinal) + setInitialReplicaAdmitted(t, f, tc.queuedOrdinal) + f.pass(false) + require.Len(t, f.live("server", tc.queuedOrdinal), int(tc.size)) + for range 8 { + f.admit() + for ordinal := range int(tc.replicas) { + if len(f.live("server", ordinal)) > 0 { + setInitialReplicaAdmitted(t, f, ordinal) + } + } + f.pass(false) + } + _, holding = f.slot("server") + require.False(t, holding, "every replacement has gained fresh admission") + for ordinal := range int(tc.replicas) { + settled := f.live("server", ordinal) + require.Len(t, settled, int(tc.size)) + for _, member := range settled { + qtyEqual(t, qty("1"), member.Spec.Containers[0].Resources.Requests[accName], "the converged request") + require.Equal(t, queue, member.Labels[kueuectrlconst.QueueLabel]) + } + } + }) + } +} + +func TestLifecycle_AnInitialQueueDoesNotIncludeExecutedOrEvictedGroups(t *testing.T) { + cases := []struct { + name, condition, reason string + status meta.ConditionStatus + past, requeue, statistics, ungated, assigned, running bool + }{ + {name: "admission revoked after execution", ungated: true, assigned: true, running: true, condition: kueue.WorkloadAdmitted, status: meta.ConditionFalse, reason: "NoReservation"}, + {name: "preempted", condition: kueue.WorkloadPreempted, status: meta.ConditionTrue}, + {name: "eviction history after reservation", condition: kueue.WorkloadEvicted, status: meta.ConditionFalse}, + {name: "requeued", condition: kueue.WorkloadRequeued, status: meta.ConditionTrue}, + {name: "previously ready", condition: kueue.WorkloadPodsReady, status: meta.ConditionTrue}, + {name: "waiting for recovery", condition: kueue.WorkloadPodsReady, status: meta.ConditionFalse, reason: kueue.WorkloadWaitForRecovery}, + {name: "zero-second prior execution", past: true}, + {name: "requeue state", requeue: true}, + {name: "eviction statistics", statistics: true}, + {name: "gate already removed", ungated: true}, + {name: "assigned node", assigned: true}, + {name: "runtime status", running: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newLifecycleFixture(t, newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 1 })) + f.pass(false) + setInitialReplicaQueue(t, f, 0) + standInForKueue(t, f.cli, false) + pod := f.live("server", 0)[0] + if tc.ungated { + pod.Spec.SchedulingGates = nil + } + if tc.assigned { + pod.Spec.NodeName = "assigned-node" + } + require.NoError(t, f.cli.Update(context.Background(), pod)) + if tc.running { + pod.Status.Phase = core.PodRunning + pod.Status.ContainerStatuses = []core.ContainerStatus{{Name: pod.Spec.Containers[0].Name}} + require.NoError(t, f.cli.Status().Update(context.Background(), pod)) + } + wl := f.workloadNamed(pod.Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, wl) + if tc.condition != "" { + wl.Status.Conditions = []meta.Condition{initialQueueCondition(tc.condition, tc.status, tc.reason)} + } + if tc.past { + wl.Status.AccumulatedPastExecutionTimeSeconds = ptr.To(int32(0)) + } + if tc.requeue { + wl.Status.RequeueState = &kueue.RequeueState{} + } + if tc.statistics { + wl.Status.SchedulingStats = &kueue.SchedulingStats{Evictions: []kueue.WorkloadSchedulingStatsEviction{{Reason: "Preempted", Count: 1}}} + } + require.NoError(t, f.cli.Status().Update(context.Background(), wl)) + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = []string{"sleep", "100"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + for range 6 { + f.pass(false) + } + _, holding := f.slot("server") + require.False(t, holding) + members := f.live("server", 0) + require.Len(t, members, 1) + require.Equal(t, pod.UID, members[0].UID) + preserved := f.workloadNamed(wl.Name) + require.NotNil(t, preserved) + require.Equal(t, wl.UID, preserved.UID) + }) + } +} + +func TestLifecycle_InitialQueueEligibilityReadsTheServerGate(t *testing.T) { + f := newLifecycleFixture(t, newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 1 })) + f.pass(false) + setInitialReplicaQueue(t, f, 0) + standInForKueue(t, f.cli, false) + stale := f.live("server", 0)[0] + standing := stale.DeepCopy() + standing.Spec.SchedulingGates = nil + require.NoError(t, f.cli.Update(context.Background(), standing)) + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = []string{"sleep", "100"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + cache := ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + List: func(ctx context.Context, next ctrlcli.WithWatch, list ctrlcli.ObjectList, opts ...ctrlcli.ListOption) error { + if err := next.List(ctx, list, opts...); err != nil { + return err + } + if pods, ok := list.(*core.PodList); ok { + for i := range pods.Items { + if pods.Items[i].UID == stale.UID { + pods.Items[i] = *stale.DeepCopy() + } + } + } + return nil + }, + }) + r := &ModelDeploymentReconciler{Client: cache, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64)} + _, err := reconcileModelDeploymentWith(t, r) + require.NoError(t, err) + _, holding := f.slot("server") + require.False(t, holding) + require.Len(t, f.live("server", 0), 1) + require.Equal(t, stale.UID, f.live("server", 0)[0].UID) +} + +func setInitialReplicaQueue(t *testing.T, f *lifecycleFixture, ordinal int) { + t.Helper() + members := f.live("server", ordinal) + require.NotEmpty(t, members) + for _, pod := range members { + pod.Spec.SchedulingGates = []core.PodSchedulingGate{{Name: kueuepodconst.SchedulingGateName}} + require.NoError(t, f.cli.Update(context.Background(), pod)) + pod.Status.Phase = core.PodPending + require.NoError(t, f.cli.Status().Update(context.Background(), pod)) + } +} + +func setInitialReplicaAdmitted(t *testing.T, f *lifecycleFixture, ordinal int) { + t.Helper() + members := f.live("server", ordinal) + require.NotEmpty(t, members) + for _, pod := range members { + pod.Spec.SchedulingGates = nil + pod.Spec.NodeName = "assigned-node" + require.NoError(t, f.cli.Update(context.Background(), pod)) + pod.Status.Phase = core.PodRunning + require.NoError(t, f.cli.Status().Update(context.Background(), pod)) + } + wl := f.workloadNamed(members[0].Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, wl) + wl.Status.Conditions = []meta.Condition{initialQueueCondition(kueue.WorkloadAdmitted, meta.ConditionTrue, "Admitted")} + require.NoError(t, f.cli.Status().Update(context.Background(), wl)) +} + +func initialQueueCondition(kind string, status meta.ConditionStatus, reason string) meta.Condition { + if reason == "" { + reason = kind + } + return meta.Condition{Type: kind, Status: status, Reason: reason, LastTransitionTime: meta.Now()} +} + +func initialQueueSiblingUIDs(t *testing.T, f *lifecycleFixture, selected int) map[int][]types.UID { + t.Helper() + uids := map[int][]types.UID{} + for ordinal := range int(f.md.Spec.Roles[0].Replicas) { + if ordinal == selected { + continue + } + members := f.live("server", ordinal) + require.NotEmpty(t, members) + for _, member := range members { + uids[ordinal] = append(uids[ordinal], member.UID) + } + wl := f.workloadNamed(members[0].Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, wl) + uids[ordinal] = append(uids[ordinal], wl.UID) + } + return uids +} + +func TestLifecycle_InitialQueueSlotAcceptsASecondResourceAndPoolEdit(t *testing.T) { + f := newLifecycleFixture(t, newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 1 + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{Accelerator: ptr.To(resource.MustParse("2"))} + }), newRenderInstanceTypeB()) + f.pass(false) + setInitialReplicaQueue(t, f, 0) + standInForKueue(t, f.cli, false) + first := getModelDeployment(t, f.cli) + first.Spec.Roles[0].Resources.Accelerator = ptr.To(resource.MustParse("1")) + require.NoError(t, f.cli.Update(context.Background(), first)) + f.pass(false) + f.admit() + f.pass(false) + setInitialReplicaQueue(t, f, 0) + standInForKueue(t, f.cli, false) + queued := f.live("server", 0)[0] + oldWorkload := f.workloadNamed(queued.Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, oldWorkload) + second := getModelDeployment(t, f.cli) + second.Spec.Roles[0].Resources.Accelerator = ptr.To(resource.MustParse("3")) + second.Spec.Roles[0].InstanceType = "a100-8x" + require.NoError(t, f.cli.Update(context.Background(), second)) + f.pass(false) + f.admit() + f.pass(false) + setInitialReplicaQueue(t, f, 0) + standInForKueue(t, f.cli, false) + for range 3 { + f.pass(false) + } + members := f.live("server", 0) + require.Len(t, members, 1) + require.NotEqual(t, queued.UID, members[0].UID) + accName := nodefeature.GetAcceleratableResourceName(nodefeature.ManufacturerNVIDIA, workercore.DeviceAllocationModeExclusive) + qtyEqual(t, qty("3"), members[0].Spec.Containers[0].Resources.Requests[accName], "superseding request") + wl := f.workloadNamed(members[0].Labels[kueuepodconst.GroupNameLabel]) + require.NotNil(t, wl) + require.NotEqual(t, oldWorkload.UID, wl.UID) + require.Equal(t, kueue.LocalQueueName("queue-for-a100-8x"), wl.Spec.QueueName) + require.Len(t, wl.Spec.PodSets, 1) + qtyEqual(t, qty("3"), wl.Spec.PodSets[0].Template.Spec.Containers[0].Resources.Requests[accName], "superseding composition") + slot, holding := f.slot("server") + require.True(t, holding) + require.Equal(t, 0, slot.Ordinal) +} + +// TestInitialQueuedReplica_CachedProgressSkipsTheServerRead locks in the cheap pre-filter: a member +// whose cached status already shows a node assignment cannot be in the initial queue, so the replica +// is refused without a quorum read per member. +func TestInitialQueuedReplica_CachedProgressSkipsTheServerRead(t *testing.T) { + f := newLifecycleFixture(t, newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 1 })) + f.pass(false) + setInitialReplicaQueue(t, f, 0) + + member := f.live("server", 0)[0].DeepCopy() + member.Spec.NodeName = "node-a" + reads := 0 + reader := ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + Get: func(ctx context.Context, next ctrlcli.WithWatch, key ctrlcli.ObjectKey, obj ctrlcli.Object, opts ...ctrlcli.GetOption) error { + reads++ + return next.Get(ctx, key, obj, opts...) + }, + }) + r := &ModelDeploymentReconciler{Client: f.cli, APIReader: reader, Recorder: ctrlrecord.NewFakeRecorder(64)} + + queued, err := r.initialQueuedReplica(context.Background(), modelDeploymentReplicaView{ + Role: "server", Ordinal: 0, Seated: true, Members: []*core.Pod{member}, + }, nil) + require.NoError(t, err) + require.False(t, queued) + require.Zero(t, reads, "a member with a node assignment must be refused before any server read") +} diff --git a/pkg/worker/controllers/worker/model_deployment_joint_admission.go b/pkg/worker/controllers/worker/model_deployment_joint_admission.go index 3bdcc99ea..ea8ef475c 100644 --- a/pkg/worker/controllers/worker/model_deployment_joint_admission.go +++ b/pkg/worker/controllers/worker/model_deployment_joint_admission.go @@ -408,7 +408,7 @@ type jointHeld struct { func (r *ModelDeploymentJointAdmissionReconciler) jointVerdict( ctx context.Context, md *workercore.ModelDeployment, ) (jointHeld, error) { - byGroup, liveByGroup, err := modelDeploymentReplicaGroups(ctx, r.Client, md) + byGroup, liveByGroup, shapes, err := modelDeploymentReplicaGroups(ctx, r.Client, md) if err != nil { return jointHeld{}, err } @@ -457,7 +457,12 @@ func (r *ModelDeploymentJointAdmissionReconciler) jointVerdict( // as present sends the verdict to the message below, which tells an operator to look // for a quota problem that does not exist. At one member per replica the two readings // are the same test, which is why this was a comparison against zero. - if liveByGroup[group].Len() < modelDeploymentRoleSize(role) { + // + // The group's own declared shape decides, never the role's size: a role being edited + // describes the replica yet to be built, and a group the members cannot reconcile reads + // as assembling too. + if shape := shapes[group]; !shape.modelDeploymentReplicaIsWhole() || + liveByGroup[group].Len() < shape.Total { roleAssembling = true } if anyWorkloadHoldsQuotaFor(wlList.Items, members) || @@ -574,18 +579,25 @@ func anyWorkloadOwnsAny(wls []kueue.Workload, members sets.Set[types.UID]) bool // counts every one, since a terminating Pod is still owned and the Workload holding it is still the // one a sibling's event has to reach. Collapsing the two either reads a rebuilding group as complete // or drops a Workload from the mapping while its last Pod is leaving. +// modelDeploymentReplicaGroups splits this deployment's Pods by their Kueue group, and reads each +// group's deployed shape off the same Pods. +// +// The shape is read with the same reader the other observations use, seats included, so a group +// whose members all claim one seat is not mistaken for one still assembling. func modelDeploymentReplicaGroups( ctx context.Context, cli ctrlcli.Client, md *workercore.ModelDeployment, -) (all, live map[string]sets.Set[types.UID], err error) { +) (all, live map[string]sets.Set[types.UID], shapes map[string]modelDeploymentReplicaShape, err error) { podList := new(core.PodList) if err = cli.List(ctx, podList, ctrlcli.InNamespace(md.Namespace), ctrlcli.MatchingLabels{modelDeploymentLabelKeyInstance: md.Name}, ctrlclix.WithoutQuorum); err != nil { - return nil, nil, fmt.Errorf("list replicas: %w", err) + return nil, nil, nil, fmt.Errorf("list replicas: %w", err) } all, live = make(map[string]sets.Set[types.UID]), make(map[string]sets.Set[types.UID]) + shapes = make(map[string]modelDeploymentReplicaShape) + members := make(map[string][]*core.Pod) for i := range podList.Items { pod := &podList.Items[i] if !modelDeploymentOwns(pod, md) { @@ -595,6 +607,7 @@ func modelDeploymentReplicaGroups( if all[group] == nil { all[group] = sets.New[types.UID]() } + members[group] = append(members[group], pod) all[group].Insert(pod.UID) if pod.DeletionTimestamp != nil { continue @@ -605,7 +618,13 @@ func modelDeploymentReplicaGroups( live[group].Insert(pod.UID) } - return all, live, nil + // Seats are validated with the same reader the other observations use. + for group, groupMembers := range members { + shapes[group] = modelDeploymentDeployedReplicaShape( + modelDeploymentReplicaView{Members: groupMembers}) + } + + return all, live, shapes, nil } // anyWorkloadHoldsQuotaFor reports whether one of these Workloads owns any of the group's replicas @@ -774,7 +793,7 @@ func (r *ModelDeploymentJointAdmissionReconciler) jointSiblings( // Every replica, terminating ones included: this maps an event to the siblings that have to see // it, and a Workload still holding a leaving Pod is still one of them. - byGroup, _, err := modelDeploymentReplicaGroups(ctx, r.Client, md) + byGroup, _, _, err := modelDeploymentReplicaGroups(ctx, r.Client, md) if err != nil { logger.Error(err, "index the deployment's replicas for sibling mapping") return nil diff --git a/pkg/worker/controllers/worker/model_deployment_joint_admission_test.go b/pkg/worker/controllers/worker/model_deployment_joint_admission_test.go index 74425b480..5d4d34b9b 100644 --- a/pkg/worker/controllers/worker/model_deployment_joint_admission_test.go +++ b/pkg/worker/controllers/worker/model_deployment_joint_admission_test.go @@ -22,6 +22,7 @@ import ( ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuepod "sigs.k8s.io/kueue/pkg/controller/jobs/pod" kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" kueueworkload "sigs.k8s.io/kueue/pkg/workload" @@ -1063,7 +1064,29 @@ func TestModelDeploymentJointAdmission_AnAbsentReplicaIsReadByItsDeparture(t *te // subresources for the convergence loop, and the Workload's and the check's for the barrier's // verdict writes. func newRolloutClient(objs ...ctrlcli.Object) ctrlcli.Client { + allocated := 0 return ctrlfake.NewClientBuilder(). + WithInterceptorFuncs(ctrlinterceptor.Funcs{ + Create: func(ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.CreateOption, + ) error { + switch obj.(type) { + case *core.Pod, *kueue.Workload: + if obj.GetUID() == "" { + allocated++ + obj.SetUID(types.UID(fmt.Sprintf("rollout-created-%d", allocated))) + } + } + return next.Create(ctx, obj, opts...) + }, + }). + WithIndex(&core.Pod{}, kueuepod.PodGroupNameCacheKey, func(obj ctrlcli.Object) []string { + group := obj.GetLabels()[kueuepodconst.GroupNameLabel] + if group == "" { + return nil + } + return []string{group} + }). WithScheme(scheme.Scheme). WithStatusSubresource( &workercore.ModelDeployment{}, &workercore.KVCachePoolBinding{}, @@ -1073,19 +1096,14 @@ func newRolloutClient(objs ...ctrlcli.Object) ctrlcli.Client { Build() } -// stampReplicaUIDs gives every replica a distinct UID, standing in for the API server's half of a -// create: the fake client assigns none, and every ownership question in this controller is answered -// by UID -- a fleet of empty UIDs makes one Workload "own" every replica, and a barrier test that -// cannot tell replicas apart proves nothing. -func stampReplicaUIDs(t *testing.T, cli ctrlcli.Client) { +// assertReplicaUIDs checks the identities assigned at creation, before any owner comparisons. +func assertReplicaUIDs(t *testing.T, cli ctrlcli.Client) { t.Helper() - - for i, pod := range replicaPods(t, cli) { - if pod.UID != "" { - continue - } - pod.UID = types.UID(fmt.Sprintf("uid-%s-%d", pod.Name, i)) - require.NoError(t, cli.Update(context.Background(), &pod)) + seen := sets.New[types.UID]() + for _, pod := range replicaPods(t, cli) { + require.NotEmpty(t, pod.UID) + require.False(t, seen.Has(pod.UID), "every incarnation has a distinct identity") + seen.Insert(pod.UID) } } @@ -1111,12 +1129,10 @@ func armReplicaFinalizers(t *testing.T, cli ctrlcli.Client) { func composeWorkloadFor(t *testing.T, cli ctrlcli.Client, pod core.Pod) { t.Helper() - wl := new(kueue.Workload) - wl.Name, wl.Namespace = pod.Labels[kueuepodconst.GroupNameLabel], pod.Namespace - wl.UID = types.UID("uid-" + wl.Name) - wl.OwnerReferences = []meta.OwnerReference{{ - APIVersion: "v1", Kind: "Pod", Name: pod.Name, UID: pod.UID, - }} + wl, err := kueueComposeGroup(context.Background(), cli, pod.Labels[kueuepodconst.GroupNameLabel]) + require.NoError(t, err) + require.NotNil(t, wl, "complete authoritative members compose their own Workload") + wl.Status.Conditions = []meta.Condition{{ Type: kueue.WorkloadQuotaReserved, Status: meta.ConditionTrue, @@ -1178,7 +1194,7 @@ func rolloutSetup(t *testing.T, cli ctrlcli.Client) { _, err := reconcileModelDeployment(t, cli) require.NoError(t, err) - stampReplicaUIDs(t, cli) + assertReplicaUIDs(t, cli) armReplicaFinalizers(t, cli) for _, pod := range replicaPods(t, cli) { composeWorkloadFor(t, cli, pod) @@ -1221,6 +1237,13 @@ func TestModelDeploymentJointAdmission_AnAdmittedWorkloadIsSkippedThroughARollou require.NoError(t, cli.Update(ctx, changed)) for round := 0; round < 40; round++ { + for _, wl := range rolloutWorkloads(t, cli) { + if wl.DeletionTimestamp == nil { + continue + } + wl.Finalizers = nil + require.NoError(t, cli.Update(ctx, &wl)) + } // KUEUE RELEASES A DEPARTED REPLICA'S FINALIZER ONCE ITS WORKLOAD IS GONE. The converger // deletes the pair together, and the fake cluster holds the Pod until this clears it -- the // compressed shape of the drain a real cluster measures in tens of seconds. @@ -1235,7 +1258,7 @@ func TestModelDeploymentJointAdmission_AnAdmittedWorkloadIsSkippedThroughARollou _, err := reconcileModelDeployment(t, cli) require.NoError(t, err, "round %d", round) - stampReplicaUIDs(t, cli) + assertReplicaUIDs(t, cli) // THE BARRIER RUNS WHILE THE NEWEST REPLICAS HAVE NO WORKLOAD YET: Kueue composes on its // own events, and this loop composes at the round's end. The gap is the state the skip @@ -1318,6 +1341,13 @@ func TestModelDeploymentJointAdmission_ARolloutRunsToCompletionWithTheBarrierAct admittedMidReplacement := false for round := 0; round < 40; round++ { + for _, wl := range rolloutWorkloads(t, cli) { + if wl.DeletionTimestamp == nil { + continue + } + wl.Finalizers = nil + require.NoError(t, cli.Update(ctx, &wl)) + } for _, pod := range replicaPods(t, cli) { if pod.DeletionTimestamp == nil { continue @@ -1348,7 +1378,7 @@ func TestModelDeploymentJointAdmission_ARolloutRunsToCompletionWithTheBarrierAct _, err := reconcileModelDeployment(t, cli) require.NoError(t, err, "round %d", round) - stampReplicaUIDs(t, cli) + assertReplicaUIDs(t, cli) // KUEUE COMPOSES PER GROUP ON ITS OWN EVENTS, and one composition per round is the // interleaving where a sibling is judged while another ordinal's recomposition has not @@ -1425,7 +1455,7 @@ func anyReplicaMidReplacement(t *testing.T, cli ctrlcli.Client) bool { t.Helper() md := getModelDeployment(t, cli) - byGroup, liveByGroup, err := modelDeploymentReplicaGroups(context.Background(), cli, md) + byGroup, liveByGroup, _, err := modelDeploymentReplicaGroups(context.Background(), cli, md) require.NoError(t, err) wlList := new(kueue.WorkloadList) @@ -1567,3 +1597,79 @@ func TestModelDeploymentJointAdmissionCheckReconciler_ExpectedWriteFailuresAreQu }) } } + +// TestModelDeploymentJointAdmission_AnEditedRoleIsNotStillAssembling is the joint barrier's half of +// this task. +// +// THE ROLE HAS BEEN EDITED AND THE REPLICA HAS NOT. The prefiller now declares four members per +// replica; the replica standing was rendered at two and holds both of them, with a Workload holding +// its quota. Reading the live count against the ROLE's size says the group has not reached its +// declared total, which is the exact state Kueue composes no Workload for -- so the verdict reports +// a deployment still assembling, and a barrier that reports assembling HOLDS. A perfectly healthy, +// already-admitted deployment would sit behind a configuration change for as long as the rollout +// takes. +// +// The group declares two of its own accord, on every member, and that is the figure the verdict is +// entitled to compare against. +func TestModelDeploymentJointAdmission_AnEditedRoleIsNotStillAssembling(t *testing.T) { + md := jointDeployment("qwen", "h20-8x", "a100-8x") + md.Spec.Roles[0].ReplicaSize = 4 + + group := modelDeploymentReplicaGroupName(md, "prefill", 0) + first := jointGroupPod("qwen-prefill-0-0", group, "qwen") + second := jointGroupPod("qwen-prefill-0-1", group, "qwen") + for _, member := range []*core.Pod{first, second} { + member.Annotations = map[string]string{kueuepodconst.GroupTotalCountAnnotation: "2"} + } + decoder := jointGroupPod("qwen-decode-0", + modelDeploymentReplicaGroupName(md, "decode", 0), "qwen") + decoder.Annotations = map[string]string{kueuepodconst.GroupTotalCountAnnotation: "1"} + + cli := newJointClient(jointCheckObject(), md, first, second, decoder, + jointWorkload("wl-first", true, first), jointWorkload("wl-second", true, decoder)) + + r := &ModelDeploymentJointAdmissionReconciler{Client: cli} + held, err := r.jointVerdict(context.Background(), md) + require.NoError(t, err) + + assert.Equal(t, kueue.CheckStateReady, held.State, + "both roles hold quota; an edited replica size is not a reason to hold a settled set") +} + +// TestModelDeploymentJointAdmission_AGroupOfOneSeatIsNotAssembling is the control for the barrier +// reading a group's own shape rather than a bare member count. +// +// A GROUP WHOSE MEMBERS ALL CLAIM ONE SEAT has not reached a total of two; it is one member counted +// twice. Reducing the total on its own called that group assembled, and the assembled branch tells +// an operator to wait for it to fill -- which is an instruction with no exit for a group that never +// will. The shape reader refuses the group as one this operator cannot classify instead, which is +// the answer the members actually support. +func TestModelDeploymentJointAdmission_AGroupOfOneSeatIsNotAssembling(t *testing.T) { + md := jointDeployment("qwen", "h20-8x", "a100-8x") + md.Spec.Roles[0].ReplicaSize = 2 + + group := modelDeploymentReplicaGroupName(md, "prefill", 0) + seated := func(name, seat string) *core.Pod { + pod := jointGroupPod(name, group, "qwen") + pod.Labels[modelDeploymentMemberIndexLabel] = seat + pod.Annotations = map[string]string{kueuepodconst.GroupTotalCountAnnotation: "2"} + + return pod + } + decoder := jointGroupPod("qwen-decode-0", + modelDeploymentReplicaGroupName(md, "decode", 0), "qwen") + decoder.Annotations = map[string]string{kueuepodconst.GroupTotalCountAnnotation: "1"} + + cli := newJointClient(jointCheckObject(), md, seated("qwen-prefill-0-0", "0"), + seated("qwen-prefill-0-1", "0"), decoder, + jointWorkload("wl-decoder", true, decoder)) + + r := &ModelDeploymentJointAdmissionReconciler{Client: cli} + held, err := r.jointVerdict(context.Background(), md) + require.NoError(t, err) + + assert.Equal(t, kueue.CheckStatePending, held.State, + "the decode role is admitted, so the only thing left to hold is the unclassifiable group") + assert.NotContains(t, held.Message, "quota", + "a duplicate seat is not a quota problem, and the message must not send an operator there") +} diff --git a/pkg/worker/controllers/worker/model_deployment_lifecycle_test.go b/pkg/worker/controllers/worker/model_deployment_lifecycle_test.go new file mode 100644 index 000000000..625e8718d --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_lifecycle_test.go @@ -0,0 +1,1198 @@ +package worker + +import ( + "context" + "errors" + "fmt" + "maps" + "slices" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + ctrlrecord "k8s.io/client-go/tools/record" + "k8s.io/utils/ptr" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" + kueuepod "sigs.k8s.io/kueue/pkg/controller/jobs/pod" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + + worker "gpustack.ai/gpustack/api/worker/v1" + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" +) + +// These are the observable lifecycle cases: each drives real reconcile passes over a fake cluster +// and asserts what a reader could see afterwards. The function-level composition controls in the +// sibling file cannot stand in for them, because what they prove is that a check rejects a +// Workload, not that the controller reaches the point where the check runs. + +// lifecycleFixture drives a deployment through passes, with the Kueue handshake between them. +type lifecycleFixture struct { + t *testing.T + md *workercore.ModelDeployment + // cli is the watch-capable client the fake client builds, so a case can wrap the very one the + // passes ran against rather than a second one over an empty cluster. + cli ctrlcli.WithWatch + past [][]string + // allocated counts the identities this fixture has handed out, which is what keeps two + // incarnations of one ordinal from being the same object to every comparison made here. + allocated int +} + +func newLifecycleFixture(t *testing.T, md *workercore.ModelDeployment, extra ...ctrlcli.Object) *lifecycleFixture { + t.Helper() + + return newLifecycleFixtureAround(t, md, newCompositionClient, extra...) +} + +// newLifecycleFixtureAround builds the fixture over a client the case builds itself, so a write can +// be made to fail the way a real one does rather than described in prose. +func newLifecycleFixtureAround( + t *testing.T, md *workercore.ModelDeployment, build func(...ctrlcli.Object) ctrlcli.WithWatch, + extra ...ctrlcli.Object, +) *lifecycleFixture { + t.Helper() + + objs := make([]ctrlcli.Object, 0, 2+len(extra)) + objs = append(objs, md, newRenderInstanceType()) + objs = append(objs, extra...) + // An object read from a server always carries a resource version, and the lifecycle writes its + // replacement intent through a real optimistic lock that needs one to compare against. + if md.ResourceVersion == "" { + md.ResourceVersion = "1" + } + cli := build(objs...) + stamped := getModelDeployment(t, cli) + stamped.ResourceVersion = "1" + require.NoError(t, cli.Update(context.Background(), stamped)) + + f := &lifecycleFixture{t: t, md: md} + f.cli = ctrlinterceptor.NewClient(cli, ctrlinterceptor.Funcs{ + Create: func(ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.CreateOption, + ) error { + switch obj.(type) { + case *core.Pod: + if obj.GetUID() == "" { + obj.SetUID(f.nextUID("pod")) + } + case *kueue.Workload: + if obj.GetUID() == "" { + obj.SetUID(f.nextUID("wl")) + } + } + return next.Create(ctx, obj, opts...) + }, + }) + return f +} + +// nextUID allocates an identity the way an API server would, once per object rather than once per +// name: two Pods of one ordinal and two Workloads of one group name are told apart by this and +// nothing else. +func (f *lifecycleFixture) nextUID(kind string) types.UID { + f.allocated++ + + return types.UID(fmt.Sprintf("%s-%03d", kind, f.allocated)) +} + +// groupsByName returns the live members of every group standing in the cluster, keyed by the group +// name the members carry. +func (f *lifecycleFixture) groupsByName() map[string][]*core.Pod { + f.t.Helper() + + groups := map[string][]*core.Pod{} + for _, pod := range replicaPods(f.t, f.cli) { + if pod.DeletionTimestamp != nil { + continue + } + group := pod.Labels[kueuepodconst.GroupNameLabel] + member := pod + groups[group] = append(groups[group], &member) + } + for group := range groups { + slices.SortFunc(groups[group], func(a, b *core.Pod) int { return strings.Compare(a.Name, b.Name) }) + } + + return groups +} + +// workloadNamed returns the Workload holding a group name, or nil when the name is free. +func (f *lifecycleFixture) workloadNamed(group string) *kueue.Workload { + f.t.Helper() + + wl := new(kueue.Workload) + if err := f.cli.Get(context.Background(), + ctrlcli.ObjectKey{Namespace: "team-a", Name: group}, wl); err != nil { + return nil + } + + return wl +} + +// pass constructs a new reconciler on every call, including the restart cases. +func (f *lifecycleFixture) pass(_ bool) { + f.t.Helper() + + r := &ModelDeploymentReconciler{ + Client: f.cli, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64), + } + _, err := reconcileModelDeploymentWith(f.t, r) + require.NoError(f.t, err, "pass %d", len(f.past)) + f.past = append(f.past, replicaNames(f.t, f.cli)) +} + +// passExpectingFailure runs a pass whose create is about to fail, and requires that it reported it. +// A failed create is still a pass that looked: the error is what tells the next one to try again. +func (f *lifecycleFixture) passExpectingFailure(fresh bool) { + f.t.Helper() + + r := &ModelDeploymentReconciler{ + Client: f.cli, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64), + } + if fresh { + r = &ModelDeploymentReconciler{ + Client: f.cli, APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64), + } + } + _, err := reconcileModelDeploymentWith(f.t, r) + require.Error(f.t, err, "pass %d reported the create that never landed", len(f.past)) + f.past = append(f.past, replicaNames(f.t, f.cli)) +} + +// admit runs the Kueue half of the handshake through Kueue's own constructor: departures finish, +// and every complete group composes its +// Workload and is admitted. +func (f *lifecycleFixture) admit() { + f.t.Helper() + ctx := context.Background() + + // A delete is accepted while the finalizer is on the Pod, so the group stays occupied until the + // finalizer comes off. That is the window the vacancy proof waits on. + for _, pod := range replicaPods(f.t, f.cli) { + if pod.DeletionTimestamp == nil || !slices.Contains(pod.Finalizers, kueuepodconst.PodFinalizer) { + continue + } + released := pod.DeepCopy() + released.Finalizers = nil + require.NoError(f.t, f.cli.Update(ctx, released)) + } + + // A composed Workload carries Kueue's finalizer too, so the group name stays held until it comes + // off. The vacancy proof reads the server rather than the cache for the same reason. + departing := new(kueue.WorkloadList) + require.NoError(f.t, f.cli.List(ctx, departing, ctrlcli.InNamespace("team-a"))) + for i := range departing.Items { + if departing.Items[i].DeletionTimestamp == nil { + continue + } + released := departing.Items[i].DeepCopy() + released.Finalizers = nil + require.NoError(f.t, f.cli.Update(ctx, released)) + } + + for _, group := range slices.Sorted(maps.Keys(f.groupsByName())) { + members := f.groupsByName()[group] + if held := f.workloadNamed(group); held != nil { + // A Workload already holding this name that owns none of the members standing now is not + // adopted. The constructor composes nothing under a taken name, so writing over one would + // assert something Kueue never does; the refusal is stated rather than worked around + // because refusing it is the property under test. + uids := sets.New[types.UID]() + for _, member := range members { + uids.Insert(member.UID) + } + require.True(f.t, modelDeploymentWorkloadOwnsAny(held, uids), + "the fixture will not let the pre-existing workload %q absorb the members standing now", + group) + + continue + } + + // AN INCOMPLETE GROUP, OR ONE WHOSE MEMBERS DISAGREE ON THEIR TOTAL, COMPOSES NOTHING. That + // is Kueue's own answer, and a pass that created one of those groups is a pass whose next + // step has to wait rather than a fixture failure. + composed, err := kueueComposeGroup(ctx, f.cli, group) + if err != nil || composed == nil { + continue + } + + composed.UID = f.nextUID("wl") + require.NoError(f.t, f.cli.Create(ctx, composed)) + + // Kueue puts the finalizer on the members of a group it has composed, and that is what makes + // the departure above take a pass rather than completing at once. + for _, member := range members { + held := new(core.Pod) + require.NoError(f.t, f.cli.Get(ctx, ctrlcli.ObjectKeyFromObject(member), held)) + if slices.Contains(held.Finalizers, kueuepodconst.PodFinalizer) { + continue + } + held.Finalizers = append(held.Finalizers, kueuepodconst.PodFinalizer) + require.NoError(f.t, f.cli.Update(ctx, held)) + } + + admitted := new(kueue.Workload) + require.NoError(f.t, f.cli.Get(ctx, + ctrlcli.ObjectKey{Namespace: "team-a", Name: group}, admitted)) + admitted.Status.Conditions = []meta.Condition{{ + Type: kueue.WorkloadAdmitted, + Status: meta.ConditionTrue, + Reason: "Admitted", + LastTransitionTime: meta.Now(), + }} + require.NoError(f.t, f.cli.Status().Update(ctx, admitted)) + } +} + +// live returns the live members at one ordinal. +func (f *lifecycleFixture) live(role string, ordinal int) []*core.Pod { + f.t.Helper() + + var members []*core.Pod + for _, pod := range replicaPods(f.t, f.cli) { + if pod.DeletionTimestamp != nil || modelDeploymentPodRole(&pod) != role { + continue + } + if at, ok := modelDeploymentPodOrdinal(&pod); ok && at == ordinal { + copied := pod.DeepCopy() + members = append(members, copied) + } + } + + return members +} + +// slot returns the role's recorded replacement slot and whether it has one. +func (f *lifecycleFixture) slot(role string) (modelDeploymentReplacementSlot, bool) { + f.t.Helper() + + return modelDeploymentReplacementSlotsOf(getModelDeployment(f.t, f.cli)).slotFor(role) +} + +// images reports the container image of every live member at one ordinal. +func (f *lifecycleFixture) images(role string, ordinal int) []string { + f.t.Helper() + + seen := make([]string, 0, len(f.live(role, ordinal))) + for _, member := range f.live(role, ordinal) { + seen = append(seen, member.Spec.Containers[0].Image) + } + + return seen +} + +// TestLifecycle_AQueuedReplacementHoldsTheRoleAcrossPassesAndRestarts covers the cadence rule: one +// unresolved replacement per role, and it survives a controller restart because the intent was +// persisted before anything was deleted. +func TestLifecycle_AQueuedReplacementHoldsTheRoleAcrossPassesAndRestarts(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 3 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + before := f.live("server", 2) + require.Len(t, before, 1, "three replicas of one member each") + + // The edit that makes every replica outdated. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + + f.pass(false) + slot, held := f.slot("server") + require.True(t, held, "selecting a replacement records the intent before deleting anything") + assert.Len(t, slot.MemberUIDs, 1, "and the slot captures the member it is about to remove") + assert.Equal(t, modelDeploymentReplacementCleanup, slot.Phase) + + // Kueue releases the finalizer the captured member holds, so it finishes leaving. The ordinal is + // empty from here on, and the replacement comes back queued behind the vacancy. + f.admit() + require.Empty(t, f.live("server", slot.Ordinal), "the captured member has finished leaving") + + // The next passes must not turn over another healthy replica, and a reconciler built fresh must + // reach the same conclusion from the persisted slot. + for pass := 0; pass < 3; pass++ { + f.pass(pass%2 == 0) + assert.Len(t, f.live("server", 1), 1, + "pass %d: the healthy replica beside the queued one is untouched", pass) + assert.Len(t, f.live("server", 0), 1, + "pass %d: and so is the one below it", pass) + } + assert.Contains(t, f.images("server", 2), "vllm/vllm-openai:v0.26.0", + "while the replacement itself is built from the new spec") + + // The admission releases the role, and only then does the next turnover happen. The pass that + // releases it may open the next slot in the same breath, so what is asserted is the turnover + // itself rather than the absence of a slot. + // Once the replacement is admitted the role is free again, so the remaining replicas turn over + // in their own turns rather than all at once. + for pass := 0; pass < 8; pass++ { + if allAtImage(f, "vllm/vllm-openai:v0.26.0", 3) { + break + } + f.pass(pass%2 == 0) + f.admit() + } + + for ordinal := range 3 { + members := f.live("server", ordinal) + require.Len(t, members, 1, "ordinal %d never empties", ordinal) + assert.Equal(t, "vllm/vllm-openai:v0.26.0", members[0].Spec.Containers[0].Image, + "ordinal %d carries the edit once the role has settled", ordinal) + } +} + +// TestLifecycle_PersistedIntentSurvivesAnInterruptionBetweenDeleteAndCreate covers the window the +// slot exists for: the members are gone, the record is not, and nothing observable names the +// replica being replaced. +func TestLifecycle_PersistedIntentSurvivesAnInterruptionBetweenDeleteAndCreate(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + before := getModelDeployment(t, f.cli) + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + + f.pass(false) + slot, held := f.slot("server") + require.True(t, held) + + // Everything the pass issued happens, and then the controller dies. + f.admit() + require.Len(t, f.live("server", slot.Ordinal), 0, "the departing member is gone from the cluster") + + // A fresh controller over the same cluster finds the ordinal empty and nothing else to go on. + f.pass(true) + after, stillHeld := f.slot("server") + require.True(t, stillHeld, + "an empty ordinal names no replica, so the slot is the only thing that can complete this one") + assert.Equal(t, slot.Ordinal, after.Ordinal, "and it is the same ordinal, not a new selection") + assert.NotEmpty(t, after.MemberHashes, "which now records what the replacement is being built from") + + f.pass(true) + f.admit() + f.pass(true) + assert.Equal(t, []string{"vllm/vllm-openai:v0.26.0"}, f.images("server", slot.Ordinal), + "and the ordinal comes back built from the new spec") + assert.Equal(t, before.Generation, getModelDeployment(t, f.cli).Generation, + "through all of it the spec itself never moved again") +} + +// TestLifecycle_ASecondEditSupersedesTheQueuedConfiguration covers the supersession rule: the +// queued configuration is discarded and rebuilt in the same slot, without a second one opening and +// without waiting for an admission that can never come. +func TestLifecycle_ASecondEditSupersedesTheQueuedConfiguration(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + + first := getModelDeployment(t, f.cli) + first.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), first)) + + f.pass(false) + slot, held := f.slot("server") + require.True(t, held) + f.admit() + require.Empty(t, f.live("server", slot.Ordinal), "the captured member has finished leaving") + + // A second edit lands while the first replacement is still queued. + second := getModelDeployment(t, f.cli) + second.Spec.Roles[0].Image = "vllm/vllm-openai:v0.27.0" + require.NoError(t, f.cli.Update(context.Background(), second)) + + sibling := 1 - slot.Ordinal + var siblingUID types.UID + if held := f.live("server", sibling); len(held) == 1 { + siblingUID = held[0].UID + } + + for pass := 0; pass < 4; pass++ { + f.pass(pass%2 == 0) + } + + _, stillHeld := f.slot("server") + assert.True(t, stillHeld, "the role is still committed to the same ordinal") + assert.Equal(t, "vllm/vllm-openai:v0.27.0", f.images("server", slot.Ordinal)[0], + "and the obsolete queued configuration was replaced rather than waited for") + + for pass := 0; pass < 8; pass++ { + if allAtImage(f, "vllm/vllm-openai:v0.27.0", 2) { + break + } + f.pass(pass%2 == 0) + f.admit() + } + + for _, ordinal := range []int{slot.Ordinal, sibling} { + members := f.live("server", ordinal) + require.Len(t, members, 1, "ordinal %d keeps serving throughout", ordinal) + assert.Equal(t, "vllm/vllm-openai:v0.27.0", members[0].Spec.Containers[0].Image, + "ordinal %d ends on the newest spec, never on the obsolete one", ordinal) + } + if siblingUID != types.UID("") { + assert.NotEqual(t, siblingUID, f.live("server", sibling)[0].UID, + "while the obsolete sibling was turned over in its own turn rather than kept") + } +} + +// TestLifecycle_ACleanupIsBoundedToWhatTheSlotCaptured covers UID authority: a member that took +// the name after the capture is not the one the slot may delete. +func TestLifecycle_ACleanupIsBoundedToWhatTheSlotCaptured(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + + // The members the slot may capture are read before the pass that removes one, because after + // that pass its name is free and nothing else names the identity the slot may delete. + before := map[int]*core.Pod{} + for ordinal := range 2 { + members := f.live("server", ordinal) + require.Len(t, members, 1) + before[ordinal] = members[0] + } + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot, held := f.slot("server") + require.True(t, held) + captured := before[slot.Ordinal] + assert.Equal(t, []types.UID{captured.UID}, slot.MemberUIDs, + "and the slot names the identity rather than the name") + + // The captured member finishes leaving, which frees its name the way a same-named recreate is + // only possible after. + f.admit() + require.Empty(t, f.live("server", slot.Ordinal)) + + // A different Pod takes the departing member's name and seat, as a same-named recreate would. + stranger := captured.DeepCopy() + stranger.GenerateName = "" + stranger.UID = types.UID("pod-someone-else") + stranger.ResourceVersion = "" + require.NoError(t, f.cli.Create(context.Background(), stranger)) + + f.pass(false) + f.admit() + f.pass(false) + + survivor := new(core.Pod) + require.NoError(t, f.cli.Get(context.Background(), + ctrlcli.ObjectKey{Namespace: "team-a", Name: captured.Name}, survivor)) + assert.Equal(t, types.UID("pod-someone-else"), survivor.UID, + "the cleanup deleted the identity the slot captured, not the name") +} + +// TestLifecycle_AForeignWorkloadOnTheGroupNameIsNeverDeleted covers the other half of that +// authority: the vacancy proof waits on an object this operator cannot prove it composed, and does +// not remove it to make room. +func TestLifecycle_AForeignWorkloadOnTheGroupNameIsNeverDeleted(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot, held := f.slot("server") + require.True(t, held, "the highest ordinal turns over first") + + // Something else holds the name the replacement's own group is composed under. It owns none of + // the replica's members, so nothing about it is this operator's to delete or to adopt. The + // slot's own Workload is given up first, because a name still held is not one a stranger could + // take -- and because a foreign Workload only arrives where this operator's has gone. + f.admit() + group := modelDeploymentReplicaGroupName(f.md, "server", slot.Ordinal) + require.Nil(t, f.workloadNamed(group), "the group name is free before the stranger takes it") + foreign := &kueue.Workload{} + foreign.Name, foreign.Namespace = group, "team-a" + foreign.UID = types.UID("wl-not-ours") + require.NoError(t, f.cli.Create(context.Background(), foreign)) + + for pass := range 4 { + f.pass(false) + assert.Equal(t, slot.Ordinal, slotOrdinal(t, f), + "pass %d: the vacancy proof waits rather than choosing another replica", pass) + } + + surviving := new(kueue.Workload) + require.NoError(t, f.cli.Get(context.Background(), + ctrlcli.ObjectKey{Namespace: "team-a", Name: group}, surviving)) + assert.Equal(t, types.UID("wl-not-ours"), surviving.UID, + "a Workload on the group's name that owns none of its members is not this operator's to delete") +} + +// slotOrdinal returns the ordinal a role's slot names, or -1 when it holds none. +func slotOrdinal(t *testing.T, f *lifecycleFixture) int { + t.Helper() + + slot, held := f.slot("server") + if !held { + return -1 + } + + return slot.Ordinal +} + +// TestLifecycle_ABrokenReplicaWaitsBehindTheSlotAndIsRebuiltWholeAfterwards covers a real member +// loss while another replica is being replaced. The broken replica is never topped up from the +// current render -- that would put a member of one configuration beside a member of another -- and +// it is not dismantled either while the role is committed elsewhere. Once the slot is admitted it +// is rebuilt whole, ahead of any healthy old replica. +func TestLifecycle_ABrokenReplicaWaitsBehindTheSlotAndIsRebuiltWholeAfterwards(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].Replicas = 3 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + require.Len(t, f.live("server", 0), 2, "size two renders two members") + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + slot, held := f.slot("server") + require.True(t, held) + require.Equal(t, 2, slot.Ordinal, "the highest ordinal turns over first") + + // While that slot waits, a DIFFERENT replica loses a member for real. + lost := f.live("server", 0) + require.NoError(t, f.cli.Delete(context.Background(), lost[1].DeepCopy())) + healthy := f.live("server", 1) + require.Len(t, healthy, 2) + + for pass := range 3 { + f.pass(pass%2 == 0) + assertOneConfiguration(t, f, 0, + fmt.Sprintf("pass %d: the broken replica is never mixed", pass)) + assert.Len(t, f.live("server", 0), 1, + "pass %d: and is not topped up from the render the spec states now", pass) + assert.Len(t, f.live("server", 1), 2, "pass %d: a healthy sibling is untouched", pass) + assert.Equal(t, healthy[0].UID, f.live("server", 1)[0].UID, + "pass %d: down to the member", pass) + assert.Equal(t, slot.Ordinal, f.slotMust("server").Ordinal, + "pass %d: the slot still names its own ordinal", pass) + } + + // The slot's replacement is admitted, and the broken replica is what the role turns over next: + // the pass that releases the role spends it on the ordinal that is down, not on the one that is + // still serving. + f.admit() + f.pass(false) + + stillServing := f.live("server", 1) + require.Len(t, stillServing, 2, "the healthy old replica keeps serving") + assert.Equal(t, healthy[0].UID, stillServing[0].UID, "down to the member") + assert.Equal(t, "vllm/vllm-openai:v0.25.1", stillServing[0].Spec.Containers[0].Image, + "and it was not taken ahead of the broken one") + + for pass := range 10 { + if allAtImage(f, "vllm/vllm-openai:v0.26.0", 6) { + break + } + f.pass(pass%2 == 0) + f.admit() + assertOneConfiguration(t, f, 0, + fmt.Sprintf("settling pass %d: the broken ordinal is never mixed either", pass)) + } + + for ordinal := range 3 { + members := f.live("server", ordinal) + require.Len(t, members, 2, "ordinal %d is whole again", ordinal) + assert.Equal(t, "vllm/vllm-openai:v0.26.0", members[0].Spec.Containers[0].Image, + "ordinal %d carries the edit the replacement was built from", ordinal) + } +} + +// assertOneConfiguration states that the live members of an ordinal all run the same image, which is +// the property a top-up from a newer render would break. An empty ordinal satisfies it: a replica +// with nothing serving has mixed nothing. +func assertOneConfiguration(t *testing.T, f *lifecycleFixture, ordinal int, message string) { + t.Helper() + + seen := sets.New[string]() + for _, member := range f.live("server", ordinal) { + seen.Insert(member.Spec.Containers[0].Image) + } + assert.LessOrEqual(t, seen.Len(), 1, message) +} + +// TestLifecycle_APartialCreateIsFilledOnlyInTheReplacingGroup covers a create that lands for one +// member of a replica and not the other. The group is then short of its total, which is a group +// Kueue composes no Workload for, so the slot would wait on it forever unless the missing member is +// filled -- and it is filled only because that group is the one the replacement is being built in. +func TestLifecycle_APartialCreateIsFilledOnlyInTheReplacingGroup(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + }) + + // The second member's create fails once, and then stops failing. The replica renders its + // members for one group under one name each, so the failure lands on the replica being + // replaced rather than on a member chosen by the case. + failing := false + f := newLifecycleFixtureAround(t, md, func(objs ...ctrlcli.Object) ctrlcli.WithWatch { + return ctrlfake.NewClientBuilder(). + WithScheme(scheme.Scheme). + WithIndex(&core.Pod{}, kueuepod.PodGroupNameCacheKey, func(obj ctrlcli.Object) []string { + pod, ok := obj.(*core.Pod) + if !ok { + return nil + } + group := pod.Labels[kueuepodconst.GroupNameLabel] + if group == "" { + return nil + } + + return []string{group} + }). + WithIndex(&core.Pod{}, "spec.nodeName", func(obj ctrlcli.Object) []string { + pod, ok := obj.(*core.Pod) + if !ok || pod.Spec.NodeName == "" { + return nil + } + + return []string{pod.Spec.NodeName} + }). + WithStatusSubresource(&workercore.ModelDeployment{}, &workercore.KVCachePoolBinding{}, + &workercore.Devices{}, &kueue.Workload{}). + WithInterceptorFuncs(ctrlinterceptor.Funcs{ + Create: func( + c context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.CreateOption, + ) error { + pod, ok := obj.(*core.Pod) + if !ok || !failing || + pod.Labels[modelDeploymentMemberIndexLabel] != "1" { + return next.Create(c, obj, opts...) + } + + return errors.New("the create was refused") + }, + }). + WithObjects(objs...). + Build() + }) + + f.pass(false) + f.admit() + for ordinal := range 2 { + require.Len(t, f.live("server", ordinal), 2, "the role is whole before the edit") + } + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + slot, held := f.slot("server") + require.True(t, held) + f.admit() + require.Empty(t, f.live("server", slot.Ordinal), "the captured member has finished leaving") + + // The group comes back one member short of its total. + failing = true + f.passExpectingFailure(false) + assert.Len(t, f.live("server", slot.Ordinal), 1, + "the create that landed left the replica one member short") + + // And the next pass fills exactly that one member, leaving the one already there alone. + failing = false + f.pass(false) + members := f.live("server", slot.Ordinal) + require.Len(t, members, 2, "the missing member is filled in the group the replacement is built in") + assertOneConfiguration(t, f, slot.Ordinal, "and the filled group is one configuration") + assert.Len(t, f.live("server", 1-slot.Ordinal), 2, + "while the sibling replica the slot is not about keeps both of its members") + + f.admit() + f.pass(false) + assert.Empty(t, f.answering()[slot.Ordinal], "an unadmitted member does not take an endpoint") + for _, member := range members { + assert.Equal(t, "vllm/vllm-openai:v0.26.0", member.Spec.Containers[0].Image, + "and both members carry the configuration the replacement was built from") + } +} + +// TestLifecycle_TwoIncarnationsOfOneReplicaGetDistinctIdentities pins the premise every comparison +// above rests on. A replaced replica and the one that took its seat share a name and a group, so +// only the identities the API server assigns tell them apart -- and a fixture that gave one pair of +// identities to two objects would pass a cleanup that deleted the wrong one. +func TestLifecycle_TwoIncarnationsOfOneReplicaGetDistinctIdentities(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 1 + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + before := f.live("server", 0) + require.Len(t, before, 2) + beforeWorkload := f.workloadNamed(modelDeploymentReplicaGroupName(f.md, "server", 0)) + require.NotNil(t, beforeWorkload) + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + + for range 6 { + f.pass(false) + f.admit() + if allAtImage(f, "vllm/vllm-openai:v0.26.0", 2) { + break + } + } + after := f.live("server", 0) + require.Len(t, after, 2) + afterWorkload := f.workloadNamed(modelDeploymentReplicaGroupName(f.md, "server", 0)) + require.NotNil(t, afterWorkload) + + beforeUIDs := sets.New[types.UID]() + for _, member := range before { + beforeUIDs.Insert(member.UID) + } + for _, member := range after { + assert.False(t, beforeUIDs.Has(member.UID), + "%s is a different object from the member that held this seat", member.Name) + } + assert.NotEqual(t, beforeWorkload.UID, afterWorkload.UID, + "and so is the Workload the group composed for each of them") + assert.False(t, modelDeploymentWorkloadOwnsAny(afterWorkload, beforeUIDs), + "the Workload composed for the replacement owns none of the members it replaced") +} + +// allAtImage reports whether the role holds its declared replica count and every member carries an +// image. An empty set is not convergence: a role with nothing serving has not settled, it has +// stopped. +func allAtImage(f *lifecycleFixture, image string, replicas int) bool { + live := 0 + for _, pod := range replicaPods(f.t, f.cli) { + if modelDeploymentPodRole(&pod) != "server" || pod.DeletionTimestamp != nil { + continue + } + if pod.Spec.Containers[0].Image != image { + return false + } + live++ + } + + return live == replicas +} + +// slotMust returns a role's slot, failing the test when there is none. +func (f *lifecycleFixture) slotMust(role string) modelDeploymentReplacementSlot { + f.t.Helper() + + slot, held := f.slot(role) + require.True(f.t, held, "expected an active replacement slot for role %q", role) + + return slot +} + +// setReady puts the live members of one ordinal in or out of the Ready condition, which is the +// unconditional half of every routing decision. +func (f *lifecycleFixture) setReady(ordinal int, ready bool) { + f.t.Helper() + + for _, member := range f.live("server", ordinal) { + ready := ready + stored := new(core.Pod) + require.NoError(f.t, f.cli.Get(context.Background(), + ctrlcli.ObjectKeyFromObject(member), stored)) + if ready { + stored.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} + } else { + stored.Status.Conditions = nil + } + require.NoError(f.t, f.cli.Status().Update(context.Background(), stored)) + } +} + +// setEligible puts the engine-eligibility key on the live members of one ordinal, or takes it off. +// A key that is absent is a member this operator has not found qualified, which is the state the +// negative controls need. +func (f *lifecycleFixture) setEligible(ordinal int, present bool) { + f.t.Helper() + + for _, member := range f.live("server", ordinal) { + patched := new(core.Pod) + require.NoError(f.t, f.cli.Get(context.Background(), + ctrlcli.ObjectKeyFromObject(member), patched)) + modelDeploymentApplyLabel(patched, modelDeploymentLabelKeyEndpointEligible, present) + require.NoError(f.t, f.cli.Patch(context.Background(), patched, ctrlcli.MergeFrom(member))) + } +} + +// answering reports the routing label of every live member, keyed by the replica it sits in. A +// member without the key reports the empty string, which is how the Service selector reads it. +func (f *lifecycleFixture) answering() map[int]string { + f.t.Helper() + + seen := map[int]string{} + for _, member := range f.live("server", 0) { + ordinal, ok := modelDeploymentPodOrdinal(member) + if ok { + seen[ordinal] = member.Labels[modelDeploymentLabelKeyAPIAnswering] + } + } + for _, member := range f.live("server", 1) { + ordinal, ok := modelDeploymentPodOrdinal(member) + if ok { + seen[ordinal] = member.Labels[modelDeploymentLabelKeyAPIAnswering] + } + } + + return seen +} + +// frontService is the Service clients address the deployment through. +func (f *lifecycleFixture) frontService() core.Service { + f.t.Helper() + + svc := new(core.Service) + require.NoError(f.t, f.cli.Get(context.Background(), + ctrlcli.ObjectKey{Namespace: "team-a", Name: f.md.Name}, svc)) + + return *svc +} + +// routesByMembership reports whether the front Service selects on the routing label at all. A +// homogeneous role does not, and keeps its ordinary selector. +func (f *lifecycleFixture) routesByMembership() bool { + f.t.Helper() + + _, present := f.frontService().Spec.Selector[modelDeploymentLabelKeyAPIAnswering] + + return present +} + +// TestLifecycle_ReplicasGrowingAndShrinkingDuringAReplacement covers both directions of a count +// change while a slot is open: an ordinal the count no longer declares cancels the slot it holds +// and is not created again, and an ordinal that re-enters the count is created from the spec the +// role states now rather than from the configuration it left. +func TestLifecycle_ReplicasGrowingAndShrinkingDuringAReplacement(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 3 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot, held := f.slot("server") + require.True(t, held, "the highest ordinal turns over first") + require.Equal(t, 2, slot.Ordinal, "which is the one the smaller count no longer declares") + + // The count shrinks past the ordinal the slot owns. The slot is canceled rather than carried + // on an ordinal nothing declares, the ordinal is not created again, and the two the count still + // declares are then rebuilt from the new spec in their own turns. + scaled := getModelDeployment(t, f.cli) + scaled.Spec.Roles[0].Replicas = 2 + require.NoError(t, f.cli.Update(context.Background(), scaled)) + + for pass := range 4 { + f.pass(pass%2 == 0) + assert.Empty(t, f.live("server", 2), + "pass %d: an ordinal the spec no longer declares is not recreated", pass) + } + + for pass := range 10 { + if allAtImage(f, "vllm/vllm-openai:v0.26.0", 2) { + break + } + f.pass(pass%2 == 0) + f.admit() + assert.Empty(t, f.live("server", 2), + fmt.Sprintf("pass %d: nor once the role is free again", pass)) + } + require.True(t, allAtImage(f, "vllm/vllm-openai:v0.26.0", 2), "the role settles at two replicas") + + // The count grows back. The ordinal that returns is created from the spec the role states now. + grown := getModelDeployment(t, f.cli) + grown.Spec.Roles[0].Replicas = 3 + require.NoError(t, f.cli.Update(context.Background(), grown)) + + for pass := range 6 { + f.pass(pass%2 == 0) + if len(f.live("server", 2)) > 0 { + break + } + } + members := f.live("server", 2) + require.Len(t, members, 1, "the returning ordinal is created") + assert.Equal(t, "vllm/vllm-openai:v0.26.0", members[0].Spec.Containers[0].Image, + "from the spec the role states now, not the one it left") +} + +// TestLifecycle_AFourFieldEditIsOneReplacementNotFour covers the contract the whole task exists for: +// changing size, resources, instance type and command together still turns over one replica at a +// time, and each replica comes back carrying all four. +func TestLifecycle_AFourFieldEditIsOneReplacementNotFour(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md, newRenderInstanceType(func(it *worker.InstanceType) { + it.Name = "h20-2x" + it.Spec.UnitResources = workercore.InstanceTypeUnitResources{CPU: "4", RAM: "16Gi"} + it.Status.Entrance = "queue-for-h20-2x" + })) + + f.pass(false) + f.admit() + + // The four fields the role makes mutable, each observable in the rendered Pod: two members per + // replica, host resources sized for two cards, the narrower pool's queue, and the command line + // the role supplies in full. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].ReplicaSize = 2 + edited.Spec.Roles[0].InstanceType = "h20-2x" + edited.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: ptr.To(resource.MustParse("2")), + } + edited.Spec.Roles[0].Command = []string{"python", "-m", "vllm.entrypoints.openai.api_server"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + + // One pass turns over exactly one replica, whatever the edit touched. + f.pass(false) + _, held := f.slot("server") + require.True(t, held) + assert.Len(t, f.live("server", 0), 1, "the sibling is untouched by a four-field edit") + assert.Empty(t, f.live("server", 1), "and one replica is on its way out") + + for range 6 { + f.pass(false) + f.admit() + } + + f.pass(false) + for _, ordinal := range []int{0, 1} { + members := f.live("server", ordinal) + require.Len(t, members, 2, "ordinal %d is rebuilt at the new member count", ordinal) + for _, member := range members { + assert.Equal(t, "vllm/vllm-openai:v0.25.1", member.Spec.Containers[0].Image, + "a field the edit did not name is unchanged") + assert.Equal(t, []string{"python", "-m", "vllm.entrypoints.openai.api_server"}, + member.Spec.Containers[0].Command, "the command is the one the role supplied") + assert.Equal(t, "queue-for-h20-2x", member.Labels[kueuectrlconst.QueueLabel], + "and the group is submitted to the pool the role named") + // Four cards' worth of unit resources are one: the narrower pool's four CPU by the two + // cards the role asked for, where the unedited render asked for one card of the wider. + assert.Equal(t, int64(8), member.Spec.Containers[0].Resources.Limits.Cpu().Value(), + "the host resources follow the pool the edit named and the cards it asked for") + } + } +} + +// TestLifecycle_SiblingRolesAreNobodyElsesCost covers a two-role deployment: the role being edited +// spends its own replicas and leaves the other's alone. +func TestLifecycle_SiblingRolesAreNobodyElsesCost(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + sibling := md.Spec.Roles[0] + sibling.Name = "decode" + sibling.Replicas = 1 + md.Spec.Roles = append(md.Spec.Roles, sibling) + + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + + before := f.live("decode", 0) + require.Len(t, before, 1) + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + + for pass := 0; pass < 3; pass++ { + f.pass(false) + f.admit() + } + + after := f.live("decode", 0) + require.Len(t, after, 1) + assert.Equal(t, before[0].UID, after[0].UID, + "the sibling role is not a replacement candidate and keeps its member") + assert.Equal(t, "vllm/vllm-openai:v0.25.1", after[0].Spec.Containers[0].Image, + "built from the image the sibling role's edit never changed") +} + +// TestLifecycle_APartialCreateWhoseResponseIsLostIsNotRepeated covers the create gate against the +// server rather than the cache: a create the server accepted and never answered is found, not +// repeated. +func TestLifecycle_APartialCreateWhoseResponseIsLostIsNotRepeated(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + server := f.cli + hidden := true + lost := 0 + attempts := 0 + view := ctrlinterceptor.NewClient(server, ctrlinterceptor.Funcs{ + List: func(ctx context.Context, next ctrlcli.WithWatch, list ctrlcli.ObjectList, + opts ...ctrlcli.ListOption, + ) error { + if err := next.List(ctx, list, opts...); err != nil { + return err + } + if pods, ok := list.(*core.PodList); ok && hidden { + pods.Items = nil + } + return nil + }, + Create: func(ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.CreateOption, + ) error { + if _, ok := obj.(*core.Pod); ok { + attempts++ + } + if err := next.Create(ctx, obj, opts...); err != nil { + return err + } + if _, ok := obj.(*core.Pod); ok { + lost++ + return errors.New("connection lost: the create's response never arrived") + } + return nil + }, + }) + r := &ModelDeploymentReconciler{Client: view, APIReader: server, Recorder: ctrlrecord.NewFakeRecorder(64)} + _, err := reconcileModelDeploymentWith(t, r) + require.ErrorContains(t, err, "connection lost") + require.Equal(t, 4, lost, "all members were accepted by the authoritative server") + before := replicaPods(t, server) + require.Len(t, before, 4) + uids := sets.New[types.UID]() + for _, pod := range before { + require.NotEmpty(t, pod.UID, "identity exists before any lifecycle read") + uids.Insert(pod.UID) + } + require.Len(t, uids, 4, "every create has a distinct identity") + require.Empty(t, replicaPods(t, view), "the cache remains behind on retry") + + _, err = reconcileModelDeploymentWith(t, r) + require.NoError(t, err, "the uncached gate finds successful writes despite lost responses") + assert.Equal(t, 4, lost, "no successful create was repeated") + assert.Equal(t, 4, attempts, "no duplicate create was attempted") + after := replicaPods(t, server) + require.Len(t, after, 4, "the authoritative member set did not grow") + for _, pod := range after { + assert.True(t, uids.Has(pod.UID)) + } + + // Composition uses the authoritative complete groups and Kueue's pinned constructor. + f.admit() + workloadUIDs := sets.New[types.UID]() + for ordinal := range 2 { + wl := f.workloadNamed(modelDeploymentReplicaGroupName(md, "server", ordinal)) + require.NotNil(t, wl) + require.NotEmpty(t, wl.UID) + workloadUIDs.Insert(wl.UID) + for _, pod := range f.live("server", ordinal) { + assert.True(t, modelDeploymentWorkloadOwnsAny(wl, sets.New(pod.UID))) + } + } + require.Len(t, workloadUIDs, 2) + assert.Empty(t, uids.Intersection(workloadUIDs), "Pod and Workload identities are distinct") + hidden = false + _, err = reconcileModelDeploymentWith(t, r) + require.NoError(t, err, "the cache eventually catches up") + assert.Equal(t, 4, lost) + assert.True(t, allAtImage(f, "vllm/vllm-openai:v0.25.1", 4)) + for ordinal := range 2 { + members := f.live("server", ordinal) + require.Len(t, members, 2) + for _, pod := range members { + assert.True(t, uids.Has(pod.UID)) + } + assert.True(t, workloadUIDs.Has(f.workloadNamed(modelDeploymentReplicaGroupName(md, "server", ordinal)).UID)) + } +} + +// TestLifecycle_SizingDownDoesNotTrimTheHealthyGroup covers the arithmetic the size edit depends +// on: a group built at four members is not excess while the spec asks for two. +func TestLifecycle_SizingDownDoesNotTrimTheHealthyGroup(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].ReplicaSize = 4 + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + + f.pass(false) + f.admit() + require.Len(t, f.live("server", 0), 4, "the group is built at the size the spec asked for") + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].ReplicaSize = 2 + require.NoError(t, f.cli.Update(context.Background(), edited)) + + f.pass(false) + slot, held := f.slot("server") + require.True(t, held) + sibling := 1 - slot.Ordinal + + // Three passes with nothing admitted behind the first replacement: the group that is not being + // replaced must still be whole, because a desired-size difference is not surplus. + for pass := range 3 { + f.pass(false) + assert.Len(t, f.live("server", sibling), 4, + "pass %d: the healthy group keeps all four members while one replica is replaced", pass) + } + + for range 8 { + if len(f.live("server", slot.Ordinal)) == 2 && len(f.live("server", sibling)) == 2 { + break + } + f.pass(false) + f.admit() + } + assert.Len(t, f.live("server", slot.Ordinal), 2, "the replaced replica comes back at the new size") + assert.Len(t, f.live("server", sibling), 2, "and so does the one behind it, in its own turn") +} diff --git a/pkg/worker/controllers/worker/model_deployment_pod_group.go b/pkg/worker/controllers/worker/model_deployment_pod_group.go index 7ebb946dd..411ddc9ac 100644 --- a/pkg/worker/controllers/worker/model_deployment_pod_group.go +++ b/pkg/worker/controllers/worker/model_deployment_pod_group.go @@ -3,6 +3,7 @@ package worker import ( "context" "fmt" + "slices" core "k8s.io/api/core/v1" kerrors "k8s.io/apimachinery/pkg/api/errors" @@ -89,6 +90,24 @@ const ( // selector is a conjunction of equalities, so the term is present-or-absent; the value exists // because a selector needs one, and "true" states what presence means. modelDeploymentEndpointEligibleValue = "true" + + // modelDeploymentLabelKeyAPIAnswering marks a Pod whose replicas answer the deployment's HTTP + // API under the routing rules of the mode they were created with. + // + // IT IS ROUTING MEMBERSHIP, NOT ENGINE QUALIFICATION. The eligibility key above says this + // member's endpoints serve; this one says which Service this member is behind. They differ for + // exactly one reason: a role mid-edit holds replicas built under different command modes, and + // each mode keeps the routing it already had. A take-over replica answers under the legacy + // rules whether or not this operator can verify its engine, and a managed replica answers under + // the qualification rules, which the eligibility key already carries. + // + // It is written by the reconciler at runtime and never rendered, so it stays out of the + // spec-hash: turning a member's routing on or off must not roll the replica that serves. + modelDeploymentLabelKeyAPIAnswering = "modeldeployment." + systemname.LabelPrefix + "api-answering" + + // modelDeploymentAPIAnsweringValue is the only value the answering key carries, for the same + // reason the eligibility key has one. + modelDeploymentAPIAnsweringValue = "true" ) // ModelDeploymentPodGroupMeta is the Kueue group metadata one replica's Pod carries. @@ -214,6 +233,131 @@ func modelDeploymentAnsweringShapeOf(externalDP bool, size int) modelDeploymentA return modelDeploymentAnsweringLeader } +// modelDeploymentReplicaRunsCommand reports whether every member of a replica was built with a +// command line this operator wrote. +// +// A replica whose members disagree, or which carries no engine container to read, is left out: the +// mode it runs in is then not established, and the reader that asks must not be told it is managed. +func modelDeploymentReplicaRunsCommand(members []*core.Pod) bool { + if len(members) == 0 { + return false + } + for _, member := range members { + managed, ok := modelDeploymentMemberExecution(member) + if !ok || !managed { + return false + } + } + + return true +} + +// modelDeploymentAnyMemberRunsCommand reports whether any member standing for a role is one this +// operator built. A role mid-transition holds replicas of both modes at once, and the gate belongs +// to the managed ones still serving, so one is enough. +func modelDeploymentAnyMemberRunsCommand(members []*core.Pod) bool { + for _, member := range members { + if managed, ok := modelDeploymentMemberExecution(member); ok && managed { + return true + } + } + + return false +} + +// modelDeploymentDeployedAnsweringShape derives the answering shape from what a replica's members +// are RUNNING, rather than from the role the spec states now. +// +// IT MATTERS WHILE A REPLICA IS BEING REPLACED. A role edited from four members to one, or from an +// External data-parallel arrangement to a leader-served one, leaves its old replicas serving under +// the shape they were built at. Reading the desired role instead would withdraw the followers of a +// four-member replica the moment the spec asked for one, or expose every member of a rank group the +// moment the spec asked for a leader -- on replicas that are serving perfectly well and that nothing +// has replaced yet. +// +// The total is the one the members themselves declare, and the parallelism is read off their own +// command lines. Both are facts about the replica in front of us rather than about a role that may +// have moved on. +func modelDeploymentDeployedAnsweringShape( + view modelDeploymentReplicaView, engine string, +) modelDeploymentAnsweringShape { + total, reason := modelDeploymentReplicaDeclaredTotal(view.Members) + if reason != "" || total <= 0 { + // A replica whose members disagree on what they are has no shape to read. Holding it is the + // honest answer, and holding keeps whatever membership it already carries. + return modelDeploymentAnsweringUnknown + } + + externalDP, known := modelDeploymentMembersExternalDP(view.Members, engine) + if !known { + // A failed read is not a reading of internal data parallelism. Holding keeps the membership + // the replica already carries instead of withdrawing followers on a guess. + return modelDeploymentAnsweringUnknown + } + + return modelDeploymentAnsweringShapeOf(externalDP, total) +} + +// modelDeploymentMembersExternalDP reports whether the members of a replica carry a data-parallel +// load balance, read from the command lines they are running. +// +// IT READS THE MEMBERS AND NOT THE ROLE for the same reason the shape does: the role states what the +// next replica will be built as, and these members are running now. +// +// known is false when a managed member was consulted and none of them could be read, so the answer +// is not an established "leader-served". Members this operator did not build carry no managed load +// balance, which is an established answer. +func modelDeploymentMembersExternalDP(members []*core.Pod, engine string) (externalDP, known bool) { + consulted := false + for _, member := range members { + managed, readable := modelDeploymentMemberExecution(member) + if !readable || !managed { + continue + } + consulted = true + container, found := modelDeploymentDrainContainerOf(member, modelDeploymentMainContainerName) + if !found { + continue + } + // THE ARGV IS THE COMMAND AND THE ARGS TOGETHER, because that is what the engine is run + // with. A managed replica's parallelism is declared as extra arguments and rendered onto the + // command line, so reading the arguments alone would report every replica as leader-served + // and expose the followers of an External-DP group. + argv := slices.Concat(container.Command, container.Args) + reading, err := scanModelDeploymentParallelism(engine, argv, + modelDeploymentDeclaredEnv(container.Env)) + if err != nil { + continue + } + shape, _ := modelDeploymentLoadBalance(reading.declared) + + return shape != workercore.ModelDeploymentLoadBalanceInternal && + shape != workercore.ModelDeploymentLoadBalanceUnknown, true + } + + return false, !consulted +} + +// modelDeploymentDeclaredEnv reads a container's environment back into the declared form the +// parallelism scan takes. +// +// ONLY VARIABLES WHOSE VALUE IS WRITTEN LITERALLY ARE READ. A value Kubernetes fills in at run time +// is not a declaration this operator can act on, and reading its name as if it were would let any +// Pod carrying that name change how its replica is understood. +func modelDeploymentDeclaredEnv(env []core.EnvVar) []workercore.ModelDeploymentEnvVar { + declared := make([]workercore.ModelDeploymentEnvVar, 0, len(env)) + for _, variable := range env { + if variable.ValueFrom != nil { + continue + } + declared = append(declared, workercore.ModelDeploymentEnvVar{ + Name: variable.Name, Value: variable.Value, + }) + } + + return declared +} + // modelDeploymentAnsweringMemberLeader reports whether a Pod is the member a leader-served replica's // answer comes from. // @@ -319,9 +463,9 @@ func ModelDeploymentPodGroup( // THE TOTAL IS THE REPLICA'S SIZE, AND A REPLICA COUNT CHANGE NEVER MOVES IT: the group // is this replica and nobody else's, so adding or removing replicas adds or removes // whole groups rather than editing the total any running member carries. That is what - // turns a resize into a trim rather than a rebuild, and it survives sizes above one - // only because the size itself is frozen at creation -- a mutable size would put this - // number back under two writers, which is the defect the per-replica split removed. + // turns a resize into a rebuild of one replica at a time, and it holds for sizes above one + // because a member created at the old size keeps its old total until its replica is + // recreated -- the two totals coexist, and nothing writes the running one. kueuepodconst.GroupTotalCountAnnotation: strconvx.Itoa(modelDeploymentRoleSize(role)), // THE ROLE HASH IS LOAD-BEARING, NOT COSMETIC. Kueue reads this annotation verbatim when // present and otherwise derives a digest of the Pod spec's SHAPE -- containers, @@ -450,21 +594,6 @@ func modelDeploymentGroupPods(pods []core.Pod, skipDeparting bool) []modelDeploy return views } -// modelDeploymentReplicaIsComplete reports whether a replica holds every member it declares. -// -// AN INCOMPLETE REPLICA IS NOT A PARTLY WORKING ONE. Kueue composes no Workload at all for a pod -// group short of its declared total, so such a replica holds no quota, is admitted by nothing and -// serves nothing -- it is not "most of the way there", it is absent with some Pods lying around. -// Every figure that counts replicas therefore excludes it rather than discounting it. -func modelDeploymentReplicaIsComplete(view modelDeploymentReplicaView, size int) bool { - if !view.Seated { - // A Pod that claims no ordinal declares one member, itself. - return len(view.Members) == 1 - } - - return len(view.Members) == size -} - // modelDeploymentPodDescription names a Pod the way its own shape makes true: a replica when a // replica is one Pod, and a member of one when it is not. // diff --git a/pkg/worker/controllers/worker/model_deployment_render_test.go b/pkg/worker/controllers/worker/model_deployment_render_test.go index a7b1fa899..02db8ca2f 100644 --- a/pkg/worker/controllers/worker/model_deployment_render_test.go +++ b/pkg/worker/controllers/worker/model_deployment_render_test.go @@ -2170,7 +2170,7 @@ func TestRenderModelDeploymentPod_Probes(t *testing.T) { // THE GATE AND THE SERVICE MUST NAME ONE PORT, and the comparison is against the // rendered Service rather than against the helper the render used -- comparing a value // with the function that produced it would hold however either side drifted. - svc := renderModelDeploymentService(md) + svc := renderModelDeploymentService(md, nil) require.Len(t, svc.Spec.Ports, 1) assert.Equal(t, svc.Spec.Ports[0].TargetPort.IntVal, c.ReadinessProbe.HTTPGet.Port.IntVal, "the port the gate grades and the port the Service sends traffic to are one fact") diff --git a/pkg/worker/controllers/worker/model_deployment_replacement_read_error_test.go b/pkg/worker/controllers/worker/model_deployment_replacement_read_error_test.go new file mode 100644 index 000000000..35d2ce8b4 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_replacement_read_error_test.go @@ -0,0 +1,84 @@ +package worker + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + + "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" +) + +// A failed read of the server is an error to the caller, not a wait reason. A wait reason ends the +// pass without a requeue, and a read that failed changes nothing in the cluster, so no event would +// wake the slot again. +func TestReplacementReads_ServerReadFailureIsAnErrorNotAWaitReason(t *testing.T) { + errRead := errors.New("api server unavailable") + + testCases := []struct { + name string + fails func(list ctrlcli.ObjectList) bool + }{ + {name: "listing the members fails", fails: func(list ctrlcli.ObjectList) bool { + _, ok := list.(*core.PodList) + + return ok + }}, + {name: "listing the workloads fails", fails: func(list ctrlcli.ObjectList) bool { + _, ok := list.(*kueue.WorkloadList) + + return ok + }}, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + reader := ctrlfake.NewClientBuilder(). + WithScheme(scheme.Scheme). + WithInterceptorFuncs(ctrlinterceptor.Funcs{ + List: func( + ctx context.Context, c ctrlcli.WithWatch, list ctrlcli.ObjectList, + opts ...ctrlcli.ListOption, + ) error { + if tc.fails(list) { + return errRead + } + + return c.List(ctx, list, opts...) + }, + }). + Build() + r := &ModelDeploymentReconciler{Client: reader, APIReader: reader} + md := newRenderDeployment() + slot := modelDeploymentReplacementSlot{Role: "server", Ordinal: 0} + + vacant, _, err := r.replacementVacant(context.Background(), md, slot) + assert.False(t, vacant) + require.ErrorIs(t, err, errRead) + }) + } + + t.Run("admission read of the members fails", func(t *testing.T) { + reader := ctrlfake.NewClientBuilder(). + WithScheme(scheme.Scheme). + WithInterceptorFuncs(ctrlinterceptor.Funcs{ + List: func(context.Context, ctrlcli.WithWatch, ctrlcli.ObjectList, ...ctrlcli.ListOption) error { + return errRead + }, + }). + Build() + r := &ModelDeploymentReconciler{Client: reader, APIReader: reader} + + admitted, _, err := r.replacementAdmitted(context.Background(), newRenderDeployment(), + modelDeploymentReplacementSlot{Role: "server", Ordinal: 0}, nil) + assert.False(t, admitted) + require.ErrorIs(t, err, errRead) + }) +} diff --git a/pkg/worker/controllers/worker/model_deployment_replacement_slot.go b/pkg/worker/controllers/worker/model_deployment_replacement_slot.go new file mode 100644 index 000000000..bcafbd767 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_replacement_slot.go @@ -0,0 +1,1226 @@ +package worker + +import ( + "context" + "encoding/json" + "fmt" + "maps" + "slices" + "strings" + + core "k8s.io/api/core/v1" + kerrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" + resourcehelper "k8s.io/component-helpers/resource" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrllog "sigs.k8s.io/controller-runtime/pkg/log" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" + utilpod "sigs.k8s.io/kueue/pkg/util/pod" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubeapistatus" + "gpustack.ai/gpustack/pkg/kubemeta" + "gpustack.ai/gpustack/pkg/nodefeature" + "gpustack.ai/gpustack/pkg/utils/strconvx" +) + +// modelDeploymentReplacementSlotsAnnotation carries each role's active replacement slot. +// +// An annotation rather than a status field: it is bookkeeping, not observed state, and it covers +// the window between deleting a replica's members and creating its replacement, in which the +// ordinal is simply empty and no observation can name what is being replaced. +const modelDeploymentReplacementSlotsAnnotation = "modeldeployment.gpustack.ai/replacement-slots" + +// modelDeploymentReplacementPhase is how far one slot has been observed to go. Cleanup carries +// the identities the slot captured; WaitingForAdmission carries the render its replacement is +// being built from. +type modelDeploymentReplacementPhase string + +const ( + // modelDeploymentReplacementCleanup is a slot whose old members are still leaving. + modelDeploymentReplacementCleanup modelDeploymentReplacementPhase = "Cleanup" + + // modelDeploymentReplacementWaitingForAdmission holds the role until its replacement is + // admitted. + modelDeploymentReplacementWaitingForAdmission modelDeploymentReplacementPhase = "WaitingForAdmission" +) + +// modelDeploymentReplacementSlot is one role's active replacement. At most one exists per role: +// a role whose slot this pass did not clear turns over no second healthy old replica. +type modelDeploymentReplacementSlot struct { + // Role is the role whose replica is being replaced. + Role string `json:"role"` + + // Ordinal is the slot within that role being replaced. + Ordinal int `json:"ordinal"` + + // Phase is how far the slot has been observed to go. + Phase modelDeploymentReplacementPhase `json:"phase"` + + // MemberUIDs are the identities the slot may delete. The derived group name locates + // candidates; a captured UID is what licenses deleting one. + MemberUIDs []types.UID `json:"memberUIDs,omitempty"` + + // WorkloadUID is the departing group's Workload. Empty while none has been observed. + WorkloadUID types.UID `json:"workloadUID,omitempty"` + + // MemberHashes are the fingerprints the replacement is being built from, recorded once the + // old group has left. A later edit differing from them supersedes it in this same slot. + MemberHashes []string `json:"memberHashes,omitempty"` + + // CurrentMemberUIDs and CurrentWorkloadUID retain creation observations through member loss. + CurrentMemberUIDs []types.UID `json:"currentMemberUIDs,omitempty"` + CurrentWorkloadUID types.UID `json:"currentWorkloadUID,omitempty"` +} + +// modelDeploymentReplacementSlots is every role's slot, keyed by role name. +type modelDeploymentReplacementSlots map[string]modelDeploymentReplacementSlot + +// slotFor returns a role's slot and whether it has one. +func (s modelDeploymentReplacementSlots) slotFor(role string) (modelDeploymentReplacementSlot, bool) { + slot, held := s[role] + + return slot, held +} + +// held reports whether a role has an unresolved replacement. +func (s modelDeploymentReplacementSlots) held(role string) bool { + _, present := s.slotFor(role) + + return present +} + +// modelDeploymentReplacementSlotsOf returns the recorded slots. A missing or unreadable record is +// no slots, which is how a deployment that never held one behaves. +func modelDeploymentReplacementSlotsOf(md *workercore.ModelDeployment) modelDeploymentReplacementSlots { + raw, present := md.Annotations[modelDeploymentReplacementSlotsAnnotation] + if !present { + return nil + } + + slots := modelDeploymentReplacementSlots{} + if err := json.Unmarshal([]byte(raw), &slots); err != nil { + return nil + } + + return slots +} + +// setSlot returns the slots with one role's slot written, leaving every other role's alone. +func (s modelDeploymentReplacementSlots) setSlot( + slot modelDeploymentReplacementSlot, +) modelDeploymentReplacementSlots { + next := make(modelDeploymentReplacementSlots, len(s)+1) + for role, held := range s { + next[role] = held + } + next[slot.Role] = slot + + return next +} + +// clearRole returns the slots with one role's slot removed, or nil once none remain. +func (s modelDeploymentReplacementSlots) clearRole(role string) modelDeploymentReplacementSlots { + next := make(modelDeploymentReplacementSlots, len(s)) + for name, held := range s { + if name != role { + next[name] = held + } + } + if len(next) == 0 { + return nil + } + + return next +} + +// replacementDescription names one slot in words. +func replacementDescription(slot modelDeploymentReplacementSlot) string { + return fmt.Sprintf("role %s replica %d", slot.Role, slot.Ordinal) +} + +// replacementCapturesMember reports whether a Pod is one the slot captured. +func replacementCapturesMember(slot modelDeploymentReplacementSlot, uid types.UID) bool { + return slices.Contains(slot.MemberUIDs, uid) +} + +// replacementCapturesWorkload reports whether a Workload is the one the slot captured. A Workload +// the slot never saw carries no authority, whatever its name reads. +func replacementCapturesWorkload(slot modelDeploymentReplacementSlot, wl *kueue.Workload) bool { + return slot.WorkloadUID != "" && slot.WorkloadUID == wl.UID +} + +// loadReplacementSlots reads the current record from the API server rather than the cache, so +// another role's slot written by a concurrent pass survives this pass's write. +func (r *ModelDeploymentReconciler) loadReplacementSlots( + ctx context.Context, md *workercore.ModelDeployment, +) (*workercore.ModelDeployment, modelDeploymentReplacementSlots, error) { + current := new(workercore.ModelDeployment) + if err := r.APIReader.Get(ctx, ctrlcli.ObjectKeyFromObject(md), current); err != nil { + return nil, nil, fmt.Errorf("read the deployment's replacement slots: %w", err) + } + + // The record names a role of this deployment, so a delete and recreate between the read and + // the write would leave it describing replicas of an object that no longer exists. + if current.UID != md.UID { + return nil, nil, fmt.Errorf( + "the deployment was replaced while this pass ran: read UID %s, expected %s", + current.UID, md.UID) + } + + return current, modelDeploymentReplacementSlotsOf(current), nil +} + +// writeReplacementSlots persists the slots, or writes nothing when they already say what this +// pass reached. +// +// The base is the server's own record and the patch carries the resourceVersion it was read at, +// so two passes writing different roles cannot overwrite one another and a record that moved +// underneath is a Conflict rather than a silent loss. The caller's object carries the new record +// and version forward, so a second write in the same pass is checked against the server. +func (r *ModelDeploymentReconciler) writeReplacementSlots( + ctx context.Context, current *workercore.ModelDeployment, + slots modelDeploymentReplacementSlots, carried *workercore.ModelDeployment, +) error { + previous := modelDeploymentReplacementSlotsOf(current) + if kubemeta.DeepEqual(previous, slots) { + return nil + } + + base := current.DeepCopy() + candidate := base.DeepCopy() + if candidate.Annotations == nil { + candidate.Annotations = map[string]string{} + } + if len(slots) == 0 { + // The key is removed rather than emptied: a merge patch that merely omits it leaves + // whatever was there, and a finished record must not survive naming a replacement. + delete(candidate.Annotations, modelDeploymentReplacementSlotsAnnotation) + } else { + raw, err := json.Marshal(slots) + if err != nil { + return fmt.Errorf("encode the replacement slots: %w", err) + } + candidate.Annotations[modelDeploymentReplacementSlotsAnnotation] = string(raw) + } + encoded := candidate.Annotations[modelDeploymentReplacementSlotsAnnotation] + + optimistic := ctrlcli.MergeFromWithOptions(base, ctrlcli.MergeFromWithOptimisticLock{}) + if err := r.Client.Patch(ctx, candidate, optimistic); err != nil { + if kerrors.IsConflict(err) { + return fmt.Errorf("the replacement slots were written against a stale object: %w", err) + } + + return fmt.Errorf("write the replacement slots: %w", err) + } + + if len(slots) == 0 { + delete(carried.Annotations, modelDeploymentReplacementSlotsAnnotation) + } else { + if carried.Annotations == nil { + carried.Annotations = map[string]string{} + } + carried.Annotations[modelDeploymentReplacementSlotsAnnotation] = encoded + } + carried.ResourceVersion = candidate.ResourceVersion + + return nil +} + +// beginReplacementSlot records a role's replacement before anything is deleted. The captured +// identities are the only record of what the departing replica was; once its members are gone an +// empty ordinal names no replica to recover. +func (r *ModelDeploymentReconciler) beginReplacementSlot( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, +) error { + current, observed, err := r.loadReplacementSlots(ctx, md) + if err != nil { + return err + } + if held, present := observed.slotFor(slot.Role); present && held.Ordinal == slot.Ordinal { + return nil + } + + return r.writeReplacementSlots(ctx, current, observed.setSlot(slot), md) +} + +// advanceReplacementSlot persists a change to a slot this pass did not create. +func (r *ModelDeploymentReconciler) advanceReplacementSlot( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, +) error { + current, observed, err := r.loadReplacementSlots(ctx, md) + if err != nil { + return err + } + + return r.writeReplacementSlots(ctx, current, observed.setSlot(slot), md) +} + +// endReplacementSlot removes a slot that has finished. +func (r *ModelDeploymentReconciler) endReplacementSlot( + ctx context.Context, md *workercore.ModelDeployment, role string, +) error { + current, observed, err := r.loadReplacementSlots(ctx, md) + if err != nil { + return err + } + if !observed.held(role) { + return nil + } + + return r.writeReplacementSlots(ctx, current, observed.clearRole(role), md) +} + +// replacementCapturedIdentities builds the slot a selection pass writes: the members it is about +// to delete and the Workload that owns them, captured from the objects as they are now. +func replacementCapturedIdentities( + view modelDeploymentReplicaView, workloads []kueue.Workload, +) modelDeploymentReplacementSlot { + memberUIDs := make([]types.UID, 0, len(view.Members)) + for _, member := range view.Members { + memberUIDs = append(memberUIDs, member.UID) + } + slices.Sort(memberUIDs) + + slot := modelDeploymentReplacementSlot{ + Role: view.Role, + Ordinal: view.Ordinal, + Phase: modelDeploymentReplacementCleanup, + MemberUIDs: memberUIDs, + } + if wl := replicaOwningWorkload(view.Members, workloadPointers(workloads)); wl != nil { + slot.WorkloadUID = wl.UID + } + + return slot +} + +// workloadPointers views a listed Workload list as the pointers the ownership helpers take. +func workloadPointers(workloads []kueue.Workload) []*kueue.Workload { + pointers := make([]*kueue.Workload, 0, len(workloads)) + for i := range workloads { + pointers = append(pointers, &workloads[i]) + } + + return pointers +} + +// replicaOwningWorkload returns the Workload claiming any of a replica's members. A Workload that +// merely carries the group's name is not this replica's. +func replicaOwningWorkload(members []*core.Pod, workloads []*kueue.Workload) *kueue.Workload { + pods := make([]core.Pod, 0, len(members)) + for _, member := range members { + pods = append(pods, *member) + } + + byMember := modelDeploymentReplicaWorkloads(pods, workloads) + for _, member := range members { + if wl := byMember[member.UID]; wl != nil { + return wl + } + } + + return nil +} + +// replacementMemberDeletes returns the captured members still standing. The caller deletes each +// under its captured UID, so a Pod that took the name after the capture is not removed. +func replacementMemberDeletes( + slot modelDeploymentReplacementSlot, members []*core.Pod, +) []*core.Pod { + deletes := make([]*core.Pod, 0, len(slot.MemberUIDs)) + for _, member := range members { + if replacementCapturesMember(slot, member.UID) { + deletes = append(deletes, member) + } + } + slices.SortFunc(deletes, func(a, b *core.Pod) int { return strings.Compare(a.Name, b.Name) }) + + return deletes +} + +// replacementWorkloadDeletes returns the Workloads the slot may delete: the one it captured, and a +// Workload that names one of the members it captured as an owner. The second covers a slot written +// before the informer delivered the departing group's Workload, which carries no WorkloadUID but is +// still provably that group's. A matching name, or an ownerless shape, is not authority. +func replacementWorkloadDeletes( + slot modelDeploymentReplacementSlot, workloads []*kueue.Workload, +) []*kueue.Workload { + members := sets.New[types.UID](slot.MemberUIDs...) + + var deletes []*kueue.Workload + for _, wl := range workloads { + if replacementCapturesWorkload(slot, wl) || modelDeploymentWorkloadOwnsAny(wl, members) { + deletes = append(deletes, wl) + } + } + + return deletes +} + +// workloadOwnsGroupName reports whether a Workload stands on a group's derived name. It locates +// an object; it never authorizes deleting or adopting one. +func workloadOwnsGroupName(wl *kueue.Workload, group string) bool { + if wl.Name == group { + return true + } + for _, ps := range wl.Spec.PodSets { + if ps.Name == kueue.PodSetReference(group) { + return true + } + } + + return false +} + +// replacementOccupancyOnServer lists the API server's members of a role at one ordinal, +// terminating ones included: a departing member still on the server holds the slot. +func (r *ModelDeploymentReconciler) replacementOccupancyOnServer( + ctx context.Context, md *workercore.ModelDeployment, role string, ordinal int, +) ([]core.Pod, error) { + podList := new(core.PodList) + if err := r.APIReader.List(ctx, podList, + ctrlcli.InNamespace(md.Namespace), + ctrlcli.MatchingLabels{ + modelDeploymentLabelKeyName: modelDeploymentLabelValueName, + modelDeploymentLabelKeyInstance: md.Name, + modelDeploymentLabelKeyComponent: role, + }); err != nil { + return nil, fmt.Errorf("list the members of role %q on the api server: %w", role, err) + } + + held := make([]core.Pod, 0, len(podList.Items)) + for i := range podList.Items { + pod := &podList.Items[i] + if !modelDeploymentOwns(pod, md) || modelDeploymentPodRole(pod) != role { + continue + } + if pod.Labels[modelDeploymentReplicaOrdinalLabel] == strconvx.Itoa(ordinal) { + held = append(held, *pod) + } + } + + return held, nil +} + +// modelDeploymentPartialGroupMembers reports which members a replacement slot's group is still +// missing, and whether filling them is this operator's to do. +// +// A PARTIAL CREATE LEAVES THE REPLICA'S GROUP SHORT OF ITS TOTAL, and Kueue composes no Workload for +// an incomplete group, so a slot waiting on one waits forever. Filling it is safe only under three +// conditions, and each of them is checked rather than assumed: +// +// - the group is declared at the size this pass renders it, so the member to add agrees with the +// members already there about what the group is; +// - every member standing carries this pass's own render, so the group is the configuration the +// spec states now rather than a survivor of an earlier one; +// - the group holds no Workload, so nothing has been composed against it and nothing admitted is +// being extended. +// +// A group failing any of them is not filled here. A replica that lost a member under an earlier +// render waits behind the slot and is replaced whole, because adding a member from the current spec +// beside one from the previous spec is a group whose members disagree about what they are. +// +// The occupancy and the Workloads are read on the API server, because the state this question lives +// in is exactly the window the cache lags in: the members were created moments ago. +func (r *ModelDeploymentReconciler) modelDeploymentPartialGroupMembers( + ctx context.Context, md *workercore.ModelDeployment, + slot modelDeploymentReplacementSlot, want []*core.Pod, +) ([]*core.Pod, bool, error) { + members, err := r.replacementOccupancyOnServer(ctx, md, slot.Role, slot.Ordinal) + if err != nil { + return nil, false, err + } + + standing := replacementPodPointers(members) + for _, member := range standing { + if member.DeletionTimestamp != nil { + return nil, false, nil + } + } + + // Nothing standing is not a partial create: an empty ordinal is the create gate's own case, and + // it waits for the vacancy to be proved before anything is built there. + if len(standing) == 0 || len(standing) >= len(want) { + return nil, false, nil + } + + // Every member standing must be seated on a seat this render has, and must carry this render's + // own fingerprint for it. The group being short is expected here; the group being made of + // something else is not. + seated := make(map[int]string, len(standing)) + for _, member := range standing { + seat := modelDeploymentPodMemberIndex(member) + if seat >= len(want) { + return nil, false, nil + } + if _, duplicated := seated[seat]; duplicated { + return nil, false, nil + } + seated[seat] = member.Annotations[modelDeploymentPodSpecHashAnnotation] + } + for index, rendered := range want { + if carried, present := seated[index]; present && + carried != rendered.Annotations[modelDeploymentPodSpecHashAnnotation] { + return nil, false, nil + } + } + + total, totalReason := modelDeploymentReplicaDeclaredTotal(standing) + if totalReason != "" || total != len(want) { + return nil, false, nil + } + + workloads, err := r.findModelDeploymentGroupWorkloads(ctx, md, members) + if err != nil { + return nil, false, err + } + if len(workloads) > 0 { + return nil, false, nil + } + + missing := make([]*core.Pod, 0, len(want)-len(standing)) + for index, rendered := range want { + if _, present := seated[index]; !present { + missing = append(missing, rendered) + } + } + + return missing, len(missing) > 0, nil +} + +// modelDeploymentLiveRoleSelectors reads the selector each role's Service is published with, so a +// deployment whose eligibility was enabled keeps routing by it on the passes that did not decide it. +// +// The read is uncached. A selector published by the previous pass is the fact being asked about, and +// the informer lags the writer on exactly the pass that published it. +func (r *ModelDeploymentReconciler) modelDeploymentLiveRoleSelectors( + ctx context.Context, md *workercore.ModelDeployment, +) (map[string]map[string]string, error) { + selectors := make(map[string]map[string]string, len(md.Spec.Roles)) + for i := range md.Spec.Roles { + role := &md.Spec.Roles[i] + svc := new(core.Service) + err := r.APIReader.Get(ctx, ctrlcli.ObjectKey{ + Namespace: md.Namespace, Name: md.Name + "-" + role.Name, + }, svc) + if err != nil { + if kerrors.IsNotFound(err) { + continue + } + + return nil, fmt.Errorf("read the service of role %q: %w", role.Name, err) + } + selectors[role.Name] = maps.Clone(svc.Spec.Selector) + if svc.Annotations[modelDeploymentEligibilityActiveAnnotation] == modelDeploymentEndpointEligibleValue { + selectors[role.Name][modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue + } + } + + return selectors, nil +} + +// replacementVacant reports whether the slot's ordinal is provably free on the server. +// +// The reads are uncached because the informer lags the writer in exactly this window: a departing +// member the cache still lists, a create whose response was lost, an old Workload already gone. +// A lost response is not absence. +func (r *ModelDeploymentReconciler) replacementVacant( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, +) (vacant bool, why string, err error) { + occupied, err := r.replacementOccupancyOnServer(ctx, md, slot.Role, slot.Ordinal) + if err != nil { + return false, "", err + } + if len(occupied) > 0 { + names := make([]string, 0, len(occupied)) + for i := range occupied { + names = append(names, occupied[i].Name) + } + slices.Sort(names) + + return false, fmt.Sprintf("the server still holds %s", strings.Join(names, ", ")), nil + } + + // An occupied name holds the slot even by a Workload this operator cannot prove it composed. + // Such a Workload is never deleted and never adopted, but a replacement composed beside it + // would collide, so the wait stands until it goes. + blocking, err := r.replacementGroupWorkload(ctx, md, slot) + if err != nil { + return false, "", err + } + if blocking != nil { + if !blocking.DeletionTimestamp.IsZero() { + return false, fmt.Sprintf("workload %s is still terminating", blocking.Name), nil + } + + return false, fmt.Sprintf("workload %s still occupies the group", blocking.Name), nil + } + + return true, "", nil +} + +// replacementGroupWorkload returns the Workload still standing on a slot's derived group name, +// and nil when the group holds none. +func (r *ModelDeploymentReconciler) replacementGroupWorkload( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, +) (*kueue.Workload, error) { + group := modelDeploymentReplicaGroupName(md, slot.Role, slot.Ordinal) + wlList := new(kueue.WorkloadList) + if err := r.APIReader.List(ctx, wlList, ctrlcli.InNamespace(md.Namespace)); err != nil { + return nil, fmt.Errorf("list workloads: %w", err) + } + for i := range wlList.Items { + if workloadOwnsGroupName(&wlList.Items[i], group) { + return &wlList.Items[i], nil + } + } + + return nil, nil +} + +// replacementAdmitted reports whether the slot's ordinal now holds a complete current group whose +// Workload is admitted. Quota reservation is not admission: a reserved group is a request Kueue +// agreed to consider, and releasing the role on it would release it while the replacement holds +// nothing. +func (r *ModelDeploymentReconciler) replacementAdmitted( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, + want []*core.Pod, +) (admitted bool, why string, err error) { + members, err := r.replacementOccupancyOnServer(ctx, md, slot.Role, slot.Ordinal) + if err != nil { + return false, "", err + } + + standing := make([]*core.Pod, 0, len(members)) + for i := range members { + if members[i].DeletionTimestamp != nil { + return false, fmt.Sprintf("member %s is on its way out", members[i].Name), nil + } + standing = append(standing, &members[i]) + } + + if reason := replacementMembersMatch(want, standing); reason != "" { + return false, reason, nil + } + + wl, err := r.replacementGroupWorkload(ctx, md, slot) + if err != nil { + return false, "", err + } + if wl == nil { + return false, "the group holds no workload yet", nil + } + + if reason := replacementAdmissionHolds(wl, standing, want); reason != "" { + return false, reason, nil + } + + return true, "", nil +} + +// replacementAdmissionHolds reports whether an admitted Workload is fresh admission for the group +// its current members make, and why it is not when it is not. +// +// Ownership alone is not enough: a recreated Pod takes the same name and the same group, so a +// Workload composed for the departing replica can own the members that replaced it. An admission is +// fresh when its scheduling inputs describe the group standing now -- same members, count, +// template, reservation and queue. +func replacementAdmissionHolds(wl *kueue.Workload, standing, want []*core.Pod) string { + if !kubeapistatus.ConditionType(kueue.WorkloadAdmitted).IsTrue(wl) { + return fmt.Sprintf("workload %s is not admitted yet", wl.Name) + } + + ours := sets.New[types.UID]() + for _, member := range standing { + ours.Insert(member.UID) + } + if !modelDeploymentWorkloadOwnsAny(wl, ours) { + return fmt.Sprintf("workload %s owns none of the replica's current members", wl.Name) + } + for _, member := range standing { + if !modelDeploymentWorkloadOwnsAny(wl, sets.New(member.UID)) { + return fmt.Sprintf("workload %s does not own member %s", wl.Name, member.Name) + } + } + + if reason := replacementCompositionHolds(wl, standing); reason != "" { + return reason + } + + // The queue is compared unconditionally: a Workload naming none has not proved it submitted + // where its members were sent. + if queue := replacementQueueName(want); queue != "" && string(wl.Spec.QueueName) != queue { + return fmt.Sprintf("workload %s is queued on %q rather than %q", wl.Name, wl.Spec.QueueName, queue) + } + + return "" +} + +// replacementCompositionHolds compares a Workload's scheduling inputs against the composition its +// own current members produce, and names the first difference. +// +// The composition is rebuilt from the members standing now rather than copied from the Workload: +// grouped by role hash as Kueue groups them, each count read from the members carrying it. +func replacementCompositionHolds(wl *kueue.Workload, members []*core.Pod) string { + pinned := replacementPinnedPodSets(members) + // Members sharing a role hash compose one podset carrying that count, so the number of + // podsets is the number of distinct hashes among them, not the number of members. + if len(wl.Spec.PodSets) != len(pinned) { + return fmt.Sprintf("workload %s describes %d podsets for members composing %d", + wl.Name, len(wl.Spec.PodSets), len(pinned)) + } + live := make(map[kueue.PodSetReference]kueue.PodSet, len(wl.Spec.PodSets)) + for _, ps := range wl.Spec.PodSets { + live[ps.Name] = ps + } + grouped := map[kueue.PodSetReference][]*core.Pod{} + for _, member := range members { + hash, err := replacementRoleHash(member) + if err != nil { + continue + } + grouped[hash] = append(grouped[hash], member) + } + + for _, want := range pinned { + got, present := live[want.Name] + if !present { + return fmt.Sprintf("workload %s carries no podset %s", wl.Name, want.Name) + } + if got.Count != want.Count { + return fmt.Sprintf("workload %s asks for %d pods in podset %s, not the %d its members compose to", + wl.Name, got.Count, want.Name, want.Count) + } + if reason := replacementTemplateMatches(wl, got, grouped[want.Name]); reason != "" { + return reason + } + } + + return "" +} + +// replacementTemplateMatches reports whether a Workload's PodSet describes one of a group's members. +// +// Any member matches, not only the one Kueue listed first: its List carries no order and rank-specific +// members share a role hash, so a template composed from m1 is correct even when the reader sees m0 +// first. +func replacementTemplateMatches( + wl *kueue.Workload, got kueue.PodSet, members []*core.Pod, +) string { + for _, member := range members { + template := core.PodTemplateSpec{Spec: *member.Spec.DeepCopy()} + if replacementPodShapesMatch(got.Template.Spec, template.Spec) { + // Kueue states no separate request on a PodSet: what it reserves on is the request it + // derives from the template's containers. + if resourceQuantityListEqual( + replacementPodRequests(&core.Pod{Spec: got.Template.Spec}), + replacementPodRequests(&core.Pod{Spec: template.Spec})) { + return "" + } + + return fmt.Sprintf("workload %s reserves resources in podset %s that its members do not declare", + wl.Name, got.Name) + } + } + + return fmt.Sprintf("workload %s describes a pod in podset %s that its members do not match", + wl.Name, got.Name) +} + +// replacementPinnedPodSets rebuilds the PodSets a set of members composes to, following Kueue's +// own rule: grouped by the role hash read verbatim, each count taken from the members carrying it. +func replacementPinnedPodSets(members []*core.Pod) []kueue.PodSet { + order := make([]kueue.PodSetReference, 0, len(members)) + grouped := make(map[kueue.PodSetReference][]*core.Pod, len(members)) + for _, member := range members { + hash, err := replacementRoleHash(member) + if err != nil { + continue + } + if _, present := grouped[hash]; !present { + order = append(order, hash) + } + grouped[hash] = append(grouped[hash], member) + } + + pinned := make([]kueue.PodSet, 0, len(order)) + for _, hash := range order { + group := grouped[hash] + pinned = append(pinned, replacementPodSetFor(hash, group)) + } + + return pinned +} + +// replacementPodSetFor is the PodSet one role hash composes to: the count of the members carrying +// it, and the template of whichever of them Kueue listed first. +// +// Which member that is does not matter, and the pinned set keeps all of them because Kueue's List +// carries no order: rank-specific members share one role hash by design, and a template taken from +// whichever member happened to come first would read a valid placement as a stale one. +func replacementPodSetFor(hash kueue.PodSetReference, members []*core.Pod) kueue.PodSet { + return kueue.PodSet{ + Name: hash, + Count: int32(len(members)), + Template: core.PodTemplateSpec{Spec: *members[0].Spec.DeepCopy()}, + } +} + +// replacementRoleHash is the group name Kueue gives a PodSet: the annotation this operator writes, +// read verbatim when present, and otherwise the digest Kueue derives from the Pod's shape. +func replacementRoleHash(pod *core.Pod) (kueue.PodSetReference, error) { + if hash, present := pod.Annotations[kueuepodconst.RoleHashAnnotation]; present && hash != "" { + return kueue.NewPodSetReference(hash), nil + } + + hash, err := utilpod.GenerateRoleHash(&pod.Spec) + if err != nil { + return "", err + } + + return kueue.NewPodSetReference(hash), nil +} + +// replacementPodRequests is what a member contributes to its PodSet's reservation. +// +// IT IS KUBERNETES' OWN HELPER rather than a rule written here, because the rule is not this +// operator's to state: requests win over limits, several containers sum, and init containers carry +// the pod maximum. A hand-rolled aggregation agreeing with it on one shape is not agreement, and +// the reservation being compared is the one Kueue will charge. +func replacementPodRequests(pod *core.Pod) core.ResourceList { + return resourcehelper.PodRequests(pod, resourcehelper.PodResourcesOptions{}) +} + +// replacementPodShapesMatch compares the Pod specs Kueue reserved quota against, ignoring what +// admission is expected to have changed between the two. +func replacementPodShapesMatch(got, want core.PodSpec) bool { + actual, member := replacementComparablePod(got), replacementComparablePod(want) + actual.Affinity = replacementWorkloadAffinity(actual.Affinity, member.Affinity) + + return kubemeta.DeepEqual(actual, member) +} + +// replacementWorkloadAffinity removes only Workload webhook pins absent from the member. +// Fit and model-manager pins constrain Kueue placement but are not copied to the Pod. +// Required terms and all other affinity must still describe the member's placement. +func replacementWorkloadAffinity(got, want *core.Affinity) *core.Affinity { + if got == nil || got.NodeAffinity == nil || got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution == nil { + return got + } + var wanted []core.NodeSelectorTerm + if want != nil && want.NodeAffinity != nil && want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil { + wanted = want.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution.NodeSelectorTerms + } + required := got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution + if len(required.NodeSelectorTerms) != max(1, len(wanted)) { + return got + } + for i := range required.NodeSelectorTerms { + term := &required.NodeSelectorTerms[i] + var original []core.NodeSelectorRequirement + if len(wanted) > 0 { + original = wanted[i].MatchExpressions + } + before := len(term.MatchExpressions) + term.MatchExpressions = slices.DeleteFunc(term.MatchExpressions, func(e core.NodeSelectorRequirement) bool { + if !nodefeature.IsFitLabelKey(e.Key) && e.Key != ModelManagerRegisteredLabel { + return false + } + return !slices.ContainsFunc(original, func(existing core.NodeSelectorRequirement) bool { + return kubemeta.DeepEqual(existing, e) + }) + }) + if len(wanted) == 0 && before > 0 && len(term.MatchExpressions) == 0 && len(term.MatchFields) == 0 { + got.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution = nil + } + } + if kubemeta.DeepEqual(got.NodeAffinity, &core.NodeAffinity{}) { + got.NodeAffinity = nil + } + if kubemeta.DeepEqual(got, &core.Affinity{}) { + return nil + } + + return got +} + +// replacementComparablePod reduces a Pod spec to what the operator rendered, which is what Kueue +// reserved on. +// +// The PodSet holds the pre-admission template, so the live Pod carries the placement the cluster made +// after Kueue read it: a node selector and tolerations the admission chain added, a scheduling gate +// it consumed, and the node the scheduler bound. Those are one configuration seen before and after, +// so they are compared out. Everything the operator renders is still compared, so a template +// describing a different configuration is still refused. +func replacementComparablePod(spec core.PodSpec) core.PodSpec { + reduced := *spec.DeepCopy() + reduced.Volumes = nil + reduced.InitContainers = nil + reduced.EphemeralContainers = nil + reduced.RestartPolicy = "" + reduced.TerminationGracePeriodSeconds = nil + reduced.ActiveDeadlineSeconds = nil + reduced.DNSPolicy = "" + reduced.NodeSelector = nil + reduced.Tolerations = nil + reduced.SchedulingGates = nil + reduced.NodeName = "" + for i := range reduced.Containers { + reduced.Containers[i].TerminationMessagePath = "" + reduced.Containers[i].TerminationMessagePolicy = "" + reduced.Containers[i].ImagePullPolicy = "" + // Resources are compared on their own, since that is what Kueue reserves on. + reduced.Containers[i].Resources = core.ResourceRequirements{} + } + + return reduced +} + +// resourceQuantityListEqual compares two resource lists by value rather than by representation. +func resourceQuantityListEqual(got, want core.ResourceList) bool { + if len(got) != len(want) { + return false + } + for name, quantity := range want { + other, present := got[name] + if !present { + return false + } + if quantity.Cmp(other) != 0 { + return false + } + } + + return true +} + +// replacementMembersMatch reports whether a standing group is the render this pass produced, and +// names the first difference. The rendered fingerprint covers the template and the request +// together, so a replica rebuilt from any other configuration does not match. +func replacementMembersMatch(want, standing []*core.Pod) string { + if len(standing) != len(want) { + return fmt.Sprintf("the replica holds %d of the %d members it is rendered at", + len(standing), len(want)) + } + + seats := make(map[int]*core.Pod, len(standing)) + for _, member := range standing { + seat := modelDeploymentPodMemberIndex(member) + if _, duplicated := seats[seat]; duplicated { + return fmt.Sprintf("two members sit at index %d", seat) + } + seats[seat] = member + } + + for index, rendered := range want { + member, seated := seats[index] + if !seated { + return fmt.Sprintf("no member sits at index %d", index) + } + if member.Annotations[modelDeploymentPodSpecHashAnnotation] != + rendered.Annotations[modelDeploymentPodSpecHashAnnotation] { + return fmt.Sprintf("member %s does not carry the render this pass produced", member.Name) + } + } + + return "" +} + +// replacementQueueName is the LocalQueue a rendered replica submits to, read off the render so +// this check and the label Kueue admits the group by cannot disagree. +func replacementQueueName(want []*core.Pod) string { + for _, rendered := range want { + if entrance := rendered.Labels[kueuectrlconst.QueueLabel]; entrance != "" { + return entrance + } + } + + return "" +} + +// replacementRenderHashes are the fingerprints a render produced for one replica, in member order. +func replacementRenderHashes(want []*core.Pod) []string { + hashes := make([]string, 0, len(want)) + for _, rendered := range want { + hashes = append(hashes, rendered.Annotations[modelDeploymentPodSpecHashAnnotation]) + } + + return hashes +} + +// replacementSuperseded reports whether the slot's replacement was rendered from a configuration +// the spec has since replaced. +// +// The comparison is on rendered fingerprints, not on the generation: a replica-count edit moves +// the generation without moving any member's render, and treating that as a configuration change +// would tear down a healthy replacement and rebuild it identically. +func replacementSuperseded(slot modelDeploymentReplacementSlot, want []*core.Pod) bool { + if len(slot.MemberHashes) == 0 { + // The slot has not recorded its target render yet, so nothing is known to be obsolete. + return false + } + if len(slot.MemberHashes) != len(want) { + return true + } + + return !slices.Equal(slot.MemberHashes, replacementRenderHashes(want)) +} + +// modelDeploymentReplacementProgress is what one pass did about a role's active slot. +type modelDeploymentReplacementProgress struct { + // slot is the record to carry into the rest of the pass. + slot modelDeploymentReplacementSlot + + // cleared is that the replacement holds an admission and the role no longer holds a slot. + cleared bool + + // requeue is that nothing else will wake this deployment before the slot can move. + requeue bool +} + +// resolveReplacementSlot advances one role's active slot by one step. +// +// The order of the questions is the lifecycle: an ordinal the spec no longer declares is finished +// by cleanup alone; then the captured members are removed; then vacancy is proved on the server, +// because a lost delete response is not absence; then a superseded replacement is rebuilt in this +// same slot; then an admitted replacement releases the role. +func (r *ModelDeploymentReconciler) resolveReplacementSlot( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, + role *workercore.ModelDeploymentRole, want []*core.Pod, +) (modelDeploymentReplacementProgress, error) { + logger := ctrllog.FromContext(ctx) + progress := modelDeploymentReplacementProgress{slot: slot} + + // A REPLICA COUNT THAT NO LONGER DECLARES THIS ORDINAL CANCELS THE SLOT, and the cancellation + // runs through the same cleanup: the members and the Workload the slot captured still have to + // leave before the record can be dropped, or the finalizer and the reservation outlive it. + if slot.Ordinal >= int(role.Replicas) { + members, err := r.replacementOccupancyOnServer(ctx, md, slot.Role, slot.Ordinal) + if err != nil { + return progress, err + } + if err = r.cleanReplacementSlot(ctx, md, slot, replacementPodPointers(members)); err != nil { + return progress, err + } + vacant, why, err := r.replacementVacant(ctx, md, slot) + if err != nil { + return progress, err + } + if !vacant { + progress.requeue = true + logger.V(3).Info("waiting for a cancelled replacement to leave", + "slot", replacementDescription(slot), "because", why) + + return progress, nil + } + progress.cleared = true + + return progress, nil + } + + // Whatever the slot captured is removed first, on its own authority. Members that arrived + // afterwards belong to a replacement this slot is waiting on and are left standing. + captured, err := r.replacementOccupancyOnServer(ctx, md, slot.Role, slot.Ordinal) + if err != nil { + return progress, err + } + // A CAPTURED MEMBER ALREADY ON ITS WAY OUT IS LEFT TO ITS FINALIZER. Re-issuing the delete + // would say nothing and would keep this pass reporting that it issued a cleanup it had not, + // which is the one state the vacancy question below is asked from. + mine := replacementStandingMemberDeletes(slot, replacementPodPointers(captured)) + if slot.Phase == modelDeploymentReplacementCleanup { + if err = r.cleanReplacementSlot(ctx, md, slot, mine); err != nil { + return progress, err + } + if len(mine) > 0 { + progress.requeue = true + return progress, nil + } + } + + standing := replacementUncapturedMembers(slot, captured) + wl, wlErr := r.replacementGroupWorkload(ctx, md, slot) + if wlErr != nil { + return progress, wlErr + } + if slot.Phase == modelDeploymentReplacementWaitingForAdmission { + present := sets.New[types.UID]() + broken := false + for _, member := range standing { + present.Insert(member.UID) + broken = broken || member.DeletionTimestamp != nil + } + for _, uid := range slot.CurrentMemberUIDs { + broken = broken || !present.Has(uid) + } + // A composed group cannot be filled beside its existing reservation. + ownsCurrent := wl != nil && (wl.UID == slot.CurrentWorkloadUID || + modelDeploymentWorkloadOwnsAny(wl, present.Union(sets.New(slot.CurrentMemberUIDs...)))) + if ownsCurrent && len(standing) < len(want) { + broken = true + } + if broken || replacementSuperseded(slot, want) { + next := modelDeploymentReplacementSlot{ + Role: slot.Role, Ordinal: slot.Ordinal, Phase: modelDeploymentReplacementCleanup, + MemberUIDs: sets.List(present.Union(sets.New(slot.CurrentMemberUIDs...))), + WorkloadUID: slot.CurrentWorkloadUID, + } + if ownsCurrent { + next.WorkloadUID = wl.UID + } + progress.slot = next + if err = r.advanceReplacementSlot(ctx, md, next); err != nil { + return progress, err + } + if err = r.cleanReplacementSlot(ctx, md, next, standing); err != nil { + return progress, err + } + progress.requeue = true + return progress, nil + } + // Persist identities before admission or cleanup can remove their observable evidence. + slot.CurrentMemberUIDs = sets.List(present) + if ownsCurrent { + slot.CurrentWorkloadUID = wl.UID + } + if err = r.advanceReplacementSlot(ctx, md, slot); err != nil { + return progress, err + } + progress.slot = slot + } + + // The replacement exists once a member the slot did not capture is standing. From there the + // slot waits on admission rather than on vacancy, and the two questions are different: vacancy + // is about the old group leaving, admission about the new one being granted quota. + if len(standing) == 0 { + vacant, why, err := r.replacementVacant(ctx, md, slot) + if err != nil { + return progress, err + } + if !vacant { + // Something still holds the ordinal. That is the ordinary waiting state -- a member + // draining behind its finalizer, a Workload not yet gone -- and it ends through the Pod + // and Workload events rather than through a timer. + logger.V(3).Info("waiting for the replaced group to leave", + "slot", replacementDescription(slot), "because", why) + + return progress, nil + } + + // The target render is recorded once the old group has left, which is what a later edit is + // compared against. + if slot.Phase == modelDeploymentReplacementCleanup || len(slot.MemberHashes) == 0 { + progress.slot = modelDeploymentReplacementSlot{ + Role: slot.Role, + Ordinal: slot.Ordinal, + Phase: modelDeploymentReplacementWaitingForAdmission, + MemberHashes: replacementRenderHashes(want), + } + if err = r.advanceReplacementSlot(ctx, md, progress.slot); err != nil { + return progress, err + } + } + + return progress, nil + } + + admitted, reason, err := r.replacementAdmitted(ctx, md, slot, want) + if err != nil { + return progress, err + } + if admitted { + progress.cleared = true + + return progress, nil + } + logger.V(3).Info("waiting for the replacement to be admitted", + "slot", replacementDescription(slot), "because", reason) + + return progress, nil +} + +// replacementStandingMemberDeletes returns the captured members still standing, skipping those +// already terminating. A member past its delete is on its way out whatever this pass does. +func replacementStandingMemberDeletes( + slot modelDeploymentReplacementSlot, members []*core.Pod, +) []*core.Pod { + deletes := make([]*core.Pod, 0, len(members)) + for _, member := range replacementMemberDeletes(slot, members) { + if member.DeletionTimestamp == nil { + deletes = append(deletes, member) + } + } + + return deletes +} + +// replacementUncapturedMembers returns the members on a slot's ordinal that it did not capture: +// the replacement, wherever an earlier pass managed to create it. +func replacementUncapturedMembers( + slot modelDeploymentReplacementSlot, live []core.Pod, +) []*core.Pod { + pointers := replacementPodPointers(live) + uncaptured := make([]*core.Pod, 0, len(pointers)) + for _, member := range pointers { + if !replacementCapturesMember(slot, member.UID) { + uncaptured = append(uncaptured, member) + } + } + slices.SortFunc(uncaptured, func(a, b *core.Pod) int { return strings.Compare(a.Name, b.Name) }) + + return uncaptured +} + +// replacementPodPointers views a list of Pod values as the pointers the cleanup helpers take. +func replacementPodPointers(pods []core.Pod) []*core.Pod { + pointers := make([]*core.Pod, 0, len(pods)) + for i := range pods { + pointers = append(pointers, &pods[i]) + } + + return pointers +} + +// cleanReplacementSlot removes what a slot captured: its members under their own UIDs, then the +// Workload holding them. +// +// A member and its Workload are deleted only under captured UIDs. The group name locates +// vacancy blockers and never authorizes cleanup, so a Workload this operator +// cannot prove it composed is left standing and the vacancy proof waits on it. +func (r *ModelDeploymentReconciler) cleanReplacementSlot( + ctx context.Context, md *workercore.ModelDeployment, slot modelDeploymentReplacementSlot, + members []*core.Pod, +) error { + for _, member := range replacementMemberDeletes(slot, members) { + uid := member.UID + if err := r.Client.Delete(ctx, member, ctrlcli.Preconditions{UID: &uid}); err != nil && + !kerrors.IsNotFound(err) { + return fmt.Errorf("delete replaced member %s: %w", member.Name, err) + } + } + + // A SLOT WHOSE MEMBERS ARE ALL GONE STILL OWES ITS WORKLOAD A DELETE. That Workload is what + // releases Kueue's finalizer on the Pods and gives the reservation back, and reading it as + // already done because the members left is what strands a group holding quota forever. + listed := new(kueue.WorkloadList) + if err := r.APIReader.List(ctx, listed, ctrlcli.InNamespace(md.Namespace)); err != nil { + return fmt.Errorf("list replaced workloads: %w", err) + } + workloads := make([]*kueue.Workload, 0, len(listed.Items)) + for i := range listed.Items { + workloads = append(workloads, &listed.Items[i]) + } + for _, wl := range replacementWorkloadDeletes(slot, workloads) { + uid := wl.UID + if err := r.Client.Delete(ctx, wl, ctrlcli.Preconditions{UID: &uid}); err != nil && + !kerrors.IsNotFound(err) { + return fmt.Errorf("delete replaced replica's workload %s: %w", wl.Name, err) + } + } + + return nil +} diff --git a/pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go b/pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go new file mode 100644 index 000000000..cafd26110 --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go @@ -0,0 +1,463 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/resource" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrlrecord "k8s.io/client-go/tools/record" + kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/nodefeature" + "gpustack.ai/gpustack/pkg/utils/strconvx" +) + +// composeReplicaGroup builds the Workload a set of group members composes to, following the rule +// replacementCompositionHolds checks against: one PodSet per role hash, count from the members +// carrying it, and the reservation read off the template's containers. +// +// The negative controls below are what matter here. Each asserts that the production admission +// proof rejects a Workload that IS admitted and DOES own the current members while disagreeing on +// count, template, reservation or queue, so they test this task's checks rather than the fixture. + +// composeReplicaGroup builds the Workload a set of group members composes to. +func composeReplicaGroup(md *workercore.ModelDeployment, role string, ordinal int, + members []*core.Pod, +) *kueue.Workload { + wl := &kueue.Workload{} + wl.Name, wl.Namespace = modelDeploymentReplicaGroupName(md, role, ordinal), md.Namespace + wl.UID = types.UID("wl-" + wl.Name) + wl.Spec.QueueName = kueue.LocalQueueName(members[0].Labels[kueuectrlconst.QueueLabel]) + wl.Spec.PodSets = replacementPinnedPodSets(members) + // Kueue owns EVERY member, one reference each, and the admission proof requires all of them: + // a Workload claiming one member of a group is admitting a different group. + for _, member := range members { + wl.OwnerReferences = append(wl.OwnerReferences, meta.OwnerReference{ + APIVersion: "v1", Kind: "Pod", Name: member.Name, UID: member.UID, + }) + } + + return wl +} + +// admit records the Workload as admitted, which is the state a replacement has to reach before its +// slot may be released. +func admit(wl *kueue.Workload) *kueue.Workload { + wl.Status.Conditions = []meta.Condition{{ + Type: kueue.WorkloadAdmitted, + Status: meta.ConditionTrue, + Reason: "Admitted", + LastTransitionTime: meta.Now(), + }} + + return wl +} + +// TestReplacementAdmission_AnHonestCompositionIsAccepted is the positive case the negative controls +// are read against: a Workload composed from the current members releases the slot. +func TestReplacementAdmission_AnHonestCompositionIsAccepted(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2) + + wl := admit(composeReplicaGroup(md, "server", 0, members)) + + reason := replacementAdmissionHolds(wl, members, members) + assert.Empty(t, reason, "a Workload composed from these members is fresh admission") +} + +func TestReplacementAdmission_WorkloadAffinityPins(t *testing.T) { + slicePin := core.NodeSelectorRequirement{ + Key: nodefeature.FitSlicedMaxFreeUnitsLabelKey("nvidia-test-gpu"), + Operator: core.NodeSelectorOpGt, Values: []string{"799999"}, + } + sharedPin := core.NodeSelectorRequirement{ + Key: nodefeature.FitSharedFreeCardsLabelKey("nvidia-test-gpu"), + Operator: core.NodeSelectorOpGt, Values: []string{"1"}, + } + pluginPin := core.NodeSelectorRequirement{ + Key: ModelManagerRegisteredLabel, Operator: core.NodeSelectorOpIn, Values: []string{"true"}, + } + volumePin := core.NodeSelectorRequirement{ + Key: core.LabelTopologyZone, Operator: core.NodeSelectorOpIn, Values: []string{"zone-a"}, + } + otherVolumePin := volumePin.DeepCopy() + otherVolumePin.Values = []string{"zone-b"} + otherSlicePin := slicePin.DeepCopy() + otherSlicePin.Values = []string{"399999"} + cases := []struct { + name string + member, workload []core.NodeSelectorRequirement + wantAccepted bool + }{ + {name: "slice fit pin only", workload: []core.NodeSelectorRequirement{slicePin}, wantAccepted: true}, + {name: "shared fit pin only", workload: []core.NodeSelectorRequirement{sharedPin}, wantAccepted: true}, + {name: "model manager pin only", workload: []core.NodeSelectorRequirement{pluginPin}, wantAccepted: true}, + { + name: "volume affinity with fit pin", member: []core.NodeSelectorRequirement{volumePin}, + workload: []core.NodeSelectorRequirement{volumePin, slicePin}, wantAccepted: true, + }, + { + name: "changed volume affinity with fit pin", member: []core.NodeSelectorRequirement{volumePin}, + workload: []core.NodeSelectorRequirement{*otherVolumePin, slicePin}, + }, + {name: "unrelated affinity", workload: []core.NodeSelectorRequirement{volumePin}}, + {name: "empty term matches no node", workload: []core.NodeSelectorRequirement{}}, + { + name: "member fit constraint preserved", member: []core.NodeSelectorRequirement{slicePin}, + workload: []core.NodeSelectorRequirement{slicePin, pluginPin}, wantAccepted: true, + }, + { + name: "changed member fit constraint refused", member: []core.NodeSelectorRequirement{slicePin}, + workload: []core.NodeSelectorRequirement{*otherSlicePin}, + }, + } + affinity := func(expressions []core.NodeSelectorRequirement) *core.Affinity { + if expressions == nil { + return nil + } + return &core.Affinity{NodeAffinity: &core.NodeAffinity{ + RequiredDuringSchedulingIgnoredDuringExecution: &core.NodeSelector{ + NodeSelectorTerms: []core.NodeSelectorTerm{{MatchExpressions: expressions}}, + }, + }} + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + md := newRenderDeployment() + members := replicaMembersAt(t, md, "server", 0) + members[0].Spec.Affinity = affinity(tc.member) + wl := admit(composeReplicaGroup(md, "server", 0, members)) + wl.Spec.PodSets[0].Template.Spec.Affinity = affinity(tc.workload) + reason := replacementAdmissionHolds(wl, members, members) + if tc.wantAccepted { + assert.Empty(t, reason, "a Workload-only admission pin must not hold the replacement") + } else { + assert.Contains(t, reason, "do not match", "a different placement still holds the replacement") + } + }) + } +} + +// TestReplacementAdmission_AFollowerRepresentativeIsStillFreshAdmission covers the representative +// Kueue happened to list. Its List carries no order, and rank-specific members share one role hash +// by design, so a template composed from m1 is correct even when the reader sees m0 first. +func TestReplacementAdmission_AFollowerRepresentativeIsStillFreshAdmission(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2) + + wl := admit(composeReplicaGroup(md, "server", 0, members)) + wl.Spec.PodSets = replacementPinnedPodSets(members) + wl.Spec.PodSets[0].Template = core.PodTemplateSpec{Spec: *members[1].Spec.DeepCopy()} + + assert.Empty(t, replacementAdmissionHolds(wl, members, members), + "a template composed from any member of the group is a template of that group") +} + +// TestReplacementAdmission_ARequestOnlyPodIsReservedAsKubernetesReadsIt covers the reservation rule +// this operator does not own: a request smaller than the limit is reserved on the request, and +// several containers sum. +func TestReplacementAdmission_ARequestOnlyPodIsReservedAsKubernetesReadsIt(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + + withRequests := members[0].DeepCopy() + withRequests.Spec.Containers[0].Resources = core.ResourceRequirements{ + Requests: core.ResourceList{core.ResourceCPU: resource.MustParse("4")}, + Limits: core.ResourceList{core.ResourceCPU: resource.MustParse("16")}, + } + sidecar := core.Container{Name: "sidecar", Image: "busybox"} + sidecar.Resources.Requests = core.ResourceList{core.ResourceCPU: resource.MustParse("2")} + withRequests.Spec.Containers = append(withRequests.Spec.Containers, sidecar) + + wl := admit(composeReplicaGroup(md, "server", 0, members)) + wl.Spec.PodSets = replacementPinnedPodSets([]*core.Pod{withRequests}) + + group := []*core.Pod{withRequests} + assert.Empty(t, replacementAdmissionHolds(wl, group, group), + "a reservation of request plus sidecar is the reservation the members compose to") + + // The same pod shape, reserving less: the fault is the reservation, not the template. + understated := wl.DeepCopy() + understated.Spec.PodSets[0].Template.Spec.Containers[1].Resources.Requests = core.ResourceList{core.ResourceCPU: resource.MustParse("1")} + assert.Contains(t, replacementAdmissionHolds(understated, group, group), "reserves resources", + "a reservation short of what the sidecar requests reserves for a smaller Pod than the group") + + // A reservation that drops the sidecar outright is a different shape, and is refused as one. + truncated := wl.DeepCopy() + truncated.Spec.PodSets[0].Template.Spec.Containers = withRequests.Spec.Containers[:1] + assert.Contains(t, replacementAdmissionHolds(truncated, group, group), "do not match", + "and a template naming fewer containers describes a Pod the group does not have") +} + +// TestReplacementAdmission_AnAdmittedPlacementIsNotAStaleReservation covers the mutation admission +// makes: the Workload holds the pre-admission template, the live Pod carries what the admission +// chain added and consumed. +func TestReplacementAdmission_AnAdmittedPlacementIsNotAStaleReservation(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2) + + // The Workload is composed by Kueue's own constructor from these complete members, because a + // PodSet is the pre-admission template: it is a copy taken before the scheduler and the + // admission chain touched anything. + ctx := context.Background() + cli := newCompositionClient() + group := modelDeploymentReplicaGroupName(md, "server", 0) + stamped := make([]*core.Pod, 0, len(members)) + for i, member := range members { + stamped = append(stamped, createMember(t, ctx, cli, member, + "pod-placed-"+strconvx.Itoa(i))) + } + + wl, err := kueueComposeGroup(ctx, cli, group) + require.NoError(t, err) + require.NotNil(t, wl) + require.Len(t, wl.Spec.PodSets, 1) + require.Equal(t, int32(2), wl.Spec.PodSets[0].Count, + "the count is read from the members rather than declared anywhere") + wl.UID = types.UID("wl-" + wl.Name) + for _, member := range stamped { + wl.OwnerReferences = append(wl.OwnerReferences, meta.OwnerReference{ + APIVersion: "v1", Kind: "Pod", Name: member.Name, UID: member.UID, + }) + } + wl = admit(wl) + + // What the cluster then does to the members: the admission chain adds a selector and a + // toleration and consumes a gate, and the scheduler fills in the node it placed the member on. + placed := make([]*core.Pod, 0, len(stamped)) + for i, member := range stamped { + placedMember := member.DeepCopy() + placedMember.Spec.NodeSelector = map[string]string{"nvidia.com/gpu.present": "true"} + placedMember.Spec.Tolerations = []core.Toleration{{ + Key: "nvidia.com/gpu", Operator: core.TolerationOpExists, Effect: core.TaintEffectNoSchedule, + }} + placedMember.Spec.SchedulingGates = nil + placedMember.Spec.NodeName = "worker-" + strconvx.Itoa(i) + placed = append(placed, placedMember) + } + + assert.Empty(t, replacementAdmissionHolds(wl, placed, placed), + "an added selector, an added toleration, a consumed gate and a filled node are one "+ + "configuration seen before and after admission, not two") + + changed := wl.DeepCopy() + changed.Spec.PodSets[0].Template.Spec.Containers[0].Image = "vllm/vllm-openai:v0.0.1" + assert.Contains(t, replacementAdmissionHolds(changed, placed, placed), "do not match", + "while a rendered field the live Pods do not carry is still refused") + + shortened := wl.DeepCopy() + shortened.Spec.PodSets[0].Template.Spec.Containers[0].Resources.Requests["nvidia.com/gpu"] = resource.MustParse("2") + assert.Contains(t, replacementAdmissionHolds(shortened, placed, placed), "reserves", + "and a Workload reserving a different amount is still refused") +} + +// TestReplacementAdmission_ControlsRejectAWorkloadThatIsAdmittedButDescribesAnotherGroup is the +// negative control set, and each case answers one question: a Workload that Kueue has admitted and +// that owns the current members still releases nothing unless its scheduling inputs describe the +// group those members make. +func TestReplacementAdmission_ControlsRejectAnAdmittedButStaleWorkload(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + + cases := []struct { + name string + mutate func(*kueue.Workload) + wantText string + }{ + { + name: "an admitted workload on the wrong queue", + mutate: func(wl *kueue.Workload) { + wl.Spec.QueueName = kueue.LocalQueueName("queue-for-someone-else") + }, + wantText: "queued on", + }, + { + name: "an admitted workload naming no queue at all", + mutate: func(wl *kueue.Workload) { + wl.Spec.QueueName = "" + }, + wantText: "queued on", + }, + { + name: "an admitted workload reserving too few pods", + mutate: func(wl *kueue.Workload) { + wl.Spec.PodSets[0].Count = 1 + }, + wantText: "asks for", + }, + { + name: "an admitted workload describing a template its members do not match", + mutate: func(wl *kueue.Workload) { + wl.Spec.PodSets[0].Template.Spec.Containers[0].Image = "vllm/vllm-openai:v0.0.1" + }, + wantText: "do not match", + }, + { + name: "an admitted workload reserving resources its members do not declare", + mutate: func(wl *kueue.Workload) { + // The request, not the limit: Kubernetes reads the request when both are named, so + // a limit-only difference is not a difference in the reservation. + wl.Spec.PodSets[0].Template.Spec.Containers[0].Resources.Requests = core.ResourceList{core.ResourceMemory: resource.MustParse("512Gi")} + }, + wantText: "reserves resources", + }, + { + name: "an admitted workload carrying a podset for nothing else", + mutate: func(wl *kueue.Workload) { + wl.Spec.PodSets = append(wl.Spec.PodSets, kueue.PodSet{ + Name: "a-podset-for-another-shape", + Count: 1, + }) + }, + wantText: "podsets", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + members := replicaMembersAt(t, md, "server", 0) + require.Len(t, members, 2) + + wl := admit(composeReplicaGroup(md, "server", 0, members)) + tc.mutate(wl) + + reason := replacementAdmissionHolds(wl, members, members) + assert.NotEmpty(t, reason, + "an admitted workload describing another group releases nothing") + assert.Contains(t, reason, tc.wantText, + "and the reason names the disagreement, so an operator reading the status can act on it") + }) + } +} + +// TestReplacementAdmission_AStaleWorkloadDoesNotAdmitANewGroup covers the stale case directly: the +// ownership is current-looking, because a recreated Pod takes the same group, and only the +// scheduling inputs reveal that the Workload was composed before it. +func TestReplacementAdmission_AStaleWorkloadDoesNotAdmitANewGroup(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }) + old := replicaMembersAt(t, md, "server", 0) + + stale := admit(composeReplicaGroup(md, "qwen", 0, old)) + + // The old members leave and the group is rebuilt at one member per replica, so the workload + // still owns the names the new members took but describes the group that is gone. + edited := md.DeepCopy() + edited.Spec.Roles[0].ReplicaSize = 1 + fresh := replicaMembersAt(t, edited, "server", 0) + require.Len(t, fresh, 1) + + reason := replacementAdmissionHolds(stale, fresh, fresh) + assert.NotEmpty(t, reason, "an admission of the previous group is not an admission of this one") +} + +// replicaMembersAt runs a pass and returns the members it rendered for one ordinal, each with a +// distinct UID so an ownership match cannot pass on two empty values. +func replicaMembersAt( + t *testing.T, md *workercore.ModelDeployment, role string, ordinal int, +) []*core.Pod { + t.Helper() + + cli := newModelDeploymentClient(md, newRenderInstanceType()) + for pass := 0; pass < 4; pass++ { + _, err := reconcileModelDeploymentWith(t, &ModelDeploymentReconciler{ + Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(64), + }) + require.NoError(t, err, "pass %d", pass) + } + + members := make([]*core.Pod, 0, 4) + for _, pod := range replicaPods(t, cli) { + if modelDeploymentPodRole(&pod) != role { + continue + } + if at, ok := modelDeploymentPodOrdinal(&pod); !ok || at != ordinal { + continue + } + live := pod.DeepCopy() + live.UID = types.UID("pod-" + live.Name) + require.NoError(t, cli.Update(context.Background(), live)) + members = append(members, live) + } + require.NotEmpty(t, members, "the render produced no member for ordinal %d", ordinal) + + return members +} + +// TestReplacementWorkloadDeletes_OnlyProvableOwnershipIsDeletable pins the contract that the group +// name locates vacancy blockers and never authorizes cleanup. A Workload is deletable only when the +// slot captured its UID, or when it names a member the slot captured as an owner, which is how a +// slot written while the Workload was still unseen reaches it. +func TestReplacementWorkloadDeletes_OnlyProvableOwnershipIsDeletable(t *testing.T) { + ownedBy := func(name string, uid, pod types.UID) *kueue.Workload { + wl := &kueue.Workload{ObjectMeta: meta.ObjectMeta{Name: name, UID: uid}} + if pod != "" { + wl.OwnerReferences = []meta.OwnerReference{{APIVersion: "v1", Kind: "Pod", Name: "p", UID: pod}} + } + + return wl + } + captured := ownedBy("captured", "wl-captured", "") + orphan := ownedBy("orphan", "wl-orphan", "") + byMember := ownedBy("by-member", "wl-by-member", "pod-captured") + byStranger := ownedBy("by-stranger", "wl-by-stranger", "pod-other") + + slot := modelDeploymentReplacementSlot{WorkloadUID: "wl-captured", MemberUIDs: []types.UID{"pod-captured"}} + deletes := replacementWorkloadDeletes(slot, []*kueue.Workload{orphan, captured, byStranger, byMember}) + require.Len(t, deletes, 2) + assert.ElementsMatch(t, []types.UID{"wl-captured", "wl-by-member"}, []types.UID{deletes[0].UID, deletes[1].UID}) + + // Cache lag at capture time: no WorkloadUID, but the member still proves whose Workload it is. + lagged := modelDeploymentReplacementSlot{MemberUIDs: []types.UID{"pod-captured"}} + got := replacementWorkloadDeletes(lagged, []*kueue.Workload{orphan, byStranger, byMember}) + require.Len(t, got, 1) + assert.Equal(t, byMember.UID, got[0].UID) + + assert.Empty(t, replacementWorkloadDeletes(modelDeploymentReplacementSlot{}, []*kueue.Workload{orphan, byStranger}), + "a slot that captured nothing deletes nothing") +} + +// TestCleanReplacementSlot_ASlotWrittenBeforeTheWorkloadWasSeenStillReleasesIt drives the whole +// cleanup rather than the filter. The slot was written while the informer had not yet delivered +// the departing group's Workload, so it holds no WorkloadUID, and its members are already gone. The +// Workload is what Kueue's finalizer on the Pods and the reservation hang off, so leaving it +// standing strands the group holding quota. It must go, and nothing that merely shares the +// namespace may go with it. +func TestCleanReplacementSlot_ASlotWrittenBeforeTheWorkloadWasSeenStillReleasesIt(t *testing.T) { + ctx := context.Background() + md := newRenderDeployment() + + owned := composeReplicaGroup(md, "server", 0, []*core.Pod{{ + ObjectMeta: meta.ObjectMeta{Name: "gone-0", Namespace: md.Namespace, UID: "pod-gone"}, + }}) + owned.Spec.QueueName = "q" + stranger := &kueue.Workload{ObjectMeta: meta.ObjectMeta{ + Name: "someone-elses", Namespace: md.Namespace, UID: "wl-stranger", + OwnerReferences: []meta.OwnerReference{{APIVersion: "v1", Kind: "Pod", Name: "theirs", UID: "pod-theirs"}}, + }} + orphan := &kueue.Workload{ObjectMeta: meta.ObjectMeta{Name: "orphan", Namespace: md.Namespace, UID: "wl-orphan"}} + + cli := newJointClient(owned, stranger, orphan) + r := &ModelDeploymentReconciler{Client: cli, APIReader: cli, Recorder: ctrlrecord.NewFakeRecorder(8)} + slot := modelDeploymentReplacementSlot{Role: "server", Ordinal: 0, MemberUIDs: []types.UID{"pod-gone"}} + require.Empty(t, slot.WorkloadUID, "the case is a slot that captured no Workload") + + require.NoError(t, r.cleanReplacementSlot(ctx, md, slot, nil)) + + left := new(kueue.WorkloadList) + require.NoError(t, cli.List(ctx, left)) + names := make([]string, 0, len(left.Items)) + for i := range left.Items { + names = append(names, left.Items[i].Name) + } + assert.ElementsMatch(t, []string{"someone-elses", "orphan"}, names, + "only the Workload owned by a captured member is released") +} diff --git a/pkg/worker/controllers/worker/model_deployment_retirement.go b/pkg/worker/controllers/worker/model_deployment_retirement.go index bff6fcd7f..9b82b7a40 100644 --- a/pkg/worker/controllers/worker/model_deployment_retirement.go +++ b/pkg/worker/controllers/worker/model_deployment_retirement.go @@ -928,7 +928,28 @@ func (r *ModelDeploymentReconciler) observeModelDeploymentRetirementDrained( // The disaggregation shape is asked only where the spec can still answer it. A role the spec // no longer declares cannot be read, and an absent answer is not a disaggregated one: the // replica's own Pods say what they run, and that is what the drain reader needs anyway. - if role != nil && modelDeploymentRoleExternalDP(md, role) { + // + // THE MEMBERS ANSWER, NOT THE ROLE. The role describes the replica about to replace this one, so + // a flag added to the role would refuse to drain a leader-served replica that is still running a + // supported command, and clearing it would let a replica that IS disaggregated through a guard + // written to catch exactly that. What the replica is is on the members. + // + // A COMMAND NO MEMBER CAN SUPPLY IS A HOLD rather than a fallback to the role. The role's answer + // here would be about a group nobody has read, and this guard exists to avoid deleting live + // traffic on a shape it has not established. + externalDP, renderReason := modelDeploymentRetirementRenderedExternalDP(md, plan.Target.Members) + if renderReason != "" { + return renderReason, false + } + // A member whose command cannot be read holds; the role is not consulted here at all. + if len(plan.Target.Members) == 0 { + return "the replica has no members to read", false + } + if !externalDP.known { + return "a member carries no engine container to read, so the replica's own command line could " + + "not be established", false + } + if externalDP.answer { return "a disaggregated replica has no verifiable decoder-side release", false } @@ -937,7 +958,7 @@ func (r *ModelDeploymentReconciler) observeModelDeploymentRetirementDrained( // serve no metrics endpoint; reading them would hold the operation on gauges that do not exist. // The set the reservation bound is untouched, so the delete preconditions still cover every // member this operation admitted. - answering, reason := r.modelDeploymentRetirementDrainMembers(md, role, plan) + answering, reason := r.modelDeploymentRetirementDrainMembers(md, plan) if reason != "" { return reason, false } @@ -962,22 +983,49 @@ func (r *ModelDeploymentReconciler) observeModelDeploymentRetirementDrained( return "", true } -// modelDeploymentRetirementAnsweringShape classifies the replica from what the drain actually holds, -// which is the member Pods and, when the spec still names it, the role. -// -// THE ROLE IS AN INPUT AND NOT A REQUIREMENT. A removal the spec has already made often concerns a -// role it no longer declares, so requiring one would make that case the one the protocol can never -// finish. Everything the role would have said is either also on the Pods or already in hand: the -// balance shape is read from the member's own rendered command by the same helper the reader uses, -// and the size is the count of the members the reservation bound. +// modelDeploymentRetirementRenderedExternalDP reads whether a replica is disaggregated from every +// member's own rendered command. // -// WHERE BOTH SOURCES EXIST THEY MUST AGREE. The role's own book and the Pods it rendered should say -// the same thing, and a replica where they do not is one this operator cannot reason about; naming a -// winner between them would be picking the reading that happens to let the deletion proceed. +// EVERY MEMBER IS ASKED. A member whose engine container is not there has not been read, and a +// replica is not classifiable from the members that happened to be legible. The answer is reported +// alongside whether every member gave one, because callers hold on the missing case. +func modelDeploymentRetirementRenderedExternalDP( + md *workercore.ModelDeployment, members []*core.Pod, +) (modelDeploymentRetirementCommand, string) { + var answer modelDeploymentRetirementCommand + var missing bool + for _, member := range members { + externalDP, known := modelDeploymentRetirementMemberExternalDP(md, member) + if !known { + missing = true + + continue + } + if answer.known && answer.answer != externalDP { + return modelDeploymentRetirementCommand{}, "the replica's members disagree on whether it " + + "is disaggregated, so its shape could not be established" + } + answer = modelDeploymentRetirementCommand{answer: externalDP, known: true} + } + // ONE UNREAD MEMBER MAKES THE WHOLE REPLICA UNREADABLE, and a later readable member does not + // repair that: the members this operator could not read are still the ones it would delete. + answer.known = answer.known && !missing + + return answer, "" +} + +// modelDeploymentRetirementCommand is what the members said, and whether all of them said it. +type modelDeploymentRetirementCommand struct { + answer bool + known bool +} + +// modelDeploymentRetirementAnsweringShape classifies the replica from what the drain actually holds: +// the member Pods. func modelDeploymentRetirementAnsweringShape( - md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, members []*core.Pod, + md *workercore.ModelDeployment, members []*core.Pod, ) (modelDeploymentAnsweringShape, string) { - externalDP, size, reason := modelDeploymentRetirementReplicaFacts(md, role, members) + externalDP, size, reason := modelDeploymentRetirementReplicaFacts(md, members) if reason != "" { return modelDeploymentAnsweringUnknown, reason } @@ -986,66 +1034,37 @@ func modelDeploymentRetirementAnsweringShape( } // modelDeploymentRetirementReplicaFacts are the two facts the shape is derived from, or the reason -// they could not be established. -// -// THE POD IS ITS OWN SECOND SOURCE for both facts, and where the role is present both are read and -// required to agree. The balance shape comes from the member's rendered command through the same -// helper the drain reader uses, so there is one answer to "is this replica disaggregated" in the tree -// rather than two that can drift. The size is the count of the members the reservation actually bound, -// which is what the drain can read and what the leader search below then runs over. +// they could not be established. Both are read from the members; a replica no member can answer for +// is refused rather than answered from the role. func modelDeploymentRetirementReplicaFacts( - md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, members []*core.Pod, + md *workercore.ModelDeployment, members []*core.Pod, ) (externalDP bool, size int, reason string) { size = len(members) if size == 0 { return false, 0, "the replica has no members to read" } - // The members carry the shape in their own rendered commands, and a replica whose members - // disagree is not one this operator can classify. A member with no engine container to read has - // no opinion, and an absent opinion is not a disagreement; that member is held later, by the - // reader, with the reason that says which container was missing. - podExternalDP, podReadable := modelDeploymentRetirementMemberExternalDP(md, members[0]) - if podReadable { - for _, member := range members[1:] { - other, ok := modelDeploymentRetirementMemberExternalDP(md, member) - if !ok { - continue - } - if other != podExternalDP { - return false, 0, "the replica's members disagree on whether it is disaggregated, " + - "so its shape could not be established" - } - } + // Every member must answer; skipping one would classify the replica from the members that + // happened to be legible. + command, cmdReason := modelDeploymentRetirementRenderedExternalDP(md, members) + if cmdReason != "" { + return false, 0, cmdReason } - - // A role the spec no longer declares is the ordinary case for a dropped role, and it carries no - // facts at all, so the members are all there is. A replica whose own rendered command cannot be - // read then has nothing to classify it by, and that is a hold rather than a guess. - if role == nil { - if !podReadable { - return false, 0, "the role is no longer declared and a member carries no engine " + - "container to read, so the replica's shape could not be established" - } - - return podExternalDP, size, "" + if !command.known { + return false, 0, "a member carries no engine container to read, so the replica's own command " + + "line could not be established" } - // Where the role can answer, it answers, and the members are the check on it rather than the - // source. The two should say the same thing, and naming a winner between them when they do not - // would be picking whichever reading lets the deletion proceed. - roleExternalDP := modelDeploymentRoleExternalDP(md, role) - if podReadable && roleExternalDP != podExternalDP { - return false, 0, "the role and the replica's own rendered command disagree on whether it is " + - "disaggregated, so its shape could not be established" - } - if declared := modelDeploymentRoleSize(role); declared != size { - return false, 0, fmt.Sprintf( - "the role declares %d members and the replica holds %d, so its shape could not be "+ - "established", declared, size) + // The replica is measured against its own declared shape, not the role: a role being edited + // describes the replica about to be built. Seats are included because the leader search below + // counts only seat zero, so a group of three declaring three with seats 0, 1 and 1 has a leader + // and would otherwise drain. + shape := modelDeploymentDeployedReplicaShape(modelDeploymentReplicaView{Members: members}) + if !shape.modelDeploymentReplicaIsWhole() { + return false, 0, shape.Reason } - return roleExternalDP, size, "" + return command.answer, size, "" } // modelDeploymentRetirementMemberExternalDP reads one member's own balance shape, and reports false @@ -1072,15 +1091,15 @@ func modelDeploymentRetirementMemberExternalDP( // report success with a member still running. What the drain reads is a measurement, and a // measurement is this function's business alone. // -// THE CLASSIFIER'S OWN REASON IS THE HOLD REASON. A replica whose members disagree, whose role -// declares a size it does not have, or whose command cannot be read at all, is a different +// THE CLASSIFIER'S OWN REASON IS THE HOLD REASON. A replica whose members disagree, whose members +// do not match their declared total, or whose command cannot be read at all, is a different // operator problem from one another, and collapsing them into a single "shape could not be // established" would leave the reservation naming a symptom with no cause behind it. func (r *ModelDeploymentReconciler) modelDeploymentRetirementDrainMembers( - md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, + md *workercore.ModelDeployment, plan *modelDeploymentRetirementPlan, ) ([]*core.Pod, string) { - shape, reason := modelDeploymentRetirementAnsweringShape(md, role, plan.Target.Members) + shape, reason := modelDeploymentRetirementAnsweringShape(md, plan.Target.Members) if reason != "" { return nil, reason } diff --git a/pkg/worker/controllers/worker/model_deployment_retirement_release_test.go b/pkg/worker/controllers/worker/model_deployment_retirement_release_test.go index 1ae0efe49..fd48c386c 100644 --- a/pkg/worker/controllers/worker/model_deployment_retirement_release_test.go +++ b/pkg/worker/controllers/worker/model_deployment_retirement_release_test.go @@ -71,18 +71,29 @@ func releaseHolder(t *testing.T, mutate ...func(*core.Pod)) *core.Pod { ObjectMeta: meta.ObjectMeta{Name: "qwen-server-one", Namespace: "team-a", UID: "member-1"}, Spec: core.PodSpec{ NodeName: releaseNode, - Containers: []core.Container{{ - Name: "vllm", - Image: "vllm/vllm-openai:v0.25.1", - // The holder asks for the card it is recorded as holding. A recorded claim with no - // matching request would be a record this allocator never wrote. - Resources: core.ResourceRequirements{ - Requests: core.ResourceList{ - nodefeature.GetAcceleratableResourceName( - nodefeature.ManufacturerNVIDIA, workercore.DeviceAllocationModeExclusive): resource.MustParse("1"), + // THE ENGINE CONTAINER IS THE FIRST ONE, because it is the one the retirement guards read + // and a holder carrying none describes a Pod this operator never rendered. The card + // request lives on the second container so both facts stay on the same Pod. + Containers: []core.Container{ + { + Name: modelDeploymentMainContainerName, + Image: "vllm/vllm-openai:v0.25.1", + Command: shapePlainCommand, + }, + { + Name: "vllm", + Image: "vllm/vllm-openai:v0.25.1", + // The holder asks for the card it is recorded as holding. A recorded claim with no + // matching request would be a record this allocator never wrote. + Resources: core.ResourceRequirements{ + Requests: core.ResourceList{ + nodefeature.GetAcceleratableResourceName( + nodefeature.ManufacturerNVIDIA, + workercore.DeviceAllocationModeExclusive): resource.MustParse("1"), + }, }, }, - }}, + }, }, Status: core.PodStatus{Phase: core.PodRunning, PodIP: "10.0.8.1"}, } @@ -718,6 +729,12 @@ func TestModelDeploymentRetirementRelease_EmptyWorkloadUIDHolds(t *testing.T) { // cannot be shown to be anything but an object that still owns the target's member. unidentified := releaseWorkload("", "member-1", "qwen-server-one") require.NoError(t, scenario.cli.Create(context.Background(), unidentified)) + // Corrupt the evidence explicitly after the fixture's server assigned a create identity. + unidentified.UID = "" + require.NoError(t, scenario.cli.Update(context.Background(), unidentified)) + observed := new(kueue.Workload) + require.NoError(t, scenario.cli.Get(context.Background(), ctrlcli.ObjectKeyFromObject(unidentified), observed)) + require.Empty(t, observed.UID, "the negative control presents unidentifiable evidence") scenario.settleLedger() plan, completed := scenario.settle() diff --git a/pkg/worker/controllers/worker/model_deployment_retirement_test.go b/pkg/worker/controllers/worker/model_deployment_retirement_test.go index 8107d3b63..cf0a5f9c0 100644 --- a/pkg/worker/controllers/worker/model_deployment_retirement_test.go +++ b/pkg/worker/controllers/worker/model_deployment_retirement_test.go @@ -130,6 +130,25 @@ func retirementDeployment(mutate ...func(*workercore.ModelDeployment)) *workerco } // retirementReservation builds the persisted operation the FSM re-enters at. +// renderedMember is healthPod carrying the engine container the render writes. +// +// The retirement guards read each member's own rendered command, so a fixture without a container +// describes a Pod this operator never rendered. A command may be given; the default is the plain +// leader-served one the render emits for an ordinary role. +func renderedMember(role string, ordinal, member int, ready *bool, uid string, command ...string) core.Pod { + pod := healthPod(role, ordinal, member, ready, uid) + if len(command) == 0 { + command = shapePlainCommand + } + pod.Spec.Containers = []core.Container{{ + Name: modelDeploymentMainContainerName, + Image: "vllm/vllm-openai:v0.29.0", + Command: command, + }} + + return pod +} + func retirementReservation( state workercore.ModelDeploymentRetirementState, role string, ordinal int32, uids []string, mutate ...func(*workercore.ModelDeploymentRetirementStatus), @@ -250,7 +269,7 @@ func TestProductionDrainReaderHoldsUntilTheTransportLands(t *testing.T) { // naive reader fails: it scraped two of the four SGLang gauges, summed them, and reported a zero. func TestDrainNeverCountsAnIncompleteReadAsIdle(t *testing.T) { md := retirementDeployment() - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} testCases := []struct { name string @@ -367,7 +386,7 @@ func TestDrainNeedsTwoConsecutiveCompleteZeros(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { md := retirementDeployment() - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} md = reserve(md, retirementReservation( workercore.ModelDeploymentRetirementStateDraining, "server", 1, []string{"member-1"})) // The member is seeded, because the release capture reads the target back from the @@ -427,7 +446,7 @@ func TestDrainHoldsOnAnythingThatIsNotAnObservation(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { md := retirementDeployment() - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} md = reserve(md, retirementReservation( workercore.ModelDeploymentRetirementStateDraining, "server", 1, []string{"member-1"})) cli := newModelDeploymentClient(md.DeepCopy()) @@ -452,12 +471,15 @@ func TestDrainHoldsOnAnythingThatIsNotAnObservation(t *testing.T) { // that work while the decoder that will serve it reports none. Reading the prefill gauges as the // decoder-side release would delete a decode cluster in the middle of a prefill. func TestDisaggregatedRetirementIsRefusedRatherThanMeasuredByThePrefiller(t *testing.T) { - // The shape is declared the way the tree reads it: an external load-balance argument on the - // role, which is what makes the group disaggregated rather than merely multi-rank. + // THE MEMBER CARRIES THE ARGUMENT, which is what the refusal now reads. The role declares the + // same thing because a rendered replica and its role agree, and declaring it on the role alone + // is no longer sufficient to refuse. md := retirementDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ExtraArgs = []string{"--data-parallel-external-lb"} }) - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{ + renderedMember("server", 1, 0, healthBool(true), "member-1", shapeExternalDPCommand...), + } md = reserve(md, retirementReservation( workercore.ModelDeploymentRetirementStateDraining, "server", 1, []string{"member-1"})) cli := newModelDeploymentClient(md.DeepCopy()) @@ -477,7 +499,7 @@ func TestDisaggregatedRetirementIsRefusedRatherThanMeasuredByThePrefiller(t *tes // does: the operation stops with every member, the Workload and the capacity still standing. func TestBudgetExpiryAbortsAndRetains(t *testing.T) { md := retirementDeployment() - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} exhausted := retirementReservation( workercore.ModelDeploymentRetirementStateDraining, "server", 1, []string{"member-1"}, func(r *workercore.ModelDeploymentRetirementStatus) { @@ -511,7 +533,7 @@ func TestAnAbortedReservationIsNotRetriedUnchanged(t *testing.T) { md := retirementDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 1 }) - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} md = reserve(md, retirementReservation( workercore.ModelDeploymentRetirementStateAborted, "server", 1, []string{"member-1"}, func(r *workercore.ModelDeploymentRetirementStatus) { r.Reason = "the drain budget expired" }, @@ -537,7 +559,7 @@ func TestAnAbortedReservationIsNotRetriedUnchanged(t *testing.T) { // deployment that restarts often would never finish at all. func TestRestartResumesFromThePersistedState(t *testing.T) { md := retirementRouterBacked(retirementDeployment()) - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} for i := range pods { pods[i].Status.Phase = core.PodRunning pods[i].Status.PodIP = "10.0.2.1" @@ -1489,7 +1511,7 @@ func TestAProtocolThatIsNotAbortedIgnoresItsOwnRetryDirective(t *testing.T) { }) md = reserve(md, retirementReservation( workercore.ModelDeploymentRetirementStateWithdrawing, "server", 1, []string{"member-1"})) - pods := []core.Pod{healthPod("server", 1, 0, healthBool(true), "member-1")} + pods := []core.Pod{renderedMember("server", 1, 0, healthBool(true), "member-1")} cli := newModelDeploymentClient(md.DeepCopy()) r := holdReconciler(cli, &scriptedDrainReader{}, time.Now()) @@ -2545,7 +2567,7 @@ func newRetirementReleaseFixture( keeper.Status.Phase = core.PodRunning keeper.Status.PodIP = "10.0.7.2" - workload := admittedReplicaWorkload(target, true) + workload := admittedReplicaWorkload([]*core.Pod{target}, true) cli := newModelDeploymentClient( md, newRenderInstanceType(), target, keeper, router, workload) r := retirementRouterReconciler(cli, &scriptedDrainReader{}, time.Now(), []core.Pod{*target, *keeper}) diff --git a/pkg/worker/controllers/worker/model_deployment_routing_test.go b/pkg/worker/controllers/worker/model_deployment_routing_test.go new file mode 100644 index 000000000..4bdd2790b --- /dev/null +++ b/pkg/worker/controllers/worker/model_deployment_routing_test.go @@ -0,0 +1,504 @@ +package worker + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + core "k8s.io/api/core/v1" + kerrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/types" + ctrlrecord "k8s.io/client-go/tools/record" + ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" + ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + workercore "gpustack.ai/gpustack/api/worker/v1alpha1" +) + +// Routing membership across a command-mode change, in both directions. +// +// One selector cannot express the union a transition needs. A managed replica answers only when +// its engine is qualified, while a take-over replica answers whenever it is Ready -- the rule it +// has always had. Selecting the managed replicas while the take-over ones are still serving would +// drop them from the pool the moment the first replacement was admitted, and selecting every Pod +// would hand back managed replicas whose engine was never qualified. So a private routing label +// carries membership, the Service selector names it while the two modes coexist, and each replica +// keeps the rule its own deployed mode has always had. + +// settleRouting brings a fixture to a state where every replica is Ready and, for a managed role, +// qualified -- the state a running deployment is in and the one a transition starts from. +func settleRouting(t *testing.T, f *lifecycleFixture, replicas int) { + t.Helper() + + f.pass(false) + f.admit() + for ordinal := range replicas { + f.setReady(ordinal, true) + f.setEligible(ordinal, true) + } + f.pass(false) +} + +// TestRouting_ManagedToTakeoverKeepsTheOldManagedReplicas covers the direction where the role +// starts managed and becomes take-over. The old managed replicas are not withdrawn when the first +// replacement appears: they keep serving under their existing qualification until their own turn +// comes, and one that is not qualified is not selected on the strength of being old. +func TestRouting_ManagedToTakeoverKeepsTheOldManagedReplicas(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + assert.False(t, f.routesByMembership(), + "a role whose replicas are all managed does not route on the membership label") + assert.Equal(t, map[int]string{0: modelDeploymentAPIAnsweringValue, 1: modelDeploymentAPIAnsweringValue}, + f.answering(), "and both qualified managed replicas answer") + + // The role takes over its command line. The highest ordinal turns over first. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + + f.pass(false) + f.admit() + f.pass(false) + slot := f.slotMust("server") + f.setReady(slot.Ordinal, true) + f.pass(false) + + assert.True(t, f.routesByMembership(), + "the two modes cannot be selected by one rule, so the selector names the routing label") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[0], + "the old managed replica keeps answering while it is still serving") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[slot.Ordinal], + "and the take-over replica built beside it answers under the rule it always had") + + // The negative control: an old managed replica whose engine is not answering does not keep an + // endpoint merely for being the one that is still old. The qualification is re-derived from the + // engine every pass rather than read back off the Pod, so the control is taken on the member's + // readiness, which every eligibility rule already treats as a definite fault. + f.setReady(0, false) + f.pass(false) + assert.Empty(t, f.answering()[0], + "an old managed replica that is not ready is not selected while the transition is on") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[slot.Ordinal], + "and the take-over replica beside it is untouched by that decision") + + // Once the transition is over, the label leaves the selector again: a homogeneous role keeps + // its ordinary routing. + for range 8 { + if allAtImage(f, "vllm/vllm-openai:v0.25.1", 2) && !f.routesByMembership() { + break + } + f.pass(false) + f.admit() + } + assert.False(t, f.routesByMembership(), + "and ordinary homogeneous routing is unchanged once no replica of the old mode is left") +} + +// TestRouting_TakeoverToManagedKeepsTheOldTakeoverReplicas covers the other direction. A take-over +// replica answers whenever it is Ready, so the old ones stay selectable for as long as they run; +// the new managed ones answer only once their engine is qualified. +func TestRouting_TakeoverToManagedKeepsTheOldTakeoverReplicas(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + assert.False(t, f.routesByMembership(), + "a role whose replicas are all take-over does not route on the membership label either") + assert.Equal(t, map[int]string{0: modelDeploymentAPIAnsweringValue, 1: modelDeploymentAPIAnsweringValue}, + f.answering(), "and both take-over replicas answer") + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = nil + require.NoError(t, f.cli.Update(context.Background(), edited)) + + f.pass(false) + f.admit() + f.pass(false) + slot := f.slotMust("server") + + // The negative control, taken while the old take-over replica is still serving: the managed + // replacement is not answering yet, and neither managed rule nor take-over rule may select it. + f.setReady(slot.Ordinal, false) + f.pass(false) + + assert.True(t, f.routesByMembership(), "the two modes coexist, so the selector names the label") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[0], + "the old take-over replica is retained rather than withdrawn when a replacement appears") + assert.Empty(t, f.answering()[slot.Ordinal], + "while a managed replica that is not yet qualified answers nothing") + + // Once it answers it is selected, and the old take-over replica keeps answering beside it. + f.setReady(slot.Ordinal, true) + f.setEligible(slot.Ordinal, true) + f.pass(false) + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[slot.Ordinal], + "the managed replacement answers once its engine is qualified") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[0], + "and the old take-over replica is still selectable until its own turn") +} + +// TestRouting_TheReplicaBeingReplacedIsNotSelected covers the exclusion the transition needs: a +// replica on its way out must stop receiving traffic before it is deleted, whichever mode it is in. +func TestRouting_TheReplicaBeingReplacedIsNotSelected(t *testing.T) { + for _, command := range [][]string{nil, {"vllm", "serve", "qwen", "--port", "8000"}} { + name := "managed" + if command != nil { + name = "takeover" + } + t.Run(name, func(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].Command = command + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot, held := f.slot("server") + require.True(t, held) + assert.Empty(t, f.answering()[slot.Ordinal], + "the replica this pass selected stops answering before it is deleted") + assert.Equal(t, modelDeploymentAPIAnsweringValue, f.answering()[1-slot.Ordinal], + "and the replica beside it is unaffected") + }) + } +} + +// TestRouting_AnExternalDataParallelRoleAnswersOnItsRanksOnly covers the shape the transition has +// to preserve on a role whose members are ranks of one engine: the ranks answer under the existing +// rules, and a withdrawn leader answers nothing. +func TestRouting_AnExternalDataParallelRoleAnswersOnItsRanksOnly(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].ExtraArgs = []string{ + "--data-parallel-size", "2", + "--data-parallel-rank", "$(GPUSTACK_MEMBER_INDEX)", + } + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + // AN EXTERNAL DATA-PARALLEL REPLICA ANSWERS ON EVERY RANK. The shape is read off the replicas the + // cluster is running, so both members of a two-rank group answer -- which is what the rank + // arrangement means, and what reading the rank from the spec instead would quietly withdraw. + for ordinal := range 2 { + members := f.live("server", ordinal) + require.Len(t, members, 2) + for _, member := range members { + assert.Equal(t, modelDeploymentAPIAnsweringValue, + member.Labels[modelDeploymentLabelKeyAPIAnswering], + "ordinal %d member %s answers: every rank of an External-DP group serves", + ordinal, member.Labels[modelDeploymentMemberIndexLabel]) + } + } + + // The transition itself: a replica is replaced while its sibling keeps answering on both ranks. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot := f.slotMust("server") + assert.Empty(t, f.answering()[slot.Ordinal], + "the replica under replacement stops answering, ranks included") + for _, member := range f.live("server", 1-slot.Ordinal) { + assert.Equal(t, modelDeploymentAPIAnsweringValue, + member.Labels[modelDeploymentLabelKeyAPIAnswering], + "while the untouched replica keeps every rank answering") + } +} + +// TestRouting_ALeaderServedReplicaAnswersOnItsDeployedSize covers the same rule for the other +// direction of a size edit. A replica still running four members answers on its leader, because +// that is the shape it was built at -- and the follower is withdrawn the moment the spec asks for +// one member, which is capacity this operator never chose to give up. +func TestRouting_ALeaderServedReplicaAnswersOnItsDeployedSize(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + md.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + // A take-over replica answers under the legacy rule, so the shape is the only thing deciding + // which of its members are selected. + leaders := 0 + for _, member := range f.live("server", 0) { + if member.Labels[modelDeploymentLabelKeyAPIAnswering] == modelDeploymentAPIAnsweringValue { + leaders++ + } + } + assert.Equal(t, 1, leaders, "a leader-served replica of two members answers on its leader") + + // The size drops to one. The running replica is not the one being replaced, so it keeps + // answering on its own shape until its own turn comes. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].ReplicaSize = 1 + require.NoError(t, f.cli.Update(context.Background(), edited)) + f.pass(false) + + slot, held := f.slot("server") + require.True(t, held) + sibling := 1 - slot.Ordinal + stillTwo := 0 + for _, member := range f.live("server", sibling) { + if member.Labels[modelDeploymentLabelKeyAPIAnswering] == modelDeploymentAPIAnsweringValue { + stillTwo++ + } + } + assert.Equal(t, 1, stillTwo, + "the replica the spec no longer sizes is still leader-served, so its follower stays unselected") +} + +// TestRouting_TheLabelWriteIsBoundToTheObservedPod covers the window between reading a member and +// writing its membership. A name is not an identity: a take-over replica that is deleted and +// recreated as a managed one under the same name must not inherit the membership the controller read +// off the replica that is gone. +func TestRouting_TheLabelWriteIsBoundToTheObservedPod(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 1 + md.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + f.setReady(0, true) + f.pass(false) + require.Equal(t, map[int]string{0: modelDeploymentAPIAnsweringValue}, f.answering(), + "the take-over replica answers under the legacy rule") + + // The member goes unready, so this pass has a write to make: it withdraws the membership it + // carries. The replacement arrives in the window between the observation and that write. + takeover := f.live("server", 0) + require.Len(t, takeover, 1) + f.setReady(0, false) + + managed := takeover[0].DeepCopy() + managed.UID = types.UID("pod-the-managed-one") + delete(managed.Annotations, "prometheus.io/port") + delete(managed.Labels, modelDeploymentLabelKeyAPIAnswering) + managed.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} + + swapped := new(bool) + r := &ModelDeploymentReconciler{ + Client: f.interceptReplace(t, takeover[0], managed, swapped), + APIReader: f.cli, Recorder: ctrlrecord.NewFakeRecorder(64), + } + _, err := reconcileModelDeploymentWith(t, r) + require.True(t, *swapped, "the member was replaced in the window under test") + if err != nil { + assert.True(t, kerrors.IsConflict(err), + "the write is refused as a conflict rather than landing on a stranger: %v", err) + } + + survivor := new(core.Pod) + require.NoError(t, f.cli.Get(context.Background(), + ctrlcli.ObjectKeyFromObject(managed), survivor)) + assert.NotContains(t, survivor.Labels, modelDeploymentLabelKeyAPIAnswering, + "and the replacement carries no membership read off the replica it replaced") +} + +// interceptReplace returns a client that swaps one Pod for another the first time that Pod is +// patched, which is the window a stale cached object is written through. +func (f *lifecycleFixture) interceptReplace( + t *testing.T, old, replacement *core.Pod, swapped *bool, +) ctrlcli.Client { + t.Helper() + + return ctrlinterceptor.NewClient(f.cli, ctrlinterceptor.Funcs{ + Patch: func( + ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + patch ctrlcli.Patch, opts ...ctrlcli.PatchOption, + ) error { + pod, ok := obj.(*core.Pod) + if !ok || *swapped || pod.Name != old.Name { + return next.Patch(ctx, obj, patch, opts...) + } + + *swapped = true + // The member holds Kueue's finalizer, so the delete is accepted and the object stays + // until it is released -- which is the one thing this window is not about. + held := new(core.Pod) + require.NoError(t, f.cli.Get(ctx, ctrlcli.ObjectKeyFromObject(old), held)) + held.Finalizers = nil + require.NoError(t, f.cli.Update(ctx, held)) + require.NoError(t, f.cli.Delete(ctx, held)) + arriving := replacement.DeepCopy() + arriving.Finalizers = nil + arriving.ResourceVersion = "" + require.NoError(t, f.cli.Create(ctx, arriving)) + + return next.Patch(ctx, obj, patch, opts...) + }, + }) +} + +// A Ready managed replica with revoked qualification stays unselected during a command change. +// Drive the membership writer and compare complete Service selectors. +// Qualified leaders remain selected; qualification is injected at the writer's input. +func TestRouting_ARevokedManagedReplicaKeepsTheGateWhileTheRoleTurnsOver(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + f.pass(false) + f.admit() + require.Len(t, f.live("server", 0), 2, "setup requires a replica of two members") + f.setReady(0, true) + managed := f.live("server", 0) + require.Len(t, managed, 2) + + // The role is turning over, and its replicas have not. One replica is still the managed pair the + // operator built, and that pair keeps the gate. + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + role := &getModelDeployment(t, f.cli).Spec.Roles[0] + require.NotEmpty(t, role.Command, "setup requires the role to have gained a command line") + + require.True(t, modelDeploymentReplicaRunsCommand(managed), + "setup requires the replica still to be one this operator built") + + // The gate is asked of the mode the replica runs. Asking it of the role instead is the defect. + assert.False(t, modelDeploymentEligibilitySelectorActive(md, role, nil, true), + "the role's own command line would take the gate away") + assert.True(t, modelDeploymentEligibilitySelectorActive(md, + modelDeploymentEffectiveRoleForEligibility(role, managed), nil, true), + "while a replica still running the operator's command line keeps it") + + // And the writer publishes that gate: the role Service keeps the term it was narrowed by while + // a managed replica of the role is still standing. + // The published Services are read against the deployment as the cluster now holds it, with the + // eligibility condition decided, which is what a narrowed selector means. + published := getModelDeployment(t, f.cli) + ModelDeploymentConditionEndpointEligibility.True(published, + modelDeploymentReasonEndpointsQualified, "every group qualified") + require.NoError(t, f.cli.Status().Update(context.Background(), published)) + + r := &ModelDeploymentReconciler{Client: f.cli, APIReader: f.cli} + cases := []struct { + name string + state modelDeploymentGroupForwardState + verdict modelDeploymentLegVerdict + want bool + }{ + {name: "revoked", state: modelDeploymentGroupForwardFailed, verdict: modelDeploymentLegFailed}, + {name: "qualified", state: modelDeploymentGroupForwardVerified, verdict: modelDeploymentLegVerified, want: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + byMember := make(map[types.UID]modelDeploymentInstanceQualification, len(managed)) + for _, member := range managed { + byMember[member.UID] = modelDeploymentInstanceQualification{ + GroupForward: modelDeploymentGroupForward{State: tc.state}, + Legs: []modelDeploymentQualificationLeg{{ + Name: modelDeploymentLegGroupForward, Verdict: tc.verdict, + }}, + } + } + require.NoError(t, r.convergeModelDeploymentEndpointEligibility(context.Background(), published, + replicaPods(t, f.cli), byMember, nil)) + actual := replicaPods(t, f.cli) + require.NoError(t, r.syncModelDeploymentService(context.Background(), published, true, actual)) + for _, name := range []string{md.Name, md.Name + "-server"} { + svc := new(core.Service) + require.NoError(t, f.cli.Get(context.Background(), + ctrlcli.ObjectKey{Namespace: md.Namespace, Name: name}, svc)) + for _, member := range f.live("server", 0) { + assert.Equal(t, tc.want && modelDeploymentAnsweringMemberLeader(member), + labels.SelectorFromSet(svc.Spec.Selector).Matches(labels.Set(member.Labels)), + "%s selects only qualified answering members", name) + } + } + }) + } + + // A Ready, revoked, managed member answers nothing under that gate, whichever mode it runs. + view := modelDeploymentReplicaView{Members: managed} + for _, member := range managed { + require.True(t, podIsReady(member), + "setup requires the member to stay Ready: readiness is not what withdrew it") + assert.False(t, modelDeploymentPodAnswersAPI(member, view, md.Spec.Engine.Name, + false, true, nil), + "a managed member whose qualification was revoked answers nothing while the gate is active") + if modelDeploymentAnsweringMemberLeader(member) { + assert.True(t, modelDeploymentPodAnswersAPI(member, view, md.Spec.Engine.Name, + false, false, nil), + "and the legacy rule is kept only for a role whose gate was never activated") + } + } +} + +// TestRouting_APeerServiceIsNeverNarrowedByTheTransition covers the Services a transition must +// leave alone. A member's peers are addressed by DNS through the headless Services, and narrowing +// those would break the collective that is still running. +func TestRouting_APeerServiceIsNeverNarrowedByTheTransition(t *testing.T) { + md := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 2 + md.Spec.Roles[0].ReplicaSize = 2 + }) + f := newLifecycleFixture(t, md) + settleRouting(t, f, 2) + + headless := new(core.ServiceList) + require.NoError(t, f.cli.List(context.Background(), headless, ctrlcli.InNamespace("team-a"))) + before := make(map[string]map[string]string, len(headless.Items)) + for i := range headless.Items { + before[headless.Items[i].Name] = headless.Items[i].Spec.Selector + } + require.NotEmpty(t, before) + + edited := getModelDeployment(t, f.cli) + edited.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen", "--port", "8000"} + require.NoError(t, f.cli.Update(context.Background(), edited)) + for range 3 { + f.pass(false) + f.admit() + f.pass(false) + if f.routesByMembership() { + break + } + } + require.True(t, f.routesByMembership(), "the two modes coexist at the point under test") + + // The front Services are the ones the transition re-points, and the peer Services are not. A + // member's peers are addressed by DNS through the headless Services, and narrowing those would + // break the collective that is still running. + front := map[string]bool{f.md.Name: true} + for i := range f.md.Spec.Roles { + front[f.md.Name+"-"+f.md.Spec.Roles[i].Name] = true + } + + after := new(core.ServiceList) + require.NoError(t, f.cli.List(context.Background(), after, ctrlcli.InNamespace("team-a"))) + peers := 0 + for i := range after.Items { + require.Contains(t, before, after.Items[i].Name, "a Service appeared that no pass rendered") + if front[after.Items[i].Name] { + assert.Contains(t, after.Items[i].Spec.Selector, modelDeploymentLabelKeyAPIAnswering, + "%s names the routing label while the two modes coexist", after.Items[i].Name) + + continue + } + peers++ + assert.Equal(t, before[after.Items[i].Name], after.Items[i].Spec.Selector, + "%s keeps the selector the collective resolves its peers through", after.Items[i].Name) + } + assert.Positive(t, peers, "the deployment has peer Services to leave alone") +} diff --git a/pkg/worker/controllers/worker/model_deployment_service.go b/pkg/worker/controllers/worker/model_deployment_service.go index 52a2c339b..79aec1760 100644 --- a/pkg/worker/controllers/worker/model_deployment_service.go +++ b/pkg/worker/controllers/worker/model_deployment_service.go @@ -8,6 +8,7 @@ import ( core "k8s.io/api/core/v1" meta "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" "gpustack.ai/gpustack/pkg/kubemeta" @@ -26,26 +27,41 @@ import ( // entry read as "not known to be Ascend": the KV event ports are part of the render only where the // replicas behind the Service publish, and that decision is the connector's. func renderModelDeploymentServices( - md *workercore.ModelDeployment, manufacturers map[string]string, + md *workercore.ModelDeployment, manufacturers map[string]string, pods []core.Pod, ) []*core.Service { svcs := make([]*core.Service, 0, len(md.Spec.Roles)+1) - svcs = append(svcs, renderModelDeploymentService(md)) + svcs = append(svcs, renderModelDeploymentService(md, pods)) for i := range md.Spec.Roles { role := &md.Spec.Roles[i] - svcs = append(svcs, renderModelDeploymentRoleService(md, role, manufacturers)) - - // ONE HEADLESS SERVICE PER REPLICA, AND ONLY ABOVE SIZE ONE. It is what publishes the - // members' names in DNS, so a replica whose only member has nobody to address needs none -- - // rendering one anyway would put an object per replica on the cluster for no consumer. + standing := modelDeploymentReplicaServiceOrdinals(pods, md, role) + svcs = append(svcs, + renderModelDeploymentRoleService(md, role, manufacturers, + modelDeploymentStandingMembers(pods, role, standing), + modelDeploymentNeedsDeployedRouting(pods, md, role))) + + // ONE HEADLESS SERVICE PER REPLICA, AND ONLY WHERE A REPLICA HAS PEERS TO ADDRESS. It is + // what publishes the members' names in DNS, so a replica whose members are alone needs none + // -- rendering one anyway would put an object per replica on the cluster for no consumer. // // They come AFTER the role's own Service in this list, which matters because the caller - // garbage collects by comparing this list against what exists: a replica's Service must be - // derivable from the spec alone, and a scale-down drops the entries its ordinals no longer - // reach. - if modelDeploymentRoleSize(role) == 1 { - continue - } + // garbage collects by comparing this list against what exists. A replica keeps its Service + // while it is desired or while its deployed members still need peer records. + // + // AN ORDINAL GETS ONE WHEN THE DESIRED SIZE SAYS IT WILL HAVE PEERS OR WHILE A MEMBER OF IT + // IS STILL RUNNING. A role edited from four members to one leaves the four-member replica + // serving while its replacement is built, and deleting its Service at that moment takes the + // peer records out from under a running collective: the members cannot resolve each other, + // and a collective that cannot resolve its peers does not fail visibly -- it hangs. So the + // retention is by observed membership and ends when the last such member has gone, one pass + // after the last departure. + ordinals := standing for ordinal := range int(role.Replicas) { + if modelDeploymentRoleSize(role) == 1 { + break + } + ordinals[ordinal] = true + } + for _, ordinal := range slices.Sorted(maps.Keys(ordinals)) { svcs = append(svcs, renderModelDeploymentReplicaService(md, role, ordinal)) } } @@ -53,6 +69,37 @@ func renderModelDeploymentServices( return svcs } +// modelDeploymentReplicaServiceOrdinals returns the ordinals whose running members still need peer +// records, whether or not the desired size would render them. +// +// A TERMINATING MEMBER COUNTS. The Service publishes unready addresses precisely so a member can +// find its peers before it is ready, and deleting the record while a departing member still runs +// is the same loss one instant later. +// +// THE EVIDENCE IS THE GROUP TOTAL THE MEMBER CARRIES, not the member index: the index is written at +// every size, so its presence says a Pod was rendered, not that it has peers. +func modelDeploymentReplicaServiceOrdinals( + pods []core.Pod, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, +) map[int]bool { + ordinals := make(map[int]bool) + for i := range pods { + pod := &pods[i] + if !modelDeploymentOwns(pod, md) || modelDeploymentPodRole(pod) != role.Name { + continue + } + total, err := strconvx.Atoi[int](pod.Annotations[kueuepodconst.GroupTotalCountAnnotation]) + if err != nil || total <= 1 { + // A replica of one member has nobody to address. + continue + } + if ordinal, ok := modelDeploymentPodOrdinal(pod); ok { + ordinals[ordinal] = true + } + } + + return ordinals +} + // renderModelDeploymentReplicaService renders the headless Service one replica's members are // published behind, which is the subdomain each of them names. // @@ -98,10 +145,60 @@ func renderModelDeploymentReplicaService( // configured as a producer and one configured as a consumer, which is the silent wrong answer this // spec's whole kind field exists to prevent. Per-role addressability is what this task adds; a real // front door needs the router, and that is a later spec. -func renderModelDeploymentService(md *workercore.ModelDeployment) *core.Service { +// modelDeploymentEffectiveRoleForEligibility returns the role as the replicas standing in this +// cluster see it. +// +// The gate is about the replicas serving, not about the role the spec states now. A role turned into +// a take-over one while its old managed replicas are still running has not lost the group-forward +// observation for them: their command lines are the ones this operator wrote. Clearing the desired +// command line while any such replica stands keeps the gate, and the selector narrowed by it, from +// being withdrawn on the pass that turned the role over. +func modelDeploymentEffectiveRoleForEligibility( + role *workercore.ModelDeploymentRole, members []*core.Pod, +) *workercore.ModelDeploymentRole { + if !modelDeploymentAnyMemberRunsCommand(members) { + return role + } + effective := *role + effective.Command = nil + + return &effective +} + +// modelDeploymentStandingMembers returns the live members of the replicas of one role that still +// hold a seat, which is what the gate and the membership rules are read from. +func modelDeploymentStandingMembers( + pods []core.Pod, role *workercore.ModelDeploymentRole, standing map[int]bool, +) []*core.Pod { + members := make([]*core.Pod, 0, len(standing)) + for _, view := range modelDeploymentGroupPods(pods, true) { + if modelDeploymentPodRole(view.Members[0]) != role.Name || !standing[view.Ordinal] { + continue + } + members = append(members, view.Members...) + } + + return members +} + +func renderModelDeploymentService( + md *workercore.ModelDeployment, pods []core.Pod, +) *core.Service { svc := renderModelDeploymentServiceFor(md, &md.Spec.Roles[0], md.Name) - modelDeploymentFrontLeadersOnly(svc, &md.Spec.Roles[0]) - if modelDeploymentEligibilitySelectorActive(md, &md.Spec.Roles[0], nil, modelDeploymentEligibilityDecided(md)) { + modelDeploymentFrontLeadersOnly(svc, &md.Spec.Roles[0], + modelDeploymentRoleHasMultiMemberReplica(pods, md, &md.Spec.Roles[0])) + modelDeploymentSelectAnsweringMembers(svc, modelDeploymentNeedsDeployedRouting(pods, md, &md.Spec.Roles[0])) + // ONLY THE FRONT ROLE'S OWN STANDING MEMBERS: the Service routes Roles[0] alone, so a sibling + // role's managed Pod, or a Pod already leaving, must not lift the take-over refusal for it. + var frontStanding []*core.Pod + for _, view := range modelDeploymentGroupPods(pods, true) { + if modelDeploymentPodRole(view.Members[0]) == md.Spec.Roles[0].Name { + frontStanding = append(frontStanding, view.Members...) + } + } + if modelDeploymentEligibilitySelectorActive(md, + modelDeploymentEffectiveRoleForEligibility(&md.Spec.Roles[0], frontStanding), nil, + modelDeploymentEligibilityDecided(md)) { modelDeploymentSelectEligibleEndpoints(svc, md, &md.Spec.Roles[0]) } @@ -120,14 +217,95 @@ func renderModelDeploymentService(md *workercore.ModelDeployment) *core.Service // AT SIZE ONE THE SELECTOR IS LEFT EXACTLY AS IT WAS, and not merely equivalently: a single-Member // replica's Pod carries no member-index label, so adding the term there would select nothing and // empty every Service of every deployment already running. -func modelDeploymentFrontLeadersOnly(svc *core.Service, role *workercore.ModelDeploymentRole) { - if modelDeploymentRoleSize(role) == 1 { +// +// WHILE A ROLE IS MID-EDIT, THE TERMS COME FROM WHAT IS STANDING RATHER THAN FROM THE DESIRED +// SIZE. A replica edited from four members down to one still has four members running, and a +// selector computed from the new size alone would drop the leader term and front all four of them +// -- which is the exact failure above, reappearing for a deployment that was working. So the term +// is applied whenever the desired size is above one OR any actual member of the role carries a +// member index, and dropped only once the last such member has gone. +func modelDeploymentFrontLeadersOnly( + svc *core.Service, role *workercore.ModelDeploymentRole, multiMemberStanding bool, +) { + if modelDeploymentRoleSize(role) == 1 && !multiMemberStanding { return } svc.Spec.Selector[modelDeploymentMemberIndexLabel] = strconvx.Itoa(modelDeploymentLeaderMemberIndex) } +// modelDeploymentNeedsDeployedRouting reports whether desired routing differs from a running member. +// Use observed membership from the first edit, including before a replacement appears. +func modelDeploymentNeedsDeployedRouting( + pods []core.Pod, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, +) bool { + desiredManaged := len(role.Command) == 0 + desiredExternal := desiredManaged && modelDeploymentRoleExternalDP(md, role) + for i := range pods { + pod := &pods[i] + if !modelDeploymentOwns(pod, md) || modelDeploymentPodRole(pod) != role.Name { + continue + } + if pod.DeletionTimestamp != nil { + continue + } + managed, readable := modelDeploymentMemberExecution(pod) + if !readable { + continue + } + if managed != desiredManaged { + return true + } + if !managed { + continue + } + // A member whose parallelism cannot be read says nothing about the routing it was built with. + if external, known := modelDeploymentMembersExternalDP( + []*core.Pod{pod}, md.Spec.Engine.Name); known && external != desiredExternal { + return true + } + } + + return false +} + +// modelDeploymentSelectAnsweringMembers narrows an HTTP Service to the members whose replicas +// answer it under the mode they were built with. +// +// Use the term while a running mode or answering shape differs from the desired configuration. +// Remove it once every member follows the desired routing rule. +func modelDeploymentSelectAnsweringMembers( + svc *core.Service, mixed bool, +) { + if !mixed { + return + } + + svc.Spec.Selector[modelDeploymentLabelKeyAPIAnswering] = modelDeploymentAPIAnsweringValue + delete(svc.Spec.Selector, modelDeploymentLabelKeyEndpointEligible) + delete(svc.Spec.Selector, modelDeploymentMemberIndexLabel) +} + +// modelDeploymentRoleHasMultiMemberReplica reports whether any actual member of a role still +// belongs to a replica built at more than one member, which is what makes it a member rather than +// a whole replica. +func modelDeploymentRoleHasMultiMemberReplica( + pods []core.Pod, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, +) bool { + for i := range pods { + pod := &pods[i] + if !modelDeploymentOwns(pod, md) || modelDeploymentPodRole(pod) != role.Name { + continue + } + if total, err := strconvx.Atoi[int](pod.Annotations[kueuepodconst.GroupTotalCountAnnotation]); err == nil && + total > 1 { + return true + } + } + + return false +} + // modelDeploymentGroupForwardActivatable reports whether this role's group-forward capability // can ever reach Verified or NotApplicable — the two states that activate eligibility. A shape // the operator has not verified (an unlisted engine version, or a prefill/decode role) answers @@ -202,12 +380,23 @@ func modelDeploymentEligibilitySelectorActive( liveSelector[modelDeploymentLabelKeyEndpointEligible] == modelDeploymentEndpointEligibleValue } +// modelDeploymentEligibilityActiveAnnotation retains gate activation while mixed Services use membership. +const modelDeploymentEligibilityActiveAnnotation = "modeldeployment.gpustack.ai/eligibility-active" + // modelDeploymentSelectEligibleEndpoints selects qualified HTTP endpoints. // External roles serve every rank; Internal roles retain their leader restriction. // Headless peer Services never use endpoint eligibility. func modelDeploymentSelectEligibleEndpoints( svc *core.Service, md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, ) { + if svc.Spec.Selector[modelDeploymentLabelKeyAPIAnswering] == modelDeploymentAPIAnsweringValue { + // Membership already combines each deployed mode's qualification and answering shape. + if svc.Annotations == nil { + svc.Annotations = map[string]string{} + } + svc.Annotations[modelDeploymentEligibilityActiveAnnotation] = modelDeploymentEndpointEligibleValue + return + } svc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] = modelDeploymentEndpointEligibleValue // Every qualified External-DP member serves HTTP. The eligibility label already // excludes unhealthy groups, so the legacy leader restriction must not hide ranks. @@ -239,11 +428,14 @@ func modelDeploymentSelectEligibleEndpoints( // two objects away from the field that caused it. func renderModelDeploymentRoleService( md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole, - manufacturers map[string]string, + manufacturers map[string]string, standingMembers []*core.Pod, mixedCommandMode bool, ) *core.Service { svc := renderModelDeploymentServiceFor(md, role, md.Name+"-"+role.Name) - modelDeploymentFrontLeadersOnly(svc, role) - if modelDeploymentEligibilitySelectorActive(md, role, nil, modelDeploymentEligibilityDecided(md)) { + modelDeploymentFrontLeadersOnly(svc, role, len(standingMembers) > 0) + modelDeploymentSelectAnsweringMembers(svc, mixedCommandMode) + if modelDeploymentEligibilitySelectorActive(md, + modelDeploymentEffectiveRoleForEligibility(role, standingMembers), nil, + modelDeploymentEligibilityDecided(md)) { modelDeploymentSelectEligibleEndpoints(svc, md, role) } if modelDeploymentPublishesKVEvents(md, role, manufacturers[role.Name]) { @@ -441,6 +633,18 @@ func modelDeploymentRoleEndpoint( // must not discard — the allocated ClusterIP above all, which every client that resolved it is // still using. func alignModelDeploymentService(actual, expected *core.Service) (changed bool) { + key := modelDeploymentEligibilityActiveAnnotation + if actual.Annotations[key] != expected.Annotations[key] { + if expected.Annotations[key] == "" { + delete(actual.Annotations, key) + } else { + if actual.Annotations == nil { + actual.Annotations = map[string]string{} + } + actual.Annotations[key] = expected.Annotations[key] + } + changed = true + } if !maps.Equal(actual.Spec.Selector, expected.Spec.Selector) { actual.Spec.Selector = expected.Spec.Selector changed = true diff --git a/pkg/worker/controllers/worker/model_deployment_service_test.go b/pkg/worker/controllers/worker/model_deployment_service_test.go index 1127d4e07..9247d58a1 100644 --- a/pkg/worker/controllers/worker/model_deployment_service_test.go +++ b/pkg/worker/controllers/worker/model_deployment_service_test.go @@ -9,10 +9,12 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" core "k8s.io/api/core/v1" + meta "k8s.io/apimachinery/pkg/apis/meta/v1" ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubemeta" "gpustack.ai/gpustack/pkg/systemmeta" "gpustack.ai/gpustack/pkg/worker/kvcache/inject" ) @@ -119,7 +121,7 @@ func TestModelDeploymentService_OnePerRoleBesideTheDeploymentWide(t *testing.T) func TestRenderModelDeploymentService_KVEventPorts(t *testing.T) { md := routedModelDeployment() - services := renderModelDeploymentServices(md, nil) + services := renderModelDeploymentServices(md, nil, nil) require.Len(t, services, 3) portsByName := func(service *core.Service) map[string]int32 { @@ -163,8 +165,8 @@ func TestModelDeploymentService_RemovingARoleRemovesItsService(t *testing.T) { // TestModelDeploymentService_AReplicaServiceIsCreatedAndReclaimedWithItsReplica runs the whole // convergence rather than the renderer, because the question here is about the prune path: a -// headless Service is derived from an ordinal, so scaling down has to reclaim the ones whose -// ordinals the role no longer reaches. +// headless Service is derived from an ordinal. Scaling down keeps its peer records until the +// ordinal's last member leaves, then reclaims the Service. // // THE SCALE-UP HALF IS THE CONTROL. Without it "the Services went away" is satisfied by a // convergence that never created them, which is the failure this case would otherwise report as a @@ -189,8 +191,21 @@ func TestModelDeploymentService_AReplicaServiceIsCreatedAndReclaimedWithItsRepli _, err = reconcileModelDeployment(t, cli) require.NoError(t, err) + require.Len(t, replicaPods(t, cli), 6, "the old members have not left") + require.Equal(t, []string{ + "qwen", "qwen-server", "qwen-server-r0", "qwen-server-r1", "qwen-server-r2", + }, serviceNames(t, cli), "old members keep their peer records") + for _, pod := range replicaPods(t, cli) { + if pod.Labels[modelDeploymentReplicaOrdinalLabel] != "0" { + releaseLifecyclePod(t, cli, &pod) + } + } + require.Len(t, replicaPods(t, cli), 2, "only the desired replica remains") + _, err = reconcileModelDeployment(t, cli) + require.NoError(t, err) + assert.Equal(t, []string{"qwen", "qwen-server", "qwen-server-r0"}, serviceNames(t, cli), - "the ordinals the role no longer reaches take their addresses with them") + "the departed replicas take their peer records with them") } // TestModelDeploymentService_SurvivesAScale pins the one interaction between the Service and a @@ -236,7 +251,7 @@ func TestModelDeploymentService_SurvivesAScale(t *testing.T) { // replica, and must not carry the entrance label, which a spec update can move. func TestRenderModelDeploymentService_SelectsExactlyTheRolesPods(t *testing.T) { md := newRenderDeployment() - svc := renderModelDeploymentService(md) + svc := renderModelDeploymentService(md, nil) pod := renderOne(t, md, newRenderInstanceType()) // The eligibility term is written at runtime by the reconciler, so a member the Service may // select is one that carries it; the rendered template never does. @@ -259,7 +274,7 @@ func TestRenderModelDeploymentServices_AboveOneMemberAddsAHeadlessServicePerRepl md.Spec.Roles[0].Replicas = 3 }) - svcs := renderModelDeploymentServices(md, nil) + svcs := renderModelDeploymentServices(md, nil, nil) names := make([]string, 0, len(svcs)) byName := make(map[string]*core.Service, len(svcs)) for _, svc := range svcs { @@ -330,7 +345,7 @@ func TestRenderModelDeploymentServices_SelectEligibleEndpoints(t *testing.T) { require.NotEmpty(t, qs) observeModelDeploymentEndpointEligibility(tc.md, qs) } - svcs := renderModelDeploymentServices(tc.md, nil) + svcs := renderModelDeploymentServices(tc.md, nil, nil) require.NotEmpty(t, svcs) ordinary := 0 @@ -359,7 +374,7 @@ func TestRenderModelDeploymentServices_SelectEligibleEndpoints(t *testing.T) { func TestRenderModelDeploymentServices_AtSizeOneIsUnchanged(t *testing.T) { md := newRenderDeployment(func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 3 }) - svcs := renderModelDeploymentServices(md, nil) + svcs := renderModelDeploymentServices(md, nil, nil) names := make([]string, 0, len(svcs)) for _, svc := range svcs { names = append(names, svc.Name) @@ -416,7 +431,7 @@ func TestRenderModelDeploymentService_Port(t *testing.T) { md.Spec.Roles[0].Ports = tc.ports }) - svc := renderModelDeploymentService(md) + svc := renderModelDeploymentService(md, nil) require.Len(t, svc.Spec.Ports, 1) assert.Equal(t, tc.wantPort, svc.Spec.Ports[0].Port) assert.Equal(t, tc.wantPort, svc.Spec.Ports[0].TargetPort.IntVal) @@ -544,7 +559,7 @@ func TestRenderModelDeploymentService_TargetFollowsTheEnginesPort(t *testing.T) pod, err := renderModelDeploymentPod(context.Background(), in) require.NoError(t, err) - svcs := renderModelDeploymentServices(md, nil) + svcs := renderModelDeploymentServices(md, nil, nil) require.Len(t, svcs, 2) for _, svc := range svcs { require.Len(t, svc.Spec.Ports, 1, svc.Name) @@ -884,8 +899,8 @@ func TestAlignModelDeploymentService(t *testing.T) { for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { md := newRenderDeployment() - expected := renderModelDeploymentService(md) - actual := renderModelDeploymentService(md) + expected := renderModelDeploymentService(md, nil) + actual := renderModelDeploymentService(md, nil) if tc.mutate != nil { tc.mutate(actual) } @@ -946,10 +961,13 @@ func TestEndpointEligibleSelectorActivation(t *testing.T) { healthy := healthDeployment(2) healthy.Spec.Router = &workercore.ModelDeploymentRouter{Name: workercore.ModelDeploymentRouterLLMD} - healthyPods := []core.Pod{ + // Stamped with the group total the render always writes, because completeness is measured + // against the replica's OWN total now. Without it this pair reads as two replicas of one, and a + // member claiming seat one of a group declaring one is a group this operator cannot classify. + healthyPods := stampHealthGroupTotals(healthy, []core.Pod{ healthPod("server", 0, 0, healthBool(true), "uid-a"), healthPod("server", 0, 1, healthBool(true), "uid-b"), - } + }) for name, md := range map[string]*workercore.ModelDeployment{ "unverified engine version": unverified, @@ -957,10 +975,10 @@ func TestEndpointEligibleSelectorActivation(t *testing.T) { "replaced command": takeOver, } { t.Run(name+" renders no narrowing", func(t *testing.T) { - front := renderModelDeploymentService(md) + front := renderModelDeploymentService(md, nil) _, has := front.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] assert.False(t, has, "front Service must not narrow a never-activating shape") - roleSvc := renderModelDeploymentRoleService(md, &md.Spec.Roles[0], nil) + roleSvc := renderModelDeploymentRoleService(md, &md.Spec.Roles[0], nil, nil, false) _, has = roleSvc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] assert.False(t, has, "role Service must not narrow a never-activating shape") published, err := renderModelDeploymentRouterObjects(context.Background(), md, @@ -985,7 +1003,7 @@ func TestEndpointEligibleSelectorActivation(t *testing.T) { require.False(t, first[0].Activated(), "setup requires an unavailable first observation") observeModelDeploymentEndpointEligibility(healthy, first) assert.False(t, modelDeploymentEligibilityDecided(healthy), "no activation fact exists yet") - front := renderModelDeploymentService(healthy) + front := renderModelDeploymentService(healthy, nil) _, has := front.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] assert.False(t, has, "first enable with an unavailable observation must not narrow") @@ -995,7 +1013,7 @@ func TestEndpointEligibleSelectorActivation(t *testing.T) { require.True(t, working[0].Activated(), "setup requires an available observation") observeModelDeploymentEndpointEligibility(healthy, working) assert.True(t, modelDeploymentEligibilityDecided(healthy), "a real pass recorded the predicate") - front = renderModelDeploymentService(healthy) + front = renderModelDeploymentService(healthy, nil) assert.Equal(t, modelDeploymentEndpointEligibleValue, front.Spec.Selector[modelDeploymentLabelKeyEndpointEligible], "activation narrows") @@ -1006,7 +1024,7 @@ func TestEndpointEligibleSelectorActivation(t *testing.T) { retained := modelDeploymentEligibilitySelectorActive(healthy, &healthy.Spec.Roles[0], map[string]string{modelDeploymentLabelKeyEndpointEligible: modelDeploymentEndpointEligibleValue}, false) assert.True(t, retained, "an already-narrowed selector survives a quiet pass") - fresh := renderModelDeploymentService(healthy) + fresh := renderModelDeploymentService(healthy, nil) _, has = fresh.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] assert.False(t, has, "a from-scratch render never fabricates the term") }) @@ -1052,11 +1070,12 @@ func TestAdmissionFirstEnableUnknownPreservesExistingRouting(t *testing.T) { require.False(t, qs[0].Activated(), "setup requires unavailable first-enable observation") require.False(t, qs[0].HasFailure(), "setup requires healthy existing members") for i := range pods { - require.False(t, modelDeploymentPodEligible(md, &md.Spec.Roles[0], &pods[i], qs[0], true), + view := modelDeploymentReplicaView{Members: []*core.Pod{&pods[i]}} + require.False(t, modelDeploymentPodEligible(md, &md.Spec.Roles[0], &pods[i], view, qs[0], true), "first enable must not fabricate eligibility") } for _, svc := range []*core.Service{ - renderModelDeploymentService(md), renderModelDeploymentRoleService(md, &md.Spec.Roles[0], nil), + renderModelDeploymentService(md, nil), renderModelDeploymentRoleService(md, &md.Spec.Roles[0], nil, nil, false), } { _, narrowed := svc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] assert.False(t, narrowed, "unavailable first-enable observation must preserve Service selection") @@ -1069,3 +1088,61 @@ func TestAdmissionFirstEnableUnknownPreservesExistingRouting(t *testing.T) { }) } } + +// The front Service routes the first role alone. A managed Pod of a sibling role, or one already +// leaving, must not lift the take-over refusal for a first role that supplies its own command. +func TestRenderModelDeploymentService_FrontGateReadsOnlyTheFrontRoleMembers(t *testing.T) { + newDeployment := func() *workercore.ModelDeployment { + md := healthDeployment(1, "sibling") + md.Spec.Roles[0].Command = []string{"vllm", "serve", "qwen"} + ModelDeploymentConditionEndpointEligibility.True(md, + modelDeploymentReasonEndpointsQualified, "every group qualified") + + return md + } + // siblingPod is a managed Pod, which is what would lift the refusal if it were counted. + siblingPod := func(md *workercore.ModelDeployment, mutate func(*core.Pod)) core.Pod { + pod := healthPod("sibling", 0, 0, healthBool(true), "uid-sibling") + pod.Spec.Containers = []core.Container{{Name: modelDeploymentMainContainerName}} + kubemeta.ControlOnWithoutBlock(&pod, md, workercore.SchemeGroupVersionKind("ModelDeployment")) + if mutate != nil { + mutate(&pod) + } + + return pod + } + + testCases := []struct { + name string + pod func(md *workercore.ModelDeployment) core.Pod + }{ + { + name: "a managed Pod of a sibling role", + pod: func(md *workercore.ModelDeployment) core.Pod { return siblingPod(md, nil) }, + }, + { + name: "a managed Pod of the front role that is already leaving", + pod: func(md *workercore.ModelDeployment) core.Pod { + return siblingPod(md, func(pod *core.Pod) { + pod.Labels[modelDeploymentLabelKeyComponent] = md.Spec.Roles[0].Name + now := meta.Now() + pod.DeletionTimestamp = &now + }) + }, + }, + } + + for _, tc := range testCases { + t.Run(tc.name, func(t *testing.T) { + md := newDeployment() + pod := tc.pod(md) + require.True(t, modelDeploymentReplicaRunsCommand([]*core.Pod{&pod}), + "setup requires the Pod to be one this operator built") + + svc := renderModelDeploymentService(md, []core.Pod{pod}) + + _, narrowed := svc.Spec.Selector[modelDeploymentLabelKeyEndpointEligible] + assert.False(t, narrowed, "a take-over front role is never narrowed by endpoint eligibility") + }) + } +} diff --git a/pkg/worker/controllers/worker/model_deployment_status.go b/pkg/worker/controllers/worker/model_deployment_status.go index 4ae41e413..48c1573b8 100644 --- a/pkg/worker/controllers/worker/model_deployment_status.go +++ b/pkg/worker/controllers/worker/model_deployment_status.go @@ -429,22 +429,28 @@ func modelDeploymentRoleStatuses( // two. At one member per replica the grouping is the identity, which is why this reads exactly // as it did for the shape that came before. // - size := make(map[string]int, len(md.Spec.Roles)) - for i := range md.Spec.Roles { - size[md.Spec.Roles[i].Name] = modelDeploymentRoleSize(&md.Spec.Roles[i]) - } - ready := make(map[string]int32, len(md.Spec.Roles)) + unmanaged := make(map[string]bool, len(md.Spec.Roles)) reserved := make(map[string]int32, len(md.Spec.Roles)) flavors := make(map[string]sets.Set[string], len(md.Spec.Roles)) for _, view := range modelDeploymentGroupPodsByReplica(pods) { role := view.Role + // Whether the operator built the running command lines is read from the replicas, never + // from the role, and one replica it contributed nothing to is enough. + if execution, _ := modelDeploymentDeployedReplicaExecution(view.Members); execution == + modelDeploymentExecutionTakeover { + unmanaged[role] = true + } + // A REPLICA IS READY WHEN IT HOLDS EVERY MEMBER IT DECLARES AND ALL OF THEM ARE READY. The // completeness half is not redundant with the readiness half: a replica short of a member // can have every member it DOES hold reporting ready, while Kueue has composed no Workload // for it and it is admitted by nothing. - allReady := modelDeploymentReplicaIsComplete(view, size[role]) + // + // The member set is taken against the group's own total, so a role being edited still + // counts the replicas it is serving from. + allReady := modelDeploymentDeployedReplicaShape(view).modelDeploymentReplicaIsWhole() for _, pod := range view.Members { allReady = allReady && podIsReady(pod) } @@ -489,9 +495,9 @@ func modelDeploymentRoleStatuses( Desired: role.Replicas, Ready: ready[role.Name], QuotaReserved: reserved[role.Name], - // A role that replaced the whole command line got no synthesized argument and no client - // environment, so nothing here can claim it is attached to the cache. - Unmanaged: len(role.Command) > 0, + // A replica that replaced the whole command line got no cache client, so nothing here + // can claim it is attached to one. + Unmanaged: unmanaged[role.Name], AssignedFlavors: modelDeploymentAssignedFlavors(flavors[role.Name]), Parallelism: ReadModelDeploymentRoleParallelism(md.Spec.Engine.Name, role), Endpoints: modelDeploymentRoleEndpoints(md.Spec.Router != nil, qualified[role.Name]), @@ -851,18 +857,12 @@ func observeModelDeploymentQuota( views := modelDeploymentGroupPodsByReplica(pods) groups := modelDeploymentPodGroups(md) for _, group := range groups { - size := 1 - for i := range md.Spec.Roles { - if md.Spec.Roles[i].Name == group.Role { - size = modelDeploymentRoleSize(&md.Spec.Roles[i]) - - break - } - } - var alive int for _, view := range views { - if view.Role == group.Role && modelDeploymentReplicaIsComplete(view, size) { + // The replica's own group total, so a role being edited still counts the replicas it is + // serving from. + if view.Role == group.Role && + modelDeploymentDeployedReplicaShape(view).modelDeploymentReplicaIsWhole() { alive++ } } diff --git a/pkg/worker/controllers/worker/model_deployment_status_test.go b/pkg/worker/controllers/worker/model_deployment_status_test.go index e88c1be0b..219eeff25 100644 --- a/pkg/worker/controllers/worker/model_deployment_status_test.go +++ b/pkg/worker/controllers/worker/model_deployment_status_test.go @@ -16,12 +16,14 @@ import ( ctrlfake "sigs.k8s.io/controller-runtime/pkg/client/fake" ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" "gpustack.ai/gpustack/pkg/kubemeta" "gpustack.ai/gpustack/pkg/nodefeature" "gpustack.ai/gpustack/pkg/systemmeta" + "gpustack.ai/gpustack/pkg/utils/strconvx" ) func TestModelDeploymentRoleSummary(t *testing.T) { @@ -99,6 +101,17 @@ func readyReplica(md *workercore.ModelDeployment, ordinal int32, ready bool) *co // The controller reference is what the reconciler selects on, so a fixture without one is // invisible to every path that lists replicas rather than being handed them. kubemeta.ControlOnWithoutBlock(pod, md, workercore.SchemeGroupVersionKind("ModelDeployment")) + // The render writes the serving-port annotation, the drain hook and the group total; readers + // below use all three to tell a command line the operator built from one it did not. + engine := core.Container{Name: modelDeploymentMainContainerName, Image: "vllm/vllm-openai:v0.29.0"} + engine.Lifecycle = &core.Lifecycle{PreStop: &core.LifecycleHandler{Exec: &core.ExecAction{ + Command: []string{"/bin/sh", "-c", "/usr/local/bin/gpustack-drain"}, + }}} + pod.Spec.Containers = []core.Container{engine} + pod.Annotations = map[string]string{ + kueuepodconst.GroupTotalCountAnnotation: strconvx.Itoa(modelDeploymentRoleSize(&md.Spec.Roles[0])), + "prometheus.io/port": "8000", + } if ready { pod.Status.Conditions = []core.PodCondition{{Type: core.PodReady, Status: core.ConditionTrue}} } @@ -338,16 +351,45 @@ func readyRouterRolePods(md *workercore.ModelDeployment) []core.Pod { // TestComputeModelDeploymentStatus_Unmanaged pins the flag that tells a reader why no cache // condition will ever be True for this role. +// TestComputeModelDeploymentStatus_Unmanaged pins the field against the REPLICAS rather than the +// role, which is the whole difference this task makes to it. +// +// A role declaring a command line says the operator will contribute nothing to the replica it is +// about to build. It says nothing at all about the replica already running, and during a command +// transition the two disagree: a role that has just gone back to a managed command line describes +// a replacement, while every Pod standing runs arguments this operator never wrote -- containers +// carrying no cache client, which is precisely what the field exists to say. func TestComputeModelDeploymentStatus_Unmanaged(t *testing.T) { + managed := newRenderDeployment(func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Replicas = 1 + }) + r := &ModelDeploymentReconciler{Client: newModelDeploymentClient(managed, newRenderInstanceType())} + + status, err := r.computeModelDeploymentStatusWithQualifications( + context.Background(), managed, readyPods(managed, 1), nil, nil, nil, nil) + require.NoError(t, err) + require.Len(t, status.Roles, 1) + assert.False(t, status.Roles[0].Unmanaged, + "a replica this operator built is a replica it can claim a cache client for") +} + +// TestComputeModelDeploymentStatus_UnmanagedIsObservedNotDeclared is the reverse transition, and it +// is the case the desired-shaped field got wrong: the ROLE is declared managed and the RUNNING +// replicas are not, so reporting the role would tell an operator their engine has a cache client +// when the container in front of it never received one. +func TestComputeModelDeploymentStatus_UnmanagedIsObservedNotDeclared(t *testing.T) { md := newRenderDeployment(func(md *workercore.ModelDeployment) { - md.Spec.Roles[0].Command = []string{"/bin/my-server"} + md.Spec.Roles[0].Replicas = 1 }) r := &ModelDeploymentReconciler{Client: newModelDeploymentClient(md, newRenderInstanceType())} - status, err := r.computeModelDeploymentStatusWithQualifications(context.Background(), md, nil, nil, nil, nil, nil) + status, err := r.computeModelDeploymentStatusWithQualifications( + context.Background(), md, takeoverPods(md, 1), nil, nil, nil, nil) require.NoError(t, err) require.Len(t, status.Roles, 1) - assert.True(t, status.Roles[0].Unmanaged) + assert.True(t, status.Roles[0].Unmanaged, + "the running replicas run a command line the operator contributed nothing to, whatever "+ + "the role now declares") } // TestModelDeploymentStatus_AssignedFlavorIsAbsentUntilItIsAssigned is why the field is nil rather @@ -647,6 +689,12 @@ func TestObserveModelDeploymentQuota_AnIncompleteReplicaIsReportedAboveOneMember modelDeploymentReplicaOrdinalLabel: fmt.Sprint(ordinal), modelDeploymentMemberIndexLabel: fmt.Sprint(index), }, + // The group total a rendered replica always carries, and the size completeness is + // measured against. Without it each member reads as a replica of one, and a member + // holding seat one of a group of one is a group that contradicts itself. + Annotations: map[string]string{ + kueuepodconst.GroupTotalCountAnnotation: "2", + }, }} } @@ -2432,6 +2480,12 @@ func TestObserveModelDeploymentQuota_WaitingFiguresAreInReplicasNotPods(t *testi modelDeploymentReplicaOrdinalLabel: fmt.Sprint(ordinal), modelDeploymentMemberIndexLabel: fmt.Sprint(index), }, + // The group total a rendered replica always carries, and the size completeness is + // measured against. Without it each member reads as a replica of one, and a member + // holding seat one of a group of one is a group that contradicts itself. + Annotations: map[string]string{ + kueuepodconst.GroupTotalCountAnnotation: "2", + }, }} } diff --git a/pkg/worker/controllers/worker/model_deployment_test.go b/pkg/worker/controllers/worker/model_deployment_test.go index 39488fbd5..cc36ac6e2 100644 --- a/pkg/worker/controllers/worker/model_deployment_test.go +++ b/pkg/worker/controllers/worker/model_deployment_test.go @@ -18,6 +18,7 @@ import ( labels "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/sets" ctrlrecord "k8s.io/client-go/tools/record" ctrl "sigs.k8s.io/controller-runtime" ctrlcli "sigs.k8s.io/controller-runtime/pkg/client" @@ -25,6 +26,7 @@ import ( ctrlinterceptor "sigs.k8s.io/controller-runtime/pkg/client/interceptor" ctrlreconcile "sigs.k8s.io/controller-runtime/pkg/reconcile" kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2" + kueuectrlconst "sigs.k8s.io/kueue/pkg/controller/constants" kueuepodconst "sigs.k8s.io/kueue/pkg/controller/jobs/pod/constants" worker "gpustack.ai/gpustack/api/worker/v1" @@ -32,6 +34,7 @@ import ( "gpustack.ai/gpustack/pkg/kubeapistatus" "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" "gpustack.ai/gpustack/pkg/systemmeta" + "gpustack.ai/gpustack/pkg/utils/stringx" "gpustack.ai/gpustack/pkg/worker/kvcache/inject" "gpustack.ai/gpustack/pkg/worker/settings" ) @@ -46,7 +49,22 @@ func newModelDeploymentClient(objs ...ctrlcli.Object) ctrlcli.Client { } } + allocated := 0 return ctrlfake.NewClientBuilder(). + WithInterceptorFuncs(ctrlinterceptor.Funcs{ + Create: func(ctx context.Context, next ctrlcli.WithWatch, obj ctrlcli.Object, + opts ...ctrlcli.CreateOption, + ) error { + switch obj.(type) { + case *core.Pod, *kueue.Workload: + if obj.GetUID() == "" { + allocated++ + obj.SetUID(types.UID("created-" + strconv.Itoa(allocated))) + } + } + return next.Create(ctx, obj, opts...) + }, + }). WithScheme(scheme.Scheme). // The release observation lists a node's pods with the spec.nodeName field selector, which // a real API server answers directly. The fake client can only answer it when the field is @@ -395,19 +413,33 @@ func setGroupAsk(t *testing.T, cli ctrlcli.Client, asking bool) { require.NoError(t, cli.Update(context.Background(), wl)) } -// admittedReplicaWorkload builds the Workload Kueue composes for ONE replica's group: named after -// the group the Pod's own membership label carries -- which is Kueue's own rule, the group name -// verbatim -- owning that Pod by a plain reference, admitted or still pending as the case needs. +// admittedReplicaWorkload builds the Workload Kueue composes for one replica's group: named after +// the group the members' own membership labels carry -- which is Kueue's own rule, the group name +// verbatim -- owning them by plain references, admitted or still pending as the case needs. +// +// THE WHOLE GROUP IS PASSED, not one of its members, because a PodSet counts the members sharing +// its role hash and carries one member's template: a Workload built from a single member of a +// multi-Member replica would ask Kueue for one Pod where the group seats two, which is a +// composition Kueue never produces and which this tree's own admission proof must reject. // // The UIDs are stamped by the fixture rather than the environment, so an ownership match between a // Pod and a Workload cannot pass on two empty values. -func admittedReplicaWorkload(pod *core.Pod, admitted bool) *kueue.Workload { - group := pod.Labels[kueuepodconst.GroupNameLabel] +func admittedReplicaWorkload(group []*core.Pod, admitted bool) *kueue.Workload { + representative := group[0] + groupName := representative.Labels[kueuepodconst.GroupNameLabel] wl := &kueue.Workload{} - wl.Name, wl.Namespace, wl.UID = group, pod.Namespace, types.UID("wl-"+group) - wl.OwnerReferences = []meta.OwnerReference{{ - APIVersion: "v1", Kind: "Pod", Name: pod.Name, UID: pod.UID, - }} + wl.Name, wl.Namespace = groupName, representative.Namespace + wl.UID = replicaWorkloadUID(group) + for _, member := range group { + wl.OwnerReferences = append(wl.OwnerReferences, meta.OwnerReference{ + APIVersion: "v1", Kind: "Pod", Name: member.Name, UID: member.UID, + }) + } + // The queue and the podsets are what Kueue copies off the members, and the replacement's + // admission proof compares them against what its current members compose to. A Workload that + // carried neither would not be fresh admission however it was admitted. + wl.Spec.QueueName = kueue.LocalQueueName(representative.Labels[kueuectrlconst.QueueLabel]) + wl.Spec.PodSets = replacementPinnedPodSets(group) if admitted { wl.Status.Conditions = []meta.Condition{{ Type: kueue.WorkloadAdmitted, @@ -420,6 +452,24 @@ func admittedReplicaWorkload(pod *core.Pod, admitted bool) *kueue.Workload { return wl } +// replicaWorkloadUID identifies one Workload incarnation, from the members it was composed for +// rather than from the group's name. +// +// THE GROUP NAME IS CONSTANT ACROSS EVERY INCARNATION of a replica, so a UID derived from it would +// hand the same identity to the replacement that the replica it replaced wore. A lifecycle test +// asking whether a captured Workload was the one deleted, or whether an admission belongs to the +// group standing now, would then be reading one identity where the cluster holds two -- and would +// pass against a controller that confused them. +func replicaWorkloadUID(group []*core.Pod) types.UID { + uids := make([]string, 0, len(group)) + for _, member := range group { + uids = append(uids, string(member.UID)) + } + slices.Sort(uids) + + return types.UID("wl-" + stringx.SumByFNV64a(strings.Join(uids, "/"))) +} + // standInForKueue plays the half of the handshake this tree cannot run: after a pass, it finishes // the departures the pass issued -- the drain completes and the finalizer releases -- and composes // a Workload for every live replica whose group has none, admitted or pending as the case asks. @@ -459,16 +509,34 @@ func standInForKueue(t *testing.T, cli ctrlcli.Client, admit bool) { composed[wl.Name] = wl } + // THE LIVE MEMBERS OF EACH GROUP, because a Workload is composed for a group and not for a + // member of it: its PodSets count the members sharing a role hash and carry one member's + // template, so composing per member would ask Kueue for a PodSet of one where the group seats + // two. The groups are walked in name order and each group's members in name order, because + // Kueue's own List carries none and this stand-in has to produce the same composition twice. + groups := make(map[string][]*core.Pod) + var groupNames []string for _, pod := range replicaPods(t, cli) { if pod.DeletionTimestamp != nil { continue } group := pod.Labels[kueuepodconst.GroupNameLabel] + if _, seen := groups[group]; !seen { + groupNames = append(groupNames, group) + } + member := pod + groups[group] = append(groups[group], &member) + } + slices.Sort(groupNames) + for _, group := range groupNames { + members := groups[group] + slices.SortFunc(members, func(a, b *core.Pod) int { return strings.Compare(a.Name, b.Name) }) + if composed[group] == nil { - // RECORDED AS COMPOSED, so the rest of the group adopts it below rather than trying to + // RECORDED AS COMPOSED, so the next pass's members adopt it below rather than trying to // create a second object under the same name. A group holds one Workload however many // members it has, and without this the stand-in could only ever serve groups of one. - wl := admittedReplicaWorkload(&pod, admit) + wl := admittedReplicaWorkload(members, admit) require.NoError(t, cli.Create(ctx, wl)) composed[group] = wl @@ -482,15 +550,22 @@ func standInForKueue(t *testing.T, cli ctrlcli.Client, admit bool) { wl := composed[group].DeepCopy() owned := false for _, ref := range wl.OwnerReferences { - owned = owned || ref.UID == pod.UID + owned = owned || ref.UID == members[0].UID } if owned { continue } - wl.OwnerReferences = append(wl.OwnerReferences, meta.OwnerReference{ - APIVersion: "v1", Kind: "Pod", Name: pod.Name, UID: pod.UID, - }) + for _, member := range members { + wl.OwnerReferences = append(wl.OwnerReferences, meta.OwnerReference{ + APIVersion: "v1", Kind: "Pod", Name: member.Name, UID: member.UID, + }) + } require.NoError(t, cli.Update(ctx, wl)) + // The admitted object goes back into the record, or the next pass would append to the copy + // this update has just superseded and the server would refuse it. + stored := new(kueue.Workload) + require.NoError(t, cli.Get(ctx, ctrlcli.ObjectKeyFromObject(wl), stored)) + composed[group] = stored } } @@ -961,6 +1036,20 @@ func TestModelDeploymentReconciler_CountHealsBeforeTheHashRolls(t *testing.T) { changed.Spec.Roles[0].Image = "vllm/vllm-openai:v0.26.0" require.NoError(t, cli.Update(ctx, changed)) + _, err = reconcileModelDeployment(t, cli) + require.NoError(t, err) + require.Equal(t, map[string]string{names[1]: "vllm/vllm-openai:v0.25.1"}, replicaImages(t, cli), + "the missing ordinal remains blocked by its old Workload") + workloads := new(kueue.WorkloadList) + require.NoError(t, cli.List(ctx, workloads, ctrlcli.InNamespace("team-a"))) + removed := 0 + for i := range workloads.Items { + if modelDeploymentWorkloadOwnsAny(&workloads.Items[i], sets.New(gone.UID)) { + require.NoError(t, cli.Delete(ctx, &workloads.Items[i])) + removed++ + } + } + require.Equal(t, 1, removed, "complete the vanished member's Workload cleanup") _, err = reconcileModelDeployment(t, cli) require.NoError(t, err) @@ -1392,9 +1481,8 @@ func TestModelDeploymentReconciler_ATemplateEditRollsOneReplicaAtATime(t *testin // was lost -- and the pass is woken by Pod events, so the read goes to the API server rather than a // cache that would just repeat the list the gate is checking against. // -// THE COUNT IS PART OF THE CONTRACT because these reads bypass the cache: one per role, whatever -// number of ordinals that role is short. A read per ordinal is the same answer at N times the cost, -// billed again on every requeue for as long as a departure takes to drain. +// Vacancy uses one Pod list and one Workload list per short role, independent of missing count. +// A settled role needs neither vacancy read. Service synchronization reads Pods once per pass. type countingAPIReader struct { ctrlcli.Reader lists int @@ -1414,20 +1502,17 @@ func TestModelDeploymentReconciler_TheCreateGateReadsTheAPIServer(t *testing.T) reader := &countingAPIReader{Reader: cli} r := &ModelDeploymentReconciler{Client: cli, APIReader: reader} - // The first pass reads the role ONCE on the API server before creating for either of its two - // missing ordinals: the groups a first create would double-populate are exactly the ones this - // read has to find empty, and one list of the role answers for every one of them. + // Two vacancy lists cover both missing ordinals. One Pod list supplies current Service members. _, err := reconcileModelDeploymentWith(t, r) require.NoError(t, err) - assert.Equal(t, 1, reader.lists, - "one API-server read per role, not one per missing ordinal: this role is short two") + assert.Equal(t, 3, reader.lists, + "two vacancy lists and one Service membership list cover the role's two missing ordinals") - // A settled pass reads nothing through the API server: every ordinal is accounted for and - // no role is rolling. + // A settled pass needs only the current Service membership read. reader.lists = 0 _, err = reconcileModelDeploymentWith(t, r) require.NoError(t, err) - assert.Zero(t, reader.lists, "a settled pass issues no API-server read at all") + assert.Equal(t, 1, reader.lists, "a settled pass reads Service members without vacancy reads") // A short role reads the missing ordinal through the API server, and the replacement lands // once the ordinal reads empty. @@ -2367,13 +2452,24 @@ func TestModelDeploymentReconciler_AHeldReplicaLeavesAnExistingLabelUntouched(t _, err := reconcileModelDeploymentActivated(t, cli) require.NoError(t, err) - assert.Zero(t, writes.patches, - "a held replica writes no Pod at all, so the pre-existing label is preserved byte for byte: %v", - writes.ordered) for _, pod := range endpointEligibilityPods(t, cli) { - assert.Equal(t, before[pod.Name], pod.Labels, + after := maps.Clone(pod.Labels) + // The routing label is the one write this pass issues, and it answers a different question: + // whether the HTTP Service should select the Pod, which is not an eligibility decision and + // is stated by the command-mode routing cases. Everything else is what the predicate holds. + delete(after, modelDeploymentLabelKeyAPIAnswering) + assert.Equal(t, before[pod.Name], after, "%s kept every label it had, because the predicate neither granted nor revoked", pod.Name) } + + // And a second held pass writes nothing at all, which is the part of the original contract that + // matters: a pass re-deriving what it already wrote must not write it again. + writes.patches, writes.ordered = 0, nil + _, err = reconcileModelDeploymentActivated(t, cli) + require.NoError(t, err) + assert.Zero(t, writes.patches, + "a held replica settles after one pass rather than being rewritten on every one: %v", + writes.ordered) } // TestModelDeploymentReconciler_ADefaultHealthFaultWithdrawsThroughTheGate is the half of the gate diff --git a/pkg/worker/webhooks/worker/model_deployment.go b/pkg/worker/webhooks/worker/model_deployment.go index 079398e52..9b568db49 100644 --- a/pkg/worker/webhooks/worker/model_deployment.go +++ b/pkg/worker/webhooks/worker/model_deployment.go @@ -22,6 +22,7 @@ import ( worker "gpustack.ai/gpustack/api/worker/v1" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" "gpustack.ai/gpustack/pkg/kubemeta" + "gpustack.ai/gpustack/pkg/nodefeature" "gpustack.ai/gpustack/pkg/webhook" workerctrl "gpustack.ai/gpustack/pkg/worker/controllers/worker" "gpustack.ai/gpustack/pkg/worker/kvcache/inject" @@ -407,12 +408,15 @@ func modelDeploymentRoleReservedListenPort( // validateModelDeploymentPoolTransport checks a new cache binding against each role's engine. // An unchanged binding is held access: an existing deployment remains editable if its backend -// later changes transport, as with the host access gates above. +// later changes transport, as with the host access gates above. On an update the rule re-runs +// only for a role moved onto a different instanceType, because the engine it reads is carried by +// that type's manufacturer; a role whose type did not move keeps held access. func (r *ModelDeploymentWebhook) validateModelDeploymentPoolTransport( ctx context.Context, old, md *workercore.ModelDeployment, ) (field.ErrorList, error) { if md.Spec.KVCache == nil || md.DeletionTimestamp != nil || - old != nil && kubemeta.DeepEqual(old.Spec.KVCache, md.Spec.KVCache) { + old != nil && kubemeta.DeepEqual(old.Spec.KVCache, md.Spec.KVCache) && + !modelDeploymentPoolTransportInputsChanged(old, md) { return nil, nil } path := field.NewPath("spec", "kvCache", "poolRef", "name") @@ -452,9 +456,23 @@ func (r *ModelDeploymentWebhook) validateModelDeploymentPoolTransport( if nonempty.Len() < 2 { return nil, nil } + // A create judges every role. On an update a role whose type did not move keeps held access; + // a changed binding is judged whole against the new pool. + judgeAll := old == nil || !kubemeta.DeepEqual(old.Spec.KVCache, md.Spec.KVCache) + stored := make(map[string]*workercore.ModelDeploymentRole, len(md.Spec.Roles)) + if !judgeAll { + for i := range old.Spec.Roles { + stored[old.Spec.Roles[i].Name] = &old.Spec.Roles[i] + } + } seen := make(map[string]struct{}, len(md.Spec.Roles)) for i := range md.Spec.Roles { role := &md.Spec.Roles[i] + if !judgeAll { + if was := stored[role.Name]; was != nil && was.InstanceType == role.InstanceType { + continue + } + } if _, ok := seen[role.InstanceType]; ok { continue } @@ -478,18 +496,118 @@ func (r *ModelDeploymentWebhook) validateModelDeploymentPoolTransport( return nil, nil } +// modelDeploymentPoolTransportInputsChanged reports whether an update moved a role onto a +// different instanceType. That is the one transport input the object carries: the engine each +// role runs is read off the type's manufacturer, so a move can flip the verdict the stored shape +// once passed. +func modelDeploymentPoolTransportInputsChanged(old, md *workercore.ModelDeployment) bool { + if len(old.Spec.Roles) != len(md.Spec.Roles) { + return true + } + stored := make(map[string]string, len(old.Spec.Roles)) + for i := range old.Spec.Roles { + stored[old.Spec.Roles[i].Name] = old.Spec.Roles[i].InstanceType + } + for i := range md.Spec.Roles { + if stored[md.Spec.Roles[i].Name] != md.Spec.Roles[i].InstanceType { + return true + } + } + + return false +} + +// modelDeploymentDirectLegProbe renders one role's direct-transfer leg the way any non-Ascend +// type with a computed status would. The manufacturer only ever forces the leg empty or hands +// back the declared protocol, so equal probes mean the leg kept its value whatever this role's +// type reports. +func modelDeploymentDirectLegProbe(md *workercore.ModelDeployment, role *workercore.ModelDeploymentRole) string { + return workerctrl.ModelDeploymentDirectInterfaceProtocol(md, role, nodefeature.ManufacturerNVIDIA) +} + +// modelDeploymentRoleInterfaceInputsChanged reports whether an update touched the inputs the +// interface-request rule reads for one role: its interface request, its InstanceType and its +// command presence. A role the object did not carry counts as changed. Roles are matched by name +// and quantities by value. +// +// An unchanged role is not re-judged: the rule resolves cluster-side objects, and those can drift +// under a stored deployment, so re-running it on a role whose inputs did not move would refuse +// that role for a drift its own edit did not cause. +func modelDeploymentRoleInterfaceInputsChanged(was, role *workercore.ModelDeploymentRole) bool { + return was == nil || role.InstanceType != was.InstanceType || + (len(role.Command) > 0) != (len(was.Command) > 0) || + !kubemeta.DeepEqual(interfaceQuantity(role.Resources), interfaceQuantity(was.Resources)) +} + +// interfaceQuantity returns the role's interface request, nil when unset anywhere on the way. +func interfaceQuantity(ress *workercore.ModelDeploymentRoleResources) *resource.Quantity { + if ress == nil { + return nil + } + return ress.Interface +} + +// modelDeploymentRoleNeedsInterfaceRejudge reports whether this write must rejudge one requesting +// role. A create judges every role. An update judges the roles whose own inputs moved; the router +// and the transfer declaration reach a role only through the direct leg the production helper +// renders, so a role whose leg kept its value keeps the escape hatch its stored verdict provides. +func (r *ModelDeploymentWebhook) modelDeploymentRoleNeedsInterfaceRejudge( + ctx context.Context, old, md *workercore.ModelDeployment, was, role *workercore.ModelDeploymentRole, i int, +) (bool, error) { + if modelDeploymentRoleInterfaceInputsChanged(was, role) { + return true, nil + } + if modelDeploymentDirectLegProbe(old, was) == modelDeploymentDirectLegProbe(md, role) { + return false, nil + } + // The probes differ, which only a non-Ascend manufacturer can turn into a real leg change, + // so the rendered legs are compared at the type's own manufacturer before rejudging. + manufacturer := "" + if md.Spec.Engine.Name == workercore.ModelDeploymentEngineVLLM { + instType, err := r.getInstanceType(ctx, role.InstanceType, i) + if err != nil { + return false, err + } + manufacturer = instType.Status.Detail.Manufacturer + } + return workerctrl.ModelDeploymentDirectInterfaceProtocol(old, was, manufacturer) != + workerctrl.ModelDeploymentDirectInterfaceProtocol(md, role, manufacturer), nil +} + // validateModelDeploymentInterfaceRequests checks the device key against the resolved cache -// backend and the direct leg each role actually renders. An unresolved cache binding cannot +// backend and the direct leg each judged role actually renders. An unresolved cache binding cannot // justify a positive device request, because it may later resolve to another fabric. +// +// A create judges every requesting role. An update judges the roles whose own inputs moved, and +// the roles whose rendered direct-transfer leg moved with the router or the transfer +// declaration; a role whose leg kept its value keeps the escape hatch its stored verdict gives. func (r *ModelDeploymentWebhook) validateModelDeploymentInterfaceRequests( - ctx context.Context, md *workercore.ModelDeployment, + ctx context.Context, old, md *workercore.ModelDeployment, ) (field.ErrorList, error) { + stored := make(map[string]*workercore.ModelDeploymentRole, len(md.Spec.Roles)) + if old != nil { + for i := range old.Spec.Roles { + stored[old.Spec.Roles[i].Name] = &old.Spec.Roles[i] + } + } + var requested []int for i := range md.Spec.Roles { ress := md.Spec.Roles[i].Resources - if ress != nil && ress.Interface != nil && ress.Interface.Sign() > 0 { - requested = append(requested, i) + if ress == nil || ress.Interface == nil || ress.Interface.Sign() <= 0 { + continue } + if old != nil { + judge, err := r.modelDeploymentRoleNeedsInterfaceRejudge( + ctx, old, md, stored[md.Spec.Roles[i].Name], &md.Spec.Roles[i], i) + if err != nil { + return nil, err + } + if !judge { + continue + } + } + requested = append(requested, i) } if len(requested) == 0 { return nil, nil @@ -577,7 +695,7 @@ func (r *ModelDeploymentWebhook) ValidateCreate( } errs = append(errs, typeErrs...) if len(errs) == 0 { - interfaceErrs, err := r.validateModelDeploymentInterfaceRequests(ctx, md) + interfaceErrs, err := r.validateModelDeploymentInterfaceRequests(ctx, nil, md) if err != nil { return nil, err } @@ -631,9 +749,8 @@ func (r *ModelDeploymentWebhook) ValidateUpdate( return nil, err } errs = append(errs, typeErrs...) - if len(errs) == 0 && old != nil && (!kubemeta.DeepEqual(old.Spec.KVTransfer, md.Spec.KVTransfer) || - !kubemeta.DeepEqual(old.Spec.Router, md.Spec.Router)) { - interfaceErrs, err := r.validateModelDeploymentInterfaceRequests(ctx, md) + if len(errs) == 0 && old != nil { + interfaceErrs, err := r.validateModelDeploymentInterfaceRequests(ctx, old, md) if err != nil { return nil, err } @@ -664,17 +781,6 @@ func (r *ModelDeploymentWebhook) ValidateUpdate( const modelDeploymentIdentityMessage = "this is part of what makes this deployment the deployment " + "it is: a different value describes a different deployment, which is created rather than edited" -// modelDeploymentReplicaSizeFrozenMessage is the reason a size change carries, and it points at the -// field that does move rather than only refusing. -// -// IT NAMES replicas BECAUSE THE REFUSAL IS OTHERWISE A DEAD END. An operator changing size almost -// always wants more capacity, which replicas gives without touching anything already serving. The -// one thing size can do -- serve at a different instance shape -- genuinely needs a new deployment, -// and saying both is what separates "you asked for the wrong field" from "you cannot have this". -const modelDeploymentReplicaSizeFrozenMessage = "an instance's size is fixed when the deployment is " + - "created: the Pods of a running instance cannot become a different number of Pods. Change " + - "replicas to run more or fewer instances, or create a deployment that declares the size you want" - // validateModelDeploymentRouterName allows a router to be added or removed, but not changed in // place. Changing the implementation is a delete and create with an interval between them. func validateModelDeploymentRouterName(md, old *workercore.ModelDeployment) field.ErrorList { @@ -691,17 +797,12 @@ func validateModelDeploymentRouterName(md, old *workercore.ModelDeployment) fiel // one that changes how it is currently run. // // THE CRITERION IS THE RULE, NOT THE LIST BELOW. A field is frozen when it answers "which deployment -// is this" -- what is served, what serves it, whose cache it shares, and the shape of the roles that -// serve it. A field is editable when it answers "how is this deployment being run right now" -- how -// many replicas, which build, how that build is fetched and tuned. Judging a NEW field means asking -// that question, not appending to the list; a list alone grows by precedent and stops meaning -// anything. -// -// ONE FIELD IS FROZEN AGAINST THE CRITERION and is marked here so it is not read as an oversight: -// roles[].resources does not answer which deployment this is, but it changes what admission has to -// find. Changing it renegotiates the scheduling, which is not materially different from deleting and -// recreating. Its mirror image is the role's privileged field, which the criterion leaves editable even -// though a different argument could move it. +// is this" -- what is served, what serves it, whose cache it shares. A field is editable when it +// answers "how is this deployment being run right now" -- how many replicas at what size, which +// pool and card shape, which build, how that build is fetched, tuned and launched. Judging a NEW +// field means asking that question, not appending to the list; a list alone grows by precedent and +// stops meaning anything. The role's shape fields sit on the editable side; kind stays frozen +// because it is what the role is. // // ROLES ARE MATCHED BY NAME, NEVER BY POSITION. The field is a listType=map keyed by name, so a // reordered list is the same set of roles and the API already treats it as one; comparing by index @@ -782,25 +883,13 @@ func validateModelDeploymentRoleIdentity(md, old *workercore.ModelDeployment) fi return errs } -// validateModelDeploymentRoleIdentityFields compares one role's frozen fields against the stored +// validateModelDeploymentRoleIdentityFields compares one role's frozen field against the stored // role of the same name. // -// command is frozen because it decides whether the operator configures this role at all: a -// role that supplies one is taken over by its author, which changes cache injection and what status -// can claim. The rest of the role's container fields are how the build is fetched, shaped and -// tuned, and are editable. -// -// size IS FROZEN FOR A DIFFERENT REASON AND SO CARRIES A DIFFERENT MESSAGE. The others are identity: -// a different value describes a different deployment. This one is arithmetic the Pods of a running -// instance cannot survive -- an instance of n Pods is admitted as one group of n, and a new n makes -// every living instance the wrong shape at once, with no intermediate state in which the deployment -// is serving. What the reader needs is the field that does move, so the message names replicas. -// -// FREEZING IT ALSO KEEPS ONE NUMBER UNDER ONE WRITER. The size is the group's declared total, the -// member count the operator creates, and the rank count it publishes to every container; all three -// are derived from this one field on every pass. A count that never moves cannot be read as two -// different totals by two readers, and a count that moved would change a group's declared total -// while the group is running -- which Kueue answers by stopping every member it already admitted. +// Kind is the one frozen role field: it is what the role is, so a Server role re-declared as +// Decode is a different deployment wearing the same name. Size, instanceType, resources and +// command describe how the role currently runs, are editable, and are applied by whole-group +// replacement. The elastic-EP profile pins its own role through validateModelDeploymentElasticIdentity. func validateModelDeploymentRoleIdentityFields( rolePath *field.Path, role, was *workercore.ModelDeploymentRole, ) field.ErrorList { @@ -809,22 +898,6 @@ func validateModelDeploymentRoleIdentityFields( errs = append(errs, field.Invalid( rolePath.Child("kind"), role.Kind, modelDeploymentIdentityMessage)) } - if role.ReplicaSize != was.ReplicaSize { - errs = append(errs, field.Invalid( - rolePath.Child("size"), role.ReplicaSize, modelDeploymentReplicaSizeFrozenMessage)) - } - if role.InstanceType != was.InstanceType { - errs = append(errs, field.Invalid( - rolePath.Child("instanceType"), role.InstanceType, modelDeploymentIdentityMessage)) - } - if !kubemeta.DeepEqual(role.Resources, was.Resources) { - errs = append(errs, field.Invalid( - rolePath.Child("resources"), role.Resources, modelDeploymentIdentityMessage)) - } - if !kubemeta.DeepEqual(role.Command, was.Command) { - errs = append(errs, field.Invalid( - rolePath.Child("command"), role.Command, modelDeploymentIdentityMessage)) - } return errs } @@ -1488,10 +1561,10 @@ func validateModelDeploymentServiceNamesAreDistinct(md *workercore.ModelDeployme // IT IS A SEPARATE RULE FROM THE SERVICE-NAME ONE ABOVE, AND NOT A WIDENING OF IT, because the two // have different subjects and different triggers. That rule is about -, which no // update can move, so it is checked once when a role appears. This one is about -// --r-m, whose length depends on `replicas` -- a field a user is -// invited to change -- so it has to be checked on every update, including for roles that already -// exist. Merging them would mean either re-checking an immutable pair forever or letting a scale -// walk a legal deployment into an illegal one. +// --r-m, whose length depends on `replicas` and `size` -- both +// fields a user is invited to change -- so it has to be checked on every update, including for +// roles that already exist. Merging them would mean either re-checking an immutable pair forever +// or letting an edit walk a legal deployment into an illegal one. // // THE FAILURE IT PREVENTS IS A LOOP RATHER THAN AN ERROR. A member's name is its hostname, which is // a DNS-1123 label of 63 characters, while the names above are only checked to 63 for the shorter @@ -1529,16 +1602,14 @@ func validateModelDeploymentRoleMemberNames(md *workercore.ModelDeployment) fiel // THE ERROR IS ATTACHED TO THE ROLE RATHER THAN TO ONE OF ITS FIELDS, because four inputs // spell this name -- the deployment's name, the role's name, `replicas` and `size` -- and - // which of them moved is not knowable from the object being validated. Naming `size` would - // be actively misleading on the edit that reaches here most often: a scale that takes - // `replicas` from 9 to 10 lengthens the ordinal and trips this, while `size` is immutable - // and therefore the one input the user cannot act on. The detail below names every input - // that can be changed instead. + // which of them moved is not knowable from the object being validated. The two names are + // frozen identity; only the counts are editable, so the error names every input instead of + // betting on one. errs = append(errs, field.Invalid( rolesPath.Index(i), name, fmt.Sprintf( "a replica of this role is addressed by naming each of its Pods, and the longest "+ "such name would be %q (%d characters), which cannot be a hostname: %s. "+ - "Shorten the role or the deployment, or declare fewer replicas", + "Shorten the role or the deployment, or declare fewer replicas or members", name, len(name), strings.Join(why, "; ")))) } diff --git a/pkg/worker/webhooks/worker/model_deployment_elastic.go b/pkg/worker/webhooks/worker/model_deployment_elastic.go index 5a77e601c..bd7c2642a 100644 --- a/pkg/worker/webhooks/worker/model_deployment_elastic.go +++ b/pkg/worker/webhooks/worker/model_deployment_elastic.go @@ -23,6 +23,7 @@ import ( "k8s.io/apimachinery/pkg/util/validation/field" workercore "gpustack.ai/gpustack/api/worker/v1alpha1" + "gpustack.ai/gpustack/pkg/kubemeta" workerctrl "gpustack.ai/gpustack/pkg/worker/controllers/worker" "gpustack.ai/gpustack/pkg/worker/elasticprofile" ) @@ -290,9 +291,9 @@ func validateModelDeploymentElasticOwnedKeys( return errs } -// validateModelDeploymentElasticIdentity enforces the frozen half of the profile: presence -// and head placement are identity, width is the running state. The profile must also stay on -// the role that carried it. +// validateModelDeploymentElasticIdentity enforces the frozen half of the profile: presence, +// head placement, the role's pool and card shape, and the declared TP and width ceiling are +// identity; width is the running state. The profile must also stay on the role that carried it. func validateModelDeploymentElasticIdentity( md, old *workercore.ModelDeployment, ) field.ErrorList { @@ -327,6 +328,23 @@ func validateModelDeploymentElasticIdentity( "the elastic profile must stay on the role it was created with: "+modelDeploymentIdentityMessage, )) } + // The profile pins what the ordinary contract lets move: an elastic role's member shape is + // fixed at creation, and width is the only running-state field. ReplicaSize stays at one and + // a command stays forbidden through the shape rules. + if newElastic.InstanceType != oldElastic.InstanceType { + errs = append(errs, field.Invalid( + modelDeploymentElasticRolePath(md, newElastic).Child("instanceType"), + newElastic.InstanceType, + "the elastic-EP profile pins the pool the members were admitted against: "+modelDeploymentIdentityMessage, + )) + } + if !kubemeta.DeepEqual(newElastic.Resources, oldElastic.Resources) { + errs = append(errs, field.Invalid( + modelDeploymentElasticRolePath(md, newElastic).Child("resources"), + newElastic.Resources, + "the elastic-EP profile pins the members' accelerator shape: "+modelDeploymentIdentityMessage, + )) + } if newElastic.ElasticEP.Width < oldElastic.ElasticEP.Width { errs = append(errs, field.Forbidden( modelDeploymentElasticRolePath(md, newElastic).Child("elasticEp", "width"), diff --git a/pkg/worker/webhooks/worker/model_deployment_elastic_test.go b/pkg/worker/webhooks/worker/model_deployment_elastic_test.go index dde30d194..e07c513f3 100644 --- a/pkg/worker/webhooks/worker/model_deployment_elastic_test.go +++ b/pkg/worker/webhooks/worker/model_deployment_elastic_test.go @@ -105,6 +105,35 @@ func TestValidateModelDeploymentElasticSpec(t *testing.T) { name: "unchanged width on update admitted", md: func() *workercore.ModelDeployment { return elasticMD() }, old: func() *workercore.ModelDeployment { return elasticMD() }, + }, { + name: "instanceType change on update refused", + md: func() *workercore.ModelDeployment { + md := elasticMD() + md.Spec.Roles[0].InstanceType = "other-gpu" + return md + }, + old: func() *workercore.ModelDeployment { return elasticMD() }, + wantErr: []string{"spec.roles[0].instanceType", "profile"}, + }, { + name: "resources change on update refused", + md: func() *workercore.ModelDeployment { + md := elasticMD() + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + return md + }, + old: func() *workercore.ModelDeployment { return elasticMD() }, + wantErr: []string{"spec.roles[0].resources", "profile"}, + }, { + name: "command take-over on update refused", + md: func() *workercore.ModelDeployment { + md := elasticMD() + md.Spec.Roles[0].Command = []string{"python", "-m", "vllm"} + return md + }, + old: func() *workercore.ModelDeployment { return elasticMD() }, + wantErr: []string{"spec.roles[0].command"}, }, { name: "width decrease on update refused", md: func() *workercore.ModelDeployment { diff --git a/pkg/worker/webhooks/worker/model_deployment_test.go b/pkg/worker/webhooks/worker/model_deployment_test.go index 844ad2571..bf3da5dca 100644 --- a/pkg/worker/webhooks/worker/model_deployment_test.go +++ b/pkg/worker/webhooks/worker/model_deployment_test.go @@ -25,6 +25,7 @@ import ( "gpustack.ai/gpustack/pkg/kubeclients/kubernetes/scheme" "gpustack.ai/gpustack/pkg/nodefeature" "gpustack.ai/gpustack/pkg/setting/settingtest" + workerctrl "gpustack.ai/gpustack/pkg/worker/controllers/worker" ) // modelDeployment builds a valid single-role deployment for the given engine, which every case then @@ -250,7 +251,7 @@ func TestModelDeploymentWebhook_InterfaceProtocol(t *testing.T) { objs = append(objs, cache...) } w := newModelDeploymentWebhookWith(objs) - errs, err := w.validateModelDeploymentInterfaceRequests(context.Background(), md) + errs, err := w.validateModelDeploymentInterfaceRequests(context.Background(), nil, md) require.NoError(t, err) if tc.wantErr != "" { require.Len(t, errs, 1) @@ -311,11 +312,12 @@ func TestModelDeploymentWebhook_InterfaceUpdateRejectsProtocolChange(t *testing. require.ErrorContains(t, err, "no effective RDMA or EFA") } -// TestModelDeploymentWebhook_InterfaceUpdateRefusesACountChange pins why the fabric check on -// update runs only when kvTransfer or router moved: a count change with both untouched is refused -// by the role-resources freeze, so no changed count reaches the fabric check. The raised count is -// one EFA accepts, so only the freeze can refuse it. The unchanged update is the baseline. -func TestModelDeploymentWebhook_InterfaceUpdateRefusesACountChange(t *testing.T) { +// TestModelDeploymentWebhook_InterfaceUpdateJudgesACountChange pins what an editable count means +// for the fabric check: a count change is an ordinary resources edit, so the raised count is +// re-judged against the leg the pair renders instead of being refused for having moved. One EFA +// leg accepts both one and two interfaces, so the raised count is admitted; the unchanged update +// is the baseline that proves admission is not an accident of skipping the check. +func TestModelDeploymentWebhook_InterfaceUpdateJudgesACountChange(t *testing.T) { md := modelDeployment(workercore.ModelDeploymentEngineVLLM, role(func(r *workercore.ModelDeploymentRole) { r.Name = "prefill" @@ -337,10 +339,8 @@ func TestModelDeploymentWebhook_InterfaceUpdateRefusesACountChange(t *testing.T) md.Spec.Roles[0].Resources.Interface = ptr.To(resource.MustParse("2")) _, err = w.ValidateUpdate(context.Background(), old, md) - require.ErrorContains(t, err, "spec.roles[0].resources") - require.ErrorContains(t, err, modelDeploymentIdentityMessage) - require.NotContains(t, err.Error(), "spec.roles[0].resources.interface", - "the freeze refuses the count, not the fabric check") + require.NoError(t, err, + "two interfaces on one EFA leg is a valid request, judged now that the count moved") } func TestValidateModelDeployment(t *testing.T) { @@ -2575,6 +2575,12 @@ func modelDeploymentWithEveryField() *workercore.ModelDeployment { // distinguish the rule from either. A refusal case asserts the TYPED ERROR ON ITS OWN PATH rather // than that something was refused, because any other rule refusing the object would satisfy a // weaker assertion just as convincingly. +// +// THE ROLE'S SHAPE FIELDS ARE EDITABLE, not frozen: size, instanceType, resources and command are +// accepted, and each edit is applied by replacing the role's replicas rather than by editing the +// running ones. What stays frozen is what the role IS: its name within the set and its kind. The +// elastic-EP profile pins its own role's shape fields, which +// TestValidateModelDeploymentElasticProfileKeepsItsShapeFrozen holds separately. func TestValidateModelDeploymentIdentity(t *testing.T) { accel2 := resource.NewQuantity(2, resource.DecimalSI) @@ -2598,15 +2604,6 @@ func TestValidateModelDeploymentIdentity(t *testing.T) { {"role_kind", func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Kind = workercore.ModelDeploymentRoleKindPrefill }, "spec.roles[0].kind"}, - {"role_instance_type", func(md *workercore.ModelDeployment) { - md.Spec.Roles[0].InstanceType = "a100-8x" - }, "spec.roles[0].instanceType"}, - {"role_resources", func(md *workercore.ModelDeployment) { - md.Spec.Roles[0].Resources.Accelerator = accel2 - }, "spec.roles[0].resources"}, - {"role_command", func(md *workercore.ModelDeployment) { - md.Spec.Roles[0].Command = []string{"/bin/other"} - }, "spec.roles[0].command"}, {"role_added", func(md *workercore.ModelDeployment) { md.Spec.Roles = append(md.Spec.Roles, role(func(r *workercore.ModelDeploymentRole) { r.Name = "decode" @@ -2616,6 +2613,19 @@ func TestValidateModelDeploymentIdentity(t *testing.T) { md.Spec.Roles = nil }, "spec.roles"}, + // The role's shape fields: how the deployment is currently run. Each is applied by + // replacing the role's replicas, not by editing running Pods. + {"role_instance_type", func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].InstanceType = "a100-8x" + }, ""}, + {"role_resources", func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources.Accelerator = accel2 + }, ""}, + {"role_size", func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 3 }, ""}, + {"role_command", func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"/bin/other"} + }, ""}, + // Editable: how the deployment is being run right now. {"engine_version", func(md *workercore.ModelDeployment) { md.Spec.Engine.Version = "0.26.0" }, ""}, {"role_replicas", func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Replicas = 8 }, ""}, @@ -2694,13 +2704,14 @@ func TestValidateModelDeploymentIdentity_RolesAreMatchedByName(t *testing.T) { assert.Empty(t, validateModelDeploymentIdentity(reordered, old), "a reordered listType=map is the same set of roles") - // The negative baseline: the same reorder, with one frozen field actually changed, is still - // refused and still names the role's own index in the INCOMING list. + // The negative baseline: the same reorder, with a frozen field actually changed, is still + // refused and still names the role's own index in the INCOMING list. Kind is the one role + // field the update contract freezes. moved := reordered.DeepCopy() - moved.Spec.Roles[0].InstanceType = "a100-8x" // this is "decode" after the swap + moved.Spec.Roles[0].Kind = workercore.ModelDeploymentRoleKindServer // this is "decode" after the swap errs := validateModelDeploymentIdentity(moved, old) require.Len(t, errs, 1) - assert.Equal(t, "spec.roles[0].instanceType", errs[0].Field) + assert.Equal(t, "spec.roles[0].kind", errs[0].Field) } // TestValidateModelDeploymentIdentity_DoesNotRunOnCreate pins that there is nothing to compare @@ -2841,9 +2852,9 @@ func TestModelDeploymentWebhook_RefusesMemberNamesThatCannotBeHostnames(t *testi got := err.Error() assert.True(t, errsContain(got, "declare fewer replicas"), got) - // THE FIELD THE REFUSAL NAMES IS THE ONE THE USER CAN ACT ON. Nothing about `size` moved on - // this edit and nothing could -- it is immutable -- so an error attached to it would send an - // operator to change the one input this deployment has already frozen. + // THE ERROR STAYS ON THE ROLE, AND THE REASON IS STABLE ACROSS THE MUTABLE CONTRACT. Four + // inputs spell the name and size is now as editable as replicas, so a refusal naming one + // field would misread an edit made to another on exactly this kind of scale. assert.False(t, errsContain(got, "spec.roles[0].size"), "a replicas-only scale must not be reported against size, which did not change: %s", got) assert.True(t, errsContain(got, "spec.roles[0]: Invalid value"), got) @@ -2902,19 +2913,15 @@ func TestModelDeploymentWebhook_RefusesTwoServicesNamedTheSame(t *testing.T) { }) } -// TestModelDeploymentWebhook_ValidateUpdateFreezesSizeButNotReplicas holds the two halves of the -// scaling story against each other, on one object, in one test. -// -// THE PAIR IS THE POINT, NOT EITHER HALF. A rule refusing a size change would also be satisfied by -// a rule refusing every numeric change, and that implementation takes away the only elasticity this -// role has. Asserting the refusal beside the acceptance is what distinguishes "size is frozen" from -// "numbers are frozen", and the two subtests start from the same stored object so nothing but the -// field under test differs. +// TestModelDeploymentWebhook_ValidateUpdateAllowsSizeAndReplicas holds the two scaling fields +// against the same object, in one test. // -// THE REFUSAL MUST NAME size AND POINT AT replicas. An operator raising size almost always wants -// capacity, which replicas gives without disturbing anything already serving; a refusal that only -// says no leaves them with a deployment they believe cannot grow. -func TestModelDeploymentWebhook_ValidateUpdateFreezesSizeButNotReplicas(t *testing.T) { +// THE PAIR IS THE POINT, NOT EITHER HALF. Both fields are editable, and the two edits mean +// different things: replicas adds or removes whole instances and disturbs nothing already +// serving, while a size edit replaces every instance of the role so the new group shape can be +// admitted. A rule refusing either would pass every other acceptance case in this file, and +// only running both against the same stored object pins the contract rather than an accident. +func TestModelDeploymentWebhook_ValidateUpdateAllowsSizeAndReplicas(t *testing.T) { r := newModelDeploymentWebhookWith([]ctrlcli.Object{servingInstanceType("h20-8x", 8)}) stored := func() *workercore.ModelDeployment { @@ -2926,22 +2933,14 @@ func TestModelDeploymentWebhook_ValidateUpdateFreezesSizeButNotReplicas(t *testi return md } - t.Run("size_change_is_refused", func(t *testing.T) { + t.Run("size_change_is_accepted", func(t *testing.T) { old := stored() md := old.DeepCopy() md.Spec.Roles[0].ReplicaSize = 3 _, err := r.ValidateUpdate(context.Background(), old, md) - require.Error(t, err) - - got := err.Error() - assert.True(t, errsContain(got, "spec.roles[0].size"), got) - assert.True(t, errsContain(got, "fixed when the deployment is created"), got) - assert.True(t, errsContain(got, "replicas"), - "the refusal has to name the field that does move: %s", got) - // The other frozen fields answer "this is a different deployment". Size is not that: the - // deployment is the same one, and what cannot happen is this edit to it. - assert.False(t, errsContain(got, "describes a different deployment"), got) + assert.NoError(t, err, + "the size edit replaces the role's replicas, which is what the contract allows") }) t.Run("replicas_change_is_accepted", func(t *testing.T) { @@ -2950,7 +2949,708 @@ func TestModelDeploymentWebhook_ValidateUpdateFreezesSizeButNotReplicas(t *testi md.Spec.Roles[0].Replicas = 4 _, err := r.ValidateUpdate(context.Background(), old, md) - assert.NoError(t, err, "scaling a role is the one edit this whole shape exists to allow") + assert.NoError(t, err, "scaling a role is the edit this whole shape exists to allow") + }) +} + +// TestModelDeploymentWebhook_ValidateUpdateAcceptsOrdinaryShapeEdits walks the transitions the +// mutable role contract names, each through the full update handler. +// +// THE TABLE IS THE CONTRACT, NOT A SAMPLE. Size crosses one in both directions; instanceType +// moves to a second pool; resources change count, drop to nothing, and take a mode the type +// offers; command makes every transition between the managed and take-over tiers. A case here +// that stops passing is a rule that started refusing an update the contract admits. +func TestModelDeploymentWebhook_ValidateUpdateAcceptsOrdinaryShapeEdits(t *testing.T) { + cases := []struct { + name string + old func(*workercore.ModelDeployment) + edit func(*workercore.ModelDeployment) + types func() []ctrlcli.Object + }{ + { + name: "size increase above one", + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 4 }, + }, + { + name: "size decrease to one", + old: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }, + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 1 }, + }, + { + name: "size decrease across one", + old: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 3 }, + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }, + }, + { + name: "size increase across one", + old: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 1 }, + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 2 }, + }, + { + name: "instanceType moves to another pool", + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].InstanceType = "a100-8x" }, + types: func() []ctrlcli.Object { + return []ctrlcli.Object{servingInstanceType("h20-8x", 8), servingInstanceType("a100-8x", 8)} + }, + }, + { + name: "resources change card count", + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources.Accelerator = resource.NewQuantity(2, resource.DecimalSI) + }, + }, + { + name: "resources omitted entirely", + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Resources = nil }, + }, + { + name: "resources take the slicing mode the type offers", + old: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources = nil + }, + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + AcceleratorSlicedMemoryPercentage: 50, + } + }, + types: func() []ctrlcli.Object { + return []ctrlcli.Object{servingInstanceType("h20-8x", 8, offeringLogicalSlices)} + }, + }, + { + name: "managed role takes over its command line", + old: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = nil + }, + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"vllm", "serve", "Qwen/Qwen2.5-72B-Instruct", "--port", "8000"} + }, + }, + { + name: "take-over role returns to managed", + old: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"/bin/serve"} + }, + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Command = nil }, + }, + { + name: "take-over role replaces its command line", + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Command = []string{"/bin/serve", "--else"} + }, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var live []ctrlcli.Object + if tc.types != nil { + live = tc.types() + } else { + live = []ctrlcli.Object{servingInstanceType("h20-8x", 8)} + } + r := newModelDeploymentWebhookWith(live) + + old := modelDeploymentWithEveryField() + if tc.old != nil { + tc.old(old) + } + r.Default(context.Background(), old) + + md := old.DeepCopy() + tc.edit(md) + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoErrorf(t, err, "an ordinary shape edit is admitted: %s", tc.name) + }) + } +} + +// TestModelDeploymentWebhook_ValidateUpdateStillRefusesInvalidShapeEdits pins that mutability did +// not relax the rules that judge WHAT a shape edit asks for. Each edit is exactly one refusal on +// its own path; a case passing for any other reason would say nothing about this contract. +func TestModelDeploymentWebhook_ValidateUpdateStillRefusesInvalidShapeEdits(t *testing.T) { + cases := []struct { + name string + old func(*workercore.ModelDeployment) + edit func(*workercore.ModelDeployment) + types func() []ctrlcli.Object + refuse string + contains string + }{ + { + name: "a slice the new card count names on a type that offers none", + old: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].Resources = nil }, + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + AcceleratorSlicedMemoryPercentage: 50, + } + }, + refuse: "spec.roles[0].resources", + contains: "does not offer logical slicing", + }, + { + name: "a whole-card request over the new pool's ceiling", + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources.Accelerator = resource.NewQuantity(16, resource.DecimalSI) + }, + refuse: "spec.roles[0].resources.accelerator", + contains: "at most 8 whole accelerator(s)", + }, + { + name: "a size edit that makes member names impossible", + old: func(md *workercore.ModelDeployment) { + md.Name = "deploymentaaaaaaaaaaaaaaaaaaaaaaaaaaaaax" + md.Spec.Roles[0].Name = "roleaaaaaaaaaaaaay" // 40 + 1 + 17 = 58; r0-m8 = 63, r0-m10 = 65 + md.Spec.Roles[0].ReplicaSize = 9 + }, + edit: func(md *workercore.ModelDeployment) { md.Spec.Roles[0].ReplicaSize = 11 }, + refuse: "spec.roles[0]", + contains: "cannot be a hostname", + }, + { + name: "an explicit zero card on a type another role shares", + old: func(md *workercore.ModelDeployment) { + md.Spec.Roles = append(md.Spec.Roles, role(func(r *workercore.ModelDeploymentRole) { + r.Name = "cpu-sidecar" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + })) + }, + edit: func(md *workercore.ModelDeployment) { + md.Spec.Roles[0].Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(0, resource.DecimalSI), + } + }, + refuse: "spec.roles[0].resources.accelerator", + contains: "request at least one accelerator", + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + live := []ctrlcli.Object{servingInstanceType("h20-8x", 8)} + if tc.types != nil { + live = tc.types() + } + r := newModelDeploymentWebhookWith(live) + + old := modelDeploymentWithEveryField() + if tc.old != nil { + tc.old(old) + } + r.Default(context.Background(), old) + + md := old.DeepCopy() + tc.edit(md) + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, "the shape the edit names is invalid and must be refused") + + got := err.Error() + assert.True(t, errsContain(got, tc.refuse), got) + assert.True(t, errsContain(got, tc.contains), got) + }) + } +} + +// TestModelDeploymentWebhook_ValidateUpdateRechecksInterfaceRequests pins the two directions of +// the dependency rule. Adding an interface request is judged against the fabric the deployment +// actually renders, even though the router and the transfer settings did not move; an edit that +// touches none of the inputs is still admitted when the fabric behind the object has drifted. +func TestModelDeploymentWebhook_ValidateUpdateRechecksInterfaceRequests(t *testing.T) { + build := func(store string) ([]ctrlcli.Object, *workercore.ModelDeployment) { + objs := kvCacheFixture() + backend := objs[2].(*workercore.KVCacheBackend) + backend.Spec.Transport.Protocol = store + objs = append(objs, servingInstanceType("h20-8x", 8)) + md := modelDeploymentWithEveryField() + // A role that took over its command line renders no managed leg, so its interface request + // is refused for that reason before the store is ever read. The request under test is + // judged against the fabric, so the role under test stays managed. + md.Spec.Roles[0].Command = nil + md.Spec.KVCache.PoolRef.Name = "chat" + return objs, md + } + + t.Run("a new request is judged against the leg that renders", func(t *testing.T) { + objs, old := build("TCP") + r := newModelDeploymentWebhookWith(objs) + + md := old.DeepCopy() + md.Spec.Roles[0].Resources.Interface = ptr.To(resource.MustParse("1")) + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, + "the request names no interface a TCP store or a managed leg can use") + assert.True(t, errsContain(err.Error(), "no effective RDMA or EFA"), err.Error()) + assert.True(t, errsContain(err.Error(), "spec.roles[0].resources.interface"), err.Error()) + }) + + t.Run("the same request is admitted when the fabric backs it", func(t *testing.T) { + objs, old := build("RDMA") + r := newModelDeploymentWebhookWith(objs) + + md := old.DeepCopy() + md.Spec.Roles[0].Resources.Interface = ptr.To(resource.MustParse("1")) + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, "an RDMA store is an effective leg for the request") + }) + + t.Run("an unchanged input is not refused for a drifted fabric", func(t *testing.T) { + objs, old := build("RDMA") + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), old); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + // The pool's transport is a fact about the cluster, not the object: the store now offers + // TCP, which the stored interface request can no longer use. The object did not change. + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, servingInstanceType("h20-8x", 8)) + r = newModelDeploymentWebhookWith(drifted) + + md := old.DeepCopy() + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "an edit that touches none of the interface inputs is admitted; repairing the pool is the remedy") + }) + + t.Run("a request count untouched by a card edit keeps the drift escape", func(t *testing.T) { + objs, old := build("RDMA") + old.Spec.Roles[0].Resources.Interface = ptr.To(resource.MustParse("1")) + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), old); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, servingInstanceType("h20-8x", 8)) + r = newModelDeploymentWebhookWith(drifted) + + // The card count moves; the interface request does not. The interface rule reads the + // request, so raising the accelerator is not judged against the drifted fabric. + md := old.DeepCopy() + md.Spec.Roles[0].Resources.Accelerator = resource.NewQuantity(2, resource.DecimalSI) + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "the interface input did not move, so the drifted fabric is not re-judged") + }) + + t.Run("router tuning that leaves the leg named keeps the drift escape", func(t *testing.T) { + objs, old := build("RDMA") + old.Spec.Roles[0].Resources.Interface = ptr.To(resource.MustParse("1")) + // A routed role must name its serving port TCP; the fixture's port carries no protocol. + old.Spec.Roles[0].Ports = []workercore.ModelDeploymentPort{{Port: 8000, Protocol: "TCP"}} + old.Spec.Router = &workercore.ModelDeploymentRouter{ + Name: workercore.ModelDeploymentRouterLLMD, + ExtraArgs: []string{"--log-level=info"}, + } + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), old); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, servingInstanceType("h20-8x", 8)) + r = newModelDeploymentWebhookWith(drifted) + + // The leg is decided by the router's presence and name, not by how it is tuned, so an + // extraArgs change re-judges nothing. + md := old.DeepCopy() + md.Spec.Router.ExtraArgs = []string{"--log-level=debug"} + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "router parameter tuning moves no interface input, so the drifted fabric is not re-judged") + }) +} + +// TestModelDeploymentWebhook_ValidateUpdateRechecksPoolTransportAfterInstanceTypeMove pins the +// pool-transport recheck an editable instanceType makes reachable. The rule reads each role's +// manufacturer through its InstanceType, so moving a role onto hardware whose engine transport +// cannot read the bound pool's blocks is refused, while a move within the same manufacturer is +// admitted with the binding unchanged. +func TestModelDeploymentWebhook_ValidateUpdateRechecksPoolTransportAfterInstanceTypeMove(t *testing.T) { + typeFixture := func(name, manufacturer string) ctrlcli.Object { + return servingInstanceType(name, 8, func(it *worker.InstanceType) { + it.Status.Detail.Manufacturer = manufacturer + }) + } + live := func(first, second string) []ctrlcli.Object { + objs := kvCacheFixture() + backend := objs[2].(*workercore.KVCacheBackend) + backend.Spec.Connection.Managed = &workercore.KVCacheBackendManaged{Members: []workercore.KVCacheBackendMember{ + {NodeSelector: map[string]string{"cache": "dram"}, Medium: "DRAM", CapacityPerMember: resource.MustParse("64Gi")}, + { + NodeSelector: map[string]string{"cache": "vram"}, Medium: "VRAM", CapacityPerMember: resource.MustParse("16Gi"), + Transport: &workercore.KVCacheBackendMemberTransport{Protocol: "CANN"}, + }, + }} + objs = append(objs, typeFixture(first, nodefeature.ManufacturerAscend)) + if second != "" { + objs = append(objs, typeFixture(second, nodefeature.ManufacturerNVIDIA)) + } + + return objs + } + + stored := func(name string) *workercore.ModelDeployment { + md := modelDeployment(workercore.ModelDeploymentEngineVLLM) + md.Spec.KVCache.PoolRef.Name = "chat" + md.Spec.Roles[0].InstanceType = name + + return md + } + + t.Run("a move across manufacturers is judged against the bound pool", func(t *testing.T) { + old := stored("ascend-type") + r := newModelDeploymentWebhookWith(live("ascend-type", "nvidia-type")) + if _, err := r.ValidateCreate(context.Background(), old); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + md := old.DeepCopy() + md.Spec.Roles[0].InstanceType = "nvidia-type" + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, "the mixed pool cannot serve the NVIDIA half of this binding") + assert.True(t, errsContain(err.Error(), "spec.kvCache.poolRef.name"), err.Error()) + }) + + t.Run("a move within one manufacturer is admitted", func(t *testing.T) { + old := stored("ascend-type") + objs := live("ascend-type", "") + objs = append(objs, typeFixture("ascend-type-2", nodefeature.ManufacturerAscend)) + r := newModelDeploymentWebhookWith(objs) + + md := old.DeepCopy() + md.Spec.Roles[0].InstanceType = "ascend-type-2" + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, "the constrained engine still reads every offered protocol") + }) +} + +// TestModelDeploymentWebhook_InterfaceRecheckJudgesOnlyTheMovedRoles pins the per-role scope of +// the interface recheck on a two-role object: a role whose own inputs did not move keeps the +// escape hatch its stored verdict provides when the fabric drifts, a role whose inputs did move +// is judged, and a shared router or transfer move rejudges every requesting role. +func TestModelDeploymentWebhook_InterfaceRecheckJudgesOnlyTheMovedRoles(t *testing.T) { + // build stores a two-role object whose server role requests an interface against an RDMA + // fabric, and drifted swaps the webhook's cluster for one whose store drifted to TCP. + build := func(t *testing.T) (*ModelDeploymentWebhook, *workercore.ModelDeployment) { + t.Helper() + objs := kvCacheFixture() + objs[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "RDMA" + objs = append(objs, servingInstanceType("h20-8x", 8)) + md := modelDeployment(workercore.ModelDeploymentEngineVLLM, + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "server" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + Interface: ptr.To(resource.MustParse("1")), + } + }), + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "worker" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + }), + ) + md.Spec.KVCache.PoolRef.Name = "chat" + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), md); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, servingInstanceType("h20-8x", 8)) + + return newModelDeploymentWebhookWith(drifted), md + } + + t.Run("a sibling's command edit does not rejudge an unchanged request", func(t *testing.T) { + r, old := build(t) + + md := old.DeepCopy() + md.Spec.Roles[1].Command = []string{"/bin/serve"} + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "the worker role's interface inputs did not move, so the drifted fabric is not re-judged against the server role") + }) + + t.Run("the moved role's own request is judged against the drifted fabric", func(t *testing.T) { + r, old := build(t) + + md := old.DeepCopy() + md.Spec.Roles[1].Resources.Interface = ptr.To(resource.MustParse("1")) + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, "the new request names no interface a TCP store can use") + assert.True(t, errsContain(err.Error(), "spec.roles[1].resources.interface"), err.Error()) + assert.True(t, errsContain(err.Error(), "no effective RDMA or EFA"), err.Error()) + }) + + t.Run("the requesting role's command edit rejudges its own request", func(t *testing.T) { + r, old := build(t) + + md := old.DeepCopy() + md.Spec.Roles[0].Command = []string{"/bin/serve"} + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, "a takeover command renders no managed leg for the request") + assert.True(t, errsContain(err.Error(), "spec.roles[0].resources.interface"), err.Error()) + }) + + t.Run("a shared transfer move rejudges every requesting role", func(t *testing.T) { + md := modelDeployment(workercore.ModelDeploymentEngineVLLM, + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "prefill" + r.Kind = workercore.ModelDeploymentRoleKindPrefill + r.Resources = &workercore.ModelDeploymentRoleResources{ + Interface: ptr.To(resource.MustParse("1")), + } + }), + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "decode" + r.Kind = workercore.ModelDeploymentRoleKindDecode + }), + ) + md.Spec.KVCache = nil + md.Spec.KVTransfer = &workercore.ModelDeploymentKVTransfer{Protocol: "EFA"} + md.Spec.Router = &workercore.ModelDeploymentRouter{Name: workercore.ModelDeploymentRouterLLMD} + r := newModelDeploymentWebhookWith([]ctrlcli.Object{servingInstanceType("h20-8x", 8)}) + if _, err := r.ValidateCreate(context.Background(), md); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + newer := md.DeepCopy() + newer.Spec.KVTransfer.Protocol = "TCP" + + _, err := r.ValidateUpdate(context.Background(), md, newer) + require.Error(t, err, "the transfer protocol is shared, so the unchanged prefill is rejudged") + assert.True(t, errsContain(err.Error(), "spec.roles[0].resources.interface"), err.Error()) + }) + + t.Run("a router that renders no direct leg keeps the drift escape", func(t *testing.T) { + r, old := build(t) + + md := old.DeepCopy() + md.Spec.Router = &workercore.ModelDeploymentRouter{Name: workercore.ModelDeploymentRouterVLLM} + + // The leg the rule rejudges on a shared edit is the one the production helper renders, + // and for plain Server roles it is empty on both sides of the edit: a router fronts no + // pair here, so the edit moves no effective direct input. + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(old, &old.Spec.Roles[0], + nodefeature.ManufacturerNVIDIA)) + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(md, &md.Spec.Roles[0], + nodefeature.ManufacturerNVIDIA)) + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "the added router cannot create a direct leg, so the drifted fabric is not re-judged") + }) + + t.Run("a transfer declaration no pair renders keeps the drift escape", func(t *testing.T) { + objs := kvCacheFixture() + objs[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "RDMA" + objs = append(objs, servingInstanceType("h20-8x", 8)) + md := modelDeployment(workercore.ModelDeploymentEngineVLLM, + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "server" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + Interface: ptr.To(resource.MustParse("1")), + } + }), + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "worker" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + }), + ) + md.Spec.KVCache.PoolRef.Name = "chat" + md.Spec.KVTransfer = &workercore.ModelDeploymentKVTransfer{Protocol: "EFA"} + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), md); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, servingInstanceType("h20-8x", 8)) + r = newModelDeploymentWebhookWith(drifted) + + newer := md.DeepCopy() + newer.Spec.KVTransfer.Protocol = "TCP" + + // Without a pair the declaration renders no leg on either side, so the protocol the + // object carries is inert for this role and the drifted fabric is not re-judged. + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(md, &md.Spec.Roles[0], + nodefeature.ManufacturerNVIDIA)) + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(newer, &newer.Spec.Roles[0], + nodefeature.ManufacturerNVIDIA)) + + _, err := r.ValidateUpdate(context.Background(), md, newer) + assert.NoError(t, err, + "the transfer declaration renders no leg, so the drifted fabric is not re-judged") + }) + + t.Run("a transfer edit a pair cannot render keeps the Ascend escape", func(t *testing.T) { + ascend := func(name string) ctrlcli.Object { + return servingInstanceType(name, 8, func(it *worker.InstanceType) { + it.Status.Detail.Manufacturer = nodefeature.ManufacturerAscend + }) + } + objs := kvCacheFixture() + objs[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "RDMA" + objs = append(objs, ascend("ascend-type")) + md := modelDeployment(workercore.ModelDeploymentEngineVLLM, + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "prefill" + r.Kind = workercore.ModelDeploymentRoleKindPrefill + r.InstanceType = "ascend-type" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + Interface: ptr.To(resource.MustParse("1")), + } + }), + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "decode" + r.Kind = workercore.ModelDeploymentRoleKindDecode + r.InstanceType = "ascend-type" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + }), + ) + md.Spec.KVCache.PoolRef.Name = "chat" + md.Spec.KVTransfer = &workercore.ModelDeploymentKVTransfer{Protocol: "EFA"} + md.Spec.Router = &workercore.ModelDeploymentRouter{Name: workercore.ModelDeploymentRouterLLMD} + r := newModelDeploymentWebhookWith(objs) + if _, err := r.ValidateCreate(context.Background(), md); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + drifted := kvCacheFixture() + drifted[2].(*workercore.KVCacheBackend).Spec.Transport.Protocol = "TCP" + drifted = append(drifted, ascend("ascend-type")) + r = newModelDeploymentWebhookWith(drifted) + + newer := md.DeepCopy() + newer.Spec.KVTransfer.Protocol = "TCP" + + // The helper renders no leg on Ascend whatever the routed pair declares, so the protocol + // edit moves no effective direct input for this role either. + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(md, &md.Spec.Roles[0], + nodefeature.ManufacturerAscend)) + assert.Empty(t, workerctrl.ModelDeploymentDirectInterfaceProtocol(newer, &newer.Spec.Roles[0], + nodefeature.ManufacturerAscend)) + + _, err := r.ValidateUpdate(context.Background(), md, newer) + assert.NoError(t, err, + "the Ascend pair renders no direct leg, so the drifted fabric is not re-judged") + }) +} + +// TestModelDeploymentWebhook_PoolTransportRecheckJudgesOnlyTheMovedRole pins the per-role scope +// of the pool-transport recheck: moving one role onto another type of the same manufacturer is +// judged for that role alone, so an unchanged role is not refused because the bound pool drifted +// into a mix its engine cannot read. +func TestModelDeploymentWebhook_PoolTransportRecheckJudgesOnlyTheMovedRole(t *testing.T) { + typeFixture := func(name, manufacturer string) ctrlcli.Object { + return servingInstanceType(name, 8, func(it *worker.InstanceType) { + it.Status.Detail.Manufacturer = manufacturer + }) + } + // live assembles the cluster the webhook reads; second adds the member group whose transport + // turns the pool into a CANN/RDMA mix, which is the drift an unconstrained engine cannot serve. + live := func(second string, types ...ctrlcli.Object) []ctrlcli.Object { + objs := kvCacheFixture() + backend := objs[2].(*workercore.KVCacheBackend) + backend.Spec.Transport.Protocol = "CANN" + if second != "" { + backend.Spec.Connection.Managed = &workercore.KVCacheBackendManaged{Members: []workercore.KVCacheBackendMember{ + {NodeSelector: map[string]string{"cache": "dram"}, Medium: "DRAM", CapacityPerMember: resource.MustParse("64Gi")}, + { + NodeSelector: map[string]string{"cache": "vram"}, Medium: "VRAM", CapacityPerMember: resource.MustParse("16Gi"), + Transport: &workercore.KVCacheBackendMemberTransport{Protocol: second}, + }, + }} + } + return append(objs, types...) + } + stored := func() *workercore.ModelDeployment { + md := modelDeployment(workercore.ModelDeploymentEngineVLLM, + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "server" + r.InstanceType = "nvidia-type" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + }), + role(func(r *workercore.ModelDeploymentRole) { + r.Name = "worker" + r.InstanceType = "ascend-type" + r.Resources = &workercore.ModelDeploymentRoleResources{ + Accelerator: resource.NewQuantity(1, resource.DecimalSI), + } + }), + ) + md.Spec.KVCache.PoolRef.Name = "chat" + + return md + } + build := func(t *testing.T, types ...ctrlcli.Object) (*ModelDeploymentWebhook, *workercore.ModelDeployment) { + t.Helper() + old := stored() + r := newModelDeploymentWebhookWith(live("", types...)) + if _, err := r.ValidateCreate(context.Background(), old); err != nil { + t.Fatalf("the stored object was valid at creation: %s", err) + } + + return newModelDeploymentWebhookWith(live("RDMA", types...)), old + } + + t.Run("an Ascend move spares the unchanged NVIDIA role", func(t *testing.T) { + r, old := build(t, + typeFixture("nvidia-type", nodefeature.ManufacturerNVIDIA), + typeFixture("ascend-type", nodefeature.ManufacturerAscend), + typeFixture("ascend-type-2", nodefeature.ManufacturerAscend), + ) + + md := old.DeepCopy() + md.Spec.Roles[1].InstanceType = "ascend-type-2" + + _, err := r.ValidateUpdate(context.Background(), old, md) + assert.NoError(t, err, + "the NVIDIA role's type did not move, so the pool's drift into a mixed offering is not re-judged against it") + }) + + t.Run("the moved role is judged against the drifted pool", func(t *testing.T) { + r, old := build(t, + typeFixture("nvidia-type", nodefeature.ManufacturerNVIDIA), + typeFixture("ascend-type", nodefeature.ManufacturerAscend), + typeFixture("nvidia-type-2", nodefeature.ManufacturerNVIDIA), + ) + + md := old.DeepCopy() + md.Spec.Roles[1].InstanceType = "nvidia-type-2" + + _, err := r.ValidateUpdate(context.Background(), old, md) + require.Error(t, err, "the Ascend role moved onto an engine that cannot read a CANN/RDMA pool") + assert.True(t, errsContain(err.Error(), "spec.kvCache.poolRef.name"), err.Error()) }) } diff --git a/specs/2026-10-08-model-deployment-role-mutability.md b/specs/2026-10-08-model-deployment-role-mutability.md new file mode 100644 index 000000000..00ef81748 --- /dev/null +++ b/specs/2026-10-08-model-deployment-role-mutability.md @@ -0,0 +1,579 @@ +# Spec: ModelDeployment Role Configuration Mutability + +Status: Shipped +Type: Feature + +Supersedes: the size-freeze rule in "Role Replica Admission Unit", Summary and Motivation / Goals 7. +Also supersedes ordinary-role freezes in "KV Cache and ModelDeployment — Per-Group Transport, API Consolidation, and Replaceable Replicas". +The affected sections are Motivation / Non-Goals, Notes / Constraints / Caveats, and Boundaries. +Only size, resources, instanceType, and command mutability changes. +Preserve the earlier specifications as historical records. + +## Summary + +Allow ordinary ModelDeployment roles to update size, resources, instanceType, and command. +Apply changed configuration through whole-replica Recreate with fresh Kueue admission. +Preserve healthy old replicas until their replacement starts. +Keep role identity and validation rules. +Document shared prefill/decode effects and preserve Elastic EP restrictions. + +## Motivation + +### Goals + +- Let workload owners change instance shape and accelerator requests without creating another deployment. +- Support pool changes and managed-command transitions. +- Replace affected replicas one at a time. +- Preserve unrelated Pods, Workloads, and serving selection. +- Keep quota accounting, group cleanup, and update status accurate. + +The source baseline is commit `5ca7e36bc1d6667654aeb98faa2034cf913e9b37`. +The supplied handoff requests mutable role configuration. +Source inspection found these additional requirements: + +- The [update webhook](../pkg/worker/webhooks/worker/model_deployment.go) freezes all four fields. + Interface validation currently reruns only for router or transfer changes. +- [Convergence](../pkg/worker/controllers/worker/model_deployment.go) compares existing member counts with desired size. + A size difference therefore enters incomplete-group repair and bypasses the admission guard. +- [Qualification](../pkg/worker/controllers/worker/model_deployment_health.go) also uses desired size. + All outdated ordinals enter pending replacement. + Active qualification selectors can therefore withdraw every old managed Server endpoint together. +- [Service rendering](../pkg/worker/controllers/worker/model_deployment_service.go) uses desired size for peer DNS and leader selection. + A decrease to one member can remove old peer Services and expose old followers through legacy selectors. +- [Retirement](../pkg/worker/controllers/worker/model_deployment_retirement.go) requires old replicas to match the current role's shape. + Size or command changes can prevent classification. +- [The existing pair test](../pkg/worker/controllers/worker/model_deployment_test.go), TestModelDeployment_DegreeEditRollsThePair, asserts shared effects. + Ascend TP/DP changes move both roles' hashes; ordinary argument changes affect one role. + +These are observations from the source baseline before implementation. + +### Non-Goals + +- Changing the role map key or allowing role additions, removals, renames, or kind changes. +- Relaxing model, engine, cache, or router identity rules. +- Adding surge settings, revision APIs, or another workload controller. +- Resizing a running engine collective in place. +- Adding Elastic EP configuration replacement. +- Guaranteeing uninterrupted inference during Recreate or shared transfer changes. + +## Proposal + +Make the four fields editable for ordinary Server, Prefill, and Decode roles. +New configuration determines replacement Pods and their admission. +Existing Pods keep their deployed configuration. + +Different ordinals can run different configurations during replacement. +Replace every member of a selected replica and remove its old Workload. +Create its replacement after the old group and Workload leave. +Continue only after fresh Kueue admission. + +A role is affected when its rendered configuration changes. +Shared transfer configuration can affect both roles. +Keep an unrelated role unchanged when its rendered configuration stays unchanged. + +### User Stories + +#### Story 1 + +As a workload owner, I want editable size and resources, so that I can change instance shape and allocation. + +#### Story 2 + +As a workload owner, I want editable instanceType and command, so that I can change placement and execution. + +#### Story 3 + +As a prefill/decode operator, I want gradual replacement, so that unrelated capacity remains available. + +#### Story 4 + +As an operator, I want documented shared effects and constraints, so that I can predict update behavior. + +### Core Features & Acceptance Criteria + +#### Mutability and identity + +- Accept valid size, resources, instanceType, and command updates on ordinary roles. +- Cover size increases, decreases, and transitions across one member. +- Cover accelerator count, slicing, partition, and interface changes when valid. +- Cover managed-to-command, command-to-managed, and command-to-command transitions. +- Keep roles keyed by unique name. List reordering preserves identity. +- Reject additions, removals, renames, and kind changes. +- Allow repeated Server roles. Allow at most one Prefill and one Decode. +- Reject mixing Server with Prefill or Decode. +- Keep size within 1 through 64 and preserve all configuration validity rules. + +#### Replacement and admission + +- Replace whole replicas. Do not trim or append members within an old admitted group. +- Permit at most one unresolved configuration replacement per affected role. +- Wait for fresh replacement admission before deleting another healthy old replica. +- A queued replacement does not authorize further healthy deletions. +- Read an old replica's completeness from its deployed shape. +- Desired-size differences alone do not qualify for incomplete-group repair. +- Preserve recovery for actual member loss and interrupted writes. +- Do not require obsolete, unadmitted configuration to become admitted before it can be replaced. +- Apply this rule to complete groups waiting for first admission, including those without a replacement slot. +- Select these groups before admitted siblings and use the same single-slot cleanup path. +- Verify current member admission gates and exclude earlier execution, eviction, and preemption history. +- Recovery preserves replacement limits and does not spend additional healthy serving capacity. +- Reconciliation retries and restarts never overlap old and new members in one ordinal. +- Combined replica-count and configuration edits preserve these ownership and cadence rules. + +#### Cleanup and mixed configurations + +- Remove every selected old member and its Workload, including when old size exceeds new size. +- Verify old group vacancy and Workload removal before creating a replacement. +- Compose a new Workload with the new count, template, resource request, and queue. +- Do not reuse the old reservation for changed scheduling inputs. +- Keep untouched replicas' Workloads and reservations. +- Preserve accurate resource attribution during pool changes and mixed configurations. +- Distinguish outdated configuration from an active replacement. +- Keep healthy old replicas eligible until their individual replacement starts. +- Preserve peer DNS until all old members that need it have left. +- Keep legacy Service leader selection correct while multi-member old replicas remain. +- Give new multi-member replicas the required peer DNS and rank metadata. +- Preserve External DP answering-member rules. +- Read old replica size and execution shape for retirement. +- Desired configuration differences alone do not prevent retirement. +- Preserve health revocation, drain, ownership, and unsupported-observation rules. +- Joint admission recognizes existing groups by their deployed shape. +- A configuration edit does not park or revoke an already-admitted sibling by itself. + +#### Shared effects, validation, and documentation + +- For independent edits, unchanged roles retain Pod UIDs, hashes, member sets, and Workload UIDs. +- Their serving selection and peer DNS remain correct. +- For shared edits, replace all roles whose rendered configuration changes. +- Preserve Ascend transfer TP/DP propagation, including command-supplied degrees. +- Keep shared configuration in fingerprints. +- Document possible transfer incompatibility while both roles converge. +- Retain defaulting, resource-mode, parallel-width, interface, transport, port, naming, host-access, and owned-key checks. +- Recheck dependencies changed by resource, pool, and command updates. +- Preserve escape hatches for unchanged configuration when external dependencies drift. +- Rejected updates change no child resources. +- Preserve Elastic EP's managed command, size=1, replicas=1, and accelerator-to-TP constraint. +- Preserve Elastic EP presence, TP, maximum DP, and width-update restrictions. +- Ordinary relaxation does not enable previously forbidden Elastic EP instanceType or resources edits. +- Update the owning guide, API comments, and generated descriptions. +- Explain role identity, shared effects, cadence, quota waits, and Recreate downtime. +- Explain merge-patch list replacement and omitted-field effects under the mutable contract. +- Register this spec in the Model Deployment module map. +- Verify actual Pods, Workloads, selectors, endpoint membership, and peer Services. +- Hash comparisons alone do not establish safe replacement. + +### Notes / Constraints / Caveats + +- Use the existing Go, Kubernetes, controller-runtime, and Kueue stack. +- Keep public resources at worker.gpustack.ai/v1; v1alpha1 remains internal storage. +- Keep size, resources, and instanceType as structured scheduling inputs. +- Use rendered fingerprints to identify configuration differences. +- Use observed group metadata and ownership to establish deployed facts. +- Fresh admission can wait for capacity. Keep additional healthy replicas while it waits. +- A single-replica role can lose capacity during Recreate. +- Admission advances replacement before engine readiness. Multiple replicas do not guarantee continuous serving capacity. +- Changed ports, commands, hardware, or shared transfer configuration can disrupt requests. +- An unchanged sibling Pod does not guarantee complete inference while its peer is unavailable. +- Elastic EP keeps its existing behavior outside the new replacement contract. + +### Boundaries + +- **Always:** preserve identity, validation, whole-group replacement, ownership, quota accounting, and cleanup. +- **Always:** distinguish observed state, desired configuration, replacement progress, and serving capacity. +- **Ask first:** expand Elastic EP replacement, change public identity rules, or run cluster builds, deployments, and end-to-end verification. +- **Never:** mix old and new members within one admitted group or reuse old scheduling admission. +- **Never:** weaken fingerprints or health checks to conceal incompatible configuration. + +### Risks and Mitigations + +- Desired size condemns healthy old groups → evaluate deployed completeness separately. +- Repair bypasses rollout cadence → reserve repair for actual broken membership. +- Peer DNS or leader selection changes early → retain old dependencies through group departure. +- A queued replacement drains remaining capacity → wait for fresh admission before further healthy replacement. +- Changed requests reuse stale admission → remove the old Workload and compose a new one. +- Command changes confuse retirement → read old execution from deployed Pods. +- Shared degrees become stale on one side → retain coupled replacement and document the compatibility window. +- Elastic edits are accepted but ignored → preserve profile-specific refusals. +- A restart loses replacement selection → persist the active slot before deleting members. +- Stale named Workloads lose their owners → bind cleanup to proven member and Workload UIDs. +- A shape edit interrupts retirement → classify the reserved members from their deployed facts. +- Surplus arithmetic trims old groups → identify duplicate seats separately from desired-size differences. + +## Design Details + +### Commands + +Validation runs locally, using the current worktree's source. +The user confirmed unit tests, lint, generation, and compilation. +The user separately authorized cluster tests on owned temporary accelerator infrastructure. + +Run focused task tests from the repository root: + +```bash +GODEBUG=gotypesalias=0 CGO_ENABLED=1 go test -tags 'goccy netgo' -race -count=1 ./api/worker/v1alpha1 ./pkg/worker/webhooks/worker ./pkg/worker/controllers/worker +make lint +make lint docs +make build +``` + +Collect package coverage with `go test -cover` during final validation. +General lint can edit source. Run it after all parallel writers settle. +Regenerate after its final source changes. + +Generation requires a resolved checkout path ending in `gpustack.ai/gpustack`. +Use a detached local generation worktree under a task-specific directory beneath `${HOME}/.gpustack-gen/`. +Pin it to the current source commit after local source checks pass. +Confirm its commit and edited source content before invoking `make generate`. +Copy only generator outputs back to the implementation worktree. +Run generation twice and compare output contents to prove deterministic parity. +Preserve the shared main checkout. + +The build uses the host platform unless BUILD_PLATFORMS is set. +Use the authorized remote builder for packaged runtime validation. +Publish a development image tied to the source commit and record its tested digest. +Run gpustack-operator-e2e against the isolated temporary cluster. + +### Project Structure + +- `api/worker/v1alpha1/model_deployment.go`: internal contracts and generation inputs. +- `api/worker/v1/model_deployment.go`: public resource type. +- `pkg/worker/webhooks/worker/model_deployment*.go`: identity, profile, and configuration validation. +- `pkg/worker/controllers/worker/model_deployment*.go`: render, rollout, admission, health, retirement, Services, and status. +- `docs/modules/model-deployment/deployment.md`: owning update and rollout guide. +- `docs/modules/model-deployment/prefill-decode.md`: shared transfer behavior. +- `docs/modules/model-deployment/shutdown.md`: departure behavior. +- `docs/modules/model-deployment/status.md`: progress and capacity interpretation. +- `docs/modules/model-deployment/elastic-ep.md`: retained profile restrictions. +- `docs/README.md`: shared routing. +- `specs/`: versioned design records. + +### Code Style + +Use focused Go helpers, explicit errors, table-driven tests, and fake clients. +Assert observable state. Keep reconciliation level-based and repeatable. +Avoid a public revision field for internal bookkeeping. + +Existing [group cleanup](../pkg/worker/controllers/worker/model_deployment_pod_group.go) treats repeated absence as success: + +```go +for _, wl := range wls { + if err = r.Client.Delete(ctx, wl); err != nil && !kerrors.IsNotFound(err) { + return fmt.Errorf("delete workload %s: %w", wl.Name, err) + } +} +``` + +### Implementation Design + +Keep desired rendering separate from deployed replica facts. +Leave modelDeploymentRoleSize as the desired-size helper. +Add a private replica reader for observed membership and execution. + +Read the group total, member indices, ownership, queue, and matching Workload. +Require consistent totals and unique member seats. +Preserve the legacy single-member fallback only when its metadata supports that shape. +Unknown or contradictory observations do not authorize deleting healthy capacity. +Unreadable shape or execution keeps health and cache observations Unknown. +It does not grant new endpoint eligibility. +Preserve existing eligibility through the established unknown-observation policy. +Readable member loss remains a positive health failure. + +Use deployed facts in health, retirement, joint admission, ready counts, and cache observation. +Preserve deployed command mode. A desired managed command does not make an old takeover Pod managed. +A changed desired size or command does not rewrite those facts. +Preserve all supported-version, health-revocation, and drain guards. + +Record one active replacement slot per affected role before deleting its old members. +Use private controller metadata on ModelDeployment when observation alone cannot recover that slot. +Keep the record small and use optimistic concurrency. +It is internal bookkeeping, not a user setting or public revision API. + +Recover the same slot through partial deletion, vacancy, creation, and admission. +Read Pods and Workloads through the API reader before reusing an ordinal. +Remove old Workloads by proven ownership and verify their absence. +A derived group name locates a Workload; it does not authorize deleting a foreign object. +Bind cleanup to captured member and Workload UIDs. +A new Workload must own the replacement members and describe their scheduling inputs. + +Clear the active slot only after complete current members receive fresh Workload admission. +A newer configuration can replace an obsolete queued configuration in the same active slot. +It does not authorize another healthy old replica's deletion. +Real membership failures remain repairable without spending other healthy capacity. +A broken old replica can wait for the role's active replacement to receive admission. +Then replace the broken replica as a whole before selecting another healthy old replica. +Do not add new-configuration members beside surviving old-configuration members, even when their sizes match. +An interrupted create can fill missing members only within its current replacement configuration. +Record new member UIDs during creation, before the next reconciliation observation. +If the active replacement loses members, recover that same slot as a whole. +Retain its captured Workload identity when all members disappear or a cleanup response is lost. +Replica-count edits cancel or finish only the affected slot after cleanup. + +Qualification distinguishes an outdated replica from a selected replacement. +Keep old healthy eligibility until selection. +During command-mode changes, preserve each deployed mode's routing rules. +Managed replicas keep engine qualification. Takeover replicas keep legacy Service membership and remain Unmanaged. +Preserve deployed answering selection from the first edit, before the replacement configuration appears. +If a shared selector needs private routing labels, write those labels before switching the Service. +Routing membership does not establish engine health or cache qualification. +Retain peer Services required by actual members, including terminating members. +Read those members uncached before synchronizing Services, so informer lag cannot retire their peer DNS. +Keep leader selection valid while old multi-member and new single-member groups coexist. +Preserve External DP answering-member selection. +Role-level parallelism remains declared configuration; it does not describe every running replica during replacement. + +Revalidate interface and transport inputs when their dependencies change. +Compare roles by name, so list reordering does not trigger a false dependency change. +Retain unchanged-configuration escape hatches for external dependency drift. +Keep Elastic EP restrictions through an explicit profile-specific identity path. + +### Implementation Plan + +The user authorized unattended implementation with mcode and qwen. +Keep the current working branch and use local validation. +Workers own disjoint files. The coordinator owns commits and generated artifacts. +The implementation tasks do not create a PR. +Cluster acceptance follows separate user authorization. + +- [x] **T1 · Deployed replica facts and observation** + Blocked by: None + Owns: `pkg/worker/controllers/worker/model_deployment_pod_group*.go`, `pkg/worker/controllers/worker/model_deployment_group_shape*.go`, `pkg/worker/controllers/worker/model_deployment_render*.go`, `pkg/worker/controllers/worker/model_deployment_health*.go`, `pkg/worker/controllers/worker/model_deployment_retirement*.go`, `pkg/worker/controllers/worker/model_deployment_joint_admission*.go`, `pkg/worker/controllers/worker/model_deployment_cache_attached*.go`, `pkg/worker/controllers/worker/model_deployment_status*.go`, `pkg/worker/controllers/worker/model_deployment_answering_shape_test.go`, `pkg/worker/controllers/worker/model_deployment_service_test.go` + Gate: review + Acceptance: Read deployed size and execution without desired-shape substitution. Reject duplicate seats and contradictory group totals. Preserve existing unchanged-shape behavior and unknown-observation guards. Demonstrate old healthy groups remain complete and observable after a desired size or command edit. + Verify: `GODEBUG=gotypesalias=0 CGO_ENABLED=1 go test -tags 'goccy netgo' -race -count=1 ./pkg/worker/controllers/worker` + +Checkpoint: validate the replica reader and its negative controls before changing replacement cadence. + +- [x] **T2 · Ordinary role update contract** + Blocked by: None + Owns: `api/worker/v1alpha1/model_deployment.go`, `api/worker/v1alpha1/model_deployment_test.go`, `pkg/worker/webhooks/worker/model_deployment*.go`, `docs/modules/model-deployment/deployment.md`, `docs/modules/model-deployment/prefill-decode.md`, `docs/modules/model-deployment/shutdown.md`, `docs/modules/model-deployment/status.md`, `docs/modules/model-deployment/elastic-ep.md` + Gate: review + Acceptance: Accept all four ordinary fields while preserving name-map identity and kind rules. Cover size bounds, resources, pool dependencies, and all command transitions. Revalidate changed dependencies without rejecting unchanged inputs due to external drift. Preserve every Elastic EP refusal. Document mixed configurations, whole-group Recreate, shared effects, merge-patch omission, and quota waits. + Verify: `GODEBUG=gotypesalias=0 CGO_ENABLED=1 go test -tags 'goccy netgo' -race -count=1 ./api/worker/v1alpha1 ./pkg/worker/webhooks/worker` + +- [x] **T3 · Recoverable whole-replica replacement** + Blocked by: T1 + Owns: `pkg/worker/controllers/worker/model_deployment*.go`, supporting `go.mod` and `go.sum` updates through the coordinator + Gate: review + Acceptance: Persist and recover one unresolved slot per affected role. Verify vacancy and old Workload absence before creation. Wait for fresh admission before another healthy deletion. Replace obsolete queued configuration within the same slot. Preserve actual member-loss recovery, untouched roles, mixed-shape Services, peer DNS, truthful status, and joint admission. Cover lost responses, restarts, partial deletion, repeated edits, and concurrent replica-count changes with observable object assertions. + Verify: `GODEBUG=gotypesalias=0 CGO_ENABLED=1 go test -tags 'goccy netgo' -race -count=1 ./pkg/worker/controllers/worker` + +Checkpoint: T2 and T3 must both pass before the mutable contract is ready for use. + +- [x] **T4 · Integrated validation and generated contract** + Blocked by: T2, T3 + Owns: `api/**/zz_generated.*`, `api/**/generated.pb.go`, `api/**/generated.proto`, `pkg/worker/webhooks/worker/zz_generated.*`, `pkg/kubeclients/**`, `specs/2026-10-08-model-deployment-role-mutability.md` + Gate: review + Acceptance: Review both Standards and Spec against the integrated diff. Resolve material defects through the owning tasks. Run local race tests, general lint, code generation, documentation lint, and compilation. Confirm a second generation leaves no artifact drift. Record runtime scenarios still awaiting user testing. Stop before PR creation. + Verify: run the complete Commands sequence and review the generated diff. + +- [x] **T5 · Workload placement pins and replacement admission** + Blocked by: T3 + Owns: `pkg/worker/controllers/worker/model_deployment_replacement_slot.go`, `pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go` + Gate: review + Acceptance: Accept Workload-only fit and model-manager pins. Preserve member placement constraints, template, resource, owner, count and queue checks. Demonstrate RED/GREEN regressions and compare an actual admitted slice snapshot without modifying it. + Verify: focused admission regressions, `make test`, `make lint`, and `make build`. + +- [x] **T6 · Authorized runtime acceptance** + Blocked by: T5 + Owns: `.agents/skills/gpustack-operator-e2e/cases/case-115.sh`, runtime evidence, affected guides and this spec + Gate: review + Acceptance: Include T5 in the tested image. Verify whole-card to slice, percentage changes, and slice to whole-card replacement with fresh objects and slot release. Finish the role-update scenarios and record their runtime limits. + Verify: case 115, actual engine inference, role-update receipts, documentation lint, and fixture cleanup readback. + +### Test Plan + +[x] The involved component owners may require updates to existing tests before this enhancement can land. + +#### Prerequisite testing updates + +Add an honest Kueue composition fixture for new rollout tests. +It waits for the declared member total and consistent scheduling inputs. +Reuse the pinned Kueue group constructor when building PodSets. +Keep its required API dependencies at the versions selected by the existing Kueue pin. +Preserve role-hash grouping; member commands can differ within one role. +Give every Pod and Workload a distinct UID. +Keep existing Workload ownership and finalizers visible. +Prove the fixture rejects incomplete groups and stale reservation adoption. +Use API-reader/client separation and intercepted writes for lost-response tests. + +#### Unit tests + +Measured package coverage appears under Local validation. +No pre-change baseline was measured. + +- `api/worker/v1alpha1`: size defaults and bounds; unchanged role list-map identity. +- `pkg/worker/webhooks/worker`: ordinary four-field edits, role reorder, invalid identity/kind, resource modes, command transitions, changed interface/transport dependencies, drift escape hatches, and Elastic restrictions. +- `pkg/worker/controllers/worker`: consistent deployed facts, legacy groups, duplicate seats, missing members, old execution, qualification, status attribution, joint admission, cache observation, peer Services, and leader/External DP selectors. +- `pkg/worker/controllers/worker`: preserved pair fingerprint tests for independent and shared Ascend changes. + +Each changed contract has a repeatable regression: + +| Contract | Local regression source | Live check | +|---|---|---| +| Ordinary size, resources, instanceType and command updates | [Webhook update cases](../pkg/worker/webhooks/worker/model_deployment_test.go) and [four-field lifecycle cases](../pkg/worker/controllers/worker/model_deployment_lifecycle_test.go) | Case 115; real-engine inference is checked separately | +| Initially queued resource or pool updates without a replacement slot | [Initial-queue regressions](../pkg/worker/controllers/worker/model_deployment_initial_queue_test.go) | Case 115 queued resource reduction | +| Independent P/D edits preserve the sibling | [Sibling lifecycle cases](../pkg/worker/controllers/worker/model_deployment_lifecycle_test.go) | Case 115 CPU P/D size and command checks | +| Shared rendered degrees may replace both roles | [Pair fingerprint cases](../pkg/worker/controllers/worker/model_deployment_test.go) | Shared Ascend behavior remains outside NVIDIA runtime evidence | +| Queued edits, restart recovery and fresh admission | [Lifecycle recovery cases](../pkg/worker/controllers/worker/model_deployment_lifecycle_test.go) and [admission controls](../pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go) | Case 115 checks current admission and complete group identities | +| Workload-only placement pins release an admitted replacement | [Admission pin regressions](../pkg/worker/controllers/worker/model_deployment_replacement_slot_test.go) | Case 115 whole-card to slice, percentage changes and slice to whole-card | +| Role map identity, kind rules and Elastic EP restrictions | [API schema cases](../api/worker/v1alpha1/model_deployment_test.go), [identity cases](../pkg/worker/webhooks/worker/model_deployment_test.go) and [Elastic cases](../pkg/worker/webhooks/worker/model_deployment_elastic_test.go) | Existing admission cases remain controls; case 115 does not exercise Elastic EP | + +#### Integration tests + +Use fake-client reconciliation sequences for these local integration checks: + +- Change size through increase, decrease, one-to-many, and many-to-one. +- Wait with a queued replacement while every other healthy old group remains. +- Replace resources and instanceType with a new Workload request and queue. +- Exercise managed-to-command, command-to-managed, and command-to-command updates. +- Restart after intent persistence, partial member deletion, old Workload deletion, and partial creation. +- Recover lost delete/create responses and an API-reader/cache disagreement. +- Edit configuration again while its replacement is queued. +- Reduce resources or change pools while a complete initial group waits without a replacement slot. +- Keep admitted siblings intact when an initial queued group changes configuration. +- Reject initial-queue classification after execution, eviction, preemption, or removal of the admission gate. +- Combine configuration changes with replica growth and shrink. +- Preserve unrelated P/D child UIDs, selectors, endpoints, and DNS. +- Replace both roles when command-supplied shared degrees alter both renders. +- Recover genuine member loss during an active configuration replacement. +- Recover all active replacement members disappearing before the next pass without losing Workload cleanup authority. +- Repair an ordinal-less legacy Pod while an active replacement holds its slot. +- Retain peer DNS when the cache omits standing or terminating members still present on the API server. +- Compare complete HTTP Service selectors against deployed answering members, including the first edit, mixed modes, qualification revocation, and Unknown observations. + +These tests exercise reconciliation and objects. They do not establish runtime engine or Kueue behavior. + +#### Local validation + +Checks passed on implementation commit `dc81b055f77cd07ebde0382e531dacba442d0a51`. + +- `make test`: all 99 packages with tests passed, with race detection, shuffled order, and coverage. +- `make lint`: passed after all writers settled. +- `make lint docs`: passed, including site links and module routing. +- `make generate`: two passes passed on the exact implementation commit in a detached generation worktree. + All 1,468 generated file contents matched between passes and matched the implementation worktree. + No artifact drift remained after the final source lint. +- `make build`: passed for the host target, darwin/arm64, with CGO enabled and goccy/netgo tags. +- `git diff --check`: passed. + +Measured coverage: controller 84.2%; webhook 90.2%; API storage package 4.9%; repository total 25.5%. +The repository aggregate includes generated code. No pre-change baseline supports a coverage comparison. + +Standards review corrections aligned Service reads and status documentation with their implemented contracts. +Spec review corrections passed regressions for legacy Pods, first-edit routing, and peer DNS during cache lag. +HTTP routing tests assert complete selectors against both qualified and revoked members. + +These results establish local object behavior and build validity. +The role-update runtime checks are recorded below. +Local results alone do not establish Kueue, engine, network, or inference behavior. + +#### e2e tests + +The user authorized the operator e2e procedure before PR creation. +Keep the tested source commit, image digest, role-update results, and fixture cleanup receipts in runtime evidence. + +Use a deployment with at least two replicas and enough accelerators for multi-member groups. +Observe Pod and Workload UIDs, PodSet counts/templates, queue assignments, admission, and reserved resources. +Cover all size transitions, pool/resource updates, command transitions, queue contention, and interrupted cleanup. + +Case 115 supplies repeatable live scheduling checks for ordinary role updates. +It covers an initially queued resource edit without a replacement slot, complete size replacements, and takeover command changes. +Logical slice capacity enables whole-card to slice, percentage changes, and slice to whole-card updates. +These legs require fresh Pods and Workloads, matching slice resources, and replacement slot release. +A second accelerator group enables its cross-pool InstanceType leg. +A CPU type enables P/D size and command updates while verifying the sibling keeps both Pod and Workload UIDs. +Each update reads the same deployment UID, fresh Pod and Workload UIDs, PodSet count and template, queue, and current admission. +Its initially queued resource leg needs two total whole cards; its size leg needs two free cards. +A one-card pool still checks command, cross-pool, and CPU P/D updates. +It asserts cleanup and reports missing geometry as a limited SKIP. +It uses placeholder commands, so engine inference remains a separate runtime check. + +Local lifecycle regressions cover queued replacements across passes and restarts, a second edit, captured cleanup, simultaneous four-field updates, and sibling-role identity. +The initially queued regressions include positive replacements and execution-history guards. +Webhook tests retain role identity, kind rules, configuration validity, and Elastic EP constraints. +Verify role endpoints select answering members and old peer DNS lasts through departure. +Verify independent P/D edits preserve the sibling and shared Ascend edits roll both roles. +Exercise real joint admission and finalizer delays. +Keep a supported Elastic EP deployment as the unchanged-profile control. +Record inference interruption and shared transfer incompatibility without claiming continuous service. + +#### Recorded runtime checks + +The following earlier campaign checks completed on implementation commit +`8fba3f3ac6635accd0ace52c05e0dcaa463ef740`. +The running operator image digest is +`sha256:db20543ed10da503749391602b422b8d098d40ad751342a818e3ed1fa7052f16`. +The engine fixture uses Qwen/Qwen2.5-0.5B-Instruct with vLLM 0.29.0 on NVIDIA RTX PRO 6000 Blackwell Server Edition. +Each accelerator node has one whole GPU. + +| Check | Observed result | Evidence limit | +|---|---|---| +| Initially queued accelerator request changes from two to one | Same deployment UID; obsolete Pod and Workload removed; fresh group admitted; ordinary and SSE inference passed | The pool had two total cards across one-card nodes; no replacement slot existed before the edit | +| Two-replica Server size changes from one to two and back | Each selected replica was replaced completely; fresh admission, member selection and inference passed | Four accelerator nodes were available for the two-member shape | +| One-replica Server size ladder, one to two to three to two to one | Fresh groups admitted and ordinary and SSE inference passed at each step | Recreate has a serving gap; these checks do not establish continuous inference | +| Managed command to takeover, takeover to managed, and takeover argument changes | Deployed commands, group replacement and ordinary and SSE inference passed | The takeover fixture supplies its own pipeline-parallel launch; operator status remains unmanaged | +| Case 115 cross-pool InstanceType update | Fresh Workload uses the new queue; allocation UUIDs belong to the selected physical node and accelerator group | Placeholder command; no engine inference in this leg | +| Case 115 Prefill size increase, decrease and command update | Decode Pod and Workload UIDs remain unchanged while Prefill receives a fresh admitted group | CPU placeholders establish sibling identity preservation, not live P/D inference or DNS resolution | + +Case 115 also verifies full cleanup, including Workloads whose owner Pod has already disappeared. +Its final two-pool run passed all applicable legs and cleanup. +The two one-card pools cannot run its two-card resource and size legs; those rows remain explicit SKIPs. +Earlier checks ran these legs while the larger pool was available. + +Case 115 also completed on worker commit `836d929e6cdee3f1d00ba9d536cf6a8ebd976b97`. +Its worker image digest is `sha256:243f604133518d4a0720505e1e0674c2e606bcf58fc0cb1bd98f2eaabd5cc554`. +Device Manager and model-manager retained the preceding image during this check. +The worker's binary version and running image identity were read from the deployed container. + +| Check | Observed result | Evidence limit | +|---|---|---| +| Whole card to 50% slice, 50% to 40%, then whole card | Fresh complete groups and admitted Workloads; matching slice requests and limits; replacement slots released | Ubuntu shell placeholders establish scheduling convergence, not engine inference | +| Command and cross-pool updates | Same deployment identity; fresh member and Workload identities; allocation belongs to the selected accelerator group | Two disjoint one-card pools; initial queued reduction and Server size legs explicitly skipped | +| Prefill size one to two and back, then command update | Fresh Prefill groups; Decode Pod and Workload identities retained; owned objects absent after cleanup | CPU placeholders; no P/D transfer or serving continuity claim | + +The first slice run used a Busybox placeholder whose container exited with code 127. +Its admitted replacement released the slot, but the Ready check failed. +The corrected Ubuntu fixture passed all applicable legs and cleanup. +The failed run remains in the campaign evidence. +Local checks on the updated worker also passed `make lint`, `make test`, and `make build`. +The test run covered all 99 packages with tests, with race detection and shuffled order. + +#### Final role-update checks + +The final operator image includes the uncommitted peer DNS fix above worker commit +`836d929e6cdee3f1d00ba9d536cf6a8ebd976b97`. +Its digest is `sha256:ea82d32d32e9f5ec18c8e3dd53a5d59ce9ac7a1f760b573a0ea3faa77d041d79`. +The binary version still reports the committed worker revision. +Source hashes tie the tested Go files to the image build inputs. +The three Go files match the reviewed patch; full race/shuffle tests, lint and build passed. +Peer Services cover desired ordinals and standing members, including surplus members awaiting retirement. +Services are reclaimed after their members leave. + +Case 115 completed with two whole cards in one pool: 13 PASS and one cross-pool SKIP. +The initially queued two-card request changed to one card and admitted without an existing replacement slot. +The size one-to-two-to-one sequence and whole-card-to-slice transitions also passed. +The prior two-pool run supplies the separate cross-pool result. +Real inference passed the Server slice-update sequence and the Prefill whole-card-to-slice update. +The role-update fixtures were removed after their checks; placeholders retain their stated limits. + +Shared Ascend degree changes remain covered by local regressions only. +This hardware has no Ascend devices. Role updates involving native RDMA transport were not tested live. +Hardware partition resource changes remain covered by local validation only. + +## Alternatives + +- Remove only webhook freezes: rejected because cadence, health, retirement, and Services still assume immutable size. +- Require another deployment: rejected because it does not provide updates to the existing object. +- Resize admitted groups in place: rejected because membership and reservation belong to the old configuration. +- Add a rollout API or another controller: deferred because existing replica admission provides the required boundary. +- Use only per-pass deletion counts: rejected because queued groups can accumulate unresolved replacements across passes. +- Use private controller metadata: chosen for replacement progress that must survive an empty ordinal. +- Guarantee isolation for every edit: rejected because shared transfer configuration must remain consistent. +- Include Elastic EP replacement: deferred by the confirmed scope decision. + +## Open Questions + +No product-scope questions remain. +The implementation plan and local test environment are resolved. +The authorized role-update runtime checks have ended with the evidence limits recorded above. +Final code review is complete; this branch is prepared for PR submission.