Design: in-cluster scaling channel for PD slices #5

Activity

1. remotego

   changed the title [-]Design: in-cluster scaling channel for PD slices (helm upgrade is too heavyweight)[/-] [+]Design: in-cluster scaling channel for PD slices[/+] on Oct 19, 2026
2. lumianph

   assigned

   aceforeverd

   on Oct 22, 2026
3. huskcass commented on Oct 22, 2026

   

   huskcass

   on Oct 22, 2026

   More actions

   Proposal: a SliceSet workload CRD owned by llm-operator (scale channel = SliceSet.spec.replicas)

   Summary. None of A–D gives us the right ownership model. What is missing is a StatefulSet-shaped workload whose unit is a whole PD slice: today the chart expands one slice's objects with range $s := until $slices loops, so the slice count only exists at render time. We propose a new namespaced CRD, SliceSet (serving.modelsphere.dev/v1alpha1), reconciled by llm-operator:

   • spec.replicas = slice count, with a /scale subresource. The chart renders SliceSet + LLMScaler(targetRef: SliceSet), and the scaler writes one integer. No helm involvement per scale step.
   • spec.template = the full shape of one slice: engines (a DS spec), router (Deployment spec), per-role headless Services, per-slice PDBs.
   • Ordinal identity s<i> like StatefulSet. Scale-up fills the lowest missing ordinal; scale-down removes the highest and tears it down in a fixed order (out of routing → router → engines).
   • Each ordinal is its own DisaggregatedSet with slices: 1. We do not wrap one DS with slices: N (reasons below).
   • Release-scoped singletons stay in the chart: front-door Service, mc-master, ModelRoute, ServiceMonitor, LLMSLORequirement, LLMScaler.
   • helm upgrade (via LLMService) stays the path for shape changes. Slice count becomes an operational knob.

   What we checked (sources)

   • DS has no /scale subresource. DisaggregatedSet only declares +kubebuilder:subresource:status (types.go#L402). KEP-849 explicitly rejected /scale on DS mapped to spec.slices ("could ship in a future KEP"; KEP-849 Alt 2). The only DS-level /scale is the per-role DisaggregatedSetRoleScaler (scaling.mode: External). Its webhook rejects slices > 1 while any role is External (webhook#L155-L160).
   • Slices roll independently and concurrently, with no cross-slice budget. The controller loops for slice := range sliceCount { reconcileSlice(...) }, and each slice runs its own revision transition (controller#L173). Combined with our chart default maxSurge: 0, maxUnavailable: <role replicas> (atomic per role, pd-ds.yaml#L62), an image bump today takes every slice down at once. The chart comment's assumption that HA lives "at the tier above (slices…)" does not hold under DS semantics.
   • Slice scale-down deletes the highest slices' LWS directly, with no drain (KEP-846 Scale-Down, controller#L446). slices is excluded from the revision hash, so changing it never triggers a rollout.
   • Labels: pods carry disaggregatedset.x-k8s.io/{name,role,slice,revision} (+ subrole) and env DISAGGREGATEDSET_{NAME,ROLE,SLICE,REVISION}. LWS names are <ds>-<slice>-<revision>-<role> and rotate on every rollout.
   • Upstream direction: KEP-766 removed DS-created Services (Service Orchestration). KEP-846 Non-Goal ci: bump actions/checkout from 6.0.2 to 7.0.1 #3 excludes cross-slice Service/traffic routing (Non-Goals). Per-slice routing is ours to own, outside DS.
   • Chart (helm-charts @ bc27272): each slice is 2 headless Services (-prefill-hl-s<i>, -decode-hl-s<i>, selector on DS labels + worker-index: "0") + router Deployment -router-s<i> + PDBs (-<role>-s<i>, and -router-s<i> when router replicas ≥ 2). There is no per-slice router ClusterIP Service: the front-door Service selects component: router across all slices (service.yaml#L20). replicaCount == spec.slices. PD + scaler is currently rejected at render time (pd-guards.yaml#L37-L38), because the LWS names rotate.
   • llm-operator (@ e0bbf23): LLMService upgrades with helm v4, ServerSideApply=true, and ForceConflicts only when annotated for the current generation (helm.go#L204-L209). It re-applies only when the values hash changes and watches only LLMService. LLMScaler reads spec.replicas and status.{replicas,updatedReplicas,readyReplicas,terminatingReplicas,observedGeneration} off an unstructured target and writes spec.replicas with a plain Update (guards.go#L238, llmscaler_controller.go#L314). Its rollout guard only tracks Deployment.

   Answers to the four key questions

   1. Scaling channel: the scaler patches a CRD replica field: SliceSet.spec.replicas, through the existing LLMScaler (or HPA/KEDA via /scale). The render layer stays split. The chart owns the shape. The SliceSet controller owns the fan-out: ordinals, per-slice objects, ordering. The scaler never renders anything and never touches the LLMService. An external scaler targeting the LLMService would invert ownership and loop: scaler → LLMService → helm upgrade → re-rendered scaler. It would also make every scale step a full release upgrade.
   2. Values blob storage: not needed for scaling. The rest of the values already live in-cluster in LLMService.spec.layers. The decision pipeline only needs serviceId → slice count, which the Custom provider already reads. We would not add a per-release values ConfigMap.
   3. Committing APIs: yes, but narrowly. We commit to "a slice = one DS spec + one router pod template + per-role headless Services + optional PDBs, with ordinal naming". The router's script, args, probes and image stay chart-owned inside the pod template, so that layer keeps evolving under the chart. mc-master is not promoted.
   4. Failure reporting: on SliceSet.status. It has replicas/readyReplicas/updatedReplicas/terminatingReplicas/observedGeneration (the exact fields LLMScaler already reads, so readyReplicas means ready slices), per-ordinal slices[] entries, and Available/Progressing/ReplicaFailure conditions naming the stuck ordinal. LLMScaler mirrors it in currentReplicas. Helm release state never reflects runtime health.

   API sketch

   apiVersion: serving.modelsphere.dev/v1alpha1
   kind: SliceSet
   metadata:
     name: qwen3-pd                      # = chart fullname
   spec:
     replicas: 3                         # omitted by the chart when a scaler owns it (see below)
     selector:
       matchLabels: { app: qwen3-pd-sglang }
     updateStrategy:
       type: RollingUpdate               # | OnDelete
       rollingUpdate:
         maxUnavailable: 1               # slices; int or %
         partition: 0                    # ordinals >= partition get the new revision (canary)
     scaleDown:
       drainSeconds: 0                   # extra hold after the router is out of rotation
     minReadySeconds: 0
     revisionHistoryLimit: 5
     template:                           # ONE slice; controller stamps out s0..s{N-1}
       metadata:
         labels: { app: qwen3-pd-sglang }
       engines:                          # typed: lws DisaggregatedSetSpec minus slices
         placementPolicy: { type: None }
         roles:
         - name: prefill
           spec: { replicas: 2, leaderWorkerTemplate: { ... } }   # from _pod.tpl, unchanged
         - name: decode
           spec: { replicas: 1, leaderWorkerTemplate: { ... } }
       upstreamServices:                 # one headless Service per role, per slice
       - role: prefill
         ports: [{ name: http-prefill, port: 30001, targetPort: http-prefill }]
       - role: decode
         ports: [{ name: http-decode, port: 30002, targetPort: http-decode }]
       router:                           # typed: replicas + strategy + PodTemplateSpec
         replicas: 1
         progressDeadlineSeconds: 7200
         template: { ... }               # wait-upstreams, drift-liveness, grace 600 -- chart-owned
         affinityToRole: prefill         # controller injects required podAffinity to this slice's prefill leader
       disruptionBudgets:                # optional, rendered per slice
         engines: { maxUnavailable: 1 }
         router: { minAvailable: 1 }     # only applied when router.replicas >= 2
   status:
     observedGeneration: 7
     replicas: 3
     readyReplicas: 2
     updatedReplicas: 3
     terminatingReplicas: 0
     currentRevision: qwen3-pd-6c9f…
     updateRevision: qwen3-pd-6c9f…
     selector: app=qwen3-pd-sglang,serving.modelsphere.dev/component=router   # for /scale
     slices:
     - { ordinal: 0, revision: 6c9f…, ready: true }
     - { ordinal: 1, revision: 6c9f…, ready: true }
     - { ordinal: 2, revision: 6c9f…, ready: false, reason: EnginesPending, message: "prefill 1/2 placed" }
     conditions: [ { type: Available, status: "False", reason: SliceNotReady, message: "s2 not ready" } ]
   # +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector

   Per ordinal i, the controller owns (controller ownerRefs, label serving.modelsphere.dev/slice: "<i>"):
   DisaggregatedSet <name>-s<i> (slices: 1), Services <name>-<role>-hl-s<i> (selector disaggregatedset.x-k8s.io/name=<name>-s<i>, role=<role>, leaderworkerset.sigs.k8s.io/worker-index="0", publishNotReadyAddresses: true), Deployment <name>-router-s<i>, PDBs <name>-<role>-s<i> / <name>-router-s<i>. Names stay identical to today except the DS. Into router containers the controller injects SLICE_ORDINAL, SLICE_<ROLE>_HL (FQDN) and SLICE_<ROLE>_WANT, so the chart's router script reads env instead of {{ $s }}.

   Template fields are typed (lws DisaggregatedRoleSpec, PodTemplateSpec, a small Service-port/PDB subset), not generic object templates. That gives schema validation and clean SSA diffs, keeps ordinal substitution to a few known fields, and allows a narrow RBAC (DS, Deployments, Services, PDBs) instead of */*.

   Wiring and ownership

   flowchart TD
     LS[LLMService] -->|helm v4 install/upgrade, SSA, shape changes only| R[(helm release)]
     R --> SS[SliceSet  spec.template, no spec.replicas]
     R --> SC[LLMScaler  targetRef: SliceSet]
     R --> FD[front-door Service  component=router]
     R --> MC[mc-master Deployment+Service]
     R --> MR[ModelRoute / ServiceMonitor / LLMSLORequirement]
     DEC[decision server] -.->|slice count| SC
     SC -->|writes spec.replicas| SS
     SS -->|owns, per ordinal i| DS[DisaggregatedSet name-s_i  slices=1]
     SS --> HL[headless Services name-role-hl-s_i]
     SS --> RT[router Deployment name-router-s_i]
     SS --> PDB[PDBs per slice]
     DS -->|lws controller| LWS[LWS per role/revision]
     FD -.selects.-> RT
   

   Loading

   Who owns spec.replicas: when scaler.enabled, the chart omits spec.replicas (the CRD defaults it to 1 on create, and LLMScaler lifts it to minReplicas). This is the HPA practice. It matters concretely here: LLMService applies with SSA and no force, so a rendered replicas would make every later upgrade either fail on a field-manager conflict with the scaler, or (with the force annotation) snap the slice count back. Turning the scaler on for an existing release follows the documented SSA hand-off (k8s: migrating to HPA): the scaler writes once, then the chart drops the field. When scaler.enabled=false, the chart renders replicas: {{ .Values.replicaCount }} exactly as today.

   Small llm-operator changes beyond the new controller: (a) LLMScaler's rollout guard also tracks SliceSet (generation == observedGeneration && updatedReplicas == replicas). (b) Optionally, LLMScaler writes through /scale or SSA with its own field manager instead of a full-object Update. (c) Drop the pd.enabled && scaler.enabled render guard.

   Semantics

   • Scale up (N → M): for each missing ordinal, lowest first, create the headless Services and the DS (slices: 1) at the update revision, then the router Deployment. Its wait-upstreams gate and required podAffinity already handle ordering. The router joins the front door only when its readiness passes. Parallel creation across new ordinals is the default (podManagementPolicy: Parallel-like), since slices are independent.

   • Scale down (M → N): highest ordinal first, one at a time (or maxStepReplicas from LLMScaler):

     1. Take the slice out of rotation: delete the router Deployment. Terminating endpoints leave the front door immediately, and the existing 600 s grace drains in-flight streams.
     2. Wait until the router pods are gone, plus scaleDown.drainSeconds.
     3. Delete the DS (engines go via their own grace).
     4. Delete the Services and PDBs.

     A PDB never blocks this, because it is a controller delete and not an eviction. Status reports terminatingReplicas until the ordinal is gone, so LLMScaler's settle guard works as-is.

   • Template update: a SliceSet revision hash (ControllerRevision) covers template. With RollingUpdate, ordinals >= partition are updated highest-first, at most maxUnavailable slices not-ready at a time. "Updating a slice" means patching that ordinal's DS spec, so the DS planner does the cross-role, same-version rollout inside the slice. The router is updated in place. A slice counts as updated + ready when its DS is Available at the new revision and its router Deployment is available. After an engines-only change, the controller bumps a router pod-template annotation so the router re-resolves upstreams once, instead of relying only on drift-liveness. Changing replicas never triggers a rollout (same as DS slices). New slices created mid-rollout use the update revision. OnDelete is available for manual, slice-by-slice upgrades.

   Why one DS per ordinal (slices: 1) rather than one DS with slices: N, or raw LWS

   	One DS, slices: N	One DS per ordinal	LWS directly
   Cross-slice rollout budget (maxUnavailable, partition, canary)	impossible: DS rolls all slices concurrently	SliceSet decides which ordinals get the new spec	yes, but we own everything
   Ordered slice teardown	DS deletes slice LWS immediately, so we would have to sequence around it via slices--	natural: delete router, then that DS	yes
   Cross-role same-version rollout inside a slice	DS planner	DS planner (reused)	re-implement ~5k lines of planner/executor
   Per-role autoscaling inside a slice (KEP-849 External)	blocked: webhook rejects slices > 1	allowed today	n/a
   Placement (KEP-848)	ExclusiveSlice spreads slices of one DS	ExclusiveTopology works across DSs; ExclusiveSlice spread must be injected by SliceSet	ours
   Objects	1 DS	N DS (cheap control objects)	R×N LWS + revisions

   KEP-846 rejected "multiple DS objects" because there was "no single object to manage or scale". SliceSet is that single object, so the objection does not apply. Managing LWS directly gives us nothing that per-ordinal DS does not, and costs the hardest part of DS.

   Compared with A–D

   • A (router/mc-master as DS roles + sliceServices): goes against KEP-766/846, needs upstream negotiation, bends a singleton into an LWS role, and still has DS's all-slices-at-once rollout.
   • B (SliceServices sidecar CRD): fixes Services but not the router's lifecycle, rollout budget or teardown order, and splits one slice's state across two controllers.
   • C (PDRelease): right instinct (an operator owns the fan-out), but it re-implements the chart's whole render and makes the chart optional. SliceSet takes only the part that varies with N and leaves the rest of the release in the chart.
   • D (helm template | kubectl apply): has the ownership reversal described in Q1. A second writer forks the LLMService's helm-v4 SSA release (conflicting field managers, history that no longer matches the cluster), every scale step is a full re-render, and the scaler needs chart + values + render credentials.

   SliceSet is the only option that makes slice count a single in-cluster write while keeping the chart as the single shape owner.

   Open questions

   1. Name and group: SliceSet in serving.modelsphere.dev, or in a separate workloads. group? "Slice" also collides with TPU/JobSet vocabulary, so PDSliceSet is the conservative alternative.
   2. Scale-down victim: strict highest ordinal (StatefulSet), or prefer a not-ready ordinal first (ReplicaSet-like) and accept holes in the ordinal space?
   3. Slice spread: should SliceSet inject per-ordinal anti-affinity that mimics DS ExclusiveSlice across its child DSs, or require ExclusiveTopology?
   4. Name budget: LWS becomes <release>-s<i>-0-<rev8>-<role>. Check the 63-char limit for the longest release names, or give the child DS a short hash name.
   5. mc-master: stays release-scoped. Revisit only if per-slice stores are ever needed, via template.sidecarServices.
   6. Per-role scaling within a slice (P:D ratio): later, via DS External roles on each child DS, with SliceSet fanning out one aggregate target. This is out of scope for v1alpha1.
   7. Migration of existing releases: rename the DS from <release> to <release>-s0 (a one-time engine restart per release), or adopt the existing DS as ordinal 0.
4. aceforeverd commented on Oct 23, 2026

   

   aceforeverd

   on Oct 23, 2026

   Collaborator

   More actions

   temp solution in swiss side only: modelsphere/swiss#30

   It build scale loop inside swiss, and goes directly into swiss's apply API
5. remotego commented on Nov 2, 2026

   

   remotego

   on Nov 2, 2026

   CollaboratorAuthor

   More actions

   closed. see modelsphere/swiss#30
6. remotego

   closed this as duplicateon Nov 2, 2026