feat: serve JobSLORequirement routes; follow the pdengine and scale-to-zero CRD bounds

CHANGELOG.md

@@ -8,12 +8,23 @@ no leading `v`.
 ## [Unreleased]
 
 ### Added
+- `JobSLORequirement` routes. The index watches both kinds, and a route
+  resolves across them; a job route's body has `queue` (`maxDepth`, `promql`)
+  in place of `ttft` / `otps`. A cluster without the JobSLO CRD serves the LLM
+  routes as before. The RBAC in `config/rbac/` grants `jobslorequirements`.
+- The `pdengine` bound type on `LLMSLORequirement` (helm-charts#95).
+- Bound values of 0 (scale-to-zero), which the CRD has allowed since
+  helm-charts#20; they were refused with `400` before.
 - CI on every pull request: pytest and the end-to-end smoke run on Python 3.10
   and 3.12, a docker build, and a gate for the license text and committed
   credentials.
 - Dependabot for the Python dependencies and the GitHub Actions.
 - `NOTICE`.
 
+### Changed
+- A write the CRD's rules reject (API server `422`) is `400 invalid_request`
+  with the API server's message, not `502`.
+
 ### Removed
 - The internal GitLab pipeline (`.gitlab-ci.yml`).
 

README.md

@@ -3,7 +3,8 @@
 A small HTTP API for reading and changing the SLO of an LLM inference service,
 without handing the caller access to Kubernetes.
 
-The SLO itself lives in an `LLMSLORequirement` custom resource, which
+The SLO itself lives in an `LLMSLORequirement` custom resource (or a
+`JobSLORequirement`, for a queue-driven job service), which
 [slo-scaler-decision-gen](https://github.com/modelsphere/slo-scaler-decision-gen)
 reads to decide replica counts. slo-api is a front door onto those resources:
 callers address a service by its route name, send plain JSON, and never see a
@@ -21,11 +22,11 @@ namespace, a CR name or a kubeconfig.
   the service. The RBAC in `config/rbac/` grants exactly
   `get / list / watch / patch / update` and nothing else.
 - **It only touches the SLO fields** — `priority`, `minimumDeployment`,
-  `maximumDeployment`, `ttft`, `otps`. Anything else in the spec, written by
-  other controllers, survives a write.
-- **A route resolves through `spec.serviceId`.** If two namespaces have a CR with
-  the same `serviceId`, writes are refused with `409` rather than guessing; reads
-  return the first one seen.
+  `maximumDeployment`, `ttft`, `otps` (`queue` on a job route). Anything else in
+  the spec, written by other controllers, survives a write.
+- **A route resolves through `spec.serviceId`**, across both kinds. If two CRs
+  have the same `serviceId` (two namespaces, or one of each kind), reads and
+  writes are refused with `409` rather than guessing.
 - Writes merge onto the **live** CR, read fresh from the API server rather than
   from the watch cache, so a concurrent change is not silently reverted.
 
@@ -56,8 +57,25 @@ boolean:
 }
 ```
 
-`highPriority: true` writes `priority: 10`; `false` writes `0`. Metric types are
-`avg`, `p50`, `p80`, `p90`, `p95`, `p99`. Errors come back as
+`highPriority: true` writes `priority: 10`; `false` writes `0`. A bound's
+`value` may be 0: `{min: 0, max: 0}` turns the service off (scale-to-zero). Metric types are
+`avg`, `p50`, `p80`, `p90`, `p95`, `p99`. A bound's `type` is `replica`,
+`concurrency` (minimum only) or `pdengine` (a PD service's bounds counted in
+prefill + decode engines; both bounds must then be `pdengine`).
+
+A route backed by a `JobSLORequirement` has `queue` in place of `ttft` and
+`otps`, and its bounds are `replica` only (`concurrency` for the minimum):
+
+```json
+{"route": "img-gen", "queue": {"maxDepth": 8, "promql": "sum(jobs_pending{service=\"img-gen\"})"}}
+```
+
+`queue.promql` may be left out of a `PUT` to keep the CR's own. `DELETE` keeps
+`queue` (the CRD requires it) and resets the rest.
+
+The CRD's own cross-field rules (minimum at most maximum, matching bound types,
+...) are checked by the API server against the merged spec; a write that breaks
+one is `400` with the API server's message. Errors come back as
 `{"error": {"code", "message", "field"}}`, and an `X-Request-Id` header on the
 request is echoed on the response.
 

config/rbac/rbac.yaml

@@ -16,7 +16,7 @@ rules:
   # Both groups: the CRDs moved from inference.x-k8s.io, and slo-api picks
   # whichever the cluster serves at startup (slo_api.resolve_group).
   - apiGroups: ["inference.modelsphere.dev", "inference.x-k8s.io"]
-    resources: ["llmslorequirements"]
+    resources: ["llmslorequirements", "jobslorequirements"]
     verbs: ["get", "list", "watch", "patch", "update"]
 ---
 apiVersion: rbac.authorization.k8s.io/v1

docs/openapi.yaml

@@ -4,25 +4,27 @@ info:
   version: 0.2.1
   summary: Read and change the SLO of an LLM inference service by its route.
   description: |
-    slo-api is a front door onto `LLMSLORequirement` custom resources, which
+    slo-api is a front door onto `LLMSLORequirement` and `JobSLORequirement`
+    custom resources, which
     slo-scaler-decision-gen reads to size each service. Callers address a
     service by its route and send plain JSON; they never see a namespace, a CR
     name or a kubeconfig.
 
     ```
-    client -- Bearer token --> slo-api --> LLMSLORequirement CRs --> decision-gen
+    client -- Bearer token --> slo-api --> {LLM,Job}SLORequirement CRs --> decision-gen
                                (patch only; never creates or deletes a CR)
     ```
 
     ## Routes
 
-    A route is a CR's `spec.serviceId`. The CR's lifecycle belongs to whatever
+    A route is a CR's `spec.serviceId`, of either kind. The CR's lifecycle belongs to whatever
     deployed the service (for swiss, the chart at install, with the release
     name as the serviceId). Here a route is only read and patched:
 
     - An unknown route is `404`, for writes too. Nothing here creates a CR.
-    - A serviceId found in more than one namespace is `409` for reads and
-      writes alike, with the candidates in `error.details`.
+    - A serviceId found on more than one CR (two namespaces, or one of each
+      kind) is `409` for reads and writes alike, with the candidates in
+      `error.details`.
 
     ## Readiness
 
@@ -33,9 +35,11 @@ info:
     ## Fields
 
     slo-api owns five fields of the CR spec: `priority`, `minimumDeployment`,
-    `maximumDeployment`, `ttft` and `otps`. Every other field, written by other
-    controllers, survives a write. `priority` (0-10 in the CR) is exposed as
-    the boolean `highPriority`.
+    `maximumDeployment`, `ttft` and `otps` -- on a `JobSLORequirement` route,
+    `queue` in place of `ttft` and `otps`, and bounds of type `replica` (or
+    `concurrency` for the minimum) only. A section of the other kind is `400`.
+    Every other field, written by other controllers, survives a write.
+    `priority` (0-10 in the CR) is exposed as the boolean `highPriority`.
 
     ## Errors
 
@@ -68,9 +72,9 @@ paths:
       summary: List routes
       description: |
         Routes in name order, read from the watch index rather than the API
-        server. Items leave out `ttft` and `otps`; read one route for those.
+        server. Items leave out `ttft`, `otps` and `queue`; read one route for those.
 
-        A route found in more than one namespace counts in `total` but is left
+        A route found on more than one CR counts in `total` but is left
         out of `items`, so a page can be shorter than `pageSize`.
       operationId: listConfigs
       parameters:
@@ -138,9 +142,12 @@ paths:
         the write, so a concurrent change to another field is not reverted.
         `spec.serviceId` is kept.
 
+        A `queue.promql` left out is kept, the same way.
+
         The CRD's own rules, such as `minimumDeployment.value` at most
-        `maximumDeployment.value`, are checked by the API server. A body that
-        breaks one comes back as `502`, not `400`.
+        `maximumDeployment.value`, or both bounds of the same `type`, are
+        checked by the API server on the merged spec. A body that breaks one
+        comes back as `400` with the API server's message.
       operationId: putConfig
       requestBody:
         required: true
@@ -176,7 +183,8 @@ paths:
       description: |
         Resets the five fields to the CRD defaults; the CR stays. `priority`
         becomes 0 and `minimumDeployment` `{type: replica, value: 1}`;
-        `maximumDeployment`, `ttft` and `otps` are removed.
+        `maximumDeployment`, `ttft` and `otps` are removed. A
+        `JobSLORequirement` keeps its `queue`, which its CRD requires.
 
         decision-gen treats a requirement with no `maximumDeployment` as opted
         out of autoscaling, so after a reset it stops sizing the service.
@@ -297,14 +305,14 @@ components:
           example:
             error: { code: not_found, message: "route 'ghost' not registered" }
     Ambiguous:
-      description: The route is the serviceId of CRs in more than one namespace.
+      description: The route is the serviceId of more than one CR.
       content:
         application/json:
           schema: { $ref: "#/components/schemas/Error" }
           example:
             error:
               code: conflict
-              message: "route 'my-model' maps to multiple LLMSLORequirement CRs; cannot auto-select a target"
+              message: "route 'my-model' maps to multiple SLO requirement CRs; cannot auto-select a target"
               details:
                 - { namespace: team-a, name: my-model }
                 - { namespace: team-b, name: my-model }
@@ -332,10 +340,11 @@ components:
           description: Absent means decision-gen does not autoscale the service.
         ttft: { $ref: "#/components/schemas/TTFT" }
         otps: { $ref: "#/components/schemas/OTPS" }
+        queue: { $ref: "#/components/schemas/Queue" }
 
     SLOSummary:
       type: object
-      description: A list item. `ttft` and `otps` are left out.
+      description: A list item. `ttft`, `otps` and `queue` are left out.
       required: [route, highPriority]
       properties:
         route: { type: string }
@@ -369,24 +378,40 @@ components:
           type: object
           required: [value]
           properties:
-            type: { type: string, enum: [replica, concurrency], default: replica }
-            value: { type: integer, minimum: 1 }
+            type: { type: string, enum: [replica, concurrency, pdengine], default: replica }
+            value: { type: integer, minimum: 0 }
         maximumDeployment:
           type: object
           required: [value]
           properties:
-            type: { type: string, enum: [replica], default: replica }
-            value: { type: integer, minimum: 1 }
+            type: { type: string, enum: [replica, pdengine], default: replica }
+            value: { type: integer, minimum: 0 }
         ttft: { $ref: "#/components/schemas/TTFT" }
         otps: { $ref: "#/components/schemas/OTPS" }
+        queue:
+          $ref: "#/components/schemas/Queue"
+          description: JobSLORequirement routes only. `promql` may be left out to keep the stored one.
 
     Bound:
       type: object
-      description: A replica bound as stored. The CRD allows 0 (scale to zero), which this API does not write.
+      description: |
+        A bound as stored. 0 is scale-to-zero: a minimum of 0 needs a maximum of
+        0, which turns the service off (a CRD rule, so breaking it is `400`).
+        `pdengine` counts a PD service's prefill plus decode engines
+        rather than replicas, and needs both bounds to be `pdengine`.
       properties:
-        type: { type: string, enum: [replica, concurrency] }
+        type: { type: string, enum: [replica, concurrency, pdengine] }
         value: { type: integer, minimum: 0 }
 
+    Queue:
+      type: object
+      description: Queue SLO of a `JobSLORequirement`.
+      additionalProperties: false
+      required: [maxDepth]
+      properties:
+        maxDepth: { type: integer, minimum: 0, description: Target pending jobs per replica. }
+        promql: { type: string, minLength: 1, description: PromQL that decision-gen evaluates to the queue depth. }
+
     TTFT:
       description: |
         Time to first token, a ceiling in seconds: a metric is violated when the

src/slo_api/__init__.py

@@ -1,4 +1,4 @@
-"""slo-api: HTTP facade over LLMSLORequirement CRs.
+"""slo-api: HTTP facade over LLMSLORequirement and JobSLORequirement CRs.
 
 Exposes a route-keyed REST API and translates to/from the underlying
 custom resource. Never creates new CRs; only get/patch/replace/delete
@@ -9,7 +9,10 @@
 _log = _logging.getLogger("slo-api")
 
 VERSION = "v1alpha1"
-PLURAL = "llmslorequirements"
+LLM_PLURAL = "llmslorequirements"
+JOB_PLURAL = "jobslorequirements"
+# Both kinds share one route space, as decision-gen sizes them side by side.
+PLURALS = (LLM_PLURAL, JOB_PLURAL)
 
 # The CRDs moved from inference.x-k8s.io to inference.modelsphere.dev. Both are
 # listed so this runs against a cluster on either, in the order to prefer them: