diff --git a/deploy/helm/openplan/README.md b/deploy/helm/openplan/README.md index c711934a..7e154a76 100644 --- a/deploy/helm/openplan/README.md +++ b/deploy/helm/openplan/README.md @@ -24,7 +24,10 @@ real TLS, real keys and an OIDC provider reachable over HTTPS. executor's security settings, so the chart doesn't work there. Use standard node pools. - **An ingress controller** with an IngressClass, and a TLS certificate for - the host you serve openplan on. + the host you serve openplan on. The chart creates a standard Ingress with + no controller-specific annotations, so any controller that serves one + works. Don't start on ingress-nginx: Kubernetes retired it in March 2026, + and it gets no more releases or security fixes. ### The services you bring @@ -117,11 +120,8 @@ api: credentialEncryptionKeySecret: {name: openplan-credentials, key: credential-encryption-key} trustedCASecret: openplan-trusted-ca # only for an internal CA executor: - resources: - limits: - memory: 41Gi # see "Sizing the executor" ingress: - className: nginx + className: # kubectl get ingressclass; omit for the cluster default host: openplan.example.com tlsSecretName: openplan-tls ``` @@ -137,8 +137,8 @@ kubectl -n openplan rollout status deployment/openplan-executor ``` `helm install` succeeds before any pod is up. Wait for the executor too: if -it can't be scheduled or refuses to start, the UI still works, but every -plan fails after a minute and every apply after ten, for want of an executor. +it can't be scheduled or refuses to start, the UI still works, but every plan +and apply waits ten minutes for an executor and then fails. **6. Sign in** at `https://` as the user whose `sub` you set in `api.rootSubject`. @@ -171,10 +171,11 @@ the key in it. | `api.terraformTimeout` | openplan's default | The longest one Terraform command may run, as a duration such as `90m`. | | `api.githubApp.id`, `api.githubApp.privateKeySecret` | none | A GitHub App that clones private template repositories. Set both or neither. | | `api.trustedCASecret` | none | A Secret whose `ca.crt` the api trusts, for Dex behind an internal CA. | -| `executor.replicas` | `1` | Executors. Each runs up to `executor.maxSessions` runs at once. | -| `executor.resources` | none | Requests and limits for the executor container. | -| `executor.maxSessions` | `20` | Runs at once on one executor, each in its own sandbox. | -| `executor.sandbox.memoryMiB` | `2048` | Memory per sandbox, in MiB. | +| `api.resources` | 2 CPU, 2Gi, requests equal to limits | Requests and limits for the api container. Override requests and limits together. | +| `executor.replicas` | `1` | Executors. Each runs `executor.maxSessions` runs at once, one by default, so this is how many runs go at once. | +| `executor.resources` | 2 CPU, 2Gi, requests equal to limits | Requests and limits for the executor container. Override requests and limits together; see "Sizing the executor". | +| `executor.maxSessions` | `1` | Runs at once on one executor, each in its own sandbox. | +| `executor.sandbox.memoryMiB` | `1536` | Memory per sandbox, in MiB. See "Sizing the executor" before you raise it. | | `executor.sandbox.cpus`, `executor.sandbox.pids` | `2`, `1024` | CPUs and processes per sandbox. | | `executor.sandbox.subnet` | `10.237.0.0/24` | The IPv4 subnet the sandboxes' addresses come from, `/29` or larger. Move it if it collides with your pod or service CIDR. | | `executor.runRoot.sizeLimit` | `8Gi` | Disk for the workspaces of the runs in flight on one executor. | @@ -187,19 +188,39 @@ the key. ## Sizing the executor -Every sandbox's memory counts against the executor pod. Set the executor's -memory limit to at least `executor.sandbox.memoryMiB` × `executor.maxSessions`, -plus about 1 GiB for the executor itself. The defaults, 2048 MiB × 20 -sessions, need 40 GiB of runs, so a limit around 41 GiB. To fit a smaller -node, lower `executor.maxSessions`, then the limit. +By default each executor pod gets 2 CPU and 2 GiB, reserved on its node +(requests equal limits, so Kubernetes runs it as `Guaranteed`), and runs one +plan or apply at a time in a 1536 MiB sandbox. The other 512 MiB is for the +executor itself. + +**More runs at once: add replicas.** Set `executor.replicas` to the number of +runs you want going at the same time. Each replica needs another 2 CPU and +2 GiB on some node. + +**When runs outnumber executors**, a new run shows "waiting for executor" and +waits for a free one, up to 10 minutes, whether it's a plan or an apply. If +none frees up in that time, the run fails and you start it again. + +**A heavy module: give each run more memory.** If a run's sandbox runs out of +memory, the run fails, and an apply can stop halfway and leave the state +locked. Raise `executor.sandbox.memoryMiB`, and raise the executor's memory +with it, so that memory is at least `executor.maxSessions` × +`executor.sandbox.memoryMiB` + 512Mi. Set requests and limits together: Helm +merges the map, so setting only `limits.memory` keeps the 2Gi request and the +pod is no longer `Guaranteed`. For 3072 MiB per run: -A limit set without a request is also the request, so the scheduler needs a -node with that much allocatable memory. With the defaults' 41 GiB limit on a -smaller node, the executor stays `Pending` ("Insufficient memory"). With no -`executor.resources` at all, the pod schedules anywhere, but its runs can -together claim more memory than the node has, and as a `BestEffort` pod the -executor is the first the kubelet evicts under memory pressure, failing every -run on it. +```yaml +executor: + sandbox: + memoryMiB: 3072 + resources: + requests: + cpu: "2" + memory: 3584Mi + limits: + cpu: "2" + memory: 3584Mi +``` The run root holds the checked-out template and providers of every run in flight. When it passes `executor.runRoot.sizeLimit`, the kubelet evicts the @@ -260,9 +281,9 @@ how it went. `deploy/helm/evaluation/` in the repository sets up everything above on Docker Desktop's Kubernetes, so you can try the chart without your own -services: single-pod Postgres, Temporal, Dex, Valkey and MinIO, ingress-nginx, -a local CA, fresh keys and the credentials Secret. It is for evaluation only: one -replica of everything, no backups, no persistence you should trust. +services: single-pod Postgres, Temporal, Dex, Valkey and MinIO, Traefik, +a local CA, fresh keys and the credentials Secret. It is for evaluation +only: one replica of everything, no backups, no persistence you should trust. From a checkout, with Docker Desktop's Kubernetes on and selected: @@ -278,8 +299,9 @@ release you checked out without its `v`, such as `1.0.0`. Then open `https://openplan.localhost` and sign in as `admin@openplan.local` / `admin-local-only`. `setup.sh` prints how to trust -the local CA on macOS, so the browser stops warning. Docker Desktop's VM has -about 8 GiB, so the harness runs two sessions at once. To remove everything: +the local CA on macOS, so the browser stops warning. The harness runs the +chart's defaults, which fit in Docker Desktop's VM of about 8 GiB: one run at +a time, and a second run waits for the first. To remove everything: ```sh ./deploy/helm/evaluation/teardown.sh diff --git a/deploy/helm/openplan/templates/api-deployment.yaml b/deploy/helm/openplan/templates/api-deployment.yaml index 88a88338..6c821b39 100644 --- a/deploy/helm/openplan/templates/api-deployment.yaml +++ b/deploy/helm/openplan/templates/api-deployment.yaml @@ -75,6 +75,10 @@ spec: httpGet: path: /healthz port: http + {{- with .Values.api.resources }} + resources: + {{- toYaml . | nindent 12 }} + {{- end }} {{- if .Values.api.trustedCASecret }} volumeMounts: - name: trusted-ca diff --git a/deploy/helm/openplan/templates/executor-deployment.yaml b/deploy/helm/openplan/templates/executor-deployment.yaml index 891c520c..5fb4c1a8 100644 --- a/deploy/helm/openplan/templates/executor-deployment.yaml +++ b/deploy/helm/openplan/templates/executor-deployment.yaml @@ -25,6 +25,15 @@ spec: command: ["sysctl", "-w", "net.ipv4.ip_forward=1"] securityContext: privileged: true + # A pod is Guaranteed only if every container, init included, has + # requests equal to limits; sysctl needs next to nothing. + resources: + requests: + cpu: 50m + memory: 32Mi + limits: + cpu: 50m + memory: 32Mi containers: - name: executor image: {{ include "openplan.image" $component }} diff --git a/deploy/helm/openplan/values.yaml b/deploy/helm/openplan/values.yaml index bf3d3ee9..566df0af 100644 --- a/deploy/helm/openplan/values.yaml +++ b/deploy/helm/openplan/values.yaml @@ -109,21 +109,44 @@ api: # Optional. The name of an existing Secret whose ca.crt the api trusts in # addition to the public roots, for an OIDC issuer behind an internal CA. trustedCASecret: "" + # requests equal limits, so the pod is Guaranteed and never uses more than + # the node reserved for it. Override requests and limits together: Helm + # merges maps, so setting only limits.memory keeps the 2Gi request and the + # pod drops to Burstable. + resources: + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "2" + memory: 2Gi # The executor gets Temporal, the executor's Valkey user and S3 from above, and # nothing else: no database, no keys, no OIDC secret. executor: replicas: 1 - # Set a memory limit of at least sandbox.memoryMiB times maxSessions, plus - # the executor's own headroom: every sandbox's memory is counted in this pod. - resources: {} - # The runs, each in its own sandbox, one executor holds at once. Empty means - # the executor's default. - maxSessions: "" + # Every sandbox's memory is counted in this pod, so memory must be at least + # maxSessions times sandbox.memoryMiB plus 512Mi for the executor itself; + # the default fits the chart's one 1536 MiB session. + # requests equal limits, so the pod is Guaranteed and never uses more than + # the node reserved for it. Override requests and limits together: Helm + # merges maps, so setting only limits.memory keeps the 2Gi request and the + # pod drops to Burstable. + resources: + requests: + cpu: "2" + memory: 2Gi + limits: + cpu: "2" + memory: 2Gi + # The runs, each in its own sandbox, one executor holds at once. One fits + # the default 2Gi pod; the executor's own default (20 sessions of 2048 MiB) + # would need 40 GiB. More runs at once come from replicas. + maxSessions: 1 # Each sandbox's limits. Empty means the executor's default. sandbox: # MiB of memory. - memoryMiB: "" + memoryMiB: 1536 cpus: "" pids: "" # The IPv4 subnet, /29 or larger, the sandboxes' addresses come from. Move