diff --git a/docs/guides/postgres/scaling/vertical-scaling/in-place/index.md b/docs/guides/postgres/scaling/vertical-scaling/in-place/index.md new file mode 100644 index 0000000000..5022d3d8b3 --- /dev/null +++ b/docs/guides/postgres/scaling/vertical-scaling/in-place/index.md @@ -0,0 +1,379 @@ +--- +title: In-Place Vertical Scaling Postgres +menu: + docs_{{ .version }}: + identifier: guides-postgres-scaling-vertical-inplace + name: In-Place Vertical Scaling + parent: guides-postgres-scaling-vertical + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# In-Place Vertical Scaling Postgres + +This guide will show you how to use `KubeDB-Ops-Manager` to update the CPU and +memory of a running `Postgres` instance **in place**, that is, without recreating +the Pods and without a primary failover. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line + tool must be configured to communicate with your cluster. If you do not already + have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- In-place resize requires a Kubernetes cluster with the **in-place pod resize** + feature available (the `pods/resize` subresource; container-level in-place resize + is GA in Kubernetes v1.35). On clusters that do not support it, KubeDB + automatically falls back to the regular restart-based vertical scaling, so the + request still completes (see [Eligibility and fallback](#eligibility-and-fallback)). + +- Install `KubeDB-Provisioner` and `KubeDB-Ops-Manager` in your cluster following + the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Postgres](/docs/guides/postgres/concepts/postgres.md) + - [PostgresOpsRequest](/docs/guides/postgres/concepts/opsrequest.md) + - [Vertical Scaling Overview](/docs/guides/postgres/scaling/vertical-scaling/overview/index.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` +throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/guides/postgres/scaling/vertical-scaling/in-place/yamls](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +## In-Place vs Restart vertical scaling + +The default vertical scaling path patches the `PetSet` and then **evicts and +recreates** every Pod (replicas first, then a primary failover, then the primary) +so each Pod starts with the new resources. It is safe but disruptive: every resize +is a full rolling restart plus a failover. + +In-place vertical scaling instead asks the kubelet to change the running +container's cgroup limits through the `pods/resize` subresource, so: + +- **CPU** changes (in either direction) take effect live, with no Pod restart and + no failover. +- A **memory increase** grows the cgroup live. Because PostgreSQL sets + `shared_buffers` only at startup, that GUC is left unchanged; the reloadable, + memory-derived GUCs (`effective_cache_size`, `work_mem`, `maintenance_work_mem`) + are re-applied live so the database can take advantage of the extra memory. + +You opt in per request with `spec.verticalScaling.mode: InPlace`. The mode +defaults to `Restart`, so existing OpsRequests behave exactly as before. + +## Deploy Postgres + +Below is the YAML of a 3-replica `Postgres` cluster we are going to create. Using a +cluster (replicas > 1) lets us confirm that in-place scaling keeps the same primary +(no failover). We also enable [auto-tuning](/docs/guides/postgres/configuration/pgtune.md) +(`spec.configuration.tuning`) so that the memory-derived parameters are managed by +KubeDB — this is what lets the in-place memory increase re-apply the reloadable GUCs +live (see [In-Place memory increase](#in-place-memory-increase)). + +```yaml +apiVersion: kubedb.com/v1 +kind: Postgres +metadata: + name: pg + namespace: demo +spec: + version: "13.13" + replicas: 3 + standbyMode: Hot + configuration: + tuning: + profile: oltp + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut +``` + +Let's create the `Postgres` cr we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/postgres.yaml +postgres.kubedb.com/pg created +``` + +Wait for the `Postgres` to become `Ready`, + +```bash +$ kubectl get postgres -n demo pg +NAME VERSION STATUS AGE +pg 13.13 Ready 4m16s +``` + +Let's check the `pg-0` Pod's postgres container resources (the postgres container +is the first container, so its index is `0`), and note the Pod's UID, its +`restartCount`, and the role of each Pod, so we can compare after the resize. + +```bash +$ kubectl get pod -n demo pg-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "memory": "1Gi" + }, + "requests": { + "cpu": "500m", + "memory": "1Gi" + } +} + +$ kubectl get pod -n demo pg-0 -o jsonpath='{.metadata.uid}{"\n"}' +6f0a4c9e-2d2b-4d8e-9d4a-2b9c1f5a7e10 + +$ kubectl get pods -n demo -l app.kubernetes.io/instance=pg -L kubedb.com/role +NAME READY STATUS RESTARTS AGE ROLE +pg-0 2/2 Running 0 4m51s primary +pg-1 2/2 Running 0 3m50s standby +pg-2 2/2 Running 0 3m46s standby +``` + +KubeDB provisions the Pods with a `resizePolicy` of `NotRequired` for CPU and +memory, which is what lets the kubelet resize them without a restart: + +```bash +$ kubectl get pod -n demo pg-0 -o json | jq '.spec.containers[0].resizePolicy' +[ + { + "resourceName": "cpu", + "restartPolicy": "NotRequired" + }, + { + "resourceName": "memory", + "restartPolicy": "NotRequired" + } +] +``` + +## In-Place CPU scaling (no restart, no failover) + +In order to update the resources in place, create a `PostgresOpsRequest` with +`spec.verticalScaling.mode: InPlace`. Below is the YAML we are going to apply; it +raises the CPU request and limit to `1` and keeps memory at `1Gi`. + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: PostgresOpsRequest +metadata: + name: pg-scale-vertical-inplace + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: pg + verticalScaling: + mode: InPlace + postgres: + resources: + requests: + memory: "1Gi" + cpu: "1" + limits: + memory: "1Gi" + cpu: "1" +``` + +Here, + +- `spec.databaseRef.name` specifies the `pg` `Postgres` database. +- `spec.type` specifies that we are performing `VerticalScaling`. +- `spec.verticalScaling.mode: InPlace` requests the in-place path. (Omitting `mode`, + or setting it to `Restart`, uses the default restart-based path.) +- `spec.verticalScaling.postgres` is the desired postgres container resources. + +Let's create it, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace.yaml +postgresopsrequest.ops.kubedb.com/pg-scale-vertical-inplace created +``` + +**Wait for the OpsRequest to succeed:** + +```bash +$ watch kubectl get postgresopsrequest -n demo pg-scale-vertical-inplace +NAME TYPE STATUS AGE +pg-scale-vertical-inplace VerticalScaling Successful 1m12s +``` + +Unlike the restart path, the `KubeDB-Ops-Manager` does not evict any Pod here. It +patches the `PetSet` template, the PetSet controller drives the kubelet resize on +each Pod, and the operator waits for the resize to be actuated. The `describe` +output reflects this — there is no `PauseDatabase`/eviction-driven restart of the +members for the resize: + +```bash +$ kubectl get postgresopsrequest -n demo pg-scale-vertical-inplace -o yaml | yq '.status.conditions[].type' +Progressing +UpdatePetSets +VerticalScale +Successful +``` + +**Verify the Pods were resized in place (not recreated):** + +The clearest proof is that the Pod UID and `restartCount` are unchanged, while the +container resources are updated: + +```bash +$ kubectl get pod -n demo pg-0 -o jsonpath='{.metadata.uid}{"\n"}' +6f0a4c9e-2d2b-4d8e-9d4a-2b9c1f5a7e10 # same UID as before — the Pod was not recreated + +$ kubectl get pods -n demo -l app.kubernetes.io/instance=pg -L kubedb.com/role +NAME READY STATUS RESTARTS AGE ROLE +pg-0 2/2 Running 0 9m12s primary # still primary — no failover +pg-1 2/2 Running 0 8m11s standby +pg-2 2/2 Running 0 8m07s standby + +$ kubectl get pod -n demo pg-0 -o json | jq '.spec.containers[0].resources' +{ + "limits": { + "cpu": "1", + "memory": "1Gi" + }, + "requests": { + "cpu": "1", + "memory": "1Gi" + } +} +``` + +You can also confirm the kubelet actually applied the change by checking the +allocated/actual resources reported in the Pod status: + +```bash +$ kubectl get pod -n demo pg-0 -o json | jq '.status.containerStatuses[0].resources' +{ + "limits": { + "cpu": "1", + "memory": "1Gi" + }, + "requests": { + "cpu": "1", + "memory": "1Gi" + } +} +``` + +The `RESTARTS` column stays at `0`, the UID is unchanged, and `pg-0` is still the +primary: the CPU was scaled with no restart and no failover. + +## In-Place memory increase + +A memory **increase** is also done in place. Create a `PostgresOpsRequest` with +`spec.verticalScaling.mode: InPlace` and the default +`spec.verticalScaling.memoryPolicy: ResizeOnly`: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: PostgresOpsRequest +metadata: + name: pg-scale-vertical-inplace-mem + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: pg + verticalScaling: + mode: InPlace + memoryPolicy: ResizeOnly + postgres: + resources: + requests: + memory: "2Gi" + cpu: "1" + limits: + memory: "2Gi" + cpu: "1" +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace-memory.yaml +postgresopsrequest.ops.kubedb.com/pg-scale-vertical-inplace-mem created +``` + +With `memoryPolicy: ResizeOnly` (the default for `InPlace`): + +- the memory cgroup is grown live (no restart); +- `shared_buffers` is **left unchanged** — it is a restart-only PostgreSQL + parameter, and KubeDB intentionally does not change it here so a later restart + will not surprise-jump it; +- the reloadable, memory-derived GUCs (`effective_cache_size`, `work_mem`, + `maintenance_work_mem`) are re-applied live with `ALTER SYSTEM SET ...; SELECT + pg_reload_conf();`, computed from the new memory. This applies only when KubeDB + manages the tuning, i.e. `spec.configuration.tuning` is set on the `Postgres` + (we enabled it above). + +You can see the difference — `shared_buffers` keeps the value it started with at +`1Gi`, while `effective_cache_size` now tracks the `2Gi`: + +```bash +$ kubectl exec -it -n demo pg-0 -c postgres -- psql -U postgres -c \ + "SHOW shared_buffers; SHOW effective_cache_size;" + shared_buffers +---------------- + 256MB # unchanged — restart-only + + effective_cache_size +---------------------- + 1536MB # re-applied live (75% of the new 2Gi) +``` + +If you want `shared_buffers` itself to track the new memory, use +`spec.verticalScaling.memoryPolicy: Retune` instead — that requires a restart and +KubeDB runs it on the restart path (see below). + +## Eligibility and fallback + +In-place is **per request**: KubeDB runs the request in place only if **every** +requested change is eligible, otherwise the whole request falls back to the +restart-based path (and the OpsRequest still completes). A change is in-place +eligible when it is: + +- a **CPU** change in any direction, or +- a **memory increase** with `memoryPolicy: ResizeOnly`. + +The request falls back to the restart path when any of the following is true: + +| Situation | Why | +| --- | --- | +| `spec.verticalScaling.memoryPolicy: Retune` | `shared_buffers` is restart-only, so retuning it needs a restart. | +| A **memory decrease** | A live shrink can be rejected by the kubelet or risk an OOM kill, so KubeDB does it via restart. | +| The cluster does not support in-place resize, or the kubelet reports the resize `Infeasible` (e.g. the node cannot fit the larger request) | KubeDB recreates the Pod so the scheduler can place it on a node with room. | + +When a request that asked for `InPlace` falls back, KubeDB records an +`InPlaceResizeEligible=false` condition on the OpsRequest with the reason, and then +runs the regular vertical scaling. This covers the arbiter, the read replicas, and +the primary group — each resizes in place when eligible. + +## Autoscaler + +The KubeDB Postgres compute autoscaler can emit these requests for you. When the +autoscaler is configured to use in-place mode, the `PostgresOpsRequest`s it creates +carry `mode: InPlace` (with `memoryPolicy: ResizeOnly`), so routine CPU autoscaling +becomes non-disruptive. See the +[compute autoscaling guide](/docs/guides/postgres/autoscaler/compute/cluster.md). + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete postgresopsrequest -n demo pg-scale-vertical-inplace pg-scale-vertical-inplace-mem +kubectl delete postgres -n demo pg +kubectl delete ns demo +``` diff --git a/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace-memory.yaml b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace-memory.yaml new file mode 100644 index 0000000000..def5876336 --- /dev/null +++ b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace-memory.yaml @@ -0,0 +1,20 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: PostgresOpsRequest +metadata: + name: pg-scale-vertical-inplace-mem + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: pg + verticalScaling: + mode: InPlace + memoryPolicy: ResizeOnly + postgres: + resources: + requests: + memory: "2Gi" + cpu: "1" + limits: + memory: "2Gi" + cpu: "1" diff --git a/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace.yaml b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace.yaml new file mode 100644 index 0000000000..243cebfc04 --- /dev/null +++ b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/pg-vertical-scaling-inplace.yaml @@ -0,0 +1,19 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: PostgresOpsRequest +metadata: + name: pg-scale-vertical-inplace + namespace: demo +spec: + type: VerticalScaling + databaseRef: + name: pg + verticalScaling: + mode: InPlace + postgres: + resources: + requests: + memory: "1Gi" + cpu: "1" + limits: + memory: "1Gi" + cpu: "1" diff --git a/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/postgres.yaml b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/postgres.yaml new file mode 100644 index 0000000000..56d5396bf3 --- /dev/null +++ b/docs/guides/postgres/scaling/vertical-scaling/in-place/yamls/postgres.yaml @@ -0,0 +1,21 @@ +apiVersion: kubedb.com/v1 +kind: Postgres +metadata: + name: pg + namespace: demo +spec: + version: "13.13" + replicas: 3 + standbyMode: Hot + configuration: + tuning: + profile: oltp + storageType: Durable + storage: + storageClassName: "standard" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + deletionPolicy: WipeOut diff --git a/docs/guides/postgres/scaling/vertical-scaling/overview/index.md b/docs/guides/postgres/scaling/vertical-scaling/overview/index.md index c15fe56164..87f595a852 100644 --- a/docs/guides/postgres/scaling/vertical-scaling/overview/index.md +++ b/docs/guides/postgres/scaling/vertical-scaling/overview/index.md @@ -51,4 +51,10 @@ The vertical scaling process consists of the following steps: 9. After successful updating of the `Postgres` resources, the `KubeDB` Ops Manager resumes the `Postgres` object so that the `KubeDB` Provisioner operator resumes its usual operations. +In step 7 above, the Ops Manager recreates the Pods (replicas first, then the primary, with a failover) so they start with the new resources. + +## In-Place Vertical Scaling + +By default the resize recreates the Pods. On clusters that support Kubernetes in-place pod resize, you can instead set `spec.verticalScaling.mode: InPlace` on the `PostgresOpsRequest` to change CPU/memory of the **running** Pods through the `pods/resize` subresource — without recreating the Pods and without a primary failover. CPU changes (either direction) and memory increases take effect live; `shared_buffers` (a restart-only parameter) is left unchanged while the reloadable, memory-derived GUCs are re-applied live. A memory decrease, `memoryPolicy: Retune`, or a cluster that does not support in-place resize automatically falls back to the recreate path, so the request still completes. The mode defaults to `Restart`, so existing requests are unchanged. See [In-Place Vertical Scaling](/docs/guides/postgres/scaling/vertical-scaling/in-place/index.md). + In the next doc, we are going to show a step by step guide on updating resources of Postgres database using vertical scaling operation.