Resize pods in place on resource-only template changes#50
Open
tamalsaha wants to merge 2 commits into
Open
Conversation
When a pod differs from the update revision only in container or pod-level resources, resize the running pod via the pods/resize subresource instead of deleting it, then relabel it to the update revision so revision accounting converges without a restart. Gated by the InPlaceVerticalScaling feature gate (default on); falls back to delete-and-recreate when the resize is unsupported or reported infeasible. Signed-off-by: Tamal Saha <tamal@appscode.com>
![kodiakhq[bot]](https://avatars.githubusercontent.com/in/29196?s=60&v=4){kind=small}
Two correctness fixes to the in-place resize path: - onlyResourcesDiffer now renders the pod's current revision and the update revision and compares the WHOLE pod spec (resources zeroed), instead of only the container slices against the live pod. This prevents a revision that changes a non-container field (volumes, affinity, tolerations, nodeSelector, securityContext, ...) from being misclassified as resources-only and resized + relabeled in place, which would silently drop that change. - The MaxUnavailable update path now counts an in-place resize against the maxUnavailable budget (deletedPods++), so a potentially disruptive resize (RestartContainer resizePolicy or a readiness blip) cannot be applied to every stale pod in a single pass. Threads currentRevision through updatePetSetAfterInvariantEstablished. Adds tests covering non-container template changes. Signed-off-by: Tamal Saha <tamal@appscode.com>
tamalsaha
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Teaches the PetSet controller to actuate a resource-only template change in place via the Kubernetes
pods/resizesubresource instead of deleting and recreating the pod. Generic infrastructure — every KubeDB database that uses PetSet inherits it.onlyResourcesDifferrenders the pod's current revision and the update revision (viaApplyRevision), zeroes resources on both, and deep-compares the whole PodSpec. Comparing two rendered revisions (not the live pod vs a template) cancels apiserver defaulting noise; comparing the full PodSpec (not just container slices) guarantees any non-resource change (volumes, affinity, tolerations, nodeSelector, securityContext, labels, ...) is ineligible and is never silently dropped.ResizeStatefulPodissuesUpdateResize, waits for kubelet actuation (thePodResizePending/PodResizeInProgressconditions andContainerStatuses[].Resources), then patches the pod'scontroller-revision-hashlabel to the update revision so revision accounting converges without a restart.updatePetSetandupdatePetSetAfterInvariantEstablished). In the MaxUnavailable path the in-place resize consumes the maxUnavailable budget, so a potentially disruptive resize (aRestartContainerresizePolicy or a readiness blip) cannot hit every stale pod at once.isResizeUnsupported:pods/resizeabsent, or the cluster'sInPlacePodVerticalScalinggate off) or reportedInfeasible; transient errors requeue (never delete).InPlaceVerticalScalingfeature gate (Beta, default on). With the gate off, behavior is byte-for-byte the current delete-and-recreate.Tests
onlyResourcesDiffertruth table including non-container changes (nodeSelector, volume) being ineligible even alongside a resource change;resizeStatetransitions;isResizeUnsupportedmatrix; gate on/off. Full controller suite green.Part of a coordinated set
Foundation PR (independent of apimachinery). Used at runtime by kubedb/postgres#911. Safe to ship on any cluster — unsupported clusters fall back.