fix(controller): measure durationThreshold from readiness, not the last exit - #71
Merged
Merged
Conversation
…st exit podExceedsDurationThreshold measured from the container's last termination time. Kubelet restarts a crashing container with backoff capped at a few minutes and rewrites FinishedAt on every restart, so for a steady crash loop that timestamp is always recent and a threshold above the cap could never be reached. durationThreshold was therefore inert for CrashLoopBackOff, the operator's headline failure reason: crash loops were gated on restartThreshold alone. The field only ever worked for containers that never started, which reach the RestartCount == 0 branch and fall back to the pod creation time. The README promised something the code did not do. Measures from the readiness condition instead, which flips to False when the container first stops serving and stays there while it keeps failing. A container that recovers and fails again restarts the clock, which is the intended meaning. ContainersReady is preferred over Ready because Ready can also be held false by readiness gates, which say nothing about the containers. A pod with no readiness condition yet returns no verdict rather than falling back to the creation timestamp, which would fire immediately on an old pod that only just broke. The test that covered this asserted a state kubelet cannot produce: 15 restarts with a termination an hour old. Replaced with the state a live kubelet does produce, plus cases for a recovered pod and a missing condition. Verified against the previous implementation that the new crash loop test fails on it. Closes #64 Signed-off-by: Simon Lauger <simon@lauger.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
durationThresholddid not work forCrashLoopBackOff, the operator's headline failure reason.podExceedsDurationThresholdmeasured from the container's last termination time. Kubelet restarts a crashing container with exponential backoff capped at a few minutes and rewritesFinishedAton every restart, so for a pod in a steady crash loop that timestamp is always recent. A threshold above the backoff cap could never be reached, and the default is 30m.Crash loops were therefore gated on
restartThresholdalone. The field only ever had an effect for containers that never started, which reach theRestartCount == 0branch and fall back to the pod creation time. The README promised "how long a pod must be failing before action", which is not what happened.The fix measures from the readiness condition, which does not bounce with the backoff: it flips to
Falsewhen the container first stops serving and stays there while it keeps failing. A container that recovers and later fails again restarts the clock, which is the intended meaning rather than a flaw.ContainersReadyis preferred overReady, sinceReadycan also be held false by readiness gates that say nothing about the containers.A pod with no readiness condition yet returns no verdict rather than falling back to the creation timestamp. That fallback would fire immediately on a pod that is hours old and only just broke, which is exactly the false positive the previous code's
RestartCount == 0guard existed to prevent. Deferring to the next evaluation costs one interval and cannot cause a wrong scale-down.Note this makes the operator act in cases where it previously did not. The two thresholds are ORed, so workloads that were waiting for
restartThresholdmay now be caught by the duration path first. That is the documented behaviour finally taking effect, but it is a real behaviour change worth knowing about before the next release.Closes #64
Test plan
The existing test asserted a state kubelet cannot produce: 15 restarts with a termination an hour old. After 15 restarts the container is in the capped backoff, so the last exit is minutes old at most. It passed for that reason and never exercised the bug.
FinishedAt, hour-old readiness transition.ImagePullBackOfffixture, which carried no conditions at all although kubelet publishes readiness as soon as it starts syncing a pod.make cipasses; coverage 61.0 percent.