Skip to content

fix(controller): measure durationThreshold from readiness, not the last exit - #71

Merged
slauger merged 1 commit into
developfrom
fix/duration-threshold
Sep 6, 2026
Merged

slauger merged 1 commit into
developfrom
fix/duration-threshold

Conversation

@slauger

@slauger slauger commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Summary

durationThreshold did not work for CrashLoopBackOff, the operator's headline failure reason.

podExceedsDurationThreshold measured from the container's last termination time. Kubelet restarts a crashing container with exponential backoff capped at a few minutes and rewrites FinishedAt on every restart, so for a pod in a steady crash loop that timestamp is always recent. A threshold above the backoff cap could never be reached, and the default is 30m.

Crash loops were therefore gated on restartThreshold alone. The field only ever had an effect for containers that never started, which reach the RestartCount == 0 branch and fall back to the pod creation time. The README promised "how long a pod must be failing before action", which is not what happened.

The fix measures from the readiness condition, which does not bounce with the backoff: it flips to False when the container first stops serving and stays there while it keeps failing. A container that recovers and later fails again restarts the clock, which is the intended meaning rather than a flaw. ContainersReady is preferred over Ready, since Ready can also be held false by readiness gates that say nothing about the containers.

A pod with no readiness condition yet returns no verdict rather than falling back to the creation timestamp. That fallback would fire immediately on a pod that is hours old and only just broke, which is exactly the false positive the previous code's RestartCount == 0 guard existed to prevent. Deferring to the next evaluation costs one interval and cannot cause a wrong scale-down.

Note this makes the operator act in cases where it previously did not. The two thresholds are ORed, so workloads that were waiting for restartThreshold may now be caught by the duration path first. That is the documented behaviour finally taking effect, but it is a real behaviour change worth knowing about before the next release.

Closes #64

Test plan

The existing test asserted a state kubelet cannot produce: 15 restarts with a termination an hour old. After 15 restarts the container is in the capped backoff, so the last exit is minutes old at most. It passed for that reason and never exercised the bug.

  • Replaced it with the state a live kubelet does produce: recent FinishedAt, hour-old readiness transition.
  • Verified the regression test actually catches the bug by temporarily reintroducing the previous implementation alongside the new one: the old logic returns false on that pod, the new one returns true.
  • Added a case for a container that recovered two minutes ago, which must not count as broken for hours, and one for a pod with no readiness condition.
  • Corrected the unrealistic ImagePullBackOff fixture, which carried no conditions at all although kubelet publishes readiness as soon as it starts syncing a pod.
  • make ci passes; coverage 61.0 percent.

…st exit

podExceedsDurationThreshold measured from the container's last
termination time. Kubelet restarts a crashing container with backoff
capped at a few minutes and rewrites FinishedAt on every restart, so for
a steady crash loop that timestamp is always recent and a threshold
above the cap could never be reached.

durationThreshold was therefore inert for CrashLoopBackOff, the
operator's headline failure reason: crash loops were gated on
restartThreshold alone. The field only ever worked for containers that
never started, which reach the RestartCount == 0 branch and fall back to
the pod creation time. The README promised something the code did not do.

Measures from the readiness condition instead, which flips to False when
the container first stops serving and stays there while it keeps
failing. A container that recovers and fails again restarts the clock,
which is the intended meaning. ContainersReady is preferred over Ready
because Ready can also be held false by readiness gates, which say
nothing about the containers.

A pod with no readiness condition yet returns no verdict rather than
falling back to the creation timestamp, which would fire immediately on
an old pod that only just broke.

The test that covered this asserted a state kubelet cannot produce: 15
restarts with a termination an hour old. Replaced with the state a live
kubelet does produce, plus cases for a recovered pod and a missing
condition. Verified against the previous implementation that the new
crash loop test fails on it.

Closes #64

Signed-off-by: Simon Lauger <simon@lauger.de>
@slauger
slauger merged commit 475a8d8 into develop Sep 6, 2026
14 checks passed
@slauger
slauger deleted the fix/duration-threshold branch September 6, 2026 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(controller): durationThreshold never fires for CrashLoopBackOff

1 participant