Skip to content

feat(api): correlate failures by container name across replicas - #77

Merged
slauger merged 1 commit into
developfrom
feat/container-correlation
Sep 6, 2026
Merged

slauger merged 1 commit into
developfrom
feat/container-correlation

Conversation

@slauger

@slauger slauger commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Summary

allReplicasFailing worked at pod level: every replica had to be failing in some way. That conflates two different situations. The same container broken in every replica points at a systematic cause, a bad image or a missing Secret. Different containers failing in different pods may just be unrelated problems that happen to coincide.

Adds failureCorrelation with two values: Pod, the existing behaviour and the default, and Container, which additionally requires the same container name to be the failing one in every replica. Since the operator never scales anything back up, having a stricter option available is worth the field.

A new field rather than an enum on allReplicasFailing, which would have changed the type of a released field and needed a v1alpha2.

Two things fell out of the implementation:

  • The three per-kind branches in allReplicasFailing ran identical sibling loops. They now share one evaluator, so they cannot drift apart. That was already a latent risk before this change.
  • isMoreRestrictive needed the new dimension. Container correlation acts in fewer situations, so a policy using it ranks below one that does not. Without this the winner selection would no longer be a total order and overlapping policies could resolve inconsistently.

Worth being plain about the value: this is a caution knob, not a fix. If all replicas are broken, pod correlation is a perfectly defensible signal. The container mode exists for people who would rather under-act than over-act, which given the absence of a scale-up path is a reasonable preference.

Closes #70

Test plan

  • Table test across both modes with a two-container pod: same container failing everywhere, and different containers failing in different replicas. Pod mode acts in both cases; container mode acts only in the first.
  • Test that the restrictiveness ordering places pod correlation above container correlation and is not symmetric.
  • make ci passes; coverage rose from 64.1 to 65.8 percent.

allReplicasFailing worked at pod level, requiring every replica to be
failing in some way. That conflates a systematic cause, the same
container broken in every replica because of a bad image or a missing
Secret, with unrelated failures that happen to coincide.

Adds failureCorrelation with Pod, the existing behaviour and the
default, and Container, which additionally requires the same container
name to be the failing one everywhere. Since the operator never scales
anything back up, a stricter option is worth having.

A new field rather than an enum on allReplicasFailing, which would have
changed the type of a released field.

The three per-kind branches ran identical sibling loops, so they now
share one evaluator and cannot drift apart. isMoreRestrictive gains the
new dimension: container correlation acts in fewer situations, so a
policy using it ranks below one that does not, keeping the winner
selection a total order.

Closes #70

Signed-off-by: Simon Lauger <simon@lauger.de>
@slauger
slauger merged commit 283bc3a into develop Sep 6, 2026
14 checks passed
@slauger
slauger deleted the feat/container-correlation branch September 6, 2026 10:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(api): correlate failures by container name across replicas

1 participant