Reconcile loops in 0.10.0 cause ~660 logs/s

[birksl]

Description

Hi,

Thanks for creating the operator :)

We want to start using the operator in production, but during testing we found that the operator generated roughly 50% of the log volume on our test cluster. I worked with Claude to create a reproduction test with envtest, as well as implementing a fix. Claude wrote most of the code, but I have reviewed and tested it in our dev environment.

I have a PR stack with a fix for this plus a few other fixes I found, at birksl#15 if it is of interest. The PR descriptions contain further root cause analysis.

I believe we experienced the issue because our postgres and clickhouse secrets contained unexpected characters and paths, which langfuse itself is happy to consume, but the operator probes fail on. That is likely what led to the status conflicts since the probes report unhealthy but langfuse itself was running fine.

Steps to Reproduce

This branch adds an envtest test for reproducing the reconcile churn: https://github.com/birksl/langfuse-operator/tree/repro/reconcile-churn

Run make test after checking out the branch. The run takes ~100s and will report 600+ reconciliations over a 15s span.

Expected Behavior

I expect the controller to set a status explaining why it couldn't reconcile my instance.

Actual Behavior

A flood of identical logs going through all the reconciliation steps over and over again.

Environment

- Project version: v0.10.0
- OS: talos 1.12.6
- Kubernetes: v1.34.6
- Deployment method: Helm via fluxcd HelmRelease

Additional Context

My initial fix was to stop the reconciliation from churning due to status conflict. A second wave of fixes was to make the probes mimic langfuses behavior, or more clearly report why a probe failed.

[schneidermr]

Thanks for this — the report and the root-cause analysis in the PR
descriptions are unusually thorough, and made it quick to verify.

I've traced the churn through the code and your diagnosis holds up exactly.
status.phase has three writers, and two of them can never agree:
updateStatus() falls back to Pending, determineOverallHealth() falls
back to Degraded, neither value is in the other's set, and both end in an
unconditional Status().Update() while both controllers use a bare
For(&LangfuseInstance{}) with no predicate. So each write bumps
resourceVersion and wakes the sibling, forever.

That also explains why you saw it in production while Langfuse itself was
healthy: with the Deployments ready but the probes failing on your secrets,
the two writers land on Running and Degraded rather than the
Pending/Degraded pair your envtest run measures (no kubelet there, so
the Deployments never go ready). Same defect, different pair.

Two asks before we can take any of this:

1. Please open the PRs against this repo — #15–#21 all currently target
   birksl/langfuse-operator, so there's nothing here to review or merge.

2. Please base them on main, not the v0.10.0 tag. A tag is a fixed
   point; anything built on it silently misses everything that has landed
   since. Concretely: main now carries the missing pods rule in the chart
   ClusterRole — the same one-line fix as your PR feat: improve operator capabilities #1, but with a generator
   that regenerates the chart's rules from config/rbac/role.yaml and a unit
   test that fails if the two ever drift again. So drop your feat: improve operator capabilities #1 and start
   the stack at #15; the rest should rebase onto main without conflicts,
   since none of them touch the files that change.

Five stacked PRs is a lot to land at once, but we're happy to take them in
order rather than as one squashed change.

A few questions while you're at it:

1. Did you have the pods RBAC fix applied when you measured the 660/s in
   production? The health monitor short-circuits on phase: Migrating and
   only on that, so an instance wedged by the missing permission shouldn't
   flood at all. I'm trying to work out whether fixing the RBAC unmasks
   this churn for everyone still on 0.10.0 — which would change how loudly we
   need to flag it in the release notes.

2. What did the probe-hostile secrets actually look like? Redacted shape
   is fine — which characters, which paths, which probe failed, and what it
   reported. I'd like a regression test built on the real shape rather than
   one I invent.

3. Of the PRs already merged into your fork's main (#9–#14), which do you
   consider part of this fix, and which are separate? Some look independently
   useful and I don't want to lose them.

4. #21 looks like a re-roll of your merged #6 — is that intentional, or an
   artifact of rebasing onto v0.10.0?