Skip to content

operator: the Secret informer caches every Secret in the cluster #577

Description

@slauger

Summary

The Certificate controller watches Secrets (internal/controller/certificate_controller.go:258-259):

Owns(&corev1.Secret{}).
Watches(&corev1.Secret{}, enqueueCertificatesForSecret(mgr.GetClient()))

A watch needs an informer, and cmd/main.go configures no ByObject selector for Secrets. In the default cluster-wide mode the manager therefore runs a ListWatch over every Secret in every namespace and holds them all in memory. The operator uses no uncached APIReader anywhere, so all Secret reads go through that cache too.

The chart ships memory: 256Mi as the limit (charts/openvox-operator/values.yaml:71).

Why it matters

The operator only ever cares about a handful of Secrets: the ones it creates, plus a few the user points it at. It pays for all of them.

In a busy cluster this is not a rounding error. Helm stores one Secret per release revision, each holding a gzipped manifest; add pull secrets, TLS secrets and service account tokens across a few hundred namespaces and the working set reaches hundreds of megabytes. The operator is then OOMKilled with nothing in its own logs pointing at the cause.

Not visible on a small test cluster: the e2e cluster holds 11 Secrets totalling well under a megabyte.

Why it is not a one-liner

Scoping the cache with a label selector bounds the informer, but the operator also reads Secrets it does not own and which carry no operator label:

  • spec.image.pullSecrets on Config, Server and Database
  • spec.postgres.credentialsSecretRef on Database
  • the token, bearer and basic auth secrets on ReportProcessor
  • spec.external.caSecretRef and tlsSecretRef on CertificateAuthority

A selector alone would break all of those. The change needs both halves:

  1. Cache.ByObject[&corev1.Secret{}] with a label selector matching app.kubernetes.io/managed-by: openvox-operator, so the informer only tracks the operator's own Secrets.
  2. An uncached reader (mgr.GetAPIReader()) for the user-supplied Secrets above. Those are read rarely, so the extra API call is cheap.

Worth checking while there: whether Owns(&corev1.Secret{}) plus the explicit Watches are both needed, since the label selector has to cover whatever the watch relies on.

Verification

A memory comparison before and after on a cluster with a few hundred Secrets would show whether the change does what it claims. controller_runtime_webhook and the Go runtime metrics the operator already exposes are enough to measure it.

Related

Found during the second review pass on 2026-09-03, raised independently by the operator-expert and SRE perspectives. Related to #510 in that both are about the operator's own runtime posture rather than the resources it manages.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions