Phase: Phase 15 — Performance and scale
Proposal: docs/proposals/next-phases.md §15
Priority: P3 · Size: S
Problem
controller-runtime exports workqueue depth, add rate, latency, and work duration for every controller, plus reconcile totals and errors. The operator registers the standard metrics registry, so these are already being scraped — they are simply not on any shipped dashboard or alert.
Workqueue depth and latency are the leading indicator of a reconcile backlog: they rise before phases go stale, before propagation lags, and before anyone notices a problem through the operator's own metrics.
Goal
The signal that predicts trouble is visible next to the signals that report it.
Design
- A "Controller health" row in the shipped Grafana dashboard: per-controller queue depth, add rate, work duration p50/p99, and reconcile error rate, alongside the existing conversion panels.
- An alert on sustained queue depth and on a rising reconcile error rate, shipped off by default with the rest of the pack.
- Document the relationship in
docs/observability.md: queue depth rising → phases go stale → ConversionPropagated lags. That causal chain is what makes the panels useful rather than decorative.
- Check whether
--max-concurrent-reconciles is exposed; if not, add it, because the dashboard will inevitably prompt the question and there should be a lever.
Acceptance criteria
Notes
Smallest item in the phase; good first issue for someone who wants to learn the observability stack.
Generated by Claude Code
Phase: Phase 15 — Performance and scale
Proposal:
docs/proposals/next-phases.md§15Priority: P3 · Size: S
Problem
controller-runtime exports workqueue depth, add rate, latency, and work duration for every controller, plus reconcile totals and errors. The operator registers the standard metrics registry, so these are already being scraped — they are simply not on any shipped dashboard or alert.
Workqueue depth and latency are the leading indicator of a reconcile backlog: they rise before phases go stale, before propagation lags, and before anyone notices a problem through the operator's own metrics.
Goal
The signal that predicts trouble is visible next to the signals that report it.
Design
docs/observability.md: queue depth rising → phases go stale →ConversionPropagatedlags. That causal chain is what makes the panels useful rather than decorative.--max-concurrent-reconcilesis exposed; if not, add it, because the dashboard will inevitably prompt the question and there should be a lever.Acceptance criteria
PrometheusRuleand covered bymake test-prometheusdocs/observability.mdexplains the leading-indicator relationship--max-concurrent-reconcilesexposed and chart-configurable if it is not alreadyNotes
Smallest item in the phase; good first issue for someone who wants to learn the observability stack.
Generated by Claude Code