Repository navigation
feat: Health-check load balancer backends - #715
Conversation
9d4436c to
2b11f2a
Compare
kevwilliams
left a comment
There was a problem hiding this comment.
Hold: the probe reconciler retriggers itself on its own status writes, so the documented 5s pause between sweeps never happens and status is written in a tight loop.
SetupWithManager registers For(NetworkLoadBalancerBackendHealth) with no predicate. Every sweep rewrites status (LastProbeTime always changes), the update bumps resourceVersion, and the watch event re-enqueues the object immediately, ahead of the RequeueAfter of 5s. With the dialer-less registration in cell.go, every probe fails at once, so each health object becomes a back-to-back Get plus Status().Update loop limited only by API latency. With a real dialer it still runs sweeps back to back, doubling probe and write volume against the design. Nothing creates these objects yet, so this is latent, but it should be closed before a producer lands.
Suggested fix: add builder.WithPredicates(predicate.GenerationChangedPredicate{}) to For() so only spec changes and the RequeueAfter drive sweeps, plus a test asserting a status-only update does not enqueue.
Everything else checks out: the bounds match NSO#611 (64 targets, 8 per object, 32 per process, 2s timeout after slot acquisition, 5s pause, 39s lifetime), results fail closed on generation, route, fingerprint or expiry mismatch, probe targets are pinned to the live interface address and reject loopback, link-local, multicast and mapped addresses, and the threshold, freshness and slot tests fail under mutation.
Each location checks its load balancer backends with TCP, HTTP or HTTPS probes, and only backends with a recent successful check are eligible for traffic, so a failed backend stops receiving new connections on its own. Key changes: - Add a cell-local backend health record and probe runner - Fail closed when a probe fails, is missing or goes stale - Require a matching attachment UID before probing a backend - Report that probing is unavailable until a location can reach VPC backends
A sweep's own status write re-enqueued the health object immediately, so sweeps ran back to back and skipped the pause between them. Only spec changes and the scheduled requeue now start a sweep. Key changes: - Filter backend health events to generation changes - Test that a status-only update does not start another sweep
2b11f2a to
86aad00
Compare
|
Fixed. Health checks now react only to spec changes, so status writes no longer start another sweep and the 5 second pause holds. A new test covers it. Generated by Claude Code |
kevwilliams
left a comment
There was a problem hiding this comment.
Verified the predicate drops status-only updates; the test covers the filter only, so unwiring it from For() still passes.
ecv
left a comment
There was a problem hiding this comment.
Approve.
Ran at 86aad00: internal/nlbhealth tests and the NLB controller tests pass with envtest 1.31.0.
CI at this head is green.
The new commit filters events to generation changes in internal/controller/networkloadbalancerbackendhealth_controller.go, so a status write no longer retriggers a sweep, and the new test covers status-only, spec and create events.
The PR body now says a location reports it cannot probe, which settles my wording point.
Still open: with no dialer (controller line 105) every pass stamps LastProbeTime (internal/nlbhealth/health.go:138), so each health object rewrites its status every 5s through the requeue.
Nothing creates these objects yet, so skip probing when the dialer is nil in a follow-up, tracked with the backend health producer in #720.
Nit: about 10 comment lines across health.go and the controller against the repo CLAUDE.md zero-comment rule.
Stack: based on #714, merge after it.
Summary
Each location checks its load balancer backends with TCP, HTTP or HTTPS probes, and only backends with a recent successful check receive traffic. A backend that fails, hasn't been checked yet, or has a stale result is left out. Until a location can reach backends inside their VPC, it reports that it can't probe and marks no backend healthy. Customer-set probe timing isn't passed through yet.
Test plan
Depends on #714. Replaces #629. Related to #611 and datum-cloud/enhancements#884