Skip to content

feat: Health-check load balancer backends - #715

Merged
scotwells merged 2 commits into
codex/nlb-source-identityfrom
codex/nlb-health
Oct 25, 2026
Merged

scotwells merged 2 commits into
codex/nlb-source-identityfrom
codex/nlb-health

Conversation

@scotwells

@scotwells scotwells commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Each location checks its load balancer backends with TCP, HTTP or HTTPS probes, and only backends with a recent successful check receive traffic. A backend that fails, hasn't been checked yet, or has a stale result is left out. Until a location can reach backends inside their VPC, it reports that it can't probe and marks no backend healthy. Customer-set probe timing isn't passed through yet.

Test plan

  • Health check tests pass, including 64 backends over two full rounds
  • Stale results expire and remove the backend
  • A busy location doesn't wrongly mark healthy backends as failed
  • Lint passes

Depends on #714. Replaces #629. Related to #611 and datum-cloud/enhancements#884

@scotwells
scotwells requested a review from a team as a code owner October 23, 2026 15:55
@scotwells
scotwells requested review from ecv and kevwilliams October 23, 2026 15:55
@scotwells scotwells self-assigned this Oct 23, 2026
@cla-assistant

cla-assistant Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@kevwilliams kevwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hold: the probe reconciler retriggers itself on its own status writes, so the documented 5s pause between sweeps never happens and status is written in a tight loop.

SetupWithManager registers For(NetworkLoadBalancerBackendHealth) with no predicate. Every sweep rewrites status (LastProbeTime always changes), the update bumps resourceVersion, and the watch event re-enqueues the object immediately, ahead of the RequeueAfter of 5s. With the dialer-less registration in cell.go, every probe fails at once, so each health object becomes a back-to-back Get plus Status().Update loop limited only by API latency. With a real dialer it still runs sweeps back to back, doubling probe and write volume against the design. Nothing creates these objects yet, so this is latent, but it should be closed before a producer lands.

Suggested fix: add builder.WithPredicates(predicate.GenerationChangedPredicate{}) to For() so only spec changes and the RequeueAfter drive sweeps, plus a test asserting a status-only update does not enqueue.

Everything else checks out: the bounds match NSO#611 (64 targets, 8 per object, 32 per process, 2s timeout after slot acquisition, 5s pause, 39s lifetime), results fail closed on generation, route, fingerprint or expiry mismatch, probe targets are pinned to the live interface address and reject loopback, link-local, multicast and mapped addresses, and the threshold, freshness and slot tests fail under mutation.

Each location checks its load balancer backends with TCP, HTTP or HTTPS
probes, and only backends with a recent successful check are eligible for
traffic, so a failed backend stops receiving new connections on its own.

Key changes:
- Add a cell-local backend health record and probe runner
- Fail closed when a probe fails, is missing or goes stale
- Require a matching attachment UID before probing a backend
- Report that probing is unavailable until a location can reach VPC backends
A sweep's own status write re-enqueued the health object immediately, so
sweeps ran back to back and skipped the pause between them. Only spec
changes and the scheduled requeue now start a sweep.

Key changes:
- Filter backend health events to generation changes
- Test that a status-only update does not start another sweep

Copy link
Copy Markdown
Contributor Author

Fixed. Health checks now react only to spec changes, so status writes no longer start another sweep and the 5 second pause holds. A new test covers it.


Generated by Claude Code

@kevwilliams kevwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the predicate drops status-only updates; the test covers the filter only, so unwiring it from For() still passes.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.
Ran at 86aad00: internal/nlbhealth tests and the NLB controller tests pass with envtest 1.31.0.
CI at this head is green.
The new commit filters events to generation changes in internal/controller/networkloadbalancerbackendhealth_controller.go, so a status write no longer retriggers a sweep, and the new test covers status-only, spec and create events.
The PR body now says a location reports it cannot probe, which settles my wording point.
Still open: with no dialer (controller line 105) every pass stamps LastProbeTime (internal/nlbhealth/health.go:138), so each health object rewrites its status every 5s through the requeue.
Nothing creates these objects yet, so skip probing when the dialer is nil in a follow-up, tracked with the backend health producer in #720.
Nit: about 10 comment lines across health.go and the controller against the repo CLAUDE.md zero-comment rule.
Stack: based on #714, merge after it.

@scotwells
scotwells merged commit c0bce97 into main Oct 25, 2026
10 checks passed
@scotwells
scotwells deleted the codex/nlb-health branch October 25, 2026 22:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants