Spread service endpoints trust producer health reports that never expire #1099

Activity

1. privateip

   added

   enhancementNew feature or requestNew feature or request

   needs triageIssues that need to be triaged before implementingIssues that need to be triaged before implementing

   galactic-routerRelated to galactic-routerRelated to galactic-router

   on Oct 25, 2026
2. privateip

   self-assigned this

   on Oct 27, 2026
3. privateip commented on Oct 27, 2026

   

   privateip

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   Confirmed on main (509c9c9).

   Entries in status.producers carry no freshness. serviceroute.Compile and serviceroute.ProgrammingPlan both read every entry for the current attachment, whenever it was written (planner.go#L389-L412). The writer, ServiceProducerHealthReconciler, rewrites an entry only when its decision changes. Nothing marks an entry as no longer current, so both symptoms reproduce:

   • Lost node: with a PreferNodeLocal Spread endpoint, a remote consumer stays on the first producer whose entry says Healthy, even after that entry's node stopped writing.
   • Stuck drain: in a node's local set, an Unhealthy entry written before its router stopped keeps the producer draining with no time limit.

   Two planner tests show both. Each sets an entry's last renewal to older than any plausible lifetime. Both fail on the unchanged planner:

   producers with an expired Unhealthy decision = producer-a(draining),producer-z, want all eligible
   remote producer with a-remote's report expired = a-remote, want z-remote
   

   Plan

   • API: add an optional renewTime to each status.producers entry. Each node renews its entries every 15 seconds, even when no decision changed. An entry is current for 60 seconds after its renewTime. It is expired after that, or when renewTime is unset.
   • Writer: ServiceProducerHealthReconciler sets renewTime on every apply. It skips the write while its entries are unchanged and younger than the renew interval, and requeues for the next renewal.
   • Planner (Compile and ProgrammingPlan):
     • An expired entry never drains a local producer.
     • A remote consumer prefers, in order: a producer with a current Healthy or Unknown entry (or no entry), then one with an expired entry, then the first producer.
     • A lost node's producers therefore stop receiving other nodes' consumers within 60 seconds of their node's last renewal, whenever a currently reported producer exists. If none exists, availability wins, as it does today when every producer is Unhealthy.
   • Policy controller:
     • Requeues for the next entry expiry, so a node that has no intents yet picks up a producer that moves onto it.
     • Ignores endpoint updates that only renew entries that are still current, so the periodic renewal does not trigger a reconcile on every node.
   • Operators: renewTime shows which reports are current. The private services guide documents the bound, the ordering and the upgrade behavior: routers without renewal write no renewTime, so upgraded nodes treat those entries as expired until the rollout finishes.
   • Tests: planner tests for both symptoms and for the expiry helper, an envtest check of renewal and its write suppression, and a unit test of the watch filter.
4. privateip commented on Oct 27, 2026

   

   privateip

   on Oct 27, 2026

   CollaboratorAuthor

   More actions

   The fix is up for review in #1139.
5. privateip

   added a commit that references this issue on Oct 27, 2026

   fix: Reconcile when a producer report loses its renewal

   ...

   7b2c8f8
6. privateip

   closed this as completedin #1139on Oct 27, 2026