fix: Expire producer health reports a node stops renewing

api/network/v1alpha1/serviceroute_types.go

@@ -1,6 +1,8 @@
 package v1alpha1
 
 import (
+	"time"
+
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 )
@@ -128,6 +130,18 @@ const (
 // endpoint that a node can hold.
 const ServiceEndpointMaxProducersPerNode = 16
 
+const (
+	// ServiceEndpointProducerReportRenewInterval is how often a node renews
+	// its entries in a Spread endpoint's status.producers while its decisions
+	// do not change.
+	ServiceEndpointProducerReportRenewInterval = 15 * time.Second
+
+	// ServiceEndpointProducerReportLifetime is how long after its renewTime
+	// an entry in status.producers is current. Nodes route on an entry only
+	// while it is current.
+	ServiceEndpointProducerReportLifetime = 60 * time.Second
+)
+
 // ServiceEndpointHealthCheck configures the TCP health check of an endpoint's
 // producers. A probe opens a TCP connection to the producer attachment's first
 // IPv6 address, not the endpoint's address, on Port, through the producer's
@@ -188,6 +202,9 @@ type ServiceEndpointStatus struct {
 	// under its own field manager, and removes them when the producer stops
 	// being selected. An entry whose attachmentUID no longer matches the
 	// attachment of that name describes an earlier attachment and is ignored.
+	// Each node renews its entries every 15 seconds; an entry not renewed
+	// for 60 seconds is expired, because its node or its galactic-router has
+	// stopped, and no node routes on its state.
 	// +optional
 	// +listType=map
 	// +listMapKey=namespace
@@ -238,6 +255,12 @@ type ServiceEndpointProducerStatus struct {
 	// LastTransitionTime is when State last changed.
 	// +kubebuilder:validation:Required
 	LastTransitionTime metav1.Time `json:"lastTransitionTime"`
+
+	// RenewTime is when the producer's node last confirmed this entry. The
+	// entry is current for 60 seconds after it, and expired after that or
+	// when RenewTime is unset.
+	// +optional
+	RenewTime *metav1.Time `json:"renewTime,omitempty"`
 }
 
 // ServiceEndpointDeliveryMode defines the placement contract for a service.

config/crd/galactic.datumapis.com_serviceendpoints.yaml

@@ -276,6 +276,9 @@ spec:
                   under its own field manager, and removes them when the producer stops
                   being selected. An entry whose attachmentUID no longer matches the
                   attachment of that name describes an earlier attachment and is ignored.
+                  Each node renews its entries every 15 seconds; an entry not renewed
+                  for 60 seconds is expired, because its node or its galactic-router has
+                  stopped, and no node routes on its state.
                 items:
                   description: ServiceEndpointProducerStatus is one producer's health
                     decision.
@@ -317,6 +320,13 @@ spec:
                         or ProbeUnavailable.
                       maxLength: 64
                       type: string
+                    renewTime:
+                      description: |-
+                        RenewTime is when the producer's node last confirmed this entry. The
+                        entry is current for 60 seconds after it, and expired after that or
+                        when RenewTime is unset.
+                      format: date-time
+                      type: string
                     state:
                       description: |-
                         State is the decision. Healthy and Unknown producers receive new

docs/api/network-gateway.md

@@ -687,6 +687,7 @@ _Appears in:_
 | `reason` _string_ | Reason is ProbeSucceeded, ProbeFailed, Pending or ProbeUnavailable. |  | MaxLength: 64 <br />Required: \{\} <br /> |
 | `message` _string_ | Message is the last probe error, when there is one. |  | MaxLength: 1024 <br /> |
 | `lastTransitionTime` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v/#time-v1-meta)_ | LastTransitionTime is when State last changed. |  | Required: \{\} <br /> |
+| `renewTime` _[Time](https://kubernetes.io/docs/reference/generated/kubernetes-api/v/#time-v1-meta)_ | RenewTime is when the producer's node last confirmed this entry. The<br />entry is current for 60 seconds after it, and expired after that or<br />when RenewTime is unset. |  |  |
 
 
 #### ServiceEndpointReference
@@ -744,7 +745,7 @@ _Appears in:_
 
 | Field | Description | Default | Validation |
 | --- | --- | --- | --- |
-| `producers` _[ServiceEndpointProducerStatus](#serviceendpointproducerstatus) array_ | Producers holds the health decision of every producer of a Spread<br />endpoint. Each node writes the entries of the producers on that node,<br />under its own field manager, and removes them when the producer stops<br />being selected. An entry whose attachmentUID no longer matches the<br />attachment of that name describes an earlier attachment and is ignored. |  | MaxItems: 4096 <br /> |
+| `producers` _[ServiceEndpointProducerStatus](#serviceendpointproducerstatus) array_ | Producers holds the health decision of every producer of a Spread<br />endpoint. Each node writes the entries of the producers on that node,<br />under its own field manager, and removes them when the producer stops<br />being selected. An entry whose attachmentUID no longer matches the<br />attachment of that name describes an earlier attachment and is ignored.<br />Each node renews its entries every 15 seconds; an entry not renewed<br />for 60 seconds is expired, because its node or its galactic-router has<br />stopped, and no node routes on its state. |  | MaxItems: 4096 <br /> |
 
 
 #### ServiceRouteAuthorization

docs/private-services/README.md

@@ -84,19 +84,29 @@ TCP connection from `fe80::/10` on the Private path. The node publishes the
 decision for each of its producers in the endpoint's `status.producers`:
 
 ```console
-$ kubectl get serviceendpoint resolver -o jsonpath='{range .status.producers[*]}{.name}{"\t"}{.node}{"\t"}{.state}{"\t"}{.reason}{"\n"}{end}'
-resolver-a   node-1   Healthy     ProbeSucceeded
-resolver-b   node-1   Unhealthy   ProbeFailed
+$ kubectl get serviceendpoint resolver -o jsonpath='{range .status.producers[*]}{.name}{"\t"}{.node}{"\t"}{.state}{"\t"}{.reason}{"\t"}{.renewTime}{"\n"}{end}'
+resolver-a   node-1   Healthy     ProbeSucceeded   2026-01-01T12:00:45Z
+resolver-b   node-1   Unhealthy   ProbeFailed      2026-01-01T12:00:45Z
 ```
 
-What the decision does:
+The node renews its entries every 15 seconds, whether or not a decision
+changed, and sets each entry's `renewTime`. An entry is current for 60 seconds
+after its `renewTime`. After that it is expired: its node is down, or its
+`galactic-router` has stopped, and its `state` may no longer be true. An entry
+with no `renewTime` is expired too.
+
+What a current decision does:
 
 | State       | New flows | Existing TCP connections and replies |
 | ----------- | --------- | ------------------------------------ |
 | `Healthy`   | yes       | yes                                  |
 | `Unknown`   | yes       | yes                                  |
 | `Unhealthy` | no        | yes                                  |
 
+An expired entry never drains a producer on its own node, so a producer
+reported `Unhealthy` just before its node's router stopped takes new flows
+again once the entry expires.
+
 When every producer on a node is `Unhealthy`, all of them keep taking new
 flows, because an empty set would drop every request. A producer that crashes
 stops getting new flows within `intervalSeconds × unhealthyThreshold` plus
@@ -111,9 +121,14 @@ longest request, then stop it. Its in-flight requests and open connections
 finish while new ones go to the other producers.
 
 When the consumer's node has no producer, `PreferNodeLocal` sends the consumer
-to one remote producer, the first by namespace and name that its node has not
-reported `Unhealthy`, through the SRv6 service tunnel as with `Single`. A
-remote consumer therefore does not spread. While a health change reaches the
+to one remote producer, through the SRv6 service tunnel as with `Single`. It is
+the first producer by namespace and name whose node currently reports it
+`Healthy` or `Unknown`, or has no entry for it. If there is none, it is the
+first producer whose entry expired, and failing that the first producer. So
+when a node goes down while its producers' attachments still read ready,
+consumers on other nodes move off those producers within 60 seconds of the
+node's last renewal, as long as another node currently reports a producer that
+can take them. A remote consumer does not spread. While a health change reaches the
 consumer's node and the producer's node at different moments, the two can
 briefly disagree on that producer, and its requests are refused until they
 agree.
@@ -127,6 +142,19 @@ the connection resets. Changing an endpoint between `Single` and `Spread`
 replaces its routes, which can drop a request in flight and moves existing TCP
 connections.
 
+A `galactic-router` without report renewal writes no `renewTime`, so while
+routers are being upgraded, consumers on upgraded nodes treat the entries of
+nodes that are not yet upgraded as expired, and send remote consumers to
+them only when no current entry offers a producer. This lasts until the
+rollout finishes. A rolled-back router ignores `renewTime`.
+
+Apply the `ServiceEndpoint` CRD with `renewTime` before upgrading
+`galactic-router`. Without it the API server prunes the field from every
+report, so every entry reads as expired and no `Unhealthy` producer drains.
+Nodes compare `renewTime` with their own clock, so keep node clocks in sync:
+a skew near 45 seconds expires live reports early or keeps a lost node's
+reports current for that much longer.
+
 Spread needs a `galactic-cni` whose datapath has the producer set map. A
 `galactic-router` that finds the map missing refuses the endpoint's Spread
 routes on that node, retrying, and keeps every other route working. Roll out