Add a delay to the shutdown-manager before failing healthchecks

[mwain]

Description

We run Envoy Gateway v1.7.0 on GKE behind the external passthrough L4 load balancer (with externalTrafficPolicy: Local).
Envoy pod terminations can produce a few seconds where new connections are reset or hang until they time out.

Seems that the cause is two timers racing. When a pod terminates, the shutdown-manager immediately calls healthcheck/fail, the readiness probe fails within ~5s, and kube-proxy stops routing to the pod.
But the load balancer deregisters nodes via its own health check, which on GKE runs every 3 seconds and needs 2 failures (not configurable), so it takes ~6-12s.
In between, the LB keeps sending connections that the node can no longer route.

Kubernetes handles this window via KEP-1669.
When no other ready endpoints remain, kube-proxy keeps routing to terminating pods that still pass their readiness probe.
Failing readiness at the start of shutdown defeats this.

We use distroless images, so patching the preStop hook with a sleep is not an option.

Proposal

Add a drainDelay field to ShutdownConfig that delays the healthcheck/fail call.
During the delay the pod stays ready and keeps serving, so the load balancer has time to deregister it.
Defaults to 0s, which keeps the current behavior.

Impl

Proposal has been implemented in #9211

Relevant Links

• Need a setting to delay start of shutdown sequence #4156 (same race on AWS NLB)
• GKE L4 health check parameters
• KEP-1669

[consideRatio]

Brilliant - this is exactly what I was arriving to create an issue for.

I think it would help avoid no disruption in a setup like mine with GKE and Cilium (aka. dataplane v2), using envoy gateway, having a Service configured with externalTrafficPolicy: Local (default).

I can resolve this quite well by using externalTrafficPolicy: Cluster, but its a tradeoff with regards to retaining the source IP that I'd rather not make.

[consideRatio]

Introducing drainDelay via #9211 could be a good fix, but I've started to think twice since we delay two mechanisms by delaying shutdown-manager's call to the /healthcheck/fail endpoint on the envoy container. Instead of always delaying both mechanisms, could it be better to delay just one?

The two mechanisms:

• Connection draining: enovy container nudges connections to terminate and re-establish
• Accepting new connections: envoy container readiness endpoint starts failing --> EndpointSlice ready=false and serving=false --> not a valid target for new connections

Key assumption for following discussion:

• EndpointSlice describing a specific endpoint with terminating=bool, ready=bool, and serving=bool, traffic will instruct kube-proxy and cilium-agent etc to setup new connections to non-terminating (but serving) endpoints, and only fall back to terminating ones if forced to. This relates to KEP-1669

Alternative

Let shutdown-manager initiate connection draining directly as it already does, but allow delaying the transition to serving=false via a failing readiness probe.

In practice, it seems like a shutdown-manager initiated POST to envoy's /drain_listeners?graceful&skip_exit followed by a delayed POST to /healthcheck/fail could accomplish this.

A conservative approach to enable this alternative would be to introduce a readinessFailureDelay set to 0 by default, preserving existing behavior exactly. If it was set to something like 40s, it could split apart the call to /healthcheck/fail as two separate calls, where /healthcheck/fail was delayed.

A more direct approach would be to let the readinessFailureDelay above not be configurable, but instead determined by minDrainDuration. It may be the better default, but turn out to be unsuitable in some situations.

[consideRatio]

AI revised version of my self-written comment above

AI revision of my own text

Introducing drainDelay via #9211 could be a good fix, but I
have started to wonder if it delays two different mechanisms that may be better
controlled separately.

In the current shutdown-manager flow, delaying the call to Envoy's
/healthcheck/fail also delays the start of Envoy's drain behavior. Instead of
always delaying both mechanisms, could it be better to start graceful drain
immediately and delay only the readiness/serving transition?

The two mechanisms are:

• Connection draining: Envoy nudges accepted HTTP connections away from the
  terminating pod. For HTTP/1.1 this can mean Connection: close; for HTTP/2
  this can mean GOAWAY.
• Removal from new Service traffic: Envoy's readiness endpoint starts
  failing, the pod becomes unready, and the EndpointSlice endpoint eventually
  becomes ready=false and serving=false. At that point the endpoint should
  not be a valid target for new Service-selected connections.

Key assumption for the following discussion:

• EndpointSlice describes an endpoint with terminating, ready, and
  serving conditions. Service proxies such as kube-proxy and GKE Dataplane
  V2 / Cilium should prefer non-terminating serving endpoints and only fall
  back to terminating serving endpoints when no better local endpoint is
  available. This relates to
  KEP-1669: Proxy Terminating Endpoints.

Alternative

Let shutdown-manager initiate connection draining immediately, while allowing a
delay before the terminating endpoint becomes serving=false through the
failing readiness probe.

In practice, it seems like this could be implemented as:

1. POST /drain_listeners?graceful&skip_exit
2. Wait for a configured delay.
3. POST /healthcheck/fail
4. Continue waiting for accepted connections until drainTimeout, with the
   existing minDrainDuration / exitAtConnections behavior.

This would let Envoy start graceful HTTP drain behavior immediately while
keeping Kubernetes readiness passing for a bounded window. During that window,
external load balancer straggler connections can still reach the terminating
pod if the node dataplane needs to fall back to a local terminating endpoint.

For GKE Service type LoadBalancer with externalTrafficPolicy: Local, this
matters because the external load balancer does not probe Envoy directly. It
probes the Service-specific healthCheckNodePort on the node, answered by the
node dataplane agent. That health state can lag behind pod termination and
EndpointSlice updates, so the load balancer may still send new connections to a
node for a bounded window after an Envoy pod starts terminating.

Naming

A conservative API could introduce a separate field, defaulting to zero:

shutdown:
  healthcheckFailDelay: 40s

With the default 0s, existing behavior would be preserved. If set, Envoy
Gateway would start compatible graceful listener drain behavior immediately, but
delay the /healthcheck/fail call by that duration.

Other possible names:

• readinessFailureDelay
• preReadinessDrainDelay
• healthcheckFailDelay

I slightly prefer readinessFailureDelay from an operator perspective, because
the Kubernetes-visible effect is delaying pod readiness failure and
EndpointSlice serving=false. I can also see healthcheckFailDelay being more
natural from an Envoy Gateway implementation perspective, because the delayed
operation is specifically /healthcheck/fail.

Relationship To minDrainDuration

A more direct approach would be to avoid another user-facing duration and derive
the healthcheck-fail delay from minDrainDuration.

That would make minDrainDuration mean something like:

1. Start graceful drain immediately.
2. Keep readiness passing for at least minDrainDuration.
3. Then call /healthcheck/fail.
4. Keep draining accepted connections until drainTimeout.

The tradeoff is that this changes the meaning of minDrainDuration more than a
separate field would. A separate healthcheckFailDelay /
readinessFailureDelay is more explicit and easier to reason about alongside
drainTimeout.

Why This Is Useful

For external L4 load balancers whose backend removal lags behind Kubernetes
endpoint updates, there are two different shutdown windows:

• New LB straggler connections: new connections that still arrive at a node
  before the external load balancer has converged.
• Already accepted connections: existing connections/requests that Envoy
  has already accepted and should be allowed to complete.

Delaying the whole drain sequence protects the first window by keeping Envoy
fully live for longer, but it also delays HTTP drain signals such as
Connection: close and GOAWAY.

Starting graceful drain immediately while delaying /healthcheck/fail could
protect both windows:

• new LB straggler connections can still be accepted and served because
  EndpointSlice can remain serving=true;
• those same connections are discouraged from sticking around;
• already accepted work still has drainTimeout to complete.

Caveats / Questions

• Does POST /drain_listeners?graceful&skip_exit have the desired behavior for
  Envoy Gateway's generated HTTP listeners: accept and serve new downstream
  connections while discouraging reuse?
• Envoy Gateway's /ready endpoint is also served by an Envoy listener. A
  split-phase implementation should verify that graceful listener drain does
  not itself make the readiness listener fail before /healthcheck/fail is
  called.
• Keeping readiness true during pod termination should probably remain opt-in.
  If healthy non-terminating pods are available and connections can persist,
  becoming non-ready promptly is often preferable so new Service traffic moves
  away from the terminating pod.

[arkodg]

curious which version of Cilium is running, is it v1.17 or v1.18 , imo cilium/cilium#40969 (comment) is the issue in Cilium which proactively removes endpoints from its bfp map, when an endpoint is not Ready

Envoy Gateway fails the endpoint asap so it can signal the peer to shift traffic to a healthy/new pod asap (using Connection: close / GOAWAY)

if this is the core issue youre hitting, it probably does make sense to implement some workaround until cloud providers have upgraded to v1.18 and higher

[consideRatio]

Envoy Gateway fails the endpoint asap so it can signal the peer to shift traffic to a healthy/new pod asap (using Connection: close / GOAWAY)

Im new to envoy gateway, but what does "signal to peer mean" here? Is it related to cluster internal routing controlled by / proxied through envoy gateway pods? Wont envoy gateway avoid directing new connections to the pod if its terminating?

[arkodg]

Envoy Gateway fails the endpoint asap so it can signal the peer to shift traffic to a healthy/new pod asap (using Connection: close / GOAWAY)

Im new to envoy gateway, but what does "signal to peer mean" here? Is it related to cluster internal routing controlled by / proxied through envoy gateway pods? Wont envoy gateway avoid directing new connections to the pod if its terminating?

The shutdown logic does 2 things here

1. fails healthchecks so
   • proxy readiness checks fail
   • any Cloud L3/L4 LB checks using a configured health check ( configured in CTP) also fail
     This is signalling to the Cloud LB peer to not forward new connections to the terminating proxy, but continue to route existing connections to it, so requests on them can be allowed to finish (iirc cilium proactively cleans up L3 connectivity to this endpoint to this terminating proxy which is the issue)
2. starts off listener drains, which starts L7 draining by sending Connection: close / GOAWAY so the L7 peer to close the connection and start a new one

[consideRatio]

I hear there may be a reason to fail readiness checks to signal to some LBs, but i know some LBs wont need it as being in a termenating state is enough - like a common one for GCP based GKE clusters Service resources of type loadbalancer - they dont need the readiness probe to start failing - and doing so can rather cause issues.

To me, that makes the case for not delaying both mechanisms (fail readiness AND start sending GOAWAY) at the same time, but only the failing of readiness checks.

• delay just failing readiness: feat: add healthCheckFailureDelay to shutdown config #9363
• delay GOAWAY and failing readiness: feat: add drain delay to graceful shutdown process #9211