Best Practices for Load Shedding in Envoy Gateway

[am36166]

Hello,

I'm looking for guidance on implementing load shedding in Envoy Gateway to protect backend services under high load.

In networking, mechanisms such as Random Early Detection (RED) and traffic policing proactively prevent congestion by dropping or limiting traffic before the network becomes saturated. I'm looking for the equivalent approach for HTTP/API traffic using Envoy Gateway.

Specifically, I'd like to know:

Does Envoy Gateway support proactive load shedding based on resource pressure (e.g., request latency, concurrency, queue depth, CPU utilization, or other overload signals)?
Is Envoy's adaptive concurrency filter currently supported and configurable through Envoy Gateway?
What is the recommended way to reject excess requests before backend services become overloaded?
Are there best practices for combining:
Local or global rate limiting
Circuit breakers
Adaptive concurrency
Overload Manager
Kubernetes HPA
If some of these capabilities are not yet exposed by Envoy Gateway, what is the recommended production approach today?

Our goal is not only to enforce rate limits, but to gracefully shed load when the platform approaches its safe operating capacity, ensuring that the system remains responsive instead of allowing latency to grow until services become unavailable.

If there are existing examples, documentation, or recommended configuration patterns for this use case, I would greatly appreciate being pointed to them.

Thank you!

[1991santhu]

Went looking in the repo for the specific mechanisms you listed, since the answer differs a lot between them.

Overload Manager is present, but as fixed bootstrap config rather than a user-facing API. internal/xds/bootstrap/bootstrap.yaml.tpl configures it directly:

overload_manager:
  refresh_interval: 0.25s
  resource_monitors:
  - name: "envoy.resource_monitors.global_downstream_max_connections"
    typed_config:
      max_active_downstream_connections: 50000
  {{- with .OverloadManager.MaxHeapSizeBytes }}
  - name: "envoy.resource_monitors.fixed_heap"
    ...
  actions:
  - name: "envoy.overload_actions.shrink_heap"
  - name: "envoy.overload_actions.stop_accepting_requests"

So there is memory-pressure shedding via shrink_heap and stop_accepting_requests, plus a downstream connection ceiling. What you can configure through EG is the heap size. The thresholds and the choice of actions are template constants. The admin/stats listener sets bypass_overload_manager: true, so observability survives shedding, which is the right call and worth knowing if you're watching a gateway under pressure.

Adaptive concurrency isn't in the typed API, but it is reachable. No translator path emits envoy.filters.http.adaptive_concurrency; the only repo hits are egctl testdata fixtures. You can get at it through EnvoyPatchPolicy, which patches the generated xDS directly, and the Overload Manager thresholds can be overridden via the EnvoyProxy bootstrap. So you aren't blocked. But a patch policy is an explicit escape hatch, schema-unvalidated and coupled to Envoy's xDS layout across upgrades, which makes it a fine answer to "can I do this at all" and a poor place for something you depend on.

Circuit breakers and rate limiting are both there, via BackendTrafficPolicy (connection, request and pending-request limits) and local plus global rate limiting.

Which leaves a real gap against what you're asking for. The mechanisms EG exposes are all static thresholds: a request-per-second limit, a max-connections number, a heap ceiling. Every one needs you to know the safe number in advance and keep it correct as the backend changes. Adaptive concurrency is the one that infers the limit from observed latency instead, which is the RED analogy you drew: react to the onset of congestion rather than to a preconfigured line.

For today the closest workable combination is probably BackendTrafficPolicy pending-request and connection limits sized below what the backend can absorb, with Overload Manager as a last-resort backstop for the gateway itself and HPA reacting on a slower timescale. That sheds load, but at a line you picked rather than at the point the backend actually degrades.

Is there interest in exposing adaptive concurrency through BackendTrafficPolicy? It looks like the missing piece here, and the filter already exists upstream in Envoy, so the work would be API surface and translation rather than new data-plane capability. Glad to help with that if maintainers think it's worth its own issue.

---

Edited: my original wording said adaptive concurrency is "not exposed", which overstated it. EnvoyPatchPolicy reaches it today. Follow-up comment below records the change.

[1991santhu]

One correction to what I wrote above. I said adaptive concurrency is "not exposed". That's true of the typed API but it overstates the position. EnvoyPatchPolicy exists and can patch the generated xDS directly, so envoy.filters.http.adaptive_concurrency is reachable today without waiting for anything. Same for the Overload Manager thresholds I described as template constants; the bootstrap is overridable through the EnvoyProxy resource.

So the accurate version of my answer is that every mechanism reachable through the typed API is a static threshold, and the latency-driven one is reachable only by dropping to a patch. That's a real difference for you rather than a technicality. EnvoyPatchPolicy is explicitly an escape hatch, it's schema-unvalidated, and it couples your config to Envoy's xDS layout across upgrades, so it's a reasonable way to answer "can I do this at all" and a poor place for something you depend on long term.

The substance of the question is intact: whether adaptive concurrency deserves first-class representation in BackendTrafficPolicy alongside the circuit breakers and rate limits already there, so it stops being an escape-hatch feature. I'd still be glad to help with that if maintainers want it. But you aren't blocked in the meantime and I shouldn't have implied you were.