Summary
When a client reconnects to an existing stateful gRPC session, grpc-proxy hands the session back to its worker by publishing a reconnect message on stateful_session.reconnect.<requestId>. This is a plain core NATS publish, which succeeds whether or not anything is subscribed.
If the worker that owned the session is no longer there, the message is discarded by the server and the publish still returns success. The proxy reports a healthy rejoin, waits for a worker that cannot arrive, and the client is left holding a session cookie it presents again on every retry. No error is surfaced on any path, so nothing recovers.
Impact
Sessions in this state produce sustained client-visible errors and do not self-heal. The only reliable remedy today is restarting or rolling over the function, which is operator action for what should be an automatic recovery.
Why it does not recover on its own
The recovery mechanism already exists. StreamDirector.ServeHTTP clears the client's request id cookie when a request fails with ErrSessionNotFound, and a client that drops the cookie opens a fresh session on its next request. That path is never reached, because the rejoin has no way to learn that the session is dead and so never produces ErrSessionNotFound.
Two smaller gaps contribute:
- The worker's reconnect listener does not acknowledge receipt, so subscription interest is the only available liveness signal.
- The worker subscribes to the reconnect subject only after its first CONNECT completes, leaving a window during session establishment where a live session has nothing listening.
Precedent
The stateless polling path already does this correctly. polling_request uses request/reply and maps a no-responders answer onto "no worker picked this up", and the caller turns that into a clear client-visible error. The worker's polling listener acknowledges receipt. The stateful reconnect path is the same shape minus the acknowledgement and the delivery check.
Proposed fix
Make the rejoin a request rather than a publish, treat a confirmed no-responders answer as ErrSessionNotFound, acknowledge reconnects on the worker side, and subscribe before the first CONNECT.
Out of scope
This addresses sessions that cannot be handed back to a worker. It does not address worker CONNECT tokens expiring while work waits for a concurrency slot, which is a separate failure mode.
Summary
When a client reconnects to an existing stateful gRPC session,
grpc-proxyhands the session back to its worker by publishing a reconnect message onstateful_session.reconnect.<requestId>. This is a plain core NATS publish, which succeeds whether or not anything is subscribed.If the worker that owned the session is no longer there, the message is discarded by the server and the publish still returns success. The proxy reports a healthy rejoin, waits for a worker that cannot arrive, and the client is left holding a session cookie it presents again on every retry. No error is surfaced on any path, so nothing recovers.
Impact
Sessions in this state produce sustained client-visible errors and do not self-heal. The only reliable remedy today is restarting or rolling over the function, which is operator action for what should be an automatic recovery.
Why it does not recover on its own
The recovery mechanism already exists.
StreamDirector.ServeHTTPclears the client's request id cookie when a request fails withErrSessionNotFound, and a client that drops the cookie opens a fresh session on its next request. That path is never reached, because the rejoin has no way to learn that the session is dead and so never producesErrSessionNotFound.Two smaller gaps contribute:
Precedent
The stateless polling path already does this correctly.
polling_requestuses request/reply and maps a no-responders answer onto "no worker picked this up", and the caller turns that into a clear client-visible error. The worker's polling listener acknowledges receipt. The stateful reconnect path is the same shape minus the acknowledgement and the delivery check.Proposed fix
Make the rejoin a request rather than a publish, treat a confirmed no-responders answer as
ErrSessionNotFound, acknowledge reconnects on the worker side, and subscribe before the first CONNECT.Out of scope
This addresses sessions that cannot be handed back to a worker. It does not address worker CONNECT tokens expiring while work waits for a concurrency slot, which is a separate failure mode.