Remote streamable-http session goes stale → gateway returns silent "success" with empty results (writes lost), no error, no reconnect

[gabiudrescu]

I'm a product owner with enough technical depth to be dangerous, leaning on Claude to do the deeper digging - so treat the analysis below as "carefully investigated, but I haven't read your Go." I've been running the gateway in front of Atlassian's hosted MCP (some context on that setup here: https://productowner.ro/blog/almost-converted-mcp-json-to-toon/) and hit a reproducible failure that silently loses writes. Reporting it carefully because the failure mode is easy to miss.

TL;DR

After a remote MCP server's streamable-http session goes stale, the gateway keeps proxying calls but they return empty results reported as success - in tens of milliseconds, with no error logged anywhere, even at --verbose. Reads coming back blank is annoying; the real problem is on writes: an updateConfluencePage / editJiraIssue reports success but never actually applies the change - the upstream version doesn't increment, and the edit is lost with no signal. I've lost a Jira sprint write and two Confluence pushes this way before noticing.

/mcp reconnect (which respawns the gateway process) restores it for a while, then it recurs.

This is adjacent to #412 but distinct: #412 is the gateway as a server dropping client connections with a hard error. This is the gateway as a client to a remote upstream, where the failure is a silent empty success - worse, because nothing surfaces it.

Plain-language vs. technical

What I see	What's actually happening
🔴 A write "succeeds" but didn't take.	updateConfluencePage/editJiraIssue return success; upstream version unchanged. Silent data loss, no error.
🟠 Reads suddenly come back empty.	Tool calls "completed successfully" but with empty content.
🟡 It happens after the gateway's been up a little while, under load.	Works fine for ~10-15 min after a fresh start, then stops returning real data - frequently in the middle of a run of write calls.
⚪ Reconnecting "fixes" it temporarily.	/mcp reconnect mints a fresh upstream session → healthy again for a while.

Environment

• docker mcp plugin v0.42.1 (gateway advertises serverVersion Docker AI MCP Gateway 2.0.1 on the wire)
• Docker Desktop 29.5.2, macOS (Apple Silicon)
• Client: Claude Code (stdio transport to the gateway)
• Remote server: Atlassian's hosted MCP - https://mcp.atlassian.com/v1/mcp, transport=streamable-http, authenticated via the gateway's built-in OAuth (docker mcp oauth authorize)
• Gateway launched with --verbose and --servers <list incl. the remote>

Reproduction

1. Add a remote streamable-http MCP server that uses the gateway's OAuth (I used atlassian-remote).
2. Run the gateway and exercise the remote's tools normally (reads + writes) over ~10-20 minutes.
3. Observe healthy calls completing in ~0.6-2s.
4. After a while (for me ~11-15 min from a fresh process, sooner under a burst of writes), every call to that remote starts "completing successfully" in 40-70ms with empty content. Writes report success but don't commit.
5. /mcp reconnect → healthy again for another ~10-15 min.

Evidence (from the Claude Code MCP debug log, identifiers scrubbed)

Healthy (fresh process), real upstream round-trips:

11:08:30  Calling MCP tool: <remote>__searchJiraIssuesUsingJql
11:08:31  completed successfully in 800ms
11:09:00  Calling MCP tool: <remote>__editJiraIssue
11:09:01  completed successfully in 1s

Wedged (~6 min later, same process), instant empty "successes":

11:15:38  Calling MCP tool: <remote>__searchJiraIssuesUsingJql
11:15:38  completed successfully in 67ms
11:15:46  ... 62ms
11:15:50  ... 58ms
11:17:30  Calling MCP tool: <remote>__updateConfluencePage
11:17:30  completed successfully in 61ms      <-- write "succeeded"; page version did NOT change
11:17:38  Calling MCP tool: <remote>__getConfluencePage
11:17:38  completed successfully in 48ms      <-- empty

Two things I think matter:

• --verbose emits nothing per-request. The only stderr I get is the startup banner (config read, image pulls, tool listing, OAuth loop start). During the wedge: zero stderr - no 401, no "session not found", no HTTP status. So from the logs there's no error to act on; the gateway just returns empty.
• OAuth refresh is running, yet it still wedges. The startup banner shows Starting OAuth notification monitor and Started OAuth provider loop for <remote>. So this doesn't look like access-token expiry (refresh is active) - it looks like the upstream streamable-http session itself being recycled/invalidated, with the gateway not re-establishing it and not detecting that responses are empty.

This lines up with the same root class reported elsewhere - server-side streamable-http session invalidation with no client-side re-establish: github/gh-aw#23153, anomalyco/opencode#25137, NousResearch/hermes-agent#13383.

What seems to be the right fix (and why I think it stalled)

The keepalive direction looks correct. A commit referenced from #412 - 71b0a90 "fix(sessions): pong response resets inactivity timer (MCP-spec liveness…)" - is exactly the shape I'd expect (don't let an idle session lapse). But as far as I can tell it never landed: the commit 404s (the fork it lived on is no longer reachable), there's no open PR carrying it, and it's in no release (latest is v0.42.2, whose only session-related change is "reuse containers per session" - local containers, not the remote upstream).

Two complementary asks, in priority order:

1. Don't report empty upstream responses as success. When a streamable-http call to a remote returns empty/no-content where data is expected (or the session is gone), surface an error to the client instead of a silent empty success. Silent success on writes is the data-loss vector; even without a full reconnect fix, failing loudly would stop the bleeding.
2. Re-establish a recycled upstream session (the keepalive/71b0a90 direction, plus reconnect-on-stale-session), so it doesn't wedge in the first place.

Confidence

High confidence on the symptom and timing, medium on the exact mechanism - the session-recycling call is inferred from behavior + logs, not source. Happy to provide a fuller --verbose log, test a patch against my setup (I can reproduce on demand), or help however's useful.

[gabiudrescu]

Proposed fix (code-grounded), plus one new piece of evidence

Following up on my own issue. Since filing, I cloned the gateway and read the relevant paths, and I captured the one thing the original report was missing: what a legitimate empty result actually looks like on the wire. That turns out to be the key to a safe trigger, so I'm laying out a concrete fix below. File/line references are against current main; the function names are the stable part.

New evidence: legitimate "no results" is structurally non-empty

The worry with "treat empty as stale" is false positives: a normal search that simply matches nothing. So I drove the gateway over stdio (docker mcp gateway run --servers atlassian-remote) and ran a JQL guaranteed to match nothing. The raw result:

"result": {
  "content": [
    { "type": "text", "text": "{\n  \"issues\": [],\n  \"isLast\": true\n}" }
  ]
}

A healthy "zero rows" response is structurally non-empty: content has one text block whose payload happens to be an empty list. The wedge, by contrast, produces a result whose content array is itself empty (no blocks), with isError absent, in 40-70ms.

That gives a trigger that does not fire on legitimate no-match queries: content has zero blocks (or is otherwise degenerate), not "the payload represents an empty list." The two are different things, and only the first is the wedge.

Where it goes wrong in the code

1. The tool handler never inspects the result. In mcpServerToolHandler (pkg/gateway/handlers.go), the call is result, err := client.Session().CallTool(ctx, params) and the only check is if err != nil. On err == nil it returns result straight back as success, regardless of whether result.Content is empty or result.IsError is set. So an empty upstream response is forwarded as a successful tool result.

2. The dead remote client stays cached. For a remote server, clientPool.longLived() returns true unconditionally (pkg/gateway/clientpool.go: "Remote servers are always treated as long-lived since no container is involved"). So the gateway keeps one client per remote in keptClients, and AcquireClient returns that same cached client on every later call. There is exactly one eviction line, delete(cp.keptClients, key), and it only runs when building the client fails (getter.GetClient returns an error). A client that builds fine and later returns empty is never evicted. Hence the wedge persists until the whole process is respawned (/mcp reconnect), which is what clears the map.

Proposed fix

The two asks in the issue map cleanly onto code that, in part, already exists.

Ask 1 (stop the bleeding): do not forward an empty upstream success as success.
In the tool handler, after CallTool returns err == nil, add a check: if the result is structurally empty (no content blocks) and isError is not set, do not return it as a successful tool result. This alone converts silent data loss into a visible failure, which is the most important fix for writes.

Ask 2 (heal it): re-establish just the stale upstream session, then retry.
On detecting that empty-success, evict and rebuild only the affected server's client:

• Evict with the eviction that already exists: delete(cp.keptClients, key). Because the cache key is clientKey{serverName, session}, this drops only that one remote's client; every other server behind the gateway keeps its live session. This is a per-server restart, not a whole-gateway respawn.
• Rebuild by calling AcquireClient again: with the key gone, the getter == nil branch constructs a fresh client (a fresh OAuth'd streamable-http session).
• Retry the call once against the fresh client.

Read vs write on the retry. A blind retry is unsafe for a write: the first attempt may have committed upstream and only lost its response, so re-running it could double-apply. The signal for whether a retry is safe is already on the wire. Every tool in this remote advertises MCP tool annotations, and they pass through the gateway intact (I read them via tools/list through the gateway). Captured examples:

tool	readOnlyHint	idempotentHint
searchJiraIssuesUsingJql	true	true
getConfluencePage	true	true
editJiraIssue	(absent)	true
updateConfluencePage	(absent)	true
createJiraIssue	(absent)	(absent)

That yields a three-tier retry rule:

• readOnlyHint: true -> auto-retry once and return the fresh result transparently (reads).
• not read-only but idempotentHint: true -> still safe to auto-retry, re-applying the same edit converges (editJiraIssue, updateConfluencePage).
• neither -> do not auto-retry; surface the failure (per Ask 1) and let the client decide. This is exactly the duplicate-creating case (createJiraIssue).

These are server-supplied hints, not guarantees, which is fine for an auto-retry decision; and "surface when the hint is absent" stays safe if a server omits them.

These annotations are optional, and the fix does not depend on them. In the SDK the gateway compiles against (go-sdk/mcp/protocol.go), Tool.Annotations is a nil-able pointer (*ToolAnnotations, omitempty) and the individual hints are optional, so a server may send none. The SDK's own documented defaults all lean safe (absent readOnlyHint -> not read-only; idempotentHint default false; destructiveHint default true), i.e. an absent hint means "do not auto-retry." That cleanly splits the fix by what it relies on:

• Ask 1 (do not report empty-success as success) is annotation-independent. It protects every server, hints or not. This is the data-loss fix and must not depend on annotations.
• Ask 2's auto-retry is a convenience layer that activates only when a tool affirmatively declares readOnlyHint: true or idempotentHint: true. For a server that omits them, it falls back to surfacing the failure: no auto-heal, but no silent loss either, and no worse than today.

So the correctness guarantee never rests on the upstream being a well-behaved MCP build; only the auto-heal convenience does, and it fails safe.

Why this is minimal

The per-server cache (keptClients keyed by server) and the eviction line (delete(cp.keptClients, key)) already exist; the fix calls them on a new trigger (empty-success) instead of only on a build failure. The genuinely new code is (a) the structurally-empty detector in the tool handler and (b) a retry-once wrapper gated on read/write.

What I have not confirmed

I have ground truth for the healthy empty result (captured above). I have not captured the exact wedge bytes yet (whether content is [], omitted, or otherwise degenerate), because that needs catching it live. I have a wire-logging patch ready for headerRoundTripper.RoundTrip in pkg/mcp/remote.go (gated behind an env var, redacts the bearer token, captures non-SSE bodies only) and can run it to record the exact wedge response if that would help pin the detector's precise condition. Happy to do that, or to test a patch against my setup, which reproduces on demand.