feat(mcp): stateless spec modern fanout routing and server discovery ( 28-07-2026 ) - #2545
Conversation
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2545 +/- ##
==========================================
+ Coverage 86.33% 86.39% +0.05%
==========================================
Files 183 184 +1
Lines 24509 24945 +436
==========================================
+ Hits 21160 21550 +390
- Misses 2168 2194 +26
- Partials 1181 1201 +20 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
…dd-modern-fanout-logic Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
|
/retest |
| headerVersion := r.Header.Get(mcpProtocolVersionHeader) | ||
| if headerVersion == "" { | ||
| headerVersion = protocolVersion20260728 | ||
| } | ||
| if !isSupportedVersion(headerVersion) { | ||
| errType = metrics.MCPErrorUnsupportedProtocolVersion | ||
| err = fmt.Errorf("unsupported protocol version: %s", headerVersion) | ||
| onErrorResponse(w, http.StatusBadRequest, fmt.Sprintf("unsupported protocol version: %s", headerVersion)) | ||
| return |
There was a problem hiding this comment.
Since this PR depends on #2518 , should this handler consume the validated result from detectClientEra instead of independently validating or inferring the version?
Inline with that, what's the intended request flow after rebase? Is it servePOST` → `detectClientEra` → reject or dispatch a validated request → `serveModernPOST?
Broader question: can we try and centralize validation to ensure it's consistent and there's one place to make changes/evolve it? Also, have cleaner separation of concerns.
There was a problem hiding this comment.
Since this PR depends on #2518 , should this handler consume the validated result from detectClientEra instead of independently validating or inferring the version?
Right, I think we should avoid these version checks here, and detectClientEra function in the previous PR. Nevertheless, all the necessary checks are already present in #2518 itself.
Inline with that, what's the intended request flow after rebase? Is it servePOST
→detectClientEra→ reject or dispatch a validated request →serveModernPOST?
yes
Broader question: can we try and centralize validation to ensure it's consistent and there's one place to make changes/evolve it? Also, have cleaner separation of concerns.
the detectClientEra function in the previous PR takes care of that
| var results []*mcp.DiscoverResult | ||
| for _, backend := range routeConfig.backends { | ||
| backendStartAt := time.Now() | ||
| result, err := m.discoverBackend(ctx, route, backend) |
There was a problem hiding this comment.
is it possible for a request to receive a backend (and it's tools/prompts/resources) that isn't authorized?
Can the backend selection be a separate step that produces the backend set consumed downstream (discovery, fanout)? Trying to see if we can clearly define the input interface here. I was thinking a validated request + authorized backend set for the request.
There was a problem hiding this comment.
is it possible for a request to receive a backend (and it's tools/prompts/resources) that isn't authorized? Can the backend selection be a separate step that produces the backend set consumed downstream (discovery, fanout)? Trying to see if we can clearly define the input interface here. I was thinking a validated request + authorized backend set for the request.
Agree, earlier on initialize we used to call newSession() which used to set the selected authorized backends, and tools/list prompts/list.... used to iterate through the selected backends returned from newSession(). Since in modern, we dont have any initialize and every request is stateless, reevaluating every request is actually the right stateless behavior, a new token can change which backends are visible. Legacy cannot do that mid session because the set is fixed at initialize.
| if err != nil { | ||
| m.l.Warn("server/discover failed for backend", | ||
| slog.String("backend", backend.Name), | ||
| slog.String("error", err.Error())) | ||
| backendMetrics.RecordMethodErrorCount(ctx, req.Method, nil, metrics.MCPStatusError) | ||
| backendMetrics.RecordRequestErrorDuration(ctx, backendStartAt, errorType(err), nil) | ||
| continue | ||
| } | ||
| if span != nil { | ||
| span.RecordRouteToBackend(backend.Name, "", true) | ||
| } | ||
| backendMetrics.RecordMethodCount(ctx, req.Method, nil) | ||
| backendMetrics.RecordRequestDuration(ctx, backendStartAt, nil) | ||
| results = append(results, result) | ||
| } | ||
| if len(results) == 0 { | ||
| m.l.Error("server/discover failed for all backends", slog.String("route", route)) | ||
| onErrorResponse(w, http.StatusInternalServerError, "failed to discover any backend") | ||
| return handlerResult{}, errors.New("failed to discover any backend") | ||
| } | ||
| merged := mergeDiscoverResults(results) |
There was a problem hiding this comment.
what happens when there are partial discovery results? when there are temporary failures on a backend?
could it create a different backend set for a similar request? trying to see what the best possible response should be when we have that ambiguity
There was a problem hiding this comment.
For initialize in the previous spec, this was the standard we followed during fanout;
- if one of the backends failed, we don't return any error and pass the request if atleast one of the underlying backend returned a 200.
- and only throw the error, if all underlying backends fail to establish session. And I think it makes sense in a multiplexing scenario
- Ref: https://github.com/envoyproxy/ai-gateway/blob/79770ba7777caaa2870fa0c8cd0eeb1a2aa08ba5/internal/mcpproxy/mcpproxy.go#L224
| } | ||
| for _, v := range r.SupportedVersions { | ||
| if !slices.Contains(versions, v) { | ||
| versions = append(versions, v) |
There was a problem hiding this comment.
should we intersect against the gateway's supported versions here? to prevent accidentally advertising versions the gateway doesn't support
There was a problem hiding this comment.
Good point.
planning to not have any hardcoded logic for showing supported versions. Was thinking to directly return the merged list of supported versions from all the backends beneath. This PR #2543 handles that logic, and would want to use the same for showing supported versions.
| var rpcResp map[string]json.RawMessage | ||
| if err := json.Unmarshal(jsonPayload, &rpcResp); err != nil { | ||
| return nil, fmt.Errorf("parse response: %w (body: %.200s)", err, string(jsonPayload)) | ||
| } | ||
| if errField, ok := rpcResp["error"]; ok && string(errField) != "null" { | ||
| return nil, fmt.Errorf("backend error: %s", string(errField)) | ||
| } | ||
| result, ok := rpcResp["result"] | ||
| if !ok || string(result) == "null" { | ||
| return nil, fmt.Errorf("backend returned no result") | ||
| } | ||
| return result, nil |
There was a problem hiding this comment.
should we add some validation logic here?
some to things to validate:
- The payload is a valid JSON-RPC 2.0 response.
- The response ID matches the upstream request ID.
- Exactly one of
resultorerroris present. - Required modern fields are present and valid, including
resultType. - A malformed response is recorded as a backend failure and is not included in the aggregated result.
There was a problem hiding this comment.
Out of all the validations suggested, i think we can up few,
but validations like valid JSON-RPC 2.0 does not make sense for a proxy. Because once we are able to pull up result from the response, these extra checks do not change aggregation behavior
Exactly one of result or error is present.
Also for this, we anyways inject the resultType in the ensureResultType function on the outbound response. resultType is only mattered during MRTR cases like input_required, but given that we are deferring those changes in this phase, lets keep it simple?
swiftdiaries
left a comment
There was a problem hiding this comment.
Thanks for the context and the work here! Areas I focused on for the review:
- request validation at ingress
- backend selection logic
- discovery logic
- response handling
P.S Tried my best with the small context I have of the project and the code. So feel free to disregard things if they are not relevant
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
✅ Deploy Preview for theagentrouter ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Signed-off-by: Hritik Raj <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
| _, selectedBackends, err := m.resolveModernRouteBackends(w, route) | ||
| if err != nil { | ||
| return handlerResult{}, err | ||
| } | ||
|
|
There was a problem hiding this comment.
since we are calling it at every handler, can we call it at some top level?
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
|
/retest |
| @@ -648,9 +655,45 @@ func decodeCapabilityFlags(hex string) *mcpsdk.ServerCapabilities { | |||
| // If ANY backend supports a capability, the merged result includes it. | |||
| // Sub-fields like ListChanged and Subscribe are OR'd across all backends. | |||
| func (s *session) mergedCapabilities() *mcpsdk.ServerCapabilities { | |||
There was a problem hiding this comment.
do you need the separate helper method unionServerCapabilities for this?
| }() | ||
|
|
||
| route := r.Header.Get(internalapi.MCPRouteHeader) | ||
| if route == "" { |
There was a problem hiding this comment.
so in legacy, every session remembers which route it points to inside its session id. But its not the case with modern cause it being stateless, and if we skip this check, a missing header would look route <name> not found and return 404 masking the gateway bug
| // Start a tracing span for the request, mirroring the legacy path. The span | ||
| // is closed in recordPOSTCompletion. Fan-out handlers additionally record | ||
| // per-backend routing on the span via RecordRouteToBackend. | ||
| if params := modernParamsForHeaderMetadata(req); params != nil { |
There was a problem hiding this comment.
start span only for supportedMethod. Similar to legacy.go?
| // against this request. The returned map is the only backend set discovery and | ||
| // list fan-out may talk to. Writes 404 (unknown route) or 403 (no matching | ||
| // backends) on failure. | ||
| func (m *mcpRequestContext) resolveModernRouteBackends(w http.ResponseWriter, route filterapi.MCPRouteName) (*mcpProxyConfigRoute, map[filterapi.MCPBackendName]filterapi.MCPBackend, error) { |
There was a problem hiding this comment.
forwardHeader extraction before authorizing against backend selector?
There was a problem hiding this comment.
I think you also need to extract per backend forwardHeader. Can you please compare this with legacy flow to find what's missing and fix it?
| m.l.Warn("server/discover failed for backend", | ||
| slog.String("backend", backend.Name), | ||
| slog.String("error", err.Error())) | ||
| backendMetrics.RecordMethodErrorCount(ctx, req.Method, nil, metrics.MCPStatusError) |
There was a problem hiding this comment.
is this already being recorded in recordPOSTCompletion?
There was a problem hiding this comment.
Nope, we set perBackendMetrics=true which ends the span when it goes to defer block. So its not a duplicate
|
|
||
| // modernParamsForHeaderMetadata best-effort parses params for methods where | ||
| // addMCPHeaders can enrich upstream metadata (tool name/resource URI). | ||
| func modernParamsForHeaderMetadata(req *jsonrpc.Request) mcp.Params { |
There was a problem hiding this comment.
can this be reused from legacy?
There was a problem hiding this comment.
those helpers aren’t in legacy
| return nil | ||
| } | ||
|
|
||
| func writeJSONRPCResult(w http.ResponseWriter, id jsonrpc.ID, result any) { |
There was a problem hiding this comment.
same, can this be reused from legacy?
There was a problem hiding this comment.
those helpers aren’t in legacy
|
Reviewed this alongside #2661/#2663 — really solid refactor overall (nice that One thing worth a look before this lands: Two possible fixes for that part: add the same Separately, even short of total failure, it'd be worth surfacing which backends dropped out of a partial result and why — not just a server-side log line and a metric. Right now a caller who gets 8 tools back from a 10-backend route has no way to tell "that's everything" from "2 backends silently failed." |
|
+1 on the error when no backends are selected / all backends failed (+tests for both complete/partial failures) |
Thats a good point and would help in debugging in great detail. But in my opinion that can be done in another PR because if we are doing that, legacy code also need to stay consistent of logging partial backend failures and it will need few refactorings out of the scope of this PR. |
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
Signed-off-by: Hritik003 <hritik.raj@nutanix.com>
|
LGTM, this took a while thanks for being patient! |
Description
This PR adds phase0 support for modern fanout routing as per
validates modern protocol headers, fans out requests across backends, and aggregates responses into a single JSON-RPC result. It also extracts capability aggregation into a shared helper (unionServerCapabilities) so both stateful and stateless paths use the same capability-union behavior, and bumps github.com/modelcontextprotocol/go-sdk to v1.7.0 for modern MCP support.
Related Issues/PRs
Dependent PR
Special notes for reviewers (if applicable)
Please focus review on fanout/merge behavior and protocol compliance in internal/mcpproxy/modern.go, especially:
this PR also has some common code with feat(mcp): stateless spec era detection and legacy split (28-07-2026) #2518 which will be adjusted and rebased once the first PR goes in.
Testing
Built and deployed the changes to local cluster, tested the functionality using MCP inspector.