Skip to content

binding-mcp: add telemetry event for south-route cache-hydration failure #2391

Description

@jfallows

Problem

McpEventContext (added in #2070 / #2074) covers session lifecycle, bearer-auth rejection, and elicitation timeout — but it defines nothing for the proxy's own south-route cache-hydration path, which is a distinct area #2070 didn't survey. Right now every failure mode in that path is silent by design:

  • McpProxyCacheHydrater.normalizeItems() catches JsonException while reordering a cached list response and falls back to the unreordered items with no log, no event:
    catch (JsonException ex)
    {
        result = items;
    }
  • McpClientFactory's onDecodeParseError / onDecodeInvalidResponse overrides (the abstract hooks fired when a south server's initialize/list response fails to parse or is structurally invalid) do nothing but cleanupNet(traceId, authorization) — no log, no event.
  • McpProxyCacheManager.onError(int kind) — the terminal callback once every pending route for a given list kind (tools/prompts/resources) has settled as failed — only calls scheduleHydrateRetry(kind), which arms an exponential-backoff Signaler retry, again with no log, no event.

The net effect: a south server that never successfully populates the cache for a given kind — whether from a single malformed response, a connection that resets mid-stream, or a persistently incompatible payload shape — retries forever with backoff and leaves zero trace anywhere in the event system. The only symptom visible to an operator is that kind's tools/resources/prompts never appearing in tools/list et al., with no way to distinguish "still warming up," "upstream is down," and "upstream's response doesn't parse" from each other after the fact.

Criteria (same framework used in #2070)

Applying the same five criteria #2070 established for this binding's event surface:

  1. Fires alongside a real consequence — yes: a hydration retry is scheduled (or a normalizeItems fallback is taken) every time this fires; never a disconnected observation.
  2. traceId=0 for config/attach-time diagnostics, real traceId for stream-scoped events — this is attach-scoped (a south route, not a single request), so traceId=0 fits, matching how binding-kafka/binding-mqtt treat connection-level diagnostics.
  3. Structured fields point at the actionable resource — the failing route's kind (tools/prompts/resources) and a reason enum, not a bare "something happened."
  4. Collapse sub-causes into one event + a reason enum — parse error, invalid response, and route-settle failure all collapse into one HYDRATE_FAILED event with a McpHydrateError reason, mirroring AUTHORIZATION_FAILED's McpAuthorizationError pattern rather than three near-duplicate event types.
  5. No event without a confirmed call site — all three sites above are read from current code, not inferred from names.

Proposed event

scope event
{
    enum McpEventType (uint8)
    {
        SESSION_ESTABLISHED (1),
        SESSION_CLOSED (2),
        AUTHORIZATION_FAILED (3),
        ELICITATION_TIMEOUT (4),
        HYDRATE_FAILED (5)
    }

    enum McpHydrateError (uint8)
    {
        PARSE_ERROR (0),
        INVALID_RESPONSE (1),
        ROUTE_FAILED (2)
    }

    struct McpHydrateFailedEx extends core::stream::Extension
    {
        string16 kind;
        McpHydrateError error;
        int64 retryAt = -1;
    }

    union McpEventEx switch (McpEventType)
    {
        case SESSION_ESTABLISHED: McpSessionEstablishedEx sessionEstablished;
        case SESSION_CLOSED: McpSessionClosedEx sessionClosed;
        case AUTHORIZATION_FAILED: McpAuthorizationFailedEx authorizationFailed;
        case ELICITATION_TIMEOUT: McpElicitationTimeoutEx elicitationTimeout;
        case HYDRATE_FAILED: McpHydrateFailedEx hydrateFailed;
    }
}

Call sites (confirmed by reading current code)

Fires from Reason Available context
McpClientFactory's onDecodeParseError override PARSE_ERROR traceId, authorization, routedId from the enclosing stream
McpClientFactory's onDecodeInvalidResponse override INVALID_RESPONSE same
McpProxyCacheManager.onError(int kind), immediately before scheduleHydrateRetry(kind) ROUTE_FAILED kind, and the computed next retry time from the backoff calculation already in scheduleHydrateRetry

McpProxyCacheHydrater.normalizeItems()'s fallback is a degraded-but-successful outcome (the unreordered list is still served), not a hydration failure, so it's deliberately left out of this event — flagging it here only so it isn't mistaken for an omission.

Plan

  1. Add the HYDRATE_FAILED addition to the scope event block in specs/binding-mcp.spec/src/main/resources/META-INF/zilla/mcp.idl
  2. Extend McpEventContext/McpEventFormatter/McpEventFormatterFactory in runtime/binding-mcp
  3. Wire the three call sites above
  4. Unit tests covering the new event's construction/firing, matching existing *EventContext/*EventFormatter coverage

Following the repo's test-first discipline, tests land before the implementation wiring.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions