proxy: no request-outcome metric to distinguish routing (500) / upstream (502) / client-closed (499) failures #3757

Summary

The reverse proxy (packages/shared/pkg/proxy, used by both client-proxy and the orchestrator sandbox proxy) has no request-outcome metric. Its only metrics are connection gauges (*.proxy.server.connections.open, *.proxy.pool.connections.open, *.proxy.pool.size). There is no counter that distinguishes how a proxied request ended:

• routing failure (destination could not be resolved) — answered 500
• upstream unreachable (destination resolved, TCP to the node failed) — answered 502
• client closed the connection before the server responded — surfaces as 499 in the gin layer, but the proxy data-plane has no equivalent counter
• sandbox not found
• too many incoming connections (connection-limit shed)
• success

Motivation

This is the exact blind spot behind the 2026-10-17 node-churn incident: during a scale-down, sandboxes on removed nodes surfaced as Failed to route request to sandbox, and because routing failures (500) and upstream-unreachable failures (502) were indistinguishable at the metric layer, the only way to tell them apart was scraping logs. #3691 / #3692 split the message; this issue adds the metric so the split is visible on dashboards and alertable (e.g. a spike in upstream_unreachable by node = stale routing records pointing at a dead node).

Proposal

Add a request-outcome counter to the shared proxy, recorded at the points where a request's fate is decided:

• handler() pre-routing terminal branches (sandbox_not_found, too_many_connections, routing_failed, client_closed)
• the pool ErrorHandler (upstream_unreachable, client_closed, routing_failed)
• ModifyResponse (ok)

Keep the shared proxy telemetry-agnostic: thread an OnRequestOutcome(ctx, outcome string) callback into reverseproxy.New (mirroring the existing ConnectionLimitConfig callbacks), and have each service own its counter + name:

• client_proxy.request.outcome
• orchestrator.proxy.request.outcome

labeled outcome = ok | routing_failed | upstream_unreachable | client_closed | sandbox_not_found | too_many_connections.

Refs #3691.