Agent crash-loops with misleading 'OTEL collector not reachable / Protocol mismatch' — actually authbridge-proxy resets OTLP egress (collector missing from NO_PROXY)

[yoavkatz]

Summary

Agent pods crash-loop with a misleading "OTEL is enabled but collector is not reachable: Protocol mismatch" error. The OTEL collector is actually healthy and reachable — the real cause is that the agent's OTLP telemetry egress is forced through the authbridge-proxy sidecar, which is not configured to route the collector host and resets the connection. The exgentic library then misreports the reset ([Errno 104] Connection reset by peer) as a protocol/reachability problem, and the agent exits 1 → CrashLoopBackOff.

Symptoms

Agent container logs:

Starting Exgentic A2A Agent
Agent: tool_calling
...
Error ─────────────────────────────────────────────
 OTEL is enabled but collector is not reachable: Protocol mismatch:
 configured as HTTP but server
 http://otel-collector.kagenti-system.svc.cluster.local:8335 may be gRPC.
 Error: [Errno 104] Connection reset by peer
stream closed: EOF for team1/exgentic-a2a-tool-calling-...-parallel-1 (agent)

Pod state: CrashLoopBackOff, container agent Ready=false with repeated restarts, while sidecar authbridge-proxy is Ready=true.

Root cause

The error message is misleading — it is not a protocol mismatch and the collector is reachable.

The agent pod contains:

• agent container — the workload (ghcr.io/exgentic/exgentic-a2a-tool_calling:latest)
• authbridge-proxy sidecar (ghcr.io/kagenti/kagenti-extensions/authbridge:...) listening on 127.0.0.1:8081
• proxy-init initContainer (ghcr.io/kagenti/kagenti-extensions/proxy-init:...) that configures egress through the proxy

Relevant agent env:

OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector.kagenti-system.svc.cluster.local:8335
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
EXGENTIC_OTEL_ENABLED:        true
HTTP_PROXY:                   http://127.0.0.1:8081
HTTPS_PROXY:                  http://127.0.0.1:8081
NO_PROXY:                     127.0.0.1,localhost

Because otel-collector.kagenti-system.svc.cluster.local is not in NO_PROXY, the OTLP HTTP exporter routes through the authbridge-proxy. The authbridge is not configured to route/authorize the collector host, so it resets the connection. The exgentic OTEL preflight check surfaces that reset as a bogus "Protocol mismatch: configured as HTTP but server may be gRPC," then the agent hard-fails and exits.

Verification that the collector itself is fine

The collector runs the OTLP HTTP receiver on port 8335 (relocated from the default 4318 via a launch flag):

/otelcol-contrib --config=/etc/otelcol-config/base.yaml \
  --set receivers::otlp::protocols::http::endpoint=0.0.0.0:8335

Probing directly (from a plain pod in kagenti-system with no proxy):

POST http://otel-collector:8335/v1/traces   -> HTTP 200   (healthy OTLP HTTP receiver)
GET  http://otel-collector:8335/             -> HTTP 404   (expected for OTLP HTTP)
GET  http://otel-collector:4317/v1/traces    -> 000        (4317 is the gRPC port, resets over HTTP)

So the endpoint URL and protocol configured on the agent are correct; the collector is reachable when the proxy is bypassed.

How to reproduce

1. Deploy an exgentic A2A workload (e.g. exgentic-a2a-tool-calling / gsm8k) into a namespace where pods get the kagenti authbridge-proxy sidecar + proxy-init initContainer, with EXGENTIC_OTEL_ENABLED=true and OTEL_EXPORTER_OTLP_ENDPOINT pointing at the in-cluster OTEL collector on :8335.
2. Observe the agent container crash-loop with the "collector is not reachable: Protocol mismatch" error above.
3. Confirm the collector is actually healthy from a proxy-less pod:

   kubectl run otel-probe --rm -i --restart=Never --image=curlimages/curl -n kagenti-system -- \
     curl -s -o /dev/null -w "%{http_code}\n" -X POST \
     http://otel-collector:8335/v1/traces -H "Content-Type: application/json" -d '{}'
   # -> 200
   

4. Confirm the collector host is not excluded from proxying:

   kubectl get pod <agent-pod> -n team1 \
     -o jsonpath='{range .spec.containers[?(@.name=="agent")].env[*]}{.name}={.value}{"\n"}{end}' \
     | grep -i proxy
   # NO_PROXY=127.0.0.1,localhost   (collector host absent)
   

Suggested fixes (any one)

1. Add the collector to NO_PROXY so telemetry bypasses the authbridge (preferred — telemetry does not need auth-bridging):

   NO_PROXY=127.0.0.1,localhost,otel-collector.kagenti-system.svc.cluster.local,.svc.cluster.local
   

2. Add otel-collector.kagenti-system.svc.cluster.local:8335 to the authbridge-proxy allowlist/route config if telemetry is intended to be proxied.
3. Make the exgentic OTEL preflight non-fatal (warn instead of exit 1) and/or fix the misleading "Protocol mismatch" message so a proxy reset is reported accurately.

The env vars and sidecar injection are set by the deployment/platform layer, not by exgentic_a2a_runner (the runner only reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment), so the NO_PROXY fix belongs in the workload-harness deployment tooling.

[huang195]

@yoavkatz I would suggest add the following in the authbridge-config to let otel traffic go through:

listener:
  skip_hosts:
    - "otel-collector.kagenti-system.svc.cluster.local"
    - "otel-collector"          # add the short name too, if the exporter ever uses it

[yoavkatz]

Fix: bake OTEL-collector skip_hosts into the namespace AuthBridge base before the agent is deployed

Motivation. When AuthBridge is enabled, the injected proxy sidecar intercepts the agent's outbound OTLP export to the OTEL collector and resets the connection — surfacing as the misleading "collector not reachable / protocol mismatch" crash-loop. The collector needs to be on the proxy's bypass list (listener.skip_hosts).

Why not patch the per-agent config after deploy. The rossoctl operator webhook force-overwrites the per-agent authbridge-config-<agent> ConfigMap (server-side apply, ForceOwnership) on every pod admission, so anything patched there post-deploy races the operator and triggers redeployments/reload churn. There's also no per-agent inline config field in POST /api/v1/agents or the AgentRuntime CR.

The fix. deploy-agent.sh now, in a new Step 7.5 (runs before the agent POST, only when AuthBridge is enabled and kubectl is available), merges listener.skip_hosts into the namespace-level authbridge-runtime-config ConfigMap:

listener:
  skip_hosts:
    - otel-collector.rossoctl-system.svc.cluster.local
    - otel-collector

The operator reads that namespace base at admission and merges the listener map into each per-agent CM (it only injects reverse_proxy_addr/reverse_proxy_backend/forward_proxy_addr), so the OTEL bypass is present from the pod's first boot — no post-deploy patch, no reload race. The merge is idempotent (union + sort + de-dupe, preserves any existing skip_hosts and the pipeline: block) so redeploys hit a no-op. Every failure mode (missing PyYAML, missing namespace CM, bad YAML, failed apply) fails hard with exit 1.

Caveats. The change is namespace-wide (affects all agents in the namespace — appropriate since the collector is shared). skip_hosts is an AuthBridge-binary listener key and hasn't been verified against the sidecar's config schema; if the sidecar rejects the config, check the sidecar container logs for a reload/parse error.