Route OTEL collector around authbridge sidecar via skip_hosts#45
Open
yoavkatz wants to merge 1 commit into
Open
Route OTEL collector around authbridge sidecar via skip_hosts#45yoavkatz wants to merge 1 commit into
yoavkatz wants to merge 1 commit into
Conversation
Agent pods crash-loop with a misleading "OTEL is enabled but collector is not reachable: Protocol mismatch" error. The real cause is that the agent's OTLP egress is forced through the authbridge sidecar, which is not configured to route the collector host and resets the connection. The exgentic library then misreports the reset as a protocol/reachability problem and exits 1 → CrashLoopBackOff. deploy-agent.sh: bake OTEL-collector skip_hosts into the namespace authbridge base (authbridge-runtime-config) BEFORE the agent is created, so the sidecar has it from first pod boot — no post-deploy patch, no reload race. The operator webhook force-overwrites per-agent ConfigMaps but honors the namespace base and merges it into each per-agent CM, so a listener.skip_hosts added here rides through untouched. The merge is idempotent (union + sort + de-dupe) for byte-stable re-runs. wait-for-reload.sh: during a rollout, probe every sidecar-bearing pod each poll and succeed as soon as any reports the target SHA, instead of assuming .items[0]. The old assumption often picked the old/terminating or agent-only pod and either found no sidecar or waited forever on a config that would never advance. Fixes rossoctl#34 Signed-off-by: Yoav Katz <katz@il.ibm.com>
Member
|
Tested locally and works for me |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Agent pods crash-loop with a misleading "OTEL is enabled but collector is not reachable: Protocol mismatch" error (#34). The collector is actually healthy — the real cause is that the agent's OTLP egress is forced through the
authbridgesidecar, which isn't configured to route the collector host and resets the connection. The exgentic library misreports the reset ([Errno 104] Connection reset by peer) as a protocol/reachability problem and the agent exits 1 →CrashLoopBackOff.Changes
deploy-agent.sh— bake OTEL-collectorskip_hostsinto the namespace authbridge base (authbridge-runtime-config) before the agent is created, so the sidecar has it from first pod boot — no post-deploy patch of the running agent, no reload race.config.yamlinto each per-agent CM, so alistener.skip_hostsadded here rides through untouched.authbridge/wait-for-reload.sh— during a rollout, probe every sidecar-bearing pod each poll and succeed as soon as any reports the target SHA, instead of assuming.items[0].Notes
skip_hostsis an AuthBridge-binary listener key; it is not defined in operator source and has not been verified against the sidecar's config schema. If the sidecar rejects the merged config, areload failed/ parse error will appear in the sidecar container logs — the key name/shape may need correcting. This is documented inline.Fixes #34