fix(observability): receive otlp on every interface in jaeger by Tiago0507 · Pull Request #248 · MicroTodoSuite/microservice-app-gitops

Tiago0507 · 2026-09-28T23:05:35Z

What changes

infrastructure/jaeger/config.yaml (economical) and infrastructure/profiles/full/jaeger/components/elasticsearch/config.yaml (full) now declare the OTLP receiver endpoints explicitly: grpc: endpoint: 0.0.0.0:4317 and http: endpoint: 0.0.0.0:4318.
tests/contract/observability.sh requires both endpoints in both files.
spec 006 gains T055.

Why

On the rebuilt lex-mts-eco-eks-main Jaeger received no trace. Every service exporter failed, for example auth-api:

traces export: exporter export timeout: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 172.20.248.99:4317: connect: connection refused"

172.20.248.99 is the jaeger-collector Service. Jaeger's own startup log shows why:

otlpreceiver@v0.155.0/otlp.go:120  Starting GRPC server  ... "endpoint": "127.0.0.1:4317"
otlpreceiver@v0.155.0/otlp.go:175  Starting HTTP server  ... "endpoint": "127.0.0.1:4318"

curl http://localhost:16686/api/services returned only ["jaeger"]. Jaeger 2.x is built on the OpenTelemetry Collector, whose receivers default to localhost when no endpoint is set, so the configuration's bare grpc: and http: listened only inside the pod. The traces that specs 006 (T038, T039), 010 (T020) and the full profile depend on could never arrive.

Tasks

gitops specs/006-observability-platform-foundation T055 (added and delivered here).

The task register is updated in this pull request
Every task ticked here was verified by locating its artifact

How it is verified

microservice-app-ai-agents/scripts/check-mcp.sh gitops . passes (aws-knowledge 5 tools, microsoft-learn 3 tools). Neither server documents Jaeger or the OpenTelemetry Collector's receiver defaults, so the evidence is the running binary's own log above and its validate command.

The contract fails before the fix (a107491) and passes after it (4f02789):

$ tests/contract/observability.sh
FAIL: Jaeger's OTLP gRPC receiver must listen on every interface, not the 127.0.0.1 default (infrastructure/jaeger/config.yaml)
$ tests/contract/observability.sh
PASS: observability platform static contract (prometheus, grafana, jaeger, loki); registration asserted

Both rendered configurations pass jaeger validate with the pinned Jaeger 2.20.0 image (jaegertracing/jaeger@sha256:46a88626..., --network none). A negative control shows the command does reject a malformed receiver:

infrastructure/jaeger validate exit=0
infrastructure/profiles/full/jaeger/aws validate exit=0
negative control (grpc: endpont: 0.0.0.0:4317) exit=1
Error: failed to get config: cannot unmarshal the configuration ... 'receivers' error reading configuration for "otlp"

validate does not check the address itself (a bad port also exits 0), so the live proof is the next sync. The existing NetworkPolicy jaeger-allow-otlp-and-query already admits 4317 and 4318, and every business pod's allow-tracing-egress already targets it.

Infrastructure changes: the documentation consulted through the required MCP servers is listed above

Risk and rollback

Jaeger restarts once with the new configuration. The query UI and storage are unchanged. Rollback is a revert of the merge commit.

What this PR does not do

It does not change retention, storage, or the Service and NetworkPolicy, which were already correct.

Merge method: merge commit. This PR carries a Spec-Driven Development test/fix pair (docs #40).

Approval required: one maintainer approval (EstebanGZam or Juanmadiaz45).

🤖 Generated with Claude Code

…face Jaeger 2.20 bound its OTLP receivers to 127.0.0.1:4317 and :4318 on the live eks-dev cluster, and every service's exporter failed with dial tcp 172.20.248.99:4317: connect: connection refused, so Jaeger listed no service. The contract now requires an explicit 0.0.0.0 endpoint in the economical and full configurations and fails: FAIL: Jaeger's OTLP gRPC receiver must listen on every interface, not the 127.0.0.1 default (infrastructure/jaeger/config.yaml)

Both Jaeger configurations now declare 0.0.0.0:4317 and 0.0.0.0:4318 for the OTLP receiver instead of relying on the 127.0.0.1 default, so the jaeger-collector Service reaches it.

## What changes - The Jaeger pod template in `infrastructure/jaeger/jaeger-allinone.yaml` carries `microtodosuite.io/config-sha256: d6c1087f...`, and the full profile's `infrastructure/profiles/full/jaeger/components/elasticsearch/deployment.yaml` carries `60dc29a5...`. Each value is the sha256 of the `jaeger-config` data that root renders. - `tests/contract/observability.sh` renders `infrastructure/jaeger` and both full Jaeger roots, and requires the annotation to equal that hash. A configuration change that does not update the annotation fails CI. - spec 006 gains T056. ## Why #248 fixed Jaeger's OTLP listen address, and ArgoCD synced it: the live `jaeger-config` ConfigMap on `eks-dev` holds `0.0.0.0:4317`. But `jaeger-config` is a plain ConfigMap, so nothing restarted Jaeger, and the running pod (140 minutes old) still logs `"endpoint": "127.0.0.1:4317"`. The services' traces still cannot arrive. With the hash in the pod template, every configuration change is a pod-template change, and ArgoCD rolls Jaeger, including on this merge. ## Tasks gitops specs/006-observability-platform-foundation T056 (added and delivered here). - [x] The task register is updated in this pull request - [x] Every task ticked here was verified by locating its artifact ## How it is verified `microservice-app-ai-agents/scripts/check-mcp.sh gitops .` passes. No MCP server documents Kubernetes ConfigMap rollout semantics for Jaeger; the evidence is the live state quoted above. The contract fails before the fix and passes after it, locally with kustomize 5.8.1 and yq 4.53.6: ```text $ tests/contract/observability.sh FAIL: Jaeger's pod template must carry microtodosuite.io/config-sha256: d6c1087fe0bd59ee7e11148638c08adbf4c5bd54e74d009af808c0c85c617ad5, the sha256 of the jaeger-config it renders, so a config change rolls the pod (found '' in infrastructure/jaeger) $ tests/contract/observability.sh PASS: observability platform static contract (prometheus, grafana, jaeger, loki); registration asserted $ tests/platform/observability-full.bats PASS: full-profile observability roots (prometheus, grafana) keep encrypted volumes ... store Jaeger spans in Elasticsearch for 3 days ... ``` - [x] Infrastructure changes: the documentation consulted through the required MCP servers is listed above ## Risk and rollback Jaeger restarts once. Economical spans live on the PVC, so they survive. Rollback is a revert of the merge commit. ## What this PR does not do - It does not convert `jaeger-config` to a `configMapGenerator`. The full profile patches it by name from another root, and a hashed name would move that coupling. **Merge method**: merge commit. This PR carries a Spec-Driven Development test/fix pair (docs #40). **Approval required**: one maintainer approval (EstebanGZam or Juanmadiaz45). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

## What changes - The Jaeger Deployment in `infrastructure/jaeger/jaeger-allinone.yaml` carries `argocd.argoproj.io/sync-options: Replace=true`. - `tests/contract/observability.sh` requires that sync option. - spec 006 gains T058. ## Why #250 set `strategy: Recreate` so Jaeger can restart despite Badger's lock. ArgoCD synced it on `eks-dev`, and the API server rejected it: ```text Deployment.apps "jaeger" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'. Retrying attempt #2 $ kubectl -n observability get deploy jaeger -o jsonpath='{.spec.strategy}' {"rollingUpdate":{"maxSurge":"25%","maxUnavailable":"25%"},"type":"RollingUpdate"} ``` The Application syncs with `ServerSideApply=true`. A server-side apply merges the manifest into the live object, which keeps the `rollingUpdate` defaults the API server filled in when the Deployment was created, and those are invalid next to `Recreate`. Replacing the object drops them. Jaeger therefore still runs the old pod listening on `127.0.0.1:4317`, beside a new pod that crash-loops on the Badger lock. ## Tasks gitops specs/006-observability-platform-foundation T058 (added and delivered here). - [x] The task register is updated in this pull request - [x] Every task ticked here was verified by locating its artifact ## How it is verified `microservice-app-ai-agents/scripts/check-mcp.sh gitops .` passes. No MCP server covers Argo CD, so the source is Argo CD's own documentation at the vendored version, `docs/user-guide/sync-options.md` at `v3.5.0` (read with `gh api`). The section "Replace Resource Instead Of Applying Changes" says `Replace=true` "can also be configured at individual resource level" with this exact annotation, and line 358 says "`Replace=true` takes precedence over `ServerSideApply=true`." ```text $ tests/contract/observability.sh FAIL: Jaeger's Deployment must carry argocd.argoproj.io/sync-options Replace=true, so a server-side sync can move it off RollingUpdate (found '' in infrastructure/jaeger) $ tests/contract/observability.sh PASS: observability platform static contract (prometheus, grafana, jaeger, loki); registration asserted $ tests/platform/observability-full.bats PASS: full-profile observability roots ... ``` - [x] Infrastructure changes: the documentation consulted through the required MCP servers is listed above ## Risk and rollback ArgoCD replaces the Deployment object once. Under `Recreate`, the old pod stops and releases the Badger lock, and the new pod starts with the `0.0.0.0` OTLP endpoints from #248. The spans on the PVC are kept. The documentation warns that `Replace` can recreate resources; for a single-replica Deployment on a persistent volume that means the same short gap `Recreate` already accepts. Rollback is a revert of the merge commit. ## What this PR does not do - It does not change the Application-wide sync options; only this Deployment is replaced. **Merge method**: merge commit. This PR carries a Spec-Driven Development test/fix pair (docs #40). **Approval required**: one maintainer approval (EstebanGZam or Juanmadiaz45). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Tiago0507 added 3 commits September 28, 2026 18:02

fix(observability): receive otlp on every interface in jaeger

4f02789

Both Jaeger configurations now declare 0.0.0.0:4317 and 0.0.0.0:4318 for the OTLP receiver instead of relying on the 127.0.0.1 default, so the jaeger-collector Service reaches it.

docs(specs): record the jaeger otlp listen address as delivered

776d42f

Tiago0507 requested review from EstebanGZam and Juanmadiaz45 September 28, 2026 23:05

Juanmadiaz45 approved these changes Sep 28, 2026

View reviewed changes

Tiago0507 merged commit 2bd89e4 into main Sep 28, 2026
4 checks passed

Tiago0507 deleted the fix/jaeger-otlp-listen-address branch September 28, 2026 23:11

Tiago0507 mentioned this pull request Sep 28, 2026

fix(observability): roll jaeger when its configuration changes #249

Merged

3 tasks done

Tiago0507 mentioned this pull request Sep 28, 2026

fix(observability): let argocd replace the jaeger deployment #251

Merged

3 tasks done

Navigation Menu

Repository navigation

fix(observability): receive otlp on every interface in jaeger - #248

fix(observability): receive otlp on every interface in jaeger#248
Tiago0507 merged 3 commits into
mainfrom
fix/jaeger-otlp-listen-address

Tiago0507 commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Tiago0507 commented Sep 28, 2026

What changes

Why

Tasks

How it is verified

Risk and rollback

What this PR does not do

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants