tracing, metric, log, otel: complete native OpenTelemetry support for traces, metrics and logs

[applike-ss]

Summary

Completes the switch to native OpenTelemetry in gosoline for traces, metrics, and logs, emitting into an OTEL Collector → gateway → ClickHouse (ClickStack) pipeline. Everything is configurable through the native gosoline cfg system. Transport is OTLP/gRPC by default with TLS/mTLS support; the existing Prometheus pull path is left unchanged.

Relates to justtrack/devops/tickets#961 (native OTEL ingestion) and #962 (OTEL metric naming).

What's included

Shared core — new pkg/otel

• BuildResource derives the OTEL resource (service.name, service.namespace, deployment.environment, plus configurable attributes) from the app identity, so all three signals share identical resource attributes for correlation.
• BuildTraceExporter / BuildMetricExporter / BuildLogExporter: OTLP gRPC/HTTP exporter builders with gzip, headers, retry, and TLS/mTLS.
• ExporterSettings splits host/port (with optional full endpoint override) so the node IP can be injected from pod metadata (status.hostIP); Address() composes via net.JoinHostPort.

Traces (pkg/tracing)

• Added otel_grpc and stdout span exporters; the exporter registry now returns sdktrace.SpanExporter.
• Trace provider uses the shared resource and supports configurable propagators (tracecontext, baggage).
• Legacy otel_http (config under tracing.otel.http.*) kept for back-compat.

Metrics (pkg/metric) — bridge, not rewrite

• New otel writer (WriterTypeOtel) bridges the existing Datum pipeline to an OTEL MeterProvider + OTLP exporter via a PeriodicReader. Kind→instrument (counter/gauge/histogram) and CloudWatch unit→UCUM mapping. Identity goes to resource attributes; Dimensions become metric attributes.
• Runs alongside cloudwatch/prometheus writers (operator picks via metric.writers). Existing Prometheus writer/pull endpoint unchanged.

Metric naming (#962)

• FormatOtelMetricName enforces lowercase, dot-delimited, snake_case-within-segment names; ToUcumUnit maps units. Identity and units are not part of the name (units live on the instrument; identity in resource attributes).

Logs (pkg/log)

• New otel log handler bridges to the OTEL Logs SDK (BatchProcessor + OTLP exporter). Level→severity and field→attribute mapping; trace/span context is attached from ctx for trace↔log correlation.

Docs & example

• pkg/otel/AGENTS.md, updated pkg/tracing/AGENTS.md, and examples/otel/config.dist.yml showing all three signals.

Resilience notes

OTLP exporters use background processors with bounded queues — the hot path only enqueues and never blocks on network I/O. On collector outage they retry/backoff then drop (lost telemetry, never app instability). Recommended deployment: push to a node-local collector agent that buffers/retries to the gateway.

Testing

• go build ./... clean.
• go vet clean on changed packages.
• go test ./pkg/otel/ ./pkg/metric/ ./pkg/tracing/ ./pkg/log/ pass.
• New unit tests: metric naming + UCUM mapping, exporter Address() (host/port/endpoint, IPv6), resource builder (identity → attributes).
• golangci-lint clean on all new code (2 remaining findings are pre-existing in untouched files).

Follow-ups / not in scope

• Collector/gateway/ClickHouse config (already set up, owned by infra).
• Exemplars for metric→trace drill-down (resource-attribute + time correlation for now).
• Explicit shutdown-flush hook (currently relies on PeriodicReader / BatchProcessor intervals).

🤖 Draft — opened for review.