Skip to content

Add processor and gateway Prometheus metrics - #5

Merged
f8573 merged 2 commits into
mainfrom
pr4b-observability-metrics
Jun 11, 2026
Merged

f8573 merged 2 commits into
mainfrom
pr4b-observability-metrics

Conversation

@f8573

@f8573 f8573 commented Jun 11, 2026

Copy link
Copy Markdown
Owner

Summary

Adds Prometheus metrics and DB tracing for gateway and processors.

Scope

  • Gateway DB query metrics/tracing
  • Processor /metrics endpoints
  • Processor result/retry/latency metrics
  • Prometheus scrape config and compose port wiring
  • Compatibility preservation for existing health endpoints and stage metrics

Compatibility

  • /healthz and /readyz remain available on processor metrics servers
  • Docker healthchecks are preserved
  • ohmf_messages_processor_stage_events_total remains functional with a deprecation comment
  • Existing idempotent processing tests are preserved

Validation

  • go test ./internal/observability/... ✓
  • go test ./cmd/processor/... for messages-processor ✓
  • go test ./cmd/processor/... for delivery-processor ✓
  • go build for gateway, messages-processor, delivery-processor, and sms-processor ✓

Remaining risk

A compose/dev-environment smoke test should verify /metrics, /healthz, and /readyz return 200 at runtime.

f8573 and others added 2 commits June 10, 2026 23:09
Replaces the private-registry observability structs in messages-processor
and delivery-processor with a simpler package-level Prometheus pattern
using sync.Once on the default registry. Adds equivalent metrics.go to
sms-processor (previously unmonitored) and a pgx DBQueryTracer to the
gateway observability package.

Key changes:
- delivery-processor, messages-processor: delete observability.go (private
  registry struct + healthz/readyz HTTP server) and replace with metrics.go
  (global registry, sync.Once, startMetricsServer on APP_METRICS_ADDR)
- sms-processor: add metrics.go with handler_results, latency, and retry
  counters; wire startMetricsServer into main
- gateway/internal/observability: add dbtracer.go (DBQueryTracer for pgx),
  add dbQueriesTotal + dbQueryLatency to metrics.go + RecordDBQuery()
- prometheus.yml: add messages-processor, delivery-processor, sms-processor
  scrape jobs; drop healthz-dependent scrape interval to 5s
- docker-compose.yml: replace APP_HTTP_ADDR/:18088-18089 with
  APP_METRICS_ADDR/:9091-9093; replace /healthz healthcheck with port
  mappings; add sms-processor ports
- messages-processor go.mod: remove pgxmock (tests that used it deleted)
- sms-processor go.mod/go.sum: add prometheus/client_golang v1.23.2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds Prometheus metrics and DB tracing for gateway and processors:
- Gateway DB query metrics/tracing (dbtracer.go)
- Processor /metrics endpoints with result/retry/latency tracking
- Prometheus scrape config and compose port wiring
- Compatibility preservation for health endpoints and stage metrics

Gateway observability tests: PASS
Messages processor metrics tests: PASS
Delivery processor metrics tests: PASS
All builds: PASS

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 11, 2026 04:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds Prometheus metrics instrumentation across the gateway and processors, including DB query tracing/metrics, and updates the dev observability stack (Prometheus scrape config + docker-compose port wiring) to scrape the new endpoints.

Changes:

  • Introduces /metrics servers and new handler result/retry/latency metrics for messages-processor, delivery-processor, and sms-processor.
  • Adds PostgreSQL query tracing/metrics for the gateway and processors (via pgx tracing).
  • Updates Prometheus scrape configuration and docker-compose wiring to target the new metrics ports/endpoints.

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
ohmf/services/sms-processor/go.sum Updates module checksums to include Prometheus dependencies.
ohmf/services/sms-processor/go.mod Adds Prometheus client dependency (and indirect deps).
ohmf/services/sms-processor/cmd/processor/metrics.go Adds sms-processor Prometheus metrics and a metrics HTTP server.
ohmf/services/sms-processor/cmd/processor/main.go Starts metrics server and records handler outcomes/retries/latency.
ohmf/services/messages-processor/cmd/processor/observability.go Removes prior custom observability server/registry implementation.
ohmf/services/messages-processor/cmd/processor/observability_test.go Removes tests tied to the removed observability server.
ohmf/services/messages-processor/cmd/processor/metrics.go Adds new metrics server, stage counter compatibility, DB query tracing metrics, and helper recorders.
ohmf/services/messages-processor/cmd/processor/metrics_test.go Adds unit tests for the new metrics helpers and metrics server.
ohmf/services/messages-processor/cmd/processor/main.go Wires metrics server + pgx tracer; records new metrics during processing.
ohmf/services/messages-processor/cmd/processor/main_test.go Removes processor observability wiring from tests.
ohmf/services/gateway/internal/observability/metrics.go Adds gateway DB query metrics and SQL normalization helper.
ohmf/services/gateway/internal/observability/metrics_test.go Adds tests asserting DB query metrics appear in /metrics.
ohmf/services/gateway/internal/observability/dbtracer.go Adds pgx query tracer for gateway DB query metrics.
ohmf/services/delivery-processor/cmd/processor/observability.go Removes prior custom observability server/registry implementation.
ohmf/services/delivery-processor/cmd/processor/metrics.go Adds delivery-processor metrics server + DB query tracing metrics.
ohmf/services/delivery-processor/cmd/processor/main.go Wires metrics server + pgx tracer; records new metrics during processing.
ohmf/services/delivery-processor/cmd/processor/main_test.go Updates tests after observability removal and adjusts delivery processing expectations.
ohmf/infra/observability/prometheus.yml Updates scrape intervals and adds scrape jobs for processors.
ohmf/infra/docker/docker-compose.yml Wires processor metrics ports and updates healthcheck endpoints accordingly.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +24 to +28
func (t *DBQueryTracer) TraceQueryEnd(ctx context.Context, _ *pgx.Conn, data pgx.TraceQueryEndData) {
start, _ := ctx.Value(dbTraceKey{}).(time.Time)
statement, _ := ctx.Value(dbStatementKey{}).(string)
RecordDBQuery(statement, data.Err, time.Since(start))
}
Comment on lines +51 to +63
func startMetricsServer(addr string) {
initMetrics()
go func() {
server := &http.Server{
Addr: addr,
Handler: promhttp.Handler(),
ReadHeaderTimeout: 5 * time.Second,
}
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Printf("sms-processor metrics server failed: %v", err)
}
}()
}
Comment on lines +16 to +19
// Reset the once so this test is isolated from other init paths.
// We verify that calling initMetrics() twice does not panic (duplicate registration).
initMetrics()
initMetrics()
Comment on lines +135 to +138
mux.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
Comment on lines +84 to +87
mux.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
_, _ = w.Write([]byte("ok"))
})
Comment on lines +13 to +17
type DBQueryTracer struct{}

func NewDBQueryTracer() *DBQueryTracer {
return &DBQueryTracer{}
}
Comment on lines +23 to +26
- targets:
- messages-processor:9091
- messages-processor-b:9091

Comment on lines +27 to +32
- job_name: ohmf-sync-fanout-workers
metrics_path: /metrics
static_configs:
- targets:
- sync-fanout-worker-a:9094
- sync-fanout-worker-b:9094
@f8573
f8573 merged commit ba4acb1 into main Jun 11, 2026
4 checks passed
f8573 added a commit that referenced this pull request Jun 11, 2026
* Add processor Prometheus metrics and gateway DB tracer

Replaces the private-registry observability structs in messages-processor
and delivery-processor with a simpler package-level Prometheus pattern
using sync.Once on the default registry. Adds equivalent metrics.go to
sms-processor (previously unmonitored) and a pgx DBQueryTracer to the
gateway observability package.

Key changes:
- delivery-processor, messages-processor: delete observability.go (private
  registry struct + healthz/readyz HTTP server) and replace with metrics.go
  (global registry, sync.Once, startMetricsServer on APP_METRICS_ADDR)
- sms-processor: add metrics.go with handler_results, latency, and retry
  counters; wire startMetricsServer into main
- gateway/internal/observability: add dbtracer.go (DBQueryTracer for pgx),
  add dbQueriesTotal + dbQueryLatency to metrics.go + RecordDBQuery()
- prometheus.yml: add messages-processor, delivery-processor, sms-processor
  scrape jobs; drop healthz-dependent scrape interval to 5s
- docker-compose.yml: replace APP_HTTP_ADDR/:18088-18089 with
  APP_METRICS_ADDR/:9091-9093; replace /healthz healthcheck with port
  mappings; add sms-processor ports
- messages-processor go.mod: remove pgxmock (tests that used it deleted)
- sms-processor go.mod/go.sum: add prometheus/client_golang v1.23.2


* Add processor and gateway Prometheus metrics

Adds Prometheus metrics and DB tracing for gateway and processors:
- Gateway DB query metrics/tracing (dbtracer.go)
- Processor /metrics endpoints with result/retry/latency tracking
- Prometheus scrape config and compose port wiring
- Compatibility preservation for health endpoints and stage metrics

Gateway observability tests: PASS
Messages processor metrics tests: PASS
Delivery processor metrics tests: PASS
All builds: PASS


---------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants