Skip to content

Export queue drop and drain metrics for replay and overload analysis #42

Description

@jillesvangurp

Summary

The current OTLP instrumentation shows queue depths and ingest rates, but it does not expose the overload signals we actually needed during the OpenSky replay benchmark.

During the 2026-04-08 benchmark run:

  • the 10 Hz replay expanded a five-minute Germany capture to 1,581,177 replayed updates
  • the runtime started dropping work on the decision queue long before host CPU was saturated
  • queue drops were visible in logs but not as first-class metrics, which made it harder to dashboard overload behavior directly in SigNoz

Missing telemetry

Add explicit OTLP metrics for:

  • native queue dropped item counter
  • decision queue dropped item counter
  • event-bus outbound dropped item counter if applicable
  • websocket outbound dropped item counter if applicable
  • queue enqueue/dequeue counters by stage
  • queue drain rate by stage
  • stage worker concurrency / active worker count where measurable
  • collision evaluation count and fence evaluation count as separate counters from the generic processing timings
  • per-stage success, drop, and skip outcomes so dashboards can show where overload is happening without reading logs

Acceptance criteria

  • queue drops are exported as monotonic counters with low-cardinality stage labels
  • default SigNoz dashboards include overload panels for drop rate and cumulative drop count
  • the local benchmark workflow can assert that selected runs remain drop-free without parsing container logs
  • docs are updated with the new metrics and dashboard intent

Context

This came out of the OpenSky Germany replay benchmark that now lives in connectors/replay/benchmarks/opensky-germany-2026-04-08/.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions