Summary
The current OTLP instrumentation shows queue depths and ingest rates, but it does not expose the overload signals we actually needed during the OpenSky replay benchmark.
During the 2026-04-08 benchmark run:
- the
10 Hz replay expanded a five-minute Germany capture to 1,581,177 replayed updates
- the runtime started dropping work on the decision queue long before host CPU was saturated
- queue drops were visible in logs but not as first-class metrics, which made it harder to dashboard overload behavior directly in SigNoz
Missing telemetry
Add explicit OTLP metrics for:
- native queue dropped item counter
- decision queue dropped item counter
- event-bus outbound dropped item counter if applicable
- websocket outbound dropped item counter if applicable
- queue enqueue/dequeue counters by stage
- queue drain rate by stage
- stage worker concurrency / active worker count where measurable
- collision evaluation count and fence evaluation count as separate counters from the generic processing timings
- per-stage success, drop, and skip outcomes so dashboards can show where overload is happening without reading logs
Acceptance criteria
- queue drops are exported as monotonic counters with low-cardinality stage labels
- default SigNoz dashboards include overload panels for drop rate and cumulative drop count
- the local benchmark workflow can assert that selected runs remain drop-free without parsing container logs
- docs are updated with the new metrics and dashboard intent
Context
This came out of the OpenSky Germany replay benchmark that now lives in connectors/replay/benchmarks/opensky-germany-2026-04-08/.
Summary
The current OTLP instrumentation shows queue depths and ingest rates, but it does not expose the overload signals we actually needed during the OpenSky replay benchmark.
During the
2026-04-08benchmark run:10 Hzreplay expanded a five-minute Germany capture to1,581,177replayed updatesMissing telemetry
Add explicit OTLP metrics for:
Acceptance criteria
Context
This came out of the OpenSky Germany replay benchmark that now lives in
connectors/replay/benchmarks/opensky-germany-2026-04-08/.