Skip to content

observability: dispatch latency and source reconnects #243

Description

@mfw78

Renames the metrics that still say event and subscription, last in the program, in one step with no dual-emit.

#239 has landed.
This lands after #241.

Renames

Old New Surface
nexum_runtime_event_latency_seconds nexum_runtime_dispatch_latency_seconds metric name, metrics.rs
nexum_runtime_stream_reconnects_total nexum_runtime_source_reconnects_total metric name, metrics.rs
label kind on the reconnect counter source_kind metric label, runtime/event_loop.rs, two sites
help "Chain subscription reconnects by chain." "Chain source reconnects by chain." metric help, metrics.rs
help "Events dropped before dispatch, by reason." "Triggers dropped before dispatch, by reason." metric help, metrics.rs

All of it is operator-visible and all of it breaks in one deploy.

Why there is no dual-emit window

An earlier version of this issue shipped both names for a release so operators could move dashboards, then deleted the old ones in a follow-up.

That window protects nobody. Nothing is published, there is no tag, no operator is running this, and no dashboard or alert rule exists outside docs/production.md, which this issue updates in the same commit. A migration window with no one to migrate is machinery that has to be built, documented, remembered and then removed.

So both names rename once, like the label, and the follow-up deletion issue is folded in here rather than filed.

The reasoning that made labels rename-once applies to the names too, once there is nothing to protect. A Prometheus series is its full label set including __name__, so a rename ends the old series at its cumulative value and starts a new one at zero. rate() and increase() lose the delta across the deploy. Dual-emitting only defers that cost in exchange for carrying two names; with no consumer, it is cost without the exchange.

Note the deploy-boundary reset in the changelog anyway. It is true even with no dashboards, and it is the kind of thing a reader of the release notes wants stated rather than discovered.

Why the reconnect metric is in the list

nexum_runtime_source_reconnects_total is here because #241 renames this layer to SOURCE.

Renaming only the label leaves a metric whose name says stream and whose label says source, which is the disagreement this program exists to remove, manufactured rather than inherited.

Already landed

The event_kind dispatch label is not in the table because it renamed to trigger_kind with the guest-ABI rename, at every emit site and in docs/production.md.

It was pulled forward because that same change renamed the chain-log trigger to event, which would have produced event_kind="event" and asserted that "event" is both the genus and one of its species. ADR-0019 already required the label rename, so it landed there rather than breaking the same series twice.

Divergences the source rename left for this issue to close

The source rename froze this metric and its labels, which was right, but it left three surfaces disagreeing until this lands. Each must be resolved here.

The tracing kind field and the metric kind label now disagree for one reconnect task. NextTrigger::StreamPanic carries "event", so the "reconnect task ended unexpectedly" warn emits kind="event", while the counter for the same task emits kind="chain-log". An operator correlating a log line to a Prometheus series sees two values for one source. The block path agrees in both, so only the event path acquired the mismatch.

Resolve it by deciding what the label VALUE should be, which the earlier "does not change" note did not settle. The value chain-log was justified as naming the RPC stream rather than a trigger, and that reasoning holds for the transport. But the tracing field on the same task now says event, so pick one and make both surfaces carry it.

Two operator warnings in supervisor/cursors.rs still read "failed to persist chain-log cursor" and "failed to open module store for chain-log cursor". These are free prose in a warn!, not key strings, and they interleave with reconnect lines that now all read "event source". The store key chainlog_cursor:<keccak> is exempt and stays; the prose does not have to.

The reconnect metric's help text reads "Source stream reconnects by kind and chain." but the event-path emission also carries a module label, so the help under-describes the label set. Fix the help to match what is emitted.

Does not change

nexum_runtime_dispatch_dropped_total.
It already names the right concept.

The label VALUES on the reconnect counter.
It carries kind = "chain-log", and that value names the RPC stream rather than a trigger.
The transport layer deliberately still spells logs, because eth_getLogs returns logs.
Renaming the value to event would be the same category error in reverse.
The label name becomes source_kind; its values stay block and chain-log.

Must land with it

The METRICS table and every emit site for each renamed name.
nexum_runtime_event_latency_seconds has one emit site, in supervisor/dispatch.rs.
nexum_runtime_stream_reconnects_total has two, both in runtime/event_loop.rs.
The test every_emitted_name_is_in_the_table_and_every_entry_is_emitted in metrics.rs asserts the table and the emitted set match in both directions, so it holds you to the table rows.

docs/production.md, the metric table in section 6 and all six alert rules in section 7, in the same commit as the emit change.

Coverage warning

Metric is { name, kind, help } in metrics.rs and the test scans for "nexum_runtime_ string literals only.
Labels are not in the table and not in the test, so the pin holds you to nothing for the label row.
Two alert rules touch a renamed name: NexumDispatchLatency reads nexum_runtime_event_latency_seconds_bucket, and NexumReconnectStorm reads nexum_runtime_stream_reconnects_total.
Two rows of the section 6 metric table name the same two metrics, and the reconnect row also documents the kind label and its values in prose.
Review the label row and the docs prose by reading.

Done when

Both metrics emit under their new names only, with no old name anywhere in the tree.

The bare kind label on the reconnect counter is gone.

Every alert rule in docs/production.md section 7 references a series that exists, verified by querying it and not by reading it.

No event or subscription survives in a metric name, label or help string.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:observabilityMetrics, gauges, health endpoints, structured loggingdebtRefactor/cleanup: typed replacements for stringly code, dedup, right-sizing

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions