Give the reconnect log lines the field an operator needs to correlate them with the counter that alerted.
The gap
NexumReconnectStorm fires on nexum_runtime_source_reconnects_total, whose series carry source_kind = "chain-log" for the event path and "block" for the block path.
An operator seeing that alert greps the logs for the value it names. On the event path they find only the terminal reconnect task ended unexpectedly warn, because that is the one line carrying a kind field. Every individual reconnect is logged by info!(module, chain_id, attempt, start_block, "event source reopened") in runtime/event_loop.rs, immediately above the counter increment, and it carries no kind field at all.
So the storm's occurrences are invisible to the obvious search, and the one line that does match uses prose ("event source") that does not contain the value the alert reported.
The same shape elsewhere on the path
event source open, event source backfilling a large gap and event source gap exceeds max_lookback in runtime/event_loop.rs, and the two cursor warns in supervisor/cursors.rs that read event source cursor. None carries the kind.
Scope
Add the kind field to every log line on the source path that pairs with a counter increment, spelled and valued exactly as the metric label is, so one value finds both surfaces.
Decide whether the prose should carry the value too. A line reading "event source reopened" beside a field saying chain-log is legible if the field is there, and a reader greps the field; that is a judgement call rather than an obvious yes.
Why it is not the metric rename's job
The metric rename made chain-log canonical across the metric label and the tracing field on the panic path, which is what it was scoped to do. This is the reopen path, which has no field to make agree, so it needs a field added rather than a value aligned. That is new work, not a missed rename, and folding it in would have widened a rename into an observability change.
Done when
An operator can take the value from an alert, grep it, and find every reconnect it counted.
Give the reconnect log lines the field an operator needs to correlate them with the counter that alerted.
The gap
NexumReconnectStormfires onnexum_runtime_source_reconnects_total, whose series carrysource_kind = "chain-log"for the event path and"block"for the block path.An operator seeing that alert greps the logs for the value it names. On the event path they find only the terminal
reconnect task ended unexpectedlywarn, because that is the one line carrying akindfield. Every individual reconnect is logged byinfo!(module, chain_id, attempt, start_block, "event source reopened")inruntime/event_loop.rs, immediately above the counter increment, and it carries no kind field at all.So the storm's occurrences are invisible to the obvious search, and the one line that does match uses prose ("event source") that does not contain the value the alert reported.
The same shape elsewhere on the path
event source open,event source backfilling a large gapandevent source gap exceeds max_lookbackinruntime/event_loop.rs, and the two cursor warns insupervisor/cursors.rsthat readevent source cursor. None carries the kind.Scope
Add the kind field to every log line on the source path that pairs with a counter increment, spelled and valued exactly as the metric label is, so one value finds both surfaces.
Decide whether the prose should carry the value too. A line reading "event source reopened" beside a field saying
chain-logis legible if the field is there, and a reader greps the field; that is a judgement call rather than an obvious yes.Why it is not the metric rename's job
The metric rename made
chain-logcanonical across the metric label and the tracing field on the panic path, which is what it was scoped to do. This is the reopen path, which has no field to make agree, so it needs a field added rather than a value aligned. That is new work, not a missed rename, and folding it in would have widened a rename into an observability change.Done when
An operator can take the value from an alert, grep it, and find every reconnect it counted.