Skip to content

fix: restore the heartbeats_heartbeat_received_total counter - #243

Merged
gi8lino merged 2 commits into
containeroo:masterfrom
cplieger:fix/heartbeat-received-metric
Jul 23, 2026
Merged

fix: restore the heartbeats_heartbeat_received_total counter#243
gi8lino merged 2 commits into
containeroo:masterfrom
cplieger:fix/heartbeat-received-metric

Conversation

@cplieger

@cplieger cplieger commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

heartbeats_heartbeat_received_total is registered, documented in the README, and covered by registry tests, but nothing increments it anymore: since v0.22.0 no production series ever appears on /metrics, no matter how many bumps arrive.

Regression trace

  • v0.21.0 and earlier incremented it in the bump path, right next to the status gauge: internal/heartbeat/actor.go line 150 at 1f2b091 (v0.20.0); the same call is still present at the v0.21.0 tag.
  • refactor app #147 (released as v0.22.0) decomposed the actor into manager/runner/service. The new receive path kept the history recording but the IncHeartbeatReceived call fell out; the state gauge survived because it lives in the sender.
  • Since then the only callers of IncHeartbeatReceived are its own tests. Nothing ever touches a child of the labeled CounterVec (no WithLabelValues on any code path), so no series exists and the metric silently vanishes from the exposition. Confirmed live on v0.23.0: bumps accepted ({"status":"ok"}, stage_transition logged) with no received_total series ever appearing.

The fix

The increment lives in service.Update, the single place that sees every bump and can tell the three outcomes apart:

  • Accepted bump: counted.
  • Mailbox-full bump: counted too. State.UpdateSeen has already recorded last-seen and payload before the mailbox send is attempted, and history logs the event as received (enqueued: false), so the delivery genuinely arrived; only the runner wake-up was dropped.
  • Unknown id: not counted, deliberately. The id becomes a metric label, and counting arbitrary request paths would let callers mint unbounded label cardinality.

Plumbing: NewService takes the metrics registry (same pattern as manager.NewManager), run.go passes the existing metricsReg, and the increment is nil-guarded so callers without metrics keep working. The HTTP handler is untouched apart from what was already there.

Tests:

  • internal/heartbeat/service: accepted + mailbox-full render heartbeats_heartbeat_received_total{heartbeat="api"} 2 on the registry's exposition; an unknown id creates no series; a nil-metrics service doesn't panic.
  • internal/handler (heartbeat_test.go was an empty placeholder): accepted bump reaches the service and returns 200, a service error maps to 404, a missing id is a 400 that never reaches the service.

Verification

  • gofmt, go vet, and go test pass for ./internal/handler, ./internal/heartbeat/service, and ./internal/metrics (frontend built first for the web/dist embed).
  • A full go test ./... currently fails on master in the packages importing internal/notify; that's the pre-existing notifykit build break tracked in master does not compile; the PR test gate never runs #242, unrelated to this change. I baseline-checked the identical failures occur without this diff.

One adjacent thing I noticed but didn't touch: the README's metrics list says heartbeats_heartbeat_last_status while the code registers heartbeats_heartbeat_last_state.

Checklist

  • The PR has a meaningful title (changelog-ready).
  • PR contains a single logical change.
  • Documentation: the README already documents this metric; the change makes the code match it.
  • Labels: I can't set labels on this repo; bug fits.
  • Linked related issues/PRs: refactor app #147 (where the regression landed), master does not compile; the PR test gate never runs #242 (why full-tree tests can't run on master right now).

@cplieger cplieger changed the title fix: increment heartbeats_heartbeat_received_total on accepted bumps fix: restore the heartbeats_heartbeat_received_total counter Jul 23, 2026
@gi8lino

gi8lino commented Jul 23, 2026

Copy link
Copy Markdown
Member

hi @cplieger

thank you for your contribution!

Regards!

@gi8lino
gi8lino merged commit 8e38e14 into containeroo:master Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants