Skip to content

Ops metrics: /metrics + reconnect + rate-limit counters - #50

Merged
cvsloane merged 6 commits into
mainfrom
feat/ops-metrics
Feb 16, 2026
Merged

cvsloane merged 6 commits into
mainfrom
feat/ops-metrics

Conversation

@cvsloane

@cvsloane cvsloane commented Feb 16, 2026 •

Copy link
Copy Markdown
Owner

Adds Prometheus /metrics endpoints and minimal operational counters.

  • control-plane: GET /metrics (optional METRICS_TOKEN), Clawdbot allow/deny counters by reason
  • agentd: GET /metrics on hooks listener (providers.claude.hooks_http_listen), reconnect/backoff counters
  • Docs: reliability primitives inventory + ops monitoring note

Tests/build:

  • agents/agentd: go test ./...
  • services/control-plane: pnpm --filter @agent-command/control-plane build + typecheck + lint

Summary by CodeRabbit

  • New Features

    • Prometheus metrics endpoint at GET /metrics (agent and control-plane) exposing WebSocket state, reconnect attempts/backoff, and a new decision counter for notification gating. METRICS_TOKEN optional bearer auth supported.
    • Metrics emitted for WebSocket lifecycle (connect/reconnect success/failure, reconnect attempts).
  • Documentation

    • New Reliability Primitives guide and updated operations/agentd docs describing metrics and monitoring.
  • Chores

    • Go toolchain bumped to 1.23.0 and Prometheus client dependencies added.

@coderabbitai

coderabbitai Bot commented Feb 16, 2026 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@cvsloane has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 12 minutes and 4 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📝 Walkthrough

Walkthrough

Adds Prometheus metrics across agentd and the control plane: new agentd metrics package, WebSocket instrumentation, control-plane metrics module and routes with optional token auth, clawdbot decision telemetry and typed decision reasons, plus related docs and dependency updates.

Changes

Cohort / File(s) Summary
Dependency & Toolchain
agents/agentd/go.mod, services/control-plane/package.json
Bump Go toolchain (1.22 → 1.23.0) and expand indirect Go module pins; add prom-client dependency to control-plane Node deps.
Agentd: Metrics Package
agents/agentd/internal/metrics/metrics.go
New Prometheus metrics package: registry, collectors, HTTP Handler, and exported functions to set/connect/reconnect counters and gauges.
Agentd: HTTP Wiring
agents/agentd/internal/providers/claude.go
Register metrics.Handler() on the Claude provider HTTP mux (adds GET /metrics endpoint).
Agentd: WebSocket Instrumentation
agents/agentd/internal/ws/client.go
Instrument WebSocket lifecycle with metrics calls for connect/disconnect, reconnect attempts, successes, and failures.
Control Plane: Metrics Core
services/control-plane/src/metrics.ts
New prom-client registry, default metrics, and clawdbot decision counter with exported types and record helper.
Control Plane: Routes & Config
services/control-plane/src/routes/metrics.ts, services/control-plane/src/index.ts, services/control-plane/src/config.ts, services/control-plane/.env.example
Add GET /metrics route with optional Bearer or x-metrics-token auth, register routes in index, add METRICS_TOKEN config and example env var.
Control Plane: Clawdbot Logic
services/control-plane/src/services/clawdbot.ts
Refactor shouldNotify to return structured NotifyDecision (allowed or blocked with coded reason), add dedupe/cooldown checks, and record decisions via metrics helper.
Docs & Ops
docs/reliability-primitives.md, docs/agentd.md, docs/operations.md, docs/README.md, project_status.md
Add reliability-primitives doc; document GET /metrics exposure and scraping/auth guidance; link docs and update project status.

Sequence Diagram(s)

sequenceDiagram
    participant WS as WebSocket Client
    participant Metrics as Agentd Metrics
    participant Registry as Prometheus Registry
    participant HTTP as Agentd HTTP Handler
    participant Scraper as Prometheus Scraper

    WS->>Metrics: SetWSConnected(true)
    Metrics->>Registry: update connected gauge
    WS->>Metrics: RecordWSReconnectAttempt(delayMs)
    Metrics->>Registry: increment reconnect attempts / observe delay
    WS->>Metrics: RecordWSReconnectSuccess()
    Metrics->>Registry: increment success counter
    WS->>Metrics: SetWSConnected(false)
    Metrics->>Registry: update connected gauge

    Scraper->>HTTP: GET /metrics
    HTTP->>Registry: registry.metrics()
    Registry-->>HTTP: formatted metrics
    HTTP-->>Scraper: 200 text/plain (Prometheus format)
Loading
sequenceDiagram
    participant Queue as Notification Queue
    participant Claw as ClawdbotNotifier
    participant Metrics as Control-plane Metrics
    participant Counter as Decision Counter

    Queue->>Claw: queueNotification(event)
    Claw->>Claw: shouldNotify() (dedupe, cooldown, rate-limit)
    alt blocked
        Claw->>Metrics: recordClawdbotNotificationDecision(blocked, reason, eventType, provider)
        Metrics->>Counter: increment decision=blocked, reason=...
        Counter-->>Metrics: updated
        Metrics-->>Claw: ack
    else allowed
        Claw->>Metrics: recordClawdbotNotificationDecision(allowed, -, eventType, provider)
        Metrics->>Counter: increment decision=allowed
        Counter-->>Metrics: updated
        Metrics-->>Claw: ack
    end
    Claw-->>Queue: return decision
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Poem

🐰 I twitch my whiskers, counters hum,

Reconnects counted, metrics come,
Decisions tallied, reasons shown,
Endpoints standing, seeds are sown,
Observatory carrots, neatly grown. 🥕✨

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and concisely summarizes the main change: adding operational metrics endpoints (/metrics) with reconnect and rate-limit counters across control-plane and agentd services.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feat/ops-metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@agents/agentd/internal/metrics/metrics.go`:
- Around line 47-60: The initOnce function currently calls
prometheus.MustRegister with collectors.NewGoCollector() and
collectors.NewProcessCollector(...), which will panic due to duplicate
registration with the default registry; fix by removing those two default
collectors from the prometheus.MustRegister call and only register your custom
metrics (wsConnected, wsReconnecting, wsReconnectAttemptsTotal,
wsReconnectFailuresTotal, wsReconnectSuccessTotal, wsReconnectBackoffSeconds),
or alternatively create and use a dedicated prometheus.NewRegistry() and
register all collectors (including NewGoCollector and NewProcessCollector) on
that registry instead of the global prometheus registry; update initOnce
accordingly to use one of these two approaches and ensure references to the
metric variables (wsConnected, wsReconnecting, etc.) remain registered on the
chosen registry.
🧹 Nitpick comments (4)
docs/operations.md (1)

13-13: Consider mentioning that agentd's /metrics is unauthenticated.

The note only describes the control-plane's METRICS_TOKEN protection. Since agentd's /metrics has no token gate, operators binding hooks to a non-loopback address should be aware they need to protect it at the network/proxy level. A brief note here (or a cross-reference to the agentd docs) would help.

services/control-plane/src/routes/metrics.ts (1)

24-26: Consider using a constant-time comparison for the token check.

provided !== expected is vulnerable to timing side-channel attacks. While the risk is low for a metrics endpoint, using crypto.timingSafeEqual would be more robust, especially if the same pattern gets copied elsewhere.

🔒 Proposed fix
+import { timingSafeEqual } from 'node:crypto';
 import type { FastifyInstance } from 'fastify';
 import { config } from '../config.js';
 import { registry } from '../metrics.js';
-      if (provided !== expected) {
+      if (!provided || !timingSafeEqual(Buffer.from(provided), Buffer.from(expected))) {
         return reply.status(401).send('Unauthorized\n');
       }
services/control-plane/src/index.ts (1)

64-68: Auth-bypass list is growing — consider extracting to a set for readability.

Line 66 is becoming unwieldy. A small array/set of prefixes would be easier to maintain and less error-prone when adding future unauthenticated routes.

♻️ Example refactor
+  const publicPrefixes = ['/health', '/metrics', '/v1/agent/connect', '/v1/ui/stream', '/v1/ui/terminal', '/v1/voice/transcribe'];
+
   app.addHook('onRequest', async (request, reply) => {
     const url = request.url;
-    if (url.startsWith('/health') || url.startsWith('/metrics') || url.startsWith('/v1/agent/connect') || url.startsWith('/v1/ui/stream') || url.startsWith('/v1/ui/terminal') || url.startsWith('/v1/voice/transcribe')) {
+    if (publicPrefixes.some((p) => url.startsWith(p))) {
       return;
     }
services/control-plane/src/services/clawdbot.ts (1)

245-274: Metrics recording looks thorough — note the gap for !config.enabled.

All decision paths inside the provider and shouldNotify checks are instrumented. However, the early return at line 239 (if (!config.enabled) return) silently skips recording. If you ever want visibility into "how many events were dropped because clawdbot was off," you'd need to record there too. Not necessarily worth changing now, but worth keeping in mind.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@agents/agentd/go.mod`:
- Line 3: The go.mod contains an invalid patch-level Go version directive ("go
1.23.0"); update the module directive in go.mod to a valid major.minor
form—replace "go 1.23.0" with either "go 1.22" (to match the commit message and
pin compatibility with prometheus client_golang v1.22.0) or "go 1.23" if you
intend to upgrade; ensure the file's go directive uses only major.minor syntax.

Comment thread agents/agentd/go.mod Resolved
@cvsloane
cvsloane merged commit b0a3818 into main Feb 16, 2026
8 checks passed
@cvsloane
cvsloane deleted the feat/ops-metrics branch February 16, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant