Repository navigation
Ops metrics: /metrics + reconnect + rate-limit counters - #50
Conversation
|
Warning Rate limit exceeded
⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📝 WalkthroughWalkthroughAdds Prometheus metrics across agentd and the control plane: new agentd metrics package, WebSocket instrumentation, control-plane metrics module and routes with optional token auth, clawdbot decision telemetry and typed decision reasons, plus related docs and dependency updates. Changes
Sequence Diagram(s)sequenceDiagram
participant WS as WebSocket Client
participant Metrics as Agentd Metrics
participant Registry as Prometheus Registry
participant HTTP as Agentd HTTP Handler
participant Scraper as Prometheus Scraper
WS->>Metrics: SetWSConnected(true)
Metrics->>Registry: update connected gauge
WS->>Metrics: RecordWSReconnectAttempt(delayMs)
Metrics->>Registry: increment reconnect attempts / observe delay
WS->>Metrics: RecordWSReconnectSuccess()
Metrics->>Registry: increment success counter
WS->>Metrics: SetWSConnected(false)
Metrics->>Registry: update connected gauge
Scraper->>HTTP: GET /metrics
HTTP->>Registry: registry.metrics()
Registry-->>HTTP: formatted metrics
HTTP-->>Scraper: 200 text/plain (Prometheus format)
sequenceDiagram
participant Queue as Notification Queue
participant Claw as ClawdbotNotifier
participant Metrics as Control-plane Metrics
participant Counter as Decision Counter
Queue->>Claw: queueNotification(event)
Claw->>Claw: shouldNotify() (dedupe, cooldown, rate-limit)
alt blocked
Claw->>Metrics: recordClawdbotNotificationDecision(blocked, reason, eventType, provider)
Metrics->>Counter: increment decision=blocked, reason=...
Counter-->>Metrics: updated
Metrics-->>Claw: ack
else allowed
Claw->>Metrics: recordClawdbotNotificationDecision(allowed, -, eventType, provider)
Metrics->>Counter: increment decision=allowed
Counter-->>Metrics: updated
Metrics-->>Claw: ack
end
Claw-->>Queue: return decision
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@agents/agentd/internal/metrics/metrics.go`:
- Around line 47-60: The initOnce function currently calls
prometheus.MustRegister with collectors.NewGoCollector() and
collectors.NewProcessCollector(...), which will panic due to duplicate
registration with the default registry; fix by removing those two default
collectors from the prometheus.MustRegister call and only register your custom
metrics (wsConnected, wsReconnecting, wsReconnectAttemptsTotal,
wsReconnectFailuresTotal, wsReconnectSuccessTotal, wsReconnectBackoffSeconds),
or alternatively create and use a dedicated prometheus.NewRegistry() and
register all collectors (including NewGoCollector and NewProcessCollector) on
that registry instead of the global prometheus registry; update initOnce
accordingly to use one of these two approaches and ensure references to the
metric variables (wsConnected, wsReconnecting, etc.) remain registered on the
chosen registry.
🧹 Nitpick comments (4)
docs/operations.md (1)
13-13: Consider mentioning that agentd's/metricsis unauthenticated.The note only describes the control-plane's
METRICS_TOKENprotection. Since agentd's/metricshas no token gate, operators binding hooks to a non-loopback address should be aware they need to protect it at the network/proxy level. A brief note here (or a cross-reference to the agentd docs) would help.services/control-plane/src/routes/metrics.ts (1)
24-26: Consider using a constant-time comparison for the token check.
provided !== expectedis vulnerable to timing side-channel attacks. While the risk is low for a metrics endpoint, usingcrypto.timingSafeEqualwould be more robust, especially if the same pattern gets copied elsewhere.🔒 Proposed fix
+import { timingSafeEqual } from 'node:crypto'; import type { FastifyInstance } from 'fastify'; import { config } from '../config.js'; import { registry } from '../metrics.js';- if (provided !== expected) { + if (!provided || !timingSafeEqual(Buffer.from(provided), Buffer.from(expected))) { return reply.status(401).send('Unauthorized\n'); }services/control-plane/src/index.ts (1)
64-68: Auth-bypass list is growing — consider extracting to a set for readability.Line 66 is becoming unwieldy. A small array/set of prefixes would be easier to maintain and less error-prone when adding future unauthenticated routes.
♻️ Example refactor
+ const publicPrefixes = ['/health', '/metrics', '/v1/agent/connect', '/v1/ui/stream', '/v1/ui/terminal', '/v1/voice/transcribe']; + app.addHook('onRequest', async (request, reply) => { const url = request.url; - if (url.startsWith('/health') || url.startsWith('/metrics') || url.startsWith('/v1/agent/connect') || url.startsWith('/v1/ui/stream') || url.startsWith('/v1/ui/terminal') || url.startsWith('/v1/voice/transcribe')) { + if (publicPrefixes.some((p) => url.startsWith(p))) { return; }services/control-plane/src/services/clawdbot.ts (1)
245-274: Metrics recording looks thorough — note the gap for!config.enabled.All decision paths inside the provider and
shouldNotifychecks are instrumented. However, the early return at line 239 (if (!config.enabled) return) silently skips recording. If you ever want visibility into "how many events were dropped because clawdbot was off," you'd need to record there too. Not necessarily worth changing now, but worth keeping in mind.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@agents/agentd/go.mod`:
- Line 3: The go.mod contains an invalid patch-level Go version directive ("go
1.23.0"); update the module directive in go.mod to a valid major.minor
form—replace "go 1.23.0" with either "go 1.22" (to match the commit message and
pin compatibility with prometheus client_golang v1.22.0) or "go 1.23" if you
intend to upgrade; ensure the file's go directive uses only major.minor syntax.
Adds Prometheus /metrics endpoints and minimal operational counters.
Tests/build:
Summary by CodeRabbit
New Features
Documentation
Chores