Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions agents/agentd/go.mod
Original file line number Diff line number Diff line change
@@ -1,13 +1,24 @@
module github.com/agent-command/agentd

go 1.22
go 1.22.0

require (
github.com/creack/pty v1.1.24
github.com/fsnotify/fsnotify v1.7.0
github.com/google/uuid v1.6.0
github.com/gorilla/websocket v1.5.3
github.com/prometheus/client_golang v1.22.0
gopkg.in/yaml.v3 v3.0.1
)

require golang.org/x/sys v0.4.0 // indirect
require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.62.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
golang.org/x/sys v0.30.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
)
42 changes: 39 additions & 3 deletions agents/agentd/go.sum
Original file line number Diff line number Diff line change
@@ -1,14 +1,50 @@
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA=
github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
golang.org/x/sys v0.4.0 h1:Zr2JFtRQNX3BCZ8YtxRE9hNJYC8J6I1MVbMg6owUp18=
golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_golang v1.22.0 h1:rb93p9lokFEsctTys46VnV1kLCDpVZ0a/Y92Vm0Zc6Q=
github.com/prometheus/client_golang v1.22.0/go.mod h1:R7ljNsLXhuQXYZYtw6GAE9AZg8Y7vEW5scdCXrWRXC0=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.62.0 h1:xasJaQlnWAeyHdUBeGjXmutelfJHWMRr+Fg4QszZ2Io=
github.com/prometheus/common v0.62.0/go.mod h1:vyBcEuLSvWos9B1+CyL7JZ2up+uFzXhkqml0W5zIY1I=
github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0learggepc=
github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
98 changes: 98 additions & 0 deletions agents/agentd/internal/metrics/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
package metrics

import (
"net/http"
"sync"

"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
)

var once sync.Once

var (
wsConnected = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "agentd_ws_connected",
Help: "Whether agentd is currently connected to the control plane (1=yes, 0=no).",
})

wsReconnecting = prometheus.NewGauge(prometheus.GaugeOpts{
Name: "agentd_ws_reconnecting",
Help: "Whether agentd is currently in a reconnect loop (1=yes, 0=no).",
})

wsReconnectAttemptsTotal = prometheus.NewCounter(prometheus.CounterOpts{
Name: "agentd_ws_reconnect_attempts_total",
Help: "Total WebSocket reconnect attempts to the control plane.",
})

wsReconnectFailuresTotal = prometheus.NewCounter(prometheus.CounterOpts{
Name: "agentd_ws_reconnect_failures_total",
Help: "Total WebSocket reconnect failures to the control plane.",
})

wsReconnectSuccessTotal = prometheus.NewCounter(prometheus.CounterOpts{
Name: "agentd_ws_reconnect_success_total",
Help: "Total WebSocket reconnect successes to the control plane.",
})

wsReconnectBackoffSeconds = prometheus.NewHistogram(prometheus.HistogramOpts{
Name: "agentd_ws_reconnect_backoff_seconds",
Help: "Backoff delay (sleep) used before each reconnect attempt.",
Buckets: []float64{0.25, 0.5, 1, 2, 5, 10, 30, 60},
})
)

func initOnce() {
once.Do(func() {
prometheus.MustRegister(
wsConnected,
wsReconnecting,
wsReconnectAttemptsTotal,
wsReconnectFailuresTotal,
wsReconnectSuccessTotal,
wsReconnectBackoffSeconds,
)
})
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

func Handler() http.Handler {
initOnce()
return promhttp.Handler()
}

func SetWSConnected(connected bool) {
initOnce()
if connected {
wsConnected.Set(1)
return
}
wsConnected.Set(0)
}

func SetWSReconnecting(reconnecting bool) {
initOnce()
if reconnecting {
wsReconnecting.Set(1)
return
}
wsReconnecting.Set(0)
}

func RecordWSReconnectAttempt(delayMs int) {
initOnce()
wsReconnectAttemptsTotal.Inc()
if delayMs > 0 {
wsReconnectBackoffSeconds.Observe(float64(delayMs) / 1000.0)
}
}

func RecordWSReconnectFailure() {
initOnce()
wsReconnectFailuresTotal.Inc()
}

func RecordWSReconnectSuccess() {
initOnce()
wsReconnectSuccessTotal.Inc()
}
2 changes: 2 additions & 0 deletions agents/agentd/internal/providers/claude.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
"time"

"github.com/agent-command/agentd/internal/config"
"github.com/agent-command/agentd/internal/metrics"
"github.com/google/uuid"
)

Expand Down Expand Up @@ -63,6 +64,7 @@ func (p *ClaudeProvider) Start() error {
mux := http.NewServeMux()
mux.HandleFunc("/v1/hooks/claude", p.handleHook)
mux.HandleFunc("/v1/hooks/codex", p.handleCodexHook)
mux.Handle("/metrics", metrics.Handler())

p.server = &http.Server{
Addr: p.cfg.HooksHTTPListen,
Expand Down
44 changes: 29 additions & 15 deletions agents/agentd/internal/ws/client.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,28 @@ import (
"sync/atomic"
"time"

"github.com/gorilla/websocket"
"github.com/agent-command/agentd/internal/metrics"
"github.com/agent-command/agentd/internal/queue"
"github.com/gorilla/websocket"
)

type MessageHandler func(msgType string, payload json.RawMessage)

type Client struct {
url string
token string
hostID string
backoff []int
conn *websocket.Conn
mu sync.Mutex
seq atomic.Int64
lastAckedSeq int64
onMessage MessageHandler
done chan struct{}
reconnecting bool
queue *queue.Queue
stateDir string
onConnect func()
url string
token string
hostID string
backoff []int
conn *websocket.Conn
mu sync.Mutex
seq atomic.Int64
lastAckedSeq int64
onMessage MessageHandler
done chan struct{}
reconnecting bool
queue *queue.Queue
stateDir string
onConnect func()
}

func NewClient(url, token, hostID string, backoff []int) *Client {
Expand Down Expand Up @@ -76,6 +77,8 @@ func (c *Client) Connect() error {

c.conn = conn
c.reconnecting = false
metrics.SetWSConnected(true)
metrics.SetWSReconnecting(false)

// Start reader goroutine
go c.reader()
Expand All @@ -95,6 +98,8 @@ func (c *Client) reader() {
}
c.mu.Unlock()

metrics.SetWSConnected(false)

// Attempt reconnection
c.reconnect()
}()
Expand Down Expand Up @@ -174,19 +179,25 @@ func (c *Client) reconnect() {
c.reconnecting = true
c.mu.Unlock()

metrics.SetWSReconnecting(true)
defer metrics.SetWSReconnecting(false)

for i, delay := range c.backoff {
select {
case <-c.done:
return
case <-time.After(time.Duration(delay) * time.Millisecond):
}

metrics.RecordWSReconnectAttempt(delay)
log.Printf("Reconnection attempt %d/%d", i+1, len(c.backoff))

if err := c.Connect(); err == nil {
metrics.RecordWSReconnectSuccess()
log.Printf("Reconnected successfully")
return
}
metrics.RecordWSReconnectFailure()
}

// Keep trying with max backoff
Expand All @@ -198,10 +209,13 @@ func (c *Client) reconnect() {
case <-time.After(time.Duration(maxDelay) * time.Millisecond):
}

metrics.RecordWSReconnectAttempt(maxDelay)
if err := c.Connect(); err == nil {
metrics.RecordWSReconnectSuccess()
log.Printf("Reconnected successfully")
return
}
metrics.RecordWSReconnectFailure()
}
}

Expand Down
1 change: 1 addition & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,6 @@ Reference:

Operations:
- [Operations](operations.md)
- [Reliability Primitives](reliability-primitives.md)
- [Troubleshooting](troubleshooting.md)
- [Implementation Notes](implementation.md)
7 changes: 7 additions & 0 deletions docs/agentd.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,13 @@ Disable these if you want read only mode on a host.
agentd can handle provider specific hooks and usage parsing.
See [Provider Hooks](hooks.md) for setup details.

### Metrics

agentd exposes Prometheus metrics at `GET /metrics` on the same HTTP listener as provider hooks (`providers.claude.hooks_http_listen`).

- Default: `http://127.0.0.1:7777/metrics`
- If you bind hooks to a non-loopback address, treat `/metrics` as sensitive (firewall or reverse proxy auth).

### Provider usage polling

agentd can run periodic commands to pull usage data from provider CLIs or APIs
Expand Down
1 change: 1 addition & 0 deletions docs/operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ This guide covers operational best practices for running Agent Commander.
## Monitoring

- Watch `/health` for control plane status.
- Scrape `/metrics` for operational counters (reconnect/backoff, Clawdbot rate-limit decisions). If `METRICS_TOKEN` is set on the control plane, Prometheus must send `Authorization: Bearer <token>` (or `x-metrics-token`).
- Monitor database connections and disk usage.
- Track agentd logs for reconnect loops or tmux errors.

Expand Down
60 changes: 60 additions & 0 deletions docs/reliability-primitives.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Reliability Primitives

This doc inventories how **retry/backoff**, **rate limiting**, and **ops metrics** currently work in Agent Command, and proposes a migration path toward consistent, observable behavior.

## Inventory (Current State)

### Retry + Backoff

**agentd (Go)**
- WebSocket reconnect loop: `agents/agentd/internal/ws/client.go`
- Policy: fixed backoff schedule from config `control_plane.reconnect_backoff_ms` (default `[250, 500, 1000, 2000, 5000]` ms) and then retries forever at the max delay.
- No jitter.
- No per-attempt timeout (relies on the underlying dial behavior).
- Ad-hoc “retry once” for Claude usage command parsing: `agents/agentd/cmd/agentd/main.go` (`maybeRetryClaudeUsage`)
- If output looks incomplete, re-runs the command using `script -c` to force a TTY.
- No backoff.

**control-plane (TypeScript)**
- No shared retry helper (outbound HTTP calls generally rely on per-call timeouts).
- One “retry for race” pattern exists in group creation logic (check-then-recheck on unique collision): `services/control-plane/src/routes/groups.ts` and `services/control-plane/src/db/index.ts`.

### Rate Limiting

**control-plane (TypeScript)**
- In-memory per-user throttle for Clawdbot notifications: `services/control-plane/src/services/clawdbot.ts`
- Default: `maxPerHour = 30` with additional dedupe + session cooldown rules.
- Not distributed-safe (assumes a single control-plane instance for correct enforcement).

### Metrics / Observability

**Before 2026-02-16:** operational metrics (Prometheus-style) were not exposed; only session analytics existed in Postgres (`migrations/009_session_analytics.sql`).

**After 2026-02-16:** operational `/metrics` endpoints exist:
- control-plane: `GET /metrics` (token optional via `METRICS_TOKEN`)
- agentd: `GET /metrics` served from the hooks HTTP server (same listen address as `providers.claude.hooks_http_listen`)

## Standard Primitives (Recommended)

### Go (agentd)
1. **Retry/backoff standard:** exponential backoff with jitter and a hard cap.
2. **Timeout standard:** every outbound operation should be bounded by context deadlines.
3. **Metrics standard:** Prometheus counters/histograms for reconnect attempts, failures, and backoff delays.

### TypeScript (control-plane)
1. **Retry/backoff standard:** one shared helper (jitter, caps, per-attempt timeout, and explicit “non-retryable” classification).
2. **Rate limiting standard:** if control-plane is ever scaled horizontally, move throttles to Redis-backed primitives (in-memory is fine only when single-instance is guaranteed).
3. **Metrics standard:** Prometheus counters/histograms for decisioning (allowed/blocked + reason) and key outbound call failures.

## Migration Plan (Incremental)

1. **(Done) Add operational metrics endpoints and key counters**
- control-plane `/metrics` + Clawdbot allow/deny reasons
- agentd `/metrics` + reconnect/backoff counters
2. **Centralize retry/backoff**
- Introduce a small `retry` helper in each language (repo-local), then migrate high-value call sites first.
3. **Rate limiting hardening (only if/when needed)**
- If control-plane becomes multi-instance: add Redis and migrate Clawdbot throttle to distributed-safe limits.
4. **Dashboards/alerts**
- Alert on reconnect storms, reconnect failure rates, and sustained Clawdbot rate-limit blocks.

Loading
Loading