diff --git a/docker-compose.cloud.yml b/docker-compose.cloud.yml index e6c845c7d..7ede1078e 100644 --- a/docker-compose.cloud.yml +++ b/docker-compose.cloud.yml @@ -87,6 +87,15 @@ x-backend: &backend - SENTRY_ENVIRONMENT=${SENTRY_ENVIRONMENT:-production} - SENTRY_TRACES_SAMPLE_RATE=${SENTRY_TRACES_SAMPLE_RATE:-0.05} - SENTRY_VERIFY_TOKEN=${SENTRY_VERIFY_TOKEN:-} + # Prometheus /metrics — a separate listener, off by default (docs/operations/observability.md). + # METRICS_TOKEN is required once METRICS_ENABLED=true; the listener refuses to start without it. + # METRICS_TOOL_LABELS is always ignored here regardless of its value — no series may carry a + # connector id or tool name across tenant workspaces. + - METRICS_ENABLED=${METRICS_ENABLED:-false} + - METRICS_PORT=${METRICS_PORT:-9464} + - METRICS_HOST=${METRICS_HOST:-0.0.0.0} + - METRICS_TOKEN=${METRICS_TOKEN:-} + - METRICS_TOOL_LABELS=${METRICS_TOOL_LABELS:-false} - NEXT_PUBLIC_API_URL=https://${DOMAIN} - DATABASE_URL=postgresql://amcp:${POSTGRES_PASSWORD}@postgres:5432/anythingmcp - REDIS_URL=redis://redis:6379 diff --git a/docker-compose.yml b/docker-compose.yml index 68ffb68fc..b31342d81 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -110,6 +110,13 @@ services: - OPENAI_API_KEY=${OPENAI_API_KEY:-} - OPENROUTER_API_KEY=${OPENROUTER_API_KEY:-} - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY:-} + # Prometheus /metrics — a separate listener, off by default (docs/operations/observability.md). + # METRICS_TOKEN is required once METRICS_ENABLED=true; the listener refuses to start without it. + - METRICS_ENABLED=${METRICS_ENABLED:-false} + - METRICS_PORT=${METRICS_PORT:-9464} + - METRICS_HOST=${METRICS_HOST:-0.0.0.0} + - METRICS_TOKEN=${METRICS_TOKEN:-} + - METRICS_TOOL_LABELS=${METRICS_TOOL_LABELS:-false} depends_on: postgres: diff --git a/docs/deployment.md b/docs/deployment.md index fafbdd84e..53d172041 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -359,6 +359,11 @@ curl -s http://localhost:4000/api/mcp-api-keys \ | `DOMAIN` | No | Domain for Caddy SSL certificate (e.g., `example.com`) | | `ACME_EMAIL` | No | Email for Let's Encrypt certificate notifications | | `APP_BIND_IP` | No | Bind IP for app ports (default: `0.0.0.0`, set `127.0.0.1` behind Caddy) | +| `METRICS_ENABLED` | No | Start a second, separate `/metrics` listener for Prometheus (default: `false`) — see [observability.md](operations/observability.md) | +| `METRICS_PORT` | No | Port for the metrics listener (default: `9464`) | +| `METRICS_HOST` | No | Bind host for the metrics listener (default: `0.0.0.0`) | +| `METRICS_TOKEN` | **Required when `METRICS_ENABLED=true`** | Bearer token required to scrape `/metrics`, compared with `timingSafeEqual` | +| `METRICS_TOOL_LABELS` | No | Add `connector_id`/`tool` labels to tool-call metrics (default: `false`; ignored on Cloud) | --- diff --git a/docs/operations/observability.md b/docs/operations/observability.md index 8d87b4746..4d249e72e 100644 --- a/docs/operations/observability.md +++ b/docs/operations/observability.md @@ -176,6 +176,41 @@ Use the *raw* request body, not a re-serialized JSON object, since re-serializin The webhook URL is validated with the same guard (`SSRF_GUARD`, private/loopback/metadata-address blocking) used for every other outbound call this app makes on a user's behalf, both when it's saved and again on every dispatch, and redirects are not followed. On AnythingMCP Cloud the URL must use `https`, and it ignores both the env (`SSRF_ALLOWED_HOSTS`) and the admin-editable SSRF allowlist, which name hosts only the platform may reach. On a self-hosted instance the operator's allowlist applies, so an internal Mattermost or Slack proxy can receive alerts. +## 6. Prometheus metrics (opt-in) + +Off unless `METRICS_ENABLED=true`. When on, the backend starts a **second, separate `node:http` listener** for `/metrics` — it never shares the main app's port 4000, because `/health*` is proxied publicly on both Cloud and self-hosted (`deploy/cloud/Caddyfile`, [deployment.md](../deployment.md)), and `/metrics` must not be. + +| Env | Default | | +|---|---|---| +| `METRICS_ENABLED` | `false` | Starts the listener. | +| `METRICS_PORT` | `9464` | Metrics listener port. | +| `METRICS_HOST` | `0.0.0.0` | Metrics listener bind host — a Prometheus container needs to reach it over the compose network, so the port is deliberately **not** published in `docker-compose.yml`. | +| `METRICS_TOKEN` | unset | **Required** `Authorization: Bearer` token, compared with `timingSafeEqual`, whenever `METRICS_ENABLED=true` — on Cloud and self-hosted alike. Without it the listener refuses to start and logs an error; the default bind host `0.0.0.0` would otherwise serve `/metrics` to anyone who can reach the host. | +| `METRICS_TOOL_LABELS` | `false` | Adds `connector_id` (cuid, not name) and `tool` labels to the tool-call metrics. Self-hosted only — **always ignored on Cloud**, even if set, so no series can carry a connector or tool identity across tenant workspaces. | + +### Metrics (prefix `anythingmcp_`) + +- `tool_calls_total{connector_type,status}` — counter, `status` is `success` or `error`. +- `tool_call_duration_seconds{connector_type,status}` — histogram, buckets from 0.05 s to 60 s (tool calls can run 20 s or more). +- `tool_call_upstream_errors_total{connector_type,upstream_status_class}` — only incremented for an actual upstream error; `upstream_status_class` is `4xx` or `5xx`. +- `tool_calls_in_flight`, `mcp_requests_in_flight`, `registered_tools` — gauges mirrored from the in-process `processGauges` singleton at scrape time (not polled on an interval). +- The standard `prom-client` Node runtime metrics (`collectDefaultMetrics()`: event loop lag, heap, GC, etc.). + +### Sample Prometheus scrape config + +```yaml +scrape_configs: + - job_name: anythingmcp + scheme: http + bearer_token: '' + static_configs: + - targets: ['app:9464'] +``` + +### Cloud vs self-hosted + +On Cloud, metrics are instance-wide only: no series may carry an organization id, connector name, tool name or user, which is why `METRICS_TOOL_LABELS` has no effect there. Per-organization usage already exists in the database (`AuditService.getBreakdowns`) and is unaffected by this endpoint. On self-hosted, the operator who sets `METRICS_TOOL_LABELS=true` is the only consumer of their own instance's metrics, so per-connector/per-tool labels are allowed. `METRICS_TOKEN` is required on both: the listener's default bind host is `0.0.0.0`. + ## Fatal crash reports (always on in the Docker image) A fatal V8 error, such as an out-of-memory abort, ends the backend before any JavaScript handler runs, so the log has no stack. `start.sh` therefore starts the backend with `--report-on-fatalerror`: on such a crash Node writes a diagnostic report (`report..