From 080aa5ca475329f33e0756a9e152534c85ae93f6 Mon Sep 17 00:00:00 2001 From: rangarajan19 Date: Mon, 19 Oct 2026 11:16:53 +0530 Subject: [PATCH 1/2] Add opt-in Prometheus /metrics endpoint for tool calls (#1003) Adds a second, separate node:http listener (off by default) serving tool-call counters, a duration histogram, upstream-error classification and mirrored process gauges via prom-client. Required METRICS_TOKEN and dropped per-tool/per-connector labels on Cloud keep it instance-wide and leak-free across tenant workspaces. --- docs/deployment.md | 5 + docs/operations/observability.md | 35 +++ package-lock.json | 165 +++++++------- packages/backend/package.json | 1 + packages/backend/src/common/metrics.spec.ts | 208 ++++++++++++++++++ packages/backend/src/common/metrics.ts | 203 +++++++++++++++++ packages/backend/src/common/secrets.util.ts | 8 + .../src/health/sentry-verify.controller.ts | 8 +- packages/backend/src/main.ts | 6 + .../src/mcp-server/dynamic-mcp-tools.ts | 9 + 10 files changed, 566 insertions(+), 82 deletions(-) create mode 100644 packages/backend/src/common/metrics.spec.ts create mode 100644 packages/backend/src/common/metrics.ts diff --git a/docs/deployment.md b/docs/deployment.md index fafbdd84e..68c701efb 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -359,6 +359,11 @@ curl -s http://localhost:4000/api/mcp-api-keys \ | `DOMAIN` | No | Domain for Caddy SSL certificate (e.g., `example.com`) | | `ACME_EMAIL` | No | Email for Let's Encrypt certificate notifications | | `APP_BIND_IP` | No | Bind IP for app ports (default: `0.0.0.0`, set `127.0.0.1` behind Caddy) | +| `METRICS_ENABLED` | No | Start a second, separate `/metrics` listener for Prometheus (default: `false`) — see [observability.md](operations/observability.md) | +| `METRICS_PORT` | No | Port for the metrics listener (default: `9464`) | +| `METRICS_HOST` | No | Bind host for the metrics listener (default: `0.0.0.0`) | +| `METRICS_TOKEN` | No on self-hosted, **required on Cloud** | Bearer token required to scrape `/metrics`, compared with `timingSafeEqual` | +| `METRICS_TOOL_LABELS` | No | Add `connector_id`/`tool` labels to tool-call metrics (default: `false`; ignored on Cloud) | --- diff --git a/docs/operations/observability.md b/docs/operations/observability.md index 8d87b4746..f0f5a04bb 100644 --- a/docs/operations/observability.md +++ b/docs/operations/observability.md @@ -176,6 +176,41 @@ Use the *raw* request body, not a re-serialized JSON object, since re-serializin The webhook URL is validated with the same guard (`SSRF_GUARD`, private/loopback/metadata-address blocking) used for every other outbound call this app makes on a user's behalf, both when it's saved and again on every dispatch, and redirects are not followed. On AnythingMCP Cloud the URL must use `https`, and it ignores both the env (`SSRF_ALLOWED_HOSTS`) and the admin-editable SSRF allowlist, which name hosts only the platform may reach. On a self-hosted instance the operator's allowlist applies, so an internal Mattermost or Slack proxy can receive alerts. +## 6. Prometheus metrics (opt-in) + +Off unless `METRICS_ENABLED=true`. When on, the backend starts a **second, separate `node:http` listener** for `/metrics` — it never shares the main app's port 4000, because `/health*` is proxied publicly on both Cloud and self-hosted (`deploy/cloud/Caddyfile`, [deployment.md](../deployment.md)), and `/metrics` must not be. + +| Env | Default | | +|---|---|---| +| `METRICS_ENABLED` | `false` | Starts the listener. | +| `METRICS_PORT` | `9464` | Metrics listener port. | +| `METRICS_HOST` | `0.0.0.0` | Metrics listener bind host — a Prometheus container needs to reach it over the compose network, so the port is deliberately **not** published in `docker-compose.yml`. | +| `METRICS_TOKEN` | unset | Required `Authorization: Bearer` token, compared with `timingSafeEqual`. **Required on Cloud** — without it the listener refuses to start and logs an error. Optional on self-hosted. | +| `METRICS_TOOL_LABELS` | `false` | Adds `connector_id` (cuid, not name) and `tool` labels to the tool-call metrics. Self-hosted only — **always ignored on Cloud**, even if set, so no series can carry a connector or tool identity across tenant workspaces. | + +### Metrics (prefix `anythingmcp_`) + +- `tool_calls_total{connector_type,status}` — counter, `status` is `success` or `error`. +- `tool_call_duration_seconds{connector_type,status}` — histogram, buckets from 0.05 s to 60 s (tool calls can run 20 s or more). +- `tool_call_upstream_errors_total{connector_type,upstream_status_class}` — `upstream_status_class` is `4xx`, `5xx` or `none`. +- `tool_calls_in_flight`, `mcp_requests_in_flight`, `registered_tools` — gauges mirrored from the in-process `processGauges` singleton at scrape time (not polled on an interval). +- The standard `prom-client` Node runtime metrics (`collectDefaultMetrics()`: event loop lag, heap, GC, etc.). + +### Sample Prometheus scrape config + +```yaml +scrape_configs: + - job_name: anythingmcp + scheme: http + bearer_token: '' + static_configs: + - targets: ['anythingmcp-backend:9464'] +``` + +### Cloud vs self-hosted + +On Cloud, metrics are instance-wide only: no series may carry an organization id, connector name, tool name or user, which is why `METRICS_TOOL_LABELS` has no effect there and `METRICS_TOKEN` is mandatory. Per-organization usage already exists in the database (`AuditService.getBreakdowns`) and is unaffected by this endpoint. On self-hosted, the operator who sets `METRICS_TOOL_LABELS=true` is the only consumer of their own instance's metrics, so per-connector/per-tool labels are allowed. + ## Fatal crash reports (always on in the Docker image) A fatal V8 error, such as an out-of-memory abort, ends the backend before any JavaScript handler runs, so the log has no stack. `start.sh` therefore starts the backend with `--report-on-fatalerror`: on such a crash Node writes a diagnostic report (`report..