Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,4 @@ dist/

# Debugging screenshots and scratch artifacts captured during dev
*.png
packages/cli/src/build-info.ts
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,45 @@ This section accumulates work going into the first tagged customer release.
Phase tags (A, B, C) match the customer-trial roadmap and the commit prefix
convention used in the git log.

### June 2026 audit sweep (PRs #1–#16)

A 14-agent review (2026-06-02) produced 28 findings; all shipped by 2026-06-09:
SDK contract + rate-limit fixes, low-tier hardening, a green Playwright smoke
test in CI, a repo-wide lint pass + CI Lint job, `eslint-plugin-react-hooks`
7.1.x upgrade, `checkSameOrigin` CSRF guard rolled out to all cookie-auth
mutation routes, repo-identity normalization and user attribution fixed at
write time, SSRF hardening for 6to4/NAT64 IPv6 wrappers, Usage-page backend
segmentation (Bedrock vs direct) with a per-user spend table, and
better-sqlite3 12.11.1 + CI Node 22 for Node 26 support.

### July 2026 revival audit (PRs #17–#23)

A follow-up 4-agent review (2026-07-11) found the cost pipeline and several
security gaps; all critical/high findings shipped the same day:

- **Cost pipeline repaired (#17).** The model pricing table had gone stale —
current models (Opus 4.8, Sonnet 5, Fable 5) silently priced at $0 and
Opus 4.6/4.7 at 3× their real rate; the transcript reader double-counted
usage 2–7× (Claude Code writes one line per content block; now deduped by
`message.id`); and transcript paths were mis-derived for project dirs with
non-alphanumeric characters (hooks now use the payload's `transcript_path`).
Unknown models now warn on stderr instead of failing open at $0.
- **Four unauthenticated GET routes gated (#18).** `runs/[id]/{agents,metrics,
policies}` and `policies/[id]/results` leaked cross-tenant data; they now
enforce the same owner-or-admin + 404-on-non-owner rules as their siblings,
and policy results are scoped to the member's own runs.
- **Dependency advisories cleared (#19, #21).** Next.js 16.1.6 → 16.2.10 and
drizzle-orm 0.39 → 0.45.2 clear every high advisory in `npm audit`.
- **Usage page org-wide section works for the first time (#22).** The Admin
API proxies sent parameters the API doesn't accept and the page parsed
fields that don't exist in the responses; the routes now send RFC 3339
`starting_at`, follow pagination, and normalize amounts (decimal-string
cents) server-side. The Activity Summary card (which could only render
zeros) was removed pending Claude Code Analytics API integration.
- **Policies are deletable again (#23).** Deleting any evaluated policy hit a
foreign-key constraint and 500'd; `deletePolicy` now cascades its
`policy_results` in a transaction and reports the count in the audit log.

### Added

#### Phase A — trial-blocking foundations
Expand Down
18 changes: 11 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,24 +60,24 @@ core ← db ← cli
```

- **@agentops/core** — Domain types, policy engine, scoring algorithm, and builder functions for all entities. No external dependencies. All types use `readonly` and branded ID types (RunId, JobId, SessionId, etc.) for type safety.
- **@agentops/db** — SQLite persistence via Drizzle ORM + better-sqlite3. Eight tables: `runs`, `policies`, `policy_results`, `run_metrics`, `jobs`, `sessions`, `events`, `locks`. Complex fields stored as JSON columns. DB defaults to `~/.agentops/agentops.db` (override with `AGENTOPS_DB_PATH`).
- **@agentops/cli** — CLI entry point (`agentops`). Commands: `init`, `serve`, `setup`, `hook`, `run`, `policy`, `report`, `wrap`, `watch`, `link`, `pr`, `job`, `session`, `events`, `lock`, `dispatch`. Supports `--json` output and `--db-path` override. `init` bootstraps the DB (`--seed` for sample data, `--clean` to reset). `serve` starts the dashboard server (`--port` to override 3000). `setup` configures Claude Code hooks (`--global`, `--uninstall`, `--dry-run`). `hook` handles Claude Code hook events (session-start, pre-tool-use, post-tool-use, session-end) — reads JSON from stdin, manages state via temp files, evaluates policies in real-time, can block risky tool calls (exit code 2). `wrap` emits real-time events during execution. Helper modules: `format.ts` (output formatting), `git.ts` (git integration), `github.ts` (GitHub API).
- **@agentops/db** — SQLite persistence via Drizzle ORM + better-sqlite3. Sixteen tables: `runs`, `policies`, `policy_results`, `run_metrics`, `jobs`, `sessions`, `events`, `locks`, `users`, `api_tokens`, `auth_sessions`, `device_codes`, `webhooks`, `webhook_deliveries`, `audit_log`, `user_budgets`. Complex fields stored as JSON columns. DB defaults to `~/.agentops/agentops.db` (override with `AGENTOPS_DB_PATH`). WAL mode with `foreign_keys = ON` — deletes of parent rows must cascade children first (see `deletePolicy`, `deleteOldRuns`).
- **@agentops/cli** — CLI entry point (`agentops`). Commands: `init`, `serve`, `setup`, `hook`, `login`, `doctor`, `user`, `admin`, `cleanup`, `run`, `policy`, `report`, `wrap`, `watch`, `link`, `pr`, `job`, `session`, `events`, `lock`, `dispatch`. Supports `--json` output and `--db-path` override. `init` bootstraps the DB (`--seed` for sample data, `--seed-policies` for the starter policy set, `--clean` to reset). `serve` starts the dashboard server (`--port` to override 3000). `setup` configures Claude Code hooks (`--global`, `--uninstall`, `--dry-run`). `hook` handles Claude Code hook events (session-start, pre-tool-use, post-tool-use, user-prompt-submit, stop, subagent-stop, session-end) — reads JSON from stdin, manages state under `~/.agentops/state/`, evaluates policies in real-time, can block risky tool calls (exit code 2), and reads cost/token usage from the Claude Code transcript (deduped by `message.id`; unknown models warn on stderr instead of pricing at $0). `login` runs the device-flow auth against a dashboard; `doctor` diagnoses a local install. Helper modules: `format.ts` (output formatting), `git.ts` (git integration), `github.ts` (GitHub API), `transcript.ts` (usage/cost from Claude Code transcripts), `pricing` lives in core. Note: `job`, `lock`, `dispatch`, `wrap`, and `watch` operate on orchestration machinery that Claude Code hooks never populate — treat them as experimental/vestigial.
- **@agentops/sdk** — Lightweight HTTP client for agent runtimes to talk to the AgentOps server. Depends only on `@agentops/core` for types. Uses native `fetch`. Provides `AgentOpsClient` class (via `createClient()` factory) with methods: `createSession`, `startRun`, `reportAction`, `reportArtifact`, `reportMetrics`, `checkPolicy`, `heartbeat`, `completeRun`, `failRun`. Also exports `PolicyMiddleware` for pre-flight policy checks before actions. Throws typed `AgentOpsError` with status codes.
- **@agentops/web** — Next.js 16 App Router dashboard with React 19, Tailwind CSS 4. API routes under `src/app/api/` organized by resource (runs, sessions, policies, events, analytics, admin, stats, sdk). Jobs, Locks, and Coordination pages were removed from the dashboard because hooks do not populate this data. Sidebar nav: Runs | Sessions | Events | Analytics | Usage | Policies | Settings. Includes inbound SDK routes under `/api/sdk/` for agent runtime communication and mutation routes for dashboard controls (approve/block runs). The Usage page (`/usage`) proxies to the Anthropic Admin API for cost/usage data; requires the `ANTHROPIC_ADMIN_API_KEY` env var. Admin API routes live at `/api/admin/{status,cost,analytics}`. Server components access SQLite directly via a singleton lazy-loaded DB instance (`src/lib/db.ts`). Uses `serverExternalPackages: ["better-sqlite3"]` in next.config.ts. Path alias: `@/*` → `./src/*`. Dark theme by default.
- **@agentops/web** — Next.js 16 App Router dashboard with React 19, Tailwind CSS 4. API routes under `src/app/api/` organized by resource (runs, sessions, policies, events, analytics, admin, stats, sdk, auth, budgets, webhooks). Jobs, Locks, and Coordination pages were removed from the dashboard because hooks do not populate this data. Sidebar nav: Runs | Sessions | Events | Analytics | Usage | Policies | Settings. **Every API route is authenticated** (`src/lib/auth.ts`): bearer tokens or session cookies, `admin`/`member` roles, members are scoped to their own runs/sessions (`resolveViewScope`), non-owners get 404 (not 403) to avoid ID enumeration, mutations additionally pass `checkSameOrigin` CSRF checks — new routes must follow this pattern (regression tests live in `api/__tests__/auth-gaps.test.ts`). Inbound SDK routes under `/api/sdk/` use bearer auth + per-token rate limits. The Usage page (`/usage`) always shows the local hook-captured rollup (incl. Bedrock-vs-direct backend split and per-user attribution); when `ANTHROPIC_ADMIN_API_KEY` is set it additionally shows org-wide cost/tokens via `/api/admin/{status,cost,analytics}`, which normalize the Anthropic Admin API's reports (RFC 3339 `starting_at`, paginated, amounts are decimal-string cents) server-side. Server components access SQLite directly via a singleton lazy-loaded DB instance (`src/lib/db.ts`). Uses `serverExternalPackages: ["better-sqlite3"]` in next.config.ts. Path alias: `@/*` → `./src/*`. Dark theme by default.

### Core Domain Concepts

**Run** is the central abstraction — an immutable, timestamped record of one autonomous execution containing: Goal, Agents (with roles: Lead/Implementer/Reviewer/CI/Policy), Environment, Actions, Artifacts, Metrics, Evaluations, Decisions, and optional GitHub links. Run modifications always return new objects; never mutate in-place.

**Job** (`core/src/job.ts`) is a dispatchable unit of work. Jobs have priority (Critical/High/Normal/Low), retry policies, and concurrency limits. A Job produces one or more Runs and tracks lifecycle: Queued → Dispatched → Running → Completed/Failed. The **Dispatcher** (`core/src/dispatcher.ts`) provides pure functions for queue ordering, dispatch decisions based on concurrency limits, and session matching.
**Job** (`core/src/job.ts`) is a dispatchable unit of work. Jobs have priority (Critical/High/Normal/Low), retry policies, and concurrency limits. A Job produces one or more Runs and tracks lifecycle: Queued → Dispatched → Running → Completed/Failed. The **Dispatcher** (`core/src/dispatcher.ts`) provides pure functions for queue ordering, dispatch decisions based on concurrency limits, and session matching. ⚠️ **Vestigial:** nothing in the hook-driven product path creates Jobs or dispatches them (hook-created sessions always carry a `currentRunId`, so `matchSession` can never select one); this layer is exercised only by its own tests and the experimental `job`/`dispatch` CLI commands.

**Session** (`core/src/session.ts`) represents an agent runtime's lifecycle: Provisioning → Active → Paused → Terminated. Sessions track the current Run being executed, completed Runs, resource usage (memory, CPU, token/cost budgets), and heartbeats for liveness detection.

**Event System** (`core/src/events.ts`) provides a typed event bus. `EVENT_TYPES` defines all events (job.queued, run.started, session.terminated, policy.violated, cost.threshold, etc.). `EventBus` class supports subscribe/unsubscribe/publish with wildcard `"*"` subscriptions. Events are persisted in the `events` table for audit trail. The web SSE endpoint reads from the events table.

**Coordination** (`core/src/coordination.ts`) handles multi-agent resource management. Lock lifecycle (createLock, releaseLock, isLockExpired, isLockHeld), conflict detection (checkConflicts checks for overlapping repo/path/branch locks), branch isolation (generateWorkBranch), and work partitioning (partitionByPath).
**Coordination** (`core/src/coordination.ts`) handles multi-agent resource management. Lock lifecycle (createLock, releaseLock, isLockExpired, isLockHeld), conflict detection (checkConflicts checks for overlapping repo/path/branch locks), branch isolation (generateWorkBranch), and work partitioning (partitionByPath). ⚠️ **Vestigial:** locks are only ever created via the experimental `lock` CLI command; no runtime path uses them.

**Policy Engine** (`core/src/policy.ts`) evaluates 6 policy types: PathRestriction, FileLimitCount, CostCeiling, RequiredApproval, TestEnforcement, RiskyOpFlag. New policy types must extend the `PolicyConfig` union type.
**Policy Engine** (`core/src/policy.ts`) evaluates 8 policy types: PathRestriction, FileLimitCount, TestEnforcement, RiskyOpFlag, SecretDetection, BranchProtection, ToolRestriction, CostCeiling. Policies run in two modes: **guard** (`evaluatePreToolPolicies` — real-time, called from the PreToolUse hook, can block a tool call) and **check** (post-hoc evaluation of a completed Run). New policy types must extend the `PolicyConfig` union type and usually need both a guard and a check implementation.

**Scoring** (`core/src/scoring.ts`) computes a ScoreCard across 5 dimensions (Correctness, RegressionRisk, ScopeRisk, PolicyCompliance, Unknowns) as 0–1 ratios, producing a MergeRecommendation of Merge | Block | Review.

Expand All @@ -93,6 +93,8 @@ core ← db ← cli
- Repository pattern: all DB functions take `(db: AgentOpsDb, ...)` as first arg, named `insertX`, `getX`, `listX`, `updateX`
- Tests colocated in `src/__tests__/` directories, pattern `*.test.ts`
- Web API routes use `force-dynamic` to ensure fresh data on every request
- Every web API route authenticates via `src/lib/auth.ts` helpers (`requireUser`/`requireAdmin`/`requireOwnedRun`); mutations also call `checkSameOrigin`. Add coverage to `auth-gaps.test.ts` when adding routes
- Web tests import `@agentops/core`/`db` from their built `dist/` — after switching branches or editing core/db, run root `npm run build` first or web tests will exercise stale code
- CLI commands registered via `registerXCommands(program)` pattern in `cli/src/index.ts`
- CLI auto-detects git repo/branch; can override with `--repo` and `--branch` flags
- DB defaults to `~/.agentops/agentops.db`; override with `AGENTOPS_DB_PATH` env var or `--db-path` CLI flag
Expand All @@ -102,4 +104,6 @@ core ← db ← cli
### Environment Variables

- `AGENTOPS_DB_PATH` — Override default SQLite database location (`~/.agentops/agentops.db`)
- `ANTHROPIC_ADMIN_API_KEY` — Anthropic Admin API key for the Usage page. Set before starting the dashboard (`agentops serve`) to enable cost/usage tracking from the Anthropic API. Obtain from the Anthropic Console under Organization Settings.
- `AGENTOPS_FAIL_CLOSED` — Set to `1`/`true` to make hooks block tool calls when enforcement can't be verified (offline dashboard, rejected token). Default is fail-open so AgentOps never bricks a working Claude Code session.
- `CLAUDE_CODE_USE_BEDROCK` — Read (not set) by the hooks to tag a session's spend as `bedrock` vs direct `anthropic` for the Usage page's backend split.
- `ANTHROPIC_ADMIN_API_KEY` — Anthropic Admin API key for the org-wide half of the Usage page. Set before starting the dashboard (`agentops serve`). Obtain from the Anthropic Console under Organization Settings. Model pricing for the local rollup lives in `core/src/pricing.ts` (`ANTHROPIC_PRICING`) — **it must be refreshed when new Claude models ship**, or their sessions warn on stderr and record $0.
8 changes: 5 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,9 +77,11 @@ services:
depends_on:
- dashboard
volumes:
# Read-only mount of the SQLite db. Litestream reads the WAL frames
# but never writes back to the live database.
- ./agentops-data:/data:ro
# Read-write mount: Litestream must be able to open the database with
# write access — it takes the WAL checkpoint lock and maintains its own
# -litestream sidecar files next to the db. A :ro mount here breaks
# replication outright (it fails at startup, it doesn't degrade).
- ./agentops-data:/data
- ./litestream.yml:/etc/litestream.yml:ro
environment:
AGENTOPS_S3_BUCKET: ${AGENTOPS_S3_BUCKET}
Expand Down
6 changes: 5 additions & 1 deletion litestream.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@ dbs:
# Prefix within the bucket. Lets one bucket back up multiple DBs.
path: ${AGENTOPS_S3_PREFIX:-agentops}
region: ${AWS_REGION:-us-east-1}
# Optional endpoint override for S3-compatible stores.
# Using an S3-compatible store (MinIO, R2, Wasabi)? Set
# AGENTOPS_S3_ENDPOINT in .env AND uncomment the next line — the
# env var alone does nothing while this stays commented. Leave it
# commented for real AWS S3 (an empty endpoint can break the AWS
# default resolution).
# endpoint: ${AGENTOPS_S3_ENDPOINT}
# 14 days of point-in-time recovery. Tune for your RPO needs;
# the cost of WAL frames at AgentOps scale is sub-cent / day.
Expand Down
1 change: 1 addition & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
],
"scripts": {
"prebuild": "node scripts/write-build-info.mjs",
"pretest": "node scripts/write-build-info.mjs",
"build": "tsc",
"lint": "eslint src",
"test": "vitest run"
Expand Down
8 changes: 0 additions & 8 deletions packages/cli/src/build-info.ts

This file was deleted.

7 changes: 4 additions & 3 deletions packages/cli/src/commands/serve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,10 @@ export function registerServeCommand(program: Command): void {
console.log(`AgentOps dashboard starting at http://${displayHost}:${port}`);
if (host !== "127.0.0.1" && host !== "localhost") {
console.warn(
`WARNING: dashboard is binding to ${host}. The API is currently unauthenticated — `
+ `anyone with network reach can read and mutate data. Bind to 127.0.0.1 unless `
+ `you have added auth.`,
`Note: dashboard is binding to ${host} and reachable from the network. `
+ `The API requires authentication (run 'agentops login' to create the first `
+ `admin), but traffic is plain HTTP — put it behind TLS for anything beyond `
+ `a trusted LAN (see docker-compose.yml's caddy profile).`,
);
}

Expand Down
Loading
Loading