AgentOps Eval Studio is an AI agent engineering workspace for deterministic tool orchestration, code-level guardrails, trace inspection, and scenario evaluation. It is built to show the parts that separate a real agent system from a chat demo: permissions, policy checks, memory retrieval, output validation, audit traces, and regression tests.
- Deterministic observe -> decide -> act -> validate -> handoff loop.
- Tool registry with
read,review, andwrite-dry-runpermission classes. - Policy classifier that blocks destructive or approval-bypass instructions before tool use.
- Local memory retrieval for policy/runbook grounding.
- Signal inspection and plan review before handoff generation.
- Trace persistence under
traces/for local replay and debugging. - React UI for running goals, viewing quality gates, and inspecting trace events.
- API tests plus scenario eval suite in CI.
npm install
npm run devOpen http://localhost:5173. The API defaults to http://localhost:4420.
npm test # API tests plus scenario evals
npm run eval # deterministic scenario suite
npm run build # production React bundle
npm run start # serve API and built frontend| Method | Endpoint | Description |
|---|---|---|
GET |
/api/health |
Service health and run mode |
GET |
/api/tools |
Tool catalog and permission classes |
GET |
/api/metrics/runtime |
Runtime request counters |
GET |
/api/metrics/scorecard |
Operational readiness score and checks |
POST |
/api/runs |
Execute a dry-run agent run and persist trace |
npm testverifies API contracts, blocked requests, dry-run quality, and eval scenarios.npm run evalreplays realistic safe, blocked, and review-required goals.- GitHub Actions runs tests and build on pull requests and
main. - Architecture notes live in docs/ARCHITECTURE.md.
MIT
This repository now includes contribution guidelines, a security policy, operational runbook notes, PR review gates, and automated readiness checks. See docs/ENTERPRISE_READINESS.md and docs/OPERATIONS.md.