AgentTrace stores full prompts, tool payloads and customer records, which is the most sensitive data an AI team has. This document says what the project does about that, what it does not do, and where the gaps are. It describes the code in this repository, not a certified product.

The demo's customer data is synthetic (generated by Era). No real personal data is used.

• Redaction at the source. The SDK redacts before anything leaves the process: a key-name denylist (authorization, api_key, password, token, secret, cookie) plus patterns for API keys, bearer tokens, emails, phone numbers and card-like numbers. A hit becomes [REDACTED:<type>:<hash8>], so equal secrets still compare equal in a diff.
• Redacted again on ingest, in case an SDK is misconfigured. Redaction is idempotent.
• Size limits. Every stored text field is cut to 32 KB (with its length and sha256 kept), and request bodies over 5 MB are refused.
• Memory holds summaries only. Agent Memory gets a short redacted paragraph about a verified root cause, never prompts or payloads.
• Deletion. An organization admin can delete a run. That removes its steps, every fork of it, the experiments run on it, its root-cause reports, and the Agent Memory patterns derived from it. The deletion is written to the audit log first and must succeed.

• The workspace comes only from the verified identity: the Clerk session token (organization id) or the hashed ingest key. It is never read from a request body, and every query filters on it. Tests prove a run in one organization is a 404 for another.
• Clerk tokens are verified against Clerk's published keys (RS256 only), with issuer, expiry and authorized-party checks.
• Roles. Members view runs and start forks and experiments. Admins also delete runs and read the audit log.
• Ingest keys are made by organization admins in the app (Connect your agents). A key is shown once and stored as a sha256 hash with a short display prefix. Admins can stop a key at any time, and making keys is rate limited and audited.
• Audit log. Experiment launches, root-cause runs and deletions are recorded with the actor, workspace and time. Admins can read it at GET /v1/audit.
• Imported logs and the saved-example copy go through the same redaction and 32 KB limit as recorded runs, are rate limited and audited, and stay inside the caller's own workspace. Imported runs are view only: forking and root-cause analysis only run workflows that are registered on the server.
• OpenTelemetry ingest (POST /v1/traces) takes the same ingest key, redacts and clamps exactly like ingest, refuses anything but JSON, caps the decoded body at 5 MB (a gzip bomb is stopped while decompressing), 50 traces per request and 300 steps per trace, and never lets a trace id from one workspace overwrite another's run. It has no rate limit of its own yet, as POST /v1/import has none for keys; both are listed under known gaps below.
• Code checks (submit_verdict, oracle=) run in the caller's own process, never on our servers. Only the check's name, its verdict and a redacted reason of at most 300 characters are stored; a check that raises stores only the exception type.
• Key scopes. A key holds ingest, worker, read and spend as an admin chose when making it, and each route checks the one it needs. A key made before scopes can only ingest. A key that only sends runs cannot read them; only a spend key can cost tokens, and it is charged to the key: the per-minute and daily experiment limits and the audit log treat key:<id> as the user.
• Worker routes (/v1/worker/*) take a worker key. A job is claimed with compare-and-set so two workers cannot take one, a result must present the claim it was given (a stale or forged claim is refused), a result is recorded once, and what a worker sends is redacted and clamped like ingest. Our server never calls a worker. We cannot see inside a worker's code: the owner states with safe_to_rerun=True that re-running is safe.
• Agent routes (/v1/agent/*, used by MCP and the CLI) only read and propose. Run content in their responses is clipped and marked as data, not instructions, because it was recorded from other programs and may contain text aimed at an agent. State values are never returned, only key names.
• Alert webhooks make requests to an address an admin typed in, so the address is checked when saved and again when used: https only, the standard port, no user name in it, and the host must resolve only to public addresses; Slack addresses must be on hooks.slack.com; redirects are not followed, and the connection goes to the very address that passed the check (the name cannot be swung to a private address in between). The address is a secret and is only shown as its host and last four characters. Messages to Slack are escaped so run text cannot ping a channel or plant a link, a generic post is signed with an HMAC the receiver can check, a run alerts once, a workspace gets at most 10 alerts a minute, and the likely cause (which costs tokens) is opt-in per webhook and charged to it.
• The CI gate only re-runs an experiment the key's workspace already has, under a spend key and the usual limits. It fails closed: a gate that cannot ask or cannot finish exits non-zero.
• Public demo routes serve one fixed workspace chosen by server configuration, have no write endpoints, and are cached.
• In production the interactive API docs are switched off, and cross-origin access is limited to the configured web origin.

• Forks run only registered workflows. There is no endpoint that runs arbitrary code.
• A side-effecting tool cannot run in a live fork unless a mock is registered, and a fork that could call one is refused up front.
• Spend limits: at most MAX_N runs per arm, a token ceiling per experiment, a daily experiment cap per user, and a per-minute rate limit on the endpoints that spend tokens (starting an experiment, running an analysis).
• The trace is untrusted input. The root-cause agent receives it inside <trace> tags with an instruction to treat it as data, has no tool that acts on anything, must return a schema-validated answer, and any evidence quote it gives is checked verbatim against the trace. Recalled memory is treated the same way.
• The agent proposes; a person launches experiments. Nothing is applied to your prompts or code.

• It does not claim deterministic replay of live steps. Recorded steps replay exactly; forked steps re-run live and are sampled.
• It does not apply fixes to production code or prompts.
• It does not train models on your traces.
• It does not send traces anywhere except SurrealDB (storage), Anthropic (analysis and live steps) and, as a redacted summary, Agent Memory.
• It does not replace monitoring or alerting, and it holds no compliance certification (SOC 2, ISO 27001 or similar).

These are real and deliberate cuts, not oversights:

• Redaction is heuristic. Patterns can miss free-form secrets in prose, and can redact harmless text (a part code shaped like a phone number). A bare 10-digit number is not treated as a phone number.
• Rate limits are in memory, so they are per server process and reset on restart. The daily cap is counted from the database and holds across restarts.
• The body-size check relies on Content-Length. A client that streams without one is not stopped by the app; put a reverse-proxy limit in front for that.
• Key limits are per server process. Every call made with a key is limited to 600 a minute per key (a 429 with Retry-After), and requests are bounded by size, trace count and step count, but the counter is in memory, so it resets on restart and is not shared between servers. A leaked key should still be revoked. The OpenTelemetry endpoint also reads a chunked body without a length in full before checking its size; the reverse-proxy limit above covers it.
• No automatic retention sweep. Workspaces record a retention period, but nothing expires data on its own. Deletion is manual, by an admin.
• No automatic key rotation and no redaction-rule endpoints. Keys are made and stopped by hand; changing redaction rules means changing the code.
• Experiments run inside the API process. They survive the request that started them, not a restart.
• No security headers or CSP are set on the web app yet.
• Sign-up is open on the Clerk instance, so anyone can create an organization. The per-user caps and the Anthropic spend limit are what bound the cost.
• The Era token used for the demo workflow expires and must be refreshed by hand.

Please open a private security advisory on the GitHub repository rather than a public issue. Include steps to reproduce and what you could access. Do not include real customer data.

Secrets live in environment variables (a local .env, which is gitignored, and the Render dashboard). .env.example lists names only. Run a secret scan such as gitleaks over the repository and its history before it is made public.