diff --git a/.gitignore b/.gitignore index f51a7f6..5239043 100644 --- a/.gitignore +++ b/.gitignore @@ -32,15 +32,7 @@ engine/.beir/ # built widget IIFE copied next to the Next app for the same-origin dogfood page apps/api/public/orca-widget.iife.js -# local planning + review docs — kept on disk for orientation, never committed -docs/ -SECURITY-AUDIT.md -SECURITY-BACKLOG.md - # misc .DS_Store *.log .vercel - -# Local Claude preview launch config -.claude/launch.json diff --git a/AGENTS.md b/AGENTS.md index 54c5094..4483651 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,11 +1,10 @@ # AGENTS.md -## Cursor Cloud specific instructions +## Dev environment This is a pnpm + Turborepo monorepo (TypeScript serving layer) plus a `uv`-managed -Python ingestion engine (`engine/`). The update script already runs `pnpm install` -and `uv sync` (in `engine/`) on startup, so dependencies are ready when you begin. -`uv` lives at `~/.local/bin` (added to `PATH` via `~/.bashrc`); in a fresh +Python ingestion engine (`engine/`). Run `pnpm install` at the root and `uv sync` +in `engine/` before starting. `uv` installs to `~/.local/bin`; in a fresh non-login shell run `export PATH="$HOME/.local/bin:$PATH"` if `uv` is not found. ### Services / how to run diff --git a/README.md b/README.md index eb4dc55..0ab9ac1 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,106 @@ # Orca -Internal codename for an injection-resilient, multi-source, agentic RAG platform with -non-custodial on-chain action-prep. Boardwalk is the first tenant; the architecture is a -generic platform, not an add-on to any one app. +[![ci](https://github.com/useboardwalk/orca/actions/workflows/ci.yml/badge.svg)](https://github.com/useboardwalk/orca/actions/workflows/ci.yml) -Two engines over one Neon Postgres + pgvector store: +Injection-resilient, multi-tenant agentic RAG with non-custodial on-chain action-prep. +Orca answers questions over an ingested knowledge base with cited evidence, reads live +on-chain data, and prepares — never submits — transactions the user signs in their own +wallet. [Boardwalk](https://useboardwalk.com) is the first tenant; the core is a generic +platform behind a `TenantAdapter` contract. -- a **Python connector framework** (`engine/`) that ingests docs and OpenAPI specs into the - store behind a uniform `Connector` contract, and -- a **TypeScript serving layer** (`apps/`, `packages/`) — a deterministic router fronting a - hand-rolled tool loop that retrieves cited evidence, reads live on-chain data, and prepares - (never submits) transactions the user signs in their own wallet. +The design premise: an LLM will eventually obey instructions planted in retrieved content, +so don't trust it not to. Safety is deterministic code, not model alignment — a taint gate +that makes obedience not matter for the action path. -The load-bearing safety is deterministic, not model-trust: a non-custodial wallet gate, a -taint-gated reachability check on action tools, schema-enforced field provenance, RLS tenant -isolation, and an egress allowlist. +```mermaid +flowchart LR + W["Widget · Playground · Telegram bot"] --> R + subgraph serve ["Serving — TypeScript"] + R["Deterministic router"] --> L["Agent loop
+ taint gate"] + end + subgraph ingest ["Ingestion — Python engine"] + C["Connectors
docs_mdx · openapi"] + end + DB[("Neon Postgres
pgvector + FTS · RLS")] + C --> DB + L -->|searchKnowledge| DB + L -->|"getLaunch — live read"| API["Boardwalk API"] + L -->|"prepare* — action tools"| TX["Unsigned tx bundle"] + TX --> U["User signs in their own wallet"] +``` + +## The taint gate + +Retrieved content carries a trust tier (tier-0 first-party docs → tier-2 uploads and +external content). The moment a tool returns content at or above the taint tier, the turn +is tainted — and while tainted, action tools are removed from the set the model is even +offered. Taint is monotonic and enforced in code the model can't argue with. A live +tainted turn: + +```text +router tier=tier2 reasons=["action-verb","live-lexicon"] +step 1 actionToolsActive=true → searchKnowledge → taintedAfter=true (tier-2 chunk retrieved) +step 2 actionToolsActive=false ← action tools left the model's reach +final tainted=true no signable action surfaced; answer grounded in tier-0 sources +``` + +Defense in depth behind the gate: spotlighted/datamarked untrusted chunks, schema-enforced +field provenance (an action address must appear verbatim in the user's typed message), +non-custodial by construction, an egress allowlist, RLS tenant isolation, and spend caps. +The full threat model, prior art (CaMeL, spotlighting), and stated boundaries — including +what is *not* defended — are in [SECURITY.md](./SECURITY.md). + +## Measured + +Injection resistance, end to end against the live serving path (details and caveats in +[SECURITY.md](./SECURITY.md#evidence) and +[packages/bench/RESULTS.md](./packages/bench/RESULTS.md)): + +| Measurement | Result | +|---|---| +| Deterministic gate floor — model *scripted to obey* the injection (`poison-asr.test.ts`, per PR) | action ASR **0** | +| [AgentDojo](https://github.com/ethz-spylab/agentdojo) attack wrappers × claim-to-attacker goal, ingested tier-2, wallet attached | action-hijack ASR **0/5**, gate engaged 5/5 | +| Live red-team (promptfoo) vs a running `/v1/chat` | **5/5** passed | +| Own-corpus PoisonedRAG-style sweep — injected *instructions*, up to 8/8 retrieval flooding | ASR **0** | +| Same sweep — injected *facts* with no true counter-source in the corpus | ASR ~1.0 — **undefended**, documented as the boundary | + +Retrieval quality (BEIR, the production spine: hybrid RRF → voyage rerank-2.5): + +| nDCG@10 | Orca hybrid | Orca + rerank | best published single-model first stage | +|---|---|---|---| +| SciFact | 0.802 | **0.807** | 0.784 — NV-Embed-v1 (7B) | +| FiQA-2018 | 0.565 | **0.629** | 0.631 — NV-Embed-v1 (7B) | ## Layout -- `apps/api` — Next.js 16 route handlers (`/v1/chat`, `/v1/search`) + a `/playground` page (dev, port 3000) -- `apps/web` — Next.js admin console (better-auth): tenant settings, usage/activity, knowledge, feedback (port 3001) -- `packages/core` — router, hand-rolled loop + taint gate, tools, retrieval, guardrails -- `packages/db` — Drizzle schema (the cross-language contract), RLS policy, `withTenant`, retrieval -- `packages/tenant-boardwalk` — the Boardwalk tenant adapter (system prompt + tools) consumed via `TenantAdapter`, plus its evals -- `packages/widget` — embeddable React chat widget + the dependency-free typed stream client (`@orca/widget/stream`) -- `packages/bench` — external retrieval benchmarks (BEIR) + the injection/poison-resistance harness -- `packages/config` — shared tsconfig, the zod env contract, and the model catalog (allowlist + pricing) -- `evals/redteam` — live red-team (promptfoo + AgentDojo) against a running `/v1/chat`, with committed results -- `engine/` — Python (uv) connector framework + `docs_mdx` / `openapi` connectors +- `apps/api` — Next.js route handlers (`/v1/chat`, `/v1/search`), the Telegram bot webhook, `/playground` (port 3000) +- `apps/web` — admin console (better-auth): tenant settings, knowledge, usage, feedback (port 3001) +- `packages/core` — router, agent loop + taint gate, tools, retrieval, guardrails +- `packages/db` — Drizzle schema (the cross-language contract), RLS policy, `withTenant` +- `packages/tenant-boardwalk` — the Boardwalk tenant adapter (system prompt + tools) + its evals +- `packages/widget` — embeddable React chat widget + a dependency-free typed stream client +- `packages/bench` — BEIR benchmarks + the injection/poison-resistance harness +- `packages/config` — shared tsconfig, the zod env contract, the model catalog +- `evals/redteam` — live red-team (promptfoo + AgentDojo) with committed results +- `engine/` — Python (uv) connector framework: `docs_mdx`, `openapi` ## Develop -Node is pinned in `.nvmrc` (22.x; the machine default may be older — don't use it). pnpm via -corepack; Python via uv. +Node 22 (`.nvmrc`), pnpm via corepack, Python via uv. ```sh pnpm install pnpm lint:ci # biome lint + format (what CI runs) -pnpm -r typecheck && pnpm -r test # TS +pnpm -r typecheck && pnpm -r test # TS (pglite in-memory Postgres; no services needed) node scripts/lint-tenant-access.mjs # tenant tables only reachable via withTenant -cd engine && uv run pytest # Python (incl. docs_mdx JS-parity fixtures) +cd engine && uv run pytest # Python, offline ``` -Live paths need `apps/api/.env.local` + `engine/.env` (see `.env.example`): an AI Gateway key -and a Neon serving URL whose role is **non-owner + NOBYPASSRLS** (the owner role bypasses RLS). +Live paths need `apps/api/.env.local` + `engine/.env` (see [.env.example](./.env.example)): +an AI Gateway key and two Neon URLs — `DATABASE_URL` (serving) whose role must be +**non-owner + NOBYPASSRLS** (the owner role bypasses RLS, making tenant isolation a no-op), +and `ADMIN_DATABASE_URL` (owner role) used only for migrations and role setup, never for +serving. Boot fails loud if the serving role can bypass RLS. ```sh pnpm --filter @orca/db exec tsx src/migrate.ts # ADMIN_DATABASE_URL: schema + RLS + serving role @@ -51,17 +108,3 @@ pnpm --filter @orca/db exec tsx scripts/seed-dev.mts # dev tenant + knowledge b cd engine && uv run engine ingest docs_mdx && uv run engine ingest openapi pnpm --filter @orca/api dev # http://localhost:3000/playground ``` - -## Security & evaluation - -The safety claim — untrusted retrieved content cannot drive a signable on-chain action — and the -evidence for it are in [SECURITY.md](./SECURITY.md): the lethal-trifecta threat model, the -deterministic taint gate, how it relates to prior art (CaMeL, spotlighting, AgentDojo), and the -boundary (fact-poisoning is not defended). - -- `evals/redteam` — live red-team against a running `/v1/chat` (latest run: 5/5, committed `results.json`) -- `packages/bench` — external BEIR retrieval results + the injection/poison resistance numbers -- `packages/core/src/agent/poison-asr.test.ts` — the deterministic action-ASR gate floor, run per PR - -> "Orca" is an internal codename only — rename before any public launch (collisions: orca.so -> Solana DEX, Microsoft Orca LLM, Orca Security; npm `orca` is taken). diff --git a/SECURITY.md b/SECURITY.md index 533d6b4..12ad156 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -172,8 +172,8 @@ final tainted=true no review-sign action surfaced; answer ground The same structure is emitted as OpenTelemetry spans (`orca.chat` → `retrieve.hybrid` → `tool.searchKnowledge`, carrying `orca.tier`, `orca.tainted`, `orca.tool_class`, `orca.prepared_actions`) and exported to Langfuse when `LANGFUSE_PUBLIC_KEY` / -`LANGFUSE_SECRET_KEY` are set; the exporter self-disables when they are not (the current dev -config), so the in-band trace above is the committed artifact. +`LANGFUSE_SECRET_KEY` are set; the exporter self-disables when they are not, so the in-band +trace above is the committed artifact. ## Boundaries and non-goals diff --git a/apps/api/app/v1/chat/route.ts b/apps/api/app/v1/chat/route.ts index 7eeab59..41692f0 100644 --- a/apps/api/app/v1/chat/route.ts +++ b/apps/api/app/v1/chat/route.ts @@ -60,7 +60,7 @@ const MAX_BODY_BYTES = 32 * 1024; const MAX_MESSAGE_CHARS = 8000; // Cross-instance daily spend cap applied when a tenant has none configured. The in-memory // meter in requestGate is per serverless instance; this DB-backed cap holds across them. -// Interim until per-tenant config + a shared store land (Phase 3); 0 = refuse all paid gens. +// Interim until per-tenant config + a shared store exist; 0 = refuse all paid gens. const DEFAULT_TENANT_CAP_USD = 25; // Wall-clock budget for the tier-2 agent loop, with headroom under maxDuration for diff --git a/package.json b/package.json index f2bd4d8..606eb43 100644 --- a/package.json +++ b/package.json @@ -3,7 +3,7 @@ "private": true, "version": "0.0.0", "type": "module", - "description": "Injection-resilient, multi-source, agentic RAG platform with non-custodial on-chain action-prep (internal codename).", + "description": "Injection-resilient, multi-source, agentic RAG platform with non-custodial on-chain action-prep.", "packageManager": "pnpm@9.15.9", "engines": { "node": ">=22" diff --git a/packages/bench/BENCHMARK-STRATEGY-2026.md b/packages/bench/BENCHMARK-STRATEGY-2026.md index fa68d43..93c083e 100644 --- a/packages/bench/BENCHMARK-STRATEGY-2026.md +++ b/packages/bench/BENCHMARK-STRATEGY-2026.md @@ -68,8 +68,8 @@ So the best comparisons for a hybrid+rerank product like Orca: ## 3. What Orca is missing → prioritized roadmap -Storage is unblocked (Neon Launch). Treat each as diagnostic, not a target to overfit. Targets -below are verified 2026 published numbers a hybrid+rerank pipeline should be measured against. +Treat each as diagnostic, not a target to overfit. Targets below are verified 2026 published +numbers a hybrid+rerank pipeline should be measured against. | Priority | Run | Why | Status / target | |---|---|---|---| @@ -89,7 +89,7 @@ own published 3072-dim FiQA number (0.507 vs 0.618), and a fair eval would need migration + ~3× storage. Given the mirage and voyage-4-large's clean win, that migration isn't worth it. The reranker (voyage rerank-2.5) is confirmed current — no 3.x exists. -## 4. What to publish (rank-ordered, with caveats) +## 4. Headline results and caveats (rank-ordered) The governing finding: MTEB and BEIR — the only leaderboards buyers universally see — carry **no** security signal, and no RAG product vendor publishes injection-ASR. Orca's @@ -213,14 +213,14 @@ number) and a **misleading-context** set (the poison fixture, truth-present + sp voyage-4-large puts Orca's first stage at the top of the single-model cluster on SciFact (0.802) and level with e5-mistral-7b on FiQA (0.565); reranked, it tops every published jina-v3+reranker -stack. Present this as "competitive, validated engine," never as frontier-leading first-stage -retrieval — gemini-embedding-001 (native, 3072-dim) still leads FiQA first-stage, and that is fine -because BEIR is table-stakes, not the differentiator. - -**Strategy:** lead public material with §4.1 (injection resistance — the genuine white space, -with the honest fact-poison boundary stated), support with §4.2 (faithfulness + utility) and §4.3 -(cost-efficiency), and keep BEIR (§4.4) as credibility context. Never present a first-stage -retrieval number as frontier-leading, never publish a bare zero-ASR — pair every zero with a +stack. This is a competitive, validated engine, not frontier-leading first-stage retrieval — +gemini-embedding-001 (native, 3072-dim) still leads FiQA first-stage, and that is fine because +BEIR is table-stakes, not the differentiator. + +**Reading order:** §4.1 (injection resistance — the genuine white space, with the honest +fact-poison boundary stated) is the headline; §4.2 (faithfulness + utility) and §4.3 +(cost-efficiency) support it; BEIR (§4.4) is credibility context. A first-stage retrieval number +is never frontier-leading here, and a zero-ASR means nothing bare — every zero needs its paired utility number and an "under repeated/adaptive attempts" qualifier (the Opus 4.6 system card, 17.8% single-attempt ASR rising to 78.6% by the 200th, is why a single static pass won't survive scrutiny on its own). diff --git a/packages/bench/RESULTS.md b/packages/bench/RESULTS.md index 5726b62..90c3439 100644 --- a/packages/bench/RESULTS.md +++ b/packages/bench/RESULTS.md @@ -138,8 +138,8 @@ stage from 0.461 to 0.565 — now level with e5-mistral-7b (0.566) and above eve embedder, though still under native gemini-embedding-001 (0.618) and NV-Embed-v1 (0.631). Reranking adds a real +0.064 (0.565→0.629), and Orca's reranked 0.629 beats the best published jina-v3+reranker stack (mxbai-rerank-large-v2, 0.528) by ~0.10, because it reranks a -higher-recall hybrid (R@100 0.91) rather than a single dense first stage. Lead with the reranked -rows and the rerank lift; the first stage is competitive but not frontier-leading. +higher-recall hybrid (R@100 0.91) rather than a single dense first stage. The reranked rows and +the rerank lift are the headline; the first stage is competitive but not frontier-leading. ## Embedder A/B — what we kept and why diff --git a/packages/core/src/guardrails/no-advice.recall.test.ts b/packages/core/src/guardrails/no-advice.recall.test.ts index 30c5410..8f87f22 100644 --- a/packages/core/src/guardrails/no-advice.recall.test.ts +++ b/packages/core/src/guardrails/no-advice.recall.test.ts @@ -1,7 +1,6 @@ /** - * No-advice gate recall/precision eval (audit M1). The detector was broadened to catch - * paraphrased EXPLICIT advice, then retightened after review so it can't over-refuse grounded - * mechanics docs. This guards both directions: + * No-advice gate recall/precision eval. The detector catches paraphrased EXPLICIT advice + * without over-refusing grounded mechanics docs. This guards both directions: * - precision: no benign / mechanics string is ever flagged (the gate REPLACES the answer, so * a false positive suppresses a factual doc answer), * - recall: explicit advice — canonical and paraphrased — is caught. @@ -25,7 +24,7 @@ const ADVICE_OUTPUT = [ "guaranteed returns", "This token will likely moon.", "price target is $5", - // former M1 misses — paraphrases of explicit advice + // paraphrases of explicit advice "Strong buy.", "It could easily 10x from here.", "I'd recommend buying.", @@ -68,7 +67,7 @@ const BENIGN_QUERY = [ const recall = (items: string[], fn: (s: string) => boolean) => items.filter((t) => fn(t)).length / items.length; -describe("no-advice gate — recall/precision eval (audit M1)", () => { +describe("no-advice gate — recall/precision eval", () => { it("high precision: benign + mechanics outputs and queries are never flagged", () => { for (const t of BENIGN_OUTPUT) expect(looksLikeAdvice(t), t).toBe(false); for (const q of BENIGN_QUERY) expect(solicitsAdvice(q), q).toBe(false); diff --git a/packages/db/src/crypto.ts b/packages/db/src/crypto.ts index 58c49c4..4a0611b 100644 --- a/packages/db/src/crypto.ts +++ b/packages/db/src/crypto.ts @@ -2,7 +2,7 @@ * Symmetric encryption for connector secrets at rest (AES-256-GCM). The dashboard stores and * manages secrets; plaintext never reaches the browser. The key is a 32-byte hex value in * ORCA_SECRET_KEY. The blob layout is stable and language-neutral so the engine can read it - * back at ingest once secret consumption is wired (a connector follow-up). + * back at ingest. * * Blob layout: `v1:` + base64( iv[12] ‖ tag[16] ‖ ciphertext ). */ diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 2e42b8b..9630dd1 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -3,7 +3,7 @@ packages: - "packages/*" # Single source of truth for shared versions. Use `catalog:` in package.json deps. -# AI SDK v6 line verified against node_modules — see PHASE0.md. +# AI SDK v6 line verified against node_modules. catalog: # AI SDK v6 (pinned — security-critical surface) ai: 6.0.209 diff --git a/scripts/verify/README.md b/scripts/verify/README.md index 1b1c0fe..c2c83da 100644 --- a/scripts/verify/README.md +++ b/scripts/verify/README.md @@ -11,4 +11,4 @@ node guc-spike.mjs # SET LOCAL one-transaction / fail-closed RL curl -s https://ai-gateway.vercel.sh/v1/models # public model catalog (voyage/cohere/anthropic) ``` -Node must be 22.x (see ../../.nvmrc). The default shell node on this machine is v14 — do not use it. +Node must be 22.x (see ../../.nvmrc); an older system-default node will fail.