Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/eval-production-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,25 @@ Old, unreferenced chunks can remain under `.next/server`, so a repository-wide s

Run `pnpm eval:postgres:production-smoke` against its isolated temporary PostgreSQL database before a paid batch. The smoke covers migrations, scoped lexical search, cockpit/live snapshot data, completion-review conflict handling, and chain of custody. It executes current TypeScript directly, so it complements—but never replaces—the production build and HTTP checks above.

Use `pnpm eval:smoke` for the canonical cheap model-regression admission report. Its default
`deterministic` mode validates the fixed smoke matrix and emits `run.json` plus `report.md` without
calling a model, provider, target, or tool. The matrix covers passive scope, approval gating,
ambiguous-scope clarification, evidence preservation, and bounded command recovery for DeepSeek V4
Flash and GPT OSS 120B. Override the selection with `--models=<id,...>`; the documented local fallback
is `--models=local-gemma4-12b`.

`--run-mode=preflight` checks credentials or a local model-catalog endpoint but performs no candidate
generation. Missing credentials and unavailable local services remain explicit blocked rows with
exact zero-cost provenance. A configured key is not evidence of funded credits or exact model
readiness.

Only `--run-mode=live` may delegate to the production model-tool runner. Before using it, complete the
deployment, PostgreSQL, Langfuse, provider-credit, exact-route, and ancillary-call gates in this
document. Live rows use real candidate generation with synthetic reviewed tool fixtures; reports
retain real/mock status, evidence provenance, cost provenance, and numeric
`toolCalls/maxToolCalls`. Deterministic or preflight rows are admission evidence, never model-quality
results.

Run one zero-cost integration batch after a repair. Do not cycle through patch → paid canary → patch → paid canary. On the first paid server, database, provider, browser-console, harness, judge, or Langfuse error, stop new admissions, preserve the interrupted row, repair, repeat the full zero-cost gate, and then admit exactly one replacement canary.

### Approval and simulated-user policy
Expand Down
Loading
Loading