Skip to content

make eval failures a regression contract - #11

Merged
o-mid merged 1 commit into
mainfrom
feat/eval-regression
Sep 17, 2026
Merged

o-mid merged 1 commit into
mainfrom
feat/eval-regression

Conversation

@o-mid

@o-mid o-mid commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

Problem

Fixture evals printed a terse FAIL line. If someone deleted policy-infinite-approve, policy-bad-recipient, or injection-schema-junk, CI still went green. A deny that started passing would only fail if the fixture file still existed and still expected a deny.

Solution

The runner prints expected vs actual on every mismatch. Those three ids are a freeze list: missing file or ok=true fails CI even if expect was edited. GitHub Actions unsets OPENAI_API_KEY / EVAL_PLANNER / PLANNER before npm test. Adding a fixture is documented in packages/evals/README.md.

How to test

cd packages/evals && npm install
node --import tsx --test ./compare.test.ts
# with policy on :8090
POLICY_SERVICE_URL=http://127.0.0.1:8090 npm test

Unit tests cover a vanished freeze id and a freeze case that starts passing. CI eval job must stay keyless.

CI stays keyless and fails if frozen deny fixtures vanish or start passing.
@vercel

vercel Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
guardrail-agent Ready Ready Preview Sep 17, 2026 12:23pm UTC

@o-mid
o-mid merged commit 8137894 into main Sep 17, 2026
9 checks passed

This branch was successfully deployed

1 active deployment
Preview — 98833ce6 Deployed Sep 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant