Skip to content

add a public scorecard for the firewall contract - #18

Open
o-mid wants to merge 2 commits into
feat/break-it-arenafrom
feat/public-evals
Open

o-mid wants to merge 2 commits into
feat/break-it-arenafrom
feat/public-evals

Conversation

@o-mid

@o-mid o-mid commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Problem

The fixture job checks canned plans. It does not fail when the payment-firewall cases start passing, and nothing in the repo shows the last real run of those cases.

Solution

packages/evals/attacks/v1 holds eleven stored plans for the standing invoice. CI posts them at policy. A missing code, an allow on a deny case, or a missing frozen id fails the job. /scorecard and docs/scorecard.md copy the local firewall run (11 held, 0 failed) and the mock-planner traces. A live model was not called. The nightly workflow runs only on this repository and skips when the Anthropic key, the model, or the token prices are unset. That skip is not a score.

How to test

  • Start policy on :8090 and run npm test in packages/evals.
  • Open /scorecard. The table matches packages/evals/scorecard/firewall.json.
  • EVAL_PLANNER=mock npm run test:attacks replays the mock traces. There is no live-model number to check.

o-mid added 2 commits November 7, 2026 11:30
CI now posts the stored attack plans at policy and fails if a named code stops firing. The scorecard records that local run and a mock-planner pass. A live model was not called.
The firewall and mock traces now name the commit that added the cases. delayed-instruction latency on this pass is 1 ms. A live model was still not called.
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
guardrail-agent Ready Ready Preview Nov 7, 2026 9:43am UTC

This branch was successfully deployed

1 active deployment
Preview — 6ae9244c Deployed Nov 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant