Repository navigation

Add adrian-eval: a scored, gateable eval harness for the judge - #98

Open

Ankit1478 wants to merge 22 commits into

secureagentics:mainfrom

Ankit1478:feat/judge-eval-harness

Open

Add adrian-eval: a scored, gateable eval harness for the judge#98
Ankit1478 wants to merge 22 commits into
secureagentics:mainfrom
Ankit1478:feat/judge-eval-harness

Commits on Oct 8, 2026
- Add adrian-eval, an eval harness for the judge classifier
  Ankit1478
  committed
  06d90b4
- Flag also_ok answers that change the action tier
  Ankit1478
  committed
  388a5cc
- Add low-harm M2.d cases with near-miss twins
  Ankit1478
  committed
  6437b38
- Add -runs N to repeat the eval and report unstable cases
  Ankit1478
  committed
  8f6ae31
- Retry network errors and judge cases in parallel
  Ankit1478
  committed
  1c9a813
- Add a pass/fail gate with thresholds and an exit code
  Ankit1478
  committed
  0759a63
- Add -compare to diff two saved reports
  Ankit1478
  committed
  38fff12
- Record run metadata, token usage and cost in every report
  Ankit1478
  committed
  f9e877d
- Add a README for adrian-eval
  Ankit1478
  committed
  d349616
- Record all run settings and save every report automatically
  Ankit1478
  committed
  162983f
- Add a label guide and grow the eval dataset to 266 cases
  Ankit1478
  committed
  f26fc50
- Balance the held-out set so it measures false positives
  Ankit1478
  committed
  64a54e3
Commits on Oct 9, 2026
- Support multi-step cases so chained attacks can be tested
  Ankit1478
  committed
  0d746cd
- Judge cases against agent profiles from a real database
  Ankit1478
  committed
  771bb7f
- Fix three harness bugs that distorted the numbers
  Ankit1478
  committed
  7a1ada2
- Harden the eval harness against reporting the wrong thing
  Ankit1478
  committed
  8ceb3e3
- Run the Go backend's tests on pull requests
  Ankit1478
  committed
  a69a183
- Document the harness features the README had fallen behind
  Ankit1478
  committed
  7f9aa22
- Cover the multi-run gate, which decides every real gated run
  Ankit1478
  committed
  b7325ef
- Say why an errored case counts on one side of the gate and not the other
  Ankit1478
  committed
  38d2efd
- Treat reading special-category data as a breach on the first record
  Ankit1478
  committed
  62274bb
- Gate missed blocks on the worst run, not the average
  Ankit1478
  committed
  0954b1f

Commits

Commits on Oct 8, 2026

Commits on Oct 9, 2026