Skip to content

feat(judging): add multi-sample pass^k grading - #289

Merged
slowdini merged 1 commit into
devfrom
feat/multi-sample-judging
Aug 24, 2026
Merged

feat(judging): add multi-sample pass^k grading#289
slowdini merged 1 commit into
devfrom
feat/multi-sample-judging

Conversation

@slowdini

Copy link
Copy Markdown
Owner

What and why

Repeated LLM verdicts over one execution expose judge consistency without paying to rerun the agent. This adds assertion-level and campaign-wide judge sampling, then carries the individual verdicts, vote proportion, and pass^k endpoint through grading and benchmark aggregation.

The implementation builds on the bounded judge evidence bundle from #288: every sample for a run receives the exact same persisted judge-evidence.md, so sampling varies the judgment rather than the evidence under review.

Closes #262.
Part of #244.

Strategy

  • Resolve the effective count from llm_judge.samples, then run --judge-samples, then the compatibility default of one.
  • Fan one authored LLM assertion out to independently resumable __sample-N judge tasks while reusing one bounded evidence bundle. The framework-owned __skill_invoked meta-check remains single-shot.
  • Treat a missing response as a failure for that sample only, preserving the other responses and their evidence and confidence values.
  • Store sampled assertions as ordered verdicts plus passed, failed, total, vote proportion p, and pass_power_k = p^N. Mixed gradings weight each authored assertion equally: binary assertions contribute zero or one, and sampled assertions contribute their derived endpoint.
  • Aggregate per-condition vote-proportion and pass^k statistics, deltas, and pooled per-assertion votes. Fully binary campaigns retain the existing Fisher exact sample-size floor; sampled campaigns print the non-binary endpoints without presenting them as significance tests.

An eval can opt into sampling directly:

{
  "id": "clear-review",
  "type": "llm_judge",
  "rubric": "The review identifies the most important defect and explains its impact.",
  "samples": 10
}

Operators can instead set a campaign default:

eval-magic run --skill ./my-skill --harness claude-code \
  --judge-samples 10

The assertion value takes precedence over the command-line default. With six passing verdicts out of 10, the retained vote proportion is 0.6 and pass^k is 0.6^10, approximately 0.006047.

Compatibility and safety boundaries

  • An effective count of one preserves the response filename, binary assertion fields, grading summary, and benchmark passed / n rollup.
  • Sampled task filenames are validated against authored assertion IDs, preventing ambiguous paths such as an authored quality__sample-1 colliding with a generated sample for quality.
  • The judge evidence remains bounded, persisted once, and read-only. Sampling does not rerun the agent, rebuild the bundle, or expose held-out command-check results.
  • Mode B uses the same sample-aware artifacts and response paths. Promotion retains the evidence bundle and benchmark data, while existing provenance fields continue to carry the source revision.
  • The schemas accept the sampled artifact variants while retaining the legacy binary variants for existing consumers and historical data.

Verification

  • cargo fmt --check
  • cargo build
  • cargo test — 1,300 tests passed
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

Review focus

The main review surface is the compatibility split between one-sample binary artifacts and multi-sample artifacts, plus the equal-assertion weighting used when a run mixes binary and sampled assertions.

Reuse each run's bounded evidence across independent judge verdicts and retain vote proportions plus pass^k without changing the one-sample artifact contract.
@slowdini
slowdini merged commit c4af136 into dev Aug 24, 2026
7 checks passed
@slowdini
slowdini deleted the feat/multi-sample-judging branch August 24, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-sample judging (pass^k)

1 participant