Skip to content

feat(judging): add multi-sample pass^k grading #289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
slowdini merged 1 commit into from
Aug 24, 2026
Merged

feat(judging): add multi-sample pass^k grading #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/developer_overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ focused internal notes instead of duplicating their details.
4. `eval-magic ingest` reads the harness outputs, transcript evidence, guard denials, and final
task state. Runner-owned deterministic checks and diff-scope evidence are collected here.
5. `eval-magic grade` evaluates runner-owned assertions, writes one bounded `judge-evidence.md`
per recorded run, and emits tasks for assertions that require an LLM. Each task inlines the
exact bundle for its run. `eval-magic dispatch --judges` runs those judge tasks through the
selected harness.
per recorded run, and emits one or more tasks for assertions that require an LLM. Every sample
inlines the exact same bundle for its run. `eval-magic dispatch --judges` runs those judge tasks
through the selected harness.
6. `eval-magic finalize` checks that required work is complete and writes the final per-run and
benchmark artifacts. `eval-magic aggregate` combines campaigns when a larger comparison is
needed.
Expand Down
53 changes: 53 additions & 0 deletions docs/guides/judging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,59 @@ files are injected, and keeping the bundle at that boundary prevents a judge fro
runner-owned mutations with agent work. Mechanical assertion results remain runner-owned and are
merged during `finalize`.

## Sample an LLM judge

An authored `llm_judge` assertion can request several independent verdicts for the same run:

```json
{
"id": "clear-review",
"type": "llm_judge",
"rubric": "The review identifies the most important defect and explains its impact.",
"samples": 10
}
```

Use `run --judge-samples N` to set a campaign-wide default. An assertion's `samples` field takes
precedence over that default. The effective count must be at least one. The framework-injected
`__skill_invoked` meta-check is not substantive grading and remains single-shot.

Each sample is a separate judge task and response, but every sample for a run receives the exact
same bounded `judge-evidence.md`. The agent is not rerun, and eval-magic does not rebuild or expand
the evidence between samples. This measures agreement among repeated judgments of one execution;
it does not estimate how reliably the agent would succeed across repeated executions.

For a sampled assertion with `N` verdicts, `grading.json` reports:

- each verdict in order, including its evidence and confidence
- vote counts and the pass proportion `p = passed / N`
- `pass_power_k = p^N`, the estimated probability that all `N` judgments pass under an
independent-draw assumption

For example, 6 / 10 passing verdicts produce a vote proportion of `0.6` and pass^k of
`0.6^10`, approximately `0.006047`. This is a stricter judge-consistency endpoint than majority
vote. It is not a statistical significance test. Anthropic's
[eval overview](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) explains
the pass^k interpretation in the broader agent-evaluation context. Correlated judge behavior means
`p^N` is a consistency score rather than a calibrated probability, so retain and inspect the
individual verdicts.

Multi-sample prompts and responses add `__sample-N` to the assertion id in their filenames, and
`judge-tasks.json` records `sample_index` and `sample_count`. `dispatch --judges` skips every
nonempty response independently, so rerunning fills only missing samples. During `finalize`, a
missing response fails that sample and leaves the other samples intact.

When a run mixes sampled and binary assertions, each authored assertion has equal weight in the
run summary. A binary assertion contributes either 0 or 1; a sampled assertion contributes its
vote proportion to `vote_proportion` and its `p^N` value to `pass_power_k`. `benchmark.json`
reports both endpoints by condition, their deltas, and pooled per-assertion vote counts. The run
plan prints these non-binary endpoints instead of a Fisher exact floor. Fully binary campaigns
retain the Fisher sample-size line.

An effective sample count of one preserves the binary artifact contract: the legacy response
filename, assertion-level `passed`, `evidence`, and `confidence`, binary grading summary, and
per-assertion `passed` / `n` benchmark rollup remain unchanged.

## How the bounds work

Each evidence bundle is at most 98,304 bytes (96 KiB). The complete judge prompt, including its
Expand Down
37 changes: 34 additions & 3 deletions schema/benchmark.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://slow-powers.dev/schemas/benchmark.schema.json",
"title": "Benchmark",
"description": "Output of evals:aggregate. The before/after comparison across the two conditions, with per-condition stats, per-assertion pass counts, the a-b delta, and validity warnings. Lives at <workspace>/iteration-N/benchmark.json.",
"description": "Output of evals:aggregate. The before/after comparison across the two conditions, with per-condition grading stats, per-assertion binary pass or sampled vote counts, the a-b delta, and validity warnings. Lives at <workspace>/iteration-N/benchmark.json.",
"type": "object",
"required": [
"generated",
Expand Down Expand Up @@ -39,12 +39,17 @@
},
"assertions": {
"type": "object",
"description": "Observed substantive assertion pass counts, keyed by eval id, assertion id, then condition. Omitted from historical benchmarks generated before this rollup was available.",
"description": "Observed substantive assertion results, keyed by eval id, assertion id, then condition. Binary assertions carry passed/n; sampled assertions carry pooled votes, run count, samples per run, and pass^k. Omitted from historical benchmarks generated before this rollup was available.",
"additionalProperties": {
"type": "object",
"additionalProperties": {
"type": "object",
"additionalProperties": { "$ref": "#/definitions/assertionCount" }
"additionalProperties": {
"oneOf": [
{ "$ref": "#/definitions/assertionCount" },
{ "$ref": "#/definitions/sampledAssertionCount" }
]
}
}
}
},
Expand Down Expand Up @@ -72,6 +77,8 @@
"properties": {
"direction": { "type": "string" },
"pass_rate": { "type": "number" },
"vote_proportion": { "type": "number", "description": "Condition A mean vote proportion minus condition B mean vote proportion." },
"pass_power_k": { "type": "number", "description": "Condition A mean pass^k minus condition B mean pass^k." },
"duration_ms": { "type": "number" },
"total_tokens": { "type": "number" }
}
Expand Down Expand Up @@ -175,6 +182,28 @@
}
}
},
"sampledAssertionCount": {
"type": "object",
"required": ["votes", "samples_per_run", "run_count", "pass_power_k"],
"additionalProperties": false,
"properties": {
"votes": { "$ref": "#/definitions/voteCount" },
"samples_per_run": { "type": "integer", "minimum": 2 },
"run_count": { "type": "integer", "minimum": 1 },
"pass_power_k": { "type": "number", "minimum": 0, "maximum": 1, "description": "Pooled vote proportion raised to samples_per_run." }
}
},
"voteCount": {
"type": "object",
"required": ["passed", "failed", "total", "proportion"],
"additionalProperties": false,
"properties": {
"passed": { "type": "integer", "minimum": 0 },
"failed": { "type": "integer", "minimum": 0 },
"total": { "type": "integer", "minimum": 2 },
"proportion": { "type": "number", "minimum": 0, "maximum": 1, "description": "Passed votes divided by total votes across all runs in this cell." }
}
},
"stats": {
"type": "object",
"required": ["mean", "stddev", "n"],
Expand All @@ -200,6 +229,8 @@
"additionalProperties": false,
"properties": {
"pass_rate": { "$ref": "#/definitions/stats" },
"vote_proportion": { "$ref": "#/definitions/stats", "description": "Per-run equal-weight mean of substantive assertion vote proportions; present when the campaign contains sampled grading." },
"pass_power_k": { "$ref": "#/definitions/stats", "description": "Per-run equal-weight mean of substantive assertion pass^k values; present when the campaign contains sampled grading." },
"duration_ms": { "$ref": "#/definitions/stats" },
"total_tokens": { "$ref": "#/definitions/stats" },
"skill_invocation_n": { "type": "integer" },
Expand Down
5 changes: 5 additions & 0 deletions schema/evals.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,11 @@
"model": {
"type": "string",
"description": "Optional judge model override. When absent, defaults to the run-level judge model recorded in conditions.json, or the harness default when no run-level model was selected."
},
"samples": {
"type": "integer",
"minimum": 1,
"description": "Independent judge verdicts requested for this assertion. Overrides run --judge-samples and defaults to 1."
}
}
},
Expand Down
154 changes: 104 additions & 50 deletions schema/grading.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,75 +9,129 @@
"properties": {
"assertion_results": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "passed", "evidence"],
"additionalProperties": false,
"properties": {
"id": {
"type": "string",
"description": "Matches the assertion id in evals.json."
},
"passed": { "type": "boolean" },
"evidence": {
"type": "string",
"description": "Direct quote or specific reference from the run record. Vague summaries are not evidence."
},
"confidence": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Judge confidence. Low confidence (< 0.7) flags this result for human review. Always 1.0 for transcript_check results."
},
"grader": {
"type": "string",
"enum": ["transcript_check", "llm_judge", "command_check", "diff_scope"],
"description": "Which grader produced this result."
}
}
}
"items": { "$ref": "#/definitions/assertionResult" }
},
"summary": {
"$ref": "#/definitions/gradingSummary"
},
"meta_results": {
"type": "array",
"description": "Framework-injected meta-assertions (e.g. skill-invocation check). Reserved id prefix: __ (double underscore). Tracked separately from substantive assertion_results so they do not pollute the skill effectiveness pass_rate.",
"items": { "$ref": "#/definitions/binaryAssertionResult" }
},
"meta_summary": {
"type": "object",
"required": ["passed", "failed", "total", "pass_rate"],
"additionalProperties": false,
"properties": {
"passed": { "type": "integer", "minimum": 0 },
"failed": { "type": "integer", "minimum": 0 },
"total": { "type": "integer", "minimum": 0 },
"pass_rate": { "type": "number", "minimum": 0, "maximum": 1 }
"skill_invoked": {
"description": "True when the skill-invocation meta-check passed; false when the judge found no evidence the skill influenced behavior; null when no skill was loaded for this run.",
"type": ["boolean", "null"]
}
}
}
},
"definitions": {
"assertionResult": {
"oneOf": [
{ "$ref": "#/definitions/binaryAssertionResult" },
{ "$ref": "#/definitions/sampledAssertionResult" }
]
},
"meta_results": {
"type": "array",
"description": "Framework-injected meta-assertions (e.g. skill-invocation check). Reserved id prefix: __ (double underscore). Tracked separately from substantive assertion_results so they do not pollute the skill effectiveness pass_rate.",
"items": {
"type": "object",
"required": ["id", "passed", "evidence"],
"additionalProperties": false,
"properties": {
"id": { "type": "string" },
"passed": { "type": "boolean" },
"evidence": { "type": "string" },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 },
"grader": {
"type": "string",
"enum": ["transcript_check", "llm_judge", "command_check", "diff_scope"]
}
"binaryAssertionResult": {
"type": "object",
"required": ["id", "passed", "evidence"],
"additionalProperties": false,
"properties": {
"id": {
"type": "string",
"description": "Matches the assertion id in evals.json."
},
"passed": { "type": "boolean" },
"evidence": {
"type": "string",
"description": "Direct quote or specific reference from the run record. Vague summaries are not evidence."
},
"confidence": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "Judge confidence. Low confidence (< 0.7) flags this result for human review. Always 1.0 for transcript_check results."
},
"grader": {
"type": "string",
"enum": ["transcript_check", "llm_judge", "command_check", "diff_scope"],
"description": "Which grader produced this result."
}
}
},
"meta_summary": {
"sampledAssertionResult": {
"type": "object",
"required": ["id", "grader", "votes", "judge_samples"],
"additionalProperties": false,
"properties": {
"id": { "type": "string", "description": "Matches the authored llm_judge assertion id in evals.json." },
"grader": { "const": "llm_judge", "description": "Sampled results are produced only by authored LLM-judge assertions." },
"votes": { "$ref": "#/definitions/judgeVotes" },
"judge_samples": {
"type": "array",
"minItems": 2,
"description": "Every requested verdict in sample-index order. A missing response is retained as a failed sample rather than failing the whole assertion.",
"items": { "$ref": "#/definitions/judgeSample" }
}
}
},
"judgeVotes": {
"type": "object",
"required": ["passed", "failed", "total", "proportion", "pass_power_k"],
"additionalProperties": false,
"properties": {
"passed": { "type": "integer", "minimum": 0 },
"failed": { "type": "integer", "minimum": 0 },
"total": { "type": "integer", "minimum": 2 },
"proportion": { "type": "number", "minimum": 0, "maximum": 1, "description": "Passed divided by total." },
"pass_power_k": { "type": "number", "minimum": 0, "maximum": 1, "description": "proportion raised to total: the estimated probability every requested judgment passes." }
}
},
"judgeSample": {
"type": "object",
"required": ["sample_index", "passed", "evidence", "confidence"],
"additionalProperties": false,
"properties": {
"sample_index": { "type": "integer", "minimum": 1 },
"passed": { "type": "boolean" },
"evidence": { "type": "string" },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 }
}
},
"gradingSummary": {
"oneOf": [
{ "$ref": "#/definitions/binaryGradingSummary" },
{ "$ref": "#/definitions/sampledGradingSummary" }
]
},
"binaryGradingSummary": {
"type": "object",
"required": ["passed", "failed", "total", "pass_rate"],
"additionalProperties": false,
"properties": {
"passed": { "type": "integer", "minimum": 0 },
"failed": { "type": "integer", "minimum": 0 },
"total": { "type": "integer", "minimum": 0 },
"skill_invoked": {
"description": "True when the skill-invocation meta-check passed; false when the judge found no evidence the skill influenced behavior; null when no skill was loaded for this run.",
"type": ["boolean", "null"]
}
"pass_rate": { "type": "number", "minimum": 0, "maximum": 1 }
}
},
"sampledGradingSummary": {
"type": "object",
"required": ["total", "pass_rate", "vote_proportion", "pass_power_k"],
"additionalProperties": false,
"properties": {
"total": { "type": "integer", "minimum": 0 },
"pass_rate": { "type": "number", "minimum": 0, "maximum": 1, "description": "Compatibility alias for vote_proportion in a sampled grading." },
"vote_proportion": { "type": "number", "minimum": 0, "maximum": 1, "description": "Equal-weight mean of each substantive assertion's binary result or sampled vote proportion." },
"pass_power_k": { "type": "number", "minimum": 0, "maximum": 1, "description": "Equal-weight mean of each substantive assertion's binary result or sampled pass^k value." }
}
}
}
Expand Down
14 changes: 14 additions & 0 deletions schema/judge-tasks.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@
"dispatch_prompt_bytes",
"dispatch_prompt_byte_limit"
],
"dependencies": {
"sample_index": ["sample_count"],
"sample_count": ["sample_index"]
},
"additionalProperties": false,
"properties": {
"eval_id": { "type": "string" },
Expand All @@ -50,6 +54,16 @@
"description": "1-based run index within a multi-run (eval, condition) cell; absent for single-run cells."
},
"assertion_id": { "type": "string" },
"sample_index": {
"type": "integer",
"minimum": 1,
"description": "1-based verdict index for an assertion requesting more than one judge sample; absent for the legacy single-verdict shape."
},
"sample_count": {
"type": "integer",
"minimum": 2,
"description": "Total verdicts requested for this sampled assertion; absent together with sample_index when the effective count is 1."
},
"rubric": { "type": "string" },
"model": {
"type": ["string", "null"],
Expand Down
Loading
Loading