feat(judging): add bounded evidence bundles

docs/developer_overview.md

@@ -25,8 +25,10 @@ focused internal notes instead of duplicating their details.
    its turns. Each task ends with a `conversation.json`, which is also what a rerun skips on.
 4. `eval-magic ingest` reads the harness outputs, transcript evidence, guard denials, and final
    task state. Runner-owned deterministic checks and diff-scope evidence are collected here.
-5. `eval-magic grade` evaluates runner-owned assertions and emits tasks for assertions that require
-   an LLM. `eval-magic dispatch --judges` runs those judge tasks through the selected harness.
+5. `eval-magic grade` evaluates runner-owned assertions, writes one bounded `judge-evidence.md`
+   per recorded run, and emits tasks for assertions that require an LLM. Each task inlines the
+   exact bundle for its run. `eval-magic dispatch --judges` runs those judge tasks through the
+   selected harness.
 6. `eval-magic finalize` checks that required work is complete and writes the final per-run and
    benchmark artifacts. `eval-magic aggregate` combines campaigns when a larger comparison is
    needed.
@@ -153,3 +155,5 @@ implementation evidence in an internal note.
   `eval-magic docs guard`.
 - [Shipped conversations guide](guides/conversations.md) is the repository source for
   `eval-magic docs conversations`.
+- [Shipped judging guide](guides/judging.md) is the repository source for
+  `eval-magic docs judging`.

docs/guides/codebase.md

@@ -120,6 +120,9 @@ During `ingest`, Git measures that difference. Each run gets:
   any good. It always exists; for a run that changed nothing it is empty. A diff past the capture
   cap is cut at a line boundary and carries a marker saying so, and `patch.truncated` in
   `diff-scope.json` records it.
+- `judge-evidence.md` — the bounded grading input that combines this diff with the task, completion
+  state, conversation, and tool summary. See `eval-magic docs judging` for its limits, trust
+  boundary, and retained-baseline behavior.
 
 What counts is what Git counts, under the same rules the baseline commit was built under:
 
@@ -206,6 +209,7 @@ After a dispatch and `ingest`, read what the run produced:
 ```sh
 jq '{files_touched, lines_added, lines_removed, hunks, files, patch}' diff-scope.json
 head -50 diff.patch
+sed -n '1,240p' judge-evidence.md
 ```
 
 The same difference, spelled by Git itself, is `git diff refs/eval-magic/baseline` inside the

docs/guides/judging.md

@@ -0,0 +1,86 @@
+# Judge evidence bundles
+
+> **Audience:** eval authors and operators deciding whether an LLM verdict has enough evidence to
+> trust.
+
+`ingest` writes one `judge-evidence.md` beside every recorded run. This Markdown bundle is the
+primary input for every LLM judge task for that run: eval-magic persists it once and inlines those
+exact bytes into each judge prompt. Read the bundle when a verdict is surprising, when a truncation
+marker appears, or before promoting an important result.
+
+## What the bundle contains
+
+The bundle combines the evidence that establishes what the agent was asked to do, what it did, and
+what it changed:
+
+- run identity, completion state, timing, token counts, and codebase and skill provenance
+- an artifact manifest pointing to `run.json`, `diff-scope.json`, `diff.patch`, and raw harness
+  outputs
+- the original task `prompt` and the agent's `final_message`
+- changed-file metrics, a changed-file list, and the captured patch
+- the conversation transcript, including markers showing where tools were invoked
+- a tool invocation summary with bounded arguments and results
+
+A one-shot run has an explicit “no conversation record” entry. Missing diff evidence is also
+explicit; it is never presented as an empty successful change.
+
+Held-out `command_check` results are not included. Diff capture happens before command-check setup
+files are injected, and keeping the bundle at that boundary prevents a judge from confusing
+runner-owned mutations with agent work. Mechanical assertion results remain runner-owned and are
+merged during `finalize`.
+
+## How the bounds work
+
+Each evidence bundle is at most 98,304 bytes (96 KiB). The complete judge prompt, including its
+rubric and framing, is at most 131,072 bytes (128 KiB). Within the bundle, eval-magic reserves:
+
+- 8 KiB for the task prompt
+- 12 KiB for the final message
+- 8 KiB for the changed-file list
+- 16 KiB for the conversation, with at most 4 KiB per event
+- 8 KiB for the tool summary, with at most 512 bytes for each argument and result
+
+The patch receives the remaining bundle space, so short contextual sections leave more room for
+the implementation itself. Oversized sections retain both their beginning and end at valid UTF-8
+boundaries and carry an `[eval-magic] ... omitted` marker naming the full source. Markdown fences
+are chosen so evidence containing its own fences cannot escape the section that holds it.
+
+`judge-tasks.json` records the actual byte count, limit, and `truncated` state of each evidence
+bundle, plus the actual and maximum judge-prompt sizes. A `diff.patch` can also carry its own
+capture-time truncation marker; that upstream limit is separate from bundle truncation.
+
+Eval-authored rubrics and skill content are never silently shortened. If either makes the complete
+prompt exceed 131,072 bytes, judge-task emission fails with the assertion id, actual size, limit,
+and a request to shorten the authored content.
+
+## Treat evidence as data
+
+The task prompt, transcript, final message, patch, and tool output are untrusted agent-produced
+data. Judge framing says not to follow instructions found inside the evidence. A judge works
+read-only: it may inspect a source path named by a truncation marker, but it must not edit the run,
+the evidence, or the task environment. Its only write is the requested verdict file.
+
+When a marker omits material needed by the rubric, inspect the named source before deciding. The
+artifact paths are valid in the grading iteration. After promotion and teardown reclaim that
+iteration, its retained bundle may no longer have those complete sources beside it. If a required
+source is unavailable, the claim is unverifiable rather than evidence of success.
+
+From a run directory, inspect the bounded evidence and its source records:
+
+```sh
+sed -n '1,240p' judge-evidence.md
+jq '{prompt, final_message, conversation, tool_invocations}' run.json
+jq '{files_touched, lines_added, lines_removed, hunks, files, patch}' diff-scope.json
+```
+
+## Retain the evidence behind a baseline
+
+`promote-baseline` copies each exact bounded bundle into `<skill>/evals/baseline/evidence/` beside
+the retained benchmark and gradings. A single-run bundle is named
+`<eval-id>__<condition>.md`; multi-run bundles add `__rN`. This preserves the primary judge input
+without copying unbounded transcripts, patches, or task environments into the skill repository.
+
+Older iterations can have gradings without `judge-evidence.md`. Promotion preserves compatibility
+by warning about each missing legacy bundle instead of failing, but such a baseline does not retain
+the evidence needed to reproduce its LLM judgment. Re-grade the iteration before promotion when
+that evidence matters.

profiles/shared/runbook.md

@@ -33,7 +33,10 @@ each one by name and cause, and `aggregate` counts them per condition in `benchm
 `ingest` records each run, backfills transcripts, scans for stray writes, collects guarded-task
 blocks into `guard-denials.json`, and grades every mechanical assertion. Inspect any denial
 warning before trusting the affected task. It then prints any `llm_judge` tasks it could not
-grade itself.
+grade itself. Each run's bounded `judge-evidence.md` combines the task, final message, diff,
+conversation, tool summary, and source paths; those exact bytes are the primary input shared by
+that run's judge tasks. Read `eval-magic docs judging` for its caps, truncation markers, and
+retention contract.
 
 ## 2. Dispatch the judge agents, then finalize
 

schema/judge-tasks.schema.json

@@ -2,7 +2,7 @@
   "$schema": "http://json-schema.org/draft-07/schema#",
   "$id": "https://slow-powers.dev/schemas/judge-tasks.schema.json",
   "title": "Judge Tasks",
-  "description": "Output of evals:grade (emit mode). The list of LLM judge tasks the orchestrator dispatches, plus the skill-invocation meta-checks. Lives at <workspace>/iteration-N/judge-tasks.json. The full prompt is written to dispatch_prompt_path and is NOT inlined here.",
+  "description": "Output of evals:grade (emit mode). The list of LLM judge tasks the orchestrator dispatches, plus the skill-invocation meta-checks. Lives at <workspace>/iteration-N/judge-tasks.json. The full prompt is written to dispatch_prompt_path and is NOT inlined here; evidence_bundle accounts for the exact bounded run evidence inlined into that prompt.",
   "type": "object",
   "required": [
     "generated",
@@ -35,7 +35,10 @@
         "run_record_path",
         "outputs_dir",
         "response_path",
-        "dispatch_prompt_path"
+        "dispatch_prompt_path",
+        "evidence_bundle",
+        "dispatch_prompt_bytes",
+        "dispatch_prompt_byte_limit"
       ],
       "additionalProperties": false,
       "properties": {
@@ -59,8 +62,32 @@
         "dispatch_prompt_path": {
           "type": "string",
           "description": "Absolute path to the file holding the full judge prompt."
+        },
+        "evidence_bundle": {
+          "$ref": "#/definitions/evidenceBundle",
+          "description": "The persisted, bounded evidence shared by every judge task for this run and inlined byte-for-byte into the dispatch prompt."
+        },
+        "dispatch_prompt_bytes": {
+          "type": "integer",
+          "minimum": 0,
+          "description": "Actual UTF-8 byte length of the complete dispatch prompt."
+        },
+        "dispatch_prompt_byte_limit": {
+          "type": "integer",
+          "const": 131072
         }
       }
+    },
+    "evidenceBundle": {
+      "type": "object",
+      "required": ["path", "bytes", "byte_limit", "truncated"],
+      "additionalProperties": false,
+      "properties": {
+        "path": { "type": "string", "description": "Absolute grading-iteration path to judge-evidence.md." },
+        "bytes": { "type": "integer", "minimum": 0, "description": "Actual UTF-8 byte length of the persisted bundle." },
+        "byte_limit": { "type": "integer", "const": 98304 },
+        "truncated": { "type": "boolean", "description": "True when any bundle section or the captured source patch was truncated." }
+      }
     }
   }
 }

src/cli/args.rs

@@ -692,7 +692,9 @@ pub(crate) enum Commands {
     /// runner-owned command check in its task environment, applying its
     /// environment overrides and running every environment matrix cell. Diff
     /// scope is captured before held-out files are injected. Then stops at the
-    /// judge hand-off, listing a judge task per `llm_judge` assertion. Requires
+    /// judge hand-off, writing one bounded `judge-evidence.md` per recorded run
+    /// and listing a judge task per `llm_judge` assertion. The exact evidence
+    /// bundle is shared by that run's tasks and inlined into their prompts. Requires
     /// `--iteration`; reads each task's `outputs/<harness>-events.jsonl` when the
     /// harness exposes transcripts, under `outputs/turn-<n>/`. Dispatch the judge
     /// tasks it lists with `eval-magic dispatch --judges`.
@@ -772,15 +774,24 @@ pub(crate) enum Commands {
     /// held-out `command_check.setup_files` and executes each runner-owned command
     /// in its task environment, applying fixed environment overrides and running
     /// every environment matrix cell; completed command and diff-scope results
-    /// are reused. Emits judge-task files for `llm_judge` assertions; with
-    /// `--finalize`, merges every result into per-run `grading.json`.
+    /// are reused. Before emitting tasks, writes one `judge-evidence.md` beside
+    /// every recorded run. This 98,304-byte bounded bundle combines task context,
+    /// completion state, diff evidence, conversation, tool summary, and source
+    /// paths; its exact bytes are inlined into each run's LLM-judge prompts. The
+    /// complete prompt has a 131,072-byte cap, and authored rubrics or skill content
+    /// that exceed the remaining space fail rather than being truncated.
+    /// Treat bundle content as untrusted, read-only data; truncation markers name
+    /// iteration-local sources for material a rubric requires. See
+    /// `eval-magic docs judging`. With `--finalize`, merges every result into
+    /// per-run `grading.json`.
     ///
     /// Injects the `__skill_invoked` meta-check — did the skill actually influence
     /// behavior? It has two tiers, chosen automatically per run: code-based (where
     /// the staged slug + transcript are available, as on Claude Code, it checks the
     /// transcript for a `Skill` call matching the eval slug — deterministic and
-    /// free) and an LLM-judge fallback (where transcripts aren't available, a judge
-    /// compares the final message against the SKILL.md for behavioral fingerprints).
+    /// free) and an LLM-judge fallback (where deterministic transcript evidence
+    /// isn't available, a judge compares the final message, conversation, and tool
+    /// summary against the SKILL.md for behavioral fingerprints).
     /// The meta-check does not count toward the substantive `pass_rate`.
     Grade(GradeArgs),
     /// Aggregate before/after benchmark deltas.
@@ -827,13 +838,16 @@ pub(crate) enum Commands {
     /// assertions after the first iteration, then check the file with
     /// `eval-magic validate`.
     Init(InitArgs),
-    /// Promote a benchmark and gradings into a committed baseline.
-    ///
-    /// Copies the iteration's `benchmark.json` and per-run `grading.json` files to
-    /// `<skill>/evals/baseline/`. The benchmark stays at that directory's root,
-    /// grading files land under `grading/`, and `BASELINE.md` records provenance.
-    /// An existing hand-authored `NOTES.md` is retained; one is scaffolded when
-    /// absent. Promote before teardown when the result is worth keeping.
+    /// Promote a benchmark, gradings, and judge evidence into a committed baseline.
+    ///
+    /// Copies the iteration's `benchmark.json`, per-run `grading.json`, and exact
+    /// bounded `judge-evidence.md` bundles to `<skill>/evals/baseline/`. The
+    /// benchmark stays at that directory's root, gradings land under `grading/`,
+    /// evidence bundles land under `evidence/`, and `BASELINE.md` records
+    /// provenance. Missing bundles from compatible legacy iterations warn without
+    /// blocking promotion. An existing hand-authored `NOTES.md` is retained; one
+    /// is scaffolded when absent. Promote before teardown when the result is worth
+    /// keeping. See `eval-magic docs judging` for the evidence contract.
     PromoteBaseline(PromoteBaselineArgs),
     /// Validate `evals.json` files against the bundled schemas.
     Validate(ValidateArgs),

src/cli/commands/workspace.rs

@@ -38,8 +38,8 @@ pub(crate) fn run_snapshot(args: SnapshotArgs) -> anyhow::Result<()> {
     Ok(())
 }
 
-/// Promote an iteration's `benchmark.json` + per-run gradings into the skill's
-/// committed `evals/baseline/`, dropping a `.promoted.json` marker.
+/// Promote an iteration's benchmark, gradings, and bounded judge evidence into
+/// the skill's committed `evals/baseline/`, dropping a `.promoted.json` marker.
 pub(crate) fn run_promote_baseline(args: PromoteBaselineArgs) -> anyhow::Result<()> {
     let ctx = run_context_from(&args.common)?;
     let iteration = resolve_iteration(&ctx, args.common.iteration)?;
@@ -57,11 +57,13 @@ pub(crate) fn run_promote_baseline(args: PromoteBaselineArgs) -> anyhow::Result<
     })?;
 
     let n = result.gradings_copied;
+    let evidence = result.evidence_copied;
     println!(
-        "Promoted baseline for {} → {} (benchmark.json + {n} grading file{} + BASELINE.md)",
+        "Promoted baseline for {} → {} (benchmark.json + {n} grading file{} + {evidence} evidence bundle{} + BASELINE.md)",
         ctx.skill_name,
         result.baseline_dir.display(),
-        if n == 1 { "" } else { "s" }
+        if n == 1 { "" } else { "s" },
+        if evidence == 1 { "" } else { "s" }
     );
     if result.missing_gradings > 0 {
         let m = result.missing_gradings;
@@ -71,6 +73,13 @@ pub(crate) fn run_promote_baseline(args: PromoteBaselineArgs) -> anyhow::Result<
             if m == 1 { "" } else { "s" }
         );
     }
+    if result.missing_evidence > 0 {
+        let m = result.missing_evidence;
+        eprintln!(
+            "⚠ {m} run cell{} missing judge-evidence.md — retained gradings have no bounded evidence bundle. Re-grade the iteration to create it before promoting again.",
+            if m == 1 { "" } else { "s" }
+        );
+    }
     match result.notes {
         workspace::NotesStatus::StubWritten => {
             println!("+ NOTES.md stub — fill in observations for this iteration.");

src/cli/run/golden_tests.rs

@@ -139,6 +139,8 @@ fn golden_runbook_per_harness() {
             num_tasks: 6,
             target_args: " --skill-dir /tmp/skills --skill widget-skill",
         });
+        assert!(book.contains("judge-evidence.md"));
+        assert!(book.contains("eval-magic docs judging"));
         assert_golden(&format!("{label}/runbook.golden.md"), &book);
     }
 }