Skip to content

feat(judging): add bounded evidence bundles - #288

Merged
slowdini merged 1 commit into
devfrom
feat/bounded-judge-evidence
Aug 24, 2026
Merged

feat(judging): add bounded evidence bundles#288
slowdini merged 1 commit into
devfrom
feat/bounded-judge-evidence

Conversation

@slowdini

Copy link
Copy Markdown
Owner

Closes #261
Part of #244

Why

Realistic codebase evals produce long transcripts and large tool payloads. The prior judge prompt serialized the entire run.json into every task while exposing the implementation only through an outputs-directory listing. That made prompt size unbounded and left code-quality rubrics without the code changes they need to judge.

This change makes a persisted, diff-aware evidence bundle the shared grading input for every LLM assertion on a run.

Strategy

  • Render one judge-evidence.md beside every recorded run before creating assertion tasks.
  • Include task context, completion state, source provenance, artifact paths, final_message, changed-file metrics and list, diff.patch, the multi-turn conversation, and a trimmed tool summary.
  • Persist the bundle once and inline those exact bytes into every judge prompt for the run.
  • Record bundle and complete-prompt byte accounting in judge-tasks.json.
  • Keep the deterministic __skill_invoked check on the full transcript; only its LLM fallback uses the bounded view.

The evidence bundle is capped at 98,304 bytes (96 KiB), and the complete judge prompt is capped at 131,072 bytes (128 KiB). Prompt, final message, changed files, conversation events, and tool fields have fixed budgets; the patch receives the remaining bundle budget. Oversized content keeps UTF-8-safe head and tail excerpts with visible source-aware markers. Collision-safe fences prevent embedded Markdown from escaping a section.

Eval-authored rubrics and skill content are never silently shortened. Emission fails with the assertion ID, actual size, cap, and remediation when authored content exceeds the remaining prompt budget.

Evidence and safety boundary

Judge framing treats the task, transcript, final message, patch, and tool output as untrusted data. Judges may inspect named source files read-only when a truncation marker omits material required by the rubric, but may write only the requested verdict.

Held-out command_check results stay out of the bundle. Diff capture precedes command-check setup-file injection, so including those results would mix runner-owned post-capture mutations into the agent evidence. Mechanical checks remain runner-owned and merge during finalize.

Compatibility and retention

  • Existing eval definitions and final-message rubrics require no configuration changes.
  • One-shot runs explicitly report that no conversation is available; missing diff evidence is explicit rather than presented as an empty successful change.
  • New-skill and revision modes use the same bundle path; integration coverage exercises both revision arms.
  • promote-baseline retains exact bounded bundles under evals/baseline/evidence/, using filenames that match retained gradings.
  • Compatible legacy iterations without bundles still promote with a warning. Promotion removes a stale prior bundle rather than associating it with a grading it never informed.
  • Full transcripts, patches, outputs, and task environments remain iteration-local; the bounded primary judge input is the durable artifact.

Before and after

Before After
Every judge prompt embedded the complete run.json, including unbounded tool arguments and results. Every run has one size-accounted judge-evidence.md, shared byte-for-byte by its judge tasks.
Judges saw only an outputs-directory listing for the implementation. Judges receive the changed-file list and patch directly, with paths to complete sources when truncation matters.

Schema and documentation

judge-tasks.json tasks add required evidence metadata and complete-prompt byte accounting. The shipped eval-magic docs judging guide documents contents, caps, truncation behavior, source inspection, and baseline retention. CLI help, the codebase guide, developer overview, generated runbooks, and goldens point to the same contract.

Verification

  • cargo fmt --check
  • cargo build
  • cargo test — 1,288 tests passed
  • cargo clippy --all-targets -- -D warnings
  • git diff --check

The pre-commit hook reran formatting and clippy; the pre-push hook reran the full test suite.

Review focus

  • Whether the fixed section budgets and patch-remainder policy provide the right evidence balance.
  • Whether the untrusted/read-only prompt boundary is sufficiently explicit.
  • Whether exact bounded-bundle retention is the right durable baseline artifact while unbounded sources remain iteration-local.

Persist one diff-aware evidence bundle per run and inline its exact
bounded contents into every LLM judge prompt.

Retain bundles during baseline promotion so reviewable evidence survives
teardown.
@slowdini
slowdini merged commit 6707dec into dev Aug 24, 2026
7 checks passed
@slowdini
slowdini deleted the feat/bounded-judge-evidence branch August 24, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bounded, diff-aware judge evidence bundle

1 participant