feat(judging): add bounded evidence bundles - #288
Merged
Conversation
Persist one diff-aware evidence bundle per run and inline its exact bounded contents into every LLM judge prompt. Retain bundles during baseline promotion so reviewable evidence survives teardown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #261
Part of #244
Why
Realistic codebase evals produce long transcripts and large tool payloads. The prior judge prompt serialized the entire
run.jsoninto every task while exposing the implementation only through an outputs-directory listing. That made prompt size unbounded and left code-quality rubrics without the code changes they need to judge.This change makes a persisted, diff-aware evidence bundle the shared grading input for every LLM assertion on a run.
Strategy
judge-evidence.mdbeside every recorded run before creating assertion tasks.final_message, changed-file metrics and list,diff.patch, the multi-turn conversation, and a trimmed tool summary.judge-tasks.json.__skill_invokedcheck on the full transcript; only its LLM fallback uses the bounded view.The evidence bundle is capped at 98,304 bytes (96 KiB), and the complete judge prompt is capped at 131,072 bytes (128 KiB). Prompt, final message, changed files, conversation events, and tool fields have fixed budgets; the patch receives the remaining bundle budget. Oversized content keeps UTF-8-safe head and tail excerpts with visible source-aware markers. Collision-safe fences prevent embedded Markdown from escaping a section.
Eval-authored rubrics and skill content are never silently shortened. Emission fails with the assertion ID, actual size, cap, and remediation when authored content exceeds the remaining prompt budget.
Evidence and safety boundary
Judge framing treats the task, transcript, final message, patch, and tool output as untrusted data. Judges may inspect named source files read-only when a truncation marker omits material required by the rubric, but may write only the requested verdict.
Held-out
command_checkresults stay out of the bundle. Diff capture precedes command-check setup-file injection, so including those results would mix runner-owned post-capture mutations into the agent evidence. Mechanical checks remain runner-owned and merge duringfinalize.Compatibility and retention
promote-baselineretains exact bounded bundles underevals/baseline/evidence/, using filenames that match retained gradings.Before and after
run.json, including unbounded tool arguments and results.judge-evidence.md, shared byte-for-byte by its judge tasks.Schema and documentation
judge-tasks.jsontasks add required evidence metadata and complete-prompt byte accounting. The shippedeval-magic docs judgingguide documents contents, caps, truncation behavior, source inspection, and baseline retention. CLI help, the codebase guide, developer overview, generated runbooks, and goldens point to the same contract.Verification
cargo fmt --checkcargo buildcargo test— 1,288 tests passedcargo clippy --all-targets -- -D warningsgit diff --checkThe pre-commit hook reran formatting and clippy; the pre-push hook reran the full test suite.
Review focus