More realistic eval environment - full codebase testbed, multi-skill workflows

[slowdini]

The main usage of eval-magic so far has been in our companion project, slow-powers, where we're attempting to create useful, publishable data about the value of the skills the plugin ships. This has proven to be very difficult, with a recent run using the lowest tier of model power (codex luna) on an eval case based on a bug encountered in a real-world scenario, ceiling on all cases. I believe this is because we simply cannot effectively simulate the complexity of a real task, done in a real codebase. This update seeks to avoid that weakness by simply providing an actual, real code environment, and supporting prompts that involve an agent executing real tasks against the codebase.

Proposal

We already create isolated environments for eval runs. Instead of populating them with small fixtures, they should be initialized with a full codebase, one custom-designed for the eval run, or a project of a reasonable type and complexity. Instead of doing what slow-powers is currently trying, and giving agents fragmentary "transcripts" of ongoing sessions, with the expectation of essentially testing a single turn, we should expect an eval run to start with a realistic prompt, potentially have a dynamic number of turns, and end with the task fully done.

There are a number of benefits to this approach, beyond just the fact the we're now running a real task.

Robust judging

While some mechanical judging is useful (and cheap), it often fails to capture the essence of what we're really trying to test, with the kind of behavior that's most interesting to test often being the hardest to judge. I'll walk through an explanatory example. A hypothetical skill like create-pr might contain instructions along the lines of: check if there's a PR template to follow, use a default if not, always wait for user approval, etc. Most of these would be easy to write eval cases for, and we could reasonably expect those cases to succeed - an agent without the skill has no reason to wait for final user approval, might not know to check for templates, and so on. This is also the least interesting behavior to test, often bordering on completely deterministic and trustworthy on sight.

Such a skill might also contain an instruction like: write a clear, concise PR description. This is much more interesting to test, and still easy to test reliably - a simulated "create a PR" state that matches the kind of case an agent might encounter in the wild is possible, even without a full codebase and a real PR to create. Running that case a number of times and comparing the with and without skill results should be expected to determine if anything in the "clear and concise description" instructions hold their weight, or if the agent would write the same kind of description without any instruction.

A skill like slow-powers:working-with-tdd also contains instructions that are relatively deterministic and trustable: write the test first, it must fail, and so on. (Pressure scenarios - did the agent follow its clear instructions under difficult or unexpected circumstances - are a different type of eval, not relevant to this discussion). The interesting thing to test, however, is actually: is the code itself better when following TDD, than without (better test coverage, code organization, test clarity, etc)? Under a limited, simulated environment, such a test can't be verified. Even haiku is a powerful coding model, and it's not going to have any trouble working through a toy problem, and producing reasonable code.

A realistic environment gives us the circumstances and output artifacts (that is, the code changes) that a judge needs to answer important performance questions that can't reasonably be simulated.

Multiple judges

With a question like "is the code better?", we'd likely see more value by running the test once, and judging it multiple times. The more "opinionated" a question is, the more useful this reverse grading seems. If 6/10 judges grade no-skill work as "good", and 10/10 judges grade with-skill work as "good", we'd count that as marked improvement (see Reference 1, "pass^k" grading). This should be much cheaper, since judging is far simpler than executing the task, and we have a clear pattern for how to judge success for this type of question.

Interactive judging

This also allows the agent driving eval-magic itself to do heuristic judging, without actual eval cases known ahead of time. For instance, if we have a test case like "implement this feature...", the user can ask the agent to have eval-magic run both the no-skill and with-skill sides of the test, then ask vague questions like "did the skill make the code better?" or "did the skill confuse the agent at any point?". This won't be statistically reliable information, but it can lead to the discovery of worthwhile test cases to run real evals for, and give the user an early sense of how the skill is working.

This generally seems like an interesting way to begin drafting actual eval cases, with two comparable session transcripts, given the exact same starting prompt, providing straightforward insight into the function of the skill under test.

Multiple skills tested at once

This is specifically helpful to slow-powers, but should also be generally useful. With a real test case, we could see the full suite of slow-powers skills working together, with the final code being a useful product by which to judge the plugin as a whole. This feels especially useful combined with interactive judging to come up with eval cases for the most useful aspects of the suite.

This is also likely to be a way to catch places where skills conflict or cause confusion, something that is again unlikely to come up in a toy test case.

Issues and considerations

I can see a few things that need deeper investigation, or seem likely to be a source of issues.

Legacy maintenance

My feeling is this will present a number of breaking changes, and that there's no reason to support legacy modes of evals. If we can architecturally handle all our evals in essentially the same way, we should aim for that goal. The simplicity will be a big benefit in the long run.

Sourcing codebases

I think this should be a simple part of the eval configuration, with the option to use a local directory or an online git repo. This is an opportunity to provide some nice UX. We should provide a default fixture for a simple codebase (maybe just by creating it on github), and suggest using the eval-magic codebase itself as a complex fixture. Instructions for what fixtures should be used should be part of the committed eval files in some way, and recorded as part of any run report.

Guard updates

We may need to consider the details of the what should be guarded, and what should be allowed, during this kind of eval run. I'm not sure the scope of this change, but opportunities to simplify the guard are always good.

Multiple runs

While the "one run, many judges" approach is nice, we still need to support statistically significant eval runs, which will likely require running the actual task multiple times. We don't want to check out a dozen copies of a codebase to run the eval though. We might be able to make use of git for this, but we should keep in mind that git actions can be an expected part of an eval itself.

Multiple turns

As stated, we want to support an unknown, dynamic number of turns. I think there's a convenient happy path for most responses an agent-under-test might expect to receive - Did the agent ask a multiple choice question, with a recommended option? Always choose the recommendation. For more complex questions, we may need some kind of "question answering agent", designed to quickly generate a comprehensible response, always tending towards "choose whatever is recommended/simplest/easiest".

Notes

• Although I'm generally referring to mode A style runs, as that's what's being tested in slow-powers, most of this should equally apply to mode B runs.
• This obviously presents a large change overall. The first goal should be splitting this into smaller tasks, and answering as many open questions as we can.

References

1. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents - We should think about our evals as essentially evaluating agents. The skill(s) under test essentially constitute an "agent", being handed a particular task.
2. https://github.com/slowdini/slow-powers

---

Breakdown

Feasibility was assessed against the current tree and the work split into 15 sub-tickets.
Verdict: feasible, no blocking unknowns. Most of what this issue needs already exists in some
form — private per-(eval, condition, run) environments, a runner-owned Git repo per env, a working
multi-turn driver with native session resume, deterministic + LLM grading, guard and stray-write
accounting, cross-campaign aggregation. The work is reshaping those, not building new subsystems.

Sub-tickets

Foundation

• Source a real codebase into each task environment #252 — Source a real codebase into each task environment (start here)
• Source skills as copied inputs and relocate the eval home #253 — Source skills as copied inputs and relocate the eval home (needs Source a real codebase into each task environment #252)
• Provision environments from a per-iteration codebase cache #254 — Provision environments from a per-iteration codebase cache
• Measure and capture diffs with Git instead of copying a baseline #255 — Measure and capture diffs with Git instead of copying a baseline (needs Source a real codebase into each task environment #252)

Execution

• Runner-driven dispatch for every task #256 — Runner-driven dispatch for every task
• Dynamic conversation turns with a heuristic responder #257 — Dynamic conversation turns with a heuristic responder (needs Runner-driven dispatch for every task #256)
• LLM answering agent for free-form questions #258 — LLM answering agent for free-form questions (needs Dynamic conversation turns with a heuristic responder #257)

Environment fidelity

• Write-guard policy for real development work #259 — Write-guard policy for real development work
• Harness config shipped by a sourced codebase #260 — Harness config shipped by a sourced codebase

Judging

• Bounded, diff-aware judge evidence bundle #261 — Bounded, diff-aware judge evidence bundle (needs Measure and capture diffs with Git instead of copying a baseline #255)
• Multi-sample judging (pass^k) #262 — Multi-sample judging (pass^k)
• compare: paired evidence for interactive judging #263 — compare: paired evidence for interactive judging

Breadth

• Skill sets as the treatment (multi-skill evals) #264 — Skill sets as the treatment (multi-skill evals)

Onboarding and cleanup

• Default codebase fixtures and shipped docs #265 — Default codebase fixtures and shipped docs
• Retire the fixture-only eval model #266 — Retire the fixture-only eval model (last)

#252, #256, #259, #260, and #265 are independently startable today.

Frictions found during assessment

Friction	Evidence	Ticket
Task repos must be freshly git inited; verify_task_repository rejects any remote and any dirty state. A cloned codebase violates both.	src/cli/run/orchestrate/git.rs:166-218	#252
Diff-scope's baseline is a full copy of every file in the env, and it never produces a diff artifact.	src/pipeline/diff_scope.rs:18-21	#255
One fully materialized env per (group, condition, run) via plain fs::copy. --runs 10 = 20 checkouts.	orchestrate/envs.rs:63, src/core/fs.rs:128	#254
Only scripted-turn tasks are runner-driven; one-shot tasks are dispatched by hand from RUNBOOK.md.	conversation.rs:89-93	#256
No per-dispatch timeout anywhere; execute_round runs to completion.	conversation.rs:367-387	#256
Judge prompts inline the entire run.json, including every tool invocation's args and results. Unbounded on a real task.	grade/judge_tasks.rs:166-218	#261
The guard denies npm/pip install unless the command text happens to mention an allowed root.	src/sandbox/policy.rs:55-96,262-264	#259
A cloned real repo carries .claude/, .agents/, AGENTS.md, and checked-in skills into both arms.	staging/mod.rs:121-123	#260
skill_name is a single string and conditions are binary — no way to make a skill suite the treatment.	schema/evals.schema.json:10, core/types.rs:225	#264
The workspace defaults to cwd/.eval-magic, so eval artifacts land inside the user's skills repo, and the live skill source is reachable from the env by a short relative path.	src/core/context.rs:221, detect_stray_writes.rs:180-229	#253

What is not a problem

The write guard is already fully decoupled from the workspace's location. marker_allowed_roots
(src/sandbox/install.rs:75) returns exactly one allowed root — the isolated env — and the rationale
is already documented there: "Scoping to the env — not the parent .eval-magic/ or the host temp
directory — keeps the guard boundary identical to the isolation boundary." Nothing about write
containment depends on the workspace sitting next to a skill, so no ticket should be justified as
"simplifying the guard by relocating the workspace". Guard friction during a run is a separate
problem with a separate cause — BASH_MUTATION_PATTERNS firing on ordinary dev commands — and that
is #259.

Decisions

1. The runner drives every dispatch. dispatch-task becomes the universal execution path; a new
   eval-magic dispatch runs a whole iteration with concurrency, timeouts, and resume. (Runner-driven dispatch for every task #256)
2. Task envs are real Git repos with history, remotes stripped, hooks disabled, marked with an
   eval-magic/baseline ref. A local-directory source that is not already a repo still gets
   git inited, so every run is guaranteed a repo without requiring one up front. (Source a real codebase into each task environment #252)
3. Turn responder: heuristics first; the LLM answering agent is a separate follow-up. (Dynamic conversation turns with a heuristic responder #257, LLM answering agent for free-form questions #258)
4. codebase is declarable at the top level as a default and overridable per eval, mirroring how
   runs already works. (Source a real codebase into each task environment #252)
5. A git source requires an explicit ref. The runner records the resolved SHA and keys the
   cache on it, so re-running an iteration needs no network. (Source a real codebase into each task environment #252, Provision environments from a per-iteration codebase cache #254)
6. A sourced codebase keeps its own CLAUDE.md/AGENTS.md and config — that realism is the
   point. Skills it makes discoverable get a distinct codebase-sourced finding class, plus an
   opt-in codebase.exclude_skill_sources that moves them aside symmetrically for the
   self-referential case (evaluating slow-powers against the slow-powers repo). (Harness config shipped by a sourced codebase #260)
7. files/files_root survive as a documented overlay on the codebase — seeding a task-specific
   file into a real repo is a live use, not legacy baggage. Only isolation is removed, and it is
   already a documented no-op. (Retire the fixture-only eval model #266)
8. Skills become a sourced, copied input and the eval home moves out of the user's repo. The
   justification is input symmetry and provenance, not guard simplification — on raw code
   complexity this is close to a wash. Today a report can pin the codebase SHA while the skill side
   is "whatever was on disk at the time", which is not a reproducible published claim. (Source skills as copied inputs and relocate the eval home #253)

Cross-cutting requirements

Applied to every sub-ticket rather than tracked separately:

• Mode B parity. Acceptance holds for both Mode::NewSkill and Mode::Revision, and the
  snapshot/promote path keeps working against codebase-backed iterations.
• Provenance reaches the report. Codebase and skill source + resolved SHA land in
  conditions.json, each run.json, benchmark.json, and the BASELINE.md built by
  promote.rs:231 — not just in the workspace.