[ENHANCEMENT] Add coding-task evaluation framework on the agent-evals substrate #2084

Background

PR #1909 adds scripts/agent-evals/. This is a runner that starts real VS Code with the extension in an isolated profile. It runs one registered fixture, collects events over IPC, and writes a metadata-only trace.json. The first fixture is the Solheim provider smoke test (scripts/solheim-smoke/). Tracking issue: #1896.

The smoke test shows that Zoo can talk to a model. It does not show that Zoo can do useful work, or what that work costs. Nothing measures the effect of a context, tool-output, or compaction change. A change can cut tokens and still break tasks. Today nobody can see this.

Goal

A maintainer runs a fixed set of coding tasks on two exact commits. The maintainer then reads one report that compares correctness and cost.

Boundaries between the issues

• PR feat(evals): add trusted fixture runner and execution traces (substrate only) #1909 is the execution substrate. It runs a real Zoo task and produces a trace.
• This issue is the evaluation framework. It grades runs, repeats them, and compares commits. It produces a self-contained result.
• [ENHANCEMENT] Run the agent evaluation advisory report during release preparation #2085 is release orchestration. It triggers runs, shows reports, keeps baselines, and promotes them.

This issue must not assume a release process. A maintainer must be able to run it locally or on demand.

Scope

1. Add five frozen coding fixtures. Each fixture has a known-good workspace, one prompt, a tool policy, and execution limits. Use the fixture contract in scripts/agent-evals/contract.ts.
2. Add graders that need no model. See "Grading".
3. Add a comparison step. It runs the same fixtures on a baseline commit and a candidate commit with identical pinned inputs. See "Pinned inputs". This paired execution is for development experiments. Release comparisons reuse stored baseline results. See [ENHANCEMENT] Run the agent evaluation advisory report during release preparation #2085.
4. Run each fixture three times per commit. Report paired differences with uncertainty.
5. Add a JSON report and a Markdown report.

Fixture isolation

• Each fixture has an immutable initial-state digest.
• Each run builds the workspace from that state in a fresh directory. A run cannot inherit changes from another run.
• The current contract allows up to 64 inline files and 262,144 characters of file content. Select small tasks that fit. Do not expand the substrate to clone arbitrary repositories in this issue.

Grading

• The agent gets a realistic task in an ordinary repository.
• After the agent stops, the grader runs hidden acceptance tests that the agent could not read or change. The hidden tests are copied into a fresh copy of the final workspace.
• The grader runs outside the agent's tool-accessible environment. It has no provider credential. It has no network access, because it runs code that the agent wrote.
• Treat the agent's code as untrusted. A separate directory is not enough isolation. Run the grader in a disposable container with CPU, memory, and time limits, a restricted filesystem, and no network. If you use another method, state why it gives equal isolation.
• The grader has its own time limit. A grader that hangs or crashes is an invalid evaluation. It is not an incorrect solution.
• The agent never judges its own work. Do not use a model as a grader.
• Visible tests in the workspace can exist. A passing visible test is not enough.

Each fixture ships with two reference solutions: one known-correct, and one deliberately incorrect but plausible. For example, the incorrect solution passes the visible test and breaks an edge case. Each grader must pass the first and fail the second.

Run outcomes

Report one outcome for each run:

• correct: the hidden tests pass.
• incorrect: the agent stopped and the hidden tests fail. This includes a run where the agent reported completion.
• timed out: the agent hit an execution limit.
• grading failure: the grader hung, crashed, or exceeded its resource limits on the agent's code.
• infrastructure failure: VS Code, IPC, or the provider failed for a reason not caused by the agent.

Count grading failures and infrastructure failures separately. Do not count them as incorrect.

Pinned inputs

A paired comparison pins and records these values:

• Exact baseline and candidate commit SHAs.
• Fixture suite identifier and content digest.
• Nonsecret settings digest and model identifier.
• Grader version.
• Runner environment: OS, Node version, VS Code version.

If any pinned value differs between the two sides, mark the comparison as incompatible.

Metrics per run

• Input, output, and cached tokens, when the provider reports them. Mark missing values as missing. Never write zero for a missing value.
• Prompt size for each request. Show reported input tokens and estimated context size as separate values, with labels.
• Tool calls and tool failures.
• Duration and stop reason.
• Run outcome.
• Estimated cost. Record the price-table identifier and date. State how cache reads and writes are priced.
• Cost per successful task: total cost of all attempts (successful and not) divided by the number of correct runs. If no run is correct, or if cost data is incomplete, report it as unavailable. Never report zero.
• Unsupported telemetry, such as duplicate-call detection or HTTP retry counts, is reported as unavailable. Do not infer it from unrelated events.

Runs with incomplete token or cost accounting are flagged. Exclude them from aggregate cost comparisons unless a documented partial-data policy applies.

Statistics

• Use three repetitions per fixture per commit. This is 30 coding runs for one comparison.
• Report success rate, paired differences, and uncertainty.
• Do not claim statistical significance. Three repetitions can find large regressions. They cannot show small improvements.

Security

• Keep the rule from feat(evals): add trusted fixture runner and execution traces (substrate only) #1909. Only trusted code runs with the provider credential.
• Do not run fixtures or extension code from an outside pull request with the credential.
• Keep each run in an isolated VS Code profile.

Out of scope for this issue

• Release workflow changes. See [ENHANCEMENT] Run the agent evaluation advisory report during release preparation #2085.
• Model-based judging.
• Hard release gates.

Later phases

Start these only after the first five fixtures give stable results.

Together with the smoke test, this is a 21-task suite.

Optimization experiments, in this order. Each one compares a baseline with a candidate on the same fixtures:

1. Instrument per-request context and token use. Find the real cost drivers.
2. Detect non-progress loops and repeated work.
3. Compare tool-output filtering policies.
4. Compare compaction strategies. Measure both savings and lost context.
5. Test dynamic tool-schema loading. Measure repeated prompt overhead.

These experiments need only this issue. They do not need #2085.

Acceptance

• Each fixture has an immutable initial-state digest and runs in a fresh directory. Repeated runs cannot inherit workspace changes.
• Each grader runs in a disposable container (or a documented equal) with resource limits, a restricted filesystem, no credential, and no network.
• A grader that hangs on agent code is reported as grading failure, not incorrect.
• Each grader passes its known-correct solution and fails its deliberately incorrect solution.
• A plausible incorrect solution fails grading even when the agent reports completion.
• Reports distinguish correct, incorrect, timed out, grading failure, and infrastructure failure.
• Incomplete accounting is reported explicitly and excluded from aggregate cost comparisons unless a documented partial-data policy applies.
• A paired comparison records commit SHAs, fixture and settings digests, model configuration, grader version, and runner environment. It flags incompatible pairs.
• The report shows cost per successful task, or unavailable.
• The default is three repetitions per fixture per commit. The report shows paired differences and uncertainty and makes no significance claim.
• Unsupported telemetry is marked unavailable.
• A run on a deliberately broken commit fails at least one grader.
• Unit tests cover the graders and the report logic without VS Code or a provider.

Depends on

PR #1909 merged, and the first approved live smoke run on main (open item in #1896).