[ENHANCEMENT] Run the agent evaluation advisory report during release preparation #2085

Background

PR #1909 adds scripts/agent-evals/, a runner that starts real VS Code and runs one fixture. It also adds a manual workflow, solheim-provider-smoke.yml. The job runs only on refs/heads/main, behind the protected environment final-vscode-review-smoke. Issue #2084 adds a coding-task evaluation framework. It produces a self-contained, graded, paired report. Tracking issue: #1896.

That framework helps only if maintainers see its result before a release. Without a trigger, nobody runs it. Maintainers decide each release by hand. They have no data on correctness or cost against the last approved release.

Goal

After release preparation is merged to main, and before the stable release is published, the evaluation runs on the exact main commit. The result appears in the workflow summary. A maintainer compares it with the last approved baseline and acknowledges it.

Boundary with #2084

• [ENHANCEMENT] Add coding-task evaluation framework on the agent-evals substrate #2084 produces a self-contained evaluation result. Nothing in it is release-specific.
• This issue only does four things: trigger runs, fetch baselines, show reports, and promote baselines.

Trust decision

Evaluate approved main commits only. Keep the main-only credential environment from #1909.

Do not run on a release branch in this issue. Two reasons:

• A main-only environment cannot run a release branch.
• Push access to a branch in this repository does not make its code trusted. A person who can push to the branch can change the harness to expose the provider key.

Evaluating release branches before merge needs more security work: controls on branch updates, workflow changes, dependencies, and credential exposure. Treat it as a separate future issue.

The existing release workflows

I read these workflows. They do not fit as the trigger.

Add a separate workflow, agent-evals.yml. Start it by hand (workflow_dispatch) on main. Document the step in the release process: after the release-preparation merge and before the tag.

Scope

1. Add agent-evals.yml, with manual dispatch on main.
2. Record the exact candidate commit SHA. Verify before reporting that it has not changed.
3. Compare the candidate with the stored baseline results. Write the comparison to the workflow summary. Do not rerun the baseline release. Reuse its approved, immutable results. This keeps the historical evidence and avoids paid model runs.
4. Keep the JSON report, the Markdown report, and the traces as artifacts for 90 days.
5. Keep approved baselines in a durable place that outlives workflow artifact expiry. Keep transient traces and videos separate from the durable manifest and report. The implementer chooses the storage and records the choice in this issue.
6. Add an explicit baseline promotion action.
7. Add a step to the release process documentation: the maintainer reads the report and acknowledges it before publishing.

Baselines

A baseline is an immutable, approved record with these fields:

• Exact commit SHA and extension version.
• Fixture suite identifier and content digest.
• Model identifier and nonsecret settings digest.
• Grader version.
• Runner environment identity.
• Individual run results and accounting completeness.
• Price-table identifier and date.

Two baseline kinds exist:

• Product baseline: the last approved Zoo release. It compares product performance across releases.
• Measurement baseline: the pinned environment, model configuration, and fixture suite. It keeps comparisons valid.

Storage must keep these invariants, whatever mechanism the implementer picks: immutable identities, integrity verification, durability, traceability to an approved release, and authorized promotion.

Rules:

• If the model, fixture suite, grader, or runner environment differs, flag the comparison as incompatible. Do not show it as a normal result. A maintainer then sets a new measurement baseline.
• Promotion needs an explicit action from an authorized maintainer after they review the report.
• A passing release does not promote itself.

Rules

• The report is advisory. It is not a required status check. It never blocks publishing.
• Use a protected environment with required reviewer approval and the main-only restriction.
• Use the same credential rules as the smoke workflow: contents: read and restricted child environments.
• Never run on a pull request from a fork.
• Record a video only on failure. Do not record every run.

Out of scope

• Changes to the evaluation tasks or graders. See [ENHANCEMENT] Add coding-task evaluation framework on the agent-evals substrate #2084.
• Evaluating release branches before merge.
• Public publishing of results. Consider this after several releases of stable data.
• Hard release gates.

Acceptance

• A maintainer can start the workflow by hand on main, and on no other ref.
• The workflow evaluates an exact commit SHA and fails the report if the SHA changed during the run.
• The workflow runs only the candidate. It reads the baseline from storage and never reruns the baseline commit.
• The workflow summary shows baseline against candidate for correctness, tokens, tool calls, and estimated cost.
• The report flags incompatible comparisons (different model, fixture suite, grader, or environment).
• Baseline manifests are immutable and include all fields listed above.
• Only an authorized maintainer can promote a baseline. No run promotes automatically.
• Approved baseline reports stay available after workflow artifact expiry.
• A workflow test proves that fork pull requests, other refs, and modified candidate inputs cannot reach the credential step.
• The release process documentation names the step that triggers the evaluation and the maintainer acknowledgement.

Depends on

#2084.