Add local Workshop trajectory evals

[AshishKumar4]

This PR adds pnpm evals on top of WorkshopAgentSession. An eval defines one or more prompts and deterministic checks against the Gadget the agent created. The runner starts an isolated Workshop under local workerd, runs each model and trial, calls the generated Gadget through its RPC API, and cleans up the workspace afterward.

The report shows whether the task passed, which checks failed, and the evidence returned by each check. It also records the canonical conversation, timings, model turns, tool calls, tool errors, final-step tokens, observed chat cost, model, trial, Git commit, and prompt hash.

The first eval asks the agent to create a Project Doc. Only model inference leaves the machine. This PR does not add a production target, Braintrust, or an LLM quality judge.

---

Wall time: 0.45 seconds

Wall time: 2.19 seconds

Wall time: 0.48 seconds

[github-actions]

Preview: pr361-evals-local-runner

https://pr361-evals-local-runner-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[ask-bonk]

1. [P1] Redirects bypass egress filtering (packages/integration-tests/src/network-interceptor.ts:63): realFetch(request) follows redirects internally, allowing a permitted endpoint to redirect to any host without another allow check. Disable redirects or validate every hop.

2. [P1] Vitest cancellation is ignored (packages/workshop-evals/src/harness.ts:36): The supplied abort signal is discarded. A timed-out eval can continue model inference while retaining workerd and the global fetch interceptor. Cancellation should stop the session and trigger bounded cleanup.

3. [P2] Gateway model overrides fail for unsuggested models (packages/workshop-evals/src/target.ts:104): Only direct mode registers input.model. Gateway mode exposes only SUGGESTED_MODELS, so a valid Workers AI model supplied through WORKSHOP_EVAL_MODELS fails availability validation before inference.

4. [P2] Multi-tool transcript ordering is incorrect (packages/workshop-evals/src/transcript.ts:28-57): Results are emitted immediately after each call, producing call A, result A, call B, result B. The actual assistant batch is call A, call B, result A, result B, affecting trajectory display and order-sensitive analysis.

5. [P2] Project-doc verification accepts content without detail (packages/workshop-evals/evals/project-doc.eval.ts:34-35): Length plus punctuation anywhere accepts items such as Scope............., so a trial can pass without the requested sentence of placeholder detail.

github run

[ask-bonk]

1. [P2] Successful tool outputs are omitted (packages/workshop-evals/src/transcript.ts:50): Tools such as readFile and writeFile lack a persisted output field, so their model-visible results become empty events. Reports therefore omit critical trajectory evidence.

2. [P2] Failed later turns lose their prompt (packages/workshop-evals/src/harness.ts:110): If a subsequent runTurn() rejects before producing history, the fallback is skipped because prior events exist. The report omits the attempted turn entirely.

3. [P2] GITHUB_SHA can mislabel eval results (packages/workshop-evals/src/config.ts:29): Ambient GITHUB_SHA bypasses dirty-tree validation and is never compared with HEAD, allowing results to be attributed to code other than what executed.

4. [P2] Fire-and-forget duplicate checks cause unhandled rejections (packages/workshop-evals/src/verifier.ts:48): Duplicate detection rejects before registering the promise in #pending. An unawaited duplicate, despite fire-and-forget checks being supported, can fail Vitest as an unhandled rejection.

5. [P2] Ordered lists satisfy the bullet-list check (packages/workshop-evals/evals/project-doc.eval.ts:71): Matching every <li> also accepts <ol> content, allowing a numbered list to pass the explicitly requested bullet-list requirement.

github run

[ask-bonk]

Posted 1 actionable inline finding.

github run

[ask-bonk]

@AshishKumar4 Bonk workflow failed. Check the logs for details.

View workflow run · To retry, trigger Bonk again.

[ask-bonk]

LGTM!

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

1. [P2] Valid HTML lists can fail verification (packages/workshop-evals/evals/project-doc.eval.ts:71): HTML permits omitted </li> tags, but this regex requires them. A valid three-bullet document can render correctly yet produce zero matches. Parse the HTML as a DOM instead.

github run

[ask-bonk]

LGTM!

github run

[ask-bonk]

Posted 2 actionable inline findings.

github run

[Maximo-Guk]

To be honest this should probably also check for CF_AI_GATEWAY_USE_BINDING=false / not existing rather than just empty api token

[AshishKumar4]

tbh my agent actually proposed that but why would we want that though? feels redundant state?

[Maximo-Guk]

Because we use this elsewhere in the repo, and what if you have an API token already in env vars but you want to opt out?