Skip to content

feat(evals): add coding-task evaluation framework (#2084) #2154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
edelauna wants to merge 21 commits into
base: main
Choose a base branch
from
Open

feat(evals): add coding-task evaluation framework (#2084) #2154

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f69baee
feat(evals): add coding-task evaluation framework (#2084)
edelauna Oct 23, 2026
f49cdb2
fix(evals): secure coding comparisons and correct grading and cost ac…
edelauna Oct 23, 2026
95a525c
fix(evals): enforce trust checks and reliable comparison cleanup and …
edelauna Oct 24, 2026
bb1758c
fix(evals): pass test fixture paths as arguments instead of embedding…
edelauna Oct 24, 2026
12794d7
fix(evals): enforce isolated grading and safe interruption and result…
edelauna Oct 24, 2026
4f0e44e
fix(agent-evals): reject dirty checkouts and clear stale grader digests
edelauna Oct 26, 2026
d26ad55
fix(agent-evals): prevent child startup after interruption and preser…
edelauna Oct 26, 2026
02ce7ff
fix(agent-evals): detect dirty checkouts despite Git settings that hi…
edelauna Oct 27, 2026
980095b
fix(agent-evals): isolate side builds and invalidate stale results be…
edelauna Oct 27, 2026
8210012
fix(evals): wait for preparation cleanup after repeated interrupts
edelauna Oct 30, 2026
9028fde
fix(evals): address review feedback on isolation, cleanup, and reports
edelauna Nov 2, 2026
9fd42e9
fix(evals): address review feedback on isolation, cleanup, and reports
edelauna Nov 2, 2026
fceb8a7
fix(evals): close review gaps in cleanup and run accounting
edelauna Nov 2, 2026
6f381c8
fix(evals): bind grader images and isolate host caches
edelauna Nov 3, 2026
87901d1
fix(evals): clean up partial side publication on failure
edelauna Nov 3, 2026
801c43a
fix(agent-evals): prevent credential leaks to grading subprocesses
edelauna Nov 3, 2026
3ce847d
fix(evals): probe container engines and preserve Docker endpoint sett…
edelauna Nov 5, 2026
a17059b
fix(evals): qualify grader image and test output permissions
edelauna Nov 6, 2026
42788a0
fix(evals): address review feedback and failing CI assertions
edelauna Nov 6, 2026
3c26401
fix(evals): reuse verified private VS Code cache and preserve proxy s…
edelauna Nov 7, 2026
e0427c6
fix(evals): disable host commands and lock comparison output
edelauna Nov 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
"solheim-smoke:check-types": "tsc -p scripts/solheim-smoke/tsconfig.json",
"test:agent-evals:unit": "node --import tsx --test scripts/agent-evals/*.test.ts",
"agent-evals:check-types": "tsc -p scripts/agent-evals/tsconfig.json",
"coding-evals:side": "node --import tsx scripts/agent-evals/coding-side.mts",
"coding-evals:compare": "node --import tsx scripts/agent-evals/coding-compare.mts",
"test:mutation-ci": "node --test scripts/stryker-diff.test.mjs",
"lifecycle:model-check": "tsx scripts/check-task-lifecycle.ts && tsx scripts/check-task-store-concurrency.ts && tsx scripts/check-provider-handoff-scheduler.ts && pnpm cleanup-protocol:model-check && pnpm parser-scope:model-check && tsx scripts/check-completion-persistence.ts && tsx scripts/check-delegated-mode-readers.ts && pnpm tool-turn:model-check",
"webview-recovery:model-check": "node --import tsx scripts/check-webview-recovery.ts",
Expand Down
109 changes: 99 additions & 10 deletions scripts/agent-evals/README.md
View file
Open in desktop

Large diffs are not rendered by default.

Loading
Loading