feat: add Terminal-Bench 4.0 as a separate Modal-backed benchmark

[abhinav-pola]

TL;DR

Adds terminal_bench_4 (Terminal-Bench v4.0.0, 55 single-container tasks, 3 of them on Modal H100s) as a new benchmark alongside the unchanged terminal_bench (2.1).

What changed?

• harbor/sandbox.ts, harbor/modal-sandbox.ts: CreateSessionInput gains optional gpu and env, forwarded to client.sandboxes.create as gpu / env. Existing callers are unaffected (both optional, dropped via definedValues).
• terminal-bench-4/schema.ts: Zod schema for the 4.0 task.toml shape, validating only the fields the harness consumes (agent.timeout_sec, verifier.{timeout_sec,environment_mode,environment,collect,env}, environment.{cpus,memory_mb,storage_mb,gpus,gpu_types,allow_internet,env}, artifacts, metadata.category). environment_mode must be "separate".
• terminal-bench-4/tasks-source.ts: pinned checkout of harbor-framework/terminal-bench at 452bf305 (the v4.0.0 tag), override via BENCH_TERMINAL_BENCH_4_TASKS_DIR.
• terminal-bench-4/dataset.ts: streams terminal_bench_4-<taskId> samples, excludes the 11 docker-compose tasks, maps gpus/gpu_types to a Modal gpu string (toModalGpu: "H100", "A100:2", throws when a GPU count has no type), carries agent and verifier SandboxResources, artifacts, collect hooks in sample metadata.
• terminal-bench-4/session.ts: two-sandbox flow. Agent sandbox (task image, task resources, GPU, internet forced on) runs the ori agent, then collect hooks. Artifacts (/logs/artifacts plus declared paths) are tarred, downloaded, uploaded into a fresh verifier sandbox (verifier image, verifier resources and env) and extracted at the same absolute paths. /tests/test.sh runs, reward read from /logs/verifier/reward.txt.
• terminal-bench-4/solver.ts, benchmark.ts, images.ts: solver wiring, benchmark definition (temperature 0, 1 epoch, degradeSolverErrors: true, reuses the terminal-bench scorer), deterministic image tags <imageRepo>/<task>:<commit12> and <imageRepo>/<task>-verifier:<commit12>.
• benchmark-config.ts, benchmark-meta.ts, registry.ts, cli/index.ts: TerminalBench4OptionsSchema = TerminalBenchOptionsSchema.extend({ imageRepo }), meta, registry and CLI entries.
• scripts/build-terminal-bench-4-images.ts: one-off docker buildx build --platform linux/amd64 --push of agent and verifier images for every runnable task (--repo, --task, --concurrency, --dry-run).
• terminal-bench-4/README.md: provenance, license (Apache-2.0), execution model, GPU mapping, image strategy, the 11 excluded compose tasks.

Why?

Terminal-Bench 4.0 is a different task set from 2.1 with separate verifier containers, Dockerfile-only environments and H100 tasks, so it cannot be a version bump of the existing integration. Requested in Slack by @abhinav-pola.

How to test

Deterministic (no network):

bun test src/benchmarks/terminal-bench-4

Against the pinned checkout (clones once, or point at an existing checkout):

BENCH_TERMINAL_BENCH_4_TASKS_DIR=/path/to/terminal-bench@452bf305 bun test src/benchmarks/terminal-bench-4

Expected: the pinned-checkout tests report 55 runnable tasks, 11 compose tasks, and exactly fp8-rmsnorm-gemm, jax-speedrun-gpu, math-eval-grader mapped to H100.

Image build dry run:

bun scripts/build-terminal-bench-4-images.ts --dry-run --task hello-world

A real run (bun run bench --benchmark terminal_bench_4 --model <model> --limit 20 --concurrency 4) has not been executed. It needs the 110 task images pushed to a registry and Modal credentials, neither of which was available in this session. See Reviewer focus.

Benchmark impact

New benchmark only. terminal_bench (2.1) code paths are untouched apart from two new optional fields on CreateSessionInput that it does not set.

A terminal_bench_4 score covers 55 of the 66 official tasks, so it is not an official Terminal-Bench 4.0 number. The README states this.

Reviewer focus

• DEFAULT_TERMINAL_BENCH_4_IMAGE_REPO is ghcr.io/openrouterteam/terminal-bench-4. This is an assumption, not a confirmed registry. Change it or override with imageRepo in the run config.
• storage_mb is parsed and validated but not passed to Modal because SandboxCreateParams in modal 0.8.0 has no disk-size field. jax-speedrun-gpu declares 1 TB.
• Agent sandboxes always have internet (the agent CLI needs OpenRouter). Tasks with allow_internet = false record agentNetworkForced: true in metadata rather than being blocked.
• collect hooks that name a compose sidecar service fail the sample explicitly instead of being skipped.
• No end-to-end Modal run yet. Once images are pushed I will run the smoke command and an H100 task and report per-sample results here.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/646dc7c269b049f888a62b6003dacd62
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/646dc7c269b049f888a62b6003dacd62?variant=devin

---

[devin-ai-integration]

Original prompt from Abhinav

SYSTEM:
<latest_message>
Abhinav Pola (U090K0G7JF3) [ts=1788463791.018289]: @Devin whats the difference between terminal bench 2.1 and 4.0
</latest_message>

=== BEGIN THREAD HISTORY (in #ecosystem-agents) ===
Abhinav Pola (U090K0G7JF3) [ts=1788463791.018289]: @Devin whats the difference between terminal bench 2.1 and 4.0
=== END THREAD HISTORY ===
Channel ID: C0BU53A7VEH
Thread URL: https://openrouter.slack.com/archives/C0BU53A7VEH/p1788463791018289?thread_ts=1788463791.018289&amp;cid=C0BU53A7VEH

The <latest_message> is the message that you should use to guide your goals + task for this session, and you should use the rest of the slack thread as context.
A [ts=...] marker on a Slack message is that message's timestamp. To act on a specific message with the slack tool (e.g. adding an emoji reaction via the reaction command), pass that value as timestamp along with the Channel ID — no extra lookup call is needed.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring