feat(bench): run Frontier-Bench on Modal, where all 74 tasks fit - #1048
Merged
Conversation
The lane could only ever produce a development baseline. A submittable run must cover all 74 tasks, four of which want a GPU, without overriding any task's declared resources — so it needs a machine that can hand a container 32 GB and an H100 on demand. This laptop's Docker VM is 10 GB with no GPU, and no amount of scheduling fixes that: the plan was 48/74. FB_ENV now selects the backend. On modal the plan is 74/74. Three things change with the backend, each because a docker assumption is false there rather than merely cautious: * Nothing is excluded for capacity. Modal builds a sandbox per trial from that trial's own declared resources, so "does this task fit" is the wrong question. Carrying the laptop's exclusions over would produce a 48-task result that looks exactly like a complete one. * GPU tasks default to on. Harbor passes each task's gpus/gpu_types straight through. Defaulting them off there would silently drop the four tasks a submission is required to cover — the same plausible-wrong-number failure this lane exists to prevent, inverted. --no-allow-gpu still wins when said explicitly, which is why the flag is tri-state rather than store_true. * warm_images.sh skips. The builds happen in the cloud; a local pull warms a cache no trial reads. What does not change: the glibc-2.17 assert runs on both. Modal is a different scheduler, not a different libc, and a binary that cannot exec would fail there the same way it did on 2026-07-31 — as a reward-0 row. Also wires --upload, opt-in behind FB_UPLOAD with private visibility by default. The leaderboard requires publicly readable trials, so this publishes every trajectory and every command the agent ran, permanently. That is a deliberate act and must never be inherited from a shell that had it set for an earlier run. primary.sh now names *every* reason a run would be refused as a submission — tier, attempts, any excluded task, upload off, visibility private — rather than the first. An operator who fixes one and reruns only to be told about the next has paid for a full run to learn something one line could have said. setup_venv.sh installs the modal extra and passes --clear, so it is idempotent; it previously failed the moment a venv existed, which is exactly when you want to run it. 27 planner tests (6 new for the backend), shell syntax and shellcheck clean. Nothing under bench/harbor_adapter/ is touched.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
Contributor
Reviewer's GuideAdds support for running Frontier-Bench on a Modal backend with per-task sandboxes, introduces backend-aware planning and concurrency overrides, makes GPU usage and uploading explicitly configurable and opt-in, and improves posture reporting and venv setup while keeping the Docker path and Harbor adapter tree unchanged. Sequence diagram for submission-shaped run with upload and posture reportingsequenceDiagram
actor Operator
participant EnvSh as env.sh
participant Fetch as fetch_dataset.sh
participant PlanPy as plan.py
participant Primary as primary.sh
participant Harbor as harbor run
Operator->>EnvSh: set FB_ENV=modal FB_ATTEMPTS=5 FB_CONCURRENCY=32
Operator->>EnvSh: set FB_UPLOAD=1 FB_UPLOAD_VISIBILITY=public
EnvSh-->>Operator: exports FB_ENV, FB_CONCURRENCY, FB_UPLOAD, FB_UPLOAD_VISIBILITY
Operator->>Fetch: run fetch_dataset.sh
Fetch->>PlanPy: python3 plan.py --backend $FB_ENV --concurrency $FB_CONCURRENCY
PlanPy->>PlanPy: _elastic_capacity() when backend=="modal"
PlanPy-->>Fetch: write plan.json (backend=modal, tiers, excluded)
Operator->>Primary: run primary.sh all fb-submission-01
Primary->>Primary: compute EXCLUDED_N from plan.json
Primary->>Primary: build UPLOAD_FLAGS from FB_UPLOAD / FB_UPLOAD_VISIBILITY
Primary-->>Operator: print posture=submission-shaped or development-baseline
Primary->>Harbor: harbor run --env $FB_ENV $INCLUDES $UPLOAD_FLAGS
Harbor-->>Primary: exit status
Primary-->>Operator: print harbor_exit and finished timestamp
Flow diagram for backend selection and planning on docker vs modalflowchart TD
Operator["Operator sets FB_ENV and FB_CONCURRENCY"] --> Env["env.sh loads FB_ENV, FB_ALLOW_GPU"]
Env --> Fetch["fetch_dataset.sh"]
Fetch -->|calls| PlanPy["plan.py build_plan(dataset_dir, allow_gpu, memory_headroom_mb, backend, concurrency)"]
PlanPy -->|backend=docker| HostCap["_host_capacity"]
PlanPy -->|backend=modal| ElasticCap["_elastic_capacity"]
PlanPy --> PlanOut["plan.json (backend, tiers.concurrency, excluded)"]
PlanOut --> Sentinel["sentinel.sh harbor run --env $FB_ENV"]
PlanOut --> Primary["primary.sh harbor run --env $FB_ENV"]
Primary --> Warm["warm_images.sh (skips when FB_ENV=modal)"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
macanderson
marked this pull request as ready for review
August 1, 2026 00:26
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-on to #1031. That PR landed the Frontier-Bench lane but it could only ever produce a development baseline — on this laptop's 10 GB / 6-CPU Docker VM the plan is 48 of 74. A submittable run must cover all 74 tasks, four of which want a GPU, without overriding any task's declared resources. No amount of scheduling fixes that; it needs a machine that can hand a container 32 GB and an H100 on demand.
FB_ENVnow selects the backend. Onmodalthe plan is 74/74 with zero exclusions.Three things change with the backend
Each because a docker assumption is false there, not merely cautious:
gpus/gpu_typesstraight through. Defaulting them off would silently drop the four tasks a submission is required to cover — the same plausible-wrong-number failure this lane exists to prevent, inverted.--no-allow-gpustill wins when said explicitly, which is why the flag is tri-state rather thanstore_true.warm_images.shskipsWhat does not change: the glibc-2.17 portability assert runs on both. Modal is a different scheduler, not a different libc — a binary that cannot exec would fail there the same way it did on 2026-07-31, as an ordinary reward-0 row.
Upload, opt-in
--uploadis now wired behindFB_UPLOAD=1, private by default. The leaderboard requires publicly readable trials — CI re-derives every trial from the uploaded job and a maintainer's LLM judge reads every trajectory — soFB_UPLOAD_VISIBILITY=publicpublishes the agent's full reasoning and every command it ran, permanently. That is a deliberate act, and it must never be inherited from a shell that had it set for an earlier run.Posture reporting
primary.shnow names every reason a run would be refused as a submission, not the first:An operator who fixes one gap and reruns only to be told about the next has paid for a full run to learn something one line could have said.
Also
setup_venv.shinstalls themodalextra and passes--clear, so it is idempotent — it previously failed the moment a venv existed, which is exactly when you want to run it (after changing the Harbor pin or adding an extra). Installing the extra unconditionally means switching backends is one variable rather than a rebuild, and lets the preflight tell "modal is not authenticated" apart from "the SDK was never installed" — different problems, different fixes.Verified
warm_images.shcorrectly skips.--no-allow-gpuon Modal → 70/74, so the operator override genuinely wins.~/.modal.toml, workspace reachable), and the preflight distinguishes missing-SDK from unauthenticated.bench/harbor_adapter/is touched — the digest-frozen tree stays untouched, same as feat(bench): a Frontier-Bench lane that keeps the adapter untouched #1031.Not done
No paid trial has run. The sentinel now works against either backend but is still unexecuted; that is the next step and it costs money. A real submission is ~370 multi-hour trials with no spend cap — a four-figure decision that deserves the same preregistration the Terminal-Bench claim got. The upstream intake (PR #1405) is also still unmerged.
Summary by Sourcery
Add a selectable backend for Frontier-Bench runs (docker vs Modal), enabling full 74-task planning on Modal while keeping docker behavior unchanged, and wire upload posture and reporting around this.
New Features:
Enhancements:
Tests: