Six CAD tasks, 100 Core cases. The model works in a sandbox, writes one Python program per round, and hands in geometry or a schematic. Scoring is executed geometry against the reference, deterministic, with no LLM judge.
| Task | Input → Output | Metric |
|---|---|---|
| T1 | a part drawing → the part (CadQuery) | part_v1 |
| T2 | parts (STEP) + assembly drawing → the assembly | asm_v1 |
| T3 | four rendered views → the part | part_v1 |
| T4 | views of a mechanism → every part + the assembly | avg_part × asm_v1 |
| T5 | part drawings + parts + assembly drawing → the assembly | avg_part × asm_v1 |
| T6 | photos of a PCB → its schematic | ecad_v2 |
Needs Docker, libgl1, and a Hugging Face account with
Core access.
git clone https://github.com/BenchCAD-org/benchcad-2-main && cd benchcad-2-main
export HF_TOKEN=... GEMINI_API_KEY=... # or ANTHROPIC_API_KEY / OPENAI_API_KEY
tools/run_core.sh --model gemini/<model-id> --effort low,medium,highOne command installs the environment, builds the sandbox, downloads and verifies the
dataset, runs all 100 cases and scores them. Run it again to resume. The summary is
results/core/<model>/core_summary.json; send that file to report a score.
| Flag | Meaning |
|---|---|
--model |
anthropic/<id>, openai/<id>, gemini/<id>, xai/<id>, openrouter/<id> |
--effort |
comma-separated levels, one run each (default: the provider's full ladder) |
--rounds |
default 60, the published setting; anything else is a smoke run |
--token-budget |
output tokens per case, thinking included; default 500000, the published setting (0 = none); anything else is a smoke run |
--cases |
a task or case directory, for a subset |
--set public |
run the Public set instead of Core |
--dry-run |
check keys, dataset access and scorers; no API calls |
How each provider's effort level is sent, auth options and the other flags:
docs/RUNNING.md.
| Version | Tag | Use |
|---|---|---|
| 2.1 | v2.1 |
Recommended. Reasoning carried across rounds; 60 rounds and 500k output tokens per case, both stated to the model, provisional. Task-6 scorer rule B; case files unchanged. |
| 2.1-beta | v2.1-beta |
Superseded: reasoning carried; 40 rounds and 500k output tokens per case. |
| 2.0 | v2.0-core1 |
Only to reproduce the published reference lines (30 rounds, thinking not carried). |
What each version pins (code commit, dataset revisions, scorer digest, limits):
releases/.
Don't mix versions or run from main for a reported score; check out a tag
(git checkout v2.1). Every record and core_summary.json carry harness_version;
scores from different versions are not comparable.
BenchCAD/benchcad-2.0-public:
100 more cases, same tasks and scorers, no overlap with Core. Accept its terms once
(one click, no review), then add --set public. Its scores are for trying the harness,
not for reporting. Nine samples also ship in examples/, no download needed.
60 rounds and 500k output tokens per case, thinking included: the case ends at a submit or at
whichever limit runs out first, checked between calls. The prompt states both limits, and that a
case that ends without a submit scores 0. (Since 2026-10-18, 2.1; 2.1-beta: 40 rounds and
500k; 2.0, before 2026-10-16: 30 rounds, no token budget.) Each observation starts
Round N/60 · output tokens spent / 500,000. A reply the harness throws away and asks again for
(no executable block, a broken stream) does not count against the budget. A case that ends without a submit has no answer. A call that fails for infrastructure reasons (network, server error,
stalled stream) is retried and does not count as a round. (Changed 2026-10-12.)
Each round the model gets its earlier reasoning back, as its provider returned it:
Anthropic's thinking blocks with their signatures, OpenAI's encrypted reasoning items,
Gemini's thought signatures, OpenRouter's reasoning_details. (Changed 2026-10-15; before,
only the text went back.)
results/core_reference.json holds the published lines: harness 2.0, 30 rounds, all 100 Core cases,
run before the 2026-10-15 change above -- a different setting from today's 60 rounds and 500k tokens.
Claude lines are one rep; openai/gpt-6-astra is the mean of 3 reps (T6: 1 rep), with
the spread in noise_note. run_core prints the published line beside yours. To rerun
them in their own setting, check out v2.0-core1 first:
git checkout v2.0-core1
tools/run_core.sh --model anthropic/claude-opus-5-5 # ANTHROPIC_API_KEY
tools/run_core.sh --model openai/gpt-6-astra # OPENAI_API_KEYdocs/RUNNING.md: providers, flags, clocks, running your own agent.docs/METRICS.md: every metric.docs/CASE_FORMAT.md: a case and a submission.envs/<task>/TASK.md: the prompt each task gives the model.
Contact: benchcad.team@gmail.com