eval: claude arm needs its own experiment run_limits defaults (900/1200 is codex-sized)

[nikhil-maryala]

Follow-up to rockymadden's review on #3369 (finding 2).

tests/experiments/nightly.yaml hands every task turn_timeout: 900 / task_timeout: 1200, sized for the codex arm. The claude-code arm streams the whole session as one turn, so turn_timeout is its effective wall budget — and 20+ BPMN tasks (plus other suites) still inherit the default. Result: per-task budget PRs keep recurring (#3150 → #3369) each time a task's claude duration drifts across the codex-sized cap.

Proposal: a claude-specific experiment config (e.g. nightly-claude.yaml) or per-agent run_limits support, defaulting the claude arm to ~1800/2400, so the next slow task is a config line instead of a third PR of this shape.

Evidence for the sizing: #3369's analysis — tool execution is 16–50s of 900–1800s walls, the rest is model generation; three-day nightly history shows tasks passing with 3–37s of margin under the current defaults.

🤖 Generated with Claude Code

[nikhil-maryala]

Implementation up: #3416 — nightly-claude.yaml (run_limits 1800/2400, everything else pinned identical to nightly.yaml by a sync guard that verifiably fails on drift), per-arm selection in run-coder-eval.yml, and a 2700/3600 bump for smart_triage (today's other victim, whose explicit limits outrank any default).

Fresh data since this issue was filed — the treadmill produced victims 4 and 5 in today's (09-18) claude nightly:

• multi-city-weather ERROR at 905s vs the inherited 900s default (exactly the class this issue describes)
• smart-triage ERROR at 1805s vs its explicit 1800s (1.0 on both 09-16 and 09-17)

Plus the measurement from #3379's verify: variable_scopes_nested legitimately takes 2354s on the claude arm.

The one line I can't land from the skills repo: coder_eval_uipath's daily.sh also passes -e experiments/nightly.yaml for every arm. It needs the claude branch of the same selection, e.g.:

[ "$ARM" = "claude" ] && EXPERIMENT=experiments/nightly-claude.yaml || EXPERIMENT=experiments/nightly.yaml

(spelled however daily.sh names its arm variable). @rockymadden — that half is yours; until it lands the VM claude arm just keeps today's shared defaults, so there's no broken intermediate state.

[nikhil-maryala]

Runner-side update from #3416's review: @rockymadden points out daily.sh:1049 hardcodes the experiment path for the auth-safe rewrite on provisioned runs (greps it for exactly one writable mount), so the per-arm choice has to feed that rewrite — and that eval_runner/src/eval_runner/skills/cli.py:107 already honours EVAL_RUNNER_SKILLS_EXPERIMENT, which is likely the cheaper mechanism than editing the script: export it per-arm (experiments/nightly-claude.yaml on the claude arm) and the rewrite path follows. Superseding my earlier one-line daily.sh suggestion with that.