Add SWE-bench agent runner

[Bjjj834]

Summary

Builds on the SWE-bench Verified adapter from #30. Adds runner support for real agent attempts on SWE-bench tasks plus conversion of official harness reports back into CodeBench metrics. This still does not include final large-scale SWE-bench experiment results — it is the attempt/evaluation infrastructure for producing them.

What's included

• Runner (src/codebench/swebench_runner.py) + CLI (scripts/run_swebench_agent.py):
  • Loads only agent-safe task fields — load_agent_safe_tasks() is the single choke point that drops hidden_reference (gold patch, test_patch, FAIL_TO_PASS, PASS_TO_PASS) at parse time; prompts, logs, and attempt records never contain gold data (enforced by a poisoned-sentinel test).
  • Checks out each target repo at base_commit — cached clone per repo under workspaces/repos/, isolated detached git worktree per attempt.
  • Runs any agent via a pluggable --agent-cmd shell template ({prompt_file}/{workdir}/{instance_id}/{repo} placeholders), with a built-in noop mode for dry runs; per-attempt timeout, --resume, and --attempts N for independent rollouts.
  • Captures git diff as model_patch — git add -A + diff against the explicit base commit (robust to agent-created commits; includes new files). Failed/timed-out attempts are recorded with empty patches so they still count as rollouts.
  • Exports official SWE-bench prediction JSONL — one file per attempt index (instance_id, model_name_or_path, model_patch), reusing write_predictions_jsonl() from Add SWE-bench Verified adapter #30.
• Results conversion (src/codebench/swebench_results.py) + CLI (scripts/convert_swebench_report.py):
  • Converts SWE-bench harness reports into CodeBench results — per-instance tests_status becomes an ExecutionResult (F2P+P2P totals, P2P failures as regression_count, resolved as execution_success); errored/empty-patch instances become zero-score rollouts.
  • Pooling one harness run per attempt yields exactly the rollout shape reliability_at_k() consumes — real rollouts replacing synthetic ones in the H-experiment pipeline, with no public API changes.
• Docs (docs/swebench_experiment.md): updated end-to-end workflow (sample → attempts → harness → reliability@k).
• pyproject.toml: optional eval = ["swebench>=2.1"] extra; no new hard dependencies.

Testing

• 92/92 tests pass (17 new). All new tests run offline: local fixture git repos, a scripted fake agent, and fixture harness reports — no network, no Docker.
• Leakage test: a sample poisoned with sentinel strings in the gold fields verifies the sentinels appear nowhere in prompts, logs, patches, metas, or predictions.
• Live smoke test: both CLIs end-to-end against a local fixture repo — real diff round-tripped into official prediction format, and fixture harness reports converted to correct reliability@k values.

🤖 Generated with Claude Code