Skip to content

Add SWE-bench Verified adapter - #30

Merged
Bjjj834 merged 1 commit into
mainfrom
feature/swebench-verified-adapter
Jul 13, 2026
Merged

Add SWE-bench Verified adapter#30
Bjjj834 merged 1 commit into
mainfrom
feature/swebench-verified-adapter

Conversation

@Bjjj834

@Bjjj834 Bjjj834 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds SWE-bench Verified dataset integration to CodeBench, replacing synthetic sample tasks with real repository-level tasks for benchmark runs. This PR is infrastructure only — it contains no agent automation and no SWE-bench evaluation results yet.

What's included

  • Adapter (src/codebench/swebench_adapter.py): loads SWE-bench/SWE-bench_Verified from Hugging Face, converts instances into the existing CodeTask format (no public API changes), and writes prediction files in the official SWE-bench harness format (instance_id, model_name_or_path, model_patch).
  • Sampling script (scripts/prepare_swebench_sample.py): writes a small task sample (default 5 tasks; --limit, --difficulty, --output for 30/50-task runs later).
  • Prediction-template script (scripts/create_swebench_predictions_template.py): generates a harness-compatible predictions JSONL with empty model_patch stubs.
  • Docs (docs/swebench_experiment.md): full workflow — prepare sample → run agents → fill predictions → evaluate with the official harness → map results back into CodeBench metrics.

Leakage prevention

The gold patch, test_patch, FAIL_TO_PASS, and PASS_TO_PASS fields never reach agent-facing output:

  • CodeTask.test_file and CodeTask.reference_solution hold placeholders, so a serialized task cannot leak the solution.
  • Agents see only the issue description, repo name, base commit, difficulty, and tags (agent_prompt_payload()).
  • Gold data is stored solely in a clearly-marked hidden_reference block for internal sanity checks; the generated sample/template files are gitignored.
  • A programmatic audit of the generated sample confirmed no gold content in any agent-facing field.

Testing

  • 19 new offline adapter tests (fake instances, no network): field mapping, difficulty mapping (<15 min fix→easy … >4 hours→hard, unknown→medium), leakage checks, sampling limits, and official prediction format.
  • Full suite passes: 75 passed (56 pre-existing + 19 new).
  • End-to-end smoke test against the real Hugging Face dataset verified sampling and template generation.

🤖 Generated with Claude Code

@Bjjj834
Bjjj834 merged commit e2a0190 into main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant