A Python coding benchmark that scores model–harness configurations, not models in isolation.
Most code benchmarks score a model. MiHaCoBench scores the final artifact produced by a complete coding system — model, prompt handling, file I/O, planning, tool use, and dependency management included. It ships 95 tasks across 11 categories, each graded by an independent pytest grader against the public contract (never internal names or layout).
See RUBRIC.md for the authoritative scoring methodology.
git clone https://github.com/Hangyu8123/MiHaCoBench.git
cd MiHaCoBench
./scripts/setup.sh # macOS / Linux / WSL / Git Bash
.\scripts\setup.ps1 # Windows PowerShellOne command creates .venv on a suitable interpreter (Python 3.10+, preferring
3.11), installs the dependencies, and runs the preflight check. Then:
# Validate a category first — the full suite takes ~30 minutes
python run_benchmark.py --self-check --category easy
# Validate all 95 graders (gold passes, broken fails)
python run_benchmark.py --self-check # or just: python run_benchmark.py
# Score your own coding harness, end to end
python run_benchmark.py \
--run-harness "claude -p 'Read TASK.md and implement it' --permission-mode acceptEdits" \
--workspace ../mihaco-candidateManual install instead of the setup script
python3.11 -m venv .venv
source .venv/bin/activate # Windows: .\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
python run_benchmark.py --preflightEach category stresses a different ability. Tasks are weighted when aggregating so easy wins cannot inflate the total score.
| Category | Count | What it probes | Weight |
|---|---|---|---|
easy |
4 | Single-file (~100 LOC), stdlib / ≤4 packages — baseline competence | 1 |
complex |
10 | Multi-file / multi-class systems (gold ≈400–800 LOC) using SQLAlchemy, jinja2, networkx, or pandas | 5 |
data_analysis |
7 | Load → analyse → correct statistics → visualise | 3 |
algorithmic |
12 | Time / space complexity + readability (2 medium, 10 hard) | 4 / 8 |
long_horizon |
10 | Dependency chains of 2–30 steps — a wrong early step cascades | steps/2 |
ml |
5 | scikit-learn tasks with held-out, leakage-resistant grading | 3 |
debug |
6 | Fix a planted bug (SWE-bench FAIL_TO_PASS / PASS_TO_PASS style) | 2 |
swe_bench |
10 | Multi-file mini-repo fault localization — symptom crosses a module boundary | 6 |
compositional |
11 | Compose ≥2 libraries under a precise contract with full exception-path coverage (BigCodeBench style) | 4 |
competitive |
12 | Contest-level algorithms with a hard complexity gate + adversarial inputs (LiveCodeBench / APPS style) | 8 |
harness |
8 | Broad from-scratch implementations (expression evaluator, SQL engine, interval map, transactional KV store, …) | 4–6 |
Overall weighted score = Σ(weight × partial) / Σ weight.
MiHaCoBench scores whatever produced the solutions — Claude Code, GitHub Copilot, OpenAI Codex, your own agent, or a human.
--run-harness does the whole loop in one command — scaffold an isolated
workspace, run your agent CLI once per task folder, then score the result:
python run_benchmark.py --run-harness "<your-agent-cli>" --workspace ../mihaco-candidateYour command runs with the task folder as its working directory, so the
portable contract is just: read TASK.md, write the solution here, read nothing
outside the folder.
┌─ 1. Setup ────┐ ┌─ 2. Scaffold ─────────┐ ┌─ 3. Drive your harness ──┐ ┌─ 4. Score ───┐
│ scripts/setup │→ │ Visible task inputs │→ │ Agent reads TASK.md, │→ │ weighted + │
│ │ │ in folders OUTSIDE repo│ │ writes the solution file │ │ results.json │
└───────────────┘ └────────────────────────┘ └───────────────────────────┘ └──────────────┘
└──────────────── all three: --run-harness ─────────────────────────────┘
The command string is parsed with POSIX quoting rules on every platform, so
one invocation works identically on Windows and POSIX — and Windows .cmd shims
(how npm installs claude and codex) are resolved automatically.
| Flag | Purpose |
|---|---|
--workspace DIR |
Where the scaffold lives. Must be outside the repo. |
--harness-timeout SECS |
Per-task budget (default 900). |
--resume |
Reuse an existing workspace, skipping tasks already solved — for interrupted or expensive runs. |
--no-score |
Stop after the agent runs; score later with --candidate-root. |
Combine with --category / --task to try one slice before committing to all 95.
Each run's output is written to <workspace>-logs/, beside the workspace rather
than inside it, so one task's agent can never read another's transcript.
Placeholders. If your CLI needs the values explicitly, {task_id},
{category}, {module}, {task_dir}, and {task_md} are substituted per task.
Substitution happens after argument splitting, so a path containing spaces
stays a single argument.
python run_benchmark.py --workspace ../mihaco-candidate --run-harness \
"claude -p 'Read TASK.md and implement the solution it specifies. Write the solution file(s) into this directory. Do not read files outside it.' --permission-mode acceptEdits"Add --bare inside the command for a config-isolated run (skips global
CLAUDE.md/MCP/hooks).
python run_benchmark.py --workspace ../mihaco-candidate --run-harness \
"codex exec --sandbox workspace-write --ask-for-approval never 'Read TASK.md and implement the solution it specifies. Write the file(s) here.'"Copilot has no headless CLI, so scaffold and drive it interactively:
python run_benchmark.py --scaffold-candidate ../mihaco-candidateOpen the scaffold in VS Code, switch Copilot Chat to Agent mode, and prompt:
"Read TASK.md in this folder and implement the solution; write the file(s)
here." Then score with --candidate-root ../mihaco-candidate.
The agent under test may see only task-declared visible inputs: TASK.md,
optional data/, and optional validated starter files. It must not see graders,
expected/, gold/broken references, task.json, or other tasks. Both
--scaffold-candidate and --run-harness enforce this by refusing targets
inside the repository.
--list-tasks prints the required solution module for every task. Most expect
solution.py; the complex tasks expect a named facade module (e.g.
c01 → queue_api.py); long_horizon tasks expect a solution.py exposing the
--step K --in <prev> --out <out> CLI.
All modes accept --category <cat> and --task <substring> to filter.
| Mode | Description |
|---|---|
--preflight |
Import every required package; report what is missing |
--self-check (default) |
Validate every grader: must PASS on gold, FAIL on broken |
--candidate-root DIR |
Score a harness's solutions; writes results.json |
--scaffold-candidate DIR |
Create an isolated workspace with visible task inputs |
--run-harness CMD |
Scaffold → run CMD per task folder → score, in one command |
--list-tasks |
Print every task's category, id, weight, solution module, and TASK.md path |
Each task is graded in an isolated pytest subprocess (JUnit-XML parsed back) under a per-category time budget, so a crashing or looping solution cannot take down the runner.
Example output:
SELF-CHECK — validating 95 graders (must PASS on gold, FAIL on broken)
[PASS] easy e01_csv_pulse gold 11/11 broken 9/11
...
SELF-CHECK: 95/95 graders valid.
For each task the runner records:
| Metric | Definition |
|---|---|
| partial | passed / runnable (where runnable = total − skipped); gives partial credit |
| strict | 1 only if every runnable grader test passes |
| weighted | Σ(weight × partial) / Σ weight using the category weights above |
--candidate-root writes results.json with per-task passed/total/partial/strict,
per-category totals, strict_total (e.g. 95/95), and the overall weighted_partial.
Readability and empirical Big-O fit are advisory — reported, never gating. Complexity is enforced by feasibility: a wrong-complexity solution physically times out on a large adversarial input.
Graders never hard-code a path. They call grading_utils.resolve_solution_dir,
which honours, in order:
PYBENCH_SOLUTION_DIR— absolute path to this task's solution.PYBENCH_CANDIDATE_ROOT— resolved as<root>/<category>/<task_id>/.- The default gold under
_solutions/;PYBENCH_VARIANT=brokenselects the broken reference.
The same grader runs unchanged against gold, broken, or any candidate.
- Grader integrity. Every grader must pass on a hidden gold reference and
fail on a deliberately broken one.
--self-checkenforces this for all 95 tasks — it is the benchmark's own correctness test. - Isolation. Gold and broken references live under
_solutions/, separate fromtasks/. The agent sees only task-declared visible inputs. - Reproducibility. Committed datasets, fixed seeds,
PYTHONHASHSEED=0,MPLBACKEND=Agg, pinned dependencies, and an offline Python 3.11.5 environment. - Contamination resistance. Domain-specific entrypoint names (e.g.
compute_cohort_retention, notsolve), twists on classic problems, hidden references, and creation dates on everyTASK.mdfor temporal filtering.
A fully-specified task can be solved in one shot by a strong model, so this suite includes tasks specifically built to separate harnesses:
- Multi-file fault localization — the bug's symptom and root cause live in different modules (SWE-bench style).
- Spec-density traps — constraint-dense contracts with a single hidden ambiguity that a hasty pass silently drops.
- Hard complexity gates — a naive or wrong-complexity solution physically times out on a large adversarial input.
- Multi-library composition + exception paths — compose ≥3 libraries with full, typed exception-path coverage.
- Statistical surface-form twists — the memorised answer is wrong (e.g. unpaired vs paired t-test).
- Long state cascades — a wrong early step poisons everything downstream.
- Broad implementation features — large systems (SQL engine with joins, transactional KV store with savepoints) that stress sustained coherence.
Interpretation boundary. A pass does not prove the harness planned, ran tests, or recovered from feedback. The current runner scores the final artifact, not the execution trace. Process claims require a controlled model-held-constant experiment with trajectory telemetry.
run_benchmark.py # entry point: runner / self-validator / scorer
conftest.py # pytest root: sys.path bootstrap + marker registration
pytest.ini # pytest config
requirements.txt # pinned dependency union
RUBRIC.md # full scoring methodology
_lib/
grading_utils.py # shared grading kernel (all primitives, once)
AUTHORING_GUIDE.md # task-authoring contract
tasks/<category>/<task_id>/
TASK.md # the ONLY file shown to the agent under test
task.json # manifest (id, category, weight, packages, entrypoints)
grader/test_*.py # independent pytest grader (public contract only)
data/ expected/ # committed inputs + precomputed ground truth (where needed)
_solutions/<category>/<task_id>/ # GOLD reference (hidden)
_solutions/<category>/<task_id>__broken/ # deliberately-broken reference
benchmarks/agentic/ # separate agentic over-engineering harness (see below)
tasks/<category>/<task_id>/
TASK.md # prompt + exact public contract — the only thing the agent sees
task.json # machine-readable manifest (weight, packages, steps, entrypoints)
grader/test_*.py # written against the contract, not against any solution
data/ # committed deterministic inputs (data_analysis, ml, long_horizon)
expected/ # precomputed ground-truth fixtures (when applicable)
The grader imports the solution by file path via _lib/grading_utils.py, so a
candidate's internal module layout never matters — only the public contract does.
Alongside the correctness suite, the repo ships a second harness under
benchmarks/agentic/ that ports the agentic minimalism /
over-engineering benchmark from
ponytail (MIT). Where the
correctness suite grades solution.py pass/fail, this one runs real headless
Claude Code sessions and measures whether a harness keeps code minimal without
dropping safety or completeness.
- 27 self-contained tasks — safety (7 adversarial-input tasks), quality (4 reuse/trace-before-fix tasks), and open/vibe LOC-only tasks.
- Metrics —
correct+safedeterministic gates, source LOC / file count (tests excluded), cost / tokens / duration / turns from the CLI JSON, plus two auditable LLM judges (over-engineering 0–3, completeness 0–3). - Integrity —
python benchmarks/agentic/run.py --selftestvalidates all good/bad reference pairs offline before any API spend.
This harness is independent of run_benchmark.py and does not affect the
correctness suite. See benchmarks/agentic/README.md.
CONTRIBUTING.md— dev setup, adding a task, PR checklist.RUBRIC.md— authoritative per-task scoring and grading mechanics._lib/AUTHORING_GUIDE.md— contract for authoring new tasks._lib/grading_utils.py— shared grading kernel (solution resolution, import-by-path, complexity/memory gates, plot validation, long-horizon provenance).
Released under the MIT License — © 2026 Hang Yu.