MiMo-CodeHarness: Multi-Agent Evaluation Harness for Real-World Code Repositories

A repository-level CodeAgent evaluation harness powered by Xiaomi MiMo Pro API.
It scans real code repositories, reasons over cross-file dependencies, generates coding tasks, runs model-based agents, checks patches/build signals, records token usage, and produces dashboards and technical reports.

中文说明

Current status: v0.12.0 beta. The installable CLI supports independent patch verification, bounded external-Agent repair, Python/Node/JUnit acceptance, recovery and portable evidence. Start with Quick Start or the independent-user kit. Live-model validation, three independent user walkthroughs and maintainer license/final sign-off remain pending; this is not a stable release. See project status and the product plan.

---

1. Why this project?

Most LLM coding evaluations focus on isolated algorithm problems or short code snippets. However, real engineering work requires much more than writing one function:

• understanding a real repository structure;
• locating relevant files across modules;
• reasoning about cross-file dependencies;
• generating repository-level tasks;
• applying or checking patch-style modifications;
• running lightweight build/test checks;
• recording token usage and evaluation evidence;
• producing reproducible reports.

MiMo-CodeHarness was built to evaluate these repository-level CodeAgent capabilities in a more engineering-oriented workflow.

This project is especially designed for AI + Software Engineering + IoT/Embedded/Tooling scenarios. It uses Xiaomi MiMo Pro API as the main model backend and applies the harness to real repositories such as an STM32 RFID access control project and a Python SlideNotes GUI tool.

---

2. What is an Agent Harness?

This project is not just a single LLM API demo.

A simple API demo usually looks like:

Prompt -> LLM response

MiMo-CodeHarness runs a structured evaluation pipeline:

Real Repository
-> Repository Scanner
-> Dependency Reasoner
-> Task Generator
-> Model Runner
-> Patch Checker
-> Build/Test Checker
-> Evidence-aware Scorer
-> Dashboard / Technical Report / Token Log

In other words, this project works as a harness: it provides the task environment, execution pipeline, evidence collection, scoring logic, and reproducible report generation for CodeAgent evaluation.

---

3. Core Features

Repository scanning

• Counts files, languages, suffixes, and source/documentation roles.
• Extracts representative files from the target repository.
• Supports multiple code types including C/C++/embedded C, Python, JavaScript/TypeScript, ArkTS, Markdown, JSON, and shell scripts.

Dependency reasoning

• Extracts cross-file relations such as C includes, Python imports, Markdown links, and symbol-level references.
• Produces dependency summaries and structured dependency files.

Task generation

Generates repository-level tasks such as:

• repository architecture understanding;
• cross-file dependency explanation;
• bug localization;
• test plan generation;
• patch-style maintainability improvement;
• engineering risk analysis.

Model execution

• Uses Xiaomi MiMo Pro API through environment variables.
• Records model outputs, response time, and token usage.
• API keys are never stored in source code.
• A shared runtime supports mock, OpenAI-compatible, Anthropic and mapped generic HTTP final-text responses, with explicit usage/completion states, bounded deadlines/retries and recovery-aware budgets. See provider configuration.

Patch and build/test checks

• Extracts patch-style outputs when available.
• Applies patch checks in isolated worktrees.
• Runs lightweight build/test signals such as Python syntax check, JSON parsing, and C/C++ brace scan.
• For embedded projects, the current build/test check is not a full Keil firmware compilation; it is a lightweight static safety check.

Evidence-aware scoring and hard verification

The descriptive score measures repository grounding, rubric coverage, answer completeness, patch evidence, and build/test signals. v0.4 now reports an independent hard verdict for every candidate: PASS, FAIL, INSUFFICIENT_EVIDENCE, ERROR, or NOT_APPLICABLE.

A high score cannot override a failed or unverified patch. Each applied candidate is tested in its own faithful workspace and compared with an untouched baseline. Static-only checks no longer count as behavior-test success. See Verification Pipeline v0.4.

LLM Judge and human review remain optional evidence and do not replace executable verification.

---

4. Historical v0.3.1 Case Study Results

The following results preserve the original final clean runs for provenance. Cite them only as historical descriptive scores. For current executable acceptance and reference/regression controls, see real Python defects, Node/JUnit checks and the acceptance matrix.

Case Study	Repository Type	Files	Dependency Edges	Tasks	Avg. Score	Estimated Tokens	Status
STM32 RFID Access Control	Embedded C / IoT	16	86	6	78.76	41,520	Legacy / unverified
SlideNotes GUI Tool	Python GUI / Document Export Tool	91	505	6	66.71	Legacy / unverified

These historical v0.3.1 scores predate baseline-vs-candidate verification and must not be cited as verified patch success.

Historical result directories:

outputs/mimo_codeharness_v02/final_stm32_v031_clean__stm32_rfid/
outputs/mimo_codeharness_v02/final_slidenotes_v031_clean__slidenotes/

Each final result directory contains:

dashboard.html
TECHNICAL_REPORT.md
PAPER_DRAFT.md
token_usage.csv
evaluation_scores.csv
model_outputs.jsonl
repo_summary.json
dependency_summary.json
tasks.json

---

5. Example Outputs

Dashboard

Each case study produces an HTML dashboard showing:

• repository file count;
• dependency edge count;
• estimated token usage;
• task count;
• average score;
• build/test status;
• task-level score table.

Open a dashboard locally:

ii "outputs\mimo_codeharness_v02\final_stm32_v031_clean__stm32_rfid\dashboard.html"

or:

ii "outputs\mimo_codeharness_v02\final_slidenotes_v031_clean__slidenotes\dashboard.html"

Technical report

Each case study also generates a technical report:

TECHNICAL_REPORT.md

The report summarizes repository structure, dependency analysis, model evaluation results, token usage, and engineering observations.

---

6. Quick Start

6.1 Clone this repository

git clone https://github.com/hongzuoj-pixel/MiMo-CodeHarness.git
cd MiMo-CodeHarness

6.2 Install the CLI

Python 3.10+ is required. The harness itself is stdlib-only.

python -m pip install -e ".[test]"
mimo --version

For Python behavior verification, install the test extra with python -m pip install -e ".[test]" and install your project's own test dependencies in the same interpreter.

The legacy python src/... entry points remain available, but new users should use the mimo CLI.

6.3 Initialize and diagnose a project

Start in explicit offline mode. This creates mimo-harness.json and .mimo/models.json; it never calls a model API.

mimo init --mode offline
mimo doctor
mimo run --name first-run --limit-tasks 1
mimo report

Generated run artifacts are written under .mimo/runs/. Existing run directories and configuration files are not overwritten implicitly.

6.4 Verify an existing patch without an API key

verify materializes the selected Git baseline and applies the patch only to an isolated candidate workspace. The source checkout is not modified.

git diff > fix.patch
mimo verify --base HEAD --patch fix.patch --run-tests
mimo report

Omit --run-tests for static checks only. That intentionally yields INSUFFICIENT_EVIDENCE, not a false pass. For --task-type bug_fix, configure explicit target_tests: each target must fail a baseline assertion and pass in the candidate. See project checks and pytest evidence for a complete configuration example.

Pytest results now include stable case IDs, execution phases, and JUnit XML. Missing tests, new skip/xfail, changed baseline acceptance inputs, collection errors, and empty suites cannot produce a verified pass. Existing baseline skips are listed explicitly and do not count as executed acceptance tests.

Execution environments

The default local backend uses the current interpreter with a clean environment and per-run home/temp directories. Use execution.mode: "venv" for fresh baseline/candidate dependency environments, or the Linux Docker backend for resource and network controls. Preparation commands are explicit, logs are streamed, and managed descendants are cleaned up on timeout, cancellation and normal exit. An interrupted CLI run retains evidence and returns 130.

See execution isolation and configuration and the end-to-end acceptance matrix. Docker is never silently replaced with local execution.

External CLI Agent repair

mimo repair --agent-config agent.json --task-file issue.txt --max-rounds 3 --name fix-001 runs a finite proposal → independent pytest acceptance → failure-feedback loop. Configure explicit defect target_tests first. Each round starts from the original baseline, preserves its cumulative patch and JSON/JUnit evidence, and cannot weaken the baseline acceptance files to obtain PASS. The original project remains available for review.

Agent execution uses explicit argv, separate credentials, timeout/output limits and durable invocation/time reservations shared across recovery branches. resume, report, export and the PASS-only gate support the complete repair workflow. External CLI tokens/costs are unavailable; configure actual spending limits in the Agent/provider.

Start with the complete free two-round example and Agent protocol. v0.10 also supplies a pinned Codex adapter and profile generator; live-model repair quality and provider billing remain separate acceptance work.

Try a real defect without a model account

mimo case-init --case boltons-write --output .mimo/cases/boltons
mimo case-check --directory .mimo/cases/boltons --name first

The prepared case reproduces an upstream defect, accepts its reference fix, rejects a regression, reruns recovery and exports portable evidence. It uses no API key or model call. A second independent case and exact source/fix identities are documented in real defect acceptance. Choose new output directories/run names to repeat; existing evidence is preserved.

Node/JUnit checks and product acceptance

v0.11 adds native Node and generic JUnit case evidence, including a pinned real ms defect. Both support independent repair, recovery and portable gates; a zero command exit still cannot establish PASS.

mimo case-init --case ms-invalid-number --output .mimo/cases/ms
mimo case-check --directory .mimo/cases/ms --name first
mimo readiness

readiness returns 3 while external acceptance is pending. Use trial-init/trial-run for an explicitly budgeted live model trial and walkthrough-init for each independent new user. Blank templates and deterministic controls do not count as those observations. See the acceptance guide and support/upgrade boundaries.

v0.12 adds walkthrough-pack: distribute one exact wheel, offline Python dependencies and three blank user-study slots. A cross-platform bootstrap creates a private environment, verifies the installed engine and preserves feedback when resuming interrupted setup. Actual user observations remain separate.

Continue an interrupted run

mimo resume --run .mimo/runs/first-run --name continued-run
mimo report --run .mimo/runs/continued-run

CLI runs save versioned input fingerprints and individual model checkpoints. Recovery preserves the original run, validates source/config/environment and evidence integrity, reuses completed model responses, and rebuilds/retests in fresh workspaces. Changed inputs return an error instead of reusing an old verdict. See run identities and recovery.

Export evidence and enforce a CI gate

mimo export --run .mimo/runs/first-run --output evidence.zip
mimo gate --bundle evidence.zip --json

Extract the ZIP and open index.html for an offline report with portable JSON/JUnit/log links. The strict gate recomputes the verdict and only returns success for an intact PASS; a successful export or NOT_APPLICABLE is not approval. See evidence bundles, exit codes and the opt-in PR workflow. Required merge checks must be enabled by the repository maintainer.

6.5 Configure Xiaomi MiMo API

Do not write your API key into source code or commit it to GitHub.

PowerShell example:

$env:MIMO_API_KEY='YOUR_MIMO_API_KEY'
$env:MIMO_BASE_URL='https://api.xiaomimimo.com/v1'
$env:MIMO_MODEL='mimo-v2.5-pro'

Create API-mode configuration in a project that has not been initialized yet:

mimo init --mode api
mimo doctor
mimo run --execute --apply-patches --run-tests

doctor fails closed when API mode has no real provider or its required key is missing. It never prints the secret value. To replace an existing generated configuration, review it first and then use mimo init --mode api --force deliberately.

The default is one network attempt. Configure retries and request/token/estimated-cost reservations in .mimo/models.json; resumed branches share their original budget. Incomplete or truncated model answers cannot be applied as completed patches. See model providers and execution limits, the reviewed product plan and remaining acceptance gates.

The legacy API connectivity check is still available:

python src\test_mimo_api_connection.py

Expected result:

[OK] API connected successfully.

---

7. Run Final Clean Case Studies

7.1 STM32 RFID case

python src\run_real_repo_case_studies.py --cases config\case_stm32_clean.local.json --suite-name final_stm32_v031_clean --config config\api_models_mimo.json --execute --clone-missing --apply-patches --limit-cases 1

Check result:

Get-Content outputs\mimo_codeharness_v02\final_stm32_v031_clean\case_study_summary.csv
Test-Path "outputs\mimo_codeharness_v02\final_stm32_v031_clean__stm32_rfid\dashboard.html"

7.2 SlideNotes case

python src\run_real_repo_case_studies.py --cases config\case_slidenotes_clean.local.json --suite-name final_slidenotes_v031_clean --config config\api_models_mimo.json --execute --clone-missing --apply-patches --limit-cases 1

Check result:

Get-Content outputs\mimo_codeharness_v02\final_slidenotes_v031_clean\case_study_summary.csv
Test-Path "outputs\mimo_codeharness_v02\final_slidenotes_v031_clean__slidenotes\dashboard.html"

---

8. Repository Structure

MiMo-CodeHarness/
├── pyproject.toml
├── mimo-harness.json          # generated by `mimo init`
├── .mimo/
│   ├── models.json            # generated model configuration, never a secret
│   └── runs/                  # generated and gitignored
├── config/
│   ├── api_models_mimo.json
│   ├── case_stm32_clean.local.json
│   └── case_slidenotes_clean.local.json
├── src/
│   ├── mimo_harness/
│   │   ├── cli.py
│   │   ├── repair.py
│   │   ├── structured_checks.py
│   │   ├── product_acceptance.py
│   │   ├── user_kit.py
│   │   ├── study_bootstrap.py
│   │   └── __main__.py
│   ├── run_real_repo_case_studies.py
│   ├── run_mimo_codeharness_v02.py
│   ├── test_mimo_api_connection.py
│   ├── validate_mimo_codeharness_v02.py
│   ├── scoring_v03.py
│   └── verification_v04.py
├── docs/
│   ├── PRODUCT_PLAN.md
│   ├── PRODUCT_ACCEPTANCE_CN.md
│   ├── INDEPENDENT_USER_KIT_CN.md
│   ├── CODEX_AGENT.md
│   ├── REAL_DEFECT_CASES.md
│   ├── STRUCTURED_CHECKS.md
│   └── ACCEPTANCE_MATRIX.md
├── tests/
│   ├── test_cli_repair.py
│   ├── test_product_acceptance.py
│   ├── test_user_kit.py
│   └── ci_user_kit.py
├── .github/workflows/
│   └── verification.yml
├── outputs/
│   └── mimo_codeharness_v02/
│       ├── final_stm32_v031_clean__stm32_rfid/
│       └── final_slidenotes_v031_clean__slidenotes/
├── PROJECT_STATUS.md
├── README.md
└── README_CN.md

---

9. Current Limitations

This project now has an installable beta CLI and evidence-backed verification semantics, but it is not yet a fully sandboxed industrial benchmark service.

Current limitations:

• A candidate receives INSUFFICIENT_EVIDENCE when no behavior test is detected or executed.
• The embedded STM32 case still does not perform full Keil firmware compilation; static checks alone are never reported as verified success.
• Local and venv execution run trusted project code with host permissions. Docker execution is currently supported on non-root Linux Docker workers; Windows Docker/WSL is not certified.
• The package is installable from the repository but is not yet published to PyPI.
• LLM Judge and human review are not used in the final clean scores.
• Scores should be interpreted as evidence-aware automatic evaluation results, not absolute ground truth.
• More repositories and more repeated trials are needed for stronger benchmark-level conclusions.

---

10. Roadmap

The product plan records implementation and acceptance separately. Python/Node/JUnit case evidence, run manifests, bounded repair/recovery, portable gates and the independent-user kit are implemented.

Before a stable release:

1. Run the prepared live-model trial with an explicit model and spending allowance; retain the actual outcome and available usage.
2. Observe three independent new users, resolve their reported obstacles and rerun the affected acceptance checks.
3. Have the maintainer choose the project license, review the source-bound CI/model/user evidence and complete final release sign-off. A stable GitHub Release/PyPI publication has not been performed.

Later extensions, prioritized after actual acceptance feedback, include broader repository/framework coverage, Keil/hardware adapters, additional container platforms, generalized cross-run caches, reusable project PR integration and a local/multi-model web interface. Judge calibration and research reports remain optional later work. These extensions are not part of the implemented v0.12 feature set.

---

11. Safety Notes

• Do not commit API keys.
• Use environment variables for all model credentials.
• Review generated patches before applying them to production code.
• Treat automatic scores as descriptive evidence, not final truth.
• Use the hard verification verdict and process exit code as the automation gate.