rewire.it benchmarks

MFASS correction, September 2026: The archived mfass-v1 baseline used a sequence field reverse-complemented for 7,770 variants while keeping the original assay coordinate. Its baseline comparisons and split-cost claims are superseded. See the validated assay-oriented mfass-v2 results in the MFASS benchmark README. V1 result files are preserved for audit.

Independent, reproducible benchmarks for AI models in biology.

Every benchmark here runs from public data, on open weights, on hardware a reader can afford, and reports what it cost as well as what it scored. The point is not to rank models. It is to give anyone a floor they can measure against and re-run themselves.

Project repositories

This is the public rewire-bio/rewire-benchmarks repository. It owns benchmark runners, model adapters, baseline implementations, evaluation protocols, split manifests and reproducible run artifacts.

Repository	Responsibility
rewire-benchmarks (this repository)	Execute declared evaluations and preserve reproducible artifacts
rewire-bio/rewire-database (private initially)	Review evidence, publish versioned records, and deploy the benchmark database
rewire-bio/rewire.it (private)	Write and publish articles and their illustrations

Each repository has an independent build and deployment. Running an evaluation here does not publish a database result or deploy either website. Repository ownership has changed; existing commits, personal attribution and archived scientific artifacts remain intact.

From a run to a published result

1. Commit the protocol, configuration, split identity, provenance and reviewable outputs in this repository. Keep downloaded source data, model weights and credentials out of Git.
2. In the database repository, identify the source using the full commit SHA and the exact artifact paths and SHA-256 hashes. Do not ingest mutable branch names or a moving “latest” file as evidence.
3. Stage and review the result in the database. Check the evaluation protocol, metrics, coverage, uncertainty and evidence locations against the pinned artifacts. A successful runner exit is not publication approval.
4. Publish accepted records through a versioned database release. Keep their evidence links pinned to the runner revision. Corrections receive new records or versions with explicit supersession; archived artifacts remain available.

The database's collection tools own this ingestion and review process. This repository does not require website or contribution-service credentials. For the current MFASS v2 evidence, retain the reviewed revision bee9133b83f3aedaf2bbb9013f1875515845607e; a documentation update does not change that scientific source identity.

Baselines, model evaluations and HPC

The baseline programme provides registered controls, frozen ESM-2 examples and an explicit private submission queue. The Nextflow workflow prepares, runs, rescores and exports model matrices, with separate profiles for local execution and future HPC/cloud runs. Implemented adapters do not imply complete catalogue coverage; blocked baseline choices and unexecuted deployment profiles remain explicit.

House rules

These are enforced in rewirebench, not merely described here. A result that breaks one is refused rather than written.

1. A trivial baseline always runs, and is given a fair budget. A leaderboard without a floor is misleading, and in this field the floor frequently wins.
2. The grouping rule and the independent-group count are published with every result. Splits leak in ways that do not show up in the output file.
3. Contamination is stated for every pretrained method: what was checked, and what is unknown. "unknown" is an acceptable answer. Silence is not, because a reader cannot tell it apart from clean.
4. Coverage reconciles against the original denominator. A method that cannot score an input has a coverage problem, not a negative prediction.
5. Throughput sits beside accuracy, measured end to end rather than on the forward pass alone.
6. Configuration is recorded for every pretrained method: checkpoint revision, pooling, context in both bases and tokens, precision, batch size.

Benchmarks

Benchmark	Task	Labels	Prevalence	Status
mfass-v1	Splice-variant prioritisation	MFASS minigene exon recognition	3.78%	Archived; baseline superseded after sequence correction
mfass-v2	Splice-variant prioritisation	Same functional cohort and canonical exon/gene split	3.78%	Corrected baseline and local DNABERT-2; specialists carried unchanged

MFASS-v2 results

The source assay's original_seq and natural_seq differ at the variant's assay-coordinate position for all 27,733 eligible rows. The legacy sequence field is reversed for 7,770 of them; mfass-v2 validates the pair and uses the assay-oriented mutant. The v1 result files are retained for audit, but v1 baseline comparisons and split-cost claims are withdrawn.

Method	Protocol	P@100	AP	AUROC	Coverage
Corrected k-mer/position baseline	supervised on MFASS train	0.610	0.286	0.778	8,324/8,324
SpliceAI 1.3.1	zero-shot specialist, unchanged	0.640	0.299	0.806	8,194/8,324
Pangolin, mask=False	zero-shot specialist, unchanged	0.650	0.389	0.876	8,301/8,324
DNABERT-2 117M frozen pair + logistic head	supervised head on MFASS train	0.030	0.045	0.550	8,324/8,324

Paired whole-exon/gene-group resampling on common variants shows Pangolin still exceeds the corrected baseline on AP and AUROC. SpliceAI's corrected AUROC difference is +0.028 with a 95% interval [-0.004, +0.064], so the old claim of clear separation is invalid. The locally run DNABERT-2 frozen-pair protocol is worse than the trivial baseline: P@100 -0.580 [-0.713, -0.464], AP -0.241 [-0.304, -0.182], and AUROC -0.228 [-0.300, -0.160]. This is one declared representation and train-only head, not a model-wide verdict. Full method, access, coverage, timing, and source limits are in the MFASS benchmark report, with exact artifact hashes in the run provenance manifest.

Running

Requires uv.

uv sync --package mfass --extra dnabert2-pilot

# Public author-maintained source tables; their exact hashes are in the MFASS README.
curl -L -o benchmarks/mfass/data/snv_data_clean.txt \
  https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_data_clean.txt
curl -L -o benchmarks/mfass/data/snv_func_annot.txt \
  https://raw.githubusercontent.com/KosuriLab/MFASS/master/processed_data/snv/snv_func_annot.txt

uv run --package mfass --extra dnabert2-pilot mfass-build --check
uv run --package mfass --extra dnabert2-pilot mfass-build
# Use the checked-in split-v2.tsv; do not overwrite it with a new draw.
uv run --package mfass --extra dnabert2-pilot mfass-baseline
uv run --package mfass --extra dnabert2-pilot mfass-dnabert2-pilot
# If the pilot passes its 12-hour projection and 1 GiB disk reserve:
uv run --package mfass --extra dnabert2-pilot mfass-dnabert2

Tests:

uv run --group dev pytest -q

Adding a benchmark

1. benchmarks/<name>/ with a pyproject.toml depending on rewirebench, added to the workspace.
2. A build_dataset step that asserts the cohort against the source publication's own totals and refuses to write on drift. Cohort definitions move silently otherwise.
3. A split step using rewirebench.splits.connected_components, reporting the naive concatenated-key count beside the real group count so key inflation stays visible.
4. A trivial baseline, before any pretrained model.
5. Results written through rewirebench.results.write_result, which enforces the house rules.

Licensing and data

The code in this repository is MIT licensed. The benchmark data is not ours and is not redistributed.

• Code (packages/, benchmarks/*/src/): MIT, see LICENSE.
• Source data: downloaded at run time from the original authors and gitignored. MFASS comes from KosuriLab/MFASS, which declares no licence, so it remains the authors' work under their terms. Cite Chong et al., Molecular Cell 2018.
• Derived results (benchmarks/*/results/, benchmarks/*/splits/): committed so a reader can verify a comparison without rerunning a specialist. The prediction tables carry the upstream assay label beside our scores, for attribution and verification only. Anyone building on the labels themselves should take them from the original repository, which is the authoritative copy.
• Model weights: never redistributed. SpliceAI is PolyForm Strict 1.0.0 code with CC BY-NC 4.0 weights; Pangolin is GPL v3. Both are installed from upstream at run time under their own terms.

Reproducibility

Results carry the git revision, UTC timestamp, Python version and platform. Data directories are gitignored and fully regenerated by the commands above; split manifests and results are committed so a change in either is visible in a diff.

Source files are pinned by SHA-256 in the cohort summary, so an upstream edit shows up as a reconciliation failure rather than as a quietly different number.

Python SDK and private models

The SDK provides local prepare, run, evaluate, export and explicit submit interfaces. Bring a private adapter, keyed predictions or frozen embeddings.

Protocol	Supported scope	Instructions
MFASS v2	Corrected assay pairs, scalar scores and frozen paired embeddings	MFASS
ProteinGym	v1.3 zero-shot DMS substitutions	ProteinGym
Genomic Benchmarks	Nine sequence classification datasets	Genomic Benchmarks
TDC ADMET	22 dataset-specific ADMET tasks	TDC ADMET
FLIP2	Seven datasets, 16 archived fitness splits, one split per evaluation	FLIP2
DART-Eval	Task 1 paired regulatory elements and controls; authenticated source access	DART-Eval
mRNABench Sample	Four MRL datasets, six targets scored separately	mRNABench

Start with the SDK guide and HPC packaging. Use rewirebench inspect PROTOCOL_ID for exact supported inputs and configurations. The new sequence environment adds parquet and HDF5 readers without downloading model weights. Podman and Apptainer use the same interfaces. A supported protocol is not a claim that every published evaluation has been reproduced.

Submission goes to verified-email private review, never directly to publication. As of 22 September 2026, production intake is enabled for verified contributors; notifications remain paused. Track submissions on the signed-in contribution page. The latest published SDK is v0.5.0. Release validation records exact source, artifact hashes and tested platforms. Existing scientific artifacts are preserved.

Use with an agent

The bio-model-benchmarks plugin gives a coding agent two things: a skill that explains how to find published results and run each protocol with rewirebench, and the read-only benchmark knowledge graph as an MCP server named rewire-benchmark-graph. Runs stay local. The skill never exports or submits a result unless you ask for it.

In Claude Code:

/plugin marketplace add rewire-bio/rewire-benchmarks
/plugin install bio-model-benchmarks@rewire-benchmarks

Claude Code 2.1.275 or later can do both in one step:

/plugin install bio-model-benchmarks --marketplace rewire-bio/rewire-benchmarks

The MCP server needs uv; it downloads and verifies the current graph bundle on first start. Then ask, for example, "Reproduce the published MFASS-v2 baseline and compare it with the result in the benchmark graph."

For other MCP clients, add the server:

{
  "mcpServers": {
    "rewire-benchmark-graph": {
      "command": "uvx",
      "args": [
        "--from", "git+https://github.com/rewire-bio/rdf-kg-mcp@v0.1.0",
        "rdf-kg-mcp", "serve", "https://benchmarks.rewirebio.io/agents/kg.tar.gz"
      ]
    }
  }
}

That URL redirects to the latest graph release. For an exact, reproducible graph, use a release asset instead, such as https://github.com/rewire-bio/rewire-benchmark-data/releases/download/kg/2026-10-12-cba623554ce3/kg-2026-10-12-cba623554ce3.tar.gz. Then give the agent the skill: copy skills/bio-model-benchmarks into the client's skills directory, or paste SKILL.md into its instructions and keep the reference/ files beside it. The copy under plugins/ is kept byte-identical by tests/test_agent_plugin.py; edit skills/bio-model-benchmarks and copy it across.

Cleaning local reference data

Preview old expanded reference files whose adjacent gzip archives contain exactly the same bytes:

uv run rewirebench clean --root .
uv run rewirebench clean --root . --apply

Stop local benchmark runs before applying. The default minimum age is seven days; override it with --older-than-days. The command verifies both copies with SHA-256 and keeps the compressed source. It only considers FASTA, GTF and GFF files under benchmarks/*/data/. Results, release archives, annotation databases and other datasets are retained. Symlinks and files that change during verification are skipped. Reported bytes are logical file sizes; actual freed space can differ.

Restore an expanded file before running a benchmark that needs it:

gzip -dk benchmarks/mfass/data/ref/GRCh38.primary_assembly.genome.fa.gz

Python callers can use rewirebench.cleanup.clean(root, older_than_days=7, apply=False).