kevinhyj · Sep 9, 2026 · Sep 9, 2026
diff --git a/README.md b/README.md
@@ -169,14 +169,14 @@ paper experiment. See [dated validation records](docs/RELEASE_STATUS.md).
 |---|---|
 | `eva/`, `tools/` | Model, tokenizer and inference CLIs |
 | `training/`, `finetune/`, `config/` | Training entry points and configurations |
-| `scripts/`, `notebooks/` | Reproduction tools, analysis and benchmark inputs |
-| `reproduction/` | Versioned inputs, historical sources and validation records |
+| `scripts/` | Command-line workflows and automation |
+| [`notebooks/`](notebooks/README.md) | Interactive analysis, plots and supporting data |
+| [`reproduction/`](reproduction/README.md) | Fixed inputs, protocols, reference outputs and historical code |
 | `docs/`, `tests/` | Guides and regression tests |
 
-The wheel contains the importable model and CLI packages. Use the complete
+Notebooks and command-line workflows share the reproduction resources.
+The wheel contains the importable model and CLI packages; use the complete
 source checkout for training, notebooks and paper-reproduction inputs.
-Historical sources under `reproduction/` are labeled separately and may preserve
-old paths; use the documented entry points for current workflows.
 
 </details>
 

diff --git a/docs/BENCHMARK_PROTOCOLS.md b/docs/BENCHMARK_PROTOCOLS.md
@@ -27,7 +27,13 @@ archive calculation from a completed model reproduction.
 
 ## Comparison-model execution, not plotting
 
-`reproduction/benchmark/run_competitor.py` supplies fail-closed parameterized entry points. It does not download models, select checkpoints or infer original paper protocols. Dependencies must be installed in the chosen model's compatible environment; environment conflicts are not hidden by fallback implementations.
+`reproduction/benchmark/run_competitor.py` accepts locally prepared checkpoints
+for its supported scoring interfaces. Models are loaded from `--model-dir`;
+the RNA interface uses `local_files_only=True`. Prepare the model, tokenizer and
+dependencies in the chosen model's compatible environment before running it.
+For other model interfaces, use the upstream scorer and the common prediction
+evaluator described below. Record checkpoint versions and scoring settings with
+your results; a specific paper comparison uses that paper's protocol.
 
 1. Snapshot an already acquired, version-pinned local model. Keep the manifest **outside** the model directory; subsequent scoring requires the exact file set and hashes.
 
@@ -72,6 +78,32 @@ The evaluator rejects missing/duplicate IDs, mismatched sequences, nonfinite val
 
 The portable RNA scorer passed a real CPU forward/save/reload test with a randomly initialized tiny BERT and character tokenizer, including mask-batch equality and explicit length-policy checks. This tests the execution machinery only, not any released competitor's predictions.
 
+### Using your own model outputs
+
+You can run a third-party model in its own environment and evaluate its exported
+predictions without loading that model in EVA. The metric step uses Python's
+standard library and requires no network connection or model weights.
+
+| File | Required CSV columns |
+|---|---|
+| Predictions | `variant_id,sequence,score` |
+| Labels | `variant_id,sequence,label` |
+
+Supply one row per assay variant in each file. IDs and sequences must match;
+all scores and labels must be finite, and both vectors must vary. Record the
+scoring convention, model version and environment alongside the CSV. The
+evaluator calculates signed Spearman using the supplied scores.
+
+```bash
+python reproduction/benchmark/run_competitor.py evaluate \
+  --predictions results/my_model/predictions.csv \
+  --labels data/verified_labels.csv --output results/my_model/metrics.json
+```
+
+Use the [benchmark notebook](../notebooks/prediction/benchmark_reproduction.ipynb)
+and its evaluation manifest when comparing keyed predictions across multiple
+datasets. This gives local model deployments a common analysis entry point.
+
 ### Coverage and remaining external resources
 
 | Family | Recovered execution source | Remaining requirement / verification boundary |
@@ -101,6 +133,6 @@ The separately recovered `RNAVerse/checkpoint/clm` metadata supplies a fifth con
 
 The [current resource index](OFFICIAL_MODEL_RESOURCES.md#recovered-comparison-model-files-september-9-2026)
 records recovered RNA-FM/RNABERT/RNA-MSM snapshots and additional cached
-CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. Public download/runtime
-bindings remain incomplete. These new identities supplement the historical
-requirements above; they do not imply newly executed competitor benchmarks.
+CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. These can identify matching
+local files; the environment and scoring protocol must be recorded separately.
+The records describe recovered artifacts, not additional benchmark runs.
diff --git a/docs/OFFICIAL_MODEL_RESOURCES.md b/docs/OFFICIAL_MODEL_RESOURCES.md
@@ -1,46 +1,42 @@
-# Official comparison-model resources
+# Comparison-model resources
 
-Links checked on September 9, 2026. These are resource and runtime entry points,
-not new claims of paper checkpoint identity. Use the recorded original revision
-when it is available; unverified current weights are not substituted. Exact
-execution gaps are listed in [PAPER_WORKFLOWS.md](PAPER_WORKFLOWS.md).
+Third-party models can be prepared locally and run in their own environments.
+Use the supported [local-checkpoint interfaces](BENCHMARK_PROTOCOLS.md#comparison-model-execution-not-plotting),
+or export predictions from an upstream scorer to the
+[common metric evaluator](BENCHMARK_PROTOCOLS.md#using-your-own-model-outputs).
+The benchmark step does not require downloading a model from its host.
 
-| Method | Upstream resources | Environment boundary |
+| Method | Resources | Components to prepare locally |
 |---|---|---|
-| RNA-FM, RNABERT, RNA-MSM | [RNA-FM upstream](https://github.com/ml4bio/RNA-FM); the historical Multimolecule ports use three revisions that still return HTTP 404 | Matching cached checkpoint/tokenizer hashes are now recorded below; public download and the original Multimolecule runtime remain unresolved. |
-| GENA-LM | [Official code and models](https://github.com/AIRI-Institute/GENA_LM) | DNA tokenizer/custom-code runtime; recorded snapshot resolves but model-name binding is unresolved. |
-| ERNIE-RNA | [Official repository and checkpoint instructions](https://github.com/Bruce-ywj/ERNIE-RNA/blob/main/README.md) | Use its fairseq environment and original pretrained checkpoint, not a structure fine-tuned model. |
-| Evo2 | [Official models and runtime](https://github.com/arcinstitute/evo2) | Separate Vortex/Transformer Engine environment; exact paper model and runtime revisions remain required. |
-| CodonFM | [Official repository and model downloads](https://github.com/NVIDIA-BioNeMo/CodonFM) | Cached 600M/1B weights and config hashes are recorded below; public revision/runtime linkage is still required. |
-| AIDO.RNA | [ModelGenerator](https://github.com/genbio-ai/ModelGenerator), [AIDO model collection](https://github.com/genbio-ai/AIDO) | Match the paper backbone and ModelGenerator API version; current releases do not prove compatibility with the historical worker. |
-| GenerRNA | [Official model checkpoint and tokenizer](https://huggingface.co/pfnet/GenerRNA) | Cached model_updated.pt and tokenizer-config hashes are recorded below; the complete tokenizer/runtime and public-version binding remain required. |
-| CodonGPT | The paper-linked NanilTx/codonGPT_pub repository returned HTTP 404 in this check | Obtain the original repository/version and weights; a similarly named software project is not a substitute. |
-| ESM | [Official model downloads and fair-esm environment](https://github.com/facebookresearch/esm) | Recover exact model size/ensemble and original ProteinGym reference/DMS inputs. |
-| ProGen3 | [Official models, Docker runtime and scoring CLI](https://github.com/Profluent-AI/progen3) | Official scorer averages forward/reverse likelihood; verify this against the original EVA comparison protocol and fix the corresponding weight revision. |
+| RNA-FM, RNABERT, RNA-MSM | [RNA-FM upstream](https://github.com/ml4bio/RNA-FM); [recorded model identities](../reproduction/release/recovered_rna_snapshots_20260909.json) | Model/config/tokenizer snapshot and a compatible Transformers/Multimolecule environment; choose the appropriate masking, length and sequence/MSA protocol. |
+| GENA-LM | [Official code and models](https://github.com/AIRI-Institute/GENA_LM) | Checkpoint, DNA tokenizer and any required custom model code. |
+| ERNIE-RNA | [Official repository](https://github.com/Bruce-ywj/ERNIE-RNA/blob/main/README.md) | ERNIE/fairseq environment and the pretrained checkpoint selected for the comparison. |
+| Evo2 | [Official models and runtime](https://github.com/arcinstitute/evo2) | Selected model size/checkpoint and its Vortex/Transformer Engine environment. |
+| CodonFM | [Official repository](https://github.com/NVIDIA-BioNeMo/CodonFM) | Checkpoint, configuration, CodonFM environment and preprocessing settings. |
+| AIDO.RNA | [ModelGenerator](https://github.com/genbio-ai/ModelGenerator), [AIDO](https://github.com/genbio-ai/AIDO) | ModelGenerator environment, matching API and local checkpoint. |
+| GenerRNA | [Official checkpoint and tokenizer](https://huggingface.co/pfnet/GenerRNA) | Model, complete tokenizer and the upstream likelihood-scoring implementation. |
+| CodonGPT | [Preserved scoring worker](../reproduction/benchmark/upstream/rna/external_scripts/compute_codongpt_ll.py) | Compatible checkpoint/tokenizer, Transformers version and codon preprocessing. |
+| ESM | [Official models and fair-esm](https://github.com/facebookresearch/esm) | Local weights and companion files, reference/DMS inputs and the selected scoring strategy or ensemble. |
+| ProGen3 | [Official models and scoring CLI](https://github.com/Profluent-AI/progen3) | Local weights and the official runtime; record the scorer's forward/reverse convention. |
 
-`reproduction/release/competitor_resource_verification.json` records the
-read-only Hugging Face revision checks and available LFS object hashes.
-No third-party weight download or inference is claimed by this resource index.
+Use whichever local model version your benchmark specifies and record its
+identity, environment and scoring protocol with the output. Reproducing a
+particular published row additionally requires that row's checkpoint and
+protocol. [Paper workflow coverage](PAPER_WORKFLOWS.md) records the established
+historical bindings and the components still needed for that purpose.
 
 ## Recovered comparison-model files: September 9, 2026
 
-The original RNA-FM, RNABERT and RNA-MSM cache snapshots were recovered and
-all 15 weight/config/tokenizer files were hashed. Their model configurations
-record Transformers 4.50.0; the exact Multimolecule runtime is still unbound.
-The [recovered identity record](../reproduction/release/recovered_rna_snapshots_20260909.json)
-contains each historical revision and every file SHA256. Ready-to-use manifests
-for the existing `--model-manifest` interface are in
-`reproduction/release/model_manifests/`. Keep a manifest outside its model folder.
+The [RNA snapshot record](../reproduction/release/recovered_rna_snapshots_20260909.json)
+contains revision identifiers and SHA256 values for 15 RNA-FM, RNABERT and
+RNA-MSM weight/config/tokenizer files. Their saved configs record Transformers
+4.50.0; the exact historical Multimolecule version has not been established.
+Matching local files can use the supplied manifests in
+`reproduction/release/model_manifests/`. Keep each manifest outside its model
+folder. For a different local checkpoint, create a manifest with the
+[existing snapshot command](BENCHMARK_PROTOCOLS.md#comparison-model-execution-not-plotting).
 
-The historical public revisions still return 404. Current public weights have
-different file hashes; no tensor-level equivalence is claimed and no current
-weight is substituted. The recovered files are not rehosted in this source
-repository. Original environment/protocol correspondence and a working public
-download remain required before these methods can be claimed fully reproducible.
-
-The [additional recovered-asset record](../reproduction/release/recovered_competitor_assets_20260909.json)
-adds 13 file identities from CodonFM 600M/1B, ERNIE-RNA, AIDO.RNA and GenerRNA
-caches. These include configurations and the ERNIE environment-file hash; the
-two ERNIE copies have identical hashes. A local cached weight is not itself a
-public download or proof that its full environment/assay protocol is recovered.
-No additional competitor inference was performed in this repository update.
+The [additional asset record](../reproduction/release/recovered_competitor_assets_20260909.json)
+contains 13 file identities from CodonFM 600M/1B, ERNIE-RNA, AIDO.RNA and
+GenerRNA caches, including an ERNIE environment-file hash. These records help
+identify local assets; execution scope is documented in the workflow guide.
diff --git a/docs/PAPER_WORKFLOWS.md b/docs/PAPER_WORKFLOWS.md
@@ -22,44 +22,43 @@ this is not the complete set of manuscript experiments or all plotted models.
 The preserved original configuration is
 `reproduction/benchmark/upstream/rna/config.yaml`. Historical absolute paths
 identify earlier resources; do not execute that configuration unchanged.
-Use separate method environments to avoid incompatible dependencies.
-The entries below distinguish recovered version identifiers from missing ones.
+Use separate method environments and local checkpoint paths, or export
+predictions from an upstream scorer to the
+[common evaluator](BENCHMARK_PROTOCOLS.md#using-your-own-model-outputs).
+The version identifiers below describe the original comparison records. New
+benchmarks can use other locally prepared versions, recorded with their results.
 
 | Method | Model resource identified in original source | Entry and remaining requirement |
 |---|---|---|
-| RNA-FM | `multimolecule/rnafm`, revision `2a98c38e8219d666ad35d04a689218ab50a36293` | `run_competitor.py rna-mlm`; Multimolecule registration and the method's original masking/length settings are required. The historical revision currently returns HTTP 404; no public version is substituted or newly run. |
-| RNABERT | `multimolecule/rnabert`, revision `7a9b5d5d5a96931ff056828c0968479098506cea` | Same bounded masked-LM entry; historical revision currently returns HTTP 404. |
-| RNA-MSM | `multimolecule/rnamsm`, revision `ad61a4915674596557b3c3c4fe4fd7162d36acc8` | Historical revision currently returns HTTP 404; sequence/MSA protocol correspondence is also unverified. |
+| RNA-FM | `multimolecule/rnafm`, revision `2a98c38e8219d666ad35d04a689218ab50a36293` | `run_competitor.py rna-mlm`; Multimolecule registration and the method's original masking/length settings are required. Use a locally prepared model/tokenizer snapshot and record its file hashes. |
+| RNABERT | `multimolecule/rnabert`, revision `7a9b5d5d5a96931ff056828c0968479098506cea` | Use the same bounded masked-LM entry with a local snapshot and compatible runtime. |
+| RNA-MSM | `multimolecule/rnamsm`, revision `ad61a4915674596557b3c3c4fe4fd7162d36acc8` | Use a local snapshot; the original sequence/MSA protocol correspondence still requires confirmation for the paper comparison. |
 | GENA-LM | Local snapshot names `AIRI-Institute/gena-lm-bert-base-t2t-multi`, revision `4633e5a1ada905bb7afee6877d71cc12578a95a5` | Original model-name field differs from local snapshot name. Resolve that binding before downloading or running a replacement. |
-| ERNIE-RNA | `ERNIE-RNA_pretrain.pt` | Preserved ERNIE adapter and worker; cached weight and environment-file hashes are now recorded; matching public release and runtime execution remain unverified. |
+| ERNIE-RNA | `ERNIE-RNA_pretrain.pt` | Preserved ERNIE adapter and worker; cached weight and environment-file hashes are now recorded; the exact paper runtime has not been verified. |
 | Evo2 1B, 7B, 40B | `evo2_1b_base`, `evo2_7b_base`, `evo2_40b_base` | Preserved Evo adapter; exact revisions and official compatible runtime still required. Do not replace the original scoring objective. |
-| CodonFM 600M and 1B | Original 600m/1b model selectors | Preserved `score_codonfm.py` and adapter; cached weight/config identities are now recorded; original runtime and public version binding remain unresolved. |
-| AIDO.RNA | Original model-weight folder and ModelGenerator framework | Preserved AIDO worker/adapter; cached weight-shard/config hashes are now recorded; exact public snapshot and environment binding remain required. |
-| GenerRNA | `model_updated.pt` and separate tokenizer | Preserved GenerRNA worker/adapter; cached weight and tokenizer-config hashes are now recorded; complete tokenizer/runtime and public version binding remain required. |
+| CodonFM 600M and 1B | Original 600m/1b model selectors | Preserved `score_codonfm.py` and adapter; cached weight/config identities are now recorded; the original runtime and paper checkpoint binding remain to be established. |
+| AIDO.RNA | Original model-weight folder and ModelGenerator framework | Preserved AIDO worker/adapter; cached weight-shard/config hashes are now recorded; the paper snapshot and environment binding remain required. |
+| GenerRNA | `model_updated.pt` and separate tokenizer | Preserved GenerRNA worker/adapter; cached weight and tokenizer-config hashes are now recorded; the complete tokenizer/runtime and paper checkpoint binding remain required. |
 | CodonGPT | Original worker `compute_codongpt_ll.py` | Weight revision and original environment still required. |
 | ESM protein models | `upstream/protein/evaluate_dms_esm_domain.py` | Strict native entry `run_competitor.py protein-esm`; exact reference/DMS files and model ensemble must be supplied. |
 | ProGen3 and other unmatched protein rows | Existing summary-table entries | Missing producer/version must be recovered; a table row is not an inference implementation. |
 
-The three historical Multimolecule revision IDs returned HTTP 404 in the
-current public API check. They are retained as original configuration evidence,
-not presented as working download commands. The GENA-LM revision resolves,
-but its original model-name/snapshot mismatch still needs clarification.
-Do not substitute current `main` weights and call them the paper model.
-Once an original snapshot is recovered, its files can be recorded with:
+Record the exact files used by a local model before scoring. A supplied
+manifest identifies a matching historical snapshot; a newly generated manifest
+identifies the checkpoint you have chosen for a new benchmark:
 
 ```bash
 python reproduction/benchmark/run_competitor.py snapshot \
   --model-dir checkpoint/rnafm --output checkpoint/rnafm-manifest.json
 ```
 
 [Official resource links](OFFICIAL_MODEL_RESOURCES.md) identify upstream model
-and runtime sources. These links do not establish which upstream version
-produced an EVA paper result.
+and runtime sources. For an exact paper comparison, use the corresponding checkpoint and scoring
+protocol from the original run.
 
 See [BENCHMARK_PROTOCOLS.md](BENCHMARK_PROTOCOLS.md) for the parameterized
 masked-LM and native ESM interfaces. The snapshot helper records identity; it
 does not determine whether a checkpoint is the one used for a paper result.
-No unverified version is silently filled with a current `main` revision.
 
 The recovered Evo2 7B protein row remains a separate candidate table under
 `reproduction/benchmark_release/recovered_reference/`. It is not substituted
@@ -71,6 +70,6 @@ to be documented before those comparisons can be called complete.
 
 The [current resource index](OFFICIAL_MODEL_RESOURCES.md#recovered-comparison-model-files-september-9-2026)
 records recovered RNA-FM/RNABERT/RNA-MSM snapshots and additional cached
-CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. Public download/runtime
-bindings remain incomplete. These new identities supplement the historical
-requirements above; they do not imply newly executed competitor benchmarks.
+CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. These can identify matching
+local files; the environment and scoring protocol must be recorded separately.
+The records describe recovered artifacts, not additional benchmark runs.