From 6919809d85dd9fe1c7ee6932dac73d83a09fd7b3 Mon Sep 17 00:00:00 2001 From: Yanjie Huang Date: Wed, 9 Sep 2026 00:11:26 -0700 Subject: [PATCH] Document local model benchmarking and explain reproduction folders --- README.md | 10 ++--- docs/BENCHMARK_PROTOCOLS.md | 40 ++++++++++++++++-- docs/OFFICIAL_MODEL_RESOURCES.md | 72 +++++++++++++++----------------- docs/PAPER_WORKFLOWS.md | 41 +++++++++--------- docs/RELEASE_STATUS.md | 5 ++- docs/REPRODUCTION.md | 11 +++++ docs/RESOURCES.md | 27 ++++++------ notebooks/README.md | 31 ++++++++++++++ reproduction/README.md | 31 ++++++++++++++ scripts/check_repository.py | 3 +- 10 files changed, 185 insertions(+), 86 deletions(-) create mode 100644 notebooks/README.md create mode 100644 reproduction/README.md diff --git a/README.md b/README.md index 140f19c..795578f 100644 --- a/README.md +++ b/README.md @@ -169,14 +169,14 @@ paper experiment. See [dated validation records](docs/RELEASE_STATUS.md). |---|---| | `eva/`, `tools/` | Model, tokenizer and inference CLIs | | `training/`, `finetune/`, `config/` | Training entry points and configurations | -| `scripts/`, `notebooks/` | Reproduction tools, analysis and benchmark inputs | -| `reproduction/` | Versioned inputs, historical sources and validation records | +| `scripts/` | Command-line workflows and automation | +| [`notebooks/`](notebooks/README.md) | Interactive analysis, plots and supporting data | +| [`reproduction/`](reproduction/README.md) | Fixed inputs, protocols, reference outputs and historical code | | `docs/`, `tests/` | Guides and regression tests | -The wheel contains the importable model and CLI packages. Use the complete +Notebooks and command-line workflows share the reproduction resources. +The wheel contains the importable model and CLI packages; use the complete source checkout for training, notebooks and paper-reproduction inputs. -Historical sources under `reproduction/` are labeled separately and may preserve -old paths; use the documented entry points for current workflows. diff --git a/docs/BENCHMARK_PROTOCOLS.md b/docs/BENCHMARK_PROTOCOLS.md index 48823cd..a5a2bba 100644 --- a/docs/BENCHMARK_PROTOCOLS.md +++ b/docs/BENCHMARK_PROTOCOLS.md @@ -27,7 +27,13 @@ archive calculation from a completed model reproduction. ## Comparison-model execution, not plotting -`reproduction/benchmark/run_competitor.py` supplies fail-closed parameterized entry points. It does not download models, select checkpoints or infer original paper protocols. Dependencies must be installed in the chosen model's compatible environment; environment conflicts are not hidden by fallback implementations. +`reproduction/benchmark/run_competitor.py` accepts locally prepared checkpoints +for its supported scoring interfaces. Models are loaded from `--model-dir`; +the RNA interface uses `local_files_only=True`. Prepare the model, tokenizer and +dependencies in the chosen model's compatible environment before running it. +For other model interfaces, use the upstream scorer and the common prediction +evaluator described below. Record checkpoint versions and scoring settings with +your results; a specific paper comparison uses that paper's protocol. 1. Snapshot an already acquired, version-pinned local model. Keep the manifest **outside** the model directory; subsequent scoring requires the exact file set and hashes. @@ -72,6 +78,32 @@ The evaluator rejects missing/duplicate IDs, mismatched sequences, nonfinite val The portable RNA scorer passed a real CPU forward/save/reload test with a randomly initialized tiny BERT and character tokenizer, including mask-batch equality and explicit length-policy checks. This tests the execution machinery only, not any released competitor's predictions. +### Using your own model outputs + +You can run a third-party model in its own environment and evaluate its exported +predictions without loading that model in EVA. The metric step uses Python's +standard library and requires no network connection or model weights. + +| File | Required CSV columns | +|---|---| +| Predictions | `variant_id,sequence,score` | +| Labels | `variant_id,sequence,label` | + +Supply one row per assay variant in each file. IDs and sequences must match; +all scores and labels must be finite, and both vectors must vary. Record the +scoring convention, model version and environment alongside the CSV. The +evaluator calculates signed Spearman using the supplied scores. + +```bash +python reproduction/benchmark/run_competitor.py evaluate \ + --predictions results/my_model/predictions.csv \ + --labels data/verified_labels.csv --output results/my_model/metrics.json +``` + +Use the [benchmark notebook](../notebooks/prediction/benchmark_reproduction.ipynb) +and its evaluation manifest when comparing keyed predictions across multiple +datasets. This gives local model deployments a common analysis entry point. + ### Coverage and remaining external resources | Family | Recovered execution source | Remaining requirement / verification boundary | @@ -101,6 +133,6 @@ The separately recovered `RNAVerse/checkpoint/clm` metadata supplies a fifth con The [current resource index](OFFICIAL_MODEL_RESOURCES.md#recovered-comparison-model-files-september-9-2026) records recovered RNA-FM/RNABERT/RNA-MSM snapshots and additional cached -CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. Public download/runtime -bindings remain incomplete. These new identities supplement the historical -requirements above; they do not imply newly executed competitor benchmarks. +CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. These can identify matching +local files; the environment and scoring protocol must be recorded separately. +The records describe recovered artifacts, not additional benchmark runs. diff --git a/docs/OFFICIAL_MODEL_RESOURCES.md b/docs/OFFICIAL_MODEL_RESOURCES.md index 58a83e7..2a8568b 100644 --- a/docs/OFFICIAL_MODEL_RESOURCES.md +++ b/docs/OFFICIAL_MODEL_RESOURCES.md @@ -1,46 +1,42 @@ -# Official comparison-model resources +# Comparison-model resources -Links checked on September 9, 2026. These are resource and runtime entry points, -not new claims of paper checkpoint identity. Use the recorded original revision -when it is available; unverified current weights are not substituted. Exact -execution gaps are listed in [PAPER_WORKFLOWS.md](PAPER_WORKFLOWS.md). +Third-party models can be prepared locally and run in their own environments. +Use the supported [local-checkpoint interfaces](BENCHMARK_PROTOCOLS.md#comparison-model-execution-not-plotting), +or export predictions from an upstream scorer to the +[common metric evaluator](BENCHMARK_PROTOCOLS.md#using-your-own-model-outputs). +The benchmark step does not require downloading a model from its host. -| Method | Upstream resources | Environment boundary | +| Method | Resources | Components to prepare locally | |---|---|---| -| RNA-FM, RNABERT, RNA-MSM | [RNA-FM upstream](https://github.com/ml4bio/RNA-FM); the historical Multimolecule ports use three revisions that still return HTTP 404 | Matching cached checkpoint/tokenizer hashes are now recorded below; public download and the original Multimolecule runtime remain unresolved. | -| GENA-LM | [Official code and models](https://github.com/AIRI-Institute/GENA_LM) | DNA tokenizer/custom-code runtime; recorded snapshot resolves but model-name binding is unresolved. | -| ERNIE-RNA | [Official repository and checkpoint instructions](https://github.com/Bruce-ywj/ERNIE-RNA/blob/main/README.md) | Use its fairseq environment and original pretrained checkpoint, not a structure fine-tuned model. | -| Evo2 | [Official models and runtime](https://github.com/arcinstitute/evo2) | Separate Vortex/Transformer Engine environment; exact paper model and runtime revisions remain required. | -| CodonFM | [Official repository and model downloads](https://github.com/NVIDIA-BioNeMo/CodonFM) | Cached 600M/1B weights and config hashes are recorded below; public revision/runtime linkage is still required. | -| AIDO.RNA | [ModelGenerator](https://github.com/genbio-ai/ModelGenerator), [AIDO model collection](https://github.com/genbio-ai/AIDO) | Match the paper backbone and ModelGenerator API version; current releases do not prove compatibility with the historical worker. | -| GenerRNA | [Official model checkpoint and tokenizer](https://huggingface.co/pfnet/GenerRNA) | Cached model_updated.pt and tokenizer-config hashes are recorded below; the complete tokenizer/runtime and public-version binding remain required. | -| CodonGPT | The paper-linked NanilTx/codonGPT_pub repository returned HTTP 404 in this check | Obtain the original repository/version and weights; a similarly named software project is not a substitute. | -| ESM | [Official model downloads and fair-esm environment](https://github.com/facebookresearch/esm) | Recover exact model size/ensemble and original ProteinGym reference/DMS inputs. | -| ProGen3 | [Official models, Docker runtime and scoring CLI](https://github.com/Profluent-AI/progen3) | Official scorer averages forward/reverse likelihood; verify this against the original EVA comparison protocol and fix the corresponding weight revision. | +| RNA-FM, RNABERT, RNA-MSM | [RNA-FM upstream](https://github.com/ml4bio/RNA-FM); [recorded model identities](../reproduction/release/recovered_rna_snapshots_20260909.json) | Model/config/tokenizer snapshot and a compatible Transformers/Multimolecule environment; choose the appropriate masking, length and sequence/MSA protocol. | +| GENA-LM | [Official code and models](https://github.com/AIRI-Institute/GENA_LM) | Checkpoint, DNA tokenizer and any required custom model code. | +| ERNIE-RNA | [Official repository](https://github.com/Bruce-ywj/ERNIE-RNA/blob/main/README.md) | ERNIE/fairseq environment and the pretrained checkpoint selected for the comparison. | +| Evo2 | [Official models and runtime](https://github.com/arcinstitute/evo2) | Selected model size/checkpoint and its Vortex/Transformer Engine environment. | +| CodonFM | [Official repository](https://github.com/NVIDIA-BioNeMo/CodonFM) | Checkpoint, configuration, CodonFM environment and preprocessing settings. | +| AIDO.RNA | [ModelGenerator](https://github.com/genbio-ai/ModelGenerator), [AIDO](https://github.com/genbio-ai/AIDO) | ModelGenerator environment, matching API and local checkpoint. | +| GenerRNA | [Official checkpoint and tokenizer](https://huggingface.co/pfnet/GenerRNA) | Model, complete tokenizer and the upstream likelihood-scoring implementation. | +| CodonGPT | [Preserved scoring worker](../reproduction/benchmark/upstream/rna/external_scripts/compute_codongpt_ll.py) | Compatible checkpoint/tokenizer, Transformers version and codon preprocessing. | +| ESM | [Official models and fair-esm](https://github.com/facebookresearch/esm) | Local weights and companion files, reference/DMS inputs and the selected scoring strategy or ensemble. | +| ProGen3 | [Official models and scoring CLI](https://github.com/Profluent-AI/progen3) | Local weights and the official runtime; record the scorer's forward/reverse convention. | -`reproduction/release/competitor_resource_verification.json` records the -read-only Hugging Face revision checks and available LFS object hashes. -No third-party weight download or inference is claimed by this resource index. +Use whichever local model version your benchmark specifies and record its +identity, environment and scoring protocol with the output. Reproducing a +particular published row additionally requires that row's checkpoint and +protocol. [Paper workflow coverage](PAPER_WORKFLOWS.md) records the established +historical bindings and the components still needed for that purpose. ## Recovered comparison-model files: September 9, 2026 -The original RNA-FM, RNABERT and RNA-MSM cache snapshots were recovered and -all 15 weight/config/tokenizer files were hashed. Their model configurations -record Transformers 4.50.0; the exact Multimolecule runtime is still unbound. -The [recovered identity record](../reproduction/release/recovered_rna_snapshots_20260909.json) -contains each historical revision and every file SHA256. Ready-to-use manifests -for the existing `--model-manifest` interface are in -`reproduction/release/model_manifests/`. Keep a manifest outside its model folder. +The [RNA snapshot record](../reproduction/release/recovered_rna_snapshots_20260909.json) +contains revision identifiers and SHA256 values for 15 RNA-FM, RNABERT and +RNA-MSM weight/config/tokenizer files. Their saved configs record Transformers +4.50.0; the exact historical Multimolecule version has not been established. +Matching local files can use the supplied manifests in +`reproduction/release/model_manifests/`. Keep each manifest outside its model +folder. For a different local checkpoint, create a manifest with the +[existing snapshot command](BENCHMARK_PROTOCOLS.md#comparison-model-execution-not-plotting). -The historical public revisions still return 404. Current public weights have -different file hashes; no tensor-level equivalence is claimed and no current -weight is substituted. The recovered files are not rehosted in this source -repository. Original environment/protocol correspondence and a working public -download remain required before these methods can be claimed fully reproducible. - -The [additional recovered-asset record](../reproduction/release/recovered_competitor_assets_20260909.json) -adds 13 file identities from CodonFM 600M/1B, ERNIE-RNA, AIDO.RNA and GenerRNA -caches. These include configurations and the ERNIE environment-file hash; the -two ERNIE copies have identical hashes. A local cached weight is not itself a -public download or proof that its full environment/assay protocol is recovered. -No additional competitor inference was performed in this repository update. +The [additional asset record](../reproduction/release/recovered_competitor_assets_20260909.json) +contains 13 file identities from CodonFM 600M/1B, ERNIE-RNA, AIDO.RNA and +GenerRNA caches, including an ERNIE environment-file hash. These records help +identify local assets; execution scope is documented in the workflow guide. diff --git a/docs/PAPER_WORKFLOWS.md b/docs/PAPER_WORKFLOWS.md index 6089152..13b7a52 100644 --- a/docs/PAPER_WORKFLOWS.md +++ b/docs/PAPER_WORKFLOWS.md @@ -22,30 +22,30 @@ this is not the complete set of manuscript experiments or all plotted models. The preserved original configuration is `reproduction/benchmark/upstream/rna/config.yaml`. Historical absolute paths identify earlier resources; do not execute that configuration unchanged. -Use separate method environments to avoid incompatible dependencies. -The entries below distinguish recovered version identifiers from missing ones. +Use separate method environments and local checkpoint paths, or export +predictions from an upstream scorer to the +[common evaluator](BENCHMARK_PROTOCOLS.md#using-your-own-model-outputs). +The version identifiers below describe the original comparison records. New +benchmarks can use other locally prepared versions, recorded with their results. | Method | Model resource identified in original source | Entry and remaining requirement | |---|---|---| -| RNA-FM | `multimolecule/rnafm`, revision `2a98c38e8219d666ad35d04a689218ab50a36293` | `run_competitor.py rna-mlm`; Multimolecule registration and the method's original masking/length settings are required. The historical revision currently returns HTTP 404; no public version is substituted or newly run. | -| RNABERT | `multimolecule/rnabert`, revision `7a9b5d5d5a96931ff056828c0968479098506cea` | Same bounded masked-LM entry; historical revision currently returns HTTP 404. | -| RNA-MSM | `multimolecule/rnamsm`, revision `ad61a4915674596557b3c3c4fe4fd7162d36acc8` | Historical revision currently returns HTTP 404; sequence/MSA protocol correspondence is also unverified. | +| RNA-FM | `multimolecule/rnafm`, revision `2a98c38e8219d666ad35d04a689218ab50a36293` | `run_competitor.py rna-mlm`; Multimolecule registration and the method's original masking/length settings are required. Use a locally prepared model/tokenizer snapshot and record its file hashes. | +| RNABERT | `multimolecule/rnabert`, revision `7a9b5d5d5a96931ff056828c0968479098506cea` | Use the same bounded masked-LM entry with a local snapshot and compatible runtime. | +| RNA-MSM | `multimolecule/rnamsm`, revision `ad61a4915674596557b3c3c4fe4fd7162d36acc8` | Use a local snapshot; the original sequence/MSA protocol correspondence still requires confirmation for the paper comparison. | | GENA-LM | Local snapshot names `AIRI-Institute/gena-lm-bert-base-t2t-multi`, revision `4633e5a1ada905bb7afee6877d71cc12578a95a5` | Original model-name field differs from local snapshot name. Resolve that binding before downloading or running a replacement. | -| ERNIE-RNA | `ERNIE-RNA_pretrain.pt` | Preserved ERNIE adapter and worker; cached weight and environment-file hashes are now recorded; matching public release and runtime execution remain unverified. | +| ERNIE-RNA | `ERNIE-RNA_pretrain.pt` | Preserved ERNIE adapter and worker; cached weight and environment-file hashes are now recorded; the exact paper runtime has not been verified. | | Evo2 1B, 7B, 40B | `evo2_1b_base`, `evo2_7b_base`, `evo2_40b_base` | Preserved Evo adapter; exact revisions and official compatible runtime still required. Do not replace the original scoring objective. | -| CodonFM 600M and 1B | Original 600m/1b model selectors | Preserved `score_codonfm.py` and adapter; cached weight/config identities are now recorded; original runtime and public version binding remain unresolved. | -| AIDO.RNA | Original model-weight folder and ModelGenerator framework | Preserved AIDO worker/adapter; cached weight-shard/config hashes are now recorded; exact public snapshot and environment binding remain required. | -| GenerRNA | `model_updated.pt` and separate tokenizer | Preserved GenerRNA worker/adapter; cached weight and tokenizer-config hashes are now recorded; complete tokenizer/runtime and public version binding remain required. | +| CodonFM 600M and 1B | Original 600m/1b model selectors | Preserved `score_codonfm.py` and adapter; cached weight/config identities are now recorded; the original runtime and paper checkpoint binding remain to be established. | +| AIDO.RNA | Original model-weight folder and ModelGenerator framework | Preserved AIDO worker/adapter; cached weight-shard/config hashes are now recorded; the paper snapshot and environment binding remain required. | +| GenerRNA | `model_updated.pt` and separate tokenizer | Preserved GenerRNA worker/adapter; cached weight and tokenizer-config hashes are now recorded; the complete tokenizer/runtime and paper checkpoint binding remain required. | | CodonGPT | Original worker `compute_codongpt_ll.py` | Weight revision and original environment still required. | | ESM protein models | `upstream/protein/evaluate_dms_esm_domain.py` | Strict native entry `run_competitor.py protein-esm`; exact reference/DMS files and model ensemble must be supplied. | | ProGen3 and other unmatched protein rows | Existing summary-table entries | Missing producer/version must be recovered; a table row is not an inference implementation. | -The three historical Multimolecule revision IDs returned HTTP 404 in the -current public API check. They are retained as original configuration evidence, -not presented as working download commands. The GENA-LM revision resolves, -but its original model-name/snapshot mismatch still needs clarification. -Do not substitute current `main` weights and call them the paper model. -Once an original snapshot is recovered, its files can be recorded with: +Record the exact files used by a local model before scoring. A supplied +manifest identifies a matching historical snapshot; a newly generated manifest +identifies the checkpoint you have chosen for a new benchmark: ```bash python reproduction/benchmark/run_competitor.py snapshot \ @@ -53,13 +53,12 @@ python reproduction/benchmark/run_competitor.py snapshot \ ``` [Official resource links](OFFICIAL_MODEL_RESOURCES.md) identify upstream model -and runtime sources. These links do not establish which upstream version -produced an EVA paper result. +and runtime sources. For an exact paper comparison, use the corresponding checkpoint and scoring +protocol from the original run. See [BENCHMARK_PROTOCOLS.md](BENCHMARK_PROTOCOLS.md) for the parameterized masked-LM and native ESM interfaces. The snapshot helper records identity; it does not determine whether a checkpoint is the one used for a paper result. -No unverified version is silently filled with a current `main` revision. The recovered Evo2 7B protein row remains a separate candidate table under `reproduction/benchmark_release/recovered_reference/`. It is not substituted @@ -71,6 +70,6 @@ to be documented before those comparisons can be called complete. The [current resource index](OFFICIAL_MODEL_RESOURCES.md#recovered-comparison-model-files-september-9-2026) records recovered RNA-FM/RNABERT/RNA-MSM snapshots and additional cached -CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. Public download/runtime -bindings remain incomplete. These new identities supplement the historical -requirements above; they do not imply newly executed competitor benchmarks. +CodonFM, ERNIE-RNA, AIDO.RNA and GenerRNA file hashes. These can identify matching +local files; the environment and scoring protocol must be recorded separately. +The records describe recovered artifacts, not additional benchmark runs. diff --git a/docs/RELEASE_STATUS.md b/docs/RELEASE_STATUS.md index 0dedf4e..6518a56 100644 --- a/docs/RELEASE_STATUS.md +++ b/docs/RELEASE_STATUS.md @@ -62,8 +62,9 @@ a demonstrated cause of the difference. ## Remaining coverage and publication work - Recovered comparison-model hashes are in the [resource index](OFFICIAL_MODEL_RESOURCES.md). - Public downloads, original runtimes or producers remain missing for some methods; - cached artifact identities alone do not close those gaps. + Local checkpoints and user-generated predictions are supported. Exact paper + runtimes, scoring implementations or checkpoint bindings remain unverified for + some methods; file identities alone do not establish those correspondences. - Training machinery is tested at small scale. Original dense-model code and some paper-specific training/run bindings are not available. - Some figure-specific input/command mappings and complete SAE generation diff --git a/docs/REPRODUCTION.md b/docs/REPRODUCTION.md index 84cabc1..ebb6651 100644 --- a/docs/REPRODUCTION.md +++ b/docs/REPRODUCTION.md @@ -116,6 +116,17 @@ machinery; it does not rerun large-scale pretraining. ## Other analyses and historical sources +[Notebooks](../notebooks/README.md) provide interactive analysis and plotting; +[reproduction resources](../reproduction/README.md) hold fixed inputs, protocols, +reference outputs and historical implementations. They work together with the +command-line entry points in `scripts/`. For example, the benchmark notebook +calls `scripts/benchmark_release.py` with manifests and tables from +`reproduction/benchmark_release/`. + +Third-party models can run from local checkpoints or in their own upstream +environments. Export keyed predictions to use the common metric evaluator; +see [using your own model outputs](BENCHMARK_PROTOCOLS.md#using-your-own-model-outputs). + [Benchmark protocols](BENCHMARK_PROTOCOLS.md), [essentiality](ESSENTIALITY_REPRODUCTION.md) and [SAE analyses](SAE_REPRODUCTION.md) distinguish stored-result calculations from inference. Sources under `reproduction/benchmark/upstream/` and diff --git a/docs/RESOURCES.md b/docs/RESOURCES.md index 8e9429a..03974d9 100644 --- a/docs/RESOURCES.md +++ b/docs/RESOURCES.md @@ -57,21 +57,18 @@ currently resolves to a record with eight split archive parts, not a single Its files are not evidence that it contains the latest main-branch repairs. The checked metadata and advertised checksums are retained in the JSON record. -## Comparison models and access checks + -[Official comparison-model resources](OFFICIAL_MODEL_RESOURCES.md) links the -upstream code and records unresolved historical bindings. Run the independent -access report from the repository root with Python 3.10 or 3.11: +## Comparison models -```bash -python scripts/check_resource_links.py --output results/resource-access.json -``` +Prepare third-party models in their own compatible environments. You can use +locally downloaded or cached weights with the supported comparison-model CLI, +or run an upstream scorer and submit its prediction CSV for metric calculation. +Benchmark evaluation does not require a live connection to the model host. -Exit 0 means the report was written. Inspect each `access` field: network -errors and unavailable revisions are reported without changing CPU regression -status. The tool retrieves metadata only; it neither downloads model bodies -nor replaces missing weights. CPU CI performs offline file/hash checks. - -The [September 9 access report](../reproduction/release/resource_access_20260909.json) -is a dated observation. External services and their current default branches -can change. Each data/model resource retains its upstream license and terms. +[Model resources](OFFICIAL_MODEL_RESOURCES.md) lists the required components; +[benchmark protocols](BENCHMARK_PROTOCOLS.md) documents local model paths and +the prediction format. Record the model version, scoring settings and input +identities with each run. Matching a specific published result additionally +requires its checkpoint and protocol. External resources retain their upstream +license and terms. diff --git a/notebooks/README.md b/notebooks/README.md new file mode 100644 index 0000000..43dd6c0 --- /dev/null +++ b/notebooks/README.md @@ -0,0 +1,31 @@ +# Analysis notebooks + +Use this directory to explore results interactively, inspect intermediate +outputs and generate plots. Notebook folders also contain supporting scripts +and data; the directory is organized by research task, not by file extension. + +| Task | Start here | +|---|---| +| Benchmark summaries and metrics from your own predictions | [Benchmark notebook](prediction/benchmark_reproduction.ipynb) | +| RNA generation tutorial | [Generation notebook](generation/generation_tutorial.ipynb) | +| Interpretability analysis | [Analysis guide](interpretability_analysis/README.md) | +| Design case studies | [Design notebooks](design/) | + +## How this relates to reproduction + +Notebooks and command-line tools are two ways to work through a computation. +They share scripts and resources rather than defining separate benchmarks. +For example, `prediction/benchmark_reproduction.ipynb` calls +[`scripts/benchmark_release.py`](../scripts/benchmark_release.py), which reads +the manifests and reference tables in +[`reproduction/benchmark_release/`](../reproduction/benchmark_release/). +That notebook performs CPU analysis; fresh model inference has its own command. + +[`reproduction/`](../reproduction/README.md) holds fixed inputs, protocols, +reference outputs and dated records, together with preserved historical code. +Some data and helper scripts remain next to their notebooks to keep existing +paths working. Both directories participate in paper reproduction. + +Start with the [reproduction guide](../docs/REPRODUCTION.md) for a complete +workflow, or [benchmark protocols](../docs/BENCHMARK_PROTOCOLS.md) to evaluate +predictions from a model you run in your own environment. diff --git a/reproduction/README.md b/reproduction/README.md new file mode 100644 index 0000000..d9971eb --- /dev/null +++ b/reproduction/README.md @@ -0,0 +1,31 @@ +# Reproduction resources + +This directory keeps the resources that identify a computation: input and model +manifests, scoring protocols, reference results, file hashes and dated validation +records. It also contains the portable comparison-model runner and preserved +historical implementations. It is not a second set of analysis notebooks. + +| Folder | Contents | +|---|---| +| [milena_14b/](milena_14b/) | Fixed EVA checkpoint/input identities and expected Milena outputs | +| [benchmark/](benchmark/) | Comparison-model CLI, fixtures, protocols and historical scoring sources | +| [benchmark_release/](benchmark_release/) | Benchmark reference tables and evaluation manifests used by scripts and notebooks | +| [essentiality/](essentiality/) | Inputs, archived results and provenance for essentiality calculations | +| [sae/](sae/) | SAE protocols, manifests and archived analysis resources | +| [release/](release/) | Dated software validation and model-resource identity records | + +## How the folders work together + +- [`notebooks/`](../notebooks/README.md) provides interactive analysis and plots, + with supporting data and helpers beside the notebooks. +- [`scripts/`](../scripts/) provides command-line entry points for repeatable runs. +- This directory supplies fixed resources and reference evidence for those runs. + +For example, `scripts/reproduce_milena.py` uses the manifest here, reads the assay +inputs under `notebooks/prediction/data/`, and calls the preserved scoring +implementation under `benchmark/`. Its output is written to the directory you +select. The [task guide](../docs/REPRODUCTION.md) documents the complete command. + +Files under `upstream/` and `archive/` preserve historical versions, while +current commands are listed in the task guides. These folders retain their +existing paths because manifests and scripts refer to them. diff --git a/scripts/check_repository.py b/scripts/check_repository.py index 6c87728..e083c7e 100644 --- a/scripts/check_repository.py +++ b/scripts/check_repository.py @@ -29,7 +29,8 @@ def check_links(): docs = [ROOT / 'README.md', ROOT / 'README_BENCHMARK_SUPPLEMENT.md', ROOT / 'CONTRIBUTING.md'] docs += list((ROOT / 'docs').glob('*.md')) docs += [ROOT / 'training/pretrain/README.md', ROOT / 'finetune/aptamer/script/README.md', - ROOT / 'reproduction/milena_14b/expected/README.md'] + ROOT / 'reproduction/milena_14b/expected/README.md', + ROOT / 'notebooks/README.md', ROOT / 'reproduction/README.md'] for path in docs: text = re.sub(r'```.*?```', '', path.read_text(), flags=re.S) links = re.findall(r'\]\(([^\s)]+)(?:\s+[^)]*)?\)', text)