Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
22 changes: 11 additions & 11 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
checkpoint/

# Output data
data/output/
examples/data/output/

# Python cache
__pycache__/
Expand All @@ -15,19 +15,19 @@ __pycache__/
*_bak*

# Docker files
# NOTE: docker/ must stay in the build context: docker/Dockerfile COPYs
# docker/requirements.txt and docker/entrypoint.sh. Excluding it breaks the
# documented `docker build -f docker/Dockerfile -t eva:latest .` command.
# NOTE: scripts/docker/ must stay in the build context: scripts/docker/Dockerfile COPYs
# scripts/docker/requirements.txt and scripts/docker/entrypoint.sh. Excluding it breaks the
# documented `docker build -f scripts/docker/Dockerfile -t eva:latest .` command.
Dockerfile
# Keep the published recipe available to COPY docker/ and image-internal QA.
!docker/Dockerfile
docker/Dockerfile_bak*
docker/EVA.def
docker/smoke_test.sh
# Keep the published recipe available to COPY scripts/docker/ and image-internal QA.
!scripts/docker/Dockerfile
scripts/docker/Dockerfile_bak*
scripts/docker/EVA.def
scripts/docker/smoke_test.sh
.dockerignore

# Figures are not part of the image (34 MB of build context)
fig/
docs/assets/

# IDE / OS
.git/
Expand All @@ -45,4 +45,4 @@ dist/
build/
*.egg-info/
.pytest_cache/
notebooks/interpretability_analysis/sae_repro_release/outputs/
examples/notebooks/interpretability_analysis/sae_repro_release/outputs/
2 changes: 1 addition & 1 deletion .github/workflows/resources.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
branches: [main]
paths:
- 'scripts/check_resource_links.py'
- 'reproduction/release/resource_links.json'
- 'examples/reproduction/release/resource_links.json'
- '.github/workflows/resources.yml'

permissions:
Expand Down
11 changes: 5 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
# Reproduction guides are source deliverables, not generated documentation.
reference/
!reproduction/benchmark_release/reference/
!reproduction/benchmark_release/reference/**
!examples/reproduction/benchmark_release/reference/
!examples/reproduction/benchmark_release/reference/**
checkpoint/*
!checkpoint/.gitkeep
data/output/**
examples/data/output/**
eva_latest.tar
eva_latest.tar.gz
notebooks/interpretability_analysis/intermediate_data/*.npz
notebooks/tools/visualization/UMAP/taxid_phylum_mapping.json
examples/notebooks/interpretability_analysis/intermediate_data/*.npz
examples/notebooks/tools/visualization/UMAP/taxid_phylum_mapping.json
__pycache__/
*.py[cod]
*.egg-info/
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@

## Unreleased

### Repository layout — 2026-09-09

Tutorials and paper reproduction resources now share `examples/`; fine-tuning
lives under `training/finetune/`. Example configurations and data are under
`examples/config/` and `examples/data/`, illustrations under `docs/assets/`, and
container recipes under `scripts/docker/`. The benchmark supplement is now
`docs/BENCHMARK_SUPPLEMENT.md`. Create `checkpoint/` when downloading weights.
Model imports and the `eva-predict`, `eva-generate` and `eva-evolve` commands keep
their existing names. Updated guides show the new source-file paths.

### Other improvements

- Organize task guides, resource information and reviewer evidence around a concise README.
- Separate successful Milena execution from optional strict reference comparison.
- Add CPU regression, packaging, documentation and citation checks, with separate external-resource reporting.
Expand Down
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<a id="eva-a-long-context-generative-foundation-model-for-versatile-rna-design"></a>
<p align="center">
<img src="fig/github_logo.svg" alt="EVA — RNA foundation model" width="100%">
<img src="docs/assets/github_logo.svg" alt="EVA — RNA foundation model" width="100%">
</p>

<h1 align="center">EVA</h1>
Expand Down Expand Up @@ -28,7 +28,7 @@ workflows. Model weights and large datasets are downloaded separately.

<a id="why-use-eva"></a>
<p align="center">
<img src="fig/readme_overview.svg" alt="1.4B-parameter MoE flagship model; 8,192-token context window; trained on OpenRNA v1" width="100%">
<img src="docs/assets/readme_overview.svg" alt="1.4B-parameter MoE flagship model; 8,192-token context window; trained on OpenRNA v1" width="100%">
</p>

<a id="start-here"></a><a id="our-journey-with-eva-starts-here-"></a>
Expand Down Expand Up @@ -91,7 +91,7 @@ From the repository root, build and enter the runtime **on the host**:

```bash
mkdir -p checkpoint results
docker build -f docker/Dockerfile -t eva:local .
docker build -f scripts/docker/Dockerfile -t eva:local .
docker run --rm -it --gpus device=0 --name eva-repro \
-v "$PWD":/eva -w /eva eva:local bash
```
Expand Down Expand Up @@ -167,14 +167,15 @@ paper experiment. See [dated validation records](docs/RELEASE_STATUS.md).

| Directory | Contents |
|---|---|
| `eva/`, `tools/` | Model, tokenizer and inference CLIs |
| `training/`, `finetune/`, `config/` | Training entry points and configurations |
| `scripts/` | Command-line workflows and automation |
| [`notebooks/`](notebooks/README.md) | Interactive analysis, plots and supporting data |
| [`reproduction/`](reproduction/README.md) | Fixed inputs, protocols, reference outputs and historical code |
| `docs/`, `tests/` | Guides and regression tests |

Notebooks and command-line workflows share the reproduction resources.
| `eva/` | Model and tokenizer |
| `tools/` | Scoring, generation and evolution CLIs |
| [`training/`](training/README.md) | Pretraining, midtraining, fine-tuning and evaluation |
| [`examples/`](examples/README.md) | Notebooks, paper benchmarks, sample data and configurations |
| `scripts/` | Reproduction commands, checks and container recipes |
| `docs/` | Guides and illustrations |
| `tests/` | Regression tests |

Start with [examples](examples/README.md) to choose a tutorial or benchmark.
The wheel contains the importable model and CLI packages; use the complete
source checkout for training, notebooks and paper-reproduction inputs.

Expand Down
6 changes: 0 additions & 6 deletions README_BENCHMARK_SUPPLEMENT.md

This file was deleted.

Empty file removed checkpoint/.gitkeep
Empty file.
20 changes: 0 additions & 20 deletions config/tools_config/config_score_test.yaml

This file was deleted.

18 changes: 9 additions & 9 deletions docs/BENCHMARK_PROTOCOLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
The representative current-submission workflow is documented in
[REPRODUCTION.md](REPRODUCTION.md) and fixes the public EVA-1.4B model,
all 135 Milena samples, and the original sum-score convention. Its observed
output and numerical comparison are in `reproduction/milena_14b/expected/`.
output and numerical comparison are in `examples/reproduction/milena_14b/expected/`.
The current label file is retained as confirmed by the author. Historical
FASTA header metadata is not treated as an independent assay ground truth.

Historical source files and archived predictions remain unchanged with hashes
in `reproduction/benchmark/provenance.json` and `historical_14b_manifest.json`.
in `examples/reproduction/benchmark/provenance.json` and `historical_14b_manifest.json`.
[September 5 validation notes](HISTORICAL_BENCHMARK_VALIDATION.md) preserve
older findings, which must not be substituted for current validation status.

Expand All @@ -27,7 +27,7 @@ archive calculation from a completed model reproduction.

## Comparison-model execution, not plotting

`reproduction/benchmark/run_competitor.py` accepts locally prepared checkpoints
`examples/reproduction/benchmark/run_competitor.py` accepts locally prepared checkpoints
for its supported scoring interfaces. Models are loaded from `--model-dir`;
the RNA interface uses `local_files_only=True`. Prepare the model, tokenizer and
dependencies in the chosen model's compatible environment before running it.
Expand All @@ -38,14 +38,14 @@ your results; a specific paper comparison uses that paper's protocol.
1. Snapshot an already acquired, version-pinned local model. Keep the manifest **outside** the model directory; subsequent scoring requires the exact file set and hashes.

```bash
python reproduction/benchmark/run_competitor.py snapshot \
python examples/reproduction/benchmark/run_competitor.py snapshot \
--model-dir /models/rnafm-pinned --output /models/rnafm-manifest.json
```

2. RNA masked-LM scoring implements leave-one-token-out mean log likelihood, excluding tokenizer special tokens. It uses bounded mask batches, rejects unknown tokens/nonfinite logits, and has no fallback tokenizer. The original adapter's truncation is only reproduced when explicitly requested; the default is an error on an oversized sequence. `max-tokens` includes special tokens and must match the selected model.

```bash
python reproduction/benchmark/run_competitor.py rna-mlm \
python examples/reproduction/benchmark/run_competitor.py rna-mlm \
--input data/test.fasta --model-dir /models/rnafm-pinned \
--model-manifest /models/rnafm-manifest.json --max-tokens 1024 \
--sequence-type rna --register-multimolecule --mask-batch-size 1 \
Expand All @@ -57,7 +57,7 @@ Remove `--dry-run` for actual scoring. `--length-policy truncate` is an explicit
3. Protein ESM scoring uses the recovered native wt-/masked-marginal implementation, preceded by strict reference/variant validation and followed by finite/count checks. It requires one matching ProteinGym reference entry; no inferred-WT fallback or out-of-range mutation skipping is permitted.

```bash
python reproduction/benchmark/run_competitor.py protein-esm \
python examples/reproduction/benchmark/run_competitor.py protein-esm \
--input /data/ProteinGym/ASSAY.csv --reference /data/ProteinGym/DMS_substitutions.csv \
--model-dir /models/esm1v-pinned --model-manifest /models/esm1v-manifest.json \
--weight-file esm1v_t33_650M_UR90S_1.pt --strategy wt-marginals \
Expand All @@ -69,7 +69,7 @@ The snapshot must contain companion files required by `fair-esm`. Actual ESM exe
4. Metrics for RNA predictions require an independently verified label CSV with `variant_id,sequence,label`, not a bare positional vector:

```bash
python reproduction/benchmark/run_competitor.py evaluate \
python examples/reproduction/benchmark/run_competitor.py evaluate \
--predictions outputs/rnafm_test/predictions.csv --labels data/verified_labels.csv \
--output outputs/rnafm_metrics.json
```
Expand All @@ -95,12 +95,12 @@ scoring convention, model version and environment alongside the CSV. The
evaluator calculates signed Spearman using the supplied scores.

```bash
python reproduction/benchmark/run_competitor.py evaluate \
python examples/reproduction/benchmark/run_competitor.py evaluate \
--predictions results/my_model/predictions.csv \
--labels data/verified_labels.csv --output results/my_model/metrics.json
```

Use the [benchmark notebook](../notebooks/prediction/benchmark_reproduction.ipynb)
Use the [benchmark notebook](../examples/notebooks/prediction/benchmark_reproduction.ipynb)
and its evaluation manifest when comparing keyed predictions across multiple
datasets. This gives local model deployments a common analysis entry point.

Expand Down
20 changes: 10 additions & 10 deletions docs/BENCHMARK_RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,22 @@ python -B -m unittest discover -s tests -p test_benchmark_release.py -v
python scripts/benchmark_release.py --help
```

Open `notebooks/prediction/benchmark_reproduction.ipynb` to follow the same entry
Open `examples/notebooks/prediction/benchmark_reproduction.ipynb` to follow the same entry
points. Its three code cells have been executed both sequentially as Python and
in an isolated Jupyter kernel. The reference-analysis section ran; the optional
prediction-manifest section was explicitly unconfigured. This does not validate
model inference or arbitrary user-supplied prediction files.
The observed notebook-test dependency versions are recorded in
`reproduction/benchmark_release/jupyter-validation-macos-py312.lock.txt`.
`examples/reproduction/benchmark_release/jupyter-validation-macos-py312.lock.txt`.
That lock is for the tested macOS/Python 3.12 environment, not a universal
installation recipe or an EVA GPU environment.

## A. Published summary tables to diagnostic figures

```bash
python scripts/benchmark_release.py audit-reference \
--manifest reproduction/benchmark_release/reference_manifest.json \
--data-root reproduction/benchmark_release/reference \
--manifest examples/reproduction/benchmark_release/reference_manifest.json \
--data-root examples/reproduction/benchmark_release/reference \
--output results/reference_audit --plot
```

Expand Down Expand Up @@ -73,8 +73,8 @@ not fresh inference or confirmation of its final manuscript/checkpoint lineage.

```bash
python scripts/benchmark_release.py audit-reference \
--manifest reproduction/benchmark_release/recovered_reference_manifest.json \
--data-root reproduction/benchmark_release/recovered_reference \
--manifest examples/reproduction/benchmark_release/recovered_reference_manifest.json \
--data-root examples/reproduction/benchmark_release/recovered_reference \
--output results/recovered_reference_audit --plot
```

Expand All @@ -96,7 +96,7 @@ Do not generate IDs after independently sorting files, join by row number, use
another model to fill missing predictions, or infer experimental readouts from
sequence names. The label and prediction ID sets must match exactly.

Complete `reproduction/benchmark_release/evaluation_manifest.template.json` in a
Complete `examples/reproduction/benchmark_release/evaluation_manifest.template.json` in a
new file. Its null fields are deliberately unusable until real values are
provided. Specify all intended models and assays; the required scope is their
Cartesian product. For genuinely different model scopes, use separately labeled
Expand Down Expand Up @@ -155,9 +155,9 @@ The source contains these entry points; see REPRODUCTION.md for the new pinned 1
|---|---|---|
| EVA representative assay | `scripts/reproduce_dms.py` | Resolve original label/invocation mismatch; do not call the previous 21M run a numerical match |
| Historical EVA archive | `scripts/reproduce_historical_benchmark.py` | Arithmetic only; unresolved label provenance stays visible |
| Bounded RNA MLM / native ESM | `reproduction/benchmark/run_competitor.py` | Correct per-model objective, weights, runtime and assay manifest; not interchangeable with all competing methods |
| Historical multi-model adapters | `reproduction/benchmark/upstream/rna/` | Port original verified configs and external workers; legacy absolute paths are not reader instructions |
| Protein model adapters | `reproduction/benchmark/upstream/protein/` | Original reference/codon/domain/ensemble conventions and exact model versions; not all methods are complete |
| Bounded RNA MLM / native ESM | `examples/reproduction/benchmark/run_competitor.py` | Correct per-model objective, weights, runtime and assay manifest; not interchangeable with all competing methods |
| Historical multi-model adapters | `examples/reproduction/benchmark/upstream/rna/` | Port original verified configs and external workers; legacy absolute paths are not reader instructions |
| Protein model adapters | `examples/reproduction/benchmark/upstream/protein/` | Original reference/codon/domain/ensemble conventions and exact model versions; not all methods are complete |

Those files belong to the existing full-source repair, not this small overlay.
Their presence is not a fresh execution result. This supplement performs no GPU
Expand Down
6 changes: 6 additions & 0 deletions docs/BENCHMARK_SUPPLEMENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Benchmark evaluation component

The benchmark supplement has been integrated into this candidate. Start with
[the reproduction guide](REPRODUCTION.md). CPU summary analysis and keyed
prediction evaluation are documented in [BENCHMARK_RELEASE.md](BENCHMARK_RELEASE.md).
See [release status](RELEASE_STATUS.md) for actual validation and publication status.
2 changes: 1 addition & 1 deletion docs/ESSENTIALITY_REPRODUCTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ python scripts/reproduce_essentiality.py \
The original source root is currently available on `EVA_a100` at
`/data/yanjie_huang/enzyme1_server/eva/EVA1`. It is not a public dataset URL.
The required dataset checksum and directory layout are in
`reproduction/essentiality/source_manifest.json`. The input contains 95,538 CDS
`examples/reproduction/essentiality/source_manifest.json`. The input contains 95,538 CDS
records from five species; these records must not be described as 95,538 unique
genes without resolving transcript and duplicate-annotation identifiers.

Expand Down
8 changes: 4 additions & 4 deletions docs/HISTORICAL_BENCHMARK_VALIDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ author-confirmed submission. Use REPRODUCTION.md for the current workflow.

## What was recovered

`reproduction/benchmark/provenance.json` identifies 95 unchanged historical files by source host, original path, byte size and SHA256. These include RNA adapters and their scoring workers, protein reverse-translation/ESM workers, the historical EVA model/tokenizer/loader support code, four training configurations, released Milena inputs, archived predictions and an earlier validation report. This is a source snapshot, not a claim that every external model/environment has been released or rerun. The original server was read-only.
`examples/reproduction/benchmark/provenance.json` identifies 95 unchanged historical files by source host, original path, byte size and SHA256. These include RNA adapters and their scoring workers, protein reverse-translation/ESM workers, the historical EVA model/tokenizer/loader support code, four training configurations, released Milena inputs, archived predictions and an earlier validation report. This is a source snapshot, not a claim that every external model/environment has been released or rerun. The original server was read-only.

All three released EVA-21M files at `GENTEL-Lab/EVA`, revision `514db6705637c1ec963b728768fc9b34728699ee`, match the historical `30M_1130_midtrain/checkpoint_86006` files byte-for-byte:

Expand Down Expand Up @@ -71,10 +71,10 @@ To repeat on a single explicitly selected GPU with this compatible environment:

```bash
CUDA_VISIBLE_DEVICES=7 PYTHONDONTWRITEBYTECODE=1 \
python reproduction/benchmark/run_historical_14b.py \
--source-dir reproduction/benchmark/historical_14b \
python examples/reproduction/benchmark/run_historical_14b.py \
--source-dir examples/reproduction/benchmark/historical_14b \
--checkpoint /models/historical-rnaverse-clm \
--archive reproduction/benchmark/fixtures/milena/eva_1_4b_wotag_scores.json \
--archive examples/reproduction/benchmark/fixtures/milena/eva_1_4b_wotag_scores.json \
--expected-weights-sha256 323c13d571d0be87e450cb7b103bf8b28396e55e877519e2d2fb3c434875d420 \
--memory-limit-gib 4 --device cuda:0 --output outputs/milena_14b_fresh
```
Expand Down
Loading
Loading