diff --git a/results/ablm-eval/1.0.2.txt b/results/ablm-eval/1.0.2.txt new file mode 100644 index 0000000..c9ccf76 --- /dev/null +++ b/results/ablm-eval/1.0.2.txt @@ -0,0 +1,66 @@ +--- +tool: ablm-eval +version: 1.0.2 +image: ghcr.io/brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +digest: sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +runner: sbatch --partition=flyloft --gres=gpu:1 --time=00:30:00 +host: sunk.6af0ac-eu-south-03b-briney-b200.coreweave.app +started: 2026-11-20T01:08:02Z +exit: 0 +--- +Job 1077: SUNK job dashboard: https://cks-grafana.coreweave.com/d/slurm-job-metrics/slurm-job-metrics?var-slurm_cluster=slurm&var-job_id=1077 +pyxis: importing docker image: ghcr.io#brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +node: slurm-b200-204-039 +GPU 0: NVIDIA B200 (UUID: GPU-e3ae7af6-08de-017b-aedd-15ab9be767f6) +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ +┃ name ┃ repository ┃ revision ┃ separator ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ +│ balm-paired │ brineylab/BALM-paired │ 78e6a59276db │ │ +│ balm-unpaired │ brineylab/BALM-unpaired │ 8fd18a5b47bf │ single chains only │ +│ ft-esm │ brineylab/ft-ESM │ 6594c8b61de1 │ │ +│ balm-moe │ brineylab/BALM-MoE │ 51842e69ca3f │ │ +│ currab │ brineylab/CurrAb │ 92e28534663e │ │ +│ preferential-250k │ brineylab/preferential-250k │ 72ebb7b3e0f1 │ │ +│ uniform-250k │ brineylab/uniform-250k │ bb2fd2338300 │ │ +│ uniform-350k │ brineylab/uniform-350k │ 43a1e4fd35cf │ │ +│ scaling-350m-full │ brineylab/350M_full_checkpoint-500000 │ 9cd9e8af5138 │ │ +└───────────────────┴───────────────────────────────────────┴──────────────┴────────────────────┘ +tiny: brineylab/8M_full_checkpoint-500000 @ 01e6532f9091 + Fetching 7 files: 0%| | 0/7 [00:00/models.json` records the run's models and separators; each task writes its config to `/_/config.json` and one parquet per model under `results/` (the per-position file is `_-per-position-inference.parquet`). With more than one model, comparison plots and summary CSVs land beside them. The per-position parquet keeps the dataset's own columns plus `sequence` (chains joined by the separator), `tokenized_sequence`, `tokenized_seq_wo_special`, `loss` (one cross-entropy per non-special position: each position masked in turn, the true residue's negative log-probability), `perplexity` (`exp(loss)` per position), `probabilities` (the full next-token distribution per position), `prediction_tokens` and `prediction` (the model's top prediction per position). The other tasks write the analogous columns for their metrics — `inference` one mean loss per sequence, `mutation_prediction` and `naturalness` their per-mutation scores, `classification` per-fold metrics, `routing` the MoE expert assignments. +- **Out:** `/models.json` records the run's models and separators; each task writes its config to `/_/config.json` and one parquet per model under `results/` (the per-position file is `_-per-position-inference.parquet`). With more than one model, comparison plots and summary CSVs land beside them. The per-position parquet keeps the dataset's own columns plus `sequence` (chains joined by the separator), `tokenized_sequence`, `tokenized_seq_wo_special`, `loss` (one cross-entropy per non-special position: each position masked in turn, the true residue's negative log-probability), `perplexity` (`exp(loss)` per position), `probabilities` (the full next-token distribution per position), `prediction_tokens` and `prediction` (the model's top prediction per position). The other tasks write the analogous columns for their metrics — `inference` one mean loss per sequence, `mutation_prediction` and `naturalness` their per-mutation scores, `classification` one CSV per model at `/_classification/results/` with the fold's test metrics, `routing` the MoE expert assignments. ## Compute -One GPU per job: the models are small enough that one is faster than several through the DataParallel the code falls back to when it sees more. Per-position inference costs one forward pass per sequence position (each sequence becomes one batch of L masked copies), so its cost grows as sequence count times the square of length; the other inference tasks cost one pass per sequence. Measured inside the image on one b200, downloading each model once into `--cache-dir` and running `per_position` on the 8-pair fixture (heavy chains 124-134 residues, light 107-113): +One GPU per job: the models are small enough that one is faster than several through the DataParallel the code falls back to when it sees more. Per-position inference costs one forward pass per sequence position (each sequence becomes one batch of L masked copies), so its cost grows as sequence count times the square of length; the other inference tasks cost one pass per sequence. Measured on one b200 in a venv carrying the image's pinned package set on CPython 3.12, the image's Python — the image's twin, since the image cannot be measured before it publishes — downloading each model once into `--cache-dir` and running `per_position` on the 8-pair fixture (heavy chains 124-134 residues, light 107-113): | model | separator | mean per-residue log-probability | wall time | | --- | --- | --- | --- | -| `balm-paired` | `` | -0.3063 | 16.9 s | -| `balm-unpaired` | single chains only | -0.3713 | 12.6 s | -| `ft-esm` | `` | -0.2964 | 26.2 s | -| `balm-moe` | `` | does not load (see below) | 1.3 s | -| `currab` | `` | -0.2723 | 21.0 s | -| `preferential-250k` | `` | -0.2817 | 14.8 s | -| `uniform-250k` | `` | -0.2924 | 15.9 s | -| `uniform-350k` | `` | -0.2739 | 17.5 s | -| `scaling-350m-full` | `` | -0.2915 | 19.3 s | +| `balm-paired` | `` | -0.3063 | 18.5 s | +| `balm-unpaired` | single chains only | -0.3713 | 12.7 s | +| `ft-esm` | `` | -0.2964 | 25.9 s | +| `balm-moe` | `` | -0.2848 | 15.9 s | +| `currab` | `` | -0.2723 | 20.1 s | +| `preferential-250k` | `` | -0.2817 | 13.5 s | +| `uniform-250k` | `` | -0.2924 | 12.8 s | +| `uniform-350k` | `` | -0.2739 | 14.6 s | +| `scaling-350m-full` | `` | -0.2915 | 14.4 s | -Each row spans the line that announces the model — printed just before its download — to its results parquet, so its weights download is inside the number. `balm-paired` and `ft-esm` ran inside the one-config run of the eight paired models; that run stops at the first model that fails, and it stopped at `balm-moe`, the third of its eight, 56.4 s in, so `currab` through `scaling-350m-full` each ran in a config of their own. `balm-unpaired` ran the same eight heavy chains as an unpaired task, that run 19.5 s. `balm-moe`'s row is its own-config retry, which found its weights already fetched: it does not load — transformers 5.23 hands its special tokens down to BALM's `BalmTokenizer` as `AddedToken` objects, and the `WordLevel` that tokenizer builds on the pinned tokenizers 0.23.3 wants a plain `str` (`TypeError: 'AddedToken' object is not an instance of 'str'`, while processing `unk_token`) — so it is reported as a failed row, not dropped or worked around. +Each row spans the line that announces the model — printed just before its download — to its results parquet, so its weights download is inside the number. All nine load under AbLM-Eval's strict weight check and finish: the eight paired models ran in one config (154.9 s cold, to the final comparison), and `balm-unpaired` ran the same eight heavy chains as an unpaired task (29.6 s cold). The per-residue log-probabilities are deterministic — for the eight models the AbLM-Eval 32a3961 bump leaves alone they equal the 1.0.1 image's measurements — while wall times vary run to run. -Classification fine-tuning would cost one forward and backward pass per training step plus one predict pass over the test fold, but on the pinned versions it never reaches one: the HD-CoV check — fold 0 of the MoE paper's data (Zenodo record 19411192), one fold, 10 training steps, the 8M checkpoint, through `ablm-eval run` on one GPU with AbLM-Eval's default accelerate config — died at model construction on every attempt (22.0 s, 19.9 s and 27.5 s wall inside the image for the 8M checkpoint, the same with the task's `attention_classifier: false`, and `balm-paired`), before a single training step; the gotcha below says why. +Classification fine-tuning costs one forward and backward pass per training step plus one predict pass over the test fold. The HD-CoV check — fold 0 of the MoE paper's data (Zenodo record 19411192), one fold, through `ablm-eval run` on one GPU with AbLM-Eval's default accelerate config (DeepSpeed ZeRO 0, bf16, one process) and its default training arguments (no `training_args` in the task: warmup_steps 0.1, bf16, batch 32, 3 epochs of the fold's 20,257 rows, about 1,900 steps), measured in the same venv as the table above — trains and writes scores for both the 8M checkpoint (139.4 s wall from a cold cache, fetch to final metrics) and `balm-moe`, whose attention classifier is on by default (752.4 s); their test metrics on the fold's 5,065 test rows are from that short default-args run, not model quality: the 8M checkpoint's accuracy 0.7765, precision 0.7827, recall 0.7654, AUROC 0.8613, AUPR 0.8651, F1 0.7740, MCC 0.5531, and `balm-moe`'s accuracy 0.7822, precision 0.7816, recall 0.7832, AUROC 0.8569, AUPR 0.8374, F1 0.7824, MCC 0.5645. -The shed's test (`test/run.sh`) took 2.5 minutes of job wall on one b200 in the candidate's run (image import, the registry check, the 8M checkpoint fetch, `per_position` and the reference comparison); its `--overwrite` exercise — the refused rerun, the recomputation, the comparison again — adds well under a minute, the checkpoint being already in the cache. +The shed's test (`test/run.sh`) took 2.5 minutes of job wall on one b200 in the 1.0.1 candidate's run (image import, the registry check, the 8M checkpoint fetch, `per_position` and the reference comparison); the test is unchanged in 1.0.2, and its `--overwrite` exercise — the refused rerun, the recomputation, the comparison again — adds well under a minute, the checkpoint being already in the cache. ## Running it @@ -84,13 +84,13 @@ On a laptop, from the directory holding `config.yaml` and the data (the tasks us ```bash podman run --rm --device nvidia.com/gpu=all -v "$PWD":/work -w /work \ - ghcr.io/brineylab/toolshed/ablm-eval:1.0.1 ablm-eval run config.yaml --cache-dir /work/cache + ghcr.io/brineylab/toolshed/ablm-eval:1.0.2 ablm-eval run config.yaml --cache-dir /work/cache ``` On a Slurm cluster with pyxis: ```bash -srun --gres=gpu:1 --container-image=ghcr.io#brineylab/toolshed/ablm-eval:1.0.1 \ +srun --gres=gpu:1 --container-image=ghcr.io#brineylab/toolshed/ablm-eval:1.0.2 \ --container-mounts="$PWD":/work --container-workdir=/work \ ablm-eval run config.yaml --cache-dir /work/cache ``` @@ -104,6 +104,6 @@ The model files land in `--cache-dir` (the default `.ablm-eval-cache` sits in th - **The separator comes from the model:** each registered model carries the separator it was trained with and applies it to every task; setting `separator` in a task config alongside such a model is refused. Only full model entries with `separator` of their own, and `balm-unpaired` (which leaves the task's separator alone), differ. - **`num_proc` defaults to 128** dataset-preparation workers; on a node with fewer cores, set it to what the node has. - **Rich wraps at 80 columns** when a terminal is not a tty: export `COLUMNS=1000` before `ablm-eval models` in scripts, or the table's repository and revision columns wrap. -- **Classification (does not work on the pinned versions):** it fine-tunes a head with transformers' `Trainer` through accelerate — run non-interactively it uses accelerate's default config (DeepSpeed ZeRO 0, bf16, one process) without prompting — but it dies before training: AbLM-Eval's train script passes `attention_classifier` into `AutoModelForSequenceClassification.from_pretrained` unconditionally, and on transformers 5.23 a kwarg that is neither a recognized config field nor an attribute of the checkpoint's config is forwarded to the model constructor, so every model that loads here (each a plain esm or roberta config) fails with `TypeError: ... got an unexpected keyword argument 'attention_classifier'`. The HD-CoV check tried it three ways on one GPU — the 8M checkpoint, the same with the task's `attention_classifier: false` (the kwarg is passed whatever its value), and `balm-paired`, whose published config.json is a plain roberta config — and all three failed identically (22.0 s, 19.9 s and 27.5 s wall, inside the image). The only registered model whose config defines the field is `balm-moe`, which fails earlier at its tokenizer (the failed row above). A classification task with no `training_args` dies even earlier, at `TrainingArguments`: transformers 5.23 removed `warmup_ratio`, which AbLM-Eval's fallback still passes — spell the warmup as `warmup_steps` (an int, or a float in [0, 1) for the ratio). Fixing classification needs a change in AbLM-Eval itself — pass `attention_classifier` only when the checkpoint's config defines it, or stop passing it — so until upstream changes, use the image's other tasks. One trap is the caller's, not AbLM-Eval's: under Slurm/pyxis the caller's environment is passed into the container, and an ambient `LOCAL_RANK`/`RANK` with no `WORLD_SIZE` makes transformers' `TrainingArguments` — built while the CLI validates the config, before any task — init `torch.distributed` at the `env://` rendezvous and die (`environment variable WORLD_SIZE expected, but not set`); unset them for the CLI, which is a single process that launches its own training through accelerate, and sets the training subprocess's variables itself. -- **The test** (`test/run.sh`) runs `per_position` through the CLI on AbLM-Eval's own fixture `test/paired-mini.parquet` — the 8 public pairs from Jaffe et al. 2022 (figshare 19617633), heavy chains 124-134 and light 107-113 residues, so each sequence builds a real masked batch of 107-134 rows — with the 8M checkpoint `brineylab/8M_full_checkpoint-500000` @ `01e6532f` spelled out as a full model entry with separator ``, fetched at test time. It compares the pooled mean per-residue log-probability and each sequence's mean with `test/reference.json`, computed once with `evaluate_ablms` (AbLM-Eval's Python API) at commit `641cff8` on the same fixture, model and pinned versions, to 1e-4. It also checks every registered model's name is listed by `ablm-eval models`, and exercises the documented `--overwrite`: the same config rerun without it is refused (`The directory '...' exists and is not empty!`), and rerun with it the results are recomputed and the comparison passes again. +- **Classification's `attention_classifier` follows the model:** the task field is unset by default — a model whose config defines it (`balm-moe` only) gets it on, every other model gets nothing, and setting it on a model whose config does not define it is refused with a `ValueError` naming the model. A task with no `training_args` trains on AbLM-Eval's defaults (warmup_steps 0.1, bf16, batch 32, 3 epochs, report_to none); spell out `training_args` only to depart from them. One trap is the caller's, not AbLM-Eval's: under Slurm/pyxis the caller's environment is passed into the container, and an ambient `LOCAL_RANK`/`RANK` with no `WORLD_SIZE` kills a CLI validating a config whose task carries `training_args` — transformers' `TrainingArguments`, built while the config is validated, inits `torch.distributed` at the `env://` rendezvous and dies (`environment variable WORLD_SIZE expected, but not set`); unset them for the CLI, which is a single process that launches its own training through accelerate and sets the training subprocess's variables itself. A task with no `training_args` builds no `TrainingArguments` at validation, so it does not hit the trap. +- **The test** (`test/run.sh`) runs `per_position` through the CLI on AbLM-Eval's own fixture `test/paired-mini.parquet` — the 8 public pairs from Jaffe et al. 2022 (figshare 19617633), heavy chains 124-134 and light 107-113 residues, so each sequence builds a real masked batch of 107-134 rows — with the 8M checkpoint `brineylab/8M_full_checkpoint-500000` @ `01e6532f` spelled out as a full model entry with separator ``, fetched at test time. It compares the pooled mean per-residue log-probability and each sequence's mean with `test/reference.json`, computed once with `evaluate_ablms` (AbLM-Eval's Python API) at commit `641cff8` on the same fixture, model and pinned versions, to 1e-4 — unchanged for 1.0.2: the AbLM-Eval bump touches no per-position inference path, and the reference still matches. It also checks every registered model's name is listed by `ablm-eval models`, and exercises the documented `--overwrite`: the same config rerun without it is refused (`The directory '...' exists and is not empty!`), and rerun with it the results are recomputed and the comparison passes again. - **Licenses:** AbLM-Eval is MIT; every model the registry names (and the 8M test checkpoint) is MIT-licensed on the Hub. The image carries no weights — it downloads them at run time — so a run needs the network and a place to put `--cache-dir`.