From b7d3f8809a0767278e6d0a4369d86b41e1534051 Mon Sep 17 00:00:00 2001 From: Sarah Burbach <106550373+smburbach@users.noreply.github.com> Date: Fri, 20 Nov 2026 01:03:44 +0000 Subject: [PATCH 1/2] ablm-eval 1.0.2: the AbLM-Eval bump to 32a3961 (BALM at a91d7b4 through its pin) makes balm-moe load and classification train on the default training arguments, so the skill drops the classification-broken flag and the warmup_ratio workaround, carries the all-models table with balm-moe and the classification outcomes as measured, and keeps the launch-variable gotcha. --- tools/ablm-eval/Containerfile | 8 +++--- tools/ablm-eval/SKILL.md | 46 +++++++++++++++++------------------ 2 files changed, 28 insertions(+), 26 deletions(-) diff --git a/tools/ablm-eval/Containerfile b/tools/ablm-eval/Containerfile index afd61ea..cf83295 100644 --- a/tools/ablm-eval/Containerfile +++ b/tools/ablm-eval/Containerfile @@ -1,4 +1,4 @@ -# AbLM-Eval (github.com/smburbach/AbLM-Eval, MIT) at commit 641cff8: evaluation +# AbLM-Eval (github.com/smburbach/AbLM-Eval, MIT) at commit 32a3961: evaluation # of antibody language models -- inference, per-position inference, mutation # prediction, naturalness, classification (fine-tuning a classification head # over folds) and MoE routing -- from a YAML config, on the lab's published @@ -40,9 +40,11 @@ RUN pip uninstall -y --break-system-packages torchaudio torchvision \ # (any pyarrow 21 or later needs numpy 2 and breaks the import). pyyaml, # matplotlib, seaborn and tqdm are pinned at the versions the same set resolved # to in the campaign's Python 3.12 build of it. BALM rides AbLM-Eval's own -# dependency pin, git 36a8001. +# dependency pin, git a91d7b4 -- the commit that fixes BalmTokenizer on +# transformers 5.23 (which passes special tokens as AddedToken objects), so +# balm-moe loads. RUN pip install --no-cache-dir --break-system-packages \ - "ablm-eval @ git+https://github.com/smburbach/AbLM-Eval@641cff87aa688967d61a5beaacb8c9b51b29c78e" \ + "ablm-eval @ git+https://github.com/smburbach/AbLM-Eval@32a396150cc16303f7c06b7f0631f103f493b728" \ numpy==1.26.4 \ pyarrow==20.0.0 \ transformers==5.23.0 \ diff --git a/tools/ablm-eval/SKILL.md b/tools/ablm-eval/SKILL.md index d4b6b23..5e2e11e 100644 --- a/tools/ablm-eval/SKILL.md +++ b/tools/ablm-eval/SKILL.md @@ -1,15 +1,15 @@ --- name: ablm-eval -description: "AbLM-Eval: score antibody datasets with the lab's antibody language models from one YAML config — per-position and whole-sequence log-likelihoods, mutation and naturalness prediction, MoE routing, and classification fine-tuning over CV folds (classification is broken on the pinned versions; see Gotchas). Reach for it to score or compare the lab's published models (BALM-paired, BALM-unpaired, ft-ESM, BALM-MoE, CurrAb, the preferential/uniform checkpoints, the 350M scaling model) on paired or single-chain antibody data." +description: "AbLM-Eval: score antibody datasets with the lab's antibody language models from one YAML config — per-position and whole-sequence log-likelihoods, mutation and naturalness prediction, MoE routing, and classification fine-tuning over CV folds. Reach for it to score, compare or fine-tune the lab's published models (BALM-paired, BALM-unpaired, ft-ESM, BALM-MoE, CurrAb, the preferential/uniform checkpoints, the 350M scaling model) on paired or single-chain antibody data." metadata: - version: 1.0.1 + version: 1.0.2 --- # ablm-eval -Image: `ghcr.io/brineylab/toolshed/ablm-eval:1.0.1` +Image: `ghcr.io/brineylab/toolshed/ablm-eval:1.0.2` -AbLM-Eval (github.com/smburbach/AbLM-Eval at commit `641cff8`, MIT) runs its evaluation tasks on the lab's published antibody language models from one YAML config. The nine lab models are registered by name, each pinned to a full Hugging Face commit and carrying the chain separator it was trained with. A general-purpose image (DESIGN §1): it carries no weights and sets no `HF_HUB_OFFLINE`, because it downloads the models a config names at run time, into `--cache-dir`. +AbLM-Eval (github.com/smburbach/AbLM-Eval at commit `32a3961`, MIT) runs its evaluation tasks on the lab's published antibody language models from one YAML config: inference, per-position inference, mutation prediction, naturalness, classification (fine-tuning a head over CV folds) and MoE routing. The nine lab models are registered by name, each pinned to a full Hugging Face commit and carrying the chain separator it was trained with. A general-purpose image (DESIGN §1): it carries no weights and sets no `HF_HUB_OFFLINE`, because it downloads the models a config names at run time, into `--cache-dir`. ## Commands @@ -37,7 +37,7 @@ A config names any of them as a bare string, or spells a model out in full — ` ## Inputs and outputs -- **In:** a YAML config with `output_dir`, `models` and `tasks`. Tasks are keyed `inference`, `per_position`, `mutation_prediction`, `naturalness`, `classification`, `routing`, and share fields for `data_path` (CSV or parquet), `antibody_datatype` (`paired` or `unpaired`), `dataset_name` (names the task's output directory; distinct per task), `num_proc` (dataset preparation workers; default 128), and `dataset_columns` (overrides for the expected column names). Paired data defaults to `sequence_id`, `sequence_aa_heavy`, `sequence_aa_light`, with optional `cdr_mask_heavy`/`cdr_mask_light` (a `0`/`1` string per residue, CDRs `1`) and `v_mutation_count_aa_heavy`/`v_mutation_count_aa_light` (a `0`/`1` string per residue, mutated `1`); unpaired data defaults to `sequence_aa` with optional `locus`, `cdr_mask`, `v_mutation_count_aa`. A small config, whole: +- **In:** a YAML config with `output_dir`, `models` and `tasks`. Tasks are keyed `inference`, `per_position`, `mutation_prediction`, `naturalness`, `classification`, `routing`, and share fields for `data_path` (CSV or parquet), `antibody_datatype` (`paired` or `unpaired`), `dataset_name` (names the task's output directory; distinct per task), `num_proc` (dataset preparation workers; default 128), and `dataset_columns` (overrides for the expected column names). Paired data defaults to `sequence_id`, `sequence_aa_heavy`, `sequence_aa_light`, with optional `cdr_mask_heavy`/`cdr_mask_light` (a `0`/`1` string per residue, CDRs `1`) and `v_mutation_count_aa_heavy`/`v_mutation_count_aa_light` (a `0`/`1` string per residue, mutated `1`); unpaired data defaults to `sequence_aa` with optional `locus`, `cdr_mask`, `v_mutation_count_aa`. A classification task takes `data_path` as `{fold: {train: path, test: path}}`, with `num_folds` (default 5) and `num_classes` (default 2); leave `training_args` unset to train on AbLM-Eval's defaults (see Gotchas). A small config, whole: ```yaml output_dir: results @@ -54,29 +54,29 @@ A config names any of them as a bare string, or spells a model out in full — ` num_proc: 1 ``` -- **Out:** `/models.json` records the run's models and separators; each task writes its config to `/_/config.json` and one parquet per model under `results/` (the per-position file is `_-per-position-inference.parquet`). With more than one model, comparison plots and summary CSVs land beside them. The per-position parquet keeps the dataset's own columns plus `sequence` (chains joined by the separator), `tokenized_sequence`, `tokenized_seq_wo_special`, `loss` (one cross-entropy per non-special position: each position masked in turn, the true residue's negative log-probability), `perplexity` (`exp(loss)` per position), `probabilities` (the full next-token distribution per position), `prediction_tokens` and `prediction` (the model's top prediction per position). The other tasks write the analogous columns for their metrics — `inference` one mean loss per sequence, `mutation_prediction` and `naturalness` their per-mutation scores, `classification` per-fold metrics, `routing` the MoE expert assignments. +- **Out:** `/models.json` records the run's models and separators; each task writes its config to `/_/config.json` and one parquet per model under `results/` (the per-position file is `_-per-position-inference.parquet`). With more than one model, comparison plots and summary CSVs land beside them. The per-position parquet keeps the dataset's own columns plus `sequence` (chains joined by the separator), `tokenized_sequence`, `tokenized_seq_wo_special`, `loss` (one cross-entropy per non-special position: each position masked in turn, the true residue's negative log-probability), `perplexity` (`exp(loss)` per position), `probabilities` (the full next-token distribution per position), `prediction_tokens` and `prediction` (the model's top prediction per position). The other tasks write the analogous columns for their metrics — `inference` one mean loss per sequence, `mutation_prediction` and `naturalness` their per-mutation scores, `classification` one CSV per model at `/_classification/results/` with the fold's test metrics, `routing` the MoE expert assignments. ## Compute -One GPU per job: the models are small enough that one is faster than several through the DataParallel the code falls back to when it sees more. Per-position inference costs one forward pass per sequence position (each sequence becomes one batch of L masked copies), so its cost grows as sequence count times the square of length; the other inference tasks cost one pass per sequence. Measured inside the image on one b200, downloading each model once into `--cache-dir` and running `per_position` on the 8-pair fixture (heavy chains 124-134 residues, light 107-113): +One GPU per job: the models are small enough that one is faster than several through the DataParallel the code falls back to when it sees more. Per-position inference costs one forward pass per sequence position (each sequence becomes one batch of L masked copies), so its cost grows as sequence count times the square of length; the other inference tasks cost one pass per sequence. Measured on one b200 in a venv carrying the image's pinned package set on CPython 3.12, the image's Python — the image's twin, since the image cannot be measured before it publishes — downloading each model once into `--cache-dir` and running `per_position` on the 8-pair fixture (heavy chains 124-134 residues, light 107-113): | model | separator | mean per-residue log-probability | wall time | | --- | --- | --- | --- | -| `balm-paired` | `` | -0.3063 | 16.9 s | -| `balm-unpaired` | single chains only | -0.3713 | 12.6 s | -| `ft-esm` | `` | -0.2964 | 26.2 s | -| `balm-moe` | `` | does not load (see below) | 1.3 s | -| `currab` | `` | -0.2723 | 21.0 s | -| `preferential-250k` | `` | -0.2817 | 14.8 s | -| `uniform-250k` | `` | -0.2924 | 15.9 s | -| `uniform-350k` | `` | -0.2739 | 17.5 s | -| `scaling-350m-full` | `` | -0.2915 | 19.3 s | +| `balm-paired` | `` | -0.3063 | 18.5 s | +| `balm-unpaired` | single chains only | -0.3713 | 12.7 s | +| `ft-esm` | `` | -0.2964 | 25.9 s | +| `balm-moe` | `` | -0.2848 | 15.9 s | +| `currab` | `` | -0.2723 | 20.1 s | +| `preferential-250k` | `` | -0.2817 | 13.5 s | +| `uniform-250k` | `` | -0.2924 | 12.8 s | +| `uniform-350k` | `` | -0.2739 | 14.6 s | +| `scaling-350m-full` | `` | -0.2915 | 14.4 s | -Each row spans the line that announces the model — printed just before its download — to its results parquet, so its weights download is inside the number. `balm-paired` and `ft-esm` ran inside the one-config run of the eight paired models; that run stops at the first model that fails, and it stopped at `balm-moe`, the third of its eight, 56.4 s in, so `currab` through `scaling-350m-full` each ran in a config of their own. `balm-unpaired` ran the same eight heavy chains as an unpaired task, that run 19.5 s. `balm-moe`'s row is its own-config retry, which found its weights already fetched: it does not load — transformers 5.23 hands its special tokens down to BALM's `BalmTokenizer` as `AddedToken` objects, and the `WordLevel` that tokenizer builds on the pinned tokenizers 0.23.3 wants a plain `str` (`TypeError: 'AddedToken' object is not an instance of 'str'`, while processing `unk_token`) — so it is reported as a failed row, not dropped or worked around. +Each row spans the line that announces the model — printed just before its download — to its results parquet, so its weights download is inside the number. All nine load under AbLM-Eval's strict weight check and finish: the eight paired models ran in one config (154.9 s cold, to the final comparison), and `balm-unpaired` ran the same eight heavy chains as an unpaired task (29.6 s cold). The per-residue log-probabilities are deterministic — for the eight models the AbLM-Eval 32a3961 bump leaves alone they equal the 1.0.1 image's measurements — while wall times vary run to run. -Classification fine-tuning would cost one forward and backward pass per training step plus one predict pass over the test fold, but on the pinned versions it never reaches one: the HD-CoV check — fold 0 of the MoE paper's data (Zenodo record 19411192), one fold, 10 training steps, the 8M checkpoint, through `ablm-eval run` on one GPU with AbLM-Eval's default accelerate config — died at model construction on every attempt (22.0 s, 19.9 s and 27.5 s wall inside the image for the 8M checkpoint, the same with the task's `attention_classifier: false`, and `balm-paired`), before a single training step; the gotcha below says why. +Classification fine-tuning costs one forward and backward pass per training step plus one predict pass over the test fold. The HD-CoV check — fold 0 of the MoE paper's data (Zenodo record 19411192), one fold, through `ablm-eval run` on one GPU with AbLM-Eval's default accelerate config (DeepSpeed ZeRO 0, bf16, one process) and its default training arguments (no `training_args` in the task: warmup_steps 0.1, bf16, batch 32, 3 epochs of the fold's 20,257 rows, about 1,900 steps), measured in the same venv as the table above — trains and writes scores for both the 8M checkpoint (139.4 s wall from a cold cache, fetch to final metrics) and `balm-moe`, whose attention classifier is on by default (752.4 s); their test metrics on the fold's 5,065 test rows are from that short default-args run, not model quality: the 8M checkpoint's accuracy 0.7765, precision 0.7827, recall 0.7654, AUROC 0.8613, AUPR 0.8651, F1 0.7740, MCC 0.5531, and `balm-moe`'s accuracy 0.7822, precision 0.7816, recall 0.7832, AUROC 0.8569, AUPR 0.8374, F1 0.7824, MCC 0.5645. -The shed's test (`test/run.sh`) took 2.5 minutes of job wall on one b200 in the candidate's run (image import, the registry check, the 8M checkpoint fetch, `per_position` and the reference comparison); its `--overwrite` exercise — the refused rerun, the recomputation, the comparison again — adds well under a minute, the checkpoint being already in the cache. +The shed's test (`test/run.sh`) took 2.5 minutes of job wall on one b200 in the 1.0.1 candidate's run (image import, the registry check, the 8M checkpoint fetch, `per_position` and the reference comparison); the test is unchanged in 1.0.2, and its `--overwrite` exercise — the refused rerun, the recomputation, the comparison again — adds well under a minute, the checkpoint being already in the cache. ## Running it @@ -84,13 +84,13 @@ On a laptop, from the directory holding `config.yaml` and the data (the tasks us ```bash podman run --rm --device nvidia.com/gpu=all -v "$PWD":/work -w /work \ - ghcr.io/brineylab/toolshed/ablm-eval:1.0.1 ablm-eval run config.yaml --cache-dir /work/cache + ghcr.io/brineylab/toolshed/ablm-eval:1.0.2 ablm-eval run config.yaml --cache-dir /work/cache ``` On a Slurm cluster with pyxis: ```bash -srun --gres=gpu:1 --container-image=ghcr.io#brineylab/toolshed/ablm-eval:1.0.1 \ +srun --gres=gpu:1 --container-image=ghcr.io#brineylab/toolshed/ablm-eval:1.0.2 \ --container-mounts="$PWD":/work --container-workdir=/work \ ablm-eval run config.yaml --cache-dir /work/cache ``` @@ -104,6 +104,6 @@ The model files land in `--cache-dir` (the default `.ablm-eval-cache` sits in th - **The separator comes from the model:** each registered model carries the separator it was trained with and applies it to every task; setting `separator` in a task config alongside such a model is refused. Only full model entries with `separator` of their own, and `balm-unpaired` (which leaves the task's separator alone), differ. - **`num_proc` defaults to 128** dataset-preparation workers; on a node with fewer cores, set it to what the node has. - **Rich wraps at 80 columns** when a terminal is not a tty: export `COLUMNS=1000` before `ablm-eval models` in scripts, or the table's repository and revision columns wrap. -- **Classification (does not work on the pinned versions):** it fine-tunes a head with transformers' `Trainer` through accelerate — run non-interactively it uses accelerate's default config (DeepSpeed ZeRO 0, bf16, one process) without prompting — but it dies before training: AbLM-Eval's train script passes `attention_classifier` into `AutoModelForSequenceClassification.from_pretrained` unconditionally, and on transformers 5.23 a kwarg that is neither a recognized config field nor an attribute of the checkpoint's config is forwarded to the model constructor, so every model that loads here (each a plain esm or roberta config) fails with `TypeError: ... got an unexpected keyword argument 'attention_classifier'`. The HD-CoV check tried it three ways on one GPU — the 8M checkpoint, the same with the task's `attention_classifier: false` (the kwarg is passed whatever its value), and `balm-paired`, whose published config.json is a plain roberta config — and all three failed identically (22.0 s, 19.9 s and 27.5 s wall, inside the image). The only registered model whose config defines the field is `balm-moe`, which fails earlier at its tokenizer (the failed row above). A classification task with no `training_args` dies even earlier, at `TrainingArguments`: transformers 5.23 removed `warmup_ratio`, which AbLM-Eval's fallback still passes — spell the warmup as `warmup_steps` (an int, or a float in [0, 1) for the ratio). Fixing classification needs a change in AbLM-Eval itself — pass `attention_classifier` only when the checkpoint's config defines it, or stop passing it — so until upstream changes, use the image's other tasks. One trap is the caller's, not AbLM-Eval's: under Slurm/pyxis the caller's environment is passed into the container, and an ambient `LOCAL_RANK`/`RANK` with no `WORLD_SIZE` makes transformers' `TrainingArguments` — built while the CLI validates the config, before any task — init `torch.distributed` at the `env://` rendezvous and die (`environment variable WORLD_SIZE expected, but not set`); unset them for the CLI, which is a single process that launches its own training through accelerate, and sets the training subprocess's variables itself. -- **The test** (`test/run.sh`) runs `per_position` through the CLI on AbLM-Eval's own fixture `test/paired-mini.parquet` — the 8 public pairs from Jaffe et al. 2022 (figshare 19617633), heavy chains 124-134 and light 107-113 residues, so each sequence builds a real masked batch of 107-134 rows — with the 8M checkpoint `brineylab/8M_full_checkpoint-500000` @ `01e6532f` spelled out as a full model entry with separator ``, fetched at test time. It compares the pooled mean per-residue log-probability and each sequence's mean with `test/reference.json`, computed once with `evaluate_ablms` (AbLM-Eval's Python API) at commit `641cff8` on the same fixture, model and pinned versions, to 1e-4. It also checks every registered model's name is listed by `ablm-eval models`, and exercises the documented `--overwrite`: the same config rerun without it is refused (`The directory '...' exists and is not empty!`), and rerun with it the results are recomputed and the comparison passes again. +- **Classification's `attention_classifier` follows the model:** the task field is unset by default — a model whose config defines it (`balm-moe` only) gets it on, every other model gets nothing, and setting it on a model whose config does not define it is refused with a `ValueError` naming the model. A task with no `training_args` trains on AbLM-Eval's defaults (warmup_steps 0.1, bf16, batch 32, 3 epochs, report_to none); spell out `training_args` only to depart from them. One trap is the caller's, not AbLM-Eval's: under Slurm/pyxis the caller's environment is passed into the container, and an ambient `LOCAL_RANK`/`RANK` with no `WORLD_SIZE` kills a CLI validating a config whose task carries `training_args` — transformers' `TrainingArguments`, built while the config is validated, inits `torch.distributed` at the `env://` rendezvous and dies (`environment variable WORLD_SIZE expected, but not set`); unset them for the CLI, which is a single process that launches its own training through accelerate and sets the training subprocess's variables itself. A task with no `training_args` builds no `TrainingArguments` at validation, so it does not hit the trap. +- **The test** (`test/run.sh`) runs `per_position` through the CLI on AbLM-Eval's own fixture `test/paired-mini.parquet` — the 8 public pairs from Jaffe et al. 2022 (figshare 19617633), heavy chains 124-134 and light 107-113 residues, so each sequence builds a real masked batch of 107-134 rows — with the 8M checkpoint `brineylab/8M_full_checkpoint-500000` @ `01e6532f` spelled out as a full model entry with separator ``, fetched at test time. It compares the pooled mean per-residue log-probability and each sequence's mean with `test/reference.json`, computed once with `evaluate_ablms` (AbLM-Eval's Python API) at commit `641cff8` on the same fixture, model and pinned versions, to 1e-4 — unchanged for 1.0.2: the AbLM-Eval bump touches no per-position inference path, and the reference still matches. It also checks every registered model's name is listed by `ablm-eval models`, and exercises the documented `--overwrite`: the same config rerun without it is refused (`The directory '...' exists and is not empty!`), and rerun with it the results are recomputed and the comparison passes again. - **Licenses:** AbLM-Eval is MIT; every model the registry names (and the 8M test checkpoint) is MIT-licensed on the Hub. The image carries no weights — it downloads them at run time — so a run needs the network and a place to put `--cache-dir`. From 8a317dc7a022d55717e1a32f67ccc38ba54642c2 Mon Sep 17 00:00:00 2001 From: Sarah Burbach <106550373+smburbach@users.noreply.github.com> Date: Thu, 19 Nov 2026 17:10:00 -0800 Subject: [PATCH 2/2] ablm-eval 1.0.2's record: its candidate passed on coreweave-barcelona --- results/ablm-eval/1.0.2.txt | 66 +++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 results/ablm-eval/1.0.2.txt diff --git a/results/ablm-eval/1.0.2.txt b/results/ablm-eval/1.0.2.txt new file mode 100644 index 0000000..c9ccf76 --- /dev/null +++ b/results/ablm-eval/1.0.2.txt @@ -0,0 +1,66 @@ +--- +tool: ablm-eval +version: 1.0.2 +image: ghcr.io/brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +digest: sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +runner: sbatch --partition=flyloft --gres=gpu:1 --time=00:30:00 +host: sunk.6af0ac-eu-south-03b-briney-b200.coreweave.app +started: 2026-11-20T01:08:02Z +exit: 0 +--- +Job 1077: SUNK job dashboard: https://cks-grafana.coreweave.com/d/slurm-job-metrics/slurm-job-metrics?var-slurm_cluster=slurm&var-job_id=1077 +pyxis: importing docker image: ghcr.io#brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/ablm-eval@sha256:8c932603546af070394b0747658992b298f6874913d8b43b61233389d64c88a9 +node: slurm-b200-204-039 +GPU 0: NVIDIA B200 (UUID: GPU-e3ae7af6-08de-017b-aedd-15ab9be767f6) +┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓ +┃ name ┃ repository ┃ revision ┃ separator ┃ +┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩ +│ balm-paired │ brineylab/BALM-paired │ 78e6a59276db │ │ +│ balm-unpaired │ brineylab/BALM-unpaired │ 8fd18a5b47bf │ single chains only │ +│ ft-esm │ brineylab/ft-ESM │ 6594c8b61de1 │ │ +│ balm-moe │ brineylab/BALM-MoE │ 51842e69ca3f │ │ +│ currab │ brineylab/CurrAb │ 92e28534663e │ │ +│ preferential-250k │ brineylab/preferential-250k │ 72ebb7b3e0f1 │ │ +│ uniform-250k │ brineylab/uniform-250k │ bb2fd2338300 │ │ +│ uniform-350k │ brineylab/uniform-350k │ 43a1e4fd35cf │ │ +│ scaling-350m-full │ brineylab/350M_full_checkpoint-500000 │ 9cd9e8af5138 │ │ +└───────────────────┴───────────────────────────────────────┴──────────────┴────────────────────┘ +tiny: brineylab/8M_full_checkpoint-500000 @ 01e6532f9091 + Fetching 7 files: 0%| | 0/7 [00:00