Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions datasets/jaffe-paired/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
name: jaffe-paired
description: "Paired human antibody heavy and light chains from four donors' sorted B cells (Jaffe et al. 2022), annotated with abstar, clustered at 85% identity and split by cluster, train reduced at 95%. Reach for it to train or evaluate a paired-chain antibody model."
metadata:
version: 1.0.0
tools: datascience@1.2.0
license: mit
---
# jaffe-paired

Hub: `brineylab/jaffe-paired`, tag `v1.0.0`

## Files

`train.parquet`, `eval.parquet`, `test.parquet`: one row per pair, a heavy and a light chain from one droplet. Columns:

- `pair_id`: `<barcode>_<sample>` (`AAACCTGAGACACGAC-1_1279049`).
- `sample`, `donor` (1 to 4), `flow_class` (`naive`, `unswitched`, `switched`, `plasmablast`, `unswitched + naive`, `switched + naive`): the sample's row of the source's `test_data.csv`.
- `cluster_85`: the `pair_id` of the pair's 85% cluster representative. In eval and test every row is its own representative.
- `cluster_95` (train only): the `pair_id` of its 95% representative; every train row is its own.
- Every field abstar 0.9.0 writes, once per chain: the heavy chain's with `:0` appended (`v_call:0`, `cdr3_aa:0`, `sequence_vdj_aa:0`), the light chain's with `:1`. `sequence_vdj_aa` is the variable domain, FWR1 through FWR4.

## Reading it

```python
from huggingface_hub import snapshot_download
import polars as pl
path = snapshot_download("brineylab/jaffe-paired", repo_type="dataset", revision="v1.0.0")
train = pl.read_parquet(f"{path}/train.parquet", columns=["pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1"])
```

Name the columns you need: each file holds some 340.

## Recipe

**Source:** Jaffe et al. 2022, figshare article 19617633 version 3 (DOI `10.6084/m9.figshare.19617633.v3`, MIT), pulled 2026-11-06: `test_data.tar.gz` (file 34846953, MD5 `4d87c4da2d407ba95180be2cb1c91044`), of which each sample's Cell Ranger `vdj_b/filtered_contig.fasta` (cell-associated contigs, productive, high-confidence, and full-length by Cell Ranger), and `test_data.csv` (file 34846938, MD5 `3ccae335a8a03b5a771dc0eae97f7af6`), the 94 samples' donors and flow classes. The recipe refuses a download whose MD5 differs.

**Steps**, one job in `datascience@1.2.0`:

1. Annotate every contig with abstar 0.9.0 (`germline_database = "human"`). 0.9.0 matters: it writes `sequence_vdj_aa`, the variable domain, which the clustering uses; in 0.9.0 `sequence_aa` is the translated input, as AIRR 2.0 defines it.
2. Pair within each sample: a droplet's barcode is the contig id up to its first underscore. Keep a droplet only with exactly one heavy chain (`IGH`) and one light (`IGK` or `IGL`), both productive by abstar. The build prints what it dropped, by shape (`dropped 1H2L: <n>`) and for productivity (`dropped unproductive: <n>`); the record carries those lines.
3. Cluster every pair with `mmseqs easy-cluster --min-seq-id 0.85 -c 0.8 --cov-mode 0` (`identity`, `coverage`; 80% of both sequences) on the heavy chain's `sequence_vdj_aa` joined to the light chain's. One `easy-cluster` run over every pair: `datascience@1.2.0`'s mmseqs (master, commit 2a3dafc) carries the fix to the prefilter overflow (MMseqs2 #1092) that kills release 18 and abutils' bundled build on about 1.1M of these sequences.
4. Split by cluster `[0.96, 0.02, 0.02]`: the representatives, sorted, are shuffled with `random.Random(seed = 1)`, and whole clusters go to train until it holds 96% of the pairs, then to eval until 2%, then to test. No 85% cluster spans two splits. Eval and test keep the 85% representatives.
5. Re-cluster every train pair at 95% (`train_identity`), the same way; train keeps the 95% representatives.

`params.toml` also holds `limit = 0`, the number of samples for a developer's dry run; 0 is all.

**The test** checks the three files and their columns, each split's rows against a floor, no `pair_id` or `cluster_85` in two splits, both chains productive, `locus:0` `IGH` and `locus:1` `IGK` or `IGL`, and each `sequence_vdj_aa` free of gaps and stops and ending with its chain's `fwr4_aa`.

## Compute

One job, CPU only: 32 CPUs and 200G asked, on coreweave-rtx6000 `hpc-low` (Xeon Platinum 8562Y+). From nothing it takes about 4 h 20 min: the download (12 GB) 6 min; abstar and the pairing, sample by sample, 2 h 10 min; the 85% clustering of all 1,389,546 pairs 7 min; the 95% clustering of train's 1.33M 1 h 50 min. Peak memory 81 GB, in the annotation. Disk: 1.5 GB of files and 19 GB of stages under `<out>/.work/`, the tarball 12 GB of it; a rerun resumes from the last finished stage.

## Gotchas

- Barcodes are unique only within a sample; `pair_id` carries the sample for that reason.
- The droplet rule drops about 13% of droplets by shape (215,163 of 1,618,287 across the 94 samples) and 13,578 more for productivity. The rate varies by sample: 1279049's is 8.7% (389 with two lights, 299 with two of each, 49 with two heavies, 118 with one chain, of 9,808).
- Eval and test hold one pair per 85% cluster, so they are far fewer rows than 2% of the pairs: they are cluster-level held-out sets.
- `donor` makes a held-out-donor split possible; that is another dataset, a copy of this one with the split changed.
- With podman on a laptop, run with `--pids-limit=-1`: abstar's workers pass podman's default limit of 2048 threads.
187 changes: 187 additions & 0 deletions datasets/jaffe-paired/recipe/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
#!/usr/bin/env -S python3 -I
"""recipe/build <out>: paired human antibodies from Jaffe et al. 2022 (figshare 19617633 v3), annotated with
abstar, one heavy and one light chain per droplet, clustered with mmseqs2 at `identity`, split by cluster into
train, eval, and test, train reduced at `train_identity`; written as <out>/{train,eval,test}.parquet.

Stages under <out>/.work/, each skipped when a rerun finds it finished. Prints `dropped <why>: <n>` lines,
then `pairs <split>: <n>` lines last.
"""
import hashlib
import json
import os
import random
import shutil
import subprocess
import sys
import tomllib
import urllib.request
from pathlib import Path

import polars as pl

# figshare file id: (name, size, MD5) of article 19617633 version 3
SOURCE = {34846953: ("test_data.tar.gz", 11941014666, "4d87c4da2d407ba95180be2cb1c91044"),
34846938: ("test_data.csv", 1981, "3ccae335a8a03b5a771dc0eae97f7af6")}
FASTA = "test_data/*/outs/per_sample_outs/*/vdj_b/filtered_contig.fasta"
LIGHT = ("IGK", "IGL")

params = tomllib.loads((Path(__file__).parent / "params.toml").read_text())
out = Path(sys.argv[1]).resolve()
work = out / ".work"
cpus = len(os.sched_getaffinity(0)) # the job's CPUs, not the node's


def run(*cmd):
subprocess.run([str(c) for c in cmd], check=True, stdout=subprocess.DEVNULL)


def download(dest: Path) -> None:
dest.mkdir(parents=True, exist_ok=True)
for fid, (name, size, md5) in SOURCE.items():
path = dest / name
if path.exists():
continue
part, h = path.with_suffix(".part"), hashlib.md5()
req = urllib.request.Request(f"https://ndownloader.figshare.com/files/{fid}", headers={"User-Agent": "brineylab-datasets"})
with urllib.request.urlopen(req) as r, open(part, "wb") as f:
for block in iter(lambda: r.read(1 << 22), b""):
h.update(block)
f.write(block)
if h.hexdigest() != md5 or part.stat().st_size != size:
part.unlink()
sys.exit(f"{fid}: MD5 {h.hexdigest()} is not {md5}")
part.rename(path)


def extract(tarball: Path, dest: Path) -> None:
"""Each sample's filtered_contig.fasta as dest/<sample>.fasta."""
if dest.exists():
return
tmp = dest.with_suffix(".tmp")
shutil.rmtree(tmp, ignore_errors=True)
tmp.mkdir(parents=True)
run("tar", "-xzf", tarball, "-C", tmp, "--wildcards", FASTA)
for f in tmp.glob(FASTA):
f.rename(tmp / f"{f.parts[-6]}.fasta")
shutil.rmtree(tmp / "test_data")
tmp.rename(dest)


def annotate(fasta: Path, dest: Path) -> Path:
"""abstar's parquet for one sample, named <sample>.parquet."""
done = dest / f"{fasta.stem}.parquet"
if not done.exists():
tmp = dest.with_name(dest.name + ".tmp")
shutil.rmtree(tmp, ignore_errors=True)
run("abstar", "run", fasta, tmp, "--germline_database", params["germline_database"], "-o", "parquet",
"--n_processes", cpus, "--mmseqs_threads", cpus, "--no-copy-inputs")
dest.mkdir(parents=True, exist_ok=True)
(tmp / "parquet" / done.name).rename(done)
shutil.rmtree(tmp)
return done


def pair(sample: str, fasta: Path, annotated: Path, dest: Path) -> dict:
"""One row per droplet with one heavy and one light chain, both productive, as dest/<sample>.parquet;
returns the droplets dropped, by why."""
counts_path = dest / f"{sample}.json"
if counts_path.exists():
return json.loads(counts_path.read_text())
ids = [line[1:].split()[0] for line in fasta.read_text().splitlines() if line.startswith(">")]
ab = pl.read_parquet(annotated)
contigs = (pl.DataFrame({"sequence_id": ids})
.join(ab.select("sequence_id", "locus"), on="sequence_id", how="left")
.with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode")))
shapes = (contigs.group_by("barcode")
.agg(pl.col("locus").is_null().sum().alias("unannotated"),
(pl.col("locus") == "IGH").sum().alias("h"),
pl.col("locus").is_in(LIGHT).sum().alias("l"))
.with_columns(pl.when(pl.col("unannotated") > 0).then(pl.lit("unannotated"))
.otherwise(pl.format("{}H{}L", "h", "l")).alias("shape")))
counts = {k: v for k, v in shapes.group_by("shape").len().iter_rows() if k != "1H1L"}
keep = shapes.filter(pl.col("shape") == "1H1L").select("barcode")
chains = ab.with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode")).join(keep, on="barcode")
heavy = chains.filter(pl.col("locus") == "IGH")
light = chains.filter(pl.col("locus").is_in(LIGHT))
pairs = heavy.rename({c: f"{c}:0" for c in ab.columns}).join(light.rename({c: f"{c}:1" for c in ab.columns}), on="barcode")
productive = pairs.filter(pl.col("productive:0") & pl.col("productive:1"))
counts["unproductive"] = pairs.height - productive.height
dest.mkdir(parents=True, exist_ok=True)
(productive.with_columns(pl.format("{}_{}", "barcode", pl.lit(sample)).alias("pair_id"), pl.lit(sample).alias("sample"))
.drop("barcode").write_parquet(dest / f"{sample}.parquet"))
counts_path.write_text(json.dumps(counts))
return counts


def easy_cluster(seqs: dict, identity: float, dest: Path) -> dict:
"""Each id's representative, mmseqs2 easy-cluster at `identity` and `coverage` of both sequences."""
tsv = dest / "clu_cluster.tsv"
if not tsv.exists():
shutil.rmtree(dest, ignore_errors=True)
dest.mkdir(parents=True)
with open(dest / "seqs.fasta", "w") as f:
f.writelines(f">{i}\n{s}\n" for i, s in seqs.items())
run("mmseqs", "easy-cluster", dest / "seqs.fasta", dest / "clu", dest / "tmp", "--min-seq-id", identity,
"-c", params["coverage"], "--cov-mode", 0, "--threads", cpus)
shutil.rmtree(dest / "tmp")
return {member: rep for rep, member in (line.split("\t") for line in tsv.read_text().splitlines())}


def cluster(pairs: pl.DataFrame, identity: float, dest: Path) -> pl.DataFrame:
"""pair_id and its representative's pair_id, clustering the heavy chain's sequence_vdj_aa joined to the
light chain's."""
seqs = {pid: h + l for pid, h, l in pairs.select("pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1").iter_rows()}
reps = easy_cluster(seqs, identity, dest)
return pl.DataFrame({"rep": list(reps.values()), "pair_id": list(reps)})


def main() -> None:
download(work / "download")
extract(work / "download" / "test_data.tar.gz", work / "fasta")
samples = pl.read_csv(work / "download" / "test_data.csv", infer_schema=False).rename({"dataset": "sample"})
samples = samples.with_columns(pl.col("donor").cast(pl.Int64))
if params["limit"]:
samples = samples.head(params["limit"])
dropped = {}
for sample in samples["sample"]:
fasta = work / "fasta" / f"{sample}.fasta"
for why, n in pair(sample, fasta, annotate(fasta, work / "abstar" / sample), work / "pairs").items():
dropped[why] = dropped.get(why, 0) + n
for why, n in sorted(dropped.items(), key=lambda kv: (kv[0] == "unproductive", -kv[1], kv[0])):
print(f"dropped {why}: {n}")
pairs = pl.concat([pl.read_parquet(work / "pairs" / f"{s}.parquet") for s in samples["sample"]], how="diagonal_relaxed")
pairs = pairs.join(samples, on="sample")
print(f"pairs kept: {pairs.height}")

c85 = cluster(pairs, params["identity"], work / "mm85").rename({"rep": "cluster_85"})
sizes = dict(c85.group_by("cluster_85").len().iter_rows())
reps = sorted(sizes)
random.Random(params["seed"]).shuffle(reps)
total, (f_train, f_eval, _) = pairs.height, params["split"]
assigned, n_train, n_eval = {}, 0, 0
for rep in reps:
if n_train < f_train * total:
assigned[rep], n_train = "train", n_train + sizes[rep]
elif n_eval < f_eval * total:
assigned[rep], n_eval = "eval", n_eval + sizes[rep]
else:
assigned[rep] = "test"
split = pl.DataFrame({"cluster_85": list(assigned), "split": list(assigned.values())})
pairs = pairs.join(c85, on="pair_id").join(split, on="cluster_85")

front = ["pair_id", "sample", "donor", "flow_class", "cluster_85"]
chains = [c for c in pairs.columns if c.endswith((":0", ":1"))]
train = pairs.filter(pl.col("split") == "train")
c95 = cluster(train, params["train_identity"], work / "mm95").rename({"rep": "cluster_95"})
train = train.join(c95, on="pair_id").filter(pl.col("pair_id") == pl.col("cluster_95"))
files = {"train": train.select(*front, "cluster_95", *chains)}
for name in ("eval", "test"):
files[name] = pairs.filter((pl.col("split") == name) & (pl.col("pair_id") == pl.col("cluster_85"))).select(*front, *chains)
for name, df in files.items():
df.sort("pair_id").write_parquet(out / f"{name}.parquet")
for name, df in files.items():
print(f"pairs {name}: {df.height}")


if __name__ == "__main__":
main()
10 changes: 10 additions & 0 deletions datasets/jaffe-paired/recipe/params.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
pulled = "2026-11-06"
article = 19617633
article_version = 3
germline_database = "human"
identity = 0.85
coverage = 0.8
train_identity = 0.95
split = [0.96, 0.02, 0.02]
seed = 1
limit = 0 # samples, for a developer's dry run; 0 is all, and the published version has 0
64 changes: 64 additions & 0 deletions datasets/jaffe-paired/recipe/test
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env -S python3 -I
"""recipe/test <out>: exit 0 iff <out> holds a version of jaffe-paired. Reads the parquet footers and only the
columns each check needs; one line per failure, naming the check and the first offending pair_id."""
import sys
from pathlib import Path

import polars as pl

FLOORS = {"train": 1_000_000, "eval": 1_200, "test": 4_300} # rows: 90% of the first full build's, to two figures
FRONT = ["pair_id", "sample", "donor", "flow_class", "cluster_85"]
FIELDS = ["sequence_id", "locus", "productive", "v_call", "j_call", "sequence_vdj", "sequence_vdj_aa", "cdr3_aa", "fwr4_aa"]

out, problems = Path(sys.argv[1]), []
frames, chain_columns = {}, None
for split, floor in FLOORS.items():
path = out / f"{split}.parquet"
if not path.is_file():
problems.append(f"files: {split}.parquet is missing")
continue
lf = pl.scan_parquet(path)
columns = lf.collect_schema().names()
want = FRONT + (["cluster_95"] if split == "train" else []) + [f"{f}:{i}" for f in FIELDS for i in (0, 1)]
missing = [c for c in want if c not in columns]
if missing:
problems.append(f"columns: {split}.parquet lacks {missing}")
continue
chains = sorted(c for c in columns if c.endswith((":0", ":1")))
if {c[:-2] for c in chains if c.endswith(":0")} != {c[:-2] for c in chains if c.endswith(":1")}:
problems.append(f"columns: {split}.parquet's heavy and light fields differ")
if chain_columns is not None and chains != chain_columns:
problems.append(f"columns: {split}.parquet's chain fields are not train's")
chain_columns = chain_columns or chains
rows = lf.select(pl.len()).collect().item()
if rows < floor:
problems.append(f"rows: {split}.parquet has {rows}, under its floor {floor}")
df = lf.select("pair_id", "cluster_85", *[f"{f}:{i}" for f in ("locus", "productive", "sequence_vdj_aa", "fwr4_aa") for i in (0, 1)]).collect()
frames[split] = df.select("pair_id", "cluster_85")
bad = {
"productive": ~(pl.col("productive:0") & pl.col("productive:1")),
"locus": (pl.col("locus:0") != "IGH") | ~pl.col("locus:1").is_in(["IGK", "IGL"]),
}
for i in (0, 1):
seq = pl.col(f"sequence_vdj_aa:{i}")
bad[f"sequence_vdj_aa:{i} gap or stop"] = seq.str.contains(r"[-.*]")
bad[f"sequence_vdj_aa:{i} ends with fwr4_aa:{i}"] = ~seq.str.ends_with(pl.col(f"fwr4_aa:{i}"))
for check, expr in bad.items():
hit = df.filter(expr.fill_null(True))
if hit.height:
problems.append(f"{check}: {hit.height} rows in {split}.parquet, first {hit['pair_id'][0]}")
print(f"{split}.parquet {rows} rows")

for key in ("pair_id", "cluster_85"):
seen = {}
for split, df in frames.items():
for value in df[key].unique():
if value in seen:
problems.append(f"{key}: {value} is in {seen[value]} and {split}")
break
seen[value] = split
for split, df in frames.items():
if df["pair_id"].n_unique() != df.height:
problems.append(f"pair_id: {split}.parquet repeats one")
print("\n".join(problems) or "ok")
sys.exit(1 if problems else 0)
Loading
Loading