diff --git a/datasets/jaffe-paired/SKILL.md b/datasets/jaffe-paired/SKILL.md new file mode 100644 index 0000000..fa438a6 --- /dev/null +++ b/datasets/jaffe-paired/SKILL.md @@ -0,0 +1,60 @@ +--- +name: jaffe-paired +description: "Paired human antibody heavy and light chains from four donors' sorted B cells (Jaffe et al. 2022), annotated with abstar, clustered at 85% identity and split by cluster, train reduced at 95%. Reach for it to train or evaluate a paired-chain antibody model." +metadata: + version: 1.0.0 + tools: datascience@1.2.0 + license: mit +--- +# jaffe-paired + +Hub: `brineylab/jaffe-paired`, tag `v1.0.0` + +## Files + +`train.parquet`, `eval.parquet`, `test.parquet`: one row per pair, a heavy and a light chain from one droplet. Columns: + +- `pair_id`: `_` (`AAACCTGAGACACGAC-1_1279049`). +- `sample`, `donor` (1 to 4), `flow_class` (`naive`, `unswitched`, `switched`, `plasmablast`, `unswitched + naive`, `switched + naive`): the sample's row of the source's `test_data.csv`. +- `cluster_85`: the `pair_id` of the pair's 85% cluster representative. In eval and test every row is its own representative. +- `cluster_95` (train only): the `pair_id` of its 95% representative; every train row is its own. +- Every field abstar 0.9.0 writes, once per chain: the heavy chain's with `:0` appended (`v_call:0`, `cdr3_aa:0`, `sequence_vdj_aa:0`), the light chain's with `:1`. `sequence_vdj_aa` is the variable domain, FWR1 through FWR4. + +## Reading it + +```python +from huggingface_hub import snapshot_download +import polars as pl +path = snapshot_download("brineylab/jaffe-paired", repo_type="dataset", revision="v1.0.0") +train = pl.read_parquet(f"{path}/train.parquet", columns=["pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1"]) +``` + +Name the columns you need: each file holds some 340. + +## Recipe + +**Source:** Jaffe et al. 2022, figshare article 19617633 version 3 (DOI `10.6084/m9.figshare.19617633.v3`, MIT), pulled 2026-11-06: `test_data.tar.gz` (file 34846953, MD5 `4d87c4da2d407ba95180be2cb1c91044`), of which each sample's Cell Ranger `vdj_b/filtered_contig.fasta` (cell-associated contigs, productive, high-confidence, and full-length by Cell Ranger), and `test_data.csv` (file 34846938, MD5 `3ccae335a8a03b5a771dc0eae97f7af6`), the 94 samples' donors and flow classes. The recipe refuses a download whose MD5 differs. + +**Steps**, one job in `datascience@1.2.0`: + +1. Annotate every contig with abstar 0.9.0 (`germline_database = "human"`). 0.9.0 matters: it writes `sequence_vdj_aa`, the variable domain, which the clustering uses; in 0.9.0 `sequence_aa` is the translated input, as AIRR 2.0 defines it. +2. Pair within each sample: a droplet's barcode is the contig id up to its first underscore. Keep a droplet only with exactly one heavy chain (`IGH`) and one light (`IGK` or `IGL`), both productive by abstar. The build prints what it dropped, by shape (`dropped 1H2L: `) and for productivity (`dropped unproductive: `); the record carries those lines. +3. Cluster every pair with `mmseqs easy-cluster --min-seq-id 0.85 -c 0.8 --cov-mode 0` (`identity`, `coverage`; 80% of both sequences) on the heavy chain's `sequence_vdj_aa` joined to the light chain's. One `easy-cluster` run over every pair: `datascience@1.2.0`'s mmseqs (master, commit 2a3dafc) carries the fix to the prefilter overflow (MMseqs2 #1092) that kills release 18 and abutils' bundled build on about 1.1M of these sequences. +4. Split by cluster `[0.96, 0.02, 0.02]`: the representatives, sorted, are shuffled with `random.Random(seed = 1)`, and whole clusters go to train until it holds 96% of the pairs, then to eval until 2%, then to test. No 85% cluster spans two splits. Eval and test keep the 85% representatives. +5. Re-cluster every train pair at 95% (`train_identity`), the same way; train keeps the 95% representatives. + +`params.toml` also holds `limit = 0`, the number of samples for a developer's dry run; 0 is all. + +**The test** checks the three files and their columns, each split's rows against a floor, no `pair_id` or `cluster_85` in two splits, both chains productive, `locus:0` `IGH` and `locus:1` `IGK` or `IGL`, and each `sequence_vdj_aa` free of gaps and stops and ending with its chain's `fwr4_aa`. + +## Compute + +One job, CPU only: 32 CPUs and 200G asked, on coreweave-rtx6000 `hpc-low` (Xeon Platinum 8562Y+). From nothing it takes about 4 h 20 min: the download (12 GB) 6 min; abstar and the pairing, sample by sample, 2 h 10 min; the 85% clustering of all 1,389,546 pairs 7 min; the 95% clustering of train's 1.33M 1 h 50 min. Peak memory 81 GB, in the annotation. Disk: 1.5 GB of files and 19 GB of stages under `/.work/`, the tarball 12 GB of it; a rerun resumes from the last finished stage. + +## Gotchas + +- Barcodes are unique only within a sample; `pair_id` carries the sample for that reason. +- The droplet rule drops about 13% of droplets by shape (215,163 of 1,618,287 across the 94 samples) and 13,578 more for productivity. The rate varies by sample: 1279049's is 8.7% (389 with two lights, 299 with two of each, 49 with two heavies, 118 with one chain, of 9,808). +- Eval and test hold one pair per 85% cluster, so they are far fewer rows than 2% of the pairs: they are cluster-level held-out sets. +- `donor` makes a held-out-donor split possible; that is another dataset, a copy of this one with the split changed. +- With podman on a laptop, run with `--pids-limit=-1`: abstar's workers pass podman's default limit of 2048 threads. diff --git a/datasets/jaffe-paired/recipe/build b/datasets/jaffe-paired/recipe/build new file mode 100755 index 0000000..cd8e999 --- /dev/null +++ b/datasets/jaffe-paired/recipe/build @@ -0,0 +1,187 @@ +#!/usr/bin/env -S python3 -I +"""recipe/build : paired human antibodies from Jaffe et al. 2022 (figshare 19617633 v3), annotated with +abstar, one heavy and one light chain per droplet, clustered with mmseqs2 at `identity`, split by cluster into +train, eval, and test, train reduced at `train_identity`; written as /{train,eval,test}.parquet. + +Stages under /.work/, each skipped when a rerun finds it finished. Prints `dropped : ` lines, +then `pairs : ` lines last. +""" +import hashlib +import json +import os +import random +import shutil +import subprocess +import sys +import tomllib +import urllib.request +from pathlib import Path + +import polars as pl + +# figshare file id: (name, size, MD5) of article 19617633 version 3 +SOURCE = {34846953: ("test_data.tar.gz", 11941014666, "4d87c4da2d407ba95180be2cb1c91044"), + 34846938: ("test_data.csv", 1981, "3ccae335a8a03b5a771dc0eae97f7af6")} +FASTA = "test_data/*/outs/per_sample_outs/*/vdj_b/filtered_contig.fasta" +LIGHT = ("IGK", "IGL") + +params = tomllib.loads((Path(__file__).parent / "params.toml").read_text()) +out = Path(sys.argv[1]).resolve() +work = out / ".work" +cpus = len(os.sched_getaffinity(0)) # the job's CPUs, not the node's + + +def run(*cmd): + subprocess.run([str(c) for c in cmd], check=True, stdout=subprocess.DEVNULL) + + +def download(dest: Path) -> None: + dest.mkdir(parents=True, exist_ok=True) + for fid, (name, size, md5) in SOURCE.items(): + path = dest / name + if path.exists(): + continue + part, h = path.with_suffix(".part"), hashlib.md5() + req = urllib.request.Request(f"https://ndownloader.figshare.com/files/{fid}", headers={"User-Agent": "brineylab-datasets"}) + with urllib.request.urlopen(req) as r, open(part, "wb") as f: + for block in iter(lambda: r.read(1 << 22), b""): + h.update(block) + f.write(block) + if h.hexdigest() != md5 or part.stat().st_size != size: + part.unlink() + sys.exit(f"{fid}: MD5 {h.hexdigest()} is not {md5}") + part.rename(path) + + +def extract(tarball: Path, dest: Path) -> None: + """Each sample's filtered_contig.fasta as dest/.fasta.""" + if dest.exists(): + return + tmp = dest.with_suffix(".tmp") + shutil.rmtree(tmp, ignore_errors=True) + tmp.mkdir(parents=True) + run("tar", "-xzf", tarball, "-C", tmp, "--wildcards", FASTA) + for f in tmp.glob(FASTA): + f.rename(tmp / f"{f.parts[-6]}.fasta") + shutil.rmtree(tmp / "test_data") + tmp.rename(dest) + + +def annotate(fasta: Path, dest: Path) -> Path: + """abstar's parquet for one sample, named .parquet.""" + done = dest / f"{fasta.stem}.parquet" + if not done.exists(): + tmp = dest.with_name(dest.name + ".tmp") + shutil.rmtree(tmp, ignore_errors=True) + run("abstar", "run", fasta, tmp, "--germline_database", params["germline_database"], "-o", "parquet", + "--n_processes", cpus, "--mmseqs_threads", cpus, "--no-copy-inputs") + dest.mkdir(parents=True, exist_ok=True) + (tmp / "parquet" / done.name).rename(done) + shutil.rmtree(tmp) + return done + + +def pair(sample: str, fasta: Path, annotated: Path, dest: Path) -> dict: + """One row per droplet with one heavy and one light chain, both productive, as dest/.parquet; + returns the droplets dropped, by why.""" + counts_path = dest / f"{sample}.json" + if counts_path.exists(): + return json.loads(counts_path.read_text()) + ids = [line[1:].split()[0] for line in fasta.read_text().splitlines() if line.startswith(">")] + ab = pl.read_parquet(annotated) + contigs = (pl.DataFrame({"sequence_id": ids}) + .join(ab.select("sequence_id", "locus"), on="sequence_id", how="left") + .with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode"))) + shapes = (contigs.group_by("barcode") + .agg(pl.col("locus").is_null().sum().alias("unannotated"), + (pl.col("locus") == "IGH").sum().alias("h"), + pl.col("locus").is_in(LIGHT).sum().alias("l")) + .with_columns(pl.when(pl.col("unannotated") > 0).then(pl.lit("unannotated")) + .otherwise(pl.format("{}H{}L", "h", "l")).alias("shape"))) + counts = {k: v for k, v in shapes.group_by("shape").len().iter_rows() if k != "1H1L"} + keep = shapes.filter(pl.col("shape") == "1H1L").select("barcode") + chains = ab.with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode")).join(keep, on="barcode") + heavy = chains.filter(pl.col("locus") == "IGH") + light = chains.filter(pl.col("locus").is_in(LIGHT)) + pairs = heavy.rename({c: f"{c}:0" for c in ab.columns}).join(light.rename({c: f"{c}:1" for c in ab.columns}), on="barcode") + productive = pairs.filter(pl.col("productive:0") & pl.col("productive:1")) + counts["unproductive"] = pairs.height - productive.height + dest.mkdir(parents=True, exist_ok=True) + (productive.with_columns(pl.format("{}_{}", "barcode", pl.lit(sample)).alias("pair_id"), pl.lit(sample).alias("sample")) + .drop("barcode").write_parquet(dest / f"{sample}.parquet")) + counts_path.write_text(json.dumps(counts)) + return counts + + +def easy_cluster(seqs: dict, identity: float, dest: Path) -> dict: + """Each id's representative, mmseqs2 easy-cluster at `identity` and `coverage` of both sequences.""" + tsv = dest / "clu_cluster.tsv" + if not tsv.exists(): + shutil.rmtree(dest, ignore_errors=True) + dest.mkdir(parents=True) + with open(dest / "seqs.fasta", "w") as f: + f.writelines(f">{i}\n{s}\n" for i, s in seqs.items()) + run("mmseqs", "easy-cluster", dest / "seqs.fasta", dest / "clu", dest / "tmp", "--min-seq-id", identity, + "-c", params["coverage"], "--cov-mode", 0, "--threads", cpus) + shutil.rmtree(dest / "tmp") + return {member: rep for rep, member in (line.split("\t") for line in tsv.read_text().splitlines())} + + +def cluster(pairs: pl.DataFrame, identity: float, dest: Path) -> pl.DataFrame: + """pair_id and its representative's pair_id, clustering the heavy chain's sequence_vdj_aa joined to the + light chain's.""" + seqs = {pid: h + l for pid, h, l in pairs.select("pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1").iter_rows()} + reps = easy_cluster(seqs, identity, dest) + return pl.DataFrame({"rep": list(reps.values()), "pair_id": list(reps)}) + + +def main() -> None: + download(work / "download") + extract(work / "download" / "test_data.tar.gz", work / "fasta") + samples = pl.read_csv(work / "download" / "test_data.csv", infer_schema=False).rename({"dataset": "sample"}) + samples = samples.with_columns(pl.col("donor").cast(pl.Int64)) + if params["limit"]: + samples = samples.head(params["limit"]) + dropped = {} + for sample in samples["sample"]: + fasta = work / "fasta" / f"{sample}.fasta" + for why, n in pair(sample, fasta, annotate(fasta, work / "abstar" / sample), work / "pairs").items(): + dropped[why] = dropped.get(why, 0) + n + for why, n in sorted(dropped.items(), key=lambda kv: (kv[0] == "unproductive", -kv[1], kv[0])): + print(f"dropped {why}: {n}") + pairs = pl.concat([pl.read_parquet(work / "pairs" / f"{s}.parquet") for s in samples["sample"]], how="diagonal_relaxed") + pairs = pairs.join(samples, on="sample") + print(f"pairs kept: {pairs.height}") + + c85 = cluster(pairs, params["identity"], work / "mm85").rename({"rep": "cluster_85"}) + sizes = dict(c85.group_by("cluster_85").len().iter_rows()) + reps = sorted(sizes) + random.Random(params["seed"]).shuffle(reps) + total, (f_train, f_eval, _) = pairs.height, params["split"] + assigned, n_train, n_eval = {}, 0, 0 + for rep in reps: + if n_train < f_train * total: + assigned[rep], n_train = "train", n_train + sizes[rep] + elif n_eval < f_eval * total: + assigned[rep], n_eval = "eval", n_eval + sizes[rep] + else: + assigned[rep] = "test" + split = pl.DataFrame({"cluster_85": list(assigned), "split": list(assigned.values())}) + pairs = pairs.join(c85, on="pair_id").join(split, on="cluster_85") + + front = ["pair_id", "sample", "donor", "flow_class", "cluster_85"] + chains = [c for c in pairs.columns if c.endswith((":0", ":1"))] + train = pairs.filter(pl.col("split") == "train") + c95 = cluster(train, params["train_identity"], work / "mm95").rename({"rep": "cluster_95"}) + train = train.join(c95, on="pair_id").filter(pl.col("pair_id") == pl.col("cluster_95")) + files = {"train": train.select(*front, "cluster_95", *chains)} + for name in ("eval", "test"): + files[name] = pairs.filter((pl.col("split") == name) & (pl.col("pair_id") == pl.col("cluster_85"))).select(*front, *chains) + for name, df in files.items(): + df.sort("pair_id").write_parquet(out / f"{name}.parquet") + for name, df in files.items(): + print(f"pairs {name}: {df.height}") + + +if __name__ == "__main__": + main() diff --git a/datasets/jaffe-paired/recipe/params.toml b/datasets/jaffe-paired/recipe/params.toml new file mode 100644 index 0000000..a6a8a2d --- /dev/null +++ b/datasets/jaffe-paired/recipe/params.toml @@ -0,0 +1,10 @@ +pulled = "2026-11-06" +article = 19617633 +article_version = 3 +germline_database = "human" +identity = 0.85 +coverage = 0.8 +train_identity = 0.95 +split = [0.96, 0.02, 0.02] +seed = 1 +limit = 0 # samples, for a developer's dry run; 0 is all, and the published version has 0 diff --git a/datasets/jaffe-paired/recipe/test b/datasets/jaffe-paired/recipe/test new file mode 100755 index 0000000..c6f300d --- /dev/null +++ b/datasets/jaffe-paired/recipe/test @@ -0,0 +1,64 @@ +#!/usr/bin/env -S python3 -I +"""recipe/test : exit 0 iff holds a version of jaffe-paired. Reads the parquet footers and only the +columns each check needs; one line per failure, naming the check and the first offending pair_id.""" +import sys +from pathlib import Path + +import polars as pl + +FLOORS = {"train": 1_000_000, "eval": 1_200, "test": 4_300} # rows: 90% of the first full build's, to two figures +FRONT = ["pair_id", "sample", "donor", "flow_class", "cluster_85"] +FIELDS = ["sequence_id", "locus", "productive", "v_call", "j_call", "sequence_vdj", "sequence_vdj_aa", "cdr3_aa", "fwr4_aa"] + +out, problems = Path(sys.argv[1]), [] +frames, chain_columns = {}, None +for split, floor in FLOORS.items(): + path = out / f"{split}.parquet" + if not path.is_file(): + problems.append(f"files: {split}.parquet is missing") + continue + lf = pl.scan_parquet(path) + columns = lf.collect_schema().names() + want = FRONT + (["cluster_95"] if split == "train" else []) + [f"{f}:{i}" for f in FIELDS for i in (0, 1)] + missing = [c for c in want if c not in columns] + if missing: + problems.append(f"columns: {split}.parquet lacks {missing}") + continue + chains = sorted(c for c in columns if c.endswith((":0", ":1"))) + if {c[:-2] for c in chains if c.endswith(":0")} != {c[:-2] for c in chains if c.endswith(":1")}: + problems.append(f"columns: {split}.parquet's heavy and light fields differ") + if chain_columns is not None and chains != chain_columns: + problems.append(f"columns: {split}.parquet's chain fields are not train's") + chain_columns = chain_columns or chains + rows = lf.select(pl.len()).collect().item() + if rows < floor: + problems.append(f"rows: {split}.parquet has {rows}, under its floor {floor}") + df = lf.select("pair_id", "cluster_85", *[f"{f}:{i}" for f in ("locus", "productive", "sequence_vdj_aa", "fwr4_aa") for i in (0, 1)]).collect() + frames[split] = df.select("pair_id", "cluster_85") + bad = { + "productive": ~(pl.col("productive:0") & pl.col("productive:1")), + "locus": (pl.col("locus:0") != "IGH") | ~pl.col("locus:1").is_in(["IGK", "IGL"]), + } + for i in (0, 1): + seq = pl.col(f"sequence_vdj_aa:{i}") + bad[f"sequence_vdj_aa:{i} gap or stop"] = seq.str.contains(r"[-.*]") + bad[f"sequence_vdj_aa:{i} ends with fwr4_aa:{i}"] = ~seq.str.ends_with(pl.col(f"fwr4_aa:{i}")) + for check, expr in bad.items(): + hit = df.filter(expr.fill_null(True)) + if hit.height: + problems.append(f"{check}: {hit.height} rows in {split}.parquet, first {hit['pair_id'][0]}") + print(f"{split}.parquet {rows} rows") + +for key in ("pair_id", "cluster_85"): + seen = {} + for split, df in frames.items(): + for value in df[key].unique(): + if value in seen: + problems.append(f"{key}: {value} is in {seen[value]} and {split}") + break + seen[value] = split +for split, df in frames.items(): + if df["pair_id"].n_unique() != df.height: + problems.append(f"pair_id: {split}.parquet repeats one") +print("\n".join(problems) or "ok") +sys.exit(1 if problems else 0) diff --git a/results/jaffe-paired/1.0.0.txt b/results/jaffe-paired/1.0.0.txt new file mode 100644 index 0000000..09ec0b0 --- /dev/null +++ b/results/jaffe-paired/1.0.0.txt @@ -0,0 +1,4456 @@ +--- +dataset: jaffe-paired +version: 1.0.0 +hub: brineylab/jaffe-paired +branch: candidate-ed61fc4e51988ec49d0eb46eedbd452f22a1b467 +commit: cd02c3a0cc662942b2875a8095b159561d4ae1e5 +tree: ed61fc4e51988ec49d0eb46eedbd452f22a1b467 +tools: datascience@1.2.0 +runner: sbatch --partition=hpc-low --cpus-per-task=32 --mem=200G --time=06:00:00 +host: sunk.6af0ac-us-east-13a-briney-rtx6000pro.coreweave.app +started: 2026-11-11T02:06:03Z +exit: 0 +--- +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/928e8e90/.work/abstar/1279049.tmp/logs/abstar.log + + __ __ + ____ _/ /_ _____/ /_____ ______ + / __ `/ __ \/ ___/ __/ __ `/ ___/ +/ /_/ / /_/ (__ ) /_/ /_/ / / +\__,_/_.___/____/\__/\__,_/_/ + + +RUN PARAMETERS +=============== +PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/928e8e90/.work/abstar/1279049.tmp +GERMLINE DATABASE: human +RECEPTOR: bcr +OUTPUT FORMAT: parquet +UMI PATTERN: None +UMI LENGTH: None +MERGE: False +MERGE KWARGS: {} +INTERLEAVED FASTQ: False +CHUNKSIZE: 500 +NUM PROCESSES: 32 +COPY INPUTS TO PROJECT: False +VERBOSE: True +DEBUG: False + + + +INPUT FILES +=========== +found 1 input file: + 1279049.fasta + + +----------- + 1279049 +----------- +input contained 20,534 sequences + +germline assignment: + V | D | J | C +sequence annotation: + 0%| | 0/42 [00:00