From 723600d5c35435f40ededf1087ddef33fe2dd9b4 Mon Sep 17 00:00:00 2001 From: Bryan Briney <821533+briney@users.noreply.github.com> Date: Mon, 9 Nov 2026 06:34:01 +0000 Subject: [PATCH 1/6] jaffe-paired: the recipe from the repository's jaffe-paired branch, commit 9550667, unchanged --- datasets/jaffe-paired/SKILL.md | 60 ++++++++ datasets/jaffe-paired/recipe/build | 187 +++++++++++++++++++++++ datasets/jaffe-paired/recipe/params.toml | 10 ++ datasets/jaffe-paired/recipe/test | 64 ++++++++ 4 files changed, 321 insertions(+) create mode 100644 datasets/jaffe-paired/SKILL.md create mode 100755 datasets/jaffe-paired/recipe/build create mode 100644 datasets/jaffe-paired/recipe/params.toml create mode 100755 datasets/jaffe-paired/recipe/test diff --git a/datasets/jaffe-paired/SKILL.md b/datasets/jaffe-paired/SKILL.md new file mode 100644 index 0000000..8a003b7 --- /dev/null +++ b/datasets/jaffe-paired/SKILL.md @@ -0,0 +1,60 @@ +--- +name: jaffe-paired +description: "Paired human antibody heavy and light chains from four donors' sorted B cells (Jaffe et al. 2022), annotated with abstar, clustered at 85% identity and split by cluster, train reduced at 95%. Reach for it to train or evaluate a paired-chain antibody model." +metadata: + version: 1.0.0 + tools: datascience@1.1.0 + license: mit +--- +# jaffe-paired + +Hub: `brineylab/jaffe-paired`, tag `v1.0.0` + +## Files + +`train.parquet`, `eval.parquet`, `test.parquet`: one row per pair, a heavy and a light chain from one droplet. Columns: + +- `pair_id`: `_` (`AAACCTGAGACACGAC-1_1279049`). +- `sample`, `donor` (1 to 4), `flow_class` (`naive`, `unswitched`, `switched`, `plasmablast`, `unswitched + naive`, `switched + naive`): the sample's row of the source's `test_data.csv`. +- `cluster_85`: the `pair_id` of the pair's 85% cluster representative. In eval and test every row is its own representative. +- `cluster_95` (train only): the `pair_id` of its 95% representative; every train row is its own. +- Every field abstar 0.9.0 writes, once per chain: the heavy chain's with `:0` appended (`v_call:0`, `cdr3_aa:0`, `sequence_vdj_aa:0`), the light chain's with `:1`. `sequence_vdj_aa` is the variable domain, FWR1 through FWR4. + +## Reading it + +```python +from huggingface_hub import snapshot_download +import polars as pl +path = snapshot_download("brineylab/jaffe-paired", repo_type="dataset", revision="v1.0.0") +train = pl.read_parquet(f"{path}/train.parquet", columns=["pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1"]) +``` + +Name the columns you need: each file holds some 340. + +## Recipe + +**Source:** Jaffe et al. 2022, figshare article 19617633 version 3 (DOI `10.6084/m9.figshare.19617633.v3`, MIT), pulled 2026-11-06: `test_data.tar.gz` (file 34846953, MD5 `4d87c4da2d407ba95180be2cb1c91044`), of which each sample's Cell Ranger `vdj_b/filtered_contig.fasta` (cell-associated contigs, productive, high-confidence, and full-length by Cell Ranger), and `test_data.csv` (file 34846938, MD5 `3ccae335a8a03b5a771dc0eae97f7af6`), the 94 samples' donors and flow classes. The recipe refuses a download whose MD5 differs. + +**Steps**, one job in `datascience@1.1.0`: + +1. Annotate every contig with abstar 0.9.0 (`germline_database = "human"`). 0.9.0 matters: it writes `sequence_vdj_aa`, the variable domain, which the clustering uses; in 0.9.0 `sequence_aa` is the translated input, as AIRR 2.0 defines it. +2. Pair within each sample: a droplet's barcode is the contig id up to its first underscore. Keep a droplet only with exactly one heavy chain (`IGH`) and one light (`IGK` or `IGL`), both productive by abstar. The build prints what it dropped, by shape (`dropped 1H2L: `) and for productivity (`dropped unproductive: `); the record carries those lines. +3. Cluster every pair with `mmseqs easy-cluster --min-seq-id 0.85 -c 0.8 --cov-mode 0` (`identity`, `coverage`; 80% of both sequences) on the heavy chain's `sequence_vdj_aa` joined to the light chain's. One `easy-cluster` run over every pair: `datascience@1.1.0`'s mmseqs (master, commit 2a3dafc) carries the fix to the prefilter overflow (MMseqs2 #1092) that kills release 18 and abutils' bundled build on about 1.1M of these sequences. +4. Split by cluster `[0.96, 0.02, 0.02]`: the representatives, sorted, are shuffled with `random.Random(seed = 1)`, and whole clusters go to train until it holds 96% of the pairs, then to eval until 2%, then to test. No 85% cluster spans two splits. Eval and test keep the 85% representatives. +5. Re-cluster every train pair at 95% (`train_identity`), the same way; train keeps the 95% representatives. + +`params.toml` also holds `limit = 0`, the number of samples for a developer's dry run; 0 is all. + +**The test** checks the three files and their columns, each split's rows against a floor, no `pair_id` or `cluster_85` in two splits, both chains productive, `locus:0` `IGH` and `locus:1` `IGK` or `IGL`, and each `sequence_vdj_aa` free of gaps and stops and ending with its chain's `fwr4_aa`. + +## Compute + +One job, CPU only: 32 CPUs and 200G asked, on coreweave-rtx6000 `hpc-low` (Xeon Platinum 8562Y+). From nothing it takes about 4 h 20 min: the download (12 GB) 6 min; abstar and the pairing, sample by sample, 2 h 10 min; the 85% clustering of all 1,389,546 pairs 7 min; the 95% clustering of train's 1.33M 1 h 50 min. Peak memory 81 GB, in the annotation. Disk: 1.5 GB of files and 19 GB of stages under `/.work/`, the tarball 12 GB of it; a rerun resumes from the last finished stage. + +## Gotchas + +- Barcodes are unique only within a sample; `pair_id` carries the sample for that reason. +- The droplet rule drops about 9% of droplets (sample 1279049: 389 with two lights, 299 with two of each, 49 with two heavies, 118 with one chain, of 9,808). +- Eval and test hold one pair per 85% cluster, so they are far fewer rows than 2% of the pairs: they are cluster-level held-out sets. +- `donor` makes a held-out-donor split possible; that is another dataset, a copy of this one with the split changed. +- With podman on a laptop, run with `--pids-limit=-1`: abstar's workers pass podman's default limit of 2048 threads. diff --git a/datasets/jaffe-paired/recipe/build b/datasets/jaffe-paired/recipe/build new file mode 100755 index 0000000..cd8e999 --- /dev/null +++ b/datasets/jaffe-paired/recipe/build @@ -0,0 +1,187 @@ +#!/usr/bin/env -S python3 -I +"""recipe/build : paired human antibodies from Jaffe et al. 2022 (figshare 19617633 v3), annotated with +abstar, one heavy and one light chain per droplet, clustered with mmseqs2 at `identity`, split by cluster into +train, eval, and test, train reduced at `train_identity`; written as /{train,eval,test}.parquet. + +Stages under /.work/, each skipped when a rerun finds it finished. Prints `dropped : ` lines, +then `pairs : ` lines last. +""" +import hashlib +import json +import os +import random +import shutil +import subprocess +import sys +import tomllib +import urllib.request +from pathlib import Path + +import polars as pl + +# figshare file id: (name, size, MD5) of article 19617633 version 3 +SOURCE = {34846953: ("test_data.tar.gz", 11941014666, "4d87c4da2d407ba95180be2cb1c91044"), + 34846938: ("test_data.csv", 1981, "3ccae335a8a03b5a771dc0eae97f7af6")} +FASTA = "test_data/*/outs/per_sample_outs/*/vdj_b/filtered_contig.fasta" +LIGHT = ("IGK", "IGL") + +params = tomllib.loads((Path(__file__).parent / "params.toml").read_text()) +out = Path(sys.argv[1]).resolve() +work = out / ".work" +cpus = len(os.sched_getaffinity(0)) # the job's CPUs, not the node's + + +def run(*cmd): + subprocess.run([str(c) for c in cmd], check=True, stdout=subprocess.DEVNULL) + + +def download(dest: Path) -> None: + dest.mkdir(parents=True, exist_ok=True) + for fid, (name, size, md5) in SOURCE.items(): + path = dest / name + if path.exists(): + continue + part, h = path.with_suffix(".part"), hashlib.md5() + req = urllib.request.Request(f"https://ndownloader.figshare.com/files/{fid}", headers={"User-Agent": "brineylab-datasets"}) + with urllib.request.urlopen(req) as r, open(part, "wb") as f: + for block in iter(lambda: r.read(1 << 22), b""): + h.update(block) + f.write(block) + if h.hexdigest() != md5 or part.stat().st_size != size: + part.unlink() + sys.exit(f"{fid}: MD5 {h.hexdigest()} is not {md5}") + part.rename(path) + + +def extract(tarball: Path, dest: Path) -> None: + """Each sample's filtered_contig.fasta as dest/.fasta.""" + if dest.exists(): + return + tmp = dest.with_suffix(".tmp") + shutil.rmtree(tmp, ignore_errors=True) + tmp.mkdir(parents=True) + run("tar", "-xzf", tarball, "-C", tmp, "--wildcards", FASTA) + for f in tmp.glob(FASTA): + f.rename(tmp / f"{f.parts[-6]}.fasta") + shutil.rmtree(tmp / "test_data") + tmp.rename(dest) + + +def annotate(fasta: Path, dest: Path) -> Path: + """abstar's parquet for one sample, named .parquet.""" + done = dest / f"{fasta.stem}.parquet" + if not done.exists(): + tmp = dest.with_name(dest.name + ".tmp") + shutil.rmtree(tmp, ignore_errors=True) + run("abstar", "run", fasta, tmp, "--germline_database", params["germline_database"], "-o", "parquet", + "--n_processes", cpus, "--mmseqs_threads", cpus, "--no-copy-inputs") + dest.mkdir(parents=True, exist_ok=True) + (tmp / "parquet" / done.name).rename(done) + shutil.rmtree(tmp) + return done + + +def pair(sample: str, fasta: Path, annotated: Path, dest: Path) -> dict: + """One row per droplet with one heavy and one light chain, both productive, as dest/.parquet; + returns the droplets dropped, by why.""" + counts_path = dest / f"{sample}.json" + if counts_path.exists(): + return json.loads(counts_path.read_text()) + ids = [line[1:].split()[0] for line in fasta.read_text().splitlines() if line.startswith(">")] + ab = pl.read_parquet(annotated) + contigs = (pl.DataFrame({"sequence_id": ids}) + .join(ab.select("sequence_id", "locus"), on="sequence_id", how="left") + .with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode"))) + shapes = (contigs.group_by("barcode") + .agg(pl.col("locus").is_null().sum().alias("unannotated"), + (pl.col("locus") == "IGH").sum().alias("h"), + pl.col("locus").is_in(LIGHT).sum().alias("l")) + .with_columns(pl.when(pl.col("unannotated") > 0).then(pl.lit("unannotated")) + .otherwise(pl.format("{}H{}L", "h", "l")).alias("shape"))) + counts = {k: v for k, v in shapes.group_by("shape").len().iter_rows() if k != "1H1L"} + keep = shapes.filter(pl.col("shape") == "1H1L").select("barcode") + chains = ab.with_columns(pl.col("sequence_id").str.split("_").list.first().alias("barcode")).join(keep, on="barcode") + heavy = chains.filter(pl.col("locus") == "IGH") + light = chains.filter(pl.col("locus").is_in(LIGHT)) + pairs = heavy.rename({c: f"{c}:0" for c in ab.columns}).join(light.rename({c: f"{c}:1" for c in ab.columns}), on="barcode") + productive = pairs.filter(pl.col("productive:0") & pl.col("productive:1")) + counts["unproductive"] = pairs.height - productive.height + dest.mkdir(parents=True, exist_ok=True) + (productive.with_columns(pl.format("{}_{}", "barcode", pl.lit(sample)).alias("pair_id"), pl.lit(sample).alias("sample")) + .drop("barcode").write_parquet(dest / f"{sample}.parquet")) + counts_path.write_text(json.dumps(counts)) + return counts + + +def easy_cluster(seqs: dict, identity: float, dest: Path) -> dict: + """Each id's representative, mmseqs2 easy-cluster at `identity` and `coverage` of both sequences.""" + tsv = dest / "clu_cluster.tsv" + if not tsv.exists(): + shutil.rmtree(dest, ignore_errors=True) + dest.mkdir(parents=True) + with open(dest / "seqs.fasta", "w") as f: + f.writelines(f">{i}\n{s}\n" for i, s in seqs.items()) + run("mmseqs", "easy-cluster", dest / "seqs.fasta", dest / "clu", dest / "tmp", "--min-seq-id", identity, + "-c", params["coverage"], "--cov-mode", 0, "--threads", cpus) + shutil.rmtree(dest / "tmp") + return {member: rep for rep, member in (line.split("\t") for line in tsv.read_text().splitlines())} + + +def cluster(pairs: pl.DataFrame, identity: float, dest: Path) -> pl.DataFrame: + """pair_id and its representative's pair_id, clustering the heavy chain's sequence_vdj_aa joined to the + light chain's.""" + seqs = {pid: h + l for pid, h, l in pairs.select("pair_id", "sequence_vdj_aa:0", "sequence_vdj_aa:1").iter_rows()} + reps = easy_cluster(seqs, identity, dest) + return pl.DataFrame({"rep": list(reps.values()), "pair_id": list(reps)}) + + +def main() -> None: + download(work / "download") + extract(work / "download" / "test_data.tar.gz", work / "fasta") + samples = pl.read_csv(work / "download" / "test_data.csv", infer_schema=False).rename({"dataset": "sample"}) + samples = samples.with_columns(pl.col("donor").cast(pl.Int64)) + if params["limit"]: + samples = samples.head(params["limit"]) + dropped = {} + for sample in samples["sample"]: + fasta = work / "fasta" / f"{sample}.fasta" + for why, n in pair(sample, fasta, annotate(fasta, work / "abstar" / sample), work / "pairs").items(): + dropped[why] = dropped.get(why, 0) + n + for why, n in sorted(dropped.items(), key=lambda kv: (kv[0] == "unproductive", -kv[1], kv[0])): + print(f"dropped {why}: {n}") + pairs = pl.concat([pl.read_parquet(work / "pairs" / f"{s}.parquet") for s in samples["sample"]], how="diagonal_relaxed") + pairs = pairs.join(samples, on="sample") + print(f"pairs kept: {pairs.height}") + + c85 = cluster(pairs, params["identity"], work / "mm85").rename({"rep": "cluster_85"}) + sizes = dict(c85.group_by("cluster_85").len().iter_rows()) + reps = sorted(sizes) + random.Random(params["seed"]).shuffle(reps) + total, (f_train, f_eval, _) = pairs.height, params["split"] + assigned, n_train, n_eval = {}, 0, 0 + for rep in reps: + if n_train < f_train * total: + assigned[rep], n_train = "train", n_train + sizes[rep] + elif n_eval < f_eval * total: + assigned[rep], n_eval = "eval", n_eval + sizes[rep] + else: + assigned[rep] = "test" + split = pl.DataFrame({"cluster_85": list(assigned), "split": list(assigned.values())}) + pairs = pairs.join(c85, on="pair_id").join(split, on="cluster_85") + + front = ["pair_id", "sample", "donor", "flow_class", "cluster_85"] + chains = [c for c in pairs.columns if c.endswith((":0", ":1"))] + train = pairs.filter(pl.col("split") == "train") + c95 = cluster(train, params["train_identity"], work / "mm95").rename({"rep": "cluster_95"}) + train = train.join(c95, on="pair_id").filter(pl.col("pair_id") == pl.col("cluster_95")) + files = {"train": train.select(*front, "cluster_95", *chains)} + for name in ("eval", "test"): + files[name] = pairs.filter((pl.col("split") == name) & (pl.col("pair_id") == pl.col("cluster_85"))).select(*front, *chains) + for name, df in files.items(): + df.sort("pair_id").write_parquet(out / f"{name}.parquet") + for name, df in files.items(): + print(f"pairs {name}: {df.height}") + + +if __name__ == "__main__": + main() diff --git a/datasets/jaffe-paired/recipe/params.toml b/datasets/jaffe-paired/recipe/params.toml new file mode 100644 index 0000000..a6a8a2d --- /dev/null +++ b/datasets/jaffe-paired/recipe/params.toml @@ -0,0 +1,10 @@ +pulled = "2026-11-06" +article = 19617633 +article_version = 3 +germline_database = "human" +identity = 0.85 +coverage = 0.8 +train_identity = 0.95 +split = [0.96, 0.02, 0.02] +seed = 1 +limit = 0 # samples, for a developer's dry run; 0 is all, and the published version has 0 diff --git a/datasets/jaffe-paired/recipe/test b/datasets/jaffe-paired/recipe/test new file mode 100755 index 0000000..c6f300d --- /dev/null +++ b/datasets/jaffe-paired/recipe/test @@ -0,0 +1,64 @@ +#!/usr/bin/env -S python3 -I +"""recipe/test : exit 0 iff holds a version of jaffe-paired. Reads the parquet footers and only the +columns each check needs; one line per failure, naming the check and the first offending pair_id.""" +import sys +from pathlib import Path + +import polars as pl + +FLOORS = {"train": 1_000_000, "eval": 1_200, "test": 4_300} # rows: 90% of the first full build's, to two figures +FRONT = ["pair_id", "sample", "donor", "flow_class", "cluster_85"] +FIELDS = ["sequence_id", "locus", "productive", "v_call", "j_call", "sequence_vdj", "sequence_vdj_aa", "cdr3_aa", "fwr4_aa"] + +out, problems = Path(sys.argv[1]), [] +frames, chain_columns = {}, None +for split, floor in FLOORS.items(): + path = out / f"{split}.parquet" + if not path.is_file(): + problems.append(f"files: {split}.parquet is missing") + continue + lf = pl.scan_parquet(path) + columns = lf.collect_schema().names() + want = FRONT + (["cluster_95"] if split == "train" else []) + [f"{f}:{i}" for f in FIELDS for i in (0, 1)] + missing = [c for c in want if c not in columns] + if missing: + problems.append(f"columns: {split}.parquet lacks {missing}") + continue + chains = sorted(c for c in columns if c.endswith((":0", ":1"))) + if {c[:-2] for c in chains if c.endswith(":0")} != {c[:-2] for c in chains if c.endswith(":1")}: + problems.append(f"columns: {split}.parquet's heavy and light fields differ") + if chain_columns is not None and chains != chain_columns: + problems.append(f"columns: {split}.parquet's chain fields are not train's") + chain_columns = chain_columns or chains + rows = lf.select(pl.len()).collect().item() + if rows < floor: + problems.append(f"rows: {split}.parquet has {rows}, under its floor {floor}") + df = lf.select("pair_id", "cluster_85", *[f"{f}:{i}" for f in ("locus", "productive", "sequence_vdj_aa", "fwr4_aa") for i in (0, 1)]).collect() + frames[split] = df.select("pair_id", "cluster_85") + bad = { + "productive": ~(pl.col("productive:0") & pl.col("productive:1")), + "locus": (pl.col("locus:0") != "IGH") | ~pl.col("locus:1").is_in(["IGK", "IGL"]), + } + for i in (0, 1): + seq = pl.col(f"sequence_vdj_aa:{i}") + bad[f"sequence_vdj_aa:{i} gap or stop"] = seq.str.contains(r"[-.*]") + bad[f"sequence_vdj_aa:{i} ends with fwr4_aa:{i}"] = ~seq.str.ends_with(pl.col(f"fwr4_aa:{i}")) + for check, expr in bad.items(): + hit = df.filter(expr.fill_null(True)) + if hit.height: + problems.append(f"{check}: {hit.height} rows in {split}.parquet, first {hit['pair_id'][0]}") + print(f"{split}.parquet {rows} rows") + +for key in ("pair_id", "cluster_85"): + seen = {} + for split, df in frames.items(): + for value in df[key].unique(): + if value in seen: + problems.append(f"{key}: {value} is in {seen[value]} and {split}") + break + seen[value] = split +for split, df in frames.items(): + if df["pair_id"].n_unique() != df.height: + problems.append(f"pair_id: {split}.parquet repeats one") +print("\n".join(problems) or "ok") +sys.exit(1 if problems else 0) From 070159b379f3b3bfcade82390a93615d6cbf3394 Mon Sep 17 00:00:00 2001 From: Bryan Briney <821533+briney@users.noreply.github.com> Date: Tue, 10 Nov 2026 13:23:46 -0800 Subject: [PATCH 2/6] jaffe-paired 1.0.0's record: its candidate passed on coreweave-rtx6000 --- results/jaffe-paired/1.0.0.txt | 4456 ++++++++++++++++++++++++++++++++ 1 file changed, 4456 insertions(+) create mode 100644 results/jaffe-paired/1.0.0.txt diff --git a/results/jaffe-paired/1.0.0.txt b/results/jaffe-paired/1.0.0.txt new file mode 100644 index 0000000..767fb94 --- /dev/null +++ b/results/jaffe-paired/1.0.0.txt @@ -0,0 +1,4456 @@ +--- +dataset: jaffe-paired +version: 1.0.0 +hub: brineylab/jaffe-paired +branch: candidate-ed61fc4e51988ec49d0eb46eedbd452f22a1b467 +commit: 985ff487ed2a8003db6da5f62bb3967f65645fcb +tree: ed61fc4e51988ec49d0eb46eedbd452f22a1b467 +tools: datascience@1.1.0 +runner: sbatch --partition=hpc-low --cpus-per-task=32 --mem=200G --time=06:00:00 +host: sunk.6af0ac-us-east-13a-briney-rtx6000pro.coreweave.app +started: 2026-11-10T17:04:40Z +exit: 0 +--- +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.1.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.1.0 +LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/723600d5/.work/abstar/1279049.tmp/logs/abstar.log + + __ __ + ____ _/ /_ _____/ /_____ ______ + / __ `/ __ \/ ___/ __/ __ `/ ___/ +/ /_/ / /_/ (__ ) /_/ /_/ / / +\__,_/_.___/____/\__/\__,_/_/ + + +RUN PARAMETERS +=============== +PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/723600d5/.work/abstar/1279049.tmp +GERMLINE DATABASE: human +RECEPTOR: bcr +OUTPUT FORMAT: parquet +UMI PATTERN: None +UMI LENGTH: None +MERGE: False +MERGE KWARGS: {} +INTERLEAVED FASTQ: False +CHUNKSIZE: 500 +NUM PROCESSES: 32 +COPY INPUTS TO PROJECT: False +VERBOSE: True +DEBUG: False + + + +INPUT FILES +=========== +found 1 input file: + 1279049.fasta + + +----------- + 1279049 +----------- +input contained 20,534 sequences + +germline assignment: + V | D | J | C +sequence annotation: + 0%| | 0/42 [00:00 Date: Tue, 10 Nov 2026 21:32:51 +0000 Subject: [PATCH 3/6] jaffe-paired: name datascience@1.2.0, the version the toolshed holds The check refused datascience@1.1.0: the shed's main carries 1.2.0. The recipe is unchanged (tree ed61fc4e), and 1.2.0 holds the same abstar 0.9.0, abutils 0.6.1, and mmseqs master 2a3dafc, so the cook is unchanged. The version stays 1.0.0, since nothing is published. --- datasets/jaffe-paired/SKILL.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/datasets/jaffe-paired/SKILL.md b/datasets/jaffe-paired/SKILL.md index 8a003b7..29d0484 100644 --- a/datasets/jaffe-paired/SKILL.md +++ b/datasets/jaffe-paired/SKILL.md @@ -3,7 +3,7 @@ name: jaffe-paired description: "Paired human antibody heavy and light chains from four donors' sorted B cells (Jaffe et al. 2022), annotated with abstar, clustered at 85% identity and split by cluster, train reduced at 95%. Reach for it to train or evaluate a paired-chain antibody model." metadata: version: 1.0.0 - tools: datascience@1.1.0 + tools: datascience@1.2.0 license: mit --- # jaffe-paired @@ -35,11 +35,11 @@ Name the columns you need: each file holds some 340. **Source:** Jaffe et al. 2022, figshare article 19617633 version 3 (DOI `10.6084/m9.figshare.19617633.v3`, MIT), pulled 2026-11-06: `test_data.tar.gz` (file 34846953, MD5 `4d87c4da2d407ba95180be2cb1c91044`), of which each sample's Cell Ranger `vdj_b/filtered_contig.fasta` (cell-associated contigs, productive, high-confidence, and full-length by Cell Ranger), and `test_data.csv` (file 34846938, MD5 `3ccae335a8a03b5a771dc0eae97f7af6`), the 94 samples' donors and flow classes. The recipe refuses a download whose MD5 differs. -**Steps**, one job in `datascience@1.1.0`: +**Steps**, one job in `datascience@1.2.0`: 1. Annotate every contig with abstar 0.9.0 (`germline_database = "human"`). 0.9.0 matters: it writes `sequence_vdj_aa`, the variable domain, which the clustering uses; in 0.9.0 `sequence_aa` is the translated input, as AIRR 2.0 defines it. 2. Pair within each sample: a droplet's barcode is the contig id up to its first underscore. Keep a droplet only with exactly one heavy chain (`IGH`) and one light (`IGK` or `IGL`), both productive by abstar. The build prints what it dropped, by shape (`dropped 1H2L: `) and for productivity (`dropped unproductive: `); the record carries those lines. -3. Cluster every pair with `mmseqs easy-cluster --min-seq-id 0.85 -c 0.8 --cov-mode 0` (`identity`, `coverage`; 80% of both sequences) on the heavy chain's `sequence_vdj_aa` joined to the light chain's. One `easy-cluster` run over every pair: `datascience@1.1.0`'s mmseqs (master, commit 2a3dafc) carries the fix to the prefilter overflow (MMseqs2 #1092) that kills release 18 and abutils' bundled build on about 1.1M of these sequences. +3. Cluster every pair with `mmseqs easy-cluster --min-seq-id 0.85 -c 0.8 --cov-mode 0` (`identity`, `coverage`; 80% of both sequences) on the heavy chain's `sequence_vdj_aa` joined to the light chain's. One `easy-cluster` run over every pair: `datascience@1.2.0`'s mmseqs (master, commit 2a3dafc) carries the fix to the prefilter overflow (MMseqs2 #1092) that kills release 18 and abutils' bundled build on about 1.1M of these sequences. 4. Split by cluster `[0.96, 0.02, 0.02]`: the representatives, sorted, are shuffled with `random.Random(seed = 1)`, and whole clusters go to train until it holds 96% of the pairs, then to eval until 2%, then to test. No 85% cluster spans two splits. Eval and test keep the 85% representatives. 5. Re-cluster every train pair at 95% (`train_identity`), the same way; train keeps the 95% representatives. From 9adef3f19ea3e6d486ed759c155608808a642858 Mon Sep 17 00:00:00 2001 From: Bryan Briney <821533+briney@users.noreply.github.com> Date: Tue, 10 Nov 2026 17:58:50 -0800 Subject: [PATCH 4/6] jaffe-paired 1.0.0's record: its candidate passed on coreweave-rtx6000 --- results/jaffe-paired/1.0.0.txt | 772 ++++++++++++++++----------------- 1 file changed, 386 insertions(+), 386 deletions(-) diff --git a/results/jaffe-paired/1.0.0.txt b/results/jaffe-paired/1.0.0.txt index 767fb94..8fdc072 100644 --- a/results/jaffe-paired/1.0.0.txt +++ b/results/jaffe-paired/1.0.0.txt @@ -3,17 +3,17 @@ dataset: jaffe-paired version: 1.0.0 hub: brineylab/jaffe-paired branch: candidate-ed61fc4e51988ec49d0eb46eedbd452f22a1b467 -commit: 985ff487ed2a8003db6da5f62bb3967f65645fcb +commit: 9580c7437d6bdd43f01e11d8c2bf7c27c02bea69 tree: ed61fc4e51988ec49d0eb46eedbd452f22a1b467 -tools: datascience@1.1.0 +tools: datascience@1.2.0 runner: sbatch --partition=hpc-low --cpus-per-task=32 --mem=200G --time=06:00:00 host: sunk.6af0ac-us-east-13a-briney-rtx6000pro.coreweave.app -started: 2026-11-10T17:04:40Z +started: 2026-11-10T21:33:39Z exit: 0 --- -pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.1.0 -pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.1.0 -LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/723600d5/.work/abstar/1279049.tmp/logs/abstar.log +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/e38bf453/.work/abstar/1279049.tmp/logs/abstar.log __ __ ____ _/ /_ _____/ /_____ ______ @@ -24,7 +24,7 @@ LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired RUN PARAMETERS =============== -PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/723600d5/.work/abstar/1279049.tmp +PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/e38bf453/.work/abstar/1279049.tmp GERMLINE DATABASE: human RECEPTOR: bcr OUTPUT FORMAT: parquet @@ -55,12 +55,12 @@ input contained 20,534 sequences germline assignment: V | D | J | C sequence annotation: - 0%| | 0/42 [00:00 Date: Wed, 11 Nov 2026 02:04:55 +0000 Subject: [PATCH 5/6] jaffe-paired: the droplet rule drops about 13% of droplets, not 9 The record's own lines: 215,163 droplets dropped by the shape rule of 1,618,287 total (13.3%), and 13,578 more for productivity. About 9% is only sample 1279049's rate (855 of 9,808 = 8.7%), now scoped to it. The recipe is unchanged (tree ed61fc4e) and the version stays 1.0.0, since nothing is published. --- datasets/jaffe-paired/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasets/jaffe-paired/SKILL.md b/datasets/jaffe-paired/SKILL.md index 29d0484..fa438a6 100644 --- a/datasets/jaffe-paired/SKILL.md +++ b/datasets/jaffe-paired/SKILL.md @@ -54,7 +54,7 @@ One job, CPU only: 32 CPUs and 200G asked, on coreweave-rtx6000 `hpc-low` (Xeon ## Gotchas - Barcodes are unique only within a sample; `pair_id` carries the sample for that reason. -- The droplet rule drops about 9% of droplets (sample 1279049: 389 with two lights, 299 with two of each, 49 with two heavies, 118 with one chain, of 9,808). +- The droplet rule drops about 13% of droplets by shape (215,163 of 1,618,287 across the 94 samples) and 13,578 more for productivity. The rate varies by sample: 1279049's is 8.7% (389 with two lights, 299 with two of each, 49 with two heavies, 118 with one chain, of 9,808). - Eval and test hold one pair per 85% cluster, so they are far fewer rows than 2% of the pairs: they are cluster-level held-out sets. - `donor` makes a held-out-donor split possible; that is another dataset, a copy of this one with the split changed. - With podman on a laptop, run with `--pids-limit=-1`: abstar's workers pass podman's default limit of 2048 threads. From 8d085f57fe9645433f0671d0e98b47938b435c25 Mon Sep 17 00:00:00 2001 From: Bryan Briney <821533+briney@users.noreply.github.com> Date: Tue, 10 Nov 2026 22:27:31 -0800 Subject: [PATCH 6/6] jaffe-paired 1.0.0's record: its candidate passed on coreweave-rtx6000 --- results/jaffe-paired/1.0.0.txt | 768 ++++++++++++++++----------------- 1 file changed, 384 insertions(+), 384 deletions(-) diff --git a/results/jaffe-paired/1.0.0.txt b/results/jaffe-paired/1.0.0.txt index 8fdc072..09ec0b0 100644 --- a/results/jaffe-paired/1.0.0.txt +++ b/results/jaffe-paired/1.0.0.txt @@ -3,17 +3,17 @@ dataset: jaffe-paired version: 1.0.0 hub: brineylab/jaffe-paired branch: candidate-ed61fc4e51988ec49d0eb46eedbd452f22a1b467 -commit: 9580c7437d6bdd43f01e11d8c2bf7c27c02bea69 +commit: cd02c3a0cc662942b2875a8095b159561d4ae1e5 tree: ed61fc4e51988ec49d0eb46eedbd452f22a1b467 tools: datascience@1.2.0 runner: sbatch --partition=hpc-low --cpus-per-task=32 --mem=200G --time=06:00:00 host: sunk.6af0ac-us-east-13a-briney-rtx6000pro.coreweave.app -started: 2026-11-10T21:33:39Z +started: 2026-11-11T02:06:03Z exit: 0 --- pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 -LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/e38bf453/.work/abstar/1279049.tmp/logs/abstar.log +LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/928e8e90/.work/abstar/1279049.tmp/logs/abstar.log __ __ ____ _/ /_ _____/ /_____ ______ @@ -24,7 +24,7 @@ LOG LOCATION: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired RUN PARAMETERS =============== -PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/e38bf453/.work/abstar/1279049.tmp +PROJECT PATH: /mnt/data/flyloft/briney/jaffe-paired-publish/publish-jaffe-paired-cd402693/datasets/jaffe-paired/928e8e90/.work/abstar/1279049.tmp GERMLINE DATABASE: human RECEPTOR: bcr OUTPUT FORMAT: parquet @@ -55,12 +55,12 @@ input contained 20,534 sequences germline assignment: V | D | J | C sequence annotation: - 0%| | 0/42 [00:00