diff --git a/datasets/antigen-specific-antibody-database/SKILL.md b/datasets/antigen-specific-antibody-database/SKILL.md new file mode 100644 index 0000000..cd77058 --- /dev/null +++ b/datasets/antigen-specific-antibody-database/SKILL.md @@ -0,0 +1,60 @@ +--- +name: antigen-specific-antibody-database +description: "The complete NaturalAntibody Antigen Specific Antibody Database 2.0 in Parquet. Use it to query antibody–antigen observations together with their reported measurements, chains, confidence and provenance." +metadata: + version: 1.0.1 + tools: datascience@1.2.0 + license: cc-by-nc-4.0 +--- +# Antigen Specific Antibody Database + +Hub target: `brineylab/antigen-specific-antibody-database`, tag `v1.0.1`. This recipe targets NaturalAntibody ASD 2.0. + +## Files + +`asd.parquet` contains one row per record in the upstream Delta snapshot, including repeated observations and incomplete records. NaturalAntibody reports 1,177,819 records across 29 sources. The upstream columns keep their names and nesting. Re-encoding may change row order and physical Parquet annotations such as nullability. [ASD release and schema](https://naturalantibody.com/asd/) + +| Field | Interpretation | +| --- | --- | +| `dataset`, `confidence` | Upstream source abbreviation and ASD's confidence category for the observation. | +| `affinity_type`, `affinity`, `processed_measurement` | Upstream measurement label, original string, and ASD's numeric interpretation where available. The label gives the measurement scale and sometimes its units. Qualitative binding observations also occur here. | +| `chains.antibody_chains`, `chains.target_chains` | Maps of chain labels to sequences. ASD 2.0 documents antibody keys `heavy` and `light` and the target key `full`. | +| `chains.antibody_chains_numbered` | Upstream numbering and annotations keyed by antibody chain. | +| `nanobody`, `scfv` | ASD's antibody-format annotations. Interpret an absent light chain together with these flags. | +| `metadata.target_name`, `metadata.target_pdb`, `metadata.target_uniprot` | Target label and associated identifiers as supplied. | +| `metadata.source_url`, `metadata.article_url` | Source data and publication references when supplied. | + +The Parquet footer records the upstream release label, source URL and SHA-256 fingerprint of the input table directory. + +## Reading it + +After publication, pin the lab release and select columns with PyArrow. + +```python +from pathlib import Path +from huggingface_hub import snapshot_download +import pyarrow.parquet as pq + +root = snapshot_download("brineylab/antigen-specific-antibody-database", repo_type="dataset", revision="v1.0.1") +observations = pq.read_table(Path(root) / "asd.parquet", columns=["dataset", "affinity_type", "affinity"]) +``` + +## Recipe + +NaturalAntibody distributes ASD 2.0 under CC BY-NC 4.0. This recipe changes the storage representation from Delta Lake to Parquet. Credit NaturalAntibody and retain the upstream source and article references. [Original release](https://naturalantibody.com/asd/) · [License](https://creativecommons.org/licenses/by-nc/4.0/) + +Run `recipe/build ` in `datascience@1.2.0`. It downloads the public 2.0 release folder configured in `params.toml`, pulled 2026-11-17. For an already downloaded release, set `ASD_SOURCE` to the extracted directory containing the table. The recipe uses DuckDB's Delta reader to resolve the active snapshot, including removals. DuckDB fetches its matching official Delta extension on first use if needed. [DuckDB Delta extension](https://duckdb.org/docs/extensions/delta) + +The build compares the logical schema, row counts by source and the entire multiset of exported rows with the source before replacing the output. The comparison includes nested fields and duplicate multiplicities. A verified download is kept under `/.work/source` for reuse. The build logs the input fingerprint and UTC run time. + +Run `recipe/test ` on the standalone output to check the release metadata, documented field paths, total record count and number of sources. Each source's row count is printed for the build record. + +## Compute + +One job, CPU only: 16 CPUs and 128G asked, on bcr's `cpu` partition. Measured on the first full build: about 9 min end to end, the Google Drive download and source SHA-256 fingerprint about 8 min of it, the Delta export, the full row-by-row comparison, the test and the push about 1 min. Peak memory was not measured (flyloft did not return `sacct`); 128G was enough. Disk: the verified download kept under `/.work/source` (the upstream snapshot, 20 Parquet files plus the `_delta_log` and three CSVs, about 2.5 GB), the exported `asd.parquet` 264 MiB, and DuckDB's comparison spill under `/.work/`. + +## Gotchas + +- Units and scales vary by measurement type and source. A numeric value alone is insufficient to interpret an observation. +- Nulls, empty sequences, and literal strings such as `"nan"` retain their source meanings. Study metadata can be incomplete. +- The release total includes the large BUZZ source. NaturalAntibody excludes BUZZ from several summary statistics on its website. diff --git a/datasets/antigen-specific-antibody-database/recipe/build b/datasets/antigen-specific-antibody-database/recipe/build new file mode 100755 index 0000000..a6142a8 --- /dev/null +++ b/datasets/antigen-specific-antibody-database/recipe/build @@ -0,0 +1,78 @@ +#!/usr/bin/env -S python3 -I +"""Import the complete logical ASD Delta table. Publish the Parquet only after exact comparison.""" +import hashlib +import os +import runpy +import sys +import tempfile +import tomllib +import zipfile +from datetime import datetime, timezone +from pathlib import Path + +import duckdb +import requests + +recipe = Path(__file__).resolve().parent +params = tomllib.loads((recipe / "params.toml").read_text()) + + +def table_root(directory): + logs = sorted(path for path in directory.rglob("_delta_log") if path.is_dir()) + if len(logs) != 1: + raise ValueError(f"expected one Delta transaction log in {directory}, found {len(logs)}") + return logs[0].parent + + +def source_path(work, temp): + if "ASD_SOURCE" in os.environ: + return table_root(Path(os.environ["ASD_SOURCE"]).resolve()) + cached = work / "source" + if cached.exists(): + return table_root(cached) + fetched = temp / "source" + download = runpy.run_path(str(recipe / "drive.py"))["download_folder"] + with requests.Session() as session: + download(session, params["source_folder"].rsplit("/", 1)[1], fetched) + archives = list(fetched.rglob("*.zip")) + if not list(fetched.rglob("_delta_log")) and len(archives) == 1: + with zipfile.ZipFile(archives[0]) as archive: + archive.extractall(fetched / "unpacked") + return table_root(fetched) + + +def source_digest(root): + # Bind file contents to their relative paths, including the Delta transaction log. + snapshot = hashlib.sha256() + for path in sorted(p for p in root.rglob("*") if p.is_file()): + with path.open("rb") as stream: + digest = hashlib.file_digest(stream, "sha256").digest() + snapshot.update(path.relative_to(root).as_posix().encode() + b"\0" + digest) + return snapshot.hexdigest() + + +def build(out): + work = out / ".work" + work.mkdir(parents=True, exist_ok=True) + validate = runpy.run_path(str(recipe / "test"))["validate"] + with tempfile.TemporaryDirectory(dir=work) as stage, duckdb.connect() as db: + temp = Path(stage) + root = source_path(work, temp) + digest = source_digest(root) + print(f"ASD {params['upstream_version']} from {params['source_folder']}", flush=True) + print(f"Build UTC {datetime.now(timezone.utc).isoformat()}, source SHA-256 {digest}", flush=True) + db.execute("SET temp_directory = ?", [stage]) + source = db.table_function("delta_scan", [str(root)]) + source.create_view("upstream") + staged = temp / "asd.parquet" + db.execute("""COPY upstream TO ? (FORMAT parquet, COMPRESSION zstd, + KV_METADATA {asd_release: ?, source_url: ?, source_sha256: ?})""", + [str(staged), params["upstream_version"], params["source_folder"], digest]) + validate(db, staged, params, source) + if (temp / "source").exists() and not (work / "source").exists(): + (temp / "source").rename(work / "source") + os.replace(staged, out / "asd.parquet") + + +if __name__ == "__main__": + build(Path(sys.argv[1]).resolve()) diff --git a/datasets/antigen-specific-antibody-database/recipe/drive.py b/datasets/antigen-specific-antibody-database/recipe/drive.py new file mode 100644 index 0000000..846d079 --- /dev/null +++ b/datasets/antigen-specific-antibody-database/recipe/drive.py @@ -0,0 +1,73 @@ +"""Fetch the publisher's public Drive folder, keeping its Delta directory layout.""" +import re +from html.parser import HTMLParser +from pathlib import Path + +import requests + + +class DrivePage(HTMLParser): + """Read file names from Drive's embedded folder view and large-file confirmation form.""" + + def __init__(self, html): + super().__init__(convert_charrefs=True) + self.links = [] + self.fields = {} + self.link = None + self.label = [] + self.feed(html) + + def handle_starttag(self, tag, attrs): + attrs = dict(attrs) + if tag == "a": + self.link = attrs.get("href") + self.label = [] + if tag == "input" and attrs.get("type") == "hidden" and "name" in attrs: + self.fields[attrs["name"]] = attrs.get("value", "") + + def handle_data(self, text): + if self.link: + self.label.append(text) + + def handle_endtag(self, tag): + if tag == "a" and self.link: + self.links.append((self.link, "".join(self.label).strip())) + self.link = None + + +def download_file(session, file_id, path): + url = "https://drive.usercontent.google.com/download" + params = {"id": file_id, "export": "download", "confirm": "t"} + for _ in range(2): + with session.get(url, params=params, stream=True, timeout=120) as response: + response.raise_for_status() + if "attachment" in response.headers.get("Content-Disposition", "").lower(): + with path.open("xb") as output: + for block in response.iter_content(1024 * 1024): + output.write(block) + return + fields = DrivePage(response.text).fields + if fields.get("id") != file_id or "confirm" not in fields: + break + params.update(fields) + raise ValueError(f"Drive did not serve {path.name} as a download") + + +def download_folder(session, folder_id, dest, ancestors=()): + if folder_id in ancestors: + raise ValueError("folder cycle in Drive release") + response = session.get("https://drive.google.com/embeddedfolderview", + params={"id": folder_id}, timeout=60) + response.raise_for_status() + dest.mkdir() + for href, name in DrivePage(response.text).links: + match = re.fullmatch(r"https://drive\.google\.com/(file/d|drive/(?:u/\d+/)?folders)/([\w-]+)(?:/view)?(?:[?#].*)?", href) + if not match: + continue + if not name or name in {".", ".."} or Path(name).name != name: + raise ValueError(f"invalid release filename: {name!r}") + kind, drive_id = match.groups() + if kind == "file/d": + download_file(session, drive_id, dest / name) + else: + download_folder(session, drive_id, dest / name, (*ancestors, folder_id)) diff --git a/datasets/antigen-specific-antibody-database/recipe/params.toml b/datasets/antigen-specific-antibody-database/recipe/params.toml new file mode 100644 index 0000000..cd27bd9 --- /dev/null +++ b/datasets/antigen-specific-antibody-database/recipe/params.toml @@ -0,0 +1,4 @@ +upstream_version = "2.0" +source_folder = "https://drive.google.com/drive/folders/1iSF3BZdDu2jjxIz3sSz5wjJMbPsAkPDM" +records = 1177819 +datasets = 29 diff --git a/datasets/antigen-specific-antibody-database/recipe/test b/datasets/antigen-specific-antibody-database/recipe/test new file mode 100755 index 0000000..b795d1a --- /dev/null +++ b/datasets/antigen-specific-antibody-database/recipe/test @@ -0,0 +1,54 @@ +#!/usr/bin/env -S python3 -I +"""Check the complete Parquet release. Compare to the staged Delta snapshot during a build.""" +import re +import sys +import tomllib +from pathlib import Path + +import duckdb + + +def validate(db, parquet, params, source=None): + metadata = dict(db.sql("SELECT key, value FROM parquet_kv_metadata(?)", params=[str(parquet)]).fetchall()) + if (metadata.get(b"asd_release") != params["upstream_version"].encode() + or metadata.get(b"source_url") != params["source_folder"].encode() + or not re.fullmatch(b"[0-9a-f]{64}", metadata.get(b"source_sha256", b""))): + raise ValueError("missing or invalid ASD source provenance") + print("source SHA-256", metadata[b"source_sha256"].decode()) + output = db.read_parquet(str(parquet), hive_partitioning=False) + output.create_view("exported") + required = {"dataset", "affinity", "affinity_type", "processed_measurement", "confidence", + "nanobody", "scfv", "metadata", "chains"} + missing = required.difference(output.columns) + if missing: + raise ValueError(f"missing ASD columns: {sorted(missing)}") + # Binding these paths verifies the documented nested fields, even when values are null. + db.sql("""SELECT chains.antibody_chains, chains.target_chains, chains.antibody_chains_numbered, + metadata.target_name, metadata.target_pdb, metadata.target_uniprot, + metadata.source_url, metadata.article_url FROM exported LIMIT 0""") + counts = db.sql("SELECT dataset, count(*) FROM exported GROUP BY dataset ORDER BY dataset").fetchall() + rows = sum(count for _, count in counts) + if rows != params["records"] or len(counts) != params["datasets"]: + raise ValueError(f"unexpected ASD release size: {rows} records in {len(counts)} sources") + + if source is not None: + if source.columns != output.columns or source.types != output.types: + raise ValueError("exported field names or types differ from the Delta snapshot") + source.create_view("upstream") + source_counts = db.sql("SELECT dataset, count(*) FROM upstream GROUP BY dataset ORDER BY dataset").fetchall() + if counts != source_counts: + raise ValueError("exported record counts differ by source") + # With equal total counts, a one-way multiset difference detects any lost or changed row. + different = db.sql("""SELECT 1 FROM (FROM upstream EXCEPT ALL FROM exported) LIMIT 1""").fetchall() + if different: + raise ValueError("exported records differ from the Delta snapshot") + for dataset, count in counts: + print(f"{dataset} {count}") + print(f"{rows} records in {len(counts)} sources") + + +if __name__ == "__main__": + out = Path(sys.argv[1]) + params = tomllib.loads(Path(__file__).with_name("params.toml").read_text()) + with duckdb.connect() as db: + validate(db, out / "asd.parquet", params) diff --git a/results/antigen-specific-antibody-database/1.0.0.txt b/results/antigen-specific-antibody-database/1.0.0.txt new file mode 100644 index 0000000..6ba3c5c --- /dev/null +++ b/results/antigen-specific-antibody-database/1.0.0.txt @@ -0,0 +1,81 @@ +--- +dataset: antigen-specific-antibody-database +version: 1.0.0 +hub: brineylab/antigen-specific-antibody-database +branch: candidate-6a3a377f987368c5effae6f6adf049316d11e30a +commit: be29ffc453d6e883342d09544415a16c0c974d1f +tree: 6a3a377f987368c5effae6f6adf049316d11e30a +tools: datascience@1.2.0 +runner: sbatch --partition=cpu --cpus-per-task=16 --mem=128G --time=06:00:00 +host: bcr.scripps.edu +started: 2026-11-17T20:09:34Z +exit: 0 +--- +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +ASD 2.0 from https://drive.google.com/drive/folders/1iSF3BZdDu2jjxIz3sSz5wjJMbPsAkPDM +Build UTC 2026-11-17T20:17:31.852242+00:00, source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +aae 35 +aatp 93 +ab-bind 283 +abbd 155853 +abdesign 672 +absci 1208 +alphaseq 198703 +biomap 2725 +buzz 524346 +covid-19 27301 +dlgo 360 +flab_hie2023 146 +flab_koenig2017 4275 +flab_rosace2023 19 +flab_shanehsazzadeh2023 448 +flab_warszawski2019 2048 +genbank 2989 +hiv 48008 +htm 385 +literature 8226 +met 4000 +osh 30 +patents 190484 +rmna 10 +scmc 754 +seha 15 +skempiv2 434 +structures-antibodies 2711 +structures-nanobodies 1258 +1177819 records in 29 sources +source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +aae 35 +aatp 93 +ab-bind 283 +abbd 155853 +abdesign 672 +absci 1208 +alphaseq 198703 +biomap 2725 +buzz 524346 +covid-19 27301 +dlgo 360 +flab_hie2023 146 +flab_koenig2017 4275 +flab_rosace2023 19 +flab_shanehsazzadeh2023 448 +flab_warszawski2019 2048 +genbank 2989 +hiv 48008 +htm 385 +literature 8226 +met 4000 +osh 30 +patents 190484 +rmna 10 +scmc 754 +seha 15 +skempiv2 434 +structures-antibodies 2711 +structures-nanobodies 1258 +1177819 records in 29 sources +file asd.parquet 276804954 9bfc8f96a156b002e669d2ce477f586a72e1f94e482532e6c9df6a93527f9ae2 +commit: be29ffc453d6e883342d09544415a16c0c974d1f diff --git a/results/antigen-specific-antibody-database/1.0.1.txt b/results/antigen-specific-antibody-database/1.0.1.txt new file mode 100644 index 0000000..b924746 --- /dev/null +++ b/results/antigen-specific-antibody-database/1.0.1.txt @@ -0,0 +1,81 @@ +--- +dataset: antigen-specific-antibody-database +version: 1.0.1 +hub: brineylab/antigen-specific-antibody-database +branch: candidate-6a3a377f987368c5effae6f6adf049316d11e30a +commit: c089b95d798e063aa5b17477dacd6c3bfbfd666c +tree: 6a3a377f987368c5effae6f6adf049316d11e30a +tools: datascience@1.2.0 +runner: sbatch --partition=cpu --cpus-per-task=16 --mem=128G --time=06:00:00 +host: bcr.scripps.edu +started: 2026-11-17T20:25:54Z +exit: 0 +--- +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +ASD 2.0 from https://drive.google.com/drive/folders/1iSF3BZdDu2jjxIz3sSz5wjJMbPsAkPDM +Build UTC 2026-11-17T20:32:25.436062+00:00, source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +aae 35 +aatp 93 +ab-bind 283 +abbd 155853 +abdesign 672 +absci 1208 +alphaseq 198703 +biomap 2725 +buzz 524346 +covid-19 27301 +dlgo 360 +flab_hie2023 146 +flab_koenig2017 4275 +flab_rosace2023 19 +flab_shanehsazzadeh2023 448 +flab_warszawski2019 2048 +genbank 2989 +hiv 48008 +htm 385 +literature 8226 +met 4000 +osh 30 +patents 190484 +rmna 10 +scmc 754 +seha 15 +skempiv2 434 +structures-antibodies 2711 +structures-nanobodies 1258 +1177819 records in 29 sources +source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d +aae 35 +aatp 93 +ab-bind 283 +abbd 155853 +abdesign 672 +absci 1208 +alphaseq 198703 +biomap 2725 +buzz 524346 +covid-19 27301 +dlgo 360 +flab_hie2023 146 +flab_koenig2017 4275 +flab_rosace2023 19 +flab_shanehsazzadeh2023 448 +flab_warszawski2019 2048 +genbank 2989 +hiv 48008 +htm 385 +literature 8226 +met 4000 +osh 30 +patents 190484 +rmna 10 +scmc 754 +seha 15 +skempiv2 434 +structures-antibodies 2711 +structures-nanobodies 1258 +1177819 records in 29 sources +file asd.parquet 276804954 9bfc8f96a156b002e669d2ce477f586a72e1f94e482532e6c9df6a93527f9ae2 +commit: c089b95d798e063aa5b17477dacd6c3bfbfd666c