Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions datasets/antigen-specific-antibody-database/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
name: antigen-specific-antibody-database
description: "The complete NaturalAntibody Antigen Specific Antibody Database 2.0 in Parquet. Use it to query antibody–antigen observations together with their reported measurements, chains, confidence and provenance."
metadata:
version: 1.0.1
tools: datascience@1.2.0
license: cc-by-nc-4.0
---
# Antigen Specific Antibody Database

Hub target: `brineylab/antigen-specific-antibody-database`, tag `v1.0.1`. This recipe targets NaturalAntibody ASD 2.0.

## Files

`asd.parquet` contains one row per record in the upstream Delta snapshot, including repeated observations and incomplete records. NaturalAntibody reports 1,177,819 records across 29 sources. The upstream columns keep their names and nesting. Re-encoding may change row order and physical Parquet annotations such as nullability. [ASD release and schema](https://naturalantibody.com/asd/)

| Field | Interpretation |
| --- | --- |
| `dataset`, `confidence` | Upstream source abbreviation and ASD's confidence category for the observation. |
| `affinity_type`, `affinity`, `processed_measurement` | Upstream measurement label, original string, and ASD's numeric interpretation where available. The label gives the measurement scale and sometimes its units. Qualitative binding observations also occur here. |
| `chains.antibody_chains`, `chains.target_chains` | Maps of chain labels to sequences. ASD 2.0 documents antibody keys `heavy` and `light` and the target key `full`. |
| `chains.antibody_chains_numbered` | Upstream numbering and annotations keyed by antibody chain. |
| `nanobody`, `scfv` | ASD's antibody-format annotations. Interpret an absent light chain together with these flags. |
| `metadata.target_name`, `metadata.target_pdb`, `metadata.target_uniprot` | Target label and associated identifiers as supplied. |
| `metadata.source_url`, `metadata.article_url` | Source data and publication references when supplied. |

The Parquet footer records the upstream release label, source URL and SHA-256 fingerprint of the input table directory.

## Reading it

After publication, pin the lab release and select columns with PyArrow.

```python
from pathlib import Path
from huggingface_hub import snapshot_download
import pyarrow.parquet as pq

root = snapshot_download("brineylab/antigen-specific-antibody-database", repo_type="dataset", revision="v1.0.1")
observations = pq.read_table(Path(root) / "asd.parquet", columns=["dataset", "affinity_type", "affinity"])
```

## Recipe

NaturalAntibody distributes ASD 2.0 under CC BY-NC 4.0. This recipe changes the storage representation from Delta Lake to Parquet. Credit NaturalAntibody and retain the upstream source and article references. [Original release](https://naturalantibody.com/asd/) · [License](https://creativecommons.org/licenses/by-nc/4.0/)

Run `recipe/build <out>` in `datascience@1.2.0`. It downloads the public 2.0 release folder configured in `params.toml`, pulled 2026-11-17. For an already downloaded release, set `ASD_SOURCE` to the extracted directory containing the table. The recipe uses DuckDB's Delta reader to resolve the active snapshot, including removals. DuckDB fetches its matching official Delta extension on first use if needed. [DuckDB Delta extension](https://duckdb.org/docs/extensions/delta)

The build compares the logical schema, row counts by source and the entire multiset of exported rows with the source before replacing the output. The comparison includes nested fields and duplicate multiplicities. A verified download is kept under `<out>/.work/source` for reuse. The build logs the input fingerprint and UTC run time.

Run `recipe/test <out>` on the standalone output to check the release metadata, documented field paths, total record count and number of sources. Each source's row count is printed for the build record.

## Compute

One job, CPU only: 16 CPUs and 128G asked, on bcr's `cpu` partition. Measured on the first full build: about 9 min end to end, the Google Drive download and source SHA-256 fingerprint about 8 min of it, the Delta export, the full row-by-row comparison, the test and the push about 1 min. Peak memory was not measured (flyloft did not return `sacct`); 128G was enough. Disk: the verified download kept under `<out>/.work/source` (the upstream snapshot, 20 Parquet files plus the `_delta_log` and three CSVs, about 2.5 GB), the exported `asd.parquet` 264 MiB, and DuckDB's comparison spill under `<out>/.work/`.

## Gotchas

- Units and scales vary by measurement type and source. A numeric value alone is insufficient to interpret an observation.
- Nulls, empty sequences, and literal strings such as `"nan"` retain their source meanings. Study metadata can be incomplete.
- The release total includes the large BUZZ source. NaturalAntibody excludes BUZZ from several summary statistics on its website.
78 changes: 78 additions & 0 deletions datasets/antigen-specific-antibody-database/recipe/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
#!/usr/bin/env -S python3 -I
"""Import the complete logical ASD Delta table. Publish the Parquet only after exact comparison."""
import hashlib
import os
import runpy
import sys
import tempfile
import tomllib
import zipfile
from datetime import datetime, timezone
from pathlib import Path

import duckdb
import requests

recipe = Path(__file__).resolve().parent
params = tomllib.loads((recipe / "params.toml").read_text())


def table_root(directory):
logs = sorted(path for path in directory.rglob("_delta_log") if path.is_dir())
if len(logs) != 1:
raise ValueError(f"expected one Delta transaction log in {directory}, found {len(logs)}")
return logs[0].parent


def source_path(work, temp):
if "ASD_SOURCE" in os.environ:
return table_root(Path(os.environ["ASD_SOURCE"]).resolve())
cached = work / "source"
if cached.exists():
return table_root(cached)
fetched = temp / "source"
download = runpy.run_path(str(recipe / "drive.py"))["download_folder"]
with requests.Session() as session:
download(session, params["source_folder"].rsplit("/", 1)[1], fetched)
archives = list(fetched.rglob("*.zip"))
if not list(fetched.rglob("_delta_log")) and len(archives) == 1:
with zipfile.ZipFile(archives[0]) as archive:
archive.extractall(fetched / "unpacked")
return table_root(fetched)


def source_digest(root):
# Bind file contents to their relative paths, including the Delta transaction log.
snapshot = hashlib.sha256()
for path in sorted(p for p in root.rglob("*") if p.is_file()):
with path.open("rb") as stream:
digest = hashlib.file_digest(stream, "sha256").digest()
snapshot.update(path.relative_to(root).as_posix().encode() + b"\0" + digest)
return snapshot.hexdigest()


def build(out):
work = out / ".work"
work.mkdir(parents=True, exist_ok=True)
validate = runpy.run_path(str(recipe / "test"))["validate"]
with tempfile.TemporaryDirectory(dir=work) as stage, duckdb.connect() as db:
temp = Path(stage)
root = source_path(work, temp)
digest = source_digest(root)
print(f"ASD {params['upstream_version']} from {params['source_folder']}", flush=True)
print(f"Build UTC {datetime.now(timezone.utc).isoformat()}, source SHA-256 {digest}", flush=True)
db.execute("SET temp_directory = ?", [stage])
source = db.table_function("delta_scan", [str(root)])
source.create_view("upstream")
staged = temp / "asd.parquet"
db.execute("""COPY upstream TO ? (FORMAT parquet, COMPRESSION zstd,
KV_METADATA {asd_release: ?, source_url: ?, source_sha256: ?})""",
[str(staged), params["upstream_version"], params["source_folder"], digest])
validate(db, staged, params, source)
if (temp / "source").exists() and not (work / "source").exists():
(temp / "source").rename(work / "source")
os.replace(staged, out / "asd.parquet")


if __name__ == "__main__":
build(Path(sys.argv[1]).resolve())
73 changes: 73 additions & 0 deletions datasets/antigen-specific-antibody-database/recipe/drive.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
"""Fetch the publisher's public Drive folder, keeping its Delta directory layout."""
import re
from html.parser import HTMLParser
from pathlib import Path

import requests


class DrivePage(HTMLParser):
"""Read file names from Drive's embedded folder view and large-file confirmation form."""

def __init__(self, html):
super().__init__(convert_charrefs=True)
self.links = []
self.fields = {}
self.link = None
self.label = []
self.feed(html)

def handle_starttag(self, tag, attrs):
attrs = dict(attrs)
if tag == "a":
self.link = attrs.get("href")
self.label = []
if tag == "input" and attrs.get("type") == "hidden" and "name" in attrs:
self.fields[attrs["name"]] = attrs.get("value", "")

def handle_data(self, text):
if self.link:
self.label.append(text)

def handle_endtag(self, tag):
if tag == "a" and self.link:
self.links.append((self.link, "".join(self.label).strip()))
self.link = None


def download_file(session, file_id, path):
url = "https://drive.usercontent.google.com/download"
params = {"id": file_id, "export": "download", "confirm": "t"}
for _ in range(2):
with session.get(url, params=params, stream=True, timeout=120) as response:
response.raise_for_status()
if "attachment" in response.headers.get("Content-Disposition", "").lower():
with path.open("xb") as output:
for block in response.iter_content(1024 * 1024):
output.write(block)
return
fields = DrivePage(response.text).fields
if fields.get("id") != file_id or "confirm" not in fields:
break
params.update(fields)
raise ValueError(f"Drive did not serve {path.name} as a download")


def download_folder(session, folder_id, dest, ancestors=()):
if folder_id in ancestors:
raise ValueError("folder cycle in Drive release")
response = session.get("https://drive.google.com/embeddedfolderview",
params={"id": folder_id}, timeout=60)
response.raise_for_status()
dest.mkdir()
for href, name in DrivePage(response.text).links:
match = re.fullmatch(r"https://drive\.google\.com/(file/d|drive/(?:u/\d+/)?folders)/([\w-]+)(?:/view)?(?:[?#].*)?", href)
if not match:
continue
if not name or name in {".", ".."} or Path(name).name != name:
raise ValueError(f"invalid release filename: {name!r}")
kind, drive_id = match.groups()
if kind == "file/d":
download_file(session, drive_id, dest / name)
else:
download_folder(session, drive_id, dest / name, (*ancestors, folder_id))
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
upstream_version = "2.0"
source_folder = "https://drive.google.com/drive/folders/1iSF3BZdDu2jjxIz3sSz5wjJMbPsAkPDM"
records = 1177819
datasets = 29
54 changes: 54 additions & 0 deletions datasets/antigen-specific-antibody-database/recipe/test
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env -S python3 -I
"""Check the complete Parquet release. Compare to the staged Delta snapshot during a build."""
import re
import sys
import tomllib
from pathlib import Path

import duckdb


def validate(db, parquet, params, source=None):
metadata = dict(db.sql("SELECT key, value FROM parquet_kv_metadata(?)", params=[str(parquet)]).fetchall())
if (metadata.get(b"asd_release") != params["upstream_version"].encode()
or metadata.get(b"source_url") != params["source_folder"].encode()
or not re.fullmatch(b"[0-9a-f]{64}", metadata.get(b"source_sha256", b""))):
raise ValueError("missing or invalid ASD source provenance")
print("source SHA-256", metadata[b"source_sha256"].decode())
output = db.read_parquet(str(parquet), hive_partitioning=False)
output.create_view("exported")
required = {"dataset", "affinity", "affinity_type", "processed_measurement", "confidence",
"nanobody", "scfv", "metadata", "chains"}
missing = required.difference(output.columns)
if missing:
raise ValueError(f"missing ASD columns: {sorted(missing)}")
# Binding these paths verifies the documented nested fields, even when values are null.
db.sql("""SELECT chains.antibody_chains, chains.target_chains, chains.antibody_chains_numbered,
metadata.target_name, metadata.target_pdb, metadata.target_uniprot,
metadata.source_url, metadata.article_url FROM exported LIMIT 0""")
counts = db.sql("SELECT dataset, count(*) FROM exported GROUP BY dataset ORDER BY dataset").fetchall()
rows = sum(count for _, count in counts)
if rows != params["records"] or len(counts) != params["datasets"]:
raise ValueError(f"unexpected ASD release size: {rows} records in {len(counts)} sources")

if source is not None:
if source.columns != output.columns or source.types != output.types:
raise ValueError("exported field names or types differ from the Delta snapshot")
source.create_view("upstream")
source_counts = db.sql("SELECT dataset, count(*) FROM upstream GROUP BY dataset ORDER BY dataset").fetchall()
if counts != source_counts:
raise ValueError("exported record counts differ by source")
# With equal total counts, a one-way multiset difference detects any lost or changed row.
different = db.sql("""SELECT 1 FROM (FROM upstream EXCEPT ALL FROM exported) LIMIT 1""").fetchall()
if different:
raise ValueError("exported records differ from the Delta snapshot")
for dataset, count in counts:
print(f"{dataset} {count}")
print(f"{rows} records in {len(counts)} sources")


if __name__ == "__main__":
out = Path(sys.argv[1])
params = tomllib.loads(Path(__file__).with_name("params.toml").read_text())
with duckdb.connect() as db:
validate(db, out / "asd.parquet", params)
81 changes: 81 additions & 0 deletions results/antigen-specific-antibody-database/1.0.0.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
---
dataset: antigen-specific-antibody-database
version: 1.0.0
hub: brineylab/antigen-specific-antibody-database
branch: candidate-6a3a377f987368c5effae6f6adf049316d11e30a
commit: be29ffc453d6e883342d09544415a16c0c974d1f
tree: 6a3a377f987368c5effae6f6adf049316d11e30a
tools: datascience@1.2.0
runner: sbatch --partition=cpu --cpus-per-task=16 --mem=128G --time=06:00:00
host: bcr.scripps.edu
started: 2026-11-17T20:09:34Z
exit: 0
---
pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0
pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0
ASD 2.0 from https://drive.google.com/drive/folders/1iSF3BZdDu2jjxIz3sSz5wjJMbPsAkPDM
Build UTC 2026-11-17T20:17:31.852242+00:00, source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d
source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d
aae 35
aatp 93
ab-bind 283
abbd 155853
abdesign 672
absci 1208
alphaseq 198703
biomap 2725
buzz 524346
covid-19 27301
dlgo 360
flab_hie2023 146
flab_koenig2017 4275
flab_rosace2023 19
flab_shanehsazzadeh2023 448
flab_warszawski2019 2048
genbank 2989
hiv 48008
htm 385
literature 8226
met 4000
osh 30
patents 190484
rmna 10
scmc 754
seha 15
skempiv2 434
structures-antibodies 2711
structures-nanobodies 1258
1177819 records in 29 sources
source SHA-256 0ec836785dcafddc5ce39cb454d14c4c1907089a01f30778cb1ccaed49917a6d
aae 35
aatp 93
ab-bind 283
abbd 155853
abdesign 672
absci 1208
alphaseq 198703
biomap 2725
buzz 524346
covid-19 27301
dlgo 360
flab_hie2023 146
flab_koenig2017 4275
flab_rosace2023 19
flab_shanehsazzadeh2023 448
flab_warszawski2019 2048
genbank 2989
hiv 48008
htm 385
literature 8226
met 4000
osh 30
patents 190484
rmna 10
scmc 754
seha 15
skempiv2 434
structures-antibodies 2711
structures-nanobodies 1258
1177819 records in 29 sources
file asd.parquet 276804954 9bfc8f96a156b002e669d2ce477f586a72e1f94e482532e6c9df6a93527f9ae2
commit: be29ffc453d6e883342d09544415a16c0c974d1f
Loading
Loading