diff --git a/datasets/afdb-swissprot/SKILL.md b/datasets/afdb-swissprot/SKILL.md new file mode 100644 index 0000000..4e99bf6 --- /dev/null +++ b/datasets/afdb-swissprot/SKILL.md @@ -0,0 +1,77 @@ +--- +name: afdb-swissprot +description: "The AlphaFold Protein Structure Database's v4 predictions of every Swiss-Prot entry as sharded parquet: one row per model with its id, UniProt accession, sequence, length, mean pLDDT, and the full mmCIF text, sorted by id. Reach for it to pull AlphaFold structures, sequences or pLDDTs for Swiss-Prot proteins in bulk, without per-entry downloads from EBI." +metadata: + version: 1.0.0 + tools: datascience@1.2.0 + license: cc-by-4.0 +--- +# afdb-swissprot + +Hub: `brineylab/afdb-swissprot`, tag `v1.0.0` + +## Files + +`data/00000.parquet` … `data/00036.parquet`: 37 shards holding 542,378 rows, one row per AlphaFold DB v4 model of a Swiss-Prot entry (one model per entry, `F1`), sorted by `id` across all shards, so a point lookup reads one row group. Every shard but the last holds exactly 15,000 rows (the last 2,378); zstd, row groups of 2,000 rows. Columns: + +- `id`: the model's id, the mmCIF's `_entry.id` (`AF-P42345-F1`; a few carry an isoform suffix, `AF-P12345-1-F1`). +- `uniprot_accession`: the entry the model is of: `id` without the `AF-` and the `-F1`. +- `sequence`: the modeled sequence, one-letter codes, `_entity_poly.pdbx_seq_one_letter_code`. +- `length`: `len(sequence)`, int64. +- `mean_plddt`: the model's global pLDDT, `_ma_qa_metric_global.metric_value`, 0–100. +- `cif`: the model's full mmCIF text, decompressed, byte-for-byte as the archive holds it (large_string). + +## Reading it + +```python +from huggingface_hub import snapshot_download +import polars as pl +path = snapshot_download("brineylab/afdb-swissprot", repo_type="dataset", revision="v1.0.0") +shard = pl.read_parquet(f"{path}/data/00000.parquet", columns=["id", "sequence", "mean_plddt"]) +``` + +Selected rows back as `.cif` files (the filters prune row groups on the id and accession statistics): + +```python +from pathlib import Path +import pyarrow.parquet as pq +want = ["P42345", "Q9Y6K9"] # UniProt accessions +for shard in sorted(Path(path, "data").glob("*.parquet")): + picked = pq.read_table(shard, columns=["id", "uniprot_accession", "cif"], + filters=[("uniprot_accession", "in", want)]) + for row in picked.to_pylist(): + Path(f"{row['id']}.cif").write_text(row["cif"]) +``` + +One row's cif parsed in memory: + +```python +import io +import pyarrow.parquet as pq +from Bio.PDB import MMCIFParser +one = pq.read_table(f"{path}/data/00000.parquet", columns=["id", "cif"], + filters=[("id", "=", "AF-A0A060D764-F1")]).to_pylist()[0] +structure = MMCIFParser().get_structure(one["id"], io.StringIO(one["cif"])) +``` + +## Recipe + +**Source:** EBI's AlphaFold Protein Structure Database v4 archive of the Swiss-Prot proteome, CC BY 4.0 (cite Jumper et al., Nature 2021, and Varadi et al., NAR 2024), pulled 2026-11-17: `swissprot_cif_v4.tar` (39,471,090,688 bytes, last modified 2025-08-24) at https://ftp.ebi.ac.uk/pub/databases/alphafold/v4/swissprot_cif_v4.tar, holding 542,378 entries, one `AF--F1-model_v4.cif.gz` each. EBI publishes no checksum for it, so the recipe pins the byte size and verifies every member by decompressing it. + +One job in `datascience@1.2.0`. The tar is downloaded into `/.work/` by `download_streams` parallel HTTP range streams that join into one file, each resuming from where its slice stopped; its member names are scanned; then it is streamed again, never unpacking files to disk, and each member decompressed, checked (`_entry.id` matches the name, pLDDT in [0, 100], sequence one-letter, per-residue pLDDT count equals the sequence length) and staged: in rounds of `flush_members` members, rows go to per-shard chunk files under `/.work/buckets/`, and `/data/00000.parquet`, `00001.parquet`, … are assembled from the chunks at the end, rows sorted by id across all shards, `shard_rows` rows per shard (the last partial), zstd, row groups of `row_group_rows`. Every stage resumes: a rerun keeps the verified tar, the chunks of every finished round, and the shards already written. With `limit > 0` only the first `dry_bytes` of the tar are downloaded and only the first `limit` members are read: a developer's dry run. + +`params.toml`: the tar's `url`, `bytes`, and `entries`; `shard_rows = 15000`; `row_group_rows = 2000`; `flush_members = 2000`; `download_streams = 16`; `dry_bytes = 536870912` (a dry run's prefix download); `limit = 0`. + +**The test** reads the shards' footers and light columns, then 16 evenly spaced rows whole. From the footers and columns: the shards are `00000.parquet` onward without a gap; the schema is exactly the six columns above; every shard but the last holds exactly 15,000 rows; the total is the archive's 542,378 and never under 90% of it; every id matches `AF-[-]-F` and its `uniprot_accession`; the ids are sorted and unique across all shards; every `length` is `len(sequence)`; every `mean_plddt` is in [0, 100]. On the sampled rows: the `cif` parses with Biopython's `MMCIFParser`; its `_entry.id` is the id; its residues spell the `sequence` and number `length`; its global metric is `mean_plddt`; its atoms' B-factors (the per-atom pLDDT) mean to `mean_plddt` within 0.01; and its per-residue pLDDTs' plain mean is `mean_plddt` within 5.0. + +## Compute + +One job, CPU only: 16 CPUs and 64G asked — the 64G because a shard whose chunks arrive out of id order is sorted whole in memory (a shard's 15,000 CIFs are a few tens of GB; the tar's members are in id order, so most shards never take that path). A single connection to ftp.ebi.ac.uk moves about 0.9 MB/s from this cluster (measured, 2026-11-17), so the recipe downloads the 39.5 GB tar as 16 parallel HTTP range streams, each resumable; the dry run's first 0.5 GB moved at 7.5 MB/s (14 MB/s once warm), so the download is about an hour. The names scan of 542,378 member headers, minutes; the rows stage (decompress, parse, stage, re-compress) at the rate the build prints per round (277 members/s in the dry run, about 35 minutes for all); assembling the 37 shards reads and re-compresses every chunk once. Peak memory a few GB: one round of 2,000 members is staged at a time. Disk: about 40 GB of shards, plus under `/.work/` the 39.5 GB tar and, until assembly deletes them, about 40 GB of chunk files — a peak near 120 GB — kept so a rerun resumes (a shard's chunks are deleted once it is final). The test reads footers, light columns and 16 row groups, in about a minute. + +## Gotchas + +- PAE (predicted aligned error) matrices are out of scope: the mmCIFs do not carry them. For a v4 PAE, fetch `AF--F1-predicted_aligned_error_v4.json` from the GCS bucket `public-datasets-deepmind-alphafold-v4` (`gsutil cp` or `gcloud storage cp`); the bucket lists anonymously, but reading an object needs an authenticated account as of 2026-11-17. EBI's per-entry pages serve only the current release's PAE, and the v4 FTP directory holds no PAE archive for Swiss-Prot. +- The 39.5 GB tar is never uploaded to the Hub: the `cif` column already carries every byte of every member, and `scripts/push` leaves `/.work/` out. +- `mean_plddt` is the CIF's global metric, which in v4 is the mean of the per-residue pLDDTs **weighted by each residue's atom count** — the atoms' B-factor mean — not their plain mean: over the dry run's first 3,000 entries the global matched the atom-weighted mean to 0.005 (the metric's 2-decimal rounding) while the plain mean sat 0.21 off at the median and 2.49 at the max. The test checks the atom-weighted mean to 0.01 and the plain mean to 5.0. Per-residue pLDDT is in each row's `cif`: the atoms' B-factors, or the `_ma_qa_metric_local` loop. (AFDB's later releases, v6 on, report the plain mean as the global, so v4's numbers sit slightly off the current website's.) +- This is the v4-era snapshot, last modified 2025-08-24: Swiss-Prot has grown since (576,073 entries at 2026-11), and entries added after v4 exist only in AFDB's newer releases, not here. +- The URL serves whatever v4 tar EBI currently hosts: if EBI ever replaces it, the pinned byte size refuses the download, and a rebuild is a new version. diff --git a/datasets/afdb-swissprot/recipe/build b/datasets/afdb-swissprot/recipe/build new file mode 100755 index 0000000..aaf807a --- /dev/null +++ b/datasets/afdb-swissprot/recipe/build @@ -0,0 +1,366 @@ +#!/usr/bin/env -S python3 -I +"""recipe/build : AFDB v4's Swiss-Prot archive as sorted, sharded parquet. Downloads EBI's +swissprot_cif_v4.tar into /.work/ -- download_streams HTTP range streams at once, each +resumable from where its slice stopped, joining into one file -- and refuses a size that differs +from params (EBI publishes no checksum, so every member is verified by decompressing it). It +then scans the tar's member names and streams it again, never unpacking files to disk, extracting +per entry the id, accession, sequence, length, global pLDDT and full CIF text, and writing +/data/00000.parquet, 00001.parquet, ...: rows sorted by id across all shards, shard_rows rows +per shard (the last partial), zstd, row groups of row_group_rows. Rows stage in rounds of +flush_members members: each round's rows go to per-shard chunks under /.work/buckets/, and +the shards are assembled from the chunks at the end, so a rerun resumes: it keeps the verified +tar, the chunks of every finished round, and the shards already written. With limit > 0 only the +first dry_bytes of the tar are downloaded and only the first limit members are read: a +developer's dry run. Prints the download rate, the rows stage's rate, and the shard and row +totals last. +""" +import gzip +import os +import re +import sys +import tarfile +import threading +import time +import tomllib +import urllib.request +from bisect import bisect_left +from pathlib import Path + +import pyarrow as pa +import pyarrow.parquet as pq + +params = tomllib.loads((Path(__file__).parent / "params.toml").read_text()) +out = Path(sys.argv[1]).resolve() if len(sys.argv) == 2 else None +work, data = (out / ".work", out / "data") if out else (None, None) +SCHEMA = pa.schema([ + ("id", pa.string()), + ("uniprot_accession", pa.string()), + ("sequence", pa.string()), + ("length", pa.int64()), + ("mean_plddt", pa.float64()), + ("cif", pa.large_string()), +]) +SUFFIX = "-model_v4.cif.gz" +NAME = re.compile(r"^AF-[0-9A-Za-z]+(?:-[0-9]+)?-F\d+-model_v4\.cif\.gz$") +LETTERS = re.compile(r"^[A-Z]+$") +ROUND = re.compile(r"^r(\d+)\.parquet$") +LOCAL = "_ma_qa_metric_local." + + +def die(msg: str) -> None: + sys.exit(f"afdb-swissprot: {msg}") + + +def cif_value(txt: str, tag: str) -> str: + """A non-loop CIF tag's value: the token on its line, or the ;-block on the lines after it.""" + m = re.search(r"(?m)^" + re.escape(tag) + r"[ \t]*(.*)$", txt) + if m is None: + die(f"no {tag}") + rest = m.group(1).strip() + if rest: + if rest[0] in "'\"": + return rest[1:rest.find(rest[0], 1)] + return rest.split()[0] + after = txt[m.end():] + if not after.startswith("\n;"): + die(f"{tag}: neither a value on its line nor a ;-block after it") + end = after.find("\n;", 1) + if end < 0: + die(f"{tag}: an unterminated ;-block") + return "".join(after[2:end].split()) + + +def residue_plddts(txt: str, name: str) -> tuple[float, int]: + """(the mean of the _ma_qa_metric_local loop's metric_value column, its row count).""" + i = txt.find("\n" + LOCAL) + if i < 0: + die(f"{name}: no _ma_qa_metric_local loop") + end = txt.find("\n#", i) + if end < 0: + die(f"{name}: the _ma_qa_metric_local loop does not end") + tags, value_at, count, total = [], None, 0, 0.0 + for line in txt[i:end].splitlines(): + if line.startswith(LOCAL): + tags.append(line.strip()) + elif line.strip(): # the segment's first line is the newline find() matched: skip blanks + if value_at is None: + try: + value_at = tags.index(LOCAL + "metric_value") + except ValueError: + die(f"{name}: the local loop has no metric_value column") + parts = line.split() + if len(parts) == len(tags): + total += float(parts[value_at]) + count += 1 + if count == 0: + die(f"{name}: the local loop holds no values") + return total / count, count + + +def parse(name: str, gz: bytes) -> tuple[tuple, float, int]: + """((id, accession, sequence, length, mean_plddt, cif), the residue pLDDTs' mean, their count).""" + mid = name[: -len(SUFFIX)] + txt = gzip.decompress(gz).decode("utf-8") + if cif_value(txt, "_entry.id") != mid: + die(f"{name}: _entry.id is not {mid}") + accession = mid[3:].rsplit("-F", 1)[0] + plddt = float(cif_value(txt, "_ma_qa_metric_global.metric_value")) + if not 0.0 <= plddt <= 100.0: + die(f"{name}: mean pLDDT {plddt} is out of [0, 100]") + sequence = cif_value(txt, "_entity_poly.pdbx_seq_one_letter_code") + if not LETTERS.fullmatch(sequence): + die(f"{name}: the sequence is not one-letter codes") + mean, count = residue_plddts(txt, name) + if count != len(sequence): + die(f"{name}: {count} per-residue pLDDTs for a sequence of {len(sequence)}") + return (mid, accession, sequence, len(sequence), plddt, txt), mean, count + + +def head() -> None: + """Refuse an archive params does not pin, before anything is downloaded.""" + request = urllib.request.Request(params["url"], method="HEAD", headers={"User-Agent": "brineylab-datasets"}) + with urllib.request.urlopen(request, timeout=120) as r: + size, modified = int(r.headers["Content-Length"]), r.headers.get("Last-Modified", "?") + if size != params["bytes"]: + die(f"{params['url']} is {size} bytes, not the {params['bytes']} params.toml pins") + print(f"source: {params['url']} ({size} bytes, last modified {modified})") + + +def download() -> Path: + """The tar in .work/, fetched by download_streams parallel HTTP range streams that join into + one file, each resuming from where its slice stopped and given six attempts at a connection; + refuses a size that differs from params. With limit > 0 only the first dry_bytes are fetched.""" + name = params["url"].rsplit("/", 1)[-1] + path, part = work / name, work / f"{name}.part" + if path.is_file(): + for p in work.glob(f"{name}.part.*"): + p.unlink(missing_ok=True) + return path + total = params["dry_bytes"] if params["limit"] else params["bytes"] + streams = max(1, min(params["download_streams"], -(-total // (1 << 20)))) + chunk, attempts = -(-total // streams), 6 + fd = os.open(part, os.O_RDWR | os.O_CREAT) + os.ftruncate(fd, total) + + def pwrite_all(buf: bytes, offset: int) -> None: + view, at = memoryview(buf), offset + while len(view): + n = os.pwrite(fd, view, at) + at += n + view = view[n:] + + errors, lock = [], threading.Lock() + state = {"start": 0, "done": 0, "told": 0.0} + t0 = time.time() + + def fetch(k: int) -> None: + begin, end = k * chunk, min((k + 1) * chunk, total) + pos_file = work / f"{name}.part.{k:04d}.pos" + pos = int(pos_file.read_text()) if pos_file.is_file() else begin # absolute, as written + with lock: + state["start"] += pos - begin + for attempt in range(attempts): + if pos >= end: + break + headers = {"User-Agent": "brineylab-datasets", "Range": f"bytes={pos}-{end - 1}"} + try: + request = urllib.request.Request(params["url"], headers=headers) + with urllib.request.urlopen(request, timeout=120) as r: + if r.status != 206: + raise OSError(f"the server answered {r.status} to a range request") + while pos < end: + block = r.read(min(1 << 22, end - pos)) + if not block: + raise OSError("the connection ended inside the range") + pwrite_all(block, pos) + pos += len(block) + tmp = Path(f"{pos_file}.tmp") + tmp.write_text(f"{pos}\n") + os.replace(tmp, pos_file) + with lock: + state["done"] += len(block) + now = time.time() + if now - state["told"] >= 30 and state["done"] < total: + state["told"] = now + print(f"download: {state['done'] / 1e9:.1f} of {total / 1e9:.1f} GB " + f"({(state['done'] - state['start']) / 1e6 / max(now - t0, 1):.1f} MB/s)", + flush=True) + break + except OSError as e: + if attempt == attempts - 1: + errors.append(f"slice {k} ({begin}-{end}): {e} at {pos}") + return + time.sleep(30) + if pos != end: + errors.append(f"slice {k} ({begin}-{end}): stopped at {pos}, not {end}") + + threads = [threading.Thread(target=fetch, args=(k,)) for k in range(streams)] + for t in threads: + t.start() + for t in threads: + t.join() + os.ftruncate(fd, total) + os.fsync(fd) + os.close(fd) + if errors: + die(f"{name}: {'; '.join(errors[:3])}") + if part.stat().st_size != total: + die(f"{name}: {part.stat().st_size} bytes is not the {total} expected") + os.replace(part, path) + for p in work.glob(f"{name}.part.*"): + p.unlink() + took = time.time() - t0 + print(f"downloaded: {total / 1e9:.1f} GB in {took:.0f}s ({streams} streams, " + f"{(total - state['start']) / 1e6 / max(took, 1):.1f} MB/s of new bytes)") + return path + + +def tar_members(path: Path, done: int, wanted: int): + """(index, name, the member's gz bytes) from the tar on disk, from member done to member + wanted, without decompressing what it skips: a resumed run does not redo them.""" + with tarfile.open(path, "r:") as tf: + for i, m in enumerate(tf): + if i >= wanted: + return + if i < done: + continue + yield i, m.name, tf.extractfile(m).read() + + +def names() -> list[str]: + """The entries' ids, in the tar's order, from the member names alone.""" + ids = [] + with tarfile.open(download(), "r:") as tf: + for i, m in enumerate(tf): + if params["limit"] and i >= params["limit"]: + break + if not NAME.fullmatch(m.name): + die(f"{m.name}: not an AF--F{SUFFIX} member") + ids.append(m.name[: -len(SUFFIX)]) + wanted = params["limit"] or params["entries"] + if len(ids) != wanted: + die(f"the tar holds {len(ids)} entries, not the {wanted} expected") + if len(set(ids)) != len(ids): + seen, dupes = set(), set() + for i in ids: + (dupes if i in seen else seen).add(i) + die(f"duplicate ids in the tar: {', '.join(sorted(dupes)[:5])}") + return ids + + +def write_progress(n: int) -> None: + tmp = work / "progress.tmp" + tmp.write_text(f"{n}\n") + os.replace(tmp, work / "progress.txt") + + +def write_chunks(buffers: dict[int, list[tuple]], round_idx: int) -> None: + """One chunk file per shard that holds rows, whole or not at all: tmp then rename.""" + for shard, rows in sorted(buffers.items()): + d = work / "buckets" / f"{shard:05d}" + d.mkdir(parents=True, exist_ok=True) + tmp, final = d / f"r{round_idx:06d}.tmp", d / f"r{round_idx:06d}.parquet" + pq.write_table(pa.table({ + "id": [r[0] for r in rows], + "uniprot_accession": [r[1] for r in rows], + "sequence": [r[2] for r in rows], + "length": [r[3] for r in rows], + "mean_plddt": [r[4] for r in rows], + "cif": [r[5] for r in rows], + }, schema=SCHEMA), tmp, compression="zstd") + os.replace(tmp, final) + + +def rows_stage(sorted_ids: list[str]) -> tuple[float, int]: + """The tar's members into per-shard chunks, in rounds of flush_members members, resuming from + the members progress.txt says are durable. Returns the pLDDT agreement worst case and count.""" + flush, shard_rows = params["flush_members"], params["shard_rows"] + wanted = params["limit"] or len(sorted_ids) + if params["limit"]: # a dry run does not resume: start every stage from nothing + for p in work.glob("buckets/*/*"): + p.unlink() + done = 0 + else: + done = int((work / "progress.txt").read_text()) if (work / "progress.txt").is_file() else 0 + for p in work.glob("buckets/*/*"): # the chunks of rounds progress does not cover + m = ROUND.fullmatch(p.name) + if p.name.endswith(".tmp") or (m and int(m.group(1)) * flush >= done): + p.unlink() + buffers: dict[int, list[tuple]] = {} + worst, over, seen, t0 = 0.0, 0, done, time.time() + for i, name, gz in tar_members(download(), done, wanted): + row, residue_mean, _ = parse(name, gz) + gap = abs(row[4] - residue_mean) + worst, over = max(worst, gap), over + (gap > 5.0) + shard = bisect_left(sorted_ids, row[0]) // shard_rows + buffers.setdefault(shard, []).append(row) + seen = i + 1 + if (i + 1) % flush == 0 or i + 1 == wanted: + write_chunks(buffers, i // flush) + write_progress(i + 1) + buffers.clear() + print(f"members: {i + 1}/{wanted} ({(i + 1 - done) / max(time.time() - t0, 1):.0f}/s)", flush=True) + if seen != wanted: # pragma: no cover - the tar holds what names() counted + die(f"read {seen} members, not the {wanted} the names stage counted") + return worst, over + + +def assemble(sorted_ids: list[str]) -> None: + """data/NNNNN.parquet from each shard's chunks: rows in id order, row groups of row_group_rows; + a shard that arrived out of order is sorted. Chunks are deleted once their shard is final.""" + shard_rows, rg = params["shard_rows"], params["row_group_rows"] + nshards = -(-len(sorted_ids) // shard_rows) + data.mkdir(parents=True, exist_ok=True) + for shard in range(nshards): + final = data / f"{shard:05d}.parquet" + if final.is_file(): + continue + bdir = work / "buckets" / f"{shard:05d}" + chunks = sorted(bdir.glob("r*.parquet")) if bdir.is_dir() else [] + expected = min(shard_rows, len(sorted_ids) - shard * shard_rows) + if not chunks: + die(f"shard {shard:05d} holds no chunks") + ids = [i for p in chunks for i in pq.read_table(p, columns=["id"]).column("id").to_pylist()] + if len(ids) != expected: + die(f"shard {shard:05d} holds {len(ids)} rows, not {expected}") + part = data / f".{shard:05d}.part" + with pq.ParquetWriter(part, SCHEMA, compression="zstd") as w: + if ids == sorted(ids): + for p in chunks: # one chunk in memory at a time + w.write_table(pq.read_table(p), row_group_size=rg) + else: + print(f"shard {shard:05d} arrived out of id order; sorting", file=sys.stderr) + whole = pa.concat_tables([pq.read_table(p) for p in chunks]).sort_by("id") + w.write_table(whole, row_group_size=rg) + os.replace(part, final) + for p in chunks: + p.unlink() + + +def main() -> None: + if len(sys.argv) != 2: + die(f"usage: {Path(sys.argv[0]).name} ") + t0 = time.time() + out.mkdir(parents=True, exist_ok=True) + work.mkdir(parents=True, exist_ok=True) + if not params["limit"]: + head() + sorted_ids = sorted(names()) + print(f"entries: {len(sorted_ids)} ids, sorted and unique") + worst, over = rows_stage(sorted_ids) + print(f"rows: {worst:.3f} max |mean_plddt - residue mean|, {over} entries over 5.0") + assemble(sorted_ids) + files = sorted(data.glob("*.parquet")) + if [p.name for p in files] != [f"{i:05d}.parquet" for i in range(len(files))]: + die("data/'s shards are not 00000.parquet onward without a gap") + total = sum(pq.ParquetFile(p).metadata.num_rows for p in files) + if total != len(sorted_ids): + die(f"data/ holds {total} rows, not {len(sorted_ids)}") + print(f"shards: {len(files)}") + print(f"rows: {total}") + print(f"bytes: {sum(p.stat().st_size for p in files)}") + print(f"took: {time.time() - t0:.0f}s") + + +if __name__ == "__main__": + main() diff --git a/datasets/afdb-swissprot/recipe/params.toml b/datasets/afdb-swissprot/recipe/params.toml new file mode 100644 index 0000000..8459972 --- /dev/null +++ b/datasets/afdb-swissprot/recipe/params.toml @@ -0,0 +1,9 @@ +url = "https://ftp.ebi.ac.uk/pub/databases/alphafold/v4/swissprot_cif_v4.tar" +bytes = 39471090688 +entries = 542378 +shard_rows = 15000 +row_group_rows = 2000 +flush_members = 2000 +download_streams = 16 +dry_bytes = 536870912 +limit = 0 diff --git a/datasets/afdb-swissprot/recipe/test b/datasets/afdb-swissprot/recipe/test new file mode 100755 index 0000000..6dff0be --- /dev/null +++ b/datasets/afdb-swissprot/recipe/test @@ -0,0 +1,177 @@ +#!/usr/bin/env -S python3 -I +"""recipe/test : exit 0 iff holds a version of afdb-swissprot. From the shards' parquet +footers and light columns, without reading the CIFs whole: the shards are numbered without a gap, +the schema is exactly id, uniprot_accession, sequence, length, mean_plddt and cif (string, string, +string, int64, float64, large_string), every shard but the last is exactly shard_rows rows, the +total is the archive's entries and never under 90% of them, every id is AF-[-]-F +matching its uniprot_accession, the ids are sorted and unique across all shards, every length is +len(sequence), and every mean_plddt is in [0, 100]. 16 evenly spaced rows are then read whole and +each one's cif checked against its row: it parses with Biopython's MMCIFParser, its _entry.id is +the id, its residues spell the sequence and number length, its _ma_qa_metric_global value is +mean_plddt, its atoms' B-factors (the per-atom pLDDT) mean to mean_plddt within 0.01 -- v4's +global metric is that atom-weighted mean -- and its per-residue pLDDTs' plain mean is mean_plddt +within 5.0 (the atom weighting sits it a point or two away). One line per failure, naming the check and the +first offending row.""" +import io +import re +import sys +import tomllib +from bisect import bisect_right +from pathlib import Path + +import pyarrow as pa +import pyarrow.compute as pc +import pyarrow.parquet as pq +from Bio.PDB import MMCIFParser + +params = tomllib.loads((Path(__file__).parent / "params.toml").read_text()) +ID = re.compile(r"^AF-[0-9A-Za-z]+(?:-[0-9]+)?-F\d+$") +LOCAL = "_ma_qa_metric_local." +WANT = [("id", pa.string()), ("uniprot_accession", pa.string()), ("sequence", pa.string()), + ("length", pa.int64()), ("mean_plddt", pa.float64()), ("cif", pa.large_string())] +THREE_TO_ONE = { + "ALA": "A", "ARG": "R", "ASN": "N", "ASP": "D", "CYS": "C", "GLN": "Q", "GLU": "E", + "GLY": "G", "HIS": "H", "ILE": "I", "LEU": "L", "LYS": "K", "MET": "M", "PHE": "F", + "PRO": "P", "SER": "S", "THR": "T", "TRP": "W", "TYR": "Y", "VAL": "V", + "SEC": "U", "PYL": "O", "MSE": "M", "UNK": "X", +} + + +def local_plddts(cif: str) -> tuple[float | None, int]: + """(the mean of the cif's per-residue pLDDTs, their count), from its _ma_qa_metric_local loop.""" + i = cif.find("\n" + LOCAL) + if i < 0: + return None, 0 + end = cif.find("\n#", i) + if end < 0: + return None, 0 + tags, at, total, count = [], None, 0.0, 0 + for line in cif[i:end].splitlines(): + if line.startswith(LOCAL): + tags.append(line.strip()) + elif line.strip(): # the segment's first line is the newline find() matched: skip blanks + if at is None: + try: + at = tags.index(LOCAL + "metric_value") + except ValueError: + return None, 0 + parts = line.split() + if len(parts) == len(tags): + try: + total += float(parts[at]) + except ValueError: + return None, 0 + count += 1 + return (total / count if count else None), count + + +out, problems = Path(sys.argv[1]), [] +if params["limit"]: + problems.append(f"limit: the published version's params.toml has limit = 0, not {params['limit']}") +files = sorted((out / "data").glob("*.parquet")) if (out / "data").is_dir() else [] +if not files: + problems.append("files: data/ holds no shards") +if [p.name for p in files] != [f"{i:05d}.parquet" for i in range(len(files))]: + problems.append("files: data/'s shards are not 00000.parquet onward without a gap") + +total, last, parsed, ids, sampled = 0, 0, 0, [], 0 +for n, p in enumerate(files): + try: + pf = pq.ParquetFile(p) + except Exception as e: + problems.append(f"parses: {p.name} is not a parquet file: {e}") + continue + got = [(f.name, f.type) for f in pf.schema_arrow] + if got != WANT: + problems.append(f"columns: {p.name} has {got}, not {WANT}") + continue + parsed += 1 + last = pf.metadata.num_rows + total += last + if n + 1 < len(files) and last != params["shard_rows"]: + problems.append(f"rows: {p.name} holds {last}, not {params['shard_rows']}") + light = pf.read(columns=[c for c, _ in WANT if c != "cif"]) + nulls = sum(col.null_count for col in light.columns) # a Table has no null_count; its columns do + if nulls: + problems.append(f"nulls: {p.name} holds {nulls} nulls") + shard_ids = light.column("id").to_pylist() + ids += shard_ids + for i, s in zip(shard_ids, light.column("uniprot_accession").to_pylist()): + if not ID.fullmatch(i) or i[3:].rsplit("-F", 1)[0] != s: + problems.append(f"id: {p.name} holds {i!r} with uniprot_accession {s!r}, not AF-[-]-F matching it") + break + same = pc.equal(light.column("length").cast(pa.int32()), pc.utf8_length(light.column("sequence"))) + if not pc.all(same).as_py(): + i = same.to_pylist().index(False) + problems.append(f"length: {p.name} row {i} ({shard_ids[i]}) is {light.column('length')[i].as_py()}, not len(sequence) {len(light.column('sequence')[i].as_py())}") + ok = pc.and_(pc.greater_equal(light.column("mean_plddt"), 0.0), pc.less_equal(light.column("mean_plddt"), 100.0)) + if not pc.all(ok).as_py(): + i = ok.to_pylist().index(False) + problems.append(f"mean_plddt: {p.name} row {i} ({shard_ids[i]}) holds {light.column('mean_plddt')[i].as_py()}, not in [0, 100]") + +if parsed == len(files) and files and total: + if not 0 < last <= params["shard_rows"]: + problems.append(f"rows: the last shard holds {last}, not at most {params['shard_rows']} and at least 1") + floor = params["entries"] * 9 // 10 + if total < floor: + problems.append(f"rows: the shards hold {total}, under the floor of {floor}, 90% of the archive's {params['entries']}") + elif total != params["entries"]: + problems.append(f"rows: the shards hold {total}, not the archive's {params['entries']}") + bad = next((k for k in range(1, len(ids)) if ids[k] <= ids[k - 1]), None) + if bad is not None: + problems.append(f"sorted: {ids[bad]} at row {bad} does not follow {ids[bad - 1]}; ids are sorted and unique across all shards") + by_shard = {} + for pos in sorted({k * total // 16 for k in range(16)}): + by_shard.setdefault(pos // params["shard_rows"], []).append(pos % params["shard_rows"]) + for shard, offs in sorted(by_shard.items()): + pf = pq.ParquetFile(files[shard]) + starts = [0] + for g in range(pf.metadata.num_row_groups): + starts.append(starts[-1] + pf.metadata.row_group(g).num_rows) + groups = {} + for off in offs: + g = bisect_right(starts, off) - 1 + if g not in groups: + groups[g] = pf.read_row_group(g) + sampled += 1 + row = groups[g].slice(off - starts[g], 1) + rid, seq = row.column("id")[0].as_py(), row.column("sequence")[0].as_py() + length, plddt = row.column("length")[0].as_py(), row.column("mean_plddt")[0].as_py() + cif = row.column("cif")[0].as_py() + where = f"{files[shard].name} {rid}" + try: + structure = MMCIFParser().get_structure(rid, io.StringIO(cif)) + except Exception as e: + problems.append(f"cif: {where} does not parse: {e}") + continue + residues = list(structure.get_residues()) + if len(residues) != length: + problems.append(f"length: {where}'s cif has {len(residues)} residues, not length {length}") + try: + spelled = "".join(THREE_TO_ONE[r.get_resname()] for r in residues) + except KeyError: + problems.append(f"sequence: {where}'s cif has a residue outside the 20 standard amino acids") + else: + if spelled != seq: + problems.append(f"sequence: {where}'s residues spell a sequence the row does not") + b = [a.bfactor for a in structure.get_atoms()] + if abs(sum(b) / len(b) - plddt) > 0.01: + problems.append(f"mean_plddt: {where}'s atoms' B-factors mean {sum(b) / len(b):.3f}, not mean_plddt {plddt} to 0.01") + e = re.search(r"(?m)^_entry\.id\s+(\S+)", cif) + if e is None or e.group(1) != rid: + problems.append(f"id: {where}'s cif's _entry.id is {e.group(1) if e else 'missing'}, not {rid}") + m = re.search(r"(?m)^_ma_qa_metric_global\.metric_value\s+(\S+)", cif) + try: + global_ = float(m.group(1)) if m else None + except ValueError: + global_ = None + if global_ is None or abs(global_ - plddt) > 1e-9: + problems.append(f"mean_plddt: {where}'s cif's global metric is {m.group(1) if m else 'missing'}, not mean_plddt {plddt}") + mean, count = local_plddts(cif) + if mean is None or count != length or abs(mean - plddt) > 5.0: + problems.append(f"mean_plddt: {where}'s per-residue pLDDTs ({count}) mean {mean}, not mean_plddt {plddt} to 5.0") + +print(f"sampled: {sampled} rows read whole") +print(f"data: {len(files)} shards, {total} rows") +print("\n".join(problems) or "ok") +sys.exit(1 if problems else 0) diff --git a/results/afdb-swissprot/1.0.0.txt b/results/afdb-swissprot/1.0.0.txt new file mode 100644 index 0000000..b3a93d0 --- /dev/null +++ b/results/afdb-swissprot/1.0.0.txt @@ -0,0 +1,454 @@ +--- +dataset: afdb-swissprot +version: 1.0.0 +hub: brineylab/afdb-swissprot +branch: candidate-016e0d4ad3fd2c595104bd31eca8c3b4dbc4a4f0 +commit: c949a24043c96fe77db8118c6c120d6d1ae6b9d1 +tree: 016e0d4ad3fd2c595104bd31eca8c3b4dbc4a4f0 +tools: datascience@1.2.0 +runner: sbatch --partition=cpu --cpus-per-task=16 --mem=64G --time=12:00:00 +host: bcr.scripps.edu +started: 2026-11-18T00:56:10Z +exit: 0 +--- +pyxis: importing docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +pyxis: imported docker image: ghcr.io#brineylab/toolshed/datascience:1.2.0 +source: https://ftp.ebi.ac.uk/pub/databases/alphafold/v4/swissprot_cif_v4.tar (39471090688 bytes, last modified Sun, 24 Aug 2025 11:07:54 GMT) +download: 0.0 of 39.5 GB (1.2 MB/s) +download: 0.5 of 39.5 GB (15.5 MB/s) +download: 1.0 of 39.5 GB (15.0 MB/s) +download: 1.4 of 39.5 GB (14.7 MB/s) +download: 1.8 of 39.5 GB (14.3 MB/s) +download: 2.2 of 39.5 GB (14.3 MB/s) +download: 2.6 of 39.5 GB (14.2 MB/s) +download: 3.0 of 39.5 GB (14.1 MB/s) +download: 3.5 of 39.5 GB (14.1 MB/s) +download: 3.9 of 39.5 GB (14.2 MB/s) +download: 4.4 of 39.5 GB (14.3 MB/s) +download: 4.8 of 39.5 GB (14.3 MB/s) +download: 5.2 of 39.5 GB (14.2 MB/s) +download: 5.6 of 39.5 GB (14.1 MB/s) +download: 6.0 of 39.5 GB (14.1 MB/s) +download: 6.5 of 39.5 GB (14.1 MB/s) +download: 6.9 of 39.5 GB (14.1 MB/s) +download: 7.3 of 39.5 GB (14.1 MB/s) +download: 7.7 of 39.5 GB (13.9 MB/s) +download: 8.0 of 39.5 GB (13.8 MB/s) +download: 8.4 of 39.5 GB (13.7 MB/s) +download: 8.8 of 39.5 GB (13.7 MB/s) +download: 9.2 of 39.5 GB (13.7 MB/s) +download: 9.5 of 39.5 GB (13.4 MB/s) +download: 9.9 of 39.5 GB (13.4 MB/s) +download: 10.3 of 39.5 GB (13.5 MB/s) +download: 10.6 of 39.5 GB (13.3 MB/s) +download: 10.9 of 39.5 GB (13.2 MB/s) +download: 11.3 of 39.5 GB (13.2 MB/s) +download: 11.6 of 39.5 GB (13.1 MB/s) +download: 12.1 of 39.5 GB (13.1 MB/s) +download: 12.4 of 39.5 GB (13.0 MB/s) +download: 12.7 of 39.5 GB (13.0 MB/s) +download: 13.1 of 39.5 GB (13.0 MB/s) +download: 13.5 of 39.5 GB (12.9 MB/s) +download: 13.8 of 39.5 GB (12.9 MB/s) +download: 14.2 of 39.5 GB (12.8 MB/s) +download: 14.6 of 39.5 GB (12.9 MB/s) +download: 15.0 of 39.5 GB (12.9 MB/s) +download: 15.4 of 39.5 GB (12.9 MB/s) +download: 15.8 of 39.5 GB (12.9 MB/s) +download: 16.2 of 39.5 GB (12.9 MB/s) +download: 16.6 of 39.5 GB (12.9 MB/s) +download: 16.9 of 39.5 GB (12.9 MB/s) +download: 17.3 of 39.5 GB (12.9 MB/s) +download: 17.6 of 39.5 GB (12.8 MB/s) +download: 18.0 of 39.5 GB (12.8 MB/s) +download: 18.4 of 39.5 GB (12.8 MB/s) +download: 18.7 of 39.5 GB (12.8 MB/s) +download: 19.1 of 39.5 GB (12.8 MB/s) +download: 19.6 of 39.5 GB (12.8 MB/s) +download: 19.9 of 39.5 GB (12.8 MB/s) +download: 20.4 of 39.5 GB (12.9 MB/s) +download: 20.8 of 39.5 GB (12.9 MB/s) +download: 21.2 of 39.5 GB (12.9 MB/s) +download: 21.6 of 39.5 GB (12.9 MB/s) +download: 21.9 of 39.5 GB (12.8 MB/s) +download: 22.3 of 39.5 GB (12.8 MB/s) +download: 22.6 of 39.5 GB (12.8 MB/s) +download: 23.0 of 39.5 GB (12.8 MB/s) +download: 23.4 of 39.5 GB (12.8 MB/s) +download: 23.7 of 39.5 GB (12.8 MB/s) +download: 24.1 of 39.5 GB (12.8 MB/s) +download: 24.6 of 39.5 GB (12.8 MB/s) +download: 25.0 of 39.5 GB (12.8 MB/s) +download: 25.4 of 39.5 GB (12.8 MB/s) +download: 25.8 of 39.5 GB (12.8 MB/s) +download: 26.2 of 39.5 GB (12.8 MB/s) +download: 26.7 of 39.5 GB (12.9 MB/s) +download: 27.0 of 39.5 GB (12.8 MB/s) +download: 27.4 of 39.5 GB (12.9 MB/s) +download: 27.8 of 39.5 GB (12.8 MB/s) +download: 28.2 of 39.5 GB (12.9 MB/s) +download: 28.7 of 39.5 GB (12.9 MB/s) +download: 29.1 of 39.5 GB (12.9 MB/s) +download: 29.4 of 39.5 GB (12.9 MB/s) +download: 29.8 of 39.5 GB (12.9 MB/s) +download: 30.2 of 39.5 GB (12.8 MB/s) +download: 30.6 of 39.5 GB (12.9 MB/s) +download: 31.1 of 39.5 GB (12.9 MB/s) +download: 31.3 of 39.5 GB (12.8 MB/s) +download: 31.7 of 39.5 GB (12.8 MB/s) +download: 32.1 of 39.5 GB (12.8 MB/s) +download: 32.4 of 39.5 GB (12.8 MB/s) +download: 32.8 of 39.5 GB (12.8 MB/s) +download: 33.1 of 39.5 GB (12.7 MB/s) +download: 33.5 of 39.5 GB (12.8 MB/s) +download: 33.9 of 39.5 GB (12.8 MB/s) +download: 34.2 of 39.5 GB (12.7 MB/s) +download: 34.5 of 39.5 GB (12.7 MB/s) +download: 35.0 of 39.5 GB (12.8 MB/s) +download: 35.3 of 39.5 GB (12.7 MB/s) +download: 35.7 of 39.5 GB (12.7 MB/s) +download: 36.0 of 39.5 GB (12.7 MB/s) +download: 36.4 of 39.5 GB (12.7 MB/s) +download: 36.7 of 39.5 GB (12.7 MB/s) +download: 36.9 of 39.5 GB (12.6 MB/s) +download: 37.2 of 39.5 GB (12.6 MB/s) +download: 37.4 of 39.5 GB (12.5 MB/s) +download: 37.6 of 39.5 GB (12.5 MB/s) +download: 37.8 of 39.5 GB (12.4 MB/s) +download: 37.9 of 39.5 GB (12.3 MB/s) +download: 38.1 of 39.5 GB (12.2 MB/s) +download: 38.2 of 39.5 GB (12.2 MB/s) +download: 38.3 of 39.5 GB (12.1 MB/s) +download: 38.5 of 39.5 GB (12.0 MB/s) +download: 38.6 of 39.5 GB (11.9 MB/s) +download: 38.8 of 39.5 GB (11.9 MB/s) +download: 38.9 of 39.5 GB (11.8 MB/s) +download: 39.0 of 39.5 GB (11.7 MB/s) +download: 39.1 of 39.5 GB (11.6 MB/s) +download: 39.1 of 39.5 GB (11.5 MB/s) +download: 39.2 of 39.5 GB (11.4 MB/s) +download: 39.3 of 39.5 GB (11.4 MB/s) +download: 39.3 of 39.5 GB (11.3 MB/s) +download: 39.4 of 39.5 GB (11.2 MB/s) +download: 39.4 of 39.5 GB (11.1 MB/s) +download: 39.4 of 39.5 GB (11.0 MB/s) +downloaded: 39.5 GB in 3612s (16 streams, 10.9 MB/s of new bytes) +entries: 542378 ids, sorted and unique +members: 2000/542378 (301/s) +members: 4000/542378 (315/s) +members: 6000/542378 (345/s) +members: 8000/542378 (358/s) +members: 10000/542378 (368/s) +members: 12000/542378 (365/s) +members: 14000/542378 (372/s) +members: 16000/542378 (377/s) +members: 18000/542378 (382/s) +members: 20000/542378 (387/s) +members: 22000/542378 (384/s) +members: 24000/542378 (382/s) +members: 26000/542378 (383/s) +members: 28000/542378 (384/s) +members: 30000/542378 (386/s) +members: 32000/542378 (389/s) +members: 34000/542378 (389/s) +members: 36000/542378 (391/s) +members: 38000/542378 (393/s) +members: 40000/542378 (394/s) +members: 42000/542378 (393/s) +members: 44000/542378 (394/s) +members: 46000/542378 (395/s) +members: 48000/542378 (397/s) +members: 50000/542378 (397/s) +members: 52000/542378 (397/s) +members: 54000/542378 (397/s) +members: 56000/542378 (398/s) +members: 58000/542378 (399/s) +members: 60000/542378 (398/s) +members: 62000/542378 (399/s) +members: 64000/542378 (399/s) +members: 66000/542378 (399/s) +members: 68000/542378 (400/s) +members: 70000/542378 (400/s) +members: 72000/542378 (400/s) +members: 74000/542378 (401/s) +members: 76000/542378 (400/s) +members: 78000/542378 (401/s) +members: 80000/542378 (401/s) +members: 82000/542378 (401/s) +members: 84000/542378 (402/s) +members: 86000/542378 (402/s) +members: 88000/542378 (403/s) +members: 90000/542378 (403/s) +members: 92000/542378 (403/s) +members: 94000/542378 (403/s) +members: 96000/542378 (403/s) +members: 98000/542378 (404/s) +members: 100000/542378 (405/s) +members: 102000/542378 (405/s) +members: 104000/542378 (405/s) +members: 106000/542378 (405/s) +members: 108000/542378 (405/s) +members: 110000/542378 (406/s) +members: 112000/542378 (406/s) +members: 114000/542378 (406/s) +members: 116000/542378 (406/s) +members: 118000/542378 (405/s) +members: 120000/542378 (405/s) +members: 122000/542378 (405/s) +members: 124000/542378 (405/s) +members: 126000/542378 (406/s) +members: 128000/542378 (406/s) +members: 130000/542378 (406/s) +members: 132000/542378 (407/s) +members: 134000/542378 (407/s) +members: 136000/542378 (407/s) +members: 138000/542378 (407/s) +members: 140000/542378 (407/s) +members: 142000/542378 (407/s) +members: 144000/542378 (407/s) +members: 146000/542378 (407/s) +members: 148000/542378 (407/s) +members: 150000/542378 (407/s) +members: 152000/542378 (407/s) +members: 154000/542378 (407/s) +members: 156000/542378 (407/s) +members: 158000/542378 (407/s) +members: 160000/542378 (407/s) +members: 162000/542378 (407/s) +members: 164000/542378 (407/s) +members: 166000/542378 (408/s) +members: 168000/542378 (408/s) +members: 170000/542378 (407/s) +members: 172000/542378 (407/s) +members: 174000/542378 (406/s) +members: 176000/542378 (405/s) +members: 178000/542378 (404/s) +members: 180000/542378 (400/s) +members: 182000/542378 (398/s) +members: 184000/542378 (397/s) +members: 186000/542378 (395/s) +members: 188000/542378 (395/s) +members: 190000/542378 (395/s) +members: 192000/542378 (394/s) +members: 194000/542378 (393/s) +members: 196000/542378 (392/s) +members: 198000/542378 (391/s) +members: 200000/542378 (390/s) +members: 202000/542378 (389/s) +members: 204000/542378 (390/s) +members: 206000/542378 (390/s) +members: 208000/542378 (389/s) +members: 210000/542378 (390/s) +members: 212000/542378 (390/s) +members: 214000/542378 (391/s) +members: 216000/542378 (391/s) +members: 218000/542378 (391/s) +members: 220000/542378 (391/s) +members: 222000/542378 (392/s) +members: 224000/542378 (391/s) +members: 226000/542378 (390/s) +members: 228000/542378 (390/s) +members: 230000/542378 (390/s) +members: 232000/542378 (389/s) +members: 234000/542378 (389/s) +members: 236000/542378 (388/s) +members: 238000/542378 (388/s) +members: 240000/542378 (387/s) +members: 242000/542378 (386/s) +members: 244000/542378 (386/s) +members: 246000/542378 (385/s) +members: 248000/542378 (385/s) +members: 250000/542378 (385/s) +members: 252000/542378 (384/s) +members: 254000/542378 (384/s) +members: 256000/542378 (383/s) +members: 258000/542378 (383/s) +members: 260000/542378 (383/s) +members: 262000/542378 (384/s) +members: 264000/542378 (384/s) +members: 266000/542378 (385/s) +members: 268000/542378 (386/s) +members: 270000/542378 (386/s) +members: 272000/542378 (386/s) +members: 274000/542378 (387/s) +members: 276000/542378 (388/s) +members: 278000/542378 (387/s) +members: 280000/542378 (387/s) +members: 282000/542378 (387/s) +members: 284000/542378 (386/s) +members: 286000/542378 (386/s) +members: 288000/542378 (386/s) +members: 290000/542378 (386/s) +members: 292000/542378 (385/s) +members: 294000/542378 (385/s) +members: 296000/542378 (385/s) +members: 298000/542378 (385/s) +members: 300000/542378 (384/s) +members: 302000/542378 (385/s) +members: 304000/542378 (384/s) +members: 306000/542378 (384/s) +members: 308000/542378 (383/s) +members: 310000/542378 (383/s) +members: 312000/542378 (383/s) +members: 314000/542378 (383/s) +members: 316000/542378 (383/s) +members: 318000/542378 (383/s) +members: 320000/542378 (383/s) +members: 322000/542378 (383/s) +members: 324000/542378 (383/s) +members: 326000/542378 (383/s) +members: 328000/542378 (382/s) +members: 330000/542378 (382/s) +members: 332000/542378 (382/s) +members: 334000/542378 (382/s) +members: 336000/542378 (382/s) +members: 338000/542378 (382/s) +members: 340000/542378 (382/s) +members: 342000/542378 (382/s) +members: 344000/542378 (382/s) +members: 346000/542378 (382/s) +members: 348000/542378 (382/s) +members: 350000/542378 (382/s) +members: 352000/542378 (382/s) +members: 354000/542378 (383/s) +members: 356000/542378 (383/s) +members: 358000/542378 (383/s) +members: 360000/542378 (382/s) +members: 362000/542378 (382/s) +members: 364000/542378 (382/s) +members: 366000/542378 (382/s) +members: 368000/542378 (382/s) +members: 370000/542378 (382/s) +members: 372000/542378 (382/s) +members: 374000/542378 (381/s) +members: 376000/542378 (381/s) +members: 378000/542378 (380/s) +members: 380000/542378 (379/s) +members: 382000/542378 (380/s) +members: 384000/542378 (379/s) +members: 386000/542378 (378/s) +members: 388000/542378 (378/s) +members: 390000/542378 (378/s) +members: 392000/542378 (378/s) +members: 394000/542378 (378/s) +members: 396000/542378 (378/s) +members: 398000/542378 (378/s) +members: 400000/542378 (378/s) +members: 402000/542378 (377/s) +members: 404000/542378 (377/s) +members: 406000/542378 (377/s) +members: 408000/542378 (376/s) +members: 410000/542378 (376/s) +members: 412000/542378 (376/s) +members: 414000/542378 (376/s) +members: 416000/542378 (375/s) +members: 418000/542378 (375/s) +members: 420000/542378 (375/s) +members: 422000/542378 (375/s) +members: 424000/542378 (375/s) +members: 426000/542378 (374/s) +members: 428000/542378 (374/s) +members: 430000/542378 (374/s) +members: 432000/542378 (373/s) +members: 434000/542378 (373/s) +members: 436000/542378 (373/s) +members: 438000/542378 (373/s) +members: 440000/542378 (373/s) +members: 442000/542378 (373/s) +members: 444000/542378 (373/s) +members: 446000/542378 (373/s) +members: 448000/542378 (373/s) +members: 450000/542378 (373/s) +members: 452000/542378 (372/s) +members: 454000/542378 (372/s) +members: 456000/542378 (372/s) +members: 458000/542378 (372/s) +members: 460000/542378 (372/s) +members: 462000/542378 (372/s) +members: 464000/542378 (371/s) +members: 466000/542378 (371/s) +members: 468000/542378 (371/s) +members: 470000/542378 (371/s) +members: 472000/542378 (371/s) +members: 474000/542378 (370/s) +members: 476000/542378 (370/s) +members: 478000/542378 (370/s) +members: 480000/542378 (370/s) +members: 482000/542378 (369/s) +members: 484000/542378 (369/s) +members: 486000/542378 (369/s) +members: 488000/542378 (369/s) +members: 490000/542378 (369/s) +members: 492000/542378 (369/s) +members: 494000/542378 (368/s) +members: 496000/542378 (368/s) +members: 498000/542378 (368/s) +members: 500000/542378 (368/s) +members: 502000/542378 (368/s) +members: 504000/542378 (368/s) +members: 506000/542378 (367/s) +members: 508000/542378 (367/s) +members: 510000/542378 (367/s) +members: 512000/542378 (367/s) +members: 514000/542378 (366/s) +members: 516000/542378 (366/s) +members: 518000/542378 (366/s) +members: 520000/542378 (366/s) +members: 522000/542378 (366/s) +members: 524000/542378 (365/s) +members: 526000/542378 (365/s) +members: 528000/542378 (365/s) +members: 530000/542378 (365/s) +members: 532000/542378 (364/s) +members: 534000/542378 (364/s) +members: 536000/542378 (363/s) +members: 538000/542378 (363/s) +members: 540000/542378 (363/s) +members: 542000/542378 (363/s) +members: 542378/542378 (363/s) +rows: 4.531 max |mean_plddt - residue mean|, 0 entries over 5.0 +shards: 37 +rows: 542378 +bytes: 36215798389 +took: 6219s +sampled: 16 rows read whole +data: 37 shards, 542378 rows +ok +file data/00000.parquet 1001798339 c581a38a6ce97f5d13d0a7dc461b9d425e46b8b7c518504819322d6a03372a87 +file data/00001.parquet 954694158 5e8adc16f987c48b6c3734b16138aff988f6c32d2ba4001788a5c0c24ab5e4f1 +file data/00002.parquet 912981339 5ff280a7e98b9c8348ad67dfc2c885734a6c6ff7f66c335fd8b84ece7ae605c6 +file data/00003.parquet 926936420 775b1e9b6426f09f35c4d29479a801819e18271edc56bc556a4350620c492f96 +file data/00004.parquet 915527233 73c9a581a86cadc162ed94944da0209f208525cc66be065ca6342ee81e4123f0 +file data/00005.parquet 885330212 b24b42f3ecd6f4c3d9de40b23ba4c7efff768e061aad14edd9ec7d6f446b7511 +file data/00006.parquet 885264602 dd17b617ec963cc52112f9babd84193d48e5ebe2a995a783f3020940088a8736 +file data/00007.parquet 924284099 d6a969ff13d0c8832230d8f7e3123586fe35ffbe00141b9299d2f23bb58438da +file data/00008.parquet 868604930 6a14a4ed3d7f2dd2a5a722a3d1586486c57a9a028c34a43cd7938cdffa5bbfd9 +file data/00009.parquet 853262203 6f2fd042a8e5a683c98858f2a968e89202e3e61f866c2ea6f6b2302097091922 +file data/00010.parquet 854654157 b3dbde98cd0b5b2fa1b11ecb926730be44ff67ecae693f762279a1b2215fc63c +file data/00011.parquet 1079528390 059b78211d2aced5bf9f10fb95a7033092ec48b7a19dd4376461a31186e203dd +file data/00012.parquet 1122537331 f73b80012e76266fe17490086f4e8f65db5edc11f1cd6aad47c7396110622e08 +file data/00013.parquet 1003903260 bc1ae4ad7729fa92cee814d1c2d9d4d3e6fa98b4d9dbbcb74c88762cc4569a49 +file data/00014.parquet 840736539 4b30960d81f469003482583394b2cb5b2786c9ccddbc13d693fb05cea72e92b5 +file data/00015.parquet 1087831564 3edebd932a566daa3bb1dce39b23ad388aa767b6fa99988a285d4c3c0dfeb22e +file data/00016.parquet 1062639743 86a671dcf051b48e52cc629222cd20813171291833901fe420f3d1c2e6315e9a +file data/00017.parquet 801292075 c71d2a79b47de5adebdc5d8fe3fb57443c6ec881b0435ec415b2a87906f3e6ab +file data/00018.parquet 889128168 f23bb58055d7ae7151f3a4da1a2f7f5129072bcab734a4ede4c946d182d4291b +file data/00019.parquet 1035782935 63f9251fdf5ab3e653d8037ca334af5a6e4039df913be60a956a678c9b41a2b1 +file data/00020.parquet 1043249604 067e573c262b725afdf4137d7d26bd7181bfebeaedcae66a4f7d98d35b493185 +file data/00021.parquet 981572063 4740a25db634e864a617bf8a6afcda90a8315ade630fcd09b3030025d9cdaa63 +file data/00022.parquet 954158226 6c05d93876e3a013d1b684513065b85040e73d18588934936e8c480f342298b2 +file data/00023.parquet 936711933 0b87ad6759309c99d5dc71cb178440cf912b3ad9383b19f29dd15ff2267d4c35 +file data/00024.parquet 1011326828 18fff86d28b01edce5ddc0b7dd0d75820648e9f03f97c4e487e6e4e2a9458640 +file data/00025.parquet 1142544799 53c5d7d4bfbff39b084960862b7a5332547bfe3157a1b93b43721fd53f24feaf +file data/00026.parquet 1065272066 42f94f1a6439731a027411e924cc6d804408273ea3a276b3bfd092ccf59e7a56 +file data/00027.parquet 1115247633 f6a80a1c80bd35854f2a833df7f0e96a2b3e088fd351982f4ea218b9ce0641f4 +file data/00028.parquet 1081265556 32d6f2f4e975c442493e64c8b1b2d91a160fe1f27a9b3d139a2ca26e568933d4 +file data/00029.parquet 1020360375 6eb479deeeb0ebae0c4b810f463efe5ff733a740de5064a956fb538963e56b02 +file data/00030.parquet 1117184983 c12a667e6fb1cbcaae432d4cf8be1c9a7012ece3444378b46faefd7dcb285630 +file data/00031.parquet 1079045138 3378c85d44422b31d0c236ef4b6d876a263a2ed546bb7db9ffe086cdfd6338ac +file data/00032.parquet 1086785973 404a87538549c615a3a5900b965dd42a0f0409adbbca2ddfe84639dced209cf6 +file data/00033.parquet 1092575188 0b644770755e2959906c6d885ff0c53e59ed97b890e9cdd76a37274b20dbcfe8 +file data/00034.parquet 1176149686 dd82566ed24c628957045931fda6e36ec73aa5b8e7da99c78cc39f3ae4325800 +file data/00035.parquet 1230973492 6f8eca9e625a880dbab90f209ff73811fa1027c827121f327611715c7642b526 +file data/00036.parquet 174657149 608ff25063dd38bb0f95f3b2288b848002e3c5c66c8d996a003017c6fa7459d7 +commit: c949a24043c96fe77db8118c6c120d6d1ae6b9d1