Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions datasets/uniref100/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
name: uniref100
description: "UniRef100 of UniProt release 2026_04 as sharded two-column parquet: one row per entry (a 100%-identity cluster's representative), sequence_id and sequence, in the FASTA's order. Reach for it to pre-train or evaluate a protein language model on UniRef100 at a pinned release."
metadata:
version: 1.0.0
tools: datascience@1.2.0
license: cc-by-4.0
---
# uniref100

Hub: `brineylab/uniref100`, tag `v1.0.0`

## Files

`data/00000.parquet` … `data/00227.parquet`: 228 shards holding 227,086,028 rows, one row per UniRef100 entry, in the FASTA's order. Every shard but the last holds exactly 1,000,000 rows (the last 86,028); zstd, row groups of about 100,000 rows. Columns:

- `sequence_id`: the FASTA header's first token, the entry id (`UniRef100_A0A5A9P0L4`).
- `sequence`: the entry's amino acids, its lines after the header joined.

## Reading it

```python
from huggingface_hub import snapshot_download
import polars as pl
path = snapshot_download("brineylab/uniref100", repo_type="dataset", revision="v1.0.0")
shard = pl.read_parquet(f"{path}/data/00000.parquet", columns=["sequence_id", "sequence"])
```

Rows keep the FASTA's order, so a subset of shards is a contiguous slice of the release, not a sample; shuffle yourself for training.

## Recipe

**Source:** UniRef release 2026_04 (published 2026-11-04), CC BY 4.0, pulled 2026-11-11: `uniref100.fasta.gz` (63,065,088,280 bytes, MD5 `923e539321e9bfe6a9b74d343284d600`) at https://ftp.uniprot.org/pub/databases/uniprot/uniref/uniref100/uniref100.fasta.gz, holding the release's 227,086,028 entries. The recipe refuses a download whose size or MD5 differs.

One job in `datascience@1.2.0`: download the FASTA into `<out>/.work/`, resuming a partial one by range; then stream it, never holding the whole file in memory, into `<out>/data/00000.parquet`, `00001.parquet`, … in the FASTA's order, `shard_rows` rows per shard (the last one partial), zstd, row groups of about 100k rows. Stages resume: a rerun keeps the verified FASTA and the shards already written, and a finished build only rereads the footers.

`params.toml`: `release = "2026_04"`; the file's `url`, `bytes`, and `md5`; `sequences = 227086028`, the release's count; `shard_rows = 1000000`.

**The test** reads the shards' parquet footers and three sampled row groups, in seconds: the shards are `00000.parquet` onward without a gap; the schema is exactly `sequence_id` and `sequence`, strings; every shard but the last holds exactly 1,000,000 rows; the total is exactly the release's 227,086,028; and in the samples every `sequence_id` starts with `UniRef100_` and every `sequence` is uppercase letters only.

## Compute

One job, CPU only: 8 CPUs and 32G asked, on bcr's `cpu` partition. From nothing it takes about 82 min: the download (58.7 GB, one stream from ftp.uniprot.org at about 25 MB/s) 42 min; the MD5 check and the 228 shards 40 min. Peak memory 2.6 GB. Disk: 48.2 GB of shards and 58.7 GB under `<out>/.work/`, the FASTA all of it, kept so a rerun resumes; the test reads only footers and three row groups, in 1 s.

## Gotchas

- The URL serves UniProt's current release only: once 2026_05 is out, the pinned MD5 refuses it, and a rebuild is a new version.
- The FASTA header's other fields (`n=`, `Tax=`, `TaxID=`, `RepID=`) are dropped; cluster sizes and taxonomy are in UniRef's XML, not here.
133 changes: 133 additions & 0 deletions datasets/uniref100/recipe/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
#!/usr/bin/env -S python3 -I
"""recipe/build <out>: UniRef100 of UniProt release 2026_04 as sharded two-column parquet. Downloads
uniref100.fasta.gz into <out>/.work/, refusing a size or MD5 that differs from params, then streams it, never
holding the whole file in memory, into <out>/data/00000.parquet, 00001.parquet, ...: the FASTA's order,
shard_rows rows per shard (the last one partial), zstd, row groups of about 100k rows. sequence_id is the
header's first token, sequence the amino acids of the header's lines joined. Stages resume: a rerun keeps
the verified FASTA and the shards already written. Prints the shard and row totals last.
"""
import gzip
import hashlib
import os
import sys
import time
import tomllib
import urllib.request
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

params = tomllib.loads((Path(__file__).parent / "params.toml").read_text())
out = Path(sys.argv[1]).resolve()
work, data = out / ".work", out / "data"
SCHEMA = pa.schema([("sequence_id", pa.string()), ("sequence", pa.string())])


def shard_paths() -> list[Path]:
"""data/'s shards in order; a gap ends the list, so a partial build resumes at its end."""
paths, i = [], 0
while (data / f"{i:05d}.parquet").is_file():
paths.append(data / f"{i:05d}.parquet")
i += 1
return paths


def rows(paths: list[Path]) -> int:
return sum(pq.ParquetFile(p).metadata.num_rows for p in paths)


def download() -> Path:
"""The FASTA in .work/, resumed by range from where a partial one stopped and given three attempts at
a connection; refuses a size or MD5 that differs from params."""
name = params["url"].rsplit("/", 1)[-1]
path, part = work / name, Path(f"{work / name}.part")
if path.is_file():
return path
for attempt in range(3):
have = part.stat().st_size if part.exists() else 0
if have >= params["bytes"]:
break
try:
headers = {"User-Agent": "brineylab-datasets"}
if have:
headers["Range"] = f"bytes={have}-"
request = urllib.request.Request(params["url"], headers=headers)
with urllib.request.urlopen(request, timeout=120) as r, open(part, "ab") as f:
if have and r.status != 206:
sys.exit(f"{params['url']} answered {r.status} to a range request at {have} bytes")
while block := r.read(1 << 22):
f.write(block)
break
except OSError as e:
if attempt == 2:
raise
print(f"{name}: {e}; retrying from {part.stat().st_size} bytes", file=sys.stderr)
time.sleep(30)
if part.stat().st_size != params["bytes"]:
sys.exit(f"{name}: {part.stat().st_size} bytes is not {params['bytes']}")
digest = hashlib.md5()
with open(part, "rb") as f:
while block := f.read(1 << 24):
digest.update(block)
if digest.hexdigest() != params["md5"]:
sys.exit(f"{name}: MD5 {digest.hexdigest()} is not {params['md5']}")
part.rename(path)
return path


def records(gz: Path):
"""(sequence_id, sequence) per FASTA record, in the file's order; the whole file is never held."""
with gzip.open(gz, "rt", encoding="ascii") as f:
sid, parts = None, []
for line in f:
if line.startswith(">"):
if sid is not None:
yield sid, "".join(parts)
sid, parts = line[1:].split(None, 1)[0], []
else:
parts.append(line.rstrip())
if sid is not None:
yield sid, "".join(parts)


def write(i: int, ids: list[str], seqs: list[str]) -> None:
"""Shard i whole or not at all: written under .work/ then moved into data/."""
pq.write_table(pa.table({"sequence_id": ids, "sequence": seqs}, schema=SCHEMA), work / "shard.part",
row_group_size=100_000, compression="zstd")
os.replace(work / "shard.part", data / f"{i:05d}.parquet")


def shard(gz: Path, first: int) -> None:
"""data/'s shards from first on, in the FASTA's order, skipping the rows of the shards already there."""
data.mkdir(parents=True, exist_ok=True)
skip = first * params["shard_rows"]
i, ids, seqs = first, [], []
for sid, seq in records(gz):
if skip:
skip -= 1
continue
ids.append(sid)
seqs.append(seq)
if len(ids) == params["shard_rows"]:
write(i, ids, seqs)
i, ids, seqs = i + 1, [], []
if ids:
write(i, ids, seqs)


def main() -> None:
if len(sys.argv) != 2:
sys.exit(f"usage: {Path(sys.argv[0]).name} <out>")
out.mkdir(parents=True, exist_ok=True)
work.mkdir(parents=True, exist_ok=True)
paths = shard_paths()
if rows(paths) != params["sequences"]:
shard(download(), len(paths))
paths = shard_paths()
print(f"shards: {len(paths)}")
print(f"rows: {rows(paths)}")


if __name__ == "__main__":
main()
6 changes: 6 additions & 0 deletions datasets/uniref100/recipe/params.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
release = "2026_04"
url = "https://ftp.uniprot.org/pub/databases/uniprot/uniref/uniref100/uniref100.fasta.gz"
bytes = 63065088280
md5 = "923e539321e9bfe6a9b74d343284d600"
sequences = 227086028
shard_rows = 1000000
72 changes: 72 additions & 0 deletions datasets/uniref100/recipe/test
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#!/usr/bin/env -S python3 -I
"""recipe/test <out>: exit 0 iff <out> holds a version of uniref100. From the shards' parquet footers and
three sampled row groups, in seconds: the shards are numbered without a gap, the schema is exactly
sequence_id and sequence (strings), every shard but the last is exactly shard_rows rows, the total is
exactly the release's count, and in the samples every sequence_id starts with UniRef100_ and every sequence
is uppercase letters only. One line per failure, naming the check and the first offending row."""
import re
import sys
import tomllib
from pathlib import Path

import pyarrow as pa
import pyarrow.parquet as pq

params = tomllib.loads((Path(__file__).parent / "params.toml").read_text())
PREFIX = "UniRef" + Path(__file__).resolve().parent.parent.name.removeprefix("uniref") + "_"
LETTERS = re.compile(r"[A-Z]+")

out, problems = Path(sys.argv[1]), []
files = sorted((out / "data").glob("*.parquet")) if (out / "data").is_dir() else []
if not files:
problems.append("files: data/ holds no shards")
if [p.name for p in files] != [f"{i:05d}.parquet" for i in range(len(files))]:
problems.append("files: data/'s shards are not 00000.parquet onward without a gap")

total, last, parsed = 0, 0, 0
for n, p in enumerate(files):
try:
pf = pq.ParquetFile(p)
except Exception as e:
problems.append(f"parses: {p.name} is not a parquet file: {e}")
continue
schema = pf.schema_arrow
if schema.names != ["sequence_id", "sequence"] or any(schema.field(c).type != pa.string() for c in schema.names):
problems.append(f"columns: {p.name} has {schema}, not exactly sequence_id and sequence as strings")
continue
parsed += 1
shard_rows = pf.metadata.num_rows
total += shard_rows
last = shard_rows
if n + 1 < len(files) and shard_rows != params["shard_rows"]:
problems.append(f"rows: {p.name} holds {shard_rows}, not {params['shard_rows']}")

if parsed:
if not 0 < last <= params["shard_rows"]:
problems.append(f"rows: the last shard holds {last}, not at most {params['shard_rows']} and at least 1")
if total != params["sequences"]:
problems.append(f"rows: the shards hold {total} in all, not the release's {params['sequences']}")
picks = sorted({0, len(files) // 2, len(files) - 1})
sampled = 0
for n in picks:
p = files[n]
try:
pf = pq.ParquetFile(p)
except Exception:
continue
groups = pf.metadata.num_row_groups
g = 0 if n == 0 else groups - 1 if n == len(files) - 1 else groups // 2
table = pf.read_row_group(g)
ids, seqs = table.column("sequence_id").to_pylist(), table.column("sequence").to_pylist()
sampled += len(ids)
bad = next((s for s in ids if not s.startswith(PREFIX)), None)
if bad is not None:
problems.append(f"sequence_id: {p.name} row group {g} holds {bad!r}, not starting with {PREFIX}")
bad = next((s for s in seqs if s is None or not LETTERS.fullmatch(s)), None)
if bad is not None:
problems.append(f"sequence: {p.name} row group {g} holds {bad!r}, not uppercase letters only")
print(f"sampled: {len(picks)} row groups ({sampled} rows)")

print(f"data: {len(files)} shards, {total} rows")
print("\n".join(problems) or "ok")
sys.exit(1 if problems else 0)
Loading
Loading