Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 106 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Adding a dataset

A dataset is built because a question needs it. Before you start, `ls datasets`: a sibling may be one parameter away from what you need, and a variant of it is a copy with that parameter changed and its own name.

## The layout

Copy the nearest sibling's directory to `datasets/<name>/`, where `<name>` is lowercase words joined by hyphens, and change what differs.

- `SKILL.md`: the skill, the dataset's whole description.
- `recipe/build`: the program. It reads `recipe/params.toml` beside it and writes the version's files into the directory it is given.
- `recipe/params.toml`: every parameter the recipe reads.
- `recipe/test`: checks an output directory; exits 0 iff it holds a version of this dataset.

The record, `results/<name>/<version>.txt`, is written by `scripts/record` after you cook and push; it lives outside the dataset's directory so committing it leaves the recipe alone.

## The conventions

- **The bytes are on the Hub, never in git.** Parquet where the data is tabular, sharded when a file would pass a few GB; every file under 200 GB, under 100k files. Anything the recipe stages for itself goes under a directory whose name starts with `.` (`<out>/.work/`), which `scripts/push` leaves out.
- **A recipe is instructions, and whoever cooks it runs it**, anywhere: a laptop, a Slurm job, a flyloft campaign. Nothing in this repository runs, schedules, or orders a recipe; it files, records, checks, and promotes what a cook made.
- **A recipe runs in toolshed images and never pip-installs**, so its tools are pinned and the same images serve every cook. Name them in the skill's `metadata.tools`. Most recipes need only `datascience`, the shed's general-purpose image: the scientific Python stack, abstar, mmseqs2, DIAMOND, `huggingface_hub`, and git, with network access.
- **The identity is the recipe's inputs**: the source and its pull date, every parameter, the tools' versions. The skill states them in prose and the record carries them. A change to any of them is a new version; a change that makes the result incomparable (a threshold, a species) is a new dataset.
- **Never a credential in the repository.** `scripts/push` reads `HF_TOKEN` from the environment and never prints it; `scripts/push`, `scripts/record`, and `check` refuse a file that holds what looks like one.

## The skill

Frontmatter has five fields a machine reads:

```markdown
---
name: <name>
description: "<what it is, and when to reach for it>"
metadata:
version: 1.0.0
tools: datascience@1.0.0
license: <the Hub's license id, e.g. mit or cc-by-4.0>
---
```

`name` is the directory's and the Hub repository's, `description` is the catalogue line, `metadata.version` is the Hub tag (`v` prefixed), `metadata.tools` is one or more `name@version` separated by single spaces (a string: the skill format's `metadata` maps strings to strings), and `metadata.license` goes on the card. Then the body, in this order: `Hub:` (the repository and tag), `## Files` (each file, its rows, its columns), `## Reading it` (the `snapshot_download` line at this version), `## Recipe` (the source, its pull date, every parameter, the tools, what the test checks), `## Compute` (CPUs, memory, hours, disk, measured), `## Gotchas`. The body is also the Hub's card. No declared schemas: prose.

## The recipe's shape

One job is the default: `recipe/build <out>` does everything, in one image, on one node. Split a recipe only when one job is wrong for it:

- **a very long step whose resumption matters** (days of annotation you would not redo because a later step failed), or
- **steps with different compute** (CPU clustering, then GPU embedding).

A split recipe is several programs in `recipe/` (`recipe/build` is the first) and prose in the skill stating their order, each one's tool, and each one's compute. Its record is one record: `metadata.tools` lists every tool its steps use, and the record's body holds each step's output in order. **Chaining datasets** is the other way to split: a recipe may start from another dataset's published version, read with `snapshot_download` at a named tag.

Whatever the shape, a recipe that stages its work under `<out>/.work/` and skips a stage already finished there resumes for free.

A recipe may read a `limit` from `params.toml` for a developer's dry run; the published version has none, or 0.

## The test

`recipe/test <out>` judges the real output: the files are there, each parses, the columns are the skill's, each split has at least the rows the recipe expects (a floor written in the test, about 90% of what the first full build gave), and whatever else the recipe knows (no id in two splits, no unannotated row). It runs in seconds on hundreds of GB, because it reads parquet footers and samples. It is not the recipe on a slice: the build is the real thing and the test judges it.

## Cooking it

Commit the dataset directory first: `scripts/push` names the candidate by the committed recipe's tree and refuses uncommitted or git-ignored changes under `datasets/<name>/`. Cook from a full clone; in a `git worktree`, `.git` points at the main repository, which a container must also mount or `push` cannot tell whether the recipe is committed.

**On a laptop**, for a small dataset, with podman, from the clone's root (`hf auth login` once beforehand; the token is read from its file, never typed or printed):

```bash
img=ghcr.io/brineylab/toolshed/datascience:1.0.0; out=$(mktemp -d); started=$(date -u +%FT%TZ)
run() { podman run --rm --pids-limit=-1 -e HF_TOKEN -v "$PWD":/work -v "$out":/out -w /work "$img" "$@"; }
run datasets/<name>/recipe/build /out
run datasets/<name>/recipe/test /out > "$out.test"; code=$?
if [ "$code" = 0 ]; then
HF_TOKEN=$(< ~/.cache/huggingface/token) run python3 -I scripts/push <name> /out > "$out.push"
cat "$out.test" "$out.push" | scripts/record <name> "$(sed -n 's/^commit: //p' "$out.push")" "$code" podman "$(hostname)" "$started"
fi
```

**On a Slurm cluster with pyxis**, from a clone on the login node, with `out` a directory on shared storage with room for the output and its staging. Pyxis passes your environment into the container, so export the token there, and never `HF_HUB_OFFLINE`, which stops the recipe's downloads and the push:

```bash
export HF_TOKEN=$(< ~/.cache/huggingface/token); out=<a directory on shared storage>; mkdir -p "$out"; started=$(date -u +%FT%TZ)
srun --partition=<p> --cpus-per-task=<n> --mem=<m> --time=<t> \
--container-image=ghcr.io#brineylab/toolshed/datascience:1.0.0 \
--container-mounts="$PWD":"$PWD","$out":"$out" --container-workdir="$PWD" \
bash -c "echo host: \$(hostname -f) && datasets/<name>/recipe/build $out && datasets/<name>/recipe/test $out && python3 -I scripts/push <name> $out" > "$out.run" 2> "$out.err"
code=$?
[ "$code" = 0 ] && scripts/record <name> "$(sed -n 's/^commit: //p' "$out.run")" "$code" "srun --partition=<p> --cpus-per-task=<n> --mem=<m>" "$(sed -n 's/^host: //p' "$out.run")" "$started" < "$out.run"
```

On a laptop, `--pids-limit=-1` lifts podman's default of 2048 processes and threads per container, which abstar passes with a dozen worker processes: each starts polars' thread pools, sized to the machine. Mount `$out` and anything else the recipe reads (a shared data directory), or the job writes into the container's own filesystem, which goes with the job. The `host:` line is the compute node's, where the build and test ran. On coreweave-rtx6000, a CPU-only recipe takes `--partition=hpc-low`: a GPU node's spare CPUs, at most 32 CPUs and 240G. A failed build or test is fixed, never filed: when `code` is not 0, read `$out.err`, fix the recipe, commit, and cook again.

`scripts/push` uploads the output to the Hub's candidate branch, `candidate-<tree>` on `<org>/<name>`, replacing an earlier candidate of the same tree, writes the card from the skill, and prints one `file <path> <size> <sha256>` line per file and `commit: <sha>` last. `scripts/record` writes `results/<name>/<version>.txt` from that output. Commit it: `git add results/<name>/<version>.txt`.

## The record

The record's header names the Hub repository, the candidate branch and its head commit, the recipe's tree, the tools, the runner, the host, the start time, and the exit code; its body is the recipe's test output and push's manifest. `check` proves it on every pull request: the exit code is 0, the tree is the committed recipe's, the tools are the skill's, and the Hub's commit holds exactly the record's files with the record's sha256. Nobody can claim a version exists without one.

A change after cooking changes the tree, so cook, push, and record again. A prose-only change to the skill leaves the tree alone, but it still bumps the version: push the same output again (the Hub skips the files it holds, so only the card moves) and record the new version.

## Versions

Any change under `datasets/<name>/` bumps `metadata.version`, prose included. A published tag is never moved, and `check` refuses a pull request whose tag is already on the Hub. Readers always name a version, never `main`.

## The pull request

1. Push your branch and open the pull request, the record committed in it.
2. `check` runs: the dataset's rules, an unpublished version, each tool in the toolshed at its version, and a record of a passing test whose files the Hub holds exactly.
3. Someone other than you reviews it, a critic agent included: does the skill describe the files the record lists and state the recipe's inputs; does the record show the test passed where the build ran; is the version new; are the tools the ones the record shows. They merge with a merge commit (`gh pr merge --merge`).
4. `promote` tags the record's commit `v<version>` on the Hub and deletes the candidate branch.
36 changes: 36 additions & 0 deletions datasets/hello/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
name: hello
description: "300 synthetic antibody pairs of random amino acids in three splits: the repository's smoke dataset. Reach for it to check that reading a dataset works from where you are, never to train."
metadata:
version: 1.0.0
tools: datascience@1.0.0
license: mit
---
# hello

Hub: `brineylab/hello`, tag `v1.0.0`

## Files

`train.parquet`, `eval.parquet`, `test.parquet`: 240, 30, and 30 rows, one per pair, columns `pair_id` (`hello-0000` to `hello-0299`), `sequence_heavy` (110 to 130 amino acids), `sequence_light` (105 to 115). The letters are uniform draws from the twenty amino acids: no sequence is an antibody.

## Reading it

```python
from huggingface_hub import snapshot_download
import polars as pl
path = snapshot_download("brineylab/hello", repo_type="dataset", revision="v1.0.0")
train = pl.read_parquet(f"{path}/train.parquet")
```

## Recipe

No source: `recipe/build` draws `pairs = 300` pairs from `random.Random(seed = 1)`, shuffles them, and splits them `[0.8, 0.1, 0.1]` in that order. It runs in `datascience@1.0.0`. The test checks the three files, their columns, their row counts, and that no `pair_id` is in two splits.

## Compute

Seconds on a laptop, one CPU, kilobytes of disk.

## Gotchas

- It exists to exercise filing, checking, promoting, and reading; its sequences mean nothing.
23 changes: 23 additions & 0 deletions datasets/hello/recipe/build
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/usr/bin/env -S python3 -I
"""recipe/build <out>: `pairs` synthetic antibody pairs from params.toml's seed, shuffled and split by
`split` into train, eval, and test parquet files in <out>."""
import random
import sys
import tomllib
from pathlib import Path

import polars as pl

AA = "ACDEFGHIKLMNPQRSTVWY"
params = tomllib.loads((Path(__file__).parent / "params.toml").read_text())
out = Path(sys.argv[1])
out.mkdir(parents=True, exist_ok=True)
rng = random.Random(params["seed"])
chain = lambda lo, hi: "".join(rng.choice(AA) for _ in range(rng.randint(lo, hi)))
rows = [(f"hello-{i:04d}", chain(110, 130), chain(105, 115)) for i in range(params["pairs"])]
rng.shuffle(rows)
n_train, n_eval = (round(f * len(rows)) for f in params["split"][:2])
splits = {"train": rows[:n_train], "eval": rows[n_train:n_train + n_eval], "test": rows[n_train + n_eval:]}
for split, part in splits.items():
pl.DataFrame(part, schema=["pair_id", "sequence_heavy", "sequence_light"], orient="row").write_parquet(out / f"{split}.parquet")
print(f"{split}.parquet {len(part)} rows")
3 changes: 3 additions & 0 deletions datasets/hello/recipe/params.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
seed = 1
pairs = 300
split = [0.8, 0.1, 0.1]
26 changes: 26 additions & 0 deletions datasets/hello/recipe/test
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env -S python3 -I
"""recipe/test <out>: exit 0 iff <out> holds hello's three splits, with its columns and row counts and no
pair_id in two splits."""
import sys
from pathlib import Path

import polars as pl

out, seen, problems = Path(sys.argv[1]), {}, []
for split, rows in (("train", 240), ("eval", 30), ("test", 30)):
path = out / f"{split}.parquet"
if not path.is_file():
problems.append(f"{split}.parquet is missing")
continue
df = pl.read_parquet(path)
if df.columns != ["pair_id", "sequence_heavy", "sequence_light"]:
problems.append(f"{split}.parquet has columns {df.columns}")
if df.height != rows:
problems.append(f"{split}.parquet has {df.height} rows, not {rows}")
for pid in df["pair_id"]:
if pid in seen:
problems.append(f"{pid} is in {seen[pid]} and {split}")
seen[pid] = split
print(f"{split}.parquet {df.height} rows")
print("\n".join(problems) or "ok")
sys.exit(1 if problems else 0)
21 changes: 21 additions & 0 deletions results/hello/1.0.0.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
dataset: hello
version: 1.0.0
hub: brineylab/hello
branch: candidate-22a46dde029a3283d7b17a1b29a48d7f01821605
commit: 78253ef713accf01fde4ee2ec9ac97ebdfab4126
tree: 22a46dde029a3283d7b17a1b29a48d7f01821605
tools: datascience@1.0.0
runner: podman
host: strix
started: 2026-11-06T19:03:56Z
exit: 0
---
train.parquet 240 rows
eval.parquet 30 rows
test.parquet 30 rows
ok
file eval.parquet 6602 fc311054c3da6268f2dcb7cb9a751697a5542f6fdd0ea5befe7bb7a6a39a234d
file test.parquet 6505 98c0a809593d563ea422663e21185e8b08631be9b7a593a3eda2d0034a0b5234
file train.parquet 35145 5382f5f36ce977c410e4401ef5dc4c8755e8aa66d5cee8b4e217809473da5b72
commit: 78253ef713accf01fde4ee2ec9ac97ebdfab4126
Loading