diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..02178d4 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,106 @@ +# Adding a dataset + +A dataset is built because a question needs it. Before you start, `ls datasets`: a sibling may be one parameter away from what you need, and a variant of it is a copy with that parameter changed and its own name. + +## The layout + +Copy the nearest sibling's directory to `datasets//`, where `` is lowercase words joined by hyphens, and change what differs. + +- `SKILL.md`: the skill, the dataset's whole description. +- `recipe/build`: the program. It reads `recipe/params.toml` beside it and writes the version's files into the directory it is given. +- `recipe/params.toml`: every parameter the recipe reads. +- `recipe/test`: checks an output directory; exits 0 iff it holds a version of this dataset. + +The record, `results//.txt`, is written by `scripts/record` after you cook and push; it lives outside the dataset's directory so committing it leaves the recipe alone. + +## The conventions + +- **The bytes are on the Hub, never in git.** Parquet where the data is tabular, sharded when a file would pass a few GB; every file under 200 GB, under 100k files. Anything the recipe stages for itself goes under a directory whose name starts with `.` (`/.work/`), which `scripts/push` leaves out. +- **A recipe is instructions, and whoever cooks it runs it**, anywhere: a laptop, a Slurm job, a flyloft campaign. Nothing in this repository runs, schedules, or orders a recipe; it files, records, checks, and promotes what a cook made. +- **A recipe runs in toolshed images and never pip-installs**, so its tools are pinned and the same images serve every cook. Name them in the skill's `metadata.tools`. Most recipes need only `datascience`, the shed's general-purpose image: the scientific Python stack, abstar, mmseqs2, DIAMOND, `huggingface_hub`, and git, with network access. +- **The identity is the recipe's inputs**: the source and its pull date, every parameter, the tools' versions. The skill states them in prose and the record carries them. A change to any of them is a new version; a change that makes the result incomparable (a threshold, a species) is a new dataset. +- **Never a credential in the repository.** `scripts/push` reads `HF_TOKEN` from the environment and never prints it; `scripts/push`, `scripts/record`, and `check` refuse a file that holds what looks like one. + +## The skill + +Frontmatter has five fields a machine reads: + +```markdown +--- +name: +description: "" +metadata: + version: 1.0.0 + tools: datascience@1.0.0 + license: +--- +``` + +`name` is the directory's and the Hub repository's, `description` is the catalogue line, `metadata.version` is the Hub tag (`v` prefixed), `metadata.tools` is one or more `name@version` separated by single spaces (a string: the skill format's `metadata` maps strings to strings), and `metadata.license` goes on the card. Then the body, in this order: `Hub:` (the repository and tag), `## Files` (each file, its rows, its columns), `## Reading it` (the `snapshot_download` line at this version), `## Recipe` (the source, its pull date, every parameter, the tools, what the test checks), `## Compute` (CPUs, memory, hours, disk, measured), `## Gotchas`. The body is also the Hub's card. No declared schemas: prose. + +## The recipe's shape + +One job is the default: `recipe/build ` does everything, in one image, on one node. Split a recipe only when one job is wrong for it: + +- **a very long step whose resumption matters** (days of annotation you would not redo because a later step failed), or +- **steps with different compute** (CPU clustering, then GPU embedding). + +A split recipe is several programs in `recipe/` (`recipe/build` is the first) and prose in the skill stating their order, each one's tool, and each one's compute. Its record is one record: `metadata.tools` lists every tool its steps use, and the record's body holds each step's output in order. **Chaining datasets** is the other way to split: a recipe may start from another dataset's published version, read with `snapshot_download` at a named tag. + +Whatever the shape, a recipe that stages its work under `/.work/` and skips a stage already finished there resumes for free. + +A recipe may read a `limit` from `params.toml` for a developer's dry run; the published version has none, or 0. + +## The test + +`recipe/test ` judges the real output: the files are there, each parses, the columns are the skill's, each split has at least the rows the recipe expects (a floor written in the test, about 90% of what the first full build gave), and whatever else the recipe knows (no id in two splits, no unannotated row). It runs in seconds on hundreds of GB, because it reads parquet footers and samples. It is not the recipe on a slice: the build is the real thing and the test judges it. + +## Cooking it + +Commit the dataset directory first: `scripts/push` names the candidate by the committed recipe's tree and refuses uncommitted or git-ignored changes under `datasets//`. Cook from a full clone; in a `git worktree`, `.git` points at the main repository, which a container must also mount or `push` cannot tell whether the recipe is committed. + +**On a laptop**, for a small dataset, with podman, from the clone's root (`hf auth login` once beforehand; the token is read from its file, never typed or printed): + +```bash +img=ghcr.io/brineylab/toolshed/datascience:1.0.0; out=$(mktemp -d); started=$(date -u +%FT%TZ) +run() { podman run --rm --pids-limit=-1 -e HF_TOKEN -v "$PWD":/work -v "$out":/out -w /work "$img" "$@"; } +run datasets//recipe/build /out +run datasets//recipe/test /out > "$out.test"; code=$? +if [ "$code" = 0 ]; then + HF_TOKEN=$(< ~/.cache/huggingface/token) run python3 -I scripts/push /out > "$out.push" + cat "$out.test" "$out.push" | scripts/record "$(sed -n 's/^commit: //p' "$out.push")" "$code" podman "$(hostname)" "$started" +fi +``` + +**On a Slurm cluster with pyxis**, from a clone on the login node, with `out` a directory on shared storage with room for the output and its staging. Pyxis passes your environment into the container, so export the token there, and never `HF_HUB_OFFLINE`, which stops the recipe's downloads and the push: + +```bash +export HF_TOKEN=$(< ~/.cache/huggingface/token); out=; mkdir -p "$out"; started=$(date -u +%FT%TZ) +srun --partition=

--cpus-per-task= --mem= --time= \ + --container-image=ghcr.io#brineylab/toolshed/datascience:1.0.0 \ + --container-mounts="$PWD":"$PWD","$out":"$out" --container-workdir="$PWD" \ + bash -c "echo host: \$(hostname -f) && datasets//recipe/build $out && datasets//recipe/test $out && python3 -I scripts/push $out" > "$out.run" 2> "$out.err" +code=$? +[ "$code" = 0 ] && scripts/record "$(sed -n 's/^commit: //p' "$out.run")" "$code" "srun --partition=

--cpus-per-task= --mem=" "$(sed -n 's/^host: //p' "$out.run")" "$started" < "$out.run" +``` + +On a laptop, `--pids-limit=-1` lifts podman's default of 2048 processes and threads per container, which abstar passes with a dozen worker processes: each starts polars' thread pools, sized to the machine. Mount `$out` and anything else the recipe reads (a shared data directory), or the job writes into the container's own filesystem, which goes with the job. The `host:` line is the compute node's, where the build and test ran. On coreweave-rtx6000, a CPU-only recipe takes `--partition=hpc-low`: a GPU node's spare CPUs, at most 32 CPUs and 240G. A failed build or test is fixed, never filed: when `code` is not 0, read `$out.err`, fix the recipe, commit, and cook again. + +`scripts/push` uploads the output to the Hub's candidate branch, `candidate-` on `/`, replacing an earlier candidate of the same tree, writes the card from the skill, and prints one `file ` line per file and `commit: ` last. `scripts/record` writes `results//.txt` from that output. Commit it: `git add results//.txt`. + +## The record + +The record's header names the Hub repository, the candidate branch and its head commit, the recipe's tree, the tools, the runner, the host, the start time, and the exit code; its body is the recipe's test output and push's manifest. `check` proves it on every pull request: the exit code is 0, the tree is the committed recipe's, the tools are the skill's, and the Hub's commit holds exactly the record's files with the record's sha256. Nobody can claim a version exists without one. + +A change after cooking changes the tree, so cook, push, and record again. A prose-only change to the skill leaves the tree alone, but it still bumps the version: push the same output again (the Hub skips the files it holds, so only the card moves) and record the new version. + +## Versions + +Any change under `datasets//` bumps `metadata.version`, prose included. A published tag is never moved, and `check` refuses a pull request whose tag is already on the Hub. Readers always name a version, never `main`. + +## The pull request + +1. Push your branch and open the pull request, the record committed in it. +2. `check` runs: the dataset's rules, an unpublished version, each tool in the toolshed at its version, and a record of a passing test whose files the Hub holds exactly. +3. Someone other than you reviews it, a critic agent included: does the skill describe the files the record lists and state the recipe's inputs; does the record show the test passed where the build ran; is the version new; are the tools the ones the record shows. They merge with a merge commit (`gh pr merge --merge`). +4. `promote` tags the record's commit `v` on the Hub and deletes the candidate branch. diff --git a/datasets/hello/SKILL.md b/datasets/hello/SKILL.md new file mode 100644 index 0000000..f03e1d3 --- /dev/null +++ b/datasets/hello/SKILL.md @@ -0,0 +1,36 @@ +--- +name: hello +description: "300 synthetic antibody pairs of random amino acids in three splits: the repository's smoke dataset. Reach for it to check that reading a dataset works from where you are, never to train." +metadata: + version: 1.0.0 + tools: datascience@1.0.0 + license: mit +--- +# hello + +Hub: `brineylab/hello`, tag `v1.0.0` + +## Files + +`train.parquet`, `eval.parquet`, `test.parquet`: 240, 30, and 30 rows, one per pair, columns `pair_id` (`hello-0000` to `hello-0299`), `sequence_heavy` (110 to 130 amino acids), `sequence_light` (105 to 115). The letters are uniform draws from the twenty amino acids: no sequence is an antibody. + +## Reading it + +```python +from huggingface_hub import snapshot_download +import polars as pl +path = snapshot_download("brineylab/hello", repo_type="dataset", revision="v1.0.0") +train = pl.read_parquet(f"{path}/train.parquet") +``` + +## Recipe + +No source: `recipe/build` draws `pairs = 300` pairs from `random.Random(seed = 1)`, shuffles them, and splits them `[0.8, 0.1, 0.1]` in that order. It runs in `datascience@1.0.0`. The test checks the three files, their columns, their row counts, and that no `pair_id` is in two splits. + +## Compute + +Seconds on a laptop, one CPU, kilobytes of disk. + +## Gotchas + +- It exists to exercise filing, checking, promoting, and reading; its sequences mean nothing. diff --git a/datasets/hello/recipe/build b/datasets/hello/recipe/build new file mode 100755 index 0000000..426b511 --- /dev/null +++ b/datasets/hello/recipe/build @@ -0,0 +1,23 @@ +#!/usr/bin/env -S python3 -I +"""recipe/build : `pairs` synthetic antibody pairs from params.toml's seed, shuffled and split by +`split` into train, eval, and test parquet files in .""" +import random +import sys +import tomllib +from pathlib import Path + +import polars as pl + +AA = "ACDEFGHIKLMNPQRSTVWY" +params = tomllib.loads((Path(__file__).parent / "params.toml").read_text()) +out = Path(sys.argv[1]) +out.mkdir(parents=True, exist_ok=True) +rng = random.Random(params["seed"]) +chain = lambda lo, hi: "".join(rng.choice(AA) for _ in range(rng.randint(lo, hi))) +rows = [(f"hello-{i:04d}", chain(110, 130), chain(105, 115)) for i in range(params["pairs"])] +rng.shuffle(rows) +n_train, n_eval = (round(f * len(rows)) for f in params["split"][:2]) +splits = {"train": rows[:n_train], "eval": rows[n_train:n_train + n_eval], "test": rows[n_train + n_eval:]} +for split, part in splits.items(): + pl.DataFrame(part, schema=["pair_id", "sequence_heavy", "sequence_light"], orient="row").write_parquet(out / f"{split}.parquet") + print(f"{split}.parquet {len(part)} rows") diff --git a/datasets/hello/recipe/params.toml b/datasets/hello/recipe/params.toml new file mode 100644 index 0000000..e525260 --- /dev/null +++ b/datasets/hello/recipe/params.toml @@ -0,0 +1,3 @@ +seed = 1 +pairs = 300 +split = [0.8, 0.1, 0.1] diff --git a/datasets/hello/recipe/test b/datasets/hello/recipe/test new file mode 100755 index 0000000..207a24f --- /dev/null +++ b/datasets/hello/recipe/test @@ -0,0 +1,26 @@ +#!/usr/bin/env -S python3 -I +"""recipe/test : exit 0 iff holds hello's three splits, with its columns and row counts and no +pair_id in two splits.""" +import sys +from pathlib import Path + +import polars as pl + +out, seen, problems = Path(sys.argv[1]), {}, [] +for split, rows in (("train", 240), ("eval", 30), ("test", 30)): + path = out / f"{split}.parquet" + if not path.is_file(): + problems.append(f"{split}.parquet is missing") + continue + df = pl.read_parquet(path) + if df.columns != ["pair_id", "sequence_heavy", "sequence_light"]: + problems.append(f"{split}.parquet has columns {df.columns}") + if df.height != rows: + problems.append(f"{split}.parquet has {df.height} rows, not {rows}") + for pid in df["pair_id"]: + if pid in seen: + problems.append(f"{pid} is in {seen[pid]} and {split}") + seen[pid] = split + print(f"{split}.parquet {df.height} rows") +print("\n".join(problems) or "ok") +sys.exit(1 if problems else 0) diff --git a/results/hello/1.0.0.txt b/results/hello/1.0.0.txt new file mode 100644 index 0000000..5eb1f62 --- /dev/null +++ b/results/hello/1.0.0.txt @@ -0,0 +1,21 @@ +--- +dataset: hello +version: 1.0.0 +hub: brineylab/hello +branch: candidate-22a46dde029a3283d7b17a1b29a48d7f01821605 +commit: 78253ef713accf01fde4ee2ec9ac97ebdfab4126 +tree: 22a46dde029a3283d7b17a1b29a48d7f01821605 +tools: datascience@1.0.0 +runner: podman +host: strix +started: 2026-11-06T19:03:56Z +exit: 0 +--- +train.parquet 240 rows +eval.parquet 30 rows +test.parquet 30 rows +ok +file eval.parquet 6602 fc311054c3da6268f2dcb7cb9a751697a5542f6fdd0ea5befe7bb7a6a39a234d +file test.parquet 6505 98c0a809593d563ea422663e21185e8b08631be9b7a593a3eda2d0034a0b5234 +file train.parquet 35145 5382f5f36ce977c410e4401ef5dc4c8755e8aa66d5cee8b4e217809473da5b72 +commit: 78253ef713accf01fde4ee2ec9ac97ebdfab4126