Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ hub.toml the Hub org this repository publishes to
scripts/check.py a dataset directory's rules, its record, and whether its version is published
scripts/push <name> <output dir> upload the cook's output and the card to the Hub's candidate branch; prints the manifest and `commit:`
scripts/record <name> <commit> <exit> <runner> <host> <started> < output write a record from the test's and push's output
scripts/promote <name> <version> after the merge, by whoever merged: tag the record's commit on the Hub, never moving a tag; delete the candidate branch
scripts/promote <name> <version> after the merge, by whoever merged: tag the record's commit on the Hub, never moving a tag; give main its files; delete the candidate branch
tests/ unit tests of the scripts, with the Hub stubbed
.github/workflows/datasets.yml check, on hosted runners
```
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ A change after cooking changes the tree, so cook, push, and record again. A pros

## Versions

Any change under `datasets/<name>/` bumps `metadata.version`, prose included. A published tag is never moved, and `check` refuses a pull request whose tag is already on the Hub. Readers always name a version, never `main`.
Any change under `datasets/<name>/` bumps `metadata.version`, prose included. A published tag is never moved, and `check` refuses a pull request whose tag is already on the Hub. Readers always name a version, never `main`, which is a copy of the version last promoted, for the Hub's page.

## The pull request

Expand All @@ -110,4 +110,4 @@ Any change under `datasets/<name>/` bumps `metadata.version`, prose included. A
ghcr.io/brineylab/toolshed/datascience:1.0.0 python3 -I scripts/promote <name> <version>
```

It tags the record's commit `v<version>` on the Hub and deletes the candidate branch; its last line says what it did. It refuses a record that is not on `origin/main`. Run it from a full clone, not a `git worktree`, whose `.git` points outside the mounted directory. CI holds no Hub token, so nothing is published until someone runs it.
It tags the record's commit `v<version>` on the Hub, gives `main` the version's files, and deletes the candidate branch; its last line says what it did. It refuses a record that is not on `origin/main`. Run it from a full clone, not a `git worktree`, whose `.git` points outside the mounted directory. CI holds no Hub token, so nothing is published until someone runs it.
10 changes: 5 additions & 5 deletions docs/DESIGN.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# The datasets repository: Design

Status: approved 2026-11-04; revised 2026-11-05 by [superpowers/specs/2026-11-05-recipes-tools-and-jaffe-paired-design.md](superpowers/specs/2026-11-05-recipes-tools-and-jaffe-paired-design.md), so that a recipe is instructions any cook runs, names its tools, and runs in the toolshed's `datascience` (§2, §3, §4, §5, §6); revised 2026-11-06 so that whoever merges promotes with their own Hub token, and CI holds none, since a Hub token is a person's, not an org's (§3, §5). This is the repository's own copy of the design that lives in flyloft as [`docs/design/datasets.md`](https://github.com/brineylab/flyloft/blob/main/docs/design/datasets.md), with the lab's decisions on flyloft issue [#174](https://github.com/brineylab/flyloft/issues/174) (D1–D7, all option A) marked by number. §1 to §4 are this repository's; §5 summarizes flyloft's side, which that document settles. Calls made in writing that nobody has discussed are marked **(call)**. The toolshed's design ([brineylab/toolshed, docs/DESIGN.md](https://github.com/brineylab/toolshed/blob/main/docs/DESIGN.md)) is the sibling this one mirrors; where this document says "as the toolshed's", that is the reference.
Status: approved 2026-11-04; revised 2026-11-05 by [superpowers/specs/2026-11-05-recipes-tools-and-jaffe-paired-design.md](superpowers/specs/2026-11-05-recipes-tools-and-jaffe-paired-design.md), so that a recipe is instructions any cook runs, names its tools, and runs in the toolshed's `datascience` (§2, §3, §4, §5, §6); revised 2026-11-06 so that whoever merges promotes with their own Hub token, and CI holds none, since a Hub token is a person's, not an org's (§3, §5); revised 2026-11-10 so that promoting gives the Hub's `main` the version's files, since the Hub's page and viewer show `main`, which was empty (§3). This is the repository's own copy of the design that lives in flyloft as [`docs/design/datasets.md`](https://github.com/brineylab/flyloft/blob/main/docs/design/datasets.md), with the lab's decisions on flyloft issue [#174](https://github.com/brineylab/flyloft/issues/174) (D1–D7, all option A) marked by number. §1 to §4 are this repository's; §5 summarizes flyloft's side, which that document settles. Calls made in writing that nobody has discussed are marked **(call)**. The toolshed's design ([brineylab/toolshed, docs/DESIGN.md](https://github.com/brineylab/toolshed/blob/main/docs/DESIGN.md)) is the sibling this one mirrors; where this document says "as the toolshed's", that is the reference.

## 1. Scenario

Expand Down Expand Up @@ -63,23 +63,23 @@ results/<name>/<version>.txt the record of the candidate's test: the Hub commi
scripts/check.py a dataset directory's rules, its record, and whether its version is published
scripts/push <name> <output dir> upload the cook's output and the card to the Hub on the candidate branch; prints the manifest and the commit
scripts/record a record from the test's and the push's output, for any cook
scripts/promote run by whoever merges: tag the merged record's commit as the version on the Hub; delete the candidate branch
scripts/promote run by whoever merges: tag the merged record's commit as the version on the Hub; give main its files; delete the candidate branch
.github/workflows/datasets.yml check
```

**Versions.** `metadata.version` is the Hub tag, `v<version>` on `<org>/<name>`, with the org from `hub.toml`, so a fork with its own org is its own catalogue and store with one edit. A published tag is never moved: any change under `datasets/<name>/` bumps the version, prose included, and CI refuses a pull request whose tag is already on the Hub. Agents and flyloft always name a version, never `main`.
**Versions.** `metadata.version` is the Hub tag, `v<version>` on `<org>/<name>`, with the org from `hub.toml`, so a fork with its own org is its own catalogue and store with one edit. A published tag is never moved: any change under `datasets/<name>/` bumps the version, prose included, and CI refuses a pull request whose tag is already on the Hub. Agents and flyloft always name a version, never `main`. `main` is for people browsing the Hub: a copy of the version last promoted, one commit per promote with the version's files, not the tagged commit itself, since the Hub moves a branch only by a commit.

**The Hub** is the store (D1). One dataset repository per dataset, public, created by the first push when it does not exist. The Hub gives what the toolshed's registry gives and more: every commit immutable, a tag per version, every file's sha256, hosting, a dataset viewer, and a download-and-cache library every agent already knows. A dataset built from public data is public from the start, which is also where reviewers look when the lab publishes. A dataset the lab could not make public is the exception, handled when it arrives (a private repository and a read token), not designed for now. A few hundred GB per dataset, the largest foreseen, is within the Hub's guidance (no per-repository cap for public repositories; files under 200 GB, parquet preferred, under 100k files). Free organizations get best-effort public storage beyond the first gigabytes, so before plan A's large dataset the lab confirms its Hub plan holds what it will publish.

**The candidate** is the Hub commit a pull request proposes to publish: the head of the branch `candidate-<tree>` on the dataset's Hub repository, `<tree>` being git's hash of the committed `datasets/<name>/recipe/` directory, so the branch names the inputs it was built from. The scripts compute it from the files, not from `.git`, so `record` and `check` need no checkout **(call)**; `push` needs one, to prove the files are the commit's. The skill is not in the tree: a prose change does not change the bytes. **Building is the cook's.** Whoever cooks a recipe runs it, and its test, where the dataset will be built, into an output directory of their choosing. `scripts/push <name> <output dir>` refuses when `datasets/<name>/` has uncommitted or ignored changes, or when git cannot say, so the candidate is named by the committed recipe's tree; it uploads the directory to the candidate branch, writes the card as a commit on it, and prints the file manifest with sizes and sha256 and then the head commit on a tagged `commit:` line, so `scripts/record` reads everything it needs from that output and nothing on the flyloft host needs the Hub library **(call)**; the Hub skips every file it already holds, so a second push of the same output moves prose only. Then `scripts/record` writes **the record**, `results/<name>/<version>.txt`: the Hub repository, the branch and its head commit, the recipe tree, the parameters, the tools, where it ran, the time, the files with their sizes and sha256, the test's output, and the exit code. The record lives outside `datasets/<name>/`, so committing it leaves the candidate alone. A person cooks under `srun` in the recipe's tool image on a cluster, or with `podman` on a laptop for a small one, as `CONTRIBUTING.md` writes out; flyloft cooks as its campaign decides (§5). CI never runs a recipe or pulls a dataset.
**The candidate** is the Hub commit a pull request proposes to publish: the head of the branch `candidate-<tree>` on the dataset's Hub repository, `<tree>` being git's hash of the committed `datasets/<name>/recipe/` directory, so the branch names the inputs it was built from. The scripts compute it from the files, not from `.git`, so `record` and `check` need no checkout **(call)**; `push` needs one, to prove the files are the commit's. The skill is not in the tree: a prose change does not change the bytes. **Building is the cook's.** Whoever cooks a recipe runs it, and its test, where the dataset will be built, into an output directory of their choosing. `scripts/push <name> <output dir>` refuses when `datasets/<name>/` has uncommitted or ignored changes, or when git cannot say, so the candidate is named by the committed recipe's tree; it uploads the directory to the candidate branch, which starts from `main`, deleting what the output lacks, writes the card as a commit on it, and prints the file manifest with sizes and sha256 and then the head commit on a tagged `commit:` line, so `scripts/record` reads everything it needs from that output and nothing on the flyloft host needs the Hub library **(call)**; the Hub skips every file it already holds, so a second push of the same output moves prose only. Then `scripts/record` writes **the record**, `results/<name>/<version>.txt`: the Hub repository, the branch and its head commit, the recipe tree, the parameters, the tools, where it ran, the time, the files with their sizes and sha256, the test's output, and the exit code. The record lives outside `datasets/<name>/`, so committing it leaves the candidate alone. A person cooks under `srun` in the recipe's tool image on a cluster, or with `podman` on a laptop for a small one, as `CONTRIBUTING.md` writes out; flyloft cooks as its campaign decides (§5). CI never runs a recipe or pulls a dataset.

**The card.** The Hub asks every large dataset for a card and serves its viewer from parquet. `scripts/push` writes `README.md` on the candidate branch from the skill: a header with the license and the name, then the skill's body. One source, two readers.

**The workflow**, on GitHub's hosted runners only, never downloading a dataset and holding no Hub token:

- `check`, on every pull request, for the datasets it changes: the skill's frontmatter parses, its name is the directory's, each of its tools is in the toolshed's catalogue at that version (read from the shed's `main` by API); `recipe/build`, `recipe/params.toml`, and `recipe/test` exist; the version's tag is not on the Hub; the record exists for this version, names a commit on the Hub whose tree holds exactly the record's files with the record's sha256 (the Hub's file listing, no download), and shows exit code 0.

**Promoting** is the merger's, as pushing is the cook's: after the merge, whoever merged pulls `main` and runs `scripts/promote <name> <version>` with their own `HF_TOKEN`, which tags the record's commit `v<version>` on the Hub, never moving a tag that exists, and deletes the candidate branch; it refuses a record that is not on `origin/main`, so only a reviewed, merged record is published, and its last line says what it did. It runs from a full clone, since a worktree's `.git` points outside the directory a container mounts. A Hub token is a person's, so CI holds none; this is the one place the repository differs from the toolshed, whose CI promotes with GitHub's own token.
**Promoting** is the merger's, as pushing is the cook's: after the merge, whoever merged pulls `main` and runs `scripts/promote <name> <version>` with their own `HF_TOKEN`, which tags the record's commit `v<version>` on the Hub, never moving a tag that exists, gives `main` the version's files in one commit, copied on the Hub (none when `main` holds them already), and deletes the candidate branch; it refuses a record that is not on `origin/main`, so only a reviewed, merged record is published, and its last line says what it did. It runs from a full clone, since a worktree's `.git` points outside the directory a container mounts. A Hub token is a person's, so CI holds none; this is the one place the repository differs from the toolshed, whose CI promotes with GitHub's own token.

**Guards.** Nothing a pull request contains runs on the lab's machines: CI runs the check script and its unit tests. Nothing that goes public holds a credential: `check.py` refuses a dataset directory or a record that holds the shape of one, in `scripts/push` before anything goes up, in `scripts/record` before anything is written, and in `check` as the backstop; `scripts/push` reads its token from `HF_TOKEN` and never prints it. The record's file list is the Hub's: what a version holds is what the record says, and `check` proves it. `main` requires `check` and no other approval: flyloft merges as the campaign's owner after its critic's review, and a person's pull request is merged by a member other than its builder.

Expand Down
26 changes: 20 additions & 6 deletions scripts/promote
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
#!/usr/bin/env -S python3 -I
"""scripts/promote <name> <version>: tag the commit results/<name>/<version>.txt names as v<version> on
<org>/<name>, then delete the candidate branch. Whoever merges the dataset's pull request runs it after
pulling main, with their own HF_TOKEN; a record that is not on origin/main as it is here is refused, so only
<org>/<name>, give main the version's files, then delete the candidate branch. Whoever merges the dataset's
pull request runs it after pulling main, with their own HF_TOKEN; a record that is not on origin/main as it is here is refused, so only
a reviewed, merged record is published (a guard against mistakes, not misuse: an org token can tag
the Hub directly). Run it from a full clone: a worktree's .git points outside a mounted directory. A tag is never moved: one already naming another commit exits 1;
one already naming the record's commit passes, so a promote run again after it succeeded passes. The last
line of stdout says what happened.
one already naming the record's commit passes, so a promote run again after it succeeded passes. main
becomes a copy of the version, one commit on it, so the Hub's page and viewer show the version last promoted;
the tag stays the version anyone names. The last line of stdout says what happened.
"""
import os
import subprocess
Expand All @@ -14,10 +15,21 @@ from pathlib import Path

sys.path.insert(0, str(Path(__file__).resolve().parent))
import check # noqa: E402
from huggingface_hub import HfApi # noqa: E402
from huggingface_hub import CommitOperationCopy, CommitOperationDelete, HfApi # noqa: E402
from huggingface_hub.errors import HfHubHTTPError # noqa: E402


def follow(api: HfApi, repo: str, tag: str, commit: str, main_head: str) -> None:
"""main gets commit's files in one commit, copied on the Hub, or nothing when it holds them already."""
def tree(rev):
return {f.path: f.blob_id for f in api.list_repo_tree(repo, revision=rev, recursive=True, repo_type="dataset") if hasattr(f, "blob_id")}
have, want = tree(main_head), tree(commit)
ops = [CommitOperationDelete(p) for p in have if p not in want]
ops += [CommitOperationCopy(src_path_in_repo=p, path_in_repo=p, src_revision=commit) for p, b in want.items() if have.get(p) != b]
if ops:
api.create_commit(repo, ops, commit_message=f"main <- {tag}", revision="main", parent_commit=main_head, repo_type="dataset")


def main() -> int:
if len(sys.argv) != 3:
print("usage: scripts/promote <name> <version>", file=sys.stderr)
Expand All @@ -33,12 +45,14 @@ def main() -> int:
rec = check.parse_record(check.ROOT / rel)
repo, tag, commit = f"{check.hub_org()}/{name}", f"v{version}", rec["commit"]
api = HfApi(token=os.environ["HF_TOKEN"])
have = {t.name: t.target_commit for t in api.list_repo_refs(repo, repo_type="dataset").tags}
refs = api.list_repo_refs(repo, repo_type="dataset")
have = {t.name: t.target_commit for t in refs.tags}
if tag in have and have[tag] != commit:
print(f"{repo} {tag} already exists at {have[tag]}, not the record's {commit}; a published tag is never moved", file=sys.stderr)
return 1
if tag not in have:
api.create_tag(repo, tag=tag, revision=commit, repo_type="dataset")
follow(api, repo, tag, commit, next(b.target_commit for b in refs.branches if b.name == "main"))
try:
api.delete_branch(repo, branch=rec["branch"], repo_type="dataset")
except HfHubHTTPError as e:
Expand Down
3 changes: 2 additions & 1 deletion scripts/push
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ def main() -> int:
# an earlier candidate of this tree: replaced, so the record's commit holds exactly this output
api.delete_branch(repo, branch=branch, repo_type="dataset")
api.create_branch(repo, branch=branch, revision="main", repo_type="dataset")
api.upload_folder(folder_path=out, repo_id=repo, repo_type="dataset", revision=branch, ignore_patterns=HIDDEN,
# the branch starts from main, which holds the last version promoted: what this output lacks goes
api.upload_folder(folder_path=out, repo_id=repo, repo_type="dataset", revision=branch, ignore_patterns=HIDDEN, delete_patterns="*",
commit_message=f"{name} {meta['version']}: the output of recipe tree {tree}")
head = api.upload_file(path_or_fileobj=card(dataset / "SKILL.md").encode(), path_in_repo="README.md", repo_id=repo,
repo_type="dataset", revision=branch, commit_message=f"{name} {meta['version']}: the card")
Expand Down
Loading
Loading