Skip to content

feat(data): subcluster member sampling (sample_members: "subcluster") - #135

Merged
briney merged 1 commit into
mainfrom
feat/subcluster-sampling
Nov 18, 2026
Merged

briney merged 1 commit into
mainfrom
feat/subcluster-sampling

Conversation

@briney

@briney briney commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Summary

Adds two-level ("double") member sampling for the upcoming UniRef and metagenomic training sets. For UniRef: pick a UniRef50 row, then a uniform UniRef90 subcluster inside it, then a uniform UniRef100 member of that subcluster.

  • data.train.<name>.sample_members now accepts false | true | "uniform" | "subcluster". true means "uniform" and behaves exactly as before. Anything else is rejected, including 1.
  • In "subcluster" mode the loader also reads a CSR subcluster_offsets column (list[int], length K+1) next to the flat member_ids / member_sequences columns:
    • it draws k = h1 % K, then j = offsets[k] + h2 % (offsets[k+1] - offsets[k]);
    • h1 is the same hash uniform mode uses, and h2 folds one more constant (_SUBCLUSTER_FOLD) into the splitmix64 _member_index;
    • so the choice depends only on (pass seed, shard, row), and resume, reshard and topology independence hold.
  • The column must exist when the dataset is built. Each row's offsets must rise strictly from 0 to len(member_ids), so empty subclusters aren't allowed; a bad row raises an error naming the shard and row.
  • The flat member columns are unchanged, so representative-only and uniform-member training both still work on the new datasets.
  • sample_members is still left out of the data fingerprint, so a branch_from that only switches the mode keeps the data cursor.
  • Docs: docs/config.md, docs/data.md (cluster-per-row format).

Test plan

  • tests/data/sequence/test_member_sampling.py:
    • subcluster mode picks the singleton subcluster about half the time (uniform mode would give about 1/10);
    • every member of the large subcluster gets drawn;
    • the two draws are jointly uniform;
    • the same member is served at world sizes 1, 2 and 3;
    • true gives the same rows as "uniform";
    • a missing column, bad offsets and unknown modes all raise errors.
  • tests/data/test_config.py: modes parse; "yes", 1 and None are rejected.
  • Fast data suite (tests/data -m "not slow", 501 tests), tests/test_docs.py, ruff, ty check src/.

🤖 Generated with Claude Code

data.train.<name>.sample_members now accepts false | true | "uniform" |
"subcluster" (true == "uniform"). Subcluster mode reads a CSR
subcluster_offsets column (length K+1) next to the flat member columns and
draws a uniform subcluster (k = h1 % K), then a uniform member within it
(h2 folds one more constant into the same splitmix64 member hash), so the
choice stays a pure function of (pass seed, shard, row) and resume,
reshard and topology independence hold. Uniform mode is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@briney
briney merged commit 4cb53ae into main Nov 18, 2026
3 checks passed
@briney
briney deleted the feat/subcluster-sampling branch November 18, 2026 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant