Repository navigation
feat(data): subcluster member sampling (sample_members: "subcluster") - #135
Merged
Merged
Conversation
data.train.<name>.sample_members now accepts false | true | "uniform" | "subcluster" (true == "uniform"). Subcluster mode reads a CSR subcluster_offsets column (length K+1) next to the flat member columns and draws a uniform subcluster (k = h1 % K), then a uniform member within it (h2 folds one more constant into the same splitmix64 member hash), so the choice stays a pure function of (pass seed, shard, row) and resume, reshard and topology independence hold. Uniform mode is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds two-level ("double") member sampling for the upcoming UniRef and metagenomic training sets. For UniRef: pick a UniRef50 row, then a uniform UniRef90 subcluster inside it, then a uniform UniRef100 member of that subcluster.
data.train.<name>.sample_membersnow acceptsfalse | true | "uniform" | "subcluster".truemeans"uniform"and behaves exactly as before. Anything else is rejected, including1."subcluster"mode the loader also reads a CSRsubcluster_offsetscolumn (list[int], length K+1) next to the flatmember_ids/member_sequencescolumns:k = h1 % K, thenj = offsets[k] + h2 % (offsets[k+1] - offsets[k]);h1is the same hash uniform mode uses, andh2folds one more constant (_SUBCLUSTER_FOLD) into the splitmix64_member_index;len(member_ids), so empty subclusters aren't allowed; a bad row raises an error naming the shard and row.sample_membersis still left out of the data fingerprint, so abranch_fromthat only switches the mode keeps the data cursor.docs/config.md,docs/data.md(cluster-per-row format).Test plan
tests/data/sequence/test_member_sampling.py:truegives the same rows as"uniform";tests/data/test_config.py: modes parse;"yes",1andNoneare rejected.tests/data -m "not slow", 501 tests),tests/test_docs.py, ruff,ty check src/.🤖 Generated with Claude Code