Repository navigation
fix(eval): seed eval masks per sequence, not per batch - #126
Merged
Merged
Conversation
Deterministic eval masking seeded one generator per batch from seed + batch_index. batch_index counts each DataLoader worker's own batches, and eval rows are striped over GPUs and workers, so the mask a sequence got changed with GPU count and worker count (about +-0.01 eval CE between 8- and 4-node runs). Workers also all started at index 0. Each row now gets its own generator, seeded from the eval seed and a blake2b hash of its sequence, and draws noise only over its own real tokens, so its mask doesn't depend on which batch, worker or GPU it lands in or on the batch's padded width. The training (dynamic) path is unchanged. The batch counter, reset_batch_index, _ResettingDataLoader and the loader_cls parameter that served it are removed. New masks differ from the old ones, so eval CE before and after this change is not directly comparable. Closes #122 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #122
Problem
Deterministic eval masking seeded one generator per batch from
seed + batch_index. The batch index counts each DataLoader worker's own batches, and eval rows are split across GPUs and workers. So a sequence's mask depended on the GPU count and the worker count. In the #58 proxy, eval CE for the same weights shifted by about ±0.01 between an 8-node run and a 4-node run. Every worker also started at index 0, so different workers reused the same seeds for different rows. Eval loss is summed over tokens across ranks and every row is scored, so the reduction was never the problem; only the seeding was.Fix
MLMCollatorgives each row its owntorch.Generator. The seed is a blake2b hash of the eval seed and the row's sequence.reset_batch_index,_ResettingDataLoader, and theloader_clsparameter on_build_dataloader.docs/data.md, Deterministic eval masking.Heads-up
Eval masks change with this PR, so eval CE logged before it and after it isn't directly comparable. The difference should be about the same size as the old GPU-count shift. Land it before the mini-prod (#121) starts, so that run's eval CE is consistent throughout.
Tests
test_deterministic_mask_depends_only_on_the_sequence: one sequence gets an identical mask whether it is collated in a full batch, alone across several calls to one collator, or in reversed order withpad_to_multiple_of=8. It fails on main and passes here.test_deterministic_mask_changes_with_seed: changing the seed still changes the masks.test_deterministic_advances_with_batch_index, since that behaviour is gone on purpose.PYTHONPATH=srcso it tests this branch and not the main checkout.tests/data/sequence/test_collate.pyhas 3 E501s that are already on main.🤖 Generated with Claude Code