Skip to content

fix(eval): seed eval masks per sequence, not per batch - #126

Merged
briney merged 2 commits into
mainfrom
fix/eval-mask-world-size
Nov 13, 2026
Merged

briney merged 2 commits into
mainfrom
fix/eval-mask-world-size

Conversation

@briney

@briney briney commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Closes #122

Problem

Deterministic eval masking seeded one generator per batch from seed + batch_index. The batch index counts each DataLoader worker's own batches, and eval rows are split across GPUs and workers. So a sequence's mask depended on the GPU count and the worker count. In the #58 proxy, eval CE for the same weights shifted by about ±0.01 between an 8-node run and a 4-node run. Every worker also started at index 0, so different workers reused the same seeds for different rows. Eval loss is summed over tokens across ranks and every row is scored, so the reduction was never the problem; only the seeding was.

Fix

  • In deterministic mode, MLMCollator gives each row its own torch.Generator. The seed is a blake2b hash of the eval seed and the row's sequence.
  • Each row draws its Gumbel noise only over its own real tokens. Padding width therefore can't shift the draw.
  • The BERT 80/10/10 replacement step uses the same per-row generator.
  • The training (dynamic) path is unchanged: it uses the global RNG in the same order, so training masks and resume behaviour are untouched.
  • Removed, since nothing else used them: the batch counter, reset_batch_index, _ResettingDataLoader, and the loader_cls parameter on _build_dataloader.
  • Updated the docstrings and docs/data.md, Deterministic eval masking.

Heads-up

Eval masks change with this PR, so eval CE logged before it and after it isn't directly comparable. The difference should be about the same size as the old GPU-count shift. Land it before the mini-prod (#121) starts, so that run's eval CE is consistent throughout.

Tests

  • New test_deterministic_mask_depends_only_on_the_sequence: one sequence gets an identical mask whether it is collated in a full batch, alone across several calls to one collator, or in reversed order with pad_to_multiple_of=8. It fails on main and passes here.
  • New test_deterministic_mask_changes_with_seed: changing the seed still changes the masks.
  • Removed test_deterministic_advances_with_batch_index, since that behaviour is gone on purpose.
  • Fast suite: 2788 passed, 500 skipped, run with PYTHONPATH=src so it tests this branch and not the main checkout.
  • ruff is clean on the changed source files. tests/data/sequence/test_collate.py has 3 E501s that are already on main.

🤖 Generated with Claude Code

briney and others added 2 commits November 13, 2026 10:29
Deterministic eval masking seeded one generator per batch from
seed + batch_index. batch_index counts each DataLoader worker's own
batches, and eval rows are striped over GPUs and workers, so the mask a
sequence got changed with GPU count and worker count (about +-0.01 eval
CE between 8- and 4-node runs). Workers also all started at index 0.

Each row now gets its own generator, seeded from the eval seed and a
blake2b hash of its sequence, and draws noise only over its own real
tokens, so its mask doesn't depend on which batch, worker or GPU it
lands in or on the batch's padded width. The training (dynamic) path is
unchanged. The batch counter, reset_batch_index, _ResettingDataLoader
and the loader_cls parameter that served it are removed.

New masks differ from the old ones, so eval CE before and after this
change is not directly comparable.

Closes #122

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@briney
briney merged commit 704e01f into main Nov 13, 2026
3 checks passed
@briney
briney deleted the fix/eval-mask-world-size branch November 13, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GPU-count-independent eval masks

1 participant