Skip to content

Consolidate determinism into configure_determinism and clarify fused AdamW behavior - #13

Merged
Sam-v6 merged 4 commits into
mainfrom
codex/make-transformer-training-process-deterministic
Dec 31, 2025
Merged

Sam-v6 merged 4 commits into
mainfrom
codex/make-transformer-training-process-deterministic

Conversation

@Sam-v6

@Sam-v6 Sam-v6 commented Dec 30, 2025

Copy link
Copy Markdown
Owner

Motivation

  • Centralize seeding and determinism knobs so reproducible runs and profiling-driven deterministic runs use a single code path.
  • Ensure the same seeded torch.Generator is produced and used for deterministic DataLoader shuffling.
  • Allow strict determinism only when requested (e.g., during profiling) to avoid unnecessary throughput loss in normal runs.
  • Make the reason to disable fused optimizer kernels in deterministic mode explicit.

Description

  • Added configure_determinism(deterministic: bool, seed: int = 42) which sets PYTHONHASHSEED, Python/NumPy/PyTorch seeds, creates and returns a seeded torch.Generator, and toggles cuDNN/TF32/deterministic algorithm flags based on deterministic.
  • Replaced the old seed setup in run_trial with g = configure_determinism(deterministic, seed=42) where deterministic is driven by args.profile.
  • Passed the returned generator g into DataLoader(..., generator=g) and set the optimizer to fused=not deterministic for AdamW, with an updated comment explaining the non-deterministic reductions.
  • Clarified inline comments around fused AdamW determinism to address earlier reviewer questioning.

Testing

  • No automated tests were executed for these changes.
  • No CI runs were invoked as part of this PR.

Codex Task

@Sam-v6
Sam-v6 merged commit 8e4dbc0 into main Dec 31, 2025
1 check passed
@Sam-v6
Sam-v6 deleted the codex/make-transformer-training-process-deterministic branch December 31, 2025 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant