Change CI to two fixed seed runs

.claude/skills/l0-test/SKILL.md

@@ -10,7 +10,7 @@ Runs the same checks that gate a pull request in `.github/workflows/cicd-main.ym
 
 1. **Lint** (host) — `pre-commit run --all-files` (ruff check, ruff format, mypy, copyright/EOF/whitespace, no-underscore-md).
 2. **L0 CPU tests** (container) — distributed muon utils at `nproc=4,8`, plus `test_scalar_optimizers.py` and `test_procrustes_step.py`.
-3. **L0 GPU tests** (container) — every `tests/test_*.py` (excluding `*_cpu.py`) twice — once with a random seed, once with `--seed=42` — then the convergence `tests/convergence/*_test.py` set.
+3. **L0 GPU tests** (container) — every `tests/test_*.py` (excluding `*_cpu.py`) twice — once with `--seed=42`, once with `--seed=0xdeadbeef` — then the convergence `tests/convergence/*_test.py` set.
 
 Stages 2 and 3 share a single NGC container session: lint runs on the host, then a single `docker run` against the image declared in `docker/Dockerfile.ci` (the `FROM` line — do **not** hard-code a tag, so the skill picks up version bumps automatically) with `--gpus all` performs both CPU and GPU passes. **All test execution happens inside the container; never run tests on the host.** Reasons:
 

CLAUDE.md

@@ -61,7 +61,7 @@ torchrun --nproc_per_node=<n> --no-python python tests/test_distributed_muon_uti
 CI test suites are encoded as shell scripts in `tests/ci/`:
 
 - `L0_Tests_CPU.sh` — all `tests/test_distributed_*_cpu.py` at 8 and 4 ranks (rekls also at 1 rank) + a couple of CPU-only tests.
-- `L0_Tests_GPU.sh` — every `tests/test_*.py` (excluding `*_cpu.py`) on `--device=cuda`, run twice (random seed, then `--seed=42`), plus the `tests/convergence/*_test.py` set.
+- `L0_Tests_GPU.sh` — every `tests/test_*.py` (excluding `*_cpu.py`) on `--device=cuda`, run twice (`--seed=42`, then `--seed=0xdeadbeef`), plus the `tests/convergence/*_test.py` set at `--seed=0xace`.
 - `L1_Tests_GPU.sh` — convergence tests at `--seed=77` plus the `moe_c4_convergence.py` runs against muon/soap with a loss target.
 
 Coverage is collected with `coverage run -p --source=emerging_optimizers ...` and aggregated; configuration in `pyproject.toml` (`[tool.coverage.*]`) excludes Triton-decorated code, `if TYPE_CHECKING`, and abstract methods.
@@ -70,7 +70,7 @@ Run a single test class/method using absl conventions, e.g. `python tests/test_o
 
 ### Authoring tests
 
-- **Every test file must define `--device` and `--seed` flags.** CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice (random seed, then `--seed=42` / `--seed=77`). Tests without these flags will be invoked with `--device=...` and fail at flag parsing. Match the existing pattern: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`.
+- **Every test file must define `--device` and `--seed` flags.** CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice with fixed seeds (`--seed=42` and `--seed=0xdeadbeef`; convergence tests use `--seed=0xace` / `--seed=77`). Tests without these flags will be invoked with `--device=...` and fail at flag parsing. Match the existing pattern: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`.
 - **Don't override `torch.testing.assert_close`'s `msg`; append to it.** The default message contains the diff/atol/rtol summary, which is invaluable when debugging CI failures. Pass `msg=` as a callable that takes the default and returns a wrapped string — never a bare string that replaces the default. Canonical pattern (from the [PyTorch 2.11 docs](https://docs.pytorch.org/docs/2.11/testing.html#module-torch.testing)):
 
   ```python

CONTRIBUTING.md

@@ -87,7 +87,7 @@ All tests should be placed under [tests](tests). We aim for 100% test coverage f
 
 We use [abseil-py](https://github.com/abseil/abseil-py/tree/main) **testing** because it is easier to launch multi process than alternatives.
 
-Every test file must define `--device` and `--seed` flags — CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice (random seed, then a fixed seed). A test without these flags will fail at flag parsing in CI. Match the pattern used in existing test files: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`.
+Every test file must define `--device` and `--seed` flags — CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice with two different fixed seeds. A test without these flags will fail at flag parsing in CI. Match the pattern used in existing test files: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`.
 
 When using `torch.testing.assert_close`, do not override the `msg` argument with a bare string — that replaces the default diff/atol/rtol summary, which is what makes CI failures debuggable. Pass `msg=` as a callable that takes the default message and returns one with your context appended. Canonical pattern (from the [PyTorch 2.11 docs](https://docs.pytorch.org/docs/2.11/testing.html#module-torch.testing)):
 

tests/ci/L0_Tests_GPU.sh

@@ -19,21 +19,17 @@ export TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=0
 mkdir -p test-results/tests/convergence
 
 error=0
-for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do
-    echo "Running $test with random seed"
-    report_name="test-results/${test}.xml"
-    coverage run -p --source=emerging_optimizers $test --device=cuda -v -2 --xml_output_file="$report_name" || error=1
-done
-
-fix_seed=42
-for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do
-    echo "Running $test with fixed seed $fix_seed"
-    report_name="test-results/${test}_seed${fix_seed}.xml"
-    coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1
+for fix_seed in 42 0xdeadbeef; do
+    for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do
+        echo "Running $test with fixed seed $fix_seed"
+        report_name="test-results/${test}_seed${fix_seed}.xml"
+        coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1
+    done
 done
 
+fix_seed=0xace
 for test in `find tests/convergence -type f -name '*_test.py'`; do
-    echo "Running $test with fixed seed"
+    echo "Running $test with fixed seed $fix_seed"
     report_name="test-results/${test}_seed${fix_seed}.xml"
     coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1
 done