diff --git a/.claude/skills/l0-test/SKILL.md b/.claude/skills/l0-test/SKILL.md index b235be4f..8a78a260 100644 --- a/.claude/skills/l0-test/SKILL.md +++ b/.claude/skills/l0-test/SKILL.md @@ -10,7 +10,7 @@ Runs the same checks that gate a pull request in `.github/workflows/cicd-main.ym 1. **Lint** (host) — `pre-commit run --all-files` (ruff check, ruff format, mypy, copyright/EOF/whitespace, no-underscore-md). 2. **L0 CPU tests** (container) — distributed muon utils at `nproc=4,8`, plus `test_scalar_optimizers.py` and `test_procrustes_step.py`. -3. **L0 GPU tests** (container) — every `tests/test_*.py` (excluding `*_cpu.py`) twice — once with a random seed, once with `--seed=42` — then the convergence `tests/convergence/*_test.py` set. +3. **L0 GPU tests** (container) — every `tests/test_*.py` (excluding `*_cpu.py`) twice — once with `--seed=42`, once with `--seed=0xdeadbeef` — then the convergence `tests/convergence/*_test.py` set. Stages 2 and 3 share a single NGC container session: lint runs on the host, then a single `docker run` against the image declared in `docker/Dockerfile.ci` (the `FROM` line — do **not** hard-code a tag, so the skill picks up version bumps automatically) with `--gpus all` performs both CPU and GPU passes. **All test execution happens inside the container; never run tests on the host.** Reasons: diff --git a/CLAUDE.md b/CLAUDE.md index 36fd7c3b..19e6ceba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,7 +61,7 @@ torchrun --nproc_per_node= --no-python python tests/test_distributed_muon_uti CI test suites are encoded as shell scripts in `tests/ci/`: - `L0_Tests_CPU.sh` — all `tests/test_distributed_*_cpu.py` at 8 and 4 ranks (rekls also at 1 rank) + a couple of CPU-only tests. -- `L0_Tests_GPU.sh` — every `tests/test_*.py` (excluding `*_cpu.py`) on `--device=cuda`, run twice (random seed, then `--seed=42`), plus the `tests/convergence/*_test.py` set. +- `L0_Tests_GPU.sh` — every `tests/test_*.py` (excluding `*_cpu.py`) on `--device=cuda`, run twice (`--seed=42`, then `--seed=0xdeadbeef`), plus the `tests/convergence/*_test.py` set at `--seed=0xace`. - `L1_Tests_GPU.sh` — convergence tests at `--seed=77` plus the `moe_c4_convergence.py` runs against muon/soap with a loss target. Coverage is collected with `coverage run -p --source=emerging_optimizers ...` and aggregated; configuration in `pyproject.toml` (`[tool.coverage.*]`) excludes Triton-decorated code, `if TYPE_CHECKING`, and abstract methods. @@ -70,7 +70,7 @@ Run a single test class/method using absl conventions, e.g. `python tests/test_o ### Authoring tests -- **Every test file must define `--device` and `--seed` flags.** CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice (random seed, then `--seed=42` / `--seed=77`). Tests without these flags will be invoked with `--device=...` and fail at flag parsing. Match the existing pattern: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`. +- **Every test file must define `--device` and `--seed` flags.** CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice with fixed seeds (`--seed=42` and `--seed=0xdeadbeef`; convergence tests use `--seed=0xace` / `--seed=77`). Tests without these flags will be invoked with `--device=...` and fail at flag parsing. Match the existing pattern: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`. - **Don't override `torch.testing.assert_close`'s `msg`; append to it.** The default message contains the diff/atol/rtol summary, which is invaluable when debugging CI failures. Pass `msg=` as a callable that takes the default and returns a wrapped string — never a bare string that replaces the default. Canonical pattern (from the [PyTorch 2.11 docs](https://docs.pytorch.org/docs/2.11/testing.html#module-torch.testing)): ```python diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 955b5fcc..8349e5ad 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -87,7 +87,7 @@ All tests should be placed under [tests](tests). We aim for 100% test coverage f We use [abseil-py](https://github.com/abseil/abseil-py/tree/main) **testing** because it is easier to launch multi process than alternatives. -Every test file must define `--device` and `--seed` flags — CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice (random seed, then a fixed seed). A test without these flags will fail at flag parsing in CI. Match the pattern used in existing test files: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`. +Every test file must define `--device` and `--seed` flags — CI invokes every test under both `--device=cpu` and `--device=cuda`, and runs each GPU test twice with two different fixed seeds. A test without these flags will fail at flag parsing in CI. Match the pattern used in existing test files: `flags.DEFINE_enum("device", "cpu", ["cpu", "cuda"], ...)`, `flags.DEFINE_integer("seed", None, ...)`, and a `setUpModule` that seeds when `FLAGS.seed is not None`. When using `torch.testing.assert_close`, do not override the `msg` argument with a bare string — that replaces the default diff/atol/rtol summary, which is what makes CI failures debuggable. Pass `msg=` as a callable that takes the default message and returns one with your context appended. Canonical pattern (from the [PyTorch 2.11 docs](https://docs.pytorch.org/docs/2.11/testing.html#module-torch.testing)): diff --git a/tests/ci/L0_Tests_GPU.sh b/tests/ci/L0_Tests_GPU.sh index 93863bb7..bb206e20 100644 --- a/tests/ci/L0_Tests_GPU.sh +++ b/tests/ci/L0_Tests_GPU.sh @@ -19,21 +19,17 @@ export TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=0 mkdir -p test-results/tests/convergence error=0 -for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do - echo "Running $test with random seed" - report_name="test-results/${test}.xml" - coverage run -p --source=emerging_optimizers $test --device=cuda -v -2 --xml_output_file="$report_name" || error=1 -done - -fix_seed=42 -for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do - echo "Running $test with fixed seed $fix_seed" - report_name="test-results/${test}_seed${fix_seed}.xml" - coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1 +for fix_seed in 42 0xdeadbeef; do + for test in `find tests -type f -name 'test_*' ! -name '*_cpu.py'`; do + echo "Running $test with fixed seed $fix_seed" + report_name="test-results/${test}_seed${fix_seed}.xml" + coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1 + done done +fix_seed=0xace for test in `find tests/convergence -type f -name '*_test.py'`; do - echo "Running $test with fixed seed" + echo "Running $test with fixed seed $fix_seed" report_name="test-results/${test}_seed${fix_seed}.xml" coverage run -p --source=emerging_optimizers $test --device=cuda --seed="$fix_seed" -v -2 --xml_output_file="$report_name" || error=1 done