Repository navigation
config(scaling): set the 170M mini-prod horizon and LR - #125
Merged
Merged
Conversation
configs/scaling.yaml now carries the 170M mini-prod recipe (#121): 2k warmup + 998k stable + 200k decay (20% of stable) = 1.2M steps, LR 0.002 from the #55 LR-vs-length power law, 8 nodes for 170M (65536 tokens per GPU, no accumulation). The stable mix stays at 0.05 rows of uniref70. The #58 Pending markers are resolved: the decay is 20% of stable, wsd_linear, to min_lr 0.0, and docs/data.md gives the anneal mix (uniref70 25% of tokens with member sampling, antibody 1.5%, deepclust70 the rest). The production (~10M-step) LR and horizon stay Pending on #70; the reference-run ranges and the antibody anneal set now wait on #121. base.yaml keeps 0.0063 as the 24k funnel base LR that sweeps and short runs start from. Refs #70 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sets
configs/scaling.yamlto the 170M mini-prod recipe (#121) and resolves the #58 Pending markers now that #58 is closed. The production (~10M-step) LR and horizon stay Pending on #70.max_steps. The decay is meant to run as a branch offcheckpoint-1000000with the anneal mix; rendered as-is, the run decays on the stable mix.tokens_per_stepis 2²².docs/mup-lr.md,docs/reproduce.md): 20% of stable,wsd_linear, down tomin_lr: 0.0.docs/data.md): uniref70 25% of tokens withsample_members: true, antibody 1.5%, deepclust70 the rest. The table gives the 1/35 proxy row fractions.tests/test_docs.pyis updated to match.base.yamlkeeps 0.0063. That is the 24k funnel base LR that sweeps and short runs start from;docs/mup-lr.mdsays so.Test plan
pytest tests/test_docs.py tests/test_scaling_config.py tests/slurm: 155 passed. The docs and scaling tests were rerun after the final edits: 10 passed.test_every_preset_plans_with_its_batch_tokens_cappasses with 170M on 8 nodes.Refs #70
🤖 Generated with Claude Code