Skip to content

config(scaling): set the 170M mini-prod horizon and LR - #125

Merged
briney merged 2 commits into
mainfrom
config/mini-prod-horizon
Nov 13, 2026
Merged

briney merged 2 commits into
mainfrom
config/mini-prod-horizon

Conversation

@briney

@briney briney commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Summary

Sets configs/scaling.yaml to the 170M mini-prod recipe (#121) and resolves the #58 Pending markers now that #58 is closed. The production (~10M-step) LR and horizon stay Pending on #70.

  • Horizon: 2k warmup + 998k stable + 200k decay (20% of stable) = 1.2M max_steps. The decay is meant to run as a branch off checkpoint-1000000 with the anneal mix; rendered as-is, the run decays on the stable mix.
  • LR 0.002: the Long 170M extrapolation campaign (LR vs training length) #55 power law gives 0.0019–0.0021 at 1.2M steps.
  • Nodes: 170M runs on 8 nodes, with 65536 tokens per GPU and no gradient accumulation. A comment notes that node counts must be powers of two, because tokens_per_step is 2²².
  • Stable-phase mix: unchanged at uniref70 0.05 rows, about 12% of tokens.
  • Decay recipe (docs/mup-lr.md, docs/reproduce.md): 20% of stable, wsd_linear, down to min_lr: 0.0.
  • Anneal mix (docs/data.md): uniref70 25% of tokens with sample_members: true, antibody 1.5%, deepclust70 the rest. The table gives the 1/35 proxy row fractions.
  • Pending markers:
  • Not changed: base.yaml keeps 0.0063. That is the 24k funnel base LR that sweeps and short runs start from; docs/mup-lr.md says so.

Test plan

  • pytest tests/test_docs.py tests/test_scaling_config.py tests/slurm: 155 passed. The docs and scaling tests were rerun after the final edits: 10 passed.
  • test_every_preset_plans_with_its_batch_tokens_cap passes with 170M on 8 nodes.

Refs #70

🤖 Generated with Claude Code

configs/scaling.yaml now carries the 170M mini-prod recipe (#121):
2k warmup + 998k stable + 200k decay (20% of stable) = 1.2M steps,
LR 0.002 from the #55 LR-vs-length power law, 8 nodes for 170M
(65536 tokens per GPU, no accumulation). The stable mix stays at
0.05 rows of uniref70.

The #58 Pending markers are resolved: the decay is 20% of stable,
wsd_linear, to min_lr 0.0, and docs/data.md gives the anneal mix
(uniref70 25% of tokens with member sampling, antibody 1.5%,
deepclust70 the rest). The production (~10M-step) LR and horizon
stay Pending on #70; the reference-run ranges and the antibody anneal
set now wait on #121.

base.yaml keeps 0.0063 as the 24k funnel base LR that sweeps and
short runs start from.

Refs #70

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@briney
briney merged commit a90e768 into main Nov 13, 2026
3 checks passed
@briney
briney deleted the config/mini-prod-horizon branch November 13, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant