Skip to content

Fix Megatron checkpoint save and resume state - #70

Merged
nightlessbaron merged 1 commit into
prodfrom
agent/checkpoint-resume-correctness
Aug 22, 2026
Merged

Fix Megatron checkpoint save and resume state#70
nightlessbaron merged 1 commit into
prodfrom
agent/checkpoint-resume-correctness

Conversation

@nightlessbaron

Copy link
Copy Markdown

What changed

  • Pass Megatron's preprocess_common_state_dict hook when saving non-LoRA checkpoints.
  • Avoid advancing the optimizer scheduler a second time after a normal checkpoint resume.
  • Continue reconstructing scheduler position for finetuning and --no-load-optim, where scheduler state is intentionally not restored.
  • Add four focused regression cases.

Why

A normal resume already restores scheduler state in load_checkpoint; stepping it again double-counts completed iterations. Checkpoint saves also need Megatron's common-state preprocessing to produce the expected distributed checkpoint state.

Validation

  • Focused checkpoint tests: 4 passed.
  • Black: passed.
  • Ruff: passed.
  • py_compile: passed.
  • git diff --check: passed.

Related Megatron optimizer-state fix: LLM360/Megatron-LM#6.

@nightlessbaron
nightlessbaron marked this pull request as ready for review August 21, 2026 22:28
@nightlessbaron
nightlessbaron requested a review from a team August 21, 2026 22:28
@nightlessbaron
nightlessbaron force-pushed the agent/checkpoint-resume-correctness branch from 3471303 to f3b1193 Compare August 21, 2026 22:28
@nightlessbaron
nightlessbaron merged commit 648d8bc into prod Aug 22, 2026
4 checks passed
@nightlessbaron
nightlessbaron deleted the agent/checkpoint-resume-correctness branch August 22, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants