Skip to content

fix(verl): stop admitting batches at the epoch limit - #694

Open
Zhengyang Liu (liuzhengyang699) wants to merge 1 commit into
microsoft:mainfrom
liuzhengyang699:fix/respect-training-epoch-limit
Open

Zhengyang Liu (liuzhengyang699) wants to merge 1 commit into
microsoft:mainfrom
liuzhengyang699:fix/respect-training-epoch-limit

Conversation

@liuzhengyang699

Copy link
Copy Markdown

Fixes #693.

With one epoch, two dataloader batches, and a larger explicit step budget, fit() currently trains on three batches: the iterator starts the next epoch before the outer loop sees that the limit has been exceeded. Async refills can cross the same boundary.

Bound input collection at the dataloader helper. Synchronous training stops before another rollout; asynchronous training allows a partial final refill and finishes groups already admitted. Its completion target is capped to the available groups so the last collection cannot wait for inputs that will never arrive. The explicit training-step limit still takes precedence.

When data exhaustion ends training, enabled final validation and checkpointing use the last successful update, without repeating a scheduled validation/save or counting another optimizer step. The configuration docs describe these limits. Existing checkpoint-resume bookkeeping is unchanged.

Twelve CPU regressions cover one/two epochs, sync and async collection, exact/partial refills, carry-over draining, final saves, skipped collections, and step-limit precedence. The carry-only case also exercises the real _train_step() path through empty DataProto construction. Nine of the initial 11 cases fail on unchanged main.

Validation

  • uv run --locked --no-sync pytest -v --durations=20 tests — 253 passed, 4 skipped.
  • uv run --locked --no-sync pyright — 0 errors, 0 warnings.
  • uv run --locked --no-sync pre-commit run --all-files --show-diff-on-failure — passed; the new test file was also checked explicitly.
  • uv run --locked --no-sync ruff check ., uv run --locked --no-sync ruff format --check ., and uv run --locked --no-sync python scripts/check_headers.py — passed.
  • UV_PROJECT_ENVIRONMENT=/tmp/agent-lightning-epoch-docs uv run --locked --no-sync mkdocs build --strict, uv build --no-sources, and git diff --cached --check — passed.

The development environment used the locked dev extra/group and verl-cpu group; the separate docs environment used the locked docs group. Independent review also exercised a real StatefulDataLoader across two async epochs.

This was found during a follow-up audit with AI assistance. AI assistance was also used for the fix, tests, and review. Validation used CPU tests with worker operations stubbed; no GPU rollout or end-to-end training was run.

@liuzhengyang699
Zhengyang Liu (liuzhengyang699) marked this pull request as ready for review November 7, 2026 06:31
Copilot AI balanced review requested due to automatic review settings November 7, 2026 06:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unconditional optional Torch and VERL imports in the new regression test cause collection failures when the verl-cpu dependency group is absent and should be guarded with pytest.importorskip.

1 open issue
What changed in this PR

Fixes #693 by stopping VERL training inputs at the epoch limit while allowing asynchronous groups already admitted to finish.

Changes:

  • Bounds dataloader reads and async completion targets, preserving the explicit step limit.
  • Runs final validation and checkpointing at the last successful update without duplicating scheduled runs.
  • Documents training limits and adds CPU regressions for exhaustion, partial refills, and carry-over draining.
File Description
tests/​verl/​test_training_epoch_limit.py Adds regression coverage for epoch limits and finalization.
docs/​20-trainer-configuration.md Explains training limits and final partial collections.
agentlightning/​verl/​trainer.py Bounds input collection, drains admitted groups, and finalizes on exhaustion.

Review effort: Balanced

Comment on lines +10 to +14
import numpy as np
import pytest
import torch
from omegaconf import OmegaConf
from verl import DataProto

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Training can admit batches beyond trainer.total_epochs

2 participants