Skip to content

fix no_shard training convergency and add unittest for no_shard#3754

Open
wplf wants to merge 7 commits intoNVIDIA:mainfrom
wplf:fix-mfsdp-no-shard
Open

fix no_shard training convergency and add unittest for no_shard#3754
wplf wants to merge 7 commits intoNVIDIA:mainfrom
wplf:fix-mfsdp-no-shard

Conversation

@wplf
Copy link
Copy Markdown
Member

@wplf wplf commented Mar 9, 2026

What does this PR do ?

Fixes no_shard convergence by correcting grad_norm and all_reduce logic.

  1. fix no_shard grad_norm calculation
  2. fix no_shard all_reduce grad continue bug
  3. Make no_shard to be compatiable without overlap_param_gather and overlap_grad_reduce
  4. add unittest for no_shard
  5. add assert False for no_shard and init_model_with_meta_device follow fully_shard.py:326

Grad_norm and loss convergency and unittest result is below.

image image image

⚠️ For major changes (either in lines of code or in its impact), please make sure to first share a design doc with the team. If you're unsure what's the best way to do so, contact the @mcore-oncall.

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Code review

Feel free to message or comment the @mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!

All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.

Step 1: Mark PR as "Ready for Review"

  1. When your PR is ready, click Ready for Review.
  2. An oncall reviewer is auto-assigned and expert reviewers are notified based on your changes.
    • Some PRs may jump straight to step 2. This is determined by .github/CODEOWNERS.

⚠️ Only mark as ready once merge-conflicts are resolved and the CI is passing.
Final Review might get declined if these requirements are not fulfilled.

Step 2: Final Review

For PRs that change megatron/core, once all expert reviewers have approved, the Final Review label is applied automatically and final reviewers are assigned.

For PRs outside megatron/core, this step is skipped.

Step 3: Approved

Once all required reviewers have approved, the Approved label is applied automatically.

Merge

Any member of mcore-engineers will be able to merge your PR.

For MRs into `dev` branch The proposed review process for `dev` branch is under active discussion.

MRs are mergable after one approval by either eharper@nvidia.com or zijiey@nvidia.com.

@wplf wplf requested review from a team as code owners March 9, 2026 15:54
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented Mar 9, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci marked this pull request as draft March 9, 2026 15:55
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 9, 2026

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@wplf wplf marked this pull request as ready for review March 9, 2026 15:55
@svcnvidia-nemo-ci svcnvidia-nemo-ci requested a review from a team March 9, 2026 15:55
@wplf wplf force-pushed the fix-mfsdp-no-shard branch from 920e8d0 to d2fde24 Compare March 9, 2026 16:09
Copy link
Copy Markdown
Member

@cspades cspades left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

wplf added 2 commits March 22, 2026 07:32
add unittest for no_shard and add empty cache to avoid OOM

add meta_device_check for no_shard following fully_shard.py:326

Signed-off-by: jinliangl <jinliangl@nvidia.com>
@wplf wplf reopened this Mar 22, 2026
@wplf wplf enabled auto-merge March 31, 2026 07:13
@wplf wplf disabled auto-merge March 31, 2026 07:22
@wplf wplf enabled auto-merge March 31, 2026 07:22
@svcnvidia-nemo-ci svcnvidia-nemo-ci added Approved All necessary approvals have been made and removed Final Review PR is in the "final review" stage labels Apr 14, 2026
@Phlip79 Phlip79 removed the request for review from a team April 15, 2026 16:18
@yaox12 yaox12 disabled auto-merge April 20, 2026 04:33
@yaox12 yaox12 enabled auto-merge April 20, 2026 04:33
@yaox12
Copy link
Copy Markdown
Member

yaox12 commented Apr 20, 2026

/ok to test e6a1239

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved All necessary approvals have been made module: megatron-fsdp

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants