Skip to content

From NVIDIA Megatron-LM for visibility #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
RaymondLi0 wants to merge 6,041 commits into
base: multi-query-attention
Choose a base branch
from
Open

From NVIDIA Megatron-LM for visibility #18

RaymondLi0 wants to merge 6,041 commits into from

Conversation

@RaymondLi0
Copy link
Collaborator

@RaymondLi0 RaymondLi0 commented Jan 24, 2023

No description provided.

@RaymondLi0 RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12
@RaymondLi0 RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12
jalbericiola and others added 28 commits December 5, 2025 20:38
@jalbericiola @jon-barker
fix for sequence packing plus sequence parallel: padding the sequence…
416687f 
… to a multiple of TP (#2574)

Co-authored-by: Jon Barker <19699370+jon-barker@users.noreply.github.com>
@ko3n1g
Revert "Make check_large_grads non-fatal (#2307)"
32ebde7 
This reverts commit bcf07a2.
@sidsingh-nvidia
Torch symmetric - new latency optimized NVLS communication kernels fo…
972d9b6 
…r sequence parallelism (#1997)
@BestJuly @chtruong814
[Main] Support MTP packed-seq in main branch (#2173)
8c4df6b 
Signed-off-by: Li Tao <lit@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@deepakn94
Various quality-of-life improvements in training loop (#2580)
8a5f379 
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
@deepakn94 @rogerwaleffe
Support TP greater than num_kv_heads by supporting QKV activation sub…
f7dfb99 
…-sharding (#2565)

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Roger Waleffe <rwaleffe@nvidia.com>
@santhnm2
Fix FA3 import (#2577)
dfc3913 
Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>
@cms42 @Li-dongyang
@ko3n1g @ericharper
Fix runaway Etpt in straggler detector by resetting FLOPs accumulator (…
5232820 
…#1755)

Co-authored-by: Li Ruixiao <cgruixiao@outlook.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
@AAnoosheh
Rename TensorRT Model Optimizer to Model Optimizer (#2373)
4cf809c 
Signed-off-by: Asha Anoosheh <aanoosheh@nvidia.com>
@ko3n1g
Reapply "Make check_large_grads non-fatal (#2307)"
e2199af 
This reverts commit 32ebde7.
@Victarry
Fix aux loss scale when CP is enabled. (#2237)
f6e0d42
@BestJuly @ko3n1g
Save memory using main_param for moe in param_l2_norm (#2249)
01aad93 
Signed-off-by: lit <lit@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
@deepakn94 @venmugil @ericharper
Changes to support latent MoEs (#2296)
b51db3e 
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
Co-authored-by: Venmugil Elango <velango@nvidia.com>
Co-authored-by: Venmugil Elango <498703+venmugil@users.noreply.github.com>
Co-authored-by: Eric Harper <eharper@nvidia.com>
@pablo-garay
update API compat check baseline to b51db3e (#2588)
03b6d31 
Signed-off-by: Pablo Garay <pagaray@nvidia.com>
@tdene
Fix invalid argument failing tests on main (#2589)
f4957d1
@yobibyte
Add openmathinstruct config. (#2586)
2bc35c5
@yobibyte
Move model configs to github. (#2587)
e2cf81c
@iuyo5678 @TencentXiaowei
fix: Assign tokenizer to Encoder.tokenizer in legacy mode (#2498)
8d18afd 
Co-authored-by: drakezhang <drakezhang@tencent.com>
@wplf @yaox12 @ko3n1g
Delete redundant import in yaml_arguments.py (#2139)
c21bf6e 
Co-authored-by: Xin Yao <xiny@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
@CodersAcademy006 @yaox12
Fix world size mismatch causing distributed init deadlock (Issue #2458)…
dfb78dc 
… (#2571)

Co-authored-by: Xin Yao <xiny@nvidia.com>
@tdene
Improve performance of request_metadata logic (#2378)
4a4f23a
@JungHoyoun @yaox12
Fix broken Table of Contents links in README.md (#1954)
7b11553 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@gautham-kollu @yaox12
Add minor log update (#2080)
9fba363 
Signed-off-by: gkollu <gkollu@nvidia.com>
Co-authored-by: Xin Yao <xiny@nvidia.com>
@janbernloehr @yaox12
Fix link to NeMo performance summary documentation (#2190)
bd32927 
Co-authored-by: Xin Yao <xiny@nvidia.com>
@wdykas
Prep for refit (#2590)
ef12f16 
Co-authored-by: root <root@gpu-h100-0435.cm.cluster>
Co-authored-by: root <root@gpu-h100-0012.cm.cluster>
Co-authored-by: root <root@gpu-h100-0426.cm.cluster>
Co-authored-by: root <root@gpu-h100-0188.cm.cluster>
Co-authored-by: root <root@gpu-h100-0013.cm.cluster>
Co-authored-by: root <root@gpu-h100-0032.cm.cluster>
Co-authored-by: root <root@gpu-h100-0240.cm.cluster>
Co-authored-by: root <root@gpu-h100-0089.cm.cluster>
@pablo-garay
feat: API compat: ignore ParameterMovedBreakage for __init__ methods (#…
a2aafe3 
…2595)

Signed-off-by: Pablo Garay <pagaray@nvidia.com>
@ko3n1g
Revert "Prep for refit (#2590)"
5c54bb6 
This reverts commit ef12f16.
@vignesh1507 @yaox12
Fix NameError in pretrain_retro.py (add import_module), remove unused… (
be4baad 
#2084)

Co-authored-by: Xin Yao <xiny@nvidia.com>
ko3n1g and others added 30 commits January 15, 2026 23:13
@ko3n1g
ci: Fix GB200 change (#2969)
4fa27a1 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@tdene
Update golden values for reshard test (#2971)
a0908fc
@ko3n1g
chore: Update golden values (#2973)
ad2ab2c 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci(hotfix): _TENSORBOARD_PATH
99ea287 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@jkamalu @nvda-bnorick
Enable phase transition iterations (#2938)
4da7da9 
Co-authored-by: Brandon Norick <bnorick@nvidia.com>
@ko3n1g
gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_no_a2a_ovlp_fine_grained_of…
710ff97 
…floading

Signed-off-by: oliver könig <okoenig@nvidia.com>
@jon-barker
add missing import in rl_utils.py (#2915)
048825c
@ChenhanYu
Pass through --trust-remote-code and add this to all Nemotron model c…
7f85843 
…onfigs (#2939)
@wdykas @ericharper
Cuda 13 UVM (#2957)
15ea236 
Co-authored-by: Eric Harper <eharper@nvidia.com>
@Wohox @Phlip79
[Main] Partial CUDA Graph support for EP Overlap (#2184)
1501c0f 
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@duncanriach @ericharper @Phlip79
Add sequence packing support for hybrid model (#2913)
03c0727 
Co-authored-by: Eric Harper <eharper@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
@xuwchen
docs(megatron-fsdp): add Megatron-FSDP user guide (#2396)
55103f9
@kunlunl @Phlip79
DeepSeek V3.2 support (#2440)
382eeea 
Signed-off-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>
Co-authored-by: Philip Petrakian <pgpetrak@gmail.com>
@dimapihtar
fully remove zarr support (#2944)
faa6037 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@ko3n1g
chore: Standardize setuptools version (#2975)
b213e10 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Run functional tests on main (#2983)
5df1954 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci(fix): CI_COMMIT_BRANCH on forks (#2982)
b96cadb 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@yuzhongw-nvidia @ko3n1g
[main] feat(moe): Support gated delta net for Qwen3-Next (1/4) (#1989)
20d66d5 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Co-authored-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Add more gb200 nightly tests (#2981)
92f052b 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci(hotfix): gpt3_mcore_te_tp1_pp2_resume_torch_dist_rope_embeddings
4ebd1ad 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci(hotfix): move test to nightly
b50da25 
Signed-off-by: oliver könig <okoenig@nvidia.com>
@yuzhongw-nvidia @jstjohn @deepakn94
[main] feat(moe): Support apply wd to qk layernorm for Qwen3-Next (4/…
622a06a 
…4) (#2753)

Signed-off-by: John St. John <jstjohn@nvidia.com>
Co-authored-by: John St. John <jstjohn@nvidia.com>
Co-authored-by: Deepak Narayanan <2724038+deepakn94@users.noreply.github.com>
@asolergi-nv @ko3n1g
Set token_dtype_code init value in GPTDatasetConfig to fix CI (#2912
b305422 
)

Co-authored-by: oliver könig <okoenig@nvidia.com>
@hexinw-nvidia @rhewett-nv @chtruong814
Re-submit "Various fixes to in-job restarter and better time accounti…
c40b6ea 
…ng of startup operations" (#2954)

Co-authored-by: Russell Hewett <rhewett@nvidia.com>
Co-authored-by: Charlie Truong <chtruong@nvidia.com>
@Phlip79
Use slack-sdk in a different manner (#2950)
100e11e
@lmcafee-nvidia
Inference | Move assert active_request_count > 0. (#2958)
b705148
@parthmannan @xiaoyao0115 @kunlunl
Hybrid Context Parallel Feature (#2282)
98d8c56 
Signed-off-by: tailaim <tailaim@nvidia.com>
Signed-off-by: Parth Mannan <pmannan@nvidia.com>
Co-authored-by: Mcore Bot <mcore-bot@nvidia.com>
Co-authored-by: tailaim <tailaim@nvidia.com>
Co-authored-by: kunlunl <kunlunl@nvidia.com>
Co-authored-by: Kunlun Li <94586211+kunlunl@users.noreply.github.com>
@yuzhongw-nvidia
[main] ci(moe): Add --apply-wd-to-qk-layernorm flag to the gdn test…
829e461 
… case (#2995)
@ko3n1g
ci: Disable step time on `gpt3_moe_mcore_te_tp2_pp2_ep4_etp1_no_mtp_n… (
a191791 
#2991)

Signed-off-by: oliver könig <okoenig@nvidia.com>
@ko3n1g
ci: Fix workflows on main (#2990)
b49d810 
Signed-off-by: oliver könig <okoenig@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.