[ROADMAP][2026 Q3] Megatron Core MoE Roadmap

[buptzyb]

Description

The focus for Megatron Core MoE is to provide comprehensive support for latest MoE architectures, advanced parallelism strategies, and performance optimizations for Blackwell. This is a tentative roadmap and subject to change.

• ⚠️ This Roadmap is based on the dev branch; please see the details in its README.
• 🎉 Our technical report for Megatron Core MoE is published in https://arxiv.org/abs/2603.07685
• Previous Roadmaps
  • [ROADMAP][2026 Q2] Megatron Core MoE Roadmap #4815
  • [ROADMAP][Updated on April 07] Megatron Core MoE Roadmap #1729

---

Megatron MoE Supported Features

Model Support

• DeepSeek
  • ✅ DeepSeek-V2
  • ✅ DeepSeek-V3, including MTP
  • ✅ DeepSeek-V3.2
  • ✅ DeepSeek-V4
• Qwen
  • ✅ Qwen2-57B-A14B
  • ✅ Qwen3-235B-A22B
  • ✅ Qwen3.5
  • 🚀 Qwen3.5-VL
• Kimi
  • ✅ Kimi-K2
  • 🚀 Kimi-K3
• GLM
  • 🚀 GLM-5.2

Core MoE Functionality

• ✅ Token dropless MoE - Advanced routing without token dropping
• ✅ Top-K Router with flexible K selection
• ✅ Load balancing losses for expert load balancing optimization
• ✅ Shared-expert support

Advanced Parallelism

• Basic Parallelism
  • ✅ Expert Parallel (EP) with 3D parallelism integration
  • ✅ Context Parallel (CP) and Dynamic Context Parallel (DCP) for long-sequence training and imbalanced sequence workloads
  • ✅ Pipeline Parallelism (PP) with VPP and flexible stage configuration for workload balancing
• Parallelism Composition and Mapping
  • ✅ Full parallelism combo: EP + DP + TP + PP + CP + SP + VPP support
  • ✅ Parallel Folding - Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training
• Data- and Model-Parallel State Sharding
  • ✅ Megatron FSDP / HSDP with expert-parallel support
  • 🚀 Megatron FSDP v2 expanded MoE and ZeRO-mode coverage
  • 🚀 Generalized Tensor Parallelism (GTP) dev integration, bringing per-weight model-parallel sharding and grouped-expert prefetch from main into the dev MoE stack ([DEV] Generalized Tensor Parallelism (GTP) #6882)

Data and Sequence Handling

• ✅ Variable-length dataset ingestion for packed-sequence training
• ✅ DP×CP-aware sequence packing and batch scheduling, including DP-balanced and DCP-aware scheduling
• ✅ End-to-end packed-sequence / THD execution and metadata propagation across TP/PP/CP, including MTP

Optimizations

• Memory
  • ✅ Memory Efficient token permutation
  • ✅ Pipeline-aware fine-grained activation offloading
  • ✅ Fine-grained Recomputations (mla, moe, mlp, moe_act, norm)
  • ✅ Optimizer CPU offload, with a configurable fraction of parameter updates executed on CPU and overlapped transfers
  • ✅ Optimizer-state and master-weight offload with GPU-side updates, using chunked staging to reduce peak optimizer memory
• Communication
  • ✅ DeepEP support for H100 and B200
  • ✅ HybridEP support for GB200
  • ✅ DeepEP v2 token-dispatcher integration
  • ✅ NCCL EP base dispatcher support
  • ✅ 1F1B EP A2A Overlap - Hiding Expert Parallel Communication with 1F1B Pipeline Schedule
  • ✅ DP/PP/TP/EP Communication Overlapping
• Computation
  • ✅ Advanced fusions for Router, Permutation, MLA/RoPE, FP8 casting, and memory-efficient linear cross-entropy on Blackwell
  • ✅ cuDNN fused Attention and FlashAttn integration
  • ✅ GroupedGEMM and Gradient Accumulation Fusion
  • ✅ Production-ready CUDA Graph support for MoE, including partial capture with EP overlap and THD training, with 🚀 chunk-wise and fixed-GA full-iteration THD capture in progress

Optimizer Support

• ✅ Distributed Optimizer for MoE with optimizer-state sharding across dense and expert data-parallel groups (ZeRO-1 equivalent)
• ✅ Muon with LayerWise Distributed Optimizer, with non-matrix parameters handled by distributed Adam or Lion
• ✅ Precision-aware distributed Adam with configurable main-parameter, gradient, and optimizer-state dtypes, including BF16 optimizer states for FP8 training; 🚀 precision-aware Muon for BF16, blockwise FP8, and MXFP8 training, with checkpoint and evaluation validation in progress

Precision Support

• ✅ GroupedGEMM including FP8/MXFP8 support
• ✅ FP8 training full support
• ✅ FP8 primary weights and parameter all-gather across Distributed Optimizer and Megatron-FSDP
• ✅ Fine-grained precision selection for supported model components via per-module TE recipes; 🚀 BF16/MXFP8 options for specialized DeepSeek-V4 hybrid-attention components, including the DSA indexer and grouped output projection, are in progress
• 🚀 NVFP4 Training

Developer Experience

• ✅ MoE Model Zoo with pre-training best practices
• ✅ MCore2HF Converter for ecosystem compatibility in megatron-bridge
• ✅ Distributed Checkpointing Support
• ✅ Runtime Upcycling Support for efficient model scaling
• ✅ Layer-wise logging for detailed monitoring
• ✅ Single-GPU memory and kernel profiling for distributed MoE layouts with Fake Process Group
• ✅ DeepSeek-V3 GB200 performance reproduction guide
• ✅ Megatron Core MoE and advanced-feature guides, covering MoE architecture, parallelism, CUDA Graph, memory/offload, optimizer/FSDP, pipeline layouts, and data loading
• ✅ Public MoE training recipes

---

(2026 Q3) Megatron MoE Roadmap

Model Supports

DeepSeek-V4

See #4468 for detailed tracker.

Functionality

• Convergence
  • E2E convergence verification
  • Deterministic and checkpoint-restart validation (Determinism roadmap tracking #5785)
• Long Context
  • 64K-to-1M context training curriculum validation
• Muon Support
  • Extend precision-aware optimizer support from distributed Adam to Muon, and validate BF16, blockwise FP8, and MXFP8 training together with checkpoint and evaluation correctness
  • Complete decoupled-optimizer checkpoint and LayerWise parameter-gather fixes ([Dev] Align decoupled Muon FP8 parameter-gather paths with main #6015)
• Correctness and numerical-stability fixes for hybrid attention and DSA ([DSv4] Use the full CSA denominator for unfused indexer loss #5960, [Dev] Keep the mHC mapping computation in fp32 on the fused cuTile path #6172, [dev] [Deepseek-V4] Fix fused CSA indexer loss normalization and compact attention indices #6349, [dev] Refactor CSA structure: Move CSA implementation helpers into csa_utils dir #6372)

Performance Optimization

• mHC
  • mHC selective recompute with CUDA Graph under EP overlap ([Dev] Support mHC selective recompute with CUDA graphs under EP a2a overlap #5841)
• Hybrid Attention
  • Fused Attention Kernel for Dense MQA
  • Complete BF16/MXFP8 compact DSA indexer integration and validation ([dev] [DeepSeek-V4] Add compact BF16 and MXFP8 DSA indexer support #5992, cuDNN Frontend #370)
  • Complete BF16/MXFP8 batched GEMM ([Dev] Add batched gemm for DSv4HybridAttention #5722, TE #3160)
  • Dispatch THD CSA compressor pooling to fused cuDNN Frontend kernels (Dispatch the CSA Compressor gated pooling to the cudnn-frontend fused kernels (THD) #5984)
• ClampedSwiGLU
  • Additional fused kernel in cuDNN Frontend
• Long Context and Memory
  • Complete load-balanced CP indexer support, building on delivered CP communication overlap ([dev] Load-balanced context-parallel DSA indexer #6058, [dev] Overlap CP communication for indexer and compressed KV #5691)
  • Optimize MTP contiguous packed-CP rolls ([dev] Optimize MTP contiguous packed-CP rolls #6246)
  • Reduce indexer and hybrid-attention memory, including inverse-RoPE fusion (Fuse DSv4 output inverse RoPE into CSA sparse-attn Functions #6185)

E2E Training Recipes and Validation

• Refresh public 4K DeepSeek-V4 Flash and Pro training recipes ([Dev] Update moe recipes #6335)
• Complete 4K dense training convergence validation (recipe delivered in [Dev] DeepSeek-V4-Flash recipe 20260610 #5266)
• Complete 16K dense training convergence validation
• Complete 64K sparse training convergence validation
• Complete 1M sparse training convergence validation

Long-Term Topics

• Anticipatory Routing
• Flexible Activation Checkpointing

Qwen3.5

Performance Optimization

• Gated Delta Rule (GDN) Optimization
  • Refactor shared GDN/GDN2 implementation foundations ([dev] Refactor: extract and split common logic between GDN & GDN2 (cherry-pick #5843) #6088)
  • GDN kernel fusion on Blackwell
  • Gated RMSNorm implementation in TE
• Memory Optimization
  • GDN fine-grained activation offloading
• Enable and validate partial/full-iteration CUDA Graph paths for the full Qwen3.5 model, including THD and DCP combinations

Long Context Training

• Add mixed-attention CP layout handling for SBHD and THD workloads ([dev] moe(perf): Refactor CP layout organization for Qwen3.5-style hybrid attention model #6387)
• Validate THD, CP, DCP, and CUDA Graph combinations for Qwen3.5

Qwen3.5-VL and Multimodal Training

• Complete the Qwen3.5-VL HybridModel migration ([Dev] feat(multimodal): migrate Qwen3.5-VL to HybridModel #6315)
• Complete Qwen3.5-VL pipeline-parallel support and validation ([Dev] Add pipeline-parallel support to multimodal training in multimodal_dev code path #6081)
• Complete full-model Qwen3.5-VL MDP validation, closing the remaining encoder/decoder CP and full-iteration CUDA Graph gaps and qualifying composition with DP/EP overlap, THD/DCP, and recompute
• Complete and standardize the variable-length multimodal data path, including the Energon dataloader and vision-sequence packing ( [DEV] Add variable-length multimodal mock dataset for Qwen3.5 training #5927)
• Evaluate packing and global-batch scheduling strategies to reduce padding and data imbalance for dynamic-resolution multimodal workloads
• Prepare and validate the Qwen3.5-VL recipe with Megatron FSDP v2, gated on readiness of the required multimodal FSDP v2 path (Integrate NeMo Automodel Qwen3.5-35B-A3B recipe with MFSDP v2 #6139)
• Complete multimodal CI and model-level validation ([Dev] ci: guard examples/multimodal_dev with a unit-test bucket #6083, [Dev] Account for Qwen3.5 vision encoder FLOPs #6173)

GLM-5.2

See #6392 for detailed tracker.

Functionality and Model/Optimizer Parity

• Land the ordinary DSA baseline and backend selection ([Dev] Migrate main-first DSA support to dev #6020)
• Add latent-attention normalization and indexer-projection precision controls ([Dev] Add configurable attention latent normalization epsilon #6204, [Dev] Add TileLang DSA and indexer weights projection precision controls #6279)
• Establish per-head Muon and packed-MLA optimizer parity
• Complete repeated-MTP IndexShare/KVShare sharing, end-to-end TV training, and future-row alignment ([Dev] Share DSA indexer output and latent KV across repeated MTP steps #6472, [Dev] Add end-to-end TV loss for MTP #6473, [Dev] Add absolute row alignment for MTP sequence rolls #6741)

Performance, Memory, and CUDA Graph

• Skip invalid slots in sparse-indexer backward (fix(dsa): skip invalid slots in sparse indexer backward #6166)
• Correct DSA FLOPs accounting and resolve the CP1 indexer-scoring path ([Dev] fix: account for DSA top-k sparsity and indexer cost in FLOPs #6753, [dev](perf): DSA pre-attention: plan-driven scorer dispatch, host-built CP layout metadata #6206)
• Fuse standard RoPE packing for MLA and DSA ([Dev] Fuse standard RoPE packing for MLA and DSA #6343)
• Make DSA sparse-attention backward fixed-shape for empty-row cases ([Dev] Fix DSA sparse attention backward empty rows #6910)
• Enable low-precision AbsorbedMLA up-projection recompute ([Dev] Allow FP8/FP4 with AbsorbedMLA up-projection recompute #6178)
• Close full-model memory safety with selective recompute and activation offload
• Reduce DP-balanced THD rerouting memory ([dev] Replace DP balance all-to-all rerouting with all-gather #6378)
• Validate DSA, repeated-MTP, and whole-MoE paged-stash CUDA Graph composition ([Dev] Support paged stash with TE whole-MoE CUDA graphs #6022)

Long Context and Parallelism

• Validate HybridModel execution and 1F1B overlap composition, including logical-layer scheduling, dense/MoE composition, pipeline boundaries, deferred weight-gradient computation, repeated-MTP sharing, and Distributed Optimizer
• Qualify long-context packed-THD training with static and dynamic CP, including correctness, checkpoint resume, memory, and performance

E2E Training Recipes and Delivery

• Validate model and optimizer checkpoint continuity, including repeated-MTP sharing, Muon state, and low-precision parameters
• Qualify reproducible full-model BF16 and MXFP8 recipes on GB200 and GB300, including correctness, memory margin, performance evidence, and short convergence

Kimi-K3

See #6872 for detailed tracker.

Functionality and Model Integration

• Establish the baseline Quantile Balancing path for MoE routing (Implement Quantile Balancing in MoE #5349)
• Complete Kimi-K3 global-batch Quantile Balancing in Megatron Core and its supporting dense-routing-map integration, including fused CUDA Graph readiness ([Dev] Add global-batch quantile balancing #6637, [Dev] Enable dense routing maps for Flex dispatch #6614, Transformer Engine #3395, Transformer Engine #3426)
• Add optional output normalization for latent MoE ([ENHANCEMENT] Optional output normalization for latent MoE #6448, [MoE] Add optional output normalization for latent MoE #6449)
• Integrate latent-MoE RMSNorm into Megatron Core training ([Dev] Add RMSNorm latent up-projection #6804)
• Complete end-to-end Kimi-K3 SiTU-GLU integration across dense, routed-expert, and shared-expert FFNs, building on merged Transformer Engine and cuDNN Frontend foundations ([Dev] Add SiTU-GLU through Transformer Engine #6673, Transformer Engine #3402, cuDNN Frontend #645, cuDNN Frontend #670)
• Add baseline KDA and Gated MLA hybrid-model support in MCore ([Dev] Add Ling-V3 Tiny hybrid model training support #6556)
• Complete KDA low-rank projections and extended MLA integration and validation ([dev] feat(moe): Add KDA Low-Rank Projections and Extend MLA Support #6877)
• Attention Residual support in MCore #6840

Performance and CUDA Graph

• Migrate GDN kernel fusion works to KDA
• MLA latent CP ([dev] moe(perf): MLA Latent Context Parallel #6829)
• Validate end-to-end CUDA Graph execution for the Kimi-K3 KDA path in composition with the MCore MoE stack, building on graph-capable backend foundations (cuDNN Frontend #556)

Training Validation and Recipes

• Establish a reproducible Kimi-K3 training recipe and complete initial correctness and convergence validation

P1 items

• Per-head Muon
• Multimodal Support (MoonViT-V2)
• MXFP4 Support

General Optimizations

Packed Sequence and Dynamic Context Parallel (DCP)

• Validate representative variable-length packed/THD workloads end to end with static CP and DCP
• Mature DP-balanced scheduling for production-scale packed variable-length workloads
• Complete DCP efficiency and load-imbalance metrics
• Reduce DCP communication-memory overhead and support arbitrary CP sizes, including non-power-of-two configurations ([dev] Reuse parent communicators for dynamic context parallelism #6801, TE #3420)
• Develop workload-model-driven DCP configuration optimization and publish representative guidance for variable-length THD workloads across supported attention backends

CUDA Graph

• Enable and validate end-to-end CUDA Graph execution for KDA-based models
• Enable and validate end-to-end CUDA Graph execution for DSA-based models (GLM-5.2 training support tracking #6392)
• Padding-aware GEMM for CUDA Graph-padded variable-length workloads, avoiding tail-padding compute while preserving replay-stable buffer shapes
• Chunk-wise whole-block CUDA Graph support for THD ([dev] Add chunk-wise (whole-block) CUDA graph support for THD training #5258)
• DCP partial CUDA Graph capture and shared-pool slot reuse, including characterization and reduction of model-dependent graph-memory overhead ([dev] partial cuda graph support for dynamic cp #5618, [dev] Share CUDA graph slots across dynamic CP variants #6479, TE #3353)
• Megatron FSDP v2 integration with partial CUDA Graph capture, including end-to-end MoE composition validation
• Paged stash for whole-MoE CUDA Graph capture ([Dev] Support paged stash with TE whole-MoE CUDA graphs #6022)
• Full-iteration THD CUDA Graph under fixed gradient accumulation ([dev] support thd full iter graph under fixed GA #5807)

MegaMoE

• Advance MegaMoE integration and correctness/determinism qualification across representative MoE workloads, including MOK and CuTe DSL paths (mok megakernel integration #6572, TE #3350)
• Benchmark MegaMoE variants against one another and against established EP-overlap baselines across representative workloads, covering performance, memory, scalability, and composition constraints

Token-Imbalance Scheduling and Load Balancing

• Build a common benchmark for runtime MoE token imbalance, comparing the current baseline with ECHO (MoE ECHO: Unlocking Sync-Free, Full CUDA-Graph Support for Dropless MoE via Elastic Cloning #2368), UltraEP, MoonEP (MoonEP support #6134), and related approaches
• Design and evaluate a common runtime-aware MoEScheduler framework, adapt ECHO to the common interface, and establish unified correctness, performance, and integration validation across UltraEP, MoonEP, and ECHO backends

Communication Optimization

• Full activation recompute with EP A2A overlap ([Dev] Support full activation recompute for EP A2A overlap (VPP-stage recompute) #5869)
• Per-layer activation recompute with EP A2A overlap ([Dev] Recompute EP A2A overlap activations per layer instead of per model chunk #6311)
• HybridModel EP-overlap integration, checkpoint compatibility, and performance/memory parity with the GPTModel path for the DeepSeek-V3 proxy workload
• HybridEP robustness for zero-copy, small-token, ultra-sparse, and paged-stash workloads

Megatron FSDP

• Characterize and improve Megatron FSDP v2 MoE memory and execution behavior across TP/EP, PP/VPP and 1F1B overlap, no-shard/ZeRO modes, heterogeneous dense/expert sharding, memory-safe initialization from scratch, and LayerWise Muon bucket overlap

GEMM Optimization

• Validate device-initiated GroupedGEMM with paged-stash composition across representative MoE workloads, including correctness, performance, and in-flight buffer memory behavior
• Enable and validate 2D MXFP8 quantization for Megatron Core GEMM paths on Blackwell, building on merged Transformer Engine support (TE #2634)
• Reduce GroupedGEMM and grouped-MLP host and launch overhead, including ragged and small-token expert workloads (TE #2897, TE #3099)
• Complete low-precision grouped-expert kernel and fusion paths (TE #3315, TE #3359, TE #3261)

Optimizer

• Add optimizer-state and master-weight offload with GPU-side updates, using chunked staging to reduce peak optimizer memory for Distributed Adam and compact-layout BF16 LayerWise Muon ([dev] Add chunked optimizer-state and master-weight offload #6244)
• Complete parameter-structure-aware Muon updates for fused attention projections and gated-MLP weights (Add per-head Muon QKV orthogonalization and fix TP>num_query_groups with Muon. Generalize to MLA and GQA #6326, Split gated FC1 updates in Muon #6688)
• Complete explicit optimizer assignment for MoE routers and other parameters excluded from Muon, building on corrected LayerWise buffer routing (Add option to skip Muon for MoE routers #6504, Fix LayerWise buffer routing to respect use_muon = False #6642)

Release Qualification and Branch Alignment

This section separates dev release qualification from bidirectional branch alignment. Branch-alignment items are capability-level: inclusion does not constitute a Q3 delivery commitment or imply active ownership, and PR-level implementation details are intentionally omitted.

Dev Release Qualification

• Publish a qualified dev-branch release tag, gated by convergence and performance validation of one canonical configuration each for DeepSeek-V3 4K, DeepSeek-V4 Flash 64K, and Qwen3.5-397B

Dev-to-Main Promotion

Already on main

• Establish data and sequence-handling foundations on main, including THD/DCP plumbing, sequence packing, variable-length data handling, and CP-aware CUDA Graph compatibility
• Establish distributed-execution foundations on main, including schedule planning, expert-parallel communication, Megatron FSDP integration and overlap, checkpointing, and core sharding modes
• Establish precision and heterogeneous-model foundations on main, including low-precision parameter movement, heterogeneous model-parallel axes, and grouped-module initialization

Remaining promotion gaps

• Complete model and HybridModel promotion readiness for DeepSeek-V4, Qwen3.5, Kimi-K3, and shared HybridModel integration
• Complete expert-parallel backend and overlap readiness, including dispatcher, checkpoint, recompute, and training composition
• Complete THD, sequence-packing, DCP, and CUDA Graph readiness for variable-length and long-context workloads
• Complete remaining Megatron FSDP v2 readiness across parallelism modes, pipeline schedules, optimizer checkpointing, and grouped-expert weights
• Close optimizer, precision, normalization, initialization, and gradient-handling parity gaps

Main-to-Dev Synchronization

• Integrate the main-branch GTP stack into dev and validate it with the dev MoE stack

Ongoing Long-term Features

• E2E Performance optimization for DeepSeek-V4, Qwen3.5 and other fine-grained MoEs
• Extreme-sparsity MoE execution and scaling, including large expert counts, small-token dispatch, and router/dispatcher co-design
• Migration from GPTModel to HybridModel
• CUDA Graph readiness across key MoE training workloads
• Long-context MoE training scalability and productionization
• RL readiness for MoE models, including policy-weight refit, rollout-training consistency, and end-to-end recipe validation
• NVFP4 MoE training maturity, including numerical stability, convergence, and production recipe readiness
• Production-ready, topology-aware MoE load balancing through runtime scheduling and expert placement

(2026 Q2) Highlights

Model & Architecture

• DeepSeek-V4 architecture baseline, including hybrid attention, Hash MoE, ClampedSwiGLU, mHC, MTP, and HybridModel (DeepSeek-V4 training support #4468, [dev] Enable DeepSeek-v4 hybrid_model Part (3/N) #5762)
• DeepSeek-V4 THD and context-parallel support ([Dev] add support for deepep/hybridep dispatcher under thd format training #4816, [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention #5011, [dev] [DeepSeek-v4] Context Parallel support #5087)
• DeepSeek-V4-Flash 4K training recipe ([Dev] DeepSeek-V4-Flash recipe 20260610 #5266)
• DeepSeek-V4 HF <-> MCore checkpoint conversion (Megatron Bridge #3562)
• Qwen3.5 architecture, packed sequence, and checkpoint conversion ([dev] [3/5] Qwen3.5 support: SharedExpertMLP meta init #4749, [Dev] fix(mtp): use padded cu_seqlens in MTP roll for THD with CP #4494, [dev] [2/5] Qwen3.5 support: FSDP DTensor Bridge checkpoint compatibility #4748)
• Qwen3.5 GDN context parallel, fused mRoPE, OOTB E2E THD/FSDP example, and heterogeneous encoder-decoder parallelism foundations (Fuse per-sequence AlltoAll into a unified one in GDN forward #4913, [dev] moe(feat): support chunkwise context parallelism for GDN. #3282, [Dev] Add Triton fused mRoPE for Qwen3.5-VL #5962, [dev] [5/5] Qwen3.5 support: Qwen3.5-VL training example #4751, Add named layouts to HyperCommGrid for heterogeneous parallelism #5148, Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path #5150, Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) #5260)

Performance & Optimizers

• mHC, hybrid-attention, and ClampedSwiGLU fusion foundations ([dev] mHC kernel fusion  #3828, [dev]: faster implementation of mHC fused kernels #4624, [dev] [DeepSeek-v4] Part 4: Fusion Kernels for DSv4 Hybrid Attention #4894, [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing #5130, TE #2938)
• Correct mHC boundaries in EP-overlap schedules ([dev]: Fix mHC boundaries in EP overlap schedule #5471)
• LayerWise optimizer, Newton–Schulz integration, and FP8-aware Muon parameter gather (Integrate LayerWiseDistributedOptimizer with DDP buffer infrastructure #4509, [dev] bump emerging optimizers to v0.3.0 #5320, [dev] Add experimental decoupled compact LayerWise DDP layout for Muon #5388, [dev] Muon fp8 param gather for decoupled layout #5470)
• Qwen3.5 pre-GDR, chunk-wise CP pre-GDR, and selective recompute ([dev] moe(perf): Pre-GDR kernel fusion #5361, [dev] moe(perf): Support fused pre-GDR path for chunkwise CP. #5638, feat(ssm): whole-module 'gdn' selective recompute for GatedDeltaNet #5296)

Parallelism, Dispatch, and CUDA Graph

• Variable-length THD, real/padded-length propagation, and DCP foundations (varlendataset for thd e2e and benchmark #4832, [codex] Preserve real and padded THD sequence lengths #5541, [codex] Propagate SBHD padding masks across pipeline stages #5544, Minor improvements for Dynamic-cp #4226)
• DeepEP v2 and NCCL EP dispatcher integration ([Dev] Add DeepEP v2 flex dispatcher backend #4793, NCCL EP support #5129)
• FSDP communication with EP A2A overlap ([Dev][feat] Support A2A Overlap for Megatron-FSDP #3796)
• CUDA Graph interface refactor and partial THD capture ([Dev] Refactor CUDA graph API: decompose cuda_graph_scope into full_iteration impl, inference scope, and per-layer capture modules #4293, [dev] partial cuda graph support for thd format training. #4359)

---

Call for Community Contributions

• Model implementations - Additional MoE model variants
• Performance testing - Performance tests across different platforms and workloads
• Documentation and tutorials - Best practices and optimization guides
• Bug fixes

---

This roadmap reflects the collective efforts of NVIDIA and our collaborators.

Credits: MCore MoE Team and @sbhavani

Labels: roadmap, moe, call-for-contribution

[zupengwang]

Hi maintainers — before prototyping the roadmap item “Reduce peak MoE communication memory in return A2A/unpermute paths” (the replacement requested after #4746), could you clarify the intended acceptance boundary?

1. Target branch: should a community replacement target dev, which [ROADMAP][2026 Q3] Megatron Core MoE Roadmap #6757 names as the feature-development baseline, or main, which Reduce MoE peak memory in unfused EP combine path #4746 targeted? The two branches are currently diverged.
2. Performance baseline: must the replacement beat or match moe_permute_fusion=True on step time while reducing peak memory, or is an improvement limited to the unfused fallback path still useful? Reduce MoE peak memory in unfused EP combine path #4746 was closed after its author found that enabling moe_permute_fusion performed better.
3. Hardware/world-size evidence: would initial single-node BF16/NCCL evidence on SM86 with world sizes 2, 3, and 6 be acceptable for correctness, peak allocated/reserved memory, and warmed-up step-time comparisons, with upstream 8-GPU/Hopper CI covering the larger matrix? The prior PR’s distributed test used TP1/EP8 and TP2/EP4, which cannot be reproduced exactly on a 6-GPU host.
4. Replacement scope: should the first PR address only the return A2A + unpermute transient buffers, or also include the SequentialMLP output preallocation and unfused sort_chunks_by_idxs split/cat removal from Reduce MoE peak memory in unfused EP combine path #4746? May CUDA Graph, shared-expert, drop_and_pad, and fused-permutation modes explicitly retain the existing path initially?
5. Ownership/design: is there an active internal branch or preferred design that a community contribution should build on?

I can keep the first change opt-in, independently reviewable, and backed by forward/backward parity plus repeated distributed memory/performance measurements.

[buptzyb]

@zupengwang Thanks for checking. To clarify, optimizing the unfused path doesn't seem to be a high-priority direction to me. We recommend using an optimized dispatcher such as HybridEP.

Could you clarify what concrete use case on your side is motivating the work on the unfused path? Is there a supported configuration where the fused or optimized paths are unavailable or unsuitable, or where they introduce a specific correctness, compatibility, or memory issue? cc @HollowMan6