Bump sglang to v0.5.18 - #2714
Merged
Merged
Conversation
Megatron dev reimplemented PR #6's MTP-in-RL support natively: - process_mtp_loss derives MTP labels from input_ids when labels is None (RL). - config.mtp_detach_heads detaches output head + MTP embedding gradients. So on the miles side: set config.mtp_detach_heads=True when enable_mtp_training, and stop passing the now-unsupported mtp_kwargs to GPTModel.forward (labels=None + input_ids derivation is equivalent to mtp_labels=batch['tokens']).
… lora test mock Matches the dev Megatron arg rename used in model.py (use_gloo_process_groups).
…put_with_bias
New dev split the MLP into _forward_mlp_output_with_bias + _forward_post_mlp and dropped
_sglang_pre_mlp_residual (true-on-policy was deferred). Anchor pre_mlp_residual on the
unconditional nvtx_range_push("mlp") (residual set in all branches); keep pre_mlp_layernorm_output
and mlp_output_with_bias anchors. Will re-verify after the true-on-policy migration.
…bias The megatron true-on-policy merge relocated the _sglang_pre_mlp_residual extraction out of _forward_mlp into _forward_mlp_output_with_bias, so the BSHD source-patcher anchor 'residual = getattr(self, "_sglang_pre_mlp_residual", hidden_states)' no longer exists. Mirror the THD config: target _forward_mlp_output_with_bias and prepend the pre_mlp_residual dump at the mlp nvtx push (matches the new structure). Fixes test_run_megatron + test_miles_dumper source-patch failures.
…_module) New dev's build_module passes name= to every spec-built module (megatron base Attention.__init__ accepts it). miles' custom attention classes overriding __init__ must accept + thread it: DSAMultiLatentAttention/DSAMLASelfAttention (glm5, DeepSeek-V3.2), HuggingfaceAttention base, qwen3_5/qwen3_next Attention. Mirrors the DeepSeekV4Attention name fix. Fixes 'DSAMLASelfAttention.__init__() got an unexpected keyword argument name'.
…lders New dev removed the moe_use_legacy_grouped_gemm argument and the legacy grouped-GEMM concept (grouped_mlp_modules now takes only moe_use_grouped_gemm). miles' model_provider and glm4 spec builders read args.moe_use_legacy_grouped_gemm, which no longer exists -> AttributeError on the Namespace. Drop the kwarg (dev spec fns take **kwargs / the new grouped path is the only one).
New Megatron renamed MultiTokenPredictionLayerSubmodules.transformer_layer ->
mtp_model_layer, so MTP weight names are now mtp.layers.{i}.mtp_model_layer.*.
miles' MTP weight converters (mcore<->hf) hardcoded 'transformer_layer' ->
'Unknown MTP component' on rollout weight export (test_mimo_7B_mtp_only_grad).
Accept both names across export (mimo/qwen3_next/deepseekv3/glm4moe/qwen3_5 + fp8/
mxfp8/nvfp4 quantizers) and mbridge import (mimo/qwen3_next/qwen3_5/glm4moe_lite/glm4moe).
New dev rewrote MTPLossLoggingHelper: it stores loss_sums/num_tokens (or loss_values) and only computes tracker['values'] after reduce_loss_in_tracker() (which also does the cross-rank all-reduce). miles read tracker['values'] directly (never populated) -> 'values' not in tracker -> mtp_losses unbound -> UnboundLocalError at extra_metrics. Call reduce_loss_in_tracker(), read values/loss_values, init mtp_losses=None + guard the metric. Fixes test_mimo_7B_mtp_only_grad past weight-export.
New Megatron reads disable_{param,grad}_buffers_cpu_backup from
DistributedDataParallelConfig; _ParamAndGradBuffer no longer takes them
as ctor kwargs.
…rd-only run-megatron skips DDP wrap
Rebuilds were driven by whether the PR diff touched docker paths, so every rerun of a Dockerfile-touching PR paid for an identical multi-arch build. Hash the build inputs, stamp the hash on the published tag as a label, and compare against it: a PR keeps one pr-<num> tag and rebuilds it only when the inputs actually move, when the tag is missing, or on the one-shot rebuild-ci-image label. resolve-ci-image now selects the PR image whenever that tag is current rather than only when this run built it, so reusing an image does not silently fall back to dev.
Also scope pull-requests: write to the docker-build call instead of the whole workflow; only that job removes the one-shot label.
…' into megatron-bump-20260819
New dev returns functools.partial builders for the mlp and shared-experts submodules instead of ModuleSpec, and MoELayer now passes name= to the shared-experts builder.
Megatron now rejects the combination: fake QAT swaps in straight-through weight tensors while TE's fused wgrad accumulation writes main_grad onto the original ones, so the quantized weights' gradients would be dropped.
Megatron's dsa variant now builds AbsorbedMLASelfAttention, which reads linear_kv_up_proj.weight directly; the pinned Bridge's LoRALinear has no such property. Upstream Bridge added one that returns the LoRA-effective weight.
Megatron's Mamba mixer exposes conv1d as conv1d_weight/conv1d_bias parameters; the pinned Bridge maps only the conv1d.* module names, so the weight sync skips them silently. Upstream Bridge accepts both spellings.
New Megatron renamed the MTP submodule to mtp_model_layer, so MTP expert weights missed the expert-offset rewrite and ranks broadcast differing tensor sets during weight sync, deadlocking the EP collectives.
Megatron now gates the HF tokenizer on args.trust_remote_code, which miles never set, so checkpoints shipping custom tokenizer code (Kimi) failed to load. miles already passes trust_remote_code=True everywhere it loads a checkpoint itself.
The private _keep_fp32 attribute was only honoured by the fp32 snapshot that rode in on the DSv4 megatron commit, which the base bump defers. Upstream now owns this: mark_keep_in_fp32 exempts the tensor from Float16Module's cast, so the scale survives the bf16 round trip through the weight sync.
TopKRouter.forward gained packed_seq_params upstream, which MoELayer passes positionally. Separately, the plain bridge provider never forwarded gradient_accumulation_fusion, so --no-gradient-accumulation-fusion could not reach models built through it (the LoRA bridge path already synced it).
# Conflicts: # .github/workflows/_build-pr-ci-image.yml
Megatron now resolves te_general_gemm, which it could not on TE 2.17 before the bump, and TE multiplies in the operand dtype -- so --moe-router-dtype fp32 became a bf16 multiply. The last ulp is enough for top-k to pick a different expert, and on qwen3.5 that flipped one token in layer 0 and cascaded to a 15% logprob drift. 25 model scripts ask for fp32 routing, so default it here rather than per model.
DetProcessGroup registers the NCCL backend for cuda so device lookups work, which means a collective it does not override reaches the C++ base and runs on NCCL -- silently, with NCCL's reduction order instead of the fixed-order fold the group exists to provide. torch 2.13 routes dist.reduce_scatter_tensor through reduce_scatter_single, which was not overridden, so test_det_process_group's bitwise assertion saw NCCL's 6e-8 reduction error. all_to_all_single and monitored_barrier are left alone: neither reduces, so the base dispatch returns the same bits.
The bump moved WHEELS_TAG_X86 to cu130-torch213-x86_64 and taught docker-build.yml to fingerprint cu130-torch213-aarch64, but left the Dockerfile's arm64 default on cu130-aarch64 — a July release whose wheels were built against torch 2.9. An arm64 image therefore installed torch-2.9 ABI extensions (TransformerEngine, flash-attn, apex, mamba) on top of the v0.5.18 base's torch 2.13. Verified by building the aarch64 image with the corrected tag and running it on a GB300 (sm_103): transformer_engine.pytorch imports, te.Linear bf16 fwd+bwd and fp8_autocast produce finite output, FA2 runs, and nvidia-cutlass-dsl resolves 4.6.2. The two docs that quote these ARG defaults still named the pre-torch-2.13 releases for both arches; update them to match.
A skill under .claude/skills/ ships its own harness and runs it itself, so the files it carries are no more this runner's to collect than the sglang and Megatron-LM checkouts CI clones into the workspace -- which the same function already skips by asking git instead of walking the tree. The mechanical-refactor-verify update in #1890 brought 25 such files, and the rule has failed on every PR branched from main since. Widening _KNOWN_ORPHANS instead would work against what that set is for: it is meant to shrink, and each entry is a promise to move a file that belongs under tests/. These do not.
…ng manager allgather_into_tensor_coalesced waits on each inner _allgather_base, but inside an active _coalescing_manager the inner collectives are batched and hand back None -- the manager waits on the aggregate at exit. The all_gather_single forwarding added for torch 2.13 routed the coalescing flush into this method, so the unconditional .wait() raised AttributeError on rank 0 of test_det_process_group's coalescing check. Wait only when a real Work comes back (outside a manager, where nobody else will wait on it).
The 09912d8 push produced no PR Test run (only the fork-only approval gate, which correctly skips for an in-repo branch), so nothing validated that commit. Re-run against yueming/bump-ci-fixes, pinned in the PR body, which carries the dp-capture layout fix and the DSV4 fusion auto-enable revert.
# Conflicts: # tests/ci/test/test_ci_discovery_coverage.py
Forwarding all_gather_single / all_gather_single_coalesced was overreach on my part: only the reducing collectives need the fixed-order fold, and all_gather moves data without summing, so the C++ base dispatch to the inner NCCL backend already returns identical bits. Worse, the coalesced forward reissued per-tensor _allgather_base calls on the inner group while the outer _coalescing_manager was mid-flush, which faulted in NCCL -- rank 3 died with SIGSEGV in test_det_process_group_multi_gpu. Keep the reduce_scatter pair, which is what the bitwise assertion actually needs, and restore allgather_into_tensor_coalesced to its original form now that nothing routes a coalescing flush into it.
Picks up sglang yueming/bump-ci-fixes aac166b1c0, which fixes the 'MQALayer' object has no attribute 'tp_size' crash the DSV4 session-server test hit once the fusion-autoenable deadlock stopped masking it.
… flush torch 2.13's _coalescing_manager.__exit__ calls allgather_into_tensor_coalesced directly -- not through all_gather_single_coalesced -- so removing that forwarding stopped the NCCL fault but left the flush landing here, where the unconditional .wait() on a batched (None) Work raises AttributeError. Wait only on a real Work; the manager waits on the aggregate.
The SIGSEGV in test_det_process_group_multi_gpu was in _check_delegation's dist.all_to_all_single, not in the coalescing checks I had been chasing: torch 2.13 dispatches it to all_to_all_single, which this wrapper does not define, so it reached the C++ base and faulted in NCCL. I had listed the name as missing when the reduce_scatter_single gap was found and left it alone, reasoning that a collective without a reduction returns the same bits either way -- true of the values, but the base dispatch does not survive the trip on a wrapper group. Forward it to alltoall_base, which already delegates to the inner group. Validated on a 4-GPU devbox running the CI image: the whole file goes from the segfault to 24 passed.
|
❌ The execution job failed; inspect the workflow run for the failing step. |
|
❌ The execution job failed; inspect the workflow run for the failing step. |
|
✅ |
test_mtp1_spec_v2_r3 failed the last round on a sampling NaN. Three isolated /rerun-test runs of that file on the same image gave three different outcomes -- a NaN timeout, an NCCL init error, and a pass -- so it is flaky rather than a regression from this bump. Confirm against a full round.
Collaborator
Author
|
All CI in |
yueming-yuan
removed
run-ci-image
bypass-fastfail
They recorded how the bug was found rather than what a reader needs at the line. The one on allgather_into_tensor_coalesced also still described the _start/_end_coalescing forwarding that the final fix dropped.
….5.18 base The v0.5.18 base ships sglang's own pins (flashinfer 0.6.17, cutlass-dsl 4.6.2), so both override blocks now downgrade instead of fix: the stale flashinfer 0.6.15.post1 breaks the trtllm MoE runner on Blackwell (ActivationType.Situ is 0.6.16+). The apache-tvm-ffi reconcile stays: it guards against this file's own later installs, not the base.
Zhichenzzz
approved these changes
Aug 27, 2026
| self.group.bound_device_id = dev | ||
|
|
||
|
|
||
| def _forward_remaining_collectives(): |
Contributor
There was a problem hiding this comment.
LGTM if @yueming-yuan tested this under GB300 nodes
# Conflicts: # tests/fast/utils/chat_template_utils/test_deepseek_v4.py
guapisolo
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebases the
sglang-milesstack ontov0.5.18and repoints the miles images at it. Targets #2673 so the Megatron and sglang bumps are validated together.sglang side
sglang-miles-v0.5.18= 35 commits onv0.5.18, replayed fromsglang-miles-v0.5.17rather than from production: production is still the v0.5.16 stack, so going through v0.5.17 halves the upstream drift (736 commits instead of 1317) and carries forward resolutions a green CI already validated. Production gained no commits in the meantime, so nothing is skipped. Rollback points:sglang-miles-v0.5.16-final(production) andsglang-miles-v0.5.17.Only 9 of 35 picks conflicted. Where v0.5.18 has since removed machinery that the v0.5.17 resolutions leaned on, the newer shape wins and our behaviour was re-expressed on top of it:
tokenizer_manager: v0.5.18 dropped the parallel-samplingchild_rid_to_logical_rid/lifecycle_id/_remove_req_statemachinery. Our abort-by-prefix gate, pre-dispatch flagging and LoRA lease release were rebuilt on v0.5.18's simpler form; the dispatch paths keep upstream's new CUDA-VMM prepare/cancel structure with our abort gate ahead of the reservation.forward_batch_info: v0.5.18 split mrope into decode/extend helpers and already forces the text-only path onrl_on_policy_target— the same intent as ours. Took its structure and added our contract switch at both decision points, sincetrue_on_policy_contractis a separate switch fromrl_on_policy_target.deepseek_v2: v0.5.18 moved the shared-expert fusion decision into the loader plus ashared_experts_fusion_disable_reasonclassmethod that carries our SBO/TBO and DeepEP reasons, sodetermine_num_fused_shared_expertsis upstream's no-arg version.compile_utils: kept upstream'senvs.SGLANG_DG_CACHE_DIRdefault and layered ourSGLANG_DG_CACHE_DIR_PER_PROCESSsuffix on it (colocated RL engines would otherwise race on one JIT cache).glm4_moe,dsa_indexer: each side adds a distinct method, both kept.Post-pick repairs: restored
_post_load_weights(the pick dropped the definition but kept its call sites) andget_execinglm4_moe_nextn; a mangleddetermine_num_fused_shared_expertsremnant was replaced with upstream's.ruffF821 is zero across every file the stack touches, andsys.exit(pytest.main(...))now matches the repo convention in the parallelism-context test.miles side
docker/Dockerfile:SGLANG_IMAGE_TAG=v0.5.18,SGLANG_BRANCH=sglang-miles-v0.5.18(switched to the stablesglang-milesname at merge time).docker/build.py: release variant pinned tov0.5.18-cu129.pyproject.tomlrewrite block was re-checked against v0.5.18's upstream Dockerfile — the three sed patterns and the three markers are unchanged, so it still applies as-is.Rust files are untouched; local rustfmt/clippy hooks fail only for want of
cargoon this machine.