Feat/sdpo ema teacher - #293
Draft
ZiyiTsang wants to merge 281 commits into
Draft
Conversation
# 🐛 Bug Fix ## Strip consecutive `<|image_pad|>` tokens in pre-tokenized prompts - Add `QwenVLImageProcessor._strip_image_token` static helper that collapses runs of `<|image_pad|>` (token id 151655) into a single placeholder while leaving the surrounding `<|vision_start|>`/`<|vision_end|>` markers intact. - Apply the helper to `prompt` before calling `load_mm_data` in `process_mm_data_async`, so pre-tokenized `input_ids` (where each image is already expanded to N image-pad tokens, one per visual patch) no longer collide with `load_mm_data` re-expanding the placeholder itself. Without the collapse, the pipeline saw `N x M` image-pad tokens and miscounted positions, breaking mrope bookkeeping. - Raw text (`str`) prompts are passed through unchanged.
# 🐛 Bug Fix ## Disable torch.compile around GatedDeltaNet QKV prep - Wrap `_prepare_qkv_for_gated_delta_rule` with `torch._dynamo.config.patch(disable=True)` in the Megatron patch - Avoids torch.compile failure on the Qwen3.6 GatedDeltaNet path --- # ⭐ Feature ## Generalize unsplit-forward path to text-only Qwen3.6 / Qwen3.5 - Detect `Qwen3VLModel` at model build and set `args.uses_unsplit_forward`; the bridge model does CP+SP splitting internally for both VL and text-only Qwen3.5/3.6 sharing the same architecture - Route unsplit tokens + tp*cp*2-aligned `cu_seqlens` through `forward_only` / `train_one_step` whenever the flag is on, not just for VL inputs - Propagate the flag through `data.get_batch`, `loss.compute_advantages_and_returns`, `log_rollout_data`, and `stream_dataloader.post_process_rollout_data` so padding stays consistent ## Add Qwen3.6-35B-A3B 8xGPU DAPO-math training script - New `scripts/training/text/run-qwen36-35B-A3B-8xgpu.sh` for sync GRPO training with TP=2/PP=2/CP=2/EP=4 and partial rollout
# 🐛 Bug Fix ## Propagate fp16 to SGLang Mamba conv dtype - `relax/distributed/ray/genrm.py` and `relax/distributed/ray/rollout.py`: pass `SGLANG_MAMBA_CONV_DTYPE=float16` to the engine env when `--fp16` is set, so Qwen3.6 hybrid-Mamba layers use the matching dtype in rollout/GenRM - `scripts/training/multimodal/run-qwen3-vl-30B-A3B-8xgpu.sh`: add `--fp16 --use-rollout-routing-replay --use-slime-router` - `scripts/training/multimodal/run-qwen35-35B-A3B-8xgpu.sh`: add `--use-rollout-routing-replay --use-slime-router`; document why fp16 is intentionally left disabled for Qwen3.5 ## Skip routing replay for MTP layers - `relax/utils/training/routing_replay.py`: MTP routers exist in training but rollout (sglang) does not run MTP, so there is nothing to record or replay against. Install a pre-hook that clears the global `ROUTING_REPLAY` (so `compute_topk` falls through to the original impl) and skip registration in `all_routing_replays` to keep the per-layer accounting consistent - Guard `compute_topk` against `ROUTING_REPLAY is None` ## Detect Ray 2.x head node by internal resource - `relax/utils/utils.py::get_serve_url`: Ray 2.x auto-registers `node:__internal_head__` on the head node; legacy setups also tag it with a custom `head` resource. Accept either when scanning `ray.nodes()` so head IP discovery works on both --- # 📝 Documentation ## Announce Qwen3.6 support in README - `README.md` / `README_zh.md`: add 05/11/2026 news entry noting Qwen3.6 series (text + VLM) support
# ⭐ Feature ## Auto-detect shared-GPU colocate sub-mode for GenRM - Pick mode from GPU allocation: `R+G==A` keeps the existing split layout; `R==G==A` activates the new shared layout where rollout and genrm overlap on the same bundles. Other combinations are now rejected at startup with a clear error. - Drop the bundle offset for genrm in shared mode so both engines schedule on the same `[0, A)` bundles, and lower genrm Ray fractional `num_gpus` default from 0.2 to 0.1 to leave room alongside rollout. - Plumb `mem_fraction_static` through `--genrm-engine-config`; rollout keeps using `--sglang-mem-fraction-static`. Two engines can now split each GPU independently. - Onload rollout weights and genrm KV in parallel inside `update_weights()` so both engines come back together before the next rollout step. --- # 📝 Documentation ## Document the new GenRM colocate sub-mode (en + zh) - Add a second ASCII architecture diagram for the shared layout and a sub-mode auto-detection table. - Introduce a Shared-mode launch example with `mem_fraction_static` settings and a warning to keep the per-GPU sum < 1.0. - Update Best Practices with sub-mode selection guidance and OOM troubleshooting for shared mode. - Fix stale defaults in the sampling-config table (temperature 0.2 -> 0.1, max_response_len 1024 -> 4096) and add `ep_size` / `mem_fraction_static` to the engine-config table. - Update the example launch script to demonstrate shared mode (rollout 0.5 + genrm 0.3, both on 8 GPU). --- # 🔩 Chore ## Add py-spy multi-PID dump helper - `scripts/tools/_pyspy_dump.sh` runs `py-spy dump` over a list of PIDs in one ray-job submission, used by the debug-hang skill to avoid per-PID submission overhead.
This commit integrates the glm_moe_dsa model, updates Megatron backend, sets up corresponding training scripts, and modifies entrypoint scripts (local.sh, ray-job.sh, spmd-multinode.sh) to support environment variable overrides, keeping the cleanup logic intact.
# 🔩 Chore ## Update torch_memory_saver dependency - Switch source repo from `fzyzcjy/torch_memory_saver` to `redai-infra/torch_memory_saver` fork - Pin to commit `afc13785c50119048e2dd8ac497cc9e29ec75bd4` - Set `TMS_CUDA_MAJOR=12` build-time env var for CUDA 12 compatibility
# ♻️ Refactor
## Split path env vars in launcher scripts
- Introduce `MODEL_DIR` (HF weights / `--ref-load`) and `DATA_DIR`
(`PROMPT_SET` / `--eval-prompt-data`) alongside `EXP_DIR`
(`--load` / `--save`) across 31 training and example scripts
- Each variable is overridable independently; `MODEL_DIR` and
`DATA_DIR` fall back to `EXP_DIR`, while `EXP_DIR` falls back to
`MODEL_DIR` to preserve the legacy `export MODEL_DIR=/root` flow
- Wire `omni-16xgpu-async` defaults (`HF_CHECKPOINT`, `PROMPT_SET`,
`EVAL_PROMPT_DATA`) through the new vars instead of `/path/to/...`
placeholders so the convention is uniform
---
# 📝 Documentation
## Document the path variable convention
- Add a tip block in `docs/{en,zh}/guide/customize-training.md`
describing the three directories and the fallback chain
- Update the example `--hf-checkpoint` / `--ref-load` snippet to use
`${MODEL_DIR}` instead of `${EXP_DIR}`
…eyes script # 🔩 Chore ## Align launch argument order in deepeyes run script - Remove duplicate `--rollout-num-gpus-per-engine` definition in `examples/deepeyes/run_deepeyes.sh`. - Keep the effective `--rollout-num-gpus-per-engine 2` from the SGLang config section and remove the earlier conflicting value.
# 🐛 Bug Fix ## Preserve DeepEyes multimodal state across partial resume - Keep `sample.multimodal_inputs` as the read-only dataset input instead of appending observation images into the shared shallow-copy reference - Make `_prepare_initial_inputs()` return the initial processor output as local `init_mm_train` without overwriting `sample.multimodal_train_inputs` - Keep observation images in `current_image_data` and append observation processor chunks to `multimodal_train_inputs_buffer` - Route the budget-exhausted early return through `_finalize_sample()` so resumed samples leave `generate()` with merged multimodal train inputs
# ⭐ Feature ## Auto-enable true-on-policy mode in fully-async - Auto-enable `--true-on-policy-mode` in `slime_validate_args` when `fully_async` and `rollout_batch_size * n_samples_per_prompt == global_batch_size`, since the train forward log_probs equal what actor_fwd would produce - Add `ROLES_FULLY_ASYNC_ON_POLICY` (no actor_fwd) and route to it from `process_role` when the mode is on - Recompute `old_log_probs` inline in `policy_loss_function` via `log_probs.detach()`, recovering vanilla PG (ratio ≡ 1) while keeping TIS valid against rollout_log_probs - Drop `log_probs` from required data fields in `MegatronTrainRayActor` and `Advantages` when actor_fwd is absent; treat `rollout_log_probs` as kl-zero template in advantages compute - Make `Controller` fully-async weight-sync skip `actor_fwd` recv when the role is not registered, and skip the actor_fwd HTTP probe in the actor health check - Fall back to `rollout_log_probs` for entropy logging in `log_rollout_data` when `log_probs` is unavailable --- # 🔩 Chore ## Tune training scripts and runtime env - Add `NVSHMEM_BOOTSTRAP_UID_SOCK_IFNAME` default in `scripts/entrypoint/local.sh` - Drop `actor_fwd`/`reference` from resource specs in async scripts (35B/9B text, 9B openr1mm-mm) now that true-on-policy mode handles them - Add `--log-probs-max-tokens-per-gpu` and adjust parallelism/recompute/MoE-dispatcher knobs across qwen3/qwen35/qwen36 scripts to fit larger micro-batches
# 🐛 Bug Fix ## PR #1889: validate MoE HF config when dense layers exist - Add `_has_dense_moe_layers` / `_is_moe_config` helpers in `relax/backends/megatron/arguments.py` - Validate `moe_intermediate_size` and `shared_expert_intermediate_size` - Skip `intermediate_size` check when model is pure MoE with no dense layer - Ref: THUDM/slime#1889 ## PR #1880: fix per-actor HTTP POST concurrency split - Replace node-count-only divisor with `len(nodes) * num_gpus_per_node` - Use ceiling division `(c + n - 1) // n` to avoid losing concurrency budget - File: `relax/utils/http_utils.py` - Ref: THUDM/slime#1880 ## PR #1873: avoid blocking the asyncio event loop on ray.get - Replace `asyncio.to_thread(ray.get, obj_ref)` with direct `await obj_ref` - Removes the hard ThreadPoolExecutor cap on parallel POSTs - File: `relax/utils/http_utils.py` - Ref: THUDM/slime#1873 ## PR #1888: actor save_model must wake_up / sleep, not just reload PG - `save_model` was calling `reload_process_groups()` without resuming `torch_memory_saver`, leaving GPU tensors unreachable when NCCL is triggered during checkpoint save - Use `self.wake_up()` / `self.sleep()` instead so PG and TMS stay in sync - File: `relax/backends/megatron/actor.py` - Ref: THUDM/slime#1888 ## PR #1882: disaggregate PPO must reconnect rollout NCCL across sleep - Add `disconnect_rollout_engines` to `UpdateWeightFromDistributed` - In `actor.sleep()`, tear down weight-sync NCCL group for disaggregate PPO (`use_critic` + not `colocate`) before `destroy_process_groups()` - In `actor.update_weights()`, force `wake_up` + `connect_rollout_engines` + `sleep` when the disconnect path was taken - Drop the buggy `args.use_critic` GPU-offset branch in `sglang_engine.get_base_gpu_id` - Auto-enable `offload_train` when `use_critic` is on - Files: `relax/backends/megatron/actor.py`, `relax/backends/megatron/weight_update/update_weight_from_distributed.py`, `relax/backends/sglang/sglang_engine.py`, `relax/utils/arguments.py` - Ref: THUDM/slime#1882 ## PR #1878: reinitialize critic output_layer when ckpt shape mismatches - Detect missing or shape-mismatched `output_layer.{weight,bias}` in the critic checkpoint metadata before / after `load_checkpoint` - Reinitialize with `normal_(0, 0.02)` for weight and zero for bias, plus `optimizer.reload_model_params()` for fp16/bf16 master sync - Gated on `role == "critic"`, no effect on actor or non-PPO algorithms - File: `relax/backends/megatron/model.py` - Ref: THUDM/slime#1878 --- # ⭐ Feature ## PR #1890: add missing spec / prefix-cache rollout metrics - Wire `_compute_spec_metrics` and `_compute_prefix_cache_metrics` into `compute_metrics_from_samples` - File: `relax/distributed/ray/rollout.py` - Ref: THUDM/slime#1890 --- # 🔩 Chore ## PR #1862: improve `slice_log_prob_with_cp` assert message - Include `len(log_prob)`, `response_length`, `total_length` in the assert so failures are diagnosable - File: `relax/backends/megatron/cp_utils.py` - Ref: THUDM/slime#1862 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# ⭐ Feature ## Add --custom-prompt-path for prompt transformation hook - Add CLI argument in add_data_arguments (arguments.py) - Load custom function via load_function in data_source.py - Thread custom_prompt_func through build_messages, process_raw_sample, BaseDataset, Dataset, and StreamingDataset - Custom function is called after prompt extraction, before conversation/multimodal processing ## Add --image-resize-scale-factor for image dimension alignment control - Add CLI argument in add_data_arguments (arguments.py) - Add image_resize_scale_factor field to MultimodalConfig with from_args propagation and getter function - Update fetch_image with 3-way logic: None uses default patch_factor, 0 disables alignment, positive int uses custom value --- # 📝 Documentation ## Update configuration reference docs - Add --custom-prompt-path to Dataset table (EN + ZH) - Add --image-resize-scale-factor to Multimodal Data table (EN + ZH) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
# ⭐ Feature ## Add hybrid training mode combining async data pipeline with colocate weight sharing - Introduce `--hybrid` flag that sets `fully_async=True` and `colocate=True` so actor/ref/actor_fwd share GPUs via TensorBackuper+_switch_model while rollout runs on a separate GPU placement group with streaming transfer queue - Add `train_hybrid()` method in MegatronTrainRayActor: collects sub-batches from transfer queue, runs ref/teacher/actor forward per sub-batch, merges all sub-batches, computes advantages with correct global normalization, then trains on the full merged batch - Register hybrid mode in `process_role()` to use ROLES_COLOCATE (actor + rollout only, no separate reference/actor_fwd services) - Update controller to skip shared placement groups and fully-async DCS weight sync setup when hybrid is active - Skip actor_fwd health probe in `_check_services_health` for hybrid mode to avoid spurious warnings - Add `train_hybrid()` dispatch in RayTrainGroup and Actor component - Validate argument combinations: `--hybrid` is the supported way to combine async pipeline with colocate weight sharing; bare `--fully-async --colocate` now raises ValueError - Set `offload_train=False`, `offload_rollout=False`, and `compute_advantages_and_returns=True` for hybrid mode - Add Qwen3-4B 8xGPU hybrid-async training launch script
# ⭐ Feature
## Add `relax/utils/visualize` rollout result viewer
- Add web viewer adapted from rlsp/utils/visualize: FastAPI + single-page UI
for browsing `<save>/rollout_result/{train,eval}/{step}.jsonl`, with
step dropdown, sample nav, sort by reward / response_length, sample-info
card, and prompt / response / label rendering with chat-template / tool-call
/ `<think>` highlighting
- Auto-discover `train/` and `eval/` subdirs and render a tab toggle when both
exist; fall back to a single anonymous bucket for flat dirs
- Add terminal UI mode (`--tui`) adapted from redaccel/verl reward_viewer_v2:
sync-load the first step then stream remaining steps via a daemon thread
(newest first), with step / sample / dataset / sort dropdowns, field
filter, fuzzy search (`f`/`enter`/`esc`), vim-style page nav, and
text/table render toggle
- Default theme switched to dark; header shows the Relax wordmark linking
to the GitHub repo plus a GitHub icon
- Mask multimodal pad tokens (`<|image_pad|>`, etc.) by default in the TUI
via `--mask-str`
## Add `relax/entrypoints/visualize` thin wrapper
- One command for both modes: `python -m relax.entrypoints.visualize <dir>`
(web) and `... --tui` (terminal)
- `DATA_DIR` is a required positional argument
- TUI dependencies (`textual`, `rich`) are lazy-imported; clear error
message if missing
---
# 📝 Documentation
## Add bilingual rollout result viewer guide
- Add `docs/{en,zh}/guide/rollout-result-viewer.md` covering data layout,
launch command, flags, page features, terminal UI key bindings, and
reverse-proxy notes
- Register the new pages under the existing "Operations & Debugging" /
"运维与调试" sidebar group in `docs/.vitepress/config.mts`
- Add `docs/public/relax-viewer.png` screenshot (palette-compressed PNG
to stay under the 500 KB pre-commit limit)
# 🐛 Bug Fix
## Fix MODEL_DIR/EXP_DIR initialization in qwen35-9B hybrid-async script
- Replace buggy `EXP_DIR="${MODEL_DIR:=...}"` side-effect assignment with separate `EXP_DIR`/`MODEL_DIR`/`DATA_DIR` defaults, matching `run-qwen35-9B-8xgpu-openr1mm-async.sh`
- Point `--hf-checkpoint` / `--ref-load` at `${MODEL_DIR}` and `PROMPT_SET` at `${DATA_DIR}` so model and dataset roots can be overridden independently of the experiment output dir
(cherry picked from commit 466c779)
(cherry picked from commit 8034d15)
# 🔩 Chore ## Align DeepEyes dataset inputs - Align fp16, GenRM, and partial-rollout scripts with examples/deepeyes/run_deepeyes.sh - Use the same Deepeyes v1 training shards as the main script - Use the same thinklite reasoning accuracy eval slice as the main script (cherry picked from commit df613b4)
# 📝 Documentation ## Add bilingual Hybrid training mode guide - Add `docs/en/guide/hybrid-training.md` and `docs/zh/guide/hybrid-training.md` describing the hybrid execution mode (streaming TransferQueue + in-process TensorBackuper weight sharing) - Cover mode comparison vs Colocate / Fully Async, role layout (`ROLES_COLOCATE` with disjoint actor/rollout placement groups), `--hybrid` flag resolution, and the three-phase `train_hybrid` loop - Document required and optional flags (`--hybrid`, `--num-iters-per-train-update`, `--max-staleness`, `--balance-data`) and the default overrides applied in `relax/utils/arguments.py` - Include the 8-GPU multimodal reference launch from `scripts/training/multimodal/run-qwen35-9B-8xgpu-openr1mm-hybrid-async.sh` and troubleshooting tips (stalled sub-batches, balance-data rejection) - List planned next steps: integrate DCS for weight sync, split `train_actor` by `num_iters_per_train_update` ## Register pages in VitePress sidebar - Add Hybrid Training Mode under the Advanced group in both `en` and `zh` sidebars in `docs/.vitepress/config.mts` (cherry picked from commit 7f78ff7)
# ⭐ Feature ## Add JSON provider config dump - Keep the existing transformer_config.pkl dump for compatibility - Also write transformer_config.json next to it for easier inspection - Convert non-JSON-safe values recursively and fall back to str() when needed (cherry picked from commit 598c15c)
# 📝 Documentation ## Add hybrid mode to bilingual README - Add Hybrid bullet to Highlights section in both README.md and README_zh.md - Add 05/26/2026 News entry pointing to the Hybrid Training guide - Expand Architecture section from two to three execution modes with Hybrid description (separate PG + in-process ref/actor_fwd via TensorBackuper + _switch_model) - Add Hybrid Training doc link to the "Learn more" line (cherry picked from commit 6a262cb)
# ⚡ Performance
## Pre-fault HF safetensors into page cache once per node
- Add `_warm_hf_checkpoint_page_cache(source_path)` in `relax/backends/megatron/checkpoint.py`, invoked from `_load_checkpoint_hf` before `AutoBridge.from_hf_pretrained`
- Eliminates the dominant NFS-mmap small-read bottleneck during `bridge.load_hf_weights` (`aten::cat` was running at ~20 MB/s, accounting for ~65% of init CPU time on 30B-A3B-class MoE models)
- Explicit per-node coordination: `LOCAL_RANK == 0` runs `cat <ckpt>/*.{safetensors,bin} > /dev/null`, other local ranks poll a marker under `/dev/shm`
- Advisory `flock` wraps the rank-0 path so two Relax jobs sharing a host and ckpt do not duplicate the warmup
- Marker lives in `/dev/shm` (tmpfs) so it naturally clears on reboot, avoiding stale-marker / cold-cache mismatches
- Warmup is best-effort: missing path, non-zero `cat` exit, or wait timeout only log a warning, never a correctness gate
- Configurable wait via `RELAX_HF_WARMUP_TIMEOUT_S` (default 1800s)
---
# ✅ Tests
## Reshape repro profiler around the bridge progress loop
- Replace `_maybe_profile` contextmanager in `scripts/tools/repro_megatron_bridge_load.py` with `_install_bridge_progress_profiler` that monkey-patches `MegatronModelBridge._with_progress_tracking`
- Profiles a fixed `RELAX_REPRO_PROFILE_STEPS` window of conversion tasks (default 50) after `RELAX_REPRO_PROFILE_WARMUP` warmup tasks (default 5), then dumps trace/operator-table/stacks/metadata immediately
- Add `RELAX_REPRO_PROFILE_EXIT_AFTER_DUMP` early-exit knob so a long load can be cut short once the profile window is captured
- `scripts/tools/repro_qwen35_moe_bridge_load_tp4pp2.sh`: default `RELAX_REPRO_PROFILE=0`, set `PYTHONPATH=$REPO_ROOT`, default profile dir to `/tmp/relax/profile`
(cherry picked from commit 335001c)
# ✨ Feature - Propagate virtual pipeline size into Megatron-Bridge providers. - Derive vp_stage from Megatron virtual pipeline state for provider wrappers. - Round dynamic microbatch counts up to the VPP group multiple. - Add Qwen3.6-35B 8xGPU VPP trial settings. --- # ✅ Tests - Add focused VPP provider and microbatch rounding regressions. - Verified with focused pytest and pre-commit. (cherry picked from commit 460aa2d)
# 🐛 Bug Fix ## Resume aborted DeepEyes samples by status - Detect aborted samples from sample status and response length - Preserve multimodal rollout state needed for continued generation - Keep off-policy masking controlled by the existing partial-rollout mask flag ## Align resumed generation budgets - Track current-turn generated tokens separately from context budget - Apply the smaller active budget to resumed inference calls - Clear turn-local resume metadata when the turn completes ## Repair rollout prefetch and abort handoff - Wait for aborted samples to return to the buffer before the next fetch - Submit the next synchronous prefetch after transfer tasks complete (cherry picked from commit 867ed47)
# ⭐ Feature ## Add INT4 QAT weight sync pipeline - Add BridgeConverter to unify HF→Megatron weight conversion for bridge and DCS backends - Add fake INT4 quantization CUDA kernel for QAT forward pass - Add compressed-tensors INT4 quantizer processor for weight repacking - Add quantization_config ignore-list augmentation for non-quantized namespaces - Add `--sglang-hf-checkpoint` arg to let INT4 QAT point SGLang at original INT4 weights - Add `--rollout-engine-init-timeout` arg with progress-bar wait for engine startup - Add Kimi K2.6 model config and INT4 training launch scripts (text + multimodal) - Add MoE INT4→BF16 offline cast tool (`relax/tools/quant_cast/convert_moe_int4_to_bf16.py`) ## Add Kimi K2.5-style multimodal processor adapters - Add processor kwargs adaptation for K2.5-style VLM chat processors - Add placeholder expansion and response token sanitization for K2.5 vision tokens - Add multimodal train_inputs remapping for K2.5 pixel_values/grid_thws --- # ♻️ Refactor ## Refactor weight update broadcast into bucketed pipeline - Extract param-info bucketing, GPU loading, PP/EP broadcast into composable functions - Add quantized-weight broadcast phase with metadata encoding for INT4 triplets - Consolidate DCS device_direct backend to reuse BridgeConverter --- # ✅ Tests - Add test_broadcast_quantized for INT4 weight broadcast round-trip - Add test_processing_utils for K2.5 processor adapter functions - Update test_dcs_weight_conversion and test_state_machine for new APIs (cherry picked from commit ec24de0)
# 🐛 Bug Fix ## Make overlap grad/param sync setup idempotent in train() - Relax invokes `train()` once per rollout (upstream Megatron calls it once per run); re-assigning `config.no_sync_func` / `config.param_sync_func` after rollout 0 trips the "no_sync_func must be None" assertion. - Guard the sync-func wiring so it only runs when the slot is still `None` — works for both `--overlap-grad-reduce` and `--overlap-param-gather --align-param-gather`. - Leave forward pre-hooks enabled on exit; disabling them here would empty `DDP.remove_forward_pre_hook_handles` and the next `train()` would `KeyError` on the second `disable_forward_pre_hook` call. - Drop the now-dead `pre_hook_enabled` flag. --- # 📝 Documentation ## Document distributed-optimizer and overlap flags - Add `--use-distributed-optimizer`, `--overlap-grad-reduce`, `--overlap-param-gather` to optimizer tables (EN + ZH). - Add compatibility matrix covering text dense, dense VL (CP=1 vs CP>1), and MoE. (cherry picked from commit 8425002)
(cherry picked from commit f1e8764)
- **cp_utils: cast `get_cp_local_num_tokens` return to `torch.int`**
- Both branches (`cp_size == 1` and `cp_size > 1`) returned float32
(`loss_mask.sum()`), crashing Megatron's `schedules.py:687`
`total_num_tokens += num_tokens` (int32 accumulator) with
`RuntimeError: result type Float can't be cast to the desired output type Int`
- SDPO trainings crashed deterministically at `Actor training step 0`
14+ consecutive times overnight; fixed by matching the accumulator dtype
- **sglang backend: harden router-arg fallbacks and read-only ServerArgs**
- `validate_args`: DP/PP/EP sizes now default to 1 when RouterArgs no
longer exposes them (sglang >= 0.5.17), via `getattr` fallbacks
- `launch_server_process`: copy `ServerArgs` with `dataclasses.replace`
instead of mutating the resolved (read-only) instance
- `_compute_server_args`: disable CUDA graph capture when memory saver
is on (`disable_cuda_graph: args.offload_rollout`) — graphs conflict
with memory saver / tvm-ffi map_dataclass_to_tuple
# 🔧 chore
- **sdpo example launchers: memory-safety flags for non-TE bshd path**
- PERF_ARGS gains `--optimizer-cpu-offload` +
`--use-precision-aware-optimizer` (HybridDeviceOptimizer, Adam state
resident on CPU, ~16GB/rank freed) and `--recompute-granularity full
--recompute-method uniform --recompute-num-layers 1` (whole-layer
activation checkpointing; unfused attention softmax for 4096-token
sequences previously OOM'd with 36GB free after wake_up)
- sciknoweval material/physics + toolalpaca/tooluse examples: bump
`--num-rollout 100`, `--rollout-batch-size 4`, `--n-samples-per-prompt 8`,
`--global-batch-size 32` to match the biology/chemistry configuration
- Rename all 6 SDPO launchers from 2xgpu to 4xgpu - Switch from Qwen3-4B to Qwen3-8B model for all examples - Upgrade TP from 2 to 4, rollout DP=3 + 1 teacher GPU - Add --selective-offload to bypass torch_memory_saver VMM OOM - Bump rollout-batch-size=32, global-batch-size=256 - Set rollout-max-response-len=8192, opd-log-prob-top-k=16 - Add opd-teacher-timeout-s=600 for long generation - Unify PERF_ARGS: optimizer-cpu-offload, precision-aware-optimizer, full recompute, bshd qkv format, micro-batch-size=1 - Add WANDB_ARGS section and eval config to material/physics/tooluse - Generate eval.jsonl via prepare_data.py --eval-ratio 0.1 - sglang backend: harden router-arg fallbacks, read-only ServerArgs, disable CUDA graph when memory saver is on - Add biology training wrapper script for env setup
- update_weight_from_tensor.py: drop EMA error-sync refactor, restore PR-base CPU serialization - loss.py: inline _get_loss_num_tokens, drop copyright header - model_provider.py: restore apply_rope_fusion bridge key - data.py: keep OPD sample-mask folding, revert step_local_sample_counts move - launchers: 4-GPU TP=4 colocate tuning (rollout-batch 8, gbs 32, selective-offload, expandable_segments)
- actor.py: drop actor_ema_weight_updater, _snapshot_student_and_step_ema_teacher, _publish_sdpo_teacher_ema; restore no-arg update_weights() (static only) - components/actor.py, actor_group.py, train_actor.py: drop publish_sdpo_teacher_ema param - teacher_manager.py: drop get_weight_update_engines_and_lock / _teacher_gpu_offsets - tensor_backper.py: drop ema() from TensorBackuper hierarchy - opd_utils.py: drop is_sdpo_teacher_ema_enabled, --sdpo-teacher-update-mode, --sdpo-teacher-ema-alpha and EMA validation; keep prompt routing - launchers: drop --sdpo-teacher-update-mode static / EMA comments - sglang_rollout.py: drop redundant _validate_sdpo_sample (OpdManager.prefill validates) - tests: drop EMA tests, restore test_opd_loss_aggregation.py
# 🐛 Bug Fix ## Repair SDPO launcher arguments - Pass eval prompt dataset name and path as separate argv items (--eval-prompt-data is nargs="+"); a single string hit the legacy aime branch and raised FileNotFoundError at the first eval - Add --attention-backend flash to all colocate launchers (TE fused attention cuDNN crash on first train step) - Set --eval-interval 5 for sciknoweval-biology
- source sdpo/env.sh and drop required WANDB_API_KEY in biology launcher - scale num-rollout/rollout-batch-size/global-batch-size and eval samples
- restore configs/env.yaml (clean upstream template) so the PR does not delete it; local secret-bearing copy stays on disk untracked-by-intent - drop sdpo-only .gitignore entries (configs/env.yaml, wandb/, scripts/training/sdpo/) to match upstream/main - remove tests/utils/opd/test_opd_legacy_regression.py (not needed)
- ruff, ruff-format, mdformat, docformatter, clang-format fixes - covers sdpo README, megatron data.py, opd/metrics/tracking utils
# 🐛 Bug Fix ## Score tool-use trajectories as ordered (Action, Input) pairs - _extract_tool_calls parses (Action, Action Input) pairs in document order instead of merging every input into one flat dict; _golden_tool_call returns the same ordered pair structure - Correctness is now format_ok and predicted pairs == golden pairs, so step order and per-step Action<->Input pairing matter: 37% of the tooluse rows are multi-step and 12% carry the same input key with different values across steps, which the old merged-dict comparison could not distinguish from a correct trajectory # ✅ Tests - Add order-swap, cross-step value-swap and exact multi-step cases
# ♻️ Refactor ## Route every algorithm difference through EnvironmentFeedback hooks - Split feedback implementations per algorithm: feedback.py (base + OPD), opsd_feedback.py (OPSD), sdpo/feedback.py (SDPO); base-class defaults reproduce plain OPD so OPD/MOPD need zero overrides - Delete all 21 is_sdpo branches from OpdManager together with the is_sdpo_feedback / is_sdpo_prompt_routing_enabled reflection; --opd-feedback-class now defaults to OPDFeedback - SDPO preflight (text-only check, stale-payload clearing, teacher prompt presence) moves into SDPOFeedback.prepare_teacher_prompts; teacher fetch/response failures degrade like plain OPD and surface via the assembly-time check_transfer_channels validation - GenerateState takes its feedback from OpdManager; the generate_and_rm_group call sites are unchanged - Rebind example launchers: vision_opd/mopd -> opsd_feedback. OPSDFeedback, sdpo -> sdpo.feedback.*, math_opd uses the default ## Drop the OPD top-K context-parallel slicing - Delete slice_opd_topk_rollout_fields and its call sites; launch validation rejects top-K token selection combined with CP > 1 / dynamic CP / allgather-cp - Restore cp_utils get_cp_local_num_tokens to upstream shape (the int cast lives on fix/cp-num-tokens-int) and restore the stream_dataloader / metrics / tracking churn to upstream - Gate _apply_opd_sample_mask behind use_opd so non-OPD runs skip it # ✅ Tests - Update payload / feedback / arguments tests to the hook contract - Add OPD no-op contract, SDPO escalation hooks, default-class resolution, top-K x CP rejection and SDPO launch validation coverage
# ♻️ Refactor ## Make EnvironmentFeedback concrete with behavioral defaults - Replace the abstractmethod pair with defaults: record_sample_feedback records reward-dict env feedback and prepare_teacher_prompts clears sample.teacher_prompt/opd_sample_mask; OPDFeedback becomes an explicit alias subclass ## Route the OPSD dataset privilege through OPSDFeedback - process_raw_sample stores the rendered teacher prompt under metadata["opd_teacher_prompt"] instead of Sample(teacher_prompt=...) - OPSDFeedback.prepare_teacher_prompts assigns metadata["opd_teacher_prompt"] -> sample.teacher_prompt at rollout time; samples without it keep the OpsdWorker student-prompt fallback ## Run both feedback hooks on every OPD rollout path - Add _record_feedback_and_prefill in sglang_rollout; the group_rm branch and both non-group branches (multi-sample / single-sample) now share record -> prepare -> prefill - Delete SDPOFeedback.record_sample_feedback override and its module helper (base default is equivalent) --- # ✅ Tests ## Pin the new feedback contract - Base recording applies to OPD/OPSD/SDPO subclasses alike - OPSD assigns privilege from metadata; missing key leaves teacher_prompt None - process_raw_sample surfaces teacher prompt as metadata instead of the Sample field
# Conflicts: # relax/backends/megatron/model.py
# 🐛 Bug Fix ## Restore Megatron/sglang-free import isolation - Stub sglang_router and relax.backends.sglang.arguments before importing relax.utils.arguments so the CPU CI runners (no sglang installed) can collect the opd teacher colocate test suite - Keep the sdpo-specific assertions (per-token loss, feedback class, allgather-cp rejection) on top of the restored fixture
# ♻️ Refactor ## Remove unused TopkWorker.TRANSFER_FIELDS - Superseded by topk_transfer_fields(); no consumers remain # ✅ Tests ## Remove ghost attrs from colocate arguments test - Drop opd_mask_on_success and opd_log_prob_dump_dir kwargs that the argument parser no longer defines
# ⭐ Feature ## Unify the sdpo example launchers - Source examples/on_policy_distillation/sdpo/env.sh in every 4xgpu launcher - Add --teacher-sglang-enable-weights-cpu-backup to every launcher - Align training and eval params with the biology script (5000 rollouts, batch 32, 8 samples per prompt, global batch 256, eval every 5 iters with 16 samples per eval prompt) - Wire the eval split into the toolalpaca launcher --- # ♻️ Refactor ## Stop writing unused choices metadata - prepare_data.py no longer emits a never-consumed choices field --- # 📝 Documentation ## Update the sdpo README - Correct model size, resource layout, config table and eval flow - Document the env.sh contract sourced by every launcher
# ⭐ Feature ## Restore SDPO EMA teacher update mode - Recover the EMA teacher path (previously stripped for the static-only PR base) and re-fit it onto the current feedback-strategy architecture - Megatron actor: gate on `is_sdpo_teacher_ema_enabled`, snapshot `actor_ema` in the multi-tag backuper, refresh it after each student step via `_snapshot_student_and_step_ema_teacher`, and publish EMA weights to the managed colocated teacher after the student publish via `update_weights(publish_sdpo_teacher_ema=...)` - TeacherManager: `get_weight_update_engines_and_lock` exposes teacher engines with a Ray weight-sync lock and per-replica GPU offsets - TensorBackuper: add `ema()` to the backuper hierarchy - update_weight_from_tensor: restore error-sync refactor (candidate weight version committed only after cross-rank success, shared abort path) and torch_memory_saver CPU serialization - CLI: `--sdpo-teacher-update-mode` / `--sdpo-teacher-ema-alpha` with validation for managed single colocated teacher, megatron backend, no MOPD/external URL/hybrid/fully-async/LoRA; auto-enables the weights backuper - SDPOFeedback: restore `is_sdpo_feedback` marker used for gating ## Adaptation to current branch - Old prompt-routing validation block is not re-added; those checks now live in `SDPOFeedback.validate_launch_args`, EMA-only checks are layered on top of it --- # ✅ Tests ## Restore EMA test suites - test_sdpo_teacher_weight_sync.py: publish ordering, failure abort, uncommitted-version retry, CPU serialization, snapshot refresh - test_tensor_backuper_ema.py: EMA oracle/alpha/integral-buffer semantics (guarded by CUDA skipif) - teacher_manager engine/lock tests and EMA arguments tests - Guarded megatron imports with importorskip per project convention
# ✅ Tests ## Fix stale method name in EMA actor tests - The recovered EMA tests called `_backup_actor_and_update_sdpo_teacher_ema`, a name that already diverged from the implementation (`_snapshot_student_and_step_ema_teacher`) at the original branch point - Rename the calls; assertions and event ordering unchanged - Verified both tests pass with megatron on PYTHONPATH
# 🐛 Bug Fix ## Fix Float-to-Int cast failure in megatron schedule - Megatron's `forward_backward_no_pipelining` accumulates `total_num_tokens += num_tokens` into an Int tensor; the Float tensor returned by `get_cp_local_num_tokens` raised `RuntimeError: result type Float can't be cast to the desired output type Int` at step 0 of colocate SDPO training - Cast the cp_size == 1 sum and the CP>1 totals to Int; loss-mask sums are integral so the cast is lossless ## Verification - Step-0 failure reproduced on 4xH100 Qwen3-8B colocate SDPO run - Unit check: returns int32 tensor with correct unmasked count
# ♻️ Refactor ## Remove unused teacher image payload builder - Drop `build_teacher_preexpanded_image_data` from `opd_utils`; no remaining callers ## Drop unused sdpo package re-exports - Empty out `relax.utils.opd.sdpo` `__init__`; consumers import from `sdpo.constants` / `sdpo.validation` directly
# ⭐ Feature ## Merge EMA teacher launcher into the biology script - Select teacher update mode (static/ema) by comment in one launcher - EMA active by default (`--sdpo-teacher-update-mode ema`, alpha 0.01); static mode available via the commented alternative flags and naming lines - Removes the need for a separate EMA launcher
# ✅ Tests ## Match RLOO normalizer oracle to Int token counts - `get_cp_local_num_tokens` returns Int since 91b51a9; the float64 mask-sum oracle made `allclose` raise `Int did not match Double` - Cast the expected token count to `torch.int` so the gradient-oracle test matches the production contract
# ✅ Tests ## Guard sdpo teacher weight sync tests on megatron availability - CPU CI has no megatron package; the publish/coordination tests imported `update_weight_from_tensor`, which needs `megatron.core` at module level - Add `pytest.importorskip(megatron.core)` so they skip cleanly instead of erroring (and avoid mp.spawn hangs) in megatron-less environments
# ♻️ Refactor ## Remove GenerateState.feedback alias - Delete the `self.feedback` mirror of `opd_manager.feedback` in GenerateState - Access `state.opd_manager.feedback` directly inside `_record_feedback_and_prefill` - Rollout behavior is unchanged; the OPD injection point stays between rm and prefill
# ♻️ Refactor ## Revert feedback-class flags in existing recipes - Drop `--opd-feedback-class OPSDFeedback` from the three mopd and three vision_opd launchers: none passes `--opd-teacher-prompt-key`, so the flag never had an effect there and MOPD/vision recipes are not OPSD - Restore `--sglang-disable-cuda-graph` in the three mopd launchers - The 2B mopd launcher now matches main byte-for-byte
# ⭐ Feature ## Interface-level feedback parameter schema - Declare the constructor schema once on EnvironmentFeedback (teacher_prompt_key, success_reward_threshold); the selected subclass binds the same --opd-feedback-kwargs dict at construction time - Add load_feedback(path, kwargs) with loud TypeError on unknown fields; OPD default stays OPDFeedback with empty kwargs - OPSDFeedback now requires teacher_prompt_key and validates the sglang teacher path in validate_launch_args - SDPOFeedback consumes success_reward_threshold (default 1.0) instead of a hardcoded score >= 1.0 boundary --- # 🐛 Bug Fix ## Drop the --opd-teacher-prompt-key engine flag - The flag silently lost its effect when the teacher prompt moved to the feedback strategy; the key now enters only via --opd-feedback-kwargs - OpsdWorker.from_args activates OPSD from image keys alone; prompt-key routing is owned by OPSDFeedback --- # ♻️ Refactor ## Collapse empty SDPO domain subclasses - Merge SciKnowEvalSDPOFeedback and ToolUseSDPOFeedback (both were `pass`) into GoldenAnswerSDPOFeedback for static golden-answer text datasets; keep CodeSDPOFeedback as the raising placeholder - Point the six SDPO launchers and the README at the merged class - Ingestion reads teacher_prompt_key from --opd-feedback-kwargs in data_source; rendering and metadata hand-off are unchanged --- # ✅ Tests ## Cover kwargs binding and the merged classes - Add load_feedback binding/error tests and a success-threshold test - Update payload, feedback, and arguments tests for the new flag and class
Sync the feedback-kwargs refactor into the EMA branch: interface-level parameter schema, --opd-feedback-kwargs binding, existing-recipe reverts, and the GoldenAnswerSDPOFeedback consolidation. One semantic fixup on top of the merge: the EMA arguments test now points at the merged class name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
总结
本 PR 在 #237（SDPO 静态 teacher 基座）之上恢复并重接 SDPO EMA teacher 更新模式：teacher
不再只能是冻结 snapshot，每个 actor 优化步后将学生权重按指数滑动平均写入
actor_emasnapshot（
actor_ema ← (1−α)·actor_ema + α·actor），并在学生权重发布到 rollout 之后，经现有
UpdateWeightFromTensor通路把actor_ema发布到 Relax 托管的 colocated teacherSGLang engines。对应原版 lasgroup/SDPO 的
teacher_regularization="ema"+teacher_update_rate（本 PR 暴露为--sdpo-teacher-update-mode ema+--sdpo-teacher-ema-alpha，默认 0.01）。static保持默认，不传 EMA 参数时行为与 #237完全一致。
本 PR 同时包含 #237 的全部改动（feedback 接口层参数化、
--opd-feedback-kwargs、已有recipe 还原等，经 merge
8a4d773同步），以及若干支撑性修复（见「其他」）。Static 与 EMA teacher 的差异
actor_ema滑动平均 snapshot，每步发布TensorBackuper.ema(source_tag="actor", target_tag="actor_ema", alpha)backup("actor")之后同点执行 EMA 步；学生发布 rollout 之后发布 teacherMegatronTrainRayActor._snapshot_student_and_step_ema_teacher()/_publish_sdpo_teacher_ema()UpdateWeightFromTensor，权重 getter 换成actor_ematagupdate_weights(publish_sdpo_teacher_ema=True)OpdManager）TeacherManager拿 engines + Ray 权重锁 + 每副本 GPU offset，连接一次后复用get_weight_update_engines_and_lock()weight_version仅在全 rank 成功后提交；任一失败走共享 abort，版本号不前移，可重试update_weight_from_tensor错误同步重构actor_ema权重 snapshot（CPU backuper）TensorBackuper，自动开启--sdpo-teacher-update-mode ema --sdpo-teacher-ema-alpha 0.01flowchart TD classDef existing fill:#eef3f8,stroke:#64748b,stroke-width:1px,color:#1e293b classDef changed fill:#fee2e2,stroke:#dc2626,stroke-width:2px,color:#991b1b classDef shared fill:#eef3f8,stroke:#2563eb,stroke-width:2px,color:#1e293b OPT["Megatron optimizer step"]:::existing SNAP["_snapshot_student_and_step_ema_teacher()<br/>backup('actor') + ema(actor → actor_ema)"]:::changed PUB["update_weights()<br/>student weights → rollout engines<br/>(UpdateWeightFromTensor)"]:::existing CONN["首次连接：TeacherManager<br/>get_weight_update_engines_and_lock()<br/>engines + Ray Lock + GPU offsets"]:::changed TPUB["UpdateWeightFromTensor.update_weights()<br/>actor_ema → teacher SGLang engines"]:::changed ROLLOUT["next rollout：teacher 以 EMA 权重打分"]:::shared OPT --> SNAP --> PUB --> TPUB --> ROLLOUT TPUB -.-> CONNEMA 步位于学生 snapshot 的同一调用点（原
weights_backuper.backup("actor")），因此静态路径的 snapshot 语义不变；EMA 发布位于学生发布之后、
flush_metrics之前，保证下一步rollout 看到的 teacher 权重与本步学生版本对应。非浮点 tensor 直接 copy（保持 buffer /
整数统计量语义），
alpha=1退化为整份 copy。组件与设计模式
mul_(1-α).add_(α)，零额外显式模型TensorBackuper.ema()；_snapshot_student_and_step_ema_teacher()UpdateWeightFromTensor类，仅换权重 getter 与目标 engines；engines/lock 连接一次缓存_publish_sdpo_teacher_ema()；actor_ema_weight_updater.connect_rollout_engines()Lock、每副本 GPU offset（rollout 区之后按副本偏移）；shutdown 时回收锁TeacherManager.get_weight_update_engines_and_lock()weight_version全 rank 成功才提交；_synchronize_update_error汇聚各 rank 异常，任一失败共享 abort，不留下半提交状态update_weight_from_tensor._synchronize_update_error()/_abort_weight_update()torch_memory_saver下发布走 CPU 序列化路径，避免 resident 权重峰值update_weight_from_tensoruse_opd ∧ mode=ema ∧ group_rm ∧ feedback 类标记 is_sdpo_feedback才启用 EMA 路径is_sdpo_teacher_ema_enabled(args)；SDPOFeedback.is_sdpo_feedback = Trueenable_weights_backuper=True，用户无需显式传validate_opd_args参数与校验
--sdpo-teacher-update-mode=ema的约束（违反即启动报错）：--use-opd+--group-rm+ SDPO feedback 类（is_sdpo_feedback标记）；非 SFT--colocate、单一 Relax 托管 teacher（--teacher-hf-checkpoint）--opd-teacher-routes）、外部 teacher URL、--hybrid、fully-async、LoRA--enable-weights-backuper；EMA-only 校验叠加在 No.41 SDPO实现与复现 #237 的SDPOFeedback.validate_launch_args之上使用方式
biology launcher 单脚本切换模式（EMA 默认开）：
变更
CLI 与兼容性
--sdpo-teacher-update-mode（新增）static（默认，行为同 #237）/ema。--sdpo-teacher-ema-alpha（新增）--enable-weights-backuper文件变更
relax/backends/megatron/actor.pyactor_emasnapshot 初始化、_snapshot_student_and_step_ema_teacher()、_publish_sdpo_teacher_ema()、update_weights(publish_sdpo_teacher_ema=)。relax/utils/training/tensor_backper.pyema()（含 α/shape/dtype 校验；非浮点直接 copy；Noop backuper 明确报错）。relax/distributed/ray/teacher_manager.pyget_weight_update_engines_and_lock()：engines + Ray 权重锁 + 每副本 GPU offset；shutdown 回收锁。relax/backends/megatron/weight_update/update_weight_from_tensor.pyweight_version跨 rank 成功后提交、共享 abort 路径、CPU 序列化；供学生与 EMA teacher 发布共用。relax/utils/opd/opd_utils.pyis_sdpo_teacher_ema_enabled()gating、两个新 flag 及启动校验。relax/utils/opd/sdpo/feedback.pyis_sdpo_feedback类标记（EMA gating 用）。examples/on_policy_distillation/sdpo/run-sciknoweval-biology-4xgpu-colocate.sh验证
可复制命令
新增测试
tests/backends/megatron/weight_update/test_sdpo_teacher_weight_sync.pytests/utils/training/test_tensor_backuper_ema.pytests/distributed/ray/test_teacher_manager.pytests/utils/test_arguments_opd_teacher_colocate.pyimportorskip(megatron.core)干净 skip，不挂 mp.spawn。单元与集成测试结果
tests/utils/+tests/engine/全量test_sdpo_teacher_weight_sync（CPU CI，无 megatron）importorskip）* 2 个失败为
tests/utils/test_megatron_peft_utils.py桥接前缀用例，在不含本 PR 改动的干净 HEAD 上同样失败，属环境预存问题，与本 PR 无关。
实际训练结果
EMA 模式的多步真实训练（
run-sciknoweval-biology-4xgpu-colocate.sh，ema默认档）指标口径与 #237 的 static 基线一致（eval acc / valid distillation ratio /
throughput），便于对照 static vs ema。
风险与回退
已知限制
非 fully-async；MOPD 与外部 teacher URL 不在范围。
actor_ema是常驻 CPU snapshot（multi-tag backuper），内存占用随模型规模线性增加。情况下的发布中途故障恢复依赖「候选版本成功才提交」语义，真实多节点待验收。
风险
test_sdpo_teacher_weight_sync.py的顺序断言覆盖。ema()对非浮点 tensor 直接 copy，测试覆盖。关闭开关或回退方式
--sdpo-teacher-update-mode ema时本 PR 的全部 EMA 路径不激活，行为与 No.41 SDPO实现与复现 #237 完全一致。
其他
8a4d773同步 No.41 SDPO实现与复现 #237（155c4e0）全部改动；两 PR 的评审基线一致。get_cp_local_num_tokens返回值 cast 为torch.int（修复 4×H100 colocate SDPO runstep-0 的 Float→Int 崩溃）；
relax.utils.opd.sdpo的无效 re-export；importorskip守卫对齐。检查清单
weight_version跨 rank 成功才提交，失败共享 abort、可重试。--enable-weights-backuper自动开启，用户无需显式传。tests/utils + tests/engine656 passed（2 个失败为环境预存，干净 HEAD 复现）。More info: issue #86