Skip to content

[model] fix: Map DeepSeek V4 hash layer config - #1

Closed
bobboli wants to merge 2 commits into
miles-dsv4-direct-hffrom
fix/dsv4-r3-config
Closed

[model] fix: Map DeepSeek V4 hash layer config#1
bobboli wants to merge 2 commits into
miles-dsv4-direct-hffrom
fix/dsv4-r3-config

Conversation

@bobboli

@bobboli bobboli commented Aug 18, 2026

Copy link
Copy Markdown
Owner

Summary

  • map the DeepSeek V4 hash-layer count to the Megatron-Core dsv4_n_hash_layers field
  • read the same field when exporting the Hugging Face config
  • add import and export regression coverage

The previous code assigned moe_n_hash_layers dynamically. The Megatron-LM runtime defines the hash-layer count as TransformerConfig.dsv4_n_hash_layers and reads it in Router._init_routing_mode() to select hash routing. It does not read moe_n_hash_layers.

As a result, the trainer constructed learned routers for the first three layers while SGLang used deterministic hash routing. That caused both the R3 replay failure and the large rollout/trainer mismatch.

Verification

  • uv run python -m pytest tests/unit_tests/models/deepseek/test_deepseek_v4_bridge.py -q: 16 passed
  • targeted pre-commit hooks: passed

@bobboli bobboli added the bug Something isn't working label Aug 18, 2026
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
Signed-off-by: Bo Li <22713281+bobboli@users.noreply.github.com>
@bobboli

bobboli commented Aug 19, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #3.

@bobboli bobboli closed this Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant