You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature] Compact IntDim-L/G experts and add a ragged Qwen vLLM GPU backend #29
IntDim-L/G can keep different neuron counts in each expert, but the current exporter only supports uniform-width structural pruning. Zero masking retains the original dense matrix shapes, so stock vLLM does not skip those computations.
Scope
Implement a first GPU-only Qwen MoE path on the pinned unified environment (PyTorch 2.13.0+cu130, Transformers 5.17.0, vLLM 0.29.0):
Versioned checkpoint metadata describing each layer's per-expert widths and compact weight layout.
Physical compaction using the existing calibration-based IntDim-L/G selection, with an explicit compatible HF reload path.
An opt-in vLLM model integration and a Triton GPU expert kernel that reads the actual expert widths; preserve routing, attention and shared experts.
Start with BF16, single GPU / TP=1; reject unsupported configurations explicitly.
Acceptance
GPU tests compare compact expert outputs, full-model logits and reloaded weights against the corresponding zero-mask reference.
Cover both layer and global selection, unequal/unaligned widths and invalid metadata.
Run vLLM inference on B200 using a small Qwen MoE fixture, then a complete pretrained Qwen MoE checkpoint if resources permit. Report fixture and pretrained evidence separately, including exact environment, settings and limitations.
Record reproducible commands and committed test code. Successful inference does not by itself establish a throughput speedup.
Related: #25, #9. Existing uniform-width IntDim-E and stock runtime behavior must remain available.
Activity