English · 简体中文
📚 Engineering notes (Chinese) · 🐋 mini-DeepInfer · 🌟 Merged PRs
I write inference code, debug numerical issues, inspect performance traces, and turn what I learn into engineering notes.
Lately, most of my work revolves around SGLang, Ascend 910C, and DeepSeek:
- 🧩 Understanding scheduling, KV caches, MoE, and speculative decoding.
- 🔬 Debugging operator precision, caches, and graph execution.
- 🚀 Working on expert load balancing, memory use, and inference performance.
- ✍️ Building small runnable projects and writing guides to go with them.
LLM inference engineer · SGLang & Ascend NPU contributor
SGLang #47574 · Dynamic EPLB on Ascend 🎉 Merged · 2026-11-02
Added dynamic expert load balancing on Ascend to SGLang, connecting valid-route statistics, expert-weight migration, and bounded temporary memory.
Follow-up: SGLang #47595 · Open
| Area | Change | PR |
|---|---|---|
| DeepSeek-V4 memory | Bound decode SWA storage by request slots | SGLang #47740 |
| Numerical precision | Fix Softplus accuracy in GDN gating and decode | sgl-kernel-npu #933 |
| Sparse attention | Exclude padded lanes from non-power-of-two TopK reduction | sgl-kernel-npu #935 |
| Quantization | Preserve AWQ fallback weight dtype; correct GPTQ zero-point offset | #47639 / #47637 |
| Benchmark metrics | Record PD decode handoff time for non-streaming requests | SGLang #47772 |
These PRs were Open when checked on 2026-11-02. Follow each PR for its latest status.
📦 More sglang contributions (12 PRs)
- #47721 · Bound DeepSeek-V4 RoPE tables by serving context · Open
- #47720 · Bound DeepSeek-V4 prefill query intermediates · Open
- #47718 · Release temporary DeepSeek-V4 RoPE tables · Open
- #47713 · dsv4: Reuse verify sequence lengths · Open
- #47665 · dsv4: Generate compression positions with stepped arange · Open
- #47663 · dsv4: Reuse CPU query lengths for prefill metadata · Open
- #47659 · MLA: preserve NZ cache data during CPU retraction · Open
- #47656 · moe: Honor SwiGLU clamp for BF16 and INT8 experts · Open
- #47654 · sparse-kv: Preserve shared prefixes in physical host slots · Open
- #47647 · dsv4: Avoid unused packed indexer cache allocation · Open
- #47634 · attention: Slice MLA latent K after TP all-gather · Open
- #47629 · moe: honor simple TopK scoring and routing scale · Open
📦 More sgl-kernel-npu contributions (8 PRs)
- #936 · include dtypes in legacy add RMSNorm cache key · Open
- #934 · compute weightless RMSNorm in float32 · Open
- #932 · normalize strided gating inputs · Open
- #930 · preserve tiling configurations across graph replays · Open
- #926 · include sparse attributes in the tiling cache key · Open
- #925 · handle cache-location windows larger than 16 tokens · Open
- #924 · handle fully masked leading tiles in argmax probability · Open
- #922 · preserve arch22 accumulation order across batch sizes · Open
All upstream PRs · Merged upstream PRs
Learn DeepInfer, one request at a time.
A small, runnable teaching inference engine: explore scheduling, batching, and KV slot reclamation on CPU, then generate text with real Qwen3.5 weights on Ascend 910C while learning execution across iterations and speculative decoding.
From getting models to run to understanding how they run.
Chinese engineering notes on SGLang internals, DeepSeek-V4 Attention / MoE / DSpark, Ascend deployment, profiling, and tuning, with code, measured results, and the reasoning behind each conclusion.
Let's talk inference, read some code, or catch a bug together 🐋
DeepSeek whale girl · Community fan art