Repository navigation
hy4flash: improve single-GPU NVFP4 decode on hybrid CPUs - #466
Conversation
Reuse the startup sizing policy and respect explicit worker and affinity settings. In a fixed-tier ABBA on one RTX 5090 at 475 W, the conversation fixture went from 64.7/49.3 to 38.8/39.7 ms per window (15.4/20.3 to 25.8/25.2 passes/s). The 85-window frozen-tier replay kept the same greedy picks.
If the first ring allocation fails, try one more segment before lending the entire cache. A forced failure on NVFP4 retained 2,494 of the 4,715 original expert slots and completed prefill and decode. Cover repeated bounded lending and retained bytes in the tier test. Start counters with the first prompt chunk and report Engram reader wait and row normalization separately.
jesdga95
left a comment
There was a problem hiding this comment.
Thanks for this, a few changes before I can merge.
- Please move the worker sizing into the expert tier's default (the pool_workers fallback in src/core/expert_tier.cpp) instead of Hy4's tier_spec, using the same auto_pool_workers sizing Flash-Next gets. DeepSeek uses the same tier, and a hybrid CPU is short on workers regardless of quant or peer cards, so the NVFP4 only and no peer conditions can go. That also drops the engine_session link and the program include from the Hy4 model.
- The ring retry looks good, keep it. I have a change landing in the same open_prompt code soon, so a rebase might be needed.
- Please split the Engram timing changes into a separate PR so this one is just the fix.
For reference, I measured similar numbers to your patched run on my 5090 alone without the patch, so it looks like this mostly brings hybrid CPUs in line with everyone else. Still a great catch!
- The fallback in ExpertTier::open now runs program::auto_pool_workers, the sizing Flash-Next's plan applies: a hybrid CPU gets its P-cores minus the host thread plus half the E-cores, and the cgroup quota caps it. DeepSeek uses the same tier and a hybrid CPU is short on workers whatever the quant or the peer cards, so the per-model NVFP4 and no-peer conditions go with the Hy4 tier_spec block. - engine_session moves from model_hy4flash to expert_tier, the target that names auto_pool_workers. - The tier test starts on 15 workers here (8 P-cores + 16 E-cores), the default docs/configuration/index.md already names.
The prompt log keeps the one Engram figure it had; the split into waiting for rows and normalizing, and its scoping to the prompt, move to a separate PR so this one is only the ring retry.
|
All three, pushed as
The diff against Two notes. The branch log carries the two earlier commits whose Engram timing and Hy4-side sizing are superseded by the last two commits; no amends or force-pushes here, so the final diff is the thing to read. And you were right about the measurement: the patched run is mostly bringing a hybrid CPU in line, not beating a plain one. |
Shared code stays model-agnostic (model_layering): the comment said which model's plan applies this sizing, and the checker reads it as shared code naming a model. It now names the plan (program::auto_plan) instead.
|
Thanks, this looks much better! CI is red because of the model_layering test, the new comment in expert_tier.cpp names Flash-Next and shared code can't name a model. Rewording it fixes that. One small ask while you're there, could you move auto_pool_workers down next to detect_cpu_topology in kernels/cpu instead of linking engine_session into expert_tier? It's a pure function so it should be an easy move, and it keeps the tier from pulling in the program layer. |
auto_pool_workers moves from the program's auto plan to kernels::cpu, next to physical_cores and detect_cpu_topology, as one pure function over the machine's counts and the cgroup quota. program::auto_plan keeps its entry point as a one-line delegation, so the plan's callers are unchanged, and expert_tier calls the kernel rule directly: the tier no longer links engine_session or includes the program layer.
|
Both done, as
Verified: full |

Changes
Performance
A cold
basalt-benchA/B/B/A comparison on one RTX 5090 at a 475 W power limit used HY4-Flash NVFP4, seed 42, and 4,715 fixed expert-cache slots. Baselinead274aa8dcwas compared withee70bd4183.The individual conversation runs were 31.01 / 39.94 tok/s at baseline and 57.39 / 53.46 tok/s with the patch. The prefill difference was within the observed run-to-run spread.
The full default seeded suite on the patched build at 475 W and 4,920 automatically sized slots reached 58.55 tok/s conversation, 74.59 tok/s structured, 57.79 tok/s prose, and 3,602.79 tok/s engine prefill. The changes are in the host runtime, so a CUDA kernel microbenchmark does not apply.
Validation