Skip to content

hy4flash: improve single-GPU NVFP4 decode on hybrid CPUs - #466

Merged
jesdga95 merged 6 commits into
jesdga95:mainfrom
giveen:codex/hy4flash-performance
Nov 19, 2026
Merged

jesdga95 merged 6 commits into
jesdga95:mainfrom
giveen:codex/hy4flash-performance

Conversation

@giveen

@giveen giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Changes

  • Use the shared CPU pool sizing logic for single-GPU HY4-Flash NVFP4 on hybrid CPUs. The benchmark host selected 15 workers instead of 23. Explicit worker overrides still take precedence.
  • Retry a failed prompt ring allocation with one additional 256 MiB expert-tier loan before lending the full tier. Reset the prompt counters at the first chunk and measure Engram row waits and normalization separately.

Performance

A cold basalt-bench A/B/B/A comparison on one RTX 5090 at a 475 W power limit used HY4-Flash NVFP4, seed 42, and 4,715 fixed expert-cache slots. Baseline ad274aa8dc was compared with ee70bd4183.

basalt-bench --model models/Hy4-flash-NVFP4.basalt -t conv -p 16384 --seed 42 -- --expert-cache 4715
Metric Baseline mean Patched mean Change
Conversation decode 35.48 tok/s 55.43 tok/s +56.2%
Decode window 57.01 ms 39.23 ms −31.2%
Decode passes 17.87/s 25.50/s +42.7%
Prefill (14,155 actual tokens) 3,550.65 tok/s 3,578.34 tok/s +0.8%

The individual conversation runs were 31.01 / 39.94 tok/s at baseline and 57.39 / 53.46 tok/s with the patch. The prefill difference was within the observed run-to-run spread.

The full default seeded suite on the patched build at 475 W and 4,920 automatically sized slots reached 58.55 tok/s conversation, 74.59 tok/s structured, 57.79 tok/s prose, and 3,602.79 tok/s engine prefill. The changes are in the host runtime, so a CUDA kernel microbenchmark does not apply.

Validation

  • All 12 selected Release tests passed, including HY4-Flash kernel parity, CPU pool and expert-tier tests, and prompt chunk invariance on the real NVFP4 model.
  • A pinned-tier greedy replay across 85 verify windows produced identical picks for 256 tokens. A 128-token real-model generation also passed.
  • With the first prompt ring allocation forced to fail, the bounded retry retained 2,494 of 4,715 expert slots and completed prefill and decode.

Reuse the startup sizing policy and respect explicit worker and affinity settings.
In a fixed-tier ABBA on one RTX 5090 at 475 W, the conversation fixture went
from 64.7/49.3 to 38.8/39.7 ms per window (15.4/20.3 to 25.8/25.2 passes/s).
The 85-window frozen-tier replay kept the same greedy picks.
If the first ring allocation fails, try one more segment before lending
the entire cache. A forced failure on NVFP4 retained 2,494 of the 4,715
original expert slots and completed prefill and decode. Cover repeated
bounded lending and retained bytes in the tier test.

Start counters with the first prompt chunk and report Engram reader
wait and row normalization separately.
@giveen

giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author
image

@jesdga95 jesdga95 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, a few changes before I can merge.

  1. Please move the worker sizing into the expert tier's default (the pool_workers fallback in src/core/expert_tier.cpp) instead of Hy4's tier_spec, using the same auto_pool_workers sizing Flash-Next gets. DeepSeek uses the same tier, and a hybrid CPU is short on workers regardless of quant or peer cards, so the NVFP4 only and no peer conditions can go. That also drops the engine_session link and the program include from the Hy4 model.
  2. The ring retry looks good, keep it. I have a change landing in the same open_prompt code soon, so a rebase might be needed.
  3. Please split the Engram timing changes into a separate PR so this one is just the fix.

For reference, I measured similar numbers to your patched run on my 5090 alone without the patch, so it looks like this mostly brings hybrid CPUs in line with everyone else. Still a great catch!

- The fallback in ExpertTier::open now runs program::auto_pool_workers, the sizing Flash-Next's plan applies: a hybrid CPU gets its P-cores minus the host thread plus half the E-cores, and the cgroup quota caps it. DeepSeek uses the same tier and a hybrid CPU is short on workers whatever the quant or the peer cards, so the per-model NVFP4 and no-peer conditions go with the Hy4 tier_spec block.
- engine_session moves from model_hy4flash to expert_tier, the target that names auto_pool_workers.
- The tier test starts on 15 workers here (8 P-cores + 16 E-cores), the default docs/configuration/index.md already names.
The prompt log keeps the one Engram figure it had; the split into waiting for rows and normalizing, and its scoping to the prompt, move to a separate PR so this one is only the ring retry.
@giveen

giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

All three, pushed as 154437fdc and fefa395d7:

  1. The worker sizing moved into the tier. ExpertTier::open's pool_workers fallback now runs program::auto_pool_workers with the same inputs and affinity mapping Flash-Next's plan uses, so a hybrid CPU gets its P-cores minus the host thread plus half the E-cores, capped by the cgroup quota (15 workers here, 8 P + 16 E; the old fallback said 23). The NVFP4 and no-peer conditions are gone with the tier_spec block, so DeepSeek gets the same pick. engine_session moved from model_hy4flash to expert_tier, and the auto_plan/cpu_quota includes left the Hy4 model. docs/configuration/index.md already named this default, so the engine now matches it.
  2. The ring retry is untouched.
  3. The Engram timing is now its own PR: hy4flash: the prompt line splits the Engram's host step into waiting for rows and normalizing #468. This one is only the fix.

The diff against main is now 7 files, +60 −22 (the retry plus the tier sizing). Verified: full cmake --build build -j clean, expert_tier_test and auto_plan_test pass.

Two notes. The branch log carries the two earlier commits whose Engram timing and Hy4-side sizing are superseded by the last two commits; no amends or force-pushes here, so the final diff is the thing to read. And you were right about the measurement: the patched run is mostly bringing a hybrid CPU in line, not beating a plain one.

Shared code stays model-agnostic (model_layering): the comment said which model's plan applies this sizing, and the checker reads it as shared code naming a model. It now names the plan (program::auto_plan) instead.
@jesdga95

Copy link
Copy Markdown
Owner

Thanks, this looks much better! CI is red because of the model_layering test, the new comment in expert_tier.cpp names Flash-Next and shared code can't name a model. Rewording it fixes that. One small ask while you're there, could you move auto_pool_workers down next to detect_cpu_topology in kernels/cpu instead of linking engine_session into expert_tier? It's a pure function so it should be an easy move, and it keeps the tier from pulling in the program layer.

auto_pool_workers moves from the program's auto plan to kernels::cpu, next to physical_cores and detect_cpu_topology, as one pure function over the machine's counts and the cgroup quota. program::auto_plan keeps its entry point as a one-line delegation, so the plan's callers are unchanged, and expert_tier calls the kernel rule directly: the tier no longer links engine_session or includes the program layer.
@giveen

giveen commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Both done, as 14a3d76f1 and 88984fa73.

  • The layering test: the comment named Flash-Next in shared code; it now names the plan instead, and tools/check_model_layering.py passes locally.
  • auto_pool_workers moved to kernels::cpu (src/kernels/cpu/pool.cpp, declared in pool.hpp next to physical_cores/detect_cpu_topology), as one pure function over the counts + cgroup quota + affinity. program::auto_pool_workers is now a one-line delegation, so choose_auto and its callers are unchanged; ExpertTier::open calls the kernel rule directly and the engine_session link and the program include both leave expert_tier.

Verified: full cmake --build build -j clean, expert_tier_test ok (15 workers here), auto_plan_test ok, layering check clean.

@jesdga95
jesdga95 merged commit 465af81 into jesdga95:main Nov 19, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants