Repository navigation
peer-ada: the head, GDN and prompt-stage planners refuse a peer of another architecture (mmq_nvfp4.cu joins the sm_89 list)Peer ada arch guard - #473
Open
erkrali wants to merge 2 commits into
Conversation
…other architecture
The qwen4exp head split (hs_plan, hs_plan_two, hs_init), the GDN mixer split (gs_plan_bw, gs_init)
and the prompt-time layer split (split_want in qfn_run) choose their peer work from head types,
tensor shapes and measured bandwidth. None of them asks what architecture the peer card has, and
a -DPEER_ADA=ON build compiles for sm_89 only the expert-window decode kernels and the prompt
mmq/iota instances. With a Blackwell main card and an Ada peer the planners price work whose
kernels the peer does not have, and the launch fails with "no kernel image is available for
execution on the device": at load for the two-stage head split, at the first decode window for the
exact head split, and for the prompt split during the request, because kv_stream's check() exits.
Gate them on the two cards' compute capability: hs_plan and hs_plan_two return 0, gs_plan_bw
returns 0, hs_init and gs_init return before arming anything, and qfn_run clears split_want for the
pair. The head and the mixers stay whole on CUDA0 and the peer keeps the job it was built for -
its experts' rows of a prompt chunk, through the mmq kernels. A peer of the main card's own
architecture is unaffected, and a single card never reaches a new line.
deepseek41 already refuses an older peer in its model layer ("the DeepSeek peer-tier kernels are
built for Blackwell only"). This is the same check placed where qwen4exp decides, so start-time
pricing, hs_peer_bytes()/gs_peer_bytes() accounting and the decode-window dispatch stay consistent
without touching launch sites. Each declined decision logs one line: a fallback and a lost
function, which is what it is.
The peer list covers 18 of the 19 files in src/prefill/mmq/instances/; NVFP4's instance is the missing one. An Ada peer asked to compute its experts' rows of a prompt chunk reads them through a kernel that was never built for it, and the mmq arm fails at cudaFuncSetAttribute with "no kernel image is available for execution on the device". Only NVFP4 packs hit it: every type a UD-Q4_K_XL pack reads is on the list. The instance is the Q3_K-layout Q8_1/DQ dot (mmq.cuh: "one scale per 16 values"), not the Blackwell FP4 tensor-core path, and it needs no change to build for sm_89: nvcc -gencode arch=compute_89,code=sm_89 on the file alone emits sm_89 SASS for mul_mat_q_case<(ggml_type)40>, with the same 614 mul_mat_q symbols mmq_q3_k.cu produces in 2.41 MB against its 2.45 MB.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #462.
-DPEER_ADA=ONcompiles forsm_89the kernels an older card needs to serve as the expert tier: theexpert-window decode kernels and the prompt
mmq/iotainstances. The three qwen4exp planners thathand the peer other work never ask what architecture it has — they price from head type, tensor
shapes and measured bandwidth — so with a Blackwell main card and an Ada peer they arm kernels the
peer's binary does not contain. Each one dies the same way, and two of them die during a request.
deepseek41refuses an older peer in its model layer already (model.cpp: "the DeepSeek peer-tierkernels are built for Blackwell only", and
docs/configuration/gpus.mdsays so in prose). This isthe same check, in qwen4exp, where its decisions are actually made.
What changed
card_cc(dev)helper indecode/verify/common.hpp,and one comparison at each decision:
hs_planandhs_plan_tworeturn 0,gs_plan_bwreturns 0,hs_initandgs_initreturn before they arm anything, andqfn_runclearssplit_wantfor thepair (its comment already says the layer split is not taken "on three or more unlike cards"; a
two-card pair was never asked). The head and the mixers stay whole on CUDA0, the prompt keeps its
layers on the main card, and the peer keeps the job it was built for — its experts' rows of a
chunk, through the
mmqkernels. Putting it in the planners rather than at the launch sites keepsthe start-time pricing and the
hs_peer_bytes()/gs_peer_bytes()accounting in step with whatwill actually run. Each declined decision logs one line: a fallback and a lost function.
mmq_nvfp4.cujoins the peer list. That list covers 18 of the 19 files insrc/prefill/mmq/instances/, so the one prompt job an Ada peer is contracted to do wasunavailable to NVFP4 packs specifically. The instance is the Q3_K-layout Q8_1/DQ dot
(
mmq.cuh: "one scale per 16 values"), not the Blackwell FP4 tensor-core path, and it needed nochange at all:
nvcc -gencode arch=compute_89,code=sm_89on the file alone emits sm_89 SASS formul_mat_q_case<(ggml_type)40>, with the same 614mul_mat_qsymbolsmmq_q3_k.cuproduces in2.41 MB against its 2.45 MB.
They are separate commits so either can be dropped independently.
What it does now
An NVFP4 or a
UD-Q4_K_XLpack with a Q8_0 output head loads and serves on a 5060 Ti + 4060 Ti pairinstead of failing at load (
q80p_vhead_prepare) or, for the prompt split, partway through a longrequest (
kv_stream'scheck()callsstd::exit(1), which behind a supervisor reads as an enginethat restarts itself for no reason). The declined work is logged, not silent:
Parity
The guard is inert on every configuration that works today. A single card never reaches a new line:
hs_initandgs_initreturn ondm_->pdev < 0first, andqfn_run's new loop has no second cardto look at. A peer of the main card's own architecture takes the same branch it took before, so an
alike pair — the 5090 + 5060 Ti the validated rows run on — is bit-for-bit the old plan.
UD-IQ1_S,the one Ada-peer row on the table, already returned from
hs_initon itsvproxybefore any of thenew lines; its plan is unchanged.
Verification
Measured on a 5060 Ti 16 GB (main) + RTX 4060 Ti 16 GB (peer), PCIe 4.0 x8, Ryzen 7 5700X, 125 GB
RAM, CUDA 13, a
-DPEER_ADA=ONbuild of63cb68acarrying the first commit:basalt-bench -t prose,conv,struct -r 3, conv / struct / prose t/s. With the guard the two-card run exists at all;--no-peer-prefillwas still needed for NVFP4 before the second commit.Both commits then build together: a whole-tree
-DPEER_ADA=ONbuild at this branch is the image thisbox runs today, and its binary names
sm_120andsm_89as it always did — the second commit costsone object and no build time worth mentioning.
What that image has been asked to do with it is everything else: the
no-peer-prefill: truethat had beenin the config as a workaround went out the same day, so the peer has been taking its experts' rows of live
prompts on this pair since, on live traffic, with no
mmqerror since it went in.That is proof the instance is correct enough to run and not proof it is worth much: the 88 / 174 / 84 above
were measured with the workaround on, and the difference the added rows make is still unmeasured here.
One number from that state, for scale: the run that hit this bug before the fix had reached 90.5 t/s of
conv before it died, against the 88 measured with the peer's prompt share off.
The rows that would settle it are
--communityones, which cannot pass the workaround (the preset refusesengine flags) and so run this path by necessity. We can post both — single-card and Ada-peer — as soon as
#463 lets
--communitystart here at all.The runs above are informal (no NVML row, so no power limit or clocks with them), and we have run no
quality gate. Both are offered rather than assumed.
Known limit
Nothing makes an Ada peer able to run the head split's or the mixers' kernels; this declines the work
instead of arming it, which is what
deepseek41does and what the peer's VRAM is better spent on.If the FP4
mmqpath ever grows Blackwell-only instructions,mmq_nvfp4.cuwill stop being aone-line addition for
sm_89; today it is the same dotmmq_q3_k.cucompiles.