Skip to content

peer-ada: the head, GDN and prompt-stage planners refuse a peer of another architecture (mmq_nvfp4.cu joins the sm_89 list)Peer ada arch guard - #473

Open
erkrali wants to merge 2 commits into
jesdga95:mainfrom
erkrali:peer-ada-arch-guard
Open

erkrali wants to merge 2 commits into
jesdga95:mainfrom
erkrali:peer-ada-arch-guard

Conversation

@erkrali

@erkrali erkrali commented Nov 19, 2026

Copy link
Copy Markdown

Closes #462.

-DPEER_ADA=ON compiles for sm_89 the kernels an older card needs to serve as the expert tier: the
expert-window decode kernels and the prompt mmq/iota instances. The three qwen4exp planners that
hand the peer other work never ask what architecture it has — they price from head type, tensor
shapes and measured bandwidth — so with a Blackwell main card and an Ada peer they arm kernels the
peer's binary does not contain. Each one dies the same way, and two of them die during a request.

deepseek41 refuses an older peer in its model layer already (model.cpp: "the DeepSeek peer-tier
kernels are built for Blackwell only", and docs/configuration/gpus.md says so in prose). This is
the same check, in qwen4exp, where its decisions are actually made.

What changed

  1. The planners decline an unlike peer. A card_cc(dev) helper in decode/verify/common.hpp,
    and one comparison at each decision: hs_plan and hs_plan_two return 0, gs_plan_bw returns 0,
    hs_init and gs_init return before they arm anything, and qfn_run clears split_want for the
    pair (its comment already says the layer split is not taken "on three or more unlike cards"; a
    two-card pair was never asked). The head and the mixers stay whole on CUDA0, the prompt keeps its
    layers on the main card, and the peer keeps the job it was built for — its experts' rows of a
    chunk, through the mmq kernels. Putting it in the planners rather than at the launch sites keeps
    the start-time pricing and the hs_peer_bytes() / gs_peer_bytes() accounting in step with what
    will actually run. Each declined decision logs one line: a fallback and a lost function.
  2. mmq_nvfp4.cu joins the peer list. That list covers 18 of the 19 files in
    src/prefill/mmq/instances/, so the one prompt job an Ada peer is contracted to do was
    unavailable to NVFP4 packs specifically. The instance is the Q3_K-layout Q8_1/DQ dot
    (mmq.cuh: "one scale per 16 values"), not the Blackwell FP4 tensor-core path, and it needed no
    change at all: nvcc -gencode arch=compute_89,code=sm_89 on the file alone emits sm_89 SASS for
    mul_mat_q_case<(ggml_type)40>, with the same 614 mul_mat_q symbols mmq_q3_k.cu produces in
    2.41 MB against its 2.45 MB.

They are separate commits so either can be dropped independently.

What it does now

An NVFP4 or a UD-Q4_K_XL pack with a Q8_0 output head loads and serves on a 5060 Ti + 4060 Ti pair
instead of failing at load (q80p_vhead_prepare) or, for the prompt split, partway through a long
request (kv_stream's check() calls std::exit(1), which behind a supervisor reads as an engine
that restarts itself for no reason). The declined work is logged, not silent:

basalt verify: the head stays whole on CUDA0: the peer is not the main card's architecture
basalt verify: the GDN mixers stay whole on CUDA0: the peer is not the main card's architecture
basalt generate: prompt split: off (NVIDIA GeForce RTX 4060 Ti is not the main card's architecture): the tier cards keep their chunks' expert rows

Parity

The guard is inert on every configuration that works today. A single card never reaches a new line:
hs_init and gs_init return on dm_->pdev < 0 first, and qfn_run's new loop has no second card
to look at. A peer of the main card's own architecture takes the same branch it took before, so an
alike pair — the 5090 + 5060 Ti the validated rows run on — is bit-for-bit the old plan. UD-IQ1_S,
the one Ada-peer row on the table, already returned from hs_init on its vproxy before any of the
new lines; its plan is unchanged.

Verification

Measured on a 5060 Ti 16 GB (main) + RTX 4060 Ti 16 GB (peer), PCIe 4.0 x8, Ryzen 7 5700X, 125 GB
RAM, CUDA 13, a -DPEER_ADA=ON build of 63cb68a carrying the first commit: basalt-bench -t prose,conv,struct -r 3, conv / struct / prose t/s. With the guard the two-card run exists at all;
--no-peer-prefill was still needed for NVFP4 before the second commit.

pack one card + Ada peer
NVFP4-DQ6_K 58 ± 2 / 86 ± 0 / 61 ± 1 88 ± 2 / 174 ± 2 / 84 ± 2
UD-Q4_K_XL 42 ± 1 / 63 ± 0 / 44 ± 1 64 ± 2 / 134 ± 1 / 66 ± 1

Both commits then build together: a whole-tree -DPEER_ADA=ON build at this branch is the image this
box runs today, and its binary names sm_120 and sm_89 as it always did — the second commit costs
one object and no build time worth mentioning.

What that image has been asked to do with it is everything else: the no-peer-prefill: true that had been
in the config as a workaround went out the same day, so the peer has been taking its experts' rows of live
prompts on this pair since, on live traffic, with no mmq error since it went in.
That is proof the instance is correct enough to run and not proof it is worth much: the 88 / 174 / 84 above
were measured with the workaround on, and the difference the added rows make is still unmeasured here.
One number from that state, for scale: the run that hit this bug before the fix had reached 90.5 t/s of
conv before it died, against the 88 measured with the peer's prompt share off.

The rows that would settle it are --community ones, which cannot pass the workaround (the preset refuses
engine flags) and so run this path by necessity. We can post both — single-card and Ada-peer — as soon as
#463 lets --community start here at all.

The runs above are informal (no NVML row, so no power limit or clocks with them), and we have run no
quality gate. Both are offered rather than assumed.

Known limit

Nothing makes an Ada peer able to run the head split's or the mixers' kernels; this declines the work
instead of arming it, which is what deepseek41 does and what the peer's VRAM is better spent on.
If the FP4 mmq path ever grows Blackwell-only instructions, mmq_nvfp4.cu will stop being a
one-line addition for sm_89; today it is the same dot mmq_q3_k.cu compiles.

…other architecture

The qwen4exp head split (hs_plan, hs_plan_two, hs_init), the GDN mixer split (gs_plan_bw, gs_init)
and the prompt-time layer split (split_want in qfn_run) choose their peer work from head types,
tensor shapes and measured bandwidth.  None of them asks what architecture the peer card has, and
a -DPEER_ADA=ON build compiles for sm_89 only the expert-window decode kernels and the prompt
mmq/iota instances.  With a Blackwell main card and an Ada peer the planners price work whose
kernels the peer does not have, and the launch fails with "no kernel image is available for
execution on the device": at load for the two-stage head split, at the first decode window for the
exact head split, and for the prompt split during the request, because kv_stream's check() exits.

Gate them on the two cards' compute capability: hs_plan and hs_plan_two return 0, gs_plan_bw
returns 0, hs_init and gs_init return before arming anything, and qfn_run clears split_want for the
pair.  The head and the mixers stay whole on CUDA0 and the peer keeps the job it was built for -
its experts' rows of a prompt chunk, through the mmq kernels.  A peer of the main card's own
architecture is unaffected, and a single card never reaches a new line.

deepseek41 already refuses an older peer in its model layer ("the DeepSeek peer-tier kernels are
built for Blackwell only").  This is the same check placed where qwen4exp decides, so start-time
pricing, hs_peer_bytes()/gs_peer_bytes() accounting and the decode-window dispatch stay consistent
without touching launch sites.  Each declined decision logs one line: a fallback and a lost
function, which is what it is.
The peer list covers 18 of the 19 files in src/prefill/mmq/instances/; NVFP4's instance is the
missing one.  An Ada peer asked to compute its experts' rows of a prompt chunk reads them through a
kernel that was never built for it, and the mmq arm fails at cudaFuncSetAttribute with "no kernel
image is available for execution on the device".  Only NVFP4 packs hit it: every type a UD-Q4_K_XL
pack reads is on the list.

The instance is the Q3_K-layout Q8_1/DQ dot (mmq.cuh: "one scale per 16 values"), not the Blackwell
FP4 tensor-core path, and it needs no change to build for sm_89: nvcc -gencode
arch=compute_89,code=sm_89 on the file alone emits sm_89 SASS for mul_mat_q_case<(ggml_type)40>,
with the same 614 mul_mat_q symbols mmq_q3_k.cu produces in 2.41 MB against its 2.45 MB.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant