feat(dflash2): execute the NVFP4-encoded DFlash2 module on the nvfp4full artifact - #13
Open
cometkim wants to merge 5 commits into
Open
feat(dflash2): execute the NVFP4-encoded DFlash2 module on the nvfp4full artifact#13cometkim wants to merge 5 commits into
cometkim wants to merge 5 commits into
Conversation
…ilu_mul route The A16-only policy threw above T=16 because only the fused small-T family was registered. The route now decomposes larger A16 extents into the generic A16 linear plus silu_mul, materializing the gate/up projection in caller workspace; the capacity query accounts for it. W4A4 routes and the fused A16 family through T=16 are unchanged.
The context_kv_materialize replay updates counts, slots, and positions with copies on the legacy default stream and then launches the recorded executable on a cudaStreamNonBlocking stream. Nothing ordered the two, so replays could consume capture-time buffer contents and produce wrong K/V values and footprints - deterministically on this box for the batch-8 and W=1/8/16 replay cells, the first full-suite run after the rebase. A device synchronization after the copies fixes it; the Op and kernels were correct.
The pre-rebase v2 images (fork object names, weight-only NVFP4 matrices) now run the DFlash2 lane through the same schedule as the upstream W8G32_F16S module, dispatched on the module weight format: - ops::linear registers the five weight-only drafter problems (feature projection, qkv, attention output, conv projection, selector) with A16-only routing - AllowA4 throws because the module carries no activation-quant divisor sites. The gemv route expands inline; the small-T expansions live in their own instantiation TU through the shared launch header (the build-speed pattern). - attn_input_proj's three-output form accepts the NVFP4 [6144,5120] parent, writing q/k/v directly through a split-output policy at every positive T (32-token chunks above the small-T family). - rmsnorm_dynamic_grouped_conv_prepare and linear_dynamic_grouped_conv_add accept NVFP4 kernel-projection and projection parents; the prepare route materializes the dynamic coefficients through the generic A16 linear and a dedicated finish kernel, the add route reuses the format-neutral finish extracted from the W8 TU. - context_kv_materialize accepts NVFP4 key/value parents: one MMA family serves every routed column count, staging raw E2M1 codes and applying the stored E4M3 scales in FP32 per 16-value group with the payload divisor folded in. The column mapping, key scratch, fused key head, and key post kernel move to shared headers used by both formats. - candidate_selector_path takes Weight codebooks (BF16_CTRL or NVFP4); the NVFP4 family decodes gathered rows from the registered K16M128x4 blocked scale plane - the pre-rebase selector read that plane row-major, misdecoding every group past the first tile boundary. - the fork-format binder produces an executable DFlash2Plan (payload divisor, input divisor fixed at 1.0F), context key/value become 128-row-aligned NVFP4 row views, and the coherent-proposal layout covers the NVFP4 swiglu scratch across both module formats. Verified against independent FP64 oracles (attn_input, conv pair, context materialize, selector via the BF16 family plus the artifact reconstruction), the fixed context suite, and greedy smokes on both v2 artifacts: nvfp4full DFlash2 5.50 tok/round @ 64.3% and MTP3 both coherent; qat lanes run.
…codings The v2 images were never released, so their module directory is renamed in place to the upstream sections (attention_conv/base_kernel & co., candidate_selector/*) - a one-shot local rewrite; the renamed JSON fits the existing alignment slack, so payload offsets and bytes never move and a header backup stays beside each artifact. The converters already emit the upstream names, so regenerated artifacts match. The binder collapses to one bind_dflash2: matrices, conv projections, and codebooks dispatch per object on the declared format (W8G32_F16S/BF16 or weight-only NVFP4 via the new Binder::declared_format lookup), norms and conv base kernels stay BF16, and the fork-format detection gate and its plan builder are gone. Verified: the patched v2 artifacts reproduce their pre-rename acceptance exactly (nvfp4full DFlash2 5.50 tok/round @ 64.3%, qat 2.50 @ 21.4%, MTP lanes unchanged) and the W8 splice artifact runs the same binder (5.75 @ 67.9%).
Update the artifact row to the current v2 image (19,406,942,468 bytes, SHA abb1e120...) and replace the 'DFlash2 unsupported on nvfp4full' note: the module rides the same image in a weight-only NVFP4 encoding that the unified binder from this branch executes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Unlocks
--spec dflash2on thenvfp4fullartifact, flipping the "DFlash2 is currently unsupported on it" note (46cb1cd).The published nvfp4full v2 image (cometkim/Qwen3.8-27B-nvfp4full-NInfer, now 19,406,942,468 bytes / SHA
abb1e120…) carries its 66-object DFlash2 companion module in a weight-only NVFP4 encoding — matrices only; norms and conv base kernels stay BF16 — while #8 integrated the upstreamW8G32_F16Sschema. This branch adds the NVFP4 module's execution routes.The patch
Four engine commits cherry-picked from cometkim/ninfer
feat/dflash2(stacked directly onto this repo's master), plus a README update:fix(ops): nvfp4 linear_swiglu A16 beyond T=16 through a linear-then-silu_mul routefix(test): order the replay inputs before the non-blocking graph launchfeat(dflash2): the NVFP4 draft module end to end — A16-only drafter linears, the 3-output attention-input projection, the dynamic-conv pair, thecontext_kv_materializefamily, and the selector walk reading the K16M128x4 blocked scale planerefactor(artifact): onebind_dflash2contract across both encodings — per-object dispatch on the declared format viaBinder::declared_format, so upstream-schemaW8G32_F16Sartifacts (groupwise-int / nvfp4) keep working through the same binderUpstream's DFlash2 schedule — the stochastic selector walk, K1..15 × B1..8, both proposal heads — is untouched.
Verified (RTX 5090, sm_120a)
aa64ee53); the only conflict isbindings.cpp, resolved by keeping the nvfp4full binder from feat: port qwen3.8-27b nvfp4full support (target + converter) onto latest master #7 and adding the module weight helperninfer_dflash2_nvfp4_routes_testpassesninfer.exeon the published v2 artifact with--spec dflash2 --draft-tokens 7exits 0 and drafts (greedy smoke prompt: 7.50 tok/round acceptance length)The artifact's model card documents this exact patch set under Engine support.