Skip to content

cuda: emit banked HC MMQ activation images - #152

Draft
GenerelSchwerz wants to merge 27 commits into
reference/upstream-kernels-20261001from
kernel/hc-banked-mmq-emission-20261008
Draft

GenerelSchwerz wants to merge 27 commits into
reference/upstream-kernels-20261001from
kernel/hc-banked-mmq-emission-20261008

Conversation

@GenerelSchwerz

@GenerelSchwerz GenerelSchwerz commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Change

HC_POST normalization and eligible affine gates can emit banked MMQ activation images directly, retaining the original F32 outputs and producing F16/BF16 images when an existing reader needs them. Readers requiring different D4, DS4 and D2S6 layouts can now share the same producer instead of running another activation converter for each layout. These layouts serve multiple weight quantizations; this is not limited to FP4 weights.

The multiple-layout extension adds 120 lines and removes 12 across three existing CUDA files. Its separately committed prerequisite adapts the published #133 (77fb0d934e2f10e4d5b4e00f87e89f675a018d32) bundle helpers, including their #130 prerequisite, to this branch: 279 additions and 20 deletions. The original banked implementation depends on #149 (15b12ec519711b001b6485c317e39478b5109be9) and its prerequisites. These are explicit dependencies, not additive speedups. The PR base remains reference/upstream-kernels-20261001 at official upstream dcd387a412ca54e172a8d60eb71ef6753850c8ca.

Coverage and limits

  • Dynamic bank/sample/stream dimensions; identity or combination HC_POST, weighted normalization, eligible SCALE and affine gates. Compatible whole-buffer reshapes, flat/chunk normalization, padding and intervening independent work are covered. Repeated readers share each existing layout image; three layout slots do not limit reader or expert count.
  • The new banked bundle uses four-element writes with the existing per-layout quantization primitives. Normalization accumulation, scalar expressions, rounding and positive-zero padding retain their original behavior. Existing banked single-image device functions and matrix readers are unchanged.
  • Admission uses the existing NVIDIA capability, small-grid and bounded normalization checks. Banked multiple-image emission additionally requires reader width at least 256. MMVQ priority, large grids, wide frames, biased SCALE, incompatible/partial readers, mutations, live intermediates and concurrency exclusions retain their existing paths. Physical ranges, NONE inputs, input versions and standard allocation dependencies remain authoritative.
  • Floating readers that select MMF consume F32 directly. Typed producer images are created only for readers whose original selector requires them, including the separately checked cuBLAS cases.
  • This runs through ordinary upstream graph allocation, dispatch and capture. Private executor/allocator integration, routed expert images, native FP4 images, complete HC pipeline fusion and MTP policy are outside this leaf. No model-name whitelist is introduced.

Matched component measurements

RTX 5070 Ti. The exact prior PR152 plus the adapted published bundle prerequisite is the control; only the new banked leaf differs. Three rotated AB/BA pairs per graph, with the full captured producer/reader graph timed:

Component Less GPU time Faster pairs
N512, plain, two layouts 5.24% 3/3
N512, combination, three layouts 8.73% 3/3
N1024, SCALE, two layouts 5.48% 3/3
N2560, three layouts 7.72% 3/3
Chunk normalization, three layouts 8.84% 3/3
Affine, three layouts 6.88% 3/3
N320, partial quantization block, two layouts 10.50% 3/3

The complete utility gate contains 24 graphs and 72 pairs. All 15 eligible cases were faster in 3/3 pairs; two N256 cases were near flat at 0.09% and 0.31%. Nine unchanged controls stayed within 0.047%. These are component measurements, not serving, native-implementation parity, Flash Next, MTP, older-GPU, universal or additive speedup claims.

Validation

  • Normal upstream scheduler allocation/capture, 24 utility graphs, 21 further boundary graphs and two actual cuBLAS typed-image graphs. 188 fresh CPU/control datasets and 188 matching candidate datasets; original outputs and retained inputs match byte for byte, with the existing CPU error limit of 5e-4 unchanged.
  • 744 complete quantized consumer-image checks, including padding/metadata, plus 16 complete F16/BF16 image checks against the original CUDA converters. Typed images are inspected after the producer and before its original consumers.
  • Five focused memcheck/racecheck/synccheck jobs pass. Existing RMS_NORM, RMS_NORM_MUL_ADD and DSV4_HC_POST backend tests pass with both libraries and fusion enabled/disabled: 384 test executions.
  • Both changed modules audited: 827 pre-existing device functions, their resource records and 2,411,504 encoded instruction words are exact. The 72 added functions report LOCAL0; five have 8..32-byte stack frames. Eight host/norm translation units compile for SM75/80/89/90; older-device runtime is untested.
  • Whole-suite ordered build/GPU locks; -j18 with two heavy build slots; verified hard 12 GiB host memory, zero swap, process/time limits and complete owned-process teardown. Existing validated objects were reused. No model server was launched.

Failed fixture checks and their diagnoses are preserved. The first typed probe correctly found that MMF readers did not require typed images; a later standalone-kernel name classification error was corrected in the probe. Completed controls were adopted rather than repeated. Neither required production changes or relaxed tolerances.

Exact commands, source/object/library hashes, original outputs, CPU references, image bytes and teardown proofs are retained in results/upstream-kernel-series-20261001/HC-BANKED-MULTILAYOUT-PRODUCER-20261011/normal-source4/isolation-v1/FINAL-QUALIFIED-V1.json and FINAL-RESULTS-V1.md. Earlier single-image evidence remains under HYPERCONNECTION/HC-BANKED-MMQ-EMISSION in the same benchmark workspace.

Requirements

  • Contributing guidelines reviewed, subject to the repository owner's maintained-fork policy.
  • AI usage disclosure: YES. Codex implemented, tested and reviewed these changes under explicit fork-only owner authorization. The owner retains review and maintenance responsibility and handles any upstream submission.

Carry the reviewed kernel sources from PR125, PR129 and PR132 as the
matching control for the separate tensor-affine HC follow-up.

Assisted-by: Codex
Preserve gate intermediates, original rounding and live HC outputs.
Retain strict input alias checks and standard allocation dependencies.

Assisted-by: Codex
Use the exact tested control sources from published PR130, PR129 and PR132.

Assisted-by: Codex
Reuse the allocated affine match to find the planned HC_POST and retain prepared emission priority. Reject expanded normalization cuts whose terminal is not written by this kernel.

Assisted-by: Codex
Assisted-by: Codex
Adapt the published PR133 and PR130 bundle helpers to the PR152 parent.
Keep the original banked single-image API and writer unchanged.

Assisted-by: Codex
Reuse the original normalization and quantization primitives to prepare
D4, DS4 and D2S6 images for eligible banked consumers in one producer.
Keep existing single-image kernels and conservative dispatch contracts.

Assisted-by: Codex
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant