[Perf][SM70] Integrate AWQ decode defaults with safe capability fallback - #526
Merged
yangzhuxinyzx merged 14 commits intoSep 6, 2026
Merged
Conversation
Reuse the existing active-stage operator for TP4 g32 M2-8 decode, align warmup admission, and annotate scratch mutations. Keep C1 and unmatched contracts on existing routes; add focused coverage without new public operators or GEMM kernels. Validated with 51 GPU-directed tests, 32 dynamic-route operator comparisons, and QSA-fixed matched model runs. Single-run pure-decode gains: C4x64K +5.62%, C8x16K +2.24%. Natural-EOS AWQ outputs match 17/18; the retained and/+ near-tie and W13 rounding differences are not claimed bitwise exact. No precision-default changes or deployment. Co-authored-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Separate implementation-only and existing-autotune gains; document EOS rounding limitations, capacity and startup observations, rollback, and the dual-format C1 follow-up. Co-authored-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Record the new main integration gate separately from frozen GPU results; keep prefill and grouped decode admission disjoint. Co-authored-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Assisted-by: OpenAI Codex Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Use the retained engine log instead of extrapolating cache block count. No production or benchmark changes. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Add the typed inference operator, prepared-bank decoder, FP32 reduction kernels and standalone GPU tests without changing model dispatch. Keep 3B scalar loading here so cooperative metadata access can be reviewed separately. Assisted-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> (cherry picked from commit 27a27d3) (cherry picked from commit 5b4135f)
Gate the native operator to the existing Qwen3.8 TP4 E512 M1 contract, retaining default-off behavior and neighboring routes. Document numerical attribution, observed quality changes, and incomplete production acceptance. Assisted-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> (cherry picked from commit 37fb5d2)
Evaluate both AWQ arithmetic paths against an independent fixed-weight reference; distinguish existing quantization error, local arithmetic bounds, and task-level quality. Assisted-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> (cherry picked from commit 6f73eac)
Load 3-byte W13 metadata cooperatively without changing dequantization or reduction order. Keep scalar W2 and the 4-byte path unchanged; document the component-only validation scope. Assisted-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com> (cherry picked from commit 636db1d)
Default VLLM_SM70_AWQ_MOE_COMPACT_METADATA to 1 for the supported Qwen3.8 TP4 E512 native-g32 shape. A default-on request that the build or layer cannot serve falls back to the 4-byte layout with an info log; an explicit =1 still fails closed. The decision is a pure helper covered by unit tests. Same-contract regression: engine-reported KV cache 427,385 -> 497,983 tokens, C1 x 64K pure decode within 0.2% of the 4-byte layout, C4/C8 within noise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
…ration-20260906-071600 Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…integration-20260906-071600 Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
yangzhuxinyzx
marked this pull request as ready for review
September 6, 2026 07:33
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Owner-authorized integration of #519/#520/#521/#522 on main, preserving all original commits and author attribution. This is not a competing kernel implementation. Adds default-on M1 admission with fallback for unsupported layer shapes and old native builds, plus the #509 loading-test interaction fix.
Changes
Test Plan and Result
At head d01b371, integrated main 96954be:
Performance and numerical decision
Reuse, do not relabel, original same-contract evidence: C1 x 64K pure decode 50.0326 -> 59.0313 tok/s; grouped-route C4/C8 gains are separate measurements and not additive. Compact metadata saves approximately 0.88 GiB/card and its default-change model comparisons preserve throughput within recorded noise. The recorded 65-case task screen and 1152 independently checked local samples support bounded-shape promotion under the owner's policy, not greedy equality or statistical noninferiority. Preserve both the GSM loss and IFEval gain; neither is hidden or declared proof of zero regression. No new speed measurement is claimed.
AI assistance: OpenAI Codex. Integration and merges are explicitly requested by the repository owner; independent human execution of these new tests is not asserted. Draft until final CI and base/head verification.