Skip to content

[Perf][SM70] Integrate AWQ decode defaults with safe capability fallback - #526

Merged
yangzhuxinyzx merged 14 commits into
mainfrom
codex/v100-awq-sep06-integration-20260906-071600
Sep 6, 2026
Merged

[Perf][SM70] Integrate AWQ decode defaults with safe capability fallback#526
yangzhuxinyzx merged 14 commits into
mainfrom
codex/v100-awq-sep06-integration-20260906-071600

Conversation

@yangzhuxinyzx

@yangzhuxinyzx yangzhuxinyzx commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Purpose

Owner-authorized integration of #519/#520/#521/#522 on main, preserving all original commits and author attribution. This is not a competing kernel implementation. Adds default-on M1 admission with fallback for unsupported layer shapes and old native builds, plus the #509 loading-test interaction fix.

Changes

  • Keep the reviewed QPN kernels, dequantization/FP16 boundaries, grouped dispatch and cooperative metadata implementation unchanged.
  • Enable QPN M1 automatically only for the supported TP4/E512/native-g32 geometry and available native operator. No checkpoint-name/path identity admission.
  • Unset flag + unsupported layer/native build retains the old path; explicit 1 fails closed; explicit 0 restores the old path. Restart is required for graph changes.
  • Repair the loading lifecycle test stub for the new grouped flag. Preserve [Bugfix][SM70] Release unused AWQ/NVFP4 MoE conversion cache between layers #509 cleanup in production.
  • Include main's [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494 logical-order QSA repair, which was a baseline dependency of the submitted model comparisons.

Test Plan and Result

At head d01b371, integrated main 96954be:

  • Targeted CPU quantization/admission/loading suites: 118 passed. Run .venv/bin/python -m pytest --confcutdir=tests/quantization -q on test_awq_qpn_sm70.py, test_sm70_awq_active_grouped_decode.py, test_sm70_awq_compact_metadata.py, test_sm70_moe_load_cache_release.py, test_awq_sm70_fallback.py, test_sm70_awq_indexed_prefill.py, test_awq_sm70_persistent_cap.py and test_sm70_awq_prefill_exact_dense.py.
  • Unmodified integration had 67 passes and one failure in the [Bugfix][SM70] Release unused AWQ/NVFP4 MoE conversion cache between layers #509 env stub; fixed, not suppressed.
  • Applicable local pre-commit hooks pass.
  • Fresh CUDA 12.8 / SM70 compile of awq_sm70_gemm.cu and production torch_bindings.cpp passes with ENABLE_SM70_TURBOMIND.
  • Fresh reviewed awq_qpn_m1_sm70.cu compiles and links in an isolated test fragment using the production declaration/schema. This is not a complete release wheel. Initial test-binding header/macro omissions were corrected without changing production CUDA source.
  • Fresh GPU execution is pending available GPU leases. Two CUDA-only cases skipped with CUDA hidden are NOT counted as passes. Existing contributor GPU/graph and independent FP64 evidence remains separately attributed in docs.
  • No full E2E rerun, model deployment, GPU preemption or wheel release.

Performance and numerical decision

Reuse, do not relabel, original same-contract evidence: C1 x 64K pure decode 50.0326 -> 59.0313 tok/s; grouped-route C4/C8 gains are separate measurements and not additive. Compact metadata saves approximately 0.88 GiB/card and its default-change model comparisons preserve throughput within recorded noise. The recorded 65-case task screen and 1152 independently checked local samples support bounded-shape promotion under the owner's policy, not greedy equality or statistical noninferiority. Preserve both the GSM loss and IFEval gain; neither is hidden or declared proof of zero regression. No new speed measurement is claimed.

AI assistance: OpenAI Codex. Integration and merges are explicitly requested by the repository owner; independent human execution of these new tests is not asserted. Draft until final CI and base/head verification.

Leonccaa and others added 14 commits September 5, 2026 01:28
Reuse the existing active-stage operator for TP4 g32 M2-8 decode, align warmup admission, and annotate scratch mutations. Keep C1 and unmatched contracts on existing routes; add focused coverage without new public operators or GEMM kernels.

Validated with 51 GPU-directed tests, 32 dynamic-route operator comparisons, and QSA-fixed matched model runs. Single-run pure-decode gains: C4x64K +5.62%, C8x16K +2.24%. Natural-EOS AWQ outputs match 17/18; the retained and/+ near-tie and W13 rounding differences are not claimed bitwise exact. No precision-default changes or deployment.

Co-authored-by: OpenAI Codex
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Separate implementation-only and existing-autotune gains; document EOS rounding limitations, capacity and startup observations, rollback, and the dual-format C1 follow-up.

Co-authored-by: OpenAI Codex

Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Record the new main integration gate separately from frozen GPU results; keep prefill and grouped decode admission disjoint.

Co-authored-by: OpenAI Codex

Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Co-authored-by: OpenAI Codex

Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Assisted-by: OpenAI Codex

Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Use the retained engine log instead of extrapolating cache block count. No production or benchmark changes.

Co-authored-by: OpenAI Codex <noreply@openai.com>

Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
Add the typed inference operator, prepared-bank decoder, FP32 reduction kernels and standalone GPU tests without changing model dispatch. Keep 3B scalar loading here so cooperative metadata access can be reviewed separately.

Assisted-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
(cherry picked from commit 27a27d3)
(cherry picked from commit 5b4135f)
Gate the native operator to the existing Qwen3.8 TP4 E512 M1 contract, retaining default-off behavior and neighboring routes. Document numerical attribution, observed quality changes, and incomplete production acceptance.

Assisted-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
(cherry picked from commit 37fb5d2)
Evaluate both AWQ arithmetic paths against an independent fixed-weight reference; distinguish existing quantization error, local arithmetic bounds, and task-level quality.

Assisted-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
(cherry picked from commit 6f73eac)
Load 3-byte W13 metadata cooperatively without changing dequantization or reduction order. Keep scalar W2 and the 4-byte path unchanged; document the component-only validation scope.

Assisted-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
(cherry picked from commit 636db1d)
Default VLLM_SM70_AWQ_MOE_COMPACT_METADATA to 1 for the supported Qwen3.8 TP4
E512 native-g32 shape. A default-on request that the build or layer cannot
serve falls back to the 4-byte layout with an info log; an explicit =1 still
fails closed. The decision is a pure helper covered by unit tests.

Same-contract regression: engine-reported KV cache 427,385 -> 497,983 tokens,
C1 x 64K pure decode within 0.2% of the 4-byte layout, C4/C8 within noise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Leonccaa <166551845+Leonccaa@users.noreply.github.com>
…ration-20260906-071600

Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
…integration-20260906-071600

Signed-off-by: yangzhuxinyzx <153831768+yangzhuxinyzx@users.noreply.github.com>
@yangzhuxinyzx
yangzhuxinyzx marked this pull request as ready for review September 6, 2026 07:33
@yangzhuxinyzx
yangzhuxinyzx merged commit a4567d7 into main Sep 6, 2026
2 checks passed
@yangzhuxinyzx

Copy link
Copy Markdown
Contributor Author

已通过完整静态 CI 34019173319 并合并,main 合并提交 a4567d7。最终集成树与本地验证树一致。#520/#521/#522 自动标记 merged。更正提交祖先说明:#519 的等价 kernel 已由后继分支的 9ae1a29 提交携带,但原 PR 的独立提交 5b4135f 不在祖先链,所以 #519 继续 open;没有把它误记成已合并。118 项 CPU 回归和编译结果见正文,不宣称重复运行 GPU/E2E。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants