Phase 5a: int8 weight-only quantization

[bsreecharanreddy]

Summary

• Designs and implements Phase 5a: self-computed, per-output-channel int8 weight-only quantization extending the Triton grouped-GEMM kernel itself (src/dispatch/kernels/quantization.py, grouped_gemm_int8.py) -- not sourced from bitsandbytes/AWQ, per the design doc's explicit scope decision.
• Kernel-level correctness gate passed on a real NVIDIA L40 (RunPod Secure Cloud): the int8 kernel's first real-hardware run, 15/15 GPU tests against an independent quantize-then-dequantize reference; the existing bf16 kernels re-verified at 25/25.
• Measured on the real deepseek-ai/deepseek-moe-16b-base model (bf16, single L40, unbatched eager decode): int8 kernel +70.9% over stock -- a throughput tie with the bf16 naive kernel it's built on (-1.2%), as expected since weight-only quantization saves memory bandwidth, not FLOPs. 49.89% expert-weight memory reduction. Perfect model-level top-1/mutual-top-k agreement at every tested position, stronger than the design doc's own bar required.
• Three real bugs found and fixed, none caught by earlier per-task review:
  • Found live during the GPU rental: patch_moe_infer_quantized quantized each layer's weights into int8 but never freed the original bf16 weights (stack_expert_weights re-points every expert's Linear.weight at a view into a shared bf16 tensor), so the model held both copies at once and OOMed a 44GB L40. Fixed with _free_expert_weights plus a regression test (and, after the final review flagged it as insufficient, a second test using a weakref to prove the underlying tensor is actually collected, not just the views).
  • Found by the final whole-branch review: quantize_per_channel_int8 computed its arithmetic in the weight's own bf16/fp16 dtype rather than float32, widening round-trip error (bf16: up to 1.5x the ideal bound).
  • Found by the same review: the zero-channel NaN guard clamped every small-magnitude channel's scale toward a dtype-dependent floor, not just the all-zero case -- for fp16 specifically, this silently threw away int8 range on real, nonzero, CLI-reachable data (--dtype float16 --moe-kernel quantized).
• One gap recorded honestly rather than hidden: the design doc's mutation check on the kernel (the same bar Phase 1 met) wasn't run this session -- documented as deferred, not fabricated.
• README, CLAUDE.md, and CHANGELOG.md all refreshed -- CLAUDE.md and CHANGELOG.md had drifted stale since Phase 1 and were caught by the final review.
• $1.9543 of the $5 cap, across three pods including two RunPod Community Cloud pods that hit a real host-level GPU passthrough bug (cuInit() returning CUDA_ERROR_UNKNOWN even against the base image's own stock torch) before a Secure Cloud pod worked.

Test plan

• make check green (lint, mypy --strict, 123 tests) throughout every commit
• Kernel-level correctness gate passed on real hardware: 15/15 int8, 25/25 bf16
• Model-level agreement measured and reported exactly as it came back (perfect, not assumed)
• Two real precision bugs found by the final whole-branch review, both fixed with regression tests proving they have teeth
• Findings, cost records (all three pods), memory-footprint measurement, and runbook committed: docs/findings/2026-09-17-phase-5a-quantization-run.md