Skip to content

Optimize orthogonal CQ4 LM head layout - #662

Closed
ncylich wants to merge 1 commit into
v2from
feature/cq4-interleaved-lmhead
Closed

ncylich wants to merge 1 commit into
v2from
feature/cq4-interleaved-lmhead

Conversation

@ncylich

@ncylich ncylich commented May 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR adds first-class support for full-width orthogonal CQ4 LM heads stored in the existing 4-row interleaved layout.

Key changes:

  • Add a fast orthogonal CQ4 INTERLEAVED_4ROW LM-head matmul path for decode.
  • Support tied token embedding lookup from the same interleaved orthogonal CQ4 weight file.
  • Teach the CQ writer/QDQ reader to explicitly preserve orthogonal interleaved storage.
  • Prefer the interleaved CQ4 orthogonal layout for token embeddings, decoder token embeddings, and output heads in converter policy.
  • Add a temporary conversion helper for patching existing full-width orthogonal CQ4 artifacts into the interleaved layout without a full reconversion.
  • Remove reliance on the old separate LM-head-only path by routing through the general CQ4 interleaved layout.

Validation

Ran:

  • PYTHONPATH=/Users/noahcylich/Documents/Desert/cactus-v2/python pytest python/cactus/convert/tests/test_cq.py python/cactus/convert/tests/test_policy.py python/tests/test_transpile_weight_compat.py -q
  • cactus-graph/build/test_ops
  • cactus-graph/build/test_io
  • cactus-kernels/build/test_matmul
  • cactus build --android
  • Local e2e generation sanity checks for temp interleaved Gemma, LFM, and Qwen bundles using Explain shrodingers cat.
  • Pixel Gemma shared-KV short generation using the patched interleaved tied embedding/LM-head file.
  • Pixel Gemma shared-KV 256-token prompt / 32-token decode benchmark comparing interleaved and row-major tied weights with the same rebuilt branch binary.
  • Final focused rerun after review edits: cactus-graph/build/test_ops and python -m py_compile python/cactus/convert/interleave_orthogonal_cq4.py.

Generation checks produced coherent outputs. On the Pixel run, interleaved was slightly faster than row-major in the same slow device/build path (decode_tps: 2.67 vs 2.15; prefill_tps: 9.57 vs 9.24).

Add support for using the existing 4-row interleaved CQ4 layout with full-width orthogonal output heads. The LM-head path now dispatches to an optimized interleaved decode kernel while preserving the scalar fallback for unsupported shapes.

Teach CQ weight export, QDQ, and runtime loading about explicit INTERLEAVED_4ROW storage for orthogonal CQ4 tensors. The loader validates malformed interleaved tensors strictly, including metadata byte counts, embedding lookup can read tied interleaved weights, and the converter policy selects this layout for large output heads and tied token embeddings.

Add a temporary artifact patching tool for rewriting existing full-width orthogonal CQ4 weight files to the interleaved layout without running a full reconversion. This keeps current artifacts testable while the converter pipeline is updated.

Validation: pytest python/cactus/convert/tests/test_cq.py python/cactus/convert/tests/test_policy.py python/tests/test_transpile_weight_compat.py -q; cactus-graph/build/test_ops; cactus-graph/build/test_io; cactus-kernels/build/test_matmul; cactus build --android; local e2e generation checks for Gemma, LFM, and Qwen temp interleaved bundles; Pixel Gemma shared-KV generation and token benchmark with interleaved and row-major tied weights.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
@ncylich
ncylich force-pushed the feature/cq4-interleaved-lmhead branch from 61b7bc2 to 4b0ccb3 Compare May 26, 2026 00:33
@ncylich ncylich closed this May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant