[liqlos]

Problem and scope

Related Issue: #107. Cooperative-load follow-up to #127.

Q4_K/Q6_K vector decoders reload packed words needed by neighboring lanes.

Implementation

Share those words through explicit eight-lane Q4 and four-lane Q6 shuffles, preserving decoded values, dot-product arithmetic and rounding. Replace the CMake option and sm86 guard with device-profile selection by type and width.

The RTX 3090 built-in dense profile selects Q4_K T1 and Q6_K T1–8. Q4_K T2–8 stay scalar in that profile: the original T4 screen was neutral and T8 regressed. MoE-vector and decode-up have separate calibration. Unconfigured routes stay scalar. A shared plain/fused schedule requires both forms to win.

Verification

RTX 3090 at 350 W, Ubuntu 24.04/WSL2, CUDA 13.1.115, GCC 13.3, Release/sm86, based on e1f2d961. Jonathan Q4_K_M, 4096 prompt + 256 decode tokens, INT8 KV, MTP3, n-gram drafting off, context capacity 65536, prefill chunk 1024, graphs on, prefix reuse off. Both arms used the same lab artifact with legacy tokenizer metadata updated. Tensor bytes were unchanged.

Same executable, scalar versus selected profiles: eight balanced ABBA/BAAB blocks, 32 fresh processes, one warmup and three measured repetitions per process. Speedup = scalar/selected − 1. The 95% CIs use eight block log ratios after averaging repetitions within each process (t, df=7). Load/warmup and profiling are excluded.

Phase	Speedup	95% CI
Prefill (primary)	+0.14%	[−0.15%, +0.43%]
Decode (secondary)	+8.12%	[7.78%, 8.45%]
Whole generation (secondary)	+3.60%	[3.30%, 3.91%]

All eight decode and whole-generation blocks were positive. None were discarded. The original prefill result was −0.106% [−0.199%, −0.012%]. A separate two-block screen was −0.192%. The confirmation CI crosses zero. The earlier slowdown remains unexplained. Zero effect is not established.

Exact dense/MoE scalar-versus-cooperative checks, CPU expert tests, the public GGUF FP64 oracle and a 128-token CLI comparison passed. The direct CPU/GPU expert parity suite was not rerun. Affected CUDA units compile for sm70/75/80/86/89/120a. GPU execution was tested only on RTX 3090. Other contexts and full MoE-model performance remain unmeasured.

Reproduction commands

Recorded test runs used a focused runner linking the public test OBJECT targets, with scalar/cooperative profiles installed in-process. These are the equivalent public entrypoints. The full ninfer_tests executable was not run. The cooperative test selects both routes internally. The standalone GGUF oracle below uses scalar defaults.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DBUILD_TESTING=ON -DNINFER_BUILD_BENCHMARKS=ON
cmake --build build -j --target ninfer_tests ninfer-calibrate ninfer_bench
build/tests/ninfer_tests ninfer_device_profile_test
build/tests/ninfer_tests ninfer_linear_gguf_cooperative_test
build/tests/ninfer_tests ninfer_linear_gguf_test
build/tests/ninfer_tests ninfer_gguf_expert_cpu_test
build/apps/ninfer-calibrate --only gguf_ --out calibrated.json
NINFER_DEVICE_PROFILES="$PWD/ARM-profile.json" build/bench/ninfer_bench \
  --weights "$ARTIFACT" --corpus bench/fixtures/bench_corpus.ids \
  --prompt-gen 4096,256 --repetitions 3 --warmup 1 --max-ctx 65536 \
  --prefill-chunk 1024 --kv-dtype int8 --spec mtp --draft-tokens 3 \
  --ngram-draft-tokens 0 --output json --output-file RUN.json

Set ARTIFACT to the compatible Jonathan artifact. Replace ARM with scalar or selected and RUN.json with a unique report filename. Make scalar/selected copies of calibrated.json, setting all six GGUF routes to scalar through width 8 (width 4 for gguf_moe_up/*). In selected, override gguf_vector/q4_k with [[1,"cooperative"],[8,"scalar"]] and gguf_vector/q6_k with [[8,"cooperative"]]. Other routes must match.

The benchmark command runs one fresh process. For the reported CI, use scalar/selected as A/B, alternate ABBA/BAAB for eight blocks, leave four seconds between processes and save each report separately. Average repetitions per process, form each block’s log ratio from its two scalar and two selected process means, then use a t interval over the eight ratios.

[iamwavecut]

Thanks, great numbers! The shuffles look right, and the explicit masks are exactly what we asked for.

A few changes before we merge:

• Please drop the CMake option and the __CUDA_ARCH__ == 860 check. We'd like this on every card, picked per (type, T) through device-profile routes, like q6_head or q4_linear_swiglu. Your 3090 results can go into its profile (Q6_K T1–8, Q4_K T1), and the calibration tool can measure the route on other cards.
• The MoE decode kernel runs the same Decoder loop. Could you enable it there too? MoE models spend most of their decode time in experts, and the CPU expert parity still holds.
• Prefill is −0.11% with a CI that excludes zero. Any idea where that comes from?

[liqlos]

Thanks! I’ve switched this to device profiles and added calibration for dense and MoE kernels. The CMake option and sm86 guard are gone.

I also reran both routes in the same binary. There’s no statistically significant prefill difference in that run, but I haven’t pinned down what caused the earlier slowdown. I’ve kept the earlier measurements in the description too.

[iamwavecut]

Thanks, this looks good! We added three small commits on top: the GGUF FP64 oracle test now runs Q4_K/Q6_K with both load schedules, a docs/comment cleanup, and RTX 3090 MoE Q6_K seeds from calibration. On a 3090, Qwen3.8-27B GSQ-RCO IQ3_S decode went +0.7% plain and +1.2% with MTP3, with identical greedy output. Merging once CI is green.