LLM170

Pure-Rust LLM inference engine — single track CUDA + W4A16 (int4 sym g128, f16 activations) for the NVIDIA CMP 170HX 64GB (sm_80). serve/infer run W4A16 directories (compressed-tensors / AutoRound) end-to-end on GPU with an int8 KV cache and CUDA-graph decode — 27B dense (g128) and 35B-A3B MoE (g32).

Benchmarks

Environment (CMP 170HX)

item	value
GPU	4x NVIDIA CMP 170HX (GA100, sm_80, 64 GB each)
driver	610.57.04 (unlock patch, pinned)
PCIe	Gen2 x16 all 4
power limit	180 W (fixed)
SM / L2	70 SM; 32 MiB L2; persisting max 20 MiB
host	AMD EPYC 7452 32C/64T, 107 GB RAM

Qwen3.8-27B-W4A16 (g128, dense GDN)

row	pp512	pp1024	pp4096	tg128@4k	np4
CUDA (170HX, 180 W, 2026-10-18)	1258.83	1248.77	1178.92	42.66	59.50

Qwen3.6-35B-A3B-W4A16 (g32, MoE resident)

row	pp512	pp1024	pp4096	tg128@4k	np4
CUDA (170HX, 180 W, 2026-10-20 mma3)	4653.47	4607.67	4356.50	120.41	189.08

Gemma-4-26B-A4B-it-QAT-W4A16 (g32, MoE resident)

row	pp512	pp1024	pp4096	tg128@4k	np4
CUDA (170HX, 180 W, 2026-10-20 attn splits64)	4565.01	4308.14	3652.48	66.76	—

mode (27B, 512tok prompt · 128 gen)	t/s	note
greedy (no spec)	37.4	baseline
MTP	41.6	+11.2%
n-gram	52.6	+40.6% (3회 중 1회 ~42 — 드래프트 고갈)

mode (35B, 512tok prompt · 128 gen)	t/s	note
greedy (no spec)	114.2	baseline
n-gram	114.1	±0% (MoE spec ≈ neutral)

serve (27B)	value
TTFT cold / warm (prefix reuse)	1054 ms / 305 ms (3.5x)
prefix reuse ratio	0.582 (2048 tok, 34% reused)

Build & run

Requires Rust 1.99+. No GPU toolchain needed to build.

cargo build --release

cargo run --release -- serve --model <w4a16_dir> --kv-mem 24G --port 8080
cargo run --release -- bench --model <w4a16_dir> --pp 512,1024,4096 --tg 128 --np 4
cargo run --release -- w4a16-gpu <w4a16_dir> --prompt-tokens 148678,65233,202419 --n-predict 8
cargo run --release -- w4a16-ref <w4a16_dir> --prompt-tokens 148678,65233,202419 --n-predict 8
cargo run --release -- w4a16-load <w4a16_dir>

Repository layout

crates/core        W4A16 loader, CPU reference (quant lane), qwen35 CPU engine
crates/diag        diagnostics (env flags, dumps)
crates/backend-gpu rawcuda — CUDA driver bindings + kernel assets (.cu/fatbin)
crates/server      CLI + HTTP server (continuous batching)

Kernel assets: scripts/build_cuda_kernels.sh (nvcc, sm_80); kernel outputs must match the CPU reference (crates/core/src/quant/lane.rs). Pre-commit gate: scripts/preflight.sh · numeric gate: w4a16-eval (goldens · ΔPPL band · needle).