NPU

NPU (Strix Halo XDNA 2)

The NPU is one shared device: 0000:c6:00.1, 8 columns, AIE2P tiles. This page records measured facts. Every number has a committed source, linked in its row.

What the hardware multiplies natively

From the toolchain's own AIE API header (mlir_aie/include/aie_api/detail/mmul.hpp, table of matrix-multiply modes, XDNA2 row):

Operand widths	Native modes	MACs per instruction
8b x 4b (int8 activations x int4 weights)	4x16x16	1024
8b x 8b	4x8x8, 8x8x8	256 / 512
16b x 8b	4x4x8, 8x4x8, 4x8x8, 2x8x8	<= 256
bf16 x bf16	4x8x8	256

There is no ternary, 2-bit or 1-bit multiply mode. The consequences:

• 4-bit is the NPU's sweet spot. int4 weights are the narrowest the hardware multiplies, at twice the int8 rate. Below 4 bits, weights have to be unpacked on-core, which saves bytes but buys no math.
• A ternary model converts to 4-bit exactly. A weight in {-1, 0, +1} x s is exact in int4 and in q4_1 (min -s, step s, nibbles 0/1/2). The only loss is how the scale is stored.

Bonsai-8B in 4-bit on the NPU

Bonsai-8B is a 1-bit model. Packed as 4-bit Q4NX, it ran through 1bit-MONSTER's npu-infer full-ELF int8 x int4 path on 2026-09-23, compared with an independent CPU reference decoded from the same container:

argmax ("The capital of France is")	12095 " Paris", matches
correlation with the reference	0.9984 (Qwen3-8B on the same path: 0.9916)
top-10 overlap	10/10

That harness is a correctness test, at 280 s/token. A serving path for 8B still needs a runlist lane like the Qwen3-0.6B one below. One trap: Bonsai pads only its lm_head to 151680 rows, and its embedding keeps the real 151669.

The fast lane (serving path in this repo)

npu/ runs Qwen3-0.6B as one runlist per token (28 layers plus the lm head) on full ELFs generated in C++, with no xclbin (PRs #8, #9):

• Accuracy: logits bit-identical to the reference lane, 24/24 steps.
• Speed: 11.0 ms/token (91 tok/s), and 87 tok/s served through Lemonade's onebit recipe.

Our own layer kernel (dx)

Our own layer kernel (dx) is in development in a private repository. Numbers will be published when it lands.

Qwen3.6-35B-A3B MoE on the NPU (closed source)

Engine PR #85; docs/npu.md, "Private routes". The route is a private add-on (1bit-MONSTER/npu-kernels), built into 1bit with -DONEBIT_NPU_PRIVATE. Measured on Strix Halo:

• Parity: against the fp64 reference, it passes on 3 positions (argmax 846 / 198 / 3710).
• Speed: 16.3-16.5 tok/s decode and about 21 tok/s prompt.
• Serving: 1bit serve --device npu answers "The capital of France is Paris."

On the same box Vulkan decodes the Q8_0 GGUF at 53 tok/s. So for this model the NPU is a concurrent second stream: about 11 tok/s alongside a Vulkan run, drawing about 75-80 W less package power.

Reading an ERT timeout

The NPU is shared, and one faulting kernel makes the driver's watchdog reset every hardware context. Before blaming a kernel, read the watchdog dump:

journalctl -k | grep -E "IO_PAGE_FAULT|tdr_work|Dumping ctx"
aie2_dump_ctx: Dumping ctx ctx.<pid>.1, hwctx 15, sub=12011, comp=12010   <- still progressing: collateral
aie2_dump_ctx: Dumping ctx ctx.<pid>.1, hwctx 14, sub=1, comp=0           <- stuck on its first job

• Progressing (sub = comp + 1) means the context was killed by someone else's hang.
• Stuck at sub=1, comp=0, right after an IO_PAGE_FAULT, marks the one that caused it.

See Working on the shared NPU.