NPU fast lane: attention reads stale KV rows from the previous request (nondeterministic output)

NPU fast lane: attention reads stale KV rows from the previous request

1bit serve --device npu on Qwen3-0.6B gives different text for identical greedy requests in one server process.

Repro (strixhalo, own NPU runtime, amdxdna 0.17.0 freshly reloaded, no TDRs in the window)

kat through 1bit serve --device npu with cache_prompt: false on every request, --repeat 3: capital, arithmetic and varied-75w give 2 different texts over 3 identical requests.
In-process: calling the lane's generate() repeatedly on the same prompt in one process, the step-0 logits differ from the first call by max |diff| 1.23 to 1.84. The deltas repeat exactly across processes for the same call index, so they come from state carried between calls. Separate processes are bit-identical to each other.

Cause

A probe that zeroes every layer's KV buffer before each generate() makes all repeats bit-identical; zeroing the activation and logits buffers does not. The per-layer KV buffers (npu/lane.cpp, allocated once for kMaxContext) are reused across requests, and attention reads rows past the current position, which still hold the previous request's keys and values.

Fix direction

Prefer the masking the attention control code expects over zero-filling: zero K rows would still add exp(0) terms to the softmax denominator. Verify against an f64 attention reference, gate on kat repeat-determinism, and check the dx route for the same pattern.

🤖 Generated with Claude Code

Status: the cause and the fix direction look right to me, and this changes how the published fast-lane numbers should be read.

Numbers: the fast-lane decode figures from the 2026-10-11 NPU matrix (73.5 / 66.9 / 44.5 tok/s at 128 / 512 / 2048 prompt, about 84 on short chat) were measured with repeated requests in one server lifetime, so some of those requests ran with stale KV rows. Throughput should be unaffected, because the work per token is the same, but output correctness in those runs is unproven. I'll mark them provisional in npu-re until a rerun after the fix.
Gate for the fix: besides the f64 attention reference, run the same in-process generate() repeat probe that found this: step-0 logits must be bit-identical across 10 repeats on the same prompt, in one process, with no KV zeroing. Then rerun kat --repeat 3 through 1bit serve --device npu with cache_prompt: false.
Masking over zero-fill is right for exactness: a zero K row adds exp(0 - max) to the softmax denominator, so zero-filling would only hide the nondeterminism, not remove it.

npu: zero the fast lane's KV rows between generations (#400)

Fixed and verified on the device — merged as `31091dc`

The fix (row-scoped KV zeroing in Lane::begin(), npu/full_elf.cpp/h, tests/npu_full_elf_test.cpp, docs/npu.md) was open as PR #404; it is now merged. Verification ran on strixhalo with the NPU held exclusively (accel0 fuser empty before and after, TDR delta 0, Tctl 69→89 C over the window, load 10-13):

gate	result
G0 — base, pre-fix	`RESULT FAIL: 2 of 2 repeats differ from rep 0`, first differing step 0, max abs diff 2.379 / 2.146 — the probe detects the defect
G1 — fix, 10 in-process repeats, each preceded by a different-length prior (5 … 2000 tokens)	`RESULT PASS: 0 of 9 repeats differ from rep 0`; all ten produce the identical token sequence, bit-identical to rep 0 — since rep 0 is the process's fresh-KV request, this is also the fresh-process bit-identity check
G2 — f64 attention reference	exact causal attention 90.3 % top-1 (KL 0.1646) vs 9.7 % (KL 9.18) for a softmax that counts the zero rows → the lane excludes masked rows, it does not zero-weight them
G3 — `kat` through `1bit serve --device npu`, `--repeat 3`, `cache_prompt: false`	0 of 9 cases nondeterministic (varied-75w/300w/675w/1050w, needle, prefix-reuse); 7/9 `expect_ok`, the 2 failures are expected-string mismatches (`FRANCE` vs the literal `Paris`; arithmetic truncated before `391`), not instability. The serve process held 1 `accel0` fd, i.e. the NPU-selecting route opened the device and served

A correction to this issue's stated rationale

The issue's fix direction — "prefer masking over zero-filling, because zero K rows would still add exp(0) terms to the softmax denominator" — is refuted by the author's own later measurement in the same tree, and G2 confirms which mechanism is real:

the control code already masks scores of rows >= N; writing into K rows past the position changed nothing;
the residual contamination comes from V values in the position's 8-row block-float group, where 8 values share an exponent — masking cannot remove that, only matching a fresh process's zeros can;
hence "with no KV zeroing" cannot literally be met by any host-side fix, and the achievable gate is determinism plus fresh-process bit-identity. That gate passes.

Recorded rather than smoothed over: the merged fix is row-scoped zeroing, not the masking the issue asked for, and the reason is above.

dx route

Code-level answer only: dx::Decoder is persistent with KV zeroed once at construction and every request restarts at position 0 — the same structural pattern. Whether it manifests cannot be settled without a dx runtime probe, which this window did not have. Stated as an open item, not as a pass.

Evidence: team/evidence/400-npu-kv-stale.md, raw logs in team/evidence/400-raw/v400c/ (including the first launch's thermal abort at its own 92 C guard).

Navigation Menu

Repository navigation

NPU fast lane: attention reads stale KV rows from the previous request (nondeterministic output) #400