test(qwen36): tier invariants on the fake backend, and the reservation leak they found

[kreuzzelg]

Stacked on #1344 (crichalchemist's three tier fixes). Its three commits show in this diff until it lands; the one commit of ours is d0ecbf1. I will rebase the moment #1344 merges.

Why

#1339, #1340 and #1341 were each found by reading, and #1344 gives each a regression test that pins its scenario. This adds the rules those scenarios are instances of, checked directly against tests/qwen36_fake_cuda.h in the plain CPU build — no GPU, no toolkit — so the next drift fails in make test-c (and under test-asan) before anyone reads the tier again. The tier is mine (#713); two of the three bugs were mine; these are the tests I should have shipped with it.

The three rules

1. Budget accounting balances. On every device used <= budget, and once the queue is drained used == resident × exp_bytes: every reservation is consumed by an upload or handed back. Checked on one and two devices, across an LFRU swap (budget-neutral by construction, so used must not move), and across the path where a planned expert is reported without weights.

2. Shutdown wakes every waiter at once. All four cv_take sleepers — the uploader's victim wait, qt_note_block, qt_note_planned, qt_fill_wait — parked simultaneously behind a full queue and an open group. qt_shutdown has to bring every one of them home, under a watchdog thread rather than alarm() so the test runs on MinGW. Afterwards no slot may still read as queued, and the abandoned swaps' victim keeps its tensor and its resident flag.

3. Issue geometry under random routing. Random resident sets, random K up to the row limit, one to three devices, 200 seeds each: every device block inside the replica buffer, pairwise disjoint, at its device's slot; the mask names exactly the routed experts that were resident on a device whose issue succeeded; hits + misses add up to everything routed. The row limit is read from the array the rows index (sizeof G.is_k[0] / sizeof G.is_k[0][0]), so the stride cannot drift from it without failing here. Under ASan this is a fuzz for the #1339 class.

What rule 1 found, and the fix

qt_plan_fill reserves budget and sets planned=1. qt_note_planned returned early on NULL weights without undoing either, and the warmstart did not call it at all when the loader came back empty (if (planned[gi] && wg)). The bytes stayed out of the budget for the life of the process, and if(resident||queued||planned) continue never reconsidered the expert. #1331 was this leak for every expert of an int8 container — the class survived its fix.

Fix, 19 lines: qt_note_planned hands the reservation back when it receives no weights; tier_warmstart reports every planned expert, with or without them.

Verified

test_qwen36_tier_invariants with the fix	ok; ASan+UBSan: 0 diagnostics
the same test without the fix	8 FAIL, all the leak; 0 sanitizer diagnostics — a red test, not a crash
the four #1344 tests, test_qwen36_ctx, make qwen36	unchanged, ok
make test-c (the full suite, ~120 binaries)	ALL PASS, 8 m 43 s

Two notes from writing it, kept in the file:

• The watchdog's first draft passed its timeout by pointer into the arming function's frame. ASan flagged the stack-use-after-return — fix(pilot): the prefetch worker outlives the Model it dereferences #1277's class — in the test before it could flag anything in the tier. Static state now, disarmed after a successful shutdown.
• "Queue empty" is not "uploader done": the last dequeued expert is still queued=1 until its upload returns. The test waits for both. test_qwen36_tier_multidev checks residency right after qt_fill_wait() and passes on timing; worth the same two-line wait there, @crichalchemist, if you are touching the file for the watchdog anyway.

Not in here on purpose: the QT_MAX_ROWS constant and the int8 warmstart-line / docs correction I suggested on #1344 — they belong there, and here they would collide.

🤖 Generated with Claude Code