fix(qwen36 tier): qt_fill_wait returns only when the last upload is resident

[crichalchemist]

Root-cause fix for the race @kreuzzelg found in #1360 ("queue empty is not uploader done") and covered there on the test side.

The defect

qt_fill_wait waited for the upload queue to drain (qn == 0). The uploader frees the ring slot when it dequeues an entry, before it calls the backend, so the queue is empty while the last expert is still being copied to the device: the caller sees it queued=1, resident=0. The engine's warmstart frees the RAM int8 copy of every planned expert right after qt_fill_wait, trusting that they are all in VRAM.

The fix

Count what is in flight, not what is queued. enqueue_locked increments G.inflight; the uploader decrements it only once the slot is resident (or the upload failed, or the entry was abandoned at shutdown) and broadcasts cv_take; qt_fill_wait blocks on inflight. Same lock, same condition variable, one extra broadcast per completed upload, nothing new on the decode path.

The test, in the shape #1344 set

One commit, one test that fails before the fix. tests/test_qwen36_tier_fill_wait.c makes the race deterministic instead of one-in-fifteen: the fake backend gains a third hook, fake_upload_hook, the test sleeps 3 ms per tensor in it, and reads queued/resident under G.mx the instant the wait returns.

	before	after
test_qwen36_tier_fill_wait	fails 5/5 (all three assertions)	passes 10/10
test_qwen36_tier_multidev, poll removed	—	passes 20/20
fill_wait, multidev, shutdown under ASan+UBSan	—	clean

The poll bdeb338 added to the multidev test is removed: the wait is now the guarantee. The other tier tests and make qwen36 are unchanged and green.

Relation to the other open PRs

• Merges cleanly with qwen36 tier: QT_MAX_ROWS, placement pin, ownership-based int8 free (follow-up to #1344) #1388.
• qwen36: Vulkan expert tier, and staged device-local uploads for cards without Resizable BAR #1338 carries the same inflight counter as part of its Vulkan work (the Vulkan backend has no completion signal of its own, so it needed one). Once this lands I will rebase qwen36: Vulkan expert tier, and staged device-local uploads for cards without Resizable BAR #1338 onto it and drop its copy, which shrinks that PR.

Verified on macOS 13 x86_64 (2017 iMac), Apple clang, fake CUDA backend.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[kreuzzelg]

Reviewed as the author of the tier and of the #1360 poll this removes. The fix is the right shape and I checked the accounting against every exit of the uploader.

Accounting. Every entry that enqueue_locked counts leaves through exactly one of the two decrements: the abandon-at-shutdown branch (th_stop && issue_open) or the common tail after the three uploads, which also covers the "device genuinely full" failure. qt_shutdown sets th_stop and joins the uploader, and the uploader only returns on qn == 0, so nothing stays in the ring uncounted; a waiter parked in qt_fill_wait at shutdown leaves on th_stop as before. The extra cv_take broadcasts land on while-guarded waits (queue space, issue_open), so spurious wakeups cost nothing. Nothing on the decode path changes.

On real hardware, two cards (RTX 3070 + Quadro RTX 4000, driver 580.173, CUDA 12.0), the PR applied on dev 6f9117f:

	result
test_qwen36_tier_fill_wait	10/10
test_qwen36_tier_multidev (poll removed)	20/20
shutdown, int8_engine, invariants, autoplace, fp8, int8	all pass
CI's qwen36_tiny fixture, int8 container (--ebits 8), COLI_CUDA=1, default warmstart, ./qwen36 8 8 ref_full.json	40/40 runs, 16/16 tokens, uploads 64 / miss(CPU) 0
same, int4 container (--ebits 4), ./qwen36 8 4	20/20 runs, 16/16 tokens

One caveat for anyone repeating the hardware runs: on dev as of today the default warmstart segfaults about one run in twelve for a reason unrelated to this PR (ehit_mark's lazy allocation racing in the parallel warmstart, details in #1331). I ran the table above with a local lock on that allocation; without it the crash would look like a tier fault and it is not one.

Two small remarks, neither blocking:

1. The new test proves the wait on the success path. The failure path (ok == 0, the "device genuinely full" branch) decrements too, but no test pins it; a fake_upload_hook that fails the third tensor of the last expert would cover it in a few lines. Fine as a follow-up.
2. qwen36_tier.h:142 has the non-CUDA stub static inline void qt_fill_wait(void){}; its comment could say the same as the new one at line 116 so the two declarations do not drift.

Approve. Once this is in I will rebase my qwen38 tier branch onto it; the dense-trunk residents there do not go through the uploader, so nothing else to reconcile.