Skip to content

fix(qwen36): the pre-init device-count probe must load the backend DLL (#1577) - #1579

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
ZhiyangK:fix/1577-available-device-count-load-dll
Sep 17, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
ZhiyangK:fix/1577-available-device-count-load-dll

Conversation

@ZhiyangK

Copy link
Copy Markdown

Summary

coli_cuda_available_device_count() is called by qwen36_tier.c before coli_cuda_init()
the count decides which device indices are handed to init — but the wrapper gated on
g_cuda.available, a flag only init sets. On Windows that answered 0 on every host, so the tier
fell back to the CPU path unless COLI_GPUS/COLI_GPU was set by hand (#1577).

This is the fix #1577 proposed, plus the regression test that issue said it did not have.

Changes

c/backend_loader.c

c/tests/test_backend_loader.py

  • The generated stub backend answers the two device-count exports with distinct non-zero values
    (2 and 5) and can now be built without the optional export. 0 is both a legitimate device count
    and what the g_cuda.available gate returned without loading anything, so a test that cannot
    separate those two cases cannot see this bug.
  • The generated harness prints the count with the backend-module state on both sides of the
    probe, then still runs init.
  • New LoaderPreInitAvailabilityTest, four cases:
    1. backend present, no init, no COLI_GPUS → the count comes from the DLL (2), the harness
      observes the DLL being loaded by the probe (0 → 2 → 1), and init still succeeds after;
    2. no backend → 0 devices, nothing loaded, no crash, init still reached;
    3. DLL without the export (new oldvariant fixture) → falls back to device_count() (5);
    4. HIP control → a HIP host still refuses on COLI_HIP_RUNTIME_DIR before init and leaves the
      backend unmapped.

Test plan

$ python c/tests/test_backend_loader.py
Ran 82 tests in 75.4s
OK

Reverting the one-line change in coli_cuda_available_device_count() makes cases 1 and 3 fail
(0 instead of 2/5) — which is what these tests exist for.

The build path from #1533 is exercised too:

$ make -C c qwen36.exe CUDA_DLL=1 ARCH=native
gcc ... -DCOLI_CUDA qwen36.c qwen36_tier.c backend_loader.o -o qwen36.exe -lm -fopenmp -static -lpsapi

— the tier is compiled in by the Makefile alone (no variable overrides) and links against the
patched loader with no missing symbol.

Environment: Windows 11, MinGW w64devkit 2.10.0, Python 3.11. The fixture skips honestly where
gcc/objdump are absent, and loads no real vendor runtime.

Fixes #1577

coli_cuda_available_device_count() is asked by qwen36_tier.c BEFORE
coli_cuda_init -- the count decides which device indices are handed to init
-- so gating the wrapper on g_cuda.available answered 0 on every Windows host
and the tier silently fell back to the CPU path unless COLI_GPUS or COLI_GPU
was set explicitly (JustVugg#1577).

Load the DLL on demand, exactly as coli_cuda_attention_project_ragged already
does, and keep the fallback for a DLL predating the export. The "no lock
needed" note above coli_cuda_load() now names all three callers and records
why the new one is still inside the single-threaded start-up window: the tier
asks while it is still choosing the devices to hand to init.

Tests: the generated stub backend answers the two device-count exports with
distinct non-zero values (2 and 5) and can be built without the optional one,
so LoaderPreInitAvailabilityTest can tell "the DLL answered" from "nothing
answered" -- the difference the g_cuda.available gate hid. Four cases
(backend present / absent / older DLL / HIP control); reverting the one-line
change fails the two that need the DLL.

Fixes JustVugg#1577
@ZhiyangK

Copy link
Copy Markdown
Author

Real-hardware verification of the fix on a native Windows host (no WSL), following the
reproduction in #1577 with no COLI_GPUS and no COLI_GPU in the environment at all —
i.e. the exact case the issue reports, and the one --auto-tier cannot reach because the
resource planner writes COLI_GPU itself.

$ SNAP=<qwen36 i4-gs64 container> N_NEW=8 COLI_CUDA=1 CUDA_DENSE=1 CUDA_EXPERT_GB=12 \
    qwen36.exe 256 8 prompt.txt          # COLI_GPUS / COLI_GPU unset

[qtier] COLI_GPUS unset: selecting 1 visible device(s)
[CUDA] device 0: NVIDIA GeForce RTX 5060 Ti, 17.1 GB VRAM, sm_120
[place] auto: dev 0 holds 1207.4 MB of trunk (lmhead yes, 30 dnproj layers), 10.82 GB left for experts
[qtier] dev 0: budget 10.82 GB for experts (~6139 experts) after trunk
[qtier] CUDA VRAM expert tier active: 1 device(s), 1.80 MB/expert
[CUDA] mode: routed experts (qwen36 VRAM tier)

Before the change the first line read selecting 0 visible device(s) and the tier took the CPU
path, which is what #1577 reports.

Host: Windows 11, RTX 5060 Ti (sm_120), CUDA 12.9, VS 2022 MSVC, MinGW w64devkit 2.10.0,
make cuda-dll CUDA_ARCH=sm_120 + make qwen36.exe CUDA_DLL=1 ARCH=native — the latter compiles
qwen36_tier.c in from the Makefile alone, without the QWEN36_TIER_SRC override #1533 needed.

Unrelated to this PR: on a 32 GB host this container still dies at slot allocation
(Error: OOM allocating slot weights) because cap=256 wants the full 10240 experts
(40 layers x 256) resident; device selection now works, memory sizing is the separate wall
tracked elsewhere. Both hosts here have 32 GB, so #1577's own log (all 10240 experts in RAM)
is worth a second look if you expected this to fit smaller machines.

@JustVugg

Copy link
Copy Markdown
Owner

Reviewed together with #1542 and #1580, full comparison in #1577 (comment).

Short version: the load fix and the precedent you cite are right, and your test is the best of the three. The one thing to change is the fallback you kept:

if(!g_cuda.available_device_count) return g_cuda.device_count();   /* a DLL from before the export */

coli_cuda_device_count is return g_nctx, the count of initialised contexts, and the single caller (qwen36_tier.c:524) asks before init. So that line can only ever return 0, silently, which is the bug this PR fixes wearing a different hat. #1542 replaces it with a message naming the missing symbol and pointing at COLI_GPUS.

Take that behaviour into this branch and I will merge it.

IcarusAegis added a commit to IcarusAegis/colibri that referenced this pull request Sep 17, 2026
Adapt DLL module-state probes and the HIP refusal control from ZhiyangK's PR JustVugg#1579. Preserve initialized-context semantics, verify missing-export diagnostics and explicit initialization, and distinguish a valid zero-device response. Clarify the three loader callers and their startup ordering.
IcarusAegis added a commit to IcarusAegis/colibri that referenced this pull request Sep 17, 2026
Extend the discovery fixture with separate visible-device and initialized-context state. Cover CUDA/HIP discovery, explicit device forwarding, old DLL diagnostics, failure cleanup, runtime ownership, and repeated calls without changing the original direct-init tests.

Exercise real qt_init and qt_init_fp8 selection paths with a stateful backend and add both suites to the mandatory Windows CI job. Adapt module-state and HIP control ideas from ZhiyangK's JustVugg#1579.

Validation: 105 focused tests passed without skips; make check passed (Python: 913 passed, 100 skipped); seven regression mutations detected. On RTX 4070 Ti SUPER, nine CUDA test programs and sixteen real DLL/tier scenarios passed. Multi-GPU, AMD HIP hardware, and full-model inference remain unverified.

Co-authored-by: ZhiyangK <ZhiyangK@users.noreply.github.com>
@JustVugg
JustVugg merged commit 4dbc4c1 into JustVugg:dev Sep 17, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants