Fix Windows Qwen36 CUDA DLL tier build - #1580
Merged
Merged
Conversation
expert_cache_init sizes the LRU expert-slot cache as MemAvailable - 3 GB.
MemAvailable counts reclaimable page cache as free, so on a machine where the
model already fits in RAM the engine allocates almost all of it as anonymous
memory and evicts the very model pages the slot cache then has to re-read from
disk. Both caches hold the same bytes; the cheap copy is the one that loses.
The budget now leaves the model room to stay in page cache and takes only what
is left: total - model - 8% margin, still clamped by MemAvailable and still
overridable with GLM53_EXPERT_GB. When the model cannot fit in RAM at all, page
cache cannot help, so the previous take-what-is-available behaviour is kept --
and "could not measure the total" takes that same safe path.
Cold A/B on the rome box (GLM-5.3-Flash int4-g64, 265.6 GB RAM, 8-core EPYC
7F32, NVMe), model shards evicted with posix_fadvise(DONTNEED) before EVERY
run, same prompt, greedy decode so both binaries emit the same 60 tokens:
tok/s hit rate slots/layer anon resident
old (dev) 1.120 / 1.106 72.5% 288 171.2 GB
new 1.290 / 1.292 65.3% 102 60.6 GB
+15.6%. Note the shape: the hit rate FALLS while tok/s rises, because a miss
that lands in page cache is far cheaper than the LRU hit it replaced. Hits and
misses are bit-identical between reps (greedy decode, same routing), so the
difference is placement and nothing else.
Two unit bugs are fixed here, both found while rebasing:
* the original patch computed the total with kb/1048576 (GiB) and compared it
against a model size computed from byte counts (/1e9, true GB). It
subtracted true GB from GiB.
* /proc/meminfo's "kB" is KiB, so compat's kb/1e6 understates true GB by
2.3%. That was harmless while "available" was only ever compared against
itself, and stops being harmless the moment it is weighed against a model
size in real GB.
Both fields are now true GB, which is why the auto-selected budget on this box
is 61.1 GB and not the 44.3 GB the first version of this PR reported.
Addressing the review notes:
* total and available now come from one call and one pass, via a new
compat_meminfo_gb(total, avail) in compat.h; compat_mem_available_gb()
delegates to it, so there is still exactly one definition of "available".
The second /proc/meminfo scan is gone.
* that also settles the JustVugg#1456 interaction: total is measured the same
cross-platform way as available (sysctl hw.memsize on macOS,
GlobalMemoryStatusEx on Windows), so the two cannot disagree about what
they mean. Before this, memory_total_gb returned 0 on macOS and Windows,
which silently took the fallback branch and disabled this fix entirely on
both platforms.
* test_mem_available.c stated in its own comment that the "available <=
physical RAM" bound had to be skipped on macOS for want of a total. It no
longer has to be, and it also now pins the wrapper against the one-pass
helper.
Not covered: the model >= total - margin fallback is not exercised on hardware
here. No model on this box exceeds RAM, and unprivileged user namespaces are
disabled, so /proc/meminfo cannot be faked to force it. The branch is reasoned
and unchanged from the behaviour it replaces, but it is untested at runtime and
should be read as such.
Four other places still hand-roll their own total-RAM query with their own
#ifdef ladder and their own definition (telemetry.h via sysctl, olmoe.c via
sysconf(_SC_PHYS_PAGES), kimi_k3.c, tests/test_mem_available.c). Consolidating
them onto compat_meminfo_gb is a separate change and deliberately not done here.
Add converter="convert_olmoe_merged.py" and converter_accepts=() to the OLMoE FamilyDescriptor. Empty converter_accepts is intentional: the converter takes no precision flags (--ebits / --group-size etc.).
cmd_convert hardcodes --outdir as the output flag. Without this rename, the converter rejects the command as unknown argument.
- test_olmoe_checkpoint_reaches_convert_olmoe_merged: asserts OLMoE dispatches to convert_olmoe_merged.py with no precision flags. - Relax test_every_declared_converter_states_what_it_accepts to accept empty converter_accepts (genuine "takes nothing" is valid).
… device coli_metal_init() gave up whenever MTLCreateSystemDefaultDevice() returned nil. That call resolves the *display* device, so it is nil on a host with no window-server session (ssh, CI, a launchd job) and on Intel Macs whose GPU is not the system default -- the card is still present and MTLCopyAllDevices() enumerates it. The backend reported "Metal unavailable" on hardware that runs every kernel correctly. Fall back to the enumeration, preferring a non-low-power device so a dual-GPU Mac does not land on the iGPU. This is the same discrete>integrated ranking backend_vulkan.c already applies when picking a physical device. Measured on an Intel Mac (macOS 14.5) with a discrete AMD Radeon RX 6900 XT, where MTLCreateSystemDefaultDevice() returns nil and MTLCopyAllDevices() returns the card: before: make metal-test -> "Metal unavailable (skipping)" after: make metal-test -> "metal backend tests: ok" The full battery passes on that card: rmsnorm/silu/add, quantized GEMV int8/int4/int2/f32, grouped int4 (fmt=4), FP8 passthrough (fmt=8), E8/IQ3 (fmt=6), batched and ragged moe_block, large-batch GEMM, the fused MLA attention cases, and top-8 select bitwise serial==parallel. Errors against the CPU reference are 1e-8..3e-5, in family with an Apple Silicon run. The card reports threadExecutionWidth 32, so the r_top8_par width guard stays satisfied and is not exercised by this change. `make check` passes (861 tests, 78 skipped) and `make colibri METAL=1` builds warning-free. The default CPU build is untouched: backend_metal.mm is only compiled under METAL=1.
hip-test is `$(MAKE) cuda-test HIP=1`, so the -x none added for hipcc also reached nvcc on CUDA hosts. nvcc accepts only c, c++ and cu for -x and fails the invocation on anything else, and CI never executes cuda-test, so the breakage would have surfaced only on the next CUDA box to run the suite. Wraps it in $(if $(filter 1,$(HIP)),...) as suggested in review. -fPIC on the reference object is unconditional and stays as it was. Verified both directions on this host: with HIP=1 the flag appears in the recipe, without it the expansion is empty. `make hip-test HIP=1 HIP_ARCH=gfx1151 ROCM_HOME=/usr` still builds and passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019WbKLB3C3vuXSZWDw5w83j
The same positions evaluated two ways must produce the same logits. Arm A
pushes the whole sequence through step_all in one batched prefill; arm B
prefills a prefix and then walks the continuation one token at a time through
the KV cache, exactly as generate() does. The two arms share weights, so a
disagreement beyond float accumulation order is a KV-addressing or masking
defect in one of them.
It needs no reference PREDICTIONS -- no tf_pred, no second implementation --
because it compares the engine against itself. Tokens come from either source:
SNAP=<model> REF=<ref.json> CONSIST=1 ./colibri 64 16 16 # prompt_ids/full_ids
SNAP=<model> PROMPT="..." CONSIST=1 ./colibri # no ref file at all
The PROMPT form is what lets the check run on a model no fixture exists for. It
tokenizes with the same [gMASK]<sop> prefix run_text applies -- skip that and
the sequence is out-of-distribution, both arms agree on garbage, and the check
cannot fail -- then splits at CONSIST_NP (default halfway).
The gate is the largest RELATIVE logit gap, which is what separates the two
failure classes: reordered f32 accumulation over the hidden dim lands near
D*eps, a misaddressed KV row lands at O(1). Argmax flips are reported but NOT
gated: a flip requires |a[ia]-a[ib]| < 2*gap by construction, so "the flip is
explained by the gap" holds for every flip and would be an assertion that
cannot fail.
Measured on GLM-5.2, 744B int4-gs64, D=6144, 15 tokens split 8/7:
CONSIST_TOL=1e-2 (default) worst relative gap 3.902e-04 OK, exit 0
CONSIST_TOL=1e-6 worst relative gap 2.762e-04 FAIL, exit 1
D*eps at D=6144 is 7.3e-04, so the observed gap sits just under the float
reordering noise floor and ~3 orders of magnitude below where a KV defect would
land. The second run is the positive control: the gate fires and exits non-zero,
so the passing result is not an assertion that cannot fail. The two gaps differ
because expert cache state changes accumulation order between runs; both sit far
below the default tolerance.
make check passes (921 tests, 85 skipped).
CPU and Metal already cap that product. The Vulkan path used the same qmatmul_gate_up.comp without a limit, so GLM-5.3 decode could NaN on fp16 overflow. limit<=0 stays unclamped for GLM-5.2, per dispatch. Fixes JustVugg#1520
…lease mv() gates Metal on w->resident (glm53.c:961) but Vulkan on format alone (:970). Mat.resident is documented as "eligible for persistent accelerator wrapping" (:645) and set only by the two resident-trunk loaders (:788, :914); routed experts are "streamed, not resident" (:562). The weight arena never reclaims slices per-tensor, which is safe under its documented invariant that wrapped tensors "live for the process" (backend_vulkan.c:455-464). Streamed expert Mats are stack temporaries from expert_mats (:1432-1443) whose cached vk handle dies at scope exit, so every expert-cache miss burns a slice that can never be reused. Measured on GLM-5.3-Flash int4 (194.7 GB), 16 greedy tokens, one RTX 5090: the registry requests 33.57 GB on a 32 GB card (6,689 tensors, VRAM 98.5% full) before the fix, 4.51 GB (530 tensors) after. Growth is ~1.82 GB/token and strictly linear in run length. mat_release additionally frees mat->vk, mirroring mat->metal and the function's stated total-release contract. It reclaims the host tensor, its VkBuffers and the counters; arena slices remain non-reclaimable by design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018MKPGjJHSf3VpEkrarkYCM
Contributor
Author
|
Real-hardware validation on Windows 11 Pro / Ryzen 7 7800X3D / 64 GB RAM / RTX 4070 Ti 12 GB ( Model:
CUDA startup confirmed: |
This was referenced Sep 17, 2026
Owner
|
Thank you for this, and the two Windows gaps you describe are real. The problem is the base, not the patch: this targets Concretely, ifneq (,$(filter 1,$(CUDA) $(CUDA_DLL) $(HIP) $(HIP_DLL)))
QWEN36_TIER_SRC = qwen36_tier.cwith a comment citing #1533 for the same symptom you hit. Merged as is against Could you retarget onto |
# Conflicts: # c/Makefile # c/qwen36.c
--out stays for backward compat with existing callers (oracles, scripts). --outdir added as alias so cmd_convert also works. Fixes: CI failures in OLMoE tiny oracle, make check, container image.
…stVugg#1581) `coli plan` and `coli doctor` print a VRAM tier; `--auto-tier` is documented as applying that plan; and then environment_for_plan drops the tier whenever CUDA was not switched on, which `--auto-tier` alone never does. Nothing said so. The drop itself is the contract and stays: --gpu and --vram are what select a CUDA-capable build, and merely asking for a plan must not turn a CPU-only sibling binary into an attempted CUDA launch. The silence was the defect. On the box in JustVugg#1581 it was the difference between 11.8 and 21 tok/s, with the plan on screen promising the tier that the launch had just discarded. So the launcher now names what it is not applying, how much of it there is, and the flag that would use it. It stays quiet when the tier was not real anyway -- no device qualified to drive placement, or a zero budget -- because a warning that fires on every CPU-only launch is one people learn to scroll past. The notice sits outside the try that wraps plan construction on purpose: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers both directions and the malformed plan. It caught a real mistake on the way: GB lives in resource_plan, not in the launcher, so the first version raised NameError on the one path that mattered.
… it (JustVugg#1581) `coli plan` and `coli doctor` print a VRAM tier. `--auto-tier` is documented as applying the plan they print. On the sibling-engine path it then dropped the tier whenever COLI_CUDA was not already "1", which `--auto-tier` on its own never sets, and dropped it in silence: no [PLAN] line, no [CUDA] banner, just half the throughput. On the reporter's RTX 4060 Ti that was 11.8 tok/s against 21.0 once `--gpu auto` was added, found only by watching nvidia-smi. The drop was right once. cuda_binary() could inspect only the GLM binary, so a working qwen36 CUDA build looked CPU-only to it, and requiring an explicit flag was safer than risking a CPU-only sibling launched as a CUDA one. Since JustVugg#1533 the check takes the engine that will actually run, so the risk that contract was written against can be tested instead of assumed -- and assuming it costs the user the tier they were just shown. So plan_cuda_enabled() now asks the same question the GLM path has always asked (`has_cuda=cuda_binary()`), of the right binary: COLI_CUDA=1 --gpu/--vram already validated the build; take it at its word COLI_CUDA=0 --gpu none is still the off switch otherwise ask the family's engine, and DeepSeek V4 its own probe, because cuda_binary() rejects valid V4 CUDA builds (JustVugg#1219) A CPU-only build still never gets a CUDA launch: that is now what the check says rather than what the flag assumed. The notice stays, for the two ways a real tier can still go unused, and says which one happened -- "drop --gpu none" and "rebuild with CUDA=1" are not interchangeable advice. It stays quiet when no device qualified to drive placement or the budget was zero, since neither was going to be applied with the GPU on either. The call sits outside the try that wraps plan construction: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers the decision in both directions (a GPU build gets the tier without a flag; a CPU-only build does not; --gpu none beats a GPU build; an explicit --gpu is taken at its word without a second probe; V4 uses its own probe) and the notice's two messages and its silences.
…evel sgb_array inlined the item schema twice, once for the first item and once for each ","-item, so every level of array nesting doubled the GBNF text: 8 levels of arrays overflowed GR_MAX_RULES and ran without a grammar, and a 567-byte schema of 22 levels compiled to 180 MB in about a second. The compiler accepts 32 levels. An item that holds an array is now queued as one composite rule, jitem<N>, emitted after the root rule and referenced by name at both places. Every other item is inlined as before, so a schema without an array inside an array's items compiles to the same GBNF as before. The language the grammar accepts is unchanged.
gr__normalize advanced a caller past a rule reference and pushed the callee, but left the caller on the stack even when that reference was the last symbol of its alternate. A repetition compiles to R ::= I R, so every repeat of x* or x+ kept one finished frame. After ~60 repeats the stack hit GR_MAX_DEPTH, gr_accept returned -1, and gr_feed turned grammar drafting off for the rest of the output: at the 60th byte of a JSON string, the 56th item of an array, 60 bytes of whitespace, or the 52nd row of the NDJSON grammar in test_grammar.c. Pop the caller when the reference ends its alternate. The frame had nothing left to resume and would be popped as soon as the callee finished, so the accepted language, the admissible bytes and the forced spans are unchanged. Left recursion and epsilon cycles are still caught by the depth argument.
Contributor
Author
|
Thank you for the clarification. I’ll retarget the PR to I’ll remove the portions already covered by #1533, then preserve and revalidate any remaining |
LTCjRet
force-pushed
the
fix/windows-qwen36-cuda-dll
branch
from
September 17, 2026 17:58
b690cfa to
7b81b52
Compare
Contributor
Author
|
Rebased onto the current The PR now retains only:
The parity test passes on the rebased tree. Thank you for the guidance. |
…-rollback fix(cuda): reclaim partially uploaded expert tensors
…budget fix(cuda): budget expert scale allocations separately
…rror fix(cuda): propagate Qwen expert collection failures
…leanup fix(cuda): unwind failed Qwen tier initialization and preserve active state
…t-unwind fix(cuda): unwind initialization failures and preserve active contexts
…hutdown fix(cuda): release Qwen tier resources on shutdown
qwen36: offload batched attention projections to CUDA
…refill qwen36: batch resident DeltaNet input projections
…sets, both cuda-test additions
…tVugg#1689) On a merged-/usr system /bin/coli is a link to /usr/bin/coli; abspath kept the alias and the installed launcher looked for /libexec/colibri. realpath first. tests/test_install_closure.py runs the launcher through such an alias, and checks that every root module tools/pack_python.py reaches is in the handwritten make install list (the v41_dsml.py gap of 1.12.0, fixed by JustVugg#1610, is the case it would have caught).
qwen38: CPU prefill batching — runtime chunk/workspace knobs, wider expert load batch, parallel QSA prefill
…s-metrics feat(serve): expose Prometheus admission and generation metrics
…nchmark feat(tools): add reproducible HTTP serving benchmark
…gpu-backend-claims docs/qwen38, README: remove stale "no GPU backend" claims (GPU tier merged in 1.12.0)
…t8 down in one slab) The JustVugg#1370 experiment: Q4_K_M keeps down/output in Q6_K, our gs64 int4 loses 2.5 % perplexity against it, and all-int8 experts close the gap (7.153 against 7.147). The first number JustVugg asked for is down alone at more bits with everything else unchanged. --down-bits N (5..8, with --ebits <= 4) quantizes down_proj at N bits (--down-gs groups along its input, 0 = per row) and writes one uint8 slab per expert: [gate int4 packed | up int4 packed | down int8], 2*inter*hidden bytes -- neither the int4 (1.5x) nor the int8 (3x) size, so the engine can tell the layouts apart from the bytes, as it does today. qs stays [gate | up | down]. meta gains expert_down_bits / expert_down_gs. The selftest covers the mixed round trip byte for byte. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
… CPU path The slab is told apart by size (2*inter*hidden bytes); gate|up unpack into the slot's int8 block as the int4 container does, down's int8 rows are copied behind them, the scales come as [gate | up | down] with down_proj's own group size (expert_down_gs, 0 = per row) from qwen36_meta.json. matmul_qd dispatches down_proj on its own layout; gate/up stay on matmul_qe. The VRAM tier takes one format per expert, so COLI_CUDA=1 is refused with a line on a mixed container until the tier learns it. Tiny fixture (inter 64, gs 64): int8 16/16 tokens, int4 gs64 12/16, mixed 16/16; cap 1, cap 2, PILOT=1 WIDE=2 and ASan/UBSan clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
…JustVugg#1370 numbers in "Which container?" Tiny job: the inter-64 fixture converted with --down-bits 8 must be detected by size, give identical ids at cap 1/2/8 (cap 1 recycles the one slot after every expert, where a wrong slab offset shows), and refuse COLI_CUDA=1 with the line. Docs: the knob, the four perplexity figures, and the verdict -- down alone is a quarter of the gap, the tier does not take the layout yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
coli: resolve the launcher symlink before deriving libexec; install closure test
…pipeline kimi: keep CUDA SiTU expert intermediates on device
…atch CHANGELOG 1.12.1: notes for the eighteen pull requests merged today
qwen36: --down-bits, the mixed expert layout (int4 gate/up, int8 down) -- the knob behind the JustVugg#1370 numbers
fix: resolve static analysis bugs (measure redefinition, dead imports, missing f-string, unused variable)
…-upgrade-to-3.24 ci: upgrade Alpine 3.21 to 3.24 in musl job (EOL Nov 1, 2026)
…isites plus the alias and .build-config
Owner
|
Merged current dev into your branch: the qwen36 rule's prerequisite list had grown on dev (decode_batch.h, idot.h), so I kept dev's line and added your alias block and the .build-config prerequisite on top. The loader part of the change was already on dev via #1533, which is why the net diff is the Makefile, the parity test and the docs. Merging once CI is green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This fixes two Windows-specific gaps that prevented qwen36.exe from using the CUDA DLL tier.
Changes
Add a Windows qwen36 alias so make qwen36 CUDA_DLL=1 reaches the intended qwen36.exe rule instead of GNU Make’s implicit C-file rule.
Compile qwen36_tier.c under CUDA_DLL=1, linking its CUDA ABI calls through backend_loader.o.
Add the missing coli_cuda_available_device_count loader typedef, optional DLL resolution, and forwarding wrapper. The wrapper loads the DLL before querying because Qwen tier device discovery occurs before coli_cuda_init().
Add a .build-config prerequisite so a prior CPU-only qwen36.exe rebuilds when CUDA-DLL configuration changes.
Add loader/header parity coverage and Windows Qwen CUDA-tier documentation.
Validation
On Windows 11 with an RTX 4070 Ti (sm_89), CUDA 13.2, MSVC Build Tools, and MSYS2 MinGW GCC:
make qwen36 CUDA_DLL=1 ARCH=native
successfully compiled qwen36.c, qwen36_tier.c, and backend_loader.o.
The new loader-parity test passed. A real Qwen3.6-35B-A3B int4-gs64 smoke test confirmed:
[gpu] MoE experts -> CUDA VRAM tier
[qtier] resident 4889/10240 experts
The existing coli_cuda.dll exported coli_cuda_available_device_count and was reused unchanged.
Compatibility
CUDA_DLL=1backend: fixes the Qwen3.6 CUDA-tier build and runtime path.CUDA=1andHIP=1build recipes are unchanged.coli_cuda.dllstill loads; automatic device discovery returns zero in that case, while an explicitCOLI_GPUS=0selection remains available.