Integrazione dev + #1582 auto-tier VRAM + #1580 build Windows CUDA + #1588 + tetto RAM_GB qwen36 - #7
Merged
Merged
Conversation
cmd_run fed the one-shot prompt to olmoe's stdin chat loop through a text-mode pipe, so Python encoded it in the locale's preferred encoding while the engine tokenizes the bytes as UTF-8. On cp949/cp932 (or the C locale) a prompt with an accented letter or an emoji died with UnicodeEncodeError before the engine started; where the code page could encode it (e on cp1252, Hangul on cp949) the engine got non-UTF-8 bytes. Hand the engine the UTF-8 bytes, as the V4 path already does with its prompt file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
autotune._run captured the engine's stdout and stderr in text mode, i.e. in the locale's code page. The GLM engine writes UTF-8 -- with PROF=1, set on every calibration and replay run, its "[prefill] layer ... ·" progress line and the em-dash profile verdict. On cp949/cp932 or the C locale the reader thread raised UnicodeDecodeError, proc.stdout came back None, and `coli tune` died in calibration with a TypeError before measuring anything. Decode the engine's output as UTF-8 with errors="replace", as the serve-protocol path in openai_server already does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cale's code page run_fresh_engine drove olmoe, inkling and colibri over a text-mode pipe, so the prompt was encoded, and the engine's output decoded, in the locale's preferred encoding while the engines speak UTF-8. On cp949/cp932 or the C locale a non-ASCII --prompt died with UnicodeEncodeError, and an engine line such as colibri's "[prefill] layer ... ·" made the reader thread raise, leaving proc.stdout None and killing the datapoint with a TypeError. Use encoding="utf-8", errors="replace" on that call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
convert_inkling_dense_int4.py measured free space with os.statvfs, which does not exist on Windows, so every Windows run died with AttributeError right after the plan, before writing anything. Past that, its "…" and "✅" progress lines raised UnicodeEncodeError on a stdout that cannot encode them (cp949/cp932, the C locale): the run exited 1 after writing the container and never printed the quantization-error report. Use shutil.disk_usage for free space, and escape what stdout cannot encode, as rans_verify.py and k3_repack.py already do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CudaTestMakefileTest shells out to make to dry-run the cuda-test recipe, but unlike its siblings it never checks that make is there. Without it the six recipe tests raise FileNotFoundError from CreateProcess/exec and the file reports six errors -- an absent prerequisite dressed up as a regression. test_makefile_platform.py already has the shape: resolve make once with shutil.which, invoke it by that path, and skipUnless on the result. test_makefile_cuda_scope.py and test_backend_loader.py guard the same way. This file is the only one of the five that reaches for make and does not. test_setup_openmp_probe_does_not_require_tmp only reads setup.sh, so it moves to its own class instead of being skipped along with the rest -- a class-level guard would have cost that coverage. Running make through the resolved absolute path rather than the bare name also matches the sibling, and avoids relying on PATHEXT resolution. python -m unittest discover -s tests, on a host with no make: before FAILED (errors=6) 819 ran, 132 skipped after OK 813 ran, 138 skipped With make present every test still runs; nothing is skipped on CI.
… refuses what it cannot place Qwen/Qwen3.8-2.4T-A95B declares the same model_type as Qwen3.6-35B-A3B (qwen3_5_moe_text) and resolves to this engine today, where the banner announces it as "Qwen3.6-35B-A3B · 35B" (JustVugg#1045). Fixed on its own merits, with the pieces that make the 2.4T shape a verified one rather than a guessed one -- without a single weight of it. A model_type resolves to exactly one family, so a second descriptor is not an option. Instead `DisplayVariant` pairs a geometry (config keys that must match exactly) with a name, and `display_for(resolved)` picks the first match: 40/256/2048 -> "Qwen3.6-35B-A3B · 35B", 92/512/8192 -> "Qwen3.8-2.4T-A95B · 2.4T". A config matching neither -- a tiny fixture, a third sibling -- is named by its own model_type and measured geometry, never by a sibling's parameter count. The registry refuses a variant that matches everything, and a family whose static display_name is not among its variants (the READMEs are held to that name). `coli`'s banner reads the family config (text_config-aware), so the VL-wrapped 35B repo and the flat 2.4T config both display right. The converter selected `<prefix>layers.<i>.` and four globals and let the rest fall through unmentioned. That skipped the `mtp.*` head and the `visual.*` tower by accident, and would have copied an unfamiliar tensor inside a layer as f16 without a word. `tools/qwen36_tensor_kinds.py` now classifies every name before the first shard is read: layer tensors by an exact suffix list, `mtp.*` / `visual.*` skipped ON PURPOSE with a count and a reason, anything else refused with the names. `test_qwen36_tensor_kinds` pins the contract to the safetensors indexes of both real checkpoints (1609 and 1045 tensors) and to the per-expert layout transformers saves. One correction found on the way: the comment claimed the real 35B ships per-expert tensors. Both real checkpoints ship the FUSED layout (`experts.gate_up_proj` / `down_proj`); only the tiny fixture is per-expert. `make_qwen36_tiny.py --geometry qwen38-2p4t`: 92 layers / interval 4 / 512 experts top-10 / 16:1 attention heads / 8:1 DeltaNet heads, and the saved shard rewritten into what the checkpoints ship -- fused experts plus a one-layer `mtp.*` head with `mtp_num_hidden_layers: 1`. The converter has to split the one and skip the other; the engine has to match the transformers reference. The default preset reproduces the shipped fixture's reference exactly. - old fixture: reference identical, converted shard bit-identical (cmp), 16/16 at cap 1/2/8 - 2.4T geometry: 19 mtp tensors skipped and said so, 1587 placed in 92 layers, 16/16 at cap 1/2/512, 0.35 s per run - ASan/UBSan (nm confirms 31 __asan symbols): no diagnostic on either fixture under PILOT=1 WIDE=2 - registry planner on the real 2.4T config: KV 1.44 GiB @8k, 46.0 GiB @256k, DeltaNet state 0.55 GiB context-free -- the numbers quoted in JustVugg#1045; the engine's KV cache is F32 (ensure_kv: falloc), so they stand - python suite: 743 tests, OK (skipped=35) - the new CI step run locally word for word `--seed 3` for the 2.4T rung: the default seed collapses this geometry's reference to one repeated token, which a shape error could still reproduce; seed 3 gives twelve distinct tokens over sixteen. Not done here, and said in docs/qwen36.md: running the 2.4T. The warmstart keeps every expert in RAM by design, ~1.4 TB of int4 for this checkpoint; that needs the disk-streaming design, not this one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLj9ctDNPGTsYxgBuDmy5a
coli_cuda_available_device_count() is asked by qwen36_tier.c BEFORE coli_cuda_init -- the count decides which device indices are handed to init -- so gating the wrapper on g_cuda.available answered 0 on every Windows host and the tier silently fell back to the CPU path unless COLI_GPUS or COLI_GPU was set explicitly (JustVugg#1577). Load the DLL on demand, exactly as coli_cuda_attention_project_ragged already does, and keep the fallback for a DLL predating the export. The "no lock needed" note above coli_cuda_load() now names all three callers and records why the new one is still inside the single-threaded start-up window: the tier asks while it is still choosing the devices to hand to init. Tests: the generated stub backend answers the two device-count exports with distinct non-zero values (2 and 5) and can be built without the optional one, so LoaderPreInitAvailabilityTest can tell "the DLL answered" from "nothing answered" -- the difference the g_cuda.available gate hid. Four cases (backend present / absent / older DLL / HIP control); reverting the one-line change fails the two that need the DLL. Fixes JustVugg#1577
) The official template writes "\n<tool_call>". GLM-5.3 does not: on an assistant turn that is nothing but a tool call it writes "</think><tool_call>", with no newline between them. render_chat_glm53 wrote the newline anyway, so the replayed prefix carried one token the model had never produced. One token is the whole cost. The reuse gate in glm53.c is all-or-nothing, so the next turn threw away the entire cached prefix and re-prefilled from scratch. The reporter measured that on a 3k-token agent history at 2.3 tok/s: twenty minutes, every turn after a bare tool call. A turn that also carries text is left exactly as it was, and that is not an oversight. There the model's own trailing newline is stripped by .strip() and put back by the renderer, the tokens line up, and it is the case that works today. Only the bare turn changes, which is the only one that was broken. tests/test_openai_server.py asserts both halves: "</think><tool_call>" with no newline for the bare turn, and the separator still present for the turn with text, so a later tidy-up cannot quietly break the working case while fixing nothing.
…f8-prompt fix(coli): coli run on OLMoE sent the prompt in the locale's code page
…ine-output fix(autotune): coli tune on GLM died decoding the engine's UTF-8 output
fix(datapoint): fresh-process mode talked to legacy engines in the locale's code page
…vert-windows fix(inkling): the dense int4 converter could not finish on Windows
…criptor feat(qwen36): name a checkpoint by its geometry, and a converter that refuses what it cannot place
candidate_steps offered cuda-pipe-1, cuda-pipe-2 and cuda-sync whenever the plan listed a GPU, whatever the engine. COLI_CUDA_PIPE is read in colibri.c only, and COLI_CUDA_ASYNC only inside expert_group_impl, which is reached from colibri.c alone. On any other engine those candidates were identical runs: each cost a full engine start, and a timing wobble could be accepted as a gain and written into the saved profile. Gate the CUDA sweep on arch == "glm", the same rule JustVugg#898 applied to PIPE and DIRECT. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
numa-on was offered to every engine on a multi-socket host, and the docstring called COLI_NUMA engine-agnostic. The only reader is colibri.c's numa_init, which mbinds colibri's own expert slabs; no other engine reads the variable. Gate it on arch == "glm" with the CUDA pair and correct the docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
push_id and bpe_piece grow their token-id/BPE-symbol buffers with a bare realloc() and no NULL check, so an allocation failure under memory pressure overwrote the buffer pointer with NULL and the very next store wrote through it. Both are reached from encode_text on every inference request's prompt text (serve_one -> encode_text -> push_id/bpe_piece). qwen38.c's push_id/bpe_piece already guard this exact growth pattern (q38_encode_realloc / q38_encode_oom); qwen36.c never got the equivalent fix. This mirrors the project's established OOM-guard idiom (checked realloc, fprintf to stderr, exit(1)) used in st.h for the JustVugg#798 defect class. Adds tests/test_qwen36_encode_oom.c, which injects a real allocation failure via the shadow-realloc technique from tests/test_798_guards.c and asserts both call sites exit(1) with a message instead of crashing on the NULL write; verified red (SIGSEGV in the forked child) against the unpatched function before this fix, green after.
_auto_tune recommends DRAFT, PIPE, COLI_CUDA_PIPE, COLI_NUMA and PIN_GB. colibri.c is the only reader of all five. glm53 was excluded for that reason, but every other family still got them: `coli plan` and `coli doctor` printed them under "auto-tune:", and --auto-tier exported them into an engine environment that never reads them. Return no knobs for any engine group other than colibri-core, the exclusion glm53 already had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…GEMM DLL dsv4_cuda_available checked only coli_cuda_dsv4.dll next to the engine on Windows. backend_loader_dsv4.c loads coli_cuda_dsv4_dg.dll first and coli_cuda_dsv4.dll second, and doctor's cuda_linkage accepts either. So with only the DeepGEMM build installed, `coli doctor` reported the GPU engine as available while `coli chat/serve/run --gpu` and `--vram` exited with "--gpu needs the CUDA build". Accept either name, as the loader and doctor do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
generation_options read `(response_format.get("json_schema") or {})
.get("schema")`. A truthy json_schema that is not an object (a JSON
string, a list, a number) raised AttributeError, which do_POST turns
into HTTP 500 "The colibri engine failed to process the request." on
both /v1/chat/completions and /v1/completions. OpenAI SDKs retry a 5xx,
and the message blames the engine for a malformed request.
Take .get only from a dict, so the existing 400
"`response_format.json_schema.schema` must be an object." covers it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JSON can spell a lone UTF-16 surrogate ("\ud83d"), which is what a
client produces when it cuts a string between the two halves of an
emoji. json.loads accepts it, but no UTF-8 can carry it, and the first
thing Engine.generate does is prompt.encode("utf-8"). The
UnicodeEncodeError reached do_POST's catch-all, so /v1/chat/completions,
/v1/completions and /v1/messages answered HTTP 500 "The colibri engine
failed to process the request.". Undecodable UTF-8 in the raw body is
already a 400 in read_json; this is the same invalid text, escaped.
read_json now also rejects a body whose strings cannot be encoded as
UTF-8, with a 400 that says why. Paired surrogate escapes (an escaped
emoji) are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…stVugg#1581) `coli plan` and `coli doctor` print a VRAM tier; `--auto-tier` is documented as applying that plan; and then environment_for_plan drops the tier whenever CUDA was not switched on, which `--auto-tier` alone never does. Nothing said so. The drop itself is the contract and stays: --gpu and --vram are what select a CUDA-capable build, and merely asking for a plan must not turn a CPU-only sibling binary into an attempted CUDA launch. The silence was the defect. On the box in JustVugg#1581 it was the difference between 11.8 and 21 tok/s, with the plan on screen promising the tier that the launch had just discarded. So the launcher now names what it is not applying, how much of it there is, and the flag that would use it. It stays quiet when the tier was not real anyway -- no device qualified to drive placement, or a zero budget -- because a warning that fires on every CPU-only launch is one people learn to scroll past. The notice sits outside the try that wraps plan construction on purpose: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers both directions and the malformed plan. It caught a real mistake on the way: GB lives in resource_plan, not in the launcher, so the first version raised NameError on the one path that mattered.
… it (JustVugg#1581) `coli plan` and `coli doctor` print a VRAM tier. `--auto-tier` is documented as applying the plan they print. On the sibling-engine path it then dropped the tier whenever COLI_CUDA was not already "1", which `--auto-tier` on its own never sets, and dropped it in silence: no [PLAN] line, no [CUDA] banner, just half the throughput. On the reporter's RTX 4060 Ti that was 11.8 tok/s against 21.0 once `--gpu auto` was added, found only by watching nvidia-smi. The drop was right once. cuda_binary() could inspect only the GLM binary, so a working qwen36 CUDA build looked CPU-only to it, and requiring an explicit flag was safer than risking a CPU-only sibling launched as a CUDA one. Since JustVugg#1533 the check takes the engine that will actually run, so the risk that contract was written against can be tested instead of assumed -- and assuming it costs the user the tier they were just shown. So plan_cuda_enabled() now asks the same question the GLM path has always asked (`has_cuda=cuda_binary()`), of the right binary: COLI_CUDA=1 --gpu/--vram already validated the build; take it at its word COLI_CUDA=0 --gpu none is still the off switch otherwise ask the family's engine, and DeepSeek V4 its own probe, because cuda_binary() rejects valid V4 CUDA builds (JustVugg#1219) A CPU-only build still never gets a CUDA launch: that is now what the check says rather than what the flag assumed. The notice stays, for the two ways a real tier can still go unused, and says which one happened -- "drop --gpu none" and "rebuild with CUDA=1" are not interchangeable advice. It stays quiet when no device qualified to drive placement or the budget was zero, since neither was going to be applied with the GPU on either. The call sits outside the try that wraps plan construction: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers the decision in both directions (a GPU build gets the tier without a flag; a CPU-only build does not; --gpu none beats a GPU build; an explicit --gpu is taken at its word without a second probe; V4 uses its own probe) and the notice's two messages and its silences.
fix(qwen36): a truncated multibyte tail made the tokenizer read past the prompt
…ble-race fix(st): publish the lazy shard-mapping table without a data race
…ice-count-load-dll fix(qwen36): the pre-init device-count probe must load the backend DLL (JustVugg#1577)
…-non-object fix(serve): a non-object json_schema answered 500 "engine failed"
…rrogate fix(serve): an unpaired surrogate escape answered 500 "engine failed"
…-dll fix(coli): --gpu on DeepSeek V4 refused an install with only the DeepGEMM DLL
…bri-only fix(planner): auto-tune advised colibri.c-only knobs for every engine
…lm-only fix(autotune): coli tune swept GLM-only CUDA and NUMA knobs on every engine
…akefile-skip-without-make tests: skip the cuda-test recipe checks when make is absent
glm53: replay a bare tool-call turn as the model wrote it (JustVugg#1576)
# Conflicts: # .github/workflows/ci.yml # c/backend_loader.c # c/qwen36.c
# Conflicts: # c/Makefile
Same defect class as glm53 (#A3): cap came from argv alone, tier_warmstart loaded ALL experts into RAM with no ceiling. RAM_GB is now a whole-process ceiling: measured RSS + activations/KV reserve subtracted, argv cap clamped with named terms, refuse-to-start (exit 2) when even cap=1 overflows MemAvailable unless COLI_RAM_OVERCOMMIT=1. Warmstart loads the planned VRAM set first, then only what the cap allows; the rest stays in the container until LRU demand. Telemetry at init and every 64 loads. (cherry picked from commit 00c58f8)
- c/coli: ask about the accelerator LAST. JustVugg#1582 lets --auto-tier turn the VRAM tier on from the built binary alone, after the --gpu/--vram block, so the Windows hot-team defaults were seeded for launches that do use the GPU -- exactly what their own comment forbids. Moved the block to the end of env_for_engine, with a test that fails without the move. - c/compat.h: compat_fadvise waited on a pending warm-up read through an OVERLAPPED with no hEvent, i.e. on the file handle, which a shared fd signals for ANY completed read; it could return while this read was still in flight and then free the buffer. It now goes through compat_read_at and its per-thread event (helpers moved above it), which also resets the event before each read instead of relying on ReadFile doing it. - ISTRUZIONI.md: current hardware (64 GB at 5200, SSD in M.2_3 at Gen4 x4, BIOS 3881), the measured numbers, and the reordered open items. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
…'s one-line diagnostic The conflict resolution in 0c6cb27 kept the fork's JustVugg#1542 body (return 0 when the DLL lacks coli_cuda_available_device_count) over dev's fallback. That contradicts tests/test_backend_loader.py's test_the_probe_falls_back_for_a_dll_without_the_export, which arrived with the dev merge and fails on the Windows loader job. The fallback is upstream's contract, so it stays; what the fork keeps is the stderr line naming the stale DLL, printed once. Before init device_count() is 0, so tests/test_cuda_loader_discovery.py's expectations are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… bytes back The cherry-picked RAM_GB warmstart gates both passes on loads[l] >= m->cache[l].cap, so when RAM_GB pushes the per-layer cache below what qt_plan_fill planned, the excess PLANNED experts are skipped without ever reaching qt_note_planned. That is the JustVugg#1331 leak: the tier reserved their bytes and "if (resident || queued || planned) continue" never reconsiders them, so the budget stays spent for the life of the process -- in exactly the configuration RAM_GB exists for. Pass 0 now always reports: with weights when the cache had room, with NULLs when it did not, which is the path qt_note_planned already documents as handing the bytes back (c/qwen36_tier.c:992-998). Pass 1 (unplanned experts) keeps the cap. Found by review of the cherry-pick; worth reporting upstream on JustVugg#1564. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Aggiorna il fork al
devdi JustVugg/colibri (cc756a6, 29 commit) e aggiunge tre PR aperte piu' un commit isolato.devmtp.*/visual.*che salta), JustVugg#1587/JustVugg#1589 (serve: 500 su JSON malformato), JustVugg#1549/JustVugg#1557 (gia' nel fork, ora anche upstream)--auto-tierapplica il tier VRAM anche ai motori non-GLM: chi l'ha segnalata e' passato da 11,8 a 21 tok/smake qwen36,qwen36_tier.csotto CUDA_DLL=1,.build-configcome prerequisito (niente eseguibile CPU-only rimasto indietro)push_id/bpe_piecerifiutano un realloc fallito invece di scrivere su NULL00c58f8da JustVugg#1564RAM_GBnella cache esperti e nel warmstart (oggi il warmstart carica tutti gli esperti senza tetto). Solo questo commit: il resto della PR e' un feature branch in corso. Attivo solo seRAM_GBe' impostataConflitti risolti a mano
.github/workflows/ci.yml(2 blocchi): tenuti sia i nostri step (identita' della quantizzazione al caricamento, ASan con dense int8 + embed int8) sia quelli del dev (fixture Qwen3.8-2.4T, loop ASan sulle due fixture, riuso KV).c/backend_loader.c: tenuto il corpo del fork (fix(windows): load CUDA backend before device discovery JustVugg/colibri#1542: messaggio esplicito e 0 quando la DLL non esporta il simbolo, con test dedicato intests/test_cuda_loader_discovery.py); commento riscritto perche' descriveva il fallback del dev.c/qwen36.c: un soloomp_tune.h, piu' il nuovokv_prefix.h.c/Makefile(da fix(qwen36): refuse loudly on a failed encode-buffer realloc JustVugg/colibri#1588): tenute entrambe le regole di test, ognuna con la sua ricetta.Verifiche locali
🤖 Generated with Claude Code