Detect Metal GPUs in macOS resource plans - #1556
Merged
Merged
Conversation
numa-on was offered to every engine on a multi-socket host, and the docstring called COLI_NUMA engine-agnostic. The only reader is colibri.c's numa_init, which mbinds colibri's own expert slabs; no other engine reads the variable. Gate it on arch == "glm" with the CUDA pair and correct the docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_auto_tune recommends DRAFT, PIPE, COLI_CUDA_PIPE, COLI_NUMA and PIN_GB. colibri.c is the only reader of all five. glm53 was excluded for that reason, but every other family still got them: `coli plan` and `coli doctor` printed them under "auto-tune:", and --auto-tier exported them into an engine environment that never reads them. Return no knobs for any engine group other than colibri-core, the exclusion glm53 already had. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…GEMM DLL dsv4_cuda_available checked only coli_cuda_dsv4.dll next to the engine on Windows. backend_loader_dsv4.c loads coli_cuda_dsv4_dg.dll first and coli_cuda_dsv4.dll second, and doctor's cuda_linkage accepts either. So with only the DeepGEMM build installed, `coli doctor` reported the GPU engine as available while `coli chat/serve/run --gpu` and `--vram` exited with "--gpu needs the CUDA build". Accept either name, as the loader and doctor do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
generation_options read `(response_format.get("json_schema") or {})
.get("schema")`. A truthy json_schema that is not an object (a JSON
string, a list, a number) raised AttributeError, which do_POST turns
into HTTP 500 "The colibri engine failed to process the request." on
both /v1/chat/completions and /v1/completions. OpenAI SDKs retry a 5xx,
and the message blames the engine for a malformed request.
Take .get only from a dict, so the existing 400
"`response_format.json_schema.schema` must be an object." covers it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JSON can spell a lone UTF-16 surrogate ("\ud83d"), which is what a
client produces when it cuts a string between the two halves of an
emoji. json.loads accepts it, but no UTF-8 can carry it, and the first
thing Engine.generate does is prompt.encode("utf-8"). The
UnicodeEncodeError reached do_POST's catch-all, so /v1/chat/completions,
/v1/completions and /v1/messages answered HTTP 500 "The colibri engine
failed to process the request.". Undecodable UTF-8 in the raw body is
already a 400 in read_json; this is the same invalid text, escaped.
read_json now also rejects a body whose strings cannot be encoded as
UTF-8, with a 400 that says why. Paired surrogate escapes (an escaped
emoji) are unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--out stays for backward compat with existing callers (oracles, scripts). --outdir added as alias so cmd_convert also works. Fixes: CI failures in OLMoE tiny oracle, make check, container image.
…stVugg#1581) `coli plan` and `coli doctor` print a VRAM tier; `--auto-tier` is documented as applying that plan; and then environment_for_plan drops the tier whenever CUDA was not switched on, which `--auto-tier` alone never does. Nothing said so. The drop itself is the contract and stays: --gpu and --vram are what select a CUDA-capable build, and merely asking for a plan must not turn a CPU-only sibling binary into an attempted CUDA launch. The silence was the defect. On the box in JustVugg#1581 it was the difference between 11.8 and 21 tok/s, with the plan on screen promising the tier that the launch had just discarded. So the launcher now names what it is not applying, how much of it there is, and the flag that would use it. It stays quiet when the tier was not real anyway -- no device qualified to drive placement, or a zero budget -- because a warning that fires on every CPU-only launch is one people learn to scroll past. The notice sits outside the try that wraps plan construction on purpose: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers both directions and the malformed plan. It caught a real mistake on the way: GB lives in resource_plan, not in the launcher, so the first version raised NameError on the one path that mattered.
… it (JustVugg#1581) `coli plan` and `coli doctor` print a VRAM tier. `--auto-tier` is documented as applying the plan they print. On the sibling-engine path it then dropped the tier whenever COLI_CUDA was not already "1", which `--auto-tier` on its own never sets, and dropped it in silence: no [PLAN] line, no [CUDA] banner, just half the throughput. On the reporter's RTX 4060 Ti that was 11.8 tok/s against 21.0 once `--gpu auto` was added, found only by watching nvidia-smi. The drop was right once. cuda_binary() could inspect only the GLM binary, so a working qwen36 CUDA build looked CPU-only to it, and requiring an explicit flag was safer than risking a CPU-only sibling launched as a CUDA one. Since JustVugg#1533 the check takes the engine that will actually run, so the risk that contract was written against can be tested instead of assumed -- and assuming it costs the user the tier they were just shown. So plan_cuda_enabled() now asks the same question the GLM path has always asked (`has_cuda=cuda_binary()`), of the right binary: COLI_CUDA=1 --gpu/--vram already validated the build; take it at its word COLI_CUDA=0 --gpu none is still the off switch otherwise ask the family's engine, and DeepSeek V4 its own probe, because cuda_binary() rejects valid V4 CUDA builds (JustVugg#1219) A CPU-only build still never gets a CUDA launch: that is now what the check says rather than what the flag assumed. The notice stays, for the two ways a real tier can still go unused, and says which one happened -- "drop --gpu none" and "rebuild with CUDA=1" are not interchangeable advice. It stays quiet when no device qualified to drive placement or the budget was zero, since neither was going to be applied with the GPU on either. The call sits outside the try that wraps plan construction: it only prints, and a display helper that stumbled inside it would be reported as "invalid resource plan" and take the launch down with it. tests/test_auto_tier_vram_notice.py covers the decision in both directions (a GPU build gets the tier without a flag; a CPU-only build does not; --gpu none beats a GPU build; an explicit --gpu is taken at its word without a second probe; V4 uses its own probe) and the notice's two messages and its silences.
fix(qwen36): a truncated multibyte tail made the tokenizer read past the prompt
…ble-race fix(st): publish the lazy shard-mapping table without a data race
…ice-count-load-dll fix(qwen36): the pre-init device-count probe must load the backend DLL (JustVugg#1577)
…-non-object fix(serve): a non-object json_schema answered 500 "engine failed"
…rrogate fix(serve): an unpaired surrogate escape answered 500 "engine failed"
…-dll fix(coli): --gpu on DeepSeek V4 refused an install with only the DeepGEMM DLL
…bri-only fix(planner): auto-tune advised colibri.c-only knobs for every engine
…lm-only fix(autotune): coli tune swept GLM-only CUDA and NUMA knobs on every engine
…akefile-skip-without-make tests: skip the cuda-test recipe checks when make is absent
glm53: replay a bare tool-call turn as the model wrote it (JustVugg#1576)
…evel sgb_array inlined the item schema twice, once for the first item and once for each ","-item, so every level of array nesting doubled the GBNF text: 8 levels of arrays overflowed GR_MAX_RULES and ran without a grammar, and a 567-byte schema of 22 levels compiled to 180 MB in about a second. The compiler accepts 32 levels. An item that holds an array is now queued as one composite rule, jitem<N>, emitted after the root rule and referenced by name at both places. Every other item is inlined as before, so a schema without an array inside an array's items compiles to the same GBNF as before. The language the grammar accepts is unchanged.
gr__normalize advanced a caller past a rule reference and pushed the callee, but left the caller on the stack even when that reference was the last symbol of its alternate. A repetition compiles to R ::= I R, so every repeat of x* or x+ kept one finished frame. After ~60 repeats the stack hit GR_MAX_DEPTH, gr_accept returned -1, and gr_feed turned grammar drafting off for the rest of the output: at the 60th byte of a JSON string, the 56th item of an array, 60 bytes of whitespace, or the 52nd row of the NDJSON grammar in test_grammar.c. Pop the caller when the reference ends its alternate. The frame had nothing left to resume and would be popped as soon as the callee finished, so the accepted language, the admissible bytes and the forced spans are unchanged. Left recursion and epsilon cycles are still caught by the depth argument.
`coli stop` scans /proc to find the serve wrapper and its engine. The scan
is guarded against /proc being absent, so it does not crash on macOS or
the BSDs -- it iterates an empty list and prints
nothing running — no serve on port 8000, no SERVE engines
while the serve and its multi-GB engine are both still running. That is a
worse failure than the crash it replaced. This command exists because the
engine re-execs itself for OpenMP tuning and no longer carries the name it
was started with, which is how two ghost engines once OOM'd a box; a
silent success is an invitation to walk away from exactly that situation.
Discovery is now ported rather than merely guarded. /proc is still used
where it exists. Elsewhere `ps` is asked once for the whole table, and its
space-separated output is normalised into the NUL-separated byte form that
_serve_cmdline_matches_port and _serve_environ_matches_port already parse,
so the matchers themselves are untouched and the port-aware behaviour is
identical on both paths. The environment is read only for processes whose
name already matches an engine, so the common case adds no work.
Verified on macOS 15 against a live `coli serve`: before the change the
command reported "nothing running" with the process alive; after it, the
process is listed and stopped, and with nothing running it still correctly
reports "nothing running".
tests/test_coli_stop_portable.py covers the discovery contract: it plants a
decoy serve and asserts cmd_stop finds it, which fails on the unported code
for the right reason -- "Discovery must be PORTED, not just guarded."
Note: tests/test_openai_server.py already fails on this commit's parent and
is unrelated to this change.
Windows discovery is pidfile-only. cmd_stop has two kinds of target, and the engine one is confirmed by reading another process's environment for SERVE=1 and COLI_SERVE_PORT. Neither tasklist nor wmic can read another process's environment block, so a Windows scan could only ever find the coli serve wrapper and never port-confirm the engine behind it. Discovering the wrapper and then killing it while the engine survives is the ghost-engine failure this command exists to prevent, so a partial port would be worse here than an honest skip. cmd_stop reads the pidfile before it scans and that path does not depend on /proc, so stop continues to work on Windows for the supported flow. The other three tests in this file are unchanged and still run on win32.
No engine build on dev writes the identity-bound wire form
("SCORE <ordinal> <sha256> <exact> <contlen> <greedy>"); SCORE_EVIDENCE
is read by nothing (0 files vs. 13 for the coli_omp_tune positive
control). Remove the dead binding machinery -- _SCORE_EVIDENCE_RE,
parse_score_evidence_result, ScoreStdoutClassifier's request_digests/
binding_mode/mode tracking, the SCORE_EVIDENCE=1 env set, and the
BOUND/UNBOUND reporting -- and the docstring paragraphs describing it.
The per-request durability append and the full stdout line
classification (banner/load preamble lifecycle, foreign-line refusal)
are unchanged.
Bring the binding back when an engine actually writes that record.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Split reused verbatim from upstream/p0-engine-preamble (b6070718), verified there by 18,093 old-vs-new behavioural comparisons with zero divergences. Keeps PreambleError, parse_engine_banner, parse_engine_loaded, parse_engine_preamble and their module constants. Removes ManifestFormError and canonical_manifest_bytes, which travel to JustVugg#1356 with their only consumer (check_ablate_evidence.py, removed from this PR next). The module docstring no longer describes the absent half. This transiently breaks check_ablate_evidence.py's import of the removed names; the next commit removes that module from this PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
matmul_fp8 computes one output row per pass. Computing four, each with its own accumulator, is bit-exact and substantially faster. Bit-exact by construction: each row's sequence of additions is unchanged, same operands in the same column order with the same intermediate roundings. Only the interleaving of four independent accumulator chains differs, and interleaving independent chains cannot alter any row's result. The gain is four independent floating-point dependency chains hiding latency, plus one load of xs[i] feeding four rows. A tail of fewer than four rows clamps the spare indices onto the last valid row and guards the stores, which avoids a separate remainder loop. Measured on an Apple M3 Max at p256 / 40 tokens, N=2, both arms deterministic: decode 49.97s to 41.18s and 0.7805 to 0.9471 tok/s, plus 21.34 percent, with time to first token down 17.0 percent. The patched arm reproduces the unmodified engine's generated-text md5 exactly, so bit-exactness is measured rather than argued.
check_ablate_evidence.py, its test module, and the three release.yml hunks that copy and gate it (the cp into dist/tools, the test -f presence gate, the ast.parse packaged-file check) move to JustVugg#1356, which owns their writer (canonical_manifest_bytes/ManifestFormError, removed from engine_evidence.py the previous commit). test_eval_glm.py's shared byte-limit test drops its ABLATE-side assertions (module gone) and keeps the EVAL-side ones unchanged; test_pack_python.py's HumanOnlyToolsShipExplicitly class is removed verbatim rather than edited, since check_ablate_evidence.py was one of its two pinned entries -- it is carried to colibri_lab/dispatch/2026-09-17-program-d-r2/CARRIED_TO_P2_test_pack_python_class.txt for JustVugg#1356 to restore alongside its own copy of the checker. test_engine_evidence_is_needed_by_the_real_tree, this branch's own addition to that file, is untouched. This is a forward commit that deletes -- intended, not a mistake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The four-accumulator form is algebraically identical to the one-row loop, but that is a statement about the C abstract machine, not about codegen. Under the project's default -O3 -march=native, GCC contracts the multiply-adds differently in the four-chain shape than in the one-chain shape, and the result drifts by about one unit in the last place. That breaks the byte-exact contract tests/test_qwen38_native_weights.c pins against its own independent reference, and a one-ulp logit difference can flip an argmax in the token-exact gates, so a tolerance comparison is not available on this path. Constraining contraction for this kernel restores byte-exactness on x86-64 with AVX2 and FMA: the cited test goes from 179 of 258 rows differing to a clean pass. The guard is scoped with push_options and pop_options so no other kernel in this header changes. clang is deliberately left alone. There the reference and this kernel already contract identically and the test passes; forcing contraction off under clang makes them disagree and the same test fails on arm64. Applying the constraint unconditionally would have broken the platform that was already correct. Rejected alternative, recorded because it is the first thing a reviewer will try: rounding each product into a named float before accumulating does not work, because GCC contracts across statements. It still fails 179 of 258.
…-rollback fix(cuda): reclaim partially uploaded expert tensors
…budget fix(cuda): budget expert scale allocations separately
…rror fix(cuda): propagate Qwen expert collection failures
…leanup fix(cuda): unwind failed Qwen tier initialization and preserve active state
…t-unwind fix(cuda): unwind initialization failures and preserve active contexts
…hutdown fix(cuda): release Qwen tier resources on shutdown
qwen36: offload batched attention projections to CUDA
…refill qwen36: batch resident DeltaNet input projections
…sets, both cuda-test additions
…tVugg#1689) On a merged-/usr system /bin/coli is a link to /usr/bin/coli; abspath kept the alias and the installed launcher looked for /libexec/colibri. realpath first. tests/test_install_closure.py runs the launcher through such an alias, and checks that every root module tools/pack_python.py reaches is in the handwritten make install list (the v41_dsml.py gap of 1.12.0, fixed by JustVugg#1610, is the case it would have caught).
qwen38: CPU prefill batching — runtime chunk/workspace knobs, wider expert load batch, parallel QSA prefill
…s-metrics feat(serve): expose Prometheus admission and generation metrics
…nchmark feat(tools): add reproducible HTTP serving benchmark
…gpu-backend-claims docs/qwen38, README: remove stale "no GPU backend" claims (GPU tier merged in 1.12.0)
…t8 down in one slab) The JustVugg#1370 experiment: Q4_K_M keeps down/output in Q6_K, our gs64 int4 loses 2.5 % perplexity against it, and all-int8 experts close the gap (7.153 against 7.147). The first number JustVugg asked for is down alone at more bits with everything else unchanged. --down-bits N (5..8, with --ebits <= 4) quantizes down_proj at N bits (--down-gs groups along its input, 0 = per row) and writes one uint8 slab per expert: [gate int4 packed | up int4 packed | down int8], 2*inter*hidden bytes -- neither the int4 (1.5x) nor the int8 (3x) size, so the engine can tell the layouts apart from the bytes, as it does today. qs stays [gate | up | down]. meta gains expert_down_bits / expert_down_gs. The selftest covers the mixed round trip byte for byte. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
… CPU path The slab is told apart by size (2*inter*hidden bytes); gate|up unpack into the slot's int8 block as the int4 container does, down's int8 rows are copied behind them, the scales come as [gate | up | down] with down_proj's own group size (expert_down_gs, 0 = per row) from qwen36_meta.json. matmul_qd dispatches down_proj on its own layout; gate/up stay on matmul_qe. The VRAM tier takes one format per expert, so COLI_CUDA=1 is refused with a line on a mixed container until the tier learns it. Tiny fixture (inter 64, gs 64): int8 16/16 tokens, int4 gs64 12/16, mixed 16/16; cap 1, cap 2, PILOT=1 WIDE=2 and ASan/UBSan clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
…JustVugg#1370 numbers in "Which container?" Tiny job: the inter-64 fixture converted with --down-bits 8 must be detected by size, give identical ids at cap 1/2/8 (cap 1 recycles the one slot after every expert, where a wrong slab offset shows), and refuse COLI_CUDA=1 with the line. Docs: the knob, the four perplexity figures, and the verdict -- down alone is a quarter of the gap, the tier does not take the layout yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X2HPETYXNwT4RmQkdMMyPS
coli: resolve the launcher symlink before deriving libexec; install closure test
…pipeline kimi: keep CUDA SiTU expert intermediates on device
…atch CHANGELOG 1.12.1: notes for the eighteen pull requests merged today
qwen36: --down-bits, the mixed expert layout (int4 gate/up, int8 down) -- the knob behind the JustVugg#1370 numbers
fix: resolve static analysis bugs (measure redefinition, dead imports, missing f-string, unused variable)
…-upgrade-to-3.24 ci: upgrade Alpine 3.21 to 3.24 in musl job (EOL Nov 1, 2026)
…e in the VRAM branch
Owner
|
Merged current dev into your branch: format_plan's VRAM line now also prints the int8 trunk bytes on dev, so the VRAM branch of your Metal/VRAM split keeps that; the Metal branch is yours as written. The 72 planner tests pass on the merge. Merging once CI is green. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
While diagnosing slow
coli chatinference on an M1 Pro, we ran:The output incorrectly said:
This was unexpected because
make metal-testsucceeded on the same machine, confirming that the Metal backend and Apple GPU were available.Cause
build_plan()callsdiscover_gpus(). Before this change, discovery only tried:nvidia-smirocm-smiNeither command can detect Apple Metal GPUs, so
coli plansilently reported no supported GPU and showed the CPU path.Fix
Add a macOS-specific discovery path using:
coli plannow detects and displays the Apple Metal GPU. It is reported as unified memory, without inventing a separate VRAM budget or applying CUDA-style placement settings.Summary
Validation