Repository navigation
External PRs, batch of 2026-10-16 (56 PRs) - #2112
Merged
Merged
Conversation
Since #700 (b956472), route_trace.h writes two header records, `-1 <n_layers> <n_experts>` and `-2 <version> <engine hash>`, at the top of every non-empty history and every STATS dump. The engine readers and tools/mirror_plan.py drop them (negative layer, as documented in docs/routing-telemetry.md). Four Python tools parsed them as experts: - tools/placement_balance.py and tools/placement_crossval.py: the engine hash becomes one expert with billions of selections. On the shipped c/profiles/qwen36-35b.coli_usage, placement_balance counted 2,580,437,419 selections instead of 796,160 and recommended a 1-slot GPU prefix holding 99.97% of traffic (52.34% for 2,048 slots after the fix). - tools/expert_atlas/analyze.py: the hash inflates every run total, so the per-run normalisation stops correcting for verbose categories, and "-1:92" / "-2:1" are emitted as experts into experts.json (the web dashboard input). On a synthetic 3-category sweep the verbose category took 42 of 48 labels instead of 17 and strong specialists fell from 9 to 3. - tools/expert_atlas/validate.py: every own-set/best-other share collapses to 0.00%, so the held-out accuracy comes from tie-breaking. The expert_atlas sweep writes its per-prompt files through STATS=, so any sweep run since #700 is affected. Skip negative-layer records in all four readers. Regression tests feed each tool a history with the header and require the same result as without it.
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
AUTOPIN and the CUDA tier size the pin by how much memory is free, but nothing shows what that memory buys. usage_coverage.py reads .coli_usage / STATS files and prints the coverage curve: the share of past selections the best pinned set of a given size would have served. - concentration: top 1/10/30% share, Gini, effective number of experts - coverage at a list of budgets (GB with --expert-mb, else % of experts) - smallest budget for each coverage target (default 70/80/90%) - knee: budget maximising coverage minus share of bytes - --min-gain G: where one more GB adds fewer than G points - --vram-gb/--ram-gb: VRAM (less --reserve-gb, default 2 like CUDA_RESERVE_GB) then RAM, in pin_load's order, and what stopping at the knee would free and cost - --holdout: score the ranking on other histories - --row-mb: per-layer widths (int8 MTP row); ranks by selections per byte and shows the gap to pin_load's count ranking Ranking by selections per byte and taking the longest prefix that fits is the optimal pin for equal-width experts and within one expert of optimal otherwise. Header records are read for the dimensions, never counted. On the shipped Qwen3.8 profile, the experts covering 70% of its own selections cover 57.7% of a separate session's (docs/experiments/vulkan-qwen38-mtp-2026-10-05/history.usage.gz); the docs say so next to the in-sample numbers. Standard library only. Tests cover parsing, targets, knee, lambda point, budget fitting, mixed widths, the VRAM/RAM plan, holdout and the shipped Qwen3.6 profile.
A clustered run has no way to see what the network costs it: PROF carries disk service, wait, matmul, attention and head, and the wire's share lands in "other". distributed-llama prints per token G (the whole step), I (local inference), T (transfer) and the bytes each way; this gives colibri the same figures on the lines a dashboard already reads. g_cluster_ns/tx/rx accumulate on the coordinator only, around the socket I/O cluster_moe_batch does (headers included; cluster_item() and the weighted accumulate stay outside the timed calls), through two timed wrappers over cluster_io/cluster_u32. A worker's own I/O is not counted. ProfBase snapshots them, so every window is the request's, like edisk. With CLUSTER_WORKERS set: - PROF continues " <cluster_s> <tx_bytes> <rx_bytes>" after n_fw. Unlike edisk this is a wall-time phase: the compute thread does nothing else while it waits. - STAT continues " cluster_g_ms= cluster_i_ms= cluster_t_ms= cluster_tx_kb= cluster_rx_kb=", per emitted token over the line's own window, on the mux DONE line and on the single-slot line (which already carries key=value). - the PROF=1 report prints a "[PROF] cluster:" line next to expert I/O. Without a cluster the counters stay 0 and every line is byte-identical. openai_server.py keeps the extra PROF fields on the profile turn (cluster_transfer_s, cluster_tx_bytes, cluster_rx_bytes) and the STAT key=values on the stats dict; a bare positional STAT tail is still ignored. test_cluster_protocol drives cluster_moe_batch over a socketpair and asserts the counters move by exactly the bytes that crossed it (60 out, 52 in) and not at all for the raw round trip; test_openai_server covers both parsers.
release 2.0.0: dev to main
The bf16 shared-expert matmul has three CPU paths: a four-row AVX2 tile for prefill, its single-row remainder, and the scalar loop that decode and the non-AVX2 build use. The tile was written as a separate multiply and add so it would match the scalar loop "bit for bit", but which of the two the optimizer fuses into an FMA is its own decision: with -ffp-contract=fast, GCC 15.2 fuses the tile under -mtune=skylake and leaves it alone under the generic tuning of x86-64-v3. On a Skylake host the default ARCH=native build therefore rounded prefill rows differently from the same rows decoded one at a time, and test_inkling_shared_batch failed in bf16 while CI, which runs on other tunings, passed. Spell the rounding out instead: _mm_fmadd_ps in the tile, fmaf in both scalar loops, so every path performs the same single rounding whatever the compiler decides to contract, as gsgemv.h already does for the same reason. The tile now requires __FMA__ as well as __AVX2__, like the other FMA kernels; a build without it takes the scalar loop. The test also runs batches of 4 and 5 next to 3 and the default, so the tile is compared with no remainder and with a remainder of 1. Before this change the default, 4 and 5 fail with ARCH=native, ARCH=skylake, ARCH=haswell and ARCH=x86-64-v3 EXTRA_CFLAGS=-mtune=skylake on an i7-7700K; after it every one of those passes, as do x86-64-v3, x86-64 and -ffp-contract=off. Fixes #1864.
Tests only. Freeze four falsey malformed JSON types, four truthy rejection controls and six absent/null/empty-list legacy fallback controls. Existing HTTP fixture and all original methods unchanged. No production/workflow changes or inference/hardware claim. Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Keep explicit falsey non-array tools available for the existing named400 type validation. Only absent/null/empty-list tools retain legacy-functions fallback. Preserve all frozen native HTTP cases, existing error contracts and all original workflows. Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Tests only: freeze four truthy non-object decoded arguments, malformed-JSON rejection and two object prompt controls through the actual gateway CLI subprocess with COLI_TOOL_FALLBACK=1, owned HTTP and existing mock SERVE engine. Preserve all original fixture owners, production files, manifests and workflows. No model quality, pretrained checkpoint, actual GPU or Windows-owner claim. Accepted1597 GLM52/53 repair remains unchanged; this qualifies the remaining opt-in helper caller independently. Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Use the existing named client error for truthy non-object history arguments in the opt-in fallback renderer. Preserve falsey empty-call compatibility, valid object rendering, default-off behavior and accepted GLM/native Qwen paths. Preserve all seven frozen scenarios and original fixture/workflows. Native full Python BEFORE has exactly four public HTTP500 failures and three new controls, zero errors; other before lanes are retained with actual statuses. Require unchanged full114 native AFTER before upstream publication. No checkpoint/GPU/modelquality claim. Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
With its default expert kernel, qwen36 repacks every INT4 routed expert it reads from disk from the container's signed nibble pairs into the planar layout of the expert kernels (xf_repack_pairs_signed, called by load_expert_merged). x86 has an AVX2 path; on ARM the repack ran the scalar loop. The NEON path builds the 32 planar bytes of a 64-element block with two loads, shifts, two zips and two stores: the same bytes as the scalar loop. The zips are AArch64 instructions, so 32-bit ARM keeps the scalar loop. test_expert_ffn checks the repack against its definition with every byte value in every block position, on whichever path the build takes (AVX2, NEON or scalar), and that an AArch64 build compiles the NEON path. A NEON path left out of the build fails it, as does a wrong NEON constant (0x89 for 0x88).
The record of the previous commit against dev on Qwen3.6-35B-A3B with the CPU engine: two manifests (generation after a 512-token prompt, and that prompt; both an improvement), the summary, the raw logs of every run, a first series of the same runs, the harness that produced them, the repack timed alone, called as load_expert_merged calls it, and four more runs with the process's memory footprint read every second. Every logit is the same in all runs of both versions.
Adds web/src/i18n/pt-BR.ts with all 252 keys of en.ts, and registers the
locale in index.ts (import, LOCALES, DICTS) and in the i18n.test.ts
dictionary map. Locale detection already routes navigator.language "pt*"
to it, since detectLocale() matches on the language prefix.
Translation was produced with the local Hunyuan-MT model (hy-mt2:1.8b) and
then reviewed by hand. The review was necessary: the model handles full
sentences well, but short UI labels came back as noun phrases for
imperatives, and the brand wordmark ("colibrì") was translated as the
animal. Those entries were rewritten.
Placeholders are unchanged; `npm test` (97 tests) and `npm run build` both
pass.
serve_sample, the chat sampler for temperature > 0 (the gateway's default is 0.7 with top_p 0.9), took the probabilities of all 248,320 tokens and sorted all of them with qsort to find the top_p nucleus. It now puts the probabilities into buckets by binary exponent, takes the buckets from the top until their total passes top_p of the sum, plus one more for rounding, and sorts only those tokens. The walk over the sorted prefix and the draw are as before, so the same rand() value draws the same token, unless tokens of exactly equal probability sit where the walk or the draw stops: they are now in token id order, which the full sort left to qsort. With top_p 1 every token is in the nucleus, and when the candidates do not reach the cut the whole vocabulary is needed: both take serve_sample_sorted, the full sort as before. Its comparator now also orders ties by token id, which makes that sort 0.8 ms slower on the M1; in exchange the token drawn no longer depends on the C library's qsort. The greedy path (temperature 0) is unchanged. The sampler moves to qwen36_sample.h, with a test against a full sort (tests/test_qwen36_sample.c: 1612 draws at Qwen3.6's vocabulary size, temperatures 0.2-2 and top_p 0.3-1, one nucleus made only of ties, seeds spread over the range of rand() and runs of four draws from one srand(); with the tie order reversed it fails 127 of them, with one more rand() per draw 205, and with the last token of the nucleus never drawn 72).
The record of the previous commit against dev: the time of a draw on the logits of every token of three chat replies of Qwen3.6-35B-A3B at the gateway's default (temperature 0.7, top_p 0.9), and on the last token of three prompts at five settings; the manifest; the raw output of every run; the prompts, the replies and the SHA-256 of the logits; the programs that saved the replies, timed the sampler, counted the tokens it sorts and compared the draws; and the separate measurement of what the comparator that orders ties by token id costs a full sort. Every run of a version draws the same tokens; between the versions only draws between two tokens of exactly the same probability differ.
Adds README.pt-BR.md, a translation of README.md, and links it from the language bar of the other six READMEs. The translation was done by hand from the English source, not from a machine translation pass, and it was checked against the original rather than assumed: - the 122 link targets are identical and in the same order; - the 8 code blocks are byte-identical; - the 9 table separator rows are intact; - <p>, <img>, </p> and <a href> counts match. One thing the translation had to handle: Markdown derives an anchor from the heading text, so translating "## Which model for my machine" would break every `(#which-model-for-my-machine)` link. Four explicit `<a id="...">` anchors were added before the translated headings that are link targets, the same device the file already uses for `get-started` and `the-one-step-way`. Every internal `](#...)` resolves. Proper nouns, command names, environment variables, file paths and the technical terms the project uses as such are left untranslated, as in the existing translations.
The GLM engine can hand a layer's routed batch-union to disk-backed expert workers on other machines (docs/cluster.md); the DeepSeek V4.1 engine could not, and it is the one the workers were for -- a 475 GB container on a 64 GB Mac, the experts resident across several. This carries the same protocol in deepseek_v41.c: the COLIEX01 magic, network-order u32 headers, raw f32 rows, one request per layer per worker, expert (eid, layer) on worker (eid + layer) % n -- byte for byte what colibri.c sends. A dsv41 coordinator talks to dsv41 workers only. What moves is exactly what moe_run_at hands to expert_ffn_rows for the backbone: the fp4 routed experts. The worker loads the config and the shard index, keeps one slot per layer through the same expert_slot_at, and runs the same matmul_mxfp4 kernel, so a row comes back as the bytes the local path produces; the coordinator writes it into contrib[d] = w * y where the local loop would have, and the rank-ordered sum after it is the same sum. Routing, the shared expert, attention, the DSA indexer, the engram tables, the vision tower and the DSpark stages' own experts stay on the coordinator; a v1 request names a backbone layer and nothing else. Off unless CLUSTER_WORKERS names a worker; the single-machine path is untouched, and colibri.c is not edited (its cluster block has open PRs against it -- a shared header is the follow-up once those land). Pinned two ways, as the GLM engine is: tests/test_dsv41_cluster_protocol.c holds the wire contract over a socketpair, and tests/test_dsv41_cluster_sharding.py decodes the tiny V4.1 fixture locally and through a worker at cap=1 and cap=8 and requires the same tokens, no [mismatch] line on either side, the coordinator's own "connected to 1 expert worker(s)" and fewer expert bytes read than the baseline -- so a coordinator that ignored CLUSTER_WORKERS cannot pass it. Perturbing the worker's output by one float fails both caps. `coli cluster worker` accepts the family; ci.yml runs the gate the way cluster-parity runs GLM's.
…layer>, default 1) The worker kept one expert slot per layer, so every routed read past the first of a layer was a cold SSD read plus the round trip the coordinator waits on, while the single-machine path runs a RAM-plan cache (48 slots/layer on a 64 GB Mac). Measured on two M1 Max MacBooks over wired gigabit with DeepSeek-V4.1- Flash: 0.46 tok/s single, 0.36 with the worker -- delegation halved the expert reads and the cold cache gave the saving straight back. CLUSTER_WORKER_CACHE=<n> sizes the worker's per-layer LRU like --cap does the coordinator's; default 1 keeps today's memory footprint.
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
With the int8 dense trunk the engine converted the whole embedding table to f32 at start and kept it, 248,320 x 2048 floats or 2 GB of RAM on the 35B, to read one row per token. qwen36_embedding.h maps the checkpoint's embed_tokens tensor read-only and converts a token's row when the forward reads it, with st.h's bf16/f16 helpers: the same f32 values, so the logits are unchanged. The mapped pages take no memory until they are read, and the system can drop them again. The decode step of several conversations at once (KV_SLOTS) reads the rows through q36_embed_row too. The engine says on stderr when it reads the rows in place. COLI_EMBED_MMAP=0 keeps the table. COLI_KEEP_F32, COLI_DENSE_BITS=16 (which keeps the table in f16) and a Vulkan run (COLI_VULKAN=1: qwen36_chain.h rebuilds the state from the table) keep their own paths, and a tensor it cannot map (another dtype or shape) falls back to the table. A VK=1 build that runs on the CPU, as the Linux and Windows release binaries do without COLI_VULKAN=1, reads the rows in place. The name is resolved as for the other dense tensors, with or without the language_model. prefix. The Segment and Edge adapters do not load the embedding through this path. tests/test_qwen36_embedding.c checks the rows against st.h's reader for every 16-bit pattern and for f32 special values, the error paths, and that every mapping is released; swapping the bf16 and f16 conversions makes it fail. A new step of the qwen36 tiny CI job runs the fixture with the int8 trunk both ways, which must give the same layer outputs, logits, ids and exit status, and serves four conversations at once.
At every start the engine reads the dense matrices and quantizes them to int8 with qw_quantize: 1.9 GB of int8 on the 35B, each matrix through an f32 copy. qwen36-canonical (tools/qwen36_canonical.c) does that once and writes canonical.qc beside the model (the format is in qwen36_dense_file.h: a header, the table of the matrices, then each matrix's int8 rows and scales on a 16 KiB boundary). When the file is there, or QWEN_DENSE_CACHE names it, the engine reads the matrices from it instead of converting them: the same bytes, so the same logits, without the conversion and its f32 copies. They are copied into ordinary memory, as after a conversion, and the file is closed once they are: the CPU reads every dense matrix at every token, and file pages would be dropped by the expert reads and read again (on an 8 GB M1, with a 64-token prompt and 8 generated tokens, using them in place made the prompt 5% and generation 29% slower than dev's conversion). The file is checked before it is used: the list of matrices and their shapes, the quantizer and the first and last 128 bytes of every source matrix go into a fingerprint, so a file made from other weights (a full fine-tune of the same shape too) or for another quantizer is not used; it samples the source, it is not a checksum. A file beside the model that does not match is set aside with the reason and the matrices are converted; one named by QWEN_DENSE_CACHE stops the start. It is used only where every dense matrix is the int8 copy the engine would make: the whole model, COLI_DENSE_BITS=8, no COLI_KEEP_F32, a run on the CPU (not COLI_CUDA=1 or COLI_VULKAN=1) and not in the adapters. A VK=1 build running on the CPU, as the Linux and Windows release binaries do without COLI_VULKAN=1, uses it. The list of matrices follows the loader (no router in a dense model, no shared expert where there is none), and the tool reads names with or without the language_model. prefix. tests/test_qwen36_dense_file.c puts synthetic BF16, F16 and F32 matrices through the tool's own writer, in one block and in blocks of a few rows (the same bytes), and back through the engine's reader, and checks that every kind of wrong file is refused with its reason. It pins the quantizer to QDF_QUANTIZER. A changed fingerprint, quantizer, scale copy, block offset or header each make it fail. A new step of the qwen36 tiny CI job writes the file for the fixture and runs the engine with and without it (the same layer outputs, logits, ids and exit status), then with a damaged file, named and beside the model.
The record of the previous commit against dev on Qwen3.6-35B-A3B with the CPU engine: prompts of 64 and 512 tokens, each followed by 32 generated tokens. Two manifests, one for the generation after each prompt (both an improvement on this 8 GB Mac), the summary, the raw logs of every run with the start time, the peak memory and the swap counters, and the harness. Every logit and every generated token is the same in all runs of both versions. The commit measured differs from the previous one by four small changes to how the embedding is loaded (a line on stderr, how COLI_EMBED_MMAP is read, the tensor's name resolved as for the other dense tensors, the table kept only when Vulkan is on at run time), the test's output, the docs, a CI step and the commit message, and by the dev under it (94dee9d then, 1509ea5 now: the harness compiles to the same code section from both).
The record of the previous commit against the embedding change it is stacked on, on Qwen3.6-35B-A3B with the CPU engine: prompts of 64 and 512 tokens, each followed by 32 generated tokens. The manifest (the generation after the 512-token prompt: no change), the summary, the raw logs of every run with the start time, the peak memory and the swap counters, the tool's output when it wrote the file, and the harness. Every logit and every generated token is the same in all runs of both versions. The commit measured differs from the previous one by the changes its manifest lists. on-dev-alone.txt is the same change measured on dev itself, without the embedding change: there the generation after the prompt varied too much to compare and both versions wrote to swap in every run; on the embedding change neither happens, which is why this change is measured on it. in-place.txt is a variant that used the matrices in place in the file: slower, with the matrices' pages read from the disk again.
…nment-20261013
…tatus-20261013
…terisks-20261013
# Conflicts: # docs/ENVIRONMENT.md
# Conflicts: # .github/workflows/ci.yml
…rides-20261008 # Conflicts: # c/setup_flow.py # c/tests/test_setup_flow.py
…readiness-20261014 # Conflicts: # c/setup_flow.py
# Conflicts: # c/tests/test_openai_server.py
This was referenced Oct 16, 2026
…on AVX2 builds only
- coli: an image path may start with a drive and either slash (C:/... as Python on
MSYS2 and many Windows tools write it), never with a letter inside a word
("Nota:/tmp/x.png" stays /tmp/x.png). #2078's apostrophe tests failed on the
Windows runner, whose temporary folder is D:/a/_temp/...: only the part after the
drive was recognised, and it stopped at the apostrophe.
- tests/test_olmoe_matmul_q.c: #2039's FUSED3 section calls moe_fused3, which exists
with the AVX2 kernels only; macOS on ARM did not compile the test.
JustVugg
marked this pull request as ready for review
October 16, 2026 13:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
56 reviewed external PRs in one branch, each joined with a merge commit (no squash), so every one of them closes as merged when this lands, and one CI run covers them all.
Included
Conflicts resolved in this branch
cmd_setup's ready path keeps dev's steps (split directories, release CUDA). An explicit--backend,--no-gpu,--hostor--portnow skips that path, as fix(setup): honor explicit overrides on a ready saved configuration #1993 intends.config_readychecks both the shards in the split directories (dev) and the post-install outputs (fix(setup): require generated tokenizer before reporting ready #2079).docs/ENVIRONMENT.mdare kept.ci.ymlkeeps the embedding-in-place andcanonical.qcsteps and addstests.test_qwen36_text_stopsto the prefix-serve line.tests/test_matmul_f16rule no longer lists its headers by hand: the-MMDfiles carry them, andtest_makefile_depsrefuses hand lists.Not in this batch
Checked locally
On a Linux box (Zen 4):
make test-cis green.make test-pythonis green: 2,597 tests, 227 skipped for missing optional packages.