Repository navigation
ci: integration of the fifth release batch (merge after #1931) - #1932
Merged
Merged
Conversation
`required` was prompt-level on four renderers (GLM-5.2, DeepSeek V4, DeepSeek V4.1, Kimi K3) and dropped on the floor by four others that accept it, declare the tools, and then say nothing about using them: Qwen3.6, Qwen3.8, GLM-5.3, and OLMoE under COLI_TOOL_FALLBACK=1. A client sending `required` to those got a 200 and a tool block with no instruction in it, so the model answered in prose and the caller could not tell the difference from a refusal. That is the "accepted in silence" outcome #1698 rules out. The instruction moves into one constant, and all four renderers append it after the tool block -- outside `_glm53_tool_block`, which stays byte-pinned against jinja2, and between the tool block and the client's own system text for the two Qwen families, whose template puts the tool block first. The two engines with no tool block to attach it to still answer 400. Tests pin the property per renderer: `required` is `auto` plus exactly that one line, and the line follows the last declared function. Kimi K3 keeps its own wording, because its wire format carries a dedicated tool-choice message rather than prose. docs/api.md also gains the missing GLM-5.3-Flash row in the tool-calling table, and says plainly that `required` is a prompt-level instruction rather than a sampling constraint. No engine or wire-format change: part of #1698.
Both conflicts are the keep-both unions predicted on #1805. - c/tests/test_openai_server.py: dev's #1802 added `_keepalive_choice` to the `from openai_server import (...)` list; this branch added `TOOL_CHOICE_REQUIRED_INSTRUCTION` to the same closing group. Both names are needed and the list keeps its shape, so the union is the two of them plus the shared `stop_policy, tune_child_env` tail. - docs/api.md: dev's #1804 added the `strict_tool_calls` paragraph to the tool-calling section; this branch added the `tool_choice: "required"` and forced-choice paragraphs. They are about different request fields and neither refers to the other, so both stand, with `required` and the forced choice first since the strict-mode paragraph reads as a follow-on to them.
Catch up past #1821 so the branch includes the serve-variant and MiMo-Pro docs that landed after the first merge.
#1813 landed MiMo after this branch was cut. render_chat_mimo now appends the same required-instruction line as the other block renderers; the matrix row matches that, so the "every engine" claim in the required paragraph is true in the table too.
serve_one threw the slot's prefix away for every request that carried an
image (`if (shared) { /* niente riuso con un'immagine */ slot_reset }`):
a multi-turn chat about one screenshot re-prefilled the whole
conversation, picture included, on every turn. The reset was there for a
reason: the gateway expands every image into the same run of
image_token_id placeholders, so two requests that differ only in the
picture have identical token ids, and slot_shared() compared ids alone.
Reproduced against the base engine's own functions (slot_remember on
[7, IMG, IMG, IMG, 9], slot_shared on the same ids plus one token):
5 of 5 positions "shared" with a different picture pending.
Each slot now keeps, beside its ids, a keyed sequence (kv_image_key.h):
a token id as itself, an image position as a 64-bit FNV-1a of the patch
bytes and their grid with the top bit set, so it can never equal an id
(2^-63 per pair of different pictures). slot_shared() compares keys, so
the same picture resent agrees on every position and a different one
diverges at its first placeholder; pins are restored only when the keyed
agreement covers them, since their ids match any picture. The tower is
then asked only for the placeholders outside the reused prefix, and not
at all when the prefix holds them all; a prefix that holds more image
positions than the tower produces for this grid does not describe this
picture and starts over, as before. A placeholder that arrived without a
frame keys as "no content" and matches only the same.
tests/test_glm53_image_key.c is the state-identity fixture: identical
placeholders with different bytes share nothing (agreement stops at the
first placeholder), identical bytes share everything and the reused
prefix reports its 3 tower outputs, a truncated payload or another grid
is another picture, ids up to INT_MAX never read as an image, a frameless
placeholder matches only a frameless one, and the text rules (extend,
diverge, equal length, empty slot) are unchanged. On the real fixture,
GLM53_VERBOSE=1's REUSE line shows `extend` instead of `image` for the
second turn of a conversation that resends its picture.
… shapes at open The container records one quantBits/quantGroupSize for every expert, and coli_qpack_affine_view (c/qpack.c:649-662) was the first place that pair was held against a section's shapes: after qq_open had sized and allocated every expert slot from the stride (c/qwen36_qpack.c:248-258) and read expert (0,0). A module packed at another width than the manifest says, or a manifest carried over from another export, was refused late and by a generic message. coli_qpack_open now checks every <module>.weight U32 section against its <module>.scales: weight.last * (32 / bits) must equal scales.last * group_size. A disagreement is refused by section name with both widths, before a layer file is opened. A declared quantization the affine contract cannot size by (not 4/8-bit, or no group size) is refused at open too, by its numbers. A manifest with no quantization metadata is left to the affine view as before. Mixed expert quant across modules can only reach the reader as one module disagreeing with the single-valued manifest, so it is the same refusal, named. tests/test_qpack.c: a width mismatch under 4-bit/g8 and under 8-bit/g4, a container whose up_proj was packed at 8 bits beside a 4-bit gate_proj, a 2-bit manifest, and a two-module container that still opens. The leading-dim fixture keeps its view-time purpose with agreeing widths.
The one-shot and PPL=1 modes read prompt_ids and full_ids from the JSON reference and index with them, and nothing checked either. full_ids' length sizes the token buffer and the KV cache, and attention() writes one score per position into sc[4096], a stack array. CTX caps chat and serve at 4096 for that reason; these two paths had no cap, so a reference past 4096 tokens wrote past the array. full_ids no longer than prompt_ids, or an empty prompt, sized the buffers shorter than the prefill writes. An id outside [0, vocab) read past the embedding table, and under PPL=1 past the logits. main() now refuses such a reference after the load and before the run, with a message that names the problem: 1 <= prompt < full, full_ids at most 4096 tokens (OLMoE's max_position_embeddings too), every id inside the vocabulary. tests/test_olmoe_reference_bounds.py runs the engine on the tiny OLMoE container, in the OLMoE tiny oracle job: a reference that fills the 4096 positions runs; one token more, a reference past the score buffer (one shot and PPL=1), full_ids shorter than or equal to the prompt, an empty prompt, a prompt id equal to the vocabulary and a negative last full id are refused by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0121smTcyMpR19LBmeELVUwm
The three engines' attention scored every cached key with the one-chain loop `acc += qv[d]*kv[d]`. Without -ffast-math the compiler cannot reassociate it, so it ran one dependent add per element. This is the same scalar reduction #442 named for the router, on a path that grows with the context. Its bits also depended on the toolchain: x86 clang fuses the add, gcc does not. The score is now dot_f32_lanes() from matmul_f32.h (#1798), which all three already include for their matmuls: 32 fixed lanes, the same bits on every vector width and compiler. The value mix is untouched. tests/bench_attn_f32.c times one decode token's attention core at the Qwen3.6-35B-A3B and OLMoE shapes, scalar score against lanes. It is built on demand and is not a gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0121smTcyMpR19LBmeELVUwm
load_t() allocated each dense tensor from the element count in the safetensors header, and the forward used it with the config's dimensions: q/k/v/o as [hidden, hidden], the router as [num_experts, hidden], embed_tokens and lm_head as [vocab_size, hidden], each norm as [hidden]. Nothing compared the two. A tensor shorter than its config shape was read past its end, by the matmul or by rmsnorm_row(); a key projection written for grouped-query attention, which this engine does not run, is one. A tensor of another shape with enough elements (extra rows, a transposed router, a flat projection) was used as if it had the right one. load_t() now takes the shape the forward uses and refuses any other, rank included, naming both: "<name>: shape [32, 64], expected [64, 64] from config.json". Exact, like the routed experts in load_expert_merged(). The Edge adapter's three boundary tensors go through the same check. tests/test_olmoe_dense_shape.py copies the tiny OLMoE container, rewrites one dense tensor with another shape and runs the engine on the container's reference, in the OLMoE tiny oracle job. The container as converted runs token-exact; short, long, transposed, flat and extra-axis tensors of every kind the engine loads are refused by name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0121smTcyMpR19LBmeELVUwm
A load-time failure in st.h ended in fprintf(stderr) + exit(1); the gateway,
reading stdout for READY, saw EOF and raised "colibri engine exited
unexpectedly" (c/openai_server.py:4659) for an out-of-memory host, an
unreadable path, a truncated shard and an unsupported dtype alike.
load_fail.h: ColiLoadFailKind {nomem, io, format, unsupported},
coli_load_fail_kind_from_errno() (ENOMEM/EAGAIN/ENOBUFS are nomem, every
other errno io), and coli_load_fail(kind, fmt, ...): the stderr line as
before and, with SERVE set, `LOAD_FAIL kind=<kind> <detail>` on stdout
before exit(1). st.h's open/fstat/pread/opendir, short-read, header, dtype
and tensor-malloc exits say their kind through it. serve_codec.h carries the
header; docs/serve_protocol.md documents the line.
openai_server.py: Engine keeps the last 4 KiB the engine wrote before READY;
when the engine exits instead, a LOAD_FAIL line there becomes
EngineLoadError(kind, detail), and serve() logs
`[gateway] engine load failed: kind=<kind> <detail>` and exits 1 instead of
a traceback. read_engine_turn hands over the sentinel-length tail it held
back at EOF, so the line is whole. An engine that says nothing is still
"exited unexpectedly".
tests/test_serve_codec.c pins the errno mapping, the kind words and the
line's bytes; tests/test_openai_server.py EngineLoadFailTest pins the four
kinds, the silent engine, the parser and the serve() log line.
Qwen3.8's DeltaNet gates its norm with sigmoid(z) where Qwen3.6 uses silu(z), and its in_proj is two matrices (qkv, z) where qwen36 fused them: coli_cuda_dn_create takes gate_sigmoid, coli_cuda_dn_step an optional projz (NULL = fused). Same five entry points, loader and fake follow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…e handles qwen38's trunk items are dense handles (dnqkv, dnz, dnout), not the fused dnproj tensor qwen36 uploads; this variant joins the layer to those three, all on one device, with the gate mode the engine names. The step uses the tensors recorded at init on either path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d end to end The same hand-over as qwen36's Q36_DN_GPU: for every DeltaNet layer whose dnqkv, dnz and dnout the placer put on one card, the conv ring, the recurrent state (48 x 128 x 128 per layer) and the conv/norm weights go there too; a decode token computes the two gates on the CPU and hands x to qt_dn_gpu_step. The host arrays stay canonical: a prompt, an MTP verify (snap_after takes its snapshot on the CPU path), a pin snapshot and the prompt cache pull first; a reset, a pin restore and a rejected draft's rollback invalidate the card's copy; a failing step turns the layer off and the CPU continues from what the card holds. Opt-in, beside the Vulkan chain's own hooks, which the CUDA tier keeps off. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the layer on the card documented The fixture's oracle with the whole dense trunk on the fake tier and the two DeltaNet layers stepped on the fake card (sigmoid gate, split in_proj): 8/8 tokens, cosine 0.99998; the same run on the CPU for comparison. Runs from qwen38-dn-gpu-check next to qwen38-tier-engine-check in CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r's own buffers Every expert of a model has the same geometry, so a tier swap can write the newcomer into the victim's device buffers instead of cudaFree + cudaMalloc, each of which synchronises the device under the async expert groups. Same steps as the upload (H2D, the int4 offset-to-signed pass, the scales), into the buffers already there; a failure half-way leaves the content undefined and the caller frees the tensor. Optional in the DLL loader (RESOLVE_OPT): a backend that predates it answers 0 and the caller uploads as before. The fake backend implements it and can be told to refuse, so the fallback is testable without a card. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ting The warmstart fills VRAM from the heat file, i.e. from what earlier prompts routed to, and the LFRU tick corrects that one expert per sixteen tokens and device. qt_replan(layer, counts, cap) takes one layer's routing counts over this prompt's rows and swaps residents the prompt never routed to for its most-routed non-residents of that layer: budget-neutral, the victim-first swap the tick uses, in strict count order, only while the newcomer's count beats the victim's. What the upload queue takes starts at once, the rest is pending and drains on later calls and four per token on the decode ticks, so nothing blocks. layer < 0 takes the whole table and plans per device. The swap writes the newcomer into the victim's buffers when the backend has coli_cuda_tensor_overwrite, and frees-then-uploads otherwise. Evicted slots are reported through qt_evicted_take so the engine can rebuild their RAM copies ahead of the miss path. qt_stats_mark snapshots the counters at the prefill/decode boundary and qt_stats then prints the decode-only hit rate, the number a re-plan moves; the swap line says how many were in place. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ill, layer by layer After each prefill layer's routing loop moe() hands that layer's counts over the prompt rows to qt_replan, so the swaps of layer L upload while layers L+1.. still compute; QT_PREFILL_REPLAN_MAX caps them per layer (24: offline at B=96 that reaches the uncapped plan's hit rate with two thirds of the swaps). At the start of every step the experts the tier evicted get their RAM int8 copies back, in parallel, before the layers run: on the int4 container the warmstart freed them, and left to the miss path 2,000 victims cost ~7 ms/token over the next 300 tokens in slot_ensure_int8. The prefill/decode boundary is marked for the decode-only hit rate. Measured on the 3070 (35B per-row int4, heat file from six other prompts, four held-out prompts, 300 tokens): decode VRAM hit rate 55-63 -> 73-75 % with the trunk on the CPU, 42-50 -> 60-67 % with the trunk in VRAM, MoE phase -2 to -5 ms/token, whole token -8..-10 % with the trunk in VRAM on three prompts of four; TTFT +0.1..1.1 s for the 960 swaps' uploads. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ow-the-prompt section Fake backend, in make check: counts order the plan and only a newcomer that beats its victim is swapped; the cap holds; a layer plan leaves other layers alone while the whole-table form trades across layers; swaps are budget- neutral and in place (the newcomer takes the victim's tensor pointers, no new upload); evicted experts are reported once, with their layer; with the overwrite refused the swap still lands by free-then-upload. Docs: the offline table that motivates it, the mechanism, the measured runs, the TTFT price and its follow-up, the two knobs in ENVIRONMENT.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…verwrite 47 mandatory + 10 optional, 57 exports: the new entry point is optional on purpose, a DLL predating it leaves the pointer NULL and the tier frees and uploads as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…im on every libc qsort is not stable and glibc's and msvcrt's disagree on ties: two never- routed residents with equal heat came out in a different order on the Windows runner, and the test's "4 in, 0 out" saw 1 go instead. The slot index breaks the last tie in both comparators; the plan is now the same on every platform. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
moe_routed ran every prompt row through its 8 experts one at a time: three matmul_q calls per (row, rank), each its own OpenMP region, and the expert's weights read again for every row. For S > 1 it now goes through moe_by_expert(): per block of OLMOE_PREFILL_ROWS rows, the rows are routed in the same order, their (row, rank) pairs are grouped by expert with a counting sort, each expert is fetched once and matmul_q_rows multiplies it against all of its rows (the same inner loop as matmul_q, one region per matrix, each weight row read once for all rows), then every rank joins its row in rank order, as before. So the sum's order and the bits are the one-row loop's. A decode step (S == 1) keeps the one-row loop; IDOT and FUSED3 keep their one-row kernels inside the grouping. The block's other pairs on an expert count as hits, so hits + misses still equal (row, rank) pairs. tests/test_olmoe_moe_by_expert.c builds a one-layer model in memory and checks with memcmp that a prompt of 2..293 rows (one to three blocks) equals the same rows fed one at a time, with norm_topk off and on, by default and with IDOT and FUSED3 forced on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J8Nc7wgKDTE4dZDJUt6XBr
The automatic cache budget (cap 0) subtracted the KV of the whole context (CTX, 1.07 GB at 4096 positions on OLMoE-1B-7B) before a single position was written. The KV's pages are faulted in as positions are written, so a request that writes a few hundred positions never touches most of that memory, and on a machine where the experts do not fit in RAM it was cache slots that never existed. The cache and the KV now share one room. Before each forward, step() calls kv_room_fit(): once the KV's pages no longer fit beside the slots, every layer's cap comes down and its least recently used slots are freed (a pinned one only when nothing else is left, never below one slot per layer; malloc_trim on glibc, which keeps freed blocks otherwise). An explicit cap and the Segment ranges are unchanged. Only which experts sit in RAM changes, the logits do not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DvqUhursKoJqB1wge9kvu8
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
# Conflicts: # c/olmoe.c
# Conflicts: # docs/serve_protocol.md
# Conflicts: # c/qwen38.c # c/qwen38_core.h
# Conflicts: # c/backend_loader.c # c/qwen36.c
# Conflicts: # c/glm53.c # c/tests/vulkan_engines.sh
JustVugg
changed the title
…g/release-batch-4
compat.h requires it on Windows (#error otherwise), as the Makefile's CFLAGS always pass it; the Windows UCRT64 job failed in the test's setUpClass.
#1865 no longer sets the KV aside in the automatic cache (the room is RAM_GB less what is resident less 0.5 GB), so the gate's RAM_GB, which added the KV of 4096 positions, gave the experts that much room and every one of them fit with the host copies too: the cache had nothing to grow into. RAM_GB now leaves the experts half of their bytes beside the dense weights: 27 slots/layer with the host copies, 64 on the device only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One CI run for the external PRs reviewed for the release, on top of the fourth batch (#1931): this branch is
integ/release-batch-4plus the PRs below, merged in this order.tool_choice: "required"instructs the model on every engine #1805. Yoruxyv: docs: add Indonesian README #1907. rayhanhanaputra: fix(web): render GFM tables in chat instead of one run-on paragraph #1905.Conflicts resolved in the merges, against what dev has since:
c/kv_persist.h(fix(kv): preserve retry state after persistence failures #1889): its header check and syncs, through harden config, grammar and model-dir file handling #1930's symlink-safe opens.c/olmoe.c(security(olmoe): hold every dense tensor to the shape config.json gives #1865):load_t's shape check beside the automatic cap's record; the dense-only reload passes the shape too. With olmoe: lend the KV's room to the expert cache until the context uses it #1873 the KV is not set aside in the automatic budget, so the cap's regrowth after the dense weights leave for the device no longer subtracts it and raises the shared room instead.c/qwen38*.h(qwen38: the DeltaNet layer runs on the card (Q38_DN_GPU=1) -- the qwen36 step with a sigmoid gate and a split in_proj #1870): the card's DeltaNet copy is invalidated on dev's multi-draft rollback and pin restore; a verify is told bysnap_rows(dev's), notsnap_after.c/backend_loader.c,c/qwen36.c(qwen36 tier: re-plan the VRAM residents from the prompt's own prefill routing (QT_PREFILL_REPLAN=1) #1769 after qwen38: the DeltaNet layer runs on the card (Q38_DN_GPU=1) -- the qwen36 step with a sigmoid gate and a split in_proj #1870): both entry points, both lines.c/glm53.c,c/tests/vulkan_engines.sh(fix(vulkan): count dense-chain work and dispatch GLM-5.3 f32 residents #1867): f32 residents with dev's device-only and partial-chain checks; the pin harness keeps dev's tolerance and adds the per-row GEMV.docs/serve_protocol.md(serve: say why the model did not load, as a kind the gateway can name #1861): both paragraphs.One commit of mine on top: a zero-width space removed from
docs/api.md.Several of these PRs conflict with dev as they stand, so they cannot be merged one by one. When this run is green and #1931's PRs are in dev, this branch is merged into dev with a merge commit: it carries every PR's own commits, and GitHub marks each of them merged.