[Avicennasis]

The expert-worker protocol moves every routed row as raw f32 both ways. On a LAN that is the cost that scales with workers: distributed-llama's answer is to sync activations as 8-bit blocks, and today on two M1 Max MacBooks over wired 1 GbE its own per-token line showed ~28 ms/token on the wire for 712 kB/token, half of each step. This ports that idea, opt-in, with the token-exact default untouched.

Protocol. COLI_CLUSTER_VERSION stays 1 and its bytes are unchanged. New v2 header is magic version act layer D I n; act is f32 (0) or q8 (1). A q8 row is its ceil(D/32) f32 block scales followed by its D int8 values, qrow_i8 per block of 32 (the engine's own activation rounding), both directions. The coordinator speaks v1 unless COLI_CLUSTER_ACT=q8 (default f32; anything else is refused by name at start), so a cluster of mixed binaries keeps working by default. The worker accepts v1 and v2, echoes version and act in the reply, and refuses a v3 request or an unknown act by name. When a q8 coordinator meets a v1 worker, which refuses v2 by closing, the failure names the protocol versions, the worker's host:port and the COLI_CLUSTER_ACT=f32 escape. Byte counters on cluster_io; the coordinator prints [CLUSTER] transfer: N bytes sent, M bytes received (activations q8) at exit.

Also fixed on the way: the worker's response-header writes break'd on a peer gone mid-header without freeing the batch; they now take the same free path a bad request does.

Tests. test_cluster_protocol: the v1 round trip is untouched; new test_q8_codec_matches_qrow_i8 (D=256/70/1: wire bytes equal qrow_i8 per block, decode within s/2, zero row exact), test_v2_f32_rows_are_raw, test_wire_round_trip_q8 (v2 header over a socketpair, the worker checks the wire bytes, round-trip error ≤ 2·amax/127, and asserts the path is really lossy). test_cluster_sharding.py: the f32 token-exact gates are unchanged; new ClusterQ8DriftTest on both fixtures asserts the f32 delegated run stays oracle-exact and the q8 run scores the same positions with mismatches ≤ Q8_MAX_MISMATCHES (0, measured, documented as no-slack: raise it with a measurement, not silently) while moving < 1/3 of the bytes.

Measured on the tiny GLM oracle (hidden 256, 2 MoE layers, top-2, 32 positions): f32 8203 B/position, q8 2316 B/position (3.54×), oracle argmax identical at all 32 positions on fmt6 and fmt4. make colibri tests/test_cluster_protocol warning-free; pytest tests/test_cluster_sharding.py: 4 passed.

Open choice: the block scale is f32 (the tree has no f32→f16 encoder); f16 like distributed-llama would be another ~6% for ~15 lines if you want it.

[Avicennasis]

CI on this PR has no verdict yet: of its 114 checks, 112 are CANCELLED and 2 SUCCESS (read from the check rollup on 2026-10-09); no job failed. The branch still merges cleanly against dev 1509ea5b (git merge-tree --write-tree against the fetched head produces a tree with no conflicts), and what merged into dev since the base in the files this PR touches (one line, in c/colibri.c) is in other regions than this change, so the reviewed head is left as it is rather than rebased for the sake of a re-run. Could you re-run the checks?

For context: #1968 is closed as superseded by #1774 (glm53.c has its own COLIG53X workers now), and #1964, #1969 and #1970 are being rebased onto 1509ea5b because they conflict.

[JustVugg]

Thanks for the series. #1962 and #1969 are in the external batch of 2026-10-16 (#2112). #1963, #1964, #1967 and #1970 could not go in: they conflict with #1962 and with each other in cluster_moe_batch and tests/test_cluster_protocol.c, and #1964 no longer applies on #1963 as it is now. Once #2112 lands, could you rebase the four as one stack on dev, in the order you intend (we read it as #1963, #1964, #1970, then #1967)? Each will then get its own review against the merged #1962.

[Avicennasis]

Done. The four, plus #2120 (which was stacked on #1970), are rebased as one stack on dev 33b952c (the #2112 merge), in your order with #2120 on top: #1963 → #1964 → #1970 → #1967 → #2120. Each branch carries exactly its own commits over the one below; commit messages unchanged.

PR	new head	commits over dev	conflicts and how they were resolved
#1963	7a4ba6c1	2	cluster_moe_batch: #1962's cluster_xfer/cluster_xfer_u32 wrappers against the v2 header and cluster_act_send/recv. #1962's rule won — every byte cluster_moe_batch moves goes through the counted wrappers: the header words (act included) through cluster_xfer_u32, the rows through two new coordinator-side variants cluster_xfer_act_send/recv (the cluster_act_* bodies with cluster_xfer inside, so only the socket call is in the timed window; the q8 encode/decode stays outside like cluster_item()). The worker keeps the raw codec calls. #1963's own g_cluster_tx_bytes/rx_bytes in cluster_io and the [CLUSTER] transfer: exit line stay separate from #1962's pair on purpose: profile_reset() zeroes #1962's counters after prefill in the one-shot paths, and the exit line (and the q8 gate that reads it) needs a total that is never reset. Tests: your test_moe_batch_counts_the_wire and the q8 tests landed at the same spot; both kept, yours first in main (it asserts the counters start at zero). Its 60/52 bytes still hold at every tip: f32 without a shared buffer is v1 bytes throughout the stack.
#1964	7f4d0525	4	Same lines again (cluster_version_for, the shared item's id, the want check): same rule. cluster_hello keeps raw cluster_io — it runs at connect, outside cluster_moe_batch, which is the scope your comment on cluster_account gives the counters. cluster_exchange (second commit) is the coordinator's wire, so its I/O calls use the wrappers. docs/cluster.md: #1969's "Engines" section now follows the sentence #1964 rewords. Your test's cluster_moe_batch(…) call gained the new , NULL (sh_out) argument.
#1970	e1ff5cde	5	cluster_exchange only: the batch rows sent once and the per-item index lists go through cluster_xfer_act_send / cluster_xfer, counted like every other byte.
#1967	50b1f8df	6	The one real merge. #1967 rewrote the v1-only inline exchange into two phases; on the stack the exchange lives in cluster_exchange() (#1964) and speaks v1/v2 (#1970). So cluster_exchange is now cluster_send() + cluster_recv(), each checking and setting w->failed (#1964's failed-link contract holds for both halves; cluster_exchange stays as their composition, which is what the tests drive), and cluster_moe_batch runs #1967's two phases over the stack's items: every worker's request out, then every reply in, in worker order. On a failure it prints the stack's line (protocol version, activations, the v1-worker hint) rather than the old plain one. Its fan-out test fails on the serial engine exactly as the body says (Assertion 'bar.late == 0' failed — I hit it by accident with the test merged before the engine was) and passes here: 2 workers holding 200 ms each, the batch took 204 ms (served in turn: 400 ms).
#2120	364c492c	7	Written on #1970, now on #1967, so the two compose: all[] is built once in uniq[] order, each worker gets shallow copies (a per-worker which[]), phase 1 sends to every worker, phase 2 receives into items[].y in worker order and hands each y back under its uniq[] index, and the single weighted sum runs afterwards in uniq[] order. cluster_recv/cluster_exchange lose out, as in #2120.

Gates, same invocations on today's dev (33b952c, unchanged worktree) and on the stack tip, Linux/gcc 13 -O3 -march=native:

• make -C c check — dev: exit 0, 205 C test binaries (201 distinct) / 0 FAILED:, Python Ran 2673 tests … OK (skipped=216). Tip 364c492: exit 0, 205 (201) / 0 FAILED:, Ran 2680 tests … OK (skipped=216) — the +7 are the stack's own Python tests, all run. 0 compiler warnings on both. make test-python at the tip, run separately: 2680, OK (skipped=216).
• At every tip: make -C c and tests/test_cluster_protocol warning-free, ok 3/3 (6 / 10 / 11 / 12 / 13 cases vs dev's 3); tests.test_cluster_sharding OK — 4 tests at cluster: opt-in q8 activations on the wire (COLI_CLUSTER_ACT=q8), protocol v2 #1963/cluster: v2 hello that refuses v1 workers by name, and the shared expert on the worker #1964/cluster: a v2 request sends the batch rows once, each expert naming its rows by index #1970/cluster: a layer's requests to every worker before any reply is read #1967 with the bodies' numbers unchanged (q8 2316 → 2317 with the hello → 1758 B/position, mismatches 0; rows-once sent 4102 → 2080 B/position), 9 at the tip.
• cluster: the workers' replies are summed in uniq[] order, bit-exact for any worker count #2120's gate at the tip: baseline vs 1, 2, 3 and 4 workers on both fixtures, 8192 logits, 0 differ every time.

One thing for CI: #1967's timing assertion (elapsed < 1.5 × hold) is load-sensitive — at load ~80 here it tripped in 1 of 10 standalone runs (the batch at 200–232 ms in the other nine; the bar.late barrier assertion, the deterministic one, never did), as its description anticipates. The first lines of #1967 and #2120 now say what they sit on.