Repository navigation
cluster: v2 hello that refuses v1 workers by name, and the shared expert on the worker - #1964
Merged
Merged
Conversation
This was referenced Oct 6, 2026
Avicennasis
force-pushed
the
p/dense-sharding-design
branch
from
October 9, 2026 21:07
a0eb920 to
601dfca
Compare
Contributor
Author
|
Rebased onto What changed in the rebase: one conflict, in One commit added on top, Ran here on the same Linux box, against
The push should restart the cancelled matrix. |
Owner
…tocol v2 The expert-worker protocol moves every routed row as raw f32 both ways, so on a LAN the transfer, not the expert FFN, bounds a delegated layer. distributed-llama syncs its activations as blocks of 32 int8 behind one scale for a ~4x cut. Same idea here, behind an env switch and off by default: the cluster parity gate is token-exact and int8 rows are not. - COLI_CLUSTER_ACT=q8: the coordinator speaks a v2 header (magic, version 2, act, layer, D, I, n) whose rows are ceil(D/32) f32 block scales followed by D int8 -- qrow_i8 per block, the engine's own activation rounding, so no new quantizer -- and the worker answers in the same format. f32, the default, sends the v1 bytes unchanged, so a cluster of mixed binaries keeps working. - The worker accepts v1 and v2, names the versions it speaks when it refuses one, and refuses an unknown act word by name. A coordinator whose v2 request meets a closed socket names the worker and the COLI_CLUSTER_ACT=f32 escape hatch instead of just "failed". The worker's response-header writes free the batch when the peer is gone mid-header, as a bad request already did; the act word had widened that leak. - The coordinator reports bytes sent and received at exit, so the cut is measurable: on the tiny GLM oracle, 8203 -> 2316 B/position (3.54x) with the same argmax at every position. - test_cluster_protocol: the q8 codec byte for byte against qrow_i8 (partial last block, zero row, decode within half a block scale), a v2 q8 round trip over a socketpair, and a v2 header with act=f32 still carrying raw rows.
test_cluster_sharding's f32 gate is unchanged: the same assertions, now on shared _expert_worker/_coordinator helpers. ClusterQ8DriftTest runs the delegated oracle with COLI_CLUSTER_ACT=q8 on both fixtures and asserts that the f32 delegation stays exact, that the q8 run scores the same positions and mismatches at most Q8_MAX_MISMATCHES of them (measured 0 on fmt6 and fmt4; the bound carries no slack, raise it with a measurement), and that q8 moves less than a third of f32's bytes; it prints bytes per position for the record.
…ert on the worker
On top of the q8 protocol v2 (COLI_CLUSTER_ACT=q8, header magic version act
layer D I n): the default still speaks v1 bytes, so a cluster of mixed engine
builds keeps working. The coordinator speaks v2 when it needs q8 rows or the
shared expert, and v2 gains two words v1 has none for:
- hello: an empty v2 request (n==0, act as the coordinator will speak it) at
connect time, sent only when v2 will be spoken at all. A worker built from
an older engine closes on version 2, and cluster_init refuses it by name
("expert worker HOST:PORT closed on the COLIEX01 v2 hello: it runs an older
engine (v1); rebuild the worker, or run this coordinator without
COLI_CLUSTER_ACT=q8 and CLUSTER_SHARED=1") before any token is generated. A
worker answers the hello before its model checks, with an empty response in
the same act; its refusals of an unknown version or act are unchanged.
- item eid 0xFFFFFFFF: the layer's shared expert. With CLUSTER_SHARED=1 the
coordinator adds one item of every row to the layer's first routed request,
on the worker layer%n picks, in the request's act; the worker loads the
three shared matrices once (qt_load + qt_planarize, the coordinator's own
resident branch) and runs FASE E's CPU sequence (matmul_qt, siluf,
matmul_qt); the coordinator adds the rows where FASE E adds its own, so
with f32 rows the f32 order of the sum does not move. Off by default.
GLM-5.2: 2.83 B params a token off the coordinator. A v2 request may carry
65 items (64 routed + the shared one); v1 stays at 64.
DUMP=<path> (raw f32 logits rows) was compiled only under COLI_VULKAN; it is
now on every build so the cluster gate can compare runs byte for byte.
Tests:
- tests/test_cluster_protocol.c: the hello against a v1 worker (refused, by
name and version, read back from stderr) and a v2 worker (accepted); the
real cluster_moe_batch over a socketpair with the shared item (a v2 request)
and without it (v1 bytes, the default).
- tests/test_cluster_sharding.py: the f32 parity gate gains a third run with
CLUSTER_SHARED=1 that must match the local run's teacher-forcing signature,
show zero oracle mismatches, and match the local run's logits bytes; the
worker's stderr must name every sparse layer it loaded the shared expert
for. A 0.1 % scale on the worker's shared output flips no token on the tiny
fixtures but fails the byte comparison; zeroing it fails the oracle on
27/32 and 30/32 positions. The q8 drift gate is unchanged; its transfer
summary now includes the hello's 56 bytes per worker per run.
…nnect The coordinator's exchange with a worker -- the request header and items, then the reply's header and items -- is one function now, cluster_exchange, and a transport or protocol failure in it (a short read, EOF mid-reply, a magic, version, act, status, item id or row count that is not the one asked for) marks that worker's link failed. Every later exchange on a failed link returns -1 before touching the socket: nothing is sent, nothing is read, so the bytes a worker still sends after a failed exchange -- the rest of a reply the coordinator stopped reading, or a well-formed reply behind it -- are never read as the answer to a later request. A link comes back only by reconnecting. The hello's refusals mark the link the same way. cluster_moe_batch is unchanged in what it does: it builds the items, calls the exchange, and on -1 prints the worker by name and exits, as before. The rule is ds4's after a one-sided transport failure (antirez/ds4 f0962e3): refuse everything on the link, never consume drained bytes as a reply. tests/test_cluster_protocol.c pins it over a socketpair in two shapes. A worker whose reply header claims two items for a one-item request, with a well-formed 52-byte reply queued right behind it: the first exchange returns -1, the second returns -1 at once and the 52 bytes are still on the link, unread, and the worker saw no second request. A worker whose reply stops after the first of two rows and shuts its write side: the same, with nothing pending. Without the link guard the second exchange reads the queued reply as its own (the test fails on rc == -1); with it, 8 of 8 runs pass. 0 warnings.
Avicennasis
force-pushed
the
p/dense-sharding-design
branch
from
October 16, 2026 16:58
601dfca to
7f4d052
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1963: the first two commits here are that PR; the change to review is the last commit (
a0eb920d). Once #1963 lands this diff is only that commit.Two things, both on the v2 header #1963 introduces (
magic version act layer D I n), with the default still v1 bytes:A hello at connect, only when v2 will be spoken. With
COLI_CLUSTER_ACT=q8orCLUSTER_SHARED=1the coordinator sends a v2 request withn==0to each worker at connect and expectsmagic 2 act 0 0back before any model work. A v1 worker closes on it, and the coordinator says so instead of failing later mid-layer:[CLUSTER] expert worker HOST:PORT closed on the COLIEX01 v2 hello: it runs an older engine (v1); rebuild the worker, or run this coordinator without COLI_CLUSTER_ACT=q8 and CLUSTER_SHARED=1. A default f32 run sends no hello and no v2 bytes, so mixed binaries keep working as before.The shared expert on the worker,
CLUSTER_SHARED=1(off by default). The layer's shared expert rides the layer's first routed request as one more item,eid=0xFFFFFFFF(v2 only; a v1 request with it is refused by name). The worker loads it once with the coordinator's ownqt_load+qt_planarize, runs FASE E's CPU sequence, and the coordinator adds the rows at FASE E's own add point. Its rows obeyact, so they are q8 under q8. This removes 2.83 B (GLM-5.2) / 1.06 B (GLM-5.3) params per token from the coordinator's dense reads for +2·S·D·4 bytes per layer, bit-identical in f32.The gate got stricter on the way.
DUMP=(raw f32 logits rows) is lifted out of#ifdef COLI_VULKANso the CPU build can compare bytes;test_cluster_sharding.pynow runs baseline / delegated /CLUSTER_SHARED=1and asserts equal teacher-forcing signatures, 0 oracle mismatches, and identical logits bytes. A negative control (×1.001 on the worker's shared output) flips no token but fails the bytes assertion on both fixtures; the token-only gate was blind to it.Tests (gcc 13.3, Linux, 0 warnings):
test_cluster_protocol8 cases, the 5 from #1963 plus: hello refuses a v1 worker by name and version (read back from captured stderr), hello accepts a v2 worker, and realcluster_moe_batchover a socketpair is v2 with the sentinel item when a shared buffer is given and v1 bytes without one.pytest tests/test_cluster_sharding.py: 4 passed (fmt6/fmt4 parity with the shared-expert run, and #1963's q8 drift gates; the hello adds 56 B per run, so q8 reads 2317 B/position there instead of 2316).Background: this is the first step of a dense-sharding study (what distributed-llama's row/column split would buy on colibri's expert-worker protocol). Headline from the configs: on GLM-5.x, dense TP loses on 1 GbE and is a wash on Thunderbolt until experts are fully resident; where it is a capacity requirement is Kimi K3's ~54 B dense params per token. Two v1 properties worth fixing before any of that:
cluster_moe_batchserves workers serially, and every routed item carries its own copy of the row (top-8 sends the same row 8 times). Happy to file those as follow-ups if you want them.