Skip to content

glm53: expert workers over the COLIEX01 wire, as colibri.c has them - #1968

Closed
Avicennasis wants to merge 1 commit into
JustVugg:devfrom
Avicennasis:p/glm53-cluster
Closed

Avicennasis wants to merge 1 commit into
JustVugg:devfrom
Avicennasis:p/glm53-cluster

Conversation

@Avicennasis

Copy link
Copy Markdown
Contributor

The expert-worker cluster mode exists in one engine, colibri.c. This gives glm53.c the same thing, same wire (COLIEX01 v1, byte-identical header and item layout), env-gated and off by default; the single-machine path is byte-identical (built the pristine glm53.c beside it and cmp'd stdout on the tiny and the streaming-int4 fixtures: identical, counters included). Delegation applies only when m->streaming (the int4 U8 container); resident-f32 experts never reach the hook, and the worker refuses such a checkpoint by name. KDA/MLA/indexer/mHC untouched; the shared expert stays on the coordinator.

One deliberate difference from colibri.c, coordinator-only: adding each worker's replies into out worker-by-worker changes the per-token float summation order against the local union-order loop. Measured here: 1 worker bit-identical; 2 workers same tokens but 66/128 last_logits differ at ~1e-8. glm53.c's cluster_moe_batch collects every worker's y first and sums in union order, so it is bit-exact for any worker count. The same order dependence exists in colibri.c (its gate is token-only, so it never saw it); I left that file alone because #1962/#1963/#1964 are open on its cluster block and will send the ~15-line fix after.

Worker: load_cfg + st_init + glm53_mirror_setup + the prefix probe (mirrors model_load_range), then expert_geometry + expert_table_init, which validates every piece up front, so an absent expert fails at start, not on first use. It also rejects layer < first_dense (dense layers have zeroed eref entries here). coli cluster worker learns the glm53 argv (cap only; precision is GLM53_BITS); WORKER_FAMILIES=("glm","glm53").

Gate (tests/test_glm53_cluster_sharding.py, GLM53_TINY_STREAM): glm53.c has no [ORACLE] mismatch mode, so the gate compares against ref.json: baseline equals the f32 oracle's teacher-forcing and greedy ids per position; delegated equals baseline byte-for-byte on teacher_forcing, greedy and last_logits, and via every prompt prefix 1..8 on last_logits at every position; delegation acted (two workers, experts hits 0 miss 0 bytes 0 on the coordinator); an unreachable worker makes the engine refuse, never fall back silently. The fixture is the text-only streaming pair from make_glm53_streaming_pair.py: the multimodal int4 CI fixture flips one teacher-forced token against its f32 ref on the pristine engine (int4 rounding on a 128-wide random model), so it cannot carry a zero-mismatch gate. Negative control: a 1e-3 nudge on the worker's output passed the first, last-position-only gate; the per-prefix per-position logit parity catches it (1 failed), reverted, 2 passed.

make glm53 colibri tests/test_glm53_cluster_protocol 0 warnings; protocol test ok; glm53 gate 2 passed; the GLM-5.2 gate still 2 passed; test_makefile_deps 8 passed; test_cluster.py + test_glm53_oracles.py 9 passed. check.yml's glm53-oracle job generates the text int4 pair and runs the gate.

Note: a sibling PR for deepseek_v41.c touches the same three need_worker_model lines in c/coli; whichever lands second is a trivial rebase.

GLM-5.2's local cluster mode (docs/cluster.md: CLUSTER_WORKERS, EXPERT_WORKER=1,
one request per layer per worker carrying the routed batch-union, the experts
sharded by (eid + layer) % workers) lived in colibri.c only. glm53.c now speaks
the same wire, byte for byte -- the COLIEX01 magic, the v1 header, the per-item
layout -- so the protocol stays one thing; a glm53 coordinator reaches glm53
workers, since the expert math (the int4 gs64 pieces, the clamped SwiGLU) is the
family's. Off unless CLUSTER_WORKERS is set: without it the engine's stdout is
byte-identical to the previous build's on the tiny f32 and int4 fixtures
(teacher_forcing, last_logits, greedy, the expert counters).

In glm53.c: cluster_io, cluster_u32, cluster_connect_one and cluster_init as in
colibri.c; cluster_item and cluster_moe_batch over ffn_layer_ex's chosen, weight
and union_ids (every token has topk entries here, there is no keff); the hook
once the union is built, by blocks of 64, in place of the cache-sized block loop
(the Vulkan tier, taken before it, wins over it as in colibri.c, and so does the
resident f32 path, which never streams); cluster_worker_run, which loads the
config, opens the shards, indexes the experts (expert_table_init checks every
piece's presence and length up front, so a request can never name a tensor the
worker cannot resolve), keeps one Slot per layer and answers with expert_mats +
mlp3_rows; EXPERT_WORKER and CLUSTER_WORKERS in main, before the SERVE and CLI
paths, the model from SNAP or --model. KDA, the indexer, the attention and the
shared expert stay on the coordinator. A worker serves the int4 container only
and says so.

One thing differs from colibri.c: the coordinator collects every worker's answer
first and adds them into `out` in union order afterwards. Adding worker by worker
gave the same tokens with logits off in the last bit (one worker exact; two
workers, 66 of 128 last logits apart at ~1e-8), because the CPU loop adds a
token's experts in union order and float addition is not associative.

coli: `cluster worker` accepts glm53 models (WORKER_FAMILIES) and hands that
engine the cap alone, as it reads every bare number as the cap and takes its
precision from GLM53_BITS; the GLM-5.2 launch keeps its ebits and dbits.

Gates: tests/test_glm53_cluster_protocol.c (the wire over a socketpair, the shape
of test_cluster_protocol.c, plus the constants and the byte order) and
tests/test_glm53_cluster_sharding.py (the int4 pair of the tiny text oracle,
tools/make_glm53_streaming_pair.py over tools/make_glm53_tiny.py's output, with
and without two workers: the baseline reproduces ref.json's teacher-forced and
greedy tokens at every position, the delegated run prints the same
teacher_forcing, greedy and last_logits bytes and, through every prefix of the
prompt, the same logits at every position, and the coordinator reads no expert
from its own disk). A worker nudging one row of its answer by 1e-3 fails it at
position 0. The multimodal int4 pair is not the fixture: int4 rounding flips one
of its teacher-forced tokens against the f32 reference, the text pair keeps them
all. The GLM-5.3 CI job runs the gate after the oracles.
@Avicennasis

Copy link
Copy Markdown
Contributor Author

Closing this one: superseded by #1774.

#1774 (merged into dev 2026-10-08, merge 8da806d6, with #1977 and #1978 stacked on it) gives glm53.c expert workers on its own wire, COLIG53X. It already has what this PR proposed for glm53 over COLIEX01 v1, and more: every request of a layer goes out before any reply is read (poll), GLM53_CLUSTER_TIMEOUT bounds the connect, every message in flight and each layer's reply, ownership is a deterministic weighted hash of (layer, expert) with shares from a disk-bandwidth handshake that also rejects a worker holding a different model, CLUSTER_WORKER_BIND, and the replies are added back in the local loop's order, so a cluster run prints what a local run prints. A second glm53 worker path on the COLIEX01 wire would be two protocols for one engine, which is not what was taken, so this branch stops here. The p/glm53-cluster head 44d9ca35 stays on the fork for reference.

The remaining colibri.c-side proposals from the same batch are #1964 (v2 hello refusing a v1 worker by name, shared expert on the worker) and #1970 (a v2 request sends the batch rows once); they are being rebased onto dev 1509ea5b. #1962, #1963 and #1967 merge cleanly against that head as they stand. The union-order summation point about colibri.c's cluster_moe_batch noted in the description above is still open and is not part of any of those.

@Avicennasis Avicennasis closed this Oct 9, 2026
Avicennasis added a commit to Avicennasis/colibri-contrib that referenced this pull request Oct 16, 2026
…or any worker count

The local loop adds a row's experts in uniq[] order. cluster_exchange added
each worker's replies into out as they arrived, and cluster_moe_batch calls
it worker by worker, so a row routed to experts on two workers got worker
0's experts first -- a different f32 summation order from two workers on
(one worker's replies are already in uniq[] order, so one was exact). Same
tokens, logits off in the last bits: the gate in tests/test_cluster_sharding
compares logits bytes with one worker, so it never saw it (JustVugg#1968's body
named it for colibri.c and promised the fix).

Now cluster_moe_batch builds the block's items once, in uniq[base+j] order,
sends each worker the ones (eid+layer)%n assigns it, and holds every reply
in its item (ClusterItem gains y; cluster_exchange no longer takes out) until
the last worker has answered; one loop then weights the rows into out, item
by item in that order -- the local loop's. The shared item still lands in
sh_out for FASE E. Allocation stays bounded by the block's 64 items (+ the
shared one); a failed exchange still marks the link failed, leaves no reply
rows behind, and ends the run by name as before.

Measured on the tiny GLM oracle (32 positions, top-2, D=256, 8192 logits
dumped), delegated vs local, every run token-exact against the oracle --
before: 1 worker 0 logits differ; 2 workers 6860 (fmt6) / 6692 (fmt4) differ,
max 7.2e-7; 3 workers 3543 / 5046; 4 workers 6932 / 6736. After: 1, 2 and 3 workers on fmt6 and 2 and 3 on fmt4, 0 of 8192 differ (the
module's 9 tests OK). With top-2 a row's two terms commute unless the add is
contracted into an FMA (GCC's default here, -O3 -march=native): the previous
head built with -ffp-contract=off shows 0 differ with 2 and 3 workers too,
so the protocol test below gives one row three experts, red on any build.

tests/test_cluster_protocol: one row routed to experts 42, 43 and 45 of
layer 7 over two workers ((eid+7)%2 puts 43 and 45 on worker 0, 42 on
worker 1), every expert answering 1.0, gates 1, 2^-24, 2^-24: uniq[] order
gives 1.0 and worker-major 1+2^-23, and the test checks its own numbers
tell the orders apart. On the previous head it aborts on out[d] == expect;
with the fix 3 of 3 runs ok, 0 warnings. The failed-link cases keep their contract with the new
signature (items[0].y stays NULL). tests/test_cluster_sharding gains
ClusterWorkerCountParityTest: the logits bytes of 1, 2 and 3 workers on
fmt6 and 2 and 3 on fmt4 against the local run, counting the floats that
differ. docs/cluster.md says the order.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant