Repository navigation
glm53: expert workers over the COLIEX01 wire, as colibri.c has them - #1968
Avicennasis wants to merge 1 commit into
Conversation
GLM-5.2's local cluster mode (docs/cluster.md: CLUSTER_WORKERS, EXPERT_WORKER=1, one request per layer per worker carrying the routed batch-union, the experts sharded by (eid + layer) % workers) lived in colibri.c only. glm53.c now speaks the same wire, byte for byte -- the COLIEX01 magic, the v1 header, the per-item layout -- so the protocol stays one thing; a glm53 coordinator reaches glm53 workers, since the expert math (the int4 gs64 pieces, the clamped SwiGLU) is the family's. Off unless CLUSTER_WORKERS is set: without it the engine's stdout is byte-identical to the previous build's on the tiny f32 and int4 fixtures (teacher_forcing, last_logits, greedy, the expert counters). In glm53.c: cluster_io, cluster_u32, cluster_connect_one and cluster_init as in colibri.c; cluster_item and cluster_moe_batch over ffn_layer_ex's chosen, weight and union_ids (every token has topk entries here, there is no keff); the hook once the union is built, by blocks of 64, in place of the cache-sized block loop (the Vulkan tier, taken before it, wins over it as in colibri.c, and so does the resident f32 path, which never streams); cluster_worker_run, which loads the config, opens the shards, indexes the experts (expert_table_init checks every piece's presence and length up front, so a request can never name a tensor the worker cannot resolve), keeps one Slot per layer and answers with expert_mats + mlp3_rows; EXPERT_WORKER and CLUSTER_WORKERS in main, before the SERVE and CLI paths, the model from SNAP or --model. KDA, the indexer, the attention and the shared expert stay on the coordinator. A worker serves the int4 container only and says so. One thing differs from colibri.c: the coordinator collects every worker's answer first and adds them into `out` in union order afterwards. Adding worker by worker gave the same tokens with logits off in the last bit (one worker exact; two workers, 66 of 128 last logits apart at ~1e-8), because the CPU loop adds a token's experts in union order and float addition is not associative. coli: `cluster worker` accepts glm53 models (WORKER_FAMILIES) and hands that engine the cap alone, as it reads every bare number as the cap and takes its precision from GLM53_BITS; the GLM-5.2 launch keeps its ebits and dbits. Gates: tests/test_glm53_cluster_protocol.c (the wire over a socketpair, the shape of test_cluster_protocol.c, plus the constants and the byte order) and tests/test_glm53_cluster_sharding.py (the int4 pair of the tiny text oracle, tools/make_glm53_streaming_pair.py over tools/make_glm53_tiny.py's output, with and without two workers: the baseline reproduces ref.json's teacher-forced and greedy tokens at every position, the delegated run prints the same teacher_forcing, greedy and last_logits bytes and, through every prefix of the prompt, the same logits at every position, and the coordinator reads no expert from its own disk). A worker nudging one row of its answer by 1e-3 fails it at position 0. The multimodal int4 pair is not the fixture: int4 rounding flips one of its teacher-forced tokens against the f32 reference, the text pair keeps them all. The GLM-5.3 CI job runs the gate after the oracles.
|
Closing this one: superseded by #1774. #1774 (merged into The remaining |
…or any worker count The local loop adds a row's experts in uniq[] order. cluster_exchange added each worker's replies into out as they arrived, and cluster_moe_batch calls it worker by worker, so a row routed to experts on two workers got worker 0's experts first -- a different f32 summation order from two workers on (one worker's replies are already in uniq[] order, so one was exact). Same tokens, logits off in the last bits: the gate in tests/test_cluster_sharding compares logits bytes with one worker, so it never saw it (JustVugg#1968's body named it for colibri.c and promised the fix). Now cluster_moe_batch builds the block's items once, in uniq[base+j] order, sends each worker the ones (eid+layer)%n assigns it, and holds every reply in its item (ClusterItem gains y; cluster_exchange no longer takes out) until the last worker has answered; one loop then weights the rows into out, item by item in that order -- the local loop's. The shared item still lands in sh_out for FASE E. Allocation stays bounded by the block's 64 items (+ the shared one); a failed exchange still marks the link failed, leaves no reply rows behind, and ends the run by name as before. Measured on the tiny GLM oracle (32 positions, top-2, D=256, 8192 logits dumped), delegated vs local, every run token-exact against the oracle -- before: 1 worker 0 logits differ; 2 workers 6860 (fmt6) / 6692 (fmt4) differ, max 7.2e-7; 3 workers 3543 / 5046; 4 workers 6932 / 6736. After: 1, 2 and 3 workers on fmt6 and 2 and 3 on fmt4, 0 of 8192 differ (the module's 9 tests OK). With top-2 a row's two terms commute unless the add is contracted into an FMA (GCC's default here, -O3 -march=native): the previous head built with -ffp-contract=off shows 0 differ with 2 and 3 workers too, so the protocol test below gives one row three experts, red on any build. tests/test_cluster_protocol: one row routed to experts 42, 43 and 45 of layer 7 over two workers ((eid+7)%2 puts 43 and 45 on worker 0, 42 on worker 1), every expert answering 1.0, gates 1, 2^-24, 2^-24: uniq[] order gives 1.0 and worker-major 1+2^-23, and the test checks its own numbers tell the orders apart. On the previous head it aborts on out[d] == expect; with the fix 3 of 3 runs ok, 0 warnings. The failed-link cases keep their contract with the new signature (items[0].y stays NULL). tests/test_cluster_sharding gains ClusterWorkerCountParityTest: the logits bytes of 1, 2 and 3 workers on fmt6 and 2 and 3 on fmt4 against the local run, counting the floats that differ. docs/cluster.md says the order.
The expert-worker cluster mode exists in one engine,
colibri.c. This givesglm53.cthe same thing, same wire (COLIEX01 v1, byte-identical header and item layout), env-gated and off by default; the single-machine path is byte-identical (built the pristineglm53.cbeside it andcmp'd stdout on the tiny and the streaming-int4 fixtures: identical, counters included). Delegation applies only whenm->streaming(the int4 U8 container); resident-f32 experts never reach the hook, and the worker refuses such a checkpoint by name. KDA/MLA/indexer/mHC untouched; the shared expert stays on the coordinator.One deliberate difference from
colibri.c, coordinator-only: adding each worker's replies intooutworker-by-worker changes the per-token float summation order against the local union-order loop. Measured here: 1 worker bit-identical; 2 workers same tokens but 66/128last_logitsdiffer at ~1e-8.glm53.c'scluster_moe_batchcollects every worker'syfirst and sums in union order, so it is bit-exact for any worker count. The same order dependence exists incolibri.c(its gate is token-only, so it never saw it); I left that file alone because #1962/#1963/#1964 are open on its cluster block and will send the ~15-line fix after.Worker:
load_cfg+st_init+glm53_mirror_setup+ the prefix probe (mirrorsmodel_load_range), thenexpert_geometry+expert_table_init, which validates every piece up front, so an absent expert fails at start, not on first use. It also rejectslayer < first_dense(dense layers have zeroederefentries here).coli cluster workerlearns the glm53 argv (caponly; precision isGLM53_BITS);WORKER_FAMILIES=("glm","glm53").Gate (
tests/test_glm53_cluster_sharding.py,GLM53_TINY_STREAM):glm53.chas no[ORACLE] mismatchmode, so the gate compares againstref.json: baseline equals the f32 oracle's teacher-forcing and greedy ids per position; delegated equals baseline byte-for-byte on teacher_forcing, greedy andlast_logits, and via every prompt prefix 1..8 onlast_logitsat every position; delegation acted (two workers,experts hits 0 miss 0 bytes 0on the coordinator); an unreachable worker makes the engine refuse, never fall back silently. The fixture is the text-only streaming pair frommake_glm53_streaming_pair.py: the multimodal int4 CI fixture flips one teacher-forced token against its f32 ref on the pristine engine (int4 rounding on a 128-wide random model), so it cannot carry a zero-mismatch gate. Negative control: a 1e-3 nudge on the worker's output passed the first, last-position-only gate; the per-prefix per-position logit parity catches it (1 failed), reverted, 2 passed.make glm53 colibri tests/test_glm53_cluster_protocol0 warnings; protocol test ok; glm53 gate 2 passed; the GLM-5.2 gate still 2 passed;test_makefile_deps8 passed;test_cluster.py+test_glm53_oracles.py9 passed. check.yml's glm53-oracle job generates the text int4 pair and runs the gate.Note: a sibling PR for
deepseek_v41.ctouches the same threeneed_worker_modellines inc/coli; whichever lands second is a trivial rebase.