Repository navigation
cluster: the coordinator's time on the wire, per turn on PROF and STAT - #1962
Conversation
A clustered run has no way to see what the network costs it: PROF carries disk service, wait, matmul, attention and head, and the wire's share lands in "other". distributed-llama prints per token G (the whole step), I (local inference), T (transfer) and the bytes each way; this gives colibri the same figures on the lines a dashboard already reads. g_cluster_ns/tx/rx accumulate on the coordinator only, around the socket I/O cluster_moe_batch does (headers included; cluster_item() and the weighted accumulate stay outside the timed calls), through two timed wrappers over cluster_io/cluster_u32. A worker's own I/O is not counted. ProfBase snapshots them, so every window is the request's, like edisk. With CLUSTER_WORKERS set: - PROF continues " <cluster_s> <tx_bytes> <rx_bytes>" after n_fw. Unlike edisk this is a wall-time phase: the compute thread does nothing else while it waits. - STAT continues " cluster_g_ms= cluster_i_ms= cluster_t_ms= cluster_tx_kb= cluster_rx_kb=", per emitted token over the line's own window, on the mux DONE line and on the single-slot line (which already carries key=value). - the PROF=1 report prints a "[PROF] cluster:" line next to expert I/O. Without a cluster the counters stay 0 and every line is byte-identical. openai_server.py keeps the extra PROF fields on the profile turn (cluster_transfer_s, cluster_tx_bytes, cluster_rx_bytes) and the STAT key=values on the stats dict; a bare positional STAT tail is still ignored. test_cluster_protocol drives cluster_moe_batch over a socketpair and asserts the counters move by exactly the bytes that crossed it (60 out, 52 in) and not at all for the raw round trip; test_openai_server covers both parsers.
|
CI on this PR has no verdict yet: of its 114 checks, 85 are CANCELLED and 29 SUCCESS (read from the check rollup on 2026-10-09); no job failed. The branch still merges cleanly against For context: #1968 is closed as superseded by #1774 ( |
A clustered run has no way to see what the network costs it: PROF carries disk service, wait, matmul, attention and head, and the wire's share lands in "other". This adds the coordinator's time blocked in socket I/O inside
cluster_moe_batch, and the bytes each way, to the lines a dashboard already reads. The shape follows what distributed-llama prints per token (G whole step, I local inference, T transfer, S/R bytes), which is the comparison people make when they cluster Macs.What changes, only when
CLUSTER_WORKERSis set (without a cluster every line is byte-identical, the counters stay 0):PROFcontinues<cluster_s> <tx_bytes> <rx_bytes>aftern_fw(window totals, like edisk, but a wall-time phase: the compute thread does nothing else while it waits).STATcontinuescluster_g_ms= cluster_i_ms= cluster_t_ms= cluster_tx_kb= cluster_rx_kb=per emitted token over the line's own window, on the mux DONE line and the single-slot line that already carries key=value.PROF=1stderr report gets a[PROF] cluster:line next to expert I/O.openai_server.pykeeps the three PROF fields on the profile turn and the five STAT keys on the stats dict; a bare positional STAT tail is still ignored.Mechanically:
g_cluster_ns/tx/rxaccumulate around the 15 coordinator-sidecluster_io/cluster_u32calls through two timed wrappers; headers included,cluster_item()and the weighted accumulate outside; a worker's own I/O is not counted.ProfBasesnapshots them so each window is the request's.g_cluster_nmoves out of the!_WIN32guard (always 0 there) so the emission sites need no ifdefs.Tests:
test_cluster_protocolnow drives the realcluster_moe_batchover a socketpair and asserts the counters move by exactly the bytes that crossed it (60 out, 52 in),ns > 0after and 0 before, and the outputs are the worker'sw*2*x. Twotest_openai_servercases cover the PROF tail and the STAT suffix parsing, with the existing positional fields and the bare 8th-field case intact.make colibri tests/test_cluster_protocolwarning-free;pytest tests/test_openai_server.py: 427 passed.For scale: on two M1 Max MacBooks over wired 1 GbE today, distributed-llama's own line showed ~28 ms/token on the wire for 712 kB/token with Qwen3-30B-A3B, half of each step. That is the number this makes visible for colibri's expert workers.