Skip to content

cluster: the coordinator's time on the wire, per turn on PROF and STAT - #1962

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
Avicennasis:p/cluster-transfer-timing
Oct 16, 2026
Merged

JustVugg merged 1 commit into
JustVugg:devfrom
Avicennasis:p/cluster-transfer-timing

Conversation

@Avicennasis

Copy link
Copy Markdown
Contributor

A clustered run has no way to see what the network costs it: PROF carries disk service, wait, matmul, attention and head, and the wire's share lands in "other". This adds the coordinator's time blocked in socket I/O inside cluster_moe_batch, and the bytes each way, to the lines a dashboard already reads. The shape follows what distributed-llama prints per token (G whole step, I local inference, T transfer, S/R bytes), which is the comparison people make when they cluster Macs.

What changes, only when CLUSTER_WORKERS is set (without a cluster every line is byte-identical, the counters stay 0):

  • PROF continues <cluster_s> <tx_bytes> <rx_bytes> after n_fw (window totals, like edisk, but a wall-time phase: the compute thread does nothing else while it waits).
  • STAT continues cluster_g_ms= cluster_i_ms= cluster_t_ms= cluster_tx_kb= cluster_rx_kb= per emitted token over the line's own window, on the mux DONE line and the single-slot line that already carries key=value.
  • PROF=1 stderr report gets a [PROF] cluster: line next to expert I/O.
  • openai_server.py keeps the three PROF fields on the profile turn and the five STAT keys on the stats dict; a bare positional STAT tail is still ignored.

Mechanically: g_cluster_ns/tx/rx accumulate around the 15 coordinator-side cluster_io/cluster_u32 calls through two timed wrappers; headers included, cluster_item() and the weighted accumulate outside; a worker's own I/O is not counted. ProfBase snapshots them so each window is the request's. g_cluster_n moves out of the !_WIN32 guard (always 0 there) so the emission sites need no ifdefs.

Tests: test_cluster_protocol now drives the real cluster_moe_batch over a socketpair and asserts the counters move by exactly the bytes that crossed it (60 out, 52 in), ns > 0 after and 0 before, and the outputs are the worker's w*2*x. Two test_openai_server cases cover the PROF tail and the STAT suffix parsing, with the existing positional fields and the bare 8th-field case intact. make colibri tests/test_cluster_protocol warning-free; pytest tests/test_openai_server.py: 427 passed.

For scale: on two M1 Max MacBooks over wired 1 GbE today, distributed-llama's own line showed ~28 ms/token on the wire for 712 kB/token with Qwen3-30B-A3B, half of each step. That is the number this makes visible for colibri's expert workers.

A clustered run has no way to see what the network costs it: PROF carries
disk service, wait, matmul, attention and head, and the wire's share lands
in "other". distributed-llama prints per token G (the whole step), I (local
inference), T (transfer) and the bytes each way; this gives colibri the same
figures on the lines a dashboard already reads.

g_cluster_ns/tx/rx accumulate on the coordinator only, around the socket I/O
cluster_moe_batch does (headers included; cluster_item() and the weighted
accumulate stay outside the timed calls), through two timed wrappers over
cluster_io/cluster_u32. A worker's own I/O is not counted. ProfBase snapshots
them, so every window is the request's, like edisk.

With CLUSTER_WORKERS set:
- PROF continues " <cluster_s> <tx_bytes> <rx_bytes>" after n_fw. Unlike
  edisk this is a wall-time phase: the compute thread does nothing else
  while it waits.
- STAT continues " cluster_g_ms= cluster_i_ms= cluster_t_ms= cluster_tx_kb=
  cluster_rx_kb=", per emitted token over the line's own window, on the mux
  DONE line and on the single-slot line (which already carries key=value).
- the PROF=1 report prints a "[PROF] cluster:" line next to expert I/O.
Without a cluster the counters stay 0 and every line is byte-identical.

openai_server.py keeps the extra PROF fields on the profile turn
(cluster_transfer_s, cluster_tx_bytes, cluster_rx_bytes) and the STAT
key=values on the stats dict; a bare positional STAT tail is still ignored.

test_cluster_protocol drives cluster_moe_batch over a socketpair and asserts
the counters move by exactly the bytes that crossed it (60 out, 52 in) and
not at all for the raw round trip; test_openai_server covers both parsers.
@Avicennasis

Copy link
Copy Markdown
Contributor Author

CI on this PR has no verdict yet: of its 114 checks, 85 are CANCELLED and 29 SUCCESS (read from the check rollup on 2026-10-09); no job failed. The branch still merges cleanly against dev 1509ea5b (git merge-tree --write-tree against the fetched head produces a tree with no conflicts), and what merged into dev since the base in the files this PR touches (+542/-36 over c/colibri.c, c/openai_server.py and c/tests/test_openai_server.py) is in other regions than this change, so the reviewed head is left as it is rather than rebased for the sake of a re-run. Could you re-run the checks?

For context: #1968 is closed as superseded by #1774 (glm53.c has its own COLIG53X workers now), and #1964, #1969 and #1970 are being rebased onto 1509ea5b because they conflict.

@JustVugg
JustVugg merged commit 18c225b into JustVugg:dev Oct 16, 2026
29 of 114 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants