docs: read the expert pool's width A-B-A-B, and its floor - #314
Merged
Merged
Conversation
`### How wide the expert pool, at 256K` rested on one reading an arm. Four processes -- pool 148 twice, then 288 twice -- at 32768 tokens, chunk 4096, `DEEPSEEK_V41_EXPERT_DEAL=id`, one 4096-token chunk each, put a floor under it and move where the win is. Same-setting repeats differ by 0.364 s on `quiet`, the uninstrumented chunk, which is 1.3% of its own value; the pooled lever is -2.254 s there, -8.1%, or 6.2x that floor. It also sizes the working rule that a single-run `quiet` or `moe.routed` move under ~10% is not an effect: on this evidence the threshold is conservative by most of an order of magnitude, and two repeats bound a spread from below, so the floor column is a floor. The lever is not where the section said it was. `staged` is 2178 rows in both 148 arms against 2174 in both 288 arms, `routed.upload` moves 3.843 -> 3.832 s -- the same bytes -- and `_issue_chunk` is called 40 times, once a layer, in every arm, so neither the row-hit accounting nor `_chunk_bounds` cutting the forward into fewer pieces is what is being bought. The routed sub-phases net +0.11 s against the wider pool, the wrong sign; the ~1.0 s that does appear is in the `moe` block's own body. The memory arithmetic still sizes the pool, and its own allocation column reproduces it (16118 MiB at 148 rows against 18631 at 288, +2513 against the 2512 the row size predicts); what the phases do not support is the explanation of the speedup. The token column separates by setting and by nothing else: both 148 processes print one top-8 and both 288 processes print another, sharing the first four ids in order and permuting the 5th-8th. `topk` on the logits, so it is the logits that moved, and the same-setting pairs are exact repeats. Measured on a later tree than the table above, so this is a second reading of the lever and not a re-measurement of it. No code change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pool-width block already recorded that the A-B-A-B refutes its own stated mechanism. Two more things say so, and neither needed a run. `_issue_chunk` is called once per bound `_chunk_bounds` returns, so the 40 calls the A-B-A-B counts over 40 layers mean the whole 4096-row batch is one bound at either width. One is the floor, so the "fewer and bigger chunks" the section proposes cannot happen at 32768 rather than merely not having happened -- and the phase table above, read on an earlier tree, counted 158 calls for the same row, about four bounds a layer. The 0.2% staged figure is the more interesting one, because the pool is what it should move. `pool_lru` is one LRU arena a card shared by all forty layers, so 148 pool rows is about 3.7 rows a layer against the ~54 a chunk stages in both arms. Both widths sit inside the region where a sweep of the layer's working set thrashes the cache, where capacity buys almost nothing until it spans the whole set: 140 more rows bought 4 of 2178 misses. The +2513 MiB is real and `_chunk_bounds`' rule is real, but the pool is not the lever they are attached to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
### How wide the expert pool, at 256Krested on one reading an arm. Four processes -- pool 148 twice, then 288 twice -- at--at 32768 --chunk 4096 --max-seq-len 41024withDEEPSEEK_V41_EXPERT_DEAL=id, one 4096-token chunk each, put a floor under it and move where the win is.quiet-- the same width at 36864, taps offmoemoe.routedrouted.resolverouted.stagerouted.uploadattnThe floor
Two processes of the same setting differ by 0.364 s on
quiet, 1.3% of its own value, so the -8.1% is 6.2x the floor and the lever stands. It also sizes the working rule that a single-runquietormoe.routedmove under ~10% is not an effect: on this evidence the threshold is conservative by most of an order of magnitude, sincemoe.routed's own floor is 2.2% andquiet's is 1.3%. Two repeats bound a spread from below, so the floor column is a floor.The lever is not where the section said
The section explained the win as a wider pool making more draws hit, so
_chunk_boundscuts the forward into fewer and bigger chunks. The phases do not show that:stagedis 2178 rows in both 148 arms against 2174 in both 288 arms -- 0.2%; a wider pool did not make more draws hit.routed.uploadmoves 3.843 -> 3.832 s, the same bytes._issue_chunkis called 40 times, once a layer, in every arm, so the forward was not cut into fewer pieces either.The ~1.0 s that does appear sits in the
moeblock's own body, the part its three children do not cover, so the mechanism is a per-layer issue/wait effect at the MoE boundary. The row-size arithmetic still sizes the pool and this run's allocation column reproduces it -- 16118 MiB at 148 rows against 18631 at 288, +2513 MiB against the 2512 the row size predicts -- but the phases do not support the explanation of the speedup, which is now stated as open.The token column
All four ranks print one top-8 a process (
torch.topk(logits[0].float(), 8), no sampling). Both 148 processes print[455, 1, 223, 8077, 764, 330, 343, 334]and both 288 processes print[455, 1, 223, 8077, 330, 334, 764, 343]: the same eight ids, the first four in the same order, the 5th-8th permuted. So the same-setting pairs are exact repeats and the setting is what moved the tail.Scope
Measured on a later tree than the table above, so this is a second reading of the same lever and not a re-measurement of it. Documentation only; no code change.
mkdocs build --strictclean.🤖 Generated with Claude Code