Skip to content

program: the layer split and its cutoff points are command-line arguments - #212

Open
gtrak wants to merge 2 commits into
jesdga95:mainfrom
gtrak:pr-layer-split-cut
Open

gtrak wants to merge 2 commits into
jesdga95:mainfrom
gtrak:pr-layer-split-cut

Conversation

@gtrak

@gtrak gtrak commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

What changed

Two command-line surfaces for the layer split's cut points:

  • --layer-split K1,K2,.. — the first layer of each later stage (8,20,34 for four stages of 8/12/14/14). An
    explicit cut also implies the forced mode.
  • --layer-split-forced — take the split even when the price model picks the expert tier.

Why

The machinery for explicit cut points already exists upstream: the split_at search, the K1,K2.. parse and its
validation are all in src/models/qwen4exp/load/qfn_run.cpp (and the bounds check in load/sessions.cpp). It is
unreachable from any user input — no parser assigns o.layer_split, and the price model only ever writes
"auto". The option's own error messages ("--layer-split K[,K2..]|auto needs --serve, rising K from 2, …")
describe a flag nobody can pass, so the search always cuts the layers evenly.

An even cut leaves the last stage carrying its layers plus the output head while every earlier stage waits on
it. Skewing the cut fixes that, and it is not a small effect.

The forced mode also matters independently: on a box where the price model prefers the expert tier, there is
currently no way to ask for the split at all. That is how a run silently ends up in a different architecture than
intended. On a 4-card box the line reads:

multi-gpu: 4 cards alike; ms per window: expert tier 24.4, layer split 28.3, tensor parallel 22.8; picked expert tier

Results

4x RTX 5060 Ti (sm_120), Qwen3.8-Flash-Next, 180 W — not the 400 W standard, needs a re-run.

Sweeping the cut moved decode from 92.5 t/s to 105–112 t/s. The driver is not layer count but expert-cache
coverage per stage
: a stage's cache is what its dense weights leave over, so more layers means fewer resident
experts and each miss streams from RAM.

  • CUDA2 at 43% coverage: 97–101 t/s
  • CUDA2 at 29% coverage: collapsed to 77–80 t/s
  • CUDA2 at 65% coverage: 106–112 t/s

Wattage says the same thing: an idle stage waiting on its successor draws ~42 W, a saturated one 73 W. The shipped
cut draws 50/51/54/71 W with no card pinned.

Under fp16 the cut's cost follows QSA count rather than layer count (fp16 allocates KV per card for that card's QSA
layers), so 8,20,34 keeps the best decode (107.6–111.6) at the same prefill as 8,22,34.

Rebased on current main

Re-ported onto current main, where the options, the flag parser and the loader moved to src/models/qwen4exp/.
The change is now four small hunks: the split_forced field, the two flags in flags.cpp, the mechanism choice in
load/qfn_run.cpp, and the help text.

The original second commit (a split_cuts helper with its own parse/validation test) is dropped: upstream now
parses the cut inline (ints()), enforces monotonicity (split_at[i] >= 2 && > split_at[i-1]) and bounds
(split_at[i] < n_layers), and requires one device per cut. A second copy of that logic would be a second place
for the rules to drift. Verified on the box:

layer split forced on (the price model picked expert tier)
layer split across 4 GPUs: CUDA0, then CUDA1,2,3 (split 8,20,34)

Tests

None yet — this is a CLI surface over machinery that is already exercised. A unit test over the parse and the
validation (monotone cut points, auto vs explicit) is still owed if the inline rules are to be tested directly.

Quality check

Not run: no Q8_0 reference or .kld base on this machine. The cut changes which card computes which layers, which
moves rounding, so the PPL/KLD gate is the correct one here.

Development note

Engine, kernels and tests were developed with DeepSeek Flash 4.1 and GLM 5.3 Flash, and dogfooded against
Qwen3.8-Flash-Next NVFP4 served by this engine — with a human driving every decision and measurement, no
autonomous agent loops.

…ents

--layer-split K1,K2,.. names the first layer of each later stage, and
--layer-split-forced takes the split over the price model's expert-tier pick.
An explicit --layer-split implies the forced mode.

Re-ported onto the tree that moved the model's options and flag parser to
src/models/qwen4exp/: the machinery (the split_at search, the K1,K2.. parse and
its validation) is upstream and unchanged, it is simply unreachable - no parser
assigns o.layer_split, so the search always cuts the layers evenly, and the price
model's expert-tier pick on a 4-card box leaves no way to ask for the split at
all.
…g could set

configure.cpp has always read Options::split_device, parsed it with the same cut
parser as --layer-split, validated the cards distinct and in 1..n_dev, and used the
order to place stage i on card split_devs[i].  The parser case was dropped with the
other stage options in 0.1.0 and never restored, so the field was reachable only
from a caller: passing the flag hit the unknown-argument path.

Restores the case and documents it.  Verified live: '--layer-split 8,20,34
--split-device 2,3,1' logs 'layer split across 4 GPUs: CUDA0, then CUDA2,3,1'.

Measured on the production recipe (kv int8, prefill 12288, budget 37) it changes
nothing -- default 1,2,3 and 1,2,3 / 2,3,1 / 3,2,1 all read 123/212/109 t/s.  That
is the expected result: the cards share one memory bandwidth and the window is
memory-bound, so a card's clock or thermal edge has no compute-bound share to help.
The flag is a placement control, not a speed one.
@gtrak
gtrak force-pushed the pr-layer-split-cut branch from 51414d1 to 65a52dd Compare November 19, 2026 19:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant