Repository navigation
Conversation
gtrak
marked this pull request as ready for review
November 2, 2026 17:50
gtrak
force-pushed
the
pr-layer-split-cut
branch
from
November 10, 2026 16:07
0914a4e to
09bf2f2
Compare
gtrak
force-pushed
the
pr-layer-split-cut
branch
from
November 13, 2026 16:01
09bf2f2 to
51414d1
Compare
…ents --layer-split K1,K2,.. names the first layer of each later stage, and --layer-split-forced takes the split over the price model's expert-tier pick. An explicit --layer-split implies the forced mode. Re-ported onto the tree that moved the model's options and flag parser to src/models/qwen4exp/: the machinery (the split_at search, the K1,K2.. parse and its validation) is upstream and unchanged, it is simply unreachable - no parser assigns o.layer_split, so the search always cuts the layers evenly, and the price model's expert-tier pick on a 4-card box leaves no way to ask for the split at all.
…g could set configure.cpp has always read Options::split_device, parsed it with the same cut parser as --layer-split, validated the cards distinct and in 1..n_dev, and used the order to place stage i on card split_devs[i]. The parser case was dropped with the other stage options in 0.1.0 and never restored, so the field was reachable only from a caller: passing the flag hit the unknown-argument path. Restores the case and documents it. Verified live: '--layer-split 8,20,34 --split-device 2,3,1' logs 'layer split across 4 GPUs: CUDA0, then CUDA2,3,1'. Measured on the production recipe (kv int8, prefill 12288, budget 37) it changes nothing -- default 1,2,3 and 1,2,3 / 2,3,1 / 3,2,1 all read 123/212/109 t/s. That is the expected result: the cards share one memory bandwidth and the window is memory-bound, so a card's clock or thermal edge has no compute-bound share to help. The flag is a placement control, not a speed one.
gtrak
force-pushed
the
pr-layer-split-cut
branch
from
November 19, 2026 19:09
51414d1 to
65a52dd
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Two command-line surfaces for the layer split's cut points:
--layer-split K1,K2,..— the first layer of each later stage (8,20,34for four stages of 8/12/14/14). Anexplicit cut also implies the forced mode.
--layer-split-forced— take the split even when the price model picks the expert tier.Why
The machinery for explicit cut points already exists upstream: the
split_atsearch, theK1,K2..parse and itsvalidation are all in
src/models/qwen4exp/load/qfn_run.cpp(and the bounds check inload/sessions.cpp). It isunreachable from any user input — no parser assigns
o.layer_split, and the price model only ever writes"auto". The option's own error messages ("--layer-split K[,K2..]|autoneeds--serve, rising K from 2, …")describe a flag nobody can pass, so the search always cuts the layers evenly.
An even cut leaves the last stage carrying its layers plus the output head while every earlier stage waits on
it. Skewing the cut fixes that, and it is not a small effect.
The forced mode also matters independently: on a box where the price model prefers the expert tier, there is
currently no way to ask for the split at all. That is how a run silently ends up in a different architecture than
intended. On a 4-card box the line reads:
Results
4x RTX 5060 Ti (sm_120), Qwen3.8-Flash-Next, 180 W — not the 400 W standard, needs a re-run.
Sweeping the cut moved decode from 92.5 t/s to 105–112 t/s. The driver is not layer count but expert-cache
coverage per stage: a stage's cache is what its dense weights leave over, so more layers means fewer resident
experts and each miss streams from RAM.
Wattage says the same thing: an idle stage waiting on its successor draws ~42 W, a saturated one 73 W. The shipped
cut draws 50/51/54/71 W with no card pinned.
Under fp16 the cut's cost follows QSA count rather than layer count (fp16 allocates KV per card for that card's QSA
layers), so
8,20,34keeps the best decode (107.6–111.6) at the same prefill as8,22,34.Rebased on current main
Re-ported onto current
main, where the options, the flag parser and the loader moved tosrc/models/qwen4exp/.The change is now four small hunks: the
split_forcedfield, the two flags inflags.cpp, the mechanism choice inload/qfn_run.cpp, and the help text.The original second commit (a
split_cutshelper with its own parse/validation test) is dropped: upstream nowparses the cut inline (
ints()), enforces monotonicity (split_at[i] >= 2 && > split_at[i-1]) and bounds(
split_at[i] < n_layers), and requires one device per cut. A second copy of that logic would be a second placefor the rules to drift. Verified on the box:
Tests
None yet — this is a CLI surface over machinery that is already exercised. A unit test over the parse and the
validation (monotone cut points,
autovs explicit) is still owed if the inline rules are to be tested directly.Quality check
Not run: no Q8_0 reference or
.kldbase on this machine. The cut changes which card computes which layers, whichmoves rounding, so the PPL/KLD gate is the correct one here.
Development note
Engine, kernels and tests were developed with DeepSeek Flash 4.1 and GLM 5.3 Flash, and dogfooded against
Qwen3.8-Flash-Next NVFP4 served by this engine — with a human driving every decision and measurement, no
autonomous agent loops.