fix(speculative): decline the Qwen 3.5 DFlash burst the probe cannot vet - #1944
Merged
Merged
Conversation
…t wrapper in process Issue #1935's residual at widths 2 and 4 sits in the served burst rather than in the target's forward: every in-process arm of prefill, verify and rewind is exact over the same transcript, including a replay of the served run's own 79 rounds and 56 rewinds. Two instruments narrow what is left. The round loop gains a per-round `debug` transcript carrying the round index, the block width, the bonus, the drafter's proposals, the target's block argmax, the accepted count and the emitted tokens. Every field is already on the host, so an enabled callsite costs no device work. With it a served run's round algebra is checkable offline against a classic transcript, with no GPU and nothing replayed: whether each round's bonus follows from the previous round's emission, whether `accepted` is the longest common prefix, and whether the emitted tokens are what the caches hold. That separates a round-loop or wrapper defect from a target-forward one before any replay is built. The probe tests gain an arm that calls `run_dflash_on_target` itself, with the real drafter loaded through a `WorkerDrafterSlot`. That is the function the server burst calls once it has resolved the model variant, so the prefill, the first-bonus sample, `DFlashGenerator::run` and the drafter are the served ones rather than an approximation. It reproduces `--ignore-eos` the way admission does, as a -inf bias on the merged end-of-generation set, so the arm is not accidentally kinder than the served configuration. A reproduction here is a reproducer without HTTP; exactness here puts the difference in the server process around the call. Refs #1935
…scripts Three served arms on one binary for issue #1935's residual: classic as the null arm, and verify widths 4 and 2, each with two requests against the same server process. Raw completions, per-token logprobs, server logs carrying the per-round transcript, and the classic reference ids. What the data establishes, before any in-process arm runs. The round algebra holds at both widths and in all four bursts: every round's bonus is the previous round's last emitted token, `accepted` is the longest common prefix of the drafter's proposals and the target's block argmax, and the emitted tokens are exactly `draft[:accepted] + [target[accepted]]`. So the emitted stream is the target's own argmax under a self-consistent cache history, and neither the round loop nor the burst wrapper is misbookkeeping anything. Widths 2 and 4 disagree with classic at the same five logprob indices before the divergence, by the same one reporting step each, and swap the same token at 105, which also rules out anything that depends on round structure: width 4 runs 79 rounds with 56 rewinds and width 2 runs 114 with 28. At the divergence the target's block argmax is 5741 where classic's is 11439, and both report logprob -2.125. The logprobs are quantized to 0.125, so that is a near-tie rather than a proven exact one; the next arm has to compare logit bytes rather than reported logprobs. The harness is three scripts. One runs an arm, one checks the round algebra offline with no GPU, one reads the arms against the null arm. The record that interprets all of this lands with the fix. Refs #1935
…2 and 4 residual The host, the method and the part of issue #1935's residual the served arms answer on their own: the round algebra holds in all four speculative bursts, the residual does not track round structure, and the logprobs place the difference just below the reporting floor rather than at a rollback or a round boundary. The sections that name a cause are deliberately absent until the in-process arms run. What is here is what the data supports without them. Refs #1935
…hands the kernel `Qwen3NextAttention::attend_per_position` makes a verify block reproduce single-token decode by attending one query position at a time, each to its own causal prefix. The arithmetic is right and the layout is not: a one-row slice of a `[B, H, T, D]` block has its heads `T * D` apart, while single-token decode hands the same attention call a freshly built `[B, H, 1, D]` whose heads are `D` apart. Same values, same shape, different strides, and on CUDA this family's `head_dim` 256 reaches the fused `sdpa_vector` kernel (issue #675), which is not bit-equal across the two. That is why a served greedy DFlash burst parted from classic decode at every verify width alike. The key and value slices need no copy: both paths slice the same cache buffer to `prefix + i + 1`, so their strides already match. The query row is now copied to the decode layout, under `MLXCEL_QWEN35_ATTEND_CONTIGUOUS` so the pre-fix slicing is one environment variable away for an A/B. The copy is `[B, H, 1, D]` per row per attention layer. Three arms identify the mechanism, each ruling out what the others cannot. At `T = 1` the speculative and classic forwards agree in all 496640 logit bytes at every one of 200 steps, which is where the one-row slice IS the whole tensor and the layouts coincide; that rules out the projections, the gated-delta layers, the prefill and the caches. At `T = 4` the divergence reproduces in process with no drafter and no server, at exactly the served position, which kills the standing hypothesis that the drafter's interleaved work was the cause. And `MLXCEL_SDPA_VECTOR_LARGE_D=0`, which routes both paths off the fused kernel, makes classic and width 4 byte-identical over all 200 tokens. Record and raw data: `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. The served A/B of this change, and the probe's own false pass at widths 2 and 4, land next. Refs #1935
…ck in bytes The query-layout reading of issue #1935's residual is wrong and the change it produced is reverted, including its environment variable. Measured served on GB10 at temperature 0: copying the per-row query slice to the layout single-token decode hands the attention call leaves the width 2 and 4 completions exactly where they were (`3e60b1574c`, first differing token 105), and so does copying the key and value slices with it. The layouts were not what the fused kernel is sensitive to here. Recording the negative result rather than the change. What the same session did establish, and what the next arm has to explain: `MLXCEL_SDPA_VECTOR_LARGE_D=0` collapses the divergence. Served, it makes classic and width 4 byte-identical over all 200 tokens; in process, replaying the served run's own 79 rounds and 56 rewinds, it takes the target from 1 disagreement of 201 to 0. That switch decides only whether `head_dim` 256 reaches the fused `sdpa_vector` kernels, so the difference is in what those kernels do with the two calls rather than in the round loop, the rollback, the gated-delta scan, the quantized matmul or the drafter. Two instruments for that. The exactness probe's prompt length becomes `MLXCEL_MTP_PROBE_PROMPT_LEN`, defaulting to 256 rather than 8: the old length is why the probe reported byte-identity at widths 2 and 4 on a checkpoint whose served burst diverges there, since a short key sequence does not reach the behaviour the probe exists to catch. And a new `#[ignore]` arm drives the served round structure over the recorded transcript comparing logit bytes per kept row rather than argmaxes, then walks every layer's captured hidden state on the first differing row, which is what separates the full attention layers from the gated-delta ones. Refs #1935
…clined run pass The record's cause section said the residual was the per-row tensor layouts. The A/B says otherwise: copying the query slice, and copying the key and value slices with it, leave the served width 2 and 4 completions exactly where they were. Reading CUDA's `supports_sdpa_vector` and `sdpa_vector` confirms it, since both calls satisfy the same admission predicate and the same copy predicates and neither reaches the 2-pass split. The section now records that as a negative result and states what the kill switch does establish: the fused `sdpa_vector` path is the discriminator, in two independent arms, and the difference is in what MLX does around the call rather than in what is handed to it. `assert_server_byte_equality_at_widths` no longer fails when every width declines. It failed on the reasoning that an all-declined run says nothing about verify-path parity, and on this pairing on CUDA that outcome is the contract holding through the gate rather than a hole in the test. What the test must not allow is a decline for some other reason standing in for a measured verdict, so each declined width now has to carry the exactness probe's own decline line; a multimodal payload, an adopted prompt-cache prefix or a drafter from the wrong family still fails the arm. Refs #1935
…ive result The arms that ruled the per-row tensor layouts out, and the harness that produced them, alongside the `MLXCEL_SDPA_VECTOR_LARGE_D` A/B they sit against. The reading they rule out is the obvious one for "same math, one kernel, two answers", so a later reader needs to see it measured rather than argued. Also the formatting fix the byte-bisect arm needed. Refs #1935
…annot see the hazard A probe that cannot observe a hazard reports a pass, which is worse than one that fails, and that is what the exactness gate was doing at DFlash verify widths 2 and 4 on this family (issue #1935). `probe_block_chain_exactness` now returns `NotRun` with a reason, before its draws, when the checkpoint and host are the configuration measured to break byte-identity: CUDA, a `head_dim` of 256 or 288, and `MLXCEL_SDPA_VECTOR_LARGE_D` left enabled. `mtp_exactness_gate` already treats every non-`Equal` verdict as a decline, so both Qwen 3.5 `DFlashTargetModel` impls decline at every width and the request is served by classic decode instead. Why the probe cannot see it. It prefills, runs one block and compares, and the difference does not exist yet there. Measured on `qwen3.5-4b-4bit`, the first row whose logits differ is thirty-six tokens past a 158-token prompt and seventy past a 256-token one, and a byte-level bisect puts the first differing layer at 15, a full attention layer. Repeating that bisect with five accept patterns puts the first difference at the same absolute position in all five, so the trigger is a position in the sequence rather than a number of rounds or rewinds. Sweeping the probe's own prompt length over 8, 32, 64, 128, 256 and 512 leaves its verdict byte-identity at widths 2 and 4 at every length, so lengthening the prefix is not the answer either. The length is now `MLXCEL_MTP_PROBE_PROMPT_LEN`, defaulting to what it always was, so that sweep can be repeated on another checkpoint. This narrows where DFlash engages on this family, deliberately. The alternative is the status quo, in which an operator who turns on speculative decoding gets different greedy text than without it and nothing says so. Both escapes stay: `MLXCEL_MTP_ALLOW_INEXACT=1` engages the burst and logs the forfeit, and `MLXCEL_SDPA_VECTOR_LARGE_D=0` buys the contract back instead of forfeiting it, at the cost of classic decode's own fused attention kernel. Record, with the served arms, the kill-switch A/B, the length sweep and the bisect: `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. Refs #1935
`needless_range_loop` and `collapsible_if` on the new diagnostic, caught by `cargo clippy -p mlxcel --lib --tests -- -D warnings`, which is the gate CI runs and which a `--lib`-only check never reaches because the arm is `cfg(test)`. Refs #1935
…dcoding them The four shell arms carried the absolute scratchpad and worktree paths of the session that produced them, which makes them unrunnable by anyone else and quietly wrong for whoever tries. They now derive the harness directory from their own location, the repo from git, and the output directory and binaries from overridable variables, so a rerun can point at a binary built anywhere. Refs #1935
…ess gate MLX reads `MLXCEL_SDPA_VECTOR_LARGE_D` as an integer defaulting to 1, and the gate now reads it more narrowly than mlxcel's own documented spelling: only a value that parses to zero counts as off. Erring narrow declines a burst that might have been safe. Erring wide would report "safe" for a process in which MLX still takes the fused kernels, which is the one direction a safety gate must not be wrong in. Also records the session's driver budget in the record's host block: the boot's cumulative `NV_ERR_NO_MEMORY` count was 0 before and 0 after, across roughly a dozen server starts and two dozen in-process model loads. Refs #1935
…n it `greedy_parity_dflash_qwen35_4b` passes at widths 2, 4, 8 and 16 on GB10 with every width declined by a measured verdict and every response equal to the drafter-less baseline. It used to fail at all four. The record also names a weakness in that test rather than leaving it to be discovered: its chat prompt at its token budget returns an empty `content` on this checkpoint, baseline included, so its byte comparison of `content` compares two empty strings and the weight is carried by the completion-token equality and the decline assertions. That is not introduced here and not what this issue is about, but the test is weaker evidence than its name suggests. The byte-identity claims in the record rest on the `/v1/completions` arms, which return 823 characters compared by sha256. Refs #1935
inureyes
added
status:review
5 tasks
inureyes
changed the title
…cord Five patterns were run, not four; the list and the per-pattern round numbers after it were already five long. Refs #1935
Names the two files added after it was first written, and records that two arms of the probe-length sweep aborted with this host's known CUDA `cudaStreamEndCapture` artifact rather than producing a result. The lengths that completed all report the same verdict, and 512 completed. Refs #1935
Member
Author
|
CI is green. The first run's |
inureyes
added a commit
that referenced
this pull request
Sep 21, 2026
…ill matters The reading section quoted the classic brackets' full spread (0.40 tok/s) in one sentence and the shift between their means (0.27) in the next without naming either, which invites a reader to add them. Both now say which they are. And the closing paragraph said the width was academic for this pairing because PR #1944's gate declines the burst on CUDA. It is not: `MLXCEL_MTP_ALLOW_INEXACT=1` is a documented escape and an operator who sets it gets exactly this curve, so the default still decides what they run at. The paragraph now says plainly that on this evidence 4 is not the right number for this pairing on this host and 3 is. Refs #1797, #1935
Before choosing what PR #1944 should be, a question had to be settled: which side of the comparison does CUDA's fused `sdpa_vector` kernel actually serve. Both. `supports_sdpa_vector` gates on the query row count of the individual call, and `attend_per_position` slices the verify block to one row per call precisely so each row computes what a decode step computes, so classic decode and every verify row alike arrive with `q.shape(2) == 1`. Measured rather than inferred: `MLXCEL_SDPA_VECTOR_LARGE_D=0` moves both sides, and the burst's only sub-4-row attention calls are those verify rows. That kills the idea of disabling the fused path only for the verify block: it would put the two sides on different kernels, which is the one configuration guaranteed not to be byte-identical, since byte-identity here is exactly the two sides agreeing on a kernel. Seven arms in one session on a verified `origin/main` binary, n = 3, classic brackets at both ends and overlapping, with the 1.29 tok/s union spread reported as the resolution floor and every smaller difference called unresolved. Byte-identical speculation is achievable at 1.13x by running the process on the fallback; without the byte-identity requirement it is 1.22x; declining is 1.00x. The fused kernel is worth about 8% to the burst and nothing measurable to classic decode at this prompt length, which is not a general result because the fallback's cost grows with key length. The fallback costs the burst twice, in per-row attention and in acceptance (0.599 to 0.526 at width 3), and it changes what classic decode produces, so it is not a free correctness switch. Refs #1935
…binary The record's closing claim, that an operator can have byte-identical speculation today by selecting the fallback because the guard stands down when it is off, was an argument about code I wrote rather than a measurement. Three arms on PR #1944's own binary, verified to carry the guard before they ran, settle it: at the shipped default the guard declines, no burst runs and the server returns exactly today's classic text, while with the fallback selected the guard stands down, the burst runs, and its text is byte-identical to the drafter-less server in the same configuration. One binary carries both options, so the choice between declining and byte-identical speculation is an environment variable rather than a build. Refs #1935
Member
Author
|
Hold this from merging until the numbers below are read. The alternative of disabling the fused Which side the fused kernel serves: both. So the flag does not move classic onto the block's kernel. It moves both onto the materializing fallback, and byte-identity is restored because the fallback is consistent across the two call sites while the fused kernel is not. Disabling the fused path only in verify would therefore put the two sides on different kernels, which is the one configuration guaranteed not to be byte-identical. It is not an option, and I did not build an arm for it. The options, one session, n = 3, classic brackets overlapping, 1.29 tok/s resolution floor. On an
The fused kernel is worth 4.39 tok/s to the width-3 burst and nothing measurable to classic decode at this prompt length (0.13, inside the floor), which is not a general result: #675 records that the fallback's cost grows with key length. The fallback also costs the burst acceptance, 0.599 to 0.526 at width 3. Both options are reachable from this PR's binary, measured on it: at the shipped default the guard declines and the server returns exactly today's classic text; with the fallback selected the guard stands down, the burst runs, and its text is byte-identical to the drafter-less server in the same configuration. What the numbers support. A as the default, because it is the only option that leaves both today's output and today's non-speculative performance untouched, and because B's price is paid by every request on the process rather than by the ones that asked for speculation. B belongs in the documentation as the operator recipe for byte-identical speculation, not as a default, and needs no new code. Record: |
inureyes
added a commit
that referenced
this pull request
Sep 21, 2026
#1945) * docs(benchmarks): re-measure the draft block width on main after the #1939 prefill fix Issue #1797 seeded a measured default of 4 for `(12, 1, Affine)` on a DFlash drafter from a sweep taken before PR #1939 fixed the burst's prompt prefill. That fix changes the state every round starts from, which changes acceptance, which changes throughput per width, and nobody had looked since. The same harness, the same pairing, widths 2, 3, 4 and 6 at n = 3 with the classic brackets at both ends, on a binary built from `origin/main` and verified to carry none of the issue #1935 branch's gate. The ordering moved. Width 3 runs at 1.19x classic and its slowest run clears width 4's fastest by 0.80 tok/s, twice the session's full bracket spread, with the drift working against width 3 rather than for it since width 4 ran later. Width 2 and width 4 overlap and this sweep does not order them. Width 6 is below every classic run. Acceptance falls monotonically as the block widens (0.754, 0.599, 0.511, 0.432) against a per-round device-sync cost that rises (19.9, 24.8, 30.3, 47.4 ms), and the product of those peaks at 3 rather than at 4. The record states the drift rather than printing a clean table over it: the two classic brackets do not overlap, by about half a percent, and that is the resolution floor for everything between them. Nothing is differenced against the #1797 table, which ran on a different binary and a different tree. This record does not change the default; it files the evidence so the number can be re-examined against 3. Refs #1797, #1935 * docs(benchmarks): separate the two drift numbers and say the width still matters The reading section quoted the classic brackets' full spread (0.40 tok/s) in one sentence and the shift between their means (0.27) in the next without naming either, which invites a reader to add them. Both now say which they are. And the closing paragraph said the width was academic for this pairing because PR #1944's gate declines the burst on CUDA. It is not: `MLXCEL_MTP_ALLOW_INEXACT=1` is a documented escape and an operator who sets it gets exactly this curve, so the default still decides what they run at. The paragraph now says plainly that on this evidence 4 is not the right number for this pairing on this host and 3 is. Refs #1797, #1935
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue #1935's residual at DFlash verify widths 2 and 4 is genuine numerics in CUDA's fused
sdpa_vectorpath, not a defect in the round loop, the rollback or the drafter, and the exactness gate that was supposed to catch it could not see it. The burst now declines on the configuration that was measured to break byte-identity, and says why.This narrows where DFlash engages on Qwen 3.5. On CUDA with
head_dim256 or 288 the burst is declined at every width and requests are served by classic decode. The alternative is the status quo, in which an operator who turns on speculative decoding gets different greedy text than without it and nothing says so. Both escapes stay:MLXCEL_MTP_ALLOW_INEXACT=1engages the burst and logs the forfeit, andMLXCEL_SDPA_VECTOR_LARGE_D=0buys byte-identity back rather than forfeiting it.What the evidence says
MLXCEL_SDPA_VECTOR_LARGE_D=0decides only whetherhead_dim256 reaches the fusedsdpa_vectorkernels (#675). Served on GB10 at temperature 0 it makes classic and width 4 byte-identical over all 200 tokens; in process it takes the block from 1 disagreement of 201 against the chain to 0. Nothing else that switch touches is a candidate, and everything else that was a candidate stops mattering when it is off.The round loop is exonerated by algebra rather than by argument: a new per-round
debugtranscript makes a served run checkable offline with no GPU, and the algebra holds in all four bursts. The drafter is dead as a hypothesis, which is the one this issue left standing: the divergence reproduces in process with no drafter and no server, at exactly the served position. AtT = 1the speculative and classic forwards agree in all 496640 logit bytes at every one of 200 steps, so it is a property of a multi-row verify block and of nothing else. A byte-level bisect puts the first differing row thirty-six tokens past the prompt and the first differing layer at 15, a full attention layer, and five accept patterns put it at the same absolute position, so the trigger is a position in the sequence rather than a round structure.Two readings are recorded as measured wrong rather than argued away: the per-row tensor layouts (copying the query slice, and the key and value slices with it, changes nothing), and the probe's prompt length (swept over 8 to 512, the verdict is byte-identity at every length).
What changed
Qwen35Model::probe_block_chain_exactnessreturnsNotRunwith a reason on the measured-unsafe configuration, whichmtp_exactness_gatealready treats as a decline.MLXCEL_MTP_PROBE_PROMPT_LEN, default unchanged, so the sweep can be repeated elsewhere.DFlashGenerator::runemits a per-rounddebugtranscript; every field is already on the host, so an enabled callsite costs no device work.assert_server_byte_equality_at_widthsaccepts an all-declined outcome, and now requires each decline to carry the exactness probe's own verdict rather than any other decline reason.#[ignore]-gated diagnostics and a record with its raw data underdocs/benchmark_results/.Test plan
MLXCEL_SDPA_VECTOR_LARGE_Dand query-layout A/Bs.--test-threads=1: one-row byte walk, replay of the served round structure, the burst wrapper with the real drafter, and the byte-level per-layer bisect.cargo fmt,cargo clippy -p mlxcel --lib --tests -- -D warnings.greedy_parity_dflash_qwen35_4bat widths 2, 4, 8 and 16.Closes #1935