From 5cc25ccaddfbdea5f221782bb14c514e0e6faeee Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:00:19 +0900 Subject: [PATCH 01/16] test(speculative): log the DFlash round transcript and drive the burst wrapper in process Issue #1935's residual at widths 2 and 4 sits in the served burst rather than in the target's forward: every in-process arm of prefill, verify and rewind is exact over the same transcript, including a replay of the served run's own 79 rounds and 56 rewinds. Two instruments narrow what is left. The round loop gains a per-round `debug` transcript carrying the round index, the block width, the bonus, the drafter's proposals, the target's block argmax, the accepted count and the emitted tokens. Every field is already on the host, so an enabled callsite costs no device work. With it a served run's round algebra is checkable offline against a classic transcript, with no GPU and nothing replayed: whether each round's bonus follows from the previous round's emission, whether `accepted` is the longest common prefix, and whether the emitted tokens are what the caches hold. That separates a round-loop or wrapper defect from a target-forward one before any replay is built. The probe tests gain an arm that calls `run_dflash_on_target` itself, with the real drafter loaded through a `WorkerDrafterSlot`. That is the function the server burst calls once it has resolved the model variant, so the prefill, the first-bonus sample, `DFlashGenerator::run` and the drafter are the served ones rather than an approximation. It reproduces `--ignore-eos` the way admission does, as a -inf bias on the merged end-of-generation set, so the arm is not accidentally kinder than the served configuration. A reproduction here is a reproducer without HTTP; exactness here puts the difference in the server process around the call. Refs #1935 --- .../src/drafter/dflash/round_loop.rs | 20 +++ src/models/qwen3_5_dflash_probe_tests.rs | 128 ++++++++++++++++++ 2 files changed, 148 insertions(+) diff --git a/src/lib/mlxcel-core/src/drafter/dflash/round_loop.rs b/src/lib/mlxcel-core/src/drafter/dflash/round_loop.rs index 22befedc1..468e430bd 100644 --- a/src/lib/mlxcel-core/src/drafter/dflash/round_loop.rs +++ b/src/lib/mlxcel-core/src/drafter/dflash/round_loop.rs @@ -835,6 +835,26 @@ impl DFlashGenerator { let (accepted, new_tokens) = speculative_walk(&draft_tokens, &target_tokens, budget); diagnostics.walk_time_ms += phase_start.elapsed().as_secs_f64() * 1000.0; self.accept_lens.push(accepted as u32); + // Per-round transcript, at `debug` (issue #1935). Every field is + // already on the host, so an enabled callsite costs no device work + // and a disabled one costs the level check: `bonus` followed by + // `draft_tokens` IS the verify input, and `target_tokens` is the + // block argmax the walk just consumed. With this a served run's + // round algebra (whether the next round's bonus follows from this + // round's emission, whether the kept prefix is what the caches + // hold) is checkable offline against a classic transcript, with no + // GPU and nothing replayed. That is what separates a round-loop or + // wrapper defect from a target-forward one. + tracing::debug!( + round = diagnostics.rounds, + bs, + bonus, + ?draft_tokens, + ?target_tokens, + accepted, + ?new_tokens, + "DFlash round transcript" + ); diagnostics.rounds += 1; diagnostics.proposed_tokens += draft_tokens.len(); diagnostics.accepted_tokens += accepted; diff --git a/src/models/qwen3_5_dflash_probe_tests.rs b/src/models/qwen3_5_dflash_probe_tests.rs index 1ced612b9..caafb19d1 100644 --- a/src/models/qwen3_5_dflash_probe_tests.rs +++ b/src/models/qwen3_5_dflash_probe_tests.rs @@ -725,3 +725,131 @@ fn round_loop_cache_dynamics_with_rollback_match_the_chain() { ); } } + +/// The drafter checkpoint the served arms pair this target with; override +/// with `MLXCEL_Q35_PROBE_DRAFTER`. +const DEFAULT_DRAFTER: &str = "models/mlx/qwen3.5-4b-dflash"; + +fn drafter_dir() -> String { + std::env::var("MLXCEL_Q35_PROBE_DRAFTER").unwrap_or_else(|_| DEFAULT_DRAFTER.to_string()) +} + +/// **Diagnostic 7.** Drive the served burst wrapper itself, with the real +/// drafter bound, and compare its emitted ids against the classic transcript. +/// +/// Every earlier arm drives the target alone: prefill, a verify block, a +/// rewind, or a replay of the served run's own accept sequence. All of them +/// are exact, and the served burst is not, so what separates them is the one +/// thing they omit: the drafter, executing its own MLX work between the +/// target's forwards, and the wrapper that carries the first bonus, the +/// budget and the emission. +/// +/// This arm omits only the server process. It calls +/// [`crate::server::batch::dflash_target::run_dflash_on_target`], which is the +/// function `run_dflash_burst` calls once it has resolved the model variant, +/// so the prefill, the first-bonus sample, `DFlashGenerator::run` and the +/// drafter slot are the served ones rather than an approximation of them. +/// +/// A reproduction here is the reproducer the record asks for, under a +/// debugger and without HTTP. Exactness here instead says the difference is +/// the server process around this call, and the next arm has to look there. +#[test] +#[ignore = "needs the real Qwen 3.5 4B checkpoint, the DFlash drafter, a GPU and a recorded transcript"] +fn served_burst_wrapper_with_real_drafter_matches_the_chain() { + use mlxcel_core::generate::{LanguageModel, SamplingConfig}; + use mlxcel_core::sampling::LogprobsConfig; + use std::sync::atomic::AtomicBool; + + let Some((model, dir)) = load_text_model() else { + return; + }; + let draft_dir = drafter_dir(); + if !std::path::Path::new(&draft_dir).exists() { + eprintln!("[1935] skipping: drafter {draft_dir} not on disk"); + return; + } + let Some(prompt) = env_ids("MLXCEL_Q35_PROBE_PROMPT") else { + eprintln!("[1935] skipping: set MLXCEL_Q35_PROBE_PROMPT and MLXCEL_Q35_PROBE_REFERENCE"); + return; + }; + let reference = env_ids("MLXCEL_Q35_PROBE_REFERENCE").expect("MLXCEL_Q35_PROBE_REFERENCE"); + let text = text_model(&model); + + for block_size in widths() { + // The served arm's own configuration: `--ignore-eos` is a -inf bias on + // every end-of-generation id (`admission.rs`), and the burst still + // receives the merged EOS set, so both are reproduced here rather than + // approximated by an empty stop set. + let eos = mlxcel_core::generation_policy::merged_eos_token_ids( + model.eos_token_ids(), + &Vec::new(), + ); + let mut sampling = SamplingConfig { + temperature: 0.0, + top_k: 1, + ..SamplingConfig::default() + }; + sampling.token_bias.suppress_tokens(&eos); + + let dispatch = crate::server::SpeculativeDispatch::DFlash { + draft_model_path: std::path::PathBuf::from(&draft_dir), + block_size: block_size as u32, + block_size_source: crate::cli::draft_block_policy::BlockSizeSource::Override, + user_requested_explicit_kind: true, + }; + let mut slot = + crate::server::batch::speculative_burst::WorkerDrafterSlot::from_dispatch(&dispatch); + slot.ensure_loaded().expect("drafter must load"); + let owned = slot.take().expect("drafter present after ensure_loaded"); + + let cancel = AtomicBool::new(false); + let logprobs = LogprobsConfig::default(); + let max_tokens = reference.len(); + let run = crate::server::batch::dflash_target::run_dflash_on_target( + text, + &prompt, + &sampling, + &[], + &eos, + owned, + block_size as u32, + max_tokens, + &mut slot, + &cancel, + &logprobs, + ); + let run = match run { + Ok(r) => r, + Err(_) => { + eprintln!( + "[1935] block {block_size}: the burst declined or errored rather than \ + running; nothing to compare" + ); + continue; + } + }; + let n = run.tokens.len().min(reference.len()); + let first = (0..n) + .find(|&i| run.tokens[i] != reference[i]) + .map(|i| i as i64) + .unwrap_or(-1); + let disagreements = (0..n).filter(|&i| run.tokens[i] != reference[i]).count(); + eprintln!( + "[1935] {dir} + {draft_dir}, block {block_size}: burst emitted {} ids against \ + {} reference; {disagreements} disagree over {n} compared, first at {first} \ + (-1 means none)", + run.tokens.len(), + reference.len(), + ); + if first >= 0 { + let i = first as usize; + let lo = i.saturating_sub(3); + let hi = (i + 4).min(n); + eprintln!( + "[1935] around {i}: reference {:?} against burst {:?}", + &reference[lo..hi], + &run.tokens[lo..hi], + ); + } + } +} From 6cfd29656bf3b800b4ccc74fd39bf6a10bb36a05 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:19:40 +0900 Subject: [PATCH 02/16] test(speculative): record the served DFlash arms and their round transcripts Three served arms on one binary for issue #1935's residual: classic as the null arm, and verify widths 4 and 2, each with two requests against the same server process. Raw completions, per-token logprobs, server logs carrying the per-round transcript, and the classic reference ids. What the data establishes, before any in-process arm runs. The round algebra holds at both widths and in all four bursts: every round's bonus is the previous round's last emitted token, `accepted` is the longest common prefix of the drafter's proposals and the target's block argmax, and the emitted tokens are exactly `draft[:accepted] + [target[accepted]]`. So the emitted stream is the target's own argmax under a self-consistent cache history, and neither the round loop nor the burst wrapper is misbookkeeping anything. Widths 2 and 4 disagree with classic at the same five logprob indices before the divergence, by the same one reporting step each, and swap the same token at 105, which also rules out anything that depends on round structure: width 4 runs 79 rounds with 56 rewinds and width 2 runs 114 with 28. At the divergence the target's block argmax is 5741 where classic's is 11439, and both report logprob -2.125. The logprobs are quantized to 0.125, so that is a near-tie rather than a proven exact one; the next arm has to compare logit bytes rather than reported logprobs. The harness is three scripts. One runs an arm, one checks the round algebra offline with no GPU, one reads the arms against the null arm. The record that interprets all of this lands with the fix. Refs #1935 --- .../README.md | 21 + .../arms/arm.w2.json | 857 ++++++++++++++++++ .../arms/arm.w4.json | 857 ++++++++++++++++++ .../arms/arm.wclassic.json | 851 +++++++++++++++++ .../arms/server.w2.log | 276 ++++++ .../arms/server.w4.log | 206 +++++ .../arms/server.wclassic.log | 36 + .../arms/w2_ids.json | 1 + .../arms/w4_ids.json | 1 + .../classic_ids.json | 1 + .../harness/algebra.py | 153 ++++ .../harness/compare.py | 106 +++ .../harness/transcript.py | 199 ++++ .../identity.txt | 20 + .../prompt_ids.json | 1 + 15 files changed, 3586 insertions(+) create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w2.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.wclassic.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w2.log create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w4.log create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.wclassic.log create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w2_ids.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w4_ids.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/classic_ids.json create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/algebra.py create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/compare.py create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/transcript.py create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/identity.txt create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/prompt_ids.json diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md new file mode 100644 index 000000000..032f078c7 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md @@ -0,0 +1,21 @@ +# Issue #1935 residual at widths 2 and 4: served arms and round transcripts (GB10, 2026-09-21) + +Raw data for the residual PR #1939 left open: on `models/mlx/qwen3.5-4b-4bit` with `models/mlx/qwen3.5-4b-dflash`, greedy served output at verify widths 2 and 4 parts from classic decode at generated token 105. + +`identity.txt` is the host, binary and checkpoint identity for every arm here. One binary, one session, three arms, two requests each. + +## What is in here + +`arms/arm..json` is one served arm: the request bodies' completion text, the per-token logprob tokens and values, and the usage block, for both requests against the same server process. `arms/server..log` is that arm's server log at `RUST_LOG=info,mlxcel_core::drafter::dflash::round_loop=debug`, which carries the per-round `DFlash round transcript` line the round loop emits. + +`prompt_ids.json` and `classic_ids.json` are the 158 prompt ids and the 200 classic generated ids, from `MLXCEL_PRINT_TOKEN_IDS=1 mlxcel generate -m models/mlx/qwen3.5-4b-4bit --no-chat-template --temp 0 --max-tokens 200`. They are what the in-process probe arms take as `MLXCEL_Q35_PROBE_PROMPT` and `MLXCEL_Q35_PROBE_REFERENCE`. Note the prompt file ends in a newline and shell command substitution strips it: passing the prompt without that newline tokenizes to 157 ids, not 158, and produces a different first token, so the CLI reference then matches nothing. + +`arms/w4_ids.json` and `arms/w2_ids.json` are the emitted id streams reconstructed from the round transcripts by `harness/algebra.py`. + +## Harness + +`harness/transcript.py` runs one arm: one server, `--ignore-eos --max-batch-size 1`, n non-streaming `/v1/completions` at temperature 0 with logprobs, against the same process so a cross-request difference would show. Identity only, so its host gate is the foreign-model check alone. + +`harness/algebra.py` checks a run's round algebra offline against the transcript lines, with no GPU: that each round's bonus is the previous round's last emitted token, that `accepted` is the longest common prefix of the drafter's proposals and the target's block argmax, and that the emitted tokens are `draft[:accepted] + [target[accepted]]`. Rounds are logged 0-based, which is where a burst boundary is. + +`harness/compare.py` reads the arms against the classic null arm: the first differing token index, and whether the logprobs agree before it. diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w2.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w2.json new file mode 100644 index 000000000..0c58b90ee --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w2.json @@ -0,0 +1,857 @@ +{ + "tag": "w2", + "width": "2", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18935", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "2" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.w2.log", + "extra_env": "", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + { + "index": 1, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4.json new file mode 100644 index 000000000..8bdbb7bbf --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4.json @@ -0,0 +1,857 @@ +{ + "tag": "w4", + "width": "4", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18935", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "4" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.w4.log", + "extra_env": "", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + { + "index": 1, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.wclassic.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.wclassic.json new file mode 100644 index 000000000..adb471341 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.wclassic.json @@ -0,0 +1,851 @@ +{ + "tag": "wclassic", + "width": "classic", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18935", + "--ignore-eos", + "--max-batch-size", + "1" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.wclassic.log", + "extra_env": "", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.5, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -0.875, + 0.0, + -0.375, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.375, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.625, + -0.75, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.25, + -2.0, + -0.5, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + -0.25, + -1.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.375, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + -0.25, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + -0.75, + -1.875, + -1.25, + -0.375, + -0.5, + 0.0, + -0.75, + -0.375, + -0.375, + -0.625, + -0.125, + -0.5, + -0.125, + 0.0, + 0.0, + -0.5, + -1.0, + -0.25, + -0.625, + -0.625, + -0.5, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + }, + { + "index": 1, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.5, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -0.875, + 0.0, + -0.375, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.375, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.625, + -0.75, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.25, + -2.0, + -0.5, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + -0.25, + -1.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.375, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + -0.25, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + -0.75, + -1.875, + -1.25, + -0.375, + -0.5, + 0.0, + -0.75, + -0.375, + -0.375, + -0.625, + -0.125, + -0.5, + -0.125, + 0.0, + 0.0, + -0.5, + -1.0, + -0.25, + -0.625, + -0.625, + -0.5, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w2.log b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w2.log new file mode 100644 index 000000000..cddf7b30a --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w2.log @@ -0,0 +1,276 @@ +2026-09-21T05:11:23.000424Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T05:11:23.000468Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T05:11:23.000598Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T05:11:23.000672Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T05:11:23.000674Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T05:11:23.000676Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T05:11:23.336121Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T05:11:23.336155Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T05:11:23.336723Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T05:11:23.679701Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T05:11:23.680399Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T05:11:24.038388Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.358s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.357948284 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T05:11:24.038468Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=2, block_size_from=operator override, explicit_kind=true)) +2026-09-21T05:11:24.038876Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T05:11:24.038890Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T05:11:24.038912Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T05:11:25.537808Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=2 +2026-09-21T05:11:25.537844Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T05:11:25.538237Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T05:11:25.539213Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T05:11:25.562732Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=2 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.920665 first_bonus_ms=23.482434 first_hidden_ms=0.01 bind_reset_ms=0.040048 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T05:11:25.562788Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[] +2026-09-21T05:11:25.562916Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1523 +2026-09-21T05:11:25.562922Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1523.795185 burst_active_ms=1523.795185 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T05:11:25.562978Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T05:11:25.563071Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T05:11:25.563942Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T05:11:25.564692Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18935 +2026-09-21T05:11:25.564698Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T05:11:25.564701Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T05:11:25.564703Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T05:11:25.564704Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T05:11:25.564706Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T05:11:25.564707Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T05:11:25.564708Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T05:11:25.564709Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T05:11:25.564711Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T05:11:25.564712Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T05:11:25.564713Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T05:11:25.564714Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T05:11:27.684468Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=0 bs=2 bonus=285 draft_tokens=[364] target_tokens=[4071, 7383] accepted=0 new_tokens=[4071] +2026-09-21T05:11:27.745020Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=1 bs=2 bonus=4071 draft_tokens=[34237] target_tokens=[30097, 30056] accepted=0 new_tokens=[30097] +2026-09-21T05:11:27.796371Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=2 bs=2 bonus=30097 draft_tokens=[264] target_tokens=[264, 8240] accepted=1 new_tokens=[264, 8240] +2026-09-21T05:11:27.846021Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=3 bs=2 bonus=8240 draft_tokens=[314] target_tokens=[314, 30056] accepted=1 new_tokens=[314, 30056] +2026-09-21T05:11:27.895068Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=4 bs=2 bonus=30056 draft_tokens=[71483] target_tokens=[364, 198] accepted=0 new_tokens=[364] +2026-09-21T05:11:27.943596Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=5 bs=2 bonus=364 draft_tokens=[1754] target_tokens=[21979, 4621] accepted=0 new_tokens=[21979] +2026-09-21T05:11:27.991723Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=6 bs=2 bonus=21979 draft_tokens=[286] target_tokens=[286, 71483] accepted=1 new_tokens=[286, 71483] +2026-09-21T05:11:28.039967Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=7 bs=2 bonus=71483 draft_tokens=[285] target_tokens=[198, 653] accepted=0 new_tokens=[198] +2026-09-21T05:11:28.088152Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=8 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 30056] accepted=1 new_tokens=[285, 30056] +2026-09-21T05:11:28.136041Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=9 bs=2 bonus=30056 draft_tokens=[283] target_tokens=[283, 2958] accepted=1 new_tokens=[283, 2958] +2026-09-21T05:11:28.183774Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=10 bs=2 bonus=2958 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:28.230565Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=11 bs=2 bonus=285 draft_tokens=[364] target_tokens=[364, 585] accepted=1 new_tokens=[364, 585] +2026-09-21T05:11:28.277760Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=12 bs=2 bonus=585 draft_tokens=[303] target_tokens=[303, 2020] accepted=1 new_tokens=[303, 2020] +2026-09-21T05:11:28.324152Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=13 bs=2 bonus=2020 draft_tokens=[7] target_tokens=[1153, 16] accepted=0 new_tokens=[1153] +2026-09-21T05:11:28.372909Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=14 bs=2 bonus=1153 draft_tokens=[6469] target_tokens=[6469, 76753] accepted=1 new_tokens=[6469, 76753] +2026-09-21T05:11:28.421959Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=15 bs=2 bonus=76753 draft_tokens=[198] target_tokens=[1590, 5015] accepted=0 new_tokens=[1590] +2026-09-21T05:11:28.470496Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=16 bs=2 bonus=1590 draft_tokens=[198] target_tokens=[198, 309] accepted=1 new_tokens=[198, 309] +2026-09-21T05:11:28.517379Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=17 bs=2 bonus=309 draft_tokens=[7383] target_tokens=[7383, 283] accepted=1 new_tokens=[7383, 283] +2026-09-21T05:11:28.564229Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=18 bs=2 bonus=283 draft_tokens=[638] target_tokens=[638, 8658] accepted=1 new_tokens=[638, 8658] +2026-09-21T05:11:28.612334Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=19 bs=2 bonus=8658 draft_tokens=[1889] target_tokens=[21510, 8] accepted=0 new_tokens=[21510] +2026-09-21T05:11:28.661666Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=20 bs=2 bonus=21510 draft_tokens=[2972] target_tokens=[348, 318] accepted=0 new_tokens=[348] +2026-09-21T05:11:28.710249Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=21 bs=2 bonus=348 draft_tokens=[318] target_tokens=[318, 17] accepted=1 new_tokens=[318, 17] +2026-09-21T05:11:28.757240Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=22 bs=2 bonus=17 draft_tokens=[2972] target_tokens=[2972, 585] accepted=1 new_tokens=[2972, 585] +2026-09-21T05:11:28.803746Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=23 bs=2 bonus=585 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:28.850845Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=24 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 7383] accepted=1 new_tokens=[309, 7383] +2026-09-21T05:11:28.897785Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=25 bs=2 bonus=7383 draft_tokens=[283] target_tokens=[283, 1262] accepted=1 new_tokens=[283, 1262] +2026-09-21T05:11:28.945185Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=26 bs=2 bonus=1262 draft_tokens=[58308] target_tokens=[58308, 11] accepted=1 new_tokens=[58308, 11] +2026-09-21T05:11:28.992439Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=27 bs=2 bonus=11 draft_tokens=[638] target_tokens=[638, 6469] accepted=1 new_tokens=[638, 6469] +2026-09-21T05:11:29.039546Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=28 bs=2 bonus=6469 draft_tokens=[76753] target_tokens=[21510, 8] accepted=0 new_tokens=[21510] +2026-09-21T05:11:29.088280Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=29 bs=2 bonus=21510 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:29.137349Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=30 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 413] accepted=1 new_tokens=[309, 413] +2026-09-21T05:11:29.184885Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=31 bs=2 bonus=413 draft_tokens=[585] target_tokens=[638, 835] accepted=0 new_tokens=[638] +2026-09-21T05:11:29.231380Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=32 bs=2 bonus=638 draft_tokens=[39281] target_tokens=[1130, 5206] accepted=0 new_tokens=[1130] +2026-09-21T05:11:29.280977Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=33 bs=2 bonus=1130 draft_tokens=[81480] target_tokens=[3143, 25] accepted=0 new_tokens=[3143] +2026-09-21T05:11:29.330843Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=34 bs=2 bonus=3143 draft_tokens=[25] target_tokens=[25, 198] accepted=1 new_tokens=[25, 198] +2026-09-21T05:11:29.378234Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=35 bs=2 bonus=198 draft_tokens=[388] target_tokens=[388, 7383] accepted=1 new_tokens=[388, 7383] +2026-09-21T05:11:29.425316Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=36 bs=2 bonus=7383 draft_tokens=[283] target_tokens=[283, 7383] accepted=1 new_tokens=[283, 7383] +2026-09-21T05:11:29.473301Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=37 bs=2 bonus=7383 draft_tokens=[318] target_tokens=[478, 16] accepted=0 new_tokens=[478] +2026-09-21T05:11:29.524082Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=38 bs=2 bonus=478 draft_tokens=[318] target_tokens=[4055, 11134] accepted=0 new_tokens=[4055] +2026-09-21T05:11:29.572477Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=39 bs=2 bonus=4055 draft_tokens=[34593] target_tokens=[34593, 7] accepted=1 new_tokens=[34593, 7] +2026-09-21T05:11:29.620239Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=40 bs=2 bonus=7 draft_tokens=[15] target_tokens=[15, 11] accepted=1 new_tokens=[15, 11] +2026-09-21T05:11:29.667039Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=41 bs=2 bonus=11 draft_tokens=[220] target_tokens=[7383, 15] accepted=0 new_tokens=[7383] +2026-09-21T05:11:29.715857Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=42 bs=2 bonus=7383 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:29.763948Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=43 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 30056] accepted=1 new_tokens=[309, 30056] +2026-09-21T05:11:29.812759Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=44 bs=2 bonus=30056 draft_tokens=[58308] target_tokens=[1989, 8] accepted=0 new_tokens=[1989] +2026-09-21T05:11:29.860830Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=45 bs=2 bonus=1989 draft_tokens=[58308] target_tokens=[58308, 8] accepted=1 new_tokens=[58308, 8] +2026-09-21T05:11:29.908155Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=46 bs=2 bonus=8 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:29.956079Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=47 bs=2 bonus=285 draft_tokens=[460] target_tokens=[460, 30056] accepted=1 new_tokens=[460, 30056] +2026-09-21T05:11:30.004540Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=48 bs=2 bonus=30056 draft_tokens=[271] target_tokens=[1358, 262] accepted=0 new_tokens=[1358] +2026-09-21T05:11:30.053917Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=49 bs=2 bonus=1358 draft_tokens=[727] target_tokens=[727, 21979] accepted=1 new_tokens=[727, 21979] +2026-09-21T05:11:30.102287Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=50 bs=2 bonus=21979 draft_tokens=[6406] target_tokens=[6406, 3764] accepted=1 new_tokens=[6406, 3764] +2026-09-21T05:11:30.149752Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=51 bs=2 bonus=3764 draft_tokens=[3764] target_tokens=[1783, 1783] accepted=0 new_tokens=[1783] +2026-09-21T05:11:30.198019Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=52 bs=2 bonus=1783 draft_tokens=[17994] target_tokens=[7, 11] accepted=0 new_tokens=[7] +2026-09-21T05:11:30.246707Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=53 bs=2 bonus=7 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:30.293665Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=54 bs=2 bonus=262 draft_tokens=[2821] target_tokens=[2821, 11] accepted=1 new_tokens=[2821, 11] +2026-09-21T05:11:30.340285Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=55 bs=2 bonus=11 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:30.387795Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=56 bs=2 bonus=262 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:30.435652Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=57 bs=2 bonus=1783 draft_tokens=[28] target_tokens=[22064, 3581] accepted=0 new_tokens=[22064] +2026-09-21T05:11:30.483629Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=58 bs=2 bonus=22064 draft_tokens=[28] target_tokens=[25, 3581] accepted=0 new_tokens=[25] +2026-09-21T05:11:30.534749Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=59 bs=2 bonus=25 draft_tokens=[6627] target_tokens=[6627, 1783] accepted=1 new_tokens=[6627, 1783] +2026-09-21T05:11:30.584793Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=60 bs=2 bonus=1783 draft_tokens=[22064] target_tokens=[13429, 283] accepted=0 new_tokens=[13429] +2026-09-21T05:11:30.634076Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=61 bs=2 bonus=13429 draft_tokens=[283] target_tokens=[283, 2168] accepted=1 new_tokens=[283, 2168] +2026-09-21T05:11:30.681651Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=62 bs=2 bonus=2168 draft_tokens=[11] target_tokens=[11, 198] accepted=1 new_tokens=[11, 198] +2026-09-21T05:11:30.727440Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=63 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 5741] accepted=1 new_tokens=[262, 5741] +2026-09-21T05:11:30.774074Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=64 bs=2 bonus=5741 draft_tokens=[25] target_tokens=[25, 8132] accepted=1 new_tokens=[25, 8132] +2026-09-21T05:11:30.820716Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=65 bs=2 bonus=8132 draft_tokens=[283] target_tokens=[12385, 2168] accepted=0 new_tokens=[12385] +2026-09-21T05:11:30.870894Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=66 bs=2 bonus=12385 draft_tokens=[283] target_tokens=[283, 2168] accepted=1 new_tokens=[283, 2168] +2026-09-21T05:11:30.918196Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=67 bs=2 bonus=2168 draft_tokens=[11] target_tokens=[11, 198] accepted=1 new_tokens=[11, 198] +2026-09-21T05:11:30.963728Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=68 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 2972] accepted=1 new_tokens=[262, 2972] +2026-09-21T05:11:31.009882Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=69 bs=2 bonus=2972 draft_tokens=[1371] target_tokens=[9386, 1783] accepted=0 new_tokens=[9386] +2026-09-21T05:11:31.058246Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=70 bs=2 bonus=9386 draft_tokens=[262] target_tokens=[11, 653] accepted=0 new_tokens=[11] +2026-09-21T05:11:31.113412Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=71 bs=2 bonus=11 draft_tokens=[198] target_tokens=[198, 1590] accepted=1 new_tokens=[198, 1590] +2026-09-21T05:11:31.163537Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=72 bs=2 bonus=1590 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:31.213238Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=73 bs=2 bonus=262 draft_tokens=[4071] target_tokens=[4071, 49823] accepted=1 new_tokens=[4071, 49823] +2026-09-21T05:11:31.262895Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=74 bs=2 bonus=49823 draft_tokens=[264] target_tokens=[264, 709] accepted=1 new_tokens=[264, 709] +2026-09-21T05:11:31.314348Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=75 bs=2 bonus=709 draft_tokens=[440] target_tokens=[440, 56768] accepted=1 new_tokens=[440, 56768] +2026-09-21T05:11:31.366422Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=76 bs=2 bonus=56768 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:31.416714Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=77 bs=2 bonus=1783 draft_tokens=[321] target_tokens=[321, 81480] accepted=1 new_tokens=[321, 81480] +2026-09-21T05:11:31.467677Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=78 bs=2 bonus=81480 draft_tokens=[262] target_tokens=[13, 4071] accepted=0 new_tokens=[13] +2026-09-21T05:11:31.520293Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=79 bs=2 bonus=13 draft_tokens=[271] target_tokens=[271, 262] accepted=1 new_tokens=[271, 262] +2026-09-21T05:11:31.571856Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=80 bs=2 bonus=262 draft_tokens=[17167] target_tokens=[17167, 25] accepted=1 new_tokens=[17167, 25] +2026-09-21T05:11:31.622354Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=81 bs=2 bonus=25 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:31.673432Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=82 bs=2 bonus=285 draft_tokens=[2821] target_tokens=[2821, 25] accepted=1 new_tokens=[2821, 25] +2026-09-21T05:11:31.723845Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=83 bs=2 bonus=25 draft_tokens=[561] target_tokens=[561, 709] accepted=1 new_tokens=[561, 709] +2026-09-21T05:11:31.774975Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=84 bs=2 bonus=709 draft_tokens=[310] target_tokens=[310, 21979] accepted=1 new_tokens=[310, 21979] +2026-09-21T05:11:31.827554Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=85 bs=2 bonus=21979 draft_tokens=[13] target_tokens=[13, 198] accepted=1 new_tokens=[13, 198] +2026-09-21T05:11:31.878855Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=86 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 1142] accepted=1 new_tokens=[285, 1142] +2026-09-21T05:11:31.929341Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=87 bs=2 bonus=1142 draft_tokens=[1783] target_tokens=[1783, 22064] accepted=1 new_tokens=[1783, 22064] +2026-09-21T05:11:31.977395Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=88 bs=2 bonus=22064 draft_tokens=[25] target_tokens=[25, 561] accepted=1 new_tokens=[25, 561] +2026-09-21T05:11:32.024737Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=89 bs=2 bonus=561 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:32.071494Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=90 bs=2 bonus=1783 draft_tokens=[4687] target_tokens=[4687, 310] accepted=1 new_tokens=[4687, 310] +2026-09-21T05:11:32.117104Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=91 bs=2 bonus=310 draft_tokens=[958] target_tokens=[958, 13] accepted=1 new_tokens=[958, 13] +2026-09-21T05:11:32.162240Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=92 bs=2 bonus=13 draft_tokens=[34797] target_tokens=[198, 310] accepted=0 new_tokens=[198] +2026-09-21T05:11:32.211865Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=93 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 5741] accepted=1 new_tokens=[285, 5741] +2026-09-21T05:11:32.259610Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=94 bs=2 bonus=5741 draft_tokens=[25] target_tokens=[25, 561] accepted=1 new_tokens=[25, 561] +2026-09-21T05:11:32.307126Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=95 bs=2 bonus=561 draft_tokens=[5741] target_tokens=[5741, 310] accepted=1 new_tokens=[5741, 310] +2026-09-21T05:11:32.353397Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=96 bs=2 bonus=310 draft_tokens=[958] target_tokens=[958, 364] accepted=1 new_tokens=[958, 364] +2026-09-21T05:11:32.402816Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=97 bs=2 bonus=364 draft_tokens=[8132] target_tokens=[8132, 13] accepted=1 new_tokens=[8132, 13] +2026-09-21T05:11:32.449411Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=98 bs=2 bonus=13 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:32.498078Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=99 bs=2 bonus=285 draft_tokens=[2972] target_tokens=[2972, 9386] accepted=1 new_tokens=[2972, 9386] +2026-09-21T05:11:32.545324Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=100 bs=2 bonus=9386 draft_tokens=[25] target_tokens=[25, 22484] accepted=1 new_tokens=[25, 22484] +2026-09-21T05:11:32.592443Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=101 bs=2 bonus=22484 draft_tokens=[5789] target_tokens=[5789, 310] accepted=1 new_tokens=[5789, 310] +2026-09-21T05:11:32.639684Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=102 bs=2 bonus=310 draft_tokens=[1440] target_tokens=[1440, 310] accepted=1 new_tokens=[1440, 310] +2026-09-21T05:11:32.688110Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=103 bs=2 bonus=310 draft_tokens=[279] target_tokens=[279, 709] accepted=1 new_tokens=[279, 709] +2026-09-21T05:11:32.735299Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=104 bs=2 bonus=709 draft_tokens=[13] target_tokens=[13, 271] accepted=1 new_tokens=[13, 271] +2026-09-21T05:11:32.782400Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=105 bs=2 bonus=271 draft_tokens=[262] target_tokens=[262, 5019] accepted=1 new_tokens=[262, 5019] +2026-09-21T05:11:32.829813Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=106 bs=2 bonus=5019 draft_tokens=[25] target_tokens=[25, 198] accepted=1 new_tokens=[25, 198] +2026-09-21T05:11:32.877960Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=107 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 561] accepted=1 new_tokens=[285, 561] +2026-09-21T05:11:32.925559Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=108 bs=2 bonus=561 draft_tokens=[1067] target_tokens=[1067, 314] accepted=1 new_tokens=[1067, 314] +2026-09-21T05:11:32.972739Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=109 bs=2 bonus=314 draft_tokens=[279] target_tokens=[279, 709] accepted=1 new_tokens=[279, 709] +2026-09-21T05:11:33.022427Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=110 bs=2 bonus=709 draft_tokens=[13] target_tokens=[13, 198] accepted=1 new_tokens=[13, 198] +2026-09-21T05:11:33.069232Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=111 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 4071] accepted=1 new_tokens=[262, 4071] +2026-09-21T05:11:33.116175Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=112 bs=2 bonus=4071 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:33.163171Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=113 bs=2 bonus=262 draft_tokens=[413] target_tokens=[413, 1142] accepted=1 new_tokens=[413] +2026-09-21T05:11:33.163413Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=2 rounds=114 proposed_tokens=114 accepted_tokens=86 acceptance_rate=0.7543859649122807 emitted_per_verify=1.7456140350877194 zero_accept_rounds=28 partial_accept_rounds=0 full_accept_rounds=86 prefill_verify_ms=4.85605 first_bonus_ms=596.211809 first_hidden_ms=0.004432 bind_reset_ms=0.022305000000000002 draft_ms=2901.7027970000004 verify_ms=289.97228299999995 target_argmax_sync_ms=2370.0125969999995 logprobs_ms=249.84626799999995 walk_ms=0.027280000000000006 hidden_concat_ms=0.23255 rollback_ms=13.104178 decode_ms=5844.054462 +2026-09-21T05:11:33.163473Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] +2026-09-21T05:11:33.163806Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=6445 +2026-09-21T05:11:33.163854Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=6445.789145 burst_active_ms=6445.789145 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T05:11:33.318948Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=0 bs=2 bonus=285 draft_tokens=[364] target_tokens=[4071, 7383] accepted=0 new_tokens=[4071] +2026-09-21T05:11:33.346517Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=1 bs=2 bonus=4071 draft_tokens=[34237] target_tokens=[30097, 30056] accepted=0 new_tokens=[30097] +2026-09-21T05:11:33.374909Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=2 bs=2 bonus=30097 draft_tokens=[264] target_tokens=[264, 8240] accepted=1 new_tokens=[264, 8240] +2026-09-21T05:11:33.402208Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=3 bs=2 bonus=8240 draft_tokens=[314] target_tokens=[314, 30056] accepted=1 new_tokens=[314, 30056] +2026-09-21T05:11:33.428737Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=4 bs=2 bonus=30056 draft_tokens=[71483] target_tokens=[364, 198] accepted=0 new_tokens=[364] +2026-09-21T05:11:33.457240Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=5 bs=2 bonus=364 draft_tokens=[1754] target_tokens=[21979, 4621] accepted=0 new_tokens=[21979] +2026-09-21T05:11:33.484583Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=6 bs=2 bonus=21979 draft_tokens=[286] target_tokens=[286, 71483] accepted=1 new_tokens=[286, 71483] +2026-09-21T05:11:33.513080Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=7 bs=2 bonus=71483 draft_tokens=[285] target_tokens=[198, 653] accepted=0 new_tokens=[198] +2026-09-21T05:11:33.542385Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=8 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 30056] accepted=1 new_tokens=[285, 30056] +2026-09-21T05:11:33.569523Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=9 bs=2 bonus=30056 draft_tokens=[283] target_tokens=[283, 2958] accepted=1 new_tokens=[283, 2958] +2026-09-21T05:11:33.595321Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=10 bs=2 bonus=2958 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:33.622575Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=11 bs=2 bonus=285 draft_tokens=[364] target_tokens=[364, 585] accepted=1 new_tokens=[364, 585] +2026-09-21T05:11:33.649204Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=12 bs=2 bonus=585 draft_tokens=[303] target_tokens=[303, 2020] accepted=1 new_tokens=[303, 2020] +2026-09-21T05:11:33.676506Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=13 bs=2 bonus=2020 draft_tokens=[7] target_tokens=[1153, 16] accepted=0 new_tokens=[1153] +2026-09-21T05:11:33.702620Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=14 bs=2 bonus=1153 draft_tokens=[6469] target_tokens=[6469, 76753] accepted=1 new_tokens=[6469, 76753] +2026-09-21T05:11:33.729077Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=15 bs=2 bonus=76753 draft_tokens=[198] target_tokens=[1590, 5015] accepted=0 new_tokens=[1590] +2026-09-21T05:11:33.757578Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=16 bs=2 bonus=1590 draft_tokens=[198] target_tokens=[198, 309] accepted=1 new_tokens=[198, 309] +2026-09-21T05:11:33.783737Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=17 bs=2 bonus=309 draft_tokens=[7383] target_tokens=[7383, 283] accepted=1 new_tokens=[7383, 283] +2026-09-21T05:11:33.810719Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=18 bs=2 bonus=283 draft_tokens=[638] target_tokens=[638, 8658] accepted=1 new_tokens=[638, 8658] +2026-09-21T05:11:33.837588Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=19 bs=2 bonus=8658 draft_tokens=[1889] target_tokens=[21510, 8] accepted=0 new_tokens=[21510] +2026-09-21T05:11:33.866736Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=20 bs=2 bonus=21510 draft_tokens=[2972] target_tokens=[348, 318] accepted=0 new_tokens=[348] +2026-09-21T05:11:33.897292Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=21 bs=2 bonus=348 draft_tokens=[318] target_tokens=[318, 17] accepted=1 new_tokens=[318, 17] +2026-09-21T05:11:33.923801Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=22 bs=2 bonus=17 draft_tokens=[2972] target_tokens=[2972, 585] accepted=1 new_tokens=[2972, 585] +2026-09-21T05:11:33.951142Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=23 bs=2 bonus=585 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:33.978418Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=24 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 7383] accepted=1 new_tokens=[309, 7383] +2026-09-21T05:11:34.004847Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=25 bs=2 bonus=7383 draft_tokens=[283] target_tokens=[283, 1262] accepted=1 new_tokens=[283, 1262] +2026-09-21T05:11:34.032542Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=26 bs=2 bonus=1262 draft_tokens=[58308] target_tokens=[58308, 11] accepted=1 new_tokens=[58308, 11] +2026-09-21T05:11:34.059430Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=27 bs=2 bonus=11 draft_tokens=[638] target_tokens=[638, 6469] accepted=1 new_tokens=[638, 6469] +2026-09-21T05:11:34.086893Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=28 bs=2 bonus=6469 draft_tokens=[76753] target_tokens=[21510, 8] accepted=0 new_tokens=[21510] +2026-09-21T05:11:34.115708Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=29 bs=2 bonus=21510 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:34.143453Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=30 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 413] accepted=1 new_tokens=[309, 413] +2026-09-21T05:11:34.169526Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=31 bs=2 bonus=413 draft_tokens=[585] target_tokens=[638, 835] accepted=0 new_tokens=[638] +2026-09-21T05:11:34.198590Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=32 bs=2 bonus=638 draft_tokens=[39281] target_tokens=[1130, 5206] accepted=0 new_tokens=[1130] +2026-09-21T05:11:34.227923Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=33 bs=2 bonus=1130 draft_tokens=[81480] target_tokens=[3143, 25] accepted=0 new_tokens=[3143] +2026-09-21T05:11:34.256337Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=34 bs=2 bonus=3143 draft_tokens=[25] target_tokens=[25, 198] accepted=1 new_tokens=[25, 198] +2026-09-21T05:11:34.281757Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=35 bs=2 bonus=198 draft_tokens=[388] target_tokens=[388, 7383] accepted=1 new_tokens=[388, 7383] +2026-09-21T05:11:34.309025Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=36 bs=2 bonus=7383 draft_tokens=[283] target_tokens=[283, 7383] accepted=1 new_tokens=[283, 7383] +2026-09-21T05:11:34.335190Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=37 bs=2 bonus=7383 draft_tokens=[318] target_tokens=[478, 16] accepted=0 new_tokens=[478] +2026-09-21T05:11:34.363742Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=38 bs=2 bonus=478 draft_tokens=[318] target_tokens=[4055, 11134] accepted=0 new_tokens=[4055] +2026-09-21T05:11:34.392843Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=39 bs=2 bonus=4055 draft_tokens=[34593] target_tokens=[34593, 7] accepted=1 new_tokens=[34593, 7] +2026-09-21T05:11:34.419593Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=40 bs=2 bonus=7 draft_tokens=[15] target_tokens=[15, 11] accepted=1 new_tokens=[15, 11] +2026-09-21T05:11:34.446164Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=41 bs=2 bonus=11 draft_tokens=[220] target_tokens=[7383, 15] accepted=0 new_tokens=[7383] +2026-09-21T05:11:34.474603Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=42 bs=2 bonus=7383 draft_tokens=[8] target_tokens=[8, 198] accepted=1 new_tokens=[8, 198] +2026-09-21T05:11:34.500680Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=43 bs=2 bonus=198 draft_tokens=[309] target_tokens=[309, 30056] accepted=1 new_tokens=[309, 30056] +2026-09-21T05:11:34.526706Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=44 bs=2 bonus=30056 draft_tokens=[58308] target_tokens=[1989, 8] accepted=0 new_tokens=[1989] +2026-09-21T05:11:34.555292Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=45 bs=2 bonus=1989 draft_tokens=[58308] target_tokens=[58308, 8] accepted=1 new_tokens=[58308, 8] +2026-09-21T05:11:34.582164Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=46 bs=2 bonus=8 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:34.608498Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=47 bs=2 bonus=285 draft_tokens=[460] target_tokens=[460, 30056] accepted=1 new_tokens=[460, 30056] +2026-09-21T05:11:34.635352Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=48 bs=2 bonus=30056 draft_tokens=[271] target_tokens=[1358, 262] accepted=0 new_tokens=[1358] +2026-09-21T05:11:34.664025Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=49 bs=2 bonus=1358 draft_tokens=[727] target_tokens=[727, 21979] accepted=1 new_tokens=[727, 21979] +2026-09-21T05:11:34.691802Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=50 bs=2 bonus=21979 draft_tokens=[6406] target_tokens=[6406, 3764] accepted=1 new_tokens=[6406, 3764] +2026-09-21T05:11:34.717696Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=51 bs=2 bonus=3764 draft_tokens=[3764] target_tokens=[1783, 1783] accepted=0 new_tokens=[1783] +2026-09-21T05:11:34.744737Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=52 bs=2 bonus=1783 draft_tokens=[17994] target_tokens=[7, 11] accepted=0 new_tokens=[7] +2026-09-21T05:11:34.771329Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=53 bs=2 bonus=7 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:34.798365Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=54 bs=2 bonus=262 draft_tokens=[2821] target_tokens=[2821, 11] accepted=1 new_tokens=[2821, 11] +2026-09-21T05:11:34.824964Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=55 bs=2 bonus=11 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:34.851533Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=56 bs=2 bonus=262 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:34.877955Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=57 bs=2 bonus=1783 draft_tokens=[28] target_tokens=[22064, 3581] accepted=0 new_tokens=[22064] +2026-09-21T05:11:34.906862Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=58 bs=2 bonus=22064 draft_tokens=[28] target_tokens=[25, 3581] accepted=0 new_tokens=[25] +2026-09-21T05:11:34.935325Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=59 bs=2 bonus=25 draft_tokens=[6627] target_tokens=[6627, 1783] accepted=1 new_tokens=[6627, 1783] +2026-09-21T05:11:34.961452Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=60 bs=2 bonus=1783 draft_tokens=[22064] target_tokens=[13429, 283] accepted=0 new_tokens=[13429] +2026-09-21T05:11:34.988941Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=61 bs=2 bonus=13429 draft_tokens=[283] target_tokens=[283, 2168] accepted=1 new_tokens=[283, 2168] +2026-09-21T05:11:35.016328Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=62 bs=2 bonus=2168 draft_tokens=[11] target_tokens=[11, 198] accepted=1 new_tokens=[11, 198] +2026-09-21T05:11:35.042344Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=63 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 5741] accepted=1 new_tokens=[262, 5741] +2026-09-21T05:11:35.069354Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=64 bs=2 bonus=5741 draft_tokens=[25] target_tokens=[25, 8132] accepted=1 new_tokens=[25, 8132] +2026-09-21T05:11:35.096166Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=65 bs=2 bonus=8132 draft_tokens=[283] target_tokens=[12385, 2168] accepted=0 new_tokens=[12385] +2026-09-21T05:11:35.124766Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=66 bs=2 bonus=12385 draft_tokens=[283] target_tokens=[283, 2168] accepted=1 new_tokens=[283, 2168] +2026-09-21T05:11:35.152461Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=67 bs=2 bonus=2168 draft_tokens=[11] target_tokens=[11, 198] accepted=1 new_tokens=[11, 198] +2026-09-21T05:11:35.181531Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=68 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 2972] accepted=1 new_tokens=[262, 2972] +2026-09-21T05:11:35.208160Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=69 bs=2 bonus=2972 draft_tokens=[1371] target_tokens=[9386, 1783] accepted=0 new_tokens=[9386] +2026-09-21T05:11:35.236394Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=70 bs=2 bonus=9386 draft_tokens=[262] target_tokens=[11, 653] accepted=0 new_tokens=[11] +2026-09-21T05:11:35.264856Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=71 bs=2 bonus=11 draft_tokens=[198] target_tokens=[198, 1590] accepted=1 new_tokens=[198, 1590] +2026-09-21T05:11:35.290799Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=72 bs=2 bonus=1590 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:35.317183Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=73 bs=2 bonus=262 draft_tokens=[4071] target_tokens=[4071, 49823] accepted=1 new_tokens=[4071, 49823] +2026-09-21T05:11:35.343652Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=74 bs=2 bonus=49823 draft_tokens=[264] target_tokens=[264, 709] accepted=1 new_tokens=[264, 709] +2026-09-21T05:11:35.370316Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=75 bs=2 bonus=709 draft_tokens=[440] target_tokens=[440, 56768] accepted=1 new_tokens=[440, 56768] +2026-09-21T05:11:35.396429Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=76 bs=2 bonus=56768 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:35.422886Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=77 bs=2 bonus=1783 draft_tokens=[321] target_tokens=[321, 81480] accepted=1 new_tokens=[321, 81480] +2026-09-21T05:11:35.451147Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=78 bs=2 bonus=81480 draft_tokens=[262] target_tokens=[13, 4071] accepted=0 new_tokens=[13] +2026-09-21T05:11:35.479817Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=79 bs=2 bonus=13 draft_tokens=[271] target_tokens=[271, 262] accepted=1 new_tokens=[271, 262] +2026-09-21T05:11:35.506427Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=80 bs=2 bonus=262 draft_tokens=[17167] target_tokens=[17167, 25] accepted=1 new_tokens=[17167, 25] +2026-09-21T05:11:35.533317Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=81 bs=2 bonus=25 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:35.559592Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=82 bs=2 bonus=285 draft_tokens=[2821] target_tokens=[2821, 25] accepted=1 new_tokens=[2821, 25] +2026-09-21T05:11:35.586919Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=83 bs=2 bonus=25 draft_tokens=[561] target_tokens=[561, 709] accepted=1 new_tokens=[561, 709] +2026-09-21T05:11:35.613822Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=84 bs=2 bonus=709 draft_tokens=[310] target_tokens=[310, 21979] accepted=1 new_tokens=[310, 21979] +2026-09-21T05:11:35.640914Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=85 bs=2 bonus=21979 draft_tokens=[13] target_tokens=[13, 198] accepted=1 new_tokens=[13, 198] +2026-09-21T05:11:35.669091Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=86 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 1142] accepted=1 new_tokens=[285, 1142] +2026-09-21T05:11:35.697105Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=87 bs=2 bonus=1142 draft_tokens=[1783] target_tokens=[1783, 22064] accepted=1 new_tokens=[1783, 22064] +2026-09-21T05:11:35.724235Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=88 bs=2 bonus=22064 draft_tokens=[25] target_tokens=[25, 561] accepted=1 new_tokens=[25, 561] +2026-09-21T05:11:35.752128Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=89 bs=2 bonus=561 draft_tokens=[1142] target_tokens=[1142, 1783] accepted=1 new_tokens=[1142, 1783] +2026-09-21T05:11:35.778976Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=90 bs=2 bonus=1783 draft_tokens=[4687] target_tokens=[4687, 310] accepted=1 new_tokens=[4687, 310] +2026-09-21T05:11:35.806264Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=91 bs=2 bonus=310 draft_tokens=[958] target_tokens=[958, 13] accepted=1 new_tokens=[958, 13] +2026-09-21T05:11:35.833576Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=92 bs=2 bonus=13 draft_tokens=[34797] target_tokens=[198, 310] accepted=0 new_tokens=[198] +2026-09-21T05:11:35.862369Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=93 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 5741] accepted=1 new_tokens=[285, 5741] +2026-09-21T05:11:35.889450Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=94 bs=2 bonus=5741 draft_tokens=[25] target_tokens=[25, 561] accepted=1 new_tokens=[25, 561] +2026-09-21T05:11:35.916385Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=95 bs=2 bonus=561 draft_tokens=[5741] target_tokens=[5741, 310] accepted=1 new_tokens=[5741, 310] +2026-09-21T05:11:35.944177Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=96 bs=2 bonus=310 draft_tokens=[958] target_tokens=[958, 364] accepted=1 new_tokens=[958, 364] +2026-09-21T05:11:35.971798Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=97 bs=2 bonus=364 draft_tokens=[8132] target_tokens=[8132, 13] accepted=1 new_tokens=[8132, 13] +2026-09-21T05:11:35.998710Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=98 bs=2 bonus=13 draft_tokens=[198] target_tokens=[198, 285] accepted=1 new_tokens=[198, 285] +2026-09-21T05:11:36.025304Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=99 bs=2 bonus=285 draft_tokens=[2972] target_tokens=[2972, 9386] accepted=1 new_tokens=[2972, 9386] +2026-09-21T05:11:36.052938Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=100 bs=2 bonus=9386 draft_tokens=[25] target_tokens=[25, 22484] accepted=1 new_tokens=[25, 22484] +2026-09-21T05:11:36.079139Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=101 bs=2 bonus=22484 draft_tokens=[5789] target_tokens=[5789, 310] accepted=1 new_tokens=[5789, 310] +2026-09-21T05:11:36.104843Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=102 bs=2 bonus=310 draft_tokens=[1440] target_tokens=[1440, 310] accepted=1 new_tokens=[1440, 310] +2026-09-21T05:11:36.133322Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=103 bs=2 bonus=310 draft_tokens=[279] target_tokens=[279, 709] accepted=1 new_tokens=[279, 709] +2026-09-21T05:11:36.160342Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=104 bs=2 bonus=709 draft_tokens=[13] target_tokens=[13, 271] accepted=1 new_tokens=[13, 271] +2026-09-21T05:11:36.187707Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=105 bs=2 bonus=271 draft_tokens=[262] target_tokens=[262, 5019] accepted=1 new_tokens=[262, 5019] +2026-09-21T05:11:36.214493Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=106 bs=2 bonus=5019 draft_tokens=[25] target_tokens=[25, 198] accepted=1 new_tokens=[25, 198] +2026-09-21T05:11:36.241716Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=107 bs=2 bonus=198 draft_tokens=[285] target_tokens=[285, 561] accepted=1 new_tokens=[285, 561] +2026-09-21T05:11:36.269240Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=108 bs=2 bonus=561 draft_tokens=[1067] target_tokens=[1067, 314] accepted=1 new_tokens=[1067, 314] +2026-09-21T05:11:36.295968Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=109 bs=2 bonus=314 draft_tokens=[279] target_tokens=[279, 709] accepted=1 new_tokens=[279, 709] +2026-09-21T05:11:36.321850Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=110 bs=2 bonus=709 draft_tokens=[13] target_tokens=[13, 198] accepted=1 new_tokens=[13, 198] +2026-09-21T05:11:36.347422Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=111 bs=2 bonus=198 draft_tokens=[262] target_tokens=[262, 4071] accepted=1 new_tokens=[262, 4071] +2026-09-21T05:11:36.375359Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=112 bs=2 bonus=4071 draft_tokens=[198] target_tokens=[198, 262] accepted=1 new_tokens=[198, 262] +2026-09-21T05:11:36.401850Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=113 bs=2 bonus=262 draft_tokens=[413] target_tokens=[413, 1142] accepted=1 new_tokens=[413] +2026-09-21T05:11:36.402056Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=2 rounds=114 proposed_tokens=114 accepted_tokens=86 acceptance_rate=0.7543859649122807 emitted_per_verify=1.7456140350877194 zero_accept_rounds=28 partial_accept_rounds=0 full_accept_rounds=86 prefill_verify_ms=2.756508 first_bonus_ms=121.068357 first_hidden_ms=0.003968 bind_reset_ms=0.029297 draft_ms=355.4514200000002 verify_ms=300.3111670000001 target_argmax_sync_ms=2210.4587670000005 logprobs_ms=214.60134399999998 walk_ms=0.022352 hidden_concat_ms=0.24859499999999998 rollback_ms=13.106450999999998 decode_ms=3111.659505 +2026-09-21T05:11:36.402066Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] +2026-09-21T05:11:36.402346Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3235 +2026-09-21T05:11:36.402402Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=3235.969287 burst_active_ms=3235.969287 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w4.log b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w4.log new file mode 100644 index 000000000..964a86700 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.w4.log @@ -0,0 +1,206 @@ +2026-09-21T05:11:09.507670Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T05:11:09.507712Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T05:11:09.507843Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T05:11:09.507917Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T05:11:09.507920Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T05:11:09.507922Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T05:11:09.843234Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T05:11:09.843267Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T05:11:09.843829Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T05:11:10.185589Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T05:11:10.185682Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T05:11:10.537394Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.352s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.351698343 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T05:11:10.537477Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=4, block_size_from=operator override, explicit_kind=true)) +2026-09-21T05:11:10.537894Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T05:11:10.537908Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T05:11:10.537932Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T05:11:12.169558Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=4 +2026-09-21T05:11:12.169587Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T05:11:12.170016Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T05:11:12.171109Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T05:11:12.194575Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=1.045307 first_bonus_ms=23.404116 first_hidden_ms=0.007312 bind_reset_ms=0.064849 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T05:11:12.194629Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[] +2026-09-21T05:11:12.194786Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1656 +2026-09-21T05:11:12.194794Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1656.630112 burst_active_ms=1656.630112 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T05:11:12.194832Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T05:11:12.194952Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T05:11:12.196103Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T05:11:12.196868Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18935 +2026-09-21T05:11:12.196875Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T05:11:12.196878Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T05:11:12.196880Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T05:11:12.196881Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T05:11:12.196882Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T05:11:12.196884Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T05:11:12.196885Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T05:11:12.196886Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T05:11:12.196887Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T05:11:12.196889Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T05:11:12.196890Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T05:11:12.196892Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T05:11:14.213753Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=0 bs=4 bonus=285 draft_tokens=[4071, 56, 303] target_tokens=[4071, 30097, 790, 264] accepted=1 new_tokens=[4071, 30097] +2026-09-21T05:11:14.281019Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=1 bs=4 bonus=30097 draft_tokens=[264, 314, 314] target_tokens=[264, 8240, 30056, 30056] accepted=1 new_tokens=[264, 8240] +2026-09-21T05:11:14.344898Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=2 bs=4 bonus=8240 draft_tokens=[314, 30056, 13] target_tokens=[314, 30056, 364, 271] accepted=2 new_tokens=[314, 30056, 364] +2026-09-21T05:11:14.407366Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=3 bs=4 bonus=364 draft_tokens=[1754, 71483, 71483] target_tokens=[21979, 4621, 198, 198] accepted=0 new_tokens=[21979] +2026-09-21T05:11:14.475593Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=4 bs=4 bonus=21979 draft_tokens=[286, 13, 198] target_tokens=[286, 71483, 271, 285] accepted=1 new_tokens=[286, 71483] +2026-09-21T05:11:14.539341Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=5 bs=4 bonus=71483 draft_tokens=[285, 283, 283] target_tokens=[198, 653, 39965, 283] accepted=0 new_tokens=[198] +2026-09-21T05:11:14.598731Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=6 bs=4 bonus=198 draft_tokens=[285, 283, 283] target_tokens=[285, 30056, 39965, 39965] accepted=1 new_tokens=[285, 30056] +2026-09-21T05:11:14.657847Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=7 bs=4 bonus=30056 draft_tokens=[283, 2958, 198] target_tokens=[283, 2958, 198, 285] accepted=3 new_tokens=[283, 2958, 198, 285] +2026-09-21T05:11:14.715891Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=8 bs=4 bonus=285 draft_tokens=[364, 283, 303] target_tokens=[364, 585, 303, 39965] accepted=1 new_tokens=[364, 585] +2026-09-21T05:11:14.777243Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=9 bs=4 bonus=585 draft_tokens=[303, 2020, 6686] target_tokens=[303, 2020, 1153, 1153] accepted=2 new_tokens=[303, 2020, 1153] +2026-09-21T05:11:14.837361Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=10 bs=4 bonus=1153 draft_tokens=[16, 35497, 198] target_tokens=[6469, 6469, 198, 309] accepted=0 new_tokens=[6469] +2026-09-21T05:11:14.898386Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=11 bs=4 bonus=6469 draft_tokens=[1153, 35497, 198] target_tokens=[76753, 6469, 198, 309] accepted=0 new_tokens=[76753] +2026-09-21T05:11:14.957355Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=12 bs=4 bonus=76753 draft_tokens=[198, 198, 309] target_tokens=[1590, 5015, 248069, 16026] accepted=0 new_tokens=[1590] +2026-09-21T05:11:15.017213Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=13 bs=4 bonus=1590 draft_tokens=[198, 309, 13161] target_tokens=[198, 309, 7383, 283] accepted=2 new_tokens=[198, 309, 7383] +2026-09-21T05:11:15.079448Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=14 bs=4 bonus=7383 draft_tokens=[283, 638, 348] target_tokens=[283, 638, 8658, 318] accepted=2 new_tokens=[283, 638, 8658] +2026-09-21T05:11:15.140889Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=15 bs=4 bonus=8658 draft_tokens=[1889, 471, 318] target_tokens=[21510, 8, 220, 701] accepted=0 new_tokens=[21510] +2026-09-21T05:11:15.201404Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=16 bs=4 bonus=21510 draft_tokens=[2972, 318, 17] target_tokens=[348, 318, 72, 348] accepted=0 new_tokens=[348] +2026-09-21T05:11:15.260213Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=17 bs=4 bonus=348 draft_tokens=[318, 17, 2972] target_tokens=[318, 17, 2972, 585] accepted=3 new_tokens=[318, 17, 2972, 585] +2026-09-21T05:11:15.317922Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=18 bs=4 bonus=585 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 7383] accepted=3 new_tokens=[8, 198, 309, 7383] +2026-09-21T05:11:15.377597Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=19 bs=4 bonus=7383 draft_tokens=[283, 1262, 4217] target_tokens=[283, 1262, 58308, 58308] accepted=2 new_tokens=[283, 1262, 58308] +2026-09-21T05:11:15.437597Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=20 bs=4 bonus=58308 draft_tokens=[638, 638, 76753] target_tokens=[11, 6469, 6469, 8] accepted=0 new_tokens=[11] +2026-09-21T05:11:15.497449Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=21 bs=4 bonus=11 draft_tokens=[638, 6469, 76753] target_tokens=[638, 6469, 21510, 8] accepted=2 new_tokens=[638, 6469, 21510] +2026-09-21T05:11:15.556504Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=22 bs=4 bonus=21510 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 413] accepted=3 new_tokens=[8, 198, 309, 413] +2026-09-21T05:11:15.614301Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=23 bs=4 bonus=413 draft_tokens=[585, 361, 220] target_tokens=[638, 835, 638, 15] accepted=0 new_tokens=[638] +2026-09-21T05:11:15.677499Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=24 bs=4 bonus=638 draft_tokens=[39281, 25, 198] target_tokens=[1130, 5206, 198, 388] accepted=0 new_tokens=[1130] +2026-09-21T05:11:15.741215Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=25 bs=4 bonus=1130 draft_tokens=[3143, 25, 198] target_tokens=[3143, 25, 198, 388] accepted=3 new_tokens=[3143, 25, 198, 388] +2026-09-21T05:11:15.801376Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=26 bs=4 bonus=388 draft_tokens=[7383, 1373, 4055] target_tokens=[7383, 283, 4055, 34593] accepted=1 new_tokens=[7383, 283] +2026-09-21T05:11:15.861625Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=27 bs=4 bonus=283 draft_tokens=[318, 478, 318] target_tokens=[7383, 16, 4055, 11134] accepted=0 new_tokens=[7383] +2026-09-21T05:11:15.923404Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=28 bs=4 bonus=7383 draft_tokens=[478, 318, 15] target_tokens=[478, 4055, 11134, 13] accepted=1 new_tokens=[478, 4055] +2026-09-21T05:11:15.981886Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=29 bs=4 bonus=4055 draft_tokens=[34593, 15, 15] target_tokens=[34593, 7, 16, 7] accepted=1 new_tokens=[34593, 7] +2026-09-21T05:11:16.042656Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=30 bs=4 bonus=7 draft_tokens=[15, 11, 471] target_tokens=[15, 11, 7383, 21572] accepted=2 new_tokens=[15, 11, 7383] +2026-09-21T05:11:16.106219Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=31 bs=4 bonus=7383 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 30056] accepted=3 new_tokens=[8, 198, 309, 30056] +2026-09-21T05:11:16.165698Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=32 bs=4 bonus=30056 draft_tokens=[58308, 8, 198] target_tokens=[1989, 8, 198, 285] accepted=0 new_tokens=[1989] +2026-09-21T05:11:16.226950Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=33 bs=4 bonus=1989 draft_tokens=[58308, 8, 198] target_tokens=[58308, 8, 198, 285] accepted=3 new_tokens=[58308, 8, 198, 285] +2026-09-21T05:11:16.290022Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=34 bs=4 bonus=285 draft_tokens=[460, 30056, 198] target_tokens=[460, 30056, 1358, 262] accepted=2 new_tokens=[460, 30056, 1358] +2026-09-21T05:11:16.352029Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=35 bs=4 bonus=1358 draft_tokens=[40, 3172, 2962] target_tokens=[727, 283, 1791, 279] accepted=0 new_tokens=[727] +2026-09-21T05:11:16.411236Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=36 bs=4 bonus=727 draft_tokens=[21979, 60631, 60631] target_tokens=[21979, 6406, 470, 7] accepted=1 new_tokens=[21979, 6406] +2026-09-21T05:11:16.471735Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=37 bs=4 bonus=6406 draft_tokens=[21510, 58308, 28] target_tokens=[3764, 7, 14802, 16775] accepted=0 new_tokens=[3764] +2026-09-21T05:11:16.532956Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=38 bs=4 bonus=3764 draft_tokens=[3764, 198, 198] target_tokens=[1783, 1783, 262, 262] accepted=0 new_tokens=[1783] +2026-09-21T05:11:16.596236Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=39 bs=4 bonus=1783 draft_tokens=[17994, 1783, 198] target_tokens=[7, 11, 11, 5015] accepted=0 new_tokens=[7] +2026-09-21T05:11:16.656536Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=40 bs=4 bonus=7 draft_tokens=[198, 262, 198] target_tokens=[198, 262, 2821, 262] accepted=2 new_tokens=[198, 262, 2821] +2026-09-21T05:11:16.721574Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=41 bs=4 bonus=2821 draft_tokens=[11, 198, 262] target_tokens=[11, 198, 262, 1142] accepted=3 new_tokens=[11, 198, 262, 1142] +2026-09-21T05:11:16.781016Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=42 bs=4 bonus=1142 draft_tokens=[1783, 28, 11] target_tokens=[1783, 22064, 3581, 198] accepted=1 new_tokens=[1783, 22064] +2026-09-21T05:11:16.842886Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=43 bs=4 bonus=22064 draft_tokens=[28, 28, 198] target_tokens=[25, 3581, 3581, 262] accepted=0 new_tokens=[25] +2026-09-21T05:11:16.907536Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=44 bs=4 bonus=25 draft_tokens=[3581, 283, 1371] target_tokens=[6627, 1783, 2168, 1783] accepted=0 new_tokens=[6627] +2026-09-21T05:11:16.968745Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=45 bs=4 bonus=6627 draft_tokens=[1783, 283, 283] target_tokens=[1783, 13429, 2168, 6627] accepted=1 new_tokens=[1783, 13429] +2026-09-21T05:11:17.031025Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=46 bs=4 bonus=13429 draft_tokens=[283, 2168, 1783] target_tokens=[283, 2168, 11, 22064] accepted=2 new_tokens=[283, 2168, 11] +2026-09-21T05:11:17.091608Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=47 bs=4 bonus=11 draft_tokens=[198, 262, 1866] target_tokens=[198, 262, 5741, 76753] accepted=2 new_tokens=[198, 262, 5741] +2026-09-21T05:11:17.153655Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=48 bs=4 bonus=5741 draft_tokens=[25, 25, 283] target_tokens=[25, 8132, 24461, 2168] accepted=1 new_tokens=[25, 8132] +2026-09-21T05:11:17.214410Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=49 bs=4 bonus=8132 draft_tokens=[283, 2168, 283] target_tokens=[12385, 2168, 11, 2168] accepted=0 new_tokens=[12385] +2026-09-21T05:11:17.274800Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=50 bs=4 bonus=12385 draft_tokens=[283, 2168, 11] target_tokens=[283, 2168, 11, 198] accepted=3 new_tokens=[283, 2168, 11, 198] +2026-09-21T05:11:17.334067Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=51 bs=4 bonus=198 draft_tokens=[262, 1866, 25] target_tokens=[262, 2972, 76753, 514] accepted=1 new_tokens=[262, 2972] +2026-09-21T05:11:17.395108Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=52 bs=4 bonus=2972 draft_tokens=[1371, 332, 198] target_tokens=[9386, 1783, 9386, 262] accepted=0 new_tokens=[9386] +2026-09-21T05:11:17.456694Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=53 bs=4 bonus=9386 draft_tokens=[198, 198, 262] target_tokens=[11, 1590, 1590, 16026] accepted=0 new_tokens=[11] +2026-09-21T05:11:17.518965Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=54 bs=4 bonus=11 draft_tokens=[198, 262, 2972] target_tokens=[198, 1590, 2972, 43355] accepted=1 new_tokens=[198, 1590] +2026-09-21T05:11:17.579898Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=55 bs=4 bonus=1590 draft_tokens=[198, 262, 4071] target_tokens=[198, 262, 4071, 49823] accepted=3 new_tokens=[198, 262, 4071, 49823] +2026-09-21T05:11:17.640651Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=56 bs=4 bonus=49823 draft_tokens=[264, 1142, 1142] target_tokens=[264, 709, 1783, 1783] accepted=1 new_tokens=[264, 709] +2026-09-21T05:11:17.701237Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=57 bs=4 bonus=709 draft_tokens=[440, 56768, 1783] target_tokens=[440, 56768, 1142, 321] accepted=2 new_tokens=[440, 56768, 1142] +2026-09-21T05:11:17.763285Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=58 bs=4 bonus=1142 draft_tokens=[1783, 13, 198] target_tokens=[1783, 321, 271, 262] accepted=1 new_tokens=[1783, 321] +2026-09-21T05:11:17.823155Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=59 bs=4 bonus=321 draft_tokens=[81480, 13, 13] target_tokens=[81480, 13, 271, 271] accepted=2 new_tokens=[81480, 13, 271] +2026-09-21T05:11:17.885408Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=60 bs=4 bonus=271 draft_tokens=[262, 17167, 25] target_tokens=[262, 17167, 25, 198] accepted=3 new_tokens=[262, 17167, 25, 198] +2026-09-21T05:11:17.947316Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=61 bs=4 bonus=198 draft_tokens=[285, 2821, 25] target_tokens=[285, 2821, 25, 561] accepted=3 new_tokens=[285, 2821, 25, 561] +2026-09-21T05:11:18.009394Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=62 bs=4 bonus=561 draft_tokens=[709, 310, 381] target_tokens=[709, 310, 21979, 36482] accepted=2 new_tokens=[709, 310, 21979] +2026-09-21T05:11:18.071232Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=63 bs=4 bonus=21979 draft_tokens=[13, 198, 285] target_tokens=[13, 198, 285, 1142] accepted=3 new_tokens=[13, 198, 285, 1142] +2026-09-21T05:11:18.130913Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=64 bs=4 bonus=1142 draft_tokens=[1783, 22064, 25] target_tokens=[1783, 22064, 25, 561] accepted=3 new_tokens=[1783, 22064, 25, 561] +2026-09-21T05:11:18.190314Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=65 bs=4 bonus=561 draft_tokens=[1142, 1783, 310] target_tokens=[1142, 1783, 4687, 958] accepted=2 new_tokens=[1142, 1783, 4687] +2026-09-21T05:11:18.253283Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=66 bs=4 bonus=4687 draft_tokens=[310, 958, 13] target_tokens=[310, 958, 13, 198] accepted=3 new_tokens=[310, 958, 13, 198] +2026-09-21T05:11:18.317207Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=67 bs=4 bonus=198 draft_tokens=[285, 5741, 25] target_tokens=[285, 5741, 25, 561] accepted=3 new_tokens=[285, 5741, 25, 561] +2026-09-21T05:11:18.377181Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=68 bs=4 bonus=561 draft_tokens=[5741, 310, 958] target_tokens=[5741, 310, 958, 364] accepted=3 new_tokens=[5741, 310, 958, 364] +2026-09-21T05:11:18.437115Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=69 bs=4 bonus=364 draft_tokens=[8132, 13, 13] target_tokens=[8132, 13, 198, 198] accepted=2 new_tokens=[8132, 13, 198] +2026-09-21T05:11:18.499160Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=70 bs=4 bonus=198 draft_tokens=[285, 2972, 9386] target_tokens=[285, 2972, 9386, 25] accepted=3 new_tokens=[285, 2972, 9386, 25] +2026-09-21T05:11:18.559825Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=71 bs=4 bonus=25 draft_tokens=[26804, 5789, 5789] target_tokens=[22484, 310, 310, 310] accepted=0 new_tokens=[22484] +2026-09-21T05:11:18.619294Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=72 bs=4 bonus=22484 draft_tokens=[5789, 310, 279] target_tokens=[5789, 310, 1440, 709] accepted=2 new_tokens=[5789, 310, 1440] +2026-09-21T05:11:18.682395Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=73 bs=4 bonus=1440 draft_tokens=[310, 279, 709] target_tokens=[310, 279, 709, 13] accepted=3 new_tokens=[310, 279, 709, 13] +2026-09-21T05:11:18.739957Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=74 bs=4 bonus=13 draft_tokens=[271, 262, 4071] target_tokens=[271, 262, 5019, 198] accepted=2 new_tokens=[271, 262, 5019] +2026-09-21T05:11:18.802837Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=75 bs=4 bonus=5019 draft_tokens=[25, 198, 285] target_tokens=[25, 198, 285, 561] accepted=3 new_tokens=[25, 198, 285, 561] +2026-09-21T05:11:18.863315Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=76 bs=4 bonus=561 draft_tokens=[1067, 314, 279] target_tokens=[1067, 314, 279, 709] accepted=3 new_tokens=[1067, 314, 279, 709] +2026-09-21T05:11:18.924010Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=77 bs=4 bonus=709 draft_tokens=[13, 198, 262] target_tokens=[13, 198, 262, 4071] accepted=3 new_tokens=[13, 198, 262, 4071] +2026-09-21T05:11:18.983137Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=78 bs=4 bonus=4071 draft_tokens=[198, 262, 413] target_tokens=[198, 262, 413, 1142] accepted=3 new_tokens=[198, 262, 413] +2026-09-21T05:11:18.983388Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=4.998724 first_bonus_ms=596.739476 first_hidden_ms=0.0046559999999999995 bind_reset_ms=0.029153000000000002 draft_ms=2111.526231 verify_ms=192.88915599999999 target_argmax_sync_ms=2445.562649999999 logprobs_ms=356.613477 walk_ms=0.01952 hidden_concat_ms=0.12640200000000001 rollback_ms=40.76056499999999 decode_ms=5156.371596 +2026-09-21T05:11:18.983397Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[1, 1, 2, 0, 1, 0, 1, 3, 1, 2, 0, 0, 0, 2, 2, 0, 0, 3, 3, 2, 0, 2, 3, 0, 0, 3, 1, 0, 1, 1, 2, 3, 0, 3, 2, 0, 1, 0, 0, 0, 2, 3, 1, 0, 0, 1, 2, 2, 1, 0, 3, 1, 0, 0, 1, 3, 1, 2, 1, 2, 3, 3, 2, 3, 3, 2, 3, 3, 3, 2, 3, 0, 2, 3, 2, 3, 3, 3, 3] +2026-09-21T05:11:18.983739Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5758 +2026-09-21T05:11:18.983789Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5758.670555 burst_active_ms=5758.670555 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T05:11:19.153160Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=0 bs=4 bonus=285 draft_tokens=[4071, 56, 303] target_tokens=[4071, 30097, 790, 264] accepted=1 new_tokens=[4071, 30097] +2026-09-21T05:11:19.195810Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=1 bs=4 bonus=30097 draft_tokens=[264, 314, 314] target_tokens=[264, 8240, 30056, 30056] accepted=1 new_tokens=[264, 8240] +2026-09-21T05:11:19.234978Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=2 bs=4 bonus=8240 draft_tokens=[314, 30056, 13] target_tokens=[314, 30056, 364, 271] accepted=2 new_tokens=[314, 30056, 364] +2026-09-21T05:11:19.275791Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=3 bs=4 bonus=364 draft_tokens=[1754, 71483, 71483] target_tokens=[21979, 4621, 198, 198] accepted=0 new_tokens=[21979] +2026-09-21T05:11:19.316833Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=4 bs=4 bonus=21979 draft_tokens=[286, 13, 198] target_tokens=[286, 71483, 271, 285] accepted=1 new_tokens=[286, 71483] +2026-09-21T05:11:19.359114Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=5 bs=4 bonus=71483 draft_tokens=[285, 283, 283] target_tokens=[198, 653, 39965, 283] accepted=0 new_tokens=[198] +2026-09-21T05:11:19.401414Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=6 bs=4 bonus=198 draft_tokens=[285, 283, 283] target_tokens=[285, 30056, 39965, 39965] accepted=1 new_tokens=[285, 30056] +2026-09-21T05:11:19.441455Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=7 bs=4 bonus=30056 draft_tokens=[283, 2958, 198] target_tokens=[283, 2958, 198, 285] accepted=3 new_tokens=[283, 2958, 198, 285] +2026-09-21T05:11:19.479250Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=8 bs=4 bonus=285 draft_tokens=[364, 283, 303] target_tokens=[364, 585, 303, 39965] accepted=1 new_tokens=[364, 585] +2026-09-21T05:11:19.520158Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=9 bs=4 bonus=585 draft_tokens=[303, 2020, 6686] target_tokens=[303, 2020, 1153, 1153] accepted=2 new_tokens=[303, 2020, 1153] +2026-09-21T05:11:19.563404Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=10 bs=4 bonus=1153 draft_tokens=[16, 35497, 198] target_tokens=[6469, 6469, 198, 309] accepted=0 new_tokens=[6469] +2026-09-21T05:11:19.603558Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=11 bs=4 bonus=6469 draft_tokens=[1153, 35497, 198] target_tokens=[76753, 6469, 198, 309] accepted=0 new_tokens=[76753] +2026-09-21T05:11:19.645436Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=12 bs=4 bonus=76753 draft_tokens=[198, 198, 309] target_tokens=[1590, 5015, 248069, 16026] accepted=0 new_tokens=[1590] +2026-09-21T05:11:19.685477Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=13 bs=4 bonus=1590 draft_tokens=[198, 309, 13161] target_tokens=[198, 309, 7383, 283] accepted=2 new_tokens=[198, 309, 7383] +2026-09-21T05:11:19.728087Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=14 bs=4 bonus=7383 draft_tokens=[283, 638, 348] target_tokens=[283, 638, 8658, 318] accepted=2 new_tokens=[283, 638, 8658] +2026-09-21T05:11:19.769080Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=15 bs=4 bonus=8658 draft_tokens=[1889, 471, 318] target_tokens=[21510, 8, 220, 701] accepted=0 new_tokens=[21510] +2026-09-21T05:11:19.809269Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=16 bs=4 bonus=21510 draft_tokens=[2972, 318, 17] target_tokens=[348, 318, 72, 348] accepted=0 new_tokens=[348] +2026-09-21T05:11:19.849980Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=17 bs=4 bonus=348 draft_tokens=[318, 17, 2972] target_tokens=[318, 17, 2972, 585] accepted=3 new_tokens=[318, 17, 2972, 585] +2026-09-21T05:11:19.890798Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=18 bs=4 bonus=585 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 7383] accepted=3 new_tokens=[8, 198, 309, 7383] +2026-09-21T05:11:19.931735Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=19 bs=4 bonus=7383 draft_tokens=[283, 1262, 4217] target_tokens=[283, 1262, 58308, 58308] accepted=2 new_tokens=[283, 1262, 58308] +2026-09-21T05:11:19.972325Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=20 bs=4 bonus=58308 draft_tokens=[638, 638, 76753] target_tokens=[11, 6469, 6469, 8] accepted=0 new_tokens=[11] +2026-09-21T05:11:20.013918Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=21 bs=4 bonus=11 draft_tokens=[638, 6469, 76753] target_tokens=[638, 6469, 21510, 8] accepted=2 new_tokens=[638, 6469, 21510] +2026-09-21T05:11:20.053407Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=22 bs=4 bonus=21510 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 413] accepted=3 new_tokens=[8, 198, 309, 413] +2026-09-21T05:11:20.091442Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=23 bs=4 bonus=413 draft_tokens=[585, 361, 220] target_tokens=[638, 835, 638, 15] accepted=0 new_tokens=[638] +2026-09-21T05:11:20.131650Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=24 bs=4 bonus=638 draft_tokens=[39281, 25, 198] target_tokens=[1130, 5206, 198, 388] accepted=0 new_tokens=[1130] +2026-09-21T05:11:20.172344Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=25 bs=4 bonus=1130 draft_tokens=[3143, 25, 198] target_tokens=[3143, 25, 198, 388] accepted=3 new_tokens=[3143, 25, 198, 388] +2026-09-21T05:11:20.213014Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=26 bs=4 bonus=388 draft_tokens=[7383, 1373, 4055] target_tokens=[7383, 283, 4055, 34593] accepted=1 new_tokens=[7383, 283] +2026-09-21T05:11:20.253381Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=27 bs=4 bonus=283 draft_tokens=[318, 478, 318] target_tokens=[7383, 16, 4055, 11134] accepted=0 new_tokens=[7383] +2026-09-21T05:11:20.292409Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=28 bs=4 bonus=7383 draft_tokens=[478, 318, 15] target_tokens=[478, 4055, 11134, 13] accepted=1 new_tokens=[478, 4055] +2026-09-21T05:11:20.333049Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=29 bs=4 bonus=4055 draft_tokens=[34593, 15, 15] target_tokens=[34593, 7, 16, 7] accepted=1 new_tokens=[34593, 7] +2026-09-21T05:11:20.373658Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=30 bs=4 bonus=7 draft_tokens=[15, 11, 471] target_tokens=[15, 11, 7383, 21572] accepted=2 new_tokens=[15, 11, 7383] +2026-09-21T05:11:20.415771Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=31 bs=4 bonus=7383 draft_tokens=[8, 198, 309] target_tokens=[8, 198, 309, 30056] accepted=3 new_tokens=[8, 198, 309, 30056] +2026-09-21T05:11:20.453785Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=32 bs=4 bonus=30056 draft_tokens=[58308, 8, 198] target_tokens=[1989, 8, 198, 285] accepted=0 new_tokens=[1989] +2026-09-21T05:11:20.493225Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=33 bs=4 bonus=1989 draft_tokens=[58308, 8, 198] target_tokens=[58308, 8, 198, 285] accepted=3 new_tokens=[58308, 8, 198, 285] +2026-09-21T05:11:20.532635Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=34 bs=4 bonus=285 draft_tokens=[460, 30056, 198] target_tokens=[460, 30056, 1358, 262] accepted=2 new_tokens=[460, 30056, 1358] +2026-09-21T05:11:20.574444Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=35 bs=4 bonus=1358 draft_tokens=[40, 3172, 2962] target_tokens=[727, 283, 1791, 279] accepted=0 new_tokens=[727] +2026-09-21T05:11:20.614606Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=36 bs=4 bonus=727 draft_tokens=[21979, 60631, 60631] target_tokens=[21979, 6406, 470, 7] accepted=1 new_tokens=[21979, 6406] +2026-09-21T05:11:20.655132Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=37 bs=4 bonus=6406 draft_tokens=[21510, 58308, 28] target_tokens=[3764, 7, 14802, 16775] accepted=0 new_tokens=[3764] +2026-09-21T05:11:20.695619Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=38 bs=4 bonus=3764 draft_tokens=[3764, 198, 198] target_tokens=[1783, 1783, 262, 262] accepted=0 new_tokens=[1783] +2026-09-21T05:11:20.734840Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=39 bs=4 bonus=1783 draft_tokens=[17994, 1783, 198] target_tokens=[7, 11, 11, 5015] accepted=0 new_tokens=[7] +2026-09-21T05:11:20.775231Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=40 bs=4 bonus=7 draft_tokens=[198, 262, 198] target_tokens=[198, 262, 2821, 262] accepted=2 new_tokens=[198, 262, 2821] +2026-09-21T05:11:20.817396Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=41 bs=4 bonus=2821 draft_tokens=[11, 198, 262] target_tokens=[11, 198, 262, 1142] accepted=3 new_tokens=[11, 198, 262, 1142] +2026-09-21T05:11:20.857058Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=42 bs=4 bonus=1142 draft_tokens=[1783, 28, 11] target_tokens=[1783, 22064, 3581, 198] accepted=1 new_tokens=[1783, 22064] +2026-09-21T05:11:20.899372Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=43 bs=4 bonus=22064 draft_tokens=[28, 28, 198] target_tokens=[25, 3581, 3581, 262] accepted=0 new_tokens=[25] +2026-09-21T05:11:20.938285Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=44 bs=4 bonus=25 draft_tokens=[3581, 283, 1371] target_tokens=[6627, 1783, 2168, 1783] accepted=0 new_tokens=[6627] +2026-09-21T05:11:20.978009Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=45 bs=4 bonus=6627 draft_tokens=[1783, 283, 283] target_tokens=[1783, 13429, 2168, 6627] accepted=1 new_tokens=[1783, 13429] +2026-09-21T05:11:21.017227Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=46 bs=4 bonus=13429 draft_tokens=[283, 2168, 1783] target_tokens=[283, 2168, 11, 22064] accepted=2 new_tokens=[283, 2168, 11] +2026-09-21T05:11:21.059472Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=47 bs=4 bonus=11 draft_tokens=[198, 262, 1866] target_tokens=[198, 262, 5741, 76753] accepted=2 new_tokens=[198, 262, 5741] +2026-09-21T05:11:21.100404Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=48 bs=4 bonus=5741 draft_tokens=[25, 25, 283] target_tokens=[25, 8132, 24461, 2168] accepted=1 new_tokens=[25, 8132] +2026-09-21T05:11:21.141806Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=49 bs=4 bonus=8132 draft_tokens=[283, 2168, 283] target_tokens=[12385, 2168, 11, 2168] accepted=0 new_tokens=[12385] +2026-09-21T05:11:21.181956Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=50 bs=4 bonus=12385 draft_tokens=[283, 2168, 11] target_tokens=[283, 2168, 11, 198] accepted=3 new_tokens=[283, 2168, 11, 198] +2026-09-21T05:11:21.220736Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=51 bs=4 bonus=198 draft_tokens=[262, 1866, 25] target_tokens=[262, 2972, 76753, 514] accepted=1 new_tokens=[262, 2972] +2026-09-21T05:11:21.261892Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=52 bs=4 bonus=2972 draft_tokens=[1371, 332, 198] target_tokens=[9386, 1783, 9386, 262] accepted=0 new_tokens=[9386] +2026-09-21T05:11:21.300888Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=53 bs=4 bonus=9386 draft_tokens=[198, 198, 262] target_tokens=[11, 1590, 1590, 16026] accepted=0 new_tokens=[11] +2026-09-21T05:11:21.341258Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=54 bs=4 bonus=11 draft_tokens=[198, 262, 2972] target_tokens=[198, 1590, 2972, 43355] accepted=1 new_tokens=[198, 1590] +2026-09-21T05:11:21.380697Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=55 bs=4 bonus=1590 draft_tokens=[198, 262, 4071] target_tokens=[198, 262, 4071, 49823] accepted=3 new_tokens=[198, 262, 4071, 49823] +2026-09-21T05:11:21.417647Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=56 bs=4 bonus=49823 draft_tokens=[264, 1142, 1142] target_tokens=[264, 709, 1783, 1783] accepted=1 new_tokens=[264, 709] +2026-09-21T05:11:21.458432Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=57 bs=4 bonus=709 draft_tokens=[440, 56768, 1783] target_tokens=[440, 56768, 1142, 321] accepted=2 new_tokens=[440, 56768, 1142] +2026-09-21T05:11:21.499594Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=58 bs=4 bonus=1142 draft_tokens=[1783, 13, 198] target_tokens=[1783, 321, 271, 262] accepted=1 new_tokens=[1783, 321] +2026-09-21T05:11:21.539746Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=59 bs=4 bonus=321 draft_tokens=[81480, 13, 13] target_tokens=[81480, 13, 271, 271] accepted=2 new_tokens=[81480, 13, 271] +2026-09-21T05:11:21.579244Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=60 bs=4 bonus=271 draft_tokens=[262, 17167, 25] target_tokens=[262, 17167, 25, 198] accepted=3 new_tokens=[262, 17167, 25, 198] +2026-09-21T05:11:21.616794Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=61 bs=4 bonus=198 draft_tokens=[285, 2821, 25] target_tokens=[285, 2821, 25, 561] accepted=3 new_tokens=[285, 2821, 25, 561] +2026-09-21T05:11:21.655854Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=62 bs=4 bonus=561 draft_tokens=[709, 310, 381] target_tokens=[709, 310, 21979, 36482] accepted=2 new_tokens=[709, 310, 21979] +2026-09-21T05:11:21.699246Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=63 bs=4 bonus=21979 draft_tokens=[13, 198, 285] target_tokens=[13, 198, 285, 1142] accepted=3 new_tokens=[13, 198, 285, 1142] +2026-09-21T05:11:21.737796Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=64 bs=4 bonus=1142 draft_tokens=[1783, 22064, 25] target_tokens=[1783, 22064, 25, 561] accepted=3 new_tokens=[1783, 22064, 25, 561] +2026-09-21T05:11:21.776281Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=65 bs=4 bonus=561 draft_tokens=[1142, 1783, 310] target_tokens=[1142, 1783, 4687, 958] accepted=2 new_tokens=[1142, 1783, 4687] +2026-09-21T05:11:21.817493Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=66 bs=4 bonus=4687 draft_tokens=[310, 958, 13] target_tokens=[310, 958, 13, 198] accepted=3 new_tokens=[310, 958, 13, 198] +2026-09-21T05:11:21.856998Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=67 bs=4 bonus=198 draft_tokens=[285, 5741, 25] target_tokens=[285, 5741, 25, 561] accepted=3 new_tokens=[285, 5741, 25, 561] +2026-09-21T05:11:21.895223Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=68 bs=4 bonus=561 draft_tokens=[5741, 310, 958] target_tokens=[5741, 310, 958, 364] accepted=3 new_tokens=[5741, 310, 958, 364] +2026-09-21T05:11:21.932476Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=69 bs=4 bonus=364 draft_tokens=[8132, 13, 13] target_tokens=[8132, 13, 198, 198] accepted=2 new_tokens=[8132, 13, 198] +2026-09-21T05:11:21.974038Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=70 bs=4 bonus=198 draft_tokens=[285, 2972, 9386] target_tokens=[285, 2972, 9386, 25] accepted=3 new_tokens=[285, 2972, 9386, 25] +2026-09-21T05:11:22.013485Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=71 bs=4 bonus=25 draft_tokens=[26804, 5789, 5789] target_tokens=[22484, 310, 310, 310] accepted=0 new_tokens=[22484] +2026-09-21T05:11:22.053327Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=72 bs=4 bonus=22484 draft_tokens=[5789, 310, 279] target_tokens=[5789, 310, 1440, 709] accepted=2 new_tokens=[5789, 310, 1440] +2026-09-21T05:11:22.094556Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=73 bs=4 bonus=1440 draft_tokens=[310, 279, 709] target_tokens=[310, 279, 709, 13] accepted=3 new_tokens=[310, 279, 709, 13] +2026-09-21T05:11:22.134401Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=74 bs=4 bonus=13 draft_tokens=[271, 262, 4071] target_tokens=[271, 262, 5019, 198] accepted=2 new_tokens=[271, 262, 5019] +2026-09-21T05:11:22.176861Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=75 bs=4 bonus=5019 draft_tokens=[25, 198, 285] target_tokens=[25, 198, 285, 561] accepted=3 new_tokens=[25, 198, 285, 561] +2026-09-21T05:11:22.215875Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=76 bs=4 bonus=561 draft_tokens=[1067, 314, 279] target_tokens=[1067, 314, 279, 709] accepted=3 new_tokens=[1067, 314, 279, 709] +2026-09-21T05:11:22.254503Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=77 bs=4 bonus=709 draft_tokens=[13, 198, 262] target_tokens=[13, 198, 262, 4071] accepted=3 new_tokens=[13, 198, 262, 4071] +2026-09-21T05:11:22.293919Z DEBUG mlxcel_core::drafter::dflash::round_loop: DFlash round transcript round=78 bs=4 bonus=4071 draft_tokens=[198, 262, 413] target_tokens=[198, 262, 413, 1142] accepted=3 new_tokens=[198, 262, 413] +2026-09-21T05:11:22.294115Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=3.085952 first_bonus_ms=121.52853800000001 first_hidden_ms=0.003072 bind_reset_ms=0.031648 draft_ms=247.51079799999994 verify_ms=200.43085900000003 target_argmax_sync_ms=2359.3300279999994 logprobs_ms=328.33791500000007 walk_ms=0.020400000000000005 hidden_concat_ms=0.12712400000000001 rollback_ms=39.90073200000001 decode_ms=3182.942273 +2026-09-21T05:11:22.294122Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[1, 1, 2, 0, 1, 0, 1, 3, 1, 2, 0, 0, 0, 2, 2, 0, 0, 3, 3, 2, 0, 2, 3, 0, 0, 3, 1, 0, 1, 1, 2, 3, 0, 3, 2, 0, 1, 0, 0, 0, 2, 3, 1, 0, 0, 1, 2, 2, 1, 0, 3, 1, 0, 0, 1, 3, 1, 2, 1, 2, 3, 3, 2, 3, 3, 2, 3, 3, 3, 2, 3, 0, 2, 3, 2, 3, 3, 3, 3] +2026-09-21T05:11:22.294445Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3308 +2026-09-21T05:11:22.294500Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=3308.074992 burst_active_ms=3308.074992 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.wclassic.log b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.wclassic.log new file mode 100644 index 000000000..281557ae4 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/server.wclassic.log @@ -0,0 +1,36 @@ +2026-09-21T05:11:57.305730Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T05:11:57.305772Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T05:11:57.305903Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T05:11:57.305983Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T05:11:57.305987Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T05:11:57.305990Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T05:11:57.633641Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T05:11:57.633673Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T05:11:57.634227Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T05:11:57.967749Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T05:11:57.968296Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T05:11:58.314058Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.346s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.345719487 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T05:11:58.314128Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto) +2026-09-21T05:11:58.314540Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T05:11:58.314553Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T05:11:58.314575Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T05:11:58.315718Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T05:11:59.151928Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 837ms prompt_tokens=1 cached_tokens=0 generation_time_ms=837 +2026-09-21T05:11:59.152184Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T05:11:59.152282Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T05:11:59.152856Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T05:11:59.153653Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18935 +2026-09-21T05:11:59.153675Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T05:11:59.153678Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T05:11:59.153680Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T05:11:59.153681Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T05:11:59.153682Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T05:11:59.153684Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T05:11:59.153685Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T05:11:59.153686Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T05:11:59.153687Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T05:11:59.153688Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T05:11:59.153690Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T05:11:59.153691Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T05:12:05.106418Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 4123ms prompt_tokens=158 cached_tokens=0 generation_time_ms=4123 +2026-09-21T05:12:08.825005Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3715ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3715 diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w2_ids.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w2_ids.json new file mode 100644 index 000000000..b851615a4 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w2_ids.json @@ -0,0 +1 @@ +[285, 4071, 30097, 264, 8240, 314, 30056, 364, 21979, 286, 71483, 198, 285, 30056, 283, 2958, 198, 285, 364, 585, 303, 2020, 1153, 6469, 76753, 1590, 198, 309, 7383, 283, 638, 8658, 21510, 348, 318, 17, 2972, 585, 8, 198, 309, 7383, 283, 1262, 58308, 11, 638, 6469, 21510, 8, 198, 309, 413, 638, 1130, 3143, 25, 198, 388, 7383, 283, 7383, 478, 4055, 34593, 7, 15, 11, 7383, 8, 198, 309, 30056, 1989, 58308, 8, 198, 285, 460, 30056, 1358, 727, 21979, 6406, 3764, 1783, 7, 198, 262, 2821, 11, 198, 262, 1142, 1783, 22064, 25, 6627, 1783, 13429, 283, 2168, 11, 198, 262, 5741, 25, 8132, 12385, 283, 2168, 11, 198, 262, 2972, 9386, 11, 198, 1590, 198, 262, 4071, 49823, 264, 709, 440, 56768, 1142, 1783, 321, 81480, 13, 271, 262, 17167, 25, 198, 285, 2821, 25, 561, 709, 310, 21979, 13, 198, 285, 1142, 1783, 22064, 25, 561, 1142, 1783, 4687, 310, 958, 13, 198, 285, 5741, 25, 561, 5741, 310, 958, 364, 8132, 13, 198, 285, 2972, 9386, 25, 22484, 5789, 310, 1440, 310, 279, 709, 13, 271, 262, 5019, 25, 198, 285, 561, 1067, 314, 279, 709, 13, 198, 262, 4071, 198, 262, 413] \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w4_ids.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w4_ids.json new file mode 100644 index 000000000..b851615a4 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/w4_ids.json @@ -0,0 +1 @@ +[285, 4071, 30097, 264, 8240, 314, 30056, 364, 21979, 286, 71483, 198, 285, 30056, 283, 2958, 198, 285, 364, 585, 303, 2020, 1153, 6469, 76753, 1590, 198, 309, 7383, 283, 638, 8658, 21510, 348, 318, 17, 2972, 585, 8, 198, 309, 7383, 283, 1262, 58308, 11, 638, 6469, 21510, 8, 198, 309, 413, 638, 1130, 3143, 25, 198, 388, 7383, 283, 7383, 478, 4055, 34593, 7, 15, 11, 7383, 8, 198, 309, 30056, 1989, 58308, 8, 198, 285, 460, 30056, 1358, 727, 21979, 6406, 3764, 1783, 7, 198, 262, 2821, 11, 198, 262, 1142, 1783, 22064, 25, 6627, 1783, 13429, 283, 2168, 11, 198, 262, 5741, 25, 8132, 12385, 283, 2168, 11, 198, 262, 2972, 9386, 11, 198, 1590, 198, 262, 4071, 49823, 264, 709, 440, 56768, 1142, 1783, 321, 81480, 13, 271, 262, 17167, 25, 198, 285, 2821, 25, 561, 709, 310, 21979, 13, 198, 285, 1142, 1783, 22064, 25, 561, 1142, 1783, 4687, 310, 958, 13, 198, 285, 5741, 25, 561, 5741, 310, 958, 364, 8132, 13, 198, 285, 2972, 9386, 25, 22484, 5789, 310, 1440, 310, 279, 709, 13, 271, 262, 5019, 25, 198, 285, 561, 1067, 314, 279, 709, 13, 198, 262, 4071, 198, 262, 413] \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/classic_ids.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/classic_ids.json new file mode 100644 index 000000000..c5a446427 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/classic_ids.json @@ -0,0 +1 @@ +[285, 4071, 30097, 264, 8240, 314, 30056, 364, 21979, 286, 71483, 198, 285, 30056, 283, 2958, 198, 285, 364, 585, 303, 2020, 1153, 6469, 76753, 1590, 198, 309, 7383, 283, 638, 8658, 21510, 348, 318, 17, 2972, 585, 8, 198, 309, 7383, 283, 1262, 58308, 11, 638, 6469, 21510, 8, 198, 309, 413, 638, 1130, 3143, 25, 198, 388, 7383, 283, 7383, 478, 4055, 34593, 7, 15, 11, 7383, 8, 198, 309, 30056, 1989, 58308, 8, 198, 285, 460, 30056, 1358, 727, 21979, 6406, 3764, 1783, 7, 198, 262, 2821, 11, 198, 262, 1142, 1783, 22064, 25, 6627, 1783, 13429, 283, 2168, 11, 198, 262, 11439, 198, 262, 5741, 25, 8132, 12385, 283, 2168, 11, 198, 262, 1866, 76753, 25, 514, 283, 2168, 11, 198, 262, 1866, 21510, 25, 2153, 283, 2168, 11, 198, 262, 2255, 21510, 25, 2153, 283, 2168, 11, 198, 262, 81480, 25, 1744, 283, 2168, 11, 198, 1590, 198, 262, 4071, 49823, 264, 709, 440, 56768, 1142, 1783, 321, 81480, 13, 271, 262, 1061, 48973, 369, 5222, 364, 21979, 286, 9830, 6526, 421, 2493, 3564, 4016, 310, 198, 262, 39219, 3790, 4562, 13, 1049, 5533, 56768, 1142, 1783, 440, 81480, 310, 198, 262, 5471, 35622, 286] \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/algebra.py b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/algebra.py new file mode 100755 index 000000000..049184088 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/algebra.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""Issue #1935: check a served DFlash run's round algebra offline, with no GPU. + +Reads the `DFlash round transcript` debug lines from a server log and verifies +the four properties that make the round loop's emitted stream equal to what the +target's caches say it should be: + + 1. every round's bonus is the previous round's last emitted token; + 2. `accepted` is the longest common prefix of the draft and the target argmax; + 3. `new_tokens == draft[:accepted] + [target[accepted]]`, up to the final + round's budget truncation; + 4. the emitted stream is therefore exactly the sequence the caches hold. + +A failure localizes the defect in the round loop or its wrapper. All four +holding puts the defect in the target forward or in the process around it, and +prints the round and row that produced a given emitted index so the next arm has +an anchor. +""" +import argparse +import ast +import json +import re +import sys + +LINE = re.compile( + r"DFlash round transcript\s+round=(\d+)\s+bs=(\d+)\s+bonus=(-?\d+)\s+" + r"draft_tokens=(\[[^\]]*\])\s+target_tokens=(\[[^\]]*\])\s+" + r"accepted=(\d+)\s+new_tokens=(\[[^\]]*\])" +) + + +def parse_rounds(path): + """Every burst in the log, in order. A new burst starts when `round` + restarts at 1, which is how a warm-up request is told from the measured + one without the server having to label them.""" + bursts = [] + cur = [] + for raw in open(path, errors="replace"): + m = LINE.search(raw) + if not m: + continue + r = { + "round": int(m.group(1)), + "bs": int(m.group(2)), + "bonus": int(m.group(3)), + "draft": ast.literal_eval(m.group(4)), + "target": ast.literal_eval(m.group(5)), + "accepted": int(m.group(6)), + "new": ast.literal_eval(m.group(7)), + } + if r["round"] == 0 and cur: + bursts.append(cur) + cur = [] + cur.append(r) + if cur: + bursts.append(cur) + return bursts + + +def lcp(a, b): + n = 0 + while n < len(a) and n < len(b) and a[n] == b[n]: + n += 1 + return n + + +def check(burst, label): + problems = [] + emitted = [burst[0]["bonus"]] # the first bonus the caller already delivered + origin = [("prefill", -1)] + for i, r in enumerate(burst): + if r["bonus"] != emitted[-1]: + problems.append( + f"round {r['round']}: bonus {r['bonus']} is not the previous " + f"round's last emitted token {emitted[-1]}" + ) + if len(r["draft"]) != r["bs"] - 1: + problems.append( + f"round {r['round']}: {len(r['draft'])} proposals for bs={r['bs']}" + ) + if len(r["target"]) != r["bs"]: + problems.append( + f"round {r['round']}: {len(r['target'])} target argmaxes for bs={r['bs']}" + ) + want_acc = lcp(r["draft"], r["target"][: len(r["draft"])]) + if want_acc != r["accepted"]: + problems.append( + f"round {r['round']}: accepted={r['accepted']} but the longest " + f"common prefix of draft and target is {want_acc}" + ) + want_new = r["draft"][: r["accepted"]] + [r["target"][r["accepted"]]] + last = i == len(burst) - 1 + if r["new"] != want_new and not (last and r["new"] == want_new[: len(r["new"])]): + problems.append( + f"round {r['round']}: new_tokens {r['new']} is not " + f"draft[:{r['accepted']}] + [target[{r['accepted']}]] = {want_new}" + ) + for k, tok in enumerate(r["new"]): + emitted.append(tok) + origin.append((f"round {r['round']}", k)) + return emitted, origin, problems + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("log") + ap.add_argument("--burst", type=int, default=-1, help="which burst (default: last)") + ap.add_argument("--reference", help="json file with a classic token id list") + ap.add_argument("--dump", help="write the emitted id list here") + a = ap.parse_args() + + bursts = parse_rounds(a.log) + print(f"[1935] {len(bursts)} burst(s) in {a.log}: " + f"rounds {[len(b) for b in bursts]}") + if not bursts: + return 1 + burst = bursts[a.burst] + emitted, origin, problems = check(burst, a.log) + print(f"[1935] burst {a.burst}: {len(burst)} rounds, {len(emitted)} emitted tokens, " + f"{sum(1 for r in burst if r['accepted'] < r['bs'] - 1)} rewinds") + if problems: + print(f"[1935] ROUND ALGEBRA FAILS, {len(problems)} problem(s):") + for p in problems[:20]: + print(f" - {p}") + else: + print("[1935] round algebra holds: every emitted token is the target's " + "own argmax under a self-consistent cache history") + + if a.dump: + json.dump(emitted, open(a.dump, "w")) + print(f"[1935] wrote {a.dump}") + + if a.reference: + ref = json.load(open(a.reference)) + n = min(len(ref), len(emitted)) + first = next((i for i in range(n) if ref[i] != emitted[i]), -1) + print(f"[1935] reference {len(ref)} ids against emitted {len(emitted)}: " + f"first difference at {first} (-1 means none over {n} compared)") + if first >= 0: + src, row = origin[first] + print(f"[1935] emitted[{first}]={emitted[first]} came from {src} row {row}; " + f"reference has {ref[first]}") + for r in burst: + if f"round {r['round']}" == src: + print(f"[1935] that round: bs={r['bs']} bonus={r['bonus']} " + f"draft={r['draft']} target={r['target']} " + f"accepted={r['accepted']} new={r['new']}") + break + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/compare.py b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/compare.py new file mode 100755 index 000000000..9225ec9ee --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/compare.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Issue #1935: read the served arms against the classic null arm. + +The first question is not where the divergence is but what kind it is, and the +logprobs answer that in one read: + + identical logprobs through the divergence, and the burst's token sitting in + classic's top-k at the SAME logprob -> an exact tie on identical logits, + broken differently by two selection paths; + + identical through, then a fraction apart -> same logits, last-ulp flip, + genuine numerics; + + already drifting before the divergence -> the states or the computation + differ, and the flip is where drift first crossed an argmax; + + a large gap -> state divergence. +""" +import hashlib +import json +import os +import sys + +SP = os.path.dirname(os.path.abspath(__file__)) +ARMS = os.path.join(SP, "arms") + + +def load(tag): + p = os.path.join(ARMS, f"arm.{tag}.json") + if not os.path.exists(p): + return None + return json.load(open(p)) + + +def sha(s): + return hashlib.sha256(s.encode()).hexdigest()[:10] + + +def summarize(tag, rec): + if rec is None: + print(f"[1935] {tag}: missing") + return + if rec.get("error"): + print(f"[1935] {tag}: ERROR {rec['error']}") + for r in rec.get("responses", []): + toks = r.get("tokens") or [] + print(f"[1935] {tag} request {r['index']}: sha {sha(r['text'])}, " + f"{len(r['text'])} chars, {len(toks)} logprob tokens, " + f"usage {r.get('usage')}") + + +def compare(label, ref, spec): + rt = ref["responses"][-1] + st = spec["responses"][-1] + rtok, stok = rt.get("tokens") or [], st.get("tokens") or [] + rlp, slp = rt.get("token_logprobs") or [], st.get("token_logprobs") or [] + n = min(len(rtok), len(stok)) + first = next((i for i in range(n) if rtok[i] != stok[i]), -1) + print(f"\n[1935] {label}: text sha {sha(st['text'])} against classic {sha(rt['text'])}; " + f"{len(stok)} tokens against {len(rtok)}; first differing token index {first}") + if first < 0: + print(f"[1935] {label}: token streams agree over all {n} compared positions") + return + # Do the logprobs agree BEFORE the divergence? That is the question that + # separates a selection difference from accumulated numerical drift. + exact_before = sum(1 for i in range(first) if rlp[i] == slp[i]) + print(f"[1935] {label}: logprobs bit-identical at {exact_before} of the {first} " + f"positions before the divergence") + worst = max(((abs(rlp[i] - slp[i]), i) for i in range(first)), default=(0.0, -1)) + print(f"[1935] {label}: largest pre-divergence logprob gap {worst[0]:.6e} at index {worst[1]}") + lo, hi = max(0, first - 3), min(n, first + 4) + for i in range(lo, hi): + mark = " <<<" if i == first else "" + print(f" [{i}] classic {rtok[i]!r} lp {rlp[i]:.6f} | " + f"burst {stok[i]!r} lp {slp[i]:.6f}{mark}") + for name, t in (("classic", rt), ("burst", st)): + top = (t.get("top_logprobs") or [None] * n)[first] + print(f" top-k at {first} ({name}): {top}") + + +def main(): + ref = load("wclassic") + summarize("wclassic", ref) + for tag in ("w4", "w2"): + rec = load(tag) + summarize(tag, rec) + if ref and rec and ref.get("responses") and rec.get("responses"): + compare(tag, ref, rec) + # Cross-width: the record's strongest structural clue is that these two + # agree despite entirely different round structures. + a, b = load("w4"), load("w2") + if a and b and a.get("responses") and b.get("responses"): + same = a["responses"][-1]["text"] == b["responses"][-1]["text"] + print(f"\n[1935] width 4 and width 2 texts identical: {same}") + # Within-arm: burst 1 (warm-up) against burst 2 (measured). A difference + # here would be cross-request state, which no arm has looked for. + for tag in ("wclassic", "w4", "w2"): + rec = load(tag) + if rec and len(rec.get("responses", [])) >= 2: + same = rec["responses"][0]["text"] == rec["responses"][1]["text"] + print(f"[1935] {tag}: request 0 and request 1 identical: {same}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/transcript.py b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/transcript.py new file mode 100755 index 000000000..8aa883239 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/transcript.py @@ -0,0 +1,199 @@ +#!/usr/bin/env python3 +"""Issue #1935: capture one served arm's token transcript AND its per-round +DFlash round-loop transcript, so the round algebra can be checked offline. + +One server per arm, `--ignore-eos --max-batch-size 1`, non-streaming +`/v1/completions` at temperature 0 with `logprobs`, repeated `--n` times +against the SAME server process so a cross-request state difference (burst 1 +against burst 2) shows up as two different token lists rather than as one +number. + +Identity only: nothing here is a timing measurement, so the host gate is the +foreign-model check alone. +""" +import argparse +import json +import os +import signal +import subprocess +import sys +import time +import urllib.error +import urllib.request + +FOREIGN = ("mlxcel", "mlxcel-server", "python3", "python") + + +def foreign_model_procs(): + """Processes that could be holding the GPU, matched on /proc//comm + exactly and with stopped ones dropped. Our own renamed server is excluded + by construction (its comm is mlxcel1935-serv).""" + hits = [] + for pid in os.listdir("/proc"): + if not pid.isdigit(): + continue + try: + comm = open(f"/proc/{pid}/comm").read().strip() + if comm not in ("mlxcel", "mlxcel-server"): + continue + state = open(f"/proc/{pid}/stat").read().rsplit(")", 1)[1].split()[0] + if state == "T": + continue + cwd = os.readlink(f"/proc/{pid}/cwd") + hits.append((pid, comm, cwd)) + except OSError: + continue + return hits + + +def wait_health(port, timeout=900): + t0 = time.time() + while time.time() - t0 < timeout: + try: + with urllib.request.urlopen(f"http://127.0.0.1:{port}/health", timeout=5) as r: + if r.status == 200: + return True + except Exception: + time.sleep(2) + return False + + +def first_model_id(port, timeout=30): + t0 = time.time() + while time.time() - t0 < timeout: + try: + with urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/models", timeout=5) as r: + data = json.load(r) + return data["data"][0]["id"] + except Exception: + time.sleep(1) + raise RuntimeError("no model id") + + +def complete(port, prompt, max_tokens, model, timeout=1800, want_logprobs=True): + payload = { + "model": model, + "prompt": prompt, + "max_tokens": max_tokens, + "temperature": 0, + "stream": False, + } + if want_logprobs: + payload["logprobs"] = 5 + body = json.dumps(payload).encode() + req = urllib.request.Request( + f"http://127.0.0.1:{port}/v1/completions", + data=body, + headers={"Content-Type": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.load(r) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--server", required=True) + ap.add_argument("--target", required=True) + ap.add_argument("--drafter") + ap.add_argument("--width", default="classic") + ap.add_argument("--n", type=int, default=2) + ap.add_argument("--max-tokens", type=int, default=200) + ap.add_argument("--prompt-file", required=True) + ap.add_argument("--port", type=int, default=18935) + ap.add_argument("--outdir", required=True) + ap.add_argument("--tag", default=None) + ap.add_argument("--extra-env", default="", help="k=v,k=v applied to the server process") + a = ap.parse_args() + + tag = a.tag or f"w{a.width}" + os.makedirs(a.outdir, exist_ok=True) + prompt = open(a.prompt_file).read() + + foreign = foreign_model_procs() + if foreign: + print(f"[1935] foreign model processes present, refusing: {foreign}", file=sys.stderr) + return 2 + + run_dir = os.path.join(a.outdir, "bin") + os.makedirs(run_dir, exist_ok=True) + run_bin = os.path.join(run_dir, "mlxcel1935-server") + if not os.path.exists(run_bin) or os.path.getmtime(run_bin) < os.path.getmtime(a.server): + subprocess.run(["/bin/cp", "-f", a.server, run_bin], check=True) + os.chmod(run_bin, 0o755) + + cmd = [run_bin, "-m", a.target, "--port", str(a.port), "--ignore-eos", + "--max-batch-size", "1"] + if a.width != "classic": + cmd += ["--draft-model", a.drafter, "--draft-kind", "dflash", + "--draft-block-size", str(a.width)] + + env = dict(os.environ) + env.setdefault("MLX_ENABLE_TF32", "1") + env.setdefault("MLX_CUDA_ARCHITECTURES", "121") + env["RUST_LOG"] = ( + "info,mlxcel_core::drafter::dflash::round_loop=debug," + "mlxcel::server::batch::dflash_target=debug" + ) + for pair in filter(None, a.extra_env.split(",")): + k, _, v = pair.partition("=") + env[k] = v + + log_path = os.path.join(a.outdir, f"server.{tag}.log") + log = open(log_path, "w") + srv = subprocess.Popen(cmd, env=env, stdout=log, stderr=subprocess.STDOUT, + start_new_session=True) + rec = {"tag": tag, "width": a.width, "cmd": cmd, "log": log_path, + "extra_env": a.extra_env, "responses": []} + try: + if not wait_health(a.port): + rec["error"] = "server never became healthy" + else: + model_id = first_model_id(a.port) + rec["model_id"] = model_id + for i in range(a.n): + try: + resp = complete(a.port, prompt, a.max_tokens, model_id) + except urllib.error.HTTPError as exc: + # A server that refuses `logprobs` must not cost the arm: + # the round transcript carries the ids either way, and the + # record says omitting logprobs does not move the output. + print(f"[1935] logprobs refused ({exc.code}), retrying without", + file=sys.stderr) + resp = complete(a.port, prompt, a.max_tokens, model_id, + want_logprobs=False) + choice = resp["choices"][0] + lp = choice.get("logprobs") or {} + rec["responses"].append( + { + "index": i, + "text": choice["text"], + "tokens": lp.get("tokens"), + "token_logprobs": lp.get("token_logprobs"), + "usage": resp.get("usage"), + } + ) + print(f"[1935] {tag} request {i}: " + f"{len(lp.get('tokens') or [])} tokens, " + f"{len(choice['text'])} chars", file=sys.stderr) + except Exception as exc: # noqa: BLE001 + rec["error"] = f"{type(exc).__name__}: {exc}" + finally: + try: + os.killpg(srv.pid, signal.SIGTERM) + srv.wait(timeout=120) + except Exception: + try: + os.killpg(srv.pid, signal.SIGKILL) + except Exception: + pass + log.close() + + out = os.path.join(a.outdir, f"arm.{tag}.json") + with open(out, "w") as f: + json.dump(rec, f, indent=1) + print(f"[1935] wrote {out}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/identity.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/identity.txt new file mode 100644 index 000000000..d471f1f9b --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/identity.txt @@ -0,0 +1,20 @@ +date: 2026-09-21T14:10:24+09:00 +host: spark-101 +kernel: 7.0.0-1019-nvidia +arch: aarch64 +nvidia_driver: 580.178.04 +gpu: NVIDIA GB10, 12.1 +MLX_CUDA_ARCHITECTURES: 121 +cuda_toolkit: 13.0 +mlx_pin: 81ba1c6a0e50a9268b931579c2d4f1158b9aab5a +rustc_on_path: rustc 1.97.1 (8bab26f4f 2026-07-14) +rustc_used_for_build: rustc 1.97.1 (8bab26f4f 2026-07-14) +git_commit: 5cc25ccaddfbdea5f221782bb14c514e0e6faeee +git_dirty: no +uptime: up 22 hours, 15 minutes +mem_total_gib: 121.7 +binary: /home/inureyes/Development/mlxcel/target/release/mlxcel-server +binary_sha256: cce6fdeaea36adfb8920e523b60768bc3609af7f308695f4f57e460ef4e108a0 +binary_mtime: 2026-09-21T14:05:39+09:00 +model models/mlx/qwen3.5-4b-4bit: bytes=3061131800 model_type=qwen3_5 quantization=affine +model models/mlx/qwen3.5-4b-dflash: bytes=1074861667 model_type=qwen3 quantization=none diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/prompt_ids.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/prompt_ids.json new file mode 100644 index 000000000..dc991a41b --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/prompt_ids.json @@ -0,0 +1 @@ +[12237, 49823, 29202, 440, 56768, 1142, 1783, 321, 81480, 13, 271, 35184, 12217, 748, 424, 628, 381, 18109, 2922, 1083, 2526, 3401, 421, 1902, 524, 198, 63982, 264, 4688, 23280, 15808, 1066, 310, 21979, 449, 9830, 1562, 13, 198, 12237, 271, 464, 54572, 198, 464, 39965, 198, 464, 8132, 198, 464, 4055, 198, 464, 854, 198, 1445, 795, 8588, 1120, 795, 1005, 271, 9494, 283, 8132, 13286, 5340, 591, 18562, 271, 16775, 11425, 32304, 283, 220, 15, 13, 16, 198, 16775, 6592, 32304, 283, 220, 18, 15, 13, 15, 198, 16775, 6592, 8403, 20376, 71188, 283, 220, 20, 1358, 31, 673, 1005, 925, 40838, 3497, 8, 198, 1005, 6627, 1783, 13429, 25, 198, 262, 2255, 21510, 25, 2153, 283, 11616, 11425, 32304, 198, 262, 1866, 21510, 25, 2153, 283, 11616, 6592, 32304, 198, 262, 1866, 76753, 25, 514, 283, 11616, 6592, 8403, 20376, 71188, 198, 262, 81480, 25, 1744, 283, 2912, 271, 262, 687, 30056, 1153, 1590, 198] \ No newline at end of file From 4e99a6b2bd0e134dfd4e1eb872903fe346da59e4 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:22:03 +0900 Subject: [PATCH 03/16] docs(benchmarks): record what the served arms settle about the width 2 and 4 residual The host, the method and the part of issue #1935's residual the served arms answer on their own: the round algebra holds in all four speculative bursts, the residual does not track round structure, and the logprobs place the difference just below the reporting floor rather than at a rollback or a round boundary. The sections that name a cause are deliberately absent until the in-process arms run. What is here is what the data supports without them. Refs #1935 --- ...dth-2-4-residual-qwen35-gb10-2026-09-21.md | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md new file mode 100644 index 000000000..990a5ae39 --- /dev/null +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -0,0 +1,46 @@ +# Qwen 3.5 DFlash: the residual at verify widths 2 and 4 (GB10, 2026-09-21) + +Issue #1935, continuing from `dflash-verify-divergence-qwen35-gb10-2026-09-20.md` (PR #1939). That record fixed the burst's prompt prefill, which had been computed as if it were a verify block, and left one symptom open: at widths 2 and 4 the served greedy completion still parts from classic decode, at generated token 105 rather than token 9. + +This record answers what that residual is. + +## Host + +``` +host: spark-101 +kernel: 7.0.0-1019-nvidia +arch: aarch64 +nvidia_driver: 580.178.04 +gpu: NVIDIA GB10, compute capability 12.1 +cuda_toolkit: 13.0 +mlx_pin: 81ba1c6a0e50a9268b931579c2d4f1158b9aab5a +rustc: 1.97.1 (8bab26f4f 2026-07-14), the version rust-toolchain.toml pins +MLX_CUDA_ARCHITECTURES: 121 +mem_total_gib: 121.7 +``` + +Driver budget: the boot's cumulative kernel `NV_ERR_NO_MEMORY` count was 0 before this session. Nothing here is a timing measurement, so every arm ran while the host was also compiling; greedy token streams do not depend on host load, and the two classic arms below are the control that says so. + +## Method + +Served arms: one `mlxcel-server` per arm, `--ignore-eos --max-batch-size 1`, the #1797 harness's 158-token Python prompt, two non-streaming `POST /v1/completions` per arm at `temperature 0` with `max_tokens 200` and `logprobs`, both against the same server process so a cross-request difference would show as two different token lists. In-process arms: `#[ignore]`-gated tests in `src/models/qwen3_5_dflash_probe_tests.rs`. + +The classic reference is anchored on this binary: `MLXCEL_PRINT_TOKEN_IDS=1 mlxcel generate --no-chat-template --temp 0 --max-tokens 200` reproduces the served classic completion exactly and supplies the 158 prompt ids and 200 generated ids the in-process arms take. One trap worth recording: the prompt file ends in a newline and shell command substitution strips it, which tokenizes to 157 ids instead of 158 and shifts the whole reference by one token. + +Data and harness: `data/dflash-width-2-4-residual-gb10-2026-09-21/`. + +## What the served arms establish + +| arm | completion sha256 | tokens | rounds | rewinds | first token differing from classic | +|---|---|---:|---:|---:|---:| +| classic (twice) | `2c76b0a181` | 200 | | | | +| width 4 (twice) | `3e60b1574c` | 200 | 79 | 56 | 105 | +| width 2 (twice) | `3e60b1574c` | 200 | 114 | 28 | 105 | + +**The round algebra holds, in all four speculative bursts.** Checked offline from a per-round `debug` transcript this change adds, with no GPU and nothing replayed: every round's bonus is the previous round's last emitted token, `accepted` is the longest common prefix of the drafter's proposals and the target's block argmax, and the emitted tokens are exactly `draft[:accepted] + [target[accepted]]`. So every token the burst emits is the target's own argmax under a cache history consistent with the tokens already emitted, and neither the round loop nor the burst wrapper is mis-bookkeeping anything. This is what makes the rest of the record a question about the target's forward rather than about the loop around it. + +**The residual does not track round structure.** Width 4 runs 79 rounds with 56 rewinds, width 2 runs 114 with 28, and the two emit the same 200 ids. At the divergence the two arms even reach it at different block positions: width 4 at row 2 of the block `[11, 198, 262, 1866]`, width 2 at row 1 of `[198, 262]`. Both take 5741 there; classic takes 11439. + +**The logprobs place it just below the reporting floor.** Reported logprobs on this checkpoint are quantized to 0.125. Of the 105 positions before the divergence, 100 are bit-identical between classic and both speculative arms, and the five that differ do so by exactly one step, at the same five indices (62, 68, 86, 88, 96) with the same values at both widths. At 105 itself classic's 11439 and the burst's 5741 both report -2.125. + +That pattern is a difference in the logits that is below the reporting floor almost everywhere, surfacing as a one-step logprob difference from index 62 onward and finally crossing an argmax at 105, where the top two candidates sat within a reporting step of each other. It is not a rollback, not a round-boundary effect, and not nondeterminism: both requests in each arm are byte-identical to each other. From 76c5121fae46fe7aea0ee9848c4fe448335f3675 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:30:28 +0900 Subject: [PATCH 04/16] fix(speculative): give a Qwen 3.5 verify row the query layout decode hands the kernel `Qwen3NextAttention::attend_per_position` makes a verify block reproduce single-token decode by attending one query position at a time, each to its own causal prefix. The arithmetic is right and the layout is not: a one-row slice of a `[B, H, T, D]` block has its heads `T * D` apart, while single-token decode hands the same attention call a freshly built `[B, H, 1, D]` whose heads are `D` apart. Same values, same shape, different strides, and on CUDA this family's `head_dim` 256 reaches the fused `sdpa_vector` kernel (issue #675), which is not bit-equal across the two. That is why a served greedy DFlash burst parted from classic decode at every verify width alike. The key and value slices need no copy: both paths slice the same cache buffer to `prefix + i + 1`, so their strides already match. The query row is now copied to the decode layout, under `MLXCEL_QWEN35_ATTEND_CONTIGUOUS` so the pre-fix slicing is one environment variable away for an A/B. The copy is `[B, H, 1, D]` per row per attention layer. Three arms identify the mechanism, each ruling out what the others cannot. At `T = 1` the speculative and classic forwards agree in all 496640 logit bytes at every one of 200 steps, which is where the one-row slice IS the whole tensor and the layouts coincide; that rules out the projections, the gated-delta layers, the prefill and the caches. At `T = 4` the divergence reproduces in process with no drafter and no server, at exactly the served position, which kills the standing hypothesis that the drafter's interleaved work was the cause. And `MLXCEL_SDPA_VECTOR_LARGE_D=0`, which routes both paths off the fused kernel, makes classic and width 4 byte-identical over all 200 tokens. Record and raw data: `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. The served A/B of this change, and the probe's own false pass at widths 2 and 4, land next. Refs #1935 --- .../arms/accepts_w2.txt | 1 + .../arms/accepts_w4.txt | 1 + .../arms/arm.cls-novec.json | 435 +++++++++++++++++ .../arms/arm.w4-novec.json | 441 ++++++++++++++++++ .../harness/ab_sdpav.sh | 24 + .../probe_arms.txt | 22 + .../probe_verdicts.txt | 5 + ...dth-2-4-residual-qwen35-gb10-2026-09-21.md | 29 ++ docs/environment-variables.md | 1 + src/models/qwen3_next.rs | 72 +++ 10 files changed, 1031 insertions(+) create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w2.txt create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.cls-novec.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4-novec.json create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_arms.txt create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_verdicts.txt diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w2.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w2.txt new file mode 100644 index 000000000..5a47110b9 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w2.txt @@ -0,0 +1 @@ +1,1,2,2,1,1,2,1,2,2,2,2,2,1,2,1,2,2,2,1,1,2,2,2,2,2,2,2,1,2,2,1,1,1,2,2,2,1,1,2,2,1,2,2,1,2,2,2,1,2,2,1,1,2,2,2,2,1,1,2,1,2,2,2,2,1,2,2,2,1,1,2,2,2,2,2,2,2,1,2,2,2,2,2,2,2,2,2,2,2,2,2,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2 \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt new file mode 100644 index 000000000..44095ab5f --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt @@ -0,0 +1 @@ +2,2,3,1,2,1,2,4,2,3,1,1,1,3,3,1,1,4,4,3,1,3,4,1,1,4,2,1,2,2,3,4,1,4,3,1,2,1,1,1,3,4,2,1,1,2,3,3,2,1,4,2,1,1,2,4,2,3,2,3,4,4,3,4,4,3,4,4,4,3,4,1,3,4,3,4,4,4,4 \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.cls-novec.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.cls-novec.json new file mode 100644 index 000000000..88ff8efc2 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.cls-novec.json @@ -0,0 +1,435 @@ +{ + "tag": "cls-novec", + "width": "classic", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18936", + "--ignore-eos", + "--max-batch-size", + "1" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.cls-novec.log", + "extra_env": "MLXCEL_SDPA_VECTOR_LARGE_D=0", + "responses": [ + { + "index": 0, + "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance.", + "tokens": [ + " ", + " \"\"\"", + "Y", + "ield", + " delays", + " between", + " attempts", + ".\"\"\"", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + "\n", + " ", + " for", + " _", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " yield", + " delay", + "\n", + " ", + " if", + " delay", + " >=", + " self", + ".max", + "_delay", + ":", + "\n", + " ", + " break", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing", + " herd", + " problems", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " A", + " Back", + "off", + "Policy", + " instance", + "." + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.5, + -0.125, + -1.25, + -1.375, + -0.625, + -0.625, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + -0.75, + -0.25, + 0.0, + -0.625, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.375, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + -0.75, + -0.75, + -0.5, + -0.5, + -0.125, + -0.125, + 0.0, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + -0.375, + 0.0, + -0.25, + 0.0, + -0.5, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + -0.875, + -0.375, + -1.125, + -1.0, + -0.5, + 0.0, + -1.25, + 0.0, + -0.375, + -1.375, + -0.125, + -0.125, + 0.0, + -1.625, + 0.0, + -0.875, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.75, + -0.5, + -0.125, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.5, + -0.125, + -0.625, + -0.125, + 0.0, + 0.0, + -0.375, + -0.25, + -0.125, + 0.0, + 0.0, + -1.75, + -1.125, + -1.875, + -1.5, + -0.625, + -0.875, + 0.0, + -0.875, + -0.375, + -0.5, + -0.75, + -0.25, + -0.625, + -0.125, + 0.0, + 0.0, + -0.625, + -0.875, + -0.25, + -0.75, + -0.75, + -0.75, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + -0.125, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0, + -0.125, + -0.125, + -0.125, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + -0.125, + 0.0, + 0.0, + -0.25, + -1.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4-novec.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4-novec.json new file mode 100644 index 000000000..fa8871774 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.w4-novec.json @@ -0,0 +1,441 @@ +{ + "tag": "w4-novec", + "width": "4", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18936", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "4" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.w4-novec.log", + "extra_env": "MLXCEL_SDPA_VECTOR_LARGE_D=0", + "responses": [ + { + "index": 0, + "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance.", + "tokens": [ + " ", + " \"\"\"", + "Y", + "ield", + " delays", + " between", + " attempts", + ".\"\"\"", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + "\n", + " ", + " for", + " _", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " yield", + " delay", + "\n", + " ", + " if", + " delay", + " >=", + " self", + ".max", + "_delay", + ":", + "\n", + " ", + " break", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing", + " herd", + " problems", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " A", + " Back", + "off", + "Policy", + " instance", + "." + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.5, + -0.125, + -1.25, + -1.375, + -0.625, + -0.625, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + -0.75, + -0.25, + 0.0, + -0.625, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.375, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + -0.75, + -0.75, + -0.5, + -0.5, + -0.125, + -0.125, + 0.0, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + -0.375, + 0.0, + -0.25, + 0.0, + -0.5, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + -0.875, + -0.375, + -1.125, + -1.0, + -0.5, + 0.0, + -1.25, + 0.0, + -0.375, + -1.375, + -0.125, + -0.125, + 0.0, + -1.625, + 0.0, + -0.875, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.75, + -0.5, + -0.125, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.5, + -0.125, + -0.625, + -0.125, + 0.0, + 0.0, + -0.375, + -0.25, + -0.125, + 0.0, + 0.0, + -1.75, + -1.125, + -1.875, + -1.5, + -0.625, + -0.875, + 0.0, + -0.875, + -0.375, + -0.5, + -0.75, + -0.25, + -0.625, + -0.125, + 0.0, + 0.0, + -0.625, + -0.875, + -0.25, + -0.75, + -0.75, + -0.75, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + -0.125, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0, + -0.125, + -0.125, + -0.125, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + -0.125, + 0.0, + 0.0, + -0.25, + -1.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh new file mode 100755 index 000000000..9dd5d4b46 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# A/B on MLXCEL_SDPA_VECTOR_LARGE_D. Qwen 3.5's head_dim is 256, so this gate +# decides whether a single-query attention call takes the fused sdpa_vector +# kernel or the materializing fallback. Classic decode and the burst's +# attend_per_position are both single-query calls, so if the residual is the +# two of them landing on different kernels, turning the gate off puts both on +# the fallback and they converge. If they still differ, the kernel choice is +# not the mechanism. +set -uo pipefail +SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad +WT=/home/inureyes/Development/mlxcel-wt-1935 +BIN=/home/inureyes/Development/mlxcel/target/release/mlxcel-server +H=$WT/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness +for arm in "classic:cls-novec" "4:w4-novec"; do + W=${arm%%:*}; T=${arm##*:} + python3 "$SP/transcript.py" --server "$BIN" \ + --target "$WT/models/mlx/qwen3.5-4b-4bit" \ + --drafter "$WT/models/mlx/qwen3.5-4b-dflash" \ + --width "$W" --n 1 --max-tokens 200 --tag "$T" \ + --extra-env "MLXCEL_SDPA_VECTOR_LARGE_D=0" \ + --prompt-file "$H/prompt_retry.txt" --outdir "$SP/arms" --port 18936 + echo "=== $T done ===" +done +echo "AB DONE" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_arms.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_arms.txt new file mode 100644 index 000000000..a805bb7d5 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_arms.txt @@ -0,0 +1,22 @@ +=== ARM 3: speculative against classic at ONE ROW, logit BYTES === +[1935] prefill last row: 0 of 496640 logit bytes differ +[1935] prefill: every layer's cache state is byte-identical +[1935] step 0: 0 of 496640 logit bytes differ +[1935] step 50: 0 of 496640 logit bytes differ +[1935] step 100: 0 of 496640 logit bytes differ +[1935] step 150: 0 of 496640 logit bytes differ +[1935] 0 of 200 one-row steps differ, first at -1 (-1 means none), worst 0 bytes +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 8634 filtered out; finished in 10.38s +=== ARM 1: replay the served width-4 round structure, no drafter === +[1935] capture layers [1, 8, 15, 22, 29], block argmax false +[1935] rejected rows carry the wrong token id 9999 +[1935] 79 rounds, 56 rollbacks; greedy argmax over 201 compared positions: 1 disagree, first at 105 (-1 means none) +[1935] at position 105: classic 11439 against burst 5741 +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 8634 filtered out; finished in 8.05s +=== ARM 2: the served burst wrapper with the real drafter === +[1935] models/mlx/qwen3.5-4b-4bit + models/mlx/qwen3.5-4b-dflash, block 4: burst emitted 200 ids against 200 reference; 95 disagree over 200 compared, first at 105 (-1 means none) +[1935] around 105: reference [11, 198, 262, 11439, 198, 262, 5741] against burst [11, 198, 262, 5741, 25, 8132, 12385] +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 8634 filtered out; finished in 12.98s +ALL PROBES DONE + +[exited with code 0] diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_verdicts.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_verdicts.txt new file mode 100644 index 000000000..5f7c55b10 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_verdicts.txt @@ -0,0 +1,5 @@ +[1935] width 2: verify block is byte-identical to the single-token chain +[1935] width 4: verify block is byte-identical to the single-token chain +[1935] width 8: verify block position 0 differs from the single-token chain in 229971 of 496640 logit bytes +[1935] width 16: verify block position 0 differs from the single-token chain in 229971 of 496640 logit bytes +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 8634 filtered out; finished in 3.77s diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index 990a5ae39..fbb72f8b4 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -44,3 +44,32 @@ Data and harness: `data/dflash-width-2-4-residual-gb10-2026-09-21/`. **The logprobs place it just below the reporting floor.** Reported logprobs on this checkpoint are quantized to 0.125. Of the 105 positions before the divergence, 100 are bit-identical between classic and both speculative arms, and the five that differ do so by exactly one step, at the same five indices (62, 68, 86, 88, 96) with the same values at both widths. At 105 itself classic's 11439 and the burst's 5741 both report -2.125. That pattern is a difference in the logits that is below the reporting floor almost everywhere, surfacing as a one-step logprob difference from index 62 onward and finally crossing an argmax at 105, where the top two candidates sat within a reporting step of each other. It is not a rollback, not a round-boundary effect, and not nondeterminism: both requests in each arm are byte-identical to each other. + +## The cause: a verify row and the decode step it stands for are not the same call + +`Qwen3NextAttention::attend_per_position` is what makes a verify block reproduce single-token decode. For each query position `i` of a `[B, H, T, D]` block it attends `queries[:, :, i:i+1, :]` to the causal prefix `keys[:, :, ..prefix + i + 1, :]` with no mask, so each row sees exactly the prefix a decode step would. The arithmetic is right. The layout is not. + +A one-row slice of a `[B, H, T, D]` tensor has its heads `T * D` apart. Single-token decode hands the same attention call a freshly built `[B, H, 1, D]`, whose heads are `D` apart. Same values, same shape, different strides. On CUDA this checkpoint's `head_dim` is 256, which `MLXCEL_SDPA_VECTOR_LARGE_D` (issue #675) routes to the fused `sdpa_vector` kernels, and that kernel is not bit-equal across the two layouts. The key and value slices are unaffected: both paths slice the same cache buffer to `prefix + i + 1`, so their strides already match, which is why the query row is the whole of it. + +Three arms establish this, and each rules out what the others cannot. + +**At `T = 1` the two forwards are byte-identical.** Prefilling classic with `forward_internal` and the burst with `forward_prefill_with_capture_layers`, then walking the real 200-token transcript one token at a time, every step's logits agree in all 496640 bytes and the prefill's do too. At `T = 1` the one-row slice IS the whole tensor, so the layouts coincide and the difference has nowhere to appear. This is what rules out everything that is not a block: the projections, the gated-delta layers, the rotary, the prefill, the caches. + +**At `T = 4` it reproduces with no drafter at all.** Replaying the served width-4 run's own 79 rounds and 56 rewinds in process, with the rejected rows carrying a wrong token id, the target disagrees with the classic chain at exactly one of 201 greedy positions, at 105, taking 5741 where classic takes 11439. The served position, from a replay that contains no drafter and no server. That kills the standing hypothesis that the drafter's interleaved MLX work was the cause: there is no drafter in this arm. Driving `run_dflash_on_target` with the real drafter bound reproduces the same position, which is consistent rather than additional. + +**Turning the fused kernel off makes the two byte-identical.** `MLXCEL_SDPA_VECTOR_LARGE_D=0` routes single-query attention off `sdpa_vector` for `head_dim` 256 on both paths. Served, same binary: + +| arm | completion sha256 | first token differing from classic | +|---|---|---:| +| classic, stock | `2c76b0a181` | | +| width 4, stock | `3e60b1574c` | 105 | +| classic, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | | +| width 4, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | none, 200 of 200 | + +Both arms move, which is expected and is why the row is not evidence about classic decode; what matters is that they move onto each other. A single kill switch collapsing a 95-token divergence to nothing is stronger than any of the exclusions that preceded it, and it takes the remaining candidates with it: rollback, the gated-delta scan, the quantized matmul kernel, the drafter, the round loop, the server process. None of those is disabled by this switch, and all of them stop mattering when it is off. + +## Why the gate reported the property intact + +`Qwen35Model::probe_block_chain_exactness` compares a verify block against a single-token chain in logit bytes, which is the right comparison. It prefills 8 synthetic tokens first: `PROBE_PROMPT_LEN` is 8, chosen so the attention layers "hold a real KV prefix rather than the empty-cache special case". An 8-token prefix is not a served KV layout. The cache has not grown past its first step-aligned allocation, and the difference this record is about does not appear there. + +Measured on the real checkpoint on this host, the probe reports byte-identity at widths 2 and 4 and divergence at 8 and 16, which is what PR #1939 recorded and wired the gate to. The served arms disagree with it at widths 2 and 4. So the gate's verdict was a false pass, not a correct pass that something downstream then violated, and the probe needs a prefix long enough to reproduce the layout a served request has before its verdict means anything. diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 82ba324d4..fd47cdcba 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -462,6 +462,7 @@ recommended as normal deployment settings. | `MLXCEL_SPARSE_PAGED_DUMP` | `1`/`true`/`on`/`yes` enable | off | Prints each request's selected pool rows for a fused sparse decode (#904). **Synchronizes with the device**, so it is a debugging aid only and must never be set for a timed run. | | `MLXCEL_SPARSE_PAGED_MIN_SPARSITY` | non-negative integer | `8` | Minimum `live_len / selected_rows` ratio a launch must clear before fused sparse decode is dispatched (#904). The fused kernel measured 0.67x at 2x sparsity and 1.22x at 8x against tuned dense SDPA, so a token floor alone is not sufficient. `0` disables the gate for benchmarking the declined regime; an unparseable value falls back to the default. See [`sparse-paged-decode.md`](sparse-paged-decode.md). | | `MLXCEL_SDPA_VECTOR_LARGE_D` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | **CUDA only.** Gates whether the CUDA `supports_sdpa_vector` check accepts head_dim 256/288 (gemma family, qwen3.5/3.6, baichuan-m1, paligemma2), routing their decode to the fused `sdpa_vector` kernels instead of the materializing SDPA fallback (issue #675). Disabling restores the prior fallback with no rebuild; used for the A/B in `benchmarks/cuda_gb10_sdpav_675_2026-07-06.csv`. | +| `MLXCEL_QWEN35_ATTEND_CONTIGUOUS` | `q` (default), `kv`, `all`, or `0`/`off`/`false`/`no`/`none` to disable | `q` | **Qwen 3.5 family, speculative verify only.** Which of `Qwen3NextAttention::attend_per_position`'s per-row tensors are copied to the layout single-token decode hands the same attention call (issue #1935). A verify block attends one query position at a time so each row reproduces the decode step it stands for, but a one-row slice of a `[B, H, T, D]` block has its heads `T * D` apart while decode hands the call a `[B, H, 1, D]` whose heads are `D` apart. Same values, different strides, and on CUDA the fused `sdpa_vector` kernel that `MLXCEL_SDPA_VECTOR_LARGE_D` routes `head_dim` 256/288 to is not bit-equal across them, which is how a served greedy burst parted from classic decode at every verify width alike. `q` copies the query row, which is `[B, H, 1, D]` per row per attention layer and is the shipped fix. The key and value slices already match between the two paths (both slice the same cache buffer to `prefix + i + 1`), so `kv` and `all` exist for the A/B that established that and cost one prefix-sized copy per row. `0` restores the pre-#1935 slicing without a rebuild and is the kill switch the record's A/B uses. Read once per process. See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` | non-negative integer | `0` (off) | **CUDA only. Off by default; set to `32` to enable.** #1820 measured this path behind `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s ops fallback at three of five block widths with disjoint ranges (0.943x, 0.953x, 0.915x at widths 2, 8 and 16 on a 152-token prompt) and a wash at 2634 tokens, so the shipped dispatch is #1799's alone and this is opt-in. It is kept rather than removed because the mechanism is sound (12 plan builds per generation instead of 82, greedy token ids byte-identical to exact-shape cuDNN) and a crossover above 2634 keys is plausible and unmeasured: this path's cost is flat in context while the fallback's score matrix grows with the key length. See `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-17.md`. An array-masked SDPA call with 2 to N query rows over a KV cache (the speculative verify shape: a block appended to a cache) has its cuDNN execution-plan cache key bucketed, so one plan serves every round until the cache buffer grows (issue #1820). cuDNN keys that cache on the exact shapes and strides of q, k, v and the mask, and a verify round's key length is new every round, so before this every round of every attention-layer class rebuilt a plan on the host: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing. The fix is MLX's own one-row decode canonicalization extended to these calls: k and v are unsliced to the whole cache buffer, the additive mask is widened to the same width with the new columns set to `-inf`, and the true lengths reach cuDNN through `set_padding_mask` with `set_seq_len_q` / `set_seq_len_kv`, which is what keeps the widened region out of the result. k and v reach the bucket two ways. Where they are already a leading slice of one contiguous cache buffer, they are unsliced to that buffer's own allocated extent, so nothing is reallocated and no position outside the already-allocated cache is addressed; mlxcel's KV cache grows in 256-position steps, so one plan per class covers 256 appends. Where they are not (the drafter concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room past the live length) they are copied into a zero-padded buffer rounded up to the same 256 grid, which costs a copy per call and is bounded per tensor by `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB`. Above that bound the call keeps its exact shape and pays the plan build, and a shape-eligible call whose layout declines bucketing for any reason prints a one-time warning naming this variable, because such a call gets neither this fix nor `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s. `0` disables the bucketing without a rebuild, which restores the dispatch `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` alone produced and is the kill switch used for the A/B in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md`. Non-CUDA builds ignore it. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB` | non-negative integer (MiB) | `64` | **CUDA only.** Per-tensor ceiling on the zero-padded k/v copy the bucketing arm above makes when k and v are not a leading slice of one contiguous cache buffer, which is the drafter's shape: it concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room to unslice into (issue #1820). A call whose widened k or v would exceed this keeps its exact shape and rebuilds a plan each round, and prints the one-time warning described above, because moving hundreds of MiB per layer per round would cost more than the roughly 22 ms plan build it saves. At the default 64 MiB an 8-kv-head 128-dim bf16 cache buckets up to about 32k keys. `0` disables the copy arm, leaving only the free unslice arm. | | `MLXCEL_SDPA_PLAN_DEBUG` | `1` enables | off (`0`) | **CUDA only, diagnostic.** Writes one line per cuDNN SDPA call to stderr with the key fields that decide plan reuse: the q shape, `k_len`, the cache buffer extent and row stride, the mask column count and strides, the causal and sinks flags, whether the call took the decode or the bucketed canonicalization, and whether this call built a plan or reused one, plus the resident plan count. This is how the per-shape-class key-field table and the plan-build counts in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md` were produced (issue #1820). It prints per attention call per layer, so it is a diagnostic aid and not something to leave on. | diff --git a/src/models/qwen3_next.rs b/src/models/qwen3_next.rs index 113d5267e..6e670cdc4 100644 --- a/src/models/qwen3_next.rs +++ b/src/models/qwen3_next.rs @@ -591,6 +591,53 @@ pub(crate) struct Qwen3NextAttention { pub(crate) mrope: Option, } +/// Which of `attend_per_position`'s per-row tensors are copied to the layout +/// single-token decode hands the same attention call (issue #1935). +#[derive(Clone, Copy)] +pub(crate) struct AttendContiguity { + /// Copy the one-row query slice. Cheap: `[B, H, 1, D]`. + pub(crate) queries: bool, + /// Copy the key and value prefix slices. Expensive: one prefix-sized copy + /// per row, so this exists for the A/B rather than for production. + pub(crate) keys_values: bool, +} + +/// `MLXCEL_QWEN35_ATTEND_CONTIGUOUS`: `q` (the default), `kv`, `all`, or `0`. +/// +/// Default `q`, because a verify row and the decode step it stands for must be +/// the same computation and on CUDA they are not while the query row keeps the +/// block's strides. `0` restores the pre-#1935 slicing without a rebuild, which +/// is the kill switch the record's A/B uses. +pub(crate) fn attend_per_position_contiguity() -> AttendContiguity { + static RESOLVED: std::sync::OnceLock = std::sync::OnceLock::new(); + *RESOLVED.get_or_init(|| { + match std::env::var("MLXCEL_QWEN35_ATTEND_CONTIGUOUS") + .unwrap_or_default() + .trim() + .to_ascii_lowercase() + .as_str() + { + "0" | "off" | "false" | "no" | "none" => AttendContiguity { + queries: false, + keys_values: false, + }, + "kv" => AttendContiguity { + queries: false, + keys_values: true, + }, + "all" => AttendContiguity { + queries: true, + keys_values: true, + }, + // "q", "" and anything unrecognised take the shipped default. + _ => AttendContiguity { + queries: true, + keys_values: false, + }, + } + }) +} + impl Qwen3NextAttention { pub(crate) fn forward( &self, @@ -825,15 +872,40 @@ impl Qwen3NextAttention { let l_kv = k_shape[2]; let prefix_len = l_kv - l_q; + let contiguity = attend_per_position_contiguity(); let mut out: Option> = None; for i in 0..l_q { // queries[:, :, i:i+1, :] let q_i = mlxcel_core::slice(queries, &[0, 0, i, 0], &[b, n_q_heads, i + 1, head_dim]); + // A one-row slice of an `[B, H, L_q, D]` tensor has its heads + // `L_q * D` apart, while single-token decode hands the same call a + // freshly built `[B, H, 1, D]` whose heads are `D` apart. Same + // values, different layout, and on CUDA the fused `sdpa_vector` + // kernel this checkpoint's head_dim 256 reaches (issue #675) is not + // bit-equal across the two. Copying the row to the decode layout is + // what makes a verify row reproduce the decode step it stands for. + // See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. + let q_i = if contiguity.queries { + mlxcel_core::contiguous(&q_i, false) + } else { + q_i + }; // keys/values[:, :, : prefix_len + i + 1, :] let kv_len = prefix_len + i + 1; let k_i = mlxcel_core::slice(keys, &[0, 0, 0, 0], &[b, n_kv_heads, kv_len, head_dim]); let v_i = mlxcel_core::slice(values, &[0, 0, 0, 0], &[b, n_kv_heads, kv_len, v_shape[3]]); + // The key and value slices are leading slices in both paths, so + // their layout already matches; the copy is available for the A/B + // that establishes that, and costs a full prefix copy per row. + let (k_i, v_i) = if contiguity.keys_values { + ( + mlxcel_core::contiguous(&k_i, false), + mlxcel_core::contiguous(&v_i, false), + ) + } else { + (k_i, v_i) + }; // Single-query attention, no mask: the K/V slice is already the // exact causal prefix, matching the single-token decode call. let attn_i = mlxcel_core::layers::attention(&q_i, &k_i, &v_i, self.scale, None, 0.0, 0); From 1e88fd6bf1ecd59af05b1ad15a52a66ae5e6583a Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:38:32 +0900 Subject: [PATCH 05/16] test(speculative): revert the query-layout attempt and bisect the block in bytes The query-layout reading of issue #1935's residual is wrong and the change it produced is reverted, including its environment variable. Measured served on GB10 at temperature 0: copying the per-row query slice to the layout single-token decode hands the attention call leaves the width 2 and 4 completions exactly where they were (`3e60b1574c`, first differing token 105), and so does copying the key and value slices with it. The layouts were not what the fused kernel is sensitive to here. Recording the negative result rather than the change. What the same session did establish, and what the next arm has to explain: `MLXCEL_SDPA_VECTOR_LARGE_D=0` collapses the divergence. Served, it makes classic and width 4 byte-identical over all 200 tokens; in process, replaying the served run's own 79 rounds and 56 rewinds, it takes the target from 1 disagreement of 201 to 0. That switch decides only whether `head_dim` 256 reaches the fused `sdpa_vector` kernels, so the difference is in what those kernels do with the two calls rather than in the round loop, the rollback, the gated-delta scan, the quantized matmul or the drafter. Two instruments for that. The exactness probe's prompt length becomes `MLXCEL_MTP_PROBE_PROMPT_LEN`, defaulting to 256 rather than 8: the old length is why the probe reported byte-identity at widths 2 and 4 on a checkpoint whose served burst diverges there, since a short key sequence does not reach the behaviour the probe exists to catch. And a new `#[ignore]` arm drives the served round structure over the recorded transcript comparing logit bytes per kept row rather than argmaxes, then walks every layer's captured hidden state on the first differing row, which is what separates the full attention layers from the gated-delta ones. Refs #1935 --- docs/environment-variables.md | 1 - src/models/qwen3_5.rs | 38 +++- src/models/qwen3_5_dflash_probe_tests.rs | 239 +++++++++++++++++++++++ src/models/qwen3_next.rs | 72 ------- 4 files changed, 271 insertions(+), 79 deletions(-) diff --git a/docs/environment-variables.md b/docs/environment-variables.md index fd47cdcba..82ba324d4 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -462,7 +462,6 @@ recommended as normal deployment settings. | `MLXCEL_SPARSE_PAGED_DUMP` | `1`/`true`/`on`/`yes` enable | off | Prints each request's selected pool rows for a fused sparse decode (#904). **Synchronizes with the device**, so it is a debugging aid only and must never be set for a timed run. | | `MLXCEL_SPARSE_PAGED_MIN_SPARSITY` | non-negative integer | `8` | Minimum `live_len / selected_rows` ratio a launch must clear before fused sparse decode is dispatched (#904). The fused kernel measured 0.67x at 2x sparsity and 1.22x at 8x against tuned dense SDPA, so a token floor alone is not sufficient. `0` disables the gate for benchmarking the declined regime; an unparseable value falls back to the default. See [`sparse-paged-decode.md`](sparse-paged-decode.md). | | `MLXCEL_SDPA_VECTOR_LARGE_D` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | **CUDA only.** Gates whether the CUDA `supports_sdpa_vector` check accepts head_dim 256/288 (gemma family, qwen3.5/3.6, baichuan-m1, paligemma2), routing their decode to the fused `sdpa_vector` kernels instead of the materializing SDPA fallback (issue #675). Disabling restores the prior fallback with no rebuild; used for the A/B in `benchmarks/cuda_gb10_sdpav_675_2026-07-06.csv`. | -| `MLXCEL_QWEN35_ATTEND_CONTIGUOUS` | `q` (default), `kv`, `all`, or `0`/`off`/`false`/`no`/`none` to disable | `q` | **Qwen 3.5 family, speculative verify only.** Which of `Qwen3NextAttention::attend_per_position`'s per-row tensors are copied to the layout single-token decode hands the same attention call (issue #1935). A verify block attends one query position at a time so each row reproduces the decode step it stands for, but a one-row slice of a `[B, H, T, D]` block has its heads `T * D` apart while decode hands the call a `[B, H, 1, D]` whose heads are `D` apart. Same values, different strides, and on CUDA the fused `sdpa_vector` kernel that `MLXCEL_SDPA_VECTOR_LARGE_D` routes `head_dim` 256/288 to is not bit-equal across them, which is how a served greedy burst parted from classic decode at every verify width alike. `q` copies the query row, which is `[B, H, 1, D]` per row per attention layer and is the shipped fix. The key and value slices already match between the two paths (both slice the same cache buffer to `prefix + i + 1`), so `kv` and `all` exist for the A/B that established that and cost one prefix-sized copy per row. `0` restores the pre-#1935 slicing without a rebuild and is the kill switch the record's A/B uses. Read once per process. See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` | non-negative integer | `0` (off) | **CUDA only. Off by default; set to `32` to enable.** #1820 measured this path behind `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s ops fallback at three of five block widths with disjoint ranges (0.943x, 0.953x, 0.915x at widths 2, 8 and 16 on a 152-token prompt) and a wash at 2634 tokens, so the shipped dispatch is #1799's alone and this is opt-in. It is kept rather than removed because the mechanism is sound (12 plan builds per generation instead of 82, greedy token ids byte-identical to exact-shape cuDNN) and a crossover above 2634 keys is plausible and unmeasured: this path's cost is flat in context while the fallback's score matrix grows with the key length. See `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-17.md`. An array-masked SDPA call with 2 to N query rows over a KV cache (the speculative verify shape: a block appended to a cache) has its cuDNN execution-plan cache key bucketed, so one plan serves every round until the cache buffer grows (issue #1820). cuDNN keys that cache on the exact shapes and strides of q, k, v and the mask, and a verify round's key length is new every round, so before this every round of every attention-layer class rebuilt a plan on the host: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing. The fix is MLX's own one-row decode canonicalization extended to these calls: k and v are unsliced to the whole cache buffer, the additive mask is widened to the same width with the new columns set to `-inf`, and the true lengths reach cuDNN through `set_padding_mask` with `set_seq_len_q` / `set_seq_len_kv`, which is what keeps the widened region out of the result. k and v reach the bucket two ways. Where they are already a leading slice of one contiguous cache buffer, they are unsliced to that buffer's own allocated extent, so nothing is reallocated and no position outside the already-allocated cache is addressed; mlxcel's KV cache grows in 256-position steps, so one plan per class covers 256 appends. Where they are not (the drafter concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room past the live length) they are copied into a zero-padded buffer rounded up to the same 256 grid, which costs a copy per call and is bounded per tensor by `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB`. Above that bound the call keeps its exact shape and pays the plan build, and a shape-eligible call whose layout declines bucketing for any reason prints a one-time warning naming this variable, because such a call gets neither this fix nor `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s. `0` disables the bucketing without a rebuild, which restores the dispatch `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` alone produced and is the kill switch used for the A/B in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md`. Non-CUDA builds ignore it. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB` | non-negative integer (MiB) | `64` | **CUDA only.** Per-tensor ceiling on the zero-padded k/v copy the bucketing arm above makes when k and v are not a leading slice of one contiguous cache buffer, which is the drafter's shape: it concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room to unslice into (issue #1820). A call whose widened k or v would exceed this keeps its exact shape and rebuilds a plan each round, and prints the one-time warning described above, because moving hundreds of MiB per layer per round would cost more than the roughly 22 ms plan build it saves. At the default 64 MiB an 8-kv-head 128-dim bf16 cache buckets up to about 32k keys. `0` disables the copy arm, leaving only the free unslice arm. | | `MLXCEL_SDPA_PLAN_DEBUG` | `1` enables | off (`0`) | **CUDA only, diagnostic.** Writes one line per cuDNN SDPA call to stderr with the key fields that decide plan reuse: the q shape, `k_len`, the cache buffer extent and row stride, the mask column count and strides, the causal and sinks flags, whether the call took the decode or the bucketed canonicalization, and whether this call built a plan or reused one, plus the resident plan count. This is how the per-shape-class key-field table and the plan-build counts in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md` were produced (issue #1820). It prints per attention call per layer, so it is a diagnostic aid and not something to leave on. | diff --git a/src/models/qwen3_5.rs b/src/models/qwen3_5.rs index 62b059ee7..d903cd0e4 100644 --- a/src/models/qwen3_5.rs +++ b/src/models/qwen3_5.rs @@ -50,12 +50,38 @@ use mlxcel_core::{MlxArray, UniquePtr, concatenate}; use serde::Deserialize; use std::path::Path; -/// Prompt length the exactness probe prefills before comparing arms. +/// Prompt length the exactness probe prefills before comparing arms, unless +/// `MLXCEL_MTP_PROBE_PROMPT_LEN` overrides it. /// -/// Long enough that the attention layers hold a real KV prefix rather -/// than the empty-cache special case, short enough that the probe stays -/// a fraction of a second on a 27B target. -const PROBE_PROMPT_LEN: usize = 8; +/// This was 8 until issue #1935, on the reasoning that 8 is enough for the +/// attention layers to hold a real KV prefix rather than the empty-cache +/// special case. It is not. At 8 the probe reported byte-identity at verify +/// widths 2 and 4 on `qwen3.5-4b-4bit` while a served burst at those widths +/// diverged from classic decode, because the difference the probe exists to +/// catch is a kernel one and short key sequences do not reach the code path +/// that carries it. A probe that prefills a served-sized prefix catches it. +/// The cost is one prefill per draw per width at worker startup, paid once +/// and memoized by `mtp_exactness_gate`. +/// +/// See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. +const DEFAULT_PROBE_PROMPT_LEN: usize = 256; + +/// The probe's prompt length for this process. +/// +/// `MLXCEL_MTP_PROBE_PROMPT_LEN` moves it without a rebuild, which is what the +/// #1935 record's length sweep uses and what an operator whose checkpoint makes +/// the default prefill too slow at startup can reach for. Values below 2 are +/// ignored: the probe needs a prefix to attend to. +fn probe_prompt_len() -> usize { + static RESOLVED: std::sync::OnceLock = std::sync::OnceLock::new(); + *RESOLVED.get_or_init(|| { + std::env::var("MLXCEL_MTP_PROBE_PROMPT_LEN") + .ok() + .and_then(|v| v.trim().parse::().ok()) + .filter(|&n| n >= 2) + .unwrap_or(DEFAULT_PROBE_PROMPT_LEN) + }) +} /// Independent synthetic inputs the exactness probe compares before it /// is allowed to report equality. @@ -1631,7 +1657,7 @@ impl Qwen35Model { let salt = draw * 977 + 1; let wrap = |i: usize, stride: usize, offset: usize| ((i * stride + offset + salt) % vocab) as i32; - let prompt: Vec = (0..PROBE_PROMPT_LEN).map(|i| wrap(i, 7, 1)).collect(); + let prompt: Vec = (0..probe_prompt_len()).map(|i| wrap(i, 7, 1)).collect(); let block: Vec = (0..block_size).map(|i| wrap(i, 13, 3)).collect(); let as_input = diff --git a/src/models/qwen3_5_dflash_probe_tests.rs b/src/models/qwen3_5_dflash_probe_tests.rs index caafb19d1..603076b93 100644 --- a/src/models/qwen3_5_dflash_probe_tests.rs +++ b/src/models/qwen3_5_dflash_probe_tests.rs @@ -853,3 +853,242 @@ fn served_burst_wrapper_with_real_drafter_matches_the_chain() { } } } + +/// **Diagnostic 8.** Is the speculative forward byte-identical to the classic +/// forward at one row, on the real transcript, with both caches prefilled the +/// way their own paths prefill them? +/// +/// Diagnostic 3 asks a version of this and cannot answer it: it prefills the +/// speculative arm with `forward_speculative`, which is the per-position +/// prefill issue #1935 fixed, so every step it compares runs on caches that +/// already differ. This arm prefills classic with `forward_internal` and the +/// burst with `forward_prefill_with_capture_layers`, which the #1939 record +/// measured byte-identical, and only then walks one token at a time. +/// +/// What it separates: the exactness probe compares `forward_speculative` over +/// a block against `forward_speculative` one row at a time, so a difference +/// between `forward_speculative` at one row and `forward_internal` at one row +/// is invisible to it and would make a passing probe meaningless. Bytes, not +/// argmaxes: an argmax comparison stays silent until a difference happens to +/// cross a boundary, which on this checkpoint is what token 105 is. +#[test] +#[ignore = "needs the real Qwen 3.5 4B checkpoint, a GPU and a recorded transcript"] +fn speculative_t1_forward_versus_classic_on_the_real_transcript() { + let Some((model, dir)) = load_text_model() else { + return; + }; + let text = text_model(&model); + let Some(prompt) = env_ids("MLXCEL_Q35_PROBE_PROMPT") else { + eprintln!("[1935] skipping: set MLXCEL_Q35_PROBE_PROMPT and MLXCEL_Q35_PROBE_REFERENCE"); + return; + }; + let reference = env_ids("MLXCEL_Q35_PROBE_REFERENCE").expect("MLXCEL_Q35_PROBE_REFERENCE"); + let capture = probe_capture_layer_ids(); + eprintln!( + "[1935] {dir}: one-row speculative against classic over {} reference ids, \ + capture layers {capture:?}", + reference.len() + ); + + let mut classic_caches = text.make_internal_caches(); + let classic_prefill = text.forward_internal(&ids_of(&prompt), None, &mut classic_caches, None); + let mut burst_caches = text.make_speculative_caches(); + let burst_prefill = + text.forward_prefill_with_capture_layers(&ids_of(&prompt), &mut burst_caches, &capture); + + let last = prompt.len() as i32 - 1; + let c0 = row_bytes(&classic_prefill, last); + let b0 = row_bytes(&burst_prefill.logits, last); + eprintln!( + "[1935] prefill last row: {} of {} logit bytes differ", + differing(&c0, &b0), + c0.len() + ); + match compare_caches(&classic_caches, &burst_caches) { + None => eprintln!("[1935] prefill: every layer's cache state is byte-identical"), + Some((layer, why)) => { + eprintln!("[1935] prefill: FIRST cache divergence at layer {layer}: {why}") + } + } + + // Walk the whole transcript rather than stopping at the first difference: + // how many steps differ, and from where, is what tells accumulated drift + // apart from a one-off. + let mut first_diff: i64 = -1; + let mut differing_steps = 0usize; + let mut worst = 0usize; + for (i, token) in reference.iter().enumerate() { + let c = text.forward_internal(&ids_of(&[*token]), None, &mut classic_caches, None); + let s = text.forward_speculative(&ids_of(&[*token]), &mut burst_caches, &capture); + let cb = row_bytes(&c, 0); + let sb = row_bytes(&s.logits, 0); + let d = differing(&cb, &sb); + if d > 0 { + if first_diff < 0 { + first_diff = i as i64; + if let Some((layer, why)) = compare_caches(&classic_caches, &burst_caches) { + eprintln!("[1935] step {i}: FIRST cache divergence at layer {layer}: {why}"); + } else { + eprintln!( + "[1935] step {i}: logits differ in {d} bytes while every layer's \ + cache state is still byte-identical, so the difference is in the \ + forward rather than inherited" + ); + } + } + differing_steps += 1; + worst = worst.max(d); + } + if i % 50 == 0 { + eprintln!("[1935] step {i}: {d} of {} logit bytes differ", cb.len()); + } + } + eprintln!( + "[1935] {differing_steps} of {} one-row steps differ, first at {first_diff} \ + (-1 means none), worst {worst} bytes", + reference.len() + ); +} + +/// **Diagnostic 9.** Where, in bytes and in which layer, a verify block first +/// stops reproducing the single-token chain on the real transcript. +/// +/// Diagnostic 2 asks the same question on a 64-token synthetic prompt and +/// answers "nowhere", which is also what the shipped exactness probe answers +/// on its 8-token one, and both are wrong about a served request. This arm +/// drives the served run's own round structure over the recorded transcript +/// and compares logit BYTES per kept row rather than argmaxes, so it sees the +/// difference from the first round that has one instead of from the round +/// where it happens to cross a boundary. +/// +/// Both arms prefill through `forward_prefill_with_capture_layers` and the +/// chain arm steps one token at a time through `forward_speculative`, which +/// Diagnostic 8 measured byte-identical to `forward_internal` at one row over +/// this whole transcript. So any difference found here belongs to the block. +/// +/// On the first differing row it walks every layer's captured hidden state and +/// names the first one that differs, which is what separates the full +/// attention layers from the gated-delta ones. +#[test] +#[ignore = "needs the real Qwen 3.5 4B checkpoint, a GPU and a recorded transcript"] +fn block_versus_chain_byte_bisect_on_the_real_transcript() { + let Some((model, dir)) = load_text_model() else { + return; + }; + let text = text_model(&model); + let Some(prompt) = env_ids("MLXCEL_Q35_PROBE_PROMPT") else { + eprintln!("[1935] skipping: set MLXCEL_Q35_PROBE_PROMPT and MLXCEL_Q35_PROBE_REFERENCE"); + return; + }; + let reference = env_ids("MLXCEL_Q35_PROBE_REFERENCE").expect("MLXCEL_Q35_PROBE_REFERENCE"); + let block_size: usize = std::env::var("MLXCEL_Q35_PROBE_BLOCK") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(4); + let accepts: Vec = std::env::var("MLXCEL_Q35_PROBE_ACCEPTS") + .ok() + .map(|v| { + v.split(',') + .filter(|s| !s.trim().is_empty()) + .map(|s| s.trim().parse::().expect("accept count")) + .collect() + }) + .unwrap_or_else(|| vec![1, 2, 3]); + let wrong: Option = std::env::var("MLXCEL_Q35_PROBE_WRONG") + .ok() + .filter(|v| v != "0") + .map(|v| v.parse::().unwrap_or(9999)); + let layers = text.num_layers(); + let capture: Vec = (0..layers).collect(); + eprintln!( + "[1935] {dir}, {layers} layers, block {block_size}, {} accepts, prompt {}, \ + reference {}", + accepts.len(), + prompt.len(), + reference.len() + ); + + let mut chain_caches = text.make_speculative_caches(); + let _ = text.forward_prefill_with_capture_layers(&ids_of(&prompt), &mut chain_caches, &capture); + let mut burst_caches = text.make_speculative_caches(); + let _ = text.forward_prefill_with_capture_layers(&ids_of(&prompt), &mut burst_caches, &capture); + + let mut i = 0usize; + let mut round = 0usize; + let mut first_reported = false; + let mut differing_rows = 0usize; + while i < reference.len() { + let end = (i + block_size).min(reference.len()); + let rows = end - i; + let keep = accepts[round % accepts.len()].clamp(1, rows); + let mut fed: Vec = reference[i..end].to_vec(); + if let Some(w) = wrong { + for slot in fed.iter_mut().skip(keep) { + *slot = w; + } + } + let burst = text.forward_speculative(&ids_of(&fed), &mut burst_caches, &capture); + for r in 0..keep { + let chain = + text.forward_speculative(&ids_of(&[fed[r]]), &mut chain_caches, &capture); + let cb = row_bytes(&chain.logits, 0); + let bb = row_bytes(&burst.logits, r as i32); + let d = differing(&cb, &bb); + if d > 0 { + differing_rows += 1; + if !first_reported { + first_reported = true; + eprintln!( + "[1935] FIRST byte difference at round {round} row {r} \ + (emitted index {}, absolute position {}): {d} of {} logit bytes", + i + r + 1, + prompt.len() + i + r, + cb.len() + ); + let mut first_layer = None; + for layer in 0..layers { + let c = row_bytes(&chain.hidden_states[layer], 0); + let b = row_bytes(&burst.hidden_states[layer], r as i32); + let dl = differing(&c, &b); + if dl > 0 { + if first_layer.is_none() { + first_layer = Some(layer); + eprintln!( + "[1935] FIRST layer divergence at layer {layer} ({}): \ + {dl} of {} hidden bytes", + if text.config.is_linear_layer(layer) { + "gated-delta" + } else { + "full attention" + }, + c.len() + ); + } + } + } + if first_layer.is_none() { + eprintln!( + "[1935] every layer's hidden state agrees on this row, so the \ + difference is after the last captured layer (the norm or the \ + LM head)" + ); + } + } + } + } + if keep < rows { + text.rollback_speculative_cache( + &mut burst_caches, + &burst.gdn_states, + &[keep as i32 - 1], + rows as i32, + ); + } + i += keep; + round += 1; + } + eprintln!( + "[1935] {round} rounds, {differing_rows} of {} kept rows differ in logit bytes", + reference.len() + ); +} diff --git a/src/models/qwen3_next.rs b/src/models/qwen3_next.rs index 6e670cdc4..113d5267e 100644 --- a/src/models/qwen3_next.rs +++ b/src/models/qwen3_next.rs @@ -591,53 +591,6 @@ pub(crate) struct Qwen3NextAttention { pub(crate) mrope: Option, } -/// Which of `attend_per_position`'s per-row tensors are copied to the layout -/// single-token decode hands the same attention call (issue #1935). -#[derive(Clone, Copy)] -pub(crate) struct AttendContiguity { - /// Copy the one-row query slice. Cheap: `[B, H, 1, D]`. - pub(crate) queries: bool, - /// Copy the key and value prefix slices. Expensive: one prefix-sized copy - /// per row, so this exists for the A/B rather than for production. - pub(crate) keys_values: bool, -} - -/// `MLXCEL_QWEN35_ATTEND_CONTIGUOUS`: `q` (the default), `kv`, `all`, or `0`. -/// -/// Default `q`, because a verify row and the decode step it stands for must be -/// the same computation and on CUDA they are not while the query row keeps the -/// block's strides. `0` restores the pre-#1935 slicing without a rebuild, which -/// is the kill switch the record's A/B uses. -pub(crate) fn attend_per_position_contiguity() -> AttendContiguity { - static RESOLVED: std::sync::OnceLock = std::sync::OnceLock::new(); - *RESOLVED.get_or_init(|| { - match std::env::var("MLXCEL_QWEN35_ATTEND_CONTIGUOUS") - .unwrap_or_default() - .trim() - .to_ascii_lowercase() - .as_str() - { - "0" | "off" | "false" | "no" | "none" => AttendContiguity { - queries: false, - keys_values: false, - }, - "kv" => AttendContiguity { - queries: false, - keys_values: true, - }, - "all" => AttendContiguity { - queries: true, - keys_values: true, - }, - // "q", "" and anything unrecognised take the shipped default. - _ => AttendContiguity { - queries: true, - keys_values: false, - }, - } - }) -} - impl Qwen3NextAttention { pub(crate) fn forward( &self, @@ -872,40 +825,15 @@ impl Qwen3NextAttention { let l_kv = k_shape[2]; let prefix_len = l_kv - l_q; - let contiguity = attend_per_position_contiguity(); let mut out: Option> = None; for i in 0..l_q { // queries[:, :, i:i+1, :] let q_i = mlxcel_core::slice(queries, &[0, 0, i, 0], &[b, n_q_heads, i + 1, head_dim]); - // A one-row slice of an `[B, H, L_q, D]` tensor has its heads - // `L_q * D` apart, while single-token decode hands the same call a - // freshly built `[B, H, 1, D]` whose heads are `D` apart. Same - // values, different layout, and on CUDA the fused `sdpa_vector` - // kernel this checkpoint's head_dim 256 reaches (issue #675) is not - // bit-equal across the two. Copying the row to the decode layout is - // what makes a verify row reproduce the decode step it stands for. - // See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. - let q_i = if contiguity.queries { - mlxcel_core::contiguous(&q_i, false) - } else { - q_i - }; // keys/values[:, :, : prefix_len + i + 1, :] let kv_len = prefix_len + i + 1; let k_i = mlxcel_core::slice(keys, &[0, 0, 0, 0], &[b, n_kv_heads, kv_len, head_dim]); let v_i = mlxcel_core::slice(values, &[0, 0, 0, 0], &[b, n_kv_heads, kv_len, v_shape[3]]); - // The key and value slices are leading slices in both paths, so - // their layout already matches; the copy is available for the A/B - // that establishes that, and costs a full prefix copy per row. - let (k_i, v_i) = if contiguity.keys_values { - ( - mlxcel_core::contiguous(&k_i, false), - mlxcel_core::contiguous(&v_i, false), - ) - } else { - (k_i, v_i) - }; // Single-query attention, no mask: the K/V slice is already the // exact causal prefix, matching the single-token decode call. let attn_i = mlxcel_core::layers::attention(&q_i, &k_i, &v_i, self.scale, None, 0.0, 0); From 481e2dbb849cbacf028f6f15d87ca9e404c026f7 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:42:01 +0900 Subject: [PATCH 06/16] docs(benchmarks): correct the cause section and let a measured all-declined run pass The record's cause section said the residual was the per-row tensor layouts. The A/B says otherwise: copying the query slice, and copying the key and value slices with it, leave the served width 2 and 4 completions exactly where they were. Reading CUDA's `supports_sdpa_vector` and `sdpa_vector` confirms it, since both calls satisfy the same admission predicate and the same copy predicates and neither reaches the 2-pass split. The section now records that as a negative result and states what the kill switch does establish: the fused `sdpa_vector` path is the discriminator, in two independent arms, and the difference is in what MLX does around the call rather than in what is handed to it. `assert_server_byte_equality_at_widths` no longer fails when every width declines. It failed on the reasoning that an all-declined run says nothing about verify-path parity, and on this pairing on CUDA that outcome is the contract holding through the gate rather than a hole in the test. What the test must not allow is a decline for some other reason standing in for a measured verdict, so each declined width now has to carry the exactness probe's own decline line; a multimodal payload, an adopted prompt-cache prefix or a drafter from the wrong family still fails the arm. Refs #1935 --- ...dth-2-4-residual-qwen35-gb10-2026-09-21.md | 26 ++++++++----- tests/speculative_parity.rs | 39 ++++++++++++++++--- 2 files changed, 49 insertions(+), 16 deletions(-) diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index fbb72f8b4..003ab4c78 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -45,19 +45,15 @@ Data and harness: `data/dflash-width-2-4-residual-gb10-2026-09-21/`. That pattern is a difference in the logits that is below the reporting floor almost everywhere, surfacing as a one-step logprob difference from index 62 onward and finally crossing an argmax at 105, where the top two candidates sat within a reporting step of each other. It is not a rollback, not a round-boundary effect, and not nondeterminism: both requests in each arm are byte-identical to each other. -## The cause: a verify row and the decode step it stands for are not the same call +## The cause: the verify block's attention does not reproduce decode's, on one kernel -`Qwen3NextAttention::attend_per_position` is what makes a verify block reproduce single-token decode. For each query position `i` of a `[B, H, T, D]` block it attends `queries[:, :, i:i+1, :]` to the causal prefix `keys[:, :, ..prefix + i + 1, :]` with no mask, so each row sees exactly the prefix a decode step would. The arithmetic is right. The layout is not. +`Qwen3NextAttention::attend_per_position` is what makes a verify block reproduce single-token decode. For each query position `i` of a `[B, H, T, D]` block it attends `queries[:, :, i:i+1, :]` to the causal prefix `keys[:, :, ..prefix + i + 1, :]` with no mask, so each row sees exactly the prefix a decode step would, and `target_verify && l > 1` is the only condition that selects it. Classic decode takes the `l == 1` arm of the same match and reaches the same `layers::attention` entry point with the same absence of a mask. -A one-row slice of a `[B, H, T, D]` tensor has its heads `T * D` apart. Single-token decode hands the same attention call a freshly built `[B, H, 1, D]`, whose heads are `D` apart. Same values, same shape, different strides. On CUDA this checkpoint's `head_dim` is 256, which `MLXCEL_SDPA_VECTOR_LARGE_D` (issue #675) routes to the fused `sdpa_vector` kernels, and that kernel is not bit-equal across the two layouts. The key and value slices are unaffected: both paths slice the same cache buffer to `prefix + i + 1`, so their strides already match, which is why the query row is the whole of it. +One switch decides whether those two calls agree. -Three arms establish this, and each rules out what the others cannot. +**`MLXCEL_SDPA_VECTOR_LARGE_D=0` collapses the divergence, in two independent arms.** That switch does one thing: it decides whether `head_dim` 256 and 288 are accepted by CUDA's `supports_sdpa_vector` gate, and so whether a single-query attention call takes the fused `sdpa_vector` kernels or the materializing fallback (issue #675). Both classic decode and every row of `attend_per_position` are single-query calls, so both move together. -**At `T = 1` the two forwards are byte-identical.** Prefilling classic with `forward_internal` and the burst with `forward_prefill_with_capture_layers`, then walking the real 200-token transcript one token at a time, every step's logits agree in all 496640 bytes and the prefill's do too. At `T = 1` the one-row slice IS the whole tensor, so the layouts coincide and the difference has nowhere to appear. This is what rules out everything that is not a block: the projections, the gated-delta layers, the rotary, the prefill, the caches. - -**At `T = 4` it reproduces with no drafter at all.** Replaying the served width-4 run's own 79 rounds and 56 rewinds in process, with the rejected rows carrying a wrong token id, the target disagrees with the classic chain at exactly one of 201 greedy positions, at 105, taking 5741 where classic takes 11439. The served position, from a replay that contains no drafter and no server. That kills the standing hypothesis that the drafter's interleaved MLX work was the cause: there is no drafter in this arm. Driving `run_dflash_on_target` with the real drafter bound reproduces the same position, which is consistent rather than additional. - -**Turning the fused kernel off makes the two byte-identical.** `MLXCEL_SDPA_VECTOR_LARGE_D=0` routes single-query attention off `sdpa_vector` for `head_dim` 256 on both paths. Served, same binary: +Served, one binary, temperature 0: | arm | completion sha256 | first token differing from classic | |---|---|---:| @@ -66,7 +62,17 @@ Three arms establish this, and each rules out what the others cannot. | classic, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | | | width 4, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | none, 200 of 200 | -Both arms move, which is expected and is why the row is not evidence about classic decode; what matters is that they move onto each other. A single kill switch collapsing a 95-token divergence to nothing is stronger than any of the exclusions that preceded it, and it takes the remaining candidates with it: rollback, the gated-delta scan, the quantized matmul kernel, the drafter, the round loop, the server process. None of those is disabled by this switch, and all of them stop mattering when it is off. +Both arms move, which is expected and is why this table is not evidence about classic decode; what matters is that they move onto each other. In process, replaying the served width-4 run's own 79 rounds and 56 rewinds against a classic chain computed in the same run, the same switch takes the target from 1 disagreement of 201 greedy positions to 0. + +A single kill switch collapsing a 95-token divergence to nothing is stronger than any of the exclusions that preceded it, and it takes the remaining candidates with it. Rollback, the gated-delta scan, the quantized matmul kernel, the drafter, the round loop and the server process are none of them disabled by this switch, and all of them stop mattering when it is off. + +**Two more arms place it at `T > 1` and nowhere else.** At `T = 1`, prefilling classic with `forward_internal` and the burst with `forward_prefill_with_capture_layers` and then walking the real 200-token transcript one token at a time, every step's logits agree in all 496640 bytes and the prefill's do too. `attend_per_position` at `T = 1` slices a one-row tensor, so its call is classic's call, and nothing differs. At `T = 4` the divergence reproduces in process with no drafter and no server, at exactly the served position, taking 5741 where classic takes 11439. That kills the hypothesis PR #1939 left standing, that the drafter's interleaved MLX work was the cause: there is no drafter in that arm. + +### What it is not: the per-row tensor layouts + +The obvious reading of "same math, one kernel, two answers" is that the two calls hand the kernel differently-laid-out tensors. A one-row slice of a `[B, H, T, D]` block does carry the block's strides, and single-token decode does hand the same call a `[B, H, 1, D]` built from a one-row forward. Measured, that is not the difference. Copying the per-row query slice to a fresh contiguous array leaves the served width 2 and 4 completions exactly where they were (`3e60b1574c`, first differing token 105), and copying the key and value slices along with it changes nothing either. The change that produced those arms is reverted; the negative result is recorded here instead. + +Reading the CUDA gate confirms why. `supports_sdpa_vector` admits a call when `q.shape(2) < 4`, which both satisfy, and `sdpa_vector`'s own `q_copy_unless` and `kv_copy_unless` predicates accept both layouts without copying: a `[1, H, 1, D]` query passes on `strides[3] == 1 && strides[2] == D * H && strides[1] == D`, which the block's slice satisfies as well as decode's tensor does, and a key or value whose batch dimension is 1 is accepted whatever its strides. The 1-pass and 2-pass split is on `k.shape(2) > 1024`, which neither reaches. So the arguments the kernel sees are the same shape, the same strides and the same values, and the difference is in what MLX does around the call rather than in what is handed to it. ## Why the gate reported the property intact diff --git a/tests/speculative_parity.rs b/tests/speculative_parity.rs index a9d70cbe3..9e9cc42d2 100644 --- a/tests/speculative_parity.rs +++ b/tests/speculative_parity.rs @@ -349,6 +349,21 @@ async fn assert_server_byte_equality_at_widths( let identical = spec.content == baseline.content && spec.completion_tokens == baseline.completion_tokens; if was_declined { + // A decline only counts when the exactness gate measured it. The + // other decline reasons (a multimodal payload, an adopted + // prompt-cache prefix, a drafter from the wrong family) say + // nothing about verify-path parity, and letting one of them stand + // in for a measured verdict is how an all-declined run could pass + // while proving nothing. + assert!( + spec.logs + .contains("the block-versus-chain exactness probe did not pass"), + "[{}] b={width}: the burst declined for a reason other than the exactness \ + probe, so this arm says nothing about verify-path parity. Captured \ + logs:\n{}", + pairing.name, + spec.logs, + ); declined.push(width); assert!( identical, @@ -377,12 +392,24 @@ async fn assert_server_byte_equality_at_widths( mismatched.iter().map(|(w, _)| *w).collect::>(), baseline.content, ); - assert!( - !ran.is_empty(), - "[{}] every width declined, so this run says nothing about verify-path parity. \ - Widths tried: {widths:?}", - pairing.name, - ); + // An all-declined run used to fail here, on the reasoning that it says + // nothing about verify-path parity. It does say something, and on this + // pairing on CUDA it is the correct outcome rather than a hole in the test + // (issue #1935): the verify block is not bit-equal to classic decode at any + // width on this host, the exactness probe now measures that rather than + // reporting a false pass from an 8-token prefix, and the gate declines + // every width. What the test must not allow is a decline for any other + // reason standing in for a measured verdict, which the per-width assertion + // above rules out. `MLXCEL_MTP_ALLOW_INEXACT=1` remains the way to engage + // the burst anyway and forfeit the contract. + if ran.is_empty() { + eprintln!( + "[{}] every width declined by a measured exactness verdict, and every response \ + equalled the drafter-less baseline byte for byte. That is the contract holding \ + through the gate rather than through the verify block. Widths tried: {widths:?}", + pairing.name, + ); + } } /// byte-equality phase: spawn `mlxcel-server` twice against the From c3b88fc0710bf18ea1fb1293c181155d466b2085 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:46:22 +0900 Subject: [PATCH 07/16] test(speculative): keep the query-layout A/B arms as a recorded negative result The arms that ruled the per-row tensor layouts out, and the harness that produced them, alongside the `MLXCEL_SDPA_VECTOR_LARGE_D` A/B they sit against. The reading they rule out is the obvious one for "same math, one kernel, two answers", so a later reader needs to see it measured rather than argued. Also the formatting fix the byte-bisect arm needed. Refs #1935 --- .../README.md | 10 + .../arms/arm.fix-classic-all.json | 435 +++++++++++++++++ .../arms/arm.fix-classic.json | 435 +++++++++++++++++ .../arms/arm.fix-w2-q.json | 441 ++++++++++++++++++ .../arms/arm.fix-w4-all.json | 441 ++++++++++++++++++ .../arms/arm.fix-w4-off.json | 441 ++++++++++++++++++ .../arms/arm.fix-w4-q.json | 441 ++++++++++++++++++ .../harness/ab_fix.sh | 27 ++ src/models/qwen3_5_dflash_probe_tests.rs | 3 +- 9 files changed, 2672 insertions(+), 2 deletions(-) create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic-all.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w2-q.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-all.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-off.json create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-q.json create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md index 032f078c7..f35dd456f 100644 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md @@ -19,3 +19,13 @@ Raw data for the residual PR #1939 left open: on `models/mlx/qwen3.5-4b-4bit` wi `harness/algebra.py` checks a run's round algebra offline against the transcript lines, with no GPU: that each round's bonus is the previous round's last emitted token, that `accepted` is the longest common prefix of the drafter's proposals and the target's block argmax, and that the emitted tokens are `draft[:accepted] + [target[accepted]]`. Rounds are logged 0-based, which is where a burst boundary is. `harness/compare.py` reads the arms against the classic null arm: the first differing token index, and whether the logprobs agree before it. + +## The A/B arms + +`harness/ab_sdpav.sh` runs the `MLXCEL_SDPA_VECTOR_LARGE_D` A/B (`arms/arm.cls-novec.json`, `arms/arm.w4-novec.json`). That switch decides only whether `head_dim` 256 and 288 reach CUDA's fused `sdpa_vector` kernels, and both classic decode and every row of the verify block's per-position attention are single-query calls, so both move together. With it off the two completions are byte-identical over all 200 tokens. + +`harness/ab_fix.sh` runs the query-layout A/B (`arms/arm.fix-*.json`), which is a negative result: copying the per-row query slice to a fresh contiguous array, and copying the key and value slices with it, leave the width 2 and 4 completions exactly where they were. The change that produced those arms is reverted; the arms are kept because the reading they rule out is the obvious one. + +`probe_verdicts.txt` is `probe_block_chain_exactness` on the real checkpoint, and `probe_arms.txt` is the three in-process arms: the one-row byte walk, the replay of the served round structure with no drafter, and the served burst wrapper with the real drafter bound. + +`arms/accepts_w4.txt` and `arms/accepts_w2.txt` are the per-round kept-row counts the served runs produced, which is what `MLXCEL_Q35_PROBE_ACCEPTS` takes so an in-process replay reproduces a served run's own cache history rather than an approximation of it. diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic-all.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic-all.json new file mode 100644 index 000000000..e447d445b --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic-all.json @@ -0,0 +1,435 @@ +{ + "tag": "fix-classic-all", + "width": "classic", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-classic-all.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=all", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.5, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -0.875, + 0.0, + -0.375, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.375, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.625, + -0.75, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.25, + -2.0, + -0.5, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + -0.25, + -1.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.375, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + -0.25, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + -0.75, + -1.875, + -1.25, + -0.375, + -0.5, + 0.0, + -0.75, + -0.375, + -0.375, + -0.625, + -0.125, + -0.5, + -0.125, + 0.0, + 0.0, + -0.5, + -1.0, + -0.25, + -0.625, + -0.625, + -0.5, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic.json new file mode 100644 index 000000000..dea102dfd --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-classic.json @@ -0,0 +1,435 @@ +{ + "tag": "fix-classic", + "width": "classic", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-classic.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.5, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -0.875, + 0.0, + -0.375, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.375, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.625, + -0.75, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.25, + -2.0, + -0.5, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + -0.25, + -1.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.375, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + -0.25, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + -0.75, + -1.875, + -1.25, + -0.375, + -0.5, + 0.0, + -0.75, + -0.375, + -0.375, + -0.625, + -0.125, + -0.5, + -0.125, + 0.0, + 0.0, + -0.5, + -1.0, + -0.25, + -0.625, + -0.625, + -0.5, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w2-q.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w2-q.json new file mode 100644 index 000000000..bc2884d66 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w2-q.json @@ -0,0 +1,441 @@ +{ + "tag": "fix-w2-q", + "width": "2", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "2" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-w2-q.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-all.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-all.json new file mode 100644 index 000000000..5ef1e5e0c --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-all.json @@ -0,0 +1,441 @@ +{ + "tag": "fix-w4-all", + "width": "4", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "4" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-w4-all.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=all", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-off.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-off.json new file mode 100644 index 000000000..9e01794c0 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-off.json @@ -0,0 +1,441 @@ +{ + "tag": "fix-w4-off", + "width": "4", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "4" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-w4-off.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=0", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-q.json b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-q.json new file mode 100644 index 000000000..da365430d --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/arms/arm.fix-w4-q.json @@ -0,0 +1,441 @@ +{ + "tag": "fix-w4-q", + "width": "4", + "cmd": [ + "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18937", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "4" + ], + "log": "/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/arms/server.fix-w4-q.log", + "extra_env": "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " **", + "kwargs", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " The", + " back", + "off", + " policy", + " to", + " use", + ".", + "\n", + " ", + " logger", + ":", + " The", + " logger", + " to", + " use", + " for", + " logging", + ".", + "\n", + " ", + " **", + "kwargs", + ":", + " Additional", + " arguments", + " to", + " pass", + " to", + " the", + " function", + ".", + "\n\n", + " ", + " Returns", + ":", + "\n", + " ", + " The", + " result", + " of", + " the", + " function", + ".", + "\n", + " ", + " \"\"\"", + "\n", + " ", + " if" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.625, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -1.0, + 0.0, + -0.25, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.5, + -2.0, + -0.375, + -0.25, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.75, + -0.25, + -0.125, + -0.625, + 0.0, + 0.0, + 0.0, + -0.375, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + -0.375, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.875, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.625, + -0.125, + -0.25, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -1.375, + -0.375, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + -0.375 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh new file mode 100755 index 000000000..cd9e86575 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env bash +# A/B the per-row query-layout fix on the served path, against the classic null +# arm from the same binary. `0` is the kill switch that restores the pre-fix +# slicing, so the same binary produces both sides. +set -uo pipefail +SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad +WT=/home/inureyes/Development/mlxcel-wt-1935 +BIN=/home/inureyes/Development/mlxcel/target/release/mlxcel-server +H=$WT/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness +run() { # width tag extra-env + python3 "$SP/transcript.py" --server "$BIN" \ + --target "$WT/models/mlx/qwen3.5-4b-4bit" \ + --drafter "$WT/models/mlx/qwen3.5-4b-dflash" \ + --width "$1" --n 1 --max-tokens 200 --tag "$2" --extra-env "$3" \ + --prompt-file "$H/prompt_retry.txt" --outdir "$SP/arms" --port 18937 + echo "=== $2 done ===" +} +run classic fix-classic "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q" +run 4 fix-w4-q "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q" +run 2 fix-w2-q "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q" +run 4 fix-w4-off "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=0" +# If the query copy alone does not close it, the key and value slices are not +# the matching pair this assumed; `all` and `kv` separate those in the same +# session rather than costing another build. +run 4 fix-w4-all "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=all" +run classic fix-classic-all "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=all" +echo "FIX AB DONE" diff --git a/src/models/qwen3_5_dflash_probe_tests.rs b/src/models/qwen3_5_dflash_probe_tests.rs index 603076b93..998f56c41 100644 --- a/src/models/qwen3_5_dflash_probe_tests.rs +++ b/src/models/qwen3_5_dflash_probe_tests.rs @@ -1029,8 +1029,7 @@ fn block_versus_chain_byte_bisect_on_the_real_transcript() { } let burst = text.forward_speculative(&ids_of(&fed), &mut burst_caches, &capture); for r in 0..keep { - let chain = - text.forward_speculative(&ids_of(&[fed[r]]), &mut chain_caches, &capture); + let chain = text.forward_speculative(&ids_of(&[fed[r]]), &mut chain_caches, &capture); let cb = row_bytes(&chain.logits, 0); let bb = row_bytes(&burst.logits, r as i32); let d = differing(&cb, &bb); From 533329e99ddf8090e2a0a8ac08c6e4183a0e9b11 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 14:55:24 +0900 Subject: [PATCH 08/16] fix(speculative): decline the Qwen 3.5 DFlash burst where the probe cannot see the hazard A probe that cannot observe a hazard reports a pass, which is worse than one that fails, and that is what the exactness gate was doing at DFlash verify widths 2 and 4 on this family (issue #1935). `probe_block_chain_exactness` now returns `NotRun` with a reason, before its draws, when the checkpoint and host are the configuration measured to break byte-identity: CUDA, a `head_dim` of 256 or 288, and `MLXCEL_SDPA_VECTOR_LARGE_D` left enabled. `mtp_exactness_gate` already treats every non-`Equal` verdict as a decline, so both Qwen 3.5 `DFlashTargetModel` impls decline at every width and the request is served by classic decode instead. Why the probe cannot see it. It prefills, runs one block and compares, and the difference does not exist yet there. Measured on `qwen3.5-4b-4bit`, the first row whose logits differ is thirty-six tokens past a 158-token prompt and seventy past a 256-token one, and a byte-level bisect puts the first differing layer at 15, a full attention layer. Repeating that bisect with five accept patterns puts the first difference at the same absolute position in all five, so the trigger is a position in the sequence rather than a number of rounds or rewinds. Sweeping the probe's own prompt length over 8, 32, 64, 128, 256 and 512 leaves its verdict byte-identity at widths 2 and 4 at every length, so lengthening the prefix is not the answer either. The length is now `MLXCEL_MTP_PROBE_PROMPT_LEN`, defaulting to what it always was, so that sweep can be repeated on another checkpoint. This narrows where DFlash engages on this family, deliberately. The alternative is the status quo, in which an operator who turns on speculative decoding gets different greedy text than without it and nothing says so. Both escapes stay: `MLXCEL_MTP_ALLOW_INEXACT=1` engages the burst and logs the forfeit, and `MLXCEL_SDPA_VECTOR_LARGE_D=0` buys the contract back instead of forfeiting it, at the cost of classic decode's own fused attention kernel. Record, with the served arms, the kill-switch A/B, the length sweep and the bisect: `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. Refs #1935 --- .../harness/run_probes.sh | 23 ++++++ .../harness/run_probes2.sh | 25 ++++++ .../probe_length_sweep_and_bisect.txt | 38 ++++++++++ ...dth-2-4-residual-qwen35-gb10-2026-09-21.md | 33 +++++++- docs/environment-variables.md | 1 + src/models/qwen3_5.rs | 76 ++++++++++++++++--- 6 files changed, 183 insertions(+), 13 deletions(-) create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh create mode 100755 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_length_sweep_and_bisect.txt diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh new file mode 100755 index 000000000..7aaef3f10 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# The in-process arms, run against an already-built test binary by path so a +# concurrent rebuild of the server cannot reshape them mid-run. +set -uo pipefail +SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad +WT=/home/inureyes/Development/mlxcel-wt-1935 +TB=${TEST_BIN:-/home/inureyes/Development/mlxcel/target/release/deps/mlxcel-69c784c732fe3546} +cd "$WT" || exit 1 +export MLX_CUDA_ARCHITECTURES=121 MLX_ENABLE_TF32=1 +export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/prompt_ids.json")))))')" +export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/classic_ids.json")))))')" + +echo "=== ARM 3: speculative against classic at ONE ROW, logit BYTES ===" +"$TB" --ignored --test-threads=1 --nocapture speculative_t1_forward_versus_classic_on_the_real_transcript 2>&1 | grep -E "^\[1935\]|test result|panicked" + +echo "=== ARM 1: replay the served width-4 round structure, no drafter ===" +MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$SP/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ + "$TB" --ignored --test-threads=1 --nocapture round_loop_cache_dynamics_with_rollback_match_the_chain 2>&1 | grep -E "^\[1935\]|test result|panicked" + +echo "=== ARM 2: the served burst wrapper with the real drafter ===" +MLXCEL_Q35_PROBE_WIDTHS=4 \ + "$TB" --ignored --test-threads=1 --nocapture served_burst_wrapper_with_real_drafter_matches_the_chain 2>&1 | grep -E "^\[1935\]|test result|panicked" +echo "ALL PROBES DONE" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh new file mode 100755 index 000000000..334505e5d --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +set -uo pipefail +SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad +WT=/home/inureyes/Development/mlxcel-wt-1935 +TB=${TEST_BIN:-/home/inureyes/Development/mlxcel/target/release/deps/mlxcel-e4caea9f4277a3b5} +cd "$WT" || exit 1 +export MLX_CUDA_ARCHITECTURES=121 MLX_ENABLE_TF32=1 +export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/prompt_ids.json")))))')" +export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/classic_ids.json")))))')" +echo "test binary: $TB" + +echo "=== BISECT: where do logit bytes first differ, and in which layer ===" +MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$SP/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ + "$TB" --ignored --test-threads=1 --nocapture block_versus_chain_byte_bisect_on_the_real_transcript 2>&1 | grep -E "^\[1935\]|panicked" + +echo "=== PROBE VERDICTS against prompt length ===" +for L in 8 32 64 128 256 512; do + echo "--- MLXCEL_MTP_PROBE_PROMPT_LEN=$L ---" + S=$(date +%s.%N) + MLXCEL_MTP_PROBE_PROMPT_LEN=$L "$TB" --ignored --test-threads=1 --nocapture \ + block_chain_exactness_verdicts_on_the_real_checkpoint 2>&1 | grep -E "^\[1935\] width" + E=$(date +%s.%N) + echo " (arm wall clock including model load: $(python3 -c "print(f'{$E-$S:.1f}s')"))" +done +echo "PROBES2 DONE" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_length_sweep_and_bisect.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_length_sweep_and_bisect.txt new file mode 100644 index 000000000..f44388af1 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/probe_length_sweep_and_bisect.txt @@ -0,0 +1,38 @@ +test binary: /home/inureyes/Development/mlxcel/target/release/deps/mlxcel-69c784c732fe3546 +=== BISECT: where do logit bytes first differ, and in which layer === +[1935] FIRST byte difference at round 17 row 3 (emitted index 37, absolute position 194): 230932 of 496640 logit bytes +[1935] FIRST layer divergence at layer 15 (full attention): 1016 of 5120 hidden bytes +[1935] 79 rounds, 164 of 200 kept rows differ in logit bytes +=== PROBE VERDICTS against prompt length === +--- MLXCEL_MTP_PROBE_PROMPT_LEN=8 --- +[1935] width 2: verify block is byte-identical to the single-token chain +[1935] width 4: verify block is byte-identical to the single-token chain +[1935] width 8: verify block position 0 differs from the single-token chain in 229971 of 496640 logit bytes +[1935] width 16: verify block position 0 differs from the single-token chain in 229971 of 496640 logit bytes + (arm wall clock including model load: 4.0s) +--- MLXCEL_MTP_PROBE_PROMPT_LEN=32 --- +[1935] width 2: verify block is byte-identical to the single-token chain +[1935] width 4: verify block is byte-identical to the single-token chain +[1935] width 8: verify block position 0 differs from the single-token chain in 183229 of 496640 logit bytes +[1935] width 16: verify block position 0 differs from the single-token chain in 183229 of 496640 logit bytes + (arm wall clock including model load: 24.1s) +--- MLXCEL_MTP_PROBE_PROMPT_LEN=64 --- + (arm wall clock including model load: 3.9s) +--- MLXCEL_MTP_PROBE_PROMPT_LEN=128 --- +[1935] width 2: verify block is byte-identical to the single-token chain +/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/run_probes2.sh: line 17: 1384507 Aborted (core dumped) MLXCEL_MTP_PROBE_PROMPT_LEN=$L "$TB" --ignored --test-threads=1 --nocapture block_chain_exactness_verdicts_on_the_real_checkpoint 2>&1 + 1384508 Done | grep -E "^\[1935\] width" + (arm wall clock including model load: 4.5s) +--- MLXCEL_MTP_PROBE_PROMPT_LEN=256 --- +[1935] width 2: verify block is byte-identical to the single-token chain +[1935] width 4: verify block is byte-identical to the single-token chain +/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad/run_probes2.sh: line 17: 1384572 Aborted (core dumped) MLXCEL_MTP_PROBE_PROMPT_LEN=$L "$TB" --ignored --test-threads=1 --nocapture block_chain_exactness_verdicts_on_the_real_checkpoint 2>&1 + 1384573 Done | grep -E "^\[1935\] width" + (arm wall clock including model load: 6.4s) +--- MLXCEL_MTP_PROBE_PROMPT_LEN=512 --- +[1935] width 2: verify block is byte-identical to the single-token chain +[1935] width 4: verify block is byte-identical to the single-token chain +[1935] width 8: verify block position 0 differs from the single-token chain in 196664 of 496640 logit bytes +[1935] width 16: verify block position 0 differs from the single-token chain in 196664 of 496640 logit bytes + (arm wall clock including model load: 9.2s) +PROBES2 DONE diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index 003ab4c78..447333768 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -62,7 +62,7 @@ Served, one binary, temperature 0: | classic, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | | | width 4, `MLXCEL_SDPA_VECTOR_LARGE_D=0` | `4c37650547` | none, 200 of 200 | -Both arms move, which is expected and is why this table is not evidence about classic decode; what matters is that they move onto each other. In process, replaying the served width-4 run's own 79 rounds and 56 rewinds against a classic chain computed in the same run, the same switch takes the target from 1 disagreement of 201 greedy positions to 0. +Both arms move, which is expected and is why this table is not evidence about classic decode; what matters is that they move onto each other. The in-process arm says the same thing in the same shape: replaying the served width-4 run's own 79 rounds and 56 rewinds against a classic chain computed in the same process under the same switch, the block disagrees with the chain at 1 of 201 greedy positions with the fused kernel and at 0 without it. That is block-against-chain under each setting, not the burst reproducing the recorded classic ids, because the chain side moves with the switch too. A single kill switch collapsing a 95-token divergence to nothing is stronger than any of the exclusions that preceded it, and it takes the remaining candidates with it. Rollback, the gated-delta scan, the quantized matmul kernel, the drafter, the round loop and the server process are none of them disabled by this switch, and all of them stop mattering when it is off. @@ -74,8 +74,35 @@ The obvious reading of "same math, one kernel, two answers" is that the two call Reading the CUDA gate confirms why. `supports_sdpa_vector` admits a call when `q.shape(2) < 4`, which both satisfy, and `sdpa_vector`'s own `q_copy_unless` and `kv_copy_unless` predicates accept both layouts without copying: a `[1, H, 1, D]` query passes on `strides[3] == 1 && strides[2] == D * H && strides[1] == D`, which the block's slice satisfies as well as decode's tensor does, and a key or value whose batch dimension is 1 is accepted whatever its strides. The 1-pass and 2-pass split is on `k.shape(2) > 1024`, which neither reaches. So the arguments the kernel sees are the same shape, the same strides and the same values, and the difference is in what MLX does around the call rather than in what is handed to it. +## Where it starts, and in which layer + +A byte-level bisect over the recorded transcript, driving the served width-4 run's own round structure and comparing logit bytes per kept row rather than argmaxes, places it precisely. Both arms prefill through `forward_prefill_with_capture_layers`; the chain arm steps one token at a time through `forward_speculative`, which the one-row arm above measured byte-identical to `forward_internal` over this whole transcript. + +The first row whose logits differ is at round 17, row 3, emitted index 37, absolute position 194: 230932 of 496640 logit bytes. Walking every layer's captured hidden state on that row, the first to differ is **layer 15, a full attention layer**, in 1016 of 5120 bytes. From there it is pervasive: 164 of the 200 kept rows differ in logit bytes, while only one of them, at 105, differs in argmax. That is the shape a sub-reporting-floor difference has, and it is why an argmax comparison over 201 positions can report a single disagreement for something that is happening almost everywhere. + +**The position is fixed, and the round structure is not what sets it.** Repeating the bisect on a fully synthetic 256-token prompt with four different accept patterns (`1`, `2`, `3`, `1,3`, and `1,2,3,4`) puts the first byte difference at emitted index 71, absolute position 326, in all five, with the same 208510 differing bytes and the same first layer. The patterns reach that position at round 70, 35, 23, 35 and 28 respectively. So the trigger is a position in the sequence, not a number of rounds, a number of rewinds or a particular accept shape. At the served 158-token prompt that position is 194, thirty-six tokens past the prompt; at 256 it is 326, seventy tokens past it. + ## Why the gate reported the property intact -`Qwen35Model::probe_block_chain_exactness` compares a verify block against a single-token chain in logit bytes, which is the right comparison. It prefills 8 synthetic tokens first: `PROBE_PROMPT_LEN` is 8, chosen so the attention layers "hold a real KV prefix rather than the empty-cache special case". An 8-token prefix is not a served KV layout. The cache has not grown past its first step-aligned allocation, and the difference this record is about does not appear there. +`Qwen35Model::probe_block_chain_exactness` compares a verify block against a single-token chain in logit bytes, which is the right comparison, and the one-row arm above is what establishes that it is: `forward_speculative` at one row IS `forward_internal`, so block-against-chain is block-against-classic-decode. The probe is measuring the right pair. + +It is measuring it in the wrong place. The probe prefills, runs one block, and compares. The difference does not exist yet there. Swept over prompt lengths 8, 32, 64, 128, 256 and 512 on the real checkpoint, the verdict is byte-identity at widths 2 and 4 at **every** length, and divergence at 8 and 16 at every length. Lengthening the prefix is not the answer, and the original 8 was not the reason. + +A probe shaped like a served burst does catch it. The bisect arm, given a synthetic prompt and a synthetic accept cycle rather than the recorded transcript, reports the divergence at every accept pattern tried. What it needs is to keep going past the prompt, which the shipped probe does not do at any length. + +## What this change does about it + +It declines, and says why, rather than measuring something it cannot see. + +`probe_block_chain_exactness` now returns `NotRun` with a reason before its draws when the checkpoint and host are the configuration measured here: CUDA, a `head_dim` of 256 or 288, and `MLXCEL_SDPA_VECTOR_LARGE_D` left enabled. `mtp_exactness_gate` already treats every non-`Equal` verdict as a decline, so both Qwen 3.5 `DFlashTargetModel` impls decline the burst at every width and the request is served by classic decode, byte-identical to a drafter-less server. A probe that cannot observe a hazard reporting a pass is worse than one that fails, which is what the gate was doing at widths 2 and 4. + +Two escapes remain and both are honest. `MLXCEL_MTP_ALLOW_INEXACT=1` engages the burst anyway and logs the forfeit, which is what an operator who wants the throughput and does not need byte-identity should set. `MLXCEL_SDPA_VECTOR_LARGE_D=0` buys the contract back rather than forfeiting it: it moves single-query attention off the fused kernels on both paths, the served completions become byte-identical again, and the cost is classic decode's own fused attention kernel (issue #675 measured what that is worth). + +This is a narrowing of where DFlash engages on this family, and it is deliberate. The alternative is the status quo, in which an operator who turns on speculative decoding gets different greedy text than without it and nothing says so. + +## What remains unnamed + +What the fused `sdpa_vector` path does differently between the two calls is not identified here, and this record does not guess. What is established: the per-row query, key and value tensors are the same shape, the same strides and the same values in both paths, CUDA's `supports_sdpa_vector` admits both, `sdpa_vector`'s own copy predicates copy neither, and the 1-pass and 2-pass split is on a key length neither reaches. The difference is therefore in what MLX does around the call rather than in what is handed to it, and naming it means reading MLX's own graph-level decisions rather than mlxcel's. + +One discrepancy is recorded rather than explained. PR #1939's replay of the served width-4 accept sequence reported 0 disagreements of 201; the same arm on the same accept pattern reports 1 here, at 105. The accept vector this session used is committed at `data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt`. -Measured on the real checkpoint on this host, the probe reports byte-identity at widths 2 and 4 and divergence at 8 and 16, which is what PR #1939 recorded and wired the gate to. The served arms disagree with it at widths 2 and 4. So the gate's verdict was a false pass, not a correct pass that something downstream then violated, and the probe needs a prefix long enough to reproduce the layout a served request has before its verdict means anything. diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 82ba324d4..2a344bfaf 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -462,6 +462,7 @@ recommended as normal deployment settings. | `MLXCEL_SPARSE_PAGED_DUMP` | `1`/`true`/`on`/`yes` enable | off | Prints each request's selected pool rows for a fused sparse decode (#904). **Synchronizes with the device**, so it is a debugging aid only and must never be set for a timed run. | | `MLXCEL_SPARSE_PAGED_MIN_SPARSITY` | non-negative integer | `8` | Minimum `live_len / selected_rows` ratio a launch must clear before fused sparse decode is dispatched (#904). The fused kernel measured 0.67x at 2x sparsity and 1.22x at 8x against tuned dense SDPA, so a token floor alone is not sufficient. `0` disables the gate for benchmarking the declined regime; an unparseable value falls back to the default. See [`sparse-paged-decode.md`](sparse-paged-decode.md). | | `MLXCEL_SDPA_VECTOR_LARGE_D` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | **CUDA only.** Gates whether the CUDA `supports_sdpa_vector` check accepts head_dim 256/288 (gemma family, qwen3.5/3.6, baichuan-m1, paligemma2), routing their decode to the fused `sdpa_vector` kernels instead of the materializing SDPA fallback (issue #675). Disabling restores the prior fallback with no rebuild; used for the A/B in `benchmarks/cuda_gb10_sdpav_675_2026-07-06.csv`. | +| `MLXCEL_MTP_PROBE_PROMPT_LEN` | integer `>= 2` | `8` | Prompt length `Qwen35Model::probe_block_chain_exactness` prefills before comparing a verify block against a single-token chain. Issue #1935 swept it over 8, 32, 64, 128, 256 and 512 on `qwen3.5-4b-4bit` looking for a length at which the probe would notice the divergence a served burst has at verify widths 2 and 4, and found none: the verdict is byte-identity at every length, because what the probe misses is its own shape (one block immediately after a clean prefill) rather than a short prefix. The knob is kept so that sweep can be repeated on another checkpoint; the default is what it always was. Longer values cost one prefill per probe draw per width at worker startup, paid once and memoized. See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` | non-negative integer | `0` (off) | **CUDA only. Off by default; set to `32` to enable.** #1820 measured this path behind `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s ops fallback at three of five block widths with disjoint ranges (0.943x, 0.953x, 0.915x at widths 2, 8 and 16 on a 152-token prompt) and a wash at 2634 tokens, so the shipped dispatch is #1799's alone and this is opt-in. It is kept rather than removed because the mechanism is sound (12 plan builds per generation instead of 82, greedy token ids byte-identical to exact-shape cuDNN) and a crossover above 2634 keys is plausible and unmeasured: this path's cost is flat in context while the fallback's score matrix grows with the key length. See `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-17.md`. An array-masked SDPA call with 2 to N query rows over a KV cache (the speculative verify shape: a block appended to a cache) has its cuDNN execution-plan cache key bucketed, so one plan serves every round until the cache buffer grows (issue #1820). cuDNN keys that cache on the exact shapes and strides of q, k, v and the mask, and a verify round's key length is new every round, so before this every round of every attention-layer class rebuilt a plan on the host: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing. The fix is MLX's own one-row decode canonicalization extended to these calls: k and v are unsliced to the whole cache buffer, the additive mask is widened to the same width with the new columns set to `-inf`, and the true lengths reach cuDNN through `set_padding_mask` with `set_seq_len_q` / `set_seq_len_kv`, which is what keeps the widened region out of the result. k and v reach the bucket two ways. Where they are already a leading slice of one contiguous cache buffer, they are unsliced to that buffer's own allocated extent, so nothing is reallocated and no position outside the already-allocated cache is addressed; mlxcel's KV cache grows in 256-position steps, so one plan per class covers 256 appends. Where they are not (the drafter concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room past the live length) they are copied into a zero-padded buffer rounded up to the same 256 grid, which costs a copy per call and is bounded per tensor by `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB`. Above that bound the call keeps its exact shape and pays the plan build, and a shape-eligible call whose layout declines bucketing for any reason prints a one-time warning naming this variable, because such a call gets neither this fix nor `MLXCEL_SDPA_FALLBACK_MAX_QUERIES`'s. `0` disables the bucketing without a rebuild, which restores the dispatch `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` alone produced and is the kill switch used for the A/B in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md`. Non-CUDA builds ignore it. | | `MLXCEL_SDPA_PLAN_BUCKET_MAX_MB` | non-negative integer (MiB) | `64` | **CUDA only.** Per-tensor ceiling on the zero-padded k/v copy the bucketing arm above makes when k and v are not a leading slice of one contiguous cache buffer, which is the drafter's shape: it concatenates its proposal keys onto the cache window, so its k and v are freshly built arrays with no room to unslice into (issue #1820). A call whose widened k or v would exceed this keeps its exact shape and rebuilds a plan each round, and prints the one-time warning described above, because moving hundreds of MiB per layer per round would cost more than the roughly 22 ms plan build it saves. At the default 64 MiB an 8-kv-head 128-dim bf16 cache buckets up to about 32k keys. `0` disables the copy arm, leaving only the free unslice arm. | | `MLXCEL_SDPA_PLAN_DEBUG` | `1` enables | off (`0`) | **CUDA only, diagnostic.** Writes one line per cuDNN SDPA call to stderr with the key fields that decide plan reuse: the q shape, `k_len`, the cache buffer extent and row stride, the mask column count and strides, the causal and sinks flags, whether the call took the decode or the bucketed canonicalization, and whether this call built a plan or reused one, plus the resident plan count. This is how the per-shape-class key-field table and the plan-build counts in `docs/benchmark_results/sdpa-plan-cache-bucket-gb10-2026-09-12.md` were produced (issue #1820). It prints per attention call per layer, so it is a diagnostic aid and not something to leave on. | diff --git a/src/models/qwen3_5.rs b/src/models/qwen3_5.rs index d903cd0e4..0eca4ef8f 100644 --- a/src/models/qwen3_5.rs +++ b/src/models/qwen3_5.rs @@ -53,18 +53,22 @@ use std::path::Path; /// Prompt length the exactness probe prefills before comparing arms, unless /// `MLXCEL_MTP_PROBE_PROMPT_LEN` overrides it. /// -/// This was 8 until issue #1935, on the reasoning that 8 is enough for the -/// attention layers to hold a real KV prefix rather than the empty-cache -/// special case. It is not. At 8 the probe reported byte-identity at verify -/// widths 2 and 4 on `qwen3.5-4b-4bit` while a served burst at those widths -/// diverged from classic decode, because the difference the probe exists to -/// catch is a kernel one and short key sequences do not reach the code path -/// that carries it. A probe that prefills a served-sized prefix catches it. -/// The cost is one prefill per draw per width at worker startup, paid once -/// and memoized by `mtp_exactness_gate`. +/// Long enough that the attention layers hold a real KV prefix rather than the +/// empty-cache special case, short enough that the probe stays a fraction of a +/// second on a 27B target. +/// +/// Issue #1935 swept this over 8, 32, 64, 128, 256 and 512 on +/// `qwen3.5-4b-4bit`, looking for a length at which the probe would notice the +/// divergence a served burst has at verify widths 2 and 4. There is none: the +/// verdict is byte-identity at every length. What the probe misses is not a +/// short prefix but its own shape, one block immediately after a clean +/// prefill, and on that checkpoint the difference does not begin until several +/// dozen tokens past the prompt (position 194 at a 158-token prompt, 326 at a +/// 256-token one, in both cases whatever the accept pattern). The knob is kept +/// because that sweep is worth being able to repeat; the default is unchanged. /// /// See `docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md`. -const DEFAULT_PROBE_PROMPT_LEN: usize = 256; +const DEFAULT_PROBE_PROMPT_LEN: usize = 8; /// The probe's prompt length for this process. /// @@ -1628,6 +1632,37 @@ impl Qwen35Model { /// /// Used by: [`Self::mtp_exactness_allows`], and the /// `metal_block_vs_chain_op_parity` diagnostic's model-level sibling. + /// Whether this checkpoint on this host is the configuration issue #1935 + /// measured a verify block failing in, and which the probe cannot observe. + /// + /// `None` when the probe's verdict can be trusted. `Some(reason)` when it + /// cannot, which the caller turns into a decline. + fn cuda_sdpa_vector_verify_hazard(&self) -> Option<&'static str> { + if !mlxcel_core::cuda_is_available() { + return None; + } + // The same gate CUDA's `supports_sdpa_vector` applies, read from this + // checkpoint's geometry and from the kill switch that governs it. + let head_dim = self.config.head_dim_resolved(); + if head_dim != 256 && head_dim != 288 { + return None; + } + let fused_enabled = std::env::var("MLXCEL_SDPA_VECTOR_LARGE_D") + .map(|v| { + let v = v.trim().to_ascii_lowercase(); + !(v == "0" || v == "false" || v == "off" || v == "no") + }) + .unwrap_or(true); + if !fused_enabled { + return None; + } + Some( + "on CUDA this head_dim reaches the fused sdpa_vector kernels, where a verify \ + block's per-position attention is not bit-equal to the single-token decode it \ + stands for (issue #1935); MLXCEL_SDPA_VECTOR_LARGE_D=0 restores it", + ) + } + pub fn probe_block_chain_exactness(&self, block_size: usize) -> BlockChainExactness { if block_size < 2 { return BlockChainExactness::NotRun("block width below 2 drafts nothing"); @@ -1637,6 +1672,27 @@ impl Qwen35Model { return BlockChainExactness::NotRun("degenerate vocabulary"); } + // A configuration the probe cannot observe, which is worse than one it + // fails, because a probe that cannot see a hazard reports a pass + // (issue #1935). On CUDA, `head_dim` 256 and 288 reach the fused + // `sdpa_vector` kernels (issue #675), and there a verify block's + // per-position attention stops being bit-equal to the single-token + // decode it stands for. Measured on GB10: the served greedy + // completion parts from classic decode at every verify width, and + // `MLXCEL_SDPA_VECTOR_LARGE_D=0` makes the two byte-identical over + // all 200 tokens. The probe's own arms never see it, because the + // difference does not begin until several dozen tokens past the + // prompt while the probe compares one block immediately after a clean + // prefill, and no prompt length from 8 to 512 changes that. + // + // So this declines rather than measuring. `MLXCEL_MTP_ALLOW_INEXACT=1` + // engages the burst anyway and forfeits the contract, and + // `MLXCEL_SDPA_VECTOR_LARGE_D=0` buys the contract back at the cost of + // classic decode's fused attention kernel. + if let Some(why) = self.cuda_sdpa_vector_verify_hazard() { + return BlockChainExactness::NotRun(why); + } + for draw in 0..PROBE_DRAWS { let verdict = self.probe_one_draw(block_size, vocab, draw); if !verdict.is_equal() { From 0ef4f8eff5de071ed486c088cc4bbb90be699e43 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:01:35 +0900 Subject: [PATCH 09/16] test(speculative): satisfy clippy in the byte-bisect arm `needless_range_loop` and `collapsible_if` on the new diagnostic, caught by `cargo clippy -p mlxcel --lib --tests -- -D warnings`, which is the gate CI runs and which a `--lib`-only check never reaches because the arm is `cfg(test)`. Refs #1935 --- src/models/qwen3_5_dflash_probe_tests.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/models/qwen3_5_dflash_probe_tests.rs b/src/models/qwen3_5_dflash_probe_tests.rs index 998f56c41..a1c999e16 100644 --- a/src/models/qwen3_5_dflash_probe_tests.rs +++ b/src/models/qwen3_5_dflash_probe_tests.rs @@ -1028,8 +1028,8 @@ fn block_versus_chain_byte_bisect_on_the_real_transcript() { } } let burst = text.forward_speculative(&ids_of(&fed), &mut burst_caches, &capture); - for r in 0..keep { - let chain = text.forward_speculative(&ids_of(&[fed[r]]), &mut chain_caches, &capture); + for (r, tok) in fed.iter().enumerate().take(keep) { + let chain = text.forward_speculative(&ids_of(&[*tok]), &mut chain_caches, &capture); let cb = row_bytes(&chain.logits, 0); let bb = row_bytes(&burst.logits, r as i32); let d = differing(&cb, &bb); @@ -1049,8 +1049,8 @@ fn block_versus_chain_byte_bisect_on_the_real_transcript() { let c = row_bytes(&chain.hidden_states[layer], 0); let b = row_bytes(&burst.hidden_states[layer], r as i32); let dl = differing(&c, &b); - if dl > 0 { - if first_layer.is_none() { + if dl > 0 && first_layer.is_none() { + { first_layer = Some(layer); eprintln!( "[1935] FIRST layer divergence at layer {layer} ({}): \ From da53deaf52c3f6261c641d25a6c93e06014e227f Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:03:18 +0900 Subject: [PATCH 10/16] test(speculative): derive the residual harness's paths instead of hardcoding them The four shell arms carried the absolute scratchpad and worktree paths of the session that produced them, which makes them unrunnable by anyone else and quietly wrong for whoever tries. They now derive the harness directory from their own location, the repo from git, and the output directory and binaries from overridable variables, so a rerun can point at a binary built anywhere. Refs #1935 --- .../harness/ab_fix.sh | 12 ++++++++---- .../harness/ab_sdpav.sh | 12 ++++++++---- .../harness/run_probes.sh | 17 +++++++++++------ .../harness/run_probes2.sh | 17 +++++++++++------ 4 files changed, 38 insertions(+), 20 deletions(-) diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh index cd9e86575..4e4771790 100755 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_fix.sh @@ -3,16 +3,20 @@ # arm from the same binary. `0` is the kill switch that restores the pre-fix # slicing, so the same binary produces both sides. set -uo pipefail -SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad -WT=/home/inureyes/Development/mlxcel-wt-1935 -BIN=/home/inureyes/Development/mlxcel/target/release/mlxcel-server +# Paths are derived, not hardcoded: SP is this harness directory, WT the repo +# it lives in, and BIN/TEST_BIN are overridable so a run can point at a binary +# built anywhere. +SP=${SP:-$(cd "$(dirname "$0")" && pwd)} +WT=${WT:-$(git -C "$SP" rev-parse --show-toplevel)} +OUT=${OUT:-$SP/../arms} +BIN=${BIN:-$WT/target/release/mlxcel-server} H=$WT/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness run() { # width tag extra-env python3 "$SP/transcript.py" --server "$BIN" \ --target "$WT/models/mlx/qwen3.5-4b-4bit" \ --drafter "$WT/models/mlx/qwen3.5-4b-dflash" \ --width "$1" --n 1 --max-tokens 200 --tag "$2" --extra-env "$3" \ - --prompt-file "$H/prompt_retry.txt" --outdir "$SP/arms" --port 18937 + --prompt-file "$H/prompt_retry.txt" --outdir "$OUT" --port 18937 echo "=== $2 done ===" } run classic fix-classic "MLXCEL_QWEN35_ATTEND_CONTIGUOUS=q" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh index 9dd5d4b46..bad3da385 100755 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/ab_sdpav.sh @@ -7,9 +7,13 @@ # the fallback and they converge. If they still differ, the kernel choice is # not the mechanism. set -uo pipefail -SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad -WT=/home/inureyes/Development/mlxcel-wt-1935 -BIN=/home/inureyes/Development/mlxcel/target/release/mlxcel-server +# Paths are derived, not hardcoded: SP is this harness directory, WT the repo +# it lives in, and BIN/TEST_BIN are overridable so a run can point at a binary +# built anywhere. +SP=${SP:-$(cd "$(dirname "$0")" && pwd)} +WT=${WT:-$(git -C "$SP" rev-parse --show-toplevel)} +OUT=${OUT:-$SP/../arms} +BIN=${BIN:-$WT/target/release/mlxcel-server} H=$WT/docs/benchmark_results/data/draft-block-width-gb10-2026-09-20/harness for arm in "classic:cls-novec" "4:w4-novec"; do W=${arm%%:*}; T=${arm##*:} @@ -18,7 +22,7 @@ for arm in "classic:cls-novec" "4:w4-novec"; do --drafter "$WT/models/mlx/qwen3.5-4b-dflash" \ --width "$W" --n 1 --max-tokens 200 --tag "$T" \ --extra-env "MLXCEL_SDPA_VECTOR_LARGE_D=0" \ - --prompt-file "$H/prompt_retry.txt" --outdir "$SP/arms" --port 18936 + --prompt-file "$H/prompt_retry.txt" --outdir "$OUT" --port 18936 echo "=== $T done ===" done echo "AB DONE" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh index 7aaef3f10..15c2d2b75 100755 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes.sh @@ -2,19 +2,24 @@ # The in-process arms, run against an already-built test binary by path so a # concurrent rebuild of the server cannot reshape them mid-run. set -uo pipefail -SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad -WT=/home/inureyes/Development/mlxcel-wt-1935 -TB=${TEST_BIN:-/home/inureyes/Development/mlxcel/target/release/deps/mlxcel-69c784c732fe3546} +# Paths are derived, not hardcoded: SP is this harness directory, WT the repo +# it lives in, and BIN/TEST_BIN are overridable so a run can point at a binary +# built anywhere. +SP=${SP:-$(cd "$(dirname "$0")" && pwd)} +WT=${WT:-$(git -C "$SP" rev-parse --show-toplevel)} +OUT=${OUT:-$SP/../arms} +BIN=${BIN:-$WT/target/release/mlxcel-server} +TB=${TEST_BIN:?set TEST_BIN to the cargo lib-test binary} cd "$WT" || exit 1 export MLX_CUDA_ARCHITECTURES=121 MLX_ENABLE_TF32=1 -export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/prompt_ids.json")))))')" -export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/classic_ids.json")))))')" +export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/../prompt_ids.json")))))')" +export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/../classic_ids.json")))))')" echo "=== ARM 3: speculative against classic at ONE ROW, logit BYTES ===" "$TB" --ignored --test-threads=1 --nocapture speculative_t1_forward_versus_classic_on_the_real_transcript 2>&1 | grep -E "^\[1935\]|test result|panicked" echo "=== ARM 1: replay the served width-4 round structure, no drafter ===" -MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$SP/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ +MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$OUT/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ "$TB" --ignored --test-threads=1 --nocapture round_loop_cache_dynamics_with_rollback_match_the_chain 2>&1 | grep -E "^\[1935\]|test result|panicked" echo "=== ARM 2: the served burst wrapper with the real drafter ===" diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh index 334505e5d..9f6fa60db 100755 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/harness/run_probes2.sh @@ -1,16 +1,21 @@ #!/usr/bin/env bash set -uo pipefail -SP=/tmp/claude-1000/-home-inureyes-Development-mlxcel/a7ac83cc-0ca7-4ae0-8f19-a24a07471f2a/scratchpad -WT=/home/inureyes/Development/mlxcel-wt-1935 -TB=${TEST_BIN:-/home/inureyes/Development/mlxcel/target/release/deps/mlxcel-e4caea9f4277a3b5} +# Paths are derived, not hardcoded: SP is this harness directory, WT the repo +# it lives in, and BIN/TEST_BIN are overridable so a run can point at a binary +# built anywhere. +SP=${SP:-$(cd "$(dirname "$0")" && pwd)} +WT=${WT:-$(git -C "$SP" rev-parse --show-toplevel)} +OUT=${OUT:-$SP/../arms} +BIN=${BIN:-$WT/target/release/mlxcel-server} +TB=${TEST_BIN:?set TEST_BIN to the cargo lib-test binary} cd "$WT" || exit 1 export MLX_CUDA_ARCHITECTURES=121 MLX_ENABLE_TF32=1 -export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/prompt_ids.json")))))')" -export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/classic_ids.json")))))')" +export MLXCEL_Q35_PROBE_PROMPT="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/../prompt_ids.json")))))')" +export MLXCEL_Q35_PROBE_REFERENCE="$(python3 -c 'import json;print(",".join(map(str,json.load(open("'"$SP"'/../classic_ids.json")))))')" echo "test binary: $TB" echo "=== BISECT: where do logit bytes first differ, and in which layer ===" -MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$SP/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ +MLXCEL_Q35_PROBE_BLOCK=4 MLXCEL_Q35_PROBE_ACCEPTS="$(cat "$OUT/accepts_w4.txt")" MLXCEL_Q35_PROBE_WRONG=9999 \ "$TB" --ignored --test-threads=1 --nocapture block_versus_chain_byte_bisect_on_the_real_transcript 2>&1 | grep -E "^\[1935\]|panicked" echo "=== PROBE VERDICTS against prompt length ===" From f830f4698d050961331c986721dcd9c2a7d69d95 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:07:34 +0900 Subject: [PATCH 11/16] fix(speculative): read the fused-kernel switch narrowly in the exactness gate MLX reads `MLXCEL_SDPA_VECTOR_LARGE_D` as an integer defaulting to 1, and the gate now reads it more narrowly than mlxcel's own documented spelling: only a value that parses to zero counts as off. Erring narrow declines a burst that might have been safe. Erring wide would report "safe" for a process in which MLX still takes the fused kernels, which is the one direction a safety gate must not be wrong in. Also records the session's driver budget in the record's host block: the boot's cumulative `NV_ERR_NO_MEMORY` count was 0 before and 0 after, across roughly a dozen server starts and two dozen in-process model loads. Refs #1935 --- ...sh-width-2-4-residual-qwen35-gb10-2026-09-21.md | 2 +- src/models/qwen3_5.rs | 14 ++++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index 447333768..ade9134db 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -19,7 +19,7 @@ MLX_CUDA_ARCHITECTURES: 121 mem_total_gib: 121.7 ``` -Driver budget: the boot's cumulative kernel `NV_ERR_NO_MEMORY` count was 0 before this session. Nothing here is a timing measurement, so every arm ran while the host was also compiling; greedy token streams do not depend on host load, and the two classic arms below are the control that says so. +Driver budget: the boot's cumulative kernel `NV_ERR_NO_MEMORY` count was 0 before this session and 0 after it, across roughly a dozen server starts and two dozen in-process model loads. Nothing here is a timing measurement, so every arm ran while the host was also compiling; greedy token streams do not depend on host load, and the two classic arms below are the control that says so. ## Method diff --git a/src/models/qwen3_5.rs b/src/models/qwen3_5.rs index 0eca4ef8f..a45f343a7 100644 --- a/src/models/qwen3_5.rs +++ b/src/models/qwen3_5.rs @@ -1647,11 +1647,17 @@ impl Qwen35Model { if head_dim != 256 && head_dim != 288 { return None; } + // MLX reads this switch as `env::get_var("MLXCEL_SDPA_VECTOR_LARGE_D", 1)`, + // an integer defaulting to 1, and this deliberately reads it more + // narrowly than mlxcel's own documented spelling: only a value that + // parses to zero counts as off. Erring narrow declines a burst that + // might have been safe; erring wide would report "safe" for a process + // in which MLX still takes the fused kernels, which is the one + // direction a safety gate must not be wrong in. let fused_enabled = std::env::var("MLXCEL_SDPA_VECTOR_LARGE_D") - .map(|v| { - let v = v.trim().to_ascii_lowercase(); - !(v == "0" || v == "false" || v == "off" || v == "no") - }) + .ok() + .and_then(|v| v.trim().parse::().ok()) + .map(|v| v != 0) .unwrap_or(true); if !fused_enabled { return None; From c86574ad6648ed6d7acf7b000266ff52487c532e Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:10:07 +0900 Subject: [PATCH 12/16] docs(benchmarks): record the served-path parity result and the hole in it `greedy_parity_dflash_qwen35_4b` passes at widths 2, 4, 8 and 16 on GB10 with every width declined by a measured verdict and every response equal to the drafter-less baseline. It used to fail at all four. The record also names a weakness in that test rather than leaving it to be discovered: its chat prompt at its token budget returns an empty `content` on this checkpoint, baseline included, so its byte comparison of `content` compares two empty strings and the weight is carried by the completion-token equality and the decline assertions. That is not introduced here and not what this issue is about, but the test is weaker evidence than its name suggests. The byte-identity claims in the record rest on the `/v1/completions` arms, which return 823 characters compared by sha256. Refs #1935 --- .../parity_test.txt | 3 +++ .../dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md | 6 ++++++ 2 files changed, 9 insertions(+) create mode 100644 docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/parity_test.txt diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/parity_test.txt b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/parity_test.txt new file mode 100644 index 000000000..6ccbc3c23 --- /dev/null +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/parity_test.txt @@ -0,0 +1,3 @@ +[Qwen 3.5 4B + DFlash (b=16)] widths that ran the burst: []; widths the gate declined: [2, 4, 8, 16] +[Qwen 3.5 4B + DFlash (b=16)] every width declined by a measured exactness verdict, and every response equalled the drafter-less baseline byte for byte. That is the contract holding through the gate rather than through the verify block. Widths tried: [2, 4, 8, 16] +test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 9 filtered out; finished in 31.62s diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index ade9134db..7a5c1bb05 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -106,3 +106,9 @@ What the fused `sdpa_vector` path does differently between the two calls is not One discrepancy is recorded rather than explained. PR #1939's replay of the served width-4 accept sequence reported 0 disagreements of 201; the same arm on the same accept pattern reports 1 here, at 105. The accept vector this session used is committed at `data/dflash-width-2-4-residual-gb10-2026-09-21/arms/accepts_w4.txt`. + +## The served-path test after the change + +`greedy_parity_dflash_qwen35_4b` at widths 2, 4, 8 and 16 on GB10, against the drafter-less baseline from the same binary: `widths that ran the burst: []; widths the gate declined: [2, 4, 8, 16]`, every response equal to the baseline, `test result: ok`. It used to fail at all four. Each declined width now has to carry the exactness probe's own decline line before the arm counts, so a multimodal payload, an adopted prompt-cache prefix or a drafter from the wrong family cannot stand in for a measured verdict. + +One weakness in that test is worth naming rather than leaving for someone to discover. Its chat prompt at its token budget returns an empty `content` on this checkpoint (`completion_tokens=96, content.len()=0` in every arm, baseline included), so its byte comparison of `content` is comparing two empty strings and the weight is carried by the completion-token equality and by the decline assertions. That is not something this change introduced and it is not what this issue is about, but the test is weaker evidence than its name suggests and should either compare the reasoning channel too or use a prompt that produces content. The byte-identity claims in this record rest on the `/v1/completions` arms above, which return 823 characters and are compared by sha256, not on that test. From 2fb00b52bf22fd7cf2afb91fe75641d3958145a8 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:14:04 +0900 Subject: [PATCH 13/16] docs(benchmarks): correct the accept-pattern count in the residual record Five patterns were run, not four; the list and the per-pattern round numbers after it were already five long. Refs #1935 --- .../dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md index 7a5c1bb05..4accddf0c 100644 --- a/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-width-2-4-residual-qwen35-gb10-2026-09-21.md @@ -80,7 +80,7 @@ A byte-level bisect over the recorded transcript, driving the served width-4 run The first row whose logits differ is at round 17, row 3, emitted index 37, absolute position 194: 230932 of 496640 logit bytes. Walking every layer's captured hidden state on that row, the first to differ is **layer 15, a full attention layer**, in 1016 of 5120 bytes. From there it is pervasive: 164 of the 200 kept rows differ in logit bytes, while only one of them, at 105, differs in argmax. That is the shape a sub-reporting-floor difference has, and it is why an argmax comparison over 201 positions can report a single disagreement for something that is happening almost everywhere. -**The position is fixed, and the round structure is not what sets it.** Repeating the bisect on a fully synthetic 256-token prompt with four different accept patterns (`1`, `2`, `3`, `1,3`, and `1,2,3,4`) puts the first byte difference at emitted index 71, absolute position 326, in all five, with the same 208510 differing bytes and the same first layer. The patterns reach that position at round 70, 35, 23, 35 and 28 respectively. So the trigger is a position in the sequence, not a number of rounds, a number of rewinds or a particular accept shape. At the served 158-token prompt that position is 194, thirty-six tokens past the prompt; at 256 it is 326, seventy tokens past it. +**The position is fixed, and the round structure is not what sets it.** Repeating the bisect on a fully synthetic 256-token prompt with five different accept patterns (`1`, `2`, `3`, `1,3` and `1,2,3,4`) puts the first byte difference at emitted index 71, absolute position 326, in all five, with the same 208510 differing bytes and the same first layer. The patterns reach that position at round 70, 35, 23, 35 and 28 respectively. So the trigger is a position in the sequence, not a number of rounds, a number of rewinds or a particular accept shape. At the served 158-token prompt that position is 194, thirty-six tokens past the prompt; at 256 it is 326, seventy tokens past it. ## Why the gate reported the property intact From 6d095c863aa83a1f492797ad132ce08ac14acd2d Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 15:16:37 +0900 Subject: [PATCH 14/16] docs(benchmarks): finish the residual data README Names the two files added after it was first written, and records that two arms of the probe-length sweep aborted with this host's known CUDA `cudaStreamEndCapture` artifact rather than producing a result. The lengths that completed all report the same verdict, and 512 completed. Refs #1935 --- .../data/dflash-width-2-4-residual-gb10-2026-09-21/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md index f35dd456f..5c0d0a4a2 100644 --- a/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md +++ b/docs/benchmark_results/data/dflash-width-2-4-residual-gb10-2026-09-21/README.md @@ -29,3 +29,7 @@ Raw data for the residual PR #1939 left open: on `models/mlx/qwen3.5-4b-4bit` wi `probe_verdicts.txt` is `probe_block_chain_exactness` on the real checkpoint, and `probe_arms.txt` is the three in-process arms: the one-row byte walk, the replay of the served round structure with no drafter, and the served burst wrapper with the real drafter bound. `arms/accepts_w4.txt` and `arms/accepts_w2.txt` are the per-round kept-row counts the served runs produced, which is what `MLXCEL_Q35_PROBE_ACCEPTS` takes so an in-process replay reproduces a served run's own cache history rather than an approximation of it. + +`probe_length_sweep_and_bisect.txt` is the byte-level bisect over the recorded transcript followed by the exactness probe's verdict at prompt lengths 8, 32, 64, 128, 256 and 512. Two of those arms aborted mid-run with a CUDA `cudaStreamEndCapture` abort, which is this host's known parallelism artifact rather than a result; the lengths that completed all report the same verdict, and 512 completed. + +`parity_test.txt` is `greedy_parity_dflash_qwen35_4b` at widths 2, 4, 8 and 16 after the change. From 85f73c3fa57121bbabd7571888d2cd5376455b0c Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 17:37:12 +0900 Subject: [PATCH 15/16] docs(benchmarks): price every option for byte-identical Qwen 3.5 DFlash Before choosing what PR #1944 should be, a question had to be settled: which side of the comparison does CUDA's fused `sdpa_vector` kernel actually serve. Both. `supports_sdpa_vector` gates on the query row count of the individual call, and `attend_per_position` slices the verify block to one row per call precisely so each row computes what a decode step computes, so classic decode and every verify row alike arrive with `q.shape(2) == 1`. Measured rather than inferred: `MLXCEL_SDPA_VECTOR_LARGE_D=0` moves both sides, and the burst's only sub-4-row attention calls are those verify rows. That kills the idea of disabling the fused path only for the verify block: it would put the two sides on different kernels, which is the one configuration guaranteed not to be byte-identical, since byte-identity here is exactly the two sides agreeing on a kernel. Seven arms in one session on a verified `origin/main` binary, n = 3, classic brackets at both ends and overlapping, with the 1.29 tok/s union spread reported as the resolution floor and every smaller difference called unresolved. Byte-identical speculation is achievable at 1.13x by running the process on the fallback; without the byte-identity requirement it is 1.22x; declining is 1.00x. The fused kernel is worth about 8% to the burst and nothing measurable to classic decode at this prompt length, which is not a general result because the fallback's cost grows with key length. The fallback costs the burst twice, in per-row attention and in acceptance (0.599 to 0.526 at width 3), and it changes what classic decode produces, so it is not a free correctness switch. Refs #1935 --- .../harness/price_options.py | 184 ++++++++++++++++++ .../logs/server.classic-close.log | 38 ++++ .../logs/server.classic-novec.log | 38 ++++ .../logs/server.classic-open.log | 38 ++++ .../logs/server.w3-fused.log | 51 +++++ .../logs/server.w3-novec.log | 51 +++++ .../logs/server.w4-fused.log | 51 +++++ .../logs/server.w4-novec.log | 51 +++++ .../options.jsonl | 7 + ...sh-sdpav-options-qwen35-gb10-2026-09-21.md | 59 ++++++ 10 files changed, 568 insertions(+) create mode 100755 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/harness/price_options.py create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-close.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-novec.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-open.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-fused.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-novec.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-fused.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-novec.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/options.jsonl create mode 100644 docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/harness/price_options.py b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/harness/price_options.py new file mode 100755 index 000000000..53551e089 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/harness/price_options.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +"""Price the options for issue #1935's byte-identity problem, in one session. + +The question this answers is not "is the burst faster" but "what does each way +of getting byte-identical greedy text cost, measured against the same classic +brackets". Every arm is one `mlxcel-server` with its own environment, so the +arms differ in exactly the variable named in their tag and in nothing else. + +Reuses the #1797 harness rather than reimplementing it: the same prompt, the +same streaming request, the same discarded warm-up, and the same #1820 host +gate (sustained-quiet CPU predicate matching on `/proc//comm` with stopped +processes dropped, foreign-model check, memory floor, cumulative +NV_ERR_NO_MEMORY trip wire). Only the arm table and the per-arm environment are +new. + + price_options.py --server BIN --target DIR --drafter DIR --out run.jsonl +""" +import argparse +import hashlib +import json +import os +import signal +import subprocess +import sys +import re +import time + +HARNESS = os.path.dirname(os.path.abspath(__file__)) +S1797 = os.path.normpath( + os.path.join(HARNESS, "..", "..", "draft-block-width-gb10-2026-09-20", "harness") +) +sys.path.insert(0, S1797) +sys.path.insert( + 0, + os.path.normpath( + os.path.join(HARNESS, "..", "..", "sdpa-plan-bucket-gb10-2026-09-12", "harness") + ), +) +import hostgate # noqa: E402 + +_sweep = {} +with open(os.path.join(S1797, "sweep_server_widths.py")) as fh: + src = fh.read().split('if __name__ == "__main__"')[0] +_sweep["__file__"] = os.path.join(S1797, "sweep_server_widths.py") +exec(compile(src, _sweep["__file__"], "exec"), _sweep) +wait_health = _sweep["wait_health"] +first_model_id = _sweep["first_model_id"] +complete = _sweep["complete"] +DIAG_LINE = _sweep["DIAG_LINE"] +DIAG_FIELDS = _sweep["DIAG_FIELDS"] +DECLINE = _sweep["DECLINE"] + +# tag, draft width (None = no drafter), extra environment. +# +# The brackets are the SHIPPED configuration: no drafter, MLXCEL_SDPA_VECTOR_LARGE_D +# left at its default of on. Everything between them is priced against those. +ARMS = [ + ("classic-open", None, {}), + ("classic-novec", None, {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}), + ("w3-novec", 3, {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}), + ("w4-novec", 4, {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}), + ("w3-fused", 3, {}), + ("w4-fused", 4, {}), + ("classic-close", None, {}), +] + + +def sha(s): + return hashlib.sha256(s.encode()).hexdigest()[:10] + + +def server_cmd(a, width): + cmd = [a.server, "-m", a.target, "--port", str(a.port), "--ignore-eos", + "--max-batch-size", "1"] + if width is not None: + cmd += ["--model-draft", a.drafter, "--draft-kind", "dflash", + "--draft-block-size", str(width)] + return cmd + + +def run_arm(a, tag, width, arm_env, prompt): + gate_wait = hostgate.wait_quiet(log=sys.stderr) + driver_wait, nvrm_window, nvrm_before = hostgate.driver_gate(log=sys.stderr) + mem_wait, mem_avail = hostgate.mem_gate(log=sys.stderr) + + log_path = os.path.join(a.logdir, f"server.{tag}.log") + log = open(log_path, "w") + env = dict(os.environ) + env.setdefault("MLX_ENABLE_TF32", "1") + env.setdefault("RUST_LOG", "info") + env.setdefault("MLX_CUDA_ARCHITECTURES", "121") + env.update(arm_env) + srv = subprocess.Popen(server_cmd(a, width), env=env, stdout=log, + stderr=subprocess.STDOUT, start_new_session=True) + rec = {"arm": tag, "width": width, "arm_env": arm_env, + "cmd": server_cmd(a, width), "log": log_path, + "gate_wait_s": gate_wait, "driver_wait_s": driver_wait, + "mem_wait_s": mem_wait, "nvrm_total_before": nvrm_before, + "ci_job_running": hostgate.ci_job_running(), + "load1_before": os.getloadavg()[0]} + try: + if not wait_health(a.port): + rec["error"] = "server never became healthy" + return rec + model_id = first_model_id(a.port) + # Discarded: the first request pays MLX kernel and graph compilation. + complete(a.port, prompt, a.max_tokens, model_id) + runs = [] + for _ in range(a.n): + wall, text, chunks, ptok, ctok = complete( + a.port, prompt, a.max_tokens, model_id + ) + runs.append({"wall_s": round(wall, 3), + "e2e_tok_s": round(a.max_tokens / wall, 3), + "completion_tokens": ctok, "chars": len(text), + "sha": sha(text), "text": text}) + rec["runs"] = runs + rec["prompt_tokens"] = ptok + except Exception as exc: # noqa: BLE001 - recorded, not swallowed + rec["error"] = f"{type(exc).__name__}: {exc}" + finally: + try: + os.killpg(srv.pid, signal.SIGTERM) + srv.wait(timeout=120) + except Exception: + try: + os.killpg(srv.pid, signal.SIGKILL) + except Exception: + pass + log.close() + body = open(log_path, errors="replace").read() + rec["declined_to_classic"] = DECLINE in body + diags = [] + for m in DIAG_LINE.finditer(body): + fields = {} + for f in DIAG_FIELDS: + mm = re.search(rf"\b{f}=([0-9.]+)", m.group(1)) + if mm: + fields[f] = float(mm.group(1)) + if fields: + diags.append(fields) + rec["diagnostics"] = diags + _, nvrm_after = hostgate.nvrm_counts() + rec["nvrm_delta"] = nvrm_after - nvrm_before + return rec + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--server", required=True) + ap.add_argument("--target", required=True) + ap.add_argument("--drafter", required=True) + ap.add_argument("--n", type=int, default=3) + ap.add_argument("--max-tokens", type=int, default=200) + ap.add_argument("--prompt-file", default=os.path.join(S1797, "prompt_retry.txt")) + ap.add_argument("--port", type=int, default=18935) + ap.add_argument("--out", required=True) + ap.add_argument("--logdir", default=None) + a = ap.parse_args() + a.logdir = a.logdir or os.path.dirname(os.path.abspath(a.out)) + os.makedirs(a.logdir, exist_ok=True) + prompt = open(a.prompt_file).read() + + with open(a.out, "a") as f: + for tag, width, arm_env in ARMS: + rec = run_arm(a, tag, width, arm_env, prompt) + f.write(json.dumps(rec) + "\n") + f.flush() + rs = [r["e2e_tok_s"] for r in rec.get("runs", [])] + shas = sorted({r["sha"] for r in rec.get("runs", [])}) + print( + f"[{tag}] " + + (f"e2e {min(rs):.2f} to {max(rs):.2f} tok/s" if rs + else f"ERROR {rec.get('error')}") + + f" | sha {shas} | declined={rec.get('declined_to_classic')}" + + f" | nvrm +{rec.get('nvrm_delta')}", + flush=True, + ) + print("OPTIONS DONE", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-close.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-close.log new file mode 100644 index 000000000..459724907 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-close.log @@ -0,0 +1,38 @@ +2026-09-21T08:32:06.388765Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:32:06.388810Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:32:06.388958Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:32:06.389011Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:32:06.389013Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:32:06.389015Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:32:06.728110Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:32:06.728151Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:32:06.728726Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:32:07.076362Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:32:07.077026Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:32:07.441220Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.364s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.364153507 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:32:07.441298Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto) +2026-09-21T08:32:07.441707Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:32:07.441721Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:32:07.441747Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:32:07.442924Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:32:08.349916Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 907ms prompt_tokens=1 cached_tokens=0 generation_time_ms=907 +2026-09-21T08:32:08.350126Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:32:08.350216Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:32:08.351345Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:32:08.352138Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:32:08.352155Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:32:08.352159Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:32:08.352160Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:32:08.352161Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:32:08.352163Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:32:08.352164Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:32:08.352165Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:32:08.352167Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:32:08.352168Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:32:08.352169Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:32:08.352170Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:32:08.352172Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:32:13.046638Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 4001ms prompt_tokens=158 cached_tokens=0 generation_time_ms=4001 +2026-09-21T08:32:16.544432Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3494ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3494 +2026-09-21T08:32:20.002386Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3455ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3455 +2026-09-21T08:32:23.474290Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3470ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3470 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-novec.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-novec.log new file mode 100644 index 000000000..2c02eade2 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-novec.log @@ -0,0 +1,38 @@ +2026-09-21T08:25:56.091450Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:25:56.091535Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:25:56.091746Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:25:56.091816Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:25:56.091819Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:25:56.091821Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:25:56.414598Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:25:56.414707Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:25:56.415332Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:25:56.739114Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:25:56.739686Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:25:57.106188Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.366s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.36645844 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:25:57.106264Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto) +2026-09-21T08:25:57.106681Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:25:57.106695Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:25:57.106718Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:25:57.107906Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:25:58.193901Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 1086ms prompt_tokens=1 cached_tokens=0 generation_time_ms=1086 +2026-09-21T08:25:58.194056Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:25:58.194221Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:25:58.195192Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:25:58.196336Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:25:58.196353Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:25:58.196357Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:25:58.196358Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:25:58.196359Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:25:58.196361Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:25:58.196362Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:25:58.196363Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:25:58.196364Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:25:58.196365Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:25:58.196367Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:25:58.196368Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:25:58.196369Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:26:02.548640Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3794ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3794 +2026-09-21T08:26:06.000326Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3449ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3449 +2026-09-21T08:26:09.486522Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3483ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3483 +2026-09-21T08:26:12.970605Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3481ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3481 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-open.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-open.log new file mode 100644 index 000000000..0b7f43617 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-open.log @@ -0,0 +1,38 @@ +2026-09-21T08:24:42.438974Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:24:42.439076Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:24:42.439503Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:24:42.439592Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:24:42.439595Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:24:42.439597Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:24:42.763844Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:24:42.763975Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:24:42.765150Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:24:43.091814Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:24:43.092489Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:24:43.464169Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.372s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.371637052 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:24:43.464284Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto) +2026-09-21T08:24:43.464812Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:24:43.464826Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:24:43.464980Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:24:43.476647Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:24:44.362148Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 886ms prompt_tokens=1 cached_tokens=0 generation_time_ms=886 +2026-09-21T08:24:44.362207Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:24:44.362286Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:24:44.363335Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:24:44.364871Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:24:44.364888Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:24:44.364891Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:24:44.364892Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:24:44.364893Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:24:44.364895Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:24:44.364896Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:24:44.364897Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:24:44.364899Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:24:44.364900Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:24:44.364901Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:24:44.364902Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:24:44.364903Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:24:49.131829Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 4004ms prompt_tokens=158 cached_tokens=0 generation_time_ms=4004 +2026-09-21T08:24:52.658316Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3523ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3523 +2026-09-21T08:24:56.194555Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3534ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3534 +2026-09-21T08:24:59.688827Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3491ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3491 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-fused.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-fused.log new file mode 100644 index 000000000..848eb2182 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-fused.log @@ -0,0 +1,51 @@ +2026-09-21T08:29:39.150456Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:29:39.150503Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:29:39.150642Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:29:39.150694Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:29:39.150696Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:29:39.150698Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:29:39.494098Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:29:39.494141Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:29:39.494728Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:29:39.843483Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:29:39.844004Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:29:40.179747Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.336s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.335696474 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:29:40.179820Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=3, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:29:40.180225Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:29:40.180240Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:29:40.180264Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:29:41.735369Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=3 +2026-09-21T08:29:41.735398Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T08:29:41.735789Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T08:29:41.736727Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:29:41.757430Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.89682 first_bonus_ms=20.678595 first_hidden_ms=0.007345 bind_reset_ms=0.027488 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T08:29:41.757593Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1576 +2026-09-21T08:29:41.757600Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1577.122734 burst_active_ms=1577.122734 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:29:41.757674Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:29:41.757772Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:29:41.758328Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:29:41.759131Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:29:41.759138Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:29:41.759141Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:29:41.759143Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:29:41.759145Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:29:41.759146Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:29:41.759148Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:29:41.759149Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:29:41.759151Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:29:41.759152Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:29:41.759153Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:29:41.759155Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:29:41.759156Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:29:47.216299Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=91 proposed_tokens=182 accepted_tokens=109 acceptance_rate=0.5989010989010989 emitted_per_verify=2.1868131868131866 zero_accept_rounds=27 partial_accept_rounds=19 full_accept_rounds=45 prefill_verify_ms=4.918340000000001 first_bonus_ms=579.662405 first_hidden_ms=0.0172 bind_reset_ms=0.050048 draft_ms=2286.186407000001 verify_ms=199.053681 target_argmax_sync_ms=2301.367475 logprobs_ms=0.011712000000000007 walk_ms=0.07192099999999998 hidden_concat_ms=0.14833600000000002 rollback_ms=28.375863000000003 decode_ms=4828.916398 +2026-09-21T08:29:47.216876Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5414 +2026-09-21T08:29:47.216920Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5414.3001699999995 burst_active_ms=5414.3001699999995 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:29:50.082737Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=91 proposed_tokens=182 accepted_tokens=109 acceptance_rate=0.5989010989010989 emitted_per_verify=2.1868131868131866 zero_accept_rounds=27 partial_accept_rounds=19 full_accept_rounds=45 prefill_verify_ms=2.7434030000000003 first_bonus_ms=119.219747 first_hidden_ms=0.014352 bind_reset_ms=0.077968 draft_ms=286.79437299999995 verify_ms=204.10872799999996 target_argmax_sync_ms=2209.832928 logprobs_ms=0.009024000000000006 walk_ms=0.05416000000000001 hidden_concat_ms=0.1625119999999999 rollback_ms=27.464962000000014 decode_ms=2741.329431 +2026-09-21T08:29:50.083067Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2863 +2026-09-21T08:29:50.083113Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=2863.873815 burst_active_ms=2863.873815 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:29:52.947560Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=91 proposed_tokens=182 accepted_tokens=109 acceptance_rate=0.5989010989010989 emitted_per_verify=2.1868131868131866 zero_accept_rounds=27 partial_accept_rounds=19 full_accept_rounds=45 prefill_verify_ms=2.7185230000000002 first_bonus_ms=119.417956 first_hidden_ms=0.009056 bind_reset_ms=0.05264 draft_ms=298.5405699999999 verify_ms=202.67004100000003 target_argmax_sync_ms=2201.2436339999995 logprobs_ms=0.011056000000000003 walk_ms=0.05516800000000001 hidden_concat_ms=0.16404799999999994 rollback_ms=25.902357999999996 decode_ms=2739.88934 +2026-09-21T08:29:52.947851Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2862 +2026-09-21T08:29:52.947893Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-3 burst_wall_ms=2862.455276 burst_active_ms=2862.455276 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:29:55.838431Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=91 proposed_tokens=182 accepted_tokens=109 acceptance_rate=0.5989010989010989 emitted_per_verify=2.1868131868131866 zero_accept_rounds=27 partial_accept_rounds=19 full_accept_rounds=45 prefill_verify_ms=2.790348 first_bonus_ms=120.263655 first_hidden_ms=0.009136 bind_reset_ms=0.051088999999999996 draft_ms=295.8063200000001 verify_ms=206.54819699999993 target_argmax_sync_ms=2221.7484139999997 logprobs_ms=0.014032000000000003 walk_ms=0.05932800000000003 hidden_concat_ms=0.17508900000000005 rollback_ms=29.162923 decode_ms=2764.8306350000003 +2026-09-21T08:29:55.838732Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2888 +2026-09-21T08:29:55.838772Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-4 burst_wall_ms=2888.3210090000002 burst_active_ms=2888.3210090000002 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-novec.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-novec.log new file mode 100644 index 000000000..9bdf7ce5c --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-novec.log @@ -0,0 +1,51 @@ +2026-09-21T08:27:09.253594Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:27:09.253637Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:27:09.253774Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:27:09.253828Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:27:09.253831Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:27:09.253833Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:27:09.598990Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:27:09.599031Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:27:09.599602Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:27:09.951170Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:27:09.951659Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:27:10.286791Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.335s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.335104935 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:27:10.286865Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=3, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:27:10.287272Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:27:10.287287Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:27:10.287309Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:27:12.070299Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=3 +2026-09-21T08:27:12.070328Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T08:27:12.070719Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T08:27:12.071647Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:27:12.092390Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.8899389999999999 first_bonus_ms=20.670206999999998 first_hidden_ms=0.008943999999999999 bind_reset_ms=0.070096 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T08:27:12.092824Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1804 +2026-09-21T08:27:12.092831Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1805.319634 burst_active_ms=1805.319634 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:27:12.092876Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:27:12.092979Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:27:12.093680Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:27:12.094473Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:27:12.094480Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:27:12.094483Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:27:12.094485Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:27:12.094486Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:27:12.094488Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:27:12.094489Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:27:12.094490Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:27:12.094491Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:27:12.094493Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:27:12.094494Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:27:12.094495Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:27:12.094496Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:27:18.515758Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=97 proposed_tokens=194 accepted_tokens=102 acceptance_rate=0.5257731958762887 emitted_per_verify=2.051546391752577 zero_accept_rounds=36 partial_accept_rounds=20 full_accept_rounds=41 prefill_verify_ms=4.740577999999999 first_bonus_ms=404.693395 first_hidden_ms=0.018288 bind_reset_ms=0.057504 draft_ms=2420.6063089999993 verify_ms=214.890646 target_argmax_sync_ms=2509.9973749999986 logprobs_ms=0.011392000000000008 walk_ms=0.06558499999999999 hidden_concat_ms=0.14469 rollback_ms=35.016229 decode_ms=5191.027582000001 +2026-09-21T08:27:18.516189Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5601 +2026-09-21T08:27:18.516229Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5601.178693 burst_active_ms=5601.178693 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:27:21.626520Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=97 proposed_tokens=194 accepted_tokens=102 acceptance_rate=0.5257731958762887 emitted_per_verify=2.051546391752577 zero_accept_rounds=36 partial_accept_rounds=20 full_accept_rounds=41 prefill_verify_ms=2.8032589999999997 first_bonus_ms=115.308327 first_hidden_ms=0.007968000000000001 bind_reset_ms=0.04232 draft_ms=331.20475099999993 verify_ms=210.40432199999998 target_argmax_sync_ms=2403.131017 logprobs_ms=0.009408000000000005 walk_ms=0.05587200000000002 hidden_concat_ms=0.15855999999999992 rollback_ms=34.157559 decode_ms=2989.137984 +2026-09-21T08:27:21.626821Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3107 +2026-09-21T08:27:21.626863Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=3107.761555 burst_active_ms=3107.761555 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:27:24.713663Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=97 proposed_tokens=194 accepted_tokens=102 acceptance_rate=0.5257731958762887 emitted_per_verify=2.051546391752577 zero_accept_rounds=36 partial_accept_rounds=20 full_accept_rounds=41 prefill_verify_ms=2.779451 first_bonus_ms=115.02141400000001 first_hidden_ms=0.008048 bind_reset_ms=0.042016 draft_ms=325.98146699999995 verify_ms=210.735994 target_argmax_sync_ms=2385.599028 logprobs_ms=0.010591999999999997 walk_ms=0.051311999999999997 hidden_concat_ms=0.14853000000000005 rollback_ms=33.461309 decode_ms=2966.012552 +2026-09-21T08:27:24.713978Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3084 +2026-09-21T08:27:24.714030Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-3 burst_wall_ms=3084.260171 burst_active_ms=3084.260171 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:27:27.834266Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=97 proposed_tokens=194 accepted_tokens=102 acceptance_rate=0.5257731958762887 emitted_per_verify=2.051546391752577 zero_accept_rounds=36 partial_accept_rounds=20 full_accept_rounds=41 prefill_verify_ms=2.797403 first_bonus_ms=117.064815 first_hidden_ms=0.009680000000000001 bind_reset_ms=0.052128 draft_ms=325.772015 verify_ms=236.091736 target_argmax_sync_ms=2387.700747999999 logprobs_ms=0.011216000000000005 walk_ms=0.08652799999999998 hidden_concat_ms=0.16950400000000002 rollback_ms=37.57497000000001 decode_ms=2997.868978 +2026-09-21T08:27:27.834562Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3118 +2026-09-21T08:27:27.834617Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-4 burst_wall_ms=3118.16811 burst_active_ms=3118.16811 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-fused.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-fused.log new file mode 100644 index 000000000..8b8d06af4 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-fused.log @@ -0,0 +1,51 @@ +2026-09-21T08:30:52.106894Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:30:52.106949Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:30:52.107090Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:30:52.107177Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:30:52.107180Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:30:52.107182Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:30:52.431290Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:30:52.431330Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:30:52.431891Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:30:52.762424Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:30:52.762431Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:30:53.124896Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.362s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.362432521 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:30:53.124979Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=4, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:30:53.125404Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:30:53.125418Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:30:53.125443Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:30:54.794873Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=4 +2026-09-21T08:30:54.794904Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T08:30:54.795320Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T08:30:54.796329Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:30:54.819497Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.963684 first_bonus_ms=23.13032 first_hidden_ms=0.009584 bind_reset_ms=0.037135999999999995 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T08:30:54.819668Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1693 +2026-09-21T08:30:54.819675Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1694.0007910000002 burst_active_ms=1694.0007910000002 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:30:54.819676Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:30:54.819747Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:30:54.820325Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:30:54.821054Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:30:54.821060Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:30:54.821064Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:30:54.821066Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:30:54.821068Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:30:54.821069Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:30:54.821070Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:30:54.821072Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:30:54.821073Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:30:54.821074Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:30:54.821075Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:30:54.821077Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:30:54.821078Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:31:01.160587Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=5.007621 first_bonus_ms=594.090072 first_hidden_ms=0.008576 bind_reset_ms=0.028224 draft_ms=2137.513627 verify_ms=180.16346500000003 target_argmax_sync_ms=2426.875593 logprobs_ms=0.01651200000000002 walk_ms=0.057009 hidden_concat_ms=0.131392 rollback_ms=43.914328000000005 decode_ms=4796.881909 +2026-09-21T08:31:01.161162Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5396 +2026-09-21T08:31:01.161205Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5396.781237 burst_active_ms=5396.781237 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:31:04.142391Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=2.783356 first_bonus_ms=119.503071 first_hidden_ms=0.016176 bind_reset_ms=0.109664 draft_ms=259.72435799999994 verify_ms=184.75052900000006 target_argmax_sync_ms=2360.257531999999 logprobs_ms=0.015056000000000002 walk_ms=0.053296 hidden_concat_ms=0.13607999999999998 rollback_ms=43.843393999999996 decode_ms=2854.957313 +2026-09-21T08:31:04.142727Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2977 +2026-09-21T08:31:04.142782Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=2977.878542 burst_active_ms=2977.878542 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:31:07.101095Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=2.971469 first_bonus_ms=116.253585 first_hidden_ms=0.0077919999999999994 bind_reset_ms=0.048657 draft_ms=259.09896099999986 verify_ms=183.55604099999996 target_argmax_sync_ms=2344.5611389999995 logprobs_ms=0.014799999999999995 walk_ms=0.04793700000000001 hidden_concat_ms=0.13902400000000006 rollback_ms=41.677855000000015 decode_ms=2835.076424 +2026-09-21T08:31:07.101413Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2954 +2026-09-21T08:31:07.101465Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-3 burst_wall_ms=2954.7554800000003 burst_active_ms=2954.7554800000003 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:31:10.043540Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=79 proposed_tokens=237 accepted_tokens=121 acceptance_rate=0.510548523206751 emitted_per_verify=2.518987341772152 zero_accept_rounds=22 partial_accept_rounds=34 full_accept_rounds=23 prefill_verify_ms=2.90702 first_bonus_ms=118.738365 first_hidden_ms=0.009072 bind_reset_ms=0.050672 draft_ms=265.224331 verify_ms=181.07857299999992 target_argmax_sync_ms=2322.2688350000008 logprobs_ms=0.03046500000000001 walk_ms=0.056961000000000026 hidden_concat_ms=0.13720099999999993 rollback_ms=41.84306799999999 decode_ms=2816.590278 +2026-09-21T08:31:10.043865Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=2938 +2026-09-21T08:31:10.043916Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-4 burst_wall_ms=2938.720512 burst_active_ms=2938.720512 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-novec.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-novec.log new file mode 100644 index 000000000..6f16cbf72 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-novec.log @@ -0,0 +1,51 @@ +2026-09-21T08:28:24.136576Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:28:24.136623Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:28:24.136802Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:28:24.136879Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:28:24.136882Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:28:24.136884Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:28:24.477258Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:28:24.477364Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:28:24.478019Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:28:24.823428Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:28:24.824134Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:28:25.170692Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.347s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.346519151 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:28:25.170772Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=4, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:28:25.171172Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:28:25.171188Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:28:25.171212Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:28:26.965492Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=4 +2026-09-21T08:28:26.965522Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T08:28:26.965922Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T08:28:26.966864Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:28:26.989495Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.900691 first_bonus_ms=22.603641 first_hidden_ms=0.006848 bind_reset_ms=0.027216 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T08:28:26.989656Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1817 +2026-09-21T08:28:26.989664Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1818.2360150000002 burst_active_ms=1818.2360150000002 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:28:26.989735Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:28:26.990097Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:28:26.990768Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:28:26.991602Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18939 +2026-09-21T08:28:26.991613Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:28:26.991617Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:28:26.991618Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:28:26.991619Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:28:26.991621Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:28:26.991622Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:28:26.991623Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:28:26.991625Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:28:26.991626Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:28:26.991627Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:28:26.991628Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:28:26.991630Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:28:33.254005Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=84 proposed_tokens=250 accepted_tokens=116 acceptance_rate=0.464 emitted_per_verify=2.369047619047619 zero_accept_rounds=28 partial_accept_rounds=32 full_accept_rounds=24 prefill_verify_ms=4.904387 first_bonus_ms=403.331665 first_hidden_ms=0.011056 bind_reset_ms=0.037391999999999995 draft_ms=2162.7068690000006 verify_ms=206.86151999999996 target_argmax_sync_ms=2614.4086299999994 logprobs_ms=0.008448000000000004 walk_ms=0.067328 hidden_concat_ms=0.132128 rollback_ms=43.62668399999999 decode_ms=5037.694670999999 +2026-09-21T08:28:33.254335Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5446 +2026-09-21T08:28:33.254375Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5446.500853 burst_active_ms=5446.500853 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:28:36.438288Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=84 proposed_tokens=250 accepted_tokens=116 acceptance_rate=0.464 emitted_per_verify=2.369047619047619 zero_accept_rounds=28 partial_accept_rounds=32 full_accept_rounds=24 prefill_verify_ms=2.784267 first_bonus_ms=115.516567 first_hidden_ms=0.014832 bind_reset_ms=0.072128 draft_ms=270.062435 verify_ms=198.5988000000001 target_argmax_sync_ms=2542.9791030000006 logprobs_ms=0.00752 walk_ms=0.058688000000000025 hidden_concat_ms=0.14019300000000004 rollback_ms=40.67917099999999 decode_ms=3060.872455 +2026-09-21T08:28:36.438622Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3179 +2026-09-21T08:28:36.438668Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-2 burst_wall_ms=3179.745531 burst_active_ms=3179.745531 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:28:39.609701Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=84 proposed_tokens=250 accepted_tokens=116 acceptance_rate=0.464 emitted_per_verify=2.369047619047619 zero_accept_rounds=28 partial_accept_rounds=32 full_accept_rounds=24 prefill_verify_ms=2.8081709999999998 first_bonus_ms=117.885425 first_hidden_ms=0.015632 bind_reset_ms=0.11776 draft_ms=269.0966439999999 verify_ms=197.23528800000003 target_argmax_sync_ms=2532.2516439999986 logprobs_ms=0.006209000000000003 walk_ms=0.057056 hidden_concat_ms=0.1364799999999999 rollback_ms=40.12204599999999 decode_ms=3045.995693 +2026-09-21T08:28:39.610042Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3167 +2026-09-21T08:28:39.610090Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-3 burst_wall_ms=3167.245818 burst_active_ms=3167.245818 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:28:42.800674Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=4 rounds=84 proposed_tokens=250 accepted_tokens=116 acceptance_rate=0.464 emitted_per_verify=2.369047619047619 zero_accept_rounds=28 partial_accept_rounds=32 full_accept_rounds=24 prefill_verify_ms=2.7339949999999997 first_bonus_ms=118.044274 first_hidden_ms=0.014192000000000001 bind_reset_ms=0.079744 draft_ms=271.23432099999997 verify_ms=195.80094899999997 target_argmax_sync_ms=2551.5339200000017 logprobs_ms=0.005792000000000002 walk_ms=0.05459199999999998 hidden_concat_ms=0.133744 rollback_ms=39.57079100000001 decode_ms=3065.9240259999997 +2026-09-21T08:28:42.800963Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=3187 +2026-09-21T08:28:42.801014Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-4 burst_wall_ms=3187.151497 burst_active_ms=3187.151497 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/options.jsonl b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/options.jsonl new file mode 100644 index 000000000..35701c24f --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/options.jsonl @@ -0,0 +1,7 @@ +{"arm": "classic-open", "width": null, "arm_env": {}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-open.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.14501953125, "runs": [{"wall_s": 3.526, "e2e_tok_s": 56.721, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}, {"wall_s": 3.536, "e2e_tok_s": 56.555, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}, {"wall_s": 3.494, "e2e_tok_s": 57.238, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [], "nvrm_delta": 0} +{"arm": "classic-novec", "width": null, "arm_env": {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-novec.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.19677734375, "runs": [{"wall_s": 3.452, "e2e_tok_s": 57.944, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.486, "e2e_tok_s": 57.37, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.484, "e2e_tok_s": 57.41, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [], "nvrm_delta": 0} +{"arm": "w3-novec", "width": 3, "arm_env": {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1", "--model-draft", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", "--draft-kind", "dflash", "--draft-block-size", "3"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-novec.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.1279296875, "runs": [{"wall_s": 3.111, "e2e_tok_s": 64.298, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.087, "e2e_tok_s": 64.794, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.121, "e2e_tok_s": 64.091, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [{"block_size": 3.0, "rounds": 0.0, "proposed_tokens": 0.0, "accepted_tokens": 0.0, "acceptance_rate": 0.0, "emitted_per_verify": 0.0, "draft_ms": 0.0, "verify_ms": 0.0, "target_argmax_sync_ms": 0.0, "decode_ms": 0.0}, {"block_size": 3.0, "rounds": 97.0, "proposed_tokens": 194.0, "accepted_tokens": 102.0, "acceptance_rate": 0.5257731958762887, "emitted_per_verify": 2.051546391752577, "draft_ms": 2420.6063089999993, "verify_ms": 214.890646, "target_argmax_sync_ms": 2509.9973749999986, "decode_ms": 5191.027582000001}, {"block_size": 3.0, "rounds": 97.0, "proposed_tokens": 194.0, "accepted_tokens": 102.0, "acceptance_rate": 0.5257731958762887, "emitted_per_verify": 2.051546391752577, "draft_ms": 331.20475099999993, "verify_ms": 210.40432199999998, "target_argmax_sync_ms": 2403.131017, "decode_ms": 2989.137984}, {"block_size": 3.0, "rounds": 97.0, "proposed_tokens": 194.0, "accepted_tokens": 102.0, "acceptance_rate": 0.5257731958762887, "emitted_per_verify": 2.051546391752577, "draft_ms": 325.98146699999995, "verify_ms": 210.735994, "target_argmax_sync_ms": 2385.599028, "decode_ms": 2966.012552}, {"block_size": 3.0, "rounds": 97.0, "proposed_tokens": 194.0, "accepted_tokens": 102.0, "acceptance_rate": 0.5257731958762887, "emitted_per_verify": 2.051546391752577, "draft_ms": 325.772015, "verify_ms": 236.091736, "target_argmax_sync_ms": 2387.700747999999, "decode_ms": 2997.868978}], "nvrm_delta": 0} +{"arm": "w4-novec", "width": 4, "arm_env": {"MLXCEL_SDPA_VECTOR_LARGE_D": "0"}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1", "--model-draft", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", "--draft-kind", "dflash", "--draft-block-size", "4"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-novec.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.11865234375, "runs": [{"wall_s": 3.184, "e2e_tok_s": 62.812, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.171, "e2e_tok_s": 63.075, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}, {"wall_s": 3.191, "e2e_tok_s": 62.681, "completion_tokens": 200, "chars": 837, "sha": "4c37650547", "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance."}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [{"block_size": 4.0, "rounds": 0.0, "proposed_tokens": 0.0, "accepted_tokens": 0.0, "acceptance_rate": 0.0, "emitted_per_verify": 0.0, "draft_ms": 0.0, "verify_ms": 0.0, "target_argmax_sync_ms": 0.0, "decode_ms": 0.0}, {"block_size": 4.0, "rounds": 84.0, "proposed_tokens": 250.0, "accepted_tokens": 116.0, "acceptance_rate": 0.464, "emitted_per_verify": 2.369047619047619, "draft_ms": 2162.7068690000006, "verify_ms": 206.86151999999996, "target_argmax_sync_ms": 2614.4086299999994, "decode_ms": 5037.694670999999}, {"block_size": 4.0, "rounds": 84.0, "proposed_tokens": 250.0, "accepted_tokens": 116.0, "acceptance_rate": 0.464, "emitted_per_verify": 2.369047619047619, "draft_ms": 270.062435, "verify_ms": 198.5988000000001, "target_argmax_sync_ms": 2542.9791030000006, "decode_ms": 3060.872455}, {"block_size": 4.0, "rounds": 84.0, "proposed_tokens": 250.0, "accepted_tokens": 116.0, "acceptance_rate": 0.464, "emitted_per_verify": 2.369047619047619, "draft_ms": 269.0966439999999, "verify_ms": 197.23528800000003, "target_argmax_sync_ms": 2532.2516439999986, "decode_ms": 3045.995693}, {"block_size": 4.0, "rounds": 84.0, "proposed_tokens": 250.0, "accepted_tokens": 116.0, "acceptance_rate": 0.464, "emitted_per_verify": 2.369047619047619, "draft_ms": 271.23432099999997, "verify_ms": 195.80094899999997, "target_argmax_sync_ms": 2551.5339200000017, "decode_ms": 3065.9240259999997}], "nvrm_delta": 0} +{"arm": "w3-fused", "width": 3, "arm_env": {}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1", "--model-draft", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", "--draft-kind", "dflash", "--draft-block-size", "3"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w3-fused.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.11279296875, "runs": [{"wall_s": 2.866, "e2e_tok_s": 69.781, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}, {"wall_s": 2.865, "e2e_tok_s": 69.814, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}, {"wall_s": 2.891, "e2e_tok_s": 69.185, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [{"block_size": 3.0, "rounds": 0.0, "proposed_tokens": 0.0, "accepted_tokens": 0.0, "acceptance_rate": 0.0, "emitted_per_verify": 0.0, "draft_ms": 0.0, "verify_ms": 0.0, "target_argmax_sync_ms": 0.0, "decode_ms": 0.0}, {"block_size": 3.0, "rounds": 91.0, "proposed_tokens": 182.0, "accepted_tokens": 109.0, "acceptance_rate": 0.5989010989010989, "emitted_per_verify": 2.1868131868131866, "draft_ms": 2286.186407000001, "verify_ms": 199.053681, "target_argmax_sync_ms": 2301.367475, "decode_ms": 4828.916398}, {"block_size": 3.0, "rounds": 91.0, "proposed_tokens": 182.0, "accepted_tokens": 109.0, "acceptance_rate": 0.5989010989010989, "emitted_per_verify": 2.1868131868131866, "draft_ms": 286.79437299999995, "verify_ms": 204.10872799999996, "target_argmax_sync_ms": 2209.832928, "decode_ms": 2741.329431}, {"block_size": 3.0, "rounds": 91.0, "proposed_tokens": 182.0, "accepted_tokens": 109.0, "acceptance_rate": 0.5989010989010989, "emitted_per_verify": 2.1868131868131866, "draft_ms": 298.5405699999999, "verify_ms": 202.67004100000003, "target_argmax_sync_ms": 2201.2436339999995, "decode_ms": 2739.88934}, {"block_size": 3.0, "rounds": 91.0, "proposed_tokens": 182.0, "accepted_tokens": 109.0, "acceptance_rate": 0.5989010989010989, "emitted_per_verify": 2.1868131868131866, "draft_ms": 295.8063200000001, "verify_ms": 206.54819699999993, "target_argmax_sync_ms": 2221.7484139999997, "decode_ms": 2764.8306350000003}], "nvrm_delta": 0} +{"arm": "w4-fused", "width": 4, "arm_env": {}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1", "--model-draft", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", "--draft-kind", "dflash", "--draft-block-size", "4"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.w4-fused.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.1787109375, "runs": [{"wall_s": 2.982, "e2e_tok_s": 67.072, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}, {"wall_s": 2.958, "e2e_tok_s": 67.605, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}, {"wall_s": 2.942, "e2e_tok_s": 67.973, "completion_tokens": 200, "chars": 820, "sha": "3e60b1574c", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n logger: logging.Logger = None,\n **kwargs,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n Args:\n func: The function to retry.\n backoff_policy: The backoff policy to use.\n logger: The logger to use for logging.\n **kwargs: Additional arguments to pass to the function.\n\n Returns:\n The result of the function.\n \"\"\"\n if"}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [{"block_size": 4.0, "rounds": 0.0, "proposed_tokens": 0.0, "accepted_tokens": 0.0, "acceptance_rate": 0.0, "emitted_per_verify": 0.0, "draft_ms": 0.0, "verify_ms": 0.0, "target_argmax_sync_ms": 0.0, "decode_ms": 0.0}, {"block_size": 4.0, "rounds": 79.0, "proposed_tokens": 237.0, "accepted_tokens": 121.0, "acceptance_rate": 0.510548523206751, "emitted_per_verify": 2.518987341772152, "draft_ms": 2137.513627, "verify_ms": 180.16346500000003, "target_argmax_sync_ms": 2426.875593, "decode_ms": 4796.881909}, {"block_size": 4.0, "rounds": 79.0, "proposed_tokens": 237.0, "accepted_tokens": 121.0, "acceptance_rate": 0.510548523206751, "emitted_per_verify": 2.518987341772152, "draft_ms": 259.72435799999994, "verify_ms": 184.75052900000006, "target_argmax_sync_ms": 2360.257531999999, "decode_ms": 2854.957313}, {"block_size": 4.0, "rounds": 79.0, "proposed_tokens": 237.0, "accepted_tokens": 121.0, "acceptance_rate": 0.510548523206751, "emitted_per_verify": 2.518987341772152, "draft_ms": 259.09896099999986, "verify_ms": 183.55604099999996, "target_argmax_sync_ms": 2344.5611389999995, "decode_ms": 2835.076424}, {"block_size": 4.0, "rounds": 79.0, "proposed_tokens": 237.0, "accepted_tokens": 121.0, "acceptance_rate": 0.510548523206751, "emitted_per_verify": 2.518987341772152, "draft_ms": 265.224331, "verify_ms": 181.07857299999992, "target_argmax_sync_ms": 2322.2688350000008, "decode_ms": 2816.590278}], "nvrm_delta": 0} +{"arm": "classic-close", "width": null, "arm_env": {}, "cmd": ["/tmp/mlxcel1935opt-bin", "-m", "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", "--port", "18939", "--ignore-eos", "--max-batch-size", "1"], "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/logs/server.classic-close.log", "gate_wait_s": 0, "driver_wait_s": 0, "mem_wait_s": 0, "nvrm_total_before": 0, "ci_job_running": false, "load1_before": 0.14013671875, "runs": [{"wall_s": 3.497, "e2e_tok_s": 57.19, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}, {"wall_s": 3.458, "e2e_tok_s": 57.842, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}, {"wall_s": 3.472, "e2e_tok_s": 57.605, "completion_tokens": 200, "chars": 823, "sha": "2c76b0a181", "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering"}], "prompt_tokens": 158, "declined_to_classic": false, "diagnostics": [], "nvrm_delta": 0} diff --git a/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md new file mode 100644 index 000000000..73b5de0dd --- /dev/null +++ b/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md @@ -0,0 +1,59 @@ +# What byte-identical Qwen 3.5 DFlash costs, priced (GB10, 2026-09-21) + +Issue #1935, deciding what PR #1944 should be. The residual there is genuine numerics in CUDA's fused `sdpa_vector` path, and #1944 answers it by declining the burst. This record answers the question that has to come first, which side of the comparison that kernel actually serves, and then prices every option the dispatch allows against one set of classic brackets. + +## Which side does the fused kernel serve + +Both. It is not an asymmetry between a vector path and a block path, and that matters, because a proposal to disable the fused path "only in the verify block" rests on there being one. + +`supports_sdpa_vector` gates on the query row count **of the individual call**, not of the block: `query_sequence_length = q.shape(2)` and `supported_vector_config = sdpa_supported_head_dim && query_sequence_length < 4` (`src/lib/mlx-cpp/patches/mlx/backend/cuda/scaled_dot_product_attention.cu`). Classic decode reaches it with `l == 1`, through `Qwen3NextAttention::forward`'s third arm. The verify block reaches it through `attend_per_position`, which slices `queries[:, :, i:i+1, :]` per row precisely so each row computes what a decode step computes, so every one of its calls also has `q.shape(2) == 1`. Neither is chunked first: `materializing_sdpa_query_chunk` returns `None` below `q_len == 2`, and mlxcel's own note on `cuda_sdpa_materializes_scores` records that `supports_sdpa_vector` "only covers `q_len < 4`". + +Measured rather than inferred, from the served arms below: `MLXCEL_SDPA_VECTOR_LARGE_D=0` moves **both** sides. Classic's completion changes (`2c76b0a181` to `4c37650547`), which a flag that did not serve classic's decode call could not do. The width-3 and width-4 bursts change too (`3e60b1574c` to `4c37650547`), and the burst's only `q_len < 4` attention calls are those per-row verify ones: its prefill is one batched causal call at `q_len` 158, its first bonus is sampled from prefill logits, and its rollback does no attention. The drafter's own attention moves as well, but a drafter cannot change greedy text, because every emitted token is a target argmax. + +So `LARGE_D=0` does not move classic onto the block's kernel. It moves both off the fused kernel and onto the materializing fallback, and byte-identity is restored because the fallback is consistent across the two call sites while the fused kernel is not. + +**The consequence for "disable it only in verify": it is not an option.** It would put the verify rows on the fallback and classic decode on the fused kernel, which is the one configuration guaranteed not to be byte-identical, since byte-identity here is exactly the two sides agreeing on a kernel. It was not measured because the dispatch predicate answers it and building the knob would need an MLX rebuild to test something already decided. + +What does survive, and is not the same idea, is that the flag is a process-global `static` read once. A server process can run entirely on the fallback. That confines the cost to processes that opted into speculation instead of forfeiting the kernel for everyone, and it is priced below as option B. + +## Host and method + +One session, one binary (`origin/main`, sha `f2ba1688`, verified to carry none of #1944's gate before it ran), seven arms, n = 3 per arm after a discarded warm-up. One `mlxcel-server` per arm, `--ignore-eos --max-batch-size 1`, the #1797 harness's fixed 158-token prompt, 200 tokens per request, `MLX_ENABLE_TF32=1`. Arms differ in exactly the variable in their tag. + +Driver: `data/dflash-sdpav-options-gb10-2026-09-21/harness/price_options.py`, which reuses the #1797 harness's request path and the #1820 host gate (sustained-quiet CPU predicate matching on `/proc//comm` with stopped processes dropped, foreign-model check, memory floor, cumulative `NV_ERR_NO_MEMORY` trip wire). The driver `NV_ERR_NO_MEMORY` delta was 0 on every arm. + +The brackets are the shipped configuration: no drafter, the fused path left on. + +## Result + +Brackets: opening 56.55 to 57.24, closing 57.19 to 57.84 tok/s. **They overlap**, so this session did not drift. The union spread, 1.29 tok/s or 2.3%, is the resolution floor, and any difference smaller than it is reported below as unresolved rather than as a result. + +| arm | mean tok/s | min to max | vs classic | completion sha256 | acceptance | +|---|---:|---:|---:|---|---:| +| classic-open (fused) | 56.84 | 56.55 to 57.24 | 0.99x | `2c76b0a181` | | +| classic-novec (`LARGE_D=0`) | 57.57 | 57.37 to 57.94 | 1.01x | `4c37650547` | | +| w3, `LARGE_D=0` | 64.39 | 64.09 to 64.79 | 1.13x | `4c37650547` | 0.526 | +| w4, `LARGE_D=0` | 62.86 | 62.68 to 63.08 | 1.10x | `4c37650547` | 0.464 | +| w3, fused | 69.59 | 69.19 to 69.81 | 1.22x | `3e60b1574c` | 0.599 | +| w4, fused | 67.55 | 67.07 to 67.97 | 1.18x | `3e60b1574c` | 0.511 | +| classic-close (fused) | 57.55 | 57.19 to 57.84 | 1.01x | `2c76b0a181` | | + +Resolved against the 1.29 floor: the burst beats classic in every configuration (w3 on the fallback clears the closing bracket by 6.25 tok/s), and the fused kernel is worth 4.39 tok/s to the width-3 burst. Unresolved: width 3 against width 4 in either configuration (gaps 1.21 and 1.02), and the fused kernel's worth to classic decode (0.13). The #1945 session resolved width 3 above width 4 against a tighter 0.40 floor; this session does not contradict it, it simply cannot see it. + +Two facts the table carries that are easy to miss. **The fallback costs the burst twice**: slower per-row attention and lower acceptance (0.599 to 0.526 at width 3, 0.511 to 0.464 at width 4), because different numerics change how often the drafter's proposal survives the target's argmax. And **`LARGE_D=0` changes what classic decode produces** (`4c37650547`, not `2c76b0a181`), so the fallback is not a free correctness switch: it changes every request's output on that process, speculative or not. + +## The options, priced + +**A. Decline the burst, which is PR #1944 as written.** 1.00x, output identical to today's shipped classic (`2c76b0a181`), and nothing else in the process changes. Blast radius is the narrowest available: the guard lives in `Qwen35Model::probe_block_chain_exactness`, so it touches the Qwen 3.5 family's DFlash gating and no other family's probe, and no non-speculative request anywhere. The cost is the burst's 1.18x to 1.22x, forfeited unless an operator acts. + +**B. Run the process on the fallback (`MLXCEL_SDPA_VECTOR_LARGE_D=0`).** 1.13x at width 3, 1.10x at width 4, and byte-identical to the drafter-less server **in the same configuration**. It satisfies the contract that speculative text equals what the same server would produce without the drafter. What it does not do is preserve today's answers: the process serves `4c37650547` where it serves `2c76b0a181` today, for every request. The flag's cost to classic decode is unresolved at this prompt length, and that is not a general result: issue #675 records that the fallback's cost grows with key length while the fused kernel's does not, so a long-context server would pay more than this 158-token prompt shows. Blast radius: the flag is process-global and covers `head_dim` 256 and 288, so it reaches the gemma family, qwen3.5, qwen3.6, baichuan-m1 and paligemma2; scoping it to speculation-configured processes limits who pays but still changes every model in such a process. + +**C. Disable the fused path only for the verify block.** Not an option, for the dispatch reason above. + +**D. `MLXCEL_MTP_ALLOW_INEXACT=1`.** 1.22x at width 3, 1.18x at width 4, and not byte-identical (`3e60b1574c`). Per-process opt-in; nothing changes for anyone who does not set it. This is the status quo escape and it is what an operator who wants throughput and does not need byte-identity should use. + +## What the numbers support + +Byte-identical speculation is achievable and is worth 1.13x; dropping the byte-identity requirement is worth 1.22x; and the fused kernel is worth about 8% to the burst while being worth nothing measurable to classic decode at this context length. + +A is the better default anyway, because it is the only option that leaves today's output and today's non-speculative performance exactly as they are, and because B's price is paid by every request on the process rather than by the ones that asked for speculation. B is worth documenting as the operator recipe for byte-identical speculation rather than shipping as a default, and it needs no new code: #1944's guard stands down when the flag is off, so an operator who sets `MLXCEL_SDPA_VECTOR_LARGE_D=0` gets the burst engaged and byte-identical on the same binary. From 1595b2046ee226c34f09d4f109a5854583a82235 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 21 Sep 2026 17:42:56 +0900 Subject: [PATCH 16/16] docs(benchmarks): verify both options are reachable from the PR #1944 binary The record's closing claim, that an operator can have byte-identical speculation today by selecting the fallback because the guard stands down when it is off, was an argument about code I wrote rather than a measurement. Three arms on PR #1944's own binary, verified to carry the guard before they ran, settle it: at the shipped default the guard declines, no burst runs and the server returns exactly today's classic text, while with the fallback selected the guard stands down, the burst runs, and its text is byte-identical to the drafter-less server in the same configuration. One binary carries both options, so the choice between declining and byte-identical speculation is an environment variable rather than a build. Refs #1935 --- .../gate/arm.gate-classic-novec.json | 435 +++++++++++++++++ .../gate/arm.gate-w3-fused.json | 441 ++++++++++++++++++ .../gate/arm.gate-w3-novec.json | 441 ++++++++++++++++++ .../gate/server.gate-classic-novec.log | 35 ++ .../gate/server.gate-w3-fused.log | 38 ++ .../gate/server.gate-w3-novec.log | 44 ++ ...sh-sdpav-options-qwen35-gb10-2026-09-21.md | 14 +- 7 files changed, 1447 insertions(+), 1 deletion(-) create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-classic-novec.json create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-fused.json create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-novec.json create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log create mode 100644 docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-novec.log diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-classic-novec.json b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-classic-novec.json new file mode 100644 index 000000000..dd065d0ab --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-classic-novec.json @@ -0,0 +1,435 @@ +{ + "tag": "gate-classic-novec", + "width": "classic", + "cmd": [ + "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18941", + "--ignore-eos", + "--max-batch-size", + "1" + ], + "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log", + "extra_env": "MLXCEL_SDPA_VECTOR_LARGE_D=0", + "responses": [ + { + "index": 0, + "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance.", + "tokens": [ + " ", + " \"\"\"", + "Y", + "ield", + " delays", + " between", + " attempts", + ".\"\"\"", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + "\n", + " ", + " for", + " _", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " yield", + " delay", + "\n", + " ", + " if", + " delay", + " >=", + " self", + ".max", + "_delay", + ":", + "\n", + " ", + " break", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing", + " herd", + " problems", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " A", + " Back", + "off", + "Policy", + " instance", + "." + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.5, + -0.125, + -1.25, + -1.375, + -0.625, + -0.625, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + -0.75, + -0.25, + 0.0, + -0.625, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.375, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + -0.75, + -0.75, + -0.5, + -0.5, + -0.125, + -0.125, + 0.0, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + -0.375, + 0.0, + -0.25, + 0.0, + -0.5, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + -0.875, + -0.375, + -1.125, + -1.0, + -0.5, + 0.0, + -1.25, + 0.0, + -0.375, + -1.375, + -0.125, + -0.125, + 0.0, + -1.625, + 0.0, + -0.875, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.75, + -0.5, + -0.125, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.5, + -0.125, + -0.625, + -0.125, + 0.0, + 0.0, + -0.375, + -0.25, + -0.125, + 0.0, + 0.0, + -1.75, + -1.125, + -1.875, + -1.5, + -0.625, + -0.875, + 0.0, + -0.875, + -0.375, + -0.5, + -0.75, + -0.25, + -0.625, + -0.125, + 0.0, + 0.0, + -0.625, + -0.875, + -0.25, + -0.75, + -0.75, + -0.75, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + -0.125, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0, + -0.125, + -0.125, + -0.125, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + -0.125, + 0.0, + 0.0, + -0.25, + -1.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-fused.json b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-fused.json new file mode 100644 index 000000000..b23fea12b --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-fused.json @@ -0,0 +1,441 @@ +{ + "tag": "gate-w3-fused", + "width": "3", + "cmd": [ + "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18941", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "3" + ], + "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log", + "extra_env": "MLXCEL_SDPA_VECTOR_LARGE_D=1", + "responses": [ + { + "index": 0, + "text": " \"\"\"Generate a sequence of delays for retrying.\"\"\"\n delays = []\n for i in range(self.max_attempts):\n delay = self.base_delay * (2 ** i)\n delay = min(delay, self.max_delay)\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n delays.append(delay)\n return delays\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n logger: logging.Logger = None,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering", + "tokens": [ + " ", + " \"\"\"", + "Generate", + " a", + " sequence", + " of", + " delays", + " for", + " retry", + "ing", + ".\"\"\"", + "\n", + " ", + " delays", + " =", + " []", + "\n", + " ", + " for", + " i", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + " *", + " (", + "2", + " **", + " i", + ")", + "\n", + " ", + " delay", + " =", + " min", + "(delay", + ",", + " self", + ".max", + "_delay", + ")", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " delays", + ".append", + "(delay", + ")", + "\n", + " ", + " return", + " delays", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " logger", + ":", + " logging", + ".Logger", + " =", + " None", + ",", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing" + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.375, + -1.0, + -0.875, + 0.0, + -0.5, + -1.875, + -1.375, + -0.875, + -1.25, + 0.0, + 0.0, + -1.625, + 0.0, + -0.25, + -0.125, + 0.0, + -0.5, + -1.125, + -0.125, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + -0.625, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + 0.0, + -0.125, + 0.0, + -0.625, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.625, + -0.625, + -0.25, + -0.5, + -0.375, + -0.125, + -0.375, + 0.0, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + -0.375, + -0.375, + -1.125, + -0.625, + -0.375, + 0.0, + -0.875, + 0.0, + -0.375, + -1.125, + -0.125, + -0.25, + 0.0, + -1.5, + 0.0, + -0.75, + -0.375, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.625, + -0.75, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.25, + -2.0, + -0.5, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + -0.25, + -1.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.375, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + -0.25, + -0.125, + -0.125, + 0.0, + 0.0, + -1.375, + -0.75, + -1.875, + -1.25, + -0.375, + -0.5, + 0.0, + -0.75, + -0.375, + -0.375, + -0.625, + -0.125, + -0.5, + -0.125, + 0.0, + 0.0, + -0.5, + -1.0, + -0.25, + -0.625, + -0.625, + -0.5, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + 0.0, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-novec.json b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-novec.json new file mode 100644 index 000000000..9043d2d91 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-novec.json @@ -0,0 +1,441 @@ +{ + "tag": "gate-w3-novec", + "width": "3", + "cmd": [ + "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/bin/mlxcel1935-server", + "-m", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-4bit", + "--port", + "18941", + "--ignore-eos", + "--max-batch-size", + "1", + "--draft-model", + "/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash", + "--draft-kind", + "dflash", + "--draft-block-size", + "3" + ], + "log": "/home/inureyes/Development/mlxcel-wt-1935/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-novec.log", + "extra_env": "MLXCEL_SDPA_VECTOR_LARGE_D=0", + "responses": [ + { + "index": 0, + "text": " \"\"\"Yield delays between attempts.\"\"\"\n delay = self.base_delay\n for _ in range(self.max_attempts):\n if self.jitter:\n delay = delay + random.uniform(0, delay)\n yield delay\n if delay >= self.max_delay:\n break\n\n\ndef retry_with_backoff(\n func,\n backoff_policy: BackoffPolicy = None,\n *,\n max_attempts: int = None,\n max_delay: float = None,\n base_delay: float = None,\n jitter: bool = None,\n):\n \"\"\"Retry a function with exponential backoff and jitter.\n\n This decorator is useful for retrying HTTP calls that might fail due to\n transient network issues. It uses exponential backoff with jitter to\n avoid thundering herd problems.\n\n Args:\n func: The function to retry.\n backoff_policy: A BackoffPolicy instance.", + "tokens": [ + " ", + " \"\"\"", + "Y", + "ield", + " delays", + " between", + " attempts", + ".\"\"\"", + "\n", + " ", + " delay", + " =", + " self", + ".base", + "_delay", + "\n", + " ", + " for", + " _", + " in", + " range", + "(self", + ".max", + "_attempts", + "):", + "\n", + " ", + " if", + " self", + ".j", + "itter", + ":", + "\n", + " ", + " delay", + " =", + " delay", + " +", + " random", + ".uniform", + "(", + "0", + ",", + " delay", + ")", + "\n", + " ", + " yield", + " delay", + "\n", + " ", + " if", + " delay", + " >=", + " self", + ".max", + "_delay", + ":", + "\n", + " ", + " break", + "\n\n\n", + "def", + " retry", + "_with", + "_back", + "off", + "(", + "\n", + " ", + " func", + ",", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " Back", + "off", + "Policy", + " =", + " None", + ",", + "\n", + " ", + " *,", + "\n", + " ", + " max", + "_attempts", + ":", + " int", + " =", + " None", + ",", + "\n", + " ", + " max", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " base", + "_delay", + ":", + " float", + " =", + " None", + ",", + "\n", + " ", + " jitter", + ":", + " bool", + " =", + " None", + ",", + "\n", + "):", + "\n", + " ", + " \"\"\"", + "Retry", + " a", + " function", + " with", + " exponential", + " back", + "off", + " and", + " jitter", + ".", + "\n\n", + " ", + " This", + " decorator", + " is", + " useful", + " for", + " retry", + "ing", + " HTTP", + " calls", + " that", + " might", + " fail", + " due", + " to", + "\n", + " ", + " transient", + " network", + " issues", + ".", + " It", + " uses", + " exponential", + " back", + "off", + " with", + " jitter", + " to", + "\n", + " ", + " avoid", + " thunder", + "ing", + " herd", + " problems", + ".", + "\n\n", + " ", + " Args", + ":", + "\n", + " ", + " func", + ":", + " The", + " function", + " to", + " retry", + ".", + "\n", + " ", + " back", + "off", + "_policy", + ":", + " A", + " Back", + "off", + "Policy", + " instance", + "." + ], + "token_logprobs": [ + 0.0, + -0.625, + -1.5, + -0.125, + -1.25, + -1.375, + -0.625, + -0.625, + 0.0, + 0.0, + -1.375, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + 0.0, + -0.75, + -0.25, + 0.0, + -0.625, + -0.125, + 0.0, + 0.0, + -0.25, + 0.0, + 0.0, + -0.375, + -0.125, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -1.0, + -0.75, + -0.75, + -0.5, + -0.5, + -0.125, + -0.125, + 0.0, + -0.125, + -0.125, + 0.0, + -0.125, + 0.0, + -0.375, + 0.0, + -0.25, + 0.0, + -0.5, + 0.0, + -0.25, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.25, + -0.875, + -0.375, + -1.125, + -1.0, + -0.5, + 0.0, + -1.25, + 0.0, + -0.375, + -1.375, + -0.125, + -0.125, + 0.0, + -1.625, + 0.0, + -0.875, + -0.5, + 0.0, + 0.0, + 0.0, + -0.375, + -0.25, + 0.0, + 0.0, + -0.125, + -2.125, + 0.0, + 0.0, + -1.75, + -0.5, + -0.125, + 0.0, + -0.125, + -0.5, + 0.0, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.125, + 0.0, + 0.0, + -0.5, + 0.0, + 0.0, + 0.0, + -1.0, + -0.5, + -0.125, + -0.625, + -0.125, + 0.0, + 0.0, + -0.375, + -0.25, + -0.125, + 0.0, + 0.0, + -1.75, + -1.125, + -1.875, + -1.5, + -0.625, + -0.875, + 0.0, + -0.875, + -0.375, + -0.5, + -0.75, + -0.25, + -0.625, + -0.125, + 0.0, + 0.0, + -0.625, + -0.875, + -0.25, + -0.75, + -0.75, + -0.75, + -0.125, + 0.0, + 0.0, + -0.125, + -0.25, + -0.125, + -0.375, + 0.0, + -0.25, + -0.75, + 0.0, + -0.125, + -0.125, + -0.125, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.375, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + 0.0, + -0.75, + -0.125, + 0.0, + 0.0, + -0.25, + -1.0 + ], + "usage": { + "prompt_tokens": 158, + "completion_tokens": 200, + "total_tokens": 358, + "prompt_tokens_details": { + "cached_tokens": 0 + } + } + } + ], + "model_id": "qwen3.5-4b-4bit" +} \ No newline at end of file diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log new file mode 100644 index 000000000..362a5b9c2 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log @@ -0,0 +1,35 @@ +2026-09-21T08:41:33.424813Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:41:33.424960Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:41:33.425373Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:41:33.425543Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:41:33.425545Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:41:33.425547Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:41:33.770270Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:41:33.770411Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:41:33.771449Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:41:34.118910Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:41:34.119634Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:41:34.457333Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.338s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.337673483 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:41:34.457474Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto) +2026-09-21T08:41:34.457902Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:41:34.457916Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:41:34.458065Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:41:34.460455Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:41:35.526896Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 1068ms prompt_tokens=1 cached_tokens=0 generation_time_ms=1068 +2026-09-21T08:41:35.527225Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:41:35.527389Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:41:35.528377Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:41:35.530064Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18941 +2026-09-21T08:41:35.530080Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:41:35.530083Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:41:35.530085Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:41:35.530086Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:41:35.530087Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:41:35.530089Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:41:35.530090Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:41:35.530091Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:41:35.530092Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:41:35.530093Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:41:35.530095Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:41:35.530096Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:41:41.037539Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3968ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3968 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log new file mode 100644 index 000000000..83724cdaa --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log @@ -0,0 +1,38 @@ +2026-09-21T08:41:52.156919Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:41:52.156970Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:41:52.157100Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:41:52.157147Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:41:52.157149Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:41:52.157151Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:41:52.481846Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:41:52.481882Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:41:52.482430Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:41:52.809249Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:41:52.809761Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:41:53.167730Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.358s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.357931069 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:41:53.167812Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=3, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:41:53.168226Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:41:53.168242Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:41:53.168264Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:41:53.168481Z WARN mlxcel::models::speculative_exactness: MTP declined: exactness probe did not run: on CUDA this head_dim reaches the fused sdpa_vector kernels, where a verify block's per-position attention is not bit-equal to the single-token decode it stands for (issue #1935); MLXCEL_SDPA_VECTOR_LARGE_D=0 restores it. Disabling qmv_wide did not make it exact either. Falling back to classic decode. Set MLXCEL_MTP_ALLOW_INEXACT=1 to engage anyway and forfeit the temperature-0 byte-identity contract. block_size=3 +2026-09-21T08:41:53.168487Z WARN mlxcel::server::batch::speculative_burst: DFlash speculative dispatch declined for seq seq-0: the block-versus-chain exactness probe did not pass at block_size=3 on this host, so a 3-row verify block would not reproduce classic decode's tokens; falling back to classic decode (see the probe verdict logged above; MLXCEL_MTP_ALLOW_INEXACT=1 engages anyway and forfeits the contract) +2026-09-21T08:41:53.169449Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:41:54.111801Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 943ms prompt_tokens=1 cached_tokens=0 generation_time_ms=943 +2026-09-21T08:41:54.112243Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:41:54.112349Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:41:54.113023Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:41:54.113772Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18941 +2026-09-21T08:41:54.113777Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:41:54.113781Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:41:54.113782Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:41:54.113783Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:41:54.113785Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:41:54.113786Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:41:54.113787Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:41:54.113789Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:41:54.113790Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:41:54.113791Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:41:54.113793Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:41:54.113794Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:41:55.879733Z WARN mlxcel::server::batch::speculative_burst: DFlash speculative dispatch declined for seq seq-1: the block-versus-chain exactness probe did not pass at block_size=3 on this host, so a 3-row verify block would not reproduce classic decode's tokens; falling back to classic decode (see the probe verdict logged above; MLXCEL_MTP_ALLOW_INEXACT=1 engages anyway and forfeits the contract) +2026-09-21T08:42:00.051438Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 4171ms prompt_tokens=158 cached_tokens=0 generation_time_ms=4171 diff --git a/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-novec.log b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-novec.log new file mode 100644 index 000000000..85f16e130 --- /dev/null +++ b/docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-novec.log @@ -0,0 +1,44 @@ +2026-09-21T08:41:41.759287Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins +2026-09-21T08:41:41.759330Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0 +2026-09-21T08:41:41.759469Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None +2026-09-21T08:41:41.759525Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA) +2026-09-21T08:41:41.759527Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set) +2026-09-21T08:41:41.759530Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB +2026-09-21T08:41:42.102210Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"] +2026-09-21T08:41:42.102245Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("") think_end=Some("") think_start_tokens_len=1 think_end_tokens_len=1 +2026-09-21T08:41:42.102802Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256 +2026-09-21T08:41:42.445614Z INFO mlxcel::server::startup: Warming up model... +2026-09-21T08:41:42.445770Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model... +2026-09-21T08:41:42.805458Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.360s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.359663724 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401 +2026-09-21T08:41:42.805538Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=3, block_size_from=operator override, explicit_kind=true)) +2026-09-21T08:41:42.805953Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks) +2026-09-21T08:41:42.805969Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab) +2026-09-21T08:41:42.805992Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32 +2026-09-21T08:41:44.599988Z INFO mlxcel::models::speculative_exactness: MTP exactness probe passed: verify block is byte-identical to the single-token chain block_size=3 +2026-09-21T08:41:44.600018Z INFO mlxcel::server::batch::speculative_burst: Lazy-loading drafter from /home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash (kind=Some(Dflash)) +2026-09-21T08:41:44.600398Z INFO mlxcel::server::batch::speculative_burst: Drafter loaded (kind=dflash, 0 ms) +2026-09-21T08:41:44.601321Z INFO mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved +2026-09-21T08:41:44.624265Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=0 proposed_tokens=0 accepted_tokens=0 acceptance_rate=0.0 emitted_per_verify=0.0 zero_accept_rounds=0 partial_accept_rounds=0 full_accept_rounds=0 prefill_verify_ms=0.880893 first_bonus_ms=22.784719 first_hidden_ms=0.017792 bind_reset_ms=0.13232 draft_ms=0.0 verify_ms=0.0 target_argmax_sync_ms=0.0 logprobs_ms=0.0 walk_ms=0.0 hidden_concat_ms=0.0 rollback_ms=0.0 decode_ms=0.0 +2026-09-21T08:41:44.624358Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[] +2026-09-21T08:41:44.624631Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=1 generated_tokens=1 burst_ms=1818 +2026-09-21T08:41:44.624642Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-0 burst_wall_ms=1818.431133 burst_active_ms=1818.431133 slices=1 tokens_generated=1 rounds=0 accepted_draft_tokens=0 hol_waiters=0 +2026-09-21T08:41:44.624725Z INFO mlxcel::server::startup: Warmup complete +2026-09-21T08:41:44.625112Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support +2026-09-21T08:41:44.626701Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg. +2026-09-21T08:41:44.627519Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18941 +2026-09-21T08:41:44.627526Z INFO mlxcel::server::startup: Detected 1 GPU(s) +2026-09-21T08:41:44.627529Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin) +2026-09-21T08:41:44.627531Z INFO mlxcel::server::startup: Endpoints: +2026-09-21T08:41:44.627532Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions +2026-09-21T08:41:44.627534Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions +2026-09-21T08:41:44.627535Z INFO mlxcel::server::startup: GET /v1/models - List models +2026-09-21T08:41:44.627536Z INFO mlxcel::server::startup: POST /completion - llama-server native completion +2026-09-21T08:41:44.627537Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text +2026-09-21T08:41:44.627539Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens +2026-09-21T08:41:44.627540Z INFO mlxcel::server::startup: GET /props - Server properties +2026-09-21T08:41:44.627541Z INFO mlxcel::server::startup: GET /slots - Slot status +2026-09-21T08:41:44.627543Z INFO mlxcel::server::startup: GET /health - Health check +2026-09-21T08:41:51.453837Z INFO mlxcel::server::batch::dflash_target: DFlash diagnostics block_size=3 rounds=97 proposed_tokens=194 accepted_tokens=102 acceptance_rate=0.5257731958762887 emitted_per_verify=2.051546391752577 zero_accept_rounds=36 partial_accept_rounds=20 full_accept_rounds=41 prefill_verify_ms=4.922623 first_bonus_ms=431.598634 first_hidden_ms=0.004944 bind_reset_ms=0.023808 draft_ms=2496.762945000001 verify_ms=226.62330500000002 target_argmax_sync_ms=2461.5770899999998 logprobs_ms=324.58313399999986 walk_ms=0.033024000000000005 hidden_concat_ms=0.14614399999999997 rollback_ms=30.258836000000006 decode_ms=5547.549276 +2026-09-21T08:41:51.453882Z DEBUG mlxcel::server::batch::dflash_target: DFlash accept lengths accept_lens=[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 2, 0, 2, 1, 0, 2, 0, 0, 0, 2, 0, 1, 1, 0, 0, 2, 2, 2, 1, 2, 1, 2, 1, 1, 0, 0, 0, 0, 2, 2, 0, 1, 0, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 2, 2, 0, 1, 1, 2, 2, 1, 0, 0, 0, 0, 1, 0, 0, 2, 1, 2, 0, 2, 0, 2, 2, 0, 2, 1, 0, 1, 2, 2, 2, 2, 2, 2, 2, 2] +2026-09-21T08:41:51.454202Z INFO mlxcel::server::batch::speculative_burst: Speculative burst completed prompt_tokens=158 generated_tokens=200 burst_ms=5984 +2026-09-21T08:41:51.454301Z INFO mlxcel::server::batch::scheduler::speculative_finalize: speculative B=1 burst finalized (burst_wall_ms is the max single-tick HOL stall on concurrent rows) seq_id=seq-1 burst_wall_ms=5984.639332000001 burst_active_ms=5984.639332000001 slices=1 tokens_generated=200 rounds=0 accepted_draft_tokens=0 hol_waiters=0 diff --git a/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md b/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md index 73b5de0dd..6ff3a9591 100644 --- a/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md +++ b/docs/benchmark_results/dflash-sdpav-options-qwen35-gb10-2026-09-21.md @@ -56,4 +56,16 @@ Two facts the table carries that are easy to miss. **The fallback costs the burs Byte-identical speculation is achievable and is worth 1.13x; dropping the byte-identity requirement is worth 1.22x; and the fused kernel is worth about 8% to the burst while being worth nothing measurable to classic decode at this context length. -A is the better default anyway, because it is the only option that leaves today's output and today's non-speculative performance exactly as they are, and because B's price is paid by every request on the process rather than by the ones that asked for speculation. B is worth documenting as the operator recipe for byte-identical speculation rather than shipping as a default, and it needs no new code: #1944's guard stands down when the flag is off, so an operator who sets `MLXCEL_SDPA_VECTOR_LARGE_D=0` gets the burst engaged and byte-identical on the same binary. +A is the better default anyway, because it is the only option that leaves today's output and today's non-speculative performance exactly as they are, and because B's price is paid by every request on the process rather than by the ones that asked for speculation. B is worth documenting as the operator recipe for byte-identical speculation rather than shipping as a default, and it needs no new code. + +## That last claim, measured on the PR #1944 binary rather than argued + +The arms above run an `origin/main` binary, so they price the mechanism, not the gate. Three more arms run PR #1944's own binary (sha `9ae64508`, verified to carry the guard before it ran) to check that A and B are both reachable from the one build. + +| arm | `MLXCEL_SDPA_VECTOR_LARGE_D` | declines logged | bursts completed | completion sha256 | +|---|---|---:|---:|---| +| classic, no drafter | 0 | 0 | 0 | `4c37650547` | +| width 3 | 0 | 0 | 2 | `4c37650547` | +| width 3 | 1 (the default) | 2 | 0 | `2c76b0a181` | + +At the shipped default the guard declines, no burst runs, and the server returns exactly today's classic text. With the fallback selected the guard stands down, the burst runs, and its text is byte-identical to the drafter-less server in the same configuration. So one binary carries both options and the operator recipe for byte-identical speculation is a single environment variable.