Skip to content

fix(speculative): decline the Qwen 3.5 DFlash burst the probe cannot vet #1944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
inureyes merged 16 commits into from
Sep 21, 2026
Merged

fix(speculative): decline the Qwen 3.5 DFlash burst the probe cannot vet #1944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5cc25cc
test(speculative): log the DFlash round transcript and drive the burs…
inureyes Sep 21, 2026
6cfd296
test(speculative): record the served DFlash arms and their round tran…
inureyes Sep 21, 2026
4e99a6b
docs(benchmarks): record what the served arms settle about the width …
inureyes Sep 21, 2026
76c5121
fix(speculative): give a Qwen 3.5 verify row the query layout decode …
inureyes Sep 21, 2026
1e88fd6
test(speculative): revert the query-layout attempt and bisect the blo…
inureyes Sep 21, 2026
481e2db
docs(benchmarks): correct the cause section and let a measured all-de…
inureyes Sep 21, 2026
c3b88fc
test(speculative): keep the query-layout A/B arms as a recorded negat…
inureyes Sep 21, 2026
533329e
fix(speculative): decline the Qwen 3.5 DFlash burst where the probe c…
inureyes Sep 21, 2026
0ef4f8e
test(speculative): satisfy clippy in the byte-bisect arm
inureyes Sep 21, 2026
da53dea
test(speculative): derive the residual harness's paths instead of har…
inureyes Sep 21, 2026
f830f46
fix(speculative): read the fused-kernel switch narrowly in the exactn…
inureyes Sep 21, 2026
c86574a
docs(benchmarks): record the served-path parity result and the hole i…
inureyes Sep 21, 2026
2fb00b5
docs(benchmarks): correct the accept-pattern count in the residual re…
inureyes Sep 21, 2026
6d095c8
docs(benchmarks): finish the residual data README
inureyes Sep 21, 2026
85f73c3
docs(benchmarks): price every option for byte-identical Qwen 3.5 DFlash
inureyes Sep 21, 2026
1595b20
docs(benchmarks): verify both options are reachable from the PR #1944…
inureyes Sep 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
435 changes: 435 additions & 0 deletions ...chmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-classic-novec.json
View file
Open in desktop

Large diffs are not rendered by default.

441 changes: 441 additions & 0 deletions docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-fused.json
View file
Open in desktop

Large diffs are not rendered by default.

441 changes: 441 additions & 0 deletions docs/benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/arm.gate-w3-novec.json
View file
Open in desktop

Large diffs are not rendered by default.

35 changes: 35 additions & 0 deletions ...mark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-classic-novec.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
2026-09-21T08:41:33.424813Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins
2026-09-21T08:41:33.424960Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0
2026-09-21T08:41:33.425373Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None
2026-09-21T08:41:33.425543Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA)
2026-09-21T08:41:33.425545Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set)
2026-09-21T08:41:33.425547Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB
2026-09-21T08:41:33.770270Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"]
2026-09-21T08:41:33.770411Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("<think>") think_end=Some("</think>") think_start_tokens_len=1 think_end_tokens_len=1
2026-09-21T08:41:33.771449Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256
2026-09-21T08:41:34.118910Z INFO mlxcel::server::startup: Warming up model...
2026-09-21T08:41:34.119634Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model...
2026-09-21T08:41:34.457333Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.338s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.337673483 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401
2026-09-21T08:41:34.457474Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto)
2026-09-21T08:41:34.457902Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks)
2026-09-21T08:41:34.457916Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab)
2026-09-21T08:41:34.458065Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32
2026-09-21T08:41:34.460455Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved
2026-09-21T08:41:35.526896Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 1068ms prompt_tokens=1 cached_tokens=0 generation_time_ms=1068
2026-09-21T08:41:35.527225Z INFO mlxcel::server::startup: Warmup complete
2026-09-21T08:41:35.527389Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support
2026-09-21T08:41:35.528377Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg.
2026-09-21T08:41:35.530064Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18941
2026-09-21T08:41:35.530080Z INFO mlxcel::server::startup: Detected 1 GPU(s)
2026-09-21T08:41:35.530083Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin)
2026-09-21T08:41:35.530085Z INFO mlxcel::server::startup: Endpoints:
2026-09-21T08:41:35.530086Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions
2026-09-21T08:41:35.530087Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions
2026-09-21T08:41:35.530089Z INFO mlxcel::server::startup: GET /v1/models - List models
2026-09-21T08:41:35.530090Z INFO mlxcel::server::startup: POST /completion - llama-server native completion
2026-09-21T08:41:35.530091Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text
2026-09-21T08:41:35.530092Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens
2026-09-21T08:41:35.530093Z INFO mlxcel::server::startup: GET /props - Server properties
2026-09-21T08:41:35.530095Z INFO mlxcel::server::startup: GET /slots - Slot status
2026-09-21T08:41:35.530096Z INFO mlxcel::server::startup: GET /health - Health check
2026-09-21T08:41:41.037539Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 3968ms prompt_tokens=158 cached_tokens=0 generation_time_ms=3968
38 changes: 38 additions & 0 deletions ...benchmark_results/data/dflash-sdpav-options-gb10-2026-09-21/gate/server.gate-w3-fused.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
2026-09-21T08:41:52.156919Z WARN mlxcel::server::startup: CORS is set to allow all origins ('*') and no API key is set; this can be a security risk (cross-origin attacks). Set --api-key, or narrow --cors-origins / --allowed-origins
2026-09-21T08:41:52.156970Z INFO mlxcel::server::startup: effective KV cache mode kv_cache_mode=fp16 kv_bits=0
2026-09-21T08:41:52.157100Z INFO mlxcel::server::startup: resolved context and batch geometry (0 = the checkpoint's own trained context) ctx_size=0 ctx_size_per_slot=0 context_slots=1 kv_unified=false n_parallel=4 prefill_chunk_size=512 max_kv_size=None
2026-09-21T08:41:52.157147Z INFO mlxcel::server::startup: Runtime device: NVIDIA GPU (CUDA)
2026-09-21T08:41:52.157149Z INFO mlxcel::server::startup: CUDA graph-cache LRU capacity: MLX_CUDA_GRAPH_CACHE_SIZE=2000 (mlxcel raises MLX's default of 400 to 2000 so long-lived, shape-diverse decode does not hit the cache-thrashing abort from issue #818, unless an operator override is set)
2026-09-21T08:41:52.157151Z INFO mlxcel::server::startup: Wired memory limit: 121.7 GB
2026-09-21T08:41:52.481846Z INFO mlxcel::server::startup: DRY sequence breakers active (b10621 semantics: breaker token data derived from the vocabulary per request) breakers=["\n", ":", "\"", "*"]
2026-09-21T08:41:52.481882Z INFO mlxcel::server::startup: Tokenizer recognizes a think marker pair; defaulting chat_template kwarg `enable_thinking=true` (upstream PR #1114) think_start=Some("<think>") think_end=Some("</think>") think_start_tokens_len=1 think_end_tokens_len=1
2026-09-21T08:41:52.482430Z INFO mlxcel::server::startup: Prompt-prefix cache store enabled (+ APC, snapshots) capacity_bytes=2147483648 max_entries=1024 ttl_seconds=3600 snapshot_capacity_bytes=536870912 snapshot_max_entries=4096 snapshot_ttl_seconds=7200 min_prefix_tokens=32 apc_enabled=true apc_block_size=16 apc_hash=sha256
2026-09-21T08:41:52.809249Z INFO mlxcel::server::startup: Warming up model...
2026-09-21T08:41:52.809761Z INFO mlxcel::server::model_provider::model_worker: Model worker thread starting, loading model...
2026-09-21T08:41:53.167730Z INFO mlxcel::server::model_provider::model_worker: Model qwen3.5-4b-4bit loaded in 0.358s (resident after load: 0.00 GB) worker_model_id=qwen3.5-4b-4bit load_seconds=0.357931069 active_bytes=0 peak_bytes=0 cache_bytes=0 limit_bytes=124128085401
2026-09-21T08:41:53.167812Z INFO mlxcel::server::model_provider::model_worker: Starting BatchScheduler (max_batch_size=1, max_queue_depth=32, prefill_chunk_size=512, max_batch_prefill=4, decode_storage=auto, speculative=dflash (drafter=/home/inureyes/Development/mlxcel-wt-1935/models/mlx/qwen3.5-4b-dflash, block_size=3, block_size_from=operator override, explicit_kind=true))
2026-09-21T08:41:53.168226Z INFO mlxcel::server::model_provider::model_worker: Paged KV block budget: 3062487 blocks (32 layers, 32-token blocks)
2026-09-21T08:41:53.168242Z INFO mlxcel::server::model_provider::model_worker: Paged KV slab size: 256 blocks per layer (fused decode serves a layer only while its rows fit one slab)
2026-09-21T08:41:53.168264Z INFO mlxcel::server::batch::scheduler::paged_layout: resolved KV cache mode applied to model caches kv_cache_mode_effective=fp16 kv_cache_mode_applied_layers=32 kv_cache_mode_total_layers=32
2026-09-21T08:41:53.168481Z WARN mlxcel::models::speculative_exactness: MTP declined: exactness probe did not run: on CUDA this head_dim reaches the fused sdpa_vector kernels, where a verify block's per-position attention is not bit-equal to the single-token decode it stands for (issue #1935); MLXCEL_SDPA_VECTOR_LARGE_D=0 restores it. Disabling qmv_wide did not make it exact either. Falling back to classic decode. Set MLXCEL_MTP_ALLOW_INEXACT=1 to engage anyway and forfeit the temperature-0 byte-identity contract. block_size=3
2026-09-21T08:41:53.168487Z WARN mlxcel::server::batch::speculative_burst: DFlash speculative dispatch declined for seq seq-0: the block-versus-chain exactness probe did not pass at block_size=3 on this host, so a 3-row verify block would not reproduce classic decode's tokens; falling back to classic decode (see the probe verdict logged above; MLXCEL_MTP_ALLOW_INEXACT=1 engages anyway and forfeits the contract)
2026-09-21T08:41:53.169449Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel_core::sampling_dispatch: sampling dispatch: argmax: greedy path (temperature 0, top_k 1); no sampling kernel involved
2026-09-21T08:41:54.111801Z INFO prefill{seq_id=seq-0 prompt_len=1 cached=0 start=0}: mlxcel::server::batch::scheduler::prefill: prompt-cache: request completed during prefill: cached=0/1 prompt tokens, total 943ms prompt_tokens=1 cached_tokens=0 generation_time_ms=943
2026-09-21T08:41:54.112243Z INFO mlxcel::server::startup: Warmup complete
2026-09-21T08:41:54.112349Z INFO mlxcel::server::startup: model_type=Qwen35VLM: enabling native video_url content block support
2026-09-21T08:41:54.113023Z WARN mlxcel::server::startup: The loaded model accepts video input, but `ffmpeg` and `ffprobe` are not both on PATH, so every video request will be refused. The check is cached for the life of the process: restart the server after installing ffmpeg.
2026-09-21T08:41:54.113772Z INFO mlxcel::server::startup: Starting mlxcel server on http://127.0.0.1:18941
2026-09-21T08:41:54.113777Z INFO mlxcel::server::startup: Detected 1 GPU(s)
2026-09-21T08:41:54.113781Z INFO mlxcel::server::startup: CUDA compute capability 12.1 (sm_121); compiled for [121] (cubin)
2026-09-21T08:41:54.113782Z INFO mlxcel::server::startup: Endpoints:
2026-09-21T08:41:54.113783Z INFO mlxcel::server::startup: POST /v1/chat/completions - OpenAI chat completions
2026-09-21T08:41:54.113785Z INFO mlxcel::server::startup: POST /v1/completions - OpenAI text completions
2026-09-21T08:41:54.113786Z INFO mlxcel::server::startup: GET /v1/models - List models
2026-09-21T08:41:54.113787Z INFO mlxcel::server::startup: POST /completion - llama-server native completion
2026-09-21T08:41:54.113789Z INFO mlxcel::server::startup: POST /tokenize - Tokenize text
2026-09-21T08:41:54.113790Z INFO mlxcel::server::startup: POST /detokenize - Detokenize tokens
2026-09-21T08:41:54.113791Z INFO mlxcel::server::startup: GET /props - Server properties
2026-09-21T08:41:54.113793Z INFO mlxcel::server::startup: GET /slots - Slot status
2026-09-21T08:41:54.113794Z INFO mlxcel::server::startup: GET /health - Health check
2026-09-21T08:41:55.879733Z WARN mlxcel::server::batch::speculative_burst: DFlash speculative dispatch declined for seq seq-1: the block-versus-chain exactness probe did not pass at block_size=3 on this host, so a 3-row verify block would not reproduce classic decode's tokens; falling back to classic decode (see the probe verdict logged above; MLXCEL_MTP_ALLOW_INEXACT=1 engages anyway and forfeits the contract)
2026-09-21T08:42:00.051438Z INFO mlxcel::server::batch::scheduler::decode_tick: prompt-cache: request completed: cached=0/158 prompt tokens, total 4171ms prompt_tokens=158 cached_tokens=0 generation_time_ms=4171
Loading
Loading