RuntimeError: dflash daemon exited before weights finished loading. Check the daemon's stderr.

[DemonODG]

First, I tried splitting the target model and the draft onto different video cards. I got an error:

CUDA_VISIBLE_DEVICES=1,0 DFLASH_TARGET_GPU=0 DFLASH_DRAFT_GPU=1 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/model.safetensors --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

[init] spawning daemon: ../dflash/build/test_dflash
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=1
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 36857 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
Device 1: NVIDIA GeForce RTX 4080 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 12281 MiB
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
draft load: safetensors: 'layers.0.self_attn.k_norm.weight' shape[0]=128 expected 256
Traceback (most recent call last):
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 184, in
main()
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 123, in main
dflash = DflashClient(
^^^^^^^^^^^^^
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 91, in init
self._wait_until_loaded(timeout=boot_timeout_s, vram_mib=boot_vram_mib)
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 101, in _wait_until_loaded
raise RuntimeError(
RuntimeError: dflash daemon exited before weights finished loading. Check the daemon's stderr.

Then I used only RTX 3090 and I get the same error:

CUDA_VISIBLE_DEVICES=1 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen
/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/model.safetensors --drafter-gguf ../../ModelsIA/Qw
en/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

[init] spawning daemon: ../dflash/build/test_dflash
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
draft load: safetensors: 'layers.0.self_attn.k_norm.weight' shape[0]=128 expected 256
Traceback (most recent call last):
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 184, in
main()
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 123, in main
dflash = DflashClient(
^^^^^^^^^^^^^
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 91, in init
self._wait_until_loaded(timeout=boot_timeout_s, vram_mib=boot_vram_mib)
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 101, in _wait_until_loaded
raise RuntimeError(
RuntimeError: dflash daemon exited before weights finished loading. Check the daemon's stderr.

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Sep__8_19:17:24_PDT_2023
Cuda compilation tools, release 12.3, V12.3.52
Build cuda_12.3.r12.3/compiler.33281558_0

Please tell me the possible reasons for this error and how to fix it.

[davide221]

../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/model.safetensors Is this the correct dflash drafter like https://huggingface.co/Lucebox/Qwen3.6-27B-DFlash-GGUF

[davide221]

#195 is probably related

[DemonODG]

I removed and recompiled taking into account #195 lucebox-hub.
All three models load successfully on different video cards. This is visible via nvidia-smi.
But then, unexpectedly, this error appears:

DFLASH_TARGET_GPU=1 DFLASH_DRAFT_GPU=0 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

[init] spawning daemon: ../dflash/build/test_dflash
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=1 draft_gpu=0
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 36857 MiB):
Device 0: NVIDIA GeForce RTX 4080 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 12281 MiB
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft] loaded
[daemon] ready
Traceback (most recent call last):
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 184, in
main()
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 123, in main
dflash = DflashClient(
^^^^^^^^^^^^^
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 91, in init
self._wait_until_loaded(timeout=boot_timeout_s, vram_mib=boot_vram_mib)
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 113, in _wait_until_loaded
raise RuntimeError(
RuntimeError: dflash daemon failed to load target weights within 60s (expected VRAM > 18000 MiB). Check the daemon's stderr.

Then I tried running it on a 3090, and the error remained:

CUDA_VISIBLE_DEVICES=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/d
raft/model.safetensors --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256
[warn] BSA disabled. NIAH validation requires BSA enabled (--bsa 1 or DFLASH_FP_USE_BSA set in env); the WMMA fallback path is ~3.4x slower at long ctx and silently fails NIAH.
[init] spawning daemon: ../dflash/build/test_dflash
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft] loaded
[daemon] ready
Traceback (most recent call last):
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 184, in
main()
File "/home/dimanodg/myproject/lucebox-hub/pflash/tests/bench_niah_cpp.py", line 123, in main
dflash = DflashClient(
^^^^^^^^^^^^^
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 91, in init
self._wait_until_loaded(timeout=boot_timeout_s, vram_mib=boot_vram_mib)
File "/home/dimanodg/myproject/lucebox-hub/pflash/pflash/dflash_client.py", line 113, in _wait_until_loaded
raise RuntimeError(
RuntimeError: dflash daemon failed to load target weights within 60s (expected VRAM > 18000 MiB). Check the daemon's stderr.

[weicj]

@DemonODG I've made a small patch for the timeout issue, can you try and test if it works? #235

[reverb256]

Closing this issue as this is an external repository and should not have internal hygiene issues created.

[DemonODG]

I disabled the RTX 4080 laptop in the devices and ran the test:

DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256
[init] spawning daemon: ../dflash/build/test_dflash
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft] loaded
[daemon] ready
[park] draft released
[case 0] src=131068 keep=0.05
[park] target released
[compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ...
[drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936
[compress] drafter ready
[qwen3-0.6b-fp] layer 1/28 done (A_setup=0.000s A_alloc=0.001s A_compute=1.962s FP=0.127s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s)
[qwen3-0.6b-fp] layer 28/28 done (A_setup=0.005s A_alloc=0.003s A_compute=4.845s FP=4.220s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s)
[qwen3-0.6b-fp] forward 21.29s (S=131068, A_setup=0.00s A_alloc=0.00s A_compute=4.84s FP=4.22s B_warm=0.00s B_setup=0.00s B_alloc=0.00s B_copy_in=0.00s B_norm=0.00s B_compute=0.00s B_copy_out=0.00s) tail-score 4.32s total 25.61s
[drafter] forward+score in 26.20s S=131068
[drafter] score_and_compress total 26.20s S=131068 kept=6524 (204/4096 chunks, forced=37)
[compress] 131068 -> 6524 tokens
[case 0] compressed=6524 ratio=20.1x score_s=32.1
[loader] eos_id=248046 eos_chat_id=-1
[unpark] target restored
[drafter] freed
[unpark] draft restored
[spec-decode] tokens=193 time=5.401 s speed=35.73 tok/s steps=25 accepted=170/400 (42.5%) avg_commit=7.72
ok N=6547 gen=193 prefill_s=7.887 decode_s=5.401 decode_tok_s=35.7 stream_fd=5
[park] draft released
[case 0] raw out_ids (193): [760, 1156, 6587, 310, 1423, 279, 328, 14703, 10642, 2715, 1413, 69, 371, 7891, 1324, 1, 494, 279, 3766, 1414]
[case 0] out_with_special: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.<|im_end|>'
[case 0] gen_s=13.4 ttft=45.5 ok=True ans=4025016
[case 0] out: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.'

accuracy: 1/1

It works on one card!
Interestingly, trying to run it on one card with the environment variable CUDA_VISIBLE_DEVICES=1 (RTX 3090) returns an error (see above).

I'll keep investigating. Thanks for your help!

[weicj]

I disabled the RTX 4080 laptop in the devices and ran the test:

DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256 [init] spawning daemon: ../dflash/build/test_dflash [cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0 [test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5) ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB [loader] eos_id=248046 eos_chat_id=-1 [target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K) [draft] loaded [daemon] ready [park] draft released [case 0] src=131068 keep=0.05 [park] target released [compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ... [drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936 [compress] drafter ready [qwen3-0.6b-fp] layer 1/28 done (A_setup=0.000s A_alloc=0.001s A_compute=1.962s FP=0.127s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s) [qwen3-0.6b-fp] layer 28/28 done (A_setup=0.005s A_alloc=0.003s A_compute=4.845s FP=4.220s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s) [qwen3-0.6b-fp] forward 21.29s (S=131068, A_setup=0.00s A_alloc=0.00s A_compute=4.84s FP=4.22s B_warm=0.00s B_setup=0.00s B_alloc=0.00s B_copy_in=0.00s B_norm=0.00s B_compute=0.00s B_copy_out=0.00s) tail-score 4.32s total 25.61s [drafter] forward+score in 26.20s S=131068 [drafter] score_and_compress total 26.20s S=131068 kept=6524 (204/4096 chunks, forced=37) [compress] 131068 -> 6524 tokens [case 0] compressed=6524 ratio=20.1x score_s=32.1 [loader] eos_id=248046 eos_chat_id=-1 [unpark] target restored [drafter] freed [unpark] draft restored [spec-decode] tokens=193 time=5.401 s speed=35.73 tok/s steps=25 accepted=170/400 (42.5%) avg_commit=7.72 ok N=6547 gen=193 prefill_s=7.887 decode_s=5.401 decode_tok_s=35.7 stream_fd=5 [park] draft released [case 0] raw out_ids (193): [760, 1156, 6587, 310, 1423, 279, 328, 14703, 10642, 2715, 1413, 69, 371, 7891, 1324, 1, 494, 279, 3766, 1414] [case 0] out_with_special: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.<|im_end|>' [case 0] gen_s=13.4 ttft=45.5 ok=True ans=4025016 [case 0] out: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.'

accuracy: 1/1

It works on one card! Interestingly, trying to run it on one card with the environment variable CUDA_VISIBLE_DEVICES=1 (RTX 3090) returns an error (see above).

I'll keep investigating. Thanks for your help!

My guess is when CUDA_VISIBLE_DEVICES=1 is set, that physical GPU becomes logical GPU 0 inside the process, so DFLASH_TARGET_GPU and DFLASH_DRAFT_GPU should both be 0. Also make sure --draft-spec points to the DFlash GGUF draft model, not the old safetensors draft, otherwise it may hit the earlier shape mismatch again.

You can try this config:

CUDA_VISIBLE_DEVICES=1
DFLASH_TARGET_GPU=0
DFLASH_DRAFT_GPU=0
DFLASH_FP_USE_BSA=1
DFLASH_FP_ALPHA=0.85
python tests/bench_niah_cpp.py
--bin ../dflash/build/test_dflash
--target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf
--draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf
--drafter-arch qwen3-0.6b
--drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf
--cases /tmp/niah_128k.jsonl
--keep-ratio 0.05
--n-gen 256

[DemonODG]

@DemonODG I've made a small patch for the timeout issue, can you try and test if it works? #235

I installed 235 and the script worked on two video cards! This is amazing!!! Great job, guys! I look forward to seeing how this project develops.
weicj - Great respect!!!

DFLASH_TARGET_GPU=1 DFLASH_DRAFT_GPU=0 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

[init] spawning daemon: ../dflash/build/test_dflash
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 36857 MiB):
Device 0: NVIDIA GeForce RTX 4080 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 12281 MiB
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=1 draft_gpu=0
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft] loaded
[daemon] ready
[park] draft released
[case 0] src=131068 keep=0.05
[park] target released
[compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ...
[drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936
[compress] drafter ready
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 255.99 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 268427264
[case 0] compressed=0 ratio=131068.0x score_s=4.7
[loader] eos_id=248046 eos_chat_id=-1
[unpark] target restored
[drafter] freed
[unpark] draft restored
[dflash] Using safe (slower) cross-GPU copy via host staging (--peer-access not set or P2P unavailable for this device pair).
[spec-decode] tokens=256 time=7.099 s speed=36.06 tok/s steps=36 accepted=224/576 (38.9%) avg_commit=7.11
ok N=20 gen=256 prefill_s=3.544 decode_s=7.099 decode_tok_s=36.1 stream_fd=5
[park] draft released
[case 0] raw out_ids (256): [8160, 579, 264, 7047, 1817, 25, 271, 16, 13, 220, 2972, 2014, 53983, 2570, 5396, 64700, 198, 256, 471, 2972]
[case 0] out_with_special: 'Here's a thinking process:\n\n1. Analyze User Input:\n - User Question: "Answer the user question based on the above context."\n - Context: None provided. The prompt says "based on the above context," but there is no context provided in the prompt.\n - Task: I need to respond appropriately given the missing context.\n\n2. Identify Key Issue:\n - The user forgot to provide the context or the actual question.\n - I cannot answer a question without knowing what the question is or what context to base it on.\n\n3. Determine Response Strategy:\n - Acknowledge the missing information.\n - Politely ask the user to provide both the context and the specific question.\n - Keep it concise and helpful.\n\n4. Draft Response (Mental Refinement):\n "It looks like you forgot to include the context and the specific question you'd like me to answer. Please provide both, and I’ll be happy to help!"\n\n5. Check Against Guidelines:\n - Direct, clear, addresses the missing information.\n - No extra fluff.\n - Ready.✅\n'
[case 0] gen_s=10.8 ttft=15.5 ok=False ans=4025016
[case 0] out: 'Here's a thinking process:\n\n1. Analyze User Input:\n - User Question: "Answer the user question based on the above context."\n - Context: None provided. The prompt says "based on the above context," but there is no context provided in the prompt.\n - Task: I need to respond appropriately given the missing context.\n\n2. Identify Key Issue:\n - The user forgot to provide the context or the actual question.\n - I cannot answer a question without knowing what the question is or what context to base it on.\n\n3. Determine Response Strategy:\n - Acknowledge the missing information.\n - Politely ask the user to provide both the context and the specific question.\n - Keep it concise and helpful.\n\n4. Draft Response (Mental Refinement):\n "It looks like you forgot to include the context and the specific question you'd like me to answer. Please provide both, and I’ll be happy to help!"\n\n5. Check Against Guidelines:\n - Direct, clear, addresses the missing information.\n - No extra fluff.\n - Ready.✅\n'

accuracy: 0/1

[weicj]

@DemonODG I've made a small patch for the timeout issue, can you try and test if it works? #235

I installed 235 and the script worked on two video cards! This is amazing!!! Great job, guys! I look forward to seeing how this project develops. weicj - Great respect!!!

DFLASH_TARGET_GPU=1 DFLASH_DRAFT_GPU=0 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

[init] spawning daemon: ../dflash/build/test_dflash [test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5) ggml_cuda_init: found 2 CUDA devices (Total VRAM: 36857 MiB): Device 0: NVIDIA GeForce RTX 4080 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 12281 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB [cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=1 draft_gpu=0 [loader] eos_id=248046 eos_chat_id=-1 [target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K) [draft] loaded [daemon] ready [park] draft released [case 0] src=131068 keep=0.05 [park] target released [compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ... [drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936 [compress] drafter ready ggml_backend_cuda_buffer_type_alloc_buffer: allocating 255.99 MiB on device 0: cudaMalloc failed: out of memory alloc_tensor_range: failed to allocate CUDA0 buffer of size 268427264 [case 0] compressed=0 ratio=131068.0x score_s=4.7 [loader] eos_id=248046 eos_chat_id=-1 [unpark] target restored [drafter] freed [unpark] draft restored [dflash] Using safe (slower) cross-GPU copy via host staging (--peer-access not set or P2P unavailable for this device pair). [spec-decode] tokens=256 time=7.099 s speed=36.06 tok/s steps=36 accepted=224/576 (38.9%) avg_commit=7.11 ok N=20 gen=256 prefill_s=3.544 decode_s=7.099 decode_tok_s=36.1 stream_fd=5 [park] draft released [case 0] raw out_ids (256): [8160, 579, 264, 7047, 1817, 25, 271, 16, 13, 220, 2972, 2014, 53983, 2570, 5396, 64700, 198, 256, 471, 2972] [case 0] out_with_special: 'Here's a thinking process:\n\n1. Analyze User Input:\n - User Question: "Answer the user question based on the above context."\n - Context: None provided. The prompt says "based on the above context," but there is no context provided in the prompt.\n - Task: I need to respond appropriately given the missing context.\n\n2. Identify Key Issue:\n - The user forgot to provide the context or the actual question.\n - I cannot answer a question without knowing what the question is or what context to base it on.\n\n3. Determine Response Strategy:\n - Acknowledge the missing information.\n - Politely ask the user to provide both the context and the specific question.\n - Keep it concise and helpful.\n\n4. Draft Response (Mental Refinement):\n "It looks like you forgot to include the context and the specific question you'd like me to answer. Please provide both, and I’ll be happy to help!"\n\n5. Check Against Guidelines:\n - Direct, clear, addresses the missing information.\n - No extra fluff.\n - Ready.✅\n' [case 0] gen_s=10.8 ttft=15.5 ok=False ans=4025016 [case 0] out: 'Here's a thinking process:\n\n1. Analyze User Input:\n - User Question: "Answer the user question based on the above context."\n - Context: None provided. The prompt says "based on the above context," but there is no context provided in the prompt.\n - Task: I need to respond appropriately given the missing context.\n\n2. Identify Key Issue:\n - The user forgot to provide the context or the actual question.\n - I cannot answer a question without knowing what the question is or what context to base it on.\n\n3. Determine Response Strategy:\n - Acknowledge the missing information.\n - Politely ask the user to provide both the context and the specific question.\n - Keep it concise and helpful.\n\n4. Draft Response (Mental Refinement):\n "It looks like you forgot to include the context and the specific question you'd like me to answer. Please provide both, and I’ll be happy to help!"\n\n5. Check Against Guidelines:\n - Direct, clear, addresses the missing information.\n - No extra fluff.\n - Ready.✅\n'

accuracy: 0/1

You're welcome man, the more powerful C++ re-written OpenAI-compatible server with versatile multi-GPU support will be ready in the following two weeks, feel free to try it out then :)

[DemonODG]

I disabled the RTX 4080 laptop in the devices and ran the test:
DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256 [init] spawning daemon: ../dflash/build/test_dflash [cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0 [test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5) ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB [loader] eos_id=248046 eos_chat_id=-1 [target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K) [draft] loaded [daemon] ready [park] draft released [case 0] src=131068 keep=0.05 [park] target released [compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ... [drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936 [compress] drafter ready [qwen3-0.6b-fp] layer 1/28 done (A_setup=0.000s A_alloc=0.001s A_compute=1.962s FP=0.127s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s) [qwen3-0.6b-fp] layer 28/28 done (A_setup=0.005s A_alloc=0.003s A_compute=4.845s FP=4.220s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s) [qwen3-0.6b-fp] forward 21.29s (S=131068, A_setup=0.00s A_alloc=0.00s A_compute=4.84s FP=4.22s B_warm=0.00s B_setup=0.00s B_alloc=0.00s B_copy_in=0.00s B_norm=0.00s B_compute=0.00s B_copy_out=0.00s) tail-score 4.32s total 25.61s [drafter] forward+score in 26.20s S=131068 [drafter] score_and_compress total 26.20s S=131068 kept=6524 (204/4096 chunks, forced=37) [compress] 131068 -> 6524 tokens [case 0] compressed=6524 ratio=20.1x score_s=32.1 [loader] eos_id=248046 eos_chat_id=-1 [unpark] target restored [drafter] freed [unpark] draft restored [spec-decode] tokens=193 time=5.401 s speed=35.73 tok/s steps=25 accepted=170/400 (42.5%) avg_commit=7.72 ok N=6547 gen=193 prefill_s=7.887 decode_s=5.401 decode_tok_s=35.7 stream_fd=5 [park] draft released [case 0] raw out_ids (193): [760, 1156, 6587, 310, 1423, 279, 328, 14703, 10642, 2715, 1413, 69, 371, 7891, 1324, 1, 494, 279, 3766, 1414] [case 0] out_with_special: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.<|im_end|>' [case 0] gen_s=13.4 ttft=45.5 ok=True ans=4025016 [case 0] out: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.'
accuracy: 1/1
It works on one card! Interestingly, trying to run it on one card with the environment variable CUDA_VISIBLE_DEVICES=1 (RTX 3090) returns an error (see above).
I'll keep investigating. Thanks for your help!

My guess is when CUDA_VISIBLE_DEVICES=1 is set, that physical GPU becomes logical GPU 0 inside the process, so DFLASH_TARGET_GPU and DFLASH_DRAFT_GPU should both be 0. Also make sure --draft-spec points to the DFlash GGUF draft model, not the old safetensors draft, otherwise it may hit the earlier shape mismatch again.

You can try this config:

CUDA_VISIBLE_DEVICES=1 DFLASH_TARGET_GPU=0 DFLASH_DRAFT_GPU=0 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/niah_128k.jsonl --keep-ratio 0.05 --n-gen 256

I tested this option too. Now it works without errors!
And the result is also different. Now it finds the target number!
In my previous post, the script's output included errors allocating the buffer on the card: 0. Therefore (I assume) the accuracy is 0/1.
Now, there are no errors at all, and the result is:

CUDA_VISIBLE_DEVICES=1 DFLASH_TARGET_GPU=0 DFL
ASH_DRAFT_GPU=0 DFLASH_FP_USE_BSA=1 DFLASH_FP_ALPHA=0.85 python tests/bench_niah_cpp.py --bin ../dflash/build/test_
dflash --target ../../ModelsIA/Qwen/Qwen3.6-27B-Q4_K_M.gguf --draft-spec ../../ModelsIA/Qwen/draft/dflash-draft-3.6
-q8_0.gguf --drafter-arch qwen3-0.6b --drafter-gguf ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf --cases /tmp/ni
ah_128k.jsonl --keep-ratio 0.05 --n-gen 256
[init] spawning daemon: ../dflash/build/test_dflash
[test_dflash] arch=qwen35 daemon -> dispatching to run_qwen35_daemon (max_ctx=16384 stream_fd=5)
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24575 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24575 MiB
[cfg] seq_verify=0 fast_rollback=1 ddtree=1 budget=16 temp=1.00 chain_seed=1 fa_window=0 draft_swa=0 draft_ctx_max=4096 draft_feature_mirror=0 peer_access=0 target_gpu=0 draft_gpu=0
[loader] eos_id=248046 eos_chat_id=-1
[target] target loaded: layers [0,64) output=1, 850 tensors on GPU 14.99 GiB, tok_embd 682 MiB CPU-only (q4_K)
[draft] loaded
[daemon] ready
[park] draft released
[case 0] src=131068 keep=0.05
[park] target released
[compress] loading drafter from ../../ModelsIA/Qwen/drafter/Qwen3-0.6B-BF16.gguf ...
[drafter] loaded qwen3-0.6b BF16: n_layer=28 n_head=16 n_kv=8 n_embd=1024 n_ff=3072 head_dim=128 vocab=151936
[compress] drafter ready
[qwen3-0.6b-fp] layer 1/28 done (A_setup=0.000s A_alloc=0.001s A_compute=2.139s FP=0.146s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s)
[qwen3-0.6b-fp] layer 28/28 done (A_setup=0.019s A_alloc=0.003s A_compute=5.081s FP=4.230s B_warm=0.000s B_setup=0.000s B_alloc=0.000s B_copy_in=0.000s B_norm=0.000s B_compute=0.000s B_copy_out=0.000s)
[qwen3-0.6b-fp] forward 22.53s (S=131068, A_setup=0.02s A_alloc=0.00s A_compute=5.08s FP=4.23s B_warm=0.00s B_setup=0.00s B_alloc=0.00s B_copy_in=0.00s B_norm=0.00s B_compute=0.00s B_copy_out=0.00s) tail-score 4.34s total 26.87s
[drafter] forward+score in 27.47s S=131068
[drafter] score_and_compress total 27.47s S=131068 kept=6524 (204/4096 chunks, forced=37)
[compress] 131068 -> 6524 tokens
[case 0] compressed=6524 ratio=20.1x score_s=32.1
[loader] eos_id=248046 eos_chat_id=-1
[unpark] target restored
[drafter] freed
[unpark] draft restored
[spec-decode] tokens=193 time=5.794 s speed=33.31 tok/s steps=25 accepted=170/400 (42.5%) avg_commit=7.72
ok N=6547 gen=193 prefill_s=8.730 decode_s=5.794 decode_tok_s=33.3 stream_fd=5
[park] draft released
[case 0] raw out_ids (193): [760, 1156, 6587, 310, 1423, 279, 328, 14703, 10642, 2715, 1413, 69, 371, 7891, 1324, 1, 494, 279, 3766, 1414]
[case 0] out_with_special: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.<|im_end|>'
[case 0] gen_s=14.7 ttft=46.8 ok=True ans=4025016
[case 0] out: 'The user wants to find the "special magic qahftrxc number" from the provided text.\nI will scan the text for the phrase "special magic qahftrxc number".\nLooking through the text, I see: "The grass is green. The sky is blue. The sun is yellow. Here we go. There and bac The special magic qahftrxc number is: 4025016. k again."\nSo the number is 4025016.\nThe question asks to answer in one short sentence.\nI will formulate the answer: The special magic qahftrxc number is 4025016.\nCheck constraints: "Answer in one short sentence."\nMatches constraint.\nDone. \nOutput matches exactly.✅\n\n\nThe special magic qahftrxc number is 4025016.'

accuracy: 1/1