Feature Request: Show some love for Windows 11?

[endyjasmi]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Any chance of making the --ngram-on-disk works on windows 11? Currently it skip it. I am currently running Vulkan (compilation made using visual studio community.

Motivation

I am using my computer as my main workstation. If I can squeeze more ram out of it for my other work. The would be great.

Possible Implementation

No response

[auf1r2]

not sure about how this fork is supported on Windows, but you can try --no-op-offload --override-tensor "per_layer_token_embd=CPU" --lazy-mode on --load-mode mmap instead of --ngram-on-disk. (full command here: #44)

Also you can't use any random GGUF to make this work, model should be correctly split, so that ngram part is offloadable.

I have tested Unsloth's quants Q6, Q5 and Q4 - they all support ngram offload: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF, while ggml's quants (for example) - don't.

[endyjasmi]

Thanks for the reply. I am using Unsloth's quant (both Q4_K_XL and IQ4_XS) and also the quant from AtomicChat on a AMD Ryzen AI Max+ 395 with 128GB unified memory.

The thing's with load-mode mmap and lazy-mode on is that, eventually, Eventually it will load everything to RAM without swapping unused memory.

As this is my main computer where I do my day to day development, I encountered out of memory issue multiple time a day while doing development.

Reading the README.md where it mention --ngram-on-disk -> Keeps a model's n-gram hash-embedding table (28.8 GB on Qwen3.8-Flash-Next) off the memory budget entirely, reading only the rows each batch actually gathers. I assume this means, only required ple is read when needed, and it won't be cached on memory.

Hence, I wonder if there is a reason why this --ngram-on-disk feature is disabled for windows 11.

Also, here is the parameter I am using (I am currently using unsloth's fork)

[unsloth/Qwen3.8-Flash-Next-GGUF:Q4_K_XL]
spec-draft-model = C:\Users\EndyJasmi\.cache\huggingface\hub\models--unsloth--Qwen3.8-Flash-Next-GGUF\snapshots\38bb39ee97821de2c9009abb7e93950eec396e66\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf
override-tensor = per_layer_token_embd.weight=CPU
lazy-mode = on

;; sampling parameters ;;
temperature = 1.0
top-k = 20
top-p = 0.95
min-p = 0.0
repeat-penalty = 1.0
presence-penalty = 0.0

;; load parameters ;;
n-gpu-layers = all
ctx-size = 262144
;; max len (32768/81920)
batch-size = 8192
ubatch-size = 1024

;; slot parameters ;;
kv-unified = false
parallel = 1
fit = off
warmup = false

;; kv parameters ;;
cache-type-k = q8_0
cache-type-v = q8_0

;; speculative parameters ;;
n-gpu-layers-draft = all
spec-draft-type-k = q8_0
spec-draft-type-v = q8_0

spec-type = draft-mtp,ngram-mod
spec-draft-n-max = 5
spec-draft-n-min = 0.5

spec-ngram-mod-n-min = 48
spec-ngram-mod-n-max = 64
spec-ngram-mod-n-match = 24

;; vision parameters ;;
image-min-tokens = 1024
image-max-tokens = 4096

;; reasoning parameters ;;
reasoning = auto
reasoning-budget = -1
reasoning-preserve = true

[endyjasmi]

With the help of AI's I somehow ported the ngram-on-disk to Windows. llama-server do runs with model has unused tensor per_layer_token_embd.weight (size = 28800138240 bytes) -- ignoring.

It won't runs with draft mtp model loaded though. I am wondering if this fork have a broken mtp support for Qwen3.8 Flash Next