memory issues

[phazei]

I have a RTX3090, and I can run Qwen3 8B with no issues.

But however this is loading, seems inefficient:

(mufun) PS D:\AITools\MuFun\demo\mufun_acestep> python .\gr_app.py
ModelManager initialized. Model will be loaded on first request.
* Running on local URL:  http://127.0.0.1:7860

Received request for audio file: C:\Users\HomeStar\AppData\Local\Temp\gradio\e89fc7304bd55c990b35b7a1a7f204903815404605ee2afbf1b00c88a7425632\ace_output.wav
Loading model into VRAM...
The 8-bit optimizer is not available on your device, only available on CUDA for now.
Some weights of WhisperForConditionalGeneration were not initialized from the model checkpoint at openai/whisper-large-v3 and are newly initialized: ['proj_out.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Loading checkpoint shards:   0%|                                                                 | 0/4 [03:38<?, ?it/s]
Error loading model: [enforce fail at alloc_cpu.cpp:121] data. DefaultCPUAllocator: not enough memory: you tried to allocate 3221225472 bytes.
Scheduling model unload in 300 seconds.

I have 128gb of system ram. 190gb including pagefile. When it tried to load my system ram usage went from 20gb to 160gb.

[laitselec]

it may relate to the bitsandbytes quantization library, you can try not using it (line 40 in gr_app.py)

self.model = AutoModelForCausalLM.from_pretrained(
                    self.hf_path,
                    trust_remote_code=True,
                    torch_dtype="bfloat16",
                    device_map="auto",
              #      quantization_config=quantization_config  # comment this line
                )

if this still have problem you can further ask

FYI, the web demo(http://47.121.209.64/mufun_demo_acestep) is run on a machine with a 4070Ti(12g) and 64g ram, the system is WSL in Win11

some strange problems may occur in Windows when running ML applications, as some packages support linux best, I think WSL is a good choice

[deepblue96]

confirm the removal of quantization works, for me at least it also solve the problem of llm getting stuck on perma output loop.
small note the vram rise from 12gb to 21-22gb not a big deal for me as I have a 3090 but won't fit on low vram cards