Server wedge: generation stalls mid-decode; /health OK but /slots hangs; needs SIGKILL

[victorarthur]

Server wedge: generation stalls mid-decode; /health OK but /slots hangs; process must be SIGKILLed

Summary

Under long-running load the server occasionally stops generating mid-decode:
print_timing lines cease, GPU utilization drops to 0%, the process stays alive,
/health still returns {"status":"ok"}, but /slots hangs (no response)
and the client stream goes idle until the client-side timeout. In one instance
the process also ignored SIGTERM and required kill -9.

Observed 3 times in 2 days across two builds and two different models, with
and without MTP, so it does not appear specific to a model, speculative
decoding, or a recent regression.

Environment

• llama.cpp CUDA backend; observed on build 452 (9558fa4) and build 561 (ee4c505)
• 2× RTX 3090 (PCIe/PHB, no NVLink), driver 570.133.07, CUDA 12.8, Linux
• No Xid errors in dmesg in any incident (not a GPU fault)

Incidents

A) Qwen3.6-35B-A3B Q8_0 (MoE, no MTP), single slot, 32K ctx, long-context
eval task: generation stopped after ~286 tokens (timing lines ceased), client
timed out after 20 min, server log showed cancel task then the process kept
spinning (R state), ignored SIGTERM, needed SIGKILL. Immediately before the
stall: slot get_availabl: … selected slot by LCP similarity, f_sim_best = 0.999.

B) Qwen3.8-27B Q6_K (embedded MTP, --spec-type draft-mtp --spec-draft-n-max 3),
--parallel 2, two concurrent agent workloads: slot 1 decoding normally
(~40 tok/s, n_decoded = 8337), then all timing output stops; /health OK,
/slots hangs, GPU 0%. Restart required. Full journal excerpt (926 lines around
the stall) available on request.

C) Same symptom pattern on build 561 during a client 300 s stream-idle
timeout; restart healed it.

Server invocation (incident B)

llama-server -m Qwen3.8-27B-Q6_K.gguf -mm mmproj-F16.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --jinja -ngl 999 --ctx-size 458752 --parallel 2 \
  -ctk q8_0 -ctv q8_0 --cache-reuse 0 --no-context-shift \
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0

(--cache-reuse 0 --no-context-shift were added after incident A as a hunch;
incident B shows they do not prevent it.)

Notes

• /health responding while /slots hangs suggests the slots endpoint blocks on
  a mutex held by the stuck decode path — a liveness probe on /slots (or a
  watchdog around the decode loop) would at least make the state detectable.
• Happy to reproduce with added instrumentation and to share full journals.

[loviswaternakamoto]

Reproducing this on CPU / aarch64, with a hard receipt for the
"process ignores SIGTERM" case:

Environment

• llama.cpp tag b10991 (llama-server --version: 0.4.1-dev @930e2fa),
  CPU-only aarch64 (4 hardware threads usable), Ubuntu 24.04, kernel
  6.17.0-1020-oracle
• Model: Qwen3.5-4B (unsloth UD-Q4_K_XL MTP GGUF, draft heads in-file)
• Server: --spec-type draft-mtp --parallel 1 --cont-batching -fa on -ctk q8_0 -ctv q8_0 --ctx-size 16384 --threads 3

Symptom — identical signature to the OP

Under use, a slot wedges: the first requests answer normally (~1.6 s), then
every completion queues forever. /health keeps returning
{"status":"ok"} while /slots never answers — matching your mutex reading.
The trigger correlates with clients disconnecting mid-generation
(aborted streaming requests).

The SIGTERM receipt (systemd journal)

systemd[1]: Stopping <service>…
systemd[1]: <service>: State 'stop-sigterm' timed out. Killing.
systemd[1]: Killing process … (llama-server) with signal SIGKILL.
systemd[1]: <service>: Main process exited, code=killed, status=9/KILL

SIGTERM was ignored for the full 90 s default TimeoutStopSec; SIGKILL
required. So the wedge also blocks orderly shutdown, not just new requests.

MTP vs plain decoding, same binary + same model file

• Plain decoding (no --spec-type): a 12-case stability battery passes,
  including recovery from a deliberately aborted 400-token generation.
• With --spec-type draft-mtp: the wedge above, within minutes of
  interrupted clients.

Small sample, and given this issue reproduces without MTP on GPU, I read MTP
as an amplifier rather than the root cause — plausibly the abort path racing
draft/decode bookkeeping widens the window enough for a single-slot CPU
setup to hit it quickly.

Your /slots-not-/health liveness observation is the practical mitigation
on our side: a watchdog probing /slots with a hard timeout detects the
state immediately (healthy /slots answers in <1 ms here), which also
suggests a simple in-server detectability hook if a full fix takes a while.

[mab776]

Got this too, on completely different hardware :
Blackwell + CUDA. With the CPU/aarch64 report above that's three backends now, so it really doesn't look like a GPU thing.

My setup:

llama-server -m Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf \
  --mmproj mmproj-F16.gguf -ngl 99 -sm layer -ts 47,14 \
  -c 262144 -fa on --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1 --cache-ram 12288 \
  --spec-type ngram-mod --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 32 --jinja

Build 11012 (master 4bc272fd7 + #28770 + #28213), RTX PRO 5000 48GB + RTX 5070 Ti 16GB, driver 595.84, CUDA 13.2. Model is Qwen3.8-Flash-Next (qwen4exp, 512-expert MoE) with vision. Client is VS Code Copilot chat over SSE. That's a third spec method too (ngram-mod, where you had draft-mtp and none). To be clear I'm not using MTP, but I am still speculating, so this doesn't contradict the amplifier idea above, it adds to it.

It died mid-sentence during a long agentic session, it happened a few times, normally stop + resend message or stop + send "continue" will unfreeze it. This time I was annoyed so I've checked the logs. For this run, the server had been up 14h30 and served 87 tasks, only one never released. The whole log for that task is two lines:

868.50.896.345 I slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.988 (> 0.100 thold), f_keep = 0.986
868.50.898.090 I slot launch_slot_: id  0 | task 10962 | processing task, is_child = 0

then nothing at all. Same beginning as your incident : it wedges right after a high-similarity cache hit. Mine was 51162 of 51927 tokens served from cache, only 644 left to prefill.

Two things were different from your reports though.

/slots didn't hang for me. It answered fine and just showed the frozen state:

is_processing              = true
n_prompt_tokens            = 51927
n_prompt_tokens_processed  = 644     <- never moved
n_prompt_tokens_cache      = 51162

Polled it for ~5min and nothing moved.

And the process wasn't spinning. Your "incident A" was stuck in R and ignored SIGTERM. Mine was the opposite: all 42 threads in S, 0 CPU ticks over 6 seconds (/proc/<pid>/stat), GPU at 0%. Just blocked, not looping.

One more thing on the client side. When I looked there was no TCP connection left to the server, VS Code had already timed out and dropped it, but the slot was still is_processing. --timeout is 3600 by default so I guess the slot stays held for an hour, and with --parallel 1 that's the whole server blocked. I restarted instead of waiting so that part is a guess.

The stop + resend thing I mentioned at the start is probably the interesting part : stop is a real abort so the task gets cancelled and the next request goes through fine. That would explain why this is only annoying when you're sitting in front of the chat, and fatal when an agent is running on its own.

Happy to test a patch on Blackwell if that helps.