Repository navigation
Conversation
…thout empty submits The scheduler copied each graph input from the user synchronously: it waited for the split's backend and then wrote the input with a blocking set_tensor. On Vulkan without host-visible VRAM, each write was its own queue submission, fence wait and fence reset, six per decoded token with Qwen3.6 27B. A new optional backend function, set_tensor_async_staged, queues a write of host data after the backend's pending work and copies the data before it returns, so the user may overwrite it at once, which is what the scheduler needs for inputs. The scheduler uses it when the backend has it and falls back to the synchronous copy otherwise. Vulkan implements it with host staging memory owned by the backend context and a copy in the compute stream, freed once the backend has synchronized. ggml_vk_synchronize no longer adds an empty queue submission to signal its fence: the fence (and the wait for transfer-queue work) goes with the last compute submission, and when only transfer-queue work is outstanding the host waits for its timeline semaphore. A host-to-device copy goes on the transfer queue only when that pays off: its own submission and the compute queue's semaphore wait must cost less than the copy it lets overlap. The crossover, that overhead times the host-to-device bandwidth, is measured once per device on first use (the overhead with the host clock, the bandwidth with GPU timestamps around a staged copy) and logged; GGML_VK_COMPUTE_STREAM_COPY_MAX sets it instead. Smaller copies go in the compute stream. test-backend-sched-inputs checks that inputs are taken when the graph is queued and ordered against the work already queued on the device, and that tensors computed on the CPU reach the device on both sides of the copy size limit (--measured uses the measured limit).
Geramy
requested review from
a team,
JohannesGaessler,
ggerganov,
marty1885 and
wine99
as code owners
October 8, 2026 18:06
github-actions
Bot
added
testing
Contributor
Author
|
@pwilkin my vulkan backend stream patch, this worked really well on my eGPU with the R9700 on Mac and increased my performance a lot. I believe it will also increase performance on a normal Server or Desktop with a R9700, latency is latency and for AI it costs period. |
Upstream moved the scheduler's per-input copy into ggml_backend_sched_copy_input; the staged user-input write now lives there.
Contributor
Contributor
Author
|
Benchmarks, Vulkan backend,
On Linux with ReBAR the input writes are already plain memcpys into VRAM, and B is about 0.6% slower on decode, consistent in both rounds.
|
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
To produce one token, llama.cpp runs a compute graph. Some of its tensors are inputs
that the CPU fills in fresh for every token. Before this change, the ggml scheduler copied
each of those inputs to the GPU one at a time, and each copy blocked: wait until the GPU
backend is idle, then write the input and wait again until the write has landed
(
ggml-backend.cpp: ggml_backend_sched_compute_splits, theGGML_TENSOR_FLAG_INPUTbranch). When the CPU can write straight into GPU memory that write is just a
memcpy.When it can't, as on this Mac setup where RADV reports no CPU-visible VRAM, the Vulkan
backend's write (
ggml-vulkan-buffers.cpp: ggml_vk_buffer_write_2d) records a small copycommand, submits it to the GPU on its own, waits on a fence until the GPU finishes it, then
resets the fence. With Qwen3.6 27B that happened six times per generated token. Each round
trip costs far more than the few bytes it moves, so decode spends time waiting instead of
computing.
Solution
The patch adds an optional backend function, set_tensor_async_staged. It copies the input into staging memory that the backend owns and queues the GPU copy behind work already submitted. It returns right away, and the caller can reuse its buffer immediately. The scheduler uses this function when the backend provides it and keeps the old blocking copy otherwise. In Vulkan, these copies run in the compute stream, and the staging memory is reused between tokens. Synchronization no longer sends a separate empty submission just to signal its fence: the fence rides on the last compute submission in the common decode case. For copies of tensors the CPU computed, a measured size threshold decides between the separate transfer queue and the compute stream. It is measured once per device and can be overridden with GGML_VK_COMPUTE_STREAM_COPY_MAX. Only copies large enough to pay for the extra submission and semaphore wait go on the transfer queue; smaller copies stay in the compute stream.