server: preemption notices, asynchronous parks and exact concurrency (pin branch on b10869) - #210

Open

danielhanchen wants to merge 23 commits into

upstream/b10871-inkling-25731from

pin/server-side-preemption-b10869

Commits on Sep 9, 2026

server: preemption notices, asynchronous parks and exact concurrency, on b10869
danielhanchen
committed
afd3248
Compose the server-side preemption pin with ggml-org#25731 (inkling) so both merge onto b10871
danielhanchen
committed
2834f07
Drop the fork's own workflow and script files from the pin tree; the nightly composes them from master
danielhanchen
committed
62cf502
Pin that an exact-mode cross-sequence copy leaves both sequences as they were
danielhanchen
committed
1cfb42b
Condense the allocation granularity comment to two lines
danielhanchen
committed
17a0f3b

Commits on Sep 10, 2026

Park only when asked: --preempt-ram defaults to 0, so a server without the flag behaves as upstream
danielhanchen
committed
1d55489
Read src[5] as a page table only for the op that has one
danielhanchen
committed
0b9caab
Normalize sequence planes in every batch-invariant mode, not only under exact concurrency
danielhanchen
committed
15bcc6c
Fix the CI legs that broke on the last rebase: a portable setenv in test-exact-pages, the OpenVINO support probe's struct return, and park assertions that match the park log lines rather than the s…
danielhanchen
committed
3e7d953
Build test-exact-buft only where llama-impl.h links, queue the started-slot follower behind a slot that is seen busy, and give the notify resident time to decode on a loaded runner
danielhanchen
committed
2176de1
Condense the exact-mode weight check comment so it is not hard-wrapped
danielhanchen
committed
d0202ea
tests : give the rotation test time to finish and stop the deliberate disconnect from raising
danielhanchen
committed
a05a318
server : keep the tokens a recompute restore replays out of the request's prompt stats
danielhanchen
committed
f8b7514
server : put the preemption record inside the completed response of a streamed /v1/responses
danielhanchen
committed
0beed51
tests : condense the exact-pages preamble so it is not hard-wrapped
danielhanchen
committed
d040ae7
llama : refuse an asynchronous state transfer for a model whose states move between rows
danielhanchen
committed
c5a89f7
tests : drive the started-slot trim from the prompt lengths instead of the host's speed
danielhanchen
committed
7eb9cb7
tests : make the failed-save park a matter of lengths rather than of two requests overlapping
danielhanchen
committed
d8daa8f
server : reset the prompt counters when a recompute park restarts a prefill
danielhanchen
committed
5b66acb
server : number the preempt notices of every task of a batched request, index 0 included
danielhanchen
committed
b4433f5
tests : let the server log settle before asserting on its text
danielhanchen
committed
41abdfb
llama : keep the exact-mode host buffer comment to whole sentences
danielhanchen
committed
c5dc0f9
tests : make the two generations that do not fit overlap by their lengths, not by the client's speed
danielhanchen
committed
ae19539

Navigation Menu

Uh oh!

server: preemption notices, asynchronous parks and exact concurrency (pin branch on b10869) - #210

server: preemption notices, asynchronous parks and exact concurrency (pin branch on b10869)#210
danielhanchen wants to merge 23 commits into
upstream/b10871-inkling-25731from
pin/server-side-preemption-b10869

Commits on Sep 9, 2026

server: preemption notices, asynchronous parks and exact concurrency, on b10869

Compose the server-side preemption pin with ggml-org#25731 (inkling) so both merge onto b10871

Drop the fork's own workflow and script files from the pin tree; the nightly composes them from master

Pin that an exact-mode cross-sequence copy leaves both sequences as they were

Condense the allocation granularity comment to two lines

Commits on Sep 10, 2026

Park only when asked: --preempt-ram defaults to 0, so a server without the flag behaves as upstream

Read src[5] as a page table only for the op that has one

Normalize sequence planes in every batch-invariant mode, not only under exact concurrency

Fix the CI legs that broke on the last rebase: a portable setenv in test-exact-pages, the OpenVINO support probe's struct return, and park assertions that match the park log lines rather than the s…

Build test-exact-buft only where llama-impl.h links, queue the started-slot follower behind a slot that is seen busy, and give the notify resident time to decode on a loaded runner

Condense the exact-mode weight check comment so it is not hard-wrapped

tests : give the rotation test time to finish and stop the deliberate disconnect from raising

server : keep the tokens a recompute restore replays out of the request's prompt stats

server : put the preemption record inside the completed response of a streamed /v1/responses

tests : condense the exact-pages preamble so it is not hard-wrapped

llama : refuse an asynchronous state transfer for a model whose states move between rows

tests : drive the started-slot trim from the prompt lengths instead of the host's speed

tests : make the failed-save park a matter of lengths rather than of two requests overlapping

server : reset the prompt counters when a recompute park restarts a prefill

server : number the preempt notices of every task of a batched request, index 0 included

tests : let the server log settle before asserting on its text

llama : keep the exact-mode host buffer comment to whole sentences

tests : make the two generations that do not fit overlap by their lengths, not by the client's speed