server: --pipeline-groups, run the slots over several contexts of one model - #187

Open

danielhanchen wants to merge 18 commits into

feature/pipeline-groups

Commits on Sep 5, 2026

server: add --pipeline-groups to run the slots over several contexts
danielhanchen
committed
c238f82
server: harden the pipeline group decode loop
danielhanchen
committed
f96bb35
rpc: serialise the client connection and track the stored graph per connection
danielhanchen
committed
2bd1359
server: sample the pipeline groups in parallel and give each group its own engine lock
danielhanchen
committed
8de4320

Commits on Sep 6, 2026

server: give each pipeline group its own speculative decoding state
danielhanchen
committed
a1dd7c5

Commits on Sep 8, 2026

server: trim comments in the pipeline-groups changes
danielhanchen
committed
d98d90d

Commits on Sep 9, 2026