Repository navigation

Serve concurrent requests with a continuous-batching scheduler - #367

Merged

stikves merged 13 commits into

stikves:sukru/public-batched-stage4-scheduler

Oct 23, 2026

Merged

Serve concurrent requests with a continuous-batching scheduler#367
stikves merged 13 commits into
apple:mainfrom
stikves:sukru/public-batched-stage4-scheduler

Commits on Oct 22, 2026
- Add a continuous-batching scheduler
  stikves
  committed
  7e1c19b
- Serve concurrent requests through the batch scheduler in llm-server
  stikves
  committed
  b34cca6
- Score completions prompts in batches
  stikves
  committed
  bd06a98
- Clarify comments on batched serving
  stikves
  committed
  af1fa6d
- Simplify batched serving and cut logits copies
  stikves
  committed
  b005497
- Move batched scoring into BatchScheduler.Scorer
  stikves
  committed
  d161a4a
- Merge branch 'main' into sukru/public-batched-stage4-scheduler
  stikves
  authored
  65aaf7a
- Merge branch 'main' into sukru/public-batched-stage4-scheduler
  stikves
  authored
  2ad4ea9
Commits on Oct 23, 2026
- Rename llm-server --concurrency to --concurrent-requests
  stikves
  committed
  5daacaa
- Free a batch row when its request is cancelled or its stream is dropped
  stikves
  committed
  b4c8d2d
- Pad scoring passes to a few fixed shapes
  stikves
  committed
  acb4c05
- Include the batch size in llm-server's system_fingerprint
  stikves
  committed
  4f0f89d
- Merge branch 'main' into sukru/public-batched-stage4-scheduler
  stikves
  authored
  8ee0e2f