We recently added support for dynamic batch dimension in causal inference language models in https://github.com/apple/coreai-models/pull/334. This issue will track its integration in the runtime for * The `llm-runner` * The `llm-benchmark` * The `llm-server` for * Basic inference * More intelligent batch policies
Activity
stikves commentedon Oct 21, 2026
The first PR is for allowing generation of batched inputs, and attention masks
#335
Adds 4 low level components:
BatchedInputBuilder, which processes the new input layoutsRegionCopyableprotocol for in-place mutation of KV-cachesBatchedDecodeStepBatchedSamplerHigh level API is unchanged, except:
CoreAILanguageModelnow auto detects, and accepts batching-capable exportsInputLayout.attnMaskNameis added to help with this detection