[pull] master from ray-project:master - #1190
Merged
Merged
Conversation
…CPU KV caches (#65063) ## Description This PR adds native vLLM KV-cache offloading support to ray serve LLM KV-aware routing. KV-aware deployments use `OffloadingConnector` (enabled via `VLLM_USE_SIMPLE_KV_OFFLOAD=0` and `kv_connector_extra_config["self_describing_kv_events"] = True`) to publish CPU-tier KV events, letting the selection service account for CPU-resident prefixes and route reloads to the appropriate replica. ### KV event flow `OffloadingConnector` → `BlockStored` / `BlockRemoved` (medium="CPU") ([reference](https://github.com/vllm-project/vllm/blob/9d1aa4dda85a58c74b23ff79aa4a20594ee6f359/vllm/distributed/kv_events.py#L61)) → vLLM `KVEventBatch` over ZMQ PUB → Dynamo listener decodes and normalizes events → CPU events indexed as HostPinned tier → selection service scores GPU + CPU prefix overlap per replica ### API shape ``` from ray.serve.config import RequestRouterConfig from ray.serve.llm import LLMConfig from ray.serve.llm.request_router import KVAwareRouter llm_config = LLMConfig( model_loading_config={ "model_id": "my-model", "model_source": "my-model", }, deployment_config={ "request_router_config": RequestRouterConfig( request_router_class=KVAwareRouter, ), }, engine_kwargs={ "enable_prefix_caching": True, "kv_offloading_backend": "native", "kv_offloading_size": 8, # GiB of CPU cache per replica }, ) ``` ## Validation - **CI tests**: Added unit and release coverage for CPU-tier event ingestion, scoring, and end-to-end offload/reload behavior. Non-KV-aware routers retain user-specified KV offload configuration. - Observed KV cache offloading / reloading, and I'll add grafana dashboards for KV cache offloading observability in a follow-up. - Workload characteristic: A two-replica Ray Serve LLM workload on L4s using Qwen3-0.6B with native KV offload, capped GPU KV cache, sticky session routing, long repeated prompts, forced prefix-cache resets, and same-session follow-ups. <img width="866" height="310" alt="Screenshot 2026-07-28 at 11 33 43 PM" src="https://github.com/user-attachments/assets/65134e1a-e0fe-4877-b0cc-508458cbce3f" /> ## Related issues > Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )