Introduce v1 runtime compatibility, modern serving profile, and MP startup path

[rickisba]

Purpose

Establish a long-lived v1 compatibility line without breaking the current CacheRoute deployment based on vLLM 0.13.x and LMCache 0.3.11.

The codebase is not forked into two independent implementations. Version-dependent behavior is isolated behind runtime profiles and focused adapters so that migration work for vLLM 0.25.1 and LMCache 0.5.2 can continue while the legacy path remains runnable.

This PR consolidates the modern environment work previously staged in #150. The legacy image, root dependency profile, and current main behavior remain unchanged.

Runtime profiles

CACHEROUTE_RUNTIME_PROFILE selects the behavior:

• legacy: preserve the historical vllm@* Redis-key namespace and old behavior.
• v1: use the modern model-scoped LMCache Redis-key layout and MP startup contract.
• auto (default outside the dedicated v1 image): recognize either key layout and keep old API/CLI callers compatible.

Compatibility logic is centralized in core/runtime_compat.py. Future version checks should be added there or in profile-specific adapters rather than scattered through Scheduler, Proxy, Instance, and KDN code.

Isolated v1 serving environment

Add env/docker/cu130/ as an independent profile:

• CUDA 13.0.0 on Ubuntu 22.04;
• Python 3.12 in /opt/venv;
• PyTorch 2.11.0+cu130;
• torchvision / torchaudio 0.26.0 / 2.11.0;
• vLLM 0.25.1;
• LMCache 0.5.2;
• separately constrained CacheRoute application/development dependencies.

Files:

• env/docker/cu130/Dockerfile
• env/docker/cu130/constraints.txt
• env/docker/cu130/requirements-dev.txt
• env/docker/cu130/README.md
• env/docker/cu130/scripts/start_lmcache_mp.sh
• env/docker/cu130/scripts/start_vllm_mp.sh

The image sets CACHEROUTE_RUNTIME_PROFILE=v1 by default. It installs FFmpeg for TorchCodec and keeps Rust/Cargo and Tkinter for existing resource-agent/dashboard workflows.

The profile is additive and does not modify:

• env/docker/Dockerfile;
• root requirements.txt;
• pyproject.toml;
• legacy Scheduler, Proxy, Instance, KDN, routing, queueing, or storage behavior.

Versioned startup contract

The old and new serving commands are intentionally treated as different runtime interfaces.

Legacy:

• YAML selected through LMCACHE_CONFIG_FILE;
• historical LMCache offloading arguments;
• current legacy documentation remains valid for vLLM 0.13.x + LMCache 0.3.11.

v1:

• standalone lmcache server;
• RESP L2 adapter targeting Redis;
• vllm serve with LMCacheMPConnector in --kv-transfer-config;
• legacy LMCACHE_CONFIG_FILE, remote_url, and --kv-offloading-backend lmcache are not used.

The validated startup order is:

Redis RESP L2 -> LMCache MP :5555 -> vLLM :8000 -> CacheRoute components

Existing containers do not need to rebuild the Docker image. After switching to this branch, they can run:

export CACHEROUTE_RUNTIME_PROFILE=v1
bash env/docker/cu130/scripts/start_lmcache_mp.sh
# In another terminal after LMCache is listening:
bash env/docker/cu130/scripts/start_vllm_mp.sh

The full equivalent commands and environment-variable overrides are documented in env/docker/cu130/README.md. Both scripts passed bash -n syntax validation.

KDN compatibility changes

• Remove the operational requirement to pass --match for normal KV construction.
• Treat the historical implicit vllm@* value as a compatibility sentinel in auto and v1 modes.
• Support both legacy and modern model/path-scoped LMCache key layouts.
• Replace fixed-round stabilization with asynchronous-write-aware polling:
  • wait for the first matching key;
  • reset the quiet timer whenever the key set changes;
  • finish after a configurable quiet period;
  • keep 30 seconds as a maximum timeout, not a fixed registration delay.
• Fail explicitly when zero KV keys are captured.
• Remove partial KV directories on failure and never mark an empty build as kv_ready.
• Record runtime profile, detected key format, scan pattern, and capture timing in run_meta.json.

Backward compatibility

Strict legacy behavior remains available with:

export CACHEROUTE_RUNTIME_PROFILE=legacy

Modern deployments use the dedicated image or explicitly set:

export CACHEROUTE_RUNTIME_PROFILE=v1

Generic deployments may retain:

export CACHEROUTE_RUNTIME_PROFILE=auto

Existing CLI/API callers that still supply the old default vllm@* remain compatible.

Homepage entry plan

The repository homepage should present a version selector before the current Requirements and full-deployment sections, rather than mixing modern commands into the existing legacy walkthrough.

Proposed homepage block:

Runtime path	Status	Serving stack	Entry point
Legacy stable	Existing supported path	CUDA 12.8 / vLLM 0.13.x / LMCache 0.3.11	env/README.md and the existing full single-machine walkthrough
Modern v1	Migration/development path	CUDA 13.0 / vLLM 0.25.1 / LMCache 0.5.2	env/docker/cu130/README.md
Compatibility design	Architecture reference	legacy, v1, and auto profiles	docs/runtime_compatibility_v1.md

The homepage should also include a visible warning that startup commands cannot be mixed:

• legacy uses the YAML/offloading path;
• v1 uses lmcache server + LMCacheMPConnector;
• users with an already-built modern container can switch branches, export CACHEROUTE_RUNTIME_PROFILE=v1, and use the repository scripts without rebuilding.

The detailed legacy walkthrough should remain intact and be labeled Legacy stable full deployment. The modern homepage entry should stay concise and link to the dedicated v1 guide instead of duplicating its full commands.

Validation completed

Runtime compatibility tests:

• profile aliases;
• legacy behavior;
• modern LMCache key detection;
• mixed-generation filtering;
• strict v1 reinterpretation of the historical CLI default.

KDN migration validation:

• LMCache 0.5.2 asynchronous Redis writes were captured successfully;
• 104 keys were persisted locally;
• dump/restore preserved 1,090,519,040 bytes exactly;
• key/value round-trip verification reported no missing, mismatched, or extra entries.

Modern environment validation inherited from the tested #150 profile:

• Python 3.12.13;
• PyTorch 2.11.0+cu130 and CUDA runtime 13.0;
• vLLM 0.25.1;
• LMCache 0.5.2;
• Transformers 5.12.1;
• Redis client 8.0.1;
• successful pip check and serving-stack imports;
• successful startup of the existing CacheRoute Scheduler/KDN/Proxy/Instance workflow.

Validation boundary

Byte-for-byte Redis dump/restore proves KDN storage integrity, but does not yet prove that an injected block is consumed as an LMCache MP hit. End-to-end validation must compare LMCache hit-token and remote-read metrics before this PR is considered complete.

Planned follow-up work

1. Add Instance-side LMCache observability and metric collection.
2. Validate injected-cache hits using LMCache metrics rather than latency inference.
3. Verify prompt/token-prefix and key namespace compatibility across KDN and Instance.
4. Add a dual-profile integration matrix for legacy and v1 environments.
5. Implement the planned root README runtime-selector block without replacing the existing legacy walkthrough.
6. Continue isolating subsequent vLLM/LMCache API differences behind compatibility adapters.

Review status

This PR contains the isolated modern environment, the validated MP startup interface, and the first KDN runtime migration slice. Full LMCache MP cache-consumption validation is still pending.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdf27f491b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Allow the quiet period after the first key arrives

When asynchronous Redis writes produce their first key near first_key_timeout_s—for example at 29 seconds with the default 1.5-second quiet period—the same deadline also terminates stabilization at 30 seconds, so the build raises and deletes otherwise valid output before the key set can become quiet. The first-key deadline should stop applying once a key appears, with subsequent polling governed by the quiet period.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Honor an explicitly requested wildcard match

When an operator explicitly passes --match '*' for debugging or a custom backend, scan_match is still "*", so this branch proceeds to profile classification and discards unrecognized keys (and legacy keys under v1) rather than returning everything matched by Redis. Check whether requested_match was explicitly supplied before applying automatic-discovery filtering so the existing CLI/debug workflow remains authoritative.

AGENTS.md reference: AGENTS.md:L255-L259

Useful? React with 👍 / 👎.