-
Notifications
You must be signed in to change notification settings - Fork 4
env: add isolated CUDA 13 / vLLM 0.25.1 / LMCache 0.5.2 profile #150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
+340
−0
Closed
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
6f95a49
env: add CUDA 13 serving constraints
rickisba dc9ee16
env: add CUDA 13 application dependency profile
rickisba 843e89e
env: add isolated CUDA 13 development image
rickisba 66a9a26
docs: document additive CUDA 13 environment profile
rickisba File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,119 @@ | ||
| ARG CUDA_VERSION=13.0.0 | ||
| ARG PYTHON_VERSION=3.12 | ||
| ARG RUST_TOOLCHAIN=stable | ||
|
|
||
| FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04 | ||
|
|
||
| SHELL ["/bin/bash", "-o", "pipefail", "-c"] | ||
|
|
||
| ARG PYTHON_VERSION | ||
| ARG RUST_TOOLCHAIN | ||
|
|
||
| ENV DEBIAN_FRONTEND=noninteractive | ||
| ENV RUSTUP_HOME=/opt/rustup | ||
| ENV CARGO_HOME=/opt/cargo | ||
| ENV PIP_DISABLE_PIP_VERSION_CHECK=1 | ||
| ENV PYTHONDONTWRITEBYTECODE=1 | ||
| ENV PYTHONUNBUFFERED=1 | ||
| ENV PYTHONNOUSERSITE=1 | ||
| ENV VIRTUAL_ENV=/opt/venv | ||
| ENV PATH="/opt/venv/bin:/opt/cargo/bin:${PATH}" | ||
|
|
||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| software-properties-common \ | ||
| ca-certificates \ | ||
| curl \ | ||
| git \ | ||
| git-lfs \ | ||
| wget \ | ||
| vim \ | ||
| nano \ | ||
| less \ | ||
| tmux \ | ||
| unzip \ | ||
| build-essential \ | ||
| pkg-config \ | ||
| cmake \ | ||
| ninja-build \ | ||
| libssl-dev \ | ||
| libnuma-dev \ | ||
| libibverbs-dev \ | ||
| libcurl4-openssl-dev \ | ||
| libhiredis-dev \ | ||
| libgl1 \ | ||
| ffmpeg \ | ||
| net-tools \ | ||
| iproute2 \ | ||
| iputils-ping \ | ||
| iperf \ | ||
| htop \ | ||
| jq \ | ||
| lsof \ | ||
| openssh-client && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # Ubuntu 22.04 does not provide Python 3.12 as its default interpreter. | ||
| RUN add-apt-repository ppa:deadsnakes/ppa && \ | ||
| apt-get update && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| "python${PYTHON_VERSION}" \ | ||
| "python${PYTHON_VERSION}-dev" \ | ||
| "python${PYTHON_VERSION}-venv" \ | ||
| "python${PYTHON_VERSION}-tk" && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
|
|
||
| RUN update-alternatives --install \ | ||
| /usr/bin/python3 \ | ||
| python3 \ | ||
| "/usr/bin/python${PYTHON_VERSION}" \ | ||
| 1 && \ | ||
| python3 -m venv "${VIRTUAL_ENV}" | ||
|
|
||
| # Isolate the serving stack from Ubuntu's system Python packages. | ||
| RUN python3 -m pip install --no-cache-dir --upgrade \ | ||
| "pip<26" \ | ||
| "setuptools>=77,<81" \ | ||
| wheel \ | ||
| uv | ||
|
|
||
| # Rust/Cargo are required by instance/resource_agent. | ||
| RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ | ||
| sh -s -- -y \ | ||
| --profile minimal \ | ||
| --default-toolchain "${RUST_TOOLCHAIN}" && \ | ||
| chmod -R a+rwX "${RUSTUP_HOME}" "${CARGO_HOME}" && \ | ||
| rustc --version && \ | ||
| cargo --version | ||
|
|
||
| RUN python3 -c "import sys, tkinter; print('python:', sys.executable); print('tkinter:', tkinter.TkVersion)" && \ | ||
| ffmpeg -version >/dev/null | ||
|
|
||
| COPY constraints.txt /opt/cacheroute/constraints.txt | ||
| COPY requirements-dev.txt /opt/cacheroute/requirements-dev.txt | ||
|
|
||
| # Install the CUDA 13 PyTorch wheel set before vLLM and LMCache. | ||
| RUN python3 -m pip install --no-cache-dir \ | ||
| --index-url https://download.pytorch.org/whl/cu130 \ | ||
| -c /opt/cacheroute/constraints.txt \ | ||
| torch==2.11.0 \ | ||
| torchvision==0.26.0 \ | ||
| torchaudio==2.11.0 | ||
|
|
||
| RUN python3 -m pip install --no-cache-dir \ | ||
| -c /opt/cacheroute/constraints.txt \ | ||
| vllm==0.25.1 \ | ||
| lmcache==0.5.2 | ||
|
|
||
| RUN python3 -m pip install --no-cache-dir \ | ||
| -c /opt/cacheroute/constraints.txt \ | ||
| -r /opt/cacheroute/requirements-dev.txt | ||
|
|
||
| # Validate metadata and real imports. Build steps normally do not receive GPUs, | ||
| # so torch.cuda.is_available() is intentionally checked only at container run time. | ||
| RUN python3 -m pip check && \ | ||
| python3 -c 'from importlib.metadata import version; import sys; import redis; import torch; import torchcodec; import vllm; import lmcache; print("Python serving environment:"); print("python:", sys.version); print("torch:", torch.__version__); print("torch CUDA:", torch.version.cuda); print("vllm:", version("vllm")); print("lmcache:", version("lmcache")); print("transformers:", version("transformers")); print("redis:", version("redis")); print("torchcodec:", version("torchcodec"))' | ||
|
|
||
| WORKDIR /workspace/llm-stack/CacheRoute | ||
|
|
||
| CMD ["sleep", "infinity"] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| # CUDA 13 / vLLM 0.25.1 / LMCache 0.5.2 profile | ||
|
|
||
| This directory provides an additive development-image profile for the target | ||
| serving baseline tracked by Issue #148. It does not replace or modify the | ||
| existing CUDA 12.8 / vLLM 0.13.x / LMCache 0.3.11 environment. | ||
|
|
||
| ## Target stack | ||
|
|
||
| | Component | Version | | ||
| |---|---| | ||
| | Base image | `nvidia/cuda:13.0.0-devel-ubuntu22.04` | | ||
| | Python | `3.12.x` | | ||
| | PyTorch | `2.11.0+cu130` | | ||
| | torchvision | `0.26.0+cu130` | | ||
| | torchaudio | `2.11.0+cu130` | | ||
| | vLLM | `0.25.1` | | ||
| | LMCache | `0.5.2` | | ||
|
|
||
| The image uses `/opt/venv` to isolate serving dependencies from Ubuntu system | ||
| Python packages. FFmpeg is installed because current TorchCodec wheels require | ||
| its shared libraries. Rust/Cargo and Tkinter remain available for the existing | ||
| CacheRoute resource-agent and desktop-dashboard workflows. | ||
|
|
||
| ## Files | ||
|
|
||
| - `Dockerfile`: complete CUDA 13 development image. | ||
| - `constraints.txt`: exact serving-stack versions and shared compatibility | ||
| ranges. | ||
| - `requirements-dev.txt`: CacheRoute application/development dependencies that | ||
| are compatible with the target stack. | ||
|
|
||
| `requirements-dev.txt` intentionally excludes `Booktype==1.5`. That package is | ||
| from the Python 2 era and can overwrite the modern `redis` module with invalid | ||
| Python 2 source files. | ||
|
|
||
| ## Build | ||
|
|
||
| Use this directory as the Docker build context. This keeps the build context | ||
| small and avoids sending models, KDN databases, logs, or other repository data | ||
| to the Docker daemon. | ||
|
|
||
| ```bash | ||
| cd /llm-stack/CacheRoute | ||
|
|
||
| sudo docker build \ | ||
| -f env/docker/cu130/Dockerfile \ | ||
| -t cacheroute:dev-vllm0.25.1-lmcache0.5.2-cu130 \ | ||
| env/docker/cu130 | ||
| ``` | ||
|
|
||
| The Dockerfile is compatible with both the legacy Docker builder and BuildKit. | ||
| With Buildx, the equivalent command is: | ||
|
|
||
| ```bash | ||
| sudo docker buildx build \ | ||
| --load \ | ||
| --progress=plain \ | ||
| -f env/docker/cu130/Dockerfile \ | ||
| -t cacheroute:dev-vllm0.25.1-lmcache0.5.2-cu130 \ | ||
| env/docker/cu130 | ||
| ``` | ||
|
|
||
| ## Create the development container | ||
|
|
||
| ```bash | ||
| sudo docker run -d \ | ||
| --name cacheroute-dev-cu130 \ | ||
| --gpus all \ | ||
| --network host \ | ||
| --ipc=host \ | ||
| --shm-size=64g \ | ||
| --ulimit memlock=-1 \ | ||
| --ulimit stack=67108864 \ | ||
| -e HF_HOME=/workspace/llm-stack/cache/hf \ | ||
| -e TORCH_HOME=/workspace/llm-stack/cache/torch \ | ||
| -e LMCACHE_HOME=/workspace/llm-stack/cache/lmcache \ | ||
| -e PYTHONPATH=/workspace/llm-stack/CacheRoute \ | ||
| -e PYTHONHASHSEED=0 \ | ||
| -v /llm-stack:/workspace/llm-stack \ | ||
| -w /workspace/llm-stack/CacheRoute \ | ||
| cacheroute:dev-vllm0.25.1-lmcache0.5.2-cu130 | ||
| ``` | ||
|
|
||
| Enter the container: | ||
|
|
||
| ```bash | ||
| sudo docker exec -it cacheroute-dev-cu130 bash | ||
| ``` | ||
|
|
||
| Install the mounted CacheRoute source without re-resolving the serving stack: | ||
|
|
||
| ```bash | ||
| cd /workspace/llm-stack/CacheRoute | ||
| python3 -m pip install -e . --no-deps | ||
| python3 -m pip check | ||
| ``` | ||
|
|
||
| ## Runtime sanity check | ||
|
|
||
| ```bash | ||
| python3 - <<'PY' | ||
| import sys | ||
| import torch | ||
|
|
||
| print("python:", sys.executable) | ||
| print("torch:", torch.__version__) | ||
| print("torch CUDA runtime:", torch.version.cuda) | ||
| print("CUDA available:", torch.cuda.is_available()) | ||
| print("GPU count:", torch.cuda.device_count()) | ||
| PY | ||
| ``` | ||
|
|
||
| The active interpreter should be `/opt/venv/bin/python3`, and CUDA should be | ||
| available when the container is started with `--gpus all`. | ||
|
|
||
| ## LMCache MP runtime note | ||
|
|
||
| This profile targets LMCache MP mode and vLLM `LMCacheMPConnector`. Do not use | ||
| the legacy `LMCACHE_CONFIG_FILE`, `remote_url`, or | ||
| `--kv-offloading-backend lmcache` startup path as the primary interface. | ||
|
|
||
| When using LMCache MP, either unset PyTorch expandable segments: | ||
|
|
||
| ```bash | ||
| unset PYTORCH_CUDA_ALLOC_CONF | ||
| ``` | ||
|
|
||
| or explicitly enable vLLM's CuMem allocator. The conservative validation path | ||
| uses `unset PYTORCH_CUDA_ALLOC_CONF`. | ||
|
|
||
| A minimal LMCache server uses a dedicated MP endpoint and optional RESP L2 | ||
| adapter, while vLLM connects through `--kv-transfer-config` with | ||
| `LMCacheMPConnector`. | ||
|
|
||
| ## Validation status and scope | ||
|
|
||
| The image build has been validated with: | ||
|
|
||
| - Python `3.12.13`; | ||
| - PyTorch `2.11.0+cu130`; | ||
| - vLLM `0.25.1`; | ||
| - LMCache `0.5.2`; | ||
| - Transformers `5.12.1`; | ||
| - Redis Python client `8.0.1`; | ||
| - successful `pip check`; | ||
| - successful imports of Torch, TorchCodec, vLLM, LMCache, and Redis; | ||
| - successful startup of the existing CacheRoute Scheduler, KDN, Proxy, and | ||
| Instance workflow after installing FFmpeg. | ||
|
|
||
| This environment profile does not claim that Issue #148 is complete. In | ||
| particular, migration of the KDN raw Redis dump/inject contract to LMCache MP's | ||
| RESP L2 behavior remains a runtime-integration task. This PR therefore adds the | ||
| new installation/build interface without changing the legacy environment or | ||
| CacheRoute routing, queueing, and storage behavior. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # CacheRoute CUDA 13 serving profile. | ||
| # | ||
| # Keep the binary-sensitive serving stack exact. Application dependencies are | ||
| # installed separately from requirements-dev.txt under these constraints. | ||
|
|
||
| torch==2.11.0 | ||
| torchvision==0.26.0 | ||
| torchaudio==2.11.0 | ||
| vllm==0.25.1 | ||
| lmcache==0.5.2 | ||
|
|
||
| # Shared compatibility ranges required by the target serving stack. | ||
| transformers>=5.5.3,<6.0 | ||
| sentence-transformers>=5.6,<6.0 | ||
| huggingface-hub>=1.5,<2.0 | ||
| fastapi>=0.133,<0.137 | ||
| starlette>=1.0.1,<2.0 | ||
| aiohttp>=3.13.3,<4.0 | ||
| pydantic>=2.12,<3.0 | ||
| numpy>=1.26,<2.3 | ||
| setuptools>=77,<81 | ||
| requests>=2.33,<3.0 | ||
| redis>=5.1,<9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # CacheRoute application and development dependencies for the CUDA 13 profile. | ||
| # | ||
| # PyTorch, vLLM, and LMCache are intentionally installed by the Dockerfile and | ||
| # must not be added here. This file is additive and does not replace the legacy | ||
| # root requirements.txt profile. | ||
|
|
||
| sentence-transformers>=5.6,<6.0 | ||
| faiss-cpu==1.13.1 | ||
|
|
||
| fastapi>=0.133,<0.137 | ||
| uvicorn>=0.38,<0.52 | ||
| starlette>=1.0.1,<2.0 | ||
| pydantic>=2.12,<3.0 | ||
| httpx>=0.28,<0.29 | ||
| aiohttp>=3.13.3,<4.0 | ||
| requests>=2.33,<3.0 | ||
| redis>=5.1,<9 | ||
|
|
||
| transformers>=5.5.3,<6.0 | ||
| huggingface-hub>=1.5,<2.0 | ||
| safetensors>=0.7,<0.9 | ||
| datasets>=4.4,<5.0 | ||
|
|
||
| numpy>=1.26,<2.3 | ||
| scipy>=1.16,<1.17 | ||
| pandas>=2.3,<2.4 | ||
| scikit-learn>=1.7,<1.8 | ||
| matplotlib>=3.10,<3.11 | ||
|
|
||
| jupyter_client>=8.6,<9.0 | ||
| pyzmq>=27.1,<28.0 | ||
|
|
||
| pyyaml>=6.0,<7.0 | ||
| tqdm>=4.67,<5.0 | ||
| warcio>=1.7,<2.0 | ||
| beautifulsoup4>=4.14,<5.0 | ||
|
|
||
| # Booktype 1.5 is intentionally excluded. It is a Python-2-era package that can | ||
| # overwrite the modern redis package with Python 2 source files. | ||
|
|
||
| pytest>=9.0,<10.0 | ||
| black>=24,<27 | ||
| isort>=5.13,<9 | ||
| mypy>=1.11,<3 |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When this development image later runs Ubuntu Python-based administration tools such as
add-apt-repository, replacing/usr/bin/python3with Python 3.12 makes those tools load the wrong interpreter; Ubuntu 22.04 installs extensions such asapt_pkgfor its system Python 3.10, so the commands can fail with import errors. The checkedupdate-alternatives --helpconfirms that--install <link> <name> <path>turns the supplied link into an alternatives-managed symlink. Create the virtual environment with/usr/bin/python${PYTHON_VERSION} -m venvdirectly and leave/usr/bin/python3unchanged; the existingPATHalready selects/opt/venv/bin/python3for application commands.Useful? React with 👍 / 👎.