[wip] Support ray/dynamo nightly + vLLM 0.22 (cu129) across all extras#2064
[wip] Support ray/dynamo nightly + vLLM 0.22 (cu129) across all extras#2064praateekmahajan wants to merge 3 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
praateekmahajan
force-pushed
the
praateek/all-extras-cu129
branch
from
dc21302 to
aaa42ad
Compare
|
/ok to test aaa42ad /claude review @greptileai review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| def _vllm_cu129_index_url() -> str | None: | ||
| """The vLLM cu129 wheel index for the exact version ai-dynamo[vllm] pins. | ||
|
|
||
| ai-dynamo's [vllm] extra pins an exact vllm (e.g. ``==0.22.1``) that may | ||
| differ from Curator's base vllm — the base installs ai-dynamo WITHOUT its | ||
| [vllm] extra, so its vllm comes from Curator's own pin, while the actor | ||
| venv installs ``ai-dynamo[vllm]`` and must honor ai-dynamo's pin. vLLM | ||
| publishes a per-version cu129 wheel index at ``wheels.vllm.ai/<v>/cu129``; | ||
| pointing at the pinned version means its ``+cu129`` local build sorts above | ||
| the default cu130 wheel under unsafe-best-match. Derived from ai-dynamo's | ||
| own metadata so a nightly bump (which changes the vllm pin) needs no edit. | ||
|
|
||
| Returns None if ai-dynamo (or its vllm pin) can't be found — only happens | ||
| when the dynamo backend isn't actually installed, where this is unused. | ||
| """ | ||
| try: | ||
| requirements = importlib.metadata.requires("ai-dynamo") or [] | ||
| except importlib.metadata.PackageNotFoundError: | ||
| return None | ||
| for raw in requirements: | ||
| req = Requirement(raw) | ||
| if req.name != "vllm": | ||
| continue | ||
| pinned = next((spec.version for spec in req.specifier if spec.operator in ("==", "===")), None) | ||
| if pinned: | ||
| return f"https://wheels.vllm.ai/{pinned}/{_ACTOR_VENV_CUDA_TAG}" | ||
| return None |
There was a problem hiding this comment.
Missing test coverage: _vllm_cu129_index_url() has four distinct return paths (PackageNotFoundError → None, no vllm in requirements → None, vllm present but no == pin → None, happy path → URL string) and is called at module level to build _ACTOR_VENV_UV_OPTIONS. Since a wrong URL here silently installs a cu130 vllm wheel in the actor venv (the exact failure this PR is preventing), it would be good to have unit tests — at minimum for the happy path and the "ai-dynamo not installed" fallback, using unittest.mock.patch over importlib.metadata.requires.
Greptile SummaryThis draft PR threads the Ray 3.0 nightly + ai-dynamo nightly + vLLM 0.22.0+cu129 (CUDA 12.9) stack through both
Confidence Score: 3/5Not safe to merge as-is: the ray nightly wheel entry in core dependencies is restricted to Python 3.13 only, silently dropping ray for Python 3.11 and 3.12 users the project still supports. The core ray dependency is now a direct-URL nightly wheel valid only for Python 3.13 + x86_64 + Linux. Since the project supports Python >=3.11, any 3.11 or 3.12 install gets no ray from the base dependency list — import ray in core modules would fail at runtime with no warning at install time. The ray entry in the dependencies block of pyproject.toml needs a fallback for Python 3.11/3.12. The _vllm_cu129_index_url function in vllm.py has two minor defensive-coding gaps worth addressing before merge. Important Files Changed
Reviews (1): Last reviewed commit: "Support ray/dynamo nightly + vLLM 0.22 (..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "pandas>=2.1.0", | ||
| "pyarrow", | ||
| "ray[default,data]>=2.55.1", | ||
| "ray[default,data] @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp313-cp313-manylinux2014_x86_64.whl ; python_version == '3.13' and platform_machine == 'x86_64' and platform_system != 'Darwin'", |
There was a problem hiding this comment.
The new ray entry applies only when python_version == '3.13' (plus x86_64 non-Darwin). The project declares requires-python = ">=3.11,<3.14", so Python 3.11 and 3.12 users install the package with no ray in dependencies at all. Previously the constraint was unconditional (ray[default,data]>=2.55.1). Any import ray in core nemo_curator modules will fail with ModuleNotFoundError on the two still-supported Python minor versions. A fallback entry for python_version < '3.13' (or a separate stable Ray constraint) is needed to keep the existing platforms functional.
| for raw in requirements: | ||
| req = Requirement(raw) |
There was a problem hiding this comment.
Requirement() can raise at import time
_vllm_cu129_index_url() is consumed at module-level (line 119 in _ACTOR_VENV_UV_OPTIONS), so any exception inside it becomes an import error. Only PackageNotFoundError is caught; if ai-dynamo's dist-info contains a non-standard or malformed requirement string, packaging.requirements.Requirement(raw) raises InvalidRequirement, crashing every import of this module. Wrapping the Requirement(raw) call in a try/except InvalidRequirement: continue would be sufficient.
| for raw in requirements: | ||
| req = Requirement(raw) | ||
| if req.name != "vllm": | ||
| continue | ||
| pinned = next((spec.version for spec in req.specifier if spec.operator in ("==", "===")), None) | ||
| if pinned: | ||
| return f"https://wheels.vllm.ai/{pinned}/{_ACTOR_VENV_CUDA_TAG}" |
There was a problem hiding this comment.
importlib.metadata.requires("ai-dynamo") returns every requirement across all extras. The [vllm] extra's entry looks like "vllm==0.22.x ; extra == \"vllm\"", but so would any vllm pin in a hypothetical [vllm-dev] or other extra. The loop returns on the first vllm-named req without checking req.marker for extra == "vllm", so a future ai-dynamo refactor that adds a secondary vllm constraint could silently pick the wrong index URL.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Enable the Ray 3.0 nightly + ai-dynamo nightly + vLLM 0.22 inference stack on the CUDA-12.9 image while keeping the full Curator dependency set (`uv sync --all-extras --all-groups`) resolvable and buildable. pyproject.toml: - ray: 3.0.0.dev0 nightly wheel routed per python tag (x86_64) via [tool.uv.sources] so `dependencies` stays a plain `ray[default,data]`; aarch64/other resolve ray from PyPI via the >= floor (keeps ray[llm]'s default cu130 vllm off aarch64). Rolling /latest/ wheels are re-pinned by re-locking, never frozen. - ai-dynamo and ai-dynamo-runtime >=1.3.0.dev0, both first-party so prerelease="if-necessary-or-explicit" enables the newest nightly without blanket prereleases (runtime is a transitive with stable releases, so it needs an explicit marker or uv backtracks to an older dynamo dev). - vLLM 0.22.0+cu129 via a dedicated cu129 wheel index + tool.uv.sources (default vLLM is now cu130; keep torch/vllm on CUDA 12.9). - drop nixl-cu13: ray[llm]/nixl hard-pin the CUDA-13 NIXL backend, whose eager `import nixl_ep` dlopens the absent libcudart.so.13 on cu12.9; keep the nixl meta + nixl-cu12 backend. - opencv-python -> opencv-python-headless (no libGL/GPL GUI/FFmpeg bundling; matches vllm/mistral_common/albumentations). - bump torch/torchvision/torchaudio/torchcodec to the 2.11 cu129 line. dynamo actor venv runtime_env (vllm.py): Ray builds it via a bare `uv pip install ai-dynamo[vllm]` that ignores pyproject, so force cu129 the way uv/vLLM document: --torch-backend cu129, unsafe-best-match (needed for nixl's split index resolution), and a per-version cu129 vllm index derived from ai-dynamo's own pin (`_vllm_cu129_index_url`); the --override file pins ray== and drops nixl-cu13. Signed-off-by: Praateek <praateekm@gmail.com>
praateekmahajan
force-pushed
the
praateek/all-extras-cu129
branch
from
aaa42ad to
4f0cdf4
Compare
|
/ok to test 4f0cdf4 |
…ror) The prior docker/Dockerfile stub only patched /opt/venv, but CI unit tests run from a fresh .venv (uv sync), so ray's dashboard frontend was still missing there — the dashboard process died with FrontendNotFoundError and every ray.util.state call (Xenna drives pipelines through it) failed with "Could not read 'dashboard' from GCS", erroring all xenna backends/audio/text tests. Move the stub into nemo_curator/__init__.py so it runs once on import, relative to the installed ray (works in any venv), and gate it to dev/nightly ray so published releases (which ship client/build) are untouched. Drop the redundant Dockerfile stub. Signed-off-by: Praateek <praateekm@gmail.com>
1 participant
What
Enables the Ray 3.0 nightly + ai-dynamo nightly + vLLM 0.22 (CUDA 12.9) inference stack while keeping the full Curator dependency set resolvable and buildable — validated with
uv sync --all-extras --all-groups(full container builds + every extra imports together: torch/vllm/cv2/cudf/cuml/nixl/dynamo/nemo_curator).Changes
pyproject.tomlraytracks the3.0.0.dev0nightly wheel (rolling/latest/URL).ai-dynamoandai-dynamo-runtime>=1.3.0.dev0, both first-party soprerelease="if-necessary-or-explicit"enables the newest nightly without blanket prereleases. (ai-dynamo-runtimeis a transitive with stable releases, so without an explicit marker uv backtracks to an older dynamo dev.)prerelease = "if-necessary-or-explicit"(was a blanketallow).0.22.0+cu129via a dedicated cu129 wheel index +tool.uv.sources— the default vLLM wheel is now cu130 (VLLM_MAIN_CUDA_VERSION=13.0), so torch/vllm are kept on CUDA 12.9.nixl-cu13:ray[llm]/nixlhard-pin the CUDA-13 NIXL backend, whose eagerimport nixl_epdlopens the absentlibcudart.so.13on a cu12.9 image; thenixlmeta +nixl-cu12backend remain.opencv-python→opencv-python-headless(nolibGL/GPL GUI/FFmpeg bundling; matches what vllm/mistral_common/albumentations already request).nemo_curator/core/serve/dynamo/vllm.py— the Dynamo actor venv is built by Ray'suvruntime_env via a bareuv pip install ai-dynamo[vllm]that ignorespyproject. Force cu129 the way uv/vLLM document:--torch-backend cu129,unsafe-best-match(required for nixl's split-index resolution), and a per-version cu129 vLLM index derived from ai-dynamo's own pin (so the actor honors dynamo's vLLM version as cu129); the--overridefile pinsray==<head>and dropsnixl-cu13.Notes
uv lock --refresh-package … --upgrade-package …for ray/ai-dynamo/ai-dynamo-runtime before building.docker build --ulimit nofile=1048576:1048576(rapids file count, unrelated to these deps).🤖 Generated with Claude Code