Skip to content

feat: add vllm sidecar auto-registration.#69

Merged
fdvty merged 12 commits into
xLLM-AI:mainfrom
fdvty:features/vllm-sidecar-autoregister
Jun 17, 2026
Merged

feat: add vllm sidecar auto-registration.#69
fdvty merged 12 commits into
xLLM-AI:mainfrom
fdvty:features/vllm-sidecar-autoregister

Conversation

@fdvty

@fdvty fdvty commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds a standalone vLLM sidecar (vllm_sidecar/) that auto-registers a vLLM
instance into xllm-service's etcd, replacing the manual register_vllm.sh step
from the previous PR. Components:

  • etcd_registry.py — etcd gateway client (put/lease/keepalive for the instance key)
  • health.py — polls the vLLM /health endpoint before registering
  • meta.py — builds the InstanceMetaInfo JSON (backend_type=vllm, type=DEFAULT)
  • sidecar.py — the runtime entrypoint tying them together
  • tests for meta and the sidecar runtime

Related Issues

Change Type

  • New feature
  • Bug fix / Perf / Refactor / Docs / Test / Build or CI

Pull Request Checklist

  • Commit/PR title follows <type>: <subject>.
  • Rebased onto the latest main.
  • No C++ changed; Python formatted (ruff) and unit-tested (15 cases pass).

Reviewer Notes

fdvty added 4 commits June 17, 2026 14:14
Add a Python sidecar that health-gates vLLM registration, writes InstanceMetaInfo into etcd under a lease, keeps the lease alive, and deregisters on shutdown or repeated health failures.

The sidecar uses etcd's v3 HTTP gateway via requests only, avoiding grpcio/protobuf native dependencies.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new vllm_sidecar package to automatically register and maintain the liveness of a vLLM instance in the xllm-service cluster using etcd leases. The feedback focuses on enhancing the robustness of the HTTP-based etcd client and the health prober. Key recommendations include handling pre-existing schemes in etcd endpoints, safely parsing JSON responses to prevent crashes from invalid or null fields, and using a requests.Session in the health prober to enable connection reuse and avoid socket exhaustion.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread vllm_sidecar/etcd_registry.py
Comment thread vllm_sidecar/etcd_registry.py Outdated
Comment thread vllm_sidecar/etcd_registry.py Outdated
Comment thread vllm_sidecar/etcd_registry.py Outdated
Comment thread vllm_sidecar/health.py
@fdvty
fdvty force-pushed the features/vllm-sidecar-autoregister branch from 8ed2f7f to 93fe15e Compare June 17, 2026 06:32
@fdvty

fdvty commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review! Addressed every comment, and proactively audited the whole PR for the same classes of issues:

  • etcd_registry.py — hardened the entire JSON-parsing surface against proto3 omit-empty/null: non-object 200 bodies coerce to {}, non-JSON 200 falls through to the next endpoint, lease_keepalive tolerates null result/TTL, get tolerates an omitted value, and endpoint scheme prefixes are preserved.
  • health.py — VllmHealthProbe now reuses a single requests.Session (keep-alive) instead of a socket per poll.
  • sidecar.py / meta.py / demo — fixed a stale doc comment and removed a heartbeat-only flag (--xllm-service-url) from the demo startup that this PR's sidecar doesn't accept yet. Added unit tests for each new edge case (21 tests total). Suggest Squash and merge to collapse the review fixes into the feature commit.

@DongheJin
DongheJin self-requested a review June 17, 2026 07:04
Comment thread vllm_sidecar/sidecar.py Outdated
@fdvty
fdvty merged commit ba16f4f into xLLM-AI:main Jun 17, 2026
1 check passed
@fdvty
fdvty deleted the features/vllm-sidecar-autoregister branch June 17, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants