Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/load_balancing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The `consistent_hash` policy routes requests with the same session/user identifi

```bash
# Using CLI
vllm-router --policy consistent_hash --worker-urls http://worker1:8000,http://worker2:8000
vllm-router --policy consistent_hash --worker-urls http://worker1:8000 http://worker2:8000

# Using Python
from vllm_router import Router
Expand Down Expand Up @@ -123,7 +123,7 @@ The `round_robin` policy distributes requests evenly across all healthy workers
### Configuration

```bash
vllm-router --policy round_robin --worker-urls http://worker1:8000,http://worker2:8000
vllm-router --policy round_robin --worker-urls http://worker1:8000 http://worker2:8000
```

### Behavior
Expand All @@ -148,7 +148,7 @@ The `random` policy selects a random healthy worker for each request.
### Configuration

```bash
vllm-router --policy random --worker-urls http://worker1:8000,http://worker2:8000
vllm-router --policy random --worker-urls http://worker1:8000 http://worker2:8000
```

### Behavior
Expand All @@ -172,7 +172,7 @@ The `power_of_two` policy randomly selects two workers and routes to the one wit
### Configuration

```bash
vllm-router --policy power_of_two --worker-urls http://worker1:8000,http://worker2:8000,http://worker3:8000
vllm-router --policy power_of_two --worker-urls http://worker1:8000 http://worker2:8000 http://worker3:8000
```

### Behavior
Expand Down Expand Up @@ -204,7 +204,7 @@ vllm-router --policy cache_aware \
--cache-threshold 0.5 \
--balance-abs-threshold 32 \
--balance-rel-threshold 1.1 \
--worker-urls http://worker1:8000,http://worker2:8000
--worker-urls http://worker1:8000 http://worker2:8000
```

### Parameters
Expand Down