Multi-GPU Deployment Configuration and Optimal Parallel Strategy

[Wangxingyan]

Problem

The project looks like only support matching the best local LLM for single-GPU scenarios？

However, I want to leverage multiple GPUs (e.g., 4 or 8 GPUs) to run larger models that cannot fit on a single GPU, or to achieve higher throughput via parallel inference.How to specify the target number of GPUs for deployment matching？

Proposed Solution

1. Allow users to specify target GPU count when searching for compatible models
2. Provide the optimal parallel strategy configuration for model deployment

Alternatives Considered

No response

[Andyyyy64]

Thanks for raising this. I am linking it to #65.

I want #65 to track the concrete multi-GPU simulation work: CLI parsing, multiple GPU specs, VRAM fit, and conservative speed estimates.

This issue can stay open for the deployment strategy side: tensor split, pipeline vs tensor parallel, GPU count, interconnect assumptions, and backend-specific guidance.