Feature Request: Multi-GPU simulation via --gpu flag (heterogeneous setups)

[cobra91]

Feature Request

Support simulating multiple GPUs via the --gpu flag, including heterogeneous setups (e.g., RTX 5080 + RTX 5060 Ti 16GB).

Motivation

Many users run or plan multi-GPU setups for local LLM inference. Currently whichllm only allows simulating a single GPU, making it impossible to evaluate which models would fit on a dual-GPU rig.

Common use cases:

• Dual RTX 3060 12GB — combined 24 GB VRAM, common budget build https://www.youtube.com/watch?v=3XC8BA5UNBs
• Dual RTX 3090 — 48 GB total, popular used-market combo
• RTX 4090 + RTX 3090 — asymmetric upgrades
• H100 x2, x4 — enterprise setups

Proposed CLI Syntax

# Heterogeneous dual GPU
uvx whichllm@latest --gpu "RTX 5080,RTX 5060 Ti 16GB"

# Identical GPUs with count shorthand
uvx whichllm@latest --gpu "2x RTX 4090"
uvx whichllm@latest --gpu "4x H100"

Technical Notes

The type system already partially supports this — HardwareInfo.gpu is typed as list[GPUInfo]. Physical multi-GPU detection works when multiple GPUs are present. The gap is in the CLI layer and VRAM pooling logic.

What's needed:

1. CLI parsing — _handle_gpu_option() should accept comma-separated GPU names or Nx prefix
2. VRAM pooling — combine VRAM across GPUs (total available = sum - overhead)
3. Speed estimation — for heterogeneous setups, the slowest GPU becomes the bottleneck for split-tensor inference
4. Bandwidth model — inter-GPU communication overhead (PCIe/NVLink) affects effective throughput

Environment

• uvx whichllm@latest
• Windows 11 / any OS

[0xDE57]

This would be cool.

To take it a step further, multi-GPU setups should consider PCI-e lanes. As many CPU's / motherboards cannot run multiple cards at full x16. eg:

Perhaps 1 card runs at x16 and another at x4.
Or a bifurcated x16 board to run two x8 slots.
Direct CPU connection vs chipset hop, etc....

[theodufort]

Agreed, I I have 3x RTX a4000 16gb for a total of 48gb of vram so that would be useful.
I made a local merge request but can't push, @Andyyyy64 can you add me as contributor?

Here is a preview:

[Andyyyy64]

Thanks all. I am going to use this as the umbrella issue for multi-GPU simulation.

The first useful scope is probably:

• parse multiple --gpu values, including comma-separated GPUs and 2x RTX 4090
• represent heterogeneous GPUs without pretending they are one perfect pooled device
• expose conservative VRAM fit behavior first
• keep throughput estimates low-confidence until PCIe/NVLink and tensor-split assumptions are modeled

The PCIe lane point is real. I do not want the first version to imply exact multi-GPU speed. Fit simulation can come first; speed strategy can be a separate layer.

For PRs, opening from a fork is enough. I do not need to add repo access for that.

[theodufort]

Thanks all. I am going to use this as the umbrella issue for multi-GPU simulation.

The first useful scope is probably:

• parse multiple --gpu values, including comma-separated GPUs and 2x RTX 4090
• represent heterogeneous GPUs without pretending they are one perfect pooled device
• expose conservative VRAM fit behavior first
• keep throughput estimates low-confidence until PCIe/NVLink and tensor-split assumptions are modeled

The PCIe lane point is real. I do not want the first version to imply exact multi-GPU speed. Fit simulation can come first; speed strategy can be a separate layer.

For PRs, opening from a fork is enough. I do not need to add repo access for that.

PR is ready #84