Support failover policy for multi-GPUs #14

What would you like to be added:

For some experiments, we may need the failover for GPUs, for example, if a100 is not enough, h100 is also fine, a bit more expensive but better than not.

Why is this needed:

Completion requirements:

This feature requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.