How to run such a memory intensive model

[seriouschenwannachange-art]

May I ask how you managed to run this memory intensive model. Did you use 3D parallelism?
When I was using DeepSpeed's pipeline for parallel processing, I found that due to the Batch class of the model itself, it was not very suitable for the input and output of the DeepSpeed pipeline model (I consulted DeepSpeed's official documentation, and I found that the pipeline model requires the inputs and outputs of each layer must be either a single torch.Tensor or a tuple of tensors).
What approach would you recommend in this case?

[seriouschenwannachange-art]

I experimented with converting the batch object to a tuple of tensors for input/output, but finally failed.

[seriouschenwannachange-art]

@wesselb

[wesselb]

Hey @seriouschenwannachange-art! Thanks for opening an issue.

You should be able to run inference the on a single GPU using about 40 GB of memory. Computing gradients and running a training loop should fit within 80 GB for batch size one.

We haven’t used solutions like DeepSpeed or other forms of model parallelism. It’s possible that these could work, but we have no experience with them.

[seriouschenwannachange-art]

Hey @seriouschenwannachange-art! Thanks for opening an issue.

You should be able to run inference the on a single GPU using about 40 GB of memory. Computing gradients and running a training loop should fit within 80 GB for batch size one.

We haven’t used solutions like DeepSpeed or other forms of model parallelism. It’s possible that these could work, but we have no experience with them.

Thanks for your reply!