Issue with code crashing due to high RAM usage

[morteza-mahdiani]

I am currently evaluating a 3D ResNet that has been pre-trained on short video segments. However, I encountered a problem while running the "score_model" function with an instance of the "ModelCommitment" class. It appears that the code is utilizing all available memory resources, causing it to crash.

To reproduce this issue, I modified the "preprocess_images" method located in the "activations/pytorch.py" Python file. In my modification, I pass concatenated images as a 4D input to the model. The activations model is then initialized based on this preprocessing step.

During execution, I monitored both GPU and CPU activity. Although the GPU usage is temporary, the CPU usage gradually increases until it eventually freezes the procedure. I would like to note that when using small resolutions for input images that are concatenated to create a static video frame, everything works fine. However, increasing the input image size beyond 32x32 leads to the problem mentioned above.

Please investigate this issue and provide guidance on resolving it. Thank you.

[mschrimpf]

Hi @morteza-mahdiani, I'm guessing that the collection of early-layer activations runs out of memory (it's not clear to me how this spikes the CPU usage, but it might be that your machine is trying to swap contents in and out of memory). In non-temporal models, this can already be a stressor and for temporal models you might collect activations over time too. For context, the model tools wrapper will try to find the best available layer among the layers you provided for each of the cortical regions (https://github.com/brain-score/model-tools/blob/75365b54670d3f6f63dcdf88395c0a07d6b286fc/model_tools/brain_transformation/neural.py#L86-L93).

Have you tried to pass in only a small list of layers, preferably high-level layers? Those typically have smaller activation sizes and thus should go easier on memory. If these work, you could try to submit the model to the website with all layers, we have a few 100 GB of memory available.

[morteza-mahdiani]

Hi @mschrimpf, Thank you very much for your response and suggestions. I greatly appreciate your help. I followed your advice and tried passing in a smaller list of high-level layers, including just one high-level layer, but unfortunately, I still faced the same issues. It seems that the problem lies in the heavy memory load and the challenges of fitting the regressor. While submitting to the website could be a good idea, it appears to be impossible for me to handle locally. Thank you again for your assistance.

[mschrimpf]

Another thing you could try would be to run PCA to reduce the feature dimensionality. See e.g. here for an example: https://github.com/brain-score/model-tools/blob/75365b54670d3f6f63dcdf88395c0a07d6b286fc/model_tools/brain_transformation/neural.py#L70
If computing the PCA components is too heavy for your system, you could just use a random projection or a random sub-selection of features to at least prototype the model.

[morteza-mahdiani]

Thank you for suggesting an alternative approach for reducing feature dimensionality. That seems like a good solution too.