Out of memory

[kaiw7]

Hi, I'm using 4* A100 (80G) to launch the grpo training for Qwen-omni-3B, but it shows the out of memory. Could I know what's the training config? And how to know whether flash_attn or others need to be correctly configured for saving gpu memory?

[HarryHsing]

Hi, this is the training config I used: [link]. I’ve tried Qwen-omni-7B with 4 × A100 (80G), and it works without OOM.

[kaiw7]

Did you use the num_generation=16 for training without OOM? The max value I can set is 8. In addition, this value has to been divided by (batch_size_per_divicenum_gpugrad_accumlation). Could I know whether you also set those params accordingly? Thanks a lot

[VoyageWang]

Did you use the num_generation=16 for training without OOM? The max value I can set is 8. In addition, this value has to been divided by (batch_size_per_divice_num_gpu_grad_accumlation). Could I know whether you also set those params accordingly? Thanks a lot

I met the same problem for 7B with 8 80g GPU. Can you give some suggestions @HarryHsing

[HarryHsing]

Hi, you may also need to check two things that affect memory usage:

1. Training data type (Image+Audio is lighter, Video+Audio uses much more memory)

2. Configs in qwen_omni_utils (e.g., DEFAULT_MAX_AUDIO_LEN_SEC, MAX_PIXELS, FPS, FPS_MAX_FRAMES, etc.)

Both of these can significantly impact GPU memory.

[HarryHsing]

Did you use the num_generation=16 for training without OOM? The max value I can set is 8. In addition, this value has to been divided by (batch_size_per_divice_num_gpu_grad_accumlation). Could I know whether you also set those params accordingly? Thanks a lot

You can check it out here https://github.com/HarryHsing/EchoInk/blob/main/src/scripts/run_grpo_image_audio_avqa-r1.sh

[kaiw7]

@HarryHsing Hi, I'm using the default config from the repo. I didn't change those config but still only use num_generations=8. Could I know if there are some changes in your actual running? Because I use the same bash script. Many thanks!

[HarryHsing]

@HarryHsing Hi, I'm using the default config from the repo. I didn't change those config but still only use num_generations=8. Could I know if there are some changes in your actual running? Because I use the same bash script. Many thanks!

Sorry for the late reply! I use num_generations=16 with 8 × A100.

[SeinCore]

@HarryHsing Hi, I'm using the default config from the repo. I didn't change those config but still only use num_generations=8. Could I know if there are some changes in your actual running? Because I use the same bash script. Many thanks!

Sorry for the late reply! I use num_generations=16 with 8 × A100.
CUDA_VISIBLE_DEVICES=4,5,6,7,8 torchrun --nproc_per_node="5" \ --nnodes="1" \ --node_rank="0" \ --master_addr="127.0.0.1" \ --master_port="12366" \ src/open_r1/grpo.py \ --output_dir $DATA/finetune/omni-r1 \ --model_name_or_path $DATA/model/Qwen2.5-Omni-3B \ --dataset_name $DATA/dataset/AVQA_R1/train/omni_rl_format_train.json \ --deepspeed local_scripts/zero3.json \ --max_prompt_length 16384 \ --max_completion_length 768 \ --per_device_train_batch_size 1 \ --gradient_accumulation_steps 1 \ --learning_rate 1e-5 \ --lr_scheduler_type "cosine" \ --weight_decay 0.01 \ --bf16 \ --logging_steps 1 \ --gradient_checkpointing false \ --len_control false \ --weighted_reward false\ --attn_implementation flash_attention_2 \ --max_pixels 401408 \ --num_train_epochs 1 \ --run_name $WANDB_NAME \ --save_steps 100 \ --beta 0.001 \ --max_grad_norm 5 \ --save_only_model true \ --num_generations 8 \ --model_type omni \ --use_audio_in_video true \ --use-peft true \ --lora_alpha 16 \ --lora_r 8 \ --lora_dropout 0.1 \ --lora_target_modules q_proj v_proj k_proj o_proj gate_proj up_proj down_proj \ --dataloader_num_workers 8 \
I'm using 5x 80GB A800 GPUs for LoRA fine-tuning, but OOM. Could you please advise what might be causing this?