A tutorial for new PhD students and Interns at MPI. No efficiency trick, just the minimal setup to run a sweep!
Thanks to Niccolo Ajroldi and Alexandru Meterez who showed me some of these tricks, wrote part of the code, and helped me many times in general.
Smarter yet non tutorial-like repos: https://github.com/JonasGeiping/kraken, https://github.com/alexandrumeterez/condor-bid-wrapper
-
Login into the cluster from VScode
- Open VSCode and connect to the MPI cluster using the SSH blu little square in the bottom left of the window. It might ask you to install plugins. That's fine.
- Open up a terminal,
-
Set up GitHub on cluster (to download this repo). Run these instructions in sequence:
git config --global user.name "Your_Github_Name"git config --global user.email "your_github_email@mail.com"ssh-keygen -t ed25519 -C "your_github_email@mail.com"eval "$(ssh-agent -s)"ssh-add ~/.ssh/id_ed25519cat ~/.ssh/id_ed25519.pub- Go to https://github.com/settings/keys and enter the string you got as output, save the SSH key with any name.
-
Now let's install Python, specifically Miniconda. Run these in sequence.
-
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.shchmod +x Miniforge3-Linux-x86_64.sh./Miniforge3-Linux-x86_64.shexport PATH="$HOME/miniforge3/bin:$PATH"conda init bash
-
Log out and back into condor.
- Open a terminal
- git clone this repo
- Now open the folder using the
Opencommand in VScode. You should see this folder there.
-
Create a nice environment
conda create -n test python=3.12or your preferred python version.conda activate test- Optional: Select in VSCode a Python Interpreter within the command palette -- specifically, this environment. So that you have no warnings.
- Install packages needed for this tutorial:
pip install torch torchvision einops pandas argparse itertools wandb - Optional: Create a (or log into your) Weights & Biases account, then go to https://wandb.ai/settings and create an API key. Then do
wandb loginfrom the condor terminal in your environment and paste it there.
-
Folders and permissions:
- create folders
resultsandrunsinside the repositoty folder - give permission to the executable with
chmod +x run.sh
- create folders
-
Inspect
submit.py. Around line 35 is the definition of the sweep. Put "false" in Weights and Biases if you wrongly decide not to use it. Here, we sweep over 3 learning rates, and have 2 seeds. -
Run
python submit.py. If you renamed your environment, make sure to change this inrun.sh. You should see a sequence of 6 jobs: 2 seeds, 3 learning rates. Grid search your learning rates! -
Check status with
condor_q your_mpi_username. You should see all your runs there. "I" means the job is idle, wait. If you get to the letter "R" means its running! If you see "H" you likely did something wrong. Remove the job if it stays on hold for long (see below) - it's probably wrong dependencies, missing folders, etc. With the current bid I set, you should be running on a NVIDIA A100-SXM4-40GB. You can also request a specific GPU. -
Check the
runsfolder, here are all your runs, named by uid. There are logs, errors, and stuff. Underresultsis also the CSV format. Be careful, I did not log everything in there. To improve. -
Check your wandb or the stdout: you should get something >85% on CIFAR10. Which is bad stuff but hey, takes 2 minutes to run.
-
If you want to do some heavy stuff like downloading datasets, make sure you do this within an interactive job : run
condor_submit_bid 25 -i -append request_cpus=8 -append request_memory=10000to connect to some CPUs not to slow down things. For some packages like the Mamba one, you need to actually connect to a GPU before installing. For installing vanilla stuff, CPUs or even login nodes are enough. -
To remove a job you don't like, just type
condor_rm job_id, where the job id looks something like 15501540.0. To remove all your jobs (carefully) runcondor_rm your_mpi_username. -
Always save your models to evaluate them at later stages or train more. Here, indeed, I forgot to do it.