With this repository we share
- Our extensive experimental data comparing optimizers on language models (data folder). Note that some details might be missing, you can find everything in the paper
- Our visualizations scripts, so you can better match with the paper (tuning plots folder)
- We additionally share all training curves!
- Our quadratic example folder serves as a playground to understand the Adam/SGD/Signum gap in language models!