Question when reproducing the experiment in the paper

[Kevinstone-199898]

Hi, thanks for your sharing of the code. I have a question when I try to reproduce the experiment in the paper or more specifically, this figure.

I followed the code in this repo and set the parameters of 1B models according to the paper:

I also set the global batch size the same with the paper which is 512.
I use 8 H800 with 80GB. The Validation loss during training I got is as follow:

which seems very different from the figure in the paper. And I wonder whether I am doing something wrong or why is this. I hope to get your help. thanks a lot!

[iloshchilov]

A few things contribute the difference in the figure:

1. The codebase used is different with the one used in the paper. (assuming that your runs are using the code of this repo)
2. "The vocabulary size is different and this changes the scale of loss values." as mentioned in the readme file.
3. It seems that the run that is plotted correspond to about 2k iterations and the results at 2k iterations of the second figure do not correspond to the results at 200k iterations of the first figure. In general, the speedup grows with run length.
   To my understanding, if you run both algorithms for 10k iterations, the results should be in line with the values given in the figure of readme file. To make it more affordable, I would personally run for 1k context length and try to reproduce these numbers. Only then I would run for 4k context length. However, if the runtime is not an issues, then I would just run 4k it longer.