4x4 mult model replication or checkpoints

[carlguo866]

I'm working on a project that tries to interpret the mechanisms of these models that internalize CoT and compare them against models without such internalization. To this goal, I'm interested in obtaining the checkpoints at each training epoch. However, my replication seems very qualitatively different in its hidden states from the checkpoint that this repo provides (the original checkpoint seems to have more visible structures), despite both reaching 100% accuracy in evals. I'm wondering if it is possible for you to provide the exact training hyperparameters for 4x4 mult or the epoch checkpoints. Thanks so much!

FYI for my replication, I followed the training script for 9x9 mult but modified REMOVE_PER_EPOCH to be 1 instead.

[da03]

Hmm interesting. For 4x4 we used the same hyperparameter as for 9x9, so REMOVE_PER_EPOCH should still be 8 instead of 1.