@yingwaili requests that we do the training in such away that restarting the model is basically automated. Maybe we can have some function to write a restart script in the model directory or something. Or a function that restarts training and takes as input only the directory of the checkpoint.
Items of difficulty
- restarting databases is a bit fragile - may need to be customized somehow.
@yingwaili requests that we do the training in such away that restarting the model is basically automated. Maybe we can have some function to write a restart script in the model directory or something. Or a function that restarts training and takes as input only the directory of the checkpoint.
Items of difficulty