Prediction incorrect when load saved model #88
Description
Hi,
Now I am using Keras 1.2.2 with mxnet backend on ResNet50. I observed a very weird phenomenon:
If we use mxnet as backend, when we finish training and save the model to disk (by model.save_weight(…)), then reload the model to do the prediction, we will get almost the same ouput for training data. In other word, we use the reloaded model to predict on the training data, all output will be almost the same and classify everything to one category. This shouldn’t happen since we have very high training accuracy. If we directly use trained model to predict on training data (without saving and loading), then everything becomes fine.
However, if we use Theano as backend, (keep all the code unchanged and change ~/.keras/keras.json to use Theano), then everything will be fine. The reloaded model will do its job correctly.
Have you ever seen such a weird phenomenon? Do you think we have some issue to save the model?
One more information: the way we used multi-gpu for training is directly add “context=['gpu(0)', 'gpu(1)', 'gpu(2)', 'gpu(3)']” when we are compiling the model. I am assuming there is no additional configuration needed.
Please let me know if you need more information.
Score screenshot sample:
