The best model's RMSE score doesn't match ..

[kpalaka]

The best model's RMSE score doesn't match what's given by the validation data when scored by the best model.

I pass the following to the GameTrainingDriver:

 ...
 --input-data-directories "glmm_in/train_data.avro" \
 --validation-data-directories "glmm_in/test_data.avro" \
 ...

And, receive:

Best model has RMSE score of 10.212347636694373 and following config: ....   

I'm assuming that the best model's RMSE score is computed on the validation data provided.
Yet, when I score the validation data using the best model and compute the RMSE score, the result is quite different from the best score announced.

Is the best score indeed based on the validation data?
Is a non-standard definition of RMSE being used here?
What could I be doing wrong? (Btw, the difference is significant enough to matter.)

Regards.

[kpalaka]

I have since started using GameScoringDriver CLI to do my scoring. And, the RMSE given by the scoring driver matches the RMSE output by the training driver (for the best model). Everything seemed great.

Except, when you calculate the RMSE using the predictionScore and the label values, it's exactly what I was getting previously when I used the API !! And, it doesn't match what the game scoring driver and game training driver claim.

Has anyone else noticed this issue?