Speeding up training

[Sam-v6]

What does this PR do?

Improving the speed for hyper-parameter optimization with Ray Tune

What are some specific details on the changes?

• Speed changes for training
  • Using automatic mixed precision for forward passes with torch f16
  • Allowing TF32 for matrix multiplication
  • Increasing batch size to saturate GPU utilization (done with L4 GPU in mind for HPO)
• Miscellaneous
  • Converting from os.path.join to use Path
  • Minor fixes to address some linting warnings (more to do here)
  • Added tune for 1 trial into CI pipeline
  • Dispersing over better model params for Ray Tune and updated model best params and pickled model
  • Fixing model definition in predictions

How has this been tested?

Introduced additional CI stage to cover Ray Tune (runs only 1 trial at 5 epochs), meant to test configuration

Important Notes

None

Checklist

• [ X ] Tests have been added or updated to cover the changes.
• [ X ] Documentation has been updated (if necessary)