Optuna mode bug in integration with mlflow

[PeterCalifano]

Current optuna mode cannot run multiple jobs of the optimizer in parallel due to how mlflow attempts to log data for each run. Nested runs may be required to allow asynchronous parallel jobs. Error message causing the trials to fail:

[PeterCalifano]

Development note: this issue may be solved by improving mlflow calls by adding the reference to which run is being tracked by an instance of the trainer (i.e. using its ID or name). Moreover, the mlflow.end_run within startMLflow method should be eliminated. The trainer has to store the data required to log data for the run it is processing.

Activities to perform:

• Try to remove the end_run() call to check everything works fine if no run is "running"
• If current run is seen as "running" which would cause errors, force end_run call
• Investigate the requirements and consequences of Nested=true in mlflow.start_run and verify if it can be a solution
• Integrate mlflow-side solution with Optuna mode in trainer and test with 2 optuna jobs