BUG Error in combining train & valid loader for Vision

[adwaykanhere]

When creating train+val loaders for tuning CNNs for vision data, the following traceback is received:

INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter lr. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.FLOAT for parameter momentum. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.INT for parameter epoch. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Inferred value type of ParameterType.STRING for parameter optimizer. If that is not the expected value type, you can explicity specify 'value_type' ('int', 'float', 'bool' or 'str') in parameter dict.
[INFO 05-12 21:06:37] ax.service.utils.instantiation: Created search space: SearchSpace(parameters=[RangeParameter(name='lr', parameter_type=FLOAT, range=[1e-06, 0.4], log_scale=True), RangeParameter(name='momentum', parameter_type=FLOAT, range=[0.0, 1.0]), RangeParameter(name='epoch', parameter_type=INT, range=[15, 40]), ChoiceParameter(name='optimizer', parameter_type=STRING, values=['SGD', 'Adam'], is_ordered=False, sort_values=False)], parameter_constraints=[]).
[INFO 05-12 21:06:37] ax.modelbridge.dispatch_utils: Using Bayesian optimization since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 05-12 21:06:37] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(name='Sobol+GPEI', steps=[Sobol for 8 trials, GPEI for subsequent trials]). Iterations after 8 will take longer to generate due to  model-fitting.
[INFO 05-12 21:06:37] ax.service.managed_loop: Started full optimization with 20 steps.
[INFO 05-12 21:06:37] ax.service.managed_loop: Running optimization trial 1...
[INFO 05-12 21:06:39] ax.service.managed_loop: Running optimization trial 2...
[INFO 05-12 21:06:40] ax.service.managed_loop: Running optimization trial 3...
[INFO 05-12 21:06:41] ax.service.managed_loop: Running optimization trial 4...
[INFO 05-12 21:06:43] ax.service.managed_loop: Running optimization trial 5...
[INFO 05-12 21:06:44] ax.service.managed_loop: Running optimization trial 6...
[INFO 05-12 21:06:45] ax.service.managed_loop: Running optimization trial 7...
[INFO 05-12 21:06:47] ax.service.managed_loop: Running optimization trial 8...
[INFO 05-12 21:06:48] ax.service.managed_loop: Running optimization trial 9...
[INFO 05-12 21:06:51] ax.service.managed_loop: Running optimization trial 10...
[INFO 05-12 21:06:52] ax.service.managed_loop: Running optimization trial 11...
[INFO 05-12 21:06:54] ax.service.managed_loop: Running optimization trial 12...
[INFO 05-12 21:06:56] ax.service.managed_loop: Running optimization trial 13...
[INFO 05-12 21:06:58] ax.service.managed_loop: Running optimization trial 14...
[INFO 05-12 21:07:00] ax.service.managed_loop: Running optimization trial 15...
[INFO 05-12 21:07:02] ax.service.managed_loop: Running optimization trial 16...
[INFO 05-12 21:07:04] ax.service.managed_loop: Running optimization trial 17...
[INFO 05-12 21:07:06] ax.service.managed_loop: Running optimization trial 18...
[INFO 05-12 21:07:08] ax.service.managed_loop: Running optimization trial 19...
[INFO 05-12 21:07:10] ax.service.managed_loop: Running optimization trial 20...
[INFO 05-12 21:07:10] ax.modelbridge.base: Untransformed parameter 0.40000000000000013 greater than upper bound 0.4, clamping
[WARNING 05-12 21:07:12] ax.modelbridge.cross_validation: Metric accuracy was unable to be reliably fit.
[WARNING 05-12 21:07:12] ax.service.utils.best_point: Model fit is poor; falling back on raw data for best point.
[WARNING 05-12 21:07:12] ax.service.utils.best_point: Model fit is poor and data on objective metric accuracy is noisy; interpret best points results carefully.
(array([], dtype=int64),)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-15-8f0b5b112073>](https://localhost:8080/#) in <module>()
     44 )
     45 
---> 46 run_cnn32()

[<ipython-input-14-5a5512f0afa6>](https://localhost:8080/#) in run_cnn32()
    369             for i in train_valid_indices:
    370               print(np.where(classes_new == dataset_copy.targets[i]))
--> 371               dataset_copy.targets[i] = np.where(classes_new == dataset_copy.targets[i])[0][0]
    372 
    373             train_valid_sampler = torch.utils.data.sampler.SubsetRandomSampler(train_valid_indices)

IndexError: index 0 is out of bounds for axis 0 with size 0

[PSSF23]

Was this commented out when you run create_train_loader?

  # for i in train_idxs:
  #    dataset.targets[i] = np.where(classes == dataset.targets[i])[0][0]

[adwaykanhere]

Yes, But that was because I had initially transformed train indices in create_loaders_final

[PSSF23]

You need to use deep copies of the dataset on that, too. There's no point of deep copies if you don't save the original. Then the copy becomes "original."

[adwaykanhere]

Yes, I'm using a deepcopy of the dataset for the create_train_loader and create_loaders_final. I'm still getting the error.

[PSSF23]

You have to create a deep copy each time you use those functions.

[adwaykanhere]

If I add deep copies before using each function, I get the error - IndexError: Target 7 is out of bounds.

[PSSF23]

Create deep copies inside the functions

[adwaykanhere]

I'm still getting the same error even after creating copies inside the functions