Share workers between epochs#627
Open
matham wants to merge 1 commit into
Open
Conversation
Contributor
|
Out of curiousity, have you have tried any experiments with prefetching? https://docs.pytorch.org/docs/2.12/data.html#torch.utils.data.DataLoader (prefetch_factor) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What is this PR
Why is this PR needed?
Currently, in each epoch of training pytorch ends the workers from the last epoch and spins up new worker processes for the current epoch. This results in a lot of timing overhead.
What does this PR do?
This changes so pytorch reuses the workers between epochs. By default pytorch doesn't do it, presumably because of potential memory leak issues in people's code, which if you're not careful about holding on to data, could lead worker's memory to grow infinitely. However, we are already very careful to not hold on to any memory that isn't already shared between workers because our files are so big. Therefore it's safe to re-use workers.
For a timing comparison, with main this is the log of a partial run that I stopped because it took too long:
Main
(brainglobe_test) PS C:\Users\CPLab> cellfinder_train -y "D:\training_data\561\20260121\MF1_336M_W\cuboids\training.yaml" "D:\training_data\561\20260204\MF1_354F_W\cuboids\training.yaml" "D:\training_data\561\20260203\MF1_359F_W\cuboids\training.yaml" "D:\training_data\561\20260203\MF1_351F_W\cuboids\training.yaml" "D:\training_data\561\20260204\MF1_353F_W\cuboids\training.yaml" "D:\training_data\561\20260121\MF1_334M_W\cuboids\training.yaml" "D:\training_data\561\20260122\MF1_342F_W\cuboids\training.yaml" "D:\training_data\561\20260122\MF1_340F_W\cuboids\training.yaml" "D:\training_data\640\20251016\MF1_268F_W\cuboids\training.yaml" "D:\training_data\640\20251015\MF1_272F_W\cuboids\training.yaml" "D:\training_data\640\20251015\MF1_271F_W\cuboids\training.yaml" "D:\training_data\561\20260205\MF1_355M_W\cuboids\training.yaml" "D:\training_data\561\20260123\MF1_346F_W\cuboids\training.yaml" "D:\training_data\640\20251016\MF1_273F_W\cuboids\training.yaml" "D:\training_data\640\20251012\MF1_262M_W\cuboids\training.yaml" "D:\training_data\561\20260209\MF1_339M_W\cuboids\training.yaml" "D:\training_data\561\20260208\MF1_333F_W\cuboids\training.yaml" "D:\training_data\561\20260207\MF1_347M_W\cuboids\training.yaml" "D:\training_data\640\20251011\MF1_261M_W\cuboids\training.yaml" "D:\training_data\640\20251017\MF1_274M_W\cuboids\training.yaml" "D:\training_data\640\20251017\MF1_269M_W\cuboids\training.yaml" -o "D:\models_aug90_main" --network-depth 50 --batch-size 32 --save-progress --test-fraction 0.1 --model resnet50_tv --max-workers 6 --lr-multiplier 0.5 --learning-rate 0.001 --lr-schedule 130 155 175 --augment-likelihood 90 --epochs 190 --normalize-channels --continue-training 2026-06-20 03:50:48 AM INFO 2026-06-20 03:50:48 AM - INFO - MainProcess fancylog.py:609 - Starting logging fancylog.py:609 INFO 2026-06-20 03:50:48 AM - INFO - MainProcess fancylog.py:610 - Not logging multiple processes fancylog.py:610 DEBUG 2026-06-20 03:50:48 AM - DEBUG - MainProcess prep.py:42 - No model supplied, so using the default prep.py:42 DEBUG 2026-06-20 03:50:48 AM - DEBUG - MainProcess prep.py:67 - Reading config file: prep.py:67 C:\Users\CPLab\.brainglobe\cellfinder\cellfinder.conf.custom INFO 2026-06-20 03:50:48 AM - INFO - MainProcess train_yaml.py:511 - Found 8642 images from 41 datasets in 21 train_yaml.py:511 yaml files DEBUG 2026-06-20 03:50:48 AM - DEBUG - MainProcess tools.py:38 - Creating a new instance of model: 50-layer tools.py:38 2026-06-20 03:50:49 AM DEBUG 2026-06-20 03:50:49 AM - DEBUG - MainProcess tools.py:44 - Setting model weights according to: tools.py:44 C:\Users\CPLab\.brainglobe\cellfinder\models\resnet50_tv.h5 DEBUG 2026-06-20 03:50:49 AM - DEBUG - MainProcess attrs.py:77 - Creating converter from 3 to 5 attrs.py:77 DEBUG 2026-06-20 03:50:49 AM - DEBUG - MainProcess system.py:231 - Determining the maximum number of CPU cores to system.py:231 use DEBUG 2026-06-20 03:50:49 AM - DEBUG - MainProcess system.py:236 - Number of CPU cores available is: 70 system.py:236 DEBUG 2026-06-20 03:50:49 AM - DEBUG - MainProcess system.py:263 - Setting number of processes to: 70 system.py:263 2026-06-20 03:50:50 AM INFO 2026-06-20 03:50:50 AM - INFO - MainProcess train_yaml.py:530 - Splitting data into training and train_yaml.py:530 validation datasets INFO 2026-06-20 03:50:50 AM - INFO - MainProcess train_yaml.py:542 - Using 7777 images for training and 865 train_yaml.py:542 images for validation INFO 2026-06-20 03:50:50 AM - INFO - MainProcess train_yaml.py:622 - Beginning training. train_yaml.py:622 Epoch 1/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 0s 528ms/step - accuracy: 0.5378 - loss: 0.90732026-06-20 03:57:21 AM DEBUG 2026-06-20 03:57:21 AM - DEBUG - MainProcess attrs.py:204 - Creating converter from 5 to 3 attrs.py:204 244/244 ━━━━━━━━━━━━━━━━━━━━ 342s 1s/step - accuracy: 0.5541 - loss: 0.8562 - val_accuracy: 0.4890 - val_loss: 0.6949 - learning_rate: 0.0010 Epoch 2/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 909ms/step - accuracy: 0.6960 - loss: 0.7093 - val_accuracy: 0.5179 - val_loss: 0.6912 - learning_rate: 0.0010 Epoch 3/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.6868 - loss: 0.8721 - val_accuracy: 0.5908 - val_loss: 0.6660 - learning_rate: 0.0010 Epoch 4/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.6977 - loss: 0.7870 - val_accuracy: 0.6324 - val_loss: 0.6703 - learning_rate: 0.0010 Epoch 5/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.6402 - loss: 0.8019 - val_accuracy: 0.7006 - val_loss: 0.6701 - learning_rate: 0.0010 Epoch 6/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.7160 - loss: 0.5921 - val_accuracy: 0.6347 - val_loss: 0.7911 - learning_rate: 0.0010 Epoch 7/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.7241 - loss: 0.5739 - val_accuracy: 0.7121 - val_loss: 0.5701 - learning_rate: 0.0010 Epoch 8/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.7527 - loss: 0.5458 - val_accuracy: 0.7306 - val_loss: 0.5943 - learning_rate: 0.0010 Epoch 9/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 913ms/step - accuracy: 0.7442 - loss: 0.5530 - val_accuracy: 0.7376 - val_loss: 0.5456 - learning_rate: 0.0010 Epoch 10/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.7694 - loss: 0.5109 - val_accuracy: 0.7480 - val_loss: 0.5328 - learning_rate: 0.0010 Epoch 11/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 916ms/step - accuracy: 0.7781 - loss: 0.4977 - val_accuracy: 0.7480 - val_loss: 0.5254 - learning_rate: 0.0010 Epoch 12/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.7845 - loss: 0.4683 - val_accuracy: 0.7584 - val_loss: 0.5137 - learning_rate: 0.0010 Epoch 13/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 914ms/step - accuracy: 0.7851 - loss: 0.4612 - val_accuracy: 0.7526 - val_loss: 0.5195 - learning_rate: 0.0010 Epoch 14/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 908ms/step - accuracy: 0.7904 - loss: 0.4572 - val_accuracy: 0.7665 - val_loss: 0.4919 - learning_rate: 0.0010 Epoch 15/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 908ms/step - accuracy: 0.7939 - loss: 0.4487 - val_accuracy: 0.7665 - val_loss: 0.5500 - learning_rate: 0.0010 Epoch 16/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.7940 - loss: 0.4470 - val_accuracy: 0.7699 - val_loss: 0.4969 - learning_rate: 0.0010 Epoch 17/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 908ms/step - accuracy: 0.8007 - loss: 0.4406 - val_accuracy: 0.7884 - val_loss: 0.4577 - learning_rate: 0.0010 Epoch 18/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 914ms/step - accuracy: 0.7998 - loss: 0.4322 - val_accuracy: 0.7815 - val_loss: 0.4908 - learning_rate: 0.0010 Epoch 19/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 268s 907ms/step - accuracy: 0.8092 - loss: 0.4274 - val_accuracy: 0.7838 - val_loss: 0.5494 - learning_rate: 0.0010 Epoch 20/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 911ms/step - accuracy: 0.8128 - loss: 0.4226 - val_accuracy: 0.7977 - val_loss: 0.4484 - learning_rate: 0.0010 Epoch 21/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8002 - loss: 0.4339 - val_accuracy: 0.7954 - val_loss: 0.4766 - learning_rate: 0.0010 Epoch 22/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8097 - loss: 0.4183 - val_accuracy: 0.7769 - val_loss: 0.4854 - learning_rate: 0.0010 Epoch 23/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.8166 - loss: 0.4115 - val_accuracy: 0.7884 - val_loss: 0.4686 - learning_rate: 0.0010 Epoch 24/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8170 - loss: 0.4071 - val_accuracy: 0.7642 - val_loss: 0.4746 - learning_rate: 0.0010 Epoch 25/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.8209 - loss: 0.3963 - val_accuracy: 0.8127 - val_loss: 0.4143 - learning_rate: 0.0010 Epoch 26/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.8253 - loss: 0.3919 - val_accuracy: 0.7514 - val_loss: 0.5290 - learning_rate: 0.0010 Epoch 27/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8096 - loss: 0.4167 - val_accuracy: 0.8243 - val_loss: 0.3992 - learning_rate: 0.0010 Epoch 28/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8295 - loss: 0.3876 - val_accuracy: 0.8185 - val_loss: 0.4092 - learning_rate: 0.0010 Epoch 29/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.8300 - loss: 0.3830 - val_accuracy: 0.8208 - val_loss: 0.4046 - learning_rate: 0.0010 Epoch 30/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 917ms/step - accuracy: 0.8184 - loss: 0.4069 - val_accuracy: 0.7850 - val_loss: 0.5074 - learning_rate: 0.0010 Epoch 31/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8413 - loss: 0.3723 - val_accuracy: 0.8197 - val_loss: 0.3987 - learning_rate: 0.0010 Epoch 32/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 911ms/step - accuracy: 0.8010 - loss: 0.4343 - val_accuracy: 0.7977 - val_loss: 0.5003 - learning_rate: 0.0010 Epoch 33/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 910ms/step - accuracy: 0.8262 - loss: 0.3910 - val_accuracy: 0.8243 - val_loss: 0.3973 - learning_rate: 0.0010 Epoch 34/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 268s 907ms/step - accuracy: 0.8404 - loss: 0.3683 - val_accuracy: 0.8289 - val_loss: 0.3806 - learning_rate: 0.0010 Epoch 35/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 909ms/step - accuracy: 0.8393 - loss: 0.3647 - val_accuracy: 0.8324 - val_loss: 0.3768 - learning_rate: 0.0010 Epoch 36/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8439 - loss: 0.3596 - val_accuracy: 0.8185 - val_loss: 0.3971 - learning_rate: 0.0010 Epoch 37/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 909ms/step - accuracy: 0.8480 - loss: 0.3485 - val_accuracy: 0.8532 - val_loss: 0.3607 - learning_rate: 0.0010 Epoch 38/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 917ms/step - accuracy: 0.8498 - loss: 0.3480 - val_accuracy: 0.8462 - val_loss: 0.3643 - learning_rate: 0.0010 Epoch 39/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8538 - loss: 0.3409 - val_accuracy: 0.7757 - val_loss: 0.5213 - learning_rate: 0.0010 Epoch 40/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8183 - loss: 0.4064 - val_accuracy: 0.8208 - val_loss: 0.4372 - learning_rate: 0.0010 Epoch 41/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 909ms/step - accuracy: 0.8420 - loss: 0.3651 - val_accuracy: 0.8405 - val_loss: 0.3615 - learning_rate: 0.0010 Epoch 42/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8470 - loss: 0.3538 - val_accuracy: 0.8416 - val_loss: 0.3925 - learning_rate: 0.0010 Epoch 43/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.8542 - loss: 0.3430 - val_accuracy: 0.8566 - val_loss: 0.3317 - learning_rate: 0.0010 Epoch 44/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8550 - loss: 0.3334 - val_accuracy: 0.8509 - val_loss: 0.3621 - learning_rate: 0.0010 Epoch 45/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 918ms/step - accuracy: 0.8566 - loss: 0.3361 - val_accuracy: 0.8058 - val_loss: 0.4883 - learning_rate: 0.0010 Epoch 46/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8343 - loss: 0.3831 - val_accuracy: 0.8370 - val_loss: 0.3879 - learning_rate: 0.0010 Epoch 47/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8503 - loss: 0.3471 - val_accuracy: 0.8486 - val_loss: 0.3608 - learning_rate: 0.0010 Epoch 48/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8577 - loss: 0.3354 - val_accuracy: 0.7514 - val_loss: 0.5398 - learning_rate: 0.0010 Epoch 49/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 917ms/step - accuracy: 0.8574 - loss: 0.3391 - val_accuracy: 0.8335 - val_loss: 0.3804 - learning_rate: 0.0010 Epoch 50/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8658 - loss: 0.3280 - val_accuracy: 0.8416 - val_loss: 0.3541 - learning_rate: 0.0010 Epoch 51/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.8533 - loss: 0.3377 - val_accuracy: 0.8497 - val_loss: 0.3433 - learning_rate: 0.0010 Epoch 52/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 916ms/step - accuracy: 0.8628 - loss: 0.3208 - val_accuracy: 0.8520 - val_loss: 0.3515 - learning_rate: 0.0010 Epoch 53/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 917ms/step - accuracy: 0.8659 - loss: 0.3208 - val_accuracy: 0.8543 - val_loss: 0.3331 - learning_rate: 0.0010 Epoch 54/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8659 - loss: 0.3186 - val_accuracy: 0.8382 - val_loss: 0.3708 - learning_rate: 0.0010 Epoch 55/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 913ms/step - accuracy: 0.8685 - loss: 0.3150 - val_accuracy: 0.8601 - val_loss: 0.3277 - learning_rate: 0.0010 Epoch 56/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8622 - loss: 0.3205 - val_accuracy: 0.8728 - val_loss: 0.3139 - learning_rate: 0.0010 Epoch 57/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 274s 924ms/step - accuracy: 0.8697 - loss: 0.3132 - val_accuracy: 0.8647 - val_loss: 0.3318 - learning_rate: 0.0010 Epoch 58/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 917ms/step - accuracy: 0.8667 - loss: 0.3163 - val_accuracy: 0.8578 - val_loss: 0.3215 - learning_rate: 0.0010 Epoch 59/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 919ms/step - accuracy: 0.8553 - loss: 0.3339 - val_accuracy: 0.8659 - val_loss: 0.3182 - learning_rate: 0.0010 Epoch 60/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 916ms/step - accuracy: 0.8685 - loss: 0.3150 - val_accuracy: 0.8590 - val_loss: 0.3232 - learning_rate: 0.0010 Epoch 61/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 273s 922ms/step - accuracy: 0.8660 - loss: 0.3109 - val_accuracy: 0.8590 - val_loss: 0.3175 - learning_rate: 0.0010 Epoch 62/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 912ms/step - accuracy: 0.8708 - loss: 0.3071 - val_accuracy: 0.8613 - val_loss: 0.3173 - learning_rate: 0.0010 Epoch 63/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 918ms/step - accuracy: 0.8754 - loss: 0.3009 - val_accuracy: 0.8555 - val_loss: 0.3130 - learning_rate: 0.0010 Epoch 64/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 269s 909ms/step - accuracy: 0.8593 - loss: 0.3310 - val_accuracy: 0.8324 - val_loss: 0.3747 - learning_rate: 0.0010 Epoch 65/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8649 - loss: 0.3155 - val_accuracy: 0.8717 - val_loss: 0.3109 - learning_rate: 0.0010 Epoch 66/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 917ms/step - accuracy: 0.8708 - loss: 0.3038 - val_accuracy: 0.8324 - val_loss: 0.3710 - learning_rate: 0.0010 Epoch 67/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 914ms/step - accuracy: 0.8718 - loss: 0.3054 - val_accuracy: 0.8728 - val_loss: 0.3088 - learning_rate: 0.0010 Epoch 68/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8717 - loss: 0.3016 - val_accuracy: 0.8613 - val_loss: 0.3190 - learning_rate: 0.0010 Epoch 69/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 916ms/step - accuracy: 0.8651 - loss: 0.3123 - val_accuracy: 0.8555 - val_loss: 0.3308 - learning_rate: 0.0010 Epoch 70/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 914ms/step - accuracy: 0.8727 - loss: 0.3014 - val_accuracy: 0.8775 - val_loss: 0.2996 - learning_rate: 0.0010 Epoch 71/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 915ms/step - accuracy: 0.8719 - loss: 0.3018 - val_accuracy: 0.8786 - val_loss: 0.3106 - learning_rate: 0.0010 Epoch 72/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8718 - loss: 0.3033 - val_accuracy: 0.8740 - val_loss: 0.3112 - learning_rate: 0.0010 Epoch 73/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 911ms/step - accuracy: 0.8746 - loss: 0.3026 - val_accuracy: 0.8555 - val_loss: 0.3339 - learning_rate: 0.0010 Epoch 74/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 271s 916ms/step - accuracy: 0.8650 - loss: 0.3156 - val_accuracy: 0.8740 - val_loss: 0.3079 - learning_rate: 0.0010 Epoch 75/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 270s 912ms/step - accuracy: 0.8723 - loss: 0.3009 - val_accuracy: 0.8393 - val_loss: 0.3628 - learning_rate: 0.0010 Epoch 76/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 274s 923ms/step - accuracy: 0.8706 - loss: 0.2971 - val_accuracy: 0.8740 - val_loss: 0.2964 - learning_rate: 0.0010 Epoch 77/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 272s 919ms/step - accuracy: 0.8799 - loss: 0.2886 - val_accuracy: 0.8647 - val_loss: 0.3177 - learning_rate: 0.0010 Epoch 78/190Vs running with this single change:
Branch
(brainglobe_test) PS C:\Users\CPLab> cellfinder_train -y "D:\training_data\561\20260121\MF1_336M_W\cuboids\training.yaml" "D:\training_data\561\20260204\MF1_354F_W\cuboids\training.yaml" "D:\training_data\561\20260203\MF1_359F_W\cuboids\training.yaml" "D:\training_data\561\20260203\MF1_351F_W\cuboids\training.yaml" "D:\training_data\561\20260204\MF1_353F_W\cuboids\training.yaml" "D:\training_data\561\20260121\MF1_334M_W\cuboids\training.yaml" "D:\training_data\561\20260122\MF1_342F_W\cuboids\training.yaml" "D:\training_data\561\20260122\MF1_340F_W\cuboids\training.yaml" "D:\training_data\640\20251016\MF1_268F_W\cuboids\training.yaml" "D:\training_data\640\20251015\MF1_272F_W\cuboids\training.yaml" "D:\training_data\640\20251015\MF1_271F_W\cuboids\training.yaml" "D:\training_data\561\20260205\MF1_355M_W\cuboids\training.yaml" "D:\training_data\561\20260123\MF1_346F_W\cuboids\training.yaml" "D:\training_data\640\20251016\MF1_273F_W\cuboids\training.yaml" "D:\training_data\640\20251012\MF1_262M_W\cuboids\training.yaml" "D:\training_data\561\20260209\MF1_339M_W\cuboids\training.yaml" "D:\training_data\561\20260208\MF1_333F_W\cuboids\training.yaml" "D:\training_data\561\20260207\MF1_347M_W\cuboids\training.yaml" "D:\training_data\640\20251011\MF1_261M_W\cuboids\training.yaml" "D:\training_data\640\20251017\MF1_274M_W\cuboids\training.yaml" "D:\training_data\640\20251017\MF1_269M_W\cuboids\training.yaml" -o "D:\models_aug90_main" --network-depth 50 --batch-size 32 --save-progress --test-fraction 0.1 --model resnet50_tv --max-workers 6 --lr-multiplier 0.5 --learning-rate 0.001 --lr-schedule 130 155 175 --augment-likelihood 90 --epochs 190 --normalize-channels --continue-training 2026-06-20 09:42:51 AM INFO 2026-06-20 09:42:51 AM - INFO - MainProcess fancylog.py:609 - Starting logging fancylog.py:609 INFO 2026-06-20 09:42:51 AM - INFO - MainProcess fancylog.py:610 - Not logging multiple processes fancylog.py:610 DEBUG 2026-06-20 09:42:51 AM - DEBUG - MainProcess prep.py:42 - No model supplied, so using the default prep.py:42 DEBUG 2026-06-20 09:42:51 AM - DEBUG - MainProcess prep.py:67 - Reading config file: prep.py:67 C:\Users\CPLab\.brainglobe\cellfinder\cellfinder.conf.custom 2026-06-20 09:42:52 AM INFO 2026-06-20 09:42:52 AM - INFO - MainProcess train_yaml.py:512 - Found 8642 images from 41 datasets in 21 train_yaml.py:512 yaml files DEBUG 2026-06-20 09:42:52 AM - DEBUG - MainProcess tools.py:38 - Creating a new instance of model: 50-layer tools.py:38 2026-06-20 09:42:53 AM DEBUG 2026-06-20 09:42:53 AM - DEBUG - MainProcess tools.py:44 - Setting model weights according to: tools.py:44 C:\Users\CPLab\.brainglobe\cellfinder\models\resnet50_tv.h5 DEBUG 2026-06-20 09:42:53 AM - DEBUG - MainProcess attrs.py:77 - Creating converter from 3 to 5 attrs.py:77 DEBUG 2026-06-20 09:42:53 AM - DEBUG - MainProcess system.py:231 - Determining the maximum number of CPU cores to system.py:231 use DEBUG 2026-06-20 09:42:53 AM - DEBUG - MainProcess system.py:236 - Number of CPU cores available is: 70 system.py:236 DEBUG 2026-06-20 09:42:53 AM - DEBUG - MainProcess system.py:263 - Setting number of processes to: 70 system.py:263 INFO 2026-06-20 09:42:53 AM - INFO - MainProcess train_yaml.py:531 - Splitting data into training and train_yaml.py:531 validation datasets INFO 2026-06-20 09:42:53 AM - INFO - MainProcess train_yaml.py:543 - Using 7777 images for training and 865 train_yaml.py:543 images for validation INFO 2026-06-20 09:42:53 AM - INFO - MainProcess train_yaml.py:623 - Beginning training. train_yaml.py:623 Epoch 1/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 0s 276ms/step - accuracy: 0.5302 - loss: 1.12242026-06-20 09:45:44 AM DEBUG 2026-06-20 09:45:44 AM - DEBUG - MainProcess attrs.py:204 - Creating converter from 5 to 3 attrs.py:204 244/244 ━━━━━━━━━━━━━━━━━━━━ 127s 508ms/step - accuracy: 0.5101 - loss: 1.0526 - val_accuracy: 0.4983 - val_loss: 0.6932 - learning_rate: 0.0010 Epoch 2/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 316ms/step - accuracy: 0.5356 - loss: 0.8479 - val_accuracy: 0.4983 - val_loss: 0.6933 - learning_rate: 0.0010 Epoch 3/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.6220 - loss: 0.6416 - val_accuracy: 0.5387 - val_loss: 0.6905 - learning_rate: 0.0010 Epoch 4/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.7419 - loss: 0.5644 - val_accuracy: 0.7457 - val_loss: 0.5356 - learning_rate: 0.0010 Epoch 5/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.7712 - loss: 0.4789 - val_accuracy: 0.7584 - val_loss: 0.5036 - learning_rate: 0.0010 Epoch 6/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.7889 - loss: 0.4563 - val_accuracy: 0.7757 - val_loss: 0.4721 - learning_rate: 0.0010 Epoch 7/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.7963 - loss: 0.4437 - val_accuracy: 0.7908 - val_loss: 0.5116 - learning_rate: 0.0010 Epoch 8/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.7997 - loss: 0.4384 - val_accuracy: 0.7838 - val_loss: 0.4770 - learning_rate: 0.0010 Epoch 9/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 316ms/step - accuracy: 0.8033 - loss: 0.4418 - val_accuracy: 0.7919 - val_loss: 0.4454 - learning_rate: 0.0010 Epoch 10/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8132 - loss: 0.4137 - val_accuracy: 0.7965 - val_loss: 0.4284 - learning_rate: 0.0010 Epoch 11/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8152 - loss: 0.4050 - val_accuracy: 0.8046 - val_loss: 0.4800 - learning_rate: 0.0010 Epoch 12/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8155 - loss: 0.4114 - val_accuracy: 0.8243 - val_loss: 0.3802 - learning_rate: 0.0010 Epoch 13/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8242 - loss: 0.3992 - val_accuracy: 0.8324 - val_loss: 0.4083 - learning_rate: 0.0010 Epoch 14/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8292 - loss: 0.3836 - val_accuracy: 0.8439 - val_loss: 0.3634 - learning_rate: 0.0010 Epoch 15/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 316ms/step - accuracy: 0.8301 - loss: 0.3892 - val_accuracy: 0.8289 - val_loss: 0.3755 - learning_rate: 0.0010 Epoch 16/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8359 - loss: 0.3782 - val_accuracy: 0.8289 - val_loss: 0.3765 - learning_rate: 0.0010 Epoch 17/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8341 - loss: 0.3734 - val_accuracy: 0.8370 - val_loss: 0.3601 - learning_rate: 0.0010 Epoch 18/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8409 - loss: 0.3655 - val_accuracy: 0.8532 - val_loss: 0.3421 - learning_rate: 0.0010 Epoch 19/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8388 - loss: 0.3649 - val_accuracy: 0.8370 - val_loss: 0.3656 - learning_rate: 0.0010 Epoch 20/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8480 - loss: 0.3487 - val_accuracy: 0.8254 - val_loss: 0.3522 - learning_rate: 0.0010 Epoch 21/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8400 - loss: 0.3605 - val_accuracy: 0.8266 - val_loss: 0.3842 - learning_rate: 0.0010 Epoch 22/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8440 - loss: 0.3499 - val_accuracy: 0.7919 - val_loss: 0.4327 - learning_rate: 0.0010 Epoch 23/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8523 - loss: 0.3420 - val_accuracy: 0.7676 - val_loss: 0.5858 - learning_rate: 0.0010 Epoch 24/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8537 - loss: 0.3370 - val_accuracy: 0.8647 - val_loss: 0.3213 - learning_rate: 0.0010 Epoch 25/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8487 - loss: 0.3365 - val_accuracy: 0.8324 - val_loss: 0.3537 - learning_rate: 0.0010 Epoch 26/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8592 - loss: 0.3304 - val_accuracy: 0.8671 - val_loss: 0.3015 - learning_rate: 0.0010 Epoch 27/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8577 - loss: 0.3302 - val_accuracy: 0.8786 - val_loss: 0.2923 - learning_rate: 0.0010 Epoch 28/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8550 - loss: 0.3321 - val_accuracy: 0.8601 - val_loss: 0.3254 - learning_rate: 0.0010 Epoch 29/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8604 - loss: 0.3209 - val_accuracy: 0.8578 - val_loss: 0.3081 - learning_rate: 0.0010 Epoch 30/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8611 - loss: 0.3178 - val_accuracy: 0.8728 - val_loss: 0.3059 - learning_rate: 0.0010 Epoch 31/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8614 - loss: 0.3156 - val_accuracy: 0.8682 - val_loss: 0.3050 - learning_rate: 0.0010 Epoch 32/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8668 - loss: 0.3250 - val_accuracy: 0.8636 - val_loss: 0.3285 - learning_rate: 0.0010 Epoch 33/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8634 - loss: 0.3172 - val_accuracy: 0.8451 - val_loss: 0.3915 - learning_rate: 0.0010 Epoch 34/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8614 - loss: 0.3193 - val_accuracy: 0.8613 - val_loss: 0.3147 - learning_rate: 0.0010 Epoch 35/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8688 - loss: 0.3115 - val_accuracy: 0.8844 - val_loss: 0.3085 - learning_rate: 0.0010 Epoch 36/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8629 - loss: 0.3177 - val_accuracy: 0.8809 - val_loss: 0.2930 - learning_rate: 0.0010 Epoch 37/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8710 - loss: 0.3068 - val_accuracy: 0.8763 - val_loss: 0.2969 - learning_rate: 0.0010 Epoch 38/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8676 - loss: 0.3120 - val_accuracy: 0.8486 - val_loss: 0.3535 - learning_rate: 0.0010 Epoch 39/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8656 - loss: 0.3115 - val_accuracy: 0.8855 - val_loss: 0.2919 - learning_rate: 0.0010 Epoch 40/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8715 - loss: 0.3062 - val_accuracy: 0.8913 - val_loss: 0.2747 - learning_rate: 0.0010 Epoch 41/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8708 - loss: 0.3039 - val_accuracy: 0.8751 - val_loss: 0.2848 - learning_rate: 0.0010 Epoch 42/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8704 - loss: 0.3026 - val_accuracy: 0.8855 - val_loss: 0.2935 - learning_rate: 0.0010 Epoch 43/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8755 - loss: 0.2975 - val_accuracy: 0.8879 - val_loss: 0.2961 - learning_rate: 0.0010 Epoch 44/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8704 - loss: 0.2959 - val_accuracy: 0.8590 - val_loss: 0.3402 - learning_rate: 0.0010 Epoch 45/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8709 - loss: 0.3019 - val_accuracy: 0.8786 - val_loss: 0.2839 - learning_rate: 0.0010 Epoch 46/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8750 - loss: 0.2986 - val_accuracy: 0.8671 - val_loss: 0.3265 - learning_rate: 0.0010 Epoch 47/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8737 - loss: 0.2979 - val_accuracy: 0.8763 - val_loss: 0.2866 - learning_rate: 0.0010 Epoch 48/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8772 - loss: 0.2893 - val_accuracy: 0.8543 - val_loss: 0.3583 - learning_rate: 0.0010 Epoch 49/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8767 - loss: 0.2925 - val_accuracy: 0.8705 - val_loss: 0.3216 - learning_rate: 0.0010 Epoch 50/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8784 - loss: 0.2934 - val_accuracy: 0.8890 - val_loss: 0.2855 - learning_rate: 0.0010 Epoch 51/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 322ms/step - accuracy: 0.8775 - loss: 0.2896 - val_accuracy: 0.8717 - val_loss: 0.3100 - learning_rate: 0.0010 Epoch 52/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8778 - loss: 0.2952 - val_accuracy: 0.8832 - val_loss: 0.2726 - learning_rate: 0.0010 Epoch 53/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8798 - loss: 0.2845 - val_accuracy: 0.8913 - val_loss: 0.2930 - learning_rate: 0.0010 Epoch 54/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8794 - loss: 0.2864 - val_accuracy: 0.8902 - val_loss: 0.2877 - learning_rate: 0.0010 Epoch 55/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8781 - loss: 0.2972 - val_accuracy: 0.8659 - val_loss: 0.3087 - learning_rate: 0.0010 Epoch 56/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.8746 - loss: 0.2979 - val_accuracy: 0.8821 - val_loss: 0.2841 - learning_rate: 0.0010 Epoch 57/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8799 - loss: 0.2894 - val_accuracy: 0.8867 - val_loss: 0.2768 - learning_rate: 0.0010 Epoch 58/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8802 - loss: 0.2854 - val_accuracy: 0.8162 - val_loss: 0.4278 - learning_rate: 0.0010 Epoch 59/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8799 - loss: 0.2832 - val_accuracy: 0.8624 - val_loss: 0.3162 - learning_rate: 0.0010 Epoch 60/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8825 - loss: 0.2891 - val_accuracy: 0.8358 - val_loss: 0.3700 - learning_rate: 0.0010 Epoch 61/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8818 - loss: 0.2814 - val_accuracy: 0.8867 - val_loss: 0.2813 - learning_rate: 0.0010 Epoch 62/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8830 - loss: 0.2808 - val_accuracy: 0.8867 - val_loss: 0.2764 - learning_rate: 0.0010 Epoch 63/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 322ms/step - accuracy: 0.8831 - loss: 0.2839 - val_accuracy: 0.8971 - val_loss: 0.2531 - learning_rate: 0.0010 Epoch 64/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.8813 - loss: 0.2869 - val_accuracy: 0.8844 - val_loss: 0.2739 - learning_rate: 0.0010 Epoch 65/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8825 - loss: 0.2783 - val_accuracy: 0.8844 - val_loss: 0.2882 - learning_rate: 0.0010 Epoch 66/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8786 - loss: 0.2879 - val_accuracy: 0.8902 - val_loss: 0.2752 - learning_rate: 0.0010 Epoch 67/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8835 - loss: 0.2833 - val_accuracy: 0.8832 - val_loss: 0.2893 - learning_rate: 0.0010 Epoch 68/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8839 - loss: 0.2815 - val_accuracy: 0.8855 - val_loss: 0.2888 - learning_rate: 0.0010 Epoch 69/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8809 - loss: 0.2819 - val_accuracy: 0.8832 - val_loss: 0.3089 - learning_rate: 0.0010 Epoch 70/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8795 - loss: 0.2810 - val_accuracy: 0.8451 - val_loss: 0.3292 - learning_rate: 0.0010 Epoch 71/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8775 - loss: 0.2920 - val_accuracy: 0.8543 - val_loss: 0.3484 - learning_rate: 0.0010 Epoch 72/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8823 - loss: 0.2776 - val_accuracy: 0.8936 - val_loss: 0.2958 - learning_rate: 0.0010 Epoch 73/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8812 - loss: 0.2737 - val_accuracy: 0.8960 - val_loss: 0.2676 - learning_rate: 0.0010 Epoch 74/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8861 - loss: 0.2773 - val_accuracy: 0.8902 - val_loss: 0.2707 - learning_rate: 0.0010 Epoch 75/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8834 - loss: 0.2777 - val_accuracy: 0.8902 - val_loss: 0.2740 - learning_rate: 0.0010 Epoch 76/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8875 - loss: 0.2725 - val_accuracy: 0.8717 - val_loss: 0.3133 - learning_rate: 0.0010 Epoch 77/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8690 - loss: 0.3096 - val_accuracy: 0.8624 - val_loss: 0.3353 - learning_rate: 0.0010 Epoch 78/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8771 - loss: 0.2859 - val_accuracy: 0.8809 - val_loss: 0.2943 - learning_rate: 0.0010 Epoch 79/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8863 - loss: 0.2748 - val_accuracy: 0.8624 - val_loss: 0.3393 - learning_rate: 0.0010 Epoch 80/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8710 - loss: 0.3061 - val_accuracy: 0.8717 - val_loss: 0.3253 - learning_rate: 0.0010 Epoch 81/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8825 - loss: 0.2816 - val_accuracy: 0.8913 - val_loss: 0.2791 - learning_rate: 0.0010 Epoch 82/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8807 - loss: 0.2801 - val_accuracy: 0.8890 - val_loss: 0.2790 - learning_rate: 0.0010 Epoch 83/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8859 - loss: 0.2757 - val_accuracy: 0.8867 - val_loss: 0.2839 - learning_rate: 0.0010 Epoch 84/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8916 - loss: 0.2654 - val_accuracy: 0.8740 - val_loss: 0.2979 - learning_rate: 0.0010 Epoch 85/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8868 - loss: 0.2726 - val_accuracy: 0.8405 - val_loss: 0.3807 - learning_rate: 0.0010 Epoch 86/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8874 - loss: 0.2739 - val_accuracy: 0.8960 - val_loss: 0.2645 - learning_rate: 0.0010 Epoch 87/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8888 - loss: 0.2656 - val_accuracy: 0.8162 - val_loss: 0.4405 - learning_rate: 0.0010 Epoch 88/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8885 - loss: 0.2651 - val_accuracy: 0.9017 - val_loss: 0.2685 - learning_rate: 0.0010 Epoch 89/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 323ms/step - accuracy: 0.8825 - loss: 0.2810 - val_accuracy: 0.9017 - val_loss: 0.2562 - learning_rate: 0.0010 Epoch 90/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8866 - loss: 0.2686 - val_accuracy: 0.8855 - val_loss: 0.2787 - learning_rate: 0.0010 Epoch 91/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8894 - loss: 0.2656 - val_accuracy: 0.8717 - val_loss: 0.3032 - learning_rate: 0.0010 Epoch 92/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8877 - loss: 0.2699 - val_accuracy: 0.8786 - val_loss: 0.3061 - learning_rate: 0.0010 Epoch 93/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8845 - loss: 0.2702 - val_accuracy: 0.8382 - val_loss: 0.3679 - learning_rate: 0.0010 Epoch 94/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8908 - loss: 0.2619 - val_accuracy: 0.8775 - val_loss: 0.3006 - learning_rate: 0.0010 Epoch 95/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.8889 - loss: 0.2688 - val_accuracy: 0.8913 - val_loss: 0.2855 - learning_rate: 0.0010 Epoch 96/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8826 - loss: 0.2775 - val_accuracy: 0.8960 - val_loss: 0.2669 - learning_rate: 0.0010 Epoch 97/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 323ms/step - accuracy: 0.8899 - loss: 0.2654 - val_accuracy: 0.8601 - val_loss: 0.3131 - learning_rate: 0.0010 Epoch 98/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8870 - loss: 0.2659 - val_accuracy: 0.8786 - val_loss: 0.3013 - learning_rate: 0.0010 Epoch 99/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8879 - loss: 0.2662 - val_accuracy: 0.8728 - val_loss: 0.3115 - learning_rate: 0.0010 Epoch 100/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8857 - loss: 0.2649 - val_accuracy: 0.8659 - val_loss: 0.3097 - learning_rate: 0.0010 Epoch 101/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 324ms/step - accuracy: 0.8906 - loss: 0.2638 - val_accuracy: 0.8428 - val_loss: 0.4026 - learning_rate: 0.0010 Epoch 102/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 127s 517ms/step - accuracy: 0.8911 - loss: 0.2633 - val_accuracy: 0.9052 - val_loss: 0.2611 - learning_rate: 0.0010 Epoch 103/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8889 - loss: 0.2634 - val_accuracy: 0.8601 - val_loss: 0.3428 - learning_rate: 0.0010 Epoch 104/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8911 - loss: 0.2634 - val_accuracy: 0.8925 - val_loss: 0.2569 - learning_rate: 0.0010 Epoch 105/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.8839 - loss: 0.2725 - val_accuracy: 0.8786 - val_loss: 0.2862 - learning_rate: 0.0010 Epoch 106/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8895 - loss: 0.2603 - val_accuracy: 0.9075 - val_loss: 0.2505 - learning_rate: 0.0010 Epoch 107/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8830 - loss: 0.2808 - val_accuracy: 0.8902 - val_loss: 0.2664 - learning_rate: 0.0010 Epoch 108/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8892 - loss: 0.2671 - val_accuracy: 0.8925 - val_loss: 0.2524 - learning_rate: 0.0010 Epoch 109/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8934 - loss: 0.2570 - val_accuracy: 0.9029 - val_loss: 0.2554 - learning_rate: 0.0010 Epoch 110/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8934 - loss: 0.2569 - val_accuracy: 0.8936 - val_loss: 0.2639 - learning_rate: 0.0010 Epoch 111/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8915 - loss: 0.2576 - val_accuracy: 0.8948 - val_loss: 0.2748 - learning_rate: 0.0010 Epoch 112/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 323ms/step - accuracy: 0.8904 - loss: 0.2642 - val_accuracy: 0.8636 - val_loss: 0.3099 - learning_rate: 0.0010 Epoch 113/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8926 - loss: 0.2634 - val_accuracy: 0.9040 - val_loss: 0.2563 - learning_rate: 0.0010 Epoch 114/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8893 - loss: 0.2598 - val_accuracy: 0.8809 - val_loss: 0.2918 - learning_rate: 0.0010 Epoch 115/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8935 - loss: 0.2578 - val_accuracy: 0.8913 - val_loss: 0.2640 - learning_rate: 0.0010 Epoch 116/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8917 - loss: 0.2581 - val_accuracy: 0.8590 - val_loss: 0.3306 - learning_rate: 0.0010 Epoch 117/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8928 - loss: 0.2578 - val_accuracy: 0.8960 - val_loss: 0.2639 - learning_rate: 0.0010 Epoch 118/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8876 - loss: 0.2683 - val_accuracy: 0.8277 - val_loss: 0.3881 - learning_rate: 0.0010 Epoch 119/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8925 - loss: 0.2591 - val_accuracy: 0.8867 - val_loss: 0.2730 - learning_rate: 0.0010 Epoch 120/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.8904 - loss: 0.2637 - val_accuracy: 0.8948 - val_loss: 0.2682 - learning_rate: 0.0010 Epoch 121/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8929 - loss: 0.2535 - val_accuracy: 0.8601 - val_loss: 0.3157 - learning_rate: 0.0010 Epoch 122/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8940 - loss: 0.2574 - val_accuracy: 0.9029 - val_loss: 0.2429 - learning_rate: 0.0010 Epoch 123/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.8938 - loss: 0.2528 - val_accuracy: 0.8520 - val_loss: 0.3381 - learning_rate: 0.0010 Epoch 124/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8912 - loss: 0.2634 - val_accuracy: 0.8543 - val_loss: 0.3441 - learning_rate: 0.0010 Epoch 125/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.8943 - loss: 0.2542 - val_accuracy: 0.8913 - val_loss: 0.2925 - learning_rate: 0.0010 Epoch 126/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8944 - loss: 0.2612 - val_accuracy: 0.8994 - val_loss: 0.2489 - learning_rate: 0.0010 Epoch 127/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 322ms/step - accuracy: 0.8940 - loss: 0.2528 - val_accuracy: 0.8867 - val_loss: 0.2555 - learning_rate: 0.0010 Epoch 128/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 317ms/step - accuracy: 0.8952 - loss: 0.2504 - val_accuracy: 0.8960 - val_loss: 0.2595 - learning_rate: 0.0010 Epoch 129/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8930 - loss: 0.2532 - val_accuracy: 0.8844 - val_loss: 0.3129 - learning_rate: 0.0010 Epoch 130/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8940 - loss: 0.2549 - val_accuracy: 0.9040 - val_loss: 0.2573 - learning_rate: 0.0010 Epoch 131/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8984 - loss: 0.2460 - val_accuracy: 0.8983 - val_loss: 0.2600 - learning_rate: 5.0000e-04 Epoch 132/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.8997 - loss: 0.2411 - val_accuracy: 0.9006 - val_loss: 0.2513 - learning_rate: 5.0000e-04 Epoch 133/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9011 - loss: 0.2401 - val_accuracy: 0.8960 - val_loss: 0.2585 - learning_rate: 5.0000e-04 Epoch 134/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9002 - loss: 0.2408 - val_accuracy: 0.9040 - val_loss: 0.2384 - learning_rate: 5.0000e-04 Epoch 135/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9025 - loss: 0.2423 - val_accuracy: 0.8936 - val_loss: 0.2703 - learning_rate: 5.0000e-04 Epoch 136/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9024 - loss: 0.2374 - val_accuracy: 0.8821 - val_loss: 0.2977 - learning_rate: 5.0000e-04 Epoch 137/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9042 - loss: 0.2390 - val_accuracy: 0.9052 - val_loss: 0.2422 - learning_rate: 5.0000e-04 Epoch 138/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9012 - loss: 0.2392 - val_accuracy: 0.9040 - val_loss: 0.2493 - learning_rate: 5.0000e-04 Epoch 139/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 316ms/step - accuracy: 0.8982 - loss: 0.2440 - val_accuracy: 0.9121 - val_loss: 0.2427 - learning_rate: 5.0000e-04 Epoch 140/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9003 - loss: 0.2385 - val_accuracy: 0.9064 - val_loss: 0.2389 - learning_rate: 5.0000e-04 Epoch 141/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9050 - loss: 0.2365 - val_accuracy: 0.8902 - val_loss: 0.2789 - learning_rate: 5.0000e-04 Epoch 142/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9024 - loss: 0.2336 - val_accuracy: 0.9040 - val_loss: 0.2367 - learning_rate: 5.0000e-04 Epoch 143/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9010 - loss: 0.2410 - val_accuracy: 0.9110 - val_loss: 0.2352 - learning_rate: 5.0000e-04 Epoch 144/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9007 - loss: 0.2376 - val_accuracy: 0.9052 - val_loss: 0.2407 - learning_rate: 5.0000e-04 Epoch 145/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 323ms/step - accuracy: 0.9047 - loss: 0.2336 - val_accuracy: 0.9145 - val_loss: 0.2372 - learning_rate: 5.0000e-04 Epoch 146/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9039 - loss: 0.2320 - val_accuracy: 0.8983 - val_loss: 0.2455 - learning_rate: 5.0000e-04 Epoch 147/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9009 - loss: 0.2394 - val_accuracy: 0.9064 - val_loss: 0.2399 - learning_rate: 5.0000e-04 Epoch 148/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 316ms/step - accuracy: 0.9027 - loss: 0.2349 - val_accuracy: 0.8844 - val_loss: 0.2638 - learning_rate: 5.0000e-04 Epoch 149/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9039 - loss: 0.2338 - val_accuracy: 0.8983 - val_loss: 0.2496 - learning_rate: 5.0000e-04 Epoch 150/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9029 - loss: 0.2335 - val_accuracy: 0.9110 - val_loss: 0.2388 - learning_rate: 5.0000e-04 Epoch 151/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9051 - loss: 0.2321 - val_accuracy: 0.9006 - val_loss: 0.2487 - learning_rate: 5.0000e-04 Epoch 152/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8978 - loss: 0.2400 - val_accuracy: 0.8798 - val_loss: 0.2742 - learning_rate: 5.0000e-04 Epoch 153/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9000 - loss: 0.2439 - val_accuracy: 0.8960 - val_loss: 0.2447 - learning_rate: 5.0000e-04 Epoch 154/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9078 - loss: 0.2274 - val_accuracy: 0.9075 - val_loss: 0.2469 - learning_rate: 5.0000e-04 Epoch 155/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9023 - loss: 0.2344 - val_accuracy: 0.9110 - val_loss: 0.2324 - learning_rate: 5.0000e-04 Epoch 156/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9060 - loss: 0.2302 - val_accuracy: 0.9156 - val_loss: 0.2297 - learning_rate: 2.5000e-04 Epoch 157/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9045 - loss: 0.2306 - val_accuracy: 0.9110 - val_loss: 0.2313 - learning_rate: 2.5000e-04 Epoch 158/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9038 - loss: 0.2282 - val_accuracy: 0.9098 - val_loss: 0.2324 - learning_rate: 2.5000e-04 Epoch 159/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.8994 - loss: 0.2336 - val_accuracy: 0.9098 - val_loss: 0.2368 - learning_rate: 2.5000e-04 Epoch 160/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9050 - loss: 0.2271 - val_accuracy: 0.9040 - val_loss: 0.2440 - learning_rate: 2.5000e-04 Epoch 161/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9074 - loss: 0.2213 - val_accuracy: 0.9029 - val_loss: 0.2416 - learning_rate: 2.5000e-04 Epoch 162/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9057 - loss: 0.2265 - val_accuracy: 0.9110 - val_loss: 0.2319 - learning_rate: 2.5000e-04 Epoch 163/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9070 - loss: 0.2237 - val_accuracy: 0.9006 - val_loss: 0.2499 - learning_rate: 2.5000e-04 Epoch 164/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9069 - loss: 0.2261 - val_accuracy: 0.8983 - val_loss: 0.2455 - learning_rate: 2.5000e-04 Epoch 165/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9073 - loss: 0.2261 - val_accuracy: 0.9121 - val_loss: 0.2337 - learning_rate: 2.5000e-04 Epoch 166/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9114 - loss: 0.2198 - val_accuracy: 0.9110 - val_loss: 0.2363 - learning_rate: 2.5000e-04 Epoch 167/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9056 - loss: 0.2189 - val_accuracy: 0.9017 - val_loss: 0.2481 - learning_rate: 2.5000e-04 Epoch 168/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9051 - loss: 0.2253 - val_accuracy: 0.9029 - val_loss: 0.2359 - learning_rate: 2.5000e-04 Epoch 169/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 322ms/step - accuracy: 0.9077 - loss: 0.2217 - val_accuracy: 0.9110 - val_loss: 0.2268 - learning_rate: 2.5000e-04 Epoch 170/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9118 - loss: 0.2145 - val_accuracy: 0.9133 - val_loss: 0.2447 - learning_rate: 2.5000e-04 Epoch 171/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9105 - loss: 0.2169 - val_accuracy: 0.9145 - val_loss: 0.2308 - learning_rate: 2.5000e-04 Epoch 172/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9106 - loss: 0.2192 - val_accuracy: 0.9075 - val_loss: 0.2255 - learning_rate: 2.5000e-04 Epoch 173/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9042 - loss: 0.2246 - val_accuracy: 0.8960 - val_loss: 0.2530 - learning_rate: 2.5000e-04 Epoch 174/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9060 - loss: 0.2263 - val_accuracy: 0.9110 - val_loss: 0.2295 - learning_rate: 2.5000e-04 Epoch 175/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 318ms/step - accuracy: 0.9082 - loss: 0.2195 - val_accuracy: 0.9145 - val_loss: 0.2293 - learning_rate: 2.5000e-04 Epoch 176/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 80s 322ms/step - accuracy: 0.9066 - loss: 0.2188 - val_accuracy: 0.9110 - val_loss: 0.2328 - learning_rate: 1.2500e-04 Epoch 177/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9083 - loss: 0.2187 - val_accuracy: 0.9121 - val_loss: 0.2305 - learning_rate: 1.2500e-04 Epoch 178/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9104 - loss: 0.2181 - val_accuracy: 0.9098 - val_loss: 0.2324 - learning_rate: 1.2500e-04 Epoch 179/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9118 - loss: 0.2148 - val_accuracy: 0.9064 - val_loss: 0.2319 - learning_rate: 1.2500e-04 Epoch 180/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9088 - loss: 0.2184 - val_accuracy: 0.9052 - val_loss: 0.2296 - learning_rate: 1.2500e-04 Epoch 181/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9091 - loss: 0.2179 - val_accuracy: 0.9052 - val_loss: 0.2284 - learning_rate: 1.2500e-04 Epoch 182/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9088 - loss: 0.2166 - val_accuracy: 0.9087 - val_loss: 0.2323 - learning_rate: 1.2500e-04 Epoch 183/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9101 - loss: 0.2114 - val_accuracy: 0.9052 - val_loss: 0.2312 - learning_rate: 1.2500e-04 Epoch 184/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9099 - loss: 0.2214 - val_accuracy: 0.9075 - val_loss: 0.2276 - learning_rate: 1.2500e-04 Epoch 185/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 321ms/step - accuracy: 0.9111 - loss: 0.2130 - val_accuracy: 0.9040 - val_loss: 0.2314 - learning_rate: 1.2500e-04 Epoch 186/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 78s 318ms/step - accuracy: 0.9122 - loss: 0.2121 - val_accuracy: 0.9075 - val_loss: 0.2300 - learning_rate: 1.2500e-04 Epoch 187/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9114 - loss: 0.2150 - val_accuracy: 0.8960 - val_loss: 0.2520 - learning_rate: 1.2500e-04 Epoch 188/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9087 - loss: 0.2133 - val_accuracy: 0.9133 - val_loss: 0.2240 - learning_rate: 1.2500e-04 Epoch 189/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 320ms/step - accuracy: 0.9097 - loss: 0.2121 - val_accuracy: 0.9087 - val_loss: 0.2296 - learning_rate: 1.2500e-04 Epoch 190/190 244/244 ━━━━━━━━━━━━━━━━━━━━ 79s 319ms/step - accuracy: 0.9095 - loss: 0.2142 - val_accuracy: 0.9110 - val_loss: 0.2277 - learning_rate: 1.2500e-04 2026-06-20 13:55:11 PM INFO 2026-06-20 13:55:11 PM - INFO - MainProcess train_yaml.py:646 - Saving model train_yaml.py:646 2026-06-20 13:55:16 PM INFO 2026-06-20 13:55:16 PM - INFO - MainProcess train_yaml.py:649 - Finished training, Total time taken: train_yaml.py:649 4:12:24.379061Look at the duration per epoch. It's mostly overhead! Except for the first iteration because you still have to spin up the workers (except for validation I assume - that's probably also reused so even that iteration is still faster) with the changes, each epoch is muuch faster now.
I don't know why it's so slow to spin up workers, I can only assume it's because it shares the cubes from training between them in memory or something, and there are a fair number of them.
References
We did observe how much overhead spinning up a worker can cause in the PR #493 (review). But that was about classification, for training where every epoch we need workers it makes an even bigger difference.
How has this PR been tested?
I looked at the memory usage of training with and without the changes and they looked fairly similar. And I have been using it for training.
Is this a breaking change?
No.
Does this PR require an update to the documentation?
No.
Checklist: