Update for the training data preprocessing - #968
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the training pipeline’s preprocessing and error surfacing so invalid training-folder setups are rejected earlier (before model initialization), while also refining dataset splitting and upsampling behavior and adding coverage for the new/changed logic.
Changes:
- Add preflight validation for training folders that contain no supported audio files, and propagate localized errors with placeholder payloads to the GUI.
- Refactor
random_split,random_multilabel_split, andupsamplinglogic to reduce duplication/inefficiencies and adjust shuffling behavior around validation splitting. - Add/extend tests for training-data loading validation, split correctness, and upsampling behavior; add new localization keys for new validations.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/train/test_train_loading.py | Adds regression test ensuring empty class folders are detected before model loading. |
| tests/train/test_model.py | Adds unit tests covering split and upsampling behavior. |
| birdnet_analyzer/train/utils.py | Adds _check_input_folders and runs it before model setup; fixes validation message typo. |
| birdnet_analyzer/model.py | Refactors split and upsampling implementations; adjusts training-data shuffling and enables shuffle=True in Keras fit. |
| birdnet_analyzer/gui/train.py | Formats localized validation messages with payload arguments for placeholders. |
| birdnet_analyzer/lang/en.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/de.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/fr.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/fi.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/id.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/pt-br.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/ru.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/se.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/tlh.json | Adds localization strings for new validations and multi-label upsampling restriction. |
| birdnet_analyzer/lang/zh_TW.json | Adds localization strings for new validations and multi-label upsampling restriction. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+167
to
+172
| has_audio_file = any( | ||
| entry.is_file() | ||
| and not entry.name.startswith(".") | ||
| and entry.name.rsplit(".", 1)[-1].lower() in ALLOWED_FILETYPES | ||
| for entry in os.scandir(folder_path) | ||
| ) |
Comment on lines
+242
to
+243
| train_indices = np.concatenate(train_indices) | ||
| val_indices = np.concatenate(val_indices) |
Comment on lines
+281
to
+285
| source_indices = np.flatnonzero(y == minority_label) | ||
| missing_samples = min_samples - len(source_indices) | ||
|
|
||
| while np.where(y == minority_label)[0].shape[0] + len(y_temp) < min_samples: | ||
| random_index = rng.choice(np.where(y == minority_label)[0], size=size) | ||
| x_app, y_app = apply(x, y, random_index) | ||
| for _ in range(max(0, missing_samples)): | ||
| random_index = rng.choice(source_indices, size=size) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.