Skip to content

Comments

Refactor utils/test.py: multi-path support, error handling, modular transforms#148

Open
Apprentice2907 wants to merge 5 commits intoML4SCI:mainfrom
Apprentice2907:utils-test-multi-path
Open

Refactor utils/test.py: multi-path support, error handling, modular transforms#148
Apprentice2907 wants to merge 5 commits intoML4SCI:mainfrom
Apprentice2907:utils-test-multi-path

Conversation

@Apprentice2907
Copy link

Fixes #124

What was changed

Only utils/test.py was modified. No other files touched.

Why

The original script had a hardcoded path, no error handling, commented-out
transforms, and would crash if a directory or file was missing. This made it
unreliable across different machines and environments.

Changes made

Multiple dataset paths

  • Removed hardcoded root_dir
  • Added --data_dirs CLI argument that accepts one or more paths
  • All three DeepLense dataset paths are set as defaults

Error handling

  • Script no longer crashes if a directory doesn't exist — it logs a warning and skips
  • np.load() is wrapped in try/except to handle missing or corrupted .npy files
  • Index is clamped to valid range instead of raising IndexError

Modular transforms

  • Removed commented-out transform code
  • Added build_transforms() function — crop size and normalize are now CLI flags
  • Easy to extend without touching core logic

Reproducibility

  • Added --seed, --index, --crop_size, --normalize, --output_dir CLI flags
  • Same command produces same output on any machine

Output handling

  • plots/ directory is created automatically if it doesn't exist
  • Output filename includes dataset name and index to avoid silent overwrites

Code structure

  • All logic moved into functions with a proper main() entry point
  • Script is now importable and testable

How to run

default (uses all 3 DeepLense paths, index 50)

python utils/test.py

custom

python utils/test.py --data_dirs ../Data/real_lenses_dataset/lenses --index 10
python utils/test.py --crop_size 128 --

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor utils/test.py for DeepLense Diffusion Dataset Preprocessing with Multiple Paths

1 participant