PyMorphic 🐛

A Python Framework for Metamorphic Testing in Computer Vision Robustness Validation for CI/CD Pipelines

PyMorphic is an open-source library designed to test the robustness and reliability of Computer Vision (CV) models without the need for manual labels (ground truth). Leveraging Metamorphic Testing concepts, the tool verifies if the model's behavior remains consistent under transformations that preserve the image's semantics.

---

🏛 Project Architecture

PyMorphic was built with a Framework-Agnostic philosophy. The library core operates purely on numpy.ndarray, allowing it to be used with PyTorch, TensorFlow, Keras, Scikit-learn, or any custom inference API.

The architecture follows a Composition over Inheritance pattern. A Metamorphic Relation is composed of two independent parts:

1. Operator (Input Transformation): Modifies the input image ($x \to x'$).
   • Examples: HorizontalFlip, Rotate, GaussianNoise.
2. Checker (Output Verification): Validates the consistency between the original output and the new output ($y \text{ vs } y'$).
   • Examples: ClassificationChecker, IoUChecker.

Directory Structure

projeto-pymorphic/
├── demo.py                  # Test script
└── pymorphic/               # THE LIBRARY
    ├── __init__.py
    ├── core/                # Core Logic
    │   ├── relation.py      # MetamorphicRelation (Composite Class)
    │   └── runner.py        # The Executor (PyMorphicRunner)
    ├── operators/           # INPUT: Image Transformations (x -> x')
    │   ├── base.py          # ImageOperator Interface
    │   ├── geometric.py     # Implementation (Flip, Rotate, Scale)
    │   └── photometric.py   # Implementation (Blur, Noise, Saturation)
    ├── checkers/            # OUTPUT: Verification Logic (y vs y')
    │   ├── base.py          # Checker Interface
    │   ├── classification.py # Top-K & Tolerance Logic
    │   └── segmentation.py   # Mean IoU Logic
    ├── adapters/            # Wrappers for Frameworks
    │   └── torch.py         # PyTorch Wrappers
    └── utils/
        └── io.py            # IO Utilities (Image Loading)

🎯 Supported Tasks

PyMorphic currently supports robustness testing for two major Computer Vision tasks:

1. Image Classification Verifies if the model's prediction remains stable.

• Logic: Checks if the target class rank (Top-K) is maintained and if the confidence score does not drop below a specific tolerance threshold.

• Checker: ClassificationChecker

2. Semantic Segmentation Verifies if the pixel-wise classification map remains consistent.

• Logic: Calculates the Mean Intersection over Union (mIoU) between the original mask and the transformed mask.

• Advanced Feature: Handles Geometric Consistency. If you rotate an input image by 90°, the IoUChecker can automatically rotate the expected output mask to ensure correct comparison.

• Checker: IoUChecker

⚡ Workflow Tutorial

To execute a robustness test, follow this 4-step workflow:

• Instantiate Components: Select the transformations (Operators) and the success criteria (Checkers).

• Compose Relations: Combine an Operator and a Checker into a MetamorphicRelation.

• Setup Runner: Initialize the PyMorphicRunner with your model (wrapped in an Adapter if necessary).

• Run: Execute the suite against a list of images or file paths.