ASL Hand Gesture Recognition

A comprehensive machine learning project for American Sign Language (ASL) hand gesture classification using deep learning. This project implements transfer learning with MobileNetV2 and ResNet18 architectures, featuring cross-subject validation, Grad-CAM visualizations, and extensive exploratory data analysis.

📋 Overview

This project tackles ASL alphabet recognition (36 classes: A-Z and 0-9) with a focus on:

• Person-disjoint cross-validation to ensure model generalization across different subjects
• Transfer learning with pre-trained CNNs (MobileNetV2, ResNet18)
• Model interpretability via Grad-CAM visualizations
• Comprehensive EDA including PCA, t-SNE, and UMAP embedding analysis
• Background augmentation experiments to improve robustness

🗂️ Project Structure

├── main.py                          # Main training script with k-fold cross-validation
├── cross_subject_kfold.py           # Create person-disjoint k-fold splits
├── split_dataset.py                 # Dataset splitting utilities
├── compare_background_augmentation.py # Compare training with/without background augmentation
├── gradcam.py                       # Grad-CAM visualization implementation
├── data_exploration.ipynb           # Jupyter notebook for EDA
├── embedding_visualization_*.py     # PCA, t-SNE, UMAP visualization scripts
├── library/                         # Core library modules
│   ├── model/                       # Model building utilities
│   ├── train/                       # Training loops and utilities
│   ├── data_processing/             # Data loading and augmentation
│   └── utils/                       # Helper functions
├── data/                            # Dataset directory
│   ├── asl_dataset/                 # Raw dataset (36 classes)
│   ├── by_person/                   # Data grouped by person
│   ├── by_person_kfold/             # K-fold splits for cross-validation
│   ├── train/                       # Training split
│   └── test/                        # Test split
├── runs/                            # Training outputs and checkpoints
├── gradcam_outputs/                 # Grad-CAM heatmap visualizations
└── eda_outputs/                     # EDA results and metrics

🚀 Quick Start

Installation

# Clone the repository
git clone https://github.com/verynewusername/Advanced-Machine-Learning-Course.git
cd Advanced-Machine-Learning-Course

# Install dependencies
pip install -r requirements.txt

Requirements

• Python 3.8+
• PyTorch 2.2.2
• torchvision 0.17.2
• scikit-learn
• umap-learn
• matplotlib
• numpy
• Pillow

Dataset Preparation

To split a dataset organized in a class-per-folder structure into training and testing sets:

python split_dataset.py \
  --src data/asl_dataset \
  --dst data_split \
  --test-ratio 0.2 \
  --seed 42

Training

K-Fold Cross-Validation Training:

python main.py

This runs 5-fold cross-subject validation with MobileNetV2, featuring:

• 20 epochs of head-only training
• 20 epochs of fine-tuning with early stopping
• Label smoothing (0.1)
• Automatic device selection (CUDA/MPS/CPU)

Background Augmentation Comparison:

python compare_background_augmentation.py --data-root data_split --epochs-head 20 --epochs-finetune 20

🔬 Key Features

1. Cross-Subject K-Fold Validation

Ensures robust evaluation by keeping hands from the same person in the same fold (train/val/test splits are person-disjoint):

python cross_subject_kfold.py

2. Grad-CAM Visualizations

Generate class activation maps to understand what the model focuses on:

python gradcam.py --checkpoint runs/kfold_mobilenetv2/fold0/best_model.pt \
                   --image data/test/a/example.jpg \
                   --output gradcam_outputs/

See gradcam_README.md for detailed usage.

3. Embedding Visualization

Analyze feature space using dimensionality reduction:

• PCA: python embedding_visualization_pca_mobilenet.py
• t-SNE: python embedding_visualization_tsne_mobilenet.py
• UMAP: python embedding_visualization_umap_mobilenet.py

Results include silhouette scores, Davies-Bouldin indices, and intra/inter-class similarity metrics.

4. Two-Phase Training Strategy

Phase 1 - Head Training:

• Freeze backbone (pre-trained on ImageNet)
• Train only the classification head
• Learning rate: 1e-3

Phase 2 - Fine-tuning:

• Unfreeze all layers
• Lower learning rates (backbone: 1e-4, head: 5e-4)
• Early stopping with patience=5

📊 Model Architecture

• Backbone: MobileNetV2 (default) or ResNet18
• Input: 224×224 RGB images
• Output: 36 classes (A-Z, 0-9)
• Preprocessing: ImageNet normalization
• Augmentation: Optional horizontal flip, background randomization

📈 Results

The model achieves strong performance with person-disjoint validation, demonstrating good generalization across different subjects. Results are saved in:

• runs/kfold_mobilenetv2/summary.json - Cross-validation statistics
• eda_outputs/ - Embedding analysis metrics
• Training curves and confusion matrices in respective run directories

🛠️ Configuration

Main training parameters in main.py:

cfg = {
    "seed": 42,
    "model_name": "mobilenet_v2",  # or "resnet18"
    "img_size": 224,
    "batch_size": 32,
    "label_smoothing": 0.1,
    "epochs_head": 20,
    "epochs_finetune": 20,
    "lr_head": 1e-3,
    "lr_backbone": 1e-4,
    "patience": 5,
}

📝 Dataset Format

Expected directory structure:

data/asl_dataset/
  ├── a/
  │   ├── hand1_*.jpg
  │   ├── hand2_*.jpg
  │   └── ...
  ├── b/
  └── ...

Images should be named with prefix hand{N}_ where N is the person ID.

🤝 Contributing

This project is part of an Advanced Machine Learning course. Contributions and suggestions are welcome!

📄 License

This project is available for educational purposes.

🙏 Acknowledgments

• Pre-trained models from torchvision
• ASL dataset contributors
• PyTorch and scikit-learn communities