SignLanguageRecognition is a project aimed at developing a real-time sign language recognition system. The system can detect hand movements from camera footage and recognize letters and some basic expressions in sign language. The project uses MediaPipe hand tracking library and a deep learning model trained with TensorFlow/Keras, achieving high accuracy (99%+).
- Real-Time Recognition: Analyzes user's hand movements instantly through camera
- Dual Hand Support: Can simultaneously recognize both right and left hand movements
- High Accuracy: Trained model works with 99%+ accuracy rate
- FPS Indicator: Allows real-time monitoring of system performance
- Normalized Coordinates: Normalizes hand positions relative to reference point, ensuring consistent results across different hand sizes and positions
The system currently recognizes the following signs:
| Category | Signs |
|---|---|
| Letters | C, D, E |
| Numbers | 1, 2, 3, 4, 5, 6, 8 |
| Expressions | hello, iLoveYou, no, thankYou, yes |
- Python: Main programming language
- TensorFlow/Keras: Used for developing the deep learning model
- MediaPipe: Google's open-source library for hand tracking and position detection
- OpenCV: Image processing and camera integration
- NumPy: Data manipulation and mathematical operations
The project consists of three main components:
dataLabeling.py is used to extract hand coordinates from sign language images and save them in CSV format. The MediaPipe hand recognition model detects the x and y coordinates of 21 landmark points for each hand.
The collected data is used to train a deep learning model in a Jupyter notebook environment (Sing_Language_Recognition_creating_model.ipynb). Model performance is monitored throughout the training process, and the best model is saved.
app.py uses the trained model for real-time recognition. Images from the camera are processed to detect hand movements, and model predictions are displayed on the screen.
- Install the required libraries:
pip install tensorflow opencv-python mediapipe numpy- Clone or download the project:
git clone https://github.com/baranzeyn/SignLanguageRecognition.git
cd SignLanguageRecognition- Run the application:
python app.pyThe following steps are applied to recognize sign language in the project:
- 21 key points (landmarks) are detected from hand images using MediaPipe
- The coordinates of these points are normalized relative to the reference point (wrist)
- Normalized coordinates are extracted as 42 features each for right and left hands
- The dataset with a total of 84 features is trained with a deep neural network model
The model uses a multi-layer neural network:
- Input Layer: 84 neurons (right and left hand coordinates)
- Batch Normalization
- Hidden Layer 1: 128 neurons, ReLU activation
- Dropout (0.3)
- Hidden Layer 2: 64 neurons, ReLU activation
- Dropout (0.3)
- Output Layer: 15 neurons (number of recognized signs), Softmax activation
The model was trained with Adam optimization algorithm and categorical crossentropy loss function. Early stopping and model checkpoint techniques were used to obtain the best model.
- Training Accuracy: 99.79%
- Validation Accuracy: 100%
- Number of Epochs: 69 (with early stopping)
- Batch Size: 32
The model achieved a perfect accuracy rate on the validation dataset. In real-time application, it shows high performance even under different lighting conditions and backgrounds.
- Adding more signs and words
- Sentence-level sign language recognition
- Mobile application development
- Faster and lighter model architecture
- Improving model performance with transfer learning
- Real-time translation feature
- MediaPipe team for the open-source hand tracking library
- TensorFlow team for deep learning tools