Spike-Based Hand Gesture Recognition: A Comparative Study of Neuromorphic Encoding Strategies

This repository contains the code and documentation for the Neuromorphic Engineering course project at Sant'Anna School of Advanced Studies / University of Pisa. The project explores a hybrid LabVIEW-Python pipeline to classify six fundamental left-hand gestures using the Leap Motion Controller (LMC).

We systematically compare traditional geometric approaches (floating-point spatial coordinates) against neuromorphic spike-rate coding using the bio-inspired Izhikevich neuron model.

🎯 Project Goals

1. Classify 6 Hand Gestures: Open Palm, Fist, Thumb Only, Pinky Only, Horns, and Thumb + Pinky.
2. Develop a Hybrid Pipeline: Real-time hardware acquisition and neuromorphic encoding in LabVIEW, with offline training and real-time inference in Python.
3. Compare Encoding Strategies: Analyze trade-offs between standard high-precision representations (Float-64) and sparse neuromorphic event spikes (Int-16) across multiple configurations.

🛠 Architecture & Pipeline

The system is split into two major environments working in tandem via UDP communication:

• Real-Time Acquisition & Encoding (LabVIEW): Interacts with the Leap Motion Controller to capture raw 3D joint coordinates (fingertips, palm, and hand direction). LabVIEW converts these physical distances into input currents and stimulates parallel Izhikevich neurons. The system computes real-time spike counts using different routing paradigms.

• Machine Learning & Inference (Python): Receives the generated spike trains to train optimized Random Forest Classifiers via 5-Fold Stratified Cross-Validation and Grid Search. The trained .pkl models are subsequently deployed back for live, closed-loop inference.

🧠 Neuromorphic Encoding Approaches

To handle hardware tracking ambiguities, we implemented and evaluated different spatial mapping strategies:

• Blind Spike Encoding (5 features): Raw spatial coordinates mapped directly to 5 channels.
• Summed Spike Encoding (1 feature): Aggregates activity across all 5 channels.
• Routed Spike Encoding (5 features): Applies a fixed-threshold spatial routing mechanism on the $x$-axis to reliably distinguish individual fingers, boosting accuracy significantly.
• BlindDir Spike Encoding (11 features): Combines raw finger spikes with positive and negative palm direction vectors to preserve orientation context, eliminating the need for complex spatial routing.

📊 Results & Performance

Among the simulated biological neuron types, the Fast Spiking and Thalamo-Cortical models yielded the highest classification accuracies due to their rapid temporal dynamics, which efficiently encode spatial distances into discrete spikes.

Key Trade-offs

Configuration	Channels	Spatial Routing	Test Accuracy	Data Size
Standard Distances	5 (Float64)	No	84.17%	Very High
Standard Distances + Dir	8 (Float64)	No	100.00%	Very High
Routed Neuromorphic	5 (Int16)	Yes	~97.54%	Low
BlindDir Neuromorphic	11 (Int16)	No	~98.67%	Medium

The neuromorphic approaches, particularly BlindDir and Routed, present a highly advantageous engineering trade-off: a negligible drop in accuracy (~ 1.3%) in exchange for massive data reduction and sparsity.

Neuromorphic Confusion Matrix (Thalamo1 11-Channels)

Below is the confusion matrix for one of the top-performing neuromorphic configurations (BlindDir 11ch using the Thalamo neuron), showing robust generalization across almost all gestures:

📁 Repository Structure

• LABVIEW/: Contains all .vi files for Leap Motion acquisition, dataset generation, Izhikevich simulation, UDP streaming, and inference visualization.
  • Note: The large Starting_Dataset.csv is included as a baseline.
• Python/: Contains all scripts and Jupyter Notebooks for training the Random Forest models, performing Grid Search, and generating visualizations.
  • Models/: Serialized .pkl best-performing models.
  • output/: Confusion matrices and feature importance plots.
• presentazione.pdf: The official PowerPoint presentation slides used for the exam.
• documentation_neuromorphic(2).pdf: The comprehensive project report detailing the theoretical background, methodology, and extensive results.

🚀 How to Run

1. Acquisition: Open LABVIEW/hand_dataset_generation_v002.vi to gather new LMC data.
2. Spike Conversion: Use LABVIEW/Multi_Izzie_csv.vi to convert the raw .csv into spike trains.
3. Training: Run Python/train_models.ipynb to train the Random Forest models and generate .pkl files.
4. Real-time Inference:
   • Start the Python UDP receiver script.
   • Run LABVIEW/Final_sender.vi to stream live spikes.
   • Observe the predicted class in the LabVIEW inference GUI.

---

Authors: Andrea Diano, Filippo Aspi, Riccardo Mambrini
Course: Neuromorphic Engineering, Sant'Anna School of Advanced Studies