SkillXR-AI is an interactive AR training prototype that combines real-time webcam hand tracking with a Unity environment. It maps tracked movement into an AR training surface, compares the performed path with an ideal path, gives immediate rule-based feedback, calculates a performance score, and exports session measurements to CSV.
The Python component captures webcam frames with OpenCV, detects a hand with MediaPipe Hands, derives a normalized control point from the wrist and middle-finger-base landmarks, smooths the coordinates, and sends them to Unity over UDP. Unity uses those coordinates to move a tracked object, draw the performed path, compare it with a reference path, and report deviation, speed, stability, score, and corrective guidance.
flowchart LR
A[Webcam] --> B[OpenCV]
B --> C[MediaPipe Hands]
C --> D[Python landmark processing]
D --> E[Normalized x,y]
E -->|UDP| F[Unity receiver]
F --> G[Movement and path tracking]
G --> H[Rule-based evaluation]
H --> I[Corrective feedback and score]
I --> J[CSV session logging]
The default configuration sends UTF-8 UDP datagrams to 127.0.0.1:5052. The host and port can be changed from the Python command line, while the receiver port is configurable in the Unity Inspector.
- Real-time webcam capture with OpenCV
- MediaPipe Hands landmark detection
- Smoothed, normalized hand-position output
- UDP communication from Python to Unity
- AR Foundation and ARCore configuration
- AR plane detection and placement
- Procedural ideal-path generation
- Actual movement-path drawing
- Movement-deviation evaluation
- Speed and movement-stability evaluation
- Performance scoring
- Immediate directional and corrective feedback
- Training-session start, stop, and reset controls
- CSV logging with time, position, deviation, score, and status
- Session summary with duration, average/max deviation, final score, accuracy, and grade
- Unity
2022.3.62f3 - C#
- AR Foundation
5.2.2 - ARCore XR Plugin
5.2.2 - Python
- OpenCV
- MediaPipe Hands
- UDP networking
- TextMeshPro and Unity UI
- CSV file output
Assets/
Scenes/SampleScene.unity AR training scene
Prefabs/placementObject.prefab Training surface and runtime components
Scripts/
hand_tracking.py Webcam, MediaPipe, smoothing, UDP sender
UDPReceiver.cs Configurable Unity UDP receiver
ARPlacement.cs AR plane placement and component wiring
XRHandMotionController.cs Coordinate mapping and smoothed movement
IdealPath.cs Reference-path generation
PathTracker.cs Actual-path drawing
PathEvaluator.cs Deviation, speed, stability, score, feedback
SessionLogger.cs CSV samples and session summary
UIResetManager.cs Session controls and live metrics
Images/ Portfolio screenshots and banner
Packages/manifest.json Unity package dependencies
requirements.txt Python package dependencies
Use a supported Python environment and create an isolated virtual environment:
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txtrequirements.txt intentionally lists package names without version pins because compatible installed versions were not recorded in the source project. Pin tested versions before using the system in a controlled deployment.
For Unity running on the same computer:
python Assets/Scripts/hand_tracking.pyThe defaults are host 127.0.0.1 and port 5052. To send to another machine or AR device:
python Assets/Scripts/hand_tracking.py --host <unity-device-address> --port 5052Press Esc in the OpenCV window to stop tracking.
- Open the repository in Unity
2022.3.62f3. - Allow Unity to resolve the packages in
Packages/manifest.json. - Open
Assets/Scenes/SampleScene.unity. - Confirm that the
UDPReceivercomponent uses the same port as the Python sender. - Start
hand_tracking.py, then enter Play mode to test the receiver, movement mapping, evaluation, and logging.
- The project is configured with AR Foundation and ARCore for Android AR use.
- A compatible ARCore device and camera permission are required for a device build.
- If Python and Unity run on different devices, pass the Unity device's reachable network address with
--hostand permit UDP traffic on the configured port. - Network configuration varies by router, firewall, development machine, and AR device.
Each datagram contains two comma-separated normalized floating-point coordinates:
x,y
Both values originate from MediaPipe's normalized landmark coordinates. Unity remaps them into the local training-surface coordinate system.
SessionLogger.cs writes timestamped TrainingSession_*.csv files to Unity's Application.persistentDataPath. Each sample contains:
time,x,y,z,distance,score,status
The file concludes with session duration, average deviation, maximum deviation, final score, accuracy percentage, and a performance grade.
Watch the 85-second SkillXR-AI demo on Vimeo
- Movement evaluation and coaching are deterministic and rule-based.
- The current implementation does not use a trained machine-learning scoring model.
- Webcam tracking runs in Python; AR interaction runs in Unity.
- Cross-device operation requires an appropriate network address, matching port, and firewall configuration.
- The dependency file is unpinned until a tested environment matrix is established.
The following are future possibilities, not implemented features:
- Learned movement-quality models
- Voice coaching
- Cloud analytics and reporting
- Learning-management-system integration
- Multi-user training
- Adaptive skill progression
Dr. Ayaz Muhammad Hanif
iDeveloperz
The repository currently states that the source, design, architecture, datasets, and documentation may not be copied, modified, distributed, or used commercially without explicit written permission from the author.
© 2026 Dr. Ayaz Muhammad Hanif