An advanced AI-driven fitness companion that tracks workouts, analyzes form in real time, and visualizes your body in 3D β all from a browser webcam. No wearables, no external hardware.
Proudly participating in GirlScript Summer of Code 2026!
Features Β· Demo Β· Quick Start Β· Architecture Β· Tech Stack Β· Contributing Β· FAQ
SpectraX is a full-stack fitness application that uses MediaPipe Pose Detection, Three.js, and WebSockets to deliver real-time workout tracking, intelligent form analysis, and immersive 3D body visualization β entirely in the browser.
It doesn't just track your body; it understands your movement. SpectraX counts reps, detects which exercise you're performing, scores your form, identifies anomalies with ML algorithms, and renders a live 3D skeleton that mirrors your every move.
| Traditional Fitness Trackers | SpectraX |
|---|---|
| Requires wearable hardware | Uses only a webcam |
| Basic step/heart-rate counting | Intelligent rep counting with form scoring |
| Manual exercise logging | Auto-detects exercise type via AI |
| 2D dashboards | Real-time 3D body skeleton visualization |
| No posture feedback | Real-time posture correction & anomaly detection |
| Feature | Description | |
|---|---|---|
| ποΈ | Intelligent Rep Counting | Automatically counts reps for squats, push-ups, bicep curls, shoulder press, lunges, flutter kicks, and more |
| π | Real-Time Form Analysis | Scores each rep based on joint angles and posture accuracy |
| π | Auto-Exercise Detection | AI classifies which exercise you're performing without manual selection |
| π₯ | 3D Body Mapping | Real-time skeleton rendered with Three.js and WebGL, with dynamic joint-angle vector shaders |
| Feature | Description | |
|---|---|---|
| π§ | Anomaly Detection | Pure-TypeScript ML engine (Z-Score, Modified Z-Score, Isolation Forest) flags form deviations |
| π¬ | AI Recommendations | Post-workout improvement suggestions powered by the feedback engine |
| π― | Biomechanical Stress Visualization | Volumetric fog and stress-vector shaders highlight muscle engagement during replay |
| Feature | Description | |
|---|---|---|
| π | Workout Summary | Post-session analytics with rep streaks, duration, calorie estimates, and accuracy breakdowns |
| π | Session Replay | Review your workout with full 3D replay, complete with camera orbit controls locked to pelvis tracking |
| π | Badge & Achievement System | Earn trophies and track workout streaks |
| π | Leveling System | Gain XP and level up as you work out consistently |
| Feature | Description | |
|---|---|---|
| β‘ | Real-Time Sync | Low-latency WebSocket communication with authenticated socket connections, rate limiting, and CORS hardening |
| π | Firebase Auth | Full authentication flow (signup, login, forgot password) with App Check anti-abuse protection |
| π΄ | Offline Support | PWA with service worker caching and offline queue for workout data |
| π₯οΈ | FPS Monitoring | Built-in performance overlay with adaptive throttling for lower-end devices |
Initialize your session, sign in, or browse workout history.
Select from multiple exercises with live camera preview.
Real-time pose detection ready for bicep curl tracking.
Plank hold detection with live camera feed.
Push-up rep counting with form analysis.
Review past workout sessions and track your progress over time.
graph TB
subgraph Browser["Browser - Client"]
CAM["Webcam Feed"]
MP["MediaPipe Pose - 33 Landmarks"]
EE["Exercise Engine - Rep Counting and Form Scoring"]
AD["Anomaly Detection - Z-Score / MAD / Isolation Forest"]
THREE["Three.js Renderer - 3D Skeleton + Shaders"]
UI["React 18 UI - Workout / Summary / Replay"]
SW["Service Worker - PWA + Offline Queue"]
FB["Firebase Auth + Firestore"]
end
subgraph Server["Backend - Node.js"]
EXPRESS["Express.js - REST API + Health"]
SOCKET["Socket.io - Real-Time Sync"]
PROC["Pose Processor - Angle Calculations"]
FEED["Feedback Engine - Recommendations"]
STORE["Session Storage - File-Based"]
end
CAM --> MP
MP --> EE
EE --> AD
EE --> THREE
EE --> UI
AD --> UI
THREE --> UI
UI --> SW
UI --> FB
UI <--> SOCKET
SOCKET <--> PROC
PROC --> FEED
PROC --> STORE
EXPRESS --> STORE
- π· Capture β Camera frames are captured in real time via the browser's MediaDevices API
- 𦴠Pose Estimation β MediaPipe Pose extracts 33 body landmarks with 3D coordinates
- π Angle Calculation β Joint angles are computed using landmark coordinates (with optional GPU-accelerated calculations)
- π€ Exercise Classification β AI logic auto-detects the current exercise using the activity classification service
- π’ Rep Counting β State-machine-based algorithms track movement cycles with depth classifiers per exercise
- π― Form Scoring β Posture accuracy is evaluated in real time; anomaly detection flags deviations post-session
- π₯ 3D Rendering β Three.js renders a live skeleton with volumetric fog and biomechanical stress shaders
- β‘ Sync β Socket.io synchronizes workout data between client and server with auth, rate limiting, and CORS hardening
| Category | Technologies |
|---|---|
| Frontend | React 18, TypeScript, Vite |
| 3D Rendering | Three.js, WebGL, GLSL Shaders |
| AI / ML | MediaPipe Pose, Transformers.js (@xenova/transformers) |
| Backend | Node.js, Express.js, Socket.io |
| Auth & Database | Firebase Auth, Firestore, Firebase App Check |
| State Management | React Context API, Custom Hooks |
| Testing | Vitest, React Testing Library, Supertest |
| CI / CD | GitHub Actions (Node 20.x, 22.x) |
| PWA | vite-plugin-pwa, Workbox |
| Deployment | Vercel (frontend), configurable backend |
| Styling | Vanilla CSS with modular component styles |
| Icons | Lucide React |
spectrax_1/
βββ public/ # Static assets
β βββ assets/demos/ # Exercise demo videos (squat, pushup, plank, etc.)
β βββ favicon.svg # App icon
β βββ icons.svg # UI icon sprites
β βββ model.glb # 3D body model for skeleton rendering
β
βββ assets/ # Project assets
β βββ screenshots/ # App screenshots for documentation
β
βββ src/ # Frontend source
β βββ components/ # React components
β β βββ WorkoutScreen.tsx # Main workout view with camera + pose
β β βββ CalibrationScreen.tsx # Pre-workout body alignment
β β βββ SummaryScreen.tsx # Post-workout analytics
β β βββ SummaryScreenSkeleton.tsx# Skeleton loader for summary
β β βββ ReplayScreen.tsx # Session replay UI
β β βββ Replay3DModel.tsx # 3D replay with stress shaders
β β βββ WelcomeScreen.tsx # Landing / exercise selection
β β βββ LoginScreen.tsx # Login screen
β β βββ SignUpScreen.tsx # Registration screen
β β βββ ForgotPasswordScreen.tsx # Password recovery
β β βββ UserProfileScreen.tsx # User profile view
β β βββ FitnessCalculator.tsx # BMI / calorie calculator
β β βββ TrophyRoom.tsx # Badge & achievement display
β β βββ AIRecommendations.tsx # AI workout suggestions
β β βββ BadgeNotification.tsx # Badge popup notifications
β β βββ CursorGlow.tsx # Cursor glow effect
β β βββ FpsMonitor.tsx # FPS performance monitor
β β βββ FpsOverlay.tsx # FPS overlay display
β β βββ ScrollToTopButton.tsx # Scroll-to-top button
β β βββ WorkoutPanels.tsx # Workout info panels
β β βββ HistoryPageSkeleton.tsx # Skeleton loader for history
β β βββ EmptyState.tsx # Empty state placeholder
β β βββ NotFound.tsx # 404 page
β β βββ ProtectedRoute.tsx # Auth route guard
β β βββ CameraErrorBoundary.tsx # Camera error handling
β β βββ PageErrorBoundary.tsx # Page-level error boundary
β β
β βββ services/ # Core business logic
β β βββ exerciseEngine.ts # Rep counting & form scoring
β β βββ poseService.ts # MediaPipe pose processing
β β βββ cameraService.ts # Camera initialization & management
β β βββ sessionRecorder.ts # Workout recording & serialization
β β βββ gestureService.ts # Gesture recognition
β β βββ calibrationStateEngine.ts# Calibration state machine
β β βββ calibrationLogic.ts # Calibration calculations
β β βββ calibrationVisualRenderer.ts # Calibration overlay rendering
β β βββ occlusionPredictor.ts # Landmark occlusion handling
β β βββ kinematicEngine.ts # Motion kinematics
β β βββ volumetricFogEngine.ts # 3D volumetric fog shader engine
β β βββ volumetricFogShaders.ts # GLSL shader definitions
β β βββ workoutSyncService.ts # Client-server sync logic
β β βββ angleUtils.ts # Joint angle calculations
β β βββ gpuAngleUtils.ts # GPU-accelerated angle utils
β β βββ overlayRenderer.ts # Canvas overlay rendering
β β βββ clipEngine.ts # Workout clip engine
β β βββ bodyTypeEngine.ts # Body type detection
β β βββ skeletalSense.ts # Skeletal awareness logic
β β βββ ghostService.ts # Ghost overlay service
β β βββ poseLockService.ts # Pose lock detection
β β βββ syncQueue.ts # Data sync queue
β β βββ performanceThrottleService.ts # Adaptive performance throttling
β β βββ activityClassificationService.ts # Exercise auto-detection
β β βββ wristRotationDetector.ts # Wrist rotation tracking
β β βββ Squat_depth_classifier.ts# Squat depth classification
β β βββ Pushup_depth_classifier.ts # Push-up depth classification
β β
β βββ engine/ # AI engines
β β βββ feedbackEngine.ts # Post-workout feedback generation
β β βββ recommendationEngine.ts # AI-driven workout recommendations
β β
β βββ hooks/ # Custom React hooks
β β βββ useCameraPose.ts # Camera + pose detection lifecycle
β β βββ useWorkoutSync.ts # WebSocket workout sync
β β βββ useWorkoutWebSocket.ts # WebSocket connection management
β β βββ useBadges.ts # Achievement tracking
β β βββ useLeveling.ts # XP & level progression
β β βββ useNetworkStatus.ts # Online/offline detection
β β βββ useAuth.ts # Firebase auth hook
β β βββ useDisplayConfig.ts # Display/resolution config
β β βββ useFpsCounter.ts # FPS counting hook
β β βββ useOffscreenCanvas.ts # Offscreen canvas management
β β βββ usePrefersReducedMotion.ts # Reduced motion preference
β β
β βββ workers/ # Web Workers (off-main-thread)
β β βββ poseWorker.ts # Pose processing worker
β β βββ activityWorker.ts # Activity classification worker
β β
β βββ context/ # React Context providers
β β βββ AuthContext.tsx # Firebase auth state
β β βββ SettingsContext.tsx # User preferences
β β βββ ThemeContext.tsx # Dark/light theme
β β
β βββ config/ # Configuration
β β βββ exercises.ts # Exercise definitions & parameters
β β βββ firebase.ts # Firebase initialization
β β βββ badges.ts # Badge definitions
β β βββ poseLandmarks.ts # MediaPipe landmark indices
β β
β βββ utils/ # Utility functions
β β βββ fitnessCalculations.ts # BMI, BMR, calorie math
β β βββ calorieEstimator.ts # Per-exercise calorie estimation
β β βββ streakUtils.ts # Workout streak logic
β β βββ offlineQueue.ts # Offline data queue
β β βββ avatarSkins.ts # Avatar skin definitions
β β βββ badgeIcons.ts # Badge icon mappings
β β βββ debounce.ts # Debounce utility
β β
β βββ assets/ # Frontend assets (images, SVGs)
β βββ styles/ # CSS modules
β β βββ app.css # Global app styles
β β βββ auth.css # Authentication page styles
β β βββ WelcomeScreen.css # Welcome screen styles
β β βββ FitnessCalculator.css # Calculator styles
β β
β βββ types/ # TypeScript type definitions
β β βββ badge.ts # Badge type interfaces
β β
β βββ App.tsx # Root app with routing
β βββ HistoryPage.tsx # Workout history page
β βββ SessionCard.tsx # Session card component
β βββ useWorkoutHistory.ts # Workout history hook
β βββ main.tsx # React entry point
β βββ index.css # Global CSS
β βββ style.css # Additional global styles
β
βββ server/ # Backend
β βββ src/
β β βββ index.js # Server entry point
β β βββ app.js # Express app setup
β β βββ app/ # App factory (createApp, createServer)
β β βββ socket/ # Socket.io event handlers
β β βββ modules/ # Pose processing, sessions, feedback
β β βββ middleware/ # Auth, rate limiting, CORS
β β βββ shared/ # Shared constants & utilities
β β βββ config/ # Server configuration
β β
β βββ sessions/ # Session data storage
β βββ tests/ # Backend tests (unit + integration)
β βββ index.js # Root entry (re-exports src/index.js)
β βββ .env.example # Backend env template
β βββ README.md # Backend documentation
β βββ package.json # Backend dependencies
β βββ vitest.config.js # Backend test config
β
βββ spectrax_anomaly/ # Anomaly detection module
β βββ src/ # Z-Score, MAD, Isolation Forest algos
β βββ INTEGRATION.md # Integration guide
β
βββ scripts/ # Utility scripts
β βββ setup_labels.sh # GitHub label setup script
β βββ verify-theme-security.js # Theme security verification
β
βββ .github/
β βββ workflows/ci.yml # CI pipeline (lint, build, test)
β βββ ISSUE_TEMPLATE/ # Issue templates (bug, feature, GSSoC task)
β βββ pull_request_template.md # PR template
β
βββ firestore.rules # Firestore security rules
βββ firestore.indexes.json # Firestore index definitions
βββ firebase.json # Firebase project config
βββ vercel.json # Vercel deployment config
βββ vite.config.ts # Vite + PWA + SharedArrayBuffer config
βββ tsconfig.json # TypeScript configuration
βββ tsconfig.node.json # TypeScript config for Node
βββ index.html # HTML entry point
βββ jest.config.cjs # Jest configuration
βββ .eslintrc.cjs # ESLint configuration
βββ package.json # Dependencies & scripts
# 1. Clone the repository
git clone https://github.com/Somil450/spectrax_1.git
cd spectrax_1
# 2. Install frontend dependencies
npm install
# 3. Install backend dependencies
cd server && npm install && cd ..
# 4. Set up environment variables (see below)
# 5. Start the backend (Terminal 1)
cd server && npm run dev
# 6. Start the frontend (Terminal 2)
npm run devThe frontend runs at http://localhost:5173 and the backend at http://localhost:3001.
Copy .env.example and fill in your Firebase credentials:
# Firebase Configuration (from Firebase Console)
VITE_FIREBASE_API_KEY=your_api_key_here
VITE_FIREBASE_AUTH_DOMAIN=your_project.firebaseapp.com
VITE_FIREBASE_PROJECT_ID=your_project_id
VITE_FIREBASE_STORAGE_BUCKET=your_project.appspot.com
VITE_FIREBASE_MESSAGING_SENDER_ID=your_sender_id
VITE_FIREBASE_APP_ID=your_app_id
# Firebase App Check β reCAPTCHA Enterprise site key (optional, leave empty for local dev)
VITE_APPCHECK_RECAPTCHA_KEY=
# Backend URL (client derives WebSocket URL by swapping http β ws)
VITE_BACKEND_URL=http://localhost:3001PORT=3001
ALLOWED_ORIGIN=http://localhost:5173
β οΈ Never commit.envfiles to version control. The.env.examplefile is provided as a template.
- Create a project at Firebase Console
- Enable Authentication (Email/Password provider)
- Create a Firestore database
- Copy your project config into the
.envfile - Deploy Firestore security rules:
npm install -g firebase-tools
firebase login
firebase deploy --only firestore:rulesWithout deploying these rules, the project runs in Firebase test mode (open access). Always deploy them before going to production.
- Welcome β Choose an exercise or let SpectraX auto-detect it
- Calibration β Align yourself with the camera for optimal pose tracking
- Workout β Exercise in real time! Watch reps count up, form scores update, and your 3D skeleton mirror your movement
- Summary β Review detailed post-workout analytics: rep streaks, duration, calorie burn, accuracy breakdown, and AI-generated improvement tips
- Replay β Re-watch your session in full 3D with biomechanical stress overlays
| Exercise | Detection | Rep Counting | Depth Classifier |
|---|---|---|---|
| Bodyweight Squats | β | β | β |
| Push-Ups | β | β | β |
| Bicep Curls | β | β | β |
| Shoulder Press | β | β | β |
| Lunges | β | β | β |
| Flutter Kicks | β | β | β |
| Plank (Hold) | β | β±οΈ Timer | β |
- Jumping Jacks
- Mountain Climbers
- Burpees
| Metric | Value |
|---|---|
| Pose Detection FPS | ~30 FPS |
| Rep Counting Accuracy | ~94% |
| Detection Latency | <100ms |
| Supported Resolution | 720p / 1080p |
| Pose Landmarks | 33 Keypoints (3D) |
| Web Workers | 2 (pose + activity classification) |
Performance varies by device hardware, lighting conditions, and camera quality. The built-in FPS monitor and adaptive throttle service help maintain smooth performance on lower-end devices.
| Browser | Support |
|---|---|
| Chrome (Desktop) | β Full Support |
| Edge | β Full Support |
| Firefox | β Full Support |
| Android Chrome | β Full Support |
| Safari / iOS |
Note: SpectraX requires
SharedArrayBuffersupport (viaCross-Origin-Opener-PolicyandCross-Origin-Embedder-Policyheaders), which is configured automatically in both the Vite dev server and Vercel deployment.
# Run frontend tests
npm test
# Run tests in watch mode
npm run test:watch
# Run tests with coverage
npm run test:coverage
# Run lint checks
npm run lint
# Run backend tests
cd server && npm testThe project uses Vitest with jsdom for frontend tests and React Testing Library for component testing. The backend uses Vitest with Supertest for HTTP endpoint testing.
CI runs automatically on push/PR to main via GitHub Actions, testing against Node 20.x and 22.x.
The project includes a vercel.json with:
- SPA rewrites (all routes β
index.html) - Required COOP/COEP headers for
SharedArrayBuffer - Service worker cache control
Deploy with:
npx vercelThe backend is a standalone Express + Socket.io server. Deploy to any Node.js-compatible platform (Railway, Render, Fly.io, etc.) and set the ALLOWED_ORIGIN env var to your frontend URL.
- AI-based calorie estimation improvements
- Multi-person pose tracking
- Voice-guided workout assistant
- Native mobile application
- Cloud workout history sync
- Workout recommendation engine
- Advanced analytics dashboard with charts
- Social features (workout sharing, leaderboards)
SpectraX is a GSSoC 2026 project and we welcome contributors of all experience levels!
- Read the Contributing Guide and Code of Conduct
- Find an issue β Check open issues for
good first issue,level1,level2, orlevel3labels - Follow the workflow:
# Fork & clone git clone https://github.com/YOUR_USERNAME/spectrax_1.git cd spectrax_1 # Create a feature branch git checkout -b feature/your-feature-name # Make changes, test locally, then push git push origin feature/your-feature-name
- Open a PR using the PR template and link the issue number
| Prefix | Purpose |
|---|---|
feat: |
New feature |
fix: |
Bug fix |
docs: |
Documentation |
refactor: |
Code restructuring |
test: |
Adding/updating tests |
perf: |
Performance improvement |
- Local Processing β Camera data is processed entirely in the browser; no video frames are sent to any server
- No Raw Video Storage β SpectraX never stores raw video footage
- Minimal Data β Only workout analytics and session summaries are persisted
- No Third-Party Biometric Sharing β No personal biometric data is shared externally
- Firebase App Check β Optional reCAPTCHA Enterprise integration to protect authentication endpoints from abuse
- Socket Auth & Rate Limiting β Backend enforces authentication, rate limits, and CORS policies on WebSocket connections
See the full FAQ document for detailed answers to common questions about setup, exercises, privacy, and troubleshooting.
This project is licensed under the MIT License.
SpectraX β The Future of AI Fitness
Made with β€οΈ by Somil Jain and amazing contributors
β Star this repo if you find it useful!