AdMyBrand - Live Phone Object Detection

Real-time object detection system that streams live camera feed from a phone to a browser viewer with AI-powered bounding box overlays. Built with WebRTC, MediaPipe, and Node.js.

🚀 One-Command Start

Quick Start (Recommended):

# Clone and run with Docker
git clone https://github.com/adityaatre26/Object-Detection-WebRTC.git
cd Object-Detection-WebRTC
./start.sh

Local Development:

npm install
npm run start:https

The app automatically starts with HTTPS and displays your LAN IP for phone connections.

🔄 Mode Switching

WASM Mode (Default - Fastest):

npm run start:https          # MediaPipe WASM with TFJS fallback

Server Mode (Compatibility):

npm start                    # HTTP-only for localhost testing

Production Deployment:

docker-compose up --build    # HTTPS + LAN access via Docker

📱 Phone Connection Instructions

Method 1: QR Code (Recommended)

1. Start the application using any mode above
2. Open the viewer URL on your laptop/desktop browser
3. Scan the QR code displayed on screen with your phone camera
4. Allow camera permissions when prompted
5. Point camera at objects to see live detection with bounding boxes!

Method 2: Short URL

1. Note the LAN IP displayed in terminal (e.g., https://192.168.1.100:3000)
2. Type short URL in phone browser: [IP]/phone?session=[SESSION-ID]
3. Allow camera access and start detection

Troubleshooting Connection

• HTTPS Required: Mobile browsers require HTTPS for camera access
• Same Network: Ensure both devices are on the same WiFi/LAN
• Firewall: Check that port 3000 isn't blocked
• Self-signed Certificate: Accept security warning on first visit

🔧 Running Modes

WASM Mode (Default - High Performance):

npm run start:https     # MediaPipe WASM + HTTPS for mobile

• ✅ 2-3x faster inference with MediaPipe WASM
• ✅ Automatic fallback to TensorFlow.js if WASM fails
• ✅ Cross-origin isolation enabled for optimal performance

Server Mode (Localhost Testing):

npm start              # HTTP-only for same-device testing

• ✅ Lighter resource usage for development
• ❌ No mobile camera access (HTTPS required)
• ✅ Faster startup without certificate generation

Production Mode (Docker):

./start.sh            # Docker with HTTPS + LAN network access

• ✅ Containerized deployment with host networking
• ✅ Automatic HTTPS with self-signed certificates
• ✅ Persistent metrics storage via volume mounting

📊 Performance Benchmarking

1. Connect your phone as described above
2. Click "Run 30s Bench" button in the viewer
3. Wait 30 seconds for automatic metrics collection
4. Results saved to metrics.json with:
   • End-to-end latency (median & P95)
   • Processing FPS
   • Network bandwidth (up/down kbps)

🛠 Development Setup

# Install dependencies
npm install

# Run in development
npm run dev

# Generate HTTPS certificates (for mobile testing)
HTTPS=true npm start

📋 Requirements

• Desktop: Modern browser with WebRTC support
• Phone: Mobile browser with camera access (HTTPS required for non-localhost)
• Network: Both devices on same network, or HTTPS setup for remote access

🔒 HTTPS Setup (Mobile Access)

For mobile devices not on localhost, HTTPS is required:

# Auto-generate self-signed certificate
HTTPS=true npm start

# Or provide your own certificates
HTTPS=true HTTPS_KEY=path/to/key.pem HTTPS_CERT=path/to/cert.pem npm start

🏗 Architecture

• Frontend: Vanilla JS with MediaPipe Tasks Vision / TensorFlow.js
• Backend: Node.js + Socket.IO for WebRTC signaling
• Detection: MediaPipe (primary) with TFJS fallback
• Streaming: WebRTC peer-to-peer with server-side TURN coordination

📈 Performance Features

• Adaptive Quality: Automatic resolution scaling based on performance
• Dual Detection Modes: MediaPipe (fast) → TensorFlow.js (compatibility)
• E2E Latency Tracking: Frame-level timestamping for accurate measurements
• Bandwidth Monitoring: Real-time network usage tracking

---

� Technical Report

Design Choices

WebRTC Peer-to-Peer Architecture
Selected WebRTC for ultra-low latency streaming (sub-100ms) between phone and browser. The server acts purely as a signaling relay using Socket.IO, eliminating bandwidth bottlenecks while maintaining real-time performance. This design scales horizontally as the server only handles session coordination, not media streams.

Dual Detection Engine Strategy
Implemented MediaPipe WASM as primary detector with automatic TensorFlow.js fallback. MediaPipe delivers 2-3x faster inference (60fps vs 20fps) but requires cross-origin isolation. TensorFlow.js ensures broad compatibility across all browsers and devices. Dynamic fallback provides optimal performance where possible, universal compatibility elsewhere.

Frame-Level Latency Measurement
Custom WebRTC data channel implementation tracks true end-to-end latency by timestamping frames at capture source and correlating at display destination. This provides accurate performance metrics for capture → network → processing → display pipeline, essential for real-time optimization.

Low-Resource Mode Implementation

Adaptive Resolution Scaling
Input frames automatically downscaled to maximum 320px (longest dimension) while preserving aspect ratio. Reduces compute load by ~75% with minimal accuracy impact. Processing resolution calculated dynamically:

const maxDim = 320;
const ratio = Math.min(maxDim / srcW, maxDim / srcH);
targetW = Math.round(srcW * ratio);
targetH = Math.round(srcH * ratio);

Detection Frequency Throttling
Implements skip-on-busy pattern - only processes new frames when detector is idle. Under CPU pressure, automatically reduces detection frequency (60fps → 30fps → 15fps → 10fps) while maintaining smooth video stream quality.

Memory Management

• Circular buffers for metrics collection prevent memory leaks during long sessions
• Frame timestamp correlation uses Map with automatic cleanup of stale entries
• Offscreen canvas reuse eliminates garbage collection pressure
• WebRTC connection state monitoring triggers cleanup on disconnection

Backpressure Policy

Skip-Frame Strategy
When detector is processing previous frame, skip current frame rather than queue building. Maintains real-time responsiveness over processing completeness:

if (detecting) return; // Skip frame if still processing
detecting = true;
const result = await detector.detect(canvas);
detecting = false;

Cascading Quality Degradation

1. CPU Pressure: Reduce detection FPS (60→30→15→10→5)
2. Memory Pressure: Decrease input resolution (320→240→160px)
3. Network Pressure: Lower video bitrate via WebRTC adaptation
4. Critical Failure: Circuit breaker temporarily disables detection

Circuit Breaker Pattern
After 5 consecutive detection failures, disable processing for 10 seconds to prevent cascade failures. Re-enable automatically after recovery period or manual trigger:

if (consecutiveFailures >= 5) {
  updateDetectionStatus("Circuit Breaker - Recovering");
  setTimeout(() => resetDetection(), 10000);
}

Network Adaptation
Monitor WebRTC bandwidth statistics and automatically adjust video constraints when connection becomes constrained. Prioritizes smooth streaming over detection accuracy during network pressure.

Performance Characteristics

• End-to-End Latency: 80-150ms (local network)
• Detection Throughput: 30-60 FPS (MediaPipe), 15-25 FPS (TensorFlow.js)
• Memory Usage: <100MB steady state with automatic cleanup
• CPU Usage: 15-30% on modern devices with adaptive throttling
• Network Bandwidth: 500-2000 kbps video + 5-10 kbps signaling