The broadcast booth in a browser tab. Six Gemini agents narrate any cricket screen, in any language, and sign every word in Indian Sign Language — live.
Built solo by Arun AK for Build With AI :: Agentic Premier League — GDG Cloud New Delhi · 03 May 2026.
🔗 Live demo: https://agentic-live.vercel.app 📱 Caster (point your camera at the match): https://agentic-live.vercel.app/caster 👥 Watch party (share with friends): https://agentic-live.vercel.app/viewer/ipl-room
Problem Statement 2: "Generate live commentary when you point your camera at a match. Broadcast it to multiple devices for people on a video call."
India has 1.8 crore Deaf cricket fans. They get ISL commentary on maybe 10 marquee broadcast matches a year. AGENTIC.LIVE delivers it on every match — gully cricket, school finals, your TV at home, the live screening at this hackathon.
The flow is unique:
🎤 phone mic → Gemini transcribes → Gemini generates ISL gloss
→ 3D signing avatar signs in real time
The avatar is listening to the actual broadcast audio, not reading our app's internal text. That's a real accessibility tool, not a demo trick. A Deaf viewer can join the watch-party link, watch the avatar sign what the commentators are saying, and follow along — without anyone in the room having to interpret manually.
This pipeline runs in parallel with the AI commentary pipeline, so even when our Vision agent has nothing to say, Vaani is still signing the live broadcast.
Point your phone or laptop camera at any cricket broadcast (TV / projector / live screening / actual stadium). Six Gemini agents take over:
| Agent | Job |
|---|---|
| 👁️ Vision | Watches video frames every 3 seconds. Generates Hinglish commentary in your chosen persona's voice (Bhogle / Shastri / Sidhu). |
| 📊 Stat | OCRs the on-screen scoreboard. Grounds the Vision agent so it can't hallucinate scores. |
| 🎬 Director | Decides if Vision's draft goes out, gets rewritten with stat context, or held. Prevents stale lines, cuts spam. |
| 👏 Crowd | Reads viewer emoji reactions in 5-second windows and tells the Director if the room is on fire. |
| 🌍 Translator | Per-viewer language fan-out — English, Hindi, Tamil, Bengali — so the same room serves a multilingual audience instantly. |
| 🤟 ISL Listener | Listens to the broadcast audio via mic, transcribes via Gemini, generates ISL gloss, signs through a 3D avatar. |
The commentary fans out via PartyKit to a watch-party room — anyone scans the QR / clicks the share invite and joins on their phone with their preferred language and emoji reactions.
The event is named Agentic Premier League. We literalize it — six agents argue on screen in a live dashboard:
- See the Director issue
HOLD/REWRITE/APPROVEdecisions. - See Crowd read 8 fire emojis in 5 seconds and tell Director "the room is losing it".
- See Stat OCR confidence dip and Vision fall back to generic descriptors instead of hallucinated names.
- See ISL Listener transcribe the live audio and Vaani sign it within 4 seconds.
The agentic story is visible, not invisible.
- Real accessibility, not just a feature. ISL audio→sign on any match — uncovered space. Star Sports 3 has human ISL interpreters but only on marquee broadcasts; AGENTIC.LIVE works for every match, every language, every screen.
- Cited research framing. MatchTime (EMNLP 2024 Oral) measured human commentators lag live action by 3–7 seconds. LiveCC (CVPR 2025) is the first streaming-trained video LLM. We use both as the academic punch behind "AI commentary that's faster and more grounded than human."
- Multi-agent orchestration is on screen. Judges see Director/Crowd/Stat/Vision/Translator/ISL Listener cards updating in real time on the dashboard — proof of agentic architecture, not just chat-with-PDF dressed up.
- Per-viewer language fan-out. Two judges, two phones, two different languages, one room — sub-second propagation via PartyKit.
- Persona toggle. Bhogle (calm authority), Shastri (booming catchphrases), Sidhu (theatrical Punjabi shayari) — instant swap, no reconnect.
- Anybody can use it in 30 seconds. Open the link, scan the QR, point your phone at any cricket screen. No install, no signup.
┌──────────────────────────────────────────────┐
│ PHONE OR LAPTOP CAMERA (back camera unlocked)│
│ + microphone (for ISL listener) │
└──────────┬──────────────────┬─────────────────┘
│ frames @ 3s │ audio chunks @ 6s
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ VISION AGENT │ │ ISL LISTENER │
│ Gemini Flash │ │ Gemini Flash │
│ + persona prompt│ │ audio → gloss │
└────────┬─────────┘ └─────────┬────────┘
│ │
┌──────────┐ ┌────────▼─────────┐ ┌─────────┐│
│ STAT │─▶│ DIRECTOR │◀─│ CROWD ││
│ (OCR) │ │ APPROVE/REWRITE/│ │ (emojis)││
└──────────┘ │ HOLD │ └─────────┘│
└─────────┬────────┘ │
│ │
┌──────────▼──────────┐ │
│ TRANSLATOR │ │
│ Hinglish · English │ │
│ · Hindi · Tamil │ │
│ · Bengali │ │
└──────────┬──────────┘ │
│ │
┌──────────▼─────────┐ ┌─────────▼──────────┐
│ PartyKit room │ │ iframe(vaani-gold/│
│ fanout to viewers │ │ embed) postMessage│
│ + reactions back │ │ → 3D ISL avatar │
└────────────────────┘ └────────────────────┘
🔊 Browser TTS speaks Director-approved commentary in selected persona's voice.
📱 Each viewer phone shows captions in viewer's chosen language — sub-second.
🤟 Vaani avatar signs continuously based on broadcast audio.
- Next.js 16 (App Router, React 19, TypeScript strict)
- Tailwind v4 + shadcn/ui
@google/genaiv1.51 — Gemini 2.5 Flash (vision + sidecars + audio→ISL)- PartyKit on Cloudflare Workers — sub-second pub/sub for the watch-party room
- Browser SpeechSynthesis — persona-mapped voices, free, instant, runs entirely client-side
- Vaani — 3D ISL signing avatar, integrated via its
/embedpostMessage protocol - Bun 1.3 package manager
- Vercel + Cloudflare deploys
bun install
cp .env.local.example .env.local # add your Gemini API key from aistudio.google.com
bunx partykit dev # starts the room server on :1999
bun run dev # starts Next on :3000Open http://localhost:3000/caster, click Start The Booth, allow camera + mic, point at a cricket video.
Open http://localhost:3000/viewer/ipl-room on a phone (same WiFi) to join the watch party.
For phone testing, you need HTTPS — use the deployed Vercel URL or bunx ngrok http 3000.
(walk on, hold up phone) "Two papers from 2024 and 2025 proved human commentators lag live action by 3 to 7 seconds. I built a 6-agent AI booth that's faster than humans, multilingual, and signs in ISL — by listening to the actual broadcast audio."
(tap Start The Booth on the projector laptop, point at the IPL screening) "Vision watches frames. Director gates each line. Stat reads the scoreboard. Crowd reads emoji reactions from viewer phones."
(let Bhogle speak one line — listen)
(hand judge a phone with viewer URL) "Scan this QR. Pick Tamil. Same room, different language, sub-second."
(switch persona dropdown to Sidhu, let it speak) "Persona swap, instant. Guru, what a delivery!"
(point to ISL Listener card on dashboard, then to Vaani avatar pane) "And on the right — the ISL Listener agent is listening to the broadcast audio, transcribing it, generating Indian Sign Language gloss. The 3D avatar signs in real time for 1.8 crore Deaf cricket fans who don't have ISL on every match."
(tap fire emoji on viewer phone, point to Crowd Agent card lighting up on caster) "Crowd agent just told the Director the room is losing it. Director will hype the next line."
"Built solo on Gemini Flash, PartyKit, and an integrated ISL signing avatar. Live URL on the QR. AGENTIC.LIVE."
- Event: Build With AI :: Agentic Premier League · #BWAI-APL-DELHI
- Tagged: GDG Cloud New Delhi · GDG New Delhi
- Author: Arun AK · CSE 2nd Year · Thapar Institute of Engineering and Technology
- Live demo: https://agentic-live.vercel.app
- Vaani — 3D ISL signing avatar. Integrated via its
/embedpostMessage protocol. Live at vaani-gold.vercel.app. - Reference code lifted from
google-gemini/live-api-web-console(Apache-2.0): audio worklets + Live API client patterns. - Research cited in the demo script: MatchTime (Rao et al., EMNLP 2024) and LiveCC (Chen et al., CVPR 2025).