An agent that watches your screen as you research, remembers what you read, and acts — finding papers, writing code, and building literature reviews on demand.
Built for the VideoDB Global Online Hackathon, May 2026.
Researchers context-switch constantly. You are reading a paper, a concept comes up you want to implement. You switch to a browser, lose your place, spend twenty minutes searching, open five tabs, and forget what you were originally trying to do. You watch a lecture and a technique is mentioned you want to explore further, but the moment passes and the video keeps going. The information exists. The tools exist. But nothing connects them to what you are actually doing right now.
ResearchFlow is built around one idea: the agent should know what you are working on without you having to explain it. It watches your screen, listens to your mic, and builds a live, searchable memory of your session. When you need it, you say a word or press a key and it acts. The interaction takes two seconds. The output takes thirty.
ResearchFlow runs in two integrated modes that share a single agent brain.
Video Studio is a research video workspace. You paste any YouTube lecture, conference talk, or direct video URL. The system uploads it to VideoDB, runs Whisper transcription across the full audio track, and runs Gemma 4 31B visual scene indexing across the video frames. Once indexed, the video lives in a searchable memory. You can watch it inline with a live timestamp tracker, pause at any moment and ask what was just explained, search for any concept across both the spoken transcript and the visual content simultaneously, clip any time range into a shareable segment, chain multiple clips into a compiled highlight reel, and download any output as a structured markdown document. All of this happens through a chat interface where you type naturally and the agent decides which operations to run.
Research Agent is the live session layer. It runs alongside everything else you are doing on your computer. You start a capture session from the browser, run the capture client in a second terminal, and from that point the agent is watching and listening continuously. Your screen is being indexed every ten seconds. Your mic is being transcribed in twenty-second rolling batches. When you want the agent, you say "flow" and keep talking, or press F and speak. The agent detects your query, grabs the current visual context from your screen, pulls the recent transcript, and fires a full agentic call. It responds in the chat panel and speaks the answer back to you through pyttsx3 TTS. You never leave what you were reading.
The agent can implement algorithms in Python with referenced papers, write structured literature reviews and save them as downloadable documents, search arXiv for recent preprints, search Semantic Scholar for citations and related work, find YouTube lectures on a topic, look up background on Wikipedia, and run general web searches through DuckDuckGo. Every capability is available in both modes.
VideoDB is the core infrastructure for everything that involves media in this project. Below is the complete list of SDK operations used and what they power.
Connection and collection
videodb.connect() establishes the authenticated connection at server startup. All video and stream operations go through conn.get_collection() which provides the default collection context.
Video ingestion
coll.upload(url=url) handles both YouTube URLs and direct video links, returning a video object with id, name, length, and stream URL. This is called whenever a user pastes a URL in Video Studio.
Spoken word indexing
video.index_spoken_words() runs Whisper transcription on the full audio track. The returned index ID is stored in server state and passed to every subsequent search call. This is what enables natural language search over what was said at any point in the video.
Visual scene indexing
video.index_scenes() runs with SceneExtractionType.time_based, extraction_config set to one frame every ten seconds with two keyframe selections per interval, model_name="google/gemma-4-31B-it", and the sandbox ID from the provisioned medium tier sandbox. This is what enables searching for visual content — diagrams, code on screen, paper figures, and slide text — not just spoken words.
Semantic search
video.search() is called with SearchType.semantic and either IndexType.spoken_word or IndexType.scene depending on the query. Both indexes are searched for every query in Video Studio and results are merged, deduplicated, and ranked by search score. Each result returns start time, end time, matched text, search score, and a direct stream URL to that clip.
coll.search() runs collection-wide search across all indexed videos for cross-video queries.
Clipping and compilation
video.generate_stream(timeline=[(start, end)]) generates a playable stream URL for any time range. Single clips and multi-segment highlight reels both use this, with the timeline parameter accepting a list of tuples for compiled reels.
Transcript retrieval
video.get_transcript() returns the full timestamped transcript as a list of dicts. ResearchFlow uses this for full-video summarisation and for the timestamp-windowed context retrieval when the user pauses and asks about a specific moment.
Sandbox management
conn.create_sandbox(tier="medium") provisions a GPU sandbox for running Gemma 4 31B scene indexing and Whisper transcription. conn.list_sandboxes() is called first to reuse any existing active sandbox rather than provisioning a new one on every boot. sandbox.wait_for_ready() blocks until the sandbox is confirmed ready before accepting indexing jobs.
Capture session
conn.create_capture_session() creates a desktop capture session with an end user ID and metadata. conn.generate_client_token(expires_in=7200) generates the short-lived token that the capture client uses to authenticate and begin streaming. The session ID and token are displayed in the Research Agent tab for the user to paste into the capture client terminal.
RTStream indexing
coll.get_rtstream(rtstream_id) retrieves the live stream object for a given stream ID. rtstream.index_audio() starts continuous audio indexing with Whisper in twenty-second batches. rtstream.index_visuals() starts continuous visual indexing with Gemma 4 31B in ten-second batches with one frame per interval. Both return an index ID stored in server state.
cap.get_rtstream("mic") and cap.get_rtstream("display") retrieve the mic and display RTStream objects from an active capture session. These are called after the capture client confirms it is streaming, with a retry loop of up to five attempts spaced three seconds apart to allow streams to register.
RTStream search
rtstream.search(query=query, index_id=index_id) searches the live session memory for anything matching a natural language query. This is what powers the search_live_session tool — when you ask "what was I just reading" or "find the part where they described the gradient update", it searches across everything indexed so far in the session.
ResearchFlow/
├── app.py # FastAPI server, REST endpoints, WebSocket broadcast
├── capture.py # Desktop capture client, wake word, F-key, TTS
├── agent.py # LangChain ReAct agent, Gemini 2.0 Flash, tool routing
├── tools.py # 14 agent tools across video, papers, web, documents
├── videodb_client.py # All VideoDB SDK operations, isolated from agent logic
├── sandbox_manager.py # Singleton sandbox lifecycle, reuse across requests
├── static/
│ └── index.html # Single-file frontend, YouTube IFrame API, marked.js
└── requirements.txt
FastAPI + WebSocket event bus
The server is a FastAPI application with a single WebSocket endpoint that all connected browser tabs subscribe to. Background threads (video indexing, capture indexing) cannot directly call async functions, so broadcast uses asyncio.run_coroutine_threadsafe(_broadcast(event), _main_loop) where _main_loop is captured at startup via @app.on_event("startup"). This avoids the common mistake of calling asyncio.run() inside a thread that already has a running loop, which would crash. The browser receives typed events: ingest_progress, ingest_done, ingest_error, live_event, and voice_response.
Agent loop and tool routing
The agent uses LangChain's ChatGoogleGenerativeAI with bind_tools() and a manual ReAct loop. The loop runs up to six iterations, calling tools and appending ToolMessage results until the model returns a response with no tool calls. If the loop hits the iteration limit, a final HumanMessage forces a concluding answer.
The critical design decision is intent-based tool routing. Gemini rejects requests with duplicate function declarations and performs worse with large tool lists. Rather than passing all fourteen tools on every call, _route_tools() does keyword matching on the prompt and returns four to six relevant tools per query. Implementation queries get search_live_session, search_arxiv, search_papers, search_web, and generate_document. Literature review queries get search_papers, search_arxiv, search_wikipedia, and generate_document. Lecture queries get search_youtube_lectures, search_arxiv, and search_web. This keeps each model call focused and prevents the duplicate-declaration errors that appeared when all tools were passed simultaneously.
All tools are built as closures inside build_all_tools(context). The context dict containing the live conn, sandbox ID, video ID, index IDs, current timestamp, and live transcript buffer is captured at closure creation time. This means tools always operate on the correct session state without globals.
Gemini 2.0 Flash
The agent uses gemini-2.0-flash-001 for both Video Studio and Research Agent. Earlier versions of the project used gemini-2.5-pro and gemini-2.5-flash, both of which are thinking models that require thought_signature preservation in tool call messages. LangChain's Google integration does not handle this correctly, causing failures on the second tool call in any multi-step chain. gemini-2.0-flash-001 is the stable non-thinking build with reliable tool-calling across all iterations.
_extract_text() for LangChain content blocks
Gemini occasionally returns response.content as a list of typed content blocks rather than a plain string. This is the raw multi-modal response format. Without handling it, the content gets serialised as a Python list and reaches the browser as [object Object] and crashes the TTS thread with AttributeError: 'list' object has no attribute 'replace'. The _extract_text() function handles both cases: if content is a string it passes through directly, if it is a list it extracts all blocks with type == "text" and joins them.
Capture client state machine
The capture client uses a three-state machine (idle, triggered, capturing) to prevent mic contention. The background listen loop runs continuously in one thread, entering with mic as source on each iteration. When the wake word is detected or F-key is pressed, the state transitions to triggered and a trigger event is placed on a queue. A separate trigger watcher thread picks this up and calls _capture_and_fire(), which waits 600ms before entering its own with mic as source block. This wait gives the background thread time to fully exit its context manager. Without it, the capture attempt immediately throws AudioSource already inside context manager.
The TTS engine runs in a dedicated thread with its own pyttsx3.init() instance and a queue.Queue() for text. Calling pyttsx3 from multiple threads crashes the underlying COM objects on Windows. The dedicated thread model serialises all speech output safely.
YouTube IFrame API timestamp tracking
The Video Studio player uses the YouTube IFrame Player API rather than a plain iframe. onYouTubeIframeAPIReady creates a YT.Player instance with an onStateChange handler that fires when the video is paused. On pause, ytPlayer.getCurrentTime() gives the exact timestamp in seconds, which is sent to /api/video/timestamp and stored in server state. A three-second polling interval keeps the timestamp current while playing. This means when the user pauses and asks a question, the agent's get_context_at_timestamp tool receives the precise moment, and get_transcript_at() returns the transcript within a ninety-second window around that point.
Sandbox singleton
SandboxManager is a singleton that persists across all requests in the server process. ensure_active() first calls conn.list_sandboxes() and checks for any existing active sandbox before provisioning a new one. This avoids burning sandbox credits on every server restart during development. The active check uses a safe attribute probe across is_active, active, and status to handle varying attribute names across SDK versions.
Requirements
Python 3.10 or later. A microphone and display are required for Research Agent mode.
Install
cd ResearchFlow
pip install -r requirements.txtIf pyaudio fails on Windows:
pip install pipwin
pipwin install pyaudioConfigure
Create a .env file in the project root:
VIDEODB_API_KEY=your_videodb_key
GOOGLE_API_KEY_2=your_primary_gemini_key
GOOGLE_API_KEY=your_fallback_gemini_key
VideoDB API key: https://console.videodb.io Gemini API keys: https://aistudio.google.com
# Terminal 1
python app.py
# Open http://127.0.0.1:7860Video Studio:
- Click Boot Sandbox in the header and wait for confirmation
- Paste a YouTube URL and click Ingest
- Wait for spoken word and scene indexing to complete (2 to 5 minutes)
- Chat with the agent about the video, pause to ask about the current moment, request clips or highlight reels
Research Agent:
- Boot the sandbox
- Click Start Capture Session and note the session ID and token
- Open a second terminal and run:
python capture.py- Paste the session ID and token when prompted
- Wait for the spoken confirmation "ResearchFlow ready"
- Say "flow" and ask your question, or press F and speak
- The agent responds in the chat and by voice
- Press S to stop capture
| Tool | What it does |
|---|---|
search_video |
Semantic search over spoken words and visual scenes in the current video |
get_context_at_timestamp |
Transcript window around the current pause point |
get_full_transcript |
Full timestamped transcript for summarisation or code extraction |
clip_segment |
Generate a playable stream URL for any time range |
compile_highlights |
Chain multiple clips into a single compiled stream |
search_live_session |
Search indexed audio and visual memory from the live capture session |
search_papers |
Semantic Scholar search returning titles, authors, abstracts, PDF links |
search_arxiv |
arXiv preprint search sorted by submission date |
search_youtube_lectures |
YouTube search for research lectures and talks |
search_wikipedia |
Wikipedia article retrieval with search fallback |
search_web |
DuckDuckGo instant answer and related topic search |
generate_literature_review |
Multi-source paper gathering formatted for structured synthesis |
generate_document |
Save any generated content as a downloadable markdown file |
ingest_youtube |
Add a YouTube video to the library and index it mid-session |
Python, FastAPI, VideoDB SDK (hackathon branch), LangChain, Gemini 2.0 Flash, Gemma 4 31B, Whisper large-v3-turbo, pyttsx3, SpeechRecognition, YouTube IFrame API, Semantic Scholar API, arXiv API, Wikipedia REST API, DuckDuckGo Instant Answer API