Web Browser AI Agent 🌐🤖

A local, lightweight, and extremely fast Full-Stack Web Browser AI Agent built using a Chrome Extension (Manifest V3), FastAPI, and LangChain.

The agent operates directly on the user's active, logged-in browser session. Instead of running a headless browser via heavy frameworks like Selenium or Playwright, the agent communicates with the Chrome Extension over a local WebSocket connection, executing actions directly via the browser's own scripting API.

---

✨ Features

• In-Browser Execution: Automates tasks directly on your active, logged-in tab (e.g. adding items to a cart, filling out forms, searching pages).
• Gemini Minimalist UI: A stunning, modern dark-themed chat interface matching the Google Gemini chat client layout.
• Robust DOM Serialization: Automatically parses webpage DOM structures, filtering out non-interactive elements and tagging interactive nodes with a temporary data-agent-id attribute to guarantee 100% targeting accuracy.
• Modern LangChain Loop: Uses a custom tool-calling loop with llm.bind_tools and standard message streams, validated against the pinned LangChain 1.3 dependency line in backend/constraints.txt.
• Multi-Model Support: Pre-configured for DeepSeek (deepseek-chat), with seamless fallbacks to Google Gemini (gemini-1.5-flash), OpenAI (gpt-4o-mini), or Anthropic (claude-3-5-sonnet-latest).
• Persistent History: Every chat message and browser action is logged to a local SQLite database (backend/agent_history.db), retrievable via GET /sessions and GET /sessions/{id}.
• Context-Aware Agent Loop: Caps the number of DOM elements sent per step, collapses older DOM snapshots to keep context size bounded across long tasks, and remembers a short summary of prior tasks completed in the same session.
• Step-Budgeted Execution: MAX_AGENT_STEPS limits executed browser actions, not just model reasoning turns, so multi-action runs stop at a predictable budget.
• Expanded Browser Tools: Supports navigation, keyboard events, select controls, hover menus, browser history, reload, wait steps, and visible page-text reads in addition to click/input/scroll.
• Human Approval Guard: Sensitive actions such as submit, send, delete, checkout, payment, and credential-like input require explicit sidepanel approval by default.
• Sidepanel Settings & History Viewer: Configure the backend URL and optional auth token from the extension UI, then inspect recent persisted sessions without leaving the sidepanel.
• Optional Local Auth: Set AGENT_AUTH_TOKEN to protect the WebSocket and session history endpoints on shared machines.

---

📂 Project Structure

browser-agent/
├── extension/             # Chrome Extension (Frontend UI & Execution)
│   ├── manifest.json      # MV3 configuration & permissions
│   ├── sidepanel.html     # Minimalist chat sidebar layout
│   ├── sidepanel.js       # WebSocket manager & page message broker
│   ├── sidepanel.css      # Custom dark-theme stylesheet
│   ├── background.js      # Service worker configuring side panel behavior
│   └── content.js         # Page script for DOM parsing & event dispatching
│
└── backend/               # FastAPI Backend (LangChain Brain)
    ├── main.py            # WebSocket server endpoint, routing & history API
    ├── agent.py           # LangChain tool binding & Custom Agent Loop
    ├── database.py        # SQLite persistence for sessions/messages/actions
    ├── requirements.txt   # Python package dependencies
    └── .env.example       # Template for required environment variables

---

🛠️ Installation & Setup

1. Run the FastAPI Backend

1. Navigate to the backend folder:

   cd browser-agent/backend

2. Create and activate a virtual environment:

   python -m venv venv
   # On Windows:
   .\venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt -c constraints.txt

   The backend now ships with pinned dependency versions in constraints.txt and mirrored project metadata in the repo-level pyproject.toml.

4. Configure your API keys: Copy the provided template and fill in your values:

   cp .env.example .env

   LLM_PROVIDER=deepseek
   LLM_MODEL_NAME=deepseek-chat
   LLM_MAX_TOKENS=2048
   MAX_AGENT_STEPS=15
   ACTION_TIMEOUT_SECONDS=40
   MAX_DOM_ELEMENTS=150
   REQUIRE_ACTION_APPROVAL=true
   DEEPSEEK_API_KEY=your_deepseek_api_key_here

   Only the API key matching your chosen LLM_PROVIDER (gemini, openai, anthropic, or deepseek) is required. MAX_AGENT_STEPS counts executed browser tool calls, and MAX_DOM_ELEMENTS=150 should stay aligned with the extension-side DOM capture budget. See .env.example for the full list of supported variables.

5. Start the server:

   python -m backend.main

   The server starts listening on http://127.0.0.1:8000.

---

2. Install the Chrome Extension

1. Open Google Chrome and navigate to chrome://extensions/.
2. Toggle on Developer mode in the top-right corner.
3. Click Load unpacked in the top-left corner.
4. Select the browser-agent/extension folder.
5. Pin the Web Browser AI Agent extension to your toolbar.

The extension uses activeTab, scripting, tabs, and broad host access so it can keep acting after agent-driven cross-origin navigation in the active tab. Sensitive actions are still gated by the local approval guard. The current extension defaults wait about 900ms after tab-level navigation and 700ms after DOM actions before re-reading the page state.

---

🚀 How to Use

1. Go to any public website (e.g. https://google.com or https://codeforces.com).
2. Click the Web Browser AI Agent icon in your toolbar to open the sidebar.
3. Once the status shows Connected in green, type your instruction in the prompt box (e.g. "Search for DeepSeek on Google" or "List the next Codeforces contests").
4. Click Send and watch the agent navigate, click, type, and summarize findings in real-time.

---

🗄️ Session History

Every WebSocket connection is logged as a session in backend/agent_history.db (SQLite, created automatically on first run, or AGENT_DB_PATH if set). Two read-only endpoints expose it:

• GET /sessions?limit=20&offset=0 — lists sessions with pagination metadata.
• GET /sessions/{session_id} — returns the full list of chat messages and browser actions recorded for that session.

If AGENT_AUTH_TOKEN is set, pass it as X-Agent-Token or configure the same token in the sidepanel settings.

---

✅ Development Checks

python -m py_compile backend/main.py backend/agent.py backend/database.py backend/settings.py backend/schemas.py
python -m unittest discover backend/tests
node --check extension/content.js
node --check extension/sidepanel.js