G-Axis — AI Browser Agent with Supervised Autonomy

G-Axis is a Chrome Extension + Cloud backend that lets you talk to your browser. It sees your screen through Gemini Vision, plans actions, and executes them — while you stay in control through approval gates for risky actions.

Not a web app. Not a chatbot. A real browser agent.

What Makes G-Axis Different

Feature	Traditional Automation	G-Axis
Input	Scripts / CSS selectors	Voice + text commands
Perception	DOM parsing (fragile)	Gemini Vision (screenshot-based)
Execution	Blind clicking	Visual understanding with confidence scoring
Safety	None	Policy engine with approval gates
Memory	Stateless	Remembers past tasks, learns patterns
Architecture	Monolithic script	Agent ecosystem (A2A, MCP, Agent Card)

Architecture

Chrome Extension (Side Panel + Content Script)
    |
    | WebSocket
    v
Cloud Run Backend (FastAPI + ADK)
    |
    +-- Gemini 2.5 Pro (Vision + Planning)
    +-- Policy Engine (risk scoring, approval gates)
    +-- Memory System (episodic + semantic)
    +-- Audit Store (Firestore)
    +-- Playwright (API mode)

Agent Stack Layers

Layer	Implementation
Agent Client	Chrome Extension (side panel + content script)
A2UI	WebSocket event stream -> reactive UI
A2A	REST API + Agent Card at `/.well-known/agent.json`
Agent Runtime	ADK + Coordinator FSM
Tool Access	MCP-compatible browser tools
Context/Memory	3-tier: Working + Episodic + Semantic
Infrastructure	Cloud Run + Firestore + Secret Manager

Quick Start

Prerequisites

Python 3.11+
Google Chrome
Gemini API key (get one here)

1. Backend Setup

# Clone the repo
git clone https://github.com/preethams/gaxis.git
cd gaxis

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -e .
playwright install chromium

# Set your API key
cp .env.example .env
# Edit .env and add your GOOGLE_API_KEY

# Run the server
python run.py

The server starts at http://localhost:8000. Verify with:

curl http://localhost:8000/health
# {"status": "ok", "agent": true}

2. Chrome Extension Setup

Open Chrome and go to chrome://extensions
Enable "Developer mode" (top right)
Click "Load unpacked"
Select the extension/ folder from this repo
Click the G-Axis icon in your toolbar — the side panel opens
Open Settings (gear icon) and set Backend URL to http://localhost:8000
Click "Test" to verify connection

3. Use It

Type a task: "Search Google for trending AI news"
Or click the mic button and speak
Watch the agent analyze the page, plan actions, and execute them
Approve or deny risky actions when prompted

Cloud Deployment (Google Cloud)

Automated (recommended)

# Set up GCP credentials
gcloud auth login
gcloud config set project YOUR_PROJECT_ID

# Create the API key secret
echo -n 'YOUR_GEMINI_API_KEY' | gcloud secrets create gaxis-gemini-key --data-file=-

# Deploy
./deploy.sh YOUR_PROJECT_ID

Terraform (Infrastructure as Code)

cd terraform
terraform init
terraform apply -var="project_id=YOUR_PROJECT_ID" -var="gemini_api_key=YOUR_KEY"

Update Extension

After deploying, update the Backend URL in extension settings to your Cloud Run URL: https://gaxis-XXXXX.run.app

API Reference

REST Endpoints

Method	Path	Description
GET	`/health`	Server health check
GET	`/.well-known/agent.json`	Agent Card (A2A discovery)
POST	`/api/task`	Start a new task
GET	`/api/memory/{domain}`	Get episodic memory for a domain
GET	`/api/patterns`	Get semantic patterns

WebSocket (`/ws`)

Send JSON messages:

{"type": "run_task", "instruction": "Search for flights to Tokyo", "mode": "extension"}
{"type": "approve"}
{"type": "deny"}
{"type": "screenshot", "screenshot": "<base64>", "url": "...", "title": "..."}

Receive events:

{"type": "task_started", "task_id": "task_abc123", "data": {...}}
{"type": "perception", "task_id": "...", "data": {"page_summary": "...", "elements": [...]}}
{"type": "action_planned", "task_id": "...", "data": {"action_type": "click", "x": 640, "y": 360, ...}}
{"type": "approval_needed", "task_id": "...", "data": {"risk_level": "high", "reason": "..."}}
{"type": "task_completed", "task_id": "...", "data": {"summary": "..."}}

Memory System

G-Axis remembers across sessions:

Episodic Memory: Per-domain task history. "Last time on Google Flights, the date picker needed a double click."
Semantic Memory: Cross-site patterns. "Cookie banners usually have an 'Accept All' button."
Working Memory: Current task state (dies with the task).

5 built-in patterns ship with G-Axis (cookie consent, search inputs, login flows, pagination, popup close).

Safety & Supervision

Three supervision modes:

Supervised: Approve all risky actions (default)
Semi-automatic: Only approve critical actions (passwords, payments)
Autonomous: No approvals (use with caution)

The policy engine evaluates every action for:

Risk level (none / low / medium / high / critical)
Confidence score (below threshold = require approval)
Cumulative risk (many low-risk actions in sequence = escalate)
Sensitive field detection (password, payment, PII)

Tech Stack

Component	Technology
AI Model	Gemini 2.5 Pro (vision + planning)
Agent Framework	Google ADK
Backend	Python + FastAPI + WebSocket
Browser Control	Playwright
Extension	Chrome Manifest V3
Database	Google Cloud Firestore
Deployment	Google Cloud Run
IaC	Terraform

Project Structure

gaxis/
├── backend/
│   ├── agent/core.py          # Agent coordinator with dual-mode execution
│   ├── vision/perception.py   # Gemini Vision screenshot analysis
│   ├── browser/runtime.py     # Playwright browser control
│   ├── policy/engine.py       # Risk scoring + approval gates
│   ├── memory/store.py        # 3-tier memory system
│   ├── audit/store.py         # Firestore audit logging
│   └── main.py                # FastAPI server + WebSocket + A2A
├── extension/
│   ├── manifest.json           # Chrome MV3 manifest
│   ├── background/             # Service worker (WS hub)
│   ├── sidepanel/              # Side panel UI
│   ├── content/                # Page overlays + action execution
│   ├── options/                # Settings page
│   └── shared/                 # Shared types
├── terraform/main.tf           # GCP infrastructure as code
├── deploy.sh                   # One-command Cloud Run deployment
├── Dockerfile                  # Cloud Run container
└── pyproject.toml              # Python dependencies

Reproducible Testing Instructions

Prerequisites

Google Chrome (latest)
Python 3.11+ (for local backend, optional if using Cloud Run)
A Gemini API key from Google AI Studio

Option A: Use Hosted Backend (Easiest)

No local setup needed — the backend is already deployed on Cloud Run.

Step 1: Install the Chrome Extension

git clone https://github.com/preethamtjit20-spec/gaxis.git
cd gaxis

Open Chrome → go to chrome://extensions
Enable Developer mode (toggle, top right)
Click Load unpacked → select the extension/ folder
G-Axis icon appears in your toolbar

Step 2: Open G-Axis

Click the G-Axis icon → side panel opens
The green dot indicates the backend is connected
The default backend URL points to: https://gaxis-132388856648.us-central1.run.app

Step 3: Test Voice Chat

Click the mic button (bottom of sidepanel)
A small popup window opens for mic permission — click Allow
Select a persona from the dropdown (e.g., "Friendly Buddy")
Speak naturally — G-Axis responds with voice in real-time
Try: "What's the latest news in AI?" (uses Google Search for live data)
Switch persona mid-conversation to hear different voices/personalities
Click End to finish — view analytics in My Progress

Step 4: Test Browser Automation

In the text input, type: Plan a 5-day Japan itinerary for first-timers
Press Enter — watch the activity timeline as the agent researches
A .docx document is generated with the full itinerary
Try: Schedule a team meeting tomorrow at 10am (requires Google Calendar login)

Step 5: Test Dashboard

After completing voice sessions, click My Progress
View: Level, XP, Streak, Communication Skills (5 metrics), Weekly Activity, Recent Conversations with AI-generated summaries

Option B: Run Everything Locally

# Clone
git clone https://github.com/preethamtjit20-spec/gaxis.git
cd gaxis

# Setup Python environment
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
playwright install chromium

# Set your API key
cp .env.example .env
# Edit .env → add your GOOGLE_API_KEY=AIza...

# Start backend
python run.py
# Server runs at http://localhost:8000

Then install the extension (same as Option A, Step 1).

In G-Axis Settings (gear icon), change Backend URL to http://localhost:8000.

Option C: Deploy Your Own Backend

# Authenticate with GCP
gcloud auth login
gcloud config set project YOUR_PROJECT_ID

# Create API key secret
echo -n 'YOUR_GEMINI_API_KEY' | gcloud secrets create gaxis-gemini-key --data-file=-

# Deploy (automated script)
./deploy.sh YOUR_PROJECT_ID

# Or use Terraform
cd terraform
terraform init
terraform apply -var="project_id=YOUR_PROJECT_ID" -var="gemini_api_key=YOUR_KEY"

Update Backend URL in extension settings to your Cloud Run URL.

What to Test

Feature	How to Test	Expected Result
Voice Chat	Click mic → speak	Real-time voice response with live transcript
Persona Switch	Change dropdown during voice session	New voice + personality, previous session saved
Google Search	Ask about current events in voice	Agent searches web and answers with fresh data
Browser Automation	Type "Plan a Japan trip"	Agent researches, generates .docx document
Calendar	Type "Schedule meeting tomorrow 10am"	Agent opens Calendar, fills form
Dashboard	Click "My Progress" after sessions	Level, XP, skills, weekly activity, session summaries
8 Personas	Switch between all personas	Different voices (Puck, Charon, Aoede, Kore, Fenrir)

Troubleshooting

Issue	Fix
Red dot (disconnected)	Check Backend URL in Settings. Try: `https://gaxis-132388856648.us-central1.run.app`
Mic not working	Close and reopen the mic popup. Check Chrome mic permissions at `chrome://settings/content/microphone`
Voice but no response	Check internet connection. Gemini Live requires stable connectivity
"Cannot connect to backend"	Backend may be cold-starting — wait 10 seconds and retry
Extension errors	Reload at `chrome://extensions` (click the refresh icon on G-Axis)

License

MIT

Built for the Gemini Live Agent Challenge #GeminiLiveAgentChallenge

Name		Name	Last commit message	Last commit date
Latest commit History 39 Commits
backend		backend
extension		extension
terraform		terraform
.env.example		.env.example
.gitignore		.gitignore
CLOUD_DEPLOYMENT.md		CLOUD_DEPLOYMENT.md
DEMO_SCRIPT.md		DEMO_SCRIPT.md
DEVPOST.md		DEVPOST.md
Dockerfile		Dockerfile
G-Axis_Technical_Documentation.docx		G-Axis_Technical_Documentation.docx
README.md		README.md
SUBMISSION.md		SUBMISSION.md
architecture-simple.xml		architecture-simple.xml
architecture-v3.png		architecture-v3.png
architecture-v3.xml		architecture-v3.xml
architecture.xml		architecture.xml
deploy.sh		deploy.sh
pyproject.toml		pyproject.toml
run.py		run.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

G-Axis — AI Browser Agent with Supervised Autonomy

What Makes G-Axis Different

Architecture

Agent Stack Layers

Quick Start

Prerequisites

1. Backend Setup

2. Chrome Extension Setup

3. Use It

Cloud Deployment (Google Cloud)

Automated (recommended)

Terraform (Infrastructure as Code)

Update Extension

API Reference

REST Endpoints

WebSocket (`/ws`)

Memory System

Safety & Supervision

Tech Stack

Project Structure

Reproducible Testing Instructions

Prerequisites

Option A: Use Hosted Backend (Easiest)

Option B: Run Everything Locally

Option C: Deploy Your Own Backend

What to Test

Troubleshooting

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

G-Axis — AI Browser Agent with Supervised Autonomy

What Makes G-Axis Different

Architecture

Agent Stack Layers

Quick Start

Prerequisites

1. Backend Setup

2. Chrome Extension Setup

3. Use It

Cloud Deployment (Google Cloud)

Automated (recommended)

Terraform (Infrastructure as Code)

Update Extension

API Reference

REST Endpoints

WebSocket (/ws)

Memory System

Safety & Supervision

Tech Stack

Project Structure

Reproducible Testing Instructions

Prerequisites

Option A: Use Hosted Backend (Easiest)

Option B: Run Everything Locally

Option C: Deploy Your Own Backend

What to Test

Troubleshooting

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

WebSocket (`/ws`)