Step-based Browser Agent

This project runs a Playwright-powered agent that splits your goal into steps, captures an accessibility snapshot for each step, plans actions, executes them, and caches the plan so subsequent runs reuse it.

Features

• Parse multi-step goals (e.g., "Step1: ... Step2: ...").
• For each step:
  • Capture an accessibility snapshot to accessibility_step<N>.json.
  • Plan actions via OpenAI (if configured) or fallback.
  • Cache plan in plannerCache.json keyed by host + step text.
  • Execute the plan. On later runs, skip execution if the step was already executed (unless forced).
• Automatically add a final close_browser step on the last step if missing.

Requirements

• Node.js 18+
• Playwright (installed via package.json)
• Optional: OpenAI API key for better planning

Setup

1. Install dependencies:

   npm install

2. Create a .env file (optional for OpenAI planning):

   OPENAI_API_KEY=your_openai_key
   OPENAI_MODEL=gpt-4o-mini
   HEADLESS=false
   # Set to true to force executing steps even if they've already been executed once
   FORCE_EXECUTE=false

Usage

Run the agent with a goal. If the goal contains Step1:, Step2: blocks, they'll be parsed as steps; otherwise, the entire goal is treated as a single step.

node index.js "Step1: Search ErrorsOverflow on YouTube. Step2: Open first result. Step3: Close browser."

• First run: captures snapshots, plans via LLM (if configured), executes steps, and caches the plans.
• Later runs: reuses cached plans and skips execution for already-completed steps unless FORCE_EXECUTE=true.

To force re-execution of all steps for the same goal:

$env:FORCE_EXECUTE="true"; node index.js "Step1: Search ErrorsOverflow on YouTube. Step2: Open first result. Step3: Close browser."

Or use convenient CLI flags:

# Force execution even if cache says executed
node index.js --force "Step1: Search ErrorsOverflow on YouTube. Step2: Open first result. Step3: Close browser."

# Mark cached steps as not executed (will execute next time they run)
node index.js --reset-executed "Step1: Search ErrorsOverflow on YouTube. Step2: Open first result. Step3: Close browser."

# Remove cached plans for these steps entirely (forces replanning and execution)
node index.js --clear-cache "Step1: Search ErrorsOverflow on YouTube. Step2: Open first result. Step3: Close browser."

Notes

• Plans are stored in plannerCache.json under keys of the form sha256(host|STEP|<step text>) with metadata.
• Accessibility snapshots are per-step and saved alongside the code for easy inspection.
• The agent currently starts at https://www.youtube.com by default; tailor as needed.

Troubleshooting

• If Playwright cannot launch the browser, ensure required browsers are installed:

  npx playwright install

• If planning seems poor without an API key, set OPENAI_API_KEY to enable LLM planning.