An OpenClaw skill for scraping Cloudflare-protected websites using Stagehand + Browserbase cloud browsers.
Traditional scraping methods fail on protected sites:
curl/fetch→ Blocked by Cloudflare- Playwright headless → Detected as bot
- Playwright stealth → Still blocked on many sites
This skill uses Browserbase cloud browsers which appear as real users, bypassing bot protection automatically.
# Install dependencies
npm install @browserbasehq/stagehand zod
# Set up API keys
export BROWSERBASE_API_KEY=your_key
export BROWSERBASE_PROJECT_ID=your_project_id
export GOOGLE_GENERATIVE_AI_API_KEY=your_gemini_keyGet Browserbase credentials at: https://browserbase.com
Tell OpenClaw to scrape a protected site:
Scrape all product listings from https://protected-site.com/products
Extract title, price, and URL for each item.
The skill will automatically use Stagehand + Browserbase to bypass protection.
See scripts/example_scraper.js for a complete working example that:
- Initializes Browserbase cloud browser
- Navigates to search results
- Handles Cloudflare challenges
- Extracts structured data with AI
- Saves results to JSON
Schedule regular scraping jobs:
openclaw cron add \
--name "Daily Scrape" \
--cron "0 6 * * *" \
--session isolated \
--message "Run the browserbase scraper on https://example.com"- Browserbase free tier: 1 concurrent session
- Extraction quality depends on page structure
- Some sites may still block after repeated access
Developed while building PTS Delta tracking for Porsche auction data.
MIT