Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"description": "Scrape, search, crawl, and map the web with a single command.",
"skills": [
"./skills/firecrawl-agent",
"./skills/firecrawl-cli",
"./skills/firecrawl",
"./skills/firecrawl-crawl",
"./skills/firecrawl-download",
"./skills/firecrawl-interact",
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
},
"skills": [
"./skills/firecrawl-agent",
"./skills/firecrawl-cli",
"./skills/firecrawl",
"./skills/firecrawl-crawl",
"./skills/firecrawl-download",
"./skills/firecrawl-interact",
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -667,10 +667,10 @@ firecrawl agent "Find the top 5 competitors of Notion and their pricing" --wait
firecrawl agent "Get all blog post titles and dates" --urls https://blog.example.com --max-credits 100 --wait

# Use higher accuracy model for complex extraction
firecrawl agent "Extract detailed technical specifications" --model spark-1-pro --wait --pretty
firecrawl agent "Extract detailed technical specifications" --model spark-1-pro --wait --json --pretty

# Save structured results to file
firecrawl agent "Extract contact information" --schema-file ./contact-schema.json --wait -o contacts.json --pretty
firecrawl agent "Extract contact information" --schema-file ./contact-schema.json --wait --json -o contacts.json --pretty

# Check job status without waiting
firecrawl agent abc123-def456-... --json
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
"clean": "rm -rf dist",
"prepublishOnly": "pnpm run build",
"prepare": "husky",
"format": "prettier --write \"src/**/*.{ts,json}\" \"*.{json,md}\"",
"format:check": "prettier --check \"src/**/*.{ts,json}\" \"*.{json,md}\"",
"format": "prettier --write \"src/**/*.{ts,json}\" \"skills/**/*.md\" \"*.{json,md}\"",
"format:check": "prettier --check \"src/**/*.{ts,json}\" \"skills/**/*.md\" \"*.{json,md}\"",
"type-check": "tsc --noEmit",
"test:watch": "vitest",
"test": "vitest run",
Expand Down
50 changes: 35 additions & 15 deletions skills/firecrawl-agent/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: |
AI-powered autonomous data extraction that navigates complex sites and returns structured JSON. Use this skill when the user wants structured data from websites, needs to extract pricing tiers, product listings, directory entries, or any data as JSON with a schema. Triggers on "extract structured data", "get all the products", "pull pricing info", "extract as JSON", or when the user provides a JSON schema for website data. More powerful than simple scraping for multi-page structured extraction.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl agent
Expand All @@ -21,31 +21,51 @@ AI-powered autonomous extraction. The agent navigates sites and extracts structu

```bash
# Extract structured data
firecrawl agent "extract all pricing tiers" --wait -o .firecrawl/pricing.json
firecrawl agent "extract all pricing tiers" --wait --json -o .firecrawl/pricing.json

# With a JSON schema for structured output
firecrawl agent "extract products" --schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}' --wait -o .firecrawl/products.json
firecrawl agent "extract products" --schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"number"}}}' --wait --json -o .firecrawl/products.json

# Focus on specific pages
firecrawl agent "get feature list" --urls "<url>" --wait -o .firecrawl/features.json
firecrawl agent "get feature list" --urls "<url>" --wait --json -o .firecrawl/features.json
```

## Job IDs

Without `--wait`, the command returns a job ID. A UUID positional argument is auto-detected as a status check:

```bash
# Check once (equivalent to adding --status)
firecrawl agent "<job-id>"

# Wait on an existing job, polling every 10 seconds for up to 5 minutes
firecrawl agent "<job-id>" --wait --poll-interval 10 --timeout 300

# Cancel an active job
firecrawl agent "<job-id>" --cancel
```

## Options

| Option | Description |
| ---------------------- | ----------------------------------------- |
| `--urls <urls>` | Starting URLs for the agent |
| `--model <model>` | Model to use: spark-1-mini or spark-1-pro |
| `--schema <json>` | JSON schema for structured output |
| `--schema-file <path>` | Path to JSON schema file |
| `--max-credits <n>` | Credit limit for this agent run |
| `--wait` | Wait for agent to complete |
| `--pretty` | Pretty print JSON output |
| `-o, --output <path>` | Output file path |
| Option | Description |
| --------------------------- | ----------------------------------------------------- |
| `--urls <urls>` | Starting URLs for the agent |
| `--model <model>` | Model to use: spark-1-mini or spark-1-pro |
| `--schema <json>` | JSON schema for structured output |
| `--schema-file <path>` | Path to JSON schema file |
| `--max-credits <n>` | Credit limit for this agent run |
| `--status` | Check a job ID's status |
| `--cancel` | Cancel an active job ID |
| `--wait` | Wait for agent to complete |
| `--poll-interval <seconds>` | Polling interval while waiting (default: 5 seconds) |
| `--timeout <seconds>` | Stop waiting after this duration (default: none) |
| `--json` | Output as JSON |
| `--pretty` | Pretty print JSON (`--wait` results require `--json`) |
| `-o, --output <path>` | Output file path |

## Tips

- Always use `--wait` to get results inline. Without it, returns a job ID.
- Use `--wait` for inline results; without it you get a job ID (see [Job IDs](#job-ids)).
- Use `--schema` for predictable, structured output — otherwise the agent returns freeform data.
- Agent runs consume more credits than simple scrapes. Use `--max-credits` to cap spending.
- For simple single-page extraction, prefer `scrape` — it's faster and cheaper.
Expand Down
10 changes: 6 additions & 4 deletions skills/firecrawl-crawl/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,20 @@ description: |
Bulk extract content from an entire website or site section. Use this skill when the user wants to crawl a site, extract all pages from a docs section, bulk-scrape multiple pages following links, or says "crawl", "get all the pages", "extract everything under /docs", "bulk extract", or needs content from many pages on the same site. Handles depth limits, path filtering, and concurrent extraction.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

**Prerequisite:** `crawl` requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

## When to use

- You need content from many pages on a site (e.g., all `/docs/`)
- You want to extract an entire site section
- Step 4 in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → **crawl** → interact
- Step 4 in the [workflow escalation pattern](../firecrawl/SKILL.md): search → scrape → map + scrape → **crawl** → monitor → interact

## Quick start

Expand Down Expand Up @@ -47,9 +49,9 @@ firecrawl crawl <job-id>

## Tips

- Always use `--wait` when you need the results immediately. Without it, crawl returns a job ID for async polling.
- Always use `--wait` when you need the results immediately. It has no default timeout; use `--timeout <seconds>` to bound polling. Without `--wait`, crawl returns a job ID for async polling.
- Use `--include-paths` to scope the crawl — don't crawl an entire site when you only need one section.
- Crawl consumes credits per page. Check `firecrawl credit-usage` before large crawls.
- Crawl consumes credits per page. Check `firecrawl credit-usage` before large crawls (`credit-usage` requires authentication).

## See also

Expand Down
30 changes: 17 additions & 13 deletions skills/firecrawl-download/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@ description: |
Download an entire website as local files — markdown, screenshots, or multiple formats per page. Use this skill when the user wants to save a site locally, download documentation for offline use, bulk-save pages as files, or says "download the site", "save as local files", "offline copy", "download all the docs", or "save for reference". Combines site mapping and scraping into organized local directories.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl download
# firecrawl download (invoked as `firecrawl x download`)

> **Experimental.** Convenience command that combines `map` + `scrape` to save an entire site as local files.
> **Experimental.** `download` is available under the `firecrawl x` command group.

Maps the site first to discover pages, then scrapes each one into nested directories under `.firecrawl/`. All scrape options work with download. Always pass `-y` to skip the confirmation prompt.
**Prerequisite:** `download` requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Maps the site origin first to discover pages, then scrapes each one into nested directories under `.firecrawl/`. Use `--include-paths` to scope a non-root URL to one section. Supported scrape options are listed below. Always pass `-y` to skip the confirmation prompt.

## When to use

Expand All @@ -22,24 +24,24 @@ Maps the site first to discover pages, then scrapes each one into nested directo
## Quick start

```bash
# Interactive wizard (picks format, screenshots, paths for you)
firecrawl download https://docs.example.com
# Interactive wizard (humans at a TTY only — agents must pass -y or the command blocks on a prompt)
firecrawl x download https://docs.example.com

# With screenshots
firecrawl download https://docs.example.com --screenshot --limit 20 -y
firecrawl x download https://docs.example.com --screenshot --limit 20 -y

# Multiple formats (each saved as its own file per page)
firecrawl download https://docs.example.com --format markdown,links --screenshot --limit 20 -y
firecrawl x download https://docs.example.com --format markdown,links --screenshot --limit 20 -y
# Creates per page: index.md + links.txt + screenshot.png

# Filter to specific sections
firecrawl download https://docs.example.com --include-paths "/features,/sdks"
firecrawl x download https://docs.example.com --include-paths "/features,/sdks" -y

# Skip translations
firecrawl download https://docs.example.com --exclude-paths "/zh,/ja,/fr,/es,/pt-BR"
firecrawl x download https://docs.example.com --exclude-paths "/zh,/ja,/fr,/es,/pt-BR" -y

# Full combo
firecrawl download https://docs.example.com \
firecrawl x download https://docs.example.com \
--include-paths "/features,/sdks" \
--exclude-paths "/zh,/ja" \
--only-main-content \
Expand All @@ -58,9 +60,11 @@ firecrawl download https://docs.example.com \
| `--allow-subdomains` | Include subdomain pages |
| `-y` | Skip confirmation prompt (always use in automated flows) |

## Scrape options (all work with download)
## Supported scrape options

Only the options listed below are supported:

`-f <formats>`, `-H`, `-S`, `--screenshot`, `--full-page-screenshot`, `--only-main-content`, `--include-tags`, `--exclude-tags`, `--wait-for`, `--max-age`, `--country`, `--languages`
`-f <formats>`, `-H`, `-S`, `--lockdown`, `--screenshot`, `--full-page-screenshot`, `--only-main-content`, `--include-tags`, `--exclude-tags`, `--wait-for`, `--max-age`, `--country`, `--languages`

## See also

Expand Down
37 changes: 20 additions & 17 deletions skills/firecrawl-interact/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: |
Control and interact with a live browser session on any scraped page — click buttons, fill forms, navigate flows, and extract data using natural language prompts or code. Use when the user needs to interact with a webpage beyond simple scraping: logging into a site, submitting forms, clicking through pagination, handling infinite scroll, navigating multi-step checkout or wizard flows, or when a regular scrape failed because content is behind JavaScript interaction. Also useful for authenticated scraping via profiles. Triggers on "interact", "click", "fill out the form", "log in to", "sign in", "submit", "paginated", "next page", "infinite scroll", "interact with the page", "navigate to", "open a session", or "scrape failed".
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl interact
Expand All @@ -16,7 +16,7 @@ Interact with scraped pages in a live browser session. Scrape a page first, then
- Content requires interaction: clicks, form fills, pagination, login
- `scrape` failed because content is behind JavaScript interaction
- You need to navigate a multi-step flow
- Last resort in the [workflow escalation pattern](firecrawl-cli): search → scrape → map → crawl → **interact**
- Last resort in the [workflow escalation pattern](../firecrawl/SKILL.md): search → scrape → map + scrape → crawl → monitor → **interact**
- **Never use interact for web searches** — use `search` instead

## Quick start
Expand All @@ -25,29 +25,32 @@ Interact with scraped pages in a live browser session. Scrape a page first, then
# 1. Scrape a page (scrape ID is saved automatically)
firecrawl scrape "<url>"

# 2. Interact with the page using natural language
firecrawl interact --prompt "Click the login button"
firecrawl interact --prompt "Fill in the email field with test@example.com"
firecrawl interact --prompt "Extract the pricing table"
# 2. Interact with the page using a positional prompt
firecrawl interact "Click the login button"
firecrawl interact "Fill in the email field with test@example.com"
firecrawl interact "Extract the pricing table"

# A UUID first argument is auto-detected as the scrape ID
firecrawl interact "<scrape-id>" "Extract the pricing table"

# 3. Or use code for precise control
firecrawl interact --code "agent-browser click @e5" --language bash
firecrawl interact --code "agent-browser snapshot -i" --language bash
firecrawl interact --code "click @e5" --bash
firecrawl interact --code "snapshot -i" --bash

# 4. Stop the session when done
firecrawl interact stop
```

## Options

| Option | Description |
| --------------------- | ------------------------------------------------- |
| `--prompt <text>` | Natural language instruction (use this OR --code) |
| `--code <code>` | Code to execute in the browser session |
| `--language <lang>` | Language for code: bash, python, node |
| `--timeout <seconds>` | Execution timeout (default: 30, max: 300) |
| `--scrape-id <id>` | Target a specific scrape (default: last scrape) |
| `-o, --output <path>` | Output file path |
| Option | Description |
| -------------------------------- | ------------------------------------------------- |
| `--prompt <text>` | Natural language instruction (use this OR --code) |
| `--code <code>` | Code to execute in the browser session |
| `--node` / `--python` / `--bash` | Language for `--code` (default: node) |
| `--timeout <seconds>` | Execution timeout (default: 30, max: 300) |
| `--scrape-id <id>` | Target a specific scrape (default: last scrape) |
| `-o, --output <path>` | Output file path |

## Profiles

Expand All @@ -72,7 +75,7 @@ firecrawl scrape "https://app.example.com" --profile my-app --no-save-changes
## Tips

- Always scrape first — `interact` requires a scrape ID from a previous `firecrawl scrape` call
- The scrape ID is saved automatically, so you don't need `--scrape-id` for subsequent interact calls
- The scrape ID is saved automatically, so you don't need `--scrape-id` for subsequent interact calls. Saved sessions may expire after about 10 minutes; re-scrape if the CLI warns that the session is stale
- Use `firecrawl interact stop` to free resources when done
- For parallel work, scrape multiple pages and interact with each using `--scrape-id`

Expand Down
6 changes: 4 additions & 2 deletions skills/firecrawl-map/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,20 @@ description: |
Discover and list all URLs on a website, with optional search filtering. Use this skill when the user wants to find a specific page on a large site, list all URLs, see the site structure, find where something is on a domain, or says "map the site", "find the URL for", "what pages are on", or "list all pages". Essential when the user knows which site but not which exact page.
allowed-tools:
- Bash(firecrawl *)
- Bash(npx firecrawl *)
- Bash(npx firecrawl-cli *)
---

# firecrawl map

Discover URLs on a site. Use `--search` to find a specific page within a large site.

**Prerequisite:** `map` requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

## When to use

- You need to find a specific subpage on a large site
- You want a list of all URLs on a site before scraping or crawling
- Step 3 in the [workflow escalation pattern](firecrawl-cli): search → scrape → **map** → crawl → interact
- Step 3 in the [workflow escalation pattern](../firecrawl/SKILL.md): search → scrape → **map** + scrape → crawl → monitor → interact

## Quick start

Expand Down
Loading
Loading