From 17088650f56c7368f3499cb125f3dcbec5c5f8b0 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 3 Aug 2026 13:17:16 +0000 Subject: [PATCH 1/2] Add agent quickstart docs for all SDK languages One canonical quickstart per language (Node.js, Python, Rust, Java, Elixir) covering search, scrape, and interact endpoints with complete parameter references generated from SDK source and OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01S5eo8gmHCHoSokvJbTbqqY --- agent-quickstart/elixir.mdx | 215 ++++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 238 ++++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 211 ++++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 209 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 232 +++++++++++++++++++++++++++++++++++ 5 files changed, 1105 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..5b1b6e03e --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,215 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Elixir SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Add to your `mix.exs` dependencies: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +The Elixir SDK does not use a client struct. Authentication is resolved per-call in this order: + +1. `:api_key` option passed in the trailing `opts` keyword list of any function call. +2. Application config: `Application.get_env(:firecrawl, :api_key)`. +3. If nil or empty, the key is omitted (keyless free tier for scrape, search, and interact). + +```elixir +# Option 1: Pass API key per call +Firecrawl.search_and_scrape([query: "firecrawl"], api_key: "fc-YOUR_API_KEY") + +# Option 2: Set in application config (config/config.exs) +config :firecrawl, api_key: "fc-YOUR_API_KEY" +``` + +The base URL defaults to `https://api.firecrawl.dev/v2` but can be overridden per-call with `:base_url` in `opts`. + +## When To Use What + +- **`search_and_scrape`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape_and_extract_from_url`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact_with_scrape_browser_session`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK function + +```elixir +Firecrawl.search_and_scrape(params, opts \\ []) +``` + +Bang variant: `Firecrawl.search_and_scrape!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: body}} = + Firecrawl.search_and_scrape( + query: "firecrawl web scraping API", + limit: 5, + highlights: true + ) + +for result <- body["data"]["web"] || [] do + IO.puts("#{result["url"]}: #{String.slice(result["markdown"] || "", 0..200)}") +end +``` + +### Parameters + +All parameters are passed as a keyword list. All are optional except `query`. + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `query` | `:string` | `query` | **Required.** The search query. | +| `limit` | `:integer` | `limit` | Max results to return. | +| `country` | `:string` | `country` | ISO country code for geo-targeting. | +| `location` | `:string` | `location` | Geographic location for results. | +| `tbs` | `:string` | `tbs` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `timeout` | `:integer` | `timeout` | Timeout in milliseconds. | +| `highlights` | `:boolean` | `highlights` | Generate query-relevant highlights. | +| `categories` | `list` | `categories` | Filter categories (e.g. `"github"`, `"research"`, `"pdf"`). | +| `enterprise` | `list(string)` | `enterprise` | Enterprise options (e.g. `"anon"`, `"zdr"`). | +| `exclude_domains` | `list(string)` | `excludeDomains` | Exclude results from these domains. | +| `include_domains` | `list(string)` | `includeDomains` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `ignore_invalid_urls` | `:boolean` | `ignoreInvalidURLs` | Exclude URLs invalid for other Firecrawl endpoints. | +| `scrape_options` | `:keyword_list` | `scrapeOptions` | Options applied when scraping each result. Uses same keys as `scrape_and_extract_from_url` params. | +| `sources` | `list` | `sources` | Source types (e.g. `"web"`, `"news"`, `"images"`). | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK function + +```elixir +Firecrawl.scrape_and_extract_from_url(params, opts \\ []) +``` + +Bang variant: `Firecrawl.scrape_and_extract_from_url!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: body}} = + Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"], + only_main_content: true + ) + +IO.puts(body["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. All are optional except `url`. + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `url` | `:string` | `url` | **Required.** The URL to scrape. | +| `formats` | `list` | `formats` | Output formats (e.g. `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`). | +| `actions` | `list` | `actions` | Browser actions before scraping. | +| `headers` | any | `headers` | Custom HTTP headers. | +| `include_tags` | `list(string)` | `includeTags` | HTML tags to include exclusively. | +| `exclude_tags` | `list(string)` | `excludeTags` | HTML tags to exclude. | +| `only_main_content` | `:boolean` | `onlyMainContent` | Only return main content. | +| `timeout` | `:integer` | `timeout` | Timeout in milliseconds. | +| `wait_for` | `:integer` | `waitFor` | Delay in ms before fetching content. | +| `mobile` | `:boolean` | `mobile` | Emulate a mobile device. | +| `location` | `:keyword_list` | `location` | Geolocation with `country` and `languages`. | +| `proxy` | `:basic \| :enhanced \| :auto` | `proxy` | Proxy mode. | +| `block_ads` | `:boolean` | `blockAds` | Block ads and cookie popups. | +| `max_age` | `:integer` | `maxAge` | Use cached result if younger than this (ms). | +| `min_age` | `:integer` | `minAge` | Minimum cache age (ms). | +| `parsers` | `list` | `parsers` | File processing parsers. | +| `profile` | `:keyword_list` | `profile` | Persistent browser profile. | +| `redact_pii` | `:boolean` | `redactPII` | Redact personally identifiable information. | +| `remove_base64_images` | `:boolean` | `removeBase64Images` | Remove base64 images from output. | +| `skip_tls_verification` | `:boolean` | `skipTlsVerification` | Skip TLS certificate verification. | +| `store_in_cache` | `:boolean` | `storeInCache` | Whether to cache the result. | +| `lockdown` | `:boolean` | `lockdown` | Serve only cached results. | +| `zero_data_retention` | `:boolean` | `zeroDataRetention` | Enable zero data retention. | +| `audit_metadata` | `:keyword_list` | `auditMetadata` | User attribution with required `username` key. | + +## Interact + +### Why use it + +Execute code in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK function + +```elixir +Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ []) +``` + +Bang variant: `Firecrawl.interact_with_scrape_browser_session!(job_id, params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: scrape_body}} = + Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] + ) + +job_id = scrape_body["data"]["metadata"]["jobId"] + +{:ok, %Req.Response{body: result}} = + Firecrawl.interact_with_scrape_browser_session(job_id, + code: "document.querySelector('button.load-more')?.click();", + language: :node, + timeout: 30 + ) + +IO.puts(result["stdout"]) +``` + +### Parameters + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `job_id` | `String.t()` | URL path | **Required (positional).** The scrape job ID. | +| `code` | `:string` | `code` | **Required.** Code to execute in the browser sandbox. | +| `language` | `:python \| :node \| :bash` | `language` | Runtime language. | +| `timeout` | `:integer` | `timeout` | Execution timeout in seconds. | +| `origin` | `:string` | `origin` | Request origin tag. | + +### Related function + +```elixir +Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ []) +``` + +Stops the interactive browser session. + +## Notes + +- **Naming style:** Parameters use snake_case in Elixir and are serialized to camelCase JSON keys internally. +- **OpenAPI-shaped client:** The Elixir SDK is generated from the OpenAPI spec, so function names directly mirror operation IDs (`search_and_scrape`, `scrape_and_extract_from_url`, `interact_with_scrape_browser_session`). +- **Return type:** All functions return `{:ok, %Req.Response{}} | {:error, Exception.t() | Firecrawl.Error.t()}`. Bang variants raise on error. +- **No prompt parameter:** Unlike the JS, Python, and Rust SDKs, the Elixir SDK's interact function does not support a `prompt` parameter. Use `code` only. +- **Per-call auth:** Every function accepts `:api_key` and `:base_url` in the trailing `opts` keyword list, allowing different credentials per call. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..73bad1872 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,238 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Java SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```groovy +implementation 'com.firecrawl:firecrawl-java:1.12.1' +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .build(); +``` + +Builder options: + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `apiKey` | `String` | `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property | API key. Omit for keyless free tier (rate-limited per IP). | +| `apiUrl` | `String` | `"https://api.firecrawl.dev"` | Base URL. Falls back to `FIRECRAWL_API_URL` env var. | +| `timeoutMs` | `long` | `300000` (5 min) | Per-request timeout in milliseconds. | +| `maxRetries` | `int` | `3` | Max automatic retries for transient failures. | +| `backoffFactor` | `double` | `0.5` | Exponential backoff factor for retries. | +| `asyncExecutor` | `Executor` | `ForkJoinPool.commonPool()` | Executor for async methods. | +| `httpClient` | `OkHttpClient` | — | Custom HTTP client. Overrides `timeoutMs`. | + +A convenience factory `FirecrawlClient.fromEnv()` reads the API key from the `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```java +client.search(query, options) +``` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; + +SearchData results = client.search("firecrawl web scraping API", + SearchOptions.builder() + .limit(5) + .highlights(true) + .build() +); + +for (var item : results.getWeb()) { + System.out.println(item.get("url")); +} +``` + +### Parameters + +All fields on `SearchOptions` are nullable and optional. + +| Field | Type | Description | +|-------|------|-------------| +| `sources` | `List` | Source types: `"web"`, `"news"`, `"images"`, or config maps. | +| `categories` | `List` | Filter categories: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Max results to return. | +| `tbs` | `String` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Geographic location for results. | +| `ignoreInvalidURLs` | `Boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Defaults to true. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `String` | Integration identifier. | + +An async variant is available: `client.searchAsync(query, options)` returns `CompletableFuture`. + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```java +client.scrape(url, options) +``` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; +import java.util.List; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable and optional. + +| Field | Type | Description | +|-------|------|-------------| +| `formats` | `List` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Object variants also supported for JSON extraction (`JsonFormat`), questions (`QuestionFormat`), highlights (`HighlightsFormat`). | +| `headers` | `Map` | Custom HTTP headers. | +| `includeTags` | `List` | HTML tags to include exclusively. | +| `excludeTags` | `List` | HTML tags to exclude. | +| `onlyMainContent` | `Boolean` | Only return main content, excluding navbars/footers. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `waitFor` | `Integer` | Delay in ms before fetching content. | +| `mobile` | `Boolean` | Emulate a mobile device. | +| `parsers` | `List` | File processing parsers (e.g. `"pdf"` or config maps). | +| `actions` | `List>` | Browser actions before scraping. Action types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `LocationConfig` | Geolocation with `country` and `languages`. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64 images from output. | +| `blockAds` | `Boolean` | Block ads and cookie popups. | +| `proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `maxAge` | `Long` | Use cached result if younger than this (ms). | +| `storeInCache` | `Boolean` | Whether to cache the result. | +| `lockdown` | `Boolean` | Serve only cached results. | +| `redactPII` | `Boolean` | Redact personally identifiable information. | +| `auditMetadata` | `AuditMetadata` | User attribution for SIEM logging with `username`. | +| `integration` | `String` | Integration identifier. | + +An async variant is available: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +## Interact + +### Why use it + +Execute code in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```java +client.interact(jobId, code) +client.interact(jobId, code, language, timeout) +client.interact(jobId, code, language, timeout, origin) +``` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = (String) doc.getMetadata().get("jobId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "document.querySelector('button.load-more')?.click();", + "node", + 30 +); + +System.out.println(result.getStdout()); +``` + +### Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `jobId` | `String` | — | **Required.** The scrape job ID from a prior scrape. | +| `code` | `String` | — | **Required.** Code to execute in the browser sandbox. | +| `language` | `String` | `"node"` | Runtime language: `"python"`, `"node"`, or `"bash"`. | +| `timeout` | `Integer` | `null` | Execution timeout in seconds (1-300). Default: 30 server-side. | +| `origin` | `String` | `null` | Request origin tag. | + +Async variants are available: `client.interactAsync(...)` returns `CompletableFuture`. + +### Related method + +```java +client.stopInteractiveBrowser(jobId) +``` + +Stops the interactive browser session and returns billing info as `BrowserDeleteResponse`. + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Builder pattern:** `ScrapeOptions` and `SearchOptions` use builder pattern (`ScrapeOptions.builder()...build()`). +- **Deprecated aliases:** + - `scrapeExecute()` → use `interact()` instead. + - `deleteScrapeBrowser()` → use `stopInteractiveBrowser()` instead. +- **Async methods:** Every sync method has an async counterpart suffixed with `Async` that returns `CompletableFuture`. +- **Interact limitations:** The Java SDK's `interact` method requires `code` as a string parameter. Unlike the JS and Python SDKs, it does not support a `prompt` parameter for natural-language browser instructions. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..aab935143 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,211 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Node.js/TypeScript SDK. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +## Authenticate + +```typescript +import Firecrawl from "@mendable/firecrawl-js"; + +const client = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" }); +``` + +Constructor options: + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `apiKey` | `string \| null` | `FIRECRAWL_API_KEY` env var | API key. Omit for keyless free tier (rate-limited per IP). | +| `apiUrl` | `string \| null` | `https://api.firecrawl.dev` | Base URL. Falls back to `FIRECRAWL_API_URL` env var. | +| `timeoutMs` | `number` | — | Per-request timeout in milliseconds. | +| `maxRetries` | `number` | — | Max automatic retries for transient failures. | +| `backoffFactor` | `number` | — | Exponential backoff factor for retries. | + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```typescript +client.search(query, options?) +``` + +### Example + +```typescript +const results = await client.search("firecrawl web scraping API", { + limit: 5, + scrapeOptions: { formats: ["markdown"] }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.markdown?.slice(0, 200)); +} +``` + +### Parameters + +All parameters are optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `string` | **Required (positional).** The search query. | +| `sources` | `Array<"web" \| "news" \| "images" \| { type: ... }>` | Source types to search. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer" \| { type: ... }>` | Filter by category. | +| `includeDomains` | `string[]` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. | +| `limit` | `number` | Max results to return. | +| `tbs` | `string` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `string` | Geographic location for results. | +| `ignoreInvalidURLs` | `boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `number` | Timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise zero data retention options. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Request origin tag. | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```typescript +client.scrape(url, options?) +``` + +### Example + +```typescript +const doc = await client.scrape("https://example.com", { + formats: ["markdown", "links"], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.links); +``` + +### Parameters + +All parameters are optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `string` | **Required (positional).** The URL to scrape. | +| `formats` | `FormatOption[]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Object variants: `{ type: "json", schema?, prompt? }`, `{ type: "screenshot", fullPage?, quality?, viewport? }`, `{ type: "changeTracking", modes, schema?, prompt?, tag? }`, `{ type: "attributes", selectors }`, `{ type: "question", question }`, `{ type: "highlights", query }`. | +| `headers` | `Record` | Custom HTTP headers to send with the request. | +| `includeTags` | `string[]` | HTML tags to include exclusively. | +| `excludeTags` | `string[]` | HTML tags to exclude. | +| `onlyMainContent` | `boolean` | Only return main content, excluding navbars/footers. | +| `timeout` | `number` | Timeout in milliseconds. | +| `waitFor` | `number` | Delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `Array` | File processing parsers. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `actions` | `ActionOption[]` | Browser actions to perform before scraping. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `{ country?: string, languages?: string[] }` | Geolocation for proxy routing. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64 images from output. | +| `fastMode` | `boolean` | Faster scraping with reduced accuracy. | +| `useMock` | `string` | Use a mock response. | +| `blockAds` | `boolean` | Block ads and cookie popups. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode. | +| `maxAge` | `number` | Use cached result if younger than this (ms). | +| `minAge` | `number` | Minimum cache age (ms). Set to 1 for any cached data. | +| `storeInCache` | `boolean` | Whether to cache the result. | +| `lockdown` | `boolean` | Serve only cached results. No outbound request. | +| `redactPII` | `boolean \| { mode?, entities?, replaceStyle? }` | Redact personally identifiable information. Modes: `"accurate"`, `"aggressive"`, `"fast"`. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | +| `profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser storage profile. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Request origin tag. | + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```typescript +client.interact(jobId, args) +``` + +### Example + +```typescript +const doc = await client.scrape("https://example.com", { + formats: ["markdown"], +}); + +const jobId = doc.metadata?.jobId; + +const result = await client.interact(jobId, { + code: "document.querySelector('button.load-more')?.click();", + language: "node", + timeout: 30, +}); + +console.log(result.stdout); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `jobId` | `string` | **Required (positional).** The scrape job ID from a prior scrape. | +| `code` | `string` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `string` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `"python" \| "node" \| "bash"` | Runtime language for `code`. | +| `timeout` | `number` | Execution timeout in seconds (1-300). | +| `origin` | `string` | Request origin tag. | + +### Related method + +```typescript +client.stopInteraction(jobId) +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Deprecated aliases:** + - `scrapeUrl()` → use `scrape()` instead. + - `scrapeExecute()` → use `interact()` instead. + - `stopInteractiveBrowser()` and `deleteScrapeBrowser()` → use `stopInteraction()` instead. +- **Async client:** The SDK is async by default (all methods return Promises). +- **Zod schemas:** The `json` format accepts Zod schemas in addition to plain JSON Schema objects for the `schema` field. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..d9ff92061 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,209 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Python SDK. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +client = Firecrawl(api_key="fc-YOUR_API_KEY") +``` + +Constructor parameters: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `api_key` | `str` | `FIRECRAWL_API_KEY` env var | API key. Omit for keyless free tier (rate-limited per IP). | +| `api_url` | `str` | `"https://api.firecrawl.dev"` | Base URL. | +| `timeout` | `float` | `None` | Default request timeout in seconds. | +| `max_retries` | `int` | `3` | Max automatic retries for transient failures. | +| `backoff_factor` | `float` | `0.5` | Exponential backoff factor for retries. | + +An async client is also available: `from firecrawl import AsyncFirecrawl`. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```python +client.search(query, **kwargs) +``` + +### Example + +```python +results = client.search( + "firecrawl web scraping API", + limit=5, + scrape_options=ScrapeOptions(formats=["markdown"]), +) + +for item in results.web or []: + print(item.url, item.markdown[:200] if item.markdown else "") +``` + +### Parameters + +All parameters are keyword-only and optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `str` | **Required (positional).** The search query. | +| `sources` | `list[str \| Source]` | Source types to search (e.g. `"web"`, `"news"`, `"images"`). | +| `categories` | `list[str \| Category]` | Filter by category (e.g. `"github"`, `"research"`, `"pdf"`, `"developer"`). | +| `include_domains` | `list[str]` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. | +| `limit` | `int` | Max results to return. | +| `tbs` | `str` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `str` | Geographic location for results. | +| `ignore_invalid_urls` | `bool` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `int` | Timeout in milliseconds. | +| `highlights` | `bool` | Generate query-relevant highlights. | +| `scrape_options` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `str` | Integration identifier. | +| `enterprise` | `list[str]` | Enterprise options (e.g. `"anon"`, `"zdr"`). | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```python +client.scrape(url, **kwargs) +``` + +### Example + +```python +doc = client.scrape( + "https://example.com", + formats=["markdown", "links"], + only_main_content=True, +) + +print(doc.markdown) +print(doc.links) +``` + +### Parameters + +All parameters are keyword-only and optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `str` | **Required (positional).** The URL to scrape. | +| `formats` | `list[FormatOption]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Object variants also supported for `json`, `screenshot`, `changeTracking`, `attributes`, `question`, `highlights`. | +| `headers` | `dict[str, str]` | Custom HTTP headers to send with the request. | +| `include_tags` | `list[str]` | HTML tags to include exclusively. | +| `exclude_tags` | `list[str]` | HTML tags to exclude. | +| `only_main_content` | `bool` | Only return main content, excluding navbars/footers. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Delay in ms before fetching content. | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list[str \| PDFParser]` | File processing parsers. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `actions` | `list[Action]` | Browser actions to perform before scraping. Types: `WaitAction`, `ScreenshotAction`, `ClickAction`, `WriteAction`, `PressAction`, `ScrollAction`, `ScrapeAction`, `ExecuteJavascriptAction`, `PDFAction`. | +| `location` | `Location` | Geolocation config with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64 images from output. | +| `fast_mode` | `bool` | Faster scraping with reduced accuracy. | +| `use_mock` | `str` | Use a mock response. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached result if younger than this (ms). | +| `store_in_cache` | `bool` | Whether to cache the result. | +| `lockdown` | `bool` | Serve only cached results. No outbound request. | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `profile` | `dict` | Persistent browser storage profile with `name` and optional `save_changes`. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging with `username` field. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```python +client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None) +``` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.get("jobId") + +result = client.interact( + job_id, + code="document.querySelector('button.load-more')?.click();", + language="node", + timeout=30, +) + +print(result.stdout) +``` + +### Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `job_id` | `str` | — | **Required (positional).** The scrape job ID from a prior scrape. | +| `code` | `str` | `None` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `str` | `None` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `str` | `"node"` | Runtime language: `"python"`, `"node"`, or `"bash"`. | +| `timeout` | `int` | `None` | Execution timeout in seconds (1-300). | +| `origin` | `str` | `None` | Request origin tag. | + +### Related method + +```python +client.stop_interaction(job_id) +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All parameters use snake_case. +- **Deprecated aliases:** + - `scrape_url()` → use `scrape()` instead. + - `scrape_execute()` → use `interact()` instead. + - `stop_interactive_browser()` and `delete_scrape_browser()` → use `stop_interaction()` instead. + - `FirecrawlApp` → use `Firecrawl` instead (alias still works). +- **Async client:** Use `AsyncFirecrawl` (aliased as `AsyncFirecrawlApp`) for async/await usage with the same method signatures. +- **ScrapeOptions model:** The `ScrapeOptions` Pydantic model also includes `min_age` and `redact_pii` fields which are available when passing `scrape_options` to `search()` but are not direct kwargs on `scrape()`. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..918c5b97b --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,232 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Rust SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +// Firecrawl cloud +let client = Client::new("fc-YOUR_API_KEY")?; + +// Self-hosted (API key optional) +let client = Client::new_selfhosted("https://your-instance.com", Some("fc-YOUR_API_KEY"))?; +``` + +`Client::new` requires an API key. `Client::new_selfhosted` allows an optional key for keyless free-tier usage. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```rust +client.search(query, options).await +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let response = client.search("firecrawl web scraping API", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `limit` | `Option` | Max results. Default: 5, Max: 20. | +| `sources` | `Option>` | Source types: `Web`, `News`, `Images`. | +| `categories` | `Option>` | Filter categories: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `Option` | Geographic location for results. | +| `ignore_invalid_urls` | `Option` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `highlights` | `Option` | Generate query-relevant highlights. Default: true. | +| `scrape_options` | `Option` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | + +A convenience method `search_and_scrape(query, limit)` is also available. It searches with default scrape options and returns `Vec` directly. + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```rust +client.scrape(url, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + only_main_content: Some(true), + ..Default::default() +}).await?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `formats` | `Option>` | Output formats. Enum values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(String)`, `Highlights(String)`, `Query(String)`. | +| `headers` | `Option>` | Custom HTTP headers. | +| `include_tags` | `Option>` | HTML tags to include exclusively. | +| `exclude_tags` | `Option>` | HTML tags to exclude. | +| `only_main_content` | `Option` | Only return main content. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `wait_for` | `Option` | Delay in ms before fetching content. | +| `mobile` | `Option` | Emulate a mobile device. | +| `parsers` | `Option>` | File processing parsers. `ParserConfig::Simple(String)` or `ParserConfig::Pdf { mode, max_pages }`. | +| `actions` | `Option>` | Browser actions before scraping. Types: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `location` | `Option` | Geolocation with `country` and `languages`. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64 images from output. | +| `fast_mode` | `Option` | Faster scraping with reduced accuracy. | +| `block_ads` | `Option` | Block ads. | +| `proxy` | `Option` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `max_age` | `Option` | Use cached result if younger than this (seconds). | +| `min_age` | `Option` | Minimum cache age (seconds). | +| `store_in_cache` | `Option` | Whether to cache the result. | +| `lockdown` | `Option` | Serve only cached results. | +| `redact_pii` | `Option` | Redact personally identifiable information. | +| `audit_metadata` | `Option` | User attribution for SIEM logging. | +| `profile` | `Option` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | +| `json_options` | `Option` | JSON extraction options with `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config with `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking config with `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute extraction with `selector` and `attribute`. | + +A convenience method `scrape_with_schema(url, schema, prompt)` is also available for JSON extraction. + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```rust +client.interact(job_id, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let job_id = doc.metadata.get("jobId").and_then(|v| v.as_str()).unwrap(); + +let result = client.interact(job_id, ScrapeExecuteOptions { + code: Some("document.querySelector('button.load-more')?.click();".into()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(30), + ..Default::default() +}).await?; + +println!("{:?}", result.stdout); +``` + +### Parameters + +All fields on `ScrapeExecuteOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `code` | `Option` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `Option` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `Option` | Runtime: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | + +At least one of `code` or `prompt` must be provided or the client returns `FirecrawlError::Misuse`. + +### Related method + +```rust +client.stop_interaction(job_id).await +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All struct fields use snake_case. Serialization to camelCase is handled internally by serde. +- **Deprecated aliases:** + - `scrape_execute()` → use `interact()` instead. + - `stop_interactive_browser()` and `delete_scrape_browser()` → use `stop_interaction()` instead. +- **Async:** All methods are async and return `Result`. +- **Origin auto-set:** The SDK automatically sets `origin` to `"rust-sdk@{version}"` if not explicitly provided. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/v2/client.rs` +- `firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `firecrawl/apps/rust-sdk/src/v2/search.rs` +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl-docs/api-reference/v2-openapi.json` From f26cedb89e8dafe1ff62e551596b95bbdd692ed4 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 3 Aug 2026 13:19:03 +0000 Subject: [PATCH 2/2] Add agent quickstart pages to docs navigation Register the 5 new agent-quickstart pages in the "Build with AI" tab navigation so they appear in the docs sidebar. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01S5eo8gmHCHoSokvJbTbqqY --- docs.json | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs.json b/docs.json index 4a5873a02..82d556cc3 100755 --- a/docs.json +++ b/docs.json @@ -680,6 +680,16 @@ "agents/fire-1-extract" ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Agentic Debugging", "pages": [