diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..5b1b6e03e --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,215 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Elixir SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Add to your `mix.exs` dependencies: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +The Elixir SDK does not use a client struct. Authentication is resolved per-call in this order: + +1. `:api_key` option passed in the trailing `opts` keyword list of any function call. +2. Application config: `Application.get_env(:firecrawl, :api_key)`. +3. If nil or empty, the key is omitted (keyless free tier for scrape, search, and interact). + +```elixir +# Option 1: Pass API key per call +Firecrawl.search_and_scrape([query: "firecrawl"], api_key: "fc-YOUR_API_KEY") + +# Option 2: Set in application config (config/config.exs) +config :firecrawl, api_key: "fc-YOUR_API_KEY" +``` + +The base URL defaults to `https://api.firecrawl.dev/v2` but can be overridden per-call with `:base_url` in `opts`. + +## When To Use What + +- **`search_and_scrape`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape_and_extract_from_url`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact_with_scrape_browser_session`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK function + +```elixir +Firecrawl.search_and_scrape(params, opts \\ []) +``` + +Bang variant: `Firecrawl.search_and_scrape!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: body}} = + Firecrawl.search_and_scrape( + query: "firecrawl web scraping API", + limit: 5, + highlights: true + ) + +for result <- body["data"]["web"] || [] do + IO.puts("#{result["url"]}: #{String.slice(result["markdown"] || "", 0..200)}") +end +``` + +### Parameters + +All parameters are passed as a keyword list. All are optional except `query`. + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `query` | `:string` | `query` | **Required.** The search query. | +| `limit` | `:integer` | `limit` | Max results to return. | +| `country` | `:string` | `country` | ISO country code for geo-targeting. | +| `location` | `:string` | `location` | Geographic location for results. | +| `tbs` | `:string` | `tbs` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `timeout` | `:integer` | `timeout` | Timeout in milliseconds. | +| `highlights` | `:boolean` | `highlights` | Generate query-relevant highlights. | +| `categories` | `list` | `categories` | Filter categories (e.g. `"github"`, `"research"`, `"pdf"`). | +| `enterprise` | `list(string)` | `enterprise` | Enterprise options (e.g. `"anon"`, `"zdr"`). | +| `exclude_domains` | `list(string)` | `excludeDomains` | Exclude results from these domains. | +| `include_domains` | `list(string)` | `includeDomains` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `ignore_invalid_urls` | `:boolean` | `ignoreInvalidURLs` | Exclude URLs invalid for other Firecrawl endpoints. | +| `scrape_options` | `:keyword_list` | `scrapeOptions` | Options applied when scraping each result. Uses same keys as `scrape_and_extract_from_url` params. | +| `sources` | `list` | `sources` | Source types (e.g. `"web"`, `"news"`, `"images"`). | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK function + +```elixir +Firecrawl.scrape_and_extract_from_url(params, opts \\ []) +``` + +Bang variant: `Firecrawl.scrape_and_extract_from_url!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: body}} = + Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"], + only_main_content: true + ) + +IO.puts(body["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. All are optional except `url`. + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `url` | `:string` | `url` | **Required.** The URL to scrape. | +| `formats` | `list` | `formats` | Output formats (e.g. `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`). | +| `actions` | `list` | `actions` | Browser actions before scraping. | +| `headers` | any | `headers` | Custom HTTP headers. | +| `include_tags` | `list(string)` | `includeTags` | HTML tags to include exclusively. | +| `exclude_tags` | `list(string)` | `excludeTags` | HTML tags to exclude. | +| `only_main_content` | `:boolean` | `onlyMainContent` | Only return main content. | +| `timeout` | `:integer` | `timeout` | Timeout in milliseconds. | +| `wait_for` | `:integer` | `waitFor` | Delay in ms before fetching content. | +| `mobile` | `:boolean` | `mobile` | Emulate a mobile device. | +| `location` | `:keyword_list` | `location` | Geolocation with `country` and `languages`. | +| `proxy` | `:basic \| :enhanced \| :auto` | `proxy` | Proxy mode. | +| `block_ads` | `:boolean` | `blockAds` | Block ads and cookie popups. | +| `max_age` | `:integer` | `maxAge` | Use cached result if younger than this (ms). | +| `min_age` | `:integer` | `minAge` | Minimum cache age (ms). | +| `parsers` | `list` | `parsers` | File processing parsers. | +| `profile` | `:keyword_list` | `profile` | Persistent browser profile. | +| `redact_pii` | `:boolean` | `redactPII` | Redact personally identifiable information. | +| `remove_base64_images` | `:boolean` | `removeBase64Images` | Remove base64 images from output. | +| `skip_tls_verification` | `:boolean` | `skipTlsVerification` | Skip TLS certificate verification. | +| `store_in_cache` | `:boolean` | `storeInCache` | Whether to cache the result. | +| `lockdown` | `:boolean` | `lockdown` | Serve only cached results. | +| `zero_data_retention` | `:boolean` | `zeroDataRetention` | Enable zero data retention. | +| `audit_metadata` | `:keyword_list` | `auditMetadata` | User attribution with required `username` key. | + +## Interact + +### Why use it + +Execute code in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK function + +```elixir +Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ []) +``` + +Bang variant: `Firecrawl.interact_with_scrape_browser_session!(job_id, params, opts)` raises on error. + +### Example + +```elixir +{:ok, %Req.Response{body: scrape_body}} = + Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] + ) + +job_id = scrape_body["data"]["metadata"]["jobId"] + +{:ok, %Req.Response{body: result}} = + Firecrawl.interact_with_scrape_browser_session(job_id, + code: "document.querySelector('button.load-more')?.click();", + language: :node, + timeout: 30 + ) + +IO.puts(result["stdout"]) +``` + +### Parameters + +| Parameter | Type | JSON Key | Description | +|-----------|------|----------|-------------| +| `job_id` | `String.t()` | URL path | **Required (positional).** The scrape job ID. | +| `code` | `:string` | `code` | **Required.** Code to execute in the browser sandbox. | +| `language` | `:python \| :node \| :bash` | `language` | Runtime language. | +| `timeout` | `:integer` | `timeout` | Execution timeout in seconds. | +| `origin` | `:string` | `origin` | Request origin tag. | + +### Related function + +```elixir +Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ []) +``` + +Stops the interactive browser session. + +## Notes + +- **Naming style:** Parameters use snake_case in Elixir and are serialized to camelCase JSON keys internally. +- **OpenAPI-shaped client:** The Elixir SDK is generated from the OpenAPI spec, so function names directly mirror operation IDs (`search_and_scrape`, `scrape_and_extract_from_url`, `interact_with_scrape_browser_session`). +- **Return type:** All functions return `{:ok, %Req.Response{}} | {:error, Exception.t() | Firecrawl.Error.t()}`. Bang variants raise on error. +- **No prompt parameter:** Unlike the JS, Python, and Rust SDKs, the Elixir SDK's interact function does not support a `prompt` parameter. Use `code` only. +- **Per-call auth:** Every function accepts `:api_key` and `:base_url` in the trailing `opts` keyword list, allowing different credentials per call. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..73bad1872 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,238 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Java SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```groovy +implementation 'com.firecrawl:firecrawl-java:1.12.1' +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .build(); +``` + +Builder options: + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `apiKey` | `String` | `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property | API key. Omit for keyless free tier (rate-limited per IP). | +| `apiUrl` | `String` | `"https://api.firecrawl.dev"` | Base URL. Falls back to `FIRECRAWL_API_URL` env var. | +| `timeoutMs` | `long` | `300000` (5 min) | Per-request timeout in milliseconds. | +| `maxRetries` | `int` | `3` | Max automatic retries for transient failures. | +| `backoffFactor` | `double` | `0.5` | Exponential backoff factor for retries. | +| `asyncExecutor` | `Executor` | `ForkJoinPool.commonPool()` | Executor for async methods. | +| `httpClient` | `OkHttpClient` | — | Custom HTTP client. Overrides `timeoutMs`. | + +A convenience factory `FirecrawlClient.fromEnv()` reads the API key from the `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```java +client.search(query, options) +``` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; + +SearchData results = client.search("firecrawl web scraping API", + SearchOptions.builder() + .limit(5) + .highlights(true) + .build() +); + +for (var item : results.getWeb()) { + System.out.println(item.get("url")); +} +``` + +### Parameters + +All fields on `SearchOptions` are nullable and optional. + +| Field | Type | Description | +|-------|------|-------------| +| `sources` | `List` | Source types: `"web"`, `"news"`, `"images"`, or config maps. | +| `categories` | `List` | Filter categories: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Max results to return. | +| `tbs` | `String` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Geographic location for results. | +| `ignoreInvalidURLs` | `Boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Defaults to true. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `String` | Integration identifier. | + +An async variant is available: `client.searchAsync(query, options)` returns `CompletableFuture`. + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```java +client.scrape(url, options) +``` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; +import java.util.List; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable and optional. + +| Field | Type | Description | +|-------|------|-------------| +| `formats` | `List` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Object variants also supported for JSON extraction (`JsonFormat`), questions (`QuestionFormat`), highlights (`HighlightsFormat`). | +| `headers` | `Map` | Custom HTTP headers. | +| `includeTags` | `List` | HTML tags to include exclusively. | +| `excludeTags` | `List` | HTML tags to exclude. | +| `onlyMainContent` | `Boolean` | Only return main content, excluding navbars/footers. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `waitFor` | `Integer` | Delay in ms before fetching content. | +| `mobile` | `Boolean` | Emulate a mobile device. | +| `parsers` | `List` | File processing parsers (e.g. `"pdf"` or config maps). | +| `actions` | `List>` | Browser actions before scraping. Action types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `LocationConfig` | Geolocation with `country` and `languages`. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64 images from output. | +| `blockAds` | `Boolean` | Block ads and cookie popups. | +| `proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `maxAge` | `Long` | Use cached result if younger than this (ms). | +| `storeInCache` | `Boolean` | Whether to cache the result. | +| `lockdown` | `Boolean` | Serve only cached results. | +| `redactPII` | `Boolean` | Redact personally identifiable information. | +| `auditMetadata` | `AuditMetadata` | User attribution for SIEM logging with `username`. | +| `integration` | `String` | Integration identifier. | + +An async variant is available: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +## Interact + +### Why use it + +Execute code in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```java +client.interact(jobId, code) +client.interact(jobId, code, language, timeout) +client.interact(jobId, code, language, timeout, origin) +``` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = (String) doc.getMetadata().get("jobId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "document.querySelector('button.load-more')?.click();", + "node", + 30 +); + +System.out.println(result.getStdout()); +``` + +### Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `jobId` | `String` | — | **Required.** The scrape job ID from a prior scrape. | +| `code` | `String` | — | **Required.** Code to execute in the browser sandbox. | +| `language` | `String` | `"node"` | Runtime language: `"python"`, `"node"`, or `"bash"`. | +| `timeout` | `Integer` | `null` | Execution timeout in seconds (1-300). Default: 30 server-side. | +| `origin` | `String` | `null` | Request origin tag. | + +Async variants are available: `client.interactAsync(...)` returns `CompletableFuture`. + +### Related method + +```java +client.stopInteractiveBrowser(jobId) +``` + +Stops the interactive browser session and returns billing info as `BrowserDeleteResponse`. + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Builder pattern:** `ScrapeOptions` and `SearchOptions` use builder pattern (`ScrapeOptions.builder()...build()`). +- **Deprecated aliases:** + - `scrapeExecute()` → use `interact()` instead. + - `deleteScrapeBrowser()` → use `stopInteractiveBrowser()` instead. +- **Async methods:** Every sync method has an async counterpart suffixed with `Async` that returns `CompletableFuture`. +- **Interact limitations:** The Java SDK's `interact` method requires `code` as a string parameter. Unlike the JS and Python SDKs, it does not support a `prompt` parameter for natural-language browser instructions. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..aab935143 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,211 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Node.js/TypeScript SDK. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +## Authenticate + +```typescript +import Firecrawl from "@mendable/firecrawl-js"; + +const client = new Firecrawl({ apiKey: "fc-YOUR_API_KEY" }); +``` + +Constructor options: + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `apiKey` | `string \| null` | `FIRECRAWL_API_KEY` env var | API key. Omit for keyless free tier (rate-limited per IP). | +| `apiUrl` | `string \| null` | `https://api.firecrawl.dev` | Base URL. Falls back to `FIRECRAWL_API_URL` env var. | +| `timeoutMs` | `number` | — | Per-request timeout in milliseconds. | +| `maxRetries` | `number` | — | Max automatic retries for transient failures. | +| `backoffFactor` | `number` | — | Exponential backoff factor for retries. | + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```typescript +client.search(query, options?) +``` + +### Example + +```typescript +const results = await client.search("firecrawl web scraping API", { + limit: 5, + scrapeOptions: { formats: ["markdown"] }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.markdown?.slice(0, 200)); +} +``` + +### Parameters + +All parameters are optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `string` | **Required (positional).** The search query. | +| `sources` | `Array<"web" \| "news" \| "images" \| { type: ... }>` | Source types to search. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer" \| { type: ... }>` | Filter by category. | +| `includeDomains` | `string[]` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. | +| `limit` | `number` | Max results to return. | +| `tbs` | `string` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `string` | Geographic location for results. | +| `ignoreInvalidURLs` | `boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `number` | Timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise zero data retention options. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Request origin tag. | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```typescript +client.scrape(url, options?) +``` + +### Example + +```typescript +const doc = await client.scrape("https://example.com", { + formats: ["markdown", "links"], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.links); +``` + +### Parameters + +All parameters are optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `string` | **Required (positional).** The URL to scrape. | +| `formats` | `FormatOption[]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Object variants: `{ type: "json", schema?, prompt? }`, `{ type: "screenshot", fullPage?, quality?, viewport? }`, `{ type: "changeTracking", modes, schema?, prompt?, tag? }`, `{ type: "attributes", selectors }`, `{ type: "question", question }`, `{ type: "highlights", query }`. | +| `headers` | `Record` | Custom HTTP headers to send with the request. | +| `includeTags` | `string[]` | HTML tags to include exclusively. | +| `excludeTags` | `string[]` | HTML tags to exclude. | +| `onlyMainContent` | `boolean` | Only return main content, excluding navbars/footers. | +| `timeout` | `number` | Timeout in milliseconds. | +| `waitFor` | `number` | Delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `Array` | File processing parsers. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `actions` | `ActionOption[]` | Browser actions to perform before scraping. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `{ country?: string, languages?: string[] }` | Geolocation for proxy routing. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64 images from output. | +| `fastMode` | `boolean` | Faster scraping with reduced accuracy. | +| `useMock` | `string` | Use a mock response. | +| `blockAds` | `boolean` | Block ads and cookie popups. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode. | +| `maxAge` | `number` | Use cached result if younger than this (ms). | +| `minAge` | `number` | Minimum cache age (ms). Set to 1 for any cached data. | +| `storeInCache` | `boolean` | Whether to cache the result. | +| `lockdown` | `boolean` | Serve only cached results. No outbound request. | +| `redactPII` | `boolean \| { mode?, entities?, replaceStyle? }` | Redact personally identifiable information. Modes: `"accurate"`, `"aggressive"`, `"fast"`. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | +| `profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser storage profile. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Request origin tag. | + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```typescript +client.interact(jobId, args) +``` + +### Example + +```typescript +const doc = await client.scrape("https://example.com", { + formats: ["markdown"], +}); + +const jobId = doc.metadata?.jobId; + +const result = await client.interact(jobId, { + code: "document.querySelector('button.load-more')?.click();", + language: "node", + timeout: 30, +}); + +console.log(result.stdout); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `jobId` | `string` | **Required (positional).** The scrape job ID from a prior scrape. | +| `code` | `string` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `string` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `"python" \| "node" \| "bash"` | Runtime language for `code`. | +| `timeout` | `number` | Execution timeout in seconds (1-300). | +| `origin` | `string` | Request origin tag. | + +### Related method + +```typescript +client.stopInteraction(jobId) +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Deprecated aliases:** + - `scrapeUrl()` → use `scrape()` instead. + - `scrapeExecute()` → use `interact()` instead. + - `stopInteractiveBrowser()` and `deleteScrapeBrowser()` → use `stopInteraction()` instead. +- **Async client:** The SDK is async by default (all methods return Promises). +- **Zod schemas:** The `json` format accepts Zod schemas in addition to plain JSON Schema objects for the `schema` field. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..d9ff92061 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,209 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Python SDK. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +client = Firecrawl(api_key="fc-YOUR_API_KEY") +``` + +Constructor parameters: + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `api_key` | `str` | `FIRECRAWL_API_KEY` env var | API key. Omit for keyless free tier (rate-limited per IP). | +| `api_url` | `str` | `"https://api.firecrawl.dev"` | Base URL. | +| `timeout` | `float` | `None` | Default request timeout in seconds. | +| `max_retries` | `int` | `3` | Max automatic retries for transient failures. | +| `backoff_factor` | `float` | `0.5` | Exponential backoff factor for retries. | + +An async client is also available: `from firecrawl import AsyncFirecrawl`. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```python +client.search(query, **kwargs) +``` + +### Example + +```python +results = client.search( + "firecrawl web scraping API", + limit=5, + scrape_options=ScrapeOptions(formats=["markdown"]), +) + +for item in results.web or []: + print(item.url, item.markdown[:200] if item.markdown else "") +``` + +### Parameters + +All parameters are keyword-only and optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `str` | **Required (positional).** The search query. | +| `sources` | `list[str \| Source]` | Source types to search (e.g. `"web"`, `"news"`, `"images"`). | +| `categories` | `list[str \| Category]` | Filter by category (e.g. `"github"`, `"research"`, `"pdf"`, `"developer"`). | +| `include_domains` | `list[str]` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. | +| `limit` | `int` | Max results to return. | +| `tbs` | `str` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `str` | Geographic location for results. | +| `ignore_invalid_urls` | `bool` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `int` | Timeout in milliseconds. | +| `highlights` | `bool` | Generate query-relevant highlights. | +| `scrape_options` | `ScrapeOptions` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `str` | Integration identifier. | +| `enterprise` | `list[str]` | Enterprise options (e.g. `"anon"`, `"zdr"`). | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```python +client.scrape(url, **kwargs) +``` + +### Example + +```python +doc = client.scrape( + "https://example.com", + formats=["markdown", "links"], + only_main_content=True, +) + +print(doc.markdown) +print(doc.links) +``` + +### Parameters + +All parameters are keyword-only and optional unless noted. + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `str` | **Required (positional).** The URL to scrape. | +| `formats` | `list[FormatOption]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Object variants also supported for `json`, `screenshot`, `changeTracking`, `attributes`, `question`, `highlights`. | +| `headers` | `dict[str, str]` | Custom HTTP headers to send with the request. | +| `include_tags` | `list[str]` | HTML tags to include exclusively. | +| `exclude_tags` | `list[str]` | HTML tags to exclude. | +| `only_main_content` | `bool` | Only return main content, excluding navbars/footers. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Delay in ms before fetching content. | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list[str \| PDFParser]` | File processing parsers. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `actions` | `list[Action]` | Browser actions to perform before scraping. Types: `WaitAction`, `ScreenshotAction`, `ClickAction`, `WriteAction`, `PressAction`, `ScrollAction`, `ScrapeAction`, `ExecuteJavascriptAction`, `PDFAction`. | +| `location` | `Location` | Geolocation config with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64 images from output. | +| `fast_mode` | `bool` | Faster scraping with reduced accuracy. | +| `use_mock` | `str` | Use a mock response. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached result if younger than this (ms). | +| `store_in_cache` | `bool` | Whether to cache the result. | +| `lockdown` | `bool` | Serve only cached results. No outbound request. | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `profile` | `dict` | Persistent browser storage profile with `name` and optional `save_changes`. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging with `username` field. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```python +client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None) +``` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.get("jobId") + +result = client.interact( + job_id, + code="document.querySelector('button.load-more')?.click();", + language="node", + timeout=30, +) + +print(result.stdout) +``` + +### Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `job_id` | `str` | — | **Required (positional).** The scrape job ID from a prior scrape. | +| `code` | `str` | `None` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `str` | `None` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `str` | `"node"` | Runtime language: `"python"`, `"node"`, or `"bash"`. | +| `timeout` | `int` | `None` | Execution timeout in seconds (1-300). | +| `origin` | `str` | `None` | Request origin tag. | + +### Related method + +```python +client.stop_interaction(job_id) +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All parameters use snake_case. +- **Deprecated aliases:** + - `scrape_url()` → use `scrape()` instead. + - `scrape_execute()` → use `interact()` instead. + - `stop_interactive_browser()` and `delete_scrape_browser()` → use `stop_interaction()` instead. + - `FirecrawlApp` → use `Firecrawl` instead (alias still works). +- **Async client:** Use `AsyncFirecrawl` (aliased as `AsyncFirecrawlApp`) for async/await usage with the same method signatures. +- **ScrapeOptions model:** The `ScrapeOptions` Pydantic model also includes `min_age` and `redact_pii` fields which are available when passing `scrape_options` to `search()` but are not direct kwargs on `scrape()`. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..918c5b97b --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,232 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This is the canonical quickstart for external agents integrating with Firecrawl using the official Rust SDK. Generated from SDK source and OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +// Firecrawl cloud +let client = Client::new("fc-YOUR_API_KEY")?; + +// Self-hosted (API key optional) +let client = Client::new_selfhosted("https://your-instance.com", Some("fc-YOUR_API_KEY"))?; +``` + +`Client::new` requires an API key. `Client::new_selfhosted` allows an optional key for keyless free-tier usage. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns search results grouped by source type, optionally with scraped content. +- **`scrape`** — Use when you already have a URL and want its content. Returns markdown, HTML, structured data, screenshots, or other formats. +- **`interact`** — Use when the page needs post-scrape browser actions like clicking, filling forms, or executing code in the browser sandbox. + +## Search + +### Why use it + +Search the web and optionally scrape each result in one call. Start here when you have a question or topic but not a specific URL. + +### Preferred SDK method + +```rust +client.search(query, options).await +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let response = client.search("firecrawl web scraping API", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `limit` | `Option` | Max results. Default: 5, Max: 20. | +| `sources` | `Option>` | Source types: `Web`, `News`, `Images`. | +| `categories` | `Option>` | Filter categories: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `Option` | Geographic location for results. | +| `ignore_invalid_urls` | `Option` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `highlights` | `Option` | Generate query-relevant highlights. Default: true. | +| `scrape_options` | `Option` | Options applied when scraping each result. See Scrape parameters. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | + +A convenience method `search_and_scrape(query, limit)` is also available. It searches with default scrape options and returns `Vec` directly. + +## Scrape + +### Why use it + +Fetch and extract content from a single URL. Use when you have a specific page to read. + +### Preferred SDK method + +```rust +client.scrape(url, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + only_main_content: Some(true), + ..Default::default() +}).await?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `formats` | `Option>` | Output formats. Enum values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(String)`, `Highlights(String)`, `Query(String)`. | +| `headers` | `Option>` | Custom HTTP headers. | +| `include_tags` | `Option>` | HTML tags to include exclusively. | +| `exclude_tags` | `Option>` | HTML tags to exclude. | +| `only_main_content` | `Option` | Only return main content. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `wait_for` | `Option` | Delay in ms before fetching content. | +| `mobile` | `Option` | Emulate a mobile device. | +| `parsers` | `Option>` | File processing parsers. `ParserConfig::Simple(String)` or `ParserConfig::Pdf { mode, max_pages }`. | +| `actions` | `Option>` | Browser actions before scraping. Types: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `location` | `Option` | Geolocation with `country` and `languages`. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64 images from output. | +| `fast_mode` | `Option` | Faster scraping with reduced accuracy. | +| `block_ads` | `Option` | Block ads. | +| `proxy` | `Option` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `max_age` | `Option` | Use cached result if younger than this (seconds). | +| `min_age` | `Option` | Minimum cache age (seconds). | +| `store_in_cache` | `Option` | Whether to cache the result. | +| `lockdown` | `Option` | Serve only cached results. | +| `redact_pii` | `Option` | Redact personally identifiable information. | +| `audit_metadata` | `Option` | User attribution for SIEM logging. | +| `profile` | `Option` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | +| `json_options` | `Option` | JSON extraction options with `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config with `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking config with `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute extraction with `selector` and `attribute`. | + +A convenience method `scrape_with_schema(url, schema, prompt)` is also available for JSON extraction. + +## Interact + +### Why use it + +Execute code or natural-language prompts in the browser sandbox associated with a scrape job. Use after a scrape to click buttons, fill forms, navigate, or extract additional data. + +### Preferred SDK method + +```rust +client.interact(job_id, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let job_id = doc.metadata.get("jobId").and_then(|v| v.as_str()).unwrap(); + +let result = client.interact(job_id, ScrapeExecuteOptions { + code: Some("document.querySelector('button.load-more')?.click();".into()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(30), + ..Default::default() +}).await?; + +println!("{:?}", result.stdout); +``` + +### Parameters + +All fields on `ScrapeExecuteOptions` are `Option` and default to `None`. + +| Field | Type | Description | +|-------|------|-------------| +| `code` | `Option` | Code to execute in the browser sandbox. Required if `prompt` is not provided. | +| `prompt` | `Option` | Natural-language instruction for the browser agent. Required if `code` is not provided. | +| `language` | `Option` | Runtime: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. | +| `origin` | `Option` | Request origin tag. Auto-set to `"rust-sdk@{version}"` if not provided. | + +At least one of `code` or `prompt` must be provided or the client returns `FirecrawlError::Misuse`. + +### Related method + +```rust +client.stop_interaction(job_id).await +``` + +Stops the interactive browser session and returns billing info. + +## Notes + +- **Naming style:** All struct fields use snake_case. Serialization to camelCase is handled internally by serde. +- **Deprecated aliases:** + - `scrape_execute()` → use `interact()` instead. + - `stop_interactive_browser()` and `delete_scrape_browser()` → use `stop_interaction()` instead. +- **Async:** All methods are async and return `Result`. +- **Origin auto-set:** The SDK automatically sets `origin` to `"rust-sdk@{version}"` if not explicitly provided. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/v2/client.rs` +- `firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `firecrawl/apps/rust-sdk/src/v2/search.rs` +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 4a5873a02..82d556cc3 100755 --- a/docs.json +++ b/docs.json @@ -680,6 +680,16 @@ "agents/fire-1-extract" ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Agentic Debugging", "pages": [