From 9f24e942a55507319e69d118978a9e85cb61e7f2 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 10 Aug 2026 13:30:41 +0000 Subject: [PATCH] docs: add agent quickstart guides for all SDK languages Add canonical one-file-per-language quickstart docs for external agents covering search, scrape, and interact endpoints with parameters sourced from SDK code and OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_019gGCKzr9SSjuVNj4jC1nBA --- agent-quickstart/elixir.mdx | 211 ++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 242 ++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 211 ++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 204 +++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 252 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1120 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..7de827704 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,211 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the Elixir SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +Add to your `mix.exs`: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.9"} + ] +end +``` + +## Authenticate + +The Elixir SDK is a flat module of stateless functions — there is no client struct to construct. Configure the API key globally or pass it per request. + +**Global config** (in `config.exs`): + +```elixir +config :firecrawl, api_key: "fc-YOUR_API_KEY" +``` + +**Per-request override:** + +```elixir +Firecrawl.scrape_and_extract_from_url([url: "https://example.com"], api_key: "fc-YOUR_API_KEY") +``` + +All functions accept trailing `opts` for `:api_key` and `:base_url` (defaults to `https://api.firecrawl.dev/v2`). + +## When To Use What + +- **`search_and_scrape`** — Use when you start with a query and need to discover relevant pages. Returns structured results grouped by source type. Optionally scrapes each result. +- **`scrape_and_extract_from_url`** — Use when you already have a URL and want page content in one or more formats (markdown, HTML, JSON extraction, etc.). +- **`interact_with_scrape_browser_session`** — Use when the page needs post-scrape browser actions: running code in a live browser session tied to a previous scrape job. + +## Search + +### Why use it + +Search the web for a query and optionally scrape each result. Returns results grouped by source type. Use this as the starting point when you do not yet have a specific URL. + +### Preferred SDK method + +``` +Firecrawl.search_and_scrape(params \\ [], opts \\ []) +``` + +Bang variant: `Firecrawl.search_and_scrape!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, response} = Firecrawl.search_and_scrape( + query: "firecrawl web scraping API", + limit: 5 +) + +IO.inspect(response.body) +``` + +### Parameters + +Parameters are passed as a keyword list. All are optional except `query`. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `:string` | **Required.** The search query. | +| `limit` | `:integer` | Max number of results. | +| `sources` | `{:list, :any}` | Sources to search. Default: `["web"]`. | +| `categories` | `{:list, :any}` | Categories to filter results. Default: `[]`. | +| `include_domains` | `{:list, :string}` | Restrict results to these domains. | +| `exclude_domains` | `{:list, :string}` | Exclude results from these domains. | +| `tbs` | `:string` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `:string` | Location for search results. | +| `country` | `:string` | ISO country code for geo-targeting (e.g. `"US"`). | +| `timeout` | `:integer` | Timeout in milliseconds. | +| `highlights` | `:boolean` | Generate query-relevant highlights. Default: `true`. | +| `ignore_invalid_urls` | `:boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `scrape_options` | `:keyword_list` | Options for scraping search results. | +| `enterprise` | `{:list, :string}` | Enterprise ZDR options: `["zdr"]` or `["anon"]`. | + +### Return shape + +`{:ok, %Req.Response{}}` — the response body contains results grouped by source type. + +## Scrape + +### Why use it + +Scrape a single URL and get its content in one or more formats. Use this when you have the exact URL you want content from. + +### Preferred SDK method + +``` +Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ []) +``` + +Bang variant: `Firecrawl.scrape_and_extract_from_url!(params, opts)` raises on error. + +### Example + +```elixir +{:ok, response} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "html"], + only_main_content: true +) + +IO.inspect(response.body) +``` + +### Parameters + +Parameters are passed as a keyword list. All are optional except `url`. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `:string` | **Required.** The URL to scrape. | +| `formats` | `{:list, :any}` | Output formats (e.g. `"markdown"`, `"html"`, `"json"`, `"screenshot"`). Default: `["markdown"]`. | +| `only_main_content` | `:boolean` | Only return main content. Default: `true`. | +| `include_tags` | `{:list, :string}` | HTML tags to include. | +| `exclude_tags` | `{:list, :string}` | HTML tags to exclude. | +| `headers` | `:any` | Custom HTTP headers. | +| `timeout` | `:integer` | Timeout in ms. Default: `60000`. Min 1000, max 300000. | +| `wait_for` | `:integer` | Delay in ms before fetching content. | +| `mobile` | `:boolean` | Emulate a mobile device. | +| `parsers` | `{:list, :any}` | File parser config. Default: `["pdf"]`. | +| `actions` | `{:list, :any}` | Browser actions to perform before grabbing content. | +| `location` | `:keyword_list` | Location settings (proxy, language, timezone). | +| `skip_tls_verification` | `:boolean` | Skip TLS certificate verification. | +| `remove_base64_images` | `:boolean` | Remove base64 images from markdown. Default: `true`. | +| `block_ads` | `:boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `{:in, [:basic, :enhanced, :auto]}` | Proxy type. Default: `"auto"`. | +| `max_age` | `:integer` | Max cache age in ms. Default: `172800000` (2 days). | +| `min_age` | `:integer` | Cache-only mode. Min age in ms. Set to `1` for any cache. | +| `store_in_cache` | `:boolean` | Store result in cache. Default: `true`. | +| `lockdown` | `:boolean` | Only serve cached results. | +| `redact_pii` | `:boolean` | Redact PII from content. | +| `zero_data_retention` | `:boolean` | Enable zero data retention. | +| `profile` | `:keyword_list` | Persistent browser profile. | +| `audit_metadata` | `:keyword_list` | SIEM logging attribution (`username` required). | + +## Interact + +### Why use it + +Execute code in a live browser session tied to a previous scrape job. Use this for post-scrape interactions like clicking buttons, filling forms, or running scripts. + +### Preferred SDK method + +``` +Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ []) +``` + +Bang variant: `Firecrawl.interact_with_scrape_browser_session!(job_id, params, opts)` raises on error. + +### Example + +```elixir +{:ok, scrape_response} = Firecrawl.scrape_and_extract_from_url(url: "https://example.com") +job_id = scrape_response.body["data"]["metadata"]["jobId"] + +{:ok, response} = Firecrawl.interact_with_scrape_browser_session(job_id, + code: "document.querySelector('button#submit').click();", + language: :node, + timeout: 30 +) + +IO.inspect(response.body) +``` + +### Parameters + +The first argument is the `job_id` (path parameter). Remaining parameters are passed as a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t()` | **Required** (first argument). The scrape job ID. | +| `code` | `:string` | **Required.** Code to execute in the browser session. | +| `language` | `{:in, [:python, :node, :bash]}` | Execution language. Use `:node` for JavaScript or `:bash` for agent-browser CLI commands. | +| `timeout` | `:integer` | Execution timeout in seconds. | +| `origin` | `:string` | Origin label for telemetry. | + +### Related methods + +- `Firecrawl.stop_interactive_scrape_browser_session(job_id, opts)` — Stop the browser session (`DELETE /scrape/{jobId}/interact`). + +## Notes + +- The SDK is **auto-generated from the OpenAPI spec**. Function names map 1:1 to OpenAPI operations. +- Parameter names use **snake_case** in Elixir. They are converted to camelCase for the JSON wire format automatically. +- All functions return `{:ok, %Req.Response{}} | {:error, Exception.t() | Firecrawl.Error.t()}`. +- Bang variants (e.g. `search_and_scrape!`) raise on error instead of returning an error tuple. +- There is no client struct — the module is a flat namespace of stateless functions. +- HTTP errors (4xx/5xx) are wrapped in `Firecrawl.Error` with `:status` and `:body` fields. +- Batch scraping is available via `Firecrawl.scrape_and_extract_from_urls/2` (`POST /batch/scrape`). + +## Source Of Truth + +- SDK: `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- OpenAPI: `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..d2c328796 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,242 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the Java SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +**Maven:** + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +**Gradle:** + +```groovy +implementation 'com.firecrawl:firecrawl-java:1.12.1' +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .build(); +``` + +Or from environment variables: + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +`fromEnv()` reads `FIRECRAWL_API_KEY` (env var) or `firecrawl.apiKey` (system property). + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns structured results grouped by source type (web, news, images). Optionally scrapes each result. +- **`scrape`** — Use when you already have a URL and want page content in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when the page needs post-scrape browser actions: clicking buttons, filling forms, or running code in a live browser session. + +## Search + +### Why use it + +Search the web for a query and optionally scrape each result. Returns results grouped by source type. Use this as the starting point when you do not yet have a specific URL. + +### Preferred SDK method + +``` +client.search(query) +client.search(query, options) +``` + +Async variant: `client.searchAsync(query, options)` returns `CompletableFuture`. + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; +import com.firecrawl.models.ScrapeOptions; + +SearchData results = client.search("firecrawl web scraping API", + SearchOptions.builder() + .limit(5) + .scrapeOptions(ScrapeOptions.builder() + .formats(List.of("markdown")) + .build()) + .build()); + +for (var item : results.getWeb()) { + System.out.println(item.get("url") + " " + item.get("title")); +} +``` + +### Parameters + +`SearchOptions` uses a builder pattern. All fields are optional and default to `null` (server defaults apply). + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | **Required** (method argument). The search query. | +| `sources` | `List` | Sources to query: `"web"`, `"news"`, `"images"` as strings or typed maps. Default: `["web"]`. | +| `categories` | `List` | Categories: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict results to these domains. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Max number of results. | +| `tbs` | `String` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Location for search results. | +| `ignoreInvalidURLs` | `Boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Default: `true` (server-side). | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each result. | +| `integration` | `String` | Integration identifier. | + +### Return shape + +```java +SearchData { + List> web; + List> news; + List> images; +} +``` + +## Scrape + +### Why use it + +Scrape a single URL and get its content in one or more formats. Use this when you have the exact URL you want content from. + +### Preferred SDK method + +``` +client.scrape(url) +client.scrape(url, options) +``` + +Async variant: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "html")) + .onlyMainContent(true) + .build()); + +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +`ScrapeOptions` uses a builder pattern. All fields are optional and default to `null`. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | **Required** (method argument). The URL to scrape. | +| `formats` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Also accepts typed objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). Default: `["markdown"]`. | +| `onlyMainContent` | `Boolean` | Only return main content. Default: `true`. | +| `includeTags` | `List` | HTML tags to include. | +| `excludeTags` | `List` | HTML tags to exclude. | +| `headers` | `Map` | Custom HTTP headers. | +| `timeout` | `Integer` | Timeout in milliseconds. Default: `60000`. | +| `waitFor` | `Integer` | Delay in ms before fetching content. | +| `mobile` | `Boolean` | Emulate a mobile device. | +| `parsers` | `List` | File parser config. Default: `["pdf"]`. | +| `actions` | `List>` | Browser actions (wait, click, write, press, scroll, etc.). | +| `location` | `LocationConfig` | Location and language settings (`country`, `languages`). | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64 images from markdown. Default: `true`. | +| `blockAds` | `Boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `String` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `maxAge` | `Long` | Max cache age in ms. Default: `172800000` (2 days). | +| `storeInCache` | `Boolean` | Store result in cache. Default: `true`. | +| `lockdown` | `Boolean` | Only serve cached results. | +| `redactPII` | `Boolean` | Redact PII from content. | +| `auditMetadata` | `AuditMetadata` | SIEM logging attribution. Constructed via `new AuditMetadata("username")`. | +| `integration` | `String` | Integration identifier. | + +## Interact + +### Why use it + +Execute code in a live browser session tied to a previous scrape job. Use this for post-scrape interactions like clicking buttons, filling forms, or running scripts. + +### Preferred SDK method + +``` +client.interact(jobId, code) +client.interact(jobId, code, language, timeout) +client.interact(jobId, code, language, timeout, origin) +``` + +Async variants are available for all overloads. + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; +import com.firecrawl.models.Document; + +Document doc = client.scrape("https://example.com"); +String jobId = (String) doc.getMetadata().get("jobId"); + +BrowserExecuteResponse response = client.interact( + jobId, + "document.querySelector('button#submit').click();", + "node", + 30 +); + +System.out.println(response.getStdout()); +``` + +### Parameters + +Parameters are positional (no options object). + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | **Required.** The scrape job ID from a previous `scrape` call. | +| `code` | `String` | **Required.** Code to execute in the browser session. | +| `language` | `String` | Execution language: `"python"`, `"node"`, or `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Default: `30` (server-side). | +| `origin` | `String` | Origin identifier. Auto-set to `"java-sdk@{version}"` if not provided. | + +### Related methods + +- `client.stopInteractiveBrowser(jobId)` — Stop the browser session and release resources. + +## Notes + +- All parameter names use **camelCase** (Java convention). +- The client builder also accepts `timeoutMs` (default `300000`), `maxRetries` (default `3`), `backoffFactor` (default `0.5`), and `asyncExecutor`. +- Typed format helpers: `JsonFormat.builder().prompt("...").schema(map).build()`, `QuestionFormat.builder().question("...").build()`, `HighlightsFormat.builder().query("...").build()`. +- `LocationConfig` is built via `LocationConfig.builder().country("US").languages(List.of("en")).build()`. +- **Deprecated aliases** (do not use in new code): + - `scrapeExecute()` → use `interact()` + - `deleteScrapeBrowser()` → use `stopInteractiveBrowser()` + +## Source Of Truth + +- SDK: `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java`, `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/` +- OpenAPI: `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..5ddf01fb0 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,211 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the JavaScript/TypeScript SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +## Authenticate + +```javascript +import Firecrawl from "@mendable/firecrawl-js"; + +const client = new Firecrawl("fc-YOUR_API_KEY"); +``` + +You can also pass options: + +```javascript +const client = new Firecrawl({ + apiKey: "fc-YOUR_API_KEY", + apiUrl: "https://api.firecrawl.dev", +}); +``` + +If omitted, `apiKey` falls back to the `FIRECRAWL_API_KEY` environment variable, and `apiUrl` falls back to `FIRECRAWL_API_URL`. + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns structured results grouped by source type (web, news, images). Optionally scrapes each result. +- **`scrape`** — Use when you already have a URL and want page content in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when the page needs post-scrape browser actions: clicking buttons, filling forms, running code in a live browser session, or sending a natural-language prompt to a browser agent. + +## Search + +### Why use it + +Search the web for a query and optionally scrape each result. Returns results grouped by source type. Use this as the starting point when you do not yet have a specific URL. + +### Preferred SDK method + +``` +client.search(query, options?) +``` + +### Example + +```javascript +const results = await client.search("firecrawl web scraping API", { + limit: 5, + scrapeOptions: { formats: ["markdown"] }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** The search query. | +| `limit` | `number` | Max number of results to return. | +| `sources` | `Array<"web" \| "news" \| "images" \| {type: ...}>` | Which search sources to query. Default: `["web"]`. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer" \| {type: ...}>` | Content categories to filter by. | +| `includeDomains` | `string[]` | Restrict results to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. Cannot combine with `includeDomains`. | +| `tbs` | `string` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `string` | Location for search results (e.g. `"San Francisco,California,United States"`). | +| `ignoreInvalidURLs` | `boolean` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `number` | Timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping each search result. Same shape as the `scrape` options. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise ZDR options. | +| `threatProtection` | `ThreatProtectionOptions` | Enterprise per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin identifier. | + +### Return shape + +```typescript +{ + web?: Array, + news?: Array, + images?: Array, + developer?: Array, +} +``` + +## Scrape + +### Why use it + +Scrape a single URL and get its content in one or more formats. Use this when you have the exact URL you want content from. + +### Preferred SDK method + +``` +client.scrape(url, options?) +``` + +### Example + +```javascript +const doc = await client.scrape("https://example.com", { + formats: ["markdown", "html"], + onlyMainContent: true, +}); + +console.log(doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** The URL to scrape. | +| `formats` | `FormatOption[]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts typed objects for `json`, `screenshot`, `question`, `highlights`. Default: `["markdown"]`. | +| `onlyMainContent` | `boolean` | Only return main content, excluding headers/navs/footers. Default: `true`. | +| `includeTags` | `string[]` | HTML tags to include in output. | +| `excludeTags` | `string[]` | HTML tags to exclude from output. | +| `headers` | `Record` | Custom HTTP headers sent with the request. | +| `timeout` | `number` | Timeout in milliseconds. Default: `60000`. Min 1000, max 300000. | +| `waitFor` | `number` | Delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `Array` | File parser config. Default: `["pdf"]`. | +| `actions` | `ActionOption[]` | Browser actions to perform before grabbing content (wait, click, write, press, scroll, screenshot, scrape, executeJavascript, pdf). | +| `location` | `{country?: string, languages?: string[]}` | Location and language settings. Uses appropriate proxy. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64-encoded images from markdown. Default: `true`. | +| `fastMode` | `boolean` | Enable fast mode for quicker scraping. | +| `blockAds` | `boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy type. Default: `"auto"`. | +| `maxAge` | `number` | Max cache age in ms. Use cached result if younger. Default: `172800000` (2 days). Set to `0` to bypass cache. | +| `minAge` | `number` | Cache-only mode. Min age in ms. Set to `1` for any cached version. | +| `storeInCache` | `boolean` | Store result in Firecrawl cache. Default: `true`. | +| `lockdown` | `boolean` | Only serve cached results, never make outbound requests. | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact PII from content. Pass `true` for defaults or an options object with `mode`, `entities`, `replaceStyle`. | +| `profile` | `{name: string, saveChanges?: boolean}` | Persistent browser profile across scrape/interact sessions. | +| `auditMetadata` | `{username: string}` | User attribution for SIEM logging. | +| `threatProtection` | `ThreatProtectionOptions` | Enterprise per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin identifier. | + +## Interact + +### Why use it + +Execute code or send a natural-language prompt in a live browser session tied to a previous scrape job. Use this for post-scrape interactions like clicking buttons, filling forms, or navigating multi-step flows. + +### Preferred SDK method + +``` +client.interact(jobId, args) +``` + +### Example + +```javascript +const scrapeResult = await client.scrape("https://example.com", { + formats: ["markdown"], +}); +const jobId = scrapeResult.metadata?.jobId; + +const response = await client.interact(jobId, { + code: "document.querySelector('button#submit').click();", + language: "node", + timeout: 30, +}); + +console.log(response.stdout); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | **Required.** The scrape job ID from a previous `scrape` call. | +| `code` | `string` | Code to execute in the browser session. One of `code` or `prompt` is required. | +| `prompt` | `string` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `language` | `"python" \| "node" \| "bash"` | Execution language. Default: `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. Min 1, max 300. Default: `30`. | +| `origin` | `string` | Origin identifier for telemetry. | + +### Related methods + +- `client.stopInteraction(jobId)` — Stop the browser session and release resources. + +## Notes + +- All parameter names use **camelCase**. +- The SDK auto-sets an `origin` field on requests for telemetry. +- `search` results are accessed via `.web`, `.news`, `.images`, or `.developer` — there is no `.data` property. +- The constructor also accepts `timeoutMs`, `maxRetries`, and `backoffFactor` for HTTP-level configuration. +- **Deprecated aliases** (do not use in new code): + - `scrapeUrl()` → use `scrape()` + - `scrapeExecute()` → use `interact()` + - `stopInteractiveBrowser()` / `deleteScrapeBrowser()` → use `stopInteraction()` + +## Source Of Truth + +- SDK: `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts`, `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- OpenAPI: `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..cd4f2e23f --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,204 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the Python SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +client = Firecrawl(api_key="fc-YOUR_API_KEY") +``` + +If omitted, `api_key` falls back to the `FIRECRAWL_API_KEY` environment variable. An async client is also available: + +```python +from firecrawl import AsyncFirecrawl + +client = AsyncFirecrawl(api_key="fc-YOUR_API_KEY") +``` + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns structured results grouped by source type (web, news, images). Optionally scrapes each result. +- **`scrape`** — Use when you already have a URL and want page content in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when the page needs post-scrape browser actions: clicking buttons, filling forms, running code in a live browser session, or sending a natural-language prompt to a browser agent. + +## Search + +### Why use it + +Search the web for a query and optionally scrape each result. Returns results grouped by source type. Use this as the starting point when you do not yet have a specific URL. + +### Preferred SDK method + +``` +client.search(query, **kwargs) +``` + +### Example + +```python +results = client.search( + "firecrawl web scraping API", + limit=5, + scrape_options={"formats": ["markdown"]}, +) + +for item in results.web or []: + print(item.url, item.title) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | **Required.** The search query. | +| `limit` | `int` | Max number of results. Default: `5`. | +| `sources` | `list` | Which search sources to query (`"web"`, `"news"`, `"images"` or `Source` objects). Default: `["web"]`. | +| `categories` | `list` | Content categories (`"github"`, `"research"`, `"pdf"`, `"developer"` or `Category` objects). | +| `include_domains` | `list[str]` | Restrict results to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Cannot combine with `include_domains`. | +| `tbs` | `str` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `str` | Location for search results (e.g. `"San Francisco,California,United States"`). Note: this is a plain string, not a `Location` object. | +| `ignore_invalid_urls` | `bool` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `int` | Timeout in milliseconds. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. Default: `True`. | +| `scrape_options` | `ScrapeOptions` or `dict` | Options applied when scraping each result. Same shape as `scrape` options. | +| `enterprise` | `list[str]` | Enterprise ZDR options: `["zdr"]` or `["anon"]`. | +| `threat_protection` | `ThreatProtectionOptions` | Enterprise per-request threat protection override. | +| `integration` | `str` | Integration identifier. | + +### Return shape + +```python +SearchData( + web: list[SearchResultWeb | Document] | None, + news: list[SearchResultNews | Document] | None, + images: list[SearchResultImages | Document] | None, + developer: list[SearchResultWeb | Document] | None, +) +``` + +## Scrape + +### Why use it + +Scrape a single URL and get its content in one or more formats. Use this when you have the exact URL you want content from. + +### Preferred SDK method + +``` +client.scrape(url, **kwargs) +``` + +### Example + +```python +doc = client.scrape( + "https://example.com", + formats=["markdown", "html"], + only_main_content=True, +) + +print(doc.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | **Required.** The URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts typed format objects. Default: `["markdown"]`. | +| `only_main_content` | `bool` | Only return main content, excluding headers/navs/footers. Default: `True`. | +| `include_tags` | `list[str]` | HTML tags to include in output. | +| `exclude_tags` | `list[str]` | HTML tags to exclude from output. | +| `headers` | `dict[str, str]` | Custom HTTP headers sent with the request. | +| `timeout` | `int` | Timeout in milliseconds. Default: `60000`. Min 1000, max 300000. | +| `wait_for` | `int` | Delay in ms before fetching content. | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list` | File parser config (e.g. `["pdf"]` or `[{"type": "pdf", "mode": "auto"}]`). Default: `["pdf"]`. | +| `actions` | `list` | Browser actions to perform before grabbing content (wait, click, write, press, scroll, screenshot, scrape, executeJavascript, pdf). | +| `location` | `Location` | Location and language settings (`country`, `languages`). Uses appropriate proxy. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64-encoded images from markdown. Default: `True`. | +| `fast_mode` | `bool` | Enable fast mode for quicker scraping. | +| `block_ads` | `bool` | Block ads and cookie popups. Default: `True`. | +| `proxy` | `str` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `max_age` | `int` | Max cache age in ms. Default: `172800000` (2 days). Set to `0` to bypass cache. | +| `store_in_cache` | `bool` | Store result in Firecrawl cache. Default: `True`. | +| `lockdown` | `bool` | Only serve cached results, never make outbound requests. | +| `threat_protection` | `ThreatProtectionOptions` | Enterprise per-request threat protection override. | +| `profile` | `dict` | Persistent browser profile (e.g. `{"name": "my-profile", "saveChanges": True}`). | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging (`{"username": "..."}`). | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Execute code or send a natural-language prompt in a live browser session tied to a previous scrape job. Use this for post-scrape interactions like clicking buttons, filling forms, or navigating multi-step flows. + +### Preferred SDK method + +``` +client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None, origin=None) +``` + +### Example + +```python +scrape_result = client.scrape("https://example.com", formats=["markdown"]) +job_id = scrape_result.metadata.get("jobId") + +response = client.interact( + job_id, + code="document.querySelector('button#submit').click();", + language="node", + timeout=30, +) + +print(response.stdout) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | **Required.** The scrape job ID from a previous `scrape` call. | +| `code` | `str` | Code to execute in the browser session. One of `code` or `prompt` is required. | +| `prompt` | `str` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `language` | `"python" \| "node" \| "bash"` | Execution language. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds (1–300). | +| `origin` | `str` | Origin identifier for telemetry. | + +### Related methods + +- `client.stop_interaction(job_id)` — Stop the browser session and release resources. + +## Notes + +- All parameter names use **snake_case**. +- The constructor also accepts `timeout` (in seconds), `max_retries` (default `3`), and `backoff_factor` (default `0.5`). +- `search` results are accessed via `.web`, `.news`, `.images`, or `.developer` — there is no `.data` attribute. +- `FirecrawlApp` and `AsyncFirecrawlApp` are aliases for `Firecrawl` and `AsyncFirecrawl`. +- **Deprecated aliases** (do not use in new code): + - `scrape_url()` → use `scrape()` + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + +## Source Of Truth + +- SDK: `firecrawl/apps/python-sdk/firecrawl/v2/client.py`, `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- OpenAPI: `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..f90a84d47 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,252 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl via the Rust SDK. It is generated from SDK source and the OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR_API_KEY")?; +``` + +For self-hosted instances: + +```rust +let client = Client::new_selfhosted("https://your-instance.com", Some("fc-YOUR_API_KEY"))?; +``` + +The API key is optional — omit it or pass `None` for keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search`** — Use when you start with a query and need to discover relevant pages. Returns structured results grouped by source type (web, news, images). Optionally scrapes each result. +- **`scrape`** — Use when you already have a URL and want page content in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). +- **`interact`** — Use when the page needs post-scrape browser actions: clicking buttons, filling forms, running code in a live browser session, or sending a natural-language prompt to a browser agent. + +## Search + +### Why use it + +Search the web for a query and optionally scrape each result. Returns results grouped by source type. Use this as the starting point when you do not yet have a specific URL. + +### Preferred SDK method + +``` +client.search(query, options) +``` + +A convenience wrapper is also available: + +``` +client.search_and_scrape(query, limit) -> Vec +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let results = client.search("firecrawl web scraping API", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +})?; + +if let Some(web) = results.data.web { + for item in web { + println!("{:?}", item); + } +} +``` + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. Use `..Default::default()` for unset fields. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | **Required** (method argument). The search query. | +| `limit` | `Option` | Max number of results. Default: `5`, max: `20`. | +| `sources` | `Option>` | Sources to query: `SearchSource::Web`, `News`, `Images`. Default: `[Web]`. | +| `categories` | `Option>` | Categories: `SearchCategory::Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict results to these domains. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `Option` | Location for search results. | +| `ignore_invalid_urls` | `Option` | Exclude URLs invalid for other Firecrawl endpoints. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `highlights` | `Option` | Generate query-relevant highlights. Default: `true` (server-side). | +| `scrape_options` | `Option` | Options applied when scraping each result. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +### Return shape + +```rust +SearchResponse { + success: bool, + data: SearchData { + web: Option>, + news: Option>, + images: Option>, + }, + warning: Option, +} +``` + +`SearchResultOrDocument` is an enum: `WebResult(SearchResultWeb)` or `Document(Document)`. + +## Scrape + +### Why use it + +Scrape a single URL and get its content in one or more formats. Use this when you have the exact URL you want content from. + +### Preferred SDK method + +``` +client.scrape(url, options) +``` + +A convenience wrapper for JSON extraction: + +``` +client.scrape_with_schema(url, schema, prompt) +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Html]), + only_main_content: Some(true), + ..Default::default() +})?; + +println!("{}", doc.markdown.unwrap_or_default()); +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. Use `..Default::default()` for unset fields. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | **Required** (method argument). The URL to scrape. | +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `Json`, `ChangeTracking`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. Default: `[Markdown]`. | +| `only_main_content` | `Option` | Only return main content. Default: `true`. | +| `include_tags` | `Option>` | HTML tags to include. | +| `exclude_tags` | `Option>` | HTML tags to exclude. | +| `headers` | `Option>` | Custom HTTP headers. | +| `timeout` | `Option` | Timeout in ms. Default: `60000`. | +| `wait_for` | `Option` | Delay in ms before fetching content. | +| `mobile` | `Option` | Emulate a mobile device. | +| `parsers` | `Option>` | File parser config. Default: `["pdf"]`. | +| `actions` | `Option>` | Browser automation actions (Wait, Click, Write, Press, Scroll, Screenshot, Scrape, ExecuteJavascript, Pdf). | +| `location` | `Option` | Location and language settings (`country`, `languages`). | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64 images from markdown. Default: `true`. | +| `fast_mode` | `Option` | Enable fast mode. | +| `block_ads` | `Option` | Block ads. Default: `true`. | +| `proxy` | `Option` | Proxy type: `Basic`, `Stealth`, `Enhanced`, `Auto`. Default: `Auto`. | +| `max_age` | `Option` | Max cache age in seconds. | +| `min_age` | `Option` | Cache-only mode, min age in seconds. | +| `store_in_cache` | `Option` | Store result in cache. Default: `true`. | +| `lockdown` | `Option` | Only serve cached results. | +| `redact_pii` | `Option` | Redact PII from content. | +| `profile` | `Option` | Persistent browser profile (`name`, `save_changes`). | +| `audit_metadata` | `Option` | SIEM logging attribution (`username`). | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction options (`schema`, `system_prompt`, `prompt`). Used with `Format::Json`. | +| `screenshot_options` | `Option` | Screenshot options (`full_page`, `quality`, `viewport`). | +| `change_tracking_options` | `Option` | Change tracking options (`modes`, `schema`, `prompt`, `tag`). | +| `attribute_selectors` | `Option>` | Attribute extraction selectors (`selector`, `attribute`). | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +## Interact + +### Why use it + +Execute code or send a natural-language prompt in a live browser session tied to a previous scrape job. Use this for post-scrape interactions like clicking buttons, filling forms, or navigating multi-step flows. + +### Preferred SDK method + +``` +client.interact(job_id, options) +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, ScrapeExecuteLanguage}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions::default())?; +let job_id = doc.metadata.as_ref() + .and_then(|m| m.get("jobId")) + .and_then(|v| v.as_str()) + .unwrap(); + +let response = client.interact(job_id, ScrapeExecuteOptions { + code: Some("document.querySelector('button#submit').click();".into()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(30), + ..Default::default() +})?; + +println!("{}", response.stdout.unwrap_or_default()); +``` + +### Parameters + +All fields on `ScrapeExecuteOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | **Required** (method argument). The scrape job ID. | +| `code` | `Option` | Code to execute in the browser session. One of `code` or `prompt` is required. | +| `prompt` | `Option` | Natural-language instruction for the browser agent. One of `code` or `prompt` is required. | +| `language` | `Option` | Execution language: `Python`, `Node`, `Bash`. Default: `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +### Related methods + +- `client.stop_interaction(job_id)` — Stop the browser session and release resources. + +## Notes + +- The SDK uses **snake_case** for struct fields. Serialization to the API uses camelCase via `#[serde(rename_all = "camelCase")]`. +- Options structs use `#[derive(Default)]` — construct with `..Default::default()` for unset fields. +- Method signatures accept `impl Into>`, so you can pass the options struct directly without wrapping in `Some()`. +- All public types are re-exported from the crate root (`use firecrawl::*`). +- **Deprecated aliases** (do not use in new code): + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + +## Source Of Truth + +- SDK: `firecrawl/apps/rust-sdk/src/v2/client.rs`, `firecrawl/apps/rust-sdk/src/v2/scrape.rs`, `firecrawl/apps/rust-sdk/src/v2/search.rs` +- OpenAPI: `firecrawl-docs/api-reference/v2-openapi.json`