From c6f06c8e2cd3b4d469c4e45ec0e5f2e244ec0e83 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 18 Aug 2026 13:17:59 +0000 Subject: [PATCH] Add agent quickstart docs for all SDK languages One canonical quickstart per language (Node.js, Python, Rust, Java, Elixir) covering search, scrape, and interact endpoints with confirmed parameters from SDK source and OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01VA1bTWz8x4be43M5xSHt6P --- agent-quickstart/elixir.mdx | 234 ++++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 264 ++++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 219 ++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 226 ++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 256 ++++++++++++++++++++++++++++++++++ 5 files changed, 1199 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 00000000..6e6ca43d --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,234 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +og:title: "Elixir Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Elixir SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +Add to your `mix.exs`: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.9"} + ] +end +``` + +Then run: + +```bash +mix deps.get +``` + +## Authenticate + +Configure via application config: + +```elixir +# config/config.exs +config :firecrawl, api_key: "fc-YOUR_API_KEY" +``` + +Or pass the API key per-request via the `opts` keyword: + +```elixir +Firecrawl.search_and_scrape([query: "example"], api_key: "fc-YOUR_API_KEY") +``` + +For self-hosted instances, set the base URL: + +```elixir +config :firecrawl, + api_key: "fc-YOUR_API_KEY", + base_url: "https://your-firecrawl-instance.com/v2" +``` + +The default base URL is `"https://api.firecrawl.dev/v2"`. Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search_and_scrape`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape_and_extract_from_url`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact_with_scrape_browser_session`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type. + +### Preferred SDK function + +```elixir +Firecrawl.search_and_scrape(params, opts \\ []) +Firecrawl.search_and_scrape!(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, results} = Firecrawl.search_and_scrape( + query: "firecrawl web scraping API", + limit: 5, + highlights: true +) + +IO.inspect(results) +``` + +The bang variant raises `Firecrawl.Error` on failure: + +```elixir +results = Firecrawl.search_and_scrape!( + query: "firecrawl web scraping API", + limit: 5 +) +``` + +### Parameters + +All parameters are passed as a keyword list. Only `query` is required. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `:string` | The search query (required). | +| `limit` | `:integer` | Maximum number of results to return. | +| `sources` | `{:list, :any}` | Source types to search. Defaults to `["web"]`. | +| `categories` | `{:list, :any}` | Filter results by category. | +| `include_domains` | `{:list, :string}` | Only include results from these domains. | +| `exclude_domains` | `{:list, :string}` | Exclude results from these domains. | +| `tbs` | `:string` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `:string` | Geographic location for search results (e.g. `"San Francisco,California,United States"`). | +| `country` | `:string` | ISO country code for geo-targeting (e.g. `"US"`). | +| `ignore_invalid_urls` | `:boolean` | Exclude invalid URLs from results. | +| `timeout` | `:integer` | Timeout in milliseconds. | +| `highlights` | `:boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `:keyword_list` | Options for scraping search results. Accepts the same parameters as scrape. | +| `enterprise` | `{:list, :string}` | Enterprise ZDR options: `["zdr"]` or `["anon"]`. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK function + +```elixir +Firecrawl.scrape_and_extract_from_url(params, opts \\ []) +Firecrawl.scrape_and_extract_from_url!(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, doc} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"] +) + +IO.puts(doc["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. Only `url` is required. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `:string` | The URL to scrape (required). | +| `formats` | `{:list, :any}` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"audio"`, `"video"`, etc. | +| `headers` | `:any` | Custom HTTP headers (cookies, user-agent, etc.). | +| `include_tags` | `{:list, :string}` | Only include content from these HTML tags. | +| `exclude_tags` | `{:list, :string}` | Exclude content from these HTML tags. | +| `only_main_content` | `:boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `:integer` | Timeout in milliseconds. Default: `60000`, Min: `1000`, Max: `300000`. | +| `wait_for` | `:integer` | Wait time in milliseconds before scraping. | +| `mobile` | `:boolean` | Scrape as a mobile device. | +| `parsers` | `{:list, :any}` | File processing control (e.g. PDF settings). | +| `actions` | `{:list, :any}` | Actions to execute before content extraction. | +| `location` | `:keyword_list` | Geolocation settings. Defaults to US. | +| `skip_tls_verification` | `:boolean` | Skip TLS certificate verification. | +| `remove_base64_images` | `:boolean` | Remove base64-encoded images from markdown output. | +| `block_ads` | `:boolean` | Block advertisements and cookie popups. | +| `proxy` | `{:in, [:basic, :enhanced, :auto]}` | Proxy type. | +| `max_age` | `:integer` | Use cached result if younger than this many milliseconds. Defaults to 2 days. | +| `min_age` | `:integer` | Cache-only mode; minimum age in milliseconds. | +| `store_in_cache` | `:boolean` | Whether to cache the result. | +| `lockdown` | `:boolean` | Only serve previously cached results. | +| `redact_pii` | `:boolean` | Redact personally identifiable information. | +| `audit_metadata` | `:keyword_list` | User attribution for SIEM logging. Keys: `username` (required). | +| `profile` | `:keyword_list` | Persistent browser storage across sessions. | +| `zero_data_retention` | `:boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior scrape call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK function + +```elixir +Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ []) +Firecrawl.interact_with_scrape_browser_session!(job_id, params, opts \\ []) +``` + +### Example + +```elixir +{:ok, doc} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = doc["data"]["metadata"]["jobId"] + +{:ok, result} = Firecrawl.interact_with_scrape_browser_session(job_id, + code: "document.querySelector('button.load-more').click()", + language: :node, + timeout: 30 +) + +IO.inspect(result) +``` + +### Parameters + +The `job_id` is passed as the first positional argument. Remaining parameters are a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t()` | The scrape job ID from a prior scrape call (required, first argument). | +| `code` | `:string` | Code to execute in the browser sandbox (required). | +| `language` | `{:in, [:python, :node, :bash]}` | Language for code execution. | +| `timeout` | `:integer` | Execution timeout in seconds. | +| `origin` | `:string` | Origin label for telemetry. | + +### Stopping a session + +```elixir +{:ok, result} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +## Notes + +- **Naming style:** Function names and parameters use snake_case, matching Elixir conventions. The SDK converts to camelCase for the API automatically. +- **OpenAPI-generated:** The Elixir SDK is auto-generated from the OpenAPI spec. Function names match the OpenAPI operation IDs directly (e.g. `scrape_and_extract_from_url` rather than just `scrape`). +- **Return values:** Regular functions return `{:ok, response}` or `{:error, %Firecrawl.Error{}}`. Bang variants (`!`) return the response directly or raise. +- **No deprecated aliases:** The Elixir SDK does not have deprecated method aliases. +- **Per-request options:** The trailing `opts` keyword list can include `api_key:` and `base_url:` to override the application config for a single request. +- **Origin:** The SDK automatically injects `"elixir-sdk@{version}"` as the origin in every request body. + +## Source Of Truth + +- `/firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `/firecrawl/apps/elixir-sdk/mix.exs` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 00000000..1bdb8360 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,264 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +og:title: "Java Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Java SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +**Gradle:** + +```kotlin +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +**Maven:** + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .build(); +``` + +Or from environment variables: + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +// Reads FIRECRAWL_API_KEY env var, then firecrawl.apiKey system property +``` + +Builder options: + +```java +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .apiUrl("https://api.firecrawl.dev") // default; override for self-hosted + .timeoutMs(300_000) // default: 5 minutes + .maxRetries(3) // default + .backoffFactor(0.5) // default + .build(); +``` + +Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images). + +### Preferred SDK method + +```java +client.search(query) +client.search(query, options) +``` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; + +SearchData results = client.search("firecrawl web scraping API", + SearchOptions.builder() + .limit(5) + .highlights(true) + .build() +); + +for (var result : results.getWeb()) { + System.out.println(result.get("title") + " " + result.get("url")); +} +``` + +Async variant: + +```java +CompletableFuture future = client.searchAsync("firecrawl", options); +``` + +### Parameters + +All fields on `SearchOptions` are nullable. Uses the builder pattern. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | The search query (required, first positional argument). | +| `sources` | `List` | Source types: `"web"`, `"news"`, `"images"` as strings or `{type: "web"}` maps. | +| `categories` | `List` | Filter by category: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Only include results from these domains. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Maximum number of results to return. | +| `tbs` | `String` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Geographic location for search results. | +| `ignoreInvalidURLs` | `Boolean` | Exclude invalid URLs from results. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping search result pages. | +| `integration` | `String` | Integration identifier. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```java +client.scrape(url) +client.scrape(url, options) +``` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .build() +); + +System.out.println(doc.getMarkdown()); +System.out.println(doc.getLinks()); +``` + +Async variant: + +```java +CompletableFuture future = client.scrapeAsync("https://example.com", options); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable. Uses the builder pattern. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | The URL to scrape (required, first positional argument). | +| `formats` | `List` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Also accepts format config objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). | +| `headers` | `Map` | Custom HTTP headers sent with the request. | +| `includeTags` | `List` | Only include content from these HTML tags. | +| `excludeTags` | `List` | Exclude content from these HTML tags. | +| `onlyMainContent` | `Boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `waitFor` | `Integer` | Wait time in milliseconds before scraping. | +| `mobile` | `Boolean` | Scrape as a mobile device. | +| `parsers` | `List` | File processing control (e.g. `"pdf"` or `Map.of("type", "pdf", "maxPages", 10)`). | +| `actions` | `List>` | Actions to execute before content extraction (click, write, press, scroll, wait, screenshot, scrape, executeJavascript, pdf). | +| `location` | `LocationConfig` | Geolocation config with `country` and `languages` fields. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64-encoded images from markdown output. | +| `blockAds` | `Boolean` | Block advertisements and cookie popups. | +| `proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `maxAge` | `Long` | Use cached result if younger than this many milliseconds. | +| `storeInCache` | `Boolean` | Whether to cache the result. | +| `lockdown` | `Boolean` | Only serve previously cached results. | +| `redactPII` | `Boolean` | Redact personally identifiable information. | +| `auditMetadata` | `AuditMetadata` | User attribution for SIEM logging. Constructor takes `username: String`. | +| `integration` | `String` | Integration identifier. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK method + +```java +client.interact(jobId, code) +client.interact(jobId, code, language, timeout) +client.interact(jobId, code, language, timeout, origin) +``` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); + +String jobId = (String) doc.getMetadata().get("jobId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "document.querySelector('button.load-more').click()", + "node", // language (default: "node") + 30 // timeout in seconds (default: 30) +); + +System.out.println(result.getStdout()); +``` + +Async variant: + +```java +CompletableFuture future = client.interactAsync(jobId, code); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | The scrape job ID from a prior `scrape` call (required). | +| `code` | `String` | Code to execute in the browser sandbox (required). | +| `language` | `String` | Language for code execution: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1-300). Defaults to `30`. | +| `origin` | `String` | Origin label for telemetry. Auto-set to `"java-sdk@{version}"`. | + +### Stopping a session + +```java +BrowserDeleteResponse deleteResult = client.stopInteractiveBrowser(jobId); +System.out.println(deleteResult.getCreditsBilled()); +``` + +## Notes + +- **Naming style:** All parameters use camelCase, matching Java conventions. +- **Builder pattern:** `ScrapeOptions` and `SearchOptions` use the builder pattern: `ScrapeOptions.builder().formats(...).build()`. +- **Async:** Every method has an async variant returning `CompletableFuture` (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`). +- **Deprecated aliases:** The following methods are marked `@Deprecated`: + - `scrapeExecute()` → use `interact()` + - `deleteScrapeBrowser()` → use `stopInteractiveBrowser()` + - `QueryFormat` class → use `QuestionFormat` or `HighlightsFormat` +- **Format objects:** Use `JsonFormat`, `QuestionFormat`, or `HighlightsFormat` classes for structured format options in `ScrapeOptions.formats`. +- **Search response:** `SearchData` returns `List>` for `.getWeb()`, `.getNews()`, `.getImages()`. + +## Source Of Truth + +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `/firecrawl/apps/java-sdk/build.gradle.kts` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 00000000..471d46fe --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,219 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +og:title: "Node.js Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Node.js/TypeScript SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +## Authenticate + +```typescript +import Firecrawl from "@mendable/firecrawl-js"; + +const firecrawl = new Firecrawl("fc-YOUR_API_KEY"); +``` + +Or pass an options object: + +```typescript +const firecrawl = new Firecrawl({ + apiKey: "fc-YOUR_API_KEY", + apiUrl: "https://api.firecrawl.dev", // default; override for self-hosted +}); +``` + +The SDK reads `FIRECRAWL_API_KEY` and `FIRECRAWL_API_URL` from environment variables as fallbacks. + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code, clicking elements, filling forms, or executing prompts in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images, developer). + +### Preferred SDK method + +```typescript +firecrawl.search(query, options?) +``` + +### Example + +```typescript +const results = await firecrawl.search("firecrawl web scraping API", { + limit: 5, + scrapeOptions: { + formats: ["markdown"], + }, +}); + +for (const result of results.web) { + console.log(result.title, result.url); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | The search query (required, first positional argument). | +| `limit` | `number` | Maximum number of results to return. | +| `sources` | `Array<"web" \| "news" \| "images">` | Which source types to search. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer">` | Filter results by category. | +| `includeDomains` | `string[]` | Only include results from these domains. Cannot be used with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. Cannot be used with `includeDomains`. | +| `tbs` | `string` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `string` | Geographic location for search results. | +| `ignoreInvalidURLs` | `boolean` | Exclude invalid URLs from results. | +| `timeout` | `number` | Timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping search result pages. Accepts the same parameters as `scrape`. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise options for zero data retention. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```typescript +firecrawl.scrape(url, options?) +``` + +### Example + +```typescript +const doc = await firecrawl.scrape("https://example.com", { + formats: ["markdown", "links"], +}); + +console.log(doc.markdown); +console.log(doc.links); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape (required, first positional argument). | +| `formats` | `FormatOption[]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config objects for `json`, `question`, `highlights`. | +| `headers` | `Record` | Custom HTTP headers sent with the request. | +| `includeTags` | `string[]` | Only include content from these HTML tags. | +| `excludeTags` | `string[]` | Exclude content from these HTML tags. | +| `onlyMainContent` | `boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `number` | Timeout in milliseconds. | +| `waitFor` | `number` | Wait time in milliseconds before scraping (for JS-rendered pages). | +| `mobile` | `boolean` | Scrape as a mobile device. | +| `parsers` | `Array` | File processing control (e.g. `"pdf"` or `{ type: "pdf", mode: "ocr", maxPages: 10 }`). | +| `actions` | `ActionOption[]` | Actions to execute before content extraction (click, write, press, scroll, wait, screenshot, scrape, executeJavascript, pdf). | +| `location` | `{ country?: string, languages?: string[] }` | Geolocation configuration. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64-encoded images from markdown output. | +| `fastMode` | `boolean` | Enable fast mode for quicker scraping. | +| `blockAds` | `boolean` | Block advertisements and cookie popups. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode. | +| `maxAge` | `number` | Use cached result if younger than this many milliseconds. Set to `0` to bypass cache. | +| `minAge` | `number` | Minimum cache age in milliseconds. | +| `storeInCache` | `boolean` | Whether to cache the result. | +| `lockdown` | `boolean` | Only serve previously cached results; never make an outbound request. | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | +| `profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser storage across sessions. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Interact + +### Why use it + +Interact lets you execute code or natural-language prompts in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content that requires browser interaction. + +### Preferred SDK method + +```typescript +firecrawl.interact(jobId, args) +``` + +### Example + +```typescript +const doc = await firecrawl.scrape("https://example.com", { + formats: ["markdown"], +}); + +const jobId = doc.metadata?.jobId; + +const result = await firecrawl.interact(jobId, { + code: "document.querySelector('button.load-more').click()", + language: "node", + timeout: 30, +}); + +console.log(result.output); +``` + +Or use a natural-language prompt: + +```typescript +const result = await firecrawl.interact(jobId, { + prompt: "Click the 'Load More' button and wait for new content", + timeout: 30, +}); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `string` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `string` | Natural-language instruction for the browser. Either `code` or `prompt` must be provided. | +| `language` | `"python" \| "node" \| "bash"` | Language for code execution. Defaults to `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. | +| `origin` | `string` | Origin label for telemetry. | + +### Stopping a session + +```typescript +const deleteResult = await firecrawl.stopInteraction(jobId); +console.log(deleteResult.creditsBilled); +``` + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Deprecated aliases:** The following methods still work but should not be used in new code: + - `scrapeExecute()` → use `interact()` + - `stopInteractiveBrowser()` / `deleteScrapeBrowser()` → use `stopInteraction()` + - `scrapeUrl()` → use `scrape()` +- **Search response shape:** Results are accessed via `.web`, `.news`, `.images`, `.developer` properties. Accessing `.data` throws a helpful error directing you to these properties. +- **Async client:** The same `Firecrawl` class is async-native (all methods return Promises). + +## Source Of Truth + +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 00000000..f18d869c --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,226 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +og:title: "Python Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Python SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY") +``` + +Or with additional options: + +```python +firecrawl = Firecrawl( + api_key="fc-YOUR_API_KEY", + api_url="https://api.firecrawl.dev", # default; override for self-hosted + timeout=None, # request timeout in seconds + max_retries=3, # default + backoff_factor=0.5, # default +) +``` + +The SDK reads `FIRECRAWL_API_KEY` from environment variables as a fallback. Omitting the key enables the keyless free tier (rate-limited per IP). + +An async client is also available: + +```python +from firecrawl import AsyncFirecrawl + +firecrawl = AsyncFirecrawl(api_key="fc-YOUR_API_KEY") +``` + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code, clicking elements, filling forms, or executing prompts in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images, developer). + +### Preferred SDK method + +```python +firecrawl.search(query, **kwargs) +``` + +### Example + +```python +results = firecrawl.search( + "firecrawl web scraping API", + limit=5, + scrape_options=ScrapeOptions(formats=["markdown"]), +) + +for result in results.web: + print(result.title, result.url) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | The search query (required, first positional argument). | +| `limit` | `int` | Maximum number of results to return. Default: `5` in SDK. | +| `sources` | `list[SourceOption]` | Which source types to search (`"web"`, `"news"`, `"images"`). | +| `categories` | `list[CategoryOption]` | Filter results by category (`"github"`, `"research"`, `"pdf"`, `"developer"`). | +| `include_domains` | `list[str]` | Only include results from these domains. Cannot be used with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Cannot be used with `include_domains`. | +| `tbs` | `str` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `str` | Geographic location for search results. | +| `ignore_invalid_urls` | `bool` | Exclude invalid URLs from results. | +| `timeout` | `int` | Timeout in milliseconds. Default: `300000` in SDK. | +| `highlights` | `bool` | Generate query-relevant highlights. Defaults to `True`. | +| `scrape_options` | `ScrapeOptions` | Options applied when scraping search result pages. Accepts the same parameters as `scrape`. | +| `enterprise` | `list[str]` | Enterprise options for zero data retention. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `integration` | `str` | Integration identifier. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```python +firecrawl.scrape(url, **kwargs) +``` + +### Example + +```python +doc = firecrawl.scrape( + "https://example.com", + formats=["markdown", "links"], +) + +print(doc.markdown) +print(doc.links) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape (required, first positional argument). | +| `formats` | `list[FormatOption]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"raw_html"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"change_tracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config objects. | +| `headers` | `dict[str, str]` | Custom HTTP headers sent with the request. | +| `include_tags` | `list[str]` | Only include content from these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude content from these HTML tags. | +| `only_main_content` | `bool` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait time in milliseconds before scraping (for JS-rendered pages). | +| `mobile` | `bool` | Scrape as a mobile device. | +| `parsers` | `list[str \| PDFParser]` | File processing control (e.g. `"pdf"` or `PDFParser(mode="ocr", max_pages=10)`). | +| `actions` | `list[Action]` | Actions to execute before content extraction (WaitAction, ClickAction, WriteAction, PressAction, ScrollAction, ScreenshotAction, ScrapeAction, ExecuteJavascriptAction, PDFAction). | +| `location` | `Location` | Geolocation configuration with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64-encoded images from markdown output. | +| `fast_mode` | `bool` | Enable fast mode for quicker scraping. | +| `block_ads` | `bool` | Block advertisements and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, or `"auto"`. | +| `max_age` | `int` | Use cached result if younger than this many milliseconds. | +| `store_in_cache` | `bool` | Whether to cache the result. | +| `lockdown` | `bool` | Only serve previously cached results; never make an outbound request. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. | +| `profile` | `dict` | Persistent browser storage across sessions. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Interact lets you execute code or natural-language prompts in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content that requires browser interaction. + +### Preferred SDK method + +```python +firecrawl.interact(job_id, code=None, *, prompt=None, language="node", timeout=None) +``` + +### Example + +```python +doc = firecrawl.scrape("https://example.com", formats=["markdown"]) + +job_id = doc.metadata.job_id + +result = firecrawl.interact( + job_id, + code="document.querySelector('button.load-more').click()", + language="node", + timeout=30, +) + +print(result.output) +``` + +Or use a natural-language prompt: + +```python +result = firecrawl.interact( + job_id, + prompt="Click the 'Load More' button and wait for new content", + timeout=30, +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `str` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `str` | Natural-language instruction for the browser (keyword-only). Either `code` or `prompt` must be provided. | +| `language` | `Literal["python", "node", "bash"]` | Language for code execution. Defaults to `"node"`. | +| `timeout` | `int` | Execution timeout in seconds (1-300). | +| `origin` | `str` | Origin label for telemetry. | + +### Stopping a session + +```python +delete_result = firecrawl.stop_interaction(job_id) +print(delete_result.credits_billed) +``` + +## Notes + +- **Naming style:** All parameters use snake_case. The SDK handles conversion to camelCase for the API. +- **Format strings:** Both snake_case (`"raw_html"`, `"change_tracking"`) and camelCase (`"rawHtml"`, `"changeTracking"`) format strings are accepted. +- **Deprecated aliases:** The following methods still work but should not be used in new code: + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + - `scrape_url()` → use `scrape()` +- **Search response shape:** Results are accessed via `.web`, `.news`, `.images`, `.developer` properties. Accessing `.data` raises an `AttributeError` with a helpful message directing you to these properties. +- **Async client:** Use `AsyncFirecrawl` (aliased as `AsyncFirecrawlApp`) for async/await usage. Same method signatures. +- **Class aliases:** `FirecrawlApp` is an alias for `Firecrawl`. + +## Source Of Truth + +- `/firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `/firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `/firecrawl/apps/python-sdk/firecrawl/client.py` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 00000000..30a94339 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,256 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +og:title: "Rust Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Rust SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR_API_KEY")?; +``` + +For self-hosted instances: + +```rust +let client = Client::new_selfhosted( + "https://your-firecrawl-instance.com", + Some("fc-YOUR_API_KEY"), +)?; +``` + +Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images). + +### Preferred SDK method + +```rust +client.search(query, options).await +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let response = client.search("firecrawl web scraping API", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +A convenience method is also available: + +```rust +let docs = client.search_and_scrape("firecrawl web scraping", 5).await?; +``` + +### Parameters + +All fields on `SearchOptions` are `Option` types. The struct derives `Default`. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | The search query (required, first positional argument). | +| `limit` | `Option` | Maximum number of results. Default: 5, Max: 20. | +| `sources` | `Option>` | Source types: `SearchSource::Web`, `News`, `Images`. | +| `categories` | `Option>` | Filter by category: `SearchCategory::Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Only include results from these domains. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `Option` | Geographic location for search results. | +| `ignore_invalid_urls` | `Option` | Exclude invalid URLs from results. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `highlights` | `Option` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `Option` | Options applied when scraping search result pages. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```rust +client.scrape(url, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + ..Default::default() +}).await?; + +if let Some(markdown) = &doc.markdown { + println!("{}", markdown); +} +``` + +For structured JSON extraction with a schema: + +```rust +use serde_json::json; + +let result = client.scrape_with_schema( + "https://example.com", + json!({"type": "object", "properties": {"title": {"type": "string"}}}), + Some("Extract the page title"), +).await?; +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` types. The struct derives `Default`. Fields serialize to camelCase for the API. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | The URL to scrape (required, first positional argument). | +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also object variants: `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `headers` | `Option>` | Custom HTTP headers sent with the request. | +| `include_tags` | `Option>` | Only include content from these HTML tags. | +| `exclude_tags` | `Option>` | Exclude content from these HTML tags. | +| `only_main_content` | `Option` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `wait_for` | `Option` | Wait time in milliseconds before scraping. | +| `mobile` | `Option` | Scrape as a mobile device. | +| `parsers` | `Option>` | File processing control. `ParserConfig::Simple("pdf")` or `ParserConfig::Pdf { mode, max_pages }`. | +| `actions` | `Option>` | Actions to execute before content extraction: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `location` | `Option` | Geolocation with `country` and `languages` fields. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64-encoded images from markdown output. | +| `fast_mode` | `Option` | Enable fast mode for quicker scraping. | +| `block_ads` | `Option` | Block advertisements and cookie popups. | +| `proxy` | `Option` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `max_age` | `Option` | Use cached result if younger than this many seconds. | +| `min_age` | `Option` | Minimum cache age in seconds. | +| `store_in_cache` | `Option` | Whether to cache the result. | +| `lockdown` | `Option` | Only serve previously cached results. | +| `redact_pii` | `Option` | Redact personally identifiable information. | +| `audit_metadata` | `Option` | User attribution for SIEM logging. Field: `username: String`. | +| `profile` | `Option` | Persistent browser storage. Fields: `name: String`, `save_changes: Option`. | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction options: `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config: `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking config: `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute selectors: `selector`, `attribute`. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK method + +```rust +client.interact(job_id, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.job_id.as_ref()) + .expect("job_id required for interact"); + +let result = client.interact(job_id, ScrapeExecuteOptions { + code: Some("document.querySelector('button.load-more').click()".into()), + language: None, // defaults to Node + timeout: Some(30), + ..Default::default() +}).await?; + +println!("{:?}", result.stdout); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `Option` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `Option` | Natural-language instruction for the browser. Either `code` or `prompt` must be provided. | +| `language` | `Option` | Language for code execution: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +### Stopping a session + +```rust +let delete_result = client.stop_interaction(job_id).await?; +println!("Credits billed: {:?}", delete_result.credits_billed); +``` + +## Notes + +- **Naming style:** Struct fields use snake_case in Rust. They serialize to camelCase for the API via serde. +- **Async:** All methods are async and return `Result`. +- **Deprecated aliases:** The following methods still work but are marked `#[deprecated]`: + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` +- **`max_age` / `min_age` units:** The Rust SDK uses seconds for these fields, unlike JS/Python which use milliseconds. +- **Search response:** `SearchData.web` contains `Vec`, a custom enum that distinguishes between web result metadata and full scraped documents. + +## Source Of Truth + +- `/firecrawl/apps/rust-sdk/src/v2/client.rs` +- `/firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `/firecrawl/apps/rust-sdk/src/v2/search.rs` +- `/firecrawl/apps/rust-sdk/Cargo.toml` +- `/firecrawl-docs/api-reference/v2-openapi.json`