diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 00000000..6e6ca43d --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,234 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +og:title: "Elixir Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Elixir SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +Add to your `mix.exs`: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.9"} + ] +end +``` + +Then run: + +```bash +mix deps.get +``` + +## Authenticate + +Configure via application config: + +```elixir +# config/config.exs +config :firecrawl, api_key: "fc-YOUR_API_KEY" +``` + +Or pass the API key per-request via the `opts` keyword: + +```elixir +Firecrawl.search_and_scrape([query: "example"], api_key: "fc-YOUR_API_KEY") +``` + +For self-hosted instances, set the base URL: + +```elixir +config :firecrawl, + api_key: "fc-YOUR_API_KEY", + base_url: "https://your-firecrawl-instance.com/v2" +``` + +The default base URL is `"https://api.firecrawl.dev/v2"`. Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search_and_scrape`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape_and_extract_from_url`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact_with_scrape_browser_session`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type. + +### Preferred SDK function + +```elixir +Firecrawl.search_and_scrape(params, opts \\ []) +Firecrawl.search_and_scrape!(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, results} = Firecrawl.search_and_scrape( + query: "firecrawl web scraping API", + limit: 5, + highlights: true +) + +IO.inspect(results) +``` + +The bang variant raises `Firecrawl.Error` on failure: + +```elixir +results = Firecrawl.search_and_scrape!( + query: "firecrawl web scraping API", + limit: 5 +) +``` + +### Parameters + +All parameters are passed as a keyword list. Only `query` is required. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `:string` | The search query (required). | +| `limit` | `:integer` | Maximum number of results to return. | +| `sources` | `{:list, :any}` | Source types to search. Defaults to `["web"]`. | +| `categories` | `{:list, :any}` | Filter results by category. | +| `include_domains` | `{:list, :string}` | Only include results from these domains. | +| `exclude_domains` | `{:list, :string}` | Exclude results from these domains. | +| `tbs` | `:string` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `:string` | Geographic location for search results (e.g. `"San Francisco,California,United States"`). | +| `country` | `:string` | ISO country code for geo-targeting (e.g. `"US"`). | +| `ignore_invalid_urls` | `:boolean` | Exclude invalid URLs from results. | +| `timeout` | `:integer` | Timeout in milliseconds. | +| `highlights` | `:boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `:keyword_list` | Options for scraping search results. Accepts the same parameters as scrape. | +| `enterprise` | `{:list, :string}` | Enterprise ZDR options: `["zdr"]` or `["anon"]`. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK function + +```elixir +Firecrawl.scrape_and_extract_from_url(params, opts \\ []) +Firecrawl.scrape_and_extract_from_url!(params, opts \\ []) +``` + +### Example + +```elixir +{:ok, doc} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"] +) + +IO.puts(doc["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. Only `url` is required. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `:string` | The URL to scrape (required). | +| `formats` | `{:list, :any}` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"audio"`, `"video"`, etc. | +| `headers` | `:any` | Custom HTTP headers (cookies, user-agent, etc.). | +| `include_tags` | `{:list, :string}` | Only include content from these HTML tags. | +| `exclude_tags` | `{:list, :string}` | Exclude content from these HTML tags. | +| `only_main_content` | `:boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `:integer` | Timeout in milliseconds. Default: `60000`, Min: `1000`, Max: `300000`. | +| `wait_for` | `:integer` | Wait time in milliseconds before scraping. | +| `mobile` | `:boolean` | Scrape as a mobile device. | +| `parsers` | `{:list, :any}` | File processing control (e.g. PDF settings). | +| `actions` | `{:list, :any}` | Actions to execute before content extraction. | +| `location` | `:keyword_list` | Geolocation settings. Defaults to US. | +| `skip_tls_verification` | `:boolean` | Skip TLS certificate verification. | +| `remove_base64_images` | `:boolean` | Remove base64-encoded images from markdown output. | +| `block_ads` | `:boolean` | Block advertisements and cookie popups. | +| `proxy` | `{:in, [:basic, :enhanced, :auto]}` | Proxy type. | +| `max_age` | `:integer` | Use cached result if younger than this many milliseconds. Defaults to 2 days. | +| `min_age` | `:integer` | Cache-only mode; minimum age in milliseconds. | +| `store_in_cache` | `:boolean` | Whether to cache the result. | +| `lockdown` | `:boolean` | Only serve previously cached results. | +| `redact_pii` | `:boolean` | Redact personally identifiable information. | +| `audit_metadata` | `:keyword_list` | User attribution for SIEM logging. Keys: `username` (required). | +| `profile` | `:keyword_list` | Persistent browser storage across sessions. | +| `zero_data_retention` | `:boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior scrape call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK function + +```elixir +Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ []) +Firecrawl.interact_with_scrape_browser_session!(job_id, params, opts \\ []) +``` + +### Example + +```elixir +{:ok, doc} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = doc["data"]["metadata"]["jobId"] + +{:ok, result} = Firecrawl.interact_with_scrape_browser_session(job_id, + code: "document.querySelector('button.load-more').click()", + language: :node, + timeout: 30 +) + +IO.inspect(result) +``` + +### Parameters + +The `job_id` is passed as the first positional argument. Remaining parameters are a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t()` | The scrape job ID from a prior scrape call (required, first argument). | +| `code` | `:string` | Code to execute in the browser sandbox (required). | +| `language` | `{:in, [:python, :node, :bash]}` | Language for code execution. | +| `timeout` | `:integer` | Execution timeout in seconds. | +| `origin` | `:string` | Origin label for telemetry. | + +### Stopping a session + +```elixir +{:ok, result} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +## Notes + +- **Naming style:** Function names and parameters use snake_case, matching Elixir conventions. The SDK converts to camelCase for the API automatically. +- **OpenAPI-generated:** The Elixir SDK is auto-generated from the OpenAPI spec. Function names match the OpenAPI operation IDs directly (e.g. `scrape_and_extract_from_url` rather than just `scrape`). +- **Return values:** Regular functions return `{:ok, response}` or `{:error, %Firecrawl.Error{}}`. Bang variants (`!`) return the response directly or raise. +- **No deprecated aliases:** The Elixir SDK does not have deprecated method aliases. +- **Per-request options:** The trailing `opts` keyword list can include `api_key:` and `base_url:` to override the application config for a single request. +- **Origin:** The SDK automatically injects `"elixir-sdk@{version}"` as the origin in every request body. + +## Source Of Truth + +- `/firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `/firecrawl/apps/elixir-sdk/mix.exs` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 00000000..1bdb8360 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,264 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +og:title: "Java Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Java SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +**Gradle:** + +```kotlin +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +**Maven:** + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .build(); +``` + +Or from environment variables: + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +// Reads FIRECRAWL_API_KEY env var, then firecrawl.apiKey system property +``` + +Builder options: + +```java +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR_API_KEY") + .apiUrl("https://api.firecrawl.dev") // default; override for self-hosted + .timeoutMs(300_000) // default: 5 minutes + .maxRetries(3) // default + .backoffFactor(0.5) // default + .build(); +``` + +Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images). + +### Preferred SDK method + +```java +client.search(query) +client.search(query, options) +``` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.SearchData; + +SearchData results = client.search("firecrawl web scraping API", + SearchOptions.builder() + .limit(5) + .highlights(true) + .build() +); + +for (var result : results.getWeb()) { + System.out.println(result.get("title") + " " + result.get("url")); +} +``` + +Async variant: + +```java +CompletableFuture future = client.searchAsync("firecrawl", options); +``` + +### Parameters + +All fields on `SearchOptions` are nullable. Uses the builder pattern. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | The search query (required, first positional argument). | +| `sources` | `List` | Source types: `"web"`, `"news"`, `"images"` as strings or `{type: "web"}` maps. | +| `categories` | `List` | Filter by category: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Only include results from these domains. | +| `excludeDomains` | `List` | Exclude results from these domains. | +| `limit` | `Integer` | Maximum number of results to return. | +| `tbs` | `String` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `String` | Geographic location for search results. | +| `ignoreInvalidURLs` | `Boolean` | Exclude invalid URLs from results. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `highlights` | `Boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping search result pages. | +| `integration` | `String` | Integration identifier. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```java +client.scrape(url) +client.scrape(url, options) +``` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .build() +); + +System.out.println(doc.getMarkdown()); +System.out.println(doc.getLinks()); +``` + +Async variant: + +```java +CompletableFuture future = client.scrapeAsync("https://example.com", options); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable. Uses the builder pattern. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | The URL to scrape (required, first positional argument). | +| `formats` | `List` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Also accepts format config objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). | +| `headers` | `Map` | Custom HTTP headers sent with the request. | +| `includeTags` | `List` | Only include content from these HTML tags. | +| `excludeTags` | `List` | Exclude content from these HTML tags. | +| `onlyMainContent` | `Boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `Integer` | Timeout in milliseconds. | +| `waitFor` | `Integer` | Wait time in milliseconds before scraping. | +| `mobile` | `Boolean` | Scrape as a mobile device. | +| `parsers` | `List` | File processing control (e.g. `"pdf"` or `Map.of("type", "pdf", "maxPages", 10)`). | +| `actions` | `List>` | Actions to execute before content extraction (click, write, press, scroll, wait, screenshot, scrape, executeJavascript, pdf). | +| `location` | `LocationConfig` | Geolocation config with `country` and `languages` fields. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64-encoded images from markdown output. | +| `blockAds` | `Boolean` | Block advertisements and cookie popups. | +| `proxy` | `String` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `maxAge` | `Long` | Use cached result if younger than this many milliseconds. | +| `storeInCache` | `Boolean` | Whether to cache the result. | +| `lockdown` | `Boolean` | Only serve previously cached results. | +| `redactPII` | `Boolean` | Redact personally identifiable information. | +| `auditMetadata` | `AuditMetadata` | User attribution for SIEM logging. Constructor takes `username: String`. | +| `integration` | `String` | Integration identifier. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK method + +```java +client.interact(jobId, code) +client.interact(jobId, code, language, timeout) +client.interact(jobId, code, language, timeout, origin) +``` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); + +String jobId = (String) doc.getMetadata().get("jobId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "document.querySelector('button.load-more').click()", + "node", // language (default: "node") + 30 // timeout in seconds (default: 30) +); + +System.out.println(result.getStdout()); +``` + +Async variant: + +```java +CompletableFuture future = client.interactAsync(jobId, code); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | The scrape job ID from a prior `scrape` call (required). | +| `code` | `String` | Code to execute in the browser sandbox (required). | +| `language` | `String` | Language for code execution: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1-300). Defaults to `30`. | +| `origin` | `String` | Origin label for telemetry. Auto-set to `"java-sdk@{version}"`. | + +### Stopping a session + +```java +BrowserDeleteResponse deleteResult = client.stopInteractiveBrowser(jobId); +System.out.println(deleteResult.getCreditsBilled()); +``` + +## Notes + +- **Naming style:** All parameters use camelCase, matching Java conventions. +- **Builder pattern:** `ScrapeOptions` and `SearchOptions` use the builder pattern: `ScrapeOptions.builder().formats(...).build()`. +- **Async:** Every method has an async variant returning `CompletableFuture` (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`). +- **Deprecated aliases:** The following methods are marked `@Deprecated`: + - `scrapeExecute()` → use `interact()` + - `deleteScrapeBrowser()` → use `stopInteractiveBrowser()` + - `QueryFormat` class → use `QuestionFormat` or `HighlightsFormat` +- **Format objects:** Use `JsonFormat`, `QuestionFormat`, or `HighlightsFormat` classes for structured format options in `ScrapeOptions.formats`. +- **Search response:** `SearchData` returns `List>` for `.getWeb()`, `.getNews()`, `.getImages()`. + +## Source Of Truth + +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `/firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `/firecrawl/apps/java-sdk/build.gradle.kts` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 00000000..471d46fe --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,219 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +og:title: "Node.js Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Node.js/TypeScript SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +## Authenticate + +```typescript +import Firecrawl from "@mendable/firecrawl-js"; + +const firecrawl = new Firecrawl("fc-YOUR_API_KEY"); +``` + +Or pass an options object: + +```typescript +const firecrawl = new Firecrawl({ + apiKey: "fc-YOUR_API_KEY", + apiUrl: "https://api.firecrawl.dev", // default; override for self-hosted +}); +``` + +The SDK reads `FIRECRAWL_API_KEY` and `FIRECRAWL_API_URL` from environment variables as fallbacks. + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code, clicking elements, filling forms, or executing prompts in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images, developer). + +### Preferred SDK method + +```typescript +firecrawl.search(query, options?) +``` + +### Example + +```typescript +const results = await firecrawl.search("firecrawl web scraping API", { + limit: 5, + scrapeOptions: { + formats: ["markdown"], + }, +}); + +for (const result of results.web) { + console.log(result.title, result.url); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | The search query (required, first positional argument). | +| `limit` | `number` | Maximum number of results to return. | +| `sources` | `Array<"web" \| "news" \| "images">` | Which source types to search. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer">` | Filter results by category. | +| `includeDomains` | `string[]` | Only include results from these domains. Cannot be used with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude results from these domains. Cannot be used with `includeDomains`. | +| `tbs` | `string` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `string` | Geographic location for search results. | +| `ignoreInvalidURLs` | `boolean` | Exclude invalid URLs from results. | +| `timeout` | `number` | Timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Defaults to `true`. | +| `scrapeOptions` | `ScrapeOptions` | Options applied when scraping search result pages. Accepts the same parameters as `scrape`. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise options for zero data retention. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```typescript +firecrawl.scrape(url, options?) +``` + +### Example + +```typescript +const doc = await firecrawl.scrape("https://example.com", { + formats: ["markdown", "links"], +}); + +console.log(doc.markdown); +console.log(doc.links); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape (required, first positional argument). | +| `formats` | `FormatOption[]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config objects for `json`, `question`, `highlights`. | +| `headers` | `Record` | Custom HTTP headers sent with the request. | +| `includeTags` | `string[]` | Only include content from these HTML tags. | +| `excludeTags` | `string[]` | Exclude content from these HTML tags. | +| `onlyMainContent` | `boolean` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `number` | Timeout in milliseconds. | +| `waitFor` | `number` | Wait time in milliseconds before scraping (for JS-rendered pages). | +| `mobile` | `boolean` | Scrape as a mobile device. | +| `parsers` | `Array` | File processing control (e.g. `"pdf"` or `{ type: "pdf", mode: "ocr", maxPages: 10 }`). | +| `actions` | `ActionOption[]` | Actions to execute before content extraction (click, write, press, scroll, wait, screenshot, scrape, executeJavascript, pdf). | +| `location` | `{ country?: string, languages?: string[] }` | Geolocation configuration. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64-encoded images from markdown output. | +| `fastMode` | `boolean` | Enable fast mode for quicker scraping. | +| `blockAds` | `boolean` | Block advertisements and cookie popups. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode. | +| `maxAge` | `number` | Use cached result if younger than this many milliseconds. Set to `0` to bypass cache. | +| `minAge` | `number` | Minimum cache age in milliseconds. | +| `storeInCache` | `boolean` | Whether to cache the result. | +| `lockdown` | `boolean` | Only serve previously cached results; never make an outbound request. | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `auditMetadata` | `{ username: string }` | User attribution for SIEM logging. | +| `profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser storage across sessions. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +## Interact + +### Why use it + +Interact lets you execute code or natural-language prompts in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content that requires browser interaction. + +### Preferred SDK method + +```typescript +firecrawl.interact(jobId, args) +``` + +### Example + +```typescript +const doc = await firecrawl.scrape("https://example.com", { + formats: ["markdown"], +}); + +const jobId = doc.metadata?.jobId; + +const result = await firecrawl.interact(jobId, { + code: "document.querySelector('button.load-more').click()", + language: "node", + timeout: 30, +}); + +console.log(result.output); +``` + +Or use a natural-language prompt: + +```typescript +const result = await firecrawl.interact(jobId, { + prompt: "Click the 'Load More' button and wait for new content", + timeout: 30, +}); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `string` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `string` | Natural-language instruction for the browser. Either `code` or `prompt` must be provided. | +| `language` | `"python" \| "node" \| "bash"` | Language for code execution. Defaults to `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. | +| `origin` | `string` | Origin label for telemetry. | + +### Stopping a session + +```typescript +const deleteResult = await firecrawl.stopInteraction(jobId); +console.log(deleteResult.creditsBilled); +``` + +## Notes + +- **Naming style:** All parameters use camelCase. +- **Deprecated aliases:** The following methods still work but should not be used in new code: + - `scrapeExecute()` → use `interact()` + - `stopInteractiveBrowser()` / `deleteScrapeBrowser()` → use `stopInteraction()` + - `scrapeUrl()` → use `scrape()` +- **Search response shape:** Results are accessed via `.web`, `.news`, `.images`, `.developer` properties. Accessing `.data` throws a helpful error directing you to these properties. +- **Async client:** The same `Firecrawl` class is async-native (all methods return Promises). + +## Source Of Truth + +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `/firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 00000000..f18d869c --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,226 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +og:title: "Python Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Python SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +from firecrawl import Firecrawl + +firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY") +``` + +Or with additional options: + +```python +firecrawl = Firecrawl( + api_key="fc-YOUR_API_KEY", + api_url="https://api.firecrawl.dev", # default; override for self-hosted + timeout=None, # request timeout in seconds + max_retries=3, # default + backoff_factor=0.5, # default +) +``` + +The SDK reads `FIRECRAWL_API_KEY` from environment variables as a fallback. Omitting the key enables the keyless free tier (rate-limited per IP). + +An async client is also available: + +```python +from firecrawl import AsyncFirecrawl + +firecrawl = AsyncFirecrawl(api_key="fc-YOUR_API_KEY") +``` + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code, clicking elements, filling forms, or executing prompts in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images, developer). + +### Preferred SDK method + +```python +firecrawl.search(query, **kwargs) +``` + +### Example + +```python +results = firecrawl.search( + "firecrawl web scraping API", + limit=5, + scrape_options=ScrapeOptions(formats=["markdown"]), +) + +for result in results.web: + print(result.title, result.url) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | The search query (required, first positional argument). | +| `limit` | `int` | Maximum number of results to return. Default: `5` in SDK. | +| `sources` | `list[SourceOption]` | Which source types to search (`"web"`, `"news"`, `"images"`). | +| `categories` | `list[CategoryOption]` | Filter results by category (`"github"`, `"research"`, `"pdf"`, `"developer"`). | +| `include_domains` | `list[str]` | Only include results from these domains. Cannot be used with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude results from these domains. Cannot be used with `include_domains`. | +| `tbs` | `str` | Time-based search filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `str` | Geographic location for search results. | +| `ignore_invalid_urls` | `bool` | Exclude invalid URLs from results. | +| `timeout` | `int` | Timeout in milliseconds. Default: `300000` in SDK. | +| `highlights` | `bool` | Generate query-relevant highlights. Defaults to `True`. | +| `scrape_options` | `ScrapeOptions` | Options applied when scraping search result pages. Accepts the same parameters as `scrape`. | +| `enterprise` | `list[str]` | Enterprise options for zero data retention. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `integration` | `str` | Integration identifier. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```python +firecrawl.scrape(url, **kwargs) +``` + +### Example + +```python +doc = firecrawl.scrape( + "https://example.com", + formats=["markdown", "links"], +) + +print(doc.markdown) +print(doc.links) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape (required, first positional argument). | +| `formats` | `list[FormatOption]` | Output formats. String values: `"markdown"`, `"html"`, `"rawHtml"`, `"raw_html"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"change_tracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config objects. | +| `headers` | `dict[str, str]` | Custom HTTP headers sent with the request. | +| `include_tags` | `list[str]` | Only include content from these HTML tags. | +| `exclude_tags` | `list[str]` | Exclude content from these HTML tags. | +| `only_main_content` | `bool` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait time in milliseconds before scraping (for JS-rendered pages). | +| `mobile` | `bool` | Scrape as a mobile device. | +| `parsers` | `list[str \| PDFParser]` | File processing control (e.g. `"pdf"` or `PDFParser(mode="ocr", max_pages=10)`). | +| `actions` | `list[Action]` | Actions to execute before content extraction (WaitAction, ClickAction, WriteAction, PressAction, ScrollAction, ScreenshotAction, ScrapeAction, ExecuteJavascriptAction, PDFAction). | +| `location` | `Location` | Geolocation configuration with `country` and `languages` fields. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64-encoded images from markdown output. | +| `fast_mode` | `bool` | Enable fast mode for quicker scraping. | +| `block_ads` | `bool` | Block advertisements and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, or `"auto"`. | +| `max_age` | `int` | Use cached result if younger than this many milliseconds. | +| `store_in_cache` | `bool` | Whether to cache the result. | +| `lockdown` | `bool` | Only serve previously cached results; never make an outbound request. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection configuration. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. | +| `profile` | `dict` | Persistent browser storage across sessions. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Interact lets you execute code or natural-language prompts in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content that requires browser interaction. + +### Preferred SDK method + +```python +firecrawl.interact(job_id, code=None, *, prompt=None, language="node", timeout=None) +``` + +### Example + +```python +doc = firecrawl.scrape("https://example.com", formats=["markdown"]) + +job_id = doc.metadata.job_id + +result = firecrawl.interact( + job_id, + code="document.querySelector('button.load-more').click()", + language="node", + timeout=30, +) + +print(result.output) +``` + +Or use a natural-language prompt: + +```python +result = firecrawl.interact( + job_id, + prompt="Click the 'Load More' button and wait for new content", + timeout=30, +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `str` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `str` | Natural-language instruction for the browser (keyword-only). Either `code` or `prompt` must be provided. | +| `language` | `Literal["python", "node", "bash"]` | Language for code execution. Defaults to `"node"`. | +| `timeout` | `int` | Execution timeout in seconds (1-300). | +| `origin` | `str` | Origin label for telemetry. | + +### Stopping a session + +```python +delete_result = firecrawl.stop_interaction(job_id) +print(delete_result.credits_billed) +``` + +## Notes + +- **Naming style:** All parameters use snake_case. The SDK handles conversion to camelCase for the API. +- **Format strings:** Both snake_case (`"raw_html"`, `"change_tracking"`) and camelCase (`"rawHtml"`, `"changeTracking"`) format strings are accepted. +- **Deprecated aliases:** The following methods still work but should not be used in new code: + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + - `scrape_url()` → use `scrape()` +- **Search response shape:** Results are accessed via `.web`, `.news`, `.images`, `.developer` properties. Accessing `.data` raises an `AttributeError` with a helpful message directing you to these properties. +- **Async client:** Use `AsyncFirecrawl` (aliased as `AsyncFirecrawlApp`) for async/await usage. Same method signatures. +- **Class aliases:** `FirecrawlApp` is an alias for `Firecrawl`. + +## Source Of Truth + +- `/firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `/firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `/firecrawl/apps/python-sdk/firecrawl/client.py` +- `/firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 00000000..30a94339 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,256 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +og:title: "Rust Agent Quickstart | Firecrawl" +og:description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +This file is the canonical quickstart for external agents integrating with Firecrawl using the official Rust SDK. It is generated from SDK source and OpenAPI spec. + +## Install + +Add to your `Cargo.toml`: + +```toml +[dependencies] +firecrawl = "2" +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR_API_KEY")?; +``` + +For self-hosted instances: + +```rust +let client = Client::new_selfhosted( + "https://your-firecrawl-instance.com", + Some("fc-YOUR_API_KEY"), +)?; +``` + +Omitting the API key enables the keyless free tier (rate-limited per IP). + +## When To Use What + +- **`search`** -- Use when you start with a query and need to discover relevant URLs and their content. Returns categorized results from web, news, and images. +- **`scrape`** -- Use when you already have a URL and want to extract page content in structured formats (markdown, HTML, JSON, screenshots, etc.). +- **`interact`** -- Use when the page needs post-scrape browser actions: running code in the live browser session. + +## Search + +### Why use it + +Search finds relevant pages across the web for a given query, optionally scraping the results. It returns results grouped by source type (web, news, images). + +### Preferred SDK method + +```rust +client.search(query, options).await +``` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let response = client.search("firecrawl web scraping API", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions::default()), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +A convenience method is also available: + +```rust +let docs = client.search_and_scrape("firecrawl web scraping", 5).await?; +``` + +### Parameters + +All fields on `SearchOptions` are `Option` types. The struct derives `Default`. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | The search query (required, first positional argument). | +| `limit` | `Option` | Maximum number of results. Default: 5, Max: 20. | +| `sources` | `Option>` | Source types: `SearchSource::Web`, `News`, `Images`. | +| `categories` | `Option>` | Filter by category: `SearchCategory::Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Only include results from these domains. | +| `exclude_domains` | `Option>` | Exclude results from these domains. | +| `tbs` | `Option` | Time-based search filter (e.g. `"qdr:d"` for past day). | +| `location` | `Option` | Geographic location for search results. | +| `ignore_invalid_urls` | `Option` | Exclude invalid URLs from results. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `highlights` | `Option` | Generate query-relevant highlights. Defaults to `true`. | +| `scrape_options` | `Option` | Options applied when scraping search result pages. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +## Scrape + +### Why use it + +Scrape extracts content from a single URL and returns it in one or more formats: markdown, HTML, raw HTML, JSON (via schema), screenshots, links, images, audio, video, and more. + +### Preferred SDK method + +```rust +client.scrape(url, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + ..Default::default() +}).await?; + +if let Some(markdown) = &doc.markdown { + println!("{}", markdown); +} +``` + +For structured JSON extraction with a schema: + +```rust +use serde_json::json; + +let result = client.scrape_with_schema( + "https://example.com", + json!({"type": "object", "properties": {"title": {"type": "string"}}}), + Some("Extract the page title"), +).await?; +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` types. The struct derives `Default`. Fields serialize to camelCase for the API. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | The URL to scrape (required, first positional argument). | +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also object variants: `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `headers` | `Option>` | Custom HTTP headers sent with the request. | +| `include_tags` | `Option>` | Only include content from these HTML tags. | +| `exclude_tags` | `Option>` | Exclude content from these HTML tags. | +| `only_main_content` | `Option` | Only return main content, excluding navbars, footers, etc. | +| `timeout` | `Option` | Timeout in milliseconds. | +| `wait_for` | `Option` | Wait time in milliseconds before scraping. | +| `mobile` | `Option` | Scrape as a mobile device. | +| `parsers` | `Option>` | File processing control. `ParserConfig::Simple("pdf")` or `ParserConfig::Pdf { mode, max_pages }`. | +| `actions` | `Option>` | Actions to execute before content extraction: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `location` | `Option` | Geolocation with `country` and `languages` fields. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64-encoded images from markdown output. | +| `fast_mode` | `Option` | Enable fast mode for quicker scraping. | +| `block_ads` | `Option` | Block advertisements and cookie popups. | +| `proxy` | `Option` | Proxy mode: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `max_age` | `Option` | Use cached result if younger than this many seconds. | +| `min_age` | `Option` | Minimum cache age in seconds. | +| `store_in_cache` | `Option` | Whether to cache the result. | +| `lockdown` | `Option` | Only serve previously cached results. | +| `redact_pii` | `Option` | Redact personally identifiable information. | +| `audit_metadata` | `Option` | User attribution for SIEM logging. Field: `username: String`. | +| `profile` | `Option` | Persistent browser storage. Fields: `name: String`, `save_changes: Option`. | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction options: `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config: `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking config: `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute selectors: `selector`, `attribute`. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +## Interact + +### Why use it + +Interact lets you execute code in a live browser session that was opened by a prior `scrape` call. Use it for clicking buttons, filling forms, navigating multi-step flows, or extracting dynamic content. + +### Preferred SDK method + +```rust +client.interact(job_id, options).await +``` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR_API_KEY")?; + +let doc = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.job_id.as_ref()) + .expect("job_id required for interact"); + +let result = client.interact(job_id, ScrapeExecuteOptions { + code: Some("document.querySelector('button.load-more').click()".into()), + language: None, // defaults to Node + timeout: Some(30), + ..Default::default() +}).await?; + +println!("{:?}", result.stdout); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | The scrape job ID from a prior `scrape` call (required, first positional argument). | +| `code` | `Option` | Code to execute in the browser sandbox. Either `code` or `prompt` must be provided. | +| `prompt` | `Option` | Natural-language instruction for the browser. Either `code` or `prompt` must be provided. | +| `language` | `Option` | Language for code execution: `Python`, `Node`, `Bash`. Defaults to `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. | +| `origin` | `Option` | Origin label. Auto-set to `"rust-sdk@{version}"`. | + +### Stopping a session + +```rust +let delete_result = client.stop_interaction(job_id).await?; +println!("Credits billed: {:?}", delete_result.credits_billed); +``` + +## Notes + +- **Naming style:** Struct fields use snake_case in Rust. They serialize to camelCase for the API via serde. +- **Async:** All methods are async and return `Result`. +- **Deprecated aliases:** The following methods still work but are marked `#[deprecated]`: + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` +- **`max_age` / `min_age` units:** The Rust SDK uses seconds for these fields, unlike JS/Python which use milliseconds. +- **Search response:** `SearchData.web` contains `Vec`, a custom enum that distinguishes between web result metadata and full scraped documents. + +## Source Of Truth + +- `/firecrawl/apps/rust-sdk/src/v2/client.rs` +- `/firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `/firecrawl/apps/rust-sdk/src/v2/search.rs` +- `/firecrawl/apps/rust-sdk/Cargo.toml` +- `/firecrawl-docs/api-reference/v2-openapi.json`