From 289fb5df330827050debf1a5f62ed918408673cc Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 2 Aug 2026 13:13:49 +0000 Subject: [PATCH] docs(agent-quickstart): add per-language agent quickstarts for search, scrape, interact Generate canonical quickstart docs for Node.js, Python, Rust, Java, and Elixir from latest SDK source and OpenAPI spec. Each file covers install, auth, search, scrape, interact with full parameter documentation including new fields (lockdown, redactPII, threatProtection, auditMetadata, includeDomains/excludeDomains, highlights, enterprise). SDK versions: JS v4.32.0, Python v4.34.0, Rust v2.12.1, Java v1.12.1, Elixir v1.9.1. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01LNmnX4A2jmTY5ZxuRpBKJm --- agent-quickstart/elixir.mdx | 220 ++++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 245 +++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 246 ++++++++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 244 +++++++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 246 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1201 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..4682fa3ad --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,220 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for external agents. Generated from SDK source (`:firecrawl` **v1.9.1**, `firecrawl/apps/elixir-sdk`) and the v2 OpenAPI spec. The Elixir client is auto-generated from the OpenAPI spec; function names and parameter keys reflect the spec directly. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key") +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev crawl webhooks", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of atoms, strings, or maps. Sources to search: `:web`, `:news`, `:images` or `%{type: "web" | "news" | "images"}`. + +- `categories` — list of atoms, strings, or maps. Filter by category: `:github`, `:research`, `:pdf` or `%{type: ...}`. + +- `include_domains` — list of strings. Domains to include. + +- `exclude_domains` — list of strings. Domains to exclude. + +- `limit` — integer. Max number of results. + +- `tbs` — string. Time-based filter (e.g. `qdr:d`, `qdr:w`). + +- `location` — string. Localized results. + +- `country` — string. ISO 3166-1 alpha-2 targeting (e.g. `"US"`). + +- `ignore_invalid_urls` — boolean. Drop invalid URLs. + +- `timeout` — integer. Request timeout in milliseconds. + +- `highlights` — boolean. Generate query-relevant highlights. Defaults to true. + +- `enterprise` — list of strings. Enterprise search controls: `"zdr"` for zero data retention, `"anon"` for anonymized. + +- `scrape_options` — keyword list. Scrape each search result (see Scrape parameters). + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000, + actions: [ + %{type: "click", selector: "#accept"}, + %{type: "wait", milliseconds: 750}, + %{type: "scrape"} + ] +) +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `formats` — list of format strings or format maps. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + - Map formats: `%{type: "json", prompt: ..., schema: ...}`, `%{type: "screenshot", fullPage: ..., quality: ..., viewport: ...}`, `%{type: "changeTracking", modes: [...], tag: ...}`. + +- `headers` — map. Custom request headers. + +- `include_tags` — list of strings. Include only specific HTML tags. + +- `exclude_tags` — list of strings. Exclude specific HTML tags. + +- `only_main_content` — boolean. Strip nav, footer, and other boilerplate. + +- `timeout` — integer. Timeout in milliseconds. Min 1000, default 60000, max 300000. + +- `wait_for` — integer. Wait for the page to render (milliseconds). + +- `mobile` — boolean. Mobile viewport. + +- `parsers` — list of parser strings or maps: `"pdf"` or `%{type: "pdf", mode: "fast" | "auto" | "ocr", maxPages: integer}`. + +- `actions` — list of action maps. Pre-scrape browser actions: `wait`, `screenshot`, `click` (with optional `all`), `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. + +- `location` — keyword list with `country:` and `languages:`. Geo or language-aware scraping. + +- `skip_tls_verification` — boolean. Skip TLS verification. + +- `remove_base64_images` — boolean. Drop base64 images from markdown output. + +- `block_ads` — boolean. Ad and cookie popup blocking. + +- `proxy` — atom or string: `:basic`, `:enhanced`, `:auto`. + +- `max_age` — integer. Cached data up to a maximum age (milliseconds). + +- `min_age` — integer. Cached data only if at least this old (milliseconds). + +- `store_in_cache` — boolean. Cache the result. + +- `lockdown` — boolean. Serve only previously cached results; no outbound request. + +- `redact_pii` — boolean. Redact PII from content. + +- `audit_metadata` — keyword list with `username:`. User attribution for SIEM logging. + +- `profile` — keyword list with `name:` and optional `save_changes:`. Persistent browser profile. + +- `zero_data_retention` — boolean. Zero data retention for this scrape. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + "", + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +### Parameters + +- `job_id` — string (required, first positional arg). The scrape job ID. + +- `code` — string (required). Code to run in the browser session. + +- `language` — atom or string: `:python`, `:node`, `:bash`. Defaults to `:node`. + +- `timeout` — integer. Execution timeout in seconds. + +- `origin` — string. Optional origin label for execution telemetry. + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` + +Issues `DELETE /scrape/{jobId}/interact`. A bang variant `stop_interactive_scrape_browser_session!/2` is also available. + +## Notes + +- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec. +- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- This SDK exposes code-based interactions only (no `prompt` parameter on interact). +- Every request body includes an `"origin"` field set to `"elixir-sdk@1.9.1"` for telemetry. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..768a52b01 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,245 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.12.1**, `firecrawl/apps/java-sdk`) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); +``` + +The API key is resolved from: builder `.apiKey()`, then `FIRECRAWL_API_KEY` env, then `firecrawl.apiKey` system property. A blank key is permitted — scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +Return type: `SearchData`. Access result buckets with `getWeb()`, `getNews()`, `getImages()` (each `List>`, may be null). + +### Parameters + +- `query` — String (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — `List`. Source types: `"web"`, `"news"`, `"images"` as strings or `{type: ...}` maps. + +- `options.categories` — `List`. Categories: `"github"`, `"research"`, `"pdf"`. + +- `options.includeDomains` — `List`. Domains to include. + +- `options.excludeDomains` — `List`. Domains to exclude. + +- `options.limit` — Integer. Max number of results. + +- `options.tbs` — String. Time-based filter (e.g. `qdr:d` for past day). + +- `options.location` — String. Localized results. + +- `options.ignoreInvalidURLs` — Boolean. Drop invalid URLs. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.highlights` — Boolean. Generate query-relevant highlights. Defaults to true. + +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). + +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. + +- `options.formats` — `List`. Format strings or format config objects. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Format objects: `JsonFormat.builder().prompt(...).schema(...).build()`, `QuestionFormat`, `HighlightsFormat`, or maps for screenshot/changeTracking/attributes. + +- `options.headers` — `Map`. Custom request headers. + +- `options.includeTags` — `List`. Include only specific HTML tags. + +- `options.excludeTags` — `List`. Exclude specific HTML tags. + +- `options.onlyMainContent` — Boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.waitFor` — Integer. Wait for the page to render (milliseconds). + +- `options.mobile` — Boolean. Mobile viewport. + +- `options.parsers` — `List`. File parsing controls: `"pdf"` or `{type: "pdf", maxPages: n}`. + +- `options.actions` — `List>`. Pre-scrape browser actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. + +- `options.location` — `LocationConfig` with `country` and `languages`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — Boolean. Skip TLS verification. + +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown output. + +- `options.blockAds` — Boolean. Ad and cookie popup blocking. + +- `options.proxy` — String. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom proxy URL. + +- `options.maxAge` — Long. Cached data up to a maximum age (milliseconds). + +- `options.storeInCache` — Boolean. Cache the result. + +- `options.lockdown` — Boolean. Serve only previously cached results; no outbound request. + +- `options.redactPII` — Boolean. Redact PII from content. + +- `options.auditMetadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. + +- `options.integration` — String. Integration identifier. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +- `client.interact(jobId, code)` — defaults to language `"node"`, server default timeout +- `client.interact(jobId, code, language, timeout)` — timeout in seconds (1–300), null for server default +- `client.interact(jobId, code, language, timeout, origin)` — optional origin string + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +BrowserExecuteResponse result = client.interact( + "", + "console.log(await page.title());", + "node", + 60 +); +``` + +### Parameters + +- `jobId` — String (required). The scrape job ID. + +- `code` — String (required). Code to run in the browser session. + +- `language` — String. Runtime: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. + +- `timeout` — Integer. Execution timeout in seconds (1–300). Null for server default (30s). + +- `origin` — String. Optional origin label for request attribution. + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → `BrowserDeleteResponse` + +Response fields: `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`, `getError()`. + +## Notes + +- The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike Node.js, Python, and Rust SDKs). +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Async variants are available for all methods: `scrapeAsync`, `searchAsync`, `interactAsync`, `stopInteractiveBrowserAsync`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserExecuteResponse.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserDeleteResponse.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..19a47aaa5 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,246 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for external agents. Generated from SDK source (`firecrawl` **v4.32.0**, `firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or other browser actions after a scrape has created a session. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. + +- `options.categories` — array of `"github" | "research" | "pdf" | "developer"` or `{ type: ... }`. Filters results by category. + +- `options.includeDomains` — array of strings. Domains to include. Mutually exclusive with `excludeDomains`. + +- `options.excludeDomains` — array of strings. Domains to exclude. Mutually exclusive with `includeDomains`. + +- `options.limit` — number. Max number of results. + +- `options.tbs` — string. Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). + +- `options.location` — string. Localized results (e.g. `"San Francisco,California,United States"`). + +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped. + +- `options.timeout` — number. Request timeout in milliseconds. + +- `options.highlights` — boolean. Generate query-relevant highlights. Defaults to true. + +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). + +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise search controls for zero data retention. + +- `options.threatProtection` — object. Enterprise threat protection settings with `mode`, `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. + +- `options.integration` — string. Integration identifier for server-side tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `options.formats` — array of format strings or format objects. Output formats. + - Plain string formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object formats (require at least `type`): + - `{ type: "json", prompt?: string, schema?: object | Zod }` — JSON extraction. Do not pass `"json"` as a plain string (SDK rejects it); use an object. + - `{ type: "question", question: string }` — question-answer extraction. + - `{ type: "highlights", query: string }` — relevant source-text extraction. + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` — screenshot with options. + - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` is required. + - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` — attribute extraction. + +- `options.headers` — `Record`. Custom request headers. + +- `options.includeTags` — array of strings. Include only specific HTML tags. + +- `options.excludeTags` — array of strings. Exclude specific HTML tags. + +- `options.onlyMainContent` — boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — number. Timeout in milliseconds. + +- `options.waitFor` — number. Wait for the page to render (milliseconds). + +- `options.mobile` — boolean. Mobile viewport. + +- `options.parsers` — array of parser names or objects. File parsing controls. + - `"pdf"` — enable PDF parsing. + - `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }` — PDF parser options. + +- `options.actions` — array of action objects. Pre-scrape browser actions. + - `{ type: "wait", milliseconds?: number, selector?: string }` + - `{ type: "screenshot", fullPage?, quality?, viewport? }` + - `{ type: "click", selector: string }` + - `{ type: "write", text: string }` — click to focus the input first. + - `{ type: "press", key: string }` + - `{ type: "scroll", direction: "up" | "down", selector?: string }` + - `{ type: "scrape" }` + - `{ type: "executeJavascript", script: string }` + - `{ type: "pdf", format?, landscape?, scale? }` + +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — boolean. Skip TLS verification. + +- `options.removeBase64Images` — boolean. Drop base64 images from markdown output. + +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. + +- `options.blockAds` — boolean. Ad and cookie popup blocking. + +- `options.proxy` — string. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom proxy URL. + +- `options.maxAge` — number. Cached data up to a maximum age (milliseconds). + +- `options.minAge` — number. Cached data only if at least this old (milliseconds). + +- `options.storeInCache` — boolean. Cache the result. + +- `options.lockdown` — boolean. Serve only previously cached results; no outbound request. + +- `options.redactPII` — boolean or `{ mode?: "accurate" | "aggressive" | "fast", entities?: string[], replaceStyle?: "tag" | "mask" | "remove" }`. PII redaction. + +- `options.threatProtection` — object. Threat protection settings. + +- `options.auditMetadata` — `{ username: string }`. User attribution for SIEM logging. + +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrapeId`). At least one of `code` or `prompt` is required. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); +``` + +### Parameters + +- `jobId` — string (required). The scrape job ID from `document.metadata.scrapeId`. + +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage in the `node` runtime). + +- `args.prompt` — string. Natural-language instruction for the browser agent. + +- At least one of `args.code` or `args.prompt` must be non-empty (SDK throws otherwise). + +- `args.language` — `"python" | "node" | "bash"`. Runtime language. Defaults to `"node"`. + +- `args.timeout` — number. Execution timeout in seconds. + +### Stop session + +`client.stopInteraction(jobId)` → `Promise` + +Response fields: `success`, `sessionDurationMs`, `creditsBilled`, `error`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- The package declares **Node.js >= 22** in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..4f790505f --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,244 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.34.0**, `firecrawl/apps/python-sdk`) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```py +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```py +from firecrawl.v2.types import ScrapeOptions + +results = client.search( + "site:docs.firecrawl.dev crawl webhooks", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of `"web" | "news" | "images"` or `Source` objects. Controls which sources are searched. + +- `categories` — list of `"github" | "research" | "pdf" | "developer"` or `Category` objects. Filters results by category. + +- `include_domains` — list of str. Domains to include. Mutually exclusive with `exclude_domains`. + +- `exclude_domains` — list of str. Domains to exclude. Mutually exclusive with `include_domains`. + +- `limit` — int. Max number of results. Default: 5. + +- `tbs` — str. Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). + +- `location` — str. Localized results (e.g. `"San Francisco,California,United States"`). Note: this is a plain string, unlike scrape's `Location` object. + +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped. + +- `timeout` — int. Request timeout in milliseconds. Default: 300000. + +- `highlights` — bool. Generate query-relevant highlights. Defaults to true. + +- `scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). + +- `enterprise` — list of str. Enterprise search controls: `"zdr"` for zero data retention, `"anon"` for anonymized. + +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection settings. + +- `integration` — str. Integration identifier for server-side tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```py +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) +``` + +### Parameters + +- `url` — str (required). The URL to scrape. + +- `formats` — list of format strings or format dicts. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Dict formats (require at least `type`): + - `{"type": "json", "prompt": ..., "schema": ...}` — JSON extraction. Use a dict, not the plain string `"json"`. + - `{"type": "question", "question": ...}` — question-answer extraction. + - `{"type": "highlights", "query": ...}` — relevant source-text extraction. + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` — screenshot with options. + - `{"type": "changeTracking", "modes": ["git-diff" | "json"], "schema": ..., "prompt": ..., "tag": ...}` — `modes` is required. + - `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}` — attribute extraction. + +- `headers` — dict of str to str. Custom request headers. + +- `include_tags` — list of str. Include only specific HTML tags. + +- `exclude_tags` — list of str. Exclude specific HTML tags. + +- `only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `timeout` — int. Timeout in milliseconds. + +- `wait_for` — int. Wait for the page to render (milliseconds). + +- `mobile` — bool. Mobile viewport. + +- `parsers` — list of parser names or dicts. File parsing controls. + - `"pdf"` — enable PDF parsing. + - `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int}` — PDF parser options. + +- `actions` — list of action dicts. Pre-scrape browser actions. + - `{"type": "wait", "milliseconds": ... }` or `{"type": "wait", "selector": ...}` + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` + - `{"type": "click", "selector": ...}` + - `{"type": "write", "text": ...}` — click to focus the input first. + - `{"type": "press", "key": ...}` + - `{"type": "scroll", "direction": "up" | "down", "selector": ...}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": ...}` + - `{"type": "pdf", "format": ..., "landscape": ..., "scale": ...}` + +- `location` — dict with `country` and `languages`. Geo or language-aware scraping. + +- `skip_tls_verification` — bool. Skip TLS verification. + +- `remove_base64_images` — bool. Drop base64 images from markdown output. + +- `fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `block_ads` — bool. Ad and cookie popup blocking. + +- `proxy` — str. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. + +- `max_age` — int. Cached data up to a maximum age (milliseconds). + +- `store_in_cache` — bool. Cache the result. + +- `lockdown` — bool. Serve only previously cached results; no outbound request. + +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection settings. + +- `audit_metadata` — `AuditMetadata` with `username: str`. User attribution for SIEM logging. + +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be non-empty. + +### Example + +```py +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +``` + +### Parameters + +- `job_id` — str (required). The scrape job ID from `document.metadata.scrape_id`. + +- `code` — str. Code to run in the browser session (optional if `prompt` is set). + +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent (optional if `code` is set). + +- `language` — `"python" | "node" | "bash"`. Runtime language. Defaults to `"node"`. + +- `timeout` — int. Execution timeout in seconds. + +### Stop session + +`client.stop_interaction(job_id)` → `BrowserDeleteResponse` + +Response fields: `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `ScrapeOptions` (Pydantic model used for `scrape_options` in search) also includes `min_age` and `redact_pii` fields that are not exposed as top-level kwargs on `client.scrape()`. +- All method parameters use snake_case. Pydantic models handle the camelCase wire format internally. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/methods/search.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/methods/scrape.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..ac6f31057 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,246 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.12.1**, `firecrawl/apps/rust-sdk`) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; +``` + +Return type: `SearchResponse { success, data: SearchData, warning }`. `SearchData` has `web: Option>`, `news: Option>`, `images: Option>`. Each web item is either `WebResult(SearchResultWeb)` or `Document(Document)`. + +### Parameters + +- `query` — `impl AsRef` (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — `Vec`. Sources to query: `Web`, `News`, `Images`. + +- `options.categories` — `Vec`. Filter categories: `Github`, `Research`, `Pdf`. + +- `options.include_domains` — `Vec`. Domains to include. + +- `options.exclude_domains` — `Vec`. Domains to exclude. + +- `options.limit` — u32. Max results. Default: 5, max: 20. + +- `options.tbs` — String. Time-based filter (e.g. `qdr:d`, `qdr:w`). + +- `options.location` — String. Localized results. + +- `options.ignore_invalid_urls` — bool. Drop invalid URLs. + +- `options.timeout` — u32. Timeout in milliseconds. + +- `options.highlights` — bool. Generate query-relevant highlights. Defaults to true. + +- `options.scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). + +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions, Action}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + actions: Some(vec![ + Action::Click { selector: "#accept".to_string() }, + Action::Wait { milliseconds: Some(750), selector: None }, + Action::Scrape, + ]), + ..Default::default() + }) + .await?; +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. + +- `options.formats` — `Vec`. Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. + +- `options.headers` — `HashMap`. Custom request headers. + +- `options.include_tags` — `Vec`. Include only specific HTML tags. + +- `options.exclude_tags` — `Vec`. Exclude specific HTML tags. + +- `options.only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `options.timeout` — u32. Timeout in milliseconds. + +- `options.wait_for` — u32. Wait for the page to render (milliseconds). + +- `options.mobile` — bool. Mobile viewport. + +- `options.parsers` — `Vec`. File parsing controls: `Simple("pdf".to_string())` or `Pdf { parser_type, mode, max_pages }`. + +- `options.actions` — `Vec`. Pre-scrape browser actions: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. + +- `options.location` — `LocationConfig` with `country` and `languages`. Geo or language-aware scraping. + +- `options.skip_tls_verification` — bool. Skip TLS verification. + +- `options.remove_base64_images` — bool. Drop base64 images from markdown output. + +- `options.fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `options.block_ads` — bool. Ad and cookie popup blocking. + +- `options.proxy` — `ProxyType`: `Basic`, `Stealth`, `Enhanced`, `Auto`. + +- `options.max_age` — u32. Cached data up to a maximum age (milliseconds). + +- `options.min_age` — u32. Cached data only if at least this old (milliseconds). + +- `options.store_in_cache` — bool. Cache the result. + +- `options.lockdown` — bool. Serve only previously cached results; no outbound request. + +- `options.redact_pii` — bool. Redact PII from content. + +- `options.audit_metadata` — `AuditMetadata` with `username: String`. User attribution for SIEM logging. + +- `options.profile` — `ProfileConfig` with `name: String` and `save_changes: Option`. Persistent browser profile. + +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. JSON extraction options. + +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. Screenshot options. + +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff` or `Json`), `schema`, `prompt`, `tag`. + +- `options.attribute_selectors` — `Vec` with `selector` and `attribute`. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeExecuteOptions}; + +let result = client + .interact( + "", + ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }, + ) + .await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). The scrape job ID. + +- `options.code` — `Option`. Code to run in the browser session. + +- `options.prompt` — `Option`. Natural-language instruction for the browser agent. + +- At least one of `code` or `prompt` must be non-empty; otherwise `FirecrawlError::Misuse` is returned. + +- `options.language` — `ScrapeExecuteLanguage`: `Python`, `Node`, `Bash`. Defaults to `Node`. + +- `options.timeout` — u32. Execution timeout in seconds. + +### Stop session + +`client.stop_interaction(job_id)` → `Result` + +Response fields: `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. +- `search_and_scrape(query, limit)` is a convenience helper that calls search with default `ScrapeOptions` and returns `Vec`. +- All types are exported at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`