diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..4682fa3ad --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,220 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for external agents. Generated from SDK source (`:firecrawl` **v1.9.1**, `firecrawl/apps/elixir-sdk`) and the v2 OpenAPI spec. The Elixir client is auto-generated from the OpenAPI spec; function names and parameter keys reflect the spec directly. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key") +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev crawl webhooks", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of atoms, strings, or maps. Sources to search: `:web`, `:news`, `:images` or `%{type: "web" | "news" | "images"}`. + +- `categories` — list of atoms, strings, or maps. Filter by category: `:github`, `:research`, `:pdf` or `%{type: ...}`. + +- `include_domains` — list of strings. Domains to include. + +- `exclude_domains` — list of strings. Domains to exclude. + +- `limit` — integer. Max number of results. + +- `tbs` — string. Time-based filter (e.g. `qdr:d`, `qdr:w`). + +- `location` — string. Localized results. + +- `country` — string. ISO 3166-1 alpha-2 targeting (e.g. `"US"`). + +- `ignore_invalid_urls` — boolean. Drop invalid URLs. + +- `timeout` — integer. Request timeout in milliseconds. + +- `highlights` — boolean. Generate query-relevant highlights. Defaults to true. + +- `enterprise` — list of strings. Enterprise search controls: `"zdr"` for zero data retention, `"anon"` for anonymized. + +- `scrape_options` — keyword list. Scrape each search result (see Scrape parameters). + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000, + actions: [ + %{type: "click", selector: "#accept"}, + %{type: "wait", milliseconds: 750}, + %{type: "scrape"} + ] +) +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `formats` — list of format strings or format maps. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + - Map formats: `%{type: "json", prompt: ..., schema: ...}`, `%{type: "screenshot", fullPage: ..., quality: ..., viewport: ...}`, `%{type: "changeTracking", modes: [...], tag: ...}`. + +- `headers` — map. Custom request headers. + +- `include_tags` — list of strings. Include only specific HTML tags. + +- `exclude_tags` — list of strings. Exclude specific HTML tags. + +- `only_main_content` — boolean. Strip nav, footer, and other boilerplate. + +- `timeout` — integer. Timeout in milliseconds. Min 1000, default 60000, max 300000. + +- `wait_for` — integer. Wait for the page to render (milliseconds). + +- `mobile` — boolean. Mobile viewport. + +- `parsers` — list of parser strings or maps: `"pdf"` or `%{type: "pdf", mode: "fast" | "auto" | "ocr", maxPages: integer}`. + +- `actions` — list of action maps. Pre-scrape browser actions: `wait`, `screenshot`, `click` (with optional `all`), `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. + +- `location` — keyword list with `country:` and `languages:`. Geo or language-aware scraping. + +- `skip_tls_verification` — boolean. Skip TLS verification. + +- `remove_base64_images` — boolean. Drop base64 images from markdown output. + +- `block_ads` — boolean. Ad and cookie popup blocking. + +- `proxy` — atom or string: `:basic`, `:enhanced`, `:auto`. + +- `max_age` — integer. Cached data up to a maximum age (milliseconds). + +- `min_age` — integer. Cached data only if at least this old (milliseconds). + +- `store_in_cache` — boolean. Cache the result. + +- `lockdown` — boolean. Serve only previously cached results; no outbound request. + +- `redact_pii` — boolean. Redact PII from content. + +- `audit_metadata` — keyword list with `username:`. User attribution for SIEM logging. + +- `profile` — keyword list with `name:` and optional `save_changes:`. Persistent browser profile. + +- `zero_data_retention` — boolean. Zero data retention for this scrape. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + "", + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +### Parameters + +- `job_id` — string (required, first positional arg). The scrape job ID. + +- `code` — string (required). Code to run in the browser session. + +- `language` — atom or string: `:python`, `:node`, `:bash`. Defaults to `:node`. + +- `timeout` — integer. Execution timeout in seconds. + +- `origin` — string. Optional origin label for execution telemetry. + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` + +Issues `DELETE /scrape/{jobId}/interact`. A bang variant `stop_interactive_scrape_browser_session!/2` is also available. + +## Notes + +- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec. +- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- This SDK exposes code-based interactions only (no `prompt` parameter on interact). +- Every request body includes an `"origin"` field set to `"elixir-sdk@1.9.1"` for telemetry. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..768a52b01 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,245 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.12.1**, `firecrawl/apps/java-sdk`) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); +``` + +The API key is resolved from: builder `.apiKey()`, then `FIRECRAWL_API_KEY` env, then `firecrawl.apiKey` system property. A blank key is permitted — scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +Return type: `SearchData`. Access result buckets with `getWeb()`, `getNews()`, `getImages()` (each `List>`, may be null). + +### Parameters + +- `query` — String (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — `List`. Source types: `"web"`, `"news"`, `"images"` as strings or `{type: ...}` maps. + +- `options.categories` — `List`. Categories: `"github"`, `"research"`, `"pdf"`. + +- `options.includeDomains` — `List`. Domains to include. + +- `options.excludeDomains` — `List`. Domains to exclude. + +- `options.limit` — Integer. Max number of results. + +- `options.tbs` — String. Time-based filter (e.g. `qdr:d` for past day). + +- `options.location` — String. Localized results. + +- `options.ignoreInvalidURLs` — Boolean. Drop invalid URLs. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.highlights` — Boolean. Generate query-relevant highlights. Defaults to true. + +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). + +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. + +- `options.formats` — `List`. Format strings or format config objects. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Format objects: `JsonFormat.builder().prompt(...).schema(...).build()`, `QuestionFormat`, `HighlightsFormat`, or maps for screenshot/changeTracking/attributes. + +- `options.headers` — `Map`. Custom request headers. + +- `options.includeTags` — `List`. Include only specific HTML tags. + +- `options.excludeTags` — `List`. Exclude specific HTML tags. + +- `options.onlyMainContent` — Boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.waitFor` — Integer. Wait for the page to render (milliseconds). + +- `options.mobile` — Boolean. Mobile viewport. + +- `options.parsers` — `List`. File parsing controls: `"pdf"` or `{type: "pdf", maxPages: n}`. + +- `options.actions` — `List>`. Pre-scrape browser actions: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. + +- `options.location` — `LocationConfig` with `country` and `languages`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — Boolean. Skip TLS verification. + +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown output. + +- `options.blockAds` — Boolean. Ad and cookie popup blocking. + +- `options.proxy` — String. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom proxy URL. + +- `options.maxAge` — Long. Cached data up to a maximum age (milliseconds). + +- `options.storeInCache` — Boolean. Cache the result. + +- `options.lockdown` — Boolean. Serve only previously cached results; no outbound request. + +- `options.redactPII` — Boolean. Redact PII from content. + +- `options.auditMetadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. + +- `options.integration` — String. Integration identifier. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +- `client.interact(jobId, code)` — defaults to language `"node"`, server default timeout +- `client.interact(jobId, code, language, timeout)` — timeout in seconds (1–300), null for server default +- `client.interact(jobId, code, language, timeout, origin)` — optional origin string + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +BrowserExecuteResponse result = client.interact( + "", + "console.log(await page.title());", + "node", + 60 +); +``` + +### Parameters + +- `jobId` — String (required). The scrape job ID. + +- `code` — String (required). Code to run in the browser session. + +- `language` — String. Runtime: `"python"`, `"node"`, `"bash"`. Defaults to `"node"`. + +- `timeout` — Integer. Execution timeout in seconds (1–300). Null for server default (30s). + +- `origin` — String. Optional origin label for request attribution. + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → `BrowserDeleteResponse` + +Response fields: `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`, `getError()`. + +## Notes + +- The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike Node.js, Python, and Rust SDKs). +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Async variants are available for all methods: `scrapeAsync`, `searchAsync`, `interactAsync`, `stopInteractiveBrowserAsync`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserExecuteResponse.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserDeleteResponse.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..19a47aaa5 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,246 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for external agents. Generated from SDK source (`firecrawl` **v4.32.0**, `firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or other browser actions after a scrape has created a session. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. + +- `options.categories` — array of `"github" | "research" | "pdf" | "developer"` or `{ type: ... }`. Filters results by category. + +- `options.includeDomains` — array of strings. Domains to include. Mutually exclusive with `excludeDomains`. + +- `options.excludeDomains` — array of strings. Domains to exclude. Mutually exclusive with `includeDomains`. + +- `options.limit` — number. Max number of results. + +- `options.tbs` — string. Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). + +- `options.location` — string. Localized results (e.g. `"San Francisco,California,United States"`). + +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped. + +- `options.timeout` — number. Request timeout in milliseconds. + +- `options.highlights` — boolean. Generate query-relevant highlights. Defaults to true. + +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). + +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise search controls for zero data retention. + +- `options.threatProtection` — object. Enterprise threat protection settings with `mode`, `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. + +- `options.integration` — string. Integration identifier for server-side tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `options.formats` — array of format strings or format objects. Output formats. + - Plain string formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object formats (require at least `type`): + - `{ type: "json", prompt?: string, schema?: object | Zod }` — JSON extraction. Do not pass `"json"` as a plain string (SDK rejects it); use an object. + - `{ type: "question", question: string }` — question-answer extraction. + - `{ type: "highlights", query: string }` — relevant source-text extraction. + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` — screenshot with options. + - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` is required. + - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` — attribute extraction. + +- `options.headers` — `Record`. Custom request headers. + +- `options.includeTags` — array of strings. Include only specific HTML tags. + +- `options.excludeTags` — array of strings. Exclude specific HTML tags. + +- `options.onlyMainContent` — boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — number. Timeout in milliseconds. + +- `options.waitFor` — number. Wait for the page to render (milliseconds). + +- `options.mobile` — boolean. Mobile viewport. + +- `options.parsers` — array of parser names or objects. File parsing controls. + - `"pdf"` — enable PDF parsing. + - `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }` — PDF parser options. + +- `options.actions` — array of action objects. Pre-scrape browser actions. + - `{ type: "wait", milliseconds?: number, selector?: string }` + - `{ type: "screenshot", fullPage?, quality?, viewport? }` + - `{ type: "click", selector: string }` + - `{ type: "write", text: string }` — click to focus the input first. + - `{ type: "press", key: string }` + - `{ type: "scroll", direction: "up" | "down", selector?: string }` + - `{ type: "scrape" }` + - `{ type: "executeJavascript", script: string }` + - `{ type: "pdf", format?, landscape?, scale? }` + +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — boolean. Skip TLS verification. + +- `options.removeBase64Images` — boolean. Drop base64 images from markdown output. + +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. + +- `options.blockAds` — boolean. Ad and cookie popup blocking. + +- `options.proxy` — string. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom proxy URL. + +- `options.maxAge` — number. Cached data up to a maximum age (milliseconds). + +- `options.minAge` — number. Cached data only if at least this old (milliseconds). + +- `options.storeInCache` — boolean. Cache the result. + +- `options.lockdown` — boolean. Serve only previously cached results; no outbound request. + +- `options.redactPII` — boolean or `{ mode?: "accurate" | "aggressive" | "fast", entities?: string[], replaceStyle?: "tag" | "mask" | "remove" }`. PII redaction. + +- `options.threatProtection` — object. Threat protection settings. + +- `options.auditMetadata` — `{ username: string }`. User attribution for SIEM logging. + +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile. + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrapeId`). At least one of `code` or `prompt` is required. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); +``` + +### Parameters + +- `jobId` — string (required). The scrape job ID from `document.metadata.scrapeId`. + +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage in the `node` runtime). + +- `args.prompt` — string. Natural-language instruction for the browser agent. + +- At least one of `args.code` or `args.prompt` must be non-empty (SDK throws otherwise). + +- `args.language` — `"python" | "node" | "bash"`. Runtime language. Defaults to `"node"`. + +- `args.timeout` — number. Execution timeout in seconds. + +### Stop session + +`client.stopInteraction(jobId)` → `Promise` + +Response fields: `success`, `sessionDurationMs`, `creditsBilled`, `error`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- The package declares **Node.js >= 22** in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..4f790505f --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,244 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.34.0**, `firecrawl/apps/python-sdk`) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```py +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```py +from firecrawl.v2.types import ScrapeOptions + +results = client.search( + "site:docs.firecrawl.dev crawl webhooks", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Web results are in `result.web`, news in `result.news`, images in `result.images`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of `"web" | "news" | "images"` or `Source` objects. Controls which sources are searched. + +- `categories` — list of `"github" | "research" | "pdf" | "developer"` or `Category` objects. Filters results by category. + +- `include_domains` — list of str. Domains to include. Mutually exclusive with `exclude_domains`. + +- `exclude_domains` — list of str. Domains to exclude. Mutually exclusive with `include_domains`. + +- `limit` — int. Max number of results. Default: 5. + +- `tbs` — str. Time-based filter (e.g. `qdr:d` for past day, `qdr:w` for past week). + +- `location` — str. Localized results (e.g. `"San Francisco,California,United States"`). Note: this is a plain string, unlike scrape's `Location` object. + +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped. + +- `timeout` — int. Request timeout in milliseconds. Default: 300000. + +- `highlights` — bool. Generate query-relevant highlights. Defaults to true. + +- `scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters for fields). + +- `enterprise` — list of str. Enterprise search controls: `"zdr"` for zero data retention, `"anon"` for anonymized. + +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection settings. + +- `integration` — str. Integration identifier for server-side tracking. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```py +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) +``` + +### Parameters + +- `url` — str (required). The URL to scrape. + +- `formats` — list of format strings or format dicts. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Dict formats (require at least `type`): + - `{"type": "json", "prompt": ..., "schema": ...}` — JSON extraction. Use a dict, not the plain string `"json"`. + - `{"type": "question", "question": ...}` — question-answer extraction. + - `{"type": "highlights", "query": ...}` — relevant source-text extraction. + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` — screenshot with options. + - `{"type": "changeTracking", "modes": ["git-diff" | "json"], "schema": ..., "prompt": ..., "tag": ...}` — `modes` is required. + - `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}` — attribute extraction. + +- `headers` — dict of str to str. Custom request headers. + +- `include_tags` — list of str. Include only specific HTML tags. + +- `exclude_tags` — list of str. Exclude specific HTML tags. + +- `only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `timeout` — int. Timeout in milliseconds. + +- `wait_for` — int. Wait for the page to render (milliseconds). + +- `mobile` — bool. Mobile viewport. + +- `parsers` — list of parser names or dicts. File parsing controls. + - `"pdf"` — enable PDF parsing. + - `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int}` — PDF parser options. + +- `actions` — list of action dicts. Pre-scrape browser actions. + - `{"type": "wait", "milliseconds": ... }` or `{"type": "wait", "selector": ...}` + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` + - `{"type": "click", "selector": ...}` + - `{"type": "write", "text": ...}` — click to focus the input first. + - `{"type": "press", "key": ...}` + - `{"type": "scroll", "direction": "up" | "down", "selector": ...}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": ...}` + - `{"type": "pdf", "format": ..., "landscape": ..., "scale": ...}` + +- `location` — dict with `country` and `languages`. Geo or language-aware scraping. + +- `skip_tls_verification` — bool. Skip TLS verification. + +- `remove_base64_images` — bool. Drop base64 images from markdown output. + +- `fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `block_ads` — bool. Ad and cookie popup blocking. + +- `proxy` — str. Proxy control: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. + +- `max_age` — int. Cached data up to a maximum age (milliseconds). + +- `store_in_cache` — bool. Cache the result. + +- `lockdown` — bool. Serve only previously cached results; no outbound request. + +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection settings. + +- `audit_metadata` — `AuditMetadata` with `username: str`. User attribution for SIEM logging. + +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be non-empty. + +### Example + +```py +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +``` + +### Parameters + +- `job_id` — str (required). The scrape job ID from `document.metadata.scrape_id`. + +- `code` — str. Code to run in the browser session (optional if `prompt` is set). + +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent (optional if `code` is set). + +- `language` — `"python" | "node" | "bash"`. Runtime language. Defaults to `"node"`. + +- `timeout` — int. Execution timeout in seconds. + +### Stop session + +`client.stop_interaction(job_id)` → `BrowserDeleteResponse` + +Response fields: `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `ScrapeOptions` (Pydantic model used for `scrape_options` in search) also includes `min_age` and `redact_pii` fields that are not exposed as top-level kwargs on `client.scrape()`. +- All method parameters use snake_case. Pydantic models handle the camelCase wire format internally. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/methods/search.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/methods/scrape.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..ac6f31057 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,246 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.12.1**, `firecrawl/apps/rust-sdk`) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +The API key is optional. Scrape, search, and interact fall back to a keyless free tier (rate-limited per IP). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; +``` + +Return type: `SearchResponse { success, data: SearchData, warning }`. `SearchData` has `web: Option>`, `news: Option>`, `images: Option>`. Each web item is either `WebResult(SearchResultWeb)` or `Document(Document)`. + +### Parameters + +- `query` — `impl AsRef` (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — `Vec`. Sources to query: `Web`, `News`, `Images`. + +- `options.categories` — `Vec`. Filter categories: `Github`, `Research`, `Pdf`. + +- `options.include_domains` — `Vec`. Domains to include. + +- `options.exclude_domains` — `Vec`. Domains to exclude. + +- `options.limit` — u32. Max results. Default: 5, max: 20. + +- `options.tbs` — String. Time-based filter (e.g. `qdr:d`, `qdr:w`). + +- `options.location` — String. Localized results. + +- `options.ignore_invalid_urls` — bool. Drop invalid URLs. + +- `options.timeout` — u32. Timeout in milliseconds. + +- `options.highlights` — bool. Generate query-relevant highlights. Defaults to true. + +- `options.scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). + +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions, Action}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + actions: Some(vec![ + Action::Click { selector: "#accept".to_string() }, + Action::Wait { milliseconds: Some(750), selector: None }, + Action::Scrape, + ]), + ..Default::default() + }) + .await?; +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. + +- `options.formats` — `Vec`. Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. + +- `options.headers` — `HashMap`. Custom request headers. + +- `options.include_tags` — `Vec`. Include only specific HTML tags. + +- `options.exclude_tags` — `Vec`. Exclude specific HTML tags. + +- `options.only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `options.timeout` — u32. Timeout in milliseconds. + +- `options.wait_for` — u32. Wait for the page to render (milliseconds). + +- `options.mobile` — bool. Mobile viewport. + +- `options.parsers` — `Vec`. File parsing controls: `Simple("pdf".to_string())` or `Pdf { parser_type, mode, max_pages }`. + +- `options.actions` — `Vec`. Pre-scrape browser actions: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. + +- `options.location` — `LocationConfig` with `country` and `languages`. Geo or language-aware scraping. + +- `options.skip_tls_verification` — bool. Skip TLS verification. + +- `options.remove_base64_images` — bool. Drop base64 images from markdown output. + +- `options.fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `options.block_ads` — bool. Ad and cookie popup blocking. + +- `options.proxy` — `ProxyType`: `Basic`, `Stealth`, `Enhanced`, `Auto`. + +- `options.max_age` — u32. Cached data up to a maximum age (milliseconds). + +- `options.min_age` — u32. Cached data only if at least this old (milliseconds). + +- `options.store_in_cache` — bool. Cache the result. + +- `options.lockdown` — bool. Serve only previously cached results; no outbound request. + +- `options.redact_pii` — bool. Redact PII from content. + +- `options.audit_metadata` — `AuditMetadata` with `username: String`. User attribution for SIEM logging. + +- `options.profile` — `ProfileConfig` with `name: String` and `save_changes: Option`. Persistent browser profile. + +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. JSON extraction options. + +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. Screenshot options. + +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff` or `Json`), `schema`, `prompt`, `tag`. + +- `options.attribute_selectors` — `Vec` with `selector` and `attribute`. + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeExecuteOptions}; + +let result = client + .interact( + "", + ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }, + ) + .await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). The scrape job ID. + +- `options.code` — `Option`. Code to run in the browser session. + +- `options.prompt` — `Option`. Natural-language instruction for the browser agent. + +- At least one of `code` or `prompt` must be non-empty; otherwise `FirecrawlError::Misuse` is returned. + +- `options.language` — `ScrapeExecuteLanguage`: `Python`, `Node`, `Bash`. Defaults to `Node`. + +- `options.timeout` — u32. Execution timeout in seconds. + +### Stop session + +`client.stop_interaction(job_id)` → `Result` + +Response fields: `success`, `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. +- `search_and_scrape(query, limit)` is a convenience helper that calls search with default `ScrapeOptions` and returns `Vec`. +- All types are exported at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`