From 69c60f4abe7aea28078202ccd4d505340634f6b4 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 11 Aug 2026 13:40:40 +0000 Subject: [PATCH] docs: update agent source-of-truth quickstarts to latest SDK versions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refresh all 6 agent source-of-truth files against current SDK source: - Node.js: 4.18.2 → 4.32.0 - Python: 4.22.1 → 4.34.0 - Rust: 2.0.0 → 2.12.1 - Java: 1.2.0 → 1.12.1 - Elixir: 1.0.0 → 1.9.1 New parameters across all SDKs: lockdown, redactPII/redact_pii, auditMetadata/audit_metadata, threatProtection/threat_protection, integration. New format types: product, menu, question, highlights. New search params: includeDomains/include_domains, excludeDomains/exclude_domains, highlights, developer category. Fixed stale Rust SDK source paths (src/v2/ → src/). Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_018yQHiz3LXZv7pbT6Z56WNC --- agent-source-of-truth/curl.mdx | 49 ++++++++++++++--- agent-source-of-truth/elixir.mdx | 43 +++++++++++++-- agent-source-of-truth/java.mdx | 49 +++++++++++++++-- agent-source-of-truth/node.mdx | 91 +++++++++++++++++++++++++++++--- agent-source-of-truth/python.mdx | 67 +++++++++++++++++++++-- agent-source-of-truth/rust.mdx | 59 ++++++++++++++++++--- 6 files changed, 324 insertions(+), 34 deletions(-) diff --git a/agent-source-of-truth/curl.mdx b/agent-source-of-truth/curl.mdx index 6fded6451..85729e149 100644 --- a/agent-source-of-truth/curl.mdx +++ b/agent-source-of-truth/curl.mdx @@ -75,7 +75,7 @@ curl -X POST "https://api.firecrawl.dev/v2/search" \ Successful responses include `success`, `data`, optional `warning`, `id`, and `creditsUsed`. -- `data.web`, `data.images`, `data.news`: result arrays; which keys appear depends on `sources` (by default only `data.web` is populated). +- `data.web`, `data.news`, `data.images`, `data.developer`: result arrays; which keys appear depends on `sources` and `categories` (by default only `data.web` is populated). - Web and news items include fields such as `title`, `url`, and (when `scrapeOptions` / formats request it) `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `audio`, `video`, and `metadata`. - Image items include fields such as `imageUrl`, `url`, and dimensions when available. - `warning`: optional human-readable notice. @@ -104,6 +104,17 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c - `{ "type": "github" }` - `{ "type": "research" }` - `{ "type": "pdf" }` + - `{ "type": "developer" }` + +- `includeDomains` + - Type: array of strings + - Use when: you want to restrict results to specific domains. + - Notes: mutually exclusive with `excludeDomains`. + +- `excludeDomains` + - Type: array of strings + - Use when: you want to exclude specific domains from results. + - Notes: mutually exclusive with `includeDomains`. - `limit` - Type: integer (minimum 1, maximum 100, default 5) @@ -129,6 +140,10 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c - Type: integer (milliseconds, default 60000) - Use when: you need a request timeout in milliseconds. +- `highlights` + - Type: boolean (default true) + - Use when: you want query-relevant highlights in results. + - `enterprise` - Type: array of strings (`"anon"` or `"zdr"` per item) - Use when: you need enterprise search controls. @@ -227,14 +242,19 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - `"summary"`: summary output - `"changeTracking"`: change tracking output - `"json"`: JSON extraction + - `"attributes"`: attribute extraction - `"branding"`: branding profile output + - `"product"`: product data extraction + - `"menu"`: menu/navigation extraction - `"audio"`: audio extraction - `"video"`: video extraction - - Format object fields: - - `type`: one of the format strings above - - `prompt`, `schema`: JSON extraction options for `type: "json"` - - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` - - `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"` + - Object-only format types: + - `{ "type": "json", "prompt": "...", "schema": {...} }`: at least one of `prompt` or `schema` required + - `{ "type": "question", "question": "..." }`: question-answer extraction + - `{ "type": "highlights", "query": "..." }`: relevant source-text extraction + - `{ "type": "screenshot", "fullPage": true, "quality": 80, "viewport": {...} }`: screenshot with options + - `{ "type": "changeTracking", "modes": ["git-diff"], "schema": {...}, "prompt": "...", "tag": "..." }`: `modes` required + - `{ "type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}] }`: attribute extraction - `headers` - Type: object @@ -316,6 +336,19 @@ Successful responses include `success` and `data`. Common `data` fields (dependi - Type: boolean - Use when: you want Firecrawl to cache the result. +- `lockdown` + - Type: boolean + - Use when: you want to serve only previously cached results. Returns 404 if nothing is cached. + +- `redactPII` + - Type: boolean or object + - Use when: you want to redact personally identifiable information from output. + - Object fields: `mode` (`"accurate"`, `"aggressive"`, `"fast"`), `entities` (array of `"PERSON"`, `"EMAIL"`, `"PHONE"`, `"LOCATION"`, `"FINANCIAL"`, `"SECRET"`), `replaceStyle` (`"tag"`, `"mask"`, `"remove"`) + +- `auditMetadata` + - Type: object with `username` (string, required) + - Use when: you need user attribution for SIEM logging. + - `profile` - Type: object with `name` and optional `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. @@ -489,4 +522,6 @@ curl -X POST "https://api.firecrawl.dev/v2/support/docs-search" \ - `firecrawl-docs/api-reference/v2-openapi.json` - `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` - `firecrawl/apps/python-sdk/firecrawl/v2/types.py` -- `firecrawl/apps/rust-sdk/src/v2/scrape.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` diff --git a/agent-source-of-truth/elixir.mdx b/agent-source-of-truth/elixir.mdx index 703f6d6b2..b5f770257 100644 --- a/agent-source-of-truth/elixir.mdx +++ b/agent-source-of-truth/elixir.mdx @@ -10,7 +10,7 @@ Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source Add to `mix.exs`: ```elixir -{:firecrawl, "~> 1.0.0"} +{:firecrawl, "~> 1.9.1"} ``` ## Authenticate @@ -58,6 +58,10 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or country: "US", ignore_invalid_urls: true, timeout: 60000, + highlights: true, + include_domains: ["docs.firecrawl.dev"], + exclude_domains: ["old.firecrawl.dev"], + enterprise: ["zdr"], scrape_options: [ formats: [ "markdown", @@ -97,6 +101,14 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - `"pdf"` or `:pdf` - `%{type: "github" | "research" | "pdf"}` +- `include_domains` + - Type: list of strings + - Use when: you want to restrict results to specific domains. + +- `exclude_domains` + - Type: list of strings + - Use when: you want to exclude results from specific domains. + - `limit` - Type: integer - Use when: you want to cap results. @@ -121,6 +133,10 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - Type: integer - Use when: you need a request timeout in milliseconds. +- `highlights` + - Type: boolean (default true) + - Use when: you want highlighted snippets in search results. + - `enterprise` - Type: list of strings - Use when: you need enterprise search controls. @@ -160,8 +176,11 @@ Use scrape when you already have a URL and want structured content in one or mor "markdown", "links", %{type: "json", prompt: "Extract plan names and prices."}, + %{type: "question", question: "What is the enterprise plan price?"}, + %{type: "highlights", query: "pricing tiers"}, %{type: "screenshot", fullPage: true, quality: 80, viewport: %{width: 1280, height: 720}}, - %{type: "changeTracking", modes: ["git-diff"], tag: "pricing"} + %{type: "changeTracking", modes: ["git-diff"], tag: "pricing"}, + %{type: "attributes", selectors: [%{selector: "h1", attribute: "textContent"}]} ], headers: %{"User-Agent" => "FirecrawlDocsBot/1.0"}, only_main_content: true, @@ -179,6 +198,9 @@ Use scrape when you already have a URL and want structured content in one or mor max_age: 86400000, min_age: 1, store_in_cache: true, + lockdown: false, + redact_pii: false, + audit_metadata: [username: "docs-bot"], profile: [name: "docs-session", save_changes: true], zero_data_retention: false ) @@ -209,6 +231,9 @@ Use scrape when you already have a URL and want structured content in one or mor - Format map fields: - `type`: one of the format strings above - `prompt`, `schema`: JSON extraction options for `type: "json"` + - `question`: question to answer for `type: "question"` + - `query`: search query for `type: "highlights"` + - `selectors`: list of selector maps for `type: "attributes"` - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` - `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"` @@ -294,6 +319,18 @@ Use scrape when you already have a URL and want structured content in one or mor - Type: boolean - Use when: you want Firecrawl to cache the result. +- `lockdown` + - Type: boolean + - Use when: you want to enable lockdown mode for the scrape. + +- `redact_pii` + - Type: boolean + - Use when: you want to redact personally identifiable information from the output. + +- `audit_metadata` + - Type: keyword list with `username:` (string, required) + - Use when: you need to attach audit metadata to the scrape request. + - `profile` - Type: keyword list with `name:` and optional `save_changes:` (or `saveChanges:`) - Use when: you want a persistent browser profile shared across scrapes and interactions. @@ -373,6 +410,6 @@ Use interact when a page requires browser actions or code execution after a scra ## Source Of Truth -- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/mix.exs` (version 1.9.1) - `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` - `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-source-of-truth/java.mdx b/agent-source-of-truth/java.mdx index 93da68332..a00f3ee1f 100644 --- a/agent-source-of-truth/java.mdx +++ b/agent-source-of-truth/java.mdx @@ -13,14 +13,14 @@ Maven: com.firecrawl firecrawl-java - 1.2.0 + 1.12.1 ``` Gradle: ```gradle -implementation("com.firecrawl:firecrawl-java:1.2.0") +implementation("com.firecrawl:firecrawl-java:1.12.1") ``` ## Authenticate @@ -83,6 +83,8 @@ SearchOptions options = SearchOptions.builder() .location("San Francisco,California,United States") .ignoreInvalidURLs(true) .timeout(60000) + .highlights(true) + .includeDomains(List.of("docs.firecrawl.dev")) .scrapeOptions( ScrapeOptions.builder() .formats(List.of( @@ -124,6 +126,14 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - `"pdf"`: PDF-focused results - `{type: "github" | "research" | "pdf"}`: typed category map form +- `options.includeDomains` + - Type: `List` + - Use when: you want to restrict results to specific domains. + +- `options.excludeDomains` + - Type: `List` + - Use when: you want to exclude specific domains from results. + - `options.limit` - Type: Integer - Use when: you want to cap results. @@ -144,6 +154,10 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - Type: Integer - Use when: you need a request timeout in milliseconds. +- `options.highlights` + - Type: Boolean + - Use when: you want highlighted snippets in results. Defaults to true. + - `options.scrapeOptions` - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). @@ -181,6 +195,7 @@ Document doc = client.scrape( ```java import com.firecrawl.models.ScrapeOptions; import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.AuditMetadata; List> actions = List.of( Map.of("type", "click", "selector", "#accept"), @@ -210,6 +225,9 @@ ScrapeOptions options = ScrapeOptions.builder() .proxy("auto") .maxAge(86400000L) .storeInCache(true) + .lockdown(true) + .redactPII(true) + .auditMetadata(AuditMetadata.builder().username("analyst").build()) .build(); Document doc = client.scrape("https://example.com/pricing", options); @@ -236,11 +254,15 @@ Document doc = client.scrape("https://example.com/pricing", options); - `"json"`: JSON extraction - `"attributes"`: attribute extraction - `"branding"`: branding profile output + - `"product"`: product data extraction + - `"menu"`: menu/navigation extraction - `"audio"`: audio extraction - `"video"`: video extraction - Format object fields: - `type`: one of the format strings above - `prompt`, `schema`: JSON extraction options for `type: "json"` + - `question`: question extraction for `type: "question"` (QuestionFormat) + - `query`: highlights extraction for `type: "highlights"` (HighlightsFormat) - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` - `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"` - `selectors`: array of `{selector, attribute}` for `type: "attributes"` @@ -323,6 +345,19 @@ Document doc = client.scrape("https://example.com/pricing", options); - Type: Boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: Boolean + - Use when: you want to enable lockdown mode on the scrape. + +- `options.redactPII` + - Type: Boolean + - Use when: you want personally identifiable information redacted from the output. + +- `options.auditMetadata` + - Type: `AuditMetadata` + - Use when: you need to attach audit metadata to the request. + - Fields: `username` (String) + - `options.integration` - Type: String - Use when: the API expects an integration identifier on the request. @@ -336,7 +371,7 @@ Use interact when a page requires browser actions or code execution after a scra ### Preferred SDK methods - `client.interact(jobId, code)` — uses default language `node` and API default execution timeout -- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), or null to omit and use the API default (30 seconds) +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1-300), or null to omit and use the API default (30 seconds) - `client.interact(jobId, code, language, timeout, origin)` — optional `origin` string is sent only when non-null (request attribution) ### Simple Example @@ -407,7 +442,7 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `timeout` - Type: Integer - - Use when: you need an execution timeout in seconds (1–300). Null omits the field and uses the API default. + - Use when: you need an execution timeout in seconds (1-300). Null omits the field and uses the API default. - `origin` - Type: String @@ -415,7 +450,8 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); ## Notes -- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers). +- Deprecated aliases: `scrapeExecute` -> `interact`, `deleteScrapeBrowser` -> `stopInteractiveBrowser` (and the corresponding `*Async` helpers). +- Deprecated format type: `QueryFormat` — use `QuestionFormat` (for question-based extraction) or `HighlightsFormat` (for query-based highlights) instead. - The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs). ## Source Of Truth @@ -426,6 +462,9 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/JsonFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/QuestionFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/HighlightsFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/AuditMetadata.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/LocationConfig.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserExecuteResponse.java` diff --git a/agent-source-of-truth/node.mdx b/agent-source-of-truth/node.mdx index 3d261eb0b..20a251e99 100644 --- a/agent-source-of-truth/node.mdx +++ b/agent-source-of-truth/node.mdx @@ -3,7 +3,7 @@ title: "Node.js Source of Truth" description: "Canonical Firecrawl Node.js source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.18.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. +Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.32.0** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. ## Install @@ -51,12 +51,15 @@ const results = await client.search("site:docs.firecrawl.dev webhook retries"); ```ts const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { sources: ["web", "news"], - categories: ["research"], + categories: ["research", "developer"], limit: 10, tbs: "qdr:m", location: "San Francisco,California,United States", + includeDomains: ["docs.firecrawl.dev"], ignoreInvalidURLs: true, + highlights: true, timeout: 60000, + enterprise: ["default"], scrapeOptions: { formats: [ "markdown", @@ -78,8 +81,9 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `web`: web index hits - `news`: news hits - `images`: image hits +- `developer`: developer-focused hits -**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`. +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`, developer results in `result.developer`. ### Parameters @@ -107,7 +111,18 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `"github"`: GitHub-focused results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `{ type: "github" | "research" | "pdf" }`: typed category object form + - `"developer"`: developer-focused results + - `{ type: "github" | "research" | "pdf" | "developer" }`: typed category object form + +- `options.includeDomains` + - Type: array of strings + - Use when: you want to restrict results to specific domains. + - Notes: cannot be used together with `excludeDomains`. + +- `options.excludeDomains` + - Type: array of strings + - Use when: you want to exclude specific domains from results. + - Notes: cannot be used together with `includeDomains`. - `options.limit` - Type: number @@ -125,10 +140,31 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - Type: boolean - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `options.highlights` + - Type: boolean + - Use when: you want highlighted snippets in search results. + - Notes: defaults to `true`. + - `options.timeout` - Type: number - Use when: you need a request timeout in milliseconds. +- `options.enterprise` + - Type: array of strings + - Use when: you need enterprise-grade search capabilities. + - Confirmed values: `"default"`, `"anon"`, `"zdr"` + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you want threat-level filtering on search results. + - Fields: + - `mode`: `"off"` or `"normal"` + - `riskScoreThreshold`: number + - `blacklist`: array of strings + - `whitelist`: array of strings + - `blockedTlds`: array of strings + - `failurePolicy`: `"open"` or `"closed"` + - `options.scrapeOptions` - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). The SDK runs the same validation as for `scrape` (for example plain string `"json"` in `formats` is rejected). @@ -158,6 +194,7 @@ const doc = await client.scrape("https://example.com/pricing", { formats: [ "markdown", "links", + "product", { type: "json", prompt: "Extract plan names and prices." }, { type: "screenshot", fullPage: true, quality: 80, viewport: { width: 1280, height: 720 } }, { type: "changeTracking", modes: ["git-diff"], tag: "pricing" }, @@ -182,7 +219,12 @@ const doc = await client.scrape("https://example.com/pricing", { maxAge: 86400000, minAge: 1, storeInCache: true, - profile: { name: "docs-session", saveChanges: true } + lockdown: true, + redactPII: { mode: "accurate", entities: ["EMAIL", "PHONE"], replaceStyle: "tag" }, + threatProtection: { mode: "normal", riskScoreThreshold: 50 }, + auditMetadata: { username: "bot-user" }, + profile: { name: "docs-session", saveChanges: true }, + integration: "custom-integration" }); ``` @@ -206,6 +248,8 @@ const doc = await client.scrape("https://example.com/pricing", { - `"changeTracking"`: change tracking output (for options like `modes`, use `{ type: "changeTracking", modes: [...] }` — `modes` is required on that object in typings) - `"attributes"`: attribute extraction (use `{ type: "attributes", selectors: [...] }` when passing selectors) - `"branding"`: branding profile output + - `"product"`: product data extraction + - `"menu"`: menu/navigation extraction - `"audio"`: audio extraction - `"video"`: video extraction - Object-only format types (at minimum `type` as shown): @@ -312,10 +356,44 @@ const doc = await client.scrape("https://example.com/pricing", { - Type: boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: boolean + - Use when: you want to restrict the scrape to the exact URL without following redirects or loading external resources. + +- `options.redactPII` + - Type: boolean or `RedactPIIOptions` + - Use when: you want to redact personally identifiable information from the output. + - Pass `true` for default redaction, or an options object for fine-grained control. + - `RedactPIIOptions` fields: + - `mode`: `"accurate"`, `"aggressive"`, or `"fast"` + - `entities`: array of entity types to redact — `"PERSON"`, `"EMAIL"`, `"PHONE"`, `"LOCATION"`, `"FINANCIAL"`, `"SECRET"` + - `replaceStyle`: `"tag"`, `"mask"`, or `"remove"` + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` + - Use when: you want threat-level filtering before scraping a URL. + - Fields: + - `mode`: `"off"` or `"normal"` + - `riskScoreThreshold`: number + - `blacklist`: array of strings + - `whitelist`: array of strings + - `blockedTlds`: array of strings + - `failurePolicy`: `"open"` or `"closed"` + +- `options.auditMetadata` + - Type: `AuditMetadata` + - Use when: you want to attach audit information to the scrape request. + - Fields: + - `username`: string (required) + - `options.profile` - Type: object with `name` and optional `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `options.integration` + - Type: string + - Use when: you want to tag the scrape with a specific integration identifier. + ## Interact ### Why use it @@ -379,6 +457,7 @@ const result = await client.interact("", { `ScrapeExecuteResponse` matches `BrowserExecuteResponse`. Confirmed fields include: - `success`: boolean +- `cdpUrl`: string - `output`: string (aggregated output when present) - `stdout`: string - `result`: string (alias-style field alongside stdout in typings) @@ -526,7 +605,7 @@ console.log(result.answer); ## Notes -- Deprecated client aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- Deprecated client aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`; `crawlUrl` → `crawl`; `mapUrl` → `map`. - The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. - Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. - The package declares **Node.js >= 22** in `engines`. diff --git a/agent-source-of-truth/python.mdx b/agent-source-of-truth/python.mdx index 41e594452..946e0a75d 100644 --- a/agent-source-of-truth/python.mdx +++ b/agent-source-of-truth/python.mdx @@ -3,7 +3,7 @@ title: "Python Source of Truth" description: "Canonical Firecrawl Python source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.22.1**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. +Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.34.0**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. ## Install @@ -46,10 +46,11 @@ Returns a `SearchData` model with optional lists: - `web` — web hits (`SearchResultWeb` or full `Document` when `scrape_options` hydrates content) - `news` — news hits (`SearchResultNews` or `Document`) - `images` — image hits (`SearchResultImages` or `Document`) +- `developer` — developer hits (`SearchResultWeb` or `Document`) Omitted buckets are `None` when the API did not return that key. -**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`. +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`, developer results in `result.developer`. ### Simple Example @@ -67,11 +68,13 @@ from firecrawl.v2.types import ScrapeOptions results = client.search( "site:docs.firecrawl.dev crawl webhooks", sources=["web", "news"], - categories=["research"], + categories=["research", "developer"], + include_domains=["docs.firecrawl.dev"], limit=10, tbs="qdr:m", location="San Francisco,California,United States", ignore_invalid_urls=True, + highlights=True, timeout=300000, scrape_options=ScrapeOptions( formats=[ @@ -115,7 +118,18 @@ results = client.search( - `"github"`: GitHub-focused results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `Category(type="github" | "research" | "pdf")`: typed category object form + - `"developer"`: developer-focused results + - `Category(type="github" | "research" | "pdf" | "developer")`: typed category object form + +- `include_domains` + - Type: list of str + - Use when: you want to restrict results to specific domains. + - Notes: mutually exclusive with `exclude_domains`. + +- `exclude_domains` + - Type: list of str + - Use when: you want to exclude results from specific domains. + - Notes: mutually exclusive with `include_domains`. - `limit` - Type: int @@ -134,6 +148,10 @@ results = client.search( - Type: bool - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `highlights` + - Type: bool + - Use when: you want highlighted snippets in results. + - `timeout` - Type: int - Use when: you need a request timeout in milliseconds. @@ -143,6 +161,20 @@ results = client.search( - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `integration` + - Type: str + - Use when: you need to specify an integration identifier. + +- `enterprise` + - Type: list of str + - Use when: you need enterprise features. + - Confirmed values: `"zdr"`, `"anon"` + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls. + - Fields: `mode` (`"off"` or `"normal"`), `risk_score_threshold` (0-100), `blacklist`, `whitelist`, `blocked_tlds`, `failure_policy` (`"open"` or `"closed"`) + ## Scrape ### Why use it @@ -167,6 +199,7 @@ doc = client.scrape( formats=[ "markdown", "links", + "product", {"type": "json", "prompt": "Extract plan names and prices."}, {"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}, {"type": "changeTracking", "modes": ["git-diff"], "tag": "pricing"}, @@ -188,7 +221,10 @@ doc = client.scrape( proxy="auto", max_age=86400000, store_in_cache=True, + lockdown=True, profile={"name": "docs-session", "save_changes": True}, + audit_metadata={"username": "bot-user"}, + threat_protection={"mode": "normal", "risk_score_threshold": 50}, ) ``` @@ -212,6 +248,8 @@ doc = client.scrape( - `"changeTracking"` or `"change_tracking"`: change tracking output - `"attributes"`: attribute extraction - `"branding"`: branding profile output + - `"product"`: product data extraction + - `"menu"`: menu/navigation extraction - `"audio"`: audio extraction - `"video"`: video extraction - Object-only format types: @@ -316,10 +354,28 @@ doc = client.scrape( - Type: bool - Use when: you want Firecrawl to cache the result. +- `lockdown` + - Type: bool + - Use when: you want to enable lockdown mode for the scrape. + - `profile` - Type: dict with `name` and optional `save_changes` or `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `audit_metadata` + - Type: `AuditMetadata` or dict with `username` + - Use when: you want to attach audit metadata (such as an end-user identifier) to the scrape. + - Fields: `username` (str) + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need threat protection controls. + - Fields: `mode` (`"off"` or `"normal"`), `risk_score_threshold` (0-100), `blacklist`, `whitelist`, `blocked_tlds`, `failure_policy` (`"open"` or `"closed"`) + +- `integration` + - Type: str + - Use when: you need to specify an integration identifier. + ## Interact ### Why use it @@ -507,9 +563,10 @@ print(response.json()["answer"]) ## Notes -- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`; `crawl_url` → `crawl`; `map_url` → `map`. - The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. - The bundled v2 OpenAPI snippet for `POST /v2/scrape/{jobId}/interact` may only document `code`; the Python SDK and server accept either `code` or `prompt` for this endpoint. +- `RedactPIIOptions`: `mode` (`"accurate"`, `"aggressive"`, `"fast"`), `entities` (list of `"PERSON"`, `"EMAIL"`, `"PHONE"`, `"LOCATION"`, `"FINANCIAL"`, `"SECRET"`), `replace_style` (`"tag"`, `"mask"`, `"remove"`). ## Source Of Truth diff --git a/agent-source-of-truth/rust.mdx b/agent-source-of-truth/rust.mdx index a64c654ab..4f58b623c 100644 --- a/agent-source-of-truth/rust.mdx +++ b/agent-source-of-truth/rust.mdx @@ -11,7 +11,7 @@ Canonical Firecrawl Rust source of truth for agents. Generated from SDK source a cargo add firecrawl ``` -Crate: **`firecrawl`** on crates.io. The current SDK version is **2.0.0** (verify the latest release on crates.io before pinning). +Crate: **`firecrawl`** on crates.io. The current SDK version is **2.12.1** (verify the latest release on crates.io before pinning). ## Authenticate @@ -64,7 +64,10 @@ let options = SearchOptions { limit: Some(10), tbs: Some("qdr:m".to_string()), location: Some("San Francisco,California,United States".to_string()), + include_domains: Some(vec!["docs.firecrawl.dev".to_string()]), + exclude_domains: Some(vec!["old.firecrawl.dev".to_string()]), ignore_invalid_urls: Some(true), + highlights: Some(true), timeout: Some(60000), scrape_options: Some(ScrapeOptions { formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), @@ -125,10 +128,22 @@ let results = client - Type: String - Use when: you want localized results. +- `options.include_domains` + - Type: `Vec` + - Use when: you want to restrict results to specific domains. + +- `options.exclude_domains` + - Type: `Vec` + - Use when: you want to exclude results from specific domains. + - `options.ignore_invalid_urls` - Type: bool - Use when: you want to drop URLs that cannot be scraped by other endpoints. +- `options.highlights` + - Type: bool + - Use when: you want highlighted snippets in search results. + - `options.timeout` - Type: u32 - Use when: you need a request timeout in milliseconds. @@ -170,7 +185,8 @@ let doc = client ```rust use firecrawl::{ Client, ScrapeOptions, Format, JsonOptions, ScreenshotOptions, ChangeTrackingOptions, - ChangeTrackingMode, AttributeSelector, Action, ParserConfig, ProxyType, + ChangeTrackingMode, AttributeSelector, Action, ParserConfig, ProxyType, AuditMetadata, + QuestionFormat, HighlightsFormat, }; let doc = client @@ -182,6 +198,14 @@ let doc = client Format::Screenshot, Format::ChangeTracking, Format::Attributes, + Format::Product, + Format::Menu, + Format::Question(QuestionFormat { + question: "What are the pricing tiers?".to_string(), + }), + Format::Highlights(HighlightsFormat { + query: "enterprise plan".to_string(), + }), ]), json_options: Some(JsonOptions { prompt: Some("Extract plan names and prices.".to_string()), @@ -211,6 +235,11 @@ let doc = client Action::Scrape, ]), proxy: Some(ProxyType::Auto), + lockdown: Some(true), + redact_pii: Some(true), + audit_metadata: Some(AuditMetadata { + username: "agent-user".to_string(), + }), ..Default::default() }) .await?; @@ -225,7 +254,11 @@ let doc = client - `options.formats` - Type: `Vec` - Use when: you want multiple output formats. - - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video` + - Confirmed simple values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video` + - Confirmed object variants: + - `Question(QuestionFormat)` — serialized as `{ "type": "question", "question": "..." }` + - `Highlights(HighlightsFormat)` — serialized as `{ "type": "highlights", "query": "..." }` + - Deprecated: `Query(QueryFormat)` — use `Question` or `Highlights` instead. - `options.headers` - Type: `HashMap` @@ -314,6 +347,20 @@ let doc = client - Type: bool - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: bool + - Use when: you want to enable lockdown mode for the scrape. + +- `options.redact_pii` + - Type: bool + - Use when: you want personally identifiable information redacted from the output. + - Notes: serialized as `redactPII` in the JSON payload. + +- `options.audit_metadata` + - Type: `AuditMetadata` + - Use when: you want to attach audit metadata to the scrape request. + - Confirmed fields: `username` + - `options.profile` - Type: `ProfileConfig` - Use when: you want a persistent browser profile shared across scrapes and interactions. @@ -421,11 +468,6 @@ let stopped = client.stop_interaction("").await?; - Type: u32 - Use when: you need an execution timeout in seconds. -- `options.origin` - - Type: `Option` - - Use when: you need an optional origin label for execution telemetry. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. - At least one of `options.code` or `options.prompt` must be non-empty; otherwise the SDK returns `FirecrawlError::Misuse` before calling the API. ### Response types @@ -440,6 +482,7 @@ The v2 OpenAPI spec currently models the interact request body with `code` as re ## Notes - Deprecated aliases: `scrape_execute`, `stop_interactive_browser`, and `delete_scrape_browser` map to `interact` and `stop_interaction`. +- Deprecated format: `Format::Query(QueryFormat)` — use `Format::Question(QuestionFormat)` or `Format::Highlights(HighlightsFormat)` instead. - `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. - `search_and_scrape(query, limit)` is a convenience helper: it calls `search` with default `ScrapeOptions` and returns `Vec` built from `SearchResultOrDocument::Document` entries in `data.web` (see `search.rs`). - v2 SDK exports all types at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`).