diff --git a/agent-source-of-truth/elixir.mdx b/agent-source-of-truth/elixir.mdx index 703f6d6b2..e3dc123b6 100644 --- a/agent-source-of-truth/elixir.mdx +++ b/agent-source-of-truth/elixir.mdx @@ -10,7 +10,7 @@ Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source Add to `mix.exs`: ```elixir -{:firecrawl, "~> 1.0.0"} +{:firecrawl, "~> 1.9"} ``` ## Authenticate @@ -23,6 +23,14 @@ config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") {:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key") ``` +All functions accept an optional trailing keyword list (`opts`) that supports: + +- `:api_key` — override the API key for this call. +- `:base_url` — override the default base URL (`https://api.firecrawl.dev/v2`), useful for self-hosted instances. +- Any other keys are passed through to the underlying Req HTTP client (for example, `:receive_timeout`, `:retry`). + +A nil or empty API key is allowed: `scrape`, `search`, and `interact` fall back to a keyless free tier that is rate-limited per IP. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -58,6 +66,9 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or country: "US", ignore_invalid_urls: true, timeout: 60000, + include_domains: ["docs.firecrawl.dev", "firecrawl.dev"], + exclude_domains: ["example.com"], + highlights: true, scrape_options: [ formats: [ "markdown", @@ -121,6 +132,18 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or - Type: integer - Use when: you need a request timeout in milliseconds. +- `exclude_domains` + - Type: list of strings + - Use when: you want to exclude specific domains from search results. + +- `include_domains` + - Type: list of strings + - Use when: you want to restrict search results to specific domains. + +- `highlights` + - Type: boolean + - Use when: you want query-relevant highlights returned with each result. + - `enterprise` - Type: list of strings - Use when: you need enterprise search controls. @@ -180,7 +203,10 @@ Use scrape when you already have a URL and want structured content in one or mor min_age: 1, store_in_cache: true, profile: [name: "docs-session", save_changes: true], - zero_data_retention: false + zero_data_retention: false, + lockdown: false, + redact_pii: true, + audit_metadata: [username: "admin@example.com"] ) ``` @@ -302,6 +328,19 @@ Use scrape when you already have a URL and want structured content in one or mor - Type: boolean - Use when: you want zero data retention for this scrape. +- `audit_metadata` + - Type: keyword list (keys: `username: :string`, required) + - Use when: you need SIEM logging attribution. + +- `lockdown` + - Type: boolean + - Use when: you want to serve only cached results (no live fetch). + +- `redact_pii` + - Type: boolean + - Use when: you want PII redaction applied to the scraped content. + - JSON key: `redactPII` + ## Interact ### Why use it @@ -367,8 +406,11 @@ Use interact when a page requires browser actions or code execution after a scra ## Notes -- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec. +- The Elixir client is auto-generated from the OpenAPI spec (the line 1 comment in the source confirms this). - Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- Parameters are validated at call time using NimbleOptions. +- Snake_case Elixir keys are automatically converted to camelCase JSON keys. +- The SDK auto-injects `"origin": "elixir-sdk@1.9.1"` in every request body. - This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`). ## Source Of Truth diff --git a/agent-source-of-truth/java.mdx b/agent-source-of-truth/java.mdx index 93da68332..d8ad3c97e 100644 --- a/agent-source-of-truth/java.mdx +++ b/agent-source-of-truth/java.mdx @@ -13,14 +13,14 @@ Maven: com.firecrawl firecrawl-java - 1.2.0 + 1.12.1 ``` Gradle: ```gradle -implementation("com.firecrawl:firecrawl-java:1.2.0") +implementation("com.firecrawl:firecrawl-java:1.12.1") ``` ## Authenticate @@ -33,6 +33,24 @@ FirecrawlClient client = FirecrawlClient.builder() .build(); ``` +### Builder options + +- `apiKey(String)` — API key for authentication. Falls back to `FIRECRAWL_API_KEY` env var, then `firecrawl.apiKey` system property. Null is allowed (keyless free tier). +- `apiUrl(String)` — Base URL for the Firecrawl API. Default `"https://api.firecrawl.dev"`, falls back to `FIRECRAWL_API_URL` env var. +- `timeoutMs(long)` — HTTP request timeout in milliseconds (default 300000 / 5 min). +- `maxRetries(int)` — Auto-retry count for transient failures (default 3). +- `backoffFactor(double)` — Exponential backoff factor in seconds (default 0.5). +- `asyncExecutor(Executor)` — Executor for async methods (default `ForkJoinPool.commonPool()`). +- `httpClient(OkHttpClient)` — Pre-configured OkHttp client (ignores `timeoutMs` when provided). + +### Factory method + +`FirecrawlClient.fromEnv()` — Creates a client using environment-based configuration (`FIRECRAWL_API_KEY`, `FIRECRAWL_API_URL`). Equivalent to `FirecrawlClient.builder().build()` with no explicit arguments. + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -83,6 +101,9 @@ SearchOptions options = SearchOptions.builder() .location("San Francisco,California,United States") .ignoreInvalidURLs(true) .timeout(60000) + .includeDomains(List.of("docs.firecrawl.dev", "firecrawl.dev")) + .excludeDomains(List.of("example.com")) + .highlights(true) .scrapeOptions( ScrapeOptions.builder() .formats(List.of( @@ -144,6 +165,18 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt - Type: Integer - Use when: you need a request timeout in milliseconds. +- `options.includeDomains` + - Type: `List` + - Use when: you want to restrict results to specific domains. + +- `options.excludeDomains` + - Type: `List` + - Use when: you want to exclude specific domains. + +- `options.highlights` + - Type: Boolean + - Use when: you want query-relevant highlights in the results (defaults to true). + - `options.scrapeOptions` - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). @@ -181,6 +214,9 @@ Document doc = client.scrape( ```java import com.firecrawl.models.ScrapeOptions; import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.HighlightsFormat; +import com.firecrawl.models.QuestionFormat; +import com.firecrawl.models.AuditMetadata; List> actions = List.of( Map.of("type", "click", "selector", "#accept"), @@ -198,6 +234,8 @@ ScrapeOptions options = ScrapeOptions.builder() "markdown", "links", JsonFormat.builder().prompt("Extract plan names and prices.").build(), + QuestionFormat.builder().question("What are the pricing tiers?").build(), + HighlightsFormat.builder().query("pricing").build(), Map.of("type", "screenshot", "fullPage", true, "quality", 80) )) .onlyMainContent(true) @@ -210,6 +248,9 @@ ScrapeOptions options = ScrapeOptions.builder() .proxy("auto") .maxAge(86400000L) .storeInCache(true) + .lockdown(true) + .redactPII(true) + .auditMetadata(AuditMetadata.builder().username("agent-user").build()) .build(); Document doc = client.scrape("https://example.com/pricing", options); @@ -244,6 +285,9 @@ Document doc = client.scrape("https://example.com/pricing", options); - `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"` - `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"` - `selectors`: array of `{selector, attribute}` for `type: "attributes"` + - Format object types: + - `QuestionFormat.builder().question("...").build()` — question-answer extraction + - `HighlightsFormat.builder().query("...").build()` — relevant source-text extraction - `options.headers` - Type: `Map` @@ -323,6 +367,18 @@ Document doc = client.scrape("https://example.com/pricing", options); - Type: Boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: Boolean + - Use when: you want to serve only cached results (no live fetch). + +- `options.redactPII` + - Type: Boolean + - Use when: you want PII redaction applied to the scraped content. + +- `options.auditMetadata` + - Type: `AuditMetadata` (has `username: String`) + - Use when: you need SIEM logging attribution for the scrape request. + - `options.integration` - Type: String - Use when: the API expects an integration identifier on the request. @@ -336,9 +392,11 @@ Use interact when a page requires browser actions or code execution after a scra ### Preferred SDK methods - `client.interact(jobId, code)` — uses default language `node` and API default execution timeout -- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), or null to omit and use the API default (30 seconds) +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1-300), or null to omit and use the API default (30 seconds) - `client.interact(jobId, code, language, timeout, origin)` — optional `origin` string is sent only when non-null (request attribution) +Async variants: each overload has a corresponding `interactAsync(...)` that returns `CompletableFuture`. + ### Simple Example ```java @@ -369,6 +427,8 @@ End the scrape-bound browser session when finished. **Preferred SDK method:** `client.stopInteractiveBrowser(jobId)` +Async variant: `client.stopInteractiveBrowserAsync(jobId)` returns `CompletableFuture`. + ```java import com.firecrawl.models.BrowserDeleteResponse; @@ -380,7 +440,7 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `isSuccess()`: boolean - `getStdout()`, `getStderr()`, `getResult()`, `getError()`: String (may be null) - `getExitCode()`: Integer (may be null) -- `getKilled()`: Boolean (may be null) — true when execution was stopped due to timeout +- `getKilled()`: Boolean (may be null) -- true when execution was stopped due to timeout ### Stop response (`BrowserDeleteResponse`) @@ -407,7 +467,7 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `timeout` - Type: Integer - - Use when: you need an execution timeout in seconds (1–300). Null omits the field and uses the API default. + - Use when: you need an execution timeout in seconds (1-300). Null omits the field and uses the API default. - `origin` - Type: String @@ -415,8 +475,9 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); ## Notes -- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers). +- Deprecated aliases: `scrapeExecute` -> `interact`, `deleteScrapeBrowser` -> `stopInteractiveBrowser` (and the corresponding `*Async` helpers). - The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs). +- Builder supports `fromEnv()` for environment-based configuration without explicit arguments. ## Source Of Truth @@ -430,4 +491,7 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser(""); - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserExecuteResponse.java` - `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/BrowserDeleteResponse.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/AuditMetadata.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/HighlightsFormat.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/QuestionFormat.java` - `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-source-of-truth/node.mdx b/agent-source-of-truth/node.mdx index 3d261eb0b..624fd1048 100644 --- a/agent-source-of-truth/node.mdx +++ b/agent-source-of-truth/node.mdx @@ -3,7 +3,7 @@ title: "Node.js Source of Truth" description: "Canonical Firecrawl Node.js source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.18.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. +Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.32.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. ## Install @@ -22,6 +22,34 @@ const client = new Firecrawl({ }); ``` +The constructor also accepts a plain string, which is treated as the API key: + +```ts +const client = new Firecrawl("fc-..."); +``` + +### Constructor options (`FirecrawlClientOptions`) + +- `apiKey` + - Type: `string | null` + - Falls back to the `FIRECRAWL_API_KEY` environment variable. If no key is provided the client uses the keyless free tier (rate-limited per IP). + +- `apiUrl` + - Type: `string | null` + - Falls back to the `FIRECRAWL_API_URL` environment variable, then `"https://api.firecrawl.dev"`. + +- `timeoutMs` + - Type: `number` + - Per-request timeout in milliseconds. + +- `maxRetries` + - Type: `number` + - Maximum number of automatic retries for transient failures. + +- `backoffFactor` + - Type: `number` + - Exponential backoff factor for retries. + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -57,6 +85,8 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { location: "San Francisco,California,United States", ignoreInvalidURLs: true, timeout: 60000, + includeDomains: ["docs.firecrawl.dev"], + highlights: true, scrapeOptions: { formats: [ "markdown", @@ -78,8 +108,9 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `web`: web index hits - `news`: news hits - `images`: image hits +- `developer`: developer-focused results (`Array`) -**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`. +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`, developer results in `result.developer`. ### Parameters @@ -107,7 +138,18 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - `"github"`: GitHub-focused results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `{ type: "github" | "research" | "pdf" }`: typed category object form + - `"developer"`: developer-focused results (issues, pull requests, READMEs, documentation) — results appear under `.developer` + - `{ type: "github" | "research" | "pdf" | "developer" }`: typed category object form + +- `options.includeDomains` + - Type: string[] + - Use when: you want to restrict results to specific domains. + - Notes: cannot be used together with `excludeDomains`. + +- `options.excludeDomains` + - Type: string[] + - Use when: you want to exclude specific domains from results. + - Notes: cannot be used together with `includeDomains`. - `options.limit` - Type: number @@ -129,6 +171,22 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", { - Type: number - Use when: you need a request timeout in milliseconds. +- `options.highlights` + - Type: boolean + - Use when: you want query-relevant highlights for search results. Defaults to true (server-side). + +- `options.enterprise` + - Type: array of `"default" | "anon" | "zdr"` + - Use when: you need enterprise features. Use `["zdr"]` for end-to-end Zero Data Retention or `["anon"]` for anonymized search. Must be enabled for your team. + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` (`{ mode?, riskScoreThreshold?, blacklist?, whitelist?, blockedTlds?, failurePolicy? }`) + - Use when: you need per-request threat protection override. + +- `options.integration` + - Type: string + - Use when: you need an integration identifier for server-side tracking. + - `options.scrapeOptions` - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). The SDK runs the same validation as for `scrape` (for example plain string `"json"` in `formats` is rejected). @@ -158,7 +216,10 @@ const doc = await client.scrape("https://example.com/pricing", { formats: [ "markdown", "links", + "product", { type: "json", prompt: "Extract plan names and prices." }, + { type: "question", question: "What is the enterprise plan price?" }, + { type: "highlights", query: "pricing tiers" }, { type: "screenshot", fullPage: true, quality: 80, viewport: { width: 1280, height: 720 } }, { type: "changeTracking", modes: ["git-diff"], tag: "pricing" }, { type: "attributes", selectors: [{ selector: "a", attribute: "href" }] } @@ -182,6 +243,7 @@ const doc = await client.scrape("https://example.com/pricing", { maxAge: 86400000, minAge: 1, storeInCache: true, + redactPII: { mode: "accurate", entities: ["EMAIL", "PHONE"], replaceStyle: "tag" }, profile: { name: "docs-session", saveChanges: true } }); ``` @@ -206,12 +268,15 @@ const doc = await client.scrape("https://example.com/pricing", { - `"changeTracking"`: change tracking output (for options like `modes`, use `{ type: "changeTracking", modes: [...] }` — `modes` is required on that object in typings) - `"attributes"`: attribute extraction (use `{ type: "attributes", selectors: [...] }` when passing selectors) - `"branding"`: branding profile output + - `"product"`: structured product data extraction (returns a `ProductProfile`) + - `"menu"`: structured menu data extraction (returns a `MenuProfile`) - `"audio"`: audio extraction - `"video"`: video extraction - Object-only format types (at minimum `type` as shown): - `{ type: "json", prompt?: string, schema?: JSON schema or Zod schema }`: at least one of `prompt` or `schema` is required (SDK validation). - `{ type: "question", question: string }`: question-answer style extraction. - `{ type: "highlights", query: string }`: relevant source-text extraction. + - `{ type: "query", prompt: string, mode?: "freeform" | "directQuote" }`: **deprecated** — prefer `json` or `highlights`. - `{ type: "screenshot", fullPage?, quality?, viewport? }`: same options as the string form but as an object. - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }`: `modes` is required. - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` @@ -312,6 +377,34 @@ const doc = await client.scrape("https://example.com/pricing", { - Type: boolean - Use when: you want Firecrawl to cache the result. +- `options.lockdown` + - Type: boolean + - Use when: you want to serve only cached results, never making outbound requests. + +- `options.redactPII` + - Type: `boolean | RedactPIIOptions` + - Use when: you want PII redaction on the scraped content. + - `RedactPIIOptions` fields: + - `mode`: `"accurate"` (default, model-only), `"aggressive"` (model + Presidio + spaCy), or `"fast"` (Presidio only, no model call). + - `entities`: array of `"PERSON" | "EMAIL" | "PHONE" | "LOCATION" | "FINANCIAL" | "SECRET"`. Unset means all entities. + - `replaceStyle`: `"tag"` (default, `` placeholders), `"mask"` (`*` of equal length), or `"remove"` (drop span entirely). + +- `options.threatProtection` + - Type: `ThreatProtectionOptions` (`{ mode?, riskScoreThreshold?, blacklist?, whitelist?, blockedTlds?, failurePolicy? }`) + - Use when: you need per-request threat protection override. Requires threat protection to be enabled for your team. + +- `options.auditMetadata` + - Type: `{ username: string }` + - Use when: you need SIEM logging attribution. + +- `options.integration` + - Type: string + - Use when: you need an integration identifier for server-side tracking. + +- `options.useMock` + - Type: string + - Use when: you need mock data (internal/testing). + - `options.profile` - Type: object with `name` and optional `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. @@ -374,6 +467,10 @@ const result = await client.interact("", { - Type: number - Use when: you need an execution timeout in seconds. +- `args.origin` + - Type: string + - Use when: you need an optional origin label for telemetry. + ### Return value (`interact`) `ScrapeExecuteResponse` matches `BrowserExecuteResponse`. Confirmed fields include: @@ -527,6 +624,8 @@ console.log(result.answer); ## Notes - Deprecated client aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- `scrapeUrl` is a deprecated alias for `scrape`. +- The constructor also accepts `timeoutMs`, `maxRetries`, and `backoffFactor` for transport-level control. - The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. - Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. - The package declares **Node.js >= 22** in `engines`. @@ -537,7 +636,12 @@ console.log(result.answer); - `firecrawl/apps/js-sdk/firecrawl/src/index.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/watcher.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/utils/validation.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/utils/httpClient.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/search.ts` - `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/scrape.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/browser.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/research.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/methods/monitor.ts` - `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-source-of-truth/python.mdx b/agent-source-of-truth/python.mdx index 41e594452..e96df845d 100644 --- a/agent-source-of-truth/python.mdx +++ b/agent-source-of-truth/python.mdx @@ -3,7 +3,7 @@ title: "Python Source of Truth" description: "Canonical Firecrawl Python source of truth for agents using key endpoints like search, scrape, and interact." --- -Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.22.1**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. +Canonical Firecrawl Python source of truth for agents. Generated from SDK source (`firecrawl-py` / `firecrawl` **4.35.1**) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py` unless noted. ## Install @@ -21,6 +21,16 @@ client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) # client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") ``` +### Constructor options + +- `api_key` — `str`. Falls back to the `FIRECRAWL_API_KEY` environment variable. `None` uses the keyless free tier. +- `api_url` — `str`. Default `"https://api.firecrawl.dev"`. +- `timeout` — `float`. Default request timeout in seconds for all HTTP requests. +- `max_retries` — `int`. Maximum number of retries (default `3`). +- `backoff_factor` — `float`. Exponential backoff factor (default `0.5`). + +`FirecrawlApp` is a backward-compatible alias for `Firecrawl`. For async usage, use `AsyncFirecrawl` (or its alias `AsyncFirecrawlApp`). + ## When To Use What - `search`: use when you start with a query and need discovery. @@ -46,6 +56,7 @@ Returns a `SearchData` model with optional lists: - `web` — web hits (`SearchResultWeb` or full `Document` when `scrape_options` hydrates content) - `news` — news hits (`SearchResultNews` or `Document`) - `images` — image hits (`SearchResultImages` or `Document`) +- `developer` — developer hits (e.g. GitHub, Stack Overflow) Omitted buckets are `None` when the API did not return that key. @@ -73,6 +84,7 @@ results = client.search( location="San Francisco,California,United States", ignore_invalid_urls=True, timeout=300000, + include_domains=["docs.firecrawl.dev"], scrape_options=ScrapeOptions( formats=[ "markdown", @@ -115,7 +127,8 @@ results = client.search( - `"github"`: GitHub-focused results - `"research"`: research and academic results - `"pdf"`: PDF-focused results - - `Category(type="github" | "research" | "pdf")`: typed category object form + - `"developer"`: developer-focused results (e.g. GitHub, Stack Overflow) + - `Category(type="github" | "research" | "pdf" | "developer")`: typed category object form - `limit` - Type: int @@ -143,6 +156,29 @@ results = client.search( - Type: `ScrapeOptions` - Use when: you want to scrape each search result (see Scrape parameters for fields). +- `include_domains` + - Type: list of str + - Use when: you want to restrict results to specific domains. + - Notes: mutually exclusive with `exclude_domains`. + +- `exclude_domains` + - Type: list of str + - Use when: you want to exclude specific domains from results. + - Notes: mutually exclusive with `include_domains`. + +- `highlights` + - Type: bool + - Use when: you want query-relevant highlights in results. + - Notes: default is `true` on the server. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need per-request threat protection. + +- `integration` + - Type: str + - Use when: you need an integration identifier. + ## Scrape ### Why use it @@ -167,10 +203,13 @@ doc = client.scrape( formats=[ "markdown", "links", + "product", {"type": "json", "prompt": "Extract plan names and prices."}, {"type": "screenshot", "full_page": True, "quality": 80, "viewport": {"width": 1280, "height": 720}}, {"type": "changeTracking", "modes": ["git-diff"], "tag": "pricing"}, {"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}, + {"type": "question", "question": "What is the enterprise plan price?"}, + {"type": "highlights", "query": "pricing tiers"}, ], headers={"User-Agent": "FirecrawlDocsBot/1.0"}, only_main_content=True, @@ -214,15 +253,20 @@ doc = client.scrape( - `"branding"`: branding profile output - `"audio"`: audio extraction - `"video"`: video extraction + - `"product"`: structured product data extraction + - `"menu"`: structured menu data extraction - Object-only format types: - `{"type": "json", ...}`: JSON extraction. Use an object, not the plain string `"json"`. - - `{"type": "question", "question": "..."}`: question-answer output. - - `{"type": "highlights", "query": "..."}`: relevant source-text output. + - `{"type": "question", "question": "..."}`: question-answer style extraction. + - `{"type": "highlights", "query": "..."}`: relevant source-text extraction. + - Deprecated object format: + - `{"type": "query", "prompt": "...", "mode": "freeform" | "directQuote"}`: deprecated in favor of `"question"` and `"highlights"` formats. - Format object fields: - - `type`: one of the format strings above, or `"json"`, `"question"`, or `"highlights"` for object-only formats + - `type`: one of the format strings above, or `"json"`, `"question"`, `"highlights"`, or `"query"` for object-only formats - `question`: for `type: "question"` - `query`: for `type: "highlights"` - - `prompt`: optional for `type: "json"` + - `prompt`: optional for `type: "json"` or `type: "query"` + - `mode`: for `type: "query"` — `"freeform"` or `"directQuote"` (deprecated) - `schema`: JSON schema for `type: "json"` or for change tracking JSON mode - `modes`: array of `"git-diff"` or `"json"` for `type: "changeTracking"` - `tag`: change tracking tag for `type: "changeTracking"` @@ -320,6 +364,26 @@ doc = client.scrape( - Type: dict with `name` and optional `save_changes` or `saveChanges` - Use when: you want a persistent browser profile shared across scrapes and interactions. +- `lockdown` + - Type: bool + - Use when: you want to serve only cached results, never making outbound requests. + +- `threat_protection` + - Type: `ThreatProtectionOptions` + - Use when: you need per-request threat protection. + +- `audit_metadata` + - Type: `AuditMetadata` (has `username: str`) + - Use when: you need SIEM logging attribution. + +- `integration` + - Type: str + - Use when: you need an integration identifier. + +- `use_mock` + - Type: str + - Use when: you need mock data (internal/testing). + ## Interact ### Why use it @@ -382,6 +446,10 @@ result = client.interact( - Type: int - Use when: you need an execution timeout in seconds. +- `origin` + - Type: str + - Use when: you need an optional origin label for telemetry. + ### Return value Returns `BrowserExecuteResponse`: `success`, optional `live_view_url`, `interactive_live_view_url`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `error` (API camelCase is normalized to snake_case on the model). @@ -507,7 +575,10 @@ print(response.json()["answer"]) ## Notes -- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- `FirecrawlApp` is a backward-compatible alias for `Firecrawl`. +- `AsyncFirecrawl` (and its alias `AsyncFirecrawlApp`) provide async versions of all methods. +- The constructor accepts `timeout`, `max_retries`, and `backoff_factor` for controlling HTTP request behavior. - The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. - The bundled v2 OpenAPI snippet for `POST /v2/scrape/{jobId}/interact` may only document `code`; the Python SDK and server accept either `code` or `prompt` for this endpoint. diff --git a/agent-source-of-truth/rust.mdx b/agent-source-of-truth/rust.mdx index a64c654ab..529e0b3bc 100644 --- a/agent-source-of-truth/rust.mdx +++ b/agent-source-of-truth/rust.mdx @@ -11,7 +11,7 @@ Canonical Firecrawl Rust source of truth for agents. Generated from SDK source a cargo add firecrawl ``` -Crate: **`firecrawl`** on crates.io. The current SDK version is **2.0.0** (verify the latest release on crates.io before pinning). +Crate: **`firecrawl`** on crates.io. The current SDK version is **2.12.1** (verify the latest release on crates.io before pinning). ## Authenticate @@ -66,6 +66,9 @@ let options = SearchOptions { location: Some("San Francisco,California,United States".to_string()), ignore_invalid_urls: Some(true), timeout: Some(60000), + include_domains: Some(vec!["docs.firecrawl.dev".to_string()]), + exclude_domains: Some(vec!["old.firecrawl.dev".to_string()]), + highlights: Some(true), scrape_options: Some(ScrapeOptions { formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), json_options: Some(JsonOptions { @@ -142,6 +145,23 @@ let results = client - Use when: you need an integration identifier for server-side tracking. - Notes: omit in agent-oriented examples unless your product intentionally sets it. +- `options.include_domains` + - Type: `Option>` + - Use when: you want to restrict results to specific domains. + +- `options.exclude_domains` + - Type: `Option>` + - Use when: you want to exclude specific domains. + +- `options.highlights` + - Type: `Option` + - Use when: you want query-relevant highlights (defaults to true). + +- `options.origin` + - Type: `Option` + - Use when: you need an origin label. + - Notes: auto-set to `"rust-sdk@{version}"` if not provided. Omit in agent-oriented examples unless your product intentionally sets it. + ## Scrape ### Why use it @@ -171,6 +191,7 @@ let doc = client use firecrawl::{ Client, ScrapeOptions, Format, JsonOptions, ScreenshotOptions, ChangeTrackingOptions, ChangeTrackingMode, AttributeSelector, Action, ParserConfig, ProxyType, + QuestionFormat, HighlightsFormat, }; let doc = client @@ -182,6 +203,14 @@ let doc = client Format::Screenshot, Format::ChangeTracking, Format::Attributes, + Format::Product, + Format::Menu, + Format::Question(QuestionFormat { + question: "What are the pricing tiers?".to_string(), + }), + Format::Highlights(HighlightsFormat { + query: "pricing plans".to_string(), + }), ]), json_options: Some(JsonOptions { prompt: Some("Extract plan names and prices.".to_string()), @@ -225,7 +254,7 @@ let doc = client - `options.formats` - Type: `Vec` - Use when: you want multiple output formats. - - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video` + - Confirmed values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Audio`, `Video`, `Product`, `Menu`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`, `Query(QueryFormat)` (deprecated) - `options.headers` - Type: `HashMap` @@ -344,6 +373,46 @@ let doc = client - Use when: you want attribute extraction output. - Confirmed fields: `selector`, `attribute` +- `options.lockdown` + - Type: `Option` + - Use when: you want to serve only cached results. + +- `options.redact_pii` + - Type: `Option` + - Use when: you want PII redaction. + - Notes: serialized as `"redactPII"` in the API request. + +- `options.audit_metadata` + - Type: `Option` + - Use when: you need SIEM logging attribution. + - Notes: `AuditMetadata` has a single field `username: String`. + +- `options.origin` + - Type: `Option` + - Use when: you need an origin label. + - Notes: auto-set to `"rust-sdk@{version}"` if not provided. Omit in agent-oriented examples unless your product intentionally sets it. + +### Format enum details + +- `Format::Markdown` — markdown output +- `Format::Html` — cleaned HTML output +- `Format::RawHtml` — raw HTML output +- `Format::Links` — extracted links +- `Format::Images` — extracted images +- `Format::Screenshot` — page screenshot +- `Format::Summary` — summarized content +- `Format::ChangeTracking` — change tracking output +- `Format::Json` — structured JSON extraction +- `Format::Attributes` — attribute extraction +- `Format::Branding` — branding extraction +- `Format::Audio` — audio extraction +- `Format::Video` — video extraction +- `Format::Product` — product profile output +- `Format::Menu` — menu extraction output +- `Format::Question(QuestionFormat { question: String })` — question-answer output +- `Format::Highlights(HighlightsFormat { query: String })` — relevant source-text extraction +- `Format::Query(QueryFormat { prompt: String, mode: Option })` — deprecated; modes: `QueryFormatMode::Freeform`, `QueryFormatMode::DirectQuote` + ## Interact ### Why use it @@ -423,8 +492,8 @@ let stopped = client.stop_interaction("").await?; - `options.origin` - Type: `Option` - - Use when: you need an optional origin label for execution telemetry. - - Notes: omit in agent-oriented examples unless your product intentionally sets it. + - Use when: you need an origin label for execution telemetry. + - Notes: auto-set to SDK origin if not provided. Omit in agent-oriented examples unless your product intentionally sets it. At least one of `options.code` or `options.prompt` must be non-empty; otherwise the SDK returns `FirecrawlError::Misuse` before calling the API. @@ -442,19 +511,19 @@ The v2 OpenAPI spec currently models the interact request body with `code` as re - Deprecated aliases: `scrape_execute`, `stop_interactive_browser`, and `delete_scrape_browser` map to `interact` and `stop_interaction`. - `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced formats. - `search_and_scrape(query, limit)` is a convenience helper: it calls `search` with default `ScrapeOptions` and returns `Vec` built from `SearchResultOrDocument::Document` entries in `data.web` (see `search.rs`). -- v2 SDK exports all types at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). -- Error types simplified in v2: `CrawlJobFailed(String, CrawlStatus)` → `JobFailed(String)`, `Missuse` → `Misuse`. +- Exports are at crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). +- Error types: `Misuse` (formerly `Missuse`), `JobFailed(String)` (formerly `CrawlJobFailed(String, CrawlStatus)`). ## Source Of Truth - `firecrawl/apps/rust-sdk/Cargo.toml` - `firecrawl/apps/rust-sdk/src/lib.rs` - `firecrawl/apps/rust-sdk/src/client.rs` -- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` (includes interact) - `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` - `firecrawl/apps/rust-sdk/src/crawl.rs` - `firecrawl/apps/rust-sdk/src/map.rs` - `firecrawl/apps/rust-sdk/src/batch_scrape.rs` - `firecrawl/apps/rust-sdk/src/agent.rs` -- `firecrawl/apps/rust-sdk/src/types.rs` - `firecrawl-docs/api-reference/v2-openapi.json`