From 38e28c3b4bc0cce6dc565f0d4c47f543d57289ad Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 8 Aug 2026 13:20:01 +0000 Subject: [PATCH] docs: add agent quickstart pages for all SDK languages Add one canonical quickstart file per SDK language (Node.js, Python, Rust, Java, Elixir) covering search, scrape, and interact endpoints. Each file includes parameters verified against the latest SDK source, practical examples, and language-specific notes. Added navigation entries under "Build with AI > Agent Quickstarts" in docs.json. SDK versions verified: - JS/TS: firecrawl v4.32.0 - Python: firecrawl-py v4.34.0 - Rust: firecrawl v2.12.1 - Java: firecrawl-java v1.12.1 - Elixir: firecrawl v1.9.1 Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_0184qwsYH82Fp2PqeQdP84dp --- agent-quickstart/elixir.mdx | 165 +++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 188 ++++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 169 ++++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 162 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 183 +++++++++++++++++++++++++++++++++++ docs.json | 10 ++ 6 files changed, 877 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..ab7b02c19 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,165 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for external agents. Aligned with `:firecrawl` hex package **v1.9.1** (`firecrawl/apps/elixir-sdk`) and the v2 OpenAPI spec. The Elixir client is OpenAPI-generated; function names and parameter keys come directly from the spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.search_and_scrape([query: "example"], api_key: "fc-your-api-key") +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. You can constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape(query: "site:docs.firecrawl.dev webhook retries") +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` (required) | The search query. Use `site:example.com` to limit results to a domain. | +| `sources` | `list` | Sources to search. Values: `"web"`, `"news"`, `"images"`, or atoms `:web`, `:news`, `:images`. | +| `categories` | `list` | Filter by category. Values: `"github"`, `"research"`, `"pdf"`, or atoms. | +| `include_domains` | `list(string)` | Restrict results to these domains. | +| `exclude_domains` | `list(string)` | Exclude these domains. | +| `limit` | `integer` | Cap results. | +| `tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | `string` | Location string for localized results. | +| `country` | `string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Drop URLs that cannot be scraped. | +| `timeout` | `integer` | Request timeout in milliseconds. | +| `highlights` | `boolean` | Generate query-relevant highlights. Defaults to true. | +| `scrape_options` | `keyword list` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list(string)` | Enterprise options. Values: `"zdr"`, `"anon"`. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://docs.firecrawl.dev", + formats: ["markdown"] +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` (required) | The URL to scrape. | +| `formats` | `list` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. Maps: `%{type: "json", prompt: "..."}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}`. | +| `headers` | `map` | Custom request headers. | +| `include_tags` | `list(string)` | Include only specific HTML tags. | +| `exclude_tags` | `list(string)` | Exclude specific HTML tags. | +| `only_main_content` | `boolean` | Strip nav, footer, and other boilerplate. | +| `timeout` | `integer` | Timeout in milliseconds. Min: 1000, default: 60000, max: 300000. | +| `wait_for` | `integer` | Wait for the page to render (milliseconds). | +| `mobile` | `boolean` | Use a mobile viewport. | +| `parsers` | `list` | File parsing controls. Values: `"pdf"`, `%{type: "pdf", mode: "auto", maxPages: 5}`. | +| `actions` | `list(map)` | Pre-scrape browser actions. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `keyword list` | Geo or language-aware scraping. Keys: `country:`, `languages:`. | +| `skip_tls_verification` | `boolean` | Skip TLS verification. | +| `remove_base64_images` | `boolean` | Drop base64 images from markdown output. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `atom` | Proxy mode. Values: `:basic`, `:enhanced`, `:auto`. | +| `max_age` | `integer` | Use cached data up to this age (milliseconds). | +| `min_age` | `integer` | Use cached data only if at least this old (milliseconds). | +| `store_in_cache` | `boolean` | Cache the result. | +| `lockdown` | `boolean` | Only serve cached results, never make an outbound request. | +| `redact_pii` | `boolean` | Redact personally identifiable information. | +| `profile` | `keyword list` | Persistent browser profile. Keys: `name:`, `save_changes:`. | +| `zero_data_retention` | `boolean` | Enable zero data retention for this scrape. | +| `audit_metadata` | `keyword list` | User attribution for SIEM logging. Key: `username:` (required). | + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + "", + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `string` | Scrape job ID. First positional argument. | +| `code` | `string` (required) | Code to execute in the browser session. | +| `language` | `atom \| string` | Runtime. Values: `:python`, `:node`, `:bash`. | +| `timeout` | `integer` | Execution timeout in seconds. | +| `origin` | `string` | Optional origin label for telemetry. | + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id, opts \\ [])` issues `DELETE /scrape/{jobId}/interact`. + +```elixir +{:ok, res} = Firecrawl.stop_interactive_scrape_browser_session("") +``` + +## Notes + +- The Elixir SDK is OpenAPI-generated; function names come from the spec and are not renamed. +- Every public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`). +- Per-request options (like `api_key:`, `base_url:`) are passed via the trailing `opts` keyword list. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..3ed75e1b2 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,188 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for external agents. Aligned with `firecrawl-java` **v1.12.1** (`firecrawl/apps/java-sdk`) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment/system property: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. You can constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries"); +List> web = results.getWeb(); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | The search query. Use `site:example.com` to limit results to a domain. | +| `options.sources` | `List` | Sources to search. Values: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | Filter by category. Values: `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | Restrict results to these domains. | +| `options.excludeDomains` | `List` | Exclude these domains. | +| `options.limit` | `Integer` | Cap results. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location string for localized results. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `Integer` | Request timeout in milliseconds. | +| `options.highlights` | `Boolean` | Generate query-relevant highlights. Defaults to true. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +**Return value:** `SearchData` with `getWeb()`, `getNews()`, `getImages()` (each `List>`, may be null). Do not treat `SearchData` as a directly iterable list. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.Document; + +Document doc = client.scrape( + "https://docs.firecrawl.dev", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); +System.out.println(doc.getMarkdown()); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | The URL to scrape. | +| `options.formats` | `List` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `JsonFormat.builder().prompt("...").build()`, `QuestionFormat`, `HighlightsFormat`. | +| `options.headers` | `Map` | Custom request headers. | +| `options.includeTags` | `List` | Include only specific HTML tags. | +| `options.excludeTags` | `List` | Exclude specific HTML tags. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, and other boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait for the page to render (milliseconds). | +| `options.mobile` | `Boolean` | Use a mobile viewport. | +| `options.parsers` | `List` | File parsing controls. Values: `"pdf"`, `Map.of("type", "pdf", "maxPages", 10)`. | +| `options.actions` | `List>` | Pre-scrape browser actions. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `LocationConfig` | Geo or language-aware scraping. Fields: `country`, `languages`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown output. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy mode. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `options.maxAge` | `Long` | Use cached data up to this age (milliseconds). | +| `options.storeInCache` | `Boolean` | Cache the result. | +| `options.lockdown` | `Boolean` | Only serve cached results, never make an outbound request. | +| `options.redactPII` | `Boolean` | Redact personally identifiable information. | +| `options.auditMetadata` | `AuditMetadata` | User attribution for SIEM logging. Constructor arg: `username`. | + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `node` +- `client.interact(jobId, code, language, timeout)` — `timeout` in seconds (1–300), null for API default (30s) + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +BrowserExecuteResponse result = client.interact( + "", + "console.log(await page.title());", + "node", + 60 +); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | Scrape job ID. | +| `code` | `String` | Code to run in the browser session. | +| `language` | `String` | Runtime. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Null for API default. | +| `origin` | `String` | Optional origin label for request attribution. | + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → `BrowserDeleteResponse` + +Ends the scrape-bound browser session. Response includes `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`. + +## Notes + +- The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike JS, Python, and Rust SDKs). +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Async variants are available: `scrapeAsync`, `searchAsync`, `interactAsync`, `stopInteractiveBrowserAsync` — all return `CompletableFuture`. +- Uses camelCase parameter names matching the Java convention. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..31d65a059 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,169 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for external agents. Aligned with `firecrawl` **v4.32.0** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or other browser actions after a scrape has created a session. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. You can constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries"); +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | The search query. Use `site:example.com` to limit results to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which search sources to query. | +| `options.categories` | `("github" \| "research" \| "pdf" \| "developer")[]` | Filter results by category. | +| `options.includeDomains` | `string[]` | Restrict results to these domains. Mutually exclusive with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Exclude these domains. Mutually exclusive with `includeDomains`. | +| `options.limit` | `number` | Cap the number of results. | +| `options.tbs` | `string` | Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). | +| `options.location` | `string` | Location string for localized results. | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped by other endpoints. | +| `options.timeout` | `number` | Request timeout in milliseconds. | +| `options.highlights` | `boolean` | Generate query-relevant highlights. Defaults to true server-side. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise options. `"zdr"` for zero data retention, `"anon"` for anonymized search. | + +**Return value:** `SearchData` with optional arrays `web`, `news`, `images`, `developer`. Do not access `result.data` — it throws an error directing you to use `result.web`, `result.news`, etc. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://docs.firecrawl.dev", { + formats: ["markdown"] +}); +console.log(doc.markdown); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | The URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats. Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Object formats: `{ type: "json", prompt?, schema? }`, `{ type: "question", question }`, `{ type: "highlights", query }`, `{ type: "screenshot", fullPage?, quality?, viewport? }`, `{ type: "changeTracking", modes, schema?, prompt?, tag? }`, `{ type: "attributes", selectors: [{ selector, attribute }] }`. Note: plain string `"json"` is rejected — use the object form. | +| `options.headers` | `Record` | Custom request headers. | +| `options.includeTags` | `string[]` | Include only specific HTML tags. | +| `options.excludeTags` | `string[]` | Exclude specific HTML tags. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, and other boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait for the page to render (milliseconds). | +| `options.mobile` | `boolean` | Use a mobile viewport. | +| `options.parsers` | `("pdf" \| { type: "pdf", mode?, maxPages? })[]` | File parsing controls. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `options.actions` | `ActionOption[]` | Pre-scrape browser actions. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `options.location` | `{ country?, languages? }` | Geo or language-aware scraping. | +| `options.skipTlsVerification` | `boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown output. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | Proxy mode. | +| `options.maxAge` | `number` | Use cached data up to this age (milliseconds). | +| `options.minAge` | `number` | Use cached data only if at least this old (milliseconds). | +| `options.storeInCache` | `boolean` | Cache the result. | +| `options.lockdown` | `boolean` | Only serve cached results, never make an outbound request. | +| `options.redactPII` | `boolean \| RedactPIIOptions` | Redact personally identifiable information. Options: `mode`, `entities`, `replaceStyle`. | +| `options.profile` | `{ name, saveChanges? }` | Persistent browser profile shared across scrapes and interactions. | +| `options.auditMetadata` | `{ username }` | User attribution for SIEM logging. | + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job (via `metadata.scrapeId`). The SDK requires at least one of `code` or `prompt`. For flows that go beyond quick pre-scrape tweaks, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans." +}); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to run in the browser session (e.g. Playwright `page` usage). Provide `code` or `prompt` (at least one required). | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. Provide `code` or `prompt` (at least one required). | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime language. Default: `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +### Stop session + +`client.stopInteraction(jobId)` → `Promise` + +Ends the scrape-bound browser session. Response includes `success`, `sessionDurationMs`, `creditsBilled`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- The package declares **Node.js >= 22** in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..5788e5aa3 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,162 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for external agents. Aligned with `firecrawl-py` **v4.34.0** (`firecrawl/apps/python-sdk`) and the v2 OpenAPI spec. Method names, parameters, and types match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```py +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. You can constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```py +results = client.search("site:docs.firecrawl.dev webhook retries") +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | The search query. Use `site:example.com` to limit results to a domain. | +| `sources` | `list[str \| Source]` | Which sources to search. Values: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str \| Category]` | Filter by category. Values: `"github"`, `"research"`, `"pdf"`, `"developer"`. | +| `include_domains` | `list[str]` | Restrict results to these domains. Mutually exclusive with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Mutually exclusive with `include_domains`. | +| `limit` | `int` | Cap results. Default: `5` in SDK model. | +| `tbs` | `str` | Time-based filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). | +| `location` | `str` | Location string for localized results. | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped by other endpoints. | +| `timeout` | `int` | Request timeout in milliseconds. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. Defaults to true server-side. | +| `scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options. `"zdr"` for zero data retention, `"anon"` for anonymized search. | + +**Return value:** `SearchData` with optional lists `web`, `news`, `images`, `developer`. Do not access `result.data` — it raises `AttributeError` directing you to use `result.web`, `result.news`, etc. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```py +doc = client.scrape("https://docs.firecrawl.dev", formats=["markdown"]) +print(doc.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | The URL to scrape. | +| `formats` | `list[str \| dict]` | Output formats. Strings: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"`. Objects: `{"type": "json", "prompt": ..., "schema": ...}`, `{"type": "question", "question": ...}`, `{"type": "highlights", "query": ...}`, `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}`, `{"type": "changeTracking", "modes": [...], "tag": ...}`, `{"type": "attributes", "selectors": [...]}`. Note: plain string `"json"` is rejected — use the dict form. | +| `headers` | `dict[str, str]` | Custom request headers. | +| `include_tags` | `list[str]` | Include only specific HTML tags. | +| `exclude_tags` | `list[str]` | Exclude specific HTML tags. | +| `only_main_content` | `bool` | Strip nav, footer, and other boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait for the page to render (milliseconds). | +| `mobile` | `bool` | Use a mobile viewport. | +| `parsers` | `list[str \| PDFParser]` | File parsing controls. PDF modes: `"fast"`, `"auto"`, `"ocr"`. | +| `actions` | `list[dict]` | Pre-scrape browser actions. Types: `wait`, `screenshot`, `click`, `write`, `press`, `scroll`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `Location` | Geo or language-aware scraping. Fields: `country`, `languages`. | +| `skip_tls_verification` | `bool` | Skip TLS verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown output. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Use cached data up to this age (milliseconds). | +| `store_in_cache` | `bool` | Cache the result. | +| `lockdown` | `bool` | Only serve cached results, never make an outbound request. | +| `profile` | `dict` | Persistent browser profile. Keys: `name`, `save_changes`. | +| `audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. Field: `username`. | + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. At least one of `code` or `prompt` must be non-empty. + +### Example + +```py +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | Scrape job ID from `document.metadata.scrape_id`. | +| `code` | `str` | Code to run in the browser session. Provide `code` or `prompt` (at least one required). | +| `prompt` | `str` | Natural-language instruction for the browser agent. Provide `code` or `prompt` (at least one required). Keyword-only. | +| `language` | `str` | Runtime. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds (1–300). | + +### Stop session + +`client.stop_interaction(job_id)` ends the scrape-bound browser session. Returns `BrowserDeleteResponse` with `success`, `session_duration_ms`, `credits_billed`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is an alias for `Firecrawl`; `AsyncFirecrawlApp` is an alias for `AsyncFirecrawl`. +- Pydantic models handle camelCase conversion for the API wire format; always use snake_case in Python. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..863dbffff --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,183 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for external agents. Aligned with `firecrawl` crate **v2.12.1** (`firecrawl/apps/rust-sdk`) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. You can constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::Client; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", None) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | The search query. Use `site:example.com` to limit results to a domain. | +| `options.sources` | `Vec` | Sources to search. Values: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | Filter by category. Values: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | Restrict results to these domains. | +| `options.exclude_domains` | `Vec` | Exclude these domains. | +| `options.limit` | `u32` | Cap results. | +| `options.tbs` | `String` | Time-based filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | Location string for localized results. | +| `options.ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `options.timeout` | `u32` | Request timeout in milliseconds. | +| `options.highlights` | `bool` | Generate query-relevant highlights. Defaults to true server-side. | +| `options.scrape_options` | `ScrapeOptions` | Scrape each search result (see Scrape parameters). | + +**Return value:** `SearchResponse` containing `success: bool`, `data: SearchData`, `warning: Option`. `SearchData` has `web: Option>`, `news: Option>`, `images: Option>`. + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let doc = client + .scrape("https://docs.firecrawl.dev", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | The URL to scrape. | +| `options.formats` | `Vec` | Output formats. Values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also: `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `options.headers` | `HashMap` | Custom request headers. | +| `options.include_tags` | `Vec` | Include only specific HTML tags. | +| `options.exclude_tags` | `Vec` | Exclude specific HTML tags. | +| `options.only_main_content` | `bool` | Strip nav, footer, and other boilerplate. | +| `options.timeout` | `u32` | Timeout in milliseconds. | +| `options.wait_for` | `u32` | Wait for the page to render (milliseconds). | +| `options.mobile` | `bool` | Use a mobile viewport. | +| `options.parsers` | `Vec` | File parsing controls. Values: `ParserConfig::Simple("pdf")`, `ParserConfig::Pdf { parser_type, mode?, max_pages? }`. | +| `options.actions` | `Vec` | Pre-scrape browser actions. Types: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `options.location` | `LocationConfig` | Geo or language-aware scraping. Fields: `country`, `languages`. | +| `options.skip_tls_verification` | `bool` | Skip TLS verification. | +| `options.remove_base64_images` | `bool` | Drop base64 images from markdown output. | +| `options.fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `bool` | Block ads and cookie popups. | +| `options.proxy` | `ProxyType` | Proxy mode. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | Use cached data up to this age (seconds). | +| `options.min_age` | `u32` | Use cached data only if at least this old (seconds). | +| `options.store_in_cache` | `bool` | Cache the result. | +| `options.lockdown` | `bool` | Only serve cached results, never make an outbound request. | +| `options.redact_pii` | `bool` | Redact personally identifiable information. | +| `options.profile` | `ProfileConfig` | Persistent browser profile. Fields: `name`, `save_changes`. | +| `options.audit_metadata` | `AuditMetadata` | User attribution for SIEM logging. Field: `username`. | +| `options.json_options` | `JsonOptions` | JSON extraction config. Fields: `schema`, `system_prompt`, `prompt`. | +| `options.screenshot_options` | `ScreenshotOptions` | Screenshot config. Fields: `full_page`, `quality`, `viewport`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | Change tracking config. Fields: `modes` (`GitDiff`, `Json`), `schema`, `prompt`, `tag`. | +| `options.attribute_selectors` | `Vec` | Attribute extraction. Fields: `selector`, `attribute`. | + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeExecuteOptions}; + +let result = client + .interact( + "", + ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }, + ) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | Scrape job ID. | +| `options.code` | `Option` | Code to run in the browser session. Provide `code` or `prompt` (at least one required, else `FirecrawlError::Misuse`). | +| `options.prompt` | `Option` | Natural-language instruction for the browser agent. Provide `code` or `prompt` (at least one required). | +| `options.language` | `ScrapeExecuteLanguage` | Runtime. Values: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `u32` | Execution timeout in seconds. | + +### Stop session + +`client.stop_interaction(job_id)` → `Result` + +Ends the scrape-bound browser session. Response includes `success`, `session_duration_ms`, `credits_billed`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` uses dedicated `json_options`, `screenshot_options`, `change_tracking_options` for advanced format configuration (unlike JS/Python which use inline format objects). +- `search_and_scrape(query, limit)` is a convenience helper that searches then returns `Vec`. +- All types are exported at the crate root: `use firecrawl::Client`. +- All option structs derive `Default`; use `..Default::default()` to fill unset fields. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 06d233781..60086a5a6 100755 --- a/docs.json +++ b/docs.json @@ -673,6 +673,16 @@ "features/ask" ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Cookbooks", "pages": [