From ae8c621be863ff7114db481cdcab47195385dc78 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 16 Aug 2026 13:20:14 +0000 Subject: [PATCH] docs: add agent quickstart pages for Node.js, Python, Rust, Java, and Elixir One canonical quickstart per SDK language covering search, scrape, and interact endpoints with complete parameter tables generated from SDK source and OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01QfoYYLS6N7XmYvKhpLpk7L --- agent-quickstart/elixir.mdx | 209 +++++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 225 +++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 215 ++++++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 200 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 228 ++++++++++++++++++++++++++++++++++++ docs.json | 10 ++ 6 files changed, 1087 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..879990440 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,209 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Elixir. Generated from SDK source and OpenAPI spec. The Elixir SDK is auto-generated from the OpenAPI spec, so function names mirror the API operation names. + +## Install + +Add to `mix.exs`: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.9"} + ] +end +``` + +Then run: + +```bash +mix deps.get +``` + +## Authenticate + +Add to your config: + +```elixir +config :firecrawl, api_key: "fc-YOUR-API-KEY" +``` + +Or pass `api_key` per-request: + +```elixir +Firecrawl.search_and_scrape([query: "firecrawl"], api_key: "fc-YOUR-API-KEY") +``` + +All functions accept a trailing `opts` keyword list supporting: + +| Option | Type | Description | +|---|---|---| +| `api_key` | `string` | Override the configured API key. | +| `base_url` | `string` | Override the base URL. Default: `"https://api.firecrawl.dev/v2"`. | + +Additional keys in `opts` are passed through to `Req`. + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params, opts \\ [])` + +### Example + +```elixir +{:ok, results} = Firecrawl.search_and_scrape(query: "firecrawl web scraping", limit: 5) + +for result <- results["data"]["web"] do + IO.puts("#{result["title"]} #{result["url"]}") +end +``` + +Every function also has a bang variant (`search_and_scrape!`) that raises on error instead of returning `{:error, ...}`. + +### Parameters + +All parameters are passed as a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** Search query. | +| `limit` | `integer` | Max results per source type. | +| `sources` | `list` | Sources to search. Default: `["web"]`. | +| `categories` | `list` | Category filters. | +| `include_domains` | `list(string)` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list(string)` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"` past day). | +| `location` | `string` | Geo-targeting location string. | +| `country` | `string` | ISO country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Exclude invalid URLs. | +| `timeout` | `integer` | Timeout in ms. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrape_options` | `keyword_list` | Scrape options applied to each result page. | +| `enterprise` | `list(string)` | ZDR options: `["zdr"]` or `["anon"]`. | + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params, opts \\ [])` + +### Example + +```elixir +{:ok, result} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"], + only_main_content: true +) + +IO.puts(result["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Default: `["markdown"]`. | +| `headers` | `any` | Custom HTTP headers. | +| `include_tags` | `list(string)` | HTML tags to include. | +| `exclude_tags` | `list(string)` | HTML tags to exclude. | +| `only_main_content` | `boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `integer` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `wait_for` | `integer` | Extra delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `list` | File processing controls (e.g. PDF). | +| `actions` | `list` | Browser actions before content capture. | +| `location` | `keyword_list` | Geo settings. Country defaults to `"US"`. | +| `skip_tls_verification` | `boolean` | Skip TLS certificate verification. | +| `remove_base64_images` | `boolean` | Remove base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `:basic \| :enhanced \| :auto` | Proxy type. Default: `"auto"`. | +| `max_age` | `integer` | Cache threshold in ms. Default: 2 days. | +| `min_age` | `integer` | Cache-only mode minimum age in ms. | +| `store_in_cache` | `boolean` | Store result in cache. | +| `lockdown` | `boolean` | Cache-only, no outbound requests. | +| `redact_pii` | `boolean` | Redact PII. | +| `audit_metadata` | `keyword_list` | SIEM logging. Required key: `username` (string). | +| `profile` | `keyword_list` | Persistent browser profile. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Execute code in the browser sandbox to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ [])` + +### Example + +```elixir +{:ok, result} = Firecrawl.scrape_and_extract_from_url( + url: "https://www.amazon.com", + formats: ["markdown"] +) + +scrape_id = result["data"]["metadata"]["scrapeId"] + +{:ok, response} = Firecrawl.interact_with_scrape_browser_session(scrape_id, + code: "document.querySelector('input[name=field-keywords]').value = 'iPhone 16 Pro Max'" +) + +Firecrawl.stop_interactive_scrape_browser_session(scrape_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t` | **Required.** First positional argument. Scrape job ID from response metadata. | +| `code` | `string` | **Required.** Code to execute in the browser sandbox. | +| `language` | `:python \| :node \| :bash` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `integer` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `string` | Origin label for telemetry. | + +Stop the session when done: + +```elixir +Firecrawl.stop_interactive_scrape_browser_session(scrape_id) +``` + +## Notes + +- Parameter names use **snake_case** and are passed as keyword lists. +- The Elixir SDK is **auto-generated from the OpenAPI spec**, so function names are verbose and mirror API operation names directly. +- The `interact` function requires `code` — it does not support a `prompt` parameter. Use the `code` parameter with JavaScript to control the browser. +- Every function has a **bang variant** (`!` suffix) that raises `Firecrawl.Error` instead of returning `{:error, ...}`. +- An `origin` field (`"elixir-sdk@{version}"`) is automatically injected into every request body. +- There are no deprecated aliases in the Elixir SDK. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..190ce16bb --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,225 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Java. Generated from SDK source and OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```groovy +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR-API-KEY") + .build(); +``` + +Or read from the `FIRECRAWL_API_KEY` environment variable: + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +Builder options: + +| Option | Type | Default | +|---|---|---| +| `apiKey` | `String` | `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property | +| `apiUrl` | `String` | `"https://api.firecrawl.dev"` | +| `timeoutMs` | `long` | `300000` (5 min) | +| `maxRetries` | `int` | `3` | +| `backoffFactor` | `double` | `0.5` | +| `asyncExecutor` | `Executor` | `ForkJoinPool.commonPool()` | +| `httpClient` | `OkHttpClient` | — (overrides `timeoutMs`) | + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; + +SearchData results = client.search("firecrawl web scraping", SearchOptions.builder() + .limit(5) + .build()); + +for (var result : results.getWeb()) { + System.out.println(result.get("title") + " " + result.get("url")); +} +``` + +### Parameters + +All fields on `SearchOptions` are nullable and default to null (API defaults apply). + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | **Required.** First positional argument. Search query. | +| `limit` | `Integer` | Max results per source type. | +| `sources` | `List` | Sources: `"web"`, `"news"`, `"images"` as strings or `{type: "web"}` maps. | +| `categories` | `List` | Categories: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains` | `List` | Exclude these domains. Cannot combine with `includeDomains`. | +| `tbs` | `String` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week). | +| `location` | `String` | Geo-targeting location string. | +| `ignoreInvalidURLs` | `Boolean` | Exclude invalid URLs. | +| `timeout` | `Integer` | Timeout in ms. | +| `highlights` | `Boolean` | Generate query-relevant highlights. | +| `scrapeOptions` | `ScrapeOptions` | Scrape options applied to each result page. | +| `integration` | `String` | Integration identifier. | + +Response fields: `results.getWeb()`, `results.getNews()`, `results.getImages()` each return `List>`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; + +Document result = client.scrape("https://example.com", ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .onlyMainContent(true) + .build()); + +System.out.println(result.getMarkdown()); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable and default to null (API defaults apply). + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | **Required.** First positional argument. URL to scrape. | +| `formats` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Also accepts format config objects. Default: `["markdown"]`. | +| `headers` | `Map` | Custom HTTP headers. | +| `includeTags` | `List` | HTML tags to include. | +| `excludeTags` | `List` | HTML tags to exclude. | +| `onlyMainContent` | `Boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `Integer` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `waitFor` | `Integer` | Extra delay in ms before fetching content. | +| `mobile` | `Boolean` | Emulate a mobile device. | +| `parsers` | `List` | File processing controls (e.g. `"pdf"` or `{"type":"pdf","maxPages":10}`). | +| `actions` | `List>` | Browser actions before content capture. | +| `location` | `LocationConfig` | Geo settings with `country` and `languages`. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64 images from markdown. | +| `blockAds` | `Boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `String` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `maxAge` | `Long` | Cache threshold in ms. Default: 2 days. | +| `storeInCache` | `Boolean` | Store result in cache. | +| `lockdown` | `Boolean` | Cache-only, no outbound requests. | +| `redactPII` | `Boolean` | Redact PII. | +| `auditMetadata` | `AuditMetadata` | SIEM logging with `username` field. | +| `integration` | `String` | Integration identifier. | + +Async variant: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Execute code in the browser sandbox to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document result = client.scrape("https://www.amazon.com", ScrapeOptions.builder() + .formats(List.of("markdown")) + .build()); + +String scrapeId = (String) result.getMetadata().get("scrapeId"); + +BrowserExecuteResponse response = client.interact(scrapeId, + "document.querySelector('input[name=field-keywords]').value = 'iPhone 16 Pro Max'"); + +client.stopInteractiveBrowser(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | **Required.** Scrape job ID from `result.getMetadata().get("scrapeId")`. | +| `code` | `String` | **Required.** Code to execute in the browser sandbox. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Default: `30`. | +| `origin` | `String` | Origin label. Default: `"java-sdk@{version}"`. | + +Stop the session when done: + +```java +client.stopInteractiveBrowser(scrapeId); +``` + +Async variants: `client.interactAsync(...)` and `client.stopInteractiveBrowserAsync(...)` return `CompletableFuture`. + +## Notes + +- Parameter names use **camelCase** (e.g. `onlyMainContent`, `includeTags`, `scrapeOptions`). +- All option classes use the **builder pattern**: `ScrapeOptions.builder().field(value).build()`. +- The Java `interact` method requires `code` — it does not support a `prompt` parameter. Use the `code` parameter with JavaScript to control the browser. +- `includeDomains` and `excludeDomains` on search are mutually exclusive. +- Deprecated aliases (do not use in new code): + - `scrapeExecute(...)` → use `interact(...)` + - `deleteScrapeBrowser(...)` → use `stopInteractiveBrowser(...)` + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..0d67e420e --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,215 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Node.js. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js 22+. + +## Authenticate + +```javascript +import Firecrawl from 'firecrawl'; + +const app = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" }); +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `apiKey`: + +```javascript +const app = new Firecrawl(); +``` + +Constructor options: + +| Option | Type | Default | +|---|---|---| +| `apiKey` | `string \| null` | `process.env.FIRECRAWL_API_KEY` | +| `apiUrl` | `string \| null` | `"https://api.firecrawl.dev"` | +| `timeoutMs` | `number` | — | +| `maxRetries` | `number` | — | +| `backoffFactor` | `number` | — | + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`search(query, options?)` + +### Example + +```javascript +const results = await app.search("firecrawl web scraping", { limit: 5 }); + +for (const result of results.web) { + console.log(result.title, result.url); + console.log(result.markdown); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** Search query (max 500 chars). | +| `limit` | `number` | Max results per source type. Must be positive. | +| `sources` | `Array<"web" \| "news" \| "images" \| {type: string}>` | Sources to search. Default: `[{type: "web"}]`. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer">` | Category filters. | +| `includeDomains` | `string[]` | Restrict to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude these domains. Cannot combine with `includeDomains`. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week, `"qdr:m"` past month). | +| `location` | `string` | Geo-targeting location string. | +| `ignoreInvalidURLs` | `boolean` | Exclude invalid URLs from results. | +| `timeout` | `number` | Timeout in ms. Must be positive. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrapeOptions` | `ScrapeOptions` | Scrape options applied to each result page. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise zero data retention options. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +Response groups results by source: `results.web`, `results.news`, `results.images`, `results.developer`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`scrape(url, options?)` + +### Example + +```javascript +const result = await app.scrape("https://example.com", { + formats: ["markdown", "links"], + onlyMainContent: true +}); + +console.log(result.markdown); +console.log(result.links); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** URL to scrape. | +| `formats` | `FormatOption[]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts config objects (see below). Default: `["markdown"]`. | +| `headers` | `Record` | Custom HTTP headers. | +| `includeTags` | `string[]` | HTML tags to include. | +| `excludeTags` | `string[]` | HTML tags to exclude. | +| `onlyMainContent` | `boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `number` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `waitFor` | `number` | Extra delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `Array` | File processing controls. Default: `["pdf"]`. | +| `actions` | `ActionOption[]` | Browser actions before content capture: `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `{country?: string, languages?: string[]}` | Geo settings. Country defaults to `"US"`. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64 images from markdown. Default: `true`. | +| `fastMode` | `boolean` | Enable fast mode. | +| `blockAds` | `boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | Proxy type. `"enhanced"` costs up to 5 credits. Default: `"auto"`. | +| `maxAge` | `number` | Cache threshold in ms. Returns cached version if younger. Default: 2 days. | +| `minAge` | `number` | Cache-only mode minimum age in ms. | +| `storeInCache` | `boolean` | Store result in Firecrawl cache. Default: `true`. | +| `lockdown` | `boolean` | Cache-only, no outbound requests. | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact PII. Pass `true` for defaults or `{mode?: "accurate"\|"aggressive"\|"fast", entities?: string[], replaceStyle?: "tag"\|"mask"\|"remove"}`. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection override with `mode`, `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. | +| `auditMetadata` | `{username: string}` | SIEM logging user attribution. | +| `profile` | `{name: string, saveChanges?: boolean}` | Persistent browser profile. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label. | + +#### Format config objects + +| Format | Config fields | +|---|---| +| `json` | `{type: "json", schema?: object, prompt?: string}` — structured extraction with optional JSON Schema and prompt. | +| `screenshot` | `{type: "screenshot", fullPage?: boolean, quality?: number, viewport?: {width, height}}` | +| `changeTracking` | `{type: "changeTracking", modes: ("git-diff"\|"json")[], schema?: object, prompt?: string, tag?: string}` | +| `question` | `{type: "question", question: string}` — ask a question about the page. | +| `highlights` | `{type: "highlights", query: string}` — extract text relevant to a query. | +| `attributes` | `{type: "attributes", selectors: [{selector: string, attribute: string}]}` | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`interact(jobId, args)` + +### Example + +```javascript +const result = await app.scrape("https://www.amazon.com", { formats: ["markdown"] }); +const scrapeId = result.metadata?.scrapeId; + +await app.interact(scrapeId, { prompt: "Search for iPhone 16 Pro Max" }); +const response = await app.interact(scrapeId, { + prompt: "Click on the first result and tell me the price" +}); +console.log(response.output); + +await app.stopInteraction(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | **Required.** Scrape job ID from `result.metadata.scrapeId`. | +| `code` | `string` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `string` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `string` | Origin label for telemetry. | + +Stop the session when done: + +```javascript +await app.stopInteraction(scrapeId); +``` + +## Notes + +- Parameter names use **camelCase** (e.g. `onlyMainContent`, `includeTags`, `scrapeOptions`). +- `includeDomains` and `excludeDomains` on search are mutually exclusive. +- Deprecated aliases (do not use in new code): + - `scrapeExecute()` → use `interact()` + - `stopInteractiveBrowser()` / `deleteScrapeBrowser()` → use `stopInteraction()` + - `scrapeUrl()` → use `scrape()` + - `crawlUrl()` → use `crawl()` + - `mapUrl()` → use `map()` + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..d1a96a886 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,200 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Python. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python 3.8+. + +## Authenticate + +```python +from firecrawl import Firecrawl + +app = Firecrawl(api_key="fc-YOUR-API-KEY") +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `api_key`: + +```python +app = Firecrawl() +``` + +Constructor parameters: + +| Parameter | Type | Default | +|---|---|---| +| `api_key` | `str` | `None` (falls back to `FIRECRAWL_API_KEY` env var) | +| `api_url` | `str` | `"https://api.firecrawl.dev"` | +| `timeout` | `float` | `None` | +| `max_retries` | `int` | `3` | +| `backoff_factor` | `float` | `0.5` | + +An async client is also available: `from firecrawl import AsyncFirecrawl`. + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`search(query, **kwargs)` + +### Example + +```python +results = app.search("firecrawl web scraping", limit=5) + +for result in results.web: + print(result.title, result.url) + print(result.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | **Required.** Search query (max 500 chars). | +| `limit` | `int` | Max results per source type. Default: `5` (SDK default). | +| `sources` | `list` | Sources to search: `"web"`, `"news"`, `"images"`, or `Source` objects. | +| `categories` | `list` | Category filters: `"github"`, `"research"`, `"pdf"`, `"developer"`. | +| `include_domains` | `list[str]` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `str` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week, `"qdr:m"` past month). | +| `location` | `str` | Geo-targeting location string. | +| `ignore_invalid_urls` | `bool` | Exclude invalid URLs from results. | +| `timeout` | `int` | Timeout in ms. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. | +| `scrape_options` | `ScrapeOptions` | Scrape options applied to each result page. | +| `enterprise` | `list[str]` | Enterprise zero data retention options. | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `str` | Integration identifier. | + +Response groups results by source: `results.web`, `results.news`, `results.images`, `results.developer`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`scrape(url, **kwargs)` + +### Example + +```python +result = app.scrape("https://example.com", formats=["markdown", "links"], only_main_content=True) + +print(result.markdown) +print(result.links) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | **Required.** URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config dicts. Default: `["markdown"]`. | +| `headers` | `dict[str, str]` | Custom HTTP headers. | +| `include_tags` | `list[str]` | HTML tags to include. | +| `exclude_tags` | `list[str]` | HTML tags to exclude. | +| `only_main_content` | `bool` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `int` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `wait_for` | `int` | Extra delay in ms before fetching content. | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list` | File processing controls. Accepts `"pdf"` or `{"type": "pdf", "mode": "fast"\|"auto"\|"ocr", "max_pages": int}`. | +| `actions` | `list` | Browser actions before content capture: `WaitAction`, `ClickAction`, `WriteAction`, `PressAction`, `ScrollAction`, `ScreenshotAction`, `ScrapeAction`, `ExecuteJavascriptAction`, `PDFAction`. | +| `location` | `Location` | Geo settings with `country` and `languages`. Country defaults to `"US"`. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64 images from markdown. Default: `true`. | +| `fast_mode` | `bool` | Enable fast mode. | +| `block_ads` | `bool` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `str` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `max_age` | `int` | Cache threshold in ms. Returns cached version if younger. Default: 2 days. | +| `min_age` | `int` | Cache-only mode minimum age in ms. | +| `store_in_cache` | `bool` | Store result in Firecrawl cache. Default: `true`. | +| `lockdown` | `bool` | Cache-only, no outbound requests. | +| `redact_pii` | `bool \| RedactPIIOptions` | Redact PII. Pass `True` for defaults or a `RedactPIIOptions` object. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection override. | +| `audit_metadata` | `AuditMetadata` | SIEM logging with `username` field. | +| `profile` | `dict` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`interact(job_id, code=None, *, prompt=None, language="node", timeout=None, origin=None)` + +### Example + +```python +result = app.scrape("https://www.amazon.com", formats=["markdown"]) +scrape_id = result.metadata.scrape_id + +app.interact(scrape_id, prompt="Search for iPhone 16 Pro Max") +response = app.interact(scrape_id, prompt="Click on the first result and tell me the price") +print(response.output) + +app.stop_interaction(scrape_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | **Required.** Scrape job ID from `result.metadata.scrape_id`. | +| `code` | `str` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `str` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `Literal["python", "node", "bash"]` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `str` | Origin label for telemetry. | + +Stop the session when done: + +```python +app.stop_interaction(scrape_id) +``` + +## Notes + +- Parameter names use **snake_case** (e.g. `only_main_content`, `include_tags`, `scrape_options`). +- `include_domains` and `exclude_domains` on search are mutually exclusive. +- `FirecrawlApp` is a deprecated alias for `Firecrawl`. Use `Firecrawl`. +- `AsyncFirecrawlApp` is a deprecated alias for `AsyncFirecrawl`. +- Deprecated method aliases (do not use in new code): + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + - `scrape_url()` → use `scrape()` + - `crawl_url()` → use `crawl()` + - `map_url()` → use `map()` + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..88701dec8 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,228 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Rust. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR-API-KEY")?; +``` + +For self-hosted instances: + +```rust +let client = Client::new_selfhosted("https://your-instance.example.com", Some("fc-YOUR-API-KEY"))?; +``` + +Set `FIRECRAWL_API_KEY` environment variable for keyless initialization (handled at the application level — the SDK accepts an empty or `None` key for keyless free tier). + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`client.search(query, options)` + +### Example + +```rust +use firecrawl::{Client, SearchOptions}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let response = client.search("firecrawl web scraping", SearchOptions { + limit: Some(5), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | **Required.** First positional argument. Search query. | +| `limit` | `Option` | Max results. Default: 5, Max: 20. | +| `sources` | `Option>` | Sources: `Web`, `News`, `Images`. | +| `categories` | `Option>` | Categories: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `Option>` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `Option` | Time-based filter (e.g. `"qdr:d"` past day). | +| `location` | `Option` | Geo-targeting location string. | +| `ignore_invalid_urls` | `Option` | Exclude invalid URLs. | +| `timeout` | `Option` | Timeout in ms. | +| `highlights` | `Option` | Generate query-relevant highlights. Default: `true`. | +| `scrape_options` | `Option` | Scrape options applied to each result page. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +There is also a convenience method `client.search_and_scrape(query, limit)` that returns `Vec` directly. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`client.scrape(url, options)` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let document = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + only_main_content: Some(true), + ..Default::default() +}).await?; + +if let Some(md) = document.markdown { + println!("{}", md); +} +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | **Required.** First positional argument. URL to scrape. | +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. Default: `[Markdown]`. | +| `headers` | `Option>` | Custom HTTP headers. | +| `include_tags` | `Option>` | HTML tags to include. | +| `exclude_tags` | `Option>` | HTML tags to exclude. | +| `only_main_content` | `Option` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `Option` | Timeout in ms. Default: `60000`. | +| `wait_for` | `Option` | Extra delay in ms before fetching content. | +| `mobile` | `Option` | Emulate a mobile device. | +| `parsers` | `Option>` | File processing controls (e.g. PDF with mode and max_pages). | +| `actions` | `Option>` | Browser actions before content capture. | +| `location` | `Option` | Geo settings with `country` and `languages`. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64 images from markdown. Default: `true`. | +| `fast_mode` | `Option` | Enable fast mode. | +| `block_ads` | `Option` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `Option` | Proxy type: `Basic`, `Stealth`, `Enhanced`, `Auto`. Default: `Auto`. | +| `max_age` | `Option` | Cache threshold in seconds. | +| `min_age` | `Option` | Cache-only mode minimum age in seconds. | +| `store_in_cache` | `Option` | Store result in cache. Default: `true`. | +| `lockdown` | `Option` | Cache-only, no outbound requests. | +| `redact_pii` | `Option` | Redact PII. | +| `audit_metadata` | `Option` | SIEM logging with `username` field. | +| `profile` | `Option` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction with `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config with `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking with `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute extraction with `selector` and `attribute`. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +There is also a convenience method `client.scrape_with_schema(url, schema, prompt)` for JSON extraction. + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, options)` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let document = client.scrape("https://www.amazon.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let scrape_id = document.metadata + .and_then(|m| m.get("scrapeId").and_then(|v| v.as_str().map(String::from))) + .expect("scrapeId not found"); + +let response = client.interact(&scrape_id, ScrapeExecuteOptions { + prompt: Some("Search for iPhone 16 Pro Max".into()), + ..Default::default() +}).await?; + +let response = client.interact(&scrape_id, ScrapeExecuteOptions { + prompt: Some("Click on the first result and tell me the price".into()), + ..Default::default() +}).await?; +println!("{:?}", response.output); + +client.stop_interaction(&scrape_id).await?; +``` + +### Parameters + +All fields on `ScrapeExecuteOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | **Required.** First positional argument. Scrape job ID from document metadata. | +| `code` | `Option` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `Option` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `Option` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +Stop the session when done: + +```rust +client.stop_interaction(&scrape_id).await?; +``` + +## Notes + +- All option structs use **snake_case** fields and derive `Default` — use struct literal syntax with `..Default::default()`. +- The `options` parameter on `scrape` and `search` accepts `impl Into>`, so you can pass `None` directly to skip options. +- The `origin` field is automatically set to the SDK version string if not provided. +- Deprecated aliases (do not use in new code): + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 922f9b178..606d97c25 100755 --- a/docs.json +++ b/docs.json @@ -283,6 +283,16 @@ } ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Developer Guides", "pages": [