From eb26da324c0ee3f3288d34a88db963955c8db598 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 13:15:04 +0000 Subject: [PATCH] docs(agent-quickstart): add canonical agent quickstart docs for 5 SDK languages Add one quickstart file per SDK language (Node.js, Python, Rust, Java, Elixir) covering search, scrape, and interact endpoints. Each file includes install, auth, method signatures, realistic examples, and every confirmed parameter with descriptions. Generated from SDK source and v2 OpenAPI spec. Co-Authored-By: Claude Opus 4.6 (1M context) Claude-Session: https://claude.ai/code/session_01M7VCJFZtQQ5GXHTrdvqoAL --- agent-quickstart/elixir.mdx | 237 ++++++++++++++++++++++++++++++ agent-quickstart/java.mdx | 272 ++++++++++++++++++++++++++++++++++ agent-quickstart/node.mdx | 257 ++++++++++++++++++++++++++++++++ agent-quickstart/python.mdx | 250 +++++++++++++++++++++++++++++++ agent-quickstart/rust.mdx | 284 ++++++++++++++++++++++++++++++++++++ 5 files changed, 1300 insertions(+) create mode 100644 agent-quickstart/elixir.mdx create mode 100644 agent-quickstart/java.mdx create mode 100644 agent-quickstart/node.mdx create mode 100644 agent-quickstart/python.mdx create mode 100644 agent-quickstart/rust.mdx diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..6c0da9ef7 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,237 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical Firecrawl Elixir quickstart for external agents. Generated from SDK source (`firecrawl` hex package v1.9.1) and the v2 OpenAPI spec. The Elixir SDK is auto-generated from the OpenAPI spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape( + [query: "site:docs.firecrawl.dev webhook retries"], + api_key: "fc-your-api-key" +) +``` + +There is no client struct or process. The SDK is a single flat module (`Firecrawl`) with module-level functions. Auth is configured globally via application config or per-call via opts. All functions also accept `:base_url` in opts to override the default API URL. + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + limit: 5, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +Parameters are passed as a keyword list. Snake_case keys are auto-converted to camelCase for the API. + +- `query` — string (required). The search query. + +- `sources` — list. Sources to search. Default: `["web"]`. Values: `:web`, `:news`, `:images`, or `%{type: "web"}` maps. + +- `categories` — list. Category filter. Values: `:github`, `:research`, `:pdf`, or typed maps. + +- `include_domains` — list of strings. Restrict results to these domains. + +- `exclude_domains` — list of strings. Exclude results from these domains. + +- `limit` — integer. Maximum results to return. + +- `tbs` — string. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). + +- `location` — string. Geographic location for localized results. + +- `country` — string. ISO 3166-1 alpha-2 code for geo-targeting (e.g. `"US"`). + +- `ignore_invalid_urls` — boolean. Drop URLs that cannot be scraped. + +- `highlights` — boolean. Generate query-relevant highlights. Default: true. + +- `timeout` — integer. Request timeout in milliseconds. + +- `enterprise` — list of strings. Enterprise options: `["zdr"]` for zero data retention, `["anon"]` for anonymized. + +- `scrape_options` — keyword list. Options for scraping each search result (same fields as scrape). + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true +) +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `formats` — list of format strings or format maps. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"` + - Map formats: `%{type: "json", prompt: "..."}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}` + +- `headers` — map. Custom HTTP headers. + +- `include_tags` — list of strings. Only include content from these HTML tags. + +- `exclude_tags` — list of strings. Exclude content from these HTML tags. + +- `only_main_content` — boolean. Strip nav, footer, and other boilerplate. + +- `timeout` — integer. Timeout in milliseconds. Default: 60000, min: 1000, max: 300000. + +- `wait_for` — integer. Wait for page to render (milliseconds). + +- `mobile` — boolean. Emulate mobile viewport. + +- `parsers` — list. Values: `"pdf"` or `%{type: "pdf", mode: "auto", maxPages: 5}`. + +- `actions` — list of action maps. Pre-scrape browser actions. + - `%{type: "click", selector: "#btn"}` + - `%{type: "wait", milliseconds: 1000}` or `%{type: "wait", selector: "#el"}` + - `%{type: "write", text: "hello"}` + - `%{type: "press", key: "Enter"}` + - `%{type: "scroll", direction: "down"}` + - `%{type: "scrape"}` + - `%{type: "executeJavascript", script: "..."}` + +- `location` — keyword list with `country:` and `languages:`. Geo or language-aware scraping. + +- `skip_tls_verification` — boolean. Skip TLS verification. + +- `remove_base64_images` — boolean. Drop base64 images from markdown. + +- `block_ads` — boolean. Block ads and cookie popups. + +- `proxy` — atom. Values: `:basic`, `:enhanced`, `:auto`. Enhanced costs up to 5 credits. + +- `max_age` — integer. Use cached result if younger than this (milliseconds). + +- `min_age` — integer. Cache-only mode; minimum cache age. + +- `store_in_cache` — boolean. Store result in Firecrawl cache. + +- `lockdown` — boolean. Serve from cache only. + +- `redact_pii` — boolean. Redact PII from content. + +- `profile` — keyword list with `name:` and optional `save_changes:`. Persistent browser profile. + +- `zero_data_retention` — boolean. Enable zero data retention. + +## Interact + +### Why use it + +Run code in the browser session tied to a scrape job. The Elixir SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +Stop the session when done: + +```elixir +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +- `job_id` — string (required, first positional argument). Scrape job ID. + +- `code` — string (required). Code to execute in the browser session. + +- `language` — atom. Runtime for code execution. Values: `:python`, `:node`, `:bash`. + +- `timeout` — integer. Execution timeout in seconds. + +- `origin` — string. Optional origin label for telemetry. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec via `mix run generate.exs`. Function names match the OpenAPI operation names. +- Each function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- All parameters use snake_case keyword lists. They are auto-converted to camelCase maps for the JSON API body. +- Atom values (except `true`, `false`, `nil`) are auto-converted to strings before sending. +- NimbleOptions validates all parameters at the SDK level before the HTTP call. +- There is no client struct or GenServer — the SDK constructs a fresh `Req` client per call. +- No deprecated aliases exist; function names are generated directly from the spec. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..ac2fdf774 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,272 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical Firecrawl Java quickstart for external agents. Generated from SDK source (`firecrawl-java` v1.12.1) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +API key resolution: explicit `.apiKey()` → `FIRECRAWL_API_KEY` env var → `firecrawl.apiKey` system property. A null/blank key is allowed for keyless free-tier access on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchData results = client.search( + "site:docs.firecrawl.dev webhook retries", + SearchOptions.builder() + .limit(5) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build() +); + +List> web = results.getWeb(); +``` + +**Important:** `SearchData` is not directly iterable. Access result buckets via `getWeb()`, `getNews()`, and `getImages()` (each is `List>`, may be null). + +### Parameters + +- `query` — String (required). The search query. + +- `options.sources` — `List`. Values: `"web"`, `"news"`, `"images"`, or `{type: "web"}` maps. + +- `options.categories` — `List`. Values: `"github"`, `"research"`, `"pdf"`, or typed maps. + +- `options.includeDomains` — `List`. Restrict results to these domains. + +- `options.excludeDomains` — `List`. Exclude results from these domains. + +- `options.limit` — Integer. Maximum results to return. + +- `options.tbs` — String. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). + +- `options.location` — String. Geographic location for localized results. + +- `options.ignoreInvalidURLs` — Boolean. Drop URLs that cannot be scraped. + +- `options.highlights` — Boolean. Generate query-relevant highlights. Default: true. + +- `options.timeout` — Integer. Request timeout in milliseconds. + +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each result. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; + +Document doc = client.scrape( + "https://example.com/pricing", + ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. + +- `options.formats` — `List`. Format strings or format config objects. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"` + - Config objects: `JsonFormat.builder().prompt("...").schema(schema).build()`, `QuestionFormat`, `HighlightsFormat` + - Map objects for other formats: `Map.of("type", "screenshot", "fullPage", true, "quality", 80)` + +- `options.headers` — `Map`. Custom HTTP headers. + +- `options.includeTags` — `List`. Only include content from these HTML tags. + +- `options.excludeTags` — `List`. Exclude content from these HTML tags. + +- `options.onlyMainContent` — Boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.waitFor` — Integer. Wait for page to render (milliseconds). + +- `options.mobile` — Boolean. Emulate mobile viewport. + +- `options.parsers` — `List`. Values: `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`. + +- `options.actions` — `List>`. Pre-scrape browser actions. + - `Map.of("type", "click", "selector", "#btn")` + - `Map.of("type", "wait", "milliseconds", 1000)` + - `Map.of("type", "write", "text", "hello")` + - `Map.of("type", "press", "key", "Enter")` + - `Map.of("type", "scroll", "direction", "down")` + - `Map.of("type", "scrape")` + - `Map.of("type", "executeJavascript", "script", "...")` + +- `options.location` — `LocationConfig`. Use `LocationConfig.builder().country("US").languages(List.of("en-US")).build()`. + +- `options.skipTlsVerification` — Boolean. Skip TLS verification. + +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown. + +- `options.blockAds` — Boolean. Block ads and cookie popups. + +- `options.proxy` — String. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom URL. + +- `options.maxAge` — Long. Use cached result if younger than this (milliseconds). + +- `options.storeInCache` — Boolean. Store result in Firecrawl cache. + +- `options.lockdown` — Boolean. Serve from cache only. + +- `options.redactPII` — Boolean. Redact PII from content. + +## Interact + +### Why use it + +Run code in the browser session tied to a scrape job. The Java SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `"node"` +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), null for API default (30s) +- `client.interact(jobId, code, language, timeout, origin)` — optional origin tag + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); +``` + +Stop the session when done: + +```java +import com.firecrawl.models.BrowserDeleteResponse; + +BrowserDeleteResponse stopped = client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +- `jobId` — String (required). Scrape job ID from `document.getMetadata().get("scrapeId")`. + +- `code` — String (required). Code to execute in the browser session. + +- `language` — String. Runtime for code execution. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. + +- `timeout` — Integer. Execution timeout in seconds (1–300). Null uses the API default (30s). + +- `origin` — String. Optional origin label for request attribution. + +### Return value + +`BrowserExecuteResponse` includes: `isSuccess()`, `getStdout()`, `getStderr()`, `getResult()`, `getExitCode()`, `getKilled()`, `getError()`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- All options use the builder pattern: `ScrapeOptions.builder()...build()`. Options are immutable after construction. +- The `formats` field is `List`, accepting both plain strings and typed config objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). +- Every sync method has an `*Async` counterpart returning `CompletableFuture`. +- The SDK auto-adds `origin: "java-sdk@{version}"` to requests. +- Search `location` is a plain `String` (e.g. `"US"`), while scrape `location` is a `LocationConfig` object. +- Search results (`web`, `news`, `images`) use `List>`, not strongly-typed model classes. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..02a77f151 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,257 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical Firecrawl Node.js quickstart for external agents. Generated from SDK source (`firecrawl` v4.32.0) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + limit: 5, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, `results.images`, or `results.developer`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. + +- `options.categories` — array of `"github" | "research" | "pdf" | "developer"` or typed objects. Filters results by category. + +- `options.includeDomains` — string array. Restrict results to these domains. Cannot be used with `excludeDomains`. + +- `options.excludeDomains` — string array. Exclude results from these domains. Cannot be used with `includeDomains`. + +- `options.limit` — number. Maximum results to return. + +- `options.tbs` — string. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week, `"sbd:1,qdr:m"` sorted by date past month). + +- `options.location` — string. Geographic location for localized results (e.g. `"San Francisco,California,United States"`). + +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped by other endpoints. + +- `options.highlights` — boolean. Generate query-relevant highlights. Defaults to `true`. + +- `options.timeout` — number. Request timeout in milliseconds. + +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each search result (same fields as scrape). + +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise options: `"zdr"` for zero data retention, `"anon"` for anonymized search. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `options.formats` — array of format strings or format objects. Output formats to include. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"` + - Object formats (require at least `type`): + - `{ type: "json", prompt?: string, schema?: JSONSchema | ZodSchema }` — LLM-extracted JSON. Zod schemas are auto-converted. + - `{ type: "question", question: string }` — ask a question about the page. + - `{ type: "highlights", query: string }` — find relevant source text. + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` + - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` is required. + - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` + +- `options.headers` — `Record`. Custom HTTP headers. + +- `options.includeTags` — string array. Only include content from these HTML tags. + +- `options.excludeTags` — string array. Exclude content from these HTML tags. + +- `options.onlyMainContent` — boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — number. Timeout in milliseconds. + +- `options.waitFor` — number. Wait for page to render (milliseconds). + +- `options.mobile` — boolean. Emulate mobile viewport. + +- `options.parsers` — array of `"pdf"` or `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }`. + +- `options.actions` — array of action objects for pre-scrape browser actions. + - `{ type: "wait", milliseconds }` or `{ type: "wait", selector }` + - `{ type: "click", selector }` — `all` optional to click all matches + - `{ type: "write", text }` — type text into focused input + - `{ type: "press", key }` + - `{ type: "scroll", direction: "up" | "down" }` — `selector` optional + - `{ type: "scrape" }` — scrape current page content + - `{ type: "executeJavascript", script }` + - `{ type: "screenshot" }` — `fullPage`, `quality`, `viewport` optional + - `{ type: "pdf" }` — `format`, `landscape`, `scale` optional + +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — boolean. Skip TLS verification. + +- `options.removeBase64Images` — boolean. Drop base64 images from markdown. + +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. + +- `options.blockAds` — boolean. Block ads and cookie popups. + +- `options.proxy` — `"basic" | "stealth" | "enhanced" | "auto"` or custom URL string. + +- `options.maxAge` — number. Use cached result if younger than this (milliseconds). + +- `options.minAge` — number. Cache-only mode; minimum cache age in milliseconds. + +- `options.storeInCache` — boolean. Store result in Firecrawl cache. + +- `options.lockdown` — boolean. Serve from cache only, never make outbound request. + +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile shared across scrapes and interactions. + +- `options.redactPII` — boolean or `{ mode?, entities?, replaceStyle? }`. Redact PII from returned content. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. Use `interact` for multi-step flows that go beyond quick pre-scrape actions. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +console.log(result.output); +``` + +Code execution example: + +```ts +const result = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +console.log(result.stdout); +``` + +Stop the session when done: + +```ts +await client.stopInteraction(jobId); +``` + +### Parameters + +- `jobId` — string (required). Scrape job ID from `document.metadata.scrapeId`. + +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage). Optional if `prompt` is provided. + +- `args.prompt` — string. Natural-language instruction for the browser agent. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty. + +- `args.language` — `"python" | "node" | "bash"`. Runtime for code execution. Default: `"node"`. + +- `args.timeout` — number. Execution timeout in seconds (1–300). + +### Return value + +`ScrapeExecuteResponse` includes: `success`, `output` (agent response for prompt), `stdout`, `result`, `stderr`, `exitCode`, `killed`, `liveViewUrl`, `interactiveLiveViewUrl`, `error`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed in `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema. +- The SDK adds 5000ms to user-specified `timeout` for the HTTP request. For `interact`, the padding is `timeout * 1000 + 5000` (interact timeout is in seconds). + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..27238e1bf --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,250 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical Firecrawl Python quickstart for external agents. Generated from SDK source (`firecrawl-py`) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. `FirecrawlApp` is a legacy alias for `Firecrawl`. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + limit=5, + scrape_options={"formats": ["markdown"], "only_main_content": True}, +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, `results.images`, or `results.developer`. Accessing `.data` raises an `AttributeError`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of `"web" | "news" | "images"` or `Source` objects. Controls which sources are searched. + +- `categories` — list of `"github" | "research" | "pdf" | "developer"` or `Category` objects. Filters results by category. + +- `include_domains` — list of str. Restrict results to these domains. Cannot be used with `exclude_domains`. + +- `exclude_domains` — list of str. Exclude results from these domains. Cannot be used with `include_domains`. + +- `limit` — int. Maximum results to return. Default: `5`. + +- `tbs` — str. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). + +- `location` — str. Geographic location for localized results. Note: this is a plain string, not a `Location` object. + +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped by other endpoints. + +- `highlights` — bool. Generate query-relevant highlights. Default: `True`. + +- `timeout` — int. Request timeout in milliseconds. Default: `300000`. + +- `scrape_options` — `ScrapeOptions` or dict. Options for scraping each search result (same fields as scrape). + +- `enterprise` — list of str. Enterprise options: `["zdr"]` for zero data retention, `["anon"]` for anonymized search. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +All parameters after `url` are keyword-only. + +- `url` — str (required). The URL to scrape. + +- `formats` — list of format strings or format dicts. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"` + - Dict formats (require at least `type`): + - `{"type": "json", "prompt": "...", "schema": {...}}` — LLM-extracted JSON. + - `{"type": "question", "question": "..."}` — ask a question about the page. + - `{"type": "highlights", "query": "..."}` — find relevant source text. + - `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {...}}` + - `{"type": "changeTracking", "modes": ["git-diff"], "tag": "..."}` + - `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}` + +- `headers` — dict. Custom HTTP headers. + +- `include_tags` — list of str. Only include content from these HTML tags. + +- `exclude_tags` — list of str. Exclude content from these HTML tags. + +- `only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `timeout` — int. Timeout in milliseconds. + +- `wait_for` — int. Wait for page to render (milliseconds). + +- `mobile` — bool. Emulate mobile viewport. + +- `parsers` — list of `"pdf"` or `PDFParser(mode="auto", max_pages=5)` or dicts. + +- `actions` — list of action dicts for pre-scrape browser actions. + - `{"type": "wait", "milliseconds": 1000}` or `{"type": "wait", "selector": "#el"}` + - `{"type": "click", "selector": "#btn"}` + - `{"type": "write", "text": "hello"}` + - `{"type": "press", "key": "Enter"}` + - `{"type": "scroll", "direction": "down"}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": "..."}` + - `{"type": "screenshot"}` + - `{"type": "pdf"}` + +- `location` — dict with `country` and `languages`. Geo or language-aware scraping. + +- `skip_tls_verification` — bool. Skip TLS verification. + +- `remove_base64_images` — bool. Drop base64 images from markdown. + +- `fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `block_ads` — bool. Block ads and cookie popups. + +- `proxy` — str. One of `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. + +- `max_age` — int. Use cached result if younger than this (milliseconds). + +- `store_in_cache` — bool. Store result in Firecrawl cache. + +- `lockdown` — bool. Serve from cache only, never make outbound request. + +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +print(result.output) +``` + +Code execution example: + +```python +result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) +print(result.stdout) +``` + +Stop the session when done: + +```python +client.stop_interaction(job_id) +``` + +### Parameters + +- `job_id` — str (required). Scrape job ID from `document.metadata.scrape_id`. + +- `code` — str (positional-optional). Code to run in the browser session. Optional if `prompt` is provided. + +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty. + +- `language` — `"python" | "node" | "bash"` (keyword-only). Runtime for code execution. Default: `"node"`. + +- `timeout` — int (keyword-only). Execution timeout in seconds (1–300). + +### Return value + +`BrowserExecuteResponse` includes: `success`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `live_view_url`, `interactive_live_view_url`, `error`. API camelCase fields are normalized to snake_case. + +## Notes + +- All Python parameter names use snake_case. The SDK maps them to the API's camelCase automatically. +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- Search `location` is a plain `str`, not a `Location` object (unlike scrape's `location` which is a dict/object). +- `include_domains` and `exclude_domains` are mutually exclusive — providing both raises `ValueError`. +- An async client is available via `AsyncFirecrawl` (alias `AsyncFirecrawlApp`). + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..fe41e6624 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,284 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical Firecrawl Rust quickstart for external agents. Generated from SDK source (`firecrawl` crate v2.12.1) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +All public types are re-exported from the crate root (`use firecrawl::Client`, `use firecrawl::ScrapeOptions`, etc.). + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +The API key may be `None` or empty for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions, Format}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + println!("{:?}", item); + } +} +``` + +### Parameters + +- `query` — `impl AsRef` (required). The search query. + +- `options` — `impl Into>`. Pass `None` for defaults. + +- `options.sources` — `Vec`. Values: `Web`, `News`, `Images`. + +- `options.categories` — `Vec`. Values: `Github`, `Research`, `Pdf`. + +- `options.include_domains` — `Vec`. Restrict results to these domains. + +- `options.exclude_domains` — `Vec`. Exclude results from these domains. + +- `options.limit` — `u32`. Maximum results. Default: 5, max: 20. + +- `options.tbs` — `String`. Time-based filter (e.g. `"qdr:d"`). + +- `options.location` — `String`. Geographic location for localized results. + +- `options.ignore_invalid_urls` — `bool`. Drop URLs that cannot be scraped. + +- `options.highlights` — `bool`. Generate query-relevant highlights. Default: true. + +- `options.timeout` — `u32`. Request timeout in milliseconds. + +- `options.scrape_options` — `ScrapeOptions`. Options for scraping each result. + +A convenience helper `client.search_and_scrape(query, limit)` returns `Vec` directly. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + ..Default::default() + }) + .await?; + +println!("{:?}", doc.markdown); +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. + +- `options` — `impl Into>`. Pass `None` for defaults. + +- `options.formats` — `Vec`. Output formats. + - Enum variants: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video` + - Parameterized variants: `Question(String)`, `Highlights(String)`, `Query(String)` + +- `options.headers` — `HashMap`. Custom HTTP headers. + +- `options.include_tags` — `Vec`. Only include content from these HTML tags. + +- `options.exclude_tags` — `Vec`. Exclude content from these HTML tags. + +- `options.only_main_content` — `bool`. Strip nav, footer, and other boilerplate. + +- `options.timeout` — `u32`. Timeout in milliseconds. + +- `options.wait_for` — `u32`. Wait for page to render (milliseconds). + +- `options.mobile` — `bool`. Emulate mobile viewport. + +- `options.parsers` — `Vec`. Parser configurations. + - `ParserConfig::Simple("pdf".to_string())` + - `ParserConfig::Pdf { parser_type: "pdf", max_pages: Some(5) }` + +- `options.actions` — `Vec`. Pre-scrape browser actions. + - `Action::Wait { milliseconds, selector }` + - `Action::Click { selector }` + - `Action::Write { text }` + - `Action::Press { key }` + - `Action::Scroll { direction: ScrollDirection::Down, selector }` + - `Action::Scrape` + - `Action::ExecuteJavascript { script }` + - `Action::Screenshot { full_page, quality, viewport }` + - `Action::Pdf { format, landscape, scale }` + +- `options.location` — `LocationConfig` with `country` and `languages`. + +- `options.skip_tls_verification` — `bool`. Skip TLS verification. + +- `options.remove_base64_images` — `bool`. Drop base64 images from markdown. + +- `options.fast_mode` — `bool`. Faster scrapes with reduced fidelity. + +- `options.block_ads` — `bool`. Block ads and cookie popups. + +- `options.proxy` — `ProxyType`. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. + +- `options.max_age` — `u32`. Use cached result if younger than this (seconds). + +- `options.min_age` — `u32`. Cache-only mode; minimum cache age. + +- `options.store_in_cache` — `bool`. Store result in Firecrawl cache. + +- `options.lockdown` — `bool`. Serve from cache only. + +- `options.redact_pii` — `bool`. Redact PII from content. + +- `options.profile` — `ProfileConfig` with `name` and `save_changes`. + +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. + +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. + +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff` or `Json`), `schema`, `prompt`, `tag`. + +- `options.attribute_selectors` — `Vec` with `selector` and `attribute`. + +A convenience helper `client.scrape_with_schema(url, schema, prompt)` combines scrape + JSON extraction. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +println!("{:?}", result.output); +``` + +Stop the session when done: + +```rust +client.stop_interaction(job_id).await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). Scrape job ID. + +- `options` — `ScrapeExecuteOptions` (required, not optional). + +- `options.code` — `Option`. Code to run in the browser session. Optional if `prompt` is provided. + +- `options.prompt` — `Option`. Natural-language instruction. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty, or `FirecrawlError::Misuse` is returned. + +- `options.language` — `ScrapeExecuteLanguage`. Values: `Python`, `Node`, `Bash`. Default: `Node`. + +- `options.timeout` — `u32`. Execution timeout in seconds. + +### Return value + +`ScrapeExecuteResponse` includes: `success`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `live_view_url`, `interactive_live_view_url`, `error`. All fields except `success` are `Option`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. Compiler warnings are emitted for deprecated methods. +- All methods are async and require a `tokio` runtime. +- All request/response structs serialize to camelCase via `#[serde(rename_all = "camelCase")]`. Exception: `redact_pii` is renamed to `"redactPII"`. +- The SDK auto-sets `origin` to `"rust-sdk@{version}"` on requests if not provided. +- Optional parameters use `impl Into>` — pass `None`, a bare value, or `Some(...)`. Exception: `interact()` always requires `ScrapeExecuteOptions`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`