diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..9afe5f1a9 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,213 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for external agents. Generated from SDK source (`firecrawl` hex package **v1.9.1**) and the v2 OpenAPI spec. Function names and parameters match the auto-generated SDK module. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape( + [query: "site:docs.firecrawl.dev webhook retries"], + api_key: "fc-your-api-key" +) +``` + +There is no client struct. All functions are stateless module calls. Per-call options (`api_key`, `base_url`) override application config. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs code execution in a browser session after a scrape. Note: the Elixir SDK supports code-based interactions only (no `prompt` parameter). + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +web_results = res.body["web"] +``` + +Bang variant `search_and_scrape!/2` raises on error instead of returning `{:error, _}`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to scope to a domain. +- `sources` — list of atoms, strings, or maps. Sources to search. Values: `:web`, `:news`, `:images` (or string equivalents, or `%{type: "web"}` maps). Default: `["web"]`. +- `categories` — list of atoms, strings, or maps. Filter by category. Values: `:github`, `:research`, `:pdf`. +- `include_domains` — list of strings. Restrict to these domains. +- `exclude_domains` — list of strings. Exclude these domains. +- `limit` — integer. Max results to return. +- `tbs` — string. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `location` — string. Location for localized results. Note: this is a plain string, not a keyword list. +- `country` — string. ISO 3166-1 alpha-2 country code for geo-targeting (e.g. `"US"`). +- `ignore_invalid_urls` — boolean. Drop URLs that cannot be scraped. +- `timeout` — integer. Timeout in milliseconds. +- `highlights` — boolean. Generate query-relevant highlights. Default: `true`. +- `enterprise` — list of strings. Values: `"zdr"` for zero data retention, `"anon"` for anonymized ZDR. +- `scrape_options` — keyword list. Scrape each search result (see Scrape parameters). + +## Scrape + +### Why use it + +Get structured content from a known URL in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000 +) + +markdown = res.body["data"]["markdown"] +json_data = res.body["data"]["json"] +``` + +Bang variant `scrape_and_extract_from_url!/2` raises on error. + +### Parameters + +- `url` — string (required). The URL to scrape. +- `formats` — list of format strings or format maps. Default: `["markdown"]`. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"`. + - Map formats (require `type` plus additional fields): + - `%{type: "json", prompt: ..., schema: ...}` — LLM-extracted JSON. + - `%{type: "question", question: ...}` — natural-language question. + - `%{type: "highlights", query: ...}` — relevant source text. + - `%{type: "screenshot", fullPage: true, quality: 80, viewport: %{width: 1280, height: 720}}` — screenshot with options. + - `%{type: "changeTracking", modes: ["git-diff"], tag: "..."}` — change tracking. + - `%{type: "attributes", selectors: [%{selector: "a", attribute: "href"}]}` — attribute extraction. +- `headers` — map. Custom request headers. +- `include_tags` — list of strings. HTML tags to include. +- `exclude_tags` — list of strings. HTML tags to exclude. +- `only_main_content` — boolean. Strip nav, footer, boilerplate. Default: `true`. +- `timeout` — integer. Timeout in milliseconds. Min 1000, default 60000, max 300000. +- `wait_for` — integer. Wait time in milliseconds before scraping. +- `mobile` — boolean. Use mobile viewport. +- `parsers` — list of strings or maps. Values: `"pdf"` or `%{type: "pdf", mode: "fast" | "auto" | "ocr", maxPages: n}`. +- `actions` — list of action maps. Browser actions before scraping. + - `%{type: "wait", milliseconds: n}` or `%{type: "wait", selector: "..."}` — wait for time or element. + - `%{type: "click", selector: "...", all: false}` — click element(s). + - `%{type: "write", text: "..."}` — type text into focused input. + - `%{type: "press", key: "..."}` — press a keyboard key. + - `%{type: "scroll", direction: "up" | "down"}` — scroll up or down. + - `%{type: "scrape"}` — capture current page state. + - `%{type: "executeJavascript", script: "..."}` — run JS on the page. +- `location` — keyword list with `country:` and `languages:`. Geo/language-aware scraping. +- `skip_tls_verification` — boolean. Skip TLS verification. +- `remove_base64_images` — boolean. Drop base64 images from markdown. +- `block_ads` — boolean. Block ads and cookie popups. +- `proxy` — atom. Values: `:basic`, `:enhanced`, `:auto`. +- `max_age` — integer. Use cached data if younger than this (milliseconds). +- `min_age` — integer. Cache-only mode; min age of cached data (ms). Set to `1` for any cached data. +- `store_in_cache` — boolean. Cache the result. +- `lockdown` — boolean. Serve only cached results, no outbound requests. +- `redact_pii` — boolean. Redact PII from returned content. +- `profile` — keyword list with `name:` and optional `save_changes:`. Persistent browser profile. +- `audit_metadata` — keyword list with `username:`. User attribution for SIEM logging. +- `zero_data_retention` — boolean. Enable zero data retention. + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Requires a scrape job ID from a prior scrape. + +**Note:** The Elixir SDK supports code-based interactions only. There is no `prompt` parameter for natural-language browser instructions (unlike Node.js, Python, and Rust SDKs). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` → `{:ok, Req.Response.t()} | {:error, Exception.t()}` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) + +IO.inspect(res.body) + +# End the session when done +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +Bang variant `interact_with_scrape_browser_session!/3` raises on error. + +### Parameters + +- `job_id` — string (required, first argument). Scrape job ID. +- `code` — string (required). Code to execute in the browser session. +- `language` — atom or string. `:python`, `:node`, or `:bash`. Default: `:node`. +- `timeout` — integer. Execution timeout in seconds. +- `origin` — string. Origin label for execution telemetry. + +Stop the session with `Firecrawl.stop_interactive_scrape_browser_session(job_id)`. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec. Function names are spec-derived, not hand-written aliases. +- Every public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- There are no deprecated aliases in the Elixir SDK. +- The Elixir SDK does not support `prompt` on `interact_with_scrape_browser_session` — only code-based interactions. +- Parameters use snake_case in Elixir, mapped to camelCase for the API request body. +- `location` is a keyword list (with `country:`, `languages:`) on scrape but a plain string on search. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..28a29265c --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,237 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for external agents. Generated from SDK source (`firecrawl-java` **v1.12.1**) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment automatically: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +`fromEnv()` reads `FIRECRAWL_API_KEY` env var, then falls back to `firecrawl.apiKey` system property. + +Builder options: `apiUrl(String)`, `timeoutMs(long)` (default 300000), `maxRetries(int)` (default 3), `backoffFactor(double)` (default 0.5), `asyncExecutor(Executor)`, `httpClient(OkHttpClient)`. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs code execution in a browser session after a scrape. Note: the Java SDK supports code-based interactions only (no `prompt` parameter). + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +Results are in `getWeb()`, `getNews()`, `getImages()` (each `List>`, may be null). Do not treat `SearchData` as a directly iterable list. + +### Parameters + +- `query` — String (required). The search query. Use `site:example.com` to scope to a domain. +- `options.sources` — `List`. Sources to search. Values: `"web"`, `"news"`, `"images"`, or `{type: "web" | "news" | "images"}` maps. +- `options.categories` — `List`. Filter by category. Values: `"github"`, `"research"`, `"pdf"`. +- `options.includeDomains` — `List`. Restrict to these domains. +- `options.excludeDomains` — `List`. Exclude these domains. +- `options.limit` — Integer. Max results to return. +- `options.tbs` — String. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `options.location` — String. Location for localized results. +- `options.ignoreInvalidURLs` — Boolean. Drop URLs that cannot be scraped. +- `options.timeout` — Integer. Timeout in milliseconds. +- `options.highlights` — Boolean. Generate query-relevant highlights. Default: `true`. +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `options.integration` — String. Integration identifier. + +## Scrape + +### Why use it + +Get structured content from a known URL in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. +- `options.formats` — `List`. Output formats. Accepts plain strings and typed objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. + - `JsonFormat.builder().prompt("...").schema(...).build()` — LLM-extracted JSON. + - Format maps for screenshot, changeTracking, attributes options. +- `options.headers` — `Map`. Custom request headers. +- `options.includeTags` — `List`. HTML tags to include. +- `options.excludeTags` — `List`. HTML tags to exclude. +- `options.onlyMainContent` — Boolean. Strip nav, footer, boilerplate. Default: `true`. +- `options.timeout` — Integer. Timeout in milliseconds. Default: `60000`. +- `options.waitFor` — Integer. Wait time in milliseconds before scraping. +- `options.mobile` — Boolean. Use mobile viewport. +- `options.parsers` — `List`. Parser config. Values: `"pdf"` or `{"type": "pdf", "maxPages": n}`. +- `options.actions` — `List>`. Browser actions before scraping. + - `{"type": "wait", "milliseconds": n}` or `{"type": "wait", "selector": "..."}` — wait for time or element. + - `{"type": "click", "selector": "..."}` — click an element. + - `{"type": "write", "text": "..."}` — type text into focused input. + - `{"type": "press", "key": "..."}` — press a keyboard key. + - `{"type": "scroll", "direction": "up" | "down"}` — scroll up or down. + - `{"type": "scrape"}` — capture current page state. + - `{"type": "executeJavascript", "script": "..."}` — run JS on the page. + - `{"type": "screenshot", "fullPage": true}` — take a screenshot. + - `{"type": "pdf", "format": "Letter"}` — generate PDF. +- `options.location` — `LocationConfig` with `country` and `languages`. Geo/language-aware scraping. +- `options.skipTlsVerification` — Boolean. Skip TLS verification. +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown. +- `options.blockAds` — Boolean. Block ads and cookie popups. +- `options.proxy` — String. `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. +- `options.maxAge` — Long. Use cached data if younger than this (milliseconds). +- `options.storeInCache` — Boolean. Cache the result. +- `options.lockdown` — Boolean. Serve only cached results, no outbound requests. +- `options.redactPII` — Boolean. Redact PII from returned content. +- `options.auditMetadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. +- `options.integration` — String. Integration identifier. + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. Requires a scrape job ID from a prior scrape. + +**Note:** The Java SDK supports code-based interactions only. There is no `prompt` parameter for natural-language browser instructions (unlike Node.js, Python, and Rust SDKs). + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` → `BrowserExecuteResponse` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape( + "https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build() +); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); + +// End the session when done +client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +- `jobId` — String (required). Scrape job ID. +- `code` — String (required). Code to execute in the browser session. +- `language` — String. `"python"`, `"node"`, or `"bash"`. Default: `"node"`. +- `timeout` — Integer. Execution timeout in seconds (1–300). Default: `30`. Pass `null` to use API default. +- `origin` — String (optional overload). Origin label for request attribution. + +Async variants: `interactAsync(...)` returns `CompletableFuture`. + +Stop the session with `client.stopInteractiveBrowser(jobId)`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- `QueryFormat` is deprecated; use `QuestionFormat` or `HighlightsFormat` instead. +- The Java SDK does not support `prompt` on `interact` — only code-based interactions. +- All `ScrapeOptions` built via builder are immutable after `build()`. Use `toBuilder()` to copy and modify. +- Async methods (e.g. `scrapeAsync`, `searchAsync`, `interactAsync`) return `CompletableFuture`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/Document.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..1c1f58897 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,213 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for external agents. Generated from SDK source (`firecrawl` **v4.32.0**, published as `@mendable/firecrawl-js`) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +Constructor also accepts a plain API key string: `new Firecrawl("fc-...")`. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Results are in `result.web`, `result.news`, `result.images`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to scope to a domain. +- `options.sources` — array of `"web"` | `"news"` | `"images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. +- `options.categories` — array of `"github"` | `"research"` | `"pdf"` | `"developer"`. Filters results by category. +- `options.includeDomains` — string array. Restrict to these domains. Cannot combine with `excludeDomains`. +- `options.excludeDomains` — string array. Exclude these domains. Cannot combine with `includeDomains`. +- `options.limit` — number. Max results to return. +- `options.tbs` — string. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week, `"sbd:1,qdr:m"` for sorted by date, past month). +- `options.location` — string. Location for localized results (e.g. `"San Francisco,California,United States"`). +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped by other endpoints. +- `options.timeout` — number. Timeout in milliseconds. +- `options.highlights` — boolean. Generate query-relevant highlights. Default: `true`. +- `options.scrapeOptions` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `options.enterprise` — array of `"default"` | `"anon"` | `"zdr"`. Enterprise zero data retention options. +- `options.threatProtection` — object. Enterprise threat protection override. +- `options.integration` — string. Integration identifier. + +## Scrape + +### Why use it + +Get structured content from a known URL in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. +- `options.formats` — array of format strings or format objects. Default: `["markdown"]`. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Object formats (require `type` plus additional fields): + - `{ type: "json", prompt?, schema? }` — LLM-extracted JSON. At least one of `prompt` or `schema` required. Accepts Zod schemas. + - `{ type: "question", question }` — natural-language question about the page. + - `{ type: "highlights", query }` — find relevant source text. + - `{ type: "screenshot", fullPage?, quality?, viewport? }` — screenshot with options. + - `{ type: "changeTracking", modes, schema?, prompt?, tag? }` — change tracking. `modes` required: `["git-diff"]` and/or `["json"]`. + - `{ type: "attributes", selectors }` — extract element attributes. `selectors` is `[{ selector, attribute }]`. +- `options.headers` — `Record`. Custom request headers. +- `options.includeTags` — string array. HTML tags to include. +- `options.excludeTags` — string array. HTML tags to exclude. +- `options.onlyMainContent` — boolean. Strip nav, footer, boilerplate. Default: `true`. +- `options.timeout` — number. Timeout in milliseconds. Default: `60000`. +- `options.waitFor` — number. Wait time in milliseconds before scraping. +- `options.mobile` — boolean. Use mobile viewport. +- `options.parsers` — array of `"pdf"` or `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }`. Default: `["pdf"]`. +- `options.actions` — array of action objects. Browser actions before scraping. + - `{ type: "wait", milliseconds }` or `{ type: "wait", selector }` — wait for time or element. + - `{ type: "click", selector, all? }` — click element(s). + - `{ type: "write", text }` — type text into focused input. + - `{ type: "press", key }` — press a keyboard key. + - `{ type: "scroll", direction?, selector? }` — scroll up or down. + - `{ type: "scrape" }` — capture current page state. + - `{ type: "executeJavascript", script }` — run JS on the page. + - `{ type: "screenshot", fullPage?, quality?, viewport? }` — take a screenshot. + - `{ type: "pdf", format?, landscape?, scale? }` — generate PDF. +- `options.location` — `{ country?: string, languages?: string[] }`. Geo/language-aware scraping. +- `options.skipTlsVerification` — boolean. Skip TLS verification. Default: `true`. +- `options.removeBase64Images` — boolean. Drop base64 images from markdown. Default: `true`. +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. +- `options.blockAds` — boolean. Block ads and cookie popups. Default: `true`. +- `options.proxy` — `"basic"` | `"stealth"` | `"enhanced"` | `"auto"` or a custom proxy URL. Default: `"auto"`. +- `options.maxAge` — number. Use cached data if younger than this (milliseconds). Default: `172800000` (2 days). +- `options.minAge` — number. Cache-only mode; min age of cached data (ms). Set to `1` for any cached data. +- `options.storeInCache` — boolean. Cache the result. Default: `true`. +- `options.lockdown` — boolean. Serve only cached results, no outbound requests. +- `options.redactPII` — boolean or `{ mode?, entities?, replaceStyle? }`. Redact PII from returned content. +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile across scrapes and interactions. +- `options.auditMetadata` — `{ username: string }`. User attribution for SIEM logging. +- `options.threatProtection` — object. Enterprise threat protection override. +- `options.integration` — string. Integration identifier. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a `scrapeId` from a prior scrape's `metadata`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +// Natural-language interaction +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Or code-based interaction +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +// End the session when done +await client.stopInteraction(jobId); +``` + +### Parameters + +- `jobId` — string (required). Scrape job ID from `document.metadata.scrapeId`. +- `args.code` — string. Code to execute in the browser session. +- `args.prompt` — string. Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be non-empty. +- `args.language` — `"python"` | `"node"` | `"bash"`. Execution runtime. Default: `"node"`. +- `args.timeout` — number. Execution timeout in seconds (1–300). Default: `30`. + +Stop the session with `client.stopInteraction(jobId)`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema by the SDK. +- `SearchData` has a hidden `.data` getter that throws an error directing you to use `.web`, `.news`, `.images` instead. +- The package declares Node.js >= 22 in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..eb5601d47 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,213 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for external agents. Generated from SDK source (`firecrawl-py` **v4.22.1**) and the v2 OpenAPI spec. Method names and parameters match the v2 client. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +Constructor also accepts `timeout` (seconds), `max_retries` (default 3), and `backoff_factor` (default 0.5). + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a `scrape_id` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Results are in `result.web`, `result.news`, `result.images`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to scope to a domain. +- `sources` — list of `"web"` | `"news"` | `"images"` or `Source` objects. Controls which sources are searched. +- `categories` — list of `"github"` | `"research"` | `"pdf"` | `"developer"` or `Category` objects. Filters results by category. +- `include_domains` — list of str. Restrict to these domains. Cannot combine with `exclude_domains`. +- `exclude_domains` — list of str. Exclude these domains. Cannot combine with `include_domains`. +- `limit` — int. Max results to return. Default: `5`. +- `tbs` — str. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). +- `location` — str. Location for localized results. Note: this is a plain string, not a `Location` object. +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped. +- `timeout` — int. Timeout in milliseconds. Default: `300000`. +- `highlights` — bool. Generate query-relevant highlights. Default: `True`. +- `scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `enterprise` — list of `"zdr"` | `"anon"`. Enterprise zero data retention options. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `integration` — str. Integration identifier. + +## Scrape + +### Why use it + +Get structured content from a known URL in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +- `url` — str (required). The URL to scrape. +- `formats` — list of format strings or format dicts. Default: `["markdown"]`. + - Plain strings: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. + - Dict formats (require `type` plus additional fields): + - `{"type": "json", "prompt": ..., "schema": ...}` — LLM-extracted JSON. Use a dict, not the plain string `"json"`. + - `{"type": "question", "question": ...}` — natural-language question about the page. + - `{"type": "highlights", "query": ...}` — find relevant source text. + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` — screenshot with options. + - `{"type": "changeTracking", "modes": [...], "schema": ..., "prompt": ..., "tag": ...}` — change tracking. `modes` required. + - `{"type": "attributes", "selectors": [{"selector": ..., "attribute": ...}]}` — extract element attributes. +- `headers` — dict. Custom request headers. +- `include_tags` — list of str. HTML tags to include. +- `exclude_tags` — list of str. HTML tags to exclude. +- `only_main_content` — bool. Strip nav, footer, boilerplate. Default: `True`. +- `timeout` — int. Timeout in milliseconds. Default: `60000`. +- `wait_for` — int. Wait time in milliseconds before scraping. +- `mobile` — bool. Use mobile viewport. +- `parsers` — list of `"pdf"` or `{"type": "pdf", "mode": "fast" | "auto" | "ocr", "max_pages": int}`. Default: `["pdf"]`. +- `actions` — list of action dicts. Browser actions before scraping. + - `{"type": "wait", "milliseconds": ...}` or `{"type": "wait", "selector": ...}` + - `{"type": "click", "selector": ...}` + - `{"type": "write", "text": ...}` + - `{"type": "press", "key": ...}` + - `{"type": "scroll", "direction": "up" | "down", "selector": ...}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": ...}` + - `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` + - `{"type": "pdf", "format": ..., "landscape": ..., "scale": ...}` +- `location` — dict with `country` and `languages`. Geo/language-aware scraping. +- `skip_tls_verification` — bool. Skip TLS verification. Default: `True`. +- `remove_base64_images` — bool. Drop base64 images from markdown. Default: `True`. +- `fast_mode` — bool. Faster scrapes with reduced fidelity. +- `block_ads` — bool. Block ads and cookie popups. Default: `True`. +- `proxy` — str. `"basic"`, `"stealth"`, `"enhanced"`, or `"auto"`. Default: `"auto"`. +- `max_age` — int. Use cached data if younger than this (milliseconds). Default: `172800000` (2 days). +- `store_in_cache` — bool. Cache the result. Default: `True`. +- `lockdown` — bool. Serve only cached results, no outbound requests. +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile. +- `audit_metadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. +- `threat_protection` — `ThreatProtectionOptions`. Enterprise threat protection override. +- `integration` — str. Integration identifier. + +Note: `min_age` is available on `ScrapeOptions` when passed via `scrape_options` to `search()`, but not as a top-level kwarg on `scrape()`. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a `scrape_id` from a prior scrape's metadata. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +# Natural-language interaction +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") + +# Or code-based interaction +code_result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) + +# End the session when done +client.stop_interaction(job_id) +``` + +### Parameters + +- `job_id` — str (required). Scrape job ID from `document.metadata.scrape_id`. +- `code` — str. Code to execute in the browser session (positional arg). +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be non-empty. +- `language` — `"python"` | `"node"` | `"bash"`. Execution runtime. Default: `"node"`. +- `timeout` — int. Execution timeout in seconds (1–300). Default: `30`. + +Stop the session with `client.stop_interaction(job_id)`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `FirecrawlApp` is a backward-compatible alias for `Firecrawl`. +- `AsyncFirecrawl` (also `AsyncFirecrawlApp`) provides async versions of all methods. +- `SearchData` raises `AttributeError` with a helpful message if you access `.data` — use `.web`, `.news`, `.images` instead. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..aac70050e --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,239 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for external agents. Generated from SDK source (`firecrawl` crate **v2.12.1**) and the v2 OpenAPI spec. Method names and parameters match the SDK public API. + +## Install + +```toml +[dependencies] +firecrawl = "2.12.1" +tokio = { version = "^1", features = ["full"] } +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +All public types are exported at the crate root: `use firecrawl::Client`, `use firecrawl::ScrapeOptions`, etc. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:` in the query. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + println!("{:?}", item); + } +} +``` + +### Parameters + +- `query` — `impl AsRef` (required). The search query. Use `site:example.com` to scope to a domain. +- `options.sources` — `Vec`. Sources to search. Values: `Web`, `News`, `Images`. +- `options.categories` — `Vec`. Filter by category. Values: `Github`, `Research`, `Pdf`. +- `options.include_domains` — `Vec`. Restrict to these domains. +- `options.exclude_domains` — `Vec`. Exclude these domains. +- `options.limit` — `u32`. Max results. Default: 5, max: 20. +- `options.tbs` — `String`. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). +- `options.location` — `String`. Location for localized results. +- `options.ignore_invalid_urls` — `bool`. Drop URLs that cannot be scraped. +- `options.timeout` — `u32`. Timeout in milliseconds. +- `options.highlights` — `bool`. Generate query-relevant highlights. Default: `true`. +- `options.scrape_options` — `ScrapeOptions`. Scrape each search result (see Scrape parameters). +- `options.integration` — `String`. Integration identifier. + +## Scrape + +### Why use it + +Get structured content from a known URL in one or more formats (markdown, HTML, JSON extraction, screenshots, etc.). + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + ..Default::default() + }) + .await?; + +println!("{:?}", doc.markdown); +println!("{:?}", doc.json); +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. +- `options.formats` — `Vec`. Output formats. Values: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(String)`, `Highlights(String)`. +- `options.headers` — `HashMap`. Custom request headers. +- `options.include_tags` — `Vec`. HTML tags to include. +- `options.exclude_tags` — `Vec`. HTML tags to exclude. +- `options.only_main_content` — `bool`. Strip nav, footer, boilerplate. Default: `true`. +- `options.timeout` — `u32`. Timeout in milliseconds. Default: `60000`. +- `options.wait_for` — `u32`. Wait time in milliseconds before scraping. +- `options.mobile` — `bool`. Use mobile viewport. +- `options.parsers` — `Vec`. Parser config. Values: `Simple("pdf".to_string())`, `Pdf { parser_type, mode, max_pages }`. +- `options.actions` — `Vec`. Browser actions before scraping. + - `Action::Wait { milliseconds, selector }` — wait for time or element. + - `Action::Click { selector }` — click an element. + - `Action::Write { text }` — type text into focused input. + - `Action::Press { key }` — press a keyboard key. + - `Action::Scroll { direction, selector }` — scroll up or down. + - `Action::Scrape` — capture current page state. + - `Action::ExecuteJavascript { script }` — run JS on the page. + - `Action::Screenshot { full_page, quality, viewport }` — take a screenshot. + - `Action::Pdf { format, landscape, scale }` — generate PDF. +- `options.location` — `LocationConfig` with `country` and `languages`. Geo/language-aware scraping. +- `options.skip_tls_verification` — `bool`. Skip TLS verification. +- `options.remove_base64_images` — `bool`. Drop base64 images from markdown. +- `options.fast_mode` — `bool`. Faster scrapes with reduced fidelity. +- `options.block_ads` — `bool`. Block ads and cookie popups. +- `options.proxy` — `ProxyType`. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. +- `options.max_age` — `u32`. Use cached data if younger than this (milliseconds). +- `options.min_age` — `u32`. Cache-only mode; min age of cached data. +- `options.store_in_cache` — `bool`. Cache the result. +- `options.lockdown` — `bool`. Serve only cached results, no outbound requests. +- `options.redact_pii` — `bool`. Redact PII from returned content. +- `options.profile` — `ProfileConfig` with `name` and optional `save_changes`. Persistent browser profile. +- `options.audit_metadata` — `AuditMetadata` with `username`. User attribution for SIEM logging. +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. JSON extraction config. +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. Screenshot config. +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff` | `Json`), `schema`, `prompt`, `tag`. Change tracking config. +- `options.attribute_selectors` — `Vec` with `selector` and `attribute`. Attribute extraction config. +- `options.integration` — `String`. Integration identifier. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job. Use for clicks, form fills, code execution, or natural-language browser instructions. Requires a scrape job ID from a prior scrape. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +// Natural-language interaction +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Or code-based interaction +let code_result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; + +// End the session when done +client.stop_interaction(job_id).await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). Scrape job ID. +- `options.code` — `String`. Code to execute in the browser session. +- `options.prompt` — `String`. Natural-language instruction for the browser agent. +- At least one of `code` or `prompt` must be non-empty; otherwise returns `FirecrawlError::Misuse`. +- `options.language` — `ScrapeExecuteLanguage`. Values: `Python`, `Node`, `Bash`. Default: `Node`. +- `options.timeout` — `u32`. Execution timeout in seconds (1–300). Default: `30`. + +Stop the session with `client.stop_interaction(job_id)`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` includes dedicated `json_options`, `screenshot_options`, and `change_tracking_options` for advanced format configuration. +- Convenience helper: `search_and_scrape(query, limit)` calls `search` with default `ScrapeOptions` and returns `Vec` from the web results. +- All v2 types are exported at the crate root: `use firecrawl::Client`, not `use firecrawl::v2::Client`. +- All options use `#[derive(Default)]` so `..Default::default()` fills all `None` values. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`