diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..6c0da9ef7 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,237 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical Firecrawl Elixir quickstart for external agents. Generated from SDK source (`firecrawl` hex package v1.9.1) and the v2 OpenAPI spec. The Elixir SDK is auto-generated from the OpenAPI spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape( + [query: "site:docs.firecrawl.dev webhook retries"], + api_key: "fc-your-api-key" +) +``` + +There is no client struct or process. The SDK is a single flat module (`Firecrawl`) with module-level functions. Auth is configured globally via application config or per-call via opts. All functions also accept `:base_url` in opts to override the default API URL. + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + limit: 5, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +Parameters are passed as a keyword list. Snake_case keys are auto-converted to camelCase for the API. + +- `query` — string (required). The search query. + +- `sources` — list. Sources to search. Default: `["web"]`. Values: `:web`, `:news`, `:images`, or `%{type: "web"}` maps. + +- `categories` — list. Category filter. Values: `:github`, `:research`, `:pdf`, or typed maps. + +- `include_domains` — list of strings. Restrict results to these domains. + +- `exclude_domains` — list of strings. Exclude results from these domains. + +- `limit` — integer. Maximum results to return. + +- `tbs` — string. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). + +- `location` — string. Geographic location for localized results. + +- `country` — string. ISO 3166-1 alpha-2 code for geo-targeting (e.g. `"US"`). + +- `ignore_invalid_urls` — boolean. Drop URLs that cannot be scraped. + +- `highlights` — boolean. Generate query-relevant highlights. Default: true. + +- `timeout` — integer. Request timeout in milliseconds. + +- `enterprise` — list of strings. Enterprise options: `["zdr"]` for zero data retention, `["anon"]` for anonymized. + +- `scrape_options` — keyword list. Options for scraping each search result (same fields as scrape). + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true +) +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `formats` — list of format strings or format maps. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"` + - Map formats: `%{type: "json", prompt: "..."}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}` + +- `headers` — map. Custom HTTP headers. + +- `include_tags` — list of strings. Only include content from these HTML tags. + +- `exclude_tags` — list of strings. Exclude content from these HTML tags. + +- `only_main_content` — boolean. Strip nav, footer, and other boilerplate. + +- `timeout` — integer. Timeout in milliseconds. Default: 60000, min: 1000, max: 300000. + +- `wait_for` — integer. Wait for page to render (milliseconds). + +- `mobile` — boolean. Emulate mobile viewport. + +- `parsers` — list. Values: `"pdf"` or `%{type: "pdf", mode: "auto", maxPages: 5}`. + +- `actions` — list of action maps. Pre-scrape browser actions. + - `%{type: "click", selector: "#btn"}` + - `%{type: "wait", milliseconds: 1000}` or `%{type: "wait", selector: "#el"}` + - `%{type: "write", text: "hello"}` + - `%{type: "press", key: "Enter"}` + - `%{type: "scroll", direction: "down"}` + - `%{type: "scrape"}` + - `%{type: "executeJavascript", script: "..."}` + +- `location` — keyword list with `country:` and `languages:`. Geo or language-aware scraping. + +- `skip_tls_verification` — boolean. Skip TLS verification. + +- `remove_base64_images` — boolean. Drop base64 images from markdown. + +- `block_ads` — boolean. Block ads and cookie popups. + +- `proxy` — atom. Values: `:basic`, `:enhanced`, `:auto`. Enhanced costs up to 5 credits. + +- `max_age` — integer. Use cached result if younger than this (milliseconds). + +- `min_age` — integer. Cache-only mode; minimum cache age. + +- `store_in_cache` — boolean. Store result in Firecrawl cache. + +- `lockdown` — boolean. Serve from cache only. + +- `redact_pii` — boolean. Redact PII from content. + +- `profile` — keyword list with `name:` and optional `save_changes:`. Persistent browser profile. + +- `zero_data_retention` — boolean. Enable zero data retention. + +## Interact + +### Why use it + +Run code in the browser session tied to a scrape job. The Elixir SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +Stop the session when done: + +```elixir +{:ok, _} = Firecrawl.stop_interactive_scrape_browser_session(job_id) +``` + +### Parameters + +- `job_id` — string (required, first positional argument). Scrape job ID. + +- `code` — string (required). Code to execute in the browser session. + +- `language` — atom. Runtime for code execution. Values: `:python`, `:node`, `:bash`. + +- `timeout` — integer. Execution timeout in seconds. + +- `origin` — string. Optional origin label for telemetry. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec via `mix run generate.exs`. Function names match the OpenAPI operation names. +- Each function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- All parameters use snake_case keyword lists. They are auto-converted to camelCase maps for the JSON API body. +- Atom values (except `true`, `false`, `nil`) are auto-converted to strings before sending. +- NimbleOptions validates all parameters at the SDK level before the HTTP call. +- There is no client struct or GenServer — the SDK constructs a fresh `Req` client per call. +- No deprecated aliases exist; function names are generated directly from the spec. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..ac2fdf774 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,272 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical Firecrawl Java quickstart for external agents. Generated from SDK source (`firecrawl-java` v1.12.1) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment: +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +API key resolution: explicit `.apiKey()` → `FIRECRAWL_API_KEY` env var → `firecrawl.apiKey` system property. A null/blank key is allowed for keyless free-tier access on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; + +SearchData results = client.search( + "site:docs.firecrawl.dev webhook retries", + SearchOptions.builder() + .limit(5) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build() +); + +List> web = results.getWeb(); +``` + +**Important:** `SearchData` is not directly iterable. Access result buckets via `getWeb()`, `getNews()`, and `getImages()` (each is `List>`, may be null). + +### Parameters + +- `query` — String (required). The search query. + +- `options.sources` — `List`. Values: `"web"`, `"news"`, `"images"`, or `{type: "web"}` maps. + +- `options.categories` — `List`. Values: `"github"`, `"research"`, `"pdf"`, or typed maps. + +- `options.includeDomains` — `List`. Restrict results to these domains. + +- `options.excludeDomains` — `List`. Exclude results from these domains. + +- `options.limit` — Integer. Maximum results to return. + +- `options.tbs` — String. Time-based filter (e.g. `"qdr:d"`, `"qdr:w"`). + +- `options.location` — String. Geographic location for localized results. + +- `options.ignoreInvalidURLs` — Boolean. Drop URLs that cannot be scraped. + +- `options.highlights` — Boolean. Generate query-relevant highlights. Default: true. + +- `options.timeout` — Integer. Request timeout in milliseconds. + +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each result. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; + +Document doc = client.scrape( + "https://example.com/pricing", + ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .build() +); + +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +- `url` — String (required). The URL to scrape. + +- `options.formats` — `List`. Format strings or format config objects. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"` + - Config objects: `JsonFormat.builder().prompt("...").schema(schema).build()`, `QuestionFormat`, `HighlightsFormat` + - Map objects for other formats: `Map.of("type", "screenshot", "fullPage", true, "quality", 80)` + +- `options.headers` — `Map`. Custom HTTP headers. + +- `options.includeTags` — `List`. Only include content from these HTML tags. + +- `options.excludeTags` — `List`. Exclude content from these HTML tags. + +- `options.onlyMainContent` — Boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — Integer. Timeout in milliseconds. + +- `options.waitFor` — Integer. Wait for page to render (milliseconds). + +- `options.mobile` — Boolean. Emulate mobile viewport. + +- `options.parsers` — `List`. Values: `"pdf"` or `Map.of("type", "pdf", "maxPages", 5)`. + +- `options.actions` — `List>`. Pre-scrape browser actions. + - `Map.of("type", "click", "selector", "#btn")` + - `Map.of("type", "wait", "milliseconds", 1000)` + - `Map.of("type", "write", "text", "hello")` + - `Map.of("type", "press", "key", "Enter")` + - `Map.of("type", "scroll", "direction", "down")` + - `Map.of("type", "scrape")` + - `Map.of("type", "executeJavascript", "script", "...")` + +- `options.location` — `LocationConfig`. Use `LocationConfig.builder().country("US").languages(List.of("en-US")).build()`. + +- `options.skipTlsVerification` — Boolean. Skip TLS verification. + +- `options.removeBase64Images` — Boolean. Drop base64 images from markdown. + +- `options.blockAds` — Boolean. Block ads and cookie popups. + +- `options.proxy` — String. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or a custom URL. + +- `options.maxAge` — Long. Use cached result if younger than this (milliseconds). + +- `options.storeInCache` — Boolean. Store result in Firecrawl cache. + +- `options.lockdown` — Boolean. Serve from cache only. + +- `options.redactPII` — Boolean. Redact PII from content. + +## Interact + +### Why use it + +Run code in the browser session tied to a scrape job. The Java SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — uses default language `"node"` +- `client.interact(jobId, code, language, timeout)` — `timeout` is seconds (1–300), null for API default (30s) +- `client.interact(jobId, code, language, timeout, origin)` — optional origin tag + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); + +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); +``` + +Stop the session when done: + +```java +import com.firecrawl.models.BrowserDeleteResponse; + +BrowserDeleteResponse stopped = client.stopInteractiveBrowser(jobId); +``` + +### Parameters + +- `jobId` — String (required). Scrape job ID from `document.getMetadata().get("scrapeId")`. + +- `code` — String (required). Code to execute in the browser session. + +- `language` — String. Runtime for code execution. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. + +- `timeout` — Integer. Execution timeout in seconds (1–300). Null uses the API default (30s). + +- `origin` — String. Optional origin label for request attribution. + +### Return value + +`BrowserExecuteResponse` includes: `isSuccess()`, `getStdout()`, `getStderr()`, `getResult()`, `getExitCode()`, `getKilled()`, `getError()`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- All options use the builder pattern: `ScrapeOptions.builder()...build()`. Options are immutable after construction. +- The `formats` field is `List`, accepting both plain strings and typed config objects (`JsonFormat`, `QuestionFormat`, `HighlightsFormat`). +- Every sync method has an `*Async` counterpart returning `CompletableFuture`. +- The SDK auto-adds `origin: "java-sdk@{version}"` to requests. +- Search `location` is a plain `String` (e.g. `"US"`), while scrape `location` is a `LocationConfig` object. +- Search results (`web`, `news`, `images`) use `List>`, not strongly-typed model classes. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..02a77f151 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,257 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical Firecrawl Node.js quickstart for external agents. Generated from SDK source (`firecrawl` v4.32.0) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL env var +}); +``` + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + limit: 5, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, `results.images`, or `results.developer`. + +### Parameters + +- `query` — string (required). The search query. Use `site:example.com` to limit results to a domain. + +- `options.sources` — array of `"web" | "news" | "images"` or `{ type: "web" | "news" | "images" }`. Controls which sources are searched. + +- `options.categories` — array of `"github" | "research" | "pdf" | "developer"` or typed objects. Filters results by category. + +- `options.includeDomains` — string array. Restrict results to these domains. Cannot be used with `excludeDomains`. + +- `options.excludeDomains` — string array. Exclude results from these domains. Cannot be used with `includeDomains`. + +- `options.limit` — number. Maximum results to return. + +- `options.tbs` — string. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week, `"sbd:1,qdr:m"` sorted by date past month). + +- `options.location` — string. Geographic location for localized results (e.g. `"San Francisco,California,United States"`). + +- `options.ignoreInvalidURLs` — boolean. Drop URLs that cannot be scraped by other endpoints. + +- `options.highlights` — boolean. Generate query-relevant highlights. Defaults to `true`. + +- `options.timeout` — number. Request timeout in milliseconds. + +- `options.scrapeOptions` — `ScrapeOptions`. Options for scraping each search result (same fields as scrape). + +- `options.enterprise` — array of `"default" | "anon" | "zdr"`. Enterprise options: `"zdr"` for zero data retention, `"anon"` for anonymized search. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +- `url` — string (required). The URL to scrape. + +- `options.formats` — array of format strings or format objects. Output formats to include. + - String formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"` + - Object formats (require at least `type`): + - `{ type: "json", prompt?: string, schema?: JSONSchema | ZodSchema }` — LLM-extracted JSON. Zod schemas are auto-converted. + - `{ type: "question", question: string }` — ask a question about the page. + - `{ type: "highlights", query: string }` — find relevant source text. + - `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` + - `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?, prompt?, tag? }` — `modes` is required. + - `{ type: "attributes", selectors: Array<{ selector, attribute }> }` + +- `options.headers` — `Record`. Custom HTTP headers. + +- `options.includeTags` — string array. Only include content from these HTML tags. + +- `options.excludeTags` — string array. Exclude content from these HTML tags. + +- `options.onlyMainContent` — boolean. Strip nav, footer, and other boilerplate. + +- `options.timeout` — number. Timeout in milliseconds. + +- `options.waitFor` — number. Wait for page to render (milliseconds). + +- `options.mobile` — boolean. Emulate mobile viewport. + +- `options.parsers` — array of `"pdf"` or `{ type: "pdf", mode?: "fast" | "auto" | "ocr", maxPages?: number }`. + +- `options.actions` — array of action objects for pre-scrape browser actions. + - `{ type: "wait", milliseconds }` or `{ type: "wait", selector }` + - `{ type: "click", selector }` — `all` optional to click all matches + - `{ type: "write", text }` — type text into focused input + - `{ type: "press", key }` + - `{ type: "scroll", direction: "up" | "down" }` — `selector` optional + - `{ type: "scrape" }` — scrape current page content + - `{ type: "executeJavascript", script }` + - `{ type: "screenshot" }` — `fullPage`, `quality`, `viewport` optional + - `{ type: "pdf" }` — `format`, `landscape`, `scale` optional + +- `options.location` — `{ country?: string, languages?: string[] }`. Geo or language-aware scraping. + +- `options.skipTlsVerification` — boolean. Skip TLS verification. + +- `options.removeBase64Images` — boolean. Drop base64 images from markdown. + +- `options.fastMode` — boolean. Faster scrapes with reduced fidelity. + +- `options.blockAds` — boolean. Block ads and cookie popups. + +- `options.proxy` — `"basic" | "stealth" | "enhanced" | "auto"` or custom URL string. + +- `options.maxAge` — number. Use cached result if younger than this (milliseconds). + +- `options.minAge` — number. Cache-only mode; minimum cache age in milliseconds. + +- `options.storeInCache` — boolean. Store result in Firecrawl cache. + +- `options.lockdown` — boolean. Serve from cache only, never make outbound request. + +- `options.profile` — `{ name: string, saveChanges?: boolean }`. Persistent browser profile shared across scrapes and interactions. + +- `options.redactPII` — boolean or `{ mode?, entities?, replaceStyle? }`. Redact PII from returned content. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. Use `interact` for multi-step flows that go beyond quick pre-scrape actions. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId from scrape response"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +console.log(result.output); +``` + +Code execution example: + +```ts +const result = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); + +console.log(result.stdout); +``` + +Stop the session when done: + +```ts +await client.stopInteraction(jobId); +``` + +### Parameters + +- `jobId` — string (required). Scrape job ID from `document.metadata.scrapeId`. + +- `args.code` — string. Code to run in the browser session (e.g. Playwright `page` usage). Optional if `prompt` is provided. + +- `args.prompt` — string. Natural-language instruction for the browser agent. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty. + +- `args.language` — `"python" | "node" | "bash"`. Runtime for code execution. Default: `"node"`. + +- `args.timeout` — number. Execution timeout in seconds (1–300). + +### Return value + +`ScrapeExecuteResponse` includes: `success`, `output` (agent response for prompt), `stdout`, `result`, `stderr`, `exitCode`, `killed`, `liveViewUrl`, `interactiveLiveViewUrl`, `error`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` and `deleteScrapeBrowser` → `stopInteraction`; `scrapeUrl` → `scrape`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed in `formats` (for `json` or `changeTracking`) are auto-converted to JSON Schema. +- The SDK adds 5000ms to user-specified `timeout` for the HTTP request. For `interact`, the padding is `timeout * 1000 + 5000` (interact timeout is in seconds). + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..27238e1bf --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,250 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical Firecrawl Python quickstart for external agents. Generated from SDK source (`firecrawl-py`) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client in `firecrawl/v2/client.py`. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +The API key may be omitted for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. `FirecrawlApp` is a legacy alias for `Firecrawl`. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + limit=5, + scrape_options={"formats": ["markdown"], "only_main_content": True}, +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Important:** `search()` does not return `{ data: [...] }`. Access results via `results.web`, `results.news`, `results.images`, or `results.developer`. Accessing `.data` raises an `AttributeError`. + +### Parameters + +- `query` — str (required). The search query. Use `site:example.com` to limit results to a domain. + +- `sources` — list of `"web" | "news" | "images"` or `Source` objects. Controls which sources are searched. + +- `categories` — list of `"github" | "research" | "pdf" | "developer"` or `Category` objects. Filters results by category. + +- `include_domains` — list of str. Restrict results to these domains. Cannot be used with `exclude_domains`. + +- `exclude_domains` — list of str. Exclude results from these domains. Cannot be used with `include_domains`. + +- `limit` — int. Maximum results to return. Default: `5`. + +- `tbs` — str. Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). + +- `location` — str. Geographic location for localized results. Note: this is a plain string, not a `Location` object. + +- `ignore_invalid_urls` — bool. Drop URLs that cannot be scraped by other endpoints. + +- `highlights` — bool. Generate query-relevant highlights. Default: `True`. + +- `timeout` — int. Request timeout in milliseconds. Default: `300000`. + +- `scrape_options` — `ScrapeOptions` or dict. Options for scraping each search result (same fields as scrape). + +- `enterprise` — list of str. Enterprise options: `["zdr"]` for zero data retention, `["anon"]` for anonymized search. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +All parameters after `url` are keyword-only. + +- `url` — str (required). The URL to scrape. + +- `formats` — list of format strings or format dicts. Output formats. + - String formats: `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"attributes"`, `"branding"`, `"audio"`, `"video"` + - Dict formats (require at least `type`): + - `{"type": "json", "prompt": "...", "schema": {...}}` — LLM-extracted JSON. + - `{"type": "question", "question": "..."}` — ask a question about the page. + - `{"type": "highlights", "query": "..."}` — find relevant source text. + - `{"type": "screenshot", "full_page": True, "quality": 80, "viewport": {...}}` + - `{"type": "changeTracking", "modes": ["git-diff"], "tag": "..."}` + - `{"type": "attributes", "selectors": [{"selector": "a", "attribute": "href"}]}` + +- `headers` — dict. Custom HTTP headers. + +- `include_tags` — list of str. Only include content from these HTML tags. + +- `exclude_tags` — list of str. Exclude content from these HTML tags. + +- `only_main_content` — bool. Strip nav, footer, and other boilerplate. + +- `timeout` — int. Timeout in milliseconds. + +- `wait_for` — int. Wait for page to render (milliseconds). + +- `mobile` — bool. Emulate mobile viewport. + +- `parsers` — list of `"pdf"` or `PDFParser(mode="auto", max_pages=5)` or dicts. + +- `actions` — list of action dicts for pre-scrape browser actions. + - `{"type": "wait", "milliseconds": 1000}` or `{"type": "wait", "selector": "#el"}` + - `{"type": "click", "selector": "#btn"}` + - `{"type": "write", "text": "hello"}` + - `{"type": "press", "key": "Enter"}` + - `{"type": "scroll", "direction": "down"}` + - `{"type": "scrape"}` + - `{"type": "executeJavascript", "script": "..."}` + - `{"type": "screenshot"}` + - `{"type": "pdf"}` + +- `location` — dict with `country` and `languages`. Geo or language-aware scraping. + +- `skip_tls_verification` — bool. Skip TLS verification. + +- `remove_base64_images` — bool. Drop base64 images from markdown. + +- `fast_mode` — bool. Faster scrapes with reduced fidelity. + +- `block_ads` — bool. Block ads and cookie popups. + +- `proxy` — str. One of `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. + +- `max_age` — int. Use cached result if younger than this (milliseconds). + +- `store_in_cache` — bool. Store result in Firecrawl cache. + +- `lockdown` — bool. Serve from cache only, never make outbound request. + +- `profile` — dict with `name` and optional `save_changes`. Persistent browser profile. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id from scrape response") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +print(result.output) +``` + +Code execution example: + +```python +result = client.interact( + job_id, + code="print(await page.title())", + language="python", + timeout=60, +) +print(result.stdout) +``` + +Stop the session when done: + +```python +client.stop_interaction(job_id) +``` + +### Parameters + +- `job_id` — str (required). Scrape job ID from `document.metadata.scrape_id`. + +- `code` — str (positional-optional). Code to run in the browser session. Optional if `prompt` is provided. + +- `prompt` — str (keyword-only). Natural-language instruction for the browser agent. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty. + +- `language` — `"python" | "node" | "bash"` (keyword-only). Runtime for code execution. Default: `"node"`. + +- `timeout` — int (keyword-only). Execution timeout in seconds (1–300). + +### Return value + +`BrowserExecuteResponse` includes: `success`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `live_view_url`, `interactive_live_view_url`, `error`. API camelCase fields are normalized to snake_case. + +## Notes + +- All Python parameter names use snake_case. The SDK maps them to the API's camelCase automatically. +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`; `scrape_url` → `scrape`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- Search `location` is a plain `str`, not a `Location` object (unlike scrape's `location` which is a dict/object). +- `include_domains` and `exclude_domains` are mutually exclusive — providing both raises `ValueError`. +- An async client is available via `AsyncFirecrawl` (alias `AsyncFirecrawlApp`). + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..fe41e6624 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,284 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical Firecrawl Rust quickstart for external agents. Generated from SDK source (`firecrawl` crate v2.12.1) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +All public types are re-exported from the crate root (`use firecrawl::Client`, `use firecrawl::ScrapeOptions`, etc.). + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +The API key may be `None` or empty for keyless free-tier access (rate-limited per IP) on scrape, search, and interact. + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, ScrapeOptions, Format}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", SearchOptions { + limit: Some(5), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + println!("{:?}", item); + } +} +``` + +### Parameters + +- `query` — `impl AsRef` (required). The search query. + +- `options` — `impl Into>`. Pass `None` for defaults. + +- `options.sources` — `Vec`. Values: `Web`, `News`, `Images`. + +- `options.categories` — `Vec`. Values: `Github`, `Research`, `Pdf`. + +- `options.include_domains` — `Vec`. Restrict results to these domains. + +- `options.exclude_domains` — `Vec`. Exclude results from these domains. + +- `options.limit` — `u32`. Maximum results. Default: 5, max: 20. + +- `options.tbs` — `String`. Time-based filter (e.g. `"qdr:d"`). + +- `options.location` — `String`. Geographic location for localized results. + +- `options.ignore_invalid_urls` — `bool`. Drop URLs that cannot be scraped. + +- `options.highlights` — `bool`. Generate query-relevant highlights. Default: true. + +- `options.timeout` — `u32`. Request timeout in milliseconds. + +- `options.scrape_options` — `ScrapeOptions`. Options for scraping each result. + +A convenience helper `client.search_and_scrape(query, limit)` returns `Vec` directly. + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + ..Default::default() + }) + .await?; + +println!("{:?}", doc.markdown); +``` + +### Parameters + +- `url` — `impl AsRef` (required). The URL to scrape. + +- `options` — `impl Into>`. Pass `None` for defaults. + +- `options.formats` — `Vec`. Output formats. + - Enum variants: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video` + - Parameterized variants: `Question(String)`, `Highlights(String)`, `Query(String)` + +- `options.headers` — `HashMap`. Custom HTTP headers. + +- `options.include_tags` — `Vec`. Only include content from these HTML tags. + +- `options.exclude_tags` — `Vec`. Exclude content from these HTML tags. + +- `options.only_main_content` — `bool`. Strip nav, footer, and other boilerplate. + +- `options.timeout` — `u32`. Timeout in milliseconds. + +- `options.wait_for` — `u32`. Wait for page to render (milliseconds). + +- `options.mobile` — `bool`. Emulate mobile viewport. + +- `options.parsers` — `Vec`. Parser configurations. + - `ParserConfig::Simple("pdf".to_string())` + - `ParserConfig::Pdf { parser_type: "pdf", max_pages: Some(5) }` + +- `options.actions` — `Vec`. Pre-scrape browser actions. + - `Action::Wait { milliseconds, selector }` + - `Action::Click { selector }` + - `Action::Write { text }` + - `Action::Press { key }` + - `Action::Scroll { direction: ScrollDirection::Down, selector }` + - `Action::Scrape` + - `Action::ExecuteJavascript { script }` + - `Action::Screenshot { full_page, quality, viewport }` + - `Action::Pdf { format, landscape, scale }` + +- `options.location` — `LocationConfig` with `country` and `languages`. + +- `options.skip_tls_verification` — `bool`. Skip TLS verification. + +- `options.remove_base64_images` — `bool`. Drop base64 images from markdown. + +- `options.fast_mode` — `bool`. Faster scrapes with reduced fidelity. + +- `options.block_ads` — `bool`. Block ads and cookie popups. + +- `options.proxy` — `ProxyType`. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. + +- `options.max_age` — `u32`. Use cached result if younger than this (seconds). + +- `options.min_age` — `u32`. Cache-only mode; minimum cache age. + +- `options.store_in_cache` — `bool`. Store result in Firecrawl cache. + +- `options.lockdown` — `bool`. Serve from cache only. + +- `options.redact_pii` — `bool`. Redact PII from content. + +- `options.profile` — `ProfileConfig` with `name` and `save_changes`. + +- `options.json_options` — `JsonOptions` with `schema`, `system_prompt`, `prompt`. + +- `options.screenshot_options` — `ScreenshotOptions` with `full_page`, `quality`, `viewport`. + +- `options.change_tracking_options` — `ChangeTrackingOptions` with `modes` (`GitDiff` or `Json`), `schema`, `prompt`, `tag`. + +- `options.attribute_selectors` — `Vec` with `selector` and `attribute`. + +A convenience helper `client.scrape_with_schema(url, schema, prompt)` combines scrape + JSON extraction. + +## Interact + +### Why use it + +Control the browser session tied to a scrape job with code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +println!("{:?}", result.output); +``` + +Stop the session when done: + +```rust +client.stop_interaction(job_id).await?; +``` + +### Parameters + +- `job_id` — `impl AsRef` (required). Scrape job ID. + +- `options` — `ScrapeExecuteOptions` (required, not optional). + +- `options.code` — `Option`. Code to run in the browser session. Optional if `prompt` is provided. + +- `options.prompt` — `Option`. Natural-language instruction. Optional if `code` is provided. + +- At least one of `code` or `prompt` must be non-empty, or `FirecrawlError::Misuse` is returned. + +- `options.language` — `ScrapeExecuteLanguage`. Values: `Python`, `Node`, `Bash`. Default: `Node`. + +- `options.timeout` — `u32`. Execution timeout in seconds. + +### Return value + +`ScrapeExecuteResponse` includes: `success`, `output`, `stdout`, `result`, `stderr`, `exit_code`, `killed`, `live_view_url`, `interactive_live_view_url`, `error`. All fields except `success` are `Option`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` and `delete_scrape_browser` → `stop_interaction`. Compiler warnings are emitted for deprecated methods. +- All methods are async and require a `tokio` runtime. +- All request/response structs serialize to camelCase via `#[serde(rename_all = "camelCase")]`. Exception: `redact_pii` is renamed to `"redactPII"`. +- The SDK auto-sets `origin` to `"rust-sdk@{version}"` on requests if not provided. +- Optional parameters use `impl Into>` — pass `None`, a bare value, or `Some(...)`. Exception: `interact()` always requires `ScrapeExecuteOptions`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/lib.rs` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`