diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 000000000..68250d521 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,201 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`:firecrawl` hex package) and the v2 OpenAPI spec. Function names and parameter keys are auto-generated from the OpenAPI spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# Or pass api_key per call: +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + [url: "https://example.com", formats: ["markdown"]], + api_key: "fc-your-api-key" +) +``` + +All functions accept `:api_key` and `:base_url` (default `"https://api.firecrawl.dev/v2"`) in the trailing `opts` keyword list. + +## When To Use What + +- **`search`**: start with a query and need discovery. Returns ranked results, optionally with scraped content. +- **`scrape`**: already have a URL and want page content in one or more formats. +- **`interact`**: the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Use `site:example.com` in the query to limit results to a domain. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) + +web_results = res.body["data"]["web"] +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `:string` (required) | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `{:list, :any}` | Sources: `:web`, `:news`, `:images` (atoms or strings). | +| `categories` | `{:list, :any}` | Filters: `:github`, `:research`, `:pdf` (atoms or strings). | +| `include_domains` | `{:list, :string}` | Domains to include. | +| `exclude_domains` | `{:list, :string}` | Domains to exclude. | +| `limit` | `:integer` | Maximum results. | +| `tbs` | `:string` | Time-based filter (e.g. `"qdr:d"` for past day). | +| `location` | `:string` | Geographic location for localized results. | +| `country` | `:string` | ISO 3166-1 alpha-2 country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `:boolean` | Drop URLs that cannot be scraped. | +| `timeout` | `:integer` | Timeout in milliseconds. | +| `highlights` | `:boolean` | Query-relevant highlights. Default: `true`. | +| `enterprise` | `{:list, :string}` | Enterprise options: `"zdr"`, `"anon"`. | +| `scrape_options` | `:keyword_list` | Scrape each result (see Scrape parameters). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats — markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000 +) + +markdown = res.body["data"]["markdown"] +json_data = res.body["data"]["json"] +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `:string` (required) | Target URL to scrape. | +| `formats` | `{:list, :any}` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"`. Also format maps with `type` key. | +| `headers` | `:any` | Custom HTTP headers as a map. | +| `include_tags` | `{:list, :string}` | HTML tags to include. | +| `exclude_tags` | `{:list, :string}` | HTML tags to exclude. | +| `only_main_content` | `:boolean` | Strip nav, footer, boilerplate. | +| `timeout` | `:integer` | Timeout in milliseconds. Min: 1000, max: 300000. | +| `wait_for` | `:integer` | Wait time in ms before scraping. | +| `mobile` | `:boolean` | Emulate a mobile device. | +| `parsers` | `{:list, :any}` | Parser config: `"pdf"` or `%{type: "pdf", mode: "auto", maxPages: 5}`. | +| `actions` | `{:list, :any}` | Browser actions (see action types below). | +| `location` | `:keyword_list` | `[country: "US", languages: ["en-US"]]`. | +| `skip_tls_verification` | `:boolean` | Skip TLS verification. | +| `remove_base64_images` | `:boolean` | Drop base64 images from markdown. | +| `block_ads` | `:boolean` | Block ads and cookie popups. | +| `proxy` | `{:in, [:basic, :enhanced, :auto]}` | Proxy mode. | +| `max_age` | `:integer` | Max cache age in milliseconds. | +| `min_age` | `:integer` | Min cache age in milliseconds. | +| `store_in_cache` | `:boolean` | Store result in cache. | +| `lockdown` | `:boolean` | Only serve cached results. | +| `redact_pii` | `:boolean` | Redact personally identifiable information. | +| `profile` | `:keyword_list` | `[name: "my-session", save_changes: true]`. | +| `zero_data_retention` | `:boolean` | Enable zero data retention. | +| `audit_metadata` | `:keyword_list` | `[username: "agent-x"]`. | + +### Action types + +Pass as list of maps: `%{type: "wait", milliseconds: 750}`, `%{type: "click", selector: "#accept"}`, `%{type: "write", text: "hello"}`, `%{type: "press", key: "Enter"}`, `%{type: "scroll", direction: "down"}`, `%{type: "scrape"}`, `%{type: "executeJavascript", script: "..."}`, `%{type: "screenshot"}`, `%{type: "pdf"}`. + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Elixir SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = scrape_res.body["data"]["metadata"]["scrapeId"] + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `job_id` | `String.t()` (first arg) | Scrape job ID from response metadata. | +| `code` | `:string` (required) | Code to execute in the browser session. | +| `language` | `{:in, [:python, :node, :bash]}` | Runtime for code execution. | +| `timeout` | `:integer` | Execution timeout in seconds. | + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id)` → `DELETE /scrape/{jobId}/interact`. Ends the browser session. + +## Notes + +- The Elixir SDK is auto-generated from the OpenAPI spec. Function names are verbose (e.g. `scrape_and_extract_from_url`) and come directly from operation IDs. +- Parameters use snake_case atom keys — they are automatically converted to camelCase JSON. +- Nested keyword lists are recursively camelCased (e.g. `scrape_options: [only_main_content: true]` → `{"scrapeOptions": {"onlyMainContent": true}}`). +- Atom values (except `true`/`false`/`nil`) are converted to strings (e.g. `proxy: :basic` → `"proxy": "basic"`). +- Every function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- Responses are raw `Req.Response` structs — access data via `res.body["data"]`. +- Parameters are validated client-side with NimbleOptions before sending. +- The Elixir SDK does not support `prompt` on interact — only code-based interactions. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 000000000..1414b029d --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,241 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-java`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +Requires Java 11+. + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); + +// Or from environment (uses FIRECRAWL_API_KEY env var or firecrawl.apiKey system property): +// FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +Builder options: `apiKey` (String, falls back to `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property), `apiUrl` (String, default `"https://api.firecrawl.dev"`, falls back to `FIRECRAWL_API_URL`), `timeoutMs` (long, default `300000`), `maxRetries` (int, default `3`), `backoffFactor` (double, default `0.5`). + +## When To Use What + +- **`search`**: start with a query and need discovery. Returns ranked results, optionally with scraped content. +- **`scrape`**: already have a URL and want page content in one or more formats. +- **`interact`**: the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Use `site:example.com` in the query to limit results to a domain. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; +import java.util.List; +import java.util.Map; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +### Return value + +`SearchData` has `getWeb()`, `getNews()`, `getImages()` — each returns `List>` (may be null). Result maps contain keys like `url`, `title`, `description`, and full document fields when `scrapeOptions` is provided. + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `String` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `List` | Sources: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | Filters: `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | Domains to include. | +| `options.excludeDomains` | `List` | Domains to exclude. | +| `options.limit` | `Integer` | Maximum results. | +| `options.tbs` | `String` | Time-based filter (e.g. `"qdr:d"` for past day). | +| `options.location` | `String` | Geographic location string. | +| `options.ignoreInvalidURLs` | `Boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.highlights` | `Boolean` | Query-relevant highlights. Server default: `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each result with these options (see Scrape parameters). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats — markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `String` | Target URL to scrape. | +| `options.formats` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`, etc. Also `JsonFormat`, `QuestionFormat`, `HighlightsFormat` objects. | +| `options.headers` | `Map` | Custom HTTP headers. | +| `options.includeTags` | `List` | HTML tags to include. | +| `options.excludeTags` | `List` | HTML tags to exclude. | +| `options.onlyMainContent` | `Boolean` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Integer` | Timeout in milliseconds. | +| `options.waitFor` | `Integer` | Wait time in ms before scraping. | +| `options.mobile` | `Boolean` | Emulate a mobile device. | +| `options.parsers` | `List` | Parser config: `"pdf"` or `Map.of("type", "pdf", "maxPages", 10)`. | +| `options.actions` | `List>` | Browser actions (see action types below). | +| `options.location` | `LocationConfig` | `LocationConfig.builder().country("US").languages(List.of("en-US")).build()`. | +| `options.skipTlsVerification` | `Boolean` | Skip TLS verification. | +| `options.removeBase64Images` | `Boolean` | Drop base64 images from markdown. | +| `options.blockAds` | `Boolean` | Block ads and cookie popups. | +| `options.proxy` | `String` | Proxy: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`, or custom URL. | +| `options.maxAge` | `Long` | Max cache age in milliseconds. | +| `options.storeInCache` | `Boolean` | Store result in cache. | +| `options.lockdown` | `Boolean` | Only serve cached results. | +| `options.redactPII` | `Boolean` | Redact personally identifiable information. | +| `options.auditMetadata` | `AuditMetadata` | SIEM logging: `AuditMetadata.builder().username("agent-x").build()`. | + +### Format objects + +- `JsonFormat.builder().prompt("...").schema(Map.of(...)).build()` — JSON extraction. +- `QuestionFormat.builder().question("...").build()` — question-answer extraction. +- `HighlightsFormat.builder().query("...").build()` — relevant source-text extraction. + +### Action types + +Pass as `List>`: `wait` (`milliseconds` or `selector`), `screenshot`, `click` (`selector`), `write` (`text`), `press` (`key`), `scroll` (`direction`: `"up"` | `"down"`), `scrape`, `executeJavascript` (`script`), `pdf` (`format`, `landscape`, `scale`). + +## Interact + +### Why use it + +Execute code in the browser session tied to a scrape job. The Java SDK exposes code-based interactions only (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — default language `"node"` +- `client.interact(jobId, code, language, timeout)` — explicit language and timeout + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); + +System.out.println(result.getStdout()); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `jobId` | `String` | Scrape job ID from `document.getMetadata().get("scrapeId")`. | +| `code` | `String` | Code to run in the browser session (Playwright `page` available in `node`). | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Null uses API default (30s). | + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → ends the browser session. Returns `BrowserDeleteResponse` with `isSuccess()`, `getSessionDurationMs()`, `getCreditsBilled()`. + +### Response + +`BrowserExecuteResponse` provides: `isSuccess()`, `getStdout()`, `getStderr()`, `getResult()`, `getExitCode()`, `getKilled()`, `getError()`. + +## Notes + +- All options use the Builder pattern: `ScrapeOptions.builder().formats(...).build()`. +- `formats` accepts mixed types in `List` — plain strings and typed format objects. +- Null option fields are omitted from the request, letting the API apply server-side defaults. +- `SearchOptions.location` is a plain `String`, not a `LocationConfig` (unlike `ScrapeOptions.location`). +- `SearchData` result lists are `List>` (untyped maps). +- The Java SDK does not support `prompt` on `interact` — only `code`-based interactions. +- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- Async variants available for all methods: `scrapeAsync`, `searchAsync`, `interactAsync`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 000000000..1b266a649 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,212 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`@mendable/firecrawl-js`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install @mendable/firecrawl-js +``` + +Requires Node.js >= 22. + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, +}); +``` + +Constructor options: `apiKey` (string, defaults to `FIRECRAWL_API_KEY` env var), `apiUrl` (string, defaults to `FIRECRAWL_API_URL` env var or `https://api.firecrawl.dev`), `timeoutMs` (number, default `300000`), `maxRetries` (number, default `3`), `backoffFactor` (number, default `0.5`). + +## When To Use What + +- **`search`**: start with a query and need discovery. Returns ranked results, optionally with scraped content. +- **`scrape`**: already have a URL and want page content in one or more formats. +- **`interact`**: the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Use `site:example.com` in the query to limit results to a domain. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +### Return value + +`SearchData` has optional arrays grouped by source: `web`, `news`, `images`, `developer`. Each entry is either a lightweight result or a full `Document` when `scrapeOptions` is provided. Do **not** access `result.data` — it throws an error directing you to use `.web`, `.news`, `.images`, or `.developer`. + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `string` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `("web" \| "news" \| "images")[]` | Which sources to search. | +| `options.categories` | `("github" \| "research" \| "pdf" \| "developer")[]` | Category filters for results. | +| `options.includeDomains` | `string[]` | Domains to include. Cannot use with `excludeDomains`. | +| `options.excludeDomains` | `string[]` | Domains to exclude. Cannot use with `includeDomains`. | +| `options.limit` | `number` | Maximum number of results. Must be positive. | +| `options.tbs` | `string` | Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `options.location` | `string` | Geographic location string for localized results. | +| `options.ignoreInvalidURLs` | `boolean` | Drop URLs that cannot be scraped. | +| `options.timeout` | `number` | Timeout in milliseconds. Must be positive. | +| `options.highlights` | `boolean` | Generate query-relevant highlights. Server default: `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | Scrape each result with these options (see Scrape parameters). | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | Enterprise options. `"zdr"` = zero data retention. | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats — markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `string` | Target URL to scrape. | +| `options.formats` | `FormatOption[]` | Output formats (see format options below). | +| `options.headers` | `Record` | Custom HTTP headers for the target request. | +| `options.includeTags` | `string[]` | CSS selectors/HTML tags to include. | +| `options.excludeTags` | `string[]` | CSS selectors/HTML tags to exclude. | +| `options.onlyMainContent` | `boolean` | Strip nav, footer, and other boilerplate. | +| `options.timeout` | `number` | Timeout in milliseconds. | +| `options.waitFor` | `number` | Wait time in ms before scraping (for JS-rendered pages). | +| `options.mobile` | `boolean` | Emulate a mobile device. | +| `options.parsers` | `("pdf" \| { type: "pdf", mode?: "fast" \| "auto" \| "ocr", maxPages?: number })[]` | Parser configuration. | +| `options.actions` | `ActionOption[]` | Browser actions before scraping (see action types below). | +| `options.location` | `{ country?: string, languages?: string[] }` | Geographic targeting. | +| `options.skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `options.removeBase64Images` | `boolean` | Drop base64 images from markdown output. | +| `options.fastMode` | `boolean` | Faster scrapes with reduced fidelity. | +| `options.blockAds` | `boolean` | Block ads and cookie popups. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto" \| string` | Proxy mode or custom proxy URL. | +| `options.maxAge` | `number` | Max age in ms of cached content that may be reused. `0` bypasses cache. | +| `options.minAge` | `number` | Min age in ms of cached content. | +| `options.storeInCache` | `boolean` | Store the result in Firecrawl's cache. | +| `options.profile` | `{ name: string, saveChanges?: boolean }` | Persistent browser profile across scrapes. | +| `options.redactPII` | `boolean \| { mode?: "accurate" \| "aggressive" \| "fast", entities?: string[], replaceStyle?: "tag" \| "mask" \| "remove" }` | PII redaction. | + +### Format options + +**String formats:** `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"audio"`, `"video"` + +**Object formats:** +- `{ type: "json", prompt?: string, schema?: JsonSchema | ZodSchema }` — at least one of `prompt` or `schema` required. +- `{ type: "question", question: string }` — question-answer extraction. +- `{ type: "highlights", query: string }` — relevant source-text extraction. +- `{ type: "screenshot", fullPage?: boolean, quality?: number, viewport?: { width, height } }` +- `{ type: "changeTracking", modes: ("git-diff" | "json")[], schema?: object, prompt?: string, tag?: string }` — `modes` is required. +- `{ type: "attributes", selectors: { selector: string, attribute: string }[] }` + +### Action types + +`wait` (`milliseconds` or `selector`), `screenshot`, `click` (`selector`), `write` (`text`), `press` (`key`), `scroll` (`direction`: `"up"` | `"down"`, optional `selector`), `scrape`, `executeJavascript` (`script`), `pdf` (`format`, `landscape`, `scale`). + +## Interact + +### Why use it + +Control the browser session tied to a scrape job — run code or give natural-language instructions. Requires a `scrapeId` from a prior scrape. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); + +// Or use code: +const codeResult = await client.interact(jobId, { + code: "console.log(await page.title());", + language: "node", + timeout: 60, +}); +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `jobId` | `string` | Scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | `string` | Code to run in the browser session (Playwright `page` available). | +| `args.prompt` | `string` | Natural-language instruction for the browser agent. | +| `args.language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Default: `"node"`. | +| `args.timeout` | `number` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided. + +### Stop session + +`client.stopInteraction(jobId)` → ends the browser session and returns billing info. + +## Notes + +- Deprecated aliases: `scrapeUrl` → `scrape`, `scrapeExecute` → `interact`, `stopInteractiveBrowser`/`deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client. V1 is available via `client.v1`. +- Zod schemas in `formats` are auto-converted to JSON Schema by the SDK. +- The SDK auto-injects `origin: "js-sdk@"` on every request. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 000000000..8e193c479 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,216 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl-py`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python >= 3.8. + +## Authenticate + +```python +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +``` + +Constructor parameters: `api_key` (str, defaults to `FIRECRAWL_API_KEY` env var), `api_url` (str, default `"https://api.firecrawl.dev"`), `timeout` (float, per-request timeout in seconds), `max_retries` (int, default `3`), `backoff_factor` (float, default `0.5`). + +An async variant is available as `AsyncFirecrawl`. + +## When To Use What + +- **`search`**: start with a query and need discovery. Returns ranked results, optionally with scraped content. +- **`scrape`**: already have a URL and want page content in one or more formats. +- **`interact`**: the page needs clicks, forms, or post-scrape browser actions. Requires a `scrapeId` from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Use `site:example.com` in the query to limit results to a domain. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```python +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +### Return value + +`SearchData` has optional lists grouped by source: `.web`, `.news`, `.images`, `.developer`. Each entry is either a lightweight result or a full `Document` when `scrape_options` is provided. Do **not** access `.data` — it raises `AttributeError` directing you to use `.web`, `.news`, `.images`, or `.developer`. + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `str` | Search query. Use `site:example.com` to scope to a domain. | +| `sources` | `list[str]` | Which sources to search: `"web"`, `"news"`, `"images"`. | +| `categories` | `list[str]` | Category filters: `"github"`, `"research"`, `"pdf"`, `"developer"`. | +| `include_domains` | `list[str]` | Domains to include. Cannot use with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Domains to exclude. Cannot use with `include_domains`. | +| `limit` | `int` | Maximum number of results. Default: `5`. | +| `tbs` | `str` | Time-based filter (e.g. `"qdr:d"` for past day, `"qdr:w"` for past week). | +| `location` | `str` | Geographic location string for localized results. | +| `ignore_invalid_urls` | `bool` | Drop URLs that cannot be scraped. | +| `timeout` | `int` | Timeout in milliseconds. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. Default: `True`. | +| `scrape_options` | `ScrapeOptions` | Scrape each result with these options (see Scrape parameters). | +| `enterprise` | `list[str]` | Enterprise options: `"zdr"` (zero data retention), `"anon"` (anonymized). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats — markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```python +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `str` | Target URL to scrape. | +| `formats` | `list` | Output formats (see format options below). | +| `headers` | `dict[str, str]` | Custom HTTP headers for the target request. | +| `include_tags` | `list[str]` | CSS selectors/HTML tags to include. | +| `exclude_tags` | `list[str]` | CSS selectors/HTML tags to exclude. | +| `only_main_content` | `bool` | Strip nav, footer, and other boilerplate. | +| `timeout` | `int` | Timeout in milliseconds. | +| `wait_for` | `int` | Wait time in ms before scraping (for JS-rendered pages). | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list` | Parser config: `["pdf"]` or `[PDFParser(mode="ocr", max_pages=10)]`. | +| `actions` | `list` | Browser actions before scraping (see action types below). | +| `location` | `Location` | Geographic targeting: `Location(country="US", languages=["en-US"])`. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Drop base64 images from markdown output. | +| `fast_mode` | `bool` | Faster scrapes with reduced fidelity. | +| `block_ads` | `bool` | Block ads and cookie popups. | +| `proxy` | `str` | Proxy mode: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | `int` | Max age in ms of cached content that may be reused. `0` bypasses cache. | +| `store_in_cache` | `bool` | Store the result in Firecrawl's cache. | +| `lockdown` | `bool` | Only serve previously cached results. | +| `profile` | `dict` | Persistent browser profile: `{"name": "my-session", "saveChanges": True}`. | +| `audit_metadata` | `AuditMetadata` | SIEM logging: `AuditMetadata(username="agent-x")`. | + +### Format options + +**String formats:** `"markdown"`, `"html"`, `"rawHtml"` (or `"raw_html"`), `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"` (or `"change_tracking"`), `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"` + +**Object formats (pass as dicts or typed objects):** +- `JsonFormat(prompt="...", schema={...})` or `{"type": "json", "prompt": "..."}` — at least one of `prompt` or `schema` required. +- `QuestionFormat(question="...")` or `{"type": "question", "question": "..."}` +- `HighlightsFormat(query="...")` or `{"type": "highlights", "query": "..."}` +- `ScreenshotFormat(full_page=True, quality=80)` or `{"type": "screenshot", "fullPage": True}` +- `ChangeTrackingFormat(modes=["git-diff"])` or `{"type": "changeTracking", "modes": ["git-diff"]}` +- `AttributesFormat(selectors=[...])` or `{"type": "attributes", "selectors": [...]}` + +### Action types + +`wait` (`milliseconds` or `selector`), `screenshot`, `click` (`selector`), `write` (`text`), `press` (`key`), `scroll` (`direction`: `"up"` | `"down"`, optional `selector`), `scrape`, `executeJavascript` (`script`), `pdf` (`format`, `landscape`, `scale`). + +## Interact + +### Why use it + +Control the browser session tied to a scrape job — run code or give natural-language instructions. Requires a `scrapeId` from a prior scrape. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` → `BrowserExecuteResponse` + +### Example + +```python +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.get("scrapeId") if doc.metadata else None + +result = client.interact( + job_id, + prompt="Click the pricing tab and summarize the plans.", +) + +# Or use code: +code_result = client.interact( + job_id, + code="console.log(await page.title());", + language="node", + timeout=60, +) +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `job_id` | `str` | Scrape job ID from `document.metadata["scrapeId"]`. | +| `code` | `str` | Code to run in the browser session (Playwright `page` available). | +| `prompt` | `str` | Natural-language instruction for the browser agent. | +| `language` | `str` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds (1–300). | + +At least one of `code` or `prompt` must be provided. + +### Stop session + +`client.stop_interaction(job_id)` → ends the browser session and returns billing info. + +## Notes + +- The Python SDK uses `snake_case` for all parameter names (e.g. `only_main_content`, `wait_for`, `skip_tls_verification`). The SDK handles conversion to camelCase for the API. +- `location` is a `Location` object in `scrape` but a plain `str` in `search`. +- Deprecated aliases: `scrape_url` → `scrape`, `scrape_execute` → `interact`, `stop_interactive_browser`/`delete_scrape_browser` → `stop_interaction`. +- `FirecrawlApp` is an alias for `Firecrawl` (backward compatibility). + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 000000000..d05308494 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,236 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents. Generated from SDK source (`firecrawl` crate) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```toml +[dependencies] +firecrawl = "2" +tokio = { version = "1", features = ["full"] } +``` + +All methods are async and require a Tokio runtime. + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; + +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +`Client::new(api_key)` connects to the Firecrawl cloud. `Client::new_selfhosted(api_url, api_key)` connects to a self-hosted instance with an optional API key. + +## When To Use What + +- **`search`**: start with a query and need discovery. Returns ranked results, optionally with scraped content. +- **`scrape`**: already have a URL and want page content in one or more formats. +- **`interact`**: the page needs clicks, forms, or post-scrape browser actions. Requires a scrape job ID from a prior scrape. + +## Search + +### Why use it + +Discover relevant pages from a query, then pick URLs to scrape or interact with. Use `site:example.com` in the query to limit results to a domain. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let options = SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() +}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", options) + .await?; + +if let Some(web) = results.data.web { + for item in web { + // Each item is SearchResultOrDocument::WebResult or ::Document + } +} +``` + +### Return value + +`SearchResponse` contains `success: bool`, `data: SearchData`, `warning: Option`. `SearchData` has `web: Option>`, `news: Option>`, `images: Option>`. Web results are either `SearchResultOrDocument::WebResult` (lightweight) or `::Document` (full scraped content when `scrape_options` is provided). + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `query` | `impl AsRef` | Search query. Use `site:example.com` to scope to a domain. | +| `options.sources` | `Option>` | Sources: `Web`, `News`, `Images`. | +| `options.categories` | `Option>` | Filters: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Option>` | Domains to include. | +| `options.exclude_domains` | `Option>` | Domains to exclude. | +| `options.limit` | `Option` | Maximum results. Server default: 5, max: 20. | +| `options.tbs` | `Option` | Time-based filter (e.g. `"qdr:d"` for past day). | +| `options.location` | `Option` | Geographic location for localized results. | +| `options.ignore_invalid_urls` | `Option` | Drop URLs that cannot be scraped. | +| `options.timeout` | `Option` | Timeout in milliseconds. | +| `options.highlights` | `Option` | Query-relevant highlights. Server default: `true`. | +| `options.scrape_options` | `Option` | Scrape each result with these options (see Scrape parameters). | + +## Scrape + +### Why use it + +Get structured content from a URL in one or more formats — markdown, HTML, JSON extraction, screenshots, and more. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `url` | `impl AsRef` | Target URL to scrape. | +| `options.formats` | `Option>` | Formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`. Also `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. | +| `options.headers` | `Option>` | Custom HTTP headers. | +| `options.include_tags` | `Option>` | HTML tags to include. | +| `options.exclude_tags` | `Option>` | HTML tags to exclude. | +| `options.only_main_content` | `Option` | Strip nav, footer, boilerplate. | +| `options.timeout` | `Option` | Timeout in milliseconds. | +| `options.wait_for` | `Option` | Wait time in ms before scraping. | +| `options.mobile` | `Option` | Emulate a mobile device. | +| `options.parsers` | `Option>` | Parser config: `Simple("pdf")` or `Pdf { parser_type, mode, max_pages }`. | +| `options.actions` | `Option>` | Browser actions: `Wait`, `Screenshot`, `Click`, `Write`, `Press`, `Scroll`, `Scrape`, `ExecuteJavascript`, `Pdf`. | +| `options.location` | `Option` | `LocationConfig { country, languages }`. | +| `options.skip_tls_verification` | `Option` | Skip TLS verification. | +| `options.remove_base64_images` | `Option` | Drop base64 images from markdown. | +| `options.fast_mode` | `Option` | Faster scrapes with reduced fidelity. | +| `options.block_ads` | `Option` | Block ads and cookie popups. | +| `options.proxy` | `Option` | Proxy: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `Option` | Max cache age in milliseconds. | +| `options.min_age` | `Option` | Min cache age in milliseconds. | +| `options.store_in_cache` | `Option` | Store result in cache. | +| `options.lockdown` | `Option` | Only serve cached results. | +| `options.profile` | `Option` | `ProfileConfig { name, save_changes }`. | +| `options.json_options` | `Option` | `JsonOptions { schema, system_prompt, prompt }`. | +| `options.screenshot_options` | `Option` | `ScreenshotOptions { full_page, quality, viewport }`. | +| `options.change_tracking_options` | `Option` | `ChangeTrackingOptions { modes, schema, prompt, tag }`. Modes: `GitDiff`, `Json`. | +| `options.attribute_selectors` | `Option>` | `AttributeSelector { selector, attribute }`. | + +## Interact + +### Why use it + +Control the browser session tied to a scrape job — run code or give natural-language instructions. For multi-step interactive flows, prefer `interact` over scrape-time `actions`. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, ScrapeExecuteLanguage, Format}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +// Get the scrapeId from metadata +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.get("scrapeId")) + .and_then(|v| v.as_str()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; + +// Or use code: +let code_result = client + .interact(job_id, ScrapeExecuteOptions { + code: Some("console.log(await page.title());".to_string()), + language: Some(ScrapeExecuteLanguage::Node), + timeout: Some(60), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `job_id` | `impl AsRef` | Scrape job ID from `document.metadata["scrapeId"]`. | +| `options.code` | `Option` | Code to run in the browser session. | +| `options.prompt` | `Option` | Natural-language instruction for the browser agent. | +| `options.language` | `Option` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `Option` | Execution timeout in seconds. | + +At least one of `code` or `prompt` must be provided, otherwise `FirecrawlError::Misuse` is returned. + +### Stop session + +`client.stop_interaction(job_id)` → ends the browser session. Returns `ScrapeBrowserDeleteResponse` with `success`, `session_duration_ms`, `credits_billed`. + +## Notes + +- All options use `#[derive(Default)]` — use `..Default::default()` for struct-update syntax, not a builder pattern. +- `scrape` and `search` accept `impl Into>` for options, so you can pass `None` directly instead of wrapping in `Some()`. +- Deprecated aliases: `scrape_execute` → `interact`, `stop_interactive_browser`/`delete_scrape_browser` → `stop_interaction`. +- The SDK auto-injects `origin: "rust-sdk@"` on every request. +- All types are exported at the crate root: `use firecrawl::Client` (not `use firecrawl::v2::Client`). +- `Format` enum has custom serialization for complex variants like `Question(QuestionFormat)` and `Highlights(HighlightsFormat)`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`