diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 00000000..c56c8fff --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,190 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Elixir quickstart for agents. Generated from SDK source (`firecrawl` **v1.9.1**, `firecrawl/apps/elixir-sdk`) and the v2 OpenAPI spec. + +## Install + +Add to `mix.exs`: + +```elixir +{:firecrawl, "~> 1.9"} +``` + +## Authenticate + +```elixir +# config/runtime.exs or config.exs +config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY") + +# or pass api_key per call +{:ok, res} = Firecrawl.search_and_scrape( + [query: "site:docs.firecrawl.dev webhook retries"], + api_key: "fc-your-api-key" +) +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.search_and_scrape( + query: "site:docs.firecrawl.dev webhook retries", + sources: [:web, :news], + limit: 10, + scrape_options: [ + formats: ["markdown"], + only_main_content: true + ] +) +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `query` | string (required) | You need a search query. Use `site:example.com` to scope. | +| `sources` | list of atoms or strings | You want to control sources. Values: `:web`, `:news`, `:images`. Default: `[:web]`. | +| `categories` | list of atoms, strings, or maps | You want to filter by category. Values: `:github`, `:research`, `:pdf`. | +| `include_domains` | list of strings | You want results only from specific domains. | +| `exclude_domains` | list of strings | You want to exclude specific domains. | +| `limit` | integer | You want to cap results. | +| `tbs` | string | You need a time filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | string | You want localized results. | +| `country` | string | You want ISO 3166-1 alpha-2 targeting (e.g. `"US"`). | +| `ignore_invalid_urls` | boolean | You want to drop URLs that cannot be scraped. | +| `timeout` | integer | You need a request timeout in milliseconds. | +| `highlights` | boolean | You want query-relevant highlights. Default: `true`. | +| `scrape_options` | keyword list | You want to scrape each search result inline. | +| `enterprise` | list of strings | You need enterprise controls. Values: `"zdr"`, `"anon"`. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com/pricing", + formats: [ + "markdown", + "links", + %{type: "json", prompt: "Extract plan names and prices."} + ], + only_main_content: true, + wait_for: 1000 +) +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `url` | string (required) | You want to scrape a specific page. | +| `formats` | list of strings or maps | You want multiple output formats. See format types below. | +| `headers` | map | You need custom request headers. | +| `include_tags` | list of strings | You want to include only specific HTML tags. | +| `exclude_tags` | list of strings | You want to exclude specific HTML tags. | +| `only_main_content` | boolean | You want to strip nav, footer, and boilerplate. | +| `timeout` | integer | You need a timeout in milliseconds. Default: `60000`. Min: `1000`. Max: `300000`. | +| `wait_for` | integer | You need to wait for page render (milliseconds). | +| `mobile` | boolean | You want a mobile viewport. | +| `parsers` | list of strings or maps | You need file parsing controls. | +| `actions` | list of maps | You need pre-scrape browser actions. | +| `location` | keyword list | You need geo or language-aware scraping. | +| `skip_tls_verification` | boolean | You need to skip TLS verification. | +| `remove_base64_images` | boolean | You want to drop base64 images from markdown. | +| `block_ads` | boolean | You want ad and cookie popup blocking. | +| `proxy` | atom | You need proxy control. Values: `:basic`, `:enhanced`, `:auto`. | +| `max_age` | integer | You want cached data up to a maximum age (ms). Default: 2 days. | +| `min_age` | integer | You want cached data only if at least this old (ms). | +| `store_in_cache` | boolean | You want Firecrawl to cache the result. | +| `lockdown` | boolean | You want only cached results, no outbound requests. | +| `redact_pii` | boolean | You want PII redacted from content. | +| `audit_metadata` | keyword list | You need user attribution for SIEM. Key: `username` (required). | +| `profile` | keyword list | You want a persistent browser profile. Keys: `name`, `save_changes`. | +| `zero_data_retention` | boolean | You want zero data retention for this scrape. | + +**Format strings:** `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"branding"`, `"audio"`, `"video"` + +**Format maps:** `%{type: "json", prompt: ...}`, `%{type: "screenshot", fullPage: true}`, `%{type: "changeTracking", modes: ["git-diff"]}` + +**Action types:** `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf` + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. The Elixir SDK exposes **code-based interactions only** (no `prompt` parameter). + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params \\ [], opts \\ [])` + +### Example + +```elixir +{:ok, scrape_res} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown"] +) + +job_id = get_in(scrape_res.body, ["data", "metadata", "scrapeId"]) + +{:ok, res} = Firecrawl.interact_with_scrape_browser_session( + job_id, + code: "console.log(await page.title());", + language: :node, + timeout: 60 +) +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `job_id` | string | You have a scrape job ID. | +| `code` | string (required) | You want to run code in the browser session. | +| `language` | atom or string | You need a specific runtime. Values: `:python`, `:node`, `:bash`. | +| `timeout` | integer | You need an execution timeout in seconds. | + +### Stop session + +`Firecrawl.stop_interactive_scrape_browser_session(job_id)` → ends the browser session. A bang variant `stop_interactive_scrape_browser_session!/2` is also available. + +## Notes + +- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec. +- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`. +- This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`). +- Pass `api_key:` in `opts` to override the configured key per call. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 00000000..8d4de5b7 --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,214 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Java quickstart for agents. Generated from SDK source (`firecrawl-java` **v1.12.1**, `firecrawl/apps/java-sdk`) and the v2 OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```gradle +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey(System.getenv("FIRECRAWL_API_KEY")) + .build(); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +- `client.search(query)` → `SearchData` +- `client.search(query, options)` → `SearchData` + +### Example + +```java +import com.firecrawl.models.SearchOptions; +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.SearchData; + +SearchOptions options = SearchOptions.builder() + .sources(List.of("web", "news")) + .limit(10) + .scrapeOptions( + ScrapeOptions.builder() + .formats(List.of("markdown")) + .onlyMainContent(true) + .build() + ) + .build(); + +SearchData results = client.search("site:docs.firecrawl.dev webhook retries", options); +List> web = results.getWeb(); +``` + +**Return value:** `SearchData` with `getWeb()`, `getNews()`, `getImages()` (each is `List>`, may be null). + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `query` | String | You need a search query. Use `site:example.com` to scope. | +| `options.sources` | `List` | You want to control sources. Values: `"web"`, `"news"`, `"images"`. | +| `options.categories` | `List` | You want to filter by category. Values: `"github"`, `"research"`, `"pdf"`. | +| `options.includeDomains` | `List` | You want results only from specific domains. | +| `options.excludeDomains` | `List` | You want to exclude specific domains. | +| `options.limit` | Integer | You want to cap results. | +| `options.tbs` | String | You need a time filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | String | You want localized results. | +| `options.ignoreInvalidURLs` | Boolean | You want to drop URLs that cannot be scraped. | +| `options.timeout` | Integer | You need a request timeout in milliseconds. | +| `options.highlights` | Boolean | You want query-relevant highlights. Default: `true`. | +| `options.scrapeOptions` | `ScrapeOptions` | You want to scrape each search result inline. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +- `client.scrape(url)` → `Document` +- `client.scrape(url, options)` → `Document` + +### Example + +```java +import com.firecrawl.models.ScrapeOptions; +import com.firecrawl.models.JsonFormat; +import com.firecrawl.models.Document; + +ScrapeOptions options = ScrapeOptions.builder() + .formats(List.of( + "markdown", + "links", + JsonFormat.builder().prompt("Extract plan names and prices.").build() + )) + .onlyMainContent(true) + .waitFor(1000) + .build(); + +Document doc = client.scrape("https://example.com/pricing", options); +System.out.println(doc.getMarkdown()); +System.out.println(doc.getJson()); +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `url` | String | You want to scrape a specific page. | +| `options.formats` | `List` | You want multiple output formats. Strings or format objects. | +| `options.headers` | `Map` | You need custom request headers. | +| `options.includeTags` | `List` | You want to include only specific HTML tags. | +| `options.excludeTags` | `List` | You want to exclude specific HTML tags. | +| `options.onlyMainContent` | Boolean | You want to strip nav, footer, and boilerplate. | +| `options.timeout` | Integer | You need a timeout in milliseconds. | +| `options.waitFor` | Integer | You need to wait for page render (milliseconds). | +| `options.mobile` | Boolean | You want a mobile viewport. | +| `options.parsers` | `List` | You need file parsing controls. | +| `options.actions` | `List>` | You need pre-scrape browser actions. | +| `options.location` | `LocationConfig` | You need geo or language-aware scraping. | +| `options.skipTlsVerification` | Boolean | You need to skip TLS verification. | +| `options.removeBase64Images` | Boolean | You want to drop base64 images from markdown. | +| `options.blockAds` | Boolean | You want ad and cookie popup blocking. | +| `options.proxy` | String | You need proxy control. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `options.maxAge` | Long | You want cached data up to a maximum age (ms). | +| `options.storeInCache` | Boolean | You want Firecrawl to cache the result. | +| `options.lockdown` | Boolean | You want only cached results, no outbound requests. | +| `options.redactPII` | Boolean | You want PII redacted from content. | +| `options.auditMetadata` | `AuditMetadata` | You need user attribution for SIEM. | + +**Format strings:** `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"changeTracking"`, `"json"`, `"attributes"`, `"branding"`, `"audio"`, `"video"` + +**Format objects:** `JsonFormat` (prompt, schema), `QuestionFormat` (question), `HighlightsFormat` (query) + +**Action types:** `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf` + +## Interact + +### Why use it + +Use interact when a page requires browser actions or code execution after a scrape starts. The Java SDK exposes **code-based interactions only** (no `prompt` parameter). + +### Preferred SDK method + +- `client.interact(jobId, code)` — default language `"node"` +- `client.interact(jobId, code, language, timeout)` — timeout in seconds (1–300) + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document doc = client.scrape("https://example.com", + ScrapeOptions.builder().formats(List.of("markdown")).build()); +String jobId = (String) doc.getMetadata().get("scrapeId"); + +BrowserExecuteResponse result = client.interact( + jobId, + "console.log(await page.title());", + "node", + 60 +); +System.out.println(result.getStdout()); +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `jobId` | String | You have a scrape job ID. | +| `code` | String | You want to run code in the browser session. | +| `language` | String | You need a specific runtime. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | Integer | You need an execution timeout in seconds (1–300). Null uses API default (30s). | + +### Stop session + +`client.stopInteractiveBrowser(jobId)` → ends the browser session. Returns `BrowserDeleteResponse` with `isSuccess()`, optional `getSessionDurationMs()`, `getCreditsBilled()`, `getError()`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `deleteScrapeBrowser` → `stopInteractiveBrowser`. +- The Java SDK exposes code-based interactions only — no `prompt` parameter on `interact` (unlike JS/Python/Rust). +- All methods have `Async` variants returning `CompletableFuture`. + +## Source Of Truth + +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchData.java` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 00000000..7a760439 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,229 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Node.js quickstart for agents. Generated from SDK source (`firecrawl` **v4.32.2**, `firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API. + +## Install + +```bash +npm install firecrawl +``` + +## Authenticate + +```ts +import { Firecrawl } from "firecrawl"; + +const client = new Firecrawl({ + apiKey: process.env.FIRECRAWL_API_KEY, + // apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default +}); +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or other browser actions after a scrape has created a session. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options?)` → `Promise` + +### Example + +```ts +const results = await client.search("site:docs.firecrawl.dev webhook retries", { + sources: ["web", "news"], + limit: 10, + scrapeOptions: { + formats: ["markdown"], + onlyMainContent: true, + }, +}); + +for (const item of results.web ?? []) { + console.log(item.url, item.title); +} +``` + +**Return value:** `SearchData` has optional arrays `web`, `news`, `images`, `developer`. Do **not** access `result.data` — results are grouped by source bucket. + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `query` | string | You need a search query. Use `site:example.com` to scope. | +| `options.sources` | `("web" \| "news" \| "images")[]` | You want to control which sources are searched. | +| `options.categories` | `("github" \| "research" \| "pdf" \| "developer")[]` | You want to filter by category. | +| `options.includeDomains` | `string[]` | You want results only from specific domains. | +| `options.excludeDomains` | `string[]` | You want to exclude specific domains. | +| `options.limit` | number | You want to cap results. | +| `options.tbs` | string | You need a time filter (e.g. `qdr:d`, `qdr:w`, `sbd:1,qdr:m`). | +| `options.location` | string | You want localized results. | +| `options.ignoreInvalidURLs` | boolean | You want to drop URLs that cannot be scraped. | +| `options.timeout` | number | You need a request timeout in milliseconds. | +| `options.highlights` | boolean | You want query-relevant highlights in results. | +| `options.scrapeOptions` | `ScrapeOptions` | You want to scrape each search result inline. | +| `options.enterprise` | `("default" \| "anon" \| "zdr")[]` | You need enterprise zero-data-retention controls. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options?)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com/pricing", { + formats: [ + "markdown", + "links", + { type: "json", prompt: "Extract plan names and prices." }, + ], + onlyMainContent: true, + waitFor: 1000, +}); + +console.log(doc.markdown); +console.log(doc.json); +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `url` | string | You want to scrape a specific page. | +| `options.formats` | `FormatOption[]` | You want multiple output formats. See format types below. | +| `options.headers` | `Record` | You need custom request headers. | +| `options.includeTags` | `string[]` | You want to include only specific HTML tags. | +| `options.excludeTags` | `string[]` | You want to exclude specific HTML tags. | +| `options.onlyMainContent` | boolean | You want to strip nav, footer, and boilerplate. | +| `options.timeout` | number | You need a timeout in milliseconds. | +| `options.waitFor` | number | You need to wait for page render (milliseconds). | +| `options.mobile` | boolean | You want a mobile viewport. | +| `options.parsers` | `(string \| PdfParser)[]` | You need file parsing controls (e.g. PDF). | +| `options.actions` | `ActionOption[]` | You need pre-scrape browser actions. | +| `options.location` | `{ country, languages }` | You need geo or language-aware scraping. | +| `options.skipTlsVerification` | boolean | You need to skip TLS verification. | +| `options.removeBase64Images` | boolean | You want to drop base64 images from markdown. | +| `options.fastMode` | boolean | You want faster scrapes with reduced fidelity. | +| `options.blockAds` | boolean | You want ad and cookie popup blocking. | +| `options.proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | You need proxy control. | +| `options.maxAge` | number | You want cached data up to a maximum age (ms). | +| `options.minAge` | number | You want cached data only if at least this old (ms). | +| `options.storeInCache` | boolean | You want Firecrawl to cache the result. | +| `options.lockdown` | boolean | You want only cached results, no outbound requests. | +| `options.redactPII` | `boolean \| RedactPIIOptions` | You want PII redacted from content. | +| `options.profile` | `{ name, saveChanges? }` | You want a persistent browser profile. | + +**Format types:** + +| String format | Description | +|---|---| +| `"markdown"` | Markdown content | +| `"html"` | Cleaned HTML | +| `"rawHtml"` | Raw HTML | +| `"links"` | Page links | +| `"images"` | Image URLs | +| `"screenshot"` | Screenshot output | +| `"summary"` | Summary output | +| `"changeTracking"` | Change tracking output | +| `"attributes"` | Attribute extraction | +| `"branding"` | Branding profile | +| `"product"` | Product data extraction | +| `"menu"` | Menu extraction | +| `"audio"` | Audio extraction | +| `"video"` | Video extraction | + +**Object-only formats** (require `{ type: ... }`): + +| Format object | Key fields | +|---|---| +| `{ type: "json", prompt?, schema? }` | At least one of `prompt` or `schema` required. SDK rejects plain `"json"` string. | +| `{ type: "question", question }` | Question-answer extraction. | +| `{ type: "highlights", query }` | Relevant source-text extraction. | +| `{ type: "screenshot", fullPage?, quality?, viewport? }` | Screenshot with options. | +| `{ type: "changeTracking", modes, schema?, prompt?, tag? }` | `modes` (`"git-diff"` / `"json"`) required. | +| `{ type: "attributes", selectors }` | `selectors` is `[{ selector, attribute }]`. | + +**Action types** (for `options.actions`): + +| Action | Required fields | +|---|---| +| `wait` | `milliseconds` or `selector` | +| `click` | `selector` | +| `write` | `text` (click to focus input first) | +| `press` | `key` | +| `scroll` | `direction` (`up` / `down`) | +| `screenshot` | none (all fields optional) | +| `scrape` | none | +| `executeJavascript` | `script` | +| `pdf` | none (all fields optional) | + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job. The SDK requires at least one of `code` or `prompt`. + +### Preferred SDK method + +`client.interact(jobId, args)` → `Promise` + +### Example + +```ts +const doc = await client.scrape("https://example.com", { formats: ["markdown"] }); +const jobId = doc.metadata?.scrapeId; +if (!jobId) throw new Error("Missing scrapeId"); + +const result = await client.interact(jobId, { + prompt: "Click the pricing tab and summarize the plans.", +}); +console.log(result.output); +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `jobId` | string | You have a scrape job ID from `document.metadata.scrapeId`. | +| `args.code` | string | You want to run code in the browser session. | +| `args.prompt` | string | You want the browser agent to follow a natural-language instruction. | +| `args.language` | `"python" \| "node" \| "bash"` | You need a specific runtime. Default: `"node"`. | +| `args.timeout` | number | You need an execution timeout in seconds. | + +At least one of `args.code` or `args.prompt` must be non-empty. + +### Stop session + +`client.stopInteraction(jobId)` → ends the browser session. Returns `{ success, sessionDurationMs?, creditsBilled?, error? }`. + +## Notes + +- Deprecated aliases: `scrapeExecute` → `interact`; `stopInteractiveBrowser` / `deleteScrapeBrowser` → `stopInteraction`. +- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`. +- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK. +- The package declares **Node.js >= 22** in `engines`. + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/package.json` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 00000000..8d37a0ba --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,231 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Python quickstart for agents. Generated from SDK source (`firecrawl-py` **v4.35.1**, `firecrawl/apps/python-sdk`) and the v2 OpenAPI spec. Method names, parameters, and return types match the v2 client. + +## Install + +```bash +pip install firecrawl-py +``` + +## Authenticate + +```py +import os +from firecrawl import Firecrawl + +client = Firecrawl(api_key=os.environ.get("FIRECRAWL_API_KEY")) +# client = Firecrawl(api_key="fc-...", api_url="https://api.firecrawl.dev") +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, **options)` → `SearchData` + +### Example + +```py +results = client.search( + "site:docs.firecrawl.dev webhook retries", + sources=["web", "news"], + limit=10, + scrape_options=ScrapeOptions( + formats=["markdown"], + only_main_content=True, + ), +) + +for item in results.web or []: + print(getattr(item, "url", None), getattr(item, "title", None)) +``` + +**Return value:** `SearchData` has optional lists `web`, `news`, `images`, `developer`. Do **not** access `result.data` — results are grouped by source bucket. + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `query` | str | You need a search query. Use `site:example.com` to scope. | +| `sources` | list of str or `Source` objects | You want to control which sources are searched. Values: `"web"`, `"news"`, `"images"`. | +| `categories` | list of str or `Category` objects | You want to filter by category. Values: `"github"`, `"research"`, `"pdf"`. | +| `include_domains` | list of str | You want results only from specific domains. | +| `exclude_domains` | list of str | You want to exclude specific domains. | +| `limit` | int | You want to cap results. Default: `5`. | +| `tbs` | str | You need a time filter (e.g. `qdr:d`, `qdr:w`). | +| `location` | str | You want localized results. | +| `ignore_invalid_urls` | bool | You want to drop URLs that cannot be scraped. | +| `timeout` | int | You need a request timeout in milliseconds. Default: `300000`. | +| `highlights` | bool | You want query-relevant highlights in results. | +| `scrape_options` | `ScrapeOptions` | You want to scrape each search result inline. | +| `enterprise` | list of str | You need enterprise zero-data-retention controls. Values: `"anon"`, `"zdr"`. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, **options)` → `Document` + +### Example + +```py +doc = client.scrape( + "https://example.com/pricing", + formats=[ + "markdown", + "links", + {"type": "json", "prompt": "Extract plan names and prices."}, + ], + only_main_content=True, + wait_for=1000, +) + +print(doc.markdown) +print(doc.json) +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `url` | str | You want to scrape a specific page. | +| `formats` | list of format strings or objects | You want multiple output formats. See format types below. | +| `headers` | dict of str to str | You need custom request headers. | +| `include_tags` | list of str | You want to include only specific HTML tags. | +| `exclude_tags` | list of str | You want to exclude specific HTML tags. | +| `only_main_content` | bool | You want to strip nav, footer, and boilerplate. | +| `timeout` | int | You need a timeout in milliseconds. | +| `wait_for` | int | You need to wait for page render (milliseconds). | +| `mobile` | bool | You want a mobile viewport. | +| `parsers` | list of str or `PDFParser` | You need file parsing controls (e.g. PDF). | +| `actions` | list of action objects | You need pre-scrape browser actions. | +| `location` | `Location` or dict | You need geo or language-aware scraping. | +| `skip_tls_verification` | bool | You need to skip TLS verification. | +| `remove_base64_images` | bool | You want to drop base64 images from markdown. | +| `fast_mode` | bool | You want faster scrapes with reduced fidelity. | +| `block_ads` | bool | You want ad and cookie popup blocking. | +| `proxy` | str | You need proxy control. Values: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. | +| `max_age` | int | You want cached data up to a maximum age (ms). | +| `store_in_cache` | bool | You want Firecrawl to cache the result. | +| `lockdown` | bool | You want only cached results, no outbound requests. | +| `redact_pii` | `bool \| RedactPIIOptions` | You want PII redacted from content. (Alias: `redactPII`) | +| `audit_metadata` | `AuditMetadata` | You need user attribution for SIEM. (Alias: `auditMetadata`) | +| `profile` | dict with `name` and optional `save_changes` | You want a persistent browser profile. | + +**Format types:** + +| String format | Description | +|---|---| +| `"markdown"` | Markdown content | +| `"html"` | Cleaned HTML | +| `"rawHtml"` or `"raw_html"` | Raw HTML | +| `"links"` | Page links | +| `"images"` | Image URLs | +| `"screenshot"` | Screenshot output | +| `"summary"` | Summary output | +| `"changeTracking"` or `"change_tracking"` | Change tracking output | +| `"attributes"` | Attribute extraction | +| `"branding"` | Branding profile | +| `"product"` | Product data extraction | +| `"menu"` | Menu extraction | +| `"audio"` | Audio extraction | +| `"video"` | Video extraction | + +**Object-only formats** (require `{"type": ...}`): + +| Format object | Key fields | +|---|---| +| `{"type": "json", "prompt": ..., "schema": ...}` | At least one of `prompt` or `schema`. Do not use plain `"json"` string. | +| `{"type": "question", "question": ...}` | Question-answer extraction. | +| `{"type": "highlights", "query": ...}` | Relevant source-text extraction. | +| `{"type": "screenshot", "full_page": ..., "quality": ..., "viewport": ...}` | Screenshot with options. | +| `{"type": "changeTracking", "modes": [...], "schema": ..., "tag": ...}` | `modes` (`"git-diff"` / `"json"`) required. | +| `{"type": "attributes", "selectors": [...]}` | `selectors` is `[{"selector": ..., "attribute": ...}]`. | + +**Action types** (for `actions`): + +| Action | Required fields | +|---|---| +| `wait` | `milliseconds` or `selector` | +| `click` | `selector` | +| `write` | `text` (click to focus input first) | +| `press` | `key` | +| `scroll` | `direction` (`up` / `down`) | +| `screenshot` | none (all fields optional) | +| `scrape` | none | +| `executeJavascript` | `script` | +| `pdf` | none (all fields optional) | + +## Interact + +### Why use it + +Use `interact` when a page requires browser actions or code execution after a scrape starts. Either `code` or `prompt` must be provided. + +### Preferred SDK method + +`client.interact(job_id, code=None, *, prompt=None, language="node", timeout=None)` + +`prompt` is keyword-only. + +### Example + +```py +doc = client.scrape("https://example.com", formats=["markdown"]) +job_id = doc.metadata.scrape_id if doc.metadata else None +if not job_id: + raise RuntimeError("Missing scrape_id") + +result = client.interact(job_id, prompt="Click the pricing tab and summarize the plans.") +print(result.output) +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `job_id` | str | You have a scrape job ID from `document.metadata.scrape_id`. | +| `code` | str | You want to run code in the browser session. | +| `prompt` | str (keyword-only) | You want the browser agent to follow a natural-language instruction. | +| `language` | str | You need a specific runtime. Values: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | int | You need an execution timeout in seconds. | + +At least one of `code` or `prompt` must be non-empty. + +### Stop session + +`client.stop_interaction(job_id)` → ends the browser session. Returns `BrowserDeleteResponse` with `success`, optional `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- The top-level `Firecrawl` client exposes v2 methods directly; v1 remains under `client.v1`. +- `AsyncFirecrawl` provides async versions of all methods. +- Python SDK uses `snake_case` parameter names; the SDK handles camelCase conversion for the API. + +## Source Of Truth + +- `firecrawl/apps/python-sdk/pyproject.toml` +- `firecrawl/apps/python-sdk/firecrawl/__init__.py` +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 00000000..27b58f15 --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,232 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +Canonical Firecrawl Rust quickstart for agents. Generated from SDK source (`firecrawl` **v2.12.1**, `firecrawl/apps/rust-sdk`) and the v2 OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-your-api-key")?; +// Self-hosted: +// let client = Client::new_selfhosted("http://localhost:3002", Some("fc-your-api-key"))?; +``` + +## When To Use What + +- `search`: use when you start with a query and need discovery. +- `scrape`: use when you already have a URL and want page content. +- `interact`: use when the page needs clicks, forms, or post-scrape browser actions. + +## Search + +### Why use it + +Use search to discover relevant pages from a query, then pick URLs to scrape or interact with. Constrain results to a site with `site:`, for example `site:docs.firecrawl.dev crawl webhooks`. + +### Preferred SDK method + +`client.search(query, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, SearchOptions, SearchSource, ScrapeOptions, Format}; + +let results = client + .search("site:docs.firecrawl.dev webhook retries", SearchOptions { + sources: Some(vec![SearchSource::Web, SearchSource::News]), + limit: Some(10), + scrape_options: Some(ScrapeOptions { + formats: Some(vec![Format::Markdown]), + only_main_content: Some(true), + ..Default::default() + }), + ..Default::default() + }) + .await?; + +if let Some(web) = &results.data.web { + for item in web { + // item is SearchResultOrDocument::WebResult or ::Document + } +} +``` + +**Return value:** `SearchResponse` contains `data: SearchData` with optional `web`, `news`, `images` fields. + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `query` | `impl AsRef` | You need a search query. Use `site:example.com` to scope. | +| `options.sources` | `Vec` | You want to control sources. Values: `Web`, `News`, `Images`. | +| `options.categories` | `Vec` | You want to filter by category. Values: `Github`, `Research`, `Pdf`. | +| `options.include_domains` | `Vec` | You want results only from specific domains. | +| `options.exclude_domains` | `Vec` | You want to exclude specific domains. | +| `options.limit` | `u32` | You want to cap results. | +| `options.tbs` | `String` | You need a time filter (e.g. `qdr:d`, `qdr:w`). | +| `options.location` | `String` | You want localized results. | +| `options.ignore_invalid_urls` | `bool` | You want to drop URLs that cannot be scraped. | +| `options.timeout` | `u32` | You need a request timeout in milliseconds. | +| `options.highlights` | `bool` | You want query-relevant highlights. | +| `options.scrape_options` | `ScrapeOptions` | You want to scrape each search result inline. | + +## Scrape + +### Why use it + +Use scrape when you already have a URL and want structured content in one or more formats. + +### Preferred SDK method + +`client.scrape(url, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, JsonOptions}; + +let doc = client + .scrape("https://example.com/pricing", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links, Format::Json]), + json_options: Some(JsonOptions { + prompt: Some("Extract plan names and prices.".to_string()), + ..Default::default() + }), + only_main_content: Some(true), + wait_for: Some(1000), + ..Default::default() + }) + .await?; + +println!("{:?}", doc.markdown); +println!("{:?}", doc.json); +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `url` | `impl AsRef` | You want to scrape a specific page. | +| `options.formats` | `Vec` | You want multiple output formats. See format values below. | +| `options.headers` | `HashMap` | You need custom request headers. | +| `options.include_tags` | `Vec` | You want to include only specific HTML tags. | +| `options.exclude_tags` | `Vec` | You want to exclude specific HTML tags. | +| `options.only_main_content` | `bool` | You want to strip nav, footer, and boilerplate. | +| `options.timeout` | `u32` | You need a timeout in milliseconds. | +| `options.wait_for` | `u32` | You need to wait for page render (milliseconds). | +| `options.mobile` | `bool` | You want a mobile viewport. | +| `options.parsers` | `Vec` | You need file parsing controls. | +| `options.actions` | `Vec` | You need pre-scrape browser actions. | +| `options.location` | `LocationConfig` | You need geo or language-aware scraping. | +| `options.skip_tls_verification` | `bool` | You need to skip TLS verification. | +| `options.remove_base64_images` | `bool` | You want to drop base64 images from markdown. | +| `options.fast_mode` | `bool` | You want faster scrapes with reduced fidelity. | +| `options.block_ads` | `bool` | You want ad and cookie popup blocking. | +| `options.proxy` | `ProxyType` | You need proxy control. Values: `Basic`, `Stealth`, `Enhanced`, `Auto`. | +| `options.max_age` | `u32` | You want cached data up to a maximum age (ms). | +| `options.min_age` | `u32` | You want cached data only if at least this old (ms). | +| `options.store_in_cache` | `bool` | You want Firecrawl to cache the result. | +| `options.lockdown` | `bool` | You want only cached results, no outbound requests. | +| `options.redact_pii` | `bool` | You want PII redacted from content. | +| `options.audit_metadata` | `AuditMetadata` | You need user attribution for SIEM. | +| `options.profile` | `ProfileConfig` | You want a persistent browser profile. Fields: `name`, `save_changes`. | +| `options.json_options` | `JsonOptions` | You want to configure JSON extraction. Fields: `schema`, `system_prompt`, `prompt`. | +| `options.screenshot_options` | `ScreenshotOptions` | You want to configure screenshot output. Fields: `full_page`, `quality`, `viewport`. | +| `options.change_tracking_options` | `ChangeTrackingOptions` | You want change tracking. Fields: `modes` (`GitDiff` / `Json`), `schema`, `prompt`, `tag`. | +| `options.attribute_selectors` | `Vec` | You want attribute extraction. Fields: `selector`, `attribute`. | + +**Format enum values:** + +`Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)` + +**Action enum variants:** + +| Variant | Required fields | +|---|---| +| `Wait` | `milliseconds` or `selector` | +| `Click` | `selector` | +| `Write` | `text` | +| `Press` | `key` | +| `Scroll` | `direction` (`Up` / `Down`) | +| `Screenshot` | none | +| `Scrape` | none | +| `ExecuteJavascript` | `script` | +| `Pdf` | none | + +## Interact + +### Why use it + +Use `interact` for code or natural-language control of the browser session tied to a scrape job. At least one of `code` or `prompt` must be non-empty. + +### Preferred SDK method + +`client.interact(job_id, options)` → `Result` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format, ScrapeExecuteOptions}; + +let doc = client + .scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() + }) + .await?; + +let job_id = doc.metadata + .as_ref() + .and_then(|m| m.scrape_id.as_ref()) + .expect("Missing scrapeId"); + +let result = client + .interact(job_id, ScrapeExecuteOptions { + prompt: Some("Click the pricing tab and summarize the plans.".to_string()), + ..Default::default() + }) + .await?; +``` + +### Parameters + +| Parameter | Type | Use when | +|---|---|---| +| `job_id` | `impl AsRef` | You have a scrape job ID. | +| `options.code` | `Option` | You want to run code in the browser session. | +| `options.prompt` | `Option` | You want the browser agent to follow a natural-language instruction. | +| `options.language` | `ScrapeExecuteLanguage` | You need a specific runtime. Values: `Python`, `Node`, `Bash`. Default: `Node`. | +| `options.timeout` | `u32` | You need an execution timeout in seconds. | + +At least one of `code` or `prompt` must be non-empty; otherwise returns `FirecrawlError::Misuse`. + +### Stop session + +`client.stop_interaction(job_id)` → ends the browser session. Returns `ScrapeBrowserDeleteResponse` with `success`, optional `session_duration_ms`, `credits_billed`, `error`. + +## Notes + +- Deprecated aliases: `scrape_execute` → `interact`; `stop_interactive_browser` / `delete_scrape_browser` → `stop_interaction`. +- `ScrapeOptions` uses dedicated `json_options`, `screenshot_options`, `change_tracking_options` structs (unlike the JS/Python SDKs which use inline format objects). +- `search_and_scrape(query, limit)` is a convenience helper that calls `search` with default `ScrapeOptions` and returns `Vec`. +- All types are exported at the crate root: `use firecrawl::Client`. + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/src/types.rs` +- `firecrawl-docs/api-reference/v2-openapi.json`