diff --git a/agent-quickstart/elixir.mdx b/agent-quickstart/elixir.mdx new file mode 100644 index 00000000..87999044 --- /dev/null +++ b/agent-quickstart/elixir.mdx @@ -0,0 +1,209 @@ +--- +title: "Elixir Agent Quickstart" +description: "Canonical Firecrawl Elixir quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Elixir Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Elixir. Generated from SDK source and OpenAPI spec. The Elixir SDK is auto-generated from the OpenAPI spec, so function names mirror the API operation names. + +## Install + +Add to `mix.exs`: + +```elixir +defp deps do + [ + {:firecrawl, "~> 1.9"} + ] +end +``` + +Then run: + +```bash +mix deps.get +``` + +## Authenticate + +Add to your config: + +```elixir +config :firecrawl, api_key: "fc-YOUR-API-KEY" +``` + +Or pass `api_key` per-request: + +```elixir +Firecrawl.search_and_scrape([query: "firecrawl"], api_key: "fc-YOUR-API-KEY") +``` + +All functions accept a trailing `opts` keyword list supporting: + +| Option | Type | Description | +|---|---|---| +| `api_key` | `string` | Override the configured API key. | +| `base_url` | `string` | Override the base URL. Default: `"https://api.firecrawl.dev/v2"`. | + +Additional keys in `opts` are passed through to `Req`. + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`Firecrawl.search_and_scrape(params, opts \\ [])` + +### Example + +```elixir +{:ok, results} = Firecrawl.search_and_scrape(query: "firecrawl web scraping", limit: 5) + +for result <- results["data"]["web"] do + IO.puts("#{result["title"]} #{result["url"]}") +end +``` + +Every function also has a bang variant (`search_and_scrape!`) that raises on error instead of returning `{:error, ...}`. + +### Parameters + +All parameters are passed as a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** Search query. | +| `limit` | `integer` | Max results per source type. | +| `sources` | `list` | Sources to search. Default: `["web"]`. | +| `categories` | `list` | Category filters. | +| `include_domains` | `list(string)` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list(string)` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"` past day). | +| `location` | `string` | Geo-targeting location string. | +| `country` | `string` | ISO country code (e.g. `"US"`). | +| `ignore_invalid_urls` | `boolean` | Exclude invalid URLs. | +| `timeout` | `integer` | Timeout in ms. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrape_options` | `keyword_list` | Scrape options applied to each result page. | +| `enterprise` | `list(string)` | ZDR options: `["zdr"]` or `["anon"]`. | + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`Firecrawl.scrape_and_extract_from_url(params, opts \\ [])` + +### Example + +```elixir +{:ok, result} = Firecrawl.scrape_and_extract_from_url( + url: "https://example.com", + formats: ["markdown", "links"], + only_main_content: true +) + +IO.puts(result["data"]["markdown"]) +``` + +### Parameters + +All parameters are passed as a keyword list. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Default: `["markdown"]`. | +| `headers` | `any` | Custom HTTP headers. | +| `include_tags` | `list(string)` | HTML tags to include. | +| `exclude_tags` | `list(string)` | HTML tags to exclude. | +| `only_main_content` | `boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `integer` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `wait_for` | `integer` | Extra delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `list` | File processing controls (e.g. PDF). | +| `actions` | `list` | Browser actions before content capture. | +| `location` | `keyword_list` | Geo settings. Country defaults to `"US"`. | +| `skip_tls_verification` | `boolean` | Skip TLS certificate verification. | +| `remove_base64_images` | `boolean` | Remove base64 images from markdown. | +| `block_ads` | `boolean` | Block ads and cookie popups. | +| `proxy` | `:basic \| :enhanced \| :auto` | Proxy type. Default: `"auto"`. | +| `max_age` | `integer` | Cache threshold in ms. Default: 2 days. | +| `min_age` | `integer` | Cache-only mode minimum age in ms. | +| `store_in_cache` | `boolean` | Store result in cache. | +| `lockdown` | `boolean` | Cache-only, no outbound requests. | +| `redact_pii` | `boolean` | Redact PII. | +| `audit_metadata` | `keyword_list` | SIEM logging. Required key: `username` (string). | +| `profile` | `keyword_list` | Persistent browser profile. | +| `zero_data_retention` | `boolean` | Enable zero data retention. | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Execute code in the browser sandbox to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +`Firecrawl.interact_with_scrape_browser_session(job_id, params, opts \\ [])` + +### Example + +```elixir +{:ok, result} = Firecrawl.scrape_and_extract_from_url( + url: "https://www.amazon.com", + formats: ["markdown"] +) + +scrape_id = result["data"]["metadata"]["scrapeId"] + +{:ok, response} = Firecrawl.interact_with_scrape_browser_session(scrape_id, + code: "document.querySelector('input[name=field-keywords]').value = 'iPhone 16 Pro Max'" +) + +Firecrawl.stop_interactive_scrape_browser_session(scrape_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `String.t` | **Required.** First positional argument. Scrape job ID from response metadata. | +| `code` | `string` | **Required.** Code to execute in the browser sandbox. | +| `language` | `:python \| :node \| :bash` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `integer` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `string` | Origin label for telemetry. | + +Stop the session when done: + +```elixir +Firecrawl.stop_interactive_scrape_browser_session(scrape_id) +``` + +## Notes + +- Parameter names use **snake_case** and are passed as keyword lists. +- The Elixir SDK is **auto-generated from the OpenAPI spec**, so function names are verbose and mirror API operation names directly. +- The `interact` function requires `code` — it does not support a `prompt` parameter. Use the `code` parameter with JavaScript to control the browser. +- Every function has a **bang variant** (`!` suffix) that raises `Firecrawl.Error` instead of returning `{:error, ...}`. +- An `origin` field (`"elixir-sdk@{version}"`) is automatically injected into every request body. +- There are no deprecated aliases in the Elixir SDK. + +## Source Of Truth + +- `firecrawl/apps/elixir-sdk/lib/firecrawl.ex` +- `firecrawl/apps/elixir-sdk/mix.exs` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/java.mdx b/agent-quickstart/java.mdx new file mode 100644 index 00000000..190ce16b --- /dev/null +++ b/agent-quickstart/java.mdx @@ -0,0 +1,225 @@ +--- +title: "Java Agent Quickstart" +description: "Canonical Firecrawl Java quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Java Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Java. Generated from SDK source and OpenAPI spec. + +## Install + +Maven: + +```xml + + com.firecrawl + firecrawl-java + 1.12.1 + +``` + +Gradle: + +```groovy +implementation("com.firecrawl:firecrawl-java:1.12.1") +``` + +## Authenticate + +```java +import com.firecrawl.client.FirecrawlClient; + +FirecrawlClient client = FirecrawlClient.builder() + .apiKey("fc-YOUR-API-KEY") + .build(); +``` + +Or read from the `FIRECRAWL_API_KEY` environment variable: + +```java +FirecrawlClient client = FirecrawlClient.fromEnv(); +``` + +Builder options: + +| Option | Type | Default | +|---|---|---| +| `apiKey` | `String` | `FIRECRAWL_API_KEY` env var or `firecrawl.apiKey` system property | +| `apiUrl` | `String` | `"https://api.firecrawl.dev"` | +| `timeoutMs` | `long` | `300000` (5 min) | +| `maxRetries` | `int` | `3` | +| `backoffFactor` | `double` | `0.5` | +| `asyncExecutor` | `Executor` | `ForkJoinPool.commonPool()` | +| `httpClient` | `OkHttpClient` | — (overrides `timeoutMs`) | + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`client.search(query)` or `client.search(query, options)` + +### Example + +```java +import com.firecrawl.models.SearchData; +import com.firecrawl.models.SearchOptions; + +SearchData results = client.search("firecrawl web scraping", SearchOptions.builder() + .limit(5) + .build()); + +for (var result : results.getWeb()) { + System.out.println(result.get("title") + " " + result.get("url")); +} +``` + +### Parameters + +All fields on `SearchOptions` are nullable and default to null (API defaults apply). + +| Parameter | Type | Description | +|---|---|---| +| `query` | `String` | **Required.** First positional argument. Search query. | +| `limit` | `Integer` | Max results per source type. | +| `sources` | `List` | Sources: `"web"`, `"news"`, `"images"` as strings or `{type: "web"}` maps. | +| `categories` | `List` | Categories: `"github"`, `"research"`, `"pdf"`. | +| `includeDomains` | `List` | Restrict to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains` | `List` | Exclude these domains. Cannot combine with `includeDomains`. | +| `tbs` | `String` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week). | +| `location` | `String` | Geo-targeting location string. | +| `ignoreInvalidURLs` | `Boolean` | Exclude invalid URLs. | +| `timeout` | `Integer` | Timeout in ms. | +| `highlights` | `Boolean` | Generate query-relevant highlights. | +| `scrapeOptions` | `ScrapeOptions` | Scrape options applied to each result page. | +| `integration` | `String` | Integration identifier. | + +Response fields: `results.getWeb()`, `results.getNews()`, `results.getImages()` each return `List>`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`client.scrape(url)` or `client.scrape(url, options)` + +### Example + +```java +import com.firecrawl.models.Document; +import com.firecrawl.models.ScrapeOptions; + +Document result = client.scrape("https://example.com", ScrapeOptions.builder() + .formats(List.of("markdown", "links")) + .onlyMainContent(true) + .build()); + +System.out.println(result.getMarkdown()); +``` + +### Parameters + +All fields on `ScrapeOptions` are nullable and default to null (API defaults apply). + +| Parameter | Type | Description | +|---|---|---| +| `url` | `String` | **Required.** First positional argument. URL to scrape. | +| `formats` | `List` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"screenshot"`, `"json"`, `"audio"`, `"video"`. Also accepts format config objects. Default: `["markdown"]`. | +| `headers` | `Map` | Custom HTTP headers. | +| `includeTags` | `List` | HTML tags to include. | +| `excludeTags` | `List` | HTML tags to exclude. | +| `onlyMainContent` | `Boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `Integer` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `waitFor` | `Integer` | Extra delay in ms before fetching content. | +| `mobile` | `Boolean` | Emulate a mobile device. | +| `parsers` | `List` | File processing controls (e.g. `"pdf"` or `{"type":"pdf","maxPages":10}`). | +| `actions` | `List>` | Browser actions before content capture. | +| `location` | `LocationConfig` | Geo settings with `country` and `languages`. | +| `skipTlsVerification` | `Boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `Boolean` | Remove base64 images from markdown. | +| `blockAds` | `Boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `String` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `maxAge` | `Long` | Cache threshold in ms. Default: 2 days. | +| `storeInCache` | `Boolean` | Store result in cache. | +| `lockdown` | `Boolean` | Cache-only, no outbound requests. | +| `redactPII` | `Boolean` | Redact PII. | +| `auditMetadata` | `AuditMetadata` | SIEM logging with `username` field. | +| `integration` | `String` | Integration identifier. | + +Async variant: `client.scrapeAsync(url, options)` returns `CompletableFuture`. + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Execute code in the browser sandbox to click buttons, fill forms, navigate, and extract dynamic content. + +### Preferred SDK method + +`client.interact(jobId, code)` or `client.interact(jobId, code, language, timeout)` + +### Example + +```java +import com.firecrawl.models.BrowserExecuteResponse; + +Document result = client.scrape("https://www.amazon.com", ScrapeOptions.builder() + .formats(List.of("markdown")) + .build()); + +String scrapeId = (String) result.getMetadata().get("scrapeId"); + +BrowserExecuteResponse response = client.interact(scrapeId, + "document.querySelector('input[name=field-keywords]').value = 'iPhone 16 Pro Max'"); + +client.stopInteractiveBrowser(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `String` | **Required.** Scrape job ID from `result.getMetadata().get("scrapeId")`. | +| `code` | `String` | **Required.** Code to execute in the browser sandbox. | +| `language` | `String` | Runtime: `"python"`, `"node"`, `"bash"`. Default: `"node"`. | +| `timeout` | `Integer` | Execution timeout in seconds (1–300). Default: `30`. | +| `origin` | `String` | Origin label. Default: `"java-sdk@{version}"`. | + +Stop the session when done: + +```java +client.stopInteractiveBrowser(scrapeId); +``` + +Async variants: `client.interactAsync(...)` and `client.stopInteractiveBrowserAsync(...)` return `CompletableFuture`. + +## Notes + +- Parameter names use **camelCase** (e.g. `onlyMainContent`, `includeTags`, `scrapeOptions`). +- All option classes use the **builder pattern**: `ScrapeOptions.builder().field(value).build()`. +- The Java `interact` method requires `code` — it does not support a `prompt` parameter. Use the `code` parameter with JavaScript to control the browser. +- `includeDomains` and `excludeDomains` on search are mutually exclusive. +- Deprecated aliases (do not use in new code): + - `scrapeExecute(...)` → use `interact(...)` + - `deleteScrapeBrowser(...)` → use `stopInteractiveBrowser(...)` + +## Source Of Truth + +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/client/FirecrawlClient.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/ScrapeOptions.java` +- `firecrawl/apps/java-sdk/src/main/java/com/firecrawl/models/SearchOptions.java` +- `firecrawl/apps/java-sdk/build.gradle.kts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/node.mdx b/agent-quickstart/node.mdx new file mode 100644 index 00000000..0d67e420 --- /dev/null +++ b/agent-quickstart/node.mdx @@ -0,0 +1,215 @@ +--- +title: "Node.js Agent Quickstart" +description: "Canonical Firecrawl Node.js quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Node.js Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Node.js. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +npm install firecrawl +``` + +Requires Node.js 22+. + +## Authenticate + +```javascript +import Firecrawl from 'firecrawl'; + +const app = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" }); +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `apiKey`: + +```javascript +const app = new Firecrawl(); +``` + +Constructor options: + +| Option | Type | Default | +|---|---|---| +| `apiKey` | `string \| null` | `process.env.FIRECRAWL_API_KEY` | +| `apiUrl` | `string \| null` | `"https://api.firecrawl.dev"` | +| `timeoutMs` | `number` | — | +| `maxRetries` | `number` | — | +| `backoffFactor` | `number` | — | + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`search(query, options?)` + +### Example + +```javascript +const results = await app.search("firecrawl web scraping", { limit: 5 }); + +for (const result of results.web) { + console.log(result.title, result.url); + console.log(result.markdown); +} +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `string` | **Required.** Search query (max 500 chars). | +| `limit` | `number` | Max results per source type. Must be positive. | +| `sources` | `Array<"web" \| "news" \| "images" \| {type: string}>` | Sources to search. Default: `[{type: "web"}]`. | +| `categories` | `Array<"github" \| "research" \| "pdf" \| "developer">` | Category filters. | +| `includeDomains` | `string[]` | Restrict to these domains. Cannot combine with `excludeDomains`. | +| `excludeDomains` | `string[]` | Exclude these domains. Cannot combine with `includeDomains`. | +| `tbs` | `string` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week, `"qdr:m"` past month). | +| `location` | `string` | Geo-targeting location string. | +| `ignoreInvalidURLs` | `boolean` | Exclude invalid URLs from results. | +| `timeout` | `number` | Timeout in ms. Must be positive. | +| `highlights` | `boolean` | Generate query-relevant highlights. Default: `true`. | +| `scrapeOptions` | `ScrapeOptions` | Scrape options applied to each result page. | +| `enterprise` | `Array<"default" \| "anon" \| "zdr">` | Enterprise zero data retention options. | +| `threatProtection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label for telemetry. | + +Response groups results by source: `results.web`, `results.news`, `results.images`, `results.developer`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`scrape(url, options?)` + +### Example + +```javascript +const result = await app.scrape("https://example.com", { + formats: ["markdown", "links"], + onlyMainContent: true +}); + +console.log(result.markdown); +console.log(result.links); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `string` | **Required.** URL to scrape. | +| `formats` | `FormatOption[]` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts config objects (see below). Default: `["markdown"]`. | +| `headers` | `Record` | Custom HTTP headers. | +| `includeTags` | `string[]` | HTML tags to include. | +| `excludeTags` | `string[]` | HTML tags to exclude. | +| `onlyMainContent` | `boolean` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `number` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `waitFor` | `number` | Extra delay in ms before fetching content. | +| `mobile` | `boolean` | Emulate a mobile device. | +| `parsers` | `Array` | File processing controls. Default: `["pdf"]`. | +| `actions` | `ActionOption[]` | Browser actions before content capture: `wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`, `pdf`. | +| `location` | `{country?: string, languages?: string[]}` | Geo settings. Country defaults to `"US"`. | +| `skipTlsVerification` | `boolean` | Skip TLS certificate verification. | +| `removeBase64Images` | `boolean` | Remove base64 images from markdown. Default: `true`. | +| `fastMode` | `boolean` | Enable fast mode. | +| `blockAds` | `boolean` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `"basic" \| "stealth" \| "enhanced" \| "auto"` | Proxy type. `"enhanced"` costs up to 5 credits. Default: `"auto"`. | +| `maxAge` | `number` | Cache threshold in ms. Returns cached version if younger. Default: 2 days. | +| `minAge` | `number` | Cache-only mode minimum age in ms. | +| `storeInCache` | `boolean` | Store result in Firecrawl cache. Default: `true`. | +| `lockdown` | `boolean` | Cache-only, no outbound requests. | +| `redactPII` | `boolean \| RedactPIIOptions` | Redact PII. Pass `true` for defaults or `{mode?: "accurate"\|"aggressive"\|"fast", entities?: string[], replaceStyle?: "tag"\|"mask"\|"remove"}`. | +| `threatProtection` | `ThreatProtectionOptions` | Threat protection override with `mode`, `riskScoreThreshold`, `blacklist`, `whitelist`, `blockedTlds`, `failurePolicy`. | +| `auditMetadata` | `{username: string}` | SIEM logging user attribution. | +| `profile` | `{name: string, saveChanges?: boolean}` | Persistent browser profile. | +| `integration` | `string` | Integration identifier. | +| `origin` | `string` | Origin label. | + +#### Format config objects + +| Format | Config fields | +|---|---| +| `json` | `{type: "json", schema?: object, prompt?: string}` — structured extraction with optional JSON Schema and prompt. | +| `screenshot` | `{type: "screenshot", fullPage?: boolean, quality?: number, viewport?: {width, height}}` | +| `changeTracking` | `{type: "changeTracking", modes: ("git-diff"\|"json")[], schema?: object, prompt?: string, tag?: string}` | +| `question` | `{type: "question", question: string}` — ask a question about the page. | +| `highlights` | `{type: "highlights", query: string}` — extract text relevant to a query. | +| `attributes` | `{type: "attributes", selectors: [{selector: string, attribute: string}]}` | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`interact(jobId, args)` + +### Example + +```javascript +const result = await app.scrape("https://www.amazon.com", { formats: ["markdown"] }); +const scrapeId = result.metadata?.scrapeId; + +await app.interact(scrapeId, { prompt: "Search for iPhone 16 Pro Max" }); +const response = await app.interact(scrapeId, { + prompt: "Click on the first result and tell me the price" +}); +console.log(response.output); + +await app.stopInteraction(scrapeId); +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `jobId` | `string` | **Required.** Scrape job ID from `result.metadata.scrapeId`. | +| `code` | `string` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `string` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `"python" \| "node" \| "bash"` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `number` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `string` | Origin label for telemetry. | + +Stop the session when done: + +```javascript +await app.stopInteraction(scrapeId); +``` + +## Notes + +- Parameter names use **camelCase** (e.g. `onlyMainContent`, `includeTags`, `scrapeOptions`). +- `includeDomains` and `excludeDomains` on search are mutually exclusive. +- Deprecated aliases (do not use in new code): + - `scrapeExecute()` → use `interact()` + - `stopInteractiveBrowser()` / `deleteScrapeBrowser()` → use `stopInteraction()` + - `scrapeUrl()` → use `scrape()` + - `crawlUrl()` → use `crawl()` + - `mapUrl()` → use `map()` + +## Source Of Truth + +- `firecrawl/apps/js-sdk/firecrawl/src/v2/client.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts` +- `firecrawl/apps/js-sdk/firecrawl/src/index.ts` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/python.mdx b/agent-quickstart/python.mdx new file mode 100644 index 00000000..d1a96a88 --- /dev/null +++ b/agent-quickstart/python.mdx @@ -0,0 +1,200 @@ +--- +title: "Python Agent Quickstart" +description: "Canonical Firecrawl Python quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Python Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Python. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +pip install firecrawl-py +``` + +Requires Python 3.8+. + +## Authenticate + +```python +from firecrawl import Firecrawl + +app = Firecrawl(api_key="fc-YOUR-API-KEY") +``` + +Or set the `FIRECRAWL_API_KEY` environment variable and omit `api_key`: + +```python +app = Firecrawl() +``` + +Constructor parameters: + +| Parameter | Type | Default | +|---|---|---| +| `api_key` | `str` | `None` (falls back to `FIRECRAWL_API_KEY` env var) | +| `api_url` | `str` | `"https://api.firecrawl.dev"` | +| `timeout` | `float` | `None` | +| `max_retries` | `int` | `3` | +| `backoff_factor` | `float` | `0.5` | + +An async client is also available: `from firecrawl import AsyncFirecrawl`. + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`search(query, **kwargs)` + +### Example + +```python +results = app.search("firecrawl web scraping", limit=5) + +for result in results.web: + print(result.title, result.url) + print(result.markdown) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `query` | `str` | **Required.** Search query (max 500 chars). | +| `limit` | `int` | Max results per source type. Default: `5` (SDK default). | +| `sources` | `list` | Sources to search: `"web"`, `"news"`, `"images"`, or `Source` objects. | +| `categories` | `list` | Category filters: `"github"`, `"research"`, `"pdf"`, `"developer"`. | +| `include_domains` | `list[str]` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `list[str]` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `str` | Time-based filter (e.g. `"qdr:d"` past day, `"qdr:w"` past week, `"qdr:m"` past month). | +| `location` | `str` | Geo-targeting location string. | +| `ignore_invalid_urls` | `bool` | Exclude invalid URLs from results. | +| `timeout` | `int` | Timeout in ms. Default: `300000`. | +| `highlights` | `bool` | Generate query-relevant highlights. | +| `scrape_options` | `ScrapeOptions` | Scrape options applied to each result page. | +| `enterprise` | `list[str]` | Enterprise zero data retention options. | +| `threat_protection` | `ThreatProtectionOptions` | Per-request threat protection override. | +| `integration` | `str` | Integration identifier. | + +Response groups results by source: `results.web`, `results.news`, `results.images`, `results.developer`. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`scrape(url, **kwargs)` + +### Example + +```python +result = app.scrape("https://example.com", formats=["markdown", "links"], only_main_content=True) + +print(result.markdown) +print(result.links) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `url` | `str` | **Required.** URL to scrape. | +| `formats` | `list` | Output formats: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"images"`, `"screenshot"`, `"summary"`, `"json"`, `"changeTracking"`, `"attributes"`, `"branding"`, `"product"`, `"menu"`, `"audio"`, `"video"`. Also accepts format config dicts. Default: `["markdown"]`. | +| `headers` | `dict[str, str]` | Custom HTTP headers. | +| `include_tags` | `list[str]` | HTML tags to include. | +| `exclude_tags` | `list[str]` | HTML tags to exclude. | +| `only_main_content` | `bool` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `int` | Timeout in ms. Default: `60000`. Min: `1000`, Max: `300000`. | +| `wait_for` | `int` | Extra delay in ms before fetching content. | +| `mobile` | `bool` | Emulate a mobile device. | +| `parsers` | `list` | File processing controls. Accepts `"pdf"` or `{"type": "pdf", "mode": "fast"\|"auto"\|"ocr", "max_pages": int}`. | +| `actions` | `list` | Browser actions before content capture: `WaitAction`, `ClickAction`, `WriteAction`, `PressAction`, `ScrollAction`, `ScreenshotAction`, `ScrapeAction`, `ExecuteJavascriptAction`, `PDFAction`. | +| `location` | `Location` | Geo settings with `country` and `languages`. Country defaults to `"US"`. | +| `skip_tls_verification` | `bool` | Skip TLS certificate verification. | +| `remove_base64_images` | `bool` | Remove base64 images from markdown. Default: `true`. | +| `fast_mode` | `bool` | Enable fast mode. | +| `block_ads` | `bool` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `str` | Proxy type: `"basic"`, `"stealth"`, `"enhanced"`, `"auto"`. Default: `"auto"`. | +| `max_age` | `int` | Cache threshold in ms. Returns cached version if younger. Default: 2 days. | +| `min_age` | `int` | Cache-only mode minimum age in ms. | +| `store_in_cache` | `bool` | Store result in Firecrawl cache. Default: `true`. | +| `lockdown` | `bool` | Cache-only, no outbound requests. | +| `redact_pii` | `bool \| RedactPIIOptions` | Redact PII. Pass `True` for defaults or a `RedactPIIOptions` object. | +| `threat_protection` | `ThreatProtectionOptions` | Threat protection override. | +| `audit_metadata` | `AuditMetadata` | SIEM logging with `username` field. | +| `profile` | `dict` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `str` | Integration identifier. | + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`interact(job_id, code=None, *, prompt=None, language="node", timeout=None, origin=None)` + +### Example + +```python +result = app.scrape("https://www.amazon.com", formats=["markdown"]) +scrape_id = result.metadata.scrape_id + +app.interact(scrape_id, prompt="Search for iPhone 16 Pro Max") +response = app.interact(scrape_id, prompt="Click on the first result and tell me the price") +print(response.output) + +app.stop_interaction(scrape_id) +``` + +### Parameters + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `str` | **Required.** Scrape job ID from `result.metadata.scrape_id`. | +| `code` | `str` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `str` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `Literal["python", "node", "bash"]` | Runtime for code execution. Default: `"node"`. | +| `timeout` | `int` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `str` | Origin label for telemetry. | + +Stop the session when done: + +```python +app.stop_interaction(scrape_id) +``` + +## Notes + +- Parameter names use **snake_case** (e.g. `only_main_content`, `include_tags`, `scrape_options`). +- `include_domains` and `exclude_domains` on search are mutually exclusive. +- `FirecrawlApp` is a deprecated alias for `Firecrawl`. Use `Firecrawl`. +- `AsyncFirecrawlApp` is a deprecated alias for `AsyncFirecrawl`. +- Deprecated method aliases (do not use in new code): + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + - `scrape_url()` → use `scrape()` + - `crawl_url()` → use `crawl()` + - `map_url()` → use `map()` + +## Source Of Truth + +- `firecrawl/apps/python-sdk/firecrawl/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/client.py` +- `firecrawl/apps/python-sdk/firecrawl/v2/types.py` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/agent-quickstart/rust.mdx b/agent-quickstart/rust.mdx new file mode 100644 index 00000000..88701dec --- /dev/null +++ b/agent-quickstart/rust.mdx @@ -0,0 +1,228 @@ +--- +title: "Rust Agent Quickstart" +description: "Canonical Firecrawl Rust quickstart for external agents using search, scrape, and interact." +--- + +# Firecrawl Rust Agent Quickstart + +Canonical quickstart for external agents integrating Firecrawl with Rust. Generated from SDK source and OpenAPI spec. + +## Install + +```bash +cargo add firecrawl +``` + +## Authenticate + +```rust +use firecrawl::Client; + +let client = Client::new("fc-YOUR-API-KEY")?; +``` + +For self-hosted instances: + +```rust +let client = Client::new_selfhosted("https://your-instance.example.com", Some("fc-YOUR-API-KEY"))?; +``` + +Set `FIRECRAWL_API_KEY` environment variable for keyless initialization (handled at the application level — the SDK accepts an empty or `None` key for keyless free tier). + +## When To Use What + +- **search**: Start with a query, discover relevant URLs, and get their content in one call. +- **scrape**: You already have a URL and want its page content as markdown, HTML, JSON, or other formats. +- **interact**: The page needs clicks, form fills, or post-scrape browser actions on a live session. + +## Search + +### Why use it + +Search the web for a query and get scraped content from the top results. Combines discovery and content extraction in one call. + +### Preferred SDK method + +`client.search(query, options)` + +### Example + +```rust +use firecrawl::{Client, SearchOptions}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let response = client.search("firecrawl web scraping", SearchOptions { + limit: Some(5), + ..Default::default() +}).await?; + +if let Some(web_results) = response.data.web { + for result in web_results { + println!("{:?}", result); + } +} +``` + +### Parameters + +All fields on `SearchOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `query` | `impl AsRef` | **Required.** First positional argument. Search query. | +| `limit` | `Option` | Max results. Default: 5, Max: 20. | +| `sources` | `Option>` | Sources: `Web`, `News`, `Images`. | +| `categories` | `Option>` | Categories: `Github`, `Research`, `Pdf`. | +| `include_domains` | `Option>` | Restrict to these domains. Cannot combine with `exclude_domains`. | +| `exclude_domains` | `Option>` | Exclude these domains. Cannot combine with `include_domains`. | +| `tbs` | `Option` | Time-based filter (e.g. `"qdr:d"` past day). | +| `location` | `Option` | Geo-targeting location string. | +| `ignore_invalid_urls` | `Option` | Exclude invalid URLs. | +| `timeout` | `Option` | Timeout in ms. | +| `highlights` | `Option` | Generate query-relevant highlights. Default: `true`. | +| `scrape_options` | `Option` | Scrape options applied to each result page. | +| `integration` | `Option` | Integration identifier. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +There is also a convenience method `client.search_and_scrape(query, limit)` that returns `Vec` directly. + +## Scrape + +### Why use it + +Get the content of a single URL as markdown, HTML, JSON, screenshots, or other formats. + +### Preferred SDK method + +`client.scrape(url, options)` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, Format}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let document = client.scrape("https://example.com", ScrapeOptions { + formats: Some(vec![Format::Markdown, Format::Links]), + only_main_content: Some(true), + ..Default::default() +}).await?; + +if let Some(md) = document.markdown { + println!("{}", md); +} +``` + +### Parameters + +All fields on `ScrapeOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `url` | `impl AsRef` | **Required.** First positional argument. URL to scrape. | +| `formats` | `Option>` | Output formats: `Markdown`, `Html`, `RawHtml`, `Links`, `Images`, `Screenshot`, `Summary`, `ChangeTracking`, `Json`, `Attributes`, `Branding`, `Product`, `Menu`, `Audio`, `Video`, `Question(QuestionFormat)`, `Highlights(HighlightsFormat)`. Default: `[Markdown]`. | +| `headers` | `Option>` | Custom HTTP headers. | +| `include_tags` | `Option>` | HTML tags to include. | +| `exclude_tags` | `Option>` | HTML tags to exclude. | +| `only_main_content` | `Option` | Strip navbars, footers, boilerplate. Default: `true`. | +| `timeout` | `Option` | Timeout in ms. Default: `60000`. | +| `wait_for` | `Option` | Extra delay in ms before fetching content. | +| `mobile` | `Option` | Emulate a mobile device. | +| `parsers` | `Option>` | File processing controls (e.g. PDF with mode and max_pages). | +| `actions` | `Option>` | Browser actions before content capture. | +| `location` | `Option` | Geo settings with `country` and `languages`. | +| `skip_tls_verification` | `Option` | Skip TLS certificate verification. | +| `remove_base64_images` | `Option` | Remove base64 images from markdown. Default: `true`. | +| `fast_mode` | `Option` | Enable fast mode. | +| `block_ads` | `Option` | Block ads and cookie popups. Default: `true`. | +| `proxy` | `Option` | Proxy type: `Basic`, `Stealth`, `Enhanced`, `Auto`. Default: `Auto`. | +| `max_age` | `Option` | Cache threshold in seconds. | +| `min_age` | `Option` | Cache-only mode minimum age in seconds. | +| `store_in_cache` | `Option` | Store result in cache. Default: `true`. | +| `lockdown` | `Option` | Cache-only, no outbound requests. | +| `redact_pii` | `Option` | Redact PII. | +| `audit_metadata` | `Option` | SIEM logging with `username` field. | +| `profile` | `Option` | Persistent browser profile with `name` and optional `save_changes`. | +| `integration` | `Option` | Integration identifier. | +| `json_options` | `Option` | JSON extraction with `schema`, `system_prompt`, `prompt`. | +| `screenshot_options` | `Option` | Screenshot config with `full_page`, `quality`, `viewport`. | +| `change_tracking_options` | `Option` | Change tracking with `modes`, `schema`, `prompt`, `tag`. | +| `attribute_selectors` | `Option>` | Attribute extraction with `selector` and `attribute`. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +There is also a convenience method `client.scrape_with_schema(url, schema, prompt)` for JSON extraction. + +## Interact + +### Why use it + +Control a live browser session tied to a scrape job. Click buttons, fill forms, navigate, and extract dynamic content using code or natural-language prompts. + +### Preferred SDK method + +`client.interact(job_id, options)` + +### Example + +```rust +use firecrawl::{Client, ScrapeOptions, ScrapeExecuteOptions, Format}; + +let client = Client::new("fc-YOUR-API-KEY")?; +let document = client.scrape("https://www.amazon.com", ScrapeOptions { + formats: Some(vec![Format::Markdown]), + ..Default::default() +}).await?; + +let scrape_id = document.metadata + .and_then(|m| m.get("scrapeId").and_then(|v| v.as_str().map(String::from))) + .expect("scrapeId not found"); + +let response = client.interact(&scrape_id, ScrapeExecuteOptions { + prompt: Some("Search for iPhone 16 Pro Max".into()), + ..Default::default() +}).await?; + +let response = client.interact(&scrape_id, ScrapeExecuteOptions { + prompt: Some("Click on the first result and tell me the price".into()), + ..Default::default() +}).await?; +println!("{:?}", response.output); + +client.stop_interaction(&scrape_id).await?; +``` + +### Parameters + +All fields on `ScrapeExecuteOptions` are `Option` and default to `None`. + +| Parameter | Type | Description | +|---|---|---| +| `job_id` | `impl AsRef` | **Required.** First positional argument. Scrape job ID from document metadata. | +| `code` | `Option` | Code to execute in the browser sandbox. One of `code` or `prompt` required. | +| `prompt` | `Option` | Natural-language instruction. One of `code` or `prompt` required. | +| `language` | `Option` | Runtime: `Python`, `Node`, `Bash`. Default: `Node`. | +| `timeout` | `Option` | Execution timeout in seconds. Min: 1, Max: 300. | +| `origin` | `Option` | Auto-set to SDK version if `None`. | + +Stop the session when done: + +```rust +client.stop_interaction(&scrape_id).await?; +``` + +## Notes + +- All option structs use **snake_case** fields and derive `Default` — use struct literal syntax with `..Default::default()`. +- The `options` parameter on `scrape` and `search` accepts `impl Into>`, so you can pass `None` directly to skip options. +- The `origin` field is automatically set to the SDK version string if not provided. +- Deprecated aliases (do not use in new code): + - `scrape_execute()` → use `interact()` + - `stop_interactive_browser()` / `delete_scrape_browser()` → use `stop_interaction()` + +## Source Of Truth + +- `firecrawl/apps/rust-sdk/src/client.rs` +- `firecrawl/apps/rust-sdk/src/scrape.rs` +- `firecrawl/apps/rust-sdk/src/search.rs` +- `firecrawl/apps/rust-sdk/Cargo.toml` +- `firecrawl-docs/api-reference/v2-openapi.json` diff --git a/docs.json b/docs.json index 922f9b17..606d97c2 100755 --- a/docs.json +++ b/docs.json @@ -283,6 +283,16 @@ } ] }, + { + "group": "Agent Quickstarts", + "pages": [ + "agent-quickstart/node", + "agent-quickstart/python", + "agent-quickstart/rust", + "agent-quickstart/java", + "agent-quickstart/elixir" + ] + }, { "group": "Developer Guides", "pages": [