Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion agent-source-of-truth/curl.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ curl -X POST "https://api.firecrawl.dev/v2/search" \

Successful responses include `success`, `data`, optional `warning`, `id`, and `creditsUsed`.

- `data.web`, `data.images`, `data.news`: result arrays; which keys appear depends on `sources` (by default only `data.web` is populated).
- `data.web`, `data.images`, `data.news`, `data.developer`: result arrays; which keys appear depends on `sources` and `categories` (by default only `data.web` is populated).
- Web and news items include fields such as `title`, `url`, and (when `scrapeOptions` / formats request it) `markdown`, `html`, `rawHtml`, `links`, `screenshot`, `audio`, `video`, and `metadata`.
- Image items include fields such as `imageUrl`, `url`, and dimensions when available.
- `warning`: optional human-readable notice.
Expand Down Expand Up @@ -104,6 +104,7 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `{ "type": "github" }`
- `{ "type": "research" }`
- `{ "type": "pdf" }`
- `{ "type": "developer" }`

- `limit`
- Type: integer (minimum 1, maximum 100, default 5)
Expand Down Expand Up @@ -136,6 +137,18 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `"zdr"`: end-to-end zero data retention
- `"anon"`: anonymized zero data retention

- `includeDomains`
- Type: array of strings
- Use when: you want to restrict results to specific domains. Mutually exclusive with `excludeDomains`.

- `excludeDomains`
- Type: array of strings
- Use when: you want to exclude specific domains. Mutually exclusive with `includeDomains`.

- `highlights`
- Type: boolean (default true)
- Use when: you want query-relevant highlights in results.

- `scrapeOptions`
- Type: object
- Use when: you want to scrape each search result (see Scrape parameters for fields).
Expand Down Expand Up @@ -230,11 +243,15 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- `"branding"`: branding profile output
- `"audio"`: audio extraction
- `"video"`: video extraction
- `"product"`: product profile output
- `"menu"`: menu profile output
- Format object fields:
- `type`: one of the format strings above
- `prompt`, `schema`: JSON extraction options for `type: "json"`
- `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"`
- `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"`
- `{ "type": "question", "question": "..." }`: question-answer style extraction
- `{ "type": "highlights", "query": "..." }`: relevant source-text extraction

- `headers`
- Type: object
Expand Down Expand Up @@ -324,6 +341,22 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Type: boolean
- Use when: you want zero data retention for this scrape.

- `lockdown`
- Type: boolean
- Use when: you want to serve only previously cached results (no outbound request).

- `redactPII`
- Type: boolean or object
- Use when: you want PII redaction.

- `auditMetadata`
- Type: object with `username` string
- Use when: you need SIEM logging attribution.

- `fastMode`
- Type: boolean
- Use when: you want faster scrapes with reduced fidelity.

## Interact

### Why use it
Expand Down Expand Up @@ -367,6 +400,7 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape/<jobId>/interact" \
- `code` (JSON body)
- Type: string (required in OpenAPI; min length 1, max length 100000)
- Use when: you want to run code in the scrape-bound browser sandbox.
- Notes: while OpenAPI marks `code` as required, the server also accepts `prompt` for natural-language instructions. At least one of `code` or `prompt` must be provided.

- `language` (JSON body)
- Type: string
Expand Down
31 changes: 30 additions & 1 deletion agent-source-of-truth/elixir.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source
Add to `mix.exs`:

```elixir
{:firecrawl, "~> 1.0.0"}
{:firecrawl, "~> 1.9"}
```

## Authenticate
Expand All @@ -23,6 +23,8 @@ config :firecrawl, api_key: System.get_env("FIRECRAWL_API_KEY")
{:ok, res} = Firecrawl.search_and_scrape([query: "site:docs.firecrawl.dev webhook retries"], api_key: "fc-your-api-key")
```

A nil or empty API key is permitted: `scrape`, `search`, and `interact` fall back to a keyless free tier (rate-limited per IP). Per-request options also support `:base_url` for self-hosted instances. The SDK appends an `origin` field for telemetry.

## When To Use What

- `search`: use when you start with a query and need discovery.
Expand Down Expand Up @@ -121,6 +123,18 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or
- Type: integer
- Use when: you need a request timeout in milliseconds.

- `include_domains`
- Type: list of strings
- Use when: you want to restrict results to specific domains. Mutually exclusive with `exclude_domains`.

- `exclude_domains`
- Type: list of strings
- Use when: you want to exclude specific domains. Mutually exclusive with `include_domains`.

- `highlights`
- Type: boolean
- Use when: you want query-relevant highlights. Defaults to true.

- `enterprise`
- Type: list of strings
- Use when: you need enterprise search controls.
Expand Down Expand Up @@ -302,6 +316,18 @@ Use scrape when you already have a URL and want structured content in one or mor
- Type: boolean
- Use when: you want zero data retention for this scrape.

- `lockdown`
- Type: boolean
- Use when: you want to serve only previously cached results (no outbound request).

- `redact_pii`
- Type: boolean
- Use when: you want PII redaction.

- `audit_metadata`
- Type: keyword list (keys: `username: :string`)
- Use when: you need SIEM logging attribution.

## Interact

### Why use it
Expand Down Expand Up @@ -370,6 +396,9 @@ Use interact when a page requires browser actions or code execution after a scra
- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec.
- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`.
- This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`).
- Keyless free tier: `scrape`, `search`, and `interact` work without an API key (rate-limited per IP).
- NimbleOptions validates all parameters at call time.
- Runtime dependency: `req ~> 0.5`, `nimble_options ~> 1.1`.

## Source Of Truth

Expand Down
48 changes: 45 additions & 3 deletions agent-source-of-truth/java.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,14 @@ Maven:
<dependency>
<groupId>com.firecrawl</groupId>
<artifactId>firecrawl-java</artifactId>
<version>1.2.0</version>
<version>1.12.1</version>
</dependency>
```

Gradle:

```gradle
implementation("com.firecrawl:firecrawl-java:1.2.0")
implementation("com.firecrawl:firecrawl-java:1.12.1")
```

## Authenticate
Expand All @@ -33,6 +33,18 @@ FirecrawlClient client = FirecrawlClient.builder()
.build();
```

Additional builder methods:

- `timeoutMs(long)` — HTTP request timeout in milliseconds (default 300000)
- `maxRetries(int)` — maximum retry attempts (default 3)
- `backoffFactor(double)` — exponential backoff multiplier (default 0.5)
- `asyncExecutor(Executor)` — custom executor for async methods
- `httpClient(OkHttpClient)` — custom OkHttpClient instance

Static factory: `FirecrawlClient.fromEnv()` creates a client using environment variables.

The API key can be null or blank for keyless free tier usage (rate-limited per IP).

## When To Use What

- `search`: use when you start with a query and need discovery.
Expand Down Expand Up @@ -148,6 +160,18 @@ SearchData results = client.search("site:docs.firecrawl.dev crawl webhooks", opt
- Type: `ScrapeOptions`
- Use when: you want to scrape each search result (see Scrape parameters for fields).

- `options.includeDomains`
- Type: `List<String>`
- Use when: you want to restrict results to specific domains.

- `options.excludeDomains`
- Type: `List<String>`
- Use when: you want to exclude specific domains.

- `options.highlights`
- Type: Boolean
- Use when: you want query-relevant highlights. Defaults to true.

- `options.integration`
- Type: String
- Use when: the API expects an integration identifier on the request.
Expand All @@ -165,7 +189,7 @@ Use scrape when you already have a URL and want structured content in one or mor

### Return value

`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, and additional fields when the corresponding formats are requested.
`scrape` returns `Document`. Typical getters include `getMarkdown()`, `getHtml()`, `getRawHtml()`, `getJson()`, `getMetadata()`, `getLinks()`, `getAudio()`, `getVideo()`, `getProduct()`, `getMenu()`, `getBranding()`, `getAnswer()`, `getHighlights()`, `getWarning()`, `getChangeTracking()`, and additional fields when the corresponding formats are requested.

### Simple Example

Expand Down Expand Up @@ -244,6 +268,9 @@ Document doc = client.scrape("https://example.com/pricing", options);
- `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"`
- `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"`
- `selectors`: array of `{selector, attribute}` for `type: "attributes"`
- Additional format objects:
- `QuestionFormat.builder().question("...").build()`: question-answer extraction
- `HighlightsFormat.builder().query("...").build()`: relevant source-text extraction

- `options.headers`
- Type: `Map<String, String>`
Expand Down Expand Up @@ -327,6 +354,18 @@ Document doc = client.scrape("https://example.com/pricing", options);
- Type: String
- Use when: the API expects an integration identifier on the request.

- `options.lockdown`
- Type: Boolean
- Use when: you want to serve only previously cached results.

- `options.redactPII`
- Type: Boolean
- Use when: you want PII redaction.

- `options.auditMetadata`
- Type: `AuditMetadata` (has `username: String`)
- Use when: you need SIEM logging attribution.

## Interact

### Why use it
Expand Down Expand Up @@ -417,6 +456,9 @@ BrowserDeleteResponse stopped = client.stopInteractiveBrowser("<scrapeJobId>");

- Deprecated aliases: `scrapeExecute` → `interact`, `deleteScrapeBrowser` → `stopInteractiveBrowser` (and the corresponding `*Async` helpers).
- The Java SDK exposes code-based interactions only: there is no `prompt` parameter on `interact` (unlike some other language SDKs).
- Keyless free tier: `scrape`, `search`, and `interact` work without an API key (rate-limited per IP).
- Every sync method has an async counterpart returning `CompletableFuture<T>` (e.g., `scrapeAsync`, `searchAsync`, `interactAsync`, `stopInteractiveBrowserAsync`).
- `FirecrawlClient.fromEnv()` creates a client using environment variables.

## Source Of Truth

Expand Down
55 changes: 53 additions & 2 deletions agent-source-of-truth/node.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ title: "Node.js Source of Truth"
description: "Canonical Firecrawl Node.js source of truth for agents using key endpoints like search, scrape, and interact."
---

Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.18.2** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API.
Canonical Firecrawl Node.js source of truth for agents. Aligned with `firecrawl` **v4.32.1** (`firecrawl/apps/js-sdk/firecrawl`) and the v2 OpenAPI spec. Method names, parameters, and types match the SDK public API.

## Install

Expand All @@ -19,7 +19,12 @@ import { Firecrawl } from "firecrawl";
const client = new Firecrawl({
apiKey: process.env.FIRECRAWL_API_KEY,
// apiUrl: "https://api.firecrawl.dev" // optional; falls back to FIRECRAWL_API_URL or cloud default
// maxRetries: 3, // optional; retry count for transient failures
// backoffFactor: 2 // optional; exponential backoff multiplier
});
// Or pass a plain string as the API key:
// const client = new Firecrawl("fc-key");
// API key can be omitted entirely for keyless free tier (rate-limited per IP).
```

## When To Use What
Expand Down Expand Up @@ -78,6 +83,7 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", {
- `web`: web index hits
- `news`: news hits
- `images`: image hits
- `developer`: developer-specific results (e.g., GitHub code search)

**Wrong turn to avoid:** `search()` does not return `{ data: [...] }`. Do not access `result.data`. Web results are in `result.web`, news in `result.news`, images in `result.images`.

Expand Down Expand Up @@ -107,7 +113,8 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", {
- `"github"`: GitHub-focused results
- `"research"`: research and academic results
- `"pdf"`: PDF-focused results
- `{ type: "github" | "research" | "pdf" }`: typed category object form
- `"developer"`: developer-specific results (e.g., GitHub code search)
- `{ type: "github" | "research" | "pdf" | "developer" }`: typed category object form

- `options.limit`
- Type: number
Expand All @@ -133,6 +140,26 @@ const results = await client.search("site:docs.firecrawl.dev crawl webhooks", {
- Type: `ScrapeOptions`
- Use when: you want to scrape each search result (see Scrape parameters for fields). The SDK runs the same validation as for `scrape` (for example plain string `"json"` in `formats` is rejected).

- `options.includeDomains`
- Type: array of strings
- Use when: you want to restrict results to specific domains. Mutually exclusive with `excludeDomains`.

- `options.excludeDomains`
- Type: array of strings
- Use when: you want to exclude specific domains from results. Mutually exclusive with `includeDomains`.

- `options.highlights`
- Type: boolean
- Use when: you want query-relevant highlights in results. Defaults to true.

- `options.enterprise`
- Type: array of strings
- Use when: you need enterprise search controls.
- Confirmed values:
- `"default"`
- `"anon"`: anonymized ZDR
- `"zdr"`: end-to-end zero data retention

## Scrape

### Why use it
Expand Down Expand Up @@ -208,6 +235,8 @@ const doc = await client.scrape("https://example.com/pricing", {
- `"branding"`: branding profile output
- `"audio"`: audio extraction
- `"video"`: video extraction
- `"product"`: product profile output
- `"menu"`: menu profile output
- Object-only format types (at minimum `type` as shown):
- `{ type: "json", prompt?: string, schema?: JSON schema or Zod schema }`: at least one of `prompt` or `schema` is required (SDK validation).
- `{ type: "question", question: string }`: question-answer style extraction.
Expand Down Expand Up @@ -316,6 +345,26 @@ const doc = await client.scrape("https://example.com/pricing", {
- Type: object with `name` and optional `saveChanges`
- Use when: you want a persistent browser profile shared across scrapes and interactions.

- `options.lockdown`
- Type: boolean
- Use when: you want to serve only previously cached results (no outbound request).

- `options.redactPII`
- Type: boolean or RedactPIIOptions
- Use when: you want PII redaction. RedactPIIOptions has `mode`, `entities`, `replaceStyle`.

- `options.auditMetadata`
- Type: object with `username` string
- Use when: you need SIEM logging attribution.

- `options.integration`
- Type: string
- Use when: you need an integration identifier for server-side tracking. Omit in agent-oriented examples unless your product intentionally sets it.

- `options.useMock`
- Type: string
- Use when: you want to use mock mode.

## Interact

### Why use it
Expand Down Expand Up @@ -530,6 +579,8 @@ console.log(result.answer);
- The default `Firecrawl` export is the v2 client; v1 remains under `client.v1`.
- Zod schemas passed to `formats` (for `json` or `changeTracking`) are converted to JSON Schema by the SDK.
- The package declares **Node.js >= 22** in `engines`.
- The constructor accepts a plain string as the API key: `new Firecrawl("fc-key")`.
- Keyless free tier: scrape, search, and interact work without an API key (rate-limited per IP).

## Source Of Truth

Expand Down
Loading