Skip to content

docs: refresh agent-source-of-truth with latest SDK parameters #1227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ericciarla wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

docs: refresh agent-source-of-truth with latest SDK parameters #1227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 60 additions & 11 deletions agent-source-of-truth/curl.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ curl -X POST "https://api.firecrawl.dev/v2/search" \
"query": "site:docs.firecrawl.dev crawl webhooks",
"sources": [{"type": "web"}, {"type": "news"}],
"categories": [{"type": "research"}],
"includeDomains": ["docs.firecrawl.dev"],
"highlights": true,
"limit": 10,
"tbs": "qdr:m",
"location": "San Francisco,California,United States",
Expand Down Expand Up @@ -105,8 +107,18 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `{ "type": "research" }`
- `{ "type": "pdf" }`

- `includeDomains`
- Type: array of strings (hostnames)
- Use when: you want to restrict results to specific domains.
- Notes: cannot be combined with `excludeDomains`.

- `excludeDomains`
- Type: array of strings (hostnames)
- Use when: you want to exclude results from specific domains.
- Notes: cannot be combined with `includeDomains`.

- `limit`
- Type: integer (minimum 1, maximum 100, default 5)
- Type: integer (minimum 1, maximum 100, default 10)
- Use when: you want to cap results.

- `tbs`
Expand All @@ -121,6 +133,10 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- Type: string (default `"US"`)
- Use when: you want ISO 3166-1 alpha-2 targeting (for example `"US"`).

- `highlights`
- Type: boolean (default true)
- Use when: you want query-relevant highlights for search results.

- `ignoreInvalidURLs`
- Type: boolean (default false)
- Use when: you want to drop URLs that cannot be scraped by other endpoints.
Expand All @@ -136,6 +152,11 @@ Successful responses include `success`, `data`, optional `warning`, `id`, and `c
- `"zdr"`: end-to-end zero data retention
- `"anon"`: anonymized zero data retention

- `threatProtection`
- Type: object
- Use when: you need per-request threat protection override (enterprise).
- Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (array, max 1000), `whitelist` (array, max 1000), `blockedTlds` (array, max 1000), `failurePolicy` (`"open"` | `"closed"`)

- `scrapeOptions`
- Type: object
- Use when: you want to scrape each search result (see Scrape parameters for fields).
Expand Down Expand Up @@ -175,7 +196,9 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape" \
"links",
{"type": "json", "prompt": "Extract plan names and prices."},
{"type": "screenshot", "fullPage": true, "quality": 80, "viewport": {"width": 1280, "height": 720}},
{"type": "changeTracking", "modes": ["git-diff"], "tag": "pricing"}
{"type": "changeTracking", "modes": ["git-diff"], "tag": "pricing"},
{"type": "question", "question": "What are the pricing tiers?"},
{"type": "highlights", "query": "enterprise pricing"}
],
"headers": {"User-Agent": "FirecrawlDocsBot/1.0"},
"onlyMainContent": true,
Expand All @@ -193,8 +216,7 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape" \
"maxAge": 86400000,
"minAge": 1,
"storeInCache": true,
"profile": {"name": "docs-session", "saveChanges": true},
"zeroDataRetention": false
"profile": {"name": "docs-session", "saveChanges": true}
}'
```

Expand All @@ -203,10 +225,15 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape" \
Successful responses include `success` and `data`. Common `data` fields (depending on `formats` and options):

- `markdown`, `summary`, `html`, `rawHtml`, `screenshot`, `audio`, `video`, `links`
- `answer`: present when the `question` format is requested
- `highlights`: present when the `highlights` format is requested
- `actions`: when the request included scrape-time `actions`, contains ordered results such as `screenshots`, `scrapes`, `javascriptReturns`, and `pdfs`
- `metadata`: page metadata (`title`, `sourceURL`, `url`, `statusCode`, `error`, and other extracted fields)
- `warning`: optional extraction or formatting notice
- `changeTracking`: present when the `changeTracking` format is requested
- `branding`: brand design system data when `branding` format requested
- `product`: product data when `product` format requested
- `menu`: menu data when `menu` format requested

### Parameters

Expand All @@ -228,13 +255,16 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- `"changeTracking"`: change tracking output
- `"json"`: JSON extraction
- `"branding"`: branding profile output
- `"product"`: product data extraction
- `"menu"`: menu extraction
- `"audio"`: audio extraction
- `"video"`: video extraction
- Format object fields:
- `type`: one of the format strings above
- `prompt`, `schema`: JSON extraction options for `type: "json"`
- `modes`, `schema`, `prompt`, `tag`: change tracking options for `type: "changeTracking"`
- `fullPage`, `quality`, `viewport`: screenshot options for `type: "screenshot"`
- Object format types:
- `{"type": "json", "prompt": "...", "schema": {...}}`: JSON extraction
- `{"type": "question", "question": "..."}`: question-answer extraction. Answer in `data.answer`.
- `{"type": "highlights", "query": "..."}`: relevant source-text extraction. Result in `data.highlights`.
- `{"type": "screenshot", "fullPage": bool, "quality": int, "viewport": {...}}`: screenshot options
- `{"type": "changeTracking", "modes": [...], "schema": {...}, "prompt": "...", "tag": "..."}`: change tracking

- `headers`
- Type: object
Expand Down Expand Up @@ -316,6 +346,25 @@ Successful responses include `success` and `data`. Common `data` fields (dependi
- Type: boolean
- Use when: you want Firecrawl to cache the result.

- `lockdown`
- Type: boolean
- Use when: you want to serve only previously cached results; never make outbound requests.

- `redactPII`
- Type: boolean or object
- Use when: you want to redact personally identifiable information from returned content.
- Object fields: `mode` (`"accurate"` | `"aggressive"` | `"fast"`, default `"accurate"`), `entities` (array of `"PERSON"` | `"EMAIL"` | `"PHONE"` | `"LOCATION"` | `"FINANCIAL"` | `"SECRET"`), `replaceStyle` (`"tag"` | `"mask"` | `"remove"`, default `"tag"`)

- `threatProtection`
- Type: object
- Use when: you need per-request threat protection override (enterprise).
- Fields: `mode` (`"off"` | `"normal"`), `riskScoreThreshold` (0-100), `blacklist` (array, max 1000), `whitelist` (array, max 1000), `blockedTlds` (array, max 1000), `failurePolicy` (`"open"` | `"closed"`)

- `auditMetadata`
- Type: object
- Use when: you need user attribution for SIEM logging events.
- Fields: `username` (string, required, max 1024 chars)

- `profile`
- Type: object with `name` and optional `saveChanges`
- Use when: you want a persistent browser profile shared across scrapes and interactions.
Expand Down Expand Up @@ -379,7 +428,7 @@ curl -X POST "https://api.firecrawl.dev/v2/scrape/<jobId>/interact" \

### Response

Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, and `error` (nullable).
Successful responses include `success` plus execution fields such as `stdout`, `result` (alias of stdout), `stderr`, `exitCode`, `killed`, `cdpUrl`, `liveViewUrl`, `interactiveLiveViewUrl`, and `error` (nullable).

### DELETE /scrape/{jobId}/interact

Expand Down Expand Up @@ -489,4 +538,4 @@ curl -X POST "https://api.firecrawl.dev/v2/support/docs-search" \
- `firecrawl-docs/api-reference/v2-openapi.json`
- `firecrawl/apps/js-sdk/firecrawl/src/v2/types.ts`
- `firecrawl/apps/python-sdk/firecrawl/v2/types.py`
- `firecrawl/apps/rust-sdk/src/v2/scrape.rs`
- `firecrawl/apps/rust-sdk/src/scrape.rs`
47 changes: 41 additions & 6 deletions agent-source-of-truth/elixir.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ title: "Elixir Source of Truth"
description: "Canonical Firecrawl Elixir source of truth for agents using key endpoints like search, scrape, and interact."
---

Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source and the v2 OpenAPI spec.
Canonical Firecrawl Elixir source of truth for agents. Generated from SDK source (`:firecrawl` **v1.9.1**) and the v2 OpenAPI spec.

## Install

Add to `mix.exs`:

```elixir
{:firecrawl, "~> 1.0.0"}
{:firecrawl, "~> 1.9"}
```

## Authenticate
Expand Down Expand Up @@ -52,6 +52,8 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or
query: "site:docs.firecrawl.dev crawl webhooks",
sources: [:web, :news],
categories: [:research],
include_domains: ["docs.firecrawl.dev"],
highlights: true,
limit: 10,
tbs: "qdr:m",
location: "San Francisco,California,United States",
Expand Down Expand Up @@ -97,6 +99,14 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or
- `"pdf"` or `:pdf`
- `%{type: "github" | "research" | "pdf"}`

- `include_domains`
- Type: list of strings
- Use when: you want to restrict results to specific domains.

- `exclude_domains`
- Type: list of strings
- Use when: you want to exclude results from specific domains.

- `limit`
- Type: integer
- Use when: you want to cap results.
Expand All @@ -113,6 +123,11 @@ Use search to discover relevant pages from a query, then pick URLs to scrape or
- Type: string
- Use when: you want ISO 3166-1 alpha-2 targeting (for example `"US"`).

- `highlights`
- Type: boolean
- Use when: you want query-relevant highlights for search results.
- Notes: defaults to `true` server-side.

- `ignore_invalid_urls`
- Type: boolean
- Use when: you want to drop URLs that cannot be scraped by other endpoints.
Expand Down Expand Up @@ -161,7 +176,9 @@ Use scrape when you already have a URL and want structured content in one or mor
"links",
%{type: "json", prompt: "Extract plan names and prices."},
%{type: "screenshot", fullPage: true, quality: 80, viewport: %{width: 1280, height: 720}},
%{type: "changeTracking", modes: ["git-diff"], tag: "pricing"}
%{type: "changeTracking", modes: ["git-diff"], tag: "pricing"},
%{type: "question", question: "What are the pricing tiers?"},
%{type: "highlights", query: "enterprise pricing"}
],
headers: %{"User-Agent" => "FirecrawlDocsBot/1.0"},
only_main_content: true,
Expand All @@ -179,8 +196,7 @@ Use scrape when you already have a URL and want structured content in one or mor
max_age: 86400000,
min_age: 1,
store_in_cache: true,
profile: [name: "docs-session", save_changes: true],
zero_data_retention: false
profile: [name: "docs-session", save_changes: true]
)
```

Expand All @@ -204,8 +220,14 @@ Use scrape when you already have a URL and want structured content in one or mor
- `"changeTracking"`: change tracking output
- `"json"`: JSON extraction
- `"branding"`: branding profile output
- `"product"`: product data extraction
- `"menu"`: menu extraction
- `"audio"`: audio extraction
- `"video"`: video extraction
- Format map types:
- `%{type: "json", prompt: "...", schema: %{...}}`: JSON extraction
- `%{type: "question", question: "..."}`: question-answer extraction
- `%{type: "highlights", query: "..."}`: relevant source-text extraction
- Format map fields:
- `type`: one of the format strings above
- `prompt`, `schema`: JSON extraction options for `type: "json"`
Expand Down Expand Up @@ -294,6 +316,18 @@ Use scrape when you already have a URL and want structured content in one or mor
- Type: boolean
- Use when: you want Firecrawl to cache the result.

- `lockdown`
- Type: boolean
- Use when: you want to serve only previously cached results; never make outbound requests.

- `redact_pii`
- Type: boolean
- Use when: you want to redact personally identifiable information from returned content.

- `audit_metadata`
- Type: keyword list with `username:` (required)
- Use when: you need user attribution for SIEM logging events.

- `profile`
- Type: keyword list with `name:` and optional `save_changes:` (or `saveChanges:`)
- Use when: you want a persistent browser profile shared across scrapes and interactions.
Expand Down Expand Up @@ -341,7 +375,7 @@ Use interact when a page requires browser actions or code execution after a scra
- Use when: you have a scrape job ID.

- `code`
- Type: string
- Type: string (required)
- Use when: you want to run code in the browser session.

- `language`
Expand Down Expand Up @@ -370,6 +404,7 @@ Use interact when a page requires browser actions or code execution after a scra
- The Elixir client is OpenAPI-shaped; function names and parameter keys are generated from the spec.
- Each public function has a bang (`!`) variant that raises on error instead of returning `{:error, _}`.
- This SDK exposes code-based interactions only (no `prompt` parameter on `interact_with_scrape_browser_session`).
- All parameter validation uses `NimbleOptions`.

## Source Of Truth

Expand Down
Loading