diff --git a/mcp-server/tools.mdx b/mcp-server/tools.mdx index 06e5e30cf..37a565cde 100644 --- a/mcp-server/tools.mdx +++ b/mcp-server/tools.mdx @@ -193,7 +193,7 @@ The upload command sends the file bytes to a short-lived signed upload target. I ### 5. Crawl Tool (`firecrawl_crawl`) -Start an asynchronous crawl with advanced options. +Start a crawl with advanced options. This MCP call blocks while the server polls the crawl to a terminal state, then returns the final status and collected data. ```json { @@ -249,24 +249,22 @@ Extract structured information from web pages using LLM capabilities. Supports b } ``` -Example response: +Example MCP response (`content[0].text` contains serialized JSON): ```json { "content": [ { "type": "text", - "text": { - "name": "Example Product", - "price": 99.99, - "description": "This is an example product description" - } + "text": "{\n \"name\": \"Example Product\",\n \"price\": 99.99,\n \"description\": \"This is an example product description\"\n}" } ], "isError": false } ``` +Parse `content[0].text` as JSON to access the decoded extracted payload. + #### Extract Tool Options: - `urls`: Array of URLs to extract information from diff --git a/migrate-to-v2.mdx b/migrate-to-v2.mdx index 07d4101f9..63feb28f8 100644 --- a/migrate-to-v2.mdx +++ b/migrate-to-v2.mdx @@ -85,8 +85,8 @@ og:description: "Key changes, mappings, and before/after snippets to upgrade you |------------------------------------|-------------------| | `generateLLMsText(...)` | (not in v2 SDK) | | `checkGenerateLLMsTextStatus(id)` | (not in v2 SDK) | -| `crawlUrlAndWatch(...)` | `watcher(jobId, ...)` | -| `batchScrapeUrlsAndWatch(...)` | `watcher(jobId, ...)` | +| `crawlUrlAndWatch(...)` | `watcher(jobId, options?)` | +| `batchScrapeUrlsAndWatch(...)` | `watcher(jobId, options?)` | --- diff --git a/quickstarts/nodejs.mdx b/quickstarts/nodejs.mdx index d6afb6818..dfae19db2 100644 --- a/quickstarts/nodejs.mdx +++ b/quickstarts/nodejs.mdx @@ -7,7 +7,7 @@ og:description: "Get started with Firecrawl in Node.js. Scrape, search, and inte ## Prerequisites -- Node.js 18+ +- Node.js 22+ - A Firecrawl API key — [get one free](https://www.firecrawl.dev/app/api-keys) ## Install the SDK diff --git a/quickstarts/python.mdx b/quickstarts/python.mdx index 0647f17a9..60c106afe 100644 --- a/quickstarts/python.mdx +++ b/quickstarts/python.mdx @@ -41,7 +41,7 @@ print(result.markdown) "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...", "metadata": { "title": "Example Domain", - "sourceURL": "https://example.com" + "source_url": "https://example.com" } } ``` diff --git a/reconciliation/INITIAL_DRIFT_REPORT.md b/reconciliation/INITIAL_DRIFT_REPORT.md new file mode 100644 index 000000000..0985b8340 --- /dev/null +++ b/reconciliation/INITIAL_DRIFT_REPORT.md @@ -0,0 +1,89 @@ +# Initial documentation drift report + +Initial verification date: 2026-08-02
+Narrowed pull-request verification date: 2026-08-06
+Documentation scope: base English/source files
+Publication state: [PR #1195](https://github.com/firecrawl/firecrawl-docs/pull/1195) is open; the six documentation fixes are not active until merge. Reconciliation automation is deferred to a follow-up change. + +## TL;DR + +- The first reconciliation pass classified **26 candidates**: 11 confirmed drifts, 8 generated-source lags, 2 Cloud/open-source differences, 2 contradictory documentation cases, and 3 items requiring owner decisions. +- This change fixes six high-confidence mismatches across Node.js, Python, CLI, and MCP documentation. +- This report records the source revisions and evidence used to verify those fixes; executable reconciliation is intentionally deferred. +- The remaining 20 candidates stay open or decision-dependent and provide the initial coverage backlog. + +## Classification totals + +| Classification | Count | +| --- | ---: | +| Confirmed drift | 11 | +| Generated-source lag | 8 | +| Intentional Cloud/open-source difference | 2 | +| Duplicate or contradictory documentation | 2 | +| Requires owner input | 3 | + +Severity totals are 6 high, 18 medium, and 2 low. + +## Source revisions + +- Docs base inspected during the initial pass: `firecrawl/firecrawl-docs@74685645ef42be41278f18278c708ab7547f3bb6` +- Product and bundled SDKs: `firecrawl/firecrawl@9554ad079840b0d405d5b1e5b1c57e577b4249cb` +- CLI: `firecrawl/cli@a151277b48a95f5728dc4dbc0dc7bd6d18709408` +- MCP: `firecrawl/firecrawl-mcp-server@41c257161d6b29a849fb66e097d5e5beccefdf2a` + +These revisions are evidence snapshots for the initial classification, not a claim that the repositories have remained unchanged. + +The six fixed findings were reverified against the published JavaScript SDK `4.32.0`, Python SDK `4.34.0`, CLI `1.19.29`, and MCP server `3.23.4` packages on 2026-08-06. + +## Candidate index + +| ID | Severity | Classification | Disposition | Affected documentation | Finding | +| --- | --- | --- | --- | --- | --- | +| API-001 | High | Requires owner input | Open | v2 OpenAPI | The configured product comparator is a v1 spec, so an owner-approved production v2 source is required before operation parity can be judged. | +| API-002 | High | Confirmed drift | Open | v1/v2 OpenAPI; failed webhook snippet | Crawl and batch webhook schemas advertise job-level `failed` events, while the inspected runtime completes with failed pages instead. | +| API-003 | Medium | Generated-source lag | Open | v2 OpenAPI; advanced Scrape guide | The public `attributes` format is implemented and documented in the guide but absent from the v2 request and response schemas. | +| API-004 | Medium | Generated-source lag | Open | v1 OpenAPI; account endpoint pages | Three documented account operations are registered in product routes but absent from the product v1 OpenAPI artifact. | +| API-005 | Medium | Contradictory documentation | Open | Webhook events guide; callback OpenAPI | The shared payload guide omits the always-emitted `webhookId` required by the callback schema. | +| API-006 | High | Cloud/open-source difference | Open | Webhook security; self-hosting | Self-hosted delivery is unsigned unless an HMAC secret is configured, while the security guide states signatures universally. | +| API-007 | Medium | Cloud/open-source difference | Open | Webhook overview; self-hosting | Direct self-hosted delivery is single-attempt, so the universal retry schedule does not apply to that path. | +| API-008 | Medium | Generated-source lag | Open | Webhook callback OpenAPI | The callback specification omits implemented Extract started, completed, and failed events. | +| API-009 | Low | Contradictory documentation | Open | Webhook overview and events guide | The overview omits implemented `monitor.page`, while the detailed guide and callback spec include it. | +| API-010 | Medium | Confirmed drift | Open | Webhook overview and testing | The docs require HTTPS, but the inspected request validation and delivery paths do not enforce the protocol. | +| JS-001 | High | Confirmed drift | **Fixed** | `quickstarts/nodejs.mdx` | The quickstart allowed Node.js 18+, while the pinned JavaScript SDK requires Node.js 22+. | +| JS-002 | Medium | Confirmed drift | **Fixed** | `sdks/node.mdx`; `migrate-to-v2.mdx` | V2 prose named the v1-only `crawlUrlAndWatch`; the v2 method is `watcher(jobId, options)`. | +| JS-003 | Medium | Generated-source lag | Open | Node agent source-of-truth page | The page version marker lags the pinned source manifest; published-package status was not queried. | +| JS-004 | Medium | Requires owner input | Open | Node install snippets and source-of-truth page | Workspace manifests do not unambiguously identify the canonical published package artifact. | +| PY-001 | Medium | Confirmed drift | **Fixed** | `quickstarts/python.mdx` | The quickstart used `metadata.sourceURL`, while the SDK normalizes it to `metadata.source_url`. | +| PY-002 | Medium | Generated-source lag | Open | Python agent source-of-truth page | The page version marker lags the pinned source declaration; published-package status was not queried. | +| CLI-001 | High | Confirmed drift | **Fixed** | `sdks/cli.mdx` | Single-format links output is newline-delimited, so the JSON pipeline must explicitly request JSON. | +| CLI-002 | Medium | Confirmed drift | Open | CLI Search page and snippets | Search `--pretty` is documented, but its option registration is disabled. | +| CLI-003 | Medium | Confirmed drift | Open | CLI option tables | Tables labeled “Available Options” omit implemented Scrape, Search, and Monitor flags. | +| CLI-004 | Low | Generated-source lag | Open | CLI status example | Illustrative output pins an older CLI version than the inspected source. | +| MCP-001 | High | Confirmed drift | **Fixed** | `mcp-server/tools.mdx` | `firecrawl_crawl` was described as asynchronous, but it polls to a terminal state before returning. | +| MCP-002 | Medium | Confirmed drift | **Fixed** | `mcp-server/tools.mdx` | The example showed `content[0].text` as an object, while the implementation returns a JSON-serialized string. | +| MCP-003 | Medium | Confirmed drift | Open | Local MCP guide | Local stdio supports bounded keyless Search and Scrape, although the guide states a key or self-hosted URL is always required. | +| DD-001 | Medium | Requires owner input | Open | Keyless Scrape snippets; v2 OpenAPI security | Docs and product support bounded keyless Scrape, while the OpenAPI operation declares bearer authentication. | +| DD-002 | Medium | Generated-source lag | Open | v2 OpenAPI `Formats`; Scrape snippet | The schema permits object formats only, while its description, default, docs, and verified runtime accept strings. | +| DD-003 | Medium | Generated-source lag | Open | v2 OpenAPI response metadata | `nullable` is combined with `oneOf` without a sibling `type`, preventing strict response-schema compilation. | + +## Verification status + +- JS-001 and JS-002 passed against JavaScript SDK `4.32.0`: its package requires Node.js 22 or newer, v2 exposes `watcher(jobId, opts)`, and `crawlUrlAndWatch` remains under v1. +- PY-001 passed against Python SDK `4.34.0`: v2 normalization maps `sourceURL` to `source_url`. +- CLI-001 passed against CLI `1.19.29`: single-format links output is newline-delimited, while `--json` returns a JSON object whose `links` field is an array. +- MCP-001 and MCP-002 passed against MCP server `3.23.4`: the crawl tool polls to a terminal state, and tool data is JSON-serialized into text. +- All six edited routes rendered the corrected content at 1440-by-900 desktop and 390-by-844 mobile viewports with visible main content and no horizontal overflow or browser errors. +- `mint validate` reported the same 17 pre-existing missing-import warnings as a clean checkout of current docs `main`; the narrowed change introduced no validation warning. +- `git diff --check` passed. +- No live request is required to verify the six documentation fixes in this change. + +## Coverage boundaries + +- The six fixes do not claim coverage of all 26 candidates or every documentation statement. +- Narrative claims outside the six fixed items were inventoried but are not verified by this pull request. +- Package-registry verification was limited to the four released artifacts supporting the six fixed findings; unresolved version candidates remain classified as generated-source lag rather than release verdicts. +- Cloud webhook retry behavior lives outside the pinned product repository. +- Localized documentation was not audited or edited. +- The dependency-free reconciler and any scheduling are deferred to a separately reviewed follow-up. +- Doc Detective and its npm dependency graph are deferred to a separate proposal. +- Broader semantic PR review, agent-proposed repairs, automatic branches, and automatic pull requests remain deferred to dependent changes with separate permission and security review. diff --git a/sdks/cli.mdx b/sdks/cli.mdx index bef8f84b8..d48948903 100644 --- a/sdks/cli.mdx +++ b/sdks/cli.mdx @@ -479,8 +479,8 @@ jq -r '.data.web[].url' search-results.json # Get titles from search results jq -r '.data.web[] | "\(.title): \(.url)"' search-results.json -# Extract links and process with jq -firecrawl https://example.com --format links | jq '.links[].url' +# Extract links as JSON and print one URL per line +firecrawl https://example.com --format links --json | jq -r '.links[]' # Count URLs from map firecrawl map https://example.com | wc -l diff --git a/sdks/node.mdx b/sdks/node.mdx index 30e1b350b..9888ebc68 100644 --- a/sdks/node.mdx +++ b/sdks/node.mdx @@ -111,7 +111,7 @@ To extract structured data from websites with error handling, use the `extractUr ### Crawling a Website with WebSockets -Stream crawl results in real time with the `crawlUrlAndWatch` method. You receive each page as it is crawled instead of waiting for the entire job to finish. +Stream crawl results in real time with `watcher(jobId, options)`. You receive each page as it is crawled instead of waiting for the entire job to finish.