seo: noindex stale api-reference/introduction and redirect it to v2

[claude]

Requested by Rak Ramprakash · Slack thread

Before / After

Before: Googling "firecrawl docs" can land you on docs.firecrawl.dev/api-reference/introduction — a stale v1-era API reference intro that still features the deprecated Extract endpoint. The page is in no navigation group, but it has no noindex, no canonical, and nothing redirecting off it, so Google keeps serving it above the real API reference intro. Rak hit this from a plain Google search today.

After: That URL redirects to the current reference intro (/api-reference/v2-introduction) and is marked noindex, so it drops out of the index on the next crawl.

How

Two files:

api-reference/introduction.mdx — added noindex: true to the frontmatter. The file itself stays; the repo convention is redirect-never-remove. Frontmatter style matches v1-welcome.mdx, which already carries hidden: true + noindex: true.

docs.json (redirects array) — three changes, all landing on /api-reference/v2-introduction:

source	before	after
/api-reference/v1-introduction	/api-reference/introduction	/api-reference/v2-introduction
/api-reference/introduction	(none)	/api-reference/v2-introduction
/api-reference	(none)	/api-reference/v2-introduction

Retargeting the first one avoids a two-hop redirect chain. The bare /api-reference path had no redirect at all before this — verified against the full 117-entry array.

Checks run before pushing:

• docs.json parses cleanly under both node -e "JSON.parse(...)" and python3 -m json.tool.
• No duplicate source values anywhere in the redirects array (117 → 119 entries, zero collisions).
• No redirect still points at /api-reference/introduction as a destination.
• Key order and formatting match the existing { "destination": ..., "source": ... } entries.

One thing reviewers should confirm on the preview deploy

In Mintlify, a redirect whose source matches an existing page file may not fire — the page can win over the redirect. That is exactly why the noindex on the file is here as well as the redirect: if the redirect is shadowed, the noindex still de-indexes the page; if the redirect fires, the page is unreachable anyway. Please confirm on the preview deploy which one actually takes precedence for /api-reference/introduction. If the redirect does not fire, this PR still solves the SEO problem, but the stale content stays reachable and we may want a follow-up.

Locale copies — deliberately not touched

All five non-English locale trees do have their own copy: es/, fr/, ja/, pt-BR/, and zh/ each contain api-reference/introduction.mdx, and none of them carries noindex today.

They were not edited here, because CLAUDE.md states that localized files must not be modified directly — translations are managed separately by the Locadex pipeline. There is good evidence the flag propagates on its own: every locale copy of v1-welcome.mdx already carries hidden: true + noindex: true, inherited from the base file, so noindex is a frontmatter key the translation pipeline does carry through. The locale pages should pick this up on the next translation run.

Flagging it explicitly in case a maintainer wants them updated in this PR instead — happy to add them if the pipeline assumption is wrong.

Follow-up (not part of this PR)

The broader v1 de-indexing sweep is deliberately out of scope here, but worth filing:

• No v0 or v1 page carries noindex today. Concretely: 0 of the 109 .mdx files across v1/, v0/, and api-reference/v1-endpoint/. They are only hidden from the navigation, which does not prevent crawling — hidden-from-nav and non-indexable are different things, and this PR's page is the proof.
• Any sweep must target those three file trees, not the hidden nav group. The hidden group in docs.json is not a safe proxy for "deprecated": the hidden Extract Endpoints group re-lists live v2 slugs (api-reference/endpoint/extract, api-reference/endpoint/extract-get), so de-indexing by nav group would take out current pages.

Note on timing

Merging this will not change the Google result immediately — Google needs to recrawl the URL before the stale page drops out. We have Search Console access (there is a google-site-verification token in docs.json), so the recrawl can be requested manually once this ships, rather than waiting for the organic crawl.

---

Generated by Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
firecrawl	🟢 Ready	View Preview	Aug 11, 2026, 2:09 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.