Skip to content

robots.txt: block AI training crawlers; noindex the pages.dev addresses - #11

Merged
Deityhub merged 1 commit into
devfrom
chore/robot
Oct 29, 2026
Merged

Deityhub merged 1 commit into
devfrom
chore/robot

Conversation

@Deityhub

Copy link
Copy Markdown
Owner

robots.txt (apps/web/public/robots.txt)

  • Search engines are welcome. About 40 AI training crawlers are blocked: GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai, Diffbot and others (from the ai-robots-txt list, curated by hand).
  • Allowed on purpose: Googlebot, Bingbot, Applebot, link-preview bots (facebookexternalhit, Twitterbot, LinkedInBot) and AI search or assistant fetchers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User).
  • Sends Cloudflare's Content-Signal: search=yes, ai-input=yes, ai-train=no, and names https://ream.deityhub.com/sitemap.xml. The sitemap itself comes with the SEO pull request, so release the two together.

Only ream.deityhub.com is indexed

  • _headers sends X-Robots-Tag: noindex for https://:project.pages.dev/* and the preview hosts. Cloudflare's wrangler pages dev parses the file (5 valid rules).
  • scripts/nginx-conf.ts skips rules for a whole address, so the Docker image is unaffected.

Tests (apps/web/src/crawlers.test.ts): which crawlers are allowed and blocked, the sitemap line, the noindex rules, and the NGINX output. Docs in docs/deploy.md and AGENTS.md.

robots.txt is a request, not a lock. Cloudflare's free Block AI bots setting (Security, Bots) is what enforces it; documented in docs/deploy.md.

@github-actions

Copy link
Copy Markdown
Contributor

@Deityhub
Deityhub merged commit 4732acd into dev Oct 29, 2026
3 checks passed
@Deityhub
Deityhub deleted the chore/robot branch October 29, 2026 12:20
This was referenced Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant