[GEO][P1] Explicit AI crawler policies in robots.txt #104

Problem

robots.txt (next-sitemap) only has a wildcard rule. Explicit, named rules for AI search/answer crawlers document intent and avoid accidental blocking by future wildcard changes.

Proposal

Add explicit allow policies (with /admin disallowed) for OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bingbot. Keep the news sitemap listed and mention llms.txt in a comment.

Acceptance

• Generated robots.txt contains the named policies; config test covers it.