Make the site agent-ready for AI/LLM crawlers - #2
Merged
Conversation
Add static, GitHub Pages-compatible signals for AI agents and LLM crawlers, following jlhernando.com and joost.blog: - public/robots.txt: explicit Allow + Content-Signal (ai-train/search/ ai-input=yes) for named AI agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, ...) plus a Sitemap reference. - /llms.txt and /llms-full.txt generated at build time. - Per-post Markdown (…/index.md) via page.rawInput, advertised with <link rel=alternate type=text/markdown>. - JSON-LD: BlogPosting on posts, WebSite + Person elsewhere. - Meta: page-aware <title>, canonical, Open Graph, Twitter Card, and a sitewide rel=describedby link to /llms.txt. - metadata.js: default OG image, twitter handle, sameAs social list. Also fix a stray control char (\003) in the newest post's filename that produced a broken %03 URL.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Makes
yonatanlou.github.iodiscoverable and consumable by AI agents and LLM crawlers (ChatGPT, Claude, Perplexity, Gemini, …), based on the practices in jlhernando.com and joost.blog.Constraint: the site is a static Eleventy build on GitHub Pages — no custom HTTP headers, no request-time content negotiation. So everything here is static-output only. Techniques needing an edge (HTTP
Linkheaders,Accept: text/markdownnegotiation, MCP/WebMCP) are intentionally out of scope. Full rationale indocs/superpowers/specs/2026-07-15-agent-ready-design.md.Changes
public/robots.txt— explicitAllowfor named AI agents (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, cohere-ai, Bytespider),Content-Signal: ai-train=yes, search=yes, ai-input=yesper block, and aSitemap:line./llms.txt+/llms-full.txt— build-generated index and full raw-Markdown corpus of all posts.…/index.md(page.rawInput), advertised via<link rel="alternate" type="text/markdown">.BlogPostingon posts;WebSite+Person(withsameAs/jobTitle/worksFor/alumniOf) elsewhere.<title>,author, canonical, Open Graph, Twitter Card, sitewiderel="describedby"→/llms.txt._data/metadata.js— default OG image,twitterhandle,sameAslist reused by JSON-LD.\003) in its filename, producing a broken%03URL on the live site. Renamed.Verification
npx @11ty/eleventybuild passes./robots.txt,/llms.txt,/llms-full.txt, and 7 per-post.mdfiles verified.rel="alternate" type="text/markdown"present on posts, absent on non-posts.