From a562818d2abfd4dcdf18e55c85aaae3579b2fafa Mon Sep 17 00:00:00 2001
From: Bissbert <43237892+Bissbert@users.noreply.github.com>
Date: Mon, 4 May 2026 19:17:12 +0700
Subject: [PATCH] chore(seo): advertise llms.txt to AI crawlers
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Two small discovery hints so AI crawlers (GPTBot, ClaudeBot,
PerplexityBot, Google-Extended, Applebot-Extended) and research
bots find /llms.txt without having to guess the path:
- robots.txt: add LLM-Content + Sitemap lines pointing at /llms.txt
- BaseLayout.astro: add in the head of every page
There is no central submission endpoint for llms.txt — discovery is
purely convention-based, so making the file linkable from both the
crawl-control file and every HTML page is the maximum we can do
ourselves. IndexNow already handles the search-engine side.
---
public/robots.txt | 4 ++++
src/layouts/BaseLayout.astro | 3 +++
2 files changed, 7 insertions(+)
diff --git a/public/robots.txt b/public/robots.txt
index a859c94..367b518 100644
--- a/public/robots.txt
+++ b/public/robots.txt
@@ -7,6 +7,10 @@ Allow: /
# Sitemap location
Sitemap: https://gemmology.dev/sitemap-index.xml
+# LLM crawler index (https://llmstxt.org/)
+LLM-Content: https://gemmology.dev/llms.txt
+Sitemap: https://gemmology.dev/llms.txt
+
# Disallow API endpoints (served by separate worker)
Disallow: /api/
diff --git a/src/layouts/BaseLayout.astro b/src/layouts/BaseLayout.astro
index 8bde178..c86ad2c 100644
--- a/src/layouts/BaseLayout.astro
+++ b/src/layouts/BaseLayout.astro
@@ -49,6 +49,9 @@ const fullTitle = isHome
+
+
+