Add robots.txt, llms.txt and full sitemap for AI/search discoverability

[embarquech]

The published site had no robots.txt (404), no llms.txt (404), and the
sphinx-sitemap output covers only the root project's 3 pages out of 16 URLs.

This adds three files to the published site root via html_extra_path:

• robots.txt — explicit allow-all + pointers to both sitemaps
• llms.txt — curated index of every doc section and PDF for LLM crawlers
• sitemap-docs.xml — static full sitemap (16 URLs incl. SDK subsites and PDFs), complementing the generated /sitemap.xml

Plus one line in docs/source/conf.py:

html_extra_path = ['_extra']  # robots.txt, llms.txt, sitemap-docs.xml copied to site root

Without that line Sphinx never copies the files and nothing publishes. The Deploy Sphinx to GitHub Pages workflow runs a plain sphinx-build docs/source _site, so the three files land at the site root on merge.

Verify after the Pages deploy finishes (~2 min after merge)

curl -s https://docs.cryptnox.com/robots.txt          # → 5 lines, 2 Sitemap: entries
curl -s https://docs.cryptnox.com/llms.txt | head     # → "# Cryptnox Product Documentation"
curl -s https://docs.cryptnox.com/sitemap-docs.xml | grep -c "<loc>"   # → 16

Follow-ups (separate PRs)

• html_last_updated_fmt = '%Y-%m-%d' in conf.py — visible "last updated" dates
• sphinx-sitemap config per SDK sub-project so the full sitemap maintains itself (sitemap-docs.xml is hand-written)
• Register docs.cryptnox.com in Google Search Console + submit both sitemaps

🤖 Generated with Claude Code