Fetch llms.txt and AGENTS.md weekly, and put their changes in changelogs

[lbesecker195]

Closes #75. Follows #74.

Polling, not webhooks

A GitHub webhook requires admin rights on the repository, or a GitHub App its owner installs. This registry has neither for almost any of its tens of thousands of repositories, and llms.txt is served by websites that have no webhooks at all. So each document is checked weekly with conditional requests (If-None-Match, If-Modified-Since) — an unchanged file answers 304 with no body. Verified against raw.githubusercontent.com before building on it.

What's fetched

Document	Where
llms.txt	root of the listing's website, else the domain its namespace was verified for (com.cloudflare → cloudflare.com). io.github.* names an account, not a site, so it gets none.
AGENTS.md	raw.githubusercontent.com/<owner>/<repo>/HEAD/AGENTS.md

Documents are keyed by URL, not listing: many listings share one llms.txt, and it is fetched once, recorded once, and shown on every listing that points at it. A server_documents link table, rebuilt daily, says which listing points where, so a listing's changelog, its kind counts and the sitemap all pick document changes up in SQL.

Fetching URLs publishers choose

website_url is whatever a publisher typed, so every fetch is treated as hostile until checked:

• https only, default port
• The host must resolve to a public address — otherwise a listing pointing at 169.254.169.254 would have this server read cloud metadata for it. Redirects are followed by hand so every hop gets the same check, not only the first. Covers RFC 1918, loopback, link-local, CGNAT, benchmarking, multicast, IPv6 ULA/link-local, and IPv4-mapped IPv6.
• Bodies stream with a 256 KB cap — never downloaded whole and truncated after
• HTML counts as absent — many sites answer any path with their SPA shell and a 200

What's recorded

First fetch: baseline, nothing recorded. A changed body: lines added and removed (List.myers_difference, blank lines dropped, capped at 200 lines). A file appearing where one was confirmed absent, or vanishing where one was present: recorded as such. An outage — timeout, 5xx, a bot filter's 403 — records nothing and keeps what was known. An outage is not a removal.

On the page, document changes render as a diff block with the file's URL rather than as chips, since they're lines of prose.

Running it

Documents.Scheduler: a batch of 100 every five minutes at concurrency 4, rediscovery daily. On in production unless DOCUMENTS_ENABLED=false; never under test.

Tests: 226 passing on three seeds, including a private resolver, a redirect into a private range, plain http, the size cap, the SPA fallback and an outage.

---

Pages affected:

• MCP server changelog -- registry-wide changes, now with llms.txt and AGENTS.md diffs.
• MCP Harbor -- registry home and server search.

🤖 Generated with Claude Code