index: back off failed manifest probes, read /agents.json, surface the discovery gap

[MikeyPetrillo]

What

Our crawler re-probed /.well-known/x402 on a fixed 5-minute interval regardless of outcome. An origin that had 404'd hundreds of times in a row was asked again exactly as often as a healthy one. One origin took 686 identical requests in a week, answered 404 to every one, and served a complete 17-endpoint catalogue at /agents.json that we never asked for.

Both halves are our defect: we spent their bandwidth to re-learn a fact we already had, and we listed them thinly for it.

Changes

Backoff on the manifest probe. Three consecutive failures are free (a transient blip must not cost a seller listing freshness), then 30m / 2h / 6h, capped. Any success clears it immediately, so fixing a manifest is picked up on the next crawl rather than punished. Scoped to the probe, not the origin: gating the whole origin would cost us the catalogue we can still reach in order to save them one request.

/agents.json and /llms.txt join the fallback chain. agents.json sits behind the same payment gate as openapi.json (the registry proves the origin settles, or the document itself carries a payment signal). The spec naming one path does not make the wild uniform; an index that reads only one path indexes only the sellers who read the same page we did.

llms.txt is the riskier of the two, because it is prose rather than structure. A greedy markdown scrape would turn marketing copy into phantom "tools" nobody can buy, and a fabricated listing is worse than a thin one. So the parser reads exactly one shape, the priced link-list entry, and requires a same-origin absolute URL, an explicit price on the line, and a route that survives the same non-tool path filter openapi uses. A bare URL in a sentence is not read at all.

The gap is now visible to the seller. discoveryPath is recorded per crawl and rendered on the seller card as one line: which surface answered, and which one spec-following buyers will not find. source cannot express this, it collapses both fallbacks to openapi-fallback, which would send a seller to fix the wrong file. Null for sellers already on the spec path, so it stays a signal rather than decoration.

The manifest's own catalogue is now read. Chasing the seller-facing note exposed a larger gap: we parsed /.well-known/x402 for identity and payment and threw its tool list away, so a seller enumerating every endpoint in their manifest could still be listed thinly from a third-party registry row.

Measured, not assumed: of 44 reachable manifest-sourced sellers sampled, 5 advertise more entries than we list. Live parser results against those manifests: 16 to 17, 1 to 14, 11 to 12, 3 to 5.

Four dialects handled, all taken from real manifests: {name, endpoint, price_usd, summary} objects, bare same-origin URLs, "POST /path" strings, and {resource|url|route|path} objects. Same-origin only is load-bearing: some manifests list other origins, and attributing those to the publisher would put another seller's tools under this seller's payTo and double-count one catalogue across two hosts. Two sampled origins parse to zero for that reason, which is correct for both.

Dedupe keys on method + route including the query string. Two entries differing only by ?product= are two products; collapsing them to the pathname is how a 17-tool seller read as 16.

Precedence is openapi > manifest > registry.

Verification

• scripts/test-crawl-backoff.js (11) and scripts/test-discovery-note.js (35), both wired into CI
• The llms.txt assertions are weighted toward refusals rather than acceptances: cross-origin link, unpriced link, the discovery path itself, a static asset, a duplicate route, a bare URL in prose, and empty/null/unparseable-origin inputs
• Checked against a real document, not only fixtures: 629 entries read from our own llms.txt, zero junk routes
• Every assertion mutation-tested: removing the spec-path exemption, collapsing the two fallback notes, silencing the registry-only case, dropping clear-on-success, removing each of the three llms.txt guards, and removing each of the four manifest-parser guards all kill assertions
• Full sweeps clean: 210 strict / 296 lenient tools, 526/526 through MCP, plus the 24-script offline battery
• Parser checked against the live manifests of all six sampled sellers, not only fixtures

Closes #645