fix: pool self-fetch connections; upgrade DeweySearch to 0.2.0

[phil-scott-78]

Two independent changes that surfaced while profiling a 22,178-page site (cakebuild.net) whose search index took ~3 minutes to warm up on every dev start.

1. Self-fetch connections were never pooled

HttpDispatcher.CreateClient() built a fresh HttpClientHandler on every call, and both callers — RenderedHtmlFetcher and OutputGenerationService — wrap the result in using. Nothing could pool: every self-fetch opened a new loopback socket that then sat in TIME_WAIT for four minutes.

Once the corpus exceeds the ~16k Windows ephemeral port range, the range is exhausted partway through the crawl and every remaining fetch fails with SocketError.AddressAlreadyInUse. SiteProjection.RenderOneAsync treats a failed fetch as a per-page content error, so those pages vanish from the search index and llms.txt behind a warning.

Measured on the 22,178-page site: 6,641 routes silently missing — about 30% of the corpus. The projection's own debug line read 22178 refreshed, which is the attempted count computed before rendering, so nothing in the logs suggested a problem.

Dev-serve only. Build and diag swap Kestrel for TestServer (PenningtonExtensions.cs:87), which dispatches in-memory with no sockets — static output was never affected.

Fix

Build the handler chain once and share it, handing out HttpClient(handler, disposeHandler: false) wrappers so a caller's using releases the wrapper and leaves the pool intact. MaxConnectionsPerServer is bounded at ProcessorCount * 2. The base address is still resolved per call, since on the Kestrel path it isn't known until the server binds.

New regression test drives 200 distinct paths through a real Kestrel host and counts client ports server-side — distinct on purpose, since CachingHttpHandler replays per path and repeating one URL would pass vacuously:

	distinct client ports
before	200 — FAIL
after	≤32 — PASS

After the fix the same 22,178-page crawl completes with zero socket failures and a complete index.

2. DeweySearch 0.2.0

The C# API is unchanged (IndexBuilder.Build, SearchIndex.ToFiles, SearchDocument), so no library source changes. What moves is the emitted wire format: the document table leaves index.json for cold d-{n}.json shards, leaving the entrypoint carrying only what ranking needs before a result is displayed.

Four integration tests asserted on index.json's docs array and broke. Rather than repeat the decode four times, SearchIndexReader loads the manifest plus doc shards and yields decoded IndexedDoc(Id, Url, Title, Crumbs, Priority) rows. Urls and titles are front-coded against the previous row (leading char is the shared-prefix length as c - '0'); breadcrumbs are indices into the manifest's label dictionary. It deserializes into DeweySearch's own IndexManifest/DocShard types so the field-name contract stays the package's to define, and it's now the only place in the suite that knows the layout.

Entrypoint vs. cold table on this docs site: 2,695 records ship a 43 KB index.json with the 183 KB document table held back until a query produces hits.

The search explanation page described the document table as living in index.json and is updated to match.

Verification

• Full suite: 1,244 passed, 0 failed
• Docs site builds clean: 657 pages, beck diagram renders without error
• 22,178-page crawl: 0 socket failures, complete index

Follow-ups (not in this PR)

• DocCodec has no public front-code decoder — it exports FrontCode but no inverse, so every consumer reading an index back has to reimplement it. Worth exposing on the DeweySearch side; SearchIndexReader could then drop its decode.
• RenderOneAsync swallows infrastructure failures as per-page content errors. 6,641 pages failing from one cause is indistinguishable from 6,641 individually broken pages — the same class of problem SelfFetchUnavailableException already solves for the not-started case. A failure-rate threshold that fails the projection loudly would have caught this immediately.
• Chrome rendered during corpus self-fetches is largely wasted. On the same site, hosts render full page chrome for all 22k self-fetches and SiteProjectionOptions.ContentSelector then discards it: 202 KB rendered and parsed per page to keep 8.7 KB. Guarding the host layout with CorpusFetchScope.InsideCorpusFetch took the crawl from 169.0s to 8.6s (19.6x) with a byte-identical index. That's a host-side change today; worth considering whether Pennington should make it first-class.

[github-actions]

🛰️ Docs preview: https://pr-70.pennington-dev.pages.dev

Rebuilt on every push to this PR; torn down when it closes.