perf(pipeline): stop retaining a parsed DOM per page in SiteProjection

[phil-scott-78]

The problem

RenderedPage carried an AngleSharp IElement for every page in the corpus, and the projection held it for the life of the build. An IElement keeps its whole owning document reachable through .Owner — so this retained each page's entire parsed DOM, chrome and navigation rail included, not just the selector-matched content the consumers asked for.

Measured on a real generated page, holding 200 of them:

source html: 38.2 KB
retained per page: 0.51 MB  (13.8x the source HTML)

At docs-site scale that's invisible. On a large site it isn't. A 22k-page corpus drove private memory past 26 GB at a quarter of the crawl (free RAM hit 0 on a 64 GB box) and died with OutOfMemoryException. Because OutputGenerationService.FetchPagesAsync catches everything into FetchOutcome.Error, it surfaces as a scattering of per-page failures rather than the systemic problem it is — which is how it went unnoticed. With DOTNET_GCHeapHardLimit=8GB it OOMs at the same point instead of collecting harder, confirming a genuine live set rather than GC greed.

Why this is safe

Only one consumer ever read the DOM:

Consumer	Uses	Needed the IElement?
SearchArtifactService	Sections + guard	No — HeadingSection is pure value data
BookComposer	page.Html	No — already re-parsed it
BookArtifactService	guard only	No
LinkAuditor (via AuditRunner)	page.Html	No
LlmsTxtService	Convert(page.Content, …)	Yes — the only one

BookComposer already did the right thing, with a comment saying why. The pattern just wasn't applied where it mattered.

The change

• RenderedPage — bool HasContent and an eager IReadOnlyList<HeadingSection> replace IElement? Content and Lazy<…> Sections.
• SiteProjection.RenderOneAsync — extracts sections while the element is still live, then returns; the owning document is unreachable immediately after. The eagerness is load-bearing, not an oversight, and is commented as such. RenderedHtmlFetcher's signature is untouched.
• LlmsTxtService — re-parses page.Html per iteration via an HtmlParser on BuildContext. The document is collectable straight after conversion, so peak cost tracks concurrency rather than corpus size.
• Guards in Search/Book move to HasContent.

Verification

examples/ScaleStressExample (5,000 pages, dotnet run -- build), A/B across the same commit:

	peak private memory
before	1,638 MB
after	565 MB

2.9x less, and the output is byte-identical — 30,224 files, diff -rq clean.

dotnet build clean; full suite green (1,243 passed, 4 skipped — the gated Chromium PDF test).

The 2.9x understates what content-heavy sites get: ScaleStress pages are small generated markdown, and the ratio scales with DOM-to-HTML size. On a real API-reference page with a large chrome sidebar it measured ~10x.

Breaking change

RenderedPage.Content is removed from the public API. Consumers matching on it should test HasContent and re-parse Html where they genuinely need a DOM. Nothing in this repo outside the six changed files was reading it — the compiler found every call site, and only two test files needed updating.

Note on LinkAuditor

An earlier draft of this description claimed the build "re-renders the corpus a second time" via LinkAuditor. That was wrong — BuildHtmlCache collapses repeat self-fetches, so the audit pass replays cached renders rather than re-rendering. Measured on the same 5,000-page corpus by removing the IRenderedAuditor registration:

	peak	wall
with LinkAuditor	556 MB	34.5s
without	518 MB	35.1s

~38 MB and no measurable wall-time. No opt-out is warranted; the auditor earns its keep.

Its cost does scale with links-per-page, and that is governed by SiteProjectionOptions.ContentSelector — LinkAuditor reads page.Html, which is selector-scoped. A host that leaves the selector unset audits its own chrome on every page. On one such site that measured 40 links/page instead of 6 (0.9M vs 0.1M checks corpus-wide), which is a host misconfiguration rather than a framework defect.

Formatting

Trailing-newline and whitespace churn in the diff is the repo's Rider format-on-save; the substantive change is ~30 lines.

[github-actions]

🛰️ Docs preview: https://pr-69.pennington-dev.pages.dev

Rebuilt on every push to this PR; torn down when it closes.