docs: propose bounded-memory streaming telemetry

[0YHR0]

Description

Sampled streaming inference spans retain all response chunks even when message-content capture is disabled. Anthropic folding also repeatedly copies growing strings: a local 1 MiB text stream split into 8,192 deltas allocated about 4.03 GiB across recording and finalization with capture disabled (cumulative allocation, not peak heap).

Propose request-local streaming accumulators, bounded optional content capture, continued final-usage extraction after truncation, and a phased recorder migration. Include reproducible full-span and fold-only benchmarks and document the measured environment and limitations.

Related Issues/PRs (if applicable)

Independent telemetry proposal. It does not change Envoy buffering or implement #527.

Special notes for reviewers (if applicable)

No production behavior changes. Decisions requested: incremental recorder contract, content budget and truncation semantics, and rollout order.

Both benchmarks were executed on Go 1.26.6 at 1899d8d4, the commit immediately before the toolchain upgrade; the relevant tracing and folding sources are identical to proposal base 4b317d8a. Local Go 1.27.1 download failed, so new-toolchain verification remains for CI. Exact commands and allocation results are in the proposal.

Developed with assistance from OpenAI Codex.

[netlify]

✅ Deploy Preview for theagentrouter ready!

Name	Link
🔨 Latest commit	9eeea5f
🔍 Latest deploy log	https://app.netlify.com/projects/theagentrouter/deploys/6aa4b8e689ac8a00081f4551
😎 Deploy Preview	https://deploy-preview-2668--theagentrouter.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[0YHR0]

Hi @nacx, could you take a look at the recorder contract and rollout proposed here when you have time? The independently reviewable string-copy optimization in #2678 now has passing CI. The MCP topology design in #2669 also follows up on #2562; feedback on the opt-in and operation completion boundaries would help sequence implementation.

The formatting failures in both proposals have been corrected. This PR's remaining failed build job is the Kubernetes v1.32.8 upgrade test, which timed out waiting for Envoy Pods to become ready. I attempted to rerun the failed jobs, but GitHub requires repository admin rights. Could a maintainer rerun the failed jobs in run 34667780196? I have not changed upgrade behavior in this proposal.