[A13xB0]

From upstream

This PR brings in upstream Kpa-clawbot/CoreScope#2153 by @n30nex. It is open upstream at the time of writing and addresses upstream issue Kpa-clawbot#2151. Its own two commits are cherry-picked unchanged with -x:

• 1a60606d perf(ingestor): yield between committed retention batches (perf: let live ingestion run between retention batches Kpa-clawbot/CoreScope#2151) (upstream b70a9f6b)
• a86e6861 docs(perf): report paired retention gains and service-time tradeoffs (upstream 323b18d4)

Upstream's PR also carries test: respect native permissions and fixture release context, the same change already brought in by the Kpa-clawbot#2149 PR on this fork; it is left out here so the two PRs do not apply it twice. Nothing in this PR was written for the fork.

What it changes

PruneOldPackets deletes old transmissions in bounded batches under the writer lock. Between committed batches it now calls runtime.Gosched(), after the commit and the writer-lock release, so the serial ingest consumer gets CPU to finish a packet and submit its next write before retention competes for the next batch. It does not bound a single dense batch (one transaction with very many observations). Tests use an injected hook instead of the global scheduler.

Performance

Upstream's measurement

Five alternating pairs on an Actions host, 128k transmissions / 2.05M observations, 50 ingest events/s and 20 HTTP requests/s, retention starting mid-window (full report in docs/performance/sqlite-retention-b5-2026-10-09.md, included). By median paired ratio: additional-observation ingest p95 -5.0% and p99 -8.1%; per-event service time p99 +29.1%; cleanup +0.5%. Upstream describes it as a narrow scheduling benefit with a service-time trade-off, not a general speed-up.

Our run of upstream's benchmark

BenchmarkRetentionConcurrentIngest (upstream's own, from this PR) uses only the public PruneOldPackets, so it runs unchanged against master too. 10 runs each; each run reports a JSON profile; medians:

fixture	metric	master	this PR	change
typical16 (16k tx, 256k obs)	ingest completion p95	18.18 ms	7.45 ms	-59.0%
	queue max	3.47 ms	1.06 ms	-69.5%
	retention time	708.3 ms	701.5 ms	-1.0%
dense256 (500 tx, 128k obs)	ingest completion p95	451.2 ms	274.4 ms	-39.2%
	queue p95	431.2 ms	254.4 ms	-41.0%
	retention time	710.4 ms	706.0 ms	-0.6%
singleHot (1 tx, 64k obs)	ingest completion p95	1.14 ms	1.15 ms	+0.6%
	retention time	128.1 ms	129.0 ms	+0.7%

On this machine the effect is larger than upstream's CI measurement and retention itself is not slower. singleHot is unchanged, as upstream says: one huge transaction has no batch boundary to yield at.

Not measured on pre-prod: retention runs once a day there, and the pre-prod host is in a 24-hour soak; upstream's paired run and the benchmark above are the evidence.

Tests

From upstream: retention_fanout_test.go (yields after committed full batches; no yield for disabled, empty or failed work; late fresh arrivals between batches are kept; skewed children and empty parents keep fresh evidence; a failure rolls back only the current batch), and two extra plan assertions in prune_chunked_test.go (child work must seek by transmission key, never scan observations).

All ingestor retention and prune tests pass; make test: 16 of 16 modules pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KaR4xu8gVykQBkH8yG8m6H