perf(archive): prove current full-corpus ingest and rebuild bounds

[Sinity]

Outcome

Current Polylogue master can ingest, replay, repair, and rebuild the real archive within explicit time, memory, WAL, cancellation, restart, and time-to-searchable bounds. The result is a reproducible benchmark/receipt, not another optimization based on stale measurements.

Why this issue remains open

The original June incident exposed real pathologies, including slow full-ingest chunks, multi-gigabyte WAL growth, retained RSS after interruption, and weak stage visibility. Since then, raw replay, authority reconciliation, planner statistics, derived rebuilds, FTS finalization, checkpointing, and repair paths have changed substantially.

Therefore:

Historical timings are regression evidence, not a current performance diagnosis. Remeasure current code before admitting another fix.

Benchmark matrix

Run against a declared immutable archive snapshot and record:

• hardware, code commit, schema versions, archive row/byte counts, and enabled optional lanes;
• clean full replay from retained raw authority;
• resumable restart after interruption at representative stages;
• live catch-up of changed large provider files;
• derived insight/thread/run/delegation rebuild;
• FTS/trigram/vector finalization where enabled;
• time to first searchable page and final readiness;
• peak RSS, WAL, DB growth, CPU, open files, and stage durations;
• cancellation/deadline behavior and retained resume state;
• parity and debt before/after.

Acceptance criteria

• One command or documented command sequence produces a machine-readable benchmark receipt and preserves logs/artifacts.
• Full replay and derived rebuild commit/checkpoint in bounded resumable windows; interruption does not discard all prior completed work.
• Work units are bounded by meaningful size/row/message budgets rather than only fixed file/session counts.
• Restart resumes from durable progress and converges to the same parity result as an uninterrupted run.
• Peak RSS and WAL stay inside declared operator limits, or a precise stage is named as the remaining blocker.
• Live append/catch-up does not repeatedly reparse an entire giant file when a valid bounded continuation exists.
• Time-to-searchable and final-readiness are reported separately.
• The receipt distinguishes exact parity from deferred optional work and remaining debt.
• No performance change lands solely because the historical June profile suggested it; each fix cites a reproduced current bottleneck.

Tracker authority

• GitHub: public real-scale performance envelope and closure claim.
• Bead: polylogue-20d.6 (tracker:gh-mirror).
• Related public issue: fix(ops): prove the deployed archive is coherent after rebuild and repair #2308 owns aggregate deployed archive trust; this issue supplies performance/restart evidence but does not replace its authority/reconciliation gate.
• Related public issue: feat(query): make every archive read bounded, cancellable, and resumable #3283 owns bounded/cancellable/resumable read transactions, not ingest/rebuild throughput.

Non-goals

• No generic micro-optimization campaign.
• No benchmark against only synthetic fixtures.
• No requirement that every optional derived lane block initial queryability.
• No weakening parity, evidence authority, or restart safety to improve headline throughput.

[Sinity]

Live append latency progress from the 2026-06-25 deployment pass:

• PR perf: avoid full-session active-leaf churn #2394 narrowed active-leaf clearing so append no longer rewrites every message's is_active_leaf flag.
• PR perf: expose append ingest stage timings #2395 added append stage timings.
• PR perf: split append storage timing #2396 split raw source writes from parsed index writes. Deployed evidence showed source writes were only about 15-17 ms; append.index_parsed_write was still about 5.45-5.50 s.
• PR perf: expose deep append index timings #2397 split the parsed index writer. Deployed evidence showed append.index.attachments consumed about 5.3 s even for active Codex tail appends with no new attachments.
• PR perf: scope append attachment refcounts #2398 scoped attachment refcount refresh for merge-append writes. Full-session replacements keep the global refcount refresh; appends refresh only touched attachment IDs, and no-attachment appends skip it.

Post-PR #2398 deployed evidence on polylogue 0.1.0+af63ee9:

parse_s=0.146 convergence_s=0.092
stages=append.raw_and_index_write:0.136,append.index_parsed_write:0.123,append.index.session_counts:0.040,append.index.provider_usage_rollup:0.032

parse_s=0.164 convergence_s=0.097
stages=append.raw_and_index_write:0.147,append.index_parsed_write:0.131,append.index.session_counts:0.039,append.index.provider_usage_rollup:0.031

Read-only diagnostics after deployment: report ok, no running attempts, no stale cursor writes, no overlapping source paths, FTS triggers all present, daemon RSS about 140 MB. Current residual visible cost is small and concentrated in per-session count refresh, provider usage rollup, and normal convergence.

Ref #2391