perf(retrieval): optimize serving-index reads without changing quality

[suguanYang]

Summary

• We optimize the shared v1/v2 retrieval serving-index path while preserving the existing BM25 formulas, Planner, Harvest, Control, result ordering, citations, and filtered-scope semantics.
• We persist per-channel BM25 corpus statistics, use token-selective covering-index reads, and reuse request-pinned revisions to avoid redundant scope and metadata scans.
• We add statistics-only maintenance and strict readiness checks for existing data. Missing manifests, missing/legacy indexes, and incomplete statistics fail readiness; mathematically valid zero IDF remains diagnostic.
• We add additive migrations for the covering index/statistics and a follow-up repair migration for missing or invalid concurrent-index artifacts.
• We document the production rollout, quality gate, monitoring, pause/resume, and rollback procedure. This is a follow-up to perf(retrieval): optimize map-nav snapshot and scoring #379.

Verification

• make check
• uv run pytest tests/contract/test_backfill_map_unit_indexes_contract.py tests/contract/test_retrieval_map_unit_index_contract.py tests/migrations/test_schema_contract.py -q — 28 passed
• Applied alembic upgrade heads to the restored production-shaped PostgreSQL copy; verified head c2d3e4f5a6b7.
• Statistics readiness: would_update=0 complete=2907 skipped=0 documents=2907.
• Serving readiness: 234/234 namespaces READY.
• Ten repeated classic calls across the frozen three-query set preserved chunk IDs, ordering, sources, and evidence. Maximum score delta was 1.7e-5, below the 1e-4 gate.
• Equivalent v1/v2 classic requests returned the same chunk IDs and evidence hash.
• Complete map-nav smoke finished successfully with valid cited evidence; explicit use_agentic=true and omitted use_agentic both route through map-nav.
• On the restored copy, token-selective projection reduced transferred rows from 42,794 to 13,573 and bytes from 9.53 MB to 2.93 MB. The warm index stage fell from roughly 0.16–0.8 s to 0.02–0.04 s, and revision-pin reuse removes a redundant query previously measured at roughly 0.35–0.6 s.
• We do not claim a broad end-to-end latency improvement until production p50/p95 observation is complete.

Deployment Notes

• No new runtime feature flag or environment variable is introduced.
• Run migrations before deploying the application. Expected Alembic head: c2d3e4f5a6b7.
• DevOps must follow docs/design/retrieval-serving-index-rollout-runbook.md using one one-off maintenance task at a time.
• Existing format-v2 indexes receive statistics through backfill_map_unit_statistics.py; missing or legacy indexes use the full per-document index backfill.
• The optimized reader is selected only for coherent revisions. Incomplete data keeps the exact legacy reader.
• Application rollback does not require schema downgrade; the additive index, columns, and completed statistics can remain.

Checklist

• Tests were added or updated when behavior changed
• Public docs, examples, or OpenAPI contracts were updated when needed
• Database migrations are idempotent and safe to deploy
• Logs, errors, and validation paths avoid leaking secrets or user data
• The pull request description explains any breaking or user-visible change

No public API contract or retrieval-quality change is intended.