fix(search): preserve full-document trigram storage

[seonghobae]

Descendant integration update (2026-09-05)

The search owner head remains cd8ff413d4ed8a5f2855c47a21a31db5661cd487. Its prerequisite has now propagated by ordinary merges through #1468 53ce38ed6683d01a9d113069f5ac5a8f17e133a2 (131 strict tests), #1427 cb08b1c3ea2aba8844fc29ef703c34368cc55e47 (132), and #1497 69f50ae684f50c501ff2f49be2969f1d211d7f3c (345, zero failed/skipped). The previously unchanged 8 MiB-class archive regression now passes on the combined actual migration graph, alongside retained-identity rollback and forced incompatible-writer cases. Each lane has its own exact source/tree receipt; descendant passes do not substitute for this owner's required Checks or approval.

A separate read-only Agent review of this exact owner head found no blocking source/rollback/documentation findings and ran no tests or GitHub approvals. GIN distance-order acceleration is still absent; representative search p95 and migration lock/build/storage costs, hosted Checks, independent approval, and release gates remain open. Keep Draft. The original receipt below describes the owner experiment and the earlier descendant failure as historical evidence.

---

Customer failure and scope

A valid long email, attachment, cited segment, or project document can fail to persist before restore/export/search starts. Real migrated PostgreSQL exposed this in the unchanged large-segment regression during the pending #1497 integration; ORM-only tables had hidden the migration-created index limit.

This Draft is the Naruon-owned search-schema prerequisite on #1503. Existing #1468 → #1427 → #1497 deltas stay open and will inherit it through non-force merges. No existing open PR owned the whole-document trigram repair after checking the current PR file inventory. No predecessor is closed or marked fully succeeded by this PR.

Exact source and decision

• Base fix: create workspace_entities/workspace_documents registry and provision Workspace rows #1503: 19d5860bc27e860acba940390f5792721cd99e5e (fix/workspace-document-registry-migration).
• Head: cd8ff413d4ed8a5f2855c47a21a31db5661cd487.
• Tree: 526b7c334a00020e5f3ac49d76d3fdd2b44a9665.
• Forward revision 0020_search_trigram_storage replaces the four complete normalized GiST expressions with GIN using structured Alembic operations. Published revision 0010 is unchanged; no document is shortened, hashed, omitted, copied, or rewritten.
• Ranking SQL, scores, limits, joins, and owner predicates are unchanged. RankWeave continues to own fusion and query normalization; no source or contract is copied from another service.
• Downgrade retains corrected indexes and records so valid newly stored content cannot make application rollback fail. The transactional rebuild needs measured lock/build/storage costs before deployment.

Performance remains an explicit acceptance gate: GIN does not accelerate the existing distance-only top-k ordering. This is a storage-correctness candidate, not equivalent-performance evidence or approval to ship slower search. Representative permitted data, query plans, end-to-end p95 ≤20 ms, simultaneous writers, and migration costs remain unverified. Do not add an arbitrary similarity threshold or shrink the measurement population to manufacture a pass.

RED → repair → GREEN

• PostgreSQL 16.15 isolated causal probe: 32,768-byte input, 4,097 distinct trigrams. GiST siglen=256 and 2024 both fail with SQLSTATE 54000 (12,304-byte index row, 8,191-byte maximum); GIN retains the identical full value.
• Four new real-migration surface regressions fail on the pre-repair head at the index-size boundary: 4 failed, 4 deselected. This is not a fixture/collection failure.
• Final repaired suite: 131 passed, 0 failed/skipped, -W error, 28.03 seconds, after exact uv sync --locked and fresh/repeated upgrade to 0020_search_trigram_storage.
• Tests verify historical and new records, full high-entropy inserts/updates, literal perfect far-tail word scores, actual valid GIN index presence, repeat upgrade, retained data and new large writes after downgrade, then re-upgrade.
• Existing search, PostgreSQL search, grounded answer, fusion, migration, bootstrap, scope, dependency, and Data API tests are included. Ruff and diff checks pass. Independent local Agent review reported no remaining findings; it is not a qualifying GitHub approval.
• At the time of the original owner experiment, the unchanged 8 MiB-class feat(data): add bounded tenant provenance portability #1497 archive regression had not yet been rerun with this prerequisite; the descendant update above now records that run. Its previous uncommitted integration result was 288 passed / 1 index-overflow failure. The new smaller causal test supplements, not replaces, that case.

Exact commands, image digest, scope limitations, cleanup, and local artifact hashes are in docs/doctoring/search_trigram_storage.md. Local evidence is not hosted Actions, signed-browser restore, protected merge, release, or deployment evidence. Test-only generated records are not production performance data. Existing customer resources and the main user checkout were untouched.

Decision traceability and follow-up

ADR-0020 is Proposed and records the customer failure, constraints, alternatives, rejection reasons, risks, and verification. All 160 open PR file lists were checked, including the second #1287 file page: 0008 is already used by #1418 and 0019 by #1419, so the never-committed local 0008 draft was renumbered. Historical ADR collisions elsewhere remain repair findings.

1. Collect current-head hosted checks and independent review without bypassing required gates.
2. Profile representative search and migration cost; repair any resulting performance findings at the canonical owner before acceptance/release.
3. Propagate through test(db): align PostgreSQL smoke fixtures with current schema #1468 → fix(data): align PDF DOM upload budget with sidecar #1427 → feat(data): add bounded tenant provenance portability #1497 without force, reconcile the uncommitted provenance merge revision, and rerun its unchanged large archive, retained-identity rollback, and incompatible concurrent writer cases on the actual combined migration graph.
4. Update the canonical Gap ledger in docs(architecture): repair Noema consumer boundary #1557 and recurrence guidance in the existing docs(agents): 증거 기반 작업 절차 정리 #1566 lane; do not create competing copies here.

Primary references (APA 7th)

PostgreSQL Global Development Group. (n.d.). pg_trgm—Support for similarity of text using trigram matching (PostgreSQL 16). Retrieved September 5, 2026, from https://www.postgresql.org/docs/16/pgtrgm.html

PostgreSQL Global Development Group. (n.d.). trgm_gist.c [Source code, commit ad6ffe6a1ffddf19603b13633f054f3d66ef4277]. https://github.com/postgres/postgres/blob/ad6ffe6a1ffddf19603b13633f054f3d66ef4277/contrib/pg_trgm/trgm_gist.c#L107-L143

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}