fix(fts): index-time prefix ladder + drop query-time :* (OPS-2314)

[arreyder]

Warning

DEPLOYMENT ORDERING IS LOAD-BEARING — DO NOT MERGE-AND-SHIP AS ONE STEP.
The tsvector is generated at row-write time by this generated code and stored in the row. Existing rows keep their old (:*-dependent) vectors until they are re-marshaled. The index-side prefix ladder must ship AND all FTS tsvector columns must be backfilled / re-indexed BEFORE the query-side :* removal reaches prod. Removing :* against un-backfilled rows silently drops trailing-partial recall (typeahead stops matching) with no error.

Recommended: split this into two mergeable PRs.

1. Index-side first — land the normalizeVectorDocs prefix ladder, release it, then backfill / re-index every FTS tsvector column.
2. Query-side after backfill — only then land the buildSearchQuery :* removal.

This draft is the combined proof that the two-sided change is correct and green end-to-end. It is intentionally NOT for merge as-is.

Summary

Fixes the FTS trailing-term prefix explosion (OPS-2314) by moving prefix expansion from query time to index time:

• Index side (normalizeVectorDocs, pgdb/v1/fts.go): emit the prefix ladder of each bare sub-token (securitygroup → s, se, sec, …), weighted lowerWeight(doc.Weight) to match existing gram weighting.
• Query side (buildSearchQuery, pgdb/v1/fts.go): drop || ':*' from both the non-stemmed and stemmed branches. The trailing term is now an exact single-posting-list lexeme lookup.
• New test (TestSearchTrailingTermNoPrefixExplosion): asserts the emitted tsquery contains no :*.

Root cause

• protoc-gen-pgdb fix: prefix match last search term for multi-dot queries #139 introduced the query-time to_tsquery('simple', ? || ':*') prefix match on the trailing search term.
• :* forces GIN to union the posting lists of every lexeme sharing the prefix (prod → prod, production, product, prod-xyz, …). For a common prefix this is an unbounded union → GBs scanned.
• Observed ~53s → 102ms after the fix on prod ClickHouse data.

Why the naive fix does NOT work

The obvious one-liner — just drop :* from the query — was tested against real Postgres and REGRESSES IGA-1440. The premise that the index already stores a bare per-word prefix ladder is false: the index emits full sub-token lexemes + 3-char sliding-window grams + compound-prefix ladders, but not the bare s, se, sec, … ladder. So with :* removed and no index change, TestSearchMultiDotPrefix fails on exactly the partial-trailing cases:

ad.ph02t1.admin.se   should match ad.ph02t1.admin.securitygroup  -> false
bigquery.data        should match bigquery.dataviewer            -> false
bigquery.dataV       should match bigquery.dataviewer            -> false

The two-sided change (this PR) makes the bare ladder exist at index time, so exact-match recall is preserved and the posting-list union is eliminated.

Caveats (from the validated prototype writeup)

1. tsvector size / write amplification. Prefix ladders on every bare sub-token grow the tsvector (additive with the existing compound-prefix ladder; some overlap). The 1 MB tsvectorMaxMegabytes cap truncates lexemes when exceeded, which could drop tail lexemes on very long values. Measure vector-size delta on representative large tenants before shipping the backfill; consider capping ladder length if bloat is significant. Exact-match query cost is unaffected by ladder length.
2. ENGLISH_LONG / EXACT behavior change. These types intentionally get no grams. buildSearchQuery is field-type-agnostic, so after :* removal a trailing partial on an ENGLISH_LONG / EXACT field matches only full words (no partial typeahead). Confirm no product surface relies on trailing-prefix typeahead over long-text / exact fields.
3. Byte-vs-rune slicing. The ladder uses v.value[0:i] byte slicing (matching the existing symbolsFullTokensSplitDoc prefix block). On multibyte sub-tokens this can split a rune (harmless byte-fragment lexemes; fuzz seeds pass, Postgres tolerates). Sub-tokens are lowercased identifier-ish text so risk is low; a rune-aware slice is cheap hardening for the real PR.

Test results

go test ./pgdb/v1/... -run 'FTS|Search|MultiDot' -v
--- PASS: TestSearchBigQueryDoc
--- PASS: TestSearchSymbols (+ all subtests)
--- PASS: TestSearchEmpty
--- PASS: TestSearchCamelCase
--- PASS: TestSearchSnakeCase
--- PASS: TestSearchPathsFull
--- PASS: TestSearchMultiDotPrefix                  <-- IGA-1440 GREEN with :* removed
--- PASS: TestSearchTrailingTermNoPrefixExplosion   <-- new: asserts no :* in emitted SQL
--- PASS: FuzzFullTextSearchQuery / FuzzFullTextSearchVectors (seeds)
ok  github.com/ductone/protoc-gen-pgdb/pgdb/v1   4.693s

Links

• OPS-2314 (this fix)
• protoc-gen-pgdb fix: prefix match last search term for multi-dot queries #139 — introduced the query-time :*
• IGA-1440 — trailing-partial multi-dot recall that must be preserved

🤖 Generated with Claude Code

[linear-code]

OPS-2314