fts: guard against pathological short (:*) prefix terms

[arreyder]

Problem

buildSearchQuery (in pgdb/v1/fts.go, exported via FullTextSearchQuery) appends a :* prefix operator to the trailing token of a multi-term query for typeahead. When that trailing token is very short, this is pathological.

Confirmed in production via EXPLAIN ANALYZE:

• A 1-char trailing token (e.g. c:*) matches ~52% of a tenant's entitlements.
• The GIN prefix index scan does not short-circuit against the other &&-ANDed terms, forcing a ~64s GIN prefix scan.
• End to end the search went ~72s, blowing the 15s gRPC deadline (DEADLINE_EXCEEDED) — residual of ConductorOne inc-947.
• Dropping the too-short trailing token takes the same search from ~72s → ~1.1s.

Fix

Before emitting the multi-term :* expression, if the trimmed trailing term is shorter than minPrefixLen (utf8.RuneCountInString(lastTerm) < minPrefixLen), drop it and recurse on the remaining terms.

• minPrefixLen = minWordSize (= 3). Rationale is tied to the existing minWordSize = 3: tokens shorter than this are not independently indexed on the tsvector side either, so a <3-char trailing token cannot benefit from a real prefix match anyway — it only broadens the scan.
• Recursion always makes progress (one fewer term per call) and terminates at the single-term branch, which never emits :*.

Behavior delta

• Only multi-term queries whose trailing token is <3 runes change. That trailing token is dropped; the remaining terms (including prefix matching on the new trailing term, if ≥3) are searched.
• Normal typeahead (trailing token ≥3 chars) is byte-identical — same generated SQL.
• Single-term queries are unchanged (they never emitted :*).

Tests

Added TestBuildSearchQueryShortPrefix (no DB required — inspects the generated literal/args):

• multi-term with ≥3-char trailing token still emits token:*;
• 1- and 2-char trailing tokens do not emit a bare :* (drop to the remaining terms);
• dropping a short trailing token still keeps prefix matching on the now-last full-length term;
• all-short terms recurse to the single-term path without :* (also proves recursion terminates);
• single-term unchanged.

Full pgdb/v1 suite (DB-backed, via pgtest) passes; go build ./..., go vet ./pgdb/v1, and gofmt are clean.

Rollout

Consumers pick this up via a version bump (go.mod tag bump + re-vendor) — no runtime config.

🤖 Generated with Claude Code

[arreyder]

Follow-up items from code review (deferred, non-blocking)

A multi-perspective review (bugs / test / perf) of this PR caught one blocking gap that's now fixed in d3457d8 — the guard originally checked only the raw trailing token, so stemming could still emit a short prefix on the stemmed arm (going → go:*, ads → ad:*), reopening the same pathology. The fix now guards stemmedLastTerm too and adds tests (stemmed-short, multibyte/rune-length, retained-term assertions).

The following are non-blocking follow-ups intentionally left out of this PR:

• [medium] Behavioral (pgtest) coverage. The new tests are structural (inspect the generated literal/args). The rest of this file uses the pgtest harness to prove real matching semantics. Worth adding a DB-backed case proving a dropped-token query returns the intended rows — e.g. build a vector for "admin console" and assert "admin c" matches the same rows as "admin" (dropping c neither over- nor under-matches).
• [low] O(n²) re-stemming on the recurse path. stemmedSearchText is computed at the top of buildSearchQuery (tokenize + Snowball stem of the whole joined text) on every call, including the invocation that immediately recurses after dropping a short trailing token — so an N-short-token query re-stems N, N-1, … terms. Negligible for typeahead-sized N, but moving the stem computation below the short-token guard avoids the discarded work.
• [low] minPrefixLen = 3 is a selectivity heuristic, not a guarantee. A 3-char prefix can still be broad (adm:*, use:*, acc:*). Acceptable trade-off for typeahead (raising the floor hurts legitimate short prefixes); worth a one-line comment so it's not mistaken for a hard selectivity bound.
• [low] Public-API coverage. Tests exercise the unexported buildSearchQuery directly; a short trailing token arriving via the dot-split path ("admin.c" → ["admin","c"]) through the public FullTextSearchQuery isn't covered end-to-end.
• [nit] Brittle assertion. The :* presence check couples to the exact literal template; fine for a structural test, but it would break on a benign template rewrite.

Context: this PR is the fast-follow for a ConductorOne prod incident (inc-947) where a 1-char c:* prefix caused a ~64s GIN prefix scan and a 15s gRPC deadline. Evidence: orion & c:* → ~72s; orion alone → ~1.1s; c:* alone matched ~52% of a tenant's rows.

🤖 Generated with Claude Code

[linear-code]

OPS-2279

OPS-2280

[arreyder]

Re: "search q on a list of users — how pathological is it?"

The key thing: a single-term q never becomes q:*, and this PR doesn't touch it. The :* prefix is only applied to the trailing token of a multi-term query (buildSearchQuery, the len(searchTerms) >= 2 branch). A lone q takes the single-term branch → websearch_to_tsquery('simple','q'), an exact lexeme match, no prefix. Its cost is just "how many docs contain the token q," which for a rare standalone char is small.

(Caveat so I'm not overselling it: a single common short word — e.g. we — does match a lot and can be slow, because it's a frequent content lexeme. But that's a different problem from this PR: it's handled client-side by ductone/c1#21791, which gates the indirect-access search box to ≥3 chars so we/a/q never fire a tenant-wide FTS. This PR is only about the :* prefix.)

The pathology :* guards against happens once you've typed a second word (e.g. john q → john & q:*). A GIN prefix scan expands x:* to the union of posting lists for every lexeme starting with x, and the && <other terms> AND does not short-circuit it — you pay the full prefix cost regardless of the other terms' selectivity. Prod measurement for c:*: 188s total, the GIN Bitmap Index Scan alone read 946,893 blocks to return 466 rows; a bare c:* matched ~52% of that tenant's rows. q is a rarer leading letter than c so less extreme, but it still expands to every q-word with ~zero selectivity value as a 1-char "prefix," and the worst case is unbounded (scales with table size + cache state).

Why drop the short trailing token instead of exact-matching it: content is edge-gram indexed only at minWordSize = 3, so exact-matching a <3-char token returns ~nothing — john q would go empty. Dropping it searches john, the sensible mid-type result. Anything with a ≥3-char trailing token is byte-identical to today.

Net: no regression to single-char search (already exact/near-empty for a rare char), and this only changes multi-term queries whose trailing token is 1–2 chars.