Skip to content

refactor(storage): isolate dialect search behind Bun capabilities - #1345

Open
mariusvniekerk wants to merge 6 commits into
t3code/bun-store-readsfrom
t3code/bun-search-unification
Open

refactor(storage): isolate dialect search behind Bun capabilities#1345
mariusvniekerk wants to merge 6 commits into
t3code/bun-store-readsfrom
t3code/bun-search-unification

Conversation

@mariusvniekerk

@mariusvniekerk mariusvniekerk commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Full-text and vector candidate retrieval legitimately differs by engine, but hydration, visibility, filtering, deduplication, and public pagination should not. This layer narrows dialect-specific search to capability adapters and moves the observable search pipeline into the shared Bun store.

DuckDB tool-content search now uses canonical session and message-ordinal identity instead of disposable physical row IDs. Timestamp ordering and portable search choices are supplied through narrow adapter hooks, while engine-specific relevance ranks remain allowed where the engines expose genuinely different semantics. Stack 3 of 5.

generated by a clanker

@roborev-ci

roborev-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown

roborev: Combined Review (7bb6694)

Code review passed with no Medium, High, or Critical findings.


Reviewers: 2 done | Synthesis: codex, 6s | Total: 20m56s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-search-unification branch from 7bb6694 to 5fcc6ce Compare August 7, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (5fcc6ce)

No issues found.


Reviewers: 2 done | Synthesis: codex | Total: 10m38s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-search-unification branch from 5fcc6ce to a27f90c Compare August 7, 2026 14:10
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (a27f90c)

Medium-severity issue found in remote Quack regex search.

Medium

  • internal/db/bun_search_content.go:686 — Direct QueryContext bypasses DuckDB’s Bun connection resolver, causing regex searches against a remote Quack store to query the local client database instead of the remote mirror. Stream rows through a Bun raw query that uses the resolver, and add Quack regex-search coverage.

Reviewers: 2 done | Synthesis: codex, 5s | Total: 10m13s

@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-search-unification branch from a27f90c to db3867f Compare August 7, 2026 16:34
@roborev-ci

roborev-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown

roborev: Combined Review (db3867f)

Search refactor needs a pagination-stability fix before approval.

Medium

  • internal/db/bun_search_content.go:890 — DuckDB leaves tool-call and result-event IDs null, causing tied rows to receive row_order = 0. Without further tie-breakers, paginated searches can duplicate, skip, or reorder results. Append call_index ASC, event_index ASC to the final ORDER BY.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 12m8s

Session search needs engine-specific matching and ranking, but visibility, metadata hydration, pagination, and lifecycle consistency are common storage behavior. Keeping the complete read in each backend duplicated canonical rules and let raw queries escape the guarded Bun snapshot.

Pass the operation-scoped Bun handle into narrow FTS capabilities so SQLite FTS5, PostgreSQL ILIKE, and DuckDB matching differ only at the SQL boundary while every public store resolves through one Bun hydration path.
Lexical content search needs dialect-specific matching, but source hydration, visibility, snippets, timestamps, conversation-unit ranges, ordering, and cursors are canonical behavior. Keeping those rules in three stores made parity fixes expensive and allowed each backend to drift.

Route substring and regex matching through portable Bun queries, retain only SQLite FTS5 versus portable FTS candidate selection as a capability, and delete the superseded backend scanners. Capability limits now remain operation-scoped so the shared store can fetch its cursor sentinel without collapsing the public maximum page size.
The common search layer depends on candidate windows having the same visibility, ordering, and limit semantics on every adapter. Boundary lookahead, optional SQLite FTS, missing activity timestamps, and backend-specific FTS ordering could otherwise shorten pages or reorder identical results.

Make the capability pagination contract explicit, preserve in-session substring search without a global index, use the canonical activity fallback and recency order, and reject unsupported FTS sources. Retain Unicode recent-edit matching on SQLite without returning to full edit-payload hydration.
Semantic and hybrid search need dialect-specific vector and lexical candidate generation, but visibility, unit resolution, scope filtering, fusion, hydration, redaction, and final ordering are canonical behavior. Keeping complete pipelines in SQLite and PostgreSQL duplicated those rules and made backend parity fragile.

Route both modes through BunStore capabilities, retain only SQLite FTS5 versus PostgreSQL keyword candidate SQL at the adapter boundary, and preserve backend-specific unavailable guidance. Remove the superseded concrete enrichment and pagination paths so future search changes have one owner.
Shared search must preserve Unicode matching, stable pagination, semantic range provenance, and snapshot safety across every adapter. Review exposed cases where portable SQL or transaction boundaries could silently drop valid matches, retain stale vector anchors, or exhaust the PostgreSQL connection pool.

Keep engine-specific candidate generation narrow while separating bounded session search from global FTS availability and ensuring canonical hydration owns final visibility and range behavior.
Shared search decisions should follow explicit adapter capabilities instead of backend-name checks, while DuckDB session search must use the mirror canonical message relationship even when optional source row IDs are absent.
@mariusvniekerk
mariusvniekerk force-pushed the t3code/bun-search-unification branch from db3867f to e0be781 Compare August 9, 2026 03:46
@roborev-ci

roborev-ci Bot commented Aug 9, 2026

Copy link
Copy Markdown

roborev: Combined Review (e0be781)

One medium-severity ordering bug should be fixed before merging.

Medium

  • internal/db/bun_search_content.go:820 — Content candidate ordering coalesces SQLite timestamps before converting legacy empty strings to NULL. An empty ended_at can therefore mask a valid started_at or created_at, causing recent substring or regex matches to sort last or appear on the wrong page. Normalize each timestamp before coalescing (for example, with NULLIF(..., '') in SQLite or a dialect-owned activity expression) and add an ordering test covering empty legacy timestamps.

Reviewers: 2 done | Synthesis: codex, 24s | Total: 21m23s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant