Skip to content

feat(search): collection rows join the page search as guests on the page scale - #107

Merged
ssalentin merged 9 commits into
mainfrom
feat/is-84-search-rows
Nov 15, 2026
Merged

ssalentin merged 9 commits into
mainfrom
feat/is-84-search-rows

Conversation

@ssalentin

Copy link
Copy Markdown
Owner

Implements #84: the wiki search covers collection rows next to pages, live, without a restart.

Index. Rows are a separate document set inside the one SearchIndex (rowDocs), built from property: text lines with the same tokenize/stem/compound machinery as pages. rankRows reuses the page idf, bm25 and field weights; df, vocab and avgLen stay page-only, so feeding rows never moves a page score (a golden test fingerprints page scores and order before and after). runSearch merges pages and rows into one ranked list; a row hit has an empty path and a row object with collectionId, collectionName, rowId, the matched property and an href that opens the row directly in the table view. #tag and in:folder queries stay page-only.

Live wiring. SpaceIndex.setCollectionRows remembers its inputs and re-applies them on every rebuild, so page reloads cannot drop the rows. Mdwiki seeds rows at boot only when the collections file already exists (never creates the DB as a side effect) and otherwise attaches at the first open through whenCollectionsOpen. Store events mark collections dirty and coalesce into one re-index 250 ms after the last write; flushCollectionRows runs the same converge-by-reread path, so a deleted collection loses its rows.

UI and REST. The search page renders row hits in the same list with a Collection · Property label and links to row.href; the count line reads X pages · Y rows. GET /api/v1/search and the MCP search tool pass the same objects through, so agents see row hits too; the MCP description now says so.

Tests. test/search-rows.test.ts adds 17 cases in three layers: row docs on the bare index (shape, href, idempotency, validation), a page-ranking regression fingerprint (scores, order, per_page and facets byte-identical before and after feeding rows), and the live Mdwiki path on a real temp SQLite file (boot seeding, attach-on-first-open without creating the DB, update and delete propagation, reload survival, burst coalescing under fake timers). 484 tests pass on this branch (main 459, plus 8 from the merged #83 branch, plus these 17), svelte-check 0 errors, build green.

Verification lever. scripts/search-rows-walk.sh reruns the acceptance against a copy of any instance: copy space and database into a temp dir, start the built server on a free port, search a real value, create/edit/delete a scratch row over REST and assert each change reaches the index without a restart and that #tag queries return no rows. Every write stays below the temp dir.

Evidence, measured 2026-11-15 against a copy of the live instance (7 collections, 153 rows):

search row coverage acceptance walk

The capture shows a real Rolodex phone value found at boot as one page plus four row hits in one ranked list, the scratch row searchable one second after its create, the edit replacing the old value in the index, the delete removing it, the #tag query returning no rows, and one unchanged server process across the whole walk.

Two review fixes on top of the implementation: the search page count line printed result.total as pages although total now includes row hits (now total - rows), and the MCP search description owed the row-hit sentence.

Rows live in a dedicated rowDocs map on the one SearchIndex; corpus
statistics (df, vocab, avgLen) stay page-only, so page scores are
computed exactly as before. rankRows() scores row documents with the
same idf/bm25/FIELD_WEIGHT and coverage/phrase rules, runSearch merges
the two lists into one ranked result (pages win ties) and exposes
SearchHit.row with collection/row identity and a direct-open href.
allRows() reads every row uncapped (query() keeps its 500-row limit for
the UI); the search index needs them all. whenCollectionsOpen() runs a
callback as soon as the singleton store is open without ever opening it
itself, so boot-time row indexing can attach to the first UI open
without creating a database as a side effect.
Mdwiki.start() opens an existing collections DB eagerly and indexes
every collection; without the file it attaches via whenCollectionsOpen.
store.subscribe marks collections dirty and a trailing 250 ms timer
coalesces bursts into one re-read per collection. The operation is
always 'derive collection X from the current DB state', never a delta,
so missed events, bursts and crashes converge; a vanished collection
(404) removes its rows. collectionRowsForIndex maps allRows() via the
plainCellText rule extracted from database-block.ts, so value-to-text
exists once. test/helpers pins COLLECTIONS_DB to the temp space: boot
time opening must never reach the live collections database.
The results list keys by row id for row hits, renders them as a link to
the direct-open href with a collection-name chip and the snippet, reads
the count line as 'X pages · Y rows' and keeps layer/facet counters on
page hits only.
…he MCP search tool

Review findings on the implementation: the search page printed result.total as
the page count although total now includes row hits, so it double-counted; it
now prints total minus rows as pages. The MCP search tool description gains the
sentence about row hits, the row object and the page-only tag and folder
queries, which the feature owed the machine surface.
…opied instance

The walk copies space and database into a temp dir, starts the built server on
a free port and proves over REST that a real value is searchable right after
boot, that pages and rows share one ranked list, that create, edit and delete
reach the index without a restart, and that a #tag query stays page-only. Every
write stays below the temp dir, so the source instance is never touched.
… live instance

Terminal capture of scripts/search-rows-walk.sh on 2026-11-15 against a copy of
the live instance: the phone query returns one page and four row hits in one
ranked list, the scratch row is searchable a second after its create, the edit
replaces the old value in the index, the delete removes it and the #tag query
returns no rows, all with one unchanged server process.
@ssalentin
ssalentin merged commit fe38d1b into main Nov 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant