From 19c2f016785d949f81d151ac09e5273c6ddae0df Mon Sep 17 00:00:00 2001 From: Lucas Vogel Date: Tue, 29 Sep 2026 22:31:15 -0400 Subject: [PATCH] docs: full-text indexes are rebuilt once after the engine upgrade phoenixml main 8d23b02 pins PhoenixmlDb.XQuery/Xslt 2.4.1 and records which text analysis built each full-text index. EnableIndexing() marks every container whose index doesn't match as stale, so existing indexes (which carry no record) are rebuilt once. The server rebuilds them at startup through StaleIndexRebuildService; embedded apps rebuild each container in ContainersWithStaleIndexes(). Documents the deadlock if EnableIndexing() runs inside a BeginWriteAsync transaction. Facts from the db-engine session, verified at 8d23b02 in DocumentDatabaseIndexingExtensions.cs, IndexManager.SearchFullText, StaleIndexRebuildService and Program.cs. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_019zMS2EzLV6KZG6ooYco8Uf --- docs/phoenixmldb/full-text-search.md | 19 ++++++++++++++++++- docs/release-notes.md | 12 ++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/phoenixmldb/full-text-search.md b/docs/phoenixmldb/full-text-search.md index e3cc295..b5720bf 100644 --- a/docs/phoenixmldb/full-text-search.md +++ b/docs/phoenixmldb/full-text-search.md @@ -99,7 +99,7 @@ Disposing the handle `StartFullTextIndexing` returns cancels the loop and awaits ## Rebuilding -A container's full-text index (like its other indexes) can become **stale**: written to while indexing was not enabled, or last touched by an engine version that predates it. A stale container's full-text index has no entries for whatever was written during that gap — and critically, those documents were **never queued either**, so nothing short of a rebuild can find them. Every other index-backed read path handles this by degrading to a scan; `SearchFullText` refuses instead — **it throws `InvalidOperationException` rather than return an answer that looks complete and is not**: +A container's full-text index (like its other indexes) can become **stale**: written to while indexing was not enabled, last touched by an engine version that predates stale-tracking, or built with a different text analysis than the running engine uses (see [After an engine upgrade](#after-an-engine-upgrade)). A stale container's full-text index has no entries for whatever was written during that gap — and critically, those documents were **never queued either**, so nothing short of a rebuild can find them. Every other index-backed read path handles this by degrading to a scan; `SearchFullText` refuses instead — **it throws `InvalidOperationException` rather than return an answer that looks complete and is not**: ```csharp db.ContainersWithStaleIndexes() // ["docs"] — this container needs a rebuild @@ -119,6 +119,23 @@ manager.SearchFullText(docs.Id, "walrus").Count // 1 — same answer, now serv A rebuild clears the container's previous Lucene entries and its full-text queue, then walks every document and re-enqueues it exactly as a live write would, and clears the stale flag as part of the same commit — so `SearchFullText` stops throwing the moment `RebuildIndexesAsync` returns. **You do not need a drain for that.** What the rebuild leaves behind is fresh queue entries, not a fresh Lucene index — but every one of those entries is now pending, and the `SearchFullText` guarantee above already covers a pending document: it is evaluated directly against its current stored content, so the answer is exact whether or not the worker has caught up. A drain afterward moves that work from direct evaluation into the index; it does not change what `SearchFullText` can answer, only how it answers it. +### After an engine upgrade + +A full-text index records which text analysis built it. `EnableIndexing()` compares that record with the analysis the running engine uses, and marks every container that doesn't match as stale. It does this before anything else can write through the new manager. Indexes built before this record existed carry none, so **the first time you open an existing database with indexing enabled after upgrading, every full-text index is marked stale and must be rebuilt once**. The same happens whenever a later PhoenixmlDb.XQuery release changes full-text analysis. The next one is expected to add stemming and case sensitivity. + +- **gRPC server:** nothing to do. `StaleIndexRebuildService` rebuilds stale containers in the background at startup. +- **Embedded applications:** rebuild after enabling indexing: + +```csharp +var manager = db.EnableIndexing(); +foreach (var name in db.ContainersWithStaleIndexes()) + await db.RebuildIndexesAsync(name); +``` + +Until a container is rebuilt, `SearchFullText` on it throws `InvalidOperationException`, and its other index-backed reads (path, value, metadata) scan instead of using their indexes. Those answers stay correct, just slower. + +**Don't call `EnableIndexing()` while holding a transaction from `BeginWriteAsync`.** If it has to mark a container stale, it takes the write lock, and it can't tell your transaction from another writer's, so it waits forever. + ## Where the index lives The Lucene index is stored entirely inside your LMDB environment, through a custom `Directory` that maps each Lucene file to an LMDB value keyed by container and file name. After writing documents to a full-text-indexed container and draining the queue, the on-disk footprint is unchanged from any other PhoenixmlDb database — `data.mdb` and `lock.mdb`, and nothing else. There is no `segments_N`, no `.cfs`, no separate Lucene directory to back up, replicate, or lose track of: it replicates and restores exactly as the rest of your database does. diff --git a/docs/release-notes.md b/docs/release-notes.md index 2ffd7a2..5288eef 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -33,6 +33,18 @@ that parsed JSON from `/health` must call `/health/details` instead. `/health/re database and engine checks; it used to always return `200`. See [Server Mode: Health Endpoints](phoenixmldb/deployment/server-mode.md#health-endpoints). +### Upgrade: full-text indexes are rebuilt once + +Since phoenixml `main` 8d23b02, the database uses **PhoenixmlDb.XQuery and PhoenixmlDb.Xslt +2.4.1** (Core stays at 2.0.0), and a full-text index records which text analysis built it. +Existing indexes carry no such record, so the first time a database is opened with indexing +enabled, **every full-text index is marked stale and rebuilt once**. The gRPC server does this in +the background at startup. Embedded applications must call `RebuildIndexesAsync` for each +container in `ContainersWithStaleIndexes()`; until then `SearchFullText` throws on that container +and other index-backed reads scan. Full-text query behaviour itself is unchanged by this bump; +stemming and position-based phrases are not in 2.4.1. See +[Full-Text Search: After an engine upgrade](phoenixmldb/full-text-search.md#after-an-engine-upgrade). + ## Engines: PhoenixmlDb.Xslt and PhoenixmlDb.XQuery Since 2.0.0 the two engines release together as one **train**: the Xslt and XQuery versions