Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion docs/phoenixmldb/full-text-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Disposing the handle `StartFullTextIndexing` returns cancels the loop and awaits

## Rebuilding

A container's full-text index (like its other indexes) can become **stale**: written to while indexing was not enabled, or last touched by an engine version that predates it. A stale container's full-text index has no entries for whatever was written during that gap — and critically, those documents were **never queued either**, so nothing short of a rebuild can find them. Every other index-backed read path handles this by degrading to a scan; `SearchFullText` refuses instead — **it throws `InvalidOperationException` rather than return an answer that looks complete and is not**:
A container's full-text index (like its other indexes) can become **stale**: written to while indexing was not enabled, last touched by an engine version that predates stale-tracking, or built with a different text analysis than the running engine uses (see [After an engine upgrade](#after-an-engine-upgrade)). A stale container's full-text index has no entries for whatever was written during that gap — and critically, those documents were **never queued either**, so nothing short of a rebuild can find them. Every other index-backed read path handles this by degrading to a scan; `SearchFullText` refuses instead — **it throws `InvalidOperationException` rather than return an answer that looks complete and is not**:

```csharp
db.ContainersWithStaleIndexes() // ["docs"] — this container needs a rebuild
Expand All @@ -119,6 +119,23 @@ manager.SearchFullText(docs.Id, "walrus").Count // 1 — same answer, now serv

A rebuild clears the container's previous Lucene entries and its full-text queue, then walks every document and re-enqueues it exactly as a live write would, and clears the stale flag as part of the same commit — so `SearchFullText` stops throwing the moment `RebuildIndexesAsync` returns. **You do not need a drain for that.** What the rebuild leaves behind is fresh queue entries, not a fresh Lucene index — but every one of those entries is now pending, and the `SearchFullText` guarantee above already covers a pending document: it is evaluated directly against its current stored content, so the answer is exact whether or not the worker has caught up. A drain afterward moves that work from direct evaluation into the index; it does not change what `SearchFullText` can answer, only how it answers it.

### After an engine upgrade

A full-text index records which text analysis built it. `EnableIndexing()` compares that record with the analysis the running engine uses, and marks every container that doesn't match as stale. It does this before anything else can write through the new manager. Indexes built before this record existed carry none, so **the first time you open an existing database with indexing enabled after upgrading, every full-text index is marked stale and must be rebuilt once**. The same happens whenever a later PhoenixmlDb.XQuery release changes full-text analysis. The next one is expected to add stemming and case sensitivity.

- **gRPC server:** nothing to do. `StaleIndexRebuildService` rebuilds stale containers in the background at startup.
- **Embedded applications:** rebuild after enabling indexing:

```csharp
var manager = db.EnableIndexing();
foreach (var name in db.ContainersWithStaleIndexes())
await db.RebuildIndexesAsync(name);
```

Until a container is rebuilt, `SearchFullText` on it throws `InvalidOperationException`, and its other index-backed reads (path, value, metadata) scan instead of using their indexes. Those answers stay correct, just slower.

**Don't call `EnableIndexing()` while holding a transaction from `BeginWriteAsync`.** If it has to mark a container stale, it takes the write lock, and it can't tell your transaction from another writer's, so it waits forever.

## Where the index lives

The Lucene index is stored entirely inside your LMDB environment, through a custom `Directory` that maps each Lucene file to an LMDB value keyed by container and file name. After writing documents to a full-text-indexed container and draining the queue, the on-disk footprint is unchanged from any other PhoenixmlDb database — `data.mdb` and `lock.mdb`, and nothing else. There is no `segments_N`, no `.cfs`, no separate Lucene directory to back up, replicate, or lose track of: it replicates and restores exactly as the rest of your database does.
Expand Down
12 changes: 12 additions & 0 deletions docs/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,18 @@ that parsed JSON from `/health` must call `/health/details` instead. `/health/re
database and engine checks; it used to always return `200`. See
[Server Mode: Health Endpoints](phoenixmldb/deployment/server-mode.md#health-endpoints).

### Upgrade: full-text indexes are rebuilt once

Since phoenixml `main` 8d23b02, the database uses **PhoenixmlDb.XQuery and PhoenixmlDb.Xslt
2.4.1** (Core stays at 2.0.0), and a full-text index records which text analysis built it.
Existing indexes carry no such record, so the first time a database is opened with indexing
enabled, **every full-text index is marked stale and rebuilt once**. The gRPC server does this in
the background at startup. Embedded applications must call `RebuildIndexesAsync` for each
container in `ContainersWithStaleIndexes()`; until then `SearchFullText` throws on that container
and other index-backed reads scan. Full-text query behaviour itself is unchanged by this bump;
stemming and position-based phrases are not in 2.4.1. See
[Full-Text Search: After an engine upgrade](phoenixmldb/full-text-search.md#after-an-engine-upgrade).

## Engines: PhoenixmlDb.Xslt and PhoenixmlDb.XQuery

Since 2.0.0 the two engines release together as one **train**: the Xslt and XQuery versions
Expand Down
Loading