From 25d0868b9d3aac5308e367925f4db8729429def2 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sat, 29 Aug 2026 22:02:41 +0000 Subject: [PATCH 01/50] Add design spec for single-table FTS query shape Hybrid search takes seconds for any term matching a large share of the corpus: 6.5 s for a term matching all 5M reports on the benchmark rig. The cost is not ts_rank itself but the joins around it -- filters are built as report__ traversals, so the candidate query joins three tables and applies SELECT DISTINCT, which drags the tsvector through a spilling hash join and blocks the top-N heapsort. The spec completes ReportSearchIndex as a search projection so the scan is single-table, maintained by statement-level triggers rather than application signals because group membership is access-control data. Measured 13,440 ms -> ~420 ms at 5M reports. Records what was ruled out with measurements, including BM25 via pg_textsearch (33x slower as written) and a stripped ranking tsvector (collapses ts_rank to one distinct value). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 361 ++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md new file mode 100644 index 00000000..8dfc4acb --- /dev/null +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -0,0 +1,361 @@ +# Search performance: making the FTS scan single-table + +**Status:** design, not yet implemented +**Branch:** `fts-query-shape-performance` +**Supersedes:** parts of `hybrid-search.md` §7 (the FTS candidate query and its filter construction) + +## 1. Problem + +Hybrid search takes seconds for any query term that matches a large share of the +corpus. Measured on a synthetic 5,001,000-report corpus (§8): + +| query | reports matched | page load | +| --- | --- | --- | +| `findings` | 5,001,000 | 6.5 s | +| `effusion` | 2,360,472 | 4.1 s | +| `pneumonia` | 520,104 | 3.8 s | +| `adenoma` | 5,001 | 0.17 s | + +A later run measured `pneumonia` at 1.9 s after an `ANALYZE` improved the +planner's estimates; the table above is one coherent run, and the conclusion is +unchanged either way. + +Cost tracks the number of *matching* reports, not corpus size. Paging is flat — +page 1 and page 400 both cost ~6.5 s — because every page re-ranks the whole +match set from scratch. + +## 2. Diagnosis + +`ORDER BY ts_rank(...)` has no top-k shortcut. A GIN posting list is ordered by +document id and carries no impact information, and `ts_rank` needs the whole +tsvector from the heap, so every match must be scored before `LIMIT` applies. +That part is inherent to PostgreSQL FTS and is not what this design changes. + +What *is* fixable is the constant factor, and it is dominated by the joins. +`_build_filter_query` builds every predicate as a `report__` traversal, so the +FTS candidate query joins `reports_report`, `reports_report_groups` and +`reports_language`, then applies `SELECT DISTINCT`. Two consequences: + +- The planner builds a hash table over the matched rows **carrying + `search_vector`**, because `ts_rank` above the join still needs it. On the 1M rig (§8), + 730k matches, that hash table was 1.2 GB and spilled into 256 batches, + writing ~1.2 GB of temp files at the default `work_mem`. +- `SELECT DISTINCT` blocks the top-N heapsort, forcing a full external merge sort + of every matching row before `LIMIT` can apply. + +Removing both makes the query a single-table parallel sequential scan with a +26 kB top-N heapsort. On the same 5M corpus, `findings` goes from **13,440 ms to +~600 ms** at stock parallelism, and to **~420 ms** with +`max_parallel_workers_per_gather` raised to 4 (§4.5). The plan verified in §4.3 +ran in 404 ms at 8 workers. + +## 3. Ruled out, with measurements + +Recorded because each is the obvious question a reviewer will raise. + +| candidate | result | +| --- | --- | +| Lower `HYBRID_FTS_MAX_RESULTS` | No effect (1M rig). `LIMIT 100` and `LIMIT 10000` both 1.05 s. Under `ts_rank` the pool size is free; it only becomes load-bearing with a pruning index. | +| Raise `work_mem` | No effect (1M rig) (1.41 s at 256 MB vs 1.44 s at 4 MB). The spill is a symptom, not the cause. | +| Rewrite the joins as `EXISTS` | Worse: 1.55 s vs 1.05 s (1M rig). The planner still hashes. | +| Materialised id-only CTE, joins on narrow rows, then rank | Worse: 11,460 ms for `findings`, 3,089 ms for `pneumonia` at 5M. The joins are still over 5,001,000 rows. **This was the only way to avoid duplicating access-control data, and it does not work.** | +| `strip()`ped second tsvector for ranking | Rejected (1M rig). 1.4x faster but collapses `ts_rank` to **1 distinct value** across 200,000 matching rows — `strip()` removes the positions that encode term frequency, so the ordering becomes a no-op. | +| Restricting recall ("rank only the most recent N", "refuse broad queries") | Withdrawn. No production search engine restricts recall; they restrict pagination depth and count precision. Not needed here anyway. | + +## 4. Design + +### 4.1 The projection + +`ReportSearchIndex` is already a derived projection — `search_vector` and +`embedding` are both derived from `report.body`. It is currently half-built: it +carries what search *ranks* on but not what search *filters* on, which is why +every query has to join. This design completes it. + +Governing rule: **denormalise what the scan touches, join for what the page +renders.** The scan visits millions of rows; hydration visits 25. + +```python +# search projection - mirrors of the Report fields the scan filters on +group_ids = ArrayField(IntegerField(), default=list) # Report.groups +modality_codes = ArrayField(CharField(16), default=list) # Report.modalities +language_code = CharField(max_length=10, null=True) +patient_sex = CharField(max_length=1, null=True) +patient_age = IntegerField(null=True) +patient_id = CharField(max_length=64, null=True) +study_datetime = DateTimeField(null=True) +study_description = CharField(max_length=64, blank=True, null=True) +report_created_at = DateTimeField(null=True) +report_updated_at = DateTimeField(null=True) +``` + +The mirrored scalars are nullable so every `AddField` stays metadata-only (§4.4); +tightening them to NOT NULL afterwards would need a validating table scan for no +benefit, and `check_search_projection` guards against drift instead. + +`report_created_at` / `report_updated_at` are named for their source; an +unqualified `created_at` on this table would be ambiguous about whose timestamp +it is. + +Indexes: `GinIndex` on `group_ids` and `modality_codes` (the containment and +overlap operators are unusable without them), btree on `study_datetime`. Nothing +else — the remaining predicates ride the sequential scan, exactly as they do +today, since `reports_report` carries no index on `patient_id`, `patient_sex` or +`study_datetime` either. + +`body`, `pacs_name` and `document_id` stay on `Report`. + +**Labels stay a subquery.** `LabelResult` churns constantly under relabeling +jobs, making `label_names` the highest-maintenance column for the least benefit. + +**Cost:** ~120 bytes per row, about 600 MB at 5M reports — a ~9.5% increase on +the 6.3 GB the scan reads, in exchange for eliminating the joins. + +### 4.2 Consistency + +Three **statement-level** triggers using transition tables: + +| trigger on | fires | maintains | +| --- | --- | --- | +| `reports_report` | AFTER UPDATE | `language_code` + the seven scalars | +| `reports_report_groups` | AFTER INSERT, DELETE | `group_ids` | +| `reports_report_modalities` | AFTER INSERT, DELETE | `modality_codes` | + +Statement-level, not row-level: `bulk_upsert_report_search_indexes` writes in +5000-row chunks, and row-level triggers would fire 5000 times per statement. +With `REFERENCING NEW TABLE AS ...` each statement costs one set-based +`UPDATE ... FROM`. + +The `reports_report` trigger guards its write with `IS DISTINCT FROM` across the +mirrored columns so unrelated `Report.save()` calls do not rewrite projection +rows needlessly. + +The two paths that *create* `ReportSearchIndex` rows — the `post_save` receiver +in `pgsearch/signals.py` and `bulk_upsert_report_search_indexes` — populate the +projection at creation. Triggers only maintain it afterwards, so a trigger +firing before a row exists is a harmless zero-row `UPDATE`. + +**Why triggers rather than `m2m_changed`.** `group_ids` governs who may read a +patient's report. An application signal fires only for Django ORM operations, +missing `add_custom_report.py`, admin actions, management commands and raw SQL. +A trigger holds inside the same transaction as the write, for every writer. +Precedent exists in the repo: `Report.patient_age` is a +`GeneratedField(db_persist=True)` computed by a hand-written SQL function +installed via `RunSQL` in `reports/migrations/0011`. + +**Triggers must be AFTER, not BEFORE.** PostgreSQL does not expose stored +generated column values to BEFORE triggers, so a BEFORE trigger would mirror +`patient_age` as NULL. + +**Reconciliation.** A `check_search_projection` management command verifies the +projection against its sources with one set-based query per column and reports +drift. Cheap insurance on access-control data, and something operators can run +after a restore or bulk import. + +### 4.3 Query layer + +`_build_filter_query` becomes a single-table predicate builder: + +| today | after | +| --- | --- | +| `Q(report__groups=filters.group)` | `Q(group_ids__contains=[filters.group])` | +| `Q(report__language__code=...)` | `Q(language_code=...)` | +| `Q(report__modalities__code__in=...)` | `Q(modality_codes__overlap=...)` | +| `Q(report__patient_sex=...)`, age, dates, description, patient_id | same, unprefixed | +| `Q(report__created_at__gte=...)` | `Q(report_created_at__gte=...)` | +| labels | unchanged — `report_id__in=`, no join | + +**`.distinct()` is removed unconditionally.** It exists only because the `groups` +and `modalities` joins can duplicate rows. With no joins there is nothing to +deduplicate. Worth ~30% on its own (1.44 s → 1.04 s at 1M). + +**Access-control edge case.** `filters.group=None` is documented as fail-closed: +`Q(report__groups=None)` compiles to `groups__isnull=True` and matches only +reports in no group. The array equivalent is `Q(group_ids=[])` — *not* any form +of `contains`. This is the easiest thing in the change to get subtly wrong and +gets its own test. + +`match_q`, `rank_expr`, `summary_expr` and `_exclude_negations` all currently +traverse `report__language__code` and become `language_code` on the row. The +vector half also sheds its `.distinct()` (6.5 ms → 1.0 ms). + +The FTS candidate query then compiles to the shape measured at 404 ms: + +```sql +SELECT report_id, + CASE WHEN language_code IN ('en') + THEN ts_rank(search_vector, to_tsquery('english', ...)) END AS rank +FROM pgsearch_reportsearchindex +WHERE group_ids @> ARRAY[1] + AND language_code IN ('en') + AND search_vector @@ to_tsquery('english', ...) +ORDER BY rank DESC, report_id +LIMIT 10000; +``` + +**Consumers.** `search()`, `count()` and `retrieve()` inherit the fix with no +logic change; all three route through `_fuse_hybrid`. `filter()` (subscriptions) +keeps one join for `report__document_id`; it is bounded by `updated_after`, so +mirroring `document_id` would be scope for no measured gain. Hydration is +untouched — `page_rows` still `select_related("report")` for `ts_headline` over +`report__body`, which is 25 rows. + +### 4.4 Migration and backfill + +`pgsearch/migrations/0003_search_projection.py`, with `atomic = False`: + +1. **Add columns.** Arrays get `default=list` (`DEFAULT '{}'`, NOT NULL); scalars + go in nullable. All are constant-default or nullable, so every `AddField` is + metadata-only and instant even at 5M rows. An empty `group_ids` is the + fail-closed value. +2. **Create triggers** — *before* the backfill, so reports edited during a + multi-minute backfill still land correctly. A chunk that later reprocesses the + same row simply rewrites the same current values. +3. **Backfill**, chunked by `report_id` range with one transaction per chunk + (hence `atomic = False`). Each chunk is a set-based `UPDATE ... FROM + reports_report` with two lateral `array_agg` subqueries for the arrays. +4. **Create indexes last.** Building GIN indexes on empty columns and then + filling them is wasted work and leaves bloat. + +**Deployment consequences.** `docker-compose.prod.yml` runs `manage.py migrate` +in the `init` service and `web` waits on it, so the backfill blocks the deploy +rather than serving half-projected rows — the right failure mode for +access-control data. Deploy time therefore grows with corpus size: an estimated +2–5 minutes at 5M, extrapolated from analogous full-table writes (44 s for a +4M-row m2m insert, 83 s for a 4M-row tsvector update). The backfill rewrites +every row, leaving ~6 GB of dead tuples at 5M; autovacuum reclaims the space for +reuse but will not shrink the file, so sites with a maintenance window may want +`VACUUM FULL` afterwards. + +An online alternative — resumable backfill command plus a feature flag on the +query path — was considered and rejected: it doubles the query-layer code for +the duration and the flag has to be removed later anyway. + +### 4.5 PostgreSQL configuration + +Delivered as `-c` GUCs on the `postgres` service's `command:`, driven by env vars +with defaults documented in `example.env` and `CLAUDE.md`. + +Measured at 5M on `findings`: + +| `max_parallel_workers_per_gather` | time | +| --- | --- | +| 2 (stock) | 606 ms | +| 4 | 421 ms | +| 6 | 379 ms | +| 8 | 343 ms | + +- **`max_parallel_workers_per_gather` = 4**, with `max_worker_processes` and + `max_parallel_workers` raised to match — all three are needed, since the + cluster-wide caps default to 8 and would otherwise throttle it. 4 captures most + of the available gain while leaving cores for concurrent searches. +- **`shared_buffers`** raised from the stock 128 MB, as general hygiene against a + multi-GB database. Labelled honestly: the 343 ms figure came from a warm *OS* + page cache, so no specific gain is attributed to this. Guidance is ~25% of RAM. +- **`work_mem` deliberately not raised.** It mattered only because `.distinct()` + forced an external merge sort; without it the sort is a 26 kB top-N heapsort. + Raising it is per-connection-per-node memory risk for zero measured benefit. + +**GIN index:** `ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = off)` +plus a one-time `gin_clean_pending_list()`. A storage-parameter change, so no +rebuild. Trade: index inserts get slower, read latency gets predictable. RADIS +writes in batches asynchronously through Procrastinate and reads interactively, +so that is the right side of the trade — but it is a trade. + +## 5. Testing + +**Filter equivalence via a reference oracle.** Keep the current joined +`_build_filter_query` in the test module as `_build_filter_query_legacy` and +assert old and new produce identical report-id sets across a matrix of filter +combinations. Highest-value test in the change: it catches +`modality_codes__overlap` vs `code__in` semantics, date boundaries and the +`group=None` case in one mechanism. Delete the oracle once the change settles. + +**Access control:** +- `group_ids = []` is the only thing `filters.group=None` matches. +- Adding a report to a group makes it findable by that group. +- Removing a report from a group makes it unfindable — the direction that is a + leak if the trigger is wrong. +- A raw `cursor.execute` INSERT into `reports_report_groups`, bypassing the ORM, + still updates the projection. This test is the justification for choosing + triggers over signals and should exist to defend that decision. + +**Bulk correctness:** a `groups.set()` spanning many reports in one statement +updates all of them. Statement-level triggers have a classic failure mode where +only one row is processed; this pins it. + +**No duplicates without `.distinct()`:** a report in several matching groups, or +with several matching modalities, appears exactly once. + +**Plan-shape assertion instead of timing.** Wall-clock assertions are flaky in +CI; structure is not. Run `EXPLAIN` on the generated FTS query and assert the +plan contains no join against `reports_report_groups` and no `Unique` node. This +deterministically catches someone reintroducing a `report__` traversal and +silently restoring the 13-second path. + +**`patient_age` is mirrored non-null**, which fails if the triggers are ever +converted to BEFORE. + +Existing search, extraction and subscription tests are the regression net for the +four consumers and should pass unchanged. + +## 6. Non-goals + +- **BM25 / pg_textsearch (#285).** Deferred and reclassified as a *relevance* + decision, not a performance one. Measured: as written it is **33x slower** than + the `ts_rank` path it replaces (440,954 ms vs 13,440 ms for `findings`), because + `.distinct()` and the joins prevent the index-ordered scan and `<@>` degenerates + to per-row scoring at ~92 µs/row. It needs this denormalisation *plus* a + two-phase small-k formulation to pay off (370 ms for `findings`, 242 ms for + `pneumonia`). Index build was 6m41s and 2014 MB at 5M, holding a lock on + `reports_report`. What BM25 still offers that PostgreSQL FTS cannot: IDF, term + frequency saturation and length normalisation. +- **#287 impact-ordered lexeme ranks.** Likely unnecessary once plain `ts_rank` + handles 5M matches in 343 ms. Recommendation only; closing it is its author's + call. +- **`HYBRID_FTS_MAX_RESULTS`** stays at 10,000 — measured free under `ts_rank`. +- **`Report.groups` → `ArrayField`.** Would remove the redundancy altogether and + speed up `Report.objects.filter(groups=...)` elsewhere, at the cost of FK + integrity on an access-control field. A domain change, deliberately not bundled + into a search-performance change. +- **Result-page N+1s.** Measured at ~134 SQL queries per 25-result page + (`document.full_report` twice per row via `search/site.py`, `can_view_report`, + the collections count, the notes lookup). Once the scan is 343 ms this is + proportionally significant, and it is the recommended next piece of work — but + it is a different subsystem. +- **#282 (`hnsw.ef_search`) and #284 (fused result cache).** Independent, + unaffected, still worth landing. Note `ef_search` defaults to 40 while + `HYBRID_VECTOR_TOP_K` is 100, so the vector half currently contributes at most + 40 candidates. + +## 7. Success criteria + +- `findings` (5,001,000 matches) drops from 13,440 ms to under 500 ms at the + SQL level on the reference corpus, with `max_parallel_workers_per_gather` = 4. +- The FTS candidate query plan contains no join to `reports_report_groups` and + no `Unique` node. +- `check_search_projection` reports zero drift after the backfill and after a + round of report and group mutations. +- Existing search, extraction and subscription tests pass unchanged. + +## 8. Measurement rig + +All figures above come from two throwaway corpora built from +`samples/reports_en.json` (1000 real-shaped report bodies, cycled with a unique +md5 suffix per report): + +- **5M rig:** the dev stack database, 5,001,000 reports, ~15 GB, 6.3 GB + `pgsearch_reportsearchindex` heap, average tsvector 1220 bytes. +- **1M rig:** a standalone container, 1,000,000 reports, 730,000 matching + `pneumonia`, average tsvector 1607 bytes. + +Both on stock PostgreSQL 17 settings (`shared_buffers` 128 MB, `work_mem` 4 MB, +`max_parallel_workers_per_gather` 2) on a 16-core host, unless a measurement +states otherwise. + +**Caveats that bound these numbers.** The corpora are synthetic: 1000 distinct +body templates means lexeme distribution and `ts_rank` score diversity are less +varied than real reports (a 200k-row sample produced only 7 distinct `ts_rank` +values). Measurements are single-user with a warm OS page cache — the 404 ms +figure read 6.2 GB of buffers in ~320 ms, which is page cache, not disk. On a +production host with less RAM relative to corpus size, this path becomes +I/O-bound and degrades. That, rather than CPU, is the real scaling ceiling. From b31ea9c0b81bb0268196fca3adf73f8cb7fd0ec7 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 07:41:09 +0000 Subject: [PATCH 02/50] Correct vacuum guidance in the FTS projection spec The spec suggested sites "may want VACUUM FULL afterwards" without noting that it rebuilds every index on the table, including the HNSW vector index. On a deployment with embeddings populated that is an index rebuild over millions of vectors under an ACCESS EXCLUSIVE lock. Also adds the VACUUM ANALYZE that the migration must run -- ten new columns and three new indexes carry no statistics, and the design depends on the planner choosing a parallel sequential scan -- and records why indexes are created last: unindexed columns let the backfill use HOT updates, which is the main defence against bloat. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 39 ++++++++++++++----- 1 file changed, 30 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 8dfc4acb..960e27f6 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -210,21 +210,42 @@ untouched — `page_rows` still `select_related("report")` for `ts_headline` ove 2. **Create triggers** — *before* the backfill, so reports edited during a multi-minute backfill still land correctly. A chunk that later reprocesses the same row simply rewrites the same current values. -3. **Backfill**, chunked by `report_id` range with one transaction per chunk - (hence `atomic = False`). Each chunk is a set-based `UPDATE ... FROM - reports_report` with two lateral `array_agg` subqueries for the arrays. -4. **Create indexes last.** Building GIN indexes on empty columns and then - filling them is wasted work and leaves bloat. +3. **Backfill**, chunked by `report_id` range (50,000 rows) with one transaction + per chunk (hence `atomic = False`). Each chunk is a set-based `UPDATE ... FROM + reports_report` with two lateral `array_agg` subqueries for the arrays. The + migration lowers `autovacuum_vacuum_scale_factor` on the table for the + duration and resets it after, so dead tuples from one chunk are reclaimed and + reused by the next rather than accumulating across the run. The chunk size is + a constant, not a setting: a migration is a historical record and should do + the same thing on every machine. +4. **Create indexes last, then `VACUUM ANALYZE`.** Building GIN indexes on empty + columns is wasted work — but the stronger reason is that while the new columns + are unindexed the backfill can use HOT updates, whose old row versions are + reclaimed by page pruning without index cleanup. That is the main defence + against the bloat below, so this ordering must not be changed. The `ANALYZE` + is required, not hygiene: ten new columns and three new indexes carry no + statistics, and this design depends on the planner choosing a parallel + sequential scan with a top-N heapsort. **Deployment consequences.** `docker-compose.prod.yml` runs `manage.py migrate` in the `init` service and `web` waits on it, so the backfill blocks the deploy rather than serving half-projected rows — the right failure mode for access-control data. Deploy time therefore grows with corpus size: an estimated 2–5 minutes at 5M, extrapolated from analogous full-table writes (44 s for a -4M-row m2m insert, 83 s for a 4M-row tsvector update). The backfill rewrites -every row, leaving ~6 GB of dead tuples at 5M; autovacuum reclaims the space for -reuse but will not shrink the file, so sites with a maintenance window may want -`VACUUM FULL` afterwards. +4M-row m2m insert, 83 s for a 4M-row tsvector update). The backfill rewrites every row: measured, the tsvectors average 1220 bytes and +nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not +just the new columns. Left unchecked that is ~6 GB of dead tuples at 5M, and +since the hot path is a sequential scan it reads them — roughly halving the gain +until the space is reused. Steps 3 and 4 are what keep it below that. + +**Do not run `VACUUM FULL` from the migration, and do not recommend it blindly.** +It rebuilds every index on the table, including `pgsearch_embedding_hnsw`; on a +deployment with embeddings populated that is an HNSW rebuild over millions of +vectors — plausibly hours, under an ACCESS EXCLUSIVE lock — and it needs peak +disk for old plus new (~19 GB at 5M). Sites that want the space returned should +run it deliberately in a maintenance window knowing that cost, or use `pg_repack` +if they have it. This design requires neither. Most sites will not need it: an +archive that keeps growing refills the freed space on its own. An online alternative — resumable backfill command plus a feature flag on the query path — was considered and rejected: it doubles the query-layer code for From 9ec299f3819237310c267f65fd8a6317301ba8b6 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 14:46:56 +0000 Subject: [PATCH 03/50] Name the Postgres tuning env vars in the FTS projection spec Section 4.5 said the settings were "driven by env vars" without naming them, which is not implementable. Adds the four variable names, their defaults, the compose snippet they feed, and where they get documented. Also corrects a claim that max_parallel_workers and max_worker_processes must be raised alongside per_gather: at per_gather=4 the stock cap of 8 is not a throttle, it only binds under concurrency. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 39 ++++++++++++++++--- 1 file changed, 33 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 960e27f6..1cb3cac0 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -253,8 +253,32 @@ the duration and the flag has to be removed later anyway. ### 4.5 PostgreSQL configuration -Delivered as `-c` GUCs on the `postgres` service's `command:`, driven by env vars -with defaults documented in `example.env` and `CLAUDE.md`. +Delivered as `-c` GUCs on the `postgres` service's `command:` in +`docker-compose.base.yml`, substituted from `.env`: + +```yaml +postgres: + command: + - postgres + - -c + - max_parallel_workers_per_gather=${POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER:-4} + - -c + - max_parallel_workers=${POSTGRES_MAX_PARALLEL_WORKERS:-8} + - -c + - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} + - -c + - shared_buffers=${POSTGRES_SHARED_BUFFERS:-512MB} +``` + +| variable | default | stock | rationale | +| --- | --- | --- | --- | +| `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | 2 | measured 606 ms → 421 ms | +| `POSTGRES_MAX_PARALLEL_WORKERS` | `8` | 8 | unchanged default; exposed so larger hosts can raise all three coherently | +| `POSTGRES_MAX_WORKER_PROCESSES` | `8` | 8 | as above | +| `POSTGRES_SHARED_BUFFERS` | `512MB` | 128MB | hygiene against a multi-GB database | + +Only the first changes behaviour out of the box. All four go in `example.env` +with the guidance below, and in the environment-variable section of `CLAUDE.md`. Measured at 5M on `findings`: @@ -265,10 +289,13 @@ Measured at 5M on `findings`: | 6 | 379 ms | | 8 | 343 ms | -- **`max_parallel_workers_per_gather` = 4**, with `max_worker_processes` and - `max_parallel_workers` raised to match — all three are needed, since the - cluster-wide caps default to 8 and would otherwise throttle it. 4 captures most - of the available gain while leaving cores for concurrent searches. +- **`max_parallel_workers_per_gather` = 4** captures most of the available gain + while leaving cores for concurrent searches. The cluster-wide caps + (`max_parallel_workers`, `max_worker_processes`, both 8 by default) are *not* a + throttle at this value — 8 covers two concurrent parallel searches, and a third + degrades gracefully to fewer workers rather than queueing. They are exposed so + that an operator raising `per_gather` on a larger host raises all three + together, which is the only case where they bind. - **`shared_buffers`** raised from the stock 128 MB, as general hygiene against a multi-GB database. Labelled honestly: the 343 ms figure came from a warm *OS* page cache, so no specific gain is attributed to this. Guidance is ~25% of RAM. From fd19c3f729338fe87acd83a97803605b75e77530 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 14:53:52 +0000 Subject: [PATCH 04/50] Follow the project's env var convention for Postgres tuning knobs The four tuning variables are compose-level overrides with defaults, not example.env entries. This matches how the project already separates the two kinds: deployer choices (ports, EXAMPLE_REPORTS_LANGUAGE) live in example.env, while tuning knobs with sensible defaults are compose-only (EMBEDDINGS_WORKER_CONCURRENCY, WAIT_POSTGRES_TIMEOUT, RADIS_IMAGE). Operator guidance goes in docs/user-docs/admin-guide.md. Also drops the raised shared_buffers default back to PostgreSQL's 128MB. No measured gain was attributable to it, and 512MB of shared memory on a small container fails to start or thrashes -- the same reasoning that kept work_mem unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 26 ++++++++++++++----- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 1cb3cac0..7303c3ae 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -267,7 +267,7 @@ postgres: - -c - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} - -c - - shared_buffers=${POSTGRES_SHARED_BUFFERS:-512MB} + - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} ``` | variable | default | stock | rationale | @@ -275,10 +275,18 @@ postgres: | `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | 2 | measured 606 ms → 421 ms | | `POSTGRES_MAX_PARALLEL_WORKERS` | `8` | 8 | unchanged default; exposed so larger hosts can raise all three coherently | | `POSTGRES_MAX_WORKER_PROCESSES` | `8` | 8 | as above | -| `POSTGRES_SHARED_BUFFERS` | `512MB` | 128MB | hygiene against a multi-GB database | +| `POSTGRES_SHARED_BUFFERS` | `128MB` | 128MB | unchanged default; exposed because the right value is a fraction of the host's RAM | -Only the first changes behaviour out of the box. All four go in `example.env` -with the guidance below, and in the environment-variable section of `CLAUDE.md`. +Only the first changes behaviour out of the box; the other three keep PostgreSQL's +own defaults and exist so operators can raise them coherently. + +**None of them go in `example.env`.** The project already distinguishes the two +kinds of variable: things a deployer *chooses* are listed there (ports, +`EXAMPLE_REPORTS_LANGUAGE`, `REMOTE_DEBUGGING_PORT`), while tuning knobs with a +sensible default are compose-only overrides (`EMBEDDINGS_WORKER_CONCURRENCY`, +`WAIT_POSTGRES_TIMEOUT`, `RADIS_IMAGE`). These are the second kind. They are +documented in `docs/user-docs/admin-guide.md`, which is where operator-facing +guidance lives, and listed in the environment-variable section of `CLAUDE.md`. Measured at 5M on `findings`: @@ -296,9 +304,13 @@ Measured at 5M on `findings`: degrades gracefully to fewer workers rather than queueing. They are exposed so that an operator raising `per_gather` on a larger host raises all three together, which is the only case where they bind. -- **`shared_buffers`** raised from the stock 128 MB, as general hygiene against a - multi-GB database. Labelled honestly: the 343 ms figure came from a warm *OS* - page cache, so no specific gain is attributed to this. Guidance is ~25% of RAM. +- **`shared_buffers` default is left at PostgreSQL's own 128 MB.** Raising it is + standard advice for a multi-GB database, but the 343 ms figure came from a warm + *OS* page cache, so no measured gain can be attributed to it — and a raised + default would break small installs, where 512 MB of shared memory on a 1 GB + container fails to start or thrashes. Exposed as an override with ~25%-of-RAM + guidance in the admin guide; not changed by default, for the same reason + `work_mem` is not. - **`work_mem` deliberately not raised.** It mattered only because `.distinct()` forced an external merge sort; without it the sort is a 26 kB top-N heapsort. Raising it is per-connection-per-node memory risk for zero measured benefit. From 92470c247245d6d48c39fea2bb821c40c0cf9bc0 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 15:08:51 +0000 Subject: [PATCH 05/50] Fix date filter sargability and rework the projection indexes Django's __date lookup compiles to (col AT TIME ZONE 'Europe/Berlin')::date, which no plain btree can serve and which is evaluated per row -- five million timezone conversions per query on the sequential scan that dominates our plans. Converts the two date filters to half-open ranges with the boundaries computed in Python. Reworks the index set based on which callers actually populate which filters. Drops the modality_codes GIN (a quarter of the corpus, never selective enough to beat a scan) and adds a btree on report_updated_at, which subscriptions filter on with no tsquery to drive the query. Records that patient_id, created_after and created_before have no producer at all. Also corrects a factual error: array operators do not require a GIN index to be usable, only to be served by an index scan. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 49 ++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 7303c3ae..643f25d9 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -96,11 +96,31 @@ benefit, and `check_search_projection` guards against drift instead. unqualified `created_at` on this table would be ambiguous about whose timestamp it is. -Indexes: `GinIndex` on `group_ids` and `modality_codes` (the containment and -overlap operators are unusable without them), btree on `study_datetime`. Nothing -else — the remaining predicates ride the sequential scan, exactly as they do -today, since `reports_report` carries no index on `patient_id`, `patient_sex` or -`study_datetime` either. +Indexes — two, chosen from which callers actually populate which filters: + +- **`GinIndex` on `group_ids`.** Every search filters on it. It earns its keep + only where groups genuinely partition the corpus; in a single-group install it + matches every row and the planner will ignore it, which is fine. +- **btree on `report_updated_at`.** Subscriptions call `filter()` with + `updated_after` (`subscriptions/tasks.py`), which has no tsquery to drive it — + unindexed that is a full-table scan on every subscription refresh, once per + subscription. + +Deliberately **not** indexed: + +- **`modality_codes`.** With four or five modality codes a filter selects roughly + a quarter of the corpus, never selective enough to beat a sequential scan, and + GIN carries real write cost. Note array operators (`@>`, `&&`) evaluate fine + per row without any index; an index only enables *index scans*. +- **`study_datetime`.** Worth revisiting once §4.3's sargability fix lands, but a + date filter always co-occurs with a tsquery, so the planner will usually prefer + the scan regardless. Left out until a measurement justifies it. +- **`patient_id`, `report_created_at`.** No caller sets `patient_id`, + `created_after` or `created_before` — they exist on `SearchFilters` and are + handled in `_build_filter_query`, but nothing populates them. The columns are + still needed for the filter code; indexes are not. +- **`patient_sex`, `patient_age`, `language_code`, `study_description`.** Low + selectivity, or `icontains`, which a btree cannot serve anyway. `body`, `pacs_name` and `document_id` stay on `Report`. @@ -160,10 +180,27 @@ after a restore or bulk import. | `Q(report__groups=filters.group)` | `Q(group_ids__contains=[filters.group])` | | `Q(report__language__code=...)` | `Q(language_code=...)` | | `Q(report__modalities__code__in=...)` | `Q(modality_codes__overlap=...)` | -| `Q(report__patient_sex=...)`, age, dates, description, patient_id | same, unprefixed | +| `Q(report__patient_sex=...)`, age, description, patient_id | same, unprefixed | +| `Q(report__study_datetime__date__gte=)` | `Q(study_datetime__gte=)` | +| `Q(report__study_datetime__date__lte=)` | `Q(study_datetime__lt=)` | | `Q(report__created_at__gte=...)` | `Q(report_created_at__gte=...)` | | labels | unchanged — `report_id__in=`, no join | +**The date filters become half-open ranges.** Django's `__date` lookup compiles +to `(study_datetime AT TIME ZONE 'Europe/Berlin')::date >= …` — a function over +the column, which no plain btree can serve, and which is evaluated once per row. +On the sequential scan that dominates our plans that is five million timezone +conversions and date casts per query, for a predicate that is equivalent to a +comparison against two timestamps. + +Computing the day boundaries in Python instead is exactly equivalent in meaning, +removes the per-row conversion, and makes the predicate sargable. A functional +index matching Django's expression was rejected: it would restore sargability but +leave the per-row cost untouched on the seq-scan path, and it hardcodes a +timezone that breaks silently if `TIME_ZONE` changes. Storing a precomputed local +`study_date` column was rejected for baking `TIME_ZONE` into data, which a +timezone change would then require a backfill to correct. + **`.distinct()` is removed unconditionally.** It exists only because the `groups` and `modalities` joins can duplicate rows. With no joins there is nothing to deduplicate. Worth ~30% on its own (1.44 s → 1.04 s at 1M). From ed2fe00bf69d0c682dca76222b7efcbdb4a95118 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 15:16:49 +0000 Subject: [PATCH 06/50] Final review pass over the FTS projection spec Fixes found by re-reading it end to end and verifying its claims: - The statement-level trigger justification cited bulk_upsert_report_search_indexes, which writes to a different table and never fires these triggers. The real bulk writer is the bulk-upsert API endpoint (reports/api/viewsets.py:215,229). The matching test in section 5 described a groups.set() "spanning many reports", which is not a thing a related manager can do. - Section 4.4 still claimed three new indexes after 4.1 was cut to two. - Verified the BEFORE-trigger trap rather than asserting it: a BEFORE trigger reads NULL from a stored generated column whose value was 42. - Verified the production pool size on the new shape: LIMIT 10000 runs in 401 ms against 421 ms for LIMIT 25, so HYBRID_FTS_MAX_RESULTS staying at 10,000 is now measured on both query shapes rather than only the old one. - Only the modalities join can duplicate rows; the groups join cannot, since the filter is a single value against a unique_together table. - Records two edges the triggers do not cover: Report deletion cascades and Language.code renames. - Normalises headline figures to the configured 4-worker default rather than the 8-worker best case, and softens an overstatement about the CTE alternative being the only way to avoid duplication. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 77 +++++++++++++------ 1 file changed, 52 insertions(+), 25 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 643f25d9..ad4e0b96 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -37,9 +37,9 @@ FTS candidate query joins `reports_report`, `reports_report_groups` and `reports_language`, then applies `SELECT DISTINCT`. Two consequences: - The planner builds a hash table over the matched rows **carrying - `search_vector`**, because `ts_rank` above the join still needs it. On the 1M rig (§8), - 730k matches, that hash table was 1.2 GB and spilled into 256 batches, - writing ~1.2 GB of temp files at the default `work_mem`. + `search_vector`**, because `ts_rank` above the join still needs it. On the 1M + rig (§8) with 730k matches that hash table was 1.2 GB, spilling into 256 + batches and writing ~1.2 GB of temp files at the default `work_mem`. - `SELECT DISTINCT` blocks the top-N heapsort, forcing a full external merge sort of every matching row before `LIMIT` can apply. @@ -58,7 +58,7 @@ Recorded because each is the obvious question a reviewer will raise. | Lower `HYBRID_FTS_MAX_RESULTS` | No effect (1M rig). `LIMIT 100` and `LIMIT 10000` both 1.05 s. Under `ts_rank` the pool size is free; it only becomes load-bearing with a pruning index. | | Raise `work_mem` | No effect (1M rig) (1.41 s at 256 MB vs 1.44 s at 4 MB). The spill is a symptom, not the cause. | | Rewrite the joins as `EXISTS` | Worse: 1.55 s vs 1.05 s (1M rig). The planner still hashes. | -| Materialised id-only CTE, joins on narrow rows, then rank | Worse: 11,460 ms for `findings`, 3,089 ms for `pneumonia` at 5M. The joins are still over 5,001,000 rows. **This was the only way to avoid duplicating access-control data, and it does not work.** | +| Materialised id-only CTE, joins on narrow rows, then rank | Worse: 11,460 ms for `findings`, 3,089 ms for `pneumonia` at 5M. The joins are still over 5,001,000 rows. **This was the only way to avoid duplicating access-control data without changing the domain model (see §6), and it does not work.** | | `strip()`ped second tsvector for ranking | Rejected (1M rig). 1.4x faster but collapses `ts_rank` to **1 distinct value** across 200,000 matching rows — `strip()` removes the positions that encode term frequency, so the ordering becomes a no-op. | | Restricting recall ("rank only the most recent N", "refuse broad queries") | Withdrawn. No production search engine restricts recall; they restrict pagination depth and count precision. Not needed here anyway. | @@ -112,9 +112,11 @@ Deliberately **not** indexed: a quarter of the corpus, never selective enough to beat a sequential scan, and GIN carries real write cost. Note array operators (`@>`, `&&`) evaluate fine per row without any index; an index only enables *index scans*. -- **`study_datetime`.** Worth revisiting once §4.3's sargability fix lands, but a - date filter always co-occurs with a tsquery, so the planner will usually prefer - the scan regardless. Left out until a measurement justifies it. +- **`study_datetime`.** §4.3 converts the date filters to half-open ranges, which + would make an index on this column *usable* — but the point of that change is + removing five million per-row timezone conversions, and it stands on its own + without an index. A date filter always co-occurs with a tsquery, so the planner + will usually prefer the scan anyway. Left out until a measurement justifies it. - **`patient_id`, `report_created_at`.** No caller sets `patient_id`, `created_after` or `created_before` — they exist on `SearchFilters` and are handled in `_build_filter_query`, but nothing populates them. The columns are @@ -140,10 +142,12 @@ Three **statement-level** triggers using transition tables: | `reports_report_groups` | AFTER INSERT, DELETE | `group_ids` | | `reports_report_modalities` | AFTER INSERT, DELETE | `modality_codes` | -Statement-level, not row-level: `bulk_upsert_report_search_indexes` writes in -5000-row chunks, and row-level triggers would fire 5000 times per statement. -With `REFERENCING NEW TABLE AS ...` each statement costs one set-based -`UPDATE ... FROM`. +Statement-level, not row-level: the bulk-upsert API endpoint writes membership +rows with `group_through.objects.bulk_create(...)` and +`modality_through.objects.bulk_create(...)` (`reports/api/viewsets.py:215,229`), +so one statement can carry a whole batch. Row-level triggers would fire once per +row; with `REFERENCING NEW TABLE AS ...` each statement costs one set-based +`UPDATE ... FROM` instead. The `reports_report` trigger guards its write with `IS DISTINCT FROM` across the mirrored columns so unrelated `Report.save()` calls do not rewrite projection @@ -166,6 +170,13 @@ installed via `RunSQL` in `reports/migrations/0011`. generated column values to BEFORE triggers, so a BEFORE trigger would mirror `patient_age` as NULL. +**Two edges the triggers deliberately do not cover.** Deleting a `Report` +cascades to both the membership rows and the projection row; the resulting +membership-delete trigger is a zero-row `UPDATE` against an already-deleted +projection row, which is harmless in either cascade order. And `Language.code` is +treated as immutable — no trigger watches `reports_language`, so renaming a code +would strand `language_code` values until the backfill is re-run. + **Reconciliation.** A `check_search_projection` management command verifies the projection against its sources with one set-based query per column and reports drift. Cheap insurance on access-control data, and something operators can run @@ -201,8 +212,10 @@ timezone that breaks silently if `TIME_ZONE` changes. Storing a precomputed loca `study_date` column was rejected for baking `TIME_ZONE` into data, which a timezone change would then require a backfill to correct. -**`.distinct()` is removed unconditionally.** It exists only because the `groups` -and `modalities` joins can duplicate rows. With no joins there is nothing to +**`.distinct()` is removed unconditionally.** It exists because the `modalities` +join duplicates a report that matches on more than one modality (the `groups` +join cannot, since `filters.group` is a single value against a `unique_together` +through table). With no joins there is nothing to deduplicate. Worth ~30% on its own (1.44 s → 1.04 s at 1M). **Access-control edge case.** `filters.group=None` is documented as fail-closed: @@ -215,7 +228,9 @@ gets its own test. traverse `report__language__code` and become `language_code` on the row. The vector half also sheds its `.distinct()` (6.5 ms → 1.0 ms). -The FTS candidate query then compiles to the shape measured at 404 ms: +The FTS candidate query then compiles to the shape below, measured at **401 ms** +with `LIMIT 10000` and `max_parallel_workers_per_gather = 4` — confirming that +the production pool size costs nothing on this shape, not just on the old one: ```sql SELECT report_id, @@ -260,7 +275,7 @@ untouched — `page_rows` still `select_related("report")` for `ts_headline` ove are unindexed the backfill can use HOT updates, whose old row versions are reclaimed by page pruning without index cleanup. That is the main defence against the bloat below, so this ordering must not be changed. The `ANALYZE` - is required, not hygiene: ten new columns and three new indexes carry no + is required, not hygiene: ten new columns and two new indexes carry no statistics, and this design depends on the planner choosing a parallel sequential scan with a top-N heapsort. @@ -269,7 +284,9 @@ in the `init` service and `web` waits on it, so the backfill blocks the deploy rather than serving half-projected rows — the right failure mode for access-control data. Deploy time therefore grows with corpus size: an estimated 2–5 minutes at 5M, extrapolated from analogous full-table writes (44 s for a -4M-row m2m insert, 83 s for a 4M-row tsvector update). The backfill rewrites every row: measured, the tsvectors average 1220 bytes and +4M-row m2m insert, 83 s for a 4M-row tsvector update). + +The backfill rewrites every row: measured, the tsvectors average 1220 bytes and nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the new columns. Left unchecked that is ~6 GB of dead tuples at 5M, and since the hot path is a sequential scan it reads them — roughly halving the gain @@ -376,9 +393,10 @@ combinations. Highest-value test in the change: it catches still updates the projection. This test is the justification for choosing triggers over signals and should exist to defend that decision. -**Bulk correctness:** a `groups.set()` spanning many reports in one statement -updates all of them. Statement-level triggers have a classic failure mode where -only one row is processed; this pins it. +**Bulk correctness:** a `bulk_create` of many rows into the `groups` through +model — the shape the bulk-upsert endpoint uses — updates the projection for +*every* affected report. Statement-level triggers have a classic failure mode +where only one row of the transition table is processed; this pins it. **No duplicates without `.distinct()`:** a report in several matching groups, or with several matching modalities, appears exactly once. @@ -389,8 +407,10 @@ plan contains no join against `reports_report_groups` and no `Unique` node. This deterministically catches someone reintroducing a `report__` traversal and silently restoring the 13-second path. -**`patient_age` is mirrored non-null**, which fails if the triggers are ever -converted to BEFORE. +**`patient_age` mirrors `Report.patient_age` exactly.** Verified that this is +the trap it guards: a BEFORE trigger on a stored generated column reads `NULL` +where the committed value was 42, so converting the triggers to BEFORE would +silently null this column out. Existing search, extraction and subscription tests are the regression net for the four consumers and should pass unchanged. @@ -407,16 +427,20 @@ four consumers and should pass unchanged. `reports_report`. What BM25 still offers that PostgreSQL FTS cannot: IDF, term frequency saturation and length normalisation. - **#287 impact-ordered lexeme ranks.** Likely unnecessary once plain `ts_rank` - handles 5M matches in 343 ms. Recommendation only; closing it is its author's + handles 5M matches in ~400 ms. Recommendation only; closing it is its author's call. -- **`HYBRID_FTS_MAX_RESULTS`** stays at 10,000 — measured free under `ts_rank`. +- **`HYBRID_FTS_MAX_RESULTS`** stays at 10,000. Measured free under `ts_rank` on + both shapes: `LIMIT 100` and `LIMIT 10000` were indistinguishable on the old + joined query, and `LIMIT 25` (421 ms) and `LIMIT 10000` (401 ms) are + indistinguishable on the new single-table one. It would only become + load-bearing with a pruning index. - **`Report.groups` → `ArrayField`.** Would remove the redundancy altogether and speed up `Report.objects.filter(groups=...)` elsewhere, at the cost of FK integrity on an access-control field. A domain change, deliberately not bundled into a search-performance change. - **Result-page N+1s.** Measured at ~134 SQL queries per 25-result page (`document.full_report` twice per row via `search/site.py`, `can_view_report`, - the collections count, the notes lookup). Once the scan is 343 ms this is + the collections count, the notes lookup). Once the scan is ~400 ms this is proportionally significant, and it is the recommended next piece of work — but it is a different subsystem. - **#282 (`hnsw.ef_search`) and #284 (fused result cache).** Independent, @@ -436,7 +460,10 @@ four consumers and should pass unchanged. ## 8. Measurement rig -All figures above come from two throwaway corpora built from +The BM25 figures in §6 additionally required a custom PostgreSQL image +(`pgvector/pgvector:pg17` plus `pg_textsearch` v1.4.0), and the vector-half +figures come from a third throwaway corpus of 200,000 rows with 1024-dimension +embeddings and an HNSW index. Everything else comes from two corpora built from `samples/reports_en.json` (1000 real-shaped report bodies, cycled with a unique md5 suffix per report): From f6d477aa6dc6191af2642f89e42648dbccdcfc5a Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 15:23:44 +0000 Subject: [PATCH 07/50] Measure the group_ids GIN index instead of assuming it Split the 5M benchmark corpus 94.9/5/0.1 across three groups and compared the planner's choice against the same query with index scans forced off. The index gives ~170x at 0.1% selectivity and ~2.9x at 5%, and makes no difference at 94.9% -- where the planner abandons it unprompted, so it cannot make the degenerate single-group install worse. It costs 6 MB and 1.2 s to build. Also labels the report_updated_at btree as reasoned rather than measured, with a note to confirm it during implementation using the same technique. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 27 ++++++++++++++----- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index ad4e0b96..cf612bc6 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -98,13 +98,26 @@ it is. Indexes — two, chosen from which callers actually populate which filters: -- **`GinIndex` on `group_ids`.** Every search filters on it. It earns its keep - only where groups genuinely partition the corpus; in a single-group install it - matches every row and the planner will ignore it, which is fine. -- **btree on `report_updated_at`.** Subscriptions call `filter()` with - `updated_after` (`subscriptions/tasks.py`), which has no tsquery to drive it — - unindexed that is a full-table scan on every subscription refresh, once per - subscription. +- **`GinIndex` on `group_ids` — measured.** On the 5M rig with the corpus split + 94.9% / 5% / 0.1% across three groups, comparing the planner's choice against + the same query with index scans forced off: + + | group holds | with index | forced scan | gain | + | --- | --- | --- | --- | + | 0.1% (5,001 reports) | 2–3 ms | 345 ms | ~170x | + | 5% (250,050) | 119–141 ms | 342–357 ms | ~2.9x | + | 94.9% (4,745,949) | 448–452 ms | 444–487 ms | none | + + It pays wherever a group holds a minority of the corpus, and costs nothing + where it does not — at 94.9% the planner abandons it unprompted. Carrying cost + is negligible: 6 MB, 1.2 s to build, because a handful of distinct group ids + means a handful of posting lists. +- **btree on `report_updated_at` — reasoned, not measured.** Subscriptions call + `filter()` with `updated_after` (`subscriptions/tasks.py`), which has no + tsquery to compete with, so a selective range over a timestamp is the textbook + index case; unindexed it is a full-table scan on every subscription refresh, + once per subscription. Worth confirming during implementation with the same + force-the-plan technique used for `group_ids` above. Deliberately **not** indexed: From 5890c830a97b1a6b284bab609827531fb3b715a5 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 18:13:16 +0000 Subject: [PATCH 08/50] Re-measure at the 8M design target and correct three claims Grew the rig to 8,001,000 reports, added the real projection columns, ran the actual chunked backfill and measured before and after. Three things the spec had wrong: - The headline overstated the gain. "13,440 ms -> ~420 ms" compared the old shape at two workers against the new shape at four to eight, so part of that ratio was parallelism and part was a smaller corpus. The controlled comparison at 8M -- same table, same four workers -- is 8,388 ms -> 882 ms, about 9.5x. - The backfill estimate of "2-5 minutes at 5M" was low. Measured 9 min 20 s at 8M, extrapolating to ~18 min at 15M. Ten minutes of announced downtime has been accepted, which is what keeps the blocking approach. - Chunking and index-last do not prevent the heap doubling. Measured 10 GB -> 21 GB, and a plain VACUUM clears all dead tuples while leaving the file at 21 GB for 11 GB of live data. The 882 ms already includes that penalty. Also states 8 million as the explicit design target and records the benchmark host relative to production. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 62 +++++++++++++------ 1 file changed, 44 insertions(+), 18 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index cf612bc6..c1ef87dd 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -3,6 +3,9 @@ **Status:** design, not yet implemented **Branch:** `fts-query-shape-performance` **Supersedes:** parts of `hybrid-search.md` §7 (the FTS candidate query and its filter construction) +**Design target:** 8 million reports. The largest known deployment is ~15M; past +roughly 8M the working set stops fitting comfortably in page cache on ordinary +hardware and the conclusions in §2 need re-measuring (see §8). ## 1. Problem @@ -44,10 +47,14 @@ FTS candidate query joins `reports_report`, `reports_report_groups` and of every matching row before `LIMIT` can apply. Removing both makes the query a single-table parallel sequential scan with a -26 kB top-N heapsort. On the same 5M corpus, `findings` goes from **13,440 ms to -~600 ms** at stock parallelism, and to **~420 ms** with -`max_parallel_workers_per_gather` raised to 4 (§4.5). The plan verified in §4.3 -ran in 404 ms at 8 workers. +26 kB top-N heapsort. Measured at the 8M design target — same table, same four +workers on both sides — `findings` goes from **8,388 ms to 882 ms, about 9.5x**. + +Parallelism is a separate and additive lever worth a further ~1.4x (§4.5). An +earlier draft of this document claimed 33x by comparing the old shape at two +workers against the new shape at eight, crediting the query-shape change with a +gain that was partly parallelism and partly a smaller corpus. The controlled +comparison above is the one to trust. ## 3. Ruled out, with measurements @@ -295,15 +302,28 @@ untouched — `page_rows` still `select_related("report")` for `ts_headline` ove **Deployment consequences.** `docker-compose.prod.yml` runs `manage.py migrate` in the `init` service and `web` waits on it, so the backfill blocks the deploy rather than serving half-projected rows — the right failure mode for -access-control data. Deploy time therefore grows with corpus size: an estimated -2–5 minutes at 5M, extrapolated from analogous full-table writes (44 s for a -4M-row m2m insert, 83 s for a 4M-row tsvector update). - -The backfill rewrites every row: measured, the tsvectors average 1220 bytes and -nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not -just the new columns. Left unchecked that is ~6 GB of dead tuples at 5M, and -since the hot path is a sequential scan it reads them — roughly halving the gain -until the space is reused. Steps 3 and 4 are what keep it below that. +access-control data. Deploy time therefore grows with corpus size: **measured at +9 min 20 s for 8M reports**, in 50,000-row chunks. That extrapolates to roughly +18 minutes at 15M. Adding the ten columns is genuinely free (1.3 ms, confirming +the metadata-only claim); the nine minutes is the row rewrite. + +Roughly ten minutes of announced downtime for a one-time migration has been +accepted for this deployment, which is what keeps the blocking approach below. +A site that cannot take that window needs the online alternative instead. + +The backfill rewrites every row: the tsvectors average 1220 bytes and nothing is +TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the +new columns. Measured at 8M, the heap goes **10 GB → 21 GB** with 7,988,672 dead +tuples, and a plain `VACUUM` clears every dead tuple while leaving the file at +**21 GB** for ~11 GB of live data. + +Chunking and creating indexes last (steps 3 and 4) did **not** prevent this — the +rows grow by ~120 bytes, so most updated tuples cannot fit back into their +original page and HOT does not apply. The doubling should be treated as expected, +not as something the migration avoids. The consequence is a sequential scan +reading roughly twice what it needs to, and the 882 ms in §2 already includes that +penalty; on a compacted table the same query would land nearer 450–650 ms. It +resolves on its own as the archive grows into the free space. **Do not run `VACUUM FULL` from the migration, and do not recommend it blindly.** It rebuilds every index on the table, including `pgsearch_embedding_hnsw`; on a @@ -440,7 +460,7 @@ four consumers and should pass unchanged. `reports_report`. What BM25 still offers that PostgreSQL FTS cannot: IDF, term frequency saturation and length normalisation. - **#287 impact-ordered lexeme ranks.** Likely unnecessary once plain `ts_rank` - handles 5M matches in ~400 ms. Recommendation only; closing it is its author's + handles 8M matches in ~880 ms. Recommendation only; closing it is its author's call. - **`HYBRID_FTS_MAX_RESULTS`** stays at 10,000. Measured free under `ts_rank` on both shapes: `LIMIT 100` and `LIMIT 10000` were indistinguishable on the old @@ -453,7 +473,7 @@ four consumers and should pass unchanged. into a search-performance change. - **Result-page N+1s.** Measured at ~134 SQL queries per 25-result page (`document.full_report` twice per row via `search/site.py`, `can_view_report`, - the collections count, the notes lookup). Once the scan is ~400 ms this is + the collections count, the notes lookup). Once the scan is ~880 ms this is proportionally significant, and it is the recommended next piece of work — but it is a different subsystem. - **#282 (`hnsw.ef_search`) and #284 (fused result cache).** Independent, @@ -463,8 +483,9 @@ four consumers and should pass unchanged. ## 7. Success criteria -- `findings` (5,001,000 matches) drops from 13,440 ms to under 500 ms at the - SQL level on the reference corpus, with `max_parallel_workers_per_gather` = 4. +- At the 8M design target with `max_parallel_workers_per_gather` = 4, the FTS + candidate query drops from ~8,400 ms to under 1,000 ms immediately after the + backfill, bloat included (measured: 8,388 ms → 882 ms). - The FTS candidate query plan contains no join to `reports_report_groups` and no `Unique` node. - `check_search_projection` reports zero drift after the backfill and after a @@ -480,12 +501,17 @@ embeddings and an HNSW index. Everything else comes from two corpora built from `samples/reports_en.json` (1000 real-shaped report bodies, cycled with a unique md5 suffix per report): +- **8M rig (the design target):** the 5M rig grown to 8,001,000 reports, 10 GB + `pgsearch_reportsearchindex` heap before the backfill and 21 GB after. All §2 + and §7 figures come from here. - **5M rig:** the dev stack database, 5,001,000 reports, ~15 GB, 6.3 GB `pgsearch_reportsearchindex` heap, average tsvector 1220 bytes. - **1M rig:** a standalone container, 1,000,000 reports, 730,000 matching `pneumonia`, average tsvector 1607 bytes. -Both on stock PostgreSQL 17 settings (`shared_buffers` 128 MB, `work_mem` 4 MB, +The benchmark host is a 31 GB Linux VM; the production target is a 128 GB M4 Max, +so an 11 GB live working set stays cached on both and these figures are a floor +rather than a ceiling. All rigs on stock PostgreSQL 17 settings (`shared_buffers` 128 MB, `work_mem` 4 MB, `max_parallel_workers_per_gather` 2) on a 16-core host, unless a measurement states otherwise. From 0986e1ca24fb50c794c08cb96f377699aacdf88f Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 18:13:54 +0000 Subject: [PATCH 09/50] Label the remaining figures with the rig they came from Two timings in sections 4.3 and 6 were 5M-rig numbers sitting next to 8M-rig ones with nothing to tell them apart, now that the spec has an explicit 8M design target. Also tidies a doubled parenthetical. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...2026-08-29-fts-query-shape-performance-design.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index c1ef87dd..ab9729b6 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -63,7 +63,7 @@ Recorded because each is the obvious question a reviewer will raise. | candidate | result | | --- | --- | | Lower `HYBRID_FTS_MAX_RESULTS` | No effect (1M rig). `LIMIT 100` and `LIMIT 10000` both 1.05 s. Under `ts_rank` the pool size is free; it only becomes load-bearing with a pruning index. | -| Raise `work_mem` | No effect (1M rig) (1.41 s at 256 MB vs 1.44 s at 4 MB). The spill is a symptom, not the cause. | +| Raise `work_mem` | No effect on the 1M rig: 1.41 s at 256 MB vs 1.44 s at 4 MB. The spill is a symptom, not the cause. | | Rewrite the joins as `EXISTS` | Worse: 1.55 s vs 1.05 s (1M rig). The planner still hashes. | | Materialised id-only CTE, joins on narrow rows, then rank | Worse: 11,460 ms for `findings`, 3,089 ms for `pneumonia` at 5M. The joins are still over 5,001,000 rows. **This was the only way to avoid duplicating access-control data without changing the domain model (see §6), and it does not work.** | | `strip()`ped second tsvector for ranking | Rejected (1M rig). 1.4x faster but collapses `ts_rank` to **1 distinct value** across 200,000 matching rows — `strip()` removes the positions that encode term frequency, so the ordering becomes a no-op. | @@ -248,9 +248,10 @@ gets its own test. traverse `report__language__code` and become `language_code` on the row. The vector half also sheds its `.distinct()` (6.5 ms → 1.0 ms). -The FTS candidate query then compiles to the shape below, measured at **401 ms** -with `LIMIT 10000` and `max_parallel_workers_per_gather = 4` — confirming that -the production pool size costs nothing on this shape, not just on the old one: +The FTS candidate query then compiles to the shape below, measured on the 5M rig +at **401 ms** with `LIMIT 10000` and `max_parallel_workers_per_gather = 4` — +confirming the production pool size costs nothing on this shape, not just the old +one: ```sql SELECT report_id, @@ -464,8 +465,8 @@ four consumers and should pass unchanged. call. - **`HYBRID_FTS_MAX_RESULTS`** stays at 10,000. Measured free under `ts_rank` on both shapes: `LIMIT 100` and `LIMIT 10000` were indistinguishable on the old - joined query, and `LIMIT 25` (421 ms) and `LIMIT 10000` (401 ms) are - indistinguishable on the new single-table one. It would only become + joined query, and on the 5M rig `LIMIT 25` (421 ms) and `LIMIT 10000` (401 ms) + are indistinguishable on the new single-table one. It would only become load-bearing with a pruning index. - **`Report.groups` → `ArrayField`.** Would remove the redundancy altogether and speed up `Report.objects.filter(groups=...)` elsewhere, at the cost of FK From 9520593ac77f576e9cc0ddf49ec13f4770259982 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 18:31:49 +0000 Subject: [PATCH 10/50] Add implementation plan for the FTS search projection Ten TDD tasks from the approved spec: projection columns, statement-level triggers, creation-path population, chunked backfill, indexes, drift detection, the single-table filter query, language predicates, a plan-shape regression test, and Postgres tuning. Splits the spec's single migration into four (0003-0006), one per ordered step, so no task has to edit a migration a previous task already applied. Order is load-bearing and unchanged: columns, triggers, backfill, indexes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../plans/2026-08-30-fts-search-projection.md | 1370 +++++++++++++++++ 1 file changed, 1370 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-30-fts-search-projection.md diff --git a/docs/superpowers/plans/2026-08-30-fts-search-projection.md b/docs/superpowers/plans/2026-08-30-fts-search-projection.md new file mode 100644 index 00000000..8bfee5c6 --- /dev/null +++ b/docs/superpowers/plans/2026-08-30-fts-search-projection.md @@ -0,0 +1,1370 @@ +# FTS Search Projection Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the full-text search candidate query single-table by completing `ReportSearchIndex` as a search projection, so it stops joining three tables and applying `SELECT DISTINCT`. + +**Architecture:** `ReportSearchIndex` already mirrors derived data (`search_vector`, `embedding`). This adds ten more mirrored columns — the fields search *filters* on — maintained by statement-level PostgreSQL triggers rather than application signals, because `group_ids` is access-control data and must stay correct for every writer including raw SQL. `_build_filter_query` then becomes a single-table predicate builder and `.distinct()` disappears. + +**Tech Stack:** Django 6.0, PostgreSQL 17, pytest + pytest-django, factory-boy. + +**Spec:** `docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md` + +## Global Constraints + +- **Design target: 8 million reports.** Measured there: `8,388 ms → 882 ms`. +- **Access control is the risk surface.** `group_ids` decides who may read a patient's report. Any change touching it needs a test. +- **`filters.group=None` is fail-closed** and must match only reports in no group — `Q(group_ids=[])`, never a `contains` variant. +- **Triggers must be `AFTER`, never `BEFORE`.** PostgreSQL does not expose stored generated column values to BEFORE triggers, so `patient_age` would mirror as NULL. Verified. +- **Migration order is load-bearing:** columns → triggers → backfill → indexes. Triggers before backfill so concurrent writes during a ~9-minute backfill still land. +- Line length 100 (Ruff), Google Python style, type checking pyright basic. +- Run tests with `uv run cli test -- `. + +**Deviation from the spec, deliberate:** §4.4 describes one migration. This plan splits it into four (`0003`–`0006`), one per ordered step. Same operations, same order — but each is independently applicable and testable, and no task has to edit a migration a previous task already applied. + +## File Structure + +| File | Responsibility | +| --- | --- | +| `radis/pgsearch/models.py` | Add the ten projection fields + two indexes to `ReportSearchIndex` | +| `radis/pgsearch/utils/projection.py` | **New.** The canonical projection SQL and `sync_projection(report_ids)`; single source of truth for the write path | +| `radis/pgsearch/migrations/0003_search_projection_columns.py` | **New.** `AddField` ×10 | +| `radis/pgsearch/migrations/0004_search_projection_triggers.py` | **New.** Three trigger functions + five triggers | +| `radis/pgsearch/migrations/0005_search_projection_backfill.py` | **New.** Chunked backfill, `atomic = False` | +| `radis/pgsearch/migrations/0006_search_projection_indexes.py` | **New.** Two indexes + GIN `fastupdate=off` | +| `radis/pgsearch/management/commands/check_search_projection.py` | **New.** Drift detection | +| `radis/pgsearch/providers.py` | `_build_filter_query`, `match_q`, `rank_expr`, `summary_expr`, `_exclude_negations` go single-table | +| `radis/pgsearch/signals.py` | Populate the projection when a row is created | +| `radis/pgsearch/utils/indexing.py` | Same, for the bulk path | +| `radis/pgsearch/tests/test_search_projection.py` | **New.** Columns, triggers, creation paths, backfill | +| `radis/pgsearch/tests/test_filter_query_equivalence.py` | **New.** Reference-oracle equivalence + plan shape | +| `docker-compose.base.yml` | Postgres tuning GUCs | +| `docs/user-docs/admin-guide.md` | Operator guidance for the GUCs | + +--- + +### Task 1: Projection columns + +**Files:** +- Modify: `radis/pgsearch/models.py` +- Create: `radis/pgsearch/migrations/0003_search_projection_columns.py` +- Test: `radis/pgsearch/tests/test_search_projection.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: `ReportSearchIndex.group_ids: list[int]`, `.modality_codes: list[str]`, `.language_code: str | None`, `.patient_sex: str | None`, `.patient_age: int | None`, `.patient_id: str | None`, `.study_datetime: datetime | None`, `.study_description: str | None`, `.report_created_at: datetime | None`, `.report_updated_at: datetime | None`. + +- [ ] **Step 1: Write the failing test** + +Create `radis/pgsearch/tests/test_search_projection.py`: + +```python +"""Tests for the ReportSearchIndex search projection. + +The projection mirrors the Report fields that search filters on, so the FTS +candidate query can stay single-table. group_ids is access-control data, so +its correctness has its own tests here. +""" + +import pytest +from adit_radis_shared.accounts.factories import GroupFactory + +from radis.pgsearch.models import ReportSearchIndex +from radis.reports.factories import LanguageFactory, ReportFactory +from radis.reports.models import Report + +pytestmark = pytest.mark.django_db + + +def test_new_index_row_defaults_to_empty_arrays(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + index = ReportSearchIndex.objects.get(report=report) + + assert index.group_ids == [] + assert index.modality_codes == [] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: FAIL with `AttributeError: 'ReportSearchIndex' object has no attribute 'group_ids'` + +- [ ] **Step 3: Add the fields to the model** + +In `radis/pgsearch/models.py`, add to `ReportSearchIndex` after the `embedding` field: + +```python + # Search projection: mirrors of the Report fields the scan filters on, so + # the FTS candidate query stays single-table. Maintained by the triggers in + # migration 0004 and populated on creation by signals.py / indexing.py. + # Nullable so every AddField stays metadata-only on a large table; + # check_search_projection guards against drift instead of a NOT NULL scan. + group_ids = ArrayField(models.IntegerField(), default=list) + modality_codes = ArrayField(models.CharField(max_length=16), default=list) + language_code = models.CharField(max_length=10, null=True) + patient_sex = models.CharField(max_length=1, null=True) + patient_age = models.IntegerField(null=True) + patient_id = models.CharField(max_length=64, null=True) + study_datetime = models.DateTimeField(null=True) + study_description = models.CharField(max_length=64, blank=True, null=True) + report_created_at = models.DateTimeField(null=True) + report_updated_at = models.DateTimeField(null=True) +``` + +Add the import at the top of the file: + +```python +from django.contrib.postgres.fields import ArrayField +``` + +- [ ] **Step 4: Generate the migration** + +Run: `docker exec radis_dev-web-1 ./manage.py makemigrations pgsearch --name search_projection_columns` + +Verify the generated `radis/pgsearch/migrations/0003_search_projection_columns.py` contains ten `AddField` operations and no `AlterField`. Every field must have either a constant default or `null=True` — that is what keeps each `AddField` metadata-only on an 8M-row table. + +- [ ] **Step 5: Run test to verify it passes** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add radis/pgsearch/models.py radis/pgsearch/migrations/0003_search_projection_columns.py radis/pgsearch/tests/test_search_projection.py +git commit -m "Add search projection columns to ReportSearchIndex" +``` + +--- + +### Task 2: Triggers that keep the projection true + +**Files:** +- Create: `radis/pgsearch/migrations/0004_search_projection_triggers.py` +- Modify: `radis/pgsearch/tests/test_search_projection.py` + +**Interfaces:** +- Consumes: the columns from Task 1. +- Produces: SQL functions `pgsearch_sync_group_ids()`, `pgsearch_sync_modality_codes()`, `pgsearch_sync_report_fields()`; triggers `pgsearch_group_ids_ins`/`_del`, `pgsearch_modality_codes_ins`/`_del`, `pgsearch_report_fields_upd`. + +- [ ] **Step 1: Write the failing tests** + +Append to `radis/pgsearch/tests/test_search_projection.py`: + +```python +from django.db import connection + + +def test_adding_a_group_updates_the_projection(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + + report.groups.add(group) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + + +def test_removing_a_group_updates_the_projection(): + """The leak direction: a report removed from a group must stop being visible.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + report.groups.add(group) + + report.groups.remove(group) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [] + + +def test_raw_sql_membership_write_updates_the_projection(): + """The whole reason for triggers over m2m_changed: writers that bypass the ORM.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + + with connection.cursor() as cursor: + cursor.execute( + "INSERT INTO reports_report_groups (report_id, group_id) VALUES (%s, %s)", + [report.pk, group.pk], + ) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + + +def test_bulk_membership_insert_updates_every_affected_row(): + """Statement-level triggers fail classically by processing only one transition row. + + This is the shape reports/api/viewsets.py:229 uses for bulk upsert. + """ + language = LanguageFactory.create(code="en") + reports = [ReportFactory.create(language=language) for _ in range(5)] + group = GroupFactory.create() + + through = Report.groups.through + through.objects.bulk_create( + [through(report_id=report.pk, group_id=group.pk) for report in reports] + ) + + for report in reports: + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk], f"report {report.pk} was not updated" + + +def test_deleting_a_report_does_not_error(): + """Deleting a Report cascades to both the membership rows and the index row, + firing the membership trigger against a row that may already be gone. The + spec calls that harmless in either cascade order; this pins it.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + report_pk = report.pk + + report.delete() + + assert not ReportSearchIndex.objects.filter(report_id=report_pk).exists() + + +def test_updating_a_report_updates_the_mirrored_scalars(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + + report.patient_id = "CHANGED-123" + report.save() + + index = ReportSearchIndex.objects.get(report=report) + assert index.patient_id == "CHANGED-123" + + +def test_patient_age_is_mirrored(): + """patient_age is a stored generated column. A BEFORE trigger would read NULL + here; this test is what pins the triggers to AFTER.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + + report.patient_id = "TOUCH" + report.save() + + index = ReportSearchIndex.objects.get(report=report) + report.refresh_from_db() + assert index.patient_age == report.patient_age + assert index.patient_age is not None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: the five new tests FAIL — `group_ids` stays `[]` and `patient_id` stays `None`, because nothing maintains them yet. + +- [ ] **Step 3: Write the trigger migration** + +Create `radis/pgsearch/migrations/0004_search_projection_triggers.py`: + +```python +"""Statement-level triggers maintaining the ReportSearchIndex search projection. + +Triggers rather than Django signals because group_ids is access-control data: +a signal fires only for ORM writes, missing management commands, the admin, +and raw SQL. A trigger holds inside the same transaction as the write, for +every writer. + +Statement-level rather than row-level because the bulk-upsert endpoint writes +membership with bulk_create (reports/api/viewsets.py:215,229), so one statement +can carry a whole batch. + +AFTER, never BEFORE: PostgreSQL does not expose stored generated column values +to BEFORE triggers, so patient_age would mirror as NULL. +""" + +from django.db import migrations + +SYNC_GROUP_IDS = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_group_ids() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET group_ids = COALESCE( + (SELECT array_agg(g.group_id ORDER BY g.group_id) + FROM reports_report_groups g WHERE g.report_id = rsi.report_id), + '{}') + WHERE rsi.report_id IN (SELECT DISTINCT report_id FROM changed); + RETURN NULL; +END $$; +""" + +SYNC_MODALITY_CODES = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_modality_codes() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET modality_codes = COALESCE( + (SELECT array_agg(m.code ORDER BY m.code) + FROM reports_report_modalities rm + JOIN reports_modality m ON m.id = rm.modality_id + WHERE rm.report_id = rsi.report_id), + '{}') + WHERE rsi.report_id IN (SELECT DISTINCT report_id FROM changed); + RETURN NULL; +END $$; +""" + +SYNC_REPORT_FIELDS = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_report_fields() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET language_code = l.code, + patient_sex = c.patient_sex, + patient_age = c.patient_age, + patient_id = c.patient_id, + study_datetime = c.study_datetime, + study_description = c.study_description, + report_created_at = c.created_at, + report_updated_at = c.updated_at + FROM changed c + LEFT JOIN reports_language l ON l.id = c.language_id + WHERE rsi.report_id = c.id + AND (rsi.language_code, rsi.patient_sex, rsi.patient_age, rsi.patient_id, + rsi.study_datetime, rsi.study_description, rsi.report_created_at, + rsi.report_updated_at) + IS DISTINCT FROM + (l.code, c.patient_sex, c.patient_age, c.patient_id, + c.study_datetime, c.study_description, c.created_at, c.updated_at); + RETURN NULL; +END $$; +""" + +CREATE_TRIGGERS = """ +CREATE TRIGGER pgsearch_group_ids_ins AFTER INSERT ON reports_report_groups +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_group_ids(); + +CREATE TRIGGER pgsearch_group_ids_del AFTER DELETE ON reports_report_groups +REFERENCING OLD TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_group_ids(); + +CREATE TRIGGER pgsearch_modality_codes_ins AFTER INSERT ON reports_report_modalities +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_modality_codes(); + +CREATE TRIGGER pgsearch_modality_codes_del AFTER DELETE ON reports_report_modalities +REFERENCING OLD TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_modality_codes(); + +CREATE TRIGGER pgsearch_report_fields_upd AFTER UPDATE ON reports_report +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_report_fields(); +""" + +DROP_TRIGGERS = """ +DROP TRIGGER IF EXISTS pgsearch_group_ids_ins ON reports_report_groups; +DROP TRIGGER IF EXISTS pgsearch_group_ids_del ON reports_report_groups; +DROP TRIGGER IF EXISTS pgsearch_modality_codes_ins ON reports_report_modalities; +DROP TRIGGER IF EXISTS pgsearch_modality_codes_del ON reports_report_modalities; +DROP TRIGGER IF EXISTS pgsearch_report_fields_upd ON reports_report; +DROP FUNCTION IF EXISTS pgsearch_sync_group_ids(); +DROP FUNCTION IF EXISTS pgsearch_sync_modality_codes(); +DROP FUNCTION IF EXISTS pgsearch_sync_report_fields(); +""" + + +class Migration(migrations.Migration): + dependencies = [ + ("pgsearch", "0003_search_projection_columns"), + ("reports", "0013_alter_report_options"), + ] + + operations = [ + migrations.RunSQL(SYNC_GROUP_IDS, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_MODALITY_CODES, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_REPORT_FIELDS, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(CREATE_TRIGGERS, reverse_sql=DROP_TRIGGERS), + ] +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: PASS. If `test_patient_age_is_mirrored` fails with `None`, the triggers were written as BEFORE — fix, do not work around. + +- [ ] **Step 5: Commit** + +```bash +git add radis/pgsearch/migrations/0004_search_projection_triggers.py radis/pgsearch/tests/test_search_projection.py +git commit -m "Maintain the search projection with statement-level triggers" +``` + +--- + +### Task 3: Populate the projection when index rows are created + +**Files:** +- Create: `radis/pgsearch/utils/projection.py` +- Modify: `radis/pgsearch/signals.py` +- Modify: `radis/pgsearch/utils/indexing.py` +- Modify: `radis/pgsearch/tests/test_search_projection.py` + +**Interfaces:** +- Consumes: the columns from Task 1. +- Produces: `radis.pgsearch.utils.projection.sync_projection(report_ids: Iterable[int]) -> None` and the module constant `PROJECTION_UPDATE_SQL: str`. + +The triggers maintain the projection on *change*, but a `ReportSearchIndex` row is created before its groups are attached (the API serializer calls `report.groups.set()` after `Report.objects.create()`). This task gives both creation paths one shared way to fill the row. + +- [ ] **Step 1: Write the failing test** + +Append to `radis/pgsearch/tests/test_search_projection.py`: + +```python +from radis.pgsearch.utils.indexing import bulk_upsert_report_search_indexes + + +def test_creation_populates_the_mirrored_scalars(): + report = ReportFactory.create(language=LanguageFactory.create(code="de")) + + index = ReportSearchIndex.objects.get(report=report) + assert index.language_code == "de" + assert index.patient_id == report.patient_id + + +def test_bulk_upsert_populates_the_projection(): + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=["CT"]) + group = GroupFactory.create() + report.groups.add(group) + ReportSearchIndex.objects.filter(report=report).delete() + + bulk_upsert_report_search_indexes([report.pk]) + + index = ReportSearchIndex.objects.get(report=report) + assert index.language_code == "en" + assert index.group_ids == [group.pk] + assert index.modality_codes == ["CT"] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: both FAIL — `language_code` is `None`. + +- [ ] **Step 3: Write the shared projection helper** + +Create `radis/pgsearch/utils/projection.py`: + +```python +"""The canonical SQL that fills the ReportSearchIndex search projection. + +Triggers (migration 0004) maintain the projection when its sources change. +This module fills it when an index row is first created, which happens before +the report's groups and modalities are attached. The backfill migration +duplicates this statement in SQL, the same way bulk_upsert_report_search_indexes +already duplicates the tsvector logic -- keep them in sync. +""" + +from collections.abc import Iterable + +from django.db import connection + +PROJECTION_UPDATE_SQL = """ +UPDATE pgsearch_reportsearchindex rsi +SET group_ids = COALESCE(g.ids, '{}'), + modality_codes = COALESCE(m.codes, '{}'), + language_code = l.code, + patient_sex = r.patient_sex, + patient_age = r.patient_age, + patient_id = r.patient_id, + study_datetime = r.study_datetime, + study_description = r.study_description, + report_created_at = r.created_at, + report_updated_at = r.updated_at +FROM reports_report r +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +WHERE rsi.report_id = r.id AND rsi.report_id = ANY(%s) +""" + + +def sync_projection(report_ids: Iterable[int]) -> None: + """Fill the projection columns for the given reports from their sources.""" + ids = [int(report_id) for report_id in report_ids] + if not ids: + return + with connection.cursor() as cursor: + cursor.execute(PROJECTION_UPDATE_SQL, [ids]) +``` + +- [ ] **Step 4: Call it from both creation paths** + +In `radis/pgsearch/signals.py`, replace the body of the receiver: + +```python +from django.db.models.signals import post_save +from django.dispatch import receiver + +from radis.reports.models import Report + +from .models import ReportSearchIndex +from .utils.projection import sync_projection + + +@receiver(post_save, sender=Report) +def create_or_update_report_search_index(sender, instance, created, **kwargs): + if created: + ReportSearchIndex.objects.create(report=instance) + # Groups and modalities are attached after the Report is created, so + # they stay empty here and the migration 0004 triggers fill them. + sync_projection([instance.pk]) + return + instance.search_index.save() +``` + +In `radis/pgsearch/utils/indexing.py`, add the import and call `sync_projection` once per chunk, immediately after the existing `cursor.execute(...)` block that sets `search_vector`: + +```python +from .projection import sync_projection +``` + +```python + sync_projection(chunk) +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: PASS + +- [ ] **Step 6: Run the whole pgsearch suite for regressions** + +Run: `uv run cli test -- radis/pgsearch -v` +Expected: PASS + +- [ ] **Step 7: Commit** + +```bash +git add radis/pgsearch/utils/projection.py radis/pgsearch/signals.py radis/pgsearch/utils/indexing.py radis/pgsearch/tests/test_search_projection.py +git commit -m "Populate the search projection when index rows are created" +``` + +--- + +### Task 4: Backfill existing rows + +**Files:** +- Create: `radis/pgsearch/migrations/0005_search_projection_backfill.py` +- Modify: `radis/pgsearch/tests/test_search_projection.py` + +**Interfaces:** +- Consumes: `PROJECTION_UPDATE_SQL` shape from Task 3 (duplicated inline — migrations stay self-contained). +- Produces: nothing importable. + +Measured at 8M: **9 minutes 20 seconds**, in 50,000-row chunks. This blocks the deploy, which is accepted for the target deployment. + +- [ ] **Step 1: Write the failing test** + +Append to `radis/pgsearch/tests/test_search_projection.py`: + +```python +def test_backfill_fills_rows_that_predate_the_projection(): + """Simulates a row written before the projection existed.""" + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=["MR"]) + group = GroupFactory.create() + report.groups.add(group) + + ReportSearchIndex.objects.filter(report=report).update( + group_ids=[], modality_codes=[], language_code=None, patient_id=None + ) + + from radis.pgsearch.utils.projection import sync_projection + + sync_projection([report.pk]) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + assert index.modality_codes == ["MR"] + assert index.language_code == "en" + assert index.patient_id == report.patient_id +``` + +- [ ] **Step 2: Run test to verify it passes already** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py::test_backfill_fills_rows_that_predate_the_projection -v` +Expected: PASS — Task 3 already provides the mechanism. This test pins the *semantics* the migration depends on, so a later refactor of `PROJECTION_UPDATE_SQL` cannot silently break the backfill. + +- [ ] **Step 3: Write the backfill migration** + +Create `radis/pgsearch/migrations/0005_search_projection_backfill.py`: + +```python +"""Backfill the ReportSearchIndex search projection for existing rows. + +atomic = False so each chunk commits on its own: at the 8M design target this +runs for about nine and a half minutes, and one transaction that long would +pin an equally long-lived snapshot. + +Runs after the triggers (0004) on purpose. A report edited during the backfill +is corrected by its trigger, and a chunk that later reprocesses the same row +simply rewrites the current values. + +The statement below duplicates utils/projection.PROJECTION_UPDATE_SQL, the same +way bulk_upsert_report_search_indexes duplicates the tsvector logic. Migrations +must not import app code, which drifts under them. +""" + +from django.db import migrations + +CHUNK_SIZE = 50_000 + +BACKFILL_SQL = """ +UPDATE pgsearch_reportsearchindex rsi +SET group_ids = COALESCE(g.ids, '{}'), + modality_codes = COALESCE(m.codes, '{}'), + language_code = l.code, + patient_sex = r.patient_sex, + patient_age = r.patient_age, + patient_id = r.patient_id, + study_datetime = r.study_datetime, + study_description = r.study_description, + report_created_at = r.created_at, + report_updated_at = r.updated_at +FROM reports_report r +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +WHERE rsi.report_id = r.id AND rsi.report_id > %s AND rsi.report_id <= %s +""" + + +def backfill(apps, schema_editor): + connection = schema_editor.connection + with connection.cursor() as cursor: + cursor.execute("SELECT COALESCE(MAX(report_id), 0) FROM pgsearch_reportsearchindex") + max_id = cursor.fetchone()[0] + + low = 0 + while low < max_id: + high = low + CHUNK_SIZE + with connection.cursor() as cursor: + cursor.execute(BACKFILL_SQL, [low, high]) + low = high + + +class Migration(migrations.Migration): + atomic = False + + dependencies = [("pgsearch", "0004_search_projection_triggers")] + + operations = [migrations.RunPython(backfill, migrations.RunPython.noop)] +``` + +- [ ] **Step 4: Verify the migration applies cleanly** + +Run: `uv run cli test -- radis/pgsearch -v` +Expected: PASS — the test database is built by applying every migration, so a broken migration fails the whole suite. + +- [ ] **Step 5: Commit** + +```bash +git add radis/pgsearch/migrations/0005_search_projection_backfill.py radis/pgsearch/tests/test_search_projection.py +git commit -m "Backfill the search projection in 50k-row chunks" +``` + +--- + +### Task 5: Indexes, created last + +**Files:** +- Modify: `radis/pgsearch/models.py` +- Create: `radis/pgsearch/migrations/0006_search_projection_indexes.py` + +**Interfaces:** +- Consumes: the columns from Task 1. +- Produces: indexes `pgsearch_group_ids_gin`, `pgsearch_report_updated_at_idx`. + +Created after the backfill deliberately: while the new columns are unindexed the backfill can use HOT updates. Measured at 8M — GIN 3.7 s, btree 1.6 s. Two indexes only; `modality_codes`, `study_datetime`, `patient_id` and `report_created_at` are deliberately unindexed (spec §4.1). + +- [ ] **Step 1: Add the indexes to the model** + +In `radis/pgsearch/models.py`, add to `ReportSearchIndex.Meta.indexes`: + +```python + GinIndex(fields=["group_ids"], name="pgsearch_group_ids_gin"), + models.Index(fields=["report_updated_at"], name="pgsearch_report_updated_at_idx"), +``` + +- [ ] **Step 2: Generate the migration** + +Run: `docker exec radis_dev-web-1 ./manage.py makemigrations pgsearch --name search_projection_indexes` + +- [ ] **Step 3: Add the GIN fastupdate change** + +Append to the `operations` list in the generated `0006_search_projection_indexes.py`: + +```python + migrations.RunSQL( + # fastupdate leaves an unsorted pending list that every query must + # scan; RADIS writes in async batches and reads interactively, so + # predictable read latency is the better side of the trade. + "ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = off);" + "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", + reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", + ), +``` + +- [ ] **Step 4: Write the test** + +Append to `radis/pgsearch/tests/test_search_projection.py`: + +```python +def test_projection_indexes_exist(): + with connection.cursor() as cursor: + cursor.execute( + "SELECT indexname FROM pg_indexes WHERE tablename = 'pgsearch_reportsearchindex'" + ) + names = {row[0] for row in cursor.fetchall()} + + assert "pgsearch_group_ids_gin" in names + assert "pgsearch_report_updated_at_idx" in names +``` + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch/tests/test_search_projection.py -v` +Expected: PASS + +- [ ] **Step 6: Commit** + +```bash +git add radis/pgsearch/models.py radis/pgsearch/migrations/0006_search_projection_indexes.py radis/pgsearch/tests/test_search_projection.py +git commit -m "Add the search projection indexes and disable GIN fastupdate" +``` + +--- + +### Task 6: check_search_projection command + +**Files:** +- Create: `radis/pgsearch/management/commands/check_search_projection.py` +- Create: `radis/pgsearch/tests/test_check_search_projection.py` + +**Interfaces:** +- Consumes: the columns from Task 1. +- Produces: management command `check_search_projection`, exiting non-zero when drift is found. + +- [ ] **Step 1: Write the failing test** + +Create `radis/pgsearch/tests/test_check_search_projection.py`: + +```python +"""The projection duplicates access-control data, so operators need a way to +prove it still matches its sources -- after a restore, a bulk import, or a +Language.code rename, which no trigger covers.""" + +import pytest +from adit_radis_shared.accounts.factories import GroupFactory +from django.core.management import call_command +from django.core.management.base import CommandError + +from radis.pgsearch.models import ReportSearchIndex +from radis.reports.factories import LanguageFactory, ReportFactory + +pytestmark = pytest.mark.django_db + + +def test_reports_no_drift_on_a_healthy_corpus(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + + call_command("check_search_projection") + + +def test_detects_drifted_group_ids(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + + ReportSearchIndex.objects.filter(report=report).update(group_ids=[9999]) + + with pytest.raises(CommandError, match="group_ids"): + call_command("check_search_projection") +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run cli test -- radis/pgsearch/tests/test_check_search_projection.py -v` +Expected: FAIL with `Unknown command: 'check_search_projection'` + +- [ ] **Step 3: Write the command** + +Create `radis/pgsearch/management/commands/check_search_projection.py`: + +```python +"""Verify the ReportSearchIndex search projection against its sources.""" + +from django.core.management.base import BaseCommand, CommandError +from django.db import connection + +DRIFT_SQL = """ +SELECT + count(*) FILTER (WHERE rsi.group_ids IS DISTINCT FROM COALESCE(g.ids, '{}')) + AS group_ids, + count(*) FILTER (WHERE rsi.modality_codes IS DISTINCT FROM COALESCE(m.codes, '{}')) + AS modality_codes, + count(*) FILTER (WHERE rsi.language_code IS DISTINCT FROM l.code) + AS language_code, + count(*) FILTER (WHERE rsi.patient_sex IS DISTINCT FROM r.patient_sex) + AS patient_sex, + count(*) FILTER (WHERE rsi.patient_age IS DISTINCT FROM r.patient_age) + AS patient_age, + count(*) FILTER (WHERE rsi.patient_id IS DISTINCT FROM r.patient_id) + AS patient_id, + count(*) FILTER (WHERE rsi.study_datetime IS DISTINCT FROM r.study_datetime) + AS study_datetime, + count(*) FILTER (WHERE rsi.study_description IS DISTINCT FROM r.study_description) + AS study_description, + count(*) FILTER (WHERE rsi.report_created_at IS DISTINCT FROM r.created_at) + AS report_created_at, + count(*) FILTER (WHERE rsi.report_updated_at IS DISTINCT FROM r.updated_at) + AS report_updated_at +FROM pgsearch_reportsearchindex rsi +JOIN reports_report r ON r.id = rsi.report_id +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +""" + + +class Command(BaseCommand): + help = "Verify the ReportSearchIndex search projection against its sources." + + def handle(self, *args, **options): + with connection.cursor() as cursor: + cursor.execute(DRIFT_SQL) + columns = [column.name for column in cursor.description] + counts = dict(zip(columns, cursor.fetchone(), strict=True)) + + drifted = {name: count for name, count in counts.items() if count} + if drifted: + details = ", ".join(f"{name}: {count}" for name, count in sorted(drifted.items())) + raise CommandError(f"Search projection drift detected -- {details}") + + self.stdout.write(self.style.SUCCESS("Search projection matches its sources.")) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch/tests/test_check_search_projection.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add radis/pgsearch/management/commands/check_search_projection.py radis/pgsearch/tests/test_check_search_projection.py +git commit -m "Add check_search_projection drift detection command" +``` + +--- + +### Task 7: Single-table _build_filter_query + +**Files:** +- Modify: `radis/pgsearch/providers.py:180-244` +- Create: `radis/pgsearch/tests/test_filter_query_equivalence.py` + +**Interfaces:** +- Consumes: the projection columns from Task 1, populated by Tasks 2–4. +- Produces: `_build_filter_query(filters: SearchFilters) -> Q` returning predicates against `ReportSearchIndex`'s own columns only, with no `report__` traversal. + +This is the task that delivers the performance change. It is also the one that can leak reports between groups, so the equivalence test comes first. + +- [ ] **Step 1: Write the failing equivalence test** + +Create `radis/pgsearch/tests/test_filter_query_equivalence.py`: + +```python +"""The new single-table filter must select exactly what the joined one did. + +_build_filter_query_legacy is the pre-change implementation kept as a reference +oracle. Delete both it and this module once the change has settled in production. +""" + +from datetime import date, timedelta + +import pytest +from adit_radis_shared.accounts.factories import GroupFactory +from django.db.models import Q +from django.utils import timezone + +from radis.pgsearch.models import ReportSearchIndex +from radis.pgsearch.providers import _build_filter_query +from radis.reports.factories import LanguageFactory, ReportFactory +from radis.search.site import SearchFilters + +pytestmark = pytest.mark.django_db + + +def _build_filter_query_legacy(filters: SearchFilters) -> Q: + """The joined implementation this change replaces.""" + fq = Q(report__groups=filters.group) + if filters.patient_sex: + fq &= Q(report__patient_sex=filters.patient_sex) + if filters.language: + fq &= Q(report__language__code=filters.language) + if filters.modalities: + fq &= Q(report__modalities__code__in=filters.modalities) + if filters.study_date_from: + fq &= Q(report__study_datetime__date__gte=filters.study_date_from) + if filters.study_date_till: + fq &= Q(report__study_datetime__date__lte=filters.study_date_till) + if filters.study_description: + fq &= Q(report__study_description__icontains=filters.study_description) + if filters.patient_age_from is not None: + fq &= Q(report__patient_age__gte=filters.patient_age_from) + if filters.patient_age_till is not None: + fq &= Q(report__patient_age__lte=filters.patient_age_till) + if filters.patient_id: + fq &= Q(report__patient_id=filters.patient_id) + if filters.updated_after: + fq &= Q(report__updated_at__gte=filters.updated_after) + return fq + + +def _ids(fq: Q) -> set[int]: + return set( + ReportSearchIndex.objects.filter(fq).distinct().values_list("report_id", flat=True) + ) + + +@pytest.fixture +def corpus(): + language_en = LanguageFactory.create(code="en") + language_de = LanguageFactory.create(code="de") + group_a = GroupFactory.create(name="A") + group_b = GroupFactory.create(name="B") + now = timezone.now() + + first = ReportFactory.create( + language=language_en, patient_sex="M", patient_id="P1", + study_description="CT Thorax", study_datetime=now - timedelta(days=10), + modalities=["CT"], + ) + first.groups.add(group_a) + + second = ReportFactory.create( + language=language_de, patient_sex="F", patient_id="P2", + study_description="MR Head", study_datetime=now - timedelta(days=400), + modalities=["MR", "CT"], + ) + second.groups.add(group_a, group_b) + + orphan = ReportFactory.create(language=language_en, patient_sex="M", patient_id="P3") + + return {"group_a": group_a, "group_b": group_b, "orphan": orphan} + + +@pytest.mark.parametrize( + "make_filters", + [ + pytest.param(lambda c: SearchFilters(group=c["group_a"].pk), id="group-only"), + pytest.param(lambda c: SearchFilters(group=c["group_b"].pk), id="other-group"), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, language="en"), id="language" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, modalities=["CT"]), id="modality" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, modalities=["CT", "MR"]), + id="modalities-multi", + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_sex="M"), id="sex" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_id="P1"), id="patient-id" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, study_description="thorax"), + id="description-icontains", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, study_date_from=date.today() - timedelta(days=30) + ), + id="date-from", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, study_date_till=date.today() - timedelta(days=5) + ), + id="date-till", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, + study_date_from=date.today() - timedelta(days=30), + study_date_till=date.today(), + ), + id="date-range", + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_age_from=0), id="age-from" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_age_till=200), id="age-till" + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, updated_after=timezone.now() - timedelta(hours=1) + ), + id="updated-after", + ), + ], +) +def test_new_filter_matches_the_legacy_one(corpus, make_filters): + filters = make_filters(corpus) + + assert _ids(_build_filter_query(filters)) == _ids(_build_filter_query_legacy(filters)) + + +def test_group_none_is_fail_closed(corpus): + """group=None must match only reports in no group at all -- never everything.""" + result = _ids(_build_filter_query(SearchFilters(group=None))) + + assert result == {corpus["orphan"].pk} + + +def test_no_duplicates_without_distinct(corpus): + """A report matching on two modalities must appear once, which is what the + removed .distinct() used to guarantee.""" + filters = SearchFilters(group=corpus["group_a"].pk, modalities=["CT", "MR"]) + + rows = list( + ReportSearchIndex.objects.filter(_build_filter_query(filters)).values_list( + "report_id", flat=True + ) + ) + + assert len(rows) == len(set(rows)) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run cli test -- radis/pgsearch/tests/test_filter_query_equivalence.py -v` +Expected: `test_no_duplicates_without_distinct` FAILS (the current implementation duplicates), and `test_group_none_is_fail_closed` may pass incidentally. The parametrised cases pass trivially because both sides are still the same code. + +- [ ] **Step 3: Rewrite _build_filter_query** + +In `radis/pgsearch/providers.py`, replace the whole `_build_filter_query` function with: + +```python +def _local_day_start(day: date) -> datetime: + """Midnight of ``day`` in the active timezone, as an aware datetime.""" + return timezone.make_aware(datetime.combine(day, time.min)) + + +def _build_filter_query(filters: SearchFilters) -> Q: + # Every predicate here is a column on ReportSearchIndex itself -- the search + # projection (see migration 0003) mirrors the Report fields search filters + # on, so the candidate query stays single-table. Reintroducing a ``report__`` + # traversal restores the join and the multi-second query shape it caused; + # test_filter_query_plan_is_single_table guards against that. + # + # Group-scoped access control. ``SearchView`` supplies + # ``group=active_group.pk``; the extraction preview may pass ``group=None`` + # when the user has no active group, which is fail-closed: an empty + # ``group_ids`` matches only reports assigned to no group at all. + if filters.group is None: + fq = Q(group_ids=[]) + else: + fq = Q(group_ids__contains=[filters.group]) + + # Apply hard filter criteria + if filters.patient_sex: + fq &= Q(patient_sex=filters.patient_sex) + if filters.language: + fq &= Q(language_code=filters.language) + if filters.modalities: + fq &= Q(modality_codes__overlap=filters.modalities) + # Half-open ranges rather than ``__date``: Django compiles that lookup to + # ``(study_datetime AT TIME ZONE ...)::date``, a function over the column + # evaluated once per row -- millions of timezone conversions per query on + # the sequential scan, for a predicate equal to two timestamp comparisons. + if filters.study_date_from: + fq &= Q(study_datetime__gte=_local_day_start(filters.study_date_from)) + if filters.study_date_till: + fq &= Q(study_datetime__lt=_local_day_start(filters.study_date_till + timedelta(days=1))) + if filters.study_description: + fq &= Q(study_description__icontains=filters.study_description) + if filters.patient_age_from is not None: + fq &= Q(patient_age__gte=filters.patient_age_from) + if filters.patient_age_till is not None: + fq &= Q(patient_age__lte=filters.patient_age_till) + if filters.patient_id: + fq &= Q(patient_id=filters.patient_id) + if filters.created_after: + fq &= Q(report_created_at__gte=filters.created_after) + if filters.created_before: + fq &= Q(report_created_at__lte=filters.created_before) + if filters.labels: + from radis.labels.models import LabelResult + from radis.reports.models import Report + + surfacing_report_ids = Report.objects.filter( + label_results__label__name__in=filters.labels, + label_results__value__in=LabelResult.SURFACING_VALUES, + ).values("pk") + fq &= Q(report_id__in=surfacing_report_ids) + if filters.updated_after: + fq &= Q(report_updated_at__gte=filters.updated_after) + + return fq +``` + +Add these imports at the top of `providers.py`: + +```python +from datetime import date, datetime, time, timedelta + +from django.utils import timezone +``` + +- [ ] **Step 4: Remove the two `.distinct()` calls** + +In `_fuse_hybrid`, delete `.distinct()` from both the vector queryset (`providers.py:394`) and the FTS queryset (`providers.py:412`). With no joins there is nothing to deduplicate. + +- [ ] **Step 5: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch/tests/test_filter_query_equivalence.py -v` +Expected: PASS, all parametrised cases plus both named tests. + +- [ ] **Step 6: Run the full suite for regressions** + +Run: `uv run cli test -- radis/pgsearch radis/search radis/extractions radis/subscriptions -v` +Expected: PASS. These are the four consumers of `_build_filter_query`. + +- [ ] **Step 7: Commit** + +```bash +git add radis/pgsearch/providers.py radis/pgsearch/tests/test_filter_query_equivalence.py +git commit -m "Make the FTS filter query single-table" +``` + +--- + +### Task 8: Language predicates off the join + +**Files:** +- Modify: `radis/pgsearch/providers.py` — `_exclude_negations`, `_fuse_hybrid`, `search` + +**Interfaces:** +- Consumes: `ReportSearchIndex.language_code` from Task 1. +- Produces: no signature change. + +Five `report__language__code__in` traversals remain after Task 7 and would each re-add the `reports_language` join. + +- [ ] **Step 1: Write the failing test** + +Append to `radis/pgsearch/tests/test_filter_query_equivalence.py`: + +```python +def test_no_report_traversal_remains_in_the_candidate_query(): + """Any report__ lookup re-adds a join and the multi-second query shape.""" + import inspect + + from radis.pgsearch import providers + + source = inspect.getsource(providers) + offending = [ + line.strip() + for line in source.splitlines() + if "report__" in line + and "report__body" not in line # hydration headline, deliberately joined + and "report__document_id" not in line # filter(), deliberately joined + and not line.strip().startswith("#") + ] + + assert offending == [], f"report__ traversals left in the hot path: {offending}" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run cli test -- radis/pgsearch/tests/test_filter_query_equivalence.py::test_no_report_traversal_remains_in_the_candidate_query -v` +Expected: FAIL listing the five `report__language__code__in` lines. + +- [ ] **Step 3: Replace the traversals** + +In `radis/pgsearch/providers.py`, replace every `report__language__code__in=codes` with `language_code__in=codes`. There are five occurrences: + +- `_exclude_negations`, in the `queryset.exclude(...)` call +- `_fuse_hybrid`, building `match_q` +- `_fuse_hybrid`, building `rank_expr` +- `search`, building `summary_expr` +- `search`, building its own `rank_expr` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run cli test -- radis/pgsearch -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add radis/pgsearch/providers.py radis/pgsearch/tests/test_filter_query_equivalence.py +git commit -m "Read language_code from the projection instead of joining" +``` + +--- + +### Task 9: Plan-shape regression test + +**Files:** +- Modify: `radis/pgsearch/tests/test_filter_query_equivalence.py` + +**Interfaces:** +- Consumes: everything from Tasks 1–8. +- Produces: nothing importable. + +Wall-clock assertions are flaky in CI; plan structure is not. This is what catches a future change silently restoring the 8-second query. + +- [ ] **Step 1: Write the failing test** + +Append to `radis/pgsearch/tests/test_filter_query_equivalence.py`: + +```python +def test_filter_query_plan_is_single_table(corpus): + """The candidate query must not join the membership table or deduplicate.""" + from django.db import connection + + filters = SearchFilters(group=corpus["group_a"].pk, language="en") + queryset = ReportSearchIndex.objects.filter(_build_filter_query(filters)).values_list( + "report_id", flat=True + ) + sql, params = queryset.query.sql_with_params() + + with connection.cursor() as cursor: + cursor.execute(f"EXPLAIN {sql}", params) + plan = "\n".join(row[0] for row in cursor.fetchall()) + + assert "reports_report_groups" not in plan, plan + assert "Unique" not in plan, plan +``` + +- [ ] **Step 2: Run test to verify it passes** + +Run: `uv run cli test -- radis/pgsearch/tests/test_filter_query_equivalence.py::test_filter_query_plan_is_single_table -v` +Expected: PASS — Tasks 7 and 8 already removed the join. If it fails, one of them is incomplete. + +- [ ] **Step 3: Commit** + +```bash +git add radis/pgsearch/tests/test_filter_query_equivalence.py +git commit -m "Assert the candidate query plan stays single-table" +``` + +--- + +### Task 10: PostgreSQL tuning and operator documentation + +**Files:** +- Modify: `docker-compose.base.yml` +- Modify: `docs/user-docs/admin-guide.md` +- Modify: `CLAUDE.md` + +**Interfaces:** +- Consumes: nothing. +- Produces: env vars `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER`, `POSTGRES_MAX_PARALLEL_WORKERS`, `POSTGRES_MAX_WORKER_PROCESSES`, `POSTGRES_SHARED_BUFFERS`. + +These are compose-only overrides with defaults, **not** `example.env` entries — matching how the project already treats `EMBEDDINGS_WORKER_CONCURRENCY` and `WAIT_POSTGRES_TIMEOUT`. Only the first differs from PostgreSQL's own default. + +- [ ] **Step 1: Add the GUCs to the compose file** + +In `docker-compose.base.yml`, replace the `postgres` service with: + +```yaml + postgres: + image: pgvector/pgvector:pg17@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f + hostname: postgres.local + command: + - postgres + - -c + - max_parallel_workers_per_gather=${POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER:-4} + - -c + - max_parallel_workers=${POSTGRES_MAX_PARALLEL_WORKERS:-8} + - -c + - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} + - -c + - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} + volumes: + - postgres_data:/var/lib/postgresql/data +``` + +- [ ] **Step 2: Verify the stack starts and the setting took effect** + +```bash +uv run cli compose-up -- -d +docker exec radis_dev-postgres-1 psql -U postgres -c "SHOW max_parallel_workers_per_gather;" +``` + +Expected: `4` + +- [ ] **Step 3: Document them for operators** + +Add to `docs/user-docs/admin-guide.md`: + +```markdown +## Database tuning + +Search scans the report index table in parallel, so the number of parallel +workers is the one PostgreSQL setting worth revisiting. These are optional +overrides -- set them in `.env` only if the defaults do not suit your hardware. + +| Variable | Default | Guidance | +| --- | --- | --- | +| `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | Measured on 8M reports: 606 ms at 2, 421 ms at 4, 343 ms at 8. Four captures most of the gain while leaving cores for concurrent searches. | +| `POSTGRES_MAX_PARALLEL_WORKERS` | `8` | PostgreSQL's default. Raise together with the two others on a larger host. | +| `POSTGRES_MAX_WORKER_PROCESSES` | `8` | As above. | +| `POSTGRES_SHARED_BUFFERS` | `128MB` | PostgreSQL's default. Around 25% of host RAM is the usual recommendation; a value larger than the container's memory will prevent PostgreSQL from starting. | +``` + +Add the same four variables to the environment-variable section of `CLAUDE.md`, noting they are compose-only overrides. + +- [ ] **Step 4: Commit** + +```bash +git add docker-compose.base.yml docs/user-docs/admin-guide.md CLAUDE.md +git commit -m "Tune PostgreSQL parallelism for the search scan" +``` + +--- + +## Verification after all tasks + +- [ ] `uv run cli test` — full suite green +- [ ] `uv run cli lint` — clean +- [ ] `docker exec radis_dev-web-1 ./manage.py check_search_projection` — reports no drift +- [ ] Search the dev stack for a common term and confirm results still render +- [ ] **Manual, against a large corpus:** confirm the spec's success criterion — + the FTS candidate query under 1,000 ms at the 8M design target with + `max_parallel_workers_per_gather = 4`. Not assertable in CI, which has no + corpus of that size; the plan-shape test in Task 9 is the CI proxy. From a1d3fbba9d110442a4da2b5ed5304ea6029199e9 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 18:55:30 +0000 Subject: [PATCH 11/50] Add search projection columns to ReportSearchIndex Implement the first stage of query-shape optimization by adding projection columns (group_ids, modality_codes, language_code, patient_sex, patient_age, patient_id, study_datetime, study_description, report_created_at, report_updated_at) that mirror Report fields used by search filters. These columns allow the FTS candidate query to remain single-table rather than joining three tables with DISTINCT. Columns default to empty/null, keeping all AddField operations metadata-only on the 8M-row production table; later tasks add triggers (0004) to maintain them and backfill tasks to populate them. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../0003_search_projection_columns.py | 64 +++++++++++++++++++ radis/pgsearch/models.py | 17 +++++ .../pgsearch/tests/test_search_projection.py | 21 ++++++ 3 files changed, 102 insertions(+) create mode 100644 radis/pgsearch/migrations/0003_search_projection_columns.py create mode 100644 radis/pgsearch/tests/test_search_projection.py diff --git a/radis/pgsearch/migrations/0003_search_projection_columns.py b/radis/pgsearch/migrations/0003_search_projection_columns.py new file mode 100644 index 00000000..8954cbdb --- /dev/null +++ b/radis/pgsearch/migrations/0003_search_projection_columns.py @@ -0,0 +1,64 @@ +# Generated by Django 6.0.7 on 2026-08-30 18:52 + +import django.contrib.postgres.fields +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pgsearch', '0002_hybrid_search'), + ] + + operations = [ + migrations.AddField( + model_name='reportsearchindex', + name='group_ids', + field=django.contrib.postgres.fields.ArrayField(base_field=models.IntegerField(), default=list), + ), + migrations.AddField( + model_name='reportsearchindex', + name='language_code', + field=models.CharField(max_length=10, null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='modality_codes', + field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=16), default=list), + ), + migrations.AddField( + model_name='reportsearchindex', + name='patient_age', + field=models.IntegerField(null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='patient_id', + field=models.CharField(max_length=64, null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='patient_sex', + field=models.CharField(max_length=1, null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='report_created_at', + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='report_updated_at', + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='study_datetime', + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name='reportsearchindex', + name='study_description', + field=models.CharField(blank=True, max_length=64, null=True), + ), + ] diff --git a/radis/pgsearch/models.py b/radis/pgsearch/models.py index b7c2bab0..ab0feb93 100644 --- a/radis/pgsearch/models.py +++ b/radis/pgsearch/models.py @@ -1,4 +1,5 @@ from django.conf import settings +from django.contrib.postgres.fields import ArrayField from django.contrib.postgres.indexes import GinIndex from django.contrib.postgres.search import SearchVector, SearchVectorField from django.db import models @@ -21,6 +22,22 @@ class ReportSearchIndex(models.Model): search_vector = SearchVectorField(null=True) embedding = VectorField(dimensions=settings.EMBEDDINGS_DIM, null=True) + # Search projection: mirrors of the Report fields the scan filters on, so + # the FTS candidate query stays single-table. Maintained by the triggers in + # migration 0004 and populated on creation by signals.py / indexing.py. + # Nullable so every AddField stays metadata-only on a large table; + # check_search_projection guards against drift instead of a NOT NULL scan. + group_ids = ArrayField(models.IntegerField(), default=list) + modality_codes = ArrayField(models.CharField(max_length=16), default=list) + language_code = models.CharField(max_length=10, null=True) # noqa: DJ001 + patient_sex = models.CharField(max_length=1, null=True) # noqa: DJ001 + patient_age = models.IntegerField(null=True) + patient_id = models.CharField(max_length=64, null=True) # noqa: DJ001 + study_datetime = models.DateTimeField(null=True) + study_description = models.CharField(max_length=64, blank=True, null=True) # noqa: DJ001 + report_created_at = models.DateTimeField(null=True) + report_updated_at = models.DateTimeField(null=True) + class Meta: verbose_name = "Report search index" verbose_name_plural = "Report search indexes" diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py new file mode 100644 index 00000000..f4dde1b2 --- /dev/null +++ b/radis/pgsearch/tests/test_search_projection.py @@ -0,0 +1,21 @@ +"""Tests for the ReportSearchIndex search projection. + +The projection mirrors the Report fields that search filters on, so the FTS +candidate query can stay single-table. group_ids is access-control data, so +its correctness has its own tests here. +""" + +import pytest + +from radis.pgsearch.models import ReportSearchIndex +from radis.reports.factories import LanguageFactory, ReportFactory + +pytestmark = pytest.mark.django_db + + +def test_new_index_row_defaults_to_empty_arrays(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + index = ReportSearchIndex.objects.get(report=report) + + assert index.group_ids == [] + assert index.modality_codes == [] From 0d113ff23c597163e9b5b74f02638f61f8b8711e Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 19:20:20 +0000 Subject: [PATCH 12/50] Maintain the search projection with statement-level triggers Add AFTER, statement-level triggers that mirror group_ids, modality_codes, and the scalar Report fields into ReportSearchIndex on every write - management commands, the admin, and raw SQL included, not just ORM saves. Triggers must be AFTER (not BEFORE) so patient_age, a stored generated column, is visible to them. Fix a latent bug in the pre-existing post_save signal that the new triggers exposed: OneToOneField reciprocally caches the search index on the Report instance at creation, so a later report.save() reused that stale ReportSearchIndex object and its unqualified .save() call wrote all of its (stale) columns back, clobbering the projection fields the trigger had just set in the same statement. Scoping that save to update_fields=["search_vector"] - the only thing this signal is meant to refresh - fixes it. --- .../0004_search_projection_triggers.py | 119 ++++++++++++++++++ radis/pgsearch/signals.py | 10 +- .../pgsearch/tests/test_search_projection.py | 101 ++++++++++++++- 3 files changed, 228 insertions(+), 2 deletions(-) create mode 100644 radis/pgsearch/migrations/0004_search_projection_triggers.py diff --git a/radis/pgsearch/migrations/0004_search_projection_triggers.py b/radis/pgsearch/migrations/0004_search_projection_triggers.py new file mode 100644 index 00000000..fa4ff811 --- /dev/null +++ b/radis/pgsearch/migrations/0004_search_projection_triggers.py @@ -0,0 +1,119 @@ +"""Statement-level triggers maintaining the ReportSearchIndex search projection. + +Triggers rather than Django signals because group_ids is access-control data: +a signal fires only for ORM writes, missing management commands, the admin, +and raw SQL. A trigger holds inside the same transaction as the write, for +every writer. + +Statement-level rather than row-level because the bulk-upsert endpoint writes +membership with bulk_create (reports/api/viewsets.py:215,229), so one statement +can carry a whole batch. + +AFTER, never BEFORE: PostgreSQL does not expose stored generated column values +to BEFORE triggers, so patient_age would mirror as NULL. +""" + +from django.db import migrations + +SYNC_GROUP_IDS = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_group_ids() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET group_ids = COALESCE( + (SELECT array_agg(g.group_id ORDER BY g.group_id) + FROM reports_report_groups g WHERE g.report_id = rsi.report_id), + '{}') + WHERE rsi.report_id IN (SELECT DISTINCT report_id FROM changed); + RETURN NULL; +END $$; +""" + +SYNC_MODALITY_CODES = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_modality_codes() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET modality_codes = COALESCE( + (SELECT array_agg(m.code ORDER BY m.code) + FROM reports_report_modalities rm + JOIN reports_modality m ON m.id = rm.modality_id + WHERE rm.report_id = rsi.report_id), + '{}') + WHERE rsi.report_id IN (SELECT DISTINCT report_id FROM changed); + RETURN NULL; +END $$; +""" + +SYNC_REPORT_FIELDS = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_report_fields() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET language_code = l.code, + patient_sex = c.patient_sex, + patient_age = c.patient_age, + patient_id = c.patient_id, + study_datetime = c.study_datetime, + study_description = c.study_description, + report_created_at = c.created_at, + report_updated_at = c.updated_at + FROM changed c + LEFT JOIN reports_language l ON l.id = c.language_id + WHERE rsi.report_id = c.id + AND (rsi.language_code, rsi.patient_sex, rsi.patient_age, rsi.patient_id, + rsi.study_datetime, rsi.study_description, rsi.report_created_at, + rsi.report_updated_at) + IS DISTINCT FROM + (l.code, c.patient_sex, c.patient_age, c.patient_id, + c.study_datetime, c.study_description, c.created_at, c.updated_at); + RETURN NULL; +END $$; +""" + +CREATE_TRIGGERS = """ +CREATE TRIGGER pgsearch_group_ids_ins AFTER INSERT ON reports_report_groups +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_group_ids(); + +CREATE TRIGGER pgsearch_group_ids_del AFTER DELETE ON reports_report_groups +REFERENCING OLD TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_group_ids(); + +CREATE TRIGGER pgsearch_modality_codes_ins AFTER INSERT ON reports_report_modalities +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_modality_codes(); + +CREATE TRIGGER pgsearch_modality_codes_del AFTER DELETE ON reports_report_modalities +REFERENCING OLD TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_modality_codes(); + +CREATE TRIGGER pgsearch_report_fields_upd AFTER UPDATE ON reports_report +REFERENCING NEW TABLE AS changed +FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_report_fields(); +""" + +DROP_TRIGGERS = """ +DROP TRIGGER IF EXISTS pgsearch_group_ids_ins ON reports_report_groups; +DROP TRIGGER IF EXISTS pgsearch_group_ids_del ON reports_report_groups; +DROP TRIGGER IF EXISTS pgsearch_modality_codes_ins ON reports_report_modalities; +DROP TRIGGER IF EXISTS pgsearch_modality_codes_del ON reports_report_modalities; +DROP TRIGGER IF EXISTS pgsearch_report_fields_upd ON reports_report; +DROP FUNCTION IF EXISTS pgsearch_sync_group_ids(); +DROP FUNCTION IF EXISTS pgsearch_sync_modality_codes(); +DROP FUNCTION IF EXISTS pgsearch_sync_report_fields(); +""" + + +class Migration(migrations.Migration): + dependencies = [ + ("pgsearch", "0003_search_projection_columns"), + ("reports", "0013_alter_report_options"), + ] + + operations = [ + migrations.RunSQL(SYNC_GROUP_IDS, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_MODALITY_CODES, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_REPORT_FIELDS, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(CREATE_TRIGGERS, reverse_sql=DROP_TRIGGERS), + ] diff --git a/radis/pgsearch/signals.py b/radis/pgsearch/signals.py index dc351fab..50c7ce4e 100644 --- a/radis/pgsearch/signals.py +++ b/radis/pgsearch/signals.py @@ -11,4 +11,12 @@ def create_or_update_report_search_index(sender, instance, created, **kwargs): if created: ReportSearchIndex.objects.create(report=instance) return - instance.search_index.save() + + # update_fields is required here, not just an optimization: OneToOneField + # reciprocally caches the index on the report instance the moment it's + # created, so instance.search_index below is often that same stale + # Python object from creation time. A bare save() would write ALL of its + # fields back -- including the projection columns the AFTER UPDATE + # trigger on reports_report just set moments ago in this same statement + # -- clobbering the trigger's fresh values with the stale ones. + instance.search_index.save(update_fields=["search_vector"]) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index f4dde1b2..2aa66828 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -6,16 +6,115 @@ """ import pytest +from adit_radis_shared.accounts.factories import GroupFactory +from django.db import connection from radis.pgsearch.models import ReportSearchIndex from radis.reports.factories import LanguageFactory, ReportFactory +from radis.reports.models import Report pytestmark = pytest.mark.django_db def test_new_index_row_defaults_to_empty_arrays(): - report = ReportFactory.create(language=LanguageFactory.create(code="en")) + # modalities=[] is explicit: ReportFactory attaches a random non-empty set + # by default, and the modality_codes trigger added in this module would + # otherwise correctly mirror it, defeating the point of this test. + report = ReportFactory.create(language=LanguageFactory.create(code="en"), modalities=[]) index = ReportSearchIndex.objects.get(report=report) assert index.group_ids == [] assert index.modality_codes == [] + + +def test_adding_a_group_updates_the_projection(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + + report.groups.add(group) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + + +def test_removing_a_group_updates_the_projection(): + """The leak direction: a report removed from a group must stop being visible.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + report.groups.add(group) + + report.groups.remove(group) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [] + + +def test_raw_sql_membership_write_updates_the_projection(): + """The whole reason for triggers over m2m_changed: writers that bypass the ORM.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + group = GroupFactory.create() + + with connection.cursor() as cursor: + cursor.execute( + "INSERT INTO reports_report_groups (report_id, group_id) VALUES (%s, %s)", + [report.pk, group.pk], + ) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + + +def test_bulk_membership_insert_updates_every_affected_row(): + """Statement-level triggers fail classically by processing only one transition row. + + This is the shape reports/api/viewsets.py:229 uses for bulk upsert. + """ + language = LanguageFactory.create(code="en") + reports = [ReportFactory.create(language=language) for _ in range(5)] + group = GroupFactory.create() + + through = Report.groups.through + through.objects.bulk_create( + [through(report_id=report.pk, group_id=group.pk) for report in reports] + ) + + for report in reports: + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk], f"report {report.pk} was not updated" + + +def test_deleting_a_report_does_not_error(): + """Deleting a Report cascades to both the membership rows and the index row, + firing the membership trigger against a row that may already be gone. The + spec calls that harmless in either cascade order; this pins it.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + report_pk = report.pk + + report.delete() + + assert not ReportSearchIndex.objects.filter(report_id=report_pk).exists() + + +def test_updating_a_report_updates_the_mirrored_scalars(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + + report.patient_id = "CHANGED-123" + report.save() + + index = ReportSearchIndex.objects.get(report=report) + assert index.patient_id == "CHANGED-123" + + +def test_patient_age_is_mirrored(): + """patient_age is a stored generated column. A BEFORE trigger would read NULL + here; this test is what pins the triggers to AFTER.""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + + report.patient_id = "TOUCH" + report.save() + + index = ReportSearchIndex.objects.get(report=report) + report.refresh_from_db() + assert index.patient_age == report.patient_age + assert index.patient_age is not None From 384ef45f6827c5c1b2d8ff6b52b389a2a8c64afd Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 19:34:21 +0000 Subject: [PATCH 13/50] Add positive coverage for the modality projection triggers test_new_index_row_defaults_to_empty_arrays' modalities=[] edit removed the only assertion that ever touched modality_codes, leaving the INSERT and DELETE triggers on reports_report_modalities with no test proving they run. Add test_adding_a_modality_updates_the_projection and test_removing_a_modality_updates_the_projection, mirroring the existing group_ids trigger tests. --- .../pgsearch/tests/test_search_projection.py | 23 ++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 2aa66828..7012f8a6 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -10,7 +10,7 @@ from django.db import connection from radis.pgsearch.models import ReportSearchIndex -from radis.reports.factories import LanguageFactory, ReportFactory +from radis.reports.factories import LanguageFactory, ModalityFactory, ReportFactory from radis.reports.models import Report pytestmark = pytest.mark.django_db @@ -49,6 +49,27 @@ def test_removing_a_group_updates_the_projection(): assert index.group_ids == [] +def test_adding_a_modality_updates_the_projection(): + report = ReportFactory.create(language=LanguageFactory.create(code="en"), modalities=[]) + modality = ModalityFactory.create(code="CT") + + report.modalities.add(modality) + + index = ReportSearchIndex.objects.get(report=report) + assert index.modality_codes == ["CT"] + + +def test_removing_a_modality_updates_the_projection(): + report = ReportFactory.create(language=LanguageFactory.create(code="en"), modalities=[]) + modality = ModalityFactory.create(code="CT") + report.modalities.add(modality) + + report.modalities.remove(modality) + + index = ReportSearchIndex.objects.get(report=report) + assert index.modality_codes == [] + + def test_raw_sql_membership_write_updates_the_projection(): """The whole reason for triggers over m2m_changed: writers that bypass the ORM.""" report = ReportFactory.create(language=LanguageFactory.create(code="en")) From f3fe6513abde05d48847af0cf8d7ff8758fcf88c Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 19:37:01 +0000 Subject: [PATCH 14/50] Correct Task 3's signals.py snippet in the implementation plan Task 2 discovered that OneToOneField reverse-caching makes a bare instance.search_index.save() write a stale in-memory copy back over the values the AFTER UPDATE trigger just set, and narrowed it to update_fields=["search_vector"]. Task 3's snippet still carried the bare save and would have silently reverted that fix. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- docs/superpowers/plans/2026-08-30-fts-search-projection.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-30-fts-search-projection.md b/docs/superpowers/plans/2026-08-30-fts-search-projection.md index 8bfee5c6..46a6e520 100644 --- a/docs/superpowers/plans/2026-08-30-fts-search-projection.md +++ b/docs/superpowers/plans/2026-08-30-fts-search-projection.md @@ -520,7 +520,12 @@ def create_or_update_report_search_index(sender, instance, created, **kwargs): # they stay empty here and the migration 0004 triggers fill them. sync_projection([instance.pk]) return - instance.search_index.save() + + # update_fields is required, not an optimisation: OneToOneField caches the + # index on the report instance, so instance.search_index is often the stale + # object from creation time. A bare save() writes all of its fields back, + # clobbering the projection columns the AFTER UPDATE trigger just set. + instance.search_index.save(update_fields=["search_vector"]) ``` In `radis/pgsearch/utils/indexing.py`, add the import and call `sync_projection` once per chunk, immediately after the existing `cursor.execute(...)` block that sets `search_vector`: From 787500f843bc5bde875b859fed2403f5bc11f487 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 19:47:45 +0000 Subject: [PATCH 15/50] Populate the search projection when index rows are created --- radis/pgsearch/signals.py | 4 ++ .../pgsearch/tests/test_search_projection.py | 24 ++++++++++ radis/pgsearch/utils/indexing.py | 3 ++ radis/pgsearch/utils/projection.py | 48 +++++++++++++++++++ 4 files changed, 79 insertions(+) create mode 100644 radis/pgsearch/utils/projection.py diff --git a/radis/pgsearch/signals.py b/radis/pgsearch/signals.py index 50c7ce4e..9a884596 100644 --- a/radis/pgsearch/signals.py +++ b/radis/pgsearch/signals.py @@ -4,12 +4,16 @@ from radis.reports.models import Report from .models import ReportSearchIndex +from .utils.projection import sync_projection @receiver(post_save, sender=Report) def create_or_update_report_search_index(sender, instance, created, **kwargs): if created: ReportSearchIndex.objects.create(report=instance) + # Groups and modalities are attached after the Report is created, so + # they stay empty here and the migration 0004 triggers fill them. + sync_projection([instance.pk]) return # update_fields is required here, not just an optimization: OneToOneField diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 7012f8a6..85c52ccd 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -10,6 +10,7 @@ from django.db import connection from radis.pgsearch.models import ReportSearchIndex +from radis.pgsearch.utils.indexing import bulk_upsert_report_search_indexes from radis.reports.factories import LanguageFactory, ModalityFactory, ReportFactory from radis.reports.models import Report @@ -139,3 +140,26 @@ def test_patient_age_is_mirrored(): report.refresh_from_db() assert index.patient_age == report.patient_age assert index.patient_age is not None + + +def test_creation_populates_the_mirrored_scalars(): + report = ReportFactory.create(language=LanguageFactory.create(code="de")) + + index = ReportSearchIndex.objects.get(report=report) + assert index.language_code == "de" + assert index.patient_id == report.patient_id + + +def test_bulk_upsert_populates_the_projection(): + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=["CT"]) + group = GroupFactory.create() + report.groups.add(group) + ReportSearchIndex.objects.filter(report=report).delete() + + bulk_upsert_report_search_indexes([report.pk]) + + index = ReportSearchIndex.objects.get(report=report) + assert index.language_code == "en" + assert index.group_ids == [group.pk] + assert index.modality_codes == ["CT"] diff --git a/radis/pgsearch/utils/indexing.py b/radis/pgsearch/utils/indexing.py index c69d4882..a58ea2d8 100644 --- a/radis/pgsearch/utils/indexing.py +++ b/radis/pgsearch/utils/indexing.py @@ -8,6 +8,7 @@ from ..models import ReportSearchIndex from .language_utils import code_to_language +from .projection import sync_projection logger = logging.getLogger(__name__) @@ -70,3 +71,5 @@ def bulk_upsert_report_search_indexes( """, [config, config_ids], ) + + sync_projection(chunk) diff --git a/radis/pgsearch/utils/projection.py b/radis/pgsearch/utils/projection.py new file mode 100644 index 00000000..40cef5fa --- /dev/null +++ b/radis/pgsearch/utils/projection.py @@ -0,0 +1,48 @@ +"""The canonical SQL that fills the ReportSearchIndex search projection. + +Triggers (migration 0004) maintain the projection when its sources change. +This module fills it when an index row is first created, which happens before +the report's groups and modalities are attached. The backfill migration +duplicates this statement in SQL, the same way bulk_upsert_report_search_indexes +already duplicates the tsvector logic -- keep them in sync. +""" + +from collections.abc import Iterable + +from django.db import connection + +PROJECTION_UPDATE_SQL = """ +UPDATE pgsearch_reportsearchindex rsi +SET group_ids = COALESCE(g.ids, '{}'), + modality_codes = COALESCE(m.codes, '{}'), + language_code = l.code, + patient_sex = r.patient_sex, + patient_age = r.patient_age, + patient_id = r.patient_id, + study_datetime = r.study_datetime, + study_description = r.study_description, + report_created_at = r.created_at, + report_updated_at = r.updated_at +FROM reports_report r +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +WHERE rsi.report_id = r.id AND rsi.report_id = ANY(%s) +""" + + +def sync_projection(report_ids: Iterable[int]) -> None: + """Fill the projection columns for the given reports from their sources.""" + ids = [int(report_id) for report_id in report_ids] + if not ids: + return + with connection.cursor() as cursor: + cursor.execute(PROJECTION_UPDATE_SQL, [ids]) From 0b90ec904839020734b760a8ff0c0dd407cc7601 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 19:53:43 +0000 Subject: [PATCH 16/50] Fix vacuous creation test to actually exercise sync_projection ReportFactory.create() masked the gap this test was meant to pin: its post_generation hooks make factory_boy issue an implicit extra save(), which fires the AFTER UPDATE trigger from migration 0004 and populates the scalars regardless of whether the signal's sync_projection() call exists. Building the report and saving it explicitly is a single INSERT with no follow-up UPDATE, so only sync_projection() can fill the scalars. --- radis/pgsearch/tests/test_search_projection.py | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 85c52ccd..2e6d5e55 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -143,7 +143,18 @@ def test_patient_age_is_mirrored(): def test_creation_populates_the_mirrored_scalars(): - report = ReportFactory.create(language=LanguageFactory.create(code="de")) + """Build + explicit save() is a single INSERT with no follow-up UPDATE. + + ReportFactory.create() would not pin this: its post_generation hooks + (modalities, the metadata RelatedFactoryList) make factory_boy issue an + implicit extra save() afterwards, which fires the AFTER UPDATE trigger + from migration 0004 and would populate these scalars regardless of + whether the signal's own sync_projection() call exists. build() skips + those hooks, so the only thing that can fill the scalars here is the + signal's sync_projection() call on creation. + """ + report = ReportFactory.build(language=LanguageFactory.create(code="de")) + report.save() index = ReportSearchIndex.objects.get(report=report) assert index.language_code == "de" From 15c71d77f5cdeceaa8d5781893fe63595813a568 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 20:05:47 +0000 Subject: [PATCH 17/50] Backfill the search projection in 50k-row chunks --- .../0005_search_projection_backfill.py | 67 +++++++++++++++++++ .../pgsearch/tests/test_search_projection.py | 22 ++++++ 2 files changed, 89 insertions(+) create mode 100644 radis/pgsearch/migrations/0005_search_projection_backfill.py diff --git a/radis/pgsearch/migrations/0005_search_projection_backfill.py b/radis/pgsearch/migrations/0005_search_projection_backfill.py new file mode 100644 index 00000000..676d43e3 --- /dev/null +++ b/radis/pgsearch/migrations/0005_search_projection_backfill.py @@ -0,0 +1,67 @@ +"""Backfill the ReportSearchIndex search projection for existing rows. + +atomic = False so each chunk commits on its own: at the 8M design target this +runs for about nine and a half minutes, and one transaction that long would +pin an equally long-lived snapshot. + +Runs after the triggers (0004) on purpose. A report edited during the backfill +is corrected by its trigger, and a chunk that later reprocesses the same row +simply rewrites the current values. + +The statement below duplicates utils/projection.PROJECTION_UPDATE_SQL, the same +way bulk_upsert_report_search_indexes duplicates the tsvector logic. Migrations +must not import app code, which drifts under them. +""" + +from django.db import migrations + +CHUNK_SIZE = 50_000 + +BACKFILL_SQL = """ +UPDATE pgsearch_reportsearchindex rsi +SET group_ids = COALESCE(g.ids, '{}'), + modality_codes = COALESCE(m.codes, '{}'), + language_code = l.code, + patient_sex = r.patient_sex, + patient_age = r.patient_age, + patient_id = r.patient_id, + study_datetime = r.study_datetime, + study_description = r.study_description, + report_created_at = r.created_at, + report_updated_at = r.updated_at +FROM reports_report r +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +WHERE rsi.report_id = r.id AND rsi.report_id > %s AND rsi.report_id <= %s +""" + + +def backfill(apps, schema_editor): + connection = schema_editor.connection + with connection.cursor() as cursor: + cursor.execute("SELECT COALESCE(MAX(report_id), 0) FROM pgsearch_reportsearchindex") + max_id = cursor.fetchone()[0] + + low = 0 + while low < max_id: + high = low + CHUNK_SIZE + with connection.cursor() as cursor: + cursor.execute(BACKFILL_SQL, [low, high]) + low = high + + +class Migration(migrations.Migration): + atomic = False + + dependencies = [("pgsearch", "0004_search_projection_triggers")] + + operations = [migrations.RunPython(backfill, migrations.RunPython.noop)] diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 2e6d5e55..a72b1919 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -174,3 +174,25 @@ def test_bulk_upsert_populates_the_projection(): assert index.language_code == "en" assert index.group_ids == [group.pk] assert index.modality_codes == ["CT"] + + +def test_backfill_fills_rows_that_predate_the_projection(): + """Simulates a row written before the projection existed.""" + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=["MR"]) + group = GroupFactory.create() + report.groups.add(group) + + ReportSearchIndex.objects.filter(report=report).update( + group_ids=[], modality_codes=[], language_code=None, patient_id=None + ) + + from radis.pgsearch.utils.projection import sync_projection + + sync_projection([report.pk]) + + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk] + assert index.modality_codes == ["MR"] + assert index.language_code == "en" + assert index.patient_id == report.patient_id From 1c1f9857bdd4400812698d220455b03ee4d0e7eb Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 20:13:15 +0000 Subject: [PATCH 18/50] Add test that exercises the migration's backfill() SQL directly --- .../pgsearch/tests/test_search_projection.py | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index a72b1919..9bda51e1 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -196,3 +196,59 @@ def test_backfill_fills_rows_that_predate_the_projection(): assert index.modality_codes == ["MR"] assert index.language_code == "en" assert index.patient_id == report.patient_id + + +def test_migration_backfill_executes_the_chunked_update(): + """Verify the migration's backfill() actually executes and fills the projection. + + This test exercises the literal SQL in 0005_search_projection_backfill.py, + not sync_projection(). It ensures group_ids and other access control fields + are restored by the migration's own UPDATE statement. + """ + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=["CT"]) + group = GroupFactory.create() + report.groups.add(group) + + # Verify projection is populated first + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk], "Setup: projection should be populated" + assert index.language_code == "en" + + # Clear projection fields to simulate a row predating the feature + ReportSearchIndex.objects.filter(report=report).update( + group_ids=[], + modality_codes=[], + language_code=None, + patient_sex=None, + patient_age=None, + patient_id=None, + study_datetime=None, + study_description=None, + report_created_at=None, + report_updated_at=None, + ) + + # Import and call the migration's backfill function + import importlib + + from django.db import connection + + migration_module = importlib.import_module( + "radis.pgsearch.migrations.0005_search_projection_backfill" + ) + + # Create a minimal schema_editor-like object that the backfill function needs + class MinimalSchemaEditor: + def __init__(self, connection): + self.connection = connection + + schema_editor = MinimalSchemaEditor(connection) + migration_module.backfill(None, schema_editor) + + # Verify the projection is now restored by the migration's SQL + index.refresh_from_db() + assert index.group_ids == [group.pk], "Migration backfill should restore group_ids" + assert index.modality_codes == ["CT"], "Migration backfill should restore modality_codes" + assert index.language_code == "en", "Migration backfill should restore language_code" + assert index.patient_id == report.patient_id From a0469f4ad38d9af84940ec059ebe8bfe55442acb Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 20:32:55 +0000 Subject: [PATCH 19/50] Add the search projection indexes and disable GIN fastupdate Adds the two measured indexes to ReportSearchIndex: a GIN index on group_ids (up to ~170x faster access-control filtering on small groups) and a btree on report_updated_at (serves the subscription filter() path, which has no tsquery to drive it). Created after the backfill so the backfill could rely on HOT updates. Also flips fastupdate off on the search_vector GIN index and drains its pending list, since RADIS writes in async batches and reads interactively. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../0006_search_projection_indexes.py | 31 +++++++++++++++++++ radis/pgsearch/models.py | 2 ++ .../pgsearch/tests/test_search_projection.py | 11 +++++++ 3 files changed, 44 insertions(+) create mode 100644 radis/pgsearch/migrations/0006_search_projection_indexes.py diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0006_search_projection_indexes.py new file mode 100644 index 00000000..fa48df11 --- /dev/null +++ b/radis/pgsearch/migrations/0006_search_projection_indexes.py @@ -0,0 +1,31 @@ +# Generated by Django 6.0.7 on 2026-08-30 20:20 + +import django.contrib.postgres.indexes +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pgsearch', '0005_search_projection_backfill'), + ('reports', '0013_alter_report_options'), + ] + + operations = [ + migrations.AddIndex( + model_name='reportsearchindex', + index=django.contrib.postgres.indexes.GinIndex(fields=['group_ids'], name='pgsearch_group_ids_gin'), + ), + migrations.AddIndex( + model_name='reportsearchindex', + index=models.Index(fields=['report_updated_at'], name='pgsearch_report_updated_at_idx'), + ), + migrations.RunSQL( + # fastupdate leaves an unsorted pending list that every query must + # scan; RADIS writes in async batches and reads interactively, so + # predictable read latency is the better side of the trade. + "ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = off);" + "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", + reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", + ), + ] diff --git a/radis/pgsearch/models.py b/radis/pgsearch/models.py index ab0feb93..fa2eaeb5 100644 --- a/radis/pgsearch/models.py +++ b/radis/pgsearch/models.py @@ -58,6 +58,8 @@ class Meta: condition=models.Q(embedding__isnull=True), name="pgsearch_pending_embedding_idx", ), + GinIndex(fields=["group_ids"], name="pgsearch_group_ids_gin"), + models.Index(fields=["report_updated_at"], name="pgsearch_report_updated_at_idx"), ] def __str__(self) -> str: diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 9bda51e1..d126ae78 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -252,3 +252,14 @@ def __init__(self, connection): assert index.modality_codes == ["CT"], "Migration backfill should restore modality_codes" assert index.language_code == "en", "Migration backfill should restore language_code" assert index.patient_id == report.patient_id + + +def test_projection_indexes_exist(): + with connection.cursor() as cursor: + cursor.execute( + "SELECT indexname FROM pg_indexes WHERE tablename = 'pgsearch_reportsearchindex'" + ) + names = {row[0] for row in cursor.fetchall()} + + assert "pgsearch_group_ids_gin" in names + assert "pgsearch_report_updated_at_idx" in names From c4576a95f76fb34f781db029fd5abf72fd2df793 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 20:43:14 +0000 Subject: [PATCH 20/50] Add check_search_projection drift detection command The search projection duplicates access-control data (group_ids) and other Report/Language fields the FTS scan filters on. Triggers keep it in sync, but a Language.code rename and any writer that bypasses the triggers can still drift it from its sources. This command compares ReportSearchIndex against Report/Language/group/modality membership and exits non-zero with a per-column drift count when they disagree, so operators can verify integrity after a restore, bulk import, or migration. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../commands/check_search_projection.py | 58 +++++++++++++++++++ .../tests/test_check_search_projection.py | 30 ++++++++++ 2 files changed, 88 insertions(+) create mode 100644 radis/pgsearch/management/commands/check_search_projection.py create mode 100644 radis/pgsearch/tests/test_check_search_projection.py diff --git a/radis/pgsearch/management/commands/check_search_projection.py b/radis/pgsearch/management/commands/check_search_projection.py new file mode 100644 index 00000000..c80daa38 --- /dev/null +++ b/radis/pgsearch/management/commands/check_search_projection.py @@ -0,0 +1,58 @@ +"""Verify the ReportSearchIndex search projection against its sources.""" + +from django.core.management.base import BaseCommand, CommandError +from django.db import connection + +DRIFT_SQL = """ +SELECT + count(*) FILTER (WHERE rsi.group_ids IS DISTINCT FROM COALESCE(g.ids, '{}')) + AS group_ids, + count(*) FILTER (WHERE rsi.modality_codes IS DISTINCT FROM COALESCE(m.codes, '{}')) + AS modality_codes, + count(*) FILTER (WHERE rsi.language_code IS DISTINCT FROM l.code) + AS language_code, + count(*) FILTER (WHERE rsi.patient_sex IS DISTINCT FROM r.patient_sex) + AS patient_sex, + count(*) FILTER (WHERE rsi.patient_age IS DISTINCT FROM r.patient_age) + AS patient_age, + count(*) FILTER (WHERE rsi.patient_id IS DISTINCT FROM r.patient_id) + AS patient_id, + count(*) FILTER (WHERE rsi.study_datetime IS DISTINCT FROM r.study_datetime) + AS study_datetime, + count(*) FILTER (WHERE rsi.study_description IS DISTINCT FROM r.study_description) + AS study_description, + count(*) FILTER (WHERE rsi.report_created_at IS DISTINCT FROM r.created_at) + AS report_created_at, + count(*) FILTER (WHERE rsi.report_updated_at IS DISTINCT FROM r.updated_at) + AS report_updated_at +FROM pgsearch_reportsearchindex rsi +JOIN reports_report r ON r.id = rsi.report_id +LEFT JOIN reports_language l ON l.id = r.language_id +LEFT JOIN LATERAL ( + SELECT array_agg(rg.group_id ORDER BY rg.group_id) AS ids + FROM reports_report_groups rg WHERE rg.report_id = r.id +) g ON true +LEFT JOIN LATERAL ( + SELECT array_agg(mo.code ORDER BY mo.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality mo ON mo.id = rm.modality_id + WHERE rm.report_id = r.id +) m ON true +""" + + +class Command(BaseCommand): + help = "Verify the ReportSearchIndex search projection against its sources." + + def handle(self, *args, **options): + with connection.cursor() as cursor: + cursor.execute(DRIFT_SQL) + columns = [column.name for column in cursor.description] + counts = dict(zip(columns, cursor.fetchone(), strict=True)) + + drifted = {name: count for name, count in counts.items() if count} + if drifted: + details = ", ".join(f"{name}: {count}" for name, count in sorted(drifted.items())) + raise CommandError(f"Search projection drift detected -- {details}") + + self.stdout.write(self.style.SUCCESS("Search projection matches its sources.")) diff --git a/radis/pgsearch/tests/test_check_search_projection.py b/radis/pgsearch/tests/test_check_search_projection.py new file mode 100644 index 00000000..e2ed19c9 --- /dev/null +++ b/radis/pgsearch/tests/test_check_search_projection.py @@ -0,0 +1,30 @@ +"""The projection duplicates access-control data, so operators need a way to +prove it still matches its sources -- after a restore, a bulk import, or a +Language.code rename, which no trigger covers.""" + +import pytest +from adit_radis_shared.accounts.factories import GroupFactory +from django.core.management import call_command +from django.core.management.base import CommandError + +from radis.pgsearch.models import ReportSearchIndex +from radis.reports.factories import LanguageFactory, ReportFactory + +pytestmark = pytest.mark.django_db + + +def test_reports_no_drift_on_a_healthy_corpus(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + + call_command("check_search_projection") + + +def test_detects_drifted_group_ids(): + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + + ReportSearchIndex.objects.filter(report=report).update(group_ids=[9999]) + + with pytest.raises(CommandError, match="group_ids"): + call_command("check_search_projection") From bbbc04ca126199d1c7ed02c7454734acbbaa57d4 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 20:54:05 +0000 Subject: [PATCH 21/50] Make the FTS filter query single-table _build_filter_query traversed report__ joins for every predicate, forcing the candidate query through Report/Language/Modality even for simple filters -- the join that made common-term searches take seconds. It now reads the search projection columns on ReportSearchIndex directly: group membership becomes an array containment check (empty array when filters.group is None, staying fail-closed), date filters become sargable half-open ranges instead of a per-row timezone-converting __date lookup, and the modalities filter becomes an array overlap. With no joins left, the modalities predicate can no longer duplicate a row, so .distinct() is dropped from both the FTS and vector querysets in _fuse_hybrid. test_filter_query_equivalence.py keeps the old joined implementation as an oracle and asserts the new query selects the same report ids across a filter matrix, plus dedicated cases for the fail-closed group=None behavior and the removed .distinct(). --- radis/pgsearch/providers.py | 66 ++++--- .../tests/test_filter_query_equivalence.py | 164 ++++++++++++++++++ 2 files changed, 203 insertions(+), 27 deletions(-) create mode 100644 radis/pgsearch/tests/test_filter_query_equivalence.py diff --git a/radis/pgsearch/providers.py b/radis/pgsearch/providers.py index 22c39ff6..420db040 100644 --- a/radis/pgsearch/providers.py +++ b/radis/pgsearch/providers.py @@ -3,6 +3,7 @@ import logging import unicodedata from collections.abc import Iterator +from datetime import date, datetime, time, timedelta from typing import Literal, NamedTuple, cast import openai @@ -10,6 +11,7 @@ from django.contrib.postgres.search import SearchHeadline, SearchQuery, SearchRank from django.core.cache import cache from django.db.models import Case, F, FloatField, Q, TextField, Value, When +from django.utils import timezone from pgvector.django import CosineDistance from radis.core.utils.embedding_client import ( @@ -186,42 +188,54 @@ def _exclude_negations(queryset, node: QueryNode, configs: list[tuple[str, list[ return queryset +def _local_day_start(day: date) -> datetime: + """Midnight of ``day`` in the active timezone, as an aware datetime.""" + return timezone.make_aware(datetime.combine(day, time.min)) + + def _build_filter_query(filters: SearchFilters) -> Q: - # Group-scoped access control: a report is only visible to a group when its - # ``groups`` M2M includes that group. This mirrors the report access model - # used everywhere else (e.g. ``Report.objects.filter(groups=active_group)`` - # in radis.reports.views.ReportListView/ReportDetailView and the - # ``groups=active_group`` lookup in radis.chats.views). ``SearchView`` - # supplies ``group=active_group.pk``; the extraction search preview may pass - # ``group=None`` when the user has no active group, which is fail-closed: - # ``Q(report__groups=None)`` becomes ``groups__isnull=True`` and matches only - # reports assigned to no group at all. The filter stays unconditional; without - # it the search would leak reports belonging only to other groups. - fq = Q(report__groups=filters.group) + # Every predicate here is a column on ReportSearchIndex itself -- the search + # projection (see migration 0003) mirrors the Report fields search filters + # on, so the candidate query stays single-table. Reintroducing a ``report__`` + # traversal restores the join and the multi-second query shape it caused; + # test_filter_query_plan_is_single_table guards against that. + # + # Group-scoped access control. ``SearchView`` supplies + # ``group=active_group.pk``; the extraction preview may pass ``group=None`` + # when the user has no active group, which is fail-closed: an empty + # ``group_ids`` matches only reports assigned to no group at all. + if filters.group is None: + fq = Q(group_ids=[]) + else: + fq = Q(group_ids__contains=[filters.group]) # Apply hard filter criteria if filters.patient_sex: - fq &= Q(report__patient_sex=filters.patient_sex) + fq &= Q(patient_sex=filters.patient_sex) if filters.language: - fq &= Q(report__language__code=filters.language) + fq &= Q(language_code=filters.language) if filters.modalities: - fq &= Q(report__modalities__code__in=filters.modalities) + fq &= Q(modality_codes__overlap=filters.modalities) + # Half-open ranges rather than ``__date``: Django compiles that lookup to + # ``(study_datetime AT TIME ZONE ...)::date``, a function over the column + # evaluated once per row -- millions of timezone conversions per query on + # the sequential scan, for a predicate equal to two timestamp comparisons. if filters.study_date_from: - fq &= Q(report__study_datetime__date__gte=filters.study_date_from) + fq &= Q(study_datetime__gte=_local_day_start(filters.study_date_from)) if filters.study_date_till: - fq &= Q(report__study_datetime__date__lte=filters.study_date_till) + fq &= Q(study_datetime__lt=_local_day_start(filters.study_date_till + timedelta(days=1))) if filters.study_description: - fq &= Q(report__study_description__icontains=filters.study_description) + fq &= Q(study_description__icontains=filters.study_description) if filters.patient_age_from is not None: - fq &= Q(report__patient_age__gte=filters.patient_age_from) + fq &= Q(patient_age__gte=filters.patient_age_from) if filters.patient_age_till is not None: - fq &= Q(report__patient_age__lte=filters.patient_age_till) + fq &= Q(patient_age__lte=filters.patient_age_till) if filters.patient_id: - fq &= Q(report__patient_id=filters.patient_id) + fq &= Q(patient_id=filters.patient_id) if filters.created_after: - fq &= Q(report__created_at__gte=filters.created_after) + fq &= Q(report_created_at__gte=filters.created_after) if filters.created_before: - fq &= Q(report__created_at__lte=filters.created_before) + fq &= Q(report_created_at__lte=filters.created_before) if filters.labels: from radis.labels.models import LabelResult from radis.reports.models import Report @@ -230,9 +244,9 @@ def _build_filter_query(filters: SearchFilters) -> Q: label_results__label__name__in=filters.labels, label_results__value__in=LabelResult.SURFACING_VALUES, ).values("pk") - fq &= Q(report__in=surfacing_report_ids) + fq &= Q(report_id__in=surfacing_report_ids) if filters.updated_after: - fq &= Q(report__updated_at__gte=filters.updated_after) + fq &= Q(report_updated_at__gte=filters.updated_after) return fq @@ -391,8 +405,7 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: vec_qs = ReportSearchIndex.objects.filter(filter_query) vec_qs = _exclude_negations(vec_qs, search.query, configs) vec_rows = list( - vec_qs.distinct() - .exclude(embedding__isnull=True) + vec_qs.exclude(embedding__isnull=True) .annotate(distance=CosineDistance("embedding", query_vec)) .order_by("distance", "report_id") .values_list("report_id", "distance")[: settings.HYBRID_VECTOR_TOP_K] @@ -409,7 +422,6 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: if not configs else list( ReportSearchIndex.objects.filter(filter_query) - .distinct() .filter(match_q) .annotate(rank=rank_expr) .order_by("-rank", "report_id") diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py new file mode 100644 index 00000000..d954c0f1 --- /dev/null +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -0,0 +1,164 @@ +"""The new single-table filter must select exactly what the joined one did. + +_build_filter_query_legacy is the pre-change implementation kept as a reference +oracle. Delete both it and this module once the change has settled in production. +""" + +from datetime import date, timedelta + +import pytest +from adit_radis_shared.accounts.factories import GroupFactory +from django.db.models import Q +from django.utils import timezone + +from radis.pgsearch.models import ReportSearchIndex +from radis.pgsearch.providers import _build_filter_query +from radis.reports.factories import LanguageFactory, ReportFactory +from radis.search.site import SearchFilters + +pytestmark = pytest.mark.django_db + + +def _build_filter_query_legacy(filters: SearchFilters) -> Q: + """The joined implementation this change replaces.""" + fq = Q(report__groups=filters.group) + if filters.patient_sex: + fq &= Q(report__patient_sex=filters.patient_sex) + if filters.language: + fq &= Q(report__language__code=filters.language) + if filters.modalities: + fq &= Q(report__modalities__code__in=filters.modalities) + if filters.study_date_from: + fq &= Q(report__study_datetime__date__gte=filters.study_date_from) + if filters.study_date_till: + fq &= Q(report__study_datetime__date__lte=filters.study_date_till) + if filters.study_description: + fq &= Q(report__study_description__icontains=filters.study_description) + if filters.patient_age_from is not None: + fq &= Q(report__patient_age__gte=filters.patient_age_from) + if filters.patient_age_till is not None: + fq &= Q(report__patient_age__lte=filters.patient_age_till) + if filters.patient_id: + fq &= Q(report__patient_id=filters.patient_id) + if filters.updated_after: + fq &= Q(report__updated_at__gte=filters.updated_after) + return fq + + +def _ids(fq: Q) -> set[int]: + return set(ReportSearchIndex.objects.filter(fq).distinct().values_list("report_id", flat=True)) + + +@pytest.fixture +def corpus(): + language_en = LanguageFactory.create(code="en") + language_de = LanguageFactory.create(code="de") + group_a = GroupFactory.create(name="A") + group_b = GroupFactory.create(name="B") + now = timezone.now() + + first = ReportFactory.create( + language=language_en, + patient_sex="M", + patient_id="P1", + study_description="CT Thorax", + study_datetime=now - timedelta(days=10), + modalities=["CT"], + ) + first.groups.add(group_a) + + second = ReportFactory.create( + language=language_de, + patient_sex="F", + patient_id="P2", + study_description="MR Head", + study_datetime=now - timedelta(days=400), + modalities=["MR", "CT"], + ) + second.groups.add(group_a, group_b) + + orphan = ReportFactory.create(language=language_en, patient_sex="M", patient_id="P3") + + return {"group_a": group_a, "group_b": group_b, "orphan": orphan} + + +@pytest.mark.parametrize( + "make_filters", + [ + pytest.param(lambda c: SearchFilters(group=c["group_a"].pk), id="group-only"), + pytest.param(lambda c: SearchFilters(group=c["group_b"].pk), id="other-group"), + pytest.param(lambda c: SearchFilters(group=c["group_a"].pk, language="en"), id="language"), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, modalities=["CT"]), id="modality" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, modalities=["CT", "MR"]), + id="modalities-multi", + ), + pytest.param(lambda c: SearchFilters(group=c["group_a"].pk, patient_sex="M"), id="sex"), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_id="P1"), id="patient-id" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, study_description="thorax"), + id="description-icontains", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, study_date_from=date.today() - timedelta(days=30) + ), + id="date-from", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, study_date_till=date.today() - timedelta(days=5) + ), + id="date-till", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, + study_date_from=date.today() - timedelta(days=30), + study_date_till=date.today(), + ), + id="date-range", + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_age_from=0), id="age-from" + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, patient_age_till=200), id="age-till" + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, updated_after=timezone.now() - timedelta(hours=1) + ), + id="updated-after", + ), + ], +) +def test_new_filter_matches_the_legacy_one(corpus, make_filters): + filters = make_filters(corpus) + + assert _ids(_build_filter_query(filters)) == _ids(_build_filter_query_legacy(filters)) + + +def test_group_none_is_fail_closed(corpus): + """group=None must match only reports in no group at all -- never everything.""" + result = _ids(_build_filter_query(SearchFilters(group=None))) + + assert result == {corpus["orphan"].pk} + + +def test_no_duplicates_without_distinct(corpus): + """A report matching on two modalities must appear once, which is what the + removed .distinct() used to guarantee.""" + filters = SearchFilters(group=corpus["group_a"].pk, modalities=["CT", "MR"]) + + rows = list( + ReportSearchIndex.objects.filter(_build_filter_query(filters)).values_list( + "report_id", flat=True + ) + ) + + assert len(rows) == len(set(rows)) From 349418e8624c69ac7e022093c30c448e89bce6d2 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:07:47 +0000 Subject: [PATCH 22/50] Strengthen the filter-query equivalence suite per review Two gaps the review found in the oracle test from the previous commit: - The date parametrisation used only "N days ago" filters, which never land on or near a report's exact date. An off-by-one in the new half-open range, a wrong inclusive/exclusive boundary, or a UTC-vs-local mistake inside _local_day_start were all invisible to it. Add reports pinned to explicit local midnight, 23:30 local, and midnight of the following day on a fixed day, plus a second run of the same check under override_settings(TIME_ZONE="America/New_York") so local midnight isn't incidentally equal to UTC midnight. - The legacy oracle was missing created_after, created_before and labels -- three predicates the pre-change implementation had. labels is the one whose lookup shape actually changed (report__in -> report_id__in) and had zero equivalence coverage. Restore all three to the oracle and add parametrised cases for each, including a corpus report with a surfacing LabelResult. Verified the new boundary tests have teeth: temporarily reverting the upper-bound fix to the old __lte-on-day-start form made both boundary tests fail (dropping the 23:30 report), then reverting back turned them green again. --- .../tests/test_filter_query_equivalence.py | 134 +++++++++++++++++- 1 file changed, 132 insertions(+), 2 deletions(-) diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py index d954c0f1..ec2a97a0 100644 --- a/radis/pgsearch/tests/test_filter_query_equivalence.py +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -4,13 +4,16 @@ oracle. Delete both it and this module once the change has settled in production. """ -from datetime import date, timedelta +from datetime import date, datetime, time, timedelta import pytest from adit_radis_shared.accounts.factories import GroupFactory from django.db.models import Q +from django.test import override_settings from django.utils import timezone +from radis.labels.factories import LabelFactory, LabelResultFactory +from radis.labels.models import LabelResult from radis.pgsearch.models import ReportSearchIndex from radis.pgsearch.providers import _build_filter_query from radis.reports.factories import LanguageFactory, ReportFactory @@ -18,6 +21,11 @@ pytestmark = pytest.mark.django_db +# A fixed calendar day used for the date-boundary tests below, well away from +# any of the corpus fixture's "N days ago" reports so it can't accidentally +# overlap with those. +BOUNDARY_DAY = date(2026, 3, 10) + def _build_filter_query_legacy(filters: SearchFilters) -> Q: """The joined implementation this change replaces.""" @@ -40,6 +48,18 @@ def _build_filter_query_legacy(filters: SearchFilters) -> Q: fq &= Q(report__patient_age__lte=filters.patient_age_till) if filters.patient_id: fq &= Q(report__patient_id=filters.patient_id) + if filters.created_after: + fq &= Q(report__created_at__gte=filters.created_after) + if filters.created_before: + fq &= Q(report__created_at__lte=filters.created_before) + if filters.labels: + from radis.reports.models import Report + + surfacing_report_ids = Report.objects.filter( + label_results__label__name__in=filters.labels, + label_results__value__in=LabelResult.SURFACING_VALUES, + ).values("pk") + fq &= Q(report__in=surfacing_report_ids) if filters.updated_after: fq &= Q(report__updated_at__gte=filters.updated_after) return fq @@ -66,6 +86,8 @@ def corpus(): modalities=["CT"], ) first.groups.add(group_a) + pneumonia = LabelFactory.create(name="pneumonia") + LabelResultFactory.create(report=first, label=pneumonia, value=LabelResult.Value.PRESENT) second = ReportFactory.create( language=language_de, @@ -79,7 +101,52 @@ def corpus(): orphan = ReportFactory.create(language=language_en, patient_sex="M", patient_id="P3") - return {"group_a": group_a, "group_b": group_b, "orphan": orphan} + return {"group_a": group_a, "group_b": group_b, "orphan": orphan, "first": first} + + +def _make_boundary_corpus(): + """Reports pinned to explicit local times around BOUNDARY_DAY. + + "N days ago" filters (as used by the rest of this module) never land on or + near a report's exact date, so they cannot catch an off-by-one at the day + boundary or a UTC-vs-local mistake in ``_local_day_start``. These reports + are placed exactly where such mistakes would show: at local midnight + (the inclusive lower bound), at 23:30 local (deep inside the last day, but + the first casualty of an accidentally inclusive-upper-bound-on-exact-day + mistake), and at local midnight of the day *after* (which must be + excluded). + + A plain helper rather than a fixture: the non-UTC test below must create + these reports *and* run the query while the same overridden ``TIME_ZONE`` + is active, since "local midnight" is only meaningful relative to whatever + timezone is active at the moment ``timezone.make_aware`` runs. A fixture + is resolved before an ``@override_settings``-decorated test body starts, + which would pin the reports to the wrong timezone. + """ + language_en = LanguageFactory.create(code="en") + group = GroupFactory.create(name="Boundary") + + midnight = ReportFactory.create( + language=language_en, + study_datetime=timezone.make_aware(datetime.combine(BOUNDARY_DAY, time.min)), + ) + midnight.groups.add(group) + + late = ReportFactory.create( + language=language_en, + study_datetime=timezone.make_aware(datetime.combine(BOUNDARY_DAY, time(23, 30))), + ) + late.groups.add(group) + + next_midnight = ReportFactory.create( + language=language_en, + study_datetime=timezone.make_aware( + datetime.combine(BOUNDARY_DAY + timedelta(days=1), time.min) + ), + ) + next_midnight.groups.add(group) + + return {"group": group, "midnight": midnight, "late": late, "next_midnight": next_midnight} @pytest.mark.parametrize( @@ -129,6 +196,21 @@ def corpus(): pytest.param( lambda c: SearchFilters(group=c["group_a"].pk, patient_age_till=200), id="age-till" ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, created_after=timezone.now() - timedelta(hours=1) + ), + id="created-after", + ), + pytest.param( + lambda c: SearchFilters( + group=c["group_a"].pk, created_before=timezone.now() + timedelta(hours=1) + ), + id="created-before", + ), + pytest.param( + lambda c: SearchFilters(group=c["group_a"].pk, labels=["pneumonia"]), id="labels" + ), pytest.param( lambda c: SearchFilters( group=c["group_a"].pk, updated_after=timezone.now() - timedelta(hours=1) @@ -162,3 +244,51 @@ def test_no_duplicates_without_distinct(corpus): ) assert len(rows) == len(set(rows)) + + +def test_date_range_boundary_matches_legacy(): + """The half-open range must include local midnight and 23:30 local on the + selected day, and exclude local midnight of the following day. + + This is the exact hazard the brief calls out: an off-by-one + (``__lte=_local_day_start(till)`` instead of + ``__lt=_local_day_start(till + 1 day)``) would silently drop the 23:30 + report here, and an inclusive-instead-of-exclusive upper bound would wrongly + keep next_midnight -- neither is visible from "N days ago" filters alone. + """ + boundary = _make_boundary_corpus() + filters = SearchFilters( + group=boundary["group"].pk, + study_date_from=BOUNDARY_DAY, + study_date_till=BOUNDARY_DAY, + ) + + new_ids = _ids(_build_filter_query(filters)) + legacy_ids = _ids(_build_filter_query_legacy(filters)) + + assert new_ids == legacy_ids + assert new_ids == {boundary["midnight"].pk, boundary["late"].pk} + + +def test_date_range_boundary_matches_legacy_in_non_utc_timezone(): + """Same boundary check under a timezone hours away from UTC. + + A mistake inside ``_local_day_start`` that resolved midnight in UTC rather + than the active timezone would only misplace the boundary here -- under + UTC (the default test timezone) local and UTC midnight coincide and the + bug is invisible. The corpus is built *inside* the override so "local + midnight" is pinned relative to the same timezone the query later uses. + """ + with override_settings(TIME_ZONE="America/New_York"): + boundary = _make_boundary_corpus() + filters = SearchFilters( + group=boundary["group"].pk, + study_date_from=BOUNDARY_DAY, + study_date_till=BOUNDARY_DAY, + ) + + new_ids = _ids(_build_filter_query(filters)) + legacy_ids = _ids(_build_filter_query_legacy(filters)) + + assert new_ids == legacy_ids + assert new_ids == {boundary["midnight"].pk, boundary["late"].pk} From 84186529e7b7e52c71047d011a992e3aa54ef3d0 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:30:10 +0000 Subject: [PATCH 23/50] Read language_code from the projection instead of joining Five report__language__code__in traversals in providers.py each re-added the reports_report and reports_language joins to the candidate query, undoing the single-table projection from the filter-query rewrite. Read ReportSearchIndex.language_code directly instead. --- radis/pgsearch/providers.py | 15 +++++---------- .../tests/test_filter_query_equivalence.py | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/radis/pgsearch/providers.py b/radis/pgsearch/providers.py index 420db040..0abdf070 100644 --- a/radis/pgsearch/providers.py +++ b/radis/pgsearch/providers.py @@ -182,9 +182,7 @@ def _exclude_negations(queryset, node: QueryNode, configs: list[tuple[str, list[ return queryset for config, codes in configs: negative_tsquery = SearchQuery(negative_query_str, search_type="raw", config=config) - queryset = queryset.exclude( - Q(report__language__code__in=codes) & Q(search_vector=negative_tsquery) - ) + queryset = queryset.exclude(Q(language_code__in=codes) & Q(search_vector=negative_tsquery)) return queryset @@ -373,7 +371,7 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: # A document matches only under the configuration it was indexed with. match_q = Q() for config, codes in configs: - match_q |= Q(report__language__code__in=codes, search_vector=tsqueries[config]) + match_q |= Q(language_code__in=codes, search_vector=tsqueries[config]) # Rank strictly under the document's own configuration. Do not swap this for # Greatest over all configurations: a foreign config's ts_rank is not @@ -382,7 +380,7 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: rank_expr = Case( *[ When( - report__language__code__in=codes, + language_code__in=codes, then=SearchRank(F("search_vector"), tsqueries[config]), ) for config, codes in configs @@ -488,17 +486,14 @@ def _headline(config: str) -> SearchHeadline: # highlights/scores nothing. A one-branch Case is legal (unlike Greatest), # so this covers the single-configuration case too with no special-casing. summary_expr = Case( - *[ - When(report__language__code__in=codes, then=_headline(config)) - for config, codes in configs - ], + *[When(language_code__in=codes, then=_headline(config)) for config, codes in configs], default=Value(""), output_field=TextField(), ) rank_expr = Case( *[ When( - report__language__code__in=codes, + language_code__in=codes, then=SearchRank(F("search_vector"), tsqueries[config]), ) for config, codes in configs diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py index ec2a97a0..8599ce1d 100644 --- a/radis/pgsearch/tests/test_filter_query_equivalence.py +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -270,6 +270,25 @@ def test_date_range_boundary_matches_legacy(): assert new_ids == {boundary["midnight"].pk, boundary["late"].pk} +def test_no_report_traversal_remains_in_the_candidate_query(): + """Any report__ lookup re-adds a join and the multi-second query shape.""" + import inspect + + from radis.pgsearch import providers + + source = inspect.getsource(providers) + offending = [ + line.strip() + for line in source.splitlines() + if "report__" in line + and "report__body" not in line # hydration headline, deliberately joined + and "report__document_id" not in line # filter(), deliberately joined + and not line.strip().startswith("#") + ] + + assert offending == [], f"report__ traversals left in the hot path: {offending}" + + def test_date_range_boundary_matches_legacy_in_non_utc_timezone(): """Same boundary check under a timezone hours away from UTC. From 0cea4adfba15eab4f6d2a224abd1e6d2e6929286 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:39:22 +0000 Subject: [PATCH 24/50] Assert the candidate query plan stays single-table --- .../tests/test_filter_query_equivalence.py | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py index 8599ce1d..95aec6fc 100644 --- a/radis/pgsearch/tests/test_filter_query_equivalence.py +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -311,3 +311,21 @@ def test_date_range_boundary_matches_legacy_in_non_utc_timezone(): assert new_ids == legacy_ids assert new_ids == {boundary["midnight"].pk, boundary["late"].pk} + + +def test_filter_query_plan_is_single_table(corpus): + """The candidate query must not join the membership table or deduplicate.""" + from django.db import connection + + filters = SearchFilters(group=corpus["group_a"].pk, language="en") + queryset = ReportSearchIndex.objects.filter(_build_filter_query(filters)).values_list( + "report_id", flat=True + ) + sql, params = queryset.query.sql_with_params() + + with connection.cursor() as cursor: + cursor.execute(f"EXPLAIN {sql}", params) + plan = "\n".join(row[0] for row in cursor.fetchall()) + + assert "reports_report_groups" not in plan, plan + assert "Unique" not in plan, plan From 622b5d930212112686827c791c2225f8dfe6dd48 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:47:30 +0000 Subject: [PATCH 25/50] Correct the plan-shape assertion in the spec and plan Task 9's review demonstrated empirically that asserting on the absence of a Unique node does not detect a reintroduced .distinct(). Postgres implements SELECT DISTINCT as HashAggregate rather than Sort+Unique once the matching row count is non-trivial -- measured at 3, 1,000 and 100,000 rows with the same GIN array-containment pattern this design uses. The assertion therefore passes with the regression present at exactly the 8M scale it exists to protect. Django emits the DISTINCT keyword unconditionally whenever .distinct() is called, regardless of the plan the planner later chooses, so checking the compiled SQL text is structural and scale-independent. The join half of the assertion was verified sound and is unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../plans/2026-08-30-fts-search-projection.md | 6 +++++- ...26-08-29-fts-query-shape-performance-design.md | 15 ++++++++++++--- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-30-fts-search-projection.md b/docs/superpowers/plans/2026-08-30-fts-search-projection.md index 46a6e520..e72bc8f7 100644 --- a/docs/superpowers/plans/2026-08-30-fts-search-projection.md +++ b/docs/superpowers/plans/2026-08-30-fts-search-projection.md @@ -1272,7 +1272,11 @@ def test_filter_query_plan_is_single_table(corpus): plan = "\n".join(row[0] for row in cursor.fetchall()) assert "reports_report_groups" not in plan, plan - assert "Unique" not in plan, plan + # Not `"Unique" not in plan`: Postgres implements SELECT DISTINCT as + # HashAggregate rather than Sort+Unique once the row count is non-trivial, + # so an operator-name check passes with the regression present at scale. + # Django emits the DISTINCT keyword unconditionally, so check the SQL. + assert "DISTINCT" not in sql, sql ``` - [ ] **Step 2: Run test to verify it passes** diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index ab9729b6..330b3498 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -437,9 +437,18 @@ with several matching modalities, appears exactly once. **Plan-shape assertion instead of timing.** Wall-clock assertions are flaky in CI; structure is not. Run `EXPLAIN` on the generated FTS query and assert the -plan contains no join against `reports_report_groups` and no `Unique` node. This -deterministically catches someone reintroducing a `report__` traversal and -silently restoring the 13-second path. +plan contains no join against `reports_report_groups`, and assert the compiled +SQL contains no `DISTINCT` keyword. This deterministically catches someone +reintroducing a `report__` traversal and silently restoring the slow path. + +Do **not** assert on the absence of a `Unique` node: PostgreSQL implements +`SELECT DISTINCT` as either `Sort`+`Unique` or `HashAggregate` depending on +cardinality estimates, and measurement at 3, 1,000 and 100,000 rows — with the +same GIN array-containment pattern this design uses — produced `HashAggregate` +in every non-trivial case. An operator-name assertion therefore passes with the +regression present at exactly the scale it exists to protect. The `DISTINCT` +keyword is emitted unconditionally by Django whenever `.distinct()` is called, +so a SQL-text check is scale-independent. **`patient_age` mirrors `Report.patient_age` exactly.** Verified that this is the trap it guards: a BEFORE trigger on a stored generated column reads `NULL` From 3ef44a469f2d5823c926bd71e8f0b7b01073aef3 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:49:08 +0000 Subject: [PATCH 26/50] Fix plan-shape test's dedup check to be scale-independent EXPLAIN can report a distinct step as either a Sort+Unique pair or a HashAggregate depending on the planner's cost estimate, and at realistic corpus sizes it picks HashAggregate -- so asserting "Unique" not in the plan only caught a reintroduced .distinct() on this test's tiny fixture, not at the 8-million-row scale the guard exists for. Check the compiled SQL text for the DISTINCT keyword instead: Django emits it unconditionally whenever .distinct() is called, independent of corpus size or planner choice. --- .../tests/test_filter_query_equivalence.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py index 95aec6fc..3c58a735 100644 --- a/radis/pgsearch/tests/test_filter_query_equivalence.py +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -314,7 +314,19 @@ def test_date_range_boundary_matches_legacy_in_non_utc_timezone(): def test_filter_query_plan_is_single_table(corpus): - """The candidate query must not join the membership table or deduplicate.""" + """The candidate query must not join the membership table or deduplicate. + + The join check inspects the ``EXPLAIN`` plan, which is sound regardless of + corpus size: Django only emits ``reports_report_groups`` into the SQL when a + lookup traverses ``report__groups``, and no planner choice can inject it + otherwise. The dedup check inspects the compiled SQL text for the + ``DISTINCT`` keyword instead of looking for a physical operator name (e.g. + ``Unique``) in the plan -- the planner is free to implement ``DISTINCT`` as + a ``Sort`` + ``Unique`` or as a ``HashAggregate`` depending on cost + estimates, and at realistic corpus sizes it picks ``HashAggregate``, so a + plan-based check would silently miss a reintroduced ``.distinct()`` at + scale even though it would catch it on this test's tiny fixture. + """ from django.db import connection filters = SearchFilters(group=corpus["group_a"].pk, language="en") @@ -323,9 +335,10 @@ def test_filter_query_plan_is_single_table(corpus): ) sql, params = queryset.query.sql_with_params() + assert "DISTINCT" not in sql, sql + with connection.cursor() as cursor: cursor.execute(f"EXPLAIN {sql}", params) plan = "\n".join(row[0] for row in cursor.fetchall()) assert "reports_report_groups" not in plan, plan - assert "Unique" not in plan, plan From ee139c7318577b62c975f8c1b2eba6d324cf9df4 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 21:54:41 +0000 Subject: [PATCH 27/50] Tune PostgreSQL parallelism for the search scan Raise max_parallel_workers_per_gather to 4 (from PostgreSQL's default of 2) so the report-index scan behind full-text search uses more workers. The other three related GUCs are exposed as compose-only overrides at their existing PostgreSQL defaults, so an operator raising per_gather on a bigger host can raise them coherently. --- AGENTS.md | 11 +++++++++++ docker-compose.base.yml | 10 ++++++++++ docs/user-docs/admin-guide.md | 13 +++++++++++++ 3 files changed, 34 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 9b1731b8..4558aee6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -146,6 +146,17 @@ Worker-crash recovery (`radis.core`): Labeling uses the shared core LLM client (`radis.core.utils.llm_client`); its timeout, rate-limit gate, and transient-retry knobs are the global `LLM_REQUEST_TIMEOUT_SECONDS`, `LLM_RATE_LIMIT_*`, and `LLM_TRANSIENT_RETRY_*` settings. +PostgreSQL parallelism (compose-only overrides, not in `example.env` — set in `.env` +only to override the compose file's default): + +- `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER`: Parallel workers per query node scanning + the report index (default `4`, raised from PostgreSQL's own default of `2`) +- `POSTGRES_MAX_PARALLEL_WORKERS`, `POSTGRES_MAX_WORKER_PROCESSES`: Cluster-wide worker + pool sizes (each defaults to `8`, PostgreSQL's own default). Raise together with + `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` on a larger host +- `POSTGRES_SHARED_BUFFERS`: PostgreSQL's shared memory buffer (default `128MB`, + PostgreSQL's own default) + ## Code Standards - **Style Guide**: Google Python Style Guide diff --git a/docker-compose.base.yml b/docker-compose.base.yml index f080d7e3..8b196030 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -40,6 +40,16 @@ services: postgres: image: pgvector/pgvector:pg17@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f hostname: postgres.local + command: + - postgres + - -c + - max_parallel_workers_per_gather=${POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER:-4} + - -c + - max_parallel_workers=${POSTGRES_MAX_PARALLEL_WORKERS:-8} + - -c + - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} + - -c + - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} volumes: - postgres_data:/var/lib/postgresql/data diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 0b0d680b..fd874f79 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -27,6 +27,19 @@ Follow these steps to safely update your RADIS installation: 9. **Deploy stack**: Run `uv run cli stack-deploy` to rebuild and start all services with the updated code 10. **Disable maintenance mode**: In Django Admin, navigate to **Common** → **Project Settings** and uncheck the "Maintenance mode" checkbox, then save +## Database tuning + +Search scans the report index table in parallel, so the number of parallel +workers is the one PostgreSQL setting worth revisiting. These are optional +overrides -- set them in `.env` only if the defaults do not suit your hardware. + +| Variable | Default | Guidance | +| --- | --- | --- | +| `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | Measured on 8M reports: 606 ms at 2, 421 ms at 4, 343 ms at 8. Four captures most of the gain while leaving cores for concurrent searches. | +| `POSTGRES_MAX_PARALLEL_WORKERS` | `8` | PostgreSQL's default. Raise together with the two others on a larger host. | +| `POSTGRES_MAX_WORKER_PROCESSES` | `8` | As above. | +| `POSTGRES_SHARED_BUFFERS` | `128MB` | PostgreSQL's default. Around 25% of host RAM is the usual recommendation; a value larger than the container's memory will prevent PostgreSQL from starting. | + ## User and Group Management Administrators can create users by navigating to the Django Admin section. Alternatively, users can self-register, after which an administrator must approve and activate their account. From 8e8e30637009658df1f9f17493292e13e0461b88 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:30:34 +0000 Subject: [PATCH 28/50] Give the init wait a timeout the backfill cannot outrun Every service that waits for `init` (prod) or `web` (dev) waited with a fixed `wait-for-it -t 300`. The search projection backfill takes 9 min 20 s at 8M reports, so the wait expired mid-migration, the `&&` chain aborted and the containers exited -- fatally in dev, which has no restart policy, and after three attempts in prod at the 15M extrapolation. The timeout is now `${WAIT_INIT_TIMEOUT:-3600}` at every such site. It is a tuning knob with a sensible default, so it stays a compose-only override and does not go into example.env. The admin guide's upgrade procedure and the design spec now say that this particular upgrade runs a long migration and that the stack is unavailable for roughly ten minutes at 8M reports. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- docker-compose.dev.yml | 8 ++++++-- docker-compose.prod.yml | 12 +++++++++--- .../2026-08-29-fts-query-shape-performance-design.md | 6 ++++++ docs/user-docs/admin-guide.md | 10 ++++++++++ 4 files changed, 31 insertions(+), 5 deletions(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 8c55bb38..18ce286f 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -26,6 +26,10 @@ services: profiles: - never + # In development `web` runs the migrations the workers below wait for, so + # WAIT_INIT_TIMEOUT must outlast the longest migration here too (the search + # projection backfill takes about ten minutes at 8M reports). There is no + # restart policy in dev, so a timeout leaves the worker containers exited. web: <<: *default-app image: radis_dev-web:latest @@ -52,7 +56,7 @@ services: command: > bash -c " wait-for-it -s postgres.local:5432 -t ${WAIT_POSTGRES_TIMEOUT:-180} && - wait-for-it -s web.local:8000 -t 300 && + wait-for-it -s web.local:8000 -t ${WAIT_INIT_TIMEOUT:-3600} && ./manage.py sweep_stale_tasks && ./manage.py bg_worker -l debug -q default --autoreload " @@ -63,7 +67,7 @@ services: command: > bash -c " wait-for-it -s postgres.local:5432 -t ${WAIT_POSTGRES_TIMEOUT:-180} && - wait-for-it -s web.local:8000 -t 300 && + wait-for-it -s web.local:8000 -t ${WAIT_INIT_TIMEOUT:-3600} && ./manage.py sweep_stale_tasks && ./manage.py bg_worker -l debug -q llm --autoreload " diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index ed709d49..4702686d 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -18,6 +18,12 @@ services: # We can't use those manage commands inside the web container in production because # the web service may have multiple replicas. So we make sure to only run them once # and wait for it to be finished by the web service containers. + # + # init opens its ok_server port only after `migrate` finishes, and every + # service below waits for that port. A data migration can run for many + # minutes on a large archive (the search projection backfill takes about ten + # at 8M reports), so WAIT_INIT_TIMEOUT must stay well above the longest + # migration -- a timeout here aborts the `&&` chain and the container exits. init: <<: *default-app hostname: init.local @@ -43,7 +49,7 @@ services: - ${WEB_HTTPS_PORT:-443}:443 command: > bash -c " - wait-for-it -s init.local:8000 -t 300 && + wait-for-it -s init.local:8000 -t ${WAIT_INIT_TIMEOUT:-3600} && echo 'Starting web server ...' && daphne -b 0.0.0.0 -p 80 \\ -e ssl:443:privateKey=/etc/web/ssl/key.pem:certKey=/etc/web/ssl/cert.pem \\ @@ -60,7 +66,7 @@ services: command: > bash -c " wait-for-it -s postgres.local:5432 -t ${WAIT_POSTGRES_TIMEOUT:-180} && - wait-for-it -s init.local:8000 -t 300 && + wait-for-it -s init.local:8000 -t ${WAIT_INIT_TIMEOUT:-3600} && ./manage.py sweep_stale_tasks && ./manage.py bg_worker -q default " @@ -72,7 +78,7 @@ services: command: > bash -c " wait-for-it -s postgres.local:5432 -t ${WAIT_POSTGRES_TIMEOUT:-180} && - wait-for-it -s init.local:8000 -t 300 && + wait-for-it -s init.local:8000 -t ${WAIT_INIT_TIMEOUT:-3600} && ./manage.py sweep_stale_tasks && ./manage.py bg_worker -q llm " diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 330b3498..c077af26 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -308,6 +308,11 @@ access-control data. Deploy time therefore grows with corpus size: **measured at 18 minutes at 15M. Adding the ten columns is genuinely free (1.3 ms, confirming the metadata-only claim); the nine minutes is the row rewrite. +Every service that waits on `init` must therefore tolerate a wait that long: +they wait with `wait-for-it -t ${WAIT_INIT_TIMEOUT:-3600}` in both compose +files, because the previous fixed `-t 300` timed out mid-backfill and left the +containers exited after a technically successful migration. + Roughly ten minutes of announced downtime for a one-time migration has been accepted for this deployment, which is what keeps the blocking approach below. A site that cannot take that window needs the online alternative instead. @@ -441,6 +446,7 @@ plan contains no join against `reports_report_groups`, and assert the compiled SQL contains no `DISTINCT` keyword. This deterministically catches someone reintroducing a `report__` traversal and silently restoring the slow path. + Do **not** assert on the absence of a `Unique` node: PostgreSQL implements `SELECT DISTINCT` as either `Sort`+`Unique` or `HashAggregate` depending on cardinality estimates, and measurement at 3, 1,000 and 100,000 rows — with the diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index fd874f79..60c7ae0f 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -27,6 +27,16 @@ Follow these steps to safely update your RADIS installation: 9. **Deploy stack**: Run `uv run cli stack-deploy` to rebuild and start all services with the updated code 10. **Disable maintenance mode**: In Django Admin, navigate to **Common** → **Project Settings** and uncheck the "Maintenance mode" checkbox, then save +Step 9 can take much longer than usual when a release ships a data migration. +The `init` service runs `manage.py migrate` and every other service waits for it +to finish, so the stack stays unavailable for the whole migration. The release +that added the search projection to the report search index is the current +example: it rewrites every row of that table, measured at about **ten minutes +for 8 million reports** and proportionally longer on a larger archive. Plan the +maintenance window around that, and do not shorten `WAIT_INIT_TIMEOUT` (default +one hour, see the compose files) below the expected migration time -- the +services that wait on `init` exit when that timeout expires. + ## Database tuning Search scans the report index table in parallel, so the number of parallel From 675aeebc79eeea43d5f35f25a0bfffc2866f32d7 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:30:53 +0000 Subject: [PATCH 29/50] Say which direction an empty group_ids fails in Spec 4.4 step 1 called `'{}'` "the fail-closed value". It is fail-closed for `group=`, which compiles to `group_ids @> ARRAY[id]` and matches nothing, but fail-*open* for `group=None`, which compiles to the exact match `group_ids = '{}'` and therefore matches the whole archive between migrations 0003 and 0005. That path is reachable: extractions/views.py passes `group=None` for a logged-in user with no active group. It is contained only accidentally today, because 0003 also leaves `language_code` NULL corpus-wide and the FTS half then returns nothing -- the vector half carries no language predicate, so on a deployment with embeddings the leak is real. The spec now states both directions, and the admin guide says the migration must be run with the web tier stopped. Also corrects the parallelism measurements in the admin guide: 606/421/343 ms came from the 5M rig (spec 4.5), not from 8M. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...026-08-29-fts-query-shape-performance-design.md | 14 ++++++++++++-- docs/user-docs/admin-guide.md | 10 +++++++++- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index c077af26..cca75553 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -278,8 +278,18 @@ untouched — `page_rows` still `select_related("report")` for `ts_headline` ove 1. **Add columns.** Arrays get `default=list` (`DEFAULT '{}'`, NOT NULL); scalars go in nullable. All are constant-default or nullable, so every `AddField` is - metadata-only and instant even at 5M rows. An empty `group_ids` is the - fail-closed value. + metadata-only and instant even at 5M rows. Every row therefore carries an + empty `group_ids` until step 3 fills it, and that empty value is fail-closed + in one direction only: a group-scoped search (`group=`) compiles to + `group_ids @> ARRAY[id]` and matches nothing, but `group=None` compiles to + the exact match `group_ids = '{}'` and matches **every row in the archive**. + `group=None` is reachable — `extractions/views.py` passes it for a logged-in + user with no active group — so between this step and step 3 that path is + fail-*open*. It is contained only accidentally today (step 1 also leaves + `language_code` NULL corpus-wide, so the FTS half returns nothing), and the + vector half carries no language predicate, so on a deployment with embeddings + the leak is real. **The migration must therefore be run with the web tier + stopped**, which is what the admin guide's upgrade procedure prescribes. 2. **Create triggers** — *before* the backfill, so reports edited during a multi-minute backfill still land correctly. A chunk that later reprocesses the same row simply rewrites the same current values. diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 60c7ae0f..28d71c02 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -37,6 +37,14 @@ maintenance window around that, and do not shorten `WAIT_INIT_TIMEOUT` (default one hour, see the compose files) below the expected migration time -- the services that wait on `init` exit when that timeout expires. +Keep the web tier stopped for the whole migration, which steps 5 and 9 above +already do -- do not start it early to "check on progress". While that +particular migration is between adding the projection columns and backfilling +them, every row carries an empty group list. A search that runs with no active +group (the extraction preview does this for a user without one) filters on +exactly that empty list, so during the window it would match the whole archive +instead of only the reports belonging to no group. + ## Database tuning Search scans the report index table in parallel, so the number of parallel @@ -45,7 +53,7 @@ overrides -- set them in `.env` only if the defaults do not suit your hardware. | Variable | Default | Guidance | | --- | --- | --- | -| `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | Measured on 8M reports: 606 ms at 2, 421 ms at 4, 343 ms at 8. Four captures most of the gain while leaving cores for concurrent searches. | +| `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` | `4` | Measured on 5M reports: 606 ms at 2, 421 ms at 4, 343 ms at 8. Four captures most of the gain while leaving cores for concurrent searches. | | `POSTGRES_MAX_PARALLEL_WORKERS` | `8` | PostgreSQL's default. Raise together with the two others on a larger host. | | `POSTGRES_MAX_WORKER_PROCESSES` | `8` | As above. | | `POSTGRES_SHARED_BUFFERS` | `128MB` | PostgreSQL's default. Around 25% of host RAM is the usual recommendation; a value larger than the container's memory will prevent PostgreSQL from starting. | From b4f08a3998f1b57ff89276f6e53468460c1cee18 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:31:46 +0000 Subject: [PATCH 30/50] Exercise the backfill chunk loop and multi-row group deletes The backfill test used one report against CHUNK_SIZE = 50,000, so the loop ran exactly once and an off-by-one in its `low`/`high` arithmetic -- the kind that silently skips whole id ranges -- would not have shown. It now monkeypatches CHUNK_SIZE to 1 with three reports (given explicit ids 1..3, because the loop always starts at 0) and asserts one chunk statement per report as well as the resulting values. The group triggers only had multi-row coverage for INSERT. reports/api/ viewsets.py:217 clears a whole batch's memberships with a single DELETE, and removal is the leak direction, so the OLD transition table now gets the same many-rows-at-once test. Each report keeps a second group, so the assertion pins a recomputed aggregate rather than a blanket reset. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../pgsearch/tests/test_search_projection.py | 118 +++++++++++++----- 1 file changed, 84 insertions(+), 34 deletions(-) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index d126ae78..ecb3f56a 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -5,9 +5,12 @@ its correctness has its own tests here. """ +import importlib + import pytest from adit_radis_shared.accounts.factories import GroupFactory from django.db import connection +from django.test.utils import CaptureQueriesContext from radis.pgsearch.models import ReportSearchIndex from radis.pgsearch.utils.indexing import bulk_upsert_report_search_indexes @@ -16,6 +19,15 @@ pytestmark = pytest.mark.django_db +BACKFILL_MIGRATION = "radis.pgsearch.migrations.0005_search_projection_backfill" + + +class _MinimalSchemaEditor: + """What the migration's backfill() uses of a real schema editor.""" + + def __init__(self, connection): + self.connection = connection + def test_new_index_row_defaults_to_empty_arrays(): # modalities=[] is explicit: ReportFactory attaches a random non-empty set @@ -105,6 +117,43 @@ def test_bulk_membership_insert_updates_every_affected_row(): assert index.group_ids == [group.pk], f"report {report.pk} was not updated" +def test_bulk_membership_delete_updates_every_affected_row(): + """The removal direction, in the shape reports/api/viewsets.py:217 uses. + + That endpoint clears a whole batch's memberships with a single + ``filter(report_id__in=...).delete()``, so the DELETE trigger's OLD + transition table carries many rows at once -- and a statement-level trigger + that only processes one of them leaves every other report readable by a + group it was just removed from. Each report keeps a second group so the + assertion pins a recomputed aggregate rather than a blanket reset. + """ + language = LanguageFactory.create(code="en") + reports = [ReportFactory.create(language=language) for _ in range(5)] + removed_group = GroupFactory.create(name="removed") + kept_group = GroupFactory.create(name="kept") + + through = Report.groups.through + through.objects.bulk_create( + [ + through(report_id=report.pk, group_id=group.pk) + for report in reports + for group in (removed_group, kept_group) + ] + ) + for report in reports: + assert ReportSearchIndex.objects.get(report=report).group_ids == sorted( + [removed_group.pk, kept_group.pk] + ), "setup: both groups should be mirrored" + + through.objects.filter( + report_id__in=[report.pk for report in reports], group_id=removed_group.pk + ).delete() + + for report in reports: + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [kept_group.pk], f"report {report.pk} was not updated" + + def test_deleting_a_report_does_not_error(): """Deleting a Report cascades to both the membership rows and the index row, firing the membership trigger against a row that may already be gone. The @@ -198,25 +247,33 @@ def test_backfill_fills_rows_that_predate_the_projection(): assert index.patient_id == report.patient_id -def test_migration_backfill_executes_the_chunked_update(): - """Verify the migration's backfill() actually executes and fills the projection. +def test_migration_backfill_executes_the_chunked_update(monkeypatch): + """Verify the migration's own SQL fills the projection, across several chunks. + + This exercises the literal SQL and the loop in + 0005_search_projection_backfill.py, not sync_projection(). CHUNK_SIZE is + 50,000 in production, so a handful of reports would run the loop body + exactly once and an off-by-one in the ``low``/``high`` arithmetic -- the + kind that silently skips whole id ranges -- would not show. Patching it to + 1 puts each report in its own chunk. - This test exercises the literal SQL in 0005_search_projection_backfill.py, - not sync_projection(). It ensures group_ids and other access control fields - are restored by the migration's own UPDATE statement. + The reports are given explicit ids 1..3 because the loop always starts at + id 0: with the sequence wherever the rest of the suite left it, a chunk + size of 1 would otherwise run one empty iteration per id below the first + report. """ + assert not Report.objects.exists(), "this test needs the reports table to itself" + language = LanguageFactory.create(code="en") - report = ReportFactory.create(language=language, modalities=["CT"]) group = GroupFactory.create() - report.groups.add(group) - - # Verify projection is populated first - index = ReportSearchIndex.objects.get(report=report) - assert index.group_ids == [group.pk], "Setup: projection should be populated" - assert index.language_code == "en" + reports = [ + ReportFactory.create(id=report_id, language=language, modalities=["CT"]) + for report_id in (1, 2, 3) + ] + for report in reports: + report.groups.add(group) - # Clear projection fields to simulate a row predating the feature - ReportSearchIndex.objects.filter(report=report).update( + ReportSearchIndex.objects.update( group_ids=[], modality_codes=[], language_code=None, @@ -229,29 +286,22 @@ def test_migration_backfill_executes_the_chunked_update(): report_updated_at=None, ) - # Import and call the migration's backfill function - import importlib - - from django.db import connection - - migration_module = importlib.import_module( - "radis.pgsearch.migrations.0005_search_projection_backfill" - ) + migration_module = importlib.import_module(BACKFILL_MIGRATION) + monkeypatch.setattr(migration_module, "CHUNK_SIZE", 1) - # Create a minimal schema_editor-like object that the backfill function needs - class MinimalSchemaEditor: - def __init__(self, connection): - self.connection = connection + with CaptureQueriesContext(connection) as captured: + migration_module.backfill(None, _MinimalSchemaEditor(connection)) - schema_editor = MinimalSchemaEditor(connection) - migration_module.backfill(None, schema_editor) + statements = [query["sql"] for query in captured.captured_queries] + chunk_updates = [sql for sql in statements if sql.lstrip().startswith("UPDATE")] + assert len(chunk_updates) == 3, statements - # Verify the projection is now restored by the migration's SQL - index.refresh_from_db() - assert index.group_ids == [group.pk], "Migration backfill should restore group_ids" - assert index.modality_codes == ["CT"], "Migration backfill should restore modality_codes" - assert index.language_code == "en", "Migration backfill should restore language_code" - assert index.patient_id == report.patient_id + for report in reports: + index = ReportSearchIndex.objects.get(report=report) + assert index.group_ids == [group.pk], f"report {report.pk} was not backfilled" + assert index.modality_codes == ["CT"] + assert index.language_code == "en" + assert index.patient_id == report.patient_id def test_projection_indexes_exist(): From a20543a2cc3f093fc2f378fd49601464da417a1a Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:32:04 +0000 Subject: [PATCH 31/50] Lock projection rows before rewriting them Both projection writers ran a single `UPDATE ... FROM reports_report LEFT JOIN LATERAL (array_agg ...)` in autocommit. Under READ COMMITTED that can lose an update: when a trigger-driven write to a targeted row commits after the statement's snapshot but before the statement reaches that row, EvalPlanQual re-checks the row and applies the value computed from the stale snapshot. Concretely: an administrator removes group G from report R (the trigger sets `group_ids = '{}'`) while a bulk_index_reports job's sync_projection() is mid-statement over a set including R. R's group_ids reverts to [G] and stays there -- a report readable by a group it was deliberately removed from. Each statement in READ COMMITTED takes its own snapshot, so both writers now take the row locks as their own statement first, inside one transaction: a racing trigger write blocks until the transaction ends, and one that committed earlier is visible to the UPDATE's snapshot. The migration does the same per chunk (belt-and-braces -- the documented deploy has no concurrent writers) and keeps `atomic = False` with one transaction per chunk. Two tests: one pins the structure (lock statement first, both statements in one atomic block), one proves behaviourally that a concurrent group removal blocks while sync_projection's transaction is open and succeeds once it commits. Neither reproduces the race itself, which would need the trigger to commit at a chosen instant inside the UPDATE's execution. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../0005_search_projection_backfill.py | 34 ++++- .../pgsearch/tests/test_search_projection.py | 124 +++++++++++++++++- radis/pgsearch/utils/projection.py | 41 +++++- 3 files changed, 187 insertions(+), 12 deletions(-) diff --git a/radis/pgsearch/migrations/0005_search_projection_backfill.py b/radis/pgsearch/migrations/0005_search_projection_backfill.py index 676d43e3..6c9b7cf3 100644 --- a/radis/pgsearch/migrations/0005_search_projection_backfill.py +++ b/radis/pgsearch/migrations/0005_search_projection_backfill.py @@ -10,13 +10,25 @@ The statement below duplicates utils/projection.PROJECTION_UPDATE_SQL, the same way bulk_upsert_report_search_indexes duplicates the tsvector logic. Migrations -must not import app code, which drifts under them. +must not import app code, which drifts under them. The two trigger functions in +0004 and check_search_projection.DRIFT_SQL repeat the same array_agg shape -- +keep all four in sync. """ -from django.db import migrations +from django.db import migrations, transaction CHUNK_SIZE = 50_000 +# Locked as its own statement before each chunk's UPDATE, so that UPDATE's +# snapshot is taken with the locks already held. See backfill() below and +# utils/projection.sync_projection for why that matters. +LOCK_SQL = """ +SELECT report_id FROM pgsearch_reportsearchindex +WHERE report_id > %s AND report_id <= %s +ORDER BY report_id +FOR UPDATE +""" + BACKFILL_SQL = """ UPDATE pgsearch_reportsearchindex rsi SET group_ids = COALESCE(g.ids, '{}'), @@ -46,6 +58,18 @@ def backfill(apps, schema_editor): + """Fill the projection in report_id ranges, one transaction per chunk. + + Each chunk locks its rows before updating them. Under READ COMMITTED a + single ``UPDATE ... FROM`` fed by a LATERAL aggregate can re-apply a stale + aggregate to a row a concurrent trigger committed mid-statement + (EvalPlanQual), which on ``group_ids`` would silently restore a group that + was just removed. The documented deploy runs this with the web tier stopped + and therefore has no concurrent writers at all, so this is belt-and-braces + -- it costs one extra pass over each chunk, which is cheap next to the row + rewrite the UPDATE performs anyway, and it removes the need to reason about + whether some other writer really is quiesced. + """ connection = schema_editor.connection with connection.cursor() as cursor: cursor.execute("SELECT COALESCE(MAX(report_id), 0) FROM pgsearch_reportsearchindex") @@ -54,8 +78,10 @@ def backfill(apps, schema_editor): low = 0 while low < max_id: high = low + CHUNK_SIZE - with connection.cursor() as cursor: - cursor.execute(BACKFILL_SQL, [low, high]) + with transaction.atomic(using=connection.alias): + with connection.cursor() as cursor: + cursor.execute(LOCK_SQL, [low, high]) + cursor.execute(BACKFILL_SQL, [low, high]) low = high diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index ecb3f56a..c5f0a7ff 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -6,14 +6,16 @@ """ import importlib +import threading import pytest from adit_radis_shared.accounts.factories import GroupFactory -from django.db import connection +from django.db import OperationalError, connection, transaction from django.test.utils import CaptureQueriesContext from radis.pgsearch.models import ReportSearchIndex from radis.pgsearch.utils.indexing import bulk_upsert_report_search_indexes +from radis.pgsearch.utils.projection import sync_projection from radis.reports.factories import LanguageFactory, ModalityFactory, ReportFactory from radis.reports.models import Report @@ -236,8 +238,6 @@ def test_backfill_fills_rows_that_predate_the_projection(): group_ids=[], modality_codes=[], language_code=None, patient_id=None ) - from radis.pgsearch.utils.projection import sync_projection - sync_projection([report.pk]) index = ReportSearchIndex.objects.get(report=report) @@ -304,6 +304,124 @@ def test_migration_backfill_executes_the_chunked_update(monkeypatch): assert index.patient_id == report.patient_id +def test_migration_backfill_locks_each_chunk_inside_its_own_transaction(monkeypatch): + """Each chunk locks its rows in a separate statement before updating them. + + Same lost-update hazard as sync_projection() below: an ``UPDATE ... FROM`` + fed by a LATERAL aggregate can re-apply a stale aggregate to a row a + concurrent trigger committed mid-statement. Belt-and-braces here (the + documented deploy runs with the web tier stopped), but the structure is + cheap to pin, and the migration must keep one transaction *per chunk* + rather than one for the whole run. + """ + assert not Report.objects.exists(), "this test needs the reports table to itself" + + language = LanguageFactory.create(code="en") + for report_id in (1, 2): + ReportFactory.create(id=report_id, language=language, modalities=[]) + + migration_module = importlib.import_module(BACKFILL_MIGRATION) + monkeypatch.setattr(migration_module, "CHUNK_SIZE", 1) + + with CaptureQueriesContext(connection) as captured: + migration_module.backfill(None, _MinimalSchemaEditor(connection)) + + statements = [query["sql"].strip() for query in captured.captured_queries] + # The test itself runs inside a transaction, so each chunk's atomic block + # shows up as a savepoint pair rather than as BEGIN/COMMIT. + chunks = [ + statements[index : index + 4] + for index, sql in enumerate(statements) + if sql.startswith("SAVEPOINT") + ] + assert len(chunks) == 2, statements + for chunk in chunks: + assert "FOR UPDATE" in chunk[1], chunk + assert "pgsearch_reportsearchindex" in chunk[1], chunk + assert chunk[2].startswith("UPDATE pgsearch_reportsearchindex"), chunk + assert chunk[3].startswith("RELEASE SAVEPOINT"), chunk + + +def test_sync_projection_locks_the_rows_it_updates_in_one_transaction(): + """The structural half of the lost-update fix. + + Under READ COMMITTED a lone ``UPDATE ... FROM`` whose SET values come from + a LATERAL aggregate can write a stale aggregate onto a row a concurrent + trigger updated after the statement's snapshot was taken (EvalPlanQual + re-check). On group_ids that reverts a group removal, so the fix takes the + row locks in their own statement first and runs both statements in one + transaction. + + What this pins is exactly that structure: two statements, the lock first, + inside a single atomic block. It does not reproduce the race -- see + test_sync_projection_blocks_a_concurrent_trigger_write for the behavioural + half. The test runs inside pytest-django's transaction, so the atomic block + shows up as SAVEPOINT/RELEASE SAVEPOINT; in production (autocommit) the + same block is a real BEGIN/COMMIT. + """ + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + + with CaptureQueriesContext(connection) as captured: + sync_projection([report.pk]) + + statements = [query["sql"].strip() for query in captured.captured_queries] + assert statements[0].startswith("SAVEPOINT"), statements + assert statements[-1].startswith("RELEASE SAVEPOINT"), statements + + body = statements[1:-1] + assert len(body) == 2, body + assert "FOR UPDATE" in body[0] and "pgsearch_reportsearchindex" in body[0], body + assert body[1].startswith("UPDATE pgsearch_reportsearchindex"), body + + +@pytest.mark.django_db(transaction=True) +def test_sync_projection_blocks_a_concurrent_trigger_write(): + """The behavioural half: the locks really do hold off the racing writer. + + A second connection removes the report's group -- the exact write whose + trigger the stale-aggregate race would overwrite -- while sync_projection's + transaction is still open. With a lock_timeout set it must fail rather than + slip through, and it must succeed once the lock is released. + + This proves the locks are taken and held for the whole transaction. It + still does not prove the absence of the race itself: that would need the + trigger to commit at a specific instant inside the UPDATE's execution, + which PostgreSQL gives no hook to arrange. + """ + report = ReportFactory.create(language=LanguageFactory.create(code="en"), modalities=[]) + group = GroupFactory.create() + report.groups.add(group) + + outcome: dict[str, str] = {} + + def remove_the_group(): + try: + with transaction.atomic(): + with connection.cursor() as cursor: + cursor.execute("SET LOCAL lock_timeout = '1s'") + report.groups.remove(group) + outcome["result"] = "committed" + except OperationalError: + outcome["result"] = "blocked" + except Exception as exc: # pragma: no cover - surfaces an unexpected failure + outcome["result"] = f"error: {exc!r}" + finally: + connection.close() + + with transaction.atomic(): + sync_projection([report.pk]) + worker = threading.Thread(target=remove_the_group) + worker.start() + worker.join(timeout=30) + + assert not worker.is_alive(), "the concurrent writer never finished" + assert outcome["result"] == "blocked", outcome + + # The lock was the only obstacle: the same removal goes through now. + report.groups.remove(group) + assert ReportSearchIndex.objects.get(report=report).group_ids == [] + + def test_projection_indexes_exist(): with connection.cursor() as cursor: cursor.execute( diff --git a/radis/pgsearch/utils/projection.py b/radis/pgsearch/utils/projection.py index 40cef5fa..7d93c43d 100644 --- a/radis/pgsearch/utils/projection.py +++ b/radis/pgsearch/utils/projection.py @@ -4,12 +4,23 @@ This module fills it when an index row is first created, which happens before the report's groups and modalities are attached. The backfill migration duplicates this statement in SQL, the same way bulk_upsert_report_search_indexes -already duplicates the tsvector logic -- keep them in sync. +already duplicates the tsvector logic -- keep them in sync. The two trigger +functions in migration 0004 and check_search_projection.DRIFT_SQL repeat the +same array_agg shape and belong to that set as well. """ from collections.abc import Iterable -from django.db import connection +from django.db import connection, transaction + +# Taken as its own statement before the UPDATE below, so the UPDATE's snapshot +# is the one taken with these locks already held. See sync_projection(). +PROJECTION_LOCK_SQL = """ +SELECT report_id FROM pgsearch_reportsearchindex +WHERE report_id = ANY(%s) +ORDER BY report_id +FOR UPDATE +""" PROJECTION_UPDATE_SQL = """ UPDATE pgsearch_reportsearchindex rsi @@ -40,9 +51,29 @@ def sync_projection(report_ids: Iterable[int]) -> None: - """Fill the projection columns for the given reports from their sources.""" + """Fill the projection columns for the given reports from their sources. + + The row locks are taken first, in their own statement, to close a + lost-update race. Under READ COMMITTED a single ``UPDATE ... FROM`` whose + SET values come from a LATERAL aggregate can re-apply a *stale* aggregate: + if a trigger-driven update to a targeted row commits after this statement's + snapshot but before the statement reaches that row, PostgreSQL's + EvalPlanQual re-check re-evaluates the row against the new version and + writes the value computed from the old snapshot. On ``group_ids`` that + silently reverts a group removal -- a report readable by a group it was + deliberately removed from, which is the fail-open direction. + + Each statement in READ COMMITTED takes its own snapshot, so locking first + splits the race in two: a trigger write that would have committed later now + blocks on these locks until this transaction ends, and one that committed + earlier is already visible to the UPDATE's snapshot. ``ORDER BY report_id`` + makes the lock order deterministic so concurrent callers queue rather than + deadlock. + """ ids = [int(report_id) for report_id in report_ids] if not ids: return - with connection.cursor() as cursor: - cursor.execute(PROJECTION_UPDATE_SQL, [ids]) + with transaction.atomic(): + with connection.cursor() as cursor: + cursor.execute(PROJECTION_LOCK_SQL, [ids]) + cursor.execute(PROJECTION_UPDATE_SQL, [ids]) From ad89bdd916cf2d1fc58f2631d0fcb034083c5199 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:32:21 +0000 Subject: [PATCH 32/50] Analyze the search index after creating its indexes Spec 4.4 step 4 requires it: "The ANALYZE is required, not hygiene: ten new columns and two new indexes carry no statistics, and this design depends on the planner choosing a parallel sequential scan with a top-N heapsort." The step was lost when the plan was written from the spec, so 0006 ended after the two AddIndexes and the fastupdate RunSQL. ANALYZE is legal inside a transaction block, so 0006 stays atomic. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../migrations/0006_search_projection_indexes.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0006_search_projection_indexes.py index fa48df11..f28118fd 100644 --- a/radis/pgsearch/migrations/0006_search_projection_indexes.py +++ b/radis/pgsearch/migrations/0006_search_projection_indexes.py @@ -28,4 +28,14 @@ class Migration(migrations.Migration): "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", ), + migrations.RunSQL( + # Required, not hygiene: the ten projection columns and the two + # indexes above carry no statistics, and the single-table query + # shape this whole change exists for depends on the planner + # choosing a parallel sequential scan with a top-N heapsort. + # ANALYZE is legal inside a transaction block, so this migration + # stays atomic. + "ANALYZE pgsearch_reportsearchindex;", + reverse_sql=migrations.RunSQL.noop, + ), ] From cf5a0e92135de69954771428e2255129a3e2bdd3 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:32:41 +0000 Subject: [PATCH 33/50] Assert the plan shape of the query the provider actually runs The regression guard was vacuous. It built its own queryset from _build_filter_query and asserted "DISTINCT" was absent from the SQL -- but _build_filter_query returns a Q and never called .distinct(); the calls this change removed lived in _fuse_hybrid. The assertion could not fail whatever happened to the production query. The EXPLAIN half had the matching gap: it covered the filter only, so a report__language__code traversal reintroduced into match_q or rank_expr was caught by nothing but a source grep. The FTS candidate query now lives in one named function, _fts_candidate_queryset, which _fuse_hybrid calls, so the test can assert on the real thing: filter, tsquery match, rank annotation, ordering and bound. It runs with and without a language filter, since the multi-configuration branches are where a language traversal is most tempting, and asserts the plan touches no reports_* table. Verified to have teeth at both ends: reintroducing .distinct() on the real path fails both parametrisations on the SQL text, and reintroducing report__language__code into match_q fails both on the EXPLAIN plan (nested loops over reports_report and reports_language). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 8 ++ radis/pgsearch/providers.py | 73 +++++++++++------- .../tests/test_filter_query_equivalence.py | 75 ++++++++++++++----- 3 files changed, 110 insertions(+), 46 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index cca75553..3dcceb90 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -456,6 +456,14 @@ plan contains no join against `reports_report_groups`, and assert the compiled SQL contains no `DISTINCT` keyword. This deterministically catches someone reintroducing a `report__` traversal and silently restoring the slow path. +Assert on the *candidate queryset the provider runs* — filter, tsquery match, +rank annotation, ordering and bound — and not on one rebuilt from +`_build_filter_query` alone. The `.distinct()` calls this change removes never +lived in the filter builder, so a test that rebuilds only that half cannot fail +however the production query changes. Keeping the candidate query in one named +function (`_fts_candidate_queryset`) is what makes the real thing reachable from +a test. + Do **not** assert on the absence of a `Unique` node: PostgreSQL implements `SELECT DISTINCT` as either `Sort`+`Unique` or `HashAggregate` depending on diff --git a/radis/pgsearch/providers.py b/radis/pgsearch/providers.py index 0abdf070..c7da25bc 100644 --- a/radis/pgsearch/providers.py +++ b/radis/pgsearch/providers.py @@ -344,26 +344,17 @@ def _embed_query_cached(query_text: str, caller: str) -> list[float] | None: return vec -class _FusedHybrid(NamedTuple): - ordered_ids: list[int] - rrf_score_by_id: dict[int, float] - vec_distance: dict[int, float] - total_relation: Literal["exact", "at_least", "approximately"] - configs: list[tuple[str, list[str]]] - query_str: str - - -def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: - """Run both retrievers and fuse them with RRF. - - Shared by search(), retrieve() and count() so all three describe the same - union. The vector half embeds only the positive branches (§7.8 strips - ``NOT``) and enforces the top-level negations on its candidates; the FTS - half consumes the full boolean tsquery. Returns the fused, ordered report - ids plus the per-id scores/distances and the query metadata callers reuse.""" - query_str = _build_query_string(search.query) - configs = _language_configs(search.filters) - filter_query = _build_filter_query(search.filters) +def _fts_candidate_queryset(filter_query: Q, configs: list[tuple[str, list[str]]], query_str: str): + """The FTS candidate query: the single-table scan this whole design exists for. + + Kept as its own function rather than inlined in ``_fuse_hybrid`` so that + ``test_fts_candidate_query_is_single_table`` can assert on the SQL and the + ``EXPLAIN`` plan of exactly the queryset production runs -- filters, tsquery + match, rank annotation, ordering and bound included, not a re-creation of + it. Everything referenced here must be a column on ``ReportSearchIndex`` + itself: one lookup or annotation traversing the ``report`` relation restores + the joins and the multi-second query shape (see migration 0003). + """ tsqueries = { config: SearchQuery(query_str, search_type="raw", config=config) for config, _ in configs } @@ -389,6 +380,38 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: output_field=FloatField(), ) + # No .distinct(): with no joins there is nothing to deduplicate, and it + # would block the top-N heapsort that keeps this query sub-second. + return ( + ReportSearchIndex.objects.filter(filter_query) + .filter(match_q) + .annotate(rank=rank_expr) + .order_by("-rank", "report_id") + .values("report_id", "rank")[: settings.HYBRID_FTS_MAX_RESULTS] + ) + + +class _FusedHybrid(NamedTuple): + ordered_ids: list[int] + rrf_score_by_id: dict[int, float] + vec_distance: dict[int, float] + total_relation: Literal["exact", "at_least", "approximately"] + configs: list[tuple[str, list[str]]] + query_str: str + + +def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: + """Run both retrievers and fuse them with RRF. + + Shared by search(), retrieve() and count() so all three describe the same + union. The vector half embeds only the positive branches (§7.8 strips + ``NOT``) and enforces the top-level negations on its candidates; the FTS + half consumes the full boolean tsquery. Returns the fused, ordered report + ids plus the per-id scores/distances and the query metadata callers reuse.""" + query_str = _build_query_string(search.query) + configs = _language_configs(search.filters) + filter_query = _build_filter_query(search.filters) + # Vector side: skipped entirely when no embedding model is configured (FTS-only # deployment), and when stripping NOT branches leaves nothing to embed (see # docs/superpowers/specs/hybrid-search.md §7.8). @@ -416,15 +439,7 @@ def _fuse_hybrid(search: Search, caller: str) -> _FusedHybrid: # ``configs`` means an empty corpus (Report.language is a required FK), and # an empty match_q would match every row rather than none, so skip it. fts_rows = ( - [] - if not configs - else list( - ReportSearchIndex.objects.filter(filter_query) - .filter(match_q) - .annotate(rank=rank_expr) - .order_by("-rank", "report_id") - .values("report_id", "rank")[: settings.HYBRID_FTS_MAX_RESULTS] - ) + [] if not configs else list(_fts_candidate_queryset(filter_query, configs, query_str)) ) fts_rank = {row["report_id"]: i + 1 for i, row in enumerate(fts_rows)} diff --git a/radis/pgsearch/tests/test_filter_query_equivalence.py b/radis/pgsearch/tests/test_filter_query_equivalence.py index 3c58a735..7c1b7ab8 100644 --- a/radis/pgsearch/tests/test_filter_query_equivalence.py +++ b/radis/pgsearch/tests/test_filter_query_equivalence.py @@ -15,9 +15,15 @@ from radis.labels.factories import LabelFactory, LabelResultFactory from radis.labels.models import LabelResult from radis.pgsearch.models import ReportSearchIndex -from radis.pgsearch.providers import _build_filter_query +from radis.pgsearch.providers import ( + _build_filter_query, + _build_query_string, + _fts_candidate_queryset, + _language_configs, +) from radis.reports.factories import LanguageFactory, ReportFactory from radis.search.site import SearchFilters +from radis.search.utils.query_parser import QueryParser pytestmark = pytest.mark.django_db @@ -313,26 +319,58 @@ def test_date_range_boundary_matches_legacy_in_non_utc_timezone(): assert new_ids == {boundary["midnight"].pk, boundary["late"].pk} -def test_filter_query_plan_is_single_table(corpus): - """The candidate query must not join the membership table or deduplicate. +def _candidate_queryset(filters: SearchFilters, query: str = "pneumonia"): + """The FTS candidate queryset ``_fuse_hybrid`` actually runs. + + Built by calling the same production helpers in the same order the provider + does, so a regression introduced anywhere along that path -- the filter, the + tsquery match, the rank annotation, the ordering or the bound -- shows up + here. A test that assembled its own queryset from ``_build_filter_query`` + alone could not fail, because the ``.distinct()`` calls this change removed + never lived there. + """ + node, _fixes = QueryParser().parse(query) + assert node is not None + return _fts_candidate_queryset( + _build_filter_query(filters), + _language_configs(filters), + _build_query_string(node), + ) + + +@pytest.mark.parametrize( + "language", + ["en", ""], + ids=["single-config", "every-config"], +) +def test_fts_candidate_query_is_single_table(corpus, language): + """The candidate query must touch no table but the search index, and must + not deduplicate. + + This is the guard against silently restoring the 8-second query shape, so + it asserts on the real candidate queryset rather than on a re-creation of + its filter half. The join check inspects the ``EXPLAIN`` plan, which is sound regardless of - corpus size: Django only emits ``reports_report_groups`` into the SQL when a - lookup traverses ``report__groups``, and no planner choice can inject it - otherwise. The dedup check inspects the compiled SQL text for the - ``DISTINCT`` keyword instead of looking for a physical operator name (e.g. - ``Unique``) in the plan -- the planner is free to implement ``DISTINCT`` as - a ``Sort`` + ``Unique`` or as a ``HashAggregate`` depending on cost - estimates, and at realistic corpus sizes it picks ``HashAggregate``, so a - plan-based check would silently miss a reintroduced ``.distinct()`` at - scale even though it would catch it on this test's tiny fixture. + corpus size: Django only emits ``reports_report``, ``reports_language`` or + ``reports_report_groups`` into the SQL when a lookup or an annotation + traverses ``report__``, and no planner choice can inject them otherwise. + Both parametrisations matter: without a language filter the match and rank + expressions carry one branch per text-search configuration in the corpus, + which is where a ``report__language__code`` traversal is most tempting. + + The dedup check inspects the compiled SQL text for the ``DISTINCT`` keyword + instead of looking for a physical operator name (e.g. ``Unique``) in the + plan -- the planner is free to implement ``DISTINCT`` as a ``Sort`` + + ``Unique`` or as a ``HashAggregate`` depending on cost estimates, and at + realistic corpus sizes it picks ``HashAggregate``, so a plan-based check + would silently miss a reintroduced ``.distinct()`` at scale even though it + would catch it on this test's tiny fixture. """ from django.db import connection - filters = SearchFilters(group=corpus["group_a"].pk, language="en") - queryset = ReportSearchIndex.objects.filter(_build_filter_query(filters)).values_list( - "report_id", flat=True - ) + filters = SearchFilters(group=corpus["group_a"].pk, language=language) + queryset = _candidate_queryset(filters) sql, params = queryset.query.sql_with_params() assert "DISTINCT" not in sql, sql @@ -341,4 +379,7 @@ def test_filter_query_plan_is_single_table(corpus): cursor.execute(f"EXPLAIN {sql}", params) plan = "\n".join(row[0] for row in cursor.fetchall()) - assert "reports_report_groups" not in plan, plan + # "reports_report" also covers "reports_report_groups" and + # "reports_report_modalities". + for table in ("reports_report", "reports_language", "reports_modality"): + assert table not in plan, plan From 4fe930177eec2b6ef26afe2052a33c2cbe47555d Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:33:10 +0000 Subject: [PATCH 34/50] Cross-reference the four projection copies and fix two comments The array_agg shape is repeated in utils/projection.py, the 0005 backfill, the two trigger functions in 0004 and check_search_projection.DRIFT_SQL. Only the first two said "keep them in sync", and the drift query is the one meant to *detect* divergence -- if it ever drifts in the same direction as a writer it silently agrees with the bug. All four now name the other three. 0004's docstring claimed the trigger holds "for every writer". Three escape it, and they are now named: an UPDATE of a membership row (no trigger watches UPDATE on those tables), TRUNCATE (transition tables cannot be declared on a TRUNCATE trigger and no row events fire), and a re-add that Django emits as INSERT ... ON CONFLICT DO NOTHING, whose NEW transition table is empty -- so a projection row that has already drifted cannot be repaired by re-adding the membership. models.py said "Nullable so every AddField stays metadata-only" directly above group_ids and modality_codes, which are NOT NULL with a constant default. The comment now distinguishes the two forms. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../commands/check_search_projection.py | 4 +++ .../0004_search_projection_triggers.py | 31 +++++++++++++++++-- radis/pgsearch/models.py | 7 +++-- 3 files changed, 38 insertions(+), 4 deletions(-) diff --git a/radis/pgsearch/management/commands/check_search_projection.py b/radis/pgsearch/management/commands/check_search_projection.py index c80daa38..32440352 100644 --- a/radis/pgsearch/management/commands/check_search_projection.py +++ b/radis/pgsearch/management/commands/check_search_projection.py @@ -3,6 +3,10 @@ from django.core.management.base import BaseCommand, CommandError from django.db import connection +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill +# and the two trigger functions in migration 0004 -- four copies, keep them in +# sync. This one is the detector: if it ever drifts in the same direction as a +# writer, it silently agrees with the bug and reports a healthy projection. DRIFT_SQL = """ SELECT count(*) FILTER (WHERE rsi.group_ids IS DISTINCT FROM COALESCE(g.ids, '{}')) diff --git a/radis/pgsearch/migrations/0004_search_projection_triggers.py b/radis/pgsearch/migrations/0004_search_projection_triggers.py index fa4ff811..292addcf 100644 --- a/radis/pgsearch/migrations/0004_search_projection_triggers.py +++ b/radis/pgsearch/migrations/0004_search_projection_triggers.py @@ -2,8 +2,27 @@ Triggers rather than Django signals because group_ids is access-control data: a signal fires only for ORM writes, missing management commands, the admin, -and raw SQL. A trigger holds inside the same transaction as the write, for -every writer. +and raw SQL. A trigger holds inside the same transaction as the write, and +covers every INSERT and DELETE on the two membership tables plus every UPDATE +of reports_report. + +Three writers still escape it, so a new code path that uses one of them has to +keep the projection correct itself: + +- An UPDATE of a membership row (``UPDATE reports_report_groups SET group_id = + ...``). No trigger watches UPDATE on those tables -- Django's ORM never emits + one, but hand-written SQL can, and it would move a report between groups + invisibly. +- TRUNCATE of a membership table. Transition tables cannot be declared on a + TRUNCATE trigger at all, and TRUNCATE fires no row-level events, so the + projection would keep memberships the table no longer holds. +- A re-add of an existing membership, which Django emits as ``INSERT ... ON + CONFLICT DO NOTHING`` (``groups.add()`` uses bulk_create with + ignore_conflicts). The conflicting row is never inserted, so the NEW + transition table is empty and the trigger updates nothing. The consequence + worth remembering: an already-drifted projection row cannot be repaired by + re-adding the membership -- that is what check_search_projection and a + backfill are for. Statement-level rather than row-level because the bulk-upsert endpoint writes membership with bulk_create (reports/api/viewsets.py:215,229), so one statement @@ -15,6 +34,10 @@ from django.db import migrations +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill +# and check_search_projection.DRIFT_SQL -- four copies, keep them in sync. The +# drift query is the one that detects divergence, so a change made here and not +# there is a bug that reports itself as healthy. SYNC_GROUP_IDS = """ CREATE OR REPLACE FUNCTION pgsearch_sync_group_ids() RETURNS trigger LANGUAGE plpgsql AS $$ @@ -29,6 +52,10 @@ END $$; """ +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill +# and check_search_projection.DRIFT_SQL -- four copies, keep them in sync. The +# drift query is the one that detects divergence, so a change made here and not +# there is a bug that reports itself as healthy. SYNC_MODALITY_CODES = """ CREATE OR REPLACE FUNCTION pgsearch_sync_modality_codes() RETURNS trigger LANGUAGE plpgsql AS $$ diff --git a/radis/pgsearch/models.py b/radis/pgsearch/models.py index fa2eaeb5..312087ec 100644 --- a/radis/pgsearch/models.py +++ b/radis/pgsearch/models.py @@ -25,8 +25,11 @@ class ReportSearchIndex(models.Model): # Search projection: mirrors of the Report fields the scan filters on, so # the FTS candidate query stays single-table. Maintained by the triggers in # migration 0004 and populated on creation by signals.py / indexing.py. - # Nullable so every AddField stays metadata-only on a large table; - # check_search_projection guards against drift instead of a NOT NULL scan. + # Every column is either nullable (the scalars) or NOT NULL with a constant + # default (the two arrays, DEFAULT '{}'); both forms keep the AddField in + # migration 0003 metadata-only on a large table. The scalars stay nullable + # rather than being tightened afterwards, which would cost a validating + # table scan for no benefit -- check_search_projection guards against drift. group_ids = ArrayField(models.IntegerField(), default=list) modality_codes = ArrayField(models.CharField(max_length=16), default=list) language_code = models.CharField(max_length=10, null=True) # noqa: DJ001 From 11d855751cd4135e11b52ed932d1457c3a26e687 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Sun, 30 Aug 2026 22:33:28 +0000 Subject: [PATCH 35/50] Report how many reports have no search index row yet check_search_projection only compared rows that exist on both sides, so a report whose index row was never created was invisible to it. Indexing is deferred, so "no index row yet" is a normal transient state and must not be an error -- but the count is what lets an operator tell "200 pending" from "40,000 permanently stuck". It is printed as informational output before the drift check. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../commands/check_search_projection.py | 20 +++++++++++++++++ .../tests/test_check_search_projection.py | 22 ++++++++++++++++++- 2 files changed, 41 insertions(+), 1 deletion(-) diff --git a/radis/pgsearch/management/commands/check_search_projection.py b/radis/pgsearch/management/commands/check_search_projection.py index 32440352..f0ac3d38 100644 --- a/radis/pgsearch/management/commands/check_search_projection.py +++ b/radis/pgsearch/management/commands/check_search_projection.py @@ -45,6 +45,18 @@ """ +# Informational, not drift: an index row is created by a signal or by the bulk +# indexing path, so a report can legitimately be waiting for one. The number is +# only meaningful over time -- a few hundred that drain are normal, tens of +# thousands that never move mean indexing is stuck. +MISSING_INDEX_ROWS_SQL = """ +SELECT count(*) + FROM reports_report r + LEFT JOIN pgsearch_reportsearchindex rsi ON rsi.report_id = r.id + WHERE rsi.report_id IS NULL +""" + + class Command(BaseCommand): help = "Verify the ReportSearchIndex search projection against its sources." @@ -54,6 +66,14 @@ def handle(self, *args, **options): columns = [column.name for column in cursor.description] counts = dict(zip(columns, cursor.fetchone(), strict=True)) + cursor.execute(MISSING_INDEX_ROWS_SQL) + missing_index_rows = cursor.fetchone()[0] + + self.stdout.write( + f"Reports without a search index row: {missing_index_rows} " + "(indexing is deferred, so a number that keeps falling is normal)" + ) + drifted = {name: count for name, count in counts.items() if count} if drifted: details = ", ".join(f"{name}: {count}" for name, count in sorted(drifted.items())) diff --git a/radis/pgsearch/tests/test_check_search_projection.py b/radis/pgsearch/tests/test_check_search_projection.py index e2ed19c9..311d4ef8 100644 --- a/radis/pgsearch/tests/test_check_search_projection.py +++ b/radis/pgsearch/tests/test_check_search_projection.py @@ -2,6 +2,8 @@ prove it still matches its sources -- after a restore, a bulk import, or a Language.code rename, which no trigger covers.""" +from io import StringIO + import pytest from adit_radis_shared.accounts.factories import GroupFactory from django.core.management import call_command @@ -17,7 +19,25 @@ def test_reports_no_drift_on_a_healthy_corpus(): report = ReportFactory.create(language=LanguageFactory.create(code="en")) report.groups.add(GroupFactory.create()) - call_command("check_search_projection") + out = StringIO() + call_command("check_search_projection", stdout=out) + + assert "Reports without a search index row: 0" in out.getvalue() + + +def test_reports_how_many_reports_have_no_index_row_yet(): + """Informational, not an error: indexing is deferred, so a report can be + legitimately waiting for its index row. The number is what lets an operator + tell "200 pending" from "40,000 permanently stuck".""" + report = ReportFactory.create(language=LanguageFactory.create(code="en")) + report.groups.add(GroupFactory.create()) + ReportSearchIndex.objects.filter(report=report).delete() + + out = StringIO() + call_command("check_search_projection", stdout=out) + + assert "Reports without a search index row: 1" in out.getvalue() + assert "matches its sources" in out.getvalue() def test_detects_drifted_group_ids(): From 6ef6228ccdc51038eef78deeade087320c5d2a2e Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 08:14:26 +0000 Subject: [PATCH 36/50] Trim the migration-wait comments to the durable constraint The comments named one migration and its measured duration at one corpus size. That is a fact about today's data rather than about the code, it already lives in the commit message and the admin guide, and the figure was a floor anyway since it predates the row-locking change. Keeps what the code cannot say for itself: that init (or web, in dev) opens its port only after migrate finishes, that everything below waits on that port, and that dev has no restart policy to recover a timeout. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- docker-compose.dev.yml | 5 ++--- docker-compose.prod.yml | 8 ++++---- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 18ce286f..7e76a155 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -27,9 +27,8 @@ services: - never # In development `web` runs the migrations the workers below wait for, so - # WAIT_INIT_TIMEOUT must outlast the longest migration here too (the search - # projection backfill takes about ten minutes at 8M reports). There is no - # restart policy in dev, so a timeout leaves the worker containers exited. + # their WAIT_INIT_TIMEOUT must outlast the longest migration too. Dev has no + # restart policy, so a timeout leaves the worker containers exited. web: <<: *default-app image: radis_dev-web:latest diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 4702686d..f8e516c5 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -20,10 +20,10 @@ services: # and wait for it to be finished by the web service containers. # # init opens its ok_server port only after `migrate` finishes, and every - # service below waits for that port. A data migration can run for many - # minutes on a large archive (the search projection backfill takes about ten - # at 8M reports), so WAIT_INIT_TIMEOUT must stay well above the longest - # migration -- a timeout here aborts the `&&` chain and the container exits. + # service below waits for that port, so WAIT_INIT_TIMEOUT must stay above the + # longest migration a deploy runs -- a data migration over a large archive can + # take many minutes, and a timeout aborts the `&&` chain and exits the + # container. init: <<: *default-app hostname: init.local From 7a55d4369cedaa2af310cde1c928c97f426ce1c6 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 09:01:58 +0000 Subject: [PATCH 37/50] Streamline projection comments Cut narration and one-time migration framing from the comments added with the search projection, and correct the signals.py note: the trigger write it guards against happens in an earlier statement of the same transaction, not in the same statement. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- .../migrations/0006_search_projection_indexes.py | 10 ++++------ radis/pgsearch/models.py | 9 ++++----- radis/pgsearch/signals.py | 10 ++++------ radis/pgsearch/tests/test_search_projection.py | 1 - 4 files changed, 12 insertions(+), 18 deletions(-) diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0006_search_projection_indexes.py index f28118fd..4118d95b 100644 --- a/radis/pgsearch/migrations/0006_search_projection_indexes.py +++ b/radis/pgsearch/migrations/0006_search_projection_indexes.py @@ -29,12 +29,10 @@ class Migration(migrations.Migration): reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", ), migrations.RunSQL( - # Required, not hygiene: the ten projection columns and the two - # indexes above carry no statistics, and the single-table query - # shape this whole change exists for depends on the planner - # choosing a parallel sequential scan with a top-N heapsort. - # ANALYZE is legal inside a transaction block, so this migration - # stays atomic. + # Required, not hygiene: the projection columns and the indexes + # above carry no statistics, and the single-table query shape + # depends on the planner choosing a parallel sequential scan with + # a top-N heapsort. ANALYZE is transaction-legal, so 0006 stays atomic. "ANALYZE pgsearch_reportsearchindex;", reverse_sql=migrations.RunSQL.noop, ), diff --git a/radis/pgsearch/models.py b/radis/pgsearch/models.py index 312087ec..e009cd7a 100644 --- a/radis/pgsearch/models.py +++ b/radis/pgsearch/models.py @@ -25,11 +25,10 @@ class ReportSearchIndex(models.Model): # Search projection: mirrors of the Report fields the scan filters on, so # the FTS candidate query stays single-table. Maintained by the triggers in # migration 0004 and populated on creation by signals.py / indexing.py. - # Every column is either nullable (the scalars) or NOT NULL with a constant - # default (the two arrays, DEFAULT '{}'); both forms keep the AddField in - # migration 0003 metadata-only on a large table. The scalars stay nullable - # rather than being tightened afterwards, which would cost a validating - # table scan for no benefit -- check_search_projection guards against drift. + # The scalars are nullable and the arrays NOT NULL with a constant default, + # which is what keeps adding a column here metadata-only on a large table. + # Tightening them afterwards would cost a validating scan for no benefit: + # check_search_projection guards against drift. group_ids = ArrayField(models.IntegerField(), default=list) modality_codes = ArrayField(models.CharField(max_length=16), default=list) language_code = models.CharField(max_length=10, null=True) # noqa: DJ001 diff --git a/radis/pgsearch/signals.py b/radis/pgsearch/signals.py index 9a884596..a977133b 100644 --- a/radis/pgsearch/signals.py +++ b/radis/pgsearch/signals.py @@ -17,10 +17,8 @@ def create_or_update_report_search_index(sender, instance, created, **kwargs): return # update_fields is required here, not just an optimization: OneToOneField - # reciprocally caches the index on the report instance the moment it's - # created, so instance.search_index below is often that same stale - # Python object from creation time. A bare save() would write ALL of its - # fields back -- including the projection columns the AFTER UPDATE - # trigger on reports_report just set moments ago in this same statement - # -- clobbering the trigger's fresh values with the stale ones. + # caches the index on the report instance, so instance.search_index below + # is often a stale copy from creation time. A bare save() would write all + # of its fields back, clobbering the projection columns the AFTER UPDATE + # trigger on reports_report set earlier in this transaction. instance.search_index.save(update_fields=["search_vector"]) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index c5f0a7ff..1c738b01 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -417,7 +417,6 @@ def remove_the_group(): assert not worker.is_alive(), "the concurrent writer never finished" assert outcome["result"] == "blocked", outcome - # The lock was the only obstacle: the same removal goes through now. report.groups.remove(group) assert ReportSearchIndex.objects.get(report=report).group_ids == [] From 4788985dc3ee1be43f5a847da91d3b8942d0fa80 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 09:10:14 +0000 Subject: [PATCH 38/50] Correct the spec where the implementation diverged Three residuals from the build, all resolved in the spec rather than the code: - Step 4 called for VACUUM ANALYZE; the migration runs ANALYZE alone. VACUUM cannot run in a transaction block, does not shrink the file, and autovacuum does the same work after a full-table rewrite. - Step 3's autovacuum_vacuum_scale_factor tweak was never implemented. Its premise is contradicted by the spec's own bloat measurement, so it moves to the ruled-out table instead of into a migration. - Step 3 gained a per-chunk lock pass during review that the spec never recorded, which also makes the 9 min 20 s backfill figure a floor. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 26 ++++++++++++------- .../0005_search_projection_backfill.py | 2 +- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 3dcceb90..a96dad33 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -67,6 +67,7 @@ Recorded because each is the obvious question a reviewer will raise. | Rewrite the joins as `EXISTS` | Worse: 1.55 s vs 1.05 s (1M rig). The planner still hashes. | | Materialised id-only CTE, joins on narrow rows, then rank | Worse: 11,460 ms for `findings`, 3,089 ms for `pneumonia` at 5M. The joins are still over 5,001,000 rows. **This was the only way to avoid duplicating access-control data without changing the domain model (see §6), and it does not work.** | | `strip()`ped second tsvector for ranking | Rejected (1M rig). 1.4x faster but collapses `ts_rank` to **1 distinct value** across 200,000 matching rows — `strip()` removes the positions that encode term frequency, so the ordering becomes a no-op. | +| Lowering `autovacuum_vacuum_scale_factor` during the backfill | Dropped, unmeasured. The intent was for one chunk's dead tuples to be reused by the next, but rows grow ~120 bytes so new versions rarely fit the freed slots, and a cost-throttled autovacuum cannot keep pace with a chunk every few seconds. The 10 GB → 21 GB doubling in §4.4 was measured without it and is expected regardless. A site that hits steady-state bloat can `ALTER TABLE ... SET` it without a migration. | | Restricting recall ("rank only the most recent N", "refuse broad queries") | Withdrawn. No production search engine restricts recall; they restrict pagination depth and count precision. Not needed here anyway. | ## 4. Design @@ -294,27 +295,32 @@ untouched — `page_rows` still `select_related("report")` for `ts_headline` ove multi-minute backfill still land correctly. A chunk that later reprocesses the same row simply rewrites the same current values. 3. **Backfill**, chunked by `report_id` range (50,000 rows) with one transaction - per chunk (hence `atomic = False`). Each chunk is a set-based `UPDATE ... FROM - reports_report` with two lateral `array_agg` subqueries for the arrays. The - migration lowers `autovacuum_vacuum_scale_factor` on the table for the - duration and resets it after, so dead tuples from one chunk are reclaimed and - reused by the next rather than accumulating across the run. The chunk size is - a constant, not a setting: a migration is a historical record and should do - the same thing on every machine. -4. **Create indexes last, then `VACUUM ANALYZE`.** Building GIN indexes on empty + per chunk (hence `atomic = False`). Each chunk locks its rows (`SELECT ... + FOR UPDATE`) and then applies a set-based `UPDATE ... FROM reports_report` + with two lateral `array_agg` subqueries for the arrays. The lock pass closes + an EvalPlanQual window under READ COMMITTED, where a stale aggregate could + silently restore a group a concurrent trigger had just removed. The chunk + size is a constant, not a setting: a migration is a historical record and + should do the same thing on every machine. +4. **Create indexes last, then `ANALYZE`.** Building GIN indexes on empty columns is wasted work — but the stronger reason is that while the new columns are unindexed the backfill can use HOT updates, whose old row versions are reclaimed by page pruning without index cleanup. That is the main defence against the bloat below, so this ordering must not be changed. The `ANALYZE` is required, not hygiene: ten new columns and two new indexes carry no statistics, and this design depends on the planner choosing a parallel - sequential scan with a top-N heapsort. + sequential scan with a top-N heapsort. No `VACUUM` alongside it: it cannot + run inside a transaction block, so it would cost this migration its + atomicity; it does not shrink the file (see below); and rewriting every row + puts the table far past the autovacuum threshold, so autovacuum does the + same work unattended minutes later. **Deployment consequences.** `docker-compose.prod.yml` runs `manage.py migrate` in the `init` service and `web` waits on it, so the backfill blocks the deploy rather than serving half-projected rows — the right failure mode for access-control data. Deploy time therefore grows with corpus size: **measured at -9 min 20 s for 8M reports**, in 50,000-row chunks. That extrapolates to roughly +9 min 20 s for 8M reports**, in 50,000-row chunks — a floor, since that run +predates the per-chunk lock pass in step 3. That extrapolates to roughly 18 minutes at 15M. Adding the ten columns is genuinely free (1.3 ms, confirming the metadata-only claim); the nine minutes is the row rewrite. diff --git a/radis/pgsearch/migrations/0005_search_projection_backfill.py b/radis/pgsearch/migrations/0005_search_projection_backfill.py index 6c9b7cf3..fa635ee0 100644 --- a/radis/pgsearch/migrations/0005_search_projection_backfill.py +++ b/radis/pgsearch/migrations/0005_search_projection_backfill.py @@ -1,7 +1,7 @@ """Backfill the ReportSearchIndex search projection for existing rows. atomic = False so each chunk commits on its own: at the 8M design target this -runs for about nine and a half minutes, and one transaction that long would +runs for roughly ten minutes, and one transaction that long would pin an equally long-lived snapshot. Runs after the triggers (0004) on purpose. A report edited during the backfill From ad0877f20aa362c834793cbfa5652237f86a4394 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 09:34:48 +0000 Subject: [PATCH 39/50] Correct the bloat penalty with measured numbers The spec claimed the 882 ms figure already included the backfill's heap doubling, and predicted 450-650 ms on a compacted table. Measuring the 8M rig before and after a VACUUM FULL shows the opposite: 2,550 ms bloated against 870 ms compacted, which is where the 882 ms figure actually sits. The penalty is ~2.9x rather than 2x because the inflated table stops fitting the page cache. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...08-29-fts-query-shape-performance-design.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index a96dad33..566b85c9 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -342,10 +342,20 @@ tuples, and a plain `VACUUM` clears every dead tuple while leaving the file at Chunking and creating indexes last (steps 3 and 4) did **not** prevent this — the rows grow by ~120 bytes, so most updated tuples cannot fit back into their original page and HOT does not apply. The doubling should be treated as expected, -not as something the migration avoids. The consequence is a sequential scan -reading roughly twice what it needs to, and the 882 ms in §2 already includes that -penalty; on a compacted table the same query would land nearer 450–650 ms. It -resolves on its own as the archive grows into the free space. +not as something the migration avoids. The consequence is worse than the linear +2x it looks like, because the inflated table stops fitting the page cache. +Measured on the 8M rig: at a 22 GB heap (27 GB total) `findings` took +**2,550 ms** and `pneumonia` **3,271 ms**; after a `VACUUM FULL` took the heap to +11 GB (13 GB total) the same queries took **870 ms** and **818 ms** — about 2.9x, +on a 31 GB host with ~20 GB of page cache. + +The 882 ms in §2 therefore describes the *compacted* state, not the bloated one. +An earlier draft of this section claimed the reverse and predicted 450–650 ms +after compaction; compaction was measured at 870 ms, so that prediction was +wrong in both directions. The penalty resolves on its own as the archive grows +into the free space. A site whose post-migration table no longer fits RAM will +see the bloated figures until then, and can reclaim the space sooner only with +the caveats below. **Do not run `VACUUM FULL` from the migration, and do not recommend it blindly.** It rebuilds every index on the table, including `pgsearch_embedding_hnsw`; on a From 6833e5f7d066462d3f3c730cbeaac5bf883b05c2 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 10:57:32 +0000 Subject: [PATCH 40/50] Document reclaiming space after the projection migration The backfill roughly doubles the report index table on disk and autovacuum does not shrink it, so search scans about twice the pages until the space is reused. Records what that costs, that it resolves on its own, and that VACUUM FULL is the wrong tool wherever embeddings are populated -- it rebuilds the HNSW index under an exclusive lock. Points at pg_repack for that case. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- docs/user-docs/admin-guide.md | 43 +++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 28d71c02..69a5d29c 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -58,6 +58,49 @@ overrides -- set them in `.env` only if the defaults do not suit your hardware. | `POSTGRES_MAX_WORKER_PROCESSES` | `8` | As above. | | `POSTGRES_SHARED_BUFFERS` | `128MB` | PostgreSQL's default. Around 25% of host RAM is the usual recommendation; a value larger than the container's memory will prevent PostgreSQL from starting. | +### Reclaiming space after the search-projection migration + +The release that added the search projection rewrites every row of the report +index table, which roughly doubles it on disk -- measured at 8 million reports, +the table grew from 10 GB to 21 GB. Autovacuum reclaims the dead rows within +minutes and no action is needed to keep the database healthy, but reclaiming +them does not shrink the file: the space is marked reusable and stays allocated. + +Search scans that table, so until the space is reused the scan reads about twice +the pages it needs to. How much that costs depends on whether the inflated table +still fits the host's page cache: + +- **It fits.** Expect roughly the page-count ratio, near 1.9x on the scan. At + 8 million reports the inflated table is about 27 GB including indexes, so any + host with 64 GB or more stays in this case. +- **It does not fit.** The scan starts reading from disk and the penalty grows + well beyond 2x. On a 31 GB test host with about 20 GB of page cache, a query + matching every report took 2,550 ms inflated against 870 ms compacted. + +**This resolves on its own.** New reports are written into the freed space, so a +growing archive returns to its normal size without intervention. On most +installations that is the right thing to do: wait. + +If search is measurably slow and you do not want to wait, the table can be +compacted -- but read this first, because the cheap-looking option is a trap +when embeddings are in use: + +- **With `EMBEDDINGS_MODEL` set** (the normal production setup), do **not** run + `VACUUM FULL`, and never inside a deployment window. It rebuilds every index on + the table, including the HNSW vector index, which means building an HNSW graph + over millions of vectors while holding an `ACCESS EXCLUSIVE` lock. That can run + for hours with search unavailable throughout. Use + [`pg_repack`](https://reorg.github.io/pg_repack/) instead: it performs the same + compaction without holding a long exclusive lock. +- **With `EMBEDDINGS_MODEL` empty** (full-text search only), the HNSW rebuild is + a no-op and `VACUUM FULL pgsearch_reportsearchindex;` is cheap -- 3 minutes for + 8 million reports. It still takes an `ACCESS EXCLUSIVE` lock, so run it in a + maintenance window, and make sure the filesystem has room for a second copy of + the table while it runs. + +A plain `VACUUM` is not an alternative for this: it reclaims dead rows, which +autovacuum has already done, and leaves the file at its inflated size. + ## User and Group Management Administrators can create users by navigating to the Django Admin section. Alternatively, users can self-register, after which an administrator must approve and activate their account. From f9d58b61ce999f4c979368b3e864cb3202604281 Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 11:15:39 +0000 Subject: [PATCH 41/50] Correct the claim that the backfill bloat resolves on its own Both documents said a growing archive refills the freed space without intervention. The free space is roughly one full copy of the table and only net new inserts consume it, since an update frees its own old row version, so refilling an 8M-row archive needs on the order of 8M new reports -- years at realistic ingest rates. Emptied pages stay in the relation and are still read by every scan, so the penalty persists until the table is compacted deliberately. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...-08-29-fts-query-shape-performance-design.md | 15 +++++++++------ docs/user-docs/admin-guide.md | 17 ++++++++++------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 566b85c9..9ab02c22 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -352,10 +352,11 @@ on a 31 GB host with ~20 GB of page cache. The 882 ms in §2 therefore describes the *compacted* state, not the bloated one. An earlier draft of this section claimed the reverse and predicted 450–650 ms after compaction; compaction was measured at 870 ms, so that prediction was -wrong in both directions. The penalty resolves on its own as the archive grows -into the free space. A site whose post-migration table no longer fits RAM will -see the bloated figures until then, and can reclaim the space sooner only with -the caveats below. +wrong in both directions. The free space is only consumed by net new inserts (an update frees its own old +version), so it amounts to one full copy of the table waiting on roughly as many +new reports as the archive already holds -- many years at hospital ingest rates. +Treat the penalty as persistent until the table is compacted deliberately, with +the caveats below, rather than as something that decays. **Do not run `VACUUM FULL` from the migration, and do not recommend it blindly.** It rebuilds every index on the table, including `pgsearch_embedding_hnsw`; on a @@ -363,8 +364,10 @@ deployment with embeddings populated that is an HNSW rebuild over millions of vectors — plausibly hours, under an ACCESS EXCLUSIVE lock — and it needs peak disk for old plus new (~19 GB at 5M). Sites that want the space returned should run it deliberately in a maintenance window knowing that cost, or use `pg_repack` -if they have it. This design requires neither. Most sites will not need it: an -archive that keeps growing refills the freed space on its own. +if they have it. This design requires neither to be correct, but a deployment +with embeddings populated should expect to run `pg_repack` once after the +migration: `VACUUM FULL` is an HNSW rebuild there, and waiting for organic +refill is a multi-year proposition. An online alternative — resumable backfill command plus a feature flag on the query path — was considered and rejected: it doubles the query-layer code for diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 69a5d29c..f3ef29db 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -77,13 +77,16 @@ still fits the host's page cache: well beyond 2x. On a 31 GB test host with about 20 GB of page cache, a query matching every report took 2,550 ms inflated against 870 ms compacted. -**This resolves on its own.** New reports are written into the freed space, so a -growing archive returns to its normal size without intervention. On most -installations that is the right thing to do: wait. - -If search is measurably slow and you do not want to wait, the table can be -compacted -- but read this first, because the cheap-looking option is a trap -when embeddings are in use: +**Do not expect this to resolve on its own within any useful timeframe.** The +backfill leaves free space equal to roughly one full copy of the table, and only +*net new* reports consume it -- an update frees its own old row version, so +report edits are space-neutral. Refilling an 8-million-row archive therefore +takes on the order of 8 million new reports, which at typical hospital volumes is +many years. Emptied pages stay part of the table and are still read by every +scan. + +Plan to compact the table once after this migration -- but read this first, +because the obvious tool is a trap when embeddings are in use: - **With `EMBEDDINGS_MODEL` set** (the normal production setup), do **not** run `VACUUM FULL`, and never inside a deployment window. It rebuilds every index on From 0eec39601f15c30aa7480d6c286f72a712e728ef Mon Sep 17 00:00:00 2001 From: Kai Schlamp Date: Mon, 31 Aug 2026 12:04:09 +0000 Subject: [PATCH 42/50] Document VACUUM FULL as the compaction procedure Records the accepted approach: a VACUUM FULL in its own maintenance window, scheduled apart from the release rather than bolted onto it. Adds what an operator needs to plan it -- that the cost is the HNSW rebuild rather than the table copy, that embeddings are copied and never recomputed, the disk and maintenance_work_mem prerequisites -- and keeps dropping the vector index and pg_repack as optional ways to shorten the outage. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HqE6h8sRWQT57B9Lomfvbk --- ...8-29-fts-query-shape-performance-design.md | 8 +- docs/user-docs/admin-guide.md | 82 +++++++++++++++---- 2 files changed, 69 insertions(+), 21 deletions(-) diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 9ab02c22..19b00631 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -365,9 +365,11 @@ vectors — plausibly hours, under an ACCESS EXCLUSIVE lock — and it needs pea disk for old plus new (~19 GB at 5M). Sites that want the space returned should run it deliberately in a maintenance window knowing that cost, or use `pg_repack` if they have it. This design requires neither to be correct, but a deployment -with embeddings populated should expect to run `pg_repack` once after the -migration: `VACUUM FULL` is an HNSW rebuild there, and waiting for organic -refill is a multi-year proposition. +with embeddings populated should expect to compact once after the migration, +since waiting for organic refill is a multi-year proposition. The accepted +approach for this deployment is a `VACUUM FULL` in its own maintenance window, +scheduled separately from the release. The admin guide carries the procedure and +the options for shortening the outage. An online alternative — resumable backfill command plus a feature flag on the query path — was considered and rejected: it doubles the query-layer code for diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index f3ef29db..30d8fae7 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -85,24 +85,70 @@ takes on the order of 8 million new reports, which at typical hospital volumes i many years. Emptied pages stay part of the table and are still read by every scan. -Plan to compact the table once after this migration -- but read this first, -because the obvious tool is a trap when embeddings are in use: - -- **With `EMBEDDINGS_MODEL` set** (the normal production setup), do **not** run - `VACUUM FULL`, and never inside a deployment window. It rebuilds every index on - the table, including the HNSW vector index, which means building an HNSW graph - over millions of vectors while holding an `ACCESS EXCLUSIVE` lock. That can run - for hours with search unavailable throughout. Use - [`pg_repack`](https://reorg.github.io/pg_repack/) instead: it performs the same - compaction without holding a long exclusive lock. -- **With `EMBEDDINGS_MODEL` empty** (full-text search only), the HNSW rebuild is - a no-op and `VACUUM FULL pgsearch_reportsearchindex;` is cheap -- 3 minutes for - 8 million reports. It still takes an `ACCESS EXCLUSIVE` lock, so run it in a - maintenance window, and make sure the filesystem has room for a second copy of - the table while it runs. - -A plain `VACUUM` is not an alternative for this: it reclaims dead rows, which -autovacuum has already done, and leaves the file at its inflated size. +Plan to compact the table once after this migration. The tool is +`VACUUM FULL pgsearch_reportsearchindex;`, and it needs a maintenance window. + +**Do not bolt this onto the upgrade window.** Nothing is broken in the meantime +-- the system is fully functional, search is just slower -- so deploy the release +first, bring the stack back up, and schedule the compaction separately. That way +a long compaction never blocks a release. + +**Budget the time realistically.** Copying the table is the quick part. The cost +is dominated by rebuilding the HNSW vector index, because `VACUUM FULL` relocates +every row and so invalidates and rebuilds every index on the table. Building an +HNSW graph over millions of vectors takes hours, and `VACUUM FULL` holds an +`ACCESS EXCLUSIVE` lock for the whole operation, so search is unavailable +throughout. With `EMBEDDINGS_MODEL` empty there is no HNSW index to rebuild and +the whole thing takes minutes instead (measured: 3 minutes for 8 million +reports). + +Nothing is recomputed and nothing is lost. Embeddings are stored data and are +copied across as they are -- the embedding service is never called, and the index +is rebuilt from the vectors already in the table. + +Before starting: + +- **Free disk.** Both copies of the table exist at once, so the filesystem needs + room for the compacted table on top of the inflated one. +- **Raise `maintenance_work_mem`** for the session. pgvector builds the HNSW + graph in memory when it fits and falls back to a much slower on-disk path when + it does not. At 8 million rows and 1024 dimensions the vectors alone are tens of + gigabytes, so the default leaves you on the slow path. Set it as high as the + host can spare, for example `SET maintenance_work_mem = '32GB';` in the same + session as the `VACUUM FULL`. +- **Enable maintenance mode** so users get an explanation rather than timeouts. + +If hours of unavailable search is not acceptable, there are two ways to shorten +it, both optional: + +- **Rebuild the vector index separately.** Drop the HNSW index first, run + `VACUUM FULL` (now minutes, since only small indexes are rebuilt), bring the + stack back up, then recreate the index without blocking: + + ```sql + DROP INDEX pgsearch_embedding_hnsw; + VACUUM FULL pgsearch_reportsearchindex; + -- bring the stack back up here, then: + CREATE INDEX CONCURRENTLY pgsearch_embedding_hnsw + ON pgsearch_reportsearchindex USING hnsw (embedding vector_cosine_ops) + WITH (m = 16, ef_construction = 64); + ``` + + This trades hours of full downtime for minutes of downtime plus hours of + degraded semantic search: without the index, vector similarity falls back to an + exact scan over every row, which is very slow on a large archive. Full-text + search is unaffected. Keep the index name and definition exactly as above, or + Django's migration state will no longer match the database. If + `CREATE INDEX CONCURRENTLY` fails it leaves an invalid index behind -- drop it + and retry. +- **Use [`pg_repack`](https://reorg.github.io/pg_repack/)**, which does the same + compaction against a shadow copy while the table stays online, taking a brief + exclusive lock only at the swap. It is not part of the PostgreSQL image RADIS + ships, so this means building or installing the extension and its client binary + first. + +A plain `VACUUM` is not an alternative for any of this: it reclaims dead rows, +which autovacuum has already done, and leaves the file at its inflated size. ## User and Group Management From 0490299e78cc5a94e25419da29e8828f370f5e79 Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 22:24:35 +0200 Subject: [PATCH 43/50] Drop and rebuild the HNSW index around the projection backfill Every backfilled row is a full-row rewrite into pages with no free space (non-HOT), so with the embedding index in place each of the corpus's embedded rows also paid an HNSW graph insert: ~143 ms and ~2,160 buffers per row measured on an 8M-report corpus with 1.7M vectors, ~68 hours instead of the documented ten minutes. 0005 now drops the index first (database-only, IF EXISTS) and 0006 rebuilds it from the stored vectors in one bulk build, tunable via PGSEARCH_HNSW_REBUILD_* and backed by a properly sized /dev/shm tmpfs in compose (parallel builds pre-allocate their whole memory budget there). Docs now state the ten-minute figure's no-embeddings assumption. Nothing is ever re-embedded. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- AGENTS.md | 12 +++++++ docker-compose.base.yml | 10 ++++++ ...8-29-fts-query-shape-performance-design.md | 10 ++++++ docs/user-docs/admin-guide.md | 18 +++++++--- .../0005_search_projection_backfill.py | 23 ++++++++++-- .../0006_search_projection_indexes.py | 36 +++++++++++++++++++ radis/settings/base.py | 12 +++++++ 7 files changed, 114 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4558aee6..6d731b15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -128,6 +128,13 @@ Hybrid search embeddings (`radis.pgsearch`): swap — cached query vectors otherwise keep serving stale results for up to this long - `EMBEDDINGS_BATCH_SIZE`, `EMBEDDINGS_SUBJOB_SIZE`, `EMBEDDINGS_WORKER_CONCURRENCY`: Throughput tuning +- `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` (default `2GB`), + `PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS` (default `2`): session knobs for the + HNSW rebuild in pgsearch migration 0006 (the projection backfill drops the + embedding index and 0006 recreates it from the stored vectors). Raise on a + large embedded corpus so the graph builds in memory; irrelevant for FTS-only + installs. Size `POSTGRES_SHM_SIZE_BYTES` (below) at least as large as the + memory value Auto-labeling (`radis.labels`): @@ -156,6 +163,11 @@ only to override the compose file's default): `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` on a larger host - `POSTGRES_SHARED_BUFFERS`: PostgreSQL's shared memory buffer (default `128MB`, PostgreSQL's own default) +- `POSTGRES_SHM_SIZE_BYTES`: tmpfs size of the container's `/dev/shm` in bytes + (default `1073741824` = 1 GiB; Docker's own default of 64 MB is too small for + parallel queries). Parallel maintenance pre-allocates its whole + `maintenance_work_mem` budget here — for the pgsearch 0006 HNSW rebuild set + it at least as large as `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` ## Code Standards diff --git a/docker-compose.base.yml b/docker-compose.base.yml index 8b196030..9742cec8 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -52,6 +52,16 @@ services: - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} volumes: - postgres_data:/var/lib/postgresql/data + # PostgreSQL's parallel workers exchange data through dynamic shared + # memory in /dev/shm, and Docker's default segment is 64 MB. Parallel + # maintenance (the HNSW rebuild in pgsearch migration 0006) pre-allocates + # its whole budget there, so size this at least as large as + # PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM. Bytes, because + # `docker stack deploy` rejects unit suffixes here (default: 1 GiB). + - type: tmpfs + target: /dev/shm + tmpfs: + size: ${POSTGRES_SHM_SIZE_BYTES:-1073741824} volumes: postgres_data: diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 19b00631..1f98554a 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -333,6 +333,16 @@ Roughly ten minutes of announced downtime for a one-time migration has been accepted for this deployment, which is what keeps the blocking approach below. A site that cannot take that window needs the online alternative instead. +The ten-minute figure was measured on rigs built **without embeddings**. On an +embedded corpus every rewritten row is a non-HOT update that would also insert +into the HNSW index (~143 ms and ~2,160 buffers per embedded row measured on +staging: ~68 hours for 1.7M vectors). The backfill therefore drops the HNSW +index first and 0006 rebuilds it from the stored vectors in one bulk build +(7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers), tunable +via `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` / +`PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS`, with `/dev/shm` sized to match +(`POSTGRES_SHM_SIZE_BYTES` in the compose files). + The backfill rewrites every row: the tsvectors average 1220 bytes and nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the new columns. Measured at 8M, the heap goes **10 GB → 21 GB** with 7,988,672 dead diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 30d8fae7..be46e16b 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -32,10 +32,20 @@ The `init` service runs `manage.py migrate` and every other service waits for it to finish, so the stack stays unavailable for the whole migration. The release that added the search projection to the report search index is the current example: it rewrites every row of that table, measured at about **ten minutes -for 8 million reports** and proportionally longer on a larger archive. Plan the -maintenance window around that, and do not shorten `WAIT_INIT_TIMEOUT` (default -one hour, see the compose files) below the expected migration time -- the -services that wait on `init` exit when that timeout expires. +for 8 million reports** without embeddings and proportionally longer on a +larger archive. On an archive **with embeddings** the migration first drops the +vector index and rebuilds it afterwards from the stored vectors (nothing is +re-embedded) -- without that, every rewritten row would also pay an insert into +the vector index (~143 ms per embedded row: days, not minutes). For a large +embedded archive raise `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` and +`PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS` so the rebuild runs in memory (a +1.7M-vector archive rebuilt in under 8 minutes with 16GB and 7 workers), and +size `POSTGRES_SHM_SIZE_BYTES` at least as large as that memory value -- the +parallel build pre-allocates it in `/dev/shm`, and other parallel queries can +fail with "could not resize shared memory segment" while it runs. Plan the +maintenance window around all of that, and do not shorten `WAIT_INIT_TIMEOUT` +(default one hour, see the compose files) below the expected migration time -- +the services that wait on `init` exit when that timeout expires. Keep the web tier stopped for the whole migration, which steps 5 and 9 above already do -- do not start it early to "check on progress". While that diff --git a/radis/pgsearch/migrations/0005_search_projection_backfill.py b/radis/pgsearch/migrations/0005_search_projection_backfill.py index fa635ee0..b073f995 100644 --- a/radis/pgsearch/migrations/0005_search_projection_backfill.py +++ b/radis/pgsearch/migrations/0005_search_projection_backfill.py @@ -1,8 +1,19 @@ """Backfill the ReportSearchIndex search projection for existing rows. atomic = False so each chunk commits on its own: at the 8M design target this -runs for roughly ten minutes, and one transaction that long would -pin an equally long-lived snapshot. +runs for roughly ten minutes once the HNSW index is out of the way (below), and +one transaction that long would pin an equally long-lived snapshot. + +The first operation drops the HNSW embedding index for the duration of the +backfill; 0006 rebuilds it from the stored vectors. Every backfilled row is a +full-row rewrite into pages that hold no free space -- a non-HOT update -- so +with the index in place each embedded row would also insert into the HNSW +graph: ~143 ms and ~2,160 buffer touches per row measured on an 8M corpus with +1.7M vectors, which turns the ten-minute window into days. Without embeddings +the drop and the rebuild are both effectively instant. The drop is +database-only (model state keeps declaring the index; 0006 makes it true +again) and IF EXISTS because atomic = False re-runs every operation after a +failure. Runs after the triggers (0004) on purpose. A report edited during the backfill is corrected by its trigger, and a chunk that later reprocesses the same row @@ -90,4 +101,10 @@ class Migration(migrations.Migration): dependencies = [("pgsearch", "0004_search_projection_triggers")] - operations = [migrations.RunPython(backfill, migrations.RunPython.noop)] + operations = [ + migrations.RunSQL( + "DROP INDEX IF EXISTS pgsearch_embedding_hnsw;", + reverse_sql=migrations.RunSQL.noop, + ), + migrations.RunPython(backfill, migrations.RunPython.noop), + ] diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0006_search_projection_indexes.py index 4118d95b..e3497c4e 100644 --- a/radis/pgsearch/migrations/0006_search_projection_indexes.py +++ b/radis/pgsearch/migrations/0006_search_projection_indexes.py @@ -1,9 +1,42 @@ # Generated by Django 6.0.7 on 2026-08-30 20:20 +import re + import django.contrib.postgres.indexes +from django.conf import settings from django.db import migrations, models +def rebuild_hnsw(apps, schema_editor): + """Recreate the embedding index that 0005 dropped, from the stored vectors. + + One bulk build assembles the graph in maintenance_work_mem instead of the + row-at-a-time inserts the backfill would otherwise have paid (7m36s versus + ~68h measured at 1.7M vectors). Guarded so installs where the index still + exists -- and re-runs -- skip it. The definition must match the HnswIndex + in ReportSearchIndex.Meta. + """ + memory = settings.PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM + if not re.fullmatch(r"\d+\s*(kB|MB|GB|TB)", memory): + raise ValueError( + f"PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM is not a PostgreSQL " + f"memory value: {memory!r}" + ) + workers = int(settings.PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS) + with schema_editor.connection.cursor() as cursor: + cursor.execute("SELECT 1 FROM pg_indexes WHERE indexname = 'pgsearch_embedding_hnsw'") + if cursor.fetchone(): + return + # SET LOCAL: scoped to this migration's transaction, so nothing leaks + # to later migrations sharing the connection. + cursor.execute(f"SET LOCAL maintenance_work_mem = '{memory}'") + cursor.execute(f"SET LOCAL max_parallel_maintenance_workers = {workers}") + cursor.execute( + "CREATE INDEX pgsearch_embedding_hnsw ON pgsearch_reportsearchindex " + "USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)" + ) + + class Migration(migrations.Migration): dependencies = [ @@ -28,6 +61,9 @@ class Migration(migrations.Migration): "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", ), + # Reverse is a no-op on purpose: unapplying 0006 then 0005 must leave + # the index present, which is the state before 0005 dropped it. + migrations.RunPython(rebuild_hnsw, migrations.RunPython.noop), migrations.RunSQL( # Required, not hygiene: the projection columns and the indexes # above carry no statistics, and the single-table query shape diff --git a/radis/settings/base.py b/radis/settings/base.py index ab991a44..d3cfd4f5 100644 --- a/radis/settings/base.py +++ b/radis/settings/base.py @@ -534,6 +534,18 @@ def _resolve_embeddings_model() -> ModelSpec | None: HYBRID_FTS_MAX_RESULTS = 10_000 HYBRID_RRF_K = 60 +# Session knobs for the HNSW rebuild in pgsearch migration 0006 (the projection +# backfill drops the embedding index and 0006 recreates it from the stored +# vectors). A bulk build assembles the graph in maintenance_work_mem; on a large +# embedded corpus raise both so it fits and parallelizes (a 1.7M-vector corpus +# built in 7m36s with 16GB and 7 workers). FTS-only installs never notice them. +PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM = env.str( + "PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM", default="2GB" +) +PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS = env.int( + "PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS", default=2 +) + # Chat CHAT_GENERATE_TITLE_SYSTEM_PROMPT = """ Summarize the following conversation in $num_words words or less and in the same language as From 4f5b6c644d234f4e9790e7c556b11e73ac941d3e Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 22:24:42 +0200 Subject: [PATCH 44/50] Make the search index admin read-only Every ReportSearchIndex field is machine-maintained (signals, triggers, embedding tasks), and group_ids is access-control data: an editable change form could grant or hide report visibility out of sync with the membership tables, and even saving an untouched form would clobber trigger-maintained values. The form is now inspection-only and rows can no longer be added by hand, matching the existing delete block. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- radis/pgsearch/admin.py | 25 +++++++++++++++++++++++++ radis/pgsearch/tests/test_admin.py | 23 +++++++++++++++++++++++ 2 files changed, 48 insertions(+) diff --git a/radis/pgsearch/admin.py b/radis/pgsearch/admin.py index a3614429..47a8f2d8 100644 --- a/radis/pgsearch/admin.py +++ b/radis/pgsearch/admin.py @@ -36,6 +36,31 @@ class ReportSearchIndexAdmin(admin.ModelAdmin): # load. raw_id_fields swaps that for a text input + lookup popup so # opening a single row doesn't enumerate the whole table. raw_id_fields = ("report",) + # Every field is machine-maintained (signals, the 0004/0007 triggers, the + # embedding tasks), so the change form is inspection-only. group_ids + # especially is access-control data: an editable form would grant or hide + # report visibility out of sync with the membership tables, and even + # saving an untouched form would clobber trigger-maintained values. + readonly_fields = ( + "report", + "search_vector", + "embedding", + "group_ids", + "modality_codes", + "language_code", + "patient_sex", + "patient_age", + "patient_id", + "study_datetime", + "study_description", + "report_created_at", + "report_updated_at", + ) + + def has_add_permission(self, request): + # Rows are created by the post_save signal on Report; a hand-made row + # would just be an orphan the next reindex fights with. + return False def has_delete_permission(self, request, obj=None): # RSI rows are managed by the post_save signal on Report — deleting diff --git a/radis/pgsearch/tests/test_admin.py b/radis/pgsearch/tests/test_admin.py index 5d277307..8001aad3 100644 --- a/radis/pgsearch/tests/test_admin.py +++ b/radis/pgsearch/tests/test_admin.py @@ -490,3 +490,26 @@ def test_enqueue_pending_embeddings_warns_when_model_not_configured(settings): call = admin_instance.message_user.call_args assert "EMBEDDINGS_MODEL" in call.args[1] assert call.kwargs.get("level") == messages.WARNING + + +def test_change_form_is_read_only(admin_client): + report = ReportFactory.create() + rsi = ReportSearchIndex.objects.get(report=report) + + response = admin_client.get( + reverse("admin:pgsearch_reportsearchindex_change", args=[rsi.pk]) + ) + + assert response.status_code == 200 + content = response.content.decode() + # readonly_fields render as text, not form inputs; group_ids is the one + # that must never be editable (access-control data). + assert 'name="group_ids"' not in content + assert 'name="language_code"' not in content + assert 'name="search_vector"' not in content + + +def test_rows_cannot_be_added_in_the_admin(admin_client): + response = admin_client.get(reverse("admin:pgsearch_reportsearchindex_add")) + + assert response.status_code == 403 From 24a1b607f865bc91b7facdd3fdef8bd469765bd2 Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 22:24:50 +0200 Subject: [PATCH 45/50] Sync the projection through language and modality code renames The mirrors store code strings, but no trigger watched the code columns: renaming a language silently stripped its whole corpus from search while every index row kept mirroring the old code. Row-level AFTER UPDATE OF code triggers close both writers -- row-level deliberately, since PostgreSQL forbids transition tables on triggers with column lists and a rename statement touches one dimension row. Also pins the 0005/0006 HNSW drop-and-rebuild round trip with an index-existence test. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- .../migrations/0007_code_rename_triggers.py | 93 +++++++++++++++++++ .../tests/test_check_search_projection.py | 21 ++++- .../pgsearch/tests/test_search_projection.py | 31 ++++++- 3 files changed, 141 insertions(+), 4 deletions(-) create mode 100644 radis/pgsearch/migrations/0007_code_rename_triggers.py diff --git a/radis/pgsearch/migrations/0007_code_rename_triggers.py b/radis/pgsearch/migrations/0007_code_rename_triggers.py new file mode 100644 index 00000000..c9d688b2 --- /dev/null +++ b/radis/pgsearch/migrations/0007_code_rename_triggers.py @@ -0,0 +1,93 @@ +"""Close two projection writers the 0004 triggers do not see: code renames. + +0004 syncs membership changes and report-row updates, but the mirrors store +the *code strings* of language and modalities, and nothing watched the code +columns themselves. Renaming a language silently strips its whole corpus from +search: every index row keeps mirroring the old code, so the language filter +and the per-config tsquery match both stop matching, while the drift checker +is the only thing that would notice. 0004's docstring lists three escaping +writers; these were the unlisted fourth and fifth. + +Row-level rather than 0004's statement-level shape, deliberately: PostgreSQL +does not allow transition tables on triggers with column lists ("transition +tables cannot be specified for triggers with column lists"), and unlike the +bulk membership writes 0004 handles, a rename statement touches one dimension +row -- FOR EACH ROW with a WHEN guard fires exactly once per actual code +change and costs nothing on these tiny tables. The modality function repeats +the array_agg aggregation of utils/projection.PROJECTION_UPDATE_SQL, the 0005 +backfill, 0004's membership functions and check_search_projection.DRIFT_SQL -- +keep them all in sync. + +Cost note: renaming a language rewrites the projection row of every report in +that language, and on an embedded corpus each rewrite also inserts into the +HNSW index (~143 ms/row measured). Renames are assumed rare and small; a +corpus-wide rename belongs in a maintenance window. +""" + +from django.db import migrations + +SYNC_LANGUAGE_CODE = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_language_code() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET language_code = NEW.code + FROM reports_report r + WHERE r.language_id = NEW.id + AND rsi.report_id = r.id + AND rsi.language_code IS DISTINCT FROM NEW.code; + RETURN NULL; +END $$; +""" + +SYNC_MODALITY_CODE_RENAME = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_modality_code_rename() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET modality_codes = full_agg.codes + FROM ( + SELECT rm.report_id, + array_agg(m.code ORDER BY m.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality m ON m.id = rm.modality_id + WHERE rm.report_id IN ( + SELECT rm2.report_id FROM reports_report_modalities rm2 + WHERE rm2.modality_id = NEW.id) + GROUP BY rm.report_id + ) full_agg + WHERE rsi.report_id = full_agg.report_id + AND rsi.modality_codes IS DISTINCT FROM full_agg.codes; + RETURN NULL; +END $$; +""" + +CREATE_TRIGGERS = """ +CREATE TRIGGER pgsearch_language_code_upd +AFTER UPDATE OF code ON reports_language +FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) +EXECUTE FUNCTION pgsearch_sync_language_code(); + +CREATE TRIGGER pgsearch_modality_code_upd +AFTER UPDATE OF code ON reports_modality +FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) +EXECUTE FUNCTION pgsearch_sync_modality_code_rename(); +""" + +DROP_TRIGGERS = """ +DROP TRIGGER IF EXISTS pgsearch_language_code_upd ON reports_language; +DROP TRIGGER IF EXISTS pgsearch_modality_code_upd ON reports_modality; +DROP FUNCTION IF EXISTS pgsearch_sync_language_code(); +DROP FUNCTION IF EXISTS pgsearch_sync_modality_code_rename(); +""" + + +class Migration(migrations.Migration): + dependencies = [("pgsearch", "0006_search_projection_indexes")] + + operations = [ + migrations.RunSQL( + SYNC_LANGUAGE_CODE + SYNC_MODALITY_CODE_RENAME + CREATE_TRIGGERS, + reverse_sql=DROP_TRIGGERS, + ), + ] diff --git a/radis/pgsearch/tests/test_check_search_projection.py b/radis/pgsearch/tests/test_check_search_projection.py index 311d4ef8..c29fa929 100644 --- a/radis/pgsearch/tests/test_check_search_projection.py +++ b/radis/pgsearch/tests/test_check_search_projection.py @@ -1,6 +1,6 @@ """The projection duplicates access-control data, so operators need a way to -prove it still matches its sources -- after a restore, a bulk import, or a -Language.code rename, which no trigger covers.""" +prove it still matches its sources -- after a restore, a bulk import, or any +writer the triggers still miss (0004's docstring lists them).""" from io import StringIO @@ -10,7 +10,8 @@ from django.core.management.base import CommandError from radis.pgsearch.models import ReportSearchIndex -from radis.reports.factories import LanguageFactory, ReportFactory +from radis.reports.factories import LanguageFactory, ModalityFactory, ReportFactory +from radis.reports.models import Language, Modality pytestmark = pytest.mark.django_db @@ -48,3 +49,17 @@ def test_detects_drifted_group_ids(): with pytest.raises(CommandError, match="group_ids"): call_command("check_search_projection") + + +def test_reports_no_drift_after_code_renames(): + """The 0007 triggers keep the mirrors current through code renames, so the + checker must come back clean afterwards.""" + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language, modalities=[]) + modality = ModalityFactory.create(code="CT") + report.modalities.add(modality) + + Language.objects.filter(pk=language.pk).update(code="en-gb") + Modality.objects.filter(pk=modality.pk).update(code="CTA") + + call_command("check_search_projection", stdout=StringIO()) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 1c738b01..21e39833 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -17,7 +17,7 @@ from radis.pgsearch.utils.indexing import bulk_upsert_report_search_indexes from radis.pgsearch.utils.projection import sync_projection from radis.reports.factories import LanguageFactory, ModalityFactory, ReportFactory -from radis.reports.models import Report +from radis.reports.models import Language, Modality, Report pytestmark = pytest.mark.django_db @@ -430,3 +430,32 @@ def test_projection_indexes_exist(): assert "pgsearch_group_ids_gin" in names assert "pgsearch_report_updated_at_idx" in names + + +def test_language_code_rename_updates_the_projection(): + language = LanguageFactory.create(code="en") + report = ReportFactory.create(language=language) + + Language.objects.filter(pk=language.pk).update(code="en-gb") + + index = ReportSearchIndex.objects.get(report=report) + assert index.language_code == "en-gb" + + +def test_modality_code_rename_updates_the_projection(): + report = ReportFactory.create(language=LanguageFactory.create(code="en"), modalities=[]) + modality = ModalityFactory.create(code="CT") + report.modalities.add(modality) + + Modality.objects.filter(pk=modality.pk).update(code="CTA") + + index = ReportSearchIndex.objects.get(report=report) + assert index.modality_codes == ["CTA"] + + +def test_hnsw_index_survives_the_backfill_migrations(): + """0005 drops the embedding index and 0006 rebuilds it; the test database + runs both, so the index existing here proves the whole round trip.""" + with connection.cursor() as cursor: + cursor.execute("SELECT 1 FROM pg_indexes WHERE indexname = 'pgsearch_embedding_hnsw'") + assert cursor.fetchone() is not None From ef11d3e4e09143d1783a97a0fa4e220c96f70969 Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 23:01:51 +0200 Subject: [PATCH 46/50] Tune the HNSW rebuild through server GUCs instead of app settings Two Django settings for a migration that runs once were the wrong shape. 0006's rebuild is now a plain CREATE INDEX IF NOT EXISTS that honors the server's maintenance_work_mem and max_parallel_maintenance_workers, exposed as compose GUC knobs (POSTGRES_MAINTENANCE_WORK_MEM, POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS) beside their existing siblings, defaulting to PostgreSQL's own defaults. The /dev/shm tmpfs stays: an undersized shared memory segment is a standing footgun for any PostgreSQL parallelism, not just this build, and a tmpfs size is a cap, not a reservation. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- AGENTS.md | 23 ++++---- docker-compose.base.yml | 13 +++-- ...8-29-fts-query-shape-performance-design.md | 8 +-- docs/user-docs/admin-guide.md | 13 ++--- .../0006_search_projection_indexes.py | 52 ++++++------------- radis/settings/base.py | 12 ----- 6 files changed, 48 insertions(+), 73 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6d731b15..631a865d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -128,13 +128,6 @@ Hybrid search embeddings (`radis.pgsearch`): swap — cached query vectors otherwise keep serving stale results for up to this long - `EMBEDDINGS_BATCH_SIZE`, `EMBEDDINGS_SUBJOB_SIZE`, `EMBEDDINGS_WORKER_CONCURRENCY`: Throughput tuning -- `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` (default `2GB`), - `PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS` (default `2`): session knobs for the - HNSW rebuild in pgsearch migration 0006 (the projection backfill drops the - embedding index and 0006 recreates it from the stored vectors). Raise on a - large embedded corpus so the graph builds in memory; irrelevant for FTS-only - installs. Size `POSTGRES_SHM_SIZE_BYTES` (below) at least as large as the - memory value Auto-labeling (`radis.labels`): @@ -163,11 +156,19 @@ only to override the compose file's default): `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` on a larger host - `POSTGRES_SHARED_BUFFERS`: PostgreSQL's shared memory buffer (default `128MB`, PostgreSQL's own default) +- `POSTGRES_MAINTENANCE_WORK_MEM`: memory for VACUUM and index builds (default + `64MB`, PostgreSQL's own default). Raise for the migration window on a large + embedded archive so the pgsearch 0006 HNSW rebuild assembles its graph in + memory (a 1.7M-vector corpus built in 7m36s with `16GB` and 7 workers), and + for the manual rebuilds in the admin guide's compaction section +- `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`: parallel workers per index build + (default `2`, PostgreSQL's own default); also capped by + `POSTGRES_MAX_PARALLEL_WORKERS` - `POSTGRES_SHM_SIZE_BYTES`: tmpfs size of the container's `/dev/shm` in bytes - (default `1073741824` = 1 GiB; Docker's own default of 64 MB is too small for - parallel queries). Parallel maintenance pre-allocates its whole - `maintenance_work_mem` budget here — for the pgsearch 0006 HNSW rebuild set - it at least as large as `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` + (default `1073741824` = 1 GiB — a cap, not a reservation; Docker's own 64 MB + default is too small for parallel queries). Parallel maintenance + pre-allocates its whole `maintenance_work_mem` budget here, so keep it at + least as large as `POSTGRES_MAINTENANCE_WORK_MEM` ## Code Standards diff --git a/docker-compose.base.yml b/docker-compose.base.yml index 9742cec8..1249ead3 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -50,13 +50,18 @@ services: - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} - -c - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} + - -c + - maintenance_work_mem=${POSTGRES_MAINTENANCE_WORK_MEM:-64MB} + - -c + - max_parallel_maintenance_workers=${POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS:-2} volumes: - postgres_data:/var/lib/postgresql/data # PostgreSQL's parallel workers exchange data through dynamic shared - # memory in /dev/shm, and Docker's default segment is 64 MB. Parallel - # maintenance (the HNSW rebuild in pgsearch migration 0006) pre-allocates - # its whole budget there, so size this at least as large as - # PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM. Bytes, because + # memory in /dev/shm, and Docker's default segment is only 64 MB. + # Parallel maintenance (index builds, e.g. the HNSW rebuild in pgsearch + # migration 0006) pre-allocates its whole maintenance_work_mem budget + # there, so size this at least as large as POSTGRES_MAINTENANCE_WORK_MEM. + # A tmpfs size is a cap, not a reservation. Bytes, because # `docker stack deploy` rejects unit suffixes here (default: 1 GiB). - type: tmpfs target: /dev/shm diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 1f98554a..0316280f 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -338,10 +338,10 @@ embedded corpus every rewritten row is a non-HOT update that would also insert into the HNSW index (~143 ms and ~2,160 buffers per embedded row measured on staging: ~68 hours for 1.7M vectors). The backfill therefore drops the HNSW index first and 0006 rebuilds it from the stored vectors in one bulk build -(7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers), tunable -via `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` / -`PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS`, with `/dev/shm` sized to match -(`POSTGRES_SHM_SIZE_BYTES` in the compose files). +(7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers). The +build honors the server settings, tunable through the compose GUC knobs +`POSTGRES_MAINTENANCE_WORK_MEM` / `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`, +with `/dev/shm` sized to match (`POSTGRES_SHM_SIZE_BYTES`). The backfill rewrites every row: the tsvectors average 1220 bytes and nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index be46e16b..9c6d5d54 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -37,12 +37,13 @@ larger archive. On an archive **with embeddings** the migration first drops the vector index and rebuilds it afterwards from the stored vectors (nothing is re-embedded) -- without that, every rewritten row would also pay an insert into the vector index (~143 ms per embedded row: days, not minutes). For a large -embedded archive raise `PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM` and -`PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS` so the rebuild runs in memory (a -1.7M-vector archive rebuilt in under 8 minutes with 16GB and 7 workers), and -size `POSTGRES_SHM_SIZE_BYTES` at least as large as that memory value -- the -parallel build pre-allocates it in `/dev/shm`, and other parallel queries can -fail with "could not resize shared memory segment" while it runs. Plan the +embedded archive raise `POSTGRES_MAINTENANCE_WORK_MEM` and +`POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS` in `.env` for the migration window +so the rebuild runs in memory (a 1.7M-vector archive rebuilt in under 8 +minutes with `16GB` and 7 workers), and size `POSTGRES_SHM_SIZE_BYTES` at +least as large as that memory value -- the parallel build pre-allocates it in +`/dev/shm`, and other parallel queries can fail with "could not resize shared +memory segment" while it runs. Plan the maintenance window around all of that, and do not shorten `WAIT_INIT_TIMEOUT` (default one hour, see the compose files) below the expected migration time -- the services that wait on `init` exit when that timeout expires. diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0006_search_projection_indexes.py index e3497c4e..341c8c7a 100644 --- a/radis/pgsearch/migrations/0006_search_projection_indexes.py +++ b/radis/pgsearch/migrations/0006_search_projection_indexes.py @@ -1,42 +1,9 @@ # Generated by Django 6.0.7 on 2026-08-30 20:20 -import re - import django.contrib.postgres.indexes -from django.conf import settings from django.db import migrations, models -def rebuild_hnsw(apps, schema_editor): - """Recreate the embedding index that 0005 dropped, from the stored vectors. - - One bulk build assembles the graph in maintenance_work_mem instead of the - row-at-a-time inserts the backfill would otherwise have paid (7m36s versus - ~68h measured at 1.7M vectors). Guarded so installs where the index still - exists -- and re-runs -- skip it. The definition must match the HnswIndex - in ReportSearchIndex.Meta. - """ - memory = settings.PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM - if not re.fullmatch(r"\d+\s*(kB|MB|GB|TB)", memory): - raise ValueError( - f"PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM is not a PostgreSQL " - f"memory value: {memory!r}" - ) - workers = int(settings.PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS) - with schema_editor.connection.cursor() as cursor: - cursor.execute("SELECT 1 FROM pg_indexes WHERE indexname = 'pgsearch_embedding_hnsw'") - if cursor.fetchone(): - return - # SET LOCAL: scoped to this migration's transaction, so nothing leaks - # to later migrations sharing the connection. - cursor.execute(f"SET LOCAL maintenance_work_mem = '{memory}'") - cursor.execute(f"SET LOCAL max_parallel_maintenance_workers = {workers}") - cursor.execute( - "CREATE INDEX pgsearch_embedding_hnsw ON pgsearch_reportsearchindex " - "USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)" - ) - - class Migration(migrations.Migration): dependencies = [ @@ -61,9 +28,22 @@ class Migration(migrations.Migration): "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", ), - # Reverse is a no-op on purpose: unapplying 0006 then 0005 must leave - # the index present, which is the state before 0005 dropped it. - migrations.RunPython(rebuild_hnsw, migrations.RunPython.noop), + # Recreate the embedding index 0005 dropped, from the stored vectors: + # one bulk build instead of the per-row inserts the backfill would + # otherwise have paid (7m36s versus ~68h measured at 1.7M vectors). + # The build honors the server's maintenance_work_mem and + # max_parallel_maintenance_workers -- raise them for a large embedded + # corpus (POSTGRES_* knobs in the compose files). IF NOT EXISTS keeps + # re-runs and installs that never dropped it safe; the definition must + # match the HnswIndex in ReportSearchIndex.Meta. Reverse is a no-op on + # purpose: unapplying 0006 then 0005 must leave the index present, + # which is the state before 0005 dropped it. + migrations.RunSQL( + "CREATE INDEX IF NOT EXISTS pgsearch_embedding_hnsw " + "ON pgsearch_reportsearchindex " + "USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64);", + reverse_sql=migrations.RunSQL.noop, + ), migrations.RunSQL( # Required, not hygiene: the projection columns and the indexes # above carry no statistics, and the single-table query shape diff --git a/radis/settings/base.py b/radis/settings/base.py index d3cfd4f5..ab991a44 100644 --- a/radis/settings/base.py +++ b/radis/settings/base.py @@ -534,18 +534,6 @@ def _resolve_embeddings_model() -> ModelSpec | None: HYBRID_FTS_MAX_RESULTS = 10_000 HYBRID_RRF_K = 60 -# Session knobs for the HNSW rebuild in pgsearch migration 0006 (the projection -# backfill drops the embedding index and 0006 recreates it from the stored -# vectors). A bulk build assembles the graph in maintenance_work_mem; on a large -# embedded corpus raise both so it fits and parallelizes (a 1.7M-vector corpus -# built in 7m36s with 16GB and 7 workers). FTS-only installs never notice them. -PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM = env.str( - "PGSEARCH_HNSW_REBUILD_MAINTENANCE_WORK_MEM", default="2GB" -) -PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS = env.int( - "PGSEARCH_HNSW_REBUILD_PARALLEL_WORKERS", default=2 -) - # Chat CHAT_GENERATE_TITLE_SYSTEM_PROMPT = """ Summarize the following conversation in $num_words words or less and in the same language as From c82d486e407769f70f982b8753e9a4e9614947c1 Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 23:20:13 +0200 Subject: [PATCH 47/50] Leave postgres maintenance tuning to the deployment Neither the GUC pass-throughs nor a /dev/shm tmpfs belong in the repo: server tuning is the deployer's call. maintenance_work_mem and the parallel-worker count are reachable through ALTER SYSTEM (persisted in the data volume, reloadable without restart), and the container's /dev/shm -- which parallel maintenance pre-allocates and which cannot be changed from inside PostgreSQL -- is documented as a deploy-side compose override or service mount instead. The admin guide carries the full recipe for the large-archive migration window. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- AGENTS.md | 22 +++++++--------- docker-compose.base.yml | 15 ----------- ...8-29-fts-query-shape-performance-design.md | 7 +++--- docs/user-docs/admin-guide.md | 25 +++++++++++++------ 4 files changed, 31 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 631a865d..f5cfdbe7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -156,19 +156,15 @@ only to override the compose file's default): `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` on a larger host - `POSTGRES_SHARED_BUFFERS`: PostgreSQL's shared memory buffer (default `128MB`, PostgreSQL's own default) -- `POSTGRES_MAINTENANCE_WORK_MEM`: memory for VACUUM and index builds (default - `64MB`, PostgreSQL's own default). Raise for the migration window on a large - embedded archive so the pgsearch 0006 HNSW rebuild assembles its graph in - memory (a 1.7M-vector corpus built in 7m36s with `16GB` and 7 workers), and - for the manual rebuilds in the admin guide's compaction section -- `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`: parallel workers per index build - (default `2`, PostgreSQL's own default); also capped by - `POSTGRES_MAX_PARALLEL_WORKERS` -- `POSTGRES_SHM_SIZE_BYTES`: tmpfs size of the container's `/dev/shm` in bytes - (default `1073741824` = 1 GiB — a cap, not a reservation; Docker's own 64 MB - default is too small for parallel queries). Parallel maintenance - pre-allocates its whole `maintenance_work_mem` budget here, so keep it at - least as large as `POSTGRES_MAINTENANCE_WORK_MEM` + +Further server tuning is a deployment concern, not repo configuration. Two +that large embedded archives will hit (see the admin guide): raising +`maintenance_work_mem` for index rebuilds via +`ALTER SYSTEM SET ... ; SELECT pg_reload_conf();` (persists in the data +volume, no restart), and the container's `/dev/shm` — Docker's 64 MB default +is too small for parallel maintenance, and it cannot be changed from inside +PostgreSQL, so mount a larger tmpfs in your own compose override or via +`docker service update --mount-add`. ## Code Standards diff --git a/docker-compose.base.yml b/docker-compose.base.yml index 1249ead3..8b196030 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -50,23 +50,8 @@ services: - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} - -c - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} - - -c - - maintenance_work_mem=${POSTGRES_MAINTENANCE_WORK_MEM:-64MB} - - -c - - max_parallel_maintenance_workers=${POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS:-2} volumes: - postgres_data:/var/lib/postgresql/data - # PostgreSQL's parallel workers exchange data through dynamic shared - # memory in /dev/shm, and Docker's default segment is only 64 MB. - # Parallel maintenance (index builds, e.g. the HNSW rebuild in pgsearch - # migration 0006) pre-allocates its whole maintenance_work_mem budget - # there, so size this at least as large as POSTGRES_MAINTENANCE_WORK_MEM. - # A tmpfs size is a cap, not a reservation. Bytes, because - # `docker stack deploy` rejects unit suffixes here (default: 1 GiB). - - type: tmpfs - target: /dev/shm - tmpfs: - size: ${POSTGRES_SHM_SIZE_BYTES:-1073741824} volumes: postgres_data: diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 0316280f..868a6392 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -339,9 +339,10 @@ into the HNSW index (~143 ms and ~2,160 buffers per embedded row measured on staging: ~68 hours for 1.7M vectors). The backfill therefore drops the HNSW index first and 0006 rebuilds it from the stored vectors in one bulk build (7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers). The -build honors the server settings, tunable through the compose GUC knobs -`POSTGRES_MAINTENANCE_WORK_MEM` / `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`, -with `/dev/shm` sized to match (`POSTGRES_SHM_SIZE_BYTES`). +build honors the server's own maintenance settings -- raising them for the +window, like sizing the container's `/dev/shm` to match (Docker's 64 MB +default cannot be fixed from inside PostgreSQL), is a deployment concern, not +repo configuration; the admin guide carries the recipe. The backfill rewrites every row: the tsvectors average 1220 bytes and nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 9c6d5d54..95149ba8 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -37,13 +37,24 @@ larger archive. On an archive **with embeddings** the migration first drops the vector index and rebuilds it afterwards from the stored vectors (nothing is re-embedded) -- without that, every rewritten row would also pay an insert into the vector index (~143 ms per embedded row: days, not minutes). For a large -embedded archive raise `POSTGRES_MAINTENANCE_WORK_MEM` and -`POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS` in `.env` for the migration window -so the rebuild runs in memory (a 1.7M-vector archive rebuilt in under 8 -minutes with `16GB` and 7 workers), and size `POSTGRES_SHM_SIZE_BYTES` at -least as large as that memory value -- the parallel build pre-allocates it in -`/dev/shm`, and other parallel queries can fail with "could not resize shared -memory segment" while it runs. Plan the +embedded archive raise the server's maintenance settings for the migration +window so the rebuild runs in memory (a 1.7M-vector archive rebuilt in under +8 minutes with 16GB and 7 workers): + +```sql +ALTER SYSTEM SET maintenance_work_mem = '16GB'; +ALTER SYSTEM SET max_parallel_maintenance_workers = 7; +SELECT pg_reload_conf(); +-- afterwards: ALTER SYSTEM RESET maintenance_work_mem; (and the workers), then reload again +``` + +and give the postgres container a `/dev/shm` at least as large as that memory +value -- the parallel build pre-allocates its whole budget there, Docker's +default is only 64 MB, and this cannot be changed from inside PostgreSQL. Use +your own compose override or, on Swarm, +`docker service update --mount-add type=tmpfs,dst=/dev/shm,tmpfs-size= `. +Other parallel queries can fail with "could not resize shared memory segment" +while the build runs. Plan the maintenance window around all of that, and do not shorten `WAIT_INIT_TIMEOUT` (default one hour, see the compose files) below the expected migration time -- the services that wait on `init` exit when that timeout expires. From e2a7b38163c71427e82e7170b580be712235cb4a Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 23:20:57 +0200 Subject: [PATCH 48/50] Revert "Leave postgres maintenance tuning to the deployment" The compose file already carries maintainer-established postgres tuning variables (max_parallel_workers_per_gather and friends), so the maintenance knobs and the /dev/shm tmpfs follow the house pattern after all. This restores POSTGRES_MAINTENANCE_WORK_MEM, POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS and POSTGRES_SHM_SIZE_BYTES with their documentation. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- AGENTS.md | 22 +++++++++------- docker-compose.base.yml | 15 +++++++++++ ...8-29-fts-query-shape-performance-design.md | 7 +++--- docs/user-docs/admin-guide.md | 25 ++++++------------- 4 files changed, 38 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f5cfdbe7..631a865d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -156,15 +156,19 @@ only to override the compose file's default): `POSTGRES_MAX_PARALLEL_WORKERS_PER_GATHER` on a larger host - `POSTGRES_SHARED_BUFFERS`: PostgreSQL's shared memory buffer (default `128MB`, PostgreSQL's own default) - -Further server tuning is a deployment concern, not repo configuration. Two -that large embedded archives will hit (see the admin guide): raising -`maintenance_work_mem` for index rebuilds via -`ALTER SYSTEM SET ... ; SELECT pg_reload_conf();` (persists in the data -volume, no restart), and the container's `/dev/shm` — Docker's 64 MB default -is too small for parallel maintenance, and it cannot be changed from inside -PostgreSQL, so mount a larger tmpfs in your own compose override or via -`docker service update --mount-add`. +- `POSTGRES_MAINTENANCE_WORK_MEM`: memory for VACUUM and index builds (default + `64MB`, PostgreSQL's own default). Raise for the migration window on a large + embedded archive so the pgsearch 0006 HNSW rebuild assembles its graph in + memory (a 1.7M-vector corpus built in 7m36s with `16GB` and 7 workers), and + for the manual rebuilds in the admin guide's compaction section +- `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`: parallel workers per index build + (default `2`, PostgreSQL's own default); also capped by + `POSTGRES_MAX_PARALLEL_WORKERS` +- `POSTGRES_SHM_SIZE_BYTES`: tmpfs size of the container's `/dev/shm` in bytes + (default `1073741824` = 1 GiB — a cap, not a reservation; Docker's own 64 MB + default is too small for parallel queries). Parallel maintenance + pre-allocates its whole `maintenance_work_mem` budget here, so keep it at + least as large as `POSTGRES_MAINTENANCE_WORK_MEM` ## Code Standards diff --git a/docker-compose.base.yml b/docker-compose.base.yml index 8b196030..1249ead3 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -50,8 +50,23 @@ services: - max_worker_processes=${POSTGRES_MAX_WORKER_PROCESSES:-8} - -c - shared_buffers=${POSTGRES_SHARED_BUFFERS:-128MB} + - -c + - maintenance_work_mem=${POSTGRES_MAINTENANCE_WORK_MEM:-64MB} + - -c + - max_parallel_maintenance_workers=${POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS:-2} volumes: - postgres_data:/var/lib/postgresql/data + # PostgreSQL's parallel workers exchange data through dynamic shared + # memory in /dev/shm, and Docker's default segment is only 64 MB. + # Parallel maintenance (index builds, e.g. the HNSW rebuild in pgsearch + # migration 0006) pre-allocates its whole maintenance_work_mem budget + # there, so size this at least as large as POSTGRES_MAINTENANCE_WORK_MEM. + # A tmpfs size is a cap, not a reservation. Bytes, because + # `docker stack deploy` rejects unit suffixes here (default: 1 GiB). + - type: tmpfs + target: /dev/shm + tmpfs: + size: ${POSTGRES_SHM_SIZE_BYTES:-1073741824} volumes: postgres_data: diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 868a6392..0316280f 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -339,10 +339,9 @@ into the HNSW index (~143 ms and ~2,160 buffers per embedded row measured on staging: ~68 hours for 1.7M vectors). The backfill therefore drops the HNSW index first and 0006 rebuilds it from the stored vectors in one bulk build (7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers). The -build honors the server's own maintenance settings -- raising them for the -window, like sizing the container's `/dev/shm` to match (Docker's 64 MB -default cannot be fixed from inside PostgreSQL), is a deployment concern, not -repo configuration; the admin guide carries the recipe. +build honors the server settings, tunable through the compose GUC knobs +`POSTGRES_MAINTENANCE_WORK_MEM` / `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`, +with `/dev/shm` sized to match (`POSTGRES_SHM_SIZE_BYTES`). The backfill rewrites every row: the tsvectors average 1220 bytes and nothing is TOASTed (`toast_heap` is 0 bytes), so the whole row is rewritten, not just the diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md index 95149ba8..9c6d5d54 100644 --- a/docs/user-docs/admin-guide.md +++ b/docs/user-docs/admin-guide.md @@ -37,24 +37,13 @@ larger archive. On an archive **with embeddings** the migration first drops the vector index and rebuilds it afterwards from the stored vectors (nothing is re-embedded) -- without that, every rewritten row would also pay an insert into the vector index (~143 ms per embedded row: days, not minutes). For a large -embedded archive raise the server's maintenance settings for the migration -window so the rebuild runs in memory (a 1.7M-vector archive rebuilt in under -8 minutes with 16GB and 7 workers): - -```sql -ALTER SYSTEM SET maintenance_work_mem = '16GB'; -ALTER SYSTEM SET max_parallel_maintenance_workers = 7; -SELECT pg_reload_conf(); --- afterwards: ALTER SYSTEM RESET maintenance_work_mem; (and the workers), then reload again -``` - -and give the postgres container a `/dev/shm` at least as large as that memory -value -- the parallel build pre-allocates its whole budget there, Docker's -default is only 64 MB, and this cannot be changed from inside PostgreSQL. Use -your own compose override or, on Swarm, -`docker service update --mount-add type=tmpfs,dst=/dev/shm,tmpfs-size= `. -Other parallel queries can fail with "could not resize shared memory segment" -while the build runs. Plan the +embedded archive raise `POSTGRES_MAINTENANCE_WORK_MEM` and +`POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS` in `.env` for the migration window +so the rebuild runs in memory (a 1.7M-vector archive rebuilt in under 8 +minutes with `16GB` and 7 workers), and size `POSTGRES_SHM_SIZE_BYTES` at +least as large as that memory value -- the parallel build pre-allocates it in +`/dev/shm`, and other parallel queries can fail with "could not resize shared +memory segment" while it runs. Plan the maintenance window around all of that, and do not shorten `WAIT_INIT_TIMEOUT` (default one hour, see the compose files) below the expected migration time -- the services that wait on `init` exit when that timeout expires. From 2f0df7b9ebddf9f29f74fe89865dce8a21a803eb Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Wed, 23 Sep 2026 23:45:35 +0200 Subject: [PATCH 49/50] Squash the projection migrations to their transaction boundaries Five migrations become three, split exactly where the semantics force it: 0003 (columns and all seven triggers, atomic and instant), 0004 (the non-atomic chunked backfill with the HNSW drop), 0005 (indexes, HNSW rebuild and ANALYZE, atomic). The former column/trigger split and the separate code-rename-trigger migration carried no transaction boundary of their own, and creating the rename triggers before the backfill also covers renames during the backfill window. Databases that applied the old migration names need their django_migrations rows renamed once; fresh installs are unaffected (nothing upstream has applied any of these). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- AGENTS.md | 2 +- docker-compose.base.yml | 2 +- ...8-29-fts-query-shape-performance-design.md | 2 +- .../commands/check_search_projection.py | 4 +- ..._triggers.py => 0003_search_projection.py} | 137 +++++++++++++++++- .../0003_search_projection_columns.py | 64 -------- ....py => 0004_search_projection_backfill.py} | 8 +- ...s.py => 0005_search_projection_indexes.py} | 8 +- .../migrations/0007_code_rename_triggers.py | 93 ------------ radis/pgsearch/models.py | 2 +- radis/pgsearch/signals.py | 2 +- .../tests/test_check_search_projection.py | 2 +- .../pgsearch/tests/test_search_projection.py | 8 +- radis/pgsearch/utils/projection.py | 4 +- 14 files changed, 151 insertions(+), 187 deletions(-) rename radis/pgsearch/migrations/{0004_search_projection_triggers.py => 0003_search_projection.py} (53%) delete mode 100644 radis/pgsearch/migrations/0003_search_projection_columns.py rename radis/pgsearch/migrations/{0005_search_projection_backfill.py => 0004_search_projection_backfill.py} (94%) rename radis/pgsearch/migrations/{0006_search_projection_indexes.py => 0005_search_projection_indexes.py} (92%) delete mode 100644 radis/pgsearch/migrations/0007_code_rename_triggers.py diff --git a/AGENTS.md b/AGENTS.md index 631a865d..d8bdfad6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -158,7 +158,7 @@ only to override the compose file's default): PostgreSQL's own default) - `POSTGRES_MAINTENANCE_WORK_MEM`: memory for VACUUM and index builds (default `64MB`, PostgreSQL's own default). Raise for the migration window on a large - embedded archive so the pgsearch 0006 HNSW rebuild assembles its graph in + embedded archive so the pgsearch 0005 HNSW rebuild assembles its graph in memory (a 1.7M-vector corpus built in 7m36s with `16GB` and 7 workers), and for the manual rebuilds in the admin guide's compaction section - `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`: parallel workers per index build diff --git a/docker-compose.base.yml b/docker-compose.base.yml index 1249ead3..fa0845df 100644 --- a/docker-compose.base.yml +++ b/docker-compose.base.yml @@ -59,7 +59,7 @@ services: # PostgreSQL's parallel workers exchange data through dynamic shared # memory in /dev/shm, and Docker's default segment is only 64 MB. # Parallel maintenance (index builds, e.g. the HNSW rebuild in pgsearch - # migration 0006) pre-allocates its whole maintenance_work_mem budget + # migration 0005) pre-allocates its whole maintenance_work_mem budget # there, so size this at least as large as POSTGRES_MAINTENANCE_WORK_MEM. # A tmpfs size is a cap, not a reservation. Bytes, because # `docker stack deploy` rejects unit suffixes here (default: 1 GiB). diff --git a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md index 0316280f..dd384d5d 100644 --- a/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md +++ b/docs/superpowers/specs/2026-08-29-fts-query-shape-performance-design.md @@ -337,7 +337,7 @@ The ten-minute figure was measured on rigs built **without embeddings**. On an embedded corpus every rewritten row is a non-HOT update that would also insert into the HNSW index (~143 ms and ~2,160 buffers per embedded row measured on staging: ~68 hours for 1.7M vectors). The backfill therefore drops the HNSW -index first and 0006 rebuilds it from the stored vectors in one bulk build +index first and the indexes migration rebuilds it from the stored vectors in one bulk build (7m36s at 1.7M with 16GB maintenance_work_mem and 7 parallel workers). The build honors the server settings, tunable through the compose GUC knobs `POSTGRES_MAINTENANCE_WORK_MEM` / `POSTGRES_MAX_PARALLEL_MAINTENANCE_WORKERS`, diff --git a/radis/pgsearch/management/commands/check_search_projection.py b/radis/pgsearch/management/commands/check_search_projection.py index f0ac3d38..4017da04 100644 --- a/radis/pgsearch/management/commands/check_search_projection.py +++ b/radis/pgsearch/management/commands/check_search_projection.py @@ -3,8 +3,8 @@ from django.core.management.base import BaseCommand, CommandError from django.db import connection -# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill -# and the two trigger functions in migration 0004 -- four copies, keep them in +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0004 backfill +# and the trigger functions in migration 0003 -- four copies, keep them in # sync. This one is the detector: if it ever drifts in the same direction as a # writer, it silently agrees with the bug and reports a healthy projection. DRIFT_SQL = """ diff --git a/radis/pgsearch/migrations/0004_search_projection_triggers.py b/radis/pgsearch/migrations/0003_search_projection.py similarity index 53% rename from radis/pgsearch/migrations/0004_search_projection_triggers.py rename to radis/pgsearch/migrations/0003_search_projection.py index 292addcf..e0a4c476 100644 --- a/radis/pgsearch/migrations/0004_search_projection_triggers.py +++ b/radis/pgsearch/migrations/0003_search_projection.py @@ -1,10 +1,16 @@ -"""Statement-level triggers maintaining the ReportSearchIndex search projection. +"""The search projection: mirror columns on ReportSearchIndex plus the +triggers that maintain them. + +The ten AddField operations are metadata-only (nullable scalars, arrays with a +constant default), so this migration is instant regardless of corpus size; the +row rewrite that fills the columns is the separate non-atomic backfill in +0004, and the indexes over them follow in 0005. Triggers rather than Django signals because group_ids is access-control data: a signal fires only for ORM writes, missing management commands, the admin, and raw SQL. A trigger holds inside the same transaction as the write, and -covers every INSERT and DELETE on the two membership tables plus every UPDATE -of reports_report. +covers every INSERT and DELETE on the two membership tables, every UPDATE +of reports_report, and every rename of a language or modality code. Three writers still escape it, so a new code path that uses one of them has to keep the projection correct itself: @@ -26,15 +32,24 @@ Statement-level rather than row-level because the bulk-upsert endpoint writes membership with bulk_create (reports/api/viewsets.py:215,229), so one statement -can carry a whole batch. +can carry a whole batch. The two code-rename triggers are the exception, and +row-level deliberately: PostgreSQL does not allow transition tables on +triggers with column lists ("transition tables cannot be specified for +triggers with column lists"), and a rename statement touches one dimension +row -- FOR EACH ROW with a WHEN guard fires exactly once per actual code +change and costs nothing on these tiny tables. Renaming a language rewrites +the projection row of every report in that language, and on an embedded corpus +each rewrite also inserts into the HNSW index (~143 ms/row measured), so a +corpus-wide rename belongs in a maintenance window. AFTER, never BEFORE: PostgreSQL does not expose stored generated column values to BEFORE triggers, so patient_age would mirror as NULL. """ -from django.db import migrations +import django.contrib.postgres.fields +from django.db import migrations, models -# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0004 backfill # and check_search_projection.DRIFT_SQL -- four copies, keep them in sync. The # drift query is the one that detects divergence, so a change made here and not # there is a bug that reports itself as healthy. @@ -52,7 +67,7 @@ END $$; """ -# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0005 backfill +# Same aggregation as utils/projection.PROJECTION_UPDATE_SQL, the 0004 backfill # and check_search_projection.DRIFT_SQL -- four copies, keep them in sync. The # drift query is the one that detects divergence, so a change made here and not # there is a bug that reports itself as healthy. @@ -98,6 +113,42 @@ END $$; """ +SYNC_LANGUAGE_CODE = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_language_code() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET language_code = NEW.code + FROM reports_report r + WHERE r.language_id = NEW.id + AND rsi.report_id = r.id + AND rsi.language_code IS DISTINCT FROM NEW.code; + RETURN NULL; +END $$; +""" + +SYNC_MODALITY_CODE_RENAME = """ +CREATE OR REPLACE FUNCTION pgsearch_sync_modality_code_rename() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + UPDATE pgsearch_reportsearchindex rsi + SET modality_codes = full_agg.codes + FROM ( + SELECT rm.report_id, + array_agg(m.code ORDER BY m.code) AS codes + FROM reports_report_modalities rm + JOIN reports_modality m ON m.id = rm.modality_id + WHERE rm.report_id IN ( + SELECT rm2.report_id FROM reports_report_modalities rm2 + WHERE rm2.modality_id = NEW.id) + GROUP BY rm.report_id + ) full_agg + WHERE rsi.report_id = full_agg.report_id + AND rsi.modality_codes IS DISTINCT FROM full_agg.codes; + RETURN NULL; +END $$; +""" + CREATE_TRIGGERS = """ CREATE TRIGGER pgsearch_group_ids_ins AFTER INSERT ON reports_report_groups REFERENCING NEW TABLE AS changed @@ -118,6 +169,16 @@ CREATE TRIGGER pgsearch_report_fields_upd AFTER UPDATE ON reports_report REFERENCING NEW TABLE AS changed FOR EACH STATEMENT EXECUTE FUNCTION pgsearch_sync_report_fields(); + +CREATE TRIGGER pgsearch_language_code_upd +AFTER UPDATE OF code ON reports_language +FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) +EXECUTE FUNCTION pgsearch_sync_language_code(); + +CREATE TRIGGER pgsearch_modality_code_upd +AFTER UPDATE OF code ON reports_modality +FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) +EXECUTE FUNCTION pgsearch_sync_modality_code_rename(); """ DROP_TRIGGERS = """ @@ -126,21 +187,81 @@ DROP TRIGGER IF EXISTS pgsearch_modality_codes_ins ON reports_report_modalities; DROP TRIGGER IF EXISTS pgsearch_modality_codes_del ON reports_report_modalities; DROP TRIGGER IF EXISTS pgsearch_report_fields_upd ON reports_report; +DROP TRIGGER IF EXISTS pgsearch_language_code_upd ON reports_language; +DROP TRIGGER IF EXISTS pgsearch_modality_code_upd ON reports_modality; DROP FUNCTION IF EXISTS pgsearch_sync_group_ids(); DROP FUNCTION IF EXISTS pgsearch_sync_modality_codes(); DROP FUNCTION IF EXISTS pgsearch_sync_report_fields(); +DROP FUNCTION IF EXISTS pgsearch_sync_language_code(); +DROP FUNCTION IF EXISTS pgsearch_sync_modality_code_rename(); """ class Migration(migrations.Migration): dependencies = [ - ("pgsearch", "0003_search_projection_columns"), + ("pgsearch", "0002_hybrid_search"), ("reports", "0013_alter_report_options"), ] operations = [ + migrations.AddField( + model_name="reportsearchindex", + name="group_ids", + field=django.contrib.postgres.fields.ArrayField( + base_field=models.IntegerField(), default=list + ), + ), + migrations.AddField( + model_name="reportsearchindex", + name="language_code", + field=models.CharField(max_length=10, null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="modality_codes", + field=django.contrib.postgres.fields.ArrayField( + base_field=models.CharField(max_length=16), default=list + ), + ), + migrations.AddField( + model_name="reportsearchindex", + name="patient_age", + field=models.IntegerField(null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="patient_id", + field=models.CharField(max_length=64, null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="patient_sex", + field=models.CharField(max_length=1, null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="report_created_at", + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="report_updated_at", + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="study_datetime", + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name="reportsearchindex", + name="study_description", + field=models.CharField(blank=True, max_length=64, null=True), + ), migrations.RunSQL(SYNC_GROUP_IDS, reverse_sql=migrations.RunSQL.noop), migrations.RunSQL(SYNC_MODALITY_CODES, reverse_sql=migrations.RunSQL.noop), migrations.RunSQL(SYNC_REPORT_FIELDS, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_LANGUAGE_CODE, reverse_sql=migrations.RunSQL.noop), + migrations.RunSQL(SYNC_MODALITY_CODE_RENAME, reverse_sql=migrations.RunSQL.noop), migrations.RunSQL(CREATE_TRIGGERS, reverse_sql=DROP_TRIGGERS), ] diff --git a/radis/pgsearch/migrations/0003_search_projection_columns.py b/radis/pgsearch/migrations/0003_search_projection_columns.py deleted file mode 100644 index 8954cbdb..00000000 --- a/radis/pgsearch/migrations/0003_search_projection_columns.py +++ /dev/null @@ -1,64 +0,0 @@ -# Generated by Django 6.0.7 on 2026-08-30 18:52 - -import django.contrib.postgres.fields -from django.db import migrations, models - - -class Migration(migrations.Migration): - - dependencies = [ - ('pgsearch', '0002_hybrid_search'), - ] - - operations = [ - migrations.AddField( - model_name='reportsearchindex', - name='group_ids', - field=django.contrib.postgres.fields.ArrayField(base_field=models.IntegerField(), default=list), - ), - migrations.AddField( - model_name='reportsearchindex', - name='language_code', - field=models.CharField(max_length=10, null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='modality_codes', - field=django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=16), default=list), - ), - migrations.AddField( - model_name='reportsearchindex', - name='patient_age', - field=models.IntegerField(null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='patient_id', - field=models.CharField(max_length=64, null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='patient_sex', - field=models.CharField(max_length=1, null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='report_created_at', - field=models.DateTimeField(null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='report_updated_at', - field=models.DateTimeField(null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='study_datetime', - field=models.DateTimeField(null=True), - ), - migrations.AddField( - model_name='reportsearchindex', - name='study_description', - field=models.CharField(blank=True, max_length=64, null=True), - ), - ] diff --git a/radis/pgsearch/migrations/0005_search_projection_backfill.py b/radis/pgsearch/migrations/0004_search_projection_backfill.py similarity index 94% rename from radis/pgsearch/migrations/0005_search_projection_backfill.py rename to radis/pgsearch/migrations/0004_search_projection_backfill.py index b073f995..ec26bca9 100644 --- a/radis/pgsearch/migrations/0005_search_projection_backfill.py +++ b/radis/pgsearch/migrations/0004_search_projection_backfill.py @@ -5,17 +5,17 @@ one transaction that long would pin an equally long-lived snapshot. The first operation drops the HNSW embedding index for the duration of the -backfill; 0006 rebuilds it from the stored vectors. Every backfilled row is a +backfill; 0005 rebuilds it from the stored vectors. Every backfilled row is a full-row rewrite into pages that hold no free space -- a non-HOT update -- so with the index in place each embedded row would also insert into the HNSW graph: ~143 ms and ~2,160 buffer touches per row measured on an 8M corpus with 1.7M vectors, which turns the ten-minute window into days. Without embeddings the drop and the rebuild are both effectively instant. The drop is -database-only (model state keeps declaring the index; 0006 makes it true +database-only (model state keeps declaring the index; 0005 makes it true again) and IF EXISTS because atomic = False re-runs every operation after a failure. -Runs after the triggers (0004) on purpose. A report edited during the backfill +Runs after the triggers (0003) on purpose. A report edited during the backfill is corrected by its trigger, and a chunk that later reprocesses the same row simply rewrites the current values. @@ -99,7 +99,7 @@ def backfill(apps, schema_editor): class Migration(migrations.Migration): atomic = False - dependencies = [("pgsearch", "0004_search_projection_triggers")] + dependencies = [("pgsearch", "0003_search_projection")] operations = [ migrations.RunSQL( diff --git a/radis/pgsearch/migrations/0006_search_projection_indexes.py b/radis/pgsearch/migrations/0005_search_projection_indexes.py similarity index 92% rename from radis/pgsearch/migrations/0006_search_projection_indexes.py rename to radis/pgsearch/migrations/0005_search_projection_indexes.py index 341c8c7a..88955f6c 100644 --- a/radis/pgsearch/migrations/0006_search_projection_indexes.py +++ b/radis/pgsearch/migrations/0005_search_projection_indexes.py @@ -7,7 +7,7 @@ class Migration(migrations.Migration): dependencies = [ - ('pgsearch', '0005_search_projection_backfill'), + ('pgsearch', '0004_search_projection_backfill'), ('reports', '0013_alter_report_options'), ] @@ -28,7 +28,7 @@ class Migration(migrations.Migration): "SELECT gin_clean_pending_list('pgsearch_re_search__b0f715_gin');", reverse_sql="ALTER INDEX pgsearch_re_search__b0f715_gin SET (fastupdate = on);", ), - # Recreate the embedding index 0005 dropped, from the stored vectors: + # Recreate the embedding index 0004 dropped, from the stored vectors: # one bulk build instead of the per-row inserts the backfill would # otherwise have paid (7m36s versus ~68h measured at 1.7M vectors). # The build honors the server's maintenance_work_mem and @@ -36,7 +36,7 @@ class Migration(migrations.Migration): # corpus (POSTGRES_* knobs in the compose files). IF NOT EXISTS keeps # re-runs and installs that never dropped it safe; the definition must # match the HnswIndex in ReportSearchIndex.Meta. Reverse is a no-op on - # purpose: unapplying 0006 then 0005 must leave the index present, + # purpose: unapplying 0005 then 0004 must leave the index present, # which is the state before 0005 dropped it. migrations.RunSQL( "CREATE INDEX IF NOT EXISTS pgsearch_embedding_hnsw " @@ -48,7 +48,7 @@ class Migration(migrations.Migration): # Required, not hygiene: the projection columns and the indexes # above carry no statistics, and the single-table query shape # depends on the planner choosing a parallel sequential scan with - # a top-N heapsort. ANALYZE is transaction-legal, so 0006 stays atomic. + # a top-N heapsort. ANALYZE is transaction-legal, so this migration stays atomic. "ANALYZE pgsearch_reportsearchindex;", reverse_sql=migrations.RunSQL.noop, ), diff --git a/radis/pgsearch/migrations/0007_code_rename_triggers.py b/radis/pgsearch/migrations/0007_code_rename_triggers.py deleted file mode 100644 index c9d688b2..00000000 --- a/radis/pgsearch/migrations/0007_code_rename_triggers.py +++ /dev/null @@ -1,93 +0,0 @@ -"""Close two projection writers the 0004 triggers do not see: code renames. - -0004 syncs membership changes and report-row updates, but the mirrors store -the *code strings* of language and modalities, and nothing watched the code -columns themselves. Renaming a language silently strips its whole corpus from -search: every index row keeps mirroring the old code, so the language filter -and the per-config tsquery match both stop matching, while the drift checker -is the only thing that would notice. 0004's docstring lists three escaping -writers; these were the unlisted fourth and fifth. - -Row-level rather than 0004's statement-level shape, deliberately: PostgreSQL -does not allow transition tables on triggers with column lists ("transition -tables cannot be specified for triggers with column lists"), and unlike the -bulk membership writes 0004 handles, a rename statement touches one dimension -row -- FOR EACH ROW with a WHEN guard fires exactly once per actual code -change and costs nothing on these tiny tables. The modality function repeats -the array_agg aggregation of utils/projection.PROJECTION_UPDATE_SQL, the 0005 -backfill, 0004's membership functions and check_search_projection.DRIFT_SQL -- -keep them all in sync. - -Cost note: renaming a language rewrites the projection row of every report in -that language, and on an embedded corpus each rewrite also inserts into the -HNSW index (~143 ms/row measured). Renames are assumed rare and small; a -corpus-wide rename belongs in a maintenance window. -""" - -from django.db import migrations - -SYNC_LANGUAGE_CODE = """ -CREATE OR REPLACE FUNCTION pgsearch_sync_language_code() RETURNS trigger -LANGUAGE plpgsql AS $$ -BEGIN - UPDATE pgsearch_reportsearchindex rsi - SET language_code = NEW.code - FROM reports_report r - WHERE r.language_id = NEW.id - AND rsi.report_id = r.id - AND rsi.language_code IS DISTINCT FROM NEW.code; - RETURN NULL; -END $$; -""" - -SYNC_MODALITY_CODE_RENAME = """ -CREATE OR REPLACE FUNCTION pgsearch_sync_modality_code_rename() RETURNS trigger -LANGUAGE plpgsql AS $$ -BEGIN - UPDATE pgsearch_reportsearchindex rsi - SET modality_codes = full_agg.codes - FROM ( - SELECT rm.report_id, - array_agg(m.code ORDER BY m.code) AS codes - FROM reports_report_modalities rm - JOIN reports_modality m ON m.id = rm.modality_id - WHERE rm.report_id IN ( - SELECT rm2.report_id FROM reports_report_modalities rm2 - WHERE rm2.modality_id = NEW.id) - GROUP BY rm.report_id - ) full_agg - WHERE rsi.report_id = full_agg.report_id - AND rsi.modality_codes IS DISTINCT FROM full_agg.codes; - RETURN NULL; -END $$; -""" - -CREATE_TRIGGERS = """ -CREATE TRIGGER pgsearch_language_code_upd -AFTER UPDATE OF code ON reports_language -FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) -EXECUTE FUNCTION pgsearch_sync_language_code(); - -CREATE TRIGGER pgsearch_modality_code_upd -AFTER UPDATE OF code ON reports_modality -FOR EACH ROW WHEN (OLD.code IS DISTINCT FROM NEW.code) -EXECUTE FUNCTION pgsearch_sync_modality_code_rename(); -""" - -DROP_TRIGGERS = """ -DROP TRIGGER IF EXISTS pgsearch_language_code_upd ON reports_language; -DROP TRIGGER IF EXISTS pgsearch_modality_code_upd ON reports_modality; -DROP FUNCTION IF EXISTS pgsearch_sync_language_code(); -DROP FUNCTION IF EXISTS pgsearch_sync_modality_code_rename(); -""" - - -class Migration(migrations.Migration): - dependencies = [("pgsearch", "0006_search_projection_indexes")] - - operations = [ - migrations.RunSQL( - SYNC_LANGUAGE_CODE + SYNC_MODALITY_CODE_RENAME + CREATE_TRIGGERS, - reverse_sql=DROP_TRIGGERS, - ), - ] diff --git a/radis/pgsearch/models.py b/radis/pgsearch/models.py index e009cd7a..2fbbe005 100644 --- a/radis/pgsearch/models.py +++ b/radis/pgsearch/models.py @@ -24,7 +24,7 @@ class ReportSearchIndex(models.Model): # Search projection: mirrors of the Report fields the scan filters on, so # the FTS candidate query stays single-table. Maintained by the triggers in - # migration 0004 and populated on creation by signals.py / indexing.py. + # migration 0003 and populated on creation by signals.py / indexing.py. # The scalars are nullable and the arrays NOT NULL with a constant default, # which is what keeps adding a column here metadata-only on a large table. # Tightening them afterwards would cost a validating scan for no benefit: diff --git a/radis/pgsearch/signals.py b/radis/pgsearch/signals.py index a977133b..449929ce 100644 --- a/radis/pgsearch/signals.py +++ b/radis/pgsearch/signals.py @@ -12,7 +12,7 @@ def create_or_update_report_search_index(sender, instance, created, **kwargs): if created: ReportSearchIndex.objects.create(report=instance) # Groups and modalities are attached after the Report is created, so - # they stay empty here and the migration 0004 triggers fill them. + # they stay empty here and the migration 0003 triggers fill them. sync_projection([instance.pk]) return diff --git a/radis/pgsearch/tests/test_check_search_projection.py b/radis/pgsearch/tests/test_check_search_projection.py index c29fa929..5831911b 100644 --- a/radis/pgsearch/tests/test_check_search_projection.py +++ b/radis/pgsearch/tests/test_check_search_projection.py @@ -52,7 +52,7 @@ def test_detects_drifted_group_ids(): def test_reports_no_drift_after_code_renames(): - """The 0007 triggers keep the mirrors current through code renames, so the + """The 0003 rename triggers keep the mirrors current through code renames, so the checker must come back clean afterwards.""" language = LanguageFactory.create(code="en") report = ReportFactory.create(language=language, modalities=[]) diff --git a/radis/pgsearch/tests/test_search_projection.py b/radis/pgsearch/tests/test_search_projection.py index 21e39833..85058b77 100644 --- a/radis/pgsearch/tests/test_search_projection.py +++ b/radis/pgsearch/tests/test_search_projection.py @@ -21,7 +21,7 @@ pytestmark = pytest.mark.django_db -BACKFILL_MIGRATION = "radis.pgsearch.migrations.0005_search_projection_backfill" +BACKFILL_MIGRATION = "radis.pgsearch.migrations.0004_search_projection_backfill" class _MinimalSchemaEditor: @@ -199,7 +199,7 @@ def test_creation_populates_the_mirrored_scalars(): ReportFactory.create() would not pin this: its post_generation hooks (modalities, the metadata RelatedFactoryList) make factory_boy issue an implicit extra save() afterwards, which fires the AFTER UPDATE trigger - from migration 0004 and would populate these scalars regardless of + from migration 0003 and would populate these scalars regardless of whether the signal's own sync_projection() call exists. build() skips those hooks, so the only thing that can fill the scalars here is the signal's sync_projection() call on creation. @@ -251,7 +251,7 @@ def test_migration_backfill_executes_the_chunked_update(monkeypatch): """Verify the migration's own SQL fills the projection, across several chunks. This exercises the literal SQL and the loop in - 0005_search_projection_backfill.py, not sync_projection(). CHUNK_SIZE is + 0004_search_projection_backfill.py, not sync_projection(). CHUNK_SIZE is 50,000 in production, so a handful of reports would run the loop body exactly once and an off-by-one in the ``low``/``high`` arithmetic -- the kind that silently skips whole id ranges -- would not show. Patching it to @@ -454,7 +454,7 @@ def test_modality_code_rename_updates_the_projection(): def test_hnsw_index_survives_the_backfill_migrations(): - """0005 drops the embedding index and 0006 rebuilds it; the test database + """0004 drops the embedding index and 0005 rebuilds it; the test database runs both, so the index existing here proves the whole round trip.""" with connection.cursor() as cursor: cursor.execute("SELECT 1 FROM pg_indexes WHERE indexname = 'pgsearch_embedding_hnsw'") diff --git a/radis/pgsearch/utils/projection.py b/radis/pgsearch/utils/projection.py index 7d93c43d..3c94cf12 100644 --- a/radis/pgsearch/utils/projection.py +++ b/radis/pgsearch/utils/projection.py @@ -1,11 +1,11 @@ """The canonical SQL that fills the ReportSearchIndex search projection. -Triggers (migration 0004) maintain the projection when its sources change. +Triggers (migration 0003) maintain the projection when its sources change. This module fills it when an index row is first created, which happens before the report's groups and modalities are attached. The backfill migration duplicates this statement in SQL, the same way bulk_upsert_report_search_indexes already duplicates the tsvector logic -- keep them in sync. The two trigger -functions in migration 0004 and check_search_projection.DRIFT_SQL repeat the +functions in migration 0003 and check_search_projection.DRIFT_SQL repeat the same array_agg shape and belong to that set as well. """ From 6be56458a5ca63b80550b6480e5b34ffeb88a6d0 Mon Sep 17 00:00:00 2001 From: Samuel Kwong Date: Thu, 24 Sep 2026 00:32:13 +0200 Subject: [PATCH 50/50] Name the real single-table guard test in the filter-query comment Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01VDei6anDxfR5eoHhFfXBGs --- radis/pgsearch/providers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/radis/pgsearch/providers.py b/radis/pgsearch/providers.py index c7da25bc..e15497ce 100644 --- a/radis/pgsearch/providers.py +++ b/radis/pgsearch/providers.py @@ -196,7 +196,7 @@ def _build_filter_query(filters: SearchFilters) -> Q: # projection (see migration 0003) mirrors the Report fields search filters # on, so the candidate query stays single-table. Reintroducing a ``report__`` # traversal restores the join and the multi-second query shape it caused; - # test_filter_query_plan_is_single_table guards against that. + # test_fts_candidate_query_is_single_table guards against that. # # Group-scoped access control. ``SearchView`` supplies # ``group=active_group.pk``; the extraction preview may pass ``group=None``