Refresh stale documentation

AGENTS.md

@@ -42,7 +42,7 @@ uv run cli db-backup                 # Backup database
 ### Tech Stack
 
 - **Backend**: Python 3.12+, Django 6.1+, PostgreSQL 17
-- **Search**: pg_vector (semantic), pg_search (full-text), hybrid ranking
+- **Search**: PostgreSQL full-text search (tsvector), pg_vector (semantic), hybrid RRF ranking
 - **Async**: Daphne (ASGI), Django Channels, Procrastinate (task queue)
 - **Frontend**: Django templates, Cotton components, HTMX, Alpine.js, Bootstrap 5
 - **LLM**: External OpenAI-compatible API endpoint
@@ -86,7 +86,7 @@ Analysis operations follow a Job -> Task pattern (similar to ADIT):
 - **default_worker**: General background task processor (Procrastinate queue: `default`)
 - **llm_worker**: LLM-specific task processor (Procrastinate queue: `llm`)
 - **embeddings_worker**: Embedding task processor (Procrastinate queue: `embeddings`)
-- **postgres**: PostgreSQL 17 with pg_vector and pg_search extensions (port 5432)
+- **postgres**: PostgreSQL 17 with the pg_vector extension (port 5432)
 
 ### LLM Endpoint
 
@@ -155,6 +155,9 @@ Labeling uses the shared core LLM client (`radis.core.utils.llm_client`); its ti
 - **Comments**: only where the code cannot speak for itself; explain *why*, not *what*
 - **No history in comments**: describe the code as it is, not how it changed — that
   belongs in the commit message (docstrings too)
+- **Keep the docs in sync**: when a change adds a feature or alters behaviour that the
+  docs describe (`README.md`, `docs/`, this file, in-app help templates), update them in
+  the same PR
 
 ## Key Dependencies
 
@@ -219,7 +222,7 @@ reports = response.json()
 
 ### Search Not Returning Expected Results
 
-- Check PostgreSQL extensions are installed: `pg_vector`, `pg_search`
+- Check the `vector` extension is installed in PostgreSQL
 - Verify report has `body` text indexed
 - Check search provider is configured in settings
 - Review QueryParser syntax for complex queries

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing to Our Project
 
-We're excited that you're interested in contributing to our project! This document outlines the
+Thank you for contributing to RADIS. This document outlines the
 guidelines for contributing to our codebase. We follow the Google Python Style Guide to maintain
 consistency and readability across our project.
 
@@ -17,7 +17,7 @@ cp ./example.env ./.env  # adjust the environment variables to your needs
 uv run cli compose-up -- --watch
 ```
 
-The development server of the example project will be started on <http://localhost:8000>
+The development server is then available on <http://localhost:8000>
 
 File changes will be automatically detected and the servers will be restarted. When library
 dependencies are changed, the containers will automatically be rebuilt and restarted.

README.md

@@ -35,28 +35,29 @@ The detailed documentation of RADIS can be found at <https://openradx.github.io/
 ## Features
 
 - **Intuitive Web Interface**: Simplified access to radiology reports stored in the application database through a user-friendly web portal.
-- **Advanced Text Search**: Robust search functionality combining semantic analysis and traditional keyword-based methods for precise report retrieval.
+- **Advanced Text Search**: Full-text search with boolean operators and metadata filters, optionally combined with semantic search when an embedding model is configured.
 - **Seamless PACS Viewer Integration**: Direct access to PACS viewers with deep linking to relevant studies, leveraging the viewer's capabilities for a smooth workflow.
-- **AI-Powered Search and Filtering**: Integration of large language models (LLMs) to enhance report discovery and filter options based on contextual understanding.
+- **Automated Labeling**: Large language models (LLMs) assign administrator-defined labels to reports, shown as badges and usable as a search filter.
+- **Structured Data Extraction**: Extraction jobs pull typed fields (text, numbers, booleans, selections) out of thousands of reports at once, with results exportable as CSV.
+- **Smart Notification System**: Subscription service that notifies users of new reports matching specific criteria, with optional LLM-based filter questions and data extraction per report.
+- **Report Chat**: Ask an AI assistant questions about an individual report in natural language.
 - **Bookmarking and Collection Management**: Organize reports into customizable collections with an intuitive bookmarking service for quick access and review.
 - **Custom Report Notes**: Allow users to append personalized notes to reports for additional context or annotations.
-- **Smart Notification System**: Subscription service that notifies users of new reports matching specific criteria, ensuring timely updates.
+- **API and Python Client**: Programmatic access to reports through a REST API and the `radis-client` library.
 
 ## Planned
 
-- **Automated Report Classification and Organization**: Leverage LLMs to intelligently classify, tag, and organize reports based on content and metadata.
-- **Developer-Friendly API Access**: Provide programmatic access to application features through a comprehensive API, with an optional Python client for seamless integration into workflows.
 - **Report Quality Assurance**: Tools to review and assess reports for consistency, completeness, and content accuracy, ensuring high-quality documentation.
 
 ## API Client
 
-[RADIS Client](https://pypi.org/project/radis-client/) is a Python library to search for reports on RADIS in a programmatic way. It also allows admins to feed new reports to RADIS.
+[RADIS Client](https://pypi.org/project/radis-client/) is a Python library for admins to feed reports to RADIS and to retrieve, update or delete them programmatically.
 
 ## Architectural overview
 
 RADIS employs a sophisticated multi-container architecture, optimized for local deployment using Docker Swarm mode—a feature included with all Docker installations. This local-first approach ensures compliance with the strict data security requirements inherent in hospital and research environments where sensitive patient or research data is managed. By leveraging Docker Swarm, RADIS offers seamless scalability, allowing services to be easily adjusted to meet the specific computational demands of the deployment site. To simplify the setup process, RADIS provides intuitive deployment scripts, ensuring accessibility for users with varying levels of technical expertise.
 
-The RADIS web service is built on the robust Django Python framework, with data securely stored in a PostgreSQL database. By default, RADIS harnesses PostgreSQL’s powerful full-text search capabilities, enhanced by the pg_search and pg_vector extensions, to deliver hybrid search functionality. Its modular architecture enables effortless integration with other advanced text and vector search systems, such as Vespa or ElasticSearch, through easily implementable plugins.
+The RADIS web service is built on the robust Django Python framework, with data securely stored in a PostgreSQL database. By default, RADIS uses PostgreSQL's built-in full-text search, combined with vector search through the pg_vector extension when an embedding model is configured. Other search systems, such as Vespa or ElasticSearch, can be plugged in through the search provider interface.
 
 To deliver a dynamic and responsive web interface, RADIS integrates modern JavaScript libraries HTMX and Alpine.js. For resource-intensive operations—such as batch evaluations or filtering large datasets—RADIS relies on Procrastinate, a Python-based distributed task processing library. This ensures long-running tasks are executed efficiently in the background, minimizing disruptions to user workflows while maximizing system performance.
 

docs/backups.md

@@ -1,3 +1,5 @@
 # Backups
 
-For database backups django-dbbackup app is used. The backups are done every night at 3 am by a periodic task using the dbbackup management command and stored in the `backups` directory which is mounted as a volume. The dbbackup command can also be called manually with `uv run cli db-backup`.
+Database backups are made by the django-dbbackup app. A periodic task runs the `dbbackup` management command every night at 3 am (`BACKUP_CRON`; set `BACKUP_ENABLED=false` to turn it off). Backups are written to the host directory in `BACKUP_DIR`, which is mounted into the containers as `/backups`, and the last 30 are kept.
+
+Run a backup by hand with `uv run cli db-backup`, and restore the latest one with `uv run cli db-restore`. Both need a running `web` container.

docs/dev-docs/architecture.md

@@ -12,46 +12,51 @@ RADIS inherits common functionality from **ADIT Radis Shared**, a shared library
 
 The RADIS platform provides report management, advanced search, AI-powered analysis, and collaborative features through coordinated Docker containers. Users access the system via **web browser** or **RADIS Client** (Python library for programmatic access), performing operations such as searching reports, creating collections, managing subscriptions, and analyzing reports with AI.
 
-The system consists of three main components: a Django API server handling web UI and orchestration, a PostgreSQL database storing all persistent data and serving as the task queue, and background workers executing long-running AI operations.
+The system consists of three main components: a Django API server handling web UI and orchestration, a PostgreSQL database storing all persistent data and serving as the task queue, and background workers executing long-running AI operations. In production an `init` service runs migrations, static file collection and superuser creation before the other services start.
+
+The Django apps are `core` (job/task base classes, crash recovery), `reports`, `search` (query parser, provider interface), `pgsearch` (PostgreSQL provider, embeddings), `extractions`, `labels`, `subscriptions`, `collections`, `notes` and `chats`.
 
 ## Backend Architecture
 
 **Django Web/API Server**: Central coordination engine providing REST API endpoints, authentication, user/session management, static assets, and task orchestration. Creates job/task records in PostgreSQL and schedules background work.
 
-**PostgreSQL Database**: System of record storing user accounts, reports, collections, subscriptions, task queue entries, execution history, and search indexes. Uses pg_search and pg_vector extensions for hybrid search capabilities.
+**PostgreSQL Database**: System of record storing user accounts, reports, collections, subscriptions, task queue entries, execution history, and search indexes. Full-text search uses PostgreSQL's built-in `tsvector`; the pg_vector extension stores report embeddings for semantic search.
 
-**Background Workers**: Docker containers polling PostgreSQL for tasks, executing AI-powered extractions and subscription processing using LLMs.
+**Background Workers**: Docker containers polling PostgreSQL for tasks: extractions, subscription processing and labeling with LLMs, and report embeddings.
 
 ### Procrastinate Task Queue System
 
 RADIS uses [Procrastinate](https://procrastinate.readthedocs.io/en/stable/), a PostgreSQL-based task queue storing jobs directly in the database without external message brokers. Tasks are Python functions with decorators, supporting job scheduling, prioritization, retry logic, cancellation, and periodic task execution.
 
 **RADIS Task Types**:
 
-- **Default Queue**: `process_extraction_job`, `process_subscription_job`, `subscription_launcher` (periodic), `check_disk_space`, `backup_db`
-- **LLM Queue**: `process_extraction_task`, `process_subscription_task` (AI-intensive operations)
+- **Default Queue**: `process_extraction_job`, `process_subscription_job`, `process_labeling_job`, `bulk_index_reports`, and the periodic tasks `subscription_launcher` (hourly), `incremental_label_scan` (nightly), `sweep_stale_tasks_periodic` (every minute), `retry_stalled_jobs` and `backup_db` (both from adit-radis-shared)
+- **LLM Queue**: `process_extraction_task`, `process_subscription_task`, `process_labeling_task` (AI-intensive operations)
+- **Embeddings Queue**: `embed_reports_task`
 
 ## Frontend Architecture
 
 **Web UI**: Server-side rendered with Django templates and HTMX for dynamic interactions. Uses Bootstrap 5 for styling and Alpine.js for interactive components.
 
-**RADIS Client**: Python package (`radis-client`) for programmatic API access, supporting report creation and search operations.
+**RADIS Client**: Python package (`radis-client`) to create, retrieve, update and delete reports through the REST API. There is no search API.
 
 ## Docker Container Architecture
 
 **Docker Swarm**: RADIS employs a sophisticated multi-container architecture, optimized for local deployment using Docker Swarm mode—a feature included with all Docker installations. This local-first approach ensures compliance with the strict data security requirements inherent in hospital and research environments where sensitive patient or research data is managed. By leveraging Docker Swarm, RADIS offers seamless scalability, allowing services to be easily adjusted to meet the specific computational demands of the deployment site.
 
 ### Container Types
 
-**Web Container (`radis-web-1`)**: Runs Django application serving web UI and REST API. Ports: 8000 (dev), 80/443 (prod with SSL). Handles authentication, serves static files, enqueues tasks, and manages database connections. In production, runs with 3 replicas for high availability.
+Container names are prefixed with the stack name, e.g. `radis_dev-web-1` in development and `radis_prod_web.1.<id>` under Docker Swarm.
+
+**Web Container (`web`)**: Runs Django application serving web UI and REST API. Ports: 8000 (dev), 80/443 (prod with SSL). Handles authentication, serves static files, enqueues tasks, and manages database connections. In production, runs with 3 replicas for high availability.
 
-**PostgreSQL Container (`radis-postgres-1`)**: PostgreSQL database storing all data (users, reports, collections, subscriptions, tasks, logs, Procrastinate queue). Port 5432. Uses Docker volumes for persistence.
+**PostgreSQL Container (`postgres`)**: PostgreSQL database storing all data (users, reports, collections, subscriptions, tasks, logs, Procrastinate queue). Port 5432. Uses Docker volumes for persistence.
 
-**Default Worker Container (`radis-default_worker-1`)**: Processes background tasks in the default queue (e.g., extraction job preparation, subscription job preparation, periodic subscription launcher, disk space checks, database backups).
+**Default Worker Container (`default_worker`)**: Processes background tasks in the default queue (job preparation for extractions, subscriptions and labeling, the periodic tasks listed above, database backups).
 
-**LLM Worker Container (`radis-llm_worker-1`)**: Executes AI-intensive tasks from the llm queue (extraction tasks, subscription tasks). Uses ChatClient to communicate with the configured LLM endpoint.
+**LLM Worker Container (`llm_worker`)**: Executes AI-intensive tasks from the llm queue (extraction, subscription and labeling tasks). Uses `LLMClient` (`radis.core.utils.llm_client`) to talk to the configured LLM endpoint.
 
-**Embeddings Worker Container (`radis-embeddings_worker-1`)**: Drains the embeddings
+**Embeddings Worker Container (`embeddings_worker`)**: Drains the embeddings
 queue — generating and storing report vectors for hybrid search, including operator
 backfills started by `./manage.py embed_pending` or the admin action.
 
@@ -77,7 +82,7 @@ directly instead of the client truncating a larger vector. `EMBEDDINGS_BASE_URL`
 both when the provider multiplexes models (OpenAI, Ollama, a gateway), while a
 self-hosted vLLM or SGLang serves one model per process and needs the override. The
 service has its own rate-limit gate — a 429 from the embedding gateway must not pause
-inference — and its own worker (`radis-embeddings_worker-1`) draining the `embeddings`
+inference — and its own worker (`embeddings_worker`) draining the `embeddings`
 queue, so a million-report backfill cannot starve extractions.
 
 ## Search Architecture
@@ -86,22 +91,26 @@ RADIS uses a modular search architecture allowing different search providers to
 
 **Search Provider Interface**: Defines search, retrieval, and indexing operations
 
-- **PgSearch Provider**: Default implementation using PostgreSQL full-text search with pg_search and pg_vector extensions
-- **Alternative Providers**: Vespa and ElasticSearch can be integrated through the same interface
+- **PgSearch Provider**: Default implementation using PostgreSQL full-text search and pg_vector
+- **Alternative Providers**: Other engines such as Vespa or ElasticSearch can be integrated through the same interface
 
 **Query Parser**: Parses user queries with support for:
 
-- AND/OR operators
+- `AND`, `OR` and `NOT` operators (uppercase; implicit AND between terms)
 - Phrase search ("exact match")
-- Exclusion (-term)
+- Grouping with parentheses
 - Case-insensitive matching
 
+Unsupported syntax (`-term`, `field:value`, wildcards) is stripped and the corrected query is shown to the user.
+
 **Search Filters**: Applied on top of query:
 
 - Language, modalities, study date range
 - Study description, patient sex, patient age range
-- Patient ID, group access
-- Created after timestamp (for subscriptions)
+- Patient ID, labels, group access
+- Created before/after and updated after timestamps (for subscriptions and labeling)
+
+**Hybrid ranking**: When `EMBEDDINGS_MODEL` is set, the provider runs the full-text query (up to `HYBRID_FTS_MAX_RESULTS` candidates) and a cosine-distance nearest-neighbour query on the report vectors (`HYBRID_VECTOR_TOP_K`), then fuses both lists with reciprocal rank fusion (`HYBRID_RRF_K`). `NOT` terms are removed before the query is embedded, and query embeddings are cached. If the embedding service fails, the search degrades to full-text only.
 
 **Text-search configurations**: reports are indexed with the PostgreSQL configuration for
 their own language, so stemming matches the text — an English report stores "effusion" as
@@ -114,3 +123,11 @@ does no stemming and therefore matches literally. An unset language filter is no
 to subscriptions' "All" choice: the search form's language field is optional with no
 explicit empty option, so a bookmarked or shared search URL missing `language=` — or the
 search page's first, filter-less load — resolves the same way.
+
+## Auto-Labeling
+
+The `labels` app classifies reports against administrator-defined labels. A `LabelGroup` has a Yes/No gate question; a report only gets the group's labels classified when the gate answers Yes. Each `Label` result is one of `PRESENT`, `LIKELY`, `POSSIBLE`, `ABSENT` or `UNMENTIONED`; the first three appear as badges and in the search filter. Work runs as `LabelingJob`/`LabelingTask` through the usual job/task pattern, either as a manual backfill from Django Admin or through the nightly `incremental_label_scan`, which labels reports updated since the `LabelingScanCheckpoint`. A result is stale when its label or report changed after it was generated.
+
+## Worker Crash Recovery
+
+A task left `IN_PROGRESS` by a killed worker is repaired by `radis.core.utils.recovery`: it is reset to `PENDING` and re-enqueued once its worker has sent no heartbeat for `ANALYSIS_STALLED_WORKER_GRACE_SECONDS`. The sweep runs on worker startup (`sweep_stale_tasks`) and periodically (`ANALYSIS_SWEEP_CRON`). A job whose preparation was interrupted is retried by the queue and starts preparation over, discarding half-created tasks. Embedding subjobs are plain queue jobs and rely on Procrastinate retries instead.