From b20b975cc44c0efc206d36bd1a1a8d12b33de268 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Wed, 26 Aug 2026 11:42:22 +0000
Subject: [PATCH 1/7] Refresh user-facing docs for labels, extractions,
 subscriptions and LLM config

The user and admin guides predated auto-labeling (#231, #265), the extraction
wizard rework (#252), the subscription rework (#253), hybrid search (#226, #274),
the externalized LLM configuration (#267) and worker-crash recovery (#276).

User guide: correct the search syntax (uppercase AND/OR/NOT, no minus
exclusion) and filter list (Language "All" default, Labels, no Patient ID,
sex is All/Male/Female), add a Labels section, and rewrite Subscriptions
(filter questions, extraction fields, hourly refresh, inbox, CSV) and
Extractions (three-step wizard, field types and array toggle, generated
query, live count, results and CSV, job actions).

Admin guide: add LLM and embeddings configuration, auto-labeling
administration (label groups, gate question, backfill, scan, labels_status),
worker-crash recovery, post-update steps, job verification, and fix the
urgent permission label ("Can analyze urgently").

README and index: move shipped features (labeling, API client) out of
"Planned" and list labeling, extraction and chat among the features.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016pVE1URXqr6YirSg3SmVuJ
---
 README.md                     |  11 +--
 docs/index.md                 |  11 +--
 docs/user-docs/admin-guide.md | 140 ++++++++++++++++++++++++++++------
 docs/user-docs/user-guide.md  | 121 ++++++++++++++++++++---------
 4 files changed, 213 insertions(+), 70 deletions(-)

diff --git a/README.md b/README.md
index d46704dd6..bb63e0779 100644
--- a/README.md
+++ b/README.md
@@ -35,17 +35,18 @@ The detailed documentation of RADIS can be found at <https://openradx.github.io/
 ## Features
 
 - **Intuitive Web Interface**: Simplified access to radiology reports stored in the application database through a user-friendly web portal.
-- **Advanced Text Search**: Robust search functionality combining semantic analysis and traditional keyword-based methods for precise report retrieval.
+- **Advanced Text Search**: Full-text search with boolean operators and metadata filters, optionally combined with semantic search when an embedding model is configured.
 - **Seamless PACS Viewer Integration**: Direct access to PACS viewers with deep linking to relevant studies, leveraging the viewer's capabilities for a smooth workflow.
-- **AI-Powered Search and Filtering**: Integration of large language models (LLMs) to enhance report discovery and filter options based on contextual understanding.
+- **Automated Labeling**: Large language models (LLMs) assign administrator-defined labels to reports, shown as badges and usable as a search filter.
+- **Structured Data Extraction**: Extraction jobs pull typed fields (text, numbers, booleans, selections) out of thousands of reports at once, with results exportable as CSV.
+- **Smart Notification System**: Subscription service that notifies users of new reports matching specific criteria, with optional LLM-based filter questions and data extraction per report.
+- **Report Chat**: Ask an AI assistant questions about an individual report in natural language.
 - **Bookmarking and Collection Management**: Organize reports into customizable collections with an intuitive bookmarking service for quick access and review.
 - **Custom Report Notes**: Allow users to append personalized notes to reports for additional context or annotations.
-- **Smart Notification System**: Subscription service that notifies users of new reports matching specific criteria, ensuring timely updates.
+- **API and Python Client**: Programmatic access to reports and search through a REST API and the `radis-client` library.
 
 ## Planned
 
-- **Automated Report Classification and Organization**: Leverage LLMs to intelligently classify, tag, and organize reports based on content and metadata.
-- **Developer-Friendly API Access**: Provide programmatic access to application features through a comprehensive API, with an optional Python client for seamless integration into workflows.
 - **Report Quality Assurance**: Tools to review and assess reports for consistency, completeness, and content accuracy, ensuring high-quality documentation.
 
 ## API Client
diff --git a/docs/index.md b/docs/index.md
index e1db8906a..574df9da5 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -48,11 +48,12 @@ RADIS serves as a comprehensive platform for managing radiology reports, enablin
 **RADIS acts as an intelligent hub** for radiology reports:
 
 1. **Report Storage**: Securely store radiology reports with structured metadata
-2. **Advanced Search**: Use hybrid search combining traditional text search with semantic understanding
-3. **AI-Powered Analysis**: Leverage large language models for intelligent filtering and categorization
-4. **Collections**: Organize reports into custom collections for easy access and review
-5. **Subscription**: Subscribe to searches and get notified when new matching reports arrive
-6. **Notes**: Add personal notes to reports for annotations and context
+2. **Advanced Search**: Full-text search with filters, optionally combined with semantic search for meaning-based retrieval
+3. **Automated Labeling**: Large language models assign administrator-defined labels to reports for browsing and filtering
+4. **Data Extraction**: Extract structured fields from many reports at once and export them as CSV
+5. **Subscriptions**: Subscribe to searches and get notified when new matching reports arrive, with optional AI-based filtering and extraction
+6. **Collections and Notes**: Organize reports into custom collections and add personal notes for context
+7. **Chats**: Ask an AI assistant questions about a report
 
 **Ready to modernize your radiology workflow?** RADIS combines traditional database functionality with cutting-edge AI to make radiology reports more accessible and actionable.
 
diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md
index 0b0d680b6..c3d09fe73 100644
--- a/docs/user-docs/admin-guide.md
+++ b/docs/user-docs/admin-guide.md
@@ -12,6 +12,8 @@ cp ./example.env ./.env  # copy example environment to .env
 uv run cli stack-deploy  # builds and starts Docker containers
 ```
 
+Before the first deployment, edit `.env` and set at least the secrets, `DJANGO_ALLOWED_HOSTS`, and the LLM endpoint (see [LLM and Embeddings Configuration](#llm-and-embeddings-configuration)). RADIS refuses to start without `LLM_BASE_URL` and `LLM_DEFAULT_MODEL`.
+
 ## Updating RADIS
 
 Follow these steps to safely update your RADIS installation:
@@ -22,11 +24,61 @@ Follow these steps to safely update your RADIS installation:
 4. **Backup database**: Run `uv run cli db-backup` to create a database backup
 5. **Remove stack**: Run `uv run cli stack-rm` to remove all Docker containers and services
 6. **Pull latest changes**: Run `git pull origin main` to fetch the latest code updates
-7. **Update environment**: Compare `example.env` with your `.env` file and add any new environment variables or update changed values
+7. **Update environment**: Compare `example.env` with your `.env` file and add any new environment variables or update changed values. Pay particular attention to the `LLM_*`, `EMBEDDINGS_*`, `LABELING_*`, and `ANALYSIS_*` groups, which are explained below
 8. **Pull Docker images**: Run `uv run cli compose-pull` to download the latest Docker images
 9. **Deploy stack**: Run `uv run cli stack-deploy` to rebuild and start all services with the updated code
 10. **Disable maintenance mode**: In Django Admin, navigate to **Common** → **Project Settings** and uncheck the "Maintenance mode" checkbox, then save
 
+Depending on what changed, one of these follow-up steps may be needed after the stack is up:
+
+- If you enabled or changed the embedding model, backfill the vectors of existing reports (see [Embeddings](#embeddings-hybrid-search))
+- If you added or changed labels, run a labeling backfill (see [Auto-Labeling](#auto-labeling))
+
+Tasks that were interrupted by the restart are repaired automatically (see [Background Workers](#background-workers-and-crash-recovery)).
+
+## LLM and Embeddings Configuration
+
+All AI features (chats, extractions, subscriptions, labeling, query generation) send their requests to a single OpenAI-compatible endpoint. RADIS ships no inference server; you point it at one you operate (e.g. Ollama, vLLM, SGLang, an LLM gateway) or at a hosted API. The endpoint must be reachable from inside the containers on every node of the stack; `host.docker.internal` only works in development.
+
+### LLM Settings
+
+Set these in `.env`:
+
+- `LLM_BASE_URL` (required): Base URL of the endpoint, e.g. `http://llm.internal:11434/v1`
+- `LLM_API_KEY`: API key for the endpoint. Many self-hosted providers ignore it, but it must be set
+- `LLM_DEFAULT_MODEL` (required): The model every feature uses unless overridden
+- `LLM_CHATS_MODEL`, `LLM_QUERY_GENERATION_MODEL`, `LLM_EXTRACTIONS_MODEL`, `LLM_SUBSCRIPTIONS_MODEL`, `LLM_LABELING_MODEL`: Per-feature overrides. Leave blank to use the default model. Useful to spend a stronger model where it matters (e.g. labeling) and a fast one for chat
+
+Every model setting takes the form `model[?param=value&...]`. The parameters are merged into each request body, so both standard fields and provider extensions work:
+
+```text
+LLM_DEFAULT_MODEL=qwen3:8b?temperature=0&chat_template_kwargs.enable_thinking=false
+```
+
+Values are read as JSON where possible (`temperature=0` sends a number, `reasoning_effort=none` sends the string `"none"`), and a dotted key becomes a nested object. Drop parameters the provider does not know if it rejects requests with a 400 error.
+
+Request timeout, rate-limit handling, and transient retries have sensible defaults (`LLM_REQUEST_TIMEOUT_SECONDS`, `LLM_RATE_LIMIT_*`, `LLM_TRANSIENT_RETRY_*`); override them only if needed.
+
+### Embeddings (Hybrid Search)
+
+By default RADIS runs full-text search only. Setting `EMBEDDINGS_MODEL` switches on hybrid search, where full-text and semantic (vector) results are fused into one ranking:
+
+- `EMBEDDINGS_MODEL`: The embedding model, in the same `model[?param=value]` form. Leave empty for full-text search only
+- `EMBEDDINGS_BASE_URL`, `EMBEDDINGS_API_KEY`: Default to `LLM_BASE_URL` and `LLM_API_KEY`. Set them only when embeddings are served from a different endpoint, which is the normal case for vLLM and SGLang since they serve one model per process
+- `EMBEDDINGS_DIM`: Vector width (default `1024`). This is coupled to the database schema: changing it after deployment requires dropping the embedding column, re-migrating, and re-embedding all reports. If the model spec also sets `dimensions`, both must agree; RADIS checks this at startup
+- `EMBEDDINGS_QUERY_INSTRUCTION`: Model-specific instruction prefix for search queries (some models, e.g. Qwen3-Embedding, want one)
+- `EMBEDDINGS_QUERY_CACHE_TIMEOUT_SECONDS`: How long a query's vector stays cached (default 900 s). Lower it temporarily after swapping the model, or the old vectors keep serving stale results for up to this long
+
+New and updated reports are embedded automatically by the `embeddings_worker` service. Reports that existed before the model was configured have no vector and are found by full-text search only until you backfill them:
+
+```terminal
+docker compose exec web ./manage.py embed_pending
+```
+
+The command is idempotent and resumable; `embed_cancel` stops a running backfill. When the embedding service is unreachable or rate limiting, searches silently fall back to full-text results and a warning is logged in the `web` service.
+
+See the [Contributing guide](../dev-docs/contributing.md) for how to run Ollama with a chat and an embedding model, and the troubleshooting sections there for connection problems.
+
 ## User and Group Management
 
 Administrators can create users by navigating to the Django Admin section. Alternatively, users can self-register, after which an administrator must approve and activate their account.
@@ -68,6 +120,7 @@ Each user has an **active group** that determines which reports they can current
 
 - Only reports associated with the active group are visible in searches and collections
 - This ensures proper data isolation between different departments or projects
+- Users need an active group to create subscriptions and extraction jobs; the job or subscription is bound to that group
 
 ## Report Management
 
@@ -75,45 +128,84 @@ Each user has an **active group** that determines which reports they can current
 
 Administrators can import reports programmatically via the RADIS API or using the RADIS Client library. See the **RADIS Client** section below for details.
 
-### Extraction Jobs
+Every import or update of a report re-indexes it for search, queues it for embedding (if configured), and marks its labels for re-labeling on the next scan (see below). This also applies to updates that do not change the report text, so avoid re-pushing an unchanged corpus if you want to keep LLM load down.
+
+## Auto-Labeling
+
+RADIS can label reports automatically with an LLM. Labels are defined system-wide by administrators in Django Admin; there is no in-app editor. Users see the results as badges on the report detail page and can filter searches by label.
 
-Administrators can monitor and manage extraction jobs where AI analyzes reports to extract structured data.
+### Concepts
+
+- A **Label group** has a name and a **gate question**: a Yes/No question the LLM answers once per report before any label of the group is evaluated (e.g. "Is this a CT of the chest?"). A No answer skips the whole group for that report, which is the main lever for keeping LLM cost down
+- A **Label** belongs to one group and has a name (unique across all groups), a **description** that defines the finding for the LLM, and an **active** flag. Inactive labels are neither classified nor offered in the search filter
+- For each report and active label the LLM assigns one of five results: Present, Likely, Possible, Absent, or Unmentioned. Only the first three surface as badges and in the search filter
+
+### Managing Labels
+
+1. **Access Django Admin**: Navigate to **Labels** → **Label groups** and create a group with its gate question
+2. Navigate to **Labels** → **Labels** and add labels to the group. Write the description as the definition you would give a radiologist; it is sent to the LLM verbatim
+3. Run a backfill (see below) so that existing reports are labeled
+
+Editing a label description, a group, or its gate question makes the existing results of that label or group **stale**. Stale results keep surfacing (with a greyed-out badge) until they are regenerated. Deactivating a label stops new classifications, but results already assigned remain visible on the reports.
+
+### Backfill and Periodic Scan
+
+Labeling runs as background jobs on the `llm` queue:
+
+- **Periodic scan**: Every night (`LABELING_SCAN_CRON`, default `0 2 * * *`) RADIS labels all reports that were created or updated since the last scan. The first scan after installation only records a checkpoint; reports that existed before are not labeled until you run a backfill
+- **Backfill**: In Django Admin, navigate to **Labels** → **Labeling jobs** and click **Run backfill now**. This labels every report with a missing or stale result for any active label. The job is safe to cancel and restart. Only one labeling job can be active at a time; the scan skips its tick while a backfill runs
+
+The **Labeling jobs** page shows all jobs with their trigger (Periodic scan or Manual backfill) and status; a job can be canceled from its detail page. **Label results**, **Gate answers**, and **Labeling tasks** are available read-only for inspection. A task ends with status Warning when some of its reports could not be labeled and Failure when none could, which usually points to an LLM outage.
+
+To check the corpus-wide state (checkpoint, per-label and per-gate counts, stale counts):
+
+```terminal
+docker compose exec web ./manage.py labels_status
+```
+
+Tuning variables in `.env`: `LABELING_TASK_BATCH_SIZE`, `LABELING_LLM_CONCURRENCY_LIMIT`, `LABELING_GATE_BATCH_SIZE`, `LABELING_JOB_PRIORITY`, and the optional prompt overrides `LABELING_SYSTEM_PROMPT` and `LABELING_GATE_SYSTEM_PROMPT`. Use `LLM_LABELING_MODEL` to run labeling on a different model than the other features.
+
+## Extraction Jobs
+
+Users create extraction jobs through a wizard (see the [User Guide](user-guide.md#6-extractions)). Administrators verify and monitor them.
+
+### Verifying Jobs
+
+Jobs created by regular users start in the **Unverified** state and are not processed until a staff user opens the job page and clicks **Verify Job**. Jobs created by staff users are queued immediately. Staff users see all users' jobs by appending `?all=` to the job list URL.
 
 ### Managing Extractions
 
-1. **Access Django Admin**: Navigate to **Extractions** → **Extraction Jobs**
-2. **Monitor Status**: View jobs by status (Preparing, Pending, In Progress, Success, Failure, Canceled)
-3. **View Details**: Click on a job to see:
-   - Owner and group
-   - Query and filters
-   - Output field
-   - Results data
+1. **Access Django Admin**: Navigate to **Extractions** → **Extraction jobs**
+2. **View Details**: Click on a job to see its owner and group, query and filters, status and message, and its output fields
+3. **Extraction tasks** show the individual batches of a job together with the extracted data per report
 
-### Granting Urgent Priority Permission
+Each job may cover at most 25,000 reports; the wizard refuses larger result sets.
 
-Users with urgent priority permission can skip the queue:
+### Granting Urgent Priority Permission
 
-1. Navigate to **Authentication and Authorization** → **Groups**
-2. Edit the desired group
-3. Add permission: `extractions | extraction job | Can process urgently`
-4. Save
+The permission `extractions | extraction job | Can analyze urgently` allows a job to be processed before regular jobs. The `urgent` flag itself can currently only be set on a job in Django Admin.
 
 ## Subscription Management
 
-Administrators can view and manage all user subscriptions through the Django Admin interface.
+Subscriptions are refreshed at the top of every hour. Each refresh looks at the reports created or updated since the previous refresh, applies the subscription's filters, and — if the subscription has filter questions or extraction fields — sends each report to the LLM on the `llm` queue. Users need the permission `subscriptions | subscription | Can add subscription` and an active group to create subscriptions.
 
 ### Managing Subscriptions
 
 1. **Access Django Admin**: Navigate to **Subscriptions** → **Subscriptions**
-2. **View All Subscriptions**: Click on a subscription entry to view its full details.
-3. **Delete a Subscription**: Subscriptions can be removed if necessary using the Delete action
+2. **View All Subscriptions**: Click on a subscription entry to view its full details, including its owner, group, and "last refreshed" timestamp. Changing "last refreshed" changes which reports the next refresh treats as new
+3. **Delete a Subscription**: Subscriptions can be removed if necessary using the Delete action. This also deletes the inbox items collected so far
+
+Staff users can open any user's inbox and its CSV export from the subscription list.
+
+## Background Workers and Crash Recovery
 
-### Urgent Priority Permission
+Extraction, subscription, and labeling jobs are processed by the `default_worker` and `llm_worker` services; embeddings by `embeddings_worker`. Check their logs (`docker compose logs llm_worker`) when jobs do not progress.
 
-1. Navigate to **Authentication and Authorization** → **Groups**
-2. Edit the desired group
-3. Add permission: `subscriptions | subscription | Can process urgently`
-4. Save
+When a worker container is killed mid-task (crash, out-of-memory, redeploy), the affected tasks are repaired automatically: each worker sweeps stale tasks on startup, and a periodic sweep (`ANALYSIS_SWEEP_CRON`, default every minute) requeues tasks whose worker has been silent for longer than `ANALYSIS_STALLED_WORKER_GRACE_SECONDS` (default 30; never set it lower). No manual step is needed after a deploy; to run the sweep by hand:
+
+```terminal
+docker compose exec web ./manage.py sweep_stale_tasks
+```
 
 ## System Announcements
 
diff --git a/docs/user-docs/user-guide.md b/docs/user-docs/user-guide.md
index 5ef43e173..890e3fb07 100644
--- a/docs/user-docs/user-guide.md
+++ b/docs/user-docs/user-guide.md
@@ -8,9 +8,10 @@ RADIS acts as a centralized repository for radiology reports, providing powerful
 
 1. **Reports are stored** with structured metadata (patient info, study details, modalities, etc.)
 2. **You search** using natural language queries with optional filters
-3. **RADIS processes** your query using advanced search algorithms (BM25, semantic, or hybrid search)
+3. **RADIS processes** your query with full-text search, combined with semantic (meaning-based) search when your administrator has configured it
 4. **You receive** ranked results matching your criteria
 5. **You organize** reports into collections, add notes, and subscribe to searches
+6. **AI assists you** by labeling reports, extracting structured data, and answering questions about a report
 
 ## Functionalities Overview
 
@@ -18,9 +19,9 @@ When you log into RADIS, you'll see the main/homepage with several sections:
 
 - **Search**: Search for reports using text queries and filters
 - **Collections**: Organize reports into custom collections
-- **Subscriptions**: Set up notifications for new matching reports
+- **Subscriptions**: Get notified about new reports that match your criteria, with AI-based filtering and data extraction
 - **Notes**: View and manage notes you've added to reports
-- **Extractions**: AI-powered analysis and filtering
+- **Extractions**: Extract structured data from many reports at once using AI
 - **Chats**: Interactive chat interface with AI
 
 ## Main Features
@@ -31,28 +32,53 @@ The search feature is the core of RADIS, allowing you to find relevant reports q
 
 1. Navigate to the "Search" section
 2. Enter your search query in the search box
-3. Apply filters as needed (language, modality, date range, patient demographics)
+3. Apply filters as needed on the right-hand side
 4. Review the search results
 
 #### Search Syntax
 
 - **Case-insensitive**: Search terms are not case-sensitive
-- **AND queries**: All terms must match (implicit AND between terms)
-- **OR operator**: Use capital OR between terms (e.g., "fracture OR lesion")
-- **Phrases**: Use quotes for exact matches (e.g., "lung cancer")
-- **Exclusion**: Use minus sign to exclude terms (e.g., "-metastasis")
+- **AND queries**: All terms must match (implicit AND between terms, or write `AND` explicitly)
+- **OR operator**: Use capital `OR` between terms (e.g., `fracture OR lesion`)
+- **NOT operator**: Use capital `NOT` to exclude a term (e.g., `pneumonia NOT metastasis`)
+- **Phrases**: Use quotes for exact matches (e.g., `"lung cancer"`)
+- **Grouping**: Use parentheses to combine operators (e.g., `(fracture OR lesion) NOT metastasis`)
+
+Operators must be written in capital letters; `or`, `and`, and `not` in lowercase are treated as ordinary search terms. A minus sign in front of a term does not exclude it, and `field:value` syntax is not supported — use the filters instead.
+
+If your query contains mistakes such as unbalanced quotes or parentheses, RADIS repairs it and shows a "Fixed invalid query" notice above the results with the query it actually ran.
+
+#### Semantic Search
+
+When your administrator has configured an embedding model, every search combines classic keyword matching with semantic search, which also finds reports that describe the same thing in different words. This happens automatically; there is no switch to turn it on or off. Each result shows the scores that determined its ranking.
 
 #### Available Filters
 
-- **Language**: Filter by report language
+- **Language**: Filter by report language. The default "All" searches reports in every language
 - **Modalities**: Filter by imaging modality (CT, MRI, X-ray, etc.)
-- **Study Date**: Filter by date range
-- **Study Description**: Filter by study description
-- **Patient Sex**: Filter by patient sex (M/F/U)
-- **Patient Age**: Filter by age range
-- **Patient ID**: Search for specific patient ID
+- **Labels**: Only show reports that carry at least one of the selected labels (see [Labels](#2-labels)). This filter only appears when labels have been set up
+- **Study Date**: Filter by date range (from/till)
+- **Study Description**: Filter by study description (partial match)
+- **Patient Sex**: All, Male, or Female
+- **Age Range**: Filter by patient age using the range slider (in steps of 10 years)
+
+Use "Reset filters" to clear all filters at once.
+
+#### Search Results
+
+Each result shows the patient age and sex, modalities, and study description, followed by a highlighted excerpt of the report. Click "Show full report" to read the entire text. Each result panel has buttons to open the report details, add the report to a collection, add a note, start a chat about the report, and open the study in your PACS viewer.
+
+### 2. Labels
+
+Administrators can define labels (e.g. "Pulmonary nodule" or "Fracture") that RADIS assigns to reports automatically using AI. Each label is classified as Present, Likely, or Possible when the report supports it; reports where the label is absent or not mentioned carry no badge.
+
+- Labels appear as badges on the report detail page, grouped by their label group. Hover over a badge to see whether the finding is Present, Likely, or Possible
+- A greyed-out badge means the report or the label definition changed since the label was assigned; it will be refreshed by the next labeling run
+- Use the **Labels** filter in the search to find all reports carrying a label. Selecting several labels finds reports that carry any of them
+
+Labels cannot be edited by users. If a label seems wrong or missing, contact your administrator.
 
-### 2. Collections
+### 3. Collections
 
 Collections allow you to organize reports for easy access:
 
@@ -67,7 +93,7 @@ Collections are useful for:
 - Creating teaching file collections
 - Organizing reports by project or study
 
-### 3. Notes
+### 4. Notes
 
 Add personal notes to reports for additional context:
 
@@ -76,37 +102,60 @@ Add personal notes to reports for additional context:
 3. View all your notes in the "Notes" section
 4. Notes are private to your user account
 
-### 4. Subscriptions
+### 5. Subscriptions
 
-Set up subscriptions to be notified when new reports match your criteria:
+Set up subscriptions to be notified when new reports match your criteria. RADIS checks for new reports every hour. Optionally, an AI model screens each new report with your questions and extracts data from it:
 
 1. Navigate to the "Subscriptions" section
-2. Click "Add Subscription" to set up a new subscription
-3. Enter the Subscription Name and add filter questions.
+2. Click "Add Subscription"
+3. Enter a **Name** for the subscription
+4. Narrow down the reports with the filters on the right: Patient ID, Language, Modalities, Study Description, Patient Sex, and Age Range
+5. Optionally add up to three **Filter Questions**. Each is a yes/no question about the report (e.g. "Does the report describe a new pulmonary nodule?") together with the answer that should be accepted (Yes or No). A report only enters your inbox when the AI's answer to every question matches
+6. Optionally add up to ten **Extraction Fields** to have data extracted from every matching report. Each field has a Name, a Description telling the AI what to extract, and an Output Type (Text, Numeric, Boolean, or Selection with a fixed list of options). Use the `[ ]` toggle next to the type to collect multiple values per report
+7. Check "Notify me via mail of new reports" if you want an email whenever a refresh finds new reports
+8. Save the subscription
+
+Only reports that arrive (or are updated) after the subscription was created are considered; existing reports are not searched retroactively.
+
+#### Inbox
+
+The subscription list shows how many reports each subscription has collected, with a badge for reports you have not looked at yet. Click the count to open the inbox:
+
+- Each matched report is shown with its header, a preview of the text, and — if you defined extraction fields — the extracted values
+- Sort by arrival or study date, and narrow the list with the filters on the side (Patient ID, Study Description, Study Date, Modalities)
+- Click "Download Extractions as CSV" to export the extracted values of the listed reports as a spreadsheet
+- Opening the inbox marks its reports as seen
+
+Use the "Edit" and "Delete" buttons on the subscription page to change or remove a subscription. Changes apply to future refreshes only; reports already in the inbox stay there.
+
+### 6. Extractions
+
+Extraction jobs let you pull structured data out of many reports at once. Creating a job takes three steps:
+
+**Step 1 – Define fields.** Add one to five fields that you want extracted from each report. For each field, specify:
+
+- **Name**: A short column name for the results
+- **Description**: What the AI should extract (e.g. "Largest diameter of the primary tumor in mm")
+- **Output Type**: Text, Numeric, Boolean, or Selection. For Selection, enter the allowed options (up to seven)
+- The **`[ ]` toggle** next to the type switches the field to an array, so multiple values of that type are returned per report (e.g. a list of all affected vertebrae)
+
+**Step 2 – Search query.** Enter a job title. RADIS generates a search query from your field descriptions, which you can edit or replace. Apply filters (Language, Modalities, Study Date, Study Description, Patient Sex, Age Range) to narrow down the reports. A live counter shows how many reports the job will process; a job may cover at most 25,000 reports. Click "Preview Search Results" to open the matching reports in the regular search view in a new tab.
 
-   - Write each question and select Accept when answer is Yes or No.\
-   - Add extraction fields by specifying the following for each field:
-     - Name
-     - Output Type (Boolean, Text, or Numeric)
-     - Description
+**Step 3 – Review & submit.** Check the summary, optionally tick "Notify me via Email when job is finished", and click "Create Extraction Job".
 
-4. Choose to receive email notifications
-5. New matching reports will be tracked and you'll receive notifications
+Jobs of regular users start in the "Unverified" state and are queued once an administrator has verified them. On the job page you can follow the progress of the individual tasks and, once results come in:
 
-### 5. Extraction
+- Click **View Results** to browse the extracted values in a table with one column per field
+- Click **Download CSV** to export the table
+- Use the control panel to cancel a running job, resume a canceled one, or retry failed tasks
 
-- Create extraction jobs to analyze multiple reports.
-- Add extraction fields by specifying the following for each field:
-  - Name
-  - Output Type (Boolean, Text, or Numeric)
-  - Description
-- In the next step, search query will be automatically generated based on these fields, which you can then review and refine.
+The "Extractions" menu item opens the wizard directly; click "Previous Jobs" in the wizard (or "Job List" on a job page) to see all your jobs.
 
-### 6. Chats (AI Assistant)
+### 7. Chats (AI Assistant)
 
 If enabled, the Chats feature provides an interactive AI assistant:
 
-- Ask questions about reports in natural language
+- Start a chat from a report panel to ask questions about that report in natural language
 - Get contextual answers based on report content
 - Useful for exploring report data interactively
 

From 3ecfc7fb7f93ebb6ea20a8ca69c23da5d0312e15 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Wed, 26 Aug 2026 14:01:38 +0000
Subject: [PATCH 2/7] Document the production install flow and required .env
 variables

The install section told admins to run stack-deploy right after copying
example.env, which fails: stack-deploy exits unless ENVIRONMENT=production,
and it never builds anything but deploys the image named by RADIS_IMAGE.

Introduce the production folder (an example name, use your own), pull the
image before deploying, and list the variables production must set, grouped
by secrets, hosts, LLM, SSL, email and backups, with the CLI generators for
each secret and the no-quotes rule Swarm imposes. Fix the update steps to
match: the checkbox is "Maintenance", keep STACK_NAME unchanged, and the
deploy starts the pulled image rather than rebuilding.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 docs/user-docs/admin-guide.md | 40 ++++++++++++++++++++++++++---------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md
index c3d09fe73..324ea5c6b 100644
--- a/docs/user-docs/admin-guide.md
+++ b/docs/user-docs/admin-guide.md
@@ -4,30 +4,50 @@ The Admin Guide is intended for system administrators and technical staff respon
 
 ## Installation
 
+On the server RADIS lives in a production folder that holds the checkout and the `.env` file. The name is up to you; `radis_prod` below is only an example, use whatever you called your production folder wherever this guide says "production folder":
+
 ```terminal
-Clone the repository: git clone https://github.com/openradx/radis.git
-cd radis
+git clone https://github.com/openradx/radis.git radis_prod
+cd radis_prod  # or whatever you named your production folder
 uv sync
-cp ./example.env ./.env  # copy example environment to .env
-uv run cli stack-deploy  # builds and starts Docker containers
+cp ./example.env ./.env  # set ENVIRONMENT=production and adjust the variables (see below)
+uv run cli compose-pull  # pulls the Docker image (RADIS_IMAGE, default ghcr.io/openradx/radis:latest)
+uv run cli stack-deploy  # starts the Docker Swarm stack
 ```
 
-Before the first deployment, edit `.env` and set at least the secrets, `DJANGO_ALLOWED_HOSTS`, and the LLM endpoint (see [LLM and Embeddings Configuration](#llm-and-embeddings-configuration)). RADIS refuses to start without `LLM_BASE_URL` and `LLM_DEFAULT_MODEL`.
+`stack-deploy` refuses to run unless `ENVIRONMENT=production` is set in `.env`. It does not build anything; the stack runs the pre-built image named by `RADIS_IMAGE`.
+
+### Environment Variables
+
+All settings are read from `.env`; the comments in `example.env` describe every variable. For production at least set:
+
+- `ENVIRONMENT=production`
+- Secrets: `DJANGO_SECRET_KEY`, `POSTGRES_PASSWORD`, `TOKEN_AUTHENTICATION_SALT`, `SUPERUSER_PASSWORD`, `SUPERUSER_AUTH_TOKEN` (generate with `uv run cli generate-django-secret-key`, `generate-secure-password`, `generate-auth-token`)
+- Hosts: `DJANGO_ALLOWED_HOSTS`, `DJANGO_CSRF_TRUSTED_ORIGINS`, `SITE_DOMAIN`, `SITE_NAME`
+- LLM: `LLM_BASE_URL`, `LLM_DEFAULT_MODEL` (both required, RADIS refuses to start without them; see [LLM and Embeddings Configuration](#llm-and-embeddings-configuration)). The `host.docker.internal` default from `example.env` is a development convenience that is not injected into the production stack, so point `LLM_BASE_URL` at an endpoint the stack's nodes can reach. `EMBEDDINGS_MODEL` is optional; leave it empty for full-text search only
+- SSL: `SSL_SERVER_CERT_FILE`, `SSL_SERVER_KEY_FILE`, `SSL_SERVER_CHAIN_FILE` (`uv run cli generate-certificate-chain` builds the chain from your CA-signed certificate)
+- Email: `DJANGO_EMAIL_URL`, `DJANGO_SERVER_EMAIL`, `DJANGO_ADMIN_EMAIL`, `DJANGO_ADMIN_FULL_NAME`, `SUPPORT_EMAIL`
+- Folders: `BACKUP_DIR` (see [Backups](../backups.md))
+
+Optional tuning: `RADIS_IMAGE` and `STACK_NAME` (a second stack such as staging on the same host), `BACKUP_CRON` and `BACKUP_ENABLED`, `TIME_ZONE`, the `EMBEDDINGS_*` group (see [Embeddings](#embeddings-hybrid-search)), the `LABELING_*` group (see [Auto-Labeling](#auto-labeling)), `ANALYSIS_STALLED_WORKER_GRACE_SECONDS` and `ANALYSIS_SWEEP_CRON` (see [Background Workers](#background-workers-and-crash-recovery)), `OTEL_EXPORTER_OTLP_ENDPOINT`, and `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` behind a proxy.
+
+!!! warning "No quotes in .env"
+    Values must not be wrapped in quotes; Docker Swarm treats them as part of the value, and `stack-deploy` refuses to run when it finds any.
 
 ## Updating RADIS
 
 Follow these steps to safely update your RADIS installation:
 
 1. **Verify no active jobs**
-2. **Enable maintenance mode**: In Django Admin, navigate to **Common** → **Project Settings** and check the "Maintenance mode" checkbox, then save
-3. Navigate to Production folder
+2. **Enable maintenance mode**: In Django Admin, navigate to **Common** → **Project settings**, check "Maintenance" and save
+3. **Navigate to the production folder** (e.g. `radis_prod`, or whatever you named it)
 4. **Backup database**: Run `uv run cli db-backup` to create a database backup
 5. **Remove stack**: Run `uv run cli stack-rm` to remove all Docker containers and services
 6. **Pull latest changes**: Run `git pull origin main` to fetch the latest code updates
-7. **Update environment**: Compare `example.env` with your `.env` file and add any new environment variables or update changed values. Pay particular attention to the `LLM_*`, `EMBEDDINGS_*`, `LABELING_*`, and `ANALYSIS_*` groups, which are explained below
+7. **Update environment**: Compare `example.env` with your `.env` file and add any new environment variables or update changed values. Pay particular attention to the `LLM_*`, `EMBEDDINGS_*`, `LABELING_*`, and `ANALYSIS_*` groups, which are explained below. Keep `STACK_NAME` unchanged, otherwise a second stack is deployed next to the old one
 8. **Pull Docker images**: Run `uv run cli compose-pull` to download the latest Docker images
-9. **Deploy stack**: Run `uv run cli stack-deploy` to rebuild and start all services with the updated code
-10. **Disable maintenance mode**: In Django Admin, navigate to **Common** → **Project Settings** and uncheck the "Maintenance mode" checkbox, then save
+9. **Deploy stack**: Run `uv run cli stack-deploy` to start all services with the updated image
+10. **Disable maintenance mode**: Uncheck "Maintenance" in **Project settings** and save
 
 Depending on what changed, one of these follow-up steps may be needed after the stack is up:
 

From 5759983ca809b2e0b7eb1a4d47b48fd7a6f409b8 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Wed, 26 Aug 2026 14:05:00 +0000
Subject: [PATCH 3/7] Fix the remaining admin guide claims that no longer match
 the code

Several statements described behaviour that does not exist or works
differently:

- "Can analyze urgently" is gone from extraction jobs (ExtractionJob.Meta
  overrides the base permissions) and is checked nowhere; only the urgent
  flag matters, and only before the job is queued
- "?all=" is falsy; staff need "?all=1"
- LLM_API_KEY is optional, not required
- collections are not filtered by the active group
- the subscription list is owner-only; staff reach other inboxes by URL,
  and refreshes use the owner's current active group
- embeddings_worker runs no stale-task sweep; init runs retry_stalled_jobs
- the announcement is shown to anonymous visitors too
- the token page is "Manage API Tokens", expiry is a fixed choice list
  with "Never" behind a permission, and the reports API is staff-only
- saving any label field makes its results stale, group edits only
  stale the gate answers, and the scan does not pick up label changes

The docker compose exec snippets did not work against a Swarm stack, so
they are replaced by one "Running Management Commands" subsection that
shows the docker exec form. Add the missing Email section, point the
update procedure at the backups page, and mention the pgsearch admin
pages and the labeling checkpoint admin.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 docs/user-docs/admin-guide.md | 72 +++++++++++++++++++++--------------
 1 file changed, 43 insertions(+), 29 deletions(-)

diff --git a/docs/user-docs/admin-guide.md b/docs/user-docs/admin-guide.md
index 324ea5c6b..2fbe26360 100644
--- a/docs/user-docs/admin-guide.md
+++ b/docs/user-docs/admin-guide.md
@@ -34,6 +34,27 @@ Optional tuning: `RADIS_IMAGE` and `STACK_NAME` (a second stack such as staging
 !!! warning "No quotes in .env"
     Values must not be wrapped in quotes; Docker Swarm treats them as part of the value, and `stack-deploy` refuses to run when it finds any.
 
+### Email
+
+RADIS sends mail for finished jobs, account registrations awaiting approval, subscription notifications, and critical server errors:
+
+- `DJANGO_EMAIL_URL`: SMTP server as a URL, e.g. `smtp://user:password@mail.example.com:587?tls=True`. Only used in production; in development mails are printed to the `web` log instead
+- `DJANGO_SERVER_EMAIL`: Sender address of all mail
+- `DJANGO_ADMIN_EMAIL`, `DJANGO_ADMIN_FULL_NAME`: Receive critical error reports and the "account awaiting approval" mails
+- `SUPPORT_EMAIL`: Shown to users as the address to contact for help
+
+Subjects are prefixed with `[RADIS]`. To check the settings, run the `send_test_mail` management command (optionally with a recipient address; it defaults to `DJANGO_ADMIN_EMAIL`).
+
+### Running Management Commands
+
+Some sections below ask you to run a management command (`embed_pending`, `labels_status`, `sweep_stale_tasks`, `send_test_mail`). They run inside the `web` container. In production the stack runs under Docker Swarm, so find the container first:
+
+```terminal
+docker exec -it $(docker ps -q -f name=radis_prod_web | head -n1) ./manage.py labels_status
+```
+
+`radis_prod` is the stack name (`STACK_NAME`, default `radis_prod`). In development `docker compose exec web ./manage.py labels_status` does the same.
+
 ## Updating RADIS
 
 Follow these steps to safely update your RADIS installation:
@@ -41,7 +62,7 @@ Follow these steps to safely update your RADIS installation:
 1. **Verify no active jobs**
 2. **Enable maintenance mode**: In Django Admin, navigate to **Common** → **Project settings**, check "Maintenance" and save
 3. **Navigate to the production folder** (e.g. `radis_prod`, or whatever you named it)
-4. **Backup database**: Run `uv run cli db-backup` to create a database backup
+4. **Backup database**: Run `uv run cli db-backup` to create a database backup (see [Backups](../backups.md))
 5. **Remove stack**: Run `uv run cli stack-rm` to remove all Docker containers and services
 6. **Pull latest changes**: Run `git pull origin main` to fetch the latest code updates
 7. **Update environment**: Compare `example.env` with your `.env` file and add any new environment variables or update changed values. Pay particular attention to the `LLM_*`, `EMBEDDINGS_*`, `LABELING_*`, and `ANALYSIS_*` groups, which are explained below. Keep `STACK_NAME` unchanged, otherwise a second stack is deployed next to the old one
@@ -65,7 +86,7 @@ All AI features (chats, extractions, subscriptions, labeling, query generation)
 Set these in `.env`:
 
 - `LLM_BASE_URL` (required): Base URL of the endpoint, e.g. `http://llm.internal:11434/v1`
-- `LLM_API_KEY`: API key for the endpoint. Many self-hosted providers ignore it, but it must be set
+- `LLM_API_KEY`: API key for the endpoint. Optional; many self-hosted providers ignore it
 - `LLM_DEFAULT_MODEL` (required): The model every feature uses unless overridden
 - `LLM_CHATS_MODEL`, `LLM_QUERY_GENERATION_MODEL`, `LLM_EXTRACTIONS_MODEL`, `LLM_SUBSCRIPTIONS_MODEL`, `LLM_LABELING_MODEL`: Per-feature overrides. Leave blank to use the default model. Useful to spend a stronger model where it matters (e.g. labeling) and a fast one for chat
 
@@ -89,13 +110,11 @@ By default RADIS runs full-text search only. Setting `EMBEDDINGS_MODEL` switches
 - `EMBEDDINGS_QUERY_INSTRUCTION`: Model-specific instruction prefix for search queries (some models, e.g. Qwen3-Embedding, want one)
 - `EMBEDDINGS_QUERY_CACHE_TIMEOUT_SECONDS`: How long a query's vector stays cached (default 900 s). Lower it temporarily after swapping the model, or the old vectors keep serving stale results for up to this long
 
-New and updated reports are embedded automatically by the `embeddings_worker` service. Reports that existed before the model was configured have no vector and are found by full-text search only until you backfill them:
+New and updated reports are embedded automatically by the `embeddings_worker` service. Reports that existed before the model was configured have no vector and are found by full-text search only until you backfill them with the `embed_pending` management command (see [Running Management Commands](#running-management-commands)). It is idempotent and resumable, refuses to start while another backfill is running or `EMBEDDINGS_MODEL` is empty, and takes `--limit` and `--subjob-size` options; `embed_cancel` stops a running backfill.
 
-```terminal
-docker compose exec web ./manage.py embed_pending
-```
+The same is available in Django Admin under **Pgsearch** → **Report search indexes**: the page shows the state of the embedding pipeline, has a **Cancel queued backfill** button, and offers the actions "Enqueue embedding for selected rows (NULL only)" and "Clear embeddings (NULL them)" on selected reports. **Embedding backfill runs** lists past and current backfills.
 
-The command is idempotent and resumable; `embed_cancel` stops a running backfill. When the embedding service is unreachable or rate limiting, searches silently fall back to full-text results and a warning is logged in the `web` service.
+When the embedding service is unreachable or rate limiting, searches silently fall back to full-text results and a warning is logged in the `web` service.
 
 See the [Contributing guide](../dev-docs/contributing.md) for how to run Ollama with a chat and an embedding model, and the troubleshooting sections there for connection problems.
 
@@ -138,7 +157,7 @@ RADIS uses a group-based permission system:
 
 Each user has an **active group** that determines which reports they can currently access:
 
-- Only reports associated with the active group are visible in searches and collections
+- Only reports associated with the active group are visible in searches and report views. Collections are per user and not filtered by group: a report bookmarked while one group was active stays visible in the collection after switching to another
 - This ensures proper data isolation between different departments or projects
 - Users need an active group to create subscriptions and extraction jobs; the job or subscription is bound to that group
 
@@ -146,7 +165,7 @@ Each user has an **active group** that determines which reports they can current
 
 ### Report Import and Management
 
-Administrators can import reports programmatically via the RADIS API or using the RADIS Client library. See the **RADIS Client** section below for details.
+Administrators can import reports programmatically via the RADIS API or using the RADIS Client library. The reports API only accepts tokens of staff users. See the **RADIS Client** section below for details.
 
 Every import or update of a report re-indexes it for search, queues it for embedding (if configured), and marks its labels for re-labeling on the next scan (see below). This also applies to updates that do not change the report text, so avoid re-pushing an unchanged corpus if you want to keep LLM load down.
 
@@ -166,7 +185,7 @@ RADIS can label reports automatically with an LLM. Labels are defined system-wid
 2. Navigate to **Labels** → **Labels** and add labels to the group. Write the description as the definition you would give a radiologist; it is sent to the LLM verbatim
 3. Run a backfill (see below) so that existing reports are labeled
 
-Editing a label description, a group, or its gate question makes the existing results of that label or group **stale**. Stale results keep surfacing (with a greyed-out badge) until they are regenerated. Deactivating a label stops new classifications, but results already assigned remain visible on the reports.
+Saving a label (any field, not only the description) makes its existing results **stale**; saving a group (e.g. its gate question) makes the group's gate answers stale, but not the results of its labels. Stale results keep surfacing (with a greyed-out badge) until a backfill regenerates them; the periodic scan only picks up changed reports, not changed labels. Deactivating a label stops new classifications, but results already assigned remain visible on the reports.
 
 ### Backfill and Periodic Scan
 
@@ -177,11 +196,7 @@ Labeling runs as background jobs on the `llm` queue:
 
 The **Labeling jobs** page shows all jobs with their trigger (Periodic scan or Manual backfill) and status; a job can be canceled from its detail page. **Label results**, **Gate answers**, and **Labeling tasks** are available read-only for inspection. A task ends with status Warning when some of its reports could not be labeled and Failure when none could, which usually points to an LLM outage.
 
-To check the corpus-wide state (checkpoint, per-label and per-gate counts, stale counts):
-
-```terminal
-docker compose exec web ./manage.py labels_status
-```
+To check the corpus-wide state (checkpoint, per-label and per-gate counts, stale counts), run the `labels_status` management command (see [Running Management Commands](#running-management-commands)). The checkpoint alone is also visible in Django Admin under **Labels** → **Labeling scan checkpoints**.
 
 Tuning variables in `.env`: `LABELING_TASK_BATCH_SIZE`, `LABELING_LLM_CONCURRENCY_LIMIT`, `LABELING_GATE_BATCH_SIZE`, `LABELING_JOB_PRIORITY`, and the optional prompt overrides `LABELING_SYSTEM_PROMPT` and `LABELING_GATE_SYSTEM_PROMPT`. Use `LLM_LABELING_MODEL` to run labeling on a different model than the other features.
 
@@ -191,7 +206,7 @@ Users create extraction jobs through a wizard (see the [User Guide](user-guide.m
 
 ### Verifying Jobs
 
-Jobs created by regular users start in the **Unverified** state and are not processed until a staff user opens the job page and clicks **Verify Job**. Jobs created by staff users are queued immediately. Staff users see all users' jobs by appending `?all=` to the job list URL.
+Jobs created by regular users start in the **Unverified** state and are not processed until a staff user opens the job page and clicks **Verify Job**. Jobs created by staff users are queued immediately. Staff users see all users' jobs by appending `?all=1` to the job list URL.
 
 ### Managing Extractions
 
@@ -201,9 +216,9 @@ Jobs created by regular users start in the **Unverified** state and are not proc
 
 Each job may cover at most 25,000 reports; the wizard refuses larger result sets.
 
-### Granting Urgent Priority Permission
+### Urgent Priority
 
-The permission `extractions | extraction job | Can analyze urgently` allows a job to be processed before regular jobs. The `urgent` flag itself can currently only be set on a job in Django Admin.
+A job whose **urgent** flag is set is queued ahead of regular jobs. The flag can only be set in Django Admin, and only takes effect on jobs that are not queued yet (e.g. still Unverified), because the priority is fixed when the job is queued. The permission `Can analyze urgently` that exists on subscription jobs is not checked anywhere.
 
 ## Subscription Management
 
@@ -215,17 +230,15 @@ Subscriptions are refreshed at the top of every hour. Each refresh looks at the
 2. **View All Subscriptions**: Click on a subscription entry to view its full details, including its owner, group, and "last refreshed" timestamp. Changing "last refreshed" changes which reports the next refresh treats as new
 3. **Delete a Subscription**: Subscriptions can be removed if necessary using the Delete action. This also deletes the inbox items collected so far
 
-Staff users can open any user's inbox and its CSV export from the subscription list.
+The subscription list only shows a user's own subscriptions. Staff users can still open any user's inbox and its CSV export by URL (the subscription ID is shown in Django Admin).
+
+A refresh only considers reports of the owner's **current** active group, not the group the subscription was created in. A subscription silently stops collecting reports while its owner has another group active.
 
 ## Background Workers and Crash Recovery
 
 Extraction, subscription, and labeling jobs are processed by the `default_worker` and `llm_worker` services; embeddings by `embeddings_worker`. Check their logs (`docker compose logs llm_worker`) when jobs do not progress.
 
-When a worker container is killed mid-task (crash, out-of-memory, redeploy), the affected tasks are repaired automatically: each worker sweeps stale tasks on startup, and a periodic sweep (`ANALYSIS_SWEEP_CRON`, default every minute) requeues tasks whose worker has been silent for longer than `ANALYSIS_STALLED_WORKER_GRACE_SECONDS` (default 30; never set it lower). No manual step is needed after a deploy; to run the sweep by hand:
-
-```terminal
-docker compose exec web ./manage.py sweep_stale_tasks
-```
+When a worker container is killed mid-task (crash, out-of-memory, redeploy), the affected tasks are repaired automatically: `default_worker` and `llm_worker` sweep stale tasks on startup, and a periodic sweep (`ANALYSIS_SWEEP_CRON`, default every minute) requeues tasks whose worker has been silent for longer than `ANALYSIS_STALLED_WORKER_GRACE_SECONDS` (default 30; never set it lower). Embedding subjobs are not analysis tasks; `embeddings_worker` runs no sweep, and interrupted subjobs are retried by the task queue itself. On every deploy the `init` service additionally runs `retry_stalled_jobs`, which re-enqueues queue jobs left behind by dead workers. No manual step is needed after a deploy; to run the sweep by hand, run the `sweep_stale_tasks` management command (see [Running Management Commands](#running-management-commands)).
 
 ## System Announcements
 
@@ -242,7 +255,7 @@ System administrators can inform users about important updates, maintenance sche
 ### Announcement Display
 
 - Announcements appear prominently on the main/home page
-- All logged-in users will see the announcement when they access RADIS
+- Everyone who opens the home page sees the announcement, including visitors who are not logged in
 
 #### Example Announcements
 
@@ -263,17 +276,18 @@ jobs with multiple output fields. Check out the user guide for more details.
 
 ## RADIS Client
 
-RADIS Client is a Python library for programmatic access to RADIS features without using the web interface.
+RADIS Client is a Python library to create, retrieve, update and delete reports without using the web interface. The reports API is restricted to staff users, so the token must belong to a staff account.
 
 ### Creating API Tokens
 
 To create an API token for programmatic access:
 
-1. **Navigate** to **Token Authentication** by going to **"Profile"** --> **"Manage API Token"**
-2. **Description** & **Expiry Time** : Add a description (optional) and expiry time for the token.
+1. **Navigate** to **Token Authentication** by going to **"Profile"** --> **"Manage API Tokens"**
+2. **Description** & **Expiry Time** : Add a description (optional) and choose an expiry time of 1, 7, 30 or 90 days. "Never" is only offered to users with the permission `token_authentication | token | Can generate never expiring token`.
 3. **Click** on **"Generate Token"**.
 4. This token will only be visible once, so make sure to copy it now and store it in a safe place. As you will not be able to see it again, you will have to generate a new token if you lose it.
 
 ### Revoking Tokens
 
-- **Admins** can revoke tokens by navigating to **Django Admin** --> **Token Authentication**
+- **Users** delete their own tokens on the same **Manage API Tokens** page
+- **Admins** can revoke any token by navigating to **Django Admin** --> **Token Authentication**

From 7de9da41846135e669ed5a50ba5445b63dbe1b93 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Wed, 26 Aug 2026 14:08:50 +0000
Subject: [PATCH 4/7] Fix the user guide claims that no longer match the
 application

Correct button labels ("Add new collection", "Previous Jobs" only in wizard
step 2), the notes flow (the note button is on every report panel, an empty
note deletes it), the chats section (no enable switch, report-less "New
chat", history and clear-all), and the client description (report CRUD by
staff tokens only, no search).

State what the search box does not support: wildcards, field:value tokens
(silently removed), filter-only searches, queries over 200 characters. Note
that a changed label definition is only refreshed by an admin backfill,
that the subscription Patient ID filter currently has no effect, that the
inbox CSV contains only reports with extracted values, and that query
generation can be turned off by the administrator.

Add what was missing: the report details page and patient timeline arrows,
ranking scores and pagination, collection rename/export/delete, removing a
report from a collection, the job Delete/Verify/Restart buttons, task
drill-down, and the lockable Extractions section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 docs/user-docs/user-guide.md | 46 ++++++++++++++++++++----------------
 1 file changed, 25 insertions(+), 21 deletions(-)

diff --git a/docs/user-docs/user-guide.md b/docs/user-docs/user-guide.md
index 890e3fb07..c85e0925a 100644
--- a/docs/user-docs/user-guide.md
+++ b/docs/user-docs/user-guide.md
@@ -15,7 +15,7 @@ RADIS acts as a centralized repository for radiology reports, providing powerful
 
 ## Functionalities Overview
 
-When you log into RADIS, you'll see the main/homepage with several sections:
+After logging in, the menu at the top of every page leads to these sections:
 
 - **Search**: Search for reports using text queries and filters
 - **Collections**: Organize reports into custom collections
@@ -44,9 +44,9 @@ The search feature is the core of RADIS, allowing you to find relevant reports q
 - **Phrases**: Use quotes for exact matches (e.g., `"lung cancer"`)
 - **Grouping**: Use parentheses to combine operators (e.g., `(fracture OR lesion) NOT metastasis`)
 
-Operators must be written in capital letters; `or`, `and`, and `not` in lowercase are treated as ordinary search terms. A minus sign in front of a term does not exclude it, and `field:value` syntax is not supported — use the filters instead.
+Operators must be written in capital letters; `or`, `and`, and `not` in lowercase are treated as ordinary search terms. A minus sign in front of a term does not exclude it, wildcards such as `pneumo*` are not supported (the `*` is dropped), and `field:value` syntax is not supported — the whole `field:value` token is removed; use the filters instead. A query is required (filters alone do not search) and may be at most 200 characters long.
 
-If your query contains mistakes such as unbalanced quotes or parentheses, RADIS repairs it and shows a "Fixed invalid query" notice above the results with the query it actually ran.
+If your query contains mistakes such as unbalanced quotes or parentheses, unsupported characters, or stray operators, RADIS repairs it and shows a "Fixed invalid query" notice above the results with the query it actually ran.
 
 #### Semantic Search
 
@@ -66,14 +66,16 @@ Use "Reset filters" to clear all filters at once.
 
 #### Search Results
 
-Each result shows the patient age and sex, modalities, and study description, followed by a highlighted excerpt of the report. Click "Show full report" to read the entire text. Each result panel has buttons to open the report details, add the report to a collection, add a note, start a chat about the report, and open the study in your PACS viewer.
+Each result shows the patient age and sex, modalities, study description and the scores that determined its rank, followed by a highlighted excerpt of the report. Click "Show full report" to read the entire text. Results are paginated; the page size can be changed below the list. Each result panel has buttons to open the report details, add the report to a collection, add a note, start a chat about the report, and open the study in your PACS viewer.
+
+The report details page shows all metadata of a report (document ID, patient, study, accession number, modalities, additional metadata), its labels, and the full text. The arrows next to the Patient ID open a list of the same patient's earlier reports, later reports, or all reports.
 
 ### 2. Labels
 
 Administrators can define labels (e.g. "Pulmonary nodule" or "Fracture") that RADIS assigns to reports automatically using AI. Each label is classified as Present, Likely, or Possible when the report supports it; reports where the label is absent or not mentioned carry no badge.
 
 - Labels appear as badges on the report detail page, grouped by their label group. Hover over a badge to see whether the finding is Present, Likely, or Possible
-- A greyed-out badge means the report or the label definition changed since the label was assigned; it will be refreshed by the next labeling run
+- A greyed-out badge means the report or the label definition changed since the label was assigned. A changed report is re-labeled by the next nightly run; a changed label definition is only refreshed when an administrator runs a backfill
 - Use the **Labels** filter in the search to find all reports carrying a label. Selecting several labels finds reports that carry any of them
 
 Labels cannot be edited by users. If a label seems wrong or missing, contact your administrator.
@@ -83,9 +85,9 @@ Labels cannot be edited by users. If a label seems wrong or missing, contact you
 Collections allow you to organize reports for easy access:
 
 1. Go to the "Collections" section to view your collections
-2. Click "Create Collection" to create a new collection
-3. Add reports to collections by clicking the collection button on report panels
-4. View collection contents by clicking on the collection name
+2. Click "Add new collection" to create a new collection
+3. Add reports to collections by clicking the collection button on report panels. The same dialog removes a report from a collection again and preselects the collection you used last
+4. View collection contents by clicking on the collection name. A collection page has buttons to rename, export (as an Excel spreadsheet with the report metadata and text), or delete the collection
 
 Collections are useful for:
 
@@ -97,9 +99,9 @@ Collections are useful for:
 
 Add personal notes to reports for additional context:
 
-1. Open a report detail view
-2. Click the notes button to add or edit notes
-3. View all your notes in the "Notes" section
+1. Click the note button on any report panel (in search results, collections, subscription inboxes, or the report details) to add or edit your note
+2. Save a note with empty text to delete it
+3. View and search all your notes in the "Notes" section
 4. Notes are private to your user account
 
 ### 5. Subscriptions
@@ -109,7 +111,7 @@ Set up subscriptions to be notified when new reports match your criteria. RADIS
 1. Navigate to the "Subscriptions" section
 2. Click "Add Subscription"
 3. Enter a **Name** for the subscription
-4. Narrow down the reports with the filters on the right: Patient ID, Language, Modalities, Study Description, Patient Sex, and Age Range
+4. Narrow down the reports with the filters on the right: Language, Modalities, Study Description, Patient Sex, and Age Range. The Patient ID field is shown too but currently has no effect on the refresh
 5. Optionally add up to three **Filter Questions**. Each is a yes/no question about the report (e.g. "Does the report describe a new pulmonary nodule?") together with the answer that should be accepted (Yes or No). A report only enters your inbox when the AI's answer to every question matches
 6. Optionally add up to ten **Extraction Fields** to have data extracted from every matching report. Each field has a Name, a Description telling the AI what to extract, and an Output Type (Text, Numeric, Boolean, or Selection with a fixed list of options). Use the `[ ]` toggle next to the type to collect multiple values per report
 7. Check "Notify me via mail of new reports" if you want an email whenever a refresh finds new reports
@@ -123,7 +125,7 @@ The subscription list shows how many reports each subscription has collected, wi
 
 - Each matched report is shown with its header, a preview of the text, and — if you defined extraction fields — the extracted values
 - Sort by arrival or study date, and narrow the list with the filters on the side (Patient ID, Study Description, Study Date, Modalities)
-- Click "Download Extractions as CSV" to export the extracted values of the listed reports as a spreadsheet
+- Click "Download Extractions as CSV" to export the extracted values as a spreadsheet. Only reports with extracted values are included, so the file is empty for a subscription without extraction fields
 - Opening the inbox marks its reports as seen
 
 Use the "Edit" and "Delete" buttons on the subscription page to change or remove a subscription. Changes apply to future refreshes only; reports already in the inbox stay there.
@@ -139,7 +141,7 @@ Extraction jobs let you pull structured data out of many reports at once. Creati
 - **Output Type**: Text, Numeric, Boolean, or Selection. For Selection, enter the allowed options (up to seven)
 - The **`[ ]` toggle** next to the type switches the field to an array, so multiple values of that type are returned per report (e.g. a list of all affected vertebrae)
 
-**Step 2 – Search query.** Enter a job title. RADIS generates a search query from your field descriptions, which you can edit or replace. Apply filters (Language, Modalities, Study Date, Study Description, Patient Sex, Age Range) to narrow down the reports. A live counter shows how many reports the job will process; a job may cover at most 25,000 reports. Click "Preview Search Results" to open the matching reports in the regular search view in a new tab.
+**Step 2 – Search query.** Enter a job title. Unless your administrator has turned this off, RADIS generates a search query from your field descriptions, which you can edit or replace. Apply filters (Language, Modalities, Study Date, Study Description, Patient Sex, Age Range) to narrow down the reports. A live counter shows how many reports the job will process; a job may cover at most 25,000 reports. Click "Preview Search Results" to open the matching reports in the regular search view in a new tab.
 
 **Step 3 – Review & submit.** Check the summary, optionally tick "Notify me via Email when job is finished", and click "Create Extraction Job".
 
@@ -147,20 +149,22 @@ Jobs of regular users start in the "Unverified" state and are queued once an adm
 
 - Click **View Results** to browse the extracted values in a table with one column per field
 - Click **Download CSV** to export the table
-- Use the control panel to cancel a running job, resume a canceled one, or retry failed tasks
+- Use the control panel to cancel a running job, resume a canceled one, retry failed tasks, or delete the job. Administrators additionally see buttons to verify a job and to restart it entirely
+- Click a task to see its reports and the extracted values per report
 
-The "Extractions" menu item opens the wizard directly; click "Previous Jobs" in the wizard (or "Job List" on a job page) to see all your jobs.
+The "Extractions" menu item opens the wizard directly; click "Previous Jobs" in step 2 of the wizard (or "Job List" on a job page) to see all your jobs. Administrators can lock the Extractions section temporarily, e.g. during maintenance; you then see a notice instead of the wizard.
 
 ### 7. Chats (AI Assistant)
 
-If enabled, the Chats feature provides an interactive AI assistant:
+The Chats feature provides an interactive AI assistant:
 
 - Start a chat from a report panel to ask questions about that report in natural language
-- Get contextual answers based on report content
-- Useful for exploring report data interactively
+- Click "New chat" in the "Chats" section for a general conversation that is not tied to a report
+- The "Chats" section lists your previous chats with automatically generated titles; you can delete a single chat or clear all of them
+- A message may be at most 1,000 characters long
 
 ## RADIS Client
 
-RADIS Client is a Python library that provides programmatic access to RADIS features without using the web interface.
+RADIS Client is a Python library to create, retrieve, update and delete reports without using the web interface. It does not search reports; use the web interface for that.
 
-To use the RADIS Client, you must have an API token provided by an administrator. For instructions on generating an API token, refer to the [Admin Guide](admin-guide.md).
+The reports API is restricted to staff users, so you need an API token of a staff account. For instructions on generating an API token, refer to the [Admin Guide](admin-guide.md#creating-api-tokens).

From b561f3fb0e22f23ca6fa7233ebb9d7c2cfad0a09 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Wed, 26 Aug 2026 14:13:47 +0000
Subject: [PATCH 5/7] Bring the developer docs, README and agent notes up to
 date

The architecture doc predated auto-labeling, hybrid search and worker
crash recovery: add short sections for them, list the actual queues and
tasks (no check_disk_space, plus labeling, scan, sweep and embeddings),
fix the container names, the client description (no search API), the
query parser syntax (NOT, not -term) and the filter list.

README, AGENTS.md and the architecture doc claimed a pg_search extension;
only pg_vector is installed, full-text search is PostgreSQL's own.

Maintenance: drop the JavaScript section (there is no package.json),
replace made-up service and volume names with generic commands, and point
at refresh_search_configs instead of a restart. Backups: name the
BACKUP_* settings, the retention and db-restore.

Exclude the superpowers plans and specs from the mkdocs build instead of
publishing them as orphan pages. Remove the note about the subscription
Patient ID filter from the user guide; that is a code bug to fix, not
behaviour to document.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 AGENTS.md                     |  6 ++--
 README.md                     |  6 ++--
 docs/backups.md               |  4 ++-
 docs/dev-docs/architecture.md | 53 +++++++++++++++++++++++------------
 docs/maintenance.md           | 21 ++++++--------
 docs/user-docs/user-guide.md  |  2 +-
 mkdocs.yml                    |  4 +++
 7 files changed, 58 insertions(+), 38 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 9b1731b8a..1b195b169 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -42,7 +42,7 @@ uv run cli db-backup                 # Backup database
 ### Tech Stack
 
 - **Backend**: Python 3.12+, Django 6.0+, PostgreSQL 17
-- **Search**: pg_vector (semantic), pg_search (full-text), hybrid ranking
+- **Search**: PostgreSQL full-text search (tsvector), pg_vector (semantic), hybrid RRF ranking
 - **Async**: Daphne (ASGI), Django Channels, Procrastinate (task queue)
 - **Frontend**: Django templates, Cotton components, HTMX, Alpine.js, Bootstrap 5
 - **LLM**: External OpenAI-compatible API endpoint
@@ -86,7 +86,7 @@ Analysis operations follow a Job -> Task pattern (similar to ADIT):
 - **default_worker**: General background task processor (Procrastinate queue: `default`)
 - **llm_worker**: LLM-specific task processor (Procrastinate queue: `llm`)
 - **embeddings_worker**: Embedding task processor (Procrastinate queue: `embeddings`)
-- **postgres**: PostgreSQL 17 with pg_vector and pg_search extensions (port 5432)
+- **postgres**: PostgreSQL 17 with the pg_vector extension (port 5432)
 
 ### LLM Endpoint
 
@@ -219,7 +219,7 @@ reports = response.json()
 
 ### Search Not Returning Expected Results
 
-- Check PostgreSQL extensions are installed: `pg_vector`, `pg_search`
+- Check the `vector` extension is installed in PostgreSQL
 - Verify report has `body` text indexed
 - Check search provider is configured in settings
 - Review QueryParser syntax for complex queries
diff --git a/README.md b/README.md
index bb63e0779..f870e1c5b 100644
--- a/README.md
+++ b/README.md
@@ -43,7 +43,7 @@ The detailed documentation of RADIS can be found at <https://openradx.github.io/
 - **Report Chat**: Ask an AI assistant questions about an individual report in natural language.
 - **Bookmarking and Collection Management**: Organize reports into customizable collections with an intuitive bookmarking service for quick access and review.
 - **Custom Report Notes**: Allow users to append personalized notes to reports for additional context or annotations.
-- **API and Python Client**: Programmatic access to reports and search through a REST API and the `radis-client` library.
+- **API and Python Client**: Programmatic access to reports through a REST API and the `radis-client` library.
 
 ## Planned
 
@@ -51,13 +51,13 @@ The detailed documentation of RADIS can be found at <https://openradx.github.io/
 
 ## API Client
 
-[RADIS Client](https://pypi.org/project/radis-client/) is a Python library to search for reports on RADIS in a programmatic way. It also allows admins to feed new reports to RADIS.
+[RADIS Client](https://pypi.org/project/radis-client/) is a Python library for admins to feed reports to RADIS and to retrieve, update or delete them programmatically.
 
 ## Architectural overview
 
 RADIS employs a sophisticated multi-container architecture, optimized for local deployment using Docker Swarm mode—a feature included with all Docker installations. This local-first approach ensures compliance with the strict data security requirements inherent in hospital and research environments where sensitive patient or research data is managed. By leveraging Docker Swarm, RADIS offers seamless scalability, allowing services to be easily adjusted to meet the specific computational demands of the deployment site. To simplify the setup process, RADIS provides intuitive deployment scripts, ensuring accessibility for users with varying levels of technical expertise.
 
-The RADIS web service is built on the robust Django Python framework, with data securely stored in a PostgreSQL database. By default, RADIS harnesses PostgreSQL’s powerful full-text search capabilities, enhanced by the pg_search and pg_vector extensions, to deliver hybrid search functionality. Its modular architecture enables effortless integration with other advanced text and vector search systems, such as Vespa or ElasticSearch, through easily implementable plugins.
+The RADIS web service is built on the robust Django Python framework, with data securely stored in a PostgreSQL database. By default, RADIS uses PostgreSQL's built-in full-text search, combined with vector search through the pg_vector extension when an embedding model is configured. Other search systems, such as Vespa or ElasticSearch, can be plugged in through the search provider interface.
 
 To deliver a dynamic and responsive web interface, RADIS integrates modern JavaScript libraries HTMX and Alpine.js. For resource-intensive operations—such as batch evaluations or filtering large datasets—RADIS relies on Procrastinate, a Python-based distributed task processing library. This ensures long-running tasks are executed efficiently in the background, minimizing disruptions to user workflows while maximizing system performance.
 
diff --git a/docs/backups.md b/docs/backups.md
index adea417bb..a9cdd0406 100644
--- a/docs/backups.md
+++ b/docs/backups.md
@@ -1,3 +1,5 @@
 # Backups
 
-For database backups django-dbbackup app is used. The backups are done every night at 3 am by a periodic task using the dbbackup management command and stored in the `backups` directory which is mounted as a volume. The dbbackup command can also be called manually with `uv run cli db-backup`.
+Database backups are made by the django-dbbackup app. A periodic task runs the `dbbackup` management command every night at 3 am (`BACKUP_CRON`; set `BACKUP_ENABLED=false` to turn it off). Backups are written to the host directory in `BACKUP_DIR`, which is mounted into the containers as `/backups`, and the last 30 are kept.
+
+Run a backup by hand with `uv run cli db-backup`, and restore the latest one with `uv run cli db-restore`. Both need a running `web` container.
diff --git a/docs/dev-docs/architecture.md b/docs/dev-docs/architecture.md
index 5f64b7c39..4627979f9 100644
--- a/docs/dev-docs/architecture.md
+++ b/docs/dev-docs/architecture.md
@@ -12,15 +12,17 @@ RADIS inherits common functionality from **ADIT Radis Shared**, a shared library
 
 The RADIS platform provides report management, advanced search, AI-powered analysis, and collaborative features through coordinated Docker containers. Users access the system via **web browser** or **RADIS Client** (Python library for programmatic access), performing operations such as searching reports, creating collections, managing subscriptions, and analyzing reports with AI.
 
-The system consists of three main components: a Django API server handling web UI and orchestration, a PostgreSQL database storing all persistent data and serving as the task queue, and background workers executing long-running AI operations.
+The system consists of three main components: a Django API server handling web UI and orchestration, a PostgreSQL database storing all persistent data and serving as the task queue, and background workers executing long-running AI operations. In production an `init` service runs migrations, static file collection and superuser creation before the other services start.
+
+The Django apps are `core` (job/task base classes, crash recovery), `reports`, `search` (query parser, provider interface), `pgsearch` (PostgreSQL provider, embeddings), `extractions`, `labels`, `subscriptions`, `collections`, `notes` and `chats`.
 
 ## Backend Architecture
 
 **Django Web/API Server**: Central coordination engine providing REST API endpoints, authentication, user/session management, static assets, and task orchestration. Creates job/task records in PostgreSQL and schedules background work.
 
-**PostgreSQL Database**: System of record storing user accounts, reports, collections, subscriptions, task queue entries, execution history, and search indexes. Uses pg_search and pg_vector extensions for hybrid search capabilities.
+**PostgreSQL Database**: System of record storing user accounts, reports, collections, subscriptions, task queue entries, execution history, and search indexes. Full-text search uses PostgreSQL's built-in `tsvector`; the pg_vector extension stores report embeddings for semantic search.
 
-**Background Workers**: Docker containers polling PostgreSQL for tasks, executing AI-powered extractions and subscription processing using LLMs.
+**Background Workers**: Docker containers polling PostgreSQL for tasks: extractions, subscription processing and labeling with LLMs, and report embeddings.
 
 ### Procrastinate Task Queue System
 
@@ -28,14 +30,15 @@ RADIS uses [Procrastinate](https://procrastinate.readthedocs.io/en/stable/), a P
 
 **RADIS Task Types**:
 
-- **Default Queue**: `process_extraction_job`, `process_subscription_job`, `subscription_launcher` (periodic), `check_disk_space`, `backup_db`
-- **LLM Queue**: `process_extraction_task`, `process_subscription_task` (AI-intensive operations)
+- **Default Queue**: `process_extraction_job`, `process_subscription_job`, `process_labeling_job`, `bulk_index_reports`, and the periodic tasks `subscription_launcher` (hourly), `incremental_label_scan` (nightly), `sweep_stale_tasks_periodic` (every minute), `retry_stalled_jobs` and `backup_db` (both from adit-radis-shared)
+- **LLM Queue**: `process_extraction_task`, `process_subscription_task`, `process_labeling_task` (AI-intensive operations)
+- **Embeddings Queue**: `embed_reports_task`
 
 ## Frontend Architecture
 
 **Web UI**: Server-side rendered with Django templates and HTMX for dynamic interactions. Uses Bootstrap 5 for styling and Alpine.js for interactive components.
 
-**RADIS Client**: Python package (`radis-client`) for programmatic API access, supporting report creation and search operations.
+**RADIS Client**: Python package (`radis-client`) to create, retrieve, update and delete reports through the REST API. There is no search API.
 
 ## Docker Container Architecture
 
@@ -43,15 +46,17 @@ RADIS uses [Procrastinate](https://procrastinate.readthedocs.io/en/stable/), a P
 
 ### Container Types
 
-**Web Container (`radis-web-1`)**: Runs Django application serving web UI and REST API. Ports: 8000 (dev), 80/443 (prod with SSL). Handles authentication, serves static files, enqueues tasks, and manages database connections. In production, runs with 3 replicas for high availability.
+Container names are prefixed with the stack name, e.g. `radis_dev-web-1` in development and `radis_prod_web.1.<id>` under Docker Swarm.
+
+**Web Container (`web`)**: Runs Django application serving web UI and REST API. Ports: 8000 (dev), 80/443 (prod with SSL). Handles authentication, serves static files, enqueues tasks, and manages database connections. In production, runs with 3 replicas for high availability.
 
-**PostgreSQL Container (`radis-postgres-1`)**: PostgreSQL database storing all data (users, reports, collections, subscriptions, tasks, logs, Procrastinate queue). Port 5432. Uses Docker volumes for persistence.
+**PostgreSQL Container (`postgres`)**: PostgreSQL database storing all data (users, reports, collections, subscriptions, tasks, logs, Procrastinate queue). Port 5432. Uses Docker volumes for persistence.
 
-**Default Worker Container (`radis-default_worker-1`)**: Processes background tasks in the default queue (e.g., extraction job preparation, subscription job preparation, periodic subscription launcher, disk space checks, database backups).
+**Default Worker Container (`default_worker`)**: Processes background tasks in the default queue (job preparation for extractions, subscriptions and labeling, the periodic tasks listed above, database backups).
 
-**LLM Worker Container (`radis-llm_worker-1`)**: Executes AI-intensive tasks from the llm queue (extraction tasks, subscription tasks). Uses ChatClient to communicate with the configured LLM endpoint.
+**LLM Worker Container (`llm_worker`)**: Executes AI-intensive tasks from the llm queue (extraction, subscription and labeling tasks). Uses `LLMClient` (`radis.core.utils.llm_client`) to talk to the configured LLM endpoint.
 
-**Embeddings Worker Container (`radis-embeddings_worker-1`)**: Drains the embeddings
+**Embeddings Worker Container (`embeddings_worker`)**: Drains the embeddings
 queue — generating and storing report vectors for hybrid search, including operator
 backfills started by `./manage.py embed_pending` or the admin action.
 
@@ -77,7 +82,7 @@ directly instead of the client truncating a larger vector. `EMBEDDINGS_BASE_URL`
 both when the provider multiplexes models (OpenAI, Ollama, a gateway), while a
 self-hosted vLLM or SGLang serves one model per process and needs the override. The
 service has its own rate-limit gate — a 429 from the embedding gateway must not pause
-inference — and its own worker (`radis-embeddings_worker-1`) draining the `embeddings`
+inference — and its own worker (`embeddings_worker`) draining the `embeddings`
 queue, so a million-report backfill cannot starve extractions.
 
 ## Search Architecture
@@ -86,22 +91,26 @@ RADIS uses a modular search architecture allowing different search providers to
 
 **Search Provider Interface**: Defines search, retrieval, and indexing operations
 
-- **PgSearch Provider**: Default implementation using PostgreSQL full-text search with pg_search and pg_vector extensions
-- **Alternative Providers**: Vespa and ElasticSearch can be integrated through the same interface
+- **PgSearch Provider**: Default implementation using PostgreSQL full-text search and pg_vector
+- **Alternative Providers**: Other engines such as Vespa or ElasticSearch can be integrated through the same interface
 
 **Query Parser**: Parses user queries with support for:
 
-- AND/OR operators
+- `AND`, `OR` and `NOT` operators (uppercase; implicit AND between terms)
 - Phrase search ("exact match")
-- Exclusion (-term)
+- Grouping with parentheses
 - Case-insensitive matching
 
+Unsupported syntax (`-term`, `field:value`, wildcards) is stripped and the corrected query is shown to the user.
+
 **Search Filters**: Applied on top of query:
 
 - Language, modalities, study date range
 - Study description, patient sex, patient age range
-- Patient ID, group access
-- Created after timestamp (for subscriptions)
+- Patient ID, labels, group access
+- Created before/after and updated after timestamps (for subscriptions and labeling)
+
+**Hybrid ranking**: When `EMBEDDINGS_MODEL` is set, the provider runs the full-text query (up to `HYBRID_FTS_MAX_RESULTS` candidates) and a cosine-distance nearest-neighbour query on the report vectors (`HYBRID_VECTOR_TOP_K`), then fuses both lists with reciprocal rank fusion (`HYBRID_RRF_K`). `NOT` terms are removed before the query is embedded, and query embeddings are cached. If the embedding service fails, the search degrades to full-text only.
 
 **Text-search configurations**: reports are indexed with the PostgreSQL configuration for
 their own language, so stemming matches the text — an English report stores "effusion" as
@@ -114,3 +123,11 @@ does no stemming and therefore matches literally. An unset language filter is no
 to subscriptions' "All" choice: the search form's language field is optional with no
 explicit empty option, so a bookmarked or shared search URL missing `language=` — or the
 search page's first, filter-less load — resolves the same way.
+
+## Auto-Labeling
+
+The `labels` app classifies reports against administrator-defined labels. A `LabelGroup` has a Yes/No gate question; a report only gets the group's labels classified when the gate answers Yes. Each `Label` result is one of `PRESENT`, `LIKELY`, `POSSIBLE`, `ABSENT` or `UNMENTIONED`; the first three appear as badges and in the search filter. Work runs as `LabelingJob`/`LabelingTask` through the usual job/task pattern, either as a manual backfill from Django Admin or through the nightly `incremental_label_scan`, which labels reports updated since the `LabelingScanCheckpoint`. A result is stale when its label or report changed after it was generated.
+
+## Worker Crash Recovery
+
+A task left `IN_PROGRESS` by a killed worker is repaired by `radis.core.utils.recovery`: it is reset to `PENDING` and re-enqueued once its worker has sent no heartbeat for `ANALYSIS_STALLED_WORKER_GRACE_SECONDS`. The sweep runs on worker startup (`sweep_stale_tasks`) and periodically (`ANALYSIS_SWEEP_CRON`). A job whose preparation was interrupted is retried by the queue and starts preparation over, discarding half-created tasks. Embedding subjobs are plain queue jobs and rely on Procrastinate retries instead.
diff --git a/docs/maintenance.md b/docs/maintenance.md
index 27a0c0e51..741ff359d 100644
--- a/docs/maintenance.md
+++ b/docs/maintenance.md
@@ -8,11 +8,6 @@ There are different things that can be upgraded:
   - Check outdated Python packages: `uv run cli show-outdated` (check Python section in output)
   - `uv lock --upgrade` will update packages according to their version range in `pyproject.toml`
   - Other upgrades (e.g. major versions) must be upgraded by modifying the version range in `pyproject.toml` before calling `uv lock --upgrade`
-- Javascript dependencies
-  - Check outdated Javascript packages: `uv run cli show-outdated` (check Javascript section in output)
-  - `npm update` will update packages according to their version range in `package.json`
-  - Other upgrades (e.g. major versions) must be upgraded by modifying the version range in `packages.json` before calling `npm update`
-  - After an upgrade make sure the files in `static/vendor` still link to the correct files in `node_modules`1
 - Python and uv in `Dockerfile` that builds the container where RADIS runs in
 - Dependent services in `docker-compose.base.yml`, like PostgreSQL
 - Github Codespaces development container dependencies in `.devcontainer/devcontainer.json` and `.devcontainer/Dockerfile`
@@ -31,8 +26,8 @@ the compose files no longer define:
 uv run cli compose-down -- --remove-orphans
 ```
 
-A model cache volume outlives them and can be reclaimed with
-`docker volume rm radis_dev_models_data`.
+A model cache volume of such a container outlives it; find it with `docker volume ls` and
+remove it with `docker volume rm`.
 
 **Production (Docker Swarm)**: pass `--prune`, which removes services the deployment no
 longer declares. Only do so when deploying the complete set of compose files, since Swarm
@@ -45,14 +40,16 @@ uv run cli stack-deploy -- --prune
 To look first, or to clean up without redeploying:
 
 ```terminal
-docker service ls | grep llm          # e.g. radis_llm_gpu
-docker service rm radis_llm_gpu       # use your stack name if it is not 'radis'
-docker volume rm radis_models_data    # on each node that holds model files
+docker service ls                     # look for a service the compose files do not define
+docker service rm <service>
+docker volume ls                      # and its model volume, on each node that holds one
+docker volume rm <volume>
 ```
 
 ## Search language configs
 
 RADIS reads available text search configs from Postgres (`pg_ts_config`) and auto-maps
 language codes to matching configs (falling back to `simple`). If new dictionaries/configs
-are installed in Postgres, restart RADIS to refresh the config cache, and reindex reports
-to apply the new config to existing data.
+are installed in Postgres, run `./manage.py refresh_search_configs` to clear the cache (no
+restart needed). Existing reports keep their old index until they are saved again; there is
+no reindex command.
diff --git a/docs/user-docs/user-guide.md b/docs/user-docs/user-guide.md
index c85e0925a..d51a1cd6d 100644
--- a/docs/user-docs/user-guide.md
+++ b/docs/user-docs/user-guide.md
@@ -111,7 +111,7 @@ Set up subscriptions to be notified when new reports match your criteria. RADIS
 1. Navigate to the "Subscriptions" section
 2. Click "Add Subscription"
 3. Enter a **Name** for the subscription
-4. Narrow down the reports with the filters on the right: Language, Modalities, Study Description, Patient Sex, and Age Range. The Patient ID field is shown too but currently has no effect on the refresh
+4. Narrow down the reports with the filters on the right: Patient ID, Language, Modalities, Study Description, Patient Sex, and Age Range
 5. Optionally add up to three **Filter Questions**. Each is a yes/no question about the report (e.g. "Does the report describe a new pulmonary nodule?") together with the answer that should be accepted (Yes or No). A report only enters your inbox when the AI's answer to every question matches
 6. Optionally add up to ten **Extraction Fields** to have data extracted from every matching report. Each field has a Name, a Description telling the AI what to extract, and an Output Type (Text, Numeric, Boolean, or Selection with a fixed list of options). Use the `[ ]` toggle next to the type to collect multiple values per report
 7. Check "Notify me via mail of new reports" if you want an email whenever a refresh finds new reports
diff --git a/mkdocs.yml b/mkdocs.yml
index 35cdb990f..17abad1a8 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -7,6 +7,10 @@ repo_name: openradx/radis
 repo_url: https://github.com/openradx/radis
 edit_uri: edit/main/docs/
 
+# Design notes and implementation plans are not part of the published docs.
+exclude_docs: |
+  superpowers/
+
 theme:
   name: material
   palette:

From 074abb13ec9a9dac45d041e35f05ffc196a95300 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Thu, 27 Aug 2026 08:50:41 +0000
Subject: [PATCH 6/7] Clarify the dev container setup and tone down the
 contributing intro

The dev container paragraph read as if the RADIS stack ran inside the dev
container. It uses docker-outside-of-docker, so the containers run on the
host's Docker daemon next to it; say so, and drop the "example project"
template leftover. Replace the "We're excited" opener with a plain thank
you and indent the Development vs Production admonition so it renders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 CONTRIBUTING.md               |  4 ++--
 docs/dev-docs/contributing.md | 18 +++++++++---------
 2 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 1d9dcca89..61e9e7c32 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,6 +1,6 @@
 # Contributing to Our Project
 
-We're excited that you're interested in contributing to our project! This document outlines the
+Thank you for contributing to RADIS. This document outlines the
 guidelines for contributing to our codebase. We follow the Google Python Style Guide to maintain
 consistency and readability across our project.
 
@@ -17,7 +17,7 @@ cp ./example.env ./.env  # adjust the environment variables to your needs
 uv run cli compose-up -- --watch
 ```
 
-The development server of the example project will be started on <http://localhost:8000>
+The development server is then available on <http://localhost:8000>
 
 File changes will be automatically detected and the servers will be restarted. When library
 dependencies are changed, the containers will automatically be rebuilt and restarted.
diff --git a/docs/dev-docs/contributing.md b/docs/dev-docs/contributing.md
index ecefb946b..f036a6200 100644
--- a/docs/dev-docs/contributing.md
+++ b/docs/dev-docs/contributing.md
@@ -6,14 +6,15 @@ consistency and readability across our project.
 Code Style
 We adhere to the Google Python Style [Guide](https://google.github.io/styleguide/pyguide.html).
 
-This repository includes a [Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container).
-If you open the project in VS Code after cloning, you should see a prompt:
+This repository includes a [Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container)
+with Python, uv and the Docker CLI. If you open the project in VS Code after cloning, you should see a prompt:
 
 “Reopen in Dev Container”
 
-Click it, and VS Code will automatically build and open the development environment.
-
-The development server of the example project will be started on <http://localhost:8000>
+Click it, and VS Code builds and opens the development environment. The dev container does not
+run Docker itself: it talks to the Docker daemon of your machine, so the RADIS containers started
+below run on your host next to the dev container. Docker Desktop, OrbStack or a native Docker
+installation is required either way.
 
 ## Getting Started
 
@@ -25,7 +26,7 @@ cp ./example.env ./.env  # adjust the environment variables to your needs
 uv run cli compose-up -- --watch
 ```
 
-File changes will be automatically detected and the servers will be restarted. When library dependencies are changed, the containers will automatically be rebuilt and restarted.
+The development server is then available on <http://localhost:8000>. File changes are detected automatically and the servers restart. When library dependencies change, the containers are rebuilt and restarted.
 
 ### LLM Setup
 
@@ -190,9 +191,8 @@ uv run cli compose-up  # restart containers (migrations run automatically)
 - For major database schema changes, consider backing up first: `uv run cli db-backup`
 
 !!! note "Development vs Production"
-
-**Development**: Use `uv run cli compose-up` for local development
-**Production**: Use `uv run cli stack-deploy` for production deployment with Docker Swarm
+    **Development**: Use `uv run cli compose-up` for local development.
+    **Production**: Use `uv run cli stack-deploy` for production deployment with Docker Swarm.
 
 ## Reporting Issues
 

From 561921587e3759cbaadcd282367e30d709467e27 Mon Sep 17 00:00:00 2001
From: Muhammad Humza Arain <muhammadhumza1000@gmail.com>
Date: Thu, 27 Aug 2026 11:58:49 +0000
Subject: [PATCH 7/7] Require doc updates in the same PR as the code change
 they affect

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T1vUWBPEYrirQFDLQTLxdx
---
 AGENTS.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/AGENTS.md b/AGENTS.md
index 1b195b169..1de639e27 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -155,6 +155,9 @@ Labeling uses the shared core LLM client (`radis.core.utils.llm_client`); its ti
 - **Comments**: only where the code cannot speak for itself; explain *why*, not *what*
 - **No history in comments**: describe the code as it is, not how it changed — that
   belongs in the commit message (docstrings too)
+- **Keep the docs in sync**: when a change adds a feature or alters behaviour that the
+  docs describe (`README.md`, `docs/`, this file, in-app help templates), update them in
+  the same PR
 
 ## Key Dependencies