Offer the label filter in the subscription form without losing unlabeled reports #326

Motivation

A subscription cannot be restricted to reports carrying a label, although _build_filter_query already supports SearchFilters.labels and the subscription refresh goes through it. Adding the field is small; the non-obvious part is timing. Subscriptions refresh hourly and select by Report.updated_at, while labeling runs daily and does not touch Report.updated_at. A label filter applied naively in the refresh would exclude every report that has not been labeled yet, the last_refreshed watermark would move past it, and it would never be reconsidered. (From the September 2026 ADIT/RADIS brainstorming: "Labels should be in the subscription create page".)

Current behaviour

• Subscription stores its filters as columns and an M2M but has no labels relation (radis/subscriptions/models.py:33-47). SubscriptionForm.Meta.fields and the filters Column list no labels (radis/subscriptions/forms.py:23-33, :67-76). The same form serves the create and update pages, and both call form.save() with commit (radis/subscriptions/views.py:116, :169), so an M2M would persist there.
• The refresh builds SearchFilters with updated_after=last_refreshed and no labels (radis/subscriptions/tasks.py:87-96) and runs it through the uncapped filter() path (radis/pgsearch/providers.py:547-552, see subscription_retrieval_provider is registered but never used #291). _build_filter_query already applies labels (:225-233) and updated_after (:234-235).
• refresh_time is captured before the query and last_refreshed is advanced to it unconditionally (radis/subscriptions/tasks.py:81, :122-126).
• SUBSCRIPTION_CRON is hardcoded hourly (radis/settings/base.py:699); LABELING_SCAN_CRON defaults to 02:00 daily (:749). A SCAN LabelingJob covers updated_at >= scan_from with no upper bound (radis/labels/tasks.py:19-21), the tick is skipped while any job is active (:127-138), and the checkpoint is advanced when the job is created, not when it finishes (:162-163).
• Labeling writes only GateAnswer and LabelResult rows and never saves the Report (radis/labels/labeling.py:77-104); Report.updated_at is auto_now (radis/reports/models.py:82). So "labeled" is not visible through updated_at.
• The detail page lists the filters without labels (radis/subscriptions/templates/subscriptions/subscription_detail.html:47-101).

Proposal

Form and model: Subscription.labels = ManyToManyField(Label, blank=True) plus migration 0012; the field comes from the shared create_label_field() proposed in the extraction issue; add it to the filters column, the detail page and the user guide step (docs/user-docs/user-guide.md:114).

Refresh semantics, cheapest first:

A -- pass labels straight through. Two lines in _build_subscription_job. Loses every report that arrives between a refresh and the next labeling scan. Not acceptable for a feature whose purpose is not to miss reports.

B -- select by label time instead of report time. For a label-filtered subscription, "new" means a surfacing LabelResult for one of the chosen labels was generated at or after last_refreshed. LabelResult.generated_at is auto_now (radis/labels/models.py:61) and is bumped by the update_or_create in _run_label_set (radis/labels/labeling.py:102-104), and a result is always generated after the report's updated_at. Concretely: one new field on SearchFilters (e.g. labeled_after) and one Q clause inside the existing if filters.labels: block in _build_filter_query; _build_subscription_job passes labeled_after instead of updated_after when the subscription has labels. This handles the ingest/label race, reports that failed in one scan and were fixed by a later backfill, and label definition changes, without any LabelingJob bookkeeping. Cost: a manual backfill re-labels everything and re-selects every matching report; accepted ones are deduplicated by SubscribedItem (radis/subscriptions/processors.py:72-78), previously rejected ones re-run the filter questions once.

C -- labeled-through watermark. Add updated_before to SearchFilters (mirroring created_before, radis/search/site.py:68, radis/pgsearch/providers.py:223-224), bound label-filtered refreshes to the created_at of the newest finished SCAN LabelingJob, and advance last_refreshed only to that bound. Weaknesses: a task ends WARNING when some reports failed to label (radis/labels/processors.py:37-46, rolled up in radis/core/models.py:116-138) and later scans never retry them, so counting WARNING jobs loses reports while counting only SUCCESS freezes every label-filtered subscription until a manual backfill; the first tick only sets the checkpoint (radis/labels/tasks.py:140-143) and no tick runs while no label is active (:145-148); and because ETL re-upserts bump updated_at (#257), a full re-sync makes one scan cover the whole corpus and stalls all label-filtered subscriptions for its duration.

D -- per-report readiness. Select only reports not in _needs_work_queryset (radis/labels/scope.py:8-11) and hold last_refreshed back to the oldest unready candidate. Exact, but reports rejected by the LLM filter questions leave no record (radis/subscriptions/processors.py:72-78), so they would be re-sent to the LLM every hour until the watermark moves.

Recommendation: B. It is the smallest change to _build_filter_query, needs no helper in the labels app, and its failure mode (a backfill re-runs filter questions once for rejected reports) is bounded and visible, whereas C trades lost reports for indefinite stalls depending on how job status is counted.

Open questions

1. Under B, does labeled_after replace updated_after for label-filtered subscriptions, or AND with it? Replacing is what makes the race go away; ANDing would reintroduce it.
2. ANY-of (as SearchFilters documents, radis/search/site.py:49-50) or ALL-of? Follows the separate union/intersection bullet.
3. Inactive or deleted labels: the subquery does not check Label.active (radis/pgsearch/providers.py:229-233), so a deactivated label keeps matching through stale results; the M2M cascades on delete. Warn on the detail page, drop inactive labels at run time, or block saving?
4. Latency and visibility: with B a report appears in the inbox only after the next labeling scan, up to a day later, while the user guide promises hourly checks (docs/user-docs/user-guide.md:109). Help text should say so. Should SUBSCRIPTION_CRON become env-configurable at the same time?
5. Editing labels on an existing subscription does not rescan history because last_refreshed is not reset; the help text should say new labels only affect reports labeled after the change. The other filters behave the same today.
6. Should SubscribedItemFilter on the inbox page (radis/subscriptions/filters.py:28) gain a labels filter for parity? Overlaps with the inbox-filters brainstorming bullet; decide in one place.

Implementation notes

• Keep the radis/pgsearch/providers.py change to the one clause so it rebases onto Make the FTS candidate query single-table with a search projection #292, which renames the column to report_updated_at inside _build_filter_query.
• Group scoping is unchanged: labels are system-wide (radis/labels/models.py:27-39) and the set stays bounded by Q(report__groups=filters.group) (radis/pgsearch/providers.py:200).
• Tests: form round-trip with labels; _build_subscription_job passes labels and labeled_after; a report ingested before labeling is picked up by the refresh that runs after labeling (currently untested and the whole point); unchanged behaviour without labels; _build_filter_query with the new field, following radis/labels/tests/test_search_filter.py.
• The same block of _build_subscription_job also omits patient_id (filed separately as a bug); fix that first or together.
• Incidental, same templates: subscription_detail.html:61-68 renders subscription.study_date_from / study_date_till, which do not exist on Subscription, so that row always shows a dash; _subscription_form_body.html is referenced nowhere and still renders a form.filter_fields formset that SubscriptionForm no longer has. Both could go in the same PR.

Related

• Companion issue: Offer the label filter in the extraction wizard's search step #325 (shares the form factory and M2M pattern).
• Bug: Subscription Patient ID filter is never applied when refreshing a subscription #334 (same code block).
• Labeling: report body edits don't invalidate labels — and the obvious updated_at fix would re-label on every ETL re-upsert #257 -- updated_at semantics under ETL re-upserts; the reason option C ties subscription latency to labeling throughput.
• subscription_retrieval_provider is registered but never used #291 -- subscriptions deliberately use the uncapped filter() path; the label filter must stay on it.
• Make the FTS candidate query single-table with a search projection #292 -- rewrites _build_filter_query.
• fix(labels): gate on the scoping question, not the group's display name #293 -- gate schema fix in radis/labels/labeling.py.
• Automatically generate the pre-filter query from extraction fields #138 -- auto-generated pre-filter query for subscriptions is not shipped; a label filter is a deterministic alternative.
• Allow Subscriptions feature to extract data #139 -- extraction fields in subscriptions; appears implemented on main (Subscription.output_fields, migration 0011).
• TODO.md:42 ("make sure categories app are evaluated before subscriptions") anticipated this ordering problem.