Export a collection (and optionally the current search results) as an ADIT batch transfer file #311

Motivation

The September 2026 ADIT/RADIS brainstorming asks for two things: get an ADIT batch transfer file for everything inside a collection, and export search results to such a file. Today a user who has collected reports and wants the images has to take the collection xlsx, rename two headers, delete rows without a study UID, split by PACS and add a Pseudonym column (unless allowed to transfer unpseudonymized) before ADIT accepts it. Search results cannot be exported at all. TODO.md:47 has carried "Allow to export collections to ADIT to transfer the corresponding studies" since 2024-02-08 (2d28b4d) without an issue.

Current behaviour

RADIS (30fa750):

• export_collection writes a human-readable sheet with header PACS, Patient ID, Patient Birth Date, Study Date, Study Description, Study Instance UID, Accession Number, Modalities, Content (radis/collections/utils/exporters.py:10-20) and iterates collection.reports.all() with no group scoping and no filtering of blank identifiers (:23). Allow admins to withdraw and restore reports #308 changes that loop to collection.reports.live().
• CollectionExportView serves it at collections/<pk>/export/; staff may export any collection, others only their own (radis/collections/views.py:106-128). The detail page has a single Export button (radis/collections/templates/collections/collection_detail.html:16-20). radis/collections/tests/test_exporters.py:19-29 pins the header, so the ADIT format has to be a second exporter.
• Report.study_instance_uid and accession_number are blank=True (radis/reports/models.py:77-78) and study_instance_uid has no character validator; pacs_aet is a 16-char free-text field (:49).
• Search results are rendered with pagination only (radis/search/templates/search/_search_results.html). SearchView searches with limit=page_size, scoped to the active group (radis/search/views.py:66-83). The fused candidate set is capped at HYBRID_FTS_MAX_RESULTS + HYBRID_VECTOR_TOP_K (~10,100; radis/pgsearch/providers.py:398,416, radis/settings/base.py:541-542), the same cap as Extraction jobs silently process at most ~10,100 reports regardless of how many match #290.

ADIT (f30d498f), what a file must satisfy:

• The parser maps exactly four headers, PatientID, StudyInstanceUID, SeriesInstanceUID, Pseudonym (adit/batch_transfer/parsers.py:9-14); every other column is ignored (adit/core/parsers.py:48-52). Cells are read as strings (:36), rows with all four mapped cells empty are skipped (:54-55), and one invalid row rejects the whole file (:67-72).
• patient_id and study_uid are required (adit/core/models.py:473-484, ModelSerializer at adit/core/serializers.py:17); study_uid must match ^[\d\.]+$ (adit/core/validators.py:31). A row with a PatientID but an empty StudyInstanceUID therefore fails the upload (reasoned from the code, not executed; no test in adit/batch_transfer/tests/test_parsers.py covers it).
• One PatientID per StudyInstanceUID and one Pseudonym per PatientID across rows (adit/batch_transfer/serializers.py:12-34); Pseudonym is required unless the user has batch_transfer.can_transfer_unpseudonymized (:70-73). Rows sharing a study UID are merged into one task; an empty SeriesInstanceUID means the whole study (adit/batch_transfer/parsers.py:43-72).
• Source and destination come from the job form and apply to every task (adit/batch_transfer/forms.py:76-79,155-158; the TODO there, from 2023-09-18, plans moving them into the file). Non-staff users are capped at MAX_BATCH_TRANSFER_SIZE = 500 tasks (adit/settings/base.py:514, forms.py:90).
• ADIT has no HTTP API for creating batch transfer jobs, only DICOMweb (adit/urls.py:34), so a file download is the only channel today.

Proposal

Phase 1 -- collection:

• A second exporter (suggested home radis/reports/utils/, so search can reuse it without importing from collections) that takes an iterable of reports and writes a workbook whose first columns are exactly PatientID, StudyInstanceUID, SeriesInstanceUID, Pseudonym, all as text cells. SeriesInstanceUID and Pseudonym stay empty. Informational columns after those (AccessionNumber, StudyDate, StudyDescription, ModalitiesInStudy, PACS), named as in the batch query export (adit/batch_query/utils/exporters.py:37-50), are safe because ADIT ignores them; a PACS column lets multi-PACS users filter in Excel.
• Pre-validate what ADIT validates: drop rows with an empty patient_id or study_instance_uid, or a UID not matching ^[\d\.]+$; dedupe on (patient_id, study_instance_uid) (RADIS can hold several documents per study); when one UID appears with two patient IDs keep the first and report the rest. Skipped rows go to a second sheet Skipped with document_id and reason. pd.read_excel reads only the first sheet by default (checked with pandas 3.0.6), so ADIT ignores it.
• Do not run escape_formula (radis/core/utils/csv_export.py:22) over the identifier columns; the leading apostrophe would become part of the value ADIT reads. Escape only free-text columns, or omit them.
• Entry point collections/<pk>/export-adit/ with the ownership rule of CollectionExportView, plus a second button on the detail page. A file response cannot carry a flash message, so the button could open an htmx dialog (the page already uses hx-target="#htmx-dialog" for Rename, collection_detail.html:9-12) that shows the counts (studies, skipped rows, rows per PACS) and offers one download link per pacs_aet, because ADIT takes one source per job.
• Builds on Allow admins to withdraw and restore reports #308: use the same live-report queryset.

Phase 2 -- search results, optional:

• A view that takes the same GET parameters as SearchView, rebuilds SearchFilters with group=active_group.pk, and fetches all fused ids through the retrieval provider's retrieve() (radis/pgsearch/providers.py:538-544, document ids without headlines) rather than search() with limit=None, which annotates a headline for every row (:498-525). A link next to the "N reports found" line. The set is bounded by the ~10,100 cap, so the link text must carry the total_relation == "at_least" wording that _search_results.html:4-11 already uses.
• This exports the whole capped result set, not a selection. The sibling brainstorming bullets (multi-select on the search page; create a collection from the search page) give the collection route its search front door; if those land first, phase 2 may be unnecessary.

Open questions

1. Group scoping. Both the current exporter and the proposed one write every report in the collection, while ReportDetailView (radis/reports/views.py:36-39) and can_view_report (radis/reports/templatetags/reports_extras.py:12-17) restrict to the active group. Because this file leaves RADIS with PatientID + StudyInstanceUID, filtering by active group (as search does) seems right; it would also be a behaviour change for the existing export.
2. Does Report.patient_id equal the PatientID at the source PACS? An ETL may store a RIS/HIS patient id instead. RADIS cannot check this; the export can only assume it and say so in the help text.
3. Multi-PACS collections: one file per pacs_aet (dialog links), a zip, or one file with a PACS column and a warning? pacs_aet is free text set by the ETL and not guaranteed to match an ADIT DicomServer.ae_title (adit/core/models.py:136), so RADIS can group but not validate.
4. ADIT's 500-row cap and pseudonym requirement are per instance and per user; RADIS cannot know them. Generic wording ("ADIT may limit the number of studies per job"), or nothing?
5. Pseudonym generation in RADIS: recommended out of scope. RADIS has no pseudonymization concept, and the pseudonym becomes PatientID/PatientName in the transferred images.
6. Is a synchronous export of up to ~10,100 rows acceptable, or should phase 2 be a background job like extractions (radis/extractions/tasks.py:83-98)? Not measured.
7. Is phase 2 wanted at all, or only the collection route?

Implementation notes

Options, cheapest first:

• A. Only document that ADIT ignores unknown columns, so users rename two headers by hand. Does not handle blank UIDs, dedupe, the per-PACS split or the Pseudonym column; every user repeats it.
• B. Phase 1 only. Covers the first bullet; the second bullet goes via multi-select -> collection -> export once those issues land.
• C. B plus phase 2. Small extra: one view, one link, exporter reuse.

Recommendation: B first, C if a direct search export is wanted.

Tests: exact header, identifiers as text cells, a leading-zero id such as 0062115936 survives an openpyxl round-trip, skipped-rows sheet, dedupe, per-PACS grouping, ownership 404, active-group filter. RADIS CI cannot import ADIT's parser; optionally commit a RADIS-generated fixture under adit/batch_transfer/tests/ and parse it with the existing create_batch_file fixture (adit/batch_transfer/tests/test_parsers.py:26-30) so ADIT CI guards the contract.

Docs: docs/user-docs/user-guide.md:90 describes the export button and needs the new one; CLAUDE.md asks for docs in the same PR. Remove TODO.md:47.

In flight: #308 (live-report filter, same exporter tests); the unmerged origin/downloadingExtractions branch (2025-11-05) also changes radis/collections/utils/exporters.py (10 lines). #292 moves the HYBRID_FTS_MAX_RESULTS slice but keeps it, and does not touch the limit is None branch (radis/pgsearch/providers.py:451).

Related

• Extraction jobs silently process at most ~10,100 reports regardless of how many match #290 -- same ~10,100 cap
• Allow admins to withdraw and restore reports #308 -- builds on
• Add a meaningful API for RADIS client #130 -- an API variant (reports -> batch file) belongs there
• Sibling brainstorming issues: Select search results (per page or all matching) and add them to a collection in one step #317; Allow creating a new collection from the report's Collections dialog #316; Start an ADIT batch transfer for the reports of a collection #312 (needs Add a token-authenticated REST endpoint to create and poll batch transfer jobs adit#422)
• Merge Batch Query and Batch Transfer adit#152 -- would change the batch transfer workflow and format
• Add web xlsx editor to batch transfer and batch query view adit#140 -- why ADIT prefers file upload over in-app editing
• adit/batch_transfer/forms.py:155-158 TODO -- a per-row Source column would let a multi-PACS collection be one file