theislab · xushenbo · Apr 17, 2026 · Apr 17, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,113 +8,16 @@ and this project adheres to [Semantic Versioning][].
 [keep a changelog]: https://keepachangelog.com/en/1.0.0/
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
-## [0.2.0]
-
-### Fixed
- - Assigning `.X` to a view of an X-less {class}`~ehrdata.EHRData` (e.g. one created with `layers=` only) no longer raises `TypeError: 'NoneType' object does not support item assignment`. The view is now materialised before the assignment, consistent with how AnnData handles other field modifications on views. ([#233](https://github.com/theislab/ehrdata/pull/233)) @eroell
-
-### Modified
- - {func}`~ehrdata.infer_feature_types` considers integers from 0, ..., n as numeric. It further provides a new argument `binary_as`, to steer if columns 0/1 should be considered numeric or categorical. ([#231](https://github.com/theislab/ehrdata/pull/231)) @eroell
-
-## [0.1.2]
-
-### Added
- - {func}`~ehrdata.io.from_pandas` with `format='long'` provides a new keyword argument `fill_time_gaps` that fills missing timegaps in the common case of integer time steps from 0 to n_timesteps ([#229](https://github.com/theislab/ehrdata/pull/229)) @eroell
-
-### Modified
- - {func}`~ehrdata.dt.mimic_2` column `censor_flg` switched to lifeline's convention with 1=event, 0=censored, before this dataset loader function had them vice versa since the dataset provides them as such originally. ([#227](https://github.com/theislab/ehrdata/pull/227)) @sueoglu
-
-### Fixed
- - {func}`~ehrdata.io.from_pandas` with `format='long'` misordered entries in `.X`/`.layers` with `.obs` if the input df was not sorted for the obs id keys, which is now fixed. ([#228](https://github.com/theislab/ehrdata/pull/228)) @eroell
-
-### Documentation
- - Documentation style polishing ([#223](https://github.com/theislab/ehrdata/pull/223)) @zethson
-
-## [0.1.1]
-
-### Added
- - {func}`~ehrdata.io.omop.setup_connection` can read `.parquet` files. ([#217](https://github.com/theislab/ehrdata/pull/217)) @eroell
-
-### Fixed
- - Sliceing of `EHRData` objects fixed when the backing object is an `AnnData`. ([#218](https://github.com/theislab/ehrdata/pull/218)) @eroell
-
-### Maintenance
- - More concise messages in {func}`~ehrdata.infer_feature_types`. ([#215](https://github.com/theislab/ehrdata/pull/215)) @zethson
-
-
-## [0.1.0]
+## [Unreleased]
 
 ### Added
- - {func}`~ehrdata.move_to_obs` and {func}`~ehrdata.move_to_x` are new helpers for conveniently moving variables from central 2D arrays to the `.obs` field, and vice versa. ([#199](https://github.com/theislab/ehrdata/pull/201)) @eroell
-  - {func}`~ehrdata.dt.physionet2019` as another out-of-the-box, conveniently available dataset with 40'000 ICU stays from the Physionet 2019 challenge. ([#204](https://github.com/theislab/ehrdata/pull/204)) @eroell
- - `time_precision` parameter (`"date"` or `"datetime"`) to {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` for finer temporal granularity control. ([#210](https://github.com/theislab/ehrdata/pull/210)) @eroell
-
-### Fixed
-- {func}`~ehrdata.io.read_h5ad` fixed issues when `backed=True`. ([#199](https://github.com/theislab/ehrdata/pull/199)) @eroell
-- {func}`~ehrdata.io.read_h5ad` fixed bug when `.X` is `None` and `harmonize_missing_features` is `True`. ([#206](https://github.com/theislab/ehrdata/pull/206)) @eroell
-- {func}`~ehrdata.io.omop.setup_obs` with `observation_table="person_visit_occurrence"` now supports multiple visits per patient, creating one row per visit with unique observation IDs, instead of failing with xarray conversion errors with non-unique indices. ([#210](https://github.com/theislab/ehrdata/pull/210)) @eroell
-- OMOP time interval boundaries now use half-open intervals `[start, end)` to prevent duplicate measurements at interval boundaries. ([#210](https://github.com/theislab/ehrdata/pull/210)) @eroell
-
-### Maintenance
-- Support Python3.14 ([#194](https://github.com/theislab/ehrdata/pull/194)) @Zethson
-- Address `FutureWarning`s across multiple places ([#200](https://github.com/theislab/ehrdata/pull/200)) @eroell
-- Enhanced tutorial structure ([#208](https://github.com/theislab/ehrdata/pull/208)) @eroell
-
-### Modified
-- Dataset generator function `ed.dt.ehrdata_blobs` now takes `n_cat_var` and `n_categories` arguments to generate categorical (integer encoded) time series data ([#207](https://github.com/theislab/ehrdata/pull/207)) @sueoglu
-- If `enrich_var_with_feature_info=True` in {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables`, `data_table_concept_ids` not included within the concept table are now mapped from their respective alternate `concept_id` included in the concept_relationship table to retrieve the available feature information. ([#205](https://github.com/theislab/ehrdata/pull/205)) @KilianDahm
-- {func}`~ehrdata.io.omop.setup_variables` and {func}`~ehrdata.io.omop.setup_interval_variables` with use of `"person"` now checks `birth_datetime` for meaningful behaviour and error messages. ([#210](https://github.com/theislab/ehrdata/pull/210)) @eroell
-- {func}`~ehrdata.integrations.vitessce.gen_default_config` provides convenience to generate a config directly from an `EHRData` object, and should be used instead of the previous `ehrdata.integrations.vitessce.gen_config`. ([#211](https://github.com/theislab/ehrdata/pull/211)) @eroell
-
-## [0.0.10]
-
-{class}`~ehrdata.EHRData` drops the `.R` field, and now supports 3D data storage in any slot of `.layers`. See the {doc}`tutorials/getting_started` tutorial for an introduction to this behaviour. In the future, `.X` will be enabled soon for 3D data storage as well.
-
-### Maintenance
-- Enhanced {doc}`tutorials/getting_started` ([#184](https://github.com/theislab/ehrdata/pull/184)) @eroell
-- Move from zarr<3 to zarr>=3 ([#185](https://github.com/theislab/ehrdata/pull/185)) @eroell
-
-### Fixed
-
-### Modified
-- `EHRData` drops the `.R` field in favor of using `.layers` for any 3D data arrays ([#184](https://github.com/theislab/ehrdata/pull/184)) @eroell
-- `EHRData`'s shape property will always return a 3 dimensional shape. If an `EHRData` object has flat arrays only, the third dimension will be 1. ([#184](https://github.com/theislab/ehrdata/pull/184)) @eroell
-- The following functions now take a `layer` argument: {func}`~ehrdata.io.read_csv`, {func}`~ehrdata.io.from_pandas`, {func}`~ehrdata.io.to_pandas`, {func}`~ehrdata.io.omop.setup_variables`, {func}`~ehrdata.io.omop.setup_interval_variables`, {func}`~ehrdata.dt.ehrdata_blobs`, {func}`~ehrdata.dt.physionet2012`. If it is let to its default, `None`, the `.X` field of `EHRData` is used. Since `.X` is 2D in this release, in cases with 3D data, the `layer` argument needs to be used. ([#184](https://github.com/theislab/ehrdata/pull/184)) @eroell
-- {func}`~ehrdata.io.write_zarr` now writes an `EHRData` specific store encoding, with `anndata` as a substore. This change allows to use `AnnData` with its change to consolidated Zarr metadata, and better isolates `AnnData`'s io. ([#185](https://github.com/theislab/ehrdata/pull/185)) @eroell
-- {func}`~ehrdata.io.read_zarr` is adapted to read the new store encoding, and can also deal with `AnnData` stores. ([#185](https://github.com/theislab/ehrdata/pull/185)) @eroell
-
-
-## [0.0.9]
-
-### Maintenance
-- Use custom logger & remove pydata sparse ([#176](https://github.com/theislab/ehrdata/pull/176)) @Zethson
-- Replace figshare with scverse S3 ([#177](https://github.com/theislab/ehrdata/pull/177)) @Zethson
-- Update template to v0.6.0 ([#166](https://github.com/theislab/ehrdata/pull/166)) @Zethson
-
-### Fixed
-- Fix order of `var` created in `ed.io.omop.setup_variables` and `ed.io.omop.setup_interval_variables` ([#179](https://github.com/theislab/ehrdata/pull/179)) @eroell
-
-### Modified
-- Rename `ed.pl.vitessce.gen_config` to `ed.integrations.vitessce.gen_config` ([#181](https://github.com/theislab/ehrdata/pull/181)) @eroell
-- Rename `ed.tl.omop.EHRDataset` to `ed.integrations.torch.OMOPEHRDataset` ([#181](https://github.com/theislab/ehrdata/pull/181)) @eroell
-
-
-## [0.0.8]
-
-### Fixed
-- Update duckdb imports for future ([#157](https://github.com/theislab/ehrdata/pull/157)) @eroell
-
-### Maintenance
-- Private subset method for `EHRData` ([#160](https://github.com/theislab/ehrdata/pull/160)) @eroell
-- Remove `omop` package dependency ([#160](https://github.com/theislab/ehrdata/pull/160)) @eroell
-
-## [0.0.7]
-
-### Fixed
-- Fix tests and Getting Started Notebook ([#155](https://github.com/theislab/ehrdata/pull/155)) @eroell
-
-### Maintenance
-- Update duckdb imports for future ([#155](https://github.com/theislab/ehrdata/pull/155)) @eroell
+- {func}`~ehrdata.io.source.to_ehrdata` converts canonical source tables (diagnosis, therapy, lab test, procedure) into an {class}`~ehrdata.EHRData` object
+- `ehrdata.io.source.adapters.marketscan` — adapter for IBM MarketScan commercial claims (diagnosis, therapy, procedure, patinfo, insurance, provider)
+- `ehrdata.io.source.adapters.lced` — adapter for IBM LCED linked claims-EMR data (diagnosis, therapy, lab test, habit, patinfo)
+- `ehrdata.io.source.adapters.cprd` — adapter for CPRD GOLD UK primary-care data (diagnosis, therapy, lab test, patinfo); supports Read code and prodcode vocabulary translation
+- `ehrdata.io.source.vocab` — vocabulary loaders for NDC→ingredient, RxCUI→ingredient, LOINC, Read code, and prodcode mappings
+- Source reference documentation for MarketScan, LCED, and CPRD GOLD (`docs/sources/`)
+- Tutorial notebooks: CPRD overview, LOINC mapping, ICD mapping, LCED cohort study (`docs/tutorials/`)
 
 ## [0.0.6]
 

diff --git a/docs/api/io_index.md b/docs/api/io_index.md
@@ -5,8 +5,6 @@
     :no-index:
 ```
 
-## General I/O
-
 ```{eval-rst}
 .. autosummary::
     :toctree: io
@@ -19,19 +17,10 @@
     io.write_zarr
     io.from_pandas
     io.to_pandas
-
-```
-
-## OMOP CDM
-
-```{eval-rst}
-.. autosummary::
-    :toctree: io
-    :nosignatures:
-
     io.omop.setup_connection
     io.omop.setup_obs
     io.omop.setup_variables
     io.omop.setup_interval_variables
+    io.source.to_ehrdata
 
 ```
diff --git a/docs/index.md b/docs/index.md
@@ -14,6 +14,14 @@ references.md
 
 ```
 
+```{toctree}
+:caption: 'Data Sources'
+:hidden: true
+:maxdepth: 2
+
+sources/index
+```
+
 ```{toctree}
 :caption: 'Gallery'
 :hidden: true

diff --git a/docs/sources/cprd.md b/docs/sources/cprd.md
@@ -0,0 +1,110 @@
+# CPRD GOLD
+
+CPRD GOLD (Clinical Practice Research Datalink) is a UK primary-care database derived
+from anonymised GP records.  The extract described here is the May 2018 dementia cohort
+(study DS059).  The Python adapter lives at `ehrdata.io.source.adapters.cprd`.
+
+## Raw data
+
+**Format:** Zip archives of tab-delimited `.txt` files.  Each file type (Clinical,
+Referral, Test, Additional, Therapy, Consultation, Practice) is distributed as one
+or more `.zip` files that are read directly without unpacking, using
+`extract.read_zipped_tsvs`.
+
+**Patient identifier:** `patid` — renamed to `patient_id` by the adapter.
+
+**Date format:** `DD/MM/YYYY` — the adapter passes `formats=["%d/%m/%Y"]` to
+`normalize.coerce_date`.
+
+**Scale (May 2018 extract):**
+
+| Category | Count |
+|---|---|
+| Total patients | 6,638,574 |
+| Qualified patients | 6,613,198 |
+| Total recordings | 3,726,733,860 |
+| Total visits | 939,967,599 |
+
+**Coding systems:**
+
+| Domain | Column | Standard |
+|---|---|---|
+| Diagnosis | `medcode` | CPRD internal integer → Read code via `medical.txt` |
+| Read code | `readcode` | Read code V2 (UK primary-care hierarchy) |
+| Drug | `prodcode` | CPRD product dictionary → drug substance via `product.txt` |
+
+CPRD does **not** use ICD codes.  The canonical `dxver` column is always `None` for
+CPRD outputs.
+
+## Source files
+
+| File type | Key columns | Used for |
+|---|---|---|
+| Clinical | patid, eventdate, medcode, enttype, adid | diagnosis, labtest |
+| Referral | patid, eventdate, medcode | diagnosis |
+| Test | patid, eventdate, medcode, enttype, data1–data7 | diagnosis, labtest |
+| Additional | patid, enttype, adid, data1–data7 | labtest (joined with Clinical) |
+| Therapy | patid, eventdate, prodcode, bnfcode, qty, issueseq | therapy |
+| Patient | patid, dobyr, sex, pracid | patinfo |
+| Practice | pracid, region, lcd, uts | (reference; not in canonical output) |
+
+## Vocabulary files
+
+| File | Columns | Loaded by |
+|---|---|---|
+| `medical.txt` | medcode, readcode, desc | `vocab.readcode.load_medical_map` |
+| `product.txt` | prodcode, drugsubstance, strength, … | `vocab.prodcode.load_product_map` |
+| `product.csv` | prodcode, drugsubstance.updated, … | `vocab.prodcode.load_product_map` (preferred; uses `drugsubstance.updated`) |
+
+`product.csv` is the derived file produced by CPRD's Preparation step, which normalises
+multi-ingredient names.  When it is available, `load_product_map` uses
+`drugsubstance.updated` automatically.
+
+## Canonical output tables
+
+| Canonical table | Sources | Notes |
+|---|---|---|
+| `diagnosis` | Clinical, Referral, Test | medcode translated to Read code when `medical_map` provided; `dxver = None` |
+| `therapy` | Therapy | prodcode → drug substance when `product_map` provided; `prescription_date`, `start_date`, `end_date` = NaT; no NDC or RxCUI |
+| `labtest` | Clinical+Additional (inner join on patid/adid/enttype) + Test | `data2`→value, `data3`→unit, `data4`→valuecat; `loinc = None` |
+| `patinfo` | Patient | Three canonical columns only (patient_id, dobyr, sex) |
+
+## Lab test join
+
+CPRD stores lab values across two complementary file types:
+
+- **Additional** files hold the numeric values (`data1`–`data7`) keyed by `(patid, adid, enttype)`
+  but contain no `eventdate`.
+- **Clinical** files hold `eventdate` keyed by the same composite key.
+
+`build_labtest` performs an inner join of Clinical and Additional on
+`(patid, adid, enttype)` to recover `eventdate` for each data row, then unions with
+Test files (which already contain both `eventdate` and data columns).
+
+## Usage
+
+```python
+from ehrdata.io.source.adapters import cprd
+from ehrdata.io.source.vocab import readcode, prodcode
+
+medical_map = readcode.load_medical_map("medical.txt")
+product_map = prodcode.load_product_map("product.csv")   # or product.txt
+
+diag    = cprd.build_diagnosis(clinical, referral, test_data, medical_map=medical_map)
+therapy = cprd.build_therapy(therapy_data, product_map=product_map)
+labs    = cprd.build_labtest(clinical, additional, test_data)
+patinfo = cprd.build_patinfo(patient)
+```
+
+## Notes
+
+- Read codes starting with `R` (Symptoms, Signs and Ill-Defined Conditions) are not
+  disease codes and are typically excluded from phenotype analyses (filter
+  `~dx.str.startswith("R")`).
+- Read codes starting with `ZZ` could not be mapped to any hierarchy in the original
+  extract and should be investigated before use.
+- `issueseq` in the Therapy file is a repeat-prescription counter (1 = first issue,
+  2 = first repeat, etc.).  It is not mapped to the canonical `refill` column.
+- The `practice` table provides region, last-collection date (`lcd`), and
+  up-to-standard date (`uts`) for each GP practice.  These are useful for defining
+  study observation windows but are not included in the canonical `patinfo` output.
diff --git a/docs/sources/index.md b/docs/sources/index.md
@@ -0,0 +1,12 @@
+# Data Sources
+
+Reference documentation for the three source datasets supported by the
+`ehrdata.io.source` layer.
+
+```{toctree}
+:maxdepth: 2
+
+marketscan
+lced
+cprd
+```