diff --git a/OLIDS/Release-notes/1_OLIDS_Share/v1.5.0.md b/OLIDS/Release-notes/1_OLIDS_Share/v1.5.0.md new file mode 100644 index 0000000..6d9d90f --- /dev/null +++ b/OLIDS/Release-notes/1_OLIDS_Share/v1.5.0.md @@ -0,0 +1,25 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v1.4.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/releases/tag/releases%2Fv1.4.0). +## Summary +This release includes an update to the `duckdb__format_date_to_string` function, enhancing its ability to handle postcode version date strings in the `'AUG2026'` format. This improvement increases compatibility with datasets that use this specific date representation, ensuring smoother data processing and formatting. + +### Improvements +- Updated `duckdb__format_date_to_string` to support postcode version date string format (e.g., `'AUG2026'`). [PR: #18](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/18) + > 🎯 ***Impact:*** Enables correct formatting and parsing of postcode version dates, improving data compatibility and reducing potential errors when handling date strings in this format. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 2 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #18](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/18): Updated duckdb__format_date_to_string to handle postcode version date… + - No linked issues. + +[View the diff between v1.4.0 and v1.5.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/compare/releases%2Fv1.4.0...releases%2Fv1.5.0) diff --git a/OLIDS/Release-notes/1_OLIDS_Share/v1.6.0.md b/OLIDS/Release-notes/1_OLIDS_Share/v1.6.0.md new file mode 100644 index 0000000..bfe6be9 --- /dev/null +++ b/OLIDS/Release-notes/1_OLIDS_Share/v1.6.0.md @@ -0,0 +1,25 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v1.5.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/releases/tag/releases%2Fv1.5.0). +## Summary +This release introduces a macro to standardize the handling of regular expression syntax differences between DuckDB and Snowflake. This enhancement simplifies cross-platform SQL development and improves code portability, making it easier to write queries that work seamlessly across both database systems. + +### New Features +- Add macro to handle different syntax for regex like and match between DuckDB and Snowflake. *[PR: [#19](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/19)]* + > ✨ ***Feature:*** Enables consistent usage of regex-based conditions in SQL queries across DuckDB and Snowflake, reducing the need for manual syntax adjustments and improving developer productivity. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 5 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #19](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/19): Add macro to handle different syntax for regex like and match between… + - No linked issues. + +[View the diff between v1.5.0 and v1.6.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/compare/releases%2Fv1.5.0...releases%2Fv1.6.0) diff --git a/OLIDS/Release-notes/1_OLIDS_Share/v1.7.0.md b/OLIDS/Release-notes/1_OLIDS_Share/v1.7.0.md new file mode 100644 index 0000000..eae1df2 --- /dev/null +++ b/OLIDS/Release-notes/1_OLIDS_Share/v1.7.0.md @@ -0,0 +1,53 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v1.6.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/releases/tag/releases%2Fv1.6.0). +## Summary + +This release introduces major improvements to the OLIDS_Share dbt package, focusing on automation, validation, and testing. Key highlights include automated generation of `sources.yml` files for Snowflake schemas, comprehensive NHS number validation macros with adapter-specific implementations, and a robust macro test suite with CI integration. Together, these changes enhance reliability, data quality, and developer productivity for users of the package. + +### New Features +- Automated generation of `sources.yml` files for OLIDS_PID and OLIDS_PSEUDO schemas in Snowflake, using a new dbt macro and PowerShell workflow integrated into the dbt execution process. + *[PR: [#20](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/20)]* + > ✨ ***Feature:*** This streamlines the onboarding of new views and tables, reducing manual effort and ensuring source definitions are always up to date. + +- Added NHS number validation macros, including checks for population, valid format, valid checksum, and pre-1994 format, with full support for both DuckDB and Snowflake adapters. + *[PR: [#22](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/22)]* + > ✨ ***Feature:*** These macros enable robust, adapter-specific NHS number validation, improving data integrity and compliance for downstream models. + +### Improvements +- Introduced a dedicated macro test suite and integration test project, along with a GitHub Actions workflow to automatically run dbt unit tests on DuckDB. + *[PR: [#21](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/21)]* + > 🎯 ***Impact:*** Significantly increases test coverage and CI reliability, ensuring core macros are validated in isolation and do not interfere with package consumers. + +- Added new utility macros: `append_schema_suffix` for dynamic schema naming and `remove_whitespace` for string cleaning, with updates to `format_postcode` for improved postcode formatting. + *[PR: [#21](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/21)]* + > 🎯 ***Impact:*** Enhances flexibility in schema routing and improves string handling, supporting more consistent data transformations. + +### Refactoring +- Reorganized macros into sub-folders based on functionality for better maintainability and clarity. + *[PR: [#21](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/21)]* + > ♻️ ***Refactor:*** Improves code structure, making it easier for developers to locate and manage macros as the package grows. + +- Moved integration tests to an isolated dbt sub-project to avoid dependency bloat and ensure test isolation. + *[PR: [#21](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/21)]* + > ♻️ ***Refactor:*** Keeps the main package lean and prevents test dependencies from affecting consumers. + +### Inputs + +This changelog was automatically produced from: + +- 3 Pull Requests +- 0 Issues +- 0 WorkItems +- 32 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #20](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/20): generate source yml file + - No linked issues. +- [PR #21](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/21): Add macro test suite to repository + - No linked issues. +- [PR #22](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/pull/22): Add nhs number validation macros + - No linked issues. + +[View the diff between v1.6.0 and v1.7.0](https://github.com/NHSISL/LondonDataServices.OLIDS_Share/compare/releases%2Fv1.6.0...releases%2Fv1.7.0) diff --git a/OLIDS/Release-notes/2_EMIS_OLIDS/v2.4.0.md b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.4.0.md new file mode 100644 index 0000000..2799829 --- /dev/null +++ b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.4.0.md @@ -0,0 +1,38 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.3.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/releases/tag/releases%2Fv2.3.0). +## Summary + +This release introduces a new derived appointment status column to the `PC_EMIS_Appointment` model, enhancing future compatibility and clarity in downstream data usage. Logic for identifying referral requests has been streamlined in medication models, reducing ambiguity and improving data consistency. Additionally, the OLIDS_Share dependency has been updated to revision v1.5.0, ensuring alignment with the latest platform features and fixes. + +### New Features +- Add derived column for appointment status and update tests [PR: [#117](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/117)] + > ✨ ***Feature:*** Introduces a new `APPOINTMENT_STATUS_DERIVED` column in the appointments model, preparing for future schema changes and improving clarity in appointment status handling. + +### Improvements +- Update OLIDS_Share to revision v1.5.0 [PR: [#119](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/119)] + > 🎯 ***Impact:*** Ensures the project uses the latest OLIDS_Share features and bug fixes, supporting ongoing compatibility and stability. + +### Bug Fixes +- Remove remaining references to Observation_type=referral [PR: [#118](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/118)] + > 🐞 ***Fix:*** Updates referral request logic in medication models to rely solely on `EMIS_CODE_CATEGORY_DESCRIPTION = 'Referral'`, eliminating ambiguous use of `OBSERVATION_TYPE` and improving data accuracy. + +### Inputs + +This changelog was automatically produced from: + +- 3 Pull Requests +- 0 Issues +- 0 WorkItems +- 8 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #117](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/117): Add derived column for appointment status and update tests + - No linked issues. +- [PR #118](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/118): Remove remaining references to Observation_type=referral + - No linked issues. +- [PR #119](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/119): Update OLIDS_Share to revision v1.5.0 + - No linked issues. + +[View the diff between v2.3.0 and v2.4.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.3.0...releases%2Fv2.4.0) diff --git a/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.0.md b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.0.md new file mode 100644 index 0000000..d2cff99 --- /dev/null +++ b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.0.md @@ -0,0 +1,38 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.4.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/releases/tag/releases%2Fv2.4.0). +## Summary + +This release introduces versioning to the `PC_EMIS_Episode_Of_Care` transformed model, enabling support for historic registrations and improved data lineage. The update also enhances documentation, adds new fields to the patient history model, and refines unit test coverage to ensure ongoing model reliability. Several internal improvements and bug fixes further strengthen data quality and maintainability. + +### New Features +- Added version 2 of the `PC_EMIS_Episode_Of_Care` transform to support historic registrations by consuming data from the `ADMIN_PATIENT_HISTORY` table. Introduced model versioning semantics and updated documentation to reflect changes. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]* + > ✨ ***Feature:*** Users can now access historic registration data, improving the completeness and traceability of episode-of-care records. + +### Improvements +- Enhanced the `PC_EMIS_Episode_Of_Care` model documentation with explicit version tracking and column history. Updated unit tests to ensure they target the correct model version for reliable validation as the model evolves. Added the `PATIENT_TYPE_DESCRIPTION` field to the sequenced admin patient history model and SQL transformation. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]* + > 🎯 ***Impact:*** Improves transparency, test coverage, and data clarity for end users and maintainers. + +### Bug Fixes +- Corrected macro enclosure invocation for `generate_sha256` and resolved issues in model scenarios identified by tests. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]* + > 🐞 ***Fix:*** Ensures accurate hash generation and robust model behavior, reducing the risk of data inconsistencies. + +### Refactoring +- Refactored the `PC_EMIS_Episode_Of_Care` transform by separating state change detection into its own CTE and improving SQL logic clarity. Removed the unversioned model to streamline codebase consistency. *[PR: [#116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116)]* + > ♻️ ***Refactor:*** Enhances maintainability and readability, making future updates and debugging more efficient. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 6 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #116](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/116): add PC_EMIS_Episode_Of_Care (version 2) - historic registrations + - No linked issues. + +[View the diff between v2.4.0 and v2.5.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.4.0...releases%2Fv2.5.0) diff --git a/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.1.md b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.1.md new file mode 100644 index 0000000..9f9bf78 --- /dev/null +++ b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.1.md @@ -0,0 +1,25 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.5.0](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/releases/tag/releases%2Fv2.5.0). +## Summary +This release updates the OLIDS_Share package to version 1.6.0. By referencing the latest package revision, the platform benefits from recent enhancements and bug fixes included upstream, ensuring improved stability and compatibility. + +### Improvements +- Update OLIDS_Share revision to v1.6.0 in `EMIS_OLIDS/packages.yml` to use the latest package version. *[PR: [#121](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/121)]* + > 🎯 ***Impact:*** Ensures the platform leverages the latest features and fixes from the OLIDS_Share package, improving reliability and maintainability. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 2 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #121](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/121): Update OLIDS_Share revision to v1.6.0 + - No linked issues. + +[View the diff between v2.5.0 and v2.5.1](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.5.0...releases%2Fv2.5.1) diff --git a/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.2.md b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.2.md new file mode 100644 index 0000000..395a9c1 --- /dev/null +++ b/OLIDS/Release-notes/2_EMIS_OLIDS/v2.5.2.md @@ -0,0 +1,31 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.5.1](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/releases/tag/releases%2Fv2.5.1). +## Summary + +This release focuses on improving data quality and platform stability for episode-of-care processing. It addresses a critical issue with duplicate and invalid episode records, enhancing the deduplication logic and adding comprehensive unit tests to ensure robust handling of complex data scenarios. Additionally, the OLIDS_Share revision has been updated to v1.7.0, ensuring compatibility with the latest platform standards. + +### Improvements +- Add tests and fix for duplicate episode of care IDs [PR: [#122](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/122)] + > 🎯 ***Impact:*** Improves the episode-of-care rule engine by ensuring closed episodes with differing start/end/type are handled correctly and that duplicate or overlapping episodes are deduplicated. This update enforces unique, valid episodes and adds targeted unit tests, resulting in higher data quality and more reliable downstream analytics. + +- Update OLIDS_Share revision to v1.7.0 [PR: [#123](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/123)] + > 🎯 ***Impact:*** Updates the platform dependency to the latest OLIDS_Share revision, ensuring continued compatibility and access to recent enhancements and fixes. + +### Inputs + +This changelog was automatically produced from: + +- 2 Pull Requests +- 0 Issues +- 0 WorkItems +- 7 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #122](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/122): Add tests and fix for duplicate episode of care IDs + - No linked issues. +- [PR #123](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/pull/123): Update OLIDS_Share revision to v1.7.0 + - No linked issues. + +[View the diff between v2.5.1 and v2.5.2](https://github.com/NHSISL/LondonDataservices.Transform.EMIS_OLIDS/compare/releases%2Fv2.5.1...releases%2Fv2.5.2) diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.0.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.0.md new file mode 100644 index 0000000..1bc2bd1 --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.0.md @@ -0,0 +1,64 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v2.3.2](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/releases/tag/releases%2Fv2.3.2). +## Summary + +This release delivers significant enhancements to OLIDS Enrichment, including improved data quality, new features, and important bug fixes. Highlights include a major overhaul of postcode enrichment logic, the introduction of versioned Episode of Care models, and a new derived appointment status column for better downstream compatibility. Data masking and governance have been strengthened with new schema rules and masking entries for TPP datasets. Multiple bug fixes and refactorings improve data accuracy, maintainability, and system reliability. + +### New Features +- Add versioned Episode of Care models and SQL configurations for OLIDS. *[PR: [#71](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/71)]* + > ✨ ***Feature:*** Supports multiple versions of Episode of Care tables with clear documentation and model separation, enabling smoother upgrades and better alignment with upstream changes. + +- Add APPOINTMENT_STATUS_DERIVED. *[PR: [#72](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/72), WorkItems: [AB#29866](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/29866)]* + > ✨ ***Feature:*** Introduces a new appointment status field for improved clarity and future-proofing, while maintaining backward compatibility for current users. + +### Improvements +- Users/gainhams/postcode hash fix. *[PR: [#68](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/68)]* + > 🎯 ***Impact:*** Migrates postcode enrichment to a reference-driven approach using the UKHFD dataset, enhancing data quality, governance, and maintainability. Updates dependencies, tests, and environment setup for improved reliability. + +- addition of 2 new TPP schemas for data masking. *[PR: [#73](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/73)]* + > 🎯 ***Impact:*** Adds new masking policy entries for TPP Community and Mental Health schemas, strengthening data protection and compliance. + +- masking entries for OLIDS TPP. *[PR: [#76](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/76)]* + > 🎯 ***Impact:*** Introduces masking-control rules for TPP sequenced and transformed tables, improving privacy for patient-identifiable information. + +### Bug Fixes +- Add PDS response rejection on null Matched NHS number. *[PR: [#75](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/75)]* + > 🐞 ***Fix:*** Ensures only valid PDS responses with a matched NHS number are processed, preventing incomplete or invalid records from entering downstream workflows. + +- Add QUALIFY condition to PERSON share views for version resolution. *[PR: [#77](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/77), WorkItems: [AB#30203](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/30203)]* + > 🐞 ***Fix:*** Resolves issues with duplicate person records by ensuring only the most recent version per person ID is returned, improving data accuracy for consumers. + +### Refactoring +- Refactor NDOO export loop to count pending rows from the outbound view. *[PR: [#70](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/70)]* + > ♻️ ***Refactor:*** Simplifies and improves the accuracy of pending row counts in the export pipeline by querying the outbound view directly, reducing complexity and potential errors. + +### Inputs + +This changelog was automatically produced from: + +- 8 Pull Requests +- 0 Issues +- 2 WorkItems +- 52 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #68](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/68): Users/gainhams/postcode hash fix + - No linked issues. +- [PR #70](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/70): Refactor NDOO export loop to count pending rows from the outbound view + - No linked issues. +- [PR #71](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/71): Add versioned Episode of Care models and SQL configurations for OLIDS + - No linked issues. +- [PR #72](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/72): Add APPOINTMENT_STATUS_DERIVED + - [WorkItem AB#29866](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/29866) +- [PR #73](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/73): addition of 2 new TPP schemas for data masking + - No linked issues. +- [PR #75](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/75): Add PDS response rejection on null Matched NHS number + - No linked issues. +- [PR #76](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/76): masking entries for OLIDS TPP + - No linked issues. +- [PR #77](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/77): Add QUALIFY condition to PERSON share views for version resolution + - [WorkItem AB#30203](https://dev.azure.com/NELAnalytics/aef6c175-110f-4ba4-82fc-70761f459236/_workitems/edit/30203) + +[View the diff between v2.3.2 and v3.0.0](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv2.3.2...releases%2Fv3.0.0) diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.1.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.1.md new file mode 100644 index 0000000..215ded6 --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.1.md @@ -0,0 +1,26 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v3.0.0](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/releases/tag/releases%2Fv3.0.0). +## Summary + +This patch release focuses on enhancing data masking controls and improving the reliability of postcode hash table processing. Key updates include more robust masking policies for sensitive columns, expanded masking policy coverage across additional databases, and a change in the materialization strategy for postcode hash tables to ensure consistency and downstream compatibility. These changes help strengthen data privacy and improve the maintainability of the enrichment pipeline. + +### Bug Fixes +- Fix masking entries for PDS Request, including updates to masking policies for `DATE_OF_BIRTH` and `DATE_OF_DEATH` columns, improvements to masking control logic, and changes to the materialization strategy for postcode hash tables. *[PR: [#85](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/85)]* + > 🐞 ***Fix:*** Ensures sensitive data is masked correctly in PDS request tables, applies masking policies more broadly, and standardizes postcode hash table structure for improved data privacy and downstream processing reliability. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 6 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #85](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/85): fix masking entries for PDS Request + - No linked issues. + +[View the diff between v3.0.0 and v3.0.1](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv3.0.0...releases%2Fv3.0.1) diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.2.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.2.md new file mode 100644 index 0000000..7ca531d --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.2.md @@ -0,0 +1,30 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v3.0.1](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/releases/tag/releases%2Fv3.0.1). +## Summary +This release focuses on enhancing data security and integrity. Additional columns have been masked to further protect sensitive information, and outbound data validation has been strengthened to ensure only records with valid NHS numbers are processed. These changes improve compliance and data quality across the platform. + +### Improvements +- Masked additional columns (`DATE_OF_BIRTH_DAY`, `DATE_OF_DEATH_DAY`) in PERSON_PDS_LATEST, PERSON_PDS_VERSIONED, and PERSON_PDS_VERSIONED_ENTITLED tables to enhance data privacy. *[PR: [#87](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/87)]* + > 🎯 ***Impact:*** Strengthens data protection by ensuring more sensitive fields are consistently masked across key tables. + +- Enhanced validation in the NDOO Outbound view to include only records with a valid 10-digit NHS number or a valid hashed NHS number, with support for both production and development/test environments. *[PR: [#88](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/88)]* + > 🎯 ***Impact:*** Improves data quality and compliance by filtering out records with invalid NHS numbers, reducing the risk of processing incorrect or test data in production. + +### Inputs + +This changelog was automatically produced from: + +- 2 Pull Requests +- 0 Issues +- 0 WorkItems +- 7 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #87](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/87): additional columns for masking + - No linked issues. +- [PR #88](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/88): Add valid NHS number filtered into NDOO Outbound view + - No linked issues. + +[View the diff between v3.0.1 and v3.0.2](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv3.0.1...releases%2Fv3.0.2) diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.3.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.3.md new file mode 100644 index 0000000..fe66f2f --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v3.0.3.md @@ -0,0 +1,25 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v3.0.2](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/releases/tag/releases%2Fv3.0.2). +## Summary +This release introduces enhanced masking policy governance for NDOO and PDS tables, focusing on schema naming consistency, expanded column coverage, and improved environment-specific policy management. These updates strengthen data privacy controls and ensure that development and production environments are governed more accurately and consistently. + +### Improvements +- Updated masking policy governance for NDOO and PDS tables, including schema renaming to `NDOO_REQUEST`, expanded masking to cover additional columns such as `SK_PATIENT_ID`, and improved environment-specific policy assignment for development and production. Also added a post-hook for adaptive column governance in the project configuration. *[PR: [#89](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/89)]* + > 🎯 ***Impact:*** Ensures more consistent and comprehensive data masking across environments, improving data privacy and governance while reducing manual configuration errors. + +### Inputs + +This changelog was automatically produced from: + +- 1 Pull Requests +- 0 Issues +- 0 WorkItems +- 6 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #89](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/89): NDOO masking additions + - No linked issues. + +[View the diff between v3.0.2 and v3.0.3](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv3.0.2...releases%2Fv3.0.3) diff --git a/OLIDS/Release-notes/3_OLIDS_Enrichment/v4.0.0.md b/OLIDS/Release-notes/3_OLIDS_Enrichment/v4.0.0.md new file mode 100644 index 0000000..d4e6aca --- /dev/null +++ b/OLIDS/Release-notes/3_OLIDS_Enrichment/v4.0.0.md @@ -0,0 +1,61 @@ +> [!NOTE] +> The below is a summary of the changes since the previous release [v3.0.3](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/releases/tag/releases%2Fv3.0.3). +## Summary + +This release delivers significant enhancements to the OLIDS Enrichment platform, introducing new data integration layers, improved terminology management, and expanded support for data opt-out governance. Key highlights include the addition of a "Combine" schema for unifying datasets, integration of Local Data Opt-Out (LDOO) alongside National Data Opt-Out (NDOO), and comprehensive updates to concept and mapping tables to ensure referential integrity and historical tracking. Several improvements also standardize table naming and update package dependencies, supporting both EMIS and TPP data sources. These changes collectively strengthen data consistency, governance, and usability for downstream consumers. + +### New Features + +- Add new SQL views to combine datasets ([#74](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/74)) + > ✨ ***Feature:*** Introduces a "Combine" schema with 30 new views to unify EMIS datasets, laying the groundwork for future integration of TPP Primary Care, Community, and Mental Health data. This enables more comprehensive and streamlined data analysis across sources. + +- Adding Terminology components ([#93](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/93)) + > ✨ ***Feature:*** Adds new terminology objects and updates for UPRN and PDS ingestion, improving the platform's ability to manage and process standardised clinical terminology and supporting more robust data linkage. + +- Add Local Data Opt Out (LDOO) model and tests ([#94](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/94)) + > ✨ ***Feature:*** Adds support for the Local Data Opt-Out (LDOO) data source, integrates it with existing National Data Opt-Out (NDOO) enrichment, and updates masking and sharing views. This ensures both opt-out types are governed and available for downstream use, with comprehensive unit tests for reliability. + +### Improvements + +- Concept and Concept Map Updates ([#92](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/92)) + > 🎯 ***Impact:*** Adds missing concepts, ensures only current code versions are included, and improves referential integrity between concept and concept_map tables. Introduces historic mapping tracking and new metadata columns, enhancing data accuracy and auditability. + +- Mapping table updates for TPP and adding LDOO ([#95](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/95)) + > 🎯 ***Impact:*** Updates mapping tables to support TPP Community and Mental Health ingestion and incorporates LDOO, broadening the scope of data ingestion and improving data governance. + +- Update package revisions for EMIS_OLIDS and OLIDS_Share ([#96](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/96)) + > 🎯 ***Impact:*** Updates package dependencies to the latest revisions, ensuring compatibility and access to recent features and fixes. + +### Bug Fixes + +- Standardize table names in masking policies for TPP data ([#91](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/91)) + > 🐞 ***Fix:*** Updates masking policies to use standardized table names for TPP data, aligning them with EMIS naming conventions and reducing the risk of misconfiguration or data access issues. + +### Inputs + +This changelog was automatically produced from: + +- 7 Pull Requests +- 0 Issues +- 0 WorkItems +- 67 Commits + +## Full Details + +Pull Requests included in this release: +- [PR #74](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/74): Add new SQL views to combine datasets + - No linked issues. +- [PR #91](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/91): Standardize table names in masking policies for TPP data + - No linked issues. +- [PR #92](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/92): Concept and Concept Map Updates + - No linked issues. +- [PR #93](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/93): Adding Terminology components + - No linked issues. +- [PR #94](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/94): Add Local Data Opt Out (LDOO) model and tests + - No linked issues. +- [PR #95](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/95): Mapping table updates for TPP and adding LDOO + - No linked issues. +- [PR #96](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/pull/96): Update package revisions for EMIS_OLIDS and OLIDS_Share + - No linked issues. + +[View the diff between v3.0.3 and v4.0.0](https://github.com/NHSISL/LondonDataservices.Transform.OLIDS_Enrichment/compare/releases%2Fv3.0.3...releases%2Fv4.0.0)