Skip to content

DEV - Event driven External data ingestion and analysis starting - #5300

Merged
mogmi merged 13 commits into
masterfrom
dev-external-nats
Sep 10, 2026
Merged

DEV - Event driven External data ingestion and analysis starting#5300
mogmi merged 13 commits into
masterfrom
dev-external-nats

Conversation

@diitaz93

@diitaz93 diitaz93 commented Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Provide support to message publishing using nats-py. Currently used in the new external_sample_transferred_handler and in the order storing service.

Added

  • New methods to the event publisher that establish a connection to the NATS server, publish a message and drain the connection asynchronously.
  • external_sample_transferred_handler that adds the transferred files to HK and publishes an event using the new event_publisher method to start an analysis
  • New HK API method that adds bundle and version to the DB transaction but does not commit.
  • New HK method that hard-links files to the bundle dirs and commits

Changed

  • The OrderSubmitter now recieved a NatsConfig object and a store as parameters
  • NATS env variables can now be parsed by the app_config obj

Fixed

  • Unifies all event names and payload fields in constants in cg/services/events/constants.py
  • rename directory tests/services/orders/validation_service to tests/services/orders/validation

How to prepare for test

  • Ssh to relevant server (depending on type of change)
  • Use stage: us
  • Paxa the environment: paxa
  • Install on stage (example for Hasta): 
    bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_cg -t cg -b [THIS-BRANCH-NAME] -a

How to test

  • Do ...

Expected test outcome

  • Check that ...
  • Take a screenshot and attach or copy/paste the output.

Review

  • Tests executed by
  • "Merge and deploy" approved by
    Thanks for filling in who performed the code review and the test!

This version is a

  • MAJOR - when you make incompatible API changes
  • MINOR - when you add functionality in a backwards compatible manner
  • PATCH - when you make backwards compatible bug fixes or documentation/instructions

Implementation Plan

  • Document in ...
  • Deploy this branch on ...
  • Inform to ...

diitaz93 and others added 5 commits September 2, 2026 17:28
@diitaz93
WIP
0dcb31a
@diitaz93
WIP
b651b80
@diitaz93
first commit
49b7d72
@islean @RasmusBurge-CG
@diitaz93 @mogmi @moedarrah @kedhammar
sample-transferred-handler (#5293)
b3fecdb 
---------

Co-authored-by: Rasmus Burge <80392398+RasmusBurge-CG@users.noreply.github.com>
Co-authored-by: Sebastian Diaz <juan.sebastian.diaz.boada@scilifelab.se>
Co-authored-by: Linnéa Löfdahl <linnea.lofdahl@scilifelab.se>
Co-authored-by: Moe Darrah <41540288+moedarrah@users.noreply.github.com>
Co-authored-by: kedhammar <alfred.kedhammar@scilifelab.se>
Co-authored-by: Alfred Kedhammar <89784800+kedhammar@users.noreply.github.com>
@diitaz93
Merge branch 'master' into dev-external-nats
af715dd
diitaz93
diitaz93 commented Sep 7, 2026
View reviewed changes
Comment thread cg/services/events/event_dispatching.py Outdated
diitaz93 and others added 4 commits September 9, 2026 14:33
@diitaz93 @moedarrah
@kedhammar @islean @mogmi
Create event when order with external samples is submitted (#5292)
5e1ec6a 
* WIP

* fix type hints

* Add TODOs

* WIP

* TODO

* add statusdb to order

* add passing function

* Work on submitter

* WIP

* implement tests for submit with new functionality

* WIP

* add tests

* Fix typehint for Order external_samples (#5296)

* wip red

* add app_config settings for NATS

* Work on adding NatsConfig in the app_config

* wip red

* rename nats env variables

* update renaming

* Change order of publishing of events

* red wip

* Fix failing tests

* Change logging

---------

Co-authored-by: Moe Darrah <41540288+moedarrah@users.noreply.github.com>
Co-authored-by: kedhammar <alfred.kedhammar@scilifelab.se>
Co-authored-by: islean <isak.ohlsson.angnell@gmail.com>
Co-authored-by: Linnéa Löfdahl <linnea.lofdahl@scilifelab.se>
Co-authored-by: Isak Ohlsson Ångnell <40887124+islean@users.noreply.github.com>
@islean @mogmi
@moedarrah @diitaz93
Standardise event and field names (#5303)
d9a7e28 
* Start standardising names

* Rework the field names

* Start on event names

* WIP

* Green tests

* return hint

---------

Co-authored-by: Linnéa Löfdahl <linnea.lofdahl@scilifelab.se>
Co-authored-by: Moe Darrah <41540288+moedarrah@users.noreply.github.com>
Co-authored-by: Sebastian Diaz <juan.sebastian.diaz.boada@scilifelab.se>
@diitaz93
Merge branch 'master' into dev-external-nats
dba86c1
@diitaz93
rename constants from subject to events
bd7c6bb
@diitaz93
diitaz93 marked this pull request as ready for review September 9, 2026 13:41
@diitaz93
diitaz93 requested a review from a team as a code owner September 9, 2026 13:41
islean
islean reviewed Sep 9, 2026
View reviewed changes
Comment thread cg/services/orders/submitter/service.py Outdated
islean
islean reviewed Sep 9, 2026
View reviewed changes
Comment thread cg/services/orders/validation/models/order_with_cases.py
islean
islean reviewed Sep 9, 2026
View reviewed changes
Comment thread cg/services/transfer_to_cluster_service.py Outdated
moedarrah and others added 3 commits September 9, 2026 16:44
@moedarrah
Clarify event publisher function names (#5306)
c2a8dec 
* Rename publish_command to get_publish_command

* Rename publish to publish_event
@islean
Address comments
dd9f422
@mogmi
Send in event_name to publish_command
cb31883
@sonarqubecloud

sonarqubecloud Bot commented Sep 10, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

mogmi
mogmi approved these changes Sep 10, 2026
View reviewed changes

@mogmi mogmi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐺

@mogmi
mogmi merged commit 4be50bb into master Sep 10, 2026
8 checks passed
@mogmi
mogmi deleted the dev-external-nats branch September 10, 2026 10:03
@mogmi

mogmi commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Deployed to stage:

Getting version of deploy scripts... done.
Log deploy... done.
cg, version 88.8.0
[12:06] [hiseq.clinical@hasta:/home/proj/stage/bin/miniconda3/envs] [S_base]  $

Deployed to prod:

Log deploy... done.
cg, version 88.8.0
[12:09] [hiseq.clinical@hasta:/home/proj/stage/bin/miniconda3/envs] [P_base]  $

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@islean islean islean left review comments
@mogmi mogmi mogmi approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@diitaz93 @mogmi @islean @moedarrah