Population scale merging with Jasmine for short-reads

[Solyris83]

Hi,

We are intending to test out Jasmine for a population based experiment with more than 10k sample WGS with Illumina short-reads (SR).

We are doing a test-run on 3202 WGS result on AWS OpenAccess data Illumina-Dragen processed WGS 30x coverage.
https://registry.opendata.aws/ilmn-dragen-1kgp/

The Jasmine runs on 3202 sample sv.vcf from the Dragen analysis output did not complete and timed-out after 3 days on a 64 CPU and 128 GB RAM machine.

I tried to split_jasmine the VCF as mentioned below, but the jar script failed likely due to missing CHR2 INFO tag which is not inherently output by the SV-caller.
#18

We were thinking of either

1. Running in smaller batches of maybe 500-1000 samples and merging each batches resulting VCF in a step-wise manner.
2. split by chromosome level for the whole 3202 sample and then Jasmine merge these 3202 over 24 chromosome (1-22, X and Y)

Does route 2 affect the BND events which are found abundantly (~20%) in the Dragen VCF?
And please do comment which route is more preferable?

Regards
Solyris

[mschatz]

Thanks for the feedback! I would strongly recommend option 2 (split by chromosome) for this. I would also be highly skeptical of SVs spanning different chromosomes in this (healthy) population. All or nearly all of the reported "fusions" will be false positives, most often because of transposable elements and other repeats that are shared across chromosomes. Good luck Mike

…

On Mon, Mar 24, 2025 at 5:34 AM Solyris83 ***@***.***> wrote: Hi, We are intending to test out Jasmine for a population based experiment with more than 10k sample WGS with Illumina short-reads (SR). We are doing a test-run on 3202 WGS result on AWS OpenAccess data Illumina-Dragen processed WGS 30x coverage. https://registry.opendata.aws/ilmn-dragen-1kgp/ The Jasmine runs on 3202 sample sv.vcf from the Dragen analysis output did not complete and timed-out after 3 days on a 64 CPU and 128 GB RAM machine. I tried to split_jasmine the VCF as mentioned below, but the jar script failed likely due to missing CHR2 INFO tag which is not inherently output by the SV-caller. #18 <#18> We were thinking of either 1. Running in smaller batches of maybe 500-1000 samples and merging each batches resulting VCF in a step-wise manner. 2. split by chromosome level for the whole 3202 sample and then Jasmine merge these 3202 over 24 chromosome (1-22, X and Y) Does route 2 affect the BND events which are found abundantly (~20%) in the Dragen VCF? And please do comment which route is more preferable? Regards Solyris — Reply to this email directly, view it on GitHub <#64>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABP34YD4OZWBFBYA5RMPIT2V7GSPAVCNFSM6AAAAABZUMMBWGVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE2DENRRGI3DOMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: Solyris83]*Solyris83* created an issue (mkirsche/Jasmine#64) <#64> Hi, We are intending to test out Jasmine for a population based experiment with more than 10k sample WGS with Illumina short-reads (SR). We are doing a test-run on 3202 WGS result on AWS OpenAccess data Illumina-Dragen processed WGS 30x coverage. https://registry.opendata.aws/ilmn-dragen-1kgp/ The Jasmine runs on 3202 sample sv.vcf from the Dragen analysis output did not complete and timed-out after 3 days on a 64 CPU and 128 GB RAM machine. I tried to split_jasmine the VCF as mentioned below, but the jar script failed likely due to missing CHR2 INFO tag which is not inherently output by the SV-caller. #18 <#18> We were thinking of either 1. Running in smaller batches of maybe 500-1000 samples and merging each batches resulting VCF in a step-wise manner. 2. split by chromosome level for the whole 3202 sample and then Jasmine merge these 3202 over 24 chromosome (1-22, X and Y) Does route 2 affect the BND events which are found abundantly (~20%) in the Dragen VCF? And please do comment which route is more preferable? Regards Solyris — Reply to this email directly, view it on GitHub <#64>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABP34YD4OZWBFBYA5RMPIT2V7GSPAVCNFSM6AAAAABZUMMBWGVHI2DSMVQWIX3LMV43ASLTON2WKOZSHE2DENRRGI3DOMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Solyris83]

Hi, I managed to run a test set of 3202 sample from 1000-genome project data which completed in an acceptable time (~4h on 64 CPU and 512GB RAM).

I have since then tried to scale this up to a 100,000 sample analysis and attempted to do this on chr22 as a test, and Jasmine failed to complete this
Analysis done on AWS ec2 with 192 CPU and 1.5TB ram
decided to kill the process after 6 full straight days running this and nothing to show for it.

Some observation, I have been logging into that machine to check on it on and off the 6 days

1. 1st day of running jasmine i can see full engagement of the CPU with htop, with most CPU at or near 100% utility, but after that it just died down and only show 3-4 CPU engaged at 100% in all instances in logged in to check
2. I have changed the jasmine script to include some java option to enable more memory usage (java -XX:MinRAMPercentage=85 -XX:MaxRAMPercentage=85) , jasmine process starts and is stuck to 1tb usage on htop views
3. last error printed complains about OutofMemory: java heap space BUT process did not die and continue on its merry way, which is doing nothing? I do not think i can go on making biger machine to test this as the r6a.48xlarge i am using is the largest of its kind with highest memory made available in the AWS universe?

any comment is appreciated

[hayley-power]

Did you figure out how to fix the heap space error?

[maximilianh]

Same problem here, but with SVs from around 500 assemblies.