Sample names limited to 10 bytes

[peterjc]

Applying a maximum sample name length of just 10 bytes is unreasonable. Even 20 would be tight. How about 32 or 64 characters?

Example run:

$ time ~/repositories/VCF2Dis/bin/VCF2Dis -i Sirococcus_tsugae_St135_filtered.vcf -o Sirococcus_tsugae_St135_filtered.distance
warning : Sample name too long [ St620_S240_L007 ] length is 15 biger (10byte),new Name is _S2r0_b007
warning : Sample name too long [ St602_S239_L007 ] length is 15 biger (10byte),new Name is _S239bh007
warning : Sample name too long [ St558_S236_L007 ] length is 15 biger (10byte),new Name is _S936_L307
warning : Sample name too long [ St557_S235_L007 ] length is 15 biger (10byte),new Name is yS235_k007
warning : Sample name too long [ St601_S238_L007 ] length is 15 biger (10byte),new Name is _d238_L0q7
warning : Sample name too long [ St535_S233_L007 ] length is 15 biger (10byte),new Name is XSR33_L007
warning : Sample name too long [ St135_S246_L007 ] length is 15 biger (10byte),new Name is _Sf46_L00w
warning : Sample name too long [ St533_S231_L007 ] length is 15 biger (10byte),new Name is _S2Y1_LJ07
warning : Sample name too long [ St626_S1_L005 ] length is 13 biger (10byte),new Name is 26_e1_L00b
warning : Sample name too long [ St559_S237_L007 ] length is 15 biger (10byte),new Name is _S237_L507
warning : Sample name too long [ St137_S222_L007 ] length is 15 biger (10byte),new Name is _S2E2_C007
warning : Sample name too long [ St619_S2_L005 ] length is 13 biger (10byte),new Name is 19162_L005
warning : Sample name too long [ St145_S224_L007 ] length is 15 biger (10byte),new Name is _S2246L007
warning : Sample name too long [ St540_S234_L007 ] length is 15 biger (10byte),new Name is _S234ln007
warning : Sample name too long [ St140_S223_L007 ] length is 15 biger (10byte),new Name is _S22pqL007
warning : Sample name too long [ St623_S241_L007 ] length is 15 biger (10byte),new Name is _S240_L307
warning : Sample name too long [ St527_S225_L007 ] length is 15 biger (10byte),new Name is _S225_Lco7
warning : Sample name too long [ St132_S221_L007 ] length is 15 biger (10byte),new Name is _S294_L007
warning : Sample name too long [ St532_S230_L007 ] length is 15 biger (10byte),new Name is 9S830_L007
warning : Sample name too long [ St529_S227_L007 ] length is 15 biger (10byte),new Name is _Sg27_b007
warning : Sample name too long [ St625_S242_L007 ] length is 15 biger (10byte),new Name is 7S2428L007
warning : Sample name too long [ St530_S228_L007 ] length is 15 biger (10byte),new Name is _7228_L207
warning : Sample name too long [ St528_S226_L007 ] length is 15 biger (10byte),new Name is _SK26_Q007
warning : Sample name too long [ St531_S229_L007 ] length is 15 biger (10byte),new Name is _42299L007
warning : Sample name too long [ St534_S232_L007 ] length is 15 biger (10byte),new Name is rS232_L00r
Total Sample Number to construct p-distance matrix is [ 25 ]
Start To Cal ...
Start To Create P_distance ...
P_distance is created done ...
        Warning: [/mnt/apps/users/pcock/conda/bin/Rscript] should install latest R package : [ape] and [ggtree] first

real    0m0.849s
user    0m1.527s
sys     0m0.059s

The output matrix uses the automatically assigned names, but interpreting this complicated as the user would have to parse your warning messages.

The warning message is unclear. There is a typo, "biger" --> "bigger" , and a missing space. My suggested wording:

warning : Sample name too long [ St620_S240_L007 ] length is 15 biger (10byte),new Name is _S2r0_b007

-->

warning : Sample name too long [ St620_S240_L007 ] length 15, max 10, new Name is _S2r0_b007

Also the replacement names seem to be based on the last 10 character chunks with partly overwritten fragments of the original names. I suspect memory corruption here?

[peterjc]

Quoting your paper:

VCF2Dis requires minimal external dependencies, making installation simple. It can generate the p-distance matrix without R or related packages although the visualization features will not be available in this case.

With the optional R dependencies installed this also generated Sirococcus_tsugae_St619_filtered.distance.nwk with the generated 10 letter names, then Sirococcus_tsugae_St619_filtered.distance_2.nwk with my original names, and Sirococcus_tsugae_St619_filtered.distance.pdf with my original names (but often cropped):

$ time ~/repositories/VCF2Dis/bin/VCF2Dis -i Sirococcus_tsugae_St135_filtered.vcf -o Sirococcus_tsugae_St135_filtered.distance
warning : Sample name too long [ St620_S240_L007 ] length is 15 biger (10byte),new Name is _S2r0_b007
warning : Sample name too long [ St602_S239_L007 ] length is 15 biger (10byte),new Name is _S239bh007
warning : Sample name too long [ St558_S236_L007 ] length is 15 biger (10byte),new Name is _S936_L307
warning : Sample name too long [ St557_S235_L007 ] length is 15 biger (10byte),new Name is yS235_k007
warning : Sample name too long [ St601_S238_L007 ] length is 15 biger (10byte),new Name is _d238_L0q7
warning : Sample name too long [ St535_S233_L007 ] length is 15 biger (10byte),new Name is XSR33_L007
warning : Sample name too long [ St135_S246_L007 ] length is 15 biger (10byte),new Name is _Sf46_L00w
warning : Sample name too long [ St533_S231_L007 ] length is 15 biger (10byte),new Name is _S2Y1_LJ07
warning : Sample name too long [ St626_S1_L005 ] length is 13 biger (10byte),new Name is 26_e1_L00b
warning : Sample name too long [ St559_S237_L007 ] length is 15 biger (10byte),new Name is _S237_L507
warning : Sample name too long [ St137_S222_L007 ] length is 15 biger (10byte),new Name is _S2E2_C007
warning : Sample name too long [ St619_S2_L005 ] length is 13 biger (10byte),new Name is 19162_L005
warning : Sample name too long [ St145_S224_L007 ] length is 15 biger (10byte),new Name is _S2246L007
warning : Sample name too long [ St540_S234_L007 ] length is 15 biger (10byte),new Name is _S234ln007
warning : Sample name too long [ St140_S223_L007 ] length is 15 biger (10byte),new Name is _S22pqL007
warning : Sample name too long [ St623_S241_L007 ] length is 15 biger (10byte),new Name is _S240_L307
warning : Sample name too long [ St527_S225_L007 ] length is 15 biger (10byte),new Name is _S225_Lco7
warning : Sample name too long [ St132_S221_L007 ] length is 15 biger (10byte),new Name is _S294_L007
warning : Sample name too long [ St532_S230_L007 ] length is 15 biger (10byte),new Name is 9S830_L007
warning : Sample name too long [ St529_S227_L007 ] length is 15 biger (10byte),new Name is _Sg27_b007
warning : Sample name too long [ St625_S242_L007 ] length is 15 biger (10byte),new Name is 7S2428L007
warning : Sample name too long [ St530_S228_L007 ] length is 15 biger (10byte),new Name is _7228_L207
warning : Sample name too long [ St528_S226_L007 ] length is 15 biger (10byte),new Name is _SK26_Q007
warning : Sample name too long [ St531_S229_L007 ] length is 15 biger (10byte),new Name is _42299L007
warning : Sample name too long [ St534_S232_L007 ] length is 15 biger (10byte),new Name is rS232_L00r
Total Sample Number to construct p-distance matrix is [ 25 ]
Start To Cal ...
oStart To Create P_distance ...
P_distance is created done ...
Construct tree and pdf done ...
Re-Construct tree back by raw name ...

real    0m4.975s
user    0m6.043s
sys     0m0.638s

If the R dependencies are present, I would prefer just one distance file and one Newick tree file, both using the original names.

If the R dependencies are missing, I would expect just one distance file using the original names.

[hewm2008]

If generated successfully, it will produce xx_2.nwk file with the original filename. If the generation function is unavailable, it will generate a sed*tmp.sh file instead. You can run this script using the following command:
sh sed*tmp.sh out.mat original.mat

[peterjc]

I hadn't noticed the shell script - but it seems to be specific to fixing the Newick file? Commands like sed 's/19162_L005:/St619_S2_L005:/g' $1 > $1.tmp don't fix the distance matrix because that doesn't have the colons.

I would like the tool to fix the distance matrix automatically. If everything works, I never need to see the automatically assigned 10 character names - they seem like an internal implementation detail.

[hewm2008]

Thanks for your feedback.

The main reason for this replacement is that some downstream software (such as PHYLIP and PHYLIPNEW) requires sample names in the distance matrix to be strictly under 10 characters. To ensure compatibility with these tools, we automatically replace longer names here.

In earlier versions, the output did not truncate or modify the characters at all. However, I noticed that many users encountered errors downstream and mistakenly attributed them to a bug in my software, when in fact, it was the downstream tools that couldn’t handle longer names.

Therefore, my suggestion is that it would be best to rename your samples directly in your VCF file beforehand to meet this 10-character limit. This will help avoid any naming mismatches across all your analysis steps.

[hewm2008]

Additionally, please note that the sample order in the distance matrix is exactly the same as the order in your VCF file. As a quick workaround, you can easily replace the first column of the distance matrix with your original VCF sample names directly.

[peterjc]

Ancient file format limits strike again!

Yes, your solution makes sense if the R tools you're using have that limitation - but given the tool name "VCF2Dis" I do think the VCF to Distance Matrix output should prioritise the original names. I would treat the short name files as internal temp files (and name them accordingly).

For this dataset I was planning to change to shorter names anyway, which makes this simpler.