Hello SLOW5 team,
I am one of the authors of a recently published paper in Bioinformatics Advances presenting PDZ, a new lossless compression algorithm for Oxford Nanopore raw signal data:
Efficient lossless compression of nanopore sequencing signals
DOI: 10.1093/bioadv/vbag157
Article: https://academic.oup.com/bioinformaticsadvances/article/6/1/vbag157/8707629
Benchmark repository: https://github.com/GuilleDufortFing/Nanopore-Compression-Benchmarks
PDZ was developed specifically for nanopore raw signals. Although our initial implementation was integrated into a fork of the POD5 library, the core algorithm is format-independent and could potentially be implemented as an additional signal-compression method for SLOW5/BLOW5.
For SLOW5/BLOW5, the most relevant code and reproducibility material are in the benchmark repository linked above. The POD5 fork is mainly useful as a proof of integration in that specific ecosystem, but the benchmark repository contains the implementation and tests that would be most useful for evaluating or adapting PDZ independently of POD5.
In our evaluation, PDZ improved compression over VBZ while also being faster. Across the datasets and hardware configurations evaluated in the paper, we observed:
- Compression ratio improvement over VBZ: 0.87%–2.84%, depending on the dataset.
- Compression speedup over VBZ: 1.09×–2.25×, depending on the hardware.
- Decompression speedup over VBZ: 1.01×–1.52×, depending on the hardware.
We also compared PDZ against EX-ZD-ZSTD, which is particularly relevant for SLOW5/BLOW5. In terms of compression ratio, PDZ and EX-ZD-ZSTD were very similar: the relative difference was below 0.1% for most datasets, with the largest difference in favor of EX-ZD-ZSTD being 0.366%. However, PDZ was consistently faster in our experiments:
- Compression speedup over EX-ZD-ZSTD: 1.40×–1.83×, depending on the hardware.
- Decompression speedup over EX-ZD-ZSTD: 1.39×–1.85×, depending on the hardware.
Given that SLOW5/BLOW5 provides an efficient ecosystem for nanopore raw-signal storage, we wanted to ask whether the SLOW5 maintainers would be interested in evaluating PDZ as a possible additional lossless signal-compression option.
If this is of interest, we would be happy to prepare an initial pull request with an implementation of the PDZ algorithm for the SLOW5 team to evaluate. The goal of the PR would be to provide a concrete prototype for discussion and benchmarking, rather than to assume that PDZ should be integrated without further review.
Thank you for developing and maintaining SLOW5/BLOW5 and slow5tools.
Best regards,
Guillermo Dufort y Álvarez
on behalf of the PDZ authors
Hello SLOW5 team,
I am one of the authors of a recently published paper in Bioinformatics Advances presenting PDZ, a new lossless compression algorithm for Oxford Nanopore raw signal data:
Efficient lossless compression of nanopore sequencing signals
DOI:
10.1093/bioadv/vbag157Article: https://academic.oup.com/bioinformaticsadvances/article/6/1/vbag157/8707629
Benchmark repository: https://github.com/GuilleDufortFing/Nanopore-Compression-Benchmarks
PDZ was developed specifically for nanopore raw signals. Although our initial implementation was integrated into a fork of the POD5 library, the core algorithm is format-independent and could potentially be implemented as an additional signal-compression method for SLOW5/BLOW5.
For SLOW5/BLOW5, the most relevant code and reproducibility material are in the benchmark repository linked above. The POD5 fork is mainly useful as a proof of integration in that specific ecosystem, but the benchmark repository contains the implementation and tests that would be most useful for evaluating or adapting PDZ independently of POD5.
In our evaluation, PDZ improved compression over VBZ while also being faster. Across the datasets and hardware configurations evaluated in the paper, we observed:
We also compared PDZ against EX-ZD-ZSTD, which is particularly relevant for SLOW5/BLOW5. In terms of compression ratio, PDZ and EX-ZD-ZSTD were very similar: the relative difference was below 0.1% for most datasets, with the largest difference in favor of EX-ZD-ZSTD being 0.366%. However, PDZ was consistently faster in our experiments:
Given that SLOW5/BLOW5 provides an efficient ecosystem for nanopore raw-signal storage, we wanted to ask whether the SLOW5 maintainers would be interested in evaluating PDZ as a possible additional lossless signal-compression option.
If this is of interest, we would be happy to prepare an initial pull request with an implementation of the PDZ algorithm for the SLOW5 team to evaluate. The goal of the PR would be to provide a concrete prototype for discussion and benchmarking, rather than to assume that PDZ should be integrated without further review.
Thank you for developing and maintaining SLOW5/BLOW5 and slow5tools.
Best regards,
Guillermo Dufort y Álvarez
on behalf of the PDZ authors