Multithreaded compression

[Smotrov]

It was mentioned in the announcement that "Multithreaded compression allows multiple threads to operate in unison on a single file." However, I've been struggling to find any documentation or examples that illustrate this feature. I would greatly appreciate it if someone could share a code snippet or example showing how to utilize this new functionality.
I suspect it is compress_multi, but the last parameter alloc_per_thread is unclear for me.

[danielrh]

You should pass an allocator per thread so that it can allocate from each thread. If your allocator is thread safe then you can share the same one.

[Smotrov]

Thank you @danielrh!

I'm not sure if this is correct approach.
Meanwhile I do not have a clear understanding of SliceWrapper purpose and second parameter of SendAlloc::new.

I also have 2 questions:

• What to do if I can not predict output length? Now it s 1000. When it is just Vec::new() there is an error.
• The more cores I set the bigger number is returned by compress_multi. How to explain this?

use brotli::enc::multithreading::compress_multi;
use brotli::enc::threading::SendAlloc;
use brotli::enc::backward_references::BrotliEncoderParams;
use brotli::enc::{StandardAlloc, Owned};
use brotli::enc::UnionHasher;

// Adjust number of threads
const CPU_CORES: usize = 16;

struct MySliceWrapper(Vec<u8>);

impl brotli::enc::SliceWrapper<u8> for MySliceWrapper {
    fn slice(&self) -> &[u8] {
        &self.0
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let params = BrotliEncoderParams::default();
    let data = MySliceWrapper(b"Example data to compress".to_vec());
    let mut owned_input = Owned::new(data);
    let mut output = vec![0u8; 10000]; // Adjust the size according to your needs

    let alloc = StandardAlloc::default(); // Adjust based on your allocation strategy

    let mut alloc_per_thread = Vec::with_capacity(CPU_CORES);


    for _ in 0..CPU_CORES {
        alloc_per_thread.push(SendAlloc::new(alloc,UnionHasher::Uninit));
    }

    let result = compress_multi(&params, &mut owned_input, &mut output, &mut alloc_per_thread).unwrap();
    println!("Compressed bytes: {}", result);

    Ok(())
}

[eureka-cpu]

Thank you @danielrh!

I'm not sure if this is correct approach. Meanwhile I do not have a clear understanding of SliceWrapper purpose and second parameter of SendAlloc::new.

I also have 2 questions:

• What to do if I can not predict output length? Now it s 1000. When it is just Vec::new() there is an error.
• The more cores I set the bigger number is returned by compress_multi. How to explain this?

use brotli::enc::multithreading::compress_multi;
use brotli::enc::threading::SendAlloc;
use brotli::enc::backward_references::BrotliEncoderParams;
use brotli::enc::{StandardAlloc, Owned};
use brotli::enc::UnionHasher;

// Adjust number of threads
const CPU_CORES: usize = 16;

struct MySliceWrapper(Vec);

impl brotli::enc::SliceWrapper for MySliceWrapper {
fn slice(&self) -> &[u8] {
&self.0
}
}

fn main() -> Result<(), Box> {
let params = BrotliEncoderParams::default();
let data = MySliceWrapper(b"Example data to compress".to_vec());
let mut owned_input = Owned::new(data);
let mut output = vec![0u8; 10000]; // Adjust the size according to your needs

let alloc = StandardAlloc::default(); // Adjust based on your allocation strategy

let mut alloc_per_thread = Vec::with_capacity(CPU_CORES);


for _ in 0..CPU_CORES {
    alloc_per_thread.push(SendAlloc::new(alloc,UnionHasher::Uninit));
}

let result = compress_multi(&params, &mut owned_input, &mut output, &mut alloc_per_thread).unwrap();
println!("Compressed bytes: {}", result);

Ok(())

}

Sorry to respond to this so late, you probably already realized that output is just the Writer, so you could replace it with a File for instance, or wrap the type in a type that implements Write, which I guess Vec on its own is insufficient for that purpose.

Edit: Wait, I had expected this to work the same as the compressor writer type but I guess it doesn't for some reason... @danielrh why is that?