How can I fixate a voice for an audiobook?

[WimTip]

Hi Peter,
Thanks for the Parkiet package, the quality and range of voices is impressive.
I want to use this package to create an audiobook from Max Havelaar by Multatuli (no copyright restrictions, also no audiobook on Bol.com).
For this I would like to have control over what voice is used. Can you guide me?
Again thanks for the package, keep up the good work!
Kind regards, Wim

[pevers]

Hi @WimTip , thanks! And cool project.

Generating long sequences is still a bit hard. But it can work if you stitch it carefully together.

In the generate method in inference.py you can add an audio_prompt parameter

parkiet/src/parkiet/dia/inference.py

Line 23 in d85a55b

output = model.generate(

It should be a .wav or .mp3 for example with ~ 10 seconds of audio of a voice (can be an open voice or your own).

Then you can generate part of the text which is currently limited to 30 seconds only. So you would have to batch it in small pieces and then stitch it together.

I'm working now on the next iteration to generate much longer samples with similar quality.

[0xrushi]

Hi @pevers , thanks for sharing this. Do you have any suggestions for cases where the WAV file used in the audio prompt sounds very different from the generated output?
Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

[pevers]

Hi @pevers , thanks for sharing this. Do you have any suggestions for cases where the WAV file used in the audio prompt sounds very different from the generated output? Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

Yes I noticed this as well. It is not always perfect. Longer samples work a bit better but they also fill up the context. Podcast like speech as sample is the best way to get good results.

Also, is there a way to finetune for a specific voice while preserving a full range of emotions?

Yes, you can fine-tune with a specific voice, you don't need much data. The data is synthetically generated via https://github.com/pevers/whisperd-nl . So if you have a couple of hours of voice data you should be able to transcribe it with disfluencies. The emotions are hard and I only implemented (laughs) so far.

[isilher]

Hi Wim, Peter,

We built a pipeline around Parkiet that tackles exactly this — generating consistent long-form Dutch audio with a fixed voice. We just finished a successful run of the first page of Max Havelaar (slightly redacted to modern Dutch), here's what we found was needed to get consistent results:

1. Whisper-guided prompt alignment — We transcribe the reference audio with WhisperD-NL, trim it to the first sentence, encode it as DAC codes, and prepend that sentence to every chunk's text. This anchors the voice at the start of each generation, keeping it stable across chunks.

2. Sentence-level chunking — The text is split into sentence-level chunks, respecting paragraph boundaries. Parkiet prefers lowercase prose with ... for pacing pauses, so the splitter handles that along with Dutch abbreviations (nr., dhr., mevr.) to avoid false splits. Short sentences are merged together to prevent tiny audio clips. During concatenation, silence is inserted at paragraph boundaries for natural pacing.

3. Quality validation with auto-retry — Each chunk is validated on three axes: text accuracy (Whisper round-trip fuzzy match), speaker similarity (resemblyzer embeddings vs reference), and audio quality (click/clipping/silence detection). Failures are retried up to 9 times, keeping the best attempt. This catches the hallucinations and artifacts that sampling-based generation sometimes produces.

4. Generation presets — Temperature, top_p, top_k, and cfg_scale are bundled into named presets. We found that tuning these per-voice makes a big difference — we run a calibration step that generates short samples across all presets so you can pick the one that sounds best for your reference voice.

The attached audio was generated using the fluweel preset (temperature 1.0, top_p 0.95, top_k 50, cfg_scale 3.0) — the most conservative/smooth setting we calibrated for this voice. It's not perfectly clean yet and has some odd inflections, but the aim was consistent voice and the result already gets pretty close to usable. (input voice was taken from an earlier output using Parkiet without voice cloning).

Audio: max-havelaar-v3.mp3

@pevers — one question: do you have recommendations on ideal reference audio length/style beyond "podcast-like"? We're using ~8-10s clips and getting 0.93-0.95 speaker similarity scores, but curious if there's a sweet spot.

[pevers]

Hi Wim, Peter,

We built a pipeline around Parkiet that tackles exactly this — generating consistent long-form Dutch audio with a fixed voice. We just finished a successful run of the first page of Max Havelaar (slightly redacted to modern Dutch), here's what we found was needed to get consistent results:

1. Whisper-guided prompt alignment — We transcribe the reference audio with WhisperD-NL, trim it to the first sentence, encode it as DAC codes, and prepend that sentence to every chunk's text. This anchors the voice at the start of each generation, keeping it stable across chunks.

2. Sentence-level chunking — The text is split into sentence-level chunks, respecting paragraph boundaries. Parkiet prefers lowercase prose with ... for pacing pauses, so the splitter handles that along with Dutch abbreviations (nr., dhr., mevr.) to avoid false splits. Short sentences are merged together to prevent tiny audio clips. During concatenation, silence is inserted at paragraph boundaries for natural pacing.

3. Quality validation with auto-retry — Each chunk is validated on three axes: text accuracy (Whisper round-trip fuzzy match), speaker similarity (resemblyzer embeddings vs reference), and audio quality (click/clipping/silence detection). Failures are retried up to 9 times, keeping the best attempt. This catches the hallucinations and artifacts that sampling-based generation sometimes produces.

4. Generation presets — Temperature, top_p, top_k, and cfg_scale are bundled into named presets. We found that tuning these per-voice makes a big difference — we run a calibration step that generates short samples across all presets so you can pick the one that sounds best for your reference voice.

The attached audio was generated using the fluweel preset (temperature 1.0, top_p 0.95, top_k 50, cfg_scale 3.0) — the most conservative/smooth setting we calibrated for this voice. It's not perfectly clean yet and has some odd inflections, but the aim was consistent voice and the result already gets pretty close to usable. (input voice was taken from an earlier output using Parkiet without voice cloning).

Audio: max-havelaar-v3.mp3

@pevers — one question: do you have recommendations on ideal reference audio length/style beyond "podcast-like"? We're using ~8-10s clips and getting 0.93-0.95 speaker similarity scores, but curious if there's a sweet spot.

Wow this is crazy good @isilher ! Smart way to tackle this. I trained on clips that had an average slightly higher around 15 s. But that doesn't mean that's ideal, the longer the bigger the chance of drift. It can't really recover from noise.

Really well done!

On a side note, I do think that to fully solve this we probably need a slightly different architecture, like VibeVoice, and train on just audio books instead of podcast data. I don't have the time anymore unfortunately

[isilher]

Thanks! I'll look into vibevoice. Do you have any pointers for me if I'd be interested in taking on that challenge? So far my experimentation has been standing on the shoulders of your work and my previous experience with qwen3 tts.

[pevers]

Thanks! I'll look into vibevoice. Do you have any pointers for me if I'd be interested in taking on that challenge? So far my experimentation has been standing on the shoulders of your work and my previous experience with qwen3 tts.

Get TPU research credits if you are up for a challenge or start with LoRa. You don't need that much data for single voice fine tuning