Skip to content

feat(speech): add reverb-tts pipeline - #1638

Open
IgorSwat wants to merge 7 commits into
mainfrom
@is/reverb-tts
Open

IgorSwat wants to merge 7 commits into
mainfrom
@is/reverb-tts

Conversation

@IgorSwat

Copy link
Copy Markdown
Contributor

Description

This PR adds a full pipeline (task extension, model registry, demo app) for Reverb-TTS model.

Requires v0.11.2 binaries from #1637 .

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

Run the following tests for the Reverb pipeline:

packages/react-native-executorch/__tests__/tasks/reverbTextToSpeech.test.ts
packages/react-native-executorch/__tests__/extensions/reverbUtils.test.ts
packages/react-native-executorch/__tests__/extensions/reverbPartitioner.test.ts

or play with demo app in apps/speech for a manual testing.

Screenshots

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

@IgorSwat
IgorSwat requested a review from barhanc November 16, 2026 15:16
@barhanc barhanc added the feature PRs that implement a new feature label Nov 16, 2026
@barhanc

barhanc commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

@IgorSwat Please rebase first. Edit: is the target branch correct?

@IgorSwat
IgorSwat force-pushed the @is/binaries-v0.11.2 branch from 36b3a66 to 457beb2 Compare November 16, 2026 15:51
Base automatically changed from @is/binaries-v0.11.2 to main November 16, 2026 15:56

@barhanc barhanc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will test the example app first thing tomorrow.

Comment on lines +95 to +97
// one backend nearly every model exports to. Core ML sits above MLX only to
// make the order deterministic; every group publishing both pins its winner
// explicitly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not a big problem, we should keep an xnnpack delegate as a fallback (not every app dev might want to include any other backend).

// less. All decode with the fp16 codec (the int8 Core ML one holds ~600 MB
// more on the GPU); INT4 encodes with int4 (40 MB smaller), the rest with
// int8. No XNNPACK export: the backbone needs MLX or Vulkan.
REVERB: variants({

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the rationale for keeping both int4 and int8?

Comment on lines +1343 to +1345
'standard_male',
'angry_male',
'sensual_female',

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make the number of male and female voices equal, add standard_female and maybe consider renaming these (especially the sensual_female).

Comment on lines +9 to +19
/**
* Sentence-ending punctuation.
* @category Speech / Constants
*/
export const EOS_PATTERN = /[.?!;…|।॥¿¡。.!?;]/;

/**
* Punctuation pausing within a sentence: commas, colons, dashes.
* @category Speech / Constants
*/
export const PAUSE_PATTERN = /[,:\u2014\u00AB\u00BB、,:〜~]/;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can copy these constants over to Reverb files as not to expose them in public API.

* @returns The prepared text.
*/
export function prepareText(text: string, language: ReverbLanguage): string {
const verbalizer = (verbalizers[language] ??= new Verbalizer(VERBALIS_LANGUAGES[language]()));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this caching actually bring any performance benefits, because a simpler code is preferable over minuscule perf improvements. Side question out of curiosity: why is Verbalizer a class, what state does it hold?

/** Max seconds of speech per chunk. Defaults to 20. */
readonly maxChunkDuration?: number;
/** Post-processing, or `false` for the raw decoder output. */
readonly postprocess?: ReverbPostprocessOptions | false;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The | false union here looks weird, wouldn't checking if postprocess field is undefined be sufficient?

Comment on lines +228 to +233
const frameDim = (paramSide: 'input' | 'output', tensorIdx: number, dimIdx = 1): DimRef => ({
paramSide,
tensorIdx,
dimIdx,
});
const cache = () => f16(2, CACHE_LAYERS, CACHE_HEADS, Dyn('P'), CACHE_HEAD_DIM);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think writing the spec directly without these helpers makes it clearer, even if it needs more lines---for methods with a lot of arguments they should be listed one under another with comments on the side explaining each argument, and for the constraints a simple comment describing the constraint. That might be personal preference though.

runtime?: WorkletRuntime
): Promise<ReverbTextToSpeech<K>> {
const scope = createResourceScope();
let disposed = false;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag seems redundant, native code checks for disposal and throws accordingly, so no crash is possible in that case.

Comment on lines +412 to +417
const auxTensors: Tensor[] = [];
const alloc = (...args: Parameters<typeof tensor>): Tensor => {
const t = tensor(...args);
auxTensors.push(t);
return t;
};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could use local scope here instead of this adhoc implementation.

samplePass(tokens, frames, pass, step, schedule, remaskRatio, random);
}

// 3. Polish codebooks 2-7

@barhanc barhanc Nov 16, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's a stupid question, but does "Polish" here mean the language or "polish" as in "improve"? If the latter, perhaps consider renaming this (and maybe the method name itself if it's not too much hassle) to "improve" / "enhance" / "refine".

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature PRs that implement a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants