QVAC-23544 feat[api]: CosyVoice3 zero-shot / cross-lingual voice cloning - #3872
Merged
Conversation
Consumes speech-cpp 2026-08-14 (registry baseline repoint; the umbrella port carries no version floor): with referenceAudio set, the native front-end bakes the voice from the recording at load - speech_tokenizer_v3 tokens, CAM++ speaker embedding, prompt mel - replacing the baked default voice. promptText selects the mode per the upstream frontends: verbatim transcript = zero-shot, omitted = cross-lingual (timbre only). Fail-closed everywhere: a clone request that cannot resolve the s3tok + campplus add-on GGUFs is rejected at construction by the new consistency assert, and the engine rejects missing models, unreadable or non-finite audio, and durations outside 0.5-30 s rather than silently keeping the baked voice. The JS/addon plumbing already existed (shipped reserved); this change de-reserves the docs (referenceAudio / promptText / cosyvoiceS3tokModel / cosyvoiceCampplusModel JSDoc, CosyvoiceConfig comments, README voice-cloning section + tables + model layout, CHANGELOG), adds the load-time consistency assert with both-sides unit coverage, runner and example cloning knobs, zero-shot / cross-lingual / fail-closed integration tests, and an optional COSYVOICE_CLONE_FILES download tier (separate from the all-or-nothing COSYVOICE_FILES so non-cloning tests keep their staged dirs and skip the ~300 MB download). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mexxik
added
the
run-desktop-addon-tests
Contributor
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 more from Management, Team Lead, or Member. |
Contributor
License compliance — cleanNo new dependency license findings in this PR. Warn-only (shadow) mode — this check does not block merges yet. Updated automatically by the canonical license compliance workflow. NOTICE presence (advisory)Missing NOTICE (advisory, does not block):
|
…y date The s3tok + campplus GGUFs were published under cosy_voice/2026-08-14 (qvac#3861), not the 2026-08-13 folder proposed in the handoff; the desktop lanes failed only on that MODEL_NOT_FOUND. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ogad-tether
reviewed
Aug 17, 2026
ogad-tether
left a comment
ogad-tether
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the diff. Overall solid — the risky part lives behind the speech-cpp baseline repoint, and the JS surface checks out.
Verified locally on the branch (all pass):
node scripts/check-generated.mjs—index.js/index.d.tsin sync withsrc/index.tsbrittle-bare test/unit/cosyvoice3.inference.test.js— 31/31, 88 assertsbrittle-bare test/unit/example-clone-args.test.js— 3/3, 13 assertsnode --test scripts/__tests__/mobile-test-groups.test.js+package-contract.test.jsnode scripts/validate-mobile-tests.js— up to dateprettier --check "test/**/*.js"— clean
Also traced: cosyvoiceS3tokModel / ...Path both normalize correctly, _cosyvoiceModelDir falls back to files.modelDir so _assertCosyvoiceCloneConsistent isn't over-strict, _load() already destroys the addon on a failed activate(), per-call referenceAudio stays Audio8-only, and jfk.wav is mono/11 s — inside the documented 0.5–30 s window.
Three inline comments below (one doc bug, two test-hygiene). Nothing blocking.
Non-blocking notes:
- The third integration test asserts on
/s3tok/ifrom the native error string; nothing in this repo pins that wording (CosyvoiceModel.cppvalidateModelPathsonly checks explicitly-set paths — the discovery-failure message comes from speech-cpp). Fine given ext-lib #147, just brittle to upstream rewording. test-groups.json's_commentmentionsrunParlerWerTestandrunCosyvoice3CloneTestbut not the newly-excludedrunAudio8Test.- PR description says the clone tier is
cosy_voice/2026-08-13; the code uses2026-08-14. Description typo only. - The baseline repoint moves every default-registry port, not just speech-cpp. Taking the "range contains only the three speech bumps" claim on trust — not verifiable from this repo.
- The instruct section's "cloned one" link pointed at #cosyvoice3-1, an id GitHub never emits here: only the new ### CosyVoice3 heading slugifies to cosyvoice3 (## CosyVoice3 instruct becomes cosyvoice3-instruct), so the link went nowhere. - The base-only fail-closed test unloads in a finally, so an unexpectedly successful load - the regression it guards - cannot leave a live native instance holding the models. - That test stages its dir from the base names downloadModel.js now exports instead of a hardcoded copy, and falls back to copyFileSync when linkSync cannot span filesystems, so a shared model cache on another mount or a re-quantized tier fails the assertion rather than the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oice-clone # Conflicts: # README.md
ogad-tether
previously approved these changes
Aug 17, 2026
GustavoA1604
previously approved these changes
Aug 17, 2026
…oice-clone # Conflicts: # README.md # packages/tts-ggml/CHANGELOG.md # packages/tts-ggml/vcpkg.json
Merging main renamed [Unreleased] to [0.7.1], sweeping the cloning entry into a version that shipped without it (main has no cloning). Moved it back under a fresh [Unreleased]; 0.7.1 keeps only the Audio8 OpenCL work it actually released. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Security baseline — findings detected
See the job summary and the repository Security tab for details. This comment is updated automatically by the canonical security workflow. |
GustavoA1604
approved these changes
Aug 17, 2026
The CodeQL default-setup run for this PR failed uploading its results
during the GitHub incident ('No server is currently available'), and a
GitHub-managed dynamic run cannot be retried through the Actions API, so
a fresh head is the only way to re-run it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zbig9000
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables CosyVoice3 zero-shot / cross-lingual voice cloning in
@qvac/tts-ggml, consuming speech-cpp2026-08-14(registry #313, ext-lib #147).What changes
vcpkg-configuration.jsonbaseline repoint3689b261 → 5dd51e90: the speech-cpp umbrella dependency carries no version floor, so the registry baseline is the version lever. The range contains only the three speech bumps (tts-cpp/speech-cpp Audio8 Vulkan fix, speech-cpp cloning).referenceAudio/promptText/cosyvoiceS3tokModel/cosyvoiceCampplusModelplumbing shipped earlier as reserved; the new engine makes it real. This PR de-reserves the docs and hardens the surface:_assertCosyvoiceCloneConsistent: a clone request that cannot resolve the s3tok + campplus GGUFs (no explicit paths, no model dir) is rejected at construction with an actionable message;promptTextstays optional (its absence selects cross-lingual) and legal on its own (baked-voice transcript override). Both sides unit-tested.CosyvoiceConfig.hppcomments / README (voice-cloning section, decision guide, capability matrix, options + files tables, model layout) / CHANGELOG, regeneratedindex.d.ts/index.js.runCosyvoiceTTScloning knobs +examples/cosyvoice-tts.js--reference-audio/--prompt-text.COSYVOICE_CLONE_FILES: a separate optional download tier (s3tok q8_0 + campplus f32,cosy_voice/2026-08-13) — deliberately not folded into the all-or-nothingCOSYVOICE_FILES, so existing staged dirs stay valid and non-cloning tests never pay the ~300 MB download.Semantics (mirror the upstream frontends)
referenceAudio+promptText(verbatim transcript)referenceAudioonlypromptTextonlyReference limits: mono, 0.5–30 s hard (5–15 s recommended), finite samples. The one-time bake costs ~1 s CPU at
load(); per-synthesis cost is unchanged. Every bake failure throws — no silent fallback to the baked voice.instructcomposes with a cloned voice. Mobile manifests are untouched (cloning is desktop-scoped for now; the add-on tier is not in the 6-entry mobile pin).Engine-side validation (ext-lib #147)
Asserting parity suites on CPU, Vulkan (RTX 3090), Metal (Apple Silicon): tokenizer f32/f16/q8_0 vs the upstream token stream (86-87/87), front-end round trip vs the voice.gguf enrollment fixtures, engine-level cloned-vs-baked equivalence and cross-lingual bit-equality under a pinned trajectory, fail-closed guards, mel-extraction thread-safety.