Add Echo-TTS (community model) — draft, opening early per #54 - #180
Draft
5uck1ess wants to merge 8 commits into
Draft
Add Echo-TTS (community model) — draft, opening early per #54#1805uck1ess wants to merge 8 commits into
5uck1ess wants to merge 8 commits into
Conversation
Design for porting Echo-TTS (jordand/echo-tts-base, 2.8B DiT + Fish S1-DAC) into audio.cpp as a community model. Architecture verified against upstream source and safetensors headers, not inferred. Key findings: - EchoDiT: 24 blocks, d=2048, joint attention, adaLN, byte-level text - Fixed 640-latent / 29.72s generation window - Blockwise path subdivides that window, does not extend it - Decode and encode need near-disjoint Fish submodules - 303.6M of the Fish checkpoint is regenerable buffers, not weights Staged M0-M4 with per-milestone gates and a hard Definition of Ready before the PR leaves draft.
- Resolve RoPE theta open question (10000.0, complex-valued, model.py:9) - Add timestep embedding formula - Warn RTF vs RTFx are inverses (tts-bench vs audio.cpp conventions) - Cite the actual schema validator for the M0 gate - Define 'cosine' precisely (flattened 1-D, with max-abs-error) - Add decomposition note: M0+M1 in one plan, M2/M3/M4 separate
13 tasks, each gated on executed evidence: - M0 (T1-2): spec v1 registration + draft PR - M1 (T3-13): converter, parity dumps, GGUF, assets, tokenizer, text/speaker encoders, 24-block DiT, dual-CFG Euler sampler, PCA inverse + Fish decode, crop, warm bench Every stage gates on cosine >= 0.999 vs PyTorch before the next begins. Task 12 requires a human ear check - tensor parity cannot catch a wrong flattening-point crop. PR stays draft through M1; cloning still needs an injected .npy until M2 lands native speaker encoding.
- Repo has no CMakePresets.json; --preset would fail. Use scripts/build_linux.sh or cmake --build build/linux-cuda-release. - Existing build tree pins CMAKE_CUDA_ARCHITECTURES=75 (Turing) on an sm_86 card. Task 13 now reconfigures to 86 before measuring RTF, otherwise the number is invalid. - Note AUDIOCPP_MODEL_SET=full so the family compiles in automatically.
Baseline on this branch is registered_loaders=42, verified. Also note that a 'requires a schema v1 model contract' failure means a stale binary, not a broken tree.
Plan Task 1. Spec-backed loader (no loader.cpp), schema_version 1, capabilities.clone deliberately omits long_form until M3 earns it. Verified by execution: registered_loaders 42 -> 43, echo_tts appears as 'clon (offline)', spec parses. Fix over Codex's draft: guard used VoiceTaskKind::Tts, but the family registers as a clone task, so every real invocation would have thrown. Corrected to VoiceCloning, matching confucius4_tts:185. The registration gate could not catch this - --list-loaders enumerates loaders without constructing a session.
Documents the fixed 29.72s window, why blockwise does not extend it, the CC-BY-NC-SA output restriction (with the fish_audio precedent), benchmark provenance, options, and the WhisperD text format. Also drops an unrelated .gitignore change that was accidentally swept into an earlier docs commit, so the PR diff stays scoped.
The design spec and implementation plan are our working process, not content for audio.cpp. Preserved on the local echo-tts-planning branch and still on disk; just untracked here so the PR diff stays scoped to the actual contribution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — opening early per #54 to flag the area and get direction before writing the bulk of the port.
Adds Echo-TTS as a community model: an English
zero-shot voice-cloning TTS model. A 2.8B diffusion transformer generates 80-D latents in PCA space,
decoded to 44.1 kHz by the Fish S1-DAC autoencoder. Cloning needs a reference wav, no transcript.
Echo-TTS is on the candidate list in #34 (struck through, "contributions are welcome").
What's in this commit
Scaffolding only — no model math yet:
model_specs/echo_tts.json,schema_version: 1, using the generic spec-backed loader (noloader.cpp)docs/community_models/echo_tts.mdVerified on top of current
main:registered_loadersgoes 44 → 45, andecho_tts: clon (offline)appears in--list-loaders.Plan
Each milestone is gated on cosine ≥ 0.999 parity against the reference PyTorch implementation on a
fixed seed, plus a listening check.
This PR stays in draft until the full evidence pack exists: build/run commands, parity numbers,
path-test results across safetensors/F16/Q8_0, RTF < 1.0, stable VRAM across repeated requests, and
generated wavs.
How long text is handled
Echo generates at most 640 latents (640 x 2048 / 44100 = 29.7215 s). That is a property of the
model, not the port: longer text is spoken faster to fit rather than truncated, and the tokenizer
hard-truncates past 768 UTF-8 bytes. Upstream's blockwise sampler subdivides that window rather
than extending it (
sum(block_sizes) + continuation_length < 640), and upstream notes it "hasn'tbeen thoroughly tested".
So long text goes through the framework chunker, exactly as the rest of the repo does:
engine::text::split_text_chunksis already consumed by 19session.cppfiles, andchatterboxis the closest analogue — a clone family with cached speaker conditioning that concatenates chunk
audio via
append_audio_buffer. Echo caches its speaker latent once per session, so timbre isstable across seams by construction. Planned
kDefaultTextChunkSizeis 300 codepoints (~20 sof English), leaving headroom before the model starts compressing; overridable at runtime.
I've deliberately not claimed
long_formincapabilities— it appears nowhere in the C++, andonly 5 of 22 TTS/clone families declare it. The 17 that don't include
chatterbox,fish_audio,higgs_audio_tts,index_tts2,qwen3_tts,voxcpm2, andpocket_tts. Happy to add it if you'drather it be declared.
One question
Anything you'd want structured differently before there's a lot of code to restructure — file
layout, option naming, or whether this belongs in
community_modelsat all.Licence
Echo-TTS is CC-BY-NC-SA-4.0, and the restriction covers generated audio, not just the weights
(inherited from the Fish S1-DAC dependency). Flagging it explicitly rather than leaving it to be
inferred. There's precedent in-tree —
fish_audiocarries the identical restriction from theidentical dependency — and audio.cpp's own Apache 2.0 licence is unaffected, since weights are a
separate download. Documented in the model doc so users don't ship product on non-commercial output.
Why this model
Picked by comparing every model tracked in tts-bench — 62
local TTS models benchmarked across speed, objective scores, and blind human preference — against the
existing support table.
Caveats worth stating: the cloning arena averages ~30 games per model, so gaps under ~100 Elo are
noise, and the ranking uses a single reference clip. Echo is top-3 on votes and 2nd on objective
SIM, which are independent measurements.
Compute profile suits a GGUF port — ~2.8B at 1.35× RTFx and 9.4 GB VRAM in PyTorch on an RTX 3090,
so there's real work to amortise.