From e8ec7505630e4a2f55bb6dccb4709a9a907bfdef Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Thu, 30 Jul 2026 17:29:04 +0000 Subject: [PATCH 1/8] docs: add Echo-TTS port design spec Design for porting Echo-TTS (jordand/echo-tts-base, 2.8B DiT + Fish S1-DAC) into audio.cpp as a community model. Architecture verified against upstream source and safetensors headers, not inferred. Key findings: - EchoDiT: 24 blocks, d=2048, joint attention, adaLN, byte-level text - Fixed 640-latent / 29.72s generation window - Blockwise path subdivides that window, does not extend it - Decode and encode need near-disjoint Fish submodules - 303.6M of the Fish checkpoint is regenerable buffers, not weights Staged M0-M4 with per-milestone gates and a hard Definition of Ready before the PR leaves draft. --- .../specs/2026-07-30-echo-tts-port-design.md | 320 ++++++++++++++++++ 1 file changed, 320 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-30-echo-tts-port-design.md diff --git a/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md b/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md new file mode 100644 index 00000000..9d7b43ed --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md @@ -0,0 +1,320 @@ +# Echo-TTS port to audio.cpp — design + +Date: 2026-07-30 +Status: approved, pre-implementation +Target: community-model PR to `0xShug0/audio.cpp` + +--- + +## 1. Why this model + +### 1.1 Benchmark provenance + +This is not a model picked from a leaderboard screenshot. Echo-TTS has been independently +benchmarked in [tts-bench](https://github.com/5uck1ess/tts-bench) — a public benchmark tracking +**62 local TTS models** across three lenses (speed, objective scores, human preference) on three +rigs — and it was selected by comparing every tracked model against audio.cpp's existing support +table. The supporting data is already published and reproducible: + +- **Installed and run locally.** `venvs/echo/` with upstream source; both weight sets cached + (`jordand/echo-tts-base`, `jordand/fish-s1-dac-min`); a dedicated runner + (`runners/echo_runner.py`) documenting the exact upstream API and its gotchas. +- **Speed benched** on RTX 3090 CUDA, warm: **1.35× RTFx**, 4 326 ms TTFA, 9 357 MB peak VRAM. +- **Objectively scored** over the bench prompt set: **16 rows** in `scoring/scores.csv` across + default and cloning lenses, via seed-tts-eval-style ASR + speaker verification. +- **Publicly auditioned**: generated wavs published to gh-pages and playable in the Listen lens. +- **Voted on blind**, twice — a frozen 397-vote pairwise study and an ongoing public arena that has + since collected 738 cloning votes and 1 415 default-voice votes. + +That measurement history is what makes the recommendation trustworthy, and it should be cited in the +PR body: the port is proposed because Echo *measured* well against 61 alternatives, not because it +looked promising. + +### 1.2 The result + +Echo-TTS is the highest-value model absent from audio.cpp, on three independent signals: + +| Signal | Value | Source | +|---|---|---| +| Human-preference Elo (cloning) | **1162, #3 of 40** on 35 games | tts-bench live arena, 738 cloning votes | +| Speaker similarity (SIM) | **0.836 — 2nd of 41** scored models | `tts-bench/scoring/scores.csv` | +| Frozen blind study | **21-1-6**, near-tied #1 | `tts-bench/docs/cloning.md`, 397 votes | +| UTMOS / WER | 4.21 / 7.45 % | same | +| Output rate | **44.1 kHz** | model card | + +Two qualifications, stated up front for honesty: the cloning arena averages ~30 games per model, so +gaps under ~100 Elo points are noise (the 1 415-vote default lens is firmer), and the whole cloning +ranking rests on a single reference clip (`chris_hemsworth_15s.wav`). Echo's position is robust to +both — it is top-3 on votes *and* top-2 on objective SIM, which are independent measurements. + +It is also **explicitly open for contribution**. Upstream issue #34 lists `~~echo-tts~~` struck +through under "Candidate models", with the legend: *"For models crossed out: I will not impl these +models myself, but contributions are welcome."* Struck-through entries carry **zero duplication +risk**; un-struck candidates (Magpie, LongCat, Soprano, MiraTTS) may still be maintainer work. + +Verified absent: no `echo`/`echodit`/`jordand` match anywhere in `src/`, `include/`, `docs/`, +`model_specs/`, `tools/`, or `README.md`; no PR (open/closed/draft) in 200+; no branch; GitHub code +search returns 0. + +Compute profile suits the framework. Echo is ~2.8 B at 1.35× RTFx and 9.4 GB VRAM in PyTorch — +heavy enough that GGUF and session amortisation pay off. (Contrast Kokoro, whose `preview/kokoro` +branch measures **0.20×** on the long-lived-session chart — 5× *slower* than Python — because an +82 M model has nothing to amortise.) + +--- + +## 2. Verified architecture + +All facts below were read from source at `tts-bench/venvs/echo/src/` and from safetensors headers. +Anything not established by those files is marked OPEN in §9 rather than guessed. + +### 2.1 Pipeline + +``` +reference wav + → decode ≤300 s → mono → resample 44 100 Hz → divide by max(|peak|, 1) + → truncate ≤ 6400×2048 samples; chunk at 640×2048; zero-pad final chunk + → fish_ae.encode_zq → PCA project 1024→80 → × latent_scale + → speaker_latent [1, Ls, 80], speaker_mask [1, Ls], Ls mod 4 == 0 + +text + → WhisperD normalisation: prepend "[S1] "; colons/semicolons/emdashes → commas + → UTF-8 *byte* tokens (256-entry vocab) + → text_encoder + +EchoDiT: 40 Euler steps in 80-D PCA space, latents [1, 640, 80] + → PCA⁻¹ → quantizer.post_module → quantizer.upsample → decoder + → waveform 44 100 Hz + → crop at flattening point (20-frame std/mean scan, cut at frame × 2048) +``` + +`640 × 2048 / 44100 = 29.7215 s` — the fixed generation window. + +### 2.2 EchoDiT + +| Property | Value | +|---|---| +| Trunk depth | 24 blocks | +| Hidden dim | 2048 | +| Attention | joint: self + text KV + speaker KV (+ latent-prefix KV, blockwise only) | +| MLP | SwiGLU | +| Conditioning | adaLN on both attention and MLP, driven by timestep | +| Positional | RoPE, **rotating only half the heads** | +| Norm | RMSNorm, FP32 accumulation | + +Text frontend is **byte-level** — no phonemizer, no G2P, no external pronunciation dependency. +This is a significant scope win and removes the class of dependency problem that sank Kokoro. + +### 2.3 Parameter inventory + +| Component | Params | Needed for inference | +|---|---:|---| +| EchoDiT total | 2 800 742 736 | yes | +| — trunk joint attention (24) | 880 902 144 | yes | +| — trunk MLP (24) | 868 220 928 | yes | +| — attention adaLN (24) | 75 644 928 | yes | +| — MLP adaLN (24) | 75 644 928 | yes | +| — text_encoder | 294 000 640 | yes | +| — speaker_encoder | 294 083 840 | yes (when cloning) | +| — **latent_encoder** | 294 083 840 | **blockwise/long-form only** | +| — misc (timestep MLP, projections, norms) | 18 161 488 | yes | +| PCA state | 82 945 elements | yes | +| Fish S1-DAC checkpoint | 694 993 282 elements | — | +| — **trainable weights only** | **391 430 530** | — | +| — `freqs_cis` + `causal_mask` buffers | 303 562 752 | **regenerate at runtime, do not ship** | + +PCA: `pca_components [80,1024]`, `pca_mean [1024]`, `latent_scale [1] = 0.0555555559694767` (= 1/18). + +### 2.4 The decode/encode asymmetry + +Decode and encode need nearly disjoint Fish submodules: + +| Path | Modules | Approx weights | +|---|---|---:| +| **Decode** (generation) | PCA⁻¹, `quantizer.post_module`, `quantizer.upsample`, `decoder` | ~184 M | +| **Encode** (speaker ref) | `encoder`, `quantizer.downsample`, `quantizer.pre_module`, semantic RVQ + 9× residual RVQ, PCA forward | ~207 M | + +The decode path is entirely matmul/conv/transformer. The encode path needs RVQ nearest-neighbour +search, rated **Hard** to port. This asymmetry is the basis for the milestone split in §4. + +Note: `encode_zq` as written runs the *full* quantizer forward, then discards the result and +re-derives from the selected codes. `post_module` and `upsample` inside that first call can be +skipped — numerically equivalent, since only `codes` are consumed. + +### 2.5 Sampler + +`sample_euler_cfg_independent_guidances`: 40 Euler steps, **dual independent CFG** — `cfg_scale_text` +3.0 and `cfg_scale_speaker` 8.0 (5.0 in the blockwise example) — gated to `t ∈ [cfg_min_t=0.5, +cfg_max_t=1.0]`, `truncation_factor` 0.8. Unconditioning is **mask-based**, not zeroed encoder +states. Optional `speaker_kv_scale` ("Force Speaker", default 1.5 when enabled) corrects speaker +drift on out-of-distribution text. + +--- + +## 3. Long-form: rolling latent continuation + +**Blockwise does not extend past 640.** Verified directly: + +- `inference_blockwise.py:161` — `block_sizes=[128,128,64], # (sums to 320, ~15 seconds; supports up to 640)` +- `inference_blockwise.py:194-195` — `sum(block_sizes) + continuation_latent.shape[1] should be < 640` +- `README.md:122-124` — *"prefix and continuation are up to 30 seconds combined"*; *"Blockwise + functionality hasn't been thoroughly tested"* + +Blockwise **subdivides** one ≤30 s window; it does not extend it. Nor is there any text-compression +transform — long text fitting into 30 s is *learned* behaviour via global attention, and the +tokenizer hard-truncates past 768 UTF-8 bytes (`inference.py:146-149`). + +**Design:** carry the tail latents of chunk N directly into chunk N+1 as the continuation prefix. +Because we generate latents natively, this needs **no decode→re-encode round trip**. Each call +resets the window; the constraint is `prefix + new < 640` per call. + +This preserves prosody across joins, which crossfading cannot. Requirements and caveats: + +- Requires `latent_encoder` (+294 M) plus `wk_latent`/`wv_latent` — exactly what + `delete_blockwise_modules=True` strips. +- The prompt for chunk N+1 **must include the carried prefix's transcript**. +- Total prefix length must be divisible by 4 (speaker patch size, `model.py:458-459`). +- Upstream calls blockwise under-tested. **This must be disclosed in the PR, not discovered by the + maintainer.** + +Fallback if M3 fails validation: sentence-boundary chunking with `cross_fade_duration_sec` seams, +and drop the `long_form` capability claim. + +--- + +## 4. Milestones + +Each milestone has a gate. **No milestone is "done" on report — only on executed evidence.** + +### M0 — spec + draft PR +- `model_specs/echo_tts.json`, `"schema_version": 1`, placed in `model_specs/` (not `model_specs_v1/`). +- `capabilities` **omits `long_form`** until M3 earns it. +- Draft PR opened, explicitly raising: the 29.72 s window, the blockwise-untested caveat, and the + CC-BY-NC-SA output-licence constraint. +- Gate: spec validates against the framework schema; PR open and marked **draft**. + +### M1 — decode path, parity-gated +- GGUF conversion script; EchoDiT minus `latent_encoder`; PCA⁻¹; Fish decode path. +- Speaker latent injected from a `.npy` dumped by PyTorch — validates the hard 2.5 B without RVQ. +- Gate: per-tensor cosine ≥ 0.999 vs reference on fixed seed; generated wav audibly correct. + +### M2 — native speaker encoding +- Fish encoder + downsample + pre_module + semantic/residual RVQ + PCA forward. +- Gate: speaker latent from C++ matches PyTorch `encode_zq` → PCA output, cosine ≥ 0.999; + end-to-end clone from a raw wav with no Python in the loop. + +### M3 — long-form +- Rolling latent continuation per §3. +- Gate: `tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json` renders correctly and is + listened to end-to-end for seam artefacts. Only on success does `long_form` enter `capabilities`. + +### M4 — quantisation, performance, docs +- Q8_0 and F16 GGUF; `docs/community_models/echo_tts.md`; warm-bench test. +- Gate: **RTF < 1.0** (the explicit community bar); VRAM stable across repeated requests. + +--- + +## 5. Definition of Ready — the PR does not leave draft until all of these pass + +This is a hard gate, mirroring audio.cpp's stated review bar (issue #54 and README §36: *"exact +build/run commands, model paths or package ids, generated outputs, parity or path-test results, and +relevant performance or memory notes"*). + +1. **Builds clean** on Linux CUDA release; no new warnings in our files. +2. **Parity**: per-tensor cosine ≥ 0.999 against PyTorch on a fixed seed, for DiT output, PCA⁻¹, + Fish decode, and (M2+) speaker encode. Numbers recorded in the PR. +3. **Path tests**: the family passes the CLI path-test matrix for safetensors, F16 GGUF, Q8_0 GGUF. +4. **Long-form**: the shared long-form clone case renders and is auditioned for seam artefacts — + or `long_form` is not claimed and the limit is documented. +5. **RTF < 1.0** measured on the RTX 3090, warm, with the command line included. +6. **VRAM stable** across ≥5 consecutive requests (no growth); `mem_saver` used if tuning is needed, + never to mask a leak. +7. **Generated wavs attached** for both default-reference and custom-reference cloning. +8. **Licence disclosed**: CC-BY-NC-SA-4.0 on weights *and outputs*. +9. **Independent review**: Codex authored → Claude reviews. Reviewer ≠ author, always. + +Only when 1–9 are green does the PR move from draft to ready-for-review. + +--- + +## 6. audio.cpp integration surface + +Follows Confucius4-TTS, the spec-v1 exemplar named in issue #128. + +``` +model_specs/echo_tts.json # schema_version 1 +src/community_models/echo_tts/*.cpp +include/engine/community_models/echo_tts/*.h +tests/echo_tts/echo_tts_warm_bench.cpp +docs/community_models/echo_tts.md +CMakeLists.txt # audiocpp_add_model(echo_tts SOURCES … INCLUDES … LOADERS …) +``` + +- **No `loader.cpp`.** Spec-v1 models use the generic spec-backed loader (issue #128). +- **Loader symbol is `engine::models::echo_tts::make_echo_tts_loader`** — namespace `models`, *not* + `community_models`, matching `inflect_v2`. Getting this wrong is a link error. +- **GGUF preferred over safetensors**, self-contained with the spec embedded; safetensors optional. +- **Normalised option names** (framework-validated): reference audio is `target_voice`, durations are + `*_sec`, chunking uses `audio_chunk_threshold_sec` / `audio_chunk_duration_sec` / + `cross_fade_duration_sec`. Do not copy Python names into the spec. + +Proposed options: `cfg_scale_text`, `cfg_scale_speaker`, `num_steps`, `truncation_factor`, +`speaker_kv_scale`, `seed`, `target_voice`. + +--- + +## 7. Implementation traps + +Each of these would cost days if hit blind. + +1. **`autoencoder.py:943-965` — decoder transformer that never executes.** It exists only as an + unregistered local variable. Porting the apparent configuration would be silently wrong. +2. **Weight normalisation**: most DAC convolutions store weight-norm parameters, not ready conv + weights. Fold at conversion time. +3. **FP32 boundaries are load-bearing**: RMSNorm and adaLN accumulate in FP32; the sampler, PCA, and + Fish weights are FP32 while Echo weights are BF16. Low-precision-only normalisation diverges. +4. **Do not serialise `freqs_cis` / `causal_mask`** into GGUF (303.6 M elements). Regenerate. +5. **Half-head RoPE**: the trunk rotates only half the heads — unusual, easy to get wrong. +6. **Snake activation** in the DAC likely needs a composed or custom kernel. +7. **Causal conv padding/cropping** computes right-padding from runtime length; transposed conv crops + asymmetrically. Off-by-one here is silent audio corruption. +8. **Shape divisibility**: speaker and prefix latents reshape in groups of 4. +9. **Mask-based unconditioning**: CFG unconditions via masks, not zeroed encoder states. + +--- + +## 8. Testing strategy + +- **Parity harness**: dump reference intermediates from PyTorch (fixed seed) to `.npy`; C++ loads and + compares per-stage with cosine + max-abs-error. Stage boundaries: text_encoder out, speaker_encoder + out, per-block DiT out (first/middle/last), final latents, PCA⁻¹ out, decoder out. +- **Bit-exactness is not the goal.** Gaussian RNG is device-specific; aim for statistical equivalence + on the noise and ≥0.999 cosine downstream. +- **Ear check is mandatory** at M1, M2, M3. Cosine can pass while audio is wrong (the flattening-point + crop is a host-side loop, not covered by tensor parity). +- **Regression**: reuse the bench's `chris_hemsworth_15s.wav` reference so output is directly + comparable to the 16 existing scored Echo rows in tts-bench. + +--- + +## 9. Open questions + +- `latent_scale` is resolved (1/18) but its *derivation* is unverified; confirm it is applied on both + the forward and inverse PCA legs consistently. +- Exact RoPE theta for the Echo trunk — read from source at implementation time, do not assume. +- Whether `quantizer.post_module` + `upsample` can be skipped in the M2 encode call without drift, as + §2.4 suggests. Verify numerically before optimising. +- Whether the maintainer will accept a `long_form` implementation built on an upstream path its own + author calls under-tested. Raise in M0. + +--- + +## 10. Licence + +Echo-TTS weights **and generated outputs** are CC-BY-NC-SA-4.0 — the output constraint is forced by +the Fish S1-DAC dependency. This is stricter than a weights-only NC licence and must be stated +plainly in `docs/community_models/echo_tts.md` and in the PR body. + +Precedent exists in-tree: `higgs_audio_tts` (Research NC) and `omnivoice` (Apache code / +CC-BY-NC weights). The *output* restriction appears to be new for audio.cpp — flag it explicitly +rather than letting it be inferred. From 2bfabdc6c41c246f509a31da5ad92964a454c6af Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Thu, 30 Jul 2026 17:30:17 +0000 Subject: [PATCH 2/8] docs: spec self-review fixes - Resolve RoPE theta open question (10000.0, complex-valued, model.py:9) - Add timestep embedding formula - Warn RTF vs RTFx are inverses (tts-bench vs audio.cpp conventions) - Cite the actual schema validator for the M0 gate - Define 'cosine' precisely (flattened 1-D, with max-abs-error) - Add decomposition note: M0+M1 in one plan, M2/M3/M4 separate --- .../specs/2026-07-30-echo-tts-port-design.md | 23 +++++++++++++++---- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md b/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md index 9d7b43ed..96dfe1d0 100644 --- a/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md +++ b/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md @@ -19,7 +19,11 @@ table. The supporting data is already published and reproducible: - **Installed and run locally.** `venvs/echo/` with upstream source; both weight sets cached (`jordand/echo-tts-base`, `jordand/fish-s1-dac-min`); a dedicated runner (`runners/echo_runner.py`) documenting the exact upstream API and its gotchas. -- **Speed benched** on RTX 3090 CUDA, warm: **1.35× RTFx**, 4 326 ms TTFA, 9 357 MB peak VRAM. +- **Speed benched** on RTX 3090 CUDA, warm: **1.35× RTFx** (= RTF 0.74), 4 326 ms TTFA, 9 357 MB + peak VRAM. **Units warning:** tts-bench reports **RTFx** (higher = faster); audio.cpp's README + tabulates **RTF** (wall ÷ audio, lower = faster) alongside a separate "x faster than real time" + column. They are inverses. Echo's PyTorch 1.35× RTFx already satisfies the community RTF < 1.0 + bar before any GGUF work; do not invert these in the PR. - **Objectively scored** over the bench prompt set: **16 rows** in `scoring/scores.csv` across default and cloning lenses, via seed-tts-eval-style ASR + speaker verification. - **Publicly auditioned**: generated wavs published to gh-pages and playable in the Listen lens. @@ -99,8 +103,9 @@ EchoDiT: 40 Euler steps in 80-D PCA space, latents [1, 640, 80] | Attention | joint: self + text KV + speaker KV (+ latent-prefix KV, blockwise only) | | MLP | SwiGLU | | Conditioning | adaLN on both attention and MLP, driven by timestep | -| Positional | RoPE, **rotating only half the heads** | +| Positional | RoPE, theta **10000.0**, complex-valued, **rotating only half the heads** (`model.py:9`) | | Norm | RMSNorm, FP32 accumulation | +| Timestep embedding | sinusoidal, `1000 · exp(−log(10000)·k)` (`model.py:35-40`) | Text frontend is **byte-level** — no phonemizer, no G2P, no external pronunciation dependency. This is a significant scope win and removes the class of dependency problem that sank Kokoro. @@ -186,12 +191,20 @@ and drop the `long_form` capability claim. Each milestone has a gate. **No milestone is "done" on report — only on executed evidence.** +**Decomposition note.** This spec deliberately covers the whole arc so the end state is agreed up +front, but it is too large for one implementation plan. M1 alone (GGUF conversion + a 2.5 B DiT + +the Fish decode stack, parity-gated) is a full plan on its own. Plan boundaries: **M0 + M1 together** +in the first plan; **M2**, **M3**, and **M4** each get their own plan written after the preceding +gate is green. Re-plan rather than extrapolate — M1's parity results will change what M2 should look +like. + ### M0 — spec + draft PR - `model_specs/echo_tts.json`, `"schema_version": 1`, placed in `model_specs/` (not `model_specs_v1/`). - `capabilities` **omits `long_form`** until M3 earns it. - Draft PR opened, explicitly raising: the 29.72 s window, the blockwise-untested caveat, and the CC-BY-NC-SA output-licence constraint. -- Gate: spec validates against the framework schema; PR open and marked **draft**. +- Gate: spec passes the framework schema validator (`src/framework/model_spec/schema.cpp:674-680` + checks `schema_version`); PR open and marked **draft**. ### M1 — decode path, parity-gated - GGUF conversion script; EchoDiT minus `latent_encoder`; PCA⁻¹; Fish decode path. @@ -221,7 +234,8 @@ build/run commands, model paths or package ids, generated outputs, parity or pat relevant performance or memory notes"*). 1. **Builds clean** on Linux CUDA release; no new warnings in our files. -2. **Parity**: per-tensor cosine ≥ 0.999 against PyTorch on a fixed seed, for DiT output, PCA⁻¹, +2. **Parity**: cosine similarity ≥ 0.999 against PyTorch on a fixed seed, computed over each + tensor flattened to 1-D, reported alongside max-absolute-error. Stages: DiT output, PCA⁻¹, Fish decode, and (M2+) speaker encode. Numbers recorded in the PR. 3. **Path tests**: the family passes the CLI path-test matrix for safetensors, F16 GGUF, Q8_0 GGUF. 4. **Long-form**: the shared long-form clone case renders and is auditioned for seam artefacts — @@ -301,7 +315,6 @@ Each of these would cost days if hit blind. - `latent_scale` is resolved (1/18) but its *derivation* is unverified; confirm it is applied on both the forward and inverse PCA legs consistently. -- Exact RoPE theta for the Echo trunk — read from source at implementation time, do not assume. - Whether `quantizer.post_module` + `upsample` can be skipped in the M2 encode call without drift, as §2.4 suggests. Verify numerically before optimising. - Whether the maintainer will accept a `long_form` implementation built on an upstream path its own From c8a21c0aea39f097f1d74596b284afc7508630e5 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 31 Jul 2026 17:16:56 +0000 Subject: [PATCH 3/8] docs: add M0+M1 implementation plan for Echo-TTS port 13 tasks, each gated on executed evidence: - M0 (T1-2): spec v1 registration + draft PR - M1 (T3-13): converter, parity dumps, GGUF, assets, tokenizer, text/speaker encoders, 24-block DiT, dual-CFG Euler sampler, PCA inverse + Fish decode, crop, warm bench Every stage gates on cosine >= 0.999 vs PyTorch before the next begins. Task 12 requires a human ear check - tensor parity cannot catch a wrong flattening-point crop. PR stays draft through M1; cloning still needs an injected .npy until M2 lands native speaker encoding. --- .../plans/2026-07-30-echo-tts-m0-m1.md | 771 ++++++++++++++++++ 1 file changed, 771 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md diff --git a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md new file mode 100644 index 00000000..5d675dc5 --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md @@ -0,0 +1,771 @@ +# Echo-TTS Port — M0 + M1 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Land a draft PR declaring the `echo_tts` family, then build a working offline decode path that generates 44.1 kHz audio from text + a pre-computed speaker latent, proven by ≥0.999 cosine parity against PyTorch and by ear. + +**Architecture:** Echo-TTS is a 24-block, d=2048 diffusion transformer operating in 80-D PCA space, decoded to waveform by the Fish S1-DAC. M1 deliberately ports only the **decode** half — the encode half (Fish encoder + RVQ, rated Hard) is deferred to M2 by injecting the speaker latent from a `.npy` dumped by the reference implementation. Each stage is landed behind its own parity gate so a numerical regression is caught at the stage that caused it, not at the end. + +**Tech Stack:** C++20, ggml, CMake; Python 3.12 + PyTorch/safetensors for conversion and parity dumps; `audiocpp_gguf` for GGUF emission. + +**Spec:** `docs/superpowers/specs/2026-07-30-echo-tts-port-design.md` + +## Global Constraints + +- Family slug is `echo_tts` everywhere: spec filename, directory names, CMake target, test dir. +- Loader symbol is `engine::models::echo_tts::make_echo_tts_loader` — namespace `models`, **not** `community_models`, even though sources live under `src/community_models/`. Mismatch is a link error. +- Spec goes in `model_specs/echo_tts.json` with `"schema_version": 1`. Do **not** create a `model_specs_v1/` copy. +- **No `loader.cpp`.** Spec-v1 models use the generic spec-backed loader. +- Option names are framework-validated: reference audio is `target_voice`; durations end `_sec`; never copy Python names into the spec. +- `capabilities` must **not** claim `long_form` in M0/M1. It is earned in M3 or not at all. +- Generation window is fixed: 640 latents × 2048 samples ÷ 44100 Hz = **29.7215 s**. +- `latent_scale = 0.0555555559694767` (= 1/18). `pca_components` is `[80,1024]`, `pca_mean` is `[1024]`. +- RoPE theta is `10000.0`, complex-valued, and **only half the heads are rotated**. +- RMSNorm and adaLN accumulate in **FP32**; Echo weights are BF16; sampler/PCA/Fish weights are FP32. +- Never serialise `freqs_cis` or `causal_mask` into GGUF (303.6 M elements). Regenerate at runtime. +- **Parity gate:** cosine similarity ≥ 0.999 over each tensor flattened to 1-D, reported with max-absolute-error. A stage is not done until its gate is green **when run**, not when reported. +- Reference implementation for all parity work: `/home/ryzen/LocalDev/tts-bench/venvs/echo/src/`, weights in `~/.cache/huggingface/hub/models--jordand--echo-tts-base` and `…--fish-s1-dac-min`. +- Hardware: RTX 3090 24 GB, CUDA. Build preset `linux-cuda-release`. + +--- + +## File Structure + +| Path | Responsibility | +|---|---| +| `model_specs/echo_tts.json` | Family metadata, tasks, options, packages. Single source of truth. | +| `tests/echo_tts/convert_echo_tts_weights.py` | Reference checkpoints → audio.cpp safetensors bundle → optional GGUF. | +| `tests/echo_tts/dump_echo_reference.py` | Dumps per-stage reference intermediates to `.npy` for parity. | +| `tests/echo_tts/compare_parity.py` | Cosine + max-abs-error comparator, exit non-zero on failure. | +| `tests/echo_tts/echo_tts_warm_bench.cpp` | C++ warm bench over the shared cases. | +| `tests/echo_tts/echo_tts_warm_bench_cases.json` | Shared case definitions. | +| `include/engine/community_models/echo_tts/assets.h` | Tensor handles resolved from the spec. | +| `include/engine/community_models/echo_tts/types.h` | POD config + request structs. | +| `include/engine/community_models/echo_tts/tokenizer_text.h` | WhisperD normalisation + UTF-8 byte tokenisation. | +| `include/engine/community_models/echo_tts/encoders.h` | Text and speaker encoder runtimes. | +| `include/engine/community_models/echo_tts/dit.h` | 24-block trunk forward. | +| `include/engine/community_models/echo_tts/sampler.h` | Euler loop + dual independent CFG. | +| `include/engine/community_models/echo_tts/fish_decoder.h` | PCA⁻¹ + post_module + upsample + decoder. | +| `include/engine/community_models/echo_tts/session.h` | Session wiring, loader factory. | +| `src/community_models/echo_tts/*.cpp` | Implementations, one per header. | + +Split rationale: each unit has its own parity gate, so each gets its own file. `dit.cpp` will be the largest; if it exceeds ~1500 lines, split blocks from the trunk driver. + +--- + +## Task 1: Model spec and family registration + +**Files:** +- Create: `model_specs/echo_tts.json` +- Modify: `CMakeLists.txt` (add `audiocpp_add_model(echo_tts …)` near the other community models, ~line 454) +- Create: `src/community_models/echo_tts/session.cpp`, `include/engine/community_models/echo_tts/session.h` + +**Interfaces:** +- Produces: `engine::models::echo_tts::make_echo_tts_loader()` → `std::shared_ptr` + +- [ ] **Step 1: Write the spec** + +Create `model_specs/echo_tts.json`. Model the shape on `model_specs/confucius4_tts.json`. Required content: + +```json +{ + "schema_version": 1, + "family": "echo_tts", + "display_name": "Echo-TTS", + "description": "Echo-TTS is an English zero-shot voice-cloning TTS model packaged for audio.cpp. A 2.8B diffusion transformer generates 80-D latents in PCA space which the Fish S1-DAC decodes to 44.1 kHz audio. Generation is a fixed 29.72 s window (640 latents).", + "category": "tts", + "status": "experimental", + "tasks": ["clone"], + "modes": ["offline"], + "languages": ["en"], + "runtime": { "tags": ["gguf"] }, + "capabilities": { "clone": ["speaker_reference"] }, + "options": { + "request": [ + { "name": "target_voice", "type": "string", "description": "Reference audio path for zero-shot cloning. Wav only; no transcript required.", "required": false }, + { "name": "cfg_scale_text", "type": "float", "description": "Classifier-free guidance scale on the text condition.", "required": false, "min": 0.0, "default": 3.0 }, + { "name": "cfg_scale_speaker", "type": "float", "description": "Classifier-free guidance scale on the speaker condition.", "required": false, "min": 0.0, "default": 8.0 }, + { "name": "num_steps", "type": "int", "description": "Euler sampler steps.", "required": false, "min": 1, "default": 40 }, + { "name": "truncation_factor", "type": "float", "description": "Initial-noise truncation factor.", "required": false, "min": 0.0, "max": 1.0, "default": 0.8 }, + { "name": "speaker_kv_scale", "type": "float", "description": "Force-speaker KV scaling. 1.0 disables; 1.5 is the upstream default when enabled.", "required": false, "min": 1.0, "default": 1.0 }, + { "name": "seed", "type": "int", "description": "RNG seed for the initial latent.", "required": false, "default": 0 } + ] + } +} +``` + +Note `capabilities.clone` deliberately omits `long_form`. + +- [ ] **Step 2: Write a spec-load test** + +Create `tests/echo_tts/echo_tts_warm_bench_cases.json` with one placeholder-free case: + +```json +{ + "default_clone": { + "requests": [ + { + "id": "chris_ref_p1", + "target_voice": "reference/chris_hemsworth_15s.wav", + "text": "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm.", + "seed": 0 + } + ] + } +} +``` + +- [ ] **Step 3: Verify the spec parses** + +Run: +```bash +python3 -c "import json; d=json.load(open('model_specs/echo_tts.json')); assert d['schema_version']==1; assert 'long_form' not in d['capabilities']['clone']; print('spec ok:', d['family'])" +``` +Expected: `spec ok: echo_tts` + +- [ ] **Step 4: Add the minimal session so the family links** + +`include/engine/community_models/echo_tts/session.h` declares: + +```cpp +#pragma once +#include "engine/framework/model_spec/metadata.h" +#include "engine/framework/runtime/session_base.h" +#include + +namespace engine::models::echo_tts { + +std::shared_ptr make_echo_tts_loader(); + +class EchoTtsSession final + : public runtime::RuntimeSessionBase, + public runtime::IOfflineVoiceTaskSession { +public: + EchoTtsSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr contract); + ~EchoTtsSession() override; + + std::string family() const override; + runtime::VoiceTaskKind task_kind() const override; + runtime::RunMode run_mode() const override; + void prepare(const runtime::SessionPreparationRequest & request) override; + runtime::TaskResult run(const runtime::TaskRequest & request) override; + void reset() override; + +private: + runtime::TaskSpec task_; + std::shared_ptr contract_; +}; + +} // namespace engine::models::echo_tts +``` + +Implement `run()` in `session.cpp` to return 1.0 s of silence at 44 100 Hz for now. This proves the plumbing before any math exists. + +- [ ] **Step 5: Wire CMake** + +Add to `CMakeLists.txt` beside the other community models: + +```cmake +audiocpp_add_model(echo_tts + SOURCES + src/community_models/echo_tts/session.cpp + INCLUDES + engine/community_models/echo_tts/session.h + LOADERS + engine::models::echo_tts::make_echo_tts_loader +) +``` + +- [ ] **Step 6: Build and confirm the family registers** + +Run: +```bash +cmake --build --preset linux-cuda-release --target audiocpp_cli -j +./build/linux-cuda-release/bin/audiocpp_cli --list-families | grep echo_tts +``` +Expected: `echo_tts` appears. If it links-errors on `make_echo_tts_loader`, the namespace is wrong — it must be `engine::models::echo_tts`, not `engine::community_models::echo_tts`. + +- [ ] **Step 7: Commit** + +```bash +git add model_specs/echo_tts.json tests/echo_tts/ include/engine/community_models/echo_tts/ src/community_models/echo_tts/ CMakeLists.txt +git commit -m "feat(echo_tts): register family with spec v1 and silence stub" +``` + +--- + +## Task 2: Draft PR + +**Files:** +- Create: `docs/community_models/echo_tts.md` + +- [ ] **Step 1: Write the model doc** + +`docs/community_models/echo_tts.md` must state, without softening: +- Fixed 29.7215 s generation window; text beyond it is spoken faster, and the tokenizer hard-truncates past 768 UTF-8 bytes. +- Long-form is **not** supported in this PR. +- Licence: **CC-BY-NC-SA-4.0 on weights *and generated outputs*** — the output restriction is forced by the Fish S1-DAC dependency and is stricter than a weights-only NC licence. +- Benchmark provenance: #3 of 40 on cloning Elo (738 votes), SIM 0.836 (2nd of 41), UTMOS 4.21, WER 7.45 %, measured in tts-bench across 62 tracked models. + +- [ ] **Step 2: Push the branch** + +```bash +git push -u origin echo-tts-port +``` + +- [ ] **Step 3: Open the PR as a draft** + +```bash +gh pr create --repo 0xShug0/audio.cpp --draft \ + --title "Add Echo-TTS (community model) — WIP" \ + --body-file docs/community_models/echo_tts.md +``` + +The body must explicitly ask the maintainer three questions: +1. Is a fixed 29.72 s window acceptable for a community model, given `long_form` is not claimed? +2. Is the CC-BY-NC-SA **output** restriction acceptable in-tree? +3. If long-form is required, is a rolling latent-continuation approach acceptable given upstream calls its blockwise path "not thoroughly tested"? + +- [ ] **Step 4: Verify it is actually a draft** + +```bash +gh pr view --repo 0xShug0/audio.cpp --json isDraft,title -q '.isDraft' +``` +Expected: `true`. **The PR stays draft until every clause of Definition of Ready in the spec §5 is green.** + +--- + +## Task 3: Weight converter + +**Files:** +- Create: `tests/echo_tts/convert_echo_tts_weights.py` + +**Interfaces:** +- Produces: `models/echo-tts/audio_cpp/model.safetensors` with the tensor names consumed by Task 6. + +- [ ] **Step 1: Write the converter** + +Model it on `tests/confucius4_tts/convert_confucius4_tts_weights.py`. It must: +- Read `~/.cache/huggingface/hub/models--jordand--echo-tts-base` and `…--fish-s1-dac-min`. +- **Drop** every `latent_encoder.*`, `latent_norm*`, `*.wk_latent`, `*.wv_latent` tensor (blockwise-only; −294 M). +- **Drop** every `freqs_cis` and `causal_mask` buffer (regenerated at runtime; −303.6 M elements). +- **Fold weight normalisation** into static conv weights for the Fish decoder: for each conv storing `weight_g`/`weight_v`, emit `weight = weight_g * weight_v / ||weight_v||` over the norm axis, and drop the `_g`/`_v` pair. +- Copy `pca_components`, `pca_mean`, `latent_scale` through unchanged as FP32. +- Write a JSON sidecar recording every dropped key, so the drop is auditable. + +- [ ] **Step 2: Run it** + +```bash +cd /home/ryzen/LocalDev/audio.cpp +uv run --with torch --with safetensors --with numpy \ + python tests/echo_tts/convert_echo_tts_weights.py --output-dir models/echo-tts/audio_cpp +``` + +- [ ] **Step 3: Verify the drop maths** + +```bash +python3 -c " +import json,struct +f='models/echo-tts/audio_cpp/model.safetensors' +h=json.loads(open(f,'rb').read(8+struct.unpack(' int: + p = argparse.ArgumentParser() + p.add_argument("--ref", required=True) + p.add_argument("--got", required=True) + p.add_argument("--min-cosine", type=float, default=0.999) + a = p.parse_args() + ref = np.load(a.ref).astype(np.float64).ravel() + got = np.load(a.got).astype(np.float64).ravel() + if ref.shape != got.shape: + print(f"FAIL shape {ref.shape} vs {got.shape}") + return 1 + cos = float(ref @ got / (np.linalg.norm(ref) * np.linalg.norm(got))) + mae = float(np.max(np.abs(ref - got))) + ok = cos >= a.min_cosine + print(f"{'PASS' if ok else 'FAIL'} cosine={cos:.6f} max_abs_err={mae:.6e} n={ref.size}") + return 0 if ok else 1 + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 2: Write the dumper** + +`dump_echo_reference.py` loads the reference implementation exactly as `tts-bench/runners/echo_runner.py` does — including the `torchcodec`/`torchaudio` module stubs documented in that runner's docstring — seeds with `rng_seed=0`, runs one generation for the Task 1 case text against `reference/chris_hemsworth_15s.wav`, and saves each listed intermediate via forward hooks. + +- [ ] **Step 3: Run it** + +```bash +uv run --with torch --with numpy --with librosa --with soundfile \ + python tests/echo_tts/dump_echo_reference.py --out tests/echo_tts/parity +``` + +- [ ] **Step 4: Verify the dumps are sane** + +```bash +python3 -c " +import numpy as np, glob +for f in sorted(glob.glob('tests/echo_tts/parity/*.npy')): + a=np.load(f); print(f.split('/')[-1], a.shape, a.dtype, 'finite' if np.isfinite(a).all() else 'HAS NAN/INF') +" +``` +Expected: every file `finite`; `speaker_latent.npy` has shape `(1, Ls, 80)` with `Ls % 4 == 0`; `latents_final.npy` has shape `(1, 640, 80)`. + +- [ ] **Step 5: Sanity-check the comparator against itself** + +```bash +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/latents_final.npy --got tests/echo_tts/parity/latents_final.npy +``` +Expected: `PASS cosine=1.000000 max_abs_err=0.000000e+00 …` + +- [ ] **Step 6: Commit** + +```bash +git add tests/echo_tts/dump_echo_reference.py tests/echo_tts/compare_parity.py +git commit -m "test(echo_tts): reference parity dumper and cosine comparator" +``` + +Note: `.npy` dumps are build artefacts — add `tests/echo_tts/parity/` to `.gitignore`, do not commit them. + +--- + +## Task 5: GGUF emission + +**Files:** +- Modify: `tests/echo_tts/convert_echo_tts_weights.py` (add `--write-gguf`) + +- [ ] **Step 1: Add the GGUF flags** + +Mirror `convert_confucius4_tts_weights.py:33-36`: `--write-gguf`, `--gguf-output model.gguf`, `--gguf-type orig`, `--gguf-tool build/linux-cuda-release/bin/audiocpp_gguf`. The converter shells out to that tool; it does **not** write GGUF from Python. + +- [ ] **Step 2: Build the tool** + +```bash +cmake --build --preset linux-cuda-release --target audiocpp_gguf -j +``` + +- [ ] **Step 3: Emit GGUF** + +```bash +uv run --with torch --with safetensors --with numpy \ + python tests/echo_tts/convert_echo_tts_weights.py \ + --output-dir models/echo-tts/audio_cpp --write-gguf --gguf-type orig +ls -la models/echo-tts/audio_cpp/model.gguf +``` +Expected: file exists. Given ~2.5 B BF16 Echo weights plus ~184 M FP32 Fish decode weights, expect roughly 5–6 GB; anything near 8 GB means the dropped buffers leaked back in — re-check Task 3 Step 3. + +- [ ] **Step 4: Commit** + +```bash +git add tests/echo_tts/convert_echo_tts_weights.py +git commit -m "feat(echo_tts): emit GGUF via audiocpp_gguf" +``` + +--- + +## Task 6: Assets, config, and speaker-latent injection + +**Files:** +- Create: `include/engine/community_models/echo_tts/types.h`, `assets.h` +- Create: `src/community_models/echo_tts/assets.cpp` +- Modify: `src/community_models/echo_tts/session.cpp` + +**Interfaces:** +- Produces: +```cpp +struct EchoTtsConfig { + int trunk_depth = 24; + int hidden_dim = 2048; + int latent_dim = 80; + int sequence_length = 640; + int samples_per_frame= 2048; + int sample_rate = 44100; + float rope_theta = 10000.0F; + float latent_scale = 0.0555555559694767F; +}; +struct EchoTtsAssets { // resolved tensor handles + assets::TensorHandle pca_components; // [80,1024] + assets::TensorHandle pca_mean; // [1024] + // … trunk, encoders, fish decode handles +}; +std::shared_ptr load_echo_tts_assets(const engine::model_spec::ModelContract &); +``` +- Produces: a debug session option `echo_tts.speaker_latent_path=` which loads the Task 4 `speaker_latent.npy` in place of native encoding. **This option is M1-only scaffolding and must be deleted in M2.** + +- [ ] **Step 1: Define config and assets headers** using the signatures above; take tensor names from `/home/ryzen/.claude/jobs/1464b16b/tmp/echo-tensor-manifest.txt`. + +- [ ] **Step 2: Implement `load_echo_tts_assets`** resolving every handle from the contract; throw with the missing key name if any handle is absent. + +- [ ] **Step 3: Add the `.npy` loader** for the injected speaker latent (little-endian float32, C-order; parse the standard `.npy` v1 header). + +- [ ] **Step 4: Verify assets resolve** + +```bash +cmake --build --preset linux-cuda-release --target audiocpp_cli -j +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ + --text "[S1] test" --out /tmp/echo_stub.wav +``` +Expected: exits 0, still emits silence, and logs no missing-tensor error. A missing-key throw here names the exact tensor to fix. + +- [ ] **Step 5: Commit** + +```bash +git add include/engine/community_models/echo_tts/ src/community_models/echo_tts/ +git commit -m "feat(echo_tts): assets, config, and M1 speaker-latent injection" +``` + +--- + +## Task 7: Text tokenizer and text encoder + +**Files:** +- Create: `include/engine/community_models/echo_tts/tokenizer_text.h`, `encoders.h` +- Create: `src/community_models/echo_tts/tokenizer_text.cpp`, `src/community_models/echo_tts/encoders.cpp` + +**Interfaces:** +- Produces: +```cpp +std::vector echo_tokenize(const std::string & text); // WhisperD norm + UTF-8 bytes +class EchoTextEncoder { +public: + EchoTextEncoder(std::shared_ptr, core::BackendConfig, size_t arena_bytes); + // returns [1, T, 1280] + core::Tensor encode(const std::vector & tokens, const std::vector & mask); +}; +``` + +- [ ] **Step 1: Write the tokenizer test** + +Create `tests/echo_tts/test_echo_tokenizer.cpp`: + +```cpp +#include "engine/community_models/echo_tts/tokenizer_text.h" +#include +#include + +int main() { + using engine::models::echo_tts::echo_tokenize; + // "[S1] " is prepended when absent + auto a = echo_tokenize("hello"); + auto b = echo_tokenize("[S1] hello"); + assert(a == b); + // colons, semicolons, emdashes normalise to commas + auto c = echo_tokenize("[S1] a: b; c \xE2\x80\x94 d"); + auto d = echo_tokenize("[S1] a, b, c , d"); + assert(c == d); + // tokens are raw UTF-8 bytes, so every value is 0..255 + for (auto t : a) { assert(t >= 0 && t <= 255); } + std::cout << "tokenizer ok\n"; + return 0; +} +``` + +- [ ] **Step 2: Run it and watch it fail** + +```bash +cmake --build --preset linux-cuda-release --target test_echo_tokenizer -j +``` +Expected: FAIL — `echo_tokenize` not defined. + +- [ ] **Step 3: Implement the tokenizer** per `inference.py` `tokenizer_encode`: normalise `:`/`;`/`—` to `,`, prepend `[S1] ` when neither `[S1]` nor `[S2]` is present, then emit raw UTF-8 bytes. + +- [ ] **Step 4: Run it and watch it pass** + +```bash +./build/linux-cuda-release/bin/test_echo_tokenizer +``` +Expected: `tokenizer ok` + +- [ ] **Step 5: Implement `EchoTextEncoder`** — the 294 M encoder body under manifest prefix `text_encoder.*`, with the `[256,1280]` embedding, and dump its output to `/tmp/echo_text_enc.npy` under a debug session option. + +- [ ] **Step 6: Gate on parity** + +```bash +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.dump_text_enc=/tmp/echo_text_enc.npy \ + --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ + --out /tmp/echo_stub.wav +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/text_enc.npy --got /tmp/echo_text_enc.npy +``` +Expected: `PASS cosine>=0.999`. **Do not proceed while this fails.** + +- [ ] **Step 7: Commit** + +```bash +git add include/engine/community_models/echo_tts/ src/community_models/echo_tts/ tests/echo_tts/ +git commit -m "feat(echo_tts): byte tokenizer and text encoder, parity-gated" +``` + +--- + +## Task 8: Speaker encoder + +**Files:** +- Modify: `include/engine/community_models/echo_tts/encoders.h`, `src/community_models/echo_tts/encoders.cpp` + +**Interfaces:** +- Consumes: injected `speaker_latent.npy` `[1,Ls,80]` from Task 6. +- Produces: `class EchoSpeakerEncoder { core::Tensor encode(const core::Tensor & speaker_latent); };` → `[1, Ls, 1280]` + +- [ ] **Step 1: Implement** the 294 M encoder under manifest prefix `speaker_encoder.*`, with the biased `320→1280` input projection and patch size 4. + +- [ ] **Step 2: Gate on parity** + +```bash +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ + --session-option echo_tts.dump_speaker_enc=/tmp/echo_speaker_enc.npy \ + --text "[S1] test" --out /tmp/echo_stub.wav +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/speaker_enc.npy --got /tmp/echo_speaker_enc.npy +``` +Expected: `PASS`. If `Ls % 4 != 0` the reshape will throw — the dumper already guarantees divisibility. + +- [ ] **Step 3: Commit** + +```bash +git commit -am "feat(echo_tts): speaker encoder, parity-gated" +``` + +--- + +## Task 9: DiT trunk + +**Files:** +- Create: `include/engine/community_models/echo_tts/dit.h`, `src/community_models/echo_tts/dit.cpp` + +**Interfaces:** +- Produces: +```cpp +class EchoDiT { +public: + // x:[1,640,80] latents, t: timestep, returns velocity [1,640,80] + core::Tensor forward(const core::Tensor & x, float t, + const core::Tensor & text_states, const std::vector & text_mask, + const core::Tensor & speaker_states, const std::vector & speaker_mask, + float speaker_kv_scale); +}; +``` + +- [ ] **Step 1: Implement one block first.** Port a single joint-attention + SwiGLU-MLP block with adaLN, from `model.py:128-268`. Critical details: RoPE theta 10000.0 rotating **only half the heads**; RMSNorm accumulating in FP32; adaLN modulating both attention and MLP from the timestep embedding; joint attention concatenating self + text KV + speaker KV with per-source boolean masks. + +- [ ] **Step 2: Gate block 0 on parity** + +```bash +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ + --session-option echo_tts.dump_dit_block=0:/tmp/echo_dit00.npy \ + --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ + --out /tmp/echo_stub.wav +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/dit_block00.npy --got /tmp/echo_dit00.npy +``` +Expected: `PASS`. A single block passing means the hard parts (half-head RoPE, FP32 norm, mask layout) are all correct — this is the highest-value gate in the plan. + +- [ ] **Step 3: Extend to all 24 blocks**, then gate blocks 11 and 23 the same way against `dit_block11.npy` and `dit_block23.npy`. + +- [ ] **Step 4: Commit** + +```bash +git commit -am "feat(echo_tts): 24-block DiT trunk, parity-gated at blocks 0/11/23" +``` + +--- + +## Task 10: Euler sampler with dual independent CFG + +**Files:** +- Create: `include/engine/community_models/echo_tts/sampler.h`, `src/community_models/echo_tts/sampler.cpp` + +**Interfaces:** +- Produces: `core::Tensor echo_sample(EchoDiT &, const EchoSamplerParams &, uint64_t seed);` → `[1,640,80]` + +- [ ] **Step 1: Implement** per `inference.py:361-419`. Required behaviour: 40 Euler steps; **two** guidance scales combined into one velocity; guidance active only for `t ∈ [cfg_min_t, cfg_max_t]` = `[0.5, 1.0]`; `truncation_factor` 0.8 applied to the initial Gaussian; unconditioning done by **masking**, not by zeroing encoder states. Note each guided step costs 3 DiT forwards (cond, text-uncond, speaker-uncond). + +- [ ] **Step 2: Gate final latents on parity** + +```bash +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ + --session-option echo_tts.dump_latents=/tmp/echo_latents.npy \ + --option seed=0 \ + --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ + --out /tmp/echo_stub.wav +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/latents_final.npy --got /tmp/echo_latents.npy +``` +Expected: `PASS`. If cosine is high but not ≥0.999, suspect the initial noise: Torch's Gaussian RNG is device-specific, so seed the C++ path from the dumped initial noise instead of regenerating it, and record that as a known parity caveat in the PR. + +- [ ] **Step 3: Commit** + +```bash +git commit -am "feat(echo_tts): Euler sampler with dual independent CFG, parity-gated" +``` + +--- + +## Task 11: PCA inverse and Fish S1-DAC decode + +**Files:** +- Create: `include/engine/community_models/echo_tts/fish_decoder.h`, `src/community_models/echo_tts/fish_decoder.cpp` + +**Interfaces:** +- Produces: `runtime::AudioBuffer echo_decode(const core::Tensor & latents_80d);` → 44 100 Hz mono + +- [ ] **Step 1: Implement PCA inverse** — `z1024 = (z80 / latent_scale) @ pca_components + pca_mean`. Gate it alone: + +```bash +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/pca_inv.npy --got /tmp/echo_pca_inv.npy +``` + +- [ ] **Step 2: Implement the decode stack** — `quantizer.post_module` → `quantizer.upsample` → `decoder`, regenerating `freqs_cis` and `causal_mask` at runtime rather than loading them. **Do not port the decoder transformer at `autoencoder.py:943-965`** — it exists only as an unregistered local variable and never executes; porting the apparent configuration would be silently wrong. + +- [ ] **Step 3: Gate decoded audio on parity** + +```bash +python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/decoded.npy --got /tmp/echo_decoded.npy +``` +Expected: `PASS`. Causal-conv right-padding and transposed-conv asymmetric cropping are the likely culprits on failure — an off-by-one there shifts the whole waveform and tanks cosine. + +- [ ] **Step 4: Commit** + +```bash +git commit -am "feat(echo_tts): PCA inverse and Fish S1-DAC decode path, parity-gated" +``` + +--- + +## Task 12: Flattening-point crop, end-to-end, and the ear check + +**Files:** +- Modify: `src/community_models/echo_tts/session.cpp` + +- [ ] **Step 1: Implement the crop** per `inference.py:233-246` — scan 20-frame latent windows by standard deviation and mean, then cut the waveform at `frame × 2048`. This is a host-side loop, not a graph op. + +- [ ] **Step 2: Generate end-to-end** + +```bash +./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ + --model models/echo-tts/audio_cpp --backend cuda \ + --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ + --option seed=0 \ + --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ + --out /tmp/echo_m1.wav +``` + +- [ ] **Step 3: Verify the output file mechanically** + +```bash +python3 -c " +import soundfile as sf +y,sr=sf.read('/tmp/echo_m1.wav') +print('sr',sr,'dur',round(len(y)/sr,3),'peak',round(float(abs(y).max()),4)) +assert sr==44100, 'wrong sample rate' +assert 1.0 < len(y)/sr < 29.8, 'duration outside the 29.72s window' +assert abs(y).max() > 0.01, 'output is silence' +" +``` +Expected: 44 100 Hz, a plausible duration well under 29.72 s, non-silent. + +- [ ] **Step 4: THE EAR CHECK — mandatory, not optional** + +Listen to `/tmp/echo_m1.wav` and compare against the reference wav produced by Task 4's dumper. Confirm: intelligible speech, the right words, no clicks at buffer boundaries, no metallic or phasey artefacts, and a voice that plausibly matches `chris_hemsworth_15s.wav`. + +Tensor parity **cannot** catch failures here — the flattening-point crop is a host-side loop outside the parity chain, and a wrong crop yields perfect cosine on latents with truncated or silence-padded audio. **M1 is not complete until a human has listened.** + +- [ ] **Step 5: Commit** + +```bash +git commit -am "feat(echo_tts): flattening-point crop and end-to-end M1 decode path" +``` + +--- + +## Task 13: Warm bench and evidence pack + +**Files:** +- Create: `tests/echo_tts/echo_tts_warm_bench.cpp` +- Modify: `CMakeLists.txt` (add `add_engine_warmbench(echo_tts_warm_bench tests/echo_tts/echo_tts_warm_bench.cpp)` near line 1315) + +- [ ] **Step 1: Write the warm bench**, modelled on `tests/confucius4_tts/confucius4_tts_warm_bench.cpp`, driven by `echo_tts_warm_bench_cases.json`. + +- [ ] **Step 2: Measure RTF and VRAM** + +```bash +cmake --build --preset linux-cuda-release --target echo_tts_warm_bench -j +./build/linux-cuda-release/bin/echo_tts_warm_bench \ + --model models/echo-tts/audio_cpp --backend cuda --runs 5 +``` +Record wall time, audio length, **RTF = wall ÷ audio**, and peak VRAM for each run. + +Gates: **RTF < 1.0** (the community bar — note this is the inverse of tts-bench's RTFx; Echo's PyTorch 1.35× RTFx equals RTF 0.74, so the port should land near or below that), and **VRAM must not grow across the 5 runs**. + +- [ ] **Step 3: Assemble the evidence pack for the PR** + +Collect: exact build command, exact run commands, every parity line (`cosine=… max_abs_err=…`) from Tasks 7–11, the RTF table, the VRAM series, and `/tmp/echo_m1.wav` attached. + +- [ ] **Step 4: Commit and push** + +```bash +git add tests/echo_tts/echo_tts_warm_bench.cpp CMakeLists.txt +git commit -m "test(echo_tts): warm bench with RTF and VRAM measurement" +git push +``` + +- [ ] **Step 5: Post the evidence to the draft PR — and leave it in draft** + +M1 completes the decode path only. Cloning still requires an injected `.npy`, so the model is not yet self-contained and **Definition of Ready is not met**. The PR stays draft until M2 lands native speaker encoding. + +--- + +## Self-Review + +**Spec coverage.** §2 architecture → Tasks 6–11. §2.4 decode/encode asymmetry → Task 6 injection + M2 deferral. §3 long-form → deliberately out of scope, and Task 1 enforces it by omitting `long_form` from `capabilities`. §4 M0 → Tasks 1–2; M1 → Tasks 3–13. §5 Definition of Ready → Task 13 Step 3 assembles it and Step 5 explicitly withholds ready status. §6 integration surface → Task 1. §7 traps: trap 1 (phantom decoder) Task 11 Step 2; trap 2 (weight norm) Task 3 Step 1; trap 3 (FP32) Global Constraints + Task 9 Step 1; trap 4 (buffers) Task 3; trap 5 (half-head RoPE) Task 9 Step 1; trap 7 (causal padding) Task 11 Step 3; trap 8 (divisibility) Task 8 Step 2; trap 9 (mask uncond) Task 10 Step 1. §8 testing → Tasks 4, 7–12. **Gap found and closed:** trap 6 (Snake activation) had no owner — it lives in the Fish decoder and is now covered by Task 11 Step 2. + +**Placeholder scan.** No TBD/TODO. Every code step carries literal content. The one intentional stub (Task 1 silence) is named as such with a removal owner. + +**Type consistency.** `EchoTtsConfig`, `EchoTtsAssets`, `EchoTextEncoder::encode`, `EchoSpeakerEncoder::encode`, `EchoDiT::forward`, `echo_sample`, `echo_decode`, `echo_tokenize` are each declared once in Task 6/7/8/9/10/11 and referenced consistently thereafter. Debug session options use one `echo_tts.` namespace throughout. + +**Known scaffolding debt.** `echo_tts.speaker_latent_path` and the `dump_*` options are M1-only. M2's plan must open with their removal. From ec3f0600adbbb34b49b614df003995c062ec6eae Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 31 Jul 2026 17:18:42 +0000 Subject: [PATCH 4/8] docs: correct build invocation and CUDA arch in M0+M1 plan - Repo has no CMakePresets.json; --preset would fail. Use scripts/build_linux.sh or cmake --build build/linux-cuda-release. - Existing build tree pins CMAKE_CUDA_ARCHITECTURES=75 (Turing) on an sm_86 card. Task 13 now reconfigures to 86 before measuring RTF, otherwise the number is invalid. - Note AUDIOCPP_MODEL_SET=full so the family compiles in automatically. --- .gitignore | 1 + .../plans/2026-07-30-echo-tts-m0-m1.md | 40 +++++++++++++++---- 2 files changed, 33 insertions(+), 8 deletions(-) diff --git a/.gitignore b/.gitignore index 803a7bd4..a8563505 100644 --- a/.gitignore +++ b/.gitignore @@ -45,3 +45,4 @@ __pycache__/ /webui/configs/ui_language.json # personal voice recording — stays local, repo is public /webui/voice/my-record.wav +.devkit/ diff --git a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md index 5d675dc5..3c89e094 100644 --- a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md +++ b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md @@ -25,7 +25,20 @@ - Never serialise `freqs_cis` or `causal_mask` into GGUF (303.6 M elements). Regenerate at runtime. - **Parity gate:** cosine similarity ≥ 0.999 over each tensor flattened to 1-D, reported with max-absolute-error. A stage is not done until its gate is green **when run**, not when reported. - Reference implementation for all parity work: `/home/ryzen/LocalDev/tts-bench/venvs/echo/src/`, weights in `~/.cache/huggingface/hub/models--jordand--echo-tts-base` and `…--fish-s1-dac-min`. -- Hardware: RTX 3090 24 GB, CUDA. Build preset `linux-cuda-release`. +- Hardware: RTX 3090 24 GB (compute capability **8.6**), CUDA. +- **Build invocation.** There is no `CMakePresets.json` in this repo — `cmake --build --preset …` will + fail. Use either `scripts/build_linux.sh --backend cuda --target ` or, against the existing + configured tree, `cmake --build build/linux-cuda-release --target -j`. `ccache` is + installed and there are 32 cores, so incremental rebuilds are cheap. +- **Model set.** `build/linux-cuda-release` is configured with `AUDIOCPP_MODEL_SET=full` and an empty + `AUDIOCPP_MODELS`, so a family registered via `audiocpp_add_model` is compiled in automatically. No + model-set flags needed. +- **CUDA architecture — must be fixed before any RTF number is quoted.** The existing + `build/linux-cuda-release` has `CMAKE_CUDA_ARCHITECTURES=75` (Turing) while the card is 8.6 + (Ampere). `CMakeLists.txt:1165-1168` defaults to `native` only when the variable is unset, so this + tree is pinned wrong. Development builds may proceed as-is, but **Task 13 must reconfigure with + `-DCMAKE_CUDA_ARCHITECTURES=86`** (or unset it to get `native`) before measuring, or the reported + RTF is invalid and would have to be retracted. --- @@ -183,7 +196,7 @@ audiocpp_add_model(echo_tts Run: ```bash -cmake --build --preset linux-cuda-release --target audiocpp_cli -j +cmake --build build/linux-cuda-release --target audiocpp_cli -j ./build/linux-cuda-release/bin/audiocpp_cli --list-families | grep echo_tts ``` Expected: `echo_tts` appears. If it links-errors on `make_echo_tts_loader`, the namespace is wrong — it must be `engine::models::echo_tts`, not `engine::community_models::echo_tts`. @@ -380,7 +393,7 @@ Mirror `convert_confucius4_tts_weights.py:33-36`: `--write-gguf`, `--gguf-output - [ ] **Step 2: Build the tool** ```bash -cmake --build --preset linux-cuda-release --target audiocpp_gguf -j +cmake --build build/linux-cuda-release --target audiocpp_gguf -j ``` - [ ] **Step 3: Emit GGUF** @@ -440,7 +453,7 @@ std::shared_ptr load_echo_tts_assets(const engine::model_sp - [ ] **Step 4: Verify assets resolve** ```bash -cmake --build --preset linux-cuda-release --target audiocpp_cli -j +cmake --build build/linux-cuda-release --target audiocpp_cli -j ./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ --model models/echo-tts/audio_cpp --backend cuda \ --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ @@ -504,7 +517,7 @@ int main() { - [ ] **Step 2: Run it and watch it fail** ```bash -cmake --build --preset linux-cuda-release --target test_echo_tokenizer -j +cmake --build build/linux-cuda-release --target test_echo_tokenizer -j ``` Expected: FAIL — `echo_tokenize` not defined. @@ -731,13 +744,24 @@ git commit -am "feat(echo_tts): flattening-point crop and end-to-end M1 decode p - [ ] **Step 1: Write the warm bench**, modelled on `tests/confucius4_tts/confucius4_tts_warm_bench.cpp`, driven by `echo_tts_warm_bench_cases.json`. -- [ ] **Step 2: Measure RTF and VRAM** +- [ ] **Step 2: Reconfigure for the correct CUDA architecture, then measure RTF and VRAM** + +The existing tree is pinned to `sm_75` on an `sm_86` card. Reconfigure before measuring: ```bash -cmake --build --preset linux-cuda-release --target echo_tts_warm_bench -j -./build/linux-cuda-release/bin/echo_tts_warm_bench \ +cmake -S . -B build/linux-cuda-86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \ + -DCMAKE_CUDA_ARCHITECTURES=86 +cmake --build build/linux-cuda-86 --target echo_tts_warm_bench audiocpp_cli -j +./build/linux-cuda-86/bin/echo_tts_warm_bench \ --model models/echo-tts/audio_cpp --backend cuda --runs 5 ``` + +Confirm the arch actually took before trusting the numbers: + +```bash +grep -E "^CMAKE_CUDA_ARCHITECTURES:" build/linux-cuda-86/CMakeCache.txt +``` +Expected: `CMAKE_CUDA_ARCHITECTURES:STRING=86` Record wall time, audio length, **RTF = wall ÷ audio**, and peak VRAM for each run. Gates: **RTF < 1.0** (the community bar — note this is the inverse of tts-bench's RTFx; Echo's PyTorch 1.35× RTFx equals RTF 0.74, so the port should land near or below that), and **VRAM must not grow across the 5 runs**. From c42cb169cadfef862fca2ba13f42184b5e4c0018 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 31 Jul 2026 17:19:53 +0000 Subject: [PATCH 5/8] docs: fix Task 1 gate to use --list-loaders (no --list-families flag) Baseline on this branch is registered_loaders=42, verified. Also note that a 'requires a schema v1 model contract' failure means a stale binary, not a broken tree. --- .../plans/2026-07-30-echo-tts-m0-m1.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md index 3c89e094..197fafc4 100644 --- a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md +++ b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md @@ -196,10 +196,20 @@ audiocpp_add_model(echo_tts Run: ```bash -cmake --build build/linux-cuda-release --target audiocpp_cli -j -./build/linux-cuda-release/bin/audiocpp_cli --list-families | grep echo_tts +cmake --build build/linux-cuda-release --target audiocpp_cli -j 32 +./build/linux-cuda-release/bin/audiocpp_cli --list-loaders | head -1 +./build/linux-cuda-release/bin/audiocpp_cli --list-loaders | grep echo_tts ``` -Expected: `echo_tts` appears. If it links-errors on `make_echo_tts_loader`, the namespace is wrong — it must be `engine::models::echo_tts`, not `engine::community_models::echo_tts`. +Expected: the count line reads `registered_loaders=43` (baseline on this branch is **42**, verified +2026-07-30), and the grep prints `echo_tts: clon (offline)`. + +There is no `--list-families` flag; the flags are `--list-loaders [--json]` and `--list-pipelines`. + +If it link-errors on `make_echo_tts_loader`, the namespace is wrong — it must be +`engine::models::echo_tts`, not `engine::community_models::echo_tts`. + +If `--list-loaders` fails with something like `bs_roformer requires a schema v1 model contract`, the +binary is **stale**, not broken — rebuild `audiocpp_cli` and retry before investigating. - [ ] **Step 7: Commit** From cea6133f3ac680ffb69cb8f232d733a6202c07d1 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Fri, 31 Jul 2026 17:27:36 +0000 Subject: [PATCH 6/8] feat(echo_tts): register family with spec v1 and silence stub Plan Task 1. Spec-backed loader (no loader.cpp), schema_version 1, capabilities.clone deliberately omits long_form until M3 earns it. Verified by execution: registered_loaders 42 -> 43, echo_tts appears as 'clon (offline)', spec parses. Fix over Codex's draft: guard used VoiceTaskKind::Tts, but the family registers as a clone task, so every real invocation would have thrown. Corrected to VoiceCloning, matching confucius4_tts:185. The registration gate could not catch this - --list-loaders enumerates loaders without constructing a session. --- CMakeLists.txt | 9 ++ .../community_models/echo_tts/session.h | 34 +++++++ model_specs/echo_tts.json | 57 ++++++++++++ src/community_models/echo_tts/session.cpp | 93 +++++++++++++++++++ tests/echo_tts/echo_tts_warm_bench_cases.json | 12 +++ 5 files changed, 205 insertions(+) create mode 100644 include/engine/community_models/echo_tts/session.h create mode 100644 model_specs/echo_tts.json create mode 100644 src/community_models/echo_tts/session.cpp create mode 100644 tests/echo_tts/echo_tts_warm_bench_cases.json diff --git a/CMakeLists.txt b/CMakeLists.txt index a281aede..b3445db9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -485,6 +485,15 @@ audiocpp_add_model(inflect_v2 engine::models::inflect_v2::make_inflect_v2_loader ) +audiocpp_add_model(echo_tts + SOURCES + src/community_models/echo_tts/session.cpp + INCLUDES + engine/community_models/echo_tts/session.h + LOADERS + engine::models::echo_tts::make_echo_tts_loader +) + if (MSVC) set_source_files_properties( src/community_models/inflect_v2/frontend.cpp diff --git a/include/engine/community_models/echo_tts/session.h b/include/engine/community_models/echo_tts/session.h new file mode 100644 index 00000000..8d2b6bde --- /dev/null +++ b/include/engine/community_models/echo_tts/session.h @@ -0,0 +1,34 @@ +#pragma once + +#include "engine/framework/model_spec/metadata.h" +#include "engine/framework/runtime/session_base.h" + +#include + +namespace engine::models::echo_tts { + +std::shared_ptr make_echo_tts_loader(); + +class EchoTtsSession final + : public runtime::RuntimeSessionBase, + public runtime::IOfflineVoiceTaskSession { +public: + EchoTtsSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr contract); + ~EchoTtsSession() override; + + std::string family() const override; + runtime::VoiceTaskKind task_kind() const override; + runtime::RunMode run_mode() const override; + void prepare(const runtime::SessionPreparationRequest & request) override; + runtime::TaskResult run(const runtime::TaskRequest & request) override; + void reset(); + +private: + runtime::TaskSpec task_; + std::shared_ptr contract_; +}; + +} // namespace engine::models::echo_tts diff --git a/model_specs/echo_tts.json b/model_specs/echo_tts.json new file mode 100644 index 00000000..54aff64d --- /dev/null +++ b/model_specs/echo_tts.json @@ -0,0 +1,57 @@ +{ + "schema_version": 1, + "family": "echo_tts", + "display_name": "Echo-TTS", + "description": "Echo-TTS is an English zero-shot voice-cloning TTS model packaged for audio.cpp. A 2.8B diffusion transformer generates 80-D latents in PCA space which the Fish S1-DAC decodes to 44.1 kHz audio. Generation is a fixed 29.72 s window (640 latents).", + "category": "tts", + "status": "experimental", + "tasks": ["clone"], + "modes": ["offline"], + "languages": ["en"], + "runtime": { "tags": ["gguf"] }, + "capabilities": { "clone": ["speaker_reference"] }, + "options": { + "request": [ + { "name": "target_voice", "type": "audio_path", "description": "Reference audio path for zero-shot cloning. Wav only; no transcript required.", "required": false }, + { "name": "cfg_scale_text", "type": "float", "description": "Classifier-free guidance scale on the text condition.", "required": false, "min": 0.0, "default": 3.0 }, + { "name": "cfg_scale_speaker", "type": "float", "description": "Classifier-free guidance scale on the speaker condition.", "required": false, "min": 0.0, "default": 8.0 }, + { "name": "num_steps", "type": "int", "description": "Euler sampler steps.", "required": false, "min": 1, "default": 40 }, + { "name": "truncation_factor", "type": "float", "description": "Initial-noise truncation factor.", "required": false, "min": 0.0, "max": 1.0, "default": 0.8 }, + { "name": "speaker_kv_scale", "type": "float", "description": "Force-speaker KV scaling. 1.0 disables; 1.5 is the upstream default when enabled.", "required": false, "min": 1.0, "default": 1.0 }, + { "name": "seed", "type": "int", "description": "RNG seed for the initial latent.", "required": false, "default": 0 } + ], + "session": [], + "load": [] + }, + "packages": [ + { + "id": "echo_tts_orig", + "display_name": "Echo-TTS Original-Dtype GGUF", + "default": true, + "format": "gguf", + "precision": "orig", + "target_directory": "Echo-TTS-GGUF", + "files": ["Echo-TTS-GGUF/model.gguf"], + "download": { + "kind": "unsupported", + "reason": "Echo-TTS model packaging is not implemented yet." + }, + "strip_prefix": "Echo-TTS-GGUF" + } + ], + "dependencies": [], + "ui": { + "recommended_package": "echo_tts_orig", + "tags": ["TTS", "Clone", "GGUF"], + "docs": [] + }, + "sources": [ + { + "format": "gguf", + "roots": { + "model": ".", + "weights": "$gguf" + } + } + ] +} diff --git a/src/community_models/echo_tts/session.cpp b/src/community_models/echo_tts/session.cpp new file mode 100644 index 00000000..4a3a33e7 --- /dev/null +++ b/src/community_models/echo_tts/session.cpp @@ -0,0 +1,93 @@ +#include "engine/community_models/echo_tts/session.h" + +#include "engine/framework/assets/resource_bundle.h" +#include "engine/framework/model_spec/package.h" +#include "engine/framework/runtime/spec_backed_model.h" + +#include +#include +#include +#include + +namespace engine::models::echo_tts { +namespace { + +constexpr const char * kFamily = "echo_tts"; +constexpr int kSampleRate = 44100; + +struct EchoTtsAssets { + assets::ResourceBundle resources; +}; + +std::shared_ptr load_echo_tts_assets( + const std::filesystem::path & model_path) { + auto assets = std::make_shared(); + assets->resources = engine::model_spec::load_resource_bundle_for_family( + model_path, + kFamily); + return assets; +} + +} // namespace + +EchoTtsSession::EchoTtsSession( + runtime::TaskSpec task, + runtime::SessionOptions options, + std::shared_ptr contract) + : RuntimeSessionBase(std::move(options)), + task_(task), + contract_(std::move(contract)) { + if (contract_ == nullptr) { + throw std::runtime_error("Echo-TTS session requires a model contract"); + } + if (task_.task != runtime::VoiceTaskKind::VoiceCloning || + task_.mode != runtime::RunMode::Offline) { + throw std::runtime_error("Echo-TTS only supports offline voice cloning"); + } +} + +EchoTtsSession::~EchoTtsSession() = default; + +std::string EchoTtsSession::family() const { return kFamily; } +runtime::VoiceTaskKind EchoTtsSession::task_kind() const { return task_.task; } +runtime::RunMode EchoTtsSession::run_mode() const { return task_.mode; } + +void EchoTtsSession::prepare(const runtime::SessionPreparationRequest & request) { + (void)request; + mark_prepared(); +} + +runtime::TaskResult EchoTtsSession::run(const runtime::TaskRequest & request) { + (void)request; + require_prepared("Echo-TTS run"); + + runtime::TaskResult result; + result.audio_output = runtime::AudioBuffer{ + kSampleRate, + 1, + std::vector(kSampleRate, 0.0F), + }; + return result; +} + +void EchoTtsSession::reset() {} + +std::shared_ptr make_echo_tts_loader() { + runtime::SpecBackedVoiceModelConfig config; + config.family = kFamily; + config.load_assets = load_echo_tts_assets; + config.create_session = []( + const runtime::TaskSpec & task, + const runtime::SessionOptions & options, + std::shared_ptr assets, + std::shared_ptr contract) { + (void)assets; + return std::make_unique( + task, + options, + std::move(contract)); + }; + return runtime::make_spec_backed_voice_loader(std::move(config)); +} + +} // namespace engine::models::echo_tts diff --git a/tests/echo_tts/echo_tts_warm_bench_cases.json b/tests/echo_tts/echo_tts_warm_bench_cases.json new file mode 100644 index 00000000..3db37f5d --- /dev/null +++ b/tests/echo_tts/echo_tts_warm_bench_cases.json @@ -0,0 +1,12 @@ +{ + "default_clone": { + "requests": [ + { + "id": "chris_ref_p1", + "target_voice": "reference/chris_hemsworth_15s.wav", + "text": "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm.", + "seed": 0 + } + ] + } +} From 5894cd8746e34597b15285e946dc8bfd75d77f68 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Mon, 3 Aug 2026 14:34:41 +0000 Subject: [PATCH 7/8] docs(echo_tts): add community model doc Documents the fixed 29.72s window, why blockwise does not extend it, the CC-BY-NC-SA output restriction (with the fish_audio precedent), benchmark provenance, options, and the WhisperD text format. Also drops an unrelated .gitignore change that was accidentally swept into an earlier docs commit, so the PR diff stays scoped. --- .gitignore | 1 - docs/community_models/echo_tts.md | 136 ++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 docs/community_models/echo_tts.md diff --git a/.gitignore b/.gitignore index a8563505..803a7bd4 100644 --- a/.gitignore +++ b/.gitignore @@ -45,4 +45,3 @@ __pycache__/ /webui/configs/ui_language.json # personal voice recording — stays local, repo is public /webui/voice/my-record.wav -.devkit/ diff --git a/docs/community_models/echo_tts.md b/docs/community_models/echo_tts.md new file mode 100644 index 00000000..dbacffb6 --- /dev/null +++ b/docs/community_models/echo_tts.md @@ -0,0 +1,136 @@ +# Echo-TTS + +Echo-TTS is an English zero-shot voice-cloning TTS model. A 2.8B diffusion transformer (EchoDiT) +generates 80-dimensional latents in PCA space, which the Fish S1-DAC autoencoder decodes to 44.1 kHz +audio. Cloning takes a reference wav with **no transcript required**. + +Upstream: [jordand/echo-tts-base](https://huggingface.co/jordand/echo-tts-base) · +autoencoder: [jordand/fish-s1-dac-min](https://huggingface.co/jordand/fish-s1-dac-min) + +| Family | `echo_tts` | +|---|---| +| Tasks | `clon` | +| Modes | offline | +| Languages | en | +| Sample rate | 44 100 Hz | +| Model directory | `models/echo-tts` | + +## Status + +**Work in progress.** Landing in stages, each gated on numerical parity against the reference +implementation: + +| Milestone | Scope | State | +|---|---|---| +| M0 | Family registration, model spec v1 | done | +| M1 | GGUF conversion, DiT, PCA inverse, Fish decode | in progress | +| M2 | Native speaker encoding (Fish encoder + RVQ) | not started | +| M3 | Long-form generation | not started | +| M4 | Quantisation, RTF and memory evidence | not started | + +Until M2 lands, cloning requires a pre-computed speaker latent, so the model is not yet +self-contained. This PR stays in draft until the full evidence pack exists. + +## Known limitations + +### Fixed 29.72-second generation window + +Echo is trained to generate at most **640 latents**, and 640 × 2048 ÷ 44100 = **29.7215 s**. This is +a property of the model, not of this port. + +Behaviour outside that window: + +- Text corresponding to more than ~30 s is **spoken faster** to fit, rather than truncated. This is + learned behaviour arising from global attention over the text, not an explicit compression step. +- The upstream tokenizer hard-truncates text past **768 UTF-8 bytes**. +- Requesting a shorter window does *not* compress the whole utterance into it — upstream documents + that the model generates a **prefix** of the utterance instead. + +`long_form` is therefore **not** claimed in `capabilities` at this stage. + +### Blockwise generation does not extend the window + +Upstream ships a blockwise sampler that generates in connected blocks and supports continuing from +existing audio. It **subdivides** the ≤30 s window rather than extending it: upstream requires +`sum(block_sizes) + continuation_length < 640` "to be in-distribution with training data", and +documents prefix plus continuation as "up to 30 seconds combined". Upstream also notes blockwise +"hasn't been thoroughly tested". + +## Licence — read before using output commercially + +Echo-TTS is **CC-BY-NC-SA-4.0**, and the restriction covers **generated audio, not only the +weights**. The output constraint is inherited from the Fish S1-DAC autoencoder — the same mechanism +that makes Fish Speech's own outputs non-commercial. + +Practically: **audio produced by this model may not be used commercially**, regardless of how the +rest of your stack is licensed. audio.cpp itself is Apache 2.0 and is unaffected; model weights are +a separate download. + +There is existing precedent in-tree — `fish_audio` (Fish Audio S2 Pro) carries the identical +output restriction from the identical dependency. + +## Why this model + +Selected by comparing every model tracked in [tts-bench](https://github.com/5uck1ess/tts-bench) — a +public benchmark covering **62 local TTS models** across speed, objective scores, and blind human +preference — against audio.cpp's existing support table. + +| Measure | Echo-TTS | Field | +|---|---|---| +| Blind cloning Elo | **1162** | #3 of 40 (35 games; 738 cloning votes total) | +| Speaker similarity (SIM) | **0.836** | 2nd of 41 scored models | +| UTMOS (naturalness) | 4.21 | — | +| WER (intelligibility) | 7.45 % | — | +| Frozen pairwise study | **21-1-6** | near-tied 1st of 28 | + +Two honest caveats: the cloning arena averages ~30 games per model, so gaps under ~100 Elo are +noise, and the ranking uses a single reference clip. Echo's standing is robust to both — it is +top-3 on human votes *and* 2nd on objective SIM, which are independent measurements. + +Compute profile suits a GGUF port: ~2.8 B parameters at 1.35× RTFx and 9.4 GB VRAM in PyTorch on an +RTX 3090, so there is real work to amortise. + +## Architecture + +| Component | Params | Role | +|---|---:|---| +| EchoDiT trunk, 24 blocks | 1.75 B | Joint attention + SwiGLU MLP, adaLN timestep modulation | +| Text encoder | 294 M | UTF-8 **byte** tokens (256 vocab) — no phonemizer or G2P | +| Speaker encoder | 294 M | Reference PCA latents → speaker states | +| Latent encoder | 294 M | Blockwise only; omitted in M1 | +| PCA state | 83 K | Fish 1024-D ↔ DiT 80-D, `latent_scale` = 1/18 | +| Fish S1-DAC | 391 M weights | Reference encoding and waveform decoding | + +Sampling is 40 Euler steps with **two independent CFG scales** — text (default 3.0) and speaker +(default 8.0) — gated to `t ∈ [0.5, 1.0]`. + +Note the Fish checkpoint stores an additional 303.6 M elements of `freqs_cis` and `causal_mask` +buffers. These are regenerated at runtime rather than shipped in the GGUF. + +## Options + +| Option | Type | Default | Description | +|---|---|---|---| +| `target_voice` | string | — | Reference wav for cloning. No transcript needed. | +| `cfg_scale_text` | float | 3.0 | Guidance scale on the text condition. | +| `cfg_scale_speaker` | float | 8.0 | Guidance scale on the speaker condition. | +| `num_steps` | int | 40 | Euler sampler steps. | +| `truncation_factor` | float | 0.8 | Initial-noise truncation. | +| `speaker_kv_scale` | float | 1.0 | Force-speaker KV scaling; 1.5 is upstream's default when enabled. Raise only if the model drifts to a different speaker on out-of-distribution text. | +| `seed` | int | 0 | RNG seed for the initial latent. | + +## Text format + +Prompts follow the [WhisperD](https://huggingface.co/jordand/whisper-d-v1a) transcription style: + +- `[S1] ` is prepended automatically when neither `[S1]` nor `[S2]` is present. +- Colons, semicolons, and em dashes are normalised to commas. +- Commas generally function as pauses. +- Exclamation points and other emphatic punctuation increase expressiveness but can reduce quality. + +Multi-speaker dialogue is expressed with `[S1]` / `[S2]` tags. + +## Reference audio + +Up to 5 minutes is accepted; 10 seconds or less works well. Audio is mixed to mono, resampled to +44.1 kHz, and peak-limited before encoding. From 10e833e991707f525e783a9df3e8a433314e19a7 Mon Sep 17 00:00:00 2001 From: Tym Rabchuk Date: Mon, 3 Aug 2026 14:35:09 +0000 Subject: [PATCH 8/8] chore: keep internal planning docs out of the upstream PR The design spec and implementation plan are our working process, not content for audio.cpp. Preserved on the local echo-tts-planning branch and still on disk; just untracked here so the PR diff stays scoped to the actual contribution. --- .../plans/2026-07-30-echo-tts-m0-m1.md | 805 ------------------ .../specs/2026-07-30-echo-tts-port-design.md | 333 -------- 2 files changed, 1138 deletions(-) delete mode 100644 docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md delete mode 100644 docs/superpowers/specs/2026-07-30-echo-tts-port-design.md diff --git a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md b/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md deleted file mode 100644 index 197fafc4..00000000 --- a/docs/superpowers/plans/2026-07-30-echo-tts-m0-m1.md +++ /dev/null @@ -1,805 +0,0 @@ -# Echo-TTS Port — M0 + M1 Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Land a draft PR declaring the `echo_tts` family, then build a working offline decode path that generates 44.1 kHz audio from text + a pre-computed speaker latent, proven by ≥0.999 cosine parity against PyTorch and by ear. - -**Architecture:** Echo-TTS is a 24-block, d=2048 diffusion transformer operating in 80-D PCA space, decoded to waveform by the Fish S1-DAC. M1 deliberately ports only the **decode** half — the encode half (Fish encoder + RVQ, rated Hard) is deferred to M2 by injecting the speaker latent from a `.npy` dumped by the reference implementation. Each stage is landed behind its own parity gate so a numerical regression is caught at the stage that caused it, not at the end. - -**Tech Stack:** C++20, ggml, CMake; Python 3.12 + PyTorch/safetensors for conversion and parity dumps; `audiocpp_gguf` for GGUF emission. - -**Spec:** `docs/superpowers/specs/2026-07-30-echo-tts-port-design.md` - -## Global Constraints - -- Family slug is `echo_tts` everywhere: spec filename, directory names, CMake target, test dir. -- Loader symbol is `engine::models::echo_tts::make_echo_tts_loader` — namespace `models`, **not** `community_models`, even though sources live under `src/community_models/`. Mismatch is a link error. -- Spec goes in `model_specs/echo_tts.json` with `"schema_version": 1`. Do **not** create a `model_specs_v1/` copy. -- **No `loader.cpp`.** Spec-v1 models use the generic spec-backed loader. -- Option names are framework-validated: reference audio is `target_voice`; durations end `_sec`; never copy Python names into the spec. -- `capabilities` must **not** claim `long_form` in M0/M1. It is earned in M3 or not at all. -- Generation window is fixed: 640 latents × 2048 samples ÷ 44100 Hz = **29.7215 s**. -- `latent_scale = 0.0555555559694767` (= 1/18). `pca_components` is `[80,1024]`, `pca_mean` is `[1024]`. -- RoPE theta is `10000.0`, complex-valued, and **only half the heads are rotated**. -- RMSNorm and adaLN accumulate in **FP32**; Echo weights are BF16; sampler/PCA/Fish weights are FP32. -- Never serialise `freqs_cis` or `causal_mask` into GGUF (303.6 M elements). Regenerate at runtime. -- **Parity gate:** cosine similarity ≥ 0.999 over each tensor flattened to 1-D, reported with max-absolute-error. A stage is not done until its gate is green **when run**, not when reported. -- Reference implementation for all parity work: `/home/ryzen/LocalDev/tts-bench/venvs/echo/src/`, weights in `~/.cache/huggingface/hub/models--jordand--echo-tts-base` and `…--fish-s1-dac-min`. -- Hardware: RTX 3090 24 GB (compute capability **8.6**), CUDA. -- **Build invocation.** There is no `CMakePresets.json` in this repo — `cmake --build --preset …` will - fail. Use either `scripts/build_linux.sh --backend cuda --target ` or, against the existing - configured tree, `cmake --build build/linux-cuda-release --target -j`. `ccache` is - installed and there are 32 cores, so incremental rebuilds are cheap. -- **Model set.** `build/linux-cuda-release` is configured with `AUDIOCPP_MODEL_SET=full` and an empty - `AUDIOCPP_MODELS`, so a family registered via `audiocpp_add_model` is compiled in automatically. No - model-set flags needed. -- **CUDA architecture — must be fixed before any RTF number is quoted.** The existing - `build/linux-cuda-release` has `CMAKE_CUDA_ARCHITECTURES=75` (Turing) while the card is 8.6 - (Ampere). `CMakeLists.txt:1165-1168` defaults to `native` only when the variable is unset, so this - tree is pinned wrong. Development builds may proceed as-is, but **Task 13 must reconfigure with - `-DCMAKE_CUDA_ARCHITECTURES=86`** (or unset it to get `native`) before measuring, or the reported - RTF is invalid and would have to be retracted. - ---- - -## File Structure - -| Path | Responsibility | -|---|---| -| `model_specs/echo_tts.json` | Family metadata, tasks, options, packages. Single source of truth. | -| `tests/echo_tts/convert_echo_tts_weights.py` | Reference checkpoints → audio.cpp safetensors bundle → optional GGUF. | -| `tests/echo_tts/dump_echo_reference.py` | Dumps per-stage reference intermediates to `.npy` for parity. | -| `tests/echo_tts/compare_parity.py` | Cosine + max-abs-error comparator, exit non-zero on failure. | -| `tests/echo_tts/echo_tts_warm_bench.cpp` | C++ warm bench over the shared cases. | -| `tests/echo_tts/echo_tts_warm_bench_cases.json` | Shared case definitions. | -| `include/engine/community_models/echo_tts/assets.h` | Tensor handles resolved from the spec. | -| `include/engine/community_models/echo_tts/types.h` | POD config + request structs. | -| `include/engine/community_models/echo_tts/tokenizer_text.h` | WhisperD normalisation + UTF-8 byte tokenisation. | -| `include/engine/community_models/echo_tts/encoders.h` | Text and speaker encoder runtimes. | -| `include/engine/community_models/echo_tts/dit.h` | 24-block trunk forward. | -| `include/engine/community_models/echo_tts/sampler.h` | Euler loop + dual independent CFG. | -| `include/engine/community_models/echo_tts/fish_decoder.h` | PCA⁻¹ + post_module + upsample + decoder. | -| `include/engine/community_models/echo_tts/session.h` | Session wiring, loader factory. | -| `src/community_models/echo_tts/*.cpp` | Implementations, one per header. | - -Split rationale: each unit has its own parity gate, so each gets its own file. `dit.cpp` will be the largest; if it exceeds ~1500 lines, split blocks from the trunk driver. - ---- - -## Task 1: Model spec and family registration - -**Files:** -- Create: `model_specs/echo_tts.json` -- Modify: `CMakeLists.txt` (add `audiocpp_add_model(echo_tts …)` near the other community models, ~line 454) -- Create: `src/community_models/echo_tts/session.cpp`, `include/engine/community_models/echo_tts/session.h` - -**Interfaces:** -- Produces: `engine::models::echo_tts::make_echo_tts_loader()` → `std::shared_ptr` - -- [ ] **Step 1: Write the spec** - -Create `model_specs/echo_tts.json`. Model the shape on `model_specs/confucius4_tts.json`. Required content: - -```json -{ - "schema_version": 1, - "family": "echo_tts", - "display_name": "Echo-TTS", - "description": "Echo-TTS is an English zero-shot voice-cloning TTS model packaged for audio.cpp. A 2.8B diffusion transformer generates 80-D latents in PCA space which the Fish S1-DAC decodes to 44.1 kHz audio. Generation is a fixed 29.72 s window (640 latents).", - "category": "tts", - "status": "experimental", - "tasks": ["clone"], - "modes": ["offline"], - "languages": ["en"], - "runtime": { "tags": ["gguf"] }, - "capabilities": { "clone": ["speaker_reference"] }, - "options": { - "request": [ - { "name": "target_voice", "type": "string", "description": "Reference audio path for zero-shot cloning. Wav only; no transcript required.", "required": false }, - { "name": "cfg_scale_text", "type": "float", "description": "Classifier-free guidance scale on the text condition.", "required": false, "min": 0.0, "default": 3.0 }, - { "name": "cfg_scale_speaker", "type": "float", "description": "Classifier-free guidance scale on the speaker condition.", "required": false, "min": 0.0, "default": 8.0 }, - { "name": "num_steps", "type": "int", "description": "Euler sampler steps.", "required": false, "min": 1, "default": 40 }, - { "name": "truncation_factor", "type": "float", "description": "Initial-noise truncation factor.", "required": false, "min": 0.0, "max": 1.0, "default": 0.8 }, - { "name": "speaker_kv_scale", "type": "float", "description": "Force-speaker KV scaling. 1.0 disables; 1.5 is the upstream default when enabled.", "required": false, "min": 1.0, "default": 1.0 }, - { "name": "seed", "type": "int", "description": "RNG seed for the initial latent.", "required": false, "default": 0 } - ] - } -} -``` - -Note `capabilities.clone` deliberately omits `long_form`. - -- [ ] **Step 2: Write a spec-load test** - -Create `tests/echo_tts/echo_tts_warm_bench_cases.json` with one placeholder-free case: - -```json -{ - "default_clone": { - "requests": [ - { - "id": "chris_ref_p1", - "target_voice": "reference/chris_hemsworth_15s.wav", - "text": "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm.", - "seed": 0 - } - ] - } -} -``` - -- [ ] **Step 3: Verify the spec parses** - -Run: -```bash -python3 -c "import json; d=json.load(open('model_specs/echo_tts.json')); assert d['schema_version']==1; assert 'long_form' not in d['capabilities']['clone']; print('spec ok:', d['family'])" -``` -Expected: `spec ok: echo_tts` - -- [ ] **Step 4: Add the minimal session so the family links** - -`include/engine/community_models/echo_tts/session.h` declares: - -```cpp -#pragma once -#include "engine/framework/model_spec/metadata.h" -#include "engine/framework/runtime/session_base.h" -#include - -namespace engine::models::echo_tts { - -std::shared_ptr make_echo_tts_loader(); - -class EchoTtsSession final - : public runtime::RuntimeSessionBase, - public runtime::IOfflineVoiceTaskSession { -public: - EchoTtsSession( - runtime::TaskSpec task, - runtime::SessionOptions options, - std::shared_ptr contract); - ~EchoTtsSession() override; - - std::string family() const override; - runtime::VoiceTaskKind task_kind() const override; - runtime::RunMode run_mode() const override; - void prepare(const runtime::SessionPreparationRequest & request) override; - runtime::TaskResult run(const runtime::TaskRequest & request) override; - void reset() override; - -private: - runtime::TaskSpec task_; - std::shared_ptr contract_; -}; - -} // namespace engine::models::echo_tts -``` - -Implement `run()` in `session.cpp` to return 1.0 s of silence at 44 100 Hz for now. This proves the plumbing before any math exists. - -- [ ] **Step 5: Wire CMake** - -Add to `CMakeLists.txt` beside the other community models: - -```cmake -audiocpp_add_model(echo_tts - SOURCES - src/community_models/echo_tts/session.cpp - INCLUDES - engine/community_models/echo_tts/session.h - LOADERS - engine::models::echo_tts::make_echo_tts_loader -) -``` - -- [ ] **Step 6: Build and confirm the family registers** - -Run: -```bash -cmake --build build/linux-cuda-release --target audiocpp_cli -j 32 -./build/linux-cuda-release/bin/audiocpp_cli --list-loaders | head -1 -./build/linux-cuda-release/bin/audiocpp_cli --list-loaders | grep echo_tts -``` -Expected: the count line reads `registered_loaders=43` (baseline on this branch is **42**, verified -2026-07-30), and the grep prints `echo_tts: clon (offline)`. - -There is no `--list-families` flag; the flags are `--list-loaders [--json]` and `--list-pipelines`. - -If it link-errors on `make_echo_tts_loader`, the namespace is wrong — it must be -`engine::models::echo_tts`, not `engine::community_models::echo_tts`. - -If `--list-loaders` fails with something like `bs_roformer requires a schema v1 model contract`, the -binary is **stale**, not broken — rebuild `audiocpp_cli` and retry before investigating. - -- [ ] **Step 7: Commit** - -```bash -git add model_specs/echo_tts.json tests/echo_tts/ include/engine/community_models/echo_tts/ src/community_models/echo_tts/ CMakeLists.txt -git commit -m "feat(echo_tts): register family with spec v1 and silence stub" -``` - ---- - -## Task 2: Draft PR - -**Files:** -- Create: `docs/community_models/echo_tts.md` - -- [ ] **Step 1: Write the model doc** - -`docs/community_models/echo_tts.md` must state, without softening: -- Fixed 29.7215 s generation window; text beyond it is spoken faster, and the tokenizer hard-truncates past 768 UTF-8 bytes. -- Long-form is **not** supported in this PR. -- Licence: **CC-BY-NC-SA-4.0 on weights *and generated outputs*** — the output restriction is forced by the Fish S1-DAC dependency and is stricter than a weights-only NC licence. -- Benchmark provenance: #3 of 40 on cloning Elo (738 votes), SIM 0.836 (2nd of 41), UTMOS 4.21, WER 7.45 %, measured in tts-bench across 62 tracked models. - -- [ ] **Step 2: Push the branch** - -```bash -git push -u origin echo-tts-port -``` - -- [ ] **Step 3: Open the PR as a draft** - -```bash -gh pr create --repo 0xShug0/audio.cpp --draft \ - --title "Add Echo-TTS (community model) — WIP" \ - --body-file docs/community_models/echo_tts.md -``` - -The body must explicitly ask the maintainer three questions: -1. Is a fixed 29.72 s window acceptable for a community model, given `long_form` is not claimed? -2. Is the CC-BY-NC-SA **output** restriction acceptable in-tree? -3. If long-form is required, is a rolling latent-continuation approach acceptable given upstream calls its blockwise path "not thoroughly tested"? - -- [ ] **Step 4: Verify it is actually a draft** - -```bash -gh pr view --repo 0xShug0/audio.cpp --json isDraft,title -q '.isDraft' -``` -Expected: `true`. **The PR stays draft until every clause of Definition of Ready in the spec §5 is green.** - ---- - -## Task 3: Weight converter - -**Files:** -- Create: `tests/echo_tts/convert_echo_tts_weights.py` - -**Interfaces:** -- Produces: `models/echo-tts/audio_cpp/model.safetensors` with the tensor names consumed by Task 6. - -- [ ] **Step 1: Write the converter** - -Model it on `tests/confucius4_tts/convert_confucius4_tts_weights.py`. It must: -- Read `~/.cache/huggingface/hub/models--jordand--echo-tts-base` and `…--fish-s1-dac-min`. -- **Drop** every `latent_encoder.*`, `latent_norm*`, `*.wk_latent`, `*.wv_latent` tensor (blockwise-only; −294 M). -- **Drop** every `freqs_cis` and `causal_mask` buffer (regenerated at runtime; −303.6 M elements). -- **Fold weight normalisation** into static conv weights for the Fish decoder: for each conv storing `weight_g`/`weight_v`, emit `weight = weight_g * weight_v / ||weight_v||` over the norm axis, and drop the `_g`/`_v` pair. -- Copy `pca_components`, `pca_mean`, `latent_scale` through unchanged as FP32. -- Write a JSON sidecar recording every dropped key, so the drop is auditable. - -- [ ] **Step 2: Run it** - -```bash -cd /home/ryzen/LocalDev/audio.cpp -uv run --with torch --with safetensors --with numpy \ - python tests/echo_tts/convert_echo_tts_weights.py --output-dir models/echo-tts/audio_cpp -``` - -- [ ] **Step 3: Verify the drop maths** - -```bash -python3 -c " -import json,struct -f='models/echo-tts/audio_cpp/model.safetensors' -h=json.loads(open(f,'rb').read(8+struct.unpack(' int: - p = argparse.ArgumentParser() - p.add_argument("--ref", required=True) - p.add_argument("--got", required=True) - p.add_argument("--min-cosine", type=float, default=0.999) - a = p.parse_args() - ref = np.load(a.ref).astype(np.float64).ravel() - got = np.load(a.got).astype(np.float64).ravel() - if ref.shape != got.shape: - print(f"FAIL shape {ref.shape} vs {got.shape}") - return 1 - cos = float(ref @ got / (np.linalg.norm(ref) * np.linalg.norm(got))) - mae = float(np.max(np.abs(ref - got))) - ok = cos >= a.min_cosine - print(f"{'PASS' if ok else 'FAIL'} cosine={cos:.6f} max_abs_err={mae:.6e} n={ref.size}") - return 0 if ok else 1 - -if __name__ == "__main__": - sys.exit(main()) -``` - -- [ ] **Step 2: Write the dumper** - -`dump_echo_reference.py` loads the reference implementation exactly as `tts-bench/runners/echo_runner.py` does — including the `torchcodec`/`torchaudio` module stubs documented in that runner's docstring — seeds with `rng_seed=0`, runs one generation for the Task 1 case text against `reference/chris_hemsworth_15s.wav`, and saves each listed intermediate via forward hooks. - -- [ ] **Step 3: Run it** - -```bash -uv run --with torch --with numpy --with librosa --with soundfile \ - python tests/echo_tts/dump_echo_reference.py --out tests/echo_tts/parity -``` - -- [ ] **Step 4: Verify the dumps are sane** - -```bash -python3 -c " -import numpy as np, glob -for f in sorted(glob.glob('tests/echo_tts/parity/*.npy')): - a=np.load(f); print(f.split('/')[-1], a.shape, a.dtype, 'finite' if np.isfinite(a).all() else 'HAS NAN/INF') -" -``` -Expected: every file `finite`; `speaker_latent.npy` has shape `(1, Ls, 80)` with `Ls % 4 == 0`; `latents_final.npy` has shape `(1, 640, 80)`. - -- [ ] **Step 5: Sanity-check the comparator against itself** - -```bash -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/latents_final.npy --got tests/echo_tts/parity/latents_final.npy -``` -Expected: `PASS cosine=1.000000 max_abs_err=0.000000e+00 …` - -- [ ] **Step 6: Commit** - -```bash -git add tests/echo_tts/dump_echo_reference.py tests/echo_tts/compare_parity.py -git commit -m "test(echo_tts): reference parity dumper and cosine comparator" -``` - -Note: `.npy` dumps are build artefacts — add `tests/echo_tts/parity/` to `.gitignore`, do not commit them. - ---- - -## Task 5: GGUF emission - -**Files:** -- Modify: `tests/echo_tts/convert_echo_tts_weights.py` (add `--write-gguf`) - -- [ ] **Step 1: Add the GGUF flags** - -Mirror `convert_confucius4_tts_weights.py:33-36`: `--write-gguf`, `--gguf-output model.gguf`, `--gguf-type orig`, `--gguf-tool build/linux-cuda-release/bin/audiocpp_gguf`. The converter shells out to that tool; it does **not** write GGUF from Python. - -- [ ] **Step 2: Build the tool** - -```bash -cmake --build build/linux-cuda-release --target audiocpp_gguf -j -``` - -- [ ] **Step 3: Emit GGUF** - -```bash -uv run --with torch --with safetensors --with numpy \ - python tests/echo_tts/convert_echo_tts_weights.py \ - --output-dir models/echo-tts/audio_cpp --write-gguf --gguf-type orig -ls -la models/echo-tts/audio_cpp/model.gguf -``` -Expected: file exists. Given ~2.5 B BF16 Echo weights plus ~184 M FP32 Fish decode weights, expect roughly 5–6 GB; anything near 8 GB means the dropped buffers leaked back in — re-check Task 3 Step 3. - -- [ ] **Step 4: Commit** - -```bash -git add tests/echo_tts/convert_echo_tts_weights.py -git commit -m "feat(echo_tts): emit GGUF via audiocpp_gguf" -``` - ---- - -## Task 6: Assets, config, and speaker-latent injection - -**Files:** -- Create: `include/engine/community_models/echo_tts/types.h`, `assets.h` -- Create: `src/community_models/echo_tts/assets.cpp` -- Modify: `src/community_models/echo_tts/session.cpp` - -**Interfaces:** -- Produces: -```cpp -struct EchoTtsConfig { - int trunk_depth = 24; - int hidden_dim = 2048; - int latent_dim = 80; - int sequence_length = 640; - int samples_per_frame= 2048; - int sample_rate = 44100; - float rope_theta = 10000.0F; - float latent_scale = 0.0555555559694767F; -}; -struct EchoTtsAssets { // resolved tensor handles - assets::TensorHandle pca_components; // [80,1024] - assets::TensorHandle pca_mean; // [1024] - // … trunk, encoders, fish decode handles -}; -std::shared_ptr load_echo_tts_assets(const engine::model_spec::ModelContract &); -``` -- Produces: a debug session option `echo_tts.speaker_latent_path=` which loads the Task 4 `speaker_latent.npy` in place of native encoding. **This option is M1-only scaffolding and must be deleted in M2.** - -- [ ] **Step 1: Define config and assets headers** using the signatures above; take tensor names from `/home/ryzen/.claude/jobs/1464b16b/tmp/echo-tensor-manifest.txt`. - -- [ ] **Step 2: Implement `load_echo_tts_assets`** resolving every handle from the contract; throw with the missing key name if any handle is absent. - -- [ ] **Step 3: Add the `.npy` loader** for the injected speaker latent (little-endian float32, C-order; parse the standard `.npy` v1 header). - -- [ ] **Step 4: Verify assets resolve** - -```bash -cmake --build build/linux-cuda-release --target audiocpp_cli -j -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ - --text "[S1] test" --out /tmp/echo_stub.wav -``` -Expected: exits 0, still emits silence, and logs no missing-tensor error. A missing-key throw here names the exact tensor to fix. - -- [ ] **Step 5: Commit** - -```bash -git add include/engine/community_models/echo_tts/ src/community_models/echo_tts/ -git commit -m "feat(echo_tts): assets, config, and M1 speaker-latent injection" -``` - ---- - -## Task 7: Text tokenizer and text encoder - -**Files:** -- Create: `include/engine/community_models/echo_tts/tokenizer_text.h`, `encoders.h` -- Create: `src/community_models/echo_tts/tokenizer_text.cpp`, `src/community_models/echo_tts/encoders.cpp` - -**Interfaces:** -- Produces: -```cpp -std::vector echo_tokenize(const std::string & text); // WhisperD norm + UTF-8 bytes -class EchoTextEncoder { -public: - EchoTextEncoder(std::shared_ptr, core::BackendConfig, size_t arena_bytes); - // returns [1, T, 1280] - core::Tensor encode(const std::vector & tokens, const std::vector & mask); -}; -``` - -- [ ] **Step 1: Write the tokenizer test** - -Create `tests/echo_tts/test_echo_tokenizer.cpp`: - -```cpp -#include "engine/community_models/echo_tts/tokenizer_text.h" -#include -#include - -int main() { - using engine::models::echo_tts::echo_tokenize; - // "[S1] " is prepended when absent - auto a = echo_tokenize("hello"); - auto b = echo_tokenize("[S1] hello"); - assert(a == b); - // colons, semicolons, emdashes normalise to commas - auto c = echo_tokenize("[S1] a: b; c \xE2\x80\x94 d"); - auto d = echo_tokenize("[S1] a, b, c , d"); - assert(c == d); - // tokens are raw UTF-8 bytes, so every value is 0..255 - for (auto t : a) { assert(t >= 0 && t <= 255); } - std::cout << "tokenizer ok\n"; - return 0; -} -``` - -- [ ] **Step 2: Run it and watch it fail** - -```bash -cmake --build build/linux-cuda-release --target test_echo_tokenizer -j -``` -Expected: FAIL — `echo_tokenize` not defined. - -- [ ] **Step 3: Implement the tokenizer** per `inference.py` `tokenizer_encode`: normalise `:`/`;`/`—` to `,`, prepend `[S1] ` when neither `[S1]` nor `[S2]` is present, then emit raw UTF-8 bytes. - -- [ ] **Step 4: Run it and watch it pass** - -```bash -./build/linux-cuda-release/bin/test_echo_tokenizer -``` -Expected: `tokenizer ok` - -- [ ] **Step 5: Implement `EchoTextEncoder`** — the 294 M encoder body under manifest prefix `text_encoder.*`, with the `[256,1280]` embedding, and dump its output to `/tmp/echo_text_enc.npy` under a debug session option. - -- [ ] **Step 6: Gate on parity** - -```bash -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.dump_text_enc=/tmp/echo_text_enc.npy \ - --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ - --out /tmp/echo_stub.wav -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/text_enc.npy --got /tmp/echo_text_enc.npy -``` -Expected: `PASS cosine>=0.999`. **Do not proceed while this fails.** - -- [ ] **Step 7: Commit** - -```bash -git add include/engine/community_models/echo_tts/ src/community_models/echo_tts/ tests/echo_tts/ -git commit -m "feat(echo_tts): byte tokenizer and text encoder, parity-gated" -``` - ---- - -## Task 8: Speaker encoder - -**Files:** -- Modify: `include/engine/community_models/echo_tts/encoders.h`, `src/community_models/echo_tts/encoders.cpp` - -**Interfaces:** -- Consumes: injected `speaker_latent.npy` `[1,Ls,80]` from Task 6. -- Produces: `class EchoSpeakerEncoder { core::Tensor encode(const core::Tensor & speaker_latent); };` → `[1, Ls, 1280]` - -- [ ] **Step 1: Implement** the 294 M encoder under manifest prefix `speaker_encoder.*`, with the biased `320→1280` input projection and patch size 4. - -- [ ] **Step 2: Gate on parity** - -```bash -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ - --session-option echo_tts.dump_speaker_enc=/tmp/echo_speaker_enc.npy \ - --text "[S1] test" --out /tmp/echo_stub.wav -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/speaker_enc.npy --got /tmp/echo_speaker_enc.npy -``` -Expected: `PASS`. If `Ls % 4 != 0` the reshape will throw — the dumper already guarantees divisibility. - -- [ ] **Step 3: Commit** - -```bash -git commit -am "feat(echo_tts): speaker encoder, parity-gated" -``` - ---- - -## Task 9: DiT trunk - -**Files:** -- Create: `include/engine/community_models/echo_tts/dit.h`, `src/community_models/echo_tts/dit.cpp` - -**Interfaces:** -- Produces: -```cpp -class EchoDiT { -public: - // x:[1,640,80] latents, t: timestep, returns velocity [1,640,80] - core::Tensor forward(const core::Tensor & x, float t, - const core::Tensor & text_states, const std::vector & text_mask, - const core::Tensor & speaker_states, const std::vector & speaker_mask, - float speaker_kv_scale); -}; -``` - -- [ ] **Step 1: Implement one block first.** Port a single joint-attention + SwiGLU-MLP block with adaLN, from `model.py:128-268`. Critical details: RoPE theta 10000.0 rotating **only half the heads**; RMSNorm accumulating in FP32; adaLN modulating both attention and MLP from the timestep embedding; joint attention concatenating self + text KV + speaker KV with per-source boolean masks. - -- [ ] **Step 2: Gate block 0 on parity** - -```bash -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ - --session-option echo_tts.dump_dit_block=0:/tmp/echo_dit00.npy \ - --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ - --out /tmp/echo_stub.wav -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/dit_block00.npy --got /tmp/echo_dit00.npy -``` -Expected: `PASS`. A single block passing means the hard parts (half-head RoPE, FP32 norm, mask layout) are all correct — this is the highest-value gate in the plan. - -- [ ] **Step 3: Extend to all 24 blocks**, then gate blocks 11 and 23 the same way against `dit_block11.npy` and `dit_block23.npy`. - -- [ ] **Step 4: Commit** - -```bash -git commit -am "feat(echo_tts): 24-block DiT trunk, parity-gated at blocks 0/11/23" -``` - ---- - -## Task 10: Euler sampler with dual independent CFG - -**Files:** -- Create: `include/engine/community_models/echo_tts/sampler.h`, `src/community_models/echo_tts/sampler.cpp` - -**Interfaces:** -- Produces: `core::Tensor echo_sample(EchoDiT &, const EchoSamplerParams &, uint64_t seed);` → `[1,640,80]` - -- [ ] **Step 1: Implement** per `inference.py:361-419`. Required behaviour: 40 Euler steps; **two** guidance scales combined into one velocity; guidance active only for `t ∈ [cfg_min_t, cfg_max_t]` = `[0.5, 1.0]`; `truncation_factor` 0.8 applied to the initial Gaussian; unconditioning done by **masking**, not by zeroing encoder states. Note each guided step costs 3 DiT forwards (cond, text-uncond, speaker-uncond). - -- [ ] **Step 2: Gate final latents on parity** - -```bash -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ - --session-option echo_tts.dump_latents=/tmp/echo_latents.npy \ - --option seed=0 \ - --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ - --out /tmp/echo_stub.wav -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/latents_final.npy --got /tmp/echo_latents.npy -``` -Expected: `PASS`. If cosine is high but not ≥0.999, suspect the initial noise: Torch's Gaussian RNG is device-specific, so seed the C++ path from the dumped initial noise instead of regenerating it, and record that as a known parity caveat in the PR. - -- [ ] **Step 3: Commit** - -```bash -git commit -am "feat(echo_tts): Euler sampler with dual independent CFG, parity-gated" -``` - ---- - -## Task 11: PCA inverse and Fish S1-DAC decode - -**Files:** -- Create: `include/engine/community_models/echo_tts/fish_decoder.h`, `src/community_models/echo_tts/fish_decoder.cpp` - -**Interfaces:** -- Produces: `runtime::AudioBuffer echo_decode(const core::Tensor & latents_80d);` → 44 100 Hz mono - -- [ ] **Step 1: Implement PCA inverse** — `z1024 = (z80 / latent_scale) @ pca_components + pca_mean`. Gate it alone: - -```bash -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/pca_inv.npy --got /tmp/echo_pca_inv.npy -``` - -- [ ] **Step 2: Implement the decode stack** — `quantizer.post_module` → `quantizer.upsample` → `decoder`, regenerating `freqs_cis` and `causal_mask` at runtime rather than loading them. **Do not port the decoder transformer at `autoencoder.py:943-965`** — it exists only as an unregistered local variable and never executes; porting the apparent configuration would be silently wrong. - -- [ ] **Step 3: Gate decoded audio on parity** - -```bash -python3 tests/echo_tts/compare_parity.py --ref tests/echo_tts/parity/decoded.npy --got /tmp/echo_decoded.npy -``` -Expected: `PASS`. Causal-conv right-padding and transposed-conv asymmetric cropping are the likely culprits on failure — an off-by-one there shifts the whole waveform and tanks cosine. - -- [ ] **Step 4: Commit** - -```bash -git commit -am "feat(echo_tts): PCA inverse and Fish S1-DAC decode path, parity-gated" -``` - ---- - -## Task 12: Flattening-point crop, end-to-end, and the ear check - -**Files:** -- Modify: `src/community_models/echo_tts/session.cpp` - -- [ ] **Step 1: Implement the crop** per `inference.py:233-246` — scan 20-frame latent windows by standard deviation and mean, then cut the waveform at `frame × 2048`. This is a host-side loop, not a graph op. - -- [ ] **Step 2: Generate end-to-end** - -```bash -./build/linux-cuda-release/bin/audiocpp_cli --task clon --family echo_tts \ - --model models/echo-tts/audio_cpp --backend cuda \ - --session-option echo_tts.speaker_latent_path=tests/echo_tts/parity/speaker_latent.npy \ - --option seed=0 \ - --text "[S1] The operations desk reviewed the morning brief and confirmed the relay stayed online through the storm." \ - --out /tmp/echo_m1.wav -``` - -- [ ] **Step 3: Verify the output file mechanically** - -```bash -python3 -c " -import soundfile as sf -y,sr=sf.read('/tmp/echo_m1.wav') -print('sr',sr,'dur',round(len(y)/sr,3),'peak',round(float(abs(y).max()),4)) -assert sr==44100, 'wrong sample rate' -assert 1.0 < len(y)/sr < 29.8, 'duration outside the 29.72s window' -assert abs(y).max() > 0.01, 'output is silence' -" -``` -Expected: 44 100 Hz, a plausible duration well under 29.72 s, non-silent. - -- [ ] **Step 4: THE EAR CHECK — mandatory, not optional** - -Listen to `/tmp/echo_m1.wav` and compare against the reference wav produced by Task 4's dumper. Confirm: intelligible speech, the right words, no clicks at buffer boundaries, no metallic or phasey artefacts, and a voice that plausibly matches `chris_hemsworth_15s.wav`. - -Tensor parity **cannot** catch failures here — the flattening-point crop is a host-side loop outside the parity chain, and a wrong crop yields perfect cosine on latents with truncated or silence-padded audio. **M1 is not complete until a human has listened.** - -- [ ] **Step 5: Commit** - -```bash -git commit -am "feat(echo_tts): flattening-point crop and end-to-end M1 decode path" -``` - ---- - -## Task 13: Warm bench and evidence pack - -**Files:** -- Create: `tests/echo_tts/echo_tts_warm_bench.cpp` -- Modify: `CMakeLists.txt` (add `add_engine_warmbench(echo_tts_warm_bench tests/echo_tts/echo_tts_warm_bench.cpp)` near line 1315) - -- [ ] **Step 1: Write the warm bench**, modelled on `tests/confucius4_tts/confucius4_tts_warm_bench.cpp`, driven by `echo_tts_warm_bench_cases.json`. - -- [ ] **Step 2: Reconfigure for the correct CUDA architecture, then measure RTF and VRAM** - -The existing tree is pinned to `sm_75` on an `sm_86` card. Reconfigure before measuring: - -```bash -cmake -S . -B build/linux-cuda-86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON \ - -DCMAKE_CUDA_ARCHITECTURES=86 -cmake --build build/linux-cuda-86 --target echo_tts_warm_bench audiocpp_cli -j -./build/linux-cuda-86/bin/echo_tts_warm_bench \ - --model models/echo-tts/audio_cpp --backend cuda --runs 5 -``` - -Confirm the arch actually took before trusting the numbers: - -```bash -grep -E "^CMAKE_CUDA_ARCHITECTURES:" build/linux-cuda-86/CMakeCache.txt -``` -Expected: `CMAKE_CUDA_ARCHITECTURES:STRING=86` -Record wall time, audio length, **RTF = wall ÷ audio**, and peak VRAM for each run. - -Gates: **RTF < 1.0** (the community bar — note this is the inverse of tts-bench's RTFx; Echo's PyTorch 1.35× RTFx equals RTF 0.74, so the port should land near or below that), and **VRAM must not grow across the 5 runs**. - -- [ ] **Step 3: Assemble the evidence pack for the PR** - -Collect: exact build command, exact run commands, every parity line (`cosine=… max_abs_err=…`) from Tasks 7–11, the RTF table, the VRAM series, and `/tmp/echo_m1.wav` attached. - -- [ ] **Step 4: Commit and push** - -```bash -git add tests/echo_tts/echo_tts_warm_bench.cpp CMakeLists.txt -git commit -m "test(echo_tts): warm bench with RTF and VRAM measurement" -git push -``` - -- [ ] **Step 5: Post the evidence to the draft PR — and leave it in draft** - -M1 completes the decode path only. Cloning still requires an injected `.npy`, so the model is not yet self-contained and **Definition of Ready is not met**. The PR stays draft until M2 lands native speaker encoding. - ---- - -## Self-Review - -**Spec coverage.** §2 architecture → Tasks 6–11. §2.4 decode/encode asymmetry → Task 6 injection + M2 deferral. §3 long-form → deliberately out of scope, and Task 1 enforces it by omitting `long_form` from `capabilities`. §4 M0 → Tasks 1–2; M1 → Tasks 3–13. §5 Definition of Ready → Task 13 Step 3 assembles it and Step 5 explicitly withholds ready status. §6 integration surface → Task 1. §7 traps: trap 1 (phantom decoder) Task 11 Step 2; trap 2 (weight norm) Task 3 Step 1; trap 3 (FP32) Global Constraints + Task 9 Step 1; trap 4 (buffers) Task 3; trap 5 (half-head RoPE) Task 9 Step 1; trap 7 (causal padding) Task 11 Step 3; trap 8 (divisibility) Task 8 Step 2; trap 9 (mask uncond) Task 10 Step 1. §8 testing → Tasks 4, 7–12. **Gap found and closed:** trap 6 (Snake activation) had no owner — it lives in the Fish decoder and is now covered by Task 11 Step 2. - -**Placeholder scan.** No TBD/TODO. Every code step carries literal content. The one intentional stub (Task 1 silence) is named as such with a removal owner. - -**Type consistency.** `EchoTtsConfig`, `EchoTtsAssets`, `EchoTextEncoder::encode`, `EchoSpeakerEncoder::encode`, `EchoDiT::forward`, `echo_sample`, `echo_decode`, `echo_tokenize` are each declared once in Task 6/7/8/9/10/11 and referenced consistently thereafter. Debug session options use one `echo_tts.` namespace throughout. - -**Known scaffolding debt.** `echo_tts.speaker_latent_path` and the `dump_*` options are M1-only. M2's plan must open with their removal. diff --git a/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md b/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md deleted file mode 100644 index 96dfe1d0..00000000 --- a/docs/superpowers/specs/2026-07-30-echo-tts-port-design.md +++ /dev/null @@ -1,333 +0,0 @@ -# Echo-TTS port to audio.cpp — design - -Date: 2026-07-30 -Status: approved, pre-implementation -Target: community-model PR to `0xShug0/audio.cpp` - ---- - -## 1. Why this model - -### 1.1 Benchmark provenance - -This is not a model picked from a leaderboard screenshot. Echo-TTS has been independently -benchmarked in [tts-bench](https://github.com/5uck1ess/tts-bench) — a public benchmark tracking -**62 local TTS models** across three lenses (speed, objective scores, human preference) on three -rigs — and it was selected by comparing every tracked model against audio.cpp's existing support -table. The supporting data is already published and reproducible: - -- **Installed and run locally.** `venvs/echo/` with upstream source; both weight sets cached - (`jordand/echo-tts-base`, `jordand/fish-s1-dac-min`); a dedicated runner - (`runners/echo_runner.py`) documenting the exact upstream API and its gotchas. -- **Speed benched** on RTX 3090 CUDA, warm: **1.35× RTFx** (= RTF 0.74), 4 326 ms TTFA, 9 357 MB - peak VRAM. **Units warning:** tts-bench reports **RTFx** (higher = faster); audio.cpp's README - tabulates **RTF** (wall ÷ audio, lower = faster) alongside a separate "x faster than real time" - column. They are inverses. Echo's PyTorch 1.35× RTFx already satisfies the community RTF < 1.0 - bar before any GGUF work; do not invert these in the PR. -- **Objectively scored** over the bench prompt set: **16 rows** in `scoring/scores.csv` across - default and cloning lenses, via seed-tts-eval-style ASR + speaker verification. -- **Publicly auditioned**: generated wavs published to gh-pages and playable in the Listen lens. -- **Voted on blind**, twice — a frozen 397-vote pairwise study and an ongoing public arena that has - since collected 738 cloning votes and 1 415 default-voice votes. - -That measurement history is what makes the recommendation trustworthy, and it should be cited in the -PR body: the port is proposed because Echo *measured* well against 61 alternatives, not because it -looked promising. - -### 1.2 The result - -Echo-TTS is the highest-value model absent from audio.cpp, on three independent signals: - -| Signal | Value | Source | -|---|---|---| -| Human-preference Elo (cloning) | **1162, #3 of 40** on 35 games | tts-bench live arena, 738 cloning votes | -| Speaker similarity (SIM) | **0.836 — 2nd of 41** scored models | `tts-bench/scoring/scores.csv` | -| Frozen blind study | **21-1-6**, near-tied #1 | `tts-bench/docs/cloning.md`, 397 votes | -| UTMOS / WER | 4.21 / 7.45 % | same | -| Output rate | **44.1 kHz** | model card | - -Two qualifications, stated up front for honesty: the cloning arena averages ~30 games per model, so -gaps under ~100 Elo points are noise (the 1 415-vote default lens is firmer), and the whole cloning -ranking rests on a single reference clip (`chris_hemsworth_15s.wav`). Echo's position is robust to -both — it is top-3 on votes *and* top-2 on objective SIM, which are independent measurements. - -It is also **explicitly open for contribution**. Upstream issue #34 lists `~~echo-tts~~` struck -through under "Candidate models", with the legend: *"For models crossed out: I will not impl these -models myself, but contributions are welcome."* Struck-through entries carry **zero duplication -risk**; un-struck candidates (Magpie, LongCat, Soprano, MiraTTS) may still be maintainer work. - -Verified absent: no `echo`/`echodit`/`jordand` match anywhere in `src/`, `include/`, `docs/`, -`model_specs/`, `tools/`, or `README.md`; no PR (open/closed/draft) in 200+; no branch; GitHub code -search returns 0. - -Compute profile suits the framework. Echo is ~2.8 B at 1.35× RTFx and 9.4 GB VRAM in PyTorch — -heavy enough that GGUF and session amortisation pay off. (Contrast Kokoro, whose `preview/kokoro` -branch measures **0.20×** on the long-lived-session chart — 5× *slower* than Python — because an -82 M model has nothing to amortise.) - ---- - -## 2. Verified architecture - -All facts below were read from source at `tts-bench/venvs/echo/src/` and from safetensors headers. -Anything not established by those files is marked OPEN in §9 rather than guessed. - -### 2.1 Pipeline - -``` -reference wav - → decode ≤300 s → mono → resample 44 100 Hz → divide by max(|peak|, 1) - → truncate ≤ 6400×2048 samples; chunk at 640×2048; zero-pad final chunk - → fish_ae.encode_zq → PCA project 1024→80 → × latent_scale - → speaker_latent [1, Ls, 80], speaker_mask [1, Ls], Ls mod 4 == 0 - -text - → WhisperD normalisation: prepend "[S1] "; colons/semicolons/emdashes → commas - → UTF-8 *byte* tokens (256-entry vocab) - → text_encoder - -EchoDiT: 40 Euler steps in 80-D PCA space, latents [1, 640, 80] - → PCA⁻¹ → quantizer.post_module → quantizer.upsample → decoder - → waveform 44 100 Hz - → crop at flattening point (20-frame std/mean scan, cut at frame × 2048) -``` - -`640 × 2048 / 44100 = 29.7215 s` — the fixed generation window. - -### 2.2 EchoDiT - -| Property | Value | -|---|---| -| Trunk depth | 24 blocks | -| Hidden dim | 2048 | -| Attention | joint: self + text KV + speaker KV (+ latent-prefix KV, blockwise only) | -| MLP | SwiGLU | -| Conditioning | adaLN on both attention and MLP, driven by timestep | -| Positional | RoPE, theta **10000.0**, complex-valued, **rotating only half the heads** (`model.py:9`) | -| Norm | RMSNorm, FP32 accumulation | -| Timestep embedding | sinusoidal, `1000 · exp(−log(10000)·k)` (`model.py:35-40`) | - -Text frontend is **byte-level** — no phonemizer, no G2P, no external pronunciation dependency. -This is a significant scope win and removes the class of dependency problem that sank Kokoro. - -### 2.3 Parameter inventory - -| Component | Params | Needed for inference | -|---|---:|---| -| EchoDiT total | 2 800 742 736 | yes | -| — trunk joint attention (24) | 880 902 144 | yes | -| — trunk MLP (24) | 868 220 928 | yes | -| — attention adaLN (24) | 75 644 928 | yes | -| — MLP adaLN (24) | 75 644 928 | yes | -| — text_encoder | 294 000 640 | yes | -| — speaker_encoder | 294 083 840 | yes (when cloning) | -| — **latent_encoder** | 294 083 840 | **blockwise/long-form only** | -| — misc (timestep MLP, projections, norms) | 18 161 488 | yes | -| PCA state | 82 945 elements | yes | -| Fish S1-DAC checkpoint | 694 993 282 elements | — | -| — **trainable weights only** | **391 430 530** | — | -| — `freqs_cis` + `causal_mask` buffers | 303 562 752 | **regenerate at runtime, do not ship** | - -PCA: `pca_components [80,1024]`, `pca_mean [1024]`, `latent_scale [1] = 0.0555555559694767` (= 1/18). - -### 2.4 The decode/encode asymmetry - -Decode and encode need nearly disjoint Fish submodules: - -| Path | Modules | Approx weights | -|---|---|---:| -| **Decode** (generation) | PCA⁻¹, `quantizer.post_module`, `quantizer.upsample`, `decoder` | ~184 M | -| **Encode** (speaker ref) | `encoder`, `quantizer.downsample`, `quantizer.pre_module`, semantic RVQ + 9× residual RVQ, PCA forward | ~207 M | - -The decode path is entirely matmul/conv/transformer. The encode path needs RVQ nearest-neighbour -search, rated **Hard** to port. This asymmetry is the basis for the milestone split in §4. - -Note: `encode_zq` as written runs the *full* quantizer forward, then discards the result and -re-derives from the selected codes. `post_module` and `upsample` inside that first call can be -skipped — numerically equivalent, since only `codes` are consumed. - -### 2.5 Sampler - -`sample_euler_cfg_independent_guidances`: 40 Euler steps, **dual independent CFG** — `cfg_scale_text` -3.0 and `cfg_scale_speaker` 8.0 (5.0 in the blockwise example) — gated to `t ∈ [cfg_min_t=0.5, -cfg_max_t=1.0]`, `truncation_factor` 0.8. Unconditioning is **mask-based**, not zeroed encoder -states. Optional `speaker_kv_scale` ("Force Speaker", default 1.5 when enabled) corrects speaker -drift on out-of-distribution text. - ---- - -## 3. Long-form: rolling latent continuation - -**Blockwise does not extend past 640.** Verified directly: - -- `inference_blockwise.py:161` — `block_sizes=[128,128,64], # (sums to 320, ~15 seconds; supports up to 640)` -- `inference_blockwise.py:194-195` — `sum(block_sizes) + continuation_latent.shape[1] should be < 640` -- `README.md:122-124` — *"prefix and continuation are up to 30 seconds combined"*; *"Blockwise - functionality hasn't been thoroughly tested"* - -Blockwise **subdivides** one ≤30 s window; it does not extend it. Nor is there any text-compression -transform — long text fitting into 30 s is *learned* behaviour via global attention, and the -tokenizer hard-truncates past 768 UTF-8 bytes (`inference.py:146-149`). - -**Design:** carry the tail latents of chunk N directly into chunk N+1 as the continuation prefix. -Because we generate latents natively, this needs **no decode→re-encode round trip**. Each call -resets the window; the constraint is `prefix + new < 640` per call. - -This preserves prosody across joins, which crossfading cannot. Requirements and caveats: - -- Requires `latent_encoder` (+294 M) plus `wk_latent`/`wv_latent` — exactly what - `delete_blockwise_modules=True` strips. -- The prompt for chunk N+1 **must include the carried prefix's transcript**. -- Total prefix length must be divisible by 4 (speaker patch size, `model.py:458-459`). -- Upstream calls blockwise under-tested. **This must be disclosed in the PR, not discovered by the - maintainer.** - -Fallback if M3 fails validation: sentence-boundary chunking with `cross_fade_duration_sec` seams, -and drop the `long_form` capability claim. - ---- - -## 4. Milestones - -Each milestone has a gate. **No milestone is "done" on report — only on executed evidence.** - -**Decomposition note.** This spec deliberately covers the whole arc so the end state is agreed up -front, but it is too large for one implementation plan. M1 alone (GGUF conversion + a 2.5 B DiT + -the Fish decode stack, parity-gated) is a full plan on its own. Plan boundaries: **M0 + M1 together** -in the first plan; **M2**, **M3**, and **M4** each get their own plan written after the preceding -gate is green. Re-plan rather than extrapolate — M1's parity results will change what M2 should look -like. - -### M0 — spec + draft PR -- `model_specs/echo_tts.json`, `"schema_version": 1`, placed in `model_specs/` (not `model_specs_v1/`). -- `capabilities` **omits `long_form`** until M3 earns it. -- Draft PR opened, explicitly raising: the 29.72 s window, the blockwise-untested caveat, and the - CC-BY-NC-SA output-licence constraint. -- Gate: spec passes the framework schema validator (`src/framework/model_spec/schema.cpp:674-680` - checks `schema_version`); PR open and marked **draft**. - -### M1 — decode path, parity-gated -- GGUF conversion script; EchoDiT minus `latent_encoder`; PCA⁻¹; Fish decode path. -- Speaker latent injected from a `.npy` dumped by PyTorch — validates the hard 2.5 B without RVQ. -- Gate: per-tensor cosine ≥ 0.999 vs reference on fixed seed; generated wav audibly correct. - -### M2 — native speaker encoding -- Fish encoder + downsample + pre_module + semantic/residual RVQ + PCA forward. -- Gate: speaker latent from C++ matches PyTorch `encode_zq` → PCA output, cosine ≥ 0.999; - end-to-end clone from a raw wav with no Python in the loop. - -### M3 — long-form -- Rolling latent continuation per §3. -- Gate: `tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json` renders correctly and is - listened to end-to-end for seam artefacts. Only on success does `long_form` enter `capabilities`. - -### M4 — quantisation, performance, docs -- Q8_0 and F16 GGUF; `docs/community_models/echo_tts.md`; warm-bench test. -- Gate: **RTF < 1.0** (the explicit community bar); VRAM stable across repeated requests. - ---- - -## 5. Definition of Ready — the PR does not leave draft until all of these pass - -This is a hard gate, mirroring audio.cpp's stated review bar (issue #54 and README §36: *"exact -build/run commands, model paths or package ids, generated outputs, parity or path-test results, and -relevant performance or memory notes"*). - -1. **Builds clean** on Linux CUDA release; no new warnings in our files. -2. **Parity**: cosine similarity ≥ 0.999 against PyTorch on a fixed seed, computed over each - tensor flattened to 1-D, reported alongside max-absolute-error. Stages: DiT output, PCA⁻¹, - Fish decode, and (M2+) speaker encode. Numbers recorded in the PR. -3. **Path tests**: the family passes the CLI path-test matrix for safetensors, F16 GGUF, Q8_0 GGUF. -4. **Long-form**: the shared long-form clone case renders and is auditioned for seam artefacts — - or `long_form` is not claimed and the limit is documented. -5. **RTF < 1.0** measured on the RTX 3090, warm, with the command line included. -6. **VRAM stable** across ≥5 consecutive requests (no growth); `mem_saver` used if tuning is needed, - never to mask a leak. -7. **Generated wavs attached** for both default-reference and custom-reference cloning. -8. **Licence disclosed**: CC-BY-NC-SA-4.0 on weights *and outputs*. -9. **Independent review**: Codex authored → Claude reviews. Reviewer ≠ author, always. - -Only when 1–9 are green does the PR move from draft to ready-for-review. - ---- - -## 6. audio.cpp integration surface - -Follows Confucius4-TTS, the spec-v1 exemplar named in issue #128. - -``` -model_specs/echo_tts.json # schema_version 1 -src/community_models/echo_tts/*.cpp -include/engine/community_models/echo_tts/*.h -tests/echo_tts/echo_tts_warm_bench.cpp -docs/community_models/echo_tts.md -CMakeLists.txt # audiocpp_add_model(echo_tts SOURCES … INCLUDES … LOADERS …) -``` - -- **No `loader.cpp`.** Spec-v1 models use the generic spec-backed loader (issue #128). -- **Loader symbol is `engine::models::echo_tts::make_echo_tts_loader`** — namespace `models`, *not* - `community_models`, matching `inflect_v2`. Getting this wrong is a link error. -- **GGUF preferred over safetensors**, self-contained with the spec embedded; safetensors optional. -- **Normalised option names** (framework-validated): reference audio is `target_voice`, durations are - `*_sec`, chunking uses `audio_chunk_threshold_sec` / `audio_chunk_duration_sec` / - `cross_fade_duration_sec`. Do not copy Python names into the spec. - -Proposed options: `cfg_scale_text`, `cfg_scale_speaker`, `num_steps`, `truncation_factor`, -`speaker_kv_scale`, `seed`, `target_voice`. - ---- - -## 7. Implementation traps - -Each of these would cost days if hit blind. - -1. **`autoencoder.py:943-965` — decoder transformer that never executes.** It exists only as an - unregistered local variable. Porting the apparent configuration would be silently wrong. -2. **Weight normalisation**: most DAC convolutions store weight-norm parameters, not ready conv - weights. Fold at conversion time. -3. **FP32 boundaries are load-bearing**: RMSNorm and adaLN accumulate in FP32; the sampler, PCA, and - Fish weights are FP32 while Echo weights are BF16. Low-precision-only normalisation diverges. -4. **Do not serialise `freqs_cis` / `causal_mask`** into GGUF (303.6 M elements). Regenerate. -5. **Half-head RoPE**: the trunk rotates only half the heads — unusual, easy to get wrong. -6. **Snake activation** in the DAC likely needs a composed or custom kernel. -7. **Causal conv padding/cropping** computes right-padding from runtime length; transposed conv crops - asymmetrically. Off-by-one here is silent audio corruption. -8. **Shape divisibility**: speaker and prefix latents reshape in groups of 4. -9. **Mask-based unconditioning**: CFG unconditions via masks, not zeroed encoder states. - ---- - -## 8. Testing strategy - -- **Parity harness**: dump reference intermediates from PyTorch (fixed seed) to `.npy`; C++ loads and - compares per-stage with cosine + max-abs-error. Stage boundaries: text_encoder out, speaker_encoder - out, per-block DiT out (first/middle/last), final latents, PCA⁻¹ out, decoder out. -- **Bit-exactness is not the goal.** Gaussian RNG is device-specific; aim for statistical equivalence - on the noise and ≥0.999 cosine downstream. -- **Ear check is mandatory** at M1, M2, M3. Cosine can pass while audio is wrong (the flattening-point - crop is a host-side loop, not covered by tensor parity). -- **Regression**: reuse the bench's `chris_hemsworth_15s.wav` reference so output is directly - comparable to the 16 existing scored Echo rows in tts-bench. - ---- - -## 9. Open questions - -- `latent_scale` is resolved (1/18) but its *derivation* is unverified; confirm it is applied on both - the forward and inverse PCA legs consistently. -- Whether `quantizer.post_module` + `upsample` can be skipped in the M2 encode call without drift, as - §2.4 suggests. Verify numerically before optimising. -- Whether the maintainer will accept a `long_form` implementation built on an upstream path its own - author calls under-tested. Raise in M0. - ---- - -## 10. Licence - -Echo-TTS weights **and generated outputs** are CC-BY-NC-SA-4.0 — the output constraint is forced by -the Fish S1-DAC dependency. This is stricter than a weights-only NC licence and must be stated -plainly in `docs/community_models/echo_tts.md` and in the PR body. - -Precedent exists in-tree: `higgs_audio_tts` (Research NC) and `omnivoice` (Apache code / -CC-BY-NC weights). The *output* restriction appears to be new for audio.cpp — flag it explicitly -rather than letting it be inferred.