Read aloud: speak selected text on web and desktop

[kaspesi]

Select text in the web or Electron app and a speaker button floats above the selection. Press it and the daemon speaks the selection back; press again, or clear the selection, to stop.

How it works

Uses the existing voice-mode TTS provider (local Kokoro by default, or OpenAI) rather than standing up a second provider stack — that one already has config, env vars, and docs plumbing.

Synthesis is segmented with a 2-segment prefetch. Local TTS returns raw 24 kHz PCM (~48 KB/s), so a long selection delivered in one message would be megabytes.

Piece	What it does
protocol/src/messages.ts	speech.tts.read_aloud.request → N .response segments; …cancel_read_aloud.request
server/session/voice/read-aloud-controller.ts	Sanitize → split → synthesize with 2-segment prefetch → stream
server/speech/read-aloud-text.ts	Strips markup before synthesis
server/speech/tts-text-splitter.ts	Extracted from tts-manager.ts; now shared with voice mode
client/src/daemon-client.ts	startReadAloud() returns a handle with cancel()
app/src/read-aloud/use-selection-anchor.web.ts	Endpoint rects + the clipping ancestor's visibleBox; retained during playback
app/src/read-aloud/read-aloud-placement.ts	Pure above/below/park decision and clamping
app/src/read-aloud/read-aloud-selection-bubble.web.tsx	Renders into #overlay-root; owns widths and stop intent
app/src/read-aloud/read-aloud-store.ts	Playback state machine, generation counter, speed
app/src/read-aloud/read-aloud-audio.web.ts	Reuses the voice-mode AudioEngine (playback context only, no mic)

Scope

Web and Electron only. React Native exposes no JS API for the current text selection — iOS hands it to the system edit menu, Android to an ActionMode — so there is nothing to anchor to on the mobile apps. read-aloud-selection-bubble.tsx (the non-.web file) is a deliberate return null. Closing that gap would mean a turn-level "Read aloud" button reading the whole turn; not started.

Bound to the route's host. The selection is spoken by the host it came from, never another paired daemon — the text is workspace content, so crossing hosts would disclose it across an independently paired boundary. No fallback: a route host that doesn't advertise the capability shows no button, and neither does a route with no host at all (settings, history). useReadAloudServerId is not unit-covered — it composes useLocalSearchParams with a zustand selector, so covering it needs a render harness; the invariant is a single comparison.

Capability-gated on server_info.features.readAloud (COMPAT(readAloud), v0.2.5). Hosts without it get no button at all — an upgrade prompt on every text selection would be worse than the feature being absent. No fallback path.

Verification

• npm run typecheck, npm run lint, npm run format:check — green
• 24 server tests — read-aloud-text.test.ts 9, read-aloud-controller.test.ts 9, tts-manager.test.ts 6 (that last one covers the splitter extraction, so voice mode's segmentation is still under test)
• 69 app tests — read-aloud-placement.test.ts 14, read-aloud-store.test.ts 5, voice-runtime.test.ts 18, i18n/resources.test.ts 32
• Verified in a real browser against the dev stack: button appears above the selection, press → stop square with the selection surviving the press, press again → idle, click away → button gone and playback stopped. Audio confirmed reaching the output graph by instrumenting AudioContext — two segments, non-silent buffers (peak 0.27 / 0.35), context running.
• All CI checks green. Getting there surfaced one real bug in agent-manager.test.ts, fixed in a separate commit: cancelAgentRun succeeds when the provider queues completion before rejecting the interrupt failed on server-tests (windows-latest) with ENOTEMPTY: rmdir '...\\agents'. The assertions passed — the fixture's teardown threw. AgentManager can flush an agent snapshot into agents/ while cleanup runs, and on Windows an open handle makes rmdir fail with ENOTEMPTY, which force: true does not cover. Fixed with the maxRetries: 5, retryDelay: 100 options already used in spawn.launch-regression.test.ts and run-git-command.windows-shell.test.ts; only the shared fixture is touched, since the file's other rmSync sites have not failed. Note that server-tests is load-sensitive under --fileParallelism and has separately failed with an unrelated timeout in execution-session.websocket.test.ts and, on ubuntu, inside checkout-git.commits.test.ts (passes locally 12/12) — this PR removes one failure mode, it does not make that job immune.
• Placement measured against nine real getBoundingClientRect() browser cases; see the Outcome section of docs/refactors/read-aloud-selection-anchoring-plan.md.

Known gaps — not fixed here

The GIF makes this look more finished than it is. Three things are open:

1. Segment gaps with local Kokoro. It runs ~1× realtime with a ~6 s cold start, and the splitter emits one segment per sentence — so a short first sentence drains before the next finishes synthesizing. Measured 6.1 s of silence mid-read. Three fixes, none implemented and none chosen: pack short sentences into ~250-char chunks read-aloud-side only (recommended — don't touch the shared splitter, voice mode wants the fast short first segment); buffer two segments before starting (gapless, but start moves ~6 s → ~13 s); or use OpenAI TTS, which is far faster than realtime.
2. Error detail is dropped. Unknown failure codes render a generic "Couldn't read that aloud" while the daemon's real message sits unused in failure.message.
3. Markup sanitization is unit-tested only. It has not yet been exercised against a live daemon — selecting a <spoken-input> block should speak only the inner text.

docs/refactors/read-aloud-handoff.md carries the full state, the environment gotchas (Node 22+ for the dev daemon, no daemon hot-reload, Buffer is not a browser global), and the open questions.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds selection-based read-aloud support for web and Electron.

• Adds route-host capability gating, selection anchoring, floating playback controls, speed selection, and streamed browser audio playback.
• Adds client/protocol and daemon support for segmented TTS synthesis and cancellation.
• Extracts the shared TTS text splitter and adds read-aloud sanitization, placement, store, controller, and voice-runtime tests.
• Documents the feature architecture, browser anchoring behavior, and current limitations.

Confidence Score: 4/5

This PR is not yet safe to merge because retained selections can still cross host boundaries and audio decoding can still escape an explicit stop.

Navigating directly between two capable hosts can leave the old selection anchor paired with the new route client's daemon, while the playback engine also has an uncancellable interval after dequeue and before activePlayback registration that permits a stopped segment to start later.

Files Needing Attention: packages/app/src/read-aloud/read-aloud-selection-bubble.web.tsx, packages/app/src/read-aloud/use-selection-anchor.web.ts, packages/app/src/voice/audio-engine.web.ts, packages/app/src/read-aloud/read-aloud-audio.web.ts

Important Files Changed

Filename	Overview
packages/app/src/read-aloud/read-aloud-selection-bubble.web.tsx	Adds the route-gated selection bubble, controls, placement, and playback integration; the prior retained-anchor host-binding issue remains outstanding.
packages/app/src/read-aloud/use-selection-anchor.web.ts	Adds selection capture, clipping-ancestor geometry, endpoint tracking, and detached-range retention.
packages/app/src/voice/audio-engine.web.ts	Adds mutable playback speed, while the previously reported stop-during-decode race remains outstanding.
packages/app/src/read-aloud/read-aloud-store.ts	Adds the read-aloud playback state machine, generation-based callback suppression, rate persistence, and daemon cancellation.
packages/client/src/daemon-client.ts	Adds the typed streaming read-aloud request handle and cancellation path.
packages/server/src/server/session/voice/read-aloud-controller.ts	Adds sanitized, segmented TTS synthesis with bounded prefetch and response streaming.
packages/protocol/src/messages.ts	Adds read-aloud request, response, cancellation, error, and capability message schemas.

Sequence Diagram

sequenceDiagram
  participant U as User
  participant B as Selection bubble
  participant C as Route host client
  participant D as Route host daemon
  participant T as TTS provider
  participant A as Web AudioEngine
  U->>B: Select text and press Read aloud
  B->>C: startReadAloud(selected text)
  C->>D: speech.tts.read_aloud.request
  D->>T: Synthesize prefetched segments
  T-->>D: Segment audio
  D-->>C: Stream segment responses
  C-->>B: onSegment(audio)
  B->>A: Queue and play segment
  U->>B: Stop or clear selection
  B->>C: cancel request
  B->>A: Clear queue and stop playback

Loading

_{Reviews (4): Last reviewed commit: "test(agent-manager): retry the interrupt..." | Re-trigger Greptile}

[greptile-apps]

Retained text crosses hosts

When playback of text selected on host A finishes after navigation to capable host B, the detached anchor remains available until the selection hook's next asynchronous commit, while client already refers to host B. Pressing the briefly retained bubble sends host A's text to host B for synthesis, disclosing workspace content across independently paired daemons.

How this was verified: The request combines the current route client with retained anchor text without recording or validating the anchor's originating server.

Knowledge Base Used: Paseo Mobile App (packages/app)