feat(voice): 新增基于 WebSocket 连接池的流式 TTS 消息播放 - #116
Open
juzi050 wants to merge 3 commits into
Open
Conversation
- 后端:SSE 推送音频帧,TTS 音色与音频格式由候选模型配置声明 - 前端:播放按钮 + MSE 边收边播,切会话自动停止 - 连接池:空闲驱逐配置,规避供应商空闲断开后的假活连接 - TTS:分块流式发送与帧间空闲超时,长文本合成收尾不超时
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
演示视频:ragent-语音播放功能演示
Closes #108
背景
当前聊天消息仅支持文本阅读。为了让用户直接收听已生成的 assistant 消息,本次新增端到端的流式 TTS 播放能力:后端从消息记录读取文本,通过池化 WebSocket 调用百炼 CosyVoice,并使用 SSE 将 MP3 音频帧持续推送给浏览器;前端基于 MediaSource 边接收、边缓冲、边播放。
同时抽象了可复用的 Voice WebSocket 连接与任务生命周期,避免每次播放都重新建立物理连接,并收敛成便于扩展其他基于 WebSocket 的流式语音能力。
设计参考
主要变更
接口与事件约定
GET /rag/v3/voice/play?messageId=...:建立语音播放 SSE。POST /rag/v3/voice/stop?taskId=...:停止播放任务。audio-meta→audio× N →done;audio-meta携带任务 ID,audio携带 Base64 MP3 音频帧。完整流程
/rag/v3/voice/play,后端校验并读取当前用户的消息文本。audio-meta(taskId),随后在线程池中异步调用TtsService。modelId的连接池借用 WebSocket。run-task、分块continue-task和finish-task。task-finished后发送done、关闭 SSE,并将正常连接归还池中。设计图
WebSocket 包
展示连接资源生命周期与任务生命周期如何汇聚到物理连接核心。
TTS 包
展示 TTS 业务入口、候选路由、首音频探测桥接、任务观察者及供应商 WebSocket 客户端之间的职责与继承关系。
WebSocket 单连接生命周期
展示单条物理连接从建立、执行任务、正常复用,到取消、异常及关闭的状态变化。
TTS 正常播放时序
展示播放请求、模型选择、流式音频通过 SSE 进入 MSE 播放,以及正常完成后的连接复用。
TTS 取消时序
展示用户发起停止后立即清理本地播放器,后端异步取消远端合成、销毁不可复用连接,并在取消完成后关闭 SSE。
验证结果
BaiLianTtsClientTest:8 个测试通过。RoutingTtsServiceTest:4 个测试通过。ModelSelectorTest:10 个测试通过,包含 TTS 候选和统一超时映射。npm run build通过。git diff --check通过。