feat: Dreamfinder voice agent — LiveKit audio ↔ OpenAI Realtime#11
Merged
Conversation
…ridge Dreamfinder joins Tech World as a voice-interactive bot participant. Audio flows: LiveKit AudioStream → AudioMixer → resample 48k→24k → OpenAI Realtime API WebSocket → resample 24k→48k → AudioSource → room. New files: - openai-realtime.ts: Server-side WebSocket client for OpenAI Realtime API - audio-pipeline.ts: LiveKit ↔ OpenAI bidirectional audio bridge - dreamfinder-entry.ts: DF-specific agent entry with proximity audio gating - prompts/dreamfinder.ts: System prompt + tool definitions Also: bot-config.ts gains dreamfinder entry, index.ts dispatches to DF's entry when agentName matches. ws package added for server WebSocket. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
nickmeinhold
added a commit
that referenced
this pull request
Apr 29, 2026
Two unused declarations from PR #11 (dreamfinder voice agent) were flagging the lint job. Surgical removal — unblocks CI without admin bypass, leaves the codebase healthier. - src/dreamfinder-entry.ts: drop unused `TrackPublication` type import - src/openai-realtime.ts: drop unused `SAMPLE_RATE` constant npm run lint exits 0; tsc --noEmit unchanged.
enspyr-admin
pushed a commit
that referenced
this pull request
Apr 29, 2026
…#12) * feat: oracle-request handler — bot-mediated flavor generation channel Adds the producer side of the oracle-request / oracle-response wire contract introduced in tech_world PR #307. Phase 2 of the Speech-Command Spell System uses this for `kind: 'cast_no_match'`: when a player speaks a word that doesn't match any WordId, the Flutter client asks the bot for a fresh flavor line via Claude. Wire contract (defined and tested on the consumer side in #307): client → bot topic: oracle-request, targeted to bot-claude {requestId, kind, context} bot → client topic: oracle-response, targeted to senderId {requestId, text} Phase 2 supports `kind: 'cast_no_match'` only. Phase 3's spell-combo interpretation will reuse the same channel with a new kind discriminator (~30 lines per side). Changes: - prompts/clawd.ts — ORACLE_NO_MATCH_PROMPT in the oracle's voice (not Clawd's tutor voice). One-sentence flavor lines, 8-16 words, register matches the canonical "the words swirl but find no form" line. Style references in the prompt; explicit instruction to vary every time. - bot-config.ts — oracleNoMatchPrompt field threaded through all three bots: set on clawd, null on gremlin and dreamfinder. - index.ts — handleOracleRequest dispatcher: parses the request, routes by kind, calls Claude (haiku-4-5, max_tokens: 64), publishes the response targeted to the requester. Silent on failure — Flutter falls back to its hand-crafted pool on timeout, so flavor is best- effort by design. Type-check: tsc --noEmit exits 0. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: remove pre-existing unused-vars surfaced by lint on this PR Two unused declarations from PR #11 (dreamfinder voice agent) were flagging the lint job. Surgical removal — unblocks CI without admin bypass, leaves the codebase healthier. - src/dreamfinder-entry.ts: drop unused `TrackPublication` type import - src/openai-realtime.ts: drop unused `SAMPLE_RATE` constant npm run lint exits 0; tsc --noEmit unchanged. --------- Co-authored-by: Nick Meinhold <nick@enspyr.co> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
openai-realtime.ts): Server-side connection to gpt-4o-realtime-preview with API key auth, PCM16 audio streaming, tool call handling, mood tag extraction.audio-pipeline.ts): Bidirectional bridge — LiveKit AudioStream (48kHz) → AudioMixer → AudioResampler → OpenAI (24kHz) → AudioResampler → AudioSource → room. Proximity-based: only mixes audio from players within 2 grid squares.dreamfinder-entry.ts): DF-specific entry with audio pipeline init, position tracking for proximity gating, mood publishing via data channel.dreamfinderentry — identitybot-dreamfinder, facilitator wander pace (3-8s pauses), voice tools.Test plan
npm run dev -- --bot=dreamfinder— DF connects to LiveKit room🤖 Generated with Claude Code