Live Voice: codex realtime voice over WebRTC - #2657
Open
colonelpanic8 wants to merge 32 commits into
Open
Conversation
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
|
| Filename | Overview |
|---|---|
| packages/app/src/live-voice/live-voice-runtime.ts | Implements the app-global Live Voice lifecycle, WebRTC startup, microphone lease ownership, teardown, and connection-loss handling. |
| packages/app/src/contexts/live-voice-context.tsx | Installs the singleton runtime and watches host client status and replacement events to coordinate teardown. |
| packages/server/src/server/live-voice/live-voice-coordinator.ts | Coordinates daemon-side call ownership, startup ordering, updates, and terminal cleanup. |
| packages/server/src/server/agent/providers/codex-app-server-agent.ts | Adds Codex realtime-conversation capability gating and app-server integration. |
| packages/protocol/src/messages.ts | Defines Live Voice requests, responses, updates, routing messages, and capability fields. |
| packages/app/src/live-voice/live-voice-session.web.ts | Implements browser and Electron WebRTC negotiation, media handling, and data-channel events. |
| packages/app/src/live-voice/live-voice-session.native.ts | Implements native WebRTC sessions and background-call lifetime integration. |
| packages/app/src/runtime/host-runtime.ts | Adds active-connection pinning needed to preserve exact socket ownership during calls. |
Sequence Diagram
sequenceDiagram
participant U as User
participant App as Paseo app
participant D as Owning daemon
participant C as Codex realtime
participant O as OpenAI
U->>App: Start Live Voice
App->>App: Acquire microphone lease and pin connection
App->>D: voice.live.start with WebRTC offer
D->>C: Start realtime conversation
C->>O: Establish realtime session
O-->>C: Answer SDP
C-->>D: Answer SDP
D-->>App: Start response
App->>O: WebRTC microphone audio
O-->>App: Remote speech audio
D-->>App: Finalized transcript updates
U->>App: Stop call
App->>D: voice.live.stop
App->>App: Close peer and release lease/pin
Reviews (8): Last reviewed commit: "feat(app): add Live Voice foreground deb..." | Re-trigger Greptile
colonelpanic8
added a commit
to colonelpanic8/paseo
that referenced
this pull request
Jul 30, 2026
Phase 1 promised immediate teardown on Paseo disconnect and only delivered the daemon half. The client subscribed to voice.live.update on whichever client existed at start and never observed connection state, so the daemon's owner_disconnected update could not reach the app over the socket that had just died: the runtime stayed active, the peer connection stayed open, and the app-global microphone lease stayed held, blocking dictation and voice mode until a reload. The runtime gains handleConnectionLost(serverId), which invalidates any in-flight start, tears the local half down, and reports the same owner_disconnected cause the daemon uses so the UI has one story. The provider drives it for both losses: a socket reporting anything other than connected, and the host swapping in a new client on reconnect (healthy client, dead call). Found by review on getpaseo#2657. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
Good catch on the connection-loss finding — confirmed and fixed in 6fdc5c2. It was the client half of a contract phase 1 only half-delivered: the daemon tears a call down immediately on owner disconnect, but the app subscribed to The runtime now exposes Covered by three new runtime tests: teardown frees the lease and sends no stop RPC, a loss for a different host is ignored, and a start that resolves after its connection was lost does not revive the call. Note for maintainers: the CI, Nix, and Docker workflows are sitting at |
New dotted-namespace RPC surface for Live Voice (codex realtime voice over WebRTC): voice.live.start/stop request-response pairs and the owner-targeted voice.live.update push with a discriminated event union (started, transcript, error, closed). Error/cause codes stay plain strings on the wire so new codes never break old clients. Adds the COMPAT-tagged server_info.features.liveVoice capability flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codex provider: spawn app-server with --enable realtime_conversation behind a 0.145.0 version floor, expose a per-session supportsLiveVoice capability, add thread/realtime/start|stop calls plus tolerant notification parsing routed to a dedicated realtime subscription (deliberately not AgentStreamEvent — call state is ephemeral, not timeline history), and give the transport an onClose hook so a dead codex process always produces a terminal event. New daemon-global LiveVoiceCoordinator owns one call per agent with atomic acquire (second owner gets busy), a starting/active/stopping/closed state machine, the answer SDP returned in the start response regardless of notification ordering, and exactly-once teardown from every terminal cause: client stop, owner socket detach (immediate, not the 90s retention), codex error/closed, transport death, and agent close/archive via the new AgentManager.onAgentClosing hook (which fires while the provider session is still alive, so codex is told to end the upstream realtime session). voice.live.* RPCs dispatch in session.ts with responses and updates emitted only to the owning client. Includes an end-to-end smoke script that drives a real Chrome offer through an in-process daemon against live codex. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
daemon-client grows startLiveVoice/stopLiveVoice (45s start timeout for the SDP round trip; rejections rethrown as LiveVoiceStartRejectedError with a stable errorCode). The web session module owns the full WebRTC sequence — mic, oai-events data channel before createOffer, complete ICE gathering, answer application, autoplay-policy recovery — with one idempotent cleanup on every exit path; native gets an unsupported stub via Metro platform files. A non-React runtime state machine drives the call, drops updates from stale liveSessionIds, and keeps finalized transcripts. A new app-global microphone lease makes voice mode, dictation, and live voice mutually exclusive (refuse, never interrupt), and stays held through a stop() that lands mid-negotiation so the still-open mic can't be double-acquired. Composer gains a Live Voice button and status panel, gated on web + the daemon liveVoice feature + the agent's supportsLiveVoice capability. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n it The realtime model is a conversational front end: everything it delegates becomes an ordinary turn on the attached agent session, and that session already launches with Paseo's own MCP server injected. So acting on Paseo needs no new tools (codex cannot give the realtime model client-defined tools anyway) — it needs the model to know Paseo exists and what is running. Sets three previously-unused start params: prompt (replaces the voice model's entire system prompt with one describing Paseo, delegation, and spoken-output discipline), initialItems (a developer-role snapshot of the attached session, other sessions, and workspaces), and includeStartupContext: false so codex's own context does not compete. Sections are dropped whole rather than truncated mid-list, well inside codex's 128-item / 8,192-token limits, since a half-listed set reads as complete and invites confident wrong answers. The prompt's claim that Paseo itself is actionable is conditional on AgentManager.hasPaseoMcpInjection(): with mcpInjectIntoAgents off the model is told it cannot manage workspaces or sessions, so it never promises work the session cannot do. A context build failure degrades to codex's default context instead of costing the user their call. The smoke script now runs with MCP enabled (production default) and asserts the echoed session instructions are the Paseo prompt, that tools were injected, and that the daemon seeded the snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 1 promised immediate teardown on Paseo disconnect and only delivered the daemon half. The client subscribed to voice.live.update on whichever client existed at start and never observed connection state, so the daemon's owner_disconnected update could not reach the app over the socket that had just died: the runtime stayed active, the peer connection stayed open, and the app-global microphone lease stayed held, blocking dictation and voice mode until a reload. The runtime gains handleConnectionLost(serverId), which invalidates any in-flight start, tears the local half down, and reports the same owner_disconnected cause the daemon uses so the UI has one story. The provider drives it for both losses: a socket reporting anything other than connected, and the host swapping in a new client on reconnect (healthy client, dead call). Found by review on getpaseo#2657. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tors zod-aot's discriminated-union emitter wrote the mapping key into the generated `case` label, which stringifies non-string discriminators (and any literal whose runtime value differs from its key) into something the validator can never match. Patch the emitter to resolve each option's literal value instead, alongside the existing runtime-import and output patches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…osts A Live Voice call was bound to the agent session whose codex thread hosted the realtime handshake, so talking to Paseo meant first opening some agent and the model could only act through that one session. Nothing about a voice call is workspace-shaped: the prompt already described every agent and workspace on the daemon. The daemon now spawns its own hidden internal codex session per call in the home directory and hosts the realtime session there, so `agentId` leaves `voice.live.*` entirely and a call is identified by `liveSessionId` alone. Exclusivity moves to one call per owning socket, the mid-turn `agent_busy` and `agent_not_found` rejections disappear with the attached agent, and a dying host session closes its call as `host_session_closed`. With no attached session to delegate to, the voice model instead routes work: new `voice.live.route.*` and tool-execution messages let it prompt sessions, open workspaces, and run Paseo tools on any saved host, brokered by the daemon and executed by the client that owns that connection. The client pins the exact daemon connection a call negotiated over so a reconnect cannot swap its transport mid-call. Protocol, client, server, and app land together: `voice.live.*` has not shipped, so the field is removed outright rather than deprecated, and the change is only valid once every layer agrees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The call outlives the screen it started on, so its UI cannot live in one agent's composer: the old panel rendered only for the agent that owned the call, and its transcript was an unbounded column that walked off screen on a long conversation. A call now has two homes and exactly one is active. When a visible sidebar offers a slot, a Discord-style card sits above the sidebar footer; when it does not — compact layouts inside a conversation, focus mode, a collapsed sidebar — the call falls back to a slim bar docked into the root layout's flow, so it never occludes the composer. Placement is a registration the sidebar slot claims while it is on screen rather than re-derived layout math, and it is claimed whether or not a call exists so ownership cannot flap mid-call. Both surfaces are thin readers over the app-global runtime and share their controls, status, and transcript. The transcript is collapsed to a preview by default and expands in place to a bounded, bottom-pinned scroll that follows the newest line unless the reader scrolls back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Muting is the reflex action mid-call — someone walks into the room — and reaching for a button in whichever surface currently owns the call is too slow. Ending the call deliberately keeps no keybind: it tears down the hidden host session and loses the realtime model's conversational memory, where mute is a local, instant, reversible track flip. The chord deliberately survives an editable focus scope, unlike voice mode's Space-to-mute, because you reach for it mid-keystroke; the terminal keeps first refusal on its own keys. The handler reports unhandled while no call is active so the chord stays inert and does not swallow the keystroke. Going through the declarative binding table means it shows up in the shortcuts dialog and honors user overrides like any other binding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Backgrounding the app killed the call: iOS suspends the audio session and Android reclaims the microphone, so walking away from the phone — the case hands-free voice exists for — ended the conversation. A native module holds the call alive on both platforms: an audio background mode and active AVAudioSession on iOS, a media-playback foreground service with a microphone type on Android. The native session begins the background call when the transport comes up and ends it on teardown, so the lifetime is tied to the call rather than to app state, and a platform that refuses the handoff surfaces `background_unavailable` instead of failing silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Where the realtime session lives, which socket owns a call, why media never carries an OpenAI key through the app, and the hop-by-hop route a tool call takes to reach another host are all facts you cannot recover from any one file in the implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
colonelpanic8
force-pushed
the
realtime-voice-actions
branch
from
July 30, 2026 22:48
6fdc5c2 to
00183bd
Compare
…ggregate version useHostRuntimeConnectionStatuses re-derived the whole status map from a version counter, so unrelated host churn could produce stale snapshots. Key it off each host's own connectionStatus string instead. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The always-visible launcher docked a full status bar (or sidebar card) at idle, with a raw debug dump of every host's connection state whenever live voice was unavailable — on mobile it ate the lower screen. Idle surfaces now render nothing; launching moves to a waveform icon button in the sidebar footer with an anchored start menu, and the per-host debug detail moves to Settings > Diagnostics as structured rows instead of a mono text blob. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
|
Too many files changed for review. ( Bypass the limit by tagging |
colonelpanic8
changed the title
colonelpanic8
added a commit
to colonelpanic8/paseo
that referenced
this pull request
Jul 31, 2026
Phase 1 promised immediate teardown on Paseo disconnect and only delivered the daemon half. The client subscribed to voice.live.update on whichever client existed at start and never observed connection state, so the daemon's owner_disconnected update could not reach the app over the socket that had just died: the runtime stayed active, the peer connection stayed open, and the app-global microphone lease stayed held, blocking dictation and voice mode until a reload. The runtime gains handleConnectionLost(serverId), which invalidates any in-flight start, tears the local half down, and reports the same owner_disconnected cause the daemon uses so the UI has one story. The provider drives it for both losses: a socket reporting anything other than connected, and the host swapping in a new client on reconnect (healthy client, dead call). Found by review on getpaseo#2657. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The realtime seam could only start and stop a call, so nothing that happened while the user was not speaking could ever reach the voice model. Add `realtimeAppendText` to the seam, implement it on codex via `thread/realtime/appendText`, and expose it as `LiveVoiceCoordinator.say`. Verified against a real codex realtime session: appending a developer item makes the model speak it unprompted, in its own words, without an explicit response trigger. `say` is ownership-checked rather than id-only: a client may put words in the mouth of its own call and no other, so the requesting socket must be the exact socket that owns the call. A provider failure comes back as a structured result instead of throwing, because callers deliver notifications and must not fail on a call that has since gone away. This is the delivery primitive on its own. The routed work-notification path that uses it lands separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Live Voice call gave no audible sign of starting or ending, which matters most on the surface where the screen isn't in view. Two synthesized cues, mirrored so they're distinguishable without looking: a rising two-note figure on connect, the same notes falling on disconnect. Both are generated from one tone spec, so web (Web Audio buffer) and native (WAV in the cache, played by expo-audio) sound identical. No third-party audio is added. Firing is edge-triggered off a single latch rather than derived from the phase, so snapshot churn (mute, transcripts, autoplay blocking) is silent, a start that never connected plays nothing, and stop()'s stopping -> idle pass chimes once. Native deliberately never calls setAudioModeAsync and never touches the shared two-way audio engine: WebRTC owns the session and the mic lease for the duration of a call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`isLiveVoiceBackgroundSessionSupported` was a hardcoded `true`, so an OTA bundle newer than the installed binary reported background mode as available and then threw from `requireNativeModule` when the call started. Resolve the module optionally and derive the flag from whether it is actually there. Also declare `androidx.lifecycle:lifecycle-runtime` explicitly: the module's Kotlin references `LifecycleOwner` and `Lifecycle` but only picked them up transitively.
A backgrounded call's foreground-service notification was static text. It is the call's only surface once Paseo leaves the screen, so muting or hanging up meant reopening the app — which is the thing backgrounding was supposed to avoid. The notification now carries Mute/Unmute and End call, reflects mute state in its title and text, and runs a chronometer for call duration. The service still owns no call state: a button press comes back through the Expo module as an event and the app runtime acts on it, so the runtime stays the only writer. State flows the other way through a new `update` function, pushed edge- triggered off the runtime snapshot — the runtime republishes for every transcript line, and rebuilding the notification on each would be dozens of posts a minute. Only `background` calls get this. A `foreground` call ends when the app leaves the screen, so a persistent control would outlive what it controls. iOS is unchanged; its module manages the audio session and has no notification to update, so `update` resolves silently there.
A routed Live Voice tool call has no caller agent id, deliberately, so it cannot claim an agent's workspace authority. Dropping the caller agent id also drops the agent-to-agent defaults that come with it, including background execution — so `create_agent` and `send_agent_prompt` reverted to blocking for the one caller that cannot survive a blocking call. When that happened the tool waited, `onBackgroundAgentStarted` never fired, no watch was ever registered, and the completion report the caller was promised was silently never sent. The voice model got dead air and the user got nothing, with no way for either to notice. The prompt already asked the model to pass `background: true`, but a guarantee that depends on the model remembering an optional argument is not a guarantee. Add `defaultAgentWorkToBackground` to the tool runtime context and set it whenever a routed call asked to be told the outcome. It flips the schema default rather than injecting the argument in the executor: schemas are built per runtime context, so the advertised default and the field's description stay the same fact, and tools keep owning the knowledge of which of them start work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Live Voice call only heard about work it started itself. An agent the
user launched from the app an hour ago could finish, fail, or block on a
permission request mid-call and say nothing, which is the case most worth
hearing about when you are away from the keyboard.
Add an ambient watch: with Settings -> General -> Live voice -> Report
agent activity on, the app arms every connected host that advertises
`liveVoiceAmbientAgentReports`, and those hosts report every agent that
finishes a turn, errors, or asks for permission.
The daemon-side watch is new code rather than a reuse of
`watchAgentFinish`, which watches one known agent and fires once. Here the
agent set is open and each agent keeps having turns for as long as the
call lasts, so it is one global subscription — which also picks up agents
created mid-call, and never sees internal agents, so the hidden host
cannot report on itself.
Correlation differs from a routed report, because nobody asked for these:
- By host, not requestId. There is no routed call to match, so an
unsolicited report resolves through the host the app armed. A host
reporting work the app never asked it to watch resolves to nothing.
The registration lasts the whole call instead of retiring after one
report.
- The model may stay silent. A routed report is an answer the user is
owed and its note says to speak. An unsolicited one says to use
judgement, and that saying nothing is a valid outcome.
No burst coalescing, recency filter, or event allowlist in code. What is
worth interrupting for depends on what the user is doing, so the setting
carries free text that goes into the prompt verbatim and outranks the
model's own judgement.
Teardown is call-scoped. The runtime does not await it, so a stop and the
next start overlap; disabling every host in sight let an old call's
teardown land after the new call's setup and silence it.
Also name the read tools in the prompt. Nothing told the model that
`get_agent_activity`, `get_agent_status`, or `list_pending_permissions`
existed, and "never poll for status" read as a ban on reads — so the
cheapest way to answer "what did it say?" was to prompt the agent, which
costs it a full turn and writes the question into its own history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CATEGORY_CALL was already set, but it is a ranking and DND hint — it does
not make the shade, the lock screen, or a paired watch treat the
notification as a call. CallStyle is what does, and it bridges to Wear with
call affordances rather than as a generic ongoing notification.
CallStyle owns the hang-up button, so the API 31+ path passes the End call
intent to `forOngoingCall` and adds only Mute alongside it. Below 31 the
notification keeps both explicit actions. minSdk is 29, so that fallback
covers 29 and 30 and nothing else.
The channel moves to IMPORTANCE_DEFAULT, which is what earns CallStyle its
full treatment. Two consequences worth stating:
- Importance cannot be raised on an existing channel — the user owns it
once it is created — so this takes a new channel id and deletes the old
one rather than leaving a dead entry in settings.
- DEFAULT alerts by default, and the app already plays its own connect
cue. The channel is explicitly silenced so the two do not stack.
Verified with `:paseo-background-call:compileDebugKotlin` and `lintDebug`
against a Nix-composed SDK (platform 36, JDK 21). Lint reports no NewApi
finding, so the version guard reads correctly. How CallStyle actually
renders, and whether it bridges to a watch as intended, still needs a device.
colonelpanic8
added a commit
to colonelpanic8/paseo
that referenced
this pull request
Aug 1, 2026
getpaseo#2657 grew its own CallStyle implementation while this branch carried one. Theirs is the better of the two and wins outright: it caught that a channel's importance cannot be raised once created, so moving IMPORTANCE_LOW -> DEFAULT needs a new channel id with the old one deleted. This branch's version left the channel at LOW, which would have taken CallStyle's degraded treatment without any sign that it had. Theirs also silences the channel so DEFAULT's alert does not beep over the app's own connect cue. BackgroundCallService.kt is therefore theirs verbatim, with only the two CallAudioRouter hooks re-applied. Git's auto-merge had kept both CallStyle blocks — adjacent enough to merge, wrong enough to render the call twice. Audio routing is untouched by any of this and stays: getpaseo#2657 does not select a communication device, so without CallAudioRouter the call still never reaches a Bluetooth headset's microphone, which is what puts it on a watch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Live Voice, an app-global realtime speech-to-speech mode that lets the user talk to Paseo through Codex's experimental realtime v3 API. Browser/Electron and native clients establish a WebRTC peer directly with OpenAI; the daemon carries only SDP, control messages, finalized transcripts, and routed Paseo tool calls.
Call architecture and lifecycle
--enable realtime_conversationbehind a Codex 0.145.0 version floor. The start response carries the answer SDP, avoiding a push/response race;voice.live.updatesends owner-only, monotonically sequenced started/transcript/error/closed events.Acting on Paseo across hosts
includeStartupContext: falseprevents Codex startup context from competing with it; context-build failure falls back to Codex defaults rather than failing the call.liveSessionId, selects and pins the saved target host, and executes against that daemon without a caller agent id or inherited workspace authority.live_voice_cross_host_routercapability and the target daemon'sliveVoiceToolExecutionfeature. Older clients keep local-only Live Voice behavior. If Paseo MCP injection is disabled, the prompt says the model can describe state but cannot act.App and mobile experience
Cmd+Shift+M/Ctrl+Shift+Mtoggles Live Voice mute globally (while preserving terminal key ownership and user shortcut overrides). Availability errors stay concise in the launcher; per-host connection/version/capability details live under Settings → Diagnostics. Host status subscriptions now recompute per host so unrelated connection churn cannot leave stale availability.react-native-webrtc; foreground mode remains app-bound, while background mode keeps calls alive across Home/screen lock through an iOSplayAndRecord/voiceChataudio session or an Android microphone/media-playback foreground service with an ongoing notification.Protocol and validation
voice.live.start/stop,voice.live.route, andvoice.live.tool.executerequest/response RPCs plus the owner-targetedvoice.live.updatepush. New server/client capabilities remain optional, and error/cause codes stay open strings for version skew.zod-aotgenerator so discriminated unions preserve non-string literal tags (used by booleanokresults) and recursive lazy JSON leaves fall back to runtime Zod instead of recursing into the WebSocket envelope. Regression coverage and protocol-validation documentation are included.Still out of scope
Proactive
appendSpeechnotifications, transcript deltas, a voice picker, and reconnect/resume after a real transport loss.Verification
packages/server/scripts/live-voice-smoke.ts, reported passing locally): isolated in-process daemon with MCP enabled + real Codex + real Chrome; applies the relayed answer, reaches WebRTCconnected, observessession.started, verifies the Paseo prompt/tool injection/snapshot metadata, receives owner-targeted monotonic updates, and stops cleanly.npm run typecheck,npm run lint, andnpm run formatwere reported clean; full suites remain delegated to CI per repository policy.Manual physical-device coverage still required: