Repository navigation
Stream the PC audio to the tablet - #10
Merged
Merged
Conversation
Adds protocol minor 6: the client advertises what it can play (ClientAudio), the host picks PCM or Opus (AudioConfig) and sends AudioPacket frames on the audio channel. The mux gives audio its own short queue (and its own QUIC stream) between cursor and video, dropping the oldest packets when the link stalls. The host session pumps an optional AudioBackend on its own thread; an audio failure stops only the audio.
Adds an AudioEncoder in ezd-ffmpeg (48 kHz, 10 ms frames, low-delay application) that the host uses when Opus is negotiated, and builds libopus statically into the AppImage's FFmpeg.
Adds an AudioSource that records the default output device as 48 kHz float in shared loopback mode. The PC keeps playing the sound locally.
Each session creates an "ezdisplay" sink, makes it the configured default output and records its monitor at 48 kHz. Stopping puts the previous default back unless it was changed meanwhile; the sink does not linger, so a crashed host leaves nothing behind.
The Android client advertises audio when the new Audio setting is on (default on), plays PCM through a low-latency AAudio stream behind a 20 ms jitter buffer, and decodes Opus with MediaCodec.
ServiceConfig.audio (auto, pcm, opus, off) feeds the negotiation and decides whether sessions get the platform capture and the FFmpeg Opus encoder. ezd-host takes --audio, diag reports whether capture and Opus are available, and the tray gets an Audio choice plus the negotiated stream in its live status.
# Conflicts: # android/app/src/main/kotlin/com/librestatic/ezdisplay/MainActivity.kt # crates/ezd-android/src/lib.rs # crates/ezd-client-core/src/session.rs # crates/ezd-fake-client/src/lib.rs # crates/ezd-host-linux/src/lib.rs # crates/ezd-host-service/src/platform.rs # crates/ezd-proto/src/lib.rs # crates/ezd-proto/src/msg.rs # docs/protocol.md
The prebuffer starts at 20 ms, grows on every underrun and shrinks after a calm stretch. When the PC clock runs ahead, single frames are dropped until the buffer is back near its target.
FFmpeg 9 broke three things the path relied on: scale_vulkan can no longer convert RGB to YUV, a CUDA device derived from Vulkan cannot push its context, and filters cannot allocate output from a Vulkan frames context derived from DRM. Frames now stay RGB and NVENC converts them, as on the shared-memory path; Vulkan is derived from CUDA; and captured buffers are imported into a plain Vulkan frames context. HDR needs the YUV conversion, so it takes shared memory.
AudioVolume (Control/20, protocol 0.7) carries a perceptual volume and a mute flag both ways: the tablet sends its value when audio starts, then whichever side changes its volume sends it, and echoes are dropped. A client that sends a new Hello mid-session (its app restarted on the same link) now ends the session so the host reopens it.
# Conflicts: # docs/protocol.md
FFmpeg's configure links its libopus check with pkg-config's shared flags, which leave out libm, so the check failed on floorf and sqrtf.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streams the PC's audio to the tablet (protocol 0.7).
What's in it
ClientAudio,AudioConfigand the Audio channel negotiate and carry one audio stream;AudioVolume(Control/20) keeps the volume in sync both ways. Peers below 0.7 keep working without audio.--audio auto|pcm|opus|off,audio = "..."inconfig.tomland a tray setting choose it.ezdisplaybecomes the default output for the session and the previous default comes back afterwards. Its volume and mute follow the tablet's.Hellomid-session (its app restarted) ends the session so the host reopens it.nvenc-dmabuf-rgb): frames stay RGB and NVENC converts them, Vulkan is derived from CUDA, and buffers are imported into a plain Vulkan frames context. HDR on NVENC takes shared memory.Testing
cargo fmt --check,cargo clippy --workspace --all-targets -D warnings,cargo test --workspaceand./gradlew testFullDebugUnitTest(150 tests) pass.