Skip to content

feat(linux): Linux desktop support — Part 4/6: Audio, video, and media sessions - #42

Merged
thtbee merged 12 commits into
thtbee:mainfrom
jochenkirstaetter:linux-4-media
Oct 23, 2026
Merged

thtbee merged 12 commits into
thtbee:mainfrom
jochenkirstaetter:linux-4-media

Conversation

@jochenkirstaetter

Copy link
Copy Markdown
Contributor

Summary

Part 4 of 6 for Linux desktop support (stacked on #41 / linux-3-input and #38 / linux-2-basics).

Adds Linux audio capture/render, media session (MPRIS) integration, screen capture, hardware/software H.264 video pipeline, MP4 muxer, and virtual camera (v4l2loopback) support:

  • Audio (desktop/app/src/linux/audio.rs, voice.rs):
    • PipeWire / PulseAudio (pw-cat / parec / pacat) PCM audio capture and playback workers with automatic backend selection and fallback.
    • Voice input streaming and hotkey recording integration.
  • Media Sessions & SMTC (desktop/app/src/linux/media_sessions.rs, smtc.rs):
    • In-process D-Bus (zbus) MPRIS (org.mpris.MediaPlayer2.Player) discovery, metadata polling, and transport controls (Play, Pause, Next, Previous, Seek), with playerctl CLI fallback so the 2s polling loop does not fork subprocesses.
    • Outbound MPRIS target (org.mpris.MediaPlayer2.nectarlink) exported over zbus on /org/mpris/MediaPlayer2 with PropertiesChanged emission alongside mpris-state.json.
  • Video, Screen Capture, MP4 & Virtual Camera (desktop/app/src/linux/video.rs, screen_capture.rs, mp4_muxer.rs, vcam.rs):
    • ffmpeg / gstreamer H.264 decoder/encoder and MP4 muxer backends.
    • PipeWire / X11 / Wayland screen capture stream integration.
    • v4l2loopback virtual camera frame output (desktop/app/src/linux/vcam.rs and desktop/vcam/).

Verification

  • cargo fmt --all --check
  • cargo xtask strings --check
  • cargo clippy --locked -p nectarlink-desktop -p nectarlink-vcam -p s1-qt-rust --all-targets -- -D warnings
  • cargo test --locked -p nectarlink-desktop -p nectarlink-vcam (302 + 6 unit tests pass)

…uts, network & BLE (Part 2)

Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
…-monitor parser

Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
…ode for watched PIDs

Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
…rdings (Part 4)

Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
@thtbee

thtbee commented Oct 23, 2026

Copy link
Copy Markdown
Owner

Thanks for opening this. Our notes from testing are all in one place on #38 (comment), with a section for this part; I checked them against 355c08b again just now. The ones I'd look at first:

  • pc_audio_stream.rs: the pw-record fallback has no target, so it records the default source, which is the microphone. It needs -P stream.capture.sink=true (and the sink as --target).
  • recordings.rs: convert_audio passes -- before the output file, which ffmpeg rejects, so conversion always fails.
  • linux/h264.rs: no libavcodec.so.62 (FFmpeg 8, Ubuntu 26.04).

Happy to look again whenever you push.

jochenkirstaetter and others added 5 commits October 23, 2026 14:39
… D-Bus, and packaging

Applies the Part 2 review improvements and fixes from upstream/linux-review-fixes
(b346bfe, 47e3e06, 6291b51, a815275, 0be2ddf, 20c70f3, ccb9318):
- USB ADB: retry another local TCP port only when adb reports a port bind failure (47e3e06)
- Linux platform layer: native D-Bus/Netlink/Qt integrations for clipboard, tray,
  notifications (6291b51), single-instance activation token, BlueZ/NetworkManager,
  Wake-on-LAN, BLE beacon, and clean SIGTERM/SIGINT/SIGHUP shutdown (ccb9318)
- Shell & Doctor: remove Windows settings links/explorer.exe shims (20c70f3) and
  provide plain Linux Doctor readiness checks and wording with German and Chinese
  translations (a815275)
- Release workflow & packaging: pin AppImage tools with SHA-256 verification and
  smoke-test every Linux package (.deb, AppImage, tarball) with --version (0be2ddf)

Many thanks to @thtbee for the thorough review and reference implementation on
linux-review-fixes!

Co-authored-by: Bee <302257494+thtbee@users.noreply.github.com>
Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
…and window capture

Merges updated linux-2-basics and applies the Part 3 review improvements from
upstream/linux-review-fixes (b346bfe, 7bc9b2b, a815275):
- Keyboard layout resolution (7bc9b2b): add desktop/app/src/linux/keymap.rs using
  libxkbcommon / libxkbcommon-x11 to resolve keysyms and keycodes against the
  user's active X11 or Wayland keyboard layout for text typing and shortcuts
- Input injection & window capture (b346bfe): native XTest/uinput/RemoteDesktop
  portal input handling and X11/Portal window capture improvements
- DeckPage QML & i18n (a815275): use Linux shortcut labels (Super+D, Screenshot,
  Super) with German and Chinese translations

Many thanks to @thtbee for the collaborative review and reference implementation
on linux-review-fixes!

Co-authored-by: Bee <302257494+thtbee@users.noreply.github.com>
Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
… portal, and OpenH264 ABI

Applies the Part 4 review improvements and fixes from upstream/linux-review-fixes (b346bfe, c41f185, 20c70f3):
- Wayland ScreenCast portal: open the portal's PipeWire remote connection via OpenPipeWireRemote and pass it as fd 3 to gst-launch-1.0 pipewiresrc, with session restore_token persistence (c41f185)
- Media & audio: PipeWire/PulseAudio speaker monitor capture, MPRIS D-Bus client & server, FFmpeg 8 libavcodec compatibility, v4l2loopback virtual camera, and virtual microphone
- OpenH264 ABI compatibility: auto-detect 40-byte (Debian/Ubuntu libopenh264.so.7 2.4.1) vs 56-byte (upstream 2.3+ r_psnr) SLayerBSInfo layout at runtime and install libopenh264-dev in Linux CI

Many thanks to @thtbee for the thorough review and reference implementation on linux-review-fixes!

Co-authored-by: Bee <302257494+thtbee@users.noreply.github.com>
Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>
@jochenkirstaetter

Copy link
Copy Markdown
Contributor Author

Updated with linux-3-input and integrated @thtbee's Part 4 review fixes from linux-review-fixes (cf7ab65, c9f5b10 — PipeWire ScreenCast portal pod builder, 64-bit SPA_TYPE_Fd, vcam/vmic/mp4/camera hardening), plus libopenh264-dev in CI and runtime OpenH264 v2.0/v2.3 SLayerBSInfo ABI detection so pc_mirror unit tests pass across Debian/Ubuntu libopenh264-7 and upstream libopenh264.so.8. Huge thanks @thtbee for the collaborative effort!

Signed-off-by: Jochen Kirstätter <7329802+jochenkirstaetter@users.noreply.github.com>

@thtbee thtbee left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on the final branch (0b37495) along with the rest of the series: all findings addressed, and Linux and Windows checks, the FUSE end-to-end run and the package builds pass. Thank you.

@thtbee thtbee left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving after #38 and #41 merged.

@thtbee
thtbee merged commit 2693935 into thtbee:main Oct 23, 2026
9 checks passed
@jochenkirstaetter
jochenkirstaetter deleted the linux-4-media branch October 24, 2026 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants