Screen Streaming

Rein streams the desktop screen to mobile clients using GStreamer for screen capture and video encoding, and WebRTC (werift) for real-time delivery. GStreamer captures the desktop, converts the framerate to 60 FPS, encodes the frames into H.264 video, and transmits RTP packets over a local UDP socket (127.0.0.1:5004). Rein's WebRTCManager receives these UDP packets and writes them directly into WebRTC video tracks for all connected clients.

The HTTP server handles WebSocket signaling (/ws) and host engine controls (/api/host/*). Once negotiation is complete, encoded video is relayed with minimal latency.

---

Architecture

---

GstManager (src/server/gstreamer/gstManager.ts)

GstManager manages the lifecycle of a single gst-launch-1.0 process for the host system. Screen capture is encoded once into an H.264 RTP stream and broadcast to all active WebRTC viewer sessions via WebRTCManager.

Its responsibilities include:

• Initializing the platform-specific CaptureProvider.
• Building the complete GStreamer pipeline.
• Resolving GStreamer binary paths and environment variables (GST_PLUGIN_PATH, DISPLAY, XAUTHORITY).
• Launching and monitoring the gst-launch-1.0 process.
• Reporting pipeline stdout/stderr events to the logger.
• Terminating the process gracefully (SIGTERM followed by SIGKILL timeout) when stopped.

The overall startup flow is:

Because the capture backend is selected by CaptureProvider, the rest of the processing and encoding pipeline remains identical across all supported operating systems.

---

Pipeline Construction

Every platform contributes its own capture source through CaptureProvider. After that, GstManager appends a common high-performance processing and encoding pipeline:

Processing & Encoding Parameters

Stage	Purpose & Settings
queue	max-size-buffers=1, leaky=downstream — Drops stale frames immediately if processing lags, ensuring zero buffer accumulation.
videoconvert	Converts frames into the color format expected by videorate and x264enc.
videorate	Standardizes output framerate to 60 FPS (video/x-raw,framerate=60/1).
x264enc	tune=zerolatency, speed-preset=ultrafast, key-int-max=30, byte-stream=false — Ultra-low-latency H.264 software encoder with keyframes every 30 frames (0.5s at 60 FPS).
h264parse	Formats H.264 stream into video/x-h264,profile=baseline for maximum WebRTC browser compatibility.
rtph264pay	config-interval=-1, pt=96 — Wraps H.264 NAL units into RTP packets (Payload Type 96).
udpsink	host=127.0.0.1, port=5004, sync=false, async=false — Transmits RTP packets over local UDP socket without A/V sync delay.

---

CaptureProvider (src/server/gstreamer/captureProvider.ts)

CaptureProvider abstracts platform-specific screen capture behind a unified interface:

export interface CaptureProvider {
  initialize(onFailure?: (err: Error) => void): Promise<void>
  getGStreamerSource(): Promise<string[]>
  dispose(): Promise<void>
}

During startup, createCaptureProvider() detects the operating system using os.platform() and selects the appropriate capture backend:

Platform	Provider Implementation	GStreamer Source Elements
Windows	WindowsCaptureProvider	d3d11screencapturesrc do-timestamp=true ! queue max-size-buffers=5 leaky=downstream ! d3d11convert ! d3d11download
Linux (X11)	LinuxX11CaptureProvider	ximagesrc display-name=:0 use-damage=false show-pointer=true
Linux (Wayland)	LinuxWaylandPortalCaptureProvider	pipewiresrc path=<nodeId> do-timestamp=true (via D-Bus portal)
macOS	MacOSCaptureProvider	avfvideosrc capture-screen=true capture-screen-cursor=true

---

Platform Implementations

Windows — WindowsCaptureProvider

Windows uses DirectX 11 Desktop Duplication (DXGI) through GStreamer's d3d11screencapturesrc.

Using Direct3D keeps the capture process GPU-accelerated, minimizing CPU overhead during screen capture.

---

Linux (X11) — LinuxX11CaptureProvider

On traditional X11 desktops, Rein captures the screen using ximagesrc.

ximagesrc (display-name=:0, show-pointer=true)
      │
      ▼
GStreamer Pipeline

This backend captures the X11 root window directly and includes the desktop mouse cursor in the video stream.

---

Linux (Wayland) — LinuxWaylandPortalCaptureProvider

Wayland requires applications to request screen capture permission via the XDG Desktop Portal (org.freedesktop.portal.ScreenCast).

The capture workflow is:

Internally, ImplementDbus (src/server/gstreamer/utils.ts) interacts with D-Bus services using dbus-next:

1. Connects to the session D-Bus bus.
2. Calls org.freedesktop.portal.ScreenCast.CreateSession.
3. Calls SelectSources and Start.
4. Retrieves the PipeWire node ID and passes it to pipewiresrc.
5. Listens for session closed signals and triggers onFailure callback if the portal session terminates.

---

macOS — MacOSCaptureProvider

On macOS, Rein uses AVFoundation through GStreamer's avfvideosrc.

avfvideosrc (capture-screen=true, capture-screen-cursor=true)
      │
      ▼
GStreamer Pipeline

Screen recording requires Screen Recording permission under System Settings → Privacy & Security → Screen Recording.

---

WebRTC Media Relay (WebRTCManager)

Once RTP packets arrive at 127.0.0.1:5004, WebRTCManager (src/server/webRTC.ts) distributes them to connected viewers:

const socket = dgram.createSocket("udp4")
socket.on("message", (msg) => {
  for (const client of this.clients.values()) {
    try {
      client.videoTrack.writeRtp(msg)
      client.bytesSent += msg.length
    } catch {}
  }
})
socket.bind(5004, "127.0.0.1")

This design ensures:

• Screen capture and encoding occur only once regardless of the number of connected viewers.
• Zero CPU/GPU duplication for multiple viewers.
• RTP packets are forwarded directly into werift MediaStreamTrack objects in memory.

---

Video Parameters Summary

Parameter	macOS	Linux	Windows
Codec	H.264 (x264enc)	H.264 (x264enc)	H.264 (x264enc)
Profile	Baseline	Baseline	Baseline
Tuning	zerolatency	zerolatency	zerolatency
Speed Preset	ultrafast	ultrafast	ultrafast
Bitrate / Speed	Variable ultrafast	Variable ultrafast	Variable ultrafast
Framerate	60 FPS	60 FPS	60 FPS
Keyframe Interval	Every 30 frames (0.5s)	Every 30 frames (0.5s)	Every 30 frames (0.5s)
Transport	UDP 127.0.0.1:5004	UDP 127.0.0.1:5004	UDP 127.0.0.1:5004

---

Privacy Guarantee

Rein's streaming architecture ensures desktop video stays local to your machine and network. Encoded RTP packets are sent only to local UDP port 5004 and relayed directly over encrypted WebRTC DTLS-SRTP connections to authenticated clients on your LAN. No media is ever sent to external cloud servers or third-party relays.