diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index b57f0eb..afeadba 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -1,5 +1,6 @@ -# Tests every push and pull request, builds MoonlightWebRTC.wgt, and publishes it as a GitHub -# release when a version tag (v2.0.0, v2.1.0-beta.1, ...) is pushed. +# Tests every push and pull request, builds MoonlightWebRTC.wgt (Game Mode on) and +# MoonlightWebRTC-NoGameMode.wgt, and publishes both as a GitHub release when a version tag +# (v2.0.0, v2.1.0-beta.1, ...) is pushed. name: Build and release on: @@ -103,12 +104,15 @@ jobs: throw "Unable to extract the Emscripten SDK." } - - name: Build MoonlightWebRTC.wgt + - name: Build the WGT packages shell: pwsh run: | - ./packaging/tizen/build-package.ps1 ` - -TizenCli (Join-Path $env:TIZEN_STUDIO 'tools\ide\bin\tizen.bat') ` - -EmsdkRoot (Join-Path $env:EMSDK_DIRECTORY 'emscripten-release-bundle\emsdk') + $buildArguments = @{ + TizenCli = Join-Path $env:TIZEN_STUDIO 'tools\ide\bin\tizen.bat' + EmsdkRoot = Join-Path $env:EMSDK_DIRECTORY 'emscripten-release-bundle\emsdk' + } + ./packaging/tizen/build-package.ps1 @buildArguments + ./packaging/tizen/build-package.ps1 @buildArguments -NoGameMode ./packaging/release/generate-checksums.ps1 - uses: actions/upload-artifact@v4 @@ -116,6 +120,7 @@ jobs: name: MoonlightWebRTC path: | dist/tizen/MoonlightWebRTC.wgt + dist/tizen/MoonlightWebRTC-NoGameMode.wgt dist/SHA256SUMS.txt if-no-files-found: error @@ -141,6 +146,7 @@ jobs: if [[ "$TAG" == *-* ]]; then prerelease=(--prerelease); fi gh release create "$TAG" \ release/tizen/MoonlightWebRTC.wgt \ + release/tizen/MoonlightWebRTC-NoGameMode.wgt \ release/SHA256SUMS.txt \ --repo "$GITHUB_REPOSITORY" \ --title "Moonlight WebRTC $TAG" \ diff --git a/README.md b/README.md index 189eb0d..f4e8246 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,15 @@ Sunshine-Web-RTC: capture → encode → WebRTC RTP → TV PC, so the encoder gets the bitrate the TV asked for, less only audio and RTP overhead. WebRTC on the LAN repairs losses with retransmission and keyframe requests instead. - **Faster recovery.** When the TV asks for a keyframe, the request goes directly to Sunshine's encoder. +- **No waiting for the audio or the jitter buffer.** Sunshine marks every video packet with a zero + playout delay, so the TV shows each frame the moment it is decoded instead of buffering it, as Stadia + did; the TV also asks for the shortest jitter buffer and does not hold the picture back to stay in + sync with the audio, as Moonlight does not either. Screen to screen, 1080p60 dropped from 54 to 38 ms + in Sunshine-Web-RTC's latency test. +- **Latency you can see.** The stream overlay and the top of the statistics panel show the latency from + the PC to the screen, drawing included, its session average, and the share of packets lost. The + *Performance overlay* setting, also in the stream menu, keeps them fixed at the top right of the stream, + with the frames per second actually arriving at the top left. - **Nothing extra to install or keep running.** There is no Windows service, tray application or second pairing. The TV pairs with Sunshine itself. @@ -306,6 +315,14 @@ added by hand needs it entered on the TV. Download `MoonlightWebRTC.wgt` from the [Releases page](https://github.com/mlopezsegura/moonlight-webrtc-tizen/releases) and install it with Apps2Samsung, as described in [Installing on the TV](#installing-on-the-tv). +Each release has two packages: + +- `MoonlightWebRTC.wgt` asks the TV to switch to **Game Mode** while the app is open, for the lowest input lag. +- `MoonlightWebRTC-NoGameMode.wgt` leaves the TV's picture mode alone. Use it if Game Mode causes problems on + your TV or you prefer to choose the picture mode yourself. + +Both install as the same app, so installing one replaces the other. + ## 3 — Add your PC and pair 1. Open Moonlight WebRTC on the TV. The TV looks for PCs running Sunshine-Web-RTC on the local network and @@ -439,7 +456,9 @@ The TV package needs [Samsung's Emscripten SDK](https://developer.samsung.com/sm GitHub Actions (`.github/workflows/build.yml`) runs the tests and builds `MoonlightWebRTC.wgt` on every push and pull request. To publish a release, set the version in `tizen/config.xml` and `tizen/update-check.js`, then push a matching tag, such as `v2.0.0` or `v2.0.0-beta.1` (published as a pre-release). The workflow -attaches the package and `SHA256SUMS.txt` to the release. +attaches both packages (`MoonlightWebRTC.wgt` and `MoonlightWebRTC-NoGameMode.wgt`) and `SHA256SUMS.txt` to +the release. To build the package without Game Mode locally, pass `-NoGameMode` to +`packaging/tizen/build-package.ps1`. --- diff --git a/docs/protocol.md b/docs/protocol.md index bb08860..75f1d39 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -299,6 +299,33 @@ contain that ID, so late signaling from a stopped session can be ignored. When the TV requests a keyframe (RTCP PLI), Sunshine's encoder produces one directly. +The TV application can also ask for one itself, on the `control` data channel: + +```json +{"v":1,"type":"request-keyframe","reason":"not-decoding"} +``` + +It does so when its picture stops advancing for a second while video keeps arriving, a stall the TV's +WebRTC stack does not always answer with a PLI of its own. `reason` is `not-decoding`, `not-presenting` +(frames decode but never reach the screen) or `reattached`, and is informative only. If the picture is +still stopped after 2.5 seconds the application also re-attaches the stream to its video element, which +rebuilds the renderer without touching the connection. Sunshine logs keyframe requests and IDR frames sent +at most once a second per session. + +## TV log + +Once authenticated, the TV forwards its own log to Sunshine about once a second: + +```json +{"version":2,"type":"client-log","lines":["21:04:31.512 Video stalled for 1000 ms ..."]} +``` + +Sunshine writes each line into `sunshine.log` as `WebRTC TV : `, so what the TV saw and what the host +did sit in one file. A message carries at most 100 lines; control characters are replaced and lines longer than +4000 characters are cut. Besides its normal log the TV sends a stream diagnostics line every five seconds and, +while the picture is stopped, one line per stall-watchdog sample with its decoder and network counters. A +Sunshine that does not know the message answers `unsupported-message`, and the TV then stops sending it. + ## Stop a session ```json diff --git a/packaging/release/generate-checksums.ps1 b/packaging/release/generate-checksums.ps1 index c5e0362..6675793 100644 --- a/packaging/release/generate-checksums.ps1 +++ b/packaging/release/generate-checksums.ps1 @@ -15,6 +15,10 @@ $assets = @( [pscustomobject]@{ Name = 'MoonlightWebRTC.wgt' Path = Join-Path $DistributionDirectory 'tizen\MoonlightWebRTC.wgt' + }, + [pscustomobject]@{ + Name = 'MoonlightWebRTC-NoGameMode.wgt' + Path = Join-Path $DistributionDirectory 'tizen\MoonlightWebRTC-NoGameMode.wgt' } ) diff --git a/packaging/tizen/build-package.ps1 b/packaging/tizen/build-package.ps1 index d060ff8..6a7787f 100644 --- a/packaging/tizen/build-package.ps1 +++ b/packaging/tizen/build-package.ps1 @@ -3,7 +3,9 @@ param( [string]$TizenCli = 'C:\tizen-studio\tools\ide\bin\tizen.bat', [string]$SigningProfile, # Samsung's Emscripten SDK, required for the Wake-on-LAN module; see build-wake-on-lan.ps1. - [string]$EmsdkRoot + [string]$EmsdkRoot, + # Leaves out Samsung's Game Mode metadata, for TVs or setups where Game Mode gets in the way. + [switch]$NoGameMode ) $ErrorActionPreference = 'Stop' @@ -14,7 +16,8 @@ $outputDirectory = Join-Path $repositoryRoot 'dist\tizen' $sourceDirectory = Join-Path $outputDirectory '.source' $buildDirectory = Join-Path $outputDirectory '.build' $packageDirectory = Join-Path $outputDirectory '.package' -$artifact = Join-Path $outputDirectory 'MoonlightWebRTC.wgt' +$artifactName = if ($NoGameMode) { 'MoonlightWebRTC-NoGameMode.wgt' } else { 'MoonlightWebRTC.wgt' } +$artifact = Join-Path $outputDirectory $artifactName if (-not (Test-Path -LiteralPath $TizenCli -PathType Leaf)) { throw "Tizen CLI was not found: $TizenCli" @@ -48,7 +51,10 @@ try { 'gateway-ipv4.js', 'gateway-store.js', 'index.html', + 'low-latency.js', 'preferences.js', + 'stall-watchdog.js', + 'stream-latency.js', 'tizen_web_project.yaml', 'ui.css', 'ui.js', @@ -63,6 +69,16 @@ try { } Copy-Item -LiteralPath $runtimePath -Destination $sourceDirectory } + if ($NoGameMode) { + $configPath = Join-Path $sourceDirectory 'config.xml' + $config = Get-Content -LiteralPath $configPath -Raw + $gameModePattern = '(?m)^[ \t]*]*/>\r?\n' + if ($config -notmatch $gameModePattern) { + throw "Game Mode metadata was not found in $configPath." + } + $config = $config -replace $gameModePattern, '' + [System.IO.File]::WriteAllText($configPath, $config, (New-Object System.Text.UTF8Encoding $false)) + } Copy-Item -LiteralPath (Join-Path $projectDirectory 'assets') -Destination $sourceDirectory -Recurse & (Join-Path $PSScriptRoot 'build-wake-on-lan.ps1') -OutputDirectory (Join-Path $sourceDirectory 'wasm') -EmsdkRoot $EmsdkRoot diff --git a/tests/TizenLowLatencyTest.js b/tests/TizenLowLatencyTest.js new file mode 100644 index 0000000..b1613cd --- /dev/null +++ b/tests/TizenLowLatencyTest.js @@ -0,0 +1,51 @@ +"use strict"; + +const assert = require("assert"); + +global.window = global; +require("../tizen/low-latency.js"); +const LowLatency = global.LowLatency; + +const offer = [ + "v=0", + "m=video 9 UDP/TLS/RTP/SAVPF 96", + "a=msid:moonlight video", + "a=ssrc:1 cname:sunshine", + "a=ssrc:1 msid:moonlight video", + "m=audio 9 UDP/TLS/RTP/SAVPF 111", + "a=msid:moonlight audio", + "a=ssrc:2 cname:sunshine", + "a=ssrc:2 msid:moonlight audio", + "m=application 9 UDP/DTLS/SCTP webrtc-datachannel", + "", +].join("\r\n"); + +const separated = LowLatency.separateAudioSync(offer); +assert.ok(separated.indexOf("a=msid:moonlight-audio audio\r\n") >= 0, "the audio section moves to a stream of its own"); +assert.ok(separated.indexOf("a=ssrc:2 msid:moonlight-audio audio\r\n") >= 0, "and so does its SSRC description"); +assert.ok(separated.indexOf("a=msid:moonlight video\r\n") >= 0, "the video keeps its stream"); +assert.ok(separated.indexOf("a=ssrc:1 msid:moonlight video\r\n") >= 0); +assert.strictEqual(separated.split("\r\n").length, offer.split("\r\n").length, "no line is added or lost"); +assert.strictEqual(LowLatency.separateAudioSync(separated), separated, "a separated offer stays as it is"); +assert.strictEqual(LowLatency.separateAudioSync(offer.replace(/\r\n/g, "\n")), + separated.replace(/\r\n/g, "\n"), "an offer with bare newlines keeps them"); +assert.strictEqual(LowLatency.separateAudioSync("m=audio 9 RTP 111\r\na=msid:- audio"), + "m=audio 9 RTP 111\r\na=msid:- audio", "a track without a stream has nothing to sync"); +assert.strictEqual(LowLatency.separateAudioSync(undefined), undefined); + +const modern = { jitterBufferTarget: null, playoutDelayHint: 1 }; +assert.strictEqual(LowLatency.minimizeReceiverDelay(modern), "jitterBufferTarget", "newer Chromium takes the target"); +assert.strictEqual(modern.jitterBufferTarget, 0); +assert.strictEqual(modern.playoutDelayHint, 1, "and the older hint is left alone"); + +const older = { playoutDelayHint: undefined }; +assert.strictEqual(LowLatency.minimizeReceiverDelay(older), "playoutDelayHint", "older Chromium takes the hint"); +assert.strictEqual(older.playoutDelayHint, 0); + +assert.strictEqual(LowLatency.minimizeReceiverDelay({}), null, "a TV with neither reports it"); +assert.strictEqual(LowLatency.minimizeReceiverDelay(null), null); +const refusing = {}; +Object.defineProperty(refusing, "jitterBufferTarget", { get: function () { return null; }, set: function () { throw new RangeError("no"); } }); +assert.strictEqual(LowLatency.minimizeReceiverDelay(refusing), null, "a receiver that refuses the value does not stop the stream"); + +console.log("Tizen low latency tests passed"); diff --git a/tests/TizenStallWatchdogTest.js b/tests/TizenStallWatchdogTest.js new file mode 100644 index 0000000..d78965c --- /dev/null +++ b/tests/TizenStallWatchdogTest.js @@ -0,0 +1,112 @@ +"use strict"; + +const assert = require("assert"); + +global.window = global; +require("../tizen/stall-watchdog.js"); +const StallWatchdog = global.StallWatchdog; + +// Feeds one sample every 500 ms, as the app does, and returns the actions taken. +function run(watchdog, samples) { + return samples.map(function (sample, index) { + return watchdog.sample(Object.assign({ time: index * 500 }, sample)).action; + }); +} + +function healthy(index) { + return { bytesReceived: index * 1000, framesDecoded: index * 30, framesPresented: index * 30 }; +} + +// A healthy stream is never touched. +{ + const watchdog = StallWatchdog.create(); + const actions = run(watchdog, [0, 1, 2, 3, 4, 5, 6, 7].map(healthy)); + assert.deepStrictEqual(actions, ["none", "none", "none", "none", "none", "none", "none", "none"]); + assert.strictEqual(watchdog.counters().stalls, 0); +} + +// A decoder that stops while video arrives gets a keyframe after a second, again every +// second, and the renderer re-attached once the stall reaches 2.5 s. +{ + const watchdog = StallWatchdog.create(); + const samples = [healthy(0), healthy(1)]; + for (let index = 2; index < 10; index += 1) { + samples.push({ bytesReceived: index * 1000, framesDecoded: 30, framesPresented: 30 }); + } + const actions = run(watchdog, samples); + // The stall starts at the last healthy sample (500 ms): 1 s at 1500 ms, 2.5 s at 3000 ms. + assert.deepStrictEqual(actions, + ["none", "none", "none", "keyframe", "none", "keyframe", "reattach", "none", "keyframe", "none"]); + const counters = watchdog.counters(); + assert.strictEqual(counters.stalls, 1, "one stall, however many actions it took"); + assert.strictEqual(counters.keyframeRequests, 3); + assert.strictEqual(counters.reattachments, 1); + assert.strictEqual(counters.stalled, true); + assert.strictEqual(watchdog.sample({ time: 5000, bytesReceived: 20000, framesDecoded: 60, framesPresented: 60 }).action, "none"); + assert.strictEqual(watchdog.counters().stalled, false, "the picture moving again ends the stall"); + assert.strictEqual(watchdog.counters().lastStallMs, 4500); +} + +// Frames that decode but never reach the screen are a stall too, reported as decoding. +{ + const watchdog = StallWatchdog.create(); + const samples = [healthy(0), healthy(1)]; + for (let index = 2; index < 5; index += 1) { + samples.push({ bytesReceived: index * 1000, framesDecoded: index * 30, framesPresented: 30 }); + } + let verdict = null; + samples.forEach(function (sample, index) { + const result = watchdog.sample(Object.assign({ time: index * 500 }, sample)); + if (result.action !== "none") { + verdict = result; + } + }); + assert.strictEqual(verdict.action, "keyframe"); + assert.strictEqual(verdict.decoding, true); +} + +// Nothing arriving is not the TV's stall to recover. +{ + const watchdog = StallWatchdog.create(); + const samples = [healthy(0), healthy(1)]; + for (let index = 2; index < 10; index += 1) { + samples.push({ bytesReceived: 1000, framesDecoded: 30, framesPresented: 30 }); + } + assert.ok(run(watchdog, samples).every(function (action) { return action === "none"; })); + assert.strictEqual(watchdog.counters().stalls, 0); +} + +// A counter the TV never advances is ignored rather than read as a permanent stall. +{ + const watchdog = StallWatchdog.create(); + const samples = []; + for (let index = 0; index < 10; index += 1) { + samples.push({ bytesReceived: index * 1000, framesDecoded: index * 30, framesPresented: 0 }); + } + assert.ok(run(watchdog, samples).every(function (action) { return action === "none"; }), + "a presented-frame counter stuck from the start is never trusted"); + const missing = StallWatchdog.create(); + const unreported = []; + for (let index = 0; index < 10; index += 1) { + unreported.push({ bytesReceived: index * 1000 }); + } + assert.ok(run(missing, unreported).every(function (action) { return action === "none"; }), + "a TV reporting no frame counters at all is never acted on"); +} + +// reset() forgets a previous session's stall and counters. +{ + const watchdog = StallWatchdog.create(); + const samples = [healthy(0), healthy(1)]; + for (let index = 2; index < 6; index += 1) { + samples.push({ bytesReceived: index * 1000, framesDecoded: 30, framesPresented: 30 }); + } + run(watchdog, samples); + assert.ok(watchdog.counters().stalls > 0); + watchdog.reset(); + assert.deepStrictEqual(watchdog.counters(), + { stalls: 0, keyframeRequests: 0, reattachments: 0, lastStallMs: 0, stalled: false }); + assert.strictEqual(watchdog.sample({ time: 0, bytesReceived: 0, framesDecoded: 0 }).action, "none"); +} + +console.log("TizenStallWatchdogTest passed"); diff --git a/tests/TizenStreamLatencyTest.js b/tests/TizenStreamLatencyTest.js new file mode 100644 index 0000000..a445e4b --- /dev/null +++ b/tests/TizenStreamLatencyTest.js @@ -0,0 +1,98 @@ +"use strict"; + +const assert = require("assert"); + +global.window = global; +require("../tizen/stream-latency.js"); +const StreamLatency = global.StreamLatency; + +function close(actual, expected, message) { + assert.ok(actual !== null && Math.abs(actual - expected) < 1e-6, + (message || "value") + ": expected " + expected + ", got " + actual); +} + +const latency = StreamLatency.create(); + +latency.recordPresentedFrame({ receiveTime: 1000, expectedDisplayTime: 1020, presentationTime: 1015 }); +latency.recordPresentedFrame({ receiveTime: 2000, expectedDisplayTime: 2030 }); +latency.recordPresentedFrame({ receiveTime: 0, expectedDisplayTime: 3000 }); +latency.recordPresentedFrame({ receiveTime: 4000, expectedDisplayTime: 3990 }); +latency.recordPresentedFrame({ receiveTime: 5000, expectedDisplayTime: 9000 }); +latency.recordPresentedFrame(null); + +let summary = latency.sample({ + packetsReceived: 990, + packetsLost: 10, + framesReceived: 100, + framesDropped: 2, + framesDecoded: 98, + totalDecodeTime: 0.49, + jitterBufferDelay: 1.0, + jitterBufferEmittedCount: 100, + totalProcessingDelay: 1.5, + timestamp: 10000, +}, 0.004); + +close(summary.roundTripMs, 4, "round trip"); +close(summary.networkMs, 2, "one way is half the round trip"); +close(summary.jitterBufferMs, 10, "jitter buffer per frame"); +close(summary.decodeMs, 5, "decode per frame"); +close(summary.processingMs, 15, "arrival to decoded per frame"); +close(summary.displayMs, 25, "arrival to screen ignores frames without a receive time and implausible ones"); +close(summary.displayMaxMs, 30, "worst frame to screen"); +assert.strictEqual(summary.includesDisplay, true); +close(summary.totalMs, 27, "total is network plus to screen"); +close(summary.averageTotalMs, 27); +close(summary.packetLossPercent, 1, "lost packets as a share of all sent"); +close(summary.sessionPacketLossPercent, 1); +close(summary.framesDroppedPercent, 2); +assert.strictEqual(summary.receivedFps, null, "the first sample has no interval to measure the frame rate over"); + +summary = latency.sample({ + packetsReceived: 1990, + packetsLost: 10, + framesReceived: 200, + framesDropped: 2, + framesDecoded: 198, + totalDecodeTime: 0.99, + jitterBufferDelay: 1.5, + jitterBufferEmittedCount: 200, + totalProcessingDelay: 2.5, + timestamp: 12000, +}, 0.004); +close(summary.receivedFps, 50, "a hundred frames in two seconds"); +close(summary.packetLossPercent, 0, "the window only counts its own losses"); +close(summary.sessionPacketLossPercent, 0.5, "the session counts all of them"); +close(summary.jitterBufferMs, 5); +assert.strictEqual(summary.displayMs, null, "no frame was presented in the window"); +assert.strictEqual(summary.includesDisplay, true, "but the TV does report presented frames"); +close(summary.totalMs, 12, "so the window falls back to arrival to decoded"); +close(summary.averageTotalMs, 19.5, "the average covers the whole session"); + +summary = latency.sample({ packetsReceived: 50, packetsLost: 0, framesReceived: 5, framesDecoded: 5 }, null); +close(summary.packetLossPercent, 0, "a counter that went backwards belongs to a new receiver"); +assert.strictEqual(summary.roundTripMs, null, "no round trip, no network estimate"); +assert.strictEqual(summary.networkMs, null); + +latency.reset(); +summary = latency.sample(null, null); +assert.strictEqual(summary.totalMs, null, "a reset session has nothing to report"); +assert.strictEqual(summary.averageTotalMs, null); +assert.strictEqual(summary.includesDisplay, false); +assert.strictEqual(summary.packetLossPercent, null); + +function reports(list) { + return { forEach: function (callback) { list.forEach(callback); } }; +} +assert.strictEqual(StreamLatency.roundTripSeconds(reports([ + { type: "transport", selectedCandidatePairId: "b" }, + { type: "candidate-pair", id: "a", nominated: true, state: "succeeded", currentRoundTripTime: 0.1 }, + { type: "candidate-pair", id: "b", nominated: true, state: "succeeded", currentRoundTripTime: 0.003 }, +])), 0.003, "the transport's selected pair is the one carrying the stream"); +assert.strictEqual(StreamLatency.roundTripSeconds(reports([ + { type: "candidate-pair", id: "a", nominated: false, state: "succeeded", currentRoundTripTime: 0.1 }, + { type: "candidate-pair", id: "b", nominated: true, state: "succeeded", currentRoundTripTime: 0.002 }, +])), 0.002, "without a transport report the nominated pair is"); +assert.strictEqual(StreamLatency.roundTripSeconds(reports([])), null); + +console.log("Tizen stream latency tests passed"); diff --git a/tests/TizenUiTest.js b/tests/TizenUiTest.js index d0a8fe7..d20a1ba 100644 --- a/tests/TizenUiTest.js +++ b/tests/TizenUiTest.js @@ -222,6 +222,7 @@ const persistentStorage = storage(); const preferences = global.ClientPreferences.create(persistentStorage); assert.deepStrictEqual(preferences.load(), { resolution: null, codec: null, hdr: false, bitrateKbps: null, frameRate: null, frameInterpolation: false, + performanceOverlay: false, }, "empty storage must use the current defaults"); preferences.update("resolution", "3840x2160"); preferences.update("codec", "hevc"); @@ -229,18 +230,20 @@ preferences.update("hdr", true); preferences.update("bitrateKbps", 50000); preferences.update("frameRate", 120); preferences.update("frameInterpolation", true); +preferences.update("performanceOverlay", true); assert.deepStrictEqual(global.ClientPreferences.create(persistentStorage).load(), { resolution: "3840x2160", codec: "hevc", hdr: true, bitrateKbps: 50000, - frameRate: 120, frameInterpolation: true, + frameRate: 120, frameInterpolation: true, performanceOverlay: true, }, "saved preferences must survive a simulated application reload"); assert.ok(appSource.includes("fps: Number(frameRateSelect.value) || 60,") && html.includes('id="fps-select"') && html.indexOf('src="frame-rate-support.js"') >= 0, "the frame rate comes from Settings: the TV's maximum when it announces one, otherwise the user's choice"); persistentStorage.setItem(global.ClientPreferences.STORAGE_KEY, JSON.stringify({ - resolution: 4, codec: false, hdr: "true", bitrateKbps: -1, frameRate: "120", frameInterpolation: "yes", + resolution: 4, codec: false, hdr: "true", bitrateKbps: -1, frameRate: "120", frameInterpolation: "yes", performanceOverlay: 1, })); assert.deepStrictEqual(global.ClientPreferences.create(persistentStorage).load(), { resolution: null, codec: null, hdr: false, bitrateKbps: null, frameRate: null, frameInterpolation: false, + performanceOverlay: false, }, "invalid persisted values must fall back safely"); assert.strictEqual(global.ClientPreferences.resolveSupported("av1", ["h264", "hevc"], "h264"), "h264", "unsupported persisted options must fall back to the current supported default"); diff --git a/tizen/app.js b/tizen/app.js index 162488a..67af093 100644 --- a/tizen/app.js +++ b/tizen/app.js @@ -56,6 +56,7 @@ const hdrSelect = document.getElementById("hdr-select"); const bitrateSelect = document.getElementById("bitrate-select"); const frameRateSelect = document.getElementById("fps-select"); const interpolationSelect = document.getElementById("interpolation-select"); +const performanceOverlaySelect = document.getElementById("performance-overlay-select"); const interpolationStatusElement = document.getElementById("interpolation-status"); const interpolationCanvas = document.getElementById("interpolation-canvas"); const settingsSelectorMenu = document.getElementById("settings-selector-menu"); @@ -95,6 +96,7 @@ const gatewayRemoveConfirmButton = document.getElementById("gateway-remove-confi const playButton = document.getElementById("play-button"); const continueButton = document.getElementById("continue-button"); const diagnosticsButton = document.getElementById("diagnostics-button"); +const performanceOverlayButton = document.getElementById("performance-overlay-button"); const stopButton = document.getElementById("stop-button"); const homeMessage = document.getElementById("home-message"); const gatewayStateElement = document.getElementById("gateway-state"); @@ -113,6 +115,24 @@ const overlay = { gamepad: document.getElementById("overlay-gamepad"), connection: document.getElementById("overlay-connection"), interpolation: document.getElementById("overlay-interpolation"), + recovery: document.getElementById("overlay-recovery"), + latency: document.getElementById("overlay-latency"), + packetLoss: document.getElementById("overlay-packet-loss"), +}; + +const performanceHud = { + fpsPanel: document.getElementById("performance-fps"), + fps: document.getElementById("performance-fps-value"), + fpsTarget: document.getElementById("performance-fps-target"), + latencyPanel: document.getElementById("performance-latency"), + total: document.getElementById("performance-latency-total"), + average: document.getElementById("performance-latency-average"), + network: document.getElementById("performance-latency-network"), + jitterBuffer: document.getElementById("performance-latency-jitter-buffer"), + decode: document.getElementById("performance-latency-decode"), + toScreen: document.getElementById("performance-latency-to-screen"), + packetLoss: document.getElementById("performance-packet-loss"), + framesDropped: document.getElementById("performance-frames-dropped"), }; const trackCounts = { @@ -132,6 +152,16 @@ const statistics = { receivedTransfer: document.getElementById("received-transfer"), receivedBitDepth: document.getElementById("received-bit-depth"), receivedHdr: document.getElementById("received-hdr"), + latency: { + total: document.getElementById("latency-total"), + average: document.getElementById("latency-average"), + network: document.getElementById("latency-network"), + jitterBuffer: document.getElementById("latency-jitter-buffer"), + decode: document.getElementById("latency-decode"), + toScreen: document.getElementById("latency-to-screen"), + packetLoss: document.getElementById("latency-packet-loss"), + framesDropped: document.getElementById("latency-frames-dropped"), + }, video: { framesReceived: document.getElementById("frames-received"), framesDecoded: document.getElementById("frames-decoded"), @@ -140,6 +170,11 @@ const statistics = { packetsReceived: document.getElementById("packets-received"), packetsLost: document.getElementById("packets-lost"), bytesReceived: document.getElementById("bytes-received"), + keyFramesDecoded: document.getElementById("key-frames-decoded"), + pliCount: document.getElementById("pli-count"), + nackCount: document.getElementById("nack-count"), + freezeCount: document.getElementById("freeze-count"), + stallRecovery: document.getElementById("stall-recovery"), }, audio: { packetsReceived: document.getElementById("audio-packets-received"), @@ -210,6 +245,15 @@ let lastStatisticsConsoleTime = 0; let lastStatisticsSampleTime = 0; // Half the console-log period, so that log still lands on its own five-second schedule. const HIDDEN_STATISTICS_INTERVAL_MS = 2500; +// Twice a second: a stall is declared after one second, so a slower sample would let the +// picture stay black for longer than the watchdog's own threshold. +const STALL_SAMPLE_INTERVAL_MS = 500; +const stallWatchdog = StallWatchdog.create(); +let stallSampleInProgress = false; +let presentedFrames = 0; +let presentedFrameCallbackPending = false; +const streamLatency = StreamLatency.create(); +let receiverDelayReported = false; let applications = []; let ui = null; let artworkLoader = null; @@ -240,8 +284,59 @@ const gatewayWatchers = new Map(); const GATEWAY_WATCH_GREETING_TIMEOUT_MS = 2500; const GATEWAY_WATCH_RETRY_MS = 5000; +// The TV's log, forwarded to the PC: Sunshine writes each line into its own log, next to +// what it was doing at the same moment, so a stall can be read from one file on the PC +// without a debugger attached to the TV. Lines wait here while the PC is unreachable; the +// oldest are dropped first, and how many is reported once the PC is back. +const REMOTE_LOG_FLUSH_MS = 1000; +const REMOTE_LOG_MAX_PENDING = 500; +const REMOTE_LOG_BATCH_LINES = 100; +let remoteLogLines = []; +let remoteLogDropped = 0; +let remoteLogSupported = true; + +function remoteLog(message) { + if (!remoteLogSupported) { + return; + } + remoteLogLines.push(new Date().toISOString().slice(11, 23) + " " + String(message)); + if (remoteLogLines.length > REMOTE_LOG_MAX_PENDING) { + remoteLogDropped += remoteLogLines.length - REMOTE_LOG_MAX_PENDING; + remoteLogLines = remoteLogLines.slice(remoteLogLines.length - REMOTE_LOG_MAX_PENDING); + } +} + +function flushRemoteLog() { + if (!remoteLogSupported || !gatewayAuthorized || !socket + || socket.readyState !== WebSocket.OPEN || remoteLogLines.length === 0) { + return; + } + if (remoteLogDropped > 0) { + remoteLogLines.unshift(String(remoteLogDropped) + " TV log line(s) dropped while the PC was unreachable"); + remoteLogDropped = 0; + } + while (remoteLogLines.length > 0) { + const lines = remoteLogLines.slice(0, REMOTE_LOG_BATCH_LINES); + try { + sendGatewayMessage({ type: "client-log", lines: lines }); + } catch (error) { + // Kept for the next attempt; reporting it through log() would only queue another line. + return; + } + remoteLogLines = remoteLogLines.slice(lines.length); + } +} + +// A Sunshine from before client-log answers it as an unsupported message. That is not an +// error the viewer needs to see, and it would end a launch in progress. +function isUnsupportedRemoteLog(message) { + return message.code === "unsupported-message" + && String(message.message || "").indexOf("client-log") !== -1; +} + function log(message) { console.log(message); + remoteLog(message); logElement.textContent += message + "\n"; const lines = logElement.textContent.split("\n"); if (lines.length > 80) { @@ -1066,6 +1161,7 @@ function currentPreferenceValues() { frameRate: Number.isInteger(Number(frameRateSelect.value)) && Number(frameRateSelect.value) > 0 ? Number(frameRateSelect.value) : null, frameInterpolation: interpolationSelect.value === "true", + performanceOverlay: performanceOverlaySelect.value === "true", }; } @@ -1337,6 +1433,12 @@ function setRunningApplication(appId) { } function handleGatewayError(message) { + if (isUnsupportedRemoteLog(message)) { + remoteLogSupported = false; + remoteLogLines = []; + console.log("This Sunshine does not accept the TV log; it stays on the TV only"); + return; + } if (message.requestType === "authenticate") { if (message.code === "client-disabled") { handleClientDisabled(message); @@ -1599,6 +1701,9 @@ function createPeerConnection(sessionId) { pendingRemoteCandidates = []; previousDecodedSample = null; statisticsErrorReported = false; + stallWatchdog.reset(); + streamLatency.reset(); + updateRecoveryStatus(); try { peerConnection = new RTCPeerConnection({ iceServers: [] }); @@ -1661,6 +1766,13 @@ function createPeerConnection(sessionId) { peerConnection.ontrack = function (event) { event.track.enabled = !document.hidden; + if (event.track.kind === "video") { + const applied = LowLatency.minimizeReceiverDelay(event.receiver); + if (!receiverDelayReported) { + receiverDelayReported = true; + log("Video jitter buffer: " + (applied ? applied + " set to 0" : "the TV allows no target")); + } + } const alreadyAdded = remoteStream.getTracks().some(function (track) { return track.id === event.track.id; }); @@ -1717,7 +1829,10 @@ async function handleOffer(message) { } try { - await peerConnection.setRemoteDescription({ type: "offer", sdp: message.sdp }); + await peerConnection.setRemoteDescription({ + type: "offer", + sdp: LowLatency.separateAudioSync(message.sdp), + }); while (pendingRemoteCandidates.length > 0) { await applyRemoteCandidate(pendingRemoteCandidates.shift()); } @@ -1917,6 +2032,25 @@ function handleMouseModeChanged(record, active) { ); } +function performanceOverlayEnabled() { + return performanceOverlaySelect.value === "true"; +} + +function applyPerformanceOverlay() { + const enabled = performanceOverlayEnabled(); + performanceHud.fpsPanel.hidden = !enabled; + performanceHud.latencyPanel.hidden = !enabled; + performanceOverlayButton.textContent = enabled + ? "Hide performance overlay" + : "Show performance overlay"; +} + +function togglePerformanceOverlay() { + performanceOverlaySelect.value = performanceOverlayEnabled() ? "false" : "true"; + persistCurrentPreferences(); + applyPerformanceOverlay(); +} + function toggleDiagnostics() { diagnosticsElement.hidden = !diagnosticsElement.hidden; diagnosticsButton.textContent = diagnosticsElement.hidden @@ -2968,6 +3102,7 @@ document.addEventListener("keydown", function (event) { playButton.addEventListener("click", startSelectedSession); continueButton.addEventListener("click", hideStreamMenu); diagnosticsButton.addEventListener("click", toggleDiagnostics); +performanceOverlayButton.addEventListener("click", togglePerformanceOverlay); stopButton.addEventListener("click", stopCurrentSession); resumeSessionButton.addEventListener("click", resumeRunningApplication); stopHostSessionButton.addEventListener("click", stopRunningApplication); @@ -3005,6 +3140,10 @@ hdrSelect.addEventListener("change", function () { persistCurrentPreferences(); }); bitrateSelect.addEventListener("change", persistCurrentPreferences); +performanceOverlaySelect.addEventListener("change", function () { + persistCurrentPreferences(); + applyPerformanceOverlay(); +}); interpolationSelect.addEventListener("change", function () { persistCurrentPreferences(); applyFrameInterpolation(); @@ -3461,7 +3600,7 @@ async function updateStatistics() { // hidden because it also emits the five-second stream diagnostics line, so it slows to // the slowest cadence that still keeps that line on schedule. const sampledAt = performance.now(); - if (!diagnosticsVisible() + if (!diagnosticsVisible() && !(performanceOverlayEnabled() && !streamingScreen.hidden) && sampledAt - lastStatisticsSampleTime < HIDDEN_STATISTICS_INTERVAL_MS) { return; } @@ -3481,6 +3620,8 @@ async function updateStatistics() { codecReports[report.id] = report; } }); + const latency = streamLatency.sample(inboundVideo, StreamLatency.roundTripSeconds(reports)); + displayLatency(latency); statistics.actualResolution.textContent = String(videoElement.videoWidth || 0) + " × " + String(videoElement.videoHeight || 0); @@ -3504,6 +3645,10 @@ async function updateStatistics() { statistics.video.packetsReceived.textContent = statisticValue(inboundVideo, "packetsReceived"); statistics.video.packetsLost.textContent = statisticValue(inboundVideo, "packetsLost"); statistics.video.bytesReceived.textContent = statisticValue(inboundVideo, "bytesReceived"); + statistics.video.keyFramesDecoded.textContent = optionalStatistic(inboundVideo, "keyFramesDecoded"); + statistics.video.pliCount.textContent = optionalStatistic(inboundVideo, "pliCount"); + statistics.video.nackCount.textContent = optionalStatistic(inboundVideo, "nackCount"); + statistics.video.freezeCount.textContent = optionalStatistic(inboundVideo, "freezeCount"); const codecReport = inboundVideo.codecId ? codecReports[inboundVideo.codecId] : null; const receivedCodec = codecReport && codecReport.mimeType ? String(codecReport.mimeType) @@ -3532,7 +3677,7 @@ async function updateStatistics() { if (sampleTime - lastStatisticsConsoleTime >= 5000) { lastStatisticsConsoleTime = sampleTime; - console.log("Stream diagnostics: " + JSON.stringify({ + const diagnosticsLine = "Stream diagnostics: " + JSON.stringify({ requestedWidth: selectedSession ? selectedSession.width : 0, requestedHeight: selectedSession ? selectedSession.height : 0, requestedCodec: selectedSession ? selectedSession.codec : "unknown", @@ -3552,12 +3697,16 @@ async function updateStatistics() { packetsReceived: statisticValue(inboundVideo, "packetsReceived"), packetsLost: statisticValue(inboundVideo, "packetsLost"), bytesReceived: statisticValue(inboundVideo, "bytesReceived"), + recovery: recoveryStatistics(inboundVideo), + latency: latency, audioPacketsReceived: inboundAudio ? statisticValue(inboundAudio, "packetsReceived") : 0, audioPacketsLost: inboundAudio ? statisticValue(inboundAudio, "packetsLost") : 0, audioJitter: inboundAudio ? statisticValue(inboundAudio, "jitter") : 0, gamepadConnected: activeGamepad() !== null, - })); + }); + console.log(diagnosticsLine); + remoteLog(diagnosticsLine); } } @@ -3577,6 +3726,222 @@ async function updateStatistics() { } } +function milliseconds(value) { + return value === null ? "-" : value.toFixed(1) + " ms"; +} + +function percent(value) { + return value === null ? "-" : value.toFixed(2) + " %"; +} + +// Shown in the overlay as well as the statistics panel: it is the one number a player +// checks before anything else. +function displayLatency(latency) { + const fields = statistics.latency; + fields.total.textContent = milliseconds(latency.totalMs) + + (latency.totalMs !== null && !latency.includesDisplay ? " (to decoder)" : ""); + fields.average.textContent = milliseconds(latency.averageTotalMs); + fields.network.textContent = latency.roundTripMs === null + ? "-" + : milliseconds(latency.networkMs) + " (RTT " + latency.roundTripMs.toFixed(1) + ")"; + fields.jitterBuffer.textContent = milliseconds(latency.jitterBufferMs); + fields.decode.textContent = milliseconds(latency.decodeMs); + fields.toScreen.textContent = latency.displayMs !== null + ? milliseconds(latency.displayMs) + " (max " + latency.displayMaxMs.toFixed(0) + ")" + : (latency.includesDisplay ? "-" : "unavailable"); + fields.packetLoss.textContent = percent(latency.packetLossPercent) + + (latency.sessionPacketLossPercent === null + ? "" : " (session " + percent(latency.sessionPacketLossPercent) + ")"); + fields.framesDropped.textContent = percent(latency.framesDroppedPercent); + overlay.latency.textContent = latency.averageTotalMs === null + ? "-" + : milliseconds(latency.totalMs) + " / avg " + latency.averageTotalMs.toFixed(0); + overlay.packetLoss.textContent = percent(latency.packetLossPercent); + + performanceHud.total.textContent = fields.total.textContent; + performanceHud.average.textContent = fields.average.textContent; + performanceHud.network.textContent = fields.network.textContent; + performanceHud.jitterBuffer.textContent = fields.jitterBuffer.textContent; + performanceHud.decode.textContent = fields.decode.textContent; + performanceHud.toScreen.textContent = fields.toScreen.textContent; + performanceHud.packetLoss.textContent = fields.packetLoss.textContent; + performanceHud.framesDropped.textContent = fields.framesDropped.textContent; + const targetFps = selectedSession ? selectedSession.fps : 0; + performanceHud.fps.textContent = latency.receivedFps === null + ? "-" : latency.receivedFps.toFixed(0); + performanceHud.fpsTarget.textContent = targetFps ? "of " + String(targetFps) : ""; + // Below nine tenths of what was asked for is a stream that visibly stutters. + performanceHud.fpsPanel.classList.toggle("is-low", latency.receivedFps !== null + && targetFps > 0 && latency.receivedFps < targetFps * 0.9); +} + +function optionalStatistic(report, name) { + return report && typeof report[name] === "number" ? String(report[name]) : "-"; +} + +// What the TV's own WebRTC stack did about a stall, next to what the watchdog did. A PLI +// count that does not move while the picture is black means the stack never asked the host +// for anything, and only the watchdog's request reached it. +function recoveryStatistics(inboundVideo) { + const counters = stallWatchdog.counters(); + return { + presentedFrames: presentedFrameCallbacksSupported() ? presentedFrames : null, + keyFramesDecoded: inboundVideo ? inboundVideo.keyFramesDecoded : undefined, + pliCount: inboundVideo ? inboundVideo.pliCount : undefined, + firCount: inboundVideo ? inboundVideo.firCount : undefined, + nackCount: inboundVideo ? inboundVideo.nackCount : undefined, + freezeCount: inboundVideo ? inboundVideo.freezeCount : undefined, + totalFreezesDuration: inboundVideo ? inboundVideo.totalFreezesDuration : undefined, + stalls: counters.stalls, + keyframeRequests: counters.keyframeRequests, + reattachments: counters.reattachments, + lastStallMs: counters.lastStallMs, + }; +} + +function updateRecoveryStatus() { + const counters = stallWatchdog.counters(); + let text = "No stalls"; + if (counters.stalled) { + text = "Recovering"; + } else if (counters.stalls > 0) { + text = String(counters.stalls) + (counters.stalls === 1 ? " stall" : " stalls") + + ", last " + (counters.lastStallMs / 1000).toFixed(1) + " s"; + } + overlay.recovery.textContent = text; + statistics.video.stallRecovery.textContent = String(counters.stalls) + " stalls, " + + String(counters.keyframeRequests) + " keyframes asked, " + + String(counters.reattachments) + " re-attached"; +} + +function presentedFrameCallbacksSupported() { + return typeof videoElement.requestVideoFrameCallback === "function"; +} + +// Counts frames that actually reached the screen, which framesDecoded cannot tell: a +// renderer that stopped presenting still decodes. Without the API the watchdog relies on +// framesDecoded alone. +function watchPresentedFrames() { + if (!presentedFrameCallbacksSupported() || presentedFrameCallbackPending) { + return; + } + presentedFrameCallbackPending = true; + videoElement.requestVideoFrameCallback(function (now, metadata) { + presentedFrameCallbackPending = false; + presentedFrames += 1; + streamLatency.recordPresentedFrame(metadata); + watchPresentedFrames(); + }); +} + +async function inboundVideoReport() { + const receiver = peerConnection.getReceivers().find(function (candidate) { + return candidate.track && candidate.track.kind === "video"; + }); + // A single receiver's report is a fraction of the whole connection's, which matters at + // twice a second on a TV. + const reports = receiver && typeof receiver.getStats === "function" + ? await receiver.getStats() + : await peerConnection.getStats(); + let inboundVideo = null; + reports.forEach(function (report) { + const kind = typeof report.kind === "undefined" ? report.mediaType : report.kind; + if (report.type === "inbound-rtp" && kind === "video" && !report.isRemote) { + inboundVideo = report; + } + }); + return inboundVideo; +} + +function requestKeyframe(reason) { + if (!dataChannelIsOpen(controlDataChannel)) { + return false; + } + controlDataChannel.send(JSON.stringify({ + v: GAMEPAD_PROTOCOL_VERSION, + type: "request-keyframe", + reason: reason, + })); + return true; +} + +// Rebuilds the renderer behind the video element without touching the connection: the same +// tracks are attached again, so audio and input carry on. +function reattachVideo() { + videoElement.srcObject = null; + videoElement.srcObject = remoteStream; + presentedFrameCallbackPending = false; + watchPresentedFrames(); + startPlayback(); +} + +async function sampleStall() { + if (stallSampleInProgress || !peerConnection || sessionState !== "streaming" + || document.hidden || videoElement.hidden) { + return; + } + stallSampleInProgress = true; + try { + watchPresentedFrames(); + const inboundVideo = await inboundVideoReport(); + if (!inboundVideo || !peerConnection) { + return; + } + const wasStalled = stallWatchdog.counters().stalled; + const verdict = stallWatchdog.sample({ + time: performance.now(), + bytesReceived: inboundVideo.bytesReceived, + framesDecoded: inboundVideo.framesDecoded, + framesPresented: presentedFrameCallbacksSupported() ? presentedFrames : undefined, + }); + if (verdict.stalledMs > 0) { + remoteLog("Stall sample: " + JSON.stringify({ + stalledMs: Math.round(verdict.stalledMs), + decoding: verdict.decoding, + bytesReceived: inboundVideo.bytesReceived, + packetsReceived: inboundVideo.packetsReceived, + packetsLost: inboundVideo.packetsLost, + framesReceived: inboundVideo.framesReceived, + framesDecoded: inboundVideo.framesDecoded, + framesDropped: inboundVideo.framesDropped, + keyFramesDecoded: inboundVideo.keyFramesDecoded, + framesPresented: presentedFrameCallbacksSupported() ? presentedFrames : null, + pliCount: inboundVideo.pliCount, + nackCount: inboundVideo.nackCount, + jitterBufferDelay: inboundVideo.jitterBufferDelay, + jitterBufferEmittedCount: inboundVideo.jitterBufferEmittedCount, + videoWidth: videoElement.videoWidth, + readyState: videoElement.readyState, + paused: videoElement.paused, + })); + } + if (verdict.action === "keyframe") { + const sent = requestKeyframe(verdict.decoding ? "not-presenting" : "not-decoding"); + log("Video stalled for " + String(Math.round(verdict.stalledMs)) + " ms (" + + (verdict.decoding ? "decoding, not presenting" : "not decoding") + + "), keyframe " + (sent ? "requested" : "not requested: control channel closed") + + " " + JSON.stringify(recoveryStatistics(inboundVideo))); + } else if (verdict.action === "reattach") { + log("Video still stalled after " + String(Math.round(verdict.stalledMs)) + + " ms, re-attaching the stream " + JSON.stringify(recoveryStatistics(inboundVideo))); + reattachVideo(); + requestKeyframe("reattached"); + } + if (wasStalled && !stallWatchdog.counters().stalled) { + log("Video recovered after " + String(Math.round(stallWatchdog.counters().lastStallMs)) + + " ms " + JSON.stringify(recoveryStatistics(inboundVideo))); + } + updateRecoveryStatus(); + } catch (error) { + if (!statisticsErrorReported) { + statisticsErrorReported = true; + reportError("Stall watchdog failed", error); + } + } finally { + stallSampleInProgress = false; + } +} + window.addEventListener("unhandledrejection", function (event) { reportError("Unhandled promise rejection", event.reason); }); @@ -3631,6 +3996,8 @@ setInterval(function () { updateStatistics(); updateInterpolationStatus(); }, 1000); +setInterval(sampleStall, STALL_SAMPLE_INTERVAL_MS); +setInterval(flushRemoteLog, REMOTE_LOG_FLUSH_MS); updateTrackCounts(); gamepadInputManager.updateDiagnostics(); syncGamepadUi(); @@ -3641,6 +4008,8 @@ log("Persistent storage: " + durableStorage.describe()); // preference set, and a select still holding its markup default would write that default // back over the saved value. interpolationSelect.value = savedPreferences.frameInterpolation ? "true" : "false"; +performanceOverlaySelect.value = savedPreferences.performanceOverlay ? "true" : "false"; +applyPerformanceOverlay(); restoreCachedCapabilities(); updateInterpolationStatus(); renderGateways(); diff --git a/tizen/config.xml b/tizen/config.xml index b525db1..f1abe6c 100644 --- a/tizen/config.xml +++ b/tizen/config.xml @@ -2,7 +2,7 @@ Settings
FPS60 FPSFixed
+ @@ -92,14 +93,17 @@

Preparing session

@@ -107,6 +111,9 @@

Media

Requested resolution
+ + + diff --git a/tizen/low-latency.js b/tizen/low-latency.js new file mode 100644 index 0000000..d9cfba6 --- /dev/null +++ b/tizen/low-latency.js @@ -0,0 +1,76 @@ +(function (global) { + "use strict"; + + // Two things the TV's WebRTC stack does by default that a game stream does not want. + // + // 1. Lip sync. Sunshine sends audio and video in one media stream, and WebRTC then holds + // every video frame back until the audio playing next to it catches up. The audio jitter + // buffer is the deeper of the two, so the picture inherits its delay. Moonlight never + // syncs the two either: in a game, a late picture costs more than a few milliseconds of + // audio drift. Moving the audio into a stream of its own ends the sync. + // 2. The video jitter buffer. It aims for smooth playback and grows with any jitter it sees. + // Asking for no extra delay keeps it at what the network actually needs. + + const AUDIO_STREAM_SUFFIX = "-audio"; + + // Renames the media stream of every audio section in an offer, so audio and video no longer + // share a sync group. An offer without stream ids is returned unchanged. + function separateAudioSync(sdp) { + if (typeof sdp !== "string") { + return sdp; + } + const newline = sdp.indexOf("\r\n") >= 0 ? "\r\n" : "\n"; + let inAudio = false; + return sdp.split(newline).map(function (line) { + if (line.indexOf("m=") === 0) { + inAudio = line.indexOf("m=audio") === 0; + return line; + } + if (!inAudio) { + return line; + } + return line + .replace(/^(a=msid:)(\S+)/, function (match, prefix, stream) { + return prefix + renamed(stream); + }) + .replace(/^(a=ssrc:\d+ msid:)(\S+)/, function (match, prefix, stream) { + return prefix + renamed(stream); + }); + }).join(newline); + } + + function renamed(stream) { + // "-" is the "no stream" id; there is nothing to sync and nothing to rename. + if (stream === "-" || stream.slice(-AUDIO_STREAM_SUFFIX.length) === AUDIO_STREAM_SUFFIX) { + return stream; + } + return stream + AUDIO_STREAM_SUFFIX; + } + + // Asks a receiver for the shortest jitter buffer it allows. Newer Chromium takes + // jitterBufferTarget in milliseconds; the TVs on older Chromium only know playoutDelayHint, + // in seconds. Returns which one was applied, or null when the TV has neither. + function minimizeReceiverDelay(receiver) { + if (!receiver) { + return null; + } + try { + if ("jitterBufferTarget" in receiver) { + receiver.jitterBufferTarget = 0; + return "jitterBufferTarget"; + } + if ("playoutDelayHint" in receiver) { + receiver.playoutDelayHint = 0; + return "playoutDelayHint"; + } + } catch (error) { + return null; + } + return null; + } + + global.LowLatency = { + separateAudioSync: separateAudioSync, + minimizeReceiverDelay: minimizeReceiverDelay, + }; +}(window)); diff --git a/tizen/preferences.js b/tizen/preferences.js index c2b6907..82873a3 100644 --- a/tizen/preferences.js +++ b/tizen/preferences.js @@ -9,6 +9,7 @@ bitrateKbps: null, frameRate: null, frameInterpolation: false, + performanceOverlay: false, }); function copyDefaults() { @@ -19,6 +20,7 @@ bitrateKbps: DEFAULTS.bitrateKbps, frameRate: DEFAULTS.frameRate, frameInterpolation: DEFAULTS.frameInterpolation, + performanceOverlay: DEFAULTS.performanceOverlay, }; } @@ -45,6 +47,9 @@ if (typeof candidate.frameInterpolation === "boolean") { values.frameInterpolation = candidate.frameInterpolation; } + if (typeof candidate.performanceOverlay === "boolean") { + values.performanceOverlay = candidate.performanceOverlay; + } return values; } diff --git a/tizen/stall-watchdog.js b/tizen/stall-watchdog.js new file mode 100644 index 0000000..44cccc7 --- /dev/null +++ b/tizen/stall-watchdog.js @@ -0,0 +1,123 @@ +(function (global) { + "use strict"; + + // Notices a picture that has stopped while video keeps arriving, and says what to do about it. + // + // The TV's WebRTC stack is supposed to ask for a keyframe on its own (a PLI) when it cannot + // decode, but a picture can also stop without the decoder failing: frames decode and are + // never shown, and nothing in the stack asks for anything. Seen from the couch both are a + // black screen with the audio still playing, lasting until the next keyframe the host + // happens to send for some unrelated reason. So the app watches for itself, escalating: + // first it asks the host for a keyframe, which cures a decoder waiting on a missing + // reference; then, if the picture still does not move, it re-attaches the stream to the + // video element, which rebuilds the renderer behind it. + // + // A counter is only trusted once it has been seen advancing. A TV that never reports one + // would otherwise look permanently stalled, and the watchdog would turn a healthy stream + // into one that is re-attached every few seconds. + + const STALL_MS = 1000; + const KEYFRAME_RETRY_MS = 1000; + const REATTACH_AFTER_MS = 2500; + const REATTACH_RETRY_MS = 5000; + + function isCount(value) { + return typeof value === "number" && isFinite(value); + } + + function StallWatchdog() { + this.reset(); + } + + StallWatchdog.prototype.reset = function () { + this.previous = null; + this.decodedTrusted = false; + this.presentedTrusted = false; + this.stalledSince = null; + this.stallCounted = false; + this.lastKeyframeAt = null; + this.lastReattachAt = null; + this.stalls = 0; + this.keyframeRequests = 0; + this.reattachments = 0; + this.lastStallMs = 0; + }; + + // sample: { time, bytesReceived, framesDecoded, framesPresented }, any counter may be + // missing. Returns { action: "none" | "keyframe" | "reattach", stalledMs, decoding }. + StallWatchdog.prototype.sample = function (sample) { + const previous = this.previous; + this.previous = sample; + if (!previous || !isCount(sample.time) || sample.time <= previous.time) { + return { action: "none", stalledMs: 0, decoding: true }; + } + const advanced = function (name) { + return isCount(sample[name]) && isCount(previous[name]) && sample[name] > previous[name]; + }; + const decodedAdvanced = advanced("framesDecoded"); + const presentedAdvanced = advanced("framesPresented"); + this.decodedTrusted = this.decodedTrusted || decodedAdvanced; + this.presentedTrusted = this.presentedTrusted || presentedAdvanced; + const decoding = !this.decodedTrusted || decodedAdvanced; + const presenting = !this.presentedTrusted || presentedAdvanced; + + if (decoding && presenting) { + if (this.stalledSince !== null && this.stallCounted) { + this.lastStallMs = sample.time - this.stalledSince; + } + this.stalledSince = null; + this.stallCounted = false; + this.lastKeyframeAt = null; + this.lastReattachAt = null; + return { action: "none", stalledMs: 0, decoding: true }; + } + // Nothing arriving is the network's or the host's doing, and neither a keyframe request + // nor a new renderer would bring the picture back. + if (!advanced("bytesReceived")) { + return { action: "none", stalledMs: 0, decoding: decoding }; + } + if (this.stalledSince === null) { + this.stalledSince = previous.time; + } + const stalledMs = sample.time - this.stalledSince; + if (stalledMs < STALL_MS) { + return { action: "none", stalledMs: stalledMs, decoding: decoding }; + } + if (!this.stallCounted) { + this.stallCounted = true; + this.stalls += 1; + } + if (stalledMs >= REATTACH_AFTER_MS + && (this.lastReattachAt === null || sample.time - this.lastReattachAt >= REATTACH_RETRY_MS)) { + // The app asks for a keyframe along with every re-attachment, so it restarts that clock. + this.lastReattachAt = sample.time; + this.lastKeyframeAt = sample.time; + this.reattachments += 1; + return { action: "reattach", stalledMs: stalledMs, decoding: decoding }; + } + if (this.lastKeyframeAt === null || sample.time - this.lastKeyframeAt >= KEYFRAME_RETRY_MS) { + this.lastKeyframeAt = sample.time; + this.keyframeRequests += 1; + return { action: "keyframe", stalledMs: stalledMs, decoding: decoding }; + } + return { action: "none", stalledMs: stalledMs, decoding: decoding }; + }; + + StallWatchdog.prototype.counters = function () { + return { + stalls: this.stalls, + keyframeRequests: this.keyframeRequests, + reattachments: this.reattachments, + lastStallMs: this.lastStallMs, + stalled: this.stallCounted, + }; + }; + + global.StallWatchdog = { + create: function () { return new StallWatchdog(); }, + STALL_MS: STALL_MS, + KEYFRAME_RETRY_MS: KEYFRAME_RETRY_MS, + REATTACH_AFTER_MS: REATTACH_AFTER_MS, + REATTACH_RETRY_MS: REATTACH_RETRY_MS, + }; +}(window)); diff --git a/tizen/stream-latency.js b/tizen/stream-latency.js new file mode 100644 index 0000000..530edc7 --- /dev/null +++ b/tizen/stream-latency.js @@ -0,0 +1,174 @@ +(function (global) { + "use strict"; + + // Where the time goes between Sunshine sending a frame and the TV showing it, as far as the + // TV can see. The TV cannot see the PC's capture and encode time, so every figure here + // starts when the frame leaves the PC. + // + // - Network: half the round trip, the usual one-way estimate. + // - Jitter buffer, decode: averages over the last sample window, from getStats(). + // - To screen: from the last packet of a frame arriving to the frame being composited, + // measured per frame by requestVideoFrameCallback. This is the one that counts drawing. + // A TV without that API falls back to arrival-to-decoded, and says so. + // - Total: network plus to screen. Average: the mean of every total this session. + + // A presented-frame measurement outside this range is a clock glitch, not a latency. + const MAX_PLAUSIBLE_MS = 2000; + + function create() { + let previous; + let previousTimestamp; + let presented; + let totals; + + function reset() { + previous = { + packetsReceived: 0, + packetsLost: 0, + framesReceived: 0, + framesDropped: 0, + framesDecoded: 0, + totalDecodeTime: 0, + jitterBufferDelay: 0, + jitterBufferEmittedCount: 0, + totalProcessingDelay: 0, + }; + previousTimestamp = null; + presented = { sum: 0, count: 0, max: 0, supported: false }; + totals = { sum: 0, count: 0 }; + } + + // Called from requestVideoFrameCallback with its metadata. + function recordPresentedFrame(metadata) { + if (!metadata || typeof metadata.receiveTime !== "number" || metadata.receiveTime <= 0) { + return; + } + const shownAt = typeof metadata.expectedDisplayTime === "number" + ? metadata.expectedDisplayTime + : metadata.presentationTime; + if (typeof shownAt !== "number") { + return; + } + const elapsed = shownAt - metadata.receiveTime; + if (elapsed < 0 || elapsed > MAX_PLAUSIBLE_MS) { + return; + } + presented.supported = true; + presented.sum += elapsed; + presented.count += 1; + presented.max = Math.max(presented.max, elapsed); + } + + function number(report, name) { + return report && typeof report[name] === "number" ? report[name] : null; + } + + // Change of a cumulative counter since the last sample. A counter that went backwards + // belongs to a new receiver, so the whole value is the change. + function delta(report, name) { + const value = number(report, name); + if (value === null) { + return null; + } + const change = value - previous[name]; + previous[name] = value; + return change < 0 ? value : change; + } + + function ratio(numerator, denominator, scale) { + return numerator !== null && denominator !== null && denominator > 0 + ? (numerator * scale) / denominator + : null; + } + + function sample(inboundVideo, roundTripSeconds) { + const sessionReceived = number(inboundVideo, "packetsReceived"); + const sessionLost = number(inboundVideo, "packetsLost"); + const packetsReceived = delta(inboundVideo, "packetsReceived"); + const packetsLost = delta(inboundVideo, "packetsLost"); + const framesReceived = delta(inboundVideo, "framesReceived"); + const framesDropped = delta(inboundVideo, "framesDropped"); + const framesDecoded = delta(inboundVideo, "framesDecoded"); + const decodeTime = delta(inboundVideo, "totalDecodeTime"); + const bufferDelay = delta(inboundVideo, "jitterBufferDelay"); + const emitted = delta(inboundVideo, "jitterBufferEmittedCount"); + const processingDelay = delta(inboundVideo, "totalProcessingDelay"); + + // Frames that arrived per second, over the time the reports themselves cover. + const timestamp = number(inboundVideo, "timestamp"); + const elapsedSeconds = timestamp !== null && previousTimestamp !== null + && timestamp > previousTimestamp ? (timestamp - previousTimestamp) / 1000 : null; + previousTimestamp = timestamp; + const receivedFps = ratio(framesReceived, elapsedSeconds, 1); + + const roundTripMs = typeof roundTripSeconds === "number" ? roundTripSeconds * 1000 : null; + const networkMs = roundTripMs === null ? null : roundTripMs / 2; + const jitterBufferMs = ratio(bufferDelay, emitted, 1000); + const decodeMs = ratio(decodeTime, framesDecoded, 1000); + const processingMs = ratio(processingDelay, emitted, 1000); + const displayMs = presented.count > 0 ? presented.sum / presented.count : null; + const displayMaxMs = presented.count > 0 ? presented.max : null; + const displaySupported = presented.supported; + presented.sum = 0; + presented.count = 0; + presented.max = 0; + + // Drawing included when the TV reports it; otherwise as far as the decoder. + const tvMs = displayMs !== null ? displayMs : processingMs; + let totalMs = null; + if (tvMs !== null) { + totalMs = tvMs + (networkMs || 0); + totals.sum += totalMs; + totals.count += 1; + } + + const lostShare = packetsLost !== null && packetsReceived !== null + ? ratio(packetsLost, packetsReceived + packetsLost, 100) + : null; + const sessionLostShare = sessionLost !== null && sessionReceived !== null + ? ratio(sessionLost, sessionReceived + sessionLost, 100) + : null; + + return { + roundTripMs: roundTripMs, + networkMs: networkMs, + jitterBufferMs: jitterBufferMs, + decodeMs: decodeMs, + processingMs: processingMs, + displayMs: displayMs, + displayMaxMs: displayMaxMs, + includesDisplay: displayMs !== null || displaySupported, + totalMs: totalMs, + averageTotalMs: totals.count > 0 ? totals.sum / totals.count : null, + packetLossPercent: lostShare === null ? null : Math.max(0, lostShare), + sessionPacketLossPercent: sessionLostShare === null ? null : Math.max(0, sessionLostShare), + framesDroppedPercent: ratio(framesDropped, framesReceived, 100), + receivedFps: receivedFps, + }; + } + + reset(); + return { reset: reset, recordPresentedFrame: recordPresentedFrame, sample: sample }; + } + + // The round trip of the candidate pair carrying the stream, in seconds, or null. + function roundTripSeconds(reports) { + let selectedPairId = null; + const pairs = {}; + let nominated = null; + reports.forEach(function (report) { + if (report.type === "transport" && report.selectedCandidatePairId) { + selectedPairId = report.selectedCandidatePairId; + } else if (report.type === "candidate-pair") { + pairs[report.id] = report; + if (report.nominated && report.state === "succeeded" && !nominated) { + nominated = report; + } + } + }); + const pair = (selectedPairId && pairs[selectedPairId]) || nominated; + return pair && typeof pair.currentRoundTripTime === "number" ? pair.currentRoundTripTime : null; + } + + global.StreamLatency = { create: create, roundTripSeconds: roundTripSeconds }; +}(window)); diff --git a/tizen/ui.css b/tizen/ui.css index fee6ec4..f9a8366 100644 --- a/tizen/ui.css +++ b/tizen/ui.css @@ -15,3 +15,4 @@ @media (max-width:950px) { .settings-screen { grid-template-columns:1fr; }.settings-categories { grid-template-columns:repeat(5,minmax(0,1fr)); }.settings-category { grid-template-columns:1fr; justify-items:center; gap:3px; padding:12px; font-size:17px; text-align:center; }.settings-category > span:last-child { display:none; }.settings-content { min-height:0; }.setting-row { padding:0 20px; }.applications-grid { grid-template-columns:repeat(auto-fill,minmax(190px,1fr)); } } .application-card.focusable:focus,.application-card.focusable:focus-visible { border-color:transparent !important; box-shadow:none !important; background-color:transparent !important; } .gateway-grid { grid-template-columns:repeat(auto-fill,minmax(270px,340px)); }.gateway-add-card { gap:22px; }.gateway-add-symbol { display:flex; width:82px; height:82px; align-items:center; justify-content:center; border:3px solid var(--text-secondary); border-radius:50%; color:var(--text-primary); font-size:62px; font-weight:300; line-height:1; }.gateway-editor-panel { width:min(760px,calc(100vw - 100px)); }.gateway-editor-panel > p:first-of-type { margin:0 0 16px; color:var(--text-secondary); font-size:22px; }.gateway-octets { display:flex; align-items:center; justify-content:center; gap:13px; margin:8px 0 18px; color:var(--text-secondary); font-size:34px; font-weight:700; }.gateway-octet { width:128px; min-height:82px; border:2px solid var(--panel-border); border-radius:6px; outline:0; background:#242427; color:var(--text-primary); font-size:34px; font-weight:700; text-align:center; }.gateway-port-digit { width:72px; }.gateway-octet:disabled { opacity:1; border-color:var(--accent); }.gateway-editor-hint { margin:0 0 22px; color:var(--text-secondary); font-size:19px; text-align:center; }.gateway-editor-actions { grid-template-columns:repeat(2,minmax(0,1fr)); }.gateway-editor-panel .session-dialog-message { color:#ffd0d8; } +.performance-hud { position:fixed; top:16px; margin:0; padding:8px 14px; border-radius:6px; background:rgba(0,0,0,.62); color:#fff; font-family:monospace; pointer-events:none; }.performance-hud-left { left:16px; display:flex; align-items:baseline; gap:8px; }.performance-hud-left strong { font-size:30px; }.performance-hud-left span { font-size:16px; font-weight:700; }.performance-hud-left small { color:var(--text-secondary); font-size:15px; }.performance-hud-left.is-low strong { color:var(--danger); }.performance-hud-right { right:16px; display:grid; gap:2px; min-width:250px; }.performance-hud-right div { display:flex; justify-content:space-between; gap:16px; }.performance-hud-right dt,.performance-hud-right dd { margin:0; font-size:15px; }.performance-hud-right dt { color:var(--text-secondary); }.performance-hud-right dd { font-weight:700; } diff --git a/tizen/update-check.js b/tizen/update-check.js index 104ae01..9eae9f4 100644 --- a/tizen/update-check.js +++ b/tizen/update-check.js @@ -2,7 +2,7 @@ "use strict"; // Kept equal to the version in config.xml; tests/TizenUpdateCheckTest.js checks that they match. - const APP_VERSION = "2.0.0"; + const APP_VERSION = "2.1.0"; const RELEASES_URL = "https://api.github.com/repos/mlopezsegura/moonlight-webrtc-tizen/releases?per_page=20"; const RELEASES_PAGE = "github.com/mlopezsegura/moonlight-webrtc-tizen/releases"; const STORAGE_KEY = "moonlight-webrtc.update-check";