diff --git a/CHANGELOG.md b/CHANGELOG.md index dc21fb2d1b..3ccfc2796a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## 1.0-test236 (2026-10-21) +- Found on the JAL TV (AQUOS, Android 11, MediaTek with a Mali GPU), PS5 over PlayStation Network at 720p: + - Super resolution didn't start there ("eglChooseConfig"): the GPU has no OpenGL ES 3 setup for both a window and an off-screen surface. FSR now tries several setups, and without an off-screen surface it works surfaceless. Before, it fell back to the plain picture, as designed. + - PS5 decoder: the TV's decoder (OMX.MS.HEVC.Decoder) refused the low-latency settings, and MoonCast dropped all of them. Now it drops them one step at a time (vendor settings, then MediaTek's low-latency decode, then Android's standard low-latency mode, then real-time priority), so a decoder keeps as many as it takes. The log says which level was used. +- The same JAL TV test passed: PS5 over PSN on OpenSSL 3 (connected in 84 s including waking the console), 60 fps, 25 ms network delay, no lost packets, Decode time 20.5 ms, and Put PS5 to rest from the Quick menu. + +## 1.0-test235 (2026-10-21) +- Super resolution now works on PC and Xbox streams too, not only PS5. The FSR passes are shared (`FsrPasses`), used by: + - PC: Moonlight's decoder renders into FSR's surface. FSR stays off with HDR, since its passes are 8-bit. The screen surface keeps the screen's size instead of the stream's, so FSR has room to upscale into; with Stretch, the picture is stretched by FSR. + - Xbox: WebRTC's renderer draws each frame through FSR's drawer (`FsrGlDrawer`), with WebRTC's hardware scaler off so the surface keeps the view's size. Frames that aren't hardware-decoded, and GPUs without OpenGL ES 3, are drawn by WebRTC's own drawer, as before. + - PS5: as in test234. +- In all three, FSR only takes effect when the stream is clearly smaller than the screen; otherwise the picture shows as before. + +## 1.0-test234 (2026-10-21) +- Super resolution (Settings, Picture, both apps): AMD FidelityFX Super Resolution 1.0, for the PS5 first (mockups in docs/mockups/fsr). With it on, and the stream clearly smaller than the screen (a PS5's 1080p or 720p on the Fold's 2448 × 1696, say), each frame takes three GPU passes before it shows: + - a copy of the decoder's frame at the stream's size; + - EASU, FSR's edge-adaptive upscale to the screen's size; + - RCAS, FSR's sharpening, at the new Sharpening setting (0 to 100%, 60% by default). +- How it fits in: + - The shaders are ports of AMD's ffx_fsr1.h (MIT) to OpenGL ES 3.0. Its 4-texel gathers are 4 texel reads, as ES 3.0 has no textureGather. + - Frames keep the decoder's presentation time, so Frame pacing still works. + - On a foldable, a fold only moves FSR's output; the decoder keeps drawing into FSR. + - Off by default. When it's not worth it (the stream is as big as the screen) or the GPU can't set it up, the stream shows directly, as before. +- About, Open-source licences: AMD FidelityFX Super Resolution 1.0 (MIT). + ## 1.0-test233 (2026-10-21) - PS5: frame pacing (Settings, PlayStation, Frame pacing: Lowest delay, Balanced, Smoothest), like the PC's. - Lowest delay (the default, as before): each frame shows the moment it's decoded, so frames that arrive unevenly over Wi-Fi show unevenly. diff --git a/NOTICE b/NOTICE index 2e1cd47187..266618709c 100644 --- a/NOTICE +++ b/NOTICE @@ -16,6 +16,7 @@ The app ------- Moonlight for Android (GPL-3.0): The app MoonCast grew from: PC streaming, the on-screen controller. https://github.com/moonlight-stream/moonlight-android [licences/gpl-3.0.txt] moonlight-common-c (GPL-3.0): The PC streaming protocol. https://github.com/moonlight-stream/moonlight-common-c [licences/gpl-3.0.txt] +AMD FidelityFX Super Resolution 1.0 (MIT): Super resolution: the upscaling and sharpening (FSR 1). https://github.com/GPUOpen-Effects/FidelityFX-FSR [licences/fsr1.txt] ENet (MIT): Inside moonlight-common-c: the network link. https://github.com/lsalzman/enet [licences/enet.txt] nanors (MIT): Inside moonlight-common-c: repairing lost video. https://github.com/sleepybishop/nanors [licences/nanors.txt] diff --git a/app/src/handheld/java/com/limelight/handheld/HandheldHomeActivity.kt b/app/src/handheld/java/com/limelight/handheld/HandheldHomeActivity.kt index bb38d8f687..1276e6a9b5 100644 --- a/app/src/handheld/java/com/limelight/handheld/HandheldHomeActivity.kt +++ b/app/src/handheld/java/com/limelight/handheld/HandheldHomeActivity.kt @@ -1254,6 +1254,13 @@ class HandheldHomeActivity : Activity() { else { stream.edit().putInt("seekbar_bitrate_kbps", (Math.round(v * 2) * 500).toInt()).apply(); render(force = true); null } } } + // Super resolution: AMD FSR 1.0 upscales a stream smaller than the screen, then sharpens it (PS5 for now) + choice(list, stream, com.limelight.binding.video.FsrRenderer.PREF, "Super resolution", + "Sharper picture when the stream is smaller than this screen (AMD FSR 1.0), on PC, PS5 and Xbox. Not with HDR. Uses a little of the GPU", + listOf("Off" to "off", "FSR 1.0" to "fsr1"), "off") + if (stream.getString(com.limelight.binding.video.FsrRenderer.PREF, "off") == "fsr1") + intChoice(list, com.limelight.binding.video.FsrRenderer.SHARPNESS_PREF, "Sharpening", "How crisp FSR makes edges. Lower it if you see halos", + listOf(0, 20, 40, 60, 80, 100), com.limelight.binding.video.FsrRenderer.DEFAULT_SHARPNESS) { "$it%" } choice(list, stream, "video_format", "Video codec", "H.265 and AV1 look better at the same bitrate", listOf("Auto" to "auto", "H.264" to "neverh265", "H.265" to "forceh265", "AV1" to "forceav1"), "auto") toggle(list, "checkbox_enable_hdr", "HDR", "Only when the PC, the game and this screen support it", false) diff --git a/app/src/main/assets/licences/fsr1.txt b/app/src/main/assets/licences/fsr1.txt new file mode 100644 index 0000000000..324cba594d --- /dev/null +++ b/app/src/main/assets/licences/fsr1.txt @@ -0,0 +1,19 @@ +Copyright (c) 2021 Advanced Micro Devices, Inc. All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/app/src/main/java/com/limelight/Game.java b/app/src/main/java/com/limelight/Game.java index 004784edab..7ff4d1ecc6 100644 --- a/app/src/main/java/com/limelight/Game.java +++ b/app/src/main/java/com/limelight/Game.java @@ -125,6 +125,10 @@ public class Game extends Activity implements SurfaceHolder.Callback, private boolean connected = false; private boolean autoEnterPip = false; private boolean surfaceCreated = false; + // MoonCast: Super resolution (FSR 1.0) + private boolean fsrPlanned = false; + private float fsrSharpness = 0.6f; + private volatile com.limelight.binding.video.FsrRenderer fsr; private boolean attemptedConnection = false; private int suppressPipRefCount = 0; private String pcName; @@ -405,6 +409,21 @@ public void notifyCrash(Exception e) { Toast.makeText(this, "Decoder does not support HDR10 profile", Toast.LENGTH_LONG).show(); } + // MoonCast: Super resolution (FSR 1.0) when it's on, the stream isn't HDR (FSR's passes are 8-bit) and the + // stream is clearly smaller than this screen. The surface then keeps the screen's size (no setFixedSize) + // so FSR has room to upscale into. + { + android.content.SharedPreferences sp = android.preference.PreferenceManager.getDefaultSharedPreferences(this); + android.graphics.Point real = new android.graphics.Point(); + getWindowManager().getDefaultDisplay().getRealSize(real); + fsrPlanned = com.limelight.binding.video.FsrRenderer.Companion.enabled(sp) && !willStreamHdr + && com.limelight.binding.video.FsrRenderer.Companion.worthIt(prefConfig.height, Math.min(real.x, real.y)); + fsrSharpness = com.limelight.binding.video.FsrRenderer.Companion.sharpness(sp); + if (com.limelight.binding.video.FsrRenderer.Companion.enabled(sp)) + LimeLog.info("FSR " + (fsrPlanned ? "on" : "not needed") + ": stream " + prefConfig.width + "x" + prefConfig.height + + ", screen " + real.x + "x" + real.y + (willStreamHdr ? ", HDR" : "")); + } + // Display a message to the user if HEVC was forced on but we still didn't find a decoder if (prefConfig.videoFormat == PreferenceConfiguration.FormatOption.FORCE_HEVC && !decoderRenderer.isHevcSupported()) { Toast.makeText(this, "No HEVC decoder found", Toast.LENGTH_LONG).show(); @@ -972,15 +991,16 @@ else if (!isRefreshRateGoodMatch(candidate.getRefreshRate())) { } } - if (prefConfig.stretchVideo || aspectRatioMatch) { + if (!fsrPlanned && (prefConfig.stretchVideo || aspectRatioMatch)) { // Set the surface to the size of the video streamView.getHolder().setFixedSize(prefConfig.width, prefConfig.height); } - else { + else if (!(fsrPlanned && prefConfig.stretchVideo)) { // Set the surface to scale based on the aspect ratio of the stream streamView.setDesiredAspectRatio((double)prefConfig.width / (double)prefConfig.height); streamView.setFill(prefConfig.fillVideo); } + // (FSR with Stretch: the view fills the screen and FSR stretches the picture into it) // MoonCast: Settings, Video, Zoom. Enlarges the picture for TVs that crop the edges (overscan). if (prefConfig.videoZoomPercent > 100) { @@ -2411,6 +2431,10 @@ private void stopConnection(final Runnable done) { new Thread() { public void run() { conn.stop(); + // the decoder is gone: FSR's surface can go too + com.limelight.binding.video.FsrRenderer f = fsr; + fsr = null; + if (f != null) f.release(); if (done != null) runOnUiThread(done); } }.start(); @@ -2723,6 +2747,17 @@ public void surfaceChanged(SurfaceHolder holder, int format, int width, int heig UiHelper.notifyStreamConnecting(Game.this); decoderRenderer.setRenderTarget(holder); + if (fsrPlanned && fsr == null) { + try { + fsr = new com.limelight.binding.video.FsrRenderer(prefConfig.width, prefConfig.height, fsrSharpness); + fsr.attachOutput(holder.getSurface()); + decoderRenderer.setSurfaceOverride(fsr.getInputSurface()); + LimeLog.info("FSR on: " + prefConfig.width + "x" + prefConfig.height + " to " + width + "x" + height); + } catch (RuntimeException e) { + LimeLog.warning("FSR unavailable, showing the stream directly: " + e); + fsr = null; + } + } conn.start(new AndroidAudioRenderer(Game.this, prefConfig.enableAudioFx), decoderRenderer, Game.this); } @@ -2775,6 +2810,9 @@ public void surfaceDestroyed(SurfaceHolder holder) { throw new IllegalStateException("Surface destroyed before creation!"); } + // FSR stops drawing to the screen surface before it goes + if (fsr != null) fsr.detachOutput(); + if (attemptedConnection) { // Let the decoder know immediately that the surface is gone decoderRenderer.prepareForStop(); diff --git a/app/src/main/java/com/limelight/Ps5StreamActivity.kt b/app/src/main/java/com/limelight/Ps5StreamActivity.kt index 8451be4712..bcb38a1af5 100644 --- a/app/src/main/java/com/limelight/Ps5StreamActivity.kt +++ b/app/src/main/java/com/limelight/Ps5StreamActivity.kt @@ -183,6 +183,7 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home val profile = if (away) preset.copy(bitrate = awayKbps ?: minOf(preset.bitrate, linkCap)) else preset.copy(bitrate = homeKbps) streamInfo = Triple(when (resPreset) { VideoResolutionPreset.RES_540P -> "540p"; VideoResolutionPreset.RES_720P -> "720p"; else -> "1080p" }, if (h264) "H.264" else "H.265", profile.bitrate / 1000) streamRemote = remote + videoW = profile.width; videoH = profile.height Log.i(TAG, "stream ${streamInfo.first}${profile.maxFPS} at ${profile.bitrate / 1000} Mbps (${if (remote) "PSN" else "home"}, ${if (away) "away" else "home"} profile, link ${if (link > 0) "$link Mbps" else "cable/unknown"})") if (!remote) return ConnectInfo(ps5, host, registKey, morning, profile) val account = intent.getStringExtra(EXTRA_ACCOUNT_ID)?.let { @@ -270,7 +271,9 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home } catch (e: Exception) { false } override fun surfaceCreated(holder: SurfaceHolder) { - if (started) { session?.setSurface(holder.surface); return } + screenHolder = holder + // with FSR the decoder keeps its own surface: only FSR's output moves to the new screen surface + if (started) { val f = fsr; if (f != null) f.attachOutput(holder.surface) else session?.setSurface(holder.surface); return } started = true val surface = holder.surface val previous = connectThread @@ -334,7 +337,7 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home val s = try { Session(connectInfo(remote), null, false) } finally { watch?.interrupt() } if (closed) { try { s.dispose() } catch (e: Exception) {}; return } // the screen closed during the hole-punch s.eventCallback = { ev -> onEvent(ev) } - s.setSurface(surface) + s.setSurface(decoderSurface(surface)) // Only the first picture counts as success; a quit before that is a failed attempt -> retry. // Remote: the data-channel hole-punch may fail. Home: the console may still be waking up. // Armed before start() (#38): a console still waking refuses within milliseconds, often before @@ -740,6 +743,35 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home private val menu by lazy { com.limelight.home.QuickMenu(this) } private var comboPending = false @Volatile private var streamInfo = Triple("1080p", "H.265", 0) + + // ---- Super resolution (FSR 1.0): the decoder renders into FSR's surface, FSR upscales to the screen's + @Volatile private var fsr: com.limelight.binding.video.FsrRenderer? = null + @Volatile private var screenHolder: SurfaceHolder? = null + @Volatile private var videoW = 1920 + @Volatile private var videoH = 1080 + + /** The surface the decoder renders into: the screen's, or FSR's when it's on and the stream is smaller than the screen. */ + private fun decoderSurface(screen: android.view.Surface): android.view.Surface { + val prefs = android.preference.PreferenceManager.getDefaultSharedPreferences(this) + if (prefs.getString(com.limelight.binding.video.FsrRenderer.PREF, "off") != "fsr1") return screen + val outH = screenHolder?.surfaceFrame?.height() ?: 0 + if (!com.limelight.binding.video.FsrRenderer.worthIt(videoH, outH)) { + Log.i(TAG, "FSR not needed: stream ${videoW}x$videoH, screen ${screenHolder?.surfaceFrame?.width()}x$outH") + return screen + } + fsr?.let { return it.inputSurface } + return try { + val sharp = prefs.getInt(com.limelight.binding.video.FsrRenderer.SHARPNESS_PREF, com.limelight.binding.video.FsrRenderer.DEFAULT_SHARPNESS) + val f = com.limelight.binding.video.FsrRenderer(videoW, videoH, sharp / 100f) + f.attachOutput(screen) + fsr = f + Log.i(TAG, "FSR on: ${videoW}x$videoH to ${screenHolder?.surfaceFrame?.width()}x$outH, sharpening $sharp%") + f.inputSurface + } catch (e: Exception) { + Log.w(TAG, "FSR unavailable, showing the stream directly: $e") + screen + } + } @Volatile private var streamRemote = false private val startedAt = System.currentTimeMillis() private val swapCrossCircle by lazy { @@ -897,12 +929,14 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home // into a dead surface fails, and chiaki then crashed copying a frame into a missing buffer. The release runs on a // worker (it takes chiaki's decoder lock) and the wait is bounded, so a stuck session can't freeze the app. override fun surfaceDestroyed(holder: SurfaceHolder) { + fsr?.detachOutput() // FSR stops drawing to the screen surface before it goes; the decoder keeps FSR's if (closed) { if (session != null && teardownThread == null) teardown(null) // the PS5 quit: tear down now, not in onDestroy try { teardownThread?.join(SURFACE_RELEASE_WAIT_MS) } catch (e: InterruptedException) { } return } val s = session ?: return + if (fsr != null) return // the decoder draws into FSR's surface, which stays val t = Thread({ synchronized(sessionLock) { if (!closed && synchronized(disposed) { s !in disposed }) try { s.setSurface(null) } catch (e: Exception) { } } }, "ps5-surface") t.start() @@ -937,6 +971,8 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home try { s.stop() } catch (e: Exception) { } synchronized(sessionLock) { try { s.dispose() } catch (e: Exception) { } } } + // after the decoder has let go of its surface + fsr?.release(); fsr = null if (done != null) android.os.Handler(android.os.Looper.getMainLooper()).post(done) }, "ps5-teardown") teardownThread = t diff --git a/app/src/main/java/com/limelight/XboxStreamActivity.kt b/app/src/main/java/com/limelight/XboxStreamActivity.kt index 46edab3a4e..a26c67f429 100644 --- a/app/src/main/java/com/limelight/XboxStreamActivity.kt +++ b/app/src/main/java/com/limelight/XboxStreamActivity.kt @@ -71,13 +71,18 @@ class XboxStreamActivity : Activity(), com.limelight.home.MoveToTv.LocalSession com.limelight.binding.video.MediaCodecHelper.initialize(this, com.limelight.preferences.GlPreferences.readPreferences(this).glRenderer) val r = SurfaceViewRenderer(this).also { renderer = it } + // Super resolution (FSR 1.0): WebRTC's renderer draws each frame through FSR's drawer. The surface then keeps + // the view's real size (no hardware scaler, which would shrink it to the frame's) so FSR has room to upscale + val fsrOn = com.limelight.binding.video.FsrRenderer.enabled(prefs) r.init(egl.eglBaseContext, object : RendererCommon.RendererEvents { override fun onFirstFrameRendered() { firstFrame = true; main.removeCallbacks(pictureWatchdog); showStatus(null); com.limelight.home.StreamReport.playing() } override fun onFrameResolutionChanged(w: Int, h: Int, rotation: Int) { Log.i(TAG, "picture ${w}x$h"); runOnUiThread { applyShape(w, h) } } - }) - r.setEnableHardwareScaler(true) + }, org.webrtc.EglBase.CONFIG_PLAIN, + if (fsrOn) com.limelight.binding.video.FsrGlDrawer(com.limelight.binding.video.FsrRenderer.sharpness(prefs)) else org.webrtc.GlRectDrawer()) + r.setEnableHardwareScaler(!fsrOn) + if (fsrOn) Log.i(TAG, "FSR drawer on") applyShape(0, 0) status = TextView(this).apply { setTextColor(0xFFE8ECFF.toInt()); textSize = 22f; gravity = Gravity.CENTER diff --git a/app/src/main/java/com/limelight/binding/video/FsrRenderer.kt b/app/src/main/java/com/limelight/binding/video/FsrRenderer.kt new file mode 100644 index 0000000000..6ed3aba833 --- /dev/null +++ b/app/src/main/java/com/limelight/binding/video/FsrRenderer.kt @@ -0,0 +1,521 @@ +package com.limelight.binding.video + +import android.graphics.SurfaceTexture +import android.opengl.EGL14 +import android.opengl.EGLConfig +import android.opengl.EGLContext +import android.opengl.EGLDisplay +import android.opengl.EGLExt +import android.opengl.EGLSurface +import android.opengl.GLES11Ext +import android.opengl.GLES30 +import android.os.Handler +import android.os.HandlerThread +import android.util.Log +import android.view.Surface +import java.util.concurrent.CountDownLatch +import java.util.concurrent.TimeUnit + +// Super resolution with AMD FidelityFX Super Resolution 1.0 (FSR 1, MIT licence, see assets/licences). +// The shaders are ports of ffx_fsr1.h's 32-bit FsrEasuF and FsrRcasF to OpenGL ES 3.0. ES 3.0 has no textureGather, +// so each gather is four texelFetches in gather order. + +/** + * The three FSR passes on the current GL context (OpenGL ES 3.0): copy an external OES frame into a plain texture at + * the frame's size, EASU (AMD's edge-adaptive Lanczos) to the output size, RCAS sharpening into [targetFb]'s viewport. + * GL objects are made on first use and remade when a size changes. Used by [FsrRenderer] (PS5, PC) and [FsrGlDrawer] + * (Xbox, inside WebRTC's renderer). + */ +class FsrPasses(sharpness: Float) { + private var copyProg = 0 + private var easuProg = 0 + private var rcasProg = 0 + private var copyTex = 0 + private var copyFbo = 0 + private var copyW = 0 + private var copyH = 0 + private var easuTex = 0 + private var easuFbo = 0 + private var easuW = 0 + private var easuH = 0 + private val vao = IntArray(1) + private val sharp = Math.pow(2.0, -((1.0 - sharpness.coerceIn(0f, 1f)) * 2.0)).toFloat() // stops: 100% = 0, 0% = 2 + + /** Draws one frame. [oesTex] with [texMatrix] is the frame ([inW] x [inH]); the result fills the viewport. */ + fun draw(oesTex: Int, texMatrix: FloatArray, inW: Int, inH: Int, targetFb: Int, vx: Int, vy: Int, outW: Int, outH: Int) { + if (inW <= 0 || inH <= 0 || outW <= 0 || outH <= 0) return + if (copyProg == 0) { + copyProg = program(VERTEX, COPY); easuProg = program(VERTEX, EASU); rcasProg = program(VERTEX, RCAS) + GLES30.glGenVertexArrays(1, vao, 0) + } + if (copyW != inW || copyH != inH) { + free(copyTex, copyFbo) + val (t, f) = target(inW, inH); copyTex = t; copyFbo = f; copyW = inW; copyH = inH + } + if (easuW != outW || easuH != outH) { + free(easuTex, easuFbo) + val (t, f) = target(outW, outH); easuTex = t; easuFbo = f; easuW = outW; easuH = outH + Log.i(TAG, "output ${outW}x$outH (from ${inW}x$inH)") + } + GLES30.glBindVertexArray(vao[0]) + GLES30.glDisable(GLES30.GL_BLEND) + GLES30.glActiveTexture(GLES30.GL_TEXTURE0) + + // 1. the frame into a plain texture at its own size + GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, copyFbo) + GLES30.glViewport(0, 0, inW, inH) + GLES30.glUseProgram(copyProg) + GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, oesTex) + GLES30.glUniform1i(GLES30.glGetUniformLocation(copyProg, "uTex"), 0) + GLES30.glUniformMatrix4fv(GLES30.glGetUniformLocation(copyProg, "uSt"), 1, false, texMatrix, 0) + GLES30.glDrawArrays(GLES30.GL_TRIANGLES, 0, 3) + + // 2. EASU up to the output size (constants as FsrEasuCon, kept as floats) + GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, easuFbo) + GLES30.glViewport(0, 0, outW, outH) + GLES30.glUseProgram(easuProg) + GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, copyTex) + GLES30.glUniform1i(GLES30.glGetUniformLocation(easuProg, "uTex"), 0) + val iw = inW.toFloat(); val ih = inH.toFloat() + GLES30.glUniform4f(GLES30.glGetUniformLocation(easuProg, "uCon0"), iw / outW, ih / outH, 0.5f * iw / outW - 0.5f, 0.5f * ih / outH - 0.5f) + GLES30.glUniform4f(GLES30.glGetUniformLocation(easuProg, "uCon1"), 1f / iw, 1f / ih, 1f / iw, -1f / ih) + GLES30.glUniform4f(GLES30.glGetUniformLocation(easuProg, "uCon2"), -1f / iw, 2f / ih, 1f / iw, 2f / ih) + GLES30.glUniform4f(GLES30.glGetUniformLocation(easuProg, "uCon3"), 0f, 4f / ih, 0f, 0f) + GLES30.glUniform2i(GLES30.glGetUniformLocation(easuProg, "uInSize"), inW, inH) + GLES30.glDrawArrays(GLES30.GL_TRIANGLES, 0, 3) + + // 3. RCAS sharpening into the target + GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, targetFb) + GLES30.glViewport(vx, vy, outW, outH) + GLES30.glUseProgram(rcasProg) + GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, easuTex) + GLES30.glUniform1i(GLES30.glGetUniformLocation(rcasProg, "uTex"), 0) + GLES30.glUniform1f(GLES30.glGetUniformLocation(rcasProg, "uSharp"), sharp) + GLES30.glUniform2i(GLES30.glGetUniformLocation(rcasProg, "uSize"), outW, outH) + GLES30.glUniform2i(GLES30.glGetUniformLocation(rcasProg, "uOrigin"), vx, vy) + GLES30.glDrawArrays(GLES30.GL_TRIANGLES, 0, 3) + GLES30.glBindVertexArray(0) + GLES30.glUseProgram(0) + } + + /** Frees the GL objects; call on the same GL context. */ + fun release() { + free(copyTex, copyFbo); free(easuTex, easuFbo) + copyTex = 0; easuTex = 0; copyW = 0; easuW = 0 + if (copyProg != 0) { GLES30.glDeleteProgram(copyProg); GLES30.glDeleteProgram(easuProg); GLES30.glDeleteProgram(rcasProg); copyProg = 0 } + if (vao[0] != 0) { GLES30.glDeleteVertexArrays(1, vao, 0); vao[0] = 0 } + } + + private fun free(tex: Int, fbo: Int) { + if (tex != 0) GLES30.glDeleteTextures(1, intArrayOf(tex), 0) + if (fbo != 0) GLES30.glDeleteFramebuffers(1, intArrayOf(fbo), 0) + } + + private fun target(w: Int, h: Int): Pair { + val t = IntArray(1); val f = IntArray(1) + GLES30.glGenTextures(1, t, 0) + GLES30.glBindTexture(GLES30.GL_TEXTURE_2D, t[0]) + GLES30.glTexStorage2D(GLES30.GL_TEXTURE_2D, 1, GLES30.GL_RGBA8, w, h) + GLES30.glTexParameteri(GLES30.GL_TEXTURE_2D, GLES30.GL_TEXTURE_MIN_FILTER, GLES30.GL_NEAREST) + GLES30.glTexParameteri(GLES30.GL_TEXTURE_2D, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_NEAREST) + GLES30.glGenFramebuffers(1, f, 0) + GLES30.glBindFramebuffer(GLES30.GL_FRAMEBUFFER, f[0]) + GLES30.glFramebufferTexture2D(GLES30.GL_FRAMEBUFFER, GLES30.GL_COLOR_ATTACHMENT0, GLES30.GL_TEXTURE_2D, t[0], 0) + check(GLES30.glCheckFramebufferStatus(GLES30.GL_FRAMEBUFFER) == GLES30.GL_FRAMEBUFFER_COMPLETE) { "framebuffer ${w}x$h" } + return t[0] to f[0] + } + + private fun program(vs: String, fs: String): Int { + fun shader(type: Int, src: String): Int { + val s = GLES30.glCreateShader(type) + GLES30.glShaderSource(s, src) + GLES30.glCompileShader(s) + val ok = IntArray(1) + GLES30.glGetShaderiv(s, GLES30.GL_COMPILE_STATUS, ok, 0) + check(ok[0] != 0) { "shader: " + GLES30.glGetShaderInfoLog(s) } + return s + } + val p = GLES30.glCreateProgram() + GLES30.glAttachShader(p, shader(GLES30.GL_VERTEX_SHADER, vs)) + GLES30.glAttachShader(p, shader(GLES30.GL_FRAGMENT_SHADER, fs)) + GLES30.glLinkProgram(p) + val ok = IntArray(1) + GLES30.glGetProgramiv(p, GLES30.GL_LINK_STATUS, ok, 0) + check(ok[0] != 0) { "link: " + GLES30.glGetProgramInfoLog(p) } + return p + } + + companion object { + private const val TAG = "MoonCast-FSR" + // a full-screen triangle; no vertex buffers + private const val VERTEX = """#version 300 es +out vec2 vUv; +void main() { + vec2 p = vec2(float((gl_VertexID << 1) & 2), float(gl_VertexID & 2)); + vUv = p; + gl_Position = vec4(p * 2.0 - 1.0, 0.0, 1.0); +}""" + + private const val COPY = """#version 300 es +#extension GL_OES_EGL_image_external_essl3 : require +precision mediump float; +uniform samplerExternalOES uTex; +uniform mat4 uSt; +in vec2 vUv; +out vec4 o; +void main() { o = vec4(texture(uTex, (uSt * vec4(vUv, 0.0, 1.0)).xy).rgb, 1.0); }""" + + // ffx_fsr1.h FsrEasuF (32-bit), with FsrEasuTapF and FsrEasuSetF; textureGather replaced by 4 texelFetches + private const val EASU = """#version 300 es +precision highp float; +precision highp int; +uniform highp sampler2D uTex; +uniform vec4 uCon0; +uniform vec4 uCon1; +uniform vec4 uCon2; +uniform vec4 uCon3; +uniform ivec2 uInSize; +out vec4 o; + +vec3 texel(ivec2 p) { return texelFetch(uTex, clamp(p, ivec2(0), uInSize - 1), 0).rgb; } +// textureGather at a texel corner p: x = (left, upper+1), y = (left+1, upper+1), z = (left+1, upper), w = (left, upper) +void gather(vec2 p, out vec3 x, out vec3 y, out vec3 z, out vec3 w) { + ivec2 tl = ivec2(floor(p * vec2(uInSize) + 0.5)) - 1; + x = texel(tl + ivec2(0, 1)); y = texel(tl + ivec2(1, 1)); z = texel(tl + ivec2(1, 0)); w = texel(tl); +} +float luma(vec3 c) { return c.b * 0.5 + (c.r * 0.5 + c.g); } +float rcp(float x) { return 1.0 / max(x, 1.0e-8); } + +void tap(inout vec3 aC, inout float aW, vec2 off, vec2 dir, vec2 len, float lob, float clp, vec3 c) { + vec2 v; + v.x = (off.x * ( dir.x)) + (off.y * dir.y); + v.y = (off.x * (-dir.y)) + (off.y * dir.x); + v *= len; + float d2 = min(v.x * v.x + v.y * v.y, clp); + float wB = 2.0 / 5.0 * d2 - 1.0; + float wA = lob * d2 - 1.0; + wB *= wB; + wA *= wA; + wB = 25.0 / 16.0 * wB - (25.0 / 16.0 - 1.0); + float w = wB * wA; + aC += c * w; aW += w; +} + +void set(inout vec2 dir, inout float len, vec2 pp, int corner, float lA, float lB, float lC, float lD, float lE) { + float w = 0.0; + if (corner == 0) w = (1.0 - pp.x) * (1.0 - pp.y); + if (corner == 1) w = pp.x * (1.0 - pp.y); + if (corner == 2) w = (1.0 - pp.x) * pp.y; + if (corner == 3) w = pp.x * pp.y; + float dc = lD - lC; + float cb = lC - lB; + float lenX = rcp(max(abs(dc), abs(cb))); + float dirX = lD - lB; + dir.x += dirX * w; + lenX = clamp(abs(dirX) * lenX, 0.0, 1.0); + lenX *= lenX; + len += lenX * w; + float ec = lE - lC; + float ca = lC - lA; + float lenY = rcp(max(abs(ec), abs(ca))); + float dirY = lE - lA; + dir.y += dirY * w; + lenY = clamp(abs(dirY) * lenY, 0.0, 1.0); + lenY *= lenY; + len += lenY * w; +} + +void main() { + vec2 pp = floor(gl_FragCoord.xy) * uCon0.xy + uCon0.zw; + vec2 fp = floor(pp); + pp -= fp; + vec2 p0 = fp * uCon1.xy + uCon1.zw; + vec2 p1 = p0 + uCon2.xy; + vec2 p2 = p0 + uCon2.zw; + vec2 p3 = p0 + uCon3.xy; + vec3 b, c, z0, w0, i, j, f, e, k, l, h, g, x3, y3, oo, n; + gather(p0, b, c, z0, w0); + gather(p1, i, j, f, e); + gather(p2, k, l, h, g); + gather(p3, x3, y3, oo, n); + float bL = luma(b), cL = luma(c), iL = luma(i), jL = luma(j), fL = luma(f), eL = luma(e); + float kL = luma(k), lL = luma(l), hL = luma(h), gL = luma(g), oL = luma(oo), nL = luma(n); + vec2 dir = vec2(0.0); + float len = 0.0; + set(dir, len, pp, 0, bL, eL, fL, gL, jL); + set(dir, len, pp, 1, cL, fL, gL, hL, kL); + set(dir, len, pp, 2, fL, iL, jL, kL, nL); + set(dir, len, pp, 3, gL, jL, kL, lL, oL); + vec2 dir2 = dir * dir; + float dirR = dir2.x + dir2.y; + bool zro = dirR < 1.0 / 32768.0; + dirR = inversesqrt(max(dirR, 1.0e-12)); + dirR = zro ? 1.0 : dirR; + dir.x = zro ? 1.0 : dir.x; + dir *= vec2(dirR); + len = len * 0.5; + len *= len; + float stretch = (dir.x * dir.x + dir.y * dir.y) * rcp(max(abs(dir.x), abs(dir.y))); + vec2 len2 = vec2(1.0 + (stretch - 1.0) * len, 1.0 - 0.5 * len); + float lob = 0.5 + ((1.0 / 4.0 - 0.04) - 0.5) * len; + float clp = rcp(lob); + vec3 min4 = min(min(min(f, g), j), k); + vec3 max4 = max(max(max(f, g), j), k); + vec3 aC = vec3(0.0); + float aW = 0.0; + tap(aC, aW, vec2( 0.0, -1.0) - pp, dir, len2, lob, clp, b); + tap(aC, aW, vec2( 1.0, -1.0) - pp, dir, len2, lob, clp, c); + tap(aC, aW, vec2(-1.0, 1.0) - pp, dir, len2, lob, clp, i); + tap(aC, aW, vec2( 0.0, 1.0) - pp, dir, len2, lob, clp, j); + tap(aC, aW, vec2( 0.0, 0.0) - pp, dir, len2, lob, clp, f); + tap(aC, aW, vec2(-1.0, 0.0) - pp, dir, len2, lob, clp, e); + tap(aC, aW, vec2( 1.0, 1.0) - pp, dir, len2, lob, clp, k); + tap(aC, aW, vec2( 2.0, 1.0) - pp, dir, len2, lob, clp, l); + tap(aC, aW, vec2( 2.0, 0.0) - pp, dir, len2, lob, clp, h); + tap(aC, aW, vec2( 1.0, 0.0) - pp, dir, len2, lob, clp, g); + tap(aC, aW, vec2( 1.0, 2.0) - pp, dir, len2, lob, clp, oo); + tap(aC, aW, vec2( 0.0, 2.0) - pp, dir, len2, lob, clp, n); + o = vec4(min(max4, max(min4, aC * rcp(aW))), 1.0); +}""" + + // ffx_fsr1.h FsrRcasF (32-bit, no denoise) + private const val RCAS = """#version 300 es +precision highp float; +precision highp int; +uniform highp sampler2D uTex; +uniform float uSharp; +uniform ivec2 uSize; +out vec4 o; +uniform ivec2 uOrigin; +vec3 texel(ivec2 p) { return texelFetch(uTex, clamp(p, ivec2(0), uSize - 1), 0).rgb; } +void main() { + ivec2 sp = ivec2(gl_FragCoord.xy) - uOrigin; + vec3 b = texel(sp + ivec2(0, -1)); + vec3 d = texel(sp + ivec2(-1, 0)); + vec3 e = texel(sp); + vec3 f = texel(sp + ivec2(1, 0)); + vec3 h = texel(sp + ivec2(0, 1)); + vec3 mn4 = min(min(b, d), min(f, h)); + vec3 mx4 = max(max(b, d), max(f, h)); + vec3 hitMin = min(mn4, e) / max(4.0 * mx4, vec3(1.0e-5)); + vec3 hitMax = (1.0 - max(mx4, e)) / min(4.0 * mn4 - 4.0, vec3(-1.0e-5)); + vec3 lobeRGB = max(-hitMin, hitMax); + float lobe = max(-(0.25 - 1.0 / 16.0), min(max(lobeRGB.r, max(lobeRGB.g, lobeRGB.b)), 0.0)) * uSharp; + o = vec4((lobe * (b + d + h + f) + e) / (4.0 * lobe + 1.0), 1.0); +}""" + } +} + +/** + * Super resolution for a decoder that renders into a Surface (PS5 via chiaki, PC via MediaCodec): the decoder renders + * into [inputSurface] (a SurfaceTexture) and each frame is drawn to the screen surface through [FsrPasses], on our own + * GL thread. The frame keeps the decoder's presentation time, so frame pacing still applies. The screen surface can + * come and go (a foldable's fold) with [attachOutput]/[detachOutput] while the decoder keeps its own surface. + */ +class FsrRenderer(private val videoWidth: Int, private val videoHeight: Int, sharpness: Float) { + private val thread = HandlerThread("fsr-gl").apply { start() } + private val handler = Handler(thread.looper) + private var display: EGLDisplay = EGL14.EGL_NO_DISPLAY + private var context: EGLContext = EGL14.EGL_NO_CONTEXT + private var config: EGLConfig? = null + private var pbuffer: EGLSurface = EGL14.EGL_NO_SURFACE + private var window: EGLSurface = EGL14.EGL_NO_SURFACE + private var oesTex = 0 + private val passes = FsrPasses(sharpness) + private lateinit var texture: SurfaceTexture + private val stMatrix = FloatArray(16) + @Volatile private var released = false + var frames = 0L + private set + + /** The surface the decoder renders into. */ + lateinit var inputSurface: Surface + private set + + init { + val ready = CountDownLatch(1) + var error: Throwable? = null + handler.post { + try { setUp() } catch (t: Throwable) { error = t } + ready.countDown() + } + ready.await(5, TimeUnit.SECONDS) + error?.let { release(); throw RuntimeException("FSR setup failed: ${it.message}", it) } + if (!this::inputSurface.isInitialized) { release(); throw RuntimeException("FSR setup timed out") } + } + + private fun setUp() { + display = EGL14.eglGetDisplay(EGL14.EGL_DEFAULT_DISPLAY) + val ver = IntArray(2) + check(EGL14.eglInitialize(display, ver, 0, ver, 1)) { "eglInitialize" } + // GPUs differ in what they offer (the JAL TV's Mali had no ES 3 config that does both window and pbuffer): + // ask for the most useful first, then less. Without a pbuffer, a surfaceless context stands in for it. + val es3 = EGLExt.EGL_OPENGL_ES3_BIT_KHR + val es2 = EGL14.EGL_OPENGL_ES2_BIT + val both = EGL14.EGL_WINDOW_BIT or EGL14.EGL_PBUFFER_BIT + val tries = listOf(es3 to both, es3 to EGL14.EGL_WINDOW_BIT, es2 to both, es2 to EGL14.EGL_WINDOW_BIT) + val configs = arrayOfNulls(1) + val n = IntArray(1) + var surfaces = 0 + for ((renderable, surfaceType) in tries) { + val attrs = intArrayOf(EGL14.EGL_RED_SIZE, 8, EGL14.EGL_GREEN_SIZE, 8, EGL14.EGL_BLUE_SIZE, 8, + EGL14.EGL_RENDERABLE_TYPE, renderable, EGL14.EGL_SURFACE_TYPE, surfaceType, EGL14.EGL_NONE) + if (EGL14.eglChooseConfig(display, attrs, 0, configs, 0, 1, n, 0) && n[0] > 0) { config = configs[0]; surfaces = surfaceType; break } + } + check(config != null) { "eglChooseConfig: no RGB888 window config (${EGL14.eglGetError()})" } + context = EGL14.eglCreateContext(display, config, EGL14.EGL_NO_CONTEXT, intArrayOf(EGL14.EGL_CONTEXT_CLIENT_VERSION, 3, EGL14.EGL_NONE), 0) + check(context != EGL14.EGL_NO_CONTEXT) { "eglCreateContext ES 3 (${EGL14.eglGetError()})" } + if (surfaces and EGL14.EGL_PBUFFER_BIT != 0) + pbuffer = EGL14.eglCreatePbufferSurface(display, config, intArrayOf(EGL14.EGL_WIDTH, 1, EGL14.EGL_HEIGHT, 1, EGL14.EGL_NONE), 0) + check(EGL14.eglMakeCurrent(display, pbuffer, pbuffer, context)) { "eglMakeCurrent ${if (pbuffer == EGL14.EGL_NO_SURFACE) "surfaceless" else "pbuffer"} (${EGL14.eglGetError()})" } + Log.i(TAG, "EGL: ${if (surfaces and EGL14.EGL_PBUFFER_BIT != 0) "pbuffer" else "surfaceless"}, GL ${GLES30.glGetString(GLES30.GL_VERSION)}") + val t = IntArray(1) + GLES30.glGenTextures(1, t, 0); oesTex = t[0] + GLES30.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, oesTex) + GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MIN_FILTER, GLES30.GL_LINEAR) + GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_MAG_FILTER, GLES30.GL_LINEAR) + GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_S, GLES30.GL_CLAMP_TO_EDGE) + GLES30.glTexParameteri(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, GLES30.GL_TEXTURE_WRAP_T, GLES30.GL_CLAMP_TO_EDGE) + // compile the shaders now: a GPU that can't run them fails here, and the caller shows the stream directly + passes.draw(oesTex, IDENTITY, 2, 2, 0, 0, 0, 2, 2) + texture = SurfaceTexture(oesTex).apply { + setDefaultBufferSize(videoWidth, videoHeight) + setOnFrameAvailableListener({ draw() }, handler) + } + inputSurface = Surface(texture) + Log.i(TAG, "ready: ${videoWidth}x$videoHeight, GL ${GLES30.glGetString(GLES30.GL_RENDERER)}") + } + + /** Draw to this screen surface from now on (blocks until it's in use). */ + fun attachOutput(surface: Surface) = onGl { + destroyWindow() + window = EGL14.eglCreateWindowSurface(display, config, surface, intArrayOf(EGL14.EGL_NONE), 0) + if (window == EGL14.EGL_NO_SURFACE) Log.w(TAG, "eglCreateWindowSurface failed: ${EGL14.eglGetError()}") + } + + /** Stop drawing to the screen surface before it goes away (blocks); frames are dropped until [attachOutput]. */ + fun detachOutput() = onGl { destroyWindow() } + + private fun destroyWindow() { + if (window != EGL14.EGL_NO_SURFACE) { + EGL14.eglMakeCurrent(display, pbuffer, pbuffer, context) + EGL14.eglDestroySurface(display, window) + window = EGL14.EGL_NO_SURFACE + } + } + + private fun draw() { + if (released) return + if (window == EGL14.EGL_NO_SURFACE) { + // nowhere to show it: take the frame anyway so the decoder doesn't stall + EGL14.eglMakeCurrent(display, pbuffer, pbuffer, context) + texture.updateTexImage() + return + } + if (!EGL14.eglMakeCurrent(display, window, window, context)) { Log.w(TAG, "eglMakeCurrent: ${EGL14.eglGetError()}"); return } + texture.updateTexImage() + texture.getTransformMatrix(stMatrix) + val q = IntArray(1) + EGL14.eglQuerySurface(display, window, EGL14.EGL_WIDTH, q, 0); val outW = q[0] + EGL14.eglQuerySurface(display, window, EGL14.EGL_HEIGHT, q, 0); val outH = q[0] + passes.draw(oesTex, stMatrix, videoWidth, videoHeight, 0, 0, 0, outW, outH) + // the decoder's presentation time: frame pacing keeps working + EGLExt.eglPresentationTimeANDROID(display, window, texture.timestamp) + if (!EGL14.eglSwapBuffers(display, window)) Log.w(TAG, "eglSwapBuffers: ${EGL14.eglGetError()}") + frames++ + } + + /** End it (blocks briefly). Call after the decoder has let go of [inputSurface]. */ + fun release() { + if (released) return + released = true + onGl { + destroyWindow() + if (display != EGL14.EGL_NO_DISPLAY && context != EGL14.EGL_NO_CONTEXT) { + EGL14.eglMakeCurrent(display, pbuffer, pbuffer, context) + passes.release() + } + if (this::inputSurface.isInitialized) inputSurface.release() + if (this::texture.isInitialized) texture.release() + if (display != EGL14.EGL_NO_DISPLAY) { + EGL14.eglMakeCurrent(display, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_SURFACE, EGL14.EGL_NO_CONTEXT) + if (pbuffer != EGL14.EGL_NO_SURFACE) EGL14.eglDestroySurface(display, pbuffer) + if (context != EGL14.EGL_NO_CONTEXT) EGL14.eglDestroyContext(display, context) + EGL14.eglTerminate(display) + } + display = EGL14.EGL_NO_DISPLAY + } + thread.quitSafely() + } + + private fun onGl(block: () -> Unit) { + if (Thread.currentThread() === thread) { block(); return } + val done = CountDownLatch(1) + if (!handler.post { try { block() } finally { done.countDown() } }) return + done.await(2, TimeUnit.SECONDS) + } + + companion object { + private const val TAG = "MoonCast-FSR" + + /** Settings: Super resolution (off / fsr1) and its Sharpening (0..100). */ + const val PREF = "mc_super_resolution" + const val SHARPNESS_PREF = "mc_fsr_sharpness" + const val DEFAULT_SHARPNESS = 60 + + private val IDENTITY = floatArrayOf(1f, 0f, 0f, 0f, 0f, 1f, 0f, 0f, 0f, 0f, 1f, 0f, 0f, 0f, 0f, 1f) + + /** Worth it only when the stream is clearly smaller than the screen surface. */ + fun worthIt(videoHeight: Int, outHeight: Int) = outHeight > videoHeight * 1.1 + + /** Settings, Picture, Super resolution is FSR 1.0. */ + fun enabled(prefs: android.content.SharedPreferences) = prefs.getString(PREF, "off") == "fsr1" + + /** Settings, Picture, Sharpening as 0..1. */ + fun sharpness(prefs: android.content.SharedPreferences) = prefs.getInt(SHARPNESS_PREF, DEFAULT_SHARPNESS) / 100f + } +} + +/** + * Super resolution for Xbox: WebRTC's SurfaceViewRenderer draws each frame through this GlDrawer, on its own GL thread. + * Hardware-decoded frames (OES) take the FSR passes when they're clearly smaller than the view; everything else, and + * any GPU without OpenGL ES 3, goes to WebRTC's own drawer. + */ +class FsrGlDrawer(sharpness: Float) : org.webrtc.RendererCommon.GlDrawer { + private val plain = org.webrtc.GlRectDrawer() + private val passes = FsrPasses(sharpness) + private var es3: Boolean? = null + private var logged = false + + override fun drawOes(oesTextureId: Int, texMatrix: FloatArray, frameWidth: Int, frameHeight: Int, + viewportX: Int, viewportY: Int, viewportWidth: Int, viewportHeight: Int) { + if (es3 == null) { + val v = GLES30.glGetString(GLES30.GL_VERSION) ?: "" + es3 = Regex("OpenGL ES (\\d+)").find(v)?.groupValues?.get(1)?.toIntOrNull()?.let { it >= 3 } ?: false + Log.i("MoonCast-FSR", "Xbox drawer: $v -> ${if (es3 == true) "FSR" else "plain"}") + } + if (es3 == true && FsrRenderer.worthIt(frameHeight, viewportHeight)) { + try { + if (!logged) { Log.i("MoonCast-FSR", "Xbox: ${frameWidth}x$frameHeight to ${viewportWidth}x$viewportHeight"); logged = true } + passes.draw(oesTextureId, texMatrix, frameWidth, frameHeight, 0, viewportX, viewportY, viewportWidth, viewportHeight) + return + } catch (e: RuntimeException) { + Log.w("MoonCast-FSR", "Xbox FSR failed, drawing plain: $e"); es3 = false + } + } + plain.drawOes(oesTextureId, texMatrix, frameWidth, frameHeight, viewportX, viewportY, viewportWidth, viewportHeight) + } + + override fun drawRgb(textureId: Int, texMatrix: FloatArray, frameWidth: Int, frameHeight: Int, + viewportX: Int, viewportY: Int, viewportWidth: Int, viewportHeight: Int) = + plain.drawRgb(textureId, texMatrix, frameWidth, frameHeight, viewportX, viewportY, viewportWidth, viewportHeight) + + override fun drawYuv(yuvTextures: IntArray, texMatrix: FloatArray, frameWidth: Int, frameHeight: Int, + viewportX: Int, viewportY: Int, viewportWidth: Int, viewportHeight: Int) = + plain.drawYuv(yuvTextures, texMatrix, frameWidth, frameHeight, viewportX, viewportY, viewportWidth, viewportHeight) + + override fun release() { + plain.release() + if (es3 == true) try { passes.release() } catch (e: RuntimeException) { } + } +} diff --git a/app/src/main/java/com/limelight/binding/video/MediaCodecDecoderRenderer.java b/app/src/main/java/com/limelight/binding/video/MediaCodecDecoderRenderer.java index e3a910b1d6..6400927809 100644 --- a/app/src/main/java/com/limelight/binding/video/MediaCodecDecoderRenderer.java +++ b/app/src/main/java/com/limelight/binding/video/MediaCodecDecoderRenderer.java @@ -273,6 +273,13 @@ public void setRenderTarget(SurfaceHolder renderTarget) { this.renderTarget = renderTarget; } + // MoonCast: Super resolution. The decoder renders into FSR's surface instead of the screen's + private android.view.Surface surfaceOverride; + + public void setSurfaceOverride(android.view.Surface surface) { + this.surfaceOverride = surface; + } + public MediaCodecDecoderRenderer(Activity activity, PreferenceConfiguration prefs, CrashListener crashListener, int consecutiveCrashCount, boolean meteredData, boolean requestedHdr, @@ -516,7 +523,7 @@ else if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) { LimeLog.info("Configuring with format: "+format); - videoDecoder.configure(format, renderTarget.getSurface(), null, 0); + videoDecoder.configure(format, surfaceOverride != null ? surfaceOverride : renderTarget.getSurface(), null, 0); configuredFormat = format; diff --git a/app/src/main/java/com/limelight/home/Licences.kt b/app/src/main/java/com/limelight/home/Licences.kt index d3687a976d..e8cbcd99e2 100644 --- a/app/src/main/java/com/limelight/home/Licences.kt +++ b/app/src/main/java/com/limelight/home/Licences.kt @@ -23,6 +23,7 @@ object Licences { val parts = listOf( Part("The app", "Moonlight for Android", "The app MoonCast grew from: PC streaming, the on-screen controller", "GPL-3.0", "gpl-3.0.txt", "https://github.com/moonlight-stream/moonlight-android"), Part("The app", "moonlight-common-c", "The PC streaming protocol", "GPL-3.0", "gpl-3.0.txt", "https://github.com/moonlight-stream/moonlight-common-c"), + Part("The app", "AMD FidelityFX Super Resolution 1.0", "Super resolution: the upscaling and sharpening (FSR 1)", "MIT", "fsr1.txt", "https://github.com/GPUOpen-Effects/FidelityFX-FSR"), Part("The app", "ENet", "Inside moonlight-common-c: the network link", "MIT", "enet.txt", "https://github.com/lsalzman/enet"), Part("The app", "nanors", "Inside moonlight-common-c: repairing lost video", "MIT", "nanors.txt", "https://github.com/sleepybishop/nanors"), Part("PS5", "chiaki-ng", "PS5 Remote Play, with MoonCast's changes (MoonCastDev/chiaki-ng-mooncast)", "AGPL-3.0", "agpl-3.0.txt", "https://github.com/streetpea/chiaki-ng"), diff --git a/app/src/tv/java/com/limelight/home/HomeActivity.kt b/app/src/tv/java/com/limelight/home/HomeActivity.kt index b02a0211c4..433b2419c8 100644 --- a/app/src/tv/java/com/limelight/home/HomeActivity.kt +++ b/app/src/tv/java/com/limelight/home/HomeActivity.kt @@ -983,6 +983,12 @@ class HomeActivity : Activity(), PhoneLink.Listener { val customRate = kbps != 0 && kbps / 1000 !in listOf(10, 20, 30, 50, 80) action(list, "set:customrate", "Custom bitrate", "Any value between the steps above, in Mbps", if (customRate) "${kbps / 1000} Mbps" else "Off") { customBitrate(kbps) } + choice(list, com.limelight.binding.video.FsrRenderer.PREF, "Super resolution", + "Sharper picture when the stream is smaller than this TV (AMD FSR 1.0), on PC, PS5 and Xbox. Not with HDR. Uses a little of the GPU", + listOf("Off" to "off", "FSR 1.0" to "fsr1"), "off") + if (stream.getString(com.limelight.binding.video.FsrRenderer.PREF, "off") == "fsr1") + intChoice(list, com.limelight.binding.video.FsrRenderer.SHARPNESS_PREF, "Sharpening", "How crisp FSR makes edges. Lower it if you see halos", + listOf(0, 20, 40, 60, 80, 100), com.limelight.binding.video.FsrRenderer.DEFAULT_SHARPNESS) { "$it%" } choice(list, "video_format", "Video codec", "H.265 and AV1 look better at the same bitrate", listOf("Auto" to "auto", "H.264" to "neverh265", "H.265" to "forceh265", "AV1" to "forceav1"), "auto") toggle(list, "checkbox_enable_hdr", "HDR", "Only when the PC, the game and this TV support it", false) diff --git a/docs/mockups/fsr/compare.html b/docs/mockups/fsr/compare.html index 5651f73d1f..6778bca861 100644 --- a/docs/mockups/fsr/compare.html +++ b/docs/mockups/fsr/compare.html @@ -6,16 +6,17 @@ .pane{position:absolute;top:150px;width:944px;height:944px;border-radius:14px;overflow:hidden;background:#000} .l{left:30px}.r{left:994px} .lab{position:absolute;bottom:18px;left:18px;font-size:28px;font-weight:800;background:rgba(0,0,0,.65);padding:8px 16px;border-radius:10px} -canvas{display:block} .foot{position:absolute;bottom:40px;left:0;right:0;text-align:center;font-size:22px;color:#B5BAC3} +#status{position:absolute;top:120px;left:30px;font-size:18px;color:#f88} +canvas{display:block}
Super resolution: a 720p stream on a 2448 × 1696 screen
The same frame, enlarged 3.4×; a close-up of the middle of the picture
Off: plain enlargement
FSR 1.0, sharpening 60%
-
Illustration: plain enlargement plus FSR's own sharpening pass (RCAS). Real FSR also upscales edges smarter (EASU), so it's crisper still.
+
Rendered by MoonCast's own FSR 1.0 shaders (EASU + RCAS, the code in the app), run here in WebGL 2.
+
+ \ No newline at end of file diff --git a/docs/mockups/fsr/fsr_compare.png b/docs/mockups/fsr/fsr_compare.png index c00917d3c2..030451f3a4 100644 Binary files a/docs/mockups/fsr/fsr_compare.png and b/docs/mockups/fsr/fsr_compare.png differ diff --git a/third-party/chiaki-ng b/third-party/chiaki-ng index 5d5411183d..c5fa7fad63 160000 --- a/third-party/chiaki-ng +++ b/third-party/chiaki-ng @@ -1 +1 @@ -Subproject commit 5d5411183dca913725d2d0b8210660e058bf01dc +Subproject commit c5fa7fad63d56e08fc503c3b746bec4824dfaabc