feat: Super resolution (AMD FSR 1.0) with Sharpening, on PC, PS5 and Xbox

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## 1.0-test236 (2026-10-21)
+- Found on the JAL TV (AQUOS, Android 11, MediaTek with a Mali GPU), PS5 over PlayStation Network at 720p:
+  - Super resolution didn't start there ("eglChooseConfig"): the GPU has no OpenGL ES 3 setup for both a window and an off-screen surface. FSR now tries several setups, and without an off-screen surface it works surfaceless. Before, it fell back to the plain picture, as designed.
+  - PS5 decoder: the TV's decoder (OMX.MS.HEVC.Decoder) refused the low-latency settings, and MoonCast dropped all of them. Now it drops them one step at a time (vendor settings, then MediaTek's low-latency decode, then Android's standard low-latency mode, then real-time priority), so a decoder keeps as many as it takes. The log says which level was used.
+- The same JAL TV test passed: PS5 over PSN on OpenSSL 3 (connected in 84 s including waking the console), 60 fps, 25 ms network delay, no lost packets, Decode time 20.5 ms, and Put PS5 to rest from the Quick menu.
+
+## 1.0-test235 (2026-10-21)
+- Super resolution now works on PC and Xbox streams too, not only PS5. The FSR passes are shared (`FsrPasses`), used by:
+  - PC: Moonlight's decoder renders into FSR's surface. FSR stays off with HDR, since its passes are 8-bit. The screen surface keeps the screen's size instead of the stream's, so FSR has room to upscale into; with Stretch, the picture is stretched by FSR.
+  - Xbox: WebRTC's renderer draws each frame through FSR's drawer (`FsrGlDrawer`), with WebRTC's hardware scaler off so the surface keeps the view's size. Frames that aren't hardware-decoded, and GPUs without OpenGL ES 3, are drawn by WebRTC's own drawer, as before.
+  - PS5: as in test234.
+- In all three, FSR only takes effect when the stream is clearly smaller than the screen; otherwise the picture shows as before.
+
+## 1.0-test234 (2026-10-21)
+- Super resolution (Settings, Picture, both apps): AMD FidelityFX Super Resolution 1.0, for the PS5 first (mockups in docs/mockups/fsr). With it on, and the stream clearly smaller than the screen (a PS5's 1080p or 720p on the Fold's 2448 × 1696, say), each frame takes three GPU passes before it shows:
+  - a copy of the decoder's frame at the stream's size;
+  - EASU, FSR's edge-adaptive upscale to the screen's size;
+  - RCAS, FSR's sharpening, at the new Sharpening setting (0 to 100%, 60% by default).
+- How it fits in:
+  - The shaders are ports of AMD's ffx_fsr1.h (MIT) to OpenGL ES 3.0. Its 4-texel gathers are 4 texel reads, as ES 3.0 has no textureGather.
+  - Frames keep the decoder's presentation time, so Frame pacing still works.
+  - On a foldable, a fold only moves FSR's output; the decoder keeps drawing into FSR.
+  - Off by default. When it's not worth it (the stream is as big as the screen) or the GPU can't set it up, the stream shows directly, as before.
+- About, Open-source licences: AMD FidelityFX Super Resolution 1.0 (MIT).
+
 ## 1.0-test233 (2026-10-21)
 - PS5: frame pacing (Settings, PlayStation, Frame pacing: Lowest delay, Balanced, Smoothest), like the PC's.
   - Lowest delay (the default, as before): each frame shows the moment it's decoded, so frames that arrive unevenly over Wi-Fi show unevenly.

NOTICE

@@ -16,6 +16,7 @@ The app
 -------
 Moonlight for Android (GPL-3.0): The app MoonCast grew from: PC streaming, the on-screen controller. https://github.com/moonlight-stream/moonlight-android  [licences/gpl-3.0.txt]
 moonlight-common-c (GPL-3.0): The PC streaming protocol. https://github.com/moonlight-stream/moonlight-common-c  [licences/gpl-3.0.txt]
+AMD FidelityFX Super Resolution 1.0 (MIT): Super resolution: the upscaling and sharpening (FSR 1). https://github.com/GPUOpen-Effects/FidelityFX-FSR  [licences/fsr1.txt]
 ENet (MIT): Inside moonlight-common-c: the network link. https://github.com/lsalzman/enet  [licences/enet.txt]
 nanors (MIT): Inside moonlight-common-c: repairing lost video. https://github.com/sleepybishop/nanors  [licences/nanors.txt]
 

app/src/handheld/java/com/limelight/handheld/HandheldHomeActivity.kt

@@ -1254,6 +1254,13 @@ class HandheldHomeActivity : Activity() {
                         else { stream.edit().putInt("seekbar_bitrate_kbps", (Math.round(v * 2) * 500).toInt()).apply(); render(force = true); null }
                     }
                 }
+                // Super resolution: AMD FSR 1.0 upscales a stream smaller than the screen, then sharpens it (PS5 for now)
+                choice(list, stream, com.limelight.binding.video.FsrRenderer.PREF, "Super resolution",
+                        "Sharper picture when the stream is smaller than this screen (AMD FSR 1.0), on PC, PS5 and Xbox. Not with HDR. Uses a little of the GPU",
+                        listOf("Off" to "off", "FSR 1.0" to "fsr1"), "off")
+                if (stream.getString(com.limelight.binding.video.FsrRenderer.PREF, "off") == "fsr1")
+                    intChoice(list, com.limelight.binding.video.FsrRenderer.SHARPNESS_PREF, "Sharpening", "How crisp FSR makes edges. Lower it if you see halos",
+                            listOf(0, 20, 40, 60, 80, 100), com.limelight.binding.video.FsrRenderer.DEFAULT_SHARPNESS) { "$it%" }
                 choice(list, stream, "video_format", "Video codec", "H.265 and AV1 look better at the same bitrate",
                         listOf("Auto" to "auto", "H.264" to "neverh265", "H.265" to "forceh265", "AV1" to "forceav1"), "auto")
                 toggle(list, "checkbox_enable_hdr", "HDR", "Only when the PC, the game and this screen support it", false)

app/src/main/assets/licences/fsr1.txt

@@ -0,0 +1,19 @@
+Copyright (c) 2021 Advanced Micro Devices, Inc. All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

app/src/main/java/com/limelight/Game.java

@@ -125,6 +125,10 @@ public class Game extends Activity implements SurfaceHolder.Callback,
     private boolean connected = false;
     private boolean autoEnterPip = false;
     private boolean surfaceCreated = false;
+    // MoonCast: Super resolution (FSR 1.0)
+    private boolean fsrPlanned = false;
+    private float fsrSharpness = 0.6f;
+    private volatile com.limelight.binding.video.FsrRenderer fsr;
     private boolean attemptedConnection = false;
     private int suppressPipRefCount = 0;
     private String pcName;
@@ -405,6 +409,21 @@ public void notifyCrash(Exception e) {
             Toast.makeText(this, "Decoder does not support HDR10 profile", Toast.LENGTH_LONG).show();
         }
 
+        // MoonCast: Super resolution (FSR 1.0) when it's on, the stream isn't HDR (FSR's passes are 8-bit) and the
+        // stream is clearly smaller than this screen. The surface then keeps the screen's size (no setFixedSize)
+        // so FSR has room to upscale into.
+        {
+            android.content.SharedPreferences sp = android.preference.PreferenceManager.getDefaultSharedPreferences(this);
+            android.graphics.Point real = new android.graphics.Point();
+            getWindowManager().getDefaultDisplay().getRealSize(real);
+            fsrPlanned = com.limelight.binding.video.FsrRenderer.Companion.enabled(sp) && !willStreamHdr
+                    && com.limelight.binding.video.FsrRenderer.Companion.worthIt(prefConfig.height, Math.min(real.x, real.y));
+            fsrSharpness = com.limelight.binding.video.FsrRenderer.Companion.sharpness(sp);
+            if (com.limelight.binding.video.FsrRenderer.Companion.enabled(sp))
+                LimeLog.info("FSR " + (fsrPlanned ? "on" : "not needed") + ": stream " + prefConfig.width + "x" + prefConfig.height
+                        + ", screen " + real.x + "x" + real.y + (willStreamHdr ? ", HDR" : ""));
+        }
+
         // Display a message to the user if HEVC was forced on but we still didn't find a decoder
         if (prefConfig.videoFormat == PreferenceConfiguration.FormatOption.FORCE_HEVC && !decoderRenderer.isHevcSupported()) {
             Toast.makeText(this, "No HEVC decoder found", Toast.LENGTH_LONG).show();
@@ -972,15 +991,16 @@ else if (!isRefreshRateGoodMatch(candidate.getRefreshRate())) {
             }
         }
 
-        if (prefConfig.stretchVideo || aspectRatioMatch) {
+        if (!fsrPlanned && (prefConfig.stretchVideo || aspectRatioMatch)) {
             // Set the surface to the size of the video
             streamView.getHolder().setFixedSize(prefConfig.width, prefConfig.height);
         }
-        else {
+        else if (!(fsrPlanned && prefConfig.stretchVideo)) {
             // Set the surface to scale based on the aspect ratio of the stream
             streamView.setDesiredAspectRatio((double)prefConfig.width / (double)prefConfig.height);
             streamView.setFill(prefConfig.fillVideo);
         }
+        // (FSR with Stretch: the view fills the screen and FSR stretches the picture into it)
 
         // MoonCast: Settings, Video, Zoom. Enlarges the picture for TVs that crop the edges (overscan).
         if (prefConfig.videoZoomPercent > 100) {
@@ -2411,6 +2431,10 @@ private void stopConnection(final Runnable done) {
             new Thread() {
                 public void run() {
                     conn.stop();
+                    // the decoder is gone: FSR's surface can go too
+                    com.limelight.binding.video.FsrRenderer f = fsr;
+                    fsr = null;
+                    if (f != null) f.release();
                     if (done != null) runOnUiThread(done);
                 }
             }.start();
@@ -2723,6 +2747,17 @@ public void surfaceChanged(SurfaceHolder holder, int format, int width, int heig
             UiHelper.notifyStreamConnecting(Game.this);
 
             decoderRenderer.setRenderTarget(holder);
+            if (fsrPlanned && fsr == null) {
+                try {
+                    fsr = new com.limelight.binding.video.FsrRenderer(prefConfig.width, prefConfig.height, fsrSharpness);
+                    fsr.attachOutput(holder.getSurface());
+                    decoderRenderer.setSurfaceOverride(fsr.getInputSurface());
+                    LimeLog.info("FSR on: " + prefConfig.width + "x" + prefConfig.height + " to " + width + "x" + height);
+                } catch (RuntimeException e) {
+                    LimeLog.warning("FSR unavailable, showing the stream directly: " + e);
+                    fsr = null;
+                }
+            }
             conn.start(new AndroidAudioRenderer(Game.this, prefConfig.enableAudioFx),
                     decoderRenderer, Game.this);
         }
@@ -2775,6 +2810,9 @@ public void surfaceDestroyed(SurfaceHolder holder) {
             throw new IllegalStateException("Surface destroyed before creation!");
         }
 
+        // FSR stops drawing to the screen surface before it goes
+        if (fsr != null) fsr.detachOutput();
+
         if (attemptedConnection) {
             // Let the decoder know immediately that the surface is gone
             decoderRenderer.prepareForStop();

app/src/main/java/com/limelight/Ps5StreamActivity.kt

@@ -183,6 +183,7 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
         val profile = if (away) preset.copy(bitrate = awayKbps ?: minOf(preset.bitrate, linkCap)) else preset.copy(bitrate = homeKbps)
         streamInfo = Triple(when (resPreset) { VideoResolutionPreset.RES_540P -> "540p"; VideoResolutionPreset.RES_720P -> "720p"; else -> "1080p" }, if (h264) "H.264" else "H.265", profile.bitrate / 1000)
         streamRemote = remote
+        videoW = profile.width; videoH = profile.height
         Log.i(TAG, "stream ${streamInfo.first}${profile.maxFPS} at ${profile.bitrate / 1000} Mbps (${if (remote) "PSN" else "home"}, ${if (away) "away" else "home"} profile, link ${if (link > 0) "$link Mbps" else "cable/unknown"})")
         if (!remote) return ConnectInfo(ps5, host, registKey, morning, profile)
         val account = intent.getStringExtra(EXTRA_ACCOUNT_ID)?.let {
@@ -270,7 +271,9 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
     } catch (e: Exception) { false }
 
     override fun surfaceCreated(holder: SurfaceHolder) {
-        if (started) { session?.setSurface(holder.surface); return }
+        screenHolder = holder
+        // with FSR the decoder keeps its own surface: only FSR's output moves to the new screen surface
+        if (started) { val f = fsr; if (f != null) f.attachOutput(holder.surface) else session?.setSurface(holder.surface); return }
         started = true
         val surface = holder.surface
         val previous = connectThread
@@ -334,7 +337,7 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
                 val s = try { Session(connectInfo(remote), null, false) } finally { watch?.interrupt() }
                 if (closed) { try { s.dispose() } catch (e: Exception) {}; return }   // the screen closed during the hole-punch
                 s.eventCallback = { ev -> onEvent(ev) }
-                s.setSurface(surface)
+                s.setSurface(decoderSurface(surface))
                 // Only the first picture counts as success; a quit before that is a failed attempt -> retry.
                 // Remote: the data-channel hole-punch may fail. Home: the console may still be waking up.
                 // Armed before start() (#38): a console still waking refuses within milliseconds, often before
@@ -740,6 +743,35 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
     private val menu by lazy { com.limelight.home.QuickMenu(this) }
     private var comboPending = false
     @Volatile private var streamInfo = Triple("1080p", "H.265", 0)
+
+    // ---- Super resolution (FSR 1.0): the decoder renders into FSR's surface, FSR upscales to the screen's
+    @Volatile private var fsr: com.limelight.binding.video.FsrRenderer? = null
+    @Volatile private var screenHolder: SurfaceHolder? = null
+    @Volatile private var videoW = 1920
+    @Volatile private var videoH = 1080
+
+    /** The surface the decoder renders into: the screen's, or FSR's when it's on and the stream is smaller than the screen. */
+    private fun decoderSurface(screen: android.view.Surface): android.view.Surface {
+        val prefs = android.preference.PreferenceManager.getDefaultSharedPreferences(this)
+        if (prefs.getString(com.limelight.binding.video.FsrRenderer.PREF, "off") != "fsr1") return screen
+        val outH = screenHolder?.surfaceFrame?.height() ?: 0
+        if (!com.limelight.binding.video.FsrRenderer.worthIt(videoH, outH)) {
+            Log.i(TAG, "FSR not needed: stream ${videoW}x$videoH, screen ${screenHolder?.surfaceFrame?.width()}x$outH")
+            return screen
+        }
+        fsr?.let { return it.inputSurface }
+        return try {
+            val sharp = prefs.getInt(com.limelight.binding.video.FsrRenderer.SHARPNESS_PREF, com.limelight.binding.video.FsrRenderer.DEFAULT_SHARPNESS)
+            val f = com.limelight.binding.video.FsrRenderer(videoW, videoH, sharp / 100f)
+            f.attachOutput(screen)
+            fsr = f
+            Log.i(TAG, "FSR on: ${videoW}x$videoH to ${screenHolder?.surfaceFrame?.width()}x$outH, sharpening $sharp%")
+            f.inputSurface
+        } catch (e: Exception) {
+            Log.w(TAG, "FSR unavailable, showing the stream directly: $e")
+            screen
+        }
+    }
     @Volatile private var streamRemote = false
     private val startedAt = System.currentTimeMillis()
     private val swapCrossCircle by lazy {
@@ -897,12 +929,14 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
     // into a dead surface fails, and chiaki then crashed copying a frame into a missing buffer. The release runs on a
     // worker (it takes chiaki's decoder lock) and the wait is bounded, so a stuck session can't freeze the app.
     override fun surfaceDestroyed(holder: SurfaceHolder) {
+        fsr?.detachOutput()   // FSR stops drawing to the screen surface before it goes; the decoder keeps FSR's
         if (closed) {
             if (session != null && teardownThread == null) teardown(null)   // the PS5 quit: tear down now, not in onDestroy
             try { teardownThread?.join(SURFACE_RELEASE_WAIT_MS) } catch (e: InterruptedException) { }
             return
         }
         val s = session ?: return
+        if (fsr != null) return   // the decoder draws into FSR's surface, which stays
         val t = Thread({ synchronized(sessionLock) { if (!closed && synchronized(disposed) { s !in disposed }) try { s.setSurface(null) } catch (e: Exception) { } } },
             "ps5-surface")
         t.start()
@@ -937,6 +971,8 @@ class Ps5StreamActivity : Activity(), SurfaceHolder.Callback, com.limelight.home
                 try { s.stop() } catch (e: Exception) { }
                 synchronized(sessionLock) { try { s.dispose() } catch (e: Exception) { } }
             }
+            // after the decoder has let go of its surface
+            fsr?.release(); fsr = null
             if (done != null) android.os.Handler(android.os.Looper.getMainLooper()).post(done)
         }, "ps5-teardown")
         teardownThread = t

app/src/main/java/com/limelight/XboxStreamActivity.kt

@@ -71,13 +71,18 @@ class XboxStreamActivity : Activity(), com.limelight.home.MoveToTv.LocalSession
         com.limelight.binding.video.MediaCodecHelper.initialize(this,
                 com.limelight.preferences.GlPreferences.readPreferences(this).glRenderer)
         val r = SurfaceViewRenderer(this).also { renderer = it }
+        // Super resolution (FSR 1.0): WebRTC's renderer draws each frame through FSR's drawer. The surface then keeps
+        // the view's real size (no hardware scaler, which would shrink it to the frame's) so FSR has room to upscale
+        val fsrOn = com.limelight.binding.video.FsrRenderer.enabled(prefs)
         r.init(egl.eglBaseContext, object : RendererCommon.RendererEvents {
             override fun onFirstFrameRendered() { firstFrame = true; main.removeCallbacks(pictureWatchdog); showStatus(null); com.limelight.home.StreamReport.playing() }
             override fun onFrameResolutionChanged(w: Int, h: Int, rotation: Int) {
                 Log.i(TAG, "picture ${w}x$h"); runOnUiThread { applyShape(w, h) }
             }
-        })
-        r.setEnableHardwareScaler(true)
+        }, org.webrtc.EglBase.CONFIG_PLAIN,
+            if (fsrOn) com.limelight.binding.video.FsrGlDrawer(com.limelight.binding.video.FsrRenderer.sharpness(prefs)) else org.webrtc.GlRectDrawer())
+        r.setEnableHardwareScaler(!fsrOn)
+        if (fsrOn) Log.i(TAG, "FSR drawer on")
         applyShape(0, 0)
         status = TextView(this).apply {
             setTextColor(0xFFE8ECFF.toInt()); textSize = 22f; gravity = Gravity.CENTER