Skip to content

RFI recovery frames - #276

Merged
hgaiser merged 2 commits into
hgaiser:mainfrom
porkloin:rfi-recovery-frames
Nov 18, 2026
Merged

hgaiser merged 2 commits into
hgaiser:mainfrom
porkloin:rfi-recovery-frames

Conversation

@porkloin

Copy link
Copy Markdown
Contributor

Answer Moonlight's reference frame invalidation with recovery frames instead of an IDR whenever the encoder still holds a good reference.

  • Skip an RFI request already covered by an earlier answer.
  • Send the recovering frame as frame type 5, so the client knows where decoding restarts.
  • If the screen is static, send the pending recovery frame anyway; if invalidation can't recover, fall back to an IDR.
  • Hold a 150 ms window of references (4 to 16, from the stream's fps) when the client leaves maxNumReferenceFrames to the host, but predict from one (with_active_reference_frames(1)) so encode time doesn't grow.

Depends on hgaiser/pixelforge#62 (invalidate_reference_frames returns bool, with_active_reference_frames, is_recovery_frame).

Testing

  • Unit tests pass.
  • Tested with moonshine-test-headless on a 7900 XTX (RADV), loopback, 1080p120 at 28 Mb/s, one video frame dropped every 120 and the video socket delayed 40 ms (for RFI timing this behaves like a real round trip), against current main (de05168, with pixelforge main 017ffbf):
    • HEVC: all 18 drops per stream recovered with recovery frames and 1 keyframe was sent, against 127 keyframes on main. 0 corrupt pictures.
    • AV1: 0 recovery frames and 19 keyframes (main: 127), see Known limits.
    • H.264: unchanged (Moonlight doesn't send RFI for it).
    • No speed regression on any workload or codec. Host latency p99 2.90 ms vs 2.94 ms. Recovery time is the same on both (75 ms, set by the delay).
    • Peak bitrate 110.2% of the target vs 110.0% on main.
  • Earlier probes on this branch before rebasing (HEVC, same loss pattern): at 60 ms delay all 18 drops recovered with recovery frames; at 200 ms, past the 16-reference window, every loss fell back to an IDR. 0 corrupt pictures either way.

Known limits

  • AV1: pixelforge caps the window at 7 references (~58 ms at 120 fps), so at 40 ms RTT plus detection and pipeline time AV1 got 0 recovery frames and fell back to IDRs. Follow-up: space AV1's older references out.
  • H.264: Moonlight clients don't offer RFI for H.264, so nothing changes there.
  • Recovery time is the same either way (set by the RTT); what changes is that there are no IDR bitrate spikes.

Host/Client Testing: rfi-recovery-frames@2ce1553 vs main@de05168

Hardware

Host Client
GPU AMD Radeon RX 7900 XTX (RADV NAVI31) AMD Radeon RX 7900 XTX
Driver radv Mesa 26.2.7-arch1.1 (Vulkan) radeonsi Mesa 26.2.7-arch1.1 (VAAPI)
Other GPUs N/A N/A
CPU AMD Ryzen 9 5950X 16-Core Processor AMD Ryzen 9 5950X 16-Core Processor
Memory 126 GB 126 GB
OS Fedora Linux 44 (Container Image) Arch Linux
Kernel 7.2.11-200.fc44.x86_64 7.2.11-200.fc44.x86_64
Network N/A N/A

Stream

  • rfi-recovery-frames@2ce1553 (porkloin/moonshine@agent/rfi-recovery-frames, with porkloin/pixelforge@agent/rfi-recovery-frames at d365e05), 1920x1080 at 120 fps, 28 Mb/s, streams of 20 s each, routed via loopback, with simulated chaos --loss-frame-every 120 (adds frame drops) --delay 40 (simulates 40ms client latency)
  • workloads (streams): clickflash (3), framestamp-full (21), hdrneon-cycle (4), vkcube (21); 119212 frames decoded; clock sync ±0.02 ms
  • tools: moonshine-test-headless cfac7cb, gamestream-test-client e0a987a, moonshine-otel v1.4.1

Report card

Metric test baseline Target
Decode errors 🟢 0 🟢 0 ≤ 0
Corrupt pictures 🟢 0 🟢 0 ≤ 0
Corrupt after recovery 🟢 0 🟢 0 ≤ 0
Lost frames 🟢 0 🟢 0 ≤ 0
Frame pacing p99 🟢 1.03 frames (8.5 ms) 🟢 1.03 frames (8.6 ms) ≤ 1.10 frames (9.2 ms)
Delivered frame rate 🟢 100.0% 🟢 100.0% ≥ 99.0%
Host latency p99 🟢 2.90 ms 🟢 2.94 ms ≤ 3.00 ms
Recovery time p95 🔴 9.02 frames (75.1 ms) 🔴 9.02 frames (75.1 ms) ≤ 3.00 frames (25.0 ms)
RFI recovery rate 🔴 37.5% 🔴 0.0% ≥ 95.0%
Input latency p50 🟢 0.97 frames (8.1 ms) 🟢 0.93 frames (7.8 ms) ≤ 2.00 frames (16.7 ms)
HDR switch keyframes 🟡 1.00 🟡 1.00 ≤ 0.70
FPS under GPU load no GPU-load test no GPU-load test ≥ 99.0%
Peak bitrate 🟡 110.2% 🟢 110.0% ≤ 110.0%

The loss streams' rows for HEVC and AV1 (this branch, then main):

Check Result Notes
vkcube: hevc, --loss-frame-every 120 --delay 40 ok 18 frames dropped on purpose; 2091 frames at 112.4 fps, 144 missing, 1 keyframes, 130 RFI requests, 18 recovery frames, 18 freezes (1.4 s of 18.6 s frozen)
vkcube: av1, --loss-frame-every 120 --delay 40 ok 18 frames dropped on purpose; 2088 frames at 112.3 fps, 144 missing, 19 keyframes, 128 RFI requests, 0 recovery frames, 18 freezes (1.3 s of 18.6 s frozen)
vkcube: hevc, --loss-frame-every 120 --delay 40, baseline ok 18 frames dropped on purpose; 2088 frames at 112.3 fps, 144 missing, 127 keyframes, 129 RFI requests, 0 recovery frames, 18 freezes (1.3 s of 18.6 s frozen)
vkcube: av1, --loss-frame-every 120 --delay 40, baseline ok 18 frames dropped on purpose; 2089 frames at 112.4 fps, 144 missing, 127 keyframes, 133 RFI requests, 0 recovery frames, 18 freezes (1.3 s of 18.6 s frozen)

RFI recovery rate counts H.264 streams too, where Moonlight never sends RFI, so it reads 37.5% here although HEVC recovered every loss.

Results produced by moonshine-test-headless. Results file attached.

To run the same tests on your host (loopback client):

podman container runlabel run ghcr.io/porkloin/moonshine-test-headless:latest -- --mode 1080p120 --moonshine porkloin@agent/rfi-recovery-frames --pixelforge porkloin@agent/rfi-recovery-frames --baseline --loss --chaos --loss-frame-every 120 --delay 40

To view my results in a throwaway Grafana container, download the results archive and run:
podman container runlabel run ghcr.io/porkloin/moonshine-test-headless:view path/to/results-compact-no-traces.tar.gz

moonshine-rfi-results-compact-no-traces.tar.gz

@porkloin porkloin changed the title Rfi recovery frames RFI recovery frames Nov 18, 2026
@porkloin
porkloin force-pushed the rfi-recovery-frames branch from 2ce1553 to 67e9934 Compare November 18, 2026 06:30
@porkloin

porkloin commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Note, you can also try this PR on a real device (phone, Steam Deck, etc.) using moonshine-test-headless:

podman container runlabel serve ghcr.io/porkloin/moonshine-test-headless -- --moonshine 276 --pixelforge 62

That builds this PR with hgaiser/pixelforge#62 and serves the test workloads as apps to a real Moonlight device. In Moonlight, add the host by its IP address, then type the PIN Moonlight shows into the terminal. It uses Moonshine's default ports (47984, 47989, 48010 TCP and 47998–48000 UDP), so stop any Moonshine already running on the host first.

Pick HEVC in Moonlight (H.264 doesn't use reference frame invalidation), and connect over Wi-Fi, especially forced to 2.4 GHz. With Moonlight's performance stats overlay on, this PR should recover from packet loss with far fewer keyframes, so fewer bitrate spikes and less stutter after a loss.

To compare, serve the current main branch the same way:

podman container runlabel serve ghcr.io/porkloin/moonshine-test-headless

The recovery-frame changes use pixelforge 0.13.0's API
(invalidate_reference_frames returning bool, with_active_reference_frames,
is_recovery_frame); 0.12.1 doesn't build them.
@porkloin
porkloin force-pushed the rfi-recovery-frames branch from dab0109 to e2683c4 Compare November 18, 2026 06:50

@hgaiser hgaiser left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@hgaiser
hgaiser merged commit 934c5c5 into hgaiser:main Nov 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants