Skip to content

[Linux] RTSP handshake fails when run as user #2778

Closed

Description

@ptrsr

Is there an existing issue for this?

  • I have searched the existing issues

Is your issue described in the documentation?

  • I have read the documentation

Is your issue present in the latest beta/pre-release?

Yes

Describe the Bug

Everything works as expected when running sunshine as root. However, when running sunshine as a user, moonlight is not able to connect due to "Failed to start RTSP handshake (error 110)". Sunshine then dumps its core after detecting a hang. There is no additional info in the logs, except for the ping timeout. Note that I have enabled permissions for KMS capture.

Expected Behavior

I expect a stream to be established successfully.

Additional Context

I am running Plasma 6 using Wayland.

The pairing doesn't work as smooth as when running as root either. The pairing occasionally requires the PIN to be put in more than once. It's almost like there is a networking issue, but there is no reason for one.

Host Operating System

Linux

Operating System Version

Arch Linux (kernel 6.9.3-arch1-1)

Architecture

64 bit

Sunshine commit or version

v0.23.1

Package

Arch linux package (sunshine.pkg.tar.zst)

GPU Type

AMD

GPU Model

Radeon RX 6700

GPU Driver/Mesa Version

mesa 1:24.1.2-1

Capture Method

KMX (Linux)

Config

origin_web_ui_allowed = wan
ping_timeout = 1000

Apps

No response

Relevant log output

[2024:06:30:18:32:35]: Info: Screencasting with KMS
[2024:06:30:18:32:35]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:35]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:35]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:35]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:35]: Info: Encoder [nvenc] failed
[2024:06:30:18:32:35]: Info: Trying encoder [vaapi]
[2024:06:30:18:32:35]: Info: Screencasting with KMS
[2024:06:30:18:32:35]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:35]: Info: Found monitor for DRM screencasting
[2024:06:30:18:32:35]: Info: Found connector ID [112]
[2024:06:30:18:32:35]: Info: Found cursor plane [87]
[2024:06:30:18:32:35]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:35]: Info: Color depth: 8-bit
[2024:06:30:18:32:35]: Info: Color range: [JPEG]
[2024:06:30:18:32:35]: Error: [h264_vaapi @ 0x5efa14772400] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2024:06:30:18:32:35]: Info: Retrying with fallback configuration options for [h264_vaapi] after error: Function not implemented
[2024:06:30:18:32:35]: Warning: [h264_vaapi @ 0x5efa14a5ef80] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:35]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:35]: Info: Color depth: 8-bit
[2024:06:30:18:32:35]: Info: Color range: [JPEG]
[2024:06:30:18:32:35]: Error: [hevc_vaapi @ 0x5efa14a5ef80] No usable encoding entrypoint found for profile VAProfileHEVCMain (17).
[2024:06:30:18:32:35]: Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented
[2024:06:30:18:32:35]: Warning: [hevc_vaapi @ 0x5efa14772400] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:35]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:35]: Info: Color depth: 8-bit
[2024:06:30:18:32:35]: Info: Color range: [JPEG]
[2024:06:30:18:32:35]: Error: [av1_vaapi @ 0x5efa14772400] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:35]: Info: Retrying with fallback configuration options for [av1_vaapi] after error: Function not implemented
[2024:06:30:18:32:35]: Error: [av1_vaapi @ 0x5efa14a5ef80] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:35]: Error: Could not open codec [av1_vaapi]: Function not implemented
[2024:06:30:18:32:35]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:35]: Info: Color depth: 8-bit
[2024:06:30:18:32:35]: Info: Color range: [JPEG]
[2024:06:30:18:32:35]: Error: [av1_vaapi @ 0x5efa14772400] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:35]: Info: Retrying with fallback configuration options for [av1_vaapi] after error: Function not implemented
[2024:06:30:18:32:35]: Error: [av1_vaapi @ 0x5efa152dec40] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:35]: Error: Could not open codec [av1_vaapi]: Function not implemented
[2024:06:30:18:32:35]: Info: Screencasting with KMS
[2024:06:30:18:32:35]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:35]: Info: Found monitor for DRM screencasting
[2024:06:30:18:32:35]: Info: Found connector ID [112]
[2024:06:30:18:32:35]: Info: Found cursor plane [87]
[2024:06:30:18:32:35]: Info: HDR color coding [Rec. 2020 + SMPTE 2084 PQ]
[2024:06:30:18:32:35]: Info: Color depth: 10-bit
[2024:06:30:18:32:35]: Info: Color range: [JPEG]
[2024:06:30:18:32:35]: Error: [hevc_vaapi @ 0x5efa152dec40] No usable encoding entrypoint found for profile VAProfileHEVCMain10 (18).
[2024:06:30:18:32:35]: Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented
[2024:06:30:18:32:35]: Warning: [hevc_vaapi @ 0x5efa14772400] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:35]: Info: 
[2024:06:30:18:32:35]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:06:30:18:32:35]: Info: 
[2024:06:30:18:32:35]: Info: Found H.264 encoder: h264_vaapi [vaapi]
[2024:06:30:18:32:35]: Info: Found HEVC encoder: hevc_vaapi [vaapi]
[2024:06:30:18:32:35]: Error: Failed to create client: Daemon not running
[2024:06:30:18:32:35]: Info: Configuration UI available at [https://localhost:47990]
[2024:06:30:18:32:41]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024:06:30:18:32:41]: Info: Trying encoder [nvenc]
[2024:06:30:18:32:41]: Info: Screencasting with KMS
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:41]: Info: Screencasting with KMS
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Error: Couldn't find monitor [0]
[2024:06:30:18:32:41]: Info: Encoder [nvenc] failed
[2024:06:30:18:32:41]: Info: Trying encoder [vaapi]
[2024:06:30:18:32:41]: Info: Screencasting with KMS
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Info: Found monitor for DRM screencasting
[2024:06:30:18:32:41]: Info: Found connector ID [112]
[2024:06:30:18:32:41]: Info: Found cursor plane [87]
[2024:06:30:18:32:41]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:41]: Info: Color depth: 8-bit
[2024:06:30:18:32:41]: Info: Color range: [JPEG]
[2024:06:30:18:32:41]: Error: [h264_vaapi @ 0x77a72c1b5f40] No usable encoding entrypoint found for profile VAProfileH264High (7).
[2024:06:30:18:32:41]: Info: Retrying with fallback configuration options for [h264_vaapi] after error: Function not implemented
[2024:06:30:18:32:41]: Warning: [h264_vaapi @ 0x77a72c3be880] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:41]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:41]: Info: Color depth: 8-bit
[2024:06:30:18:32:41]: Info: Color range: [JPEG]
[2024:06:30:18:32:41]: Error: [hevc_vaapi @ 0x77a72c3be880] No usable encoding entrypoint found for profile VAProfileHEVCMain (17).
[2024:06:30:18:32:41]: Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented
[2024:06:30:18:32:41]: Warning: [hevc_vaapi @ 0x77a72c1b5f40] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:41]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:41]: Info: Color depth: 8-bit
[2024:06:30:18:32:41]: Info: Color range: [JPEG]
[2024:06:30:18:32:41]: Error: [av1_vaapi @ 0x77a72c3be880] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:41]: Info: Retrying with fallback configuration options for [av1_vaapi] after error: Function not implemented
[2024:06:30:18:32:41]: Error: [av1_vaapi @ 0x77a72ccff100] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:41]: Error: Could not open codec [av1_vaapi]: Function not implemented
[2024:06:30:18:32:41]: Info: SDR color coding [Rec. 601]
[2024:06:30:18:32:41]: Info: Color depth: 8-bit
[2024:06:30:18:32:41]: Info: Color range: [JPEG]
[2024:06:30:18:32:41]: Error: [av1_vaapi @ 0x77a72ccff100] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:41]: Info: Retrying with fallback configuration options for [av1_vaapi] after error: Function not implemented
[2024:06:30:18:32:41]: Error: [av1_vaapi @ 0x77a72c3be880] No usable encoding entrypoint found for profile VAProfileAV1Profile0 (32).
[2024:06:30:18:32:41]: Error: Could not open codec [av1_vaapi]: Function not implemented
[2024:06:30:18:32:41]: Info: Screencasting with KMS
[2024:06:30:18:32:41]: Info: /dev/dri/card1 -> amdgpu
[2024:06:30:18:32:41]: Info: Found monitor for DRM screencasting
[2024:06:30:18:32:41]: Info: Found connector ID [112]
[2024:06:30:18:32:41]: Info: Found cursor plane [87]
[2024:06:30:18:32:41]: Info: HDR color coding [Rec. 2020 + SMPTE 2084 PQ]
[2024:06:30:18:32:41]: Info: Color depth: 10-bit
[2024:06:30:18:32:41]: Info: Color range: [JPEG]
[2024:06:30:18:32:41]: Error: [hevc_vaapi @ 0x77a72ccff100] No usable encoding entrypoint found for profile VAProfileHEVCMain10 (18).
[2024:06:30:18:32:41]: Info: Retrying with fallback configuration options for [hevc_vaapi] after error: Function not implemented
[2024:06:30:18:32:41]: Warning: [hevc_vaapi @ 0x77a72c085740] Driver does not support some wanted packed headers (wanted 0xd, found 0x1).
[2024:06:30:18:32:41]: Info: 
[2024:06:30:18:32:41]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2024:06:30:18:32:41]: Info: 
[2024:06:30:18:32:41]: Info: Found H.264 encoder: h264_vaapi [vaapi]
[2024:06:30:18:32:41]: Info: Found HEVC encoder: hevc_vaapi [vaapi]
[2024:06:30:18:32:41]: Info: Executing [Desktop]
[2024:06:30:18:32:42]: Error: Initial Ping Timeout
[2024:06:30:18:32:42]: Info: 192.168.1.19: Ping Timeout
[2024:06:30:18:33:16]: Fatal: Hang detected! Session failed to terminate in 10 seconds.
Trace/breakpoint trap (core dumped)

Activity

  1. ReenigneArcher commented on Jun 30, 2024

    @ReenigneArcher
    Member

    Test the pre-release version...

  2. ptrsr commented on Jun 30, 2024

    @ptrsr
    Author

    Done, same issue persists.

  3. gschintgen commented on Jul 2, 2024

    @gschintgen
    Contributor

    Why did you reduce ping_timeout by a factor of 10?

    Sunshine should terminate gracefully though.

  4. ptrsr commented on Jul 5, 2024

    @ptrsr
    Author

    I've tried setting it to 1000ms and 10000ms, it does not change anything.
    I feel like there is something weird going on regarding the UDP/TCP sockets in user mode. The PIN input has a lot of timeouts as well.

    Is there a way to get sunshine to connect with pulseaudio in root mode? I have tried running pulse audio as root but I still get access denied errors when the stream has started.

  5. ptrsr commented on Jul 12, 2024

    @ptrsr
    Author

    As a workaround, I start Sunshine as root (sudo sunshine). This allows screen capture to work. However, audio capture failed as users can not connect to the PulseAudio instance of other users by default.

    To fix this, create a .pulse folder in the home folder of the user;

    mkdir ~/.pulse
    

    And copy the default PulseAudio config;

    cp /etc/pulse/default.pa ~/.pulse/
    

    And add the following line to the ~/.pulse/default.pa file to allow TCP local connections;

    load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1
    

    Restart pulseaudio;

    pulseaudio -k
    pulseaudio -D
    

    and run sunshine using;

    sudo PULSE_SERVER=tcp:127.0.0.1:4713 sunshine
    

    Now both video and audio should work. TCP does have some overhead though, so instead I would recommend using unix sockets. I have yet to try this myself, though.

  6. gschintgen commented on Jul 13, 2024

    @gschintgen
    Contributor

    For the pairing: Do you have to enter the PIN multiple times for the same connection attempt? If not, could it simply be the result of connecting once to regular user-sunshine and then connecting to root-sunshine? Both have their separate configs, so it would be expected behaviour to do the initial pairing twice (once for each sunshine instance).

    With the binary packages there should be no need to do any manual setup steps (like setcap). There should also be no reason at all to run Sunshine as root. (I know, "should". Apparently it does not work as intended on your system.)

    As for your workaround, you could probably just reuse the command documented for the flatpak (which does need root for KMS capture):

    sudo -i PULSE_SERVER=unix:$(pactl info | awk '/Server String/{print$3}') sunshine
    
  7. Theslees commented on Aug 4, 2024

    @Theslees

    try switching to x11 @ptrsr if you're on wayland

  8. svenvvv commented on Aug 18, 2024

    @svenvvv

    I encountered a very similar problem under X11 running with AMD GPU on Arch Linux. Pairing took a lot of time as the requests kept timing out and I wasn't able to start a stream at all (RTSP handshare error 110). I did not try running as root.
    I did a bit of debugging and for me the cause of the problem was a deadlock in the tray icon library used in Sunshine.

    Try building the sunshine/sunshine-git package from AUR and edit the PKGBUILD to disable tray icon during building. This fixed it for me.
    E.g. add these to CMake configure arguments in the PKGBUILD build() command:

        -D SUNSHINE_ENABLE_TRAY=0 \
        -D SUNSHINE_REQUIRE_TRAY=0 \
    

    I don't really care for the tray icon, so I'm not going to debug it any further and just disable it, but in case someone wants to debug it then this bug was most likely already present to some degree in sunshine 0.21. Pairing and connecting used to work, but I remember having Sunshine crash fairly often with the "hang detected" message after I quit Moonlight.

  9. MichaelThomas0721 commented on Aug 18, 2024

    @MichaelThomas0721

    Thank you @svenvvv , your solution worked for me as well.
    System: hyprland (wayland) on arch with nvidia gpu

  10. Djagur commented on Oct 8, 2024

    @Djagur

    My thanks as well @svenvvv, your approach helped me to fix an issue that I had been struggling with for weeks. I'm on Wayland Fedora 40, nvidia

  11. schindlershadow commented on Nov 4, 2024

    @schindlershadow
  12. ReenigneArcher commented on Nov 4, 2024

    @ReenigneArcher
    Member

    @schindlershadow While I understand you are facing an issue, that's really not a good idea. https://github.com/LizardByte/Sunshine/security/advisories

  13. schindlershadow commented on Nov 4, 2024

    @schindlershadow

    Thanks for letting me know. I'll close off the open ports on my fw and leave it LAN only for now until this is fixed

  14. awerty-noob commented on Dec 11, 2024

    @awerty-noob

    @svenvvv Your solution is excellent! Solved on my 0.23.1
    @ReenigneArcher Any plan to fix it?

  15. ReenigneArcher commented on Dec 11, 2024

    @ReenigneArcher
    Member

    I have not looked into this. We accept PRs though.

  16. 6 remaining items

  17. moozhub commented on Jan 11, 2026

    @moozhub

    I encountered a very similar problem under X11 running with AMD GPU on Arch Linux. Pairing took a lot of time as the requests kept timing out and I wasn't able to start a stream at all (RTSP handshare error 110). I did not try running as root.
    I did a bit of debugging and for me the cause of the problem was a deadlock in the tray icon library used in Sunshine.

    Try building the sunshine/sunshine-git package from AUR and edit the PKGBUILD to disable tray icon during building. This fixed it for me.
    E.g. add these to CMake configure arguments in the PKGBUILD build() command:

        -D SUNSHINE_ENABLE_TRAY=0 \
        -D SUNSHINE_REQUIRE_TRAY=0 \
    

    I don't really care for the tray icon, so I'm not going to debug it any further and just disable it, but in case someone wants to debug it then this bug was most likely already present to some degree in sunshine 0.21. Pairing and connecting used to work, but I remember having Sunshine crash fairly often with the "hang detected" message after I quit Moonlight.

    Just wanted to chime in and say the latest build provided in the prebuilt archlinux repo had made my system run into this issue.. building my own version of trunk disabling the tray options above allowed sunshine to be usable again with no hangs. Everything was working for months with no issue then all the sudden ran into this..

    All I can point out is that my system had a number of updates that were installed along with the latest sunshine build, some that were in reference to NetworkManager which I use for my network interface management. The error I saw in the sunshine log appeared to show messages indicating that the sunshine server was unable to reach back out to the client (Initial Ping Timeout), even though the manual connection checks I did from the server to the client were working as expected, so it seems like sunshine is having problems at the network stack.

    Apologies if this is not very in-depth, just sharing my experience. How this has anything to do with the system tray, I have no idea.

    Thanks.

  18. ReenigneArcher commented on Mar 24, 2026

    @ReenigneArcher
    Member
    1. This may have been fixed by build(deps): bump third-party/tray from 4caf0d0 to 563dee4 #4894 available in pre-releases.
    2. If anyone would be willing to test out a new version of the tray library with Sunshine, that would be greatly appreciated: fix(linux): migrate to qt tray #4907 ... it uses qt instead of libnotify and appindicator
  19. lapinsami commented on May 31, 2026

    @lapinsami

    This issue (and probably #4199) still exists on the latest version (built from the latest commit). Workaround is to disable the tray in the config.

  20. Kishi85 commented on May 31, 2026

    @Kishi85
    Contributor

    This and #4199 are likely the same root cause:

    Notification sent with libnotify have a (unfortunately non-configurable) timeout for DBus of 25 seconds which exceeds the 10 seconds for the NvEnc hang detection that triggers here.

    I'm currently working on a way to run those DBus operations async in LizardByte/tray#141 but it might take a while to get thread synchronization right for that.

  21. Kishi85 commented on Jun 1, 2026

    @Kishi85
    Contributor

    Would anyone that has the time and is able to self-compile Sunshine with LizardByte/tray#141 applied to third-party/tray please give some feedback if it fixes the issue? Thanks!

  22. moozhub commented on Jun 18, 2026

    @moozhub

    @Kishi85 I took a look at the Archlinux PKGBUILD file that the authors wrote and should be able to modify it to easily build from the branch you mentioned, if you still need a tester..?

  23. ReenigneArcher commented on Jun 18, 2026

    @ReenigneArcher
    Member

    @moozhub these changes are already merged into master and available in the latest pre-release.

  24. moozhub commented on Jun 18, 2026

    @moozhub

    @ReenigneArcher .. using pre-release 2026.616.153054, still experiencing an issue. Forcefully toggling the tray option to disabled fixes it. Can provide logs if needed.

  25. Kishi85 commented on Jun 18, 2026

    @Kishi85
    Contributor

    @moozhub 2026.616.153054 does not contain the fix (b9e0739) yet. You'll unfortunately have to wait for the next pre-release build.

    Try it with v2026.617.152807 and if the issue occurs with that one please post your logs.

    EDIT: Newest pre-release contains the fix already, just missed it in the changelog upon first glance ._.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions