Skip to content

Mirror at once while the second screen starts, and stop the reconnect loop - #97

Draft
whoisaldo wants to merge 14 commits into
mainfrom
fix/second-screen-fallback
Draft

whoisaldo wants to merge 14 commits into
mainfrom
fix/second-screen-fallback

Conversation

@whoisaldo

@whoisaldo whoisaldo commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Problem

In Second screen mode the iPad often showed "Your PC answered, but no picture arrived", and over USB the PC's screen blinked and Windows played device connect and disconnect sounds every 11 seconds until the cable was pulled.

When the iPad connected, the host restarted the stream and sent no video until the virtual display had attached: about 3 s for the Enable task (6 s when a Disable from the previous attempt was still running), then up to 10 s waiting for the display. The iPad gives up 10 s after the handshake when no frame has arrived. On PCs where the display attached slowly or not at all, the iPad left first:

  • The host's fallback to mirroring (15 to 17 s in) never reached the iPad.
  • The host's "Windows didn't add the second screen" notice never appeared, because the iPad's goodbye cancelled the attach wait before it timed out.
  • Over USB the host dialed the iPad again 1 s later. Each new attempt turned the virtual display off and on again.

Fix

The fix is host-only and works with the iPad apps already released (0.3.1 and 0.3.2).

  • Mirror first. A connecting iPad gets the mirrored main screen at once from the pipeline that is already running. One worker thread makes every virtual display change, in order, and capture never waits for it, so the Enable task's length no longer matters. Once the display attaches, the stream restarts onto it with a new stream epoch, which the iPad already handles (it resets reassembly and decodes the new resolution).
  • The second screen proves itself. The capture of the virtual display must deliver a real image, read back for the encoder, within 5 s of the switch. A pipeline restart meanwhile keeps that deadline and any capture error, and the watchdog leaves the verdict to it. Until then nothing is sent from it, not even the black placeholder, so the iPad keeps showing the last mirror frame. A display that fails to duplicate, or whose acquisition only times out, goes back to the mirror. That counts as a failed start, and the PC shows its notice. On the iPad, a working switch looks like a brief pause on the last mirror frame before the second screen appears. A failed one is a pause of up to 5 s, then the live mirror again.
  • No loop. A viewer leaving does not cancel a start in progress. Every way a start ends leaves a consistent state; a mode change after Enable turns the display back off. The display stays on for 10 s after its viewer leaves, so a quick reconnect reuses it. The host changes the display at most once per 10 s, also after a mode change, Restart or Try again; only quitting turns it off sooner. Quitting stops the worker first, so no Enable starts or runs after the quit's Disable.
  • A keyframe when asked. A new session, including an iPad back within the 10 s, starts only from an IDR. NVENC and QSV answered the session's keyframe request with a plain I-frame, so the first picture waited for the GOP's next IDR: up to 0.5 s at 60 fps, but up to 22 s on a static second screen, which is resent at 1.3 fps. They now emit an IDR. NVENC also held each frame back two frames (1.5 s at that rate) and now returns each frame as it is encoded. On an RTX 5080, an iPad back within the grace period got its first frame after 140 to 280 ms (before: about 3.2 s, or none in 12 s), and a fresh connect's first mirror frame after about 60 ms (before: about 0.6 s).
  • Remembered failure. If the display does not attach or shows no picture, later connections mirror without touching the driver, and the PC shows its second-screen notice. This lasts until the user clicks Try again, restarts the stream or changes the mode.
  • Logs. Each decision writes one log line. Fallbacks carry second_screen_fallback=attach_timeout|no_frames|duplication_failed|enable_failed|no_driver, and the diagnostics upload already sends every log line, so the backend can count them. The existing "Virtual display attached" and "did not attach in time" lines keep their text.

The attach steps moved from reconcile_virtual_display into host/src/capture/second_screen.rs, behind a small driver trait so they can be tested.

Tests

  • capture::second_screen unit tests on a paused clock with a fake driver:
    • An 8 s Enable that never attaches: no pipeline restart while it runs, and the iPad's 10 s goodbye and a USB redial cancel nothing. The result is one Enable and one Disable 10 s apart, then a minute of reconnects with no driver change, until Try again.
    • After the switch: a display that only times out goes back to the mirror at 5 s, and a duplication error does so at once. A display that shows an image stays.
    • Capture generations never wait for the driver, for any mode, status or viewer, including Try again racing a thread blocked in a slow Enable.
    • A mode change during or after the start never leaves the start claimed, and Try again recovers a claim with no thread behind it.
    • The 10 s grace period, the 10 s minimum gap between toggles (including the Disable after a mode change), the switch happening once, and a quit before Enable.
    • A restart during the 5 s keeps the deadline and a capture error; the next capture generation ends a failed second screen itself; the watchdog does not restart the pipeline while the second screen proves itself; a frame that was not read back proves nothing.
    • On a worker thread: a quit while a start prepares never enables, and a quit waits for an Enable in flight. A release turns the display off whatever status it reads, unless a start was claimed since, which then reuses the display.
  • e2e_synthetic::usb_framed_stream_end_to_end over one USB tunnel, decoding whole frames:
    • With Second screen selected, the first mirror frame decodes on the running pipeline, on first connect and on a reconnect after a failed start.
    • The tunnel survives the switch onto the second screen, and the fallback to the mirror after an injected capture error.
  • encoder::tests::a_viewer_back_mid_gop_starts_from_the_keyframe_it_asked_for feeds a real encoder a viewer that leaves mid-GOP and one that comes back at the static second screen's 750 ms cadence, and passes each packet through the transport's first-frame gate. The session must open on the requested frame, which NVENC must return at once. It runs every encoder that opens on the machine: NVENC on an NVIDIA PC, libx264 everywhere. Without the NVENC options it fails on an RTX 5080 (no frame opens the session in 15 s; with only the IDR option, the frame comes out two frames late).

…start

When an iPad connected in Second screen mode, the host restarted the
pipeline and sent no video until the virtual display attached or a 10 s
wait ran out, on top of the Enable task. The iPad gives up 10 s after the
handshake, so on PCs where the display was slow or never attached it never
saw a picture, and over USB every redial turned the display off and on
again.

The running pipeline now keeps mirroring to the iPad while a separate
thread turns the display on, and the stream restarts onto it once it
attaches. A viewer leaving does not cancel the start. After a failed start
later connections mirror without touching the driver, and the PC shows its
notice, until the user clicks Try again, restarts the stream or changes
the mode. The display stays on 10 s after its viewer leaves, and the host
changes it at most once per 10 s on its own. Each decision logs one line;
fallbacks carry second_screen_fallback=<reason> for the diagnostics upload.
… for the driver

After the switch, the capture of the virtual display must deliver a real
image within 5 s. A display that fails to duplicate, or whose acquisition
only times out, goes back to the mirror, counts as a failed start and shows
the PC's notice. No black placeholder is sent from it meanwhile, so the iPad
keeps the last mirror frame.

Capture generations no longer take the driver lock or run driver tasks: a
thread turns the display off as well as on, so the mirror never waits behind
a slow Enable, also when Try again races a new generation.

Every exit of the start thread clears its claim, and a mode change after
Enable turns the display back off, so Second screen can no longer stay
stuck in Attaching. Try again also recovers a claim with no thread behind it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants