Repository navigation
video: first frame within ~150 ms after connect or takeover - #60
Merged
Merged
Conversation
After a takeover or reconnect, the frame writer kept writing to the previous session's cached track until its one-second refresh, and nothing asked the encoder for a keyframe, so a new viewer waited up to a second for frames and then up to another GOP (1s) for a decodable one; this matches the 0.3-1s first-frame times measured in testing. Read the video and audio sinks per frame instead of caching them (an uncontended RwLock read at 60/100 Hz), and force a keyframe with headers when a new video track is attached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Testing showed no first-frame gain: the first frame still arrived 0.1-1.1s after connecting, uniformly, i.e. at the next periodic IDR. The device log explains it: the keyframe was forced at ICE "connected", but the peer connection only reaches "connected" (DTLS done) afterwards, so the IDR was encoded and dropped before media could flow. Force the keyframe from the peer connection state handler on "connected", log "forced keyframe for new session <id>", and ask once more 300 ms later in case the RTP sender started slightly after. The handler captures only the session id, so it adds no reference cycle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first-frame fix did not change measured first-frame times, and the log had no timestamps to tell why. Add timestamps to the log, log every keyframe request (and the follow-up one), and probe the encoder's output so the first keyframe after a request is logged with how many ms it took. A forced IDR shows up within a frame or two; a value near the GOP length means the request was ignored and a periodic keyframe answered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The keyframe probe logged nothing in testing: every forced request was accepted, but no encoder output buffer lacked DELTA_UNIT, so the flag cannot tell IDRs apart. Check the first slice NAL type instead (H.264 only; the scan stops at the first slice) and log the delta flag along with the latency. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The device log showed every keyframe request accepted, yet IDRs only ever came on the 1-second GOP boundary. Cause, in GstVideoEncoder (1.22): an "as soon as possible" force-key-unit request stays pending until the encoder outputs a frame marked as a sync point, and while one is pending, later ASAP requests are dropped. mpph264enc never marks sync points (its IDRs carry DELTA_UNIT), so after the first request per pipeline, none had any effect. Send the request with the current running time instead; timed requests are only compared with the last forced one, so each goes through. PLI and other requests are coalesced while one is outstanding (up to a GOP) to keep the encoder's never-cleared request queue small; the request for a newly connected viewer is never coalesced. The 300 ms follow-up request is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by 4c · project thread
Summary
Before: after a takeover, reconnect or cold connect, the first video frame took 0.15–1.1 s (sometimes seconds). The frame writer kept feeding the previous session's cached track for up to a second, the keyframe for the new viewer was forced at ICE "connected" (before DTLS, so it was dropped), and after the first request per pipeline the MPP encoder ignored forced keyframes entirely, so viewers waited for the next 1 s GOP boundary.
After: the first frame arrives 100–160 ms after the PeerConnection is created, for cold connects and takeovers alike.
How
video.rs: the video and audio writers read the current sink per frame instead of caching it for a second (an uncontended RwLock read at 60/100 Hz).webrtc.rs: on peer connection state "connected" (DTLS done), request an uncoalesced keyframe for the new viewer. The handler captures only the session id, so no reference cycle (see webrtc: release taken-over sessions' UDP sockets #54).pipeline.rs: force-key-unit events carry the current running time. GstVideoEncoder keeps an "as soon as possible" request pending until the encoder outputs a sync-point frame, which mpph264enc never flags, and drops later ASAP requests while one is pending; timed requests go through each time. PLI-driven requests are coalesced while one is outstanding (up to a GOP); the new-viewer request is not.pipeline.rs: an encoder src probe detects IDRs from the H.264 bitstream (first slice NAL type) and logs how long a requested keyframe took.main.rs: log lines now carry timestamps.Testing
cargo +nightly fmt --all(CI)cargo +nightly clippy --workspace --release --all-targets -- -D warnings(CI)cargo +nightly test --workspace(CI)Checklist
#[allow(...)]suppressions and no newunwrap()/expect()in production pathsMissedTickBehavior::Delayrkvm-protoPart of a set of three video PRs (with "pause encoding while nobody is watching" and "negotiate H.264 High profile"); they merge onto
devtogether without conflicts.🤖 Generated with Claude Code
https://claude.ai/code/session_01PbdzY6PfT28goR4BhqdsXR
Generated by Claude Code