Skip to content

fix(video): stop the once-a-second keyframe stall - #79

Closed
Rust-KVM wants to merge 1 commit into
devfrom
claude/project-thread-s6tlys
Closed

Rust-KVM wants to merge 1 commit into
devfrom
claude/project-thread-s6tlys

Conversation

@Rust-KVM

Copy link
Copy Markdown
Owner

Requested by 4c

Summary

The remote picture (and the host cursor in it) hitched about once a second, even on a LAN. With no GOP configured, the encoder sent a full 4K IDR every second. Each one takes ~25–30 ms to deliver, so the next 2–3 frames arrive late.

Before: GOP = fps (one IDR per second); PLI keyframes unthrottled; the browser uses its default adaptive jitter buffer.

After: GOP = 10 s of source frames, keyframes still sent on demand (new viewer, PLI); PLI-driven keyframes at least 500 ms apart; the web client asks for jitterBufferTarget = 0 (and playoutDelayHint = 0).

How

  • pipeline.rs: the dynamic GOP is fps * DEFAULT_GOP_SECONDS (10). An explicit --video-gop / RUSTKVM_VIDEO_GOP still overrides it.
  • pipeline.rs: force_keyframe (the PLI path) skips requests within KEYFRAME_MIN_INTERVAL (500 ms) of the last forced keyframe. force_keyframe_now (new viewer) stays unconditional and records the time.
  • rkvm-web/net/peer.rs: on track, set jitterBufferTarget / playoutDelayHint to 0 on the receiver via Reflect, so no new web-sys features are needed.

Testing

Measured on hardware (RK3588, 4K60 H.264 source, static desktop). I used a WebRTC viewer that records per-frame RTP arrival without decoding, 60 s per setting:

GOP=60 (old default) GOP=600
keyframes / min 62 6
keyframe size / delivery 349 KB / 28 ms 308 KB / 24 ms
frame gap p99 46.3 ms 16.6 ms
gaps > 50 ms 23 0
  • This build ran on the device and logged Set dynamic GOP to 600 (10s) based on source 60.0fps. An 18 s sample (cut short when a browser viewer took over) gave p99 gap 18.6 ms and one keyframe.

  • The first version of the measurement client overflowed its jitter buffer and sent PLIs continuously; the device emitted ~2600 IDRs in a minute. That's why the PLI rate limit is in this PR.

  • cargo +nightly fmt --all

  • cargo +nightly clippy --workspace --release --all-targets -- -D warnings

  • cargo +nightly test --workspace: not run to completion locally. The host link step fails with cannot open /lib64/libm.so.6, which comes from this build machine and not from the diff. Leaving it to CI.

  • Frontend touched: trunk build --release OK. wasm32 clippy reports two this lifetime isn't used in the type errors in ui/button.rs / ui/field.rs. Those files are untouched here; it's a newer nightly lint.

  • Tested on an RK3588 device (above)

Checklist

  • No #[allow(...)] suppressions and no new unwrap() / expect() in production paths
  • Hot paths stay zero-copy; periodic loops use MissedTickBehavior::Delay
  • Shared protocol types live in rkvm-proto (none added)
  • RPC method set unchanged
  • Module tree / build invariants unchanged

🤖 Generated with Claude Code

With no GOP configured the encoder emitted an IDR every second. At 4K a
keyframe is 10-30x a P-frame and takes ~25-30 ms to deliver, so the next
2-3 frames arrive late every second: measured on a static desktop over 60s,
GOP=60 gave 23 frame gaps over 50 ms (p99 46 ms) against none with a 10 s
GOP (p99 16.6 ms). The remote cursor visibly stuttered with it.

- Default GOP is now 10 s of source frames; viewers still get keyframes on
  demand (new session, PLI).
- PLI-driven keyframes are at least 500 ms apart, so a client that keeps
  losing packets cannot turn every frame into an IDR (a test client did
  push ~2600 IDRs a minute).
- The web client asks for a zero jitter-buffer target so frames play as
  soon as they decode instead of the buffer growing on keyframe bursts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Rust-KVM Rust-KVM self-assigned this Oct 20, 2026
@github-actions github-actions Bot added area: video GStreamer, MPP encoding, WebRTC media rkvm-web Leptos / WASM frontend labels Oct 20, 2026

Copy link
Copy Markdown
Owner Author

Closing as a duplicate of #78, which makes the same three changes (10 s default GOP, 500 ms PLI keyframe limit, zero jitter buffer) and is already green. #78 also limits the zero jitter buffer to the video receiver, so audio keeps its buffer, and resumes encoding with force_keyframe_now, so the PLI limit can never leave a new viewer waiting a whole GOP.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: video GStreamer, MPP encoding, WebRTC media rkvm-web Leptos / WASM frontend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants