Skip to content

fix: keep the quality governor from degrading decoder-bound and new videos - #283

Merged
BGLuis merged 4 commits into
developfrom
fix/quality-governor-per-video
Nov 21, 2026
Merged

BGLuis merged 4 commits into
developfrom
fix/quality-governor-per-video

Conversation

@BGLuis

@BGLuis BGLuis commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Description

Three plays of the same SLR clip at different bitrates on Quest 3 (HEVC Main 10, 59.94 fps, streamed over SFTP) showed the adaptive quality governor making playback worse, not better:

File Resolution Video bitrate Before
4096p_original 8192x4096 ~122 Mbps (level 6.2) ~47 of 60 fps decoded; governor fell HIGH -> EMERGENCY in 6 s
2160p 4320x2160 ~16 Mbps decoded 60 fps but started in EMERGENCY (72 Hz, render scale 0.7) inherited from the previous file
4096p 8192x4096 ~37 Mbps 60 fps, HIGH, 90 Hz

Four fixes, one commit each:

  1. Decoder-bound drops no longer degrade render quality (rust/media-logic/src/quality.rs). Dropped video frames counted as render stress even with GPU time at ~1.9 ms of an 11.1 ms frame. Lowering render scale or refresh rate cannot help a decoder that cannot keep up with the bitrate. Drops now count only when GPU time or compositor pacing is outside its healthy budget, and they no longer block recovery once render is healthy.
  2. The quality level resets when a new video starts (QualityController::reset_for_new_content, called from all six playback entry points in rust/bridge/src/lib.rs). The controller is process-global; the next file kept the previous file's level and needed ~30 s per level to recover. With adaptive mode off, the user's level is kept.
  3. Network sources report the content frame rate (rust/bridge/src/lib.rs). Only local/HTTP playback called set_content_fps, so refresh-rate cadence matching never ran for SMB, FTP, SFTP, NFS or WebDAV. All six paths now go through one helper.
  4. The hardware decoder gets the frame rate (rust/core/src/playback.rs). It was configured with priority=0 but without frame-rate/operating-rate, which MediaFormat documents as the rate a realtime codec plans its resources for.

Type of change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • 🛠️ Technical refactor / Code improvement
  • 📝 Documentation update
  • 🚀 Performance optimization
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)

Affected layers

  • 🟣 Kotlin (app/) — app shell, UI panels, credentials, history, i18n
  • 🔵 C++ (native/) — OpenXR session, Vulkan/GLES render loop, controller input
  • 🦀 Rust (rust/) — demux, decode, audio, network protocols
  • 🔧 Build / CI / scripts
  • 📚 Docs only

How has this been tested?

  • Unit tests (JVM / cargo test)
  • Native/Vulkan build compiled without errors
  • Physically tested on a device (Quest 3 / Quest 3s)

Host: cargo test -p protocols -p media-logic passes; cargo fmt --check and cargo clippy -p protocols -p media-logic --all-targets --all-features -- -D warnings are clean. Four new tests in quality.rs failed before the fix; two feed the telemetry of the ~122 Mbps session. The three existing dropped-frame tests now use a marginal GPU sample so they still exercise the counter. bridge and core were cross-compiled with ./scripts/build-rust.sh (they do not build on the host).

Quest 3, after the fixes (telemetry CSV, same files):

File Quality / refresh Decoded fps Dropped fps
2160p ULTRA, 120 Hz, scale 1.0 60 0
4096p ULTRA, 120 Hz, scale 1.0 60 ~0
4096p_original ULTRA, 120 Hz, scale 1.0 median 58 mean 2.4 (p95 8.0)

Fix 4 alone, 4096p_original over the LAN, one ~2-minute run each: dropped frames 3.85 -> 2.40 per second on average (p95 10.9 -> 8.0), decoded 56.1 -> 57.1 fps. With one run per side this is within what content differences could explain. The ~122 Mbps file still does not hold a steady 60 fps; that looks like the Quest 3 hardware decoder's limit at that bitrate.

Not tested: playback speed above 1x (the decoder is told the 1x frame rate), and long sessions under thermal throttling.

Checklist:

  • My code follows the project's style guidelines (ktlint, rustfmt, clippy -D warnings).
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added new tests that prove my fix is effective or that my feature works.
  • The CI checks (Security & Workflows, Rust, C++, Android and Markdown Lint, gated by CI Gate) are passing for this PR.

If applicable

  • Kotlin never calls Rust directly — the Kotlin → JNI → C++ → C ABI → Rust path is preserved (ADR-002).
  • I changed the screen/stereo mode enum and kept it in sync across all four places: SCREEN_MODE in rust/bridge/src/lib.rs, enum class ScreenMode in native/include/screen_mode.h, the catalog in ScreenFormatCatalog.kt, and Format3D::to_screen_mode_index in rust/media-logic/src/format3d.rs.
  • I added or changed user-facing strings and updated all supported locales (values/strings.xml and all values-*/strings.xml), keeping the key order mirrored and using positional placeholders (%1$s).
  • Pure logic I added lives in media-logic (host-testable) rather than in core, where it could not be tested.
  • No credential is stored or logged in plain text.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DKhrqTrv7b11FgaPF1e7FV

BGLuis and others added 4 commits November 21, 2026 00:45
Dropped video frames lowered the quality level even when GPU time and
compositor pacing were well inside budget. On Quest 3 an 8192x4096 HEVC
Main 10 file at ~122 Mbps decoded at ~47 of 60 fps with the GPU at
~1.9 ms per frame; the governor fell from HIGH to EMERGENCY in 6 s,
cutting render scale to 0.7 and refresh to 72 Hz without helping the
decoder. Drops now count as render stress only when render is not
healthy, and no longer block recovery once it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKhrqTrv7b11FgaPF1e7FV
The quality controller is process-global and kept the level earned by
the previous file. After a decoder-bound 8K clip a 4320x2160 clip
started in EMERGENCY (72 Hz, render scale 0.7) and needed ~58 s of
30-sample recovery steps to reach MEDIUM. Every playback entry point
now resets the level, reason and stress counters, unless adaptive mode
is off and the level is the user's choice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKhrqTrv7b11FgaPF1e7FV
…twork sources

Only local and HTTP playback told the quality controller the video's
frame rate, so refresh-rate cadence matching never ran for SMB, FTP,
SFTP, NFS or WebDAV files. All six playback entry points now set it
after a successful load and clear it on failure, through one helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKhrqTrv7b11FgaPF1e7FV
The decoder was configured with realtime priority but without
KEY_FRAME_RATE or KEY_OPERATING_RATE, which MediaFormat documents as the
rate a realtime codec plans its resources for. Both are now set from the
stream's average frame rate when the container declares it.

On Quest 3 with an 8192x4096 HEVC Main 10 file at ~122 Mbps, one
~2-minute run each: late-dropped frames went from 3.85 to 2.40 per
second on average (p95 10.9 -> 8.0) and decoded fps from 56.1 to 57.1.
The stream still does not hold a steady 60 fps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DKhrqTrv7b11FgaPF1e7FV
@BGLuis
BGLuis merged commit 50aa0f7 into develop Nov 21, 2026
9 checks passed
@BGLuis
BGLuis deleted the fix/quality-governor-per-video branch November 21, 2026 05:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant