实现 Dolby Vision RPU 映射并完善解码回退 - #130
Closed
1824239290 wants to merge 43 commits into
Closed
1824239290 wants to merge 43 commits into
1824239290 wants to merge 43 commits into
Conversation
Shinokawa
requested changes
Sep 5, 2026
Member
There was a problem hiding this comment.
需求成立，建议保留 RPU 色彩映射的核心实现，修正解码策略并补齐图像验证后再合并。
Profile 5 的 base layer 不兼容普通 HDR10，现有普通 YCbCr/HDR 管线确实缺少必要的 DV 信号解释。对照 libplacebo，当前 polynomial/MMR、主要矩阵组合和 PQ 处理顺序有实质参考依据；FFmpeg → SourceColorState → uniforms → shader 的分层也符合项目现状。
当前有两个需要修复的问题，已在对应代码处评论：
- Profile 5 一律切软解的前提不成立。 本项目的 FFmpeg HEVC + D3D11VA 路径能够同时保留硬件纹理和逐帧 RPU。建议采用“硬解输出纹理 + RPU 元数据 + GPU shader 映射”的路径，并按具体后端能力决定回退；VideoToolbox 需要补真机确认，MediaCodec/HarmonyOS 原生 AVCodec 需要分别验证。
- 合法的 31/32 位系数分母被当成非法元数据拒绝。 FFmpeg 支持这些值，浮点 RPU 系数还会被统一转换成分母 32 的定点表示。已用等价系数重编码复现从有效 metadata 变成 None。P5 没有普通 HDR10 兼容基础层，这种情况下沿普通管线显示不能视为安全降级，应提供明确诊断。
本次本机验证：
- 公开 HEVC 测试基础层注入公开 P5 RPU 后，D3D11VA 输出 format=d3d11，GPU 纹理与可用 RPU 同时存在。
- 携带 RPU 的软件帧通过 wgpu 离屏渲染和 D3D11 软件上传。
- 将有效 RPU 的系数从分母 23 等价重编码到 32，元数据接受检查失败，复现第二个问题。
以上合成码流测试验证元数据传递与运行路径，不代表真实 DV 影片的色彩验收，也没有测量 4K 性能或完成 Apple/移动端真机验证。
合并前还需要补充：
- 真实 P5、P8 素材解码后的 RPU 检查，以及相同帧相同输出设置下与参考实现的像素/图像比较；覆盖硬解/软解、seek 和场景切换。
- 普通 SDR/HDR10 回归。此 PR 同时修改了通用 P010 范围换算并增加 D3D11 软件上传，影响不局限于 DV；这两部分也适合拆分独立验证。
- 修正文档支持范围。Profile 8 是单层，不能统一描述为“双层、HDR10 base layer”，例如 8.4 使用 HLG 兼容基础层。请移除或补充证据支持 DOLBY_VISION_IMPROVEMENTS.md 中的 production-ready、<0.1ms 等结论。
现有 ERIKA_DV_SAMPLE / ERIKA_DV_PROFILE_8_SAMPLE 测试仅检查容器 profile，不执行解码或渲染，缺失环境变量时直接返回；shader 名称存在性测试也不能证明跨后端颜色一致。CI 全绿目前不足以覆盖这项功能的主要正确性风险。
——来自 GPT 6.0 Astra
Comment on lines
+5877
to
+5882
| let hardware = matches!( | ||
| requested.backend, | ||
| DecoderBackend::VideoToolbox | ||
| | DecoderBackend::D3d11va | ||
| | DecoderBackend::MediaCodec | ||
| | DecoderBackend::AvCodec |
Member
There was a problem hiding this comment.
[P1] 不应因 Profile 5 一律禁用 VideoToolbox/D3D11VA 硬解
这里“硬解无法提供 RPU”的前提不适用于项目使用的 FFmpeg 通用 HEVC + hwaccel 路径。FFmpeg 在 CPU 解析 RPU，再通过 ff_dovi_attach_side_data 将解析结果附到 AVFrame；该过程不要求使用软件像素解码。
本机用 regular.hevc 的基础层替换注入 profile5.bin 的 RPU，直接调用 DecoderConfig::d3d11va()：输出 format=d3d11，d3d11va_texture().is_some() 与 dovi_metadata().is_some() 均为 true。这个实验只验证 P5 RPU 传递，不用于视觉验收。
因此此处会让本可同时获得 GPU 纹理和 RPU 的 P5 内容改走 CPU 解码、重打包与上传，增加不必要的性能和功耗成本。请保留能够传递 RPU 的硬解路径，由后续 shader 完成 DV 映射；按具体后端的能力决定回退。VideoToolbox 有相同 HEVC 解析路径的源码依据，但仍需真机确认；MediaCodec/原生 AVCodec 应分别验证。
Comment on lines
+4450
to
+4455
| let bl_bit_depth = usize::from(header.bl_bit_depth); | ||
| let coef_denom = u32::from(header.coef_log2_denom); | ||
| // Validate bit depth and coefficient denominator before using them in | ||
| // shifts below; malformed side data must be rejected, never panic. | ||
| if !(8..=16).contains(&bl_bit_depth) || coef_denom >= 31 { | ||
| return None; |
Member
There was a problem hiding this comment.
[P2] 请接受 FFmpeg 支持的 31/32 位系数分母
项目 FFmpeg 的 libavcodec/dovi_rpudec.c 对固定系数允许 coef_log2_denom 为 13..32；对于浮点系数，解码器会将它们转换为 coef_log2_denom=32 的定点表示。此处 >=31 会拒绝合法数据。
已复现：从解码帧获得一份可用 P5 RPU（分母 23），将 poly/MMR 系数等价左移 9 位并把分母改为 32，数学含义不变，但 frame.dovi_metadata() 从 Some 变为 None。当前系数缩放使用 powi，NLQ 的移位使用 u64，也没有必须排除 31/32 的移位约束。
请按 FFmpeg 数据范围校验并增加等价重编码的回归测试。否则合法 P5 内容会失去映射并沿普通色彩管线显示；P5 的基础层不兼容普通 HDR10，不能把这种结果当作安全回退。
1824239290
force-pushed
the
agent/dolby-vision-mapping
branch
from
September 6, 2026 09:32
a7ffb61 to
9ebcea3
Compare
This was referenced Sep 6, 2026
Implements Dolby Vision profile 5/8 rendering for software-decoded HEVC, following mpv's approach (libplacebo pl_shader_dovi_reshape): - erika_ffmpeg_sys: expose libavutil/dovi_meta.h to bindgen. - ffmpeg.rs: convert the decoder's per-frame AV_FRAME_DATA_DOVI_METADATA into shader-ready floats (pivots normalized by bl_bit_depth, curve coefficients by 2^-coef_log2_denom, RPU ycc_to_rgb/rgb_to_lms matrices). - pipeline.rs: SourceColorState carries per-frame DoviSourceMetadata (peak from source_max_pq, primaries forced to BT.2020); DoviUniforms packs the reshaping curves into vec4-aligned uniforms, premultiplies the HPE LMS->RGB composite and folds the 1024/1023 signal-offset correction; render graph gains a DoviReshape pass. - Shaders (WGSL/Metal/HLSL): per-component pivot/polynomial/MMR reshaping of the raw full-range base-layer signal, RPU matrix + offsets to PQ-encoded IPT, then LMS->RGB after PQ linearization; range expansion and the YCbCr matrix are bypassed for RPU frames. - VideoFrameDescriptor carries dovi_metadata for hardware-frame paths. FEL (profile 7 residual) is not composed; profiles 4/7 render the base layer reshaping only, matching current mpv without a bound enhancement layer. No C ABI changes.
The MMR extraction loop dereferenced the enumerate index instead of the mutable coefficient slot (E0614 across all CI targets), and several manually formatted blocks deviated from rustfmt. Nested call formatting was also made deterministic where line width sat near the 100-column limit.
WGSL if-expressions are not portable across validators; use a var plus a plain if statement for the packed-luma sample.
- metal layout test expects the appended dovi uniform block (size 3104, dovi at offset 144) - dovi LMS composite golden values recomputed from the RPU default matrix expressed in exact 1/16384 fractions - remaining rustfmt deviations (assert wrapping, let-row split, blank line, copy_nonoverlapping join) applied verbatim from CI output
libplacebo's pl_map_avdovi_metadata unconditionally sets BT.2020/PQ from the RPU because P5/P8 VUI tags are unreliable. Erika's dovi() builder forced primaries and peak but left the transfer untouched, so streams with an unspecified trc decoded the reshaped PQ signal through the sRGB branch. Also sync reference_white_nits to the PQ reference white so the linearization scale stays consistent.
Produces a runnable macOS binary from the fork CI so the Dolby Vision color pipeline can be exercised against real content without a local Rust toolchain.
Hardware decoders (VideoToolbox/D3D11VA/MediaCodec) cannot convey the Dolby Vision RPU, so profile 5 streams rendered with the wrong colors under the default hardware path. Read the container's DOVI configuration record (AV_PKT_DATA_DOVI_CONF) at decoder-open time and switch to the software decoder for profile 5, matching mpv's behavior; the decoder event carries the requested backend plus a fallback reason. Profile 8 stays on hardware since its base layer is HDR10-compatible. Also raise the wgpu_decode_png example's deployment target to 13.0 so the VideoToolbox hwcontext symbols resolve at link time, and add dolby_vision_profile() tests (fixture: None; ERIKA_DV_SAMPLE: Some(5)).
av_stream_get_side_data no longer exists in ffmpeg 8.x; the container's DOVI configuration record lives in AVStream.coded_side_data and is read through av_packet_side_data_get.
The example binary never calls the VideoToolbox hwcontext path that references the version-check helper, so resolve it lazily like the release dylib does instead of failing the executable link.
-Wl,-U only defers resolution and dyld then fails at launch; the __isPlatformVersionAtLeast helper from FFmpeg's VideoToolbox hwcontext must be statically provided by the compiler runtime instead.
- Add comprehensive top-level documentation (docs/dolby-vision.md) - Complete pipeline architecture explanation - Profile 5 vs 8 comparison and decode strategy - Reshaping algorithm details (polynomial and MMR) - Color transform flow and testing instructions - Enhance inline documentation - Add field-level comments to DoviSourceMetadata - Clarify signal offset scaling formula (1024/1023 ratio) - Document uniform buffer size (~3KB, 20% of mobile limits) - Expand test coverage - Add Profile 8 detection test (ERIKA_DV_PROFILE_8_SAMPLE) - Add 3 unit tests for dolby_vision_decode_fallback logic - Verify Profile 5 forces software decode - Verify Profile 8 stays on hardware decode - Improve CI maintainability - Add comment explaining libclang_rt.osx linker workaround - Reference commits acae10f and 3498b8e for context - Add validation comments for bit depth and coefficient ranges These changes enhance documentation, testing, and maintainability without modifying the core Dolby Vision mapping algorithm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tests were incorrectly expecting DecoderBackend::AvCodec instead of DecoderBackend::Software. DecoderConfig::software() returns Software variant. - dolby_vision_profile_5_falls_back_to_software_decode: expect Software - dolby_vision_software_decode_stays_software: expect Software This fixes the CI failure on macOS arm64.
dovi_lms_to_rgb(float3 linear) 中 linear 为 HLSL 插值修饰符保留字， D3DCompile(vs_4_0/ps_4_0) 报 X3000，Windows 端挂载即挂。 Metal/WGSL 不保留该词，故 macOS/iOS 未受影响。改名 lin， fxc vs_4_0/ps_4_0 均验证通过。
…ments - Fix HDR10 content being mapped to SDR in Auto output mode by promoting to apple_edr when source_is_hdr is true, defaulting unconfigured headroom to 4.0 - Bypass Mobius tone mapping in Metal shader when target_transfer is PQ (direct HDR10 PQ passthrough) - Update Metal output_status to report ActiveOutputEncoding::Hdr10Pq and 10000/203 headroom for PQ outputs - Accept 32-bit coef_denom in FFmpeg Dolby Vision RPU metadata parser - Preserve hardware decode for VideoToolbox and D3D11VA on Dolby Vision Profile 5 - Correct Profile 8 documentation to single-layer with backward-compatible base
- tone_map_nits in the D3D11 (HLSL) and wgpu (WGSL) shaders now bypasses tone mapping for PQ targets, matching the Metal shader: HDR10 PQ passthrough is exact instead of running the Mobius shoulder above 0.75 * target_peak - ffmpeg dovi parsing: remove the unreachable -1 match arm on nlq_method_idc (the FFmpeg field is u8; an absent NLQ parses as method 0 with all-neutral parameters) and refresh the dovi_metadata docs now that hardware-decoded frames also carry RPU side data
9280687 made Auto mode promote every HDR source to apple_edr, defaulting an unconfigured headroom to 4.0 — including on displays without any EDR capability, where the compositor forces the 10000-nit-coded PQ layer into the SDR range and the picture reads as blown out. Restore the pre-9280687 decision: Auto promotes an HDR source to apple_edr only when the embedder actually requests headroom above 1.0; otherwise the source stays on the SDR tone mapping path. The capi headroom defaulting (0/NaN -> 1.0 instead of 4.0) is reverted to match, so hosts that never configured headroom get the original behavior back. The HDR10 PQ passthrough itself remains available for EDR requests above 1.0 headroom.
Reverts the NLQ match-arg cleanup from 9ebcea3. bindgen maps FFmpeg's 'enum AVDOVINLQMethod nlq_method_idc' to a signed C enum, not a u8, and the decoder writes the AV_DOVI_NLQ_NONE sentinel (-1) whenever an RPU carries no NLQ definition. Real profile 5 content does exactly that, so treating -1 as unreachable made frame_dovi_metadata() reject every real Dolby Vision frame. Caught by on-device verification: raw AV_FRAME_DATA_DOVI_METADATA is present on both software and VideoToolbox hardware frames of a real P5 file, but dovi_metadata() returned None after 9ebcea3. With the -1 arm restored the same frames parse again on both backends.
Auto mode promoted every HDR source to an apple_edr(4.0) PQ layer without consulting what the attached display can present. On displays without EDR (external SDR monitors report 1.0x, and even the built-in panel of an M1 MacBook Air reports 2.0x potential at best) the compositor has no headroom to give an itur-2100-pq layer, so the 10000-nit-coded signal is forced into the SDR range and the picture looks blown out. Query the presenting screen's maximumPotentialExtendedDynamicRange- ColorComponentValue (falling back to mainScreen, then 1.0) and clamp the resolved mode: EDR requests on EDR-less displays tone map to SDR, and otherwise the requested headroom is capped at the display's real capability. The potential value is used on purpose so playback does not flip modes while the brightness slider moves. Scoped to macOS; iOS/tvOS keep the previous behavior pending device verification.
The first cut sent the -[CAMetalLayer screen] selector, which no CALayer on macOS responds to; the unrecognized-selector NSException aborted the whole process on the first HDR frame (Rust cannot catch foreign exceptions). Resolve mainScreen via NSScreen directly instead, guard the headroom selector with respondsToSelector, and document the multi-display limitation.
added 8 commits
September 7, 2026 15:17
Side-by-side comparison against Infuse showed the legacy Mobius default reading washed out on HDR masters: its knee passes everything below 0.75 * target_peak through 1:1, so a 1000-nit source keeps most of its midtones at mastered brightness while the SDR surface can only show a fraction of that range. Add ToneMapOperator::Bt2390 (code 3) and make it the default: the ITU-R BT.2390 EETF evaluated in the PQ domain, ported from libplacebo's pl_tone_map_bt2390 (default knee offset, 0 target black so the black point adaptation is a no-op). Implemented in all three shader backends with a Rust reference implementation and anchor/monotonicity tests. PQ-target (HDR passthrough) paths are unaffected: the EDR bypass runs before the operator dispatch.
ToneMapConfig::default() hardcoded Mobius, so changing ToneMapOperator::default() (as the BT.2390 commit does) never reached any pipeline — every render kept using Mobius regardless of the operator default. Derive the config default from the enum instead.
The linear gamut matrix pushes highly saturated wide-gamut colors (saturated BT.2020 greens/cyans) outside the target gamut where their negative components were hard-clipped to zero by the nits clamp — turning teal greens into neon yellow-greens with shifted hue. Blend out-of-gamut (negative) components towards their BT.709 luma, just enough to fit the gamut while preserving hue — matching libplacebo's desaturate gamut mode. Implemented in all three shader backends with a Rust reference implementation and hue-preservation tests.
The BT.2390 EETF was applied per RGB channel, so bright saturated colors lost chroma during the 10:1 HDR-to-SDR compression (saturated pink washed out toward white; measured mean saturation 48/255 vs 58 on an Infuse capture of the same frame). Apply the EETF to the pixel's BT.709 luma instead and scale RGB uniformly: hue and saturation survive the compression exactly. Values pushed above the target peak by the scaling are blended towards luma just enough to fit, matching the out-of-gamut desaturation pass.
Auto mode now negotiates with the display the playback window is on: an HDR source promotes to EDR using the presenting screen's real headroom (up to the embedder's configured cap) when the screen can present EDR, and stays on the SDR tone mapping path when it cannot — so the same embedder configuration yields HDR on an HDR display and correctly tone-mapped SDR on a non-HDR one. macOS only for now; iOS/tvOS keep the previous resolve-based behavior.
Add --seek-sequence t:pos,t:pos to the macOS native demo: the player auto-seeks through the given timeline (forward, backward, across scene changes) while logging per-seek decode/render counters and failures — covering the seek and scene-change verification the PR review asks for. Verified on real HDR10 and DV profile 5 content: five seeks each, zero failures, RPU metadata still applied after seeks. Also always activate the demo app: in smoke mode it stayed inactive, macOS App-Napped its render timer down to ~1.6Hz and stalled the presentation-driven decode.
The SDR tone map target was the 100-nit mastering value while the sRGB encode treats 1.0 as the display's white — on any real panel the whole picture lands in the top of the signal range (measured against mpv/libplacebo on the same DV profile 5 frame: our midtones were 1.33x brighter, shadows 1.47x). Follow the BT.2408 HDR reference white convention used by libplacebo/mpv: tone map HDR sources into a 203-nit peak and encode sRGB with 1.0 = 203 nits. Same-frame bucket comparison against mpv gpu-next now matches within 3-7% across all brightness ranges (was 32% off in shadows). SDR sources keep the untone-mapped 100-nit path.
An unmapped Dolby Vision frame must not degrade silently: Profile 5 has no HDR10-compatible base layer, and a rejected Profile 8 RPU displays with wrong colors. frame_dovi_metadata now reports a DoviRejectReason, exposed through Frame::dovi_unavailable_reason, and the decode loop emits a structured diagnostic on the first affected frame and every 1024 frames after. A missing RPU is only reported for Profile 5 streams; other profiles legitimately repeat RPUs across frames.
1824239290
force-pushed
the
agent/dolby-vision-mapping
branch
from
September 7, 2026 07:17
79f49aa to
5f148f7
Compare
added 10 commits
September 7, 2026 16:18
… tone mapping The RPU DM extension blocks carried in AV_FRAME_DATA_DOVI_METADATA hold level 1 per-frame min/max/avg luminance in 12-bit PQ codes, sitting in the same allocation as the header/mapping/color structures. Parse the validated ext-block region (mirroring av_dovi_get_ext) into a new DoviSourceMetadata.l1 field. SourceColorState::dovi now prefers the frame's L1 peak over the static mastering peak (source_max_pq) for nominal_peak_nits, which flows into the tone map's source peak via VideoUniforms.nits[0] on all three backends - matching libplacebo's use of the RPU CIE-Y metadata. The mastering display metadata keeps the static L0 value so output-mode negotiation never reacts to per-frame brightness. L1 blocks are optional: absent or invalid (zero peak, inverted min/max, out-of-range ext region) simply fall back to the current static behavior.
DOLBY_VISION_IMPROVEMENTS.md gains a section for the level 1 block parse and the scene-adaptive source peak it drives; docs/dolby-vision.md documents the L1 flow in the color pipeline and lists the new tests.
NSScreen.mainScreen tracks the systemwide key window, which belongs to a different app whenever this one is inactive. A player window sitting on an SDR display then negotiated EDR from whatever screen the active app's key window happened to occupy and ran PQ passthrough on the SDR panel, rendering washed-out colors - visible the moment focus moved away during multi-display playback. Resolve the headroom through the layer's hosting window instead: AppKit makes the hosting NSView the delegate of a view-assigned backing layer, so read delegate->window->screen and fall back to mainScreen only when that chain is unavailable (e.g. detached layers). Mode re-negotiation already runs per frame, so moving a window between screens now switches Sdr <-> AppleEdr live.
The tone map now mirrors libplacebo's color map: RGB in source primaries to HPE-LMS, PQ-encode, IPT, map the intensity axis through the operator's curve plus the hue-preserving chroma rule, then decode back to RGB in the target primaries (the primaries conversion happens inside the roundtrip). - default operator is spline, matching mpv gpu-next '--tone-mapping=auto'; operators add spline (4), bt.2446a (5), st2094-10 (6) - BT.2390 and the other curves gain black-point compensation: target black = peak/contrast (auto 1000:1 for SDR, 0 for HDR/EDR targets), gated so SDR-to-SDR rendering is untouched - spline/ST2094 pivots follow the Dolby Vision L1 average luminance when present (fed through tone_map_extra.y) - ToneMapConfig: curve_param + contrast_ratio wired to uniforms; the dead knee_start/desaturate fields are gone - uniforms: ipt_matrix_rows (6x4) + tone_map_extra/coeffs across WGSL/MSL/HLSL; apple.rs layout test updated - validation: naga WGSL parse test under feature=wgpu plus curve anchor, pivot, IPT-matrix and black-point tests
Ports libplacebo's pl_gamut_map_perceptual (mpv's default gamut map): a CPU-generated 48x32x256 IPT-space LUT whose texels hold the mapped (I, P+0.5, T+0.5) color. The shaders rebuild RGB in the source primaries, run the IPT roundtrip, sample the LUT in ICh space ([min,max] PQ range, chroma magnitude, hue angle) and decode to target primaries, replacing the fast gamut_compress path whenever an HDR source is tone-mapped into a different gamut. - gamut.rs: golden-section per-hue boundary search + Mobius softclip, deterministic and cached per (source, target, peak) key - uniforms: gamut_lut_enabled + packed primaries code across WGSL/MSL/HLSL - backends: wgpu/Metal/D3D11 create and bind the cached 3D texture (RGBA16F, filterable), dummy 1x1x1 binding keeps the fast path valid - tests: gating, primaries code round-trip, cross-shader markers
…draw_video The earlier helper landed inside D3d11DeviceState (whose &self cannot reach the renderer-owned cache); move it to D3d11Renderer::gamut_lut_view and thread the resulting SRV into draw_video's texture slot 2.
Contributor
Author
|
本 PR 的修订版已作为 #136 重新提交（历史已 squash 为单 commit）。 针对本 PR review 的回应已写入 #136 描述：
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
为 Erika 增加基于 FFmpeg RPU 元数据的 Dolby Vision 映射路径，按 libplacebo 的处理顺序完成曲线重塑、矩阵变换、PQ 线性化和 HDR 输出，并在无法安全映射时回退到普通 base layer。
主要改动
解码与兼容性策略
验证
测试边界
自动化测试覆盖元数据解析、映射数学、解码回退、ABI 和宿主编译；本 PR 尚未在 CI 中执行真实 Dolby Vision 样片的人工播放验收。