diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0e152fbf..038cc261 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,7 +24,7 @@ jobs: run: sudo apt-get update && sudo apt-get install -y build-essential cmake ninja-build python3 - name: Run cargo check - run: cargo check --all-targets --all-features + run: cargo check --workspace --all-targets --all-features test: name: Test Suite @@ -59,7 +59,7 @@ jobs: run: sudo apt-get update && sudo apt-get install -y build-essential cmake ninja-build python3 - name: Run clippy - run: cargo clippy --all-targets --all-features -- -D warnings + run: cargo clippy --workspace --all-targets --all-features -- -D warnings fmt: name: Rustfmt diff --git a/AGENTS.md b/AGENTS.md index f2dab8d3..d665ec9c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,8 +11,14 @@ cargo test -- --ignored # device + ffmpeg tests, ignored by default cargo bench --bench encode cargo run --example encode -- h264 cargo run --example decode -- input.264 output.yuv +cargo run -p pixelforge-tools --bin encode -- h264 --b-frames 3 +cargo run -p pixelforge-tools --bin decode -- input.264 output.yuv ``` +The examples (`examples/`) are for users: short, one use each. The tools +(`tools/`, a workspace crate that is not published) are for checking the +library, with every option that takes. The commands below use the tools. + Decoding on Intel Arc under Mesa needs the video queues enabled explicitly: `ANV_DEBUG=video-decode,video-encode`. @@ -31,7 +37,7 @@ Do not edit `README.md` directly; update the doc comments in `src/lib.rs` instea To verify the quality of the encoded videos, run: ```bash -cargo run --example encode -- h265 \ +cargo run -p pixelforge-tools --bin encode -- h265 \ && rm -f decoded.yuv \ && ffmpeg -hide_banner -loglevel error -y -i output.h265 -pix_fmt yuv420p -f rawvideo decoded.yuv \ && ffmpeg -hide_banner -loglevel info -s 320x240 -pix_fmt yuv420p -f rawvideo -i testdata/test_frames.yuv -s 320x240 -pix_fmt yuv420p -f rawvideo -i decoded.yuv -lavfi psnr -f null - @@ -41,18 +47,18 @@ To verify decoding, decode a stream and compare against ffmpeg's software decoder, which should be byte-identical: ```bash -cargo run --example decode -- tests/data/bframes.264 out.yuv \ +cargo run -p pixelforge-tools --bin decode -- tests/data/bframes.264 out.yuv \ && ffmpeg -hide_banner -loglevel error -y -i tests/data/bframes.264 -pix_fmt nv12 ref.yuv \ && cmp ref.yuv out.yuv && echo "decode matches ffmpeg" ``` -The `decode` example picks the codec from the extension (`.264`, `.265`, +The `decode` tool picks the codec from the extension (`.264`, `.265`, `.obu`, `.ivf`). For AV1 the reference is dav1d, and `-noautoscale` keeps a stream that changes size from being scaled to its first size (it applies to any codec; H.264 and H.265 streams in `tests/data` do not change size): ```bash -cargo run --example decode -- tests/data/av1/aom_altref.obu out.yuv \ +cargo run -p pixelforge-tools --bin decode -- tests/data/av1/aom_altref.obu out.yuv \ && ffmpeg -hide_banner -loglevel error -y -c:v libdav1d -i tests/data/av1/aom_altref.obu \ -noautoscale -pix_fmt nv12 ref.yuv \ && cmp ref.yuv out.yuv && echo "decode matches dav1d" @@ -60,8 +66,9 @@ cargo run --example decode -- tests/data/av1/aom_altref.obu out.yuv \ `tests/decode_streams.rs` does this for every fixture in `tests/data` (use `p010le` for the 10-bit ones), and `tests/roundtrip.rs` for each codec's own -encoder output. A run with `PIXELFORGE_NO_UNIFIED_LAYOUTS=1` set takes the -copying path below. +encoder output. Those tests take whichever path the device has; the +`PIXELFORGE_NO_UNIFIED_LAYOUTS` switch below is read by the `decode` tool +only. On NVIDIA (615.78.08) the AV1 decoder does not copy the `StdVideoAV1SequenceHeader` handed to `vkCreateVideoSessionParametersKHR`: it reads it again at every @@ -76,7 +83,7 @@ with no copy. Where it does not, they are copied into private images. Force the copying path to exercise it on hardware that would otherwise never take it: ```bash -PIXELFORGE_NO_UNIFIED_LAYOUTS=1 cargo run --example decode -- \ +PIXELFORGE_NO_UNIFIED_LAYOUTS=1 cargo run -p pixelforge-tools --bin decode -- \ tests/data/bframes.264 out.yuv && cmp ref.yuv out.yuv ``` @@ -96,27 +103,25 @@ Do not judge that one by PSNR. The hardware sampler and ffmpeg reconstruct subsampled chroma differently, so they agree almost everywhere and disagree hard on pixels sitting on a colour edge; on the synthetic test patterns that drags PSNR to ~26 dB while the images are indistinguishable. Compare the share -of pixels that agree instead. For `bframes.264` that has been 87.9% within 2 -and 33.2% exact on every device tested so far, and it is deterministic enough -to treat a change in it as a regression. +of pixels that agree instead (all three channels). For `bframes.264` that has +been 87.9% within 2 on every device tested so far, and it is deterministic +enough to treat a change in it as a regression. -`sample_planes` is the other half of that story, and the tighter test. It reads -the same frames through per-plane views, with no ycbcr conversion and no sampler -at all, and writes the samples straight through, so its output is plain NV12 and -can be compared exactly: +`tests/sample_planes.rs` is the other half of that story, and the tighter +check. It reads the same frames through per-plane views, with no ycbcr +conversion and no sampler at all, and compares the samples byte for byte with +`YuvDownloader`'s copy of each frame, on both decode paths: ```bash -cargo run --example sample_planes -- tests/data/bframes.264 out.yuv \ - && cmp ref.yuv out.yuv && echo "plane views match the decoder" +cargo test --test sample_planes -- --ignored --nocapture ``` -Where that example is worth extra attention: on a device *without* -`VK_KHR_unified_image_layouts`, its frames are pool copies, which carry no video -usage, so a mistake in how a plane view is created passes silently. The same -mistake is an immediate validation error on a device that has the extension, -because there the frame really is the decoder's DPB image. So if a device -supporting it is available, run this example there before trusting a green run -elsewhere. `RUST_LOG=debug` reports which case applies, as `pinnable=`. +On a device *without* `VK_KHR_unified_image_layouts`, its frames are pool +copies, which carry no video usage, so a mistake in how a plane view is created +passes silently. The same mistake is an immediate validation error on a device +that has the extension, because there the frame really is the decoder's DPB +image. So if a device supporting it is available, run it there under +validation before trusting a green run elsewhere. Nothing in `tests/data` changes resolution mid-stream, and that path has its own hazards, so build one and check it: @@ -125,11 +130,11 @@ own hazards, so build one and check it: ffmpeg -f lavfi -i testsrc2=size=320x240:rate=30:duration=1 -c:v libx264 -bf 2 -f h264 a.264 ffmpeg -f lavfi -i testsrc2=size=640x480:rate=30:duration=1 -c:v libx264 -bf 2 -f h264 b.264 cat a.264 b.264 > switch.264 -cargo run --example decode -- switch.264 out.yuv +cargo run -p pixelforge-tools --bin decode -- switch.264 out.yuv ``` Sixty frames, thirty at each size, and one `generation N -> N+1` line at the -changeover. Run `sample_frame` and `sample_planes` over it too: a consumer's own +changeover. Run `sample_frame` over it too: a consumer's own images are sized from a frame, so a resolution change is where they have to be rebuilt, and getting that wrong is silent rather than loud. Build the reference per segment, since ffmpeg rescales a resolution-changing stream to its first size and a whole-stream reference will @@ -181,6 +186,79 @@ PIXELFORGE_MAX_ACTIVE_REFERENCES=1 cargo test --test rfi -- --ignored --nocaptur `RUST_LOG=pixelforge::encoder::codec=debug` reports the negotiated window and per-frame count, on the `Reference window` line. +`tests/rfi.rs` runs each codec without B-frames and with the B-frames the +device has. It checks each frame's PSNR against the frames before the loss, +not only the average: a decoder that predicts from other pictures than the +encoder did loses a few dB from there on, which the average hides. That is +how the H.264 P-frame after an invalidation was found to predict from the +lost frames in a decoder (it had MMCO operations and no list modification). + +B-frames (`EncodeConfig::with_b_frames`, `tests/b_frames.rs`, the `_b_frames` +tests in `tests/roundtrip.rs`) are encoded later than they are submitted. The +planner (`src/encoder/gop.rs`) is pure code and has the unit tests for the +order. What needs a device is the stream each codec writes, and the checks are +ffmpeg's (dav1d's for AV1): decode with `-err_detect explode`, compare with +pixelforge's decoder byte for byte, and compare the frame types `ffprobe` +sees. Run the encode tool with them when touching reference lists: + +```bash +cargo run -p pixelforge-tools --bin encode -- h264 --b-frames 7 --pyramid \ + && ffmpeg -hide_banner -loglevel error -err_detect explode -i output.h264 -f null - +``` + +A wrong reference list does not always make ffmpeg report an error. It can +decode cleanly to the wrong pictures, so also check the PSNR against the source +with the command above: a B-frame that predicts from another picture than the +encoder used drops it by 5 dB or more on those frames. To see what a driver +wrote into a slice header, use +`ffmpeg -i output.h264 -c copy -bsf:v trace_headers -f null -`. + +AV1 has no B-frame type. The reference frame after a group is encoded first as +a hidden frame, whose bytes go in front of the next packet, and a +`show_existing_frame` header made on the CPU shows it later +(`src/encoder/av1/references.rs`). So AV1 packets are in display order, and +`ffprobe` reports no B-frames for them. + +What the drivers do with B-frames, as far as it was checked: + +- NVIDIA (615.78.08) reports pyramid support for H.264 and AV1, and flat only + for H.265. So the H.265 pyramid does not run in the tests on that driver. It + did decode correctly there once with the capability check forced off. +- An H.264 pyramid needs `ref_pic_list_modification` on the P-frame after it, + and drivers differ in how they write the operations they are given. NVIDIA + writes exactly the entries it gets: the end operation has to be one of them, + for the list operations, and then for the marking operations too. It ends + the marking operations by itself only in a P-slice with no list operations. + This is checked on NVIDIA only, so `implemented` in `src/encoder/bframes.rs` + offers the H.264 pyramid on that vendor only. Before offering it on another + driver, check the slice headers there with `trace_headers`. +- NVIDIA's H.264 B-frames use forward prediction only (no skip, no direct, no + list 1), so they cost more bits than P-frames. ffmpeg's `h264_vulkan` with + `-bf 2` gives the same on that GPU, so it is the driver's and not + pixelforge's. H.265 and AV1 B-frames save bits there. +- RADV and ANV were not available when B-frames were written. Nothing about + B-frames is checked on them. +- NVIDIA (615.78.08) writes its own value for some syntax, whatever + pixelforge asks for: `sps_temporal_mvp_enabled_flag = 0` (H.265), + `enable_ref_frame_mvs = 0`, `enable_jnt_comp = 0` and + `skip_mode_present = 0` (AV1). Those tools would help B-frames most, so + they are worth trying on another driver, with + `ffmpeg -c copy -bsf:v trace_headers` to see whether it writes them. + +`scripts/compare_b0.sh` builds v0.11.0 in a worktree and compares its output +with the working tree's for each codec, with CQP and CBR. Without B-frames all +streams now differ from v0.11.0, for four reasons, all intended: H.264 has its +deblocking filter on, H.264 CBR has no QP bounds, H.265 has 8x8 as its +smallest coding block, and AV1 uses CDEF and loop restoration. Before those +changes, `h265-*-gop0` was the only difference: there v0.11.0 gave the driver +a picture order count that wrapped at 512, and the count no longer wraps. Run +it after a change to the encode flow, with a base revision that has those +changes (`scripts/compare_b0.sh `), for an exact comparison. + +A test binary that builds many contexts must call `common::init_logging()` +first, also for another reason than logging: the NVIDIA driver can be loaded +only about 26 times in one process, and that call keeps it loaded. + Tests run one at a time (`RUST_TEST_THREADS=1` in `.cargo/config.toml`): the NVIDIA driver can deadlock when one thread destroys a device while another creates objects on a different one. @@ -218,7 +296,9 @@ view by image handle, and only RADV without validation showed it. ## Project Structure - `src/` - Library source code -- `examples/` - Usage examples for encoding and decoding +- `examples/` - Short usage examples, one use each +- `tools/` - Command-line tools to check the library (workspace crate, not + published) - `tests/` - Integration tests; the device + ffmpeg checks are `#[ignore]`d and run with `cargo test -- --ignored` - `benches/` - The encode timing/throughput benchmark (`cargo bench`) diff --git a/CHANGELOG.md b/CHANGELOG.md index dea8df3a..229f9ed3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,41 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- Add B-frame encoding for H.264, H.265 and AV1: `EncodeConfig::with_b_frames`, at most `MAX_B_FRAMES`. +- Add B-frame pyramids: `EncodeConfig::with_b_frame_references` and `BFrameReferences`. +- Add `VideoContext::supports_b_frames` and `BFrameSupport`. +- Add `--b-frames` and `--pyramid` to the `encode` example. +- Add `EncodeConfig::with_quality_level` and `VideoContext::encode_quality_levels`. The default stays 0, the fastest level. `Encoder::new` returns an error for a level the device does not have. +- Add `YuvDownloader` and `DownloadedFrame`, to copy decoded frames to CPU memory: the reverse of `YuvUploader`. +- Add the `encode_gpu` example: encode RGB frames that the application draws into its own Vulkan image. +- Add the `tools` workspace crate (not published), with `encode` and `decode` tools that have the options for checking the library. + +### Changed + +- The examples are short and show one use each. `encode` and `decode` lost their testing options (now in the tools), `query_capabilities` uses pixelforge's own queries, and `sample_planes` is now a test (`tests/sample_planes.rs`) that compares plane views with `YuvDownloader` on both decode paths. + +- **Breaking:** `EncodedPacket::dts` is an `i64`. With B-frames it starts below zero. +- **Breaking:** A future from `Encoder::submit` resolves with the next packet in decode order. With B-frames that is not the packet of the submitted frame. +- With B-frames, `Encoder::invalidate_reference_frames` drops every reference encoded from the first lost frame on, and the next frames predict from the references that survive. If none survives, the oldest frame that waits for its group is encoded at once as the IDR. +- With a constant QP, the B-frames of a pyramid get one QP step more for each level (a q-index step of 4 for AV1). With 7 B-frames, streams are 2% to 4% smaller at the same quality. +- The H.264 encoder writes the lowest level from 4.1 that the stream needs, by its size, frame rate, DPB and peak bitrate. It always wrote 4.1, also where the stream needed more. `max_dec_frame_buffering` is one frame smaller (NVIDIA already wrote that value). +- The H.264 and H.265 reference pictures carry their own picture type to the driver, not always P. +- With B-frames, a DMA-BUF input is always copied. +- H.265 streams with more than 256 frames between two IDR frames differ from 0.11.0: the encoder now gets the picture order count without a wrap. +- The AV1 encoder uses CDEF and loop restoration. At the same quality, streams are 3% to 10% smaller, and each frame takes longer to encode (about 0.6 ms at 1080p on an RTX 5080). + +### Fixed + +- Fix the H.264 decoder's reorder depth for streams without a VUI. +- Fix the H.264 encoder, which turned the deblocking filter off. At the same quality, streams are about 10% smaller. +- Fix the H.264 encoder's CBR and VBR, which held the QP between 18 and 42. On easy content the bitrate stayed far below the target. +- Fix H.264 reference frame invalidation: the P-frame after it predicted from the lost frames in a decoder, because it did not reorder its reference list. The quality dropped by about 8 dB from there on. +- Let the H.265 encoder use 8x8 coding blocks. The minimum was 16x16. At the same quality, streams are 3% to 18% smaller. + ## [0.11.0] - 06-11-2026 ### Added diff --git a/Cargo.lock b/Cargo.lock index 9aec45bf..7452efa9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -276,6 +276,15 @@ dependencies = [ "tracing-subscriber", ] +[[package]] +name = "pixelforge-tools" +version = "0.0.0" +dependencies = [ + "pixelforge", + "pollster", + "tracing-subscriber", +] + [[package]] name = "pollster" version = "1.0.1" diff --git a/Cargo.toml b/Cargo.toml index 0acf88bf..1e4ccd25 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,6 +9,9 @@ keywords = ["vulkan", "video", "encoding", "decoding", "h264"] categories = ["multimedia::video", "graphics"] readme = "README.md" +[workspace] +members = [".", "tools"] + [features] default = [] dmabuf = [] diff --git a/README.md b/README.md index 0fd5831b..88ce16e4 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +![pixelforge](assets/banner-readme.png) + # pixelforge [![CI](https://github.com/hgaiser/pixelforge/workflows/CI/badge.svg)](https://github.com/hgaiser/pixelforge/actions) @@ -24,14 +26,15 @@ H.265 and AV1, for both encode and decode. - **HDR support**: 10-bit encoding (P010, YUV444P10), PQ transfer function, BT.2020 color space. - **GPU-native API**: Encode directly from Vulkan images (`vk::Image`). - **Flexible configuration**: Rate control (CBR, VBR, CQP), quality levels, GOP settings. +- **B-frames**: for all three codecs, flat or as a pyramid, where the + device can encode them. Off by default, since they add latency. - **Multiple input formats**: BGRx, RGBx, BGRA, RGBA, ABGR2101010 (10-bit packed), RGBA16F (FP16). -- **Utility helpers**: [`YuvUploader`] for easy YUV data upload to GPU. +- **Utility helpers**: [`YuvUploader`] to upload YUV frames to the GPU, + and [`YuvDownloader`] to copy decoded frames back to the CPU. - **Optional DMA-BUF input**: import a producer's buffers, or allocate buffers for it, read with no copy where the driver allows, with sync_file fences (Linux only). -> **Note**: B-frame encoding is not yet implemented, so there is no option for it. - ## Supported Codecs | Codec | Encode | Decode | @@ -181,8 +184,9 @@ fn main() -> Result<(), Box> { A live frame reserves a DPB slot, so [`DecodeConfig::with_output_depth`](decoder::DecodeConfig::with_output_depth) bounds how many can be outstanding before the decoder starts copying -pictures out instead of handing over its own. Reading a frame back to the -CPU is the consumer's job; `examples/common` shows one way. +pictures out instead of handing over its own. To get a frame's samples on +the CPU (for a file, a test or software processing), copy it with +[`YuvDownloader`]. ### RGB input @@ -265,6 +269,61 @@ flight. The pixels are in one of: CPU, waits for that itself. - a `DmaBufImage`, a DMA-BUF (see below). +### B-frames + +[`EncodeConfig::with_b_frames`] puts B-frames between two reference +frames. A B-frame predicts from the frame shown before it and from the one +shown after it, which costs fewer bits, and it costs latency: the encoder +has to wait for the later frame. [`VideoContext::supports_b_frames`] says +what the device can do for a codec, and [`Encoder::new`] returns an error +for more than that. + +```rust +let b_frames = encoder.config().b_frames as usize; +let mut pending = VecDeque::new(); +for frame in frames { + pending.push_back(encoder.submit(Frame::image(frame))?); + // At most `b_frames` frames wait for the frame after their group, so + // the oldest future is only sure to resolve once more are pending. + // Waiting for it sooner never returns. + while pending.len() > b_frames { + let packet = pollster::block_on(pending.pop_front().unwrap())?; + // Write `packet.data`. The packets come in decode order. + } +} +// Encode the frames that still wait, then take the rest. +encoder.flush()?; +for future in pending { + let packet = pollster::block_on(future)?; +} +``` + +What changes with B-frames: + +- The future of the k-th submitted frame resolves with the k-th packet of + the stream, in decode order. That is the packet of another frame, and + [`EncodedPacket::pts`] says which. +- For H.264 and H.265, [`EncodedPacket::dts`] counts in decode order and + starts below zero, so it is never after `pts`. An AV1 stream has no + reordering of its own: its packets are in display order, with + `dts == pts`, and the frame encoded ahead travels hidden in an earlier + packet. +- [`Encoder::request_idr`], [`Encoder::flush`], [`Encoder::repeat`] and + [`Encoder::reconfigure`] encode the frames that wait at once, as a + shorter group. +- [`Encoder::invalidate_reference_frames`] drops every reference encoded + from the first lost frame on, also B-frames shown before it, because + the client lost those packets too. If no reference survives, the oldest + frame that waits for its group becomes the IDR at once. +- With a constant QP, a B-frame gets a slightly higher QP than the + other frames, and one step more for each level of a pyramid. +- A DMA-BUF is copied when it is submitted and never read directly, so + the producer gets its buffer back at once. + +How much B-frames save depends on the driver. On NVIDIA (615.78.08) they +save bits with H.265 and AV1. With H.264 that driver predicts a B-frame +from one direction only, and the stream gets larger than without them. + ### Encoding on your own device An application that already renders with Vulkan can give pixelforge its own @@ -319,27 +378,27 @@ cargo bench --bench encode ## Examples -Run the examples with: - ``` -# Query codec capabilities +# What this device can encode and decode cargo run --example query_capabilities -# Decode H.264 to raw YUV -cargo run --example decode -- input.264 output.yuv - -# Encode, choosing the codec (h264, h265 or av1) +# Encode frames from CPU memory (h264, h265 or av1) cargo run --example encode -- h265 -# Sample decoded frames through a ycbcr conversion (RGBA output) -cargo run --example sample_frame -- input.264 out.rgba +# Encode frames that the application draws on the GPU +cargo run --example encode_gpu -- h265 + +# Decode a stream to a raw YUV file +cargo run --example decode -- input.264 output.yuv -# Sample decoded frames through per-plane views (NV12 output) -cargo run --example sample_planes -- input.264 out.yuv +# Decode and sample the frames in a shader, with no copy (RGBA output) +cargo run --example sample_frame -- input.264 out.rgba ``` -Correctness checks that need a video device and ffmpeg are integration -tests, ignored by default: +The `tools` crate in the repository has command-line tools to check the +library, with more options: `cargo run -p pixelforge-tools --bin encode` +and `--bin decode`. Correctness checks that need a video device are +integration tests, ignored by default: ``` cargo test -- --ignored @@ -352,8 +411,7 @@ See [shader/README.md](shader/README.md) for details on editing and recompiling ## TODO's -1. [] H.265 and AV1 decoding. -1. [] B-frames support (encode). +1. [] Open GOPs. ## Contributing diff --git a/README.tpl b/README.tpl index 43a0348e..d89386f1 100644 --- a/README.tpl +++ b/README.tpl @@ -1,3 +1,5 @@ +![pixelforge](assets/banner-readme.png) + # {{crate}} [![CI](https://github.com/hgaiser/pixelforge/workflows/CI/badge.svg)](https://github.com/hgaiser/pixelforge/actions) diff --git a/assets/banner-readme.png b/assets/banner-readme.png new file mode 100644 index 00000000..acede33b Binary files /dev/null and b/assets/banner-readme.png differ diff --git a/assets/banner-social.png b/assets/banner-social.png new file mode 100644 index 00000000..42e6a547 Binary files /dev/null and b/assets/banner-social.png differ diff --git a/examples/common/mod.rs b/examples/common/mod.rs deleted file mode 100644 index 4be7a670..00000000 --- a/examples/common/mod.rs +++ /dev/null @@ -1,434 +0,0 @@ -//! Reading a decoded frame back to the CPU. -//! -//! pixelforge hands out a GPU image and stops there: it has no readback path of -//! its own, because a renderer sharing the device does not want one, and a -//! consumer that does want one knows better than the library where the pixels -//! should end up. This is that consumer's side of the deal, shared by the -//! examples that write raw YUV to a file. -//! -//! Two details are worth copying into real code: -//! -//! - A frame in `GENERAL` layout (which is what the zero-copy path hands out) -//! needs no transition to be copied from. A frame in any other layout is a -//! private image, so transitioning it is safe. -//! - The copy runs on the transfer queue, which the decoder also uses on the -//! copying fallback path. Submitting to one `VkQueue` from two threads at -//! once is undefined, so these examples drive the decoder and the readback -//! from the same thread. - -use ash::vk; -use pixelforge::decoder::{DecodeConfig, DecodedFrame, Decoder, FramePoll}; -use pixelforge::encoder::{BitDepth, Codec}; -use pixelforge::vulkan::VideoContext; -use std::fs::File; -use std::io::Write; - -/// Bytes handed to the decoder per call, standing in for a network read. -pub const CHUNK_SIZE: usize = 64 * 1024; - -/// A coded stream read from a file. -pub enum Input { - /// A raw stream that the decoder frames itself: Annex B for H.264 and - /// H.265, low-overhead OBUs for AV1. - ByteStream(Vec), - /// Whole coded frames, as a container gives them (IVF). - Frames(Vec>), -} - -/// Read `path`, and pick the codec from its extension: `.264`/`.h264`, -/// `.265`/`.h265`/`.hevc`, `.obu`/`.av1` or `.ivf` (AV1 in IVF). -pub fn open_input(path: &str) -> Result<(Codec, Input), Box> { - let extension = std::path::Path::new(path) - .extension() - .and_then(|e| e.to_str()) - .unwrap_or("") - .to_ascii_lowercase(); - let data = std::fs::read(path)?; - Ok(match extension.as_str() { - "264" | "h264" => (Codec::H264, Input::ByteStream(data)), - "265" | "h265" | "hevc" => (Codec::H265, Input::ByteStream(data)), - "obu" | "av1" => (Codec::AV1, Input::ByteStream(data)), - "ivf" => (Codec::AV1, Input::Frames(ivf_frames(&data)?)), - other => return Err(format!("unknown stream extension {other:?}").into()), - }) -} - -/// The frames of an IVF file: a 32-byte file header, then each frame behind a -/// 12-byte header holding its size and timestamp. -fn ivf_frames(data: &[u8]) -> Result>, Box> { - if data.len() < 32 || &data[..4] != b"DKIF" { - return Err("not an IVF file".into()); - } - let header_len = u16::from_le_bytes([data[6], data[7]]) as usize; - let mut frames = Vec::new(); - let mut pos = header_len; - while pos + 12 <= data.len() { - let size = u32::from_le_bytes(data[pos..pos + 4].try_into()?) as usize; - let start = pos + 12; - let frame = data.get(start..start + size).ok_or("truncated IVF frame")?; - frames.push(frame.to_vec()); - pos = start + size; - } - Ok(frames) -} - -/// The decode configuration `input` needs: byte-stream framing for a raw -/// stream, frame-aligned for frames from a container. -pub fn decode_config(codec: Codec, input: &Input) -> DecodeConfig { - match input { - Input::ByteStream(_) => DecodeConfig::new(codec).with_byte_stream(), - Input::Frames(_) => DecodeConfig::new(codec), - } -} - -/// Drive `decoder` over `input`, calling `on_frame` for every frame it emits. -/// -/// A raw stream is fed in chunks of [`CHUNK_SIZE`], standing in for network -/// reads; frames from a container are fed one per call. Frames are polled -/// after each call and again after `finish`, so the in-stream and -/// end-of-stream paths share one callback. `NeedsKeyframe` is not -/// special-cased: polling then is a no-op when nothing is ready. -pub fn decode_input( - decoder: &mut Decoder, - input: &Input, - mut on_frame: F, -) -> Result<(), Box> -where - F: FnMut(DecodedFrame) -> Result<(), Box>, -{ - let chunks: Box> = match input { - Input::ByteStream(stream) => Box::new(stream.chunks(CHUNK_SIZE)), - Input::Frames(frames) => Box::new(frames.iter().map(Vec::as_slice)), - }; - for (i, chunk) in chunks.enumerate() { - decoder.decode(chunk, i as u64)?; - while let FramePoll::Frame(frame) = decoder.try_next_frame()? { - on_frame(frame)?; - } - } - // End of stream: decodes the trailing picture, emits what reordering held - // back, and closes the source. - decoder.finish()?; - while let Some(frame) = pollster::block_on(decoder.next_frame())? { - on_frame(frame)?; - } - Ok(()) -} - -/// Write one decoded frame as NV12 to `output`, when an output is set. -pub fn write_nv12( - frame: &DecodedFrame, - readback: &mut Option, - output: &mut Option, -) -> Result<(), Box> { - if let (Some(file), Some(readback)) = (output.as_mut(), readback.as_mut()) { - let data = readback.read(frame)?; - file.write_all(&data.y)?; - file.write_all(&data.uv)?; - } - Ok(()) -} - -/// Find a memory type satisfying `flags` for `reqs`, and allocate from it. -pub fn allocate( - device: &ash::Device, - props: &vk::PhysicalDeviceMemoryProperties, - reqs: vk::MemoryRequirements, - flags: vk::MemoryPropertyFlags, -) -> Result> { - let index = (0..props.memory_type_count) - .find(|&i| { - reqs.memory_type_bits & (1 << i) != 0 - && props.memory_types[i as usize] - .property_flags - .contains(flags) - }) - .ok_or("no suitable memory type")?; - let memory = unsafe { - device.allocate_memory( - &vk::MemoryAllocateInfo::default() - .allocation_size(reqs.size) - .memory_type_index(index), - None, - ) - }?; - Ok(memory) -} - -/// One frame's pixels on the host, cropped to the visible region. -pub struct FrameData { - pub y: Vec, - pub uv: Vec, -} - -/// Copies decoded frames into host memory, reusing one staging buffer. -pub struct Readback { - /// Keeps the device alive: `VideoContext` is an `Arc` handle, and this - /// struct's objects must outlive nothing and be destroyed before the device - /// is. Holding a clone makes that ordering automatic rather than something - /// the caller has to get right. - _context: VideoContext, - device: ash::Device, - queue: vk::Queue, - pool: vk::CommandPool, - command_buffer: vk::CommandBuffer, - fence: vk::Fence, - memory_properties: vk::PhysicalDeviceMemoryProperties, - /// Staging buffer and the size actually allocated for it. - buffer: Option<(vk::Buffer, vk::DeviceMemory, usize)>, -} - -impl Readback { - pub fn new(context: &VideoContext) -> Result> { - let device = context.device().clone(); - let family = context.transfer_queue_family(); - let pool = unsafe { - device.create_command_pool( - &vk::CommandPoolCreateInfo::default() - .queue_family_index(family) - .flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER), - None, - ) - }?; - let command_buffer = unsafe { - device.allocate_command_buffers( - &vk::CommandBufferAllocateInfo::default() - .command_pool(pool) - .level(vk::CommandBufferLevel::PRIMARY) - .command_buffer_count(1), - ) - }?[0]; - let fence = unsafe { device.create_fence(&vk::FenceCreateInfo::default(), None) }?; - let memory_properties = unsafe { - context - .instance() - .get_physical_device_memory_properties(context.physical_device()) - }; - Ok(Self { - _context: context.clone(), - device, - queue: context.transfer_queue(), - pool, - command_buffer, - fence, - memory_properties, - buffer: None, - }) - } - - /// Copy `frame` into host memory and return its visible region. - pub fn read(&mut self, frame: &DecodedFrame) -> Result> { - let bytes = match frame.bit_depth { - BitDepth::Eight => 1usize, - BitDepth::Ten => 2, - }; - // NV12/P010 geometry: full-resolution luma, half-resolution interleaved - // chroma, both at the image's coded width so the copy is contiguous. - let y_stride = frame.coded_width as usize * bytes; - let y_size = y_stride * frame.coded_height as usize; - let uv_stride = y_stride; - let uv_size = uv_stride * (frame.coded_height as usize / 2); - self.ensure_capacity(y_size + uv_size)?; - let (buffer, memory, _) = self.buffer.expect("just ensured"); - - let layers = |aspect| vk::ImageSubresourceLayers { - aspect_mask: aspect, - mip_level: 0, - base_array_layer: frame.array_layer, - layer_count: 1, - }; - let regions = [ - vk::BufferImageCopy2::default() - .buffer_offset(0) - .buffer_row_length(frame.coded_width) - .buffer_image_height(frame.coded_height) - .image_subresource(layers(vk::ImageAspectFlags::PLANE_0)) - .image_extent(vk::Extent3D { - width: frame.coded_width, - height: frame.coded_height, - depth: 1, - }), - vk::BufferImageCopy2::default() - .buffer_offset(y_size as u64) - .buffer_row_length(frame.coded_width / 2) - .buffer_image_height(frame.coded_height / 2) - .image_subresource(layers(vk::ImageAspectFlags::PLANE_1)) - .image_extent(vk::Extent3D { - width: frame.coded_width / 2, - height: frame.coded_height / 2, - depth: 1, - }), - ]; - - // GENERAL is already a valid source for a transfer read, so the - // zero-copy path needs no barrier at all. Anything else is a private - // image this frame owns, so moving it is safe. - let needs_transition = frame.layout != vk::ImageLayout::GENERAL; - let planes = vk::ImageAspectFlags::PLANE_0 | vk::ImageAspectFlags::PLANE_1; - let range = vk::ImageSubresourceRange { - aspect_mask: planes, - base_mip_level: 0, - level_count: 1, - base_array_layer: frame.array_layer, - layer_count: 1, - }; - - unsafe { - self.device - .reset_command_buffer(self.command_buffer, vk::CommandBufferResetFlags::empty())?; - self.device.begin_command_buffer( - self.command_buffer, - &vk::CommandBufferBeginInfo::default() - .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT), - )?; - - if needs_transition { - let to_src = [vk::ImageMemoryBarrier2::default() - .src_stage_mask(vk::PipelineStageFlags2::NONE) - .src_access_mask(vk::AccessFlags2::NONE) - .dst_stage_mask(vk::PipelineStageFlags2::COPY) - .dst_access_mask(vk::AccessFlags2::TRANSFER_READ) - .old_layout(frame.layout) - .new_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL) - .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .image(frame.image) - .subresource_range(range)]; - self.device.cmd_pipeline_barrier2( - self.command_buffer, - &vk::DependencyInfo::default().image_memory_barriers(&to_src), - ); - } - - self.device.cmd_copy_image_to_buffer2( - self.command_buffer, - &vk::CopyImageToBufferInfo2::default() - .src_image(frame.image) - .src_image_layout(if needs_transition { - vk::ImageLayout::TRANSFER_SRC_OPTIMAL - } else { - frame.layout - }) - .dst_buffer(buffer) - .regions(®ions), - ); - - if needs_transition { - let restore = [vk::ImageMemoryBarrier2::default() - .src_stage_mask(vk::PipelineStageFlags2::COPY) - .src_access_mask(vk::AccessFlags2::TRANSFER_READ) - .dst_stage_mask(vk::PipelineStageFlags2::NONE) - .dst_access_mask(vk::AccessFlags2::NONE) - .old_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL) - .new_layout(frame.layout) - .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .image(frame.image) - .subresource_range(range)]; - self.device.cmd_pipeline_barrier2( - self.command_buffer, - &vk::DependencyInfo::default().image_memory_barriers(&restore), - ); - } - - self.device.end_command_buffer(self.command_buffer)?; - let command_buffers = [self.command_buffer]; - self.device.reset_fences(&[self.fence])?; - self.device.queue_submit( - self.queue, - &[vk::SubmitInfo::default().command_buffers(&command_buffers)], - self.fence, - )?; - self.device.wait_for_fences(&[self.fence], true, u64::MAX)?; - } - - // Crop to the visible region on the way out: the image is coded-size, - // which is padded up to the codec's macroblock granularity. - // An odd size rounds the chroma up: the last chroma sample covers - // the last luma column (and row) on its own. - let visible_y_stride = frame.width as usize * bytes; - let visible_uv_stride = frame.width.div_ceil(2) as usize * 2 * bytes; - let uv_rows = frame.height.div_ceil(2) as usize; - let mut y = Vec::with_capacity(visible_y_stride * frame.height as usize); - let mut uv = Vec::with_capacity(visible_uv_stride * uv_rows); - unsafe { - // Map the whole allocation: a sub-range map has to respect - // nonCoherentAtomSize, and WHOLE_SIZE sidesteps that. - let ptr = - self.device - .map_memory(memory, 0, vk::WHOLE_SIZE, vk::MemoryMapFlags::empty())? - as *const u8; - for row in 0..frame.height as usize { - y.extend_from_slice(std::slice::from_raw_parts( - ptr.add(row * y_stride), - visible_y_stride, - )); - } - for row in 0..uv_rows { - uv.extend_from_slice(std::slice::from_raw_parts( - ptr.add(y_size + row * uv_stride), - visible_uv_stride, - )); - } - self.device.unmap_memory(memory); - } - Ok(FrameData { y, uv }) - } - - fn ensure_capacity(&mut self, size: usize) -> Result<(), Box> { - if let Some((_, _, capacity)) = self.buffer - && capacity >= size - { - return Ok(()); - } - unsafe { - if let Some((buffer, memory, _)) = self.buffer.take() { - self.device.device_wait_idle()?; - self.device.destroy_buffer(buffer, None); - self.device.free_memory(memory, None); - } - let buffer = self.device.create_buffer( - &vk::BufferCreateInfo::default() - .size(size as u64) - .usage(vk::BufferUsageFlags::TRANSFER_DST) - .sharing_mode(vk::SharingMode::EXCLUSIVE), - None, - )?; - let reqs = self.device.get_buffer_memory_requirements(buffer); - let wanted = - vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT; - let index = (0..self.memory_properties.memory_type_count) - .find(|&i| { - reqs.memory_type_bits & (1 << i) != 0 - && self.memory_properties.memory_types[i as usize] - .property_flags - .contains(wanted) - }) - .ok_or("no host-visible memory for readback")?; - let memory = self.device.allocate_memory( - &vk::MemoryAllocateInfo::default() - .allocation_size(reqs.size) - .memory_type_index(index), - None, - )?; - self.device.bind_buffer_memory(buffer, memory, 0)?; - // Record what was allocated, not what was asked for. - self.buffer = Some((buffer, memory, reqs.size as usize)); - } - Ok(()) - } -} - -impl Drop for Readback { - fn drop(&mut self) { - unsafe { - let _ = self.device.device_wait_idle(); - if let Some((buffer, memory, _)) = self.buffer.take() { - self.device.destroy_buffer(buffer, None); - self.device.free_memory(memory, None); - } - self.device.destroy_fence(self.fence, None); - self.device.destroy_command_pool(self.pool, None); - } - } -} diff --git a/examples/decode.rs b/examples/decode.rs index 58aac2e0..b686b8e6 100644 --- a/examples/decode.rs +++ b/examples/decode.rs @@ -1,131 +1,63 @@ -//! Decode an H.264, H.265 or AV1 stream to raw YUV. +//! Decode a stream to a raw YUV file. //! -//! The codec comes from the file extension: `.264`, `.265`/`.hevc`, `.obu` -//! (AV1 OBUs) or `.ivf` (AV1 in IVF). +//! Decoded frames stay on the GPU. Here [`YuvDownloader`] copies each one to +//! CPU memory to write it to the file. To show frames, sample them where they +//! are instead, as the `sample_frame` example does. //! //! Usage: -//! cargo run --example decode -- input.264 [output.yuv] +//! cargo run --example decode -- input.264 output.yuv //! -//! Produces the same layout as `ffmpeg -i input.264 -pix_fmt nv12 out.yuv`, -//! so the output can be compared directly: -//! -//! ffmpeg -i input.264 -pix_fmt nv12 reference.yuv -//! cmp reference.yuv output.yuv +//! The codec comes from the extension: `.264`, `.265` or `.obu` (AV1). The +//! output is NV12, or P010 for a 10-bit stream. +use pixelforge::decoder::{DecodeConfig, DecodedFrame, Decoder, FramePoll}; +use pixelforge::{Codec, VideoContextBuilder, YuvDownloader}; use std::fs::File; - -#[allow(dead_code)] -mod common; -use common::{Readback, decode_input, write_nv12}; - -use pixelforge::decoder::Decoder; -use pixelforge::vulkan::VideoContextBuilder; +use std::io::Write; fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); - - let mut args = std::env::args().skip(1); - let input_path = args.next().unwrap_or_else(|| { - eprintln!("usage: decode [output.yuv]"); - std::process::exit(1); - }); - let output_path = args.next(); - let (codec, input) = common::open_input(&input_path)?; - - // PIXELFORGE_VALIDATION=1 enables the Vulkan validation layers, which - // report the exact offending call for any API misuse. - let validation = std::env::var("PIXELFORGE_VALIDATION").is_ok(); - // Set PIXELFORGE_NO_UNIFIED_LAYOUTS=1 to exercise the copying fallback on - // hardware that would otherwise take the zero-copy path. - let mut builder = VideoContextBuilder::new() - .app_name("pixelforge-decode") - .require_decode(codec) - .enable_validation(validation); - if std::env::var("PIXELFORGE_NO_UNIFIED_LAYOUTS").is_ok() { - builder = builder.without_unified_image_layouts(); - } - let context = builder.build()?; - - println!( - "Device: {}", - unsafe { std::ffi::CStr::from_ptr(context.device_properties().device_name.as_ptr()) } - .to_string_lossy() - ); - - // Frames come out in presentation order, ready to show without sorting, - // and on hardware with unified image layouts without ever being copied: a - // picture waiting its turn stays pinned in the DPB slot it was decoded - // into, and so does the one handed to this loop. - // - // 64 KB of byte stream holds several coded frames, and every frame a - // `decode` call emits is outstanding at once. The default budget of 4 is - // sized for one frame per call, so raise it: past the budget the decoder - // copies pictures out instead of handing over its own images, which is - // still correct but gives up the zero-copy path. - let config = common::decode_config(codec, &input).with_output_depth(8); - let mut output = output_path.as_ref().map(File::create).transpose()?; - // Readback lives in the consumer, not in pixelforge: the decoder hands over - // a GPU image and a renderer would sample it in place. See examples/common. - // `VideoContext` is an `Arc` handle, so cloning it shares the device. - let mut readback = output - .is_some() - .then(|| Readback::new(&context)) - .transpose()?; - let mut decoder = Decoder::new(context, config)?; - - let start = std::time::Instant::now(); - let mut frame_count = 0usize; - let mut last_info: Option<(u32, u32, i32, bool)> = None; - let mut last_generation: Option = None; - - // One thread drives both halves: feed a chunk, then take whatever has - // become ready. `Pending` means the GPU is still working on frames in - // flight, not that there are none, so they are collected on a later pass. - // A renderer would instead `split()` the decoder and await `next_frame` on - // its own thread, which is what the `Decoder::split` docs show. - // `common::decode_input` owns that loop. - decode_input(&mut decoder, &input, |frame| { - write_nv12(&frame, &mut readback, &mut output)?; - // A rebuilt session means a new set of images, so anything caching - // per-image state has to notice. Reporting it here is what makes the - // field visible in an example rather than only in the docs. - if last_generation.is_some_and(|g| g != frame.generation) { - println!( - " generation {} -> {} at frame {}: the decoder rebuilt its images", - last_generation.unwrap(), - frame.generation, - frame_count - ); - } - last_generation = Some(frame.generation); - last_info = Some(( - frame.width, - frame.height, - frame.display_order, - frame.is_keyframe, - )); - frame_count += 1; - // Dropping the frame here is what hands its DPB slot back. Holding on - // to frames is what pushes the decoder onto the copying path. + let args: Vec = std::env::args().collect(); + let [_, input, output] = args.as_slice() else { + return Err("usage: decode ".into()); + }; + let codec = match input.rsplit('.').next() { + Some("264") => Codec::H264, + Some("265") => Codec::H265, + Some("obu") => Codec::AV1, + _ => return Err(format!("unknown stream extension in {input:?}").into()), + }; + let stream = std::fs::read(input)?; + + let context = VideoContextBuilder::new().require_decode(codec).build()?; + // A file can be cut anywhere, so the decoder finds the frames in the byte + // stream itself. + let config = DecodeConfig::new(codec).with_byte_stream(); + let mut decoder = Decoder::new(context.clone(), config)?; + let mut downloader = YuvDownloader::new(&context)?; + + let mut file = File::create(output)?; + let mut write = |frame: DecodedFrame| -> Result<(), Box> { + let yuv = downloader.download(&frame)?; + file.write_all(&yuv.y)?; + file.write_all(&yuv.uv)?; + // Dropping the frame gives its image back to the decoder. Ok(()) - })?; - let decode_time = start.elapsed(); - - println!( - "Decoded {} frames in {:.1?} ({:.1} fps)", - frame_count, - decode_time, - frame_count as f64 / decode_time.as_secs_f64() - ); - if let Some((w, h, order, key)) = last_info { - println!( - "Last frame: {}x{} display_order={} keyframe={}", - w, h, order, key - ); + }; + + // Feed the stream in pieces, as if from the network, and take each frame + // the GPU has finished. + for (index, piece) in stream.chunks(64 * 1024).enumerate() { + decoder.decode(piece, index as u64)?; + while let FramePoll::Frame(frame) = decoder.try_next_frame()? { + write(frame)?; + } } - if let Some(path) = output_path { - println!("Wrote raw frames to {}", path); + // End of stream: decode the last picture and take the frames still held + // back for display order. + decoder.finish()?; + while let Some(frame) = pollster::block_on(decoder.next_frame())? { + write(frame)?; } - + println!("Wrote {output}"); Ok(()) } diff --git a/examples/encode.rs b/examples/encode.rs index 8651454f..f0288057 100644 --- a/examples/encode.rs +++ b/examples/encode.rs @@ -1,157 +1,72 @@ -//! Encode raw YUV420 frames to H.264, H.265 or AV1. +//! Encode frames from CPU memory to a file. //! -//! Demonstrates encoding with PixelForge using Vulkan Video. Loads raw YUV420 -//! frames from `testdata/test_frames.yuv` and writes the bitstream to -//! `output.h264`, `output.h265` or `output.av1` depending on the codec. +//! Each frame is made here as I420, uploaded into the encoder's input image, +//! and encoded. The packets are written as they come, which gives a raw +//! stream: Annex B for H.264 and H.265, low-overhead OBUs for AV1. //! //! Usage: //! cargo run --example encode -- [h264|h265|av1] -//! -//! The codec defaults to `h264`. use pixelforge::{ - ChromaFormat, Codec, EncodeConfig, Encoder, Frame, RateControl, VideoContextBuilder, YuvLayout, - YuvUploader, + Codec, EncodeConfig, EncodedPacket, Encoder, Frame, RateControl, VideoContextBuilder, + YuvLayout, YuvUploader, }; +use std::collections::VecDeque; use std::fs::File; -use std::io::{Read, Write}; -use std::path::Path; -use tracing_subscriber::{Layer, layer::SubscriberExt, util::SubscriberInitExt}; +use std::io::Write; -const TEST_FRAMES_PATH: &str = "testdata/test_frames.yuv"; -const WIDTH: u32 = 320; -const HEIGHT: u32 = 240; +const WIDTH: u32 = 640; +const HEIGHT: u32 = 360; +const FRAMES: u32 = 90; fn main() -> Result<(), Box> { - // Initialize tracing. - tracing_subscriber::registry() - .with( - tracing_subscriber::fmt::layer().with_filter( - tracing_subscriber::EnvFilter::try_from_default_env() - .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")), - ), - ) - .init(); - - // Pick the codec. H.265 asks for an explicit format; the others take the - // default, which is also YUV420. - let codec_arg = std::env::args() - .nth(1) - .unwrap_or_else(|| "h264".to_string()); - let (codec, config, output_name) = match codec_arg.as_str() { - "h264" => ( - Codec::H264, - EncodeConfig::h264(WIDTH, HEIGHT), - "output.h264", - ), - "h265" => ( - Codec::H265, - EncodeConfig::h265(WIDTH, HEIGHT).with_chroma_format(ChromaFormat::Yuv420), - "output.h265", - ), - "av1" => (Codec::AV1, EncodeConfig::av1(WIDTH, HEIGHT), "output.av1"), - other => { - eprintln!("unknown codec '{other}', expected one of: h264, h265, av1"); - std::process::exit(1); - } + let (codec, path) = match std::env::args().nth(1).as_deref() { + None | Some("h264") => (Codec::H264, "output.h264"), + Some("h265") => (Codec::H265, "output.h265"), + Some("av1") => (Codec::AV1, "output.obu"), + Some(other) => return Err(format!("unknown codec {other:?}").into()), }; - println!("PixelForge {codec:?} Encode Example ({output_name})\n"); - - // Load test frames. - let test_path = Path::new(TEST_FRAMES_PATH); - if !test_path.exists() { - eprintln!("Test frames not found at '{TEST_FRAMES_PATH}'"); - eprintln!( - "Generate with: ffmpeg -f lavfi -i testsrc=duration=0.5:size=320x240:rate=30 -pix_fmt yuv420p -f rawvideo testdata/test_frames.yuv" - ); - return Ok(()); - } - - let mut yuv_data = Vec::new(); - File::open(test_path)?.read_to_end(&mut yuv_data)?; - - let frame_size = (WIDTH * HEIGHT * 3 / 2) as usize; - let num_frames = yuv_data.len() / frame_size; - println!( - "Input: {num_frames} frames, {WIDTH}x{HEIGHT} YUV420, {} bytes", - yuv_data.len() - ); - - // Create video context. - let context = VideoContextBuilder::new() - .app_name("Encode Example") - .enable_validation(cfg!(debug_assertions)) - .require_encode(codec) - .build()?; - - if !context.supports_encode(codec) { - eprintln!("{codec:?} encode not supported"); - return Ok(()); - } - - // Configure encoder. - let config = config - .with_rate_control(RateControl::Cqp { qp: 26 }) + let context = VideoContextBuilder::new().require_encode(codec).build()?; + let config = EncodeConfig::new(codec, WIDTH, HEIGHT) .with_frame_rate(30, 1) - .with_gop_size(30); - - println!( - "Config: {:?}, GOP={}\n", - config.rate_control, config.gop_size - ); - + .with_rate_control(RateControl::Cbr { bitrate: 2_000_000 }); let mut encoder = Encoder::new(context, config)?; - // Create input image for uploading frames. - let mut input_image = YuvUploader::for_encoder(&encoder)?; - let mut output = File::create(output_name)?; - let mut total_bytes = 0; - - // Each `encode()` returns a future that resolves with that frame's packet. - // Keep a few in flight (so capture/upload overlaps GPU encode) and drain the - // oldest once the pipeline is full, preserving submission order. - let mut pending: std::collections::VecDeque = - std::collections::VecDeque::new(); - - let mut write_packet = |packet: pixelforge::EncodedPacket, total: &mut usize| { - *total += packet.data.len(); - output.write_all(&packet.data)?; - println!( - " pts={:<2} dts={:<2}: {:>5} bytes, {:?}{}", - packet.pts, - packet.dts, - packet.data.len(), - packet.frame_type, - if packet.is_key_frame() { " [KEY]" } else { "" } - ); - Ok::<(), Box>(()) - }; - - // Encode frames. - for i in 0..num_frames { - let frame = &yuv_data[i * frame_size..(i + 1) * frame_size]; - - // Upload YUV420 data to the input image. - input_image.upload(YuvLayout::I420, frame)?; - - // Submit the frame (async) and keep the pipeline at most ~2 deep. - pending.push_back(encoder.submit(Frame::image(input_image.source()))?); - while pending.len() > 2 { - let packet = pollster::block_on(pending.pop_front().unwrap())?; - write_packet(packet, &mut total_bytes)?; + let mut uploader = YuvUploader::for_encoder(&encoder)?; + + let mut output = File::create(path)?; + let mut write = |packet: EncodedPacket| output.write_all(&packet.data); + let mut pending = VecDeque::new(); + for index in 0..FRAMES { + // Write the frame straight into the encoder's input image. + let slot = encoder.input_slot()?; + uploader.upload_to(&slot, YuvLayout::I420, &make_frame(index))?; + // This returns before the frame is encoded. The future resolves with + // the next packet of the stream. + pending.push_back(encoder.submit(Frame::input(slot))?); + // Keep a frame in flight, so the CPU and the GPU work at once. + while pending.len() > 1 { + write(pollster::block_on(pending.pop_front().unwrap())?)?; } } - - // Flush remaining frames: barrier, then drain the outstanding futures in order. + // Encode what is still held, and take the last packets. encoder.flush()?; - while let Some(future) = pending.pop_front() { - let packet = pollster::block_on(future)?; - write_packet(packet, &mut total_bytes)?; + for future in pending { + write(pollster::block_on(future)?)?; } - - let ratio = (num_frames * frame_size) as f64 / total_bytes as f64; - println!("\nEncoded {num_frames} frames, {total_bytes} bytes, {ratio:.1}:1 compression"); - println!("Output: {output_name}"); - + println!("Wrote {FRAMES} frames to {path}"); Ok(()) } + +/// Frame `index` of a pattern that moves: I420, a full-size Y plane, then +/// quarter-size U and V planes. +fn make_frame(index: u32) -> Vec { + let (width, height) = (WIDTH as usize, HEIGHT as usize); + let mut frame = vec![128u8; YuvLayout::I420.frame_size(WIDTH, HEIGHT)]; + for y in 0..height { + for x in 0..width { + frame[y * width + x] = ((x + y + 4 * index as usize) % 256) as u8; + } + } + frame +} diff --git a/examples/encode_gpu.rs b/examples/encode_gpu.rs new file mode 100644 index 00000000..5fbb8531 --- /dev/null +++ b/examples/encode_gpu.rs @@ -0,0 +1,382 @@ +//! Encode frames that are drawn on the GPU. +//! +//! The application draws RGB frames into a Vulkan image of its own and hands +//! the image to the encoder as a [`SourceImage`]. The encoder converts each +//! frame to YUV on the GPU and encodes it. No frame goes through CPU memory, +//! and the CPU does not wait between drawing and encoding: a timeline +//! semaphore orders the drawing before the encode, and the encoder's +//! [`SourceImage::released`] points order the next drawing after it. +//! +//! The drawing here is a clear to a colour that changes and a square that +//! moves. A renderer records its own commands in the same place. +//! +//! Usage: +//! cargo run --example encode_gpu -- [h264|h265|av1] + +use ash::vk::{self, TaggedStructure as _}; +use pixelforge::{ + Codec, ColorSpec, EncodeConfig, EncodedPacket, Encoder, Frame, Handoff, RateControl, RgbFormat, + RgbSource, Source, SourceImage, TimelinePoint, VideoContext, VideoContextBuilder, +}; +use std::collections::VecDeque; +use std::fs::File; +use std::io::Write; + +const WIDTH: u32 = 640; +const HEIGHT: u32 = 360; +const FRAMES: u32 = 90; +const SQUARE: u32 = 64; + +fn main() -> Result<(), Box> { + let (codec, path) = match std::env::args().nth(1).as_deref() { + None | Some("h264") => (Codec::H264, "output.h264"), + Some("h265") => (Codec::H265, "output.h265"), + Some("av1") => (Codec::AV1, "output.obu"), + Some(other) => return Err(format!("unknown codec {other:?}").into()), + }; + + let context = VideoContextBuilder::new().require_encode(codec).build()?; + // The frames are BGRA in sRGB. The encoder converts them to the stream's + // YUV (BT.709 by default). + let config = EncodeConfig::new(codec, WIDTH, HEIGHT) + .with_frame_rate(30, 1) + .with_rate_control(RateControl::Cbr { bitrate: 2_000_000 }) + .with_source(Source::Rgb(RgbSource::new( + RgbFormat::BGRA, + ColorSpec::Srgb, + ))); + let mut encoder = Encoder::new(context.clone(), config)?; + println!("Converting with {:?}", encoder.conversion()); + + let mut canvas = Canvas::new(&context)?; + // The canvas leaves the image in GENERAL, and the encoder gives it back + // in GENERAL. + let source = SourceImage::new( + &context, + canvas.image, + Handoff::Device { + layout: vk::ImageLayout::GENERAL, + }, + )?; + + let mut output = File::create(path)?; + let mut write = |packet: EncodedPacket| output.write_all(&packet.data); + let mut pending = VecDeque::new(); + for index in 0..FRAMES { + // Draw once the encoder no longer reads the image, and encode once + // the drawing is done. Both waits are on the GPU. + let drawn = canvas.draw(index, &source.released())?; + pending.push_back(encoder.submit(Frame::image(&source).wait(&[drawn]))?); + while pending.len() > 1 { + write(pollster::block_on(pending.pop_front().unwrap())?)?; + } + } + encoder.flush()?; + for future in pending { + write(pollster::block_on(future)?)?; + } + println!("Wrote {FRAMES} frames to {path}"); + + // The source waits for the frames that read the image, so it goes before + // the image. + drop(source); + drop(canvas); + Ok(()) +} + +/// The application's side: a BGRA image, and the commands that draw into it. +struct Canvas { + context: VideoContext, + image: vk::Image, + memory: vk::DeviceMemory, + /// A white square, copied into the image at a new place each frame. + square: vk::Buffer, + square_memory: vk::DeviceMemory, + pool: vk::CommandPool, + command_buffer: vk::CommandBuffer, + /// Signalled when the last drawing is done, so the command buffer can be + /// recorded again. + fence: vk::Fence, + /// Reaches `drawn` when frame `drawn` is drawn. + timeline: vk::Semaphore, + drawn: u64, +} + +impl Canvas { + fn new(context: &VideoContext) -> Result> { + let device = context.device(); + // The encoder reads the image with its colour converter (on the + // compute queue) or with a copy (on the transfer queue), so both + // families share it. The drawing runs on the compute queue. + let mut families = vec![ + context.compute_queue_family(), + context.transfer_queue_family(), + ]; + families.dedup(); + let sharing = if families.len() > 1 { + vk::SharingMode::CONCURRENT + } else { + vk::SharingMode::EXCLUSIVE + }; + let image = unsafe { + device.create_image( + &vk::ImageCreateInfo::default() + .image_type(vk::ImageType::TYPE_2D) + .format(RgbFormat::BGRA.vk_format()) + .extent(vk::Extent3D { + width: WIDTH, + height: HEIGHT, + depth: 1, + }) + .mip_levels(1) + .array_layers(1) + .samples(vk::SampleCountFlags::TYPE_1) + .tiling(vk::ImageTiling::OPTIMAL) + // Written by the drawing; sampled by the converter, or + // copied by the encoder. + .usage( + vk::ImageUsageFlags::TRANSFER_DST + | vk::ImageUsageFlags::SAMPLED + | vk::ImageUsageFlags::TRANSFER_SRC, + ) + .sharing_mode(sharing) + .queue_family_indices(&families), + None, + ) + }?; + let memory = allocate( + context, + unsafe { device.get_image_memory_requirements(image) }, + vk::MemoryPropertyFlags::DEVICE_LOCAL, + )?; + unsafe { device.bind_image_memory(image, memory, 0) }?; + + let size = (SQUARE * SQUARE * 4) as u64; + let square = unsafe { + device.create_buffer( + &vk::BufferCreateInfo::default() + .size(size) + .usage(vk::BufferUsageFlags::TRANSFER_SRC), + None, + ) + }?; + let square_memory = allocate( + context, + unsafe { device.get_buffer_memory_requirements(square) }, + vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT, + )?; + unsafe { + device.bind_buffer_memory(square, square_memory, 0)?; + let data = device.map_memory(square_memory, 0, size, vk::MemoryMapFlags::empty())?; + std::ptr::write_bytes(data as *mut u8, 0xFF, size as usize); + device.unmap_memory(square_memory); + } + + let pool = unsafe { + device.create_command_pool( + &vk::CommandPoolCreateInfo::default() + .queue_family_index(context.compute_queue_family()) + .flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER), + None, + ) + }?; + let command_buffer = unsafe { + device.allocate_command_buffers( + &vk::CommandBufferAllocateInfo::default() + .command_pool(pool) + .command_buffer_count(1), + ) + }?[0]; + let fence = unsafe { + device.create_fence( + &vk::FenceCreateInfo::default().flags(vk::FenceCreateFlags::SIGNALED), + None, + ) + }?; + let mut timeline_info = + vk::SemaphoreTypeCreateInfo::default().semaphore_type(vk::SemaphoreType::TIMELINE); + let timeline = unsafe { + device.create_semaphore( + &vk::SemaphoreCreateInfo::default().push(&mut timeline_info), + None, + ) + }?; + Ok(Self { + context: context.clone(), + image, + memory, + square, + square_memory, + pool, + command_buffer, + fence, + timeline, + drawn: 0, + }) + } + + /// Draw frame `index` once the points in `released` are reached, and + /// return the point where the drawing is done. + fn draw( + &mut self, + index: u32, + released: &[TimelinePoint], + ) -> Result> { + let device = self.context.device(); + let cb = self.command_buffer; + let range = vk::ImageSubresourceRange { + aspect_mask: vk::ImageAspectFlags::COLOR, + base_mip_level: 0, + level_count: 1, + base_array_layer: 0, + layer_count: 1, + }; + // The first frame starts from nothing. After that the image is in + // GENERAL, where the encoder gave it back. + let old_layout = if self.drawn == 0 { + vk::ImageLayout::UNDEFINED + } else { + vk::ImageLayout::GENERAL + }; + let to_general = vk::ImageMemoryBarrier::default() + .old_layout(old_layout) + .new_layout(vk::ImageLayout::GENERAL) + .dst_access_mask(vk::AccessFlags::TRANSFER_WRITE) + .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .image(self.image) + .subresource_range(range); + // The clear and the copy both write the image. + let between = vk::ImageMemoryBarrier::default() + .old_layout(vk::ImageLayout::GENERAL) + .new_layout(vk::ImageLayout::GENERAL) + .src_access_mask(vk::AccessFlags::TRANSFER_WRITE) + .dst_access_mask(vk::AccessFlags::TRANSFER_WRITE) + .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .image(self.image) + .subresource_range(range); + let t = index as f32 / FRAMES as f32; + let background = vk::ClearColorValue { + float32: [0.2 + 0.6 * t, 0.3, 0.8 - 0.6 * t, 1.0], + }; + let x = (index * 6) % (WIDTH - SQUARE); + let square_region = vk::BufferImageCopy::default() + .image_subresource(vk::ImageSubresourceLayers { + aspect_mask: vk::ImageAspectFlags::COLOR, + mip_level: 0, + base_array_layer: 0, + layer_count: 1, + }) + .image_offset(vk::Offset3D { + x: x as i32, + y: ((HEIGHT - SQUARE) / 2) as i32, + z: 0, + }) + .image_extent(vk::Extent3D { + width: SQUARE, + height: SQUARE, + depth: 1, + }); + + unsafe { + // The command buffer is free once the last drawing is done. + device.wait_for_fences(&[self.fence], true, u64::MAX)?; + device.reset_fences(&[self.fence])?; + device.begin_command_buffer( + cb, + &vk::CommandBufferBeginInfo::default() + .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT), + )?; + device.cmd_pipeline_barrier( + cb, + vk::PipelineStageFlags::TOP_OF_PIPE, + vk::PipelineStageFlags::TRANSFER, + vk::DependencyFlags::empty(), + &[], + &[], + &[to_general], + ); + device.cmd_clear_color_image( + cb, + self.image, + vk::ImageLayout::GENERAL, + &background, + &[range], + ); + device.cmd_pipeline_barrier( + cb, + vk::PipelineStageFlags::TRANSFER, + vk::PipelineStageFlags::TRANSFER, + vk::DependencyFlags::empty(), + &[], + &[], + &[between], + ); + device.cmd_copy_buffer_to_image( + cb, + self.square, + self.image, + vk::ImageLayout::GENERAL, + &[square_region], + ); + device.end_command_buffer(cb)?; + + // Wait for the encoder's reads, and signal the next value of the + // timeline when done. + self.drawn += 1; + let wait_semaphores: Vec = + released.iter().map(|p| p.semaphore).collect(); + let wait_values: Vec = released.iter().map(|p| p.value).collect(); + let wait_stages = vec![vk::PipelineStageFlags::ALL_COMMANDS; released.len()]; + let signal_values = [self.drawn]; + let mut timeline_info = vk::TimelineSemaphoreSubmitInfo::default() + .wait_semaphore_values(&wait_values) + .signal_semaphore_values(&signal_values); + let submit = vk::SubmitInfo::default() + .wait_semaphores(&wait_semaphores) + .wait_dst_stage_mask(&wait_stages) + .command_buffers(std::slice::from_ref(&cb)) + .signal_semaphores(std::slice::from_ref(&self.timeline)) + .push(&mut timeline_info); + device.queue_submit(self.context.compute_queue(), &[submit], self.fence)?; + } + Ok(TimelinePoint::new(self.timeline, self.drawn)) + } +} + +impl Drop for Canvas { + fn drop(&mut self) { + let device = self.context.device(); + unsafe { + let _ = device.wait_for_fences(&[self.fence], true, u64::MAX); + device.destroy_semaphore(self.timeline, None); + device.destroy_fence(self.fence, None); + device.destroy_command_pool(self.pool, None); + device.destroy_buffer(self.square, None); + device.free_memory(self.square_memory, None); + device.destroy_image(self.image, None); + device.free_memory(self.memory, None); + } + } +} + +/// Allocate memory for `requirements` with `flags`. +fn allocate( + context: &VideoContext, + requirements: vk::MemoryRequirements, + flags: vk::MemoryPropertyFlags, +) -> Result> { + let memory_type = context + .find_memory_type(requirements.memory_type_bits, flags) + .ok_or("no suitable memory type")?; + let memory = unsafe { + context.device().allocate_memory( + &vk::MemoryAllocateInfo::default() + .allocation_size(requirements.size) + .memory_type_index(memory_type), + None, + ) + }?; + Ok(memory) +} diff --git a/examples/query_capabilities.rs b/examples/query_capabilities.rs index aaff6a24..2bd490c4 100644 --- a/examples/query_capabilities.rs +++ b/examples/query_capabilities.rs @@ -1,399 +1,45 @@ -//! Example: Query Codec Capabilities +//! What this device can encode and decode. //! -//! This example demonstrates how to query video codec capabilities. -//! from the Vulkan video extensions. +//! Usage: +//! cargo run --example query_capabilities -use ash::vk; -use ash::vk::TaggedStructure; -use pixelforge::{Codec, VideoContextBuilder}; -use tracing_subscriber::{Layer, layer::SubscriberExt, util::SubscriberInitExt}; +use pixelforge::{ + BitDepth, ChromaFormat, Codec, ColorRange, RgbFormat, StreamColor, VideoContextBuilder, +}; fn main() -> Result<(), Box> { - // Initialize tracing. - tracing_subscriber::registry() - .with( - tracing_subscriber::fmt::layer().with_filter( - tracing_subscriber::EnvFilter::try_from_default_env() - .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")), - ), - ) - .init(); - - println!("PixelForge Codec Capabilities Example"); - println!("======================================\n"); - - // Build the video context. - let context = VideoContextBuilder::new() - .app_name("Capabilities Example") - .app_version(1, 0, 0) - .enable_validation(cfg!(debug_assertions)) - .build()?; - - println!("Video Context Created\n"); - - // Load video queue fn to get capabilities - let video_queue_fn = ash::khr::video_queue::Instance::load(context.entry(), context.instance()); - - // Query codec support. - println!("Codec Support:"); - println!("--------------"); - - let codecs = [Codec::H264, Codec::H265, Codec::AV1]; - - for codec in codecs { - println!("\n{:?}:", codec); - - // Check encode support. - let encode_supported = context.supports_encode(codec); - println!( - " Encode: {}", - if encode_supported { - "✓ Supported" - } else { - "✗ Not supported" - } - ); - - if encode_supported { - query_detailed_capabilities(&context, codec, &video_queue_fn)?; - } - } - - Ok(()) -} - -fn query_detailed_capabilities( - context: &pixelforge::VideoContext, - codec: Codec, - video_queue_fn: &ash::khr::video_queue::Instance, -) -> Result<(), Box> { - let physical_device = context.physical_device(); - - let combinations = [ - ( - vk::VideoChromaSubsamplingFlagsKHR::TYPE_420, - vk::VideoComponentBitDepthFlagsKHR::TYPE_8, - "4:2:0 8-bit", - ), - ( - vk::VideoChromaSubsamplingFlagsKHR::TYPE_444, - vk::VideoComponentBitDepthFlagsKHR::TYPE_8, - "4:4:4 8-bit", - ), - ( - vk::VideoChromaSubsamplingFlagsKHR::TYPE_420, - vk::VideoComponentBitDepthFlagsKHR::TYPE_10, - "4:2:0 10-bit", - ), - ( - vk::VideoChromaSubsamplingFlagsKHR::TYPE_444, - vk::VideoComponentBitDepthFlagsKHR::TYPE_10, - "4:4:4 10-bit", - ), - ]; - - for (subsampling, bit_depth, desc) in combinations { - println!(" Checking {}: ", desc); - - // Construct profile info - let (mut profile_info, mut h264_profile, mut h265_profile, mut av1_profile) = match codec { - Codec::H264 => { - let profile_idc = if subsampling == vk::VideoChromaSubsamplingFlagsKHR::TYPE_444 { - ash::vk::native::StdVideoH264ProfileIdc_STD_VIDEO_H264_PROFILE_IDC_HIGH_444_PREDICTIVE - } else { - ash::vk::native::StdVideoH264ProfileIdc_STD_VIDEO_H264_PROFILE_IDC_HIGH - }; - - let h264 = - vk::VideoEncodeH264ProfileInfoKHR::default().std_profile_idc(profile_idc); - let info = vk::VideoProfileInfoKHR::default() - .video_codec_operation(vk::VideoCodecOperationFlagsKHR::ENCODE_H264) - .chroma_subsampling(subsampling) - .luma_bit_depth(bit_depth) - .chroma_bit_depth(bit_depth); - (info, Some(h264), None, None) - } - Codec::H265 => { - let profile_idc = if subsampling == vk::VideoChromaSubsamplingFlagsKHR::TYPE_444 { - ash::vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_FORMAT_RANGE_EXTENSIONS - } else if bit_depth == vk::VideoComponentBitDepthFlagsKHR::TYPE_10 { - ash::vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN_10 - } else { - ash::vk::native::StdVideoH265ProfileIdc_STD_VIDEO_H265_PROFILE_IDC_MAIN - }; - - let h265 = - vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(profile_idc); - let info = vk::VideoProfileInfoKHR::default() - .video_codec_operation(vk::VideoCodecOperationFlagsKHR::ENCODE_H265) - .chroma_subsampling(subsampling) - .luma_bit_depth(bit_depth) - .chroma_bit_depth(bit_depth); - (info, None, Some(h265), None) - } - Codec::AV1 => { - let profile = if subsampling == vk::VideoChromaSubsamplingFlagsKHR::TYPE_444 { - ash::vk::native::StdVideoAV1Profile_STD_VIDEO_AV1_PROFILE_HIGH - } else { - ash::vk::native::StdVideoAV1Profile_STD_VIDEO_AV1_PROFILE_MAIN - }; - - let av1 = vk::VideoEncodeAV1ProfileInfoKHR::default().std_profile(profile); - let info = vk::VideoProfileInfoKHR::default() - .video_codec_operation(vk::VideoCodecOperationFlagsKHR::ENCODE_AV1) - .chroma_subsampling(subsampling) - .luma_bit_depth(bit_depth) - .chroma_bit_depth(bit_depth); - (info, None, None, Some(av1)) - } - }; - - if let Some(h264) = &mut h264_profile { - profile_info = profile_info.push(h264); - } - if let Some(h265) = &mut h265_profile { - profile_info = profile_info.push(h265); - } - if let Some(av1) = &mut av1_profile { - profile_info = profile_info.push(av1); - } - - // 1. Query Video Capabilities - let mut encode_caps = vk::VideoEncodeCapabilitiesKHR::default(); - let mut caps = vk::VideoCapabilitiesKHR::default().push(&mut encode_caps); - - let mut h264_caps = vk::VideoEncodeH264CapabilitiesKHR::default(); - let mut h265_caps = vk::VideoEncodeH265CapabilitiesKHR::default(); - let mut av1_caps = vk::VideoEncodeAV1CapabilitiesKHR::default(); - match codec { - Codec::H264 => caps = caps.push(&mut h264_caps), - Codec::H265 => caps = caps.push(&mut h265_caps), - Codec::AV1 => caps = caps.push(&mut av1_caps), - } - - let result = unsafe { - (video_queue_fn - .fp() - .get_physical_device_video_capabilities_khr)( - physical_device, - &profile_info, - &mut caps, - ) - }; - - if result != vk::Result::SUCCESS { - println!(" Not Supported ({:?})", result); + let context = VideoContextBuilder::new().build()?; + let name = + unsafe { std::ffi::CStr::from_ptr(context.device_properties().device_name.as_ptr()) }; + println!("Device: {}\n", name.to_string_lossy()); + + for codec in [Codec::H264, Codec::H265, Codec::AV1] { + println!("{codec:?}"); + println!(" decode: {}", context.supports_decode(codec)); + println!(" encode: {}", context.supports_encode(codec)); + if !context.supports_encode(codec) { continue; } - println!(" Supported"); - println!( - " Max Dimensions: {}x{}", - caps.max_coded_extent.width, caps.max_coded_extent.height - ); - println!( - " Max Reference Pictures: {}", - caps.max_active_reference_pictures - ); - println!(" Max DPB Slots: {}", caps.max_dpb_slots); - // The references one frame may predict from, which can be fewer than - // the active reference pictures. - match codec { - Codec::H264 => println!( - " Max L0 References (P / B), L1: {} / {}, {}", - h264_caps.max_p_picture_l0_reference_count, - h264_caps.max_b_picture_l0_reference_count, - h264_caps.max_l1_reference_count - ), - Codec::H265 => println!( - " Max L0 References (P / B), L1: {} / {}, {}", - h265_caps.max_p_picture_l0_reference_count, - h265_caps.max_b_picture_l0_reference_count, - h265_caps.max_l1_reference_count - ), - Codec::AV1 => println!( - " Max Single References: {} (names {:#09b})", - av1_caps.max_single_reference_count, av1_caps.single_reference_name_mask - ), - } - - // 2. Query Supported Formats - let mut format_props_count = 0; - let mut format_props_list = - vk::VideoProfileListInfoKHR::default().profiles(std::slice::from_ref(&profile_info)); - - // Check for Input Image support (VIDEO_ENCODE_SRC_KHR) - let format_info = vk::PhysicalDeviceVideoFormatInfoKHR::default() - .image_usage(vk::ImageUsageFlags::VIDEO_ENCODE_SRC_KHR) - .push(&mut format_props_list); - - let result = unsafe { - (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - physical_device, - &format_info, - &mut format_props_count, - std::ptr::null_mut(), - ) - }; - - if result == vk::Result::SUCCESS { - let mut format_props = - vec![vk::VideoFormatPropertiesKHR::default(); format_props_count as usize]; - unsafe { - let _ = (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - physical_device, - &format_info, - &mut format_props_count, - format_props.as_mut_ptr(), - ); - }; - println!(" Supported Input Formats (SRC):"); - for prop in format_props { - println!(" Format: {:?}", prop.format); - } - } - - // Check for DPB Image support (VIDEO_ENCODE_DPB_KHR) - let format_info = vk::PhysicalDeviceVideoFormatInfoKHR::default() - .image_usage(vk::ImageUsageFlags::VIDEO_ENCODE_DPB_KHR) - .push(&mut format_props_list); - - let mut format_props_count = 0; - let result = unsafe { - (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - physical_device, - &format_info, - &mut format_props_count, - std::ptr::null_mut(), - ) - }; - - if result == vk::Result::SUCCESS { - let mut format_props = - vec![vk::VideoFormatPropertiesKHR::default(); format_props_count as usize]; - unsafe { - let _ = (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - physical_device, - &format_info, - &mut format_props_count, - format_props.as_mut_ptr(), - ); - }; - println!(" Supported DPB Formats (DPB):"); - for prop in format_props { + for chroma_format in [ChromaFormat::Yuv420, ChromaFormat::Yuv444] { + for bit_depth in [BitDepth::Eight, BitDepth::Ten] { + if !context.supports_encode_format(codec, chroma_format, bit_depth) { + continue; + } println!( - " Format: {:?} create flags: {:?}{}", - prop.format, - prop.image_create_flags, - if prop - .image_create_flags - .contains(vk::ImageCreateFlags::MUTABLE_FORMAT) - { - " (per-plane views available)" - } else { - "" - } + " encode {chroma_format:?} {bit_depth:?}: B-frames {:?}, {} quality levels", + context.supports_b_frames(codec, chroma_format, bit_depth), + context.encode_quality_levels(codec, chroma_format, bit_depth), ); } } - } - - print_decode_image_flags(context)?; - - Ok(()) -} - -/// Report the image creation flags H.264 decode pictures allow. -/// -/// Worth its own section because `imageCreateFlags` varies by *usage*, not just -/// by format, and the answer a consumer cares about is the one for the usage -/// pixelforge actually creates pictures with. `MUTABLE_FORMAT` there means -/// decoded frames can have their luma and chroma planes viewed separately, so a -/// renderer can read them as two ordinary textures instead of needing a -/// sampler-YCbCr conversion. Asking about the DPB usage alone gives a different, -/// and for this purpose wrong, answer: both RADV and ANV report no flags at all -/// for that. -fn print_decode_image_flags( - context: &pixelforge::VideoContext, -) -> Result<(), Box> { - if !context.supports_decode(Codec::H264) { - return Ok(()); - } - println!("\nH.264 Decode Picture Image Flags"); - println!("---------------------------------"); - - let mut h264 = vk::VideoDecodeH264ProfileInfoKHR::default() - .std_profile_idc(ash::vk::native::StdVideoH264ProfileIdc_STD_VIDEO_H264_PROFILE_IDC_HIGH) - .picture_layout(vk::VideoDecodeH264PictureLayoutFlagsKHR::PROGRESSIVE); - let profile = vk::VideoProfileInfoKHR::default() - .video_codec_operation(vk::VideoCodecOperationFlagsKHR::DECODE_H264) - .chroma_subsampling(vk::VideoChromaSubsamplingFlagsKHR::TYPE_420) - .luma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8) - .chroma_bit_depth(vk::VideoComponentBitDepthFlagsKHR::TYPE_8) - .push(&mut h264); - - let dpb = vk::ImageUsageFlags::VIDEO_DECODE_DPB_KHR; - let coincide = dpb - | vk::ImageUsageFlags::VIDEO_DECODE_DST_KHR - | vk::ImageUsageFlags::TRANSFER_SRC - | vk::ImageUsageFlags::SAMPLED; - - let video_queue_fn = ash::khr::video_queue::Instance::load(context.entry(), context.instance()); - for (label, usage) in [ - ( - "pictures pixelforge creates (DPB|DST|SRC|SAMPLED)", - coincide, - ), - ("reference-only DPB", dpb), - ] { - let profiles = [profile]; - let mut list = vk::VideoProfileListInfoKHR::default().profiles(&profiles); - let info = vk::PhysicalDeviceVideoFormatInfoKHR::default() - .image_usage(usage) - .push(&mut list); - let mut count = 0u32; - let result = unsafe { - (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - context.physical_device(), - &info, - &mut count, - std::ptr::null_mut(), - ) + // RGB input that the encoder converts itself, with no shader. + let sdr = StreamColor::Sdr { + range: ColorRange::Limited, }; - if result != vk::Result::SUCCESS || count == 0 { - println!(" {label}: unsupported"); - continue; - } - let mut props = vec![vk::VideoFormatPropertiesKHR::default(); count as usize]; - unsafe { - let _ = (video_queue_fn - .fp() - .get_physical_device_video_format_properties_khr)( - context.physical_device(), - &info, - &mut count, - props.as_mut_ptr(), - ); - } - println!(" {label}:"); - for prop in props.iter().take(count as usize) { - let planes = prop - .image_create_flags - .contains(vk::ImageCreateFlags::MUTABLE_FORMAT); - println!( - " {:?} flags: {:?} per-plane views: {}", - prop.format, prop.image_create_flags, planes - ); - } + println!( + " RGB input converted by the encoder: {}", + context.supports_rgb_encode_input(codec, RgbFormat::BGRx, sdr) + ); } Ok(()) } diff --git a/examples/sample_frame.rs b/examples/sample_frame.rs index 894eaeb9..9967d668 100644 --- a/examples/sample_frame.rs +++ b/examples/sample_frame.rs @@ -26,13 +26,10 @@ use std::fs::File; use std::io::Write; use ash::vk::{self, TaggedStructure as _}; -use pixelforge::decoder::{DecodedFrame, Decoder}; +use pixelforge::Codec; +use pixelforge::decoder::{DecodeConfig, DecodedFrame, Decoder, FramePoll}; use pixelforge::vulkan::{VideoContext, VideoContextBuilder}; -#[allow(dead_code)] -mod common; -use common::decode_input; - const SHADER: &[u8] = include_bytes!("shader/sample_frame.spv"); fn main() -> Result<(), Box> { @@ -40,11 +37,17 @@ fn main() -> Result<(), Box> { let mut args = std::env::args().skip(1); let input_path = args.next().unwrap_or_else(|| { - eprintln!("usage: sample_frame [output.rgba]"); + eprintln!("usage: sample_frame [output.rgba]"); std::process::exit(1); }); let output_path = args.next(); - let (codec, input) = common::open_input(&input_path)?; + let codec = match input_path.rsplit('.').next() { + Some("264") => Codec::H264, + Some("265") => Codec::H265, + Some("obu") => Codec::AV1, + _ => return Err(format!("unknown stream extension in {input_path:?}").into()), + }; + let stream = std::fs::read(&input_path)?; let validation = std::env::var("PIXELFORGE_VALIDATION").is_ok(); let context = VideoContextBuilder::new() @@ -63,8 +66,8 @@ fn main() -> Result<(), Box> { // picture's sharing set. Without this, sampling a decoded image from a // family the decoder did not share with is undefined. let consumer_family = context.compute_queue_family(); - let config = common::decode_config(codec, &input) - .with_output_depth(8) + let config = DecodeConfig::new(codec) + .with_byte_stream() .with_consumer_queue_family(consumer_family); let mut output = output_path.as_ref().map(File::create).transpose()?; @@ -76,7 +79,7 @@ fn main() -> Result<(), Box> { let mut count = 0usize; let start = std::time::Instant::now(); - decode_input(&mut decoder, &input, |frame| { + let mut on_frame = |frame: DecodedFrame| -> Result<(), Box> { let sampler = match &mut sampler { Some(s) => s, none => none.insert(Sampler::new(&context, consumer_family, &frame)?), @@ -87,7 +90,18 @@ fn main() -> Result<(), Box> { } count += 1; Ok(()) - })?; + }; + for (index, piece) in stream.chunks(64 * 1024).enumerate() { + decoder.decode(piece, index as u64)?; + while let FramePoll::Frame(frame) = decoder.try_next_frame()? { + on_frame(frame)?; + } + } + decoder.finish()?; + while let Some(frame) = pollster::block_on(decoder.next_frame())? { + on_frame(frame)?; + } + drop(on_frame); let elapsed = start.elapsed(); println!( diff --git a/scripts/b0_compare.rs b/scripts/b0_compare.rs new file mode 100644 index 00000000..0276baad --- /dev/null +++ b/scripts/b0_compare.rs @@ -0,0 +1,79 @@ +//! Encodes a generated sequence, for `scripts/compare_b0.sh`. +//! +//! The script copies this file into `examples/` of two builds and compares +//! what they write. It only uses the API that both builds have. +//! +//! Usage: b0_compare + +use pixelforge::{ + EncodeConfig, Encoder, Frame, RateControl, VideoContextBuilder, YuvLayout, YuvUploader, +}; +use std::io::Write; + +const WIDTH: u32 = 320; +const HEIGHT: u32 = 240; + +/// A frame that moves, so that P-frames are not empty. +fn frame(index: usize) -> Vec { + let (w, h) = (WIDTH as usize, HEIGHT as usize); + let mut data = vec![0u8; w * h * 3 / 2]; + for y in 0..h { + for x in 0..w { + data[y * w + x] = ((x + 3 * index) ^ (y + index)) as u8; + } + } + for y in 0..h / 2 { + for x in 0..w / 2 { + data[w * h + y * (w / 2) + x] = (x + index) as u8; + data[w * h + (w / 2) * (h / 2) + y * (w / 2) + x] = (y + 2 * index) as u8; + } + } + data +} + +fn main() -> Result<(), Box> { + let args: Vec = std::env::args().collect(); + if args.len() != 6 { + return Err("usage: b0_compare ".into()); + } + let config = match args[1].as_str() { + "h264" => EncodeConfig::h264(WIDTH, HEIGHT), + "h265" => EncodeConfig::h265(WIDTH, HEIGHT), + "av1" => EncodeConfig::av1(WIDTH, HEIGHT), + other => return Err(format!("unknown codec {other}").into()), + }; + let rate_control = match args[2].as_str() { + "cqp" => RateControl::Cqp { qp: 26 }, + "cbr" => RateControl::Cbr { bitrate: 500_000 }, + other => return Err(format!("unknown rate control {other}").into()), + }; + let config = config + .with_rate_control(rate_control) + .with_frame_rate(30, 1) + .with_gop_size(args[3].parse()?); + let frames: usize = args[4].parse()?; + + let context = VideoContextBuilder::new() + .app_name("b0 compare") + .require_encode(config.codec) + .build()?; + let mut encoder = Encoder::new(context, config)?; + let mut upload = YuvUploader::for_encoder(&encoder)?; + let mut output = std::fs::File::create(&args[5])?; + let mut pending = std::collections::VecDeque::new(); + for index in 0..frames { + upload.upload(YuvLayout::I420, &frame(index))?; + if index == frames / 2 { + encoder.request_idr(); + } + pending.push_back(encoder.submit(Frame::image(upload.source()))?); + while pending.len() > 2 { + output.write_all(&pollster::block_on(pending.pop_front().unwrap())?.data)?; + } + } + encoder.flush()?; + for future in pending { + output.write_all(&pollster::block_on(future)?.data)?; + } + Ok(()) +} diff --git a/scripts/compare_b0.sh b/scripts/compare_b0.sh new file mode 100755 index 00000000..8b1f08d2 --- /dev/null +++ b/scripts/compare_b0.sh @@ -0,0 +1,46 @@ +#!/usr/bin/env bash +# Checks that the encoder output without B-frames did not change. +# +# Builds a base revision (default: the v0.11.0 release commit) in a git +# worktree, encodes the same generated frames with it and with the working +# tree, and compares the streams with cmp. The bytes depend on the device and +# the driver, so both builds run here, on the same device, instead of comparing +# against a stored hash. +# +# Usage: scripts/compare_b0.sh [base revision] +set -euo pipefail + +base="${1:-b4e2d1f}" +root="$(git rev-parse --show-toplevel)" +work="$(mktemp -d)" +example="b0_compare" +trap 'rm -f "$root/examples/$example.rs"; git -C "$root" worktree remove --force "$work/base" 2>/dev/null || true; rm -rf "$work"' EXIT + +git -C "$root" worktree add --detach "$work/base" "$base" >/dev/null +cp "$root/scripts/$example.rs" "$work/base/examples/$example.rs" +cp "$root/scripts/$example.rs" "$root/examples/$example.rs" + +(cd "$work/base" && cargo build --quiet --example "$example" --target-dir "$root/target/b0-base") +(cd "$root" && cargo build --quiet --example "$example") + +failed=0 +for codec in h264 h265 av1; do + for rate in cqp cbr; do + # A GOP shorter than the sequence, and none at all: the second one + # runs past the wrap of the picture order count. + for gop in 30 0; do + name="$codec-$rate-gop$gop" + (cd "$work/base" && "$root/target/b0-base/debug/examples/$example" \ + "$codec" "$rate" "$gop" 600 "$work/$name.base") || { echo "SKIP $name (base failed)"; continue; } + (cd "$root" && "$root/target/debug/examples/$example" \ + "$codec" "$rate" "$gop" 600 "$work/$name.new") + if cmp -s "$work/$name.base" "$work/$name.new"; then + echo "same $name ($(stat -c %s "$work/$name.new") bytes)" + else + echo "DIFFERENT $name" + failed=1 + fi + done + done +done +exit "$failed" diff --git a/shader/README.md b/shader/README.md index 57e7b68f..47a0ae4f 100644 --- a/shader/README.md +++ b/shader/README.md @@ -20,6 +20,8 @@ version a Vulkan 1.1 device must accept: pixelforge also runs on a device the caller created, which can be 1.1. The `.spv` files are included in source via `include_bytes!`, so rerun this after editing a `.comp` and commit the result. -The examples carry one more, outside the library: `examples/shader/sample_frame.comp` -reads a decoded frame through a sampler-YCbCr conversion and writes RGBA, which -is what `examples/sample_frame.rs` uses to show the zero-copy render path. +Two more are outside the library. `examples/shader/sample_frame.comp` reads a +decoded frame through a sampler-YCbCr conversion and writes RGBA, which is what +`examples/sample_frame.rs` uses to show the zero-copy render path. +`tests/shader/sample_planes.comp` reads it through per-plane views, for +`tests/sample_planes.rs`. diff --git a/shader/compile.sh b/shader/compile.sh index 2f599267..1e430d5e 100755 --- a/shader/compile.sh +++ b/shader/compile.sh @@ -18,10 +18,11 @@ glslc -x glsl --target-env=vulkan1.1 -O \ FILE_SIZE=$(stat -c%s "$SPV_PATH" 2>/dev/null || stat -f%z "$SPV_PATH" 2>/dev/null) echo "Written $FILE_SIZE bytes to $SPV_PATH" -# The examples carry shaders of their own. -for name in sample_frame sample_planes; do - EXAMPLE_SRC="$PROJECT_ROOT/examples/shader/$name.comp" - EXAMPLE_SPV="$PROJECT_ROOT/examples/shader/$name.spv" +# An example and a test carry shaders of their own. +for path in examples/shader/sample_frame tests/shader/sample_planes; do + name="$(basename "$path")" + EXAMPLE_SRC="$PROJECT_ROOT/$path.comp" + EXAMPLE_SPV="$PROJECT_ROOT/$path.spv" echo "Compiling $name.comp → $name.spv" glslc -x glsl --target-env=vulkan1.1 -O "$EXAMPLE_SRC" -o "$EXAMPLE_SPV" FILE_SIZE=$(stat -c%s "$EXAMPLE_SPV" 2>/dev/null || stat -f%z "$EXAMPLE_SPV" 2>/dev/null) diff --git a/src/decoder/av1/stream_tests.rs b/src/decoder/av1/stream_tests.rs index 515f113f..64e75f48 100644 --- a/src/decoder/av1/stream_tests.rs +++ b/src/decoder/av1/stream_tests.rs @@ -20,6 +20,8 @@ //! | `aom_10bit` | 10-bit Main profile | //! | `aom_grain` | film grain (libaom denoiser) | //! | `svt_grain` | SVT-AV1 with film grain, `update_grain = 0` reuse | +//! | `svt_hier` | SVT-AV1 random access, three hierarchical levels of | +//! | | hidden frames | //! | `aom_errres` | error resilient frames, frame ids, `ref_order_hint`, | //! | | S-frames | //! | `aom_superres` | superres | @@ -50,13 +52,14 @@ macro_rules! fixture { }; } -const FIXTURES: [(&str, &[u8], &str); 10] = [ +const FIXTURES: [(&str, &[u8], &str); 11] = [ fixture!("aom_altref"), fixture!("aom_tiles"), fixture!("aom_tilegroups"), fixture!("aom_10bit"), fixture!("aom_grain"), fixture!("svt_grain"), + fixture!("svt_hier"), fixture!("aom_errres"), fixture!("aom_superres"), fixture!("aom_resize"), diff --git a/src/decoder/h264/parser.rs b/src/decoder/h264/parser.rs index 9850a723..1d581c5b 100644 --- a/src/decoder/h264/parser.rs +++ b/src/decoder/h264/parser.rs @@ -128,11 +128,24 @@ pub struct Sps { /// /// The number of frames that may precede any frame in decode order yet /// follow it in display order — i.e. the reorder buffer depth. `None` when - /// the stream does not signal it, in which case the decoder falls back to a - /// conservative bound. + /// the stream does not signal it, in which case [`Self::reorder_depth`] + /// infers it. pub max_num_reorder_frames: Option, } +/// Where a stream's reorder depth comes from. See [`Sps::reorder_depth`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReorderSource { + /// `max_num_reorder_frames` in the VUI. + Vui, + /// POC type 2: output order is decode order. + PocType, + /// A profile without B-frames. + Profile, + /// `MaxDpbFrames` of the level, which is what the standard infers. + Level, +} + impl Sps { /// Coded width in pixels (before cropping). pub fn coded_width(&self) -> u32 { @@ -171,6 +184,60 @@ impl Sps { (width, height) } + /// How many frames the reorder buffer must hold for this stream, and where + /// the number comes from. + /// + /// The VUI gives it when the stream has a bitstream restriction. Without + /// one the standard infers `MaxDpbFrames` (clause E.2.1), except where the + /// stream cannot reorder at all: POC type 2 makes output order equal decode + /// order, and the Baseline and intra-only profiles have no B-frames. + pub fn reorder_depth(&self) -> (usize, ReorderSource) { + if let Some(frames) = self.max_num_reorder_frames { + return (frames as usize, ReorderSource::Vui); + } + if self.pic_order_cnt_type == 2 { + return (0, ReorderSource::PocType); + } + // constraint_set3_flag marks the intra-only variants of the High + // profiles; 44 is CAVLC 4:4:4 Intra. + let constraint_set3 = self.constraint_set_flags & 0x10 != 0; + let intra_only = self.profile_idc == 44 + || (matches!(self.profile_idc, 100 | 110 | 122 | 244) && constraint_set3); + if self.profile_idc == 66 || intra_only { + return (0, ReorderSource::Profile); + } + (self.max_dpb_frames(), ReorderSource::Level) + } + + /// `MaxDpbFrames` (clause A.3.1): the DPB size the level allows for this + /// picture size, at most 16. + pub fn max_dpb_frames(&self) -> usize { + // MaxDpbMbs from Table A-1. Level 1b is coded as 11 with + // constraint_set3_flag (or as 9) and shares level 1's value. + let level_1b = self.level_idc == 9 + || (self.level_idc == 11 + && self.constraint_set_flags & 0x10 != 0 + && matches!(self.profile_idc, 66 | 77 | 88)); + let max_dpb_mbs: u32 = match self.level_idc { + _ if level_1b => 396, + 0..=10 => 396, + 11 => 900, + 12 | 13 | 20 => 2376, + 21 => 4752, + 22 | 30 => 8100, + 31 => 18000, + 32 => 20480, + 40 | 41 => 32768, + 42 => 34816, + 50 => 110_400, + 51 | 52 => 184_320, + // Level 6 and above; an unknown level gets the largest bound. + _ => 696_320, + }; + let frame_mbs = (self.coded_width() / 16) * (self.coded_height() / 16); + ((max_dpb_mbs / frame_mbs.max(1)) as usize).clamp(1, 16) + } + pub fn max_frame_num(&self) -> u32 { 1 << (self.log2_max_frame_num_minus4 as u32 + 4) } diff --git a/src/decoder/h264/parser_tests.rs b/src/decoder/h264/parser_tests.rs index b4c98618..1515c0fd 100644 --- a/src/decoder/h264/parser_tests.rs +++ b/src/decoder/h264/parser_tests.rs @@ -4,16 +4,29 @@ //! x264. Between them they cover the Baseline and High profiles, POC types 0 //! and 2, and streams with and without B-frames. //! +//! `bpyramid` (`bframes=3:b-pyramid=normal:weightb=1:direct=temporal`) and +//! `bstrict` (`bframes=3:b-pyramid=strict`) come from +//! `ffmpeg -c:v libx264 -crf 40 -x264-params b-adapt=0:scenecut=0:` on +//! testsrc2. `novui` and `novui_poc2` are `bframes` and `zerolatency` with the +//! VUI removed from the SPS by `tests/data/strip_vui.py`, so they carry no +//! `max_num_reorder_frames`. +//! //! These tests exercise the host-side parsing and DPB bookkeeping only; they //! need no GPU and so run everywhere. -use super::parser::{Mmco, NalType, iter_nal_units, parse_pps, parse_slice_header, parse_sps}; +use super::parser::{ + Mmco, NalType, ReorderSource, iter_nal_units, parse_pps, parse_slice_header, parse_sps, +}; const BASELINE: &[u8] = include_bytes!("../../../tests/data/base.264"); const ZEROLATENCY: &[u8] = include_bytes!("../../../tests/data/zerolatency.264"); const BFRAMES: &[u8] = include_bytes!("../../../tests/data/bframes.264"); /// 4 slices per picture: distinguishes "split per slice" from "split per picture". const MULTISLICE: &[u8] = include_bytes!("../../../tests/data/multislice.264"); +const BPYRAMID: &[u8] = include_bytes!("../../../tests/data/bpyramid.264"); +const BSTRICT: &[u8] = include_bytes!("../../../tests/data/bstrict.264"); +const NOVUI: &[u8] = include_bytes!("../../../tests/data/novui.264"); +const NOVUI_POC2: &[u8] = include_bytes!("../../../tests/data/novui_poc2.264"); /// Every fixture: one SPS, one PPS, 30 slices, first slice is an IDR. #[test] @@ -470,3 +483,87 @@ fn vui_reorder_matches_streams() { } } } + +/// Without a VUI the reorder depth is inferred: none where the stream cannot +/// reorder, and the level's DPB size otherwise. +#[test] +fn reorder_depth_without_vui() { + let sps_of = |data: &[u8]| { + let nal = iter_nal_units(data) + .find(|n| n.nal_type == NalType::Sps) + .unwrap(); + parse_sps(nal.payload()).unwrap() + }; + + for (name, data, expect) in [ + ("bframes", BFRAMES, (2, ReorderSource::Vui)), + ("bpyramid", BPYRAMID, (2, ReorderSource::Vui)), + ("bstrict", BSTRICT, (2, ReorderSource::Vui)), + ("zerolatency", ZEROLATENCY, (0, ReorderSource::Vui)), + // Level 1.3 at 320x240: MaxDpbMbs 2376 over 300 macroblocks. + ("novui", NOVUI, (7, ReorderSource::Level)), + ("novui_poc2", NOVUI_POC2, (0, ReorderSource::PocType)), + ] { + let sps = sps_of(data); + assert_eq!(sps.reorder_depth(), expect, "{name}"); + } + + // The rest of the SPS is untouched by removing the VUI. + let (with, without) = (sps_of(BFRAMES), sps_of(NOVUI)); + assert_eq!(without.max_num_reorder_frames, None); + assert_eq!( + super::parser::Sps { + max_num_reorder_frames: with.max_num_reorder_frames, + ..without + }, + with + ); + + // A Baseline stream has no B-frames, whatever its POC type and level. + let mut baseline = sps_of(BASELINE); + baseline.max_num_reorder_frames = None; + baseline.pic_order_cnt_type = 0; + assert_eq!(baseline.reorder_depth(), (0, ReorderSource::Profile)); + + // MaxDpbFrames is capped at 16, and follows the picture size. + let mut sps = sps_of(NOVUI); + sps.level_idc = 51; + assert_eq!(sps.reorder_depth(), (16, ReorderSource::Level)); + sps.level_idc = 41; + sps.pic_width_in_mbs_minus1 = 119; + sps.pic_height_in_map_units_minus1 = 67; + assert_eq!(sps.reorder_depth(), (4, ReorderSource::Level)); +} + +/// The pyramid fixtures use B-frames as references, each with its own way of +/// retiring them. +#[test] +fn pyramid_fixtures_have_reference_b_frames() { + for (name, data) in [("bpyramid", BPYRAMID), ("bstrict", BSTRICT)] { + let mut sps = None; + let mut pps = None; + let mut reference_b = 0; + let mut plain_b = 0; + for nal in iter_nal_units(data) { + match nal.nal_type { + NalType::Sps => sps = Some(parse_sps(nal.payload()).unwrap()), + NalType::Pps => pps = Some(parse_pps(nal.payload(), |_| Some(1)).unwrap()), + t if t.is_slice() => { + let header = + parse_slice_header(&nal, sps.as_ref().unwrap(), pps.as_ref().unwrap()) + .unwrap(); + if header.slice_type == super::parser::SliceType::B { + if nal.ref_idc != 0 { + reference_b += 1; + } else { + plain_b += 1; + } + } + } + _ => {} + } + } + assert!(reference_b > 0, "{name}: B-frames used as references"); + assert!(plain_b > 0, "{name}: non-reference B-frames"); + } +} diff --git a/src/decoder/h264/session.rs b/src/decoder/h264/session.rs index 53c8b50a..6e3fd882 100644 --- a/src/decoder/h264/session.rs +++ b/src/decoder/h264/session.rs @@ -184,13 +184,13 @@ impl H264Decoder { let (profile, chroma_format, bit_depth) = Self::profile_for(sps)?; // The stream needs max_num_ref_frames references plus the current - // picture. Reorder depth comes from the stream (VUI); without it, bound - // by the reference count, which is never smaller than the reorder depth - // in practice. - let reorder_depth = sps - .max_num_reorder_frames - .map(|v| v as usize) - .unwrap_or(sps.max_num_ref_frames as usize); + // picture. The reorder depth comes from the VUI, and is inferred when + // the stream has none. + let (reorder_depth, reorder_source) = sps.reorder_depth(); + debug!( + "H.264 decode: reorder depth {} (from {:?})", + reorder_depth, reorder_source + ); let needs = StreamNeeds { codec: "H.264", width: sps.coded_width(), diff --git a/src/decoder/h265/parser_tests.rs b/src/decoder/h265/parser_tests.rs index 709fe3cb..636f5b24 100644 --- a/src/decoder/h265/parser_tests.rs +++ b/src/decoder/h265/parser_tests.rs @@ -12,6 +12,7 @@ //! | `main10` | `keyint=15:bframes=2`, `-pix_fmt yuv420p10le`, 172x140 | //! | `temporal` | `temporal-layers=3:bframes=3:b-pyramid=1:keyint=20:scaling-list=default` | //! | `scaling` | `scaling-list=:bframes=2` | +//! | `bpyramid` | `bframes=4:b-pyramid=1:b-adapt=0:scenecut=0:keyint=20:min-keyint=20:open-gop=0` | //! //! `nvenc` is `hevc_nvenc -bf 3 -b_ref_mode middle -g 10 -rc constqp -qp 45`: //! unlike x265 it codes a short-term set in the SPS and has VUI HRD @@ -43,6 +44,7 @@ const NVENC: &[u8] = include_bytes!("../../../tests/data/nvenc.265"); const CRA_START: &[u8] = include_bytes!("../../../tests/data/cra_start.265"); const TEMPORAL: &[u8] = include_bytes!("../../../tests/data/temporal.265"); const SCALING: &[u8] = include_bytes!("../../../tests/data/scaling.265"); +const BPYRAMID: &[u8] = include_bytes!("../../../tests/data/bpyramid.265"); /// Every fixture, with the number of pictures it holds and the number ffmpeg /// outputs. @@ -55,6 +57,7 @@ const ALL: &[(&str, &[u8], usize, usize)] = &[ ("cra_start", CRA_START, 8, 6), ("temporal", TEMPORAL, 20, 20), ("scaling", SCALING, 10, 10), + ("bpyramid", BPYRAMID, 30, 30), ]; /// One slice segment with the parameter sets in effect for it. diff --git a/src/decoder/mod.rs b/src/decoder/mod.rs index 4129d536..862c7c4f 100644 --- a/src/decoder/mod.rs +++ b/src/decoder/mod.rs @@ -326,7 +326,7 @@ pub struct DecodedFrame { /// what the view is for, typically `SAMPLED`. Skipping it produces an /// invalid view, and only on the drivers where the frame really is the /// decoder's DPB image, so it passes on hardware that copies. See - /// `examples/sample_planes`. + /// `tests/sample_planes.rs`. pub plane_views: bool, /// Keeps this frame's storage reserved; releases it on drop. `None` for a /// frame that borrows the decoder's DPB image (see the validity rules). diff --git a/src/decoder/transfer.rs b/src/decoder/transfer.rs index 42d8d9ef..69a2333f 100644 --- a/src/decoder/transfer.rs +++ b/src/decoder/transfer.rs @@ -33,6 +33,11 @@ impl DecoderCommon { /// what the frame reports as its layout, so a consumer knows to transition /// it before reading. pub fn record_picture_copy(&self, frame: &DecodedPicture, dst_image: vk::Image) -> Result<()> { + // A picture that decoded something has waited for its slot already + // (`begin_decode_commands`). One that decoded nothing, such as an AV1 + // `show_existing_frame`, has not, and the slot's previous copy may + // still be running. + self.pipeline.wait_current_free(); let command_buffer = self.pipeline.current().transfer_command_buffer; let base_layer = frame.array_layer; let device = self.context.device().clone(); diff --git a/src/encoder/api.rs b/src/encoder/api.rs index bdca213f..73e281ff 100644 --- a/src/encoder/api.rs +++ b/src/encoder/api.rs @@ -24,6 +24,7 @@ pub(crate) trait EncoderApi: Send { fn common(&self) -> &EncoderCommon; fn input_image(&self) -> vk::Image; fn encode_frame(&mut self, input: FrameInput, sync: FrameSync) -> Result; + fn encode_frame_now(&mut self, input: FrameInput, sync: FrameSync) -> Result; fn flush(&mut self) -> Result<()>; fn request_idr(&mut self); fn invalidate_reference_frames(&mut self, first_lost_display_order: u64); @@ -40,6 +41,9 @@ impl EncoderApi for CodecEncoder { fn encode_frame(&mut self, input: FrameInput, sync: FrameSync) -> Result { CodecEncoder::encode_frame(self, input, sync) } + fn encode_frame_now(&mut self, input: FrameInput, sync: FrameSync) -> Result { + CodecEncoder::encode_frame_now(self, input, sync) + } fn flush(&mut self) -> Result<()> { CodecEncoder::flush(self) } @@ -93,6 +97,7 @@ impl Encoder { /// [`EncodeConfig::conversion`] asks for one that cannot. pub fn new(context: VideoContext, config: EncodeConfig) -> Result { config.rate_control.check(config.codec)?; + super::bframes::check_config(&config)?; let plan = plan(&context, &config)?; if let Some(format) = plan.rgb_input { super::rgb::check_rgb_input(&context, &config, format)?; @@ -201,7 +206,20 @@ impl Encoder { FrameSource::External(external) => external.fills_input_image(), _ => true, }; - let frame_slot = self.inner.common().pipeline.current_index(); + // The release of a frame written in place is its encode, which with + // B-frames may come in a later call, after the sync_file for it had + // to be handed out. + #[cfg(feature = "dmabuf")] + if frame.signal_sync_file + && matches!(frame.source, FrameSource::Input(_)) + && self.config().b_frames > 0 + { + return Err(PixelForgeError::InvalidInput( + "with B-frames, a frame written into an input slot has no release sync_file" + .to_string(), + )); + } + let frame_slot = self.inner.common().pipeline.next_index(); #[cfg_attr(not(feature = "dmabuf"), allow(unused_mut))] let mut sync = FrameSync { @@ -212,12 +230,14 @@ impl Encoder { #[cfg(feature = "dmabuf")] let slot = { // The slot's semaphores are free once the slot is. - let _ = self.inner.input_image(); - let slot = self.inner.common().pipeline.current_index(); + let slot = frame_slot; if frame.wait_sync_file.is_some() || frame.signal_sync_file { let files = match &mut self.sync_files { Some(files) => files, - files @ None => files.insert(super::external::SyncFiles::new(&self.context)?), + files @ None => files.insert(super::external::SyncFiles::new( + &self.context, + self.inner.common().pipeline.slot_count(), + )?), }; if let Some(fence) = frame.wait_sync_file { sync.waits.push(files.import_wait(slot, fence)?); @@ -330,7 +350,7 @@ impl Encoder { let common = self.inner.common(); // The encode writes nothing into its own slot's image, so the frame // stays where it is for the next repeat. - let other_slot = last != common.pipeline.current_index(); + let other_slot = last != common.pipeline.next_index(); let input = if other_slot { let slot = common.pipeline.slot(last); FrameInput::Direct { @@ -350,7 +370,10 @@ impl Encoder { } else { FrameInput::InPlace }; - let mut future = self.inner.encode_frame(input, FrameSync::default())?; + // Encoded now, with the frames that wait before it: no frame arrives + // after it to end their group, and its source may be another slot's + // image. + let mut future = self.inner.encode_frame_now(input, FrameSync::default())?; future.pts_offset = self.pts_offset; if other_slot { // The next frame may be written into that slot's image. @@ -426,6 +449,14 @@ impl Encoder { /// The window does not depend on how many references the device can /// predict from per frame. A device that predicts from one or two still /// keeps the whole window, and recovers from the older frames in it. + /// + /// With B-frames, packets come in decode order, so the client lost every + /// frame encoded after the lost one, also B-frames shown before it. The + /// encoder drops every reference from the first frame it encoded that is + /// shown at or after `first_lost_display_order`. The frames that wait for + /// their group are encoded later from the references that survive. If + /// none survives, the oldest frame that waits is encoded at once as the + /// IDR. pub fn invalidate_reference_frames(&mut self, first_lost_display_order: u64) { let local = first_lost_display_order.saturating_sub(self.pts_offset); self.inner.invalidate_reference_frames(local) diff --git a/src/encoder/av1/init.rs b/src/encoder/av1/init.rs index aa04ecec..ce6bf5da 100644 --- a/src/encoder/av1/init.rs +++ b/src/encoder/av1/init.rs @@ -1,9 +1,10 @@ use super::{Av1, MIN_BITSTREAM_BUFFER_SIZE, SUPERBLOCK_SIZE}; +use crate::encoder::references::ReferenceSet; use crate::converter::RgbFormat; use crate::encoder::EncodeConfig; use crate::encoder::codec::{ - CodecEncoder, CommonInitRequest, build_encoder_common, query_video_caps, + CodecEncoder, CommonInitRequest, build_encoder_common, check_quality_level, query_video_caps, }; use crate::encoder::profile::EncodeProfile; use crate::encoder::rgb::RgbConversionCaps; @@ -11,7 +12,7 @@ use crate::error::Result; use crate::vulkan::VideoContext; use ash::vk; use ash::vk::TaggedStructure; -use tracing::info; +use tracing::{debug, info}; impl Av1 { /// Create a new AV1 encoder. @@ -43,6 +44,7 @@ impl Av1 { capabilities = capabilities.push(&mut rgb_caps); } let caps = query_video_caps(&context, profile_info, &mut capabilities)?; + check_quality_level(&config, encode_caps.max_quality_levels)?; let rgb_caps = rgb_input.map(|_| RgbConversionCaps::from(&rgb_caps)); let init = build_encoder_common(CommonInitRequest { @@ -68,20 +70,56 @@ impl Av1 { // (required for AMD RADV). allow_layered_dpb: true, rgb_caps, + b_frame_support: crate::encoder::bframes::av1_support( + context.device_properties().vendor_id, + &av1_caps, + caps.max_active_reference_pictures, + ), })?; let reference_window = init.reference_window; let active_reference_count = init.active_reference_count; + // The past references of a frame that predicts from both directions: + // what the device allows for the names before `BWDREF`, less the one + // future reference. + let bidirectional_past_count = av1_caps + .max_bidirectional_compound_group1_reference_count + .min( + av1_caps + .max_bidirectional_compound_reference_count + .saturating_sub(1), + ) + .min(init.session_active_references.saturating_sub(1)); + debug!( + "AV1 bidirectional prediction: {} references ({} past names, {} future names), \ + name mask {:#09b}", + av1_caps.max_bidirectional_compound_reference_count, + av1_caps.max_bidirectional_compound_group1_reference_count, + av1_caps.max_bidirectional_compound_group2_reference_count, + av1_caps.bidirectional_compound_reference_name_mask + ); let common = init.common; let codec = Av1 { frame_num: 0, - order_hint: 0, header_data: None, - references: Vec::new(), - reference_window, + references: ReferenceSet::new(reference_window), active_reference_count, single_reference_name_mask: av1_caps.single_reference_name_mask, + bidirectional_reference_name_mask: av1_caps.bidirectional_compound_reference_name_mask, + bidirectional_past_count, + coded_size: super::coded_size( + config.dimensions.width, + config.dimensions.height, + av1_caps.coded_picture_alignment, + ), }; + debug!( + "AV1 coded picture alignment {}x{}: frames are coded at {}x{}", + av1_caps.coded_picture_alignment.width, + av1_caps.coded_picture_alignment.height, + codec.coded_size.width, + codec.coded_size.height + ); let mut encoder = CodecEncoder { common, codec }; let color_desc = config.color.description(); diff --git a/src/encoder/av1/mod.rs b/src/encoder/av1/mod.rs index 2e2d3147..a91802fe 100644 --- a/src/encoder/av1/mod.rs +++ b/src/encoder/av1/mod.rs @@ -13,13 +13,15 @@ mod init; mod record; +mod references; mod session_params; use crate::encoder::ColorDescription; use crate::encoder::codec::{EncoderCommon, FramePlan, PictureSetup, VideoCodec}; -use crate::encoder::pipeline::EncodeFuture; -use crate::error::Result; +use crate::encoder::references::{Reference, ReferenceSet}; +use crate::error::{PixelForgeError, Result}; use ash::vk; +use references::{FrameReferences, bidirectional_references, show_existing_frame}; /// Minimum bitstream buffer size. const MIN_BITSTREAM_BUFFER_SIZE: usize = 2 * 1024 * 1024; @@ -27,42 +29,60 @@ const MIN_BITSTREAM_BUFFER_SIZE: usize = 2 * 1024 * 1024; /// AV1 superblock size in pixels (64x64, matching use_128x128_superblock=0). pub const SUPERBLOCK_SIZE: u32 = 64; +/// What AV1 needs to know about a reference besides what every codec does. #[derive(Clone, Copy, Debug)] -pub(crate) struct ReferenceInfo { - pub dpb_slot: u8, - pub order_hint: u32, +pub(crate) struct Av1Reference { pub frame_type: u32, - /// Display order (`pts`) of this reference, for reference frame invalidation. - pub display_order: u64, } +pub(crate) type ReferenceInfo = Reference; + /// AV1-specific encoder state (multi-reference prediction). pub(crate) struct Av1 { frame_num: u32, - order_hint: u32, /// Cached sequence-header OBU (invalidated when session params change). header_data: Option>, - /// References held in the DPB, most-recent first. Each frame predicts - /// from the first `active_reference_count`; the rest are kept for - /// reference frame invalidation to fall back on. - references: Vec, - /// How many references stay in the DPB. - reference_window: u32, + /// References held in the DPB. Each frame predicts from the newest + /// `active_reference_count`; the rest are kept for reference frame + /// invalidation to fall back on. + references: ReferenceSet, /// How many of the newest references one frame predicts from. active_reference_count: u32, /// The reference names single-reference prediction may use (bit 0 is /// LAST), from `VkVideoEncodeAV1CapabilitiesKHR`. single_reference_name_mask: u32, + /// The reference names prediction from both directions may use. + bidirectional_reference_name_mask: u32, + /// How many references shown before it a frame that predicts from both + /// directions uses. It uses one shown after it. + bidirectional_past_count: u32, + /// The frame size the device codes: the display size aligned up to + /// `codedPictureAlignment`. The sequence header must allow it. + coded_size: vk::Extent2D, +} + +/// The frame size a device that codes pictures at `alignment` gives a frame +/// of `width` x `height`. The Vulkan spec lets the implementation enlarge the +/// coded frame to its alignment; the render size still crops to the display +/// size. An alignment of zero is treated as the AV1 minimum of 8. +fn coded_size(width: u32, height: u32, alignment: vk::Extent2D) -> vk::Extent2D { + let align = |value: u32, unit: u32| value.div_ceil(unit.max(1)) * unit.max(1); + vk::Extent2D { + width: align(width, alignment.width.max(8)), + height: align(height, alignment.height.max(8)), + } +} + +/// The order hint of the frame with POC `poc`: its place in display order +/// since the key frame, in the 8 bits the sequence header gives it. +fn order_hint(poc: i32) -> u32 { + (poc / 2) as u32 & 0xFF } impl Av1 { /// The references this frame predicts from: the newest held ones. fn used_references(&self) -> &[ReferenceInfo] { - let n = self - .references - .len() - .min(self.active_reference_count as usize); - &self.references[..n] + self.references.newest(self.active_reference_count as usize) } /// The DPB slot each of the 7 AV1 reference names predicts from, or -1. @@ -77,9 +97,64 @@ impl Av1 { } names } + + /// The references `plan` predicts from, under their names, and the + /// distinct ones among them. None for a key frame. + /// + /// A frame that predicts from one direction uses the newest references. + /// A B-frame uses the nearest ones shown before it and the nearest one + /// shown after it. + fn frame_references(&self, plan: &FramePlan) -> Result<(FrameReferences, Vec)> { + if plan.is_idr() || self.references.is_empty() { + let none = FrameReferences { + names: [-1; 7], + fill: [0; 2], + }; + return Ok((none, Vec::new())); + } + if !plan.is_b() { + let newest = self.references.as_slice()[0].dpb_slot as i8; + let references = FrameReferences { + names: self.reference_names(), + fill: [newest; 2], + }; + return Ok((references, self.used_references().to_vec())); + } + + let (past, future) = self.references.around(plan.pic_order_cnt()); + let slots = |list: &[ReferenceInfo]| list.iter().map(|r| r.dpb_slot).collect::>(); + let references = bidirectional_references( + &slots(&past), + &slots(&future), + self.bidirectional_reference_name_mask, + self.bidirectional_past_count as usize, + ) + .ok_or_else(|| { + PixelForgeError::InvalidInput(format!( + "AV1 B-frame {} has {} references before it and {} after it", + plan.display_order, + past.len(), + future.len() + )) + })?; + let used = self + .references + .as_slice() + .iter() + .filter(|r| references.names.contains(&i32::from(r.dpb_slot))) + .copied() + .collect(); + Ok((references, used)) + } } impl VideoCodec for Av1 { + const HIDDEN_FRAMES: bool = true; + + fn show_packet(&self, _plan: &FramePlan, dpb_slot: u8) -> Vec { + show_existing_frame(dpb_slot) + } + fn begin_picture( &mut self, common: &mut EncoderCommon, @@ -88,12 +163,21 @@ impl VideoCodec for Av1 { let is_key = plan.is_idr(); if is_key { self.frame_num = 0; - self.order_hint = 0; self.references.clear(); + } else if plan.is_reference() { + // A reference frame makes the B-frame references of its level + // and below of no more use. They go out of the list, and their + // frame buffers are taken by later frames. + self.references.drop_obsolete(plan.pyramid_level()); + } + + // A hidden frame's bitstream goes in front of the next packet, which + // has the temporal delimiter. + if plan.hidden { + return Ok(PictureSetup { header: None }); } - // Every temporal unit starts with a Temporal Delimiter OBU; key frames - // also carry the sequence header (so decoders can initialize). + // Every temporal unit starts with a temporal delimiter OBU. let mut header = vec![0x12, 0x00]; if is_key { if self.header_data.is_none() { @@ -109,54 +193,35 @@ impl VideoCodec for Av1 { }) } - fn record_picture( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + fn record_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { self.record(common, plan) } fn end_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) { - let is_key = plan.is_idr(); - // Order hint used while recording this frame (pre-increment). - let encoded_order_hint = self.order_hint; self.frame_num += 1; - self.order_hint = (self.order_hint + 1) & 0xFF; + // A frame that is not a reference goes into no frame buffer, and the + // next frame takes its DPB slot. + if !plan.is_reference() { + return; + } - let ref_info = ReferenceInfo { + let frame_type = if plan.is_idr() { + ash::vk::native::StdVideoAV1FrameType_STD_VIDEO_AV1_FRAME_TYPE_KEY + } else { + ash::vk::native::StdVideoAV1FrameType_STD_VIDEO_AV1_FRAME_TYPE_INTER + }; + self.references.insert(Reference { dpb_slot: common.current_dpb_slot, - order_hint: encoded_order_hint, + poc: plan.pic_order_cnt(), display_order: plan.display_order, - frame_type: if is_key { - ash::vk::native::StdVideoAV1FrameType_STD_VIDEO_AV1_FRAME_TYPE_KEY - } else { - ash::vk::native::StdVideoAV1FrameType_STD_VIDEO_AV1_FRAME_TYPE_INTER - }, - }; - - if is_key { - self.references.clear(); - } - self.references.insert(0, ref_info); - // Keep a sliding window of the most recent references for prediction and - // loss recovery. - while self.references.len() > self.reference_window.max(1) as usize { - self.references.pop(); - } - - // Cycle to the next available DPB slot. - let used_slots: Vec = self.references.iter().map(|r| r.dpb_slot).collect(); - let mut next_slot = (common.current_dpb_slot + 1) % common.dpb_slot_count as u8; - if used_slots.contains(&next_slot) { - for i in 0..common.dpb_slot_count as u8 { - if !used_slots.contains(&i) { - next_slot = i; - break; - } - } - } - common.current_dpb_slot = next_slot; + encode_index: plan.encode_index, + level: plan.pyramid_level(), + codec: Av1Reference { frame_type }, + }); + + // Round-robin over the frame buffers, skipping the ones in use. + let next = (common.current_dpb_slot + 1) % common.dpb_slot_count as u8; + common.current_dpb_slot = self.references.free_slot(common.dpb_slot_count, Some(next)); } fn invalidate_references( @@ -164,13 +229,10 @@ impl VideoCodec for Av1 { _common: &mut EncoderCommon, first_lost_display_order: u64, ) -> bool { - // Drop every reference at or after the first lost frame: those are - // transitively undecodable on the client. Older survivors are kept, so - // the next P-frame re-anchors LAST_FRAME to the most recent of them (see - // `calculate_reference_frame_mapping`). Returns false only when no - // reference survives, in which case the caller forces a full key frame. - self.references - .retain(|r| r.display_order < first_lost_display_order); + // Every frame buffer is declared explicitly per frame from + // `references` (ref_frame_idx + ref_order_hint), so dropping entries + // here is enough for the decoder to follow. + self.references.invalidate(first_lost_display_order); !self.references.is_empty() } @@ -196,6 +258,19 @@ mod tests { assert_eq!(SUPERBLOCK_SIZE, 64); } + #[test] + fn the_coded_size_follows_the_alignment() { + let extent = |width, height| vk::Extent2D { width, height }; + let size = |w, h, a| { + let s = coded_size(w, h, a); + (s.width, s.height) + }; + assert_eq!(size(320, 240, extent(8, 8)), (320, 240)); + assert_eq!(size(320, 240, extent(64, 64)), (320, 256)); + assert_eq!(size(1920, 1080, extent(16, 16)), (1920, 1088)); + assert_eq!(size(1920, 1080, extent(0, 0)), (1920, 1080)); + } + #[test] fn test_superblock_alignment() { let align = |v: u32| (v + SUPERBLOCK_SIZE - 1) & !(SUPERBLOCK_SIZE - 1); @@ -207,224 +282,8 @@ mod tests { } #[test] - fn test_reference_info() { - let ref_info = ReferenceInfo { - dpb_slot: 2, - order_hint: 42, - frame_type: 0, - display_order: 0, - }; - assert_eq!(ref_info.dpb_slot, 2); - assert_eq!(ref_info.order_hint, 42); - let copied = ref_info; - assert_eq!(copied.dpb_slot, ref_info.dpb_slot); - assert_eq!(copied.order_hint, ref_info.order_hint); - } - - #[test] - fn test_order_hint_wrapping() { - let mut order_hint: u32 = 254; - for _ in 0..4 { - order_hint = (order_hint + 1) & 0xFF; - } - assert_eq!(order_hint, 2); - } - - #[test] - fn test_reference_tracking() { - let mut references: Vec = Vec::new(); - let max_refs = 4usize; - for i in 0..6u8 { - let ref_info = ReferenceInfo { - dpb_slot: i % max_refs as u8, - order_hint: i as u32, - frame_type: 0, - display_order: 0, - }; - references.insert(0, ref_info); - while references.len() > max_refs { - references.pop(); - } - } - assert_eq!(references.len(), max_refs); - assert_eq!(references[0].order_hint, 5); - assert_eq!(references[max_refs - 1].order_hint, 2); - } - - #[test] - fn test_key_frame_clears_references() { - let mut references: Vec = Vec::new(); - for i in 0..3u8 { - references.insert( - 0, - ReferenceInfo { - dpb_slot: i, - order_hint: i as u32, - frame_type: 0, - display_order: 0, - }, - ); - } - assert_eq!(references.len(), 3); - references.clear(); - assert!(references.is_empty()); - references.insert( - 0, - ReferenceInfo { - dpb_slot: 0, - order_hint: 0, - frame_type: 0, - display_order: 0, - }, - ); - assert_eq!(references.len(), 1); - assert_eq!(references[0].dpb_slot, 0); - assert_eq!(references[0].order_hint, 0); - } - - #[test] - fn test_dpb_slot_reuse() { - let max_refs = 2usize; - let dpb_slot_count = 3u8; - let mut references: Vec = Vec::new(); - let mut current_dpb_slot: u8 = 0; - - let find_free_slot = |refs: &[ReferenceInfo], slot_count: u8| -> u8 { - let used: Vec = refs.iter().map(|r| r.dpb_slot).collect(); - for i in 0..slot_count { - if !used.contains(&i) { - return i; - } - } - 0 - }; - - assert_eq!(current_dpb_slot, 0); - references.insert( - 0, - ReferenceInfo { - dpb_slot: 0, - order_hint: 0, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 1); - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 1, - order_hint: 1, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 2); - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 2, - order_hint: 2, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - assert_eq!(references.len(), 2); - assert_eq!(references[0].dpb_slot, 2); - assert_eq!(references[1].dpb_slot, 1); - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 0); - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 0, - order_hint: 3, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 1); - } - - #[test] - fn test_single_reference_slot() { - let max_refs = 1usize; - let dpb_slot_count = 2u8; - let mut references: Vec = Vec::new(); - - let find_free_slot = |refs: &[ReferenceInfo], slot_count: u8| -> u8 { - let used: Vec = refs.iter().map(|r| r.dpb_slot).collect(); - for i in 0..slot_count { - if !used.contains(&i) { - return i; - } - } - 0 - }; - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 0, - order_hint: 0, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - let mut current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 1); - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 1, - order_hint: 1, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - assert_eq!(references.len(), 1); - assert_eq!(references[0].dpb_slot, 1); - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 0); - - references.insert( - 0, - ReferenceInfo { - dpb_slot: 0, - order_hint: 2, - frame_type: 0, - display_order: 0, - }, - ); - while references.len() > max_refs { - references.pop(); - } - current_dpb_slot = find_free_slot(&references, dpb_slot_count); - assert_eq!(current_dpb_slot, 1); + fn order_hint_wraps_at_8_bits() { + assert_eq!(order_hint(2 * 254), 254); + assert_eq!(order_hint(2 * 258), 2); } } diff --git a/src/encoder/av1/record.rs b/src/encoder/av1/record.rs index 432714aa..a8ea0d11 100644 --- a/src/encoder/av1/record.rs +++ b/src/encoder/av1/record.rs @@ -1,27 +1,31 @@ //! AV1 per-frame encode recording: builds the StdVideo* graph for one frame and //! submits it. AV1 uses single-reference prediction. -use super::Av1; +use super::references::{FrameReferences, ref_frame_idx}; +use super::{Av1, order_hint}; use crate::encoder::codec::{EncodeCommands, EncoderCommon, FramePlan, RateControlPlan}; -use crate::encoder::pipeline::EncodeFuture; +use crate::encoder::gop::ReferencePattern; use crate::encoder::resources::{prepare_encode_command_buffer, record_dpb_barriers}; use crate::error::Result; use ash::vk; use ash::vk::TaggedStructure; use tracing::debug; +/// What a constant quantizer index is raised by for a B-frame that is not a +/// reference. About the step that 2 is for an H.264 QP. +const B_FRAME_Q_INDEX_OFFSET: u32 = 8; + impl Av1 { - pub(super) fn record( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + pub(super) fn record(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { let is_key_frame = plan.is_idr(); + let is_b = plan.is_b(); // All frames need a setup reference slot (DPB write) per the Vulkan spec - // when maxDpbSlots > 0. - let is_reference = true; + // when maxDpbSlots > 0, also a frame that no frame predicts from. + let has_setup_slot = true; let current_dpb_slot = common.current_dpb_slot; + let order_hint = order_hint(plan.pic_order_cnt()); + let (references, used_references) = self.frame_references(plan)?; let slot = common.pipeline.current(); let command_buffer = slot.encode_command_buffer; @@ -37,7 +41,7 @@ impl Av1 { current_dpb_slot ); - let rc = RateControlPlan::new(&common.config, 128); + let rc = RateControlPlan::for_frame(&common.config, 128, plan, B_FRAME_Q_INDEX_OFFSET, 255); let qp = rc.qp; unsafe { @@ -45,7 +49,7 @@ impl Av1 { } // Only the references this frame predicts from go to the encode. The // rest of the window stays in its DPB slots, untouched. - let ref_dpb_slots: Vec = self.used_references().iter().map(|r| r.dpb_slot).collect(); + let ref_dpb_slots: Vec = used_references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( common.context.sync2(), @@ -70,7 +74,7 @@ impl Av1 { _bitfield_align_1: [], _bitfield_1: Default::default(), }; - picture_info_flags.set_show_frame(1); + picture_info_flags.set_show_frame(!plan.hidden as u32); if is_key_frame { picture_info_flags.set_error_resilient_mode(1); } else { @@ -78,7 +82,8 @@ impl Av1 { } // Without MOTION_VECTOR_SCALING all picture resource codedExtents must - // match and equal the sequence header's max_frame dimensions. + // match. The device may code the frame larger, to its + // `codedPictureAlignment`; the sequence header allows that size. let frame_extent = vk::Extent2D { width: common.config.dimensions.width, height: common.config.dimensions.height, @@ -100,7 +105,7 @@ impl Av1 { flags: reference_info_flags, frame_type, RefFrameId: current_dpb_slot as u32, - OrderHint: self.order_hint as u8, + OrderHint: order_hint as u8, reserved1: [0; 3], pExtensionHeader: std::ptr::null(), }; @@ -121,7 +126,7 @@ impl Av1 { // than one AV1 reference name below, but each slot is declared once). let mut distinct_refs: Vec<&super::ReferenceInfo> = Vec::new(); if !is_key_frame { - for r in self.used_references() { + for r in &used_references { if !distinct_refs.iter().any(|d| d.dpb_slot == r.dpb_slot) { distinct_refs.push(r); } @@ -132,9 +137,9 @@ impl Av1 { for ref_info in &distinct_refs { ref_std_infos.push(ash::vk::native::StdVideoEncodeAV1ReferenceInfo { flags: reference_info_flags, - frame_type: ref_info.frame_type, + frame_type: ref_info.codec.frame_type, RefFrameId: ref_info.dpb_slot as u32, - OrderHint: ref_info.order_hint as u8, + OrderHint: super::order_hint(ref_info.poc) as u8, reserved1: [0; 3], pExtensionHeader: std::ptr::null(), }); @@ -188,14 +193,44 @@ impl Av1 { qm_v: 0, }; - let cdef = ash::vk::native::StdVideoAV1CDEF { - cdef_damping_minus_3: 0, - cdef_bits: 0, - cdef_y_pri_strength: [0; 8], - cdef_y_sec_strength: [0; 8], - cdef_uv_pri_strength: [0; 8], - cdef_uv_sec_strength: [0; 8], + // CDEF and loop restoration. A lossless frame (quantizer index 0) + // codes neither. The encoder picks one of four CDEF strengths for each + // block; the strengths are those of the Vulkan Video samples. NVIDIA + // (615.78.08) ignores the chroma strengths and writes its own. + let lossless = rc.is_disabled() && qp == 0; + let cdef = if lossless { + ash::vk::native::StdVideoAV1CDEF { + cdef_damping_minus_3: 0, + cdef_bits: 0, + cdef_y_pri_strength: [0; 8], + cdef_y_sec_strength: [0; 8], + cdef_uv_pri_strength: [0; 8], + cdef_uv_sec_strength: [0; 8], + } + } else { + ash::vk::native::StdVideoAV1CDEF { + cdef_damping_minus_3: 2, + cdef_bits: 2, + cdef_y_pri_strength: [0, 2, 4, 9, 0, 0, 0, 0], + cdef_y_sec_strength: [0; 8], + cdef_uv_pri_strength: [0; 8], + cdef_uv_sec_strength: [0; 8], + } + }; + // Loop restoration on luma, with 64x64 restoration units + // (`LoopRestorationSize` is the log2 of the size, less 5). NVIDIA + // (615.78.08) writes SGRPROJ, whatever type it gets. + let loop_restoration = ash::vk::native::StdVideoAV1LoopRestoration { + FrameRestorationType: [ + ash::vk::native::StdVideoAV1FrameRestorationType_STD_VIDEO_AV1_FRAME_RESTORATION_TYPE_SGRPROJ, + ash::vk::native::StdVideoAV1FrameRestorationType_STD_VIDEO_AV1_FRAME_RESTORATION_TYPE_NONE, + ash::vk::native::StdVideoAV1FrameRestorationType_STD_VIDEO_AV1_FRAME_RESTORATION_TYPE_NONE, + ], + LoopRestorationSize: [1, 1, 1], }; + if !lossless { + picture_info_flags.set_UsesLr(1); + } let loop_filter = ash::vk::native::StdVideoAV1LoopFilter { flags: ash::vk::native::StdVideoAV1LoopFilterFlags { @@ -211,14 +246,14 @@ impl Av1 { }; let (ref_frame_idx, ref_order_hint, primary_ref_frame, refresh_frame_flags) = - self.calculate_reference_frame_mapping(is_key_frame, current_dpb_slot); + self.calculate_reference_frame_mapping(plan, &references, current_dpb_slot); let std_picture_info = ash::vk::native::StdVideoEncodeAV1PictureInfo { flags: picture_info_flags, frame_type, frame_presentation_time: self.frame_num, current_frame_id: current_dpb_slot as u32, - order_hint: self.order_hint as u8, + order_hint: order_hint as u8, primary_ref_frame, refresh_frame_flags, coded_denom: 0, @@ -237,7 +272,11 @@ impl Av1 { pSegmentation: std::ptr::null(), pLoopFilter: &loop_filter, pCDEF: &cdef, - pLoopRestoration: std::ptr::null(), + pLoopRestoration: if lossless { + std::ptr::null() + } else { + &loop_restoration + }, pGlobalMotion: std::ptr::null(), pExtensionHeader: std::ptr::null(), pBufferRemovalTimes: std::ptr::null(), @@ -246,17 +285,18 @@ impl Av1 { // Map the AV1 reference names this frame predicts from to DPB slots // (see `reference_names`); the others stay -1. The names that are set // must mirror `ref_frame_idx` from `calculate_reference_frame_mapping`. - let reference_name_slot_indices = if !is_key_frame && !self.references.is_empty() { - self.reference_names() - } else { - [-1i32; 7] - }; + let reference_name_slot_indices = references.names; let (prediction_mode, rate_control_group) = if is_key_frame { ( vk::VideoEncodeAV1PredictionModeKHR::INTRA_ONLY, vk::VideoEncodeAV1RateControlGroupKHR::INTRA, ) + } else if is_b { + ( + vk::VideoEncodeAV1PredictionModeKHR::BIDIRECTIONAL_COMPOUND, + vk::VideoEncodeAV1RateControlGroupKHR::BIPREDICTIVE, + ) } else { ( vk::VideoEncodeAV1PredictionModeKHR::SINGLE_REFERENCE, @@ -293,7 +333,7 @@ impl Av1 { // Begin coding: setup slot (slot_index -1) plus any active reference slots. let mut all_reference_slots = Vec::new(); - if is_reference { + if has_setup_slot { let setup_slot_for_begin = vk::VideoReferenceSlotInfoKHR::default() .slot_index(-1) .picture_resource(&setup_picture_resource) @@ -305,11 +345,35 @@ impl Av1 { // Clamp GOP values to at least 1; a value of 0 is undefined in // Vulkan and causes undefined behavior on some drivers (RADV). let gop_frames = common.config.gop_size.max(1); - let codec_rate_control = vk::VideoEncodeAV1RateControlInfoKHR::default() + let mut codec_rate_control = vk::VideoEncodeAV1RateControlInfoKHR::default() .gop_frame_count(gop_frames) .key_frame_period(gop_frames) .consecutive_bipredictive_frame_count(0) .temporal_layer_count(1); + let b_frames = common.config.b_frames; + if b_frames > 0 { + // The count of B-frames has to be below the GOP length, so a GOP + // with no end is given the longest length there is. + let gop_frames = if common.config.gop_size == 0 { + u32::MAX + } else { + common.config.gop_size + }; + let pattern = match common.gop.reference_pattern() { + Some(ReferencePattern::Flat) => { + vk::VideoEncodeAV1RateControlFlagsKHR::REFERENCE_PATTERN_FLAT + } + Some(ReferencePattern::Dyadic) => { + vk::VideoEncodeAV1RateControlFlagsKHR::REFERENCE_PATTERN_DYADIC + } + None => vk::VideoEncodeAV1RateControlFlagsKHR::empty(), + }; + codec_rate_control = codec_rate_control + .gop_frame_count(gop_frames) + .key_frame_period(gop_frames) + .consecutive_bipredictive_frame_count(b_frames.min(gop_frames - 1)) + .flags(vk::VideoEncodeAV1RateControlFlagsKHR::REGULAR_GOP | pattern); + } let src_picture_resource = vk::VideoPictureResourceInfoKHR::default() .coded_offset(vk::Offset2D { x: 0, y: 0 }) @@ -322,7 +386,7 @@ impl Av1 { .dst_buffer(bitstream_buffer) .dst_buffer_offset(0) .dst_buffer_range(bitstream_buffer_size as u64); - if is_reference { + if has_setup_slot { encode_info = encode_info.setup_reference_slot(&setup_reference_slot); } if !reference_slots.is_empty() { @@ -355,11 +419,12 @@ impl Av1 { /// header enables order hints. `STD_VIDEO_AV1_PRIMARY_REF_NONE` is 7. fn calculate_reference_frame_mapping( &self, - is_key_frame: bool, + plan: &FramePlan, + references: &FrameReferences, current_dpb_slot: u8, ) -> ([i8; 7], [u8; 8], u8, u8) { const PRIMARY_REF_NONE: u8 = 7; - if is_key_frame { + if plan.is_idr() { // Key frame refreshes all 8 reference slots; all names point to slot 0. return ([0i8; 7], [0u8; 8], PRIMARY_REF_NONE, 0xFFu8); } @@ -367,19 +432,15 @@ impl Av1 { return ([0i8; 7], [0u8; 8], PRIMARY_REF_NONE, 0x00u8); } - let mut ref_frame_idx = [0i8; 7]; let mut ref_order_hint = [0u8; 8]; // The names this frame predicts from match `reference_name_slot_indices`. // The bitstream needs a slot for every name, so the others point at - // the most recent reference, which no block then uses. - let names = self.reference_names(); - let newest = self.references[0].dpb_slot as i8; - for (idx, &slot) in ref_frame_idx.iter_mut().zip(&names) { - *idx = if slot >= 0 { slot as i8 } else { newest }; - } + // a reference on their side, which no block then uses. + let names = references.names; + let ref_frame_idx = ref_frame_idx(references); // Order hints keyed by DPB slot, for every reference still held. - for r in &self.references { - ref_order_hint[r.dpb_slot as usize] = r.order_hint as u8; + for r in self.references.as_slice() { + ref_order_hint[r.dpb_slot as usize] = super::order_hint(r.poc) as u8; } // Load the frame context from the most recent reference, under the // first name it took, and refresh only the current slot so this frame @@ -388,7 +449,12 @@ impl Av1 { .iter() .position(|&slot| slot >= 0) .map_or(PRIMARY_REF_NONE, |name| name as u8); - let refresh_flags = 1u8 << current_dpb_slot; + // A frame that no frame predicts from goes into no frame buffer. + let refresh_flags = if plan.is_reference() { + 1u8 << current_dpb_slot + } else { + 0 + }; ( ref_frame_idx, ref_order_hint, diff --git a/src/encoder/av1/references.rs b/src/encoder/av1/references.rs new file mode 100644 index 00000000..9d1cf5ba --- /dev/null +++ b/src/encoder/av1/references.rs @@ -0,0 +1,136 @@ +//! Which references an AV1 frame predicts from, under which reference names, +//! and the packet that shows a hidden frame. +//! +//! AV1 has no B-frame type. A frame that predicts from both directions is an +//! inter frame with compound prediction from a past reference name (`LAST` +//! to `GOLDEN`) and a future one (`BWDREF` to `ALTREF`). The future reference +//! was encoded before it as a hidden frame (`show_frame = 0`), and a frame +//! header with `show_existing_frame = 1` shows that frame when its turn +//! comes. + +/// The first reference name that predicts from the future, `BWDREF`. The +/// names before it (`LAST`, `LAST2`, `LAST3`, `GOLDEN`) predict from the past. +const FIRST_FUTURE_NAME: usize = 4; + +/// The references of one frame. +#[derive(Debug, PartialEq, Eq)] +pub(super) struct FrameReferences { + /// The DPB slot each of the 7 reference names predicts from, or -1. + pub names: [i32; 7], + /// The DPB slot the names that are not predicted from point at in the + /// frame header, which needs a slot for every name: one for the past + /// names and one for the future names. + pub fill: [i8; 2], +} + +/// The references of a frame that predicts from both directions. +/// +/// `past` and `future` are the DPB slots of the references shown before the +/// frame and after it, each nearest first. Up to `past_count` of the past +/// ones take the past names in `mask`, and the nearest future one takes the +/// first future name in `mask`, where bit 0 of `mask` is `LAST`. `None` if +/// a side has no reference or no name. +pub(super) fn bidirectional_references( + past: &[u8], + future: &[u8], + mask: u32, + past_count: usize, +) -> Option { + let allowed = |name: &usize| mask & (1 << name) != 0; + let mut names = [-1i32; 7]; + let past_names = (0..FIRST_FUTURE_NAME).filter(allowed); + let mut assigned = 0; + for (name, &slot) in past_names.zip(past.iter().take(past_count)) { + names[name] = i32::from(slot); + assigned += 1; + } + let future_name = (FIRST_FUTURE_NAME..7).find(allowed)?; + names[future_name] = i32::from(*future.first()?); + if assigned == 0 { + return None; + } + Some(FrameReferences { + names, + fill: [past[0] as i8, future[0] as i8], + }) +} + +/// The slot each reference name has in the frame header: its own, or the +/// fill of its side. +pub(super) fn ref_frame_idx(references: &FrameReferences) -> [i8; 7] { + let mut idx = [0i8; 7]; + for (name, slot) in idx.iter_mut().enumerate() { + *slot = match references.names[name] { + -1 => references.fill[(name >= FIRST_FUTURE_NAME) as usize], + slot => slot as i8, + }; + } + idx +} + +/// A temporal unit that shows the frame in frame buffer `slot`: a temporal +/// delimiter and a frame header with `show_existing_frame = 1`. +/// +/// The header is `show_existing_frame` (1 bit), `frame_to_show_map_idx` (3 +/// bits) and the trailing bits. Nothing else is in it, because the sequence +/// header has no decoder model and no frame ids. +pub(super) fn show_existing_frame(slot: u8) -> Vec { + debug_assert!(slot < 8); + vec![ + // OBU_TEMPORAL_DELIMITER with a size field, of size 0. + 0x12, + 0x00, + // OBU_FRAME_HEADER with a size field, of size 1. + 0x1A, + 0x01, + 0x80 | (slot & 7) << 4 | 0x08, + ] +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn nearest_references_take_the_first_names() { + // Every name allowed: LAST and LAST2 from the past, BWDREF from the + // future. + let references = bidirectional_references(&[3, 1], &[2, 5], 0x7F, 2).unwrap(); + assert_eq!(references.names, [3, 1, -1, -1, 2, -1, -1]); + assert_eq!(references.fill, [3, 2]); + assert_eq!(ref_frame_idx(&references), [3, 1, 3, 3, 2, 2, 2]); + } + + #[test] + fn only_names_in_the_mask_are_used() { + // LAST and ALTREF only. + let references = bidirectional_references(&[3, 1], &[2], 0b100_0001, 2).unwrap(); + assert_eq!(references.names, [3, -1, -1, -1, -1, -1, 2]); + } + + #[test] + fn the_past_count_limits_the_past_names() { + let references = bidirectional_references(&[3, 1, 0], &[2], 0x7F, 1).unwrap(); + assert_eq!(references.names, [3, -1, -1, -1, 2, -1, -1]); + } + + #[test] + fn both_sides_are_needed() { + assert_eq!(bidirectional_references(&[], &[2], 0x7F, 1), None); + assert_eq!(bidirectional_references(&[3], &[], 0x7F, 1), None); + assert_eq!(bidirectional_references(&[3], &[2], 0x0F, 1), None); + assert_eq!(bidirectional_references(&[3], &[2], 0x70, 1), None); + } + + #[test] + fn the_show_packet_is_what_the_decoder_reads() { + use crate::decoder::av1::obu::{ObuType, iter_obus}; + let packet = show_existing_frame(5); + let obus: Vec<_> = iter_obus(&packet).collect::>().unwrap(); + assert_eq!(obus.len(), 2); + assert_eq!(obus[0].header.obu_type, ObuType::TemporalDelimiter); + assert_eq!(obus[1].header.obu_type, ObuType::FrameHeader); + // show_existing_frame = 1, frame_to_show_map_idx = 5, trailing bits. + assert_eq!(obus[1].payload, [0b1101_1000]); + } +} diff --git a/src/encoder/av1/session_params.rs b/src/encoder/av1/session_params.rs index 89d2d0bb..756d8dfe 100644 --- a/src/encoder/av1/session_params.rs +++ b/src/encoder/av1/session_params.rs @@ -18,8 +18,10 @@ impl Av1 { desc: &ColorDescription, ) -> Result { let config = &common.config; - let width = config.dimensions.width; - let height = config.dimensions.height; + // The device may code a frame larger than the display size (see + // `coded_size`), so the sequence header allows that size. + let width = self.coded_size.width; + let height = self.coded_size.height; let frame_width_bits = 32 - (width - 1).leading_zeros(); let frame_height_bits = 32 - (height - 1).leading_zeros(); @@ -102,7 +104,7 @@ impl Av1 { 0, // frame_id_numbers_present_flag 0, // enable_superres 1, // enable_cdef - 0, // enable_restoration + 1, // enable_restoration 0, // film_grain_params_present 0, // timing_info_present_flag 0, // initial_display_delay_present_flag @@ -157,7 +159,8 @@ impl Av1 { .std_decoder_model_info(&decoder_model_info) .std_operating_points(std::slice::from_ref(&operating_point)); - let mut quality_info = vk::VideoEncodeQualityLevelInfoKHR::default().quality_level(0); + let mut quality_info = vk::VideoEncodeQualityLevelInfoKHR::default() + .quality_level(common.config.quality_level); let session_params_create_info = vk::VideoSessionParametersCreateInfoKHR::default() .video_session(common.session) .push(&mut quality_info) diff --git a/src/encoder/bframes.rs b/src/encoder/bframes.rs new file mode 100644 index 00000000..0b34dbab --- /dev/null +++ b/src/encoder/bframes.rs @@ -0,0 +1,252 @@ +//! B-frame configuration, and what a device can do with B-frames. + +use ash::vk; + +use super::{Codec, EncodeConfig}; +use crate::error::{PixelForgeError, Result}; + +/// The most B-frames [`EncodeConfig::with_b_frames`] takes. +/// +/// Vulkan reports no limit on consecutive B-frames, only whether a frame can +/// predict from both directions, so this limit is pixelforge's own. +pub const MAX_B_FRAMES: u32 = 7; + +/// How the B-frames between two reference frames are referenced. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub enum BFrameReferences { + /// No B-frame is a reference. + #[default] + Flat, + /// The middle B-frame of each group is a reference for the B-frames on + /// either side of it, and so on for each half. With fewer than three + /// B-frames there is no middle one, and this is the same as `Flat`. + Pyramid, +} + +/// What a device can do with B-frames, for one codec and profile. See +/// [`VideoContext::supports_b_frames`](crate::VideoContext::supports_b_frames). +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum BFrameSupport { + /// No B-frames. + None, + /// B-frames that are not references. + Flat, + /// B-frames that are references too. + Pyramid, +} + +/// Whether the B-frames of `config` include references, which is what +/// [`BFrameSupport::Pyramid`] is needed for. +fn needs_pyramid(config: &EncodeConfig) -> bool { + config.b_frame_references == BFrameReferences::Pyramid && config.b_frames >= 3 +} + +/// Check the B-frame settings of `config` that need no device. +pub(crate) fn check_config(config: &EncodeConfig) -> Result<()> { + if config.b_frames > MAX_B_FRAMES { + return Err(PixelForgeError::InvalidInput(format!( + "{} B-frames requested, the most is {MAX_B_FRAMES}", + config.b_frames + ))); + } + Ok(()) +} + +/// Check that a device with `support` can encode the B-frames of `config`. +pub(crate) fn check_support(config: &EncodeConfig, support: BFrameSupport) -> Result<()> { + if config.b_frames == 0 { + return Ok(()); + } + let needed = if needs_pyramid(config) { + BFrameSupport::Pyramid + } else { + BFrameSupport::Flat + }; + if support < needed { + return Err(PixelForgeError::NoSuitableDevice(format!( + "{:?} encode: {} B-frames ({:?}) need {needed:?} B-frame support, \ + and the encoder has {support:?} on this device", + config.codec, config.b_frames, config.b_frame_references + ))); + } + Ok(()) +} + +/// The B-frames the encoder itself can encode for `codec`, whatever the +/// device can. +/// +/// An H.264 pyramid depends on how the driver writes the slice header: the +/// P-frame after a pyramid has to reorder its reference list, and the +/// drivers differ in what they write for that. It is checked on NVIDIA +/// (615.78.08) only, so it is offered there only. +fn implemented(codec: Codec, vendor_id: u32) -> BFrameSupport { + const NVIDIA: u32 = 0x10de; + match codec { + Codec::H264 if vendor_id != NVIDIA => BFrameSupport::Flat, + Codec::H264 | Codec::H265 | Codec::AV1 => BFrameSupport::Pyramid, + } +} + +/// How many references one frame may use at most, as the device reports +/// it, lowered by `PIXELFORGE_MAX_ACTIVE_REFERENCES` as the encoder lowers it. +fn max_active_references(device: u32) -> u32 { + std::env::var("PIXELFORGE_MAX_ACTIVE_REFERENCES") + .ok() + .and_then(|v| v.parse::().ok()) + .map_or(device, |cap| device.min(cap)) +} + +/// The B-frame support for `codec` from what the device reports for one +/// profile, capped by what the encoder implements. +fn support( + codec: Codec, + vendor_id: u32, + device: BFrameSupport, + max_active_references_device: u32, +) -> BFrameSupport { + // A B-frame predicts from one reference on each side at least. + if max_active_references(max_active_references_device) < 2 { + return BFrameSupport::None; + } + let implemented = implemented(codec, vendor_id); + let support = device.min(implemented); + tracing::debug!("{codec:?} B-frames: the device has {device:?}, the encoder {implemented:?}"); + support +} + +/// Flat B-frames need one reference in each list. Pyramid needs a B-frame +/// in each list as well. +fn h26x_support(l0: u32, l1: u32, b_in_l0: bool, b_in_l1: bool) -> BFrameSupport { + if l0 == 0 || l1 == 0 { + BFrameSupport::None + } else if b_in_l0 && b_in_l1 { + BFrameSupport::Pyramid + } else { + BFrameSupport::Flat + } +} + +/// The B-frame support of an H.264 encode profile. +pub(crate) fn h264_support( + vendor_id: u32, + caps: &vk::VideoEncodeH264CapabilitiesKHR<'_>, + max_active_references: u32, +) -> BFrameSupport { + let device = h26x_support( + caps.max_b_picture_l0_reference_count, + caps.max_l1_reference_count, + caps.flags + .contains(vk::VideoEncodeH264CapabilityFlagsKHR::B_FRAME_IN_L0_LIST), + caps.flags + .contains(vk::VideoEncodeH264CapabilityFlagsKHR::B_FRAME_IN_L1_LIST), + ); + support(Codec::H264, vendor_id, device, max_active_references) +} + +/// The B-frame support of an H.265 encode profile. +pub(crate) fn h265_support( + vendor_id: u32, + caps: &vk::VideoEncodeH265CapabilitiesKHR<'_>, + max_active_references: u32, +) -> BFrameSupport { + let device = h26x_support( + caps.max_b_picture_l0_reference_count, + caps.max_l1_reference_count, + caps.flags + .contains(vk::VideoEncodeH265CapabilityFlagsKHR::B_FRAME_IN_L0_LIST), + caps.flags + .contains(vk::VideoEncodeH265CapabilityFlagsKHR::B_FRAME_IN_L1_LIST), + ); + support(Codec::H265, vendor_id, device, max_active_references) +} + +/// The reference names that predict from the past, as bits of a Vulkan +/// reference name mask (bit 0 is `LAST`): `LAST` to `GOLDEN`. +const AV1_PAST_NAMES: u32 = 0b000_1111; +/// The reference names that predict from the future: `BWDREF` to `ALTREF`. +const AV1_FUTURE_NAMES: u32 = 0b111_0000; + +/// The B-frame support of an AV1 encode profile. +/// +/// A B-frame is a frame with bidirectional compound prediction, from a past +/// reference name and a future one. Any AV1 frame can be a reference, so +/// there is no further condition for pyramid. +pub(crate) fn av1_support( + vendor_id: u32, + caps: &vk::VideoEncodeAV1CapabilitiesKHR<'_>, + max_active_references: u32, +) -> BFrameSupport { + let mask = caps.bidirectional_compound_reference_name_mask; + let device = if caps.max_bidirectional_compound_reference_count >= 2 + && caps.max_bidirectional_compound_group1_reference_count >= 1 + && caps.max_bidirectional_compound_group2_reference_count >= 1 + && mask & AV1_PAST_NAMES != 0 + && mask & AV1_FUTURE_NAMES != 0 + { + BFrameSupport::Pyramid + } else { + BFrameSupport::None + }; + support(Codec::AV1, vendor_id, device, max_active_references) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn h26x_support_needs_both_lists() { + assert_eq!(h26x_support(0, 1, true, true), BFrameSupport::None); + assert_eq!(h26x_support(1, 0, true, true), BFrameSupport::None); + assert_eq!(h26x_support(1, 1, false, false), BFrameSupport::Flat); + assert_eq!(h26x_support(1, 1, true, false), BFrameSupport::Flat); + assert_eq!(h26x_support(2, 1, true, true), BFrameSupport::Pyramid); + } + + #[test] + fn support_needs_two_active_references() { + assert_eq!( + support(Codec::H264, 0x10de, BFrameSupport::Pyramid, 1), + BFrameSupport::None + ); + } + + #[test] + fn an_h264_pyramid_is_for_the_driver_it_was_checked_on() { + let pyramid = |codec, vendor| support(codec, vendor, BFrameSupport::Pyramid, 4); + assert_eq!(pyramid(Codec::H264, 0x10de), BFrameSupport::Pyramid); + assert_eq!(pyramid(Codec::H264, 0x1002), BFrameSupport::Flat); + assert_eq!(pyramid(Codec::H265, 0x1002), BFrameSupport::Pyramid); + assert_eq!(pyramid(Codec::AV1, 0x1002), BFrameSupport::Pyramid); + } + + #[test] + fn check_config_limits_the_count() { + let config = EncodeConfig::h264(64, 64).with_b_frames(MAX_B_FRAMES); + assert!(check_config(&config).is_ok()); + let config = EncodeConfig::h264(64, 64).with_b_frames(MAX_B_FRAMES + 1); + assert!(check_config(&config).is_err()); + } + + #[test] + fn check_support_asks_pyramid_only_with_a_middle_frame() { + let flat = EncodeConfig::h264(64, 64).with_b_frames(3); + assert!(check_support(&flat, BFrameSupport::Flat).is_ok()); + assert!(check_support(&flat, BFrameSupport::None).is_err()); + + let pyramid = flat.with_b_frame_references(BFrameReferences::Pyramid); + assert!(check_support(&pyramid, BFrameSupport::Flat).is_err()); + assert!(check_support(&pyramid, BFrameSupport::Pyramid).is_ok()); + + let short = EncodeConfig::h264(64, 64) + .with_b_frames(2) + .with_b_frame_references(BFrameReferences::Pyramid); + assert!(check_support(&short, BFrameSupport::Flat).is_ok()); + } + + #[test] + fn no_b_frames_need_no_support() { + let config = EncodeConfig::h264(64, 64); + assert!(check_support(&config, BFrameSupport::None).is_ok()); + } +} diff --git a/src/encoder/codec.rs b/src/encoder/codec.rs index 8f243ccc..21e287ce 100644 --- a/src/encoder/codec.rs +++ b/src/encoder/codec.rs @@ -16,10 +16,11 @@ use ash::vk; use ash::vk::TaggedStructure; use crate::converter::RgbFormat; -use crate::encoder::gop::{GopFrameType, GopPosition, GopStructure}; +use crate::encoder::bframes::check_support as check_b_frame_support; +use crate::encoder::gop::{GopFrameType, GopPlanner, GopPosition}; use crate::encoder::input::{FrameInput, FrameSync, SourceOwnership}; use crate::encoder::pipeline::{ - ENCODE_PIPELINE_DEPTH, EncodeFuture, EncodePipeline, PipelineConfig, SlotPacketMetadata, + Delivery, EncodeFuture, EncodePipeline, PipelineConfig, SlotPacketMetadata, }; use crate::encoder::profile::EncodeProfile; use crate::encoder::resources::{ @@ -29,7 +30,10 @@ use crate::encoder::resources::{ upload_image_to_input, }; use crate::encoder::rgb::RgbConversionCaps; -use crate::encoder::{ColorDescription, EncodeConfig, FrameType, RateControl, StreamColor}; +use crate::encoder::{ + BFrameSupport, ColorDescription, EncodeConfig, EncodedPacket, EncodedPacketStats, FrameType, + RateControl, StreamColor, +}; use crate::error::{PixelForgeError, Result}; use crate::sync::TimelinePoint; use crate::video::{TimelineChain, allocate_command_buffers}; @@ -61,8 +65,9 @@ pub(crate) struct EncoderCommon { /// Async slot rotation + bitstream readback. pub pipeline: EncodePipeline, - /// Frame-type/POC schedule. - pub gop: GopStructure, + /// Frame types, and the order frames are encoded in. Holds the frames + /// that wait for the reference frame after them. + pub gop: GopPlanner, /// Monotonic display-order counter (presentation order). pub input_frame_num: u64, /// Monotonic encode-order counter (decode order / DTS; `0` => first frame). @@ -89,17 +94,22 @@ pub(crate) struct EncoderCommon { pub upload_command_buffers: Vec, /// Signalled by each upload; the encode of the same frame waits on it. pub upload_timeline: TimelineChain, - /// What the current frame's encode submission waits for. + /// What the encode submission of the frame being encoded waits for. pub pending_waits: Vec>, - /// What the current frame's encode submission signals besides its - /// timeline: a release the caller asked for, when the encode is what + /// What the encode submission of the frame being encoded signals besides + /// its timeline: a release the caller asked for, when the encode is what /// reads the source last. pub pending_signals: Vec>, - /// What the current frame's encode reads. + /// What the encode of the frame being encoded reads. pub source: SourceBinding, - /// When the current frame's source is free again, if something before - /// the encode was the last to read it. - pub source_released: Option, + /// Where the bitstream of the frame being encoded goes. + pub delivery: Delivery, + /// The point the last encode submission signals. `None` until the frame + /// being encoded was submitted. + pub encode_point: Option, + /// The frames that were encoded and are not shown yet, see + /// [`VideoCodec::HIDDEN_FRAMES`]. + pub hidden_frames: Vec, /// The session's video profile, for images made after creation. #[cfg_attr(not(feature = "dmabuf"), allow(dead_code))] pub profile: EncodeProfile, @@ -108,30 +118,22 @@ pub(crate) struct EncoderCommon { } impl EncoderCommon { - /// Prologue: wait until the slot we are about to record over has been read - /// back, pull the next GOP position, and snapshot the counters this frame - /// will encode with. - pub fn begin_frame(&mut self) -> FramePlan { - self.pipeline.wait_current_free(); - let gop = self.gop.get_next_frame(); - let display_order = self.input_frame_num; - self.input_frame_num += 1; - FramePlan { - gop, - display_order, - encode_index: self.encode_frame_num, - } - } - - /// Set up what the current frame's encode reads, copying the source into - /// the slot's input image first when it has to. + /// Set up what a frame's encode reads, copying the source into the input + /// image of `slot` first when it has to. Returns the frame, ready to wait + /// for its encode, and the point where its source is free again if + /// something before the encode is the last to read it. /// /// `sync.waits` go to the frame's first submission and /// `sync.release_signals` to the one that is last to read the source: /// the copy when there is one, the encode otherwise. - pub fn prepare_source(&mut self, input: FrameInput, sync: FrameSync) -> Result<()> { + pub fn prepare_source( + &mut self, + slot: usize, + input: FrameInput, + sync: FrameSync, + ) -> Result<(HeldFrame, Option)> { let (slot_image, slot_view, slot_layout) = { - let slot = self.pipeline.current(); + let slot = self.pipeline.slot(slot); ( slot.input_image, slot.input_image_view, @@ -144,14 +146,9 @@ impl EncoderCommon { extent: None, ownership: None, }; - self.source_released = None; - self.pending_signals.clear(); - match input { - FrameInput::InPlace => { - self.source = slot_binding; - self.pending_waits = sync.waits; - self.pending_signals = sync.release_signals; - } + let mut released = None; + let (source, waits, signals) = match input { + FrameInput::InPlace => (slot_binding, sync.waits, sync.release_signals), FrameInput::Copy { image, ownership } => { let (semaphore, value) = self.upload_timeline.pending_signal(); let done = vk::SemaphoreSubmitInfo::default() @@ -161,8 +158,7 @@ impl EncoderCommon { let mut signals = vec![done]; signals.extend(sync.release_signals); let params = UploadParams { - upload_command_buffer: self.upload_command_buffers - [self.pipeline.current_index()], + upload_command_buffer: self.upload_command_buffers[slot], src_image: image, src_ownership: ownership, dst_image: slot_image, @@ -177,30 +173,49 @@ impl EncoderCommon { }; upload_image_to_input(&self.context, ¶ms)?; self.upload_timeline.commit(); - self.pipeline.current_mut().input_image_layout = + self.pipeline.slot_mut(slot).input_image_layout = vk::ImageLayout::VIDEO_ENCODE_SRC_KHR; let point = TimelinePoint::new(semaphore, value); - self.pending_waits = vec![point.wait_info()]; - self.source_released = Some(point); - self.source = slot_binding; + released = Some(point); + (slot_binding, vec![point.wait_info()], Vec::new()) } FrameInput::Direct { image, view, extent, ownership, - } => { - self.source = SourceBinding { + } => ( + SourceBinding { image, view, extent: Some(extent), ownership: Some(ownership), - }; - self.pending_waits = sync.waits; - self.pending_signals = sync.release_signals; - } - } - Ok(()) + }, + sync.waits, + sync.release_signals, + ), + }; + Ok(( + HeldFrame { + slot, + source, + waits, + signals, + submitted: std::time::Instant::now(), + }, + released, + )) + } + + /// Make `frame` the frame being encoded: the codec hooks record into its + /// slot and read its source from here on. + pub fn begin_encode(&mut self, frame: HeldFrame, delivery: Delivery) { + self.pipeline.begin_encode(frame.slot); + self.source = frame.source; + self.pending_waits = frame.waits; + self.pending_signals = frame.signals; + self.delivery = delivery; + self.encode_point = None; } /// The view the current frame's encode reads. @@ -293,26 +308,26 @@ impl EncoderCommon { self.pipeline.set_pending_metadata(metadata); } - /// Submit the recorded command buffer for the current slot, mark its DPB - /// slot active, and return the future that resolves with its packet. - pub fn submit_frame(&mut self) -> Result { + /// Submit the recorded command buffer for the current slot and mark its + /// DPB slot active. [`Self::encode_point`] is what the submission signals. + pub fn submit_frame(&mut self) -> Result<()> { let encode_queue = self.context.video_encode_queue().ok_or_else(|| { PixelForgeError::NoSuitableDevice("No video encode queue available".to_string()) })?; let waits = std::mem::take(&mut self.pending_waits); let signals = std::mem::take(&mut self.pending_signals); - let mut future = self.pipeline.submit_current( + let delivery = std::mem::replace(&mut self.delivery, Delivery::Packet); + let point = self.pipeline.submit_current( self.context.device(), self.context.sync2(), encode_queue, &waits, &signals, + delivery, )?; - if let Some(point) = self.source_released.take() { - future.set_released(point); - } + self.encode_point = Some(point); self.dpb_slot_active[self.current_dpb_slot as usize] = true; - Ok(future) + Ok(()) } /// Record one frame's encode into the current slot's command buffer, @@ -320,7 +335,7 @@ impl EncoderCommon { /// /// The codec builds what differs: the reference slots, the encode info /// with its picture info, and its rate control structs. - pub fn record_encode(&mut self, cmd: EncodeCommands<'_, R, L>) -> Result + pub fn record_encode(&mut self, cmd: EncodeCommands<'_, R, L>) -> Result<()> where R: for<'x> vk::Extends> + for<'x> vk::Extends> @@ -374,8 +389,8 @@ impl EncoderCommon { // FFmpeg; required for AMD RADV). if cmd.reset { let mut codec_rc = cmd.codec_rate_control; - let mut quality_level_info = - vk::VideoEncodeQualityLevelInfoKHR::default().quality_level(0); + let mut quality_level_info = vk::VideoEncodeQualityLevelInfoKHR::default() + .quality_level(self.config.quality_level); let control_info = vk::VideoCodingControlInfoKHR::default() .flags( vk::VideoCodingControlFlagsKHR::RESET @@ -425,11 +440,6 @@ impl EncoderCommon { self.submit_frame() } - /// Epilogue: advance to the next pipeline slot. - pub fn advance(&mut self) { - self.pipeline.advance(); - } - /// The `ash` device handle. pub fn device(&self) -> &ash::Device { self.context.device() @@ -449,14 +459,53 @@ pub(crate) struct SourceBinding { pub ownership: Option, } -/// Everything a single frame needs to know about its schedule, snapshotted by -/// [`EncoderCommon::begin_frame`] so the codec hooks all see a consistent view. +/// A submitted frame that waits for its encode: its pipeline slot, which holds +/// the pixels unless the encode reads the source directly, and what the encode +/// submission needs. +pub(crate) struct HeldFrame { + pub slot: usize, + pub source: SourceBinding, + /// What the encode submission waits for. + pub waits: Vec>, + /// What the encode submission signals besides its timeline. + pub signals: Vec>, + /// When the frame was submitted, for the packet's latency. + pub submitted: std::time::Instant, +} + +/// A frame the planner holds, or hands back to be encoded. +pub(crate) struct QueuedFrame { + pub frame: HeldFrame, + /// Display order (PTS) since the encoder was made. + pub display_order: u64, +} + +/// A frame that was encoded before frames shown earlier, in a codec that +/// hides such a frame until its turn (see [`VideoCodec::HIDDEN_FRAMES`]). +pub(crate) struct HiddenFrame { + pub display_order: u64, + /// The type of the frame, which is the type of the packet that shows it. + pub frame_type: FrameType, + /// The packet that shows the frame. + pub show_packet: Vec, + /// When the frame was submitted, for the packet's latency. + pub submitted: std::time::Instant, +} + +/// Everything a single frame needs to know about its schedule, so the codec +/// hooks all see a consistent view. pub(crate) struct FramePlan { pub gop: GopPosition, /// Presentation order (PTS). pub display_order: u64, - /// Encode order (DTS); `0` marks the very first encoded frame. + /// Encode order; `0` marks the very first encoded frame. pub encode_index: u64, + /// How far decode order runs ahead of display order, which the DTS is + /// shifted back by so it is never after the PTS. + pub reorder_delay: u32, + /// Whether the frame is encoded now and shown later, by a packet of its + /// own. Only with [`VideoCodec::HIDDEN_FRAMES`]. + pub hidden: bool, } impl FramePlan { @@ -466,6 +515,13 @@ impl FramePlan { pub fn is_reference(&self) -> bool { self.gop.is_reference } + pub fn is_b(&self) -> bool { + self.gop.frame_type == GopFrameType::B + } + /// 0 for IDR and P-frames, and the depth in the pyramid for a B-frame. + pub fn pyramid_level(&self) -> u8 { + self.gop.pyramid_level + } pub fn is_first_frame(&self) -> bool { self.encode_index == 0 } @@ -477,8 +533,15 @@ impl FramePlan { match self.gop.frame_type { GopFrameType::Idr => FrameType::Idr, GopFrameType::P => FrameType::P, + GopFrameType::B => FrameType::B, } } + /// The packet's decode timestamp: the encode order, shifted back by the + /// reorder delay. Below zero for the first packets of a stream with + /// B-frames. + pub fn dts(&self) -> i64 { + self.encode_index as i64 - i64::from(self.reorder_delay) + } } /// The codec header (if any) produced by [`VideoCodec::begin_picture`]. @@ -551,17 +614,65 @@ impl RateControlPlan { } } + /// The decision for the frame `plan`. With a constant quantizer, a + /// B-frame gets a coarser one: no frame, or fewer frames, predict from + /// it, so its bits buy less. `b_offset` is what is added for a B-frame + /// that is not a reference; `max` is the largest quantizer the codec has. + pub fn for_frame( + config: &EncodeConfig, + controller_default_qp: u32, + plan: &FramePlan, + b_offset: u32, + max: u32, + ) -> Self { + let mut rc = Self::new(config, controller_default_qp); + if rc.is_disabled() && plan.is_b() { + rc.qp = (rc.qp + b_frame_qp_offset(plan, b_offset)).min(max); + } + rc + } + pub fn is_disabled(&self) -> bool { self.mode == vk::VideoEncodeRateControlModeFlagsKHR::DISABLED } } +/// What a constant quantizer is raised by for the B-frame `plan`. +/// +/// Each level of a pyramid is a step of half `b_offset` more than the level +/// above it: fewer frames predict from a deeper frame. A B-frame that is not +/// a reference gets `b_offset` at least, also in a flat group, where it is at +/// level 1. +fn b_frame_qp_offset(plan: &FramePlan, b_offset: u32) -> u32 { + let by_level = u32::from(plan.pyramid_level()) * (b_offset / 2); + if plan.is_reference() { + by_level + } else { + by_level.max(b_offset) + } +} + /// A video codec plugged into the generic [`CodecEncoder`]. /// /// Implementors are small state structs (reference lists, syntax counters, /// parameter caches). The generic driver calls these hooks in order around each /// frame; everything else (slot rotation, upload, readback, teardown) is shared. pub(crate) trait VideoCodec: Sized + Send { + /// Whether the codec has no reordering of its own, and hides a frame + /// that is encoded before frames shown earlier (AV1). + /// + /// Such a frame has no packet: its bitstream goes in front of the next + /// packet. When its turn to be shown comes, a packet from + /// [`Self::show_packet`] shows it. So the packets are in display order, + /// and their decode timestamps are their presentation timestamps. + const HIDDEN_FRAMES: bool = false; + + /// The packet that shows the hidden frame `plan`, which was just encoded + /// into `dpb_slot`. Only called with [`Self::HIDDEN_FRAMES`]. + fn show_packet(&self, _plan: &FramePlan, _dpb_slot: u8) -> Vec { + Vec::new() + } + /// Per-frame prologue: codec IDR/key-frame resets and header retrieval. /// Runs before recording; returns the packet's frame type and header. fn begin_picture( @@ -573,11 +684,8 @@ pub(crate) trait VideoCodec: Sized + Send { /// Record the codec-specific encode commands and submit the frame. Builds /// the StdVideo* graph on its own stack (so the FFI pointers stay valid until /// `cmd_encode_video`) and finishes via [`EncoderCommon::submit_frame`]. - fn record_picture( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result; + /// An error means that nothing was submitted. + fn record_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()>; /// Per-frame epilogue: advance reference lists, syntax counters and the next /// DPB slot. Runs after submission. @@ -588,11 +696,12 @@ pub(crate) trait VideoCodec: Sized + Send { /// one. /// /// `first_lost_display_order` is the display order (the `pts` reported on - /// encoded packets) of the earliest frame the client lost. In a linear P - /// chain every reference at or after that frame is transitively undecodable, - /// so all of them are dropped. Returns `true` if a usable reference survives - /// (the next P-frame can be predicted from it); `false` if none remain and - /// the caller must fall back to an IDR. + /// encoded packets) of the earliest frame the client lost. The client + /// lost every frame encoded after it too, so every reference from the + /// first one encoded that is shown at or after it is dropped (see + /// [`ReferenceSet::invalidate`](crate::encoder::references::ReferenceSet::invalidate)). + /// Returns `true` if a usable reference survives; `false` if none remain + /// and the caller must fall back to an IDR. fn invalidate_references( &mut self, common: &mut EncoderCommon, @@ -627,62 +736,245 @@ pub struct CodecEncoder { unsafe impl Send for CodecEncoder {} impl CodecEncoder { - /// The internal input image for the current slot (a `ColorConverter::convert` - /// target that avoids an intermediate copy). + /// The input image of the slot the next frame takes (a + /// `ColorConverter::convert` target that avoids an intermediate copy). pub fn input_image(&self) -> vk::Image { self.common.pipeline.input_image() } - /// Encode one frame from `input`, returning a future for its packet. - /// See [`crate::Encoder::submit`]. + /// Take one frame from `input`, returning a future for the next packet + /// of the stream. See [`crate::Encoder::submit`]. + /// + /// The frame is encoded now, or, as a B-frame, once the reference frame + /// after it is submitted. Either way the frames the planner hands back + /// are encoded here, in encode order. pub fn encode_frame(&mut self, input: FrameInput, sync: FrameSync) -> Result { - let plan = self.common.begin_frame(); - self.common.prepare_source(input, sync)?; + self.take_frame(input, sync, false) + } + + /// Take one frame and encode it now, with the frames that wait before + /// it: a frame that must not wait, because its source is another slot's + /// image ([`crate::Encoder::repeat`]). + pub fn encode_frame_now(&mut self, input: FrameInput, sync: FrameSync) -> Result { + self.take_frame(input, sync, true) + } - let setup = self.codec.begin_picture(&mut self.common, &plan)?; + fn take_frame( + &mut self, + input: FrameInput, + sync: FrameSync, + now: bool, + ) -> Result { + // Waits until the slot we are about to record over was read back. + let slot = self.common.pipeline.next_index(); + let (frame, released) = self.common.prepare_source(slot, input, sync)?; + self.common.pipeline.take_slot(); + let mut future = self.common.pipeline.new_future(); + + let display_order = self.common.input_frame_num; + self.common.input_frame_num += 1; + let queued = QueuedFrame { + frame, + display_order, + }; + let mut ready = Vec::new(); + if now { + ready.extend(self.common.gop.close()); + } + ready.extend(self.common.gop.push(queued)); + if now { + ready.extend(self.common.gop.close()); + } + let encoded = self.encode_ready(ready)?; + // A frame written in place and still waiting has nothing to release: + // the future keeps the point it was made with, which is signalled. + let point = encoded + .into_iter() + .find_map(|(order, point)| (order == display_order).then_some(point)); + if let Some(point) = released.or(point) { + future.set_released(point); + } + Ok(future) + } + + /// Encode `ready` in its order, and return the display order of each + /// frame with the point its encode submission signals. + /// + /// If an encode fails, nothing more is encoded: the frames left, and the + /// frames that wait, are dropped, the futures not yet resolved resolve + /// with an error, and the next frame is an IDR, because the reference + /// state after a failed encode is not known. + fn encode_ready( + &mut self, + ready: Vec<(QueuedFrame, GopPosition)>, + ) -> Result> { + let reorder_delay = self.common.gop.reorder_delay(); + // A frame is hidden if a frame after it in this order is shown + // before it. A group is always handed over whole, so the frames + // after it are all here. + let display_orders: Vec = ready.iter().map(|(q, _)| q.display_order).collect(); + let mut encoded = Vec::with_capacity(ready.len()); + let mut ready = ready.into_iter().enumerate(); + while let Some((index, (queued, gop))) = ready.next() { + let plan = FramePlan { + gop, + display_order: queued.display_order, + encode_index: self.common.encode_frame_num, + reorder_delay, + hidden: C::HIDDEN_FRAMES + && display_orders[index + 1..] + .iter() + .any(|&later| later < queued.display_order), + }; + let slot = queued.frame.slot; + let result = self + .encode_planned(queued.frame, &plan) + .and_then(|point| self.show_hidden_frames(&plan).map(|()| point)); + match result { + Ok(point) => encoded.push((plan.display_order, point)), + Err(e) => { + self.common.hidden_frames.clear(); + let left = ready.map(|(_, (queued, _))| queued); + let waiting = self.common.gop.abandon(); + self.common.pipeline.release(slot); + for queued in left.chain(waiting) { + self.common.pipeline.release(queued.frame.slot); + } + self.common.pipeline.drop_waiting_futures(); + return Err(e); + } + } + } + Ok(encoded) + } + + /// Encode the frames that wait now, as a shorter group. + pub fn close_group(&mut self) -> Result<()> { + let ready = self.common.gop.close(); + self.encode_ready(ready).map(|_| ()) + } + + /// After the shown frame `plan`, show the hidden frames whose turn it + /// is: those that come next in display order. Each gets a packet made + /// here, with no encode. + fn show_hidden_frames(&mut self, plan: &FramePlan) -> Result<()> { + if plan.hidden { + return Ok(()); + } + let mut next = plan.display_order + 1; + while let Some(index) = self + .common + .hidden_frames + .iter() + .position(|hidden| hidden.display_order == next) + { + let hidden = self.common.hidden_frames.swap_remove(index); + let latency = hidden.submitted.elapsed().as_nanos() as u64; + self.common.pipeline.submit_synthetic(EncodedPacket { + data: hidden.show_packet, + frame_type: hidden.frame_type, + pts: hidden.display_order, + dts: hidden.display_order as i64, + stats: EncodedPacketStats { + gpu_time_ns: None, + frame_latency_ns: latency, + wall_latency_ns: 0, + }, + })?; + next += 1; + } + Ok(()) + } + + /// Encode `frame` now, as `plan` says. Returns the point its encode + /// submission signals. An error means that nothing was submitted. + fn encode_planned(&mut self, frame: HeldFrame, plan: &FramePlan) -> Result { + let submitted = frame.submitted; + let delivery = if plan.hidden { + Delivery::Carry + } else { + Delivery::Packet + }; + self.common.begin_encode(frame, delivery); + + let setup = self.codec.begin_picture(&mut self.common, plan)?; self.common.set_pending_metadata(SlotPacketMetadata { frame_type: plan.frame_type(), pts: plan.display_order, - dts: plan.encode_index, + // With hidden frames the packets are in display order. + dts: if C::HIDDEN_FRAMES { + plan.display_order as i64 + } else { + plan.dts() + }, header: setup.header, timestamps: [0; 2], - now: std::time::Instant::now(), + now: submitted, }); - let future = self.codec.record_picture(&mut self.common, &plan)?; + let dpb_slot = self.common.current_dpb_slot; + self.codec.record_picture(&mut self.common, plan)?; self.common.encode_frame_num += 1; - self.codec.end_picture(&mut self.common, &plan); - self.common.advance(); - Ok(future) + if plan.hidden { + self.common.hidden_frames.push(HiddenFrame { + display_order: plan.display_order, + frame_type: plan.frame_type(), + show_packet: self.codec.show_packet(plan, dpb_slot), + submitted, + }); + } + self.codec.end_picture(&mut self.common, plan); + Ok(self + .common + .encode_point + .take() + .expect("record_picture submits the frame")) } - /// End-of-stream barrier: wait for all in-flight frames to be read back. + /// End-of-stream barrier: encode the frames that wait, then wait for all + /// in-flight frames to be read back. pub fn flush(&mut self) -> Result<()> { + self.close_group()?; self.common.pipeline.flush(); Ok(()) } - /// Force the next frame to be an IDR/key frame. + /// Force the next frame to be an IDR/key frame. The frames that wait + /// are encoded now, before it. pub fn request_idr(&mut self) { self.common.gop.request_idr(); + if let Err(e) = self.close_group() { + // Their futures resolve with the error. + tracing::warn!("Encoding the frames before a requested IDR failed: {e}"); + } } /// Reference frame invalidation: drop references the client lost and predict /// the next frame from a surviving reference, falling back to an IDR when no /// reference survives. See [`crate::Encoder::invalidate_reference_frames`]. + /// + /// For the IDR, the oldest frame that waits for its group is encoded as + /// one at once, so no frame is encoded from the lost references. pub fn invalidate_reference_frames(&mut self, first_lost_display_order: u64) { - if !self + if self .codec .invalidate_references(&mut self.common, first_lost_display_order) { - self.common.gop.request_idr(); + return; + } + let ready = self.common.gop.restart(); + if let Err(e) = self.encode_ready(ready) { + // Its future resolves with the error, and the next frame is an IDR. + tracing::warn!("Encoding the IDR after lost references failed: {e}"); } } /// Rebuild session parameters with a new stream colour; the next frame is /// an IDR/key frame carrying the updated header. pub fn set_color(&mut self, color: StreamColor) -> Result<()> { - // Drain in-flight encodes before mutating shared session parameters. + // Encode the frames that wait with the old parameters, and drain + // in-flight encodes before mutating shared session parameters. + self.close_group()?; self.common.pipeline.wait_all_free(); let desc = color.description(); @@ -768,6 +1060,9 @@ pub(crate) struct CommonInitRequest<'a> { /// What the hardware RGB conversion supports for this profile, queried /// alongside the codec capabilities when `rgb_input` is set. pub rgb_caps: Option, + /// What the encoder can do with B-frames for this profile, from the + /// codec's capabilities. + pub b_frame_support: BFrameSupport, } /// Result of [`build_encoder_common`]: the assembled common state plus the @@ -778,6 +1073,10 @@ pub(crate) struct CommonInit { pub reference_window: u32, /// How many of the newest of those one frame predicts from. pub active_reference_count: u32, + /// How many references one frame may use at most, which is what the + /// session was made with. More than `active_reference_count` only with + /// B-frames, which use a reference on each side. + pub session_active_references: u32, } /// The device-capability fields the generic init needs, resolved once by the @@ -792,6 +1091,23 @@ pub(crate) struct DeviceVideoCaps { pub std_header_version: vk::ExtensionProperties, } +/// Check `config.quality_level` against the device's +/// `VkVideoEncodeCapabilitiesKHR::maxQualityLevels`. +/// +/// Call it before anything else uses the level: a quality level query for a +/// level the device does not have is invalid usage. +pub(crate) fn check_quality_level(config: &EncodeConfig, max_quality_levels: u32) -> Result<()> { + if config.quality_level >= max_quality_levels { + return Err(PixelForgeError::NoSuitableDevice(format!( + "{:?} encode: quality level {} is not available, the device has levels 0 to {}", + config.codec, + config.quality_level, + max_quality_levels.saturating_sub(1) + ))); + } + Ok(()) +} + /// Call `vkGetPhysicalDeviceVideoCapabilitiesKHR` and return its raw result. pub(crate) fn query_video_capabilities( context: &VideoContext, @@ -857,6 +1173,7 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result // Capabilities were queried by the codec (which chains its codec-specific // capability struct, required by the driver) via [`query_video_caps`]. let capabilities = req.caps; + check_b_frame_support(config, req.b_frame_support)?; // Align the coded extent to lcm(codec block size, device granularity), then // clamp up to the device minimum and re-align. @@ -971,11 +1288,28 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result } // At least 1: `max_dpb_slots_supported` is at least 2, and every codec's // window cap and `MAX_DPB_SLOTS` are larger. + let gop = GopPlanner::new(config.gop_size, config.b_frames, config.b_frame_references); + // The B-frames of a pyramid that are references are held on top of the + // window, which is the P-frames. + let b_reference_count = gop.b_reference_count(); let window_limit = req .max_reference_window .min(max_dpb_slots_supported - 1) - .min(MAX_DPB_SLOTS - 1); - let reference_window = (config.max_reference_frames as usize).clamp(1, window_limit); + .min(MAX_DPB_SLOTS - 1) + .saturating_sub(b_reference_count) + .max(1); + let mut reference_window = (config.max_reference_frames as usize).clamp(1, window_limit); + // A B-frame predicts from a reference on each side, so both have to stay. + if config.b_frames > 0 && reference_window < 2 { + if window_limit < 2 { + return Err(PixelForgeError::NoSuitableDevice(format!( + "B-frames need a reference window of 2, and the device allows {window_limit} \ + with {b_reference_count} B-frame references" + ))); + } + tracing::info!("B-frames: reference window raised from {reference_window} to 2"); + reference_window = 2; + } let mut max_active_reference_pictures = reference_window .min(max_active_supported) .min(req.max_frame_references); @@ -990,9 +1324,17 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result if max_active_reference_pictures < 1 && max_active_supported >= 1 { max_active_reference_pictures = 1; } - // B-frames are not yet supported, so a reconstructed-frame slot plus the - // window is enough. - let dpb_slot_count = reference_window + 1; + // A B-frame uses a reference on each side, whatever a P-frame uses. The + // B-frame support check has made sure that the device allows two. + let session_active_references = if config.b_frames > 0 { + max_active_reference_pictures.max(2) + } else { + max_active_reference_pictures + }; + // The window, the B-frame references, and a slot for the reconstructed + // frame. A B-frame that is not a reference takes that slot too and + // leaves it free again. + let dpb_slot_count = reference_window + b_reference_count + 1; tracing::debug!( "Reference window {reference_window}, {max_active_reference_pictures} used per frame \ (device: {max_dpb_slots_supported} DPB slots, {max_active_supported} active, \ @@ -1018,7 +1360,7 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result }) .reference_picture_format(reference_picture_format) .max_dpb_slots(dpb_slot_count as u32) - .max_active_reference_pictures(max_active_reference_pictures as u32) + .max_active_reference_pictures(session_active_references as u32) .std_header_version(&std_header_version); if rgb_session.is_some() { session_create_info = session_create_info.push(&mut rgb_session_info); @@ -1072,14 +1414,16 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result bitstream_buffer_size: req.bitstream_buffer_size, profile_info, command_pool: cmd.command_pool, + held_frames: config.b_frames as usize, upload_command_buffer: cmd.upload_command_buffer, upload_fence: cmd.upload_fence, })?; - // B-frames are not yet supported, so an I-P GOP. Match the SPS's POC - // wrap (no-op for AV1, which keys off order hints). - let mut gop = GopStructure::new(config.gop_size); - gop.set_max_poc_lsb(4); + let upload_command_buffers = allocate_command_buffers( + context, + cmd.upload_command_pool, + pipeline.slot_count() as u32, + )?; let common = EncoderCommon { context: context.clone(), @@ -1105,16 +1449,14 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result command_pool: cmd.command_pool, upload_command_pool: cmd.upload_command_pool, upload_fence: cmd.upload_fence, - upload_command_buffers: allocate_command_buffers( - context, - cmd.upload_command_pool, - ENCODE_PIPELINE_DEPTH as u32, - )?, + upload_command_buffers, upload_timeline: TimelineChain::new(context)?, pending_waits: Vec::new(), pending_signals: Vec::new(), source: SourceBinding::default(), - source_released: None, + delivery: Delivery::Packet, + encode_point: None, + hidden_frames: Vec::new(), profile: req.profile, rgb_input: req.rgb_input, }; @@ -1123,5 +1465,39 @@ pub(crate) fn build_encoder_common(req: CommonInitRequest) -> Result common, reference_window: reference_window as u32, active_reference_count: max_active_reference_pictures as u32, + session_active_references: session_active_references as u32, }) } + +#[cfg(test)] +mod tests { + use super::*; + + fn b_frame(is_reference: bool, pyramid_level: u8) -> FramePlan { + FramePlan { + gop: GopPosition { + frame_type: GopFrameType::B, + pic_order_cnt: 2, + is_reference, + pyramid_level, + }, + display_order: 1, + encode_index: 2, + reorder_delay: 1, + hidden: false, + } + } + + #[test] + fn b_frame_qp_offset_grows_with_the_level() { + // A flat group, and a pyramid of 3, as before levels counted. + assert_eq!(b_frame_qp_offset(&b_frame(false, 1), 2), 2); + assert_eq!(b_frame_qp_offset(&b_frame(true, 1), 2), 1); + assert_eq!(b_frame_qp_offset(&b_frame(false, 2), 2), 2); + // A pyramid of 7: one more for each level. + assert_eq!(b_frame_qp_offset(&b_frame(true, 2), 2), 2); + assert_eq!(b_frame_qp_offset(&b_frame(false, 3), 2), 3); + // AV1 steps by 4. + assert_eq!(b_frame_qp_offset(&b_frame(false, 3), 8), 12); + } +} diff --git a/src/encoder/external.rs b/src/encoder/external.rs index bc41e982..d24f4fa5 100644 --- a/src/encoder/external.rs +++ b/src/encoder/external.rs @@ -362,7 +362,7 @@ impl Encoder { Source::Rgb(rgb) => Some(rgb.format.vk_format()), Source::Yuv => None, }; - Ok(match self.conversion() { + let plan = match self.conversion() { Conversion::None => { if config.chroma_format == ChromaFormat::Yuv422 { return Err(PixelForgeError::InvalidInput( @@ -385,6 +385,21 @@ impl Encoder { zero_copy: Access::Sample, fallback: None, }, + }; + // A B-frame is encoded only once the frame after its group arrives. + // Read directly, it would keep the producer's buffer that long, and a + // compositor has few of them. Copied, the buffer is free at once. + Ok(match plan { + InputPlan { + zero_copy: Access::Encode, + vk_format, + .. + } if config.b_frames > 0 => InputPlan { + vk_format, + zero_copy: Access::Copy, + fallback: None, + }, + plan => plan, }) } @@ -946,14 +961,13 @@ pub(crate) struct SyncFiles { } impl SyncFiles { - pub(crate) fn new(context: &VideoContext) -> Result { + pub(crate) fn new(context: &VideoContext, slots: usize) -> Result { if !context.external().sync_file { return Err(PixelForgeError::NoSuitableDevice( "sync_file fences need VK_KHR_external_semaphore_fd".to_string(), )); } let device = context.device(); - let slots = crate::encoder::pipeline::ENCODE_PIPELINE_DEPTH; let mut files = Self { context: context.clone(), fd_fn: ash::khr::external_semaphore_fd::Device::load(context.instance(), device), diff --git a/src/encoder/frame.rs b/src/encoder/frame.rs index 76812bf2..3d74ba8c 100644 --- a/src/encoder/frame.rs +++ b/src/encoder/frame.rs @@ -129,6 +129,10 @@ impl<'a> Frame<'a> { /// /// For a producer that waits on sync_files, or to attach to the DMA-BUF /// with `DMA_BUF_IOCTL_IMPORT_SYNC_FILE` before handing it back. + /// + /// With B-frames, [`Encoder::submit`](crate::Encoder::submit) returns + /// an error for a frame from [`Frame::input`] that asks for this: its + /// release is its encode, which may come in a later call. #[cfg(feature = "dmabuf")] pub fn signal_sync_file(mut self) -> Self { self.signal_sync_file = true; diff --git a/src/encoder/gop.rs b/src/encoder/gop.rs index c00b1ed8..1758a74d 100644 --- a/src/encoder/gop.rs +++ b/src/encoder/gop.rs @@ -1,7 +1,16 @@ -//! GOP (Group of Pictures) structure for video encoding. +//! GOP (Group of Pictures) planning: frame types, and the order frames are +//! encoded in. //! -//! This module provides GOP structure management for H.264 and H.265 encoders. -//! The GOP structure determines frame types (IDR or P). +//! Frames come in in display order. Without B-frames each one is encoded at +//! once, as an IDR or a P-frame. With B-frames, a group of `b_frames` frames +//! waits for the reference frame shown after it: that one is encoded first, +//! as a P-frame, and then the group, as B-frames that predict from both. +//! +//! The planner is codec-agnostic and has no Vulkan in it. It holds the frames +//! that wait, as whatever the caller gives it, and hands them back in encode +//! order with their [`GopPosition`]. + +use super::BFrameReferences; /// Frame type in GOP. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -10,6 +19,8 @@ pub enum GopFrameType { Idr, /// P frame (predicted). P, + /// B frame (predicted from both directions). + B, } impl GopFrameType { @@ -24,108 +35,591 @@ impl GopFrameType { pub struct GopPosition { /// Frame type. pub frame_type: GopFrameType, - /// POC (Picture Order Count). + /// POC (Picture Order Count): twice the frames shown since the last IDR. + /// + /// It does not wrap. Writing its least significant bits into the slice + /// header is the driver's job, and the list and RPS code sorts and + /// subtracts whole values. pub pic_order_cnt: i32, /// Whether this frame is a reference. pub is_reference: bool, + /// How deep in a B-frame pyramid the frame is: 0 for IDR and P-frames, 1 + /// for the B-frames of a flat group and for the middle B-frame of a + /// pyramid, and one more for each split below that. + pub pyramid_level: u8, } -/// GOP structure manager. +/// The held references that are of no more use once a frame of +/// `new_level` is encoded, as indices into `levels`, the pyramid levels of +/// the held references. /// -/// Manages the GOP structure for video encoding, determining frame types. -/// and their ordering. This is codec-agnostic and can be used for both -/// H.264 and H.265 encoding. -pub struct GopStructure { - /// Current frame index. - frame_index: u64, - /// Current POC. - poc: i32, - /// IDR period. +/// A B-frame that is a reference is used only by the frames between the two +/// references it was predicted from. In encode order those all come before +/// the next frame of its own level or of a level above it. So a reference +/// frame replaces every B-frame reference of its level and of the levels +/// below it, and the references the frame itself needs all stay. +pub fn obsolete_b_references(levels: &[u8], new_level: u8) -> Vec { + (0..levels.len()) + .filter(|&i| levels[i] > 0 && levels[i] >= new_level) + .collect() +} + +/// A B-frame of a group, in encode order: its index in the group's display +/// order, whether it is a reference, and its pyramid level. +type GroupEntry = (usize, bool, u8); + +/// The encode order of a group of `count` B-frames. +/// +/// Flat: display order, none a reference. Pyramid: the middle frame first, as +/// a reference, then each half the same way. A range of fewer than three +/// frames has no middle frame that frames on both sides use, so it is flat. +fn b_frame_order(count: usize, references: BFrameReferences) -> Vec { + fn split(lo: usize, len: usize, level: u8, out: &mut Vec) { + if len < 3 { + out.extend((lo..lo + len).map(|i| (i, false, level))); + return; + } + let mid = (len - 1) / 2; + out.push((lo + mid, true, level)); + split(lo, mid, level + 1, out); + split(lo + mid + 1, len - mid - 1, level + 1, out); + } + + let mut out = Vec::with_capacity(count); + match references { + BFrameReferences::Flat => out.extend((0..count).map(|i| (i, false, 1))), + BFrameReferences::Pyramid => split(0, count, 1, &mut out), + } + out +} + +/// GOP planner: takes frames in display order and gives them back in encode +/// order. +pub struct GopPlanner { + /// IDR period in frames, or 0 for no periodic IDR. idr_period: u32, - /// Max POC LSB value (log2_max_pic_order_cnt_lsb_minus4 + 4). - max_poc_lsb: u32, + b_frames: u32, + references: BFrameReferences, + /// Display position of the next frame since the last IDR, or `None` + /// before the first frame. + next_position: Option, /// Flag to force next frame to be an IDR. force_idr: bool, + /// The frames that wait for the reference frame after them, in display + /// order, with their display position since the last IDR. + waiting: Vec<(T, u32)>, + /// See [`Self::b_reference_count`]. + b_reference_count: usize, + /// See [`Self::reorder_delay`]. + reorder_delay: u32, } -impl GopStructure { - /// An I-P GOP structure, with an IDR every `gop_size` frames. - pub fn new(gop_size: u32) -> Self { +/// The reference pattern of the B-frames, as the rate control is told it. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ReferencePattern { + /// No B-frame is a reference. + Flat, + /// Each B-frame reference is in the middle of the two references it + /// predicts from, which only `2^n - 1` B-frames give. + Dyadic, +} + +impl GopPlanner { + /// A planner with an IDR every `idr_period` frames (0 for none) and + /// `b_frames` B-frames between two reference frames. + pub fn new(idr_period: u32, b_frames: u32, references: BFrameReferences) -> Self { + let order = b_frame_order(b_frames as usize, references); Self { - idr_period: gop_size, - frame_index: 0, - poc: 0, - max_poc_lsb: 16, // log2_max_pic_order_cnt_lsb_minus4=0 -> 2^4 = 16 + idr_period, + b_frames, + references, + next_position: None, force_idr: false, + waiting: Vec::new(), + b_reference_count: b_reference_count(&order), + reorder_delay: reorder_delay(&order), } } - /// Set max POC LSB (2^(log2_max_pic_order_cnt_lsb_minus4+4)). - pub fn set_max_poc_lsb(&mut self, log2_max_pic_order_cnt_lsb_minus4: u8) { - self.max_poc_lsb = 1 << (log2_max_pic_order_cnt_lsb_minus4 + 4); + /// Take the next frame in display order, and give back the frames that + /// can be encoded now, in encode order. Empty while `frame` waits. + pub fn push(&mut self, frame: T) -> Vec<(T, GopPosition)> { + let position = match self.next_position { + Some(position) + if !self.force_idr && (self.idr_period == 0 || position < self.idr_period) => + { + position + } + _ => { + // A group never crosses an IDR: what waits is closed first. + let mut out = self.close(); + self.force_idr = false; + self.next_position = Some(1); + out.push(( + frame, + GopPosition { + frame_type: GopFrameType::Idr, + pic_order_cnt: 0, + is_reference: true, + pyramid_level: 0, + }, + )); + return out; + } + }; + self.next_position = Some(position + 1); + self.waiting.push((frame, position)); + // The frame before a periodic IDR is a reference frame, so the GOP + // is closed. + let before_idr = self.idr_period > 0 && position + 1 == self.idr_period; + if self.waiting.len() > self.b_frames as usize || before_idr { + self.close() + } else { + Vec::new() + } } - /// Get the next frame position in the GOP. - pub fn get_next_frame(&mut self) -> GopPosition { - let is_idr = self.frame_index == 0 - || (self.idr_period > 0 && self.frame_index.is_multiple_of(self.idr_period as u64)) - || self.force_idr; + /// Encode the frames that wait now, as a shorter group: the last one + /// becomes the P-frame, and the others its B-frames. Empty if no frame + /// waits. + pub fn close(&mut self) -> Vec<(T, GopPosition)> { + let Some((anchor, anchor_position)) = self.waiting.pop() else { + return Vec::new(); + }; + let mut out = Vec::with_capacity(self.waiting.len() + 1); + out.push(( + anchor, + GopPosition { + frame_type: GopFrameType::P, + pic_order_cnt: poc(anchor_position), + is_reference: true, + pyramid_level: 0, + }, + )); + let mut group: Vec> = self.waiting.drain(..).map(Some).collect(); + for (index, is_reference, level) in b_frame_order(group.len(), self.references) { + let (frame, position) = group[index].take().expect("each frame once"); + out.push(( + frame, + GopPosition { + frame_type: GopFrameType::B, + pic_order_cnt: poc(position), + is_reference, + pyramid_level: level, + }, + )); + } + out + } - // Clear the force IDR flag after checking it. - self.force_idr = false; + /// Request that the next frame be an IDR frame. The frames that wait are + /// encoded before it, by the next [`Self::push`] or by [`Self::close`]. + pub fn request_idr(&mut self) { + self.force_idr = true; + } - // If IDR, reset counters. - if is_idr { - self.poc = 0; - } + /// Drop the frames that wait, and make the next frame an IDR. For when + /// the reference state is lost. + pub fn abandon(&mut self) -> Vec { + self.force_idr = true; + self.waiting.drain(..).map(|(frame, _)| frame).collect() + } - let frame_type = if is_idr { - GopFrameType::Idr - } else { - GopFrameType::P - }; + /// Make the oldest frame that waits an IDR now, and keep the others + /// waiting after it. Returns the IDR, or nothing if no frame waits, and + /// then the next frame is an IDR. For when the reference state is lost: + /// the frames that wait are then never encoded from it. + pub fn restart(&mut self) -> Vec<(T, GopPosition)> { + self.force_idr = true; + let waiting: Vec = self.waiting.drain(..).map(|(frame, _)| frame).collect(); + let mut out = Vec::new(); + for frame in waiting { + out.extend(self.push(frame)); + } + out + } - let position = GopPosition { - frame_type, - pic_order_cnt: self.poc, - is_reference: true, - }; + /// How many B-frames are held as references at once at most: one for + /// each pyramid level that has references. 0 for flat groups. + pub fn b_reference_count(&self) -> usize { + self.b_reference_count + } - // Update counters for next frame. - self.poc = (self.poc + 2) % (self.max_poc_lsb as i32 * 2); - self.frame_index += 1; + /// The largest number of frames that are encoded before a frame and + /// shown after it: how far decode order runs ahead of display order. 0 + /// without B-frames, 1 for a flat group, more for a pyramid. + pub fn reorder_delay(&self) -> u32 { + self.reorder_delay + } - position + /// The reference pattern to tell the rate control, or `None` for a + /// pyramid that is not dyadic. + pub fn reference_pattern(&self) -> Option { + if self.b_reference_count == 0 { + Some(ReferencePattern::Flat) + } else if (self.b_frames + 1).is_power_of_two() { + Some(ReferencePattern::Dyadic) + } else { + None + } } +} - /// Request that the next frame be an IDR frame. - pub fn request_idr(&mut self) { - self.force_idr = true; +/// See [`GopPlanner::b_reference_count`]; `order` is the encode order of a +/// whole group. +fn b_reference_count(order: &[GroupEntry]) -> usize { + order + .iter() + .filter(|&&(_, is_reference, _)| is_reference) + .map(|&(_, _, level)| level as usize) + .max() + .unwrap_or(0) +} + +/// See [`GopPlanner::reorder_delay`]; `order` is the encode order of a +/// whole group. +fn reorder_delay(order: &[GroupEntry]) -> u32 { + let count = order.len(); + if count == 0 { + return 0; } + // Display positions in encode order: the reference frame after the + // group, then the group. A shorter group never delays more. + let mut encode_order = vec![count]; + encode_order.extend(order.iter().map(|&(index, _, _)| index)); + encode_order + .iter() + .enumerate() + .map(|(e, &shown)| encode_order[..e].iter().filter(|&&d| d > shown).count()) + .max() + .unwrap_or(0) as u32 +} + +/// The POC of the frame at `position` since the last IDR. +fn poc(position: u32) -> i32 { + 2 * position as i32 } #[cfg(test)] mod tests { use super::*; + /// Push `frames`, numbered in display order, and return what came out, + /// as (display number, type, POC, reference). + fn run( + planner: &mut GopPlanner, + frames: std::ops::Range, + ) -> Vec<(u32, GopFrameType, i32, bool)> { + let mut out = Vec::new(); + for frame in frames { + out.extend(planner.push(frame)); + } + out.into_iter() + .map(|(f, p)| (f, p.frame_type, p.pic_order_cnt, p.is_reference)) + .collect() + } + + fn types(out: &[(u32, GopFrameType, i32, bool)]) -> String { + out.iter() + .map(|(f, t, _, r)| { + let c = match t { + GopFrameType::Idr => "I", + GopFrameType::P => "P", + GopFrameType::B if *r => "B", + GopFrameType::B => "b", + }; + format!("{c}{f}") + }) + .collect::>() + .join(" ") + } + #[test] fn test_ip_only_gop() { - let mut gop = GopStructure::new(30); - - // First frame should be IDR. - let pos = gop.get_next_frame(); - assert_eq!(pos.frame_type, GopFrameType::Idr); - assert!(pos.is_reference); - - // Next frames should be P. - for i in 1..30 { - let pos = gop.get_next_frame(); - assert_eq!(pos.frame_type, GopFrameType::P, "Frame {i} should be P"); - assert!(pos.is_reference); + let mut gop = GopPlanner::new(30, 0, BFrameReferences::Flat); + + // Each frame comes out at once. + let pos = gop.push(0); + assert_eq!(pos.len(), 1); + assert_eq!(pos[0].1.frame_type, GopFrameType::Idr); + assert!(pos[0].1.is_reference); + + for i in 1..30u32 { + let pos = gop.push(i); + assert_eq!(pos.len(), 1); + assert_eq!( + pos[0].1.frame_type, + GopFrameType::P, + "Frame {i} should be P" + ); + assert_eq!(pos[0].1.pic_order_cnt, 2 * i as i32); + assert!(pos[0].1.is_reference); } // Frame 30 should be IDR (new GOP) - let pos = gop.get_next_frame(); - assert_eq!(pos.frame_type, GopFrameType::Idr); + let pos = gop.push(30); + assert_eq!(pos[0].1.frame_type, GopFrameType::Idr); + assert_eq!(pos[0].1.pic_order_cnt, 0); + assert_eq!(gop.reorder_delay(), 0); + } + + #[test] + fn poc_does_not_wrap_without_idr() { + let mut gop = GopPlanner::new(0, 0, BFrameReferences::Flat); + let out = run(&mut gop, 0..600); + assert_eq!(out[599].2, 1198); + } + + #[test] + fn flat_groups() { + let mut gop = GopPlanner::new(0, 2, BFrameReferences::Flat); + let out = run(&mut gop, 0..7); + assert_eq!(types(&out), "I0 P3 b1 b2 P6 b4 b5"); + assert_eq!(out[2].2, 2); + assert_eq!(gop.reorder_delay(), 1); + } + + #[test] + fn pyramid_groups() { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Pyramid); + let out = run(&mut gop, 0..9); + assert_eq!(types(&out), "I0 P4 B2 b1 b3 P8 B6 b5 b7"); + assert_eq!(gop.reorder_delay(), 2); + + let mut gop = GopPlanner::new(0, 7, BFrameReferences::Pyramid); + let out = run(&mut gop, 0..9); + assert_eq!(types(&out), "I0 P8 B4 B2 b1 b3 B6 b5 b7"); + assert_eq!(gop.reorder_delay(), 3); + } + + #[test] + fn short_pyramid_is_flat() { + let mut gop = GopPlanner::new(0, 2, BFrameReferences::Pyramid); + assert_eq!(types(&run(&mut gop, 0..4)), "I0 P3 b1 b2"); + assert_eq!(gop.reorder_delay(), 1); + } + + #[test] + fn pyramid_levels() { + let mut gop = GopPlanner::new(0, 7, BFrameReferences::Pyramid); + let mut out = Vec::new(); + for frame in 0..9 { + out.extend(gop.push(frame)); + } + let levels: Vec<(u32, u8)> = out.iter().map(|(f, p)| (*f, p.pyramid_level)).collect(); + assert_eq!( + levels, + [ + (0, 0), + (8, 0), + (4, 1), + (2, 2), + (1, 3), + (3, 3), + (6, 2), + (5, 3), + (7, 3) + ] + ); + } + + #[test] + fn b_reference_count_is_the_depth_of_the_references() { + let count = |b_frames, references| { + GopPlanner::::new(0, b_frames, references).b_reference_count() + }; + for b_frames in 0..=7 { + assert_eq!(count(b_frames, BFrameReferences::Flat), 0); + } + let pyramid: Vec = (0..=7) + .map(|b_frames| count(b_frames, BFrameReferences::Pyramid)) + .collect(); + assert_eq!(pyramid, [0, 0, 0, 1, 1, 1, 2, 2]); + } + + /// Encode a long stream, dropping references by `obsolete_b_references` + /// and by a sliding window of two P-frames: every B-frame must still + /// find the nearest reference frame on each side, and the B-frame + /// references held never exceed `b_reference_count`. + #[test] + fn obsolete_references_are_never_needed() { + for b_frames in 0..=7 { + let mut gop = GopPlanner::new(0, b_frames, BFrameReferences::Pyramid); + let limit = gop.b_reference_count(); + let mut out = Vec::new(); + for frame in 0..60u32 { + out.extend(gop.push(frame)); + } + // (display number, level) of the held references. + let mut held: Vec<(u32, u8)> = Vec::new(); + // Every frame encoded so far that is a reference. + let mut all: Vec = Vec::new(); + for (frame, position) in out { + if position.is_reference { + let levels: Vec = held.iter().map(|h| h.1).collect(); + for index in obsolete_b_references(&levels, position.pyramid_level) + .into_iter() + .rev() + { + held.remove(index); + } + } + if position.frame_type == GopFrameType::B { + let before = all.iter().filter(|&&r| r < frame).max().unwrap(); + let after = all.iter().filter(|&&r| r > frame).min().unwrap(); + assert!(held.iter().any(|h| h.0 == *before), "b={b_frames} {frame}"); + assert!(held.iter().any(|h| h.0 == *after), "b={b_frames} {frame}"); + } + if position.is_reference { + held.push((frame, position.pyramid_level)); + all.push(frame); + while held.iter().filter(|h| h.1 == 0).count() > 2 { + let oldest = held.iter().position(|h| h.1 == 0).unwrap(); + held.remove(oldest); + } + assert!(held.iter().filter(|h| h.1 > 0).count() <= limit); + } + } + } + } + + #[test] + fn group_never_crosses_a_periodic_idr() { + // 8 frames per GOP, groups of 3: the last group is short. + let mut gop = GopPlanner::new(8, 2, BFrameReferences::Flat); + let out = run(&mut gop, 0..9); + assert_eq!(types(&out), "I0 P3 b1 b2 P6 b4 b5 P7 I8"); + // POC restarts at the IDR. + assert_eq!(out[8].2, 0); + } + + #[test] + fn request_idr_closes_the_group() { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Flat); + let mut out = run(&mut gop, 0..3); + assert_eq!(types(&out), "I0"); + gop.request_idr(); + // The waiting frames are closed when asked, before the IDR. + out = gop + .close() + .into_iter() + .map(|(f, p)| (f, p.frame_type, p.pic_order_cnt, p.is_reference)) + .collect(); + assert_eq!(types(&out), "P2 b1"); + let out = run(&mut gop, 3..5); + assert_eq!(types(&out), "I3"); + assert_eq!(out[0].2, 0); + } + + #[test] + fn request_idr_on_push_closes_first() { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Flat); + run(&mut gop, 0..3); + gop.request_idr(); + assert_eq!(types(&run(&mut gop, 3..4)), "P2 b1 I3"); + } + + #[test] + fn close_with_each_count_waiting() { + for waiting in 0..=3u32 { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Flat); + run(&mut gop, 0..5); + run(&mut gop, 5..5 + waiting); + let out = gop.close(); + assert_eq!(out.len(), waiting as usize); + if waiting > 0 { + assert_eq!(out[0].0, 4 + waiting); + assert_eq!(out[0].1.frame_type, GopFrameType::P); + } + assert!(gop.close().is_empty()); + } + } + + #[test] + fn restart_makes_the_oldest_waiting_frame_the_idr() { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Flat); + run(&mut gop, 0..4); + let out: Vec<_> = gop + .restart() + .into_iter() + .map(|(f, p)| (f, p.frame_type, p.pic_order_cnt, p.is_reference)) + .collect(); + assert_eq!(types(&out), "I1"); + assert_eq!(out[0].2, 0); + // The others wait after it, with POCs from it. + let out = run(&mut gop, 4..6); + assert_eq!(types(&out), "P5 b2 b3 b4"); + assert_eq!(out[0].2, 8); + + // Nothing waits: the next frame is the IDR. + assert!(gop.restart().is_empty()); + assert_eq!(types(&run(&mut gop, 6..7)), "I6"); + } + + #[test] + fn only_full_pyramids_are_dyadic() { + let pattern = |b_frames, references| { + GopPlanner::::new(0, b_frames, references).reference_pattern() + }; + assert_eq!( + pattern(3, BFrameReferences::Flat), + Some(ReferencePattern::Flat) + ); + assert_eq!( + pattern(2, BFrameReferences::Pyramid), + Some(ReferencePattern::Flat) + ); + assert_eq!( + pattern(3, BFrameReferences::Pyramid), + Some(ReferencePattern::Dyadic) + ); + assert_eq!(pattern(5, BFrameReferences::Pyramid), None); + assert_eq!( + pattern(7, BFrameReferences::Pyramid), + Some(ReferencePattern::Dyadic) + ); + } + + #[test] + fn abandon_drops_the_group() { + let mut gop = GopPlanner::new(0, 3, BFrameReferences::Flat); + run(&mut gop, 0..3); + assert_eq!(gop.abandon(), vec![1, 2]); + assert_eq!(types(&run(&mut gop, 3..4)), "I3"); + } + + #[test] + fn every_frame_once_and_decode_order_valid() { + for b_frames in 0..=7 { + for references in [BFrameReferences::Flat, BFrameReferences::Pyramid] { + for idr_period in [0, 1, 5, 16] { + let mut gop = GopPlanner::new(idr_period, b_frames, references); + let mut out = run(&mut gop, 0..50); + out.extend( + gop.close() + .into_iter() + .map(|(f, p)| (f, p.frame_type, p.pic_order_cnt, p.is_reference)), + ); + let mut shown: Vec = out.iter().map(|o| o.0).collect(); + shown.sort_unstable(); + assert_eq!(shown, (0..50).collect::>()); + + // A B-frame comes after a reference frame shown after it, + // and the decode order runs ahead by no more than the delay. + let delay = gop.reorder_delay() as usize; + for (e, (f, t, _, _)) in out.iter().enumerate() { + let ahead = out[..e].iter().filter(|o| o.0 > *f).count(); + assert!(ahead <= delay, "b={b_frames} {references:?} frame {f}"); + if *t == GopFrameType::B { + assert!(out[..e].iter().any(|o| o.0 > *f && o.3)); + } + } + // dts = encode index - delay never passes pts. + for (e, (f, _, _, _)) in out.iter().enumerate() { + assert!(e as i64 - delay as i64 <= *f as i64); + } + } + } + } } } diff --git a/src/encoder/h264/init.rs b/src/encoder/h264/init.rs index c1ece12f..e27a4786 100644 --- a/src/encoder/h264/init.rs +++ b/src/encoder/h264/init.rs @@ -3,10 +3,11 @@ use super::{H264, MB_SIZE, ReferenceInfo}; use crate::converter::RgbFormat; use crate::encoder::EncodeConfig; use crate::encoder::codec::{ - CodecEncoder, CommonInitRequest, build_encoder_common, query_video_caps, + CodecEncoder, CommonInitRequest, build_encoder_common, check_quality_level, query_video_caps, }; use crate::encoder::profile::EncodeProfile; use crate::encoder::profile::h264_profile_idc; +use crate::encoder::references::ReferenceSet; use crate::encoder::resources::MIN_BITSTREAM_BUFFER_SIZE; use crate::encoder::rgb::RgbConversionCaps; use crate::error::Result; @@ -45,13 +46,15 @@ impl H264 { capabilities = capabilities.push(&mut rgb_caps); } let caps = query_video_caps(&context, profile_info, &mut capabilities)?; + check_quality_level(&config, encode_caps.max_quality_levels)?; let rgb_caps = rgb_input.map(|_| RgbConversionCaps::from(&rgb_caps)); // The driver's preferred entropy mode is H.264-specific (some require // CAVLC for High 4:4:4 Predictive). Only after the capability query, // which fails for an unsupported profile: a quality level query for one // is invalid (VUID-VkPhysicalDeviceVideoEncodeQualityLevelInfoKHR-pVideoProfile-08259). - let preferred_entropy_cabac = query_preferred_entropy_cabac(&context, profile_info); + let preferred_entropy_cabac = + query_preferred_entropy_cabac(&context, profile_info, config.quality_level); // Scale the bitstream buffer by resolution so a single frame can't overflow (matches AV1). let bitstream_buffer_size = MIN_BITSTREAM_BUFFER_SIZE @@ -69,20 +72,29 @@ impl H264 { bitstream_buffer_size, allow_layered_dpb: true, rgb_caps, + b_frame_support: crate::encoder::bframes::h264_support( + context.device_properties().vendor_id, + &h264_caps, + caps.max_active_reference_pictures, + ), })?; let reference_window = init.reference_window; let active_reference_count = init.active_reference_count; + let b_l0_reference_count = (init.session_active_references.saturating_sub(1)) + .min(h264_caps.max_b_picture_l0_reference_count); let common = init.common; let codec = H264 { frame_num_syntax: 0, idr_pic_id: 0, - l0_references: Vec::new(), + references: ReferenceSet::new(reference_window), reference_window, active_reference_count, + b_l0_reference_count, pending_unmark_frame_nums: Vec::new(), profile_idc, preferred_entropy_cabac, + max_level_idc: h264_caps.max_level_idc, }; let mut encoder = CodecEncoder { common, codec }; @@ -117,19 +129,16 @@ impl H264 { /// The references this frame predicts from: the newest held ones. pub(super) fn used_references(&self) -> &[ReferenceInfo] { - let n = self - .l0_references - .len() - .min(self.active_reference_count as usize); - &self.l0_references[..n] + self.references.newest(self.active_reference_count as usize) } } -/// Query the driver's preferred H.264 entropy coding mode for quality level 0. +/// Query the driver's preferred H.264 entropy coding mode for `quality_level`. /// Defaults to CABAC if the query is unsupported. fn query_preferred_entropy_cabac( context: &VideoContext, profile_info: &vk::VideoProfileInfoKHR, + quality_level: u32, ) -> bool { let video_encode_instance = ash::khr::video_encode_queue::Instance::load(context.entry(), context.instance()); @@ -138,7 +147,7 @@ fn query_preferred_entropy_cabac( .push(&mut h264_quality_level_properties); let quality_level_info = vk::PhysicalDeviceVideoEncodeQualityLevelInfoKHR::default() .video_profile(profile_info) - .quality_level(0); + .quality_level(quality_level); let result = unsafe { (video_encode_instance .fp() @@ -150,7 +159,7 @@ fn query_preferred_entropy_cabac( }; if result == vk::Result::SUCCESS { debug!( - "H.264 quality level 0: preferredStdEntropyCodingModeFlag={}", + "H.264 quality level {quality_level}: preferredStdEntropyCodingModeFlag={}", h264_quality_level_properties.preferred_std_entropy_coding_mode_flag ); h264_quality_level_properties.preferred_std_entropy_coding_mode_flag != 0 diff --git a/src/encoder/h264/level.rs b/src/encoder/h264/level.rs new file mode 100644 index 00000000..c2f6985e --- /dev/null +++ b/src/encoder/h264/level.rs @@ -0,0 +1,143 @@ +//! The H.264 level a stream needs (Annex A, Table A-1). + +use ash::vk::native::StdVideoH264LevelIdc; + +/// What a level allows, from Table A-1. +struct Limits { + /// `StdVideoH264LevelIdc`, which counts the levels from 1.0 up without + /// level 1b. + level: StdVideoH264LevelIdc, + /// `MaxMBPS`: macroblocks per second. + max_mbps: u64, + /// `MaxFS`: macroblocks per frame. + max_fs: u32, + /// `MaxDpbMbs`: macroblocks in the DPB. + max_dpb_mbs: u32, + /// `MaxBR` in units of 1000 bits per second, before the profile's factor. + max_br: u64, +} + +const fn limits( + level: StdVideoH264LevelIdc, + max_mbps: u64, + max_fs: u32, + max_dpb_mbs: u32, + max_br: u64, +) -> Limits { + Limits { + level, + max_mbps, + max_fs, + max_dpb_mbs, + max_br, + } +} + +const LEVELS: [Limits; 19] = [ + limits(0, 1_485, 99, 396, 64), + limits(1, 3_000, 396, 900, 192), + limits(2, 6_000, 396, 2_376, 384), + limits(3, 11_880, 396, 2_376, 768), + limits(4, 11_880, 396, 2_376, 2_000), + limits(5, 19_800, 792, 4_752, 4_000), + limits(6, 20_250, 1_620, 8_100, 4_000), + limits(7, 40_500, 1_620, 8_100, 10_000), + limits(8, 108_000, 3_600, 18_000, 14_000), + limits(9, 216_000, 5_120, 20_480, 20_000), + limits(10, 245_760, 8_192, 32_768, 20_000), + limits(11, 245_760, 8_192, 32_768, 50_000), + limits(12, 522_240, 8_704, 34_816, 50_000), + limits(13, 589_824, 22_080, 110_400, 135_000), + limits(14, 983_040, 36_864, 184_320, 240_000), + limits(15, 2_073_600, 36_864, 184_320, 240_000), + limits(16, 4_177_920, 139_264, 696_320, 240_000), + limits(17, 8_355_840, 139_264, 696_320, 480_000), + limits(18, 16_711_680, 139_264, 696_320, 800_000), +]; + +/// Level 4.1, the lowest level the encoder writes. It was the only level +/// the encoder wrote before, and every stream that fits it keeps it. +pub(super) const LOWEST: StdVideoH264LevelIdc = 11; + +/// What a stream needs from its level. +pub(super) struct Needs { + pub width_in_mbs: u32, + pub height_in_mbs: u32, + /// Frames per second, rounded up. + pub frame_rate: u32, + /// `max_dec_frame_buffering`: the frames the DPB holds. + pub dpb_frames: u32, + /// The peak bitrate in bits per second, or 0 if not known (constant QP). + pub max_bitrate: u64, + /// `cpbBrVclFactor` of the profile (Table A-2): 1250 for High, 4000 for + /// High 4:4:4 Predictive. + pub bitrate_factor: u64, +} + +/// The lowest level from [`LOWEST`] that allows `needs`, or the highest +/// level there is if none does. +pub(super) fn level_for(needs: &Needs) -> StdVideoH264LevelIdc { + let frame_mbs = needs.width_in_mbs * needs.height_in_mbs; + let fits = |limits: &Limits| { + // A frame's sides are each at most sqrt(8 * MaxFS) macroblocks. + let max_side = ((8 * limits.max_fs) as f64).sqrt() as u32; + frame_mbs <= limits.max_fs + && needs.width_in_mbs <= max_side + && needs.height_in_mbs <= max_side + && u64::from(frame_mbs) * u64::from(needs.frame_rate) <= limits.max_mbps + && needs.dpb_frames <= (limits.max_dpb_mbs / frame_mbs).min(16) + && needs.max_bitrate <= limits.max_br * needs.bitrate_factor + }; + LEVELS + .iter() + .filter(|limits| limits.level >= LOWEST) + .find(|limits| fits(limits)) + .map_or(LEVELS[LEVELS.len() - 1].level, |limits| limits.level) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn needs(width: u32, height: u32, frame_rate: u32, dpb_frames: u32) -> Needs { + Needs { + width_in_mbs: width.div_ceil(16), + height_in_mbs: height.div_ceil(16), + frame_rate, + dpb_frames, + max_bitrate: 0, + bitrate_factor: 1250, + } + } + + #[test] + fn small_streams_keep_level_4_1() { + assert_eq!(level_for(&needs(320, 240, 30, 4)), 11); + assert_eq!(level_for(&needs(1920, 1080, 30, 4)), 11); + } + + #[test] + fn a_larger_dpb_needs_a_higher_level() { + // 4 frames of 1080p fill level 4.1's DPB. + assert_eq!(level_for(&needs(1920, 1080, 30, 5)), 13); + } + + #[test] + fn size_and_rate_need_their_levels() { + assert_eq!(level_for(&needs(1920, 1080, 60, 4)), 12); + assert_eq!(level_for(&needs(3840, 2160, 30, 4)), 14); + assert_eq!(level_for(&needs(3840, 2160, 60, 4)), 15); + } + + #[test] + fn the_bitrate_needs_its_level() { + let mut stream = needs(1920, 1080, 30, 4); + stream.max_bitrate = 80_000_000; + assert_eq!(level_for(&stream), 13); + } + + #[test] + fn too_much_for_any_level_is_the_highest() { + assert_eq!(level_for(&needs(16384, 16384, 30, 4)), 18); + } +} diff --git a/src/encoder/h264/lists.rs b/src/encoder/h264/lists.rs new file mode 100644 index 00000000..2620eff2 --- /dev/null +++ b/src/encoder/h264/lists.rs @@ -0,0 +1,132 @@ +//! The initial reference picture lists of an H.264 B-frame. +//! +//! The hardware writes the slice header from the lists it is given, and it +//! does not reorder them. A decoder builds its own initial lists from the +//! pictures it holds (clause 8.2.4.2.3), so the lists the encoder uses have +//! to be those, cut to the active counts. + +/// The initial `RefPicList0` and `RefPicList1` of a B-frame with POC +/// `current`, as indices into `pocs`, the POCs of the short-term references +/// held. +/// +/// List 0 is the references shown before the frame, nearest first, then +/// those shown after it, nearest first. List 1 is the other way around. If +/// that makes both lists the same, with more than one entry, the first two +/// entries of list 1 change places. +pub(super) fn b_frame_lists(pocs: &[i32], current: i32) -> (Vec, Vec) { + let mut before: Vec = (0..pocs.len()).filter(|&i| pocs[i] < current).collect(); + let mut after: Vec = (0..pocs.len()).filter(|&i| pocs[i] > current).collect(); + before.sort_by_key(|&i| std::cmp::Reverse(pocs[i])); + after.sort_by_key(|&i| pocs[i]); + + let l0: Vec = before.iter().chain(&after).copied().collect(); + let mut l1: Vec = after.iter().chain(&before).copied().collect(); + if l1.len() > 1 && l1 == l0 { + l1.swap(0, 1); + } + (l0, l1) +} + +/// One `ref_pic_list_modification` operation on a short-term reference: +/// whether the picture number goes up from the one before, and by how much, +/// less 1. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) struct ListModification { + pub add: bool, + pub abs_diff_pic_num_minus1: u16, +} + +/// The operations that make `frame_nums`, in this order, the start of +/// `RefPicList0` of a P-frame with `current_frame_num`, whatever its initial +/// list is. +/// +/// The initial list of a P-frame is every short-term reference the decoder +/// holds, newest first in decode order. After a pyramid of B-frames that +/// starts with the B-frame references, which the next P-frame does not +/// predict from. Each operation names a picture by its number relative to +/// the picture before (clause 8.2.4.3.1), the first relative to the current +/// picture. +pub(super) fn p_frame_list_modifications( + current_frame_num: u32, + frame_nums: &[u32], + max_frame_num: u32, +) -> Vec { + let current = current_frame_num as i32; + let mut predicted = current; + frame_nums + .iter() + .map(|&frame_num| { + // `PicNum`: a frame number above the current one is from before + // the last wrap. + let mut pic_num = frame_num as i32; + if pic_num > current { + pic_num -= max_frame_num as i32; + } + let diff = pic_num - predicted; + predicted = pic_num; + ListModification { + add: diff > 0, + abs_diff_pic_num_minus1: (diff.unsigned_abs().max(1) - 1) as u16, + } + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn modifications_skip_the_b_frame_references() { + // I0(0) P4(1) B2(2), then P8 with frame_num 3 predicts from P4 and + // I0: 3 - 2 = 1, then 1 - 1 = 0. + let ops = p_frame_list_modifications(3, &[1, 0], 256); + let sub = |n| ListModification { + add: false, + abs_diff_pic_num_minus1: n, + }; + assert_eq!(ops, [sub(1), sub(0)]); + } + + #[test] + fn modifications_follow_a_frame_num_wrap() { + // Current frame_num 1 after a wrap: 255 is picture number -1. + let ops = p_frame_list_modifications(1, &[255], 256); + assert_eq!(ops[0].abs_diff_pic_num_minus1, 1); + assert!(!ops[0].add); + } + + #[test] + fn nearest_first_on_each_side() { + // References in decode order, newest first: P6, P3, I0. Current B4. + let pocs = [12, 6, 0]; + let (l0, l1) = b_frame_lists(&pocs, 8); + assert_eq!(l0, [1, 2, 0]); + assert_eq!(l1, [0, 1, 2]); + } + + #[test] + fn references_on_both_sides_interleave_by_distance() { + // A pyramid: I0, P8, B4 held, current B2. + let pocs = [0, 16, 8]; + let (l0, l1) = b_frame_lists(&pocs, 4); + assert_eq!(l0, [0, 2, 1]); + assert_eq!(l1, [2, 1, 0]); + } + + #[test] + fn equal_lists_swap_the_first_two_of_list_1() { + // Nothing shown after the frame: both lists would be the same. + let pocs = [0, 2, 4]; + let (l0, l1) = b_frame_lists(&pocs, 6); + assert_eq!(l0, [2, 1, 0]); + assert_eq!(l1, [1, 2, 0]); + } + + #[test] + fn one_reference_is_not_swapped() { + let (l0, l1) = b_frame_lists(&[0], 2); + assert_eq!(l0, [0]); + assert_eq!(l1, [0]); + } +} diff --git a/src/encoder/h264/mod.rs b/src/encoder/h264/mod.rs index 8ed9cfe3..406de16f 100644 --- a/src/encoder/h264/mod.rs +++ b/src/encoder/h264/mod.rs @@ -6,49 +6,68 @@ //! (`record`), and the SPS/PPS generation (`session_params`). mod init; +mod level; +mod lists; mod record; mod session_params; use crate::encoder::ColorDescription; use crate::encoder::codec::{EncoderCommon, FramePlan, PictureSetup, VideoCodec}; -use crate::encoder::pipeline::EncodeFuture; +use crate::encoder::references::{Reference, ReferenceSet}; use crate::error::Result; use ash::vk; /// H.264 macroblock size in pixels. pub const MB_SIZE: u32 = 16; +/// What H.264 needs to know about a reference besides what every codec +/// does. #[derive(Clone, Copy, Debug)] -pub(crate) struct ReferenceInfo { - pub dpb_slot: u8, +pub(crate) struct H264Reference { pub frame_num: u32, - pub poc: i32, - /// Display order (`pts`) of this reference, for reference frame invalidation. - pub display_order: u64, + pub pic_type: ash::vk::native::StdVideoH264PictureType, } +pub(crate) type ReferenceInfo = Reference; + /// H.264-specific encoder state (everything the generic encoder doesn't own). pub(crate) struct H264 { /// `frame_num` syntax element (mod `max_frame_num`). frame_num_syntax: u32, /// IDR picture id, toggled each IDR. idr_pic_id: u32, - /// References held in the DPB, most-recent first. Each frame predicts - /// from the first `active_reference_count`; the rest are kept for - /// reference frame invalidation to fall back on. - l0_references: Vec, + /// References held in the DPB. Each frame predicts from the newest + /// `active_reference_count`; the rest are kept for reference frame + /// invalidation to fall back on. + references: ReferenceSet, /// How many references stay in the DPB. reference_window: u32, /// How many of the newest references one frame predicts from. active_reference_count: u32, - /// `frame_num`s of references dropped by reference frame invalidation that - /// still need to be marked unused-for-reference (via MMCO) in the next - /// P-frame's slice header, so the decoder's DPB matches the encoder's. + /// How many references shown before it a B-frame predicts from. It + /// predicts from one shown after it. + b_l0_reference_count: u32, + /// `frame_num`s of references that the next reference frame marks + /// unused-for-reference (via MMCO) in its slice header, so the decoder's + /// DPB matches the encoder's. pending_unmark_frame_nums: Vec, /// Profile IDC (cached for parameter-set recreation). profile_idc: u32, /// Whether CABAC entropy coding is preferred (from quality-level query). preferred_entropy_cabac: bool, + /// The highest level the device encodes, `VkVideoEncodeH264CapabilitiesKHR::maxLevelIdc`. + max_level_idc: ash::vk::native::StdVideoH264LevelIdc, +} + +/// The picture type of the frame `plan`. +fn picture_type(plan: &FramePlan) -> ash::vk::native::StdVideoH264PictureType { + if plan.is_idr() { + ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_IDR + } else if plan.is_b() { + ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_B + } else { + ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_P + } } impl VideoCodec for H264 { @@ -61,12 +80,31 @@ impl VideoCodec for H264 { self.frame_num_syntax = 0; self.idr_pic_id = (self.idr_pic_id + 1) & 1; // The IDR empties the DPB for the new coded video sequence. - self.l0_references.clear(); + self.references.clear(); // An IDR resets the decoder's whole DPB, so any deferred // unused-reference markings are moot. self.pending_unmark_frame_nums.clear(); } + // A reference frame makes the B-frame references of its level and + // below of no more use. The decoder drops what it is told to, so + // they are marked in this frame's slice header (see `record`). + if plan.is_reference() && !plan.is_idr() { + let level = plan.pyramid_level(); + for dropped in self.references.drop_obsolete(level) { + self.pending_unmark_frame_nums.push(dropped.codec.frame_num); + } + // A frame with marking operations has no sliding window in the + // decoder: it drops only what it is told to. So the reference + // that the window drops after this frame is marked as well. The + // frame still predicts from it: marking comes after decoding. + if !self.pending_unmark_frame_nums.is_empty() + && let Some(victim) = self.references.window_victim(level) + { + self.pending_unmark_frame_nums.push(victim.codec.frame_num); + } + } + let header = if plan.is_idr() { Some(self.build_header(common)?) } else { @@ -76,46 +114,30 @@ impl VideoCodec for H264 { Ok(PictureSetup { header }) } - fn record_picture( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + fn record_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { self.record(common, plan) } fn end_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) { - // `frame_num` carried by this frame (pre-increment), reused below for the - // reference entry even after the syntax counter advances. - let frame_num = self.frame_num_syntax; - let pic_order_cnt = plan.pic_order_cnt(); - - if plan.is_reference() { - self.frame_num_syntax = (frame_num + 1) % 256; - - // The current frame becomes a reference for subsequent frames. - self.l0_references.insert( - 0, - ReferenceInfo { - dpb_slot: common.current_dpb_slot, - frame_num, - poc: pic_order_cnt, - display_order: plan.display_order, - }, - ); - while self.l0_references.len() > self.reference_window as usize { - self.l0_references.pop(); - } - - // Pick the next free DPB slot for the reconstructed picture. - let used_slots: Vec = self.l0_references.iter().map(|r| r.dpb_slot).collect(); - for i in 0..common.dpb_slot_count as u8 { - if !used_slots.contains(&i) { - common.current_dpb_slot = i; - break; - } - } + if !plan.is_reference() { + return; } + // `frame_num` carried by this frame, before the counter advances. + let frame_num = self.frame_num_syntax; + self.frame_num_syntax = (frame_num + 1) % 256; + // The current frame becomes a reference for subsequent frames. + self.references.insert(Reference { + dpb_slot: common.current_dpb_slot, + poc: plan.pic_order_cnt(), + display_order: plan.display_order, + encode_index: plan.encode_index, + level: plan.pyramid_level(), + codec: H264Reference { + frame_num, + pic_type: picture_type(plan), + }, + }); + common.current_dpb_slot = self.references.free_slot(common.dpb_slot_count, None); } fn invalidate_references( @@ -123,22 +145,15 @@ impl VideoCodec for H264 { _common: &mut EncoderCommon, first_lost_display_order: u64, ) -> bool { - // Keep references older than the loss; every newer one is transitively - // undecodable on the client. H.264 maintains the DPB with an implicit - // sliding window, which would keep those newer pictures and desync the - // decoder's reference list from ours — so each dropped reference is - // queued to be explicitly marked unused-for-reference (MMCO) in the next - // P-frame (see `record`). - let mut survivors = Vec::with_capacity(self.l0_references.len()); - for r in self.l0_references.drain(..) { - if r.display_order < first_lost_display_order { - survivors.push(r); - } else { - self.pending_unmark_frame_nums.push(r.frame_num); - } + // H.264 maintains the DPB with an implicit sliding window, which + // would keep the lost pictures and desync the decoder's reference + // list from ours. So each dropped reference is queued to be marked + // unused-for-reference (MMCO) in the next reference frame (see + // `record`). + for lost in self.references.invalidate(first_lost_display_order) { + self.pending_unmark_frame_nums.push(lost.codec.frame_num); } - self.l0_references = survivors; - !self.l0_references.is_empty() + !self.references.is_empty() } fn create_session_params( diff --git a/src/encoder/h264/record.rs b/src/encoder/h264/record.rs index 235d42e2..c2c85077 100644 --- a/src/encoder/h264/record.rs +++ b/src/encoder/h264/record.rs @@ -1,24 +1,25 @@ //! H.264 per-frame encode recording: builds the StdVideo* graph for one frame //! and submits it. This is the part that genuinely differs from the other codecs. -use super::H264; +use super::lists::{b_frame_lists, p_frame_list_modifications}; +use super::{H264, ReferenceInfo}; use crate::encoder::codec::{EncodeCommands, EncoderCommon, FramePlan, RateControlPlan}; -use crate::encoder::pipeline::EncodeFuture; +use crate::encoder::gop::ReferencePattern; use crate::encoder::resources::{prepare_encode_command_buffer, record_dpb_barriers}; -use crate::error::Result; +use crate::error::{PixelForgeError, Result}; use ash::vk; use ash::vk::TaggedStructure; use tracing::debug; +/// What a constant QP is raised by for a B-frame that is not a reference. +const B_FRAME_QP_OFFSET: u32 = 2; + impl H264 { - pub(super) fn record( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + pub(super) fn record(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { let is_idr = plan.is_idr(); let is_reference = plan.is_reference(); + let is_b = plan.is_b(); let pic_order_cnt = plan.pic_order_cnt(); let frame_num = self.frame_num_syntax; @@ -34,11 +35,11 @@ impl H264 { frame_num, pic_order_cnt, is_idr, - self.l0_references.len(), + self.references.len(), common.current_dpb_slot ); - let rc = RateControlPlan::new(&common.config, 26); + let rc = RateControlPlan::for_frame(&common.config, 26, plan, B_FRAME_QP_OFFSET, 51); // Reference frame invalidation: emit MMCO "unmark short-term" operations // for references dropped since the last frame, so the decoder evicts the @@ -46,7 +47,7 @@ impl H264 { // surviving reference. Only valid on a non-IDR reference picture (an IDR // resets the DPB on its own). Kept alive for the whole `record` call so // the pointer handed to the encoder stays valid. - let ref_pic_marking_ops: Vec = + let mut ref_pic_marking_ops: Vec = if !is_idr && is_reference && !self.pending_unmark_frame_nums.is_empty() { self.pending_unmark_frame_nums .iter() @@ -82,7 +83,9 @@ impl H264 { } // Only the references this frame predicts from go to the encode. The // rest of the window stays in its DPB slots, untouched. - let used_references = self.used_references(); + let (list0, list1) = self.reference_lists(plan)?; + let used_references: Vec = list0.iter().chain(&list1).copied().collect(); + let used_references = used_references.as_slice(); let ref_dpb_slots: Vec = used_references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( @@ -99,15 +102,13 @@ impl H264 { let slice_type = if is_idr { ash::vk::native::StdVideoH264SliceType_STD_VIDEO_H264_SLICE_TYPE_I + } else if is_b { + ash::vk::native::StdVideoH264SliceType_STD_VIDEO_H264_SLICE_TYPE_B } else { ash::vk::native::StdVideoH264SliceType_STD_VIDEO_H264_SLICE_TYPE_P }; - let picture_type = if is_idr { - ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_IDR - } else { - ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_P - }; + let picture_type = super::picture_type(plan); // Build StdVideoEncodeH264SliceHeader. // num_ref_idx_active_override_flag is only present in P/B/SP slice headers per the H.264 @@ -118,15 +119,16 @@ impl H264 { let slice_header_flags = ash::vk::native::StdVideoEncodeH264SliceHeaderFlags { _bitfield_align_1: [], _bitfield_1: ash::vk::native::StdVideoEncodeH264SliceHeaderFlags::new_bitfield_1( - 0, // direct_spatial_mv_pred_flag + is_b as u32, // direct_spatial_mv_pred_flag use_ref_override, // num_ref_idx_active_override_flag 0, // reserved ), }; - let slice_qp_delta = match common.config.rate_control { - crate::encoder::RateControl::Cqp { qp } => (qp as i32 - 26) as i8, - _ => 0, + let slice_qp_delta = if rc.is_disabled() { + (rc.qp as i32 - 26) as i8 + } else { + 0 }; let slice_header = ash::vk::native::StdVideoEncodeH264SliceHeader { @@ -139,7 +141,9 @@ impl H264 { reserved1: 0, cabac_init_idc: ash::vk::native::StdVideoH264CabacInitIdc_STD_VIDEO_H264_CABAC_INIT_IDC_0, - disable_deblocking_filter_idc: ash::vk::native::StdVideoH264DisableDeblockingFilterIdc_STD_VIDEO_H264_DISABLE_DEBLOCKING_FILTER_IDC_ENABLED, + // The value is the syntax element `disable_deblocking_filter_idc`: + // `_DISABLED` (0) keeps the deblocking filter on. + disable_deblocking_filter_idc: ash::vk::native::StdVideoH264DisableDeblockingFilterIdc_STD_VIDEO_H264_DISABLE_DEBLOCKING_FILTER_IDC_DISABLED, pWeightTable: std::ptr::null(), }; @@ -159,47 +163,87 @@ impl H264 { // STD_VIDEO_H264_NO_REFERENCE_PICTURE = 0xFF. const NO_REFERENCE_PICTURE: u8 = 0xFF; let mut ref_list0: [u8; 32] = [NO_REFERENCE_PICTURE; 32]; - let ref_list1: [u8; 32] = [NO_REFERENCE_PICTURE; 32]; + let mut ref_list1: [u8; 32] = [NO_REFERENCE_PICTURE; 32]; + for (entry, reference) in ref_list0.iter_mut().zip(&list0) { + *entry = reference.dpb_slot; + } + for (entry, reference) in ref_list1.iter_mut().zip(&list1) { + *entry = reference.dpb_slot; + } + + // A P-frame that marks references unused (B-frame references of no + // more use, or references lost by reference frame invalidation) still + // has them at the start of the list a decoder builds for it, because + // the marking only takes effect after the frame. So it names its + // references. + let mut list0_modifications: Vec = + if !is_idr && !is_b && use_adaptive_marking { + let frame_nums: Vec = list0.iter().map(|r| r.codec.frame_num).collect(); + p_frame_list_modifications(frame_num, &frame_nums, 256) + .into_iter() + .map(|op| ash::vk::native::StdVideoEncodeH264RefListModEntry { + modification_of_pic_nums_idc: if op.add { + ash::vk::native::StdVideoH264ModificationOfPicNumsIdc_STD_VIDEO_H264_MODIFICATION_OF_PIC_NUMS_IDC_SHORT_TERM_ADD + } else { + ash::vk::native::StdVideoH264ModificationOfPicNumsIdc_STD_VIDEO_H264_MODIFICATION_OF_PIC_NUMS_IDC_SHORT_TERM_SUBTRACT + }, + abs_diff_pic_num_minus1: op.abs_diff_pic_num_minus1, + long_term_pic_num: 0, + }) + .collect() + } else { + Vec::new() + }; + + // The NVIDIA driver (615.78.08) writes the operations it is given + // and nothing more, so the operation that ends the list is one of + // them. + if !list0_modifications.is_empty() { + list0_modifications.push(ash::vk::native::StdVideoEncodeH264RefListModEntry { + modification_of_pic_nums_idc: + ash::vk::native::StdVideoH264ModificationOfPicNumsIdc_STD_VIDEO_H264_MODIFICATION_OF_PIC_NUMS_IDC_END, + abs_diff_pic_num_minus1: 0, + long_term_pic_num: 0, + }); + } + // The same driver ends the marking operations by itself only in a + // P-slice with no list operations. In a B-slice, or with list + // operations, the operation that ends them has to be given. + if use_adaptive_marking && (is_b || !list0_modifications.is_empty()) { + ref_pic_marking_ops.push(ash::vk::native::StdVideoEncodeH264RefPicMarkingEntry { + memory_management_control_operation: + ash::vk::native::StdVideoH264MemMgmtControlOp_STD_VIDEO_H264_MEM_MGMT_CONTROL_OP_END, + difference_of_pic_nums_minus1: 0, + long_term_pic_num: 0, + long_term_frame_idx: 0, + max_long_term_frame_idx_plus1: 0, + }); + } let ref_lists_info_flags = ash::vk::native::StdVideoEncodeH264ReferenceListsInfoFlags { _bitfield_align_1: [], _bitfield_1: ash::vk::native::StdVideoEncodeH264ReferenceListsInfoFlags::new_bitfield_1( - 0, // ref_pic_list_modification_flag_l0 - 0, // ref_pic_list_modification_flag_l1 - 0, // reserved + !list0_modifications.is_empty() as u32, // ref_pic_list_modification_flag_l0 + 0, // ref_pic_list_modification_flag_l1 + 0, // reserved ), }; - // Set up the reference list for P-frames. - let num_ref_l0 = if !is_idr && !self.l0_references.is_empty() { - let actual_count = self - .l0_references - .len() - .min(self.active_reference_count as usize) - .min(32); - for (i, ref_info) in self.l0_references.iter().take(actual_count).enumerate() { - ref_list0[i] = ref_info.dpb_slot; - } - actual_count - } else { - 0 - }; - let ref_lists_info = ash::vk::native::StdVideoEncodeH264ReferenceListsInfo { flags: ref_lists_info_flags, - num_ref_idx_l0_active_minus1: if num_ref_l0 > 0 { - (num_ref_l0 - 1) as u8 - } else { - 0 - }, - num_ref_idx_l1_active_minus1: 0, + num_ref_idx_l0_active_minus1: list0.len().saturating_sub(1) as u8, + num_ref_idx_l1_active_minus1: list1.len().saturating_sub(1) as u8, RefPicList0: ref_list0, RefPicList1: ref_list1, - refList0ModOpCount: 0, + refList0ModOpCount: list0_modifications.len() as u8, refList1ModOpCount: 0, refPicMarkingOpCount: ref_pic_marking_ops.len() as u8, reserved1: [0; 7], - pRefList0ModOperations: std::ptr::null(), + pRefList0ModOperations: if list0_modifications.is_empty() { + std::ptr::null() + } else { + list0_modifications.as_ptr() + }, pRefList1ModOperations: std::ptr::null(), pRefPicMarkingOperations: if ref_pic_marking_ops.is_empty() { std::ptr::null() @@ -218,7 +262,7 @@ impl H264 { PicOrderCnt: pic_order_cnt, temporal_id: 0, reserved1: [0; 3], - pRefLists: if !is_idr && (!self.l0_references.is_empty() || use_adaptive_marking) { + pRefLists: if !is_idr && (!used_references.is_empty() || use_adaptive_marking) { &ref_lists_info } else { std::ptr::null() @@ -273,9 +317,8 @@ impl H264 { ); l0_std_infos.push(ash::vk::native::StdVideoEncodeH264ReferenceInfo { flags: std_reference_info_flags, - primary_pic_type: - ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_P, - FrameNum: ref_info.frame_num, + primary_pic_type: ref_info.codec.pic_type, + FrameNum: ref_info.codec.frame_num, PicOrderCnt: ref_info.poc, long_term_pic_num: 0, long_term_frame_idx: 0, @@ -307,11 +350,7 @@ impl H264 { // Reference info for the setup slot (this frame). let std_reference_info = ash::vk::native::StdVideoEncodeH264ReferenceInfo { flags: std_reference_info_flags, - primary_pic_type: if is_idr { - ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_IDR - } else { - ash::vk::native::StdVideoH264PictureType_STD_VIDEO_H264_PICTURE_TYPE_P - }, + primary_pic_type: picture_type, FrameNum: frame_num, PicOrderCnt: pic_order_cnt, long_term_pic_num: 0, @@ -356,30 +395,23 @@ impl H264 { reference_slots_for_begin.extend_from_slice(&l0_slots); } - // Rate control. - let qp_bounds = if rc.is_disabled() { rc.qp as i32 } else { 18 }; - let qp_bounds_max = if rc.is_disabled() { rc.qp as i32 } else { 42 }; - let min_qp = vk::VideoEncodeH264QpKHR { - qp_i: qp_bounds, - qp_p: qp_bounds, - qp_b: qp_bounds, - }; - let max_qp = vk::VideoEncodeH264QpKHR { - qp_i: qp_bounds_max, - qp_p: qp_bounds_max, - qp_b: qp_bounds_max, + // Rate control. With CQP the bounds hold the QP. With CBR and VBR + // there are no bounds: a fixed range keeps the driver from reaching + // the target on easy or hard content. + let qp = vk::VideoEncodeH264QpKHR { + qp_i: rc.qp as i32, + qp_p: rc.qp as i32, + qp_b: rc.qp as i32, }; let codec_layer = vk::VideoEncodeH264RateControlLayerInfoKHR::default() - .use_min_qp(true) - .min_qp(min_qp) - .use_max_qp(true) - .max_qp(max_qp); - let codec_rate_control = vk::VideoEncodeH264RateControlInfoKHR::default() - .gop_frame_count(common.config.gop_size) - .idr_period(common.config.gop_size); + .use_min_qp(rc.is_disabled()) + .min_qp(qp) + .use_max_qp(rc.is_disabled()) + .max_qp(qp); + let codec_rate_control = self.rate_control_info(common); let first_frame = plan.is_first_frame(); - let future = common.record_encode(EncodeCommands { + common.record_encode(EncodeCommands { begin_slots: &reference_slots_for_begin, encode_info: &encode_info, rate_control: &rc, @@ -396,6 +428,79 @@ impl H264 { if use_adaptive_marking { self.pending_unmark_frame_nums.clear(); } - Ok(future) + Ok(()) + } + + /// The references this frame predicts from, as `RefPicList0` and + /// `RefPicList1`. + /// + /// A P-frame uses the newest references, in decode order, which is the + /// order a decoder puts them in. A B-frame uses the nearest ones shown + /// before it and the nearest one shown after it. + fn reference_lists( + &self, + plan: &FramePlan, + ) -> Result<(Vec, Vec)> { + if plan.is_idr() { + return Ok((Vec::new(), Vec::new())); + } + if !plan.is_b() { + return Ok((self.used_references().to_vec(), Vec::new())); + } + let current = plan.pic_order_cnt(); + let references = self.references.as_slice(); + let pocs: Vec = references.iter().map(|r| r.poc).collect(); + let before = pocs.iter().filter(|&&poc| poc < current).count(); + let after = pocs.len() - before; + if before == 0 || after == 0 { + return Err(PixelForgeError::InvalidInput(format!( + "H.264 B-frame with POC {current} has {before} references before it and \ + {after} after it" + ))); + } + let (l0, l1) = b_frame_lists(&pocs, current); + let take = |list: Vec, count: usize| -> Vec { + list.into_iter() + .take(count) + .map(|i| references[i]) + .collect() + }; + // Only references on the frame's own side of each list, so that no + // picture is in both and the count stays within the session's. + let l0_count = before.min(self.b_l0_reference_count as usize); + Ok((take(l0, l0_count), take(l1, 1))) + } + + /// The GOP the rate control is told to expect. + fn rate_control_info( + &self, + common: &EncoderCommon, + ) -> vk::VideoEncodeH264RateControlInfoKHR<'static> { + let config = &common.config; + let info = vk::VideoEncodeH264RateControlInfoKHR::default() + .gop_frame_count(config.gop_size) + .idr_period(config.gop_size); + if config.b_frames == 0 { + return info; + } + // The count of B-frames has to be below the GOP length, so a GOP + // with no end is given the longest length there is. + let gop_frame_count = if config.gop_size == 0 { + u32::MAX + } else { + config.gop_size + }; + let pattern = match common.gop.reference_pattern() { + Some(ReferencePattern::Flat) => { + vk::VideoEncodeH264RateControlFlagsKHR::REFERENCE_PATTERN_FLAT + } + Some(ReferencePattern::Dyadic) => { + vk::VideoEncodeH264RateControlFlagsKHR::REFERENCE_PATTERN_DYADIC + } + None => vk::VideoEncodeH264RateControlFlagsKHR::empty(), + }; + info.gop_frame_count(gop_frame_count) + .consecutive_b_frame_count(config.b_frames.min(gop_frame_count - 1)) + .flags(vk::VideoEncodeH264RateControlFlagsKHR::REGULAR_GOP | pattern) } } diff --git a/src/encoder/h264/session_params.rs b/src/encoder/h264/session_params.rs index f9f644ae..16043386 100644 --- a/src/encoder/h264/session_params.rs +++ b/src/encoder/h264/session_params.rs @@ -1,13 +1,14 @@ //! H.264 parameter-set generation: SPS/PPS construction and header retrieval. -use super::H264; +use super::{H264, level}; use crate::encoder::codec::EncoderCommon; use crate::encoder::resources::get_encoded_session_params; -use crate::encoder::{BitDepth, ChromaFormat, ColorDescription}; +use crate::encoder::{BitDepth, ChromaFormat, ColorDescription, RateControl}; use crate::error::{PixelForgeError, Result}; use ash::vk; use ash::vk::TaggedStructure; +use ash::vk::native::StdVideoH264LevelIdc; use std::ptr; impl H264 { @@ -71,7 +72,20 @@ impl H264 { let max_active = self.active_reference_count() as u8; // The decoder's sliding window holds as many frames as the encoder // keeps, so older ones survive for reference frame invalidation. - let window = self.reference_window() as u8; + let window = self.reference_window() as u8 + common.gop.b_reference_count() as u8; + // With B-frames a decoder has to hold frames back to show them in + // order, and the POC no longer follows from the decode order. + let reorder_delay = common.gop.reorder_delay() as u8; + // The DPB holds the references, and one frame more than the reorder + // delay: a frame that is not a reference waits there to be shown + // only while that many frames shown after it are held. + let dpb_frames = window.max(reorder_delay + 1); + let level_idc = self.level_idc(common, dpb_frames); + let pic_order_cnt_type = if config.b_frames > 0 { + ash::vk::native::StdVideoH264PocType_STD_VIDEO_H264_POC_TYPE_0 + } else { + ash::vk::native::StdVideoH264PocType_STD_VIDEO_H264_POC_TYPE_2 + }; let mut vui_flags: ash::vk::native::StdVideoH264SpsVuiFlags = unsafe { std::mem::zeroed() }; vui_flags.set_aspect_ratio_info_present_flag(1); @@ -100,8 +114,8 @@ impl H264 { matrix_coefficients: desc.matrix_coefficients, num_units_in_tick: 0, time_scale: 0, - max_num_reorder_frames: 0, - max_dec_frame_buffering: window + 1, + max_num_reorder_frames: reorder_delay, + max_dec_frame_buffering: dpb_frames, chroma_sample_loc_type_top_field: 0, chroma_sample_loc_type_bottom_field: 0, reserved1: 0, @@ -111,13 +125,13 @@ impl H264 { let sps = ash::vk::native::StdVideoH264SequenceParameterSet { flags: sps_flags, profile_idc: self.profile_idc(), - level_idc: ash::vk::native::StdVideoH264LevelIdc_STD_VIDEO_H264_LEVEL_IDC_4_1, + level_idc, chroma_format_idc, seq_parameter_set_id: 0, bit_depth_luma_minus8, bit_depth_chroma_minus8, log2_max_frame_num_minus4: 4, - pic_order_cnt_type: ash::vk::native::StdVideoH264PocType_STD_VIDEO_H264_POC_TYPE_2, + pic_order_cnt_type, offset_for_non_ref_pic: 0, offset_for_top_to_bottom_field: 0, log2_max_pic_order_cnt_lsb_minus4: 4, @@ -179,7 +193,8 @@ impl H264 { .parameters_add_info(&h264_add_info); // Chain quality level info (required by AMD RADV; matches FFmpeg). - let mut quality_level_info = vk::VideoEncodeQualityLevelInfoKHR::default().quality_level(0); + let mut quality_level_info = vk::VideoEncodeQualityLevelInfoKHR::default() + .quality_level(common.config.quality_level); let params_create_info = vk::VideoSessionParametersCreateInfoKHR::default() .video_session(common.session) @@ -207,6 +222,38 @@ impl H264 { Ok(session_params) } + /// The level of the stream: the lowest from level 4.1 that allows its + /// size, frame rate, DPB of `dpb_frames` and peak bitrate, and at most + /// the highest level the device has. + fn level_idc(&self, common: &EncoderCommon, dpb_frames: u8) -> StdVideoH264LevelIdc { + let config = &common.config; + let rate = config.frame_rate; + let max_bitrate = match config.rate_control { + RateControl::Cqp { .. } => 0, + RateControl::Cbr { bitrate } => u64::from(bitrate), + RateControl::Vbr { max, .. } => u64::from(max), + }; + let high_444 = self.profile_idc() + == ash::vk::native::StdVideoH264ProfileIdc_STD_VIDEO_H264_PROFILE_IDC_HIGH_444_PREDICTIVE; + let needed = level::level_for(&level::Needs { + width_in_mbs: common.aligned_width / 16, + height_in_mbs: common.aligned_height / 16, + frame_rate: rate.numerator.div_ceil(rate.denominator.max(1)), + dpb_frames: u32::from(dpb_frames), + max_bitrate, + bitrate_factor: if high_444 { 4000 } else { 1250 }, + }); + if needed > self.max_level_idc { + tracing::warn!( + "H.264: the stream needs level {needed} (as StdVideoH264LevelIdc), and the \ + device encodes up to {}, so the stream says that", + self.max_level_idc + ); + return self.max_level_idc; + } + needed + } + /// Retrieve the encoded SPS/PPS NAL units to prepend to an IDR frame. pub(super) fn build_header(&self, common: &EncoderCommon) -> Result> { let mut h264_get_info = vk::VideoEncodeH264SessionParametersGetInfoKHR::default() diff --git a/src/encoder/h265/init.rs b/src/encoder/h265/init.rs index 484418d6..1fe928fc 100644 --- a/src/encoder/h265/init.rs +++ b/src/encoder/h265/init.rs @@ -1,9 +1,10 @@ use super::{CTB_SIZE, H265}; +use crate::encoder::references::ReferenceSet; use crate::converter::RgbFormat; use crate::encoder::EncodeConfig; use crate::encoder::codec::{ - CodecEncoder, CommonInitRequest, build_encoder_common, query_video_caps, + CodecEncoder, CommonInitRequest, build_encoder_common, check_quality_level, query_video_caps, }; use crate::encoder::profile::EncodeProfile; use crate::encoder::profile::h265_profile_idc; @@ -50,6 +51,7 @@ impl H265 { capabilities = capabilities.push(&mut rgb_caps); } let caps = query_video_caps(&context, profile_info, &mut capabilities)?; + check_quality_level(&config, encode_caps.max_quality_levels)?; let rgb_caps = rgb_input.map(|_| RgbConversionCaps::from(&rgb_caps)); // Scale the bitstream buffer by resolution so a single frame can't overflow (matches AV1). @@ -68,16 +70,23 @@ impl H265 { bitstream_buffer_size, allow_layered_dpb: true, rgb_caps, + b_frame_support: crate::encoder::bframes::h265_support( + context.device_properties().vendor_id, + &h265_caps, + caps.max_active_reference_pictures, + ), })?; let reference_window = init.reference_window; let active_reference_count = init.active_reference_count; + let b_l0_reference_count = (init.session_active_references.saturating_sub(1)) + .min(h265_caps.max_b_picture_l0_reference_count); let common = init.common; let codec = H265 { header_data: None, - l0_references: Vec::new(), - reference_window, + references: ReferenceSet::new(reference_window), active_reference_count, + b_l0_reference_count, profile_idc, }; diff --git a/src/encoder/h265/mod.rs b/src/encoder/h265/mod.rs index 0c32ca8b..bf63aba6 100644 --- a/src/encoder/h265/mod.rs +++ b/src/encoder/h265/mod.rs @@ -10,45 +10,46 @@ mod session_params; use crate::encoder::ColorDescription; use crate::encoder::codec::{EncoderCommon, FramePlan, PictureSetup, VideoCodec}; -use crate::encoder::pipeline::EncodeFuture; +use crate::encoder::references::{Reference, ReferenceSet}; use crate::error::Result; use ash::vk; /// H.265 Coding Tree Block (CTB) size in pixels. pub const CTB_SIZE: u32 = 32; +/// What H.265 needs to know about a reference besides what every codec +/// does. #[derive(Clone, Copy, Debug)] -pub(crate) struct ReferenceInfo { - pub dpb_slot: u8, - pub poc: i32, - /// Display order (`pts`) of this reference, for reference frame invalidation. - pub display_order: u64, +pub(crate) struct H265Reference { + pub pic_type: ash::vk::native::StdVideoH265PictureType, } +pub(crate) type ReferenceInfo = Reference; + /// H.265-specific encoder state. pub(crate) struct H265 { /// Cached VPS/SPS/PPS header (invalidated when session params change). header_data: Option>, - /// References held in the DPB, most-recent first. Each frame predicts - /// from the first `active_reference_count`; the rest stay in its - /// reference picture set, unused, for reference frame invalidation to fall - /// back on. - l0_references: Vec, - /// How many references stay in the DPB. - reference_window: u32, + /// References held in the DPB. Each frame predicts from the newest + /// `active_reference_count`; the rest stay in its reference picture set, + /// unused, for reference frame invalidation to fall back on. + references: ReferenceSet, /// How many of the newest references one frame predicts from. active_reference_count: u32, + /// How many references shown before it a B-frame predicts from. It + /// predicts from one shown after it. + b_l0_reference_count: u32, profile_idc: u32, } -impl H265 { - /// The references this frame predicts from: the newest held ones. - fn used_references(&self) -> &[ReferenceInfo] { - let n = self - .l0_references - .len() - .min(self.active_reference_count as usize); - &self.l0_references[..n] +/// The picture type of the frame `plan`. +fn picture_type(plan: &FramePlan) -> ash::vk::native::StdVideoH265PictureType { + if plan.is_idr() { + ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_IDR + } else if plan.is_b() { + ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_B + } else { + ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_P } } @@ -59,9 +60,14 @@ impl VideoCodec for H265 { plan: &FramePlan, ) -> Result { if plan.is_idr() { - self.l0_references.clear(); + self.references.clear(); // All DPB slots become inactive at the start of a coded video sequence. common.dpb_slot_active.fill(false); + } else if plan.is_reference() { + // A reference frame makes the B-frame references of its level + // and below of no more use. Its reference picture set leaves + // them out, which is what makes a decoder drop them. + self.references.drop_obsolete(plan.pyramid_level()); } let header = if plan.is_idr() { @@ -76,11 +82,7 @@ impl VideoCodec for H265 { Ok(PictureSetup { header }) } - fn record_picture( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + fn record_picture(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { self.record(common, plan) } @@ -88,27 +90,17 @@ impl VideoCodec for H265 { if !plan.is_reference() { return; } - let pic_order_cnt = plan.pic_order_cnt(); - - self.l0_references.insert( - 0, - ReferenceInfo { - dpb_slot: common.current_dpb_slot, - poc: pic_order_cnt, - display_order: plan.display_order, + self.references.insert(Reference { + dpb_slot: common.current_dpb_slot, + poc: plan.pic_order_cnt(), + display_order: plan.display_order, + encode_index: plan.encode_index, + level: plan.pyramid_level(), + codec: H265Reference { + pic_type: picture_type(plan), }, - ); - while self.l0_references.len() > self.reference_window as usize { - self.l0_references.pop(); - } - - let used_slots: Vec = self.l0_references.iter().map(|r| r.dpb_slot).collect(); - for i in 0..common.dpb_slot_count as u8 { - if !used_slots.contains(&i) { - common.current_dpb_slot = i; - break; - } - } + }); + common.current_dpb_slot = self.references.free_slot(common.dpb_slot_count, None); } fn invalidate_references( @@ -116,14 +108,12 @@ impl VideoCodec for H265 { _common: &mut EncoderCommon, first_lost_display_order: u64, ) -> bool { - // Every reference at or after the first lost frame is transitively - // undecodable on the client; keep only older survivors. H.265 declares - // its short-term reference picture set explicitly every frame from - // `l0_references` (see `record`), so dropping entries here is enough for - // the decoder to follow — no extra bitstream marking is required. - self.l0_references - .retain(|r| r.display_order < first_lost_display_order); - !self.l0_references.is_empty() + // H.265 declares its short-term reference picture set explicitly every + // frame from `references` (see `record`), so dropping entries here is + // enough for the decoder to follow: no extra bitstream marking is + // required. + self.references.invalidate(first_lost_display_order); + !self.references.is_empty() } fn create_session_params( diff --git a/src/encoder/h265/record.rs b/src/encoder/h265/record.rs index 6d1228df..a4942575 100644 --- a/src/encoder/h265/record.rs +++ b/src/encoder/h265/record.rs @@ -1,24 +1,25 @@ //! H.265 per-frame encode recording: builds the StdVideo* graph for one frame //! and submits it. -use super::H265; +use super::{H265, ReferenceInfo}; +use crate::encoder::EncodeConfig; use crate::encoder::codec::{EncodeCommands, EncoderCommon, FramePlan, RateControlPlan}; -use crate::encoder::pipeline::EncodeFuture; +use crate::encoder::gop::ReferencePattern; use crate::encoder::resources::{prepare_encode_command_buffer, record_dpb_barriers}; -use crate::error::Result; +use crate::error::{PixelForgeError, Result}; use ash::vk; use ash::vk::TaggedStructure; use tracing::debug; +/// What a constant QP is raised by for a B-frame that is not a reference. +const B_FRAME_QP_OFFSET: u32 = 2; + impl H265 { - pub(super) fn record( - &mut self, - common: &mut EncoderCommon, - plan: &FramePlan, - ) -> Result { + pub(super) fn record(&mut self, common: &mut EncoderCommon, plan: &FramePlan) -> Result<()> { let is_idr = plan.is_idr(); let is_reference = plan.is_reference(); + let is_b = plan.is_b(); let pic_order_cnt = plan.pic_order_cnt(); let slot = common.pipeline.current(); @@ -33,7 +34,14 @@ impl H265 { } // Only the references this frame predicts from go to the encode. The // rest of the window stays in its DPB slots, untouched. - let used_references = self.used_references(); + let (before, after) = self.references_around(pic_order_cnt); + let (used_before, used_after) = self.used_counts(plan, before.len(), after.len())?; + let used_references: Vec = before[..used_before] + .iter() + .chain(&after[..used_after]) + .copied() + .collect(); + let used_references = used_references.as_slice(); let ref_dpb_slots: Vec = used_references.iter().map(|r| r.dpb_slot).collect(); unsafe { record_dpb_barriers( @@ -50,14 +58,12 @@ impl H265 { let slice_type = if is_idr { ash::vk::native::StdVideoH265SliceType_STD_VIDEO_H265_SLICE_TYPE_I + } else if is_b { + ash::vk::native::StdVideoH265SliceType_STD_VIDEO_H265_SLICE_TYPE_B } else { ash::vk::native::StdVideoH265SliceType_STD_VIDEO_H265_SLICE_TYPE_P }; - let picture_type = if is_idr { - ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_IDR - } else { - ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_P - }; + let picture_type = super::picture_type(plan); let slice_header_flags = ash::vk::native::StdVideoEncodeH265SliceSegmentHeaderFlags { _bitfield_align_1: [], @@ -95,37 +101,32 @@ impl H265 { pWeightTable: std::ptr::null(), }; - // Short-term reference picture set: negative (past) refs for P, plus a - // positive (future) ref for B. + // Short-term reference picture set: every held reference, shown + // before the frame (negative) or after it (positive), nearest first. + // A reference left out is one the decoder drops, so all of them stay + // in the set, but only the nearest are used by this picture. let mut delta_poc_s0_minus1 = [0u16; 16]; - let mut num_negative_pics: u8 = 0; + let mut delta_poc_s1_minus1 = [0u16; 16]; let mut used_by_curr_pic_s0_flag: u16 = 0; - - if !is_idr && !self.l0_references.is_empty() { - // max_poc = 2^(log2_max_pic_order_cnt_lsb_minus4 + 4) * 2 = 512. - let max_poc = 1i32 << 9; - let mut prev_delta_poc = 0; - for (i, ref_info) in self.l0_references.iter().enumerate() { - if i >= 15 { - break; - } - let mut delta_poc = ref_info.poc - pic_order_cnt; - if delta_poc > max_poc / 2 { - delta_poc -= max_poc; - } else if delta_poc < -max_poc / 2 { - delta_poc += max_poc; - } - let diff = prev_delta_poc - delta_poc; - delta_poc_s0_minus1[num_negative_pics as usize] = (diff - 1).max(0) as u16; - prev_delta_poc = delta_poc; - // Every held reference stays in the set, so the decoder keeps - // it too, but only the newest are used by this picture. - if i < used_references.len() { - used_by_curr_pic_s0_flag |= 1 << num_negative_pics; - } - num_negative_pics += 1; + let mut used_by_curr_pic_s1_flag: u16 = 0; + let mut previous = pic_order_cnt; + for (i, reference) in before.iter().enumerate() { + delta_poc_s0_minus1[i] = (previous - reference.poc - 1).max(0) as u16; + previous = reference.poc; + if i < used_before { + used_by_curr_pic_s0_flag |= 1 << i; + } + } + let mut previous = pic_order_cnt; + for (i, reference) in after.iter().enumerate() { + delta_poc_s1_minus1[i] = (reference.poc - previous - 1).max(0) as u16; + previous = reference.poc; + if i < used_after { + used_by_curr_pic_s1_flag |= 1 << i; } } + let num_negative_pics = before.len() as u8; + let num_positive_pics = after.len() as u8; let frame_rps = ash::vk::native::StdVideoH265ShortTermRefPicSet { flags: ash::vk::native::StdVideoH265ShortTermRefPicSetFlags { @@ -140,14 +141,14 @@ impl H265 { abs_delta_rps_minus1: 0, used_by_curr_pic_flag: 0, used_by_curr_pic_s0_flag, - used_by_curr_pic_s1_flag: 0, + used_by_curr_pic_s1_flag, reserved1: 0, reserved2: 0, reserved3: 0, num_negative_pics, - num_positive_pics: 0, + num_positive_pics, delta_poc_s0_minus1, - delta_poc_s1_minus1: [0; 16], + delta_poc_s1_minus1, }; let empty_rps = ash::vk::native::StdVideoH265ShortTermRefPicSet { @@ -173,6 +174,9 @@ impl H265 { delta_poc_s1_minus1: [0; 16], }; + // With B-frames a decoder still holds pictures to show when an IDR + // comes, and this flag would make it drop them. + let no_output_of_prior_pics = (is_idr && common.config.b_frames == 0) as u32; let picture_info_flags = ash::vk::native::StdVideoEncodeH265PictureInfoFlags { _bitfield_align_1: [], _bitfield_1: ash::vk::native::StdVideoEncodeH265PictureInfoFlags::new_bitfield_1( @@ -182,7 +186,7 @@ impl H265 { 0, // discardable_flag 0, // cross_layer_bla_flag 1, // pic_output_flag - if is_idr { 1 } else { 0 }, // no_output_of_prior_pics_flag + no_output_of_prior_pics, // no_output_of_prior_pics_flag 0, // short_term_ref_pic_set_sps_flag 0, // slice_temporal_mvp_enabled_flag 0, // reserved @@ -190,20 +194,16 @@ impl H265 { }; const NO_REFERENCE_PICTURE: u8 = 0xFF; + // The lists a decoder builds by default: list 0 is the references + // used from before the frame, list 1 those from after it. let mut ref_list0: [u8; 15] = [NO_REFERENCE_PICTURE; 15]; - let ref_list1: [u8; 15] = [NO_REFERENCE_PICTURE; 15]; - - let num_ref_l0 = if !is_idr && !self.l0_references.is_empty() { - let count = used_references.len(); - for (i, ref_info) in used_references.iter().enumerate() { - if i < 15 { - ref_list0[i] = ref_info.dpb_slot; - } - } - count.min(15) - } else { - 0 - }; + let mut ref_list1: [u8; 15] = [NO_REFERENCE_PICTURE; 15]; + for (entry, reference) in ref_list0.iter_mut().zip(&before[..used_before]) { + *entry = reference.dpb_slot; + } + for (entry, reference) in ref_list1.iter_mut().zip(&after[..used_after]) { + *entry = reference.dpb_slot; + } let ref_lists_info = ash::vk::native::StdVideoEncodeH265ReferenceListsInfo { flags: ash::vk::native::StdVideoEncodeH265ReferenceListsInfoFlags { @@ -213,12 +213,8 @@ impl H265 { 0, 0, 0, ), }, - num_ref_idx_l0_active_minus1: if num_ref_l0 > 0 { - (num_ref_l0 - 1) as u8 - } else { - 0 - }, - num_ref_idx_l1_active_minus1: 0, + num_ref_idx_l0_active_minus1: used_before.saturating_sub(1) as u8, + num_ref_idx_l1_active_minus1: used_after.saturating_sub(1) as u8, RefPicList0: ref_list0, RefPicList1: ref_list1, list_entry_l0: [0; 15], @@ -233,14 +229,14 @@ impl H265 { PicOrderCntVal: pic_order_cnt, TemporalId: 0, reserved1: [0; 7], - pRefLists: if !is_idr && !self.l0_references.is_empty() { + pRefLists: if !used_references.is_empty() { &ref_lists_info } else { std::ptr::null() }, pShortTermRefPicSet: if is_idr { &empty_rps - } else if !self.l0_references.is_empty() { + } else if !self.references.is_empty() { &frame_rps } else { &empty_rps @@ -250,7 +246,7 @@ impl H265 { short_term_ref_pic_set_idx: 0, }; - let rc = RateControlPlan::new(&common.config, 26); + let rc = RateControlPlan::for_frame(&common.config, 26, plan, B_FRAME_QP_OFFSET, 51); let constant_qp = if rc.is_disabled() { rc.qp as i32 } else { 0 }; let nalu_slice_entries = [vk::VideoEncodeH265NaluSliceSegmentInfoKHR::default() .constant_qp(constant_qp) @@ -312,7 +308,7 @@ impl H265 { let mut reference_slots_for_begin = Vec::with_capacity(17); reference_slots_for_begin.push(setup_slot_for_begin); - let has_l0_ref = !is_idr && !self.l0_references.is_empty(); + let has_l0_ref = !used_references.is_empty(); if has_l0_ref { for ref_info in used_references { @@ -325,8 +321,7 @@ impl H265 { ); let mut std_info = std_reference_info; std_info.PicOrderCntVal = ref_info.poc; - std_info.pic_type = - ash::vk::native::StdVideoH265PictureType_STD_VIDEO_H265_PICTURE_TYPE_P; + std_info.pic_type = ref_info.codec.pic_type; std_ref_infos.push(std_info); } } @@ -368,9 +363,7 @@ impl H265 { let codec_layer = vk::VideoEncodeH265RateControlLayerInfoKHR::default() .min_qp(min_qp) .max_qp(max_qp); - let codec_rate_control = vk::VideoEncodeH265RateControlInfoKHR::default() - .gop_frame_count(common.config.gop_size) - .idr_period(common.config.gop_size); + let codec_rate_control = rate_control_info(&common.config, common.gop.reference_pattern()); let encode_info = vk::VideoEncodeInfoKHR::default() .flags(vk::VideoEncodeFlagsKHR::empty()) @@ -386,7 +379,7 @@ impl H265 { "h265 submit frame {}: idr={}, num_refs={}, cur_slot={}", plan.encode_index, is_idr, - self.l0_references.len(), + self.references.len(), common.current_dpb_slot ); @@ -402,4 +395,65 @@ impl H265 { post_encode_barrier: true, }) } + + /// The held references shown before the picture with POC `current`, and + /// those shown after it, each nearest first. None for an IDR, which has + /// emptied the list. + fn references_around(&self, current: i32) -> (Vec, Vec) { + // A reference picture set holds 16 pictures at most. The window and + // the B-frame references are never more than that. + debug_assert!(self.references.len() <= 16); + self.references.around(current) + } + + /// How many of the `before` references shown before it and of the + /// `after` shown after it this frame predicts from. + fn used_counts(&self, plan: &FramePlan, before: usize, after: usize) -> Result<(usize, usize)> { + if plan.is_idr() { + return Ok((0, 0)); + } + if !plan.is_b() { + return Ok((before.min(self.active_reference_count as usize), 0)); + } + if before == 0 || after == 0 { + return Err(PixelForgeError::InvalidInput(format!( + "H.265 B-frame with POC {} has {before} references before it and {after} \ + after it", + plan.pic_order_cnt() + ))); + } + Ok((before.min(self.b_l0_reference_count as usize), 1)) + } +} + +/// The GOP the rate control is told to expect. +fn rate_control_info( + config: &EncodeConfig, + pattern: Option, +) -> vk::VideoEncodeH265RateControlInfoKHR<'static> { + let info = vk::VideoEncodeH265RateControlInfoKHR::default() + .gop_frame_count(config.gop_size) + .idr_period(config.gop_size); + if config.b_frames == 0 { + return info; + } + // The count of B-frames has to be below the GOP length, so a GOP with no + // end is given the longest length there is. + let gop_frame_count = if config.gop_size == 0 { + u32::MAX + } else { + config.gop_size + }; + let pattern = match pattern { + Some(ReferencePattern::Flat) => { + vk::VideoEncodeH265RateControlFlagsKHR::REFERENCE_PATTERN_FLAT + } + Some(ReferencePattern::Dyadic) => { + vk::VideoEncodeH265RateControlFlagsKHR::REFERENCE_PATTERN_DYADIC + } + None => vk::VideoEncodeH265RateControlFlagsKHR::empty(), + }; + info.gop_frame_count(gop_frame_count) + .consecutive_b_frame_count(config.b_frames.min(gop_frame_count - 1)) + .flags(vk::VideoEncodeH265RateControlFlagsKHR::REGULAR_GOP | pattern) } diff --git a/src/encoder/h265/session_params.rs b/src/encoder/h265/session_params.rs index cd562f67..7f7c13c7 100644 --- a/src/encoder/h265/session_params.rs +++ b/src/encoder/h265/session_params.rs @@ -20,7 +20,9 @@ impl H265 { let config = &common.config; let ctb_log2_size_y: u8 = 5; - let min_cb_log2_size_y: u8 = 4; + // 8x8, the smallest coding block. Vulkan has no capability for it, and + // a larger minimum stops the encoder from using small blocks. + let min_cb_log2_size_y: u8 = 3; let log2_min_transform_block_size: u8 = 2; let log2_max_transform_block_size: u8 = 5; @@ -52,7 +54,7 @@ impl H265 { let dec_pic_buf_mgr = ash::vk::native::StdVideoH265DecPicBufMgr { max_latency_increase_plus1: [0; 7], max_dec_pic_buffering_minus1: [(common.dpb_slot_count - 1) as u8, 0, 0, 0, 0, 0, 0], - max_num_reorder_pics: [0; 7], + max_num_reorder_pics: [common.gop.reorder_delay() as u8, 0, 0, 0, 0, 0, 0], }; let short_term_ref_pic_set = ash::vk::native::StdVideoH265ShortTermRefPicSet { @@ -335,7 +337,8 @@ impl H265 { .max_std_pps_count(1) .parameters_add_info(&h265_add_info); - let mut quality_level_info = vk::VideoEncodeQualityLevelInfoKHR::default().quality_level(0); + let mut quality_level_info = vk::VideoEncodeQualityLevelInfoKHR::default() + .quality_level(common.config.quality_level); let params_create_info = vk::VideoSessionParametersCreateInfoKHR::default() .video_session(common.session) diff --git a/src/encoder/mod.rs b/src/encoder/mod.rs index b0ad6347..6d667a5c 100644 --- a/src/encoder/mod.rs +++ b/src/encoder/mod.rs @@ -6,6 +6,7 @@ mod api; pub(crate) mod av1; +mod bframes; pub(crate) mod codec; #[cfg(feature = "dmabuf")] pub mod external; @@ -16,6 +17,7 @@ pub(crate) mod h265; pub(crate) mod input; pub(crate) mod pipeline; pub(crate) mod profile; +pub(crate) mod references; pub(crate) mod resources; pub(crate) mod rgb; mod source; @@ -419,6 +421,12 @@ pub struct EncodeConfig { /// as the device allows, which can be fewer. The device's DPB size can /// also lower it. pub max_reference_frames: u32, + /// How many B-frames go between two reference frames; see + /// [`Self::with_b_frames`]. + pub b_frames: u32, + /// Whether B-frames are references; see + /// [`Self::with_b_frame_references`]. + pub b_frame_references: BFrameReferences, /// VBV/HRD virtual buffer size in milliseconds, for /// [`RateControl::Cbr`] and [`RateControl::Vbr`]. /// Controls how much the encoder can deviate from the target bitrate @@ -447,6 +455,8 @@ pub struct EncodeConfig { pub encode_content_hint: EncodeContentHint, /// Encoder tuning mode. pub encoder_tuning_mode: EncoderTuningMode, + /// The driver's quality level; see [`Self::with_quality_level`]. + pub quality_level: u32, } impl EncodeConfig { @@ -467,6 +477,8 @@ impl EncodeConfig { }, gop_size: DEFAULT_GOP_SIZE, max_reference_frames: DEFAULT_MAX_REFERENCE_FRAMES, + b_frames: 0, + b_frame_references: BFrameReferences::Flat, virtual_buffer_size_ms: 1000, initial_virtual_buffer_size_ms: 1000, color: StreamColor::default(), @@ -475,6 +487,7 @@ impl EncodeConfig { encode_usage_hint: EncodeUsageHint::Default, encode_content_hint: EncodeContentHint::Default, encoder_tuning_mode: EncoderTuningMode::Default, + quality_level: 0, } } @@ -533,6 +546,38 @@ impl EncodeConfig { self } + /// Encode `count` B-frames between two reference frames, at most + /// [`MAX_B_FRAMES`]. 0, the default, encodes none and adds no latency. + /// + /// A B-frame predicts from a frame shown after it, so it is encoded only + /// once that frame was submitted. That adds `count` frames of latency, + /// and it changes what [`Encoder::submit`] gives back: the future of the + /// k-th submit resolves with the k-th packet of the stream, in decode + /// order, which is the packet of another frame. So a future may resolve + /// only after later submits, and `block_on(encoder.submit(frame)?)` + /// never returns. Keep the futures in a queue, and call + /// [`Encoder::flush`] at the end of the stream before draining it. + /// + /// Three more things change. [`Encoder::invalidate_reference_frames`] + /// makes the next frame an IDR. A DMA-BUF is copied at submit and not + /// read directly. And a frame written into an + /// [`InputSlot`] cannot ask for a release sync_file, + /// because the encode that reads it may come later than the call. + /// + /// [`Encoder::new`] returns an error if the device cannot encode + /// B-frames for the profile; see + /// [`VideoContext::supports_b_frames`](crate::VideoContext::supports_b_frames). + pub fn with_b_frames(mut self, count: u32) -> Self { + self.b_frames = count; + self + } + + /// Choose whether B-frames are references, see [`BFrameReferences`]. + pub fn with_b_frame_references(mut self, references: BFrameReferences) -> Self { + self.b_frame_references = references; + self + } + /// Set the VBV/HRD virtual buffer size in milliseconds. /// Smaller values produce more uniform frame sizes at the cost of /// quality variation during scene changes. @@ -590,9 +635,27 @@ impl EncodeConfig { self.encoder_tuning_mode = mode; self } + + /// Set the driver's quality level. 0 (the default) is the fastest. A + /// higher level searches more: the stream is smaller at the same quality, + /// and each frame takes longer to encode. No level adds frames of delay. + /// + /// The highest level is one less than what + /// [`VideoContext::encode_quality_levels`](crate::VideoContext::encode_quality_levels) + /// returns, and it differs between devices and codecs. [`Encoder::new`] + /// returns an error for a level the device does not have. + /// + /// On an NVIDIA RTX 5080 (driver 615.78.08) at 1080p, AV1 at level 6 + /// needs about 5% fewer bits than at level 0, and each frame takes about + /// 1.9 ms more. + pub fn with_quality_level(mut self, level: u32) -> Self { + self.quality_level = level; + self + } } pub use api::Encoder; +pub use bframes::{BFrameReferences, BFrameSupport, MAX_B_FRAMES}; pub use frame::{Frame, InputSlot}; pub use pipeline::EncodeFuture; pub use source::{Conversion, ConversionPreference, RgbSource, Source}; @@ -619,10 +682,14 @@ pub struct EncodedPacket { pub data: Vec, /// Frame type. pub frame_type: FrameType, - /// Presentation timestamp. + /// Presentation timestamp: the number of the frame in the order it was + /// submitted. pub pts: u64, - /// Decode timestamp. - pub dts: u64, + /// Decode timestamp. Equal to `pts` without B-frames, and for AV1, + /// whose packets are in display order. With B-frames in H.264 and H.265 + /// it counts the packets, and starts below zero so that it is never + /// after `pts`. + pub dts: i64, /// Stats about the packet. pub stats: EncodedPacketStats, } @@ -777,6 +844,14 @@ mod tests { assert_eq!(config.gop_size, 30); assert_eq!(config.frame_rate.numerator, 30); assert_eq!(config.frame_rate.denominator, 1); + // The fastest level, for the lowest latency. + assert_eq!(config.quality_level, 0); + } + + #[test] + fn test_with_quality_level() { + let config = EncodeConfig::av1(1920, 1080).with_quality_level(6); + assert_eq!(config.quality_level, 6); } #[test] diff --git a/src/encoder/pipeline.rs b/src/encoder/pipeline.rs index c44b41e0..b5d8a7e2 100644 --- a/src/encoder/pipeline.rs +++ b/src/encoder/pipeline.rs @@ -1,19 +1,22 @@ //! Asynchronous (push) encode pipelining, shared by all codecs. //! -//! Each in-flight frame owns an [`EncodeSlot`] (its own input image, bitstream -//! buffer, encode command buffer, fence and query pool). [`EncodePipeline`] -//! rotates through the slots so that the CPU can record and submit frame N+1 -//! while the GPU is still encoding frame N, instead of blocking on a fence after -//! every frame. +//! Each frame owns an [`EncodeSlot`] (its own input image, bitstream buffer, +//! encode command buffer, fence and query pool) from the moment it is submitted +//! until its bitstream is read back. [`EncodePipeline`] hands the slots out +//! oldest-free first, so that the CPU can record and submit frame N+1 while the +//! GPU is still encoding frame N, instead of blocking on a fence after every +//! frame. A frame that is encoded later than it was submitted (a B-frame) keeps +//! its slot, and with it its pixels, until then. //! //! Bitstream readback is performed off the calling thread: a single background //! *completion thread* waits on each submission's fence, copies the bitstream //! out, and resolves that frame's [`EncodeFuture`] the moment the GPU signals — //! rather than deferring readback until the slot is reused. This delivers each //! packet at roughly the GPU encode time instead of one or two `encode()` calls -//! later. Each `encode()` returns its own future (backed by a oneshot channel), -//! so packet delivery is paired with its submission by construction rather than -//! by a shared channel and a separate ordering convention. +//! later. Each `submit()` returns its own future (backed by a oneshot channel). +//! The futures resolve in the order they were made: the k-th future gets the +//! k-th packet, whichever frame that packet holds. The senders wait in a queue +//! until an encode takes the oldest one. //! //! The DPB images and video session are shared across slots, so encode //! submissions must still run in DPB order on the GPU. That ordering is enforced @@ -22,13 +25,13 @@ //! the timeline; the completion thread only waits on fences and reads bitstream //! buffers, so the two never race on the same Vulkan object. //! -//! Slot reuse is coordinated with a per-slot "busy" flag -//! ([`SlotSync`](crate::video::SlotSync)): a slot -//! is busy from submit until the completion thread has finished reading its -//! bitstream. The calling thread waits for the current slot to be free before -//! converting into / recording over it, which also covers the write-after-read -//! hazard on the shared input image. +//! Slot reuse is coordinated with a queue of free slots (`FreeSlots`): a +//! slot leaves it when a frame takes it and comes back once the completion +//! thread has finished reading its bitstream. The calling thread waits for a +//! free slot before converting into / recording over it, which also covers the +//! write-after-read hazard on the slot's input image. +use std::collections::VecDeque; use std::future::Future; use std::pin::Pin; use std::sync::Arc; @@ -48,17 +51,15 @@ use crate::encoder::resources::{ use crate::encoder::{BitDepth, ChromaFormat, EncodedPacket, FrameType}; use crate::error::{PixelForgeError, Result}; use crate::sync::TimelinePoint; -use crate::video::{SlotSync, TimelineChain}; +use crate::video::TimelineChain; use crate::vulkan::VideoContext; -/// A handle to the packet a single `EncodePipeline::submit_current` will -/// eventually produce. +/// A handle to one packet of the stream. /// /// Returned by `Encoder::submit`, one per submitted frame. Awaiting it yields -/// that frame's [`EncodedPacket`] once the GPU finishes encoding and the -/// completion thread reads the bitstream back. Futures resolve in submission -/// order (one submission → one readback → one packet) as long as you await them -/// in the order `encode` returned them. +/// an [`EncodedPacket`] once the GPU finishes encoding and the completion +/// thread reads the bitstream back. The futures resolve in the order `submit` +/// returned them, and the k-th one yields the k-th packet of the stream. /// /// Dropping the future before it resolves is harmless: the completion thread /// still reads the slot back (its send simply finds no receiver) and frees it. @@ -103,11 +104,12 @@ impl Future for EncodeFuture { match Pin::new(&mut self.rx).poll(cx) { Poll::Ready(Ok(result)) => Poll::Ready(result.map(|mut packet| { packet.pts += offset; - packet.dts += offset; + packet.dts += offset as i64; packet })), // The sender was dropped without sending. The completion thread only - // drops a sender after sending, so this indicates encoder teardown. + // drops a sender after sending, so the encoder was torn down, or an + // encode failed before this frame's. Poll::Ready(Err(oneshot::Canceled)) => Poll::Ready(Err(PixelForgeError::Cancelled)), Poll::Pending => Poll::Pending, } @@ -126,7 +128,7 @@ pub(crate) const ENCODE_PIPELINE_DEPTH: usize = 2; pub(crate) struct SlotPacketMetadata { pub frame_type: FrameType, pub pts: u64, - pub dts: u64, + pub dts: i64, /// Codec header (SPS/PPS, VPS/SPS/PPS, or AV1 sequence header) to prepend. /// `Some` only for frames that carry one (e.g. the IDR/key frame). pub header: Option>, @@ -144,8 +146,9 @@ struct SendPtr(*const u8); // until the slot is marked free again (after this read completes). unsafe impl Send for SendPtr {} -/// A submission handed from the calling thread to the completion thread. -struct WorkItem { +/// An encode handed from the calling thread to the completion thread, to +/// read its bitstream back. +struct Readback { slot_index: usize, fence: vk::Fence, query_pool: vk::QueryPool, @@ -158,11 +161,94 @@ struct WorkItem { timestamp_valid_bits: u32, timestamp_query_pool: vk::QueryPool, submit_time: std::time::Instant, - /// Resolves this frame's [`EncodeFuture`] once the bitstream is read back. - result_tx: oneshot::Sender>, } -/// All per-frame resources that must be private to a single in-flight encode. +/// Where an encode's bitstream goes. +pub(crate) enum Delivery { + /// Into a packet of its own, which resolves the oldest waiting future. + Packet, + /// In front of the next packet, in the same temporal unit: an AV1 frame + /// that is not shown. It resolves no future. + Carry, +} + +/// What the completion thread does with an encode's bitstream. +enum Deliver { + /// Resolves a frame's [`EncodeFuture`] once the bitstream is read back. + Packet(oneshot::Sender>), + Carry, +} + +/// Work for the completion thread, handled in the order it was sent. +enum WorkItem { + /// An encode on the GPU. + Encoded(Readback, Deliver), + /// A packet made on the CPU, with no encode and no slot: an AV1 + /// `show_existing_frame` header. + Synthetic { + packet: EncodedPacket, + result_tx: oneshot::Sender>, + }, +} + +/// The slots no frame owns, oldest first. +/// +/// The calling thread takes from the front and the completion thread puts +/// back at the end, so the front does not change under the calling thread +/// once there is one. +struct FreeSlots { + queue: std::sync::Mutex>, + cv: std::sync::Condvar, + slot_count: usize, +} + +impl FreeSlots { + fn new(slot_count: usize) -> Self { + Self { + queue: std::sync::Mutex::new((0..slot_count).collect()), + cv: std::sync::Condvar::new(), + slot_count, + } + } + + /// Wait for a free slot and return the oldest, without taking it. + fn wait_front(&self) -> usize { + let queue = self.queue.lock().unwrap_or_else(|e| e.into_inner()); + let queue = self + .cv + .wait_while(queue, |q| q.is_empty()) + .unwrap_or_else(|e| e.into_inner()); + queue[0] + } + + /// Take the oldest free slot, waiting for one. + fn take_front(&self) -> usize { + let queue = self.queue.lock().unwrap_or_else(|e| e.into_inner()); + let mut queue = self + .cv + .wait_while(queue, |q| q.is_empty()) + .unwrap_or_else(|e| e.into_inner()); + queue.pop_front().expect("waited for a free slot") + } + + fn put_back(&self, slot: usize) { + let mut queue = self.queue.lock().unwrap_or_else(|e| e.into_inner()); + queue.push_back(slot); + self.cv.notify_all(); + } + + /// Wait until every slot is free. Only returns if no frame holds a slot + /// without an encode in flight. + fn wait_all(&self) { + let queue = self.queue.lock().unwrap_or_else(|e| e.into_inner()); + let _queue = self + .cv + .wait_while(queue, |q| q.len() < self.slot_count) + .unwrap_or_else(|e| e.into_inner()); + } +} + +/// All per-frame resources that must be private to a single frame. pub(crate) struct EncodeSlot { pub input_image: vk::Image, pub input_image_memory: vk::DeviceMemory, @@ -204,21 +290,27 @@ pub(crate) struct PipelineConfig<'a> { /// input images, bitstream buffers and feedback query pools. pub profile_info: &'a vk::VideoProfileInfoKHR<'a>, pub command_pool: vk::CommandPool, + /// How many frames can wait for their encode, on top of + /// [`ENCODE_PIPELINE_DEPTH`]: the B-frames between two reference frames. + pub held_frames: usize, /// Transfer command buffer/fence reused to zero-initialize each input image. pub upload_command_buffer: vk::CommandBuffer, pub upload_fence: vk::Fence, } -/// Rotating set of [`EncodeSlot`]s plus the timeline semaphore that orders their -/// encode submissions and the completion thread that reads bitstreams back. +/// The [`EncodeSlot`]s, plus the timeline semaphore that orders their encode +/// submissions and the completion thread that reads bitstreams back. pub(crate) struct EncodePipeline { slots: Vec, + /// The slot of the frame being encoded. current_slot: usize, /// Orders encode submissions that share DPB state. timeline: TimelineChain, - /// Per-slot busy flags shared with the completion thread. - slot_sync: Arc, + /// The free slots, shared with the completion thread. + free_slots: Arc, + /// The senders of the futures no packet was assigned to yet, oldest first. + senders: VecDeque>>, /// Sends submitted work to the completion thread. Dropped on shutdown to end /// the thread. work_tx: Option>, @@ -227,14 +319,15 @@ pub(crate) struct EncodePipeline { } impl EncodePipeline { - /// Allocate the timeline semaphore, `ENCODE_PIPELINE_DEPTH` slots and spawn - /// the bitstream-readback completion thread. + /// Allocate the timeline semaphore and the slots, and spawn the + /// bitstream-readback completion thread. pub(crate) fn new(config: &PipelineConfig) -> Result { let context = config.context; let device = context.device(); + let slot_count = ENCODE_PIPELINE_DEPTH + config.held_frames; let command_buffers = - allocate_command_buffers(context, config.command_pool, ENCODE_PIPELINE_DEPTH as u32)?; + allocate_command_buffers(context, config.command_pool, slot_count as u32)?; // Timestamp queries are only legal on a queue family with non-zero // `timestampValidBits` (VUID-vkCmdWriteTimestamp-timestampValidBits-00829). @@ -253,7 +346,7 @@ impl EncodePipeline { ); } - let mut slots = Vec::with_capacity(ENCODE_PIPELINE_DEPTH); + let mut slots = Vec::with_capacity(slot_count); for &encode_command_buffer in &command_buffers { let (input_image, input_image_memory, input_image_view) = create_image( context, @@ -325,18 +418,18 @@ impl EncodePipeline { }); } - let slot_sync = Arc::new(SlotSync::new(slots.len())); + let free_slots = Arc::new(FreeSlots::new(slots.len())); let (work_tx, work_rx) = std::sync::mpsc::channel::(); // The completion thread only needs a handle to the device; the Vulkan // device handle is internally shared and safe to use from this thread // for fence waits, query reads and host-coherent buffer reads. let thread_device = device.clone(); - let thread_sync = slot_sync.clone(); + let thread_free = free_slots.clone(); let completion_thread = std::thread::Builder::new() .name("pixelforge-encode-readback".to_string()) .spawn(move || { - run_completion_thread(thread_device, work_rx, thread_sync); + run_completion_thread(thread_device, work_rx, thread_free); }) .map_err(|source| PixelForgeError::Io { what: "spawning the readback thread", @@ -347,38 +440,90 @@ impl EncodePipeline { slots, current_slot: 0, timeline: TimelineChain::new(context)?, - slot_sync, + free_slots, + senders: VecDeque::new(), work_tx: Some(work_tx), completion_thread: Some(completion_thread), }) } - /// The slot the next frame will be encoded into. + /// The slot of the frame being encoded, see [`Self::begin_encode`]. pub(crate) fn current(&self) -> &EncodeSlot { &self.slots[self.current_slot] } - pub(crate) fn current_mut(&mut self) -> &mut EncodeSlot { - &mut self.slots[self.current_slot] + /// The slot at `index`. + pub(crate) fn slot(&self, index: usize) -> &EncodeSlot { + &self.slots[index] + } + + pub(crate) fn slot_mut(&mut self, index: usize) -> &mut EncodeSlot { + &mut self.slots[index] } - /// Return the current slot's input image, first waiting until the slot is - /// free so it is safe to use as a convert/upload target (write-after-read on - /// the shared input image). + pub(crate) fn slot_count(&self) -> usize { + self.slots.len() + } + + /// The slot the next submitted frame takes, first waiting until there is + /// a free one. A free slot's last encode was read back, so its input image + /// is safe to use as a convert/upload target and its command buffers and + /// semaphores are free. + pub(crate) fn next_index(&self) -> usize { + self.free_slots.wait_front() + } + + /// The input image of the slot the next submitted frame takes, see + /// [`Self::next_index`]. pub(crate) fn input_image(&self) -> vk::Image { - self.slot_sync.wait_free(self.current_slot); - self.slots[self.current_slot].input_image + self.slots[self.next_index()].input_image + } + + /// Give the slot [`Self::next_index`] names to a frame. The frame owns it + /// until its encode was read back, or until [`Self::release`]. + pub(crate) fn take_slot(&mut self) -> usize { + self.free_slots.take_front() + } + + /// Give back the slot of a frame that will not be encoded. + pub(crate) fn release(&mut self, slot: usize) { + self.slots[slot].pending_metadata = None; + self.free_slots.put_back(slot); + } + + /// Make the future of the next packet. Its sender waits until an encode + /// takes it. + pub(crate) fn new_future(&mut self) -> EncodeFuture { + let (result_tx, result_rx) = oneshot::channel::>(); + self.senders.push_back(result_tx); + EncodeFuture { + rx: result_rx, + // A wait for 0 is satisfied at once: nothing to release yet. + released: TimelinePoint::new(self.timeline.semaphore(), 0), + pts_offset: 0, + #[cfg(feature = "dmabuf")] + release_sync_file: None, + } } - /// Wait until the current slot is free to record over and submit. - pub(crate) fn wait_current_free(&self) { - self.slot_sync.wait_free(self.current_slot); + /// Drop the senders of every future no packet was assigned to yet, so + /// those futures resolve with an error: an encode failed, and the frames + /// after it will not be encoded. + pub(crate) fn drop_waiting_futures(&mut self) { + self.senders.clear(); } /// Wait until every in-flight submission has been read back. Used before - /// mutating shared session state and at teardown. + /// mutating shared session state and at teardown. No frame may wait for + /// its encode. pub(crate) fn wait_all_free(&self) { - self.slot_sync.wait_all_free(); + self.free_slots.wait_all(); + } + + /// Encode into `slot` from here on: the slot of the frame whose turn it + /// is. + pub(crate) fn begin_encode(&mut self, slot: usize) { + self.current_slot = slot; } /// Record the metadata for the packet that the current slot will produce. @@ -390,9 +535,8 @@ impl EncodePipeline { /// Submit the current slot's recorded command buffer without waiting, and /// hand the slot to the completion thread for bitstream readback. /// - /// Chains onto the timeline semaphore so the GPU keeps encodes in DPB order, - /// and marks the slot busy until the completion thread reads it back. Returns - /// the [`EncodeFuture`] that resolves with this frame's packet. + /// Chains onto the timeline semaphore so the GPU keeps encodes in DPB + /// order. Returns the point the encode signals. pub(crate) fn submit_current( &mut self, device: &ash::Device, @@ -400,24 +544,20 @@ impl EncodePipeline { encode_queue: vk::Queue, external_waits: &[vk::SemaphoreSubmitInfo<'static>], extra_signals: &[vk::SemaphoreSubmitInfo<'static>], - ) -> Result { + delivery: Delivery, + ) -> Result { let wait = self.timeline.wait(); let signal = self.timeline.pending_signal(); let slot_index = self.current_slot; + if matches!(delivery, Delivery::Packet) && self.senders.is_empty() { + return Err(PixelForgeError::InvalidInput( + "submit_current called with no future waiting for the packet".to_string(), + )); + } // Capture the Copy handles + metadata, releasing the slot borrow before // touching the cross-thread channels. - let ( - command_buffer, - fence, - query_pool, - bitstream_ptr, - bitstream_buffer_size, - timestamp_period, - timestamp_valid_bits, - timestamp_query_pool, - metadata, - ) = { + let (command_buffer, readback) = { let slot = &mut self.slots[slot_index]; let metadata = slot.pending_metadata.take().ok_or_else(|| { PixelForgeError::InvalidInput( @@ -426,14 +566,20 @@ impl EncodePipeline { })?; ( slot.encode_command_buffer, - slot.encode_fence, - slot.query_pool, - slot.bitstream_buffer_ptr as *const u8, - slot.bitstream_buffer_size, - slot.timestamp_period, - slot.timestamp_valid_bits, - slot.timestamp_query_pool, - metadata, + Readback { + slot_index, + fence: slot.encode_fence, + query_pool: slot.query_pool, + bitstream_ptr: SendPtr(slot.bitstream_buffer_ptr as *const u8), + bitstream_buffer_size: slot.bitstream_buffer_size, + timestamp_period: slot.timestamp_period, + timestamp_valid_bits: slot.timestamp_valid_bits, + timestamp_query_pool: slot.timestamp_query_pool, + // Before the submit, not after: the driver can block in + // the submit while the GPU already runs the work. + submit_time: std::time::Instant::now(), + metadata, + }, ) }; @@ -460,15 +606,12 @@ impl EncodePipeline { ) .chain(extra_signals.iter().copied()) .collect(); - // Before the submit, not after: the driver can block in the submit - // while the GPU already runs the work. - let submit_time = std::time::Instant::now(); unsafe { submit_encode_only( device, sync2, command_buffer, - fence, + readback.fence, encode_queue, &waits, &signals, @@ -477,47 +620,35 @@ impl EncodePipeline { self.timeline.commit(); - // Mark busy *before* handing the work off, so the completion thread can - // never clear the flag before it is set. - self.slot_sync.set_busy(slot_index); - - let (result_tx, result_rx) = oneshot::channel::>(); - let work = WorkItem { - slot_index, - fence, - query_pool, - bitstream_ptr: SendPtr(bitstream_ptr), - bitstream_buffer_size, - timestamp_period, - timestamp_valid_bits, - timestamp_query_pool, - submit_time, - metadata, - result_tx, + let deliver = match delivery { + Delivery::Packet => Deliver::Packet( + self.senders + .pop_front() + .expect("checked before the submission"), + ), + Delivery::Carry => Deliver::Carry, }; if let Some(tx) = &self.work_tx { // The receiver only disconnects during shutdown, after the queue is // idle; a failed send there is benign. - let _ = tx.send(work); + let _ = tx.send(WorkItem::Encoded(readback, deliver)); } - Ok(EncodeFuture { - rx: result_rx, - released: TimelinePoint::new(signal.0, signal.1), - pts_offset: 0, - #[cfg(feature = "dmabuf")] - release_sync_file: None, - }) + Ok(TimelinePoint::new(signal.0, signal.1)) } - /// The slot at `index`. - pub(crate) fn slot(&self, index: usize) -> &EncodeSlot { - &self.slots[index] - } - - /// Index of the slot the next frame will be encoded into. - pub(crate) fn current_index(&self) -> usize { - self.current_slot + /// Resolve the oldest waiting future with a packet made on the CPU, after + /// every packet submitted before it. + pub(crate) fn submit_synthetic(&mut self, packet: EncodedPacket) -> Result<()> { + let result_tx = self.senders.pop_front().ok_or_else(|| { + PixelForgeError::InvalidInput( + "submit_synthetic called with no future waiting for the packet".to_string(), + ) + })?; + if let Some(tx) = &self.work_tx { + let _ = tx.send(WorkItem::Synthetic { packet, result_tx }); + } + Ok(()) } /// The timeline the encodes signal. Retired to the context rather than @@ -526,19 +657,14 @@ impl EncodePipeline { self.timeline.semaphore() } - /// Advance to the next slot after a frame has been submitted. - pub(crate) fn advance(&mut self) { - self.current_slot = (self.current_slot + 1) % self.slots.len(); - } - /// Barrier: wait for every in-flight frame to be read back. /// /// The completion thread resolves each frame's [`EncodeFuture`] before it - /// marks the slot free, so once every slot is free every outstanding future - /// has already been resolved. Callers await the futures `encode` returned to - /// obtain the packets themselves. + /// frees the slot, so once every slot is free every outstanding future + /// has already been resolved. Callers await the futures `submit` returned + /// to obtain the packets themselves. pub(crate) fn flush(&mut self) { - self.slot_sync.wait_all_free(); + self.free_slots.wait_all(); } /// Stop the completion thread and wait for it to finish any in-flight @@ -588,17 +714,34 @@ impl EncodePipeline { } /// Completion-thread body: wait on each submission's fence, copy its bitstream -/// out, resolve the frame's future, then mark the slot free. +/// out, resolve the oldest waiting future, then free the slot. fn run_completion_thread( device: ash::Device, work_rx: Receiver, - slot_sync: Arc, + free_slots: Arc, ) { + // The bitstream of the encodes that go in front of the next packet. + let mut carried: Option>> = None; for work in work_rx { + let (work, deliver) = match work { + WorkItem::Encoded(work, deliver) => (work, deliver), + WorkItem::Synthetic { packet, result_tx } => { + let _ = result_tx.send(Ok(packet)); + continue; + } + }; // Start the packet from any codec header, then read the encoded bitstream // straight onto it — a single copy out of the mapped buffer. let mut data = work.metadata.header.unwrap_or_default(); - let result = unsafe { + let mut result = Ok(()); + if matches!(deliver, Deliver::Packet(_)) { + match carried.take() { + Some(Ok(bytes)) => data.extend_from_slice(&bytes), + Some(Err(e)) => result = Err(e), + None => {} + } + } + let read = unsafe { wait_and_read_bitstream( &device, work.fence, @@ -608,8 +751,25 @@ fn run_completion_thread( &mut data, ) }; + let result = result.and(read); let bitstream_ready_time = std::time::Instant::now(); + let result_tx = match deliver { + Deliver::Packet(result_tx) => result_tx, + Deliver::Carry => { + carried = Some(match (carried.take(), result) { + (Some(Ok(mut bytes)), Ok(())) => { + bytes.extend_from_slice(&data); + Ok(bytes) + } + (Some(Err(e)), _) | (_, Err(e)) => Err(e), + (None, Ok(())) => Ok(data), + }); + free_slots.put_back(work.slot_index); + continue; + } + }; + let stats = super::EncodedPacketStats { gpu_time_ns: unsafe { query_timestamp_diff( @@ -638,7 +798,7 @@ fn run_completion_thread( // once all slots are observed free, every packet has already been // delivered, which `flush` relies on for completeness. A dropped // receiver (future cancelled) makes the send a no-op, which is fine. - let _ = work.result_tx.send(packet); - slot_sync.set_free(work.slot_index); + let _ = result_tx.send(packet); + free_slots.put_back(work.slot_index); } } diff --git a/src/encoder/references.rs b/src/encoder/references.rs new file mode 100644 index 00000000..17189b5d --- /dev/null +++ b/src/encoder/references.rs @@ -0,0 +1,237 @@ +//! The references an encoder holds in its DPB, and the rules for which ones +//! it keeps: the same for each codec. +//! +//! The window counts the IDR and P-frames (level 0). The B-frames of a +//! pyramid that are references are held on top of it, until a later +//! reference frame of their level or of a level above makes them of no more +//! use (see [`obsolete_b_references`]). + +use super::gop::obsolete_b_references; + +/// One reference frame in the DPB. +#[derive(Clone, Copy, Debug)] +pub(crate) struct Reference { + pub dpb_slot: u8, + /// The POC: twice the frames shown since the last IDR. + pub poc: i32, + /// Display order (`pts`) since the encoder was made. + pub display_order: u64, + /// Encode order since the encoder was made. + pub encode_index: u64, + /// 0 for an IDR or P-frame, and the pyramid level of a B-frame. + pub level: u8, + /// What only one codec needs to know about the reference. + pub codec: T, +} + +/// The references held, newest first in encode order. +pub(crate) struct ReferenceSet { + references: Vec>, + /// How many IDR and P-frames stay. + window: usize, +} + +impl ReferenceSet { + pub fn new(window: u32) -> Self { + Self { + references: Vec::new(), + window: window.max(1) as usize, + } + } + + /// The references, newest first in encode order. + pub fn as_slice(&self) -> &[Reference] { + &self.references + } + + pub fn is_empty(&self) -> bool { + self.references.is_empty() + } + + pub fn len(&self) -> usize { + self.references.len() + } + + /// The newest `count` references, in encode order. + pub fn newest(&self, count: usize) -> &[Reference] { + &self.references[..self.references.len().min(count)] + } + + /// Drop all references, for an IDR. + pub fn clear(&mut self) { + self.references.clear(); + } + + /// Before a reference frame of `level` is encoded: drop the B-frame + /// references it makes of no more use, and return them. + pub fn drop_obsolete(&mut self, level: u8) -> Vec> { + let levels: Vec = self.references.iter().map(|r| r.level).collect(); + obsolete_b_references(&levels, level) + .into_iter() + .rev() + .map(|index| self.references.remove(index)) + .collect() + } + + /// The reference that [`Self::insert`] drops for a new reference frame + /// of `level`, if any: the oldest IDR or P-frame, once the window is full. + pub fn window_victim(&self, level: u8) -> Option<&Reference> { + if level > 0 || self.references.iter().filter(|r| r.level == 0).count() < self.window { + return None; + } + self.references.iter().rev().find(|r| r.level == 0) + } + + /// Add the frame just encoded as the newest reference, and drop the + /// oldest IDR or P-frame if that makes the window too full. + pub fn insert(&mut self, reference: Reference) { + let victim = self + .window_victim(reference.level) + .map(|victim| victim.encode_index); + self.references.retain(|r| Some(r.encode_index) != victim); + self.references.insert(0, reference); + } + + /// A DPB slot that no reference holds, out of `slot_count`: `preferred` + /// if that one is free, or else the lowest free one. + pub fn free_slot(&self, slot_count: usize, preferred: Option) -> u8 { + let free = |slot: &u8| !self.references.iter().any(|r| r.dpb_slot == *slot); + preferred + .filter(|slot| (*slot as usize) < slot_count && free(slot)) + .or_else(|| (0..slot_count as u8).find(free)) + .expect("the DPB has a slot for the frame being encoded") + } + + /// The references shown before the frame with POC `poc`, and those + /// shown after it, each nearest first. + pub fn around(&self, poc: i32) -> (Vec>, Vec>) { + let mut before: Vec> = self + .references + .iter() + .filter(|r| r.poc < poc) + .copied() + .collect(); + let mut after: Vec> = self + .references + .iter() + .filter(|r| r.poc > poc) + .copied() + .collect(); + before.sort_by_key(|r| std::cmp::Reverse(r.poc)); + after.sort_by_key(|r| r.poc); + (before, after) + } + + /// Reference frame invalidation: drop the references a client that lost + /// the frame shown at `first_lost_display_order`, and every frame after + /// it, cannot decode. Returns them. + /// + /// A client loses packets in decode order. So what it lost begins with + /// the first frame encoded that is shown at or after the lost one, and + /// every reference encoded from there on is lost too. Without B-frames + /// that is the references shown at or after the lost frame. With + /// B-frames it can include a B-frame shown before it, which is the safe + /// side. + /// + /// A frame shown at or after the lost one that is not a reference was + /// encoded after a reference that is (the one after its group), and + /// that reference is held: the window keeps every P-frame encoded since. + pub fn invalidate(&mut self, first_lost_display_order: u64) -> Vec> { + let Some(first_lost) = self + .references + .iter() + .filter(|r| r.display_order >= first_lost_display_order) + .map(|r| r.encode_index) + .min() + else { + return Vec::new(); + }; + let (lost, kept) = self + .references + .drain(..) + .partition(|r| r.encode_index >= first_lost); + self.references = kept; + lost + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn reference(encode_index: u64, display_order: u64, level: u8) -> Reference<()> { + Reference { + dpb_slot: encode_index as u8, + poc: 2 * display_order as i32, + display_order, + encode_index, + level, + codec: (), + } + } + + fn shown(set: &ReferenceSet<()>) -> Vec { + set.as_slice().iter().map(|r| r.display_order).collect() + } + + #[test] + fn the_window_counts_p_frames_only() { + let mut set = ReferenceSet::new(2); + set.insert(reference(0, 0, 0)); + set.insert(reference(1, 4, 0)); + set.insert(reference(2, 2, 1)); + assert_eq!(shown(&set), [2, 4, 0]); + assert!(set.window_victim(1).is_none()); + assert_eq!(set.window_victim(0).unwrap().display_order, 0); + + assert_eq!(set.drop_obsolete(0).len(), 1); + set.insert(reference(5, 8, 0)); + assert_eq!(shown(&set), [8, 4]); + } + + #[test] + fn free_slot_prefers_the_slot_asked_for() { + let mut set = ReferenceSet::new(4); + set.insert(reference(0, 0, 0)); + set.insert(reference(2, 1, 0)); + assert_eq!(set.free_slot(4, None), 1); + assert_eq!(set.free_slot(4, Some(3)), 3); + assert_eq!(set.free_slot(4, Some(2)), 1); + } + + #[test] + fn around_sorts_nearest_first() { + let mut set = ReferenceSet::new(4); + for (e, d) in [(0, 0), (1, 8), (2, 4)] { + set.insert(reference(e, d, 0)); + } + let (before, after) = set.around(2 * 5); + let d = |list: Vec>| list.iter().map(|r| r.display_order).collect::>(); + assert_eq!(d(before), [4, 0]); + assert_eq!(d(after), [8]); + } + + #[test] + fn invalidation_without_b_frames_drops_what_is_shown_from_the_loss() { + let mut set = ReferenceSet::new(4); + for i in 0..4 { + set.insert(reference(i, i, 0)); + } + let lost: Vec = set.invalidate(2).iter().map(|r| r.display_order).collect(); + assert_eq!(lost, [3, 2]); + assert_eq!(shown(&set), [1, 0]); + assert!(set.invalidate(7).is_empty()); + } + + #[test] + fn invalidation_with_b_frames_drops_what_is_encoded_from_the_loss() { + // I0 P4 B2 b1 b3, then b3 is lost: P4 is the first frame encoded + // that is shown at or after it, so P4 and B2 go, and I0 stays. + let mut set = ReferenceSet::new(2); + set.insert(reference(0, 0, 0)); + set.insert(reference(1, 4, 0)); + set.insert(reference(2, 2, 1)); + set.invalidate(3); + assert_eq!(shown(&set), [0]); + } +} diff --git a/src/encoder/rgb.rs b/src/encoder/rgb.rs index 377a5c9a..5c675da7 100644 --- a/src/encoder/rgb.rs +++ b/src/encoder/rgb.rs @@ -188,7 +188,7 @@ impl VideoContext { ChromaFormat::Yuv420, bit_depth, Some(&mut caps), - |profile| { + |profile, _| { query_supported_video_formats( self, profile, diff --git a/src/encoder/support.rs b/src/encoder/support.rs index daa103b9..18a9b81f 100644 --- a/src/encoder/support.rs +++ b/src/encoder/support.rs @@ -1,5 +1,6 @@ //! Device support queries for encode formats. +use super::bframes::{BFrameSupport, av1_support, h264_support, h265_support}; use super::codec::query_video_capabilities; use super::profile::{av1_profile, h264_profile_idc, h265_profile_idc}; use super::rgb::rgb_conversion_profile; @@ -9,6 +10,15 @@ use ash::vk; use ash::vk::TaggedStructure; use tracing::{debug, warn}; +/// What [`VideoContext::query_encode_profile`] learns about a profile, besides +/// the profile itself. +pub(crate) struct ProfileCaps { + /// What the encoder can do with B-frames, by the rules it checks itself. + pub b_frames: BFrameSupport, + /// `VkVideoEncodeCapabilitiesKHR::maxQualityLevels`. + pub max_quality_levels: u32, +} + /// Why [`VideoContext::query_encode_profile`] has no answer to give. pub(crate) enum ProfileQueryError { /// The encoder has no profile for this combination (H.265 4:2:2). @@ -35,7 +45,7 @@ impl VideoContext { if !self.supports_encode(codec) { return false; } - let result = self.query_encode_profile(codec, chroma_format, bit_depth, None, |_| ()); + let result = self.query_encode_profile(codec, chroma_format, bit_depth, None, |_, _| ()); report( &format!("{codec:?} encode {chroma_format:?} {bit_depth:?}"), result, @@ -43,10 +53,61 @@ impl VideoContext { .is_some() } + /// Check what the device can do with B-frames for `codec` in the given + /// chroma format and bit depth: what an [`Encoder`](crate::Encoder) with + /// [`EncodeConfig::with_b_frames`](crate::EncodeConfig::with_b_frames) + /// accepts. [`BFrameSupport::None`] also where the device cannot encode + /// the profile at all, and where the encoder has no B-frames for the + /// codec yet. + pub fn supports_b_frames( + &self, + codec: Codec, + chroma_format: ChromaFormat, + bit_depth: BitDepth, + ) -> BFrameSupport { + if !self.supports_encode(codec) { + return BFrameSupport::None; + } + let result = self.query_encode_profile(codec, chroma_format, bit_depth, None, |_, caps| { + caps.b_frames + }); + report( + &format!("{codec:?} B-frames {chroma_format:?} {bit_depth:?}"), + result, + ) + .unwrap_or(BFrameSupport::None) + } + + /// The number of quality levels the device has for `codec` in the given + /// chroma format and bit depth: what + /// [`EncodeConfig::with_quality_level`](crate::EncodeConfig::with_quality_level) + /// accepts is less than this. 0 where the device cannot encode the + /// profile. + pub fn encode_quality_levels( + &self, + codec: Codec, + chroma_format: ChromaFormat, + bit_depth: BitDepth, + ) -> u32 { + if !self.supports_encode(codec) { + return 0; + } + let result = self.query_encode_profile(codec, chroma_format, bit_depth, None, |_, caps| { + caps.max_quality_levels + }); + report( + &format!("{codec:?} quality levels {chroma_format:?} {bit_depth:?}"), + result, + ) + .unwrap_or(0) + } + /// Build the profile an encoder would request for `codec`, `chroma_format` /// and `bit_depth`, ask the driver for its capabilities, and on success /// hand the profile to `then` for any further query about it. /// + /// `then` also gets what else the query says about the profile. + /// /// With `rgb_caps`, the profile is the RGB input one an encoder built with /// [`Conversion::Encoder`](crate::Conversion::Encoder) /// requests, and the driver fills in what its conversion supports. @@ -60,7 +121,7 @@ impl VideoContext { chroma_format: ChromaFormat, bit_depth: BitDepth, mut rgb_caps: Option<&mut vk::VideoEncodeRgbConversionCapabilitiesVALVE<'static>>, - then: impl FnOnce(&vk::VideoProfileInfoKHR<'_>) -> R, + then: impl FnOnce(&vk::VideoProfileInfoKHR<'_>, ProfileCaps) -> R, ) -> Result { let chroma_subsampling: vk::VideoChromaSubsamplingFlagsKHR = chroma_format.into(); let component_bit_depth: vk::VideoComponentBitDepthFlagsKHR = bit_depth.into(); @@ -75,7 +136,7 @@ impl VideoContext { // Each arm keeps its profile and capability structs alive across the // calls that use them. macro_rules! query { - ($operation:expr, $profile:expr, $codec_caps:expr) => {{ + ($operation:expr, $profile:expr, $codec_caps:expr, $b_frames:expr) => {{ let mut profile = $profile; let mut profile_info = base.video_codec_operation($operation).push(&mut profile); if rgb_caps.is_some() { @@ -89,7 +150,15 @@ impl VideoContext { caps = caps.push(rgb_caps); } match query_video_capabilities(self, &profile_info, &mut caps) { - vk::Result::SUCCESS => Ok(then(&profile_info)), + vk::Result::SUCCESS => { + let max_active = caps.max_active_reference_pictures; + let vendor_id = self.device_properties().vendor_id; + let caps = ProfileCaps { + b_frames: $b_frames(vendor_id, &codec_caps, max_active), + max_quality_levels: encode_caps.max_quality_levels, + }; + Ok(then(&profile_info, caps)) + } err => Err(ProfileQueryError::Query(err)), } }}; @@ -100,7 +169,8 @@ impl VideoContext { vk::VideoCodecOperationFlagsKHR::ENCODE_H264, vk::VideoEncodeH264ProfileInfoKHR::default() .std_profile_idc(h264_profile_idc(chroma_format)), - vk::VideoEncodeH264CapabilitiesKHR::default() + vk::VideoEncodeH264CapabilitiesKHR::default(), + h264_support ), Codec::H265 => { let profile_idc = h265_profile_idc(chroma_format, bit_depth) @@ -108,13 +178,15 @@ impl VideoContext { query!( vk::VideoCodecOperationFlagsKHR::ENCODE_H265, vk::VideoEncodeH265ProfileInfoKHR::default().std_profile_idc(profile_idc), - vk::VideoEncodeH265CapabilitiesKHR::default() + vk::VideoEncodeH265CapabilitiesKHR::default(), + h265_support ) } Codec::AV1 => query!( vk::VideoCodecOperationFlagsKHR::ENCODE_AV1, vk::VideoEncodeAV1ProfileInfoKHR::default().std_profile(av1_profile(chroma_format)), - vk::VideoEncodeAV1CapabilitiesKHR::default() + vk::VideoEncodeAV1CapabilitiesKHR::default(), + av1_support ), } } diff --git a/src/image.rs b/src/image.rs index 1140d95d..a12f7317 100644 --- a/src/image.rs +++ b/src/image.rs @@ -1,4 +1,7 @@ -//! Image utilities for creating and uploading video frames. +//! Image utilities for moving video frames between the CPU and the GPU. +//! +//! [`YuvDownloader`] does the reverse of [`YuvUploader`]: it copies decoded +//! frames back to the CPU. //! //! This module provides [`YuvUploader`], a helper for uploading YUV data from the CPU //! to the GPU. The image it creates is a transfer-only staging image (not a Vulkan Video @@ -9,6 +12,7 @@ //! encoder's [`InputSlot`] (from //! [`Encoder::input_slot`](crate::encoder::Encoder::input_slot)). +use crate::decoder::DecodedFrame; use crate::encoder::{BitDepth, ChromaFormat, Encoder, Handoff, InputSlot, Source, SourceImage}; use crate::error::{PixelForgeError, Result}; use crate::vulkan::VideoContext; @@ -672,3 +676,329 @@ impl Drop for YuvUploader { } } } + +/// A decoded frame's samples in CPU memory, from [`YuvDownloader::download`]. +/// +/// Two planes, cropped to the visible size, with no padding between rows: +/// writing [`y`](Self::y) and then [`uv`](Self::uv) gives the frame as NV12 +/// (P010 for 10 bits, each sample a little-endian 16-bit word with the value +/// in its high 10 bits). +#[derive(Debug, Clone)] +pub struct DownloadedFrame { + /// The luma plane: `height` rows of `width` samples. + pub y: Vec, + /// The chroma plane: Cb and Cr samples interleaved, at the chroma + /// format's resolution. For 4:2:0, `ceil(height / 2)` rows of + /// `ceil(width / 2)` pairs. + pub uv: Vec, + /// The visible width in pixels. + pub width: u32, + /// The visible height in pixels. + pub height: u32, + /// The chroma format of the samples. + pub chroma_format: ChromaFormat, + /// The bit depth of the samples. + pub bit_depth: BitDepth, +} + +/// Copies decoded frames from the GPU into CPU memory. +/// +/// For when the samples are needed on the CPU: to write them to a file, to +/// compare them in a test, or to process them in software. Not for showing a +/// frame: a renderer samples [`DecodedFrame::image`] where it is, and a copy +/// to the CPU only adds latency and bandwidth. +/// +/// Each [`download`](Self::download) submits a copy on the transfer queue and +/// waits for it. The decoder submits to the same queue when it copies +/// pictures out, so call it from the thread that drives the decoder. +pub struct YuvDownloader { + context: VideoContext, + command_pool: vk::CommandPool, + command_buffer: vk::CommandBuffer, + fence: vk::Fence, + /// Staging buffer, its memory and its size, made larger when a frame + /// needs more. + staging: Option<(vk::Buffer, vk::DeviceMemory, usize)>, +} + +impl YuvDownloader { + /// A downloader for frames decoded on `context`. + pub fn new(context: &VideoContext) -> Result { + let device = context.device(); + let command_pool = unsafe { + device.create_command_pool( + &vk::CommandPoolCreateInfo::default() + .queue_family_index(context.transfer_queue_family()) + .flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER), + None, + ) + } + .map_err(PixelForgeError::vk("vkCreateCommandPool"))?; + let command_buffer = unsafe { + device.allocate_command_buffers( + &vk::CommandBufferAllocateInfo::default() + .command_pool(command_pool) + .level(vk::CommandBufferLevel::PRIMARY) + .command_buffer_count(1), + ) + } + .map_err(PixelForgeError::vk("vkAllocateCommandBuffers"))?[0]; + let fence = unsafe { device.create_fence(&vk::FenceCreateInfo::default(), None) } + .map_err(PixelForgeError::vk("vkCreateFence"))?; + Ok(Self { + context: context.clone(), + command_pool, + command_buffer, + fence, + staging: None, + }) + } + + /// Copy `frame` into CPU memory, and return its visible region. + pub fn download(&mut self, frame: &DecodedFrame) -> Result { + let bytes = match frame.bit_depth { + BitDepth::Eight => 1usize, + BitDepth::Ten => 2, + }; + // How many luma samples one chroma sample covers, across and down. + let (sub_x, sub_y) = match frame.chroma_format { + ChromaFormat::Yuv420 => (2, 2), + ChromaFormat::Yuv422 => (2, 1), + ChromaFormat::Yuv444 => (1, 1), + }; + // The whole coded image, so each plane is one contiguous copy. + let (coded_width, coded_height) = (frame.coded_width, frame.coded_height); + let (chroma_width, chroma_height) = (coded_width / sub_x, coded_height / sub_y); + let y_stride = coded_width as usize * bytes; + let y_size = y_stride * coded_height as usize; + let uv_stride = chroma_width as usize * 2 * bytes; + let uv_size = uv_stride * chroma_height as usize; + let (buffer, memory) = self.staging(y_size + uv_size)?; + + let plane = |aspect, offset: usize, width, height| { + vk::BufferImageCopy::default() + .buffer_offset(offset as u64) + .image_subresource(vk::ImageSubresourceLayers { + aspect_mask: aspect, + mip_level: 0, + base_array_layer: frame.array_layer, + layer_count: 1, + }) + .image_extent(vk::Extent3D { + width, + height, + depth: 1, + }) + }; + let regions = [ + plane(vk::ImageAspectFlags::PLANE_0, 0, coded_width, coded_height), + plane( + vk::ImageAspectFlags::PLANE_1, + y_size, + chroma_width, + chroma_height, + ), + ]; + + // GENERAL, where the decoder hands over its own image, is a valid + // source for a copy. Any other layout is an image of this frame's + // own, so moving it and back is safe. + let transition = frame.layout != vk::ImageLayout::GENERAL; + let barrier = |from, to, src_access, dst_access| { + vk::ImageMemoryBarrier::default() + .old_layout(from) + .new_layout(to) + .src_access_mask(src_access) + .dst_access_mask(dst_access) + .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .image(frame.image) + .subresource_range(vk::ImageSubresourceRange { + aspect_mask: vk::ImageAspectFlags::PLANE_0 | vk::ImageAspectFlags::PLANE_1, + base_mip_level: 0, + level_count: 1, + base_array_layer: frame.array_layer, + layer_count: 1, + }) + }; + let copy_layout = if transition { + vk::ImageLayout::TRANSFER_SRC_OPTIMAL + } else { + frame.layout + }; + + let device = self.context.device(); + let command_buffer = self.command_buffer; + unsafe { + device + .reset_command_buffer(command_buffer, vk::CommandBufferResetFlags::empty()) + .map_err(PixelForgeError::vk("vkResetCommandBuffer"))?; + device + .begin_command_buffer( + command_buffer, + &vk::CommandBufferBeginInfo::default() + .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT), + ) + .map_err(PixelForgeError::vk("vkBeginCommandBuffer"))?; + if transition { + device.cmd_pipeline_barrier( + command_buffer, + vk::PipelineStageFlags::TOP_OF_PIPE, + vk::PipelineStageFlags::TRANSFER, + vk::DependencyFlags::empty(), + &[], + &[], + &[barrier( + frame.layout, + copy_layout, + vk::AccessFlags::empty(), + vk::AccessFlags::TRANSFER_READ, + )], + ); + } + device.cmd_copy_image_to_buffer( + command_buffer, + frame.image, + copy_layout, + buffer, + ®ions, + ); + if transition { + device.cmd_pipeline_barrier( + command_buffer, + vk::PipelineStageFlags::TRANSFER, + vk::PipelineStageFlags::BOTTOM_OF_PIPE, + vk::DependencyFlags::empty(), + &[], + &[], + &[barrier( + copy_layout, + frame.layout, + vk::AccessFlags::TRANSFER_READ, + vk::AccessFlags::empty(), + )], + ); + } + device + .end_command_buffer(command_buffer) + .map_err(PixelForgeError::vk("vkEndCommandBuffer"))?; + let submit = + vk::SubmitInfo::default().command_buffers(std::slice::from_ref(&command_buffer)); + device + .queue_submit(self.context.transfer_queue(), &[submit], self.fence) + .map_err(PixelForgeError::vk("vkQueueSubmit"))?; + device + .wait_for_fences(&[self.fence], true, u64::MAX) + .map_err(PixelForgeError::vk("vkWaitForFences"))?; + device + .reset_fences(&[self.fence]) + .map_err(PixelForgeError::vk("vkResetFences"))?; + } + + // Crop to the visible region: the image has the coded size, padded + // up to the codec's block size. An odd size rounds the chroma up. + let visible_y = frame.width as usize * bytes; + let visible_uv = frame.width.div_ceil(sub_x) as usize * 2 * bytes; + let uv_rows = frame.height.div_ceil(sub_y) as usize; + let mut y = Vec::with_capacity(visible_y * frame.height as usize); + let mut uv = Vec::with_capacity(visible_uv * uv_rows); + unsafe { + // The whole allocation: a smaller range has to respect + // `nonCoherentAtomSize`. + let data = device + .map_memory(memory, 0, vk::WHOLE_SIZE, vk::MemoryMapFlags::empty()) + .map_err(PixelForgeError::vk("vkMapMemory"))? as *const u8; + for row in 0..frame.height as usize { + y.extend_from_slice(std::slice::from_raw_parts( + data.add(row * y_stride), + visible_y, + )); + } + for row in 0..uv_rows { + uv.extend_from_slice(std::slice::from_raw_parts( + data.add(y_size + row * uv_stride), + visible_uv, + )); + } + device.unmap_memory(memory); + } + Ok(DownloadedFrame { + y, + uv, + width: frame.width, + height: frame.height, + chroma_format: frame.chroma_format, + bit_depth: frame.bit_depth, + }) + } + + /// The staging buffer, with room for `size` bytes. + fn staging(&mut self, size: usize) -> Result<(vk::Buffer, vk::DeviceMemory)> { + if let Some((buffer, memory, capacity)) = self.staging + && capacity >= size + { + return Ok((buffer, memory)); + } + let device = self.context.device(); + unsafe { + // Each download waits for its copy, so the old buffer is free. + if let Some((buffer, memory, _)) = self.staging.take() { + device.destroy_buffer(buffer, None); + device.free_memory(memory, None); + } + let buffer = device + .create_buffer( + &vk::BufferCreateInfo::default() + .size(size as u64) + .usage(vk::BufferUsageFlags::TRANSFER_DST) + .sharing_mode(vk::SharingMode::EXCLUSIVE), + None, + ) + .map_err(PixelForgeError::vk("vkCreateBuffer"))?; + let requirements = device.get_buffer_memory_requirements(buffer); + let Some(memory_type) = self.context.find_memory_type( + requirements.memory_type_bits, + vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT, + ) else { + device.destroy_buffer(buffer, None); + return Err(PixelForgeError::NoSuitableDevice( + "no host-visible memory to download frames into".to_string(), + )); + }; + let memory = match device.allocate_memory( + &vk::MemoryAllocateInfo::default() + .allocation_size(requirements.size) + .memory_type_index(memory_type), + None, + ) { + Ok(memory) => memory, + Err(e) => { + device.destroy_buffer(buffer, None); + return Err(PixelForgeError::vulkan("vkAllocateMemory", e)); + } + }; + if let Err(e) = device.bind_buffer_memory(buffer, memory, 0) { + device.destroy_buffer(buffer, None); + device.free_memory(memory, None); + return Err(PixelForgeError::vulkan("vkBindBufferMemory", e)); + } + self.staging = Some((buffer, memory, requirements.size as usize)); + Ok((buffer, memory)) + } + } +} + +impl Drop for YuvDownloader { + fn drop(&mut self) { + let device = self.context.device(); + unsafe { + // Each download waits for its copy, so nothing is in flight. + if let Some((buffer, memory, _)) = self.staging.take() { + device.destroy_buffer(buffer, None); + device.free_memory(memory, None); + } + device.destroy_fence(self.fence, None); + device.destroy_command_pool(self.command_pool, None); + } + } +} diff --git a/src/lib.rs b/src/lib.rs index fec57dee..9dd42d9e 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -18,14 +18,15 @@ //! - **HDR support**: 10-bit encoding (P010, YUV444P10), PQ transfer function, BT.2020 color space. //! - **GPU-native API**: Encode directly from Vulkan images (`vk::Image`). //! - **Flexible configuration**: Rate control (CBR, VBR, CQP), quality levels, GOP settings. +//! - **B-frames**: for all three codecs, flat or as a pyramid, where the +//! device can encode them. Off by default, since they add latency. //! - **Multiple input formats**: BGRx, RGBx, BGRA, RGBA, ABGR2101010 (10-bit packed), RGBA16F (FP16). -//! - **Utility helpers**: [`YuvUploader`] for easy YUV data upload to GPU. +//! - **Utility helpers**: [`YuvUploader`] to upload YUV frames to the GPU, +//! and [`YuvDownloader`] to copy decoded frames back to the CPU. //! - **Optional DMA-BUF input**: import a producer's buffers, or allocate buffers //! for it, read with no copy where the driver allows, with sync_file fences //! (Linux only). //! -//! > **Note**: B-frame encoding is not yet implemented, so there is no option for it. -//! //! # Supported Codecs //! //! | Codec | Encode | Decode | @@ -175,8 +176,9 @@ //! A live frame reserves a DPB slot, so //! [`DecodeConfig::with_output_depth`](decoder::DecodeConfig::with_output_depth) //! bounds how many can be outstanding before the decoder starts copying -//! pictures out instead of handing over its own. Reading a frame back to the -//! CPU is the consumer's job; `examples/common` shows one way. +//! pictures out instead of handing over its own. To get a frame's samples on +//! the CPU (for a file, a test or software processing), copy it with +//! [`YuvDownloader`]. //! //! ## RGB input //! @@ -262,6 +264,66 @@ //! CPU, waits for that itself. //! - a `DmaBufImage`, a DMA-BUF (see below). //! +//! ## B-frames +//! +//! [`EncodeConfig::with_b_frames`] puts B-frames between two reference +//! frames. A B-frame predicts from the frame shown before it and from the one +//! shown after it, which costs fewer bits, and it costs latency: the encoder +//! has to wait for the later frame. [`VideoContext::supports_b_frames`] says +//! what the device can do for a codec, and [`Encoder::new`] returns an error +//! for more than that. +//! +//! ```no_run +//! # use std::collections::VecDeque; +//! # use pixelforge::{Encoder, Frame, SourceImage}; +//! # fn encode(encoder: &mut Encoder, frames: &[SourceImage]) -> Result<(), Box> { +//! let b_frames = encoder.config().b_frames as usize; +//! let mut pending = VecDeque::new(); +//! for frame in frames { +//! pending.push_back(encoder.submit(Frame::image(frame))?); +//! // At most `b_frames` frames wait for the frame after their group, so +//! // the oldest future is only sure to resolve once more are pending. +//! // Waiting for it sooner never returns. +//! while pending.len() > b_frames { +//! let packet = pollster::block_on(pending.pop_front().unwrap())?; +//! // Write `packet.data`. The packets come in decode order. +//! } +//! } +//! // Encode the frames that still wait, then take the rest. +//! encoder.flush()?; +//! for future in pending { +//! let packet = pollster::block_on(future)?; +//! } +//! # Ok(()) +//! # } +//! ``` +//! +//! What changes with B-frames: +//! +//! - The future of the k-th submitted frame resolves with the k-th packet of +//! the stream, in decode order. That is the packet of another frame, and +//! [`EncodedPacket::pts`] says which. +//! - For H.264 and H.265, [`EncodedPacket::dts`] counts in decode order and +//! starts below zero, so it is never after `pts`. An AV1 stream has no +//! reordering of its own: its packets are in display order, with +//! `dts == pts`, and the frame encoded ahead travels hidden in an earlier +//! packet. +//! - [`Encoder::request_idr`], [`Encoder::flush`], [`Encoder::repeat`] and +//! [`Encoder::reconfigure`] encode the frames that wait at once, as a +//! shorter group. +//! - [`Encoder::invalidate_reference_frames`] drops every reference encoded +//! from the first lost frame on, also B-frames shown before it, because +//! the client lost those packets too. If no reference survives, the oldest +//! frame that waits for its group becomes the IDR at once. +//! - With a constant QP, a B-frame gets a slightly higher QP than the +//! other frames, and one step more for each level of a pyramid. +//! - A DMA-BUF is copied when it is submitted and never read directly, so +//! the producer gets its buffer back at once. +//! +//! How much B-frames save depends on the driver. On NVIDIA (615.78.08) they +//! save bits with H.265 and AV1. With H.264 that driver predicts a B-frame +//! from one direction only, and the stream gets larger than without them. +//! //! ## Encoding on your own device //! //! An application that already renders with Vulkan can give pixelforge its own @@ -326,27 +388,27 @@ //! //! # Examples //! -//! Run the examples with: -//! //! ```text -//! # Query codec capabilities +//! # What this device can encode and decode //! cargo run --example query_capabilities //! -//! # Decode H.264 to raw YUV -//! cargo run --example decode -- input.264 output.yuv -//! -//! # Encode, choosing the codec (h264, h265 or av1) +//! # Encode frames from CPU memory (h264, h265 or av1) //! cargo run --example encode -- h265 //! -//! # Sample decoded frames through a ycbcr conversion (RGBA output) -//! cargo run --example sample_frame -- input.264 out.rgba +//! # Encode frames that the application draws on the GPU +//! cargo run --example encode_gpu -- h265 +//! +//! # Decode a stream to a raw YUV file +//! cargo run --example decode -- input.264 output.yuv //! -//! # Sample decoded frames through per-plane views (NV12 output) -//! cargo run --example sample_planes -- input.264 out.yuv +//! # Decode and sample the frames in a shader, with no copy (RGBA output) +//! cargo run --example sample_frame -- input.264 out.rgba //! ``` //! -//! Correctness checks that need a video device and ffmpeg are integration -//! tests, ignored by default: +//! The `tools` crate in the repository has command-line tools to check the +//! library, with more options: `cargo run -p pixelforge-tools --bin encode` +//! and `--bin decode`. Correctness checks that need a video device are +//! integration tests, ignored by default: //! //! ```text //! cargo test -- --ignored @@ -359,8 +421,7 @@ //! //! # TODO's //! -//! 1. [] H.265 and AV1 decoding. -//! 1. [] B-frames support (encode). +//! 1. [] Open GOPs. //! //! # Contributing //! @@ -399,16 +460,16 @@ pub use decoder::{ #[cfg(feature = "dmabuf")] pub use encoder::external::{DmaBuf, DmaBufFormat, DmaBufImage, DmaBufModifier, DmaBufPlane}; pub use encoder::{ - BitDepth, ChromaFormat, Codec, ColorDescription, Dimensions, EncodeConfig, EncodeContentHint, - EncodeFuture, EncodeUsageHint, EncodedPacket, Encoder, EncoderTuningMode, Frame, FrameRate, - FrameType, RateControl, + BFrameReferences, BFrameSupport, BitDepth, ChromaFormat, Codec, ColorDescription, Dimensions, + EncodeConfig, EncodeContentHint, EncodeFuture, EncodeUsageHint, EncodedPacket, Encoder, + EncoderTuningMode, Frame, FrameRate, FrameType, MAX_B_FRAMES, RateControl, }; pub use encoder::{ Conversion, ConversionPreference, Handoff, InputSlot, RgbSource, Source, SourceImage, StreamColor, }; pub use error::PixelForgeError; -pub use image::{YuvLayout, YuvUploader}; +pub use image::{DownloadedFrame, YuvDownloader, YuvLayout, YuvUploader}; pub use sync::TimelinePoint; pub use vulkan::VideoContextBuilder; diff --git a/tests/adopted.rs b/tests/adopted.rs index a9b063dd..02168eeb 100644 --- a/tests/adopted.rs +++ b/tests/adopted.rs @@ -12,15 +12,15 @@ use ash::vk; use ash::vk::TaggedStructure; #[allow(dead_code)] mod common; +use common::decode_stream; use common::source::create_src_image; -use common::{Readback, decode_stream}; use pixelforge::decoder::{DecodeConfig, Decoder}; use pixelforge::encoder::Codec; use pixelforge::vulkan::{DeviceQueue, DeviceRequirements, VideoContext, VideoContextBuilder}; use pixelforge::{ - ColorSpec, ConversionPreference, EncodeConfig, Encoder, Frame, Handoff, RateControl, RgbFormat, - RgbSource, Source, SourceImage, TimelinePoint, + BFrameSupport, BitDepth, ChromaFormat, ColorSpec, ConversionPreference, EncodeConfig, Encoder, + Frame, Handoff, RateControl, RgbFormat, RgbSource, Source, SourceImage, TimelinePoint, }; /// An instance and device the test owns, destroyed on drop after everything @@ -143,14 +143,14 @@ fn decode_all( context: &VideoContext, stream: &[u8], ) -> Result>, Box> { - let mut readback = Readback::new(context)?; + let mut readback = pixelforge::YuvDownloader::new(context)?; let mut decoder = Decoder::new( context.clone(), DecodeConfig::new(Codec::H264).with_byte_stream(), )?; let mut frames = Vec::new(); decode_stream(&mut decoder, stream, |frame| { - let data = readback.read(&frame)?; + let data = readback.download(&frame)?; frames.push([data.y, data.uv].concat()); Ok(()) })?; @@ -246,13 +246,22 @@ enum Wait { /// Encode `ENCODE_FRAMES` RGB frames as H.264 on `context`, the way a capture /// pipeline would: the encoder converts each one with the colour converter's /// shader, straight into its input image. -fn encode_all(context: &VideoContext, wait: Wait) -> Result, Box> { +/// +/// With `b_frames`, some frames are encoded later than they are submitted, +/// in an order of their own: the submissions and their waits have to hold +/// for that too. +fn encode_all( + context: &VideoContext, + wait: Wait, + b_frames: u32, +) -> Result, Box> { let device = context.device(); let mut encoder = Encoder::new( context.clone(), EncodeConfig::h264(ENCODE_WIDTH, ENCODE_HEIGHT) .with_rate_control(RateControl::Cqp { qp: 22 }) .with_frame_rate(30, 1) + .with_b_frames(b_frames) .with_source(Source::Rgb(RgbSource::new( RgbFormat::BGRA, ColorSpec::Srgb, @@ -314,6 +323,8 @@ fn encode_all(context: &VideoContext, wait: Wait) -> Result, Box Result, Box Vec { + let support = context.supports_b_frames(Codec::H264, ChromaFormat::Yuv420, BitDepth::Eight); + if support == BFrameSupport::None { + vec![0] + } else { + vec![0, 2] + } +} + /// Luma PSNR between two NV12 frames of the encode size. fn luma_psnr(a: &[u8], b: &[u8]) -> f64 { let n = (ENCODE_WIDTH * ENCODE_HEIGHT) as usize; @@ -350,7 +371,12 @@ fn encode_on_a_vulkan_1_1_device() -> Result<(), Box> { .require_encode(Codec::H264) .require_decode(Codec::H264) .build()?; - let expected = encode_all(&own, Wait::Cpu)?; + // Without B-frames, and with them where the device has them. + let b_frame_counts = b_frame_counts(&own); + let mut expected = Vec::new(); + for &b_frames in &b_frame_counts { + expected.push(encode_all(&own, Wait::Cpu, b_frames)?); + } let (entry, instance) = instance_1_1()?; let builder = VideoContextBuilder::new().require_encode(Codec::H264); @@ -371,7 +397,10 @@ fn encode_on_a_vulkan_1_1_device() -> Result<(), Box> { app.physical_device, app.device.clone(), )?; - let actual = encode_all(&context, Wait::Gpu)?; + let mut actual = Vec::new(); + for &b_frames in &b_frame_counts { + actual.push(encode_all(&context, Wait::Gpu, b_frames)?); + } drop(context); // Both streams decoded by the same decoder, so any difference is the @@ -380,25 +409,28 @@ fn encode_on_a_vulkan_1_1_device() -> Result<(), Box> { // below the floor here. The floor is not equality because not every // encoder is reproducible: RADV gives bit-identical streams, while ANV has // been seen as low as 47 dB between two runs of the same synchronous path. - let expected_frames = decode_all(&own, &expected)?; - let actual_frames = decode_all(&own, &actual)?; - assert_eq!(actual_frames.len(), ENCODE_FRAMES as usize, "frame count"); - assert_eq!(expected_frames.len(), actual_frames.len(), "frame count"); - let psnrs: Vec = actual_frames - .iter() - .zip(&expected_frames) - .map(|(a, e)| luma_psnr(a, e)) - .collect(); - println!( - "adopted vs own encode, luma PSNR per frame: {:?}; streams bit-identical: {}", - psnrs, - actual == expected - ); - for (i, psnr) in psnrs.iter().enumerate() { - assert!( - *psnr >= 40.0, - "frame {i}: {psnr:.2} dB against the own-context encode" + for ((expected, actual), b_frames) in expected.iter().zip(&actual).zip(&b_frame_counts) { + let expected_frames = decode_all(&own, expected)?; + let actual_frames = decode_all(&own, actual)?; + assert_eq!(actual_frames.len(), ENCODE_FRAMES as usize, "frame count"); + assert_eq!(expected_frames.len(), actual_frames.len(), "frame count"); + let psnrs: Vec = actual_frames + .iter() + .zip(&expected_frames) + .map(|(a, e)| luma_psnr(a, e)) + .collect(); + println!( + "adopted vs own encode, {b_frames} B-frames, luma PSNR per frame: {:?}; streams \ + bit-identical: {}", + psnrs, + actual == expected ); + for (i, psnr) in psnrs.iter().enumerate() { + assert!( + *psnr >= 40.0, + "{b_frames} B-frames, frame {i}: {psnr:.2} dB against the own-context encode" + ); + } } Ok(()) } @@ -448,7 +480,8 @@ fn encode_and_decode_on_one_adopted_device() -> Result<(), Box Result<(), Box> { + common::init_logging(); + let context = VideoContextBuilder::new() + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()) + .build()?; + for codec in [Codec::H264, Codec::H265, Codec::AV1] { + if !context.supports_encode(codec) { + continue; + } + let support = context.supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight); + eprintln!("{codec:?}: {support:?}"); + for (references, needed) in [ + (BFrameReferences::Flat, BFrameSupport::Flat), + (BFrameReferences::Pyramid, BFrameSupport::Pyramid), + ] { + let config = EncodeConfig::new(codec, 320, 240) + .with_b_frames(3) + .with_b_frame_references(references); + let created = Encoder::new(context.clone(), config); + assert_eq!( + created.is_ok(), + support >= needed, + "{codec:?} {references:?}: query says {support:?}, creation gives {:?}", + created.err() + ); + } + } + Ok(()) +} + +/// More than `MAX_B_FRAMES` is an error on any device, before the device is +/// asked. +#[test] +#[ignore = "requires a Vulkan Video device"] +fn too_many_b_frames_is_an_error() -> Result<(), Box> { + common::init_logging(); + let context = VideoContextBuilder::new() + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()) + .require_encode(Codec::H264) + .build()?; + let config = EncodeConfig::h264(320, 240).with_b_frames(MAX_B_FRAMES + 1); + assert!(matches!( + Encoder::new(context, config), + Err(PixelForgeError::InvalidInput(_)) + )); + Ok(()) +} + +/// An I420 frame with a pattern that moves with `index`. +fn pattern(index: usize) -> Vec { + let (w, h) = (WIDTH as usize, HEIGHT as usize); + let mut frame = vec![128u8; w * h * 3 / 2]; + for y in 0..h { + for x in 0..w { + frame[y * w + x] = (((x + 3 * index) / 8 + y / 8) * 16 % 256) as u8; + } + } + frame +} + +/// Take the packets that are sure to be ready: all but the last `keep`. +fn drain( + pending: &mut VecDeque, + keep: usize, + packets: &mut Vec, +) -> Result<(), Box> { + while pending.len() > keep { + packets.push(pollster::block_on(pending.pop_front().expect("non-empty"))?); + } + Ok(()) +} + +/// `request_idr`, `repeat`, reference frame invalidation and `flush` in the +/// middle of a group of B-frames: no frame is lost, the IDRs are where they +/// were asked for, and ffmpeg finds no error in the stream. +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn control_calls_close_the_group() -> Result<(), Box> { + common::init_logging(); + // With a bitrate too: the rate control is told the GOP has B-frames. + let cqp = RateControl::Cqp { qp: 26 }; + let cbr = RateControl::Cbr { bitrate: 1_000_000 }; + let flat = (2, BFrameReferences::Flat); + // A pyramid with two levels of references, each call cuts it short at + // another place. + let pyramid = (7, BFrameReferences::Pyramid); + for (codec, extension, rate_control, (b_frames, references)) in [ + (Codec::H264, "h264", cqp, flat), + (Codec::H264, "h264", cbr, flat), + (Codec::H264, "h264", cqp, pyramid), + (Codec::H265, "h265", cqp, flat), + (Codec::H265, "h265", cbr, flat), + (Codec::H265, "h265", cqp, pyramid), + (Codec::AV1, "obu", cqp, flat), + (Codec::AV1, "obu", cbr, flat), + (Codec::AV1, "obu", cbr, pyramid), + ] { + let context = VideoContextBuilder::new() + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()) + .build()?; + let needed = match references { + BFrameReferences::Flat => BFrameSupport::Flat, + BFrameReferences::Pyramid => BFrameSupport::Pyramid, + }; + if !context.supports_encode(codec) + || context.supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight) < needed + { + eprintln!("{codec:?}: no {references:?} B-frames on this device, nothing to check"); + continue; + } + let config = EncodeConfig::new(codec, WIDTH, HEIGHT) + .with_rate_control(rate_control) + .with_gop_size(0) + .with_b_frames(b_frames) + .with_b_frame_references(references); + let mut encoder = Encoder::new(context, config)?; + let mut input = YuvUploader::for_encoder(&encoder)?; + + let keep = 2 + b_frames as usize; + let mut pending = VecDeque::new(); + let mut packets = Vec::new(); + // The display order of the frames that must be IDRs. + let mut idrs = vec![0u64]; + // The IDR after a loss of every reference: the oldest frame that + // waits, or the next one if none does. + let mut restart = 0..=0; + let mut submitted = 0u64; + for i in 0..40 { + match i { + // Frame 10 waits as a B-frame: it becomes a P-frame. With + // the pyramid, more frames wait before it. + 11 => { + encoder.request_idr(); + idrs.push(submitted); + } + // Frame 19 waits. The repeated frame is encoded with it. + 20 => { + pending.push_back(encoder.repeat()?); + submitted += 1; + } + // The references from before the loss survive: no IDR. + 30 => encoder.invalidate_reference_frames(submitted - 2), + // Every reference is lost. + 33 => { + encoder.invalidate_reference_frames(0); + restart = submitted - u64::from(b_frames)..=submitted; + } + // Everything submitted is encoded, and the stream goes on. + 35 => { + encoder.flush()?; + drain(&mut pending, 0, &mut packets)?; + } + _ => {} + } + input.upload(YuvLayout::I420, &pattern(i))?; + pending.push_back(encoder.submit(Frame::image(input.source()))?); + submitted += 1; + drain(&mut pending, keep, &mut packets)?; + } + encoder.flush()?; + drain(&mut pending, 0, &mut packets)?; + + let mut pts: Vec = packets.iter().map(|p| p.pts).collect(); + pts.sort_unstable(); + assert!(pts.iter().copied().eq(0..submitted), "{codec:?}: {pts:?}"); + let mut found: Vec = packets + .iter() + .filter(|p| p.is_key_frame()) + .map(|p| p.pts) + .collect(); + found.sort_unstable(); + let last = found.pop(); + assert_eq!(found, idrs, "{codec:?}: IDRs"); + assert!( + last.is_some_and(|pts| restart.contains(&pts)), + "{codec:?}: IDR after every reference is lost: {last:?}, expected in {restart:?}" + ); + assert!( + packets.iter().any(|p| p.frame_type == FrameType::B), + "{codec:?}: no B-frame" + ); + for pair in packets.windows(2) { + assert!(pair[0].dts < pair[1].dts, "{codec:?}: dts must increase"); + } + + let dir = common::scratch_dir(&format!("b-frames-control-{extension}")); + let path = dir.join(format!("out.{extension}")); + let data: Vec = packets + .iter() + .flat_map(|p| p.data.iter().copied()) + .collect(); + std::fs::write(&path, data)?; + let decoded = dir.join("out.yuv"); + let mut ffmpeg = std::process::Command::new("ffmpeg"); + ffmpeg.args(["-hide_banner", "-loglevel", "error", "-y"]); + ffmpeg.args(["-err_detect", "explode"]); + if codec == Codec::AV1 { + ffmpeg.args(["-c:v", "libdav1d"]); + } + let status = ffmpeg + .arg("-i") + .arg(&path) + .args(["-pix_fmt", "yuv420p", "-f", "rawvideo"]) + .arg(&decoded) + .status()?; + assert!(status.success(), "{codec:?}: ffmpeg found an error"); + let frame_size = (WIDTH * HEIGHT * 3 / 2) as u64; + assert_eq!( + std::fs::metadata(&decoded)?.len() / frame_size, + submitted, + "{codec:?}: frames ffmpeg decoded" + ); + } + Ok(()) +} diff --git a/tests/common/mod.rs b/tests/common/mod.rs index 5d8e85ca..a25708f4 100644 --- a/tests/common/mod.rs +++ b/tests/common/mod.rs @@ -1,28 +1,13 @@ -//! Reading a decoded frame back to the CPU, plus the integration tests' -//! fixtures and scratch space. +//! The integration tests' fixtures, decode loop and scratch space. //! -//! pixelforge hands out a GPU image and stops there: it has no readback path of -//! its own, because a renderer sharing the device does not want one, and a -//! consumer that does want one knows better than the library where the pixels -//! should end up. This is that consumer's side of the deal, shared by the -//! tests that write raw YUV to a file. -//! -//! Two details are worth copying into real code: -//! -//! - A frame in `GENERAL` layout (which is what the zero-copy path hands out) -//! needs no transition to be copied from. A frame in any other layout is a -//! private image, so transitioning it is safe. -//! - The copy runs on the transfer queue, which the decoder also uses on the -//! copying fallback path. Submitting to one `VkQueue` from two threads at -//! once is undefined, so these examples drive the decoder and the readback -//! from the same thread. +//! Decoded frames are read back with [`YuvDownloader`], from the thread that +//! drives the decoder: both submit to the transfer queue. pub mod source; use ash::vk; +use pixelforge::YuvDownloader; use pixelforge::decoder::{DecodedFrame, Decoder, FramePoll}; -use pixelforge::encoder::BitDepth; -use pixelforge::vulkan::VideoContext; use std::fs::File; use std::io::Write; @@ -37,6 +22,7 @@ use std::io::Write; pub fn init_logging() { static ONCE: std::sync::Once = std::sync::Once::new(); ONCE.call_once(|| { + keep_driver_loaded(); let filter = tracing_subscriber::EnvFilter::try_from_default_env() .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")); let _ = tracing_subscriber::fmt() @@ -46,6 +32,26 @@ pub fn init_logging() { }); } +/// Keep the Vulkan driver loaded for the life of the test process. +/// +/// The loader loads a driver's libraries when an instance is created and +/// unloads them with the last instance, so a binary whose tests each build a +/// context loads the driver once per test. The NVIDIA driver (615.78.08) can +/// be loaded only about 26 times in one process: after that the loader finds +/// no driver. One instance that is never destroyed holds the libraries. +fn keep_driver_loaded() { + let Ok(entry) = (unsafe { ash::Entry::load() }) else { + return; + }; + let app_info = vk::ApplicationInfo::default().api_version(vk::API_VERSION_1_1); + let create_info = vk::InstanceCreateInfo::default().application_info(&app_info); + // The instance is never destroyed, and the entry must not unload the + // loader under it. + if unsafe { entry.create_instance(&create_info, None) }.is_ok() { + std::mem::forget(entry); + } +} + /// Bytes handed to the decoder per call, standing in for a network read. pub const CHUNK_SIZE: usize = 64 * 1024; @@ -80,324 +86,17 @@ where /// Write one decoded frame as NV12 to `output`, when an output is set. pub fn write_nv12( frame: &DecodedFrame, - readback: &mut Option, + readback: &mut Option, output: &mut Option, ) -> Result<(), Box> { if let (Some(file), Some(readback)) = (output.as_mut(), readback.as_mut()) { - let data = readback.read(frame)?; + let data = readback.download(frame)?; file.write_all(&data.y)?; file.write_all(&data.uv)?; } Ok(()) } -/// Find a memory type satisfying `flags` for `reqs`, and allocate from it. -pub fn allocate( - device: &ash::Device, - props: &vk::PhysicalDeviceMemoryProperties, - reqs: vk::MemoryRequirements, - flags: vk::MemoryPropertyFlags, -) -> Result> { - let index = (0..props.memory_type_count) - .find(|&i| { - reqs.memory_type_bits & (1 << i) != 0 - && props.memory_types[i as usize] - .property_flags - .contains(flags) - }) - .ok_or("no suitable memory type")?; - let memory = unsafe { - device.allocate_memory( - &vk::MemoryAllocateInfo::default() - .allocation_size(reqs.size) - .memory_type_index(index), - None, - ) - }?; - Ok(memory) -} - -/// One frame's pixels on the host, cropped to the visible region. -pub struct FrameData { - pub y: Vec, - pub uv: Vec, -} - -/// Copies decoded frames into host memory, reusing one staging buffer. -pub struct Readback { - /// Keeps the device alive: `VideoContext` is an `Arc` handle, and this - /// struct's objects must outlive nothing and be destroyed before the device - /// is. Holding a clone makes that ordering automatic rather than something - /// the caller has to get right. - _context: VideoContext, - device: ash::Device, - /// Barriers through the KHR entry point, which a device created below - /// Vulkan 1.3 still has; the adopted-device tests create one. - sync2: ash::khr::synchronization2::Device, - queue: vk::Queue, - pool: vk::CommandPool, - command_buffer: vk::CommandBuffer, - fence: vk::Fence, - memory_properties: vk::PhysicalDeviceMemoryProperties, - /// Staging buffer and the size actually allocated for it. - buffer: Option<(vk::Buffer, vk::DeviceMemory, usize)>, -} - -impl Readback { - pub fn new(context: &VideoContext) -> Result> { - let device = context.device().clone(); - let family = context.transfer_queue_family(); - let pool = unsafe { - device.create_command_pool( - &vk::CommandPoolCreateInfo::default() - .queue_family_index(family) - .flags(vk::CommandPoolCreateFlags::RESET_COMMAND_BUFFER), - None, - ) - }?; - let command_buffer = unsafe { - device.allocate_command_buffers( - &vk::CommandBufferAllocateInfo::default() - .command_pool(pool) - .level(vk::CommandBufferLevel::PRIMARY) - .command_buffer_count(1), - ) - }?[0]; - let fence = unsafe { device.create_fence(&vk::FenceCreateInfo::default(), None) }?; - let memory_properties = unsafe { - context - .instance() - .get_physical_device_memory_properties(context.physical_device()) - }; - Ok(Self { - _context: context.clone(), - sync2: ash::khr::synchronization2::Device::load(context.instance(), &device), - device, - queue: context.transfer_queue(), - pool, - command_buffer, - fence, - memory_properties, - buffer: None, - }) - } - - /// Copy `frame` into host memory and return its visible region. - pub fn read(&mut self, frame: &DecodedFrame) -> Result> { - let bytes = match frame.bit_depth { - BitDepth::Eight => 1usize, - BitDepth::Ten => 2, - }; - // NV12/P010 geometry: full-resolution luma, half-resolution interleaved - // chroma, both at the image's coded width so the copy is contiguous. - let y_stride = frame.coded_width as usize * bytes; - let y_size = y_stride * frame.coded_height as usize; - let uv_stride = y_stride; - let uv_size = uv_stride * (frame.coded_height as usize / 2); - self.ensure_capacity(y_size + uv_size)?; - let (buffer, memory, _) = self.buffer.expect("just ensured"); - - let layers = |aspect| vk::ImageSubresourceLayers { - aspect_mask: aspect, - mip_level: 0, - base_array_layer: frame.array_layer, - layer_count: 1, - }; - let regions = [ - vk::BufferImageCopy::default() - .buffer_offset(0) - .buffer_row_length(frame.coded_width) - .buffer_image_height(frame.coded_height) - .image_subresource(layers(vk::ImageAspectFlags::PLANE_0)) - .image_extent(vk::Extent3D { - width: frame.coded_width, - height: frame.coded_height, - depth: 1, - }), - vk::BufferImageCopy::default() - .buffer_offset(y_size as u64) - .buffer_row_length(frame.coded_width / 2) - .buffer_image_height(frame.coded_height / 2) - .image_subresource(layers(vk::ImageAspectFlags::PLANE_1)) - .image_extent(vk::Extent3D { - width: frame.coded_width / 2, - height: frame.coded_height / 2, - depth: 1, - }), - ]; - - // GENERAL is already a valid source for a transfer read, so the - // zero-copy path needs no barrier at all. Anything else is a private - // image this frame owns, so moving it is safe. - let needs_transition = frame.layout != vk::ImageLayout::GENERAL; - let planes = vk::ImageAspectFlags::PLANE_0 | vk::ImageAspectFlags::PLANE_1; - let range = vk::ImageSubresourceRange { - aspect_mask: planes, - base_mip_level: 0, - level_count: 1, - base_array_layer: frame.array_layer, - layer_count: 1, - }; - - unsafe { - self.device - .reset_command_buffer(self.command_buffer, vk::CommandBufferResetFlags::empty())?; - self.device.begin_command_buffer( - self.command_buffer, - &vk::CommandBufferBeginInfo::default() - .flags(vk::CommandBufferUsageFlags::ONE_TIME_SUBMIT), - )?; - - if needs_transition { - let to_src = [vk::ImageMemoryBarrier2::default() - .src_stage_mask(vk::PipelineStageFlags2::NONE) - .src_access_mask(vk::AccessFlags2::NONE) - .dst_stage_mask(vk::PipelineStageFlags2::COPY) - .dst_access_mask(vk::AccessFlags2::TRANSFER_READ) - .old_layout(frame.layout) - .new_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL) - .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .image(frame.image) - .subresource_range(range)]; - self.sync2.cmd_pipeline_barrier2( - self.command_buffer, - &vk::DependencyInfo::default().image_memory_barriers(&to_src), - ); - } - - self.device.cmd_copy_image_to_buffer( - self.command_buffer, - frame.image, - if needs_transition { - vk::ImageLayout::TRANSFER_SRC_OPTIMAL - } else { - frame.layout - }, - buffer, - ®ions, - ); - - if needs_transition { - let restore = [vk::ImageMemoryBarrier2::default() - .src_stage_mask(vk::PipelineStageFlags2::COPY) - .src_access_mask(vk::AccessFlags2::TRANSFER_READ) - .dst_stage_mask(vk::PipelineStageFlags2::NONE) - .dst_access_mask(vk::AccessFlags2::NONE) - .old_layout(vk::ImageLayout::TRANSFER_SRC_OPTIMAL) - .new_layout(frame.layout) - .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) - .image(frame.image) - .subresource_range(range)]; - self.sync2.cmd_pipeline_barrier2( - self.command_buffer, - &vk::DependencyInfo::default().image_memory_barriers(&restore), - ); - } - - self.device.end_command_buffer(self.command_buffer)?; - let command_buffers = [self.command_buffer]; - self.device.reset_fences(&[self.fence])?; - self.device.queue_submit( - self.queue, - &[vk::SubmitInfo::default().command_buffers(&command_buffers)], - self.fence, - )?; - self.device.wait_for_fences(&[self.fence], true, u64::MAX)?; - } - - // Crop to the visible region on the way out: the image is coded-size, - // which is padded up to the codec's macroblock granularity. - // An odd size rounds the chroma up: the last chroma sample covers - // the last luma column (and row) on its own. - let visible_y_stride = frame.width as usize * bytes; - let visible_uv_stride = frame.width.div_ceil(2) as usize * 2 * bytes; - let uv_rows = frame.height.div_ceil(2) as usize; - let mut y = Vec::with_capacity(visible_y_stride * frame.height as usize); - let mut uv = Vec::with_capacity(visible_uv_stride * uv_rows); - unsafe { - // Map the whole allocation: a sub-range map has to respect - // nonCoherentAtomSize, and WHOLE_SIZE sidesteps that. - let ptr = - self.device - .map_memory(memory, 0, vk::WHOLE_SIZE, vk::MemoryMapFlags::empty())? - as *const u8; - for row in 0..frame.height as usize { - y.extend_from_slice(std::slice::from_raw_parts( - ptr.add(row * y_stride), - visible_y_stride, - )); - } - for row in 0..uv_rows { - uv.extend_from_slice(std::slice::from_raw_parts( - ptr.add(y_size + row * uv_stride), - visible_uv_stride, - )); - } - self.device.unmap_memory(memory); - } - Ok(FrameData { y, uv }) - } - - fn ensure_capacity(&mut self, size: usize) -> Result<(), Box> { - if let Some((_, _, capacity)) = self.buffer - && capacity >= size - { - return Ok(()); - } - unsafe { - if let Some((buffer, memory, _)) = self.buffer.take() { - self.device.device_wait_idle()?; - self.device.destroy_buffer(buffer, None); - self.device.free_memory(memory, None); - } - let buffer = self.device.create_buffer( - &vk::BufferCreateInfo::default() - .size(size as u64) - .usage(vk::BufferUsageFlags::TRANSFER_DST) - .sharing_mode(vk::SharingMode::EXCLUSIVE), - None, - )?; - let reqs = self.device.get_buffer_memory_requirements(buffer); - let wanted = - vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT; - let index = (0..self.memory_properties.memory_type_count) - .find(|&i| { - reqs.memory_type_bits & (1 << i) != 0 - && self.memory_properties.memory_types[i as usize] - .property_flags - .contains(wanted) - }) - .ok_or("no host-visible memory for readback")?; - let memory = self.device.allocate_memory( - &vk::MemoryAllocateInfo::default() - .allocation_size(reqs.size) - .memory_type_index(index), - None, - )?; - self.device.bind_buffer_memory(buffer, memory, 0)?; - // Record what was allocated, not what was asked for. - self.buffer = Some((buffer, memory, reqs.size as usize)); - } - Ok(()) - } -} - -impl Drop for Readback { - fn drop(&mut self) { - unsafe { - let _ = self.device.device_wait_idle(); - if let Some((buffer, memory, _)) = self.buffer.take() { - self.device.destroy_buffer(buffer, None); - self.device.free_memory(memory, None); - } - self.device.destroy_fence(self.fence, None); - self.device.destroy_command_pool(self.pool, None); - } - } -} - /// A scratch directory for one test, under Cargo's target temp dir so /// `cargo clean` removes it. Created if missing. pub fn scratch_dir(name: &str) -> std::path::PathBuf { diff --git a/tests/data/av1/extract_trace.py b/tests/data/av1/extract_trace.py index f5e647c7..7e62d601 100644 --- a/tests/data/av1/extract_trace.py +++ b/tests/data/av1/extract_trace.py @@ -14,6 +14,7 @@ aom_10bit: ffmpeg -c:v libaom-av1 -cpu-used 8 -pix_fmt yuv420p10le -f obu aom_grain: ffmpeg -c:v libaom-av1 -cpu-used 8 -denoise-noise-level 10 -f obu svt_grain: ffmpeg -c:v libsvtav1 -preset 10 -svtav1-params film-grain=8 -f obu + svt_hier: ffmpeg -c:v libsvtav1 -preset 10 -crf 50 -svtav1-params hierarchical-levels=3:pred-struct=2 -f obu aom_errres: aomenc --cpu-used=8 --limit=15 --obu --error-resilient=1 --sframe-dist=5 --sframe-mode=1 aom_superres: aomenc --cpu-used=8 --limit=15 --obu --superres-mode=1 --superres-denominator=12 --superres-kf-denominator=10 aom_resize: aomenc --cpu-used=8 --limit=15 --obu --resize-mode=1 --resize-denominator=12 --resize-kf-denominator=16 diff --git a/tests/data/av1/svt_hier.expected b/tests/data/av1/svt_hier.expected new file mode 100644 index 00000000..ad86f356 --- /dev/null +++ b/tests/data/av1/svt_hier.expected @@ -0,0 +1,37 @@ +F obu=6 show_existing_frame=0 frame_type=0 show_frame=1 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=0 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=117 using_qmatrix=0 segmentation_enabled=0 delta_q_present=1 delta_q_res=0 delta_lf_present=0 loop_filter_level[0]=8 loop_filter_level[1]=8 loop_filter_level[2]=4 loop_filter_level[3]=4 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=1 cdef_bits=1 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 cdef_y_pri_strength[1]=0 cdef_y_sec_strength[1]=0 tx_mode=1 reduced_tx_set=0 header_bits=94 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=8 primary_ref_frame=0 refresh_frame_flags=2 ref_frame_idx[0]=0 ref_frame_idx[1]=0 ref_frame_idx[2]=0 ref_frame_idx[3]=0 ref_frame_idx[4]=0 ref_frame_idx[5]=0 ref_frame_idx[6]=0 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=144 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=8 loop_filter_level[1]=8 loop_filter_level[2]=4 loop_filter_level[3]=4 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=1 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 cdef_y_pri_strength[1]=0 cdef_y_sec_strength[1]=0 tx_mode=1 reference_select=1 allow_warped_motion=1 reduced_tx_set=0 header_bits=139 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=4 primary_ref_frame=0 refresh_frame_flags=16 ref_frame_idx[0]=0 ref_frame_idx[1]=0 ref_frame_idx[2]=0 ref_frame_idx[3]=0 ref_frame_idx[4]=1 ref_frame_idx[5]=1 ref_frame_idx[6]=1 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=169 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=0 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=116 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=2 primary_ref_frame=4 refresh_frame_flags=32 ref_frame_idx[0]=0 ref_frame_idx[1]=0 ref_frame_idx[2]=0 ref_frame_idx[3]=0 ref_frame_idx[4]=4 ref_frame_idx[5]=4 ref_frame_idx[6]=4 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=182 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=0 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=116 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=1 primary_ref_frame=4 refresh_frame_flags=0 ref_frame_idx[0]=0 ref_frame_idx[1]=0 ref_frame_idx[2]=0 ref_frame_idx[3]=0 ref_frame_idx[4]=5 ref_frame_idx[5]=5 ref_frame_idx[6]=5 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=7 cdef_y_sec_strength[0]=3 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=5 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=3 primary_ref_frame=0 refresh_frame_flags=0 ref_frame_idx[0]=5 ref_frame_idx[1]=5 ref_frame_idx[2]=5 ref_frame_idx[3]=5 ref_frame_idx[4]=4 ref_frame_idx[5]=4 ref_frame_idx[6]=4 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=1 cdef_bits=0 cdef_y_pri_strength[0]=0 cdef_y_sec_strength[0]=0 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=4 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=6 primary_ref_frame=0 refresh_frame_flags=32 ref_frame_idx[0]=4 ref_frame_idx[1]=4 ref_frame_idx[2]=4 ref_frame_idx[3]=4 ref_frame_idx[4]=1 ref_frame_idx[5]=1 ref_frame_idx[6]=1 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=182 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=0 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=116 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=5 primary_ref_frame=4 refresh_frame_flags=0 ref_frame_idx[0]=4 ref_frame_idx[1]=4 ref_frame_idx[2]=4 ref_frame_idx[3]=4 ref_frame_idx[4]=5 ref_frame_idx[5]=5 ref_frame_idx[6]=5 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=1 cdef_bits=0 cdef_y_pri_strength[0]=0 cdef_y_sec_strength[0]=0 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=5 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=7 primary_ref_frame=0 refresh_frame_flags=0 ref_frame_idx[0]=5 ref_frame_idx[1]=5 ref_frame_idx[2]=5 ref_frame_idx[3]=5 ref_frame_idx[4]=1 ref_frame_idx[5]=1 ref_frame_idx[6]=1 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=7 cdef_y_sec_strength[0]=3 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=1 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=12 primary_ref_frame=0 refresh_frame_flags=4 ref_frame_idx[0]=1 ref_frame_idx[1]=1 ref_frame_idx[2]=1 ref_frame_idx[3]=1 ref_frame_idx[4]=1 ref_frame_idx[5]=1 ref_frame_idx[6]=1 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=164 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=10 loop_filter_level[1]=10 loop_filter_level[2]=5 loop_filter_level[3]=5 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=1 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 cdef_y_pri_strength[1]=0 cdef_y_sec_strength[1]=0 tx_mode=1 reference_select=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=139 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=10 primary_ref_frame=0 refresh_frame_flags=8 ref_frame_idx[0]=1 ref_frame_idx[1]=1 ref_frame_idx[2]=1 ref_frame_idx[3]=1 ref_frame_idx[4]=2 ref_frame_idx[5]=2 ref_frame_idx[6]=2 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=191 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=2 cdef_bits=0 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=116 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=9 primary_ref_frame=4 refresh_frame_flags=0 ref_frame_idx[0]=1 ref_frame_idx[1]=1 ref_frame_idx[2]=1 ref_frame_idx[3]=1 ref_frame_idx[4]=3 ref_frame_idx[5]=3 ref_frame_idx[6]=3 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=7 cdef_y_sec_strength[0]=3 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=3 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=11 primary_ref_frame=0 refresh_frame_flags=0 ref_frame_idx[0]=3 ref_frame_idx[1]=3 ref_frame_idx[2]=3 ref_frame_idx[3]=3 ref_frame_idx[4]=2 ref_frame_idx[5]=2 ref_frame_idx[6]=2 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=7 cdef_y_sec_strength[0]=3 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=2 header_bits=4 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=0 showable_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=14 primary_ref_frame=0 refresh_frame_flags=1 ref_frame_idx[0]=2 ref_frame_idx[1]=2 ref_frame_idx[2]=2 ref_frame_idx[3]=2 ref_frame_idx[4]=2 ref_frame_idx[5]=2 ref_frame_idx[6]=2 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=194 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=16 loop_filter_level[1]=16 loop_filter_level[2]=8 loop_filter_level[3]=8 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=15 cdef_y_sec_strength[0]=2 tx_mode=1 reference_select=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=127 +T obu=6 +F obu=6 show_existing_frame=0 frame_type=1 show_frame=1 error_resilient_mode=0 disable_cdf_update=0 allow_screen_content_tools=0 frame_size_override_flag=0 order_hint=13 primary_ref_frame=0 refresh_frame_flags=0 ref_frame_idx[0]=2 ref_frame_idx[1]=2 ref_frame_idx[2]=2 ref_frame_idx[3]=2 ref_frame_idx[4]=0 ref_frame_idx[5]=0 ref_frame_idx[6]=0 allow_high_precision_mv=0 is_filter_switchable=1 use_ref_frame_mvs=1 disable_frame_end_update_cdf=0 tile_cols_log2=0 tile_rows_log2=0 base_q_idx=200 using_qmatrix=0 segmentation_enabled=0 delta_q_present=0 loop_filter_level[0]=0 loop_filter_level[1]=0 loop_filter_sharpness=0 loop_filter_delta_enabled=0 cdef_damping_minus_3=3 cdef_bits=0 cdef_y_pri_strength[0]=7 cdef_y_sec_strength[0]=3 tx_mode=1 reference_select=1 skip_mode_present=1 allow_warped_motion=0 reduced_tx_set=0 header_bits=115 +T obu=6 +F obu=3 show_existing_frame=1 frame_to_show_map_idx=0 header_bits=4 diff --git a/tests/data/av1/svt_hier.obu b/tests/data/av1/svt_hier.obu new file mode 100644 index 00000000..f15d4bc4 Binary files /dev/null and b/tests/data/av1/svt_hier.obu differ diff --git a/tests/data/bpyramid.264 b/tests/data/bpyramid.264 new file mode 100644 index 00000000..14514db2 Binary files /dev/null and b/tests/data/bpyramid.264 differ diff --git a/tests/data/bpyramid.265 b/tests/data/bpyramid.265 new file mode 100644 index 00000000..396df83f Binary files /dev/null and b/tests/data/bpyramid.265 differ diff --git a/tests/data/bstrict.264 b/tests/data/bstrict.264 new file mode 100644 index 00000000..398f49c4 Binary files /dev/null and b/tests/data/bstrict.264 differ diff --git a/tests/data/novui.264 b/tests/data/novui.264 new file mode 100644 index 00000000..fb66a8b7 Binary files /dev/null and b/tests/data/novui.264 differ diff --git a/tests/data/novui_poc2.264 b/tests/data/novui_poc2.264 new file mode 100644 index 00000000..29542bfd Binary files /dev/null and b/tests/data/novui_poc2.264 differ diff --git a/tests/data/strip_vui.py b/tests/data/strip_vui.py new file mode 100644 index 00000000..3fbbcf26 --- /dev/null +++ b/tests/data/strip_vui.py @@ -0,0 +1,125 @@ +"""Remove the VUI from the SPS of an H.264 Annex B stream. + +x264 always writes a VUI with a bitstream restriction, so a stream without +`max_num_reorder_frames` has to be made from one that has it. The decoder then +has to infer the reorder depth (see `Sps::reorder_depth`). + +Usage: + python3 strip_vui.py in.264 out.264 + + novui.264: from bframes.264 (reorder depth from the level) + novui_poc2.264: from zerolatency.264 (POC type 2, reorder depth 0) + +Handles what x264 writes for 4:2:0 frame-coded streams: no scaling matrices. +""" + +import sys + + +class Bits: + def __init__(self, data): + self.bits = "".join(f"{b:08b}" for b in data) + self.pos = 0 + + def u(self, n): + value = int(self.bits[self.pos:self.pos + n], 2) + self.pos += n + return value + + def ue(self): + zeros = 0 + while self.bits[self.pos] == "0": + zeros += 1 + self.pos += 1 + self.pos += 1 + return (1 << zeros) - 1 + (self.u(zeros) if zeros else 0) + + +def unescape(data): + out = bytearray() + zeros = 0 + for b in data: + if zeros >= 2 and b == 3: + zeros = 0 + continue + out.append(b) + zeros = zeros + 1 if b == 0 else 0 + return bytes(out) + + +def escape(data): + out = bytearray() + zeros = 0 + for b in data: + if zeros >= 2 and b <= 3: + out.append(3) + zeros = 0 + out.append(b) + zeros = zeros + 1 if b == 0 else 0 + return bytes(out) + + +def strip(sps): + """`sps` is the NAL unit with its header byte, emulation bytes included.""" + r = Bits(unescape(sps[1:])) + profile = r.u(8) + r.u(16) + r.ue() + if profile in (100, 110, 122, 244, 44, 83, 86, 118, 128, 138, 139, 134, 135): + chroma = r.ue() + assert chroma != 3, "4:4:4 is not handled" + r.ue() + r.ue() + r.u(1) + assert r.u(1) == 0, "scaling matrices are not handled" + r.ue() + poc_type = r.ue() + if poc_type == 0: + r.ue() + else: + assert poc_type == 2, "POC type 1 is not handled" + r.ue() + r.u(1) + r.ue() + r.ue() + if r.u(1) == 0: + r.u(1) + r.u(1) + if r.u(1): + for _ in range(4): + r.ue() + assert r.u(1) == 1, "the SPS has no VUI" + # vui_parameters_present_flag = 0, then the RBSP stop bit and alignment. + bits = r.bits[:r.pos - 1] + "0" + "1" + bits += "0" * (-len(bits) % 8) + rbsp = bytes(int(bits[i:i + 8], 2) for i in range(0, len(bits), 8)) + return sps[:1] + escape(rbsp) + + +def main(): + data = open(sys.argv[1], "rb").read() + # Split at start codes, keeping each one with the NAL unit that follows. + starts = [] + i = 0 + while True: + i = data.find(b"\x00\x00\x01", i) + if i < 0: + break + starts.append(i) + i += 3 + out = bytearray(data[:starts[0]]) + for n, start in enumerate(starts): + end = starts[n + 1] if n + 1 < len(starts) else len(data) + nal = data[start + 3:end] + # A four-byte start code leaves its leading zero on the previous unit. + tail = b"" + if n + 1 < len(starts) and nal.endswith(b"\x00"): + nal, tail = nal[:-1], b"\x00" + if nal[0] & 0x1F == 7: + nal = strip(nal) + out += b"\x00\x00\x01" + nal + tail + open(sys.argv[2], "wb").write(out) + + +if __name__ == "__main__": + main() diff --git a/tests/decode_streams.rs b/tests/decode_streams.rs index e51feefb..e095d87b 100644 --- a/tests/decode_streams.rs +++ b/tests/decode_streams.rs @@ -16,9 +16,9 @@ use std::path::Path; #[allow(dead_code)] mod common; -use common::{Readback, decode_stream, write_nv12}; +use common::{decode_stream, write_nv12}; -use pixelforge::decoder::{DecodeConfig, Decoder}; +use pixelforge::decoder::{DecodeConfig, Decoder, FramePoll}; use pixelforge::{Codec, VideoContextBuilder}; /// Decode `name` from `tests/data` and compare it against ffmpeg. @@ -51,13 +51,27 @@ fn check(codec: Codec, name: &str, pix_fmt: &str) -> Result<(), Box(), + Some(pixelforge::PixelForgeError::NoSuitableDevice(_)) + ) + { + eprintln!("skipping {name}: {e}"); + return Ok(()); + } + decoded?; drop(output); assert!( @@ -68,6 +82,9 @@ fn check(codec: Codec, name: &str, pix_fmt: &str) -> Result<(), Box { $( @@ -82,9 +99,14 @@ macro_rules! stream_tests { stream_tests! { h264_bframes: Codec::H264, "bframes.264", "nv12"; + h264_bpyramid: Codec::H264, "bpyramid.264", "nv12"; + h264_bstrict: Codec::H264, "bstrict.264", "nv12"; + h264_novui: Codec::H264, "novui.264", "nv12"; + h264_novui_poc2: Codec::H264, "novui_poc2.264", "nv12"; h264_multislice: Codec::H264, "multislice.264", "nv12"; h264_zerolatency: Codec::H264, "zerolatency.264", "nv12"; h265_closedgop: Codec::H265, "closedgop.265", "nv12"; + h265_bpyramid: Codec::H265, "bpyramid.265", "nv12"; h265_cra_start: Codec::H265, "cra_start.265", "nv12"; h265_main10: Codec::H265, "main10.265", "p010le"; h265_nvenc: Codec::H265, "nvenc.265", "nv12"; @@ -101,4 +123,87 @@ stream_tests! { av1_10bit: Codec::AV1, "av1/aom_10bit.obu", "p010le"; av1_grain: Codec::AV1, "av1/aom_grain.obu", "nv12"; av1_svt_grain: Codec::AV1, "av1/svt_grain.obu", "nv12"; + av1_svt_hier: Codec::AV1, "av1/svt_hier.obu", "nv12"; +} + +/// Split an H.264 byte stream with one slice per picture into access units. +fn h264_access_units(stream: &[u8]) -> Vec<&[u8]> { + // Offsets of every start code, counting a leading zero byte as part of it. + let mut nals = Vec::new(); + let mut i = 0; + while i + 3 <= stream.len() { + if stream[i..i + 3] == [0, 0, 1] { + let start = if i > 0 && stream[i - 1] == 0 { + i - 1 + } else { + i + }; + nals.push((start, stream[i + 3] & 0x1f)); + i += 3; + } else { + i += 1; + } + } + let mut starts = Vec::new(); + let mut seen_slice = true; + for (offset, nal_type) in nals { + let is_slice = matches!(nal_type, 1 | 5); + // The first unit after a slice opens the next access unit. + if seen_slice { + starts.push(offset); + } + seen_slice = is_slice; + } + let ends = starts.iter().skip(1).copied().chain([stream.len()]); + starts + .iter() + .copied() + .zip(ends) + .map(|(start, end)| &stream[start..end]) + .collect() +} + +/// A stream that cannot reorder must give each frame back before the next one +/// is fed: the reorder buffer may add no latency to it. +fn check_no_latency(name: &str) -> Result<(), Box> { + common::init_logging(); + let context = VideoContextBuilder::new() + .app_name("pixelforge-decode-latency") + .require_decode(Codec::H264) + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()) + .build()?; + let mut decoder = Decoder::new(context, DecodeConfig::new(Codec::H264))?; + let stream = std::fs::read(Path::new("tests/data").join(name))?; + let units = h264_access_units(&stream); + assert_eq!(units.len(), 30, "{name}: access units"); + + for (i, unit) in units.iter().enumerate() { + decoder.decode(unit, i as u64)?; + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(5); + let frame = loop { + match decoder.try_next_frame()? { + FramePoll::Frame(frame) => break frame, + FramePoll::Pending if std::time::Instant::now() < deadline => { + std::thread::sleep(std::time::Duration::from_millis(1)); + } + other => panic!("{name}: no frame after access unit {i} ({other:?})"), + } + }; + assert_eq!(frame.pts, i as u64, "{name}: frame order"); + } + Ok(()) +} + +#[test] +#[ignore = "requires a Vulkan Video device"] +fn h264_zerolatency_has_no_delay() -> Result<(), Box> { + check_no_latency("zerolatency.264") +} + +/// The same stream without a VUI: POC type 2 tells the decoder that nothing +/// is reordered, where the level alone would allow a delay of several frames. +#[test] +#[ignore = "requires a Vulkan Video device"] +fn h264_novui_poc2_has_no_delay() -> Result<(), Box> { + check_no_latency("novui_poc2.264") } diff --git a/tests/dmabuf.rs b/tests/dmabuf.rs index 9f3a236f..3c06e99d 100644 --- a/tests/dmabuf.rs +++ b/tests/dmabuf.rs @@ -9,17 +9,18 @@ #[allow(dead_code)] mod common; +use common::decode_stream; use common::source::{host_buffer, one_shot}; -use common::{Readback, decode_stream}; use ash::vk; use ash::vk::TaggedStructure; use pixelforge::decoder::{DecodeConfig, Decoder}; use pixelforge::encoder::external::drm_format; use pixelforge::{ - Codec, ColorSpec, Conversion, ConversionPreference, DmaBuf, DmaBufImage, DmaBufPlane, - EncodeConfig, Encoder, Frame, Handoff, RateControl, RgbFormat, RgbSource, Source, SourceImage, - VideoContext, VideoContextBuilder, YuvLayout, YuvUploader, + BFrameSupport, BitDepth, ChromaFormat, Codec, ColorSpec, Conversion, ConversionPreference, + DmaBuf, DmaBufImage, DmaBufPlane, EncodeConfig, Encoder, Frame, Handoff, RateControl, + RgbFormat, RgbSource, Source, SourceImage, VideoContext, VideoContextBuilder, YuvLayout, + YuvUploader, }; use std::os::fd::{FromRawFd, OwnedFd}; @@ -508,6 +509,87 @@ fn allocated_and_imported_inputs_encode_the_same() -> Result<(), Box Result<(), Box> { + let context = context()?; + let (w, h) = (320, 256); + let frames: Vec> = (0..9).map(|n| nv12_frame(w, h, n)).collect(); + for codec in [Codec::H264, Codec::H265, Codec::AV1] { + if !context.supports_encode(codec) + || context.supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight) + == BFrameSupport::None + { + eprintln!("skipping {codec:?}: no B-frames on this device"); + continue; + } + let config = config(codec, w, h).with_b_frames(2); + + let mut encoder = Encoder::new(context.clone(), config.clone())?; + let mut input = YuvUploader::for_encoder(&encoder)?; + let mut pending = Vec::new(); + for frame in &frames { + input.upload(YuvLayout::Nv12, frame)?; + pending.push(encoder.submit(Frame::image(input.source()))?); + } + encoder.flush()?; + let mut reference = Vec::new(); + for future in pending { + reference.extend(pollster::block_on(future)?.data); + } + + let mut encoder = Encoder::new(context.clone(), config)?; + // The release of a frame written in place would be an encode that + // has not been submitted yet. + let slot = encoder.input_slot()?; + assert!( + encoder + .submit(Frame::input(slot).signal_sync_file()) + .is_err(), + "{codec:?}: a release sync_file for a frame written in place" + ); + let modifiers = exportable_modifiers(&encoder); + if modifiers.is_empty() { + eprintln!("skipping {codec:?}: no modifier the encoder can allocate"); + continue; + } + let allocated = encoder.allocate_input(&modifiers)?; + assert!( + !allocated.is_zero_copy(), + "{codec:?}: a buffer read directly with B-frames" + ); + let mut producer = Producer::import(&context, &allocated.export()?)?; + let mut pending = Vec::new(); + for frame in &frames { + producer.write(frame)?; + pending.push(encoder.submit(Frame::external(&allocated))?); + // The one buffer is written again for the next frame, while + // this one may still wait for its encode. Its copy is all that + // has to be done by then. + for point in allocated.released() { + let semaphores = [point.semaphore]; + let values = [point.value]; + let wait = vk::SemaphoreWaitInfo::default() + .semaphores(&semaphores) + .values(&values); + unsafe { context.device().wait_semaphores(&wait, u64::MAX)? }; + } + } + encoder.flush()?; + let mut stream = Vec::new(); + for future in pending { + stream.extend(pollster::block_on(future)?.data); + } + assert!(stream == reference, "{codec:?}: copied input differs"); + } + Ok(()) +} + /// A buffer with a modifier the encode cannot read is copied instead, and /// encodes the same. #[test] @@ -605,14 +687,14 @@ fn unaligned_sizes_are_read_at_the_display_size() -> Result<(), Box 0, "{codec:?}: encode, but no quality levels"); + let at = |level| { + Encoder::new( + context.clone(), + config(codec, ChromaFormat::Yuv420, BitDepth::Eight).with_quality_level(level), + ) + }; + assert!( + at(levels - 1).is_ok(), + "{codec:?}: the top level {} fails", + levels - 1 + ); + assert!(at(levels).is_err(), "{codec:?}: level {levels} is accepted"); + } +} diff --git a/tests/rfi.rs b/tests/rfi.rs index 82027e7a..6d8be771 100644 --- a/tests/rfi.rs +++ b/tests/rfi.rs @@ -13,6 +13,11 @@ //! reference re-signaling — H.264 MMCO, H.265 RPS, AV1 ref mapping — keeps the //! decoder's DPB in sync with the encoder's). //! +//! Each codec runs without B-frames, and with a pyramid of three B-frames +//! where the device has one (flat B-frames where it has only those). With +//! B-frames, the recovery is checked by the absence of any keyframe after the +//! first. +//! //! Ignored by default: requires a Vulkan Video device and ffmpeg. Run with //! `cargo test -- --ignored`. @@ -20,8 +25,8 @@ mod common; use pixelforge::{ - BitDepth, ChromaFormat, Codec, EncodeConfig, Encoder, Frame, RateControl, VideoContextBuilder, - YuvLayout, YuvUploader, + BFrameReferences, BFrameSupport, BitDepth, ChromaFormat, Codec, EncodeConfig, Encoder, Frame, + RateControl, VideoContextBuilder, YuvLayout, YuvUploader, }; use std::collections::VecDeque; use std::fs::File; @@ -38,6 +43,9 @@ const INVALIDATE_AT: u64 = 12; const INVALIDATE_COUNT: u64 = 2; /// PSNR below this means the decoder desynced from the encoder after recovery. const MIN_PSNR: f64 = 30.0; +/// How far below the worst frame before the loss a frame may be. B-frames +/// have a higher QP than the other frames, so this cannot be very small. +const MAX_DROP: f64 = 3.0; #[test] #[ignore = "requires a Vulkan Video device and ffmpeg"] @@ -67,11 +75,21 @@ fn rfi() -> Result<(), Box> { println!("{codec:?}: skipped (encode not supported)"); continue; } - match run_codec(&context, &dir, codec, &input_path) { - Ok(()) => {} - Err(e) => { - println!("{codec:?}: FAIL: {e}"); - failures.push(format!("{codec:?}: {e}")); + let b_frames = match context.supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight) + { + BFrameSupport::None => None, + BFrameSupport::Flat => Some(BFrameReferences::Flat), + BFrameSupport::Pyramid => Some(BFrameReferences::Pyramid), + }; + let runs = std::iter::once(None).chain(b_frames.map(Some)); + for references in runs { + let name = match references { + None => format!("{codec:?}"), + Some(references) => format!("{codec:?} B-frames {references:?}"), + }; + if let Err(e) = run_codec(&context, &dir, codec, references, &input_path) { + println!("{name}: FAIL: {e}"); + failures.push(format!("{name}: {e}")); } } } @@ -86,15 +104,17 @@ fn run_codec( context: &pixelforge::VideoContext, dir: &std::path::Path, codec: Codec, + b_frames: Option, input_path: &str, ) -> Result<(), Box> { let ext = if codec == Codec::AV1 { "obu" } else { "bin" }; + let suffix = if b_frames.is_some() { "_b" } else { "" }; let output_filename = dir - .join(format!("output_rfi_{codec:?}.{ext}")) + .join(format!("output_rfi_{codec:?}{suffix}.{ext}")) .to_string_lossy() .into_owned(); let decoded_filename = dir - .join(format!("decoded_rfi_{codec:?}.yuv")) + .join(format!("decoded_rfi_{codec:?}{suffix}.yuv")) .to_string_lossy() .into_owned(); @@ -108,6 +128,13 @@ fn run_codec( .with_bit_depth(BitDepth::Eight) // Infinite GOP so the recovery can only come from RFI, never a periodic IDR. .with_gop_size(0); + let config = match b_frames { + Some(references) => config.with_b_frames(3).with_b_frame_references(references), + None => config, + }; + // At most `b_frames` frames wait for their group, so with more futures + // pending than that the oldest is ready (this also checks that claim). + let held = config.b_frames.max(2) as usize; let mut encoder = Encoder::new(context.clone(), config)?; let mut input_image = YuvUploader::for_encoder(&encoder)?; @@ -119,15 +146,19 @@ fn run_codec( let mut output_file = File::create(&output_filename)?; let mut pending: VecDeque = VecDeque::new(); let mut recovery_is_key: Option = None; + let mut later_key_frames = 0; - let drain_one = |pending: &mut VecDeque, - output_file: &mut File, - recovery_is_key: &mut Option| + let mut drain_one = |pending: &mut VecDeque, + output_file: &mut File, + recovery_is_key: &mut Option| -> Result<(), Box> { let packet = pollster::block_on(pending.pop_front().unwrap())?; if packet.pts == INVALIDATE_AT { *recovery_is_key = Some(packet.is_key_frame()); } + if packet.pts > 0 && packet.is_key_frame() { + later_key_frames += 1; + } output_file.write_all(&packet.data)?; Ok(()) }; @@ -149,7 +180,7 @@ fn run_codec( let slot = encoder.input_slot()?; input_image.upload_to(&slot, YuvLayout::I420, &yuv_data[start..end])?; pending.push_back(encoder.submit(Frame::input(slot))?); - while pending.len() > 2 { + while pending.len() > held { drain_one(&mut pending, &mut output_file, &mut recovery_is_key)?; } } @@ -159,7 +190,13 @@ fn run_codec( drain_one(&mut pending, &mut output_file, &mut recovery_is_key)?; } drop(output_file); + drop(drain_one); + if later_key_frames > 0 { + return Err( + format!("{later_key_frames} keyframes after the first (RFI did not engage)").into(), + ); + } match recovery_is_key { Some(true) => { return Err(format!( @@ -171,27 +208,48 @@ fn run_codec( Some(false) => {} } - let psnr = decode_and_psnr(&output_filename, &decoded_filename, input_path)?; + let frames = decode_and_psnr(&output_filename, &decoded_filename, input_path)?; std::fs::remove_file(&output_filename).ok(); std::fs::remove_file(&decoded_filename).ok(); + let psnr = frames.iter().sum::() / frames.len() as f64; if psnr < MIN_PSNR { return Err(format!( "post-recovery PSNR {psnr:.2} dB below {MIN_PSNR} dB (decoder desynced)" ) .into()); } + // A decoder that predicts from other pictures than the encoder did drifts + // from there on, by a few dB at first. The frames before the loss say + // what the quality is. + let before = &frames[..INVALIDATE_AT as usize - INVALIDATE_COUNT as usize]; + let worst_before = before.iter().copied().fold(f64::INFINITY, f64::min); + for (index, frame) in frames.iter().enumerate() { + if *frame < worst_before - MAX_DROP { + return Err(format!( + "frame {index} has PSNR {frame:.2} dB, more than {MAX_DROP} dB below \ + the worst frame before the loss ({worst_before:.2} dB): the decoder desynced" + ) + .into()); + } + } - println!("{codec:?}: PASS — recovered with a P-frame, full-stream PSNR {psnr:.2} dB"); + println!( + "{codec:?} {b_frames:?}: PASS — recovered with a P-frame, full-stream PSNR {psnr:.2} dB" + ); Ok(()) } -/// Decode the bitstream to raw YUV and return its PSNR against the source. +/// Decode the bitstream to raw YUV and return the PSNR of each frame against +/// the source. fn decode_and_psnr( bitstream: &str, decoded: &str, source: &str, -) -> Result> { +) -> Result, Box> { + // The device may code a frame larger than the display size (AV1's + // `codedPictureAlignment`); compare only the display size. + let crop = format!("crop={WIDTH}:{HEIGHT}:0:0"); let status = Command::new("ffmpeg") .args([ "-hide_banner", @@ -200,6 +258,8 @@ fn decode_and_psnr( "-y", "-i", bitstream, + "-vf", + &crop, "-pix_fmt", "yuv420p", "-f", @@ -238,17 +298,33 @@ fn decode_and_psnr( "-i", decoded, "-lavfi", - "psnr", + "psnr=stats_file=-", "-f", "null", "-", ]) .output()?; - let stderr = String::from_utf8_lossy(&output.stderr); - let pos = stderr - .find("average:") - .ok_or_else(|| format!("could not parse PSNR: {stderr}"))?; - let rest = &stderr[pos + 8..]; - let end = rest.find(' ').unwrap_or(rest.len()); - Ok(rest[..end].parse()?) + if !output.status.success() { + return Err(format!( + "ffmpeg psnr failed: {}", + String::from_utf8_lossy(&output.stderr) + ) + .into()); + } + // One line per frame: `n:1 mse_avg:... psnr_avg:51.23 ...`. + let stdout = String::from_utf8_lossy(&output.stdout); + let frames: Vec = stdout + .lines() + .filter_map(|line| { + let value = line + .split_whitespace() + .find_map(|f| f.strip_prefix("psnr_avg:"))?; + // A frame equal to the source has "inf". + Some(value.parse::().unwrap_or(f64::INFINITY).min(100.0)) + }) + .collect(); + if frames.is_empty() { + return Err(format!("could not parse PSNR: {stdout}").into()); + } + Ok(frames) } diff --git a/tests/rgb_encode.rs b/tests/rgb_encode.rs index 505a2299..a8568e05 100644 --- a/tests/rgb_encode.rs +++ b/tests/rgb_encode.rs @@ -10,8 +10,8 @@ #[allow(dead_code)] mod common; +use common::decode_stream; use common::source::{SrcImage, create_src_image, create_src_image_as}; -use common::{Readback, decode_stream}; use ash::vk; use pixelforge::decoder::{DecodeConfig, Decoder}; @@ -110,14 +110,14 @@ fn decode_all( context: &VideoContext, stream: &[u8], ) -> Result>, Box> { - let mut readback = Readback::new(context)?; + let mut readback = pixelforge::YuvDownloader::new(context)?; let mut decoder = Decoder::new( context.clone(), DecodeConfig::new(Codec::H264).with_byte_stream(), )?; let mut frames = Vec::new(); decode_stream(&mut decoder, stream, |frame| { - frames.push(readback.read(&frame)?.y); + frames.push(readback.download(&frame)?.y); Ok(()) })?; Ok(frames) diff --git a/tests/roundtrip.rs b/tests/roundtrip.rs index 2fa1cb3c..71f29702 100644 --- a/tests/roundtrip.rs +++ b/tests/roundtrip.rs @@ -3,12 +3,16 @@ //! The decoder must handle anything this project's encoder can produce. That is //! not covered by decoding third-party streams: the encoder makes its own //! choices (notably explicit reference marking), so it needs its own round trip. -//! It is also where B-frames will be exercised once the encoder supports them. +//! It is also where B-frames are exercised, for each count of them the +//! device can encode. //! //! The input is planar YUV420 (I420). The decoded output is NV12 in display //! order. The assertions are that no frame is lost across the round trip, and //! that the output is byte-identical to ffmpeg's software decode of the same -//! stream. The files are left in the scratch dir for inspection. +//! stream. With B-frames, also that the timestamps are in order, that ffmpeg +//! sees the frame types the encoder reports and no error in the stream, and +//! that the quality is close to that of the stream without B-frames. The files are +//! left in the scratch dir for inspection. //! //! Ignored by default: requires a Vulkan Video device and ffmpeg. Run with //! `cargo test -- --ignored`. @@ -19,12 +23,14 @@ use std::io::{Read, Write}; #[allow(dead_code)] mod common; -use common::{Readback, decode_stream, write_nv12}; +use common::{decode_stream, write_nv12}; use pixelforge::decoder::{DecodeConfig, Decoder}; use pixelforge::{ - Codec, EncodeConfig, Encoder, Frame, RateControl, VideoContextBuilder, YuvLayout, YuvUploader, + BFrameReferences, BFrameSupport, BitDepth, ChromaFormat, Codec, EncodeConfig, Encoder, + FrameType, RateControl, VideoContextBuilder, YuvLayout, YuvUploader, }; +use pixelforge::{Frame, MAX_B_FRAMES}; const WIDTH: u32 = 320; const HEIGHT: u32 = 240; @@ -33,28 +39,181 @@ const GOP: u32 = 30; #[test] #[ignore = "requires a Vulkan Video device and ffmpeg"] fn roundtrip_h264() -> Result<(), Box> { - roundtrip(Codec::H264, "h264") + roundtrip(Codec::H264, "h264", 0, BFrameReferences::Flat, 0).map(|_| ()) } #[test] #[ignore = "requires a Vulkan Video device and ffmpeg"] fn roundtrip_h265() -> Result<(), Box> { - roundtrip(Codec::H265, "h265") + roundtrip(Codec::H265, "h265", 0, BFrameReferences::Flat, 0).map(|_| ()) } #[test] #[ignore = "requires a Vulkan Video device and ffmpeg"] fn roundtrip_av1() -> Result<(), Box> { - roundtrip(Codec::AV1, "obu") + roundtrip(Codec::AV1, "obu", 0, BFrameReferences::Flat, 0).map(|_| ()) } -fn roundtrip(codec: Codec, extension: &str) -> Result<(), Box> { - let _ = tracing_subscriber::fmt::try_init(); +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_h264_top_quality_level() -> Result<(), Box> { + roundtrip_top_quality_level(Codec::H264, "h264") +} + +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_h265_top_quality_level() -> Result<(), Box> { + roundtrip_top_quality_level(Codec::H265, "h265") +} + +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_av1_top_quality_level() -> Result<(), Box> { + roundtrip_top_quality_level(Codec::AV1, "obu") +} + +/// A round trip at the highest quality level the device has, with and +/// without B-frames. A higher level only changes what the driver searches, +/// so the stream must decode as at level 0. +fn roundtrip_top_quality_level( + codec: Codec, + extension: &str, +) -> Result<(), Box> { + common::init_logging(); + let context = VideoContextBuilder::new().require_encode(codec).build()?; + let levels = context.encode_quality_levels(codec, ChromaFormat::Yuv420, BitDepth::Eight); + let b_frames = context.supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight); + drop(context); + assert!( + levels > 0, + "{codec:?}: the device reports no quality levels" + ); + let top = levels - 1; + eprintln!("{codec:?}: quality levels 0 to {top}"); + roundtrip(codec, extension, 0, BFrameReferences::Flat, top)?; + if b_frames != BFrameSupport::None { + roundtrip(codec, extension, 3, BFrameReferences::Flat, top)?; + } + Ok(()) +} + +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_h264_b_frames() -> Result<(), Box> { + roundtrip_b_frames(Codec::H264, "h264") +} + +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_h265_b_frames() -> Result<(), Box> { + roundtrip_b_frames(Codec::H265, "h265") +} + +#[test] +#[ignore = "requires a Vulkan Video device and ffmpeg"] +fn roundtrip_av1_b_frames() -> Result<(), Box> { + roundtrip_b_frames(Codec::AV1, "obu") +} + +/// Round trips with B-frames, against the one without. Nothing to do on a +/// device that encodes none. +fn roundtrip_b_frames(codec: Codec, extension: &str) -> Result<(), Box> { + let support = VideoContextBuilder::new() + .require_encode(codec) + .build()? + .supports_b_frames(codec, ChromaFormat::Yuv420, BitDepth::Eight); + if support == BFrameSupport::None { + eprintln!("{codec:?}: no B-frames on this device, nothing to check"); + return Ok(()); + } + let baseline = roundtrip(codec, extension, 0, BFrameReferences::Flat, 0)?; + // 30 frames: none of these counts divides the 29 after the IDR, so the + // last group is a short one, closed by the flush. A pyramid starts at + // 3 B-frames, has a second level of references at 6, and at 7 a B-frame + // replaces another as a reference. + let mut cases = vec![ + (1, BFrameReferences::Flat), + (2, BFrameReferences::Flat), + (3, BFrameReferences::Flat), + (MAX_B_FRAMES, BFrameReferences::Flat), + ]; + if support == BFrameSupport::Pyramid { + cases.extend([3, 4, 6, MAX_B_FRAMES].map(|b| (b, BFrameReferences::Pyramid))); + } else { + eprintln!("{codec:?}: no B-frame pyramid on this device"); + } + for (b_frames, references) in cases { + let psnr = roundtrip(codec, extension, b_frames, references, 0)?; + // B-frames get a coarser quantizer, one step more for each level + // of a pyramid, which costs up to about 1.6 dB at a constant QP. A + // B-frame predicted from the wrong picture costs 5 dB or more. + assert!( + psnr > baseline - 2.0, + "{b_frames} B-frames ({references:?}): luma PSNR {psnr:.2} dB, \ + {baseline:.2} dB without" + ); + } + Ok(()) +} + +/// The luma PSNR of `decoded` (NV12, frames of `decoded_size`) against +/// `source` (I420, `WIDTH` x `HEIGHT`), in dB. A decoded frame larger than +/// the source is compared in its top-left `WIDTH` x `HEIGHT`. +fn luma_psnr(source: &[u8], decoded: &[u8], decoded_size: (u32, u32)) -> f64 { + let (width, height) = (WIDTH as usize, HEIGHT as usize); + let decoded_width = decoded_size.0 as usize; + let source_frame = width * height * 3 / 2; + let decoded_frame = decoded_width * decoded_size.1 as usize * 3 / 2; + let mut error = 0u64; + let mut count = 0u64; + for (a, b) in source + .chunks_exact(source_frame) + .zip(decoded.chunks_exact(decoded_frame)) + { + for row in 0..height { + let a = &a[row * width..][..width]; + let b = &b[row * decoded_width..][..width]; + for (x, y) in a.iter().zip(b) { + let d = i64::from(*x) - i64::from(*y); + error += (d * d) as u64; + } + } + count += (width * height) as u64; + } + 10.0 * (255.0f64 * 255.0 * count as f64 / error.max(1) as f64).log10() +} + +/// The frame types ffmpeg sees in the stream, in display order, one letter +/// each. +fn ffprobe_frame_types(path: &std::path::Path) -> Result> { + let output = std::process::Command::new("ffprobe") + .args(["-v", "error", "-show_frames", "-show_entries"]) + .args(["frame=pict_type", "-of", "csv=p=0"]) + .arg(path) + .output()?; + assert!(output.status.success(), "ffprobe failed"); + Ok(String::from_utf8(output.stdout)? + .split_whitespace() + .collect()) +} + +/// Encode the test frames with `b_frames` B-frames, decode them, and return +/// the luma PSNR against the source. +fn roundtrip( + codec: Codec, + extension: &str, + b_frames: u32, + references: BFrameReferences, + quality_level: u32, +) -> Result> { + common::init_logging(); let input_path = format!("testdata/test_frames_{WIDTH}x{HEIGHT}_yuv420p.yuv"); common::ensure_test_data(WIDTH, HEIGHT, "yuv420p", &input_path)?; - let dir = common::scratch_dir(&format!("roundtrip-{extension}")); + let dir = common::scratch_dir(&format!( + "roundtrip-{extension}-b{b_frames}-{references:?}-q{quality_level}" + )); let bitstream_path = dir.join(format!("out.{extension}")); let context = VideoContextBuilder::new() @@ -73,26 +232,36 @@ fn roundtrip(codec: Codec, extension: &str) -> Result<(), Box = VecDeque::new(); for i in 0..frame_count { input_image.upload(YuvLayout::I420, &yuv[i * frame_size..(i + 1) * frame_size])?; pending.push_back(encoder.submit(Frame::image(input_image.source()))?); - while pending.len() > 2 { + // A B-frame waits for the reference frame after it, so a packet may + // not come before `b_frames` more frames are submitted. + while pending.len() > 2 + b_frames as usize { let packet = pollster::block_on(pending.pop_front().expect("non-empty"))?; + packets.push((packet.pts, packet.dts, packet.frame_type)); bitstream.extend_from_slice(&packet.data); } } encoder.flush()?; while let Some(future) = pending.pop_front() { let packet = pollster::block_on(future)?; + packets.push((packet.pts, packet.dts, packet.frame_type)); bitstream.extend_from_slice(&packet.data); } + check_timestamps(&packets, frame_count, b_frames); File::create(&bitstream_path)?.write_all(&bitstream)?; println!( "Encoded {} frames -> {} bytes (gop={GOP})", @@ -105,11 +274,15 @@ fn roundtrip(codec: Codec, extension: &str) -> Result<(), Box Result<(), Box Result<(), Box 0 && codec != Codec::AV1 { + let mut by_pts = packets.clone(); + by_pts.sort_by_key(|&(pts, _, _)| pts); + let expected: String = by_pts + .iter() + .map(|&(_, _, frame_type)| match frame_type { + FrameType::B => 'B', + FrameType::P => 'P', + _ => 'I', + }) + .collect(); + // A driver may code a P-frame with B slices that predict from the + // past only (low-delay B, which ANV does for H.265). ffprobe reports + // those as B, and the stream is no less correct. + let seen = ffprobe_frame_types(&bitstream_path)?; + let agrees = seen.len() == expected.len() + && seen + .chars() + .zip(expected.chars()) + .all(|(seen, expected)| seen == expected || (expected == 'P' && seen == 'B')); + assert!(agrees, "frame types: ffprobe {seen}, encoder {expected}"); + } + + Ok(luma_psnr(&yuv, &std::fs::read(&yuv_path)?, decoded_size)) +} + +/// Every frame has one packet, the packets come in decode order, and no +/// packet is to be decoded after it is shown. With B-frames, some of them +/// are B-frames. +fn check_timestamps(packets: &[(u64, i64, FrameType)], frame_count: usize, b_frames: u32) { + let mut pts: Vec = packets.iter().map(|&(pts, _, _)| pts).collect(); + pts.sort_unstable(); + assert!( + pts.iter().copied().eq(0..frame_count as u64), + "one packet for each frame: {pts:?}" + ); + for pair in packets.windows(2) { + assert!(pair[0].1 < pair[1].1, "dts must increase: {pair:?}"); + } + for &(pts, dts, _) in packets { + assert!(dts <= pts as i64, "dts {dts} is after pts {pts}"); + } + let b_count = packets + .iter() + .filter(|&&(_, _, frame_type)| frame_type == FrameType::B) + .count(); + if b_frames == 0 { + assert_eq!(b_count, 0); + } else { + // Each group is `b_frames` long at most, with one reference frame. + assert!(b_count >= (frame_count - 1) * b_frames as usize / (b_frames as usize + 1) - 1); + } } diff --git a/examples/sample_planes.rs b/tests/sample_planes.rs similarity index 80% rename from examples/sample_planes.rs rename to tests/sample_planes.rs index edb34db1..1493f442 100644 --- a/examples/sample_planes.rs +++ b/tests/sample_planes.rs @@ -1,113 +1,118 @@ //! Read decoded frames through per-plane views, with no ycbcr conversion. //! -//! The companion to `sample_frame`, and the reason -//! [`DecodedFrame::plane_views`] exists. `sample_frame` samples the decoded -//! picture as one multi-planar image, which needs a `VkSamplerYcbcrConversion` -//! bound as a combined image sampler with an immutable sampler. Some shader -//! toolchains cannot express that at all: naga has no combined-image-sampler -//! type, so wgpu has no ycbcr sampler, and a renderer built on either had no -//! way to read a decoded frame without copying it first. +//! [`DecodedFrame::plane_views`] says a decoded picture can be read as two +//! ordinary single-plane textures, `R8_UNORM` luma and `R8G8_UNORM` chroma, +//! bound as plain `texture2D` descriptors: no conversion object, no immutable +//! sampler. That is what a renderer needs whose shader toolchain has no ycbcr +//! sampler (naga, and so wgpu). It works because pixelforge asks for +//! `MUTABLE_FORMAT` on picture images wherever the driver allows it. //! -//! This reads the same picture as two ordinary single-plane textures instead, -//! `R8_UNORM` luma and `R8G8_UNORM` chroma, bound as plain `texture2D` -//! descriptors. No conversion object, no immutable sampler, no sampler at all. -//! It works because pixelforge asks for `MUTABLE_FORMAT` on picture images -//! wherever the driver allows it. +//! The shader writes the samples straight through, so its output must equal +//! [`YuvDownloader`]'s copy of the same frame, byte for byte. //! -//! A real renderer would apply the YUV to RGB matrix in the shader. This writes -//! the samples straight through, which makes the output plain NV12 and lets it -//! be compared byte-for-byte against the decoder: +//! On a device without `VK_KHR_unified_image_layouts`, frames are copies with +//! no video usage, so a mistake in how a plane view is made passes silently. +//! On a device with it the frame is the decoder's own image, and the same +//! mistake is a validation error. So run this under validation on such a +//! device too. //! -//! cargo run --example sample_planes -- tests/data/bframes.264 out.yuv -//! ffmpeg -i tests/data/bframes.264 -pix_fmt nv12 ref.yuv -//! cmp ref.yuv out.yuv -//! -//! Unlike `sample_frame`, this comparison is exact. There is no filtering, no -//! chroma reconstruction and no colour matrix in the way. +//! Ignored by default: requires a Vulkan Video device. Run with +//! `cargo test --test sample_planes -- --ignored`. #[allow(dead_code)] mod common; -use common::decode_input; use ash::vk::{self, TaggedStructure as _}; -use pixelforge::decoder::{DecodedFrame, Decoder}; +use pixelforge::decoder::{DecodeConfig, DecodedFrame, Decoder}; use pixelforge::vulkan::{VideoContext, VideoContextBuilder}; -use std::fs::File; -use std::io::Write; +use pixelforge::{Codec, YuvDownloader}; const SHADER: &[u8] = include_bytes!("shader/sample_planes.spv"); -fn main() -> Result<(), Box> { - tracing_subscriber::fmt::init(); - - let mut args = std::env::args().skip(1); - let input_path = args.next().unwrap_or_else(|| { - eprintln!("usage: sample_planes [output.yuv]"); - std::process::exit(1); - }); - let output_path = args.next(); - let (codec, input) = common::open_input(&input_path)?; - - let validation = std::env::var("PIXELFORGE_VALIDATION").is_ok(); - let context = VideoContextBuilder::new() - .app_name("pixelforge-planes") - .require_decode(codec) - .enable_validation(validation) - .build()?; - println!( - "Device: {}", - unsafe { std::ffi::CStr::from_ptr(context.device_properties().device_name.as_ptr()) } - .to_string_lossy() - ); - - let consumer_family = context.compute_queue_family(); - let config = common::decode_config(codec, &input) - .with_output_depth(8) - .with_consumer_queue_family(consumer_family); - - let mut output = output_path.as_ref().map(File::create).transpose()?; - let mut planes: Option = None; - let mut decoder = Decoder::new(context.clone(), config)?; - let mut count = 0usize; - let start = std::time::Instant::now(); - - decode_input(&mut decoder, &input, |frame| { - if !frame.plane_views { - return Err( - "this device does not allow per-plane views of decoded pictures; \ - use the sample_frame example instead" - .into(), - ); +#[test] +#[ignore = "requires a Vulkan Video device"] +fn plane_views_match_the_decoder() -> Result<(), Box> { + common::init_logging(); + // Both paths: the decoder's own images where the device has unified + // image layouts, and copies of them. + for (codec, path, copies) in [false, true].into_iter().flat_map(|copies| { + [ + (Codec::H264, "tests/data/bframes.264", copies), + (Codec::H265, "tests/data/bpyramid.265", copies), + (Codec::AV1, "tests/data/av1/aom_altref.obu", copies), + ] + }) { + let mut builder = VideoContextBuilder::new() + .require_decode(codec) + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()); + if copies { + builder = builder.without_unified_image_layouts(); } - let reader = match &mut planes { - Some(p) => p, - none => none.insert(PlaneReader::new(&context, consumer_family, &frame)?), - }; - let (y, uv) = reader.read(&frame)?; - if let Some(file) = output.as_mut() { - file.write_all(&y)?; - file.write_all(&uv)?; + let context = builder.build()?; + if !context.supports_decode(codec) { + eprintln!("{codec:?}: no decode on this device, nothing to check"); + continue; } - count += 1; - Ok(()) - })?; - - let elapsed = start.elapsed(); - println!( - "Read {} frames through plane views in {:.1?} ({:.1} fps)", - count, - elapsed, - count as f64 / elapsed.as_secs_f64() - ); - if let Some(path) = output_path { - println!("Wrote NV12 frames to {}", path); + let consumer_family = context.compute_queue_family(); + let config = DecodeConfig::new(codec) + .with_byte_stream() + .with_consumer_queue_family(consumer_family); + let mut decoder = Decoder::new(context.clone(), config)?; + let mut downloader = YuvDownloader::new(&context)?; + let mut planes: Option = None; + let mut count = 0; + common::decode_stream(&mut decoder, &std::fs::read(path)?, |frame| { + if !frame.plane_views { + return Err(format!("{path}: no per-plane views on this device").into()); + } + let reader = match &mut planes { + Some(p) => p, + none => none.insert(PlaneReader::new(&context, consumer_family, &frame)?), + }; + let (y, uv) = reader.read(&frame)?; + let expected = downloader.download(&frame)?; + assert!(y == expected.y, "{path}: frame {count}: luma differs"); + assert!(uv == expected.uv, "{path}: frame {count}: chroma differs"); + count += 1; + Ok(()) + })?; + assert!(count > 0, "{path}: no frames"); + eprintln!( + "{codec:?} (copies: {copies}): {count} frames read through plane views, all equal" + ); + // Vulkan objects go before the device. + drop(planes); + drop(decoder); } - - drop(planes); - drop(decoder); Ok(()) } +/// Find a memory type satisfying `flags` for `reqs`, and allocate from it. +fn allocate( + device: &ash::Device, + props: &vk::PhysicalDeviceMemoryProperties, + reqs: vk::MemoryRequirements, + flags: vk::MemoryPropertyFlags, +) -> Result> { + let index = (0..props.memory_type_count) + .find(|&i| { + reqs.memory_type_bits & (1 << i) != 0 + && props.memory_types[i as usize] + .property_flags + .contains(flags) + }) + .ok_or("no suitable memory type")?; + let memory = unsafe { + device.allocate_memory( + &vk::MemoryAllocateInfo::default() + .allocation_size(reqs.size) + .memory_type_index(index), + None, + ) + }?; + Ok(memory) +} + /// Reads a decoded picture's two planes as ordinary textures. struct PlaneReader { _context: VideoContext, @@ -392,10 +397,34 @@ impl PlaneReader { 1, ); - let after = [ + let mut after = vec![ storage_to_copy(self.out[0].image), storage_to_copy(self.out[1].image), ]; + // Give the frame back in the layout it reports: the decoder, and + // anyone else reading the frame, go by `frame.layout`. + if frame.layout != vk::ImageLayout::GENERAL { + after.push( + vk::ImageMemoryBarrier2::default() + .src_stage_mask(vk::PipelineStageFlags2::COMPUTE_SHADER) + .src_access_mask(vk::AccessFlags2::NONE) + .dst_stage_mask(vk::PipelineStageFlags2::NONE) + .dst_access_mask(vk::AccessFlags2::NONE) + .old_layout(vk::ImageLayout::GENERAL) + .new_layout(frame.layout) + .src_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .dst_queue_family_index(vk::QUEUE_FAMILY_IGNORED) + .image(frame.image) + .subresource_range(vk::ImageSubresourceRange { + aspect_mask: vk::ImageAspectFlags::PLANE_0 + | vk::ImageAspectFlags::PLANE_1, + base_mip_level: 0, + level_count: 1, + base_array_layer: frame.array_layer, + layer_count: 1, + }), + ); + } device.cmd_pipeline_barrier2( self.command_buffer, &vk::DependencyInfo::default().image_memory_barriers(&after), @@ -657,7 +686,7 @@ fn make_staging( ) }?; let reqs = unsafe { device.get_buffer_memory_requirements(buffer) }; - let memory = common::allocate( + let memory = allocate( device, props, reqs, @@ -695,7 +724,7 @@ fn make_plane( ) }?; let reqs = unsafe { device.get_image_memory_requirements(image) }; - let memory = common::allocate(device, props, reqs, vk::MemoryPropertyFlags::DEVICE_LOCAL)?; + let memory = allocate(device, props, reqs, vk::MemoryPropertyFlags::DEVICE_LOCAL)?; unsafe { device.bind_image_memory(image, memory, 0) }?; let view = unsafe { device.create_image_view( diff --git a/examples/shader/sample_planes.comp b/tests/shader/sample_planes.comp similarity index 100% rename from examples/shader/sample_planes.comp rename to tests/shader/sample_planes.comp diff --git a/examples/shader/sample_planes.spv b/tests/shader/sample_planes.spv similarity index 100% rename from examples/shader/sample_planes.spv rename to tests/shader/sample_planes.spv diff --git a/tests/submit.rs b/tests/submit.rs index c83e8bf7..563c7d4e 100644 --- a/tests/submit.rs +++ b/tests/submit.rs @@ -128,14 +128,14 @@ fn repeat_encodes_the_last_frame_again() -> Result<(), Box() / y.len() as f64); Ok(()) })?; diff --git a/tests/verify_all.rs b/tests/verify_all.rs index d686ce83..732a9008 100644 --- a/tests/verify_all.rs +++ b/tests/verify_all.rs @@ -225,6 +225,9 @@ fn run_test( _ => return Err("Unsupported format".into()), }; + // The device may code a frame larger than the display size (AV1's + // `codedPictureAlignment`); compare only the display size. + let crop = format!("crop={WIDTH}:{HEIGHT}:0:0"); let status = Command::new("ffmpeg") .args([ "-hide_banner", @@ -233,6 +236,8 @@ fn run_test( "-y", "-i", &output_filename, + "-vf", + &crop, "-pix_fmt", input_pix_fmt, "-f", diff --git a/tools/Cargo.toml b/tools/Cargo.toml new file mode 100644 index 00000000..680d00a7 --- /dev/null +++ b/tools/Cargo.toml @@ -0,0 +1,12 @@ +[package] +name = "pixelforge-tools" +version = "0.0.0" +edition = "2024" +description = "Command-line tools to develop and check pixelforge. Not published." +license = "BSD-2-Clause" +publish = false + +[dependencies] +pixelforge = { path = ".." } +pollster = "1.0" +tracing-subscriber = { version = "0.3.22", features = ["env-filter"] } diff --git a/tools/src/bin/decode.rs b/tools/src/bin/decode.rs new file mode 100644 index 00000000..1203f803 --- /dev/null +++ b/tools/src/bin/decode.rs @@ -0,0 +1,149 @@ +//! Decode a stream to raw YUV, to compare with a software decoder. +//! +//! Usage: +//! cargo run -p pixelforge-tools --bin decode -- input.264 [output.yuv] +//! +//! The codec comes from the extension: `.264`/`.h264`, `.265`/`.h265`/`.hevc`, +//! `.obu`/`.av1`, or `.ivf` (AV1 in IVF). The output is NV12, or P010 for a +//! 10-bit stream, so it compares byte for byte with +//! `ffmpeg -i input.264 -pix_fmt nv12 ref.yuv` (`-pix_fmt p010le` for 10 +//! bits, and `-c:v libdav1d -noautoscale` for AV1). +//! +//! `PIXELFORGE_VALIDATION=1` turns on the validation layers. +//! `PIXELFORGE_NO_UNIFIED_LAYOUTS=1` forces the path that copies each frame +//! out of the decoder, on a device that would hand over its own images. + +use pixelforge::Codec; +use pixelforge::decoder::{DecodeConfig, Decoder, FramePoll}; +use pixelforge::{VideoContextBuilder, YuvDownloader}; +use std::io::Write; + +/// Bytes handed to the decoder per call, standing in for a network read. +const CHUNK_SIZE: usize = 64 * 1024; + +fn main() -> Result<(), Box> { + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")), + ) + .init(); + + let mut args = std::env::args().skip(1); + let input = args + .next() + .ok_or("usage: decode [output.yuv]")?; + let mut output = args.next().map(std::fs::File::create).transpose()?; + let Input { + codec, + pieces, + config, + } = open(&input)?; + + let mut builder = VideoContextBuilder::new() + .require_decode(codec) + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()); + if std::env::var("PIXELFORGE_NO_UNIFIED_LAYOUTS").is_ok() { + builder = builder.without_unified_image_layouts(); + } + let context = builder.build()?; + let mut downloader = YuvDownloader::new(&context)?; + // A frame per call can be outstanding at once, and a chunk holds several. + let mut decoder = Decoder::new(context, config.with_output_depth(8))?; + + let start = std::time::Instant::now(); + let mut count = 0usize; + let mut generation = None; + let mut on_frame = + |frame: pixelforge::decoder::DecodedFrame| -> Result<(), Box> { + if generation.is_some_and(|g| g != frame.generation) { + println!( + " generation {} -> {} at frame {count}: the decoder rebuilt its images", + generation.unwrap(), + frame.generation + ); + } + generation = Some(frame.generation); + if let Some(file) = output.as_mut() { + let yuv = downloader.download(&frame)?; + file.write_all(&yuv.y)?; + file.write_all(&yuv.uv)?; + } + count += 1; + Ok(()) + }; + for (i, chunk) in pieces.iter().enumerate() { + decoder.decode(chunk, i as u64)?; + while let FramePoll::Frame(frame) = decoder.try_next_frame()? { + on_frame(frame)?; + } + } + decoder.finish()?; + while let Some(frame) = pollster::block_on(decoder.next_frame())? { + on_frame(frame)?; + } + let elapsed = start.elapsed(); + println!( + "Decoded {count} frames in {elapsed:.1?} ({:.1} fps)", + count as f64 / elapsed.as_secs_f64() + ); + Ok(()) +} + +/// A stream read from a file. +struct Input { + codec: Codec, + /// The pieces to hand the decoder, one per call. + pieces: Vec>, + /// The configuration the pieces need. + config: DecodeConfig, +} + +/// Read the stream at `path`. +fn open(path: &str) -> Result> { + let extension = std::path::Path::new(path) + .extension() + .and_then(|e| e.to_str()) + .unwrap_or("") + .to_ascii_lowercase(); + let data = std::fs::read(path)?; + let codec = match extension.as_str() { + "264" | "h264" => Codec::H264, + "265" | "h265" | "hevc" => Codec::H265, + "obu" | "av1" | "ivf" => Codec::AV1, + other => return Err(format!("unknown stream extension {other:?}").into()), + }; + if extension == "ivf" { + // Whole frames from a container. + return Ok(Input { + codec, + pieces: ivf_frames(&data)?, + config: DecodeConfig::new(codec), + }); + } + // A raw stream, which the decoder frames itself. + Ok(Input { + codec, + pieces: data.chunks(CHUNK_SIZE).map(<[u8]>::to_vec).collect(), + config: DecodeConfig::new(codec).with_byte_stream(), + }) +} + +/// The frames of an IVF file: a 32-byte file header, then each frame behind a +/// 12-byte header holding its size and timestamp. +fn ivf_frames(data: &[u8]) -> Result>, Box> { + if data.len() < 32 || &data[..4] != b"DKIF" { + return Err("not an IVF file".into()); + } + let header_len = u16::from_le_bytes([data[6], data[7]]) as usize; + let mut frames = Vec::new(); + let mut pos = header_len; + while pos + 12 <= data.len() { + let size = u32::from_le_bytes(data[pos..pos + 4].try_into()?) as usize; + let start = pos + 12; + let frame = data.get(start..start + size).ok_or("truncated IVF frame")?; + frames.push(frame.to_vec()); + pos = start + size; + } + Ok(frames) +} diff --git a/tools/src/bin/encode.rs b/tools/src/bin/encode.rs new file mode 100644 index 00000000..0554594c --- /dev/null +++ b/tools/src/bin/encode.rs @@ -0,0 +1,126 @@ +//! Encode raw YUV420 frames, with every setting that matters for checking the +//! encoder. +//! +//! Usage: +//! cargo run -p pixelforge-tools --bin encode -- [h264|h265|av1] [options] +//! +//! Options: +//! --input PATH raw I420 frames (default testdata/test_frames.yuv) +//! --size WxH frame size (default 320x240) +//! --frames N encode at most N frames +//! --qp N constant QP or q-index (default 26) +//! --gop N frames per IDR, 0 for none (default 30) +//! --b-frames N B-frames between two reference frames (default 0) +//! --pyramid make the middle B-frames references +//! --quality-level N the driver's quality level (default 0) +//! --output PATH the stream (default output.h264, output.h265 or output.av1) +//! +//! Each packet is printed with its timestamps and type. + +use pixelforge::{ + BFrameReferences, Codec, EncodeConfig, Encoder, Frame, RateControl, VideoContextBuilder, + YuvLayout, YuvUploader, +}; +use std::collections::VecDeque; +use std::io::Write; + +fn main() -> Result<(), Box> { + tracing_subscriber::fmt() + .with_env_filter( + tracing_subscriber::EnvFilter::try_from_default_env() + .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn")), + ) + .init(); + + let mut codec = Codec::H264; + let mut input = "testdata/test_frames.yuv".to_string(); + let (mut width, mut height) = (320u32, 240u32); + let mut frames = usize::MAX; + let mut qp = 26; + let mut gop = 30; + let mut b_frames = 0; + let mut references = BFrameReferences::Flat; + let mut quality_level = 0; + let mut output = None; + + let mut args = std::env::args().skip(1); + while let Some(arg) = args.next() { + let mut value = || args.next().ok_or(format!("{arg} takes a value")); + match arg.as_str() { + "h264" => codec = Codec::H264, + "h265" => codec = Codec::H265, + "av1" => codec = Codec::AV1, + "--input" => input = value()?, + "--size" => { + let size = value()?; + let (w, h) = size.split_once('x').ok_or("--size takes WxH")?; + (width, height) = (w.parse()?, h.parse()?); + } + "--frames" => frames = value()?.parse()?, + "--qp" => qp = value()?.parse()?, + "--gop" => gop = value()?.parse()?, + "--b-frames" => b_frames = value()?.parse()?, + "--pyramid" => references = BFrameReferences::Pyramid, + "--quality-level" => quality_level = value()?.parse()?, + "--output" => output = Some(value()?), + other => return Err(format!("unknown argument {other:?}").into()), + } + } + let output = output.unwrap_or_else(|| { + match codec { + Codec::H264 => "output.h264", + Codec::H265 => "output.h265", + Codec::AV1 => "output.av1", + } + .to_string() + }); + + let data = std::fs::read(&input) + .map_err(|e| format!("{input}: {e} (make test frames with `ffmpeg -f lavfi -i testsrc=duration=1:size=320x240:rate=30 -pix_fmt yuv420p -f rawvideo {input}`)"))?; + let frame_size = YuvLayout::I420.frame_size(width, height); + let count = (data.len() / frame_size).min(frames); + + let context = VideoContextBuilder::new() + .enable_validation(std::env::var("PIXELFORGE_VALIDATION").is_ok()) + .require_encode(codec) + .build()?; + let config = EncodeConfig::new(codec, width, height) + .with_rate_control(RateControl::Cqp { qp }) + .with_gop_size(gop) + .with_b_frames(b_frames) + .with_b_frame_references(references) + .with_quality_level(quality_level); + println!("{config:?}\n"); + let mut encoder = Encoder::new(context, config)?; + let mut uploader = YuvUploader::for_encoder(&encoder)?; + let mut file = std::fs::File::create(&output)?; + + let mut total = 0; + let mut write = |packet: pixelforge::EncodedPacket| -> std::io::Result<()> { + total += packet.data.len(); + println!( + " pts={:<3} dts={:<3} {:>7} bytes {:?}", + packet.pts, + packet.dts, + packet.data.len(), + packet.frame_type + ); + file.write_all(&packet.data) + }; + let mut pending = VecDeque::new(); + for frame in data.chunks_exact(frame_size).take(count) { + let slot = encoder.input_slot()?; + uploader.upload_to(&slot, YuvLayout::I420, frame)?; + pending.push_back(encoder.submit(Frame::input(slot))?); + // At most `b_frames` frames wait for their group. + while pending.len() > b_frames as usize + 1 { + write(pollster::block_on(pending.pop_front().unwrap())?)?; + } + } + encoder.flush()?; + for future in pending { + write(pollster::block_on(future)?)?; + } + println!("\n{count} frames, {total} bytes, to {output}"); + Ok(()) +}