Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y build-essential cmake ninja-build python3

- name: Run cargo check
run: cargo check --all-targets --all-features
run: cargo check --workspace --all-targets --all-features

test:
name: Test Suite
Expand Down Expand Up @@ -59,7 +59,7 @@ jobs:
run: sudo apt-get update && sudo apt-get install -y build-essential cmake ninja-build python3

- name: Run clippy
run: cargo clippy --all-targets --all-features -- -D warnings
run: cargo clippy --workspace --all-targets --all-features -- -D warnings

fmt:
name: Rustfmt
Expand Down
132 changes: 106 additions & 26 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,14 @@ cargo test -- --ignored # device + ffmpeg tests, ignored by default
cargo bench --bench encode
cargo run --example encode -- h264
cargo run --example decode -- input.264 output.yuv
cargo run -p pixelforge-tools --bin encode -- h264 --b-frames 3
cargo run -p pixelforge-tools --bin decode -- input.264 output.yuv
```

The examples (`examples/`) are for users: short, one use each. The tools
(`tools/`, a workspace crate that is not published) are for checking the
library, with every option that takes. The commands below use the tools.

Decoding on Intel Arc under Mesa needs the video queues enabled explicitly:
`ANV_DEBUG=video-decode,video-encode`.

Expand All @@ -31,7 +37,7 @@ Do not edit `README.md` directly; update the doc comments in `src/lib.rs` instea
To verify the quality of the encoded videos, run:

```bash
cargo run --example encode -- h265 \
cargo run -p pixelforge-tools --bin encode -- h265 \
&& rm -f decoded.yuv \
&& ffmpeg -hide_banner -loglevel error -y -i output.h265 -pix_fmt yuv420p -f rawvideo decoded.yuv \
&& ffmpeg -hide_banner -loglevel info -s 320x240 -pix_fmt yuv420p -f rawvideo -i testdata/test_frames.yuv -s 320x240 -pix_fmt yuv420p -f rawvideo -i decoded.yuv -lavfi psnr -f null -
Expand All @@ -41,27 +47,28 @@ To verify decoding, decode a stream and compare against ffmpeg's software
decoder, which should be byte-identical:

```bash
cargo run --example decode -- tests/data/bframes.264 out.yuv \
cargo run -p pixelforge-tools --bin decode -- tests/data/bframes.264 out.yuv \
&& ffmpeg -hide_banner -loglevel error -y -i tests/data/bframes.264 -pix_fmt nv12 ref.yuv \
&& cmp ref.yuv out.yuv && echo "decode matches ffmpeg"
```

The `decode` example picks the codec from the extension (`.264`, `.265`,
The `decode` tool picks the codec from the extension (`.264`, `.265`,
`.obu`, `.ivf`). For AV1 the reference is dav1d, and `-noautoscale` keeps a
stream that changes size from being scaled to its first size (it applies to
any codec; H.264 and H.265 streams in `tests/data` do not change size):

```bash
cargo run --example decode -- tests/data/av1/aom_altref.obu out.yuv \
cargo run -p pixelforge-tools --bin decode -- tests/data/av1/aom_altref.obu out.yuv \
&& ffmpeg -hide_banner -loglevel error -y -c:v libdav1d -i tests/data/av1/aom_altref.obu \
-noautoscale -pix_fmt nv12 ref.yuv \
&& cmp ref.yuv out.yuv && echo "decode matches dav1d"
```

`tests/decode_streams.rs` does this for every fixture in `tests/data` (use
`p010le` for the 10-bit ones), and `tests/roundtrip.rs` for each codec's own
encoder output. A run with `PIXELFORGE_NO_UNIFIED_LAYOUTS=1` set takes the
copying path below.
encoder output. Those tests take whichever path the device has; the
`PIXELFORGE_NO_UNIFIED_LAYOUTS` switch below is read by the `decode` tool
only.

On NVIDIA (615.78.08) the AV1 decoder does not copy the `StdVideoAV1SequenceHeader`
handed to `vkCreateVideoSessionParametersKHR`: it reads it again at every
Expand All @@ -76,7 +83,7 @@ with no copy. Where it does not, they are copied into private images. Force the
copying path to exercise it on hardware that would otherwise never take it:

```bash
PIXELFORGE_NO_UNIFIED_LAYOUTS=1 cargo run --example decode -- \
PIXELFORGE_NO_UNIFIED_LAYOUTS=1 cargo run -p pixelforge-tools --bin decode -- \
tests/data/bframes.264 out.yuv && cmp ref.yuv out.yuv
```

Expand All @@ -96,27 +103,25 @@ Do not judge that one by PSNR. The hardware sampler and ffmpeg reconstruct
subsampled chroma differently, so they agree almost everywhere and disagree
hard on pixels sitting on a colour edge; on the synthetic test patterns that
drags PSNR to ~26 dB while the images are indistinguishable. Compare the share
of pixels that agree instead. For `bframes.264` that has been 87.9% within 2
and 33.2% exact on every device tested so far, and it is deterministic enough
to treat a change in it as a regression.
of pixels that agree instead (all three channels). For `bframes.264` that has
been 87.9% within 2 on every device tested so far, and it is deterministic
enough to treat a change in it as a regression.

`sample_planes` is the other half of that story, and the tighter test. It reads
the same frames through per-plane views, with no ycbcr conversion and no sampler
at all, and writes the samples straight through, so its output is plain NV12 and
can be compared exactly:
`tests/sample_planes.rs` is the other half of that story, and the tighter
check. It reads the same frames through per-plane views, with no ycbcr
conversion and no sampler at all, and compares the samples byte for byte with
`YuvDownloader`'s copy of each frame, on both decode paths:

```bash
cargo run --example sample_planes -- tests/data/bframes.264 out.yuv \
&& cmp ref.yuv out.yuv && echo "plane views match the decoder"
cargo test --test sample_planes -- --ignored --nocapture
```

Where that example is worth extra attention: on a device *without*
`VK_KHR_unified_image_layouts`, its frames are pool copies, which carry no video
usage, so a mistake in how a plane view is created passes silently. The same
mistake is an immediate validation error on a device that has the extension,
because there the frame really is the decoder's DPB image. So if a device
supporting it is available, run this example there before trusting a green run
elsewhere. `RUST_LOG=debug` reports which case applies, as `pinnable=`.
On a device *without* `VK_KHR_unified_image_layouts`, its frames are pool
copies, which carry no video usage, so a mistake in how a plane view is created
passes silently. The same mistake is an immediate validation error on a device
that has the extension, because there the frame really is the decoder's DPB
image. So if a device supporting it is available, run it there under
validation before trusting a green run elsewhere.

Nothing in `tests/data` changes resolution mid-stream, and that path has its
own hazards, so build one and check it:
Expand All @@ -125,11 +130,11 @@ own hazards, so build one and check it:
ffmpeg -f lavfi -i testsrc2=size=320x240:rate=30:duration=1 -c:v libx264 -bf 2 -f h264 a.264
ffmpeg -f lavfi -i testsrc2=size=640x480:rate=30:duration=1 -c:v libx264 -bf 2 -f h264 b.264
cat a.264 b.264 > switch.264
cargo run --example decode -- switch.264 out.yuv
cargo run -p pixelforge-tools --bin decode -- switch.264 out.yuv
```

Sixty frames, thirty at each size, and one `generation N -> N+1` line at the
changeover. Run `sample_frame` and `sample_planes` over it too: a consumer's own
changeover. Run `sample_frame` over it too: a consumer's own
images are sized from a frame, so a resolution change is where they have to be
rebuilt, and getting that wrong is silent rather than loud. Build the reference per segment, since ffmpeg rescales a
resolution-changing stream to its first size and a whole-stream reference will
Expand Down Expand Up @@ -181,6 +186,79 @@ PIXELFORGE_MAX_ACTIVE_REFERENCES=1 cargo test --test rfi -- --ignored --nocaptur
`RUST_LOG=pixelforge::encoder::codec=debug` reports the negotiated window and
per-frame count, on the `Reference window` line.

`tests/rfi.rs` runs each codec without B-frames and with the B-frames the
device has. It checks each frame's PSNR against the frames before the loss,
not only the average: a decoder that predicts from other pictures than the
encoder did loses a few dB from there on, which the average hides. That is
how the H.264 P-frame after an invalidation was found to predict from the
lost frames in a decoder (it had MMCO operations and no list modification).

B-frames (`EncodeConfig::with_b_frames`, `tests/b_frames.rs`, the `_b_frames`
tests in `tests/roundtrip.rs`) are encoded later than they are submitted. The
planner (`src/encoder/gop.rs`) is pure code and has the unit tests for the
order. What needs a device is the stream each codec writes, and the checks are
ffmpeg's (dav1d's for AV1): decode with `-err_detect explode`, compare with
pixelforge's decoder byte for byte, and compare the frame types `ffprobe`
sees. Run the encode tool with them when touching reference lists:

```bash
cargo run -p pixelforge-tools --bin encode -- h264 --b-frames 7 --pyramid \
&& ffmpeg -hide_banner -loglevel error -err_detect explode -i output.h264 -f null -
```

A wrong reference list does not always make ffmpeg report an error. It can
decode cleanly to the wrong pictures, so also check the PSNR against the source
with the command above: a B-frame that predicts from another picture than the
encoder used drops it by 5 dB or more on those frames. To see what a driver
wrote into a slice header, use
`ffmpeg -i output.h264 -c copy -bsf:v trace_headers -f null -`.

AV1 has no B-frame type. The reference frame after a group is encoded first as
a hidden frame, whose bytes go in front of the next packet, and a
`show_existing_frame` header made on the CPU shows it later
(`src/encoder/av1/references.rs`). So AV1 packets are in display order, and
`ffprobe` reports no B-frames for them.

What the drivers do with B-frames, as far as it was checked:

- NVIDIA (615.78.08) reports pyramid support for H.264 and AV1, and flat only
for H.265. So the H.265 pyramid does not run in the tests on that driver. It
did decode correctly there once with the capability check forced off.
- An H.264 pyramid needs `ref_pic_list_modification` on the P-frame after it,
and drivers differ in how they write the operations they are given. NVIDIA
writes exactly the entries it gets: the end operation has to be one of them,
for the list operations, and then for the marking operations too. It ends
the marking operations by itself only in a P-slice with no list operations.
This is checked on NVIDIA only, so `implemented` in `src/encoder/bframes.rs`
offers the H.264 pyramid on that vendor only. Before offering it on another
driver, check the slice headers there with `trace_headers`.
- NVIDIA's H.264 B-frames use forward prediction only (no skip, no direct, no
list 1), so they cost more bits than P-frames. ffmpeg's `h264_vulkan` with
`-bf 2` gives the same on that GPU, so it is the driver's and not
pixelforge's. H.265 and AV1 B-frames save bits there.
- RADV and ANV were not available when B-frames were written. Nothing about
B-frames is checked on them.
- NVIDIA (615.78.08) writes its own value for some syntax, whatever
pixelforge asks for: `sps_temporal_mvp_enabled_flag = 0` (H.265),
`enable_ref_frame_mvs = 0`, `enable_jnt_comp = 0` and
`skip_mode_present = 0` (AV1). Those tools would help B-frames most, so
they are worth trying on another driver, with
`ffmpeg -c copy -bsf:v trace_headers` to see whether it writes them.

`scripts/compare_b0.sh` builds v0.11.0 in a worktree and compares its output
with the working tree's for each codec, with CQP and CBR. Without B-frames all
streams now differ from v0.11.0, for four reasons, all intended: H.264 has its
deblocking filter on, H.264 CBR has no QP bounds, H.265 has 8x8 as its
smallest coding block, and AV1 uses CDEF and loop restoration. Before those
changes, `h265-*-gop0` was the only difference: there v0.11.0 gave the driver
a picture order count that wrapped at 512, and the count no longer wraps. Run
it after a change to the encode flow, with a base revision that has those
changes (`scripts/compare_b0.sh <revision>`), for an exact comparison.

A test binary that builds many contexts must call `common::init_logging()`
first, also for another reason than logging: the NVIDIA driver can be loaded
only about 26 times in one process, and that call keeps it loaded.

Tests run one at a time (`RUST_TEST_THREADS=1` in `.cargo/config.toml`): the
NVIDIA driver can deadlock when one thread destroys a device while another
creates objects on a different one.
Expand Down Expand Up @@ -218,7 +296,9 @@ view by image handle, and only RADV without validation showed it.

## Project Structure
- `src/` - Library source code
- `examples/` - Usage examples for encoding and decoding
- `examples/` - Short usage examples, one use each
- `tools/` - Command-line tools to check the library (workspace crate, not
published)
- `tests/` - Integration tests; the device + ffmpeg checks are `#[ignore]`d and
run with `cargo test -- --ignored`
- `benches/` - The encode timing/throughput benchmark (`cargo bench`)
Expand Down
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,41 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- Add B-frame encoding for H.264, H.265 and AV1: `EncodeConfig::with_b_frames`, at most `MAX_B_FRAMES`.
- Add B-frame pyramids: `EncodeConfig::with_b_frame_references` and `BFrameReferences`.
- Add `VideoContext::supports_b_frames` and `BFrameSupport`.
- Add `--b-frames` and `--pyramid` to the `encode` example.
- Add `EncodeConfig::with_quality_level` and `VideoContext::encode_quality_levels`. The default stays 0, the fastest level. `Encoder::new` returns an error for a level the device does not have.
- Add `YuvDownloader` and `DownloadedFrame`, to copy decoded frames to CPU memory: the reverse of `YuvUploader`.
- Add the `encode_gpu` example: encode RGB frames that the application draws into its own Vulkan image.
- Add the `tools` workspace crate (not published), with `encode` and `decode` tools that have the options for checking the library.

### Changed

- The examples are short and show one use each. `encode` and `decode` lost their testing options (now in the tools), `query_capabilities` uses pixelforge's own queries, and `sample_planes` is now a test (`tests/sample_planes.rs`) that compares plane views with `YuvDownloader` on both decode paths.

- **Breaking:** `EncodedPacket::dts` is an `i64`. With B-frames it starts below zero.
- **Breaking:** A future from `Encoder::submit` resolves with the next packet in decode order. With B-frames that is not the packet of the submitted frame.
- With B-frames, `Encoder::invalidate_reference_frames` drops every reference encoded from the first lost frame on, and the next frames predict from the references that survive. If none survives, the oldest frame that waits for its group is encoded at once as the IDR.
- With a constant QP, the B-frames of a pyramid get one QP step more for each level (a q-index step of 4 for AV1). With 7 B-frames, streams are 2% to 4% smaller at the same quality.
- The H.264 encoder writes the lowest level from 4.1 that the stream needs, by its size, frame rate, DPB and peak bitrate. It always wrote 4.1, also where the stream needed more. `max_dec_frame_buffering` is one frame smaller (NVIDIA already wrote that value).
- The H.264 and H.265 reference pictures carry their own picture type to the driver, not always P.
- With B-frames, a DMA-BUF input is always copied.
- H.265 streams with more than 256 frames between two IDR frames differ from 0.11.0: the encoder now gets the picture order count without a wrap.
- The AV1 encoder uses CDEF and loop restoration. At the same quality, streams are 3% to 10% smaller, and each frame takes longer to encode (about 0.6 ms at 1080p on an RTX 5080).

### Fixed

- Fix the H.264 decoder's reorder depth for streams without a VUI.
- Fix the H.264 encoder, which turned the deblocking filter off. At the same quality, streams are about 10% smaller.
- Fix the H.264 encoder's CBR and VBR, which held the QP between 18 and 42. On easy content the bitrate stayed far below the target.
- Fix H.264 reference frame invalidation: the P-frame after it predicted from the lost frames in a decoder, because it did not reorder its reference list. The quality dropped by about 8 dB from there on.
- Let the H.265 encoder use 8x8 coding blocks. The minimum was 16x16. At the same quality, streams are 3% to 18% smaller.

## [0.11.0] - 06-11-2026

### Added
Expand Down
9 changes: 9 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ keywords = ["vulkan", "video", "encoding", "decoding", "h264"]
categories = ["multimedia::video", "graphics"]
readme = "README.md"

[workspace]
members = [".", "tools"]

[features]
default = []
dmabuf = []
Expand Down
Loading
Loading