Repository navigation
nn::codec video decoders decode on their own thread, dav1d on frame threads: movies no longer stutter at a steady 60 fps - #2240
Closed
SailorSnoW wants to merge 1 commit into
Conversation
…d on 4 threads with 3 frames at once: a submission queues its packet and returns, as the SDK's does. Decoded inside Submit…InputBuffer, a packet held the client's thread for the whole decode; Ocarina of Time's movie reader, which also gives the shown pictures back, was held up to 120 ms, and its title movie (AV1, 1080p at 60 fps, dav1d on an M2 Pro) showed a picture again in 546 and 569 of about 2,900 frames at a steady 60 fps, now in 0 or 1 - hle_codec_video.c: cv_worker decodes the packets cv_dec_submit queues. The submission-ready event and the refusal count the pictures waiting for a released buffer, the packets queued and the one in decode together (cv_pending), so a client sees the same event whatever the decoder's speed. The decoder's thread waits for room instead of dropping a picture, and signals the sync point events itself. Stop and Flush drop the queued packets and number the stream anew, so the pictures of a packet still in decode are dropped; Finalize joins the thread. Guest calls take the decoder's lock as a host lock. One log line when a run of 600 packets left more released buffers waiting than the decoder has (`the decoder is behind the movie`), and the totals at Finalize - hle_movie_av1.c: dav1d on 4 threads, 3 frames at once (15.4 → 7.1 ms a packet over the title movie's heaviest five seconds, 122 → 42 to 76 ms the longest; more task threads on one frame gave nothing). Pictures come out later than their units, so each unit's time rides with its picture (mav1_picture.ts) and mav1_next gives the pictures after a unit, draining when no packet waits behind it or the stream ends. A drain that has given a picture goes to its end: dav1d 1.5.4's drain, left halfway by new data, retires a frame with task_thread.first past the oldest one and cancels a live frame's task rescan, and dav1d_submit_frame waits for ever (the title movie froze in 3 runs of 6 with a drain that stopped when a packet arrived) - t_codec_video.c: the event and the refusal with the packets decoded or not, which thread signals what, eight flushes with undecoded packets, a Finalize with undecoded packets, AV1 key frames through the protocol with their times in order, a mid-stream drain and 600 rounds of drains at random places - docs/reference/movies.md: `nn::codec` packets decoded inside the submission; dav1d decoded one frame at a time; the two entries they change. The Ocarina of Time note, a line in Kirby Air Riders' (the busy line's wording); symptom index regenerated Measured on an M2 Pro (Mac14,9, macOS 27.2.1), the title for 54 s, headless, the player's settings, DM_PROBE on the player's advance (main+0x3b61ed8 failed, main+0x3b61f80 good): before, 546 and 569 failed of about 2,900; after, 0 or 1 of 2,567 to 3,036 in ten runs, no frozen movie. DecodeAsync returns in 0.10 ms at the median and 1.1 ms at the 99th percentile, from 1.9 and 120 ms. In a window, 88 s of the title movie: 5,273 packets in 4.82 ms each, 2 pictures after their buffer. Checked: make -C dm/host/tests run and tsan-threads pass; t_codec_video also under ThreadSanitizer, five runs with no report; test_docs passes; the host builds with no new warning. Not checked: no regression run (the harness is not set up on this Mac, which has this game only), so Donkey Kong Bananza, Mario Kart World, Mario Tennis Fever, Kirby Air Riders and Fire Emblem Fortune's Weave were not run with it; an M1 or M2 with four performance cores and an iPad were not run; a Mac with a hardware AV1 decoder (VideoToolbox, where only the decoder's thread applies) was not measured. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
github-actions
Bot
added
host
Contributor
|
A heads-up on an overlap: this PR and @MetaIX's open #1377 and #1379 change the same AV1 path in different ways.
A trial cherry-pick of this PR's commit (b9a7fc7) onto a tree that has #1377 and #1379 gives 18 conflicting hunks across 🤖 Generated with Claude Code |
seedds
added a commit
that referenced
this pull request
Nov 12, 2026
…d on 4 threads with 3 frames at once, so movies no longer stutter at a steady 60 fps (PR #2240): a submission queues its packet and returns, as the SDK's does (its decoder is another process). Decoded inside Submit…InputBuffer, a packet held the client's thread for the whole decode; Ocarina of Time's movie reader, which also gives the shown pictures back, was held up to 120 ms, and its title movie (AV1, 1080p at 60 fps, dav1d) showed a picture again in 546 and 569 of about 2,900 frames on an M2 Pro at a steady 60 fps - hle_codec_video.c: cv_worker decodes the packets cv_dec_submit queues. The submission-ready event and the refusal count the pictures waiting for a released buffer, the packets queued and the one in decode together (cv_pending), so a client sees the same event whatever the decoder's speed. The decoder's thread waits for room instead of dropping a picture, and signals the sync point events itself. Stop and Flush drop the queued packets and number the stream anew, so the pictures of a packet still in decode are dropped; Finalize joins the thread. Guest calls take the decoder's lock as a host lock. One log line when a run of 600 packets left more released buffers waiting than the decoder has (`the decoder is behind the movie`), and the totals at Finalize - hle_movie_av1.c: dav1d on 4 threads, 3 frames at once (15.4 -> 7.1 ms a packet over the title movie's heaviest five seconds on the M2 Pro). Pictures come out later than their units, so each unit's time rides with its picture (mav1_picture.ts) and mav1_next gives the pictures after a unit, draining when no packet waits behind it or the stream ends. A drain that has given a picture goes to its end: dav1d 1.5.4's drain, left halfway by new data, leaves a live frame's task never run and dav1d_submit_frame waits for ever (a workaround of the library's bug, in the code comment and the reference entry) - t_codec_video.c: the event and the refusal with the packets decoded or not, which thread signals what, flushes and a Finalize with undecoded packets, AV1 key frames through the protocol with their times in order, a mid-stream drain and 600 rounds of drains at random places. The AV1 backend loop checks mav1_backend() rather than its pass: on a machine that reports hardware AV1 but cannot make the session (CI's, -12906) the VideoToolbox pass decodes with dav1d and gave its picture only on the drain. The four Initialize calls pass empty parameter sets: ccd834e added those arguments to both overloads and the test no longer compiled - docs/reference/movies.md: `nn::codec` packets decoded inside the submission; dav1d decoded one frame at a time; the entries they change. The Ocarina of Time note, Kirby Air Riders' busy line; symptom index regenerated PR #1377 (MetaIX) found the frame-threading cause first and carried each packet's time through Dav1dData.m the same way; it decoded on the client's thread under a lock of its own, and is superseded by this. Checked on an M4 Pro (48 GB), macOS 27.2: t_codec_video 3 runs, and 5 runs under ThreadSanitizer with no report; the rest of the host tests' run list (t_hid does not link on main: hle_hid.c's dm_gfx_addr_note, dm_gfx_bufsnap, probe_atrace) and tsan-threads; test_docs; lint_c.py tidy on both changed files, 0 findings. Games, headless, A/B with hosts from the tree at 918e9f3^ (chain cb2e91c138e3, libraries on disk) without and with this host diff: Ocarina of Time's title for 54 s with DM_AV1_DAV1D=1 and DM_PROBE on the player's advance (main+0x3b61ed8 failed, main+0x3b61f80 good): 361 failed of 3,041 before, 0 of 3,057 after, the same frame times and red lines. Mario Kart World's cup opening movie (VP9 2560x1440, libvpx): all 1,423 pictures on both, frame p95 18.8-19.3 ms in the movie on both, 3.24 ms a packet and 13.4 ms the longest, no new red line. The PR's own runs: Ocarina's title on an M2 Pro, ten runs, 0 or 1 failed advance; DecodeAsync 0.10 ms at the median and 1.1 ms at the 99th percentile, from 1.9 and 120. Not run: Fire Emblem Fortune's Weave (SDK 22's native images; its shader split on disk is older than the map), Donkey Kong Bananza, Mario Tennis Fever, Kirby Air Riders, an iPhone or iPad decoding AV1 with dav1d, regress.py (games chosen by hand: the impact map has no usage data for the codec games) Co-authored-by: SailorSnoW <49660929+SailorSnoW@users.noreply.github.com>
Owner
|
Landed on main as 09a91c5, with you as co-author. Thank you: the decoder thread is the root-cause fix (the SDK's submission only queues), and it lifts all four codecs, not just AV1. Two small changes on top:
Rechecked on an M4 Pro (48 GB, macOS 27.2), headless, one run a side with and without the host diff:
Still not run: Fire Emblem Fortune's Weave (SDK 22 native images), Bananza, Mario Tennis Fever, Kirby Air Riders, and an iPhone or iPad on dav1d. |
1 task done
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Movies stuttered in small regular jerks while the frame rate read a steady 60 fps: Ocarina of Time's title movie and cutscenes, on a Mac without a hardware AV1 decoder. The game's frames were on time; the movie's pictures were not. Two causes, both in the host and general.
Packets were decoded inside the submission. The SDK's
Submit…InputBufferqueues the packet and returns; the host decoded it on the client's thread. Ocarina's player feeds the decoder and gives the shown pictures back on one reader thread, so a long decode (up to 120 ms) left the frame loop with nothing new to show. Each decoder now decodes on a thread of its own (cv_worker). The submission-ready event and the refusal count queued packets and waiting pictures together, so a client sees the same protocol.dav1d decoded one frame at a time. It took 15.4 ms a packet over the title movie's heaviest five seconds, at a budget of 16.7 ms. It now runs 4 threads on 3 frames at once (7.1 ms). Pictures come out later than their units, so each carries its unit's time, and a drain always runs to its end: dav1d 1.5.4 deadlocks when new data interrupts one.
A log line now reports a decoder slower than its movie (
the decoder is behind the movie).Reference entries:
Checked
make -C dm/host/tests GAME=0400486024DEC000 runandtsan-threadspass.t_codec_videounder ThreadSanitizer, five runs, no report.test_docspasses. The host builds with no warning in the changed files.0400486024DEC000, v131072). The title for 54 s, headless, ten runs, counting the player's failed and goodadvancecalls withDM_PROBE. One windowed run watched for 88 s of the movie.nn::codecand were not run with it.Measurements on the title movie (AV1, 1080p at 60 fps):
DecodeAsync, median / 99th percentileDocs
docs/reference/movies.mdentries, andpython3 dm/tools/docs_symptoms.pywas run🤖 Generated with Claude Code