Repository navigation
bench: decode latency inside a client's display loop #81
Description
Why
The HEVC comparison (#80) times decodes in two ways: one frame after the other, and as a stream's frames arrive, 60 a second, with the thread asleep in between. On the M3 Max the second takes emberwave 2.6 to 4.9 times as long as the first: 2.05 against 0.42 ms at 1080p, 3.37 against 1.00 ms at 4K, into the planes. VideoToolbox's HEVC decode changes far less. With the CPU kept busy instead of asleep, emberwave's 4K decode still took 2.77 ms, so most of it is the GPU's power state between frames.
A real client sits between the two: it draws the picture every display refresh, and its network and audio threads wake between frames. How long its decode takes is not measured yet, and it decides what an emberwave client gains.
What
- In the benchmark app and its Mac front-end, time decodes while a display loop draws the last picture full screen at the display's refresh (60 and 120 Hz), frames arriving 60 a second on a thread of their own, for emberwave and HEVC alike.
- On the Mac and the iPhone.
- If emberwave's decode stays slow there, look at what a client can do about it, for example submitting the decode with the frame's draw, or macOS's Game Mode for a full-screen client.
Activity
with-developer commentedon Oct 26, 2026
A first probe, on the M3 Max with a scratch build of the Mac front-end (not kept). While the decodes ran at 60 frames a second, a second Metal queue blitted a 3456x2234 BGRA texture into another every 1/120 s, as a compositor copies a full-screen layer at the display's refresh. Two runs each, mean ms:
Frame after frame, emberwave took 1.00 to 1.04 ms at 4K and 0.42 to 0.43 ms at 1080p. So a light load at the display's refresh does not keep the GPU fast between frames, and the slowdown is likely to stay in a real client.
Still to do here:
with-developer commentedon Oct 26, 2026
A second probe, same scratch build: a kick ahead of each frame. At every 1/60 s tick a fill of a 16 MB buffer went to the GPU, as a client could send one when a frame's first packet arrives. The decode then waited 1, 2 or 4 ms, as for the rest of the frame, before it started. One run each, mean ms:
A wake-up alone does not help: the GPU is not asleep but slow. Its clocks follow how busy it has been, and a decode every 16.7 ms keeps it at a few per cent.
The remaining ways seem to be less GPU work per frame, as an FP16 pyramid would give (upstream's default, 0.25 against 0.48 ms of GPU time at 4K FP32 on this Mac), or a client whose GPU is busy anyway. A game-streaming host is in the second case, as it renders the game.
with-developer commentedon Oct 26, 2026
A third probe, same scratch build: the benchmark's thread at QoS user-interactive instead of the default, as a client's decode thread might run. Two rounds each, mean ms at 60 frames a second:
No help either. Only a thread spinning between frames shortened emberwave's paced decodes (
EMBERWAVE_BENCH_SPIN=1, BENCHMARKS.md), and a client cannot spend a core on that.with-developer commentedon Oct 26, 2026
Upstream's decoder slows down the same way (#85,
refgen bench --fps 60, the streams of BENCHMARKS.md's tables, two rounds, mean ms):That is 2.8 to 3.9 times as long, with the GPU time itself 2.1 to 3.2 times. So the slowdown belongs to GPU wavelet decoding on Apple's GPUs at a stream's pace, not to emberwave. A real display loop and the iPhone are still to be measured here.
docs(bench): emberwave's decodes at a stream's pace, next to upstream's
Merge pull request #161: bench: decode beside a display loop on the M…
with-developer commentedon Nov 6, 2026
The Mac's half is in main (#161, decision D72, BENCHMARKS.md "In a display loop, on the Mac"). Decodes beside a display loop at 60 and 120 Hz, M3 Max, two runs of the first version:
Still open here: