bench: decode beside a display loop on the Mac (#81)

ARCHITECTURE.md

@@ -1844,3 +1844,23 @@ The last frame's detail, always, gains everything on a still picture and loses u
 **Not done.** Calibrating the bandwidth term on the iPhone (a factor of 0.88 to 0.95) for other chips: it would fit one device and be an extrapolation for all the others.
 
 **Consequence.** No code in the library changes. A new measurement of the M3 Max or the iPhone is entered into the script's table, and the table in BENCHMARKS.md is rewritten with `scripts/perf_model.py --write BENCHMARKS.md`. A device measured later is added as a check.
+
+### 2026-11-06 — D72: Timing decodes beside a display loop on the Mac
+
+**Context.** The paced rows of BENCHMARKS.md time a decode with the GPU idle between frames. A client draws at every refresh instead, and its decodes share the GPU with that. Issue #81 asked what a decode then takes, for emberwave and for HEVC, and the earlier probes (a second queue blitting a texture, a kick ahead of each frame, a raised QoS) had not settled it.
+
+**Decision.** The benchmark app's Mac front-end gets a `-display` mode (`scripts/bench_app_on_mac.sh -display`), in `EmberwaveBenchMac/DisplayLoop.swift` and `DisplayRun.swift`. A borderless window covers one screen while it runs; a `CVDisplayLink` and a `CAMetalLayer` with two drawables draw the last decoded picture at the screen's refresh. The decode runs on another thread at a stream's pace. Every way (emberwave into Y, Cb and Cr textures and into RGB, HEVC at Moonlight's rate into NV12) runs without the loop and then with a 60 Hz and a 120 Hz loop in the same run, so the loop's cost is a difference within one run. The report adds the time from a frame's arrival to the display's presentation of it.
+
+**Why these choices.**
+- A `CVDisplayLink`, where a `CADisplayLink` of an app that is not in front stayed at 60 calls a second on the 120 Hz screen whatever frame rate range it asked for (62.5 a second in a minimal program, against 124 for the `CVDisplayLink`).
+- The stream arrives at 59.5 frames a second, not 60, so that frames drift across the refreshes. At 60 every frame had the same phase against the display, and the latency of a run was one number between 12 and 25 ms, set by where the run happened to start.
+- The 60 Hz loop draws every second refresh of the 120 Hz screen: the Mac's 60 Hz external monitor took no window (it never reported the window on screen), so the run falls back to the built-in screen when a screen does not take the window.
+- Two drawables, as a client that minds latency has.
+
+**Result.** Into the planes, emberwave decoded as quickly as or up to 0.8 ms quicker than HEVC into NV12 in the loop (0 to 0.3 ms at 1080p, 0.5 to 0.8 ms at 4K) (BENCHMARKS.md, "In a display loop, on the Mac"). The loop cost emberwave 0.2 to 0.5 ms and HEVC nothing, and did not keep the GPU's clock up. The time to the screen was 17 to 21 ms for all ways and does not tell them apart.
+
+**After the review.** The first runs (in BENCHMARKS.md) were made before it. It found the mean time to the screen of the 60 Hz loop dependent on the run's start phase (the stream and the loop beat every 2 s, so a run times whole beats of 119 frames now), the rows without a loop running in an application with no window on the screen (a black window is kept there now), and nothing guarding against a sleeping or locked screen (the displays are kept awake, and a window that never reaches the screen stops the run). The runs are to be repeated with the changed program, which needs the screen unlocked.
+
+**Not done.** The iPhone half of #81, which needs the phone. A real client's pipeline, with its other threads and a heavier draw. HEVC into BGRA in the loop.
+
+**Consequence.** The shared benchmark code gains a `PictureSink` type, a settable `FrameClock.rate`, and entry points that decode a prepared HEVC stream paced with a sink; the iOS app's behaviour is unchanged. The script's deployment target is macOS 14.

BENCHMARKS.md

@@ -245,7 +245,7 @@ Two more runs with `EMBERWAVE_BENCH_SPIN=1`, at 60 frames a second, the CPU awak
   - at FP32: 1.47 to 1.48 ms against 0.38 to 0.39 ms at 1080p, and 2.31 to 2.63 ms against 0.82 ms at 4K.
 
   That is 2.8 to 3.9 times as long, and its GPU time grows 2.1 to 3.2 times. The slowdown belongs to wavelet decoding on this GPU at this cadence, not to emberwave. On the same streams at that cadence, emberwave's decodes into textures still took 1.3 to 1.5 times as long as upstream's at FP32, more than frame after frame (issue #86). Decision D48 in [`ARCHITECTURE.md`](ARCHITECTURE.md) splits that time up: about 1 ms of it is Metal's own at that cadence, which any decoder pays.
-- **What a client sees.** A client draws every display refresh, and its network threads wake between frames, so it lies between these cadences. On this Mac, emberwave's decoding is quicker than HEVC's only when the GPU is kept busy. Issue #81 is to time decodes within a client's display loop.
+- **What a client sees.** A client draws every display refresh, and its network threads wake between frames, so it lies between these cadences. On this Mac, emberwave's decoding is quicker than HEVC's only when the GPU is kept busy. [In a display loop](#in-a-display-loop-on-the-mac) times that on this Mac.
 - **HEVC's decoding grows with the bytes.** At emberwave's bytes per frame it took 2.4 to 4.7 times as long as at Moonlight's rates, 12.3 to 12.7 ms at 4K. Raising HEVC's bit rate costs the client time.
 - **After a loss**, an HEVC client needs a keyframe: 600 to 730 kB at 4K, which took 5.6 to 7.8 ms to decode. Every PyroWave frame is one.
 - **BGRA** is VideoToolbox's colour conversion, which Moonlight does not use: it draws NV12. Into RGB, emberwave took 27 to 49 % less than HEVC into BGRA at 60 frames a second.
@@ -296,10 +296,45 @@ The two runs' means differed by 4 % or less, except in five rows. HEVC's encodin
 - NVENC's streams, and AV1, which Apple's devices decode but do not encode (issue #82);
 - 10-bit video;
 - FFmpeg's wrapper around VideoToolbox, through which Moonlight on the Mac decodes;
-- the wait for the display, and power;
+- the wait for the display on the iPhone, and power;
 - that HEVC's decoder leaves the GPU free to draw while emberwave's decoding uses it;
 - the pictures' quality. The report's luma PSNR only checks that the stream decodes to the pictures: about 40 dB at Moonlight's rates and 43 dB at emberwave's bytes on both devices. The grain, fresh in every picture, dominates it. [Picture quality](#picture-quality) sets a key frame of HEVC beside emberwave's frames on a picture of a game.
 
+## In a display loop, on the Mac
+
+A client draws the last picture at every refresh of its screen, and its decodes share the GPU with that. `scripts/bench_app_on_mac.sh -display` times decodes beside such a loop (issue #81, decision D72): a borderless window covers a screen while the run lasts, about three minutes, and a `CVDisplayLink` and a `CAMetalLayer` draw the last decoded picture over it at the screen's refresh, one textured triangle with two drawables. Frames arrive on the decode's thread as a stream's do and are decoded into textures, emberwave into Y, Cb and Cr textures and into an RGB texture, HEVC at Moonlight's rate into NV12 textures, 300 frames after at least 30 and a second of warm-up. Each way also runs without the loop in the same run. The loop is a 60 Hz one, every second refresh of the Mac's 120 Hz screen, and a 120 Hz one at every refresh. The stream arrives at 59.5 frames a second, a little under 60, so that its frames drift across the refreshes as a stream's do against a display: frames locked to the refresh would all take one latency, whichever phase they happened to have.
+
+Measured on 2026-11-06 from 18:30 to 18:36, with the first version of the loop (see Limits), on the Apple M3 Max of the tables above, macOS 27.0.1, on mains power, in two runs one after the other, with the library of release 0.18.1 (the code of `main` at `d6bc2b2`). The Mac was not quiet: its load average was 4 to 7 with other programs running. Mean milliseconds per frame, from a frame's arrival until its picture is in textures, the lowest and the highest of the two runs:
+
+| Decode | no display loop | loop at 60 Hz | loop at 120 Hz |
+|---|---|---|---|
+| 1080p, emberwave into Y, Cb, Cr textures | 1.64 to 1.80 | 1.93 to 2.04 | 1.84 to 2.08 |
+| 1080p, HEVC into NV12 textures | 2.11 to 2.14 | 2.14 to 2.17 | 2.07 to 2.13 |
+| 1080p, emberwave into an RGB texture | 2.06 to 2.19 | 2.39 to 2.61 | 2.35 to 2.53 |
+| 1440p, emberwave into Y, Cb, Cr textures | 2.12 to 2.17 | 2.22 to 2.40 | 2.22 to 2.24 |
+| 1440p, HEVC into NV12 textures | 2.50 to 2.61 | 2.68 to 2.73 | 2.63 to 2.77 |
+| 1440p, emberwave into an RGB texture | 2.80 to 2.90 | 2.64 to 3.06 | 3.08 to 3.26 |
+| 4K, emberwave into Y, Cb, Cr textures | 3.01 to 3.03 | 3.25 to 3.27 | 3.54 to 3.55 |
+| 4K, HEVC into NV12 textures | 4.02 to 4.05 | 4.00 to 4.04 | 4.03 to 4.07 |
+| 4K, emberwave into an RGB texture | 3.91 to 4.26 | 4.29 to 4.36 | 4.51 to 4.70 |
+
+And from a frame's arrival until the display presented it (the `presentedTime` of the drawable that first drew it), mean milliseconds, the lowest and the highest of the two runs:
+
+| To the screen | loop at 60 Hz | loop at 120 Hz |
+|---|---|---|
+| 1080p, emberwave, Y, Cb, Cr | 17.50 to 19.55 | 17.77 to 18.48 |
+| 1080p, HEVC, NV12 | 17.30 to 18.44 | 18.02 to 18.20 |
+| 4K, emberwave, Y, Cb, Cr | 18.80 to 19.58 | 19.46 to 19.89 |
+| 4K, HEVC, NV12 | 19.18 to 19.53 | 20.02 to 20.14 |
+
+- **Into the planes, emberwave decodes quicker than HEVC in the loop too.** At 4K it took 3.25 to 3.55 ms against 4.00 to 4.07 ms, at 1440p 2.22 to 2.40 against 2.63 to 2.77, at 1080p 1.84 to 2.08 against 2.07 to 2.17: a gap of 0 to 0.3 ms at 1080p, 0.3 to 0.5 ms at 1440p and 0.5 to 0.8 ms at 4K, run by run. The runs of release 0.11.0 at 60 frames a second, in the section above, had the two about even or HEVC ahead, 3.77 to 4.04 against 3.71 to 3.85 ms at 4K; the library has changed since (decisions D52 to D54 and the paced work of #86).
+- **The loop costs emberwave a little and HEVC nothing.** Beside the loop emberwave's decode into the planes took 0.2 to 0.4 ms longer at 1080p and 0.2 to 0.5 ms at 4K; HEVC's, on the media engine, did not move. The loop did not keep the GPU's clock up: the paced decodes did not get quicker beside it, they shared the GPU with its draw (a first probe, with a second queue blitting a texture, said the same, #81).
+- **The way to the screen does not tell the two apart.** Every way took 17 to 21 ms to reach the screen, the ways within about 2 ms of each other and well within the spread of one run, whose 99th percentiles were 22 to 40 ms. The picture waits for the next refresh, and the display takes a further two or so. A decode that is up to 0.8 ms quicker is a small part of that.
+- **Frames that no refresh drew.** The 60 Hz loop, at every second refresh of a 120 Hz screen, skipped 6 to 24 of 300 frames for each of the ways, as the drift between a 59.5 Hz stream and a 60 Hz loop puts two frames before one refresh now and then; the 120 Hz loop skipped at most 2 of 300.
+- **Into RGB** emberwave took 0.3 to 0.7 ms longer than into the planes at 1080p and about 1 ms longer at 4K, the colour conversion's cost; HEVC into BGRA was not timed in this mode, so there is no like-for-like row.
+- **Limits.** The Mac alone; the iPhone half of #81 needs the phone. The 60 Hz loop is a 60 frames a second client on a 120 Hz screen: the Mac's one 60 Hz screen, an external monitor, took no window (its window was never on screen), and a real 60 Hz display would add up to a refresh more to the way to the screen. The loop draws one triangle with a texture, a client draws more, and a streaming client has other threads busy. The Mac was not quiet.
+- **What a review found, after these runs.** (1) The mean time to the screen of the 60 Hz loop depends on the phase where the run happens to start: the stream and the loop beat every 2 s, 119 frames, and 300 frames are two and a half beats, so the mean wait for the next refresh moves by about 1 ms from run to run; differences under 2 ms between the ways cannot be read from that column. The decode times do not depend on it. (2) The rows without a loop ran in an application with no window on the screen, which the system may throttle after some seconds; none of those rows looks inflated, but the loop's cost may be a little larger than shown. (3) The count of late calls of the display link is of the order of an idle link's own jitter, and is left out. The program has changed since: it times whole beats (357 frames, 119 with `-short`), keeps a black window on the screen for the rows without a loop, keeps the displays awake and stops where the window never reaches the screen, and counts frames the display did not present. The runs are to be repeated with it; the Mac's screen was locked when it was written.
+
 ## Planes stored into textures
 
 Since decision D52, on Metal a decode into Y, Cb and Cr textures has the inverse transform's last levels store the planes into them, where it used to pack them into a buffer and copy them in (D41). Measured on 2026-10-29 from 19:00 to 19:20 on the Apple M3 Max of the tables above, macOS 27.0.1, on mains power, the code of `main` at `5f69efb` against that of [#103](https://github.com/with-developer/emberwave/pull/103). Nobody used the Mac, but a screen recorder and other programs ran.

README.ko.md

@@ -148,7 +148,7 @@ iPhone 17의 Safari에서는 [벤치마크 페이지](https://with-developer.git
 
 앞으로 할 일입니다. 준비되는 대로 각각 릴리스합니다.
 
-- **실사용 환경에서 측정.** 클라이언트의 화면 출력 루프 안에서 디코딩해, 화면에 나오기까지의 시간을 Mac과 iPhone에서 잽니다([#81](https://github.com/with-developer/emberwave/issues/81)). NVENC로 만든 HEVC와 AV1 스트림을 emberwave 스트림과 나란히 디코딩하고, 움직이는 게임 영상의 화질을 같은 비트레이트에서 예측 프레임을 쓰는 HEVC와 비교합니다([#82](https://github.com/with-developer/emberwave/issues/82)). 이 비교로 색차 대역 가중치를 바꿀지도 정합니다(결정 D67). 스트림 속도에서 4K는 upstream의 FP32 디코더를 따라잡았지만 1080p는 아직 0.15~0.3 ms 느리며, 이 차이를 줄입니다([#150](https://github.com/with-developer/emberwave/issues/150)).
+- **실사용 환경에서 측정.** 클라이언트의 화면 출력 루프 안에서 디코딩해, 화면에 나오기까지의 시간을 Mac과 iPhone에서 잽니다([#81](https://github.com/with-developer/emberwave/issues/81), Mac 쪽은 끝났고 결정 D72에 기록했습니다). NVENC로 만든 HEVC와 AV1 스트림을 emberwave 스트림과 나란히 디코딩하고, 움직이는 게임 영상의 화질을 같은 비트레이트에서 예측 프레임을 쓰는 HEVC와 비교합니다([#82](https://github.com/with-developer/emberwave/issues/82)). 이 비교로 색차 대역 가중치를 바꿀지도 정합니다(결정 D67). 스트림 속도에서 4K는 upstream의 FP32 디코더를 따라잡았지만 1080p는 아직 0.15~0.3 ms 느리며, 이 차이를 줄입니다([#150](https://github.com/with-developer/emberwave/issues/150)).
 - **1.0을 향해: 클라이언트.** emberwave가 코덱으로 남도록, 브라우저 스트리밍은 별도 저장소에서 만듭니다. 호스트 화면을 GPU에서 인코딩해 로컬 네트워크로 보내고, 브라우저 탭에서 WebGPU로 디코딩합니다. 지연 시간부터 잽니다. 그다음에는 PyroWave로 스트리밍하는 호스트들에 emberwave 인코더를 제안합니다. 1.0이 되려면 안정된 C·Rust API, 실제로 쓰이는 클라이언트 하나, 그리고 프레임이 실제 스트림의 간격으로 들어올 때 업스트림만큼 빠른 디코딩이 필요합니다.
 
 ## 저장소 구성

README.md

@@ -148,7 +148,7 @@ Done: the milestones to the first release, M0 to M5 ([#1](https://github.com/wit
 
 Next, each in a release when it is ready:
 
-- **Measured in real use.** Decoding inside a client's display loop, timed to the screen, on a Mac and an iPhone ([#81](https://github.com/with-developer/emberwave/issues/81)); NVENC's HEVC and AV1 streams decoded beside emberwave's, and the quality of moving game footage against HEVC with predicted frames at the same bit rate, which also tells whether chroma's band weights should change ([#82](https://github.com/with-developer/emberwave/issues/82), decision D67); and 1080p at a stream's pace, still 0.15 to 0.3 ms behind upstream's FP32 decoder where 4K has caught up ([#150](https://github.com/with-developer/emberwave/issues/150)).
+- **Measured in real use.** Decoding inside a client's display loop, timed to the screen, on a Mac and an iPhone ([#81](https://github.com/with-developer/emberwave/issues/81); the Mac's half is done, decision D72); NVENC's HEVC and AV1 streams decoded beside emberwave's, and the quality of moving game footage against HEVC with predicted frames at the same bit rate, which also tells whether chroma's band weights should change ([#82](https://github.com/with-developer/emberwave/issues/82), decision D67); and 1080p at a stream's pace, still 0.15 to 0.3 ms behind upstream's FP32 decoder where 4K has caught up ([#150](https://github.com/with-developer/emberwave/issues/150)).
 - **Toward 1.0, clients.** Streaming to a browser, in a repository of its own so that emberwave stays a codec: a host's screen encoded on its GPU, sent over the local network and decoded with WebGPU in a tab, its latency measured first. After that, emberwave's encoder offered to the hosts that stream PyroWave. 1.0 needs stable C and Rust APIs, one client in real use, and decoding at a stream's pace level with upstream's.
 
 ## Repository layout

crates/emberwave-ffi/examples/ios/EmberwaveBench/Benchmark.swift

@@ -53,6 +53,23 @@ struct Stats: CustomStringConvertible {
     }
 }
 
+/// How a decoded picture sits in Metal textures.
+enum PictureLayout {
+    /// One texture of colours.
+    case rgb
+    /// Y, Cb and Cr, a texture each: emberwave's plane textures.
+    case planes
+    /// Y, and Cb and Cr interleaved: VideoToolbox's NV12.
+    case nv12
+}
+
+/// Takes a picture the moment its decode is done: when its frame arrived on
+/// `now()`'s clock, its textures, and whether the frame is one of those
+/// timed, not the warm-up's. A display loop draws what it is given.
+typealias PictureSink = (
+    _ arrived: Double, _ layout: PictureLayout, _ textures: [MTLTexture], _ timed: Bool
+) -> Void
+
 /// A picture size and the bytes a frame of it gets: those of the native
 /// benchmarks (BENCHMARKS.md) and of the browser's benchmark page.
 struct PictureSize {
@@ -97,13 +114,22 @@ struct FrameClock {
     private var due: Double?
     private static let timers = DispatchQueue(label: "frame clock", qos: .userInteractive)
     static let spins = ProcessInfo.processInfo.environment["EMBERWAVE_BENCH_SPIN"] == "1"
+    /// Frames a second. A display-loop run detunes it a little from the
+    /// stream's 60, so that the frames drift across the display's refreshes
+    /// as a stream's do, where frames locked to the refresh would all have the
+    /// same latency, one of many.
+    let rate: Double
+
+    init(rate: Double = Double(streamFrameRate)) {
+        self.rate = rate
+    }
 
     mutating func wait() {
         guard let previous = due else {
             due = now()
             return
         }
-        let next = max(previous + 1000 / Double(streamFrameRate), now())
+        let next = max(previous + 1000 / rate, now())
         due = next
         let delay = next - now()
         guard delay > 0 else {