diff --git a/LICENSE b/LICENSE index ee95746..068d00f 100644 --- a/LICENSE +++ b/LICENSE @@ -75,5 +75,5 @@ The **licensor** is the individual or entity offering these terms, and the **sof Required Notice: Copyright 2026 TUNA-NOPE (https://github.com/TUNA-NOPE) Scope: these terms cover macbox's own files. The files in patches/ modify Reims vGPU and metal2vulkan, -which are GPL-2.0-or-later OR LGPL-3.0-or-later, and are offered under those same terms -(patches/README.md). The projects macbox builds and downloads keep their own licenses (CREDITS.md). +which are GPL-2.0-or-later OR LGPL-3.0-or-later, and are offered under those same terms; those in +patches/qemu modify QEMU and are GPL-2.0-or-later (patches/README.md). The projects macbox builds and downloads keep their own licenses (CREDITS.md). diff --git a/README.md b/README.md index 5770fe5..1bce720 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,10 @@ own. One command, `macbox`, sets it up and runs it. **Website and guides:** [tuna-nope.github.io/macbox](https://tuna-nope.github.io/macbox/): macOS, Xcode and the iOS Simulator on Linux, Expo apps, the GPU path, and AI coding agents with T3 Code. -![The macOS desktop in macbox, with two iOS 26.5 Simulators: a Liquid Glass benchmark at 1x and Safari](docs/images/hero.jpg) +![The macOS 26 desktop in macbox on Linux: Simulator.app shows an iPhone 17 Pro running Nimbus, a weather app, next to Nimbus's website in Safari](docs/images/hero.jpg) -macOS 26 Tahoe's desktop in macbox on an Intel iGPU, with two of its iOS 26.5 Simulators in front -(their screens composited onto the desktop's screenshot): GlassBench, a Liquid Glass load, and -Safari. +macOS 26 Tahoe's desktop in macbox on an Intel iGPU, captured as it is: Simulator.app with an +iPhone 17 Pro on iOS 26.5 running Nimbus, a weather app, beside its website in Safari. ``` Linux host (KVM + a Vulkan GPU + Docker) @@ -28,8 +27,8 @@ Linux host (KVM + a Vulkan GPU + Docker) - **macOS draws on your GPU.** [Reims vGPU](https://github.com/steelbrain/reims-vgpu) gives QEMU Apple's paravirtual GPU, and macOS's own driver talks to it, so Metal works with no guest driver. [metal2vulkan](https://github.com/steelbrain/metal2vulkan) runs Metal's work on Vulkan. - macbox builds both, with patches that roughly doubled the iOS Simulator's frame rate on the - GPU it was measured on (see [Performance](#performance)). + macbox builds both, with patches that took a 3x iOS Simulator from 17 to 50 fps on the GPU it + was measured on, and to 70 with the experimental Coherent switch (see [Performance](#performance)). - **The screen** opens in a window on your Linux desktop, or in a browser tab (Selkies: H.264 on the GPU, its bitrate following the link). Agents can see it and click and type into it (`macbox screenshot`, `click`, `type`), so an agent can drive the macOS installer too. @@ -91,19 +90,32 @@ floating glass toolbar, counted at the simulator's own screen (frames per second - iOS Simulator frame rate, Liquid Glass benchmark: 1x 53.5 to 94.2 fps, 2x 33.5 to 64.2, 3x 17.2 to 39.0, Reims as upstream shipped it vs the macbox build + iOS Simulator at 3x, Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 with the cohwl build, 70.0 with Coherent on -| | Reims 2ce3209, as upstream shipped it on 2026-10-26 | macbox build | -|---|---|---| -| iOS Simulator 1x (402x874), glass | 53.5 | **95.8 / 92.5** | -| iOS Simulator 2x (804x1748), glass | 33.5 | **67.8 / 60.5** | -| iOS Simulator 3x (1206x2622, an iPhone's pixels), glass | 17.2 | **38.5 / 39.5** | -| Safari at 3x, a page with a blurred floating bar | n/a | 66.4 / 64.2 | -| macOS desktop, Genie minimize at 3440x1440 (frames per animation) | ~12 | 19.5 | - -The GPU is the limit: a 3x simulator keeps this iGPU 96 to 98% busy. On a faster GPU every number -moves up. [docs/performance.md](docs/performance.md) has what the patches do, the method, what +| iOS Simulator at 3x (1206x2622, an iPhone's pixels), glass | fps | +|---|---| +| Reims 2ce3209, as upstream shipped it on 2026-10-26 | 17.2 | +| macbox, 2026-10-28: rendering guest-RAM textures in place | 38.5 / 39.5 | +| **macbox build now** (cohwl, 2026-11-05) | **50.6** (p95 21 ms) | +| macbox build with `MACBOX_COHERENT=on` (experimental) | **70.0** (p95 18 ms) | + +| | Reims 2ce3209 | macbox, 2026-10-28 | macbox now, Coherent on | +|---|---|---|---| +| iOS Simulator 1x (402x874), glass | 53.5 | 95.8 / 92.5 | not measured yet | +| iOS Simulator 2x (804x1748), glass | 33.5 | 67.8 / 60.5 | not measured yet | +| macOS desktop, dragging a window (presents/s) | | 105 (Coherent off) | 130 | +| macOS desktop, Safari scrolling (presents/s) | | 75–79 (Coherent off) | 110 | +| macOS desktop, Genie minimize at 3440x1440 (frames per animation) | ~12 | 19.5 | 29.5 / 30.5 | + +Coherent keeps the GPU's image of the simulator's and the desktop's surfaces ahead of guest RAM, +and syncs a page only when the guest's CPU touches it: the copies that were left are gone, and the +GPU drops from 97% to 70% busy at 3x. It is off by default for now: it hung the GPU once on its +first day of daily use (the next boot came up without it, as designed), and the cause isn't +settled. [docs/performance.md](docs/performance.md#coherent-experimental-off-by-default) has the details. + +The GPU is the limit: without Coherent a 3x simulator keeps this iGPU 96 to 98% busy. On a faster +GPU every number moves up. [docs/performance.md](docs/performance.md) has what the patches do, the method, what didn't help, and the known limits (an idle macOS display gets no vblank from its driver, so an animation starts about 0.1 s late: [reims-vgpu#141](https://github.com/steelbrain/reims-vgpu/issues/141)). @@ -177,7 +189,7 @@ issue or PR. See [CONTRIBUTING.md](CONTRIBUTING.md) for a few simple guidelines. | [docs/agents.md](docs/agents.md) | the agent tools, the skill, the instructions to give agents | | [docs/troubleshooting.md](docs/troubleshooting.md) | when something doesn't work | | [guest/README.md](guest/README.md) | what runs inside macOS | -| [patches/README.md](patches/README.md) | the patch series on Reims vGPU and metal2vulkan | +| [patches/README.md](patches/README.md) | the patch series on Reims vGPU, metal2vulkan and QEMU | | [site/](site/README.md) | the website's source: the guides at https://tuna-nope.github.io/macbox/guides/ | ## Credits @@ -194,7 +206,8 @@ purpose: yourself, a hobby, research, teaching, a school, a charity, a public bo needs the author's permission**: open an issue to ask. The patches in `patches/` modify Reims vGPU and metal2vulkan, and stay under those projects' terms, -`GPL-2.0-or-later OR LGPL-3.0-or-later` ([patches/README.md](patches/README.md)). Everything macbox +`GPL-2.0-or-later OR LGPL-3.0-or-later`; those in `patches/qemu` modify QEMU and are +`GPL-2.0-or-later` ([patches/README.md](patches/README.md)). Everything macbox builds or downloads keeps its own license ([CREDITS.md](CREDITS.md)). ## Disclaimer diff --git a/container/build-reims.sh b/container/build-reims.sh index d85fe91..430ebe0 100755 --- a/container/build-reims.sh +++ b/container/build-reims.sh @@ -70,6 +70,8 @@ else git -C "$R" submodule update -q --init vendor/qemu git -C "$R/vendor/qemu" fetch -q origin "$QEMU_REF" 2>/dev/null || true git -C "$R/vendor/qemu" checkout -q -f --detach "$QEMU_REF" + git -C "$R/vendor/qemu" am --abort >/dev/null 2>&1 || true + apply "$R/vendor/qemu" "$MACBOX_REPO/patches/qemu" SUFFIX=-macbox fi COMMIT=$(git -C "$R" rev-parse --short HEAD) diff --git a/docs/architecture.md b/docs/architecture.md index eb6ac3f..d4730ac 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -40,7 +40,7 @@ It runs with seccomp and AppArmor unconfined, because passt needs a user namespa `container/build-reims.sh` (run by `macbox build` in a throwaway container) checks out Reims vGPU, QEMU and metal2vulkan at the commits [versions.env](../versions.env) pins, applies -[patches/](../patches/README.md), points Reims's Cargo at the patched metal2vulkan, and builds +[patches/](../patches/README.md) to each of the three, points Reims's Cargo at the patched metal2vulkan, and builds `reims-ctl`, the library (`cargo xtask dist --platform vulkan`) and QEMU on it. Each QEMU goes into `~/.local/share/reims-ctl/qemu/--macbox` and `qemu/current` moves to it. A running VM keeps its QEMU until it restarts, and old builds stay for going back @@ -59,8 +59,10 @@ and the QEMU command line; `reims-ctl command mac` (inside the container) prints - `--video-acceleration=false` (no VideoToolbox/M2 scaler devices), a borderless window, `REIMS_VGPU_STATS=1` (a line of frame statistics a second in `logs/qemu.log`). - Before each start: `WAYLAND_DISPLAY` (sway's, or the host's), and the GPU tuning: - `M2V_LIVE=$MACBOX_HOME/run/live.conf` with `host_rt_in_place=1`, and `M2V_ARL_FACTS` on an - Arrow Lake GPU ([performance.md](performance.md)). + `M2V_LIVE=$MACBOX_HOME/run/live.conf` with `host_rt_in_place=1`, + `REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1`, `M2V_ARL_FACTS` on an Arrow Lake GPU, and with + `MACBOX_COHERENT=on` the census and Coherent switches ([performance.md](performance.md)). + The disk's options in `extra_args` are written again, so a changed `MACBOX_DISK_CACHE` applies. The system disk starts on SATA, because macOS's installer may not find a disk on NVMe, and moves to NVMe with `macbox disk nvme` once macOS is installed. @@ -130,7 +132,7 @@ versions.env the pinned upstream commits docker/ the image (Dockerfile, sway.conf) container/ scripts that run inside the container: build-reims.sh, display-up.sh, stream-up.sh lib/vmctl.py screenshots, mouse and keyboard through QMP -patches/ the series on Reims vGPU and metal2vulkan +patches/ the series on Reims vGPU, metal2vulkan and QEMU guest/ everything copied into macOS (macbox-guest and its tools) sims/ the agent tools on the host (mac-sim, sim-fps, android-sim) and their user units skills/ the simulators, phone-builds and macbox (setup) skills diff --git a/docs/images/fps-dark.svg b/docs/images/fps-dark.svg index 4285b6a..db56b45 100644 --- a/docs/images/fps-dark.svg +++ b/docs/images/fps-dark.svg @@ -1,39 +1,34 @@ - - -iOS Simulator frame rate, Liquid Glass benchmark + + +iOS Simulator at 3x (1206×2622), Liquid Glass benchmark Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better. - -Reims vGPU as upstream shipped it (2ce3209) - -macbox build - -0 - -20 - -40 - -60 - -80 - -100 -frames per second - -60 fps -1x · 402×874 - -53.5 - -94.2 -2x · 804×1748 - -33.5 - -64.2 -3x · 1206×2622 - -17.2 - -39.0 + +0 + +20 + +40 + +60 + +80 +frames per second + +60 fps +Upstream +Reims 2ce3209, 2026-10-26 + +17.2 +macbox, Oct 28 +in place rendering + +39.0 +macbox now +cohwl, the default + +50.6 ++ Coherent +experimental, off by default + +70.0 diff --git a/docs/images/fps-light.svg b/docs/images/fps-light.svg index f8706e6..6eeefda 100644 --- a/docs/images/fps-light.svg +++ b/docs/images/fps-light.svg @@ -1,39 +1,34 @@ - - -iOS Simulator frame rate, Liquid Glass benchmark + + +iOS Simulator at 3x (1206×2622), Liquid Glass benchmark Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better. - -Reims vGPU as upstream shipped it (2ce3209) - -macbox build - -0 - -20 - -40 - -60 - -80 - -100 -frames per second - -60 fps -1x · 402×874 - -53.5 - -94.2 -2x · 804×1748 - -33.5 - -64.2 -3x · 1206×2622 - -17.2 - -39.0 + +0 + +20 + +40 + +60 + +80 +frames per second + +60 fps +Upstream +Reims 2ce3209, 2026-10-26 + +17.2 +macbox, Oct 28 +in place rendering + +39.0 +macbox now +cohwl, the default + +50.6 ++ Coherent +experimental, off by default + +70.0 diff --git a/docs/images/hero.jpg b/docs/images/hero.jpg index 18ee240..7dc4f60 100644 Binary files a/docs/images/hero.jpg and b/docs/images/hero.jpg differ diff --git a/docs/images/make-chart.py b/docs/images/make-chart.py index 7b37caf..51ea13a 100644 --- a/docs/images/make-chart.py +++ b/docs/images/make-chart.py @@ -1,59 +1,54 @@ #!/usr/bin/env python3 """Writes fps-light.svg and fps-dark.svg, the README's frame-rate chart (docs/performance.md's numbers). -Colors: the validated categorical slots 1 and 2 (blue, orange) for each mode's surface.""" +Colors: the validated categorical slots 1 and 2 (blue, orange) for each mode's surface; the +experimental bar is the macbox blue at reduced opacity, and its label says so.""" import os -ROWS = [ # label, before (Reims 2ce3209), after (macbox build, mean of two runs) - ("1x · 402×874", 53.5, (95.8 + 92.5) / 2), - ("2x · 804×1748", 33.5, (67.8 + 60.5) / 2), - ("3x · 1206×2622", 17.2, (38.5 + 39.5) / 2), +ROWS = [ # label, sublabel, fps (3x, mean of the valid rounds), kind + ("Upstream", "Reims 2ce3209, 2026-10-26", 17.2, "before"), + ("macbox, Oct 28", "in place rendering", (38.5 + 39.5) / 2, "after"), + ("macbox now", "cohwl, the default", 50.6, "after"), + ("+ Coherent", "experimental, off by default", 70.0, "exp"), ] MODES = { "light": dict(surface="#fcfcfb", text="#0b0b0b", text2="#52514e", grid="#e4e3df", before="#eb6834", after="#2a78d6"), "dark": dict(surface="#1a1a19", text="#f0efec", text2="#c3c2b7", grid="#3a3a37", before="#d95926", after="#3987e5"), } -W, H = 900, 370 -X0, X1, MAX = 190, 840, 100 # bar area and its scale (fps) -BAR, GAP, GROUP = 22, 2, 30 # bar thickness, gap between a group's bars, space between groups -TOP = 112 +W, H = 900, 320 +X0, X1, MAX = 230, 840, 80 # bar area and its scale (fps) +BAR, GAP = 30, 16 # bar thickness, space between bars +TOP = 96 FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif" def x(v): return X0 + (X1 - X0) * v / MAX -def bar(y, v, color): # 4 px rounded at the data end only, square at the baseline +def bar(y, v, color, opacity=1.0): # 4 px rounded at the data end only, square at the baseline w, r = x(v) - X0, 4 return (f'') + f'Q{X0 + w},{y + BAR} {X0 + w - r},{y + BAR} H{X0} Z" fill="{color}" fill-opacity="{opacity}"/>') def svg(m): c = MODES[m] o = [f'', f'', - f'iOS Simulator frame rate, Liquid Glass benchmark', + f'iOS Simulator at 3x (1206×2622), Liquid Glass benchmark', f'Frames per second on an Intel Arrow Lake iGPU (Core Ultra 7 265). Higher is better.'] - # legend - lx = 24 - for name, col in (("Reims vGPU as upstream shipped it (2ce3209)", c["before"]), ("macbox build", c["after"])): - o.append(f'') - o.append(f'{name}') - lx += 18 + len(name) * 7.0 + 26 - # grid and axis - bottom = TOP + len(ROWS) * (2 * BAR + GAP) + (len(ROWS) - 1) * GROUP - for v in (0, 20, 40, 60, 80, 100): + bottom = TOP + len(ROWS) * BAR + (len(ROWS) - 1) * GAP + for v in (0, 20, 40, 60, 80): o.append(f'') o.append(f'{v}') o.append(f'frames per second') # the 60 fps target o.append(f'') - o.append(f'60 fps') + o.append(f'60 fps') y = TOP - for label, before, after in ROWS: - o.append(f'{label}') - for v, col in ((before, c["before"]), (after, c["after"])): - o.append(bar(y, v, col)) - o.append(f'{v:.1f}') - y += BAR + GAP - y += GROUP - GAP + for label, sub, v, kind in ROWS: + col = c["before"] if kind == "before" else c["after"] + o.append(f'{label}') + o.append(f'{sub}') + o.append(bar(y, v, col, 0.55 if kind == "exp" else 1.0)) + o.append(f'{v:.1f}') + y += BAR + GAP o.append("") return "\n".join(o) + "\n" diff --git a/docs/performance.md b/docs/performance.md index 9755f02..73b902c 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -10,29 +10,60 @@ work the translation adds per frame (Metal → Vulkan). Before macbox's patches, most of that added work was copies: macOS's framebuffer and the simulator's display surface lie in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% -with the desktop. Patch 0004 renders those textures in place ([patches/README.md](../patches/README.md)). +with the desktop. Patch 0004 renders those textures in place, and Coherent (patches 0009 onwards, +with QEMU's census) removes most of the copies left ([patches/README.md](../patches/README.md)). ## Measurements The reference host: Intel Core Ultra 7 265 (Arrow Lake iGPU), 128 GB DDR5 at 4400 MT/s, Mesa 26.2.4, a NVMe disk; the VM had 8 vCPUs and 12 GB, macOS 26.7.1, iOS 26.5, one simulator booted at a time. -iOS Simulator, GlassBench (a table scrolling under Liquid Glass bars and a floating glass toolbar), -frames per second counted at the simulator's own screen, no encoding; two runs each: +### iOS Simulator at 3x, step by step -| | Reims 2ce3209 (2026-10-26) | Reims 7198fe6 | 7198fe6 + facts + colour writes | **macbox build** | +GlassBench (a table scrolling under Liquid Glass bars and a floating glass toolbar) on an iPhone 17 +Pro at 3x (1206x2622), frames per second counted at the simulator's own screen: + +| Build | fps | GPU | +|---|---|---| +| Reims 2ce3209 as upstream shipped it (2026-10-26) | 17.2 | | +| macbox, 2026-10-28 (in place rendering, 0004) | 38.5 / 39.5 | 25.0 ms/frame | +| macbox cohwl, Coherent off (flat32) | **50.6** (p95 21.4 ms, p99 25.8 ms) | 97% busy | +| macbox cohwl, `MACBOX_COHERENT=on` | **70.0** (p95 17.7 ms, p99 22.0 ms); 63–66 in a later run | 70% busy | + +The last two rows are one A/B in one window on 2026-11-05: fresh headless boots, a quiet gate +before each round (guest background CPU and `mediaanalysisd` low), means of the valid rounds. The +first two are earlier days, on the same host but not in the same window. With Coherent, QEMU's CPU +use rises by about 1.6 cores (333% to 491%): the GPU waits less, the CPU traps more. + +flat32 also has an A/B of its own against the build before it: 41.6 → 48–50 fps at 3x. + +### Earlier measurements, the 2026-10-28 build + +| | Reims 2ce3209 (2026-10-26) | Reims 7198fe6 | 7198fe6 + facts + colour writes | macbox, 2026-10-28 | |---|---|---|---|---| -| 1x (402x874) | 53.5 | 57.0 | 63.7 | **95.8 / 92.5** (GPU 8.0 ms/frame) | -| 2x (804x1748) | 33.5 | 37.6 | 38.8 | **67.8 / 60.5** (14.1–14.9 ms) | -| 3x (1206x2622) | 17.2 | 19.2 | 21.9 | **38.5 / 39.5** (25.0 ms) | -| Safari at 3x, blurred floating bar | n/a | 39.5 | 44.5 | **66.4 / 64.2** | +| 1x (402x874) | 53.5 | 57.0 | 63.7 | 95.8 / 92.5 (GPU 8.0 ms/frame) | +| 2x (804x1748) | 33.5 | 37.6 | 38.8 | 67.8 / 60.5 (14.1–14.9 ms) | +| 3x (1206x2622) | 17.2 | 19.2 | 21.9 | 38.5 / 39.5 (25.0 ms) | +| Safari at 3x, blurred floating bar | n/a | 39.5 | 44.5 | 66.4 / 64.2 | + +1x and 2x haven't been measured again on cohwl. + +### The macOS desktop -macOS desktop at 3440x1440, a Genie minimize of a 2400x1100 window: about 12 frames per -animation before, 19.5 with the macbox build (a 0.5 s animation at 60 Hz would be 30). +| 3440x1440 desktop | Coherent off | Coherent on | +|---|---|---| +| dragging a 600x400 window (presents per second) | 105–107 | 129–130 | +| Safari scrolling a local page (presents per second) | 75–79 | 110 | +| Genie minimize of a 2400x1100 window (frames per animation; ~12 upstream) | 19.5 (the 10-28 build) | 29.5 / 30.5 | -Disk, inside macOS, emulated SATA → NVMe: a warm 4K random read 0.54 → 0.36 ms, an app's process -start 250–430 → 140–250 ms. Moving the VM's disk from a ZFS SATA SSD to an XFS NVMe halved the cold -read latency again. +The first two rows are the same A/B as above. The Genie number with Coherent comes from a smoke +check, not a matched A/B. A 0.5 s animation at 60 Hz would be 30 frames. + +### Disk + +Inside macOS, emulated SATA → NVMe: a warm 4K random read 0.54 → 0.36 ms, an app's process start +250–430 → 140–250 ms. Moving the VM's disk from a ZFS SATA SSD to an XFS NVMe halved the cold read +latency again. How they were measured: - `simfps` (in `guest/bench`, if present) counts the frames a simulator's screen produces through @@ -55,7 +86,9 @@ How they were measured: blur work. It changes nothing inside the simulator. - **CPU pinning** (`MACBOX_CPUSET`): on a hybrid CPU, keep QEMU on the performance cores. - **The disk:** NVMe in the VM (`macbox disk nvme`) and the VM's disk on an SSD or NVMe on the host. - `MACBOX_DISK_CACHE=unsafe` is faster still, at the risk spelled out in the config. + The default `MACBOX_DISK_CACHE=none`: with `writeback`, macOS's NVMe driver panicked under heavy + swap (an `IONVMeController` flush timeout), and `none` passed the same 11 GB test. + `unsafe` is faster still, at the risk spelled out in the config. ## GPU tuning switches @@ -65,6 +98,7 @@ With the macbox build (`MACBOX_GPU_TUNING=auto`, the default), `macbox up` sets: |---|---| | `M2V_LIVE=$MACBOX_HOME/run/live.conf` | metal2vulkan's live switches (patch 0003). The file holds `host_rt_in_place=1` | | `M2V_ARL_FACTS=<11 facts>` | only on an Arrow Lake GPU (PCI ids 7d67, 7dd1, 7d41, 7d51, 7d45) | +| `REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1` | skips a new Private texture's first page-up (Reims patch 0013) | `host_rt_in_place` is read when a texture is made: change it in `live.conf`, then remake the surfaces (change the resolution, or restart the VM). `MACBOX_GPU_TUNING=off` turns all of it off. @@ -73,6 +107,26 @@ On another GPU, in place rendering still runs its device check per memory kind a the check fails; it has only been measured on Arrow Lake. Measure your own GPU before and after: the facts list is specific to Arrow Lake and is never set elsewhere. +### Coherent (experimental, off by default) + +`MACBOX_COHERENT=on` in the config, then `macbox down && macbox up`. macbox then also sets +`REIMS_VGPU_CENSUS=$MACBOX_HOME/run/coherent`, `REIMS_VGPU_COHERENT=1` and a report a minute, and +adds `surface_lazy=1`, `surface_lazy_scoped=1` and `surface_coherent=1` to `live.conf`. +`MACBOX_COHERENT=off` removes them all again. + +It is off by default because it isn't settled yet: +- **One GPU hang with it on.** On 2026-11-05, the day it went into daily use, i915 + reset the GPU's render engine and QEMU stopped. The guard did its job: the next boot came up without + coherent pages. A later hang with Coherent off, on a newer development build, has the same error + code, so the cause is open, not pinned on Coherent. +- **A write-loss risk the counters flagged once,** during a test that killed Safari mid-frame + (`overwrite_risk` 1 in the census report), and that hasn't been reproduced or ruled out since. + +If a coherent run crashes, `$MACBOX_HOME/run/coherent/coherent.running` stays behind and every later +start boots without coherent pages. `macbox coherent status` says so; `macbox coherent reset` +(VM stopped) allows the next try. It needs a kernel with `UFFD_USER_MODE_ONLY` (Linux 5.11 or +later), which works with `vm.unprivileged_userfaultfd=0`. + ## Tried, no gain | | Result | @@ -85,6 +139,14 @@ the facts list is specific to Arrow Lake and is never set elsewhere. | 2 MiB hugepages or transparent hugepages for guest RAM | impossible: Reims maps guest RAM in 16 KiB views | | another vCPU model (`kvm-pv-unhalt`, no IBRS) | within noise | | QEMU's VNC | 2 to 3 frames a second: it reads every frame back from the GPU | +| `surface_lazy` without trapping, or presenting every surface | a stale desktop | +| keeping paged textures' buffers (`REIMS_VGPU_PAGED_BUFFER_CACHE`) | no reuse in practice | +| grouping submissions (`submit_grouped`) | 0.2 ms a frame or less | +| a faster idle tick (`REIMS_VGPU_VBLANK_HZ` 240 vs 480) | no gain; macOS never enables the interrupt it re-raises | +| udmabuf imports, pooled staging buffers | no gain | +| a reaper for deleted textures | Safari 5–8 fps slower, and Genie breaks | +| executing the FIFO from its own thread (fifoexec) | a GPU hang on its first day; rolled back | +| merging adjacent page runs of a mapping | negligible: 49 merges in 629,632 pages of a real trace | ## Known limits @@ -92,7 +154,7 @@ the facts list is specific to Arrow Lake and is never set elsewhere. sends no vblank while nothing is presenting, so the first frame of an animation waits a median 114 ms for display timing (8 ms when something already animates). [steelbrain/reims-vgpu#141](https://github.com/steelbrain/reims-vgpu/issues/141). -- **3x is GPU-bound** at about 35 to 40 fps on this iGPU in heavy Liquid Glass scenes. +- **3x is GPU-bound** at about 50 fps on this iGPU in heavy Liquid Glass scenes (70 with Coherent). - **x86-64 macOS ends with macOS 26.** iOS 27 simulators are arm64 only. - **Recording** uses `simctl io recordVideo`, H.264 in software in the VM: about 50 fps with one simulator animating, 18 each with three. `sim-fps` counts what the simulator wrote, and it writes diff --git a/docs/setup.md b/docs/setup.md index 355069f..5949bf0 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -37,7 +37,8 @@ node), makes an SSH key `~/.ssh/macbox_ed25519`, and adds a `Host macbox` block | `MACBOX_RESOLUTION`, `MACBOX_REFRESH` | 1920x1080, 60 | macOS's screen; more pixels cost GPU time | | `MACBOX_CPUSET` | empty | pin QEMU to some cores, e.g. a hybrid CPU's performance cores (`0-7`) | | `MACBOX_BUILD_CPUSET` | empty | cores for `macbox build`, e.g. the efficiency cores | -| `MACBOX_DISK_CACHE` | writeback | `unsafe` is faster, but a crash of the host itself can corrupt the VM's disk | +| `MACBOX_DISK_CACHE` | none | `writeback` panicked macOS's NVMe driver under heavy swap; `unsafe` is faster, but a crash of the host itself can corrupt the VM's disk. Applies at the next `macbox up` | +| `MACBOX_COHERENT` | off | `on`: Coherent, zero-copy surfaces, about 40% faster at 3x and experimental ([performance.md](performance.md#coherent-experimental-off-by-default)) | | `MACBOX_HOME` | ~/.local/share/macbox | everything macbox keeps | Then: diff --git a/llms.txt b/llms.txt index e5b922f..e80aabe 100644 --- a/llms.txt +++ b/llms.txt @@ -2,7 +2,7 @@ > Run macOS, Xcode and the real iOS Simulator on a Linux host, in Docker, with GPU acceleration (Metal through Reims vGPU and metal2vulkan on the host's Vulkan GPU), for AI coding agents such as Claude Code, Codex and T3 Code. No Mac and no cloud service: one CLI, `macbox`, builds and runs it, and `mac-sim` gives each agent checkout its own iOS Simulator to build, run, tap, screenshot, record and measure at 60 fps. -License: PolyForm Noncommercial 1.0.0 for macbox's own code (noncommercial use; commercial use needs permission); the patches stay GPL-2.0-or-later OR LGPL-3.0-or-later. +License: PolyForm Noncommercial 1.0.0 for macbox's own code (noncommercial use; commercial use needs permission); the patches stay GPL-2.0-or-later OR LGPL-3.0-or-later (GPL-2.0-or-later for the QEMU ones). Requirements: an x86-64 Linux host with KVM, AVX2, an Intel or AMD GPU with a Mesa Vulkan driver, 16 GB RAM or more, about 150 GB of SSD, and Docker. It runs macOS 26 Tahoe, Xcode 26 and iOS 26 simulators, which is the ceiling on x86-64. Apple's license allows macOS only on Apple hardware. @@ -33,7 +33,7 @@ publishing on their behalf. ## How it works - [Architecture](https://github.com/TUNA-NOPE/macbox/blob/main/docs/architecture.md): the container, QEMU and Reims vGPU, the display stream, the network, the guest, the simulator pool -- [Performance](https://github.com/TUNA-NOPE/macbox/blob/main/docs/performance.md): the metal2vulkan patches, measurements, tuning, known limits +- [Performance](https://github.com/TUNA-NOPE/macbox/blob/main/docs/performance.md): the patches (in place rendering, flat32, experimental Coherent), measurements, tuning, known limits - [Patches](https://github.com/TUNA-NOPE/macbox/blob/main/patches/README.md) - [Troubleshooting](https://github.com/TUNA-NOPE/macbox/blob/main/docs/troubleshooting.md) - [Credits](https://github.com/TUNA-NOPE/macbox/blob/main/CREDITS.md) diff --git a/macbox b/macbox index 6a767fa..1f5bfc0 100755 --- a/macbox +++ b/macbox @@ -14,7 +14,8 @@ eval "$_env" : "${MACBOX_SSH_HOST:=macbox}" "${MACBOX_SSH_PORT:=2226}" "${MACBOX_SCREEN_PORT:=6081}" : "${MACBOX_GUEST_USER:=dev}" "${MACBOX_CPUS:=8}" "${MACBOX_MEMORY:=12G}" "${MACBOX_DISK_SIZE:=160G}" : "${MACBOX_RESOLUTION:=1920x1080}" "${MACBOX_REFRESH:=60}" "${MACBOX_CPUSET:=}" -: "${MACBOX_RENDER_NODE:=/dev/dri/renderD128}" "${MACBOX_DISK_CACHE:=writeback}" "${MACBOX_GPU_TUNING:=auto}" +: "${MACBOX_RENDER_NODE:=/dev/dri/renderD128}" "${MACBOX_DISK_CACHE:=none}" "${MACBOX_GPU_TUNING:=auto}" +: "${MACBOX_COHERENT:=off}" : "${MACBOX_SIM_PREFIX:=macbox}" "${MACBOX_POOL:=full slim-1 slim-2 sharp small}" : "${MACBOX_T3_HOST_ID:=$MACBOX_SSH_HOST}" "${MACBOX_T3_SETTINGS:=$HOME/.t3/userdata/settings.json}" DOCKER="${MACBOX_DOCKER:-docker}" @@ -95,13 +96,27 @@ tuning_on() { esac } apply_tuning() { # before each start + local live="$RUN/live.conf" if tuning_on; then - [ -s "$RUN/live.conf" ] || printf '# metal2vulkan live switches (patches/metal2vulkan), re-read every second\nhost_rt_in_place=1\n' >"$RUN/live.conf" - local env=(--env "M2V_LIVE=$RUN/live.conf") + [ -s "$live" ] || printf '# metal2vulkan live switches (patches/metal2vulkan), re-read every second\nhost_rt_in_place=1\n' >"$live" + sed -i '/^surface_\(lazy\|lazy_scoped\|coherent\)=/d' "$live" + local env=(--env "M2V_LIVE=$live" --env REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1) if gpu_is_arl; then env+=(--env "M2V_ARL_FACTS=$ARL_FACTS"); else env+=(--env "M2V_ARL_FACTS="); fi + if [ "$MACBOX_COHERENT" = on ]; then + # QEMU's census traps the guest CPU's touches of the surfaces metal2vulkan renders in place. + # A run that doesn't end well leaves coherent/coherent.running, and every later start then + # boots without Coherent until that file is removed (macbox coherent reset). + mkdir -p "$RUN/coherent" + printf 'surface_lazy=1\nsurface_lazy_scoped=1\nsurface_coherent=1\n' >>"$live" + env+=(--env "REIMS_VGPU_CENSUS=$RUN/coherent" --env REIMS_VGPU_COHERENT=1 --env REIMS_VGPU_CENSUS_FLUSH_MS=60000) + else + env+=(--env REIMS_VGPU_CENSUS= --env REIMS_VGPU_COHERENT= --env REIMS_VGPU_CENSUS_FLUSH_MS=) + fi ctl set "$MACBOX_VM" "${env[@]}" >/dev/null else - ctl set "$MACBOX_VM" --env "M2V_LIVE=" --env "M2V_ARL_FACTS=" >/dev/null + [ -f "$live" ] && sed -i '/^surface_\(lazy\|lazy_scoped\|coherent\)=/d' "$live" + ctl set "$MACBOX_VM" --env "M2V_LIVE=" --env "M2V_ARL_FACTS=" --env REIMS_VGPU_PRIVATE_SKIP_PAGE_UP= \ + --env REIMS_VGPU_CENSUS= --env REIMS_VGPU_COHERENT= --env REIMS_VGPU_CENSUS_FLUSH_MS= >/dev/null fi } @@ -149,10 +164,13 @@ MACBOX_CPUSET=$MACBOX_CPUSET # Host CPUs for macbox build (taskset list, e.g. the efficiency cores); empty: all MACBOX_BUILD_CPUSET=${MACBOX_BUILD_CPUSET:-} MACBOX_RENDER_NODE=${node:-$MACBOX_RENDER_NODE} -# writeback (safe) or unsafe (faster; a crash of the host itself can corrupt the VM's disk) +# none (the default: macOS's NVMe driver panicked under heavy swap with writeback), writeback, or +# unsafe (faster; a crash of the host itself can corrupt the VM's disk) MACBOX_DISK_CACHE=$MACBOX_DISK_CACHE # auto: macbox's metal2vulkan tuning with the macbox build; on; off MACBOX_GPU_TUNING=$MACBOX_GPU_TUNING +# on: Coherent, zero-copy IOSurfaces (docs/performance.md). Experimental: one GPU hang with it on +MACBOX_COHERENT=$MACBOX_COHERENT # iOS Simulator pool: names are " ( (import: &Arc) -> u64 { + Arc::as_ptr(import) as usize as u64 @@ -351,7 +351,7 @@ index 11b17838..330c0690 100644 - range, - ..used.clone() - }); -+ // macbox: a buffer that joined a memory lies in its import but where two ++ // LOCAL (Atlas): a buffer that joined a memory lies in its import but where two + // were made over it at once (the module docs): one placed in another import than + // the memory's key's is apart. + let other = placements.is_some_and(|placements| placements.apart(used.key, key)); @@ -390,7 +390,7 @@ index 11b17838..330c0690 100644 !all.is_empty() } -+/// macbox: `used`, a use added for a key of another import of the same pages where ++/// LOCAL (Atlas): `used`, a use added for a key of another import of the same pages where +/// `other` (the module docs), with `MEMORY_WRITE` among the access types of its reads and of its +/// writes: a barrier between it and any use of the key it is added for then carries +/// `MEMORY_WRITE` on its side, which ANV takes as its full flush, so a write through one import @@ -418,7 +418,7 @@ index 11b17838..330c0690 100644 - /// Each placed key's bytes: their first byte's host address and their length. - at: KeyMap, + /// Each placed key's bytes: their first byte's host address and their length; LOCAL -+ /// (macbox): and the import it lies in ([`import_note`]; 0: not known, apart from every ++ /// (Atlas): and the import it lies in ([`import_note`]; 0: not known, apart from every + /// other). + at: KeyMap, /// Each range of bytes some key lies over, by its first byte's address and its length. @@ -428,13 +428,13 @@ index 11b17838..330c0690 100644 keys: usize, /// Whether its key lies over it still. first: bool, -+ /// macbox: the import its key lies in, kept once the key went: the key's uses still ++ /// LOCAL (Atlas): the import its key lies in, kept once the key went: the key's uses still + /// recorded lie there, and hold the import, whose address no other import takes meanwhile. + import: u64, /// The other ranges that meet it (share a byte with it), each with its key (a range's key - /// is its own while it lives), in the order the lookup by address finds them ([`seat`]). - meets: Vec<((u64, u64), ResourceKey)>, -+ /// is its own while it lives) and macbox that key's import, in the order the lookup by ++ /// is its own while it lives) and LOCAL (Atlas) that key's import, in the order the lookup by + /// address finds them ([`seat`]). + meets: Vec<((u64, u64), ResourceKey, u64)>, } @@ -473,7 +473,7 @@ index 11b17838..330c0690 100644 } } -+/// macbox: whether two imports, as placements note them (0: not known), are apart: ++/// LOCAL (Atlas): whether two imports, as placements note them (0: not known), are apart: +/// either not known, or two. +fn imports_apart(a: u64, b: u64) -> bool { + a == 0 || b == 0 || a != b @@ -489,7 +489,7 @@ index 11b17838..330c0690 100644 - /// Place `key`, lying over no bytes, over `at`. - pub(super) fn place(&mut self, key: ResourceKey, at: (u64, u64)) { + /// Place `key`, lying over no bytes, over `at`, in the import `import` names (LOCAL -+ /// (macbox)). ++ /// (Atlas)). + pub(super) fn place(&mut self, key: ResourceKey, at: (u64, u64), import: u64) { if at.1 == 0 { return; @@ -540,7 +540,7 @@ index 11b17838..330c0690 100644 } } -+ /// macbox: whether the placed keys `a` and `b` lie in two imports, or either in one not ++ /// LOCAL (Atlas): whether the placed keys `a` and `b` lie in two imports, or either in one not + /// known (the module docs); `false` where either is not placed. + pub(super) fn apart(&self, a: ResourceKey, b: ResourceKey) -> bool { + match (self.at.get(&a), self.at.get(&b)) { @@ -553,7 +553,7 @@ index 11b17838..330c0690 100644 /// share, and the bytes of it they are (counted from its range's first byte), to `each`: its /// own range's key where another, and each other range's the bytes meet, clipped to it, in - /// the order the lookup by address finds them. The ranges the bytes may meet are those that -+ /// the order the lookup by address finds them; macbox: and whether that key lies in ++ /// the order the lookup by address finds them; LOCAL (Atlas): and whether that key lies in + /// another import than `key` ([`imports_apart`]). The ranges the bytes may meet are those that /// meet the key's own range, which it keeps: they are read from there while no more than the /// classes of length the lookup by address visits (a range query each, where a kept range is @@ -667,7 +667,7 @@ index fd28d676..db2af089 100644 /// The host page of the tests that name none of their own: an import's unit. const PAGE: u64 = 0x1000; -+/// macbox: the import the tests that name none place every key in: one import. ++/// LOCAL (Atlas): the import the tests that name none place every key in: one import. +const ONE: u64 = 1; + /// A reported remap counts the ranges placed buffers lie over that have a page among its bytes @@ -789,7 +789,7 @@ index fd28d676..db2af089 100644 changed("forgotten"); } + -+/// macbox: `used` as a use added for a key of another import takes it: `MEMORY_WRITE` ++/// LOCAL (Atlas): `used` as a use added for a key of another import takes it: `MEMORY_WRITE` +/// among the access types of its reads and of its writes. +fn apart(mut used: Use) -> Use { + for side in [&mut used.read, &mut used.write] { @@ -800,7 +800,7 @@ index fd28d676..db2af089 100644 + used +} + -+/// macbox: keys over one surface's pages in two imports (one made after a remap of them): ++/// LOCAL (Atlas): keys over one surface's pages in two imports (one made after a remap of them): +/// a use of the second's bytes carries a use of the first's key with `MEMORY_WRITE` among its +/// access types, where a key in the first's import carries it as it was; a range in a third +/// import that meets the surface's last page carries the first's key so too, clipped; and a key @@ -849,7 +849,7 @@ index fd28d676..db2af089 100644 + ); +} + -+/// macbox: two buffers made at once over one memory, each importing it: the second joins ++/// LOCAL (Atlas): two buffers made at once over one memory, each importing it: the second joins +/// the first's key, and where they are placed in two imports its uses carry the first's key with +/// `MEMORY_WRITE`; in one import, as they were. +#[test] @@ -883,7 +883,7 @@ index fd28d676..db2af089 100644 + drop((one, two)); +} + -+/// macbox: a remap over a memory each of whose live imports follows its address keeps ++/// LOCAL (Atlas): a remap over a memory each of whose live imports follows its address keeps +/// it: a later buffer over its bytes lies in its import and joins its key. One whose import does +/// not, or with no import live, ends. +#[test] @@ -930,7 +930,7 @@ index 8df378c2..aaf61433 100644 //! says so and passes. use super::blit_holds::{device, over_host, until, Host, Scratch, BYTES}; -+use crate::live::LiveFacts as _; // macbox: live fact switches ++use crate::live::LiveFacts as _; // LOCAL (Atlas): live fact switches use crate::{ BlitOption, Buffer, CommandBufferStatus, CommandQueue, Device, Error, Origin, ResourceOptions, Size, Texture, TextureDescriptor, WorkCounts, @@ -939,7 +939,7 @@ index 8df378c2..aaf61433 100644 assert_eq!(remaps(&device)[..3], [reported + 2, over + 1, remapped + 1]); // A second buffer over the remapped bytes imports them anew (`host_aliases.rs`): two live - // imports over the same bytes, one placed range, which a report counts once. -+ // imports over the same bytes, one placed range, which a report counts once. macbox: ++ // imports over the same bytes, one placed range, which a report counts once. LOCAL (Atlas): + // where the driver's pointer imports follow their address, it lies in the first's import, + // which reads the new pages: one import, the same range. let again = over_host(&device, &imported, &runs).unwrap(); @@ -971,7 +971,7 @@ index c453e0b2..bda123a2 100644 //! memory a buffer was made over; the runtime orders their uses by the bytes each lies over now //! (`host_aliases.rs`'s placements), so their reads are fresh wherever one buffer's would be, two -//! queues in submission order included. -+//! queues in submission order included. macbox: the one made after a remap imports the ++//! queues in submission order included. LOCAL (Atlas): the one made after a remap imports the +//! pages anew, a second `VkDeviceMemory` of them (on a driver whose pointer imports follow their +//! address it joins the first and lies in its import instead), and its uses through that import +//! carry `MEMORY_WRITE`, so their barriers flush what a pass wrote through the other: on ANV a @@ -985,7 +985,7 @@ index c453e0b2..bda123a2 100644 Rebound, /// Made over them after the program reported their range remapped while the first lived - /// (with no remap made): it joins nothing, and imports them anew. -+ /// (with no remap made): it joins nothing, and imports them anew (macbox: on a driver ++ /// (with no remap made): it joins nothing, and imports them anew (LOCAL (Atlas): on a driver + /// whose pointer imports follow their address it joins the first instead). AfterRemap, } diff --git a/patches/metal2vulkan/0006-diagnostic-pass-timer-M2V_PASS_TIMES-and.patch b/patches/metal2vulkan/0006-diagnostic-pass-timer-M2V_PASS_TIMES-and.patch new file mode 100644 index 0000000..780b104 --- /dev/null +++ b/patches/metal2vulkan/0006-diagnostic-pass-timer-M2V_PASS_TIMES-and.patch @@ -0,0 +1,3090 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Wed, 28 Oct 2026 16:50:52 +0000 +Subject: [PATCH 06/34] diagnostic:pass timer (M2V_PASS_TIMES) + and image census (M2V_IMAGE_LOG) + +Two default-off diagnostics in metal2vulkan-runtime, to find where a macOS +guest's GPU time goes on the Arrow Lake iGPU (Mesa's INTEL_MEASURE crashes +with this runtime). Each switch is read once; unset, nothing changes: no +query pool, no command, no line. Self-contained: two new modules +(pass_times.rs, image_log.rs) and small hooks, each marked +"LOCAL (Atlas) diagnostic", so the commit can be dropped. + +M2V_PASS_TIMES=: a command buffer begun while /on exists (checked +once, as it begins recording) gets vkCmdWriteTimestamp(ALL_COMMANDS) pairs +around each unit of GPU work, cut at the recorder's routed entry points so +every recording site is covered: each render pass instance (each instance +of a split pass), each run of dispatches, each run of transfers. Queries +come from the diagnostic's own pools, reset on the host. As the command +buffer completes (completion turn, after its GPU times) one CSV line per +unit goes to /passes-.csv: +host_ns,cb_seq,unit_seq,kind,cb_label,encoder_label,attachments,split,draws,gpu_ns +with the kind (render-pass, guest-, pass:/internal:), the +queue's label, the encoder's kind and place, each attachment (role, format, +extent, samples, load/store, memoryless, image purpose), the split (i/n, +cause, inline or secondary, why not inline) and the draws/dispatches. + +M2V_IMAGE_LOG=: one CSV line per vkCreateImage (purpose, format, +extent, mips, layers, samples, tiling, DRM modifier, usage, create flags, +format list, external handle types, chained structures, memory +requirements), per vkBindImageMemory (the memory's kind: own, host-pages, +dma-buf, dedicated) and per vkDestroyImage: the fields Mesa ANV's CCS +rules read. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/buffer_texture.rs | 4 + + .../src/command_buffer/completion.rs | 2 + + .../src/command_buffer/mod.rs | 66 +- + crates/metal2vulkan-runtime/src/creation.rs | 63 +- + .../src/device/functions.rs | 179 +++ + crates/metal2vulkan-runtime/src/device/mod.rs | 8 + + crates/metal2vulkan-runtime/src/encoder.rs | 10 + + crates/metal2vulkan-runtime/src/image_log.rs | 925 +++++++++++++ + crates/metal2vulkan-runtime/src/lib.rs | 4 + + crates/metal2vulkan-runtime/src/pass_times.rs | 1194 +++++++++++++++++ + .../src/pending_dispatch.rs | 2 + + .../src/recorder/thunks.rs | 20 +- + crates/metal2vulkan-runtime/src/release.rs | 2 + + crates/metal2vulkan-runtime/src/render.rs | 7 + + .../src/render/inline_first.rs | 23 + + .../src/render/inline_splits.rs | 5 + + .../src/render/pass_steps.rs | 11 + + crates/metal2vulkan-runtime/src/texture.rs | 27 + + 18 files changed, 2535 insertions(+), 17 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/image_log.rs + create mode 100644 crates/metal2vulkan-runtime/src/pass_times.rs + +diff --git a/crates/metal2vulkan-runtime/src/buffer_texture.rs b/crates/metal2vulkan-runtime/src/buffer_texture.rs +index b062a79b..45801900 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_texture.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_texture.rs +@@ -296,6 +296,8 @@ impl Buffer { + offset: u64, + bytes_per_row: u64, + ) -> Result { ++ // LOCAL (Atlas) diagnostic: the image census's purpose (`image_log.rs`). ++ let _purpose = crate::image_log::api("buffer-texture"); + self.texture_over( + "new_texture", + descriptor, +@@ -434,6 +436,8 @@ impl Buffer { + descriptor: &TextureDescriptor, + plane: IoSurfacePlane, + ) -> Result { ++ // LOCAL (Atlas) diagnostic: the image census's purpose (`image_log.rs`). ++ let _purpose = crate::image_log::api("iosurface-plane"); + if plane.client_direction.is_some() { + return Err(Error::AppleOnly(AppleOnly::SurfaceClientDirection)); + } +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +index 635861b6..cc2497bb 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +@@ -453,6 +453,8 @@ fn finish(watch: &Watch, lists: &mut FenceLists, mut finished: Vec<(usize, Submi + for buffer in &ending { + buffer.read_stamps(); + buffer.place_gpu_times(); ++ // LOCAL (Atlas) diagnostic: the pass timer's units (`pass_times.rs`). ++ buffer.read_pass_times(); + } + } + drop(device); +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index ac0260ac..b08a9994 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -746,6 +746,8 @@ pub(super) struct State { + /// The device's trail mark as it completed (`trails.rs`): its waiters return once no trail + /// numbered below is open. 0 before, which waits for none. + trail_mark: u64, ++ /// LOCAL (Atlas) diagnostic: its pass timer's capture (`pass_times.rs`). ++ pub(super) pass_times: crate::pass_times::CbSlot, + } + + /// A batch whose end went into its last `VkCommandBuffer` ([`State::folds`]). +@@ -1855,6 +1857,8 @@ impl CommandBuffer { + replaying_predication: None, + predicated: false, + trail_mark: 0, ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ pass_times: Default::default(), + }), + changed: Condvar::new(), + gate: Arc::default(), +@@ -1950,6 +1954,11 @@ impl CommandBuffer { + self.state().uploads.bytes() + } + ++ /// LOCAL (Atlas) diagnostic: the key the pass timer finds its capture by (`pass_times.rs`). ++ pub(crate) fn pass_times_key(&self) -> usize { ++ self.0.address() ++ } ++ + /// Whether `waiting` holds this command buffer. + pub(crate) fn is(&self, waiting: &dyn Waiting) -> bool { + self.0.address() == waiting.address() +@@ -3707,12 +3716,48 @@ impl CommandBufferShared { + level: vk::CommandBufferLevel, + ) -> Result { + if let Some(pool) = &state.rerun_pool { +- return hand_out(&self.device, &mut lock(pool), level); +- } +- match self.deferred(state, level)? { +- Some(raw) => Ok(raw), +- None => self.allocate(state, level), +- } ++ let raw = hand_out(&self.device, &mut lock(pool), level)?; ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ self.pass_times_handed_out(state, raw); ++ return Ok(raw); ++ } ++ let raw = match self.deferred(state, level)? { ++ Some(raw) => raw, ++ None => self.allocate(state, level)?, ++ }; ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ self.pass_times_handed_out(state, raw); ++ Ok(raw) ++ } ++ ++ /// LOCAL (Atlas) diagnostic: `raw`, handed out, goes to the pass timer's capture of this ++ /// command buffer, decided as it is first asked (`pass_times.rs`). ++ fn pass_times_handed_out(&self, state: &mut State, raw: vk::CommandBuffer) { ++ let key = self.address(); ++ crate::pass_times::handed_out(&self.device, &self.queue, key, &mut state.pass_times, raw); ++ } ++ ++ /// LOCAL (Atlas) diagnostic: read the pass timer's units, now that its work completed and its ++ /// GPU times were placed: in the completion turn, inside its section (`pass_times.rs`). ++ pub(super) fn read_pass_times(&self) { ++ let mut state = lock(&self.state); ++ let Some(capture) = state.pass_times.take() else { ++ return; ++ }; ++ let start = match (state.times.stamped, state.times.gpu) { ++ (Stamped::Read(Some((ticks, _))), Some((host, _))) => Some((ticks, host)), ++ _ => None, ++ }; ++ let pair = state.times.pair.map(|reading| reading.pair); ++ let completed = state.times.completed; ++ drop(state); ++ let placement = crate::pass_times::Placement { ++ clock: self.device.shared().timing.clock(), ++ pair, ++ start, ++ completed, ++ }; ++ crate::pass_times::read(capture, placement); + } + + /// A deferred command buffer of `level` of its replays, where the device records them so and +@@ -3748,16 +3793,23 @@ impl CommandBufferShared { + state: &mut State, + level: vk::CommandBufferLevel, + ) -> Result { +- hand_out(&self.device, self.pool(state)?, level) ++ let raw = hand_out(&self.device, self.pool(state)?, level)?; ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ self.pass_times_handed_out(state, raw); ++ Ok(raw) + } + + /// [`CommandBuffer::encoder_primary`]: a deferred one where the device records them so, to + /// begin, whose replay takes the primary its replays' pool holds begun ahead (`recorder`). + fn encoder_primary(&self, state: &mut State) -> Result<(vk::CommandBuffer, bool), Error> { + if let Some(raw) = self.deferred(state, vk::CommandBufferLevel::PRIMARY)? { ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ self.pass_times_handed_out(state, raw); + return Ok((raw, false)); + } + if let Some(buffer) = self.pool(state)?.take_begun() { ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ self.pass_times_handed_out(state, buffer); + return Ok((buffer, true)); + } + Ok((self.allocate_raw(state)?, false)) +diff --git a/crates/metal2vulkan-runtime/src/creation.rs b/crates/metal2vulkan-runtime/src/creation.rs +index 1d760a06..b4eac3f0 100644 +--- a/crates/metal2vulkan-runtime/src/creation.rs ++++ b/crates/metal2vulkan-runtime/src/creation.rs +@@ -38,7 +38,13 @@ pub(crate) unsafe fn image( + ) -> Result { + vk_made(VkMaking::Image, Location::caller()); + // SAFETY: the caller's. +- unsafe { device.create_image(info, None) }.map_err(vulkan) ++ let made = unsafe { device.create_image(info, None) }.map_err(vulkan); ++ // LOCAL (Atlas) diagnostic: the image census (`image_log.rs`). ++ if let Ok(image) = made { ++ // SAFETY: the caller's; `image` was just made by `device`. ++ unsafe { crate::image_log::image_made(device, info, image, Location::caller()) }; ++ } ++ made + } + + /// A `VkImage` of `info`, made by `device`, or the driver's refusal of it: an image whose making +@@ -57,7 +63,12 @@ pub(crate) unsafe fn image_or_refusal( + vk_made(VkMaking::Image, Location::caller()); + // SAFETY: the caller's. + match unsafe { device.create_image(info, None) } { +- Ok(raw) => Ok(Ok(raw)), ++ Ok(raw) => { ++ // LOCAL (Atlas) diagnostic: the image census (`image_log.rs`). ++ // SAFETY: the caller's; `raw` was just made by `device`. ++ unsafe { crate::image_log::image_made(device, info, raw, Location::caller()) }; ++ Ok(Ok(raw)) ++ } + Err(vk::Result::ERROR_DEVICE_LOST) => Err(Error::DeviceLost), + Err(refusal) => Ok(Err(refusal)), + } +@@ -92,7 +103,12 @@ pub(crate) unsafe fn image_view( + ) -> VkResult { + vk_made(VkMaking::ImageView, Location::caller()); + // SAFETY: the caller's. +- unsafe { device.create_image_view(info, None) } ++ let made = unsafe { device.create_image_view(info, None) }; ++ // LOCAL (Atlas) diagnostic: the object registry (`image_log.rs`). ++ if let Ok(view) = made { ++ crate::image_log::view_made(info, view); ++ } ++ made + } + + /// `VkDeviceMemory` of `info`, allocated by `device`: one call of `vkAllocateMemory`, counted +@@ -110,7 +126,13 @@ pub(crate) unsafe fn allocate_memory( + ) -> VkResult { + vk_made(VkMaking::Memory, Location::caller()); + // SAFETY: the caller's. +- unsafe { device.allocate_memory(info, None) } ++ let made = unsafe { device.allocate_memory(info, None) }; ++ // LOCAL (Atlas) diagnostic: the image census's memory kinds (`image_log.rs`). ++ if let Ok(memory) = made { ++ // SAFETY: the caller's: `info` and its chain are valid. ++ unsafe { crate::image_log::memory_made(info, memory) }; ++ } ++ made + } + + /// `image` bound to `memory` at `offset`. +@@ -128,7 +150,12 @@ pub(crate) unsafe fn bind_image( + ) -> VkResult<()> { + vk_made(VkMaking::ImageBinding, Location::caller()); + // SAFETY: the caller's. +- unsafe { device.bind_image_memory(image, memory, offset) } ++ let bound = unsafe { device.bind_image_memory(image, memory, offset) }; ++ // LOCAL (Atlas) diagnostic: the image census (`image_log.rs`). ++ if bound.is_ok() { ++ crate::image_log::image_bound(image, memory, offset); ++ } ++ bound + } + + /// `buffer` bound to `memory` at `offset`. +@@ -154,12 +181,19 @@ pub(crate) unsafe fn bind_buffer( + /// # Safety + /// + /// As `vkCreateRenderPass`'s: `device` is live, and `info` and its chain are valid for it. ++#[track_caller] + pub(crate) unsafe fn render_pass( + device: &ash::Device, + info: &vk::RenderPassCreateInfo<'_>, + ) -> Result { + // SAFETY: the caller's. +- unsafe { device.create_render_pass(info, None) }.map_err(vulkan) ++ let made = unsafe { device.create_render_pass(info, None) }.map_err(vulkan); ++ // LOCAL (Atlas) diagnostic: the object registry (`image_log.rs`). ++ if let Ok(pass) = made { ++ // SAFETY: the caller's: `info` is valid. ++ unsafe { crate::image_log::render_pass_made(info, pass, Location::caller()) }; ++ } ++ made + } + + /// A `VkRenderPass` of `info` (`vkCreateRenderPass2`), made by `device`. +@@ -168,12 +202,19 @@ pub(crate) unsafe fn render_pass( + /// + /// As `vkCreateRenderPass2`'s: `device` is live and of Vulkan 1.2 or later, and `info` and its + /// chain are valid for it. ++#[track_caller] + pub(crate) unsafe fn render_pass2( + device: &ash::Device, + info: &vk::RenderPassCreateInfo2<'_>, + ) -> Result { + // SAFETY: the caller's. +- unsafe { device.create_render_pass2(info, None) }.map_err(vulkan) ++ let made = unsafe { device.create_render_pass2(info, None) }.map_err(vulkan); ++ // LOCAL (Atlas) diagnostic: the object registry (`image_log.rs`). ++ if let Ok(pass) = made { ++ // SAFETY: the caller's: `info` and its chain are valid. ++ unsafe { crate::image_log::render_pass2_made(info, pass, Location::caller()) }; ++ } ++ made + } + + /// A `VkFramebuffer` of `info`, made by `device`. +@@ -186,5 +227,11 @@ pub(crate) unsafe fn framebuffer( + info: &vk::FramebufferCreateInfo<'_>, + ) -> Result { + // SAFETY: the caller's. +- unsafe { device.create_framebuffer(info, None) }.map_err(vulkan) ++ let made = unsafe { device.create_framebuffer(info, None) }.map_err(vulkan); ++ // LOCAL (Atlas) diagnostic: the object registry (`image_log.rs`). ++ if let Ok(framebuffer) = made { ++ // SAFETY: the caller's: `info` is valid. ++ unsafe { crate::image_log::framebuffer_made(info, framebuffer) }; ++ } ++ made + } +diff --git a/crates/metal2vulkan-runtime/src/device/functions.rs b/crates/metal2vulkan-runtime/src/device/functions.rs +index be874eac..72ef34e8 100644 +--- a/crates/metal2vulkan-runtime/src/device/functions.rs ++++ b/crates/metal2vulkan-runtime/src/device/functions.rs +@@ -147,6 +147,8 @@ impl Vk { + regions: &[vk::BufferCopy], + ) { + (self.copies).count("vkCmdCopyBuffer", Location::caller(), regions.len()); ++ // LOCAL (Atlas) diagnostic: the caller's place, for the pass timer (`pass_times.rs`). ++ crate::pass_times::site(Location::caller()); + let splits = regions + .iter() + .filter(|region| split(region).is_some()) +@@ -194,6 +196,8 @@ impl Vk { + regions: &[vk::BufferImageCopy], + ) { + (self.copies).count("vkCmdCopyBufferToImage", Location::caller(), regions.len()); ++ // LOCAL (Atlas) diagnostic: the caller's place, for the pass timer (`pass_times.rs`). ++ crate::pass_times::site(Location::caller()); + // SAFETY: the caller's contract. + unsafe { + (self.device).cmd_copy_buffer_to_image(command_buffer, src, dst, layout, regions); +@@ -216,6 +220,8 @@ impl Vk { + regions: &[vk::BufferImageCopy], + ) { + (self.copies).count("vkCmdCopyImageToBuffer", Location::caller(), regions.len()); ++ // LOCAL (Atlas) diagnostic: the caller's place, for the pass timer (`pass_times.rs`). ++ crate::pass_times::site(Location::caller()); + // SAFETY: the caller's contract. + unsafe { + (self.device).cmd_copy_image_to_buffer(command_buffer, src, layout, dst, regions); +@@ -238,6 +244,8 @@ impl Vk { + regions: &[vk::ImageCopy], + ) { + (self.copies).count("vkCmdCopyImage", Location::caller(), regions.len()); ++ // LOCAL (Atlas) diagnostic: the caller's place, for the pass timer (`pass_times.rs`). ++ crate::pass_times::site(Location::caller()); + // SAFETY: the caller's contract. + unsafe { + (self.device).cmd_copy_image(command_buffer, src, src_layout, dst, dst_layout, regions); +@@ -245,6 +253,177 @@ impl Vk { + } + } + ++/// LOCAL (Atlas) diagnostic: the commands the pass timer labels a unit of work by, each noting its ++/// caller's place for the command's entry point to take (`pass_times.rs`), then `ash`'s. A caller ++/// holding the `ash::Device` itself records through `ash`'s, unnoted. ++impl Vk { ++ /// `vkCmdDispatch`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_dispatch`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_dispatch( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ x: u32, ++ y: u32, ++ z: u32, ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_dispatch(command_buffer, x, y, z) } ++ } ++ ++ /// `vkCmdDispatchIndirect`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_dispatch_indirect`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_dispatch_indirect( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ buffer: vk::Buffer, ++ offset: vk::DeviceSize, ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_dispatch_indirect(command_buffer, buffer, offset) } ++ } ++ ++ /// `vkCmdFillBuffer`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_fill_buffer`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_fill_buffer( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ buffer: vk::Buffer, ++ offset: vk::DeviceSize, ++ size: vk::DeviceSize, ++ data: u32, ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_fill_buffer(command_buffer, buffer, offset, size, data) } ++ } ++ ++ /// `vkCmdUpdateBuffer`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_update_buffer`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_update_buffer( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ buffer: vk::Buffer, ++ offset: vk::DeviceSize, ++ data: &[u8], ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_update_buffer(command_buffer, buffer, offset, data) } ++ } ++ ++ /// `vkCmdClearColorImage`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_clear_color_image`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_clear_color_image( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ image: vk::Image, ++ layout: vk::ImageLayout, ++ color: &vk::ClearColorValue, ++ ranges: &[vk::ImageSubresourceRange], ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_clear_color_image(command_buffer, image, layout, color, ranges) } ++ } ++ ++ /// `vkCmdClearDepthStencilImage`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_clear_depth_stencil_image`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_clear_depth_stencil_image( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ image: vk::Image, ++ layout: vk::ImageLayout, ++ value: &vk::ClearDepthStencilValue, ++ ranges: &[vk::ImageSubresourceRange], ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { ++ (self.device).cmd_clear_depth_stencil_image( ++ command_buffer, ++ image, ++ layout, ++ value, ++ ranges, ++ ) ++ } ++ } ++ ++ /// `vkCmdBlitImage`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_blit_image`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_blit_image( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ src: vk::Image, ++ src_layout: vk::ImageLayout, ++ dst: vk::Image, ++ dst_layout: vk::ImageLayout, ++ regions: &[vk::ImageBlit], ++ filter: vk::Filter, ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { ++ (self.device).cmd_blit_image( ++ command_buffer, ++ src, ++ src_layout, ++ dst, ++ dst_layout, ++ regions, ++ filter, ++ ) ++ } ++ } ++ ++ /// `vkCmdBeginRenderPass`, its caller's place noted. ++ /// ++ /// # Safety ++ /// ++ /// As `ash::Device::cmd_begin_render_pass`'s. ++ #[track_caller] ++ pub(crate) unsafe fn cmd_begin_render_pass( ++ &self, ++ command_buffer: vk::CommandBuffer, ++ begin: &vk::RenderPassBeginInfo<'_>, ++ contents: vk::SubpassContents, ++ ) { ++ crate::pass_times::site(Location::caller()); ++ // SAFETY: the caller's. ++ unsafe { (self.device).cmd_begin_render_pass(command_buffer, begin, contents) } ++ } ++} ++ + impl Deref for Vk { + type Target = ash::Device; + +diff --git a/crates/metal2vulkan-runtime/src/device/mod.rs b/crates/metal2vulkan-runtime/src/device/mod.rs +index 70a18c56..acc27e6b 100644 +--- a/crates/metal2vulkan-runtime/src/device/mod.rs ++++ b/crates/metal2vulkan-runtime/src/device/mod.rs +@@ -451,6 +451,8 @@ pub(super) struct DeviceShared { + pub(super) residency: super::host_residency::Sweeper, + /// How its command buffers are timed on the GPU (`gpu_time.rs`). + pub(super) timing: Timing, ++ /// LOCAL (Atlas) diagnostic: the pass timer, where `M2V_PASS_TIMES` is set (`pass_times.rs`). ++ pub(super) pass_times: Option>, + /// Its Vulkan views may present an image in another component layout: it has no portability + /// subset, or the subset's `imageViewFormatReinterpretation` is enabled, by the runtime on a + /// device it made or, on an adopted one, by the consumer, who says so +@@ -741,6 +743,10 @@ impl Drop for DeviceShared { + // Nor a timestamp slot: each command buffer gave its slot back as it finished, or + // with its state. + self.timing.release(self.vk()); ++ // LOCAL (Atlas) diagnostic: the pass timer's pools (`pass_times.rs`). ++ if let Some(times) = &self.pass_times { ++ times.release(self.vk()); ++ } + } + // Nor a kept barrier command buffer: the queues are idle. + self.scheduler.release_kept(self.vk()); +@@ -1825,6 +1831,8 @@ impl Device { + #[cfg(unix)] + residency: Default::default(), + timing, ++ // LOCAL (Atlas) diagnostic: the pass timer (`pass_times.rs`). ++ pass_times: crate::pass_times::DeviceTimes::new(timestamp_bits > 0, owned), + view_reinterpretation, + generated_commands, + host_image_copy, +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index f4ecfa6c..5b217e85 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -428,6 +428,14 @@ impl EncoderCore { + if let Some(problem) = state.refusal() { + return Err(invalid(call, problem)); + } ++ // LOCAL (Atlas) diagnostic: the pass timer's encoder (`pass_times.rs`). ++ crate::pass_times::encoder_opened( ++ command_buffer.device(), ++ command_buffer.command_queue(), ++ command_buffer.pass_times_key(), ++ &mut state.pass_times, ++ kind, ++ ); + // The pool's primary begun ahead where it holds one (`spares.rs`). + let (raw, begun) = command_buffer.encoder_primary(&mut state)?; + let shared = command_buffer.device().shared(); +@@ -1058,6 +1066,8 @@ impl EncoderCore { + // SAFETY: as in `barrier`. + unsafe { barrier.record(device, raw) }; + } ++ // LOCAL (Atlas) diagnostic: the program's command, to the pass timer (`pass_times.rs`). ++ let _scope = crate::pass_times::scope(crate::pass_times::Tag::Guest); + record(device, raw); + } + +diff --git a/crates/metal2vulkan-runtime/src/image_log.rs b/crates/metal2vulkan-runtime/src/image_log.rs +new file mode 100644 +index 00000000..36639493 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/image_log.rs +@@ -0,0 +1,925 @@ ++//! LOCAL (Atlas) diagnostic: the image census, `M2V_IMAGE_LOG=`, and the registry of the ++//! objects a render pass instance names, which the pass timer (`pass_times.rs`) labels its ++//! attachments from. Default off; the whole module can be dropped with its hooks (each marked ++//! `LOCAL (Atlas) diagnostic`). ++//! ++//! - **The switch** is read once. Unset (or empty), nothing is recorded, no line is formatted, and ++//! no Vulkan call is added. Set, the census appends to the file it names (made if missing). ++//! - **What the census writes.** One CSV line per event, every line the same columns ++//! ([`HEADER`]), written as it happens (a `write` per line: image creation is rare next to the ++//! work it serves, and the lines survive the process being killed): ++//! - `create`, per `VkImage` the runtime makes (`creation::image`, `creation::image_or_refusal`, ++//! the one door to `vkCreateImage`; the device's priming makes its own and is not logged): the ++//! image, its purpose (`api`: the program's call that made it where one is on the stack, ++//! `texture`, `heap-texture`, `buffer-texture`, `iosurface-plane`; `origin`: where the ++//! texture paths' image making was called from, with the texture's storage mode; `site`: the ++//! call of `creation::image` itself), format, extent, mip levels, array layers, samples, ++//! tiling (and the DRM format modifier or modifier list chained), usage, create flags, the ++//! chained `VkImageFormatListCreateInfo` (`-` where none is chained, else its view formats), ++//! the external memory handle types chained, any other structure chained, and the image's ++//! memory requirements (size, alignment, memory types; `vkGetImageMemoryRequirements`, not ++//! asked of a `DISJOINT` image). ++//! - `bind`, per `vkBindImageMemory` (`creation::bind_image`): the memory and offset, and what ++//! the memory is as the runtime allocated it (`creation::allocate_memory`): an import of host ++//! pages (`host-pages`), of a dma-buf (`dma-buf`) or another fd, an export, or its own, and ++//! whether dedicated to an image. ++//! - `destroy`, per `vkDestroyImage` (`release::object`). ++//! - **The registry** (kept while either switch is on): each image's extent, usage and purpose, ++//! each view's image and base level, each render pass's attachments (subpass 0's roles) and its ++//! making's site, each framebuffer's extent and views, and each memory's kind; entries go as ++//! their objects are released. A handle the driver reuses is registered again as it is made. ++ ++use ash::vk; ++use ash::vk::Handle; ++use std::cell::Cell; ++use std::collections::HashMap; ++use std::fmt::Write as _; ++use std::io::Write as _; ++use std::panic::Location; ++use std::path::{Path, PathBuf}; ++use std::sync::{Arc, LazyLock, Mutex, OnceLock}; ++ ++/// The census's columns, its file's first line. ++pub(crate) const HEADER: &str = "host_ns,event,image,api,origin,site,storage,format,width,height,\ ++depth,mips,layers,samples,tiling,drm_modifier,usage,flags,format_list,external,chained,mem_size,\ ++mem_align,mem_types,bind_memory,bind_offset,memory_kind\n"; ++ ++/// The census file, where `M2V_IMAGE_LOG` names one: read once. ++fn census_path() -> Option<&'static Path> { ++ static PATH: OnceLock> = OnceLock::new(); ++ PATH.get_or_init(|| { ++ std::env::var_os("M2V_IMAGE_LOG") ++ .filter(|value| !value.is_empty()) ++ .map(PathBuf::from) ++ }) ++ .as_deref() ++} ++ ++/// Whether the census is on. ++pub(crate) fn census_on() -> bool { ++ census_path().is_some() ++} ++ ++/// Whether the registry is kept: either diagnostic is on. ++pub(crate) fn registry_on() -> bool { ++ census_on() || crate::pass_times::on() ++} ++ ++/// The host's clock now, in nanoseconds (`CLOCK_MONOTONIC_RAW`, the runtime's GPU times' clock). ++#[cfg(not(unix))] ++pub(crate) fn host_ns() -> u64 { ++ 0 ++} ++ ++/// The host's clock now, in nanoseconds (`CLOCK_MONOTONIC_RAW`, the runtime's GPU times' clock). ++#[cfg(unix)] ++pub(crate) fn host_ns() -> u64 { ++ let mut now = libc::timespec { ++ tv_sec: 0, ++ tv_nsec: 0, ++ }; ++ // SAFETY: `now` is a valid timespec for the call to fill. ++ if unsafe { libc::clock_gettime(libc::CLOCK_MONOTONIC_RAW, &mut now) } != 0 { ++ return 0; ++ } ++ (now.tv_sec as u64) * 1_000_000_000 + now.tv_nsec as u64 ++} ++ ++/// `file:line` of `at`, its path from the crate's `src/` on. ++pub(crate) fn short(at: &Location<'_>) -> String { ++ let file = at.file(); ++ let file = match file.rfind("/src/") { ++ Some(i) => &file[i + 5..], ++ None => file, ++ }; ++ format!("{file}:{}", at.line()) ++} ++ ++/// The bits of `raw` by their registry names in `known`, less `prefix` and `_BIT`, `|`-joined; ++/// bits no name holds in hex; `0` for none. (`ash` is built without its `Debug` impls.) ++fn flag_names<'a>(raw: u32, known: impl Iterator, prefix: &str) -> String { ++ if raw == 0 { ++ return "0".to_owned(); ++ } ++ let mut names = Vec::new(); ++ let mut rest = raw; ++ for (bit, name) in known { ++ if bit.count_ones() == 1 && rest & bit != 0 { ++ names.push(name.trim_start_matches(prefix).replace("_BIT", "")); ++ rest &= !bit; ++ } ++ } ++ if rest != 0 { ++ names.push(format!("0x{rest:x}")); ++ } ++ names.join("|") ++} ++ ++/// `format`'s registry name less `VK_FORMAT_`, else its number. ++pub(crate) fn format_name(format: vk::Format) -> String { ++ match metal2vulkan::vk::Format::from_raw(format.as_raw()).name() { ++ Some(name) => name.trim_start_matches("VK_FORMAT_").to_owned(), ++ None => format!("format{}", format.as_raw()), ++ } ++} ++ ++fn tiling_name(tiling: vk::ImageTiling) -> String { ++ match metal2vulkan::vk::ImageTiling::from_raw(tiling.as_raw()).name() { ++ Some(name) => name.trim_start_matches("VK_IMAGE_TILING_").to_owned(), ++ None => format!("tiling{}", tiling.as_raw()), ++ } ++} ++ ++fn usage_names(usage: vk::ImageUsageFlags) -> String { ++ let known = metal2vulkan::vk::ImageUsageFlags::KNOWN.iter(); ++ flag_names( ++ usage.as_raw(), ++ known.map(|(v, n)| (v.as_raw(), *n)), ++ "VK_IMAGE_USAGE_", ++ ) ++} ++ ++fn create_names(flags: vk::ImageCreateFlags) -> String { ++ let known = metal2vulkan::vk::ImageCreateFlags::KNOWN.iter(); ++ flag_names( ++ flags.as_raw(), ++ known.map(|(v, n)| (v.as_raw(), *n)), ++ "VK_IMAGE_CREATE_", ++ ) ++} ++ ++fn external_names(types: vk::ExternalMemoryHandleTypeFlags) -> String { ++ use vk::ExternalMemoryHandleTypeFlags as T; ++ let known = [ ++ (T::OPAQUE_FD, "OPAQUE_FD"), ++ (T::OPAQUE_WIN32, "OPAQUE_WIN32"), ++ (T::OPAQUE_WIN32_KMT, "OPAQUE_WIN32_KMT"), ++ (T::D3D11_TEXTURE, "D3D11_TEXTURE"), ++ (T::D3D11_TEXTURE_KMT, "D3D11_TEXTURE_KMT"), ++ (T::D3D12_HEAP, "D3D12_HEAP"), ++ (T::D3D12_RESOURCE, "D3D12_RESOURCE"), ++ (T::HOST_ALLOCATION_EXT, "HOST_ALLOCATION"), ++ ( ++ T::HOST_MAPPED_FOREIGN_MEMORY_EXT, ++ "HOST_MAPPED_FOREIGN_MEMORY", ++ ), ++ (T::DMA_BUF_EXT, "DMA_BUF"), ++ ]; ++ flag_names( ++ types.as_raw(), ++ known.iter().map(|(v, n)| (v.as_raw(), *n)), ++ "", ++ ) ++} ++ ++fn load_name(op: vk::AttachmentLoadOp) -> String { ++ match op { ++ vk::AttachmentLoadOp::LOAD => "LOAD".into(), ++ vk::AttachmentLoadOp::CLEAR => "CLEAR".into(), ++ vk::AttachmentLoadOp::DONT_CARE => "DONT_CARE".into(), ++ vk::AttachmentLoadOp::NONE_KHR => "NONE".into(), ++ other => format!("load{}", other.as_raw()), ++ } ++} ++ ++fn store_name(op: vk::AttachmentStoreOp) -> String { ++ match op { ++ vk::AttachmentStoreOp::STORE => "STORE".into(), ++ vk::AttachmentStoreOp::DONT_CARE => "DONT_CARE".into(), ++ vk::AttachmentStoreOp::NONE => "NONE".into(), ++ other => format!("store{}", other.as_raw()), ++ } ++} ++ ++/// What the registry holds of an image. ++pub(crate) struct ImageEntry { ++ extent: vk::Extent3D, ++ usage: vk::ImageUsageFlags, ++ /// Its purpose, short: the program's call, else the texture path's caller, else the site. ++ purpose: Arc, ++} ++ ++/// A render pass attachment's role in subpass 0. ++#[derive(Clone, Copy)] ++enum Role { ++ Color(u32), ++ Resolve(u32), ++ DepthStencil, ++ DepthStencilResolve, ++ Input(u32), ++ Other(u32), ++} ++ ++/// A render pass's attachment, as made. ++#[derive(Clone, Copy)] ++struct PassAttachment { ++ role: Role, ++ format: vk::Format, ++ samples: u32, ++ load: vk::AttachmentLoadOp, ++ store: vk::AttachmentStoreOp, ++ stencil_load: vk::AttachmentLoadOp, ++ stencil_store: vk::AttachmentStoreOp, ++} ++ ++/// A render pass, as made. ++struct PassEntry { ++ attachments: Vec, ++ site: &'static Location<'static>, ++} ++ ++/// A framebuffer, as made. ++struct FramebufferEntry { ++ width: u32, ++ height: u32, ++ views: Vec, ++} ++ ++/// The objects registered. ++#[derive(Default)] ++struct Registry { ++ images: HashMap>, ++ views: HashMap, ++ passes: HashMap>, ++ framebuffers: HashMap>, ++ memories: HashMap, ++} ++ ++static REGISTRY: LazyLock> = LazyLock::new(Mutex::default); ++ ++fn registry() -> std::sync::MutexGuard<'static, Registry> { ++ REGISTRY ++ .lock() ++ .unwrap_or_else(std::sync::PoisonError::into_inner) ++} ++ ++thread_local! { ++ /// The program's call making a texture on this thread, the outermost. ++ static API: Cell> = const { Cell::new(None) }; ++ /// Where the texture paths' image making was called from, and the texture's storage mode. ++ static ORIGIN: Cell, &'static str)>> = ++ const { Cell::new(None) }; ++} ++ ++/// A purpose set on this thread until dropped ([`api`], [`origin`]). ++pub(crate) struct PurposeGuard(Option); ++ ++enum Restore { ++ Api(Option<&'static str>), ++ Origin(Option<(&'static Location<'static>, &'static str)>), ++} ++ ++impl Drop for PurposeGuard { ++ fn drop(&mut self) { ++ match self.0.take() { ++ Some(Restore::Api(previous)) => API.with(|api| api.set(previous)), ++ Some(Restore::Origin(previous)) => ORIGIN.with(|origin| origin.set(previous)), ++ None => {} ++ } ++ } ++} ++ ++/// The program's call `name` makes textures on this thread until the guard drops: the census's ++/// `api`. The outermost call holds. ++pub(crate) fn api(name: &'static str) -> PurposeGuard { ++ if !registry_on() { ++ return PurposeGuard(None); ++ } ++ let previous = API.with(|api| api.get()); ++ if previous.is_some() { ++ return PurposeGuard(None); ++ } ++ API.with(|api| api.set(Some(name))); ++ PurposeGuard(Some(Restore::Api(previous))) ++} ++ ++/// The texture paths' image making called from `at`, for a texture of `storage` mode, until the ++/// guard drops: the census's `origin`. The innermost holds. ++pub(crate) fn origin(at: &'static Location<'static>, storage: &'static str) -> PurposeGuard { ++ if !registry_on() { ++ return PurposeGuard(None); ++ } ++ let previous = ORIGIN.with(|origin| origin.replace(Some((at, storage)))); ++ PurposeGuard(Some(Restore::Origin(previous))) ++} ++ ++/// The census's line writer. ++static CENSUS: Mutex> = Mutex::new(None); ++ ++/// Append `line` to the census, opening it (and heading a new one) at the first. ++fn census_line(line: &str) { ++ let Some(path) = census_path() else { ++ return; ++ }; ++ let mut file = CENSUS ++ .lock() ++ .unwrap_or_else(std::sync::PoisonError::into_inner); ++ if file.is_none() { ++ let opened = std::fs::OpenOptions::new() ++ .create(true) ++ .append(true) ++ .open(path); ++ match opened { ++ Ok(mut opened) => { ++ if opened.metadata().map(|m| m.len() == 0).unwrap_or(false) { ++ let _ = opened.write_all(HEADER.as_bytes()); ++ } ++ *file = Some(opened); ++ } ++ Err(error) => { ++ eprintln!("metal2vulkan: M2V_IMAGE_LOG: cannot open {path:?}: {error}"); ++ return; ++ } ++ } ++ } ++ if let Some(file) = file.as_mut() { ++ let _ = file.write_all(line.as_bytes()); ++ } ++} ++ ++/// The structures chained to `next`, as a walk of their headers. ++/// ++/// # Safety ++/// ++/// `next` is null or a valid chain of Vulkan structures. ++unsafe fn chain(mut next: *const std::ffi::c_void) -> Vec<*const vk::BaseInStructure<'static>> { ++ let mut all = Vec::new(); ++ while !next.is_null() { ++ let base = next.cast::>(); ++ all.push(base); ++ // SAFETY: the caller's. ++ next = unsafe { (*base).p_next }.cast(); ++ } ++ all ++} ++ ++/// A slice of `count` from `ptr`, empty where null. ++/// ++/// # Safety ++/// ++/// `ptr` is null or points at `count` valid values. ++unsafe fn slice<'a, T>(ptr: *const T, count: u32) -> &'a [T] { ++ match ptr.is_null() || count == 0 { ++ true => &[], ++ // SAFETY: the caller's. ++ false => unsafe { std::slice::from_raw_parts(ptr, count as usize) }, ++ } ++} ++ ++/// Note `image`, made of `info` by `device` at `site` (`creation::image`): registered, and in the ++/// census its `create` line. ++/// ++/// # Safety ++/// ++/// `info` and its chain are valid; `image` is `device`'s, live. ++pub(crate) unsafe fn image_made( ++ device: &ash::Device, ++ info: &vk::ImageCreateInfo<'_>, ++ image: vk::Image, ++ site: &'static Location<'static>, ++) { ++ if !registry_on() { ++ return; ++ } ++ let api = API.with(|api| api.get()); ++ let origin = ORIGIN.with(|origin| origin.get()); ++ let site = short(site); ++ let origin_at = origin.map(|(at, _)| short(at)); ++ let purpose: Arc = match (api, &origin_at) { ++ (Some(api), _) => api.into(), ++ (None, Some(at)) => format!("<{at}").into(), ++ (None, None) => site.clone().into(), ++ }; ++ let entry = ImageEntry { ++ extent: info.extent, ++ usage: info.usage, ++ purpose, ++ }; ++ registry().images.insert(image.as_raw(), Arc::new(entry)); ++ if !census_on() { ++ return; ++ } ++ let mut format_list = "-".to_owned(); ++ let mut external = "0".to_owned(); ++ let mut drm_modifier = "-".to_owned(); ++ let mut chained = Vec::new(); ++ // SAFETY: the caller's: `info`'s chain is valid. ++ for base in unsafe { chain(info.p_next) } { ++ // SAFETY: each is a valid structure of its `s_type`. ++ unsafe { ++ match (*base).s_type { ++ vk::StructureType::IMAGE_FORMAT_LIST_CREATE_INFO => { ++ let list = &*base.cast::>(); ++ let formats = slice(list.p_view_formats, list.view_format_count); ++ let names: Vec = formats.iter().map(|f| format_name(*f)).collect(); ++ format_list = format!("{}:{}", formats.len(), names.join("|")); ++ } ++ vk::StructureType::EXTERNAL_MEMORY_IMAGE_CREATE_INFO => { ++ let ext = &*base.cast::>(); ++ external = external_names(ext.handle_types); ++ } ++ vk::StructureType::IMAGE_DRM_FORMAT_MODIFIER_EXPLICIT_CREATE_INFO_EXT => { ++ let explicit = ++ &*base.cast::>(); ++ drm_modifier = format!( ++ "0x{:x}:planes={}", ++ explicit.drm_format_modifier, explicit.drm_format_modifier_plane_count ++ ); ++ } ++ vk::StructureType::IMAGE_DRM_FORMAT_MODIFIER_LIST_CREATE_INFO_EXT => { ++ let list = ++ &*base.cast::>(); ++ let all = slice(list.p_drm_format_modifiers, list.drm_format_modifier_count); ++ let names: Vec = all.iter().map(|m| format!("0x{m:x}")).collect(); ++ drm_modifier = format!("list:{}", names.join("|")); ++ } ++ other => chained.push(format!("sType{}", other.as_raw())), ++ } ++ } ++ } ++ let requirements = match info.flags.contains(vk::ImageCreateFlags::DISJOINT) { ++ true => None, ++ // SAFETY: `image` is `device`'s, live, not disjoint. ++ false => Some(unsafe { device.get_image_memory_requirements(image) }), ++ }; ++ crate::exit_guard::called(crate::exit_guard::Call::GetImageMemoryRequirements); ++ let (size, align, types) = match requirements { ++ Some(r) => ( ++ r.size.to_string(), ++ r.alignment.to_string(), ++ format!("0x{:x}", r.memory_type_bits), ++ ), ++ None => ("-".into(), "-".into(), "-".into()), ++ }; ++ let chained = match chained.is_empty() { ++ true => "-".to_owned(), ++ false => chained.join("|").replace(',', ";"), ++ }; ++ let (origin_at, storage) = match origin { ++ Some((_, storage)) => (origin_at.unwrap_or_default(), storage), ++ None => ("-".to_owned(), "-"), ++ }; ++ let mut line = String::with_capacity(256); ++ let _ = writeln!( ++ line, ++ "{},create,0x{:x},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},{},,,", ++ host_ns(), ++ image.as_raw(), ++ api.unwrap_or("-"), ++ origin_at, ++ site, ++ storage, ++ format_name(info.format), ++ info.extent.width, ++ info.extent.height, ++ info.extent.depth, ++ info.mip_levels, ++ info.array_layers, ++ info.samples.as_raw(), ++ tiling_name(info.tiling), ++ drm_modifier, ++ usage_names(info.usage), ++ create_names(info.flags), ++ format_list, ++ external, ++ chained, ++ size, ++ align, ++ types, ++ ); ++ census_line(&line); ++} ++ ++/// Note `memory`, allocated of `info` (`creation::allocate_memory`): its kind registered. ++/// ++/// # Safety ++/// ++/// `info` and its chain are valid. ++pub(crate) unsafe fn memory_made(info: &vk::MemoryAllocateInfo<'_>, memory: vk::DeviceMemory) { ++ if !census_on() { ++ return; ++ } ++ let mut kind = "own"; ++ let mut dedicated = false; ++ // SAFETY: the caller's. ++ for base in unsafe { chain(info.p_next) } { ++ // SAFETY: each is a valid structure of its `s_type`. ++ unsafe { ++ match (*base).s_type { ++ vk::StructureType::IMPORT_MEMORY_HOST_POINTER_INFO_EXT => kind = "host-pages", ++ vk::StructureType::IMPORT_MEMORY_FD_INFO_KHR => { ++ let fd = &*base.cast::>(); ++ kind = match fd.handle_type { ++ vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT => "dma-buf", ++ _ => "fd-import", ++ }; ++ } ++ vk::StructureType::EXPORT_MEMORY_ALLOCATE_INFO if kind == "own" => { ++ kind = "own-export" ++ } ++ vk::StructureType::MEMORY_DEDICATED_ALLOCATE_INFO => { ++ let info = &*base.cast::>(); ++ dedicated = info.image != vk::Image::null(); ++ } ++ _ => {} ++ } ++ } ++ } ++ let kind = match (kind, dedicated) { ++ ("own", true) => "own-dedicated", ++ ("own-export", true) => "own-export-dedicated", ++ ("dma-buf", true) => "dma-buf-dedicated", ++ ("host-pages", true) => "host-pages-dedicated", ++ (kind, _) => kind, ++ }; ++ registry().memories.insert(memory.as_raw(), kind); ++} ++ ++/// Note `image` bound to `memory` at `offset` (`creation::bind_image`): the census's `bind` line. ++pub(crate) fn image_bound(image: vk::Image, memory: vk::DeviceMemory, offset: vk::DeviceSize) { ++ if !census_on() { ++ return; ++ } ++ let kind = registry() ++ .memories ++ .get(&memory.as_raw()) ++ .copied() ++ .unwrap_or("unknown"); ++ let line = format!( ++ "{},bind,0x{:x},,,,,,,,,,,,,,,,,,,,,,0x{:x},{},{}\n", ++ host_ns(), ++ image.as_raw(), ++ memory.as_raw(), ++ offset, ++ kind ++ ); ++ census_line(&line); ++} ++ ++/// Note `view`, made of `info` (`creation::image_view`). ++pub(crate) fn view_made(info: &vk::ImageViewCreateInfo<'_>, view: vk::ImageView) { ++ if !registry_on() { ++ return; ++ } ++ let entry = (info.image.as_raw(), info.subresource_range.base_mip_level); ++ registry().views.insert(view.as_raw(), entry); ++} ++ ++/// Note `pass`, made of `info` (`creation::render_pass`) at `site`. ++/// ++/// # Safety ++/// ++/// `info` is valid. ++pub(crate) unsafe fn render_pass_made( ++ info: &vk::RenderPassCreateInfo<'_>, ++ pass: vk::RenderPass, ++ site: &'static Location<'static>, ++) { ++ if !crate::pass_times::on() { ++ return; ++ } ++ // SAFETY: the caller's. ++ let (descriptions, subpasses) = unsafe { ++ ( ++ slice(info.p_attachments, info.attachment_count), ++ slice(info.p_subpasses, info.subpass_count), ++ ) ++ }; ++ let mut roles: Vec> = vec![None; descriptions.len()]; ++ if let Some(subpass) = subpasses.first() { ++ // SAFETY: the caller's: subpass 0's references are valid. ++ unsafe { ++ let colors = slice(subpass.p_color_attachments, subpass.color_attachment_count); ++ note_roles(&mut roles, colors.iter().map(|r| r.attachment), Role::Color); ++ if !subpass.p_resolve_attachments.is_null() { ++ let resolves = slice( ++ subpass.p_resolve_attachments, ++ subpass.color_attachment_count, ++ ); ++ note_roles( ++ &mut roles, ++ resolves.iter().map(|r| r.attachment), ++ Role::Resolve, ++ ); ++ } ++ let inputs = slice(subpass.p_input_attachments, subpass.input_attachment_count); ++ note_roles(&mut roles, inputs.iter().map(|r| r.attachment), Role::Input); ++ if let Some(depth) = subpass.p_depth_stencil_attachment.as_ref() { ++ note_role(&mut roles, depth.attachment, Role::DepthStencil); ++ } ++ } ++ } ++ let attachments = descriptions ++ .iter() ++ .enumerate() ++ .map(|(i, d)| PassAttachment { ++ role: roles[i].unwrap_or(Role::Other(i as u32)), ++ format: d.format, ++ samples: d.samples.as_raw(), ++ load: d.load_op, ++ store: d.store_op, ++ stencil_load: d.stencil_load_op, ++ stencil_store: d.stencil_store_op, ++ }) ++ .collect(); ++ let entry = PassEntry { attachments, site }; ++ registry().passes.insert(pass.as_raw(), Arc::new(entry)); ++} ++ ++/// Note `pass`, made of `info` (`creation::render_pass2`) at `site`. ++/// ++/// # Safety ++/// ++/// `info` and its subpasses' chains are valid. ++pub(crate) unsafe fn render_pass2_made( ++ info: &vk::RenderPassCreateInfo2<'_>, ++ pass: vk::RenderPass, ++ site: &'static Location<'static>, ++) { ++ if !crate::pass_times::on() { ++ return; ++ } ++ // SAFETY: the caller's. ++ let (descriptions, subpasses) = unsafe { ++ ( ++ slice(info.p_attachments, info.attachment_count), ++ slice(info.p_subpasses, info.subpass_count), ++ ) ++ }; ++ let mut roles: Vec> = vec![None; descriptions.len()]; ++ if let Some(subpass) = subpasses.first() { ++ // SAFETY: the caller's: subpass 0's references and chain are valid. ++ unsafe { ++ let colors = slice(subpass.p_color_attachments, subpass.color_attachment_count); ++ note_roles(&mut roles, colors.iter().map(|r| r.attachment), Role::Color); ++ if !subpass.p_resolve_attachments.is_null() { ++ let resolves = slice( ++ subpass.p_resolve_attachments, ++ subpass.color_attachment_count, ++ ); ++ note_roles( ++ &mut roles, ++ resolves.iter().map(|r| r.attachment), ++ Role::Resolve, ++ ); ++ } ++ let inputs = slice(subpass.p_input_attachments, subpass.input_attachment_count); ++ note_roles(&mut roles, inputs.iter().map(|r| r.attachment), Role::Input); ++ if let Some(depth) = subpass.p_depth_stencil_attachment.as_ref() { ++ note_role(&mut roles, depth.attachment, Role::DepthStencil); ++ } ++ for base in chain(subpass.p_next) { ++ if (*base).s_type == vk::StructureType::SUBPASS_DESCRIPTION_DEPTH_STENCIL_RESOLVE { ++ let resolve = ++ &*base.cast::>(); ++ if let Some(at) = resolve.p_depth_stencil_resolve_attachment.as_ref() { ++ note_role(&mut roles, at.attachment, Role::DepthStencilResolve); ++ } ++ } ++ } ++ } ++ } ++ let attachments = descriptions ++ .iter() ++ .enumerate() ++ .map(|(i, d)| PassAttachment { ++ role: roles[i].unwrap_or(Role::Other(i as u32)), ++ format: d.format, ++ samples: d.samples.as_raw(), ++ load: d.load_op, ++ store: d.store_op, ++ stencil_load: d.stencil_load_op, ++ stencil_store: d.stencil_store_op, ++ }) ++ .collect(); ++ let entry = PassEntry { attachments, site }; ++ registry().passes.insert(pass.as_raw(), Arc::new(entry)); ++} ++ ++/// Give each attachment of `indices` its role, `role` of its place among them. ++fn note_roles( ++ roles: &mut [Option], ++ indices: impl Iterator, ++ role: impl Fn(u32) -> Role, ++) { ++ for (place, index) in indices.enumerate() { ++ note_role(roles, index, role(place as u32)); ++ } ++} ++ ++/// Give attachment `index` `role`, where it is one and has none yet. ++fn note_role(roles: &mut [Option], index: u32, role: Role) { ++ if let Some(slot) = roles.get_mut(index as usize) { ++ slot.get_or_insert(role); ++ } ++} ++ ++/// Note `framebuffer`, made of `info` (`creation::framebuffer`). ++/// ++/// # Safety ++/// ++/// `info` is valid. ++pub(crate) unsafe fn framebuffer_made( ++ info: &vk::FramebufferCreateInfo<'_>, ++ framebuffer: vk::Framebuffer, ++) { ++ if !crate::pass_times::on() { ++ return; ++ } ++ let imageless = info.flags.contains(vk::FramebufferCreateFlags::IMAGELESS); ++ let views = match imageless { ++ true => Vec::new(), ++ // SAFETY: the caller's. ++ false => unsafe { slice(info.p_attachments, info.attachment_count) } ++ .iter() ++ .map(|view| view.as_raw()) ++ .collect(), ++ }; ++ let entry = FramebufferEntry { ++ width: info.width, ++ height: info.height, ++ views, ++ }; ++ registry() ++ .framebuffers ++ .insert(framebuffer.as_raw(), Arc::new(entry)); ++} ++ ++/// Forget the object `raw` released by `call` (`release::object`); an image's release is the ++/// census's `destroy` line. ++pub(crate) fn released(call: crate::exit_guard::Call, raw: u64) { ++ use crate::exit_guard::Call; ++ if !registry_on() { ++ return; ++ } ++ let mut registry = registry(); ++ match call { ++ Call::DestroyImage => { ++ registry.images.remove(&raw); ++ drop(registry); ++ if census_on() { ++ census_line(&format!( ++ "{},destroy,0x{raw:x},,,,,,,,,,,,,,,,,,,,,,,,\n", ++ host_ns() ++ )); ++ } ++ } ++ Call::DestroyImageView => { ++ registry.views.remove(&raw); ++ } ++ Call::DestroyRenderPass => { ++ registry.passes.remove(&raw); ++ } ++ Call::DestroyFramebuffer => { ++ registry.framebuffers.remove(&raw); ++ } ++ Call::FreeMemory => { ++ registry.memories.remove(&raw); ++ } ++ _ => {} ++ } ++} ++ ++/// One attachment of a render pass instance, resolved as it begins. ++struct Attached { ++ pass: PassAttachment, ++ extent: Option<(u32, u32)>, ++ memoryless: bool, ++ purpose: Option>, ++} ++ ++/// What the pass timer records of a render pass instance's objects as it begins: its attachments, ++/// each with its image's extent, whether it is memoryless (a transient attachment) and its purpose, ++/// and where its pass was made. Formatted only once its command buffer completed ([`format`]). ++pub(crate) struct PassLabel { ++ attached: Vec, ++ site: Option<&'static Location<'static>>, ++ extent: (u32, u32), ++} ++ ++impl PassLabel { ++ /// Where the pass was made, short. ++ pub(crate) fn site(&self) -> Option { ++ self.site.map(short) ++ } ++ ++ /// The attachments, `|`-joined: each its role, format, extent, samples, load/store (and the ++ /// stencil's where it has one), `memoryless`, and `@` its image's purpose. ++ pub(crate) fn format(&self) -> String { ++ let mut out = String::new(); ++ for (i, a) in self.attached.iter().enumerate() { ++ if i > 0 { ++ out.push('|'); ++ } ++ let p = &a.pass; ++ let _ = match p.role { ++ Role::Color(j) => write!(out, "c{j}"), ++ Role::Resolve(j) => write!(out, "r{j}"), ++ Role::DepthStencil => write!(out, "ds"), ++ Role::DepthStencilResolve => write!(out, "dsr"), ++ Role::Input(j) => write!(out, "in{j}"), ++ Role::Other(j) => write!(out, "a{j}"), ++ }; ++ let (w, h) = a.extent.unwrap_or(self.extent); ++ let _ = write!( ++ out, ++ ":{}:{w}x{h}:s{}:{}/{}", ++ format_name(p.format), ++ p.samples, ++ load_name(p.load), ++ store_name(p.store) ++ ); ++ if matches!(p.role, Role::DepthStencil | Role::DepthStencilResolve) ++ && has_stencil(p.format) ++ { ++ let _ = write!( ++ out, ++ ":st={}/{}", ++ load_name(p.stencil_load), ++ store_name(p.stencil_store) ++ ); ++ } ++ if a.memoryless { ++ out.push_str(":memoryless"); ++ } ++ if let Some(purpose) = &a.purpose { ++ let _ = write!(out, "@{purpose}"); ++ } ++ } ++ if out.is_empty() { ++ out.push('-'); ++ } ++ out.replace(',', ";") ++ } ++} ++ ++/// Whether `format` has a stencil aspect. ++fn has_stencil(format: vk::Format) -> bool { ++ matches!( ++ format, ++ vk::Format::S8_UINT ++ | vk::Format::D16_UNORM_S8_UINT ++ | vk::Format::D24_UNORM_S8_UINT ++ | vk::Format::D32_SFLOAT_S8_UINT ++ ) ++} ++ ++/// The label of a render pass instance of `pass` and `framebuffer`, of `area`, as it begins. ++pub(crate) fn pass_label( ++ pass: vk::RenderPass, ++ framebuffer: vk::Framebuffer, ++ area: vk::Extent2D, ++) -> PassLabel { ++ let registry = registry(); ++ let pass = registry.passes.get(&pass.as_raw()).cloned(); ++ let framebuffer = registry.framebuffers.get(&framebuffer.as_raw()).cloned(); ++ let extent = framebuffer ++ .as_ref() ++ .map_or((area.width, area.height), |f| (f.width, f.height)); ++ let Some(pass) = pass else { ++ return PassLabel { ++ attached: Vec::new(), ++ site: None, ++ extent, ++ }; ++ }; ++ let attached = pass ++ .attachments ++ .iter() ++ .enumerate() ++ .map(|(i, attachment)| { ++ let view = framebuffer.as_ref().and_then(|f| f.views.get(i)).copied(); ++ let image = view.and_then(|v| registry.views.get(&v)).copied(); ++ let entry = image.and_then(|(image, mip)| { ++ registry ++ .images ++ .get(&image) ++ .map(|entry| (Arc::clone(entry), mip)) ++ }); ++ match entry { ++ Some((entry, mip)) => Attached { ++ pass: *attachment, ++ extent: Some(( ++ (entry.extent.width >> mip).max(1), ++ (entry.extent.height >> mip).max(1), ++ )), ++ memoryless: entry ++ .usage ++ .contains(vk::ImageUsageFlags::TRANSIENT_ATTACHMENT), ++ purpose: Some(Arc::clone(&entry.purpose)), ++ }, ++ None => Attached { ++ pass: *attachment, ++ extent: None, ++ memoryless: false, ++ purpose: None, ++ }, ++ } ++ }) ++ .collect(); ++ PassLabel { ++ attached, ++ site: Some(pass.site), ++ extent, ++ } ++} +diff --git a/crates/metal2vulkan-runtime/src/lib.rs b/crates/metal2vulkan-runtime/src/lib.rs +index ca076571..9cdcc6a5 100644 +--- a/crates/metal2vulkan-runtime/src/lib.rs ++++ b/crates/metal2vulkan-runtime/src/lib.rs +@@ -125,6 +125,8 @@ mod host_rt; + #[cfg(unix)] + mod host_shared_pointers; + mod image_barrier; ++// LOCAL (Atlas) diagnostic: the image census and object registry (M2V_IMAGE_LOG). ++mod image_log; + mod image_plan; + mod in_place; + mod in_place_check; +@@ -158,6 +160,8 @@ mod nan_clear; + mod neutral; + mod noted_apart; + mod owned_kernel; ++// LOCAL (Atlas) diagnostic: the pass timer (M2V_PASS_TIMES). ++mod pass_times; + mod pending_dispatch; + mod pipeline; + mod pixel_format_info; +diff --git a/crates/metal2vulkan-runtime/src/pass_times.rs b/crates/metal2vulkan-runtime/src/pass_times.rs +new file mode 100644 +index 00000000..ca083acb +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/pass_times.rs +@@ -0,0 +1,1194 @@ ++//! LOCAL (Atlas) diagnostic: the pass timer, `M2V_PASS_TIMES=`. GPU timestamps around each ++//! unit of GPU work the runtime records, read once its command buffer completed, one CSV line per ++//! unit appended to `/passes-.csv`. Default off; the whole module can be dropped with its ++//! hooks (each marked `LOCAL (Atlas) diagnostic`). ++//! ++//! - **The switch** is read once. Unset (or empty), nothing changes: no query pool, no command, ++//! no line; each hook is one check of a `OnceLock`. ++//! - **The window.** Set, a command buffer is captured only where the file `/on` exists as the ++//! runtime begins recording it (its first encoder's opening, or its first `VkCommandBuffer` ++//! handed out): checked once per command buffer. `touch /on` opens a window, `rm` closes ++//! it; the first command buffer begun after the close flushes the lines held. ++//! - **Where the timestamps go.** Every routed command passes the recorder's entry points ++//! (`recorder/thunks.rs`), whatever the runtime records it from (an encoder, its own kernels, a ++//! pass's steps, a commit's work, the lowering thread), so the units are cut there, on each ++//! `VkCommandBuffer` the captured command buffer handed out: a render pass instance ++//! (`vkCmdBeginRenderPass` to `vkCmdEndRenderPass`, each instance of a split pass its own); a run ++//! of dispatches; a run of transfer commands (copies, blits, clears, fills, updates, query result ++//! copies). A run goes on while the commands are of its class and its label (below) and the ++//! barriers and binds between are part of it; another unit, or the `VkCommandBuffer`'s end, ends ++//! it. Each unit writes a `vkCmdWriteTimestamp` at `ALL_COMMANDS` before its first command and ++//! after its last, into two queries of a block of the diagnostic's own pools ([`BLOCK`] queries, ++//! [`POOL_BLOCKS`] blocks a pool), each block reset on the host as a command buffer takes it ++//! (`hostQueryReset`, which every device the runtime made enables), else by a ++//! `vkCmdResetQueryPool` before the unit's start. The scheduler's own command buffers of ++//! barriers (and the runtime's GPU-time timestamps) are no unit. ++//! - **Labels**, all known where the unit is recorded: ++//! - `kind`: `render-pass` (an instance of a render encoder's pass: inline first, inline after a ++//! split, or in secondaries; `render/inline_first.rs`, `inline_splits.rs`, `pass_steps.rs`), ++//! `render-pass-resolve` (the pass's instance of no draws that resolves it), `pass:` ++//! (another render pass the runtime records, at the place it began it where that goes through ++//! the device's `Vk` door, else where its `VkRenderPass` was made); `guest-` (a ++//! command of the program's encoder: `EncoderCore::command_*`, a pending dispatch), suffixed ++//! with its class where that is not the encoder's own (`guest-blit:compute`); else ++//! `internal:/`, the place that recorded its first command where it went ++//! through the device's `Vk` door (`device/functions.rs`), else the command alone. ++//! - `cb_label`: the command queue's label (no program's label reaches the runtime: Metal's ++//! `label` of a command buffer and of an encoder are not part of Reims's guest protocol, and ++//! the runtime keeps neither). `encoder_label`: the encoder's kind and place in its command ++//! buffer (`render#3`), for the `VkCommandBuffer`s handed out while it was the last opened. ++//! - `attachments`, for a render pass instance: each attachment's role in subpass 0 ++//! (`c0`, `r0`, `ds`, `dsr`, `in0`), format, extent, samples, load/store ops (the stencil's ++//! too), `memoryless` where its image is a transient attachment, and `@` its image's purpose ++//! (`image_log.rs`: the program's call that made it, else the place inside the runtime). ++//! - `split`, for a render encoder's instance: `i/n` among the encoder's instances, the split ++//! cause that began it (`work_counts::Split`; `first` for the first, `run` past the causes ++//! counted), how it records (`inline`, `inline-split`, `secondary`), and for the first, why ++//! the pass's first instance did not record inline (`work_counts::NotInline`). ++//! - `draws`: the draws of a render pass instance (its secondaries' too), the dispatches of a ++//! run of them, the commands of a run of transfers. ++//! - **Read as it completed**, in the completion turn after its GPU times (`completion.rs`): ++//! `vkGetQueryPoolResults` per block, without waiting, a unit whose timestamps are not both ++//! available (a command buffer that failed or never ran) has no `gpu_ns`. `host_ns` is the ++//! unit's start on the host's clock (`CLOCK_MONOTONIC_RAW`): placed by the calibration pair the ++//! command buffer's GPU times took where the device has one, else from its placed GPU start. ++//! The lines are held and appended every [`FLUSH_LINES`] lines or [`FLUSH_AFTER`] at the next ++//! completion, as a window closes, and as the device drops. ++ ++use crate::command_buffer::CommandBuffer; ++use crate::exit_guard::{self, Call}; ++use crate::gpu_time::{GpuClock, Pair}; ++use crate::image_log::{self, PassLabel}; ++use crate::used_resources::EncoderKind; ++use crate::work_counts::{NotInline, Split}; ++use crate::{CommandQueue, Device}; ++use ash::vk; ++use ash::vk::Handle; ++use std::cell::Cell; ++use std::collections::HashMap; ++use std::fmt::Write as _; ++use std::io::Write as _; ++use std::panic::Location; ++use std::path::{Path, PathBuf}; ++use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; ++use std::sync::{Arc, LazyLock, Mutex, MutexGuard, OnceLock, Weak}; ++use std::time::{Duration, Instant}; ++ ++/// Queries of a block a command buffer takes at once: 32 units. ++const BLOCK: u32 = 64; ++ ++/// Blocks of a query pool. ++const POOL_BLOCKS: u32 = 64; ++ ++/// Lines held before they are appended. ++const FLUSH_LINES: usize = 256; ++ ++/// Time after which the next completion appends the lines held. ++const FLUSH_AFTER: Duration = Duration::from_millis(500); ++ ++/// The CSV's columns, its first line. ++const HEADER: &str = ++ "host_ns,cb_seq,unit_seq,kind,cb_label,encoder_label,attachments,split,draws,gpu_ns\n"; ++ ++/// The directory `M2V_PASS_TIMES` names: read once. ++fn dir() -> Option<&'static Path> { ++ static DIR: OnceLock> = OnceLock::new(); ++ DIR.get_or_init(|| { ++ std::env::var_os("M2V_PASS_TIMES") ++ .filter(|value| !value.is_empty()) ++ .map(PathBuf::from) ++ }) ++ .as_deref() ++} ++ ++/// Whether the pass timer is on. ++#[inline] ++pub(crate) fn on() -> bool { ++ dir().is_some() ++} ++ ++fn lock(mutex: &Mutex) -> MutexGuard<'_, T> { ++ mutex ++ .lock() ++ .unwrap_or_else(std::sync::PoisonError::into_inner) ++} ++ ++/// The captures alive: the entry points do nothing more while there is none. ++static LIVE: AtomicUsize = AtomicUsize::new(0); ++ ++/// The next captured command buffer's number. ++static NEXT_SEQ: AtomicU64 = AtomicU64::new(0); ++ ++/// Each `VkCommandBuffer` a captured command buffer handed out, to its capture. ++static RAWS: LazyLock>>> = LazyLock::new(Mutex::default); ++ ++/// Each captured command buffer, by its address, to its capture. ++static BY_KEY: LazyLock>>> = LazyLock::new(Mutex::default); ++ ++/// What the code recording a unit says it is, on its thread ([`scope`]). ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub(crate) enum Tag { ++ /// A command of the program's encoder. ++ Guest, ++ /// A render pass instance of a render encoder's pass, begun at the given place. ++ Pass(PassAt), ++} ++ ++/// Where a render encoder's pass instance began. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub(crate) enum PassAt { ++ /// Its first instance, inline (`render/inline_first.rs`). ++ Inline, ++ /// An instance after a split, inline (`render/inline_splits.rs`). ++ InlineSplit, ++ /// An instance whose draws a secondary holds (`render/pass_steps.rs`). ++ Instance, ++ /// The instance of no draws that resolves the pass (`render/pass_steps.rs`). ++ Resolve, ++} ++ ++thread_local! { ++ static SCOPE: Cell> = const { Cell::new(None) }; ++ static SITE: Cell>> = const { Cell::new(None) }; ++} ++ ++/// A tag set on this thread until dropped ([`scope`]). ++pub(crate) struct Scope(Option>); ++ ++impl Drop for Scope { ++ fn drop(&mut self) { ++ if let Some(previous) = self.0.take() { ++ SCOPE.with(|scope| scope.set(previous)); ++ } ++ } ++} ++ ++/// Tag what this thread records with `tag` until the scope drops: the innermost holds. ++pub(crate) fn scope(tag: Tag) -> Scope { ++ if !on() { ++ return Scope(None); ++ } ++ Scope(Some(SCOPE.with(|scope| scope.replace(Some(tag))))) ++} ++ ++/// The place `at` records the next command on this thread (`Vk`'s commands, ++/// `device/functions.rs`); the command's entry point takes it. ++#[inline] ++pub(crate) fn site(at: &'static Location<'static>) { ++ if on() { ++ SITE.with(|site| site.set(Some(at))); ++ } ++} ++ ++/// A unit's class. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub(crate) enum Class { ++ Pass, ++ Compute, ++ Transfer, ++} ++ ++/// A slot's own timestamp commands (`recorder/thunks.rs`), which the units record through: into a ++/// deferred command buffer's log as any command, else the driver's. ++#[derive(Clone, Copy)] ++pub(crate) struct Stamps { ++ pub(crate) write: ++ unsafe extern "system" fn(vk::CommandBuffer, vk::PipelineStageFlags, vk::QueryPool, u32), ++ pub(crate) reset: unsafe extern "system" fn(vk::CommandBuffer, vk::QueryPool, u32, u32), ++} ++ ++/// A block of queries. ++#[derive(Clone, Copy)] ++struct Block { ++ pool: vk::QueryPool, ++ first: u32, ++} ++ ++#[derive(Default)] ++struct Blocks { ++ pools: Vec, ++ free: Vec, ++ /// The device refused a pool: no more are asked for. ++ refused: bool, ++} ++ ++/// The lines held, and where they go. ++struct Out { ++ path: PathBuf, ++ file: Option, ++ held: String, ++ lines: usize, ++ flushed: Instant, ++} ++ ++impl Out { ++ fn flush(&mut self) { ++ self.flushed = Instant::now(); ++ if self.held.is_empty() { ++ return; ++ } ++ if self.file.is_none() { ++ let opened = std::fs::OpenOptions::new() ++ .create(true) ++ .append(true) ++ .open(&self.path); ++ match opened { ++ Ok(mut file) => { ++ if file.metadata().map(|m| m.len() == 0).unwrap_or(false) { ++ let _ = file.write_all(HEADER.as_bytes()); ++ } ++ self.file = Some(file); ++ } ++ Err(error) => { ++ eprintln!( ++ "metal2vulkan: M2V_PASS_TIMES: cannot open {:?}: {error}", ++ self.path ++ ); ++ self.held.clear(); ++ self.lines = 0; ++ return; ++ } ++ } ++ } ++ if let Some(file) = self.file.as_mut() { ++ let _ = file.write_all(self.held.as_bytes()); ++ } ++ self.held.clear(); ++ self.lines = 0; ++ } ++} ++ ++/// A device's pass timer: its query pools and its lines (`DeviceShared::pass_times`). ++pub(crate) struct DeviceTimes { ++ host_reset: bool, ++ blocks: Mutex, ++ out: Mutex, ++} ++ ++impl DeviceTimes { ++ /// The pass timer of a device whose queue family writes timestamps where `timestamps`, which ++ /// resets queries on the host where `host_reset`: `None` while the switch is off. ++ pub(crate) fn new(timestamps: bool, host_reset: bool) -> Option> { ++ let dir = dir()?; ++ if !timestamps { ++ eprintln!("metal2vulkan: M2V_PASS_TIMES: the device's queue writes no timestamps"); ++ return None; ++ } ++ let path = dir.join(format!("passes-{}.csv", std::process::id())); ++ Some(Arc::new(DeviceTimes { ++ host_reset, ++ blocks: Mutex::default(), ++ out: Mutex::new(Out { ++ path, ++ file: None, ++ held: String::new(), ++ lines: 0, ++ flushed: Instant::now(), ++ }), ++ })) ++ } ++ ++ /// A block of queries, reset: a free one, else one of a pool made now; `None` once the device ++ /// refused a pool. ++ /// ++ /// # Safety ++ /// ++ /// `device` is the live device of this timer. ++ unsafe fn take(&self, device: &ash::Device) -> Option { ++ let mut blocks = lock(&self.blocks); ++ let block = match blocks.free.pop() { ++ Some(block) => block, ++ None => { ++ if blocks.refused { ++ return None; ++ } ++ let info = vk::QueryPoolCreateInfo::default() ++ .query_type(vk::QueryType::TIMESTAMP) ++ .query_count(BLOCK * POOL_BLOCKS); ++ // SAFETY: the caller's; `info` is valid. ++ let made = unsafe { device.create_query_pool(&info, None) }; ++ exit_guard::called(Call::CreateQueryPool); ++ let Ok(pool) = made else { ++ blocks.refused = true; ++ eprintln!("metal2vulkan: M2V_PASS_TIMES: the device refused a query pool"); ++ return None; ++ }; ++ blocks.pools.push(pool); ++ let rest = (1..POOL_BLOCKS).rev().map(|i| Block { ++ pool, ++ first: i * BLOCK, ++ }); ++ blocks.free.extend(rest); ++ Block { pool, first: 0 } ++ } ++ }; ++ drop(blocks); ++ if self.host_reset { ++ // SAFETY: the device enables `hostQueryReset`; the block is free: its last holder's ++ // work completed, or never ran, and nothing else uses it. ++ unsafe { device.reset_query_pool(block.pool, block.first, BLOCK) }; ++ exit_guard::called(Call::ResetQueryPool); ++ } ++ Some(block) ++ } ++ ++ fn give_back(&self, blocks: Vec) { ++ if !blocks.is_empty() { ++ lock(&self.blocks).free.extend(blocks); ++ } ++ } ++ ++ fn append(&self, text: &str, lines: usize) { ++ let mut out = lock(&self.out); ++ out.held.push_str(text); ++ out.lines += lines; ++ if out.lines >= FLUSH_LINES || out.flushed.elapsed() >= FLUSH_AFTER { ++ out.flush(); ++ } ++ } ++ ++ /// Append the lines held. ++ pub(crate) fn flush(&self) { ++ lock(&self.out).flush(); ++ } ++ ++ /// Release the pools, at the device's teardown, its lines appended first. ++ /// ++ /// # Safety ++ /// ++ /// `device` is the live device of this timer, and none of its command buffers is pending. ++ pub(crate) unsafe fn release(&self, device: &ash::Device) { ++ self.flush(); ++ let pools = { ++ let mut blocks = lock(&self.blocks); ++ blocks.free.clear(); ++ blocks.refused = true; ++ std::mem::take(&mut blocks.pools) ++ }; ++ for pool in pools { ++ // SAFETY: the caller's. ++ unsafe { crate::release::object(device, pool) }; ++ } ++ } ++} ++ ++impl Drop for DeviceTimes { ++ fn drop(&mut self) { ++ self.flush(); ++ } ++} ++ ++/// A command buffer's capture, its state's (`State::pass_times`): undecided until it begins ++/// recording, then captured or not. ++#[derive(Default)] ++pub(crate) struct CbSlot { ++ decided: bool, ++ capture: Option>, ++} ++ ++impl CbSlot { ++ /// Its capture, taken to be read. ++ pub(crate) fn take(&mut self) -> Option> { ++ self.capture.take() ++ } ++} ++ ++/// What an encoder of a captured command buffer told. ++struct EncoderNote { ++ kind: EncoderKind, ++ causes: Vec, ++ not_inline: Option, ++} ++ ++/// A `VkCommandBuffer` of a captured command buffer. ++#[derive(Default)] ++struct RawState { ++ /// The encoder last opened as it was handed out. ++ encoder: Option, ++ /// Its unit open. ++ open: Option, ++ /// A secondary's draws not yet in a unit. ++ pending_draws: u32, ++ /// The render pass instance that executed it, a secondary. ++ parent: Option, ++} ++ ++/// A unit of work. ++struct Unit { ++ class: Class, ++ tag: Option, ++ site: Option<&'static Location<'static>>, ++ command: &'static str, ++ encoder: Option, ++ label: Option, ++ count: u32, ++ /// Its block's index in the capture's and its start query's offset in it. ++ block: usize, ++ offset: u32, ++} ++ ++#[derive(Default)] ++struct CaptureState { ++ units: Vec, ++ blocks: Vec, ++ used: u32, ++ out_of_queries: bool, ++ encoders: Vec, ++ current: Option, ++ raws: HashMap, ++} ++ ++/// A captured command buffer. ++pub(crate) struct Capture { ++ times: Arc, ++ device: Device, ++ seq: u64, ++ key: usize, ++ queue_label: String, ++ state: Mutex, ++} ++ ++impl Drop for Capture { ++ fn drop(&mut self) { ++ let me: *const Capture = self; ++ let state = self ++ .state ++ .get_mut() ++ .unwrap_or_else(std::sync::PoisonError::into_inner); ++ { ++ let mut raws = lock(&RAWS); ++ for raw in state.raws.keys() { ++ if raws.get(raw).is_some_and(|w| std::ptr::eq(w.as_ptr(), me)) { ++ raws.remove(raw); ++ } ++ } ++ } ++ { ++ let mut keys = lock(&BY_KEY); ++ if keys ++ .get(&self.key) ++ .is_some_and(|w| std::ptr::eq(w.as_ptr(), me)) ++ { ++ keys.remove(&self.key); ++ } ++ } ++ self.times.give_back(std::mem::take(&mut state.blocks)); ++ LIVE.fetch_sub(1, Ordering::Relaxed); ++ } ++} ++ ++impl CaptureState { ++ /// Two queries for a unit: its block's index and the start's offset. ++ fn queries(&mut self, capture: &Capture) -> Option<(usize, u32)> { ++ if self.out_of_queries { ++ return None; ++ } ++ if self.blocks.is_empty() || self.used + 2 > BLOCK { ++ let device = capture.device.shared().vk(); ++ // SAFETY: the capture's device is live. ++ match unsafe { capture.times.take(device) } { ++ Some(block) => { ++ self.blocks.push(block); ++ self.used = 0; ++ } ++ None => { ++ self.out_of_queries = true; ++ return None; ++ } ++ } ++ } ++ let offset = self.used; ++ self.used += 2; ++ Some((self.blocks.len() - 1, offset)) ++ } ++} ++ ++/// Begin `slot`'s capture where it is undecided: a capture where the device's timer is on and ++/// `/on` exists. ++fn decide<'a>( ++ slot: &'a mut CbSlot, ++ device: &Device, ++ queue: &CommandQueue, ++ key: usize, ++) -> Option<&'a Arc> { ++ if !slot.decided { ++ slot.decided = true; ++ slot.capture = start(device, queue, key); ++ } ++ slot.capture.as_ref() ++} ++ ++fn start(device: &Device, queue: &CommandQueue, key: usize) -> Option> { ++ let times = device.shared().pass_times.as_ref()?; ++ let dir = dir()?; ++ if !dir.join("on").exists() { ++ times.flush(); ++ return None; ++ } ++ let label = queue.label().unwrap_or_else(|| "-".to_owned()); ++ let capture = Arc::new(Capture { ++ times: Arc::clone(times), ++ device: device.clone(), ++ seq: NEXT_SEQ.fetch_add(1, Ordering::Relaxed), ++ key, ++ queue_label: label.replace(',', ";"), ++ state: Mutex::default(), ++ }); ++ LIVE.fetch_add(1, Ordering::Relaxed); ++ lock(&BY_KEY).insert(key, Arc::downgrade(&capture)); ++ Some(capture) ++} ++ ++/// An encoder of `kind` opens on the command buffer at `key` of `device` and `queue`, whose capture ++/// `slot` holds (`EncoderCore::open`): `VkCommandBuffer`s handed out from now on are its. ++pub(crate) fn encoder_opened( ++ device: &Device, ++ queue: &CommandQueue, ++ key: usize, ++ slot: &mut CbSlot, ++ kind: EncoderKind, ++) { ++ if !on() { ++ return; ++ } ++ let Some(capture) = decide(slot, device, queue, key) else { ++ return; ++ }; ++ let mut state = lock(&capture.state); ++ state.current = Some(state.encoders.len() as u32); ++ state.encoders.push(EncoderNote { ++ kind, ++ causes: Vec::new(), ++ not_inline: None, ++ }); ++} ++ ++/// The command buffer at `key` of `device` and `queue`, whose capture `slot` holds, handed out ++/// `raw` (`CommandBufferShared::allocate_deferred` and kin). ++pub(crate) fn handed_out( ++ device: &Device, ++ queue: &CommandQueue, ++ key: usize, ++ slot: &mut CbSlot, ++ raw: vk::CommandBuffer, ++) { ++ if !on() { ++ return; ++ } ++ let Some(capture) = decide(slot, device, queue, key) else { ++ // A handle a capture still alive handed out before goes to no capture now. ++ if LIVE.load(Ordering::Relaxed) > 0 { ++ lock(&RAWS).remove(&raw.as_raw()); ++ } ++ return; ++ }; ++ { ++ let mut state = lock(&capture.state); ++ let encoder = state.current; ++ state.raws.insert( ++ raw.as_raw(), ++ RawState { ++ encoder, ++ ..RawState::default() ++ }, ++ ); ++ } ++ lock(&RAWS).insert(raw.as_raw(), Arc::downgrade(capture)); ++} ++ ++/// The capture of the command buffer at `key`. ++fn by_key(key: usize) -> Option> { ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return None; ++ } ++ lock(&BY_KEY).get(&key)?.upgrade() ++} ++ ++/// A render encoder of `command_buffer` split its pass for `cause` (`render.rs`). ++pub(crate) fn note_split(command_buffer: &CommandBuffer, cause: Split) { ++ if !on() { ++ return; ++ } ++ let Some(capture) = by_key(command_buffer.pass_times_key()) else { ++ return; ++ }; ++ let mut state = lock(&capture.state); ++ if let Some(at) = state.current { ++ if let Some(note) = state.encoders.get_mut(at as usize) { ++ note.causes.push(cause); ++ } ++ } ++} ++ ++/// A render encoder of `command_buffer` did not record its pass's first instance inline, for ++/// `why` (`render.rs`, `render/inline_first.rs`). ++pub(crate) fn note_not_inline(command_buffer: &CommandBuffer, why: NotInline) { ++ if !on() { ++ return; ++ } ++ let Some(capture) = by_key(command_buffer.pass_times_key()) else { ++ return; ++ }; ++ let mut state = lock(&capture.state); ++ if let Some(at) = state.current { ++ if let Some(note) = state.encoders.get_mut(at as usize) { ++ note.not_inline.get_or_insert(why); ++ } ++ } ++} ++ ++/// The capture `raw` was handed out by. ++fn capture_of(raw: vk::CommandBuffer) -> Option> { ++ lock(&RAWS).get(&raw.as_raw())?.upgrade() ++} ++ ++/// Open a unit on `cb`: its start written. ++/// ++/// # Safety ++/// ++/// `cb` is recording outside a render pass instance, by the caller alone; `st` are its slot's. ++unsafe fn open( ++ st: Stamps, ++ cb: vk::CommandBuffer, ++ capture: &Capture, ++ state: &mut CaptureState, ++ mut unit: Unit, ++) -> Option { ++ let (block, offset) = state.queries(capture)?; ++ let Block { pool, first } = state.blocks[block]; ++ // SAFETY: the caller's; the queries are the capture's, reset on the host or here. ++ unsafe { ++ if !capture.times.host_reset { ++ (st.reset)(cb, pool, first + offset, 2); ++ } ++ (st.write)( ++ cb, ++ vk::PipelineStageFlags::ALL_COMMANDS, ++ pool, ++ first + offset, ++ ); ++ } ++ unit.block = block; ++ unit.offset = offset; ++ state.units.push(unit); ++ Some(state.units.len() - 1) ++} ++ ++/// Close unit `index` on `cb`: its end written. ++/// ++/// # Safety ++/// ++/// As [`open`]'s. ++unsafe fn close(st: Stamps, cb: vk::CommandBuffer, state: &CaptureState, index: usize) { ++ let unit = &state.units[index]; ++ let Block { pool, first } = state.blocks[unit.block]; ++ // SAFETY: the caller's. ++ unsafe { ++ (st.write)( ++ cb, ++ vk::PipelineStageFlags::ALL_COMMANDS, ++ pool, ++ first + unit.offset + 1, ++ ) ++ }; ++} ++ ++/// Before `vkCmdBeginRenderPass` of `begin` on `cb`: the open run closed, a render pass unit ++/// opened. ++/// ++/// # Safety ++/// ++/// As the command's: `cb` recording outside a render pass, `begin` valid; `st` are its slot's. ++pub(crate) unsafe fn begin_pass( ++ st: Stamps, ++ cb: vk::CommandBuffer, ++ begin: *const vk::RenderPassBeginInfo<'_>, ++) { ++ let site = SITE.with(Cell::take); ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let tag = SCOPE.with(Cell::get); ++ // SAFETY: the caller's. ++ let label = unsafe { begin.as_ref() }.map(|begin| { ++ image_log::pass_label( ++ begin.render_pass, ++ begin.framebuffer, ++ begin.render_area.extent, ++ ) ++ }); ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(raw) = state.raws.get_mut(&cb.as_raw()) else { ++ return; ++ }; ++ let encoder = raw.encoder; ++ if let Some(open) = raw.open.take() { ++ // SAFETY: the caller's. ++ unsafe { close(st, cb, state, open) }; ++ } ++ let unit = Unit { ++ class: Class::Pass, ++ tag, ++ site, ++ command: "vkCmdBeginRenderPass", ++ encoder, ++ label, ++ count: 0, ++ block: 0, ++ offset: 0, ++ }; ++ // SAFETY: the caller's. ++ let opened = unsafe { open(st, cb, &capture, state, unit) }; ++ if let Some(raw) = state.raws.get_mut(&cb.as_raw()) { ++ raw.open = opened; ++ } ++} ++ ++/// After `vkCmdEndRenderPass` on `cb`: its render pass unit closed. ++/// ++/// # Safety ++/// ++/// As the command's; `st` are its slot's. ++pub(crate) unsafe fn end_pass(st: Stamps, cb: vk::CommandBuffer) { ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(open) = state.raws.get_mut(&cb.as_raw()).and_then(|r| r.open.take()) else { ++ return; ++ }; ++ // SAFETY: the caller's: the pass ended. ++ unsafe { close(st, cb, state, open) }; ++} ++ ++/// Before `vkEndCommandBuffer` of `cb`: its open unit closed. ++/// ++/// # Safety ++/// ++/// As the command's; `st` are its slot's. ++pub(crate) unsafe fn end(st: Stamps, cb: vk::CommandBuffer) { ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(open) = state.raws.get_mut(&cb.as_raw()).and_then(|r| r.open.take()) else { ++ return; ++ }; ++ // SAFETY: the caller's. ++ unsafe { close(st, cb, state, open) }; ++} ++ ++/// Before a dispatch or transfer `command` of `class` on `cb`: counted in the open run where it ++/// is of its class and label, else the run closed and another opened. ++/// ++/// # Safety ++/// ++/// As the command's: `cb` recording outside a render pass instance; `st` are its slot's. ++pub(crate) unsafe fn work(st: Stamps, cb: vk::CommandBuffer, class: Class, command: &'static str) { ++ let site = SITE.with(Cell::take); ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let tag = SCOPE.with(Cell::get); ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(raw) = state.raws.get_mut(&cb.as_raw()) else { ++ return; ++ }; ++ let encoder = raw.encoder; ++ // The program's own commands of its encoder's class run on whatever records them; other ++ // work runs by its place. ++ let kind = encoder ++ .and_then(|e| state.encoders.get(e as usize)) ++ .map(|n| n.kind); ++ let natural = tag == Some(Tag::Guest) && natural(kind, class); ++ if let Some(at) = raw.open { ++ let unit = &mut state.units[at]; ++ let same = unit.class == class ++ && unit.tag == tag ++ && unit.encoder == encoder ++ && (natural || (unit.site == site && unit.command == command)); ++ // A command inside a render pass instance (none of these may be) counts in it. ++ if same || unit.class == Class::Pass { ++ unit.count += 1; ++ return; ++ } ++ raw.open = None; ++ // SAFETY: the caller's. ++ unsafe { close(st, cb, state, at) }; ++ } ++ let unit = Unit { ++ class, ++ tag, ++ site, ++ command, ++ encoder, ++ label: None, ++ count: 1, ++ block: 0, ++ offset: 0, ++ }; ++ // SAFETY: the caller's. ++ let opened = unsafe { open(st, cb, &capture, state, unit) }; ++ if let Some(raw) = state.raws.get_mut(&cb.as_raw()) { ++ raw.open = opened; ++ } ++} ++ ++/// Before a draw on `cb`: counted in its render pass instance, or the one that executes it. ++pub(crate) fn draw(cb: vk::CommandBuffer) { ++ if LIVE.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(raw) = state.raws.get_mut(&cb.as_raw()) else { ++ return; ++ }; ++ match raw.open.or(raw.parent) { ++ Some(at) => state.units[at].count += 1, ++ None => raw.pending_draws += 1, ++ } ++} ++ ++/// Before `vkCmdExecuteGeneratedCommandsEXT` on `cb`: a draw inside a render pass instance, else a ++/// dispatch. ++/// ++/// # Safety ++/// ++/// As [`work`]'s. ++pub(crate) unsafe fn generated(st: Stamps, cb: vk::CommandBuffer, command: &'static str) { ++ let in_pass = LIVE.load(Ordering::Relaxed) > 0 ++ && capture_of(cb).is_some_and(|capture| { ++ let state = lock(&capture.state); ++ let open = state.raws.get(&cb.as_raw()).and_then(|r| r.open); ++ let in_pass = open.is_some_and(|at| state.units[at].class == Class::Pass); ++ in_pass ++ }); ++ match in_pass { ++ true => { ++ SITE.with(Cell::take); ++ draw(cb); ++ } ++ // SAFETY: the caller's. ++ false => unsafe { work(st, cb, Class::Compute, command) }, ++ } ++} ++ ++/// Before `vkCmdExecuteCommands` of `count` secondaries at `buffers` on `cb`: their draws counted ++/// in its render pass instance, and their later ones. ++/// ++/// # Safety ++/// ++/// `buffers` points at `count` command buffers. ++pub(crate) unsafe fn execute(cb: vk::CommandBuffer, count: u32, buffers: *const vk::CommandBuffer) { ++ if LIVE.load(Ordering::Relaxed) == 0 || buffers.is_null() { ++ return; ++ } ++ let Some(capture) = capture_of(cb) else { ++ return; ++ }; ++ let mut guard = lock(&capture.state); ++ let state = &mut *guard; ++ let Some(at) = state.raws.get(&cb.as_raw()).and_then(|r| r.open) else { ++ return; ++ }; ++ // SAFETY: the caller's. ++ let secondaries = unsafe { std::slice::from_raw_parts(buffers, count as usize) }; ++ let mut draws = 0; ++ for secondary in secondaries { ++ if let Some(raw) = state.raws.get_mut(&secondary.as_raw()) { ++ draws += std::mem::take(&mut raw.pending_draws); ++ raw.parent = Some(at); ++ } ++ } ++ state.units[at].count += draws; ++} ++ ++/// The routed commands' hooks (`recorder/thunks.rs`): `before` a command is logged or called, and ++/// `after`; a command named nowhere here has none. ++macro_rules! hook { ++ (before cmd_begin_render_pass, $st:expr, $cb:ident, $begin:ident, $contents:ident) => { ++ if $crate::pass_times::on() { ++ unsafe { $crate::pass_times::begin_pass($st, $cb, $begin) } ++ } ++ }; ++ (after cmd_end_render_pass, $st:expr, $cb:ident) => { ++ if $crate::pass_times::on() { ++ unsafe { $crate::pass_times::end_pass($st, $cb) } ++ } ++ }; ++ (before end_command_buffer, $st:expr, $cb:ident) => { ++ if $crate::pass_times::on() { ++ unsafe { $crate::pass_times::end($st, $cb) } ++ } ++ }; ++ (before cmd_execute_commands, $st:expr, $cb:ident, $count:ident, $buffers:ident) => { ++ if $crate::pass_times::on() { ++ unsafe { $crate::pass_times::execute($cb, $count, $buffers) } ++ } ++ }; ++ (before cmd_execute_generated_commands, $st:expr, $cb:ident $(, $a:ident)*) => { ++ if $crate::pass_times::on() { ++ unsafe { ++ $crate::pass_times::generated($st, $cb, "vkCmdExecuteGeneratedCommandsEXT") ++ } ++ } ++ }; ++ (before cmd_draw, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_indexed, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_indirect, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_indexed_indirect, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_mesh_tasks, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_mesh_tasks_indirect, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_draw_mesh_tasks_indirect_count, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@draw $cb) ++ }; ++ (before cmd_dispatch, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Compute, "vkCmdDispatch") ++ }; ++ (before cmd_dispatch_indirect, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Compute, "vkCmdDispatchIndirect") ++ }; ++ (before cmd_preprocess_generated_commands, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Compute, "vkCmdPreprocessGeneratedCommandsEXT") ++ }; ++ (before cmd_copy_buffer, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdCopyBuffer") ++ }; ++ (before cmd_copy_image, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdCopyImage") ++ }; ++ (before cmd_blit_image, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdBlitImage") ++ }; ++ (before cmd_copy_buffer_to_image, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdCopyBufferToImage") ++ }; ++ (before cmd_copy_image_to_buffer, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdCopyImageToBuffer") ++ }; ++ (before cmd_update_buffer, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdUpdateBuffer") ++ }; ++ (before cmd_fill_buffer, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdFillBuffer") ++ }; ++ (before cmd_clear_color_image, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdClearColorImage") ++ }; ++ (before cmd_clear_depth_stencil_image, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdClearDepthStencilImage") ++ }; ++ (before cmd_copy_query_pool_results, $st:expr, $cb:ident $(, $a:ident)*) => { ++ $crate::pass_times::hook!(@work $st, $cb, Transfer, "vkCmdCopyQueryPoolResults") ++ }; ++ (@draw $cb:ident) => { ++ if $crate::pass_times::on() { ++ $crate::pass_times::draw($cb) ++ } ++ }; ++ (@work $st:expr, $cb:ident, $class:ident, $name:literal) => { ++ if $crate::pass_times::on() { ++ unsafe { $crate::pass_times::work($st, $cb, $crate::pass_times::Class::$class, $name) } ++ } ++ }; ++ ($phase:ident $command:ident, $($rest:tt)*) => {}; ++} ++pub(crate) use hook; ++ ++/// Where a completed command buffer's units are placed on the host's clock ([`read`]). ++pub(crate) struct Placement<'a> { ++ /// The device's clock. ++ pub(crate) clock: Option<&'a GpuClock>, ++ /// The calibration pair its GPU times took. ++ pub(crate) pair: Option, ++ /// Its GPU start's timestamp and host time, where read and placed. ++ pub(crate) start: Option<(u64, u64)>, ++ /// The host time its completion was seen. ++ pub(crate) completed: u64, ++} ++ ++/// Whether a command of `class` is one an encoder of `kind` is made for. ++fn natural(kind: Option, class: Class) -> bool { ++ matches!( ++ (kind, class), ++ (Some(EncoderKind::Compute), Class::Compute) | (Some(EncoderKind::Blit), Class::Transfer) ++ ) ++} ++ ++/// The encoder kind's name. ++fn kind_name(kind: EncoderKind) -> &'static str { ++ match kind { ++ EncoderKind::Blit => "blit", ++ EncoderKind::Compute => "compute", ++ EncoderKind::Render => "render", ++ EncoderKind::ResourceState => "resource-state", ++ } ++} ++ ++/// Read `capture`'s units, its command buffer completed, and append their lines: in the ++/// completion turn, inside its section (`command_buffer/mod.rs`, `read_pass_times`). ++pub(crate) fn read(capture: Arc, placement: Placement<'_>) { ++ let state = lock(&capture.state); ++ if state.units.is_empty() { ++ return; ++ } ++ let device = capture.device.shared().vk(); ++ // Each block's queries: value and availability. ++ let mut values: Vec> = Vec::with_capacity(state.blocks.len()); ++ for block in &state.blocks { ++ let mut results = vec![[0u64; 2]; BLOCK as usize]; ++ // SAFETY: the block's queries were reset, and those written were written by work that ++ // completed; 64-bit values with availability, two words each. ++ let read = unsafe { ++ device.get_query_pool_results( ++ block.pool, ++ block.first, ++ &mut results, ++ vk::QueryResultFlags::TYPE_64 | vk::QueryResultFlags::WITH_AVAILABILITY, ++ ) ++ }; ++ exit_guard::called(Call::GetQueryPoolResults); ++ if !matches!(read, Ok(()) | Err(vk::Result::NOT_READY)) { ++ results.iter_mut().for_each(|r| *r = [0, 0]); ++ } ++ values.push(results); ++ } ++ let value = |block: usize, query: u32| { ++ let [value, available] = values[block][query as usize]; ++ (available != 0).then_some(value) ++ }; ++ let place = |ticks: u64| match (placement.clock, placement.pair, placement.start) { ++ (Some(clock), Some(pair), _) => clock.place(pair, ticks), ++ (Some(clock), None, Some((start, host))) => host + clock.duration(start, ticks), ++ _ => placement.completed, ++ }; ++ // Each encoder's render pass instances, for `i/n`. ++ let instance = |unit: &Unit| match unit.tag { ++ Some(Tag::Pass(at)) => unit.class == Class::Pass && at != PassAt::Resolve, ++ _ => false, ++ }; ++ let mut totals: HashMap, u32> = HashMap::new(); ++ for unit in state.units.iter().filter(|u| instance(u)) { ++ *totals.entry(unit.encoder).or_default() += 1; ++ } ++ let mut counted: HashMap, u32> = HashMap::new(); ++ let mut out = String::with_capacity(state.units.len() * 160); ++ for (index, unit) in state.units.iter().enumerate() { ++ let begin = value(unit.block, unit.offset); ++ let end = value(unit.block, unit.offset + 1); ++ let gpu = match (begin, end, placement.clock) { ++ (Some(begin), Some(end), Some(clock)) => clock.duration(begin, end).to_string(), ++ _ => String::new(), ++ }; ++ let host = begin.map_or(placement.completed, place); ++ let note = unit.encoder.and_then(|e| state.encoders.get(e as usize)); ++ let encoder = match (unit.encoder, note) { ++ (Some(e), Some(note)) => format!("{}#{e}", kind_name(note.kind)), ++ _ => "-".to_owned(), ++ }; ++ let kind = match (unit.class, unit.tag) { ++ (Class::Pass, Some(Tag::Pass(PassAt::Resolve))) => "render-pass-resolve".to_owned(), ++ (Class::Pass, Some(Tag::Pass(_))) => "render-pass".to_owned(), ++ (Class::Pass, _) => { ++ let made = unit.label.as_ref().and_then(PassLabel::site); ++ let at = unit.site.map(image_log::short).or(made); ++ format!("pass:{}", at.as_deref().unwrap_or("?")) ++ } ++ (class, Some(Tag::Guest)) => { ++ let encoder = note.map_or("?", |n| kind_name(n.kind)); ++ let what = match class { ++ Class::Compute => "compute", ++ _ => "transfer", ++ }; ++ match (natural(note.map(|n| n.kind), class), unit.site) { ++ (true, _) => format!("guest-{encoder}"), ++ (false, Some(at)) => format!( ++ "guest-{encoder}:{what}@{}/{}", ++ image_log::short(at), ++ unit.command ++ ), ++ (false, None) => format!("guest-{encoder}:{what}/{}", unit.command), ++ } ++ } ++ (_, _) => match unit.site { ++ Some(at) => format!("internal:{}/{}", image_log::short(at), unit.command), ++ None => format!("internal:{}", unit.command), ++ }, ++ }; ++ let attachments = unit ++ .label ++ .as_ref() ++ .map_or_else(|| "-".to_owned(), PassLabel::format); ++ let split = match unit.tag { ++ _ if instance(unit) => { ++ let n = totals.get(&unit.encoder).copied().unwrap_or(1); ++ let i = { ++ let i = counted.entry(unit.encoder).or_default(); ++ *i += 1; ++ *i ++ }; ++ let cause = match i { ++ 1 => "first".to_owned(), ++ _ => note ++ .and_then(|n| n.causes.get(i as usize - 2)) ++ .map_or_else(|| "run".to_owned(), |c| format!("{c:?}")), ++ }; ++ let records = match unit.tag { ++ Some(Tag::Pass(PassAt::Inline)) => "inline", ++ Some(Tag::Pass(PassAt::InlineSplit)) => "inline-split", ++ _ => "secondary", ++ }; ++ let not_inline = match (i, note.and_then(|n| n.not_inline)) { ++ (1, Some(why)) => format!(";not-inline={why:?}"), ++ _ => String::new(), ++ }; ++ format!("{i}/{n}:{cause};{records}{not_inline}") ++ } ++ Some(Tag::Pass(PassAt::Resolve)) => "resolve".to_owned(), ++ _ => "-".to_owned(), ++ }; ++ let _ = writeln!( ++ out, ++ "{host},{},{index},{kind},{},{encoder},{attachments},{},{},{gpu}", ++ capture.seq, ++ capture.queue_label, ++ split.replace(',', ";"), ++ unit.count, ++ ); ++ } ++ let lines = state.units.len(); ++ drop(state); ++ capture.times.append(&out, lines); ++} +diff --git a/crates/metal2vulkan-runtime/src/pending_dispatch.rs b/crates/metal2vulkan-runtime/src/pending_dispatch.rs +index b79bb1de..39e17ba3 100644 +--- a/crates/metal2vulkan-runtime/src/pending_dispatch.rs ++++ b/crates/metal2vulkan-runtime/src/pending_dispatch.rs +@@ -75,6 +75,8 @@ impl Recording { + command_buffer: vk::CommandBuffer, + pipelines: &[Arc], + ) { ++ // LOCAL (Atlas) diagnostic: the program's dispatch, to the pass timer (`pass_times.rs`). ++ let _scope = crate::pass_times::scope(crate::pass_times::Tag::Guest); + // SAFETY: the caller's contract; the gates lie in the command buffer's arena. + unsafe { + if let Some(preprocess) = &self.preprocess { +diff --git a/crates/metal2vulkan-runtime/src/recorder/thunks.rs b/crates/metal2vulkan-runtime/src/recorder/thunks.rs +index 0690463b..1c9970ba 100644 +--- a/crates/metal2vulkan-runtime/src/recorder/thunks.rs ++++ b/crates/metal2vulkan-runtime/src/recorder/thunks.rs +@@ -170,12 +170,16 @@ macro_rules! commands { + command_buffer: vk::CommandBuffer, + $($a: $t,)* + ) $(-> $r)? { ++ // LOCAL (Atlas) diagnostic: the pass timer's hook (`pass_times.rs`). ++ crate::pass_times::hook!(before $f, stamps::(), command_buffer $(, $a)*); + // A deferred command buffer's command goes into its log (`recorder`). + // SAFETY: the caller records `command_buffer` alone (Vulkan's external + // synchronization); its arguments are as the command takes them. + let logged = + unsafe { super::handles::record(command_buffer, |log| log.$f($($a,)*)) }; + if let Some(logged) = logged { ++ // LOCAL (Atlas) diagnostic: the pass timer's hook (`pass_times.rs`). ++ crate::pass_times::hook!(after $f, stamps::(), command_buffer $(, $a)*); + return logged; + } + let real = REAL.$f[S].load(Ordering::Acquire); +@@ -184,18 +188,30 @@ macro_rules! commands { + } + // SAFETY: the slot's device's driver gave this command (`store`), of this + // signature; the caller's arguments are as the command takes them. +- unsafe { ++ let answer = unsafe { + let real = std::mem::transmute::< + *mut c_void, + unsafe extern "system" fn(vk::CommandBuffer, $($t,)*) $(-> $r)?, + >(real); + real(command_buffer, $($a,)*) +- } ++ }; ++ // LOCAL (Atlas) diagnostic: the pass timer's hook (`pass_times.rs`). ++ crate::pass_times::hook!(after $f, stamps::(), command_buffer $(, $a)*); ++ answer + } + )* + }; + } + ++/// LOCAL (Atlas) diagnostic: slot `S`'s own timestamp commands, which the pass timer records its ++/// units' timestamps through (`pass_times.rs`). ++fn stamps() -> crate::pass_times::Stamps { ++ crate::pass_times::Stamps { ++ write: cmd_write_timestamp::, ++ reset: cmd_reset_query_pool::, ++ } ++} ++ + commands! { + v1_0 { + BeginCommandBuffer begin_command_buffer c"vkBeginCommandBuffer" +diff --git a/crates/metal2vulkan-runtime/src/release.rs b/crates/metal2vulkan-runtime/src/release.rs +index b8f18a02..684d84ec 100644 +--- a/crates/metal2vulkan-runtime/src/release.rs ++++ b/crates/metal2vulkan-runtime/src/release.rs +@@ -142,6 +142,8 @@ macro_rules! released { + const CALL: Call = Call::$call; + + unsafe fn destroy(self, device: &ash::Device, _: Door) { ++ // LOCAL (Atlas) diagnostic: the image census and registry (`image_log.rs`). ++ crate::image_log::released(Call::$call, vk::Handle::as_raw(self)); + // SAFETY: the caller's. + unsafe { device.$method(self, None) } + } +diff --git a/crates/metal2vulkan-runtime/src/render.rs b/crates/metal2vulkan-runtime/src/render.rs +index ba3449ed..de475813 100644 +--- a/crates/metal2vulkan-runtime/src/render.rs ++++ b/crates/metal2vulkan-runtime/src/render.rs +@@ -1811,6 +1811,11 @@ impl CommandBuffer { + None => EncoderCore::open(self, CALL, EncoderKind::Render, false)?, + Some(_) => EncoderCore::replaying(self, false)?, + }; ++ // LOCAL (Atlas) diagnostic: why the pass's first instance does not record inline, to the ++ // pass timer (`pass_times.rs`). ++ if let (Err(why), None) = (&opens, &again) { ++ crate::pass_times::note_not_inline(core.command_buffer(), *why); ++ } + if let Some(pass) = &rate_map { + let (map, state) = pass.taken(); + core.take_rate_map(map, state); +@@ -6368,6 +6373,8 @@ impl RenderLowering { + (false, false) => cause, + }; + self.core.device().shared().counts.instance_split(cause); ++ // LOCAL (Atlas) diagnostic: the split's cause, to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_split(self.core.command_buffer(), cause); + let previous = std::mem::replace(&mut self.current, *instance); + self.earlier.push(previous); + self.barrier_splits += usize::from(self.barrier); +diff --git a/crates/metal2vulkan-runtime/src/render/inline_first.rs b/crates/metal2vulkan-runtime/src/render/inline_first.rs +index 0b490627..e67b4acd 100644 +--- a/crates/metal2vulkan-runtime/src/render/inline_first.rs ++++ b/crates/metal2vulkan-runtime/src/render/inline_first.rs +@@ -331,6 +331,11 @@ impl RenderLowering { + if matches!(self.inline, Inline::Open { .. }) && records && !capture { + self.inline = Inline::Never; + (self.core.device().shared().counts).pass_not_inline(NotInline::Draws); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline( ++ self.core.command_buffer(), ++ NotInline::Draws, ++ ); + } + None + } +@@ -369,19 +374,30 @@ impl RenderLowering { + } => instance, + Next::New { split: true, .. } => { + counts.pass_not_inline(NotInline::Draws); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline(self.core.command_buffer(), NotInline::Draws); + return Ok(None); + } + }; + if stood.is_some() { + counts.pass_not_inline(NotInline::StandIn); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline(self.core.command_buffer(), NotInline::StandIn); + return Ok(None); + } + if !instance.shape.storage.is_empty() { + counts.pass_not_inline(NotInline::Storage); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline(self.core.command_buffer(), NotInline::Storage); + return Ok(None); + } + if writes { + counts.pass_not_inline(NotInline::MemorylessWrites); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline( ++ self.core.command_buffer(), ++ NotInline::MemorylessWrites, ++ ); + return Ok(None); + } + let plain = made +@@ -405,6 +421,8 @@ impl RenderLowering { + && !instance.shader_tests; + if !plain { + counts.pass_not_inline(NotInline::Draws); ++ // LOCAL (Atlas) diagnostic: to the pass timer (`pass_times.rs`). ++ crate::pass_times::note_not_inline(self.core.command_buffer(), NotInline::Draws); + return Ok(None); + } + let device = self.core.device().clone(); +@@ -498,7 +516,12 @@ impl RenderLowering { + // and framebuffer are live until the command buffer completes (the encoder holds them + // to its end, which retains them), the render area is the framebuffer's extent, and + // every image rests in `GENERAL`. ++ // LOCAL (Atlas) diagnostic: the pass's first instance, to the pass timer (`pass_times.rs`). ++ let scope = crate::pass_times::scope(crate::pass_times::Tag::Pass( ++ crate::pass_times::PassAt::Inline, ++ )); + unsafe { raw.cmd_begin_render_pass(held, &info, vk::SubpassContents::INLINE) }; ++ drop(scope); + device.shared().counts.render_pass_begun(PassBegun::Inline); + self.current.draws = held; + self.current.inline = true; +diff --git a/crates/metal2vulkan-runtime/src/render/inline_splits.rs b/crates/metal2vulkan-runtime/src/render/inline_splits.rs +index c7c9e642..7cd2407c 100644 +--- a/crates/metal2vulkan-runtime/src/render/inline_splits.rs ++++ b/crates/metal2vulkan-runtime/src/render/inline_splits.rs +@@ -577,7 +577,12 @@ fn record_kept( + for step in &instance.before { + step.record_in(raw, held); + } ++ // LOCAL (Atlas) diagnostic: an instance after a split, to the pass timer (`pass_times.rs`). ++ let scope = crate::pass_times::scope(crate::pass_times::Tag::Pass( ++ crate::pass_times::PassAt::InlineSplit, ++ )); + raw.cmd_begin_render_pass(held, &info, vk::SubpassContents::INLINE); ++ drop(scope); + let begun = crate::work_counts::PassBegun::InlineSplit; + shared.counts.render_pass_begun(begun); + let mut draws = kept.room.recordings[..kept.draws].iter(); +diff --git a/crates/metal2vulkan-runtime/src/render/pass_steps.rs b/crates/metal2vulkan-runtime/src/render/pass_steps.rs +index 61e2f403..9290f127 100644 +--- a/crates/metal2vulkan-runtime/src/render/pass_steps.rs ++++ b/crates/metal2vulkan-runtime/src/render/pass_steps.rs +@@ -216,11 +216,16 @@ impl PassSteps { + .framebuffer(framebuffer) + .render_area(self.render_area) + .clear_values(self.clear_values.as_slice()); ++ // LOCAL (Atlas) diagnostic: the pass's instance, to the pass timer (`pass_times.rs`). ++ let scope = crate::pass_times::scope(crate::pass_times::Tag::Pass( ++ crate::pass_times::PassAt::Instance, ++ )); + raw.cmd_begin_render_pass( + command_buffer, + &begin, + vk::SubpassContents::SECONDARY_COMMAND_BUFFERS, + ); ++ drop(scope); + } + } + +@@ -373,7 +378,13 @@ impl PassSteps { + .framebuffer(framebuffer.raw) + .render_area(self.render_area) + .clear_values(self.clear_values.as_slice()); ++ // LOCAL (Atlas) diagnostic: the resolving instance, to the pass timer ++ // (`pass_times.rs`). ++ let scope = crate::pass_times::scope(crate::pass_times::Tag::Pass( ++ crate::pass_times::PassAt::Resolve, ++ )); + raw.cmd_begin_render_pass(command_buffer, &begin, vk::SubpassContents::INLINE); ++ drop(scope); + raw.cmd_end_render_pass(command_buffer); + } + } +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 927b65f1..6a13c30a 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -1591,6 +1591,8 @@ impl Device { + /// runtime extension, for a caller's helper texture). + pub fn new_texture(&self, descriptor: &TextureDescriptor) -> Result { + const CALL: &str = "new_texture"; ++ // LOCAL (Atlas) diagnostic: the image census's purpose (`image_log.rs`). ++ let _purpose = crate::image_log::api("texture"); + self.keep_room(); + let descriptor = &self.texture_options(CALL, descriptor)?; + // An `Outside` texture takes no ID, so no ID's limit applies. +@@ -2836,6 +2838,8 @@ impl Device { + /// The image of `plan` for a texture of `layout` made with `descriptor` (and its hazard + /// tracking mode `hazard`), checked against the device's limits for it, placed, and in the + /// `GENERAL` layout. ++ // LOCAL (Atlas) diagnostic: its caller's place, for the image census (`image_log.rs`). ++ #[track_caller] + fn new_image( + &self, + plan: ImagePlan, +@@ -2855,6 +2859,8 @@ impl Device { + /// shape was written and released; those it compresses read a fixed pattern, which the + /// runtime does not give yet). The runtime's own images keep [`new_image`](Self::new_image)'s + /// start: an image its first command writes in full need not be zeroed first. ++ // LOCAL (Atlas) diagnostic: its caller's place, for the image census (`image_log.rs`). ++ #[track_caller] + fn new_texture_image( + &self, + plan: ImagePlan, +@@ -3030,6 +3036,8 @@ impl Device { + /// [`new_image`](Self::new_image)'s image, of `identity`, its owner made by `owner` of its + /// placement and `VkImage` (a heap's unaliased texture and a shadow whose copies leave an + /// aspect out, `texture/heap.rs`; a view's shadow, `texture/view_shadow.rs`). ++ // LOCAL (Atlas) diagnostic: its caller's place, for the image census (`image_log.rs`). ++ #[track_caller] + fn new_image_owned( + &self, + plan: ImagePlan, +@@ -3053,6 +3061,8 @@ impl Device { + /// any command reads it: a texture's over a buffer (`storage_over_buffer`), and a heap + /// texture's over the heap's bytes (`texture/heap.rs`, `new_heap_shadow` and + /// `heap_planes_image`). ++ // LOCAL (Atlas) diagnostic: its caller's place, for the image census (`image_log.rs`). ++ #[track_caller] + fn new_image_as_it_lies( + &self, + plan: ImagePlan, +@@ -3068,6 +3078,8 @@ impl Device { + /// `start` says so, else zeroed: on the host where it takes host copies, else by a clear + /// where one writes its format ([`clears_to_zero`](Self::clears_to_zero)), else by copies of + /// the device's zeros. (or moved alone, [`Start::Moved`].) ++ // LOCAL (Atlas) diagnostic: its caller's place, for the image census (`image_log.rs`). ++ #[track_caller] + fn new_image_starting( + &self, + plan: ImagePlan, +@@ -3080,6 +3092,17 @@ impl Device { + let held = start == Start::Held; + let moved = start == Start::Moved; + let storage = descriptor.resource_options.storage_mode()?; ++ // LOCAL (Atlas) diagnostic: where the image was asked for, for the image census ++ // (`image_log.rs`). ++ let _origin = crate::image_log::origin( ++ std::panic::Location::caller(), ++ match storage { ++ StorageMode::Shared => "Shared", ++ StorageMode::Managed => "Managed", ++ StorageMode::Private => "Private", ++ StorageMode::Memoryless => "Memoryless", ++ }, ++ ); + let class = plan.memory_class(storage, &self.properties().driver_defects); + // A `Shared` or `Managed` texture's image with host-transfer usage where the device + // copies on the host and makes and places it so (`texture/host_copy.rs`), else as +@@ -3630,6 +3653,8 @@ impl Heap { + /// Refused for a placement heap, whose textures take an offset (`Invalid`, as for its + /// buffers). + pub fn new_texture(&self, descriptor: &TextureDescriptor) -> Result { ++ // LOCAL (Atlas) diagnostic: the image census's purpose (`image_log.rs`). ++ let _purpose = crate::image_log::api("heap-texture"); + let descriptor = &self.device().texture_options("new_texture", descriptor)?; + match self.heap_type() { + HeapType::Sparse => self.device().new_sparse_texture(self, descriptor), +@@ -3664,6 +3689,8 @@ impl Heap { + descriptor: &TextureDescriptor, + offset: u64, + ) -> Result { ++ // LOCAL (Atlas) diagnostic: the image census's purpose (`image_log.rs`). ++ let _purpose = crate::image_log::api("heap-texture"); + let device = self.device(); + let descriptor = &device.texture_options("new_texture_at_offset", descriptor)?; + device.new_heap_texture(self, descriptor, Some(offset)) diff --git a/patches/metal2vulkan/0007-diagnostic-the-pass-timer-s-window-follo.patch b/patches/metal2vulkan/0007-diagnostic-the-pass-timer-s-window-follo.patch new file mode 100644 index 0000000..c03c371 --- /dev/null +++ b/patches/metal2vulkan/0007-diagnostic-the-pass-timer-s-window-follo.patch @@ -0,0 +1,44 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Wed, 28 Oct 2026 17:09:48 +0000 +Subject: [PATCH 07/34] diagnostic:the pass timer's window + follows the live switch pass_times + +A command buffer is captured where /on exists or, with M2V_LIVE set, +where its file says pass_times=1 (live.rs, re-read at most once a second), +checked once as the command buffer begins recording. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/metal2vulkan-runtime/src/pass_times.rs | 11 +++++++---- + 1 file changed, 7 insertions(+), 4 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/pass_times.rs b/crates/metal2vulkan-runtime/src/pass_times.rs +index ca083acb..9cd4574c 100644 +--- a/crates/metal2vulkan-runtime/src/pass_times.rs ++++ b/crates/metal2vulkan-runtime/src/pass_times.rs +@@ -5,9 +5,11 @@ + //! + //! - **The switch** is read once. Unset (or empty), nothing changes: no query pool, no command, + //! no line; each hook is one check of a `OnceLock`. +-//! - **The window.** Set, a command buffer is captured only where the file `/on` exists as the +-//! runtime begins recording it (its first encoder's opening, or its first `VkCommandBuffer` +-//! handed out): checked once per command buffer. `touch /on` opens a window, `rm` closes ++//! - **The window.** Set, a command buffer is captured only where the file `/on` exists, or ++//! the live switch `pass_times` is on (`live.rs`: `pass_times=1` in the file `M2V_LIVE` names, ++//! which it re-reads at most once a second), as the runtime begins recording it (its first ++//! encoder's opening, or its first `VkCommandBuffer` handed out): checked once per command ++//! buffer. `touch /on` (or `pass_times=1`) opens a window, `rm` (or `pass_times=0`) closes + //! it; the first command buffer begun after the close flushes the lines held. + //! - **Where the timestamps go.** Every routed command passes the recorder's entry points + //! (`recorder/thunks.rs`), whatever the runtime records it from (an encoder, its own kernels, a +@@ -521,7 +523,8 @@ fn decide<'a>( + fn start(device: &Device, queue: &CommandQueue, key: usize) -> Option> { + let times = device.shared().pass_times.as_ref()?; + let dir = dir()?; +- if !dir.join("on").exists() { ++ // The window: `/on`, or the live switch `pass_times=1` (`live.rs`, `M2V_LIVE`). ++ if !dir.join("on").exists() && !crate::live::on("pass_times", false) { + times.flush(); + return None; + } diff --git a/patches/metal2vulkan/0008-live-switches-tile_stored_only-and-tile_.patch b/patches/metal2vulkan/0008-live-switches-tile_stored_only-and-tile_.patch new file mode 100644 index 0000000..464dc7e --- /dev/null +++ b/patches/metal2vulkan/0008-live-switches-tile_stored_only-and-tile_.patch @@ -0,0 +1,520 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 18:45:32 +0000 +Subject: [PATCH 08/34] live switches tile_stored_only and + tile_narrow_barriers; lane gpu_cost + +Co-Authored-By: Claude Opus 5.5 +--- + crates/metal2vulkan-runtime/src/render.rs | 1 + + .../src/render/tile_dispatch.rs | 40 +++- + .../src/render/tile_writes.rs | 136 ++++++++++++ + crates/metal2vulkan-runtime/src/tracker.rs | 30 +++ + .../src/runtime_lane.rs | 1 + + .../src/runtime_lane/gpu_cost.rs | 195 ++++++++++++++++++ + 6 files changed, 393 insertions(+), 10 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/render/tile_writes.rs + create mode 100644 crates/metal2vulkan-validation/src/runtime_lane/gpu_cost.rs + +diff --git a/crates/metal2vulkan-runtime/src/render.rs b/crates/metal2vulkan-runtime/src/render.rs +index de475813..89c399b5 100644 +--- a/crates/metal2vulkan-runtime/src/render.rs ++++ b/crates/metal2vulkan-runtime/src/render.rs +@@ -413,6 +413,7 @@ mod shadows; + mod snapshot; + mod table_facts; + mod tessellation_scan; ++mod tile_writes; // LOCAL (Atlas) + mod tile_dispatch; + mod tile_memory; + mod untouched; +diff --git a/crates/metal2vulkan-runtime/src/render/tile_dispatch.rs b/crates/metal2vulkan-runtime/src/render/tile_dispatch.rs +index 5bb661dc..f2741878 100644 +--- a/crates/metal2vulkan-runtime/src/render/tile_dispatch.rs ++++ b/crates/metal2vulkan-runtime/src/render/tile_dispatch.rs +@@ -154,6 +154,8 @@ pub(super) struct TileDispatch { + /// attachment holds them: their origin and extent, inside the scratch images' extent (the + /// pixels every dispatch of the pass covers, from the origin). + copied: [[u32; 2]; 2], ++ /// LOCAL (Atlas): what the switches of `render/tile_writes.rs` took as the dispatch was made. ++ economy: super::tile_writes::Economy, + layout: vk::PipelineLayout, + sets: Vec, + /// The dynamic offsets set 0 is bound with (`binding::dynamic_writes`). +@@ -175,6 +177,17 @@ pub(super) struct TileDispatch { + /// back, and the scratch's image. + pub(super) type Staging = (vk::Image, (vk::ImageCopy, vk::ImageCopy), vk::Image); + ++/// LOCAL (Atlas): the barrier that ends the copies in or the dispatch where ++/// [`Economy::narrow`](super::tile_writes::Economy) holds: from what just ran alone, the barriers ++/// around it ordering the rest (`render/tile_writes.rs`). ++fn scratch_barrier(narrow: bool, narrowed: Barrier) -> Barrier { ++ if narrow { ++ narrowed ++ } else { ++ Barrier::everything() ++ } ++} ++ + impl TileDispatch { + /// The copies of each staged attachment of `layout` (the pass's, as it ends: a stand-in that + /// arrived after the dispatch is what the pass renders) into its scratch and back, over the +@@ -217,6 +230,7 @@ impl TileDispatch { + ) { + // SAFETY: the caller's contract. + unsafe { ++ let narrow = self.economy.narrow; + Barrier::everything().record(raw, command_buffer); + for (own, (into, _), scratch) in copies { + raw.cmd_copy_image( +@@ -228,7 +242,7 @@ impl TileDispatch { + std::slice::from_ref(into), + ); + } +- Barrier::everything().record(raw, command_buffer); ++ scratch_barrier(narrow, Barrier::copied_to_dispatch()).record(raw, command_buffer); + if let Some(push) = raw.push_descriptor().filter(|_| !self.pushed.is_empty()) { + binding::push_set( + push, +@@ -268,16 +282,19 @@ impl TileDispatch { + } + raw.cmd_dispatch(command_buffer, *x, *y, *z); + } +- Barrier::everything().record(raw, command_buffer); +- for (own, (_, back), scratch) in copies { +- raw.cmd_copy_image( ++ // LOCAL (Atlas): nothing to copy back from a dispatch that wrote no staged cell. ++ if self.economy.copies_back { ++ scratch_barrier(narrow, Barrier::dispatched_to_copy()).record(raw, command_buffer); ++ for (own, (_, back), scratch) in copies { ++ raw.cmd_copy_image( + command_buffer, +- *scratch, +- vk::ImageLayout::GENERAL, +- *own, +- vk::ImageLayout::GENERAL, +- std::slice::from_ref(back), +- ); ++ *scratch, ++ vk::ImageLayout::GENERAL, ++ *own, ++ vk::ImageLayout::GENERAL, ++ std::slice::from_ref(back), ++ ); ++ } + } + Barrier::everything().record(raw, command_buffer); + } +@@ -1199,6 +1216,8 @@ impl RenderLowering { + None => None, + }; + // Recording starts here: nothing below can be refused. ++ let economy = ++ super::tile_writes::Economy::now(&variant.pipeline().stage.spirv, cells.is_some()); + self.scratch.extend(made); + self.tile_imageblock |= aliased_past; + if let Some(made) = made_imageblock { +@@ -1220,6 +1239,7 @@ impl RenderLowering { + after: self.earlier.len() + 1, + staged, + copied, ++ economy, + layout: objects.layout(), + sets: set.into_iter().collect(), + offsets, +diff --git a/crates/metal2vulkan-runtime/src/render/tile_writes.rs b/crates/metal2vulkan-runtime/src/render/tile_writes.rs +new file mode 100644 +index 00000000..b8e11ef1 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/render/tile_writes.rs +@@ -0,0 +1,136 @@ ++//! LOCAL (Atlas): less work around a tile dispatch, under live switches, both off by default. ++//! ++//! - **`tile_stored_only`.** Three of the five tile functions a Tahoe WindowServer frame runs read ++//! the imageblock and write only a texture of their own: the translator stores no cell of such a ++//! function (`lower::access::tile`'s `Pass::Store` skips a slot it did not store), so the scratch ++//! holds what was copied into it, and its copy back rewrites the attachment's own texels. Under ++//! the switch a dispatch that writes no staged image copies nothing back. Decided from the ++//! stage's final SPIR-V ([`writes_staged`]), conservatively: only where every image the module ++//! writes is of `Unknown` format, which a staged attachment never is ++//! (`lower::access::tile::storage_format` declares its format), and nothing takes a texel ++//! pointer. A function that stores the pass's imageblock cells writes through its own storage, ++//! which the copies do not touch: it keeps them, as it does any attachment staged beside. ++//! - **`tile_narrow_barriers`.** A dispatch's copies and scratch sat between four barriers from ++//! everything to everything. The first orders everything before against everything after, its ++//! second scope holding the copies and the dispatch; the last orders the dispatch's and the ++//! copies' writes against everything after. The two between need only move the scratch's bytes: ++//! transfer writes to the dispatch's shader accesses, and its shader writes to the copies back. ++//! Under the switch they are those. The first and the last stay. ++//! ++//! The switches are read as a dispatch is encoded; the dispatch keeps what it read. ++ ++/// What a dispatch does around itself (the module docs). ++#[derive(Clone, Copy, Debug)] ++pub(super) struct Economy { ++ /// The scratch is copied back after the dispatch. ++ pub(super) copies_back: bool, ++ /// The two barriers inside the dispatch's copies are narrowed. ++ pub(super) narrow: bool, ++} ++ ++impl Economy { ++ /// The dispatch of a stage of `spirv`, `cells` where it stages the pass's imageblock storage. ++ pub(super) fn now(spirv: &[u32], cells: bool) -> Economy { ++ let stored_only = crate::live::on("tile_stored_only", false); ++ Economy { ++ copies_back: !stored_only || cells || writes_staged(spirv), ++ narrow: crate::live::on("tile_narrow_barriers", false), ++ } ++ } ++} ++ ++const OP_TYPE_IMAGE: u32 = 25; ++const OP_LOAD: u32 = 61; ++const OP_IMAGE_TEXEL_POINTER: u32 = 60; ++const OP_IMAGE_WRITE: u32 = 99; ++/// `ImageFormat::Unknown`. ++const FORMAT_UNKNOWN: u32 = 0; ++ ++/// Whether a module of `spirv` may write an image of a declared format, or reach one's texels ++/// through a pointer: `true` wherever the walk cannot tell, a staged attachment's image being one ++/// of a declared format (the module docs). ++pub(super) fn writes_staged(spirv: &[u32]) -> bool { ++ use std::collections::{HashMap, HashSet}; ++ let Some(mut rest) = spirv.get(5..) else { ++ return true; ++ }; ++ // The image types of no declared format, and the loads of them. ++ let mut unknown = HashSet::new(); ++ let mut loaded = HashMap::new(); ++ let mut written = Vec::new(); ++ while let Some(&head) = rest.first() { ++ let (words, op) = ((head >> 16) as usize, head & 0xffff); ++ if words == 0 || words > rest.len() { ++ return true; ++ } ++ let at = &rest[..words]; ++ match op { ++ OP_TYPE_IMAGE if words >= 9 => { ++ if at[8] == FORMAT_UNKNOWN { ++ unknown.insert(at[1]); ++ } ++ } ++ OP_LOAD if words >= 4 => { ++ loaded.insert(at[2], at[1]); ++ } ++ OP_IMAGE_WRITE if words >= 2 => written.push(at[1]), ++ OP_IMAGE_TEXEL_POINTER => return true, ++ _ => {} ++ } ++ rest = &rest[words..]; ++ } ++ written ++ .iter() ++ .any(|image| !loaded.get(image).is_some_and(|ty| unknown.contains(ty))) ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::writes_staged; ++ ++ fn op(code: u32, operands: &[u32]) -> Vec { ++ let mut words = vec![((operands.len() as u32 + 1) << 16) | code]; ++ words.extend(operands); ++ words ++ } ++ ++ fn module(format: u32, write: Vec) -> Vec { ++ let mut m = vec![0x0723_0203, 0x0001_0500, 0, 100, 0]; ++ // %1 = OpTypeImage %2 2D 0 0 0 2 ++ m.extend(op(25, &[1, 2, 1, 0, 0, 0, 2, format])); ++ // %3 = OpLoad %1 %4 ++ m.extend(op(61, &[1, 3, 4])); ++ m.extend(write); ++ m ++ } ++ ++ #[test] ++ fn a_write_of_a_declared_format_counts() { ++ assert!(writes_staged(&module(33, op(99, &[3, 5, 6])))); ++ } ++ ++ #[test] ++ fn a_write_of_no_declared_format_does_not() { ++ assert!(!writes_staged(&module(0, op(99, &[3, 5, 6])))); ++ } ++ ++ #[test] ++ fn no_write_does_not() { ++ assert!(!writes_staged(&module(33, Vec::new()))); ++ } ++ ++ #[test] ++ fn an_image_it_cannot_trace_counts() { ++ assert!(writes_staged(&module(0, op(99, &[9, 5, 6])))); ++ } ++ ++ #[test] ++ fn a_texel_pointer_counts() { ++ assert!(writes_staged(&module(0, op(60, &[7, 8, 3, 5, 6])))); ++ } ++ ++ #[test] ++ fn a_torn_module_counts() { ++ assert!(writes_staged(&[0x0723_0203, 0, 0, 0, 0, 5 << 16])); ++ } ++} +diff --git a/crates/metal2vulkan-runtime/src/tracker.rs b/crates/metal2vulkan-runtime/src/tracker.rs +index 37ab4133..2724bb88 100644 +--- a/crates/metal2vulkan-runtime/src/tracker.rs ++++ b/crates/metal2vulkan-runtime/src/tracker.rs +@@ -402,6 +402,36 @@ impl Barrier { + } + } + ++ /// LOCAL (Atlas): from a tile dispatch's copies into its scratch to the dispatch, which reads ++ /// and writes the scratch (`render/tile_writes.rs`). ++ pub(super) fn copied_to_dispatch() -> Barrier { ++ Barrier { ++ src: Access::new( ++ vk::PipelineStageFlags2::ALL_TRANSFER, ++ vk::AccessFlags2::TRANSFER_WRITE, ++ ), ++ dst: Access::new( ++ vk::PipelineStageFlags2::COMPUTE_SHADER, ++ vk::AccessFlags2::SHADER_READ | vk::AccessFlags2::SHADER_WRITE, ++ ), ++ } ++ } ++ ++ /// LOCAL (Atlas): from a tile dispatch to the copies back from its scratch ++ /// (`render/tile_writes.rs`). ++ pub(super) fn dispatched_to_copy() -> Barrier { ++ Barrier { ++ src: Access::new( ++ vk::PipelineStageFlags2::COMPUTE_SHADER, ++ vk::AccessFlags2::SHADER_WRITE, ++ ), ++ dst: Access::new( ++ vk::PipelineStageFlags2::ALL_TRANSFER, ++ vk::AccessFlags2::TRANSFER_READ, ++ ), ++ } ++ } ++ + /// From a fill's writes (`vkCmdFillBuffer`) to the dispatches after it, which read or write + /// what it wrote, as an indirect command too. + pub(super) fn after_fill() -> Barrier { +diff --git a/crates/metal2vulkan-validation/src/runtime_lane.rs b/crates/metal2vulkan-validation/src/runtime_lane.rs +index c2394df6..613e8e37 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane.rs +@@ -518,6 +518,7 @@ mod host_copied_textures; + mod host_memory; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] + mod host_rt_frames; ++mod gpu_cost; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd)))] + mod in_place_textures; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd)))] +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/gpu_cost.rs b/crates/metal2vulkan-validation/src/runtime_lane/gpu_cost.rs +new file mode 100644 +index 00000000..ead4119f +--- /dev/null ++++ b/crates/metal2vulkan-validation/src/runtime_lane/gpu_cost.rs +@@ -0,0 +1,195 @@ ++//! LOCAL (Atlas): what the parts of a WindowServer frame's full-target work cost on the device the ++//! cfg selects, and what the switches of `render/tile_writes.rs` change: report only, run by name. ++//! ++//! `GPU-COST` lines give the median GPU time of one command buffer of one pass. ++//! ++//! - **The render area**: a 40x40 scissored draw into a 3200x1800 target, loaded and stored, against ++//! the same draw into a 64x64 one: the most a smaller render area could save. ++//! - **A memoryless color**: the same pass with a 3200x1800 memoryless `RGBA16Float` beside it, ++//! which no draw writes. ++//! - **A tile dispatch** over 960x1856 pixels of 16x16 tiles, read-only (`cfb_tn_read4`) and ++//! writing (`cfb_tn_round4`), under each setting. ++ ++use super::no_copy_textures::{blending, floats, staged}; ++use super::tile_states::on_device; ++use crate::air_library::AirLibrary; ++use metal2vulkan::metal::{PixelFormat, PrimitiveType, TextureUsage}; ++use metal2vulkan::pipeline::{StageSource, TileRenderPipelineDescriptor}; ++use metal2vulkan_devtools::{engine_dir, llvm_assemble}; ++use metal2vulkan_runtime::{ ++ live, CommandQueue, CpuCacheMode, HazardTrackingMode, Device, LoadAction, PipelineOptions, RenderPassDescriptor, ++ RenderPipelineState, ResourceOptions, ScissorRect, Size, StorageMode, StoreAction, Texture, ++ TextureDescriptor, ++}; ++use std::time::Duration; ++ ++const WAIT: Duration = Duration::from_secs(10); ++const WARM: usize = 8; ++const RUNS: usize = 60; ++ ++fn median(mut values: Vec) -> f64 { ++ values.sort_by(f64::total_cmp); ++ values[values.len() / 2] ++} ++ ++/// The median GPU time in microseconds of a command buffer `encode` fills. ++fn timed( ++ queue: &CommandQueue, ++ mut encode: impl FnMut(&metal2vulkan_runtime::CommandBuffer), ++) -> f64 { ++ let mut times = Vec::new(); ++ for run in 0..WARM + RUNS { ++ let command_buffer = queue.command_buffer().unwrap(); ++ encode(&command_buffer); ++ command_buffer.commit().unwrap(); ++ command_buffer.wait_until_completed_for(WAIT).unwrap(); ++ assert_eq!(command_buffer.error(), None); ++ if run >= WARM { ++ times.push( ++ (command_buffer.gpu_end_time().unwrap() - command_buffer.gpu_start_time().unwrap()) * 1e6, ++ ); ++ } ++ } ++ median(times) ++} ++ ++fn target(device: &Device, format: PixelFormat, w: u64, h: u64, mode: Option) -> Texture { ++ let mut d = TextureDescriptor::texture_2d(format, w, h, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = ResourceOptions::private(); ++ if mode == Some(StorageMode::Memoryless) { ++ d.usage = TextureUsage::RENDER_TARGET; ++ d.resource_options = ResourceOptions::new( ++ StorageMode::Memoryless, ++ CpuCacheMode::DefaultCache, ++ HazardTrackingMode::Default, ++ ); ++ } ++ device.new_texture(&d).unwrap() ++} ++ ++/// The target a linear image of the device's own memory, as an in-place surface's form. ++fn linear(device: &Device, w: u64, h: u64) -> Texture { ++ let row = w * 4; ++ let own = device.new_buffer(row * h, ResourceOptions::private()).unwrap(); ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, w, h, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = own.resource_options(); ++ own.new_texture(&d, 0, row).unwrap() ++} ++ ++#[test] ++#[ignore = "a release-mode measurement: run by name"] ++fn gpu_cost_probe() { ++ on_device(|device: &Device| { ++ let queue = device.new_command_queue().unwrap(); ++ let blend = blending(device, PixelFormat::BGRA8Unorm); ++ let whole = staged(device, &floats([-1.0, -1.0, 1.0, 1.0])); ++ let color = staged(device, &floats([0.2, 0.4, 0.6, 0.5])); ++ let big = target(device, PixelFormat::BGRA8Unorm, 3200, 1800, None); ++ let small = target(device, PixelFormat::BGRA8Unorm, 64, 64, None); ++ let flat = linear(device, 3200, 1800); ++ let held = target( ++ device, ++ PixelFormat::RGBA16Float, ++ 3200, ++ 1800, ++ Some(StorageMode::Memoryless), ++ ); ++ let run = |name: &str, c0: &Texture, c1: Option<&Texture>, draw: Option| { ++ let us = timed(&queue, |command_buffer| { ++ let mut pass = RenderPassDescriptor::new(); ++ let a = &mut pass.color_attachments[0]; ++ a.texture = Some(c0.clone()); ++ a.load_action = LoadAction::Load; ++ a.store_action = StoreAction::Store; ++ if let Some(c1) = c1 { ++ let a = &mut pass.color_attachments[1]; ++ a.texture = Some(c1.clone()); ++ a.load_action = LoadAction::Clear; ++ a.store_action = StoreAction::DontCare; ++ } ++ let mut encoder = command_buffer.render_command_encoder(&pass).unwrap(); ++ encoder.set_render_pipeline_state(&blend).unwrap(); ++ encoder.set_vertex_buffer(Some(&whole), 0, 0).unwrap(); ++ encoder.set_fragment_buffer(Some(&color), 0, 0).unwrap(); ++ if let Some(side) = draw { ++ encoder ++ .set_scissor_rect(ScissorRect { ++ x: 0, ++ y: 0, ++ width: side, ++ height: side, ++ }) ++ .unwrap(); ++ } ++ encoder ++ .draw_primitives(PrimitiveType::Triangle, 0, 6, 1, 0) ++ .unwrap(); ++ encoder.end_encoding().unwrap(); ++ }); ++ eprintln!("GPU-COST {name}: {us:.1} us"); ++ }; ++ for round in 0..2 { ++ eprintln!("GPU-COST round {round}"); ++ run("64x64 target, 40x40 draw", &small, None, Some(40)); ++ run("3200x1800 target, 40x40 draw", &big, None, Some(40)); ++ run("3200x1800 linear target, 40x40 draw", &flat, None, Some(40)); ++ run("3200x1800 + memoryless, 40x40 draw", &big, Some(&held), Some(40)); ++ run("3200x1800 linear + memoryless, 40x40 draw", &flat, Some(&held), Some(40)); ++ run("3200x1800 target, full blend", &big, None, None); ++ run("3200x1800 linear target, full blend", &flat, None, None); ++ } ++ ++ let tile = |entry: &str| -> RenderPipelineState { ++ let bitcode = llvm_assemble(&engine_dir().join("cfb_tile_norm.ll")).unwrap(); ++ let library = device.new_library_with_air(&bitcode).unwrap(); ++ let function = library.new_function_with_name(entry).unwrap(); ++ let mut d = TileRenderPipelineDescriptor::new(StageSource::Function( ++ function.function_descriptor(), ++ )); ++ d.color_attachment_formats[0] = Some(PixelFormat::RGBA8Unorm); ++ d.threadgroup_size_matches_tile_size = true; ++ device ++ .new_tile_render_pipeline_state(&d, &PipelineOptions::default()) ++ .unwrap() ++ }; ++ let (w, h) = (960u64, 1856u64); ++ let attachment = target(device, PixelFormat::RGBA8Unorm, w, h, None); ++ let cells = device ++ .new_buffer(16 * (16 * h + w + 64), ResourceOptions::private()) ++ .unwrap(); ++ for round in 0..3 { ++ for (entry, state) in [ ++ ("cfb_tn_read4", tile("cfb_tn_read4")), ++ ("cfb_tn_round4", tile("cfb_tn_round4")), ++ ] { ++ for (stored, narrow) in [("0", "0"), ("1", "0"), ("0", "1"), ("1", "1")] { ++ live::set("tile_stored_only", Some(stored)); ++ live::set("tile_narrow_barriers", Some(narrow)); ++ let us = timed(&queue, |command_buffer| { ++ let mut pass = RenderPassDescriptor::new(); ++ let a = &mut pass.color_attachments[0]; ++ a.texture = Some(attachment.clone()); ++ a.load_action = LoadAction::Load; ++ a.store_action = StoreAction::Store; ++ pass.tile_width = 16; ++ pass.tile_height = 16; ++ let mut encoder = command_buffer.render_command_encoder(&pass).unwrap(); ++ encoder.set_render_pipeline_state(&state).unwrap(); ++ encoder.set_tile_buffer(Some(&cells), 0, 0).unwrap(); ++ encoder ++ .dispatch_threads_per_tile(Size::new(16, 16, 1)) ++ .unwrap(); ++ encoder.end_encoding().unwrap(); ++ }); ++ eprintln!( ++ "GPU-COST tile round {round} {entry} stored_only={stored} narrow={narrow}: {us:.1} us" ++ ); ++ } ++ } ++ } ++ live::set("tile_stored_only", None); ++ live::set("tile_narrow_barriers", None); ++ }); ++} diff --git a/patches/metal2vulkan/0009-live-switch-surface_lazy-a-large-surface.patch b/patches/metal2vulkan/0009-live-switch-surface_lazy-a-large-surface.patch new file mode 100644 index 0000000..f29edca --- /dev/null +++ b/patches/metal2vulkan/0009-live-switch-surface_lazy-a-large-surface.patch @@ -0,0 +1,732 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 19:12:44 +0000 +Subject: [PATCH 09/34] live switch surface_lazy: a large + surface's optimal image kept across command buffers + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/buffer_descriptor.rs | 1 + + .../src/buffer_texture.rs | 7 + + .../src/command_buffer/mod.rs | 4 + + crates/metal2vulkan-runtime/src/host_rt.rs | 4 + + crates/metal2vulkan-runtime/src/lib.rs | 1 + + crates/metal2vulkan-runtime/src/shadow.rs | 40 +++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 226 ++++++++++++++++++ + .../metal2vulkan-runtime/src/tests/shadow.rs | 2 + + crates/metal2vulkan-runtime/src/texture.rs | 2 + + .../metal2vulkan-runtime/src/texture/heap.rs | 3 + + .../src/texture/surface_depth.rs | 1 + + .../src/texture/surface_two_planes.rs | 1 + + .../src/texture/view_shadow.rs | 1 + + .../src/texture/ycbcr_image.rs | 3 + + .../src/runtime_lane/host_rt_frames.rs | 55 ++++- + 15 files changed, 334 insertions(+), 17 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/shadow_lazy.rs + +diff --git a/crates/metal2vulkan-runtime/src/buffer_descriptor.rs b/crates/metal2vulkan-runtime/src/buffer_descriptor.rs +index 51b73aac..3dfe1a02 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_descriptor.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_descriptor.rs +@@ -371,6 +371,7 @@ impl Device { + /// live it returns at once, and otherwise it finds the buffers over the range by address, + /// whether the range covers none of them or many, never reading them all. + pub fn host_range_remapped(&self, bytes: NonNull, length: u64) { ++ crate::shadow_lazy::everything_written(); // LOCAL (Atlas) + let start = bytes.as_ptr() as u64; + let shared = self.shared(); + // The unit an import's pages round out to: an import by pointer over a page the call +diff --git a/crates/metal2vulkan-runtime/src/buffer_texture.rs b/crates/metal2vulkan-runtime/src/buffer_texture.rs +index 45801900..c4c29b08 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_texture.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_texture.rs +@@ -1142,6 +1142,7 @@ impl Texels { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }) + } + +@@ -1312,6 +1313,7 @@ impl Texels { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + texels.staged = Some(Arc::new(copies)); + let (offset, alignment) = ( +@@ -1424,6 +1426,11 @@ impl Texels { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ // LOCAL (Atlas): `shadow_lazy.rs`. ++ lazy: super::shadow_lazy::Lazy::of( ++ self.backing() == super::shadow::Backing::Surface, ++ self.layout.row_bytes / self.texel_bytes * u64::from(self.layout.rows), ++ ), + }) + } + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index b08a9994..7b19ebee 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2505,6 +2505,10 @@ impl CommandBuffer { + failure: Option, + run: CommitRun, + ) -> Result<(), Error> { ++ // LOCAL (Atlas): a later command buffer may take its surfaces' images (`shadow_lazy.rs`). ++ if let Some(id) = self.state().shadow_carry.as_ref().map(ShadowCarry::id) { ++ crate::shadow_lazy::committed(id); ++ } + // The waiting fetched render targets take their storage form at this commit where it + // allows (`storage_form.rs`): one atomic read where none waits. The scheduler's thread + // ends the switch once it submitted the command buffer. None where nothing of it runs. +diff --git a/crates/metal2vulkan-runtime/src/host_rt.rs b/crates/metal2vulkan-runtime/src/host_rt.rs +index bffcd13f..fbf18b08 100644 +--- a/crates/metal2vulkan-runtime/src/host_rt.rs ++++ b/crates/metal2vulkan-runtime/src/host_rt.rs +@@ -259,6 +259,10 @@ impl Device { + if max_texels().is_some_and(|max| texels > max) { + return false; + } ++ // LOCAL (Atlas): an optimal image kept across command buffers instead (`shadow_lazy.rs`). ++ if crate::shadow_lazy::takes_surface(texels) { ++ return false; ++ } + if self.compile_device().policy().host_page_textures != HostPageTexturePolicy::InPlace { + return false; + } +diff --git a/crates/metal2vulkan-runtime/src/lib.rs b/crates/metal2vulkan-runtime/src/lib.rs +index 9cdcc6a5..48b3a27d 100644 +--- a/crates/metal2vulkan-runtime/src/lib.rs ++++ b/crates/metal2vulkan-runtime/src/lib.rs +@@ -203,6 +203,7 @@ mod shader_cache; + mod shadow; + mod shadow_copy_back; + mod shadow_copy_in; ++pub mod shadow_lazy; // LOCAL (Atlas) + mod shadow_merge; + mod shadow_stencil; + mod sharing; +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index 55c34686..a70e6a7c 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -215,6 +215,9 @@ pub(super) struct ShadowCopies { + /// serve is counted as one an earlier command buffer's standing stands behind, or as one of a + /// shadow that never held a standing ([`CarryTally`]). + pub(super) stood: AtomicU64, ++ /// LOCAL (Atlas): the image's standing across command buffers, where `shadow_lazy.rs` takes ++ /// the texture. ++ pub(super) lazy: Option>, + } + + /// The texels of a texture over an IOSurface's plane from which its shadow's copies in count +@@ -758,7 +761,12 @@ impl ShadowCopies { + pub(super) fn merges_back(&self) -> bool { + let bytes = matches!(&self.source, ShadowSource::Bytes { .. }); + let other_writer = self.backing != Backing::PrivateBuffer || !self.tracked; +- bytes && other_writer && !self.drops_writes && self.unmerged().is_none() ++ // LOCAL (Atlas): a lazy surface copies back (`shadow_lazy.rs`). ++ bytes ++ && other_writer ++ && !self.drops_writes ++ && self.unmerged().is_none() ++ && self.lazy.is_none() + } + + /// Whether two shadowed textures copy some of the same truth. +@@ -2167,6 +2175,8 @@ impl ShadowSync { + /// where each stands; this covers the commands of an encoder that meets none of them, whose + /// writes nothing else here would see. + pub(super) fn wrote(&mut self, uses: &[Use]) { ++ // LOCAL (Atlas): any surface's, whatever this encoder uses (`shadow_lazy.rs`). ++ super::shadow_lazy::note_writes(uses); + if self.carry.entries.is_empty() { + return; + } +@@ -2189,6 +2199,16 @@ impl ShadowSync { + carry.entries.append(&mut self.carry.entries); + for entry in &self.entries { + let touched = entry.copies.touched.fetch_add(1, Ordering::Relaxed); ++ // LOCAL (Atlas): what a later command buffer takes (`shadow_lazy.rs`). ++ if let Some(lazy) = &entry.copies.lazy { ++ if entry.standing == Standing::Current && touched == entry.met && !entry.dirty { ++ let (key, range) = entry.copies.truth(); ++ let truth = (key, entry.copies.alias(), range); ++ lazy.stand(truth, carry.id, touched + 1, entry.held, entry.stale); ++ } else { ++ lazy.fall(); ++ } ++ } + if entry.standing != Standing::Current { + continue; + } +@@ -2288,6 +2308,11 @@ struct Carried { + } + + impl ShadowCarry { ++ /// LOCAL (Atlas): the command buffer's identity (`shadow_lazy.rs`). ++ pub(super) fn id(&self) -> u64 { ++ self.id ++ } ++ + /// Carry nothing, under the same identity: every shadowed texture copies in again at its + /// next use, and a first copy in of one this command buffer stood behind still reads as that. + pub(super) fn clear(&mut self) { +@@ -2326,9 +2351,16 @@ impl ShadowSync { + .position(|carried| carried.copies.shadow_key == copies.shadow_key) + .map(|at| self.carry.entries.swap_remove(at)) + .filter(|carried| carried.touched == met); +- let (standing, held, stale) = match &carried { +- Some(carried) => (Standing::Current, carried.held, carried.stale), +- None => (Standing::Unread, None, None), ++ // LOCAL (Atlas): an earlier command buffer's image, where nothing wrote its bytes ++ // since (`shadow_lazy.rs`). ++ let lazily = match (&carried, &copies.lazy) { ++ (None, Some(lazy)) => lazy.take(self.carry.id, met), ++ _ => None, ++ }; ++ let (standing, held, stale) = match (&carried, lazily) { ++ (Some(carried), _) => (Standing::Current, carried.held, carried.stale), ++ (None, Some((held, stale))) => (Standing::Current, held, stale), ++ (None, None) => (Standing::Unread, None, None), + }; + self.entries.push(Entry { + copies: Arc::clone(copies), +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +new file mode 100644 +index 00000000..e05af1f8 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -0,0 +1,226 @@ ++//! LOCAL (Atlas): a large surface over host pages rendered in an optimal image kept across command ++//! buffers, under the live switch `surface_lazy` (default off). ++//! ++//! `host_rt_in_place` renders the macOS desktop and a Simulator's display surface straight into ++//! their pages as a linear image, where a pass costs far more than into an optimal one (a whole ++//! 3200x1800 surface blended, Arrow Lake: 5.1 ms against 1.5). The plain shadow (`shadow.rs`) pays ++//! instead a copy in and a keep and merge at every command buffer, and WindowServer's command ++//! buffers hold about one pass each: the copies cost more than the linear image. ++//! ++//! Under the switch a texture over a surface's plane of at least `surface_lazy_min_texels` ++//! (default 2,000,000) texels is a shadow ([`takes_surface`]), and three things change for it: ++//! ++//! - **No copy in where the image still holds the bytes.** An encoder's end leaves the image's ++//! standing ([`Lazy::stand`]), as `shadow::ShadowCarry` leaves it to the command buffer's later ++//! encoders; a later command buffer's encoder takes it ([`Lazy::take`]), so it copies in nothing, ++//! where nothing wrote the bytes since. Every write a command records of them, through anything ++//! else over them (`shadow::ShadowCopies::meets`, an alias of the same pages too) drops it ++//! ([`note_writes`]), as do a remap of host pages and a CPU write through a texture ++//! ([`everything_written`]). ++//! - **No keep and merge.** Its copy back writes the rectangle the commands wrote in, where their ++//! scissors bound it, or every texel. A CPU write inside that rectangle while the command buffer ++//! runs is lost, where the merge keeps it: the guest's CPU does not write a surface it renders ++//! into at the same time. ++//! - **Copy back as ever: at each encoder's end**, in the command buffer, so before its completion ++//! reaches the guest. So the guest's CPU, a screenshot, ScreenCaptureKit and the display read the ++//! pages as they read them shadowed or in place, at every command buffer's completion, and a ++//! later command's read through another import is ordered after the copy by the tracker. ++//! ++//! **What it cannot see**: a write of the guest's CPU into the pages between command buffers, which ++//! reaches the host through no call. Metal shows it to a later command buffer. Here the image misses ++//! it until the next copy in: at least every `surface_lazy_resync_ms` (default 250, 0 never) a ++//! surface's next encoder takes no standing and copies in as the plain shadow does. ++//! ++//! The switch is read as a texture is made and as an encoder ends: a texture keeps its form. ++//! ++//! **Command buffers encoded out of order**: a standing is taken only from a command buffer already ++//! committed ([`committed`]), whose commands Metal orders before the taker's. ++ ++use super::resource::ResourceKey; ++use super::tracker::{Use, UseRange}; ++use std::collections::VecDeque; ++use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; ++use std::sync::{Arc, Mutex, OnceLock, RwLock, Weak}; ++use std::time::Instant; ++ ++/// The live switch. ++const SWITCH: &str = "surface_lazy"; ++const DEFAULT_MIN_TEXELS: u64 = 2_000_000; ++const DEFAULT_RESYNC_MS: u64 = 250; ++/// The committed command buffers remembered: an older one's standing is not taken. ++const COMMITTED_KEPT: usize = 256; ++ ++/// Moved on by what may have written any surface's bytes without a command. ++static EPOCH: AtomicU64 = AtomicU64::new(0); ++/// Whether any surface is registered: every hook returns at once while none is. ++static ANY: AtomicBool = AtomicBool::new(false); ++static TAKEN: AtomicU64 = AtomicU64::new(0); ++static REFUSED: AtomicU64 = AtomicU64::new(0); ++static RESYNCS: AtomicU64 = AtomicU64::new(0); ++static ACTIVE: AtomicUsize = AtomicUsize::new(0); ++ ++fn registry() -> &'static RwLock>> { ++ static ALL: OnceLock>>> = OnceLock::new(); ++ ALL.get_or_init(RwLock::default) ++} ++ ++fn committed_ids() -> &'static Mutex> { ++ static IDS: OnceLock>> = OnceLock::new(); ++ IDS.get_or_init(Mutex::default) ++} ++ ++/// Whether a texture over host pages of `texels` stays a shadow to render lazily (`host_rt.rs`). ++pub(crate) fn takes_surface(texels: u64) -> bool { ++ crate::live::on(SWITCH, false) ++ && texels ++ >= crate::live::get("surface_lazy_min_texels") ++ .and_then(|v| v.parse().ok()) ++ .unwrap_or(DEFAULT_MIN_TEXELS) ++} ++ ++/// Note that what may hold any surface's bytes changed without a command. ++pub(crate) fn everything_written() { ++ if ANY.load(Ordering::Relaxed) { ++ EPOCH.fetch_add(1, Ordering::Relaxed); ++ } ++} ++ ++/// Note the command buffer of identity `id` committed. ++pub(crate) fn committed(id: u64) { ++ if !ANY.load(Ordering::Relaxed) { ++ return; ++ } ++ let mut ids = committed_ids().lock().unwrap(); ++ if ids.len() == COMMITTED_KEPT { ++ ids.pop_front(); ++ } ++ ids.push_back(id); ++} ++ ++/// Note a recorded command with `uses` (the host memory's among them): each of its writes of a ++/// surface's truth drops that surface's standing. A surface's own copy back counts too: its ++/// encoder's end reads the count after it ([`Lazy::stand`]). ++pub(super) fn note_writes(uses: &[Use]) { ++ if !ANY.load(Ordering::Relaxed) { ++ return; ++ } ++ let all = registry().read().unwrap(); ++ for used in uses.iter().filter(|used| !used.write.is_empty()) { ++ for lazy in all.iter().filter_map(Weak::upgrade) { ++ if lazy.meets(used) { ++ lazy.written.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++ } ++} ++ ++/// The counts so far, for a lane or a stderr line: standings taken, refused, and resyncs. ++pub fn counts() -> (u64, u64, u64) { ++ ( ++ TAKEN.load(Ordering::Relaxed), ++ REFUSED.load(Ordering::Relaxed), ++ RESYNCS.load(Ordering::Relaxed), ++ ) ++} ++ ++/// What an encoder's end left of the image. ++#[derive(Clone, Copy)] ++struct Stood { ++ /// The command buffer's identity (`shadow::ShadowCarry::id`). ++ id: u64, ++ /// `ShadowCopies::touched` as the encoder left it. ++ touched: u64, ++ held: Option<[u32; 4]>, ++ stale: Option<[u32; 4]>, ++ written: u64, ++ epoch: u64, ++} ++ ++/// One surface's standing. ++pub(super) struct Lazy { ++ truth: OnceLock<(ResourceKey, Option, UseRange)>, ++ written: AtomicU64, ++ stood: Mutex>, ++ resynced: Mutex, ++} ++ ++impl Lazy { ++ /// The state of a new shadow of `texels` texels with `surface` as its backing, where the switch ++ /// takes it. ++ pub(super) fn of(surface: bool, texels: u64) -> Option> { ++ (surface && takes_surface(texels)).then(|| { ++ Arc::new(Lazy { ++ truth: OnceLock::new(), ++ written: AtomicU64::new(0), ++ stood: Mutex::new(None), ++ resynced: Mutex::new(Instant::now()), ++ }) ++ }) ++ } ++ ++ fn meets(&self, used: &Use) -> bool { ++ self.truth.get().is_some_and(|(key, alias, range)| { ++ (used.key == *key || Some(used.key) == *alias) && used.range.overlaps(range) ++ }) ++ } ++ ++ /// Leave the image's standing: `id`'s encoder ended with the image holding the truth, inside ++ /// `held` where given, stale inside `stale`. Its own copies back are behind it. ++ pub(super) fn stand( ++ self: &Arc, ++ (key, alias, range): (ResourceKey, Option, UseRange), ++ id: u64, ++ touched: u64, ++ held: Option<[u32; 4]>, ++ stale: Option<[u32; 4]>, ++ ) { ++ if self.truth.set((key, alias, range)).is_ok() { ++ let mut all = registry().write().unwrap(); ++ all.retain(|lazy| lazy.strong_count() > 0); ++ all.push(Arc::downgrade(self)); ++ ACTIVE.store(all.len(), Ordering::Relaxed); ++ ANY.store(true, Ordering::Relaxed); ++ } ++ *self.stood.lock().unwrap() = Some(Stood { ++ id, ++ touched, ++ held, ++ stale, ++ written: self.written.load(Ordering::Relaxed), ++ epoch: EPOCH.load(Ordering::Relaxed), ++ }); ++ } ++ ++ /// The encoder's end left the image not holding the truth. ++ pub(super) fn fall(&self) { ++ *self.stood.lock().unwrap() = None; ++ } ++ ++ /// Where the image stands for an encoder that met `touched`, of a command buffer of identity ++ /// `id`: `Some((held, stale))` where it needs no copy in (the module docs). ++ pub(super) fn take(&self, id: u64, touched: u64) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { ++ let stood = (*self.stood.lock().unwrap())?; ++ let current = stood.touched == touched ++ && stood.written == self.written.load(Ordering::Relaxed) ++ && stood.epoch == EPOCH.load(Ordering::Relaxed) ++ && stood.id != id ++ && committed_ids().lock().unwrap().contains(&stood.id); ++ if !current { ++ REFUSED.fetch_add(1, Ordering::Relaxed); ++ return None; ++ } ++ let every = crate::live::get("surface_lazy_resync_ms") ++ .and_then(|v| v.parse().ok()) ++ .unwrap_or(DEFAULT_RESYNC_MS); ++ if every > 0 { ++ let mut resynced = self.resynced.lock().unwrap(); ++ if resynced.elapsed().as_millis() as u64 >= every { ++ *resynced = Instant::now(); ++ RESYNCS.fetch_add(1, Ordering::Relaxed); ++ return None; ++ } ++ } ++ TAKEN.fetch_add(1, Ordering::Relaxed); ++ Some((stood.held, stood.stale)) ++ } ++} +diff --git a/crates/metal2vulkan-runtime/src/tests/shadow.rs b/crates/metal2vulkan-runtime/src/tests/shadow.rs +index 3b723a86..cb943ca2 100644 +--- a/crates/metal2vulkan-runtime/src/tests/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/tests/shadow.rs +@@ -98,6 +98,7 @@ fn shadow_in( + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }) + } + +@@ -1430,6 +1431,7 @@ fn view_of(image: ResourceKey, levels: Range, layers: Range) -> Arc Result<(), Error> { ++ crate::shadow_lazy::everything_written(); // LOCAL (Atlas) + let upload = Upload::Slice(bytes); + self.replace(region, level, slice, upload, bytes_per_row, bytes_per_image) + } +@@ -5393,6 +5394,7 @@ impl Texture { + bytes_per_row: u64, + bytes_per_image: u64, + ) -> Result<(), Error> { ++ crate::shadow_lazy::everything_written(); // LOCAL (Atlas) + let upload = Upload::Host { + bytes, + copy: OnceCell::new(), +diff --git a/crates/metal2vulkan-runtime/src/texture/heap.rs b/crates/metal2vulkan-runtime/src/texture/heap.rs +index 1e13281c..8aba80f4 100644 +--- a/crates/metal2vulkan-runtime/src/texture/heap.rs ++++ b/crates/metal2vulkan-runtime/src/texture/heap.rs +@@ -3380,6 +3380,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Owner::HeapShadow(Box::new(HeapShadow { + placement, +@@ -3480,6 +3481,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Owner::HeapShadow(Box::new(HeapShadow { + placement, +@@ -3557,6 +3559,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Owner::HeapShadow(Box::new(HeapShadow { + placement, +diff --git a/crates/metal2vulkan-runtime/src/texture/surface_depth.rs b/crates/metal2vulkan-runtime/src/texture/surface_depth.rs +index 8617e0c5..563fbde8 100644 +--- a/crates/metal2vulkan-runtime/src/texture/surface_depth.rs ++++ b/crates/metal2vulkan-runtime/src/texture/surface_depth.rs +@@ -171,6 +171,7 @@ impl Texture { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + let storage = Storage::Shadow { + image: image.shared(), +diff --git a/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs b/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs +index 322cb108..43c812d2 100644 +--- a/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs ++++ b/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs +@@ -240,6 +240,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + self.degrade(Degradation::ShadowedTexture, || { + let [width, height, _] = layout.extent; +diff --git a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +index 654c899d..c73065f2 100644 +--- a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs ++++ b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +@@ -483,6 +483,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Owner::ViewShadow(Box::new(ViewShadow { + placement, +diff --git a/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs b/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs +index 2465608d..cf1d6816 100644 +--- a/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs ++++ b/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs +@@ -376,6 +376,7 @@ impl Device { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Ok(Storage::Shadow { + image: image.shared(), +@@ -671,6 +672,7 @@ impl Texture { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + Ok((image, copies)) + } +@@ -839,6 +841,7 @@ impl Texture { + merge_scratch: ScratchSpares::default(), + touched: AtomicU64::new(0), + stood: AtomicU64::new(0), ++ lazy: None, + }; + let storage = Storage::Shadow { + image: image.shared(), +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs b/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs +index d1c701b0..491f874d 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs +@@ -32,7 +32,7 @@ use metal2vulkan::metal::{PixelFormat, PrimitiveType, TextureUsage}; + use metal2vulkan_runtime::{ + live, Buffer, CommandBuffer, CpuCacheMode, Device, HazardTrackingMode, HostMemory, + IoSurfacePlane, LoadAction, RenderPassDescriptor, RenderPipelineState, ResourceOptions, +- StorageMode, StoreAction, Texture, TextureDescriptor, WorkCounts, ++ ScissorRect, StorageMode, StoreAction, Texture, TextureDescriptor, WorkCounts, + }; + use std::time::Instant; + +@@ -56,6 +56,13 @@ enum Shape { + /// The iOS Simulator's screen at 3x (1206x2622, rows of 4864 bytes): 42 command buffers of + /// one pass each blending a sixteenth of it, as the coordinator measured the sim's frames. + SimSmallApart, ++ /// LOCAL (Atlas): the 3200x1800 HiDPI desktop, 8 command buffers of one pass each, each ++ /// blending the whole surface. ++ HidpiWhole, ++ /// As [`Shape::HidpiWhole`], each pass scissored to a window's damage: a 1000x600 rectangle. ++ HidpiDamage, ++ /// LOCAL (Atlas): the Simulator's screen at 3x, 8 passes a frame blending all of it. ++ SimWhole, + } + + impl Shape { +@@ -66,16 +73,21 @@ impl Shape { + Shape::PassesApart => (3440, 1440, 13824, 10, true, false), + Shape::SmallApart => (3440, 1440, 13824, 10, true, true), + Shape::SimSmallApart => (1206, 2622, 4864, 42, true, true), ++ Shape::HidpiWhole | Shape::HidpiDamage => (3200, 1800, 12864, 8, true, false), ++ Shape::SimWhole => (1206, 2622, 4864, 8, true, false), + } + } + } + +-const SHAPES: [Shape; 5] = [ ++const SHAPES: [Shape; 8] = [ + Shape::Passes, + Shape::PassesViews, + Shape::PassesApart, + Shape::SmallApart, + Shape::SimSmallApart, ++ Shape::HidpiWhole, ++ Shape::HidpiDamage, ++ Shape::SimWhole, + ]; + + /// The switches' settings weighed. +@@ -87,6 +99,8 @@ enum Setting { + InPlace, + /// `shadow_compressed`: the shadow, its image eligible for the driver's compression. + Compressed, ++ /// LOCAL (Atlas): `surface_lazy`: the image kept across command buffers (`shadow_lazy.rs`). ++ Lazy, + /// For the parts: the passes into a `Private` texture of the surface's size (optimal tiling, + /// the device's own memory), no shadow. + Private, +@@ -96,19 +110,21 @@ enum Setting { + } + + impl Setting { +- fn switches(self) -> [(&'static str, &'static str); 2] { ++ fn switches(self) -> [(&'static str, &'static str); 3] { + let on = |yes: bool| if yes { "1" } else { "0" }; + [ + ("host_rt_in_place", on(self == Setting::InPlace)), + ("shadow_compressed", on(self == Setting::Compressed)), ++ ("surface_lazy", on(self == Setting::Lazy)), + ] + } + } + +-const SETTINGS: [Setting; 5] = [ ++const SETTINGS: [Setting; 6] = [ + Setting::Shadow, + Setting::InPlace, + Setting::Compressed, ++ Setting::Lazy, + Setting::Private, + Setting::LinearOwn, + ]; +@@ -158,6 +174,7 @@ fn pass( + blend: &RenderPipelineState, + rects: &[Buffer], + color: &Buffer, ++ damage: Option, + ) { + let mut descriptor = RenderPassDescriptor::new(); + let attachment = &mut descriptor.color_attachments[0]; +@@ -167,6 +184,9 @@ fn pass( + let mut encoder = command_buffer.render_command_encoder(&descriptor).unwrap(); + encoder.set_render_pipeline_state(blend).unwrap(); + encoder.set_fragment_buffer(Some(color), 0, 0).unwrap(); ++ if let Some(damage) = damage { ++ encoder.set_scissor_rect(damage).unwrap(); ++ } + for rect in rects { + encoder.set_vertex_buffer(Some(rect), 0, 0).unwrap(); + encoder +@@ -202,7 +222,7 @@ fn hash(bytes: &[u8]) -> u64 { + #[test] + #[ignore = "a release-mode measurement: run by name"] + fn host_rt_frame_costs() { +- let len = ((13824 * 1440) as usize).next_multiple_of(16384); ++ let len = ((12864 * 1800) as usize).next_multiple_of(16384); + let mapping = Mapping::anonymous(len); + // SAFETY: the mapping's bytes, which no device uses between the cases. + let bytes = || unsafe { std::slice::from_raw_parts_mut(mapping.pointer().as_ptr(), len) }; +@@ -215,7 +235,8 @@ fn host_rt_frame_costs() { + // image on ANV. + let parts = matches!(setting, Setting::Private | Setting::LinearOwn); + if (shape == Shape::PassesViews && parts) +- || (shape == Shape::SimSmallApart && setting == Setting::LinearOwn) ++ || (matches!(shape, Shape::SimSmallApart | Shape::SimWhole) ++ && setting == Setting::LinearOwn) + { + continue; + } +@@ -305,6 +326,12 @@ fn host_rt_frame_costs() { + (buffer, surface, form) + } + }; ++ let damage = (shape == Shape::HidpiDamage).then_some(ScissorRect { ++ x: 800, ++ y: 500, ++ width: 1000, ++ height: 600, ++ }); + let (mut waits, mut gpus) = (Vec::new(), Vec::new()); + let mut before = device.work_counts(); + for k in 0..WARM + FRAMES { +@@ -312,20 +339,21 @@ fn host_rt_frame_costs() { + before = device.work_counts(); + } + let mut command_buffers = vec![queue.command_buffer().unwrap()]; +- pass(&command_buffers[0], &surface, &blend, rects, &color); ++ pass(&command_buffers[0], &surface, &blend, rects, &color, damage); ++ let committed = Instant::now(); + for _ in 1..passes { + if apart { ++ // As the guest's: each committed before the next is encoded. ++ let last = command_buffers.last().unwrap(); ++ last.commit().unwrap(); + let command_buffer = queue.command_buffer().unwrap(); +- pass(&command_buffer, &surface, &blend, rects, &color); ++ pass(&command_buffer, &surface, &blend, rects, &color, damage); + command_buffers.push(command_buffer); + } else { +- pass(&command_buffers[0], &surface, &blend, rects, &color); ++ pass(&command_buffers[0], &surface, &blend, rects, &color, damage); + } + } +- let committed = Instant::now(); +- for command_buffer in &command_buffers { +- command_buffer.commit().unwrap(); +- } ++ command_buffers.last().unwrap().commit().unwrap(); + let mut gpu = 0.0; + for command_buffer in &command_buffers { + command_buffer.wait_until_completed_for(WAIT).unwrap(); +@@ -357,6 +385,7 @@ fn host_rt_frame_costs() { + device.name(), + per_frame(&before, &after), + ); ++ eprintln!("LAZY-COUNTS {:?}", metal2vulkan_runtime::shadow_lazy::counts()); + results.push((shape, setting, digest, between_kept)); + } + } diff --git a/patches/metal2vulkan/0010-surface_lazy-a-commit-does-nothing-while.patch b/patches/metal2vulkan/0010-surface_lazy-a-commit-does-nothing-while.patch new file mode 100644 index 0000000..ae1f9b9 --- /dev/null +++ b/patches/metal2vulkan/0010-surface_lazy-a-commit-does-nothing-while.patch @@ -0,0 +1,63 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 19:57:22 +0000 +Subject: [PATCH 10/34] surface_lazy: a commit does nothing + while no surface is lazy; counts on stderr + +Co-Authored-By: Claude Opus 5.5 +--- + .../metal2vulkan-runtime/src/command_buffer/mod.rs | 6 ++++-- + crates/metal2vulkan-runtime/src/shadow_lazy.rs | 14 ++++++++++---- + 2 files changed, 14 insertions(+), 6 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 7b19ebee..3c1c74f2 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2506,8 +2506,10 @@ impl CommandBuffer { + run: CommitRun, + ) -> Result<(), Error> { + // LOCAL (Atlas): a later command buffer may take its surfaces' images (`shadow_lazy.rs`). +- if let Some(id) = self.state().shadow_carry.as_ref().map(ShadowCarry::id) { +- crate::shadow_lazy::committed(id); ++ if crate::shadow_lazy::active() { ++ if let Some(id) = self.state().shadow_carry.as_ref().map(ShadowCarry::id) { ++ crate::shadow_lazy::committed(id); ++ } + } + // The waiting fetched render targets take their storage form at this commit where it + // allows (`storage_form.rs`): one atomic read where none waits. The scheduler's thread +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index e05af1f8..b1d61a39 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -85,11 +85,13 @@ pub(crate) fn everything_written() { + } + } + ++/// Whether any surface is lazy, one atomic read: while none is, a commit does nothing here. ++pub(crate) fn active() -> bool { ++ ANY.load(Ordering::Relaxed) ++} ++ + /// Note the command buffer of identity `id` committed. + pub(crate) fn committed(id: u64) { +- if !ANY.load(Ordering::Relaxed) { +- return; +- } + let mut ids = committed_ids().lock().unwrap(); + if ids.len() == COMMITTED_KEPT { + ids.pop_front(); +@@ -216,7 +218,11 @@ impl Lazy { + let mut resynced = self.resynced.lock().unwrap(); + if resynced.elapsed().as_millis() as u64 >= every { + *resynced = Instant::now(); +- RESYNCS.fetch_add(1, Ordering::Relaxed); ++ let resyncs = RESYNCS.fetch_add(1, Ordering::Relaxed) + 1; ++ if resyncs % 40 == 1 { ++ let (taken, refused, resyncs) = counts(); ++ eprintln!("surface_lazy: taken {taken} refused {refused} resyncs {resyncs}"); ++ } + return None; + } + } diff --git a/patches/metal2vulkan/0011-surface_lazy-name-the-refused-takes-scope-them-and-l.patch b/patches/metal2vulkan/0011-surface_lazy-name-the-refused-takes-scope-them-and-l.patch new file mode 100644 index 0000000..5ac4759 --- /dev/null +++ b/patches/metal2vulkan/0011-surface_lazy-name-the-refused-takes-scope-them-and-l.patch @@ -0,0 +1,1587 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 21:27:15 +0000 +Subject: [PATCH 11/34] surface_lazy: name the refused takes, scope them, and + leave a displayed surface's writes ahead + +Under surface_lazy, every remap and every replaceRegion of any texture moved +one global epoch, which refused every surface's next take. stderr now counts +takes by reason and copies in by cause, per surface too. + +New live switches, default off, nothing changes without them: +- surface_lazy_scoped: a remap or CPU write drops the standing of the + surfaces over those pages alone. +- surface_lazy_present: a surface the display presented leaves its writes + ahead in its image; a present, a command buffer using its bytes, a CPU call + through a texture or a clock copies them back, once, in a command buffer of + its own, inside the union of the scissors. +- surface_lazy_present_all: every lazy surface. +- surface_lazy_present_max_ms: the clock's bound (default 100). +- surface_lazy_min_width: keeps narrower surfaces out (default none). + +Co-Authored-By: Claude Opus 5.5 +--- + crates/metal2vulkan-runtime/src/blit.rs | 16 +- + crates/metal2vulkan-runtime/src/buffer.rs | 5 + + .../src/buffer_descriptor.rs | 5 +- + .../src/buffer_texture.rs | 6 + + .../src/command_buffer/mod.rs | 42 +- + crates/metal2vulkan-runtime/src/encoder.rs | 20 +- + crates/metal2vulkan-runtime/src/host_rt.rs | 2 +- + crates/metal2vulkan-runtime/src/shadow.rs | 39 +- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 695 +++++++++++++++++- + crates/metal2vulkan-runtime/src/texture.rs | 31 +- + crates/metal2vulkan-runtime/src/tracker.rs | 5 + + .../src/runtime_lane.rs | 2 + + .../src/runtime_lane/host_rt_frames.rs | 4 +- + .../src/runtime_lane/surface_present.rs | 336 +++++++++ + 14 files changed, 1170 insertions(+), 38 deletions(-) + create mode 100644 crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs + +diff --git a/crates/metal2vulkan-runtime/src/blit.rs b/crates/metal2vulkan-runtime/src/blit.rs +index d7ed3c16..523cac1c 100644 +--- a/crates/metal2vulkan-runtime/src/blit.rs ++++ b/crates/metal2vulkan-runtime/src/blit.rs +@@ -30,7 +30,7 @@ use super::error::{invalid, Error, Problem}; + use super::fresh_unwritten::FreshRead; + use super::replay::{BlitCall, CallLog, Replay, Resource}; + use super::resource::{ResourceKey, ResourceRef}; +-use super::shadow::{self, Covered, ShadowCopies}; ++use super::shadow::{self, Covered, ShadowCopies, ShadowCopy}; + use super::texture::Texture; + use super::tracker::{Access, Use, UseRange}; + use super::used_resources::EncoderKind; +@@ -197,6 +197,20 @@ impl CommandBuffer { + } + + impl BlitCommandEncoder { ++ /// LOCAL (Atlas): record `copy`, a surface's copy back (`shadow_lazy.rs`), holding `holds` until ++ /// it completes. Never encoded again: the encoder keeps no log. ++ pub(super) fn surface_copy_back( ++ &mut self, ++ copy: &ShadowCopy, ++ holds: Vec, ++ ) { ++ self.log = None; ++ self.core.shadow_copies(std::slice::from_ref(copy)); ++ for hold in holds { ++ self.core.retain_shared(hold); ++ } ++ } ++ + /// An encoder that encodes an ended one of `command_buffer` again (`replay.rs`): it logs + /// nothing. + pub(super) fn replaying(command_buffer: &CommandBuffer) -> Result { +diff --git a/crates/metal2vulkan-runtime/src/buffer.rs b/crates/metal2vulkan-runtime/src/buffer.rs +index 57d569a8..2100fa26 100644 +--- a/crates/metal2vulkan-runtime/src/buffer.rs ++++ b/crates/metal2vulkan-runtime/src/buffer.rs +@@ -3042,6 +3042,11 @@ impl Buffer { + } + } + ++ /// LOCAL (Atlas): where a buffer over host pages lies (`HostCell::host_range`). ++ pub(super) fn host_range(&self) -> Option> { ++ self.pages_cell()?.host_range() ++ } ++ + /// The key of the host memory a no-copy buffer joined, made over the bytes another covers + /// (`host_aliases.rs`): its uses carry one of that key too. `None` for any other buffer. + pub(super) fn alias(&self) -> Option { +diff --git a/crates/metal2vulkan-runtime/src/buffer_descriptor.rs b/crates/metal2vulkan-runtime/src/buffer_descriptor.rs +index 3dfe1a02..ce968ec2 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_descriptor.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_descriptor.rs +@@ -371,7 +371,10 @@ impl Device { + /// live it returns at once, and otherwise it finds the buffers over the range by address, + /// whether the range covers none of them or many, never reading them all. + pub fn host_range_remapped(&self, bytes: NonNull, length: u64) { +- crate::shadow_lazy::everything_written(); // LOCAL (Atlas) ++ // LOCAL (Atlas) ++ crate::shadow_lazy::remapped( ++ bytes.as_ptr() as usize..(bytes.as_ptr() as usize).saturating_add(length as usize), ++ ); + let start = bytes.as_ptr() as u64; + let shared = self.shared(); + // The unit an import's pages round out to: an import by pointer over a page the call +diff --git a/crates/metal2vulkan-runtime/src/buffer_texture.rs b/crates/metal2vulkan-runtime/src/buffer_texture.rs +index c4c29b08..d466dd62 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_texture.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_texture.rs +@@ -1428,7 +1428,13 @@ impl Texels { + stood: AtomicU64::new(0), + // LOCAL (Atlas): `shadow_lazy.rs`. + lazy: super::shadow_lazy::Lazy::of( ++ self.buffer.device(), + self.backing() == super::shadow::Backing::Surface, ++ || self.buffer.host_range(), ++ [ ++ (self.layout.row_bytes / self.texel_bytes) as u32, ++ self.layout.rows as u32, ++ ], + self.layout.row_bytes / self.texel_bytes * u64::from(self.layout.rows), + ), + }) +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 3c1c74f2..97d01cf3 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2505,12 +2505,46 @@ impl CommandBuffer { + failure: Option, + run: CommitRun, + ) -> Result<(), Error> { +- // LOCAL (Atlas): a later command buffer may take its surfaces' images (`shadow_lazy.rs`). +- if crate::shadow_lazy::active() { +- if let Some(id) = self.state().shadow_carry.as_ref().map(ShadowCarry::id) { +- crate::shadow_lazy::committed(id); ++ // LOCAL (Atlas): a later command buffer may take its surfaces' images, and what this one ++ // needs of the bytes they hold ahead goes back first (`shadow_lazy.rs`). ++ let lazy = match crate::shadow_lazy::active() { ++ true => self.state().shadow_carry.as_ref().map(ShadowCarry::id), ++ false => None, ++ }; ++ if let Some(id) = lazy { ++ if crate::shadow_lazy::ahead() { ++ let met = { ++ let state = self.state(); ++ let summaries: Vec<_> = (state.recorded.iter()) ++ .flat_map(|part| { ++ let segments = (part.segments.iter()) ++ .filter_map(|segment| segment.recorded.as_ref()); ++ std::iter::once(&part.summary) ++ .chain(segments.map(|recording| &recording.summary)) ++ }) ++ .collect(); ++ crate::shadow_lazy::met_by(&summaries) ++ }; ++ for surface in met { ++ surface.flush(); ++ } + } ++ crate::shadow_lazy::committed(id); ++ } ++ let committed = self.commit_settled_now(call, failure, run); ++ if let Some(id) = lazy { ++ crate::shadow_lazy::after_commit(id); + } ++ committed ++ } ++ ++ /// [`commit_settled`](Self::commit_settled), the surfaces' notes aside. ++ fn commit_settled_now( ++ &self, ++ call: &'static str, ++ failure: Option, ++ run: CommitRun, ++ ) -> Result<(), Error> { + // The waiting fetched render targets take their storage form at this commit where it + // allows (`storage_form.rs`): one atomic read where none waits. The scheduler's thread + // ends the switch once it submitted the command buffer. None where nothing of it runs. +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index 5b217e85..fd64a5b6 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -458,13 +458,14 @@ impl EncoderCore { + // which this one's first uses of them take instead of copying the truth in again + // (`shadow::ShadowCarry`). + let carry = state.shadows_to_carry(); ++ let id = carry.id(); + drop(state); + command_buffer + .device() + .shared() + .identifiers + .encoder_opened(kind); +- Ok(EncoderCore { ++ let mut core = EncoderCore { + open: OpenGuard { + command_buffer: command_buffer.clone(), + ended: false, +@@ -498,7 +499,19 @@ impl EncoderCore { + replaying: false, + last_set_zero: LastSetZero::default(), + _not_sync: PhantomData, +- }) ++ }; ++ // LOCAL (Atlas): this command buffer's earlier encoders may have left writes ahead in a ++ // surface's image, which this one may read through its bytes (`shadow_lazy.rs`). ++ if crate::shadow_lazy::ahead() { ++ for (lazy, copy, holds) in crate::shadow_lazy::deferred_by(id) { ++ let _own = lazy.own(); ++ core.shadow_copies(std::slice::from_ref(©)); ++ for hold in holds { ++ core.retain_shared(hold); ++ } ++ } ++ } ++ Ok(core) + } + + /// The core a segment of `command_buffer` records into at its commit, `ordered` as the +@@ -2006,7 +2019,8 @@ impl EncoderCore { + if sharing > 0 { + self.device().shared().counts.shadows_sharing_truth(sharing); + } +- let copies = self.shadows.finish(); ++ // LOCAL (Atlas): a surface may leave its writes ahead in its image (`shadow_lazy.rs`). ++ let copies = self.shadows.finish_leaving_ahead(); + let copies = self.sample_moves(copies); + self.count_shadow_copies(&copies); + self.shadow_copies(&copies); +diff --git a/crates/metal2vulkan-runtime/src/host_rt.rs b/crates/metal2vulkan-runtime/src/host_rt.rs +index fbf18b08..bbee9add 100644 +--- a/crates/metal2vulkan-runtime/src/host_rt.rs ++++ b/crates/metal2vulkan-runtime/src/host_rt.rs +@@ -260,7 +260,7 @@ impl Device { + return false; + } + // LOCAL (Atlas): an optimal image kept across command buffers instead (`shadow_lazy.rs`). +- if crate::shadow_lazy::takes_surface(texels) { ++ if crate::shadow_lazy::takes_surface(extent[0], texels) { + return false; + } + if self.compile_device().policy().host_page_textures != HostPageTexturePolicy::InPlace { +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index a70e6a7c..2853f759 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2060,7 +2060,7 @@ pub(super) struct ShadowUse { + } + + /// The rectangle bounding `a` and `b`, each `[x0, y0, x1, y1)`; `None` (anywhere) where either is. +-fn bounding(a: Option<[u32; 4]>, b: Option<[u32; 4]>) -> Option<[u32; 4]> { ++pub(super) fn bounding(a: Option<[u32; 4]>, b: Option<[u32; 4]>) -> Option<[u32; 4]> { + let ([a0, a1, a2, a3], [b0, b1, b2, b3]) = (a?, b?); + Some([a0.min(b0), a1.min(b1), a2.max(b2), a3.max(b3)]) + } +@@ -2588,6 +2588,10 @@ impl ShadowSync { + cause: Some(cause), + moves: None, + }); ++ // LOCAL (Atlas): counted by cause (`shadow_lazy.rs`). ++ if let Some(lazy) = &shadow.copies.lazy { ++ lazy.copied_in(cause); ++ } + } + entry.standing = Standing::Current; + entry.held = needs; +@@ -2617,6 +2621,10 @@ impl ShadowSync { + cause: Some(CopyCause::Bounded), + moves: None, + }); ++ // LOCAL (Atlas): as above. ++ if let Some(lazy) = &shadow.copies.lazy { ++ lazy.copied_in(CopyCause::Bounded); ++ } + } + } + // The command writes every texel before it reads any. +@@ -2763,4 +2771,33 @@ impl ShadowSync { + } + copies + } ++ ++ /// LOCAL (Atlas): [`finish`](Self::finish) at an encoder's own end, where a surface may leave ++ /// what it wrote ahead in its image instead (`shadow_lazy.rs`): one alone in the encoder over ++ /// its bytes, with nothing kept to merge. ++ pub(super) fn finish_leaving_ahead(&mut self) -> Vec> { ++ let mut copies = Vec::new(); ++ for i in 0..self.entries.len() { ++ if !self.entries[i].dirty { ++ continue; ++ } ++ let entry = &self.entries[i]; ++ let alone = (self.entries.iter().enumerate()) ++ .all(|(j, other)| j == i || !other.copies.shares_truth(&entry.copies)); ++ let left = alone ++ && entry.merge.is_none() ++ && !entry.two_ways ++ && (entry.copies.lazy.as_ref()) ++ .is_some_and(|lazy| lazy.defer(&entry.copies, entry.area, self.carry.id)); ++ match left { ++ true => { ++ let entry = &mut self.entries[i]; ++ entry.dirty = false; ++ entry.area = None; ++ } ++ false => self.copy_back(i, &mut copies), ++ } ++ } ++ copies ++ } + } +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index b1d61a39..013f6559 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -16,7 +16,7 @@ + //! where nothing wrote the bytes since. Every write a command records of them, through anything + //! else over them (`shadow::ShadowCopies::meets`, an alias of the same pages too) drops it + //! ([`note_writes`]), as do a remap of host pages and a CPU write through a texture +-//! ([`everything_written`]). ++//! ([`remapped`], [`cpu_wrote`]). + //! - **No keep and merge.** Its copy back writes the rectangle the commands wrote in, where their + //! scissors bound it, or every texel. A CPU write inside that rectangle while the command buffer + //! runs is lost, where the merge keeps it: the guest's CPU does not write a surface it renders +@@ -35,18 +35,74 @@ + //! + //! **Command buffers encoded out of order**: a standing is taken only from a command buffer already + //! committed ([`committed`]), whose commands Metal orders before the taker's. ++//! ++//! **What refuses a standing** is on stderr every [`REPORT_MS`] ms while a surface is lazy, by ++//! reason and by surface ([`report`]), so a capture names the cause of each copy in. ++//! ++//! **Under the switch `surface_lazy_scoped`** (default off) a remap or a CPU write through a texture ++//! drops the standing of the surfaces over those pages alone ([`remapped`], [`cpu_wrote`]). Off, each ++//! moves [`EPOCH`], which drops every surface's standing whatever the pages or the texture: a ++//! guest's upload into any texture and any Mach VM call that reports pages. ++//! ++//! **Under the switch `surface_lazy_present`** (default off) a surface the display presented once ++//! ([`presented`]; every lazy surface under `surface_lazy_present_all`) copies back only when ++//! something needs its bytes. Its encoders leave what they wrote ahead in the image ([`Pending`]), ++//! the rectangle bounding their writes where their scissors bound them, every texel else. Each of ++//! these copies it back, once, in a command buffer of its own ([`Lazy::flush`]): ++//! ++//! - **A present**, after the command buffer that wrote last is committed ([`after_commit`]). ++//! - **A command buffer that uses the bytes**, through any other import or a blit of the texture ++//! itself, or copies them in: before its commit ([`met_by`]), where the commit-order tracker ++//! orders the copy after the writers committed before it and before that command buffer. Its ++//! summaries hold every use the encoders recorded, the host memory's among them. ++//! - **A later encoder of the writer's own command buffer**, as it opens, inside it ++//! ([`deferred_by`]): the commit-time copy would run after that encoder. ++//! - **A CPU read or write through a texture** over the pages, which waits for it ([`cpu_reads`], ++//! [`cpu_wrote`]). ++//! - **The clock**: `surface_lazy_present_max_ms` (default 100) after the first write left ahead, on ++//! a thread of its own ([`flush_due`]), so nothing stays stale past it after the guest goes idle. ++//! ++//! Each moves only what committed command buffers wrote: a write stays out of every copy back ++//! until its commit returns, as a copy back recorded before it runs before it. ++//! ++//! **Who can read stale bytes**, each for at most that bound and only of a surface left ahead: the ++//! guest's CPU through its own mapping of the pages, which reaches the host through no call, and a ++//! host reader of the pages that goes through no command buffer. Under the switch a CPU write into ++//! the pages that lies inside the rectangle is overwritten by the copy back, where ++//! `surface_lazy` alone keeps it unless a command buffer runs meanwhile. ++//! ++//! **A texture released while ahead** is not copied back where nothing else holds its image: its ++//! pages may be another surface's by then. + ++use super::command_buffer::CommandBuffer; ++use super::device::Device; ++use super::queue::CommandQueue; + use super::resource::ResourceKey; +-use super::tracker::{Use, UseRange}; ++use super::shadow::{bounding, CopyCause, ShadowCopies, ShadowCopy, Step}; ++use super::tracker::{Summary, Use, UseRange}; ++use crate::exit_guard; ++use std::any::Any; ++use std::cell::Cell; + use std::collections::VecDeque; ++use std::ops::Range; + use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex, OnceLock, RwLock, Weak}; +-use std::time::Instant; ++use std::time::{Duration, Instant}; + + /// The live switch. + const SWITCH: &str = "surface_lazy"; + const DEFAULT_MIN_TEXELS: u64 = 2_000_000; + const DEFAULT_RESYNC_MS: u64 = 250; ++/// The live switch that drops a standing by the pages written (the module docs). ++const SCOPED: &str = "surface_lazy_scoped"; ++/// The live switch that leaves a displayed surface's writes ahead in its image (the module docs). ++const PRESENT: &str = "surface_lazy_present"; ++/// With [`PRESENT`]: every lazy surface, presented or not. ++const PRESENT_ALL: &str = "surface_lazy_present_all"; ++/// How long a write stays ahead at most, unless nothing reads it. ++const DEFAULT_PRESENT_MAX_MS: u64 = 100; ++/// How often the counts go to stderr while a surface is lazy. ++const REPORT_MS: u64 = 2000; + /// The committed command buffers remembered: an older one's standing is not taken. + const COMMITTED_KEPT: usize = 256; + +@@ -58,6 +114,49 @@ static TAKEN: AtomicU64 = AtomicU64::new(0); + static REFUSED: AtomicU64 = AtomicU64::new(0); + static RESYNCS: AtomicU64 = AtomicU64::new(0); + static ACTIVE: AtomicUsize = AtomicUsize::new(0); ++/// The takes refused, by [`Refused`]: the first test each failed. ++static REFUSED_BY: [AtomicU64; Refused::COUNT] = [const { AtomicU64::new(0) }; Refused::COUNT]; ++/// The copies in made of lazy surfaces, by [`CopyCause`]: first, beyond held, changed, bounded. ++static IN_BY: [AtomicU64; 4] = [const { AtomicU64::new(0) }; 4]; ++/// How many surfaces hold writes ahead in their images: every hook returns at once while none does. ++static AHEAD: AtomicUsize = AtomicUsize::new(0); ++static DEFERRED: AtomicU64 = AtomicU64::new(0); ++static FLUSHES: AtomicU64 = AtomicU64::new(0); ++static FLUSHED_WHOLE: AtomicU64 = AtomicU64::new(0); ++static FLUSHED_TEXELS: AtomicU64 = AtomicU64::new(0); ++static DISCARDED: AtomicU64 = AtomicU64::new(0); ++static PRESENTS: AtomicU64 = AtomicU64::new(0); ++static PRESENTS_UNMATCHED: AtomicU64 = AtomicU64::new(0); ++/// Milliseconds since the first report's chance at the last report. ++static REPORTED: AtomicU64 = AtomicU64::new(0); ++ ++thread_local! { ++ /// The surface whose own copy back this thread records: neither a write that drops its standing ++ /// nor a use that asks for a copy back. ++ static OWN: Cell<*const Lazy> = const { Cell::new(std::ptr::null()) }; ++} ++ ++/// Why a take found no standing it could use. ++#[derive(Clone, Copy)] ++enum Refused { ++ /// None stood: the surface's first use, or an encoder's end left the image not holding the truth ++ /// ([`Lazy::fall`]). Not among [`REFUSED`]'s. ++ NoStanding, ++ /// An encoder's end that used the image came between ([`ShadowCopies::touched`]). ++ Touched, ++ /// A write of the truth came between ([`note_writes`], [`remapped`], [`cpu_wrote`]). ++ Written, ++ /// [`EPOCH`] moved. ++ Epoch, ++ /// The standing is the taker's own command buffer's. ++ OwnBuffer, ++ /// The standing's command buffer is not committed. ++ Uncommitted, ++} ++ ++impl Refused { ++ const COUNT: usize = 6; ++} + + fn registry() -> &'static RwLock>> { + static ALL: OnceLock>>> = OnceLock::new(); +@@ -69,22 +168,108 @@ fn committed_ids() -> &'static Mutex> { + IDS.get_or_init(Mutex::default) + } + +-/// Whether a texture over host pages of `texels` stays a shadow to render lazily (`host_rt.rs`). +-pub(crate) fn takes_surface(texels: u64) -> bool { ++/// Whether a texture over host pages of `width` and `texels` stays a shadow to render lazily ++/// (`host_rt.rs`): `surface_lazy_min_width` (default none) keeps a narrow surface, a Simulator's ++/// portrait display among them, out, as it is made and whatever is presented later. ++pub(crate) fn takes_surface(width: u32, texels: u64) -> bool { ++ let wide = crate::live::get("surface_lazy_min_width") ++ .and_then(|v| v.parse().ok()) ++ .is_none_or(|min: u32| width >= min); + crate::live::on(SWITCH, false) ++ && wide + && texels + >= crate::live::get("surface_lazy_min_texels") + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_MIN_TEXELS) + } + +-/// Note that what may hold any surface's bytes changed without a command. +-pub(crate) fn everything_written() { +- if ANY.load(Ordering::Relaxed) { ++/// Every surface registered, none held while the caller works. ++fn surfaces() -> Vec> { ++ (registry().read().unwrap().iter()) ++ .filter_map(Weak::upgrade) ++ .collect() ++} ++ ++/// Whether two ranges of host addresses share a byte. ++fn meet(a: &Range, b: &Range) -> bool { ++ a.start < b.end && b.start < a.end ++} ++ ++/// Note a change of the pages under `bytes` without a command (`Device::host_range_remapped`). ++pub(crate) fn remapped(bytes: Range) { ++ if !ANY.load(Ordering::Relaxed) { ++ return; ++ } ++ if !crate::live::on(SCOPED, false) { ++ EPOCH.fetch_add(1, Ordering::Relaxed); ++ return; ++ } ++ for lazy in surfaces() { ++ if lazy.over(&bytes) { ++ lazy.written.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++} ++ ++/// Note a CPU write through a texture over the host bytes `bytes`, `None` for one over none: a ++/// surface over them that holds writes ahead copies them back first, so the CPU's write stays. ++pub(crate) fn cpu_wrote(bytes: impl Fn() -> Option>) { ++ if !ANY.load(Ordering::Relaxed) { ++ return; ++ } ++ cpu_reads(&bytes); ++ if !crate::live::on(SCOPED, false) { + EPOCH.fetch_add(1, Ordering::Relaxed); ++ return; ++ } ++ let Some(bytes) = bytes() else { ++ return; ++ }; ++ for lazy in surfaces() { ++ if lazy.over(&bytes) { ++ lazy.written.fetch_add(1, Ordering::Relaxed); ++ } + } + } + ++/// Before a CPU read through a texture over the host bytes `bytes`: each surface over them that ++/// holds writes ahead copies them back, which this waits for. ++pub(crate) fn cpu_reads(bytes: impl Fn() -> Option>) { ++ if !ahead() { ++ return; ++ } ++ let Some(bytes) = bytes() else { ++ return; ++ }; ++ for lazy in surfaces() { ++ if lazy.over(&bytes) { ++ if let Some(flush) = lazy.flush() { ++ let _ = flush.wait_until_completed(); ++ } ++ } ++ } ++} ++ ++/// Note that the display presents the surface over the host bytes `bytes`: it leaves its writes ++/// ahead from now on, and those it holds are copied back now, or once their last writer commits. ++pub(crate) fn presented(bytes: impl Fn() -> Option>) { ++ if !ANY.load(Ordering::Relaxed) || !crate::live::on(PRESENT, false) { ++ return; ++ } ++ let bytes = bytes(); ++ let mut matched = false; ++ for lazy in surfaces() { ++ if bytes.as_ref().is_some_and(|bytes| lazy.over(bytes)) { ++ matched = true; ++ lazy.displayed.store(true, Ordering::Relaxed); ++ lazy.presents.fetch_add(1, Ordering::Relaxed); ++ lazy.flush_once_committed(); ++ } ++ } ++ let count = if matched { &PRESENTS } else { &PRESENTS_UNMATCHED }; ++ count.fetch_add(1, Ordering::Relaxed); ++} ++ + /// Whether any surface is lazy, one atomic read: while none is, a commit does nothing here. + pub(crate) fn active() -> bool { + ANY.load(Ordering::Relaxed) +@@ -109,13 +294,178 @@ pub(super) fn note_writes(uses: &[Use]) { + let all = registry().read().unwrap(); + for used in uses.iter().filter(|used| !used.write.is_empty()) { + for lazy in all.iter().filter_map(Weak::upgrade) { +- if lazy.meets(used) { ++ if lazy.meets(used) && !lazy.own_thread() { + lazy.written.fetch_add(1, Ordering::Relaxed); + } + } + } + } + ++/// Whether any surface holds writes ahead in its image, one atomic read. ++pub(crate) fn ahead() -> bool { ++ AHEAD.load(Ordering::Relaxed) > 0 ++} ++ ++/// The surfaces to copy back before the commit of a command buffer whose encoders' uses are ++/// `summaries`: each that committed writes of lie ahead and that it uses the bytes of (the module ++/// docs). A copy in is such a use, so its command buffer's own writes wait for its commit. ++pub(crate) fn met_by(summaries: &[&Summary]) -> Vec> { ++ let mut met = Vec::new(); ++ for lazy in surfaces() { ++ let settled = (lazy.pending.lock().unwrap().as_ref()) ++ .is_some_and(|left| left.settled.is_some()); ++ let used = || (summaries.iter()).any(|s| s.all_uses().iter().any(|u| lazy.meets(u))); ++ if settled && !lazy.own_thread() && used() { ++ met.push(lazy); ++ } ++ } ++ met ++} ++ ++/// After the commit of the command buffer of identity `id`: a copy back of what it wrote may run, ++/// and the presents that waited for it do. ++pub(crate) fn after_commit(id: u64) { ++ if !ahead() { ++ return; ++ } ++ for lazy in surfaces() { ++ if let Some(left) = lazy.pending.lock().unwrap().as_mut() { ++ left.committed(id); ++ } ++ let wants = ++ lazy.flush_after ++ .compare_exchange(id, 0, Ordering::Relaxed, Ordering::Relaxed); ++ if wants.is_ok() { ++ lazy.flush(); ++ } ++ } ++} ++ ++/// What earlier encoders of the command buffer of identity `id` left ahead, for the encoder opening ++/// now to copy back: each surface, its copy and what it holds until the copy completes. ++pub(crate) fn deferred_by(id: u64) -> Vec<(Arc, ShadowCopy, Vec)> { ++ let mut left = Vec::new(); ++ for lazy in surfaces() { ++ let mut pending = lazy.pending.lock().unwrap(); ++ let Some(ahead) = pending.as_mut() else { ++ continue; ++ }; ++ let Some(at) = ahead.open.iter().position(|(open, _)| *open == id) else { ++ continue; ++ }; ++ let (_, area) = ahead.open.swap_remove(at); ++ let copy = copy_back_of(ahead.copies.clone(), area); ++ let holds = ahead.holds.clone(); ++ if ahead.settled.is_none() && ahead.open.is_empty() { ++ *pending = None; ++ drop(pending); ++ lazy.left_behind(); ++ } else { ++ drop(pending); ++ } ++ left.push((lazy, copy, holds)); ++ } ++ left ++} ++ ++/// The copies back the clock asks for (the module docs): whether any surface holds writes ahead. ++fn flush_due(max: Duration) -> bool { ++ let mut any = false; ++ for lazy in surfaces() { ++ let due = (lazy.pending.lock().unwrap().as_ref()).map(|left| { ++ let waited = left.since.elapsed(); ++ waited >= max && (left.settled.is_some() || waited >= max * 4) ++ }); ++ any |= due.is_some(); ++ if due == Some(true) { ++ let Some(_section) = exit_guard::enter() else { ++ return false; ++ }; ++ exit_guard::lowering_program_calls(|| lazy.flush()); ++ } ++ } ++ any ++} ++ ++/// The thread of the clock, started with the first write left ahead. ++fn start_clock() -> Option { ++ static CLOCK: OnceLock> = OnceLock::new(); ++ CLOCK ++ .get_or_init(|| { ++ let builder = std::thread::Builder::new().name("m2v-surface-clock".into()); ++ let run = || loop { ++ if exit_guard::closed() { ++ return; ++ } ++ let max = (crate::live::get("surface_lazy_present_max_ms")) ++ .and_then(|v| v.parse().ok()) ++ .unwrap_or(DEFAULT_PRESENT_MAX_MS) ++ .max(1); ++ match flush_due(Duration::from_millis(max)) { ++ true => std::thread::sleep(Duration::from_millis(max.div_ceil(4)).min(exit_guard::WAKE)), ++ false => std::thread::park_timeout(exit_guard::WAKE), ++ } ++ }; ++ let thread = exit_guard::spawn(builder, run).ok()?; ++ Some(thread.thread().clone()) ++ }) ++ .clone() ++} ++ ++/// The counts by reason and by surface, on stderr at most every [`REPORT_MS`] ms. ++fn report() { ++ static START: OnceLock = OnceLock::new(); ++ let now = START.get_or_init(Instant::now).elapsed().as_millis() as u64; ++ let last = REPORTED.load(Ordering::Relaxed); ++ if now < last + REPORT_MS ++ || REPORTED ++ .compare_exchange(last, now, Ordering::Relaxed, Ordering::Relaxed) ++ .is_err() ++ { ++ return; ++ } ++ let read = |count: &AtomicU64| count.load(Ordering::Relaxed); ++ let (taken, refused, resyncs) = counts(); ++ let by = REFUSED_BY.each_ref().map(read); ++ let copied = IN_BY.each_ref().map(read); ++ eprintln!( ++ "surface_lazy: taken {taken} refused {refused} resyncs {resyncs} | refused: touched {} \ ++ written {} epoch {} own {} uncommitted {} no-standing {} | in: first {} beyond-held {} \ ++ changed {} bounded {} | ahead: deferred {} flushes {} whole {} texels {} discarded {} \ ++ presents {} unmatched {}", ++ by[Refused::Touched as usize], ++ by[Refused::Written as usize], ++ by[Refused::Epoch as usize], ++ by[Refused::OwnBuffer as usize], ++ by[Refused::Uncommitted as usize], ++ by[Refused::NoStanding as usize], ++ copied[0], ++ copied[1], ++ copied[2], ++ copied[3], ++ read(&DEFERRED), ++ read(&FLUSHES), ++ read(&FLUSHED_WHOLE), ++ read(&FLUSHED_TEXELS), ++ read(&DISCARDED), ++ read(&PRESENTS), ++ read(&PRESENTS_UNMATCHED), ++ ); ++ for lazy in surfaces().iter().take(16) { ++ eprintln!( ++ "surface_lazy: {:#x} {}x{} displayed {} taken {} refused {} in {} presents {}", ++ lazy.pages.as_ref().map_or(0, |pages| pages.start), ++ lazy.size[0], ++ lazy.size[1], ++ lazy.displayed.load(Ordering::Relaxed), ++ read(&lazy.taken), ++ read(&lazy.refused), ++ read(&lazy.copied_in), ++ read(&lazy.presents), ++ ); ++ } ++} ++ + /// The counts so far, for a lane or a stderr line: standings taken, refused, and resyncs. + pub fn counts() -> (u64, u64, u64) { + ( +@@ -138,34 +488,319 @@ struct Stood { + epoch: u64, + } + ++/// What a copy back holds until it completes: its image and its bytes. Also the shape of [`Lazy`]'s ++/// queue, device and copies: named, they lie past the compiler's depth for `Send` and `Sync`, in ++/// this crate and in every crate that uses it. ++pub(crate) type Hold = Arc; ++ ++/// What a surface's encoders left ahead in its image: its bytes hold none of it yet. ++struct Pending { ++ /// The rectangle bounding what committed command buffers wrote, `Some(None)` for every texel: ++ /// what a copy back moves. `None` for nothing yet. ++ settled: Option>, ++ /// What each command buffer not committed wrote: no copy back can run after it yet, and one ++ /// over its texels would write what the image holds without it, undefined if it copied none ++ /// in. Moved to `settled` once its commit returns ([`after_commit`]), not at [`committed`], ++ /// which notes a commit about to be made. ++ open: Vec<(u64, Option<[u32; 4]>)>, ++ /// When the first of them was left, or the last copy back that left some. ++ since: Instant, ++ /// The `Arc` to copy back. ++ copies: Hold, ++ holds: Vec, ++} ++ ++impl Pending { ++ /// Note that the command buffer of identity `id` committed. ++ fn committed(&mut self, id: u64) { ++ if let Some(at) = self.open.iter().position(|(open, _)| *open == id) { ++ let (_, area) = self.open.swap_remove(at); ++ self.settled = Some(match self.settled { ++ Some(settled) => bounding(settled, area), ++ None => area, ++ }); ++ } ++ } ++} ++ ++/// A copy back of `area` of `copies`, as an encoder's end makes it. ++fn copy_back_of(copies: Hold, area: Option<[u32; 4]>) -> ShadowCopy { ++ ShadowCopy { ++ copies: copies.downcast().expect("a surface's copies"), ++ step: Step::Back, ++ kept: false, ++ area, ++ cause: None, ++ moves: None, ++ } ++} ++ ++/// The writes left ahead so far, for a lane. ++#[derive(Clone, Copy, Debug, Default)] ++pub struct AheadCounts { ++ /// The copies back encoders left for later. ++ pub deferred: u64, ++ /// The copies back made of them. ++ pub flushes: u64, ++ /// Of those, the ones of every texel. ++ pub whole: u64, ++ /// The texels those copies moved. ++ pub texels: u64, ++ /// The copies dropped, their texture gone. ++ pub discarded: u64, ++} ++ ++/// The counts of writes left ahead so far. ++pub fn ahead_counts() -> AheadCounts { ++ let read = |count: &AtomicU64| count.load(Ordering::Relaxed); ++ AheadCounts { ++ deferred: read(&DEFERRED), ++ flushes: read(&FLUSHES), ++ whole: read(&FLUSHED_WHOLE), ++ texels: read(&FLUSHED_TEXELS), ++ discarded: read(&DISCARDED), ++ } ++} ++ + /// One surface's standing. +-pub(super) struct Lazy { ++pub(crate) struct Lazy { + truth: OnceLock<(ResourceKey, Option, UseRange)>, + written: AtomicU64, + stood: Mutex>, + resynced: Mutex, ++ /// The [`Device`] its queue comes from. ++ device: Hold, ++ /// The host addresses of its pages; `None` where they lie in none now. ++ pages: Option>, ++ /// Its width and height, for the counts. ++ size: [u32; 2], ++ /// The display presented it ([`presented`]). ++ displayed: AtomicBool, ++ /// Weakly, what its copies back hold ([`Lazy::keeps`]), set as its texture is made. ++ keeps: OnceLock<[Weak; 2]>, ++ pending: Mutex>, ++ /// `pending` holds one, one atomic read. ++ ahead: AtomicBool, ++ /// The identity of the command buffer to copy back after the commit of, 0 for none. ++ flush_after: AtomicU64, ++ /// The [`CommandQueue`] of its copies back, made with its first write left ahead. ++ queue: OnceLock, ++ taken: AtomicU64, ++ refused: AtomicU64, ++ copied_in: AtomicU64, ++ presents: AtomicU64, + } + + impl Lazy { +- /// The state of a new shadow of `texels` texels with `surface` as its backing, where the switch +- /// takes it. +- pub(super) fn of(surface: bool, texels: u64) -> Option> { +- (surface && takes_surface(texels)).then(|| { ++ /// The state of a new shadow of `size` and `texels` texels of `device` with `surface` as its ++ /// backing over the host addresses `pages`, where the switch takes it. ++ pub(super) fn of( ++ device: &Device, ++ surface: bool, ++ pages: impl FnOnce() -> Option>, ++ size: [u32; 2], ++ texels: u64, ++ ) -> Option> { ++ (surface && takes_surface(size[0], texels)).then(|| { + Arc::new(Lazy { + truth: OnceLock::new(), + written: AtomicU64::new(0), + stood: Mutex::new(None), + resynced: Mutex::new(Instant::now()), ++ device: Arc::new(device.clone()), ++ pages: pages(), ++ size, ++ displayed: AtomicBool::new(false), ++ keeps: OnceLock::new(), ++ pending: Mutex::new(None), ++ ahead: AtomicBool::new(false), ++ flush_after: AtomicU64::new(0), ++ queue: OnceLock::new(), ++ taken: AtomicU64::new(0), ++ refused: AtomicU64::new(0), ++ copied_in: AtomicU64::new(0), ++ presents: AtomicU64::new(0), + }) + }) + } + ++ /// Note what its copies back hold, weakly: its image, and the bytes its texture lies over. ++ pub(crate) fn keeps(&self, image: Weak, bytes: Weak) { ++ let _ = self.keeps.set([image, bytes]); ++ } ++ + fn meets(&self, used: &Use) -> bool { + self.truth.get().is_some_and(|(key, alias, range)| { + (used.key == *key || Some(used.key) == *alias) && used.range.overlaps(range) + }) + } + ++ /// Whether its pages meet the host addresses `bytes`. ++ fn over(&self, bytes: &Range) -> bool { ++ self.pages.as_ref().is_some_and(|pages| meet(pages, bytes)) ++ } ++ ++ /// Whether this thread records this surface's own copy back. ++ fn own_thread(&self) -> bool { ++ OWN.with(Cell::get) == std::ptr::from_ref(self) ++ } ++ ++ /// Mark this thread as recording its own copy back until the mark drops. ++ pub(crate) fn own(&self) -> impl Drop { ++ struct Own(*const Lazy); ++ impl Drop for Own { ++ fn drop(&mut self) { ++ OWN.with(|own| own.set(self.0)); ++ } ++ } ++ Own(OWN.with(|own| own.replace(std::ptr::from_ref(self)))) ++ } ++ ++ /// Note a copy in of its bytes for `cause`. ++ pub(super) fn copied_in(&self, cause: CopyCause) { ++ let at = match cause { ++ CopyCause::First => 0, ++ CopyCause::BeyondHeld => 1, ++ CopyCause::Changed(_) => 2, ++ CopyCause::Bounded => 3, ++ }; ++ IN_BY[at].fetch_add(1, Ordering::Relaxed); ++ self.copied_in.fetch_add(1, Ordering::Relaxed); ++ } ++ ++ /// Leave what an encoder of the command buffer of identity `id` wrote inside `area`, `None` for ++ /// anywhere, ahead in the image: whether it does, else its copy back goes as ever. ++ pub(super) fn defer( ++ &self, ++ copies: &Arc, ++ area: Option<[u32; 4]>, ++ id: u64, ++ ) -> bool { ++ if !crate::live::on(PRESENT, false) ++ || !(self.displayed.load(Ordering::Relaxed) || crate::live::on(PRESENT_ALL, false)) ++ { ++ return false; ++ } ++ let Some(holds) = self.keeps.get() else { ++ return false; ++ }; ++ let holds: Option> = holds.iter().map(Weak::upgrade).collect(); ++ let Some(holds) = holds else { ++ return false; ++ }; ++ if self.queue.get().is_none() { ++ let device = self.device.downcast_ref::(); ++ let Some(queue) = device.and_then(|d| d.new_command_queue().ok()) else { ++ return false; ++ }; ++ let _ = self.queue.set(Arc::new(queue)); ++ } ++ let mut pending = self.pending.lock().unwrap(); ++ match &mut *pending { ++ Some(left) => match left.open.iter_mut().find(|(open, _)| *open == id) { ++ Some((_, wrote)) => *wrote = bounding(*wrote, area), ++ None => left.open.push((id, area)), ++ }, ++ None => { ++ *pending = Some(Pending { ++ settled: None, ++ open: vec![(id, area)], ++ since: Instant::now(), ++ copies: copies.clone(), ++ holds, ++ }); ++ drop(pending); ++ self.ahead.store(true, Ordering::Relaxed); ++ if AHEAD.fetch_add(1, Ordering::Relaxed) == 0 { ++ if let Some(clock) = start_clock() { ++ clock.unpark(); ++ } ++ } ++ } ++ } ++ DEFERRED.fetch_add(1, Ordering::Relaxed); ++ true ++ } ++ ++ /// `pending` was emptied. ++ fn left_behind(&self) { ++ self.ahead.store(false, Ordering::Relaxed); ++ AHEAD.fetch_sub(1, Ordering::Relaxed); ++ } ++ ++ /// Copy what it holds ahead back now, or once its last writer commits ([`after_commit`]). ++ fn flush_once_committed(self: &Arc) { ++ let pending = self.pending.lock().unwrap(); ++ let Some(left) = pending.as_ref() else { ++ return; ++ }; ++ if let Some((last, _)) = left.open.last() { ++ self.flush_after.store(*last, Ordering::Relaxed); ++ return; ++ } ++ drop(pending); ++ self.flush(); ++ } ++ ++ /// Copy what committed command buffers left ahead back, in a command buffer of its own, ++ /// committed before this returns: that command buffer, where one was made. Nothing is copied ++ /// over the pages of a texture nothing else holds (the module docs); a copy that cannot be ++ /// recorded stays for the next asker. ++ pub(crate) fn flush(self: &Arc) -> Option { ++ let (area, copies, holds) = { ++ let pending = self.pending.lock().unwrap(); ++ let left = pending.as_ref()?; ++ (left.settled, left.copies.clone(), left.holds.clone()) ++ }; ++ // The image's: `Pending`'s and this one, where neither texture nor command holds it. ++ if holds.first().is_some_and(|image| Arc::strong_count(image) <= 2) { ++ *self.pending.lock().unwrap() = None; ++ self.left_behind(); ++ DISCARDED.fetch_add(1, Ordering::Relaxed); ++ return None; ++ } ++ let area = area?; ++ let _own = self.own(); ++ let copy = copy_back_of(copies, area); ++ let texels = copy.texels_back(); ++ let flush = (|| { ++ let queue = self.queue.get()?.downcast_ref::()?; ++ let command_buffer = queue.command_buffer().ok()?; ++ let mut encoder = command_buffer.blit_command_encoder().ok()?; ++ encoder.surface_copy_back(©, holds); ++ encoder.end_encoding().ok()?; ++ command_buffer.commit().ok()?; ++ Some(command_buffer) ++ })(); ++ let Some(flush) = flush else { ++ eprintln!("surface_lazy: a copy back was not recorded"); ++ return None; ++ }; ++ self.settle(area); ++ FLUSHES.fetch_add(1, Ordering::Relaxed); ++ FLUSHED_TEXELS.fetch_add(texels, Ordering::Relaxed); ++ FLUSHED_WHOLE.fetch_add(u64::from(area.is_none()), Ordering::Relaxed); ++ Some(flush) ++ } ++ ++ /// `area` is back in the bytes, and nothing else unless more came since it was read, which ++ /// the next copy back repeats it with. ++ fn settle(&self, area: Option<[u32; 4]>) { ++ let mut pending = self.pending.lock().unwrap(); ++ let Some(left) = pending.as_mut() else { ++ return; ++ }; ++ if left.settled == Some(area) { ++ left.settled = None; ++ } ++ left.since = Instant::now(); ++ if left.settled.is_none() && left.open.is_empty() { ++ *pending = None; ++ drop(pending); ++ self.left_behind(); ++ } ++ } ++ + /// Leave the image's standing: `id`'s encoder ended with the image holding the truth, inside + /// `held` where given, stale inside `stale`. Its own copies back are behind it. + pub(super) fn stand( +@@ -201,20 +836,35 @@ impl Lazy { + /// Where the image stands for an encoder that met `touched`, of a command buffer of identity + /// `id`: `Some((held, stale))` where it needs no copy in (the module docs). + pub(super) fn take(&self, id: u64, touched: u64) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { +- let stood = (*self.stood.lock().unwrap())?; +- let current = stood.touched == touched +- && stood.written == self.written.load(Ordering::Relaxed) +- && stood.epoch == EPOCH.load(Ordering::Relaxed) +- && stood.id != id +- && committed_ids().lock().unwrap().contains(&stood.id); +- if !current { ++ report(); ++ let Some(stood) = *self.stood.lock().unwrap() else { ++ REFUSED_BY[Refused::NoStanding as usize].fetch_add(1, Ordering::Relaxed); ++ return None; ++ }; ++ let refused = if stood.touched != touched { ++ Some(Refused::Touched) ++ } else if stood.written != self.written.load(Ordering::Relaxed) { ++ Some(Refused::Written) ++ } else if stood.epoch != EPOCH.load(Ordering::Relaxed) { ++ Some(Refused::Epoch) ++ } else if stood.id == id { ++ Some(Refused::OwnBuffer) ++ } else if !committed_ids().lock().unwrap().contains(&stood.id) { ++ Some(Refused::Uncommitted) ++ } else { ++ None ++ }; ++ if let Some(refused) = refused { + REFUSED.fetch_add(1, Ordering::Relaxed); ++ REFUSED_BY[refused as usize].fetch_add(1, Ordering::Relaxed); ++ self.refused.fetch_add(1, Ordering::Relaxed); + return None; + } + let every = crate::live::get("surface_lazy_resync_ms") + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_RESYNC_MS); +- if every > 0 { ++ // Writes left ahead would be lost to a copy in, and a copy back goes over the bytes anyway. ++ if every > 0 && !self.ahead.load(Ordering::Relaxed) { + let mut resynced = self.resynced.lock().unwrap(); + if resynced.elapsed().as_millis() as u64 >= every { + *resynced = Instant::now(); +@@ -227,6 +877,7 @@ impl Lazy { + } + } + TAKEN.fetch_add(1, Ordering::Relaxed); ++ self.taken.fetch_add(1, Ordering::Relaxed); + Some((stood.held, stood.stale)) + } + } +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index dc5b4a08..9644b731 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -2429,8 +2429,15 @@ impl Device { + (object, why) + }); + self.shared().counts.shadowed_texture_made(aspects != every); ++ let image = image.shared(); ++ // LOCAL (Atlas): a copy back left for later holds them (`shadow_lazy.rs`). ++ if let Some(lazy) = &copies.lazy { ++ let (held, bytes): (Arc, Arc) = ++ (image.clone(), texels.clone()); ++ lazy.keeps(Arc::downgrade(&held), Arc::downgrade(&bytes)); ++ } + Ok(Storage::Shadow { +- image: image.shared(), ++ image, + texels, + copies: Arc::new(copies), + }) +@@ -4223,6 +4230,23 @@ impl Texture { + } + } + ++ /// LOCAL (Atlas): the host addresses of the bytes the texture lies over, where it lies over a ++ /// buffer's in host pages. ++ fn host_range(&self) -> Option> { ++ match &self.0.storage { ++ Storage::Texels(texels) ++ | Storage::Alias { texels, .. } ++ | Storage::Shadow { texels, .. } => texels.buffer.host_range(), ++ Storage::Image(_) => None, ++ } ++ } ++ ++ /// LOCAL (Atlas): tell the runtime the display presents the texture's bytes ++ /// (`shadow_lazy.rs`). Not a Metal call. ++ pub fn surface_presented(&self) { ++ crate::shadow_lazy::presented(|| self.host_range()); ++ } ++ + /// The buffer's bytes a texture buffer is, or `None` for a texture with an image. + pub(super) fn texels(&self) -> Option<&Texels> { + match &self.0.storage { +@@ -5372,7 +5396,7 @@ impl Texture { + bytes_per_row: u64, + bytes_per_image: u64, + ) -> Result<(), Error> { +- crate::shadow_lazy::everything_written(); // LOCAL (Atlas) ++ crate::shadow_lazy::cpu_wrote(|| self.host_range()); // LOCAL (Atlas) + let upload = Upload::Slice(bytes); + self.replace(region, level, slice, upload, bytes_per_row, bytes_per_image) + } +@@ -5394,7 +5418,7 @@ impl Texture { + bytes_per_row: u64, + bytes_per_image: u64, + ) -> Result<(), Error> { +- crate::shadow_lazy::everything_written(); // LOCAL (Atlas) ++ crate::shadow_lazy::cpu_wrote(|| self.host_range()); // LOCAL (Atlas) + let upload = Upload::Host { + bytes, + copy: OnceCell::new(), +@@ -5599,6 +5623,7 @@ impl Texture { + return Ok(()); + }; + self.refuse_detached_texels()?; ++ crate::shadow_lazy::cpu_reads(|| self.host_range()); // LOCAL (Atlas) + if copy + .host + .host_len() +diff --git a/crates/metal2vulkan-runtime/src/tracker.rs b/crates/metal2vulkan-runtime/src/tracker.rs +index 2724bb88..f2eb5c80 100644 +--- a/crates/metal2vulkan-runtime/src/tracker.rs ++++ b/crates/metal2vulkan-runtime/src/tracker.rs +@@ -1884,6 +1884,11 @@ impl EncoderUses { + } + + impl Summary { ++ /// LOCAL (Atlas): the tracked uses (`shadow_lazy.rs`). ++ pub(super) fn all_uses(&self) -> &[Use] { ++ &self.uses ++ } ++ + /// The tracked uses. + #[cfg(test)] + pub(super) fn uses(&self) -> &[Use] { +diff --git a/crates/metal2vulkan-validation/src/runtime_lane.rs b/crates/metal2vulkan-validation/src/runtime_lane.rs +index 613e8e37..573b692e 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane.rs +@@ -783,6 +783,8 @@ mod submit_costs; + mod submit_per_import; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd)))] + mod surface_aliases; ++#[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] ++mod surface_present; // LOCAL (Atlas) + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd)))] + mod surface_fills; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs b/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs +index 491f874d..5ed99a36 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/host_rt_frames.rs +@@ -135,7 +135,7 @@ fn median_p90(mut values: Vec) -> (f64, f64) { + (at(50), at(90)) + } + +-fn surface_buffer(device: &Device, mapping: &Mapping, len: usize) -> Buffer { ++pub(super) fn surface_buffer(device: &Device, mapping: &Mapping, len: usize) -> Buffer { + let managed = ResourceOptions::new( + StorageMode::Managed, + CpuCacheMode::DefaultCache, +@@ -168,7 +168,7 @@ fn form_of(texture: &Texture) -> &'static str { + } + } + +-fn pass( ++pub(super) fn pass( + command_buffer: &CommandBuffer, + texture: &Texture, + blend: &RenderPipelineState, +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs +new file mode 100644 +index 00000000..bf273946 +--- /dev/null ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs +@@ -0,0 +1,336 @@ ++//! LOCAL (Atlas): a surface that leaves its writes ahead in its image under the switch ++//! `surface_lazy_present` (`shadow_lazy.rs`). A texture of the surface renders through a shadow ++//! kept across command buffers; these check what its pages hold, and what every other way to its ++//! bytes reads, before and after each thing that copies the writes back. ++ ++use super::first_use_writes::CHECK_WAIT; ++use super::host_memory::{gpu_read, run, Mapping}; ++use super::host_rt_frames::{pass, surface_buffer}; ++use super::no_copy_textures::{rects, staged}; ++use super::{validated_device_with, Messages, ScratchCache}; ++use crate::gpu::TEST_DEVICE; ++use metal2vulkan::device::HostPageTexturePolicy; ++use metal2vulkan::metal::{PixelFormat, TextureUsage}; ++use metal2vulkan_runtime::shadow_lazy::{ahead_counts, counts}; ++use metal2vulkan_runtime::{ ++ live, Buffer, CommandQueue, Device, IoSurfacePlane, Region, RenderPipelineState, ScissorRect, ++ Texture, TextureDescriptor, ++}; ++use std::time::{Duration, Instant}; ++ ++const WIDTH: u64 = 1024; ++const HEIGHT: u64 = 256; ++const ROW: u64 = 4096; ++const BEFORE: u8 = 0x55; ++const RED: [f32; 4] = [1.0, 0.0, 0.0, 1.0]; ++const GREEN: [f32; 4] = [0.0, 1.0, 0.0, 1.0]; ++const RED_TEXEL: [u8; 4] = [0, 0, 255, 255]; ++const GREEN_TEXEL: [u8; 4] = [0, 255, 0, 255]; ++/// A bound no case's copy back should near. ++const SOON: Duration = Duration::from_secs(3); ++ ++/// A surface under the switches, its texture and what draws into it. ++struct Case<'d> { ++ device: &'d Device, ++ queue: CommandQueue, ++ mapping: Mapping, ++ len: usize, ++ buffer: Buffer, ++ texture: Texture, ++ opaque: RenderPipelineState, ++ rect: Buffer, ++} ++ ++impl Case<'_> { ++ /// One command buffer of a pass filling `damage`, or everything, with `color`, run to its end. ++ fn draw(&self, color: [f32; 4], damage: Option) { ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ self.pass(&command_buffer, color, damage); ++ run(&command_buffer); ++ } ++ ++ fn pass( ++ &self, ++ command_buffer: &metal2vulkan_runtime::CommandBuffer, ++ color: [f32; 4], ++ damage: Option, ++ ) { ++ let color = staged(self.device, &floats4(color)); ++ let rects = std::slice::from_ref(&self.rect); ++ pass(command_buffer, &self.texture, &self.opaque, rects, &color, damage); ++ } ++ ++ /// The texel at `x`, `y` of the pages. ++ fn page_texel(&self, x: u64, y: u64) -> [u8; 4] { ++ let at = (y * ROW + x * 4) as usize; ++ self.mapping.read(at..at + 4).try_into().unwrap() ++ } ++ ++ /// A new buffer over the same pages, as another import of the surface is. ++ fn other_import(&self) -> Buffer { ++ surface_buffer(self.device, &self.mapping, self.len) ++ } ++ ++ /// Wait until the pages hold `texel` at `x`, `y`. ++ fn pages_become(&self, x: u64, y: u64, texel: [u8; 4]) { ++ let start = Instant::now(); ++ while self.page_texel(x, y) != texel { ++ assert!( ++ start.elapsed() < SOON, ++ "the pages hold {:?} at {x}, {y}, not {texel:?}", ++ self.page_texel(x, y) ++ ); ++ std::thread::sleep(Duration::from_millis(2)); ++ } ++ } ++} ++ ++fn floats4(values: [f32; 4]) -> Vec { ++ values.iter().flat_map(|v| v.to_ne_bytes()).collect() ++} ++ ++fn switches(present: bool, all: bool, scoped: bool, max_ms: u64) { ++ let on = |yes: bool| if yes { "1" } else { "0" }; ++ live::set("surface_lazy", Some("1")); ++ live::set("surface_lazy_min_texels", Some("1")); ++ live::set("surface_lazy_resync_ms", Some("0")); ++ live::set("surface_lazy_present", Some(on(present))); ++ live::set("surface_lazy_present_all", Some(on(all))); ++ live::set("surface_lazy_scoped", Some(on(scoped))); ++ live::set("surface_lazy_present_max_ms", Some(&max_ms.to_string())); ++} ++ ++/// Run `test` on a new surface whose every byte is [`BEFORE`], on a validated device that takes ++/// textures over host pages in place, as the product's does. ++fn on_surface(test: impl FnOnce(Case<'_>)) { ++ let cache = ScratchCache::new().unwrap(); ++ let messages = Messages::default(); ++ { ++ let device = validated_device_with(TEST_DEVICE, &messages, cache.path(), |o| { ++ o.policy.host_page_textures = HostPageTexturePolicy::InPlace; ++ }) ++ .unwrap(); ++ assert!(device.settle_in_place_checks(CHECK_WAIT)); ++ let len = ((ROW * HEIGHT) as usize).next_multiple_of(16384); ++ let mapping = Mapping::anonymous(len); ++ mapping.fill(BEFORE); ++ let buffer = surface_buffer(&device, &mapping, len); ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = buffer.resource_options(); ++ let texture = buffer ++ .new_texture_with_iosurface_plane(&d, IoSurfacePlane::new(0, ROW, false)) ++ .unwrap(); ++ // SAFETY: the handle is only tested, while the texture lives. ++ assert!(unsafe { texture.vk_image() }.is_none(), "not a shadow"); ++ let rect = [-1.0f32, -1.0, 1.0, 1.0].iter().flat_map(|v| v.to_ne_bytes()); ++ test(Case { ++ device: &device, ++ queue: device.new_command_queue().unwrap(), ++ len, ++ buffer, ++ texture, ++ opaque: rects(&device, PixelFormat::BGRA8Unorm, false), ++ rect: staged(&device, &rect.collect::>()), ++ mapping, ++ }); ++ } ++ live::set("surface_lazy", None); ++ live::set("surface_lazy_present", None); ++ live::set("surface_lazy_present_all", None); ++ live::set("surface_lazy_scoped", None); ++ let reported = messages.take(); ++ assert!(reported.is_empty(), "the layer reported: {reported:#?}"); ++} ++ ++#[test] ++fn surface_present_a_presented_surface_copies_back_at_its_presents() { ++ switches(true, false, false, 60_000); ++ on_surface(|case| { ++ // Not presented yet: its copy back goes as ever. ++ case.draw(RED, None); ++ assert_eq!(case.page_texel(5, 5), RED_TEXEL); ++ case.texture.surface_presented(); ++ let before = ahead_counts(); ++ case.draw(GREEN, None); ++ assert_eq!(ahead_counts().deferred, before.deferred + 1); ++ assert_eq!(case.page_texel(5, 5), RED_TEXEL, "left ahead"); ++ case.draw(RED, None); ++ case.draw(GREEN, None); ++ assert_eq!(case.page_texel(5, 5), RED_TEXEL, "left ahead"); ++ case.texture.surface_presented(); ++ case.pages_become(5, 5, GREEN_TEXEL); ++ assert_eq!(ahead_counts().flushes, before.flushes + 1, "once for three"); ++ }); ++} ++ ++#[test] ++fn surface_present_a_reader_through_another_import_reads_the_writes() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ case.draw(RED, None); ++ assert_eq!(case.page_texel(5, 5), [BEFORE; 4], "left ahead"); ++ let read = gpu_read(&case.queue, &case.other_import(), 0..4); ++ assert_eq!(read, RED_TEXEL); ++ // And a blit of the surface's own buffer, on its own queue. ++ case.draw(GREEN, None); ++ let queue = case.device.new_command_queue().unwrap(); ++ assert_eq!(gpu_read(&queue, &case.buffer, 0..4), GREEN_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_present_a_later_encoder_of_the_writer_reads_the_writes() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ let other = case.other_import(); ++ let copy = case ++ .device ++ .new_buffer(4, metal2vulkan_runtime::ResourceOptions::shared()) ++ .unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, RED, None); ++ let mut blit = command_buffer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&other, 0, ©, 0, 4) ++ .unwrap(); ++ blit.end_encoding().unwrap(); ++ run(&command_buffer); ++ // SAFETY: a `Shared` buffer no command uses now. ++ let read = unsafe { std::slice::from_raw_parts(copy.contents().unwrap().as_ptr(), 4) }; ++ assert_eq!(read, RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_present_passes_of_one_command_buffer_end_in_the_pages() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, RED, None); ++ let half = ScissorRect { x: 0, y: 0, width: WIDTH / 2, height: HEIGHT }; ++ case.pass(&command_buffer, GREEN, Some(half)); ++ run(&command_buffer); ++ case.texture.surface_presented(); ++ case.pages_become(5, 5, GREEN_TEXEL); ++ assert_eq!(case.page_texel(WIDTH - 5, 5), RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_present_the_clock_copies_back_after_the_guest_goes_idle() { ++ switches(true, true, false, 20); ++ on_surface(|case| { ++ case.draw(RED, None); ++ case.pages_become(5, 5, RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_present_cpu_calls_see_and_keep_the_truth() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ case.draw(RED, None); ++ let mut read = [0u8; 4]; ++ case.texture ++ .get_bytes(&mut read, ROW, 0, Region::new_2d(5, 5, 1, 1), 0, 0) ++ .unwrap(); ++ assert_eq!(read, RED_TEXEL); ++ case.draw(GREEN, None); ++ case.texture ++ .replace_region(Region::new_2d(1, 1, 1, 1), 0, 0, &[1, 2, 3, 4], ROW, 0) ++ .unwrap(); ++ assert_eq!(case.page_texel(5, 5), GREEN_TEXEL, "copied back first"); ++ assert_eq!(case.page_texel(1, 1), [1, 2, 3, 4]); ++ // The next pass leaves the CPU's texel where it does not draw. ++ let right = ScissorRect { x: WIDTH / 2, y: 0, width: WIDTH / 2, height: HEIGHT }; ++ case.draw(RED, Some(right)); ++ case.texture.surface_presented(); ++ case.pages_become(WIDTH - 5, 5, RED_TEXEL); ++ assert_eq!(case.page_texel(1, 1), [1, 2, 3, 4]); ++ }); ++} ++ ++#[test] ++fn surface_present_a_copy_back_covers_the_union_of_the_scissors() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ let at = |x| ScissorRect { x, y: 0, width: 100, height: 50 }; ++ // The image holds every texel from here on, so no pass copies in. ++ case.draw(RED, None); ++ case.texture.surface_presented(); ++ case.pages_become(250, 5, RED_TEXEL); ++ let before = ahead_counts(); ++ case.draw(GREEN, Some(at(0))); ++ case.draw(GREEN, Some(at(100))); ++ case.texture.surface_presented(); ++ case.pages_become(150, 5, GREEN_TEXEL); ++ let after = ahead_counts(); ++ assert_eq!(case.page_texel(50, 5), GREEN_TEXEL); ++ assert_eq!(case.page_texel(250, 5), RED_TEXEL); ++ assert_eq!(after.flushes, before.flushes + 1); ++ assert_eq!(after.whole, before.whole, "bounded"); ++ assert_eq!(after.texels - before.texels, 200 * 50); ++ }); ++} ++ ++#[test] ++fn surface_present_a_released_texture_writes_nothing_over_its_pages() { ++ switches(true, true, false, 20); ++ on_surface(|case| { ++ let before = ahead_counts(); ++ let Case { device, queue, mapping, buffer, texture, opaque, rect, len } = case; ++ let case = Case { device, queue, mapping, len, buffer, texture, opaque, rect }; ++ case.draw(RED, None); ++ let Case { mapping, texture, buffer, .. } = case; ++ drop((texture, buffer)); ++ let start = Instant::now(); ++ while ahead_counts().discarded == before.discarded { ++ assert!(start.elapsed() < SOON, "not dropped: {:?}", ahead_counts()); ++ std::thread::sleep(Duration::from_millis(2)); ++ } ++ assert_eq!(mapping.read(20..24), [BEFORE; 4]); ++ }); ++} ++ ++/// How many takes a remap of `elsewhere` refuses. ++fn refused_by_remap(case: &Case<'_>, elsewhere: &Mapping) -> u64 { ++ case.draw(RED, None); ++ case.draw(GREEN, None); ++ let (_, before, _) = counts(); ++ case.device ++ .host_range_remapped(elsewhere.pointer(), 16384); ++ case.draw(RED, None); ++ counts().1 - before ++} ++ ++#[test] ++fn surface_lazy_scoped_a_remap_elsewhere_keeps_the_standing() { ++ let elsewhere = Mapping::anonymous(16384); ++ switches(false, false, true, 60_000); ++ on_surface(|case| assert_eq!(refused_by_remap(&case, &elsewhere), 0)); ++ // A remap of its own pages drops it all the same. ++ switches(false, false, true, 60_000); ++ on_surface(|case| { ++ case.draw(RED, None); ++ case.draw(GREEN, None); ++ let (_, before, _) = counts(); ++ case.device.host_range_remapped(case.mapping.pointer(), 16384); ++ case.draw(RED, None); ++ assert_eq!(counts().1 - before, 1); ++ }); ++} ++ ++#[test] ++fn surface_lazy_off_the_new_switches_change_nothing() { ++ let elsewhere = Mapping::anonymous(16384); ++ switches(false, false, false, 60_000); ++ on_surface(|case| { ++ let before = ahead_counts(); ++ case.texture.surface_presented(); ++ case.draw(RED, None); ++ assert_eq!(case.page_texel(5, 5), RED_TEXEL); ++ assert_eq!(ahead_counts().deferred, before.deferred); ++ // Unscoped, any remap refuses the next take. ++ assert_eq!(refused_by_remap(&case, &elsewhere), 1); ++ }); ++} diff --git a/patches/metal2vulkan/0012-surface_lazy-register-a-surface-that-leaves-writes-a.patch b/patches/metal2vulkan/0012-surface_lazy-register-a-surface-that-leaves-writes-a.patch new file mode 100644 index 0000000..ac11b7a --- /dev/null +++ b/patches/metal2vulkan/0012-surface_lazy-register-a-surface-that-leaves-writes-a.patch @@ -0,0 +1,339 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:11:40 +0000 +Subject: [PATCH 12/34] surface_lazy: register a surface that leaves writes + ahead; a second texture over the pages keeps it from doing so + +A surface registered only as it left a standing, so writes left ahead by one +that had none could not be found by any reader. Two textures over one surface +learn of each other's writes from the copy back: neither leaves any ahead, and +a copy back recorded in an unordered encoder now reaches note_writes under the +new switches, where surface_lazy alone left it to the next resync. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/metal2vulkan-runtime/src/encoder.rs | 8 ++ + crates/metal2vulkan-runtime/src/shadow.rs | 11 ++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 96 ++++++++++++++----- + .../src/runtime_lane/surface_present.rs | 46 +++++++++ + 4 files changed, 133 insertions(+), 28 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index fd64a5b6..10dc6b45 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -1678,6 +1678,10 @@ impl EncoderCore { + }; + record(self.device().shared().vk(), raw); + self.uses.record(stages, tracked, None); ++ // LOCAL (Atlas): this way past `track`, as a copy back of another texture over the bytes. ++ if writes_truth { ++ crate::shadow_lazy::note_copy_back(tracked); ++ } + self.touch(uses); + } + +@@ -1710,6 +1714,10 @@ impl EncoderCore { + let aliased = self.aliased(&uses); + let tracked = aliased.as_deref().unwrap_or(&uses); + let before = run.before(tracked, copy.writes_truth()); ++ // LOCAL (Atlas): as in `shadow_command`. ++ if copy.writes_truth() { ++ crate::shadow_lazy::note_copy_back(tracked); ++ } + // SAFETY: the held command buffer is recording outside a render pass instance (the + // caller ended the one before), and only this encoder records it; its commands retain + // the shadowed textures (and their views) and the rows' set until it completes, and +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index 2853f759..2c2917fd 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2181,7 +2181,9 @@ impl ShadowSync { + return; + } + let writes = |carried: &Carried| { +- (uses.iter()).any(|used| !used.write.is_empty() && carried.copies.meets(used)) ++ // LOCAL (Atlas): its own copy back recorded late leaves the image as it was. ++ let own = (carried.copies.lazy.as_ref()).is_some_and(|lazy| lazy.own_thread()); ++ !own && (uses.iter()).any(|used| !used.write.is_empty() && carried.copies.meets(used)) + }; + self.carry.entries.retain(|carried| !writes(carried)); + } +@@ -2784,11 +2786,14 @@ impl ShadowSync { + let entry = &self.entries[i]; + let alone = (self.entries.iter().enumerate()) + .all(|(j, other)| j == i || !other.copies.shares_truth(&entry.copies)); ++ let (key, range) = entry.copies.truth(); ++ let truth = (key, entry.copies.alias(), range); + let left = alone + && entry.merge.is_none() + && !entry.two_ways +- && (entry.copies.lazy.as_ref()) +- .is_some_and(|lazy| lazy.defer(&entry.copies, entry.area, self.carry.id)); ++ && (entry.copies.lazy.as_ref()).is_some_and(|lazy| { ++ lazy.defer(&entry.copies, truth, entry.area, self.carry.id) ++ }); + match left { + true => { + let entry = &mut self.entries[i]; +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 013f6559..5aefa2df 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -26,6 +26,10 @@ + //! pages as they read them shadowed or in place, at every command buffer's completion, and a + //! later command's read through another import is ordered after the copy by the tracker. + //! ++//! **What it does not see either**, this switch alone: another texture's copy back in an unordered ++//! encoder, until the next resync. `surface_lazy_scoped` and `surface_lazy_present` see it ++//! ([`note_copy_back`]). ++//! + //! **What it cannot see**: a write of the guest's CPU into the pages between command buffers, which + //! reaches the host through no call. Metal shows it to a later command buffer. Here the image misses + //! it until the next copy in: at least every `surface_lazy_resync_ms` (default 250, 0 never) a +@@ -65,9 +69,14 @@ + //! Each moves only what committed command buffers wrote: a write stays out of every copy back + //! until its commit returns, as a copy back recorded before it runs before it. + //! ++//! **Two textures' images over one surface** each learn of the other's writes from its copy back, ++//! so neither leaves any ahead once both are registered: the counts' `shared`. ++//! + //! **Who can read stale bytes**, each for at most that bound and only of a surface left ahead: the + //! guest's CPU through its own mapping of the pages, which reaches the host through no call, and a +-//! host reader of the pages that goes through no command buffer. Under the switch a CPU write into ++//! host reader of the pages that goes through no command buffer, and a shader that reaches the ++//! pages through addresses alone, with no use of them named, as an untracked resource may. Under ++//! the switch a CPU write into + //! the pages that lies inside the rectangle is overwritten by the copy back, where + //! `surface_lazy` alone keeps it unless a command buffer runs meanwhile. + //! +@@ -121,6 +130,8 @@ static IN_BY: [AtomicU64; 4] = [const { AtomicU64::new(0) }; 4]; + /// How many surfaces hold writes ahead in their images: every hook returns at once while none does. + static AHEAD: AtomicUsize = AtomicUsize::new(0); + static DEFERRED: AtomicU64 = AtomicU64::new(0); ++/// The copies back not left ahead, a second texture's image lying over the same pages. ++static SHARED: AtomicU64 = AtomicU64::new(0); + static FLUSHES: AtomicU64 = AtomicU64::new(0); + static FLUSHED_WHOLE: AtomicU64 = AtomicU64::new(0); + static FLUSHED_TEXELS: AtomicU64 = AtomicU64::new(0); +@@ -301,6 +312,17 @@ pub(super) fn note_writes(uses: &[Use]) { + } + } + ++/// Note a copy back recorded past [`note_writes`]: a command of an unordered encoder. A standing of ++/// another texture's image over the bytes must drop with it, which `surface_lazy` alone leaves to ++/// the next resync: the new switches close that. ++pub(crate) fn note_copy_back(uses: &[Use]) { ++ if ANY.load(Ordering::Relaxed) ++ && (crate::live::on(SCOPED, false) || crate::live::on(PRESENT, false)) ++ { ++ note_writes(uses); ++ } ++} ++ + /// Whether any surface holds writes ahead in its image, one atomic read. + pub(crate) fn ahead() -> bool { + AHEAD.load(Ordering::Relaxed) > 0 +@@ -329,13 +351,12 @@ pub(crate) fn after_commit(id: u64) { + return; + } + for lazy in surfaces() { +- if let Some(left) = lazy.pending.lock().unwrap().as_mut() { +- left.committed(id); +- } ++ let moved = (lazy.pending.lock().unwrap().as_mut()).is_some_and(|left| left.committed(id)); + let wants = + lazy.flush_after + .compare_exchange(id, 0, Ordering::Relaxed, Ordering::Relaxed); +- if wants.is_ok() { ++ // A texture that came beside it while this command buffer was open reads the copy back. ++ if wants.is_ok() || (moved && lazy.beside_another()) { + lazy.flush(); + } + } +@@ -431,8 +452,8 @@ fn report() { + eprintln!( + "surface_lazy: taken {taken} refused {refused} resyncs {resyncs} | refused: touched {} \ + written {} epoch {} own {} uncommitted {} no-standing {} | in: first {} beyond-held {} \ +- changed {} bounded {} | ahead: deferred {} flushes {} whole {} texels {} discarded {} \ +- presents {} unmatched {}", ++ changed {} bounded {} | ahead: deferred {} shared {} flushes {} whole {} texels {} \ ++ discarded {} presents {} unmatched {}", + by[Refused::Touched as usize], + by[Refused::Written as usize], + by[Refused::Epoch as usize], +@@ -444,6 +465,7 @@ fn report() { + copied[2], + copied[3], + read(&DEFERRED), ++ read(&SHARED), + read(&FLUSHES), + read(&FLUSHED_WHOLE), + read(&FLUSHED_TEXELS), +@@ -511,15 +533,17 @@ struct Pending { + } + + impl Pending { +- /// Note that the command buffer of identity `id` committed. +- fn committed(&mut self, id: u64) { +- if let Some(at) = self.open.iter().position(|(open, _)| *open == id) { +- let (_, area) = self.open.swap_remove(at); +- self.settled = Some(match self.settled { +- Some(settled) => bounding(settled, area), +- None => area, +- }); +- } ++ /// Note that the command buffer of identity `id` committed: whether it wrote here. ++ fn committed(&mut self, id: u64) -> bool { ++ let Some(at) = self.open.iter().position(|(open, _)| *open == id) else { ++ return false; ++ }; ++ let (_, area) = self.open.swap_remove(at); ++ self.settled = Some(match self.settled { ++ Some(settled) => bounding(settled, area), ++ None => area, ++ }); ++ true + } + } + +@@ -641,7 +665,7 @@ impl Lazy { + } + + /// Whether this thread records this surface's own copy back. +- fn own_thread(&self) -> bool { ++ pub(super) fn own_thread(&self) -> bool { + OWN.with(Cell::get) == std::ptr::from_ref(self) + } + +@@ -671,8 +695,9 @@ impl Lazy { + /// Leave what an encoder of the command buffer of identity `id` wrote inside `area`, `None` for + /// anywhere, ahead in the image: whether it does, else its copy back goes as ever. + pub(super) fn defer( +- &self, ++ self: &Arc, + copies: &Arc, ++ truth: (ResourceKey, Option, UseRange), + area: Option<[u32; 4]>, + id: u64, + ) -> bool { +@@ -681,6 +706,13 @@ impl Lazy { + { + return false; + } ++ // What it leaves ahead only hooks that find it can bring back, standing or not. ++ self.register(truth); ++ // Another texture's image over these pages learns of a write through the copy back. ++ if self.beside_another() { ++ SHARED.fetch_add(1, Ordering::Relaxed); ++ return false; ++ } + let Some(holds) = self.keeps.get() else { + return false; + }; +@@ -722,6 +754,15 @@ impl Lazy { + true + } + ++ /// Whether another registered texture's image lies over its pages. ++ fn beside_another(self: &Arc) -> bool { ++ let beside = |other: &Arc| { ++ !Arc::ptr_eq(other, self) ++ && (self.pages.as_ref()).is_some_and(|pages| other.over(pages)) ++ }; ++ surfaces().iter().any(beside) ++ } ++ + /// `pending` was emptied. + fn left_behind(&self) { + self.ahead.store(false, Ordering::Relaxed); +@@ -801,6 +842,17 @@ impl Lazy { + } + } + ++ /// Make it one of the surfaces every hook looks through, its truth `truth`. ++ fn register(self: &Arc, truth: (ResourceKey, Option, UseRange)) { ++ if self.truth.set(truth).is_ok() { ++ let mut all = registry().write().unwrap(); ++ all.retain(|lazy| lazy.strong_count() > 0); ++ all.push(Arc::downgrade(self)); ++ ACTIVE.store(all.len(), Ordering::Relaxed); ++ ANY.store(true, Ordering::Relaxed); ++ } ++ } ++ + /// Leave the image's standing: `id`'s encoder ended with the image holding the truth, inside + /// `held` where given, stale inside `stale`. Its own copies back are behind it. + pub(super) fn stand( +@@ -811,13 +863,7 @@ impl Lazy { + held: Option<[u32; 4]>, + stale: Option<[u32; 4]>, + ) { +- if self.truth.set((key, alias, range)).is_ok() { +- let mut all = registry().write().unwrap(); +- all.retain(|lazy| lazy.strong_count() > 0); +- all.push(Arc::downgrade(self)); +- ACTIVE.store(all.len(), Ordering::Relaxed); +- ANY.store(true, Ordering::Relaxed); +- } ++ self.register((key, alias, range)); + *self.stood.lock().unwrap() = Some(Stood { + id, + touched, +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs +index bf273946..1a1e3d8a 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_present.rs +@@ -216,6 +216,52 @@ fn surface_present_passes_of_one_command_buffer_end_in_the_pages() { + }); + } + ++#[test] ++fn surface_present_two_textures_of_a_surface_see_each_others_writes() { ++ switches(true, true, false, 60_000); ++ on_surface(|case| { ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = case.buffer.resource_options(); ++ let other = case ++ .buffer ++ .new_texture_with_iosurface_plane(&d, IoSurfacePlane::new(0, ROW, false)) ++ .unwrap(); ++ let left = ScissorRect { x: 0, y: 0, width: WIDTH / 2, height: HEIGHT }; ++ let right = ScissorRect { x: WIDTH / 2, y: 0, width: WIDTH / 2, height: HEIGHT }; ++ let red = staged(case.device, &floats4(RED)); ++ let green = staged(case.device, &floats4(GREEN)); ++ let rects = std::slice::from_ref(&case.rect); ++ // Across command buffers, then inside one, each texture over what the other wrote. ++ for apart in [true, false] { ++ let mut command_buffer = case.queue.command_buffer().unwrap(); ++ for (texture, color, damage) in [ ++ (&case.texture, &red, left), ++ (&other, &green, right), ++ (&case.texture, &green, left), ++ (&other, &red, right), ++ ] { ++ pass(&command_buffer, texture, &case.opaque, rects, color, Some(damage)); ++ if apart { ++ run(&command_buffer); ++ command_buffer = case.queue.command_buffer().unwrap(); ++ } ++ } ++ run(&command_buffer); ++ case.texture.surface_presented(); ++ other.surface_presented(); ++ case.pages_become(5, 5, GREEN_TEXEL); ++ case.pages_become(WIDTH - 5, 5, RED_TEXEL); ++ // The next round's writes start from these. ++ case.draw(RED, Some(left)); ++ case.draw(GREEN, Some(right)); ++ let read = gpu_read(&case.queue, &case.other_import(), 0..ROW); ++ assert_eq!(read[..4], RED_TEXEL); ++ assert_eq!(read[read.len() - 4..], GREEN_TEXEL); ++ } ++ }); ++} ++ + #[test] + fn surface_present_the_clock_copies_back_after_the_guest_goes_idle() { + switches(true, true, false, 20); diff --git a/patches/metal2vulkan/0013-trap-coherent-surfaces-behind-surface_co.patch b/patches/metal2vulkan/0013-trap-coherent-surfaces-behind-surface_co.patch new file mode 100644 index 0000000..5f61b6b --- /dev/null +++ b/patches/metal2vulkan/0013-trap-coherent-surfaces-behind-surface_co.patch @@ -0,0 +1,2992 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 01:11:17 +0000 +Subject: [PATCH 13/34] trap-coherent surfaces behind + surface_coherent + +A surface's copy back stays deferred with its pages poisoned through a KVM-only +view: the first guest touch syncs the pending rectangles on the CPU with +VK_EXT_host_image_copy, a write opens its page and the next GPU use copies it +in, a CPU write is kept by merging against a snapshot. Off by default. + +Fixes two things surface_lazy_present_all got wrong: a released texture's +completed writes were discarded while the surface stayed mapped, and CPU +readers saw stale bytes until the next flush. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/buffer_texture.rs | 22 + + .../src/command_buffer/mod.rs | 8 + + crates/metal2vulkan-runtime/src/lib.rs | 1 + + crates/metal2vulkan-runtime/src/memory.rs | 35 + + crates/metal2vulkan-runtime/src/shadow.rs | 50 +- + .../src/shadow_coherent.rs | 1076 +++++++++++++++++ + .../metal2vulkan-runtime/src/shadow_lazy.rs | 228 +++- + crates/metal2vulkan-runtime/src/texture.rs | 91 +- + .../src/texture/host_copy.rs | 124 +- + .../src/runtime_lane.rs | 2 + + .../src/runtime_lane/pitched_surfaces.rs | 33 + + .../src/runtime_lane/surface_coherent.rs | 812 +++++++++++++ + 12 files changed, 2453 insertions(+), 29 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/shadow_coherent.rs + create mode 100644 crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs + +diff --git a/crates/metal2vulkan-runtime/src/buffer_texture.rs b/crates/metal2vulkan-runtime/src/buffer_texture.rs +index d466dd62..7197fdc2 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_texture.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_texture.rs +@@ -1440,6 +1440,28 @@ impl Texels { + }) + } + ++ /// LOCAL (Atlas): whether a shadow of these texels is lazy (`shadow_lazy.rs`). ++ pub(super) fn lies_lazily(&self) -> bool { ++ self.backing() == super::shadow::Backing::Surface ++ && super::shadow_lazy::takes_surface( ++ (self.layout.row_bytes / self.texel_bytes) as u32, ++ self.layout.row_bytes / self.texel_bytes * u64::from(self.layout.rows), ++ ) ++ } ++ ++ /// LOCAL (Atlas): where the texels of a one-texel-a-block image of `extent` lie in host pages, ++ /// for a trapping surface (`shadow_coherent.rs`). ++ pub(super) fn surface_layout(&self, extent: [u32; 2]) -> Option { ++ let bytes = self.buffer.host_range()?; ++ Some(super::shadow_coherent::Layout { ++ base: bytes.start as u64 + self.layout.offset, ++ row: self.layout.row_pitch, ++ texel: self.texel_bytes, ++ width: extent[0], ++ height: extent[1], ++ }) ++ } ++ + /// The staging buffer of a shadowed texture or a texture buffer over the texels laid out as + /// `layout`, which start at `at` of the buffer's `VkBuffer`: the texels' bytes from the first + /// to the last, at its start, of the texture's hazard tracking, in a buffer of whole 4-byte +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 97d01cf3..b2a5ab9e 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -4332,6 +4332,14 @@ impl CommandBufferShared { + let submitted_holds; + // No rebind encodes it again from here: a group that was never submitted runs nothing. + self.device.shared().rebinds.no_replays(self); ++ // LOCAL (Atlas): its writes in surfaces' images trap before anything can tell the guest it ++ // completed (`shadow_coherent.rs`). ++ if crate::shadow_lazy::ahead() { ++ let id = lock(&self.state).shadow_carry.as_ref().map(ShadowCarry::id); ++ if let Some(id) = id { ++ crate::shadow_lazy::completing(id); ++ } ++ } + { + let mut state = lock(&self.state); + // Every trail of the work it waited for (a pipeline's making, a twin's decode) was +diff --git a/crates/metal2vulkan-runtime/src/lib.rs b/crates/metal2vulkan-runtime/src/lib.rs +index 48b3a27d..546bf564 100644 +--- a/crates/metal2vulkan-runtime/src/lib.rs ++++ b/crates/metal2vulkan-runtime/src/lib.rs +@@ -203,6 +203,7 @@ mod shader_cache; + mod shadow; + mod shadow_copy_back; + mod shadow_copy_in; ++pub mod shadow_coherent; // LOCAL (Atlas) + pub mod shadow_lazy; // LOCAL (Atlas) + mod shadow_merge; + mod shadow_stencil; +diff --git a/crates/metal2vulkan-runtime/src/memory.rs b/crates/metal2vulkan-runtime/src/memory.rs +index 25c32549..e91fe33e 100644 +--- a/crates/metal2vulkan-runtime/src/memory.rs ++++ b/crates/metal2vulkan-runtime/src/memory.rs +@@ -202,6 +202,32 @@ pub(super) fn host_incoherent(types: &[vk::MemoryType]) -> u32 { + .fold(0, |bits, (index, _)| bits | 1 << index) + } + ++/// LOCAL (Atlas): [`host_cached_placement`] for an image the host only reads (a trapping surface's, ++/// `shadow_coherent.rs`), where a type the host maps without coherence serves too: nothing writes ++/// the image through the host, whose stale lines the [`host_cached`] filter is for, and the host ++/// copy invalidates its own reads. ++pub(super) fn read_cached_placement( ++ types: &[vk::MemoryType], ++ allowed: u32, ++ class: MemoryClass, ++ unified: bool, ++) -> Option<(MemoryClass, u32)> { ++ use vk::MemoryPropertyFlags as P; ++ let cached = P::HOST_VISIBLE | P::HOST_CACHED; ++ let bits = (types.iter().enumerate()) ++ .filter(|(_, ty)| ty.property_flags.contains(cached)) ++ .fold(0, |bits, (index, _)| bits | 1 << index) ++ & allowed; ++ let has = |class: MemoryClass| !candidates(types, bits, class, unified).is_empty(); ++ if has(class) { ++ Some((class, bits)) ++ } else if unified && has(MemoryClass::Shared) { ++ Some((MemoryClass::Shared, bits)) ++ } else { ++ None ++ } ++} ++ + /// The memory class an image of `class` with host-transfer usage takes in memory the host caches, + /// and its memory types among `allowed`, as bits: the host-cached types ([`host_cached`]) of + /// `class` where it has one among them; else, on a device with unified memory (`unified`), those +@@ -1153,6 +1179,15 @@ impl Memory { + host_cached_placement(&self.types, allowed, class, self.unified) + } + ++ /// LOCAL (Atlas): [`read_cached_placement`] of this device's types. ++ pub(super) fn read_cached_placement( ++ &self, ++ allowed: u32, ++ class: MemoryClass, ++ ) -> Option<(MemoryClass, u32)> { ++ read_cached_placement(&self.types, allowed, class, self.unified) ++ } ++ + /// The memory types the host maps without coherence, as bits ([`host_incoherent`]). + pub(super) fn host_incoherent(&self) -> u32 { + host_incoherent(&self.types) +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index 2c2917fd..b84969b5 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -766,7 +766,7 @@ impl ShadowCopies { + && other_writer + && !self.drops_writes + && self.unmerged().is_none() +- && self.lazy.is_none() ++ && self.lazy.as_ref().is_none_or(|lazy| lazy.wants_merge()) + } + + /// Whether two shadowed textures copy some of the same truth. +@@ -1962,6 +1962,9 @@ struct Entry { + /// first use's copy in is the one the carry takes off the run + /// (`WorkCounts::shadow_texels_first_copy_in_carried`). + carried: bool, ++ /// LOCAL (Atlas): it took a trapping surface's standing and neither copied in nor wrote so far ++ /// (`shadow_coherent.rs`). ++ only_read: bool, + /// [`ShadowCopies::touched`] when the encoder met the shadow: its end carries the image to + /// the command buffer's later encoders only where it still stands there, no other encoder + /// having ended with the shadow in between ([`ShadowCarry`]). +@@ -2197,9 +2200,15 @@ impl ShadowSync { + let mut carry = ShadowCarry { + id: self.carry.id, + entries: Vec::new(), ++ left: self.carry.left, + }; + carry.entries.append(&mut self.carry.entries); + for entry in &self.entries { ++ // LOCAL (Atlas): the display sampled a standing: the image is as it was, and a writer ++ // of a later command buffer still takes it (`shadow_coherent.rs`). ++ if entry.only_read { ++ continue; ++ } + let touched = entry.copies.touched.fetch_add(1, Ordering::Relaxed); + // LOCAL (Atlas): what a later command buffer takes (`shadow_lazy.rs`). + if let Some(lazy) = &entry.copies.lazy { +@@ -2263,6 +2272,9 @@ pub(super) struct ShadowCarry { + /// buffer by its pointer, which another may take once it is freed. + id: u64, + entries: Vec, ++ /// LOCAL (Atlas): an encoder of it left writes ahead in a trapping surface's image, which a ++ /// later command that meets the bytes copies back (`shadow_lazy::met_by_command`). ++ left: bool, + } + + impl Default for ShadowCarry { +@@ -2272,6 +2284,7 @@ impl Default for ShadowCarry { + ShadowCarry { + id: NEXT.fetch_add(1, Ordering::Relaxed), + entries: Vec::new(), ++ left: false, + } + } + } +@@ -2359,6 +2372,8 @@ impl ShadowSync { + (None, Some(lazy)) => lazy.take(self.carry.id, met), + _ => None, + }; ++ let only_read = lazily.is_some() && (copies.lazy.as_ref()) ++ .is_some_and(|lazy| lazy.whole_image()); + let (standing, held, stale) = match (&carried, lazily) { + (Some(carried), _) => (Standing::Current, carried.held, carried.stale), + (None, Some((held, stale))) => (Standing::Current, held, stale), +@@ -2376,6 +2391,7 @@ impl ShadowSync { + stale, + met, + carried: carried.is_some(), ++ only_read, + }); + self.entries.len() - 1 + } +@@ -2510,6 +2526,15 @@ impl ShadowSync { + entry.scratch = Some(scratch); + } + let mut copies = Vec::new(); ++ // LOCAL (Atlas): what an earlier encoder left ahead in an image, where this command meets ++ // the bytes (`shadow_lazy.rs`). ++ if self.carry.left { ++ copies = super::shadow_lazy::met_by_command(self.carry.id, uses, |theirs| { ++ (shadows.iter()).any(|shadow| { ++ shadow.copies.shadow_key != theirs.shadow_key && shadow.copies.shares_truth(theirs) ++ }) ++ }); ++ } + // Merge what the command before wrote two ways; copy back what this one reads or writes + // another way: through the buffer's bytes, or through another texture over them that is + // copied in next. +@@ -2547,7 +2572,10 @@ impl ShadowSync { + // a rectangle; every texel where it writes through a shadow whose copies back move + // every texel, or writes the bytes two ways, as the merge after it compares every + // texel the keep took. +- let needs = match two_ways || !shadow.copies.cuts() { ++ // LOCAL (Atlas): a trapping surface's image holds every texel: its records' rectangles ++ // bound what may lie apart (`shadow_coherent.rs`). ++ let whole = (shadow.copies.lazy.as_ref()).is_some_and(|lazy| lazy.whole_image()); ++ let needs = match two_ways || whole || !shadow.copies.cuts() { + true => None, + false if shadow.writes && !shadow.copies.cuts_back() => None, + false if shadow.writes => bounding(shadow.loads, shadow.area), +@@ -2582,18 +2610,20 @@ impl ShadowSync { + let took = carried && cause.is_none() && !overwrites; + if let Some(cause) = cause { + if !overwrites { +- copies.push(ShadowCopy { ++ let copy = ShadowCopy { + copies: Arc::clone(&shadow.copies), + step: Step::Into, + kept: shadow.kept, + area: needs, + cause: Some(cause), + moves: None, +- }); ++ }; + // LOCAL (Atlas): counted by cause (`shadow_lazy.rs`). + if let Some(lazy) = &shadow.copies.lazy { +- lazy.copied_in(cause); ++ lazy.copied_in(cause, copy.texels_copied_in()); + } ++ copies.push(copy); ++ entry.only_read = false; + } + entry.standing = Standing::Current; + entry.held = needs; +@@ -2615,18 +2645,20 @@ impl ShadowSync { + } + }; + if let (false, Some(area)) = (overwrites, area) { +- copies.push(ShadowCopy { ++ let copy = ShadowCopy { + copies: Arc::clone(&shadow.copies), + step: Step::Into, + kept: shadow.kept, + area: Some(area), + cause: Some(CopyCause::Bounded), + moves: None, +- }); ++ }; + // LOCAL (Atlas): as above. + if let Some(lazy) = &shadow.copies.lazy { +- lazy.copied_in(CopyCause::Bounded); ++ lazy.copied_in(CopyCause::Bounded, copy.texels_copied_in()); + } ++ copies.push(copy); ++ entry.only_read = false; + } + } + // The command writes every texel before it reads any. +@@ -2729,6 +2761,7 @@ impl ShadowSync { + let i = self.entry(&shadow.copies); + // Writes Metal drops are not copied back, and the image no longer holds the bytes. + let entry = &mut self.entries[i]; ++ entry.only_read = false; + match shadow.copies.drops_writes { + true => entry.lose(Lost::Dropped), + false => { +@@ -2796,6 +2829,7 @@ impl ShadowSync { + }); + match left { + true => { ++ self.carry.left = true; + let entry = &mut self.entries[i]; + entry.dirty = false; + entry.area = None; +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +new file mode 100644 +index 00000000..22172ac7 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -0,0 +1,1076 @@ ++//! LOCAL (Atlas): a lazy surface (`shadow_lazy.rs`) whose pages trap, under the live switch ++//! `surface_coherent` (default off) and with a [`Pages`] installed: QEMU's census view ++//! (`accel/kvm/census.c` in Reims' QEMU), or a lane's. Installed by nothing and switched off, ++//! nothing here runs. ++//! ++//! A surface's writes stay in its optimal image, where nothing copies them, until something ++//! touches their pages: the display samples the image itself, so no present copies either. Each ++//! page of a tracked surface is: ++//! ++//! - **held**: write-protected, the page and the image agree; reads pass, a write traps. ++//! - **armed**: poisoned, the image is ahead; any access of the guest, in any mode, traps first. ++//! - **open**: the guest's CPU wrote it, and the image is stale there ([`State::guest`]). ++//! - **frozen**: an open page poisoned, while a sync merges into it. ++//! ++//! **A write's record** goes open (recording), running (committed), done (completed). ++//! [`completing`] runs before the command buffer's completed handlers, so a page is armed before ++//! the guest can learn the work ended. A read before that may read the older bytes, as Metal lets ++//! it: nothing promises a command's writes to the CPU before its completion. ++//! ++//! **A sync** ([`Coherent::sync_done`]) copies every done rectangle at once on the host ++//! (`VK_EXT_host_image_copy`), so the first trap in a surface brings all of it, then holds the ++//! pages. It queues no GPU work and waits for none, and its lock is never held across a GPU wait. ++//! A running record stays out of it: its pages stay as they are until it completes and arms them. ++//! So a sync moves only completed work, and may tear with a write still in flight over the same ++//! texels (the image's bytes are then partly new): that is a read racing a running command, which ++//! Metal leaves undefined too, and the running record, still pending, syncs those texels again ++//! once it completes. Completed work is never lost: a batch's writes are visible to the host at ++//! its completion (`tracker.rs`), and the record is dropped only after its copy. ++//! ++//! **A CPU write** never loses to a sync. The write stops before it lands, so the page's snapshot ++//! is what the guest overwrote: a later sync over the open page keeps every texel that differs ++//! from the snapshot ([`Coherent::merge`]). A GPU copy back over an open page cannot merge, so ++//! [`Coherent::before_flush`] waits for the records, a bounded time, and syncs on the CPU; past ++//! the bound it counts `overwrite_risk`. Writing also demotes the surface to the shadow's own keep ++//! and merge. ++//! ++//! **The next command's copy in** takes the open pages' rectangle alone, and holds them first, so ++//! a later write opens them again ([`Coherent::taking`]). ++//! ++//! **Adaptive**: a surface synced in more than `surface_coherent_demote_pct` percent of a window of ++//! its frames, or more than `surface_coherent_demote_hz` times a second (a Simulator's per-frame ++//! reader), goes back to a copy back per frame for `surface_coherent_hold_s` seconds, doubling to ++//! [`HOLD_MAX_S`] while it keeps coming back. ++//! ++//! **Unmap** ([`unmapping`]) syncs, while the pages are still the guest's, and lets go of what ++//! pending writes hold; a surface with work running answers busy, as any surface with a command ++//! running over it does. Nothing is pooled across surfaces: each image is its texture's. ++//! ++//! The report ([`report`]) gives per surface what detection, sync and fetch cost per present, ++//! beside what the copies they replace would have. ++ ++use super::shadow::bounding; ++use super::shadow_lazy::Lazy; ++use std::collections::{BTreeMap, HashSet}; ++use std::ops::Range; ++use std::sync::atomic::{AtomicU64, Ordering}; ++use std::sync::{Arc, Condvar, Mutex, MutexGuard, OnceLock, RwLock, Weak}; ++use std::time::{Duration, Instant}; ++ ++/// The live switch. ++const SWITCH: &str = "surface_coherent"; ++/// The display samples a surface's image instead of its pages. ++const DISPLAY: &str = "surface_coherent_display"; ++pub(crate) const PAGE: u64 = 4096; ++const HOLD_MAX_S: u64 = 30; ++/// Records apart at most: past it the done ones join, as the image holds every texel between. ++const RECORDS: usize = 8; ++const PROBE_GIVEN_UP: Duration = Duration::from_millis(1); ++const DEFAULT_WAIT_MS: u64 = 50; ++/// The closed surfaces' lines kept between reports. ++const ENDED_KEPT: usize = 256; ++/// A GPU copy of a surface, per MiB: 2.97 ms for 5.76 Mtexels of 4 bytes (`STUDY-virt.md`). ++const DEFAULT_COPY_NS_PER_MIB: u64 = 135_000; ++ ++/// What marks a surface's pages. Addresses are the surface's own mapping's, whole pages, ascending, ++/// one surface to a call; each call comes under the surface's lock, which [`fault`] takes. ++pub trait Pages: Send + Sync { ++ /// Every page held from now on: whether it can be, this surface. ++ fn track(&self, start: u64, len: u64) -> bool; ++ /// Every marker off the surface around `start`. ++ fn untrack(&self, start: u64); ++ /// Held pages armed: the next access traps. Others stay. The pages a CPU reached as their ++ /// markers moved come back, still held, and they may hold a write: the caller has them. ++ fn arm(&self, pages: &[u64]) -> Vec; ++ /// Armed pages, and open ones with `open_too`, held; the pages a CPU may have reached as the ++ /// markers moved come back. ++ fn hold(&self, pages: &[u64], open_too: bool) -> Vec; ++ /// A held page opened to the guest's write. ++ fn open(&self, page: u64); ++ /// Open pages poisoned, while a sync merges into them. ++ fn freeze(&self, pages: &[u64]); ++ /// Frozen pages open again. ++ fn thaw(&self, pages: &[u64]); ++} ++ ++static PAGES: RwLock>> = RwLock::new(None); ++ ++/// The last words of the surfaces that went since the last report. ++fn ended() -> &'static Mutex> { ++ static ENDED: OnceLock>> = OnceLock::new(); ++ ENDED.get_or_init(Mutex::default) ++} ++ ++/// Every tracked surface, by which a fault finds its own. ++fn all() -> &'static RwLock>> { ++ static ALL: OnceLock>>> = OnceLock::new(); ++ ALL.get_or_init(RwLock::default) ++} ++ ++/// Install what marks the pages (one call at start, a lane's `None` after). ++pub fn set_pages(pages: Option>) { ++ *PAGES.write().unwrap() = pages; ++} ++ ++fn pages() -> Option> { ++ PAGES.read().unwrap().clone() ++} ++ ++/// Whether a surface made now traps. ++pub(crate) fn enabled() -> bool { ++ crate::live::on(SWITCH, false) && PAGES.read().unwrap().is_some() ++} ++ ++fn knob(name: &str, default: u64) -> u64 { ++ crate::live::get(name).and_then(|v| v.parse().ok()).unwrap_or(default) ++} ++ ++/// Copies an image's rectangle `[x0, y0, x1, y1)` to bytes laid out in rows of the given texels, ++/// on the host: whether it did. ++pub(crate) type Reader = Box bool + Send + Sync>; ++ ++/// A surface's texels in host pages. ++#[derive(Clone, Copy, Debug)] ++pub(crate) struct Layout { ++ /// The host address of the first texel's byte. ++ pub(crate) base: u64, ++ pub(crate) row: u64, ++ pub(crate) texel: u64, ++ pub(crate) width: u32, ++ pub(crate) height: u32, ++} ++ ++impl Layout { ++ fn whole(&self) -> [u32; 4] { ++ [0, 0, self.width, self.height] ++ } ++ ++ fn first_page(&self) -> u64 { ++ self.base & !(PAGE - 1) ++ } ++ ++ fn end(&self) -> u64 { ++ (self.base + self.row * u64::from(self.height)).next_multiple_of(PAGE) ++ } ++ ++ /// The pages `area`'s texels lie in, ascending. ++ fn pages_of(&self, area: [u32; 4]) -> Vec { ++ let [x0, y0, x1, y1] = area; ++ let mut found = Vec::new(); ++ if x0 >= x1 || y0 >= y1 { ++ return found; ++ } ++ let (x0, x1) = (u64::from(x0) * self.texel, u64::from(x1) * self.texel); ++ let mut last = None; ++ for y in u64::from(y0)..u64::from(y1) { ++ let row = self.base + y * self.row; ++ let (first, end) = ((row + x0) & !(PAGE - 1), row + x1); ++ let mut page = first; ++ while page < end { ++ if last.is_none_or(|last| page > last) { ++ found.push(page); ++ last = Some(page); ++ } ++ page += PAGE; ++ } ++ } ++ found ++ } ++ ++ /// The rectangle of every texel of the row-spans `page` holds any byte of. ++ fn area_of(&self, page: u64) -> Option<[u32; 4]> { ++ let start = page.max(self.base) - self.base; ++ let end = (page + PAGE).checked_sub(self.base)?.min(self.row * u64::from(self.height)); ++ (start < end).then(|| { ++ let (y0, y1) = (start / self.row, (end - 1) / self.row + 1); ++ let (x0, x1) = match y1 - y0 { ++ 1 => ( ++ (start % self.row / self.texel).min(u64::from(self.width)), ++ ((end - 1) % self.row / self.texel + 1).min(u64::from(self.width)), ++ ), ++ _ => (0, u64::from(self.width)), ++ }; ++ [x0 as u32, y0 as u32, x1 as u32, y1 as u32] ++ }) ++ } ++ ++ fn at(&self, x: u32, y: u32) -> u64 { ++ self.base + u64::from(y) * self.row + u64::from(x) * self.texel ++ } ++} ++ ++fn meet(a: [u32; 4], b: [u32; 4]) -> bool { ++ a[0] < b[2] && b[0] < a[2] && a[1] < b[3] && b[1] < a[3] ++} ++ ++fn holds(outer: [u32; 4], inner: [u32; 4]) -> bool { ++ outer[0] <= inner[0] && outer[1] <= inner[1] && inner[2] <= outer[2] && inner[3] <= outer[3] ++} ++ ++fn texels(area: [u32; 4]) -> u64 { ++ u64::from(area[2] - area[0]) * u64::from(area[3] - area[1]) ++} ++ ++/// One write left in the image. ++#[derive(Clone, Copy, Debug)] ++struct Record { ++ /// The command buffer's identity (`shadow::ShadowCarry::id`), 0 once several joined. ++ id: u64, ++ /// Its place among records committed: what a GPU copy back moved it by. ++ seq: u64, ++ area: [u32; 4], ++} ++ ++/// One surface's counts. ++#[derive(Default)] ++struct Counts { ++ presents: AtomicU64, ++ defers: AtomicU64, ++ /// Bytes the encoders left ahead: what a copy back per encoder would have moved. ++ deferred_bytes: AtomicU64, ++ traps: AtomicU64, ++ writes: AtomicU64, ++ unique_pages: AtomicU64, ++ regions: AtomicU64, ++ bytes_detiled: AtomicU64, ++ sync_ns: AtomicU64, ++ trap_ns: AtomicU64, ++ trap_ns_max: AtomicU64, ++ timed: AtomicU64, ++ arm_ns: AtomicU64, ++ arms: AtomicU64, ++ hold_ns: AtomicU64, ++ demotions: AtomicU64, ++ eager: AtomicU64, ++ merges: AtomicU64, ++ came: AtomicU64, ++ flushes: AtomicU64, ++ flushed_bytes: AtomicU64, ++ copy_ins: AtomicU64, ++ copy_in_bytes: AtomicU64, ++ probes: AtomicU64, ++ overwrite_risk: AtomicU64, ++ sync_errors: AtomicU64, ++} ++ ++#[derive(Default)] ++struct State { ++ /// The surface's pages are marked. ++ tracked: bool, ++ /// Not trapping any more: the switch went off, a fuse, a reset, an unmap. ++ closed: bool, ++ open: Vec, ++ running: Vec, ++ done: Vec, ++ /// Records of a sequence up to this went back by a GPU copy that completed. ++ flushed_to: u64, ++ /// A GPU copy back in flight, up to this sequence: 0 for none. ++ flushing: u64, ++ seq: u64, ++ /// Rectangles whose pages are armed: nothing to do for what one holds. ++ armed: Vec<[u32; 4]>, ++ /// A probe armed every held page: the next fault holds them again. ++ /// Probes whose write has not named its page yet, and when the last began. A probe of another ++ /// surface's write leaves this one's count up, and `PROBE_GIVEN_UP` ends it. ++ probing: u32, ++ probed_at: Option, ++ /// Where the guest's CPU wrote, which the image misses. ++ guest: Option<[u32; 4]>, ++ /// Each open page, as it was before the guest's write. ++ snapshots: BTreeMap>, ++ frozen: Vec, ++ /// When the last write was: the surface keeps and merges for the hold after it. ++ wrote_at: Option, ++ /// Copies back per frame until then. ++ eager_until: Option, ++ hold_s: u64, ++ window_start: Option, ++ window_defers: u64, ++ window_syncs: u64, ++ seen: HashSet, ++} ++ ++/// A surface's trap coherence, beside its [`Lazy`]. ++pub(crate) struct Coherent { ++ state: Mutex, ++ /// Tells [`Coherent::wait_settled`] a record moved. ++ moved: Condvar, ++ layout: Layout, ++ read: Reader, ++ lazy: Weak, ++ counts: Counts, ++ made: Instant, ++} ++ ++impl Coherent { ++ pub(crate) fn new(layout: Layout, read: Reader, lazy: Weak) -> Arc { ++ Arc::new(Coherent { ++ state: Mutex::default(), ++ moved: Condvar::new(), ++ layout, ++ read, ++ lazy, ++ counts: Counts::default(), ++ made: Instant::now(), ++ }) ++ } ++ ++ fn lock(&self) -> MutexGuard<'_, State> { ++ self.state.lock().unwrap_or_else(|e| e.into_inner()) ++ } ++ ++ /// Whether it still answers for its pages. ++ pub(crate) fn is_live(&self) -> bool { ++ !self.lock().closed ++ } ++ ++ pub(crate) fn range(&self) -> Range { ++ self.layout.first_page()..self.layout.end() ++ } ++ ++ fn all_pages(&self) -> Vec { ++ (self.range().step_by(PAGE as usize)).collect() ++ } ++ ++ /// Whether the surface traps now, tracking it at its first use: its pages are held before any ++ /// copy in reads them, so nothing the guest writes after that goes unseen. ++ pub(crate) fn tracks(self: &Arc) -> bool { ++ let mut state = self.lock(); ++ if state.closed { ++ return false; ++ } ++ if state.tracked { ++ return true; ++ } ++ let range = self.range(); ++ let live = crate::live::on(SWITCH, false); ++ let marked = pages().is_some_and(|p| live && p.track(range.start, range.end - range.start)); ++ if !marked { ++ state.closed = true; ++ return false; ++ } ++ state.tracked = true; ++ all().write().unwrap().push(self.clone()); ++ true ++ } ++ ++ /// Whether it writes back as a shadow keeping and merging: a CPU write came lately. ++ pub(crate) fn wants_merge(&self) -> bool { ++ let state = self.lock(); ++ state.tracked && !state.closed && state.wrote_at.is_some_and(|at| at.elapsed() < self.hold()) ++ } ++ ++ fn hold(&self) -> Duration { ++ Duration::from_secs(knob("surface_coherent_hold_s", 2)) ++ } ++ ++ /// Whether an encoder of `id` leaves `area` ahead; if so it is recorded. ++ pub(crate) fn defers(&self, id: u64, area: Option<[u32; 4]>) -> bool { ++ let mut state = self.lock(); ++ if state.closed || !state.tracked { ++ return false; ++ } ++ let now = Instant::now(); ++ state.window_defers += 1; ++ self.roll_window(&mut state, now); ++ match state.eager_until { ++ Some(until) if now < until => { ++ self.counts.eager.fetch_add(1, Ordering::Relaxed); ++ return false; ++ } ++ Some(_) => state.eager_until = None, ++ None => {} ++ } ++ if state.wrote_at.is_some_and(|at| at.elapsed() < self.hold()) { ++ return false; ++ } ++ let whole = self.layout.whole(); ++ let area = area.unwrap_or(whole); ++ match state.open.iter_mut().find(|record| record.id == id) { ++ Some(record) => { ++ let joined = bounding(Some(record.area), Some(area)); ++ record.area = joined.unwrap_or(whole); ++ } ++ None => state.open.push(Record { id, seq: 0, area }), ++ } ++ self.counts.defers.fetch_add(1, Ordering::Relaxed); ++ let bytes = texels(area) * self.layout.texel; ++ self.counts.deferred_bytes.fetch_add(bytes, Ordering::Relaxed); ++ true ++ } ++ ++ /// The window of frames closes: too many syncs in it send the surface back to copies per frame. ++ fn roll_window(&self, state: &mut State, now: Instant) { ++ let start = *state.window_start.get_or_insert(now); ++ let frames = knob("surface_coherent_demote_window", 32); ++ let elapsed = now.duration_since(start); ++ if state.window_defers < frames && elapsed < Duration::from_secs(1) { ++ return; ++ } ++ let pct = knob("surface_coherent_demote_pct", 50); ++ let hz = knob("surface_coherent_demote_hz", 64); ++ let by_frames = state.window_defers >= frames ++ && state.window_syncs * 100 >= state.window_defers * pct; ++ let by_rate = state.window_syncs as f64 >= hz as f64 * elapsed.as_secs_f64().max(0.25); ++ if by_frames || by_rate { ++ self.demote(state, now); ++ } ++ state.window_start = Some(now); ++ state.window_defers = 0; ++ state.window_syncs = 0; ++ state.seen.clear(); ++ } ++ ++ fn demote(&self, state: &mut State, now: Instant) { ++ let last = state.eager_until; ++ let first = knob("surface_coherent_hold_s", 2).max(1); ++ state.hold_s = match last { ++ Some(until) if now < until + Duration::from_secs(state.hold_s.max(first)) => { ++ (state.hold_s * 2).clamp(first, HOLD_MAX_S) ++ } ++ _ => first, ++ }; ++ state.eager_until = Some(now + Duration::from_secs(state.hold_s)); ++ self.counts.demotions.fetch_add(1, Ordering::Relaxed); ++ // What is pending goes now, and what is running at its completion. ++ self.sync_done(state); ++ } ++ ++ /// The commit of `id` returned: its record may go back by a copy ordered after it. ++ pub(crate) fn committed(&self, id: u64) { ++ let mut state = self.lock(); ++ if let Some(at) = state.open.iter().position(|record| record.id == id) { ++ let mut record = state.open.swap_remove(at); ++ state.seq += 1; ++ record.seq = state.seq; ++ state.running.push(record); ++ } ++ self.settle_pending(&state); ++ } ++ ++ /// The command buffer `id` is done, before anything tells the guest: its pages trap from here. ++ pub(crate) fn completing(&self, id: u64) { ++ let mut state = self.lock(); ++ let found = match state.running.iter().position(|record| record.id == id) { ++ Some(at) => Some(state.running.swap_remove(at)), ++ // Its commit has not returned yet. ++ None => (state.open.iter().position(|record| record.id == id)).map(|at| { ++ let mut record = state.open.swap_remove(at); ++ state.seq += 1; ++ record.seq = state.seq; ++ record ++ }), ++ }; ++ if let Some(record) = found { ++ if record.seq > state.flushed_to { ++ self.finish_record(&mut state, record); ++ } ++ self.moved.notify_all(); ++ } ++ self.settle_pending(&state); ++ } ++ ++ fn finish_record(&self, state: &mut State, record: Record) { ++ if state.closed || !state.tracked { ++ return; ++ } ++ if state.done.len() >= RECORDS { ++ let mut joined = record; ++ for other in state.done.drain(..) { ++ joined.area = bounding(Some(joined.area), Some(other.area)).unwrap_or(joined.area); ++ joined.seq = joined.seq.max(other.seq); ++ joined.id = 0; ++ } ++ state.done.push(joined); ++ } else { ++ state.done.push(record); ++ } ++ let Some(pages_mark) = pages() else { ++ return; ++ }; ++ if state.eager_until.is_some_and(|until| Instant::now() < until) { ++ self.sync_done(state); ++ return; ++ } ++ if state.armed.iter().any(|armed| holds(*armed, record.area)) { ++ return; ++ } ++ let start = Instant::now(); ++ let list = self.layout.pages_of(record.area); ++ let refused = pages_mark.arm(&list); ++ // A page the guest wrote takes the rectangle too, by the merge. ++ let open: Vec = (state.snapshots.keys()) ++ .filter(|page| list.binary_search(page).is_ok()) ++ .copied() ++ .collect(); ++ if !open.is_empty() { ++ pages_mark.freeze(&open); ++ state.frozen.extend(open); ++ state.frozen.sort_unstable(); ++ state.frozen.dedup(); ++ } ++ if state.armed.len() == RECORDS { ++ state.armed.clear(); ++ } ++ state.armed.push(record.area); ++ let spent = start.elapsed().as_nanos() as u64; ++ self.counts.arm_ns.fetch_add(spent, Ordering::Relaxed); ++ self.counts.arms.fetch_add(1, Ordering::Relaxed); ++ // No trap comes for these: the sync is now, before anyone hears of this completion. A CPU ++ // write in the rectangle raced the GPU's; one beside it stays, and the image takes it. ++ if !refused.is_empty() { ++ self.sync_done(state); ++ self.note_came(state, refused); ++ } ++ } ++ ++ /// Copy every done rectangle into the pages, and hold them. ++ fn sync_done(&self, state: &mut State) { ++ if state.done.is_empty() { ++ self.thaw_all(state); ++ return; ++ } ++ let Some(pages_mark) = pages() else { ++ return; ++ }; ++ let start = Instant::now(); ++ let done = std::mem::take(&mut state.done); ++ let mut list = Vec::new(); ++ for record in &done { ++ let area = record.area; ++ let moved = match state.snapshots.is_empty() { ++ true => self.detile(area), ++ false => self.merge(state, &pages_mark, area), ++ }; ++ if !moved { ++ self.counts.sync_errors.fetch_add(1, Ordering::Relaxed); ++ eprintln!("surface_coherent: a sync of {area:?} failed"); ++ } ++ self.counts.regions.fetch_add(1, Ordering::Relaxed); ++ let bytes = texels(area) * self.layout.texel; ++ self.counts.bytes_detiled.fetch_add(bytes, Ordering::Relaxed); ++ list.extend(self.layout.pages_of(area)); ++ } ++ state.window_syncs += 1; ++ list.sort_unstable(); ++ list.dedup(); ++ self.thaw_all(state); ++ let held = Instant::now(); ++ let came = pages_mark.hold(&list, false); ++ self.counts.hold_ns.fetch_add(held.elapsed().as_nanos() as u64, Ordering::Relaxed); ++ self.note_came(state, came); ++ state.armed.clear(); ++ let spent = start.elapsed().as_nanos() as u64; ++ self.counts.sync_ns.fetch_add(spent, Ordering::Relaxed); ++ self.settle_pending(state); ++ } ++ ++ /// A page the guest reached as its markers moved may hold a write: the image takes it again. ++ fn note_came(&self, state: &mut State, came: Vec) { ++ self.counts.came.fetch_add(came.len() as u64, Ordering::Relaxed); ++ for page in came { ++ if let Some(area) = self.layout.area_of(page) { ++ state.guest = Some(match state.guest { ++ Some(guest) => bounding(Some(guest), Some(area)).unwrap_or(guest), ++ None => area, ++ }); ++ } ++ } ++ } ++ ++ fn thaw_all(&self, state: &mut State) { ++ if !state.frozen.is_empty() { ++ if let Some(pages_mark) = pages() { ++ pages_mark.thaw(&state.frozen); ++ } ++ state.frozen.clear(); ++ } ++ } ++ ++ fn detile(&self, area: [u32; 4]) -> bool { ++ let to = self.layout.at(area[0], area[1]) as *mut u8; ++ (self.read)(area, to, (self.layout.row / self.layout.texel) as u32) ++ } ++ ++ /// `area` into the pages, keeping what the guest wrote in open pages: every texel of them that ++ /// differs from the page's snapshot. Those pages stay frozen meanwhile, and the snapshot takes ++ /// the texels the sync moved. ++ fn merge(&self, state: &mut State, pages_mark: &Arc, area: [u32; 4]) -> bool { ++ let [x0, y0, x1, y1] = area; ++ let texel = self.layout.texel as usize; ++ let (width, height) = ((x1 - x0) as usize, (y1 - y0) as usize); ++ let row = width * texel; ++ let mut scratch = vec![0u8; row * height]; ++ if !(self.read)(area, scratch.as_mut_ptr(), width as u32) { ++ return false; ++ } ++ let list = self.layout.pages_of(area); ++ let opened: Vec = (state.snapshots.keys()) ++ .filter(|page| list.binary_search(page).is_ok() && !state.frozen.contains(page)) ++ .copied() ++ .collect(); ++ pages_mark.freeze(&opened); ++ state.frozen.extend(&opened); ++ state.frozen.sort_unstable(); ++ self.counts.merges.fetch_add(1, Ordering::Relaxed); ++ for (y, line) in (y0..y1).zip(scratch.chunks_exact(row)) { ++ for (x, source) in (x0..x1).zip(line.chunks_exact(texel)) { ++ let at = self.layout.at(x, y); ++ // SAFETY: `at` lies in the surface's mapping, whole texels of it; the guest's CPU ++ // does not reach a frozen page, and a page with no snapshot is armed or held. ++ unsafe { ++ let bytes = std::slice::from_raw_parts_mut(at as *mut u8, texel); ++ let page = at & !(PAGE - 1); ++ let offset = (at - page) as usize; ++ match state.snapshots.get_mut(&page) { ++ Some(snapshot) => { ++ let was = &mut snapshot[offset..(offset + texel).min(PAGE as usize)]; ++ if bytes[..was.len()] == *was { ++ bytes[..was.len()].copy_from_slice(&source[..was.len()]); ++ was.copy_from_slice(&source[..was.len()]); ++ } ++ } ++ None => bytes.copy_from_slice(source), ++ } ++ } ++ } ++ } ++ true ++ } ++ ++ /// Wait, with the lock let go, until no GPU copy back runs and, with `area`, no record it ++ /// overlaps does, or the bound: whether none does. The caller stopped a vCPU or an encoder, and ++ /// nothing it holds blocks the GPU. ++ fn wait_settled<'a>( ++ &'a self, ++ mut state: MutexGuard<'a, State>, ++ area: Option<[u32; 4]>, ++ ) -> (MutexGuard<'a, State>, bool) { ++ let until = Instant::now() + Duration::from_millis(knob("surface_coherent_wait_ms", DEFAULT_WAIT_MS)); ++ loop { ++ let overlaps = |area| state.running.iter().any(|record| meet(record.area, area)); ++ let busy = state.flushing != 0 || area.is_some_and(overlaps); ++ if !busy { ++ return (state, true); ++ } ++ let left = until.saturating_duration_since(Instant::now()); ++ if left.is_zero() { ++ return (state, false); ++ } ++ let (guard, _) = self.moved.wait_timeout(state, left).unwrap_or_else(|e| e.into_inner()); ++ state = guard; ++ } ++ } ++ ++ /// The guest touched `page`. ++ fn fault(&self, page: u64, write: bool) { ++ let mut state = self.lock(); ++ if !state.tracked { ++ return; ++ } ++ self.counts.traps.fetch_add(1, Ordering::Relaxed); ++ if state.seen.insert(page) { ++ self.counts.unique_pages.fetch_add(1, Ordering::Relaxed); ++ } ++ self.sync_done(&mut state); ++ let mark = pages(); ++ // Another vCPU's probe may be about to rerun on these pages: only the last of them, which ++ // lets it name its page, ends the probing. ++ let over = state.probed_at.is_some_and(|at| at.elapsed() >= PROBE_GIVEN_UP); ++ if state.probing > 0 && (over || (write && state.probing == 1)) { ++ state.probing = 0; ++ state.probed_at = None; ++ if let Some(mark) = &mark { ++ let came = mark.hold(&self.all_pages(), false); ++ self.note_came(&mut state, came); ++ } ++ state.armed.clear(); ++ } else if write { ++ state.probing = state.probing.saturating_sub(1); ++ } ++ if !write { ++ if let Some(mark) = &mark { ++ let came = mark.hold(&[page], false); ++ self.note_came(&mut state, came); ++ } ++ return; ++ } ++ self.counts.writes.fetch_add(1, Ordering::Relaxed); ++ if let Some(area) = self.layout.area_of(page) { ++ // A running write needs no wait: the merge at its completion keeps this one. A GPU ++ // copy back already on its way cannot merge. ++ let (settled, clear) = self.wait_settled(state, None); ++ state = settled; ++ self.sync_done(&mut state); ++ if !clear { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ } ++ state.guest = Some(match state.guest { ++ Some(guest) => bounding(Some(guest), Some(area)).unwrap_or(guest), ++ None => area, ++ }); ++ } ++ state.wrote_at = Some(Instant::now()); ++ if !state.snapshots.contains_key(&page) { ++ // SAFETY: the page lies in the surface's mapping; the guest's write stopped before it. ++ let was = unsafe { std::slice::from_raw_parts(page as *const u8, PAGE as usize) }; ++ state.snapshots.insert(page, was.into()); ++ } ++ if let Some(mark) = &mark { ++ mark.hold(&[page], false); ++ mark.open(page); ++ } ++ } ++ ++ /// An encoder is about to use the image: where the guest wrote is stale in it, and comes back ++ /// for this command's copy in. With no standing (`whole`) the copy in reads every texel. The ++ /// pages hold everything done first, and are held before the copy in can read them. ++ pub(crate) fn taking(&self, whole: bool) -> Option<[u32; 4]> { ++ let mut state = self.lock(); ++ if !state.tracked { ++ return None; ++ } ++ if !whole && state.guest.is_none() { ++ return None; ++ } ++ let mut stale = state.guest; ++ if whole { ++ stale = Some(self.layout.whole()); ++ } ++ if let Some(area) = stale { ++ // The copy in runs after what is running here, and reads pages that lack it: a running ++ // write's texels must reach the pages first. Over no CPU write, a refused standing's ++ // copy back (`shadow_lazy::met_by`) does that without a wait. ++ if !state.snapshots.is_empty() { ++ let (settled, clear) = self.wait_settled(state, Some(area)); ++ state = settled; ++ if !clear { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++ self.sync_done(&mut state); ++ let list: Vec = state.snapshots.keys().copied().collect(); ++ if let Some(mark) = pages() { ++ let came = mark.hold(&list, true); ++ state.guest = None; ++ self.note_came(&mut state, came); ++ } ++ state.snapshots.clear(); ++ if whole { ++ state.guest = None; ++ } ++ } ++ stale ++ } ++ ++ /// A GPU copy back of what ran is about to record: everything done went on the CPU, and what ++ /// is still running goes by the copy, up to the sequence given. `None` where nothing runs. The ++ /// lock is not held while it records. ++ pub(crate) fn before_flush(&self) -> Option { ++ let mut state = self.lock(); ++ self.sync_done(&mut state); ++ if state.running.is_empty() { ++ return None; ++ } ++ if !state.snapshots.is_empty() { ++ let area = state.guest.unwrap_or(self.layout.whole()); ++ let (settled, clear) = self.wait_settled(state, Some(area)); ++ state = settled; ++ self.sync_done(&mut state); ++ if state.running.is_empty() { ++ return None; ++ } ++ if !clear { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++ let through = state.seq; ++ state.flushing = through; ++ Some(through) ++ } ++ ++ /// The copy back of everything up to `through`, of `bytes`, recorded and committed. ++ pub(crate) fn flush_recorded(&self, bytes: u64) { ++ self.counts.flushes.fetch_add(1, Ordering::Relaxed); ++ self.counts.flushed_bytes.fetch_add(bytes, Ordering::Relaxed); ++ } ++ ++ /// The copy back was not made after all. ++ pub(crate) fn flush_failed(&self) { ++ self.lock().flushing = 0; ++ self.moved.notify_all(); ++ } ++ ++ /// That copy back completed. ++ pub(crate) fn flushed(&self, through: u64) { ++ let mut state = self.lock(); ++ if state.flushing == through { ++ state.flushing = 0; ++ } ++ state.flushed_to = state.flushed_to.max(through); ++ let (gone, kept): (Vec<_>, Vec<_>) = state.done.iter().partition(|r| r.seq <= through); ++ state.done = kept; ++ if let Some(mark) = pages() { ++ let mut list: Vec = ++ gone.iter().flat_map(|record| self.layout.pages_of(record.area)).collect(); ++ list.sort_unstable(); ++ list.dedup(); ++ let still: Vec = (state.done.iter()) ++ .flat_map(|record| self.layout.pages_of(record.area)) ++ .collect(); ++ list.retain(|page| !still.contains(page)); ++ if !list.is_empty() { ++ let came = mark.hold(&list, false); ++ self.note_came(&mut state, came); ++ state.armed.clear(); ++ } ++ } ++ self.moved.notify_all(); ++ self.settle_pending(&state); ++ } ++ ++ /// A probe: every held page armed, so the write that started it names its page. ++ fn probe(&self) { ++ let mut state = self.lock(); ++ if !state.tracked || state.closed { ++ return; ++ } ++ state.probing += 1; ++ state.probed_at = Some(Instant::now()); ++ self.counts.probes.fetch_add(1, Ordering::Relaxed); ++ if let Some(mark) = pages() { ++ let refused = mark.arm(&self.all_pages()); ++ self.note_came(&mut state, refused); ++ } ++ } ++ ++ /// Everything done into the pages and the markers off: it goes on as `surface_lazy_present` ++ /// alone, what is running copied back by the clock. ++ pub(crate) fn abandon(&self) { ++ let mut state = self.lock(); ++ self.sync_done(&mut state); ++ self.close(&mut state, true); ++ } ++ ++ fn close(&self, state: &mut State, untrack: bool) { ++ state.closed = true; ++ if state.tracked { ++ state.tracked = false; ++ if let (true, Some(mark)) = (untrack, pages()) { ++ mark.untrack(self.layout.first_page()); ++ } ++ let me = std::ptr::from_ref(self); ++ all().write().unwrap().retain(|other| !std::ptr::eq(Arc::as_ptr(other), me)); ++ } ++ state.done.clear(); ++ state.snapshots.clear(); ++ state.frozen.clear(); ++ self.moved.notify_all(); ++ if self.counts.defers.load(Ordering::Relaxed) + self.counts.traps.load(Ordering::Relaxed) > 0 { ++ let mut ended = ended().lock().unwrap_or_else(|e| e.into_inner()); ++ if ended.len() < ENDED_KEPT { ++ ended.push(self.line("closed", state)); ++ } ++ } ++ } ++ ++ /// Before an unmap: whether it may go on. Everything done goes to the pages while they are ++ /// still the guest's, and what the pending writes hold goes, so the surface's references fall; ++ /// a surface with work running stays busy. ++ fn unmapping(&self) -> bool { ++ let mut state = self.lock(); ++ self.sync_done(&mut state); ++ if !state.running.is_empty() || state.flushing != 0 { ++ return false; ++ } ++ state.open.clear(); ++ self.settle_pending(&state); ++ true ++ } ++ ++ /// Whether writes are pending in records yet: the texture's image must live on. ++ fn has_records(&self, state: &State) -> bool { ++ !(state.open.is_empty() && state.running.is_empty() && state.done.is_empty()) ++ } ++ ++ /// What the plain path's [`Lazy`] holds of this: the bounding rectangle of what ran. ++ fn settle_pending(&self, state: &State) { ++ let Some(lazy) = self.lazy.upgrade() else { ++ return; ++ }; ++ let mut settled = None; ++ for record in state.running.iter().chain(&state.done) { ++ settled = Some(match settled { ++ Some(area) => bounding(Some(area), Some(record.area)).unwrap_or(area), ++ None => record.area, ++ }); ++ } ++ lazy.settled_coherently(settled, self.has_records(state)); ++ } ++ ++ pub(crate) fn copied_in(&self, texels: u64) { ++ self.counts.copy_ins.fetch_add(1, Ordering::Relaxed); ++ let bytes = texels * self.layout.texel; ++ self.counts.copy_in_bytes.fetch_add(bytes, Ordering::Relaxed); ++ } ++ ++ /// A present: whether the display may sample the image instead of the pages. ++ pub(crate) fn presented(&self) -> bool { ++ self.counts.presents.fetch_add(1, Ordering::Relaxed); ++ let state = self.lock(); ++ state.tracked ++ && !state.closed ++ && state.eager_until.is_none_or(|until| Instant::now() >= until) ++ && crate::live::on(DISPLAY, true) ++ } ++ ++ fn timed(&self, ns: u64) { ++ self.counts.trap_ns.fetch_add(ns, Ordering::Relaxed); ++ self.counts.timed.fetch_add(1, Ordering::Relaxed); ++ self.counts.trap_ns_max.fetch_max(ns, Ordering::Relaxed); ++ } ++ ++ fn line(&self, label: &str, state: &State) -> String { ++ let read = |count: &AtomicU64| count.load(Ordering::Relaxed); ++ let c = &self.counts; ++ let copy = knob("surface_coherent_copy_ns_per_mib", DEFAULT_COPY_NS_PER_MIB) as f64; ++ let by_gpu = |bytes: u64| bytes as f64 / (1 << 20) as f64 * copy; ++ let presents = read(&c.presents).max(1) as f64; ++ let trap = read(&c.trap_ns) as f64; ++ let sync = read(&c.sync_ns) as f64; ++ let detect = (read(&c.arm_ns) as f64 + (trap - sync).max(0.0)) / presents; ++ let sync = sync / presents; ++ let fetch = by_gpu(read(&c.copy_in_bytes) + read(&c.flushed_bytes)) / presents; ++ let eager = by_gpu(read(&c.deferred_bytes)) / presents; ++ let total = detect + sync + fetch; ++ let seconds = self.made.elapsed().as_secs_f64().max(0.001); ++ format!( ++ "{{\"label\":\"{label}\",\"at\":\"{:#x}\",\"w\":{},\"h\":{},\"tracked\":{},\"closed\":{},\ ++ \"presents\":{},\"defers\":{},\"traps\":{},\"writes\":{},\"unique_pages\":{},\ ++ \"traps_per_s\":{:.2},\"regions\":{},\"bytes_detiled\":{},\"sync_ns\":{},\ ++ \"trap_ns\":{},\"trap_ns_max\":{},\"resumes\":{},\"arm_ns\":{},\"arms\":{},\ ++ \"hold_ns\":{},\"demotions\":{},\"eager_frames\":{},\"merges\":{},\"came\":{},\ ++ \"probes\":{},\"flushes\":{},\"flushed_bytes\":{},\"copy_ins\":{},\"copy_in_bytes\":{},\ ++ \"overwrite_risk\":{},\"sync_errors\":{},\"pending_done\":{},\"open_pages\":{},\ ++ \"per_present_ns\":{{\"detect\":{detect:.0},\"sync\":{sync:.0},\"fetch\":{fetch:.0},\ ++ \"total\":{total:.0},\"eager_would\":{eager:.0}}},\"pays\":{}}}\n", ++ self.layout.first_page(), ++ self.layout.width, ++ self.layout.height, ++ state.tracked, ++ state.closed, ++ read(&c.presents), ++ read(&c.defers), ++ read(&c.traps), ++ read(&c.writes), ++ read(&c.unique_pages), ++ read(&c.traps) as f64 / seconds, ++ read(&c.regions), ++ read(&c.bytes_detiled), ++ read(&c.sync_ns), ++ read(&c.trap_ns), ++ read(&c.trap_ns_max), ++ read(&c.timed), ++ read(&c.arm_ns), ++ read(&c.arms), ++ read(&c.hold_ns), ++ read(&c.demotions), ++ read(&c.eager), ++ read(&c.merges), ++ read(&c.came), ++ read(&c.probes), ++ read(&c.flushes), ++ read(&c.flushed_bytes), ++ read(&c.copy_ins), ++ read(&c.copy_in_bytes), ++ read(&c.overwrite_risk), ++ read(&c.sync_errors), ++ state.done.len(), ++ state.snapshots.len(), ++ total < eager, ++ ) ++ } ++} ++ ++fn find(addr: u64) -> Option> { ++ (all().read().unwrap().iter()).find(|one| one.range().contains(&addr)).cloned() ++} ++ ++fn tracked() -> Vec> { ++ all().read().unwrap().clone() ++} ++ ++/// The guest touched the page at `addr`, of a surface's mapping: answered once its bytes are ++/// current. A write leaves it open. ++pub fn fault(addr: u64, write: bool) { ++ let page = addr & !(PAGE - 1); ++ match find(addr) { ++ Some(surface) => surface.fault(page, write), ++ None => { ++ if let Some(mark) = pages() { ++ mark.untrack(addr); ++ } ++ } ++ } ++} ++ ++/// From a stop at `addr` to the access's rerun. ++pub fn timed(addr: u64, ns: u64) { ++ if let Some(surface) = find(addr) { ++ surface.timed(ns); ++ } ++} ++ ++/// A write hit a held page no one named: every held page armed, to name it. ++pub fn probe() { ++ for surface in tracked() { ++ surface.probe(); ++ } ++} ++ ++/// Everything back in its pages and untracked: the markers are about to go. ++pub fn abandon() { ++ for surface in tracked() { ++ surface.abandon(); ++ } ++} ++ ++/// The guest resets: every surface goes, nothing written. The vCPUs are stopped. ++pub fn reset() { ++ for surface in tracked() { ++ let mut state = surface.lock(); ++ surface.close(&mut state, false); ++ state.running.clear(); ++ state.open.clear(); ++ surface.settle_pending(&state); ++ } ++} ++ ++/// Before an unmap of the surface over `bytes`: whether it may go on (the module docs). ++pub fn unmapping(bytes: Range) -> bool { ++ tracked().iter().filter(|s| s.range().start < bytes.end && bytes.start < s.range().end).all(|s| s.unmapping()) ++} ++ ++/// The surface over `bytes` was unmapped: QEMU already dropped the marks. ++pub fn unmapped(bytes: Range) { ++ for surface in tracked() { ++ if surface.range().start < bytes.end && bytes.start < surface.range().end { ++ let mut state = surface.lock(); ++ surface.close(&mut state, false); ++ } ++ } ++} ++ ++/// Whether any surface is tracked, one lock: while none is, a completion does nothing here. ++pub(crate) fn surfaces_tracked() -> bool { ++ !all().read().unwrap().is_empty() ++} ++ ++/// One line a surface: what [`Coherent::line`] says, under `label`. A surface that went comes once. ++pub fn report(label: &str) -> String { ++ let mut lines: String = std::mem::take(&mut *ended().lock().unwrap_or_else(|e| e.into_inner())).concat(); ++ for surface in tracked() { ++ lines.push_str(&surface.line(label, &surface.lock())); ++ } ++ lines ++} +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 5aefa2df..64dd5ec4 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -81,18 +81,23 @@ + //! `surface_lazy` alone keeps it unless a command buffer runs meanwhile. + //! + //! **A texture released while ahead** is not copied back where nothing else holds its image: its +-//! pages may be another surface's by then. ++//! pages may be another surface's by then. Where they are not, a mapped surface's completed writes ++//! go missing here: this switch's stale bytes. `shadow_coherent.rs` keeps them until the unmap. ++//! ++//! **Under the switch `surface_coherent`** the surface's pages trap instead (`shadow_coherent.rs`): ++//! no clock, no copy back at a present or a later encoder, and exact news of a CPU write. + + use super::command_buffer::CommandBuffer; + use super::device::Device; + use super::queue::CommandQueue; + use super::resource::ResourceKey; + use super::shadow::{bounding, CopyCause, ShadowCopies, ShadowCopy, Step}; ++use super::shadow_coherent::{Coherent, Layout, Reader}; + use super::tracker::{Summary, Use, UseRange}; + use crate::exit_guard; + use std::any::Any; + use std::cell::Cell; +-use std::collections::VecDeque; ++use std::collections::{HashSet, VecDeque}; + use std::ops::Range; + use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex, OnceLock, RwLock, Weak}; +@@ -201,6 +206,15 @@ fn surfaces() -> Vec> { + .collect() + } + ++/// Where a standing's stale texels and the guest's writes join. ++fn bounding_stale(a: Option<[u32; 4]>, b: Option<[u32; 4]>) -> Option<[u32; 4]> { ++ match (a, b) { ++ (Some(_), Some(_)) => bounding(a, b), ++ (a, None) => a, ++ (None, b) => b, ++ } ++} ++ + /// Whether two ranges of host addresses share a byte. + fn meet(a: &Range, b: &Range) -> bool { + a.start < b.end && b.start < a.end +@@ -217,6 +231,10 @@ pub(crate) fn remapped(bytes: Range) { + } + for lazy in surfaces() { + if lazy.over(&bytes) { ++ // The surface's mapping stays while a texture holds it: what is done still lands. ++ if let Some(coherent) = lazy.coherent() { ++ coherent.abandon(); ++ } + lazy.written.fetch_add(1, Ordering::Relaxed); + } + } +@@ -274,13 +292,41 @@ pub(crate) fn presented(bytes: impl Fn() -> Option>) { + matched = true; + lazy.displayed.store(true, Ordering::Relaxed); + lazy.presents.fetch_add(1, Ordering::Relaxed); +- lazy.flush_once_committed(); ++ // What traps needs no copy back for the display, which samples its image or its pages. ++ if lazy.coherent().is_none() { ++ lazy.flush_once_committed(); ++ } + } + } + let count = if matched { &PRESENTS } else { &PRESENTS_UNMATCHED }; + count.fetch_add(1, Ordering::Relaxed); + } + ++/// The surface whose image the display may sample instead of the pages of `bytes`, under ++/// `surface_coherent`: a present copies nothing then. `Some` only where its image holds a standing ++/// a command buffer can take (the module docs), and it stays ordered through its own key. ++pub(crate) fn display_image(bytes: &Range) -> Option { ++ if !ANY.load(Ordering::Relaxed) { ++ return None; ++ } ++ let lazy = surfaces().into_iter().find(|lazy| lazy.over(bytes))?; ++ let coherent = lazy.coherent()?; ++ let takes = coherent.presented() && lazy.standing_committed() && !lazy.beside_another(); ++ takes.then(|| lazy.owner.get().and_then(Weak::upgrade)).flatten() ++} ++ ++/// A command buffer completed, before its handlers: its writes in images trap from here. ++pub(crate) fn completing(id: u64) { ++ if !crate::shadow_coherent::surfaces_tracked() { ++ return; ++ } ++ for lazy in surfaces() { ++ if let Some(coherent) = lazy.coherent() { ++ coherent.completing(id); ++ } ++ } ++} ++ + /// Whether any surface is lazy, one atomic read: while none is, a commit does nothing here. + pub(crate) fn active() -> bool { + ANY.load(Ordering::Relaxed) +@@ -333,7 +379,16 @@ pub(crate) fn ahead() -> bool { + /// docs). A copy in is such a use, so its command buffer's own writes wait for its commit. + pub(crate) fn met_by(summaries: &[&Summary]) -> Vec> { + let mut met = Vec::new(); ++ // Most surfaces stay ahead for good where the pages trap: their keys rule most of them out. ++ let keys: HashSet = (summaries.iter()) ++ .flat_map(|s| s.all_uses().iter().map(|used| used.key)) ++ .collect(); + for lazy in surfaces() { ++ if !lazy.truth.get().is_some_and(|(key, alias, _)| { ++ keys.contains(key) || alias.is_some_and(|alias| keys.contains(&alias)) ++ }) { ++ continue; ++ } + let settled = (lazy.pending.lock().unwrap().as_ref()) + .is_some_and(|left| left.settled.is_some()); + let used = || (summaries.iter()).any(|s| s.all_uses().iter().any(|u| lazy.meets(u))); +@@ -352,6 +407,10 @@ pub(crate) fn after_commit(id: u64) { + } + for lazy in surfaces() { + let moved = (lazy.pending.lock().unwrap().as_mut()).is_some_and(|left| left.committed(id)); ++ if let Some(coherent) = lazy.coherent() { ++ coherent.committed(id); ++ continue; ++ } + let wants = + lazy.flush_after + .compare_exchange(id, 0, Ordering::Relaxed, Ordering::Relaxed); +@@ -367,6 +426,10 @@ pub(crate) fn after_commit(id: u64) { + pub(crate) fn deferred_by(id: u64) -> Vec<(Arc, ShadowCopy, Vec)> { + let mut left = Vec::new(); + for lazy in surfaces() { ++ // A command that meets its bytes copies them back itself ([`met_by_command`]). ++ if lazy.coherent().is_some() { ++ continue; ++ } + let mut pending = lazy.pending.lock().unwrap(); + let Some(ahead) = pending.as_mut() else { + continue; +@@ -389,10 +452,46 @@ pub(crate) fn deferred_by(id: u64) -> Vec<(Arc, ShadowCopy, Vec)> { + left + } + ++/// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what ++/// that command buffer's earlier encoders left ahead in a trapping surface whose bytes it meets. ++/// The record stays, whose texels the copy repeats, so what the record holds lasts the copy. ++pub(super) fn met_by_command( ++ id: u64, ++ uses: &[Use], ++ shares: impl Fn(&ShadowCopies) -> bool, ++) -> Vec> { ++ let mut copies = Vec::new(); ++ for lazy in surfaces() { ++ if lazy.coherent().is_none() { ++ continue; ++ } ++ let pending = lazy.pending.lock().unwrap(); ++ let Some(ahead) = pending.as_ref() else { ++ continue; ++ }; ++ let Some((_, area)) = ahead.open.iter().find(|(open, _)| *open == id) else { ++ continue; ++ }; ++ let shared = ahead.copies.downcast_ref::().is_some_and(&shares); ++ if shared || uses.iter().any(|used| lazy.meets(used)) { ++ copies.push(copy_back_of(ahead.copies.clone(), *area)); ++ } ++ } ++ copies ++} ++ + /// The copies back the clock asks for (the module docs): whether any surface holds writes ahead. + fn flush_due(max: Duration) -> bool { + let mut any = false; + for lazy in surfaces() { ++ // Nothing stale here but what the traps answer, while they do. ++ if let Some(coherent) = lazy.coherent() { ++ if crate::live::on("surface_coherent", false) { ++ any |= lazy.ahead.load(Ordering::Relaxed); ++ continue; ++ } ++ coherent.abandon(); ++ } + let due = (lazy.pending.lock().unwrap().as_ref()).map(|left| { + let waited = left.since.elapsed(); + waited >= max && (left.settled.is_some() || waited >= max * 4) +@@ -548,7 +647,7 @@ impl Pending { + } + + /// A copy back of `area` of `copies`, as an encoder's end makes it. +-fn copy_back_of(copies: Hold, area: Option<[u32; 4]>) -> ShadowCopy { ++fn copy_back_of(copies: Hold, area: Option<[u32; 4]>) -> ShadowCopy { + ShadowCopy { + copies: copies.downcast().expect("a surface's copies"), + step: Step::Back, +@@ -613,6 +712,10 @@ pub(crate) struct Lazy { + refused: AtomicU64, + copied_in: AtomicU64, + presents: AtomicU64, ++ /// Its trap coherence (`shadow_coherent.rs`), where the texture's image takes it. ++ coherent: OnceLock>, ++ /// Weakly, the texture the display samples instead of the pages ([`display_image`]). ++ owner: OnceLock>, + } + + impl Lazy { +@@ -644,10 +747,57 @@ impl Lazy { + refused: AtomicU64::new(0), + copied_in: AtomicU64::new(0), + presents: AtomicU64::new(0), ++ coherent: OnceLock::new(), ++ owner: OnceLock::new(), + }) + }) + } + ++ /// Its pages trap, through `read` of its image, laid out as `layout`. ++ pub(crate) fn traps(self: &Arc, layout: Layout, read: Reader) { ++ let _ = self.coherent.set(Coherent::new(layout, read, Arc::downgrade(self))); ++ } ++ ++ /// The texture over it, the display's way to its image. ++ pub(crate) fn owned_by(&self, texture: Weak) { ++ let _ = self.owner.set(texture); ++ } ++ ++ /// Its trap coherence, while it holds: `None` where it never traps or no longer does. ++ pub(crate) fn coherent(&self) -> Option<&Arc> { ++ self.coherent.get().filter(|coherent| coherent.is_live()) ++ } ++ ++ /// Whether its image must stay whole: a trapping surface's copies back are any rectangle. ++ pub(super) fn whole_image(&self) -> bool { ++ self.coherent.get().is_some_and(|coherent| coherent.is_live()) ++ } ++ ++ /// Whether it writes back keeping and merging, a CPU write having come lately. ++ pub(super) fn wants_merge(&self) -> bool { ++ self.coherent().is_some_and(|coherent| coherent.wants_merge()) ++ } ++ ++ /// Whether a standing stands, from a command buffer already committed. ++ fn standing_committed(&self) -> bool { ++ let stood = *self.stood.lock().unwrap(); ++ stood.is_some_and(|stood| committed_ids().lock().unwrap().contains(&stood.id)) ++ } ++ ++ /// `settled` is what ran and is not in the pages, and whether any record stays at all. ++ pub(crate) fn settled_coherently(&self, settled: Option<[u32; 4]>, records: bool) { ++ let mut pending = self.pending.lock().unwrap(); ++ let Some(left) = pending.as_mut() else { ++ return; ++ }; ++ left.settled = settled.map(Some); ++ if !records { ++ *pending = None; ++ drop(pending); ++ self.left_behind(); ++ } ++ } ++ + /// Note what its copies back hold, weakly: its image, and the bytes its texture lies over. + pub(crate) fn keeps(&self, image: Weak, bytes: Weak) { + let _ = self.keeps.set([image, bytes]); +@@ -681,7 +831,10 @@ impl Lazy { + } + + /// Note a copy in of its bytes for `cause`. +- pub(super) fn copied_in(&self, cause: CopyCause) { ++ pub(super) fn copied_in(&self, cause: CopyCause, texels: u64) { ++ if let Some(coherent) = self.coherent() { ++ coherent.copied_in(texels); ++ } + let at = match cause { + CopyCause::First => 0, + CopyCause::BeyondHeld => 1, +@@ -701,8 +854,10 @@ impl Lazy { + area: Option<[u32; 4]>, + id: u64, + ) -> bool { +- if !crate::live::on(PRESENT, false) +- || !(self.displayed.load(Ordering::Relaxed) || crate::live::on(PRESENT_ALL, false)) ++ let coherent = self.coherent(); ++ if coherent.is_none() ++ && (!crate::live::on(PRESENT, false) ++ || !(self.displayed.load(Ordering::Relaxed) || crate::live::on(PRESENT_ALL, false))) + { + return false; + } +@@ -727,6 +882,10 @@ impl Lazy { + }; + let _ = self.queue.set(Arc::new(queue)); + } ++ // Its record is what a trap copies: no page, no deferring. ++ if coherent.is_some_and(|coherent| !coherent.defers(id, area)) { ++ return false; ++ } + let mut pending = self.pending.lock().unwrap(); + match &mut *pending { + Some(left) => match left.open.iter_mut().find(|(open, _)| *open == id) { +@@ -788,21 +947,33 @@ impl Lazy { + /// over the pages of a texture nothing else holds (the module docs); a copy that cannot be + /// recorded stays for the next asker. + pub(crate) fn flush(self: &Arc) -> Option { ++ // What is done goes on the CPU: a copy back is for what runs, which it is ordered after. ++ let coherent = self.coherent().cloned(); ++ let through = match &coherent { ++ Some(coherent) => Some(coherent.before_flush()?), ++ None => None, ++ }; + let (area, copies, holds) = { + let pending = self.pending.lock().unwrap(); + let left = pending.as_ref()?; + (left.settled, left.copies.clone(), left.holds.clone()) + }; +- // The image's: `Pending`'s and this one, where neither texture nor command holds it. +- if holds.first().is_some_and(|image| Arc::strong_count(image) <= 2) { ++ // The image's: `Pending`'s and this one, where neither texture nor command holds it. A ++ // trapping surface's pages stay the guest's until its unmap, whatever texture went. ++ if coherent.is_none() && holds.first().is_some_and(|image| Arc::strong_count(image) <= 2) { + *self.pending.lock().unwrap() = None; + self.left_behind(); + DISCARDED.fetch_add(1, Ordering::Relaxed); + return None; + } +- let area = area?; ++ let Some(area) = area else { ++ if let Some(coherent) = &coherent { ++ coherent.flush_failed(); ++ } ++ return None; ++ }; + let _own = self.own(); +- let copy = copy_back_of(copies, area); ++ let copy: ShadowCopy = copy_back_of(copies, area); + let texels = copy.texels_back(); + let flush = (|| { + let queue = self.queue.get()?.downcast_ref::()?; +@@ -810,13 +981,27 @@ impl Lazy { + let mut encoder = command_buffer.blit_command_encoder().ok()?; + encoder.surface_copy_back(©, holds); + encoder.end_encoding().ok()?; ++ if let (Some(coherent), Some(through)) = (&coherent, through) { ++ let done = coherent.clone(); ++ let handler = Box::new(move |_: &CommandBuffer| done.flushed(through)); ++ command_buffer.add_completed_handler(handler).ok()?; ++ } + command_buffer.commit().ok()?; + Some(command_buffer) + })(); + let Some(flush) = flush else { + eprintln!("surface_lazy: a copy back was not recorded"); ++ if let Some(coherent) = &coherent { ++ coherent.flush_failed(); ++ } + return None; + }; ++ if let Some(coherent) = &coherent { ++ coherent.flush_recorded(texels * 4); ++ FLUSHES.fetch_add(1, Ordering::Relaxed); ++ FLUSHED_TEXELS.fetch_add(texels, Ordering::Relaxed); ++ return Some(flush); ++ } + self.settle(area); + FLUSHES.fetch_add(1, Ordering::Relaxed); + FLUSHED_TEXELS.fetch_add(texels, Ordering::Relaxed); +@@ -882,6 +1067,24 @@ impl Lazy { + /// Where the image stands for an encoder that met `touched`, of a command buffer of identity + /// `id`: `Some((held, stale))` where it needs no copy in (the module docs). + pub(super) fn take(&self, id: u64, touched: u64) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { ++ // Tracked from here: nothing its copy in reads can change unseen. ++ let coherent = self.coherent().filter(|coherent| coherent.tracks()); ++ let taken = self.take_standing(id, touched, coherent.is_some()); ++ match coherent { ++ Some(coherent) => { ++ let stale = coherent.taking(taken.is_none()); ++ taken.map(|(held, was)| (held, bounding_stale(was, stale))) ++ } ++ None => taken, ++ } ++ } ++ ++ fn take_standing( ++ &self, ++ id: u64, ++ touched: u64, ++ tracked: bool, ++ ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { + report(); + let Some(stood) = *self.stood.lock().unwrap() else { + REFUSED_BY[Refused::NoStanding as usize].fetch_add(1, Ordering::Relaxed); +@@ -910,7 +1113,8 @@ impl Lazy { + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_RESYNC_MS); + // Writes left ahead would be lost to a copy in, and a copy back goes over the bytes anyway. +- if every > 0 && !self.ahead.load(Ordering::Relaxed) { ++ // Where the pages trap, a write is news, and no clock need look. ++ if every > 0 && !tracked && !self.ahead.load(Ordering::Relaxed) { + let mut resynced = self.resynced.lock().unwrap(); + if resynced.elapsed().as_millis() as u64 >= every { + *resynced = Instant::now(); +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 9644b731..94970fef 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -2368,7 +2368,14 @@ impl Device { + // (`shadow.rs`), so it starts as its memory lies: zeros there would never be read, and + // writing them took most of the making (the host's zeros copied into a 1307x512 image + // took 0.9 ms of 1 ms on ANV). ++ // LOCAL (Atlas): a surface whose pages trap takes copies on the host. ++ let traps = aspects == every && crate::shadow_coherent::enabled() && texels.lies_lazily(); + let image = match aspects == every { ++ true if traps => { ++ let identity = Identity::new(hazard, false); ++ let owner = Device::runtime_owner; ++ self.new_image_starting(plan, layout, descriptor, identity, Start::Traps, owner)? ++ } + true => { + let identity = Identity::new(hazard, false); + let owner = Device::runtime_owner; +@@ -2435,6 +2442,21 @@ impl Device { + let (held, bytes): (Arc, Arc) = + (image.clone(), texels.clone()); + lazy.keeps(Arc::downgrade(&held), Arc::downgrade(&bytes)); ++ if let (true, true, Some(surface)) = ++ (traps, image.takes_host_copies(), texels.surface_layout([image.plan.extent[0], image.plan.extent[1]])) ++ { ++ let reads = Arc::downgrade(&image); ++ lazy.traps( ++ surface, ++ Box::new(move |area, to, row| { ++ // SAFETY: `shadow_coherent.rs` gives `to` over `area`'s rows, in the ++ // surface's mapping or a scratch of the rectangle. ++ (reads.upgrade()).is_some_and(|image| unsafe { ++ image.host_read_to(area, to, row).unwrap_or(false) ++ }) ++ }), ++ ); ++ } + } + Ok(Storage::Shadow { + image, +@@ -2905,6 +2927,19 @@ impl Device { + texture_type: TextureType, + class: MemoryClass, + sharing: Sharing<'_>, ++ ) -> Result<(vk::Image, Placement), Error> { ++ self.placed_image_read(plan, texture_type, class, sharing, false) ++ } ++ ++ /// [`placed_image`](Self::placed_image), where `host_reads` says the host only reads the image ++ /// (LOCAL (Atlas): a trapping surface's, `memory::read_cached_placement`). ++ fn placed_image_read( ++ &self, ++ plan: &ImagePlan, ++ texture_type: TextureType, ++ class: MemoryClass, ++ sharing: Sharing<'_>, ++ host_reads: bool, + ) -> Result<(vk::Image, Placement), Error> { + let shared = self.shared(); + let raw_device = shared.vk(); +@@ -2928,6 +2963,10 @@ impl Device { + // host-transfer usage, its copies through submissions, as on a device without host copies. + let hosted = plan.usage.contains(ImageUsageFlags::HOST_TRANSFER); + let cached = match hosted { ++ true if host_reads => { ++ let memory = lock(&shared.memory); ++ memory.read_cached_placement(requirements.memory_type_bits, class) ++ } + true => { + let memory = lock(&shared.memory); + requirements.memory_type_bits &= !memory.host_incoherent(); +@@ -3015,6 +3054,7 @@ impl Device { + texture_type: TextureType, + class: MemoryClass, + staged: bool, ++ host_reads: bool, + ) -> Option<(ImagePlan, vk::Image, Placement)> { + let shared = self.shared(); + shared.host_copies.as_ref()?; +@@ -3035,8 +3075,14 @@ impl Device { + } + let mut hosted = plan.clone(); + hosted.usage |= ImageUsageFlags::HOST_TRANSFER; +- let (raw, placement) = +- (self.placed_image(&hosted, texture_type, class, shared.sharing())).ok()?; ++ let (raw, placement) = (self.placed_image_read( ++ &hosted, ++ texture_type, ++ class, ++ shared.sharing(), ++ host_reads, ++ )) ++ .ok()?; + Some((hosted, raw, placement)) + } + +@@ -3096,8 +3142,11 @@ impl Device { + start: Start, + owner: impl FnOnce(Placement, vk::Image) -> Owner, + ) -> Result { +- let held = start == Start::Held; +- let moved = start == Start::Moved; ++ // LOCAL (Atlas): a trapping surface's image is moved on the host, the copy in its first ++ // texels; held as ever where it cannot be made so (`shadow_coherent.rs`). ++ let traps = start == Start::Traps; ++ let mut held = start == Start::Held; ++ let mut moved = start == Start::Moved || traps; + let storage = descriptor.resource_options.storage_mode()?; + // LOCAL (Atlas) diagnostic: where the image was asked for, for the image census + // (`image_log.rs`). +@@ -3125,11 +3174,17 @@ impl Device { + // `VUID-vkCmdCopyBufferToImage-dstImage-07973`), and Metal makes no CPU copy of a + // multisample texture: such an image takes neither path, and a clear writes its zeros. + let one_sample = layout.samples == 1; +- let staged = !held && !private && one_sample && self.stages(&plan, layout.pixel_format); +- let hosted = match held || private || !one_sample || is_pvrtc(descriptor.pixel_format) { ++ let staged = ++ !traps && !held && !private && one_sample && self.stages(&plan, layout.pixel_format); ++ let refused = (held || private) && !traps; ++ let hosted = match refused || !one_sample || is_pvrtc(descriptor.pixel_format) { + true => None, +- false => self.host_placed_image(&plan, layout.texture_type, class, staged), ++ false => self.host_placed_image(&plan, layout.texture_type, class, staged, traps), + }; ++ if traps && hosted.is_none() { ++ held = true; ++ moved = false; ++ } + let (plan, raw, placement, host_copies) = match hosted { + Some((plan, raw, placement)) => (plan, raw, placement, true), + None => { +@@ -3814,6 +3869,9 @@ enum Start { + /// image stages, `texture/decoded_twin.rs`): on the host or the staging queue where it takes + /// their copies, as it is made; else by the next submission, as a held image's. + Moved, ++ /// LOCAL (Atlas): a trapping surface's image (`shadow_coherent.rs`): moved on the host, its ++ /// memory the host reads fast; held where it cannot be. ++ Traps, + } + + impl Start { +@@ -4130,6 +4188,13 @@ impl Texture { + if let Some(objects) = texture.0.storage.objects() { + objects.object_made(); + } ++ // LOCAL (Atlas): the display's way to a trapping surface's image (`shadow_coherent.rs`). ++ if let Storage::Shadow { copies, .. } = &texture.0.storage { ++ if let Some(lazy) = &copies.lazy { ++ let held: Arc = texture.0.clone(); ++ lazy.owned_by(Arc::downgrade(&held)); ++ } ++ } + // A texture over host pages' bytes, or in a heap over them (a view of one too): a + // detach of them makes its image present a stand-in (`stand_in.rs`). + if let Some((bytes, image, exact)) = texture.image_over_bytes() { +@@ -4243,8 +4308,18 @@ impl Texture { + + /// LOCAL (Atlas): tell the runtime the display presents the texture's bytes + /// (`shadow_lazy.rs`). Not a Metal call. +- pub fn surface_presented(&self) { ++ /// ++ /// Under `surface_coherent`, the texture for the display to sample in this one's place, where ++ /// a surface's image holds these bytes ahead of its pages: no copy back then. ++ pub fn surface_presented(&self) -> Option { + crate::shadow_lazy::presented(|| self.host_range()); ++ let bytes = self.host_range()?; ++ let owner = crate::shadow_lazy::display_image(&bytes)?; ++ let owner = Texture(owner.downcast::().ok()?); ++ let same = owner.pixel_format() == self.pixel_format() ++ && owner.width() == self.width() ++ && owner.height() == self.height(); ++ (same && owner.usage().contains(TextureUsage::SHADER_READ)).then_some(owner) + } + + /// The buffer's bytes a texture buffer is, or `None` for a texture with an image. +diff --git a/crates/metal2vulkan-runtime/src/texture/host_copy.rs b/crates/metal2vulkan-runtime/src/texture/host_copy.rs +index 0e1e751f..a5f6e1c8 100644 +--- a/crates/metal2vulkan-runtime/src/texture/host_copy.rs ++++ b/crates/metal2vulkan-runtime/src/texture/host_copy.rs +@@ -59,7 +59,7 @@ + //! lock a detach takes to present a stand-in (`detach.rs`), so it never reaches a `VkImage` that + //! went. + +-use super::Image; ++use super::{Image, Owner}; + use crate::device::lock; + use crate::error::{vulkan, Error}; + use crate::exit_guard; +@@ -293,6 +293,128 @@ impl Image { + Ok(true) + } + ++ /// LOCAL (Atlas): the rectangle `[x0, y0, x1, y1)` of level 0 of a color image into rows of ++ /// `row` texels at `to`, on the host, for a trapping surface's sync (`shadow_coherent.rs`). ++ /// Whether it did: not while the image presents a stand-in. ++ /// ++ /// # Safety ++ /// ++ /// The image [takes host copies](Self::takes_host_copies), and `to` is writable over every ++ /// row of the rectangle. ++ pub(crate) unsafe fn host_read_to( ++ &self, ++ area: [u32; 4], ++ to: *mut u8, ++ row: u32, ++ ) -> Result { ++ let shared = self.device.shared(); ++ let Some(functions) = shared.host_copies.as_ref() else { ++ return Ok(false); ++ }; ++ let device = shared.vk().handle(); ++ let detached = lock(&self.detached); ++ if detached.is_some() { ++ return Ok(false); ++ } ++ // SAFETY: the image is live; its lock is held. ++ unsafe { self.invalidate_rows(functions, area[1], area[3]) }?; ++ let region = vk::ImageToMemoryCopyEXT { ++ p_host_pointer: to.cast(), ++ memory_row_length: row, ++ memory_image_height: 0, ++ image_subresource: vk::ImageSubresourceLayers { ++ aspect_mask: vk::ImageAspectFlags::COLOR, ++ mip_level: 0, ++ base_array_layer: 0, ++ layer_count: 1, ++ }, ++ image_offset: vk::Offset3D { ++ x: area[0] as i32, ++ y: area[1] as i32, ++ z: 0, ++ }, ++ image_extent: vk::Extent3D { ++ width: area[2] - area[0], ++ height: area[3] - area[1], ++ depth: 1, ++ }, ++ ..Default::default() ++ }; ++ let info = vk::CopyImageToMemoryInfoEXT::default() ++ .src_image(self.raw.get()) ++ .src_image_layout(vk::ImageLayout::GENERAL) ++ .regions(std::slice::from_ref(®ion)); ++ // SAFETY: the caller's contract; the image is live and in `GENERAL` since it was made, and ++ // the lock keeps its own `VkImage` presented. ++ unsafe { (functions.copy_image_to_memory_ext)(device, &info) } ++ .result() ++ .map_err(vulkan)?; ++ drop(detached); ++ shared.counts.host_texel_copy(); ++ Ok(true) ++ } ++ ++ /// LOCAL (Atlas): drop what the host's caches hold of the bytes of rows `y0..y1` of level 0. An ++ /// image's memory the host caches without coherence (Arrow Lake-S's type) keeps clean lines of ++ /// an earlier read, and a read of texels the GPU wrote since returns them: 1 run in 25 read ++ /// stale texels on ANV without this, whose `vkCopyImageToMemoryEXT` does not drop them. Tiled ++ /// rows lie in bands of tiles: [`TILE_ROWS`] bounds a tile's height at 32-bit texels, so the ++ /// bands of a rectangle's rows hold its every byte, where the driver gives the row pitch a ++ /// host-transfer image's layout may give. Where it gives none, the whole image. ++ /// ++ /// # Safety ++ /// ++ /// The image is live, its memory the runtime's own, with host-transfer usage. ++ unsafe fn invalidate_rows( ++ &self, ++ functions: &DeviceFn, ++ y0: u32, ++ y1: u32, ++ ) -> Result<(), Error> { ++ const TILE_ROWS: u64 = 256; ++ let shared = self.device.shared(); ++ let Owner::Runtime { placement, .. } = &self.owner else { ++ return Ok(()); ++ }; ++ if placement.mapped.is_none() { ++ return Ok(()); ++ } ++ let subresource = vk::ImageSubresource2EXT::default().image_subresource(vk::ImageSubresource { ++ aspect_mask: vk::ImageAspectFlags::COLOR, ++ mip_level: 0, ++ array_layer: 0, ++ }); ++ let mut layout = vk::SubresourceLayout2EXT::default(); ++ // SAFETY: the caller's contract; the structures are the extension's. ++ unsafe { ++ (functions.get_image_subresource_layout2_ext)( ++ shared.vk().handle(), ++ self.raw.get(), ++ &subresource, ++ &mut layout, ++ ) ++ }; ++ let (pitch, whole) = (layout.subresource_layout.row_pitch, layout.subresource_layout.size); ++ let whole = whole.max(placement.size()); ++ let (first, last) = match pitch { ++ 0 => (0, whole), ++ _ => ( ++ u64::from(y0) / TILE_ROWS * TILE_ROWS * pitch, ++ (u64::from(y1).div_ceil(TILE_ROWS) * TILE_ROWS * pitch).min(whole), ++ ), ++ }; ++ // Vulkan bounds `nonCoherentAtomSize` by this, a power of two as it is. ++ let atom = 256; ++ let start = (placement.offset + first) / atom * atom; ++ let end = placement.offset + last; ++ let range = vk::MappedMemoryRange::default() ++ .memory(placement.memory) ++ .offset(start) ++ .size((end - start).next_multiple_of(atom)); ++ // SAFETY: the memory is mapped whole for its life, and the range lies in the image's. ++ unsafe { shared.vk().invalidate_mapped_memory_ranges(&[range]) }.map_err(vulkan) ++ } ++ + /// Copy the image's texels `copy` reaches into `bytes` on the host, laid out as `copy` writes + /// them to a buffer; the image [takes host copies](Self::takes_host_copies). Whether it did: + /// not while the image presents a stand-in. +diff --git a/crates/metal2vulkan-validation/src/runtime_lane.rs b/crates/metal2vulkan-validation/src/runtime_lane.rs +index 573b692e..aab89800 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane.rs +@@ -785,6 +785,8 @@ mod submit_per_import; + mod surface_aliases; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] + mod surface_present; // LOCAL (Atlas) ++#[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] ++mod surface_coherent; // LOCAL (Atlas) + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd)))] + mod surface_fills; + #[cfg(all(test, any(m2v_gpu, m2v_cpu_icd), target_os = "linux"))] +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/pitched_surfaces.rs b/crates/metal2vulkan-validation/src/runtime_lane/pitched_surfaces.rs +index cf7ba041..6ba4b66b 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/pitched_surfaces.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/pitched_surfaces.rs +@@ -363,6 +363,39 @@ pub(super) fn reading(device: &Device, pf: PixelFormat, fragment: &str) -> Rende + .unwrap() + } + ++/// LOCAL (Atlas): a `BGRA8Unorm` texture's first texels of `width` by `height`, sampled by a render ++/// encoder, as the display reads a frame: tight rows. ++pub(super) fn sampled_bgra8( ++ queue: &CommandQueue, ++ texture: &Texture, ++ width: u64, ++ height: u64, ++) -> Vec { ++ let device = queue.device(); ++ let layout = Layout::bgra8("sampled", width, height, width * 4); ++ let samples = reading(device, PixelFormat::BGRA8Unorm, "tob_sample"); ++ let nearest = sampler(device, SamplerMinMagFilter::Nearest); ++ drawn(queue, &layout, &samples, texture, Some(&nearest)) ++} ++ ++/// What [`sampled_bgra8`] makes, for [`drawn_small`]. ++pub(super) fn bgra8_sampling(device: &Device) -> (RenderPipelineState, SamplerState) { ++ let samples = reading(device, PixelFormat::BGRA8Unorm, "tob_sample"); ++ (samples, sampler(device, SamplerMinMagFilter::Nearest)) ++} ++ ++/// A draw that samples `texture` into a small target, run: a display's frame, with nothing read. ++pub(super) fn drawn_small( ++ queue: &CommandQueue, ++ samples: &RenderPipelineState, ++ nearest: &SamplerState, ++ texture: &Texture, ++) { ++ let layout = Layout::bgra8("small", 64, 64, 256); ++ let (command_buffer, _target) = draw_reading(queue, &layout, samples, texture, Some(nearest)); ++ run(&command_buffer); ++} ++ + /// What a draw of `pipeline` over a whole target of `layout`'s format and size reads from + /// `texture` (through `sampler`, where given): the target's texels, tight rows. + fn drawn( +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +new file mode 100644 +index 00000000..5ae338e6 +--- /dev/null ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -0,0 +1,812 @@ ++//! LOCAL (Atlas): a surface whose pages trap (`shadow_coherent.rs`). [`Fake`] stands for QEMU's ++//! marks over the pages, and each lane plays the guest through it: a read or a write of a marked ++//! page faults first, as the guest's CPU would. Beside them, a lane pins down what loses writes ++//! under `surface_lazy_present` alone, and the same steps under the traps. ++ ++use super::first_use_writes::CHECK_WAIT; ++use super::host_memory::{gpu_read, run, Mapping}; ++use super::host_rt_frames::{pass, surface_buffer}; ++use super::no_copy_textures::{rects, staged}; ++use super::{validated_device_with, Messages, ScratchCache}; ++use crate::gpu::TEST_DEVICE; ++use metal2vulkan::device::HostPageTexturePolicy; ++use super::pitched_surfaces::{bgra8_sampling, drawn_small, sampled_bgra8}; ++use metal2vulkan::metal::{PixelFormat, PrimitiveType, TextureUsage}; ++use metal2vulkan_runtime::shadow_coherent::{self, Pages}; ++use metal2vulkan_runtime::{ ++ live, Buffer, CommandBuffer, CommandQueue, Device, IoSurfacePlane, LoadAction, ++ RenderPassDescriptor, RenderPipelineState, ResourceOptions, ScissorRect, StoreAction, ++ Texture, TextureDescriptor, ++}; ++use std::collections::BTreeMap; ++use std::sync::{Arc, Mutex}; ++use std::time::{Duration, Instant}; ++ ++const WIDTH: u64 = 1024; ++const HEIGHT: u64 = 256; ++const ROW: u64 = 4096; ++const BEFORE: u8 = 0x55; ++const PAGE: u64 = 4096; ++const RED: [f32; 4] = [1.0, 0.0, 0.0, 1.0]; ++const GREEN: [f32; 4] = [0.0, 1.0, 0.0, 1.0]; ++const RED_TEXEL: [u8; 4] = [0, 0, 255, 255]; ++const GREEN_TEXEL: [u8; 4] = [0, 255, 0, 255]; ++const SOON: Duration = Duration::from_secs(3); ++ ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++enum Mark { ++ Held, ++ Armed, ++ Open, ++ Frozen, ++} ++ ++/// QEMU's page states, without the guest. ++#[derive(Default)] ++struct Fake { ++ pages: Mutex>, ++ /// Pages to answer as reached while their markers moved. ++ came: Mutex>, ++ /// Pages the next `arm` answers as reached, and leaves held. ++ refused: Mutex>, ++ /// Every page each `arm` and `hold` moved, for the counts of what moved. ++ armed: Mutex, ++} ++ ++impl Pages for Fake { ++ fn track(&self, start: u64, len: u64) -> bool { ++ let mut pages = self.pages.lock().unwrap(); ++ for page in (start..start + len).step_by(PAGE as usize) { ++ pages.entry(page).or_insert(Mark::Held); ++ } ++ true ++ } ++ ++ fn untrack(&self, start: u64) { ++ let _ = start; ++ self.pages.lock().unwrap().clear(); ++ } ++ ++ fn arm(&self, list: &[u64]) -> Vec { ++ let mut pages = self.pages.lock().unwrap(); ++ let mut refused = self.refused.lock().unwrap(); ++ for page in list { ++ if pages.get(page) == Some(&Mark::Held) && !refused.contains(page) { ++ pages.insert(*page, Mark::Armed); ++ *self.armed.lock().unwrap() += 1; ++ } ++ } ++ std::mem::take(&mut *refused) ++ } ++ ++ fn hold(&self, list: &[u64], open_too: bool) -> Vec { ++ let mut pages = self.pages.lock().unwrap(); ++ for page in list { ++ let at = pages.get(page).copied(); ++ if at == Some(Mark::Armed) || (open_too && at == Some(Mark::Open)) { ++ pages.insert(*page, Mark::Held); ++ } ++ } ++ std::mem::take(&mut *self.came.lock().unwrap()) ++ } ++ ++ fn open(&self, page: u64) { ++ let mut pages = self.pages.lock().unwrap(); ++ if pages.get(&page) == Some(&Mark::Held) { ++ pages.insert(page, Mark::Open); ++ } ++ } ++ ++ fn freeze(&self, list: &[u64]) { ++ let mut pages = self.pages.lock().unwrap(); ++ for page in list { ++ if pages.get(page) == Some(&Mark::Open) { ++ pages.insert(*page, Mark::Frozen); ++ } ++ } ++ } ++ ++ fn thaw(&self, list: &[u64]) { ++ let mut pages = self.pages.lock().unwrap(); ++ for page in list { ++ if pages.get(page) == Some(&Mark::Frozen) { ++ pages.insert(*page, Mark::Open); ++ } ++ } ++ } ++} ++ ++impl Fake { ++ fn mark(&self, page: u64) -> Option { ++ self.pages.lock().unwrap().get(&page).copied() ++ } ++ ++ fn armed_pages(&self) -> usize { ++ (self.pages.lock().unwrap().values()).filter(|mark| **mark == Mark::Armed).count() ++ } ++} ++ ++struct Case<'d> { ++ device: &'d Device, ++ queue: CommandQueue, ++ mapping: Mapping, ++ len: usize, ++ row: u64, ++ buffer: Buffer, ++ texture: Option, ++ opaque: RenderPipelineState, ++ rect: Buffer, ++ fake: Arc, ++} ++ ++impl Case<'_> { ++ fn base(&self) -> u64 { ++ self.mapping.pointer().as_ptr() as u64 ++ } ++ ++ fn range(&self) -> std::ops::Range { ++ self.base()..self.base() + self.len as u64 ++ } ++ ++ fn texture(&self) -> &Texture { ++ self.texture.as_ref().unwrap() ++ } ++ ++ fn draw(&self, color: [f32; 4], damage: Option) { ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ self.pass(&command_buffer, color, damage); ++ run(&command_buffer); ++ } ++ ++ fn pass(&self, command_buffer: &CommandBuffer, color: [f32; 4], damage: Option) { ++ let color = staged(self.device, &floats4(color)); ++ let rects = std::slice::from_ref(&self.rect); ++ pass(command_buffer, self.texture(), &self.opaque, rects, &color, damage); ++ } ++ ++ /// The pages' bytes at `x`, `y`, with no guest in the way: what an untrapped read would see. ++ fn behind(&self, x: u64, y: u64) -> [u8; 4] { ++ let at = (y * self.row + x * 4) as usize; ++ self.mapping.read(at..at + 4).try_into().unwrap() ++ } ++ ++ /// The guest reads the texel: a marked page stops it first. ++ fn read(&self, x: u64, y: u64) -> [u8; 4] { ++ let addr = self.base() + y * self.row + x * 4; ++ self.touch(addr, false); ++ self.behind(x, y) ++ } ++ ++ /// The guest writes the texel, as the page's marks let it. ++ fn write(&self, x: u64, y: u64, texel: [u8; 4]) { ++ let addr = self.base() + y * ROW + x * 4; ++ self.touch(addr, false); ++ self.touch(addr, true); ++ let at = (y * ROW + x * 4) as usize; ++ self.mapping.write(at, &texel); ++ } ++ ++ fn touch(&self, addr: u64, write: bool) { ++ let page = addr & !(PAGE - 1); ++ match (self.fake.mark(page), write) { ++ (Some(Mark::Armed | Mark::Frozen), false) => shadow_coherent::fault(addr, false), ++ (Some(Mark::Armed | Mark::Frozen | Mark::Held), true) => { ++ shadow_coherent::fault(addr, true); ++ } ++ _ => {} ++ } ++ } ++ ++ /// What the texture's image holds at `x`, `y`, its alpha gone: a pass that blends nothing over ++ /// every texel loads the image and stores it, and the guest's read then fetches that. ++ fn through_image(&self, x: u64, y: u64) -> [u8; 4] { ++ let color = staged(self.device, &floats4([0.0; 4])); ++ let rects = std::slice::from_ref(&self.rect); ++ let blending = rects_blending(self.device); ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ pass(&command_buffer, self.texture(), &blending, rects, &color, None); ++ run(&command_buffer); ++ self.read(x, y) ++ } ++ ++ /// A count of the surface's report line. ++ fn count(&self, name: &str) -> u64 { ++ let report = shadow_coherent::report("lane"); ++ let key = format!("\"{name}\":"); ++ let line = report.lines().last().unwrap_or_default(); ++ let at = line.find(&key).map_or(line.len(), |at| at + key.len()); ++ let digits: String = line[at..].chars().take_while(char::is_ascii_digit).collect(); ++ digits.parse().unwrap_or(0) ++ } ++} ++ ++fn rects_blending(device: &Device) -> RenderPipelineState { ++ rects(device, PixelFormat::BGRA8Unorm, true) ++} ++ ++fn floats4(values: [f32; 4]) -> Vec { ++ values.iter().flat_map(|v| v.to_ne_bytes()).collect() ++} ++ ++fn switches(coherent: bool) { ++ live::set("surface_lazy", Some("1")); ++ live::set("surface_lazy_min_texels", Some("1")); ++ live::set("surface_lazy_resync_ms", Some("0")); ++ live::set("surface_lazy_scoped", Some("1")); ++ live::set("surface_coherent", coherent.then_some("1")); ++ live::set("surface_coherent_hold_s", Some("1")); ++ live::set("surface_lazy_present", (!coherent).then_some("1")); ++ live::set("surface_lazy_present_all", (!coherent).then_some("1")); ++ live::set("surface_lazy_present_max_ms", Some("10")); ++} ++ ++fn scissor(x: u64, y: u64, width: u64, height: u64) -> Option { ++ Some(ScissorRect { x, y, width, height }) ++} ++ ++/// Leaves no surface or switch behind, whatever a lane did. ++struct Clean; ++ ++impl Drop for Clean { ++ fn drop(&mut self) { ++ shadow_coherent::reset(); ++ shadow_coherent::set_pages(None); ++ for switch in [ ++ "surface_lazy", ++ "surface_lazy_min_texels", ++ "surface_lazy_resync_ms", ++ "surface_lazy_scoped", ++ "surface_coherent", ++ "surface_coherent_hold_s", ++ "surface_lazy_present", ++ "surface_lazy_present_all", ++ "surface_lazy_present_max_ms", ++ ] { ++ live::set(switch, None); ++ } ++ } ++} ++ ++/// `test` on a surface whose bytes are [`BEFORE`], marks in place where `coherent`. ++fn on_surface(coherent: bool, test: impl FnOnce(Case<'_>)) { ++ on_surface_of([WIDTH, HEIGHT, ROW], coherent, true, test); ++} ++ ++fn on_surface_of( ++ [width, height, row]: [u64; 3], ++ coherent: bool, ++ lazy: bool, ++ test: impl FnOnce(Case<'_>), ++) { ++ let fake = Arc::new(Fake::default()); ++ shadow_coherent::set_pages(coherent.then(|| fake.clone() as Arc)); ++ switches(coherent); ++ live::set("surface_lazy", lazy.then_some("1")); ++ let _clean = Clean; ++ let cache = ScratchCache::new().unwrap(); ++ let messages = Messages::default(); ++ { ++ let device = validated_device_with(TEST_DEVICE, &messages, cache.path(), |o| { ++ o.policy.host_page_textures = HostPageTexturePolicy::InPlace; ++ }) ++ .unwrap(); ++ assert!(device.settle_in_place_checks(CHECK_WAIT)); ++ let len = ((row * height) as usize).next_multiple_of(16384); ++ let mapping = Mapping::anonymous(len); ++ mapping.fill(BEFORE); ++ let buffer = surface_buffer(&device, &mapping, len); ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, width, height, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = buffer.resource_options(); ++ let texture = buffer ++ .new_texture_with_iosurface_plane(&d, IoSurfacePlane::new(0, row, false)) ++ .unwrap(); ++ let rect = [-1.0f32, -1.0, 1.0, 1.0].iter().flat_map(|v| v.to_ne_bytes()); ++ test(Case { ++ device: &device, ++ queue: device.new_command_queue().unwrap(), ++ len, ++ row, ++ buffer, ++ texture: Some(texture), ++ opaque: rects(&device, PixelFormat::BGRA8Unorm, false), ++ rect: staged(&device, &rect.collect::>()), ++ mapping, ++ fake, ++ }); ++ } ++ let reported = messages.take(); ++ assert!(reported.is_empty(), "the layer reported: {reported:#?}"); ++} ++ ++#[test] ++fn surface_coherent_a_stale_page_traps_and_reads_the_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "nothing copied"); ++ assert!(case.fake.armed_pages() > 0, "armed at completion"); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ assert_eq!(case.behind(WIDTH - 1, HEIGHT - 1), RED_TEXEL, "one trap brings all of it"); ++ assert_eq!(case.fake.armed_pages(), 0); ++ assert_eq!(case.count("traps"), 1); ++ assert_eq!(case.count("regions"), 1); ++ assert_eq!(case.count("flushes"), 0, "no GPU work"); ++ }); ++} ++ ++#[test] ++fn surface_coherent_no_reader_copies_nothing() { ++ on_surface(true, |case| { ++ for frame in 0..20 { ++ case.draw(if frame % 2 == 0 { RED } else { GREEN }, None); ++ } ++ assert_eq!(case.behind(5, 5), [BEFORE; 4]); ++ assert_eq!(case.count("traps"), 0); ++ assert_eq!(case.count("flushes"), 0); ++ assert_eq!(case.count("copy_ins"), 1, "the first use's alone"); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_partial_region_moves_that_alone() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ let bytes = case.count("bytes_detiled"); ++ case.draw(GREEN, scissor(100, 100, 16, 8)); ++ assert_eq!(case.fake.armed_pages(), 8, "the rectangle's rows' pages"); ++ assert_eq!(case.behind(100, 100), RED_TEXEL); ++ assert_eq!(case.read(104, 104), GREEN_TEXEL); ++ assert_eq!(case.count("bytes_detiled") - bytes, 16 * 8 * 4); ++ assert_eq!(case.behind(99, 100), RED_TEXEL); ++ assert_eq!(case.behind(116, 100), RED_TEXEL); ++ }); ++} ++ ++/// Each sync reads the image through the host's caches, so the next reads what the GPU wrote since ++/// or a line kept from the last: a read after every write, in the worst order, reads the write. ++#[test] ++fn surface_coherent_no_sync_reads_a_line_of_an_earlier_one() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_demote_pct", Some("100000")); ++ live::set("surface_coherent_demote_hz", Some("1000000")); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ for frame in 0..600u64 { ++ let (color, texel) = match frame % 2 { ++ 0 => (GREEN, GREEN_TEXEL), ++ _ => (RED, RED_TEXEL), ++ }; ++ // A rectangle moved each frame: small, whole-row and tile-crossing ones. ++ let (x, y) = (frame * 37 % (WIDTH - 64), frame * 13 % (HEIGHT - 40)); ++ let (width, height) = (1 + frame * 7 % 64, 1 + frame * 3 % 40); ++ case.draw(color, scissor(x, y, width, height)); ++ assert_eq!(case.read(x, y), texel, "frame {frame}'s first texel"); ++ let (last_x, last_y) = (x + width - 1, y + height - 1); ++ assert_eq!(case.read(last_x, last_y), texel, "frame {frame}'s last texel"); ++ } ++ live::set("surface_coherent_demote_pct", None); ++ live::set("surface_coherent_demote_hz", None); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_screenshot_reads_every_texel_at_its_truth() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ for at in 0..8 { ++ case.draw(GREEN, scissor(at * 100, at * 30, 40, 10)); ++ } ++ for y in 0..HEIGHT { ++ for x in (0..WIDTH).step_by(7) { ++ let inside = (0..8).any(|at| { ++ (at * 100..at * 100 + 40).contains(&x) && (at * 30..at * 30 + 10).contains(&y) ++ }); ++ let want = if inside { GREEN_TEXEL } else { RED_TEXEL }; ++ assert_eq!(case.read(x, y), want, "at {x}, {y}"); ++ } ++ } ++ assert!(case.count("traps") <= 2, "{} traps", case.count("traps")); ++ assert_eq!(case.fake.armed_pages(), 0); ++ }); ++} ++ ++#[test] ++fn surface_coherent_cpu_write_then_gpu_read() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.write(7, 7, [1, 2, 3, 4]); ++ assert_eq!(case.fake.mark(case.base() + 7 * PAGE), Some(Mark::Open)); ++ assert_eq!(case.through_image(7, 7), [1, 2, 3, 0], "the image took the write"); ++ assert_eq!(case.through_image(8, 7), [0, 0, 255, 0], "and the page's other texels hold"); ++ // A later write opens it again. ++ case.write(7, 7, [5, 6, 7, 8]); ++ assert_eq!(case.through_image(7, 7), [5, 6, 7, 0]); ++ // Its rectangle alone: the first use's and the pages' own rows. ++ assert_eq!(case.count("copy_ins"), 3); ++ }); ++} ++ ++#[test] ++fn surface_coherent_gpu_write_then_cpu_read_then_gpu_again() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(1, 1), RED_TEXEL); ++ case.draw(GREEN, scissor(0, 0, 8, 8)); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL); ++ case.draw(RED, scissor(0, 0, 4, 4)); ++ assert_eq!(case.read(1, 1), RED_TEXEL); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL); ++ }); ++} ++ ++/// A write before the command buffer that holds a GPU write completes: the merge keeps it. ++#[test] ++fn surface_coherent_a_cpu_write_outlasts_a_write_in_flight() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, GREEN, scissor(0, 7, 64, 1)); ++ case.write(9, 7, [1, 2, 3, 4]); ++ assert_eq!(case.fake.mark(case.base() + 7 * PAGE), Some(Mark::Open)); ++ run(&command_buffer); ++ assert_eq!(case.fake.mark(case.base() + 7 * PAGE), Some(Mark::Frozen), "merged at a touch"); ++ assert_eq!(case.read(9, 7), [1, 2, 3, 4], "the CPU's texel"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the GPU's around it"); ++ assert_eq!(case.count("merges"), 1); ++ }); ++} ++ ++/// A write followed by a GPU reader of the bytes through another import: no flush over it. ++#[test] ++fn surface_coherent_a_cpu_write_outlives_a_copy_back() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ case.draw(GREEN, scissor(0, 0, 64, 16)); ++ case.write(9, 7, [1, 2, 3, 4]); ++ let read = gpu_read(&case.queue, &case.other_import(), 7 * ROW + 32..7 * ROW + 40); ++ assert_eq!(read[..4], GREEN_TEXEL); ++ assert_eq!(read[4..], [1, 2, 3, 4]); ++ assert_eq!(case.count("overwrite_risk"), 0); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_page_reached_as_its_markers_moved_goes_back_to_the_image() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let page = case.base() + 3 * PAGE; ++ case.fake.came.lock().unwrap().push(page); ++ case.read(0, 0); ++ // The guest's write landed unseen in the window. ++ case.mapping.write((3 * ROW) as usize, &[9, 9, 9, 9]); ++ assert_eq!(case.through_image(0, 3), [9, 9, 9, 0]); ++ assert_eq!(case.count("came"), 1); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_page_refused_its_arm_syncs_at_once_and_goes_back_to_the_image() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ case.fake.refused.lock().unwrap().push(case.base() + 3 * PAGE); ++ // The guest's write landed unseen in the window, beside the rectangle. ++ case.mapping.write((3 * ROW + 400) as usize, &[9, 9, 9, 9]); ++ case.draw(GREEN, scissor(0, 0, 16, 16)); ++ assert_eq!(case.behind(0, 3), GREEN_TEXEL, "no trap comes for that page"); ++ assert_eq!(case.fake.armed_pages(), 0, "the rectangle went with it"); ++ assert_eq!(case.count("came"), 1); ++ assert_eq!(case.through_image(100, 3), [9, 9, 9, 0]); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_reader_through_another_import_reads_the_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(gpu_read(&case.queue, &case.other_import(), 0..4), RED_TEXEL); ++ assert_eq!(case.count("flushes"), 0, "done work goes on the CPU"); ++ assert_eq!(case.fake.armed_pages(), 0); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_later_encoder_of_the_writer_reads_the_writes() { ++ on_surface(true, |case| { ++ let other = case.other_import(); ++ let copy = case.device.new_buffer(4, ResourceOptions::shared()).unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, RED, None); ++ let mut blit = command_buffer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&other, 0, ©, 0, 4).unwrap(); ++ blit.end_encoding().unwrap(); ++ run(&command_buffer); ++ // SAFETY: a `Shared` buffer no command uses now. ++ let read = unsafe { std::slice::from_raw_parts(copy.contents().unwrap().as_ptr(), 4) }; ++ assert_eq!(read, RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_passes_of_one_command_buffer_stay_ahead() { ++ on_surface(true, |case| { ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, RED, None); ++ case.pass(&command_buffer, GREEN, scissor(0, 0, WIDTH / 2, HEIGHT)); ++ run(&command_buffer); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "no copy at the second encoder"); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL); ++ assert_eq!(case.read(WIDTH - 5, 5), RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_the_display_samples_the_image() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let other = { ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::SHADER_READ; ++ d.resource_options = case.buffer.resource_options(); ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ case.other_import().new_texture_with_iosurface_plane(&d, plane).unwrap() ++ }; ++ let shown = other.surface_presented().expect("the surface's own texture"); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4]); ++ let read = sampled_bgra8(&case.queue, &shown, WIDTH, HEIGHT); ++ assert_eq!(read[..4], RED_TEXEL); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "and it copied nothing"); ++ // The next writer still takes the image. ++ let copies = case.count("copy_ins"); ++ case.draw(GREEN, scissor(0, 0, 1, 1)); ++ assert_eq!(case.count("copy_ins"), copies, "the display cost it no standing"); ++ let read = sampled_bgra8(&case.queue, &shown, WIDTH, HEIGHT); ++ assert_eq!(read[..4], GREEN_TEXEL); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "still"); ++ assert_eq!(case.read(0, 0), GREEN_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_frequent_reader_goes_back_to_copies() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_demote_window", Some("8")); ++ live::set("surface_coherent_hold_s", Some("60")); ++ for _ in 0..12 { ++ case.draw(RED, None); ++ case.read(0, 0); ++ case.draw(GREEN, None); ++ case.read(0, 0); ++ } ++ assert!(case.count("demotions") >= 1); ++ // Copied back per frame now: no trap, no armed page. ++ case.draw(RED, None); ++ assert_eq!(case.fake.armed_pages(), 0); ++ assert_eq!(case.behind(5, 5), RED_TEXEL); ++ live::set("surface_coherent_demote_window", None); ++ }); ++} ++ ++#[test] ++fn surface_coherent_unmap_during_deferred_syncs_and_lets_go() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4]); ++ assert!(shadow_coherent::unmapping(case.range()), "no work runs"); ++ assert_eq!(case.behind(5, 5), RED_TEXEL, "landed while the pages are the guest's"); ++ assert_eq!(case.fake.armed_pages(), 0); ++ shadow_coherent::unmapped(case.range()); ++ assert_eq!(shadow_coherent::report("after").matches("\"tracked\":true").count(), 0); ++ }); ++} ++ ++#[test] ++fn surface_coherent_unmap_with_work_running_is_busy() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let hold = case.device.new_shared_event().unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ command_buffer.encode_wait_for_event((&hold).into(), 1).unwrap(); ++ case.pass(&command_buffer, GREEN, None); ++ command_buffer.commit().unwrap(); ++ let start = Instant::now(); ++ assert!(!shadow_coherent::unmapping(case.range()), "work runs"); ++ assert_eq!(case.behind(5, 5), RED_TEXEL, "what is done landed"); ++ assert!(start.elapsed() < Duration::from_millis(500), "nothing waited"); ++ hold.set_signaled_value(1); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert!(shadow_coherent::unmapping(case.range())); ++ assert_eq!(case.behind(5, 5), GREEN_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_remap_over_deferred_writes_lands_them_first() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let start = std::ptr::NonNull::new(case.mapping.pointer().as_ptr().cast()).unwrap(); ++ case.device.host_range_remapped(start, case.len as u64); ++ assert_eq!(case.behind(5, 5), RED_TEXEL); ++ assert_eq!(case.fake.armed_pages(), 0, "and it stops trapping"); ++ // On as a plain lazy surface: nothing left ahead. ++ case.draw(GREEN, None); ++ assert_eq!(case.behind(5, 5), GREEN_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_reset_writes_nothing() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ shadow_coherent::reset(); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4]); ++ assert_eq!(shadow_coherent::report("after").matches("\"tracked\":true").count(), 0); ++ }); ++} ++ ++#[test] ++fn surface_coherent_switched_off_lands_everything() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ live::set("surface_coherent", None); ++ let start = Instant::now(); ++ // The report waits for a sync under way, as an unmap does. ++ while shadow_coherent::report("off").contains("\"tracked\":true") { ++ assert!(start.elapsed() < SOON, "the clock lands it"); ++ std::thread::sleep(Duration::from_millis(2)); ++ } ++ assert_eq!(case.behind(5, 5), RED_TEXEL); ++ assert_eq!(case.behind(WIDTH - 1, HEIGHT - 1), RED_TEXEL); ++ }); ++} ++ ++/// A texture of a mapped surface that goes while ahead: `surface_lazy_present` drops what it left ++/// (`shadow_lazy::Lazy::flush`), and nothing writes those pages again. ++#[test] ++fn surface_lazy_present_loses_writes_of_a_released_texture() { ++ on_surface(false, |mut case| { ++ case.draw(RED, None); ++ case.texture = None; ++ std::thread::sleep(Duration::from_millis(100)); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "the writes went with the texture"); ++ }); ++} ++ ++#[test] ++fn surface_coherent_keeps_writes_of_a_released_texture() { ++ on_surface(true, |mut case| { ++ case.draw(RED, None); ++ case.texture = None; ++ std::thread::sleep(Duration::from_millis(100)); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ }); ++} ++ ++#[test] ++fn surface_coherent_copies_back_no_texel_it_does_not_hold() { ++ on_surface(true, |case| { ++ case.draw_replacing(RED, scissor(0, 0, 8, 8)); ++ case.draw_replacing(GREEN, scissor(WIDTH - 8, HEIGHT - 8, 8, 8)); ++ gpu_read(&case.queue, &case.other_import(), 0..4); ++ assert_eq!(case.read(WIDTH / 2, HEIGHT / 2), [BEFORE; 4]); ++ assert_eq!(case.read(1, 1), RED_TEXEL); ++ assert_eq!(case.read(WIDTH - 1, HEIGHT - 1), GREEN_TEXEL); ++ }); ++} ++ ++/// With the switch off and marks installed, a surface runs as `surface_lazy` alone. ++/// What a frame costs at 3200x1800, with the display sampling the surface and, where `read`, the ++/// guest's CPU reading what changed: eager copies, `surface_lazy_present_all`, and this. ++#[test] ++fn surface_coherent_cost_of_a_frame() { ++ const W: u64 = 3200; ++ const H: u64 = 1800; ++ const R: u64 = 12800; ++ const WARM: usize = 30; ++ const FRAMES: usize = 120; ++ #[derive(Clone, Copy, Debug, PartialEq)] ++ enum Mode { ++ Eager, ++ LazyPresent, ++ Coherent, ++ } ++ fn median(mut values: Vec) -> f64 { ++ values.sort_by(f64::total_cmp); ++ values[values.len() / 2] ++ } ++ for (name, damage, read) in [ ++ ("1x1", scissor(1000, 700, 1, 1), false), ++ ("400x300", scissor(1000, 700, 400, 300), false), ++ ("1600x900", scissor(400, 400, 1600, 900), false), ++ ("whole", None, false), ++ ("400x300 cpu read", scissor(1000, 700, 400, 300), true), ++ ] { ++ for mode in [Mode::Eager, Mode::LazyPresent, Mode::Coherent] { ++ on_surface_of([W, H, R], mode == Mode::Coherent, mode != Mode::Eager, |case| { ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, W, H, false); ++ d.usage = TextureUsage::SHADER_READ; ++ d.resource_options = case.buffer.resource_options(); ++ let plane = IoSurfacePlane::new(0, R, false); ++ let other = case.other_import().new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ let (samples, nearest) = bgra8_sampling(case.device); ++ let [mut drew, mut shown, mut gpu] = [Vec::new(), Vec::new(), Vec::new()]; ++ for i in 0..WARM + FRAMES { ++ let color = if i % 2 == 0 { RED } else { GREEN }; ++ let start = Instant::now(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, color, damage); ++ run(&command_buffer); ++ let middle = Instant::now(); ++ let display = other.surface_presented().unwrap_or_else(|| other.clone()); ++ drawn_small(&case.queue, &samples, &nearest, &display); ++ if read { ++ let [x, y] = damage.map_or([0, 0], |d| [d.x, d.y]); ++ assert_eq!(case.read(x, y), if color == RED { RED_TEXEL } else { GREEN_TEXEL }); ++ } ++ if i >= WARM { ++ drew.push((middle - start).as_secs_f64() * 1e3); ++ shown.push(middle.elapsed().as_secs_f64() * 1e3); ++ let ends = command_buffer.gpu_end_time().unwrap(); ++ gpu.push((ends - command_buffer.gpu_start_time().unwrap()) * 1e3); ++ } ++ } ++ let [drew, shown, gpu] = [median(drew), median(shown), median(gpu)]; ++ eprintln!("cost {name}: {mode:?} draw {drew:.2} ms (gpu {gpu:.2}), then {shown:.2} ms"); ++ if mode == Mode::Coherent { ++ let report = shadow_coherent::report("cost"); ++ eprintln!("cost {name}: {}", report.lines().last().unwrap_or_default()); ++ } ++ }); ++ } ++ } ++} ++ ++#[test] ++fn surface_coherent_off_touches_no_page() { ++ on_surface(true, |case| { ++ live::set("surface_coherent", None); ++ let buffer = case.other_import(); ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = buffer.resource_options(); ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ let texture = buffer.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ let color = staged(case.device, &floats4(RED)); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ pass(&command_buffer, &texture, &case.opaque, std::slice::from_ref(&case.rect), &color, None); ++ run(&command_buffer); ++ assert_eq!(case.behind(5, 5), RED_TEXEL); ++ assert!(case.fake.pages.lock().unwrap().is_empty()); ++ }); ++} ++ ++impl Case<'_> { ++ fn other_import(&self) -> Buffer { ++ surface_buffer(self.device, &self.mapping, self.len) ++ } ++ ++ /// One command buffer of a pass that loads nothing, as a pass that replaces what it draws does. ++ fn draw_replacing(&self, color: [f32; 4], damage: Option) { ++ let color = staged(self.device, &floats4(color)); ++ let mut descriptor = RenderPassDescriptor::new(); ++ let attachment = &mut descriptor.color_attachments[0]; ++ attachment.texture = self.texture.clone(); ++ attachment.load_action = LoadAction::DontCare; ++ attachment.store_action = StoreAction::Store; ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ let mut encoder = command_buffer.render_command_encoder(&descriptor).unwrap(); ++ encoder.set_render_pipeline_state(&self.opaque).unwrap(); ++ encoder.set_fragment_buffer(Some(&color), 0, 0).unwrap(); ++ if let Some(damage) = damage { ++ encoder.set_scissor_rect(damage).unwrap(); ++ } ++ encoder.set_vertex_buffer(Some(&self.rect), 0, 0).unwrap(); ++ encoder.draw_primitives(PrimitiveType::Triangle, 0, 6, 1, 0).unwrap(); ++ encoder.end_encoding().unwrap(); ++ run(&command_buffer); ++ } ++} diff --git a/patches/metal2vulkan/0014-a-rehearsal-of-a-coherent-surface-and-a-.patch b/patches/metal2vulkan/0014-a-rehearsal-of-a-coherent-surface-and-a-.patch new file mode 100644 index 0000000..301c7ef --- /dev/null +++ b/patches/metal2vulkan/0014-a-rehearsal-of-a-coherent-surface-and-a-.patch @@ -0,0 +1,115 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 02:23:00 +0000 +Subject: [PATCH 14/34] a rehearsal of a coherent surface, and a + merge a device cannot lose + +Rehearsal runs a surface through every step a texture's command buffers make with +no GPU, for a boot test on a guest that runs no Metal. A merge changes a texel by +compare and swap: a device's DMA does not stop at a frozen page. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/shadow_coherent.rs | 67 ++++++++++++++++--- + 1 file changed, 58 insertions(+), 9 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 22172ac7..5ff37213 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -53,10 +53,40 @@ use super::shadow::bounding; + use super::shadow_lazy::Lazy; + use std::collections::{BTreeMap, HashSet}; + use std::ops::Range; +-use std::sync::atomic::{AtomicU64, Ordering}; ++use std::sync::atomic::{AtomicU32, AtomicU64, Ordering}; + use std::sync::{Arc, Condvar, Mutex, MutexGuard, OnceLock, RwLock, Weak}; + use std::time::{Duration, Instant}; + ++/// A surface with no GPU: every step a texture's command buffers make, for a boot test on a guest ++/// that can run no Metal (`rvg_coherent_rehearse`). `read` stands in for the image. ++pub struct Rehearsal(Arc); ++ ++impl Rehearsal { ++ pub fn new(layout: Layout, read: Reader) -> Rehearsal { ++ Rehearsal(Coherent::new(layout, read, Weak::new())) ++ } ++ ++ /// Whether the surface traps. ++ pub fn track(&self) -> bool { ++ self.0.tracks() ++ } ++ ++ /// One command buffer `id` over `area`, its running as long as `running`, and with `take` the ++ /// next use of the image: whether the surface still lives. ++ pub fn frame(&self, id: u64, area: [u32; 4], running: Duration, take: bool) -> bool { ++ if self.0.defers(id, Some(area)) { ++ self.0.committed(id); ++ std::thread::sleep(running); ++ self.0.completing(id); ++ } ++ if let Some(stale) = take.then(|| self.0.taking(false)).flatten() { ++ self.0.copied_in(texels(stale)); ++ } ++ self.0.presented(); ++ self.0.is_live() ++ } ++} ++ + /// The live switch. + const SWITCH: &str = "surface_coherent"; + /// The display samples a surface's image instead of its pages. +@@ -127,17 +157,17 @@ fn knob(name: &str, default: u64) -> u64 { + + /// Copies an image's rectangle `[x0, y0, x1, y1)` to bytes laid out in rows of the given texels, + /// on the host: whether it did. +-pub(crate) type Reader = Box bool + Send + Sync>; ++pub type Reader = Box bool + Send + Sync>; + + /// A surface's texels in host pages. + #[derive(Clone, Copy, Debug)] +-pub(crate) struct Layout { ++pub struct Layout { + /// The host address of the first texel's byte. +- pub(crate) base: u64, +- pub(crate) row: u64, +- pub(crate) texel: u64, +- pub(crate) width: u32, +- pub(crate) height: u32, ++ pub base: u64, ++ pub row: u64, ++ pub texel: u64, ++ pub width: u32, ++ pub height: u32, + } + + impl Layout { +@@ -625,8 +655,27 @@ impl Coherent { + match state.snapshots.get_mut(&page) { + Some(snapshot) => { + let was = &mut snapshot[offset..(offset + texel).min(PAGE as usize)]; +- if bytes[..was.len()] == *was { ++ // A device's DMA goes on through a frozen page: a texel changes as a ++ // whole here, or stays what it wrote. ++ let same = if let (0, Ok(new), Ok(old)) = ( ++ at % 4 + (texel as u64 ^ 4), ++ <[u8; 4]>::try_from(&source[..was.len()]), ++ <[u8; 4]>::try_from(&*was), ++ ) { ++ let word = (at as *mut u32).cast::(); ++ (*word).compare_exchange( ++ u32::from_ne_bytes(old), ++ u32::from_ne_bytes(new), ++ Ordering::Relaxed, ++ Ordering::Relaxed, ++ ).is_ok() ++ } else if bytes[..was.len()] == *was { + bytes[..was.len()].copy_from_slice(&source[..was.len()]); ++ true ++ } else { ++ false ++ }; ++ if same { + was.copy_from_slice(&source[..was.len()]); + } + } diff --git a/patches/metal2vulkan/0015-a-coherent-surface-answers-to-what-it-co.patch b/patches/metal2vulkan/0015-a-coherent-surface-answers-to-what-it-co.patch new file mode 100644 index 0000000..d3a8561 --- /dev/null +++ b/patches/metal2vulkan/0015-a-coherent-surface-answers-to-what-it-co.patch @@ -0,0 +1,725 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 03:50:09 +0000 +Subject: [PATCH 15/34] a coherent surface answers to what it + costs + +A window a compositor samples syncs whole each frame with no trap: met_by's flush, on a surface +whose image the reader cannot sample. The demotion only saw traps and frames, and below 32 fps +never fired. A ledger of what a surface spends against the copies it replaces decides now, per +sync, with its cause counted; time outs double as they were meant to, which clearing eager_until +had undone. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/mod.rs | 2 +- + .../src/shadow_coherent.rs | 176 +++++++++++++----- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 14 +- + .../src/runtime_lane/surface_coherent.rs | 170 ++++++++++++++++- + 4 files changed, 302 insertions(+), 60 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index b2a5ab9e..4812e25c 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2526,7 +2526,7 @@ impl CommandBuffer { + crate::shadow_lazy::met_by(&summaries) + }; + for surface in met { +- surface.flush(); ++ surface.flush(crate::shadow_coherent::SyncBy::GpuReader); + } + } + crate::shadow_lazy::committed(id); +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 5ff37213..985ce191 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -37,10 +37,12 @@ + //! **The next command's copy in** takes the open pages' rectangle alone, and holds them first, so + //! a later write opens them again ([`Coherent::taking`]). + //! +-//! **Adaptive**: a surface synced in more than `surface_coherent_demote_pct` percent of a window of +-//! its frames, or more than `surface_coherent_demote_hz` times a second (a Simulator's per-frame +-//! reader), goes back to a copy back per frame for `surface_coherent_hold_s` seconds, doubling to +-//! [`HOLD_MAX_S`] while it keeps coming back. ++//! **Adaptive**: each surface keeps a [`Ledger`] of what trapping spent (arms, syncs, holds, traps ++//! and the GPU copies it still needed) against what the copies back it left undone would have. A ++//! surface that spends more over [`MIN_SYNCS`] syncs goes back to a copy back per encoder for ++//! `surface_coherent_hold_s` seconds, doubling to [`HOLD_MAX_S`] while it keeps coming back. ++//! Nothing here asks who read it: a window a compositor samples each frame costs a sync with no ++//! trap at all ([`SyncBy::GpuReader`]), and pays the same as a CPU reader that traps. + //! + //! **Unmap** ([`unmapping`]) syncs, while the pages are still the guest's, and lets go of what + //! pending writes hold; a surface with work running answers busy, as any surface with a command +@@ -101,6 +103,10 @@ const DEFAULT_WAIT_MS: u64 = 50; + const ENDED_KEPT: usize = 256; + /// A GPU copy of a surface, per MiB: 2.97 ms for 5.76 Mtexels of 4 bytes (`STUDY-virt.md`). + const DEFAULT_COPY_NS_PER_MIB: u64 = 135_000; ++/// Syncs a [`Ledger`] needs to judge: a surface's first frame pays for tracking it. ++const MIN_SYNCS: u32 = 2; ++/// A ledger halves this often, so a surface's record follows what it does now. ++const LEDGER_SPAN: Duration = Duration::from_secs(1); + + /// What marks a surface's pages. Addresses are the surface's own mapping's, whole pages, ascending, + /// one surface to a call; each call comes under the surface's lock, which [`fault`] takes. +@@ -251,6 +257,36 @@ struct Record { + area: [u32; 4], + } + ++/// What made a sync run. ++#[derive(Clone, Copy)] ++pub(crate) enum SyncBy { ++ /// The guest's CPU reached an armed page. ++ Trap, ++ /// A command buffer uses the bytes through another import (`shadow_lazy::met_by`). ++ GpuReader, ++ /// A CPU read through a texture over the pages (`shadow_lazy::cpu_reads`). ++ CpuRead, ++ /// A copy in reads the pages ([`Coherent::taking`]). ++ CopyIn, ++ /// An arm a vCPU beat. ++ Refused, ++ /// Demoted: its writes go now and at each completion. ++ Demotion, ++ /// The surface leaves. ++ Unmap, ++} ++ ++const SYNC_BY: [&str; 7] = ["trap", "gpu_read", "cpu_read", "copy_in", "refused", "demotion", "unmap"]; ++ ++/// What trapping spent lately against what the copies it replaced would have, in ns. ++#[derive(Default)] ++struct Ledger { ++ spent: u64, ++ saved: u64, ++ syncs: u32, ++ since: Option, ++} ++ + /// One surface's counts. + #[derive(Default)] + struct Counts { +@@ -263,7 +299,15 @@ struct Counts { + unique_pages: AtomicU64, + regions: AtomicU64, + bytes_detiled: AtomicU64, ++ /// Whole syncs: `copy_ns`, `hold_ns` and what lies between. + sync_ns: AtomicU64, ++ sync_by: [AtomicU64; SYNC_BY.len()], ++ /// The bytes' copy alone. ++ copy_ns: AtomicU64, ++ track_ns: AtomicU64, ++ /// The [`Ledger`]'s two sides, never halved. ++ spent_ns: AtomicU64, ++ saved_ns: AtomicU64, + trap_ns: AtomicU64, + trap_ns_max: AtomicU64, + timed: AtomicU64, +@@ -313,10 +357,12 @@ struct State { + wrote_at: Option, + /// Copies back per frame until then. + eager_until: Option, ++ /// The last time out's end: a surface back soon after it stays out longer. ++ out_until: Option, + hold_s: u64, +- window_start: Option, +- window_defers: u64, +- window_syncs: u64, ++ ledger: Ledger, ++ /// The last sync's time, which a trap's own time holds. ++ synced_ns: u64, + seen: HashSet, + } + +@@ -374,12 +420,16 @@ impl Coherent { + } + let range = self.range(); + let live = crate::live::on(SWITCH, false); ++ let start = Instant::now(); + let marked = pages().is_some_and(|p| live && p.track(range.start, range.end - range.start)); + if !marked { + state.closed = true; + return false; + } + state.tracked = true; ++ let spent = start.elapsed().as_nanos() as u64; ++ self.counts.track_ns.fetch_add(spent, Ordering::Relaxed); ++ self.charge(&mut state, spent); + all().write().unwrap().push(self.clone()); + true + } +@@ -401,14 +451,15 @@ impl Coherent { + return false; + } + let now = Instant::now(); +- state.window_defers += 1; +- self.roll_window(&mut state, now); + match state.eager_until { + Some(until) if now < until => { + self.counts.eager.fetch_add(1, Ordering::Relaxed); + return false; + } +- Some(_) => state.eager_until = None, ++ Some(_) => { ++ state.eager_until = None; ++ state.ledger = Ledger::default(); ++ } + None => {} + } + if state.wrote_at.is_some_and(|at| at.elapsed() < self.hold()) { +@@ -426,44 +477,58 @@ impl Coherent { + self.counts.defers.fetch_add(1, Ordering::Relaxed); + let bytes = texels(area) * self.layout.texel; + self.counts.deferred_bytes.fetch_add(bytes, Ordering::Relaxed); ++ let saved = self.by_gpu(bytes); ++ state.ledger.saved += saved; ++ self.counts.saved_ns.fetch_add(saved, Ordering::Relaxed); + true + } + +- /// The window of frames closes: too many syncs in it send the surface back to copies per frame. +- fn roll_window(&self, state: &mut State, now: Instant) { +- let start = *state.window_start.get_or_insert(now); +- let frames = knob("surface_coherent_demote_window", 32); +- let elapsed = now.duration_since(start); +- if state.window_defers < frames && elapsed < Duration::from_secs(1) { +- return; ++ /// What a GPU copy of `bytes` costs. ++ fn by_gpu(&self, bytes: u64) -> u64 { ++ let per_mib = knob("surface_coherent_copy_ns_per_mib", DEFAULT_COPY_NS_PER_MIB); ++ ((u128::from(bytes) * u128::from(per_mib)) >> 20) as u64 ++ } ++ ++ fn charge(&self, state: &mut State, ns: u64) { ++ state.ledger.spent += ns; ++ self.counts.spent_ns.fetch_add(ns, Ordering::Relaxed); ++ } ++ ++ /// A sync ended: a surface that costs more than the copies it replaces goes back to them. ++ fn judge(&self, state: &mut State) { ++ let now = Instant::now(); ++ state.ledger.syncs += 1; ++ if now.duration_since(*state.ledger.since.get_or_insert(now)) >= LEDGER_SPAN { ++ state.ledger.spent /= 2; ++ state.ledger.saved /= 2; ++ state.ledger.since = Some(now); ++ state.seen.clear(); + } +- let pct = knob("surface_coherent_demote_pct", 50); +- let hz = knob("surface_coherent_demote_hz", 64); +- let by_frames = state.window_defers >= frames +- && state.window_syncs * 100 >= state.window_defers * pct; +- let by_rate = state.window_syncs as f64 >= hz as f64 * elapsed.as_secs_f64().max(0.25); +- if by_frames || by_rate { ++ let ledger = &state.ledger; ++ if ledger.syncs >= MIN_SYNCS ++ && ledger.spent > ledger.saved ++ && state.eager_until.is_none() ++ && knob("surface_coherent_demote", 1) != 0 ++ { + self.demote(state, now); + } +- state.window_start = Some(now); +- state.window_defers = 0; +- state.window_syncs = 0; +- state.seen.clear(); + } + + fn demote(&self, state: &mut State, now: Instant) { +- let last = state.eager_until; + let first = knob("surface_coherent_hold_s", 2).max(1); +- state.hold_s = match last { ++ state.hold_s = match state.out_until { + Some(until) if now < until + Duration::from_secs(state.hold_s.max(first)) => { + (state.hold_s * 2).clamp(first, HOLD_MAX_S) + } + _ => first, + }; +- state.eager_until = Some(now + Duration::from_secs(state.hold_s)); ++ let until = now + Duration::from_secs(state.hold_s); ++ state.eager_until = Some(until); ++ state.out_until = Some(until); ++ state.ledger = Ledger::default(); + self.counts.demotions.fetch_add(1, Ordering::Relaxed); + // What is pending goes now, and what is running at its completion. +- self.sync_done(state); ++ self.sync_done(state, SyncBy::Demotion); + } + + /// The commit of `id` returned: its record may go back by a copy ordered after it. +@@ -519,7 +584,7 @@ impl Coherent { + return; + }; + if state.eager_until.is_some_and(|until| Instant::now() < until) { +- self.sync_done(state); ++ self.sync_done(state, SyncBy::Demotion); + return; + } + if state.armed.iter().any(|armed| holds(*armed, record.area)) { +@@ -546,16 +611,17 @@ impl Coherent { + let spent = start.elapsed().as_nanos() as u64; + self.counts.arm_ns.fetch_add(spent, Ordering::Relaxed); + self.counts.arms.fetch_add(1, Ordering::Relaxed); ++ self.charge(state, spent); + // No trap comes for these: the sync is now, before anyone hears of this completion. A CPU + // write in the rectangle raced the GPU's; one beside it stays, and the image takes it. + if !refused.is_empty() { +- self.sync_done(state); ++ self.sync_done(state, SyncBy::Refused); + self.note_came(state, refused); + } + } + + /// Copy every done rectangle into the pages, and hold them. +- fn sync_done(&self, state: &mut State) { ++ fn sync_done(&self, state: &mut State, by: SyncBy) { + if state.done.is_empty() { + self.thaw_all(state); + return; +@@ -568,10 +634,12 @@ impl Coherent { + let mut list = Vec::new(); + for record in &done { + let area = record.area; ++ let copying = Instant::now(); + let moved = match state.snapshots.is_empty() { + true => self.detile(area), + false => self.merge(state, &pages_mark, area), + }; ++ self.counts.copy_ns.fetch_add(copying.elapsed().as_nanos() as u64, Ordering::Relaxed); + if !moved { + self.counts.sync_errors.fetch_add(1, Ordering::Relaxed); + eprintln!("surface_coherent: a sync of {area:?} failed"); +@@ -581,7 +649,7 @@ impl Coherent { + self.counts.bytes_detiled.fetch_add(bytes, Ordering::Relaxed); + list.extend(self.layout.pages_of(area)); + } +- state.window_syncs += 1; ++ self.counts.sync_by[by as usize].fetch_add(1, Ordering::Relaxed); + list.sort_unstable(); + list.dedup(); + self.thaw_all(state); +@@ -592,7 +660,10 @@ impl Coherent { + state.armed.clear(); + let spent = start.elapsed().as_nanos() as u64; + self.counts.sync_ns.fetch_add(spent, Ordering::Relaxed); ++ self.charge(state, spent); ++ state.synced_ns = spent; + self.settle_pending(state); ++ self.judge(state); + } + + /// A page the guest reached as its markers moved may hold a write: the image takes it again. +@@ -721,7 +792,7 @@ impl Coherent { + if state.seen.insert(page) { + self.counts.unique_pages.fetch_add(1, Ordering::Relaxed); + } +- self.sync_done(&mut state); ++ self.sync_done(&mut state, SyncBy::Trap); + let mark = pages(); + // Another vCPU's probe may be about to rerun on these pages: only the last of them, which + // lets it name its page, ends the probing. +@@ -750,7 +821,7 @@ impl Coherent { + // copy back already on its way cannot merge. + let (settled, clear) = self.wait_settled(state, None); + state = settled; +- self.sync_done(&mut state); ++ self.sync_done(&mut state, SyncBy::Trap); + if !clear { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } +@@ -797,7 +868,7 @@ impl Coherent { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } + } +- self.sync_done(&mut state); ++ self.sync_done(&mut state, SyncBy::CopyIn); + let list: Vec = state.snapshots.keys().copied().collect(); + if let Some(mark) = pages() { + let came = mark.hold(&list, true); +@@ -815,9 +886,9 @@ impl Coherent { + /// A GPU copy back of what ran is about to record: everything done went on the CPU, and what + /// is still running goes by the copy, up to the sequence given. `None` where nothing runs. The + /// lock is not held while it records. +- pub(crate) fn before_flush(&self) -> Option { ++ pub(crate) fn before_flush(&self, by: SyncBy) -> Option { + let mut state = self.lock(); +- self.sync_done(&mut state); ++ self.sync_done(&mut state, by); + if state.running.is_empty() { + return None; + } +@@ -825,7 +896,7 @@ impl Coherent { + let area = state.guest.unwrap_or(self.layout.whole()); + let (settled, clear) = self.wait_settled(state, Some(area)); + state = settled; +- self.sync_done(&mut state); ++ self.sync_done(&mut state, by); + if state.running.is_empty() { + return None; + } +@@ -842,6 +913,7 @@ impl Coherent { + pub(crate) fn flush_recorded(&self, bytes: u64) { + self.counts.flushes.fetch_add(1, Ordering::Relaxed); + self.counts.flushed_bytes.fetch_add(bytes, Ordering::Relaxed); ++ self.charge(&mut self.lock(), self.by_gpu(bytes)); + } + + /// The copy back was not made after all. +@@ -897,7 +969,7 @@ impl Coherent { + /// alone, what is running copied back by the clock. + pub(crate) fn abandon(&self) { + let mut state = self.lock(); +- self.sync_done(&mut state); ++ self.sync_done(&mut state, SyncBy::Unmap); + self.close(&mut state, true); + } + +@@ -928,7 +1000,7 @@ impl Coherent { + /// a surface with work running stays busy. + fn unmapping(&self) -> bool { + let mut state = self.lock(); +- self.sync_done(&mut state); ++ self.sync_done(&mut state, SyncBy::Unmap); + if !state.running.is_empty() || state.flushing != 0 { + return false; + } +@@ -961,6 +1033,7 @@ impl Coherent { + self.counts.copy_ins.fetch_add(1, Ordering::Relaxed); + let bytes = texels * self.layout.texel; + self.counts.copy_in_bytes.fetch_add(bytes, Ordering::Relaxed); ++ self.charge(&mut self.lock(), self.by_gpu(bytes)); + } + + /// A present: whether the display may sample the image instead of the pages. +@@ -977,13 +1050,15 @@ impl Coherent { + self.counts.trap_ns.fetch_add(ns, Ordering::Relaxed); + self.counts.timed.fetch_add(1, Ordering::Relaxed); + self.counts.trap_ns_max.fetch_max(ns, Ordering::Relaxed); ++ let mut state = self.lock(); ++ let synced = std::mem::take(&mut state.synced_ns); ++ self.charge(&mut state, ns.saturating_sub(synced)); + } + + fn line(&self, label: &str, state: &State) -> String { + let read = |count: &AtomicU64| count.load(Ordering::Relaxed); + let c = &self.counts; +- let copy = knob("surface_coherent_copy_ns_per_mib", DEFAULT_COPY_NS_PER_MIB) as f64; +- let by_gpu = |bytes: u64| bytes as f64 / (1 << 20) as f64 * copy; ++ let by_gpu = |bytes: u64| self.by_gpu(bytes) as f64; + let presents = read(&c.presents).max(1) as f64; + let trap = read(&c.trap_ns) as f64; + let sync = read(&c.sync_ns) as f64; +@@ -997,6 +1072,7 @@ impl Coherent { + "{{\"label\":\"{label}\",\"at\":\"{:#x}\",\"w\":{},\"h\":{},\"tracked\":{},\"closed\":{},\ + \"presents\":{},\"defers\":{},\"traps\":{},\"writes\":{},\"unique_pages\":{},\ + \"traps_per_s\":{:.2},\"regions\":{},\"bytes_detiled\":{},\"sync_ns\":{},\ ++ \"copy_ns\":{},\"track_ns\":{},\"sync_by\":{{{}}},\"spent_ns\":{},\"saved_ns\":{},\ + \"trap_ns\":{},\"trap_ns_max\":{},\"resumes\":{},\"arm_ns\":{},\"arms\":{},\ + \"hold_ns\":{},\"demotions\":{},\"eager_frames\":{},\"merges\":{},\"came\":{},\ + \"probes\":{},\"flushes\":{},\"flushed_bytes\":{},\"copy_ins\":{},\"copy_in_bytes\":{},\ +@@ -1017,6 +1093,14 @@ impl Coherent { + read(&c.regions), + read(&c.bytes_detiled), + read(&c.sync_ns), ++ read(&c.copy_ns), ++ read(&c.track_ns), ++ (SYNC_BY.iter().zip(&c.sync_by)) ++ .map(|(by, count)| format!("\"{by}\":{}", read(count))) ++ .collect::>() ++ .join(","), ++ read(&c.spent_ns), ++ read(&c.saved_ns), + read(&c.trap_ns), + read(&c.trap_ns_max), + read(&c.timed), +@@ -1036,7 +1120,7 @@ impl Coherent { + read(&c.sync_errors), + state.done.len(), + state.snapshots.len(), +- total < eager, ++ read(&c.spent_ns) < read(&c.saved_ns), + ) + } + } +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 64dd5ec4..d1c8eded 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -92,7 +92,7 @@ use super::device::Device; + use super::queue::CommandQueue; + use super::resource::ResourceKey; + use super::shadow::{bounding, CopyCause, ShadowCopies, ShadowCopy, Step}; +-use super::shadow_coherent::{Coherent, Layout, Reader}; ++use super::shadow_coherent::{Coherent, Layout, Reader, SyncBy}; + use super::tracker::{Summary, Use, UseRange}; + use crate::exit_guard; + use std::any::Any; +@@ -272,7 +272,7 @@ pub(crate) fn cpu_reads(bytes: impl Fn() -> Option>) { + }; + for lazy in surfaces() { + if lazy.over(&bytes) { +- if let Some(flush) = lazy.flush() { ++ if let Some(flush) = lazy.flush(SyncBy::CpuRead) { + let _ = flush.wait_until_completed(); + } + } +@@ -416,7 +416,7 @@ pub(crate) fn after_commit(id: u64) { + .compare_exchange(id, 0, Ordering::Relaxed, Ordering::Relaxed); + // A texture that came beside it while this command buffer was open reads the copy back. + if wants.is_ok() || (moved && lazy.beside_another()) { +- lazy.flush(); ++ lazy.flush(SyncBy::GpuReader); + } + } + } +@@ -501,7 +501,7 @@ fn flush_due(max: Duration) -> bool { + let Some(_section) = exit_guard::enter() else { + return false; + }; +- exit_guard::lowering_program_calls(|| lazy.flush()); ++ exit_guard::lowering_program_calls(|| lazy.flush(SyncBy::Unmap)); + } + } + any +@@ -939,18 +939,18 @@ impl Lazy { + return; + } + drop(pending); +- self.flush(); ++ self.flush(SyncBy::GpuReader); + } + + /// Copy what committed command buffers left ahead back, in a command buffer of its own, + /// committed before this returns: that command buffer, where one was made. Nothing is copied + /// over the pages of a texture nothing else holds (the module docs); a copy that cannot be + /// recorded stays for the next asker. +- pub(crate) fn flush(self: &Arc) -> Option { ++ pub(crate) fn flush(self: &Arc, by: SyncBy) -> Option { + // What is done goes on the CPU: a copy back is for what runs, which it is ordered after. + let coherent = self.coherent().cloned(); + let through = match &coherent { +- Some(coherent) => Some(coherent.before_flush()?), ++ Some(coherent) => Some(coherent.before_flush(by)?), + None => None, + }; + let (area, copies, holds) = { +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 5ae338e6..07688b3c 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -258,6 +258,8 @@ impl Drop for Clean { + "surface_lazy_scoped", + "surface_coherent", + "surface_coherent_hold_s", ++ "surface_coherent_demote", ++ "surface_coherent_copy_ns_per_mib", + "surface_lazy_present", + "surface_lazy_present_all", + "surface_lazy_present_max_ms", +@@ -369,8 +371,7 @@ fn surface_coherent_a_partial_region_moves_that_alone() { + #[test] + fn surface_coherent_no_sync_reads_a_line_of_an_earlier_one() { + on_surface(true, |case| { +- live::set("surface_coherent_demote_pct", Some("100000")); +- live::set("surface_coherent_demote_hz", Some("1000000")); ++ live::set("surface_coherent_demote", Some("0")); + case.draw(RED, None); + assert_eq!(case.read(0, 0), RED_TEXEL); + for frame in 0..600u64 { +@@ -386,8 +387,6 @@ fn surface_coherent_no_sync_reads_a_line_of_an_earlier_one() { + let (last_x, last_y) = (x + width - 1, y + height - 1); + assert_eq!(case.read(last_x, last_y), texel, "frame {frame}'s last texel"); + } +- live::set("surface_coherent_demote_pct", None); +- live::set("surface_coherent_demote_hz", None); + }); + } + +@@ -571,10 +570,15 @@ fn surface_coherent_the_display_samples_the_image() { + }); + } + ++/// What a copy back's GPU time is taken to be: 1 ns a MiB leaves nothing a surface's traps can pay ++/// for, so every sync costs more than it saved; a second a MiB, the other way. ++const COPIES_COST_NOTHING: &str = "1"; ++const COPIES_COST_A_SECOND: &str = "1000000000"; ++ + #[test] + fn surface_coherent_a_frequent_reader_goes_back_to_copies() { + on_surface(true, |case| { +- live::set("surface_coherent_demote_window", Some("8")); ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); + live::set("surface_coherent_hold_s", Some("60")); + for _ in 0..12 { + case.draw(RED, None); +@@ -587,7 +591,123 @@ fn surface_coherent_a_frequent_reader_goes_back_to_copies() { + case.draw(RED, None); + assert_eq!(case.fake.armed_pages(), 0); + assert_eq!(case.behind(5, 5), RED_TEXEL); +- live::set("surface_coherent_demote_window", None); ++ }); ++} ++ ++/// The signature of a window on real macOS: a sync of the whole surface per frame, with no trap, no ++/// copy back and no copy in, from a compositor sampling the surface through its own import. ++#[test] ++fn surface_coherent_a_compositor_syncs_every_frame_with_no_trap() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_demote", Some("0")); ++ let other = case.other_import(); ++ for frame in 0..8u64 { ++ let (color, texel) = if frame % 2 == 0 { (RED, RED_TEXEL) } else { (GREEN, GREEN_TEXEL) }; ++ case.draw(color, None); ++ assert_eq!(gpu_read(&case.queue, &other, 0..4), texel, "frame {frame}"); ++ } ++ assert_eq!(case.count("defers"), case.count("regions"), "a whole sync a defer"); ++ assert_eq!(case.count("gpu_read"), 8, "the reader's commit asked each of them"); ++ for none in ["traps", "flushes", "presents"] { ++ assert_eq!(case.count(none), 0, "{none}"); ++ } ++ assert!(case.count("copy_ins") <= 1, "the first use's alone"); ++ }); ++} ++ ++/// A window under a compositor, below the frame rate a window of frames would see, with no trap: ++/// out after two syncs, and what it shows stays right. ++#[test] ++fn surface_coherent_a_compositor_alone_demotes_a_slow_window() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); ++ live::set("surface_coherent_hold_s", Some("60")); ++ let other = case.other_import(); ++ for frame in 0..6u64 { ++ let (color, texel) = if frame % 2 == 0 { (RED, RED_TEXEL) } else { (GREEN, GREEN_TEXEL) }; ++ case.draw(color, None); ++ assert_eq!(gpu_read(&case.queue, &other, 0..4), texel, "frame {frame}"); ++ std::thread::sleep(Duration::from_millis(40)); ++ } ++ assert_eq!(case.count("traps"), 0); ++ assert_eq!(case.count("demotions"), 1, "by the ledger alone"); ++ assert!(case.count("regions") <= 3, "out quickly: {} syncs", case.count("regions")); ++ assert_eq!(case.fake.armed_pages(), 0, "eager from there"); ++ }); ++} ++ ++/// The same reader where the copies it replaces cost more: trapping pays, so it stays. ++#[test] ++fn surface_coherent_a_surface_that_pays_stays() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_A_SECOND)); ++ let other = case.other_import(); ++ for _ in 0..8 { ++ case.draw(RED, None); ++ gpu_read(&case.queue, &other, 0..4); ++ } ++ assert_eq!(case.count("demotions"), 0); ++ }); ++} ++ ++/// A display surface no one reads: defers with no sync, nothing to judge at any copy cost. ++#[test] ++fn surface_coherent_a_surface_nothing_reads_stays() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); ++ for _ in 0..64 { ++ case.draw(RED, None); ++ } ++ assert_eq!(case.count("demotions"), 0); ++ assert_eq!(case.count("regions"), 0); ++ assert!(case.fake.armed_pages() > 0, "still ahead"); ++ }); ++} ++ ++/// Back from a time out and still costing: the next one is longer. ++#[test] ++fn surface_coherent_a_surface_back_too_soon_stays_out_longer() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); ++ live::set("surface_coherent_hold_s", Some("1")); ++ let other = case.other_import(); ++ let frame = |color| { ++ case.draw(color, None); ++ gpu_read(&case.queue, &other, 0..4); ++ }; ++ (0..4).for_each(|_| frame(RED)); ++ assert_eq!(case.count("demotions"), 1); ++ std::thread::sleep(Duration::from_millis(1100)); ++ (0..4).for_each(|_| frame(GREEN)); ++ assert_eq!(case.count("demotions"), 2, "it came back, and it does not pay yet"); ++ std::thread::sleep(Duration::from_millis(1100)); ++ let eager = case.count("eager_frames"); ++ frame(RED); ++ assert_eq!(case.count("eager_frames"), eager + 1, "two seconds, this time"); ++ assert_eq!(case.count("demotions"), 2); ++ }); ++} ++ ++/// A demoted surface keeps every guarantee: a stale byte is no cheaper for being out. ++#[test] ++fn surface_coherent_a_demoted_surface_stays_correct() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); ++ live::set("surface_coherent_hold_s", Some("60")); ++ let other = case.other_import(); ++ for _ in 0..4 { ++ case.draw(RED, None); ++ gpu_read(&case.queue, &other, 0..4); ++ } ++ assert_eq!(case.count("demotions"), 1); ++ case.draw(GREEN, None); ++ assert_eq!(gpu_read(&case.queue, &other, 0..4), GREEN_TEXEL, "a GPU reader"); ++ assert_eq!(case.read(9, 9), GREEN_TEXEL, "the guest's CPU"); ++ case.write(0, 0, RED_TEXEL); ++ assert_eq!(case.read(0, 0), RED_TEXEL, "and its own write"); ++ case.draw(GREEN, scissor(4, 4, 4, 4)); ++ assert_eq!(case.read(0, 0), RED_TEXEL, "a later draw beside it keeps the write"); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL); + }); + } + +@@ -765,6 +885,44 @@ fn surface_coherent_cost_of_a_frame() { + } + } + ++/// A window: its whole surface drawn and a compositor sampling it through its own import, each ++/// frame. What a demoted surface costs beside a surface that never trapped: its image stays in ++/// memory placed for host reads. `--nocapture`. ++#[test] ++fn surface_coherent_cost_of_a_window() { ++ const W: u64 = 3160; ++ const H: u64 = 1680; ++ const R: u64 = 12640; ++ fn median(mut values: Vec) -> f64 { ++ values.sort_by(f64::total_cmp); ++ values[values.len() / 2] ++ } ++ for (name, coherent, demote) in [("off", false, "1"), ("stays", true, "0"), ("demoted", true, "1")] { ++ on_surface_of([W, H, R], coherent, coherent, |case| { ++ live::set("surface_coherent_demote", Some(demote)); ++ live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); ++ live::set("surface_coherent_hold_s", Some("60")); ++ let other = case.other_import(); ++ let [mut frames, mut gpu] = [Vec::new(), Vec::new()]; ++ for i in 0..150 { ++ let start = Instant::now(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, if i % 2 == 0 { RED } else { GREEN }, None); ++ run(&command_buffer); ++ gpu_read(&case.queue, &other, 0..4); ++ if i >= 30 { ++ frames.push(start.elapsed().as_secs_f64() * 1e3); ++ let ends = command_buffer.gpu_end_time().unwrap(); ++ gpu.push((ends - command_buffer.gpu_start_time().unwrap()) * 1e3); ++ } ++ } ++ let [frames, gpu] = [median(frames), median(gpu)]; ++ let (demotions, syncs) = (case.count("demotions"), case.count("regions")); ++ eprintln!("\nwindow {name}: frame {frames:.2} ms, gpu {gpu:.2} ms, demotions {demotions}, syncs {syncs}"); ++ }); ++ } ++} ++ + #[test] + fn surface_coherent_off_touches_no_page() { + on_surface(true, |case| { diff --git a/patches/metal2vulkan/0016-the-window-s-cost-lane-beside-the-plain-.patch b/patches/metal2vulkan/0016-the-window-s-cost-lane-beside-the-plain-.patch new file mode 100644 index 0000000..b82dcaf --- /dev/null +++ b/patches/metal2vulkan/0016-the-window-s-cost-lane-beside-the-plain-.patch @@ -0,0 +1,34 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 04:58:57 +0000 +Subject: [PATCH 16/34] the window's cost lane beside the plain + paths + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/runtime_lane/surface_coherent.rs | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 07688b3c..143ca0a8 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -897,8 +897,16 @@ fn surface_coherent_cost_of_a_window() { + values.sort_by(f64::total_cmp); + values[values.len() / 2] + } +- for (name, coherent, demote) in [("off", false, "1"), ("stays", true, "0"), ("demoted", true, "1")] { +- on_surface_of([W, H, R], coherent, coherent, |case| { ++ let modes = [ ++ ("demoted", true, true, "1"), ++ ("off", false, true, "1"), ++ ("eager", false, false, "1"), ++ ("stays", true, true, "0"), ++ ("demoted again", true, true, "1"), ++ ("off again", false, true, "1"), ++ ]; ++ for (name, coherent, lazy, demote) in modes { ++ on_surface_of([W, H, R], coherent, lazy, |case| { + live::set("surface_coherent_demote", Some(demote)); + live::set("surface_coherent_copy_ns_per_mib", Some(COPIES_COST_NOTHING)); + live::set("surface_coherent_hold_s", Some("60")); diff --git a/patches/metal2vulkan/0017-surface-coherent-a-CPU-write-opens-the-whole-surface.patch b/patches/metal2vulkan/0017-surface-coherent-a-CPU-write-opens-the-whole-surface.patch new file mode 100644 index 0000000..526b170 --- /dev/null +++ b/patches/metal2vulkan/0017-surface-coherent-a-CPU-write-opens-the-whole-surface.patch @@ -0,0 +1,169 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 06:33:18 +0000 +Subject: [PATCH 17/34] surface coherent: a CPU write opens the whole surface + +A write costs a probe of every tracked surface before its page is known. A CPU +writer going down a surface paid that once a page: MercuryPosterExtension's +alpha fill stopped at every page of the wallpaper. The first write now holds +and snapshots every page and opens them all, so a writer stops once, and the +copy in takes the whole surface. The ledger pays for it. + +Pages::probe is new: it is what names a write's page now that held pages are +read-only, and arm alone cannot. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/shadow_coherent.rs | 58 +++++++++++-------- + .../src/runtime_lane/surface_coherent.rs | 31 ++++++++-- + 2 files changed, 60 insertions(+), 29 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 985ce191..f3468992 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -27,9 +27,10 @@ + //! once it completes. Completed work is never lost: a batch's writes are visible to the host at + //! its completion (`tracker.rs`), and the record is dropped only after its copy. + //! +-//! **A CPU write** never loses to a sync. The write stops before it lands, so the page's snapshot +-//! is what the guest overwrote: a later sync over the open page keeps every texel that differs +-//! from the snapshot ([`Coherent::merge`]). A GPU copy back over an open page cannot merge, so ++//! **A CPU write** never loses to a sync. The write stops before it lands, and opens the whole ++//! surface: a CPU writer covers it page by page, and each page would cost a probe of every surface. ++//! So each page's snapshot is what the guest overwrote: a later sync over the open pages keeps ++//! every texel that differs from it ([`Coherent::merge`]). A GPU copy back over an open page cannot merge, so + //! [`Coherent::before_flush`] waits for the records, a bounded time, and syncs on the CPU; past + //! the bound it counts `overwrite_risk`. Writing also demotes the surface to the shadow's own keep + //! and merge. +@@ -121,8 +122,13 @@ pub trait Pages: Send + Sync { + /// Armed pages, and open ones with `open_too`, held; the pages a CPU may have reached as the + /// markers moved come back. + fn hold(&self, pages: &[u64], open_too: bool) -> Vec; +- /// A held page opened to the guest's write. +- fn open(&self, page: u64); ++ /// A write in flight, whose page the host has not been told, made to name it: held pages armed ++ /// and armed pages writable, so its rerun traps with its address. What [`arm`](Self::arm) ++ /// leaves cannot do that: a write to a read-only page fails before it reaches the marker. ++ /// Returns what `arm` returns. ++ fn probe(&self, pages: &[u64]) -> Vec; ++ /// Held pages opened to the guest's writes. ++ fn open(&self, pages: &[u64]); + /// Open pages poisoned, while a sync merges into them. + fn freeze(&self, pages: &[u64]); + /// Frozen pages open again. +@@ -816,30 +822,32 @@ impl Coherent { + return; + } + self.counts.writes.fetch_add(1, Ordering::Relaxed); +- if let Some(area) = self.layout.area_of(page) { +- // A running write needs no wait: the merge at its completion keeps this one. A GPU +- // copy back already on its way cannot merge. +- let (settled, clear) = self.wait_settled(state, None); +- state = settled; +- self.sync_done(&mut state, SyncBy::Trap); +- if !clear { +- self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); +- } +- state.guest = Some(match state.guest { +- Some(guest) => bounding(Some(guest), Some(area)).unwrap_or(guest), +- None => area, +- }); ++ // A running write needs no wait: the merge at its completion keeps this one. A GPU copy ++ // back already on its way cannot merge. ++ let (settled, clear) = self.wait_settled(state, None); ++ state = settled; ++ self.sync_done(&mut state, SyncBy::Trap); ++ if !clear { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } ++ state.guest = Some(self.layout.whole()); + state.wrote_at = Some(Instant::now()); +- if !state.snapshots.contains_key(&page) { +- // SAFETY: the page lies in the surface's mapping; the guest's write stopped before it. +- let was = unsafe { std::slice::from_raw_parts(page as *const u8, PAGE as usize) }; +- state.snapshots.insert(page, was.into()); ++ let start = Instant::now(); ++ let every = self.all_pages(); ++ if let Some(mark) = &mark { ++ mark.hold(&every, false); ++ } ++ for page in &every { ++ if !state.snapshots.contains_key(page) { ++ // SAFETY: the page lies in the surface's mapping; held, no guest write reaches it. ++ let was = unsafe { std::slice::from_raw_parts(*page as *const u8, PAGE as usize) }; ++ state.snapshots.insert(*page, was.into()); ++ } + } + if let Some(mark) = &mark { +- mark.hold(&[page], false); +- mark.open(page); ++ mark.open(&every); + } ++ self.charge(&mut state, start.elapsed().as_nanos() as u64); + } + + /// An encoder is about to use the image: where the guest wrote is stale in it, and comes back +@@ -960,7 +968,7 @@ impl Coherent { + state.probed_at = Some(Instant::now()); + self.counts.probes.fetch_add(1, Ordering::Relaxed); + if let Some(mark) = pages() { +- let refused = mark.arm(&self.all_pages()); ++ let refused = mark.probe(&self.all_pages()); + self.note_came(&mut state, refused); + } + } +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 143ca0a8..871c9ef6 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -90,10 +90,16 @@ impl Pages for Fake { + std::mem::take(&mut *self.came.lock().unwrap()) + } + +- fn open(&self, page: u64) { ++ fn probe(&self, list: &[u64]) -> Vec { ++ self.arm(list) ++ } ++ ++ fn open(&self, list: &[u64]) { + let mut pages = self.pages.lock().unwrap(); +- if pages.get(&page) == Some(&Mark::Held) { +- pages.insert(page, Mark::Open); ++ for page in list { ++ if pages.get(page) == Some(&Mark::Held) { ++ pages.insert(*page, Mark::Open); ++ } + } + } + +@@ -422,11 +428,28 @@ fn surface_coherent_cpu_write_then_gpu_read() { + // A later write opens it again. + case.write(7, 7, [5, 6, 7, 8]); + assert_eq!(case.through_image(7, 7), [5, 6, 7, 0]); +- // Its rectangle alone: the first use's and the pages' own rows. + assert_eq!(case.count("copy_ins"), 3); + }); + } + ++/// A CPU writer down the whole surface stops once: a probe per page would stall it for a surface ++/// this size, and on a real guest every probe walks every tracked surface. ++#[test] ++fn surface_coherent_a_cpu_writer_stops_once() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ case.draw(GREEN, None); ++ for y in 0..HEIGHT { ++ case.write(1, y, [1, 2, 3, 4]); ++ } ++ assert_eq!(case.count("writes"), 1); ++ assert_eq!(case.through_image(1, 0), [1, 2, 3, 0], "the image took the writes"); ++ assert_eq!(case.through_image(1, HEIGHT - 1), [1, 2, 3, 0]); ++ assert_eq!(case.through_image(0, HEIGHT - 1), [0, 255, 0, 0], "and what the GPU made holds"); ++ }); ++} ++ + #[test] + fn surface_coherent_gpu_write_then_cpu_read_then_gpu_again() { + on_surface(true, |case| { diff --git a/patches/metal2vulkan/0018-surface-coherent-one-image-traps-per-surface.patch b/patches/metal2vulkan/0018-surface-coherent-one-image-traps-per-surface.patch new file mode 100644 index 0000000..cbabf49 --- /dev/null +++ b/patches/metal2vulkan/0018-surface-coherent-one-image-traps-per-surface.patch @@ -0,0 +1,249 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 07:23:27 +0000 +Subject: [PATCH 18/34] surface coherent: one image traps per surface + +Two textures over one IOSurface share a census surface, and each made its own +Coherent. Both tracked. find() gave a trap to the first in the list, so the +other never heard of a CPU write or read: a CPU read missed the other's finished +GPU output, and its image kept old bytes for good, as a tracked surface has no +resync clock. Closing one untracked the pages from under the other. + +The second texture now stays lazy, its copies back keeping the pages. The +surface that traps tells it of each write, and open pages are held before it +reads, so a write that took no trap cannot be missed either. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/shadow_coherent.rs | 45 ++++++++++++++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 31 ++++++++++ + .../src/runtime_lane/surface_coherent.rs | 57 +++++++++++++++++++ + 3 files changed, 132 insertions(+), 1 deletion(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index f3468992..624c304e 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -35,6 +35,11 @@ + //! the bound it counts `overwrite_risk`. Writing also demotes the surface to the shadow's own keep + //! and merge. + //! ++//! **One image over a surface traps.** A second texture over its pages stays lazy, and its copies ++//! back keep the pages; a trap reaches the first alone, which tells the rest ++//! ([`shadow_lazy::guest_wrote`](super::shadow_lazy::guest_wrote)), and before such a texture reads, ++//! pages open to the guest are held ([`Coherent::seal`]). ++//! + //! **The next command's copy in** takes the open pages' rectangle alone, and holds them first, so + //! a later write opens them again ([`Coherent::taking`]). + //! +@@ -358,6 +363,8 @@ struct State { + guest: Option<[u32; 4]>, + /// Each open page, as it was before the guest's write. + snapshots: BTreeMap>, ++ /// The pages of `snapshots` are open: a write reaches them with no trap. ++ opened: bool, + frozen: Vec, + /// When the last write was: the surface keeps and merges for the hold after it. + wrote_at: Option, +@@ -425,10 +432,26 @@ impl Coherent { + return true; + } + let range = self.range(); ++ // One image traps per surface: a trap reaches one, and that one would untrack the pages ++ // from under another. Any other stays lazy, its copies back keeping the pages. ++ { ++ let mut list = all().write().unwrap(); ++ let taken = |one: &Arc| { ++ let other = one.range(); ++ range.start < other.end && other.start < range.end ++ }; ++ if list.iter().any(taken) { ++ state.closed = true; ++ return false; ++ } ++ list.push(self.clone()); ++ } + let live = crate::live::on(SWITCH, false); + let start = Instant::now(); + let marked = pages().is_some_and(|p| live && p.track(range.start, range.end - range.start)); + if !marked { ++ let me = Arc::as_ptr(self); ++ all().write().unwrap().retain(|other| !std::ptr::eq(Arc::as_ptr(other), me)); + state.closed = true; + return false; + } +@@ -436,7 +459,6 @@ impl Coherent { + let spent = start.elapsed().as_nanos() as u64; + self.counts.track_ns.fetch_add(spent, Ordering::Relaxed); + self.charge(&mut state, spent); +- all().write().unwrap().push(self.clone()); + true + } + +@@ -847,7 +869,27 @@ impl Coherent { + if let Some(mark) = &mark { + mark.open(&every); + } ++ state.opened = true; + self.charge(&mut state, start.elapsed().as_nanos() as u64); ++ drop(state); ++ if let Some(lazy) = self.lazy.upgrade() { ++ super::shadow_lazy::guest_wrote(&lazy); ++ } ++ } ++ ++ /// Another image over the surface is about to read it: open pages held, so it reads all the ++ /// guest wrote and the next write traps. Whether any was open. ++ pub(crate) fn seal(&self) -> bool { ++ let mut state = self.lock(); ++ if !state.tracked || !std::mem::take(&mut state.opened) { ++ return false; ++ } ++ let list: Vec = state.snapshots.keys().copied().collect(); ++ if let Some(mark) = pages() { ++ let came = mark.hold(&list, true); ++ self.note_came(&mut state, came); ++ } ++ true + } + + /// An encoder is about to use the image: where the guest wrote is stale in it, and comes back +@@ -884,6 +926,7 @@ impl Coherent { + self.note_came(&mut state, came); + } + state.snapshots.clear(); ++ state.opened = false; + if whole { + state.guest = None; + } +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index d1c8eded..0cc1eaff 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -261,6 +261,19 @@ pub(crate) fn cpu_wrote(bytes: impl Fn() -> Option>) { + } + } + ++/// The guest's CPU wrote the pages under `lazy`, the one image over them that traps: the others ++/// have only this to tell them, and with no clock their images would keep the old bytes. ++pub(crate) fn guest_wrote(lazy: &Lazy) { ++ let Some(pages) = &lazy.pages else { ++ return; ++ }; ++ for other in surfaces() { ++ if !std::ptr::eq(Arc::as_ptr(&other), lazy) && other.over(pages) { ++ other.written.fetch_add(1, Ordering::Relaxed); ++ } ++ } ++} ++ + /// Before a CPU read through a texture over the host bytes `bytes`: each surface over them that + /// holds writes ahead copies them back, which this waits for. + pub(crate) fn cpu_reads(bytes: impl Fn() -> Option>) { +@@ -1069,6 +1082,9 @@ impl Lazy { + pub(super) fn take(&self, id: u64, touched: u64) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { + // Tracked from here: nothing its copy in reads can change unseen. + let coherent = self.coherent().filter(|coherent| coherent.tracks()); ++ if coherent.is_none() { ++ self.seal_beside(); ++ } + let taken = self.take_standing(id, touched, coherent.is_some()); + match coherent { + Some(coherent) => { +@@ -1079,6 +1095,21 @@ impl Lazy { + } + } + ++ /// The surface beside it that traps gets its pages held: writes it took with no trap are news. ++ fn seal_beside(&self) { ++ let Some(pages) = &self.pages else { ++ return; ++ }; ++ if !crate::shadow_coherent::surfaces_tracked() { ++ return; ++ } ++ for other in surfaces() { ++ if other.over(pages) && other.coherent().is_some_and(|coherent| coherent.seal()) { ++ guest_wrote(&other); ++ } ++ } ++ } ++ + fn take_standing( + &self, + id: u64, +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 871c9ef6..a8a89694 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -432,6 +432,33 @@ fn surface_coherent_cpu_write_then_gpu_read() { + }); + } + ++/// Two textures over one surface: a CPU read traps, and what the second made is in the pages. ++#[test] ++fn surface_coherent_a_cpu_read_gets_what_a_second_texture_made() { ++ on_surface(true, |case| { ++ let second = case.second_texture(); ++ case.draw(RED, None); ++ case.draw_on(&second, GREEN); ++ assert_eq!(case.read(9, 9), GREEN_TEXEL); ++ }); ++} ++ ++/// A trap reaches one surface, and a second texture's image over the pages must hear of the write. ++/// The lanes' clock is off, as it is for any texture that trusts the traps. ++#[test] ++fn surface_coherent_a_second_texture_reads_each_cpu_write() { ++ on_surface(true, |case| { ++ let second = case.second_texture(); ++ case.draw(RED, None); ++ assert_eq!(case.through(&second, 7, 7), [0, 0, 255, 0], "its image holds the bytes"); ++ case.write(7, 7, [1, 2, 3, 4]); ++ assert_eq!(case.through(&second, 7, 7), [1, 2, 3, 0]); ++ // That read held the pages again: this write traps too, and it hears. ++ case.write(7, 7, [5, 6, 7, 8]); ++ assert_eq!(case.through(&second, 7, 7), [5, 6, 7, 0]); ++ }); ++} ++ + /// A CPU writer down the whole surface stops once: a probe per page would stall it for a surface + /// this size, and on a real guest every probe walks every tracked surface. + #[test] +@@ -974,6 +1001,36 @@ fn surface_coherent_off_touches_no_page() { + } + + impl Case<'_> { ++ /// Another texture over the surface's pages, as a second `MTLTexture` over an IOSurface is. ++ fn second_texture(&self) -> Texture { ++ let mut d = ++ TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = self.buffer.resource_options(); ++ self.buffer ++ .new_texture_with_iosurface_plane(&d, IoSurfacePlane::new(0, self.row, false)) ++ .unwrap() ++ } ++ ++ fn draw_on(&self, texture: &Texture, color: [f32; 4]) { ++ let color = staged(self.device, &floats4(color)); ++ let rects = std::slice::from_ref(&self.rect); ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ pass(&command_buffer, texture, &self.opaque, rects, &color, None); ++ run(&command_buffer); ++ } ++ ++ /// What `texture`'s image holds at `x`, `y`, as [`Self::through_image`] finds it. ++ fn through(&self, texture: &Texture, x: u64, y: u64) -> [u8; 4] { ++ let color = staged(self.device, &floats4([0.0; 4])); ++ let rects = std::slice::from_ref(&self.rect); ++ let blending = rects_blending(self.device); ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ pass(&command_buffer, texture, &blending, rects, &color, None); ++ run(&command_buffer); ++ self.read(x, y) ++ } ++ + fn other_import(&self) -> Buffer { + surface_buffer(self.device, &self.mapping, self.len) + } diff --git a/patches/metal2vulkan/0019-surface-lazy-an-attachment-read-copies-the-image-not.patch b/patches/metal2vulkan/0019-surface-lazy-an-attachment-read-copies-the-image-not.patch new file mode 100644 index 0000000..7ad181b --- /dev/null +++ b/patches/metal2vulkan/0019-surface-lazy-an-attachment-read-copies-the-image-not.patch @@ -0,0 +1,519 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 10:07:25 +0000 +Subject: [PATCH 19/34] surface lazy: an attachment read copies the image, not + the bytes + +On Sim, every 1206x2622 surface has one guest texture. The second image beside +each is the attachment read's usage shadow: a draw samples at its own pixel the +surface its pass renders. That shadow's truth was the bytes, with a Lazy of its +own, so the attachment copied back before each read (not alone in its encoder), +beside_another refused the rest, and the shadow copied in again. + +For a lazy surface, whose image holds every usage, the shadow's truth is now +that image. The encoder brings the image current first, inside what the copy +reads, also for a clear or a pass that names only the copy; a copy into the +image leaves the copy behind, and it goes to no later encoder, whose byte writes +it could not see. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/metal2vulkan-runtime/src/shadow.rs | 59 ++++++++++++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 3 +- + .../metal2vulkan-runtime/src/tests/shadow.rs | 83 ++++++++++++++++++ + .../src/texture/attached_reads.rs | 4 +- + .../src/texture/usage_shadow.rs | 32 ++++++- + .../src/texture/view_shadow.rs | 12 ++- + .../src/runtime_lane/surface_coherent.rs | 86 ++++++++++++++++++- + 7 files changed, 270 insertions(+), 9 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index b84969b5..fbf6af58 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -302,6 +302,10 @@ pub(super) enum ShadowSource { + /// The buffer the copies go through where Vulkan copies no texels between the two + /// images' formats (an alpha format and another, `image_plan::copies_between`). + through: Option, ++ /// LOCAL (Atlas): the copies of the shadow `image` itself is, where this one reads a lazy ++ /// surface's image (`texture/attached_reads.rs`): that image must hold the bytes first, ++ /// and each copy into it leaves this one behind ([`ShadowSync::before`]). ++ over: Option>, + }, + /// A surface's two planes in a buffer's bytes, which the shadow holds the texels Metal reads + /// of (`ycbcr.rs`): its copy in converts them, and it is never copied back. +@@ -696,6 +700,14 @@ impl ShadowCopies { + } + } + ++ /// LOCAL (Atlas): the shadow whose image this one copies ([`ShadowSource::Image`]'s `over`). ++ fn over(&self) -> Option<&Arc> { ++ match &self.source { ++ ShadowSource::Image { over, .. } => over.as_ref(), ++ _ => None, ++ } ++ } ++ + /// Whether a render pass that reaches no tile and loads this shadow's texture leaves it to + /// its truth, which keeps every texel: an image of a buffer's bytes (`ShadowTarget::Image`), + /// which the texture's own image holds, and a two-plane YCbCr texture's render shadow, +@@ -1821,6 +1833,34 @@ fn merged_back(shadow: &ShadowUse) -> bool { + shadow.writes && !shadow.overwrites && shadow.copies.merges_back() + } + ++/// LOCAL (Atlas): `shadows` with, first, a read of the shadow each copies the image of, inside what ++/// the copy reads, where one does: `None` where none does. Whatever the command does to that image ++/// (a clear takes no copy in, a pass that reaches no tile no use at all), the copy reads it as the ++/// truth, as a copy of the bytes would. ++fn reads_truths(shadows: &[ShadowUse]) -> Option> { ++ if !shadows.iter().any(|shadow| shadow.copies.over().is_some()) { ++ return None; ++ } ++ let mut with = Vec::with_capacity(shadows.len() + 1); ++ for shadow in shadows { ++ if let Some(over) = shadow.copies.over() { ++ let read = ShadowUse { ++ copies: Arc::clone(over), ++ writes: false, ++ overwrites: false, ++ kept: false, ++ area: None, ++ loads: shadow.loads, ++ }; ++ join(&mut with, &read); ++ } ++ } ++ for shadow in shadows { ++ join(&mut with, shadow); ++ } ++ Some(with) ++} ++ + /// Whether a command that uses the shadowed textures `shadows` and has `uses` writes the bytes + /// of `shadows[i]` two ways: through it, and through `uses` or another shadowed texture over + /// them (the module's rules). +@@ -2226,7 +2266,9 @@ impl ShadowSync { + // The image holds the truth as this command buffer's commands left it, whether or + // not this command buffer may carry it on. + (entry.copies.stood).store(carry.id, Ordering::Relaxed); +- if touched != entry.met { ++ // LOCAL (Atlas): a copy of another shadow's image knows no write of that image's truth ++ // but through that shadow's entry, which a later encoder makes anew. ++ if touched != entry.met || entry.copies.over().is_some() { + continue; + } + carry.entries.push(Carried { +@@ -2473,6 +2515,9 @@ impl ShadowSync { + uses: &[Use], + mut merge: impl FnMut(&ShadowCopies, bool, Option<[u32; 4]>) -> Result, + ) -> Result>, Error> { ++ // LOCAL (Atlas): a shadow that copies another's image needs that image current first. ++ let with_truths = reads_truths(shadows); ++ let shadows = with_truths.as_deref().unwrap_or(shadows); + // A copy back of an image's shadow would replace the other alias's writes, and only a + // buffer's bytes laid out as the texture's are merged: such a command is refused before + // any merge is made (a rotated texture's too). +@@ -2608,6 +2653,7 @@ impl ShadowSync { + let carried = entry.carried; + entry.carried = false; + let took = carried && cause.is_none() && !overwrites; ++ let mut moved = false; + if let Some(cause) = cause { + if !overwrites { + let copy = ShadowCopy { +@@ -2624,6 +2670,7 @@ impl ShadowSync { + } + copies.push(copy); + entry.only_read = false; ++ moved = true; + } + entry.standing = Standing::Current; + entry.held = needs; +@@ -2659,12 +2706,22 @@ impl ShadowSync { + } + copies.push(copy); + entry.only_read = false; ++ moved = true; + } + } + // The command writes every texel before it reads any. + if overwrites { + entry.held = None; + } ++ // LOCAL (Atlas): the copy above changed the image a shadow of it copied. ++ if moved { ++ let key = shadow.copies.shadow_key; ++ for other in &mut self.entries { ++ if other.copies.over().is_some_and(|over| over.shadow_key == key) { ++ other.lose(Lost::Written); ++ } ++ } ++ } + // What the first copy in of this use moved, or would have moved where the carry took + // it off: the copy `before` pushes above, or the one it would have pushed. + let first = (took || (cause == Some(CopyCause::First) && !overwrites)).then(|| { +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 0cc1eaff..1f4d8b3e 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -70,7 +70,8 @@ + //! until its commit returns, as a copy back recorded before it runs before it. + //! + //! **Two textures' images over one surface** each learn of the other's writes from its copy back, +-//! so neither leaves any ahead once both are registered: the counts' `shared`. ++//! so neither leaves any ahead once both are registered: the counts' `shared`. A draw that samples ++//! the surface it renders is no such pair: it reads a copy of the image (`texture/usage_shadow.rs`). + //! + //! **Who can read stale bytes**, each for at most that bound and only of a surface left ahead: the + //! guest's CPU through its own mapping of the pages, which reaches the host through no call, and a +diff --git a/crates/metal2vulkan-runtime/src/tests/shadow.rs b/crates/metal2vulkan-runtime/src/tests/shadow.rs +index cb943ca2..625b1dfd 100644 +--- a/crates/metal2vulkan-runtime/src/tests/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/tests/shadow.rs +@@ -1422,6 +1422,7 @@ fn view_of(image: ResourceKey, levels: Range, layers: Range) -> Arc Arc) -> Arc { ++ let mut copies = Arc::into_inner(image_shadow(texture.shadow_key, 0, 1)).expect("one holder"); ++ if let ShadowSource::Image { over, .. } = &mut copies.source { ++ *over = Some(Arc::clone(texture)); ++ } ++ Arc::new(copies) ++} ++ ++/// A copy of a shadow's image reads that image as the truth: the image takes the bytes first, ++/// though the command names only the copy, though it clears the image, and wherever the copy reads ++/// past what the command does to the image. ++#[test] ++fn a_copy_of_a_shadows_image_copies_that_shadow_in_first() { ++ let buffer = ResourceKey::new(); ++ let t = cut_shadow(buffer, 0..1024); ++ let s = image_of(&t); ++ let mut sync = ShadowSync::default(); ++ assert_eq!( ++ command(&mut sync, &[(&s, false)], &[]), ++ [(t.shadow_key, true), (s.shadow_key, true)] ++ ); ++ ++ let (mut cleared, cleared_use) = through(&t, true); ++ cleared.overwrites = true; ++ let (read, read_use) = through(&s, false); ++ let (shadows, uses) = ([cleared, read], [cleared_use, read_use]); ++ let copies = ShadowSync::::default() ++ .before(&shadows, &uses, |_, _, _| Ok(0)) ++ .unwrap(); ++ // A keep follows: such a texture merges what it writes back. ++ assert_eq!(named(&copies[..2]), [(t.shadow_key, true), (s.shadow_key, true)]); ++ ++ let (mut drawn, drawn_use) = through(&t, true); ++ (drawn.area, drawn.loads) = (Some([10, 10, 20, 20]), Some([10, 10, 20, 20])); ++ let (mut read, read_use) = through(&s, false); ++ read.loads = Some([9, 9, 21, 21]); ++ let (shadows, uses) = ([drawn, read], [drawn_use, read_use]); ++ let copies = ShadowSync::::default() ++ .before(&shadows, &uses, |_, _, _| Ok(0)) ++ .unwrap(); ++ assert_eq!( ++ (copies[0].area, copies[1].area), ++ (Some([9, 9, 21, 21]), Some([9, 9, 21, 21])) ++ ); ++} ++ ++/// What changes the image goes on to its copy: a write of the bytes reaches the image at its next ++/// copy in, and only that says the copy is behind. ++#[test] ++fn a_copy_in_of_a_shadows_image_leaves_a_copy_of_it_behind() { ++ let buffer = ResourceKey::new(); ++ let t = cut_shadow(buffer, 0..1024); ++ let s = image_of(&t); ++ let mut sync = ShadowSync::default(); ++ command(&mut sync, &[(&s, false)], &[]); ++ assert_eq!(command(&mut sync, &[(&s, false)], &[]), []); ++ assert_eq!(command(&mut sync, &[], &[bytes(buffer, 0..4, true)]), []); ++ assert_eq!( ++ command(&mut sync, &[(&s, false)], &[]), ++ [(t.shadow_key, true), (s.shadow_key, true)] ++ ); ++} ++ ++/// A copy of a shadow's image goes to no later encoder: the bytes' writes it would miss there are ++/// that shadow's to see. ++#[test] ++fn a_copy_of_a_shadows_image_is_carried_to_no_later_encoder() { ++ let buffer = ResourceKey::new(); ++ let t = cut_shadow(buffer, 0..1024); ++ let s = image_of(&t); ++ let mut sync = ShadowSync::default(); ++ command(&mut sync, &[(&s, false)], &[]); ++ let mut next: ShadowSync = ShadowSync::carrying(sync.ended()); ++ next.wrote(&[bytes(buffer, 0..4, true)]); ++ assert_eq!( ++ command(&mut next, &[(&s, false)], &[]), ++ [(t.shadow_key, true), (s.shadow_key, true)] ++ ); ++} ++ + /// Each copy's step (into the shadow, or not), area, and why a copy in is made. + fn causes(copies: &[ShadowCopy]) -> Vec<(bool, Option<[u32; 4]>, Option)> { + copies +diff --git a/crates/metal2vulkan-runtime/src/texture/attached_reads.rs b/crates/metal2vulkan-runtime/src/texture/attached_reads.rs +index fe4f68b5..659bc34f 100644 +--- a/crates/metal2vulkan-runtime/src/texture/attached_reads.rs ++++ b/crates/metal2vulkan-runtime/src/texture/attached_reads.rs +@@ -38,7 +38,9 @@ + //! in before each instance whose draws read the shadow, after the instances before it wrote + //! the attachment; before the first, ahead of the pass's load actions. Each such draw counts + //! (`WorkCounts::attachment_reads_through_copies`, the shadow served or not), and so does each +-//! copy (`WorkCounts::shadow_copies`). It is exact and announces nothing. ++//! copy (`WorkCounts::shadow_copies`). It is exact and announces nothing. LOCAL (Atlas): a lazy ++//! surface's copy takes the texture's image as its truth, not the bytes, so the attachment's ++//! writes need not go back to the bytes for it (`shadow_lazy.rs`, `ShadowCopies::over`). + //! - **Only what the instances drew.** A copy after the pass's first instance, of a buffer's + //! bytes or of an image's one level and slice, where the instances since the last copy changed + //! them only by their draws into a color attachment the pass loads (the encoder's load, +diff --git a/crates/metal2vulkan-runtime/src/texture/usage_shadow.rs b/crates/metal2vulkan-runtime/src/texture/usage_shadow.rs +index 75d4b62e..1af00981 100644 +--- a/crates/metal2vulkan-runtime/src/texture/usage_shadow.rs ++++ b/crates/metal2vulkan-runtime/src/texture/usage_shadow.rs +@@ -13,7 +13,9 @@ + //! (`ShadowSource::Image`, as a shadowed view's, `texture/view_shadow.rs`), or, for a texture + //! over a buffer (an aliasing or a shadow image), the buffer's bytes (`ShadowSource::Bytes`, + //! a second shadow of them, `shadow.rs`). The encoders copy the truth in before a command that +-//! binds the shadow and back after it writes through it, by `shadow.rs`'s rules. ++//! binds the shadow and back after it writes through it, by `shadow.rs`'s rules. LOCAL (Atlas): ++//! a lazy surface's shadow, whose image holds every usage, so that only attachment reads take ++//! this one, copies that image instead (`shadow_lazy.rs`, `ShadowSource::Image`'s `over`). + //! - **Bindings.** A slot binding (a compute dispatch's, a draw's, a tile dispatch's) whose + //! descriptor needs the storage or sampled usage the texture's image and its usage both lack + //! writes the shadow's view and names the shadow in its uses; every other binding of the +@@ -234,6 +236,32 @@ impl Texture { + if record.presented.rotation != 0 { + return Err(Error::NotImplemented(RuntimeGap::RotatedTexture)); + } ++ // LOCAL (Atlas): a lazy surface's image, where only an attachment read takes the shadow: ++ // copied from the bytes it would leave that image's writes to go back for it, each frame. ++ if let Storage::Shadow { image, copies, .. } = &self.0.storage { ++ let both = ImageUsageFlags::SAMPLED | ImageUsageFlags::STORAGE; ++ if copies.lazy.is_some() && image.plan().usage.contains(both) { ++ let root = &self.root().0.record; ++ let over = Some(Arc::clone(copies)); ++ match device.image_shadow(call, image, root, &wider, refused, over) { ++ Ok(image) => { ++ let copies = match &image.owner { ++ Owner::ViewShadow(shadow) => Arc::clone(&shadow.copies), ++ _ => return Err(internal("an image shadow without its copies")), ++ }; ++ return Ok(UsageShadow { ++ image: image.shared(), ++ copies, ++ truth: "its image", ++ announced: AtomicBool::new(false), ++ }); ++ } ++ // Formats no image copy joins: through the bytes, as any shadow. ++ Err(Error::Unsupported(_)) => {} ++ Err(error) => return Err(error), ++ } ++ } ++ } + let (image, copies, truth) = match &self.0.storage { + Storage::Image(image) => { + let base = match &image.owner { +@@ -245,7 +273,7 @@ impl Texture { + | Owner::HeapShadow(_) => return Err(refused()), + }; + let image = +- device.image_shadow(call, base, &self.root().0.record, &wider, refused)?; ++ device.image_shadow(call, base, &self.root().0.record, &wider, refused, None)?; + let copies = match &image.owner { + Owner::ViewShadow(shadow) => Arc::clone(&shadow.copies), + _ => return Err(internal("an image shadow without its copies")), +diff --git a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +index c73065f2..739b98de 100644 +--- a/crates/metal2vulkan-runtime/src/texture/view_shadow.rs ++++ b/crates/metal2vulkan-runtime/src/texture/view_shadow.rs +@@ -313,7 +313,7 @@ impl Device { + ) -> Result { + let apart = + || Error::Unsupported(self.apart_refusal(Some(base.as_ref()), view.pixel_format)); +- let image = self.image_shadow(call, base, texture, view, apart)?; ++ let image = self.image_shadow(call, base, texture, view, apart, None)?; + self.shared().counts.view_shadow_made(); + let reinterpreting = + !self.shared().view_reinterpretation && self.reinterprets(base, view.pixel_format); +@@ -354,7 +354,8 @@ impl Device { + /// record of the texture `base` was made for, whose copies move the levels and layers `view` + /// presents (`ShadowSource::Image`). Refused `refused()` where the copies cannot serve the pair + /// (no transfer usage, formats whose copy units differ), and as `new_texture` refuses the +- /// image. ++ /// image. LOCAL (Atlas): `over` is the copies of the shadow `base` is, where it is one: this ++ /// image then lies in private memory, as any shadow of a surface's bytes does. + pub(super) fn image_shadow( + &self, + call: &'static str, +@@ -362,6 +363,7 @@ impl Device { + texture: &ViewRecord, + view: &ViewRecord, + refused: impl Fn() -> Error, ++ over: Option>, + ) -> Result { + let layout = TextureLayout { + texture_type: texture.texture_type, +@@ -462,7 +464,10 @@ impl Device { + ); + let shadow_range = range(aspects); + let mut descriptor = TextureDescriptor::texture_2d(view.pixel_format, 1, 1, false); +- descriptor.resource_options = base.options; ++ descriptor.resource_options = match over { ++ Some(_) => ResourceOptions::private(), ++ None => base.options, ++ }; + descriptor.allow_gpu_optimized_contents = base.allow_gpu_optimized_contents; + let image = + self.new_image_owned(plan, &layout, &descriptor, identity, |placement, raw| { +@@ -474,6 +479,7 @@ impl Device { + shadow: raw, + regions, + through, ++ over, + }, + shadow_key: identity.key, + shadow_range, +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index a8a89694..d9393d7a 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -3,6 +3,7 @@ + //! page faults first, as the guest's CPU would. Beside them, a lane pins down what loses writes + //! under `surface_lazy_present` alone, and the same steps under the traps. + ++use super::attachment_reads::function; + use super::first_use_writes::CHECK_WAIT; + use super::host_memory::{gpu_read, run, Mapping}; + use super::host_rt_frames::{pass, surface_buffer}; +@@ -12,9 +13,10 @@ use crate::gpu::TEST_DEVICE; + use metal2vulkan::device::HostPageTexturePolicy; + use super::pitched_surfaces::{bgra8_sampling, drawn_small, sampled_bgra8}; + use metal2vulkan::metal::{PixelFormat, PrimitiveType, TextureUsage}; ++use metal2vulkan::pipeline::{ColorAttachmentDescriptor, RenderPipelineDescriptor}; + use metal2vulkan_runtime::shadow_coherent::{self, Pages}; + use metal2vulkan_runtime::{ +- live, Buffer, CommandBuffer, CommandQueue, Device, IoSurfacePlane, LoadAction, ++ live, Buffer, CommandBuffer, CommandQueue, Device, IoSurfacePlane, LoadAction, PipelineOptions, + RenderPassDescriptor, RenderPipelineState, ResourceOptions, ScissorRect, StoreAction, + Texture, TextureDescriptor, + }; +@@ -226,6 +228,15 @@ impl Case<'_> { + } + } + ++/// The bytes reach `texel` at `x`, `y` once the clock copied them back. ++fn wait_for_pages(case: &Case<'_>, x: u64, y: u64, texel: [u8; 4]) { ++ let start = Instant::now(); ++ while case.behind(x, y) != texel && start.elapsed() < SOON { ++ std::thread::sleep(Duration::from_millis(5)); ++ } ++ assert_eq!(case.behind(x, y), texel); ++} ++ + fn rects_blending(device: &Device) -> RenderPipelineState { + rects(device, PixelFormat::BGRA8Unorm, true) + } +@@ -459,6 +470,53 @@ fn surface_coherent_a_second_texture_reads_each_cpu_write() { + }); + } + ++/// A draw that reads the surface it renders reads a copy of the image, not of the pages: what lies ++/// ahead in the image stays there, though the draw reads it, and comes back at the guest's read. ++#[test] ++fn surface_coherent_a_draw_that_reads_its_surface_leaves_it_ahead() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ for _ in 0..3 { ++ case.draw(GREEN, None); ++ case.reads_itself(LoadAction::Load, [0, 0, 8, 8]); ++ assert_eq!(case.behind(1, 1), RED_TEXEL, "nothing went through the pages"); ++ } ++ assert_eq!(case.count("flushes"), 0, "no GPU work for the pages"); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL, "the draw read what lay ahead"); ++ assert_eq!(case.read(100, 100), GREEN_TEXEL); ++ }); ++} ++ ++/// A pass that clears the surface and reads it reads it as it was, a CPU write of it among that: ++/// its image takes the write though the clear would need no copy in. ++#[test] ++fn surface_coherent_a_clear_that_reads_its_surface_reads_a_cpu_write() { ++ on_surface(true, |case| { ++ case.draw(GREEN, None); ++ case.read(0, 0); ++ case.write(7, 7, [1, 2, 3, 4]); ++ case.reads_itself(LoadAction::Clear, [0, 0, 16, 16]); ++ assert_eq!(case.read(7, 7), [1, 2, 3, 4]); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL); ++ assert_eq!(case.read(100, 100), [0, 0, 0, 255], "cleared outside the draw"); ++ }); ++} ++ ++/// Without traps the image holds only what its commands asked for: a draw that reads past what its ++/// pass loads gets it all the same. ++#[test] ++fn surface_lazy_a_draw_that_reads_its_surface_reads_past_its_scissor() { ++ on_surface(false, |case| { ++ case.draw(GREEN, None); ++ wait_for_pages(&case, 1, 1, GREEN_TEXEL); ++ case.reads_itself(LoadAction::Load, [4, 4, 8, 8]); ++ case.draw(RED, scissor(4, 4, 4, 4)); ++ wait_for_pages(&case, 5, 5, RED_TEXEL); ++ assert_eq!(case.behind(3, 3), GREEN_TEXEL); ++ }); ++} ++ + /// A CPU writer down the whole surface stops once: a probe per page would stall it for a surface + /// this size, and on a real guest every probe walks every tracked surface. + #[test] +@@ -1031,6 +1089,32 @@ impl Case<'_> { + self.read(x, y) + } + ++ /// One command buffer of a pass, loaded as `load` and stored, whose one draw over the pixels ++ /// `[x0, y0, x1, y1)` writes what it reads of the surface at its own pixel. ++ fn reads_itself(&self, load: LoadAction, [x0, y0, x1, y1]: [u64; 4]) { ++ let mut d = RenderPipelineDescriptor::new(function(self.device, "tob_rect", "tob_renders.ll")); ++ d.fragment = Some(function(self.device, "tob_read", "tob_renders.ll")); ++ d.color_attachments[0] = Some(ColorAttachmentDescriptor::new(PixelFormat::BGRA8Unorm)); ++ let state = (self.device.new_render_pipeline_state(&d, &PipelineOptions::default())).unwrap(); ++ let mut descriptor = RenderPassDescriptor::new(); ++ let attachment = &mut descriptor.color_attachments[0]; ++ attachment.texture = self.texture.clone(); ++ attachment.load_action = load; ++ attachment.store_action = StoreAction::Store; ++ let (x, y) = ( ++ |p: u64| p as f32 / WIDTH as f32 * 2.0 - 1.0, ++ |p: u64| 1.0 - p as f32 / HEIGHT as f32 * 2.0, ++ ); ++ let command_buffer = self.queue.command_buffer().unwrap(); ++ let mut encoder = command_buffer.render_command_encoder(&descriptor).unwrap(); ++ encoder.set_render_pipeline_state(&state).unwrap(); ++ encoder.set_vertex_bytes(&floats4([x(x0), y(y1), x(x1), y(y0)]), 0).unwrap(); ++ encoder.set_fragment_texture(self.texture.as_ref(), 0).unwrap(); ++ encoder.draw_primitives(PrimitiveType::Triangle, 0, 6, 1, 0).unwrap(); ++ encoder.end_encoding().unwrap(); ++ run(&command_buffer); ++ } ++ + fn other_import(&self) -> Buffer { + surface_buffer(self.device, &self.mapping, self.len) + } diff --git a/patches/metal2vulkan/0020-surface-lazy-one-image-for-a-surface-s-textures.patch b/patches/metal2vulkan/0020-surface-lazy-one-image-for-a-surface-s-textures.patch new file mode 100644 index 0000000..cffc8de --- /dev/null +++ b/patches/metal2vulkan/0020-surface-lazy-one-image-for-a-surface-s-textures.patch @@ -0,0 +1,584 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 11:18:39 +0000 +Subject: [PATCH 20/34] surface lazy: one image for a surface's textures + +A compositor's sampled-only texture over a surface's pages was a linear +image in place, with no standing of its own: its read met the bytes, so +the writer's whole surface went back every frame. A texture made over +pages a lazy surface's first shadow covers now binds that image and its +copies under its own record, where the extent, levels, layers, samples, +type, hazard tracking, usage, offset and rows fit and the image takes a +view of its format. It holds the first texture's texels, as the copies' +way back is weak. Anything else keeps today's path. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/buffer_texture.rs | 14 ++ + .../metal2vulkan-runtime/src/shadow_lazy.rs | 27 ++- + crates/metal2vulkan-runtime/src/texture.rs | 17 +- + .../src/texture/rotation.rs | 2 + + .../src/texture/surface_depth.rs | 1 + + .../src/texture/surface_twin.rs | 108 ++++++++++++ + .../src/texture/surface_two_planes.rs | 1 + + .../src/texture/ycbcr_image.rs | 2 + + .../src/runtime_lane/surface_coherent.rs | 162 +++++++++++++++++- + 9 files changed, 326 insertions(+), 8 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/texture/surface_twin.rs + +diff --git a/crates/metal2vulkan-runtime/src/buffer_texture.rs b/crates/metal2vulkan-runtime/src/buffer_texture.rs +index 7197fdc2..4a8e11b7 100644 +--- a/crates/metal2vulkan-runtime/src/buffer_texture.rs ++++ b/crates/metal2vulkan-runtime/src/buffer_texture.rs +@@ -1449,6 +1449,20 @@ impl Texels { + ) + } + ++ /// LOCAL (Atlas): where these surface texels lie, as a table of a surface's textures ++ /// (`surface_twin.rs`) finds them. ++ pub(super) fn surface_key(&self) -> Option<(usize, u64)> { ++ Some((self.buffer.host_range()?.start, self.layout.offset)) ++ } ++ ++ /// LOCAL (Atlas): whether `other` lies as these do, over the same pages, tracked as they are. ++ pub(super) fn same_place(&self, other: &Texels) -> bool { ++ self.buffer.host_range() == other.buffer.host_range() ++ && self.layout == other.layout ++ && self.texel_bytes == other.texel_bytes ++ && self.tracked() == other.tracked() ++ } ++ + /// LOCAL (Atlas): where the texels of a one-texel-a-block image of `extent` lie in host pages, + /// for a trapping surface (`shadow_coherent.rs`). + pub(super) fn surface_layout(&self, extent: [u32; 2]) -> Option { +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 1f4d8b3e..ed44b544 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -142,6 +142,8 @@ static FLUSHES: AtomicU64 = AtomicU64::new(0); + static FLUSHED_WHOLE: AtomicU64 = AtomicU64::new(0); + static FLUSHED_TEXELS: AtomicU64 = AtomicU64::new(0); + static DISCARDED: AtomicU64 = AtomicU64::new(0); ++/// Textures that bound a surface's image (`surface_twin.rs`). ++static TWINS: AtomicU64 = AtomicU64::new(0); + static PRESENTS: AtomicU64 = AtomicU64::new(0); + static PRESENTS_UNMATCHED: AtomicU64 = AtomicU64::new(0); + /// Milliseconds since the first report's chance at the last report. +@@ -326,7 +328,8 @@ pub(crate) fn display_image(bytes: &Range) -> Option { + let lazy = surfaces().into_iter().find(|lazy| lazy.over(bytes))?; + let coherent = lazy.coherent()?; + let takes = coherent.presented() && lazy.standing_committed() && !lazy.beside_another(); +- takes.then(|| lazy.owner.get().and_then(Weak::upgrade)).flatten() ++ let owner = || lazy.owners.lock().unwrap().iter().find_map(Weak::upgrade); ++ takes.then(owner).flatten() + } + + /// A command buffer completed, before its handlers: its writes in images trap from here. +@@ -566,7 +569,7 @@ fn report() { + "surface_lazy: taken {taken} refused {refused} resyncs {resyncs} | refused: touched {} \ + written {} epoch {} own {} uncommitted {} no-standing {} | in: first {} beyond-held {} \ + changed {} bounded {} | ahead: deferred {} shared {} flushes {} whole {} texels {} \ +- discarded {} presents {} unmatched {}", ++ discarded {} presents {} unmatched {} twins {}", + by[Refused::Touched as usize], + by[Refused::Written as usize], + by[Refused::Epoch as usize], +@@ -585,6 +588,7 @@ fn report() { + read(&DISCARDED), + read(&PRESENTS), + read(&PRESENTS_UNMATCHED), ++ read(&TWINS), + ); + for lazy in surfaces().iter().take(16) { + eprintln!( +@@ -601,6 +605,11 @@ fn report() { + } + } + ++/// A texture bound a surface's image (`surface_twin.rs`). ++pub(crate) fn twin_bound() { ++ TWINS.fetch_add(1, Ordering::Relaxed); ++} ++ + /// The counts so far, for a lane or a stderr line: standings taken, refused, and resyncs. + pub fn counts() -> (u64, u64, u64) { + ( +@@ -685,6 +694,8 @@ pub struct AheadCounts { + pub texels: u64, + /// The copies dropped, their texture gone. + pub discarded: u64, ++ /// The textures that bound a surface's image (`surface_twin.rs`). ++ pub twins: u64, + } + + /// The counts of writes left ahead so far. +@@ -696,6 +707,7 @@ pub fn ahead_counts() -> AheadCounts { + whole: read(&FLUSHED_WHOLE), + texels: read(&FLUSHED_TEXELS), + discarded: read(&DISCARDED), ++ twins: read(&TWINS), + } + } + +@@ -728,8 +740,9 @@ pub(crate) struct Lazy { + presents: AtomicU64, + /// Its trap coherence (`shadow_coherent.rs`), where the texture's image takes it. + coherent: OnceLock>, +- /// Weakly, the texture the display samples instead of the pages ([`display_image`]). +- owner: OnceLock>, ++ /// Weakly, the textures the display may sample instead of the pages ([`display_image`]): its ++ /// own, and those that bind its image (`surface_twin.rs`). ++ owners: Mutex>>, + } + + impl Lazy { +@@ -762,7 +775,7 @@ impl Lazy { + copied_in: AtomicU64::new(0), + presents: AtomicU64::new(0), + coherent: OnceLock::new(), +- owner: OnceLock::new(), ++ owners: Mutex::new(Vec::new()), + }) + }) + } +@@ -774,7 +787,9 @@ impl Lazy { + + /// The texture over it, the display's way to its image. + pub(crate) fn owned_by(&self, texture: Weak) { +- let _ = self.owner.set(texture); ++ let mut owners = self.owners.lock().unwrap(); ++ owners.retain(|owner| owner.strong_count() > 0); ++ owners.push(texture); + } + + /// Its trap coherence, while it holds: `None` where it never traps or no longer does. +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 94970fef..4e722b99 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -177,6 +177,7 @@ pub(super) mod storage_form; + pub(super) mod surface_depth; + pub(super) mod surface_two_planes; + mod twin_decode; ++mod surface_twin; + mod usage_shadow; + mod view_shadow; + pub(super) mod wrapped; +@@ -205,6 +206,9 @@ pub(super) struct TextureState { + placement: ImagePlacement, + /// The neutral images and samplers bindings take where nothing is bound (`neutral.rs`). + pub(super) neutrals: Mutex, ++ /// LOCAL (Atlas): each surface's first shadow, for its later textures to bind ++ /// (`surface_twin.rs`). ++ twins: surface_twin::Twins, + /// Parked images, and the makings of the live recyclable ones (`recycle.rs`). + pub(super) recycled: Mutex>, + /// The placeholder images of the colors a render pass instance leaves out of its subpass for +@@ -265,6 +269,7 @@ impl TextureState { + &crate::driver_defects::of(description), + ), + neutrals: Mutex::new(Neutrals::default()), ++ twins: surface_twin::Twins::default(), + recycled: Mutex::new(Recycler::new(recycle::IMAGES)), + storage_placeholders: Mutex::new(Placeholders::default()), + usage_answers: Mutex::new(UsageAnswers::default()), +@@ -503,6 +508,9 @@ enum Storage { + image: Arc, + texels: Arc, + copies: Arc, ++ /// LOCAL (Atlas): the texels `image` and `copies` were made for, where they are another ++ /// texture's that this one binds (`surface_twin.rs`). ++ first: Option>, + }, + } + +@@ -2140,7 +2148,13 @@ impl Device { + id_use: IdUse, + ) -> Result { + let record = ViewRecord::of_descriptor(layout, descriptor); +- let storage = self.storage_over_buffer(call, Arc::new(texels), descriptor, layout)?; ++ let texels = Arc::new(texels); ++ // LOCAL (Atlas): one image for a surface's bytes (`surface_twin.rs`). ++ if let Some(storage) = self.surface_twin(&texels, descriptor, layout) { ++ return Ok(Texture::made(self, storage, record, None, id_use)); ++ } ++ let storage = self.storage_over_buffer(call, texels, descriptor, layout)?; ++ self.note_surface_first(&storage); + Ok(Texture::made(self, storage, record, None, id_use)) + } + +@@ -2462,6 +2476,7 @@ impl Device { + image, + texels, + copies: Arc::new(copies), ++ first: None, + }) + } + +diff --git a/crates/metal2vulkan-runtime/src/texture/rotation.rs b/crates/metal2vulkan-runtime/src/texture/rotation.rs +index ed90a7e7..e0570caf 100644 +--- a/crates/metal2vulkan-runtime/src/texture/rotation.rs ++++ b/crates/metal2vulkan-runtime/src/texture/rotation.rs +@@ -255,6 +255,7 @@ impl Device { + (object, reason) + }); + Ok(Storage::Shadow { ++ first: None, + image: image.shared(), + texels: rows, + copies: Arc::new(copies), +@@ -292,6 +293,7 @@ impl Device { + (object, reason) + }); + Ok(Storage::Shadow { ++ first: None, + image: image.shared(), + texels: turned, + copies: Arc::new(copies), +diff --git a/crates/metal2vulkan-runtime/src/texture/surface_depth.rs b/crates/metal2vulkan-runtime/src/texture/surface_depth.rs +index 563fbde8..56410690 100644 +--- a/crates/metal2vulkan-runtime/src/texture/surface_depth.rs ++++ b/crates/metal2vulkan-runtime/src/texture/surface_depth.rs +@@ -174,6 +174,7 @@ impl Texture { + lazy: None, + }; + let storage = Storage::Shadow { ++ first: None, + image: image.shared(), + texels: Arc::new(texels.tiled_span(base..end, bytes)), + copies: Arc::new(copies), +diff --git a/crates/metal2vulkan-runtime/src/texture/surface_twin.rs b/crates/metal2vulkan-runtime/src/texture/surface_twin.rs +new file mode 100644 +index 00000000..fb35755d +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/texture/surface_twin.rs +@@ -0,0 +1,108 @@ ++//! LOCAL (Atlas): one image for one surface's bytes. A program's textures of one IOSurface plane ++//! (each over a no-copy buffer of the surface's pages, `host_aliases.rs`) were one image each: the ++//! surface's first texture, which renders, a shadow that holds its writes ahead (`shadow_lazy.rs`), ++//! and a later texture that only reads, a linear image in place over the pages. A compositor that ++//! samples such a texture meets the bytes every frame, so the first one's writes went back whole ++//! every frame. Sim 3x: 1 such read a frame, of one of three surfaces in turn, and the Sim's own ++//! rate on Tahoe rose to 60 fps with it gone. ++//! ++//! - **Which.** A texture made over the same host pages as the live first shadow of a lazy ++//! surface ([`Twins`]), at the same offset and rows, of the same extent, levels, layers, sample ++//! count, type and hazard tracking, whose declared usage the image has, and of a pixel format ++//! the image takes a view of (`Device::image_takes_view`: the same one, its sRGB twin, one of ++//! its list). Any other keeps today's path: an image of its own, in place or a shadow. ++//! - **What it binds.** The first texture's image and copies, under its own record: its format, ++//! swizzle and usage are its views' (`binding_view`), the image's flags and usage are fixed by ++//! the first texture's plan, which is why the check above stands. Its own texels stay, for what ++//! the CPU and the program ask of it. To every encoder it is that image: one resource key for ++//! the tracker's barriers across textures, one entry of `ShadowSync`, one standing, no ++//! `beside_another`. ++//! - **Lifetime.** It holds the image, the copies and the first texture's texels (the copies name ++//! that buffer): the first texture may go first, and the surface's trapping and standing go on ++//! as they did. The display finds any live one ([`Lazy::owned_by`]). ++//! - **One device.** The tasks of a guest share one `VkDevice`; a table is a device's, so a task ++//! on another device keeps today's path. ++//! - **Order.** Only what comes after the first shadow binds it: a reader made first stays in ++//! place, and the shadow made after it stays beside it, as before. ++ ++use super::*; ++ ++/// The live switch. ++const SWITCH: &str = "surface_twin"; ++ ++/// A surface's first shadow, weakly: it goes with its last texture. ++struct First { ++ image: Weak, ++ texels: Weak, ++ copies: Weak, ++} ++ ++/// A device's first shadow of each surface plane, by the first host address of its bytes and its ++/// offset in them. ++#[derive(Default)] ++pub(super) struct Twins(Mutex>); ++ ++impl Device { ++ /// The storage of a texture of `descriptor` (checked as `layout`) over `texels` that binds a ++ /// lazy surface's first shadow (the module docs), where one takes it. ++ pub(super) fn surface_twin( ++ &self, ++ texels: &Arc, ++ descriptor: &TextureDescriptor, ++ layout: &TextureLayout, ++ ) -> Option { ++ if !crate::live::on(SWITCH, true) || !texels.lies_lazily() || texels.buffer.detached() { ++ return None; ++ } ++ if surface_depth_unit(layout.pixel_format).is_some() { ++ return None; ++ } ++ let (image, first, copies) = { ++ let twins = lock(&self.shared().textures.twins.0); ++ let found = twins.get(&texels.surface_key()?)?; ++ (found.image.upgrade()?, found.texels.upgrade()?, found.copies.upgrade()?) ++ }; ++ let facts = self.shared().textures.facts(self.compile_device().profile()); ++ let usage = texture_usage(descriptor.usage, descriptor.swizzle); ++ let plan = plan_image(layout, usage, false, &facts).ok()?; ++ let theirs = image.plan(); ++ let same = plan.extent == theirs.extent ++ && plan.levels == theirs.levels ++ && plan.layers == theirs.layers ++ && plan.samples == theirs.samples ++ && plan.image_type == theirs.image_type ++ && plan.unavailable == theirs.unavailable ++ && theirs.usage.contains(plan.usage) ++ && theirs.flags.contains(plan.flags); ++ let takes = same ++ && first.same_place(texels) ++ && !copies.converts() ++ && self.image_takes_view(&image, layout.pixel_format); ++ takes.then(|| { ++ crate::shadow_lazy::twin_bound(); ++ Storage::Shadow { ++ image, ++ texels: Arc::clone(texels), ++ copies, ++ first: Some(first), ++ } ++ }) ++ } ++ ++ /// Note `storage`, a new texture's, where it is the first shadow of a lazy surface. ++ pub(super) fn note_surface_first(&self, storage: &Storage) { ++ let Storage::Shadow { image, texels, copies, first: None } = storage else { ++ return; ++ }; ++ let Some(key) = texels.surface_key().filter(|_| copies.lazy.is_some()) else { ++ return; ++ }; ++ let mut twins = lock(&self.shared().textures.twins.0); ++ twins.retain(|_, first| first.texels.strong_count() > 0); ++ twins.entry(key).or_insert_with(|| First { ++ image: Arc::downgrade(image), ++ texels: Arc::downgrade(texels), ++ copies: Arc::downgrade(copies), ++ }); ++ } ++} +diff --git a/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs b/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs +index 43c812d2..d2daa1ef 100644 +--- a/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs ++++ b/crates/metal2vulkan-runtime/src/texture/surface_two_planes.rs +@@ -255,6 +255,7 @@ impl Device { + (object, why) + }); + let storage = Storage::Shadow { ++ first: None, + image: image.shared(), + texels: Arc::new(texels.with_alpha(alpha_layout)), + copies: Arc::new(copies), +diff --git a/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs b/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs +index cf1d6816..98bf9b65 100644 +--- a/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs ++++ b/crates/metal2vulkan-runtime/src/texture/ycbcr_image.rs +@@ -379,6 +379,7 @@ impl Device { + lazy: None, + }; + Ok(Storage::Shadow { ++ first: None, + image: image.shared(), + texels, + copies: Arc::new(copies), +@@ -844,6 +845,7 @@ impl Texture { + lazy: None, + }; + let storage = Storage::Shadow { ++ first: None, + image: image.shared(), + texels: Arc::clone(texels), + copies: Arc::new(copies), +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index d9393d7a..7da9b9dc 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -12,9 +12,12 @@ use super::{validated_device_with, Messages, ScratchCache}; + use crate::gpu::TEST_DEVICE; + use metal2vulkan::device::HostPageTexturePolicy; + use super::pitched_surfaces::{bgra8_sampling, drawn_small, sampled_bgra8}; +-use metal2vulkan::metal::{PixelFormat, PrimitiveType, TextureUsage}; ++use metal2vulkan::metal::{ ++ PixelFormat, PrimitiveType, TextureSwizzle, TextureSwizzleChannels, TextureUsage, ++}; + use metal2vulkan::pipeline::{ColorAttachmentDescriptor, RenderPipelineDescriptor}; + use metal2vulkan_runtime::shadow_coherent::{self, Pages}; ++use metal2vulkan_runtime::shadow_lazy; + use metal2vulkan_runtime::{ + live, Buffer, CommandBuffer, CommandQueue, Device, IoSurfacePlane, LoadAction, PipelineOptions, + RenderPassDescriptor, RenderPipelineState, ResourceOptions, ScissorRect, StoreAction, +@@ -678,6 +681,145 @@ fn surface_coherent_the_display_samples_the_image() { + }); + } + ++/// A compositor's texture over the surface it reads, made after the texture that renders it: one ++/// image for the bytes (`surface_twin.rs`), so its read meets no bytes. ++#[test] ++fn surface_twin_a_reader_takes_the_surfaces_image() { ++ on_surface(true, |case| { ++ let twins = shadow_lazy::ahead_counts().twins; ++ let reader = case.reader(PixelFormat::BGRA8Unorm); ++ assert_eq!(shadow_lazy::ahead_counts().twins, twins + 1); ++ case.draw(RED, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], RED_TEXEL); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "nothing went back for the read"); ++ assert_eq!(case.count("flushes"), 0); ++ case.draw(GREEN, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], GREEN_TEXEL); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "still"); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL, "and the guest reads what lay ahead"); ++ }); ++} ++ ++/// The reader on a queue of its own, as a compositor's task has, while the write it reads is in ++/// flight: the one image's key orders them. ++#[test] ++fn surface_twin_a_read_waits_for_a_write_on_another_queue() { ++ on_surface(true, |case| { ++ let reader = case.reader(PixelFormat::BGRA8Unorm); ++ let other = case.device.new_command_queue().unwrap(); ++ for (color, texel) in [(RED, RED_TEXEL), (GREEN, GREEN_TEXEL), (RED, RED_TEXEL)] { ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, color, None); ++ command_buffer.commit().unwrap(); ++ assert_eq!(sampled_bgra8(&other, &reader, WIDTH, HEIGHT)[..4], texel); ++ } ++ }); ++} ++ ++/// What the guest writes reaches a reader of the image through its traps. ++#[test] ++fn surface_twin_a_reader_reads_a_cpu_write() { ++ on_surface(true, |case| { ++ let reader = case.reader(PixelFormat::BGRA8Unorm); ++ case.draw(RED, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], RED_TEXEL); ++ case.write(0, 0, [1, 2, 3, 255]); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], [1, 2, 3, 255]); ++ }); ++} ++ ++/// A reader's format is its own view of the image: the sRGB twin decodes what the first stored. ++#[test] ++fn surface_twin_a_reader_views_the_image_in_its_own_format() { ++ on_surface(true, |case| { ++ let twins = shadow_lazy::ahead_counts().twins; ++ let reader = case.reader(PixelFormat::BGRA8UnormSRGB); ++ assert_eq!(shadow_lazy::ahead_counts().twins, twins + 1); ++ case.draw([0.5, 0.5, 0.5, 1.0], None); ++ let read = sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT); ++ // 0x80 decodes to 0.2158. ++ assert!(read[0].abs_diff(55) <= 1, "decoded to {}", read[0]); ++ assert_eq!(read[3], 255); ++ }); ++} ++ ++/// A reader's swizzle is its own too: the texture that renders keeps its channels. ++#[test] ++fn surface_twin_a_reader_keeps_its_swizzle() { ++ on_surface(true, |case| { ++ let mut d = case.reader_descriptor(PixelFormat::BGRA8Unorm); ++ d.swizzle = TextureSwizzleChannels { ++ red: TextureSwizzle::Blue, ++ green: TextureSwizzle::Green, ++ blue: TextureSwizzle::Red, ++ alpha: TextureSwizzle::Alpha, ++ }; ++ let reader = case.over_other_import(&d); ++ case.draw(RED, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], [255, 0, 0, 255]); ++ assert_eq!(sampled_bgra8(&case.queue, case.texture(), WIDTH, HEIGHT)[..4], RED_TEXEL); ++ }); ++} ++ ++/// The display finds the image through the reader once the texture it was made for went. ++#[test] ++fn surface_twin_the_display_finds_the_image_through_a_reader() { ++ on_surface(true, |case| { ++ let mapping = Mapping::anonymous(case.len); ++ mapping.fill(BEFORE); ++ let first_bytes = surface_buffer(case.device, &mapping, case.len); ++ let mut d = case.reader_descriptor(PixelFormat::BGRA8Unorm); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ let first = first_bytes.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ d.usage = TextureUsage::SHADER_READ; ++ let other = surface_buffer(case.device, &mapping, case.len); ++ let reader = other.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ case.draw_on(&first, RED); ++ drop(first); ++ let shown = reader.surface_presented().expect("the reader stands for the first"); ++ assert_eq!(sampled_bgra8(&case.queue, &shown, WIDTH, HEIGHT)[..4], RED_TEXEL); ++ assert_eq!(mapping.read(0..4), [BEFORE; 4], "and it copied nothing"); ++ }); ++} ++ ++/// A format the image takes no view of keeps a texture of its own, and reads the right texels. ++#[test] ++fn surface_twin_a_format_the_image_does_not_list_stays_apart() { ++ on_surface(true, |case| { ++ let twins = shadow_lazy::ahead_counts().twins; ++ let reader = case.reader(PixelFormat::RGBA8Unorm); ++ assert_eq!(shadow_lazy::ahead_counts().twins, twins); ++ case.draw(RED, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], [255, 0, 0, 255]); ++ }); ++} ++ ++/// A reader outlives the texture and the buffer the image was made for: the image, its standing ++/// and its way back to the pages stay. ++#[test] ++fn surface_twin_a_reader_outlives_the_first_texture() { ++ on_surface(true, |case| { ++ let mapping = Mapping::anonymous(case.len); ++ mapping.fill(BEFORE); ++ let first_bytes = surface_buffer(case.device, &mapping, case.len); ++ let mut d = case.reader_descriptor(PixelFormat::BGRA8Unorm); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ let first = first_bytes.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ d.usage = TextureUsage::SHADER_READ; ++ let other = surface_buffer(case.device, &mapping, case.len); ++ let reader = other.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ case.draw_on(&first, RED); ++ drop(first); ++ drop(first_bytes); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], RED_TEXEL); ++ assert_eq!(mapping.read(0..4), [BEFORE; 4], "it stays ahead"); ++ shadow_coherent::fault(mapping.pointer().as_ptr() as u64, false); ++ assert_eq!(mapping.read(0..4), RED_TEXEL, "and comes back through the reader's hold"); ++ }); ++} ++ + /// What a copy back's GPU time is taken to be: 1 ns a MiB leaves nothing a surface's traps can pay + /// for, so every sync costs more than it saved; a second a MiB, the other way. + const COPIES_COST_NOTHING: &str = "1"; +@@ -1070,6 +1212,24 @@ impl Case<'_> { + .unwrap() + } + ++ /// A descriptor of a texture that only reads, at the surface's size. ++ fn reader_descriptor(&self, format: PixelFormat) -> TextureDescriptor { ++ let mut d = TextureDescriptor::texture_2d(format, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::SHADER_READ; ++ d.resource_options = self.buffer.resource_options(); ++ d ++ } ++ ++ /// A texture of `descriptor` over a buffer of its own of the surface's pages. ++ fn over_other_import(&self, descriptor: &TextureDescriptor) -> Texture { ++ let plane = IoSurfacePlane::new(0, self.row, false); ++ self.other_import().new_texture_with_iosurface_plane(descriptor, plane).unwrap() ++ } ++ ++ fn reader(&self, format: PixelFormat) -> Texture { ++ self.over_other_import(&self.reader_descriptor(format)) ++ } ++ + fn draw_on(&self, texture: &Texture, color: [f32; 4]) { + let color = staged(self.device, &floats4(color)); + let rects = std::slice::from_ref(&self.rect); diff --git a/patches/metal2vulkan/0021-surface-twin-the-lifetime-lane-copies-in-through-the.patch b/patches/metal2vulkan/0021-surface-twin-the-lifetime-lane-copies-in-through-the.patch new file mode 100644 index 0000000..987209a --- /dev/null +++ b/patches/metal2vulkan/0021-surface-twin-the-lifetime-lane-copies-in-through-the.patch @@ -0,0 +1,26 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 11:42:48 +0000 +Subject: [PATCH 21/34] surface twin: the lifetime lane copies in through the + reader's hold + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/runtime_lane/surface_coherent.rs | 4 ++++ + 1 file changed, 4 insertions(+) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 7da9b9dc..7594a5a9 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -817,6 +817,10 @@ fn surface_twin_a_reader_outlives_the_first_texture() { + assert_eq!(mapping.read(0..4), [BEFORE; 4], "it stays ahead"); + shadow_coherent::fault(mapping.pointer().as_ptr() as u64, false); + assert_eq!(mapping.read(0..4), RED_TEXEL, "and comes back through the reader's hold"); ++ // A guest's write goes in through the first texture's buffer, which the reader holds. ++ shadow_coherent::fault(mapping.pointer().as_ptr() as u64, true); ++ mapping.write(0, &[1, 2, 3, 255]); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], [1, 2, 3, 255]); + }); + } + diff --git a/patches/metal2vulkan/0022-coherent-per-range-marks-host-reads-and-writes-untra.patch b/patches/metal2vulkan/0022-coherent-per-range-marks-host-reads-and-writes-untra.patch new file mode 100644 index 0000000..2a20fcb --- /dev/null +++ b/patches/metal2vulkan/0022-coherent-per-range-marks-host-reads-and-writes-untra.patch @@ -0,0 +1,1058 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 05:42:38 +0000 +Subject: [PATCH 22/34] coherent: per-range marks, host reads and writes, + untracked and cross-device readers, signals + +- Pages::track holds [start, start + len) as a range of its own (QEMU's census now does too, audit + B1); the lanes' Fake models ranges, and a touch past the rows leaves the surface trapping. +- shadow_coherent::host_reads / host_writes: a host reader outside any command buffer (Reims' + video, scaler, JPEG) syncs first; a host writer opens the surface as a CPU write (audit P1). +- met_by sees untracked uses (Summary::lazy_uses, audit P2) and a reader on another device by the + host bytes its buffers lie over. +- A shared event's signal copies back what the command buffer's encoders left ahead first: a CPU + waiter reads the pages before the completion arms them. +- A surface whose texture went retires (syncs, untracks) when a new texture over its pages comes, + so the new one traps instead of copying back per encoder for good. +- flush: a copy back that finds nothing left after before_flush unmarks the copy in flight. +- CoherentFallbackCounts in WorkCounts::coherent: every silent fallback and host access, counted. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/mod.rs | 36 ++- + .../metal2vulkan-runtime/src/host_aliases.rs | 11 + + .../src/shadow_coherent.rs | 242 ++++++++++++++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 78 ++++- + crates/metal2vulkan-runtime/src/tracker.rs | 36 +++ + .../metal2vulkan-runtime/src/work_counts.rs | 4 + + .../src/runtime_lane/surface_coherent.rs | 275 +++++++++++++++++- + 7 files changed, 653 insertions(+), 29 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 4812e25c..9c8ff0e5 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2181,6 +2181,12 @@ impl CommandBuffer { + /// becomes at least `value`. Refused while an encoder is open (Metal asserts), after commit, + /// after an abandoned encoder, and for another device's event. + pub fn encode_signal_event(&self, event: EventRef<'_>, value: u64) -> Result<(), Error> { ++ // LOCAL (Atlas): a CPU waiter on a shared event reads the pages once it is signalled, ++ // before the command buffer completes: what its encoders left ahead in a surface's image ++ // goes back first, in the command buffer (`shadow_lazy::signalled_by`). ++ if matches!(event, EventRef::Shared(_)) && crate::shadow_lazy::ahead() { ++ self.copy_back_before_signal(); ++ } + self.event_command("encode_signal_event", event, |batches, event| { + batches.signal(event, value); + }) +@@ -2313,6 +2319,26 @@ impl CommandBuffer { + } + + /// Add an event wait or signal to the batches, after the checks Metal makes. ++ /// LOCAL (Atlas): a blit encoder of the copies back of what this command buffer's encoders ++ /// left ahead (`shadow_lazy::signalled_by`), where any did; one it may not open now is left ++ /// to the signal's own refusal. ++ fn copy_back_before_signal(&self) { ++ let Some(id) = self.state().shadow_carry.as_ref().map(ShadowCarry::id) else { ++ return; ++ }; ++ let Some(copies) = crate::shadow_lazy::signalled_by(id) else { ++ return; ++ }; ++ let Ok(mut blit) = self.blit_command_encoder() else { ++ return; ++ }; ++ for (lazy, copy, holds) in copies { ++ let _own = lazy.own(); ++ blit.surface_copy_back(©, holds); ++ } ++ let _ = blit.end_encoding(); ++ } ++ + fn event_command( + &self, + call: &'static str, +@@ -2523,7 +2549,15 @@ impl CommandBuffer { + .chain(segments.map(|recording| &recording.summary)) + }) + .collect(); +- crate::shadow_lazy::met_by(&summaries) ++ // A reader on another device names its own keys: the bytes they lie over meet. ++ let aliases = &self.0.device.shared().aliases; ++ let bytes: Vec> = (summaries.iter()) ++ .flat_map(|summary| summary.lazy_uses().map(|used| used.key)) ++ .collect::>() ++ .into_iter() ++ .filter_map(|key| aliases.bytes_of(key)) ++ .collect(); ++ crate::shadow_lazy::met_by(&summaries, &bytes) + }; + for surface in met { + surface.flush(crate::shadow_coherent::SyncBy::GpuReader); +diff --git a/crates/metal2vulkan-runtime/src/host_aliases.rs b/crates/metal2vulkan-runtime/src/host_aliases.rs +index 330c0690..ad56158d 100644 +--- a/crates/metal2vulkan-runtime/src/host_aliases.rs ++++ b/crates/metal2vulkan-runtime/src/host_aliases.rs +@@ -425,6 +425,17 @@ impl HostAliases { + self.changed(); + } + ++ /// LOCAL (Atlas): the host bytes the buffer of `key` lies over now, where it lies over any ++ /// (`shadow_lazy::met_by`). ++ pub(super) fn bytes_of(&self, key: ResourceKey) -> Option> { ++ if self.placed.load(Ordering::Relaxed) == 0 { ++ return None; ++ } ++ let inner = lock(&self.inner); ++ let &((start, len), _) = inner.placements.at.get(&key)?; ++ Some(start as usize..(start + len) as usize) ++ } ++ + /// The buffer of `key` lies over no host bytes from now on (its bytes detached). + pub(super) fn unplace(&self, key: ResourceKey) { + let mut inner = lock(&self.inner); +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 624c304e..b312bc1f 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -117,9 +117,11 @@ const LEDGER_SPAN: Duration = Duration::from_secs(1); + /// What marks a surface's pages. Addresses are the surface's own mapping's, whole pages, ascending, + /// one surface to a call; each call comes under the surface's lock, which [`fault`] takes. + pub trait Pages: Send + Sync { +- /// Every page held from now on: whether it can be, this surface. ++ /// The pages of `[start, start + len)` held from now on, as one range of their own: whether ++ /// they can be. Pages of the same mapping outside it stay unmarked (a plane beside, the ++ /// allocation's tail past the rows), and a range over one tracked already is refused. + fn track(&self, start: u64, len: u64) -> bool; +- /// Every marker off the surface around `start`. ++ /// Every marker off the range [`track`](Self::track) gave that holds `start`; nothing else. + fn untrack(&self, start: u64); + /// Held pages armed: the next access traps. Others stay. The pages a CPU reached as their + /// markers moved come back, still held, and they may hold a write: the caller has them. +@@ -142,6 +144,124 @@ pub trait Pages: Send + Sync { + + static PAGES: RwLock>> = RwLock::new(None); + ++/// A silent fallback or a host access the surfaces' traps answer, counted process-wide ++/// ([`fallback_counts`]): a run counts each, where nothing else says it happened. ++#[derive(Clone, Copy)] ++pub(crate) enum Fallback { ++ /// A surface's first use found another live surface over its pages: it never traps, and its ++ /// texture copies back per encoder ([`Coherent::tracks`]). ++ ClosedByOverlap, ++ /// The marks refused the surface, or the switch was off at its first use. ++ ClosedByMarks, ++ /// A surface whose texture went was synced and untracked for a new one over its pages. ++ RetiredDead, ++ /// A fault named no surface's range: the marks there went ([`fault`]). ++ OrphanFault, ++ /// [`Coherent::defers`] said no: the surface no longer traps. ++ DeferClosed, ++ /// Said no: demoted, copying back per encoder for now. ++ DeferEager, ++ /// Said no: a CPU write came within the hold, and it keeps and merges. ++ DeferWrote, ++ /// `shadow_lazy`'s `defer` said no: no traps, and not presented under `surface_lazy_present`. ++ LazyDeferNotPresent, ++ /// Said no: another texture's image lies over the pages (`shadow_lazy`'s `SHARED`). ++ LazyDeferBeside, ++ /// Said no: what its copies back hold went. ++ LazyDeferKeepsGone, ++ /// Said no: no queue for its copies back. ++ LazyDeferNoQueue, ++ /// A surface was demoted. ++ Demotion, ++ /// A host reader outside any command buffer (Reims' video and scaler) read surface bytes. ++ HostRead, ++ /// Of those, the reads a trapping surface lay under, which it synced first. ++ HostReadSynced, ++ /// A host writer outside any command buffer wrote surface bytes. ++ HostWrite, ++ /// Of those, the writes a trapping surface lay under, which it took as a CPU write. ++ HostWriteOpened, ++ /// A copy back (`shadow_lazy`'s `flush`) found nothing left to copy after its surface had ++ /// marked a copy in flight: it is unmarked now, where it once stayed marked for good. ++ FlushNothingLeft, ++ /// A command buffer met a surface's bytes through an untracked use alone (`met_by`). ++ MetUntracked, ++} ++ ++const FALLBACKS: usize = Fallback::MetUntracked as usize + 1; ++static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; ++ ++pub(crate) fn count(what: Fallback) { ++ FALLBACK[what as usize].fetch_add(1, Ordering::Relaxed); ++} ++ ++/// The process's counts of what [`Fallback`] names, in `WorkCounts::coherent`: every device's ++/// counts show the same, as surfaces trap across devices. Not part of the API. ++#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] ++pub struct CoherentFallbackCounts { ++ /// Surfaces that never trapped: another live one lay over their pages. ++ pub closed_by_overlap: u64, ++ /// Surfaces that never trapped: the marks refused them, or the switch was off. ++ pub closed_by_marks: u64, ++ /// Surfaces whose texture went, synced and untracked for a new texture over their pages. ++ pub retired_dead: u64, ++ /// Faults that named no surface's range. ++ pub orphan_faults: u64, ++ /// Encoders' ends that copied back: their surface no longer traps. ++ pub defer_refused_closed: u64, ++ /// Encoders' ends that copied back: their surface was demoted. ++ pub defer_refused_eager: u64, ++ /// Encoders' ends that copied back: a CPU write came within the hold. ++ pub defer_refused_wrote: u64, ++ /// Lazy encoders' ends that copied back: no traps, not presented. ++ pub lazy_defer_refused_not_present: u64, ++ /// Lazy encoders' ends that copied back: another image over the pages. ++ pub lazy_defer_refused_beside: u64, ++ /// Lazy encoders' ends that copied back: what its copies hold went. ++ pub lazy_defer_refused_keeps_gone: u64, ++ /// Lazy encoders' ends that copied back: no queue. ++ pub lazy_defer_refused_no_queue: u64, ++ /// Demotions, every surface's. ++ pub demotions: u64, ++ /// Host reads of surface bytes outside any command buffer. ++ pub host_reads: u64, ++ /// Of those, over a trapping surface, which synced first. ++ pub host_reads_synced: u64, ++ /// Host writes of surface bytes outside any command buffer. ++ pub host_writes: u64, ++ /// Of those, over a trapping surface, taken as a CPU write. ++ pub host_writes_opened: u64, ++ /// Copies back that found nothing left after marking a copy in flight. ++ pub flush_nothing_left: u64, ++ /// Command buffers that met a surface through an untracked use alone. ++ pub met_untracked: u64, ++} ++ ++/// The counts of [`CoherentFallbackCounts`] so far. ++pub fn fallback_counts() -> CoherentFallbackCounts { ++ let read = |what: Fallback| FALLBACK[what as usize].load(Ordering::Relaxed); ++ CoherentFallbackCounts { ++ closed_by_overlap: read(Fallback::ClosedByOverlap), ++ closed_by_marks: read(Fallback::ClosedByMarks), ++ retired_dead: read(Fallback::RetiredDead), ++ orphan_faults: read(Fallback::OrphanFault), ++ defer_refused_closed: read(Fallback::DeferClosed), ++ defer_refused_eager: read(Fallback::DeferEager), ++ defer_refused_wrote: read(Fallback::DeferWrote), ++ lazy_defer_refused_not_present: read(Fallback::LazyDeferNotPresent), ++ lazy_defer_refused_beside: read(Fallback::LazyDeferBeside), ++ lazy_defer_refused_keeps_gone: read(Fallback::LazyDeferKeepsGone), ++ lazy_defer_refused_no_queue: read(Fallback::LazyDeferNoQueue), ++ demotions: read(Fallback::Demotion), ++ host_reads: read(Fallback::HostRead), ++ host_reads_synced: read(Fallback::HostReadSynced), ++ host_writes: read(Fallback::HostWrite), ++ host_writes_opened: read(Fallback::HostWriteOpened), ++ flush_nothing_left: read(Fallback::FlushNothingLeft), ++ met_untracked: read(Fallback::MetUntracked), ++ } ++} ++ + /// The last words of the surfaces that went since the last report. + fn ended() -> &'static Mutex> { + static ENDED: OnceLock>> = OnceLock::new(); +@@ -285,9 +405,12 @@ pub(crate) enum SyncBy { + Demotion, + /// The surface leaves. + Unmap, ++ /// A host outside any command buffer reads the pages ([`host_reads`]). ++ HostRead, + } + +-const SYNC_BY: [&str; 7] = ["trap", "gpu_read", "cpu_read", "copy_in", "refused", "demotion", "unmap"]; ++const SYNC_BY: [&str; 8] = ++ ["trap", "gpu_read", "cpu_read", "copy_in", "refused", "demotion", "unmap", "host_read"]; + + /// What trapping spent lately against what the copies it replaced would have, in ns. + #[derive(Default)] +@@ -432,16 +555,27 @@ impl Coherent { + return true; + } + let range = self.range(); ++ let meets = |one: &Arc| { ++ let other = one.range(); ++ range.start < other.end && other.start < range.end ++ }; ++ // A surface over these pages whose texture went keeps what it left ahead in its image, ++ // and its traps would bring that over what this one's copies write: it lands in the pages ++ // first and lets them go. Neither lock of `all` is held meanwhile, as `close` takes it. ++ let dead: Vec> = (all().read().unwrap().iter()) ++ .filter(|one| meets(one) && one.lazy.strong_count() == 0) ++ .cloned() ++ .collect(); ++ for one in dead { ++ one.retire(); ++ } + // One image traps per surface: a trap reaches one, and that one would untrack the pages + // from under another. Any other stays lazy, its copies back keeping the pages. + { + let mut list = all().write().unwrap(); +- let taken = |one: &Arc| { +- let other = one.range(); +- range.start < other.end && other.start < range.end +- }; +- if list.iter().any(taken) { ++ if list.iter().any(meets) { + state.closed = true; ++ count(Fallback::ClosedByOverlap); + return false; + } + list.push(self.clone()); +@@ -453,6 +587,7 @@ impl Coherent { + let me = Arc::as_ptr(self); + all().write().unwrap().retain(|other| !std::ptr::eq(Arc::as_ptr(other), me)); + state.closed = true; ++ count(Fallback::ClosedByMarks); + return false; + } + state.tracked = true; +@@ -476,12 +611,14 @@ impl Coherent { + pub(crate) fn defers(&self, id: u64, area: Option<[u32; 4]>) -> bool { + let mut state = self.lock(); + if state.closed || !state.tracked { ++ count(Fallback::DeferClosed); + return false; + } + let now = Instant::now(); + match state.eager_until { + Some(until) if now < until => { + self.counts.eager.fetch_add(1, Ordering::Relaxed); ++ count(Fallback::DeferEager); + return false; + } + Some(_) => { +@@ -491,6 +628,7 @@ impl Coherent { + None => {} + } + if state.wrote_at.is_some_and(|at| at.elapsed() < self.hold()) { ++ count(Fallback::DeferWrote); + return false; + } + let whole = self.layout.whole(); +@@ -555,6 +693,7 @@ impl Coherent { + state.out_until = Some(until); + state.ledger = Ledger::default(); + self.counts.demotions.fetch_add(1, Ordering::Relaxed); ++ count(Fallback::Demotion); + // What is pending goes now, and what is running at its completion. + self.sync_done(state, SyncBy::Demotion); + } +@@ -844,10 +983,16 @@ impl Coherent { + return; + } + self.counts.writes.fetch_add(1, Ordering::Relaxed); ++ self.wrote(state, mark); ++ } ++ ++ /// A CPU write is about to land in the pages, the guest's or a host's: the whole surface opens ++ /// and the image takes what changes, as the module docs say. ++ fn wrote(&self, state: MutexGuard<'_, State>, mark: Option>) { + // A running write needs no wait: the merge at its completion keeps this one. A GPU copy + // back already on its way cannot merge. + let (settled, clear) = self.wait_settled(state, None); +- state = settled; ++ let mut state = settled; + self.sync_done(&mut state, SyncBy::Trap); + if !clear { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); +@@ -877,6 +1022,54 @@ impl Coherent { + } + } + ++ /// A host outside any command buffer is about to read the pages (Reims' video encoder, ++ /// scaler, JPEG): what is done goes there first, and a write still running, which the guest ++ /// may have heard of by a shared event, is waited for a bounded time and goes too. ++ fn host_reads(&self) { ++ let state = self.lock(); ++ if !state.tracked { ++ return; ++ } ++ count(Fallback::HostReadSynced); ++ let mut state = state; ++ self.sync_done(&mut state, SyncBy::HostRead); ++ if !state.running.is_empty() { ++ let (settled, _) = self.wait_settled(state, Some(self.layout.whole())); ++ state = settled; ++ self.sync_done(&mut state, SyncBy::HostRead); ++ } ++ } ++ ++ /// A host outside any command buffer is about to write the pages (Reims' video decoder, ++ /// scaler, JPEG): as a CPU write of the whole surface, so the next copy in takes it. ++ fn host_writes(&self) { ++ let state = self.lock(); ++ if !state.tracked { ++ return; ++ } ++ count(Fallback::HostWriteOpened); ++ self.counts.writes.fetch_add(1, Ordering::Relaxed); ++ self.wrote(state, pages()); ++ } ++ ++ /// Its texture went, and another texture over the pages is about to track them: everything ++ /// done lands in the pages, and a write still running is waited for a bounded time, then the ++ /// marks go. What runs past the bound reaches the image alone, as for any surface untracked. ++ fn retire(&self) { ++ let mut state = self.lock(); ++ if !state.tracked { ++ return; ++ } ++ self.sync_done(&mut state, SyncBy::Unmap); ++ if !state.running.is_empty() { ++ let (settled, _) = self.wait_settled(state, Some(self.layout.whole())); ++ state = settled; ++ self.sync_done(&mut state, SyncBy::Unmap); ++ } ++ count(Fallback::RetiredDead); ++ self.close(&mut state, true); ++ } ++ + /// Another image over the surface is about to read it: open pages held, so it reads all the + /// guest wrote and the next write traps. Whether any was open. + pub(crate) fn seal(&self) -> bool { +@@ -1191,6 +1384,7 @@ pub fn fault(addr: u64, write: bool) { + match find(addr) { + Some(surface) => surface.fault(page, write), + None => { ++ count(Fallback::OrphanFault); + if let Some(mark) = pages() { + mark.untrack(addr); + } +@@ -1198,6 +1392,36 @@ pub fn fault(addr: u64, write: bool) { + } + } + ++/// A host outside any command buffer is about to read the bytes `bytes` of the library's mapping ++/// of a surface (Reims' video encoder, scaler, JPEG and the like read them so): each trapping ++/// surface over them brings them up to date first. Call it before the read, with no lock a GPU ++/// completion needs. ++pub fn host_reads(bytes: Range) { ++ count(Fallback::HostRead); ++ for surface in over(&bytes) { ++ surface.host_reads(); ++ } ++} ++ ++/// A host outside any command buffer is about to write the bytes `bytes` (Reims' video decoder, ++/// scaler, JPEG): each trapping surface over them takes it as a CPU write of the whole surface, so ++/// the next command that uses its image copies the bytes in. Call it before the write. ++pub fn host_writes(bytes: Range) { ++ count(Fallback::HostWrite); ++ for surface in over(&bytes) { ++ surface.host_writes(); ++ } ++} ++ ++/// The tracked surfaces whose pages meet `bytes`. ++fn over(bytes: &Range) -> Vec> { ++ if !surfaces_tracked() { ++ return Vec::new(); ++ } ++ let meets = |s: &&Arc| s.range().start < bytes.end && bytes.start < s.range().end; ++ all().read().unwrap().iter().filter(meets).cloned().collect() ++} ++ + /// From a stop at `addr` to the access's rerun. + pub fn timed(addr: u64, ns: u64) { + if let Some(surface) = find(addr) { +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index ed44b544..a4340670 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -93,7 +93,7 @@ use super::device::Device; + use super::queue::CommandQueue; + use super::resource::ResourceKey; + use super::shadow::{bounding, CopyCause, ShadowCopies, ShadowCopy, Step}; +-use super::shadow_coherent::{Coherent, Layout, Reader, SyncBy}; ++use super::shadow_coherent::{count, Coherent, Fallback, Layout, Reader, SyncBy}; + use super::tracker::{Summary, Use, UseRange}; + use crate::exit_guard; + use std::any::Any; +@@ -392,24 +392,40 @@ pub(crate) fn ahead() -> bool { + } + + /// The surfaces to copy back before the commit of a command buffer whose encoders' uses are +-/// `summaries`: each that committed writes of lie ahead and that it uses the bytes of (the module +-/// docs). A copy in is such a use, so its command buffer's own writes wait for its commit. +-pub(crate) fn met_by(summaries: &[&Summary]) -> Vec> { ++/// `summaries`, its buffers lying over the host bytes `bytes`: each that committed writes of lie ++/// ahead and that it uses the bytes of (the module docs). A copy in is such a use, so its command ++/// buffer's own writes wait for its commit. The bytes find a reader on another device, whose keys ++/// are its own. ++pub(crate) fn met_by(summaries: &[&Summary], bytes: &[Range]) -> Vec> { + let mut met = Vec::new(); + // Most surfaces stay ahead for good where the pages trap: their keys rule most of them out. + let keys: HashSet = (summaries.iter()) +- .flat_map(|s| s.all_uses().iter().map(|used| used.key)) ++ .flat_map(|s| s.lazy_uses().map(|used| used.key)) + .collect(); + for lazy in surfaces() { +- if !lazy.truth.get().is_some_and(|(key, alias, _)| { ++ let keyed = lazy.truth.get().is_some_and(|(key, alias, _)| { + keys.contains(key) || alias.is_some_and(|alias| keys.contains(&alias)) +- }) { ++ }); ++ let over = || bytes.iter().any(|bytes| lazy.over(bytes)); ++ if !keyed && !over() { + continue; + } + let settled = (lazy.pending.lock().unwrap().as_ref()) + .is_some_and(|left| left.settled.is_some()); +- let used = || (summaries.iter()).any(|s| s.all_uses().iter().any(|u| lazy.meets(u))); ++ if !keyed { ++ if settled && !lazy.own_thread() { ++ met.push(lazy); ++ } ++ continue; ++ } ++ // An untracked use counts: the tracker orders nothing by it, but its reads are the ++ // pages' all the same, and a trapping surface has no clock to bring them later. ++ let used = || (summaries.iter()).any(|s| s.lazy_uses().any(|u| lazy.meets(u))); + if settled && !lazy.own_thread() && used() { ++ let tracked = || (summaries.iter()).any(|s| s.all_uses().iter().any(|u| lazy.meets(u))); ++ if !tracked() { ++ count(Fallback::MetUntracked); ++ } + met.push(lazy); + } + } +@@ -469,6 +485,32 @@ pub(crate) fn deferred_by(id: u64) -> Vec<(Arc, ShadowCopy, Vec)> { + left + } + ++/// What the command buffer `id`'s encoders left ahead, for a copy back before it signals a shared ++/// event, whose CPU waiter reads the pages once it is signalled: `None` where they left nothing. ++/// A surface that does not trap copies back as the copies' encoder opens ([`deferred_by`]); one ++/// that traps keeps its record, which the copy repeats, so its pages trap again at completion. ++pub(crate) fn signalled_by(id: u64) -> Option, ShadowCopy, Vec)>> { ++ let mut copies = Vec::new(); ++ let mut any = false; ++ for lazy in surfaces() { ++ let pending = lazy.pending.lock().unwrap(); ++ let Some(ahead) = pending.as_ref() else { ++ continue; ++ }; ++ let Some((_, area)) = ahead.open.iter().find(|(open, _)| *open == id) else { ++ continue; ++ }; ++ any = true; ++ if lazy.coherent().is_some() { ++ let copy = copy_back_of(ahead.copies.clone(), *area); ++ let holds = ahead.holds.clone(); ++ drop(pending); ++ copies.push((lazy, copy, holds)); ++ } ++ } ++ any.then_some(copies) ++} ++ + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what + /// that command buffer's earlier encoders left ahead in a trapping surface whose bytes it meets. + /// The record stays, whose texels the copy repeats, so what the record holds lasts the copy. +@@ -888,6 +930,7 @@ impl Lazy { + && (!crate::live::on(PRESENT, false) + || !(self.displayed.load(Ordering::Relaxed) || crate::live::on(PRESENT_ALL, false))) + { ++ count(Fallback::LazyDeferNotPresent); + return false; + } + // What it leaves ahead only hooks that find it can bring back, standing or not. +@@ -895,18 +938,22 @@ impl Lazy { + // Another texture's image over these pages learns of a write through the copy back. + if self.beside_another() { + SHARED.fetch_add(1, Ordering::Relaxed); ++ count(Fallback::LazyDeferBeside); + return false; + } + let Some(holds) = self.keeps.get() else { ++ count(Fallback::LazyDeferKeepsGone); + return false; + }; + let holds: Option> = holds.iter().map(Weak::upgrade).collect(); + let Some(holds) = holds else { ++ count(Fallback::LazyDeferKeepsGone); + return false; + }; + if self.queue.get().is_none() { + let device = self.device.downcast_ref::(); + let Some(queue) = device.and_then(|d| d.new_command_queue().ok()) else { ++ count(Fallback::LazyDeferNoQueue); + return false; + }; + let _ = self.queue.set(Arc::new(queue)); +@@ -982,10 +1029,17 @@ impl Lazy { + Some(coherent) => Some(coherent.before_flush(by)?), + None => None, + }; +- let (area, copies, holds) = { +- let pending = self.pending.lock().unwrap(); +- let left = pending.as_ref()?; +- (left.settled, left.copies.clone(), left.holds.clone()) ++ let left = (self.pending.lock().unwrap().as_ref()) ++ .map(|left| (left.settled, left.copies.clone(), left.holds.clone())); ++ // What ran may have completed and been synced since `before_flush` marked the copy in ++ // flight: the mark must go with it, or the surface's unmap answers busy for good and every ++ // wait for a settled surface runs to its bound. ++ let Some((area, copies, holds)) = left else { ++ if let Some(coherent) = &coherent { ++ count(Fallback::FlushNothingLeft); ++ coherent.flush_failed(); ++ } ++ return None; + }; + // The image's: `Pending`'s and this one, where neither texture nor command holds it. A + // trapping surface's pages stay the guest's until its unmap, whatever texture went. +diff --git a/crates/metal2vulkan-runtime/src/tracker.rs b/crates/metal2vulkan-runtime/src/tracker.rs +index f2eb5c80..5905bd90 100644 +--- a/crates/metal2vulkan-runtime/src/tracker.rs ++++ b/crates/metal2vulkan-runtime/src/tracker.rs +@@ -1193,6 +1193,9 @@ struct Part { + /// resources have one, held inline ([`Few`]), so a command's first use of a resource + /// allocates nothing of its own. + tracked: KeyMap>>, ++ /// LOCAL (Atlas): the untracked uses, one per resource, its ranges' hull, kept only while a ++ /// lazy surface is registered (`shadow_lazy::met_by`): nothing orders by them. ++ untracked: Vec>, + stages: vk::PipelineStageFlags2, + } + +@@ -1200,11 +1203,26 @@ impl Part { + fn new() -> Part { + Part { + tracked: KeyMap::default(), ++ untracked: Vec::new(), + stages: vk::PipelineStageFlags2::NONE, + } + } + } + ++/// LOCAL (Atlas): `used` among `untracked`, one per resource, its range the hull of all of them. ++fn add_untracked(untracked: &mut Vec>, used: &Use) { ++ let found = untracked.iter_mut().find(|known| known.key == used.key); ++ if let Some(known) = found { ++ if let Some(range) = known.range.hull(&used.range) { ++ known.range = range; ++ known.read = known.read.union(used.read); ++ known.write = known.write.union(used.write); ++ return; ++ } ++ } ++ untracked.push(used.clone()); ++} ++ + impl EncoderUses { + /// The uses of a new encoder, `ordered` when Metal runs its commands in encoded order. + pub(super) fn new(ordered: bool) -> EncoderUses { +@@ -1305,6 +1323,11 @@ impl EncoderUses { + ranges.change_all(coalesce_uses); + } + } ++ if crate::shadow_lazy::active() { ++ for used in uses.iter().filter(|used| !used.tracked && !used.is_nothing()) { ++ add_untracked(&mut part.untracked, used); ++ } ++ } + } + + /// Whether a command the encoder recorded so far writes memory one of `uses` reads: an +@@ -1457,6 +1480,7 @@ impl EncoderUses { + uses.sort_by_key(|used| used.key); + Summary { + uses, ++ untracked: part.untracked, + stages: part.stages, + waits: Vec::new(), + updates: Vec::new(), +@@ -1766,6 +1790,8 @@ fn inside_uses( + pub(super) struct Summary { + /// The tracked uses, one per resource and range, sorted by resource. + uses: Vec>, ++ /// LOCAL (Atlas): the untracked uses, for `shadow_lazy::met_by` alone ([`Part`]'s). ++ untracked: Vec>, + /// Every stage the encoder's commands ran in. + stages: vk::PipelineStageFlags2, + /// The fences waited for before the encoder. +@@ -1792,6 +1818,7 @@ impl Summary { + uses.sort_by_key(|used| used.key); + Summary { + uses, ++ untracked: Vec::new(), + stages: vk::PipelineStageFlags2::ALL_COMMANDS, + waits: Vec::new(), + updates: Vec::new(), +@@ -1805,6 +1832,7 @@ impl Summary { + let every = |fence| FenceUse { fence, draws: None }; + Summary { + uses: Vec::new(), ++ untracked: Vec::new(), + stages: vk::PipelineStageFlags2::ALL_COMMANDS, + waits: waits.into_iter().map(every).collect(), + updates: updates.into_iter().map(every).collect(), +@@ -1830,6 +1858,9 @@ impl Summary { + /// range into one of the same kind, which claims more and stays exact. + pub(super) fn join(&mut self, other: Summary) { + self.stages |= other.stages; ++ for used in &other.untracked { ++ add_untracked(&mut self.untracked, used); ++ } + for used in other.uses { + if let Some(known) = self + .uses +@@ -1889,6 +1920,11 @@ impl Summary { + &self.uses + } + ++ /// LOCAL (Atlas): the tracked uses and the untracked ones (`shadow_lazy::met_by`). ++ pub(super) fn lazy_uses(&self) -> impl Iterator> { ++ self.uses.iter().chain(&self.untracked) ++ } ++ + /// The tracked uses. + #[cfg(test)] + pub(super) fn uses(&self) -> &[Use] { +diff --git a/crates/metal2vulkan-runtime/src/work_counts.rs b/crates/metal2vulkan-runtime/src/work_counts.rs +index a288274a..aac9e4c2 100644 +--- a/crates/metal2vulkan-runtime/src/work_counts.rs ++++ b/crates/metal2vulkan-runtime/src/work_counts.rs +@@ -2762,6 +2762,9 @@ pub struct WorkCounts { + /// names the copy behind a recorder's time on a driver that records each region as an + /// operation of its own (`row_copies.rs`). + pub copy_sites: [CopySite; 32], ++ /// LOCAL (Atlas): the trap-coherent surfaces' silent fallbacks and host accesses so far, the ++ /// process's (`shadow_coherent::fallback_counts`): the same on every device. ++ pub coherent: super::shadow_coherent::CoherentFallbackCounts, + /// The `vkCmdCopyBuffer` regions over 16 KiB whose size is no multiple of 4 KiB that the + /// device's functions recorded as two, the bytes up to the last whole multiple of 4 KiB and + /// the rest (`device/functions.rs`), as a driver may record such a region slowly (ANV). +@@ -4722,6 +4725,7 @@ impl Device { + object_churn: churn.0, + shadow_keeps: churn.1, + copy_sites: self.shared().vk().copy_sites(), ++ coherent: super::shadow_coherent::fallback_counts(), + copy_regions_split: self.shared().vk().copy_regions_split(), + spare_command_pools_taken: counters.spare_command_pools_taken.load(Ordering::Relaxed), + spare_command_pools_reset: counters.spare_command_pools_reset.load(Ordering::Relaxed), +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 7594a5a9..ccd8299c 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -19,9 +19,10 @@ use metal2vulkan::pipeline::{ColorAttachmentDescriptor, RenderPipelineDescriptor + use metal2vulkan_runtime::shadow_coherent::{self, Pages}; + use metal2vulkan_runtime::shadow_lazy; + use metal2vulkan_runtime::{ +- live, Buffer, CommandBuffer, CommandQueue, Device, IoSurfacePlane, LoadAction, PipelineOptions, +- RenderPassDescriptor, RenderPipelineState, ResourceOptions, ScissorRect, StoreAction, +- Texture, TextureDescriptor, ++ live, Buffer, CommandBuffer, CommandQueue, CpuCacheMode, Device, HazardTrackingMode, ++ HostMemory, IoSurfacePlane, LoadAction, PipelineOptions, RenderPassDescriptor, ++ RenderPipelineState, ResourceOptions, ScissorRect, StorageMode, StoreAction, Texture, ++ TextureDescriptor, + }; + use std::collections::BTreeMap; + use std::sync::{Arc, Mutex}; +@@ -46,10 +47,12 @@ enum Mark { + Frozen, + } + +-/// QEMU's page states, without the guest. ++/// QEMU's page states, without the guest: each track a range of its own, as the census holds them. + #[derive(Default)] + struct Fake { + pages: Mutex>, ++ /// The ranges tracked: a page outside them has no mark. ++ ranges: Mutex>>, + /// Pages to answer as reached while their markers moved. + came: Mutex>, + /// Pages the next `arm` answers as reached, and leaves held. +@@ -60,16 +63,29 @@ struct Fake { + + impl Pages for Fake { + fn track(&self, start: u64, len: u64) -> bool { ++ let mut ranges = self.ranges.lock().unwrap(); ++ let range = start..start + len; ++ if ranges.contains(&range) { ++ return true; ++ } ++ if ranges.iter().any(|other| other.start < range.end && range.start < other.end) { ++ return false; ++ } + let mut pages = self.pages.lock().unwrap(); +- for page in (start..start + len).step_by(PAGE as usize) { ++ for page in range.clone().step_by(PAGE as usize) { + pages.entry(page).or_insert(Mark::Held); + } ++ ranges.push(range); + true + } + + fn untrack(&self, start: u64) { +- let _ = start; +- self.pages.lock().unwrap().clear(); ++ let mut ranges = self.ranges.lock().unwrap(); ++ let Some(at) = ranges.iter().position(|range| range.contains(&start)) else { ++ return; ++ }; ++ let range = ranges.swap_remove(at); ++ self.pages.lock().unwrap().retain(|page, _| !range.contains(page)); + } + + fn arm(&self, list: &[u64]) -> Vec { +@@ -135,6 +151,12 @@ impl Fake { + fn armed_pages(&self) -> usize { + (self.pages.lock().unwrap().values()).filter(|mark| **mark == Mark::Armed).count() + } ++ ++ /// QEMU's unmap of the surface: every mark goes (`census_surface_unmapped`). ++ fn unmapped(&self) { ++ self.pages.lock().unwrap().clear(); ++ self.ranges.lock().unwrap().clear(); ++ } + } + + struct Case<'d> { +@@ -1304,3 +1326,242 @@ impl Case<'_> { + run(&command_buffer); + } + } ++ ++/// Audit B1, the library's side: QEMU holds a texture's rows alone, and a touch of a page past them ++/// names no surface. The marks it lets go of are that page's range's, none: the surface still traps. ++#[test] ++fn surface_coherent_a_touch_past_the_rows_leaves_the_surface_trapping() { ++ on_surface_of([WIDTH, 250, ROW], true, true, |case| { ++ case.draw(RED, None); ++ let past = case.base() + 250 * ROW; ++ assert!(past + PAGE <= case.base() + case.len as u64, "the mapping has pages past the rows"); ++ assert_eq!(case.fake.mark(past), None, "the marks hold the rows' pages alone"); ++ let armed = case.fake.armed_pages(); ++ assert!(armed > 0); ++ let orphans = shadow_coherent::fallback_counts().orphan_faults; ++ shadow_coherent::fault(past, true); ++ assert_eq!(shadow_coherent::fallback_counts().orphan_faults, orphans + 1); ++ assert_eq!(case.fake.armed_pages(), armed, "the surface's pages stay armed"); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ assert_eq!(case.count("traps"), 1); ++ }); ++} ++ ++/// Audit P1: a host reader outside any command buffer (Reims' video encoder, scaler, JPEG) reads ++/// the pages. Unasked, they are stale; asked first ([`shadow_coherent::host_reads`]), current. ++#[test] ++fn surface_coherent_a_host_reader_reads_the_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4], "the pages lack the write until asked"); ++ let before = shadow_coherent::fallback_counts(); ++ shadow_coherent::host_reads(case.range()); ++ assert_eq!(case.behind(5, 5), RED_TEXEL); ++ assert_eq!(case.behind(WIDTH - 1, HEIGHT - 1), RED_TEXEL); ++ let after = shadow_coherent::fallback_counts(); ++ assert_eq!(after.host_reads, before.host_reads + 1); ++ assert_eq!(after.host_reads_synced, before.host_reads_synced + 1); ++ assert_eq!(case.count("host_read"), 1); ++ assert_eq!(case.fake.armed_pages(), 0, "held after the sync"); ++ case.draw(GREEN, None); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL, "and a later write traps as ever"); ++ }); ++} ++ ++/// A host read over pages no surface traps changes nothing, and counts. ++#[test] ++fn surface_coherent_a_host_reader_elsewhere_costs_nothing() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let before = shadow_coherent::fallback_counts(); ++ let end = case.range().end; ++ shadow_coherent::host_reads(end..end + PAGE); ++ let after = shadow_coherent::fallback_counts(); ++ assert_eq!(after.host_reads, before.host_reads + 1); ++ assert_eq!(after.host_reads_synced, before.host_reads_synced); ++ assert_eq!(case.behind(5, 5), [BEFORE; 4]); ++ }); ++} ++ ++/// A host read while a write runs, which the guest may have heard of through a shared event: the ++/// read waits for it, a bounded time. ++#[test] ++fn surface_coherent_a_host_read_waits_for_a_write_in_flight() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ let hold = case.device.new_shared_event().unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ command_buffer.encode_wait_for_event((&hold).into(), 1).unwrap(); ++ case.pass(&command_buffer, GREEN, None); ++ command_buffer.commit().unwrap(); ++ let release = { ++ let hold = hold.clone(); ++ std::thread::spawn(move || { ++ std::thread::sleep(Duration::from_millis(10)); ++ hold.set_signaled_value(1); ++ }) ++ }; ++ shadow_coherent::host_reads(case.range()); ++ assert_eq!(case.behind(5, 5), GREEN_TEXEL); ++ release.join().unwrap(); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ }); ++} ++ ++/// Audit P1, the other way: a host writer (Reims' video decoder) writes the pages through its own ++/// mapping, no trap. Told first ([`shadow_coherent::host_writes`]), the next command takes it. ++#[test] ++fn surface_coherent_a_host_writer_reaches_the_image() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ let before = shadow_coherent::fallback_counts(); ++ shadow_coherent::host_writes(case.range()); ++ assert_eq!(shadow_coherent::fallback_counts().host_writes_opened, before.host_writes_opened + 1); ++ case.mapping.write((7 * ROW + 7 * 4) as usize, &[1, 2, 3, 4]); ++ assert_eq!(case.through_image(7, 7), [1, 2, 3, 0], "the next command took it"); ++ assert_eq!(case.through_image(8, 7), [0, 0, 255, 0], "and the GPU's texels around it"); ++ }); ++} ++ ++/// Audit P2: a texture over the pages with untracked hazards, made apart from the image, reads them ++/// in place: its command buffer must bring what lies ahead first. ++#[test] ++fn surface_coherent_an_untracked_reader_reads_the_writes() { ++ on_surface(true, |case| { ++ let untracked = ResourceOptions::new( ++ StorageMode::Managed, ++ CpuCacheMode::DefaultCache, ++ HazardTrackingMode::Untracked, ++ ); ++ // SAFETY: the mapping outlives the buffer (the case drops it first). ++ let memory = unsafe { HostMemory::new(case.mapping.pointer()) }; ++ // SAFETY: as above. ++ let bytes = unsafe { ++ case.device.new_buffer_with_bytes_no_copy(memory, case.len as u64, untracked, None) ++ } ++ .unwrap(); ++ let mut d = case.reader_descriptor(PixelFormat::BGRA8Unorm); ++ d.resource_options = untracked; ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ let reader = bytes.new_texture_with_iosurface_plane(&d, plane).unwrap(); ++ let before = shadow_coherent::fallback_counts().met_untracked; ++ case.draw(RED, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], RED_TEXEL); ++ assert_eq!(shadow_coherent::fallback_counts().met_untracked, before + 1); ++ case.draw(GREEN, None); ++ assert_eq!(sampled_bgra8(&case.queue, &reader, WIDTH, HEIGHT)[..4], GREEN_TEXEL); ++ }); ++} ++ ++/// A CPU read after the GPU signalled a shared event past the write, before its command buffer ++/// completes: the signal promises the write (audit Q4). ++#[test] ++fn surface_coherent_a_cpu_read_after_a_signal_reads_the_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ case.read(0, 0); ++ let signal = case.device.new_shared_event().unwrap(); ++ let hold = case.device.new_shared_event().unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, GREEN, None); ++ command_buffer.encode_signal_event((&signal).into(), 1).unwrap(); ++ command_buffer.encode_wait_for_event((&hold).into(), 1).unwrap(); ++ command_buffer.commit().unwrap(); ++ assert!(signal.wait_until_signaled_value(1, SOON), "the GPU reached the signal"); ++ let read = case.read(5, 5); ++ hold.set_signaled_value(1); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(read, GREEN_TEXEL, "what the signal promised"); ++ }); ++} ++ ++/// A texture made over the pages after the first one went with writes ahead: what holds those ++/// writes holds the first's image, which the new one binds (`surface_twin.rs`), and it reads them. ++#[test] ++fn surface_coherent_a_texture_made_after_the_first_went_reads_its_writes() { ++ on_surface(true, |mut case| { ++ case.draw(RED, None); ++ case.texture = None; ++ case.texture = Some(case.second_texture()); ++ case.draw(GREEN, scissor(0, 0, 8, 8)); ++ assert_eq!(case.read(100, 100), RED_TEXEL, "what the first left"); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL, "what the second drew"); ++ case.draw(RED, scissor(0, 0, 4, 4)); ++ assert_eq!(case.read(1, 1), RED_TEXEL); ++ assert_eq!(case.read(5, 5), GREEN_TEXEL); ++ }); ++} ++ ++/// A texture made over the pages after the first one went with nothing ahead, so that nothing holds ++/// its image: the first's surface stays tracked until the unmap, and the new one found it there and ++/// never trapped, copying back per encoder for good, and its image missed the guest's writes. Now ++/// the old one lets go of the pages, and the new one traps. ++#[test] ++fn surface_coherent_a_texture_made_after_the_first_went_traps() { ++ on_surface(true, |mut case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ case.texture = None; ++ case.texture = Some(case.second_texture()); ++ let before = shadow_coherent::fallback_counts(); ++ case.draw(GREEN, scissor(0, 0, 8, 8)); ++ let after = shadow_coherent::fallback_counts(); ++ assert_eq!(after.closed_by_overlap, before.closed_by_overlap, "the new one traps"); ++ assert_eq!(after.retired_dead, before.retired_dead + 1, "the old one went"); ++ assert!(case.fake.armed_pages() > 0, "and leaves its writes ahead"); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL); ++ assert_eq!(case.read(100, 100), RED_TEXEL); ++ case.write(9, 9, [1, 2, 3, 4]); ++ assert_eq!(case.through_image(9, 9), [1, 2, 3, 0], "its image takes the guest's writes"); ++ }); ++} ++ ++/// Pages unmapped, then mapped again for a new surface: it traps, with no claim of the old. ++#[test] ++fn surface_coherent_pages_mapped_again_trap_for_the_new_surface() { ++ on_surface(true, |mut case| { ++ case.draw(RED, None); ++ assert!(shadow_coherent::unmapping(case.range())); ++ shadow_coherent::unmapped(case.range()); ++ case.fake.unmapped(); ++ case.texture = None; ++ case.mapping.fill(BEFORE); ++ let bytes = case.other_import(); ++ let mut d = TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, WIDTH, HEIGHT, false); ++ d.usage = TextureUsage::RENDER_TARGET | TextureUsage::SHADER_READ; ++ d.resource_options = bytes.resource_options(); ++ let plane = IoSurfacePlane::new(0, ROW, false); ++ case.texture = Some(bytes.new_texture_with_iosurface_plane(&d, plane).unwrap()); ++ let before = shadow_coherent::fallback_counts(); ++ case.draw(GREEN, scissor(0, 0, 8, 8)); ++ let after = shadow_coherent::fallback_counts(); ++ assert_eq!(after.closed_by_overlap, before.closed_by_overlap, "the old one left no claim"); ++ assert!(case.fake.armed_pages() > 0, "the new surface traps"); ++ assert_eq!(case.read(1, 1), GREEN_TEXEL); ++ assert_eq!(case.read(100, 100), [BEFORE; 4], "and none of the old one's writes came back"); ++ }); ++} ++ ++/// A reader of the pages on another device (another task's, as `device_lend` makes): its commit ++/// must bring what lies ahead, as one on the writer's device does. ++#[test] ++fn surface_coherent_a_reader_on_another_device_reads_the_writes() { ++ on_surface(true, |case| { ++ let cache = ScratchCache::new().unwrap(); ++ let messages = Messages::default(); ++ let other = validated_device_with(TEST_DEVICE, &messages, cache.path(), |o| { ++ o.policy.host_page_textures = HostPageTexturePolicy::InPlace; ++ }) ++ .unwrap(); ++ case.draw(RED, None); ++ let bytes = surface_buffer(&other, &case.mapping, case.len); ++ let queue = other.new_command_queue().unwrap(); ++ assert_eq!(gpu_read(&queue, &bytes, 0..4), RED_TEXEL); ++ drop(bytes); ++ drop(queue); ++ drop(other); ++ let reported = messages.take(); ++ assert!(reported.is_empty(), "the layer reported: {reported:#?}"); ++ }); ++} diff --git a/patches/metal2vulkan/0023-coherent-a-signal-is-seen-once-the-writes-before-it-.patch b/patches/metal2vulkan/0023-coherent-a-signal-is-seen-once-the-writes-before-it-.patch new file mode 100644 index 0000000..15ff739 --- /dev/null +++ b/patches/metal2vulkan/0023-coherent-a-signal-is-seen-once-the-writes-before-it-.patch @@ -0,0 +1,570 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 06:59:50 +0000 +Subject: [PATCH 23/34] coherent: a signal is seen once the writes before it + are in the pages (E1) + +A shared event's value, read from the semaphore's counter or seen at the +completion turn, rose before the turn armed the pages of the command buffers it +finished, so a guest that saw the value could read a trapping surface's held +pages with their older bytes (upstream review E1). Three paths, each a lane +that failed at 76ab0722: + +- A signal after the writer's command buffer: a signal kept while a tracked + surface holds open or running writes (shadow_coherent::unarmed, which waits + for no lock) is behind the arm: no counter read observes it or any after it, + and no wait is handed its index; the completion turn does. Ported by hand + from upstream 76d89d87 (EventTimeline::keep's behind_merge, the observe_read + clamp, landing), with its timeline unit test. Counted: + CoherentFallbackCounts::signals_behind_arm. +- The completion turn arms the pages of each command buffer its batches end + (shadow_lazy::completing) before their signals are seen; complete() finds + them armed. +- A plain event's chain (writer signals an MTLEvent, a relay on another queue + waits for it and signals a shared one): copy_back_before_signal now runs + before every event signal, not only a shared event's. + +Open, its lane ignored and run by name: a command buffer committed to wait for +a shared event draws over a surface the guest writes before setting it; the +encoder's copy in was decided as it was encoded (Coherent::taking), so the +draw misses the write. It needs a copy in recorded as the held batch goes. + +Lanes (filter "surface", ANV): 129 passed 4 failed before (06:41Z), 131 +passed 1 failed after (06:49Z); the failure is ycbcr_textures, as before. +Event lanes 24/24. Runtime unit tests 1149 -> 1150 passed, the one failure +(exit_guard) as at 76ab0722. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/completion.rs | 10 ++ + .../src/command_buffer/mod.rs | 17 +- + .../src/command_buffer/submit.rs | 10 +- + crates/metal2vulkan-runtime/src/event.rs | 6 +- + .../src/event_timeline.rs | 40 ++++- + .../src/shadow_coherent.rs | 32 +++- + .../src/tests/event_timeline.rs | 39 ++++- + .../src/tests/host_settable_event_waits.rs | 4 +- + .../src/tests/scheduler.rs | 2 +- + .../src/runtime_lane/surface_coherent.rs | 155 +++++++++++++++++- + 10 files changed, 295 insertions(+), 20 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +index cc2497bb..e0335c15 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +@@ -411,6 +411,16 @@ fn finish(watch: &Watch, lists: &mut FenceLists, mut finished: Vec<(usize, Submi + batch.holds.clear(); + } + } ++ // LOCAL (Atlas): the writes in surfaces' images of the command buffers they end trap before ++ // anything, their signals among them, can tell the guest the work ended ++ // (`shadow_coherent.rs`); the command buffer's own completion finds them armed. ++ if crate::shadow_lazy::ahead() { ++ for (_, batch) in finished.iter().filter(|(_, batch)| batch.last) { ++ if let Some(id) = batch.buffer.shadow_carry_id() { ++ crate::shadow_lazy::completing(id); ++ } ++ } ++ } + // Their event signals are seen before the scheduler looks again: a batch whose submission + // checks wait for a GPU signal to land may now go. + for (_, batch) in &finished { +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 9c8ff0e5..9d984d79 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2182,9 +2182,11 @@ impl CommandBuffer { + /// after an abandoned encoder, and for another device's event. + pub fn encode_signal_event(&self, event: EventRef<'_>, value: u64) -> Result<(), Error> { + // LOCAL (Atlas): a CPU waiter on a shared event reads the pages once it is signalled, +- // before the command buffer completes: what its encoders left ahead in a surface's image +- // goes back first, in the command buffer (`shadow_lazy::signalled_by`). +- if matches!(event, EventRef::Shared(_)) && crate::shadow_lazy::ahead() { ++ // before the command buffer completes, and so does one on a shared event that a command ++ // buffer waiting for a plain one signals, or that completion: what its encoders left ++ // ahead in a surface's image goes back first, in the command buffer ++ // (`shadow_lazy::signalled_by`). ++ if crate::shadow_lazy::ahead() { + self.copy_back_before_signal(); + } + self.event_command("encode_signal_event", event, |batches, event| { +@@ -4350,6 +4352,12 @@ impl CommandBufferShared { + /// versions, what it retained and its encoders' logs, then open its gate, which releases its + /// waiters once the transition's [`Hold`] (and every other) is gone, and they wait for the + /// trails below the mark. Waits from inside these steps answer as [`own`] says. ++ /// LOCAL (Atlas): the identity of what its encoders carry of shadows (`shadow::ShadowCarry`), ++ /// the key of its writes in surfaces' images (`shadow_lazy.rs`). ++ pub(crate) fn shadow_carry_id(&self) -> Option { ++ lock(&self.state).shadow_carry.as_ref().map(ShadowCarry::id) ++ } ++ + fn complete(self: &Arc, result: Result<(), CommandBufferError>) { + // A capture replay found a primitive whose reach was not measured: its pass rewrote + // every tile, announced once the lock is let go (below). +@@ -4369,8 +4377,7 @@ impl CommandBufferShared { + // LOCAL (Atlas): its writes in surfaces' images trap before anything can tell the guest it + // completed (`shadow_coherent.rs`). + if crate::shadow_lazy::ahead() { +- let id = lock(&self.state).shadow_carry.as_ref().map(ShadowCarry::id); +- if let Some(id) = id { ++ if let Some(id) = self.shadow_carry_id() { + crate::shadow_lazy::completing(id); + } + } +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 2a90eafb..32022b28 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -1062,8 +1062,16 @@ impl CommandBufferShared { + shared.interop.wrapped.forget(image); + } + } ++ // LOCAL (Atlas): its signals are seen at its completion turn alone where a trapping ++ // surface holds writes its pages show only once their command buffer's completion turn ++ // arms them (`event_timeline.rs`, `shadow_coherent::unarmed`). ++ let behind_merge = ++ !kept.is_empty() && crate::shadow_lazy::ahead() && crate::shadow_coherent::unarmed(); ++ if behind_merge { ++ crate::shadow_coherent::count_signals_behind_arm(kept.len()); ++ } + for (event, signal, value) in &kept { +- event.keep(*signal, *value, queue); ++ event.keep(*signal, *value, queue, behind_merge); + } + // Its commands may reach every texture through its resource ID where one bound a heap + // set: the stale-reach degrade's trigger decides (`detach.rs`). +diff --git a/crates/metal2vulkan-runtime/src/event.rs b/crates/metal2vulkan-runtime/src/event.rs +index c51805a9..0e8575be 100644 +--- a/crates/metal2vulkan-runtime/src/event.rs ++++ b/crates/metal2vulkan-runtime/src/event.rs +@@ -213,8 +213,10 @@ impl EventShared { + } + + /// The batch of a kept signal was submitted. +- pub(super) fn keep(&self, index: u64, value: u64, queue: usize) { +- lock(&self.state).timeline.keep(index, value, queue); ++ pub(super) fn keep(&self, index: u64, value: u64, queue: usize, behind_merge: bool) { ++ lock(&self.state) ++ .timeline ++ .keep(index, value, queue, behind_merge); + } + + /// The semaphore reached `index`: wake the host's waits and post the notifications that came +diff --git a/crates/metal2vulkan-runtime/src/event_timeline.rs b/crates/metal2vulkan-runtime/src/event_timeline.rs +index e803a2c5..78ed95e9 100644 +--- a/crates/metal2vulkan-runtime/src/event_timeline.rs ++++ b/crates/metal2vulkan-runtime/src/event_timeline.rs +@@ -34,6 +34,17 @@ + //! + //! The held batch goes once the value is reached: a host set wakes the scheduler, and so does a + //! completion turn that observes the GPU signal (`event.rs`). ++//! ++//! **A signal behind a merge is seen at its completion turn alone** ([`EventTimeline::keep`]). ++//! LOCAL (Atlas), after upstream 76d89d87: a batch submitted while a trapping surface ++//! (`shadow_coherent.rs`) holds writes its pages do not show yet, which they show only once the ++//! completion turn of their command buffer arms them; Metal's writes before a signal are in the ++//! pages once anyone sees its value. So no read of the counter observes such a signal, nor any ++//! after it (a read of the counter is a raw read of the GPU's progress), and no wait is handed its ++//! index, the scheduler's own host wait included: the completion turn, which arms first, observes ++//! it, and wakes every host wait and held batch for it. A plain event's wait for such a signal is ++//! held as a shared event's is. The signal itself stays the batch's GPU signal: no submission is ++//! added. + + use crate::command_buffer::completion::seen; + use std::collections::VecDeque; +@@ -45,6 +56,9 @@ struct Pending { + value: u64, + /// The `VkQueue` of the batch that signals it. + queue: Q, ++ /// Seen at its batch's completion turn alone, once the surfaces' pages are armed (the module ++ /// docs). ++ behind_merge: bool, + } + + /// How a batch waits for a Metal value. +@@ -139,6 +153,12 @@ impl EventTimeline { + read: u64, + reaches: impl FnOnce(u64) -> Result, + ) -> Result { ++ // Never past a signal behind a merge, which its completion turn observes (the module ++ // docs). ++ let read = match self.pending.iter().find(|pending| pending.behind_merge) { ++ Some(pending) => read.min(pending.index - 1), ++ None => read, ++ }; + let index = seen(self.reached, None, read, reaches)?; + Ok(self.observe(index)) + } +@@ -165,14 +185,20 @@ impl EventTimeline { + /// holds on the host is one too; `None` while no submitted signal and no host set reaches it. + /// The host waits for the semaphore's index (the scheduler's thread, `scheduler/mod.rs`), and + /// a set meanwhile wakes it as any host signal does: no batch is handed that index. ++ /// A value a signal behind a merge reaches, or one after it, is `None` too: only its ++ /// completion turn observes it (the module docs). + pub(super) fn landing(&self, value: u64) -> Option { + if value <= self.value() { + return Some(WaitFor::Reached); + } +- self.pending +- .iter() +- .find(|pending| pending.value >= value) +- .map(|pending| WaitFor::Index(pending.index)) ++ let mut behind_merge = false; ++ for pending in &self.pending { ++ behind_merge |= pending.behind_merge; ++ if pending.value >= value { ++ return (!behind_merge).then_some(WaitFor::Index(pending.index)); ++ } ++ } ++ None + } + + /// Whether a wait for `value` is held on the host where a Vulkan wait for a submitted GPU +@@ -205,12 +231,14 @@ impl EventTimeline { + } + + /// The batch of a kept signal (`index` from [`EventTimeline::signal`]) of `value` on `queue` +- /// was submitted. +- pub(super) fn keep(&mut self, index: u64, value: u64, queue: Q) { ++ /// was submitted; `behind_merge` where a trapping surface held writes its pages did not show ++ /// then (the module docs). ++ pub(super) fn keep(&mut self, index: u64, value: u64, queue: Q, behind_merge: bool) { + self.pending.push_back(Pending { + index, + value, + queue, ++ behind_merge, + }); + } + +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index b312bc1f..26753f13 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -186,15 +186,23 @@ pub(crate) enum Fallback { + FlushNothingLeft, + /// A command buffer met a surface's bytes through an untracked use alone (`met_by`). + MetUntracked, ++ /// An event signal kept while a surface held writes its pages did not show yet: seen at its ++ /// completion turn alone (`event_timeline.rs`). ++ SignalBehindArm, + } + +-const FALLBACKS: usize = Fallback::MetUntracked as usize + 1; ++const FALLBACKS: usize = Fallback::SignalBehindArm as usize + 1; + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; + + pub(crate) fn count(what: Fallback) { + FALLBACK[what as usize].fetch_add(1, Ordering::Relaxed); + } + ++/// `signals` kept event signals of a batch go behind the arm ([`unarmed`]). ++pub(crate) fn count_signals_behind_arm(signals: usize) { ++ FALLBACK[Fallback::SignalBehindArm as usize].fetch_add(signals as u64, Ordering::Relaxed); ++} ++ + /// The process's counts of what [`Fallback`] names, in `WorkCounts::coherent`: every device's + /// counts show the same, as surfaces trap across devices. Not part of the API. + #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +@@ -235,6 +243,9 @@ pub struct CoherentFallbackCounts { + pub flush_nothing_left: u64, + /// Command buffers that met a surface through an untracked use alone. + pub met_untracked: u64, ++ /// Event signals seen at their completion turn alone, as a surface held writes its pages did ++ /// not show yet when they were submitted (`event_timeline.rs`). ++ pub signals_behind_arm: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -259,6 +270,7 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + host_writes_opened: read(Fallback::HostWriteOpened), + flush_nothing_left: read(Fallback::FlushNothingLeft), + met_untracked: read(Fallback::MetUntracked), ++ signals_behind_arm: read(Fallback::SignalBehindArm), + } + } + +@@ -1474,6 +1486,24 @@ pub(crate) fn surfaces_tracked() -> bool { + !all().read().unwrap().is_empty() + } + ++/// Whether a tracked surface holds writes of a command buffer not completed yet, open or running, ++/// which its pages show only once that completion arms them: a signal submitted now is seen at ++/// its completion turn alone (`event_timeline.rs`). Called by a submission, under the scheduler's ++/// lock: it waits for no lock, and a lock another thread holds answers yes. ++pub(crate) fn unarmed() -> bool { ++ let Ok(all) = all().try_read() else { ++ return true; ++ }; ++ let holds = |state: &State| { ++ state.tracked && !state.closed && !(state.open.is_empty() && state.running.is_empty()) ++ }; ++ all.iter().any(|surface| match surface.state.try_lock() { ++ Ok(state) => holds(&state), ++ Err(std::sync::TryLockError::Poisoned(state)) => holds(&state.into_inner()), ++ Err(std::sync::TryLockError::WouldBlock) => true, ++ }) ++} ++ + /// One line a surface: what [`Coherent::line`] says, under `label`. A surface that went comes once. + pub fn report(label: &str) -> String { + let mut lines: String = std::mem::take(&mut *ended().lock().unwrap_or_else(|e| e.into_inner())).concat(); +diff --git a/crates/metal2vulkan-runtime/src/tests/event_timeline.rs b/crates/metal2vulkan-runtime/src/tests/event_timeline.rs +index 2accb65c..56a961a7 100644 +--- a/crates/metal2vulkan-runtime/src/tests/event_timeline.rs ++++ b/crates/metal2vulkan-runtime/src/tests/event_timeline.rs +@@ -10,7 +10,7 @@ use crate::event_timeline::{EventTimeline, Signal, WaitFor}; + fn submit(timeline: &mut EventTimeline, value: u64, queue: usize) -> Signal { + let signal = timeline.signal(value, queue); + if let Signal::Kept { index, .. } = signal { +- timeline.keep(index, value, queue); ++ timeline.keep(index, value, queue, false); + } + signal + } +@@ -237,3 +237,40 @@ fn a_counter_read_is_observed_only_once_a_host_wait_returned_for_it() { + assert_eq!(timeline.observe_read(3, |_| Err("lost")), Err("lost")); + assert_eq!(timeline.value(), 5); + } ++ ++/// A signal kept behind a merge (its batch may write a copied host buffer, merged into the pages ++/// at its completion turn) is observed by that turn alone: a counter read past it observes only ++/// the signals before it, no wait is handed its index or a later one's (a plain event's wait is ++/// held as a shared event's is), and the completion turn's observation reaches it and those after ++/// it. ++#[test] ++fn a_signal_behind_a_merge_is_seen_at_its_completion_turn_alone() { ++ for settable in [false, true] { ++ let mut timeline = EventTimeline::new(settable); ++ for (value, behind_merge) in [(3, false), (5, true), (7, false)] { ++ let Signal::Kept { index, .. } = timeline.signal(value, 0) else { ++ panic!("kept"); ++ }; ++ timeline.keep(index, value, 0, behind_merge); ++ } ++ // The counter shows all three: the read observes the first alone. ++ assert_eq!(timeline.observe_read::<()>(3, |_| Ok(true)), Ok(true)); ++ assert_eq!(timeline.value(), 3, "{settable}"); ++ assert_eq!(timeline.observe_read::<()>(3, |_| Ok(true)), Ok(false)); ++ assert_eq!(timeline.value(), 3, "{settable}"); ++ // No wait is handed the index of the signal behind the merge, or a later one's. ++ for value in [4, 5, 6, 7] { ++ assert_eq!(timeline.landing(value), None, "{settable} {value}"); ++ assert_eq!(timeline.wait(value), None, "{settable} {value}"); ++ } ++ assert_eq!(timeline.wait(3), Some(WaitFor::Reached)); ++ // The completion turn, after the merge, observes it. ++ assert!(timeline.observe(2)); ++ assert_eq!(timeline.value(), 5, "{settable}"); ++ // A later signal not behind a merge is read and waited for as before. ++ let want = (!settable).then_some(WaitFor::Index(3)); ++ assert_eq!(timeline.wait(7), want, "{settable}"); ++ assert_eq!(timeline.observe_read::<()>(3, |_| Ok(true)), Ok(true)); ++ assert_eq!(timeline.value(), 7, "{settable}"); ++ } ++} +diff --git a/crates/metal2vulkan-runtime/src/tests/host_settable_event_waits.rs b/crates/metal2vulkan-runtime/src/tests/host_settable_event_waits.rs +index f7557c83..c5761e1d 100644 +--- a/crates/metal2vulkan-runtime/src/tests/host_settable_event_waits.rs ++++ b/crates/metal2vulkan-runtime/src/tests/host_settable_event_waits.rs +@@ -17,7 +17,7 @@ use crate::event_timeline::{EventTimeline, Signal, WaitFor}; + /// Submit a GPU signal of `value` on `queue`, kept when it is. + fn submit(timeline: &mut EventTimeline, value: u64, queue: usize) { + if let Signal::Kept { index, .. } = timeline.signal(value, queue) { +- timeline.keep(index, value, queue); ++ timeline.keep(index, value, queue, false); + } + } + +@@ -87,7 +87,7 @@ fn the_signals_of_either_kind_take_the_same_indices() { + after: None + } + ); +- timeline.keep(1, 4, 0); ++ timeline.keep(1, 4, 0, false); + assert_eq!(timeline.signal(4, 1), Signal::Dropped); + assert_eq!( + timeline.signal(5, 1), +diff --git a/crates/metal2vulkan-runtime/src/tests/scheduler.rs b/crates/metal2vulkan-runtime/src/tests/scheduler.rs +index 9a72a25a..dc9c0d7a 100644 +--- a/crates/metal2vulkan-runtime/src/tests/scheduler.rs ++++ b/crates/metal2vulkan-runtime/src/tests/scheduler.rs +@@ -155,7 +155,7 @@ impl Model { + let mut signaled = Vec::new(); + for &(event, value) in &spec.signals { + if let Signal::Kept { index, after } = self.events[event].signal(value, vk_queue) { +- self.events[event].keep(index, value, vk_queue); ++ self.events[event].keep(index, value, vk_queue, false); + self.kept[event] = index; + signaled.push((event, index)); + waits.extend(after.map(|after| (event, after))); +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index ccd8299c..795f63f4 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -21,7 +21,8 @@ use metal2vulkan_runtime::shadow_lazy; + use metal2vulkan_runtime::{ + live, Buffer, CommandBuffer, CommandQueue, CpuCacheMode, Device, HazardTrackingMode, + HostMemory, IoSurfacePlane, LoadAction, PipelineOptions, RenderPassDescriptor, +- RenderPipelineState, ResourceOptions, ScissorRect, StorageMode, StoreAction, Texture, ++ RenderPipelineState, ResourceOptions, ScissorRect, SharedEvent, SharedEventListener, ++ StorageMode, StoreAction, Texture, + TextureDescriptor, + }; + use std::collections::BTreeMap; +@@ -1565,3 +1566,155 @@ fn surface_coherent_a_reader_on_another_device_reads_the_writes() { + assert!(reported.is_empty(), "the layer reported: {reported:#?}"); + }); + } ++ ++/// The guest reads the texel at `offset` of the pages from `base` on another thread (a listener's): ++/// a marked page stops it first, as [`Case::read`]'s. ++fn guest_read_at(fake: &Fake, base: u64, offset: u64) -> [u8; 4] { ++ let addr = base + offset; ++ if matches!( ++ fake.mark(addr & !(PAGE - 1)), ++ Some(Mark::Armed | Mark::Frozen) ++ ) { ++ shadow_coherent::fault(addr, false); ++ } ++ // SAFETY: the case's mapping outlives every listener the lanes post (each waits for it). ++ unsafe { std::ptr::read_volatile(addr as *const [u8; 4]) } ++} ++ ++/// E1: a shared event signalled by a command buffer after the one that wrote, on the queue, whose ++/// completion has not been seen yet (a completed handler of an earlier command buffer holds the ++/// completions back): once the value is seen, by a read of the counter, a host wait or a listener, ++/// the writes are in the pages. Before, the value rose with the semaphore and the completion turn ++/// saw the signal before it armed the writer's pages, which stayed held with the older bytes. ++#[test] ++fn surface_coherent_a_signal_after_the_writer_reads_its_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ let (go, gate) = std::sync::mpsc::channel::<()>(); ++ let first = case.queue.command_buffer().unwrap(); ++ (first.add_completed_handler(Box::new(move |_: &CommandBuffer| { ++ let _ = gate.recv_timeout(SOON); ++ }))) ++ .unwrap(); ++ first.commit().unwrap(); ++ let writer = case.queue.command_buffer().unwrap(); ++ case.pass(&writer, GREEN, None); ++ writer.commit().unwrap(); ++ let signal = case.device.new_shared_event().unwrap(); ++ let listener = SharedEventListener::new(); ++ let (heard, hear) = std::sync::mpsc::channel(); ++ let (fake, base) = (case.fake.clone(), case.base()); ++ signal.notify_listener( ++ &listener, ++ 1, ++ Box::new(move |_: &SharedEvent, _| { ++ let _ = heard.send(guest_read_at(&fake, base, 5 * ROW + 5 * 4)); ++ }), ++ ); ++ let signaller = case.queue.command_buffer().unwrap(); ++ signaller.encode_signal_event((&signal).into(), 1).unwrap(); ++ signaller.commit().unwrap(); ++ let start = Instant::now(); ++ let mut early = None; ++ while start.elapsed() < Duration::from_millis(300) { ++ if signal.signaled_value() >= 1 { ++ early = Some(case.read(5, 5)); ++ break; ++ } ++ std::thread::yield_now(); ++ } ++ let _ = go.send(()); ++ assert!( ++ signal.wait_until_signaled_value(1, SOON), ++ "the GPU reached the signal" ++ ); ++ let late = case.read(5, 5); ++ let listened = hear.recv_timeout(SOON).unwrap(); ++ for command_buffer in [&first, &writer, &signaller] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ assert_eq!( ++ early.unwrap_or(GREEN_TEXEL), ++ GREEN_TEXEL, ++ "the counter showed the signal" ++ ); ++ assert_eq!(listened, GREEN_TEXEL, "the listener heard the signal"); ++ assert_eq!(late, GREEN_TEXEL, "a host wait returned for the signal"); ++ }); ++} ++ ++/// E1 through a plain event: the writer signals an `MTLEvent` after its pass and then waits; a ++/// command buffer on another queue waits for that event and signals a shared one. The shared ++/// event's value promises the writer's pass, whose command buffer has not completed. ++#[test] ++fn surface_coherent_a_plain_event_chain_reads_the_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(5, 5), RED_TEXEL); ++ let plain = case.device.new_event().unwrap(); ++ let signal = case.device.new_shared_event().unwrap(); ++ let hold = case.device.new_shared_event().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = case.queue.command_buffer().unwrap(); ++ case.pass(&writer, GREEN, None); ++ writer.encode_signal_event((&plain).into(), 1).unwrap(); ++ writer.encode_wait_for_event((&hold).into(), 1).unwrap(); ++ writer.commit().unwrap(); ++ let relay = other.command_buffer().unwrap(); ++ relay.encode_wait_for_event((&plain).into(), 1).unwrap(); ++ relay.encode_signal_event((&signal).into(), 1).unwrap(); ++ relay.commit().unwrap(); ++ let seen = signal.wait_until_signaled_value(1, SOON); ++ let read = case.read(5, 5); ++ hold.set_signaled_value(1); ++ writer.wait_until_completed_for(SOON).unwrap(); ++ relay.wait_until_completed_for(SOON).unwrap(); ++ assert!(seen, "the relay signalled"); ++ assert_eq!(read, GREEN_TEXEL, "what the chain promised"); ++ }); ++} ++ ++/// E1, the other way: a command buffer committed to wait for a shared event before it draws over ++/// the surface; the guest writes a texel, then sets the event. Metal's draw sees the write, so its ++/// copy in must come after the wait, not as the command was encoded. Open: the encoder's copy in ++/// is decided as it is encoded, so the write lands in the pages (kept by the merge) but the draw ++/// never sees it; a fix needs a copy in recorded as the held batch is submitted. ++#[test] ++#[ignore = "open (E1 copy in): fails at 76ab0722 and after the signal fix; run by name"] ++fn surface_coherent_a_cpu_write_before_a_signal_reaches_a_waiting_command() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let color = staged(case.device, &floats4([0.0; 4])); ++ let rects = std::slice::from_ref(&case.rect); ++ let blending = rects_blending(case.device); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ command_buffer ++ .encode_wait_for_event((&hold).into(), 1) ++ .unwrap(); ++ pass( ++ &command_buffer, ++ case.texture(), ++ &blending, ++ rects, ++ &color, ++ None, ++ ); ++ command_buffer.commit().unwrap(); ++ case.write(0, 0, [1, 2, 3, 4]); ++ hold.set_signaled_value(1); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!( ++ case.read(0, 0), ++ [1, 2, 3, 0], ++ "the draw blended over the guest's write" ++ ); ++ assert_eq!( ++ case.read(1, 0), ++ [0, 0, 255, 0], ++ "and over the GPU's texels beside it" ++ ); ++ }); ++} diff --git a/patches/metal2vulkan/0024-coherent-a-held-batch-copies-in-what-the-CPU-wrote-b.patch b/patches/metal2vulkan/0024-coherent-a-held-batch-copies-in-what-the-CPU-wrote-b.patch new file mode 100644 index 0000000..c94b9bb --- /dev/null +++ b/patches/metal2vulkan/0024-coherent-a-held-batch-copies-in-what-the-CPU-wrote-b.patch @@ -0,0 +1,453 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 09:12:51 +0000 +Subject: [PATCH 24/34] coherent: a held batch copies in what the CPU wrote + before its signal (E1 copy in) + +The last open E1 path: a command buffer committed to wait for a shared event +draws over a trapping surface the guest writes before it sets the event. The +encoder took the image's standing as it was encoded (Coherent::taking found no +CPU write then), so it recorded no copy in, and the draw ran over the image +without the write Metal's draw sees. + +The copy in is now decided as the held batch goes: + +- shadow::LateCopies (in ShadowCarry): encode_wait_for_event on a shared event + marks the command buffer; from then on an encoder that takes a trapping + surface's standing (shadow_lazy's take answered Some) notes the surface, + once. A surface an earlier encoder of the same command buffer left writes + ahead in is not noted (Lazy::left_open_by): the copy in would replace them + with the pages' bytes. A texture copied in at its use needs nothing: that + copy reads the pages as the GPU runs it. An MTLEvent wait adds nothing. +- submit_batch, after place(): record_late_copies_in takes the noted surfaces + that the batch's parts use, and for each asks Coherent::taking_late, which + answers the guest's rectangle (and holds the open pages, as taking does), or + nothing where the CPU wrote nothing. The copies go in one command buffer at + the start of the batch's first part, their uses joined to its summary, so + the barrier the tracker plans before the part orders them after earlier + writers, then a barrier before the part's own work. +- taking_late runs under the scheduler's lock and waits for no record: a write + of another command buffer still running over the rectangle is counted + (overwrite_risk, CoherentFallbackCounts::late_copies_in_busy) where taking + would wait for a bounded time. The command buffer's own records run after + the copy and are not counted as busy. +- Counted: CoherentFallbackCounts::late_copies_in, late_copies_in_busy. + +Cost where nothing changed: nothing unless a command buffer waits for a +shared event and then takes a trapping surface's standing; then one surface +lock per noted surface when its batch goes, and no command buffer. + +Lane surface_coherent_a_cpu_write_before_a_signal_reaches_a_waiting_command is +no longer ignored, and it also checks that late_copies_in moved by one. + +Lanes (filters "surface" and "event", ANV, one run): 154 passed, 1 failed, +6 ignored before; 155 passed, 1 failed, 5 ignored after. The failure is +ycbcr_textures::textures_over_any_surface_read_what_metal_reads, as before. +Runtime unit tests: 1149 passed, 2 failed. exit_guard fails as before (the +same two submit.rs Stamps lines). driver_defects' +a_set_past_the_drivers_support_is_among_its_variants_kinds stops on a missing +llvm-as in the build container, before anything this commit touches. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/mod.rs | 18 +++++- + .../src/command_buffer/submit.rs | 64 +++++++++++++++++++ + crates/metal2vulkan-runtime/src/shadow.rs | 51 +++++++++++++++ + .../src/shadow_coherent.rs | 63 +++++++++++++++--- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 26 ++++++++ + .../src/runtime_lane/surface_coherent.rs | 12 ++-- + 6 files changed, 218 insertions(+), 16 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 9d984d79..e636538b 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -1299,7 +1299,13 @@ impl State { + /// Carry no shadowed texture's image on: the work encoded next reads what another writer the + /// wait orders before it left in the truth, which every texture copies in again to hold + /// (`shadow::ShadowCarry`). +- pub(super) fn shadows_not_carried(&mut self) { ++ /// LOCAL (Atlas): the work encoded next waits for a shared event, which the host may set ++ /// after a CPU write a trapping surface's copy in must take (`shadow::LateCopies`). ++ pub(super) fn shadows_wait_for_shared_event(&mut self) { ++ (self.shadow_carry.get_or_insert_with(ShadowCarry::default)).waits_for_shared_event(); ++ } ++ ++ pub(super) fn shadows_not_carried(&mut self) { + if let Some(carry) = &mut self.shadow_carry { + carry.clear(); + } +@@ -2241,10 +2247,18 @@ impl CommandBuffer { + /// `encodeWaitForEvent:value:`: the work encoded after it starts once the event's value + /// reached `value`. Refused as [`CommandBuffer::encode_signal_event`] is. + pub fn encode_wait_for_event(&self, event: EventRef<'_>, value: u64) -> Result<(), Error> { ++ let settable = matches!(event, EventRef::Shared(_)); + self.event_command("encode_wait_for_event", event, |batches, event| { + batches.wait(event, value); + }) +- .inspect(|()| self.state().shadows_not_carried()) ++ .inspect(|()| { ++ let mut state = self.state(); ++ state.shadows_not_carried(); ++ // LOCAL (Atlas): the host may set it (`shadow::LateCopies`). ++ if settable { ++ state.shadows_wait_for_shared_event(); ++ } ++ }) + } + + /// Wait for the caller's binary semaphore `semaphore` before the work encoded after it: the +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 32022b28..f415cc30 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -458,6 +458,8 @@ impl CommandBufferShared { + state + .place(range.clone()) + .map_err(|error| recording_error(device, error))?; ++ // LOCAL (Atlas): what the CPU wrote of trapping surfaces before the signal it waited for. ++ self.record_late_copies_in(&mut state, range.clone())?; + // No work of an encoder that recorded nothing for a command (a buffer it named was + // detached), nor of one whose submission failed rebound or detached, goes to the queue. + state +@@ -1179,6 +1181,68 @@ impl CommandBufferShared { + Ok(buffer) + } + ++ /// LOCAL (Atlas): the copies in of trapping surfaces whose standing an encoder of the batch of ++ /// encoders `range` took after a wait for a shared event (`shadow::LateCopies`): each of what ++ /// the guest's CPU wrote since, in one command buffer at the start of the batch's first part, ++ /// after its waits, its uses joined to that part's summary so the barrier planned before it ++ /// orders them after earlier writers, and a barrier after them, before the part's own work. ++ /// Nothing, and no command buffer, where the CPU wrote nothing (counted: ++ /// `CoherentFallbackCounts::late_copies_in`). ++ fn record_late_copies_in( ++ &self, ++ state: &mut State, ++ range: std::ops::Range, ++ ) -> Result<(), CommandBufferError> { ++ let Some(carry) = state.shadow_carry.as_mut() else { ++ return Ok(()); ++ }; ++ let recorded = &state.recorded[range.clone()]; ++ let shadows = carry.take_late(|copies| { ++ (recorded.iter()).any(|part| { ++ (part.summary.lazy_uses()) ++ .any(|used| used.key == copies.shadow_key || copies.meets(used)) ++ }) ++ }); ++ if shadows.is_empty() || range.is_empty() { ++ return Ok(()); ++ } ++ let id = carry.id(); ++ let copies: Vec<_> = (shadows.iter()) ++ .filter_map(|copies| crate::shadow_lazy::late_copy_in(copies, id)) ++ .collect(); ++ if copies.is_empty() { ++ return Ok(()); ++ } ++ let device = &self.device; ++ let mut uses = Vec::new(); ++ for copy in &copies { ++ uses.extend(copy.uses_with(None)); ++ } ++ let mut aliased = Vec::new(); ++ if device.shared().aliases.augment_into(&uses, &mut aliased) { ++ uses = aliased; ++ } ++ let raw = self.record_raw(state, |_, buffer| { ++ let vk = device.shared().vk(); ++ // SAFETY: the command buffer is recording; the textures' images and the buffers over ++ // the pages are live (the copies hold them, kept with the command buffer), in ++ // `GENERAL`; the barrier the tracker plans before the part orders the copies after ++ // earlier writers, and this one the part's work after them. ++ unsafe { ++ for copy in &copies { ++ copy.record(vk, buffer, None); ++ } ++ Barrier::everything().record(vk, buffer); ++ } ++ Ok(()) ++ })?; ++ let first = &mut state.recorded[range.start]; ++ first.raws.insert(0, raw); ++ first.summary.join(Summary::outside_encoders(uses)); ++ state.retained.push(Box::new(copies)); ++ Ok(()) ++ } ++ + /// Record the scrub segments of encoders `range` (`Deferred::SparseScrub`), which their batch's + /// submission records with the mappings it applied. + fn record_scrubs( +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index fbf6af58..a7ab6828 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2241,6 +2241,7 @@ impl ShadowSync { + id: self.carry.id, + entries: Vec::new(), + left: self.carry.left, ++ late: std::mem::take(&mut self.carry.late), + }; + carry.entries.append(&mut self.carry.entries); + for entry in &self.entries { +@@ -2317,6 +2318,24 @@ pub(super) struct ShadowCarry { + /// LOCAL (Atlas): an encoder of it left writes ahead in a trapping surface's image, which a + /// later command that meets the bytes copies back (`shadow_lazy::met_by_command`). + left: bool, ++ /// LOCAL (Atlas): the copies in its held batches take as they go ([`LateCopies`]). ++ late: LateCopies, ++} ++ ++/// LOCAL (Atlas): the trapping surfaces (`shadow_coherent.rs`) whose standing an encoder took ++/// after a wait for a shared event, which copy in what the guest's CPU wrote since, as the batch ++/// that waited goes (`command_buffer/submit.rs`, `shadow_lazy::late_copy_in`). Metal's commands ++/// after `encodeWaitForEvent:` see the CPU's writes before the event's `setSignaledValue:`, and ++/// the copy in an encoder takes as it is encoded ([`Coherent::taking`]) sees only those before ++/// it. A wait for an `MTLEvent`, which no host call sets, adds none, nor does a texture copied in ++/// at its use: that copy reads the pages as the GPU runs it. ++/// ++/// [`Coherent::taking`]: super::shadow_coherent ++#[derive(Default)] ++pub(super) struct LateCopies { ++ /// The command buffer waits for a shared event before the work encoded from here on. ++ waited: bool, ++ shadows: Vec>, + } + + impl Default for ShadowCarry { +@@ -2327,6 +2346,7 @@ impl Default for ShadowCarry { + id: NEXT.fetch_add(1, Ordering::Relaxed), + entries: Vec::new(), + left: false, ++ late: LateCopies::default(), + } + } + } +@@ -2375,6 +2395,24 @@ impl ShadowCarry { + pub(super) fn clear(&mut self) { + self.entries.clear(); + } ++ ++ /// LOCAL (Atlas): the work encoded from here on waits for a shared event ([`LateCopies`]). ++ pub(super) fn waits_for_shared_event(&mut self) { ++ self.late.waited = true; ++ } ++ ++ /// LOCAL (Atlas): the shadows of the late copies in ([`LateCopies`]) that `met` says a batch ++ /// going now uses, taken out: each goes with the first such batch. ++ pub(super) fn take_late(&mut self, met: impl Fn(&ShadowCopies) -> bool) -> Vec> { ++ if self.late.shadows.is_empty() { ++ return Vec::new(); ++ } ++ let (taken, kept) = std::mem::take(&mut self.late.shadows) ++ .into_iter() ++ .partition(|copies| met(copies)); ++ self.late.shadows = kept; ++ taken ++ } + } + + /// What a merge's scratch keeps of a shadow's texels (`shadow_merge.rs`): the rectangle +@@ -2414,6 +2452,19 @@ impl ShadowSync { + (None, Some(lazy)) => lazy.take(self.carry.id, met), + _ => None, + }; ++ // LOCAL (Atlas): a trapping surface's standing taken after a wait for a shared ++ // event: the CPU may write before the signal, which the batch copies in as it ++ // goes ([`LateCopies`]). Not over writes an earlier encoder of this command ++ // buffer left ahead, which the copy in would replace with the pages' bytes. ++ if let (Some(_), Some(lazy), true) = (&lazily, &copies.lazy, self.carry.late.waited) { ++ let late = &mut self.carry.late.shadows; ++ if lazy.whole_image() ++ && !late.iter().any(|known| known.shadow_key == copies.shadow_key) ++ && !lazy.left_open_by(self.carry.id) ++ { ++ late.push(Arc::clone(copies)); ++ } ++ } + let only_read = lazily.is_some() && (copies.lazy.as_ref()) + .is_some_and(|lazy| lazy.whole_image()); + let (standing, held, stale) = match (&carried, lazily) { +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 26753f13..44e4d1f0 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -189,9 +189,15 @@ pub(crate) enum Fallback { + /// An event signal kept while a surface held writes its pages did not show yet: seen at its + /// completion turn alone (`event_timeline.rs`). + SignalBehindArm, ++ /// A held batch went with a copy in of what the CPU wrote after its encoders took the image's ++ /// standing ([`Coherent::taking_late`]). ++ LateCopyIn, ++ /// Of those, the copies in made while another command buffer's write over the rectangle still ++ /// ran, whose texels the copy may lose: the submission waits for nothing. ++ LateCopyInBusy, + } + +-const FALLBACKS: usize = Fallback::SignalBehindArm as usize + 1; ++const FALLBACKS: usize = Fallback::LateCopyInBusy as usize + 1; + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; + + pub(crate) fn count(what: Fallback) { +@@ -246,6 +252,11 @@ pub struct CoherentFallbackCounts { + /// Event signals seen at their completion turn alone, as a surface held writes its pages did + /// not show yet when they were submitted (`event_timeline.rs`). + pub signals_behind_arm: u64, ++ /// Held batches that went with a copy in of what the CPU wrote after their encoders took the ++ /// image's standing: a write before the signal a batch waited for. ++ pub late_copies_in: u64, ++ /// Of those, the ones made while another command buffer's write over the rectangle ran. ++ pub late_copies_in_busy: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -271,6 +282,8 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + flush_nothing_left: read(Fallback::FlushNothingLeft), + met_untracked: read(Fallback::MetUntracked), + signals_behind_arm: read(Fallback::SignalBehindArm), ++ late_copies_in: read(Fallback::LateCopyIn), ++ late_copies_in_busy: read(Fallback::LateCopyInBusy), + } + } + +@@ -1123,15 +1136,7 @@ impl Coherent { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } + } +- self.sync_done(&mut state, SyncBy::CopyIn); +- let list: Vec = state.snapshots.keys().copied().collect(); +- if let Some(mark) = pages() { +- let came = mark.hold(&list, true); +- state.guest = None; +- self.note_came(&mut state, came); +- } +- state.snapshots.clear(); +- state.opened = false; ++ self.take_guest(&mut state); + if whole { + state.guest = None; + } +@@ -1139,6 +1144,44 @@ impl Coherent { + stale + } + ++ /// The pages hold everything done, then the open ones are held: the copy in about to read ++ /// them reads all the guest wrote, and its next write traps. ++ fn take_guest(&self, state: &mut State) { ++ self.sync_done(state, SyncBy::CopyIn); ++ let list: Vec = state.snapshots.keys().copied().collect(); ++ if let Some(mark) = pages() { ++ let came = mark.hold(&list, true); ++ state.guest = None; ++ self.note_came(state, came); ++ } ++ state.snapshots.clear(); ++ state.opened = false; ++ } ++ ++ /// A held batch of the command buffer `id` is about to go, whose encoders took the image's ++ /// standing as they were encoded ([`taking`](Self::taking)): where the guest's CPU wrote ++ /// since, before the signal the batch waited for, the rectangle its copy in takes, recorded ++ /// at the batch's start (`command_buffer/submit.rs`). Called under the scheduler's lock, it ++ /// waits for no record: a write of another command buffer still running over the rectangle ++ /// is counted (`overwrite_risk`, [`Fallback::LateCopyInBusy`]), as a bounded wait past its ++ /// bound is. The command buffer's own records run after the copy. ++ pub(crate) fn taking_late(&self, id: u64) -> Option<[u32; 4]> { ++ let mut state = self.lock(); ++ if !state.tracked || state.closed { ++ return None; ++ } ++ let area = state.guest?; ++ let busy = state.flushing != 0 ++ || (state.running.iter()).any(|record| record.id != id && meet(record.area, area)); ++ if busy && !state.snapshots.is_empty() { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ count(Fallback::LateCopyInBusy); ++ } ++ self.take_guest(&mut state); ++ count(Fallback::LateCopyIn); ++ Some(area) ++ } ++ + /// A GPU copy back of what ran is about to record: everything done went on the CPU, and what + /// is still running goes by the copy, up to the sequence given. `None` where nothing runs. The + /// lock is not held while it records. +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index a4340670..76e0986b 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -511,6 +511,25 @@ pub(crate) fn signalled_by(id: u64) -> Option, ShadowCopy, + any.then_some(copies) + } + ++/// The copy in a held batch of the command buffer `id` takes as it goes, of a trapping surface ++/// an encoder of it took the standing of after a wait for a shared event (`shadow::LateCopies`): ++/// what the guest's CPU wrote since that encoding, before the signal. `None` where it wrote ++/// nothing, or the surface no longer traps. ++pub(crate) fn late_copy_in(copies: &Arc, id: u64) -> Option { ++ let lazy = copies.lazy.as_ref()?; ++ let area = lazy.coherent()?.taking_late(id)?; ++ let copy = ShadowCopy { ++ copies: Arc::clone(copies), ++ step: Step::Into, ++ kept: false, ++ area: Some(area), ++ cause: Some(CopyCause::Bounded), ++ moves: None, ++ }; ++ lazy.copied_in(CopyCause::Bounded, copy.texels_copied_in()); ++ Some(copy) ++} ++ + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what + /// that command buffer's earlier encoders left ahead in a trapping surface whose bytes it meets. + /// The record stays, whose texels the copy repeats, so what the record holds lasts the copy. +@@ -1142,6 +1161,13 @@ impl Lazy { + }); + } + ++ /// Whether an encoder of the command buffer of identity `id` left writes ahead in its image, ++ /// not committed yet. ++ pub(super) fn left_open_by(&self, id: u64) -> bool { ++ (self.pending.lock().unwrap().as_ref()) ++ .is_some_and(|left| left.open.iter().any(|(open, _)| *open == id)) ++ } ++ + /// The encoder's end left the image not holding the truth. + pub(super) fn fall(&self) { + *self.stood.lock().unwrap() = None; +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 795f63f4..031fc9ba 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -1677,11 +1677,9 @@ fn surface_coherent_a_plain_event_chain_reads_the_writes() { + + /// E1, the other way: a command buffer committed to wait for a shared event before it draws over + /// the surface; the guest writes a texel, then sets the event. Metal's draw sees the write, so its +-/// copy in must come after the wait, not as the command was encoded. Open: the encoder's copy in +-/// is decided as it is encoded, so the write lands in the pages (kept by the merge) but the draw +-/// never sees it; a fix needs a copy in recorded as the held batch is submitted. ++/// copy in must come after the wait, not as the command was encoded: the encoder took the image's ++/// standing, and the held batch copies in what the CPU wrote as it goes (`shadow::LateCopies`). + #[test] +-#[ignore = "open (E1 copy in): fails at 76ab0722 and after the signal fix; run by name"] + fn surface_coherent_a_cpu_write_before_a_signal_reaches_a_waiting_command() { + on_surface(true, |case| { + case.draw(RED, None); +@@ -1703,9 +1701,15 @@ fn surface_coherent_a_cpu_write_before_a_signal_reaches_a_waiting_command() { + None, + ); + command_buffer.commit().unwrap(); ++ let late = shadow_coherent::fallback_counts().late_copies_in; + case.write(0, 0, [1, 2, 3, 4]); + hold.set_signaled_value(1); + command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!( ++ shadow_coherent::fallback_counts().late_copies_in, ++ late + 1, ++ "the held batch copied in as it went" ++ ); + assert_eq!( + case.read(0, 0), + [1, 2, 3, 0], diff --git a/patches/metal2vulkan/0025-a-flat-grid-counts-in-32-bits-when-the-d.patch b/patches/metal2vulkan/0025-a-flat-grid-counts-in-32-bits-when-the-d.patch new file mode 100644 index 0000000..f24b839 --- /dev/null +++ b/patches/metal2vulkan/0025-a-flat-grid-counts-in-32-bits-when-the-d.patch @@ -0,0 +1,231 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 08:43:11 +0000 +Subject: [PATCH 25/34] a flat grid counts in 32 bits when the + dispatch fits + +The flat variant's invocation index and its divisions by the grid were 64-bit for every +dispatch. Intel GPUs have no 64-bit integer divider, so each of the six 64-bit div/mod became +a long software sequence per invocation. The Simulator's masked blit (SimStreamProcessorObjects +blit_texture_to_texture_with_mask, 1206 x 2622 in 32 x 32 threadgroups, one per frame) spent +about 3.6 of its 4.9 ms there on Arrow Lake. + +Now a dispatch-uniform test (nx*ny*nz*N <= 2^32, each product checked before the next can +wrap) selects a 32-bit arm; past it the previous 64-bit arm runs unchanged. Lab on ARL +(Mesa 26.2.4): the captured kernel 4.85 -> 1.13 ms, Apple's metallib through the runtime +p50 5.0-5.6 -> 1.6-2.0 ms with byte-identical outputs. + +New lane: a ragged direct dispatch (the 1206 x 2622 screen, 3-D grids) runs on the flat +variant and each thread runs once at its own position, checked against a CPU expectation. + +Co-Authored-By: Claude Opus 5.5 +(cherry picked from commit 9dbcffd80ced40c59040562a792145df7fa92cce) +--- + .../src/runtime_lane/indirect_dispatches.rs | 58 +++++++++ + crates/metal2vulkan/src/lower/dispatch.rs | 112 +++++++++++++----- + 2 files changed, 140 insertions(+), 30 deletions(-) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs b/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs +index 403185c5..49fcfac2 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs +@@ -707,3 +707,61 @@ fn an_indirect_threadgroup_dispatch_runs_on_the_gpu() { + ); + }); + } ++ ++/// A direct `dispatchThreads:` of a ragged grid runs on the flat variant (one dispatch, no split) ++/// and each of Metal's threads runs once, at its own position: the Simulator's 1206 x 2622 screen ++/// in 32 x 32 threadgroups (its masked blit's dispatch, 3.16 M threads, the flat count's 32-bit ++/// form), and ragged 3-D grids. The expectation is computed here, not by another dispatch. ++#[test] ++fn a_ragged_direct_dispatch_runs_each_thread_once_on_the_flat_variant() { ++ on_device(|device, messages| { ++ let kernel = flat_state(device); ++ for (grid, group) in [ ++ ([1206u32, 2622, 1], [32u32, 32, 1]), ++ ([1206, 2622, 1], [32, 8, 1]), ++ ([37, 19, 5], [4, 4, 2]), ++ ([1000, 3, 2], [64, 1, 1]), ++ ] { ++ let slots = u64::from(grid[0] * grid[1] * grid[2]); ++ let params = buffer_with(device, &bytes(&[1000])).unwrap(); ++ let out = buffer_with(device, &vec![0; slots as usize * 4]).unwrap(); ++ let queue = device.new_command_queue().unwrap(); ++ let command_buffer = queue.command_buffer().unwrap(); ++ let mut encoder = command_buffer.compute_command_encoder().unwrap(); ++ encoder.set_compute_pipeline_state(&kernel).unwrap(); ++ encoder.set_buffer(Some(¶ms), 0, 0).unwrap(); ++ encoder.set_buffer(Some(&out), 0, 1).unwrap(); ++ encoder.dispatch_threads(size(grid), size(group)).unwrap(); ++ encoder.end_encoding().unwrap(); ++ complete(&command_buffer, &format!("ragged {grid:?} {group:?}")); ++ let flat = command_buffer ++ .executions() ++ .iter() ++ .any(|execution| match execution.pipeline() { ++ ExecutedPipeline::Compute(pipeline) => pipeline ++ .dependencies ++ .iter() ++ .any(|dependency| dependency.fact == RuntimeFact::FlatGrid), ++ ExecutedPipeline::Render(_) => false, ++ }); ++ assert!(flat, "{grid:?} {group:?}: not on the flat variant"); ++ let words = read_buffer(&out, 0, slots * 4).unwrap(); ++ let [gx, gy, gz] = grid; ++ let common = 1000 + 11 * gx + 13 * gy + 17 * gz; ++ let mut wrong = 0u64; ++ let mut first = None; ++ for (i, w) in words.as_chunks::<4>().0.iter().enumerate() { ++ let i = i as u32; ++ let (x, y, z) = (i % gx, (i / gx) % gy, i / (gx * gy)); ++ let want = common + 3 * x + 5 * y + 7 * z; ++ let got = u32::from_le_bytes(*w); ++ if got != want { ++ wrong += 1; ++ first.get_or_insert((x, y, z, got, want)); ++ } ++ } ++ assert_eq!(wrong, 0, "{grid:?} {group:?}: first wrong {first:?}"); ++ } ++ let _ = messages.take_notices(); ++ }); ++} +diff --git a/crates/metal2vulkan/src/lower/dispatch.rs b/crates/metal2vulkan/src/lower/dispatch.rs +index e4161ffe..6a89003d 100644 +--- a/crates/metal2vulkan/src/lower/dispatch.rs ++++ b/crates/metal2vulkan/src/lower/dispatch.rs +@@ -1267,8 +1267,15 @@ impl KernelDecl { + /// A flat grid's thread position in the grid (a `uvec3`) and whether the invocation is inside + /// the grid: its index `((wz · ny + wy) · nx + wx) · N + LocalInvocationIndex` over the + /// dispatch, with `N = sx·sy·sz`, counted over `threads_per_grid` (`region[1].xyz`) `x` +- /// fastest, all in 64 bits. Inside the grid each axis's position is below its size, so it fits +- /// its `u32`; past it the position is unused. ++ /// fastest. Inside the grid each axis's position is below its size, so it fits its `u32`; ++ /// past it the position is unused. ++ /// ++ /// The count is exact in 32 bits when the dispatch has at most 2^32 invocations ++ /// (`nx · ny · nz · N <= 2^32`, so every index is below 2^32): then the index and its ++ /// divisions are 32-bit; otherwise 64-bit. The test reads only dispatch-uniform values, so ++ /// every invocation takes the same arm. Devices without a 64-bit divider (Intel's) lower each ++ /// 64-bit division to a long software sequence: on Arrow Lake the Simulator's 1206 x 2622 ++ /// masked blit ran 4.85 ms with the 64-bit form and 1.13 ms with the 32-bit one. + fn flat( + &self, + fb: &mut FunctionBuilder<'_>, +@@ -1278,42 +1285,87 @@ impl KernelDecl { + width: 64, + signed: false, + })?; +- let wide = |fb: &mut FunctionBuilder<'_>, v: SValue, axis: u32| -> BResult { +- let x = fb.composite_extract(v, &[axis])?; +- fb.convert(ConvOp::UConvert, x, u64_) +- }; ++ let u32_ = fb.module().ty(Type::Int { ++ width: 32, ++ signed: false, ++ })?; + let wg = self.load(fb, self.workgroup_id)?; + let groups = self.load(fb, self.num_workgroups)?; + let lin = self.load(fb, self.local_index)?; +- let lin = fb.convert(ConvOp::UConvert, lin, u64_)?; + let s = self.size3(fb)?; + let n = Self::volume(fb, s)?; +- let n = fb.convert(ConvOp::UConvert, n, u64_)?; +- let [wx, wy, wz] = [wide(fb, wg, 0)?, wide(fb, wg, 1)?, wide(fb, wg, 2)?]; +- let [nx, ny] = [wide(fb, groups, 0)?, wide(fb, groups, 1)?]; +- let group = fb.binary(BinOp::IMul, wz, ny)?; +- let group = fb.binary(BinOp::IAdd, group, wy)?; +- let group = fb.binary(BinOp::IMul, group, nx)?; +- let group = fb.binary(BinOp::IAdd, group, wx)?; +- let first = fb.binary(BinOp::IMul, group, n)?; +- let index = fb.binary(BinOp::IAdd, first, lin)?; + let r = region(fb)?; + let grid = fb.composite_extract(r, &[1])?; +- let [gx, gy, gz] = [wide(fb, grid, 0)?, wide(fb, grid, 1)?, wide(fb, grid, 2)?]; +- let x = fb.binary(BinOp::UMod, index, gx)?; +- let row = fb.binary(BinOp::UDiv, index, gx)?; +- let y = fb.binary(BinOp::UMod, row, gy)?; +- let z = fb.binary(BinOp::UDiv, row, gy)?; +- let inside = fb.binary(BinOp::ULessThan, z, gz)?; +- let u32_ = fb.module().ty(Type::Int { +- width: 32, +- signed: false, ++ let part = |fb: &mut FunctionBuilder<'_>, v: SValue| -> BResult<[SValue; 3]> { ++ Ok([ ++ fb.composite_extract(v, &[0])?, ++ fb.composite_extract(v, &[1])?, ++ fb.composite_extract(v, &[2])?, ++ ]) ++ }; ++ let [wx, wy, wz] = part(fb, wg)?; ++ let [nx, ny, nz] = part(fb, groups)?; ++ let [gx, gy, gz] = part(fb, grid)?; ++ // Whether every invocation's index fits in 32 bits: `a = nx·ny`, `b = a·nz`, `c = b·N` ++ // in 64 bits, each at most 2^32. While `a` and `b` are, the next product cannot wrap ++ // (`nz`, `N` < 2^32); a product past a failed test is not read. ++ let w = |fb: &mut FunctionBuilder<'_>, v: SValue| fb.convert(ConvOp::UConvert, v, u64_); ++ let limit = fb.module().constant(Constant::Scalar { ++ ty: u64_, ++ bits: ConstBits(1 << 32), + })?; +- let mut axes = [x, y, z]; +- for a in &mut axes { +- *a = fb.convert(ConvOp::UConvert, *a, u32_)?; +- } +- let pos = fb.composite_construct(self.uvec3, &axes)?; ++ let limit = fb.module().const_value(limit)?; ++ let (nx64, ny64, nz64, n64) = (w(fb, nx)?, w(fb, ny)?, w(fb, nz)?, w(fb, n)?); ++ let a = fb.binary(BinOp::IMul, nx64, ny64)?; ++ let b = fb.binary(BinOp::IMul, a, nz64)?; ++ let c = fb.binary(BinOp::IMul, b, n64)?; ++ let fa = fb.binary(BinOp::ULessThanEqual, a, limit)?; ++ let fbb = fb.binary(BinOp::ULessThanEqual, b, limit)?; ++ let fc = fb.binary(BinOp::ULessThanEqual, c, limit)?; ++ let fits = fb.binary(BinOp::LogicalAnd, fa, fbb)?; ++ let fits = fb.binary(BinOp::LogicalAnd, fits, fc)?; ++ let narrow: ArmFn<'_, ()> = Box::new(move |fb, _| { ++ let group = fb.binary(BinOp::IMul, wz, ny)?; ++ let group = fb.binary(BinOp::IAdd, group, wy)?; ++ let group = fb.binary(BinOp::IMul, group, nx)?; ++ let group = fb.binary(BinOp::IAdd, group, wx)?; ++ let first = fb.binary(BinOp::IMul, group, n)?; ++ let index = fb.binary(BinOp::IAdd, first, lin)?; ++ let x = fb.binary(BinOp::UMod, index, gx)?; ++ let row = fb.binary(BinOp::UDiv, index, gx)?; ++ let y = fb.binary(BinOp::UMod, row, gy)?; ++ let z = fb.binary(BinOp::UDiv, row, gy)?; ++ let inside = fb.binary(BinOp::ULessThan, z, gz)?; ++ Ok(ArmEnd::FallThrough(vec![x, y, z, inside])) ++ }); ++ let wide: ArmFn<'_, ()> = Box::new(move |fb, _| { ++ let w = |fb: &mut FunctionBuilder<'_>, v: SValue| fb.convert(ConvOp::UConvert, v, u64_); ++ let (wx, wy, wz) = (w(fb, wx)?, w(fb, wy)?, w(fb, wz)?); ++ let (nx, ny, n, lin) = (w(fb, nx)?, w(fb, ny)?, w(fb, n)?, w(fb, lin)?); ++ let (gx, gy, gz) = (w(fb, gx)?, w(fb, gy)?, w(fb, gz)?); ++ let group = fb.binary(BinOp::IMul, wz, ny)?; ++ let group = fb.binary(BinOp::IAdd, group, wy)?; ++ let group = fb.binary(BinOp::IMul, group, nx)?; ++ let group = fb.binary(BinOp::IAdd, group, wx)?; ++ let first = fb.binary(BinOp::IMul, group, n)?; ++ let index = fb.binary(BinOp::IAdd, first, lin)?; ++ let x = fb.binary(BinOp::UMod, index, gx)?; ++ let row = fb.binary(BinOp::UDiv, index, gx)?; ++ let y = fb.binary(BinOp::UMod, row, gy)?; ++ let z = fb.binary(BinOp::UDiv, row, gy)?; ++ let inside = fb.binary(BinOp::ULessThan, z, gz)?; ++ let x = fb.convert(ConvOp::UConvert, x, u32_)?; ++ let y = fb.convert(ConvOp::UConvert, y, u32_)?; ++ let z = fb.convert(ConvOp::UConvert, z, u32_)?; ++ Ok(ArmEnd::FallThrough(vec![x, y, z, inside])) ++ }); ++ let merged = fb.if_(&mut (), fits, narrow, Some(wide))?.0; ++ let [x, y, z, inside] = merged[..] else { ++ return Err(crate::spirv::error::BuildError::Dangling( ++ crate::spirv::error::Dangling::Value(SValue::new(u32::MAX)), ++ )); ++ }; ++ let pos = fb.composite_construct(self.uvec3, &[x, y, z])?; + Ok((pos, inside)) + } + diff --git a/patches/metal2vulkan/0026-a-flat-dispatch-past-2-32-threads-keeps-.patch b/patches/metal2vulkan/0026-a-flat-dispatch-past-2-32-threads-keeps-.patch new file mode 100644 index 0000000..7512adb --- /dev/null +++ b/patches/metal2vulkan/0026-a-flat-dispatch-past-2-32-threads-keeps-.patch @@ -0,0 +1,132 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 10:44:19 +0000 +Subject: [PATCH 26/34] a flat dispatch past 2^32 threads keeps + its 64-bit count + +New lane: a ragged direct dispatch of 65,536 x 65,601 threads (65 rows past 2^32) runs on the +flat variant, which then takes the 64-bit arm; every thread of the last column and the last row +runs once at its own position. A mutant that always takes the 32-bit arm fails it (rows 0-64 +counted twice), so the lane guards the fallback. 8 s on Arrow Lake. + +Co-Authored-By: Claude Opus 5.5 +(cherry picked from commit 5f6f2c9f80566db7116f5faf513931fe70f217ca) +--- + .../src/runtime_lane/indirect_dispatches.rs | 107 ++++++++++++++++++ + 1 file changed, 107 insertions(+) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs b/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs +index 49fcfac2..db6faaee 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/indirect_dispatches.rs +@@ -765,3 +765,110 @@ fn a_ragged_direct_dispatch_runs_each_thread_once_on_the_flat_variant() { + let _ = messages.take_notices(); + }); + } ++ ++/// A kernel that marks the grid's last column and last row: the thread at `x = gx - 1` adds 1 ++/// into `out[y]`, the thread at `y = gy - 1` into `out[gy + x]`. Over a grid of more than 2^32 ++/// threads only these edges are stored, and a position counted in 32 bits would wrap the rows past ++/// 2^32 threads onto the first ones (those counted twice, the last never). ++const FLAT_EDGES: &str = r#"define void @flat_edges(ptr addrspace(1) noundef "air-buffer-no-alias" %b0, <3 x i32> noundef %p, <3 x i32> noundef %g) { ++entry: ++ %px = extractelement <3 x i32> %p, i64 0 ++ %py = extractelement <3 x i32> %p, i64 1 ++ %gx = extractelement <3 x i32> %g, i64 0 ++ %gy = extractelement <3 x i32> %g, i64 1 ++ %lx = add i32 %gx, -1 ++ %ly = add i32 %gy, -1 ++ %ex = icmp eq i32 %px, %lx ++ br i1 %ex, label %col, label %next ++col: ++ %yi = zext i32 %py to i64 ++ %o0 = getelementptr inbounds i32, ptr addrspace(1) %b0, i64 %yi ++ %a0 = atomicrmw add ptr addrspace(1) %o0, i32 1 monotonic, align 4 ++ br label %next ++next: ++ %ey = icmp eq i32 %py, %ly ++ br i1 %ey, label %row, label %done ++row: ++ %xi = add i32 %gy, %px ++ %xw = zext i32 %xi to i64 ++ %o1 = getelementptr inbounds i32, ptr addrspace(1) %b0, i64 %xw ++ %a1 = atomicrmw add ptr addrspace(1) %o1, i32 1 monotonic, align 4 ++ br label %done ++done: ++ ret void ++} ++ ++!air.kernel = !{!0} ++!air.version = !{!20} ++!air.language_version = !{!21} ++!0 = !{ptr @flat_edges, !1, !2} ++!1 = !{} ++!2 = !{!3, !5, !6} ++!3 = !{i32 0, !"air.buffer", !"air.location_index", i32 0, i32 1, !"air.read_write", !"air.address_space", i32 1, !"air.arg_type_size", i32 4, !"air.arg_type_align_size", i32 4, !"air.arg_type_name", !"uint", !"air.arg_name", !"b0"} ++!5 = !{i32 1, !"air.thread_position_in_grid", !"air.arg_type_name", !"uint3", !"air.arg_name", !"p"} ++!6 = !{i32 2, !"air.threads_per_grid", !"air.arg_type_name", !"uint3", !"air.arg_name", !"g"} ++!20 = !{i32 2, i32 8, i32 0} ++!21 = !{!"Metal", i32 4, i32 0, i32 0} ++"#; ++ ++/// A direct dispatch of more than 2^32 threads (65,536 x 65,601: ragged, so not the exact split, and 65 rows past 2^32) runs on the ++/// flat variant's 64-bit arm, and every thread of the last column and the last row runs once at ++/// its own position, the rows past 2^32 included: the 32-bit count is taken only when the whole ++/// dispatch fits it. ++#[test] ++fn a_direct_dispatch_past_2_32_threads_keeps_the_flat_variant_s_64_bit_count() { ++ on_device(|device, messages| { ++ let library = device ++ .new_library_with_air(&llvm_assemble_text(FLAT_EDGES).unwrap()) ++ .unwrap(); ++ let function = library.new_function_with_name("flat_edges").unwrap(); ++ let kernel = device ++ .new_compute_pipeline_state( ++ &ComputePipelineDescriptor::new(StageSource::Function( ++ function.function_descriptor(), ++ )), ++ &PipelineOptions::default(), ++ ) ++ .unwrap(); ++ let (grid, group) = ([65_536u32, 65_601, 1], [32u32, 32, 1]); ++ assert!(u64::from(grid[0]) * u64::from(grid[1]) > 1 << 32); ++ let [gx, gy, _] = grid; ++ let slots = u64::from(gy + gx); ++ let out = buffer_with(device, &vec![0; slots as usize * 4]).unwrap(); ++ let queue = device.new_command_queue().unwrap(); ++ let command_buffer = queue.command_buffer().unwrap(); ++ let mut encoder = command_buffer.compute_command_encoder().unwrap(); ++ encoder.set_compute_pipeline_state(&kernel).unwrap(); ++ encoder.set_buffer(Some(&out), 0, 0).unwrap(); ++ encoder.dispatch_threads(size(grid), size(group)).unwrap(); ++ encoder.end_encoding().unwrap(); ++ complete(&command_buffer, &format!("past 2^32 {grid:?} {group:?}")); ++ let flat = command_buffer ++ .executions() ++ .iter() ++ .any(|execution| match execution.pipeline() { ++ ExecutedPipeline::Compute(pipeline) => pipeline ++ .dependencies ++ .iter() ++ .any(|dependency| dependency.fact == RuntimeFact::FlatGrid), ++ ExecutedPipeline::Render(_) => false, ++ }); ++ assert!(flat, "{grid:?} {group:?}: not on the flat variant"); ++ let words = read_buffer(&out, 0, slots * 4).unwrap(); ++ let counts: Vec = words ++ .as_chunks::<4>() ++ .0 ++ .iter() ++ .map(|w| u32::from_le_bytes(*w)) ++ .collect(); ++ let wrong: Vec<(usize, u32)> = counts ++ .iter() ++ .enumerate() ++ .filter(|(_, &c)| c != 1) ++ .map(|(i, &c)| (i, c)) ++ .take(8) ++ .collect(); ++ assert!(wrong.is_empty(), "slots not counted once (first 8, rows then columns at {gy}): {wrong:?}"); ++ let _ = messages.take_notices(); ++ }); ++} diff --git a/patches/metal2vulkan/0027-coherent-late-copies-in-keep-writes-the-pages-lack-o.patch b/patches/metal2vulkan/0027-coherent-late-copies-in-keep-writes-the-pages-lack-o.patch new file mode 100644 index 0000000..cb4743a --- /dev/null +++ b/patches/metal2vulkan/0027-coherent-late-copies-in-keep-writes-the-pages-lack-o.patch @@ -0,0 +1,619 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 15:27:30 +0000 +Subject: [PATCH 27/34] coherent: late copies in keep writes the pages lack, + one per wait, own standings after a wait + +Three limits ec9bab82 left in the held batch's copy in (shadow::LateCopies), each a lane +that failed at ec9bab82: + +- A write in flight (taking_late waits for nothing under the scheduler's lock). Another + command buffer's write over the surface ran on the GPU, not completed; the guest wrote + beside it and set the event. The copy in of the guest's whole rectangle from the pages put + their older bytes over the write in the image, which then went to the pages: lost. + Now, where a record still open or running meets the rectangle, taking_late holds the pages + and copies in the texels the CPU changed alone (each open page against its snapshot), as + rectangles (at most LATE_RECTS = 64, then bands, then the rectangle, counted busy). A texel + both wrote is a race Metal leaves undefined. No wait: the GPU orders the copy after the + writer; only the pages lacked its bytes. Counted busy still: a GPU copy back in flight, a + page with no snapshot, a shadow whose copy in is not cut. + Lane surface_coherent_a_late_copy_in_keeps_a_write_in_flight. +- An encoder after a wait for a shared event over a surface an earlier encoder of the same + command buffer left writes ahead in: its standing was refused (OwnBuffer), so it copied in + every texel from the pages at its use, losing those writes, with or without a CPU write + (and a CPU write before the signal then lost the GPU's write over it to the merge). Now, + after such a wait, a trapping surface's own standing is taken as a carried one is + (Lazy::take's waited), with no copy in at encoding; the batch's late copy in takes what the + CPU wrote, by changed texels where the command buffer's own record meets them (noted + with left_open_by). Lane surface_coherent_a_late_copy_in_keeps_the_command_buffers_own_writes. +- One note per surface: each wait for a shared event notes the surface again, with the part + it waits at (State::recorded's length), and a batch takes the notes from waits before its + end. The second of two waits left its batch with no copy in once the first limit was fixed. + Lane surface_coherent_each_wait_copies_in_what_came_before_its_signal. + +Cost where nothing changed: none. With a CPU write before the signal and a write the pages +lack over it, a compare of the snapshot pages under the scheduler's lock (memcmp per page, +per texel only on changed pages). + +Left: a standing taken after a wait (another command buffer's, or now its own) knows nothing of +a GPU write of the surface's bytes through another import encoded after it and ordered before +the signal; as before. + +Lanes (ANV): the three new ones fail at ec9bab82 and pass here. Filters "surface" and "event": +158 passed, 1 failed, 5 ignored (155 + the three; the failure is +ycbcr_textures::textures_over_any_surface_read_what_metal_reads, as before). Runtime unit tests +(shadow, batches, command_buffer, exit_guard, coherent, lazy): 135 passed, 1 failed: +exit_guard, the same two submit.rs Stamps lines as before. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/mod.rs | 5 +- + .../src/command_buffer/submit.rs | 4 +- + crates/metal2vulkan-runtime/src/shadow.rs | 57 ++++-- + .../src/shadow_coherent.rs | 165 ++++++++++++++++-- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 67 +++++-- + .../src/runtime_lane/surface_coherent.rs | 114 ++++++++++++ + 6 files changed, 359 insertions(+), 53 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index e636538b..6d3b46ca 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -1302,10 +1302,11 @@ impl State { + /// LOCAL (Atlas): the work encoded next waits for a shared event, which the host may set + /// after a CPU write a trapping surface's copy in must take (`shadow::LateCopies`). + pub(super) fn shadows_wait_for_shared_event(&mut self) { +- (self.shadow_carry.get_or_insert_with(ShadowCarry::default)).waits_for_shared_event(); ++ let part = self.recorded.len(); ++ (self.shadow_carry.get_or_insert_with(ShadowCarry::default)).waits_for_shared_event(part); + } + +- pub(super) fn shadows_not_carried(&mut self) { ++ pub(super) fn shadows_not_carried(&mut self) { + if let Some(carry) = &mut self.shadow_carry { + carry.clear(); + } +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index f415cc30..7c41b410 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -1197,7 +1197,7 @@ impl CommandBufferShared { + return Ok(()); + }; + let recorded = &state.recorded[range.clone()]; +- let shadows = carry.take_late(|copies| { ++ let shadows = carry.take_late(range.end, |copies| { + (recorded.iter()).any(|part| { + (part.summary.lazy_uses()) + .any(|used| used.key == copies.shadow_key || copies.meets(used)) +@@ -1208,7 +1208,7 @@ impl CommandBufferShared { + } + let id = carry.id(); + let copies: Vec<_> = (shadows.iter()) +- .filter_map(|copies| crate::shadow_lazy::late_copy_in(copies, id)) ++ .flat_map(|(left, copies)| crate::shadow_lazy::late_copy_in(copies, id, *left)) + .collect(); + if copies.is_empty() { + return Ok(()); +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index a7ab6828..b508e291 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2331,11 +2331,18 @@ pub(super) struct ShadowCarry { + /// at its use: that copy reads the pages as the GPU runs it. + /// + /// [`Coherent::taking`]: super::shadow_coherent ++/// ++/// Each wait starts its own notes: a surface taken after two waits copies in at each, with the ++/// first batch from its wait's place on that uses it, so a CPU write before the later signal is ++/// not left to the earlier batch's copy. + #[derive(Default)] + pub(super) struct LateCopies { +- /// The command buffer waits for a shared event before the work encoded from here on. +- waited: bool, +- shadows: Vec>, ++ /// The command buffer's part (`command_buffer::State::recorded`) the last wait for a shared ++ /// event holds back: the work encoded from here on waits for it. ++ since: Option, ++ /// Each surface noted, with the part of the wait it was noted after, and whether an earlier ++ /// encoder of the command buffer had left writes ahead in its image then. ++ shadows: Vec<(usize, bool, Arc)>, + } + + impl Default for ShadowCarry { +@@ -2396,22 +2403,28 @@ impl ShadowCarry { + self.entries.clear(); + } + +- /// LOCAL (Atlas): the work encoded from here on waits for a shared event ([`LateCopies`]). +- pub(super) fn waits_for_shared_event(&mut self) { +- self.late.waited = true; ++ /// LOCAL (Atlas): the work encoded from part `part` on waits for a shared event ++ /// ([`LateCopies`]). ++ pub(super) fn waits_for_shared_event(&mut self, part: usize) { ++ self.late.since = Some(part); + } + +- /// LOCAL (Atlas): the shadows of the late copies in ([`LateCopies`]) that `met` says a batch +- /// going now uses, taken out: each goes with the first such batch. +- pub(super) fn take_late(&mut self, met: impl Fn(&ShadowCopies) -> bool) -> Vec> { ++ /// LOCAL (Atlas): the shadows of the late copies in ([`LateCopies`]) that a batch of parts ++ /// ending at `end` takes: noted after a wait before that end, and used by the batch (`met`). ++ /// Each goes with the first such batch. ++ pub(super) fn take_late( ++ &mut self, ++ end: usize, ++ met: impl Fn(&ShadowCopies) -> bool, ++ ) -> Vec<(bool, Arc)> { + if self.late.shadows.is_empty() { + return Vec::new(); + } +- let (taken, kept) = std::mem::take(&mut self.late.shadows) ++ let (taken, kept): (Vec<_>, Vec<_>) = std::mem::take(&mut self.late.shadows) + .into_iter() +- .partition(|copies| met(copies)); ++ .partition(|(from, _, copies)| *from < end && met(copies)); + self.late.shadows = kept; +- taken ++ taken.into_iter().map(|(_, left, copies)| (left, copies)).collect() + } + } + +@@ -2449,20 +2462,26 @@ impl ShadowSync { + // LOCAL (Atlas): an earlier command buffer's image, where nothing wrote its bytes + // since (`shadow_lazy.rs`). + let lazily = match (&carried, &copies.lazy) { +- (None, Some(lazy)) => lazy.take(self.carry.id, met), ++ (None, Some(lazy)) => { ++ lazy.take(self.carry.id, met, self.carry.late.since.is_some()) ++ } + _ => None, + }; + // LOCAL (Atlas): a trapping surface's standing taken after a wait for a shared + // event: the CPU may write before the signal, which the batch copies in as it +- // goes ([`LateCopies`]). Not over writes an earlier encoder of this command +- // buffer left ahead, which the copy in would replace with the pages' bytes. +- if let (Some(_), Some(lazy), true) = (&lazily, &copies.lazy, self.carry.late.waited) { ++ // goes ([`LateCopies`]), once per wait. A standing this command buffer's own ++ // earlier encoder left is taken after the wait too (`Lazy::take`): its copy in is ++ // the batch's alone, which takes the texels the CPU changed where any write the ++ // pages lack meets them (`shadow_coherent::Coherent::taking_late`). ++ if let (Some(_), Some(lazy), Some(since)) = (&lazily, &copies.lazy, self.carry.late.since) { + let late = &mut self.carry.late.shadows; + if lazy.whole_image() +- && !late.iter().any(|known| known.shadow_key == copies.shadow_key) +- && !lazy.left_open_by(self.carry.id) ++ && !(late.iter()).any(|(from, _, known)| { ++ *from == since && known.shadow_key == copies.shadow_key ++ }) + { +- late.push(Arc::clone(copies)); ++ let left = lazy.left_open_by(self.carry.id); ++ late.push((since, left, Arc::clone(copies))); + } + } + let only_read = lazily.is_some() && (copies.lazy.as_ref()) +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 44e4d1f0..c8f5d54b 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -105,6 +105,9 @@ const HOLD_MAX_S: u64 = 30; + const RECORDS: usize = 8; + const PROBE_GIVEN_UP: Duration = Duration::from_millis(1); + const DEFAULT_WAIT_MS: u64 = 50; ++/// The rectangles a late copy in takes of the texels the guest changed, at most ++/// ([`Coherent::taking_late`]). ++const LATE_RECTS: usize = 64; + /// The closed surfaces' lines kept between reports. + const ENDED_KEPT: usize = 256; + /// A GPU copy of a surface, per MiB: 2.97 ms for 5.76 Mtexels of 4 bytes (`STUDY-virt.md`). +@@ -1160,26 +1163,162 @@ impl Coherent { + + /// A held batch of the command buffer `id` is about to go, whose encoders took the image's + /// standing as they were encoded ([`taking`](Self::taking)): where the guest's CPU wrote +- /// since, before the signal the batch waited for, the rectangle its copy in takes, recorded ++ /// since, before the signal the batch waited for, the rectangles its copy in takes, recorded + /// at the batch's start (`command_buffer/submit.rs`). Called under the scheduler's lock, it +- /// waits for no record: a write of another command buffer still running over the rectangle +- /// is counted (`overwrite_risk`, [`Fallback::LateCopyInBusy`]), as a bounded wait past its +- /// bound is. The command buffer's own records run after the copy. +- pub(crate) fn taking_late(&self, id: u64) -> Option<[u32; 4]> { ++ /// waits for no record. Where a write the pages lack meets the guest's rectangle, a record of ++ /// any command buffer still open or running (another one's in flight, or an earlier encoder ++ /// of this one that left its writes ahead), a copy of the rectangle would put the pages' ++ /// older bytes over it in the image. So then the rectangles are the texels the CPU changed ++ /// alone: each open page against its snapshot, after the pages are held, so no write lands ++ /// past the compare. A texel both wrote is a race Metal leaves undefined too. Counted as busy ++ /// (`overwrite_risk`, [`Fallback::LateCopyInBusy`]) is what that cannot cover: a GPU copy ++ /// back in flight, a page the guest reached with no snapshot, a shadow whose copy in is not ++ /// cut to a rectangle (`cut` false), or changes too scattered for [`LATE_RECTS`]. ++ /// ++ /// The command buffer's own record (`id`) counts only where an earlier encoder of it left ++ /// writes ahead before the wait (`left`): the rest of its writes run after the copy. ++ pub(crate) fn taking_late(&self, id: u64, left: bool, cut: bool) -> Vec<[u32; 4]> { + let mut state = self.lock(); + if !state.tracked || state.closed { +- return None; ++ return Vec::new(); ++ } ++ let Some(area) = state.guest else { ++ return Vec::new(); ++ }; ++ let ahead = |state: &State, rects: &[[u32; 4]]| { ++ (state.open.iter().chain(&state.running)) ++ .filter(|record| left || record.id != id) ++ .any(|record| rects.iter().any(|rect| meet(record.area, *rect))) ++ }; ++ let behind = ahead(&state, &[area]); ++ let mut busy = state.flushing != 0 && !state.snapshots.is_empty(); ++ let mut rects = vec![area]; ++ if behind && cut && !state.snapshots.is_empty() { ++ self.sync_done(&mut state, SyncBy::CopyIn); ++ let list: Vec = state.snapshots.keys().copied().collect(); ++ let came = pages().map(|mark| mark.hold(&list, true)).unwrap_or_default(); ++ let (changed, whole) = self.changed(&state, area); ++ rects = changed; ++ busy |= whole && ahead(&state, &rects); ++ state.guest = None; ++ // A page the guest reached as the marks moved came with no snapshot: its rectangle ++ // goes to the next copy in, as `take_guest`'s does. ++ self.note_came(&mut state, came); ++ state.snapshots.clear(); ++ state.opened = false; ++ } else { ++ busy |= behind; ++ self.take_guest(&mut state); + } +- let area = state.guest?; +- let busy = state.flushing != 0 +- || (state.running.iter()).any(|record| record.id != id && meet(record.area, area)); +- if busy && !state.snapshots.is_empty() { ++ if busy { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + count(Fallback::LateCopyInBusy); + } +- self.take_guest(&mut state); +- count(Fallback::LateCopyIn); +- Some(area) ++ if !rects.is_empty() { ++ count(Fallback::LateCopyIn); ++ } ++ rects ++ } ++ ++ /// The rectangles of `area`'s texels the guest changed against its open pages' snapshots, ++ /// and whether they are coarser than that: a page in `area` with no snapshot takes its whole ++ /// rectangle, and past [`LATE_RECTS`] each band of rows changed takes its bounds, then ++ /// `area` alone. ++ fn changed(&self, state: &State, area: [u32; 4]) -> (Vec<[u32; 4]>, bool) { ++ let layout = &self.layout; ++ let texel = layout.texel; ++ let mut coarse = false; ++ // Whether the bytes `[from, to)` are as their pages' snapshots hold them: `None` where a ++ // page has none. ++ let same = |from: u64, to: u64| -> Option { ++ let mut at = from; ++ while at < to { ++ let page = at & !(PAGE - 1); ++ let end = to.min(page + PAGE); ++ let was = state.snapshots.get(&page)?; ++ let was = &was[(at - page) as usize..(end - page) as usize]; ++ // SAFETY: `[at, end)` lies in the surface's mapping, in pages held: no write of ++ // the guest moves them meanwhile. ++ let now = unsafe { std::slice::from_raw_parts(at as *const u8, was.len()) }; ++ if was != now { ++ return Some(false); ++ } ++ at = end; ++ } ++ Some(true) ++ }; ++ // Per row, the runs of texels changed: `(y, x0, x1)`, ascending. ++ let mut runs: Vec<(u32, u32, u32)> = Vec::new(); ++ for y in area[1]..area[3] { ++ let row = layout.at(0, y); ++ let mut run: Option = None; ++ let mut x = area[0]; ++ while x < area[2] { ++ // The texels that start in this texel's page. ++ let page = layout.at(x, y) & !(PAGE - 1); ++ let past = ((page + PAGE - row).div_ceil(texel) as u32).clamp(x + 1, area[2]); ++ let span = same(layout.at(x, y), layout.at(past - 1, y) + texel); ++ for tx in x..past { ++ let changed = match span { ++ Some(true) => false, ++ Some(false) => { ++ let at = layout.at(tx, y); ++ same(at, at + texel) != Some(true) ++ } ++ None => { ++ coarse = true; ++ true ++ } ++ }; ++ match (changed, run) { ++ (true, None) => run = Some(tx), ++ (false, Some(from)) => { ++ runs.push((y, from, tx)); ++ run = None; ++ } ++ _ => {} ++ } ++ } ++ x = past; ++ } ++ if let Some(from) = run { ++ runs.push((y, from, area[2])); ++ } ++ } ++ // Runs of one span on consecutive rows join. ++ let mut rects: Vec<[u32; 4]> = Vec::new(); ++ let mut open: Vec = Vec::new(); ++ let mut row = None; ++ for (y, x0, x1) in runs.iter().copied() { ++ if row != Some(y) { ++ open.retain(|at| rects[*at][3] == y); ++ row = Some(y); ++ } ++ match (open.iter()).find(|at| rects[**at][0] == x0 && rects[**at][2] == x1 && rects[**at][3] == y) { ++ Some(at) => rects[*at][3] = y + 1, ++ None => { ++ open.push(rects.len()); ++ rects.push([x0, y, x1, y + 1]); ++ } ++ } ++ } ++ if rects.len() > LATE_RECTS { ++ coarse = true; ++ let mut bands: Vec<[u32; 4]> = Vec::new(); ++ for (y, x0, x1) in runs { ++ match bands.last_mut() { ++ Some(band) if band[3] >= y => { ++ *band = [band[0].min(x0), band[1], band[2].max(x1), band[3].max(y + 1)]; ++ } ++ _ => bands.push([x0, y, x1, y + 1]), ++ } ++ } ++ rects = bands; ++ if rects.len() > LATE_RECTS { ++ rects = vec![area]; ++ } ++ } ++ (rects, coarse) + } + + /// A GPU copy back of what ran is about to record: everything done went on the CPU, and what +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 76e0986b..15164b8b 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -513,21 +513,32 @@ pub(crate) fn signalled_by(id: u64) -> Option, ShadowCopy, + + /// The copy in a held batch of the command buffer `id` takes as it goes, of a trapping surface + /// an encoder of it took the standing of after a wait for a shared event (`shadow::LateCopies`): +-/// what the guest's CPU wrote since that encoding, before the signal. `None` where it wrote +-/// nothing, or the surface no longer traps. +-pub(crate) fn late_copy_in(copies: &Arc, id: u64) -> Option { +- let lazy = copies.lazy.as_ref()?; +- let area = lazy.coherent()?.taking_late(id)?; +- let copy = ShadowCopy { +- copies: Arc::clone(copies), +- step: Step::Into, +- kept: false, +- area: Some(area), +- cause: Some(CopyCause::Bounded), +- moves: None, ++/// what the guest's CPU wrote since that encoding, before the signal, one copy per rectangle ++/// ([`Coherent::taking_late`](super::shadow_coherent)). None where it wrote nothing, or the ++/// surface no longer traps. ++/// `left`: an earlier encoder of it left writes ahead in the image before the wait. ++pub(crate) fn late_copy_in(copies: &Arc, id: u64, left: bool) -> Vec { ++ let Some(lazy) = copies.lazy.as_ref() else { ++ return Vec::new(); + }; +- lazy.copied_in(CopyCause::Bounded, copy.texels_copied_in()); +- Some(copy) ++ let Some(coherent) = lazy.coherent() else { ++ return Vec::new(); ++ }; ++ let list: Vec = (coherent.taking_late(id, left, copies.cuts()).into_iter()) ++ .map(|area| ShadowCopy { ++ copies: Arc::clone(copies), ++ step: Step::Into, ++ kept: false, ++ area: Some(area), ++ cause: Some(CopyCause::Bounded), ++ moves: None, ++ }) ++ .collect(); ++ let texels = list.iter().map(ShadowCopy::texels_copied_in).sum(); ++ if !list.is_empty() { ++ lazy.copied_in(CopyCause::Bounded, texels); ++ } ++ list + } + + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what +@@ -1175,14 +1186,28 @@ impl Lazy { + + /// Where the image stands for an encoder that met `touched`, of a command buffer of identity + /// `id`: `Some((held, stale))` where it needs no copy in (the module docs). +- pub(super) fn take(&self, id: u64, touched: u64) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { ++ /// ++ /// LOCAL (Atlas): with `waited` (the command buffer waits for a shared event before this ++ /// encoder, `shadow::LateCopies`), a trapping surface's standing its own earlier encoder left ++ /// is taken too, as a carried one would be, and its copy in waits for the batch to go: a copy ++ /// in of the pages now would replace what those encoders left ahead, which the pages lack, and ++ /// the batch's copy in takes what the CPU wrote, keeping the rest ++ /// (`shadow_coherent::Coherent::taking_late`). ++ pub(super) fn take( ++ &self, ++ id: u64, ++ touched: u64, ++ waited: bool, ++ ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { + // Tracked from here: nothing its copy in reads can change unseen. + let coherent = self.coherent().filter(|coherent| coherent.tracks()); + if coherent.is_none() { + self.seal_beside(); + } +- let taken = self.take_standing(id, touched, coherent.is_some()); ++ let own = waited && coherent.is_some() && self.stood_by(id); ++ let taken = self.take_standing(id, touched, coherent.is_some(), own); + match coherent { ++ Some(_) if own => taken, + Some(coherent) => { + let stale = coherent.taking(taken.is_none()); + taken.map(|(held, was)| (held, bounding_stale(was, stale))) +@@ -1206,11 +1231,17 @@ impl Lazy { + } + } + ++ /// Whether its standing is one an encoder of the command buffer `id` left. ++ fn stood_by(&self, id: u64) -> bool { ++ (*self.stood.lock().unwrap()).is_some_and(|stood| stood.id == id) ++ } ++ + fn take_standing( + &self, + id: u64, + touched: u64, + tracked: bool, ++ own: bool, + ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { + report(); + let Some(stood) = *self.stood.lock().unwrap() else { +@@ -1223,8 +1254,10 @@ impl Lazy { + Some(Refused::Written) + } else if stood.epoch != EPOCH.load(Ordering::Relaxed) { + Some(Refused::Epoch) +- } else if stood.id == id { ++ } else if stood.id == id && !own { + Some(Refused::OwnBuffer) ++ } else if own { ++ None + } else if !committed_ids().lock().unwrap().contains(&stood.id) { + Some(Refused::Uncommitted) + } else { +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 031fc9ba..d8070e0c 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -1722,3 +1722,117 @@ fn surface_coherent_a_cpu_write_before_a_signal_reaches_a_waiting_command() { + ); + }); + } ++ ++/// Wait, a bounded time, until a held batch went with a copy in (`late_copies_in` past `from`). ++fn wait_for_late_copy(from: u64) { ++ let start = Instant::now(); ++ while shadow_coherent::fallback_counts().late_copies_in <= from && start.elapsed() < SOON { ++ std::thread::sleep(Duration::from_millis(2)); ++ } ++} ++ ++/// A blend of nothing over the case's texture (within `damage`), after a wait for `hold` at 1. ++fn blend_after( ++ case: &Case<'_>, ++ command_buffer: &CommandBuffer, ++ hold: &SharedEvent, ++ damage: Option, ++) { ++ let color = staged(case.device, &floats4([0.0; 4])); ++ let rects = std::slice::from_ref(&case.rect); ++ let blending = rects_blending(case.device); ++ command_buffer.encode_wait_for_event(hold.into(), 1).unwrap(); ++ pass(command_buffer, case.texture(), &blending, rects, &color, damage); ++} ++ ++/// A held batch's copy in (`shadow::LateCopies`) while another command buffer's write over the ++/// surface ran on the GPU and its command buffer had not completed (it waits for an event after ++/// its pass): the guest wrote beside that write, then set the event the reader waited for. The ++/// copy in takes the guest's texels alone, so the write in flight stays in the image. ++#[test] ++fn surface_coherent_a_late_copy_in_keeps_a_write_in_flight() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let other = case.device.new_command_queue().unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ case.pass(&writer, GREEN, scissor(0, 7, 64, 1)); ++ writer.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ writer.commit().unwrap(); ++ // Its pass runs on the GPU meanwhile; nothing the guest can see says when. ++ std::thread::sleep(Duration::from_millis(100)); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ let late = shadow_coherent::fallback_counts().late_copies_in; ++ case.write(9, 100, [1, 2, 3, 4]); ++ hold.set_signaled_value(1); ++ wait_for_late_copy(late); ++ gate.set_signaled_value(1); ++ for command_buffer in [&writer, &waiting] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ assert_eq!(shadow_coherent::fallback_counts().late_copies_in, late + 1); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the guest's write"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the write in flight"); ++ assert_eq!(case.read(0, 100), [0, 0, 255, 0], "the older texels"); ++ }); ++} ++ ++/// Two waits for shared events in one command buffer, each before a draw over the surface: the ++/// guest writes before each signal, the second time after the first batch went and beside its ++/// draw, and each draw sees the write before its own signal, the first draw's texels kept. Each ++/// wait notes the surface again (`shadow::LateCopies`): one note per surface left the second ++/// batch with no copy in. ++#[test] ++fn surface_coherent_each_wait_copies_in_what_came_before_its_signal() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let one = case.device.new_shared_event().unwrap(); ++ let two = case.device.new_shared_event().unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &command_buffer, &one, scissor(0, 0, WIDTH, 1)); ++ blend_after(&case, &command_buffer, &two, scissor(0, 3, WIDTH, 1)); ++ command_buffer.commit().unwrap(); ++ let late = shadow_coherent::fallback_counts().late_copies_in; ++ case.write(5, 0, [1, 2, 3, 4]); ++ one.set_signaled_value(1); ++ wait_for_late_copy(late); ++ case.write(5, 3, [5, 6, 7, 8]); ++ two.set_signaled_value(1); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(5, 0), [1, 2, 3, 0], "the write before the first signal"); ++ assert_eq!(case.read(5, 3), [5, 6, 7, 0], "the write before the second"); ++ assert_eq!(case.read(6, 3), [0, 0, 255, 0], "the older texels"); ++ }); ++} ++ ++/// A command buffer draws over a part of the surface, waits for a shared event, then draws over ++/// all of it: the guest writes beside the first draw and sets the event. The second draw sees ++/// both, so the copy in takes the guest's texels and keeps the first draw's in the image. Before, ++/// the wait let the second draw take no standing (its own command buffer's), and its copy in of ++/// the whole surface from the pages lost the first draw's texels, CPU write or not. ++#[test] ++fn surface_coherent_a_late_copy_in_keeps_the_command_buffers_own_writes() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let command_buffer = case.queue.command_buffer().unwrap(); ++ case.pass(&command_buffer, GREEN, scissor(0, 7, 64, 1)); ++ blend_after(&case, &command_buffer, &hold, None); ++ command_buffer.commit().unwrap(); ++ let late = shadow_coherent::fallback_counts().late_copies_in; ++ case.write(9, 100, [1, 2, 3, 4]); ++ hold.set_signaled_value(1); ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the guest's write"); ++ assert_eq!(case.read(8, 7), [0, 255, 0, 0], "the first draw"); ++ assert_eq!(case.read(0, 0), [0, 0, 255, 0], "the older texels"); ++ assert_eq!(shadow_coherent::fallback_counts().late_copies_in, late + 1); ++ }); ++} ++ diff --git a/patches/metal2vulkan/0028-coherent-lanes-a-standing-after-a-wait-and-a-later-G.patch b/patches/metal2vulkan/0028-coherent-lanes-a-standing-after-a-wait-and-a-later-G.patch new file mode 100644 index 0000000..edb2583 --- /dev/null +++ b/patches/metal2vulkan/0028-coherent-lanes-a-standing-after-a-wait-and-a-later-G.patch @@ -0,0 +1,142 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 18:36:18 +0000 +Subject: [PATCH 28/34] coherent lanes: a standing after a wait and a later GPU + write through another import (path 4) + +Six lanes: a command buffer waits for a shared event, then blends nothing over the whole +surface (its encoder takes the image's standing as it is encoded; "own": after a pass of its +own over a row). Encoded after it, on another queue, a command buffer writes texels of the +surface through another import (a second MTLTexture over the IOSurface, or an MTLBuffer of its +own over the pages, by a blit); the event is set after that write, by the writer on the GPU or +by the CPU once it completed. Metal orders the write before the blend through the event alone. + +At bc15b13e four of six fail (lanes-p4a, 2026-11-03 18:27Z): the buffer write (both signals, +own too) and the texture write signalled by the GPU leave the blend out of the pages, its +alpha 255 at (0,0); the write through the other import is in the pages, as the blend's record +never reached them. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/runtime_lane/surface_coherent.rs | 111 ++++++++++++++++++ + 1 file changed, 111 insertions(+) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index d8070e0c..15551a37 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -1836,3 +1836,114 @@ fn surface_coherent_a_late_copy_in_keeps_the_command_buffers_own_writes() { + }); + } + ++ ++/// What writes the surface's bytes through another import in [`gpu_write_through_another_import`]. ++#[derive(Clone, Copy, Debug)] ++enum OtherImport { ++ /// A second `MTLTexture` over the IOSurface, which draws. ++ Texture, ++ /// An `MTLBuffer` of its own over the pages, which a blit writes. ++ Buffer, ++} ++ ++/// The fourth path the late copies in left (`shadow::LateCopies`): a command buffer waits for a ++/// shared event and then blends nothing over the whole surface, so its encoder took the image's ++/// standing as it was encoded (`own`: after a pass of its own over a row, whose standing it then ++/// takes). Encoded after it, on another queue, a command buffer writes texels of the surface ++/// through another import; the event is set after that write (`by_gpu`: signalled by the writer ++/// on the GPU, else by the CPU once the writer completed). Metal orders the write before the ++/// blend through the event alone, so the blend keeps it, and the pass before the wait stays too. ++fn gpu_write_through_another_import(import: OtherImport, own: bool, by_gpu: bool) { ++ on_surface(true, |case| { ++ let second = matches!(import, OtherImport::Texture).then(|| case.second_texture()); ++ let other_buffer = matches!(import, OtherImport::Buffer).then(|| case.other_import()); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ if own { ++ case.pass(&waiting, GREEN, scissor(0, 7, 64, 1)); ++ } ++ blend_after(&case, &waiting, &hold, None); ++ waiting.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ let wrote = match import { ++ OtherImport::Texture => { ++ let blue = staged(case.device, &floats4([0.0, 0.0, 1.0, 1.0])); ++ let rects = std::slice::from_ref(&case.rect); ++ let second = second.as_ref().unwrap(); ++ pass(&writer, second, &case.opaque, rects, &blue, scissor(0, 100, 16, 1)); ++ [255, 0, 0, 0] ++ } ++ OtherImport::Buffer => { ++ let bytes: Vec = (0..16).flat_map(|_| [1u8, 2, 3, 4]).collect(); ++ let source = staged(case.device, &bytes); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ let to = other_buffer.as_ref().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, to, 100 * ROW, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ [1, 2, 3, 0] ++ } ++ }; ++ if by_gpu { ++ writer.encode_signal_event((&hold).into(), 1).unwrap(); ++ } ++ writer.commit().unwrap(); ++ let mut writer_first = true; ++ if !by_gpu { ++ writer_first = writer.wait_until_completed_for(SOON).is_ok(); ++ hold.set_signaled_value(1); ++ } ++ for command_buffer in [&writer, &waiting] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ let counts = shadow_coherent::fallback_counts(); ++ eprintln!("PATH4 report {}", shadow_coherent::report("p4")); ++ eprintln!( ++ "PATH4 {import:?} own={own} by_gpu={by_gpu} writer_first={writer_first} \ ++ late={} busy={} (9,100)={:?} (8,7)={:?} (0,0)={:?} behind(9,100)={:?}", ++ counts.late_copies_in, ++ counts.late_copies_in_busy, ++ case.read(9, 100), ++ case.read(8, 7), ++ case.read(0, 0), ++ case.behind(9, 100), ++ ); ++ assert!(writer_first, "Metal orders nothing between the two but the event"); ++ assert_eq!(case.read(9, 100), wrote, "the write through the other import"); ++ let first = if own { [0, 255, 0, 0] } else { [0, 0, 255, 0] }; ++ assert_eq!(case.read(8, 7), first, "the pass before the wait"); ++ assert_eq!(case.read(0, 0), [0, 0, 255, 0], "the older texels"); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_keeps_a_later_texture_write_before_the_signal() { ++ gpu_write_through_another_import(OtherImport::Texture, false, false); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_keeps_a_later_texture_write_signalled_by_the_gpu() { ++ gpu_write_through_another_import(OtherImport::Texture, false, true); ++} ++ ++#[test] ++fn surface_coherent_an_own_standing_after_a_wait_keeps_a_later_texture_write() { ++ gpu_write_through_another_import(OtherImport::Texture, true, false); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_keeps_a_later_buffer_write_before_the_signal() { ++ gpu_write_through_another_import(OtherImport::Buffer, false, false); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_keeps_a_later_buffer_write_signalled_by_the_gpu() { ++ gpu_write_through_another_import(OtherImport::Buffer, false, true); ++} ++ ++#[test] ++fn surface_coherent_an_own_standing_after_a_wait_keeps_a_later_buffer_write() { ++ gpu_write_through_another_import(OtherImport::Buffer, true, true); ++} diff --git a/patches/metal2vulkan/0029-coherent-a-copy-back-leaves-a-held-batch-s-record-he.patch b/patches/metal2vulkan/0029-coherent-a-copy-back-leaves-a-held-batch-s-record-he.patch new file mode 100644 index 0000000..d603000 --- /dev/null +++ b/patches/metal2vulkan/0029-coherent-a-copy-back-leaves-a-held-batch-s-record-he.patch @@ -0,0 +1,809 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 18:49:53 +0000 +Subject: [PATCH 29/34] coherent: a copy back leaves a held batch's record; + held batches copy in later writes through other imports (path 4) + +Two defects behind the path-4 lanes (ebc8cb30), both losing writes Metal keeps: + +- A GPU copy back (shadow_lazy's flush, for a command buffer that meets the bytes through + another import, met_by) covered every running record (before_flush's through = seq). A + command buffer waiting for a shared event has a batch the scheduler has not planned, which the + tracker cannot order the copy after: the copy ran before it, flushed() marked its record done, + its completion armed nothing, and its writes never reached the pages (the guest read the older + bytes). Base lanes: every failing variant made exactly one copy back, both passing none. + Now the scheduler notes each command buffer from its commit to the plan of its last batch + (shadow_lazy::planned, under the scheduler's lock, one short lock), and a copy back covers the + records up to the first one still unplanned: that one and those after it land at their + completion as ever. flushed() takes the copy's mark apart from what it covered. + Counted: flushes_held_back. +- Path 4, left by 6e193b3b: a standing taken after a wait knew nothing of a write of the + surface's bytes through another import (a buffer over the pages, another texture's copy back) + encoded after it and ordered before the signal. Each such write of a trapping surface is now + kept with its command buffer and rectangle (Lazy's Beside, from note_writes, by the byte range + against the truth's), and the held batch's late copy in also takes the rectangles written after + its standing's count by command buffers the scheduler planned all of: the copy's uses join the + batch's summary, so the tracker orders it after those writes, as Metal orders them before the + batch by the event. A write not planned yet is ordered after the batch and left (counted + late_beside_unplanned). A record still open or running over the rectangle is counted busy, + as in taking_late (the own variants count one: the command buffer's own earlier pass). + Proof that path 4 is its own defect: with the second fix off (first alone), the three buffer + variants lose the other import's write at (9,100). + +Counters: per surface late_copies, late_busy, late_compares, late_compare_ns(_max), late_beside; +a process line first in shadow_coherent::report (so coherent.jsonl carries it), with +late_copies_in(_busy), the snapshot compares (count, ns, max), late_beside(_unplanned), +flushes_held_back and the other fallbacks. A surface's line stays last. + +Lanes (ANV): the six path-4 lanes fail 4/6 at bc15b13e and pass here; unit tests +tests::coherent_late (area of a byte range, the plan set). + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/submit.rs | 11 +- + crates/metal2vulkan-runtime/src/encoder.rs | 4 +- + crates/metal2vulkan-runtime/src/shadow.rs | 25 ++- + .../src/shadow_coherent.rs | 201 +++++++++++++++++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 167 ++++++++++++++- + .../src/tests/coherent_late.rs | 47 ++++ + crates/metal2vulkan-runtime/src/tests/mod.rs | 1 + + 7 files changed, 422 insertions(+), 34 deletions(-) + create mode 100644 crates/metal2vulkan-runtime/src/tests/coherent_late.rs + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 7c41b410..4a5b8416 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -1048,6 +1048,13 @@ impl CommandBufferShared { + let handlers = (schedules && !state.submission.is_empty()) || forms; + state.submitting = handlers; + scheduler.tracker.apply(plan); ++ // LOCAL (Atlas): all of it is planned: a trapping surface's copy back planned from here ++ // is ordered after its writes (`shadow_coherent::Coherent::before_flush`). ++ if last && crate::shadow_lazy::active() { ++ if let Some(carry) = state.shadow_carry.as_ref() { ++ crate::shadow_lazy::planned(carry.id()); ++ } ++ } + held_moves.submitted(); + shared.counts.held_depths_released(held_moves.released); + let several = shared.queues.len() > 1; +@@ -1208,7 +1215,9 @@ impl CommandBufferShared { + } + let id = carry.id(); + let copies: Vec<_> = (shadows.iter()) +- .flat_map(|(left, copies)| crate::shadow_lazy::late_copy_in(copies, id, *left)) ++ .flat_map(|(left, written, copies)| { ++ crate::shadow_lazy::late_copy_in(copies, id, *left, *written) ++ }) + .collect(); + if copies.is_empty() { + return Ok(()); +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index 10dc6b45..3b04eea9 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -1680,7 +1680,7 @@ impl EncoderCore { + self.uses.record(stages, tracked, None); + // LOCAL (Atlas): this way past `track`, as a copy back of another texture over the bytes. + if writes_truth { +- crate::shadow_lazy::note_copy_back(tracked); ++ crate::shadow_lazy::note_copy_back(tracked, self.shadows.carry_id()); + } + self.touch(uses); + } +@@ -1716,7 +1716,7 @@ impl EncoderCore { + let before = run.before(tracked, copy.writes_truth()); + // LOCAL (Atlas): as in `shadow_command`. + if copy.writes_truth() { +- crate::shadow_lazy::note_copy_back(tracked); ++ crate::shadow_lazy::note_copy_back(tracked, self.shadows.carry_id()); + } + // SAFETY: the held command buffer is recording outside a render pass instance (the + // caller ended the one before), and only this encoder records it; its commands retain +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index b508e291..b89814b5 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2202,6 +2202,12 @@ impl ShadowSync { + self.entries.is_empty() + } + ++ /// LOCAL (Atlas): the identity of the command buffer the encoder records for ++ /// ([`ShadowCarry::id`]). ++ pub(super) fn carry_id(&self) -> u64 { ++ self.carry.id ++ } ++ + /// Carry no texture's image on, and let every texture the encoder met go: the encoder's + /// commands from here on read another writer's bytes, which each texture copies in again to + /// hold, and its later encoders take none of the images this one leaves (`ShadowCarry`). +@@ -2219,7 +2225,7 @@ impl ShadowSync { + /// writes nothing else here would see. + pub(super) fn wrote(&mut self, uses: &[Use]) { + // LOCAL (Atlas): any surface's, whatever this encoder uses (`shadow_lazy.rs`). +- super::shadow_lazy::note_writes(uses); ++ super::shadow_lazy::note_writes(uses, self.carry.id); + if self.carry.entries.is_empty() { + return; + } +@@ -2340,9 +2346,10 @@ pub(super) struct LateCopies { + /// The command buffer's part (`command_buffer::State::recorded`) the last wait for a shared + /// event holds back: the work encoded from here on waits for it. + since: Option, +- /// Each surface noted, with the part of the wait it was noted after, and whether an earlier +- /// encoder of the command buffer had left writes ahead in its image then. +- shadows: Vec<(usize, bool, Arc)>, ++ /// Each surface noted, with the part of the wait it was noted after, whether an earlier ++ /// encoder of the command buffer had left writes ahead in its image then, and the count of ++ /// writes of its bytes its standing was taken at (`shadow_lazy::Lazy::taken_written`). ++ shadows: Vec<(usize, bool, u64, Arc)>, + } + + impl Default for ShadowCarry { +@@ -2416,15 +2423,15 @@ impl ShadowCarry { + &mut self, + end: usize, + met: impl Fn(&ShadowCopies) -> bool, +- ) -> Vec<(bool, Arc)> { ++ ) -> Vec<(bool, u64, Arc)> { + if self.late.shadows.is_empty() { + return Vec::new(); + } + let (taken, kept): (Vec<_>, Vec<_>) = std::mem::take(&mut self.late.shadows) + .into_iter() +- .partition(|(from, _, copies)| *from < end && met(copies)); ++ .partition(|(from, _, _, copies)| *from < end && met(copies)); + self.late.shadows = kept; +- taken.into_iter().map(|(_, left, copies)| (left, copies)).collect() ++ taken.into_iter().map(|(_, left, written, copies)| (left, written, copies)).collect() + } + } + +@@ -2476,12 +2483,12 @@ impl ShadowSync { + if let (Some(_), Some(lazy), Some(since)) = (&lazily, &copies.lazy, self.carry.late.since) { + let late = &mut self.carry.late.shadows; + if lazy.whole_image() +- && !(late.iter()).any(|(from, _, known)| { ++ && !(late.iter()).any(|(from, _, _, known)| { + *from == since && known.shadow_key == copies.shadow_key + }) + { + let left = lazy.left_open_by(self.carry.id); +- late.push((since, left, Arc::clone(copies))); ++ late.push((since, left, lazy.taken_written(), Arc::clone(copies))); + } + } + let only_read = lazily.is_some() && (copies.lazy.as_ref()) +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index c8f5d54b..404e0e08 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -198,10 +198,28 @@ pub(crate) enum Fallback { + /// Of those, the copies in made while another command buffer's write over the rectangle still + /// ran, whose texels the copy may lose: the submission waits for nothing. + LateCopyInBusy, ++ /// A GPU copy back (`shadow_lazy`'s `flush`) left a running record out of what it moved: its ++ /// command buffer had batches the scheduler had not planned yet (held for an event), which ++ /// the copy is not ordered after, so the record lands at its completion as ever ++ /// ([`Coherent::before_flush`]). ++ FlushHeldBack, ++ /// A held batch's copy in took a rectangle another command buffer wrote through another ++ /// import after the batch's encoder took the image's standing, ordered before the batch by ++ /// the tracker (`shadow_lazy::late_copy_in`). ++ LateBeside, ++ /// Such a write whose command buffer was not planned yet as the batch went: the copy leaves ++ /// it, as nothing orders it before the batch. ++ LateBesideUnplanned, + } + +-const FALLBACKS: usize = Fallback::LateCopyInBusy as usize + 1; ++const FALLBACKS: usize = Fallback::LateBesideUnplanned as usize + 1; + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; ++/// The late copies in's compares of snapshot pages, every surface's ([`Coherent::taking_late`]): ++/// how many, and their ns in all and at most. Under the scheduler's lock, so the VM's smoke check ++/// reads them ([`report`]'s first line). ++static LATE_COMPARES: AtomicU64 = AtomicU64::new(0); ++static LATE_COMPARE_NS: AtomicU64 = AtomicU64::new(0); ++static LATE_COMPARE_NS_MAX: AtomicU64 = AtomicU64::new(0); + + pub(crate) fn count(what: Fallback) { + FALLBACK[what as usize].fetch_add(1, Ordering::Relaxed); +@@ -260,6 +278,13 @@ pub struct CoherentFallbackCounts { + pub late_copies_in: u64, + /// Of those, the ones made while another command buffer's write over the rectangle ran. + pub late_copies_in_busy: u64, ++ /// GPU copies back that left a running record out: its command buffer had batches unplanned. ++ pub flushes_held_back: u64, ++ /// Held batches' copies in of a rectangle another command buffer wrote through another ++ /// import after the standing was taken. ++ pub late_beside: u64, ++ /// Such writes left out: their command buffer was not planned as the batch went. ++ pub late_beside_unplanned: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -287,6 +312,9 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + signals_behind_arm: read(Fallback::SignalBehindArm), + late_copies_in: read(Fallback::LateCopyIn), + late_copies_in_busy: read(Fallback::LateCopyInBusy), ++ flushes_held_back: read(Fallback::FlushHeldBack), ++ late_beside: read(Fallback::LateBeside), ++ late_beside_unplanned: read(Fallback::LateBesideUnplanned), + } + } + +@@ -487,6 +515,17 @@ struct Counts { + probes: AtomicU64, + overwrite_risk: AtomicU64, + sync_errors: AtomicU64, ++ /// Held batches' copies in of what the CPU wrote ([`Coherent::taking_late`]), and of those ++ /// the ones counted busy. ++ late_copies: AtomicU64, ++ late_busy: AtomicU64, ++ /// Of the late copies in, the ones that compared the open pages with their snapshots under ++ /// the scheduler's lock, and what the compares took, in all and at most. ++ late_compares: AtomicU64, ++ late_compare_ns: AtomicU64, ++ late_compare_ns_max: AtomicU64, ++ /// Rectangles held batches copied in that other imports wrote ([`Coherent::taking_beside`]). ++ late_beside: AtomicU64, + } + + #[derive(Default)] +@@ -1197,7 +1236,15 @@ impl Coherent { + self.sync_done(&mut state, SyncBy::CopyIn); + let list: Vec = state.snapshots.keys().copied().collect(); + let came = pages().map(|mark| mark.hold(&list, true)).unwrap_or_default(); ++ let comparing = Instant::now(); + let (changed, whole) = self.changed(&state, area); ++ let ns = comparing.elapsed().as_nanos() as u64; ++ self.counts.late_compares.fetch_add(1, Ordering::Relaxed); ++ self.counts.late_compare_ns.fetch_add(ns, Ordering::Relaxed); ++ self.counts.late_compare_ns_max.fetch_max(ns, Ordering::Relaxed); ++ LATE_COMPARES.fetch_add(1, Ordering::Relaxed); ++ LATE_COMPARE_NS.fetch_add(ns, Ordering::Relaxed); ++ LATE_COMPARE_NS_MAX.fetch_max(ns, Ordering::Relaxed); + rects = changed; + busy |= whole && ahead(&state, &rects); + state.guest = None; +@@ -1212,14 +1259,75 @@ impl Coherent { + } + if busy { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ self.counts.late_busy.fetch_add(1, Ordering::Relaxed); + count(Fallback::LateCopyInBusy); + } + if !rects.is_empty() { ++ self.counts.late_copies.fetch_add(1, Ordering::Relaxed); + count(Fallback::LateCopyIn); + } + rects + } + ++ /// LOCAL (Atlas): the rectangle of the surface's bytes `[from, to)`, counted from its first ++ /// texel: its rows, or the texels of one row. `None` where it holds no texel. ++ pub(crate) fn area_of_bytes(&self, from: u64, to: u64) -> Option<[u32; 4]> { ++ let Layout { row, texel, width, height, .. } = self.layout; ++ if row == 0 || texel == 0 || to <= from { ++ return None; ++ } ++ let (y0, y1) = ((from / row) as u32, (to.div_ceil(row) as u32).min(height)); ++ if y0 >= y1 { ++ return None; ++ } ++ let area = match y1 - y0 { ++ 1 => { ++ let start = from - u64::from(y0) * row; ++ let end = to - u64::from(y0) * row; ++ let x0 = ((start / texel) as u32).min(width); ++ let x1 = (end.div_ceil(texel) as u32).min(width); ++ [x0, y0, x1, y1] ++ } ++ _ => [0, y0, width, y1], ++ }; ++ (area[0] < area[2]).then_some(area) ++ } ++ ++ /// Every texel of the surface. ++ pub(crate) fn whole(&self) -> [u32; 4] { ++ self.layout.whole() ++ } ++ ++ /// LOCAL (Atlas): a held batch of the command buffer `id` is about to copy in `areas`, which ++ /// other command buffers wrote through another import of the bytes after its encoder took ++ /// the image's standing, planned before it, so the tracker orders the copy after them ++ /// (`shadow_lazy::late_copy_in`). What is done goes to the pages first. A record still open ++ /// or running over them (another command buffer's, or this one's where it left writes ahead ++ /// before the wait, `left`) holds texels the pages may lack: counted busy, as in ++ /// [`taking_late`](Self::taking_late). `areas` as they go, none where the surface no longer ++ /// traps. ++ pub(crate) fn taking_beside(&self, id: u64, left: bool, areas: Vec<[u32; 4]>) -> Vec<[u32; 4]> { ++ let mut state = self.lock(); ++ if !state.tracked || state.closed || areas.is_empty() { ++ return Vec::new(); ++ } ++ self.sync_done(&mut state, SyncBy::CopyIn); ++ let busy = (state.open.iter().chain(&state.running)) ++ .filter(|record| left || record.id != id) ++ .filter(|record| record.seq == 0 || record.seq > state.flushed_to || record.id == id) ++ .any(|record| areas.iter().any(|area| meet(record.area, *area))); ++ if busy { ++ self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); ++ self.counts.late_busy.fetch_add(1, Ordering::Relaxed); ++ count(Fallback::LateCopyInBusy); ++ } ++ self.counts.late_beside.fetch_add(areas.len() as u64, Ordering::Relaxed); ++ for _ in &areas { ++ count(Fallback::LateBeside); ++ } ++ areas ++ } ++ + /// The rectangles of `area`'s texels the guest changed against its open pages' snapshots, + /// and whether they are coarser than that: a page in `area` with no snapshot takes its whole + /// rectangle, and past [`LATE_RECTS`] each band of rows changed takes its bounds, then +@@ -1324,7 +1432,15 @@ impl Coherent { + /// A GPU copy back of what ran is about to record: everything done went on the CPU, and what + /// is still running goes by the copy, up to the sequence given. `None` where nothing runs. The + /// lock is not held while it records. +- pub(crate) fn before_flush(&self, by: SyncBy) -> Option { ++ /// ++ /// LOCAL (Atlas): `Some((mark, through))`: `mark` names the copy in flight, and the copy moves ++ /// the records up to `through` alone. A record whose command buffer still has a batch the ++ /// scheduler has not planned (one held for an event, `shadow_lazy::planned`) is not covered, ++ /// nor is any after it: the tracker orders the copy after what was planned before it, so the ++ /// held batch runs after the copy, and that record lands at its completion as ever. Before, ++ /// the copy covered it, its completion then armed nothing, and its writes never reached the ++ /// pages (lane `surface_coherent_a_standing_after_a_wait_keeps_a_later_buffer_write_before_the_signal`). ++ pub(crate) fn before_flush(&self, by: SyncBy) -> Option<(u64, u64)> { + let mut state = self.lock(); + self.sync_done(&mut state, by); + if state.running.is_empty() { +@@ -1342,9 +1458,16 @@ impl Coherent { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } + } +- let through = state.seq; +- state.flushing = through; +- Some(through) ++ let held = (state.running.iter()) ++ .filter(|record| super::shadow_lazy::unplanned(record.id)) ++ .map(|record| record.seq) ++ .min(); ++ if held.is_some() { ++ count(Fallback::FlushHeldBack); ++ } ++ let mark = state.seq; ++ state.flushing = mark; ++ Some((mark, held.map_or(mark, |seq| seq - 1))) + } + + /// The copy back of everything up to `through`, of `bytes`, recorded and committed. +@@ -1360,10 +1483,11 @@ impl Coherent { + self.moved.notify_all(); + } + +- /// That copy back completed. +- pub(crate) fn flushed(&self, through: u64) { ++ /// That copy back, `mark` of [`before_flush`](Self::before_flush), completed: the records up ++ /// to `through` are in the pages. ++ pub(crate) fn flushed(&self, mark: u64, through: u64) { + let mut state = self.lock(); +- if state.flushing == through { ++ if state.flushing == mark { + state.flushing = 0; + } + state.flushed_to = state.flushed_to.max(through); +@@ -1514,7 +1638,9 @@ impl Coherent { + \"trap_ns\":{},\"trap_ns_max\":{},\"resumes\":{},\"arm_ns\":{},\"arms\":{},\ + \"hold_ns\":{},\"demotions\":{},\"eager_frames\":{},\"merges\":{},\"came\":{},\ + \"probes\":{},\"flushes\":{},\"flushed_bytes\":{},\"copy_ins\":{},\"copy_in_bytes\":{},\ +- \"overwrite_risk\":{},\"sync_errors\":{},\"pending_done\":{},\"open_pages\":{},\ ++ \"overwrite_risk\":{},\"sync_errors\":{},\"late_copies\":{},\"late_busy\":{},\ ++ \"late_compares\":{},\"late_compare_ns\":{},\"late_compare_ns_max\":{},\ ++ \"late_beside\":{},\"pending_done\":{},\"open_pages\":{},\ + \"per_present_ns\":{{\"detect\":{detect:.0},\"sync\":{sync:.0},\"fetch\":{fetch:.0},\ + \"total\":{total:.0},\"eager_would\":{eager:.0}}},\"pays\":{}}}\n", + self.layout.first_page(), +@@ -1556,6 +1682,12 @@ impl Coherent { + read(&c.copy_in_bytes), + read(&c.overwrite_risk), + read(&c.sync_errors), ++ read(&c.late_copies), ++ read(&c.late_busy), ++ read(&c.late_compares), ++ read(&c.late_compare_ns), ++ read(&c.late_compare_ns_max), ++ read(&c.late_beside), + state.done.len(), + state.snapshots.len(), + read(&c.spent_ns) < read(&c.saved_ns), +@@ -1688,9 +1820,56 @@ pub(crate) fn unarmed() -> bool { + + /// One line a surface: what [`Coherent::line`] says, under `label`. A surface that went comes once. + pub fn report(label: &str) -> String { +- let mut lines: String = std::mem::take(&mut *ended().lock().unwrap_or_else(|e| e.into_inner())).concat(); +- for surface in tracked() { ++ let ended: String = std::mem::take(&mut *ended().lock().unwrap_or_else(|e| e.into_inner())).concat(); ++ let surfaces = tracked(); ++ let mut lines = String::new(); ++ // The process's counts first, where any surface traps or went: a surface's line stays last. ++ if !ended.is_empty() || !surfaces.is_empty() { ++ lines.push_str(&process_line(label)); ++ } ++ lines.push_str(&ended); ++ for surface in surfaces { + lines.push_str(&surface.line(label, &surface.lock())); + } + lines + } ++ ++/// [`report`]'s line of the process's counts: [`fallback_counts`]' silent fallbacks, and the ++/// late copies in's compares (count, ns in all, ns at most). ++fn process_line(label: &str) -> String { ++ let c = fallback_counts(); ++ let read = |count: &AtomicU64| count.load(Ordering::Relaxed); ++ format!( ++ "{{\"process\":\"{label}\",\"late_copies_in\":{},\"late_copies_in_busy\":{},\ ++ \"late_compares\":{},\"late_compare_ns\":{},\"late_compare_ns_max\":{},\ ++ \"late_beside\":{},\"late_beside_unplanned\":{},\"flushes_held_back\":{},\ ++ \"signals_behind_arm\":{},\"met_untracked\":{},\"flush_nothing_left\":{},\ ++ \"host_reads\":{},\"host_reads_synced\":{},\"host_writes\":{},\"host_writes_opened\":{},\ ++ \"demotions\":{},\"closed_by_overlap\":{},\"closed_by_marks\":{},\"retired_dead\":{},\ ++ \"orphan_faults\":{},\"defer_refused_closed\":{},\"defer_refused_eager\":{},\ ++ \"defer_refused_wrote\":{}}}\n", ++ c.late_copies_in, ++ c.late_copies_in_busy, ++ read(&LATE_COMPARES), ++ read(&LATE_COMPARE_NS), ++ read(&LATE_COMPARE_NS_MAX), ++ c.late_beside, ++ c.late_beside_unplanned, ++ c.flushes_held_back, ++ c.signals_behind_arm, ++ c.met_untracked, ++ c.flush_nothing_left, ++ c.host_reads, ++ c.host_reads_synced, ++ c.host_writes, ++ c.host_writes_opened, ++ c.demotions, ++ c.closed_by_overlap, ++ c.closed_by_marks, ++ c.retired_dead, ++ c.orphan_faults, ++ c.defer_refused_closed, ++ c.defer_refused_eager, ++ c.defer_refused_wrote, ++ ) ++} +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 15164b8b..24d4ea28 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -187,6 +187,45 @@ fn committed_ids() -> &'static Mutex> { + IDS.get_or_init(Mutex::default) + } + ++/// LOCAL (Atlas): the command buffers committed whose last batch the scheduler has not planned ++/// yet ([`planned`]), and the last [`PLANNED_KEPT`] that it planned. ++#[derive(Default)] ++struct Plans { ++ unplanned: HashSet, ++ planned: VecDeque, ++} ++ ++/// The planned command buffers remembered: an older one counts as not planned ([`was_planned`]). ++const PLANNED_KEPT: usize = 4096; ++ ++fn plans() -> &'static Mutex { ++ static PLANS: OnceLock> = OnceLock::new(); ++ PLANS.get_or_init(Mutex::default) ++} ++ ++/// LOCAL (Atlas): the scheduler planned the last batch of the command buffer of identity `id` ++/// (`command_buffer/submit.rs`): a copy back planned after this is ordered after all of its work. ++/// Under the scheduler's lock, one short lock of its own. ++pub(crate) fn planned(id: u64) { ++ let mut plans = plans().lock().unwrap(); ++ if plans.unplanned.remove(&id) { ++ if plans.planned.len() == PLANNED_KEPT { ++ plans.planned.pop_front(); ++ } ++ plans.planned.push_back(id); ++ } ++} ++ ++/// Whether the command buffer `id` was committed and has a batch the scheduler has not planned. ++pub(crate) fn unplanned(id: u64) -> bool { ++ plans().lock().unwrap().unplanned.contains(&id) ++} ++ ++/// Whether the scheduler planned all of the command buffer `id`, lately. ++fn was_planned(id: u64) -> bool { ++ plans().lock().unwrap().planned.contains(&id) ++} ++ + /// Whether a texture over host pages of `width` and `texels` stays a shadow to render lazily + /// (`host_rt.rs`): `surface_lazy_min_width` (default none) keeps a narrow surface, a Simulator's + /// portrait display among them, out, as it is made and whatever is presented later. +@@ -334,6 +373,9 @@ pub(crate) fn display_image(bytes: &Range) -> Option { + + /// A command buffer completed, before its handlers: its writes in images trap from here. + pub(crate) fn completing(id: u64) { ++ if ANY.load(Ordering::Relaxed) { ++ planned(id); ++ } + if !crate::shadow_coherent::surfaces_tracked() { + return; + } +@@ -356,12 +398,23 @@ pub(crate) fn committed(id: u64) { + ids.pop_front(); + } + ids.push_back(id); ++ drop(ids); ++ // Before its commit hands its batches to the scheduler: none is planned yet. ++ let mut plans = plans().lock().unwrap(); ++ // A command buffer whose batches never reach the scheduler (a commit refused) stays here: ++ // bounded, the oldest is lost first, which counts it planned. ++ if plans.unplanned.len() >= PLANNED_KEPT { ++ plans.unplanned.clear(); ++ } ++ plans.unplanned.insert(id); + } + +-/// Note a recorded command with `uses` (the host memory's among them): each of its writes of a +-/// surface's truth drops that surface's standing. A surface's own copy back counts too: its +-/// encoder's end reads the count after it ([`Lazy::stand`]). +-pub(super) fn note_writes(uses: &[Use]) { ++/// Note a recorded command with `uses` (the host memory's among them) of the command buffer `id` ++/// (`shadow::ShadowCarry::id`): each of its writes of a surface's truth drops that surface's ++/// standing. A surface's own copy back counts too: its encoder's end reads the count after it ++/// ([`Lazy::stand`]). LOCAL (Atlas): a trapping surface keeps each, for a held batch whose ++/// encoder took its standing before ([`Beside`], [`late_copy_in`]). ++pub(super) fn note_writes(uses: &[Use], id: u64) { + if !ANY.load(Ordering::Relaxed) { + return; + } +@@ -369,7 +422,8 @@ pub(super) fn note_writes(uses: &[Use]) { + for used in uses.iter().filter(|used| !used.write.is_empty()) { + for lazy in all.iter().filter_map(Weak::upgrade) { + if lazy.meets(used) && !lazy.own_thread() { +- lazy.written.fetch_add(1, Ordering::Relaxed); ++ let count = lazy.written.fetch_add(1, Ordering::Relaxed) + 1; ++ lazy.note_beside(count, id, used); + } + } + } +@@ -378,11 +432,11 @@ pub(super) fn note_writes(uses: &[Use]) { + /// Note a copy back recorded past [`note_writes`]: a command of an unordered encoder. A standing of + /// another texture's image over the bytes must drop with it, which `surface_lazy` alone leaves to + /// the next resync: the new switches close that. +-pub(crate) fn note_copy_back(uses: &[Use]) { ++pub(crate) fn note_copy_back(uses: &[Use], id: u64) { + if ANY.load(Ordering::Relaxed) + && (crate::live::on(SCOPED, false) || crate::live::on(PRESENT, false)) + { +- note_writes(uses); ++ note_writes(uses, id); + } + } + +@@ -517,14 +571,29 @@ pub(crate) fn signalled_by(id: u64) -> Option, ShadowCopy, + /// ([`Coherent::taking_late`](super::shadow_coherent)). None where it wrote nothing, or the + /// surface no longer traps. + /// `left`: an earlier encoder of it left writes ahead in the image before the wait. +-pub(crate) fn late_copy_in(copies: &Arc, id: u64, left: bool) -> Vec { ++/// ++/// LOCAL (Atlas), the fourth path: with them, what other command buffers wrote of the bytes ++/// through another import (a buffer over the pages, another texture's copy back) after the ++/// encoder took the standing (`written`, [`Lazy::written`] then), where the scheduler planned ++/// that command buffer before this batch: the tracker orders the copy after its write, as Metal ++/// orders it before the batch through the event the batch waited for. A write not planned yet is ++/// ordered after the batch, and left (counted, [`Fallback::LateBesideUnplanned`]). ++pub(crate) fn late_copy_in( ++ copies: &Arc, ++ id: u64, ++ left: bool, ++ written: u64, ++) -> Vec { + let Some(lazy) = copies.lazy.as_ref() else { + return Vec::new(); + }; + let Some(coherent) = lazy.coherent() else { + return Vec::new(); + }; +- let list: Vec = (coherent.taking_late(id, left, copies.cuts()).into_iter()) ++ let mut areas = coherent.taking_late(id, left, copies.cuts()); ++ let beside = lazy.beside_since(written, id, coherent); ++ areas.extend(coherent.taking_beside(id, left, beside)); ++ let list: Vec = (areas.into_iter()) + .map(|area| ShadowCopy { + copies: Arc::clone(copies), + step: Step::Into, +@@ -783,10 +852,26 @@ pub fn ahead_counts() -> AheadCounts { + } + } + ++/// LOCAL (Atlas): the writes of a trapping surface's bytes through another import, by the ++/// [`Lazy::written`] count each made, for a held batch's copy in (`late_copy_in`): the last ++/// [`BESIDE_KEPT`], each with its command buffer's identity (0: not known) and rectangle (`None`: ++/// every texel), and the count the last one let go reached. ++#[derive(Default)] ++struct Beside { ++ writes: VecDeque<(u64, u64, Option<[u32; 4]>)>, ++ lost: u64, ++} ++ ++const BESIDE_KEPT: usize = 32; ++ + /// One surface's standing. + pub(crate) struct Lazy { + truth: OnceLock<(ResourceKey, Option, UseRange)>, + written: AtomicU64, ++ /// LOCAL (Atlas): [`Beside`], of a trapping surface. ++ beside: Mutex, ++ /// LOCAL (Atlas): [`Lazy::written`] as the last standing taken found it. ++ taken_written: AtomicU64, + stood: Mutex>, + resynced: Mutex, + /// The [`Device`] its queue comes from. +@@ -831,6 +916,8 @@ impl Lazy { + Arc::new(Lazy { + truth: OnceLock::new(), + written: AtomicU64::new(0), ++ beside: Mutex::default(), ++ taken_written: AtomicU64::new(0), + stood: Mutex::new(None), + resynced: Mutex::new(Instant::now()), + device: Arc::new(device.clone()), +@@ -1094,9 +1181,9 @@ impl Lazy { + let mut encoder = command_buffer.blit_command_encoder().ok()?; + encoder.surface_copy_back(©, holds); + encoder.end_encoding().ok()?; +- if let (Some(coherent), Some(through)) = (&coherent, through) { ++ if let (Some(coherent), Some((mark, through))) = (&coherent, through) { + let done = coherent.clone(); +- let handler = Box::new(move |_: &CommandBuffer| done.flushed(through)); ++ let handler = Box::new(move |_: &CommandBuffer| done.flushed(mark, through)); + command_buffer.add_completed_handler(handler).ok()?; + } + command_buffer.commit().ok()?; +@@ -1231,6 +1318,63 @@ impl Lazy { + } + } + ++ /// LOCAL (Atlas): a write `used` of its bytes through another import by the command buffer ++ /// `id` made the [`Lazy::written`] count `count` ([`Beside`]): kept where it traps. ++ fn note_beside(&self, count: u64, id: u64, used: &Use) { ++ let Some(coherent) = self.coherent() else { ++ return; ++ }; ++ let area = match (self.truth.get(), &used.range) { ++ (Some((_, _, UseRange::Bytes(truth))), UseRange::Bytes(bytes)) => { ++ let from = bytes.start.max(truth.start) - truth.start; ++ let to = bytes.end.min(truth.end).saturating_sub(truth.start); ++ coherent.area_of_bytes(from, to) ++ } ++ _ => None, ++ }; ++ let mut beside = self.beside.lock().unwrap(); ++ if beside.writes.len() == BESIDE_KEPT { ++ if let Some((count, _, _)) = beside.writes.pop_front() { ++ beside.lost = beside.lost.max(count); ++ } ++ } ++ beside.writes.push_back((count, id, area)); ++ } ++ ++ /// LOCAL (Atlas): the rectangles written through other imports after the count `written` by ++ /// command buffers other than `id` the scheduler planned all of ([`late_copy_in`]); one that ++ /// it has not, or one let go of, counted and left. ++ fn beside_since(&self, written: u64, id: u64, coherent: &Coherent) -> Vec<[u32; 4]> { ++ if self.written.load(Ordering::Relaxed) == written { ++ return Vec::new(); ++ } ++ let beside = self.beside.lock().unwrap(); ++ if beside.lost > written { ++ count(Fallback::LateBesideUnplanned); ++ } ++ let mut areas: Vec<[u32; 4]> = Vec::new(); ++ for (count_at, by, area) in beside.writes.iter().copied() { ++ if count_at <= written || by == id { ++ continue; ++ } ++ if by == 0 || !was_planned(by) { ++ count(Fallback::LateBesideUnplanned); ++ continue; ++ } ++ let area = area.unwrap_or_else(|| coherent.whole()); ++ if !areas.contains(&area) { ++ areas.push(area); ++ } ++ } ++ areas ++ } ++ ++ /// LOCAL (Atlas): [`Lazy::written`] as the last standing taken found it, for its note ++ /// (`shadow::LateCopies`). ++ pub(super) fn taken_written(&self) -> u64 { ++ self.taken_written.load(Ordering::Relaxed) ++ } ++ + /// Whether its standing is one an encoder of the command buffer `id` left. + fn stood_by(&self, id: u64) -> bool { + (*self.stood.lock().unwrap()).is_some_and(|stood| stood.id == id) +@@ -1286,6 +1430,7 @@ impl Lazy { + return None; + } + } ++ self.taken_written.store(stood.written, Ordering::Relaxed); + TAKEN.fetch_add(1, Ordering::Relaxed); + self.taken.fetch_add(1, Ordering::Relaxed); + Some((stood.held, stood.stale)) +diff --git a/crates/metal2vulkan-runtime/src/tests/coherent_late.rs b/crates/metal2vulkan-runtime/src/tests/coherent_late.rs +new file mode 100644 +index 00000000..0aa52845 +--- /dev/null ++++ b/crates/metal2vulkan-runtime/src/tests/coherent_late.rs +@@ -0,0 +1,47 @@ ++//! LOCAL (Atlas): the pieces of a held batch's copy in (`shadow::LateCopies`) that need no GPU: ++//! the rectangle a write of a trapping surface's bytes through another import covers ++//! (`shadow_coherent::Coherent::area_of_bytes`), and the plans a GPU copy back reads to leave a ++//! record whose command buffer still has a batch unplanned (`shadow_lazy::planned`). ++ ++use crate::shadow_coherent::{Coherent, Layout}; ++use std::sync::Weak; ++ ++fn surface() -> std::sync::Arc { ++ let layout = Layout { base: 0x10_0000, row: 4096, texel: 4, width: 1000, height: 64 }; ++ Coherent::new(layout, Box::new(|_, _, _| true), Weak::new()) ++} ++ ++#[test] ++fn a_write_inside_one_row_covers_its_texels() { ++ let surface = surface(); ++ assert_eq!(surface.area_of_bytes(4096 * 3 + 36, 4096 * 3 + 100), Some([9, 3, 25, 4])); ++ // A texel's part takes the texel. ++ assert_eq!(surface.area_of_bytes(4096 * 3 + 37, 4096 * 3 + 38), Some([9, 3, 10, 4])); ++} ++ ++#[test] ++fn a_write_over_rows_covers_them_whole() { ++ let surface = surface(); ++ assert_eq!(surface.area_of_bytes(4096 * 2 + 8, 4096 * 5 + 4), Some([0, 2, 1000, 6])); ++ assert_eq!(surface.area_of_bytes(0, 4096 * 64), Some([0, 0, 1000, 64])); ++} ++ ++#[test] ++fn a_write_past_the_texels_covers_none() { ++ let surface = surface(); ++ // The row's padding past the texels, and past the last row. ++ assert_eq!(surface.area_of_bytes(4000 + 4096, 4096 * 2), None); ++ assert_eq!(surface.area_of_bytes(4096 * 64, 4096 * 65), None); ++ assert_eq!(surface.area_of_bytes(8, 8), None); ++} ++ ++#[test] ++fn a_command_buffer_is_unplanned_from_its_commit_to_its_last_batch() { ++ // Identities no command buffer of the test process takes (`ShadowCarry::id` counts up). ++ let id = u64::MAX - 7; ++ assert!(!crate::shadow_lazy::unplanned(id)); ++ crate::shadow_lazy::committed(id); ++ assert!(crate::shadow_lazy::unplanned(id)); ++ crate::shadow_lazy::planned(id); ++ assert!(!crate::shadow_lazy::unplanned(id)); ++} +diff --git a/crates/metal2vulkan-runtime/src/tests/mod.rs b/crates/metal2vulkan-runtime/src/tests/mod.rs +index f60537af..87e655e9 100644 +--- a/crates/metal2vulkan-runtime/src/tests/mod.rs ++++ b/crates/metal2vulkan-runtime/src/tests/mod.rs +@@ -245,6 +245,7 @@ mod shader_cache; + mod shader_cache_off_thread; + mod shader_cache_queue; + mod shader_cache_time; ++mod coherent_late; + mod shadow; + #[cfg(any(m2v_gpu, m2v_cpu_icd))] + mod shadow_carried_read; diff --git a/patches/metal2vulkan/0030-coherent-exact-late-copies-beside-held-writers-copy-.patch b/patches/metal2vulkan/0030-coherent-exact-late-copies-beside-held-writers-copy-.patch new file mode 100644 index 0000000..c1188ae --- /dev/null +++ b/patches/metal2vulkan/0030-coherent-exact-late-copies-beside-held-writers-copy-.patch @@ -0,0 +1,1056 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 20:53:20 +0000 +Subject: [PATCH 30/34] coherent: exact late copies beside held writers, copy + back before late batches + +Path 4 (late_copy_in) took a write through another import only once its +writer's last batch was planned. A writer that signals after the write and +then waits for another event left its write out though Metal orders it +before the reader (late_beside_unplanned in census-r1). Each kept write now +carries the end of the parts it lies in, set as its encoder's parts are +recorded; the scheduler notes the parts planned per command buffer, and the +copy takes the write once its parts are planned. Writes left are counted by +why: late_beside_held (part held: Metal orders it after), _uncommitted, and +late_beside_unplanned now only for writes not known (ring of 256, joined +per open encoder). + +A batch planned after its commit (it waited for a shared event) that meets +a trapping surface's bytes now syncs what completed since the commit's copy +back on the CPU, and records a copy back of what runs, planned, at its +start (Coherent::before_late_batch, counted plan_syncs/plan_copies_back/ +plan_left). Before, the commit's copy back held back the reader's record +and the writer's later write was lost under the record's copy back. + +Lanes: held_writer x3, late batch copy back; surface+event 168 passed, 1 +known failure (ycbcr_textures), as at base. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/mod.rs | 13 +- + .../src/command_buffer/submit.rs | 87 ++++++- + crates/metal2vulkan-runtime/src/encoder.rs | 4 +- + crates/metal2vulkan-runtime/src/shadow.rs | 30 ++- + .../src/shadow_coherent.rs | 125 +++++++++- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 222 +++++++++++++++--- + .../src/tests/coherent_late.rs | 15 ++ + .../src/runtime_lane/surface_coherent.rs | 166 +++++++++++++ + 8 files changed, 617 insertions(+), 45 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 6d3b46ca..63c7c0b1 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -1287,7 +1287,10 @@ impl State { + /// by [`shadows_carried`](Self::shadows_carried), so an encoder abandoned without ending + /// leaves it carrying none and every texture copies in again. + pub(super) fn shadows_to_carry(&mut self) -> ShadowCarry { +- self.shadow_carry.take().unwrap_or_default() ++ let mut carry = self.shadow_carry.take().unwrap_or_default(); ++ // LOCAL (Atlas): the earlier encoders' writes lie in the parts recorded so far. ++ carry.parts_recorded(self.recorded.len()); ++ carry + } + + /// Take back what an ended encoder leaves its command buffer's later encoders +@@ -2580,6 +2583,14 @@ impl CommandBuffer { + surface.flush(crate::shadow_coherent::SyncBy::GpuReader); + } + } ++ { ++ // LOCAL (Atlas): its last encoder's writes lie in the parts it recorded. ++ let mut state = self.state(); ++ let end = state.recorded.len(); ++ if let Some(carry) = state.shadow_carry.as_mut() { ++ carry.parts_recorded(end); ++ } ++ } + crate::shadow_lazy::committed(id); + } + let committed = self.commit_settled_now(call, failure, run); +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 4a5b8416..fe00c8ce 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -458,7 +458,12 @@ impl CommandBufferShared { + state + .place(range.clone()) + .map_err(|error| recording_error(device, error))?; +- // LOCAL (Atlas): what the CPU wrote of trapping surfaces before the signal it waited for. ++ // LOCAL (Atlas): what trapping surfaces' records left in their images since the commit's ++ // copy back, then what the CPU wrote of them before the signal it waited for. ++ let waited = (state.shadow_carry.as_ref()).is_some_and(|carry| carry.waited_before(range.end)); ++ if waited { ++ self.record_copies_back_before(&mut state, range.clone())?; ++ } + self.record_late_copies_in(&mut state, range.clone())?; + // No work of an encoder that recorded nothing for a command (a buffer it named was + // detached), nor of one whose submission failed rebound or detached, goes to the queue. +@@ -1050,9 +1055,14 @@ impl CommandBufferShared { + scheduler.tracker.apply(plan); + // LOCAL (Atlas): all of it is planned: a trapping surface's copy back planned from here + // is ordered after its writes (`shadow_coherent::Coherent::before_flush`). +- if last && crate::shadow_lazy::active() { ++ // Of a batch before its last, its parts before the batch's end are: a held batch's copy in ++ // takes their writes through other imports (`shadow_lazy::late_copy_in`). ++ if crate::shadow_lazy::active() { + if let Some(carry) = state.shadow_carry.as_ref() { +- crate::shadow_lazy::planned(carry.id()); ++ match last { ++ true => crate::shadow_lazy::planned(carry.id()), ++ false => crate::shadow_lazy::planned_through(carry.id(), range.end), ++ } + } + } + held_moves.submitted(); +@@ -1195,6 +1205,77 @@ impl CommandBufferShared { + /// orders them after earlier writers, and a barrier after them, before the part's own work. + /// Nothing, and no command buffer, where the CPU wrote nothing (counted: + /// `CoherentFallbackCounts::late_copies_in`). ++ fn record_copies_back_before( ++ &self, ++ state: &mut State, ++ range: std::ops::Range, ++ ) -> Result<(), CommandBufferError> { ++ let Some(id) = state.shadow_carry.as_ref().map(|carry| carry.id()) else { ++ return Ok(()); ++ }; ++ if range.is_empty() { ++ return Ok(()); ++ } ++ let copies = { ++ let recorded = &state.recorded[range.clone()]; ++ crate::shadow_lazy::before_late_batch(id, || { ++ (recorded.iter()).flat_map(|part| part.summary.lazy_uses()).collect() ++ }) ++ }; ++ if copies.is_empty() { ++ return Ok(()); ++ } ++ let device = &self.device; ++ let mut uses = Vec::new(); ++ let mut planned = Vec::new(); ++ for (copy, _) in &copies { ++ let dispatched = copy.dispatched(device); ++ let rows = match dispatched { ++ Some(_) => None, ++ None => copy.staged_back(device), ++ }; ++ match &dispatched { ++ Some(_) => uses.extend(copy.dispatched_uses()), ++ None => uses.extend(copy.uses_with(rows.as_ref())), ++ } ++ planned.push((dispatched, rows)); ++ } ++ let mut aliased = Vec::new(); ++ if device.shared().aliases.augment_into(&uses, &mut aliased) { ++ uses = aliased; ++ } ++ let raw = self.record_raw(state, |_, buffer| { ++ let vk = device.shared().vk(); ++ // SAFETY: the command buffer is recording; the images and the buffers over the pages ++ // are live (the copies and their holds, kept with the command buffer), in `GENERAL`; ++ // the barrier the tracker plans before the part orders the copies after the records' ++ // writes, and this one the part's work after them. ++ unsafe { ++ for ((copy, _), (dispatched, rows)) in copies.iter().zip(&planned) { ++ match dispatched { ++ Some(back) => back.record(vk, buffer), ++ None => copy.record(vk, buffer, rows.as_ref()), ++ } ++ } ++ Barrier::everything().record(vk, buffer); ++ } ++ Ok(()) ++ })?; ++ let first = &mut state.recorded[range.start]; ++ first.raws.insert(0, raw); ++ first.summary.join(Summary::outside_encoders(uses)); ++ for (dispatched, rows) in planned { ++ if let Some(back) = dispatched { ++ state.retained.push(Box::new(back)); ++ } ++ if let Some(sets) = rows.and_then(|rows| rows.retained()) { ++ state.retained.push(Box::new(sets)); ++ } ++ } ++ state.retained.push(Box::new(copies)); ++ Ok(()) ++ } ++ + fn record_late_copies_in( + &self, + state: &mut State, +diff --git a/crates/metal2vulkan-runtime/src/encoder.rs b/crates/metal2vulkan-runtime/src/encoder.rs +index 3b04eea9..e3a282da 100644 +--- a/crates/metal2vulkan-runtime/src/encoder.rs ++++ b/crates/metal2vulkan-runtime/src/encoder.rs +@@ -1680,7 +1680,7 @@ impl EncoderCore { + self.uses.record(stages, tracked, None); + // LOCAL (Atlas): this way past `track`, as a copy back of another texture over the bytes. + if writes_truth { +- crate::shadow_lazy::note_copy_back(tracked, self.shadows.carry_id()); ++ self.shadows.note_copy_back(tracked); + } + self.touch(uses); + } +@@ -1716,7 +1716,7 @@ impl EncoderCore { + let before = run.before(tracked, copy.writes_truth()); + // LOCAL (Atlas): as in `shadow_command`. + if copy.writes_truth() { +- crate::shadow_lazy::note_copy_back(tracked, self.shadows.carry_id()); ++ self.shadows.note_copy_back(tracked); + } + // SAFETY: the held command buffer is recording outside a render pass instance (the + // caller ended the one before), and only this encoder records it; its commands retain +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index b89814b5..73e5c02e 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2208,6 +2208,12 @@ impl ShadowSync { + self.carry.id + } + ++ /// LOCAL (Atlas): note a copy back the encoder recorded past [`wrote`](Self::wrote) ++ /// (`shadow_lazy::note_copy_back`). ++ pub(super) fn note_copy_back(&mut self, uses: &[Use]) { ++ super::shadow_lazy::note_copy_back(uses, self.carry.id, &mut self.carry.noted); ++ } ++ + /// Carry no texture's image on, and let every texture the encoder met go: the encoder's + /// commands from here on read another writer's bytes, which each texture copies in again to + /// hold, and its later encoders take none of the images this one leaves (`ShadowCarry`). +@@ -2225,7 +2231,7 @@ impl ShadowSync { + /// writes nothing else here would see. + pub(super) fn wrote(&mut self, uses: &[Use]) { + // LOCAL (Atlas): any surface's, whatever this encoder uses (`shadow_lazy.rs`). +- super::shadow_lazy::note_writes(uses, self.carry.id); ++ super::shadow_lazy::note_writes(uses, self.carry.id, &mut self.carry.noted); + if self.carry.entries.is_empty() { + return; + } +@@ -2248,6 +2254,7 @@ impl ShadowSync { + entries: Vec::new(), + left: self.carry.left, + late: std::mem::take(&mut self.carry.late), ++ noted: std::mem::take(&mut self.carry.noted), + }; + carry.entries.append(&mut self.carry.entries); + for entry in &self.entries { +@@ -2326,6 +2333,9 @@ pub(super) struct ShadowCarry { + left: bool, + /// LOCAL (Atlas): the copies in its held batches take as they go ([`LateCopies`]). + late: LateCopies, ++ /// LOCAL (Atlas): its writes through other imports of trapping surfaces whose parts are not ++ /// all recorded yet ([`parts_recorded`](Self::parts_recorded)). ++ noted: super::shadow_lazy::Noted, + } + + /// LOCAL (Atlas): the trapping surfaces (`shadow_coherent.rs`) whose standing an encoder took +@@ -2346,6 +2356,8 @@ pub(super) struct LateCopies { + /// The command buffer's part (`command_buffer::State::recorded`) the last wait for a shared + /// event holds back: the work encoded from here on waits for it. + since: Option, ++ /// The part the first such wait holds back. ++ waited_from: Option, + /// Each surface noted, with the part of the wait it was noted after, whether an earlier + /// encoder of the command buffer had left writes ahead in its image then, and the count of + /// writes of its bytes its standing was taken at (`shadow_lazy::Lazy::taken_written`). +@@ -2361,6 +2373,7 @@ impl Default for ShadowCarry { + entries: Vec::new(), + left: false, + late: LateCopies::default(), ++ noted: Vec::new(), + } + } + } +@@ -2404,16 +2417,31 @@ impl ShadowCarry { + self.id + } + ++ /// LOCAL (Atlas): its encoders so far recorded all their parts before `end` ++ /// (`command_buffer::State::recorded`): their writes through other imports lie before it. ++ pub(super) fn parts_recorded(&mut self, end: usize) { ++ if !self.noted.is_empty() { ++ super::shadow_lazy::parts_recorded(&mut self.noted, self.id, end); ++ } ++ } ++ + /// Carry nothing, under the same identity: every shadowed texture copies in again at its + /// next use, and a first copy in of one this command buffer stood behind still reads as that. + pub(super) fn clear(&mut self) { + self.entries.clear(); + } + ++ /// LOCAL (Atlas): whether work before part `end` waits for a shared event: a batch ending there ++ /// is planned after the wait, past its commit (`shadow_lazy::before_late_batch`). ++ pub(super) fn waited_before(&self, end: usize) -> bool { ++ self.late.waited_from.is_some_and(|part| part < end) ++ } ++ + /// LOCAL (Atlas): the work encoded from part `part` on waits for a shared event + /// ([`LateCopies`]). + pub(super) fn waits_for_shared_event(&mut self, part: usize) { + self.late.since = Some(part); ++ self.late.waited_from.get_or_insert(part); + } + + /// LOCAL (Atlas): the shadows of the late copies in ([`LateCopies`]) that a batch of parts +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 404e0e08..55497598 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -207,12 +207,30 @@ pub(crate) enum Fallback { + /// import after the batch's encoder took the image's standing, ordered before the batch by + /// the tracker (`shadow_lazy::late_copy_in`). + LateBeside, +- /// Such a write whose command buffer was not planned yet as the batch went: the copy leaves +- /// it, as nothing orders it before the batch. ++ /// Such a write not known as the batch went (let go of, [`BESIDE_KEPT`] later, or its command ++ /// buffer past what is remembered): the copy leaves it, which may be stale. ++ /// ++ /// [`BESIDE_KEPT`]: super::shadow_lazy + LateBesideUnplanned, ++ /// Such a write in a part of its committed command buffer the scheduler had not planned as ++ /// the batch went (held for an event): Metal orders it after the batch, and the copy leaves it. ++ LateBesideHeld, ++ /// Such a write of a command buffer not committed as the batch went: nothing orders it before ++ /// the batch, and the copy leaves it. ++ LateBesideUncommitted, ++ /// A batch planned after its command buffer's commit (it waited for an event) meets the ++ /// surface's bytes: what completed since the commit's copy back synced first, on the CPU ++ /// ([`Coherent::before_late_batch`]). ++ PlanSync, ++ /// Such a batch: a copy back of what runs, planned, recorded at the batch's start. ++ PlanCopyBack, ++ /// Such a batch: what runs is held for an event, or the pages hold open snapshots: nothing ++ /// orders it before the batch, or no copy back may run without a wait, and it lands at its ++ /// completion as ever. ++ PlanLeft, + } + +-const FALLBACKS: usize = Fallback::LateBesideUnplanned as usize + 1; ++const FALLBACKS: usize = Fallback::PlanLeft as usize + 1; + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; + /// The late copies in's compares of snapshot pages, every surface's ([`Coherent::taking_late`]): + /// how many, and their ns in all and at most. Under the scheduler's lock, so the VM's smoke check +@@ -283,8 +301,19 @@ pub struct CoherentFallbackCounts { + /// Held batches' copies in of a rectangle another command buffer wrote through another + /// import after the standing was taken. + pub late_beside: u64, +- /// Such writes left out: their command buffer was not planned as the batch went. ++ /// Copies in that left such a write not known (let go of, or past what is remembered). + pub late_beside_unplanned: u64, ++ /// Copies in that left such a write in a part its committed command buffer had not planned: ++ /// ordered after the batch. ++ pub late_beside_held: u64, ++ /// Copies in that left such a write of a command buffer not committed: ordered after it. ++ pub late_beside_uncommitted: u64, ++ /// Batches planned after their commit that met a surface with records done: synced first. ++ pub plan_syncs: u64, ++ /// Such batches that carried a copy back of what ran, planned. ++ pub plan_copies_back: u64, ++ /// Such batches that met what ran, held, or a surface with open snapshots: left. ++ pub plan_left: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -315,6 +344,11 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + flushes_held_back: read(Fallback::FlushHeldBack), + late_beside: read(Fallback::LateBeside), + late_beside_unplanned: read(Fallback::LateBesideUnplanned), ++ late_beside_held: read(Fallback::LateBesideHeld), ++ late_beside_uncommitted: read(Fallback::LateBesideUncommitted), ++ plan_syncs: read(Fallback::PlanSync), ++ plan_copies_back: read(Fallback::PlanCopyBack), ++ plan_left: read(Fallback::PlanLeft), + } + } + +@@ -567,6 +601,9 @@ struct State { + /// The last sync's time, which a trap's own time holds. + synced_ns: u64, + seen: HashSet, ++ /// LOCAL (Atlas): copies back recorded in a batch ([`Coherent::before_late_batch`]): the ++ /// command buffer, the mark and the sequence covered, for its completion. ++ owed: Vec<(u64, u64, u64)>, + } + + /// A surface's trap coherence, beside its [`Lazy`]. +@@ -780,6 +817,17 @@ impl Coherent { + /// The command buffer `id` is done, before anything tells the guest: its pages trap from here. + pub(crate) fn completing(&self, id: u64) { + let mut state = self.lock(); ++ // LOCAL (Atlas): a copy back its batch carried landed with it. ++ if state.owed.iter().any(|owed| owed.0 == id) { ++ let (mine, kept): (Vec<_>, Vec<_>) = state.owed.iter().partition(|owed| owed.0 == id); ++ state.owed = kept; ++ drop(state); ++ // Mark 0: a copy back of `flushing` may share the mark, and stays in flight. ++ for (_, _, through) in mine { ++ self.flushed(0, through); ++ } ++ state = self.lock(); ++ } + let found = match state.running.iter().position(|record| record.id == id) { + Some(at) => Some(state.running.swap_remove(at)), + // Its commit has not returned yet. +@@ -1003,7 +1051,7 @@ impl Coherent { + let until = Instant::now() + Duration::from_millis(knob("surface_coherent_wait_ms", DEFAULT_WAIT_MS)); + loop { + let overlaps = |area| state.running.iter().any(|record| meet(record.area, area)); +- let busy = state.flushing != 0 || area.is_some_and(overlaps); ++ let busy = state.flushing != 0 || !state.owed.is_empty() || area.is_some_and(overlaps); + if !busy { + return (state, true); + } +@@ -1230,7 +1278,7 @@ impl Coherent { + .any(|record| rects.iter().any(|rect| meet(record.area, *rect))) + }; + let behind = ahead(&state, &[area]); +- let mut busy = state.flushing != 0 && !state.snapshots.is_empty(); ++ let mut busy = (state.flushing != 0 || !state.owed.is_empty()) && !state.snapshots.is_empty(); + let mut rects = vec![area]; + if behind && cut && !state.snapshots.is_empty() { + self.sync_done(&mut state, SyncBy::CopyIn); +@@ -1470,6 +1518,60 @@ impl Coherent { + Some((mark, held.map_or(mark, |seq| seq - 1))) + } + ++ /// LOCAL (Atlas): a batch of the command buffer `id` planned after its commit (it waited for an ++ /// event, under the scheduler's lock) meets the surface's bytes, which its commit's copy back ++ /// (`shadow_lazy::met_by`) covered only as far as what ran then: what completed since syncs ++ /// now, on the CPU (complete, nothing to wait for), so the batch's writes of the pages stay ++ /// after it; what runs, planned, the batch copies back first (`Some((mark, through))`, landed at ++ /// its completion). What runs held for an event is not ordered before the batch by Metal ++ /// either, and lands at its completion as ever; open snapshots ask a wait, which the ++ /// scheduler's lock cannot take: both left, counted. ++ pub(crate) fn before_late_batch(&self, id: u64) -> Option<(u64, u64)> { ++ let mut state = self.lock(); ++ if state.closed || !state.tracked { ++ return None; ++ } ++ if !state.done.is_empty() { ++ count(Fallback::PlanSync); ++ self.sync_done(&mut state, SyncBy::GpuReader); ++ } ++ let flushed_to = state.flushed_to; ++ let others = |record: &&Record| record.id != id && record.seq > flushed_to; ++ if !state.running.iter().any(|record| others(&record)) { ++ return None; ++ } ++ if !state.snapshots.is_empty() { ++ count(Fallback::PlanLeft); ++ return None; ++ } ++ let held = (state.running.iter()) ++ .filter(others) ++ .filter(|record| super::shadow_lazy::unplanned(record.id)) ++ .map(|record| record.seq) ++ .min(); ++ let mark = state.seq; ++ let through = held.map_or(mark, |seq| seq - 1); ++ if through <= flushed_to { ++ count(Fallback::PlanLeft); ++ return None; ++ } ++ if held.is_some() { ++ count(Fallback::PlanLeft); ++ } ++ // In flight beside any copy back of `flushing`: `owed` keeps the surface busy meanwhile. ++ count(Fallback::PlanCopyBack); ++ state.owed.push((id, mark, through)); ++ Some((mark, through)) ++ } ++ ++ /// LOCAL (Atlas): the copy back [`before_late_batch`](Self::before_late_batch) asked of the ++ /// batch of the command buffer `id` was not recorded after all. ++ pub(crate) fn late_batch_failed(&self, id: u64, mark: u64) { ++ let mut state = self.lock(); ++ state.owed.retain(|owed| (owed.0, owed.1) != (id, mark)); ++ self.moved.notify_all(); ++ } ++ + /// The copy back of everything up to `through`, of `bytes`, recorded and committed. + pub(crate) fn flush_recorded(&self, bytes: u64) { + self.counts.flushes.fetch_add(1, Ordering::Relaxed); +@@ -1563,7 +1665,7 @@ impl Coherent { + fn unmapping(&self) -> bool { + let mut state = self.lock(); + self.sync_done(&mut state, SyncBy::Unmap); +- if !state.running.is_empty() || state.flushing != 0 { ++ if !state.running.is_empty() || state.flushing != 0 || !state.owed.is_empty() { + return false; + } + state.open.clear(); +@@ -1842,7 +1944,9 @@ fn process_line(label: &str) -> String { + format!( + "{{\"process\":\"{label}\",\"late_copies_in\":{},\"late_copies_in_busy\":{},\ + \"late_compares\":{},\"late_compare_ns\":{},\"late_compare_ns_max\":{},\ +- \"late_beside\":{},\"late_beside_unplanned\":{},\"flushes_held_back\":{},\ ++ \"late_beside\":{},\"late_beside_unplanned\":{},\"late_beside_held\":{},\ ++ \"late_beside_uncommitted\":{},\"plan_syncs\":{},\"plan_copies_back\":{},\ ++ \"plan_left\":{},\"flushes_held_back\":{},\ + \"signals_behind_arm\":{},\"met_untracked\":{},\"flush_nothing_left\":{},\ + \"host_reads\":{},\"host_reads_synced\":{},\"host_writes\":{},\"host_writes_opened\":{},\ + \"demotions\":{},\"closed_by_overlap\":{},\"closed_by_marks\":{},\"retired_dead\":{},\ +@@ -1855,6 +1959,11 @@ fn process_line(label: &str) -> String { + read(&LATE_COMPARE_NS_MAX), + c.late_beside, + c.late_beside_unplanned, ++ c.late_beside_held, ++ c.late_beside_uncommitted, ++ c.plan_syncs, ++ c.plan_copies_back, ++ c.plan_left, + c.flushes_held_back, + c.signals_behind_arm, + c.met_untracked, +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 24d4ea28..fce5d182 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -98,7 +98,7 @@ use super::tracker::{Summary, Use, UseRange}; + use crate::exit_guard; + use std::any::Any; + use std::cell::Cell; +-use std::collections::{HashSet, VecDeque}; ++use std::collections::{HashMap, HashSet, VecDeque}; + use std::ops::Range; + use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex, OnceLock, RwLock, Weak}; +@@ -193,6 +193,23 @@ fn committed_ids() -> &'static Mutex> { + struct Plans { + unplanned: HashSet, + planned: VecDeque, ++ /// Of those not planned to their end, the end of the parts planned (`batch_range(..).end`). ++ through: HashMap, ++} ++ ++/// How a write through another import of a command buffer's parts before `part` stands against a ++/// held batch going now ([`ordered`]). ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++pub(crate) enum Order { ++ /// The scheduler planned every part it lies in: the tracker orders a copy planned now after it. ++ Before, ++ /// Committed, its part not planned: held for an event the batch going now is not ordered ++ /// after, so Metal orders the write after the batch too. ++ Held, ++ /// Not committed (or its encoder still open): nothing orders it before the batch. ++ Uncommitted, ++ /// Past what is remembered: not known. ++ Unknown, + } + + /// The planned command buffers remembered: an older one counts as not planned ([`was_planned`]). +@@ -208,6 +225,7 @@ fn plans() -> &'static Mutex { + /// Under the scheduler's lock, one short lock of its own. + pub(crate) fn planned(id: u64) { + let mut plans = plans().lock().unwrap(); ++ plans.through.remove(&id); + if plans.unplanned.remove(&id) { + if plans.planned.len() == PLANNED_KEPT { + plans.planned.pop_front(); +@@ -216,16 +234,41 @@ pub(crate) fn planned(id: u64) { + } + } + ++/// LOCAL (Atlas): the scheduler planned the parts of the command buffer `id` before `end`, a batch ++/// that does not end it (`command_buffer/submit.rs`). ++pub(crate) fn planned_through(id: u64, end: usize) { ++ let mut plans = plans().lock().unwrap(); ++ if plans.unplanned.contains(&id) { ++ let through = plans.through.entry(id).or_insert(0); ++ *through = (*through).max(end); ++ } ++} ++ ++/// How a write of the command buffer `id` in its parts before `part` (`usize::MAX`: its encoder ++/// is open) stands against the batch going now. ++pub(crate) fn ordered(id: u64, part: usize) -> Order { ++ let plans = plans().lock().unwrap(); ++ if plans.unplanned.contains(&id) { ++ return match plans.through.get(&id).is_some_and(|&through| through >= part) { ++ true => Order::Before, ++ false => Order::Held, ++ }; ++ } ++ if plans.planned.contains(&id) { ++ return Order::Before; ++ } ++ drop(plans); ++ match committed_ids().lock().unwrap().contains(&id) { ++ true => Order::Unknown, ++ false => Order::Uncommitted, ++ } ++} ++ + /// Whether the command buffer `id` was committed and has a batch the scheduler has not planned. + pub(crate) fn unplanned(id: u64) -> bool { + plans().lock().unwrap().unplanned.contains(&id) + } + +-/// Whether the scheduler planned all of the command buffer `id`, lately. +-fn was_planned(id: u64) -> bool { +- plans().lock().unwrap().planned.contains(&id) +-} +- + /// Whether a texture over host pages of `width` and `texels` stays a shadow to render lazily + /// (`host_rt.rs`): `surface_lazy_min_width` (default none) keeps a narrow surface, a Simulator's + /// portrait display among them, out, as it is made and whatever is presented later. +@@ -405,6 +448,7 @@ pub(crate) fn committed(id: u64) { + // bounded, the oldest is lost first, which counts it planned. + if plans.unplanned.len() >= PLANNED_KEPT { + plans.unplanned.clear(); ++ plans.through.clear(); + } + plans.unplanned.insert(id); + } +@@ -414,7 +458,7 @@ pub(crate) fn committed(id: u64) { + /// standing. A surface's own copy back counts too: its encoder's end reads the count after it + /// ([`Lazy::stand`]). LOCAL (Atlas): a trapping surface keeps each, for a held batch whose + /// encoder took its standing before ([`Beside`], [`late_copy_in`]). +-pub(super) fn note_writes(uses: &[Use], id: u64) { ++pub(super) fn note_writes(uses: &[Use], id: u64, noted: &mut Noted) { + if !ANY.load(Ordering::Relaxed) { + return; + } +@@ -423,20 +467,37 @@ pub(super) fn note_writes(uses: &[Use], id: u64) { + for lazy in all.iter().filter_map(Weak::upgrade) { + if lazy.meets(used) && !lazy.own_thread() { + let count = lazy.written.fetch_add(1, Ordering::Relaxed) + 1; +- lazy.note_beside(count, id, used); ++ if lazy.note_beside(count, id, used) { ++ noted.push((Arc::downgrade(&lazy), count)); ++ } + } + } + } + } + ++/// LOCAL (Atlas): the writes [`note_writes`] kept ([`Beside`]) of an encoder whose parts are not ++/// all recorded yet: their surface and count, for [`parts_recorded`]. ++pub(crate) type Noted = Vec<(Weak, u64)>; ++ ++/// LOCAL (Atlas): the encoders that made the writes `noted` recorded all their parts before ++/// `end` (`command_buffer::State::recorded`): each write lies in a part before it. ++/// `id`: their command buffer's identity. ++pub(crate) fn parts_recorded(noted: &mut Noted, id: u64, end: usize) { ++ for (lazy, count) in noted.drain(..) { ++ if let Some(lazy) = lazy.upgrade() { ++ lazy.beside_recorded(count, id, end); ++ } ++ } ++} ++ + /// Note a copy back recorded past [`note_writes`]: a command of an unordered encoder. A standing of + /// another texture's image over the bytes must drop with it, which `surface_lazy` alone leaves to + /// the next resync: the new switches close that. +-pub(crate) fn note_copy_back(uses: &[Use], id: u64) { ++pub(crate) fn note_copy_back(uses: &[Use], id: u64, noted: &mut Noted) { + if ANY.load(Ordering::Relaxed) + && (crate::live::on(SCOPED, false) || crate::live::on(PRESENT, false)) + { +- note_writes(uses, id); ++ note_writes(uses, id, noted); + } + } + +@@ -576,8 +637,8 @@ pub(crate) fn signalled_by(id: u64) -> Option, ShadowCopy, + /// through another import (a buffer over the pages, another texture's copy back) after the + /// encoder took the standing (`written`, [`Lazy::written`] then), where the scheduler planned + /// that command buffer before this batch: the tracker orders the copy after its write, as Metal +-/// orders it before the batch through the event the batch waited for. A write not planned yet is +-/// ordered after the batch, and left (counted, [`Fallback::LateBesideUnplanned`]). ++/// orders it before the batch through the event the batch waited for. A write whose part is not ++/// planned yet is ordered after the batch, by Metal too, and left ([`Lazy::beside_since`]). + pub(crate) fn late_copy_in( + copies: &Arc, + id: u64, +@@ -610,6 +671,47 @@ pub(crate) fn late_copy_in( + list + } + ++/// LOCAL (Atlas): the copies back a batch of the command buffer `id` planned after its commit (it ++/// waited for an event), its parts' uses `uses`, takes first, of the trapping surfaces whose bytes ++/// they meet ([`Coherent::before_late_batch`]): its commit's copy back ([`met_by`]) covered what ++/// ran then, and what completed or went since lies before the batch too. Under the scheduler's ++/// lock; nothing waits. With each, what it holds until it completes. ++pub(crate) fn before_late_batch<'a>( ++ id: u64, ++ uses: impl Fn() -> Vec<&'a Use>, ++) -> Vec<(ShadowCopy, Vec)> { ++ if !ahead() { ++ return Vec::new(); ++ } ++ let uses = uses(); ++ ++ let mut copies = Vec::new(); ++ for lazy in surfaces() { ++ let Some(coherent) = lazy.coherent() else { ++ continue; ++ }; ++ if lazy.own_thread() || !uses.iter().any(|used| lazy.meets(used)) { ++ continue; ++ } ++ let Some((mark, _)) = coherent.before_late_batch(id) else { ++ continue; ++ }; ++ let left = (lazy.pending.lock().unwrap().as_ref()) ++ .and_then(|left| Some((left.settled?, left.copies.clone(), left.holds.clone()))); ++ let Some((area, held, holds)) = left else { ++ count(Fallback::FlushNothingLeft); ++ coherent.late_batch_failed(id, mark); ++ continue; ++ }; ++ let copy: ShadowCopy = copy_back_of(held, area); ++ coherent.flush_recorded(copy.texels_back() * 4); ++ FLUSHES.fetch_add(1, Ordering::Relaxed); ++ FLUSHED_TEXELS.fetch_add(copy.texels_back(), Ordering::Relaxed); ++ copies.push((copy, holds)); ++ } ++ copies ++} ++ + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what + /// that command buffer's earlier encoders left ahead in a trapping surface whose bytes it meets. + /// The record stays, whose texels the copy repeats, so what the record holds lasts the copy. +@@ -856,13 +958,28 @@ pub fn ahead_counts() -> AheadCounts { + /// [`Lazy::written`] count each made, for a held batch's copy in (`late_copy_in`): the last + /// [`BESIDE_KEPT`], each with its command buffer's identity (0: not known) and rectangle (`None`: + /// every texel), and the count the last one let go reached. ++/// ++/// Each write is kept as a [`BesideWrite`]; one of the same command buffer's open encoder right ++/// after another joins it, its rectangle the two's bounding one: the pages hold the truth over all ++/// of it as the copy in reads them (it syncs what is done first, and refuses what runs). + #[derive(Default)] + struct Beside { +- writes: VecDeque<(u64, u64, Option<[u32; 4]>)>, ++ writes: VecDeque, + lost: u64, + } + +-const BESIDE_KEPT: usize = 32; ++#[derive(Clone, Copy)] ++struct BesideWrite { ++ /// The [`Lazy::written`] count it made (the last, of joined ones). ++ count: u64, ++ id: u64, ++ /// The end of its command buffer's parts it lies before; `usize::MAX` while its encoder is ++ /// open ([`parts_recorded`]). ++ part: usize, ++ area: Option<[u32; 4]>, ++} ++ ++const BESIDE_KEPT: usize = 256; + + /// One surface's standing. + pub(crate) struct Lazy { +@@ -1320,9 +1437,9 @@ impl Lazy { + + /// LOCAL (Atlas): a write `used` of its bytes through another import by the command buffer + /// `id` made the [`Lazy::written`] count `count` ([`Beside`]): kept where it traps. +- fn note_beside(&self, count: u64, id: u64, used: &Use) { ++ fn note_beside(&self, count: u64, id: u64, used: &Use) -> bool { + let Some(coherent) = self.coherent() else { +- return; ++ return false; + }; + let area = match (self.truth.get(), &used.range) { + (Some((_, _, UseRange::Bytes(truth))), UseRange::Bytes(bytes)) => { +@@ -1333,39 +1450,84 @@ impl Lazy { + _ => None, + }; + let mut beside = self.beside.lock().unwrap(); ++ if let Some(last) = beside.writes.back_mut() { ++ if last.id == id && last.part == usize::MAX { ++ last.count = count; ++ last.area = match (last.area, area) { ++ (Some(_), Some(_)) => bounding(last.area, area), ++ _ => None, ++ }; ++ return true; ++ } ++ } + if beside.writes.len() == BESIDE_KEPT { +- if let Some((count, _, _)) = beside.writes.pop_front() { +- beside.lost = beside.lost.max(count); ++ if let Some(gone) = beside.writes.pop_front() { ++ beside.lost = beside.lost.max(gone.count); ++ } ++ } ++ beside.writes.push_back(BesideWrite { count, id, part: usize::MAX, area }); ++ true ++ } ++ ++ /// LOCAL (Atlas): the write that made `count` lies in its command buffer's parts before `end`. ++ fn beside_recorded(&self, count: u64, id: u64, end: usize) { ++ let mut beside = self.beside.lock().unwrap(); ++ // Joined writes share one entry, whose count is the last's: the first at or past `count` ++ // (another's where that one was let go of). ++ if let Some(write) = beside.writes.iter_mut().find(|write| write.count >= count) { ++ if write.id == id && write.part == usize::MAX { ++ write.part = end; + } + } +- beside.writes.push_back((count, id, area)); + } + + /// LOCAL (Atlas): the rectangles written through other imports after the count `written` by +- /// command buffers other than `id` the scheduler planned all of ([`late_copy_in`]); one that +- /// it has not, or one let go of, counted and left. ++ /// command buffers other than `id` in parts the scheduler planned ([`late_copy_in`]): the ++ /// tracker orders the copy after those, as Metal orders them before the batch. The others are ++ /// ordered after the batch by Metal too, and left, counted by why ([`ordered`]); one let go of ++ /// before it was looked at is not known ([`Fallback::LateBesideUnplanned`]). + fn beside_since(&self, written: u64, id: u64, coherent: &Coherent) -> Vec<[u32; 4]> { + if self.written.load(Ordering::Relaxed) == written { + return Vec::new(); + } + let beside = self.beside.lock().unwrap(); +- if beside.lost > written { +- count(Fallback::LateBesideUnplanned); +- } ++ let mut unknown = beside.lost > written; ++ let (mut held, mut uncommitted) = (false, false); + let mut areas: Vec<[u32; 4]> = Vec::new(); +- for (count_at, by, area) in beside.writes.iter().copied() { +- if count_at <= written || by == id { ++ for write in beside.writes.iter().copied() { ++ if write.count <= written || write.id == id { + continue; + } +- if by == 0 || !was_planned(by) { +- count(Fallback::LateBesideUnplanned); +- continue; ++ match ordered(write.id, write.part) { ++ Order::Before => {} ++ Order::Held => { ++ held = true; ++ continue; ++ } ++ Order::Uncommitted => { ++ uncommitted = true; ++ continue; ++ } ++ Order::Unknown => { ++ unknown = true; ++ continue; ++ } + } +- let area = area.unwrap_or_else(|| coherent.whole()); ++ let area = write.area.unwrap_or_else(|| coherent.whole()); + if !areas.contains(&area) { + areas.push(area); + } + } ++ // One count of each per copy in. ++ for (left, why) in [ ++ (held, Fallback::LateBesideHeld), ++ (uncommitted, Fallback::LateBesideUncommitted), ++ (unknown, Fallback::LateBesideUnplanned), ++ ] { ++ if left { ++ count(why); ++ } ++ } + areas + } + +diff --git a/crates/metal2vulkan-runtime/src/tests/coherent_late.rs b/crates/metal2vulkan-runtime/src/tests/coherent_late.rs +index 0aa52845..fe1ee15b 100644 +--- a/crates/metal2vulkan-runtime/src/tests/coherent_late.rs ++++ b/crates/metal2vulkan-runtime/src/tests/coherent_late.rs +@@ -45,3 +45,18 @@ fn a_command_buffer_is_unplanned_from_its_commit_to_its_last_batch() { + crate::shadow_lazy::planned(id); + assert!(!crate::shadow_lazy::unplanned(id)); + } ++ ++#[test] ++fn a_write_is_ordered_before_once_its_parts_are_planned() { ++ use crate::shadow_lazy::{ordered, Order}; ++ let id = u64::MAX - 9; ++ assert_eq!(ordered(id, 1), Order::Uncommitted); ++ crate::shadow_lazy::committed(id); ++ assert_eq!(ordered(id, 1), Order::Held); ++ assert_eq!(ordered(id, usize::MAX), Order::Held, "its encoder still open"); ++ crate::shadow_lazy::planned_through(id, 1); ++ assert_eq!(ordered(id, 1), Order::Before); ++ assert_eq!(ordered(id, 2), Order::Held, "a part after the wait"); ++ crate::shadow_lazy::planned(id); ++ assert_eq!(ordered(id, 2), Order::Before); ++} +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index 15551a37..f2faeb89 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -1947,3 +1947,169 @@ fn surface_coherent_a_standing_after_a_wait_keeps_a_later_buffer_write_signalled + fn surface_coherent_an_own_standing_after_a_wait_keeps_a_later_buffer_write() { + gpu_write_through_another_import(OtherImport::Buffer, true, true); + } ++ ++/// Where the writer through another import holds back, in [`held_writer`]. ++#[derive(Clone, Copy, Debug, PartialEq, Eq)] ++enum WriterHeld { ++ /// It writes, signals the reader's event, then waits for `gate` before more work: its last ++ /// batch is not planned as the reader's batch goes. Metal orders the write before the reader. ++ AfterItsSignal, ++ /// It waits for `gate` before its write, which comes only after the reader completed: Metal ++ /// orders the write after the reader, whose blend it then replaces. ++ BeforeItsWrite, ++ /// As `BeforeItsWrite`, the gate set by the reader's own signal after its blend: the writer's ++ /// batch goes while the reader runs. ++ BeforeItsWriteAfterTheReadersSignal, ++} ++ ++/// The writer of [`gpu_write_through_another_import`] through a buffer of its own, with a batch ++/// held for `gate` (`held`). The reader waits for `hold`: set by the writer's signal (after its ++/// write), or by the CPU (before the write, which waits for the gate the CPU sets once the reader ++/// completed). ++fn held_writer(held: WriterHeld) { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, None); ++ if held == WriterHeld::BeforeItsWriteAfterTheReadersSignal { ++ waiting.encode_signal_event((&gate).into(), 1).unwrap(); ++ } ++ waiting.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ let bytes: Vec = (0..16).flat_map(|_| [1u8, 2, 3, 4]).collect(); ++ let source = staged(case.device, &bytes); ++ let blit_write = |writer: &CommandBuffer| { ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 100 * ROW, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ }; ++ let elsewhere = staged(case.device, &[9u8; 64]); ++ match held { ++ WriterHeld::AfterItsSignal => { ++ blit_write(&writer); ++ writer.encode_signal_event((&hold).into(), 1).unwrap(); ++ writer.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ // Later work of its own, past the surface's rows. ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&elsewhere, 0, &other_buffer, 255 * ROW + 4032, 64) ++ .unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ gate.set_signaled_value(1); ++ } ++ WriterHeld::BeforeItsWrite => { ++ writer.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ blit_write(&writer); ++ writer.commit().unwrap(); ++ hold.set_signaled_value(1); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ gate.set_signaled_value(1); ++ } ++ WriterHeld::BeforeItsWriteAfterTheReadersSignal => { ++ writer.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ blit_write(&writer); ++ writer.commit().unwrap(); ++ hold.set_signaled_value(1); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ } ++ } ++ writer.wait_until_completed_for(SOON).unwrap(); ++ let counts = shadow_coherent::fallback_counts(); ++ eprintln!( ++ "HELD {held:?} late_beside={} unplanned={} held={} uncommitted={} held_back={} busy={} \ ++ plan sync/back/left={}/{}/{} (9,100)={:?} (0,0)={:?} behind(9,100)={:?}", ++ counts.late_beside, ++ counts.late_beside_unplanned, ++ counts.late_beside_held, ++ counts.late_beside_uncommitted, ++ counts.flushes_held_back, ++ counts.late_copies_in_busy, ++ counts.plan_syncs, ++ counts.plan_copies_back, ++ counts.plan_left, ++ case.read(9, 100), ++ case.read(0, 0), ++ case.behind(9, 100), ++ ); ++ let wrote = match held { ++ WriterHeld::AfterItsSignal => [1, 2, 3, 0], ++ WriterHeld::BeforeItsWrite | WriterHeld::BeforeItsWriteAfterTheReadersSignal => { ++ [1, 2, 3, 4] ++ } ++ }; ++ assert_eq!(case.read(9, 100), wrote, "the write through the other import, in its order"); ++ assert_eq!(case.read(0, 0), [0, 0, 255, 0], "the blend"); ++ }); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_keeps_a_write_whose_writer_holds_after_its_signal() { ++ held_writer(WriterHeld::AfterItsSignal); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_leaves_a_write_held_past_the_reader() { ++ held_writer(WriterHeld::BeforeItsWrite); ++} ++ ++#[test] ++fn surface_coherent_a_standing_after_a_wait_leaves_a_write_held_past_the_readers_signal() { ++ held_writer(WriterHeld::BeforeItsWriteAfterTheReadersSignal); ++} ++ ++/// A writer through another import whose batch waits for an event the CPU sets while a held ++/// reader runs (nothing orders the two, so Metal allows either result): its batch goes planned ++/// after the reader's, and carries the copy back of the reader's record first ++/// (`shadow_lazy::before_late_batch`), so the pages hold one order or the other, whole. ++#[test] ++fn surface_coherent_a_late_batch_through_another_import_copies_back_what_runs_first() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let before = shadow_coherent::fallback_counts(); ++ let hold = case.device.new_shared_event().unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, None); ++ // Long enough to still run as the writer's batch goes. ++ for _ in 0..64 { ++ case.pass(&waiting, GREEN, scissor(0, 200, WIDTH, 1)); ++ } ++ waiting.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ writer.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ let bytes: Vec = (0..16).flat_map(|_| [1u8, 2, 3, 4]).collect(); ++ let source = staged(case.device, &bytes); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 100 * ROW, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ hold.set_signaled_value(1); ++ gate.set_signaled_value(1); ++ for command_buffer in [&writer, &waiting] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ let after = shadow_coherent::fallback_counts(); ++ let read = case.read(9, 100); ++ eprintln!( ++ "LATEBATCH sync/back/left={}/{}/{} (9,100)={read:?} (0,200)={:?}", ++ after.plan_syncs - before.plan_syncs, ++ after.plan_copies_back - before.plan_copies_back, ++ after.plan_left - before.plan_left, ++ case.read(0, 200), ++ ); ++ assert!(read == [1, 2, 3, 4] || read == [0, 0, 255, 0], "one order, whole: {read:?}"); ++ assert_eq!(case.read(0, 0), [0, 0, 255, 0], "the blend"); ++ assert_eq!(case.read(0, 200), GREEN_TEXEL, "the reader's passes"); ++ let report = shadow_coherent::report("late-batch"); ++ assert!(!report.contains("\"sync_errors\":1"), "{report}"); ++ }); ++} diff --git a/patches/metal2vulkan/0031-coherent-split-plan_left_planned-from-plan_left-lane.patch b/patches/metal2vulkan/0031-coherent-split-plan_left_planned-from-plan_left-lane.patch new file mode 100644 index 0000000..d95c2f9 --- /dev/null +++ b/patches/metal2vulkan/0031-coherent-split-plan_left_planned-from-plan_left-lane.patch @@ -0,0 +1,363 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 23:16:10 +0000 +Subject: [PATCH 31/34] coherent: split plan_left_planned from plan_left; lanes + for a reader planned in part + +r2 counted plan_left 34 (= plan_copies_back): each late batch's copy back +left a record of a command buffer with an unplanned batch. That is exact +where none of it is planned (plan_left). Where it is planned in part, its +planned writes land only at its completion, over the batch's write through +another import unless the reader's work after its wait uses the surface +(path 4 copies it in): counted plan_left_planned. Lane +..._planned_in_part passes; ..._planned_in_part_elsewhere_after shows the +gap and is ignored until records keep their areas by part. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/shadow_coherent.rs | 65 +++++++++--- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 58 ++++++---- + .../src/runtime_lane/surface_coherent.rs | 100 ++++++++++++++++++ + 3 files changed, 185 insertions(+), 38 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 55497598..55a1cfd7 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -228,9 +228,13 @@ pub(crate) enum Fallback { + /// orders it before the batch, or no copy back may run without a wait, and it lands at its + /// completion as ever. + PlanLeft, ++ /// As `PlanLeft`, a record left whose command buffer the scheduler planned in part: its planned ++ /// writes may lie before the batch and land only at its completion, over what the batch wrote ++ /// of the bytes where they meet (lane `..._planned_in_part_elsewhere_after`). Not closed yet. ++ PlanLeftPlanned, + } + +-const FALLBACKS: usize = Fallback::PlanLeft as usize + 1; ++const FALLBACKS: usize = Fallback::PlanLeftPlanned as usize + 1; + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; + /// The late copies in's compares of snapshot pages, every surface's ([`Coherent::taking_late`]): + /// how many, and their ns in all and at most. Under the scheduler's lock, so the VM's smoke check +@@ -314,6 +318,8 @@ pub struct CoherentFallbackCounts { + pub plan_copies_back: u64, + /// Such batches that met what ran, held, or a surface with open snapshots: left. + pub plan_left: u64, ++ /// Of what such batches left, records whose command buffer was planned in part. ++ pub plan_left_planned: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -349,6 +355,7 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + plan_syncs: read(Fallback::PlanSync), + plan_copies_back: read(Fallback::PlanCopyBack), + plan_left: read(Fallback::PlanLeft), ++ plan_left_planned: read(Fallback::PlanLeftPlanned), + } + } + +@@ -606,6 +613,17 @@ struct State { + owed: Vec<(u64, u64, u64)>, + } + ++/// LOCAL (Atlas): what a batch planned after its commit takes of a trapping surface ++/// ([`Coherent::before_late_batch`]). ++#[derive(Default)] ++pub(crate) struct LateBatch { ++ /// A copy back to record at its start: its mark and the sequence it covers. ++ pub(crate) flush: Option<(u64, u64)>, ++ /// The areas of records it leaves whose command buffers the scheduler planned in part ++ /// (counted, [`Fallback::PlanLeftPlanned`]). ++ pub(crate) left_planned: Vec<[u32; 4]>, ++} ++ + /// A surface's trap coherence, beside its [`Lazy`]. + pub(crate) struct Coherent { + state: Mutex, +@@ -1526,10 +1544,11 @@ impl Coherent { + /// its completion). What runs held for an event is not ordered before the batch by Metal + /// either, and lands at its completion as ever; open snapshots ask a wait, which the + /// scheduler's lock cannot take: both left, counted. +- pub(crate) fn before_late_batch(&self, id: u64) -> Option<(u64, u64)> { ++ pub(crate) fn before_late_batch(&self, id: u64) -> LateBatch { ++ let mut late = LateBatch::default(); + let mut state = self.lock(); + if state.closed || !state.tracked { +- return None; ++ return late; + } + if !state.done.is_empty() { + count(Fallback::PlanSync); +@@ -1538,11 +1557,11 @@ impl Coherent { + let flushed_to = state.flushed_to; + let others = |record: &&Record| record.id != id && record.seq > flushed_to; + if !state.running.iter().any(|record| others(&record)) { +- return None; ++ return late; + } + if !state.snapshots.is_empty() { + count(Fallback::PlanLeft); +- return None; ++ return late; + } + let held = (state.running.iter()) + .filter(others) +@@ -1551,17 +1570,32 @@ impl Coherent { + .min(); + let mark = state.seq; + let through = held.map_or(mark, |seq| seq - 1); +- if through <= flushed_to { +- count(Fallback::PlanLeft); +- return None; +- } ++ // What it leaves: records from the first held one on. One whose command buffer the ++ // scheduler planned in part (or whole, past the held one) may hold writes Metal orders ++ // before the batch, which land only at its completion: the batch's own writes of the bytes ++ // must reach the image after it ([`LateBatch::left_planned`]). One with none of it planned ++ // holds none (`PlanLeft`). + if held.is_some() { +- count(Fallback::PlanLeft); ++ late.left_planned = (state.running.iter()) ++ .filter(others) ++ .filter(|record| record.seq > through) ++ .filter(|record| { ++ !super::shadow_lazy::unplanned(record.id) ++ || super::shadow_lazy::parts_planned(record.id) ++ }) ++ .map(|record| record.area) ++ .collect(); ++ if late.left_planned.is_empty() { ++ count(Fallback::PlanLeft); ++ } ++ } ++ if through > flushed_to { ++ // In flight beside any copy back of `flushing`: `owed` keeps the surface busy meanwhile. ++ count(Fallback::PlanCopyBack); ++ state.owed.push((id, mark, through)); ++ late.flush = Some((mark, through)); + } +- // In flight beside any copy back of `flushing`: `owed` keeps the surface busy meanwhile. +- count(Fallback::PlanCopyBack); +- state.owed.push((id, mark, through)); +- Some((mark, through)) ++ late + } + + /// LOCAL (Atlas): the copy back [`before_late_batch`](Self::before_late_batch) asked of the +@@ -1946,7 +1980,7 @@ fn process_line(label: &str) -> String { + \"late_compares\":{},\"late_compare_ns\":{},\"late_compare_ns_max\":{},\ + \"late_beside\":{},\"late_beside_unplanned\":{},\"late_beside_held\":{},\ + \"late_beside_uncommitted\":{},\"plan_syncs\":{},\"plan_copies_back\":{},\ +- \"plan_left\":{},\"flushes_held_back\":{},\ ++ \"plan_left\":{},\"plan_left_planned\":{},\"flushes_held_back\":{},\ + \"signals_behind_arm\":{},\"met_untracked\":{},\"flush_nothing_left\":{},\ + \"host_reads\":{},\"host_reads_synced\":{},\"host_writes\":{},\"host_writes_opened\":{},\ + \"demotions\":{},\"closed_by_overlap\":{},\"closed_by_marks\":{},\"retired_dead\":{},\ +@@ -1964,6 +1998,7 @@ fn process_line(label: &str) -> String { + c.plan_syncs, + c.plan_copies_back, + c.plan_left, ++ c.plan_left_planned, + c.flushes_held_back, + c.signals_behind_arm, + c.met_untracked, +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index fce5d182..47c4576b 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -264,6 +264,12 @@ pub(crate) fn ordered(id: u64, part: usize) -> Order { + } + } + ++/// LOCAL (Atlas): whether the scheduler planned any part of the command buffer `id`, which has a ++/// batch it has not planned ([`planned_through`]). ++pub(crate) fn parts_planned(id: u64) -> bool { ++ (plans().lock().unwrap().through.get(&id)).is_some_and(|&through| through > 0) ++} ++ + /// Whether the command buffer `id` was committed and has a batch the scheduler has not planned. + pub(crate) fn unplanned(id: u64) -> bool { + plans().lock().unwrap().unplanned.contains(&id) +@@ -671,21 +677,20 @@ pub(crate) fn late_copy_in( + list + } + +-/// LOCAL (Atlas): the copies back a batch of the command buffer `id` planned after its commit (it +-/// waited for an event), its parts' uses `uses`, takes first, of the trapping surfaces whose bytes +-/// they meet ([`Coherent::before_late_batch`]): its commit's copy back ([`met_by`]) covered what +-/// ran then, and what completed or went since lies before the batch too. Under the scheduler's +-/// lock; nothing waits. With each, what it holds until it completes. ++/// LOCAL (Atlas): the copies a batch of the command buffer `id` planned after its commit (it waited ++/// for an event), its parts' uses `uses`, takes of the trapping surfaces whose bytes they meet ++/// ([`Coherent::before_late_batch`]): its commit's copy back ([`met_by`]) covered what ran then, ++/// and what completed or went since lies before the batch too: the copies back, with what each ++/// holds until it completes. Under the scheduler's lock; nothing waits. + pub(crate) fn before_late_batch<'a>( + id: u64, + uses: impl Fn() -> Vec<&'a Use>, + ) -> Vec<(ShadowCopy, Vec)> { ++ let mut before = Vec::new(); + if !ahead() { +- return Vec::new(); ++ return before; + } + let uses = uses(); +- +- let mut copies = Vec::new(); + for lazy in surfaces() { + let Some(coherent) = lazy.coherent() else { + continue; +@@ -693,23 +698,30 @@ pub(crate) fn before_late_batch<'a>( + if lazy.own_thread() || !uses.iter().any(|used| lazy.meets(used)) { + continue; + } +- let Some((mark, _)) = coherent.before_late_batch(id) else { +- continue; +- }; ++ let late = coherent.before_late_batch(id); + let left = (lazy.pending.lock().unwrap().as_ref()) +- .and_then(|left| Some((left.settled?, left.copies.clone(), left.holds.clone()))); +- let Some((area, held, holds)) = left else { +- count(Fallback::FlushNothingLeft); +- coherent.late_batch_failed(id, mark); +- continue; +- }; +- let copy: ShadowCopy = copy_back_of(held, area); +- coherent.flush_recorded(copy.texels_back() * 4); +- FLUSHES.fetch_add(1, Ordering::Relaxed); +- FLUSHED_TEXELS.fetch_add(copy.texels_back(), Ordering::Relaxed); +- copies.push((copy, holds)); ++ .map(|left| (left.settled, left.copies.clone(), left.holds.clone())); ++ if let Some((mark, _)) = late.flush { ++ match &left { ++ Some((Some(area), held, holds)) => { ++ let copy: ShadowCopy = copy_back_of(held.clone(), *area); ++ coherent.flush_recorded(copy.texels_back() * 4); ++ FLUSHES.fetch_add(1, Ordering::Relaxed); ++ FLUSHED_TEXELS.fetch_add(copy.texels_back(), Ordering::Relaxed); ++ before.push((copy, holds.clone())); ++ } ++ _ => { ++ count(Fallback::FlushNothingLeft); ++ coherent.late_batch_failed(id, mark); ++ } ++ } ++ } ++ // A record it leaves holds planned writes that land only at its completion: counted. ++ if !late.left_planned.is_empty() { ++ count(Fallback::PlanLeftPlanned); ++ } + } +- copies ++ before + } + + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index f2faeb89..d9e35825 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -2113,3 +2113,103 @@ fn surface_coherent_a_late_batch_through_another_import_copies_back_what_runs_fi + assert!(!report.contains("\"sync_errors\":1"), "{report}"); + }); + } ++ ++/// A reader's record left out of a late batch's copy back while its command buffer is planned in ++/// part (`shadow_coherent::Fallback::PlanLeftPlanned`): it draws over a row, signals the writer, ++/// then waits for a gate before more work. The writer, released by that signal, writes the row ++/// through another import: Metal orders the write after the draw, and the reader's later work ++/// (another row) after the gate leaves it. ++#[test] ++fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let before = shadow_coherent::fallback_counts(); ++ let first = case.device.new_shared_event().unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let reader = case.queue.command_buffer().unwrap(); ++ case.pass(&reader, GREEN, scissor(0, 100, 16, 1)); ++ reader.encode_signal_event((&first).into(), 1).unwrap(); ++ reader.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ case.pass(&reader, GREEN, scissor(0, 200, 16, 1)); ++ reader.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ writer.encode_wait_for_event((&first).into(), 1).unwrap(); ++ let bytes: Vec = (0..16).flat_map(|_| [1u8, 2, 3, 4]).collect(); ++ let source = staged(case.device, &bytes); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 100 * ROW, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ writer.wait_until_completed_for(SOON).unwrap(); ++ gate.set_signaled_value(1); ++ reader.wait_until_completed_for(SOON).unwrap(); ++ let after = shadow_coherent::fallback_counts(); ++ eprintln!( ++ "PARTLY sync/back/left/left_planned={}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", ++ after.plan_syncs - before.plan_syncs, ++ after.plan_copies_back - before.plan_copies_back, ++ after.plan_left - before.plan_left, ++ after.plan_left_planned - before.plan_left_planned, ++ after.late_copies_in_busy - before.late_copies_in_busy, ++ shadow_coherent::report("partly").contains("\"overwrite_risk\":0"), ++ case.read(9, 100), ++ case.read(9, 200), ++ ); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 4], "the write, after the reader's draw"); ++ assert_eq!(case.read(9, 200), GREEN_TEXEL, "the reader's work after the gate"); ++ }); ++} ++ ++/// As above, the reader's work after the gate leaving the surface: no copy in after its wait ++/// takes the write, and the record lands its draw over it at its completion. A known gap, counted ++/// `plan_left_planned`; closing it needs each record's area by part. ++#[test] ++#[ignore = "known gap: plan_left_planned (ADOPT.md, known limits)"] ++fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part_elsewhere_after() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let before = shadow_coherent::fallback_counts(); ++ let first = case.device.new_shared_event().unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let reader = case.queue.command_buffer().unwrap(); ++ case.pass(&reader, GREEN, scissor(0, 100, 16, 1)); ++ reader.encode_signal_event((&first).into(), 1).unwrap(); ++ reader.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ // Its work after the gate leaves the surface: no copy in after the wait. ++ let (from, to) = (staged(case.device, &[7u8; 64]), staged(case.device, &[0u8; 64])); ++ let mut blit = reader.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&from, 0, &to, 0, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ reader.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ writer.encode_wait_for_event((&first).into(), 1).unwrap(); ++ let bytes: Vec = (0..16).flat_map(|_| [1u8, 2, 3, 4]).collect(); ++ let source = staged(case.device, &bytes); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 100 * ROW, 64).unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ writer.wait_until_completed_for(SOON).unwrap(); ++ gate.set_signaled_value(1); ++ reader.wait_until_completed_for(SOON).unwrap(); ++ let after = shadow_coherent::fallback_counts(); ++ eprintln!( ++ "PARTLY-ELSEWHERE sync/back/left/left_planned={}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", ++ after.plan_syncs - before.plan_syncs, ++ after.plan_copies_back - before.plan_copies_back, ++ after.plan_left - before.plan_left, ++ after.plan_left_planned - before.plan_left_planned, ++ after.late_copies_in_busy - before.late_copies_in_busy, ++ shadow_coherent::report("partly").contains("\"overwrite_risk\":0"), ++ case.read(9, 100), ++ case.read(9, 200), ++ ); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 4], "the write, after the reader's draw"); ++ }); ++} diff --git a/patches/metal2vulkan/0032-coherent-a-late-batch-keeps-held-records-and-owes-th.patch b/patches/metal2vulkan/0032-coherent-a-late-batch-keeps-held-records-and-owes-th.patch new file mode 100644 index 0000000..974e69b --- /dev/null +++ b/patches/metal2vulkan/0032-coherent-a-late-batch-keeps-held-records-and-owes-th.patch @@ -0,0 +1,507 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 23:27:36 +0000 +Subject: [PATCH 32/34] coherent: a late batch keeps held records and owes them + the copy in of its writes + +A late batch's copy back now covers every running record. Records held for +an event (a batch not planned) are kept (State::kept): they still land at +their completion, and their command buffer owes its next batch a copy in of +what other imports wrote since the late batch's first kept write +(shadow_lazy::owed_copies_in, through path 4's beside_since/taking_beside), +so the record's landing carries the write instead of covering it. + +Closes the gap of lane ..._planned_in_part_elsewhere_after (reader writes, +signals, waits, then works off the surface; the writer's write was lost). +plan_left now counts kept records (exact); plan_left_planned only writes the +surface did not keep or open snapshots (gate 0); plan_owed_copies_in counts +the copies in. Demotion does not close it: it lands running records at +completion too. + +Lanes: 10 late/held lanes pass; surface+event 170 passed, 1 known failure +(ycbcr_textures), as at base; runtime unit tests 73 passed. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/submit.rs | 46 ++++++ + .../src/shadow_coherent.rs | 153 ++++++++++++------ + .../metal2vulkan-runtime/src/shadow_lazy.rs | 71 +++++++- + .../src/runtime_lane/surface_coherent.rs | 12 +- + 4 files changed, 226 insertions(+), 56 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index fe00c8ce..3b0831a6 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -464,6 +464,7 @@ impl CommandBufferShared { + if waited { + self.record_copies_back_before(&mut state, range.clone())?; + } ++ self.record_owed_copies_in(&mut state, range.clone())?; + self.record_late_copies_in(&mut state, range.clone())?; + // No work of an encoder that recorded nothing for a command (a buffer it named was + // detached), nor of one whose submission failed rebound or detached, goes to the queue. +@@ -1276,6 +1277,51 @@ impl CommandBufferShared { + Ok(()) + } + ++ /// LOCAL (Atlas): the copies in a late batch's copy back left this command buffer owing ++ /// (`shadow_lazy::owed_copies_in`), at the start of batch `range`'s first part, as ++ /// [`record_late_copies_in`](Self::record_late_copies_in) records its own. ++ fn record_owed_copies_in( ++ &self, ++ state: &mut State, ++ range: std::ops::Range, ++ ) -> Result<(), CommandBufferError> { ++ if range.is_empty() || !crate::shadow_coherent::owes_copies_in() { ++ return Ok(()); ++ } ++ let Some(id) = state.shadow_carry.as_ref().map(|carry| carry.id()) else { ++ return Ok(()); ++ }; ++ let copies = crate::shadow_lazy::owed_copies_in(id); ++ if copies.is_empty() { ++ return Ok(()); ++ } ++ let device = &self.device; ++ let mut uses = Vec::new(); ++ for copy in &copies { ++ uses.extend(copy.uses_with(None)); ++ } ++ let mut aliased = Vec::new(); ++ if device.shared().aliases.augment_into(&uses, &mut aliased) { ++ uses = aliased; ++ } ++ let raw = self.record_raw(state, |_, buffer| { ++ let vk = device.shared().vk(); ++ // SAFETY: as `record_late_copies_in`'s. ++ unsafe { ++ for copy in &copies { ++ copy.record(vk, buffer, None); ++ } ++ Barrier::everything().record(vk, buffer); ++ } ++ Ok(()) ++ })?; ++ let first = &mut state.recorded[range.start]; ++ first.raws.insert(0, raw); ++ first.summary.join(Summary::outside_encoders(uses)); ++ state.retained.push(Box::new(copies)); ++ Ok(()) ++ } ++ + fn record_late_copies_in( + &self, + state: &mut State, +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 55a1cfd7..91af0e1f 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -224,17 +224,27 @@ pub(crate) enum Fallback { + PlanSync, + /// Such a batch: a copy back of what runs, planned, recorded at the batch's start. + PlanCopyBack, +- /// Such a batch: what runs is held for an event, or the pages hold open snapshots: nothing +- /// orders it before the batch, or no copy back may run without a wait, and it lands at its +- /// completion as ever. ++ /// Such a batch: a record held for an event kept for its completion (its next batch copies in ++ /// what the batch wrote, [`PlanOwedCopyIn`](Self::PlanOwedCopyIn)), or open snapshots met by a ++ /// batch writing nothing of the bytes: exact. + PlanLeft, +- /// As `PlanLeft`, a record left whose command buffer the scheduler planned in part: its planned +- /// writes may lie before the batch and land only at its completion, over what the batch wrote +- /// of the bytes where they meet (lane `..._planned_in_part_elsewhere_after`). Not closed yet. ++ /// Such a batch writing the bytes through other imports in a way the surface did not keep, with ++ /// a record held, or with open snapshots (no copy back may meet those without a wait): a ++ /// record's landing may cover the write. The gate is 0. + PlanLeftPlanned, ++ /// A held record's next batch copied in other imports' writes since a late batch that kept it. ++ PlanOwedCopyIn, + } + +-const FALLBACKS: usize = Fallback::PlanLeftPlanned as usize + 1; ++const FALLBACKS: usize = Fallback::PlanOwedCopyIn as usize + 1; ++ ++/// LOCAL (Atlas): entries of [`State::owed_in`] over every surface, one atomic read for a batch. ++static OWED_IN: AtomicU64 = AtomicU64::new(0); ++ ++/// LOCAL (Atlas): whether any command buffer owes a copy in at its next batch. ++pub(crate) fn owes_copies_in() -> bool { ++ OWED_IN.load(Ordering::Relaxed) > 0 ++} + static FALLBACK: [AtomicU64; FALLBACKS] = [const { AtomicU64::new(0) }; FALLBACKS]; + /// The late copies in's compares of snapshot pages, every surface's ([`Coherent::taking_late`]): + /// how many, and their ns in all and at most. Under the scheduler's lock, so the VM's smoke check +@@ -316,10 +326,12 @@ pub struct CoherentFallbackCounts { + pub plan_syncs: u64, + /// Such batches that carried a copy back of what ran, planned. + pub plan_copies_back: u64, +- /// Such batches that met what ran, held, or a surface with open snapshots: left. ++ /// Such batches that kept a held record (owed a copy in), or met open snapshots writing nothing. + pub plan_left: u64, +- /// Of what such batches left, records whose command buffer was planned in part. ++ /// Such batches whose writes may be covered by a record's landing (gate 0). + pub plan_left_planned: u64, ++ /// Held records' next batches that copied in what a late batch wrote. ++ pub plan_owed_copies_in: u64, + } + + /// The counts of [`CoherentFallbackCounts`] so far. +@@ -356,6 +368,7 @@ pub fn fallback_counts() -> CoherentFallbackCounts { + plan_copies_back: read(Fallback::PlanCopyBack), + plan_left: read(Fallback::PlanLeft), + plan_left_planned: read(Fallback::PlanLeftPlanned), ++ plan_owed_copies_in: read(Fallback::PlanOwedCopyIn), + } + } + +@@ -611,6 +624,13 @@ struct State { + /// LOCAL (Atlas): copies back recorded in a batch ([`Coherent::before_late_batch`]): the + /// command buffer, the mark and the sequence covered, for its completion. + owed: Vec<(u64, u64, u64)>, ++ /// LOCAL (Atlas): records of a sequence a copy back covered (`flushed_to` past them) that it ++ /// left, held for an event: they land at their completion still ([`Coherent::before_late_batch`]). ++ kept: Vec, ++ /// LOCAL (Atlas): the command buffers of those records, each with the [`Lazy`] write count ++ /// after which what other command buffers wrote through other imports goes into the image at ++ /// the start of its next batch (`shadow_lazy::owed_copies_in`). ++ owed_in: Vec<(u64, u64)>, + } + + /// LOCAL (Atlas): what a batch planned after its commit takes of a trapping surface +@@ -619,9 +639,18 @@ struct State { + pub(crate) struct LateBatch { + /// A copy back to record at its start: its mark and the sequence it covers. + pub(crate) flush: Option<(u64, u64)>, +- /// The areas of records it leaves whose command buffers the scheduler planned in part +- /// (counted, [`Fallback::PlanLeftPlanned`]). +- pub(crate) left_planned: Vec<[u32; 4]>, ++} ++ ++/// LOCAL (Atlas): what a batch planned after its commit wrote of a trapping surface's bytes ++/// through other imports ([`Coherent::before_late_batch`]). ++#[derive(Clone, Copy, PartialEq, Eq)] ++pub(crate) enum Writes { ++ /// Nothing. ++ None, ++ /// Writes kept by the surface (`shadow_lazy`'s `Beside`), the first making this count. ++ Since(u64), ++ /// Writes it did not keep. ++ Unknown, + } + + /// A surface's trap coherence, beside its [`Lazy`]. +@@ -857,7 +886,7 @@ impl Coherent { + }), + }; + if let Some(record) = found { +- if record.seq > state.flushed_to { ++ if record.seq > state.flushed_to || state.kept.contains(&record.seq) { + self.finish_record(&mut state, record); + } + self.moved.notify_all(); +@@ -931,6 +960,7 @@ impl Coherent { + }; + let start = Instant::now(); + let done = std::mem::take(&mut state.done); ++ state.kept.retain(|seq| !done.iter().any(|record| record.seq == *seq)); + let mut list = Vec::new(); + for record in &done { + let area = record.area; +@@ -1380,7 +1410,12 @@ impl Coherent { + self.sync_done(&mut state, SyncBy::CopyIn); + let busy = (state.open.iter().chain(&state.running)) + .filter(|record| left || record.id != id) +- .filter(|record| record.seq == 0 || record.seq > state.flushed_to || record.id == id) ++ .filter(|record| { ++ record.seq == 0 ++ || record.seq > state.flushed_to ++ || state.kept.contains(&record.seq) ++ || record.id == id ++ }) + .any(|record| areas.iter().any(|area| meet(record.area, *area))); + if busy { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); +@@ -1544,7 +1579,7 @@ impl Coherent { + /// its completion). What runs held for an event is not ordered before the batch by Metal + /// either, and lands at its completion as ever; open snapshots ask a wait, which the + /// scheduler's lock cannot take: both left, counted. +- pub(crate) fn before_late_batch(&self, id: u64) -> LateBatch { ++ pub(crate) fn before_late_batch(&self, id: u64, writes: Writes) -> LateBatch { + let mut late = LateBatch::default(); + let mut state = self.lock(); + if state.closed || !state.tracked { +@@ -1555,49 +1590,70 @@ impl Coherent { + self.sync_done(&mut state, SyncBy::GpuReader); + } + let flushed_to = state.flushed_to; +- let others = |record: &&Record| record.id != id && record.seq > flushed_to; ++ let kept = state.kept.clone(); ++ let others = |record: &&Record| { ++ record.id != id && (record.seq > flushed_to || kept.contains(&record.seq)) ++ }; + if !state.running.iter().any(|record| others(&record)) { + return late; + } + if !state.snapshots.is_empty() { +- count(Fallback::PlanLeft); ++ count(match writes { ++ Writes::None => Fallback::PlanLeft, ++ _ => Fallback::PlanLeftPlanned, ++ }); + return late; + } +- let held = (state.running.iter()) ++ // The copy covers every record running, ordered after what of each the scheduler planned. ++ // One held for an event (its command buffer has a batch not planned) is kept: it lands at ++ // its completion as ever, and as that lands its whole area after the batch's writes of ++ // the bytes, its next batch first copies in what other command buffers wrote through ++ // other imports since `since` (the batch's first kept write), as path 4 does ++ // (`shadow_lazy::owed_copies_in`); not known, it is counted and left. ++ let held: Vec<(u64, u64)> = (state.running.iter()) + .filter(others) + .filter(|record| super::shadow_lazy::unplanned(record.id)) +- .map(|record| record.seq) +- .min(); +- let mark = state.seq; +- let through = held.map_or(mark, |seq| seq - 1); +- // What it leaves: records from the first held one on. One whose command buffer the +- // scheduler planned in part (or whole, past the held one) may hold writes Metal orders +- // before the batch, which land only at its completion: the batch's own writes of the bytes +- // must reach the image after it ([`LateBatch::left_planned`]). One with none of it planned +- // holds none (`PlanLeft`). +- if held.is_some() { +- late.left_planned = (state.running.iter()) +- .filter(others) +- .filter(|record| record.seq > through) +- .filter(|record| { +- !super::shadow_lazy::unplanned(record.id) +- || super::shadow_lazy::parts_planned(record.id) +- }) +- .map(|record| record.area) +- .collect(); +- if late.left_planned.is_empty() { +- count(Fallback::PlanLeft); +- } ++ .map(|record| (record.id, record.seq)) ++ .collect(); ++ if !held.is_empty() { ++ count(match writes { ++ Writes::Unknown => Fallback::PlanLeftPlanned, ++ _ => Fallback::PlanLeft, ++ }); + } +- if through > flushed_to { +- // In flight beside any copy back of `flushing`: `owed` keeps the surface busy meanwhile. +- count(Fallback::PlanCopyBack); +- state.owed.push((id, mark, through)); +- late.flush = Some((mark, through)); ++ for (record, seq) in &held { ++ if !state.kept.contains(seq) { ++ state.kept.push(*seq); ++ } ++ let Writes::Since(since) = writes else { ++ continue; ++ }; ++ match state.owed_in.iter_mut().find(|owed| owed.0 == *record) { ++ Some(owed) => owed.1 = owed.1.min(since), ++ None => { ++ state.owed_in.push((*record, since)); ++ OWED_IN.fetch_add(1, Ordering::Relaxed); ++ } ++ } + } ++ let mark = state.seq; ++ // In flight beside any copy back of `flushing`: `owed` keeps the surface busy meanwhile. ++ count(Fallback::PlanCopyBack); ++ state.owed.push((id, mark, mark)); ++ late.flush = Some((mark, mark)); + late + } + ++ /// LOCAL (Atlas): the write count after which the next batch of the command buffer `id` copies ++ /// in other imports' writes ([`State::owed_in`]), taken. ++ pub(crate) fn take_owed_in(&self, id: u64) -> Option { ++ let mut state = self.lock(); ++ let at = state.owed_in.iter().position(|owed| owed.0 == id)?; ++ let (_, since) = state.owed_in.swap_remove(at); ++ OWED_IN.fetch_sub(1, Ordering::Relaxed); ++ Some(since) ++ } ++ + /// LOCAL (Atlas): the copy back [`before_late_batch`](Self::before_late_batch) asked of the + /// batch of the command buffer `id` was not recorded after all. + pub(crate) fn late_batch_failed(&self, id: u64, mark: u64) { +@@ -1627,7 +1683,8 @@ impl Coherent { + state.flushing = 0; + } + state.flushed_to = state.flushed_to.max(through); +- let (gone, kept): (Vec<_>, Vec<_>) = state.done.iter().partition(|r| r.seq <= through); ++ let (gone, kept): (Vec<_>, Vec<_>) = (state.done.iter()) ++ .partition(|r| r.seq <= through && !state.kept.contains(&r.seq)); + state.done = kept; + if let Some(mark) = pages() { + let mut list: Vec = +@@ -1980,7 +2037,8 @@ fn process_line(label: &str) -> String { + \"late_compares\":{},\"late_compare_ns\":{},\"late_compare_ns_max\":{},\ + \"late_beside\":{},\"late_beside_unplanned\":{},\"late_beside_held\":{},\ + \"late_beside_uncommitted\":{},\"plan_syncs\":{},\"plan_copies_back\":{},\ +- \"plan_left\":{},\"plan_left_planned\":{},\"flushes_held_back\":{},\ ++ \"plan_left\":{},\"plan_left_planned\":{},\ ++ \"plan_owed_copies_in\":{},\"flushes_held_back\":{},\ + \"signals_behind_arm\":{},\"met_untracked\":{},\"flush_nothing_left\":{},\ + \"host_reads\":{},\"host_reads_synced\":{},\"host_writes\":{},\"host_writes_opened\":{},\ + \"demotions\":{},\"closed_by_overlap\":{},\"closed_by_marks\":{},\"retired_dead\":{},\ +@@ -1999,6 +2057,7 @@ fn process_line(label: &str) -> String { + c.plan_copies_back, + c.plan_left, + c.plan_left_planned, ++ c.plan_owed_copies_in, + c.flushes_held_back, + c.signals_behind_arm, + c.met_untracked, +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 47c4576b..602ef013 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -698,7 +698,7 @@ pub(crate) fn before_late_batch<'a>( + if lazy.own_thread() || !uses.iter().any(|used| lazy.meets(used)) { + continue; + } +- let late = coherent.before_late_batch(id); ++ let late = coherent.before_late_batch(id, lazy.writes_of(id, &uses)); + let left = (lazy.pending.lock().unwrap().as_ref()) + .map(|left| (left.settled, left.copies.clone(), left.holds.clone())); + if let Some((mark, _)) = late.flush { +@@ -716,12 +716,54 @@ pub(crate) fn before_late_batch<'a>( + } + } + } +- // A record it leaves holds planned writes that land only at its completion: counted. +- if !late.left_planned.is_empty() { ++ } ++ before ++} ++ ++/// LOCAL (Atlas): the copies in the batch of the command buffer `id` going now takes first, of ++/// each trapping surface a late batch's copy back kept a record of it for ++/// ([`Coherent::before_late_batch`]): what other command buffers wrote through other imports ++/// since, in parts the scheduler planned (path 4's `Lazy::beside_since`, `Coherent::taking_beside`). ++/// Its record's landing at completion then carries them. Under the scheduler's lock. ++pub(crate) fn owed_copies_in(id: u64) -> Vec { ++ let mut list = Vec::new(); ++ if !crate::shadow_coherent::owes_copies_in() { ++ return list; ++ } ++ for lazy in surfaces() { ++ let Some(coherent) = lazy.coherent() else { ++ continue; ++ }; ++ let Some(since) = coherent.take_owed_in(id) else { ++ continue; ++ }; ++ let held = (lazy.pending.lock().unwrap().as_ref()).map(|left| left.copies.clone()); ++ let Some(Ok(copies)) = held.map(|held| held.downcast::()) else { + count(Fallback::PlanLeftPlanned); ++ continue; ++ }; ++ let beside = lazy.beside_since(since, id, coherent); ++ let areas = coherent.taking_beside(id, false, beside); ++ if areas.is_empty() { ++ continue; ++ } ++ count(Fallback::PlanOwedCopyIn); ++ let mut texels = 0; ++ for area in areas { ++ let copy = ShadowCopy { ++ copies: Arc::clone(&copies), ++ step: Step::Into, ++ kept: false, ++ area: Some(area), ++ cause: Some(CopyCause::Bounded), ++ moves: None, ++ }; ++ texels += copy.texels_copied_in(); ++ list.push(copy); + } ++ lazy.copied_in(CopyCause::Bounded, texels); + } +- before ++ list + } + + /// The copies back a command of the command buffer `id` needs before it, its uses `uses`: of what +@@ -1481,6 +1523,27 @@ impl Lazy { + true + } + ++ /// LOCAL (Atlas): what the command buffer `id`, its uses `uses`, wrote of its bytes through ++ /// other imports, as [`Beside`] kept it. ++ fn writes_of(&self, id: u64, uses: &[&Use]) -> crate::shadow_coherent::Writes { ++ use crate::shadow_coherent::Writes; ++ if !uses.iter().any(|used| !used.write.is_empty() && self.meets(used)) { ++ return Writes::None; ++ } ++ let beside = self.beside.lock().unwrap(); ++ match (beside.writes.iter()).filter(|write| write.id == id).map(|write| write.count).min() { ++ // A joined entry's count is its last write's: the one before it is past its first. ++ Some(count) => { ++ let before = (beside.writes.iter()) ++ .take_while(|write| write.count < count) ++ .map(|write| write.count) ++ .last(); ++ Writes::Since(before.unwrap_or(beside.lost)) ++ } ++ None => Writes::Unknown, ++ } ++ } ++ + /// LOCAL (Atlas): the write that made `count` lies in its command buffer's parts before `end`. + fn beside_recorded(&self, count: u64, id: u64, end: usize) { + let mut beside = self.beside.lock().unwrap(); +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index d9e35825..cfa9d696 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -2148,11 +2148,12 @@ fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part() + reader.wait_until_completed_for(SOON).unwrap(); + let after = shadow_coherent::fallback_counts(); + eprintln!( +- "PARTLY sync/back/left/left_planned={}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", ++ "PARTLY sync/back/left/left_planned/owed={}/{}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", + after.plan_syncs - before.plan_syncs, + after.plan_copies_back - before.plan_copies_back, + after.plan_left - before.plan_left, + after.plan_left_planned - before.plan_left_planned, ++ after.plan_owed_copies_in - before.plan_owed_copies_in, + after.late_copies_in_busy - before.late_copies_in_busy, + shadow_coherent::report("partly").contains("\"overwrite_risk\":0"), + case.read(9, 100), +@@ -2164,10 +2165,10 @@ fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part() + } + + /// As above, the reader's work after the gate leaving the surface: no copy in after its wait +-/// takes the write, and the record lands its draw over it at its completion. A known gap, counted +-/// `plan_left_planned`; closing it needs each record's area by part. ++/// takes the write, and the record would land its draw over it at its completion; the writer's ++/// late batch keeps the record and owes its next batch the copy in ++/// (`shadow_lazy::owed_copies_in`). + #[test] +-#[ignore = "known gap: plan_left_planned (ADOPT.md, known limits)"] + fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part_elsewhere_after() { + on_surface(true, |case| { + let other_buffer = case.other_import(); +@@ -2200,11 +2201,12 @@ fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part_e + reader.wait_until_completed_for(SOON).unwrap(); + let after = shadow_coherent::fallback_counts(); + eprintln!( +- "PARTLY-ELSEWHERE sync/back/left/left_planned={}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", ++ "PARTLY-ELSEWHERE sync/back/left/left_planned/owed={}/{}/{}/{}/{} busy={} risk {} (9,100)={:?} (9,200)={:?}", + after.plan_syncs - before.plan_syncs, + after.plan_copies_back - before.plan_copies_back, + after.plan_left - before.plan_left, + after.plan_left_planned - before.plan_left_planned, ++ after.plan_owed_copies_in - before.plan_owed_copies_in, + after.late_copies_in_busy - before.late_copies_in_busy, + shadow_coherent::report("partly").contains("\"overwrite_risk\":0"), + case.read(9, 100), diff --git a/patches/metal2vulkan/0033-lanes-a-CPU-write-reaching-a-held-batch-past-a-later.patch b/patches/metal2vulkan/0033-lanes-a-CPU-write-reaching-a-held-batch-past-a-later.patch new file mode 100644 index 0000000..78d1800 --- /dev/null +++ b/patches/metal2vulkan/0033-lanes-a-CPU-write-reaching-a-held-batch-past-a-later.patch @@ -0,0 +1,53 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Wed, 4 Nov 2026 05:52:51 +0000 +Subject: [PATCH 33/34] lanes: a CPU write reaching a held batch past a later + command buffer of its queue (reproduces, ignored) + +Fails on f17fa833: no late copy in, (9,100) reads [0,0,255,0]. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/runtime_lane/surface_coherent.rs | 32 +++++++++++++++++++ + 1 file changed, 32 insertions(+) + +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index cfa9d696..ec44be05 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -2215,3 +2215,35 @@ fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part_e + assert_eq!(case.read(9, 100), [1, 2, 3, 4], "the write, after the reader's draw"); + }); + } ++ ++/// LOCAL (Atlas): found by lane B (`surface_inherit_a_held_batch_...`, inheritance off): a ++/// guest CPU write made while a held batch waits for its signal is lost where another command ++/// buffer of the same queue, over the surface, is committed after the write and before the ++/// signal. ++#[test] ++#[ignore = "fails on f17fa833: write lost, fix pending (lane A)"] ++fn surface_coherent_a_cpu_write_reaches_a_held_batch_past_a_later_command_buffer_of_its_queue() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ let late = shadow_coherent::fallback_counts().late_copies_in; ++ case.write(9, 100, [1, 2, 3, 4]); ++ let after = case.queue.command_buffer().unwrap(); ++ case.pass(&after, GREEN, scissor(0, 7, 64, 1)); ++ after.commit().unwrap(); ++ hold.set_signaled_value(1); ++ wait_for_late_copy(late); ++ for command_buffer in [&waiting, &after] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ let lates = shadow_coherent::fallback_counts().late_copies_in - late; ++ eprintln!("HELD-PAST-LATER late copies in {lates}, (9,100) {:?}", case.read(9, 100)); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the guest's write, under the held blend"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL); ++ }); ++} ++ diff --git a/patches/metal2vulkan/0034-coherent-writes-reach-held-batches-past-later-comman.patch b/patches/metal2vulkan/0034-coherent-writes-reach-held-batches-past-later-comman.patch new file mode 100644 index 0000000..ad5e489 --- /dev/null +++ b/patches/metal2vulkan/0034-coherent-writes-reach-held-batches-past-later-comman.patch @@ -0,0 +1,1910 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Thu, 5 Nov 2026 00:34:16 +0000 +Subject: [PATCH 34/34] coherent: writes reach held batches past later command + buffers; no waits on held records + +- Records carry their part end and publish per completed batch (completing_parts); + failed command buffers drop their records and leave the planned set. +- A later command buffer's take keeps the guest area and snapshots for a held batch + (held_guest/held_snapshots); the join no longer drops a missing side (joined, not bounding). + Done records merge into the pages before the stash. +- Held batches with no record are seen through late_waiters. +- Encoder takes and reader copy backs never wait on a record held for an event + (wait_settled skip_held); host readers still wait, bounded. +- A reader whose commit-time copy back was not made copies back at its planning + (copy_back_at_plan, from atlas/coh-d-lane3 f5239ecf); a plan-time copy back is left + where stashed snapshots hold CPU writes. +- An ordered take passes over a write count only when Beside kept every write (plain). +- done records stay bounded (join of records no copy back left behind). +- Exit guard: host image copy entry points and the image read inside the guard. +- Lanes: 5 new held-batch lanes; reproductions un-ignored. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/command_buffer/completion.rs | 12 +- + .../src/command_buffer/mod.rs | 18 +- + .../src/command_buffer/submit.rs | 7 +- + crates/metal2vulkan-runtime/src/device/mod.rs | 64 +-- + .../metal2vulkan-runtime/src/device/prime.rs | 94 ++++- + crates/metal2vulkan-runtime/src/exit_guard.rs | 84 ++-- + crates/metal2vulkan-runtime/src/shadow.rs | 42 +- + .../src/shadow_coherent.rs | 266 +++++++++++-- + .../metal2vulkan-runtime/src/shadow_lazy.rs | 98 +++-- + .../src/tests/exit_guard.rs | 13 +- + crates/metal2vulkan-runtime/src/texture.rs | 12 + + .../src/texture/host_copy.rs | 8 +- + .../src/runtime_lane/surface_coherent.rs | 365 +++++++++++++++++- + 13 files changed, 947 insertions(+), 136 deletions(-) + +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +index e0335c15..eae39133 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/completion.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/completion.rs +@@ -216,6 +216,8 @@ pub(crate) struct Submitted { + pub(crate) buffer: Hold, + /// It ends its command buffer: finishing it completes the command buffer. + pub(crate) last: bool, ++ /// Exact parts attributed to this submission; absent on synthetic failure completions. ++ pub(crate) parts: Option>, + /// The fence its submission signals after the work, when it ends its command buffer where + /// the driver needs one, or its queue's failures are relayed: the command buffer finishes + /// once it is signaled ([`fences`](super::fences)). +@@ -404,6 +406,14 @@ fn finish(watch: &Watch, lists: &mut FenceLists, mut finished: Vec<(usize, Submi + else { + return true; + }; ++ // Publish each completed batch before its signals are made visible to the guest. ++ if crate::shadow_lazy::ahead() { ++ for (_, batch) in &finished { ++ if let (Some(id), Some(parts)) = (batch.buffer.shadow_carry_id(), batch.parts.clone()) { ++ crate::shadow_lazy::completing_parts(id, parts); ++ } ++ } ++ } + // The holds of the batches that end no command buffer go first, their work completed: a + // detach that sees one of their signals detaches what only they held (`detach.rs`). + for (_, batch) in &mut finished { +@@ -415,7 +425,7 @@ fn finish(watch: &Watch, lists: &mut FenceLists, mut finished: Vec<(usize, Submi + // anything, their signals among them, can tell the guest the work ended + // (`shadow_coherent.rs`); the command buffer's own completion finds them armed. + if crate::shadow_lazy::ahead() { +- for (_, batch) in finished.iter().filter(|(_, batch)| batch.last) { ++ for (_, batch) in finished.iter().filter(|(_, batch)| batch.last && batch.parts.is_some()) { + if let Some(id) = batch.buffer.shadow_carry_id() { + crate::shadow_lazy::completing(id); + } +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +index 63c7c0b1..e553cecf 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/mod.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/mod.rs +@@ -2579,8 +2579,17 @@ impl CommandBuffer { + .collect(); + crate::shadow_lazy::met_by(&summaries, &bytes) + }; ++ // LOCAL (Atlas): a trapping surface whose records all ran unplanned leaves the ++ // copy back to this command buffer's planning (`ShadowCarry::plans_copy_back`). ++ let mut at_plan = false; + for surface in met { +- surface.flush(crate::shadow_coherent::SyncBy::GpuReader); ++ let flushed = surface.flush(crate::shadow_coherent::SyncBy::GpuReader); ++ at_plan |= flushed.is_none() && surface.coherent().is_some(); ++ } ++ if at_plan { ++ if let Some(carry) = self.state().shadow_carry.as_mut() { ++ carry.copy_back_at_plan(); ++ } + } + } + { +@@ -4254,6 +4263,7 @@ impl CommandBufferShared { + value: values[vk_queue], + buffer: self.hold(), + last: true, ++ parts: None, + fence, + signals: Vec::new(), + holds: Vec::new(), +@@ -4404,7 +4414,11 @@ impl CommandBufferShared { + // completed (`shadow_coherent.rs`). + if crate::shadow_lazy::ahead() { + if let Some(id) = self.shadow_carry_id() { +- crate::shadow_lazy::completing(id); ++ if result.is_ok() && lock(&self.state).failure.is_none() { ++ crate::shadow_lazy::completing(id); ++ } else { ++ crate::shadow_lazy::failed(id); ++ } + } + } + { +diff --git a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +index 3b0831a6..bfed8d1c 100644 +--- a/crates/metal2vulkan-runtime/src/command_buffer/submit.rs ++++ b/crates/metal2vulkan-runtime/src/command_buffer/submit.rs +@@ -460,7 +460,8 @@ impl CommandBufferShared { + .map_err(|error| recording_error(device, error))?; + // LOCAL (Atlas): what trapping surfaces' records left in their images since the commit's + // copy back, then what the CPU wrote of them before the signal it waited for. +- let waited = (state.shadow_carry.as_ref()).is_some_and(|carry| carry.waited_before(range.end)); ++ let waited = ++ (state.shadow_carry.as_ref()).is_some_and(|carry| carry.plans_copy_back(range.end)); + if waited { + self.record_copies_back_before(&mut state, range.clone())?; + } +@@ -1110,6 +1111,7 @@ impl CommandBufferShared { + value: at.value, + buffer: self.hold(), + last, ++ parts: Some(range.clone()), + fence, + signals: kept + .into_iter() +@@ -1226,6 +1228,9 @@ impl CommandBufferShared { + if copies.is_empty() { + return Ok(()); + } ++ if let Some(carry) = state.shadow_carry.as_mut() { ++ carry.copied_back_at_plan(); ++ } + let device = &self.device; + let mut uses = Vec::new(); + let mut planned = Vec::new(); +diff --git a/crates/metal2vulkan-runtime/src/device/mod.rs b/crates/metal2vulkan-runtime/src/device/mod.rs +index acc27e6b..4814fae2 100644 +--- a/crates/metal2vulkan-runtime/src/device/mod.rs ++++ b/crates/metal2vulkan-runtime/src/device/mod.rs +@@ -1448,6 +1448,32 @@ impl Device { + depth_bias: depth_bias_command, + sized_index_bind, + }; ++ // The host image copy functions, where the device enabled `hostImageCopy` and copies to ++ // and from `GENERAL` on the host: by their core names from Vulkan 1.4, else by the ++ // extension's, which the device then enabled. ++ // SAFETY: `physical` belongs to `instance`, created at Vulkan 1.2 or later. ++ let general = host_image_copy ++ && unsafe { crate::texture::host_copy::reaches_general(&instance.raw, physical) }; ++ let host_copies = general.then(|| { ++ // SAFETY: as above; the loader serves `vkGetDeviceProcAddr` once the word is put ++ // back. ++ unsafe { loader_word.restore() }; ++ let handle = raw.handle(); ++ ash::ext::host_image_copy::DeviceFn::load(|name| { ++ let bytes = name.to_bytes(); ++ let core = std::ffi::CString::new(bytes.strip_suffix(b"EXT").unwrap_or(bytes)) ++ .unwrap_or_default(); ++ // SAFETY: the device is live; a name it does not serve answers null. ++ let function = unsafe { ++ (instance.raw.get_device_proc_addr(handle, core.as_ptr())) ++ .or_else(|| instance.raw.get_device_proc_addr(handle, name.as_ptr())) ++ }; ++ function.map_or(std::ptr::null(), |function| { ++ function as *const std::ffi::c_void ++ }) ++ }) ++ }); ++ + // Before the exit hook is registered for this device (below): the first call of each + // entry point of the exit guard's table (`prime.rs`). + // SAFETY: the device is live, of `physical`, and enables timeline semaphores, and +@@ -1469,6 +1495,13 @@ impl Device { + &queues, + Sharing::new(&queue_families), + drawn.has(prime::Has { ++ host_image_copy: host_copies.is_some() && unsafe { ++ instance.raw.get_physical_device_image_format_properties( ++ physical, vk::Format::R8G8B8A8_UNORM, vk::ImageType::TYPE_2D, ++ vk::ImageTiling::OPTIMAL, vk::ImageUsageFlags::HOST_TRANSFER_EXT, ++ vk::ImageCreateFlags::empty(), ++ ).is_ok() ++ }, + timestamps: timestamp_bits > 0, + // A device the runtime created enables it (`enables.rs`). + host_query_reset: owned, +@@ -1490,6 +1523,7 @@ impl Device { + calibrated, + drawn, + ycbcr_formats.primer(), ++ host_copies.as_ref(), + ) + } { + scheduler.stop(); +@@ -1556,31 +1590,6 @@ impl Device { + dma_buf_images, + ) + }; +- // The host image copy functions, where the device enabled `hostImageCopy` and copies to +- // and from `GENERAL` on the host: by their core names from Vulkan 1.4, else by the +- // extension's, which the device then enabled. +- // SAFETY: `physical` belongs to `instance`, created at Vulkan 1.2 or later. +- let general = host_image_copy +- && unsafe { crate::texture::host_copy::reaches_general(&instance.raw, physical) }; +- let host_copies = general.then(|| { +- // SAFETY: as above; the loader serves `vkGetDeviceProcAddr` once the word is put +- // back. +- unsafe { loader_word.restore() }; +- let handle = raw.handle(); +- ash::ext::host_image_copy::DeviceFn::load(|name| { +- let bytes = name.to_bytes(); +- let core = std::ffi::CString::new(bytes.strip_suffix(b"EXT").unwrap_or(bytes)) +- .unwrap_or_default(); +- // SAFETY: the device is live; a name it does not serve answers null. +- let function = unsafe { +- (instance.raw.get_device_proc_addr(handle, core.as_ptr())) +- .or_else(|| instance.raw.get_device_proc_addr(handle, name.as_ptr())) +- }; +- function.map_or(std::ptr::null(), |function| { +- function as *const std::ffi::c_void +- }) +- }) +- }); + // The mesh draw commands, which exist where `meshShader` is enabled (it needs + // `VK_EXT_mesh_shader`). + let mesh_draws = if enabled +@@ -1691,6 +1700,11 @@ impl Device { + .flat_map(|uuid| uuid.chunks(8)) + .map(|half| u64::from_le_bytes(half.try_into().unwrap_or_default())) + .chain(functions.entry_points().iter().copied()) ++ .chain(host_copies.iter().flat_map(|f| [ ++ f.transition_image_layout_ext as usize as u64, ++ f.copy_memory_to_image_ext as usize as u64, ++ f.copy_image_to_memory_ext as usize as u64, ++ ])) + .collect(); + exit_guard::register_device(identity); + let in_process = properties.kind == DeviceKind::Cpu; +diff --git a/crates/metal2vulkan-runtime/src/device/prime.rs b/crates/metal2vulkan-runtime/src/device/prime.rs +index 4b85356b..942dc897 100644 +--- a/crates/metal2vulkan-runtime/src/device/prime.rs ++++ b/crates/metal2vulkan-runtime/src/device/prime.rs +@@ -72,7 +72,7 @@ + use crate::draw_source::SizedIndexBind; + use crate::dynamic_state::{ColorWriteMasks, ExtendedDynamicState}; + use crate::error::{vulkan, Error}; +-use crate::exit_guard::Call; ++use crate::exit_guard::{Call, CallBits}; + use crate::generated_commands::{ + Commands, ExecutionSetCreateInfo, ExecutionSetPipelineInfo, ExecutionSetToken, + LayoutCreateInfo, LayoutToken, EXECUTION_SET_PIPELINES, INDIRECT_BINDABLE, TOKEN_DISPATCH, +@@ -110,6 +110,11 @@ pub(crate) const UNPRIMED: [Call; 9] = [ + Call::GetImageSubresourceLayout, + ]; + ++/// Host copies on an image format the primer can make. ++pub(crate) const HOST_IMAGE_COPY_CALLS: [Call; 3] = [ ++ Call::TransitionImageLayout, Call::CopyMemoryToImage, Call::CopyImageToMemory, ++]; ++ + /// The table's entry points a queue family that writes no timestamps (`timestampValidBits` 0) + /// never calls, nor primes: a command buffer's GPU times (`gpu_time.rs`). + pub(crate) const TIMESTAMP_CALLS: [Call; 3] = [ +@@ -368,6 +373,7 @@ fn specialization() -> vk::SpecializationInfo<'static> { + /// calibrated timestamps, and the render encoder's commands of [`PrimedDraws`]. + #[derive(Clone, Copy)] + pub(crate) struct Has { ++ pub(crate) host_image_copy: bool, + pub(crate) timestamps: bool, + pub(crate) host_query_reset: bool, + pub(crate) device_addresses: bool, +@@ -438,6 +444,7 @@ pub(super) unsafe fn prime( + calibration: Option, + drawn: PrimedDraws, + ycbcr: Option>, ++ host_copies: Option<&ash::ext::host_image_copy::DeviceFn>, + ) -> Result<(), Error> { + let mut primer = Primer { + device, +@@ -448,7 +455,7 @@ pub(super) unsafe fn prime( + drawn, + // SAFETY: the caller's. + memory: unsafe { instance.get_physical_device_memory_properties(physical) }, +- calls: 0, ++ calls: CallBits::EMPTY, + held: Vec::new(), + }; + // SAFETY: the caller's. +@@ -462,6 +469,10 @@ pub(super) unsafe fn prime( + }) + .and_then(|()| primer.sparse(queues, has.sparse_binding)) + .and_then(|()| primer.ycbcr(ycbcr)) ++ .and_then(|()| match host_copies { ++ Some(functions) => primer.host_image_copy(instance, physical, functions), ++ None => Ok(()), ++ }) + }; + // SAFETY: the submission completed, or was never made, or the device is lost; nothing else + // uses the objects held. +@@ -494,13 +505,14 @@ pub(super) unsafe fn prime( + /// [`COLOR_WRITE_MASK_CALLS`], [`DEPTH_BIAS_CALLS`] and [`SIZED_INDEX_BIND_CALLS`] where it has + /// not their commands, and those of [`YCBCR_CONVERSION_CALLS`] where it converts no YCbCr format, + /// that `calls` (one bit each) does not hold. +-pub(crate) fn missing(calls: u128, has: Has) -> Vec { ++pub(crate) fn missing(calls: CallBits, has: Has) -> Vec { + Call::ALL + .iter() + .copied() + .filter(|&call| { +- calls & call.bit() == 0 ++ (calls & call.bit()).is_empty() + && !UNPRIMED.contains(&call) ++ && (has.host_image_copy || !HOST_IMAGE_COPY_CALLS.contains(&call)) + && (has.timestamps || !TIMESTAMP_CALLS.contains(&call)) + && ((has.timestamps && has.host_query_reset) + || !HOST_QUERY_RESET_CALLS.contains(&call)) +@@ -553,12 +565,81 @@ struct Primer<'a> { + drawn: PrimedDraws, + memory: vk::PhysicalDeviceMemoryProperties, + /// The table's entry points called, one bit each. +- calls: u128, ++ calls: CallBits, + /// What was made, released last first. + held: Vec, + } + + impl Primer<'_> { ++ /// Prime host image copy where an RGBA host-transfer image can be made. If the format is ++ /// unsupported, the calls stay unprimed and their first section registers the hook again. ++ unsafe fn host_image_copy( ++ &mut self, ++ instance: &ash::Instance, ++ physical: vk::PhysicalDevice, ++ functions: &ash::ext::host_image_copy::DeviceFn, ++ ) -> Result<(), Error> { ++ let format = vk::Format::R8G8B8A8_UNORM; ++ let usage = vk::ImageUsageFlags::HOST_TRANSFER_EXT; ++ if unsafe { ++ instance.get_physical_device_image_format_properties( ++ physical, format, vk::ImageType::TYPE_2D, vk::ImageTiling::OPTIMAL, ++ usage, vk::ImageCreateFlags::empty(), ++ ) ++ }.is_err() { ++ return Ok(()); ++ } ++ let info = vk::ImageCreateInfo::default() ++ .image_type(vk::ImageType::TYPE_2D) ++ .format(format) ++ .extent(vk::Extent3D { width: 2, height: 1, depth: 1 }) ++ .mip_levels(1).array_layers(1) ++ .samples(vk::SampleCountFlags::TYPE_1) ++ .tiling(vk::ImageTiling::OPTIMAL).usage(usage) ++ .initial_layout(vk::ImageLayout::UNDEFINED); ++ let device = self.device; ++ let image = unsafe { device.create_image(&info, None) }.map_err(vulkan)?; ++ let requirements = unsafe { device.get_image_memory_requirements(image) }; ++ let memory = match unsafe { self.allocate(requirements, vk::MemoryPropertyFlags::empty(), false) } { ++ Ok(memory) => memory, ++ Err(error) => { ++ unsafe { release::object(device, image) }; ++ return Err(error); ++ } ++ }; ++ self.hold(image); ++ unsafe { device.bind_image_memory(image, memory, 0) }.map_err(vulkan)?; ++ let subresource = vk::ImageSubresourceLayers::default() ++ .aspect_mask(vk::ImageAspectFlags::COLOR).layer_count(1); ++ let transition = vk::HostImageLayoutTransitionInfoEXT::default() ++ .image(image).old_layout(vk::ImageLayout::UNDEFINED) ++ .new_layout(vk::ImageLayout::GENERAL) ++ .subresource_range(vk::ImageSubresourceRange::default() ++ .aspect_mask(vk::ImageAspectFlags::COLOR).level_count(1).layer_count(1)); ++ unsafe { (functions.transition_image_layout_ext)(device.handle(), 1, &transition) } ++ .result().map_err(vulkan)?; ++ self.made(Call::TransitionImageLayout); ++ let input = [0u32; 2]; ++ let region = vk::MemoryToImageCopyEXT::default() ++ .host_pointer(input.as_ptr().cast()).image_subresource(subresource) ++ .image_extent(vk::Extent3D { width: 2, height: 1, depth: 1 }); ++ let write = vk::CopyMemoryToImageInfoEXT::default().dst_image(image) ++ .dst_image_layout(vk::ImageLayout::GENERAL).regions(std::slice::from_ref(®ion)); ++ unsafe { (functions.copy_memory_to_image_ext)(device.handle(), &write) } ++ .result().map_err(vulkan)?; ++ self.made(Call::CopyMemoryToImage); ++ let mut output = [0u32; 2]; ++ let region = vk::ImageToMemoryCopyEXT::default() ++ .host_pointer(output.as_mut_ptr().cast()).image_subresource(subresource) ++ .image_extent(vk::Extent3D { width: 2, height: 1, depth: 1 }); ++ let read = vk::CopyImageToMemoryInfoEXT::default().src_image(image) ++ .src_image_layout(vk::ImageLayout::GENERAL).regions(std::slice::from_ref(®ion)); ++ unsafe { (functions.copy_image_to_memory_ext)(device.handle(), &read) } ++ .result().map_err(vulkan)?; ++ self.made(Call::CopyImageToMemory); ++ Ok(()) ++ } ++ + /// `call` was made. + fn made(&mut self, call: Call) { + self.calls |= call.bit(); +@@ -962,6 +1043,9 @@ impl Primer<'_> { + .map_memory(memory, 0, vk::WHOLE_SIZE, vk::MemoryMapFlags::empty()) + .map_err(vulkan)?; + self.made(Call::MapMemory); ++ let range = vk::MappedMemoryRange::default().memory(memory).offset(0).size(vk::WHOLE_SIZE); ++ device.invalidate_mapped_memory_ranges(&[range]).map_err(vulkan)?; ++ self.made(Call::InvalidateMappedMemoryRanges); + if device_addresses { + let info = vk::BufferDeviceAddressInfo::default().buffer(buffer); + device.get_buffer_device_address(&info); +diff --git a/crates/metal2vulkan-runtime/src/exit_guard.rs b/crates/metal2vulkan-runtime/src/exit_guard.rs +index e0efb3be..fb7093ba 100644 +--- a/crates/metal2vulkan-runtime/src/exit_guard.rs ++++ b/crates/metal2vulkan-runtime/src/exit_guard.rs +@@ -267,33 +267,48 @@ static SECTIONS: AtomicU64 = AtomicU64::new(0); + /// The entry points ([`Call`]) called in sections that ended since the hook was last registered + /// at a device after a load of a shared object ([`register_device`], they were the process's), one + /// bit each. +-static CALLED: CalledWord = CalledWord([AtomicU64::new(0), AtomicU64::new(0)]); +- +-/// One bit per entry point of the table ([`Call::bit`]), as two atomic halves: the table +-/// outgrew 64 bits with the render host step's entry points. +-struct CalledWord([AtomicU64; 2]); ++static CALLED: CalledWord = CalledWord([const { AtomicU64::new(0) }; 4]); ++ ++/// A bit per entry point. Kept in two halves so the table may grow beyond 128 calls. ++#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)] ++pub(crate) struct CallBits(pub(crate) [u128; 2]); ++impl CallBits { ++ pub(crate) const EMPTY: Self = Self([0, 0]); ++ pub(crate) fn is_empty(self) -> bool { self == Self::EMPTY } ++ #[cfg(test)] ++ pub(crate) fn count_ones(self) -> u32 { self.0[0].count_ones() + self.0[1].count_ones() } ++} ++impl std::ops::BitOr for CallBits { ++ type Output = Self; ++ fn bitor(self, rhs: Self) -> Self { Self([self.0[0] | rhs.0[0], self.0[1] | rhs.0[1]]) } ++} ++impl std::ops::BitOrAssign for CallBits { ++ fn bitor_assign(&mut self, rhs: Self) { *self = *self | rhs; } ++} ++impl std::ops::BitAnd for CallBits { ++ type Output = Self; ++ fn bitand(self, rhs: Self) -> Self { Self([self.0[0] & rhs.0[0], self.0[1] & rhs.0[1]]) } ++} ++impl std::ops::Not for CallBits { ++ type Output = Self; ++ fn not(self) -> Self { Self([!self.0[0], !self.0[1]]) } ++} + ++struct CalledWord([AtomicU64; 4]); + impl CalledWord { +- /// Set `calls`; the bits set before. +- fn fetch_or(&self, calls: u128) -> u128 { +- let low = self.0[0].fetch_or(calls as u64, Ordering::Relaxed); +- let high = self.0[1].fetch_or((calls >> 64) as u64, Ordering::Relaxed); +- u128::from(low) | (u128::from(high) << 64) ++ fn fetch_or(&self, calls: CallBits) -> CallBits { ++ let halves = [calls.0[0] as u64, (calls.0[0] >> 64) as u64, ++ calls.0[1] as u64, (calls.0[1] >> 64) as u64]; ++ let old = std::array::from_fn::<_, 4, _>(|i| self.0[i].fetch_or(halves[i], Ordering::Relaxed)); ++ CallBits([u128::from(old[0]) | (u128::from(old[1]) << 64), ++ u128::from(old[2]) | (u128::from(old[3]) << 64)]) + } +- +- /// Clear every bit: a load of a shared object since (a driver or layer loaded again makes +- /// its statics again at each entry point's first call). +- fn clear(&self) { +- self.0[0].store(0, Ordering::Relaxed); +- self.0[1].store(0, Ordering::Relaxed); +- } +- +- /// The bits set. ++ fn clear(&self) { for word in &self.0 { word.store(0, Ordering::Relaxed); } } + #[cfg(test)] +- fn load(&self) -> u128 { +- let low = self.0[0].load(Ordering::Relaxed); +- let high = self.0[1].load(Ordering::Relaxed); +- u128::from(low) | (u128::from(high) << 64) ++ fn load(&self) -> CallBits { ++ let old = std::array::from_fn::<_, 4, _>(|i| self.0[i].load(Ordering::Relaxed)); ++ CallBits([u128::from(old[0]) | (u128::from(old[1]) << 64), ++ u128::from(old[2]) | (u128::from(old[3]) << 64)]) + } + } + +@@ -513,6 +528,10 @@ calls! { + CmdDrawMeshTasksIndirectCount => c"vkCmdDrawMeshTasksIndirectCountEXT", + CmdExecuteGeneratedCommands => c"vkCmdExecuteGeneratedCommandsEXT", + CmdPreprocessGeneratedCommands => c"vkCmdPreprocessGeneratedCommandsEXT", ++ InvalidateMappedMemoryRanges => c"vkInvalidateMappedMemoryRanges", ++ TransitionImageLayout => c"vkTransitionImageLayoutEXT", ++ CopyMemoryToImage => c"vkCopyMemoryToImageEXT", ++ CopyImageToMemory => c"vkCopyImageToMemoryEXT", + } + + /// The entry points a host step may call (`CommandBufferShared::host_step`): what a dispatch +@@ -610,11 +629,12 @@ pub(crate) const HOST_STEP: &[Call] = &[ + ]; + + impl Call { +- /// Its bit. The table fills 113 of the word's 128 (widened from 64): when it next outgrows the +- /// word, widen the word (here, [`CALLED`], `PENDING` and the priming's calls); do not add a +- /// second table. +- pub(crate) fn bit(self) -> u128 { +- 1 << self as u32 ++ /// The unique bit for this entry point. ++ pub(crate) fn bit(self) -> CallBits { ++ let n = self as usize; ++ let mut bits = CallBits::EMPTY; ++ bits.0[n / 128] = 1_u128 << (n % 128); ++ bits + } + } + +@@ -622,7 +642,7 @@ thread_local! { + /// The sections in progress on this thread (of every guard). + static OWN: Cell = const { Cell::new(0) }; + /// The entry points ([`Call`]) this thread called since its last section ended. +- static PENDING: Cell = const { Cell::new(0) }; ++ static PENDING: Cell = const { Cell::new(CallBits::EMPTY) }; + /// This thread is one of the runtime's own ([`spawn`]). + static RUNTIME: Cell = const { Cell::new(false) }; + /// The program's code this thread is running ([`program`]). +@@ -793,10 +813,10 @@ impl Drop for Busy<'_> { + // After the section's Vulkan calls, which may have registered a destructor. + let calls = PENDING.with(|pending| { + let calls = pending.get(); +- pending.set(0); ++ pending.set(CallBits::EMPTY); + calls + }); +- let first = calls & !CALLED.fetch_or(calls) != 0; ++ let first = !(calls & !CALLED.fetch_or(calls)).is_empty(); + let ended = SECTIONS.fetch_add(1, Ordering::Relaxed) + 1; + if (first || ended.is_power_of_two()) && !self.guard.is_closed() { + register_again(); +@@ -1523,5 +1543,5 @@ pub(crate) fn sections() -> u64 { + /// Whether a section that ended called `call`. + #[cfg(test)] + pub(crate) fn was_called(call: Call) -> bool { +- CALLED.load() & call.bit() != 0 ++ !(CALLED.load() & call.bit()).is_empty() + } +diff --git a/crates/metal2vulkan-runtime/src/shadow.rs b/crates/metal2vulkan-runtime/src/shadow.rs +index 73e5c02e..ca2a7c3e 100644 +--- a/crates/metal2vulkan-runtime/src/shadow.rs ++++ b/crates/metal2vulkan-runtime/src/shadow.rs +@@ -2255,6 +2255,7 @@ impl ShadowSync { + left: self.carry.left, + late: std::mem::take(&mut self.carry.late), + noted: std::mem::take(&mut self.carry.noted), ++ copy_back_at_plan: self.carry.copy_back_at_plan, + }; + carry.entries.append(&mut self.carry.entries); + for entry in &self.entries { +@@ -2336,6 +2337,11 @@ pub(super) struct ShadowCarry { + /// LOCAL (Atlas): its writes through other imports of trapping surfaces whose parts are not + /// all recorded yet ([`parts_recorded`](Self::parts_recorded)). + noted: super::shadow_lazy::Noted, ++ /// LOCAL (Atlas): its commit's copy back of a trapping surface it reads through another ++ /// import (`shadow_lazy::met_by`) was not made, every record running then not planned yet ++ /// (`shadow_coherent::Coherent::before_flush`): its first batch that meets the bytes copies ++ /// back as it is planned, as a late batch does ([`plans_copy_back`](Self::plans_copy_back)). ++ copy_back_at_plan: bool, + } + + /// LOCAL (Atlas): the trapping surfaces (`shadow_coherent.rs`) whose standing an encoder took +@@ -2360,7 +2366,7 @@ pub(super) struct LateCopies { + waited_from: Option, + /// Each surface noted, with the part of the wait it was noted after, whether an earlier + /// encoder of the command buffer had left writes ahead in its image then, and the count of +- /// writes of its bytes its standing was taken at (`shadow_lazy::Lazy::taken_written`). ++ /// writes of its bytes its standing was taken at (`shadow_lazy::Lazy::take`). + shadows: Vec<(usize, bool, u64, Arc)>, + } + +@@ -2374,6 +2380,7 @@ impl Default for ShadowCarry { + left: false, + late: LateCopies::default(), + noted: Vec::new(), ++ copy_back_at_plan: false, + } + } + } +@@ -2420,6 +2427,7 @@ impl ShadowCarry { + /// LOCAL (Atlas): its encoders so far recorded all their parts before `end` + /// (`command_buffer::State::recorded`): their writes through other imports lie before it. + pub(super) fn parts_recorded(&mut self, end: usize) { ++ super::shadow_lazy::coherent_parts_recorded(self.id, end); + if !self.noted.is_empty() { + super::shadow_lazy::parts_recorded(&mut self.noted, self.id, end); + } +@@ -2431,6 +2439,29 @@ impl ShadowCarry { + self.entries.clear(); + } + ++ /// LOCAL (Atlas): its commit's copy back was left to the scheduler's planning ++ /// ([`ShadowCarry::copy_back_at_plan`]). ++ pub(super) fn copy_back_at_plan(&mut self) { ++ self.copy_back_at_plan = true; ++ } ++ ++ /// LOCAL (Atlas): whether its next batch that meets a trapping surface's bytes copies back ++ /// what runs as it is planned (`shadow_lazy::before_late_batch`): its work waits for a shared ++ /// event before the batch's end, or its commit left the copy back to it. A commit's copy back ++ /// is planned when the scheduler takes it, after every command buffer committed before it, ++ /// so a record whose command buffer the scheduler had not taken in yet is left out though ++ /// nothing holds it; as this batch is planned, under the scheduler's lock, each such record ++ /// is planned or held for an event, which it keeps (lane ++ /// `surface_coherent_an_import_read_past_a_held_batch_sees_a_planned_write`). ++ pub(super) fn plans_copy_back(&self, end: usize) -> bool { ++ self.copy_back_at_plan || self.waited_before(end) ++ } ++ ++ /// LOCAL (Atlas): a batch of it recorded the copy back its commit left to it. ++ pub(super) fn copied_back_at_plan(&mut self) { ++ self.copy_back_at_plan = false; ++ } ++ + /// LOCAL (Atlas): whether work before part `end` waits for a shared event: a batch ending there + /// is planned after the wait, past its commit (`shadow_lazy::before_late_batch`). + pub(super) fn waited_before(&self, end: usize) -> bool { +@@ -2508,22 +2539,25 @@ impl ShadowSync { + // earlier encoder left is taken after the wait too (`Lazy::take`): its copy in is + // the batch's alone, which takes the texels the CPU changed where any write the + // pages lack meets them (`shadow_coherent::Coherent::taking_late`). +- if let (Some(_), Some(lazy), Some(since)) = (&lazily, &copies.lazy, self.carry.late.since) { ++ let ordered_take = lazily.as_ref().is_some_and(|(_, _, deferred, _)| *deferred); ++ if let (Some((_, _, _, written)), Some(lazy)) = (&lazily, &copies.lazy) { ++ let since = self.carry.late.since.unwrap_or(0); + let late = &mut self.carry.late.shadows; + if lazy.whole_image() ++ && (self.carry.late.since.is_some() || ordered_take) + && !(late.iter()).any(|(from, _, _, known)| { + *from == since && known.shadow_key == copies.shadow_key + }) + { + let left = lazy.left_open_by(self.carry.id); +- late.push((since, left, lazy.taken_written(), Arc::clone(copies))); ++ late.push((since, left, *written, Arc::clone(copies))); + } + } + let only_read = lazily.is_some() && (copies.lazy.as_ref()) + .is_some_and(|lazy| lazy.whole_image()); + let (standing, held, stale) = match (&carried, lazily) { + (Some(carried), _) => (Standing::Current, carried.held, carried.stale), +- (None, Some((held, stale))) => (Standing::Current, held, stale), ++ (None, Some((held, stale, _, _))) => (Standing::Current, held, stale), + (None, None) => (Standing::Unread, None, None), + }; + self.entries.push(Entry { +diff --git a/crates/metal2vulkan-runtime/src/shadow_coherent.rs b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +index 91af0e1f..7cbcbee4 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_coherent.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_coherent.rs +@@ -87,7 +87,7 @@ impl Rehearsal { + std::thread::sleep(running); + self.0.completing(id); + } +- if let Some(stale) = take.then(|| self.0.taking(false)).flatten() { ++ if let Some(stale) = take.then(|| self.0.taking(0, false)).flatten() { + self.0.copied_in(texels(stale)); + } + self.0.presented(); +@@ -101,7 +101,7 @@ const SWITCH: &str = "surface_coherent"; + const DISPLAY: &str = "surface_coherent_display"; + pub(crate) const PAGE: u64 = 4096; + const HOLD_MAX_S: u64 = 30; +-/// Records apart at most: past it the done ones join, as the image holds every texel between. ++/// Armed rectangles kept separately before the arm cache is cleared. + const RECORDS: usize = 8; + const PROBE_GIVEN_UP: Duration = Duration::from_millis(1); + const DEFAULT_WAIT_MS: u64 = 50; +@@ -476,6 +476,15 @@ impl Layout { + } + } + ++/// The rectangle bounding both, where either may be none (`bounding` wants both). ++fn joined(a: Option<[u32; 4]>, b: Option<[u32; 4]>) -> Option<[u32; 4]> { ++ match (a, b) { ++ (Some(_), Some(_)) => bounding(a, b), ++ (a, None) => a, ++ (None, b) => b, ++ } ++} ++ + fn meet(a: [u32; 4], b: [u32; 4]) -> bool { + a[0] < b[2] && b[0] < a[2] && a[1] < b[3] && b[1] < a[3] + } +@@ -491,10 +500,16 @@ fn texels(area: [u32; 4]) -> u64 { + /// One write left in the image. + #[derive(Clone, Copy, Debug)] + struct Record { +- /// The command buffer's identity (`shadow::ShadowCarry::id`), 0 once several joined. ++ /// The command buffer's identity (`shadow::ShadowCarry::id`). + id: u64, ++ /// Experimental: stable identity of this encoder's write group, unique within the surface. ++ /// Tickets must use this identity rather than the legacy commit-order sequence. ++ #[allow(dead_code)] ++ write_id: u64, + /// Its place among records committed: what a GPU copy back moved it by. + seq: u64, ++ /// Encoder parts through which this write was recorded; MAX until the encoder ends. ++ part_end: usize, + area: [u32; 4], + } + +@@ -596,6 +611,8 @@ struct State { + /// A GPU copy back in flight, up to this sequence: 0 for none. + flushing: u64, + seq: u64, ++ /// Last identity allocated; never reused while this surface exists. ++ next_write_id: u64, + /// Rectangles whose pages are armed: nothing to do for what one holds. + armed: Vec<[u32; 4]>, + /// A probe armed every held page: the next fault holds them again. +@@ -607,6 +624,16 @@ struct State { + guest: Option<[u32; 4]>, + /// Each open page, as it was before the guest's write. + snapshots: BTreeMap>, ++ /// LOCAL (Atlas): a copy of `guest`/`snapshots` kept for a held batch that is pending ++ /// (an entry in `owed`), so its `taking_late` still finds them after a later command ++ /// buffer's encoder took the image's standing (`Coherent::taking`). ++ held_guest: Option<[u32; 4]>, ++ /// LOCAL (Atlas): command buffers whose encoder took this surface after a wait for a shared ++ /// event (a late note, `shadow::LateCopies`), until their late copy in is made: a held ++ /// batch may leave no record (it only reads, or its defer was refused), and a later ++ /// command buffer's `taking` must still keep the guest area for it. ++ late_waiters: Vec, ++ held_snapshots: BTreeMap>, + /// The pages of `snapshots` are open: a write reaches them with no trap. + opened: bool, + frozen: Vec, +@@ -784,12 +811,17 @@ impl Coherent { + } + let whole = self.layout.whole(); + let area = area.unwrap_or(whole); +- match state.open.iter_mut().find(|record| record.id == id) { ++ match state.open.iter_mut().find(|record| record.id == id && record.part_end == usize::MAX) { + Some(record) => { + let joined = bounding(Some(record.area), Some(area)); + record.area = joined.unwrap_or(whole); + } +- None => state.open.push(Record { id, seq: 0, area }), ++ None => { ++ state.next_write_id = state.next_write_id.checked_add(1) ++ .expect("coherent write identity exhausted"); ++ let write_id = state.next_write_id; ++ state.open.push(Record { id, write_id, seq: 0, part_end: usize::MAX, area }); ++ } + } + self.counts.defers.fetch_add(1, Ordering::Relaxed); + let bytes = texels(area) * self.layout.texel; +@@ -852,8 +884,8 @@ impl Coherent { + /// The commit of `id` returned: its record may go back by a copy ordered after it. + pub(crate) fn committed(&self, id: u64) { + let mut state = self.lock(); +- if let Some(at) = state.open.iter().position(|record| record.id == id) { +- let mut record = state.open.swap_remove(at); ++ while let Some(at) = state.open.iter().position(|record| record.id == id) { ++ let mut record = state.open.remove(at); + state.seq += 1; + record.seq = state.seq; + state.running.push(record); +@@ -861,9 +893,60 @@ impl Coherent { + self.settle_pending(&state); + } + ++ /// Attach the completed encoder's parts to the records it left during encoding. ++ pub(crate) fn parts_recorded(&self, id: u64, end: usize) { ++ let mut state = self.lock(); ++ for record in &mut state.open { ++ if record.id == id && record.part_end == usize::MAX { ++ record.part_end = end; ++ } ++ } ++ } ++ ++ /// Publish only writes from submitted parts whose batch has completed. ++ pub(crate) fn completing_parts(&self, id: u64, parts: std::ops::Range) { ++ let mut state = self.lock(); ++ let belongs = |record: &Record| record.id == id ++ && record.part_end > parts.start && record.part_end <= parts.end; ++ while let Some(at) = state.running.iter().position(&belongs) { ++ let record = state.running.remove(at); ++ if record.seq > state.flushed_to || state.kept.contains(&record.seq) { ++ self.finish_record(&mut state, record); ++ } ++ self.moved.notify_all(); ++ } ++ // Completion can beat the post-commit bookkeeping. ++ while let Some(at) = state.open.iter().position(&belongs) { ++ let mut record = state.open.remove(at); ++ state.seq += 1; ++ record.seq = state.seq; ++ self.finish_record(&mut state, record); ++ self.moved.notify_all(); ++ } ++ self.settle_pending(&state); ++ } ++ ++ /// Drop uncompleted records after a failed command buffer; never publish future writes. ++ pub(crate) fn failed(&self, id: u64) { ++ let mut state = self.lock(); ++ state.late_waiters.retain(|&waiter| waiter != id); ++ let lost: Vec = state.running.iter().filter(|record| record.id == id) ++ .map(|record| record.seq).collect(); ++ state.open.retain(|record| record.id != id); ++ state.running.retain(|record| record.id != id); ++ state.kept.retain(|seq| !lost.contains(seq)); ++ state.owed.retain(|owed| owed.0 != id); ++ let owed_before = state.owed_in.len(); ++ state.owed_in.retain(|owed| owed.0 != id); ++ OWED_IN.fetch_sub((owed_before - state.owed_in.len()) as u64, Ordering::Relaxed); ++ self.moved.notify_all(); ++ self.settle_pending(&state); ++ } ++ + /// The command buffer `id` is done, before anything tells the guest: its pages trap from here. + pub(crate) fn completing(&self, id: u64) { + let mut state = self.lock(); ++ state.late_waiters.retain(|&waiter| waiter != id); + // LOCAL (Atlas): a copy back its batch carried landed with it. + if state.owed.iter().any(|owed| owed.0 == id) { + let (mine, kept): (Vec<_>, Vec<_>) = state.owed.iter().partition(|owed| owed.0 == id); +@@ -875,7 +958,7 @@ impl Coherent { + } + state = self.lock(); + } +- let found = match state.running.iter().position(|record| record.id == id) { ++ while let Some(found) = match state.running.iter().position(|record| record.id == id) { + Some(at) => Some(state.running.swap_remove(at)), + // Its commit has not returned yet. + None => (state.open.iter().position(|record| record.id == id)).map(|at| { +@@ -884,8 +967,8 @@ impl Coherent { + record.seq = state.seq; + record + }), +- }; +- if let Some(record) = found { ++ } { ++ let record = found; + if record.seq > state.flushed_to || state.kept.contains(&record.seq) { + self.finish_record(&mut state, record); + } +@@ -898,16 +981,27 @@ impl Coherent { + if state.closed || !state.tracked { + return; + } +- if state.done.len() >= RECORDS { +- let mut joined = record; +- for other in state.done.drain(..) { +- joined.area = bounding(Some(joined.area), Some(other.area)).unwrap_or(joined.area); +- joined.seq = joined.seq.max(other.seq); +- joined.id = 0; ++ // Experimental: retain each identity until its write is actually settled. Bounding ++ // rectangles here discarded identities needed for exact copy-back accounting. ++ state.done.push(record); ++ if state.done.len() > RECORDS { ++ // Join the records no copy back left behind: the join's seq is the latest, so a ++ // copy back drops it only once it covered all of them (`flushed`). ++ let kept = std::mem::take(&mut state.kept); ++ let (keep, join): (Vec, Vec) = ++ std::mem::take(&mut state.done).into_iter().partition(|r| kept.contains(&r.seq)); ++ state.kept = kept; ++ state.done = keep; ++ if let Some(first) = join.first().copied() { ++ let joined = join.iter().skip(1).fold(first, |mut joined, other| { ++ joined.area = bounding(Some(joined.area), Some(other.area)).unwrap_or(joined.area); ++ joined.seq = joined.seq.max(other.seq); ++ joined.part_end = joined.part_end.max(other.part_end); ++ joined.id = 0; ++ joined ++ }); ++ state.done.push(joined); + } +- state.done.push(joined); +- } else { +- state.done.push(record); + } + let Some(pages_mark) = pages() else { + return; +@@ -961,9 +1055,21 @@ impl Coherent { + let start = Instant::now(); + let done = std::mem::take(&mut state.done); + state.kept.retain(|seq| !done.iter().any(|record| record.seq == *seq)); +- let mut list = Vec::new(); ++ // Collapse only the copy operation, after taking the complete set of identities. A ++ // pending ticket still needs individual records until this point. ++ let mut areas = Vec::new(); + for record in &done { +- let area = record.area; ++ if areas.len() >= RECORDS { ++ let joined = areas.drain(..).fold(Some(record.area), |area, other| { ++ bounding(area, Some(other)) ++ }); ++ areas.push(joined.unwrap_or(record.area)); ++ } else { ++ areas.push(record.area); ++ } ++ } ++ let mut list = Vec::new(); ++ for area in areas { + let copying = Instant::now(); + let moved = match state.snapshots.is_empty() { + true => self.detile(area), +@@ -1095,10 +1201,17 @@ impl Coherent { + &'a self, + mut state: MutexGuard<'a, State>, + area: Option<[u32; 4]>, ++ skip_held: bool, + ) -> (MutexGuard<'a, State>, bool) { + let until = Instant::now() + Duration::from_millis(knob("surface_coherent_wait_ms", DEFAULT_WAIT_MS)); + loop { +- let overlaps = |area| state.running.iter().any(|record| meet(record.area, area)); ++ // With `skip_held`, a record held for an event is not waited for: on the command ++ // thread (an encoder's take, a reader's copy back) its signal comes from the guest ++ // later, so a wait only stalls. A host reader on another thread waits (the signal may ++ // be on its way). Callers that copy over a held record's area count the risk. ++ let overlaps = |area| (state.running.iter()).any(|record| { ++ meet(record.area, area) && !(skip_held && Self::held_for_event(&state, record.id)) ++ }); + let busy = state.flushing != 0 || !state.owed.is_empty() || area.is_some_and(overlaps); + if !busy { + return (state, true); +@@ -1154,7 +1267,7 @@ impl Coherent { + fn wrote(&self, state: MutexGuard<'_, State>, mark: Option>) { + // A running write needs no wait: the merge at its completion keeps this one. A GPU copy + // back already on its way cannot merge. +- let (settled, clear) = self.wait_settled(state, None); ++ let (settled, clear) = self.wait_settled(state, None, false); + let mut state = settled; + self.sync_done(&mut state, SyncBy::Trap); + if !clear { +@@ -1197,7 +1310,7 @@ impl Coherent { + let mut state = state; + self.sync_done(&mut state, SyncBy::HostRead); + if !state.running.is_empty() { +- let (settled, _) = self.wait_settled(state, Some(self.layout.whole())); ++ let (settled, _) = self.wait_settled(state, Some(self.layout.whole()), false); + state = settled; + self.sync_done(&mut state, SyncBy::HostRead); + } +@@ -1225,7 +1338,7 @@ impl Coherent { + } + self.sync_done(&mut state, SyncBy::Unmap); + if !state.running.is_empty() { +- let (settled, _) = self.wait_settled(state, Some(self.layout.whole())); ++ let (settled, _) = self.wait_settled(state, Some(self.layout.whole()), false); + state = settled; + self.sync_done(&mut state, SyncBy::Unmap); + } +@@ -1248,10 +1361,38 @@ impl Coherent { + true + } + ++ pub(crate) fn needs_ordered_take(&self, id: u64) -> bool { ++ let state = self.lock(); ++ Self::other_held(&state, id) ++ } ++ ++ /// Another command buffer than `id` is held for an event with work on this surface: a ++ /// running record, or a late note still to be made. ++ fn other_held(state: &State, id: u64) -> bool { ++ (state.running.iter().map(|record| record.id).chain(state.late_waiters.iter().copied())) ++ .any(|other| other != id && super::shadow_lazy::unplanned(other)) ++ } ++ ++ /// The command buffer `id` waits for a shared event before work on this surface, its batch ++ /// not planned: it completes only after the guest signals. (`unplanned` alone also holds a ++ /// command buffer just committed that the scheduler has not reached yet.) ++ fn held_for_event(state: &State, id: u64) -> bool { ++ state.late_waiters.contains(&id) && super::shadow_lazy::unplanned(id) ++ } ++ ++ /// LOCAL (Atlas): an encoder of the command buffer `id` took this surface after a wait for ++ /// a shared event: its batch makes a late copy in ([`taking_late`](Self::taking_late)). ++ pub(crate) fn late_noted(&self, id: u64) { ++ let mut state = self.lock(); ++ if !state.late_waiters.contains(&id) { ++ state.late_waiters.push(id); ++ } ++ } ++ + /// An encoder is about to use the image: where the guest wrote is stale in it, and comes back + /// for this command's copy in. With no standing (`whole`) the copy in reads every texel. The + /// pages hold everything done first, and are held before the copy in can read them. +- pub(crate) fn taking(&self, whole: bool) -> Option<[u32; 4]> { ++ pub(crate) fn taking(&self, id: u64, whole: bool) -> Option<[u32; 4]> { + let mut state = self.lock(); + if !state.tracked { + return None; +@@ -1268,13 +1409,50 @@ impl Coherent { + // write's texels must reach the pages first. Over no CPU write, a refused standing's + // copy back (`shadow_lazy::met_by`) does that without a wait. + if !state.snapshots.is_empty() { +- let (settled, clear) = self.wait_settled(state, Some(area)); ++ let (settled, clear) = self.wait_settled(state, Some(area), true); + state = settled; +- if !clear { ++ let held_over = (state.running.iter()).any(|record| { ++ record.id != id && meet(record.area, area) ++ && Self::held_for_event(&state, record.id) ++ }); ++ if !clear || held_over { + self.counts.overwrite_risk.fetch_add(1, Ordering::Relaxed); + } + } +- self.take_guest(&mut state); ++ // LOCAL (Atlas): if a held batch is still pending for this surface (an entry in ++ // `owed`, or a copy back in flight), keep a copy of the guest area and snapshots for ++ // its `taking_late`, which runs later, after this encoder took the standing. Without ++ // this the held batch finds `guest == None` and copies nothing in, and a CPU write ++ // that lands before its signal is lost from the image. ++ // A held batch is a command buffer that committed and still runs: its record is in ++ // `running` (another one's, since this encoder is not committed yet). Its late note ++ // needs the guest area and the snapshots this encoder is about to take. ++ let held_pending = !state.owed.is_empty() ++ || state.flushing != 0 ++ || state.running.iter().any(|record| record.id != id) ++ || state.late_waiters.iter().any(|&other| other != id); ++ if held_pending { ++ // Move, not clone: `wrote` already copied every page into `snapshots`, and a ++ // surface can be many megabytes. ++ // Done records reach the pages first, merged against the snapshots, so a CPU ++ // write under a done record's rectangle survives (`take_guest` does the same). ++ self.sync_done(&mut state, SyncBy::CopyIn); ++ let list: Vec = state.snapshots.keys().copied().collect(); ++ state.held_guest = joined(state.held_guest, state.guest); ++ for (page, snapshot) in std::mem::take(&mut state.snapshots) { ++ state.held_snapshots.entry(page).or_insert(snapshot); ++ } ++ if let Some(mark) = pages() { ++ let came = mark.hold(&list, true); ++ state.guest = None; ++ self.note_came(&mut state, came); ++ } ++ state.opened = false; ++ } else { ++ state.held_guest = None; ++ state.held_snapshots.clear(); ++ self.take_guest(&mut state); ++ } + if whole { + state.guest = None; + } +@@ -1314,10 +1492,22 @@ impl Coherent { + /// writes ahead before the wait (`left`): the rest of its writes run after the copy. + pub(crate) fn taking_late(&self, id: u64, left: bool, cut: bool) -> Vec<[u32; 4]> { + let mut state = self.lock(); ++ state.late_waiters.retain(|&waiter| waiter != id); + if !state.tracked || state.closed { + return Vec::new(); + } ++ // LOCAL (Atlas): restore a stashed guest area/snapshots a later command buffer's ++ // `taking` kept for this held batch. ++ if state.held_guest.is_some() { ++ state.guest = joined(state.guest, state.held_guest.take()); ++ // The oldest snapshot is the baseline for every write before this batch. ++ for (page, snapshot) in std::mem::take(&mut state.held_snapshots) { ++ state.snapshots.insert(page, snapshot); ++ } ++ } + let Some(area) = state.guest else { ++ state.held_guest = None; ++ state.held_snapshots.clear(); + return Vec::new(); + }; + let ahead = |state: &State, rects: &[[u32; 4]]| { +@@ -1362,6 +1552,8 @@ impl Coherent { + self.counts.late_copies.fetch_add(1, Ordering::Relaxed); + count(Fallback::LateCopyIn); + } ++ state.held_guest = None; ++ state.held_snapshots.clear(); + rects + } + +@@ -1547,9 +1739,16 @@ impl Coherent { + if state.running.is_empty() { + return None; + } ++ // Nothing a copy back could cover: every running record is held for an event, or its ++ // command buffer is not taken in by the scheduler yet. A reader's commit then leaves the ++ // copy back to its own batch's planning (`shadow::ShadowCarry::plans_copy_back`). ++ if state.running.iter().all(|record| super::shadow_lazy::unplanned(record.id)) { ++ count(Fallback::FlushHeldBack); ++ return None; ++ } + if !state.snapshots.is_empty() { + let area = state.guest.unwrap_or(self.layout.whole()); +- let (settled, clear) = self.wait_settled(state, Some(area)); ++ let (settled, clear) = self.wait_settled(state, Some(area), true); + state = settled; + self.sync_done(&mut state, by); + if state.running.is_empty() { +@@ -1597,7 +1796,9 @@ impl Coherent { + if !state.running.iter().any(|record| others(&record)) { + return late; + } +- if !state.snapshots.is_empty() { ++ // Open snapshots, or ones kept for a held batch (`held_snapshots`): the pages hold CPU ++ // writes the image lacks, which a copy back would put the image's older texels over. ++ if !state.snapshots.is_empty() || !state.held_snapshots.is_empty() || state.held_guest.is_some() { + count(match writes { + Writes::None => Fallback::PlanLeft, + _ => Fallback::PlanLeftPlanned, +@@ -1740,6 +1941,9 @@ impl Coherent { + } + state.done.clear(); + state.snapshots.clear(); ++ state.held_snapshots.clear(); ++ state.held_guest = None; ++ state.late_waiters.clear(); + state.frozen.clear(); + self.moved.notify_all(); + if self.counts.defers.load(Ordering::Relaxed) + self.counts.traps.load(Ordering::Relaxed) > 0 { +diff --git a/crates/metal2vulkan-runtime/src/shadow_lazy.rs b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +index 602ef013..f0cce7be 100644 +--- a/crates/metal2vulkan-runtime/src/shadow_lazy.rs ++++ b/crates/metal2vulkan-runtime/src/shadow_lazy.rs +@@ -326,6 +326,7 @@ pub(crate) fn remapped(bytes: Range) { + if let Some(coherent) = lazy.coherent() { + coherent.abandon(); + } ++ lazy.plain.fetch_add(1, Ordering::Relaxed); + lazy.written.fetch_add(1, Ordering::Relaxed); + } + } +@@ -347,6 +348,7 @@ pub(crate) fn cpu_wrote(bytes: impl Fn() -> Option>) { + }; + for lazy in surfaces() { + if lazy.over(&bytes) { ++ lazy.plain.fetch_add(1, Ordering::Relaxed); + lazy.written.fetch_add(1, Ordering::Relaxed); + } + } +@@ -360,6 +362,7 @@ pub(crate) fn guest_wrote(lazy: &Lazy) { + }; + for other in surfaces() { + if !std::ptr::eq(Arc::as_ptr(&other), lazy) && other.over(pages) { ++ other.plain.fetch_add(1, Ordering::Relaxed); + other.written.fetch_add(1, Ordering::Relaxed); + } + } +@@ -435,6 +438,40 @@ pub(crate) fn completing(id: u64) { + } + } + ++/// Associate the records left by the ended encoder with its last recorded part. ++pub(crate) fn coherent_parts_recorded(id: u64, end: usize) { ++ if crate::shadow_coherent::surfaces_tracked() { ++ for lazy in surfaces() { ++ if let Some(coherent) = lazy.coherent() { ++ coherent.parts_recorded(id, end); ++ } ++ } ++ } ++} ++ ++/// A batch's parts are complete; later parts of its command buffer may still be waiting. ++pub(crate) fn completing_parts(id: u64, parts: std::ops::Range) { ++ if crate::shadow_coherent::surfaces_tracked() { ++ for lazy in surfaces() { ++ if let Some(coherent) = lazy.coherent() { ++ coherent.completing_parts(id, parts.clone()); ++ } ++ } ++ } ++} ++ ++pub(crate) fn failed(id: u64) { ++ // Its batches will never be planned: it must not stay in `unplanned` (whose bound clears all). ++ planned(id); ++ if crate::shadow_coherent::surfaces_tracked() { ++ for lazy in surfaces() { ++ if let Some(coherent) = lazy.coherent() { ++ coherent.failed(id); ++ } ++ } ++ } ++} ++ + /// Whether any surface is lazy, one atomic read: while none is, a commit does nothing here. + pub(crate) fn active() -> bool { + ANY.load(Ordering::Relaxed) +@@ -475,6 +512,8 @@ pub(super) fn note_writes(uses: &[Use], id: u64, noted: &mut Noted) + let count = lazy.written.fetch_add(1, Ordering::Relaxed) + 1; + if lazy.note_beside(count, id, used) { + noted.push((Arc::downgrade(&lazy), count)); ++ } else { ++ lazy.plain.fetch_add(1, Ordering::Relaxed); + } + } + } +@@ -926,6 +965,8 @@ struct Stood { + held: Option<[u32; 4]>, + stale: Option<[u32; 4]>, + written: u64, ++ /// [`Lazy::plain`] as the encoder left it. ++ plain: u64, + epoch: u64, + } + +@@ -1031,6 +1072,7 @@ struct BesideWrite { + /// open ([`parts_recorded`]). + part: usize, + area: Option<[u32; 4]>, ++ copied: bool, + } + + const BESIDE_KEPT: usize = 256; +@@ -1039,10 +1081,11 @@ const BESIDE_KEPT: usize = 256; + pub(crate) struct Lazy { + truth: OnceLock<(ResourceKey, Option, UseRange)>, + written: AtomicU64, ++ /// LOCAL (Atlas): the writes counted in [`Lazy::written`] that left no [`Beside`] entry ++ /// (CPU writes, remaps, writes it did not keep): an ordered take may not pass over them. ++ plain: AtomicU64, + /// LOCAL (Atlas): [`Beside`], of a trapping surface. + beside: Mutex, +- /// LOCAL (Atlas): [`Lazy::written`] as the last standing taken found it. +- taken_written: AtomicU64, + stood: Mutex>, + resynced: Mutex, + /// The [`Device`] its queue comes from. +@@ -1087,8 +1130,8 @@ impl Lazy { + Arc::new(Lazy { + truth: OnceLock::new(), + written: AtomicU64::new(0), ++ plain: AtomicU64::new(0), + beside: Mutex::default(), +- taken_written: AtomicU64::new(0), + stood: Mutex::new(None), + resynced: Mutex::new(Instant::now()), + device: Arc::new(device.clone()), +@@ -1426,6 +1469,7 @@ impl Lazy { + held, + stale, + written: self.written.load(Ordering::Relaxed), ++ plain: self.plain.load(Ordering::Relaxed), + epoch: EPOCH.load(Ordering::Relaxed), + }); + } +@@ -1443,7 +1487,8 @@ impl Lazy { + } + + /// Where the image stands for an encoder that met `touched`, of a command buffer of identity +- /// `id`: `Some((held, stale))` where it needs no copy in (the module docs). ++ /// `id`: `Some((held, stale, deferred, written))` where it can reuse the standing (the ++ /// module docs). `written` belongs to this particular take, not to a shared last-take slot. + /// + /// LOCAL (Atlas): with `waited` (the command buffer waits for a shared event before this + /// encoder, `shadow::LateCopies`), a trapping surface's standing its own earlier encoder left +@@ -1456,21 +1501,26 @@ impl Lazy { + id: u64, + touched: u64, + waited: bool, +- ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { ++ ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>, bool, u64)> { + // Tracked from here: nothing its copy in reads can change unseen. + let coherent = self.coherent().filter(|coherent| coherent.tracks()); + if coherent.is_none() { + self.seal_beside(); + } ++ if let Some(coherent) = coherent.filter(|_| waited) { ++ coherent.late_noted(id); ++ } + let own = waited && coherent.is_some() && self.stood_by(id); +- let taken = self.take_standing(id, touched, coherent.is_some(), own); ++ let ordered = coherent.is_some_and(|coherent| coherent.needs_ordered_take(id)); ++ let taken = self.take_standing(id, touched, coherent.is_some(), own, ordered); ++ let deferred = taken.is_some() && ordered; + match coherent { +- Some(_) if own => taken, ++ Some(_) if own || deferred => taken.map(|(held, stale, written)| (held, stale, deferred, written)), + Some(coherent) => { +- let stale = coherent.taking(taken.is_none()); +- taken.map(|(held, was)| (held, bounding_stale(was, stale))) ++ let stale = coherent.taking(id, taken.is_none()); ++ taken.map(|(held, was, written)| (held, bounding_stale(was, stale), false, written)) + } +- None => taken, ++ None => taken.map(|(held, stale, written)| (held, stale, false, written)), + } + } + +@@ -1519,7 +1569,7 @@ impl Lazy { + beside.lost = beside.lost.max(gone.count); + } + } +- beside.writes.push_back(BesideWrite { count, id, part: usize::MAX, area }); ++ beside.writes.push_back(BesideWrite { count, id, part: usize::MAX, area, copied: false }); + true + } + +@@ -1565,12 +1615,12 @@ impl Lazy { + if self.written.load(Ordering::Relaxed) == written { + return Vec::new(); + } +- let beside = self.beside.lock().unwrap(); ++ let mut beside = self.beside.lock().unwrap(); + let mut unknown = beside.lost > written; + let (mut held, mut uncommitted) = (false, false); + let mut areas: Vec<[u32; 4]> = Vec::new(); +- for write in beside.writes.iter().copied() { +- if write.count <= written || write.id == id { ++ for write in beside.writes.iter_mut() { ++ if write.count <= written || write.id == id || write.copied { + continue; + } + match ordered(write.id, write.part) { +@@ -1589,6 +1639,7 @@ impl Lazy { + } + } + let area = write.area.unwrap_or_else(|| coherent.whole()); ++ write.copied = true; + if !areas.contains(&area) { + areas.push(area); + } +@@ -1606,10 +1657,11 @@ impl Lazy { + areas + } + +- /// LOCAL (Atlas): [`Lazy::written`] as the last standing taken found it, for its note +- /// (`shadow::LateCopies`). +- pub(super) fn taken_written(&self) -> u64 { +- self.taken_written.load(Ordering::Relaxed) ++ /// LOCAL (Atlas): every write of its bytes since `stood` was taken is kept in [`Beside`], for ++ /// a held batch's late copy in to take: none is a plain one, and none fell out of the list. ++ fn beside_covers(&self, stood: &Stood) -> bool { ++ stood.plain == self.plain.load(Ordering::Relaxed) ++ && self.beside.lock().unwrap().lost <= stood.written + } + + /// Whether its standing is one an encoder of the command buffer `id` left. +@@ -1623,7 +1675,8 @@ impl Lazy { + touched: u64, + tracked: bool, + own: bool, +- ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>)> { ++ ordered: bool, ++ ) -> Option<(Option<[u32; 4]>, Option<[u32; 4]>, u64)> { + report(); + let Some(stood) = *self.stood.lock().unwrap() else { + REFUSED_BY[Refused::NoStanding as usize].fetch_add(1, Ordering::Relaxed); +@@ -1631,7 +1684,9 @@ impl Lazy { + }; + let refused = if stood.touched != touched { + Some(Refused::Touched) +- } else if stood.written != self.written.load(Ordering::Relaxed) { ++ } else if stood.written != self.written.load(Ordering::Relaxed) ++ && !(ordered && self.beside_covers(&stood)) ++ { + Some(Refused::Written) + } else if stood.epoch != EPOCH.load(Ordering::Relaxed) { + Some(Refused::Epoch) +@@ -1667,9 +1722,8 @@ impl Lazy { + return None; + } + } +- self.taken_written.store(stood.written, Ordering::Relaxed); + TAKEN.fetch_add(1, Ordering::Relaxed); + self.taken.fetch_add(1, Ordering::Relaxed); +- Some((stood.held, stood.stale)) ++ Some((stood.held, stood.stale, stood.written)) + } + } +diff --git a/crates/metal2vulkan-runtime/src/tests/exit_guard.rs b/crates/metal2vulkan-runtime/src/tests/exit_guard.rs +index d1867621..b3b929be 100644 +--- a/crates/metal2vulkan-runtime/src/tests/exit_guard.rs ++++ b/crates/metal2vulkan-runtime/src/tests/exit_guard.rs +@@ -5,7 +5,7 @@ + //! it (Linux). Each test uses a guard of its own; only the fork tests use the process's, which + //! nothing here closes. + +-use crate::exit_guard::{self, hook, Busy, Call, Guard, InProcess, InProcessDevices}; ++use crate::exit_guard::{self, hook, Busy, Call, CallBits, Guard, InProcess, InProcessDevices}; + use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + use std::sync::{mpsc, Arc, Barrier, Mutex, Weak}; + use std::time::{Duration, Instant}; +@@ -639,11 +639,11 @@ fn a_forked_child_s_exit_does_not_wait_for_its_parent_s_sections() { + /// Each entry point has a bit of its own. + #[test] + fn every_entry_point_has_a_bit_of_its_own() { +- let mut bits = 0_u128; ++ let mut bits = CallBits::EMPTY; + for &call in Call::ALL { + let bit = call.bit(); + assert_eq!(bit.count_ones(), 1, "{call:?} has no bit of the word"); +- assert_eq!(bits & bit, 0, "{call:?} shares a bit"); ++ assert_eq!(bits & bit, CallBits::EMPTY, "{call:?} shares a bit"); + bits |= bit; + } + } +@@ -2090,6 +2090,7 @@ fn the_priming_check_names_what_was_not_made() { + SIZED_INDEX_BIND_CALLS, SPARSE_CALLS, TIMESTAMP_CALLS, UNPRIMED, YCBCR_CONVERSION_CALLS, + }; + let has = |timestamps, device_addresses| Has { ++ host_image_copy: true, + timestamps, + host_query_reset: true, + device_addresses, +@@ -2105,15 +2106,15 @@ fn the_priming_check_names_what_was_not_made() { + sized_index_bind: true, + ycbcr_conversion: true, + }; +- let all = Call::ALL.iter().fold(0, |bits, call| bits | call.bit()); ++ let all = Call::ALL.iter().fold(CallBits::EMPTY, |bits, call| bits | call.bit()); + assert!(missing(all, has(true, true)).is_empty()); + let made = Call::ALL + .iter() + .filter(|call| !UNPRIMED.contains(call)) +- .fold(0, |bits, call| bits | call.bit()); ++ .fold(CallBits::EMPTY, |bits, call| bits | call.bit()); + assert!(missing(made, has(true, true)).is_empty()); + assert_eq!( +- missing(0, has(true, true)).len(), ++ missing(CallBits::EMPTY, has(true, true)).len(), + Call::ALL.len() - UNPRIMED.len() + ); + for &call in Call::ALL { +diff --git a/crates/metal2vulkan-runtime/src/texture.rs b/crates/metal2vulkan-runtime/src/texture.rs +index 4e722b99..f8ac72a6 100644 +--- a/crates/metal2vulkan-runtime/src/texture.rs ++++ b/crates/metal2vulkan-runtime/src/texture.rs +@@ -2969,6 +2969,18 @@ impl Device { + // SAFETY: `raw` is a live image of this device. + let requirements = unsafe { raw_device.get_image_memory_requirements(raw) }; + let mut requirements = shared.textures.placement().place(requirements); ++ // A trapping image may be invalidated by the host while other images occupy the same ++ // memory block. Reserve whole non-coherent atoms for it so that invalidating its entire ++ // placement never includes another placement's bytes. Vulkan limits the atom size to 256. ++ if host_reads && plan.usage.contains(ImageUsageFlags::HOST_TRANSFER) { ++ let Some(size) = requirements.size.checked_next_multiple_of(256) else { ++ // SAFETY: the image was just created and is not bound or in use. ++ unsafe { release::object(raw_device, raw) }; ++ return Err(Error::OutOfMemory(MemoryKind::Device)); ++ }; ++ requirements.size = size; ++ requirements.alignment = requirements.alignment.max(256); ++ } + // Its host copies read and write through the host's mapping of its memory, which the + // host reads slowly where it does not cache it: such memory of the class where the + // device has it, else `Shared` memory's with unified memory +diff --git a/crates/metal2vulkan-runtime/src/texture/host_copy.rs b/crates/metal2vulkan-runtime/src/texture/host_copy.rs +index a5f6e1c8..e55672d5 100644 +--- a/crates/metal2vulkan-runtime/src/texture/host_copy.rs ++++ b/crates/metal2vulkan-runtime/src/texture/host_copy.rs +@@ -62,7 +62,7 @@ + use super::{Image, Owner}; + use crate::device::lock; + use crate::error::{vulkan, Error}; +-use crate::exit_guard; ++use crate::exit_guard::{self, Call}; + use ash::ext::host_image_copy::DeviceFn; + use ash::vk; + use metal2vulkan::error::RuntimeDriverDefect; +@@ -166,6 +166,7 @@ pub(super) unsafe fn transition( + layer_count: vk::REMAINING_ARRAY_LAYERS, + }); + // SAFETY: the caller's contract; the image is in `UNDEFINED`, which the move leaves. ++ if exit_guard::in_section() { exit_guard::called(Call::TransitionImageLayout); } + unsafe { (functions.transition_image_layout_ext)(device, 1, &transition) } + .result() + .map_err(vulkan) +@@ -200,6 +201,7 @@ unsafe fn write( + .dst_image_layout(vk::ImageLayout::GENERAL) + .regions(®ions); + // SAFETY: the caller's contract. ++ if exit_guard::in_section() { exit_guard::called(Call::CopyMemoryToImage); } + unsafe { (functions.copy_memory_to_image_ext)(device, &info) } + .result() + .map_err(vulkan) +@@ -307,6 +309,7 @@ impl Image { + to: *mut u8, + row: u32, + ) -> Result { ++ let Some(_busy) = exit_guard::enter() else { return Ok(false); }; + let shared = self.device.shared(); + let Some(functions) = shared.host_copies.as_ref() else { + return Ok(false); +@@ -346,6 +349,7 @@ impl Image { + .regions(std::slice::from_ref(®ion)); + // SAFETY: the caller's contract; the image is live and in `GENERAL` since it was made, and + // the lock keeps its own `VkImage` presented. ++ if exit_guard::in_section() { exit_guard::called(Call::CopyImageToMemory); } + unsafe { (functions.copy_image_to_memory_ext)(device, &info) } + .result() + .map_err(vulkan)?; +@@ -412,6 +416,7 @@ impl Image { + .offset(start) + .size((end - start).next_multiple_of(atom)); + // SAFETY: the memory is mapped whole for its life, and the range lies in the image's. ++ exit_guard::called(Call::InvalidateMappedMemoryRanges); + unsafe { shared.vk().invalidate_mapped_memory_ranges(&[range]) }.map_err(vulkan) + } + +@@ -447,6 +452,7 @@ impl Image { + .regions(std::slice::from_ref(®ion)); + // SAFETY: as `host_write`'s; the copy writes within `bytes`, which the caller sized to + // its layout. ++ if exit_guard::in_section() { exit_guard::called(Call::CopyImageToMemory); } + unsafe { (functions.copy_image_to_memory_ext)(device, &info) } + .result() + .map_err(vulkan)?; +diff --git a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +index ec44be05..693a9fd8 100644 +--- a/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs ++++ b/crates/metal2vulkan-validation/src/runtime_lane/surface_coherent.rs +@@ -19,11 +19,10 @@ use metal2vulkan::pipeline::{ColorAttachmentDescriptor, RenderPipelineDescriptor + use metal2vulkan_runtime::shadow_coherent::{self, Pages}; + use metal2vulkan_runtime::shadow_lazy; + use metal2vulkan_runtime::{ +- live, Buffer, CommandBuffer, CommandQueue, CpuCacheMode, Device, HazardTrackingMode, +- HostMemory, IoSurfacePlane, LoadAction, PipelineOptions, RenderPassDescriptor, +- RenderPipelineState, ResourceOptions, ScissorRect, SharedEvent, SharedEventListener, +- StorageMode, StoreAction, Texture, +- TextureDescriptor, ++ live, BlitOption, Buffer, CommandBuffer, CommandQueue, CpuCacheMode, Device, ++ HazardTrackingMode, HostMemory, IoSurfacePlane, LoadAction, Origin, PipelineOptions, Region, ++ RenderPassDescriptor, RenderPipelineState, ResourceOptions, ScissorRect, SharedEvent, ++ SharedEventListener, Size, StorageMode, StoreAction, Texture, TextureDescriptor, + }; + use std::collections::BTreeMap; + use std::sync::{Arc, Mutex}; +@@ -2221,7 +2220,6 @@ fn surface_coherent_a_late_batch_keeps_its_write_past_a_reader_planned_in_part_e + /// buffer of the same queue, over the surface, is committed after the write and before the + /// signal. + #[test] +-#[ignore = "fails on f17fa833: write lost, fix pending (lane A)"] + fn surface_coherent_a_cpu_write_reaches_a_held_batch_past_a_later_command_buffer_of_its_queue() { + on_surface(true, |case| { + case.draw(RED, None); +@@ -2247,3 +2245,358 @@ fn surface_coherent_a_cpu_write_reaches_a_held_batch_past_a_later_command_buffer + }); + } + ++/// A GPU write through another import invalidates the standing before a later command buffer ++/// of the held batch's queue is encoded. Both writes precede the held batch's signal. ++#[test] ++fn surface_coherent_a_cpu_write_reaches_a_held_batch_when_the_later_standing_was_invalidated() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ case.write(9, 100, [1, 2, 3, 4]); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ let source = staged(case.device, &[9u8, 8, 7, 6]); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 200 * ROW + 9 * 4, 4).unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ writer.wait_until_completed_for(SOON).unwrap(); ++ let after = case.queue.command_buffer().unwrap(); ++ case.pass(&after, GREEN, scissor(0, 7, 64, 1)); ++ after.commit().unwrap(); ++ hold.set_signaled_value(1); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ after.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "CPU write under the blend"); ++ assert_eq!(case.read(9, 200), [9, 8, 7, 0], "other import under the blend"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the later draw"); ++ }); ++} ++ ++/// Another import must reach a held blend even when no guest CPU write opened the pages. ++#[test] ++fn surface_coherent_a_gpu_write_reaches_a_held_batch_past_a_later_command_buffer() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let writer = other.command_buffer().unwrap(); ++ let source = staged(case.device, &[9u8, 8, 7, 6]); ++ let mut blit = writer.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&source, 0, &other_buffer, 200 * ROW + 9 * 4, 4).unwrap(); ++ blit.end_encoding().unwrap(); ++ writer.commit().unwrap(); ++ writer.wait_until_completed_for(SOON).unwrap(); ++ let after = case.queue.command_buffer().unwrap(); ++ case.pass(&after, GREEN, scissor(0, 7, 64, 1)); ++ after.commit().unwrap(); ++ hold.set_signaled_value(1); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ after.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(9, 200), [9, 8, 7, 0], "import write under the blend"); ++ assert_eq!(case.read(0, 100), [0, 0, 255, 0], "untouched blend"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "later draw"); ++ }); ++} ++ ++/// Two CPU writes before the signal must both reach the held blend. ++#[test] ++fn surface_coherent_two_cpu_writes_reach_a_held_batch_past_a_later_command_buffer() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ case.write(9, 100, [1, 2, 3, 4]); ++ let after = case.queue.command_buffer().unwrap(); ++ case.pass(&after, GREEN, scissor(0, 7, 64, 1)); ++ after.commit().unwrap(); ++ case.write(10, 100, [5, 6, 7, 8]); ++ hold.set_signaled_value(1); ++ waiting.wait_until_completed_for(SOON).unwrap(); ++ after.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "first CPU write under the blend"); ++ assert_eq!(case.read(10, 100), [5, 6, 7, 0], "second CPU write under the blend"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "later draw"); ++ }); ++} ++ ++/// The event separates the GPU write from a CPU write that restores its original value. ++#[test] ++fn surface_coherent_a_cpu_write_restoring_the_old_value_after_a_gpu_signal_is_visible() { ++ on_surface(true, |case| { ++ live::set("surface_coherent_hold_s", Some("0")); ++ live::set("surface_coherent_demote", Some("0")); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let old = [1, 2, 3, 4]; ++ case.write(9, 100, old); ++ let signal = case.device.new_shared_event().unwrap(); ++ let hold = case.device.new_shared_event().unwrap(); ++ let command = case.queue.command_buffer().unwrap(); ++ let deferred = shadow_lazy::ahead_counts().deferred; ++ case.pass(&command, GREEN, scissor(9, 100, 1, 1)); ++ assert!(shadow_lazy::ahead_counts().deferred > deferred, "GPU write must stay in the shadow"); ++ command.encode_signal_event((&signal).into(), 1).unwrap(); ++ blend_after(&case, &command, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ command.commit().unwrap(); ++ assert!(signal.wait_until_signaled_value(1, SOON), "GPU write reached the signal"); ++ case.write(9, 100, old); ++ hold.set_signaled_value(1); ++ command.wait_until_completed_for(SOON).unwrap(); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the later CPU write under the blend"); ++ }); ++} ++ ++/// The bytes at `at` of a `Shared` buffer no command uses now. ++fn shared_texel(buffer: &Buffer, at: usize) -> [u8; 4] { ++ // SAFETY: a `Shared` buffer of at least `at + 4` bytes, no command using it. ++ let bytes = unsafe { std::slice::from_raw_parts(buffer.contents().unwrap().as_ptr(), at + 4) }; ++ bytes[at..at + 4].try_into().unwrap() ++} ++ ++/// A held batch whose encoder left no record: a CPU write just before it, within the hold, makes ++/// the surface keep and merge instead of deferring (`Coherent::wants_merge`; else ++/// `Fallback::DeferWrote`), so the blend copies back as it goes. A ++/// second CPU write made while the batch waits, then a later command buffer of its queue over ++/// another row, committed before the signal: the blend sees both writes, under it, not lost to ++/// the eager copy back or to the later command buffer's take. ++#[test] ++fn surface_coherent_a_cpu_write_reaches_a_held_batch_whose_blend_left_no_record() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ case.write(9, 100, [1, 2, 3, 4]); ++ let defers = case.count("defers"); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ let records = case.count("defers") - defers; ++ case.write(9, 101, [5, 6, 7, 8]); ++ let after = case.queue.command_buffer().unwrap(); ++ case.pass(&after, GREEN, scissor(8, 7, 1, 1)); ++ after.commit().unwrap(); ++ hold.set_signaled_value(1); ++ for command_buffer in [&waiting, &after] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ eprintln!( ++ "NO-RECORD records {records} (9,100)={:?} (9,101)={:?} (8,7)={:?}", ++ case.read(9, 100), ++ case.read(9, 101), ++ case.read(8, 7) ++ ); ++ assert_eq!(records, 0, "the write within the hold left the blend no record"); ++ assert_eq!(case.read(9, 101), [5, 6, 7, 0], "the write while held, under the blend"); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the write before, under the blend"); ++ assert_eq!(case.read(0, 100), [0, 0, 255, 0], "the older texels, blended"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the later draw"); ++ }); ++} ++ ++/// A done record whose rectangle bounds texels it never wrote: one encoder on another queue ++/// draws at (0,100) and (20,100), the CPU wrote (9,100) before. A held command buffer of that ++/// queue keeps a record of its own running, so the next take on the first queue (a draw at ++/// (8,7)) syncs the done record while another batch is held: its copy back of the rectangle must ++/// not put the image's older bytes over the CPU write between the two draws. ++#[test] ++fn surface_coherent_a_cpu_write_survives_a_done_record_synced_by_a_later_take() { ++ on_surface(true, |case| { ++ // Defer at once, a CPU write just before or not: the draws leave records. ++ live::set("surface_coherent_hold_s", Some("0")); ++ live::set("surface_coherent_demote", Some("0")); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ case.write(9, 100, [1, 2, 3, 4]); ++ let other = case.device.new_command_queue().unwrap(); ++ let green = staged(case.device, &floats4(GREEN)); ++ let two = other.command_buffer().unwrap(); ++ let mut descriptor = RenderPassDescriptor::new(); ++ let attachment = &mut descriptor.color_attachments[0]; ++ attachment.texture = case.texture.clone(); ++ attachment.load_action = LoadAction::Load; ++ attachment.store_action = StoreAction::Store; ++ let mut encoder = two.render_command_encoder(&descriptor).unwrap(); ++ encoder.set_render_pipeline_state(&case.opaque).unwrap(); ++ encoder.set_fragment_buffer(Some(&green), 0, 0).unwrap(); ++ encoder.set_vertex_buffer(Some(&case.rect), 0, 0).unwrap(); ++ for x in [0, 20] { ++ encoder.set_scissor_rect(ScissorRect { x, y: 100, width: 1, height: 1 }).unwrap(); ++ encoder.draw_primitives(PrimitiveType::Triangle, 0, 6, 1, 0).unwrap(); ++ } ++ encoder.end_encoding().unwrap(); ++ two.commit().unwrap(); ++ two.wait_until_completed_for(SOON).unwrap(); ++ let gate = case.device.new_shared_event().unwrap(); ++ let held = other.command_buffer().unwrap(); ++ held.encode_wait_for_event((&gate).into(), 1).unwrap(); ++ case.pass(&held, GREEN, scissor(0, 200, 16, 1)); ++ held.commit().unwrap(); ++ let later = case.queue.command_buffer().unwrap(); ++ case.pass(&later, GREEN, scissor(8, 7, 1, 1)); ++ later.commit().unwrap(); ++ gate.set_signaled_value(1); ++ for command_buffer in [&held, &later] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ eprintln!( ++ "DONE-RECORD (9,100)={:?} behind={:?} (0,100)={:?} (20,100)={:?}", ++ case.read(9, 100), ++ case.behind(9, 100), ++ case.read(0, 100), ++ case.read(20, 100) ++ ); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 4], "the CPU write between the two draws"); ++ assert_eq!(case.read(0, 100), GREEN_TEXEL, "the first draw"); ++ assert_eq!(case.read(20, 100), GREEN_TEXEL, "the second draw"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the later take's draw"); ++ assert_eq!(case.read(9, 200), GREEN_TEXEL, "the held draw"); ++ }); ++} ++ ++/// A read through another import, on another queue, of a texel a draw of that queue wrote just ++/// before, while a batch of the first queue is held for an event with a blend over the surface: ++/// nothing waits for the held batch, so the read completes before the signal and sees the draw. ++/// The planned write's copy back must not wait behind, or be lost to, the held record. ++#[test] ++fn surface_coherent_an_import_read_past_a_held_batch_sees_a_planned_write() { ++ on_surface(true, |case| { ++ let other_buffer = case.other_import(); ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, None); ++ waiting.commit().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let drawn = other.command_buffer().unwrap(); ++ case.pass(&drawn, GREEN, scissor(0, 7, 1, 1)); ++ let staging = case.device.new_buffer(4, ResourceOptions::shared()).unwrap(); ++ let reader = other.command_buffer().unwrap(); ++ let mut blit = reader.blit_command_encoder().unwrap(); ++ blit.copy_from_buffer_to_buffer(&other_buffer, 7 * ROW, &staging, 0, 4).unwrap(); ++ blit.end_encoding().unwrap(); ++ drawn.commit().unwrap(); ++ reader.commit().unwrap(); ++ let before_signal = reader.wait_until_completed_for(SOON).is_ok(); ++ let read = shared_texel(&staging, 0); ++ hold.set_signaled_value(1); ++ for command_buffer in [&drawn, &reader, &waiting] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ eprintln!("IMPORT-READ before_signal={before_signal} staging={read:?}"); ++ assert!(before_signal, "the read waits for nothing held"); ++ assert_eq!(read, GREEN_TEXEL, "the read saw the draw before it on its queue"); ++ assert_eq!(case.read(0, 7), [0, 255, 0, 0], "the blend after the signal, over the draw"); ++ assert_eq!(case.read(1, 7), [0, 0, 255, 0], "the older texels, blended"); ++ }); ++} ++ ++/// A held batch's blend has a record running; then the standing is refused for a reason other ++/// than a write (`shadow_lazy`'s `EPOCH` moved: a CPU upload into an unrelated texture, the ++/// switch `surface_lazy_scoped` off), with a CPU write open. The next encoder of the queue copies ++/// in the whole surface: it must neither wait out the held record (a bounded wait of ~50 ms) ++/// nor lose the blend or the CPU write (the guest area kept for the held batch, `held_guest`). ++#[test] ++fn surface_coherent_a_refused_standing_past_a_held_batch_keeps_its_blend() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let waiting = case.queue.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ case.write(9, 100, [1, 2, 3, 4]); ++ // Refuse the standing by the epoch: an upload into a texture over no surface, unscoped. ++ let unrelated = case ++ .device ++ .new_texture(&TextureDescriptor::texture_2d(PixelFormat::BGRA8Unorm, 4, 4, false)) ++ .unwrap(); ++ live::set("surface_lazy_scoped", None); ++ unrelated.replace_region(Region::new_2d(0, 0, 1, 1), 0, 0, &[7u8; 4], 4, 0).unwrap(); ++ live::set("surface_lazy_scoped", Some("1")); ++ let green = staged(case.device, &floats4(GREEN)); ++ let rects = std::slice::from_ref(&case.rect); ++ let after = case.queue.command_buffer().unwrap(); ++ let start = Instant::now(); ++ pass(&after, case.texture(), &case.opaque, rects, &green, scissor(0, 7, 64, 1)); ++ after.commit().unwrap(); ++ let spent = start.elapsed(); ++ hold.set_signaled_value(1); ++ for command_buffer in [&waiting, &after] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ eprintln!( ++ "REFUSED-PAST-HELD spent={spent:?} (9,100)={:?} (0,100)={:?} (8,7)={:?} report {}", ++ case.read(9, 100), ++ case.read(0, 100), ++ case.read(8, 7), ++ shadow_coherent::report("refused") ++ ); ++ assert_eq!(case.read(9, 100), [1, 2, 3, 0], "the CPU write, under the blend"); ++ assert_eq!(case.read(0, 100), [0, 0, 255, 0], "the held blend"); ++ assert_eq!(case.read(8, 7), GREEN_TEXEL, "the later draw"); ++ assert!(spent < Duration::from_millis(25), "the later encoder waited {spent:?}"); ++ }); ++} ++ ++/// A take is ordered (`Coherent::needs_ordered_take`) while a batch of another queue is held with ++/// a record over the surface: the standing is taken though a write came since, and its copy in ++/// is left to the batch. A CPU write made after that held batch was committed, then a command ++/// buffer of the first queue whose encoder (a blit of the texture) reuses the standing: what it ++/// reads holds the CPU write, which Metal shows to any command buffer committed after it. ++#[test] ++fn surface_coherent_a_cpu_write_after_a_standing_is_not_hidden_by_an_ordered_take() { ++ on_surface(true, |case| { ++ case.draw(RED, None); ++ assert_eq!(case.read(0, 0), RED_TEXEL); ++ let hold = case.device.new_shared_event().unwrap(); ++ let other = case.device.new_command_queue().unwrap(); ++ let waiting = other.command_buffer().unwrap(); ++ blend_after(&case, &waiting, &hold, scissor(0, 64, WIDTH, HEIGHT - 64)); ++ waiting.commit().unwrap(); ++ case.write(9, 7, [1, 2, 3, 4]); ++ let staging = case.device.new_buffer(4, ResourceOptions::shared()).unwrap(); ++ let reader = case.queue.command_buffer().unwrap(); ++ let mut blit = reader.blit_command_encoder().unwrap(); ++ blit.copy_from_texture_to_buffer( ++ case.texture(), ++ 0, ++ 0, ++ Origin::new(9, 7, 0), ++ Size::new(1, 1, 1), ++ &staging, ++ 0, ++ 4, ++ 4, ++ BlitOption::None, ++ ) ++ .unwrap(); ++ blit.end_encoding().unwrap(); ++ reader.commit().unwrap(); ++ let before_signal = reader.wait_until_completed_for(SOON).is_ok(); ++ let read = shared_texel(&staging, 0); ++ hold.set_signaled_value(1); ++ for command_buffer in [&reader, &waiting] { ++ command_buffer.wait_until_completed_for(SOON).unwrap(); ++ } ++ eprintln!("ORDERED-TAKE before_signal={before_signal} staging={read:?}"); ++ assert!(before_signal, "the read waits for nothing held"); ++ assert_eq!(read, [1, 2, 3, 4], "the CPU write, in what the blit read"); ++ assert_eq!(case.read(9, 7), [1, 2, 3, 4], "the CPU write, in the pages"); ++ assert_eq!(case.read(0, 100), [0, 0, 255, 0], "the held blend"); ++ }); ++} diff --git a/patches/qemu/0001-the-touch-census-trap-the-guest-CPU-s-fi.patch b/patches/qemu/0001-the-touch-census-trap-the-guest-CPU-s-fi.patch new file mode 100644 index 0000000..97ff86d --- /dev/null +++ b/patches/qemu/0001-the-touch-census-trap-the-guest-CPU-s-fi.patch @@ -0,0 +1,1670 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 21:48:54 +0000 +Subject: [PATCH 1/9] the touch census: trap the guest CPU's + first touch of large IOSurfaces + +Off unless REIMS_VGPU_CENSUS is set. Memslots move to a KVM-only view of guest RAM; arms poison +through a user-mode-only userfaultfd; classifies read or write by write-protecting a trapped page +and rerunning its instruction. Observation only: guest pages stay the truth. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 1283 +++++++++++++++++++++++++++++++++++ + accel/kvm/kvm-all.c | 109 ++- + accel/kvm/meson.build | 1 + + hw/display/apple-gfx.c | 67 ++ + include/system/kvm_census.h | 91 +++ + 5 files changed, 1543 insertions(+), 8 deletions(-) + create mode 100644 accel/kvm/census.c + create mode 100644 include/system/kvm_census.h + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +new file mode 100644 +index 0000000000..1d77d3521a +--- /dev/null ++++ b/accel/kvm/census.c +@@ -0,0 +1,1283 @@ ++/* ++ * LOCAL (Atlas): the touch census. How often does the guest CPU touch large IOSurfaces? ++ * ++ * Observation only: rendering stays in place and the guest's pages stay the truth, so nothing can ++ * go stale. With REIMS_VGPU_CENSUS unset this file makes no mapping, no userfaultfd and no thread. ++ * ++ * Memslots move to a view of guest RAM, the "KVM-only view": a second shared mapping of the same ++ * memfd, which only KVM ever uses. QEMU's devices, NVMe, virtio and every GPU import stay on the ++ * main mapping. No user-mode code here or in QEMU reads through the view: its address lives in ++ * this file and in the memslot, and QEMU's address spaces never see it. So a SIGBUS from a marker ++ * can only come from KVM's fault path inside KVM_RUN, on a vCPU thread. KVM's emulator and other ++ * kernel accesses never raise one: a failed copy becomes KVM_EXIT_MMIO at the RAM's gpa, which ++ * QEMU completes through the main mapping, correctly, or an -EFAULT (below). ++ * ++ * A page's cycle, on userfaultfd opened with UFFD_USER_MODE_ONLY, which vm.unprivileged_userfaultfd ++ * = 0 allows and which needs neither /dev/userfaultfd nor a capability: ++ * ++ * armed UFFDIO_POISON. Any guest access: KVM's hwpoison path, SIGBUS with the exact address. ++ * The vCPU logs the touch, drops the marker and write-protects the page, then runs again. ++ * held Write-protected. The same instruction reruns: a read passes, which classifies it. A ++ * write comes back at once as a bare -EFAULT, no address: the page is this vCPU's ++ * pending page while RIP has not moved, and it opens. ++ * open Nothing on the page. ++ * ++ * A write to a page held since an earlier read gives -EFAULT with nothing to name it, so it ++ * opens every held page and counts one upgrade. Every Nth present the census's thread re-arms ++ * what was touched: WP off, then DONTNEED, then poison. That order matters: MADV_DONTNEED drops a ++ * poison marker but on a file mapping it keeps a uffd-wp one, and POISON refuses a slot that holds ++ * either (EEXIST). ++ * ++ * UFFDIO_POISON and UFFDIO_WRITEPROTECT work on PTEs, so 98k Reims views and the 12 GiB slot add no ++ * VMAs beyond this one mapping: arming splits nothing, and vm.max_map_count sees +1. ++ * ++ * Registering WP makes the view's read faults map read-only (vma_wants_writenotify), so the first ++ * write after a read of each page takes one extra host fault, once per page per mapping. ++ * ++ * Locks: the fault path takes none, and only touches per-page atomics, madvise and uffd ioctls. It ++ * queues no GPU work and calls nothing in Reims. census.lock guards the surfaces, never waits on a ++ * vCPU or the GPU, and is taken only on the rare -EFAULT with no pending page. Each page's word ++ * is the owner's: whoever moves it to ST_WORK alone changes that page's markers. ++ * ++ * A failure must never take the VM with it: an -EFAULT nothing answers disarms the whole census ++ * (census_fuse) before it is allowed to reach QEMU's fatal path. ++ * ++ * SPDX-License-Identifier: GPL-2.0-or-later ++ */ ++ ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#include "system/kvm_census.h" ++ ++#ifndef MIN ++#define MIN(a, b) ((a) < (b) ? (a) : (b)) ++#define MAX(a, b) ((a) > (b) ? (a) : (b)) ++#endif ++ ++#define PG 4096ULL ++#define PG_SHIFT 12 ++#define RING_SIZE 8192 /* a power of two */ ++#define TOP_RIPS 8 ++#define RIP_SEEN_SLOTS (1 << 16) ++#define POISON_TRIES 4 ++#define SPIN_LIMIT 100000 /* about 0.5 s of repeated SIGBUS on one page: real poison */ ++/* An -EFAULT unanswered this long is no race: first the markers go, then QEMU's fatal path. */ ++#define EFAULT_GAP_NS 5000000 ++#define FUSE_AFTER_NS 20000000 ++#define FATAL_AFTER_NS 120000000 ++ ++/* A page's word: the state's three bits, then the mapping id. 0 is a page never armed. */ ++enum { ST_NONE, ST_ARMED, ST_HELD, ST_OPEN, ST_SKIP, ST_WORK, ST_STALE }; ++#define PW(st, id) (((uint32_t)(st) << 29) | (id)) ++#define PW_ST(w) ((w) >> 29) ++#define PW_ID(w) ((w) & 0x1fffffffu) ++ ++enum { REC_TOUCH, REC_WRITE, REC_UPGRADE }; ++ ++typedef struct Rec { ++ uint64_t rip; ++ uint64_t cr3; ++ uint32_t page; ++ uint32_t serial; ++ uint16_t id; ++ uint8_t kind; ++ uint8_t cpl; ++ uint8_t ncode; ++ uint8_t code[15]; ++} Rec; ++ ++struct CensusVcpu { ++ int index; ++ int fd; ++ void *opaque; ++ CensusReadGpa *read_gpa; ++ /* The page this vCPU trapped on last, while its instruction may be rerunning. */ ++ int64_t pending; ++ uint64_t pending_rip; ++ void *spin_addr; ++ uint32_t spins; ++ uint64_t efault_first; ++ uint64_t efault_last; ++ uint64_t seen[256]; ++ _Atomic uint32_t head; ++ _Atomic uint32_t tail; ++ Rec ring[RING_SIZE]; ++}; ++ ++typedef struct Run { ++ uint32_t first; ++ uint32_t count; ++} Run; ++ ++typedef struct RipCount { ++ uint64_t rip; ++ uint64_t count; ++} RipCount; ++ ++/* The running totals of a surface, and of the tick written next. */ ++typedef struct Tally { ++ uint64_t traps, writes, upgrades; ++ uint64_t pages_r, pages_w, pages_skip; ++ uint64_t windows_touched, windows_saturated; ++ uint64_t cpl0, cpl3; ++ uint64_t vcpus; ++ RipCount rips[TOP_RIPS]; ++} Tally; ++ ++typedef struct Surface { ++ bool live; ++ uint32_t serial; ++ Run *runs; ++ uint32_t run_count; ++ uint64_t armed_pages; ++ uint64_t overlaps; ++ uint64_t foreign; ++ uint64_t presents_at_map; ++ uint64_t windows_at_map; ++ uint64_t max_window_pages; ++ uint64_t presents_at_tick; ++ Tally total; ++ Tally tick; ++ _Atomic uint64_t presents; ++ _Atomic uint32_t window_traps; ++ _Atomic uint32_t saturate; ++} Surface; ++ ++typedef struct Line { ++ struct Line *next; ++ char text[]; ++} Line; ++ ++static struct { ++ int wanted; /* -1 not read yet */ ++ bool active; ++ _Atomic bool dead; ++ char dir[3584]; ++ uint8_t *view; ++ uint8_t *main_map; ++ uint64_t len; ++ uint64_t pages; ++ int uffd; ++ _Atomic uint32_t *pg; ++ Surface *surfaces; ++ CensusVcpu *vcpus[CENSUS_MAX_VCPUS]; ++ ++ pthread_mutex_t lock; ++ sem_t wake; ++ pthread_t thread; ++ Line *lines_head, **lines_tail; ++ FILE *ticks, *facts, *rips; ++ uint64_t *rips_seen; ++ uint32_t next_serial; ++ ++ bool rw; ++ _Atomic uint32_t every; ++ uint32_t min_pixels; ++ uint32_t saturate_at; ++ uint32_t flush_ms; ++ uint32_t window_ms; ++ ++ _Atomic uint64_t presents; ++ uint64_t presents_at_window; ++ uint64_t presents_at_tick; ++ uint64_t window_ns; ++ uint64_t windows; ++ uint64_t tick_ns; ++ ++ _Atomic uint64_t traps, writes, upgrades, spurious, dropped, errors; ++ uint64_t overlaps; ++} census = { ++ .wanted = -1, ++ .uffd = -1, ++ .lock = PTHREAD_MUTEX_INITIALIZER, ++}; ++ ++static uint64_t now_ns(void) ++{ ++ struct timespec t; ++ ++ clock_gettime(CLOCK_MONOTONIC, &t); ++ return t.tv_sec * 1000000000ULL + t.tv_nsec; ++} ++ ++static uint32_t env_u32(const char *name, uint32_t fallback) ++{ ++ const char *v = getenv(name); ++ ++ return v && *v ? (uint32_t)strtoul(v, NULL, 0) : fallback; ++} ++ ++bool census_wanted(void) ++{ ++ if (census.wanted < 0) { ++ const char *dir = getenv(CENSUS_ENV); ++ ++ census.wanted = dir && *dir; ++ if (census.wanted) { ++ snprintf(census.dir, sizeof(census.dir), "%s", dir); ++ } ++ } ++ return census.wanted; ++} ++ ++bool census_active(void) ++{ ++ return census.active; ++} ++ ++bool census_view_holds(const void *addr) ++{ ++ return census.view && ++ (uint64_t)((const uint8_t *)addr - census.view) < census.len; ++} ++ ++void *census_hva(void *ram) ++{ ++ uint64_t off = (uint8_t *)ram - census.main_map; ++ ++ return census.active && off < census.len ? census.view + off : ram; ++} ++ ++void *census_main_addr(void *addr) ++{ ++ return census_view_holds(addr) ? ++ census.main_map + ((uint8_t *)addr - census.view) : addr; ++} ++ ++/* ---- markers ------------------------------------------------------------------------------ */ ++ ++static void count_error(const char *what, uint64_t page) ++{ ++ if (atomic_fetch_add(&census.errors, 1) < 16) { ++ fprintf(stderr, "census: %s at page %#" PRIx64 ": %s\n", what, page, strerror(errno)); ++ } ++} ++ ++static int page_drop(uint64_t page, uint64_t count) ++{ ++ return madvise(census.view + (page << PG_SHIFT), count << PG_SHIFT, MADV_DONTNEED); ++} ++ ++static int page_wp(uint64_t page, uint64_t count, bool on) ++{ ++ struct uffdio_writeprotect w = { ++ .range = { (uint64_t)census.view + (page << PG_SHIFT), count << PG_SHIFT }, ++ /* The kernel takes DONTWAKE only for WP off. Nothing ever waits on this uffd. */ ++ .mode = on ? UFFDIO_WRITEPROTECT_MODE_WP : UFFDIO_WRITEPROTECT_MODE_DONTWAKE, ++ }; ++ ++ int r; ++ ++ for (int tries = 0; tries < POISON_TRIES; tries++) { ++ r = ioctl(census.uffd, UFFDIO_WRITEPROTECT, &w); ++ if (r == 0 || errno != EAGAIN) { ++ break; ++ } ++ } ++ return r; ++} ++ ++static void store_pages(uint64_t first, uint64_t count, uint32_t state, uint32_t id) ++{ ++ for (uint64_t p = first; p < first + count; p++) { ++ atomic_store(&census.pg[p], PW(state, id)); ++ } ++} ++ ++/* ++ * Poisons pages this thread owns (ST_WORK), then gives each its state: armed, or skipped for a ++ * page a vCPU faulted back in between the DONTNEED and the POISON, so that a marker exists only ++ * where the word says armed. EEXIST names that race, and the page's next window arms it. EAGAIN ++ * is a PTE table a concurrent MADV_DONTNEED just freed, and a retry finds a new one. ++ */ ++static void poison_run(uint64_t first, uint64_t count, uint32_t id) ++{ ++ uint64_t at = first, end = first + count; ++ int tries = 0; ++ ++ while (at < end) { ++ struct uffdio_poison z = { ++ .range = { (uint64_t)census.view + (at << PG_SHIFT), (end - at) << PG_SHIFT }, ++ }; ++ uint64_t done; ++ ++ if (ioctl(census.uffd, UFFDIO_POISON, &z) == 0) { ++ store_pages(at, end - at, ST_ARMED, id); ++ return; ++ } ++ done = z.updated > 0 ? (uint64_t)z.updated >> PG_SHIFT : 0; ++ store_pages(at, done, ST_ARMED, id); ++ at += done; ++ if (at == end) { ++ return; ++ } ++ if ((errno == EEXIST || errno == EAGAIN) && ++tries < POISON_TRIES) { ++ if (errno == EEXIST) { ++ page_drop(at, 1); ++ } ++ continue; ++ } ++ if (errno != EEXIST && errno != EAGAIN) { ++ count_error("poison", at); ++ } ++ store_pages(at, 1, ST_SKIP, id); ++ at++; ++ tries = 0; ++ } ++} ++ ++static void arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_held) ++{ ++ if (any_held && page_wp(first, count, false)) { ++ count_error("WP off for arming", first); ++ } ++ page_drop(first, count); ++ poison_run(first, count, id); ++} ++ ++/* ---- the fault path, on a vCPU's thread ----------------------------------------------------- */ ++ ++CensusVcpu *census_vcpu(int index, int fd, void *opaque, CensusReadGpa *read_gpa) ++{ ++ CensusVcpu *vc; ++ ++ if (index < 0 || index >= CENSUS_MAX_VCPUS) { ++ return NULL; ++ } ++ vc = census.vcpus[index]; ++ if (!vc) { ++ vc = calloc(1, sizeof(*vc)); ++ vc->index = index; ++ vc->fd = fd; ++ vc->opaque = opaque; ++ vc->read_gpa = read_gpa; ++ vc->pending = -1; ++ census.vcpus[index] = vc; ++ } ++ return vc; ++} ++ ++static void ring_push(CensusVcpu *vc, const Rec *r) ++{ ++ uint32_t head = atomic_load_explicit(&vc->head, memory_order_relaxed); ++ uint32_t tail = atomic_load_explicit(&vc->tail, memory_order_acquire); ++ ++ if (head - tail == RING_SIZE) { ++ atomic_fetch_add(&census.dropped, 1); ++ return; ++ } ++ vc->ring[head & (RING_SIZE - 1)] = *r; ++ atomic_store_explicit(&vc->head, head + 1, memory_order_release); ++} ++ ++/* RIP, CPL and CR3 at the access, and the bytes at RIP the first time this vCPU sees it. */ ++static void snapshot(CensusVcpu *vc, Rec *r) ++{ ++ struct kvm_regs regs; ++ struct kvm_sregs sregs; ++ ++ if (ioctl(vc->fd, KVM_GET_REGS, ®s) == 0) { ++ r->rip = regs.rip; ++ } ++ if (ioctl(vc->fd, KVM_GET_SREGS, &sregs) == 0) { ++ r->cpl = sregs.cs.dpl; ++ r->cr3 = sregs.cr3; ++ } ++ if (vc->read_gpa && vc->seen[(r->rip ^ (r->rip >> 8)) & 255] != r->rip) { ++ unsigned have = 0; ++ ++ vc->seen[(r->rip ^ (r->rip >> 8)) & 255] = r->rip; ++ while (have < sizeof(r->code)) { ++ uint64_t at = r->rip + have; ++ unsigned n = MIN(sizeof(r->code) - have, PG - (at & (PG - 1))); ++ struct kvm_translation t = { .linear_address = at }; ++ ++ if (ioctl(vc->fd, KVM_TRANSLATE, &t) || !t.valid) { ++ break; ++ } ++ vc->read_gpa(vc->opaque, t.physical_address, r->code + have, n); ++ have += n; ++ } ++ r->ncode = have; ++ } ++} ++ ++int census_on_sigbus(CensusVcpu *vc, void **addr) ++{ ++ uint64_t off = (uint8_t *)*addr - census.view; ++ uint64_t page; ++ uint32_t w, id; ++ Rec r = { .kind = REC_TOUCH }; ++ ++ if (!census.view || !vc || off >= census.len) { ++ return CENSUS_NOT_OURS; ++ } ++ page = off >> PG_SHIFT; ++ w = atomic_load(&census.pg[page]); ++ if (w == 0) { ++ *addr = census.main_map + off; ++ return CENSUS_NOT_OURS; ++ } ++ if (PW_ST(w) != ST_ARMED || ++ !atomic_compare_exchange_strong(&census.pg[page], &w, PW(ST_WORK, PW_ID(w)))) { ++ /* ++ * Another vCPU took this page, or a thread is mid-way through its markers: the fault ++ * reruns. Only real poison on a page once armed keeps coming back. ++ */ ++ if (vc->spin_addr == *addr) { ++ if (++vc->spins > SPIN_LIMIT) { ++ *addr = census.main_map + off; ++ return CENSUS_NOT_OURS; ++ } ++ } else { ++ vc->spin_addr = *addr; ++ vc->spins = 0; ++ } ++ atomic_fetch_add(&census.spurious, 1); ++ return CENSUS_HANDLED; ++ } ++ vc->spin_addr = NULL; ++ id = PW_ID(w); ++ snapshot(vc, &r); ++ page_drop(page, 1); ++ if (census.rw && page_wp(page, 1, true) == 0) { ++ vc->pending = page; ++ vc->pending_rip = r.rip; ++ atomic_store(&census.pg[page], PW(ST_HELD, id)); ++ } else { ++ if (census.rw) { ++ count_error("WP on after a trap", page); ++ } ++ atomic_store(&census.pg[page], PW(ST_OPEN, id)); ++ } ++ r.page = page; ++ r.id = id; ++ r.serial = census.surfaces[id].serial; ++ ring_push(vc, &r); ++ atomic_fetch_add(&census.traps, 1); ++ if (atomic_fetch_add(&census.surfaces[id].window_traps, 1) + 1 == census.saturate_at) { ++ atomic_store(&census.surfaces[id].saturate, 1); ++ sem_post(&census.wake); ++ } ++ return CENSUS_HANDLED; ++} ++ ++/* ++ * Moves the pages of s in the states of mask to ST_WORK, and runs fn over each contiguous run of ++ * them; fn ends the work by giving each page its next state. wait: wait out a page another ++ * thread is working on, as an unmap must; otherwise that page is left for the next pass. ++ */ ++typedef void RunFn(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg); ++ ++static uint32_t take_pages(Surface *s, uint32_t id, uint32_t mask, bool wait, RunFn *fn, void *arg) ++{ ++ uint32_t taken = 0; ++ ++ for (uint32_t i = 0; i < s->run_count; i++) { ++ uint64_t first = 0, count = 0; ++ bool any_held = false; ++ ++ for (uint64_t p = s->runs[i].first; p < s->runs[i].first + s->runs[i].count; p++) { ++ uint32_t w = atomic_load(&census.pg[p]); ++ ++ while (wait && PW_ID(w) == id && PW_ST(w) == ST_WORK) { ++ sched_yield(); ++ w = atomic_load(&census.pg[p]); ++ } ++ if (PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { ++ if (count && first + count != p) { ++ fn(first, count, id, any_held, arg); ++ count = 0; ++ any_held = false; ++ } ++ first = count ? first : p; ++ count++; ++ taken++; ++ any_held |= PW_ST(w) == ST_HELD; ++ } else if (count) { ++ fn(first, count, id, any_held, arg); ++ count = 0; ++ any_held = false; ++ } ++ } ++ if (count) { ++ fn(first, count, id, any_held, arg); ++ } ++ } ++ return taken; ++} ++ ++static void open_held(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ if (page_wp(first, count, false)) { ++ count_error("WP off", first); ++ } ++ store_pages(first, count, ST_OPEN, id); ++} ++ ++/* Everything off: the unmap's and the reset's work, done before either returns. */ ++static void clear_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ if (any_held) { ++ page_wp(first, count, false); ++ } ++ page_drop(first, count); ++ store_pages(first, count, ST_STALE, 0x1fffffffu); ++} ++ ++#define MASK(st) (1u << (st)) ++#define TOUCHED (MASK(ST_HELD) | MASK(ST_OPEN) | MASK(ST_SKIP)) ++ ++static void census_fuse(const char *why) ++{ ++ if (atomic_exchange(&census.dead, true)) { ++ return; ++ } ++ fprintf(stderr, "census: %s: every marker dropped, the census is off\n", why); ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ Surface *s = &census.surfaces[id]; ++ ++ if (s->live) { ++ take_pages(s, id, MASK(ST_ARMED) | TOUCHED, true, clear_run, NULL); ++ } ++ } ++} ++ ++bool census_on_efault(CensusVcpu *vc) ++{ ++ Rec r = { .kind = REC_UPGRADE }; ++ uint32_t opened = 0; ++ uint64_t now; ++ ++ if (!census.active || !vc) { ++ return false; ++ } ++ snapshot(vc, &r); ++ if (vc->pending >= 0 && vc->pending_rip == r.rip) { ++ uint64_t page = vc->pending; ++ uint32_t w = atomic_load(&census.pg[page]); ++ ++ vc->pending = -1; ++ if (PW_ST(w) == ST_HELD && ++ atomic_compare_exchange_strong(&census.pg[page], &w, PW(ST_WORK, PW_ID(w)))) { ++ uint32_t id = PW_ID(w); ++ ++ open_held(page, 1, id, false, NULL); ++ r.kind = REC_WRITE; ++ r.page = page; ++ r.id = id; ++ r.serial = census.surfaces[id].serial; ++ ring_push(vc, &r); ++ atomic_fetch_add(&census.writes, 1); ++ vc->efault_first = 0; ++ return true; ++ } ++ } ++ vc->pending = -1; ++ /* A write to a page held from an earlier read: no address, so every held page opens. */ ++ pthread_mutex_lock(&census.lock); ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ Surface *s = &census.surfaces[id]; ++ ++ if (s->live) { ++ uint32_t n = take_pages(s, id, MASK(ST_HELD), false, open_held, NULL); ++ ++ s->total.upgrades += n; ++ s->tick.upgrades += n; ++ opened += n; ++ } ++ } ++ if (opened) { ++ pthread_mutex_unlock(&census.lock); ++ vc->efault_first = 0; ++ r.page = opened; ++ ring_push(vc, &r); ++ atomic_fetch_add(&census.upgrades, 1); ++ return true; ++ } ++ /* ++ * Nothing was held: another thread had the page, and the -EFAULT should not return. Should it ++ * keep returning, something answers it that no write protection explains. ++ */ ++ now = now_ns(); ++ if (!vc->efault_first || now - vc->efault_last > EFAULT_GAP_NS) { ++ vc->efault_first = now; ++ } ++ vc->efault_last = now; ++ if (now - vc->efault_first > FUSE_AFTER_NS) { ++ census_fuse("an -EFAULT no held page explains"); ++ } ++ pthread_mutex_unlock(&census.lock); ++ if (now - vc->efault_first > FATAL_AFTER_NS) { ++ return false; ++ } ++ usleep(20); ++ return true; ++} ++ ++/* ---- surfaces, from the library's threads ------------------------------------------------- */ ++ ++static void queue_line(const char *format, ...) ++{ ++ va_list a, b; ++ int n; ++ Line *line; ++ ++ va_start(a, format); ++ va_copy(b, a); ++ n = vsnprintf(NULL, 0, format, a); ++ va_end(a); ++ line = malloc(sizeof(*line) + n + 1); ++ vsnprintf(line->text, n + 1, format, b); ++ va_end(b); ++ line->next = NULL; ++ *census.lines_tail = line; ++ census.lines_tail = &line->next; ++} ++ ++static double census_time(void) ++{ ++ return now_ns() / 1e9; ++} ++ ++static void rip_add(RipCount *top, uint64_t rip) ++{ ++ RipCount *low = &top[0]; ++ ++ for (int i = 0; i < TOP_RIPS; i++) { ++ if (top[i].rip == rip) { ++ top[i].count++; ++ return; ++ } ++ if (top[i].count < low->count) { ++ low = &top[i]; ++ } ++ } ++ low->rip = rip; ++ low->count = 1; ++} ++ ++static void drain_rec(const Rec *r, int vcpu) ++{ ++ Surface *s = &census.surfaces[r->id]; ++ ++ if (r->ncode) { ++ uint64_t *slot = &census.rips_seen[(r->rip * 0x9e3779b97f4a7c15ULL) >> 48]; ++ ++ if (*slot != r->rip) { ++ char hex[sizeof(r->code) * 2 + 1]; ++ ++ *slot = r->rip; ++ for (int i = 0; i < r->ncode; i++) { ++ snprintf(hex + i * 2, 3, "%02x", r->code[i]); ++ } ++ queue_line("{\"rip\":\"%#" PRIx64 "\",\"cpl\":%u,\"cr3\":\"%#" PRIx64 ++ "\",\"bytes\":\"%s\"}\n", r->rip, r->cpl, r->cr3, hex); ++ } ++ } ++ if (r->kind == REC_UPGRADE) { ++ queue_line("{\"upgrade\":%u,\"t\":%.3f,\"rip\":\"%#" PRIx64 "\",\"vcpu\":%d}\n", ++ r->page, census_time(), r->rip, vcpu); ++ return; ++ } ++ if (r->serial != s->serial) { ++ return; ++ } ++ if (r->kind == REC_WRITE) { ++ s->total.writes++; ++ s->tick.writes++; ++ return; ++ } ++ for (Tally *t = &s->total; ; t = &s->tick) { ++ t->traps++; ++ t->cpl0 += r->cpl == 0; ++ t->cpl3 += r->cpl != 0; ++ t->vcpus |= 1ULL << vcpu; ++ rip_add(t->rips, r->rip); ++ if (t == &s->tick) { ++ break; ++ } ++ } ++} ++ ++static void drain_locked(void) ++{ ++ for (int i = 0; i < CENSUS_MAX_VCPUS; i++) { ++ CensusVcpu *vc = census.vcpus[i]; ++ uint32_t tail, head; ++ ++ if (!vc) { ++ continue; ++ } ++ tail = atomic_load_explicit(&vc->tail, memory_order_relaxed); ++ head = atomic_load_explicit(&vc->head, memory_order_acquire); ++ while (tail != head) { ++ drain_rec(&vc->ring[tail & (RING_SIZE - 1)], i); ++ tail++; ++ } ++ atomic_store_explicit(&vc->tail, tail, memory_order_release); ++ } ++} ++ ++static void rips_text(char *out, size_t size, const RipCount *top) ++{ ++ size_t at = 0; ++ ++ out[0] = 0; ++ for (int i = 0; i < TOP_RIPS && at + 48 < size; i++) { ++ if (top[i].count) { ++ at += snprintf(out + at, size - at, "%s[\"%#" PRIx64 "\",%" PRIu64 "]", ++ at ? "," : "", top[i].rip, top[i].count); ++ } ++ } ++} ++ ++static void surface_end(Surface *s, uint32_t id, const char *event) ++{ ++ char rips[512]; ++ ++ rips_text(rips, sizeof(rips), s->total.rips); ++ queue_line("{\"ev\":\"%s\",\"t\":%.3f,\"s\":%u,\"id\":%u,\"traps\":%" PRIu64 ++ ",\"writes\":%" PRIu64 ",\"upgrades\":%" PRIu64 ",\"pages_r\":%" PRIu64 ++ ",\"pages_w\":%" PRIu64 ",\"pages_skip\":%" PRIu64 ",\"cpl0\":%" PRIu64 ++ ",\"cpl3\":%" PRIu64 ",\"vcpus\":\"%#" PRIx64 "\",\"presents\":%" PRIu64 ++ ",\"frames\":%" PRIu64 ",\"windows\":%" PRIu64 ",\"windows_touched\":%" PRIu64 ++ ",\"windows_saturated\":%" PRIu64 ",\"max_window_pages\":%" PRIu64 ++ ",\"rips\":[%s]}\n", ++ event, census_time(), s->serial, id, s->total.traps, s->total.writes, ++ s->total.upgrades, s->total.pages_r, s->total.pages_w, s->total.pages_skip, ++ s->total.cpl0, s->total.cpl3, s->total.vcpus, atomic_load(&s->presents), ++ atomic_load(&census.presents) - s->presents_at_map, ++ census.windows - s->windows_at_map, s->total.windows_touched, ++ s->total.windows_saturated, s->max_window_pages, rips); ++} ++ ++static void surface_close(Surface *s, uint32_t id, const char *event) ++{ ++ drain_locked(); ++ take_pages(s, id, MASK(ST_ARMED) | TOUCHED, true, clear_run, NULL); ++ surface_end(s, id, event); ++ s->live = false; ++ free(s->runs); ++ s->runs = NULL; ++ s->run_count = 0; ++} ++ ++void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t page_count, ++ uint64_t page_size) ++{ ++ Surface *s; ++ uint32_t id = f->mapping_id; ++ uint64_t pixels = f->width * f->height; ++ size_t capacity = page_count ? page_count : 1; ++ ++ if (!census.active || atomic_load(&census.dead) || id == 0 || id >= CENSUS_MAX_SURFACES || ++ pixels < census.min_pixels) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ s = &census.surfaces[id]; ++ if (s->live) { ++ surface_close(s, id, "replaced"); ++ } ++ atomic_store(&s->presents, 0); ++ atomic_store(&s->window_traps, 0); ++ atomic_store(&s->saturate, 0); ++ memset(&s->total, 0, sizeof(s->total)); ++ memset(&s->tick, 0, sizeof(s->tick)); ++ s->serial = ++census.next_serial; ++ s->armed_pages = s->overlaps = s->foreign = s->max_window_pages = 0; ++ s->presents_at_map = s->presents_at_tick = atomic_load(&census.presents); ++ s->windows_at_map = census.windows; ++ s->runs = calloc(capacity, sizeof(*s->runs)); ++ s->live = true; ++ ++ /* A page already armed for a surface over the same memory stays with that one. */ ++ for (size_t i = 0; i < page_count; i++) { ++ uint64_t off = pages[i] - (uint64_t)census.main_map; ++ uint64_t first = off >> PG_SHIFT, count = page_size >> PG_SHIFT; ++ ++ if (off >= census.len || off + page_size > census.len || off & (PG - 1)) { ++ s->foreign += count; ++ continue; ++ } ++ for (uint64_t p = first; p < first + count; p++) { ++ uint32_t w = atomic_load(&census.pg[p]); ++ ++ if ((w != 0 && PW_ST(w) != ST_STALE) || ++ !atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { ++ s->overlaps++; ++ census.overlaps++; ++ continue; ++ } ++ if (s->run_count && s->runs[s->run_count - 1].first + ++ s->runs[s->run_count - 1].count == p) { ++ s->runs[s->run_count - 1].count++; ++ } else { ++ if (s->run_count == capacity) { ++ capacity *= 2; ++ s->runs = realloc(s->runs, sizeof(*s->runs) * capacity); ++ } ++ s->runs[s->run_count].first = p; ++ s->runs[s->run_count].count = 1; ++ s->run_count++; ++ } ++ s->armed_pages++; ++ } ++ } ++ for (uint32_t i = 0; i < s->run_count; i++) { ++ arm_run(s->runs[i].first, s->runs[i].count, id, false); ++ } ++ queue_line("{\"ev\":\"map\",\"t\":%.3f,\"s\":%u,\"id\":%u,\"format\":\"%#x\",\"w\":%" PRIu64 ++ ",\"h\":%" PRIu64 ",\"row\":%" PRIu64 ",\"alloc\":%" PRIu64 ",\"protection\":%" ++ PRIu64 ",\"address_format\":%u,\"compression\":%u,\"read_only\":%d," ++ "\"host_only\":%d,\"armed_pages\":%" PRIu64 ",\"runs\":%u,\"overlaps\":%" PRIu64 ++ ",\"foreign\":%" PRIu64 "}\n", ++ census_time(), s->serial, id, f->pixel_format, f->width, f->height, ++ f->bytes_per_row, f->alloc_size, f->protection_options, f->address_format, ++ f->compression_type, f->read_only, f->host_only, s->armed_pages, s->run_count, ++ s->overlaps, s->foreign); ++ pthread_mutex_unlock(&census.lock); ++} ++ ++void census_surface_unmapped(uint32_t mapping_id) ++{ ++ if (!census.active || mapping_id == 0 || mapping_id >= CENSUS_MAX_SURFACES) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ if (census.surfaces[mapping_id].live) { ++ surface_close(&census.surfaces[mapping_id], mapping_id, "unmap"); ++ } ++ pthread_mutex_unlock(&census.lock); ++} ++ ++void census_surface_used(uint32_t mapping_id, uint64_t usage, uint64_t storage_mode) ++{ ++ if (!census.active || mapping_id == 0 || mapping_id >= CENSUS_MAX_SURFACES) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ if (census.surfaces[mapping_id].live) { ++ queue_line("{\"ev\":\"use\",\"s\":%u,\"usage\":\"%#" PRIx64 "\",\"storage\":%" PRIu64 "}\n", ++ census.surfaces[mapping_id].serial, usage, storage_mode); ++ } ++ pthread_mutex_unlock(&census.lock); ++} ++ ++void census_presented(uint64_t mapping_id) ++{ ++ uint64_t n; ++ ++ if (!census.active) { ++ return; ++ } ++ n = atomic_fetch_add(&census.presents, 1) + 1; ++ if (mapping_id > 0 && mapping_id < CENSUS_MAX_SURFACES && ++ census.surfaces[mapping_id].live) { ++ atomic_fetch_add(&census.surfaces[mapping_id].presents, 1); ++ } ++ if (n % atomic_load(&census.every) == 0) { ++ sem_post(&census.wake); ++ } ++} ++ ++void census_reset(void) ++{ ++ if (!census.active) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ if (census.surfaces[id].live) { ++ surface_close(&census.surfaces[id], id, "reset"); ++ } ++ } ++ for (int i = 0; i < CENSUS_MAX_VCPUS; i++) { ++ if (census.vcpus[i]) { ++ census.vcpus[i]->pending = -1; ++ } ++ } ++ pthread_mutex_unlock(&census.lock); ++} ++ ++/* ---- the census's thread ------------------------------------------------------------------- */ ++ ++typedef struct WindowCount { ++ uint64_t r, w, skip; ++} WindowCount; ++ ++static void rearm_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ arm_run(first, count, id, any_held); ++} ++ ++static void count_pages(Surface *s, uint32_t id, WindowCount *c) ++{ ++ for (uint32_t i = 0; i < s->run_count; i++) { ++ for (uint64_t p = s->runs[i].first; p < s->runs[i].first + s->runs[i].count; p++) { ++ uint32_t w = atomic_load(&census.pg[p]); ++ ++ if (PW_ID(w) == id) { ++ c->r += PW_ST(w) == ST_HELD; ++ c->w += PW_ST(w) == ST_OPEN; ++ c->skip += PW_ST(w) == ST_SKIP; ++ } ++ } ++ } ++} ++ ++static void saturate_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ page_drop(first, count); ++ store_pages(first, count, ST_SKIP, id); ++} ++ ++/* ++ * A surface read whole in a window has answered for that window: the rest of its markers go at ++ * once, so that a recorder pays a few hundred traps a window rather than thousands. ++ */ ++static void saturate_locked(void) ++{ ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ Surface *s = &census.surfaces[id]; ++ ++ if (s->live && atomic_exchange(&s->saturate, 0)) { ++ take_pages(s, id, MASK(ST_ARMED), false, saturate_run, NULL); ++ s->total.windows_saturated++; ++ s->tick.windows_saturated++; ++ } ++ } ++} ++ ++static void close_window_locked(void) ++{ ++ census.windows++; ++ census.presents_at_window = atomic_load(&census.presents); ++ census.window_ns = now_ns(); ++ if (atomic_load(&census.dead)) { ++ return; ++ } ++ drain_locked(); ++ saturate_locked(); ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ Surface *s = &census.surfaces[id]; ++ WindowCount c = { 0 }; ++ uint64_t pages; ++ ++ if (!s->live || !atomic_exchange(&s->window_traps, 0)) { ++ continue; ++ } ++ count_pages(s, id, &c); ++ pages = c.r + c.w + c.skip; ++ s->total.pages_r += c.r; ++ s->tick.pages_r += c.r; ++ s->total.pages_w += c.w; ++ s->tick.pages_w += c.w; ++ s->total.pages_skip += c.skip; ++ s->tick.pages_skip += c.skip; ++ s->total.windows_touched++; ++ s->tick.windows_touched++; ++ s->max_window_pages = MAX(s->max_window_pages, pages); ++ take_pages(s, id, TOUCHED, false, rearm_run, NULL); ++ /* The lock is the fallback path's and the map's: never held for a whole pass of arming. */ ++ pthread_mutex_unlock(&census.lock); ++ sched_yield(); ++ pthread_mutex_lock(&census.lock); ++ } ++} ++ ++void census_close_window(void) ++{ ++ pthread_mutex_lock(&census.lock); ++ close_window_locked(); ++ pthread_mutex_unlock(&census.lock); ++} ++ ++static void tick_locked(void) ++{ ++ uint64_t presents = atomic_load(&census.presents); ++ char rips[512]; ++ ++ drain_locked(); ++ queue_line("{\"t\":%.3f,\"presents\":%" PRIu64 ",\"frames\":%" PRIu64 ",\"windows\":%" PRIu64 ++ ",\"traps\":%" PRIu64 ",\"writes\":%" PRIu64 ",\"upgrades\":%" PRIu64 ++ ",\"spurious\":%" PRIu64 ",\"dropped\":%" PRIu64 ",\"errors\":%" PRIu64 ++ ",\"dead\":%d}\n", ++ census_time(), presents, presents - census.presents_at_tick, census.windows, ++ atomic_load(&census.traps), atomic_load(&census.writes), ++ atomic_load(&census.upgrades), atomic_load(&census.spurious), ++ atomic_load(&census.dropped), atomic_load(&census.errors), ++ (int)atomic_load(&census.dead)); ++ census.presents_at_tick = presents; ++ for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { ++ Surface *s = &census.surfaces[id]; ++ uint64_t own; ++ ++ if (!s->live) { ++ continue; ++ } ++ own = atomic_load(&s->presents); ++ if (s->tick.traps || s->tick.upgrades || s->tick.windows_touched) { ++ rips_text(rips, sizeof(rips), s->tick.rips); ++ queue_line("{\"tick\":%u,\"t\":%.3f,\"presents\":%" PRIu64 ",\"traps\":%" PRIu64 ++ ",\"writes\":%" PRIu64 ",\"upgrades\":%" PRIu64 ",\"pages_r\":%" PRIu64 ++ ",\"pages_w\":%" PRIu64 ",\"pages_skip\":%" PRIu64 ++ ",\"windows_touched\":%" PRIu64 ",\"windows_saturated\":%" PRIu64 ++ ",\"cpl0\":%" PRIu64 ",\"cpl3\":%" PRIu64 ",\"rips\":[%s]}\n", ++ s->serial, census_time(), own - s->presents_at_tick, s->tick.traps, ++ s->tick.writes, s->tick.upgrades, s->tick.pages_r, s->tick.pages_w, ++ s->tick.pages_skip, s->tick.windows_touched, s->tick.windows_saturated, ++ s->tick.cpl0, s->tick.cpl3, rips); ++ memset(&s->tick, 0, sizeof(s->tick)); ++ } ++ s->presents_at_tick = own; ++ } ++} ++ ++static void write_lines(void) ++{ ++ Line *line; ++ ++ pthread_mutex_lock(&census.lock); ++ line = census.lines_head; ++ census.lines_head = NULL; ++ census.lines_tail = &census.lines_head; ++ pthread_mutex_unlock(&census.lock); ++ while (line) { ++ Line *next = line->next; ++ FILE *to = census.ticks; ++ ++ if (strncmp(line->text, "{\"ev\"", 5) == 0) { ++ to = census.facts; ++ } else if (strncmp(line->text, "{\"rip\"", 6) == 0) { ++ to = census.rips; ++ } ++ fputs(line->text, to); ++ free(line); ++ line = next; ++ } ++ fflush(census.ticks); ++ fflush(census.facts); ++ fflush(census.rips); ++} ++ ++/* /mark: "every N" sets the re-arm period; anything else is a label in the timeline. */ ++static void read_mark(void) ++{ ++ char path[sizeof(census.dir) + 8], text[128] = ""; ++ FILE *f; ++ size_t n; ++ unsigned long every; ++ ++ snprintf(path, sizeof(path), "%s/mark", census.dir); ++ f = fopen(path, "r"); ++ if (!f) { ++ return; ++ } ++ n = fread(text, 1, sizeof(text) - 1, f); ++ fclose(f); ++ unlink(path); ++ text[n] = 0; ++ text[strcspn(text, "\n")] = 0; ++ pthread_mutex_lock(&census.lock); ++ if (sscanf(text, "every %lu", &every) == 1 && every > 0) { ++ atomic_store(&census.every, every); ++ queue_line("{\"mark\":\"every %lu\",\"t\":%.3f}\n", every, census_time()); ++ } else { ++ for (char *c = text; *c; c++) { ++ *c = *c == '"' || *c == '\\' || (unsigned char)*c < 32 ? '_' : *c; ++ } ++ tick_locked(); ++ queue_line("{\"mark\":\"%s\",\"t\":%.3f}\n", text, census_time()); ++ } ++ pthread_mutex_unlock(&census.lock); ++} ++ ++static void *census_thread(void *unused) ++{ ++ sigset_t all; ++ ++ sigfillset(&all); ++ pthread_sigmask(SIG_BLOCK, &all, NULL); ++ pthread_setname_np(pthread_self(), "reims-census"); ++ census.window_ns = census.tick_ns = now_ns(); ++ for (;;) { ++ struct timespec until; ++ uint64_t now, every = atomic_load(&census.every); ++ ++ clock_gettime(CLOCK_REALTIME, &until); ++ until.tv_nsec += 50000000; ++ if (until.tv_nsec >= 1000000000) { ++ until.tv_sec++; ++ until.tv_nsec -= 1000000000; ++ } ++ sem_timedwait(&census.wake, &until); ++ now = now_ns(); ++ read_mark(); ++ pthread_mutex_lock(&census.lock); ++ drain_locked(); ++ saturate_locked(); ++ if (atomic_load(&census.presents) - census.presents_at_window >= every || ++ now - census.window_ns >= census.window_ms * 1000000ULL) { ++ close_window_locked(); ++ } ++ if (now - census.tick_ns >= census.flush_ms * 1000000ULL) { ++ tick_locked(); ++ census.tick_ns = now; ++ } ++ pthread_mutex_unlock(&census.lock); ++ write_lines(); ++ } ++ return unused; ++} ++ ++void census_flush(void) ++{ ++ if (!census.active) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ tick_locked(); ++ pthread_mutex_unlock(&census.lock); ++ write_lines(); ++} ++ ++static void census_at_exit(void) ++{ ++ census_flush(); ++} ++ ++void census_counts(CensusCounts *out) ++{ ++ *out = (CensusCounts) { ++ .traps = atomic_load(&census.traps), ++ .writes = atomic_load(&census.writes), ++ .upgrades = atomic_load(&census.upgrades), ++ .spurious = atomic_load(&census.spurious), ++ .dropped = atomic_load(&census.dropped), ++ .errors = atomic_load(&census.errors), ++ .overlaps = census.overlaps, ++ .windows = census.windows, ++ .dead = atomic_load(&census.dead), ++ }; ++} ++ ++int census_page_state(const void *main_addr) ++{ ++ uint64_t off = (const uint8_t *)main_addr - census.main_map; ++ ++ return off < census.len ? (int)PW_ST(atomic_load(&census.pg[off >> PG_SHIFT])) : -1; ++} ++ ++static FILE *open_output(const char *name) ++{ ++ char path[sizeof(census.dir) + 32]; ++ ++ snprintf(path, sizeof(path), "%s/%s", census.dir, name); ++ return fopen(path, "ae"); ++} ++ ++static bool census_refuse(const char *what) ++{ ++ fprintf(stderr, "census: %s: %s; the census is off\n", what, strerror(errno)); ++ census.wanted = 0; ++ if (census.view) { ++ munmap(census.view, census.len); ++ census.view = NULL; ++ } ++ if (census.uffd >= 0) { ++ close(census.uffd); ++ census.uffd = -1; ++ } ++ return false; ++} ++ ++bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) ++{ ++ struct uffdio_api api = { ++ .api = UFFD_API, ++ .features = UFFD_FEATURE_POISON | UFFD_FEATURE_WP_HUGETLBFS_SHMEM | ++ UFFD_FEATURE_WP_UNPOPULATED, ++ }; ++ struct uffdio_register reg = { .mode = UFFDIO_REGISTER_MODE_WP }; ++ ++ mkdir(census.dir, 0777); ++ census.ticks = open_output("ticks.jsonl"); ++ census.facts = open_output("surfaces.jsonl"); ++ census.rips = open_output("rips.jsonl"); ++ if (!census.ticks || !census.facts || !census.rips) { ++ return census_refuse("the output directory"); ++ } ++ census.uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK | UFFD_USER_MODE_ONLY); ++ if (census.uffd < 0) { ++ return census_refuse("userfaultfd(UFFD_USER_MODE_ONLY)"); ++ } ++ if (ioctl(census.uffd, UFFDIO_API, &api)) { ++ return census_refuse("UFFDIO_API, poison and shmem write protection"); ++ } ++ census.view = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offset); ++ if (census.view == MAP_FAILED) { ++ census.view = NULL; ++ return census_refuse("the view"); ++ } ++ /* A fork's child, which only execs, needs none of this. */ ++ madvise(census.view, len, MADV_DONTFORK); ++ reg.range.start = (uint64_t)census.view; ++ reg.range.len = len; ++ if (ioctl(census.uffd, UFFDIO_REGISTER, ®)) { ++ return census_refuse("UFFDIO_REGISTER"); ++ } ++ census.main_map = main_map; ++ census.len = len; ++ census.pages = len >> PG_SHIFT; ++ census.pg = calloc(census.pages, sizeof(*census.pg)); ++ census.surfaces = calloc(CENSUS_MAX_SURFACES, sizeof(*census.surfaces)); ++ census.rips_seen = calloc(RIP_SEEN_SLOTS, sizeof(*census.rips_seen)); ++ if (!census.pg || !census.surfaces || !census.rips_seen) { ++ return census_refuse("memory"); ++ } ++ census.lines_tail = &census.lines_head; ++ census.rw = env_u32("REIMS_VGPU_CENSUS_RW", 1); ++ census.every = env_u32("REIMS_VGPU_CENSUS_EVERY", 32); ++ census.min_pixels = env_u32("REIMS_VGPU_CENSUS_MIN_PIXELS", 1000000); ++ census.saturate_at = env_u32("REIMS_VGPU_CENSUS_SATURATE", 256); ++ census.flush_ms = env_u32("REIMS_VGPU_CENSUS_FLUSH_MS", 5000); ++ census.window_ms = env_u32("REIMS_VGPU_CENSUS_WINDOW_MS", 2000); ++ if (!census.every) { ++ census.every = 1; ++ } ++ sem_init(&census.wake, 0, 0); ++ census.active = true; ++ if (pthread_create(&census.thread, NULL, census_thread, NULL)) { ++ census.active = false; ++ return census_refuse("the census's thread"); ++ } ++ atexit(census_at_exit); ++ queue_line("{\"started\":%.3f,\"pid\":%d,\"bytes\":%" PRIu64 ",\"rw\":%d,\"every\":%u," ++ "\"min_pixels\":%u,\"saturate\":%u}\n", census_time(), getpid(), len, census.rw, ++ census.every, census.min_pixels, census.saturate_at); ++ fprintf(stderr, "census: on, writing to %s: %" PRIu64 " MiB view at %p\n", census.dir, ++ len >> 20, census.view); ++ return true; ++} +diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c +index 83cbd120a8..77952192cb 100644 +--- a/accel/kvm/kvm-all.c ++++ b/accel/kvm/kvm-all.c +@@ -47,6 +47,7 @@ + #include "qapi/qapi-types-common.h" + #include "qapi/qapi-visit-common.h" + #include "system/reset.h" ++#include "system/kvm_census.h" + #include "qemu/guest-random.h" + #include "system/hw_accel.h" + #include "kvm-cpus.h" +@@ -366,6 +367,77 @@ int kvm_physical_memory_addr_from_host(KVMState *s, void *ram, + return ret; + } + ++/* LOCAL (Atlas): reads guest memory for the census's instruction bytes. */ ++static void kvm_census_read_gpa(void *opaque, uint64_t gpa, void *buf, unsigned len) ++{ ++ address_space_read(&address_space_memory, gpa, MEMTXATTRS_UNSPECIFIED, buf, len); ++} ++ ++static CensusVcpu *kvm_census_vcpu(CPUState *cpu) ++{ ++ return census_vcpu(cpu->cpu_index, cpu->kvm_fd, cpu, kvm_census_read_gpa); ++} ++ ++/* ++ * REIMS_VGPU_CENSUS_SELFTEST=:, in hex, bytes of the RAM block named "ram": arms ++ * a surface no library told of, so that a guest with no GPU device can drive the census. ++ */ ++static void kvm_census_selftest(void) ++{ ++ const char *test = getenv("REIMS_VGPU_CENSUS_SELFTEST"); ++ uint64_t offset, length; ++ RAMBlock *block; ++ g_autofree uint64_t *pages = NULL; ++ CensusSurface facts = { .mapping_id = 1, .width = 4096, .height = 4096 }; ++ ++ if (!test || sscanf(test, "%" SCNx64 ":%" SCNx64, &offset, &length) != 2) { ++ return; ++ } ++ block = qemu_ram_block_by_name("ram"); ++ if (!block) { ++ return; ++ } ++ pages = g_new(uint64_t, length >> 12); ++ for (uint64_t i = 0; i < length >> 12; i++) { ++ pages[i] = (uint64_t)block->host + offset + (i << 12); ++ } ++ census_surface_mapped(&facts, pages, length >> 12, 4096); ++} ++ ++static void kvm_census_reset(void *opaque) ++{ ++ census_reset(); ++ kvm_census_selftest(); ++} ++ ++/* ++ * LOCAL (Atlas): the address KVM gets for main-mapping address ram: the census's KVM-only view ++ * for a memslot of guest RAM, when the census is on (accel/kvm/census.c); otherwise ram, and ++ * nothing is mapped. slot->ram stays the main mapping's, for kvm_physical_memory_addr_from_host. ++ * ROMs, pflash and every other block have no fd of this kind, or are too small to hold a surface. ++ */ ++static void *kvm_census_hva(void *ram) ++{ ++ ram_addr_t offset; ++ RAMBlock *block; ++ ++ if (likely(!census_wanted())) { ++ return ram; ++ } ++ if (!census_active()) { ++ block = qemu_ram_block_from_host(ram, false, &offset); ++ if (!block || block->fd < 0 || !qemu_ram_is_shared(block) || ++ block->page_size != 4096 || block->used_length < 256 * MiB) { ++ return ram; ++ } ++ if (!census_start(block->fd, block->fd_offset, block->host, block->used_length)) { ++ return ram; ++ } ++ qemu_register_reset(kvm_census_reset, NULL); ++ } ++ return census_hva(ram); ++} ++ + static int kvm_set_user_memory_region(KVMMemoryListener *kml, KVMSlot *slot, bool new) + { + KVMState *s = kvm_state; +@@ -374,7 +446,7 @@ static int kvm_set_user_memory_region(KVMMemoryListener *kml, KVMSlot *slot, boo + + mem.slot = slot->slot | (kml->as_id << 16); + mem.guest_phys_addr = slot->start_addr; +- mem.userspace_addr = (unsigned long)slot->ram; ++ mem.userspace_addr = (unsigned long)kvm_census_hva(slot->ram); + mem.flags = slot->flags; + mem.guest_memfd = slot->guest_memfd; + mem.guest_memfd_offset = slot->guest_memfd_offset; +@@ -3474,11 +3546,23 @@ int kvm_cpu_exec(CPUState *cpu) + + #ifdef KVM_HAVE_MCE_INJECTION + if (unlikely(have_sigbus_pending)) { +- bql_lock(); +- kvm_arch_on_sigbus_vcpu(cpu, pending_sigbus_code, +- pending_sigbus_addr); +- have_sigbus_pending = false; +- bql_unlock(); ++ /* LOCAL (Atlas): a census marker runs the vCPU again, off the BQL. */ ++ if (census_active() && kvm_census_vcpu(cpu) && ++ census_on_sigbus(kvm_census_vcpu(cpu), &pending_sigbus_addr) == ++ CENSUS_HANDLED) { ++ have_sigbus_pending = false; ++ if (run_ret == -EINTR) { ++ ret = 0; ++ continue; ++ } ++ } else { ++ qatomic_set(&cpu->exit_request, true); ++ bql_lock(); ++ kvm_arch_on_sigbus_vcpu(cpu, pending_sigbus_code, ++ pending_sigbus_addr); ++ have_sigbus_pending = false; ++ bql_unlock(); ++ } + } + #endif + +@@ -3489,6 +3573,12 @@ int kvm_cpu_exec(CPUState *cpu) + ret = EXCP_INTERRUPT; + break; + } ++ /* LOCAL (Atlas): a write to a page the census holds write-protected. */ ++ if (run_ret == -EFAULT && run->exit_reason != KVM_EXIT_MEMORY_FAULT && ++ census_active() && census_on_efault(kvm_census_vcpu(cpu))) { ++ ret = 0; ++ continue; ++ } + if (!(run_ret == -EFAULT && run->exit_reason == KVM_EXIT_MEMORY_FAULT)) { + fprintf(stderr, "error: kvm run failed %s\n", + strerror(-run_ret)); +@@ -3993,7 +4083,10 @@ int kvm_on_sigbus_vcpu(CPUState *cpu, int code, void *addr) + have_sigbus_pending = true; + pending_sigbus_addr = addr; + pending_sigbus_code = code; +- qatomic_set(&cpu->exit_request, true); ++ /* LOCAL (Atlas): the census's markers are answered on the way out of KVM_RUN. */ ++ if (!census_view_holds(addr)) { ++ qatomic_set(&cpu->exit_request, true); ++ } + return 0; + #else + return 1; +@@ -4009,7 +4102,7 @@ int kvm_on_sigbus(int code, void *addr) + * we can only get action optional here. + */ + assert(code != BUS_MCEERR_AR); +- kvm_arch_on_sigbus_vcpu(first_cpu, code, addr); ++ kvm_arch_on_sigbus_vcpu(first_cpu, code, census_main_addr(addr)); + return 0; + #else + return 1; +diff --git a/accel/kvm/meson.build b/accel/kvm/meson.build +index 397a1fe1fd..a154754a04 100644 +--- a/accel/kvm/meson.build ++++ b/accel/kvm/meson.build +@@ -5,3 +5,4 @@ kvm_ss.add(files( + )) + + specific_ss.add_all(when: 'CONFIG_KVM', if_true: kvm_ss) ++system_ss.add(when: 'CONFIG_LINUX', if_true: files('census.c')) +diff --git a/hw/display/apple-gfx.c b/hw/display/apple-gfx.c +index de8a240f95..935c5850d1 100644 +--- a/hw/display/apple-gfx.c ++++ b/hw/display/apple-gfx.c +@@ -32,6 +32,9 @@ + #include "ui/input.h" + #include "apple-gfx.h" + #include "trace.h" ++#ifdef __linux__ ++#include "system/kvm_census.h" ++#endif + + static const AppleGFXDisplayMode apple_gfx_default_modes[] = { + { 1920, 1080, 60 }, +@@ -1001,6 +1004,69 @@ static bool apple_gfx_parse_resolution(const char *str, uint16_t *width, + return true; + } + ++#ifdef __linux__ ++/* ++ * LOCAL (Atlas): the library's map, unmap, texture and present news for the touch census ++ * (accel/kvm/census.c). The two surface calls run on the IOSurface device's thread; presented on a ++ * present's, where it takes no lock. ++ */ ++static void apple_gfx_census_mapped(void *ctx, const RvgCensusSurface *surface, ++ const uint64_t *pages, size_t page_count, ++ uint64_t page_size) ++{ ++ CensusSurface facts = { ++ .mapping_id = surface->mapping_id, ++ .pixel_format = surface->pixel_format, ++ .width = surface->width, ++ .height = surface->height, ++ .bytes_per_row = surface->bytes_per_row, ++ .alloc_size = surface->alloc_size, ++ .protection_options = surface->protection_options, ++ .address_format = surface->address_format, ++ .compression_type = surface->compression_type, ++ .read_only = surface->read_only, ++ .host_only = surface->host_only, ++ }; ++ ++ census_surface_mapped(&facts, pages, page_count, page_size); ++} ++ ++static void apple_gfx_census_unmapped(void *ctx, uint32_t mapping_id) ++{ ++ census_surface_unmapped(mapping_id); ++} ++ ++static void apple_gfx_census_used(void *ctx, uint32_t mapping_id, uint64_t usage, ++ uint64_t storage_mode) ++{ ++ census_surface_used(mapping_id, usage, storage_mode); ++} ++ ++static void apple_gfx_census_presented(void *ctx, uint64_t mapping_id) ++{ ++ census_presented(mapping_id); ++} ++ ++static void apple_gfx_census_register(void) ++{ ++ static const RvgCensusCallbacks callbacks = { ++ .struct_size = sizeof(RvgCensusCallbacks), ++ .mapped = apple_gfx_census_mapped, ++ .unmapped = apple_gfx_census_unmapped, ++ .used = apple_gfx_census_used, ++ .presented = apple_gfx_census_presented, ++ }; ++ ++ if (census_active()) { ++ rvg_census_set_callbacks(&callbacks); ++ } ++} ++#else ++static void apple_gfx_census_register(void) ++{ ++} ++#endif ++ + bool apple_gfx_common_realize(AppleGFXState *s, DeviceState *dev, + RvgDeviceConfig *config, Error **errp) + { +@@ -1093,6 +1159,7 @@ bool apple_gfx_common_realize(AppleGFXState *s, DeviceState *dev, + s->displays[i].cursor_show = true; + } + ++ apple_gfx_census_register(); + status = rvg_device_create(config, &s->pgdev); + if (status != RVG_OK) { + error_setg(errp, "apple-gfx: rvg_device_create: %s", +diff --git a/include/system/kvm_census.h b/include/system/kvm_census.h +new file mode 100644 +index 0000000000..e242236452 +--- /dev/null ++++ b/include/system/kvm_census.h +@@ -0,0 +1,91 @@ ++/* ++ * LOCAL (Atlas): the touch census. See accel/kvm/census.c. ++ * ++ * SPDX-License-Identifier: GPL-2.0-or-later ++ */ ++ ++#ifndef KVM_CENSUS_H ++#define KVM_CENSUS_H ++ ++#include ++#include ++#include ++ ++#define CENSUS_ENV "REIMS_VGPU_CENSUS" ++#define CENSUS_MAX_SURFACES 8192 ++#define CENSUS_MAX_VCPUS 64 ++ ++/* What a surface's map tells the census, the descriptor's fields as they came. */ ++typedef struct CensusSurface { ++ uint32_t mapping_id; ++ uint32_t pixel_format; ++ uint64_t width; ++ uint64_t height; ++ uint64_t bytes_per_row; ++ uint64_t alloc_size; ++ uint64_t protection_options; ++ uint32_t address_format; ++ uint32_t compression_type; ++ bool read_only; ++ bool host_only; ++} CensusSurface; ++ ++typedef struct CensusVcpu CensusVcpu; ++ ++/* Reads guest memory for the instruction bytes a record keeps; never armed memory. */ ++typedef void CensusReadGpa(void *opaque, uint64_t gpa, void *buf, unsigned len); ++ ++/* REIMS_VGPU_CENSUS is set, and nothing has refused it yet. */ ++bool census_wanted(void); ++/* The view exists and traps are being counted. */ ++bool census_active(void); ++ ++/* ++ * Maps the KVM-only view of [main, main + len), the mapping of fd at offset, registers it with ++ * userfaultfd and starts the census's thread. On failure, says why on stderr and turns the census ++ * off; memslots then stay on the main mapping and the VM runs as it would without the variable. ++ */ ++bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len); ++/* The address memslots use for main-mapping address ram: the view's when it holds ram. */ ++void *census_hva(void *ram); ++/* Whether addr lies in the view. Async-signal-safe. */ ++bool census_view_holds(const void *addr); ++/* The main mapping's address for a view address, or addr itself. */ ++void *census_main_addr(void *addr); ++ ++CensusVcpu *census_vcpu(int index, int fd, void *opaque, CensusReadGpa *read_gpa); ++ ++enum { CENSUS_NOT_OURS, CENSUS_HANDLED }; ++/* ++ * On the vCPU's thread, outside the signal handler, after KVM_RUN returned. CENSUS_HANDLED: run ++ * the vCPU again. CENSUS_NOT_OURS: not a marker of the census's, as for real poison, and *addr ++ * is the address to go on with, the main mapping's for an address in the view. ++ */ ++int census_on_sigbus(CensusVcpu *vc, void **addr); ++/* KVM_RUN gave a bare -EFAULT: true when the census answers for it, and the vCPU runs again. */ ++bool census_on_efault(CensusVcpu *vc); ++ ++/* The guest's reset: every marker goes. vCPUs are stopped. */ ++void census_reset(void); ++ ++/* From the library's threads. None waits on a vCPU, and presented takes no lock. */ ++void census_surface_mapped(const CensusSurface *facts, const uint64_t *pages, size_t page_count, ++ uint64_t page_size); ++void census_surface_unmapped(uint32_t mapping_id); ++void census_surface_used(uint32_t mapping_id, uint64_t usage, uint64_t storage_mode); ++void census_presented(uint64_t mapping_id); ++ ++/* Writes everything counted so far. For a test: the census's thread does this on its own. */ ++void census_flush(void); ++ ++/* What a test reads. */ ++typedef struct CensusCounts { ++ uint64_t traps, writes, upgrades, spurious, dropped, overlaps, errors, windows, dead; ++} CensusCounts; ++void census_counts(CensusCounts *out); ++/* Runs the end of a window now, whatever the presents count. */ ++void census_close_window(void); ++/* Whether the page at this main-mapping address carries a marker of the census's, by its state. */ ++int census_page_state(const void *main_addr); ++ ++#endif diff --git a/patches/qemu/0002-census-kvm-all-include-qemu-units.h.patch b/patches/qemu/0002-census-kvm-all-include-qemu-units.h.patch new file mode 100644 index 0000000..cf7f572 --- /dev/null +++ b/patches/qemu/0002-census-kvm-all-include-qemu-units.h.patch @@ -0,0 +1,22 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:01:18 +0000 +Subject: [PATCH 2/9] census: kvm-all: include qemu/units.h + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/kvm-all.c | 1 + + 1 file changed, 1 insertion(+) + +diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c +index 77952192cb..7a84a3e33b 100644 +--- a/accel/kvm/kvm-all.c ++++ b/accel/kvm/kvm-all.c +@@ -20,6 +20,7 @@ + #include + + #include "qemu/atomic.h" ++#include "qemu/units.h" + #include "qemu/option.h" + #include "qemu/config-file.h" + #include "qemu/error-report.h" diff --git a/patches/qemu/0003-census-meson-build-census.c-on-any-Linux-host.patch b/patches/qemu/0003-census-meson-build-census.c-on-any-Linux-host.patch new file mode 100644 index 0000000..ee4db1c --- /dev/null +++ b/patches/qemu/0003-census-meson-build-census.c-on-any-Linux-host.patch @@ -0,0 +1,24 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:01:53 +0000 +Subject: [PATCH 3/9] census: meson: build census.c on any Linux host + +CONFIG_LINUX is not a source-set key, so the file was never built. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/meson.build | 4 +++- + 1 file changed, 3 insertions(+), 1 deletion(-) + +diff --git a/accel/kvm/meson.build b/accel/kvm/meson.build +index a154754a04..359ea27f83 100644 +--- a/accel/kvm/meson.build ++++ b/accel/kvm/meson.build +@@ -5,4 +5,6 @@ kvm_ss.add(files( + )) + + specific_ss.add_all(when: 'CONFIG_KVM', if_true: kvm_ss) +-system_ss.add(when: 'CONFIG_LINUX', if_true: files('census.c')) ++if host_os == 'linux' ++ system_ss.add(files('census.c')) ++endif diff --git a/patches/qemu/0004-coherent-pages-on-the-census-s-KVM-only-.patch b/patches/qemu/0004-coherent-pages-on-the-census-s-KVM-only-.patch new file mode 100644 index 0000000..324900b --- /dev/null +++ b/patches/qemu/0004-coherent-pages-on-the-census-s-KVM-only-.patch @@ -0,0 +1,1142 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 01:11:17 +0000 +Subject: [PATCH 4/9] coherent pages on the census's KVM-only + view + +Under REIMS_VGPU_COHERENT=1 the library owns tracked pages' states: held, +armed, open, frozen. Faults reach it through ops, QEMU's own RAM accesses +through census_coherent_access. An arm drops the mapping before the write +protection, so a page POISON refuses may be written, and goes back held and +to the library rather than being skipped. Marks read line by line. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 603 ++++++++++++++++++++++++++++++++++-- + hw/display/apple-gfx.c | 97 ++++++ + include/system/kvm_census.h | 54 ++++ + system/physmem.c | 14 +- + 4 files changed, 739 insertions(+), 29 deletions(-) + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +index 1d77d3521a..3a3740675d 100644 +--- a/accel/kvm/census.c ++++ b/accel/kvm/census.c +@@ -42,6 +42,25 @@ + * A failure must never take the VM with it: an -EFAULT nothing answers disarms the whole census + * (census_fuse) before it is allowed to reach QEMU's fatal path. + * ++ * Under REIMS_VGPU_COHERENT=1 the same view and markers serve metal2vulkan's trap-coherent ++ * surfaces (shadow_lazy.rs and shadow_coherent.rs there), and nothing is sampled. The library then ++ * owns every tracked page's state, and moves it only under its surface's lock: ++ * ++ * held Write-protected: pages and image agree. Reads pass; a write opens the page. ++ * armed Poison: the image is ahead of the page. Any access stops in census_on_sigbus, which ++ * asks the library to sync (ops.fault) and reruns the instruction. ++ * open No marker: the guest's CPU may have written, and the image is stale here. ++ * frozen Poison over an open page, while a sync merges into it. ++ * ++ * A write to a held page gives a bare -EFAULT. Where the vCPU has just trapped on the page, the ++ * pending page names it. Where it has not, ops.probe arms every held page, nothing having changed ++ * in them, and the rerun's SIGBUS gives the exact address: so a write always has its page. ++ * ARMED -> HELD drops the poison and then write-protects, and POISON and WP cannot swap in one ++ * step, so a vCPU can reach the page between them: census_coh_hold reads pagemap afterwards, and ++ * reports a page that came in meanwhile as written, for a copy in of it. QEMU's own accesses (a ++ * device's DMA, KVM's emulator through an MMIO exit) use the main mapping and raise no fault, so ++ * address_space_map and the direct RAM paths call census_coherent_access. ++ * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +@@ -79,6 +98,7 @@ + #define TOP_RIPS 8 + #define RIP_SEEN_SLOTS (1 << 16) + #define POISON_TRIES 4 ++#define FREEZE_TRIES 4096 + #define SPIN_LIMIT 100000 /* about 0.5 s of repeated SIGBUS on one page: real poison */ + /* An -EFAULT unanswered this long is no race: first the markers go, then QEMU's fatal path. */ + #define EFAULT_GAP_NS 5000000 +@@ -86,7 +106,7 @@ + #define FATAL_AFTER_NS 120000000 + + /* A page's word: the state's three bits, then the mapping id. 0 is a page never armed. */ +-enum { ST_NONE, ST_ARMED, ST_HELD, ST_OPEN, ST_SKIP, ST_WORK, ST_STALE }; ++enum { ST_NONE, ST_ARMED, ST_HELD, ST_OPEN, ST_SKIP, ST_WORK, ST_STALE, ST_FROZEN }; + #define PW(st, id) (((uint32_t)(st) << 29) | (id)) + #define PW_ST(w) ((w) >> 29) + #define PW_ID(w) ((w) & 0x1fffffffu) +@@ -113,6 +133,8 @@ struct CensusVcpu { + /* The page this vCPU trapped on last, while its instruction may be rerunning. */ + int64_t pending; + uint64_t pending_rip; ++ /* The instruction a probe arms the held pages for, whose next fault is its write. */ ++ uint64_t probe_rip; + void *spin_addr; + uint32_t spins; + uint64_t efault_first; +@@ -157,6 +179,11 @@ typedef struct Surface { + uint64_t presents_at_tick; + Tally total; + Tally tick; ++ /* Coherent: the pages in the surface's order, and where its texels' bytes start. */ ++ uint32_t *pages; ++ uint64_t page_total; ++ uint64_t view_start; ++ bool tracked; + _Atomic uint64_t presents; + _Atomic uint32_t window_traps; + _Atomic uint32_t saturate; +@@ -196,6 +223,13 @@ static struct { + uint32_t flush_ms; + uint32_t window_ms; + ++ bool coherent; ++ CensusCoherentOps ops; ++ /* A page's index in its surface. */ ++ uint32_t *coh_index; ++ int pagemap; ++ FILE *report; ++ + _Atomic uint64_t presents; + uint64_t presents_at_window; + uint64_t presents_at_tick; +@@ -211,6 +245,8 @@ static struct { + .lock = PTHREAD_MUTEX_INITIALIZER, + }; + ++_Atomic uint64_t census_coherent_pages; ++ + static uint64_t now_ns(void) + { + struct timespec t; +@@ -234,11 +270,17 @@ bool census_wanted(void) + census.wanted = dir && *dir; + if (census.wanted) { + snprintf(census.dir, sizeof(census.dir), "%s", dir); ++ census.coherent = env_u32("REIMS_VGPU_COHERENT", 0); + } + } + return census.wanted; + } + ++bool census_coherent(void) ++{ ++ return census_wanted() && census.coherent; ++} ++ + bool census_active(void) + { + return census.active; +@@ -421,6 +463,70 @@ static void snapshot(CensusVcpu *vc, Rec *r) + } + } + ++static void census_fuse(const char *why); ++static void coherent_report(const char *label); ++ ++/* Where the library knows the page: its address in the surface's own mapping of the texels. */ ++static uint64_t coh_addr(uint32_t id, uint64_t page) ++{ ++ return census.surfaces[id].view_start + ((uint64_t)census.coh_index[page] << PG_SHIFT); ++} ++ ++static uint64_t vcpu_rip(CensusVcpu *vc) ++{ ++ struct kvm_regs regs; ++ ++ return ioctl(vc->fd, KVM_GET_REGS, ®s) == 0 ? regs.rip : 0; ++} ++ ++static int coherent_on_sigbus(CensusVcpu *vc, void **addr, uint64_t page, uint32_t w) ++{ ++ uint32_t st = PW_ST(w), id = PW_ID(w); ++ uint64_t start, rip; ++ bool write; ++ ++ if (st != ST_ARMED && st != ST_FROZEN) { ++ /* A thread moved the page meanwhile, and the access reruns; real poison stays armed. */ ++ if (vc->spin_addr == *addr) { ++ if (++vc->spins > SPIN_LIMIT) { ++ *addr = census.main_map + ((uint8_t *)*addr - census.view); ++ return CENSUS_NOT_OURS; ++ } ++ } else { ++ vc->spin_addr = *addr; ++ vc->spins = 0; ++ } ++ atomic_fetch_add(&census.spurious, 1); ++ return CENSUS_HANDLED; ++ } ++ vc->spin_addr = NULL; ++ start = now_ns(); ++ rip = vcpu_rip(vc); ++ write = vc->probe_rip && vc->probe_rip == rip; ++ vc->probe_rip = 0; ++ census.ops.fault(census.ops.ctx, coh_addr(id, page), write); ++ /* The library answers for its pages; a page still marked here would trap forever. */ ++ w = atomic_load(&census.pg[page]); ++ if (PW_ST(w) == ST_ARMED || PW_ST(w) == ST_FROZEN) { ++ uint64_t at = coh_addr(id, page); ++ ++ if (PW_ST(w) == ST_ARMED) { ++ census_coh_hold(&at, 1, false, NULL, 0); ++ } else { ++ census_coh_thaw(&at, 1); ++ } ++ } ++ vc->pending = write ? -1 : (int64_t)page; ++ vc->pending_rip = rip; ++ vc->efault_first = 0; ++ atomic_fetch_add(&census.traps, 1); ++ if (write) { ++ atomic_fetch_add(&census.writes, 1); ++ } ++ census.ops.timed(census.ops.ctx, coh_addr(id, page), now_ns() - start); ++ return CENSUS_HANDLED; ++} ++ + int census_on_sigbus(CensusVcpu *vc, void **addr) + { + uint64_t off = (uint8_t *)*addr - census.view; +@@ -437,6 +543,9 @@ int census_on_sigbus(CensusVcpu *vc, void **addr) + *addr = census.main_map + off; + return CENSUS_NOT_OURS; + } ++ if (census.coherent) { ++ return coherent_on_sigbus(vc, addr, page, w); ++ } + if (PW_ST(w) != ST_ARMED || + !atomic_compare_exchange_strong(&census.pg[page], &w, PW(ST_WORK, PW_ID(w)))) { + /* +@@ -546,7 +655,7 @@ static void clear_run(uint64_t first, uint64_t count, uint32_t id, bool any_held + } + + #define MASK(st) (1u << (st)) +-#define TOUCHED (MASK(ST_HELD) | MASK(ST_OPEN) | MASK(ST_SKIP)) ++#define TOUCHED (MASK(ST_HELD) | MASK(ST_OPEN) | MASK(ST_SKIP) | MASK(ST_FROZEN)) + + static void census_fuse(const char *why) + { +@@ -554,6 +663,10 @@ static void census_fuse(const char *why) + return; + } + fprintf(stderr, "census: %s: every marker dropped, the census is off\n", why); ++ /* The surfaces' pages catch up first: nothing marks them stale from here on. */ ++ if (census.coherent && census.ops.abandon) { ++ census.ops.abandon(census.ops.ctx); ++ } + for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { + Surface *s = &census.surfaces[id]; + +@@ -563,15 +676,64 @@ static void census_fuse(const char *why) + } + } + ++/* An -EFAULT: is it worth another try? Unanswered, the fuses go as in census_on_efault. */ ++static bool efault_again(CensusVcpu *vc, bool at_once) ++{ ++ uint64_t now = now_ns(); ++ ++ if (!vc->efault_first || now - vc->efault_last > EFAULT_GAP_NS) { ++ vc->efault_first = now; ++ } ++ vc->efault_last = now; ++ if (now - vc->efault_first > FUSE_AFTER_NS) { ++ census_fuse("an -EFAULT no held page explains"); ++ } ++ if (now - vc->efault_first > FATAL_AFTER_NS) { ++ return false; ++ } ++ if (!at_once) { ++ usleep(20); ++ } ++ return true; ++} ++ ++static bool coherent_on_efault(CensusVcpu *vc) ++{ ++ uint64_t rip = vcpu_rip(vc); ++ ++ if (vc->pending >= 0 && vc->pending_rip == rip) { ++ uint64_t page = vc->pending; ++ uint32_t w = atomic_load(&census.pg[page]); ++ ++ vc->pending = -1; ++ if (PW_ST(w) == ST_HELD) { ++ census.ops.fault(census.ops.ctx, coh_addr(PW_ID(w), page), true); ++ atomic_fetch_add(&census.writes, 1); ++ vc->efault_first = 0; ++ return true; ++ } ++ } ++ vc->pending = -1; ++ if (atomic_load(&census_coherent_pages) && !atomic_load(&census.dead)) { ++ vc->probe_rip = rip; ++ atomic_fetch_add(&census.upgrades, 1); ++ census.ops.probe(census.ops.ctx); ++ return efault_again(vc, true); ++ } ++ return efault_again(vc, false); ++} ++ + bool census_on_efault(CensusVcpu *vc) + { + Rec r = { .kind = REC_UPGRADE }; + uint32_t opened = 0; +- uint64_t now; + + if (!census.active || !vc) { + return false; + } ++ if (census.coherent) { ++ return coherent_on_efault(vc); ++ } + snapshot(vc, &r); + if (vc->pending >= 0 && vc->pending_rip == r.rip) { + uint64_t page = vc->pending; +@@ -619,22 +781,331 @@ bool census_on_efault(CensusVcpu *vc) + * Nothing was held: another thread had the page, and the -EFAULT should not return. Should it + * keep returning, something answers it that no write protection explains. + */ +- now = now_ns(); +- if (!vc->efault_first || now - vc->efault_last > EFAULT_GAP_NS) { +- vc->efault_first = now; ++ pthread_mutex_unlock(&census.lock); ++ return efault_again(vc, false); ++} ++ ++/* ---- coherent pages, each under its surface's lock in the library -------------------------- */ ++ ++void census_coherent_set_ops(const CensusCoherentOps *ops) ++{ ++ census.ops = *ops; ++} ++ ++/* The surface whose texels' mapping holds addr, and its page there. */ ++static Surface *coh_surface(uint64_t addr, uint32_t *id, uint64_t *index) ++{ ++ static _Atomic uint32_t last; ++ uint32_t at = atomic_load(&last); ++ ++ for (uint32_t tried = 0; tried < CENSUS_MAX_SURFACES; tried++, at = (at + 1) % CENSUS_MAX_SURFACES) { ++ Surface *s = &census.surfaces[at]; ++ ++ if (s->live && s->pages && addr - s->view_start < s->page_total << PG_SHIFT) { ++ atomic_store(&last, at); ++ *id = at; ++ *index = (addr - s->view_start) >> PG_SHIFT; ++ return s; ++ } + } +- vc->efault_last = now; +- if (now - vc->efault_first > FUSE_AFTER_NS) { +- census_fuse("an -EFAULT no held page explains"); ++ return NULL; ++} ++ ++/* ++ * Calls fn over the runs of contiguous pages of the n addresses' surface, each run of those whose ++ * state is in mask, moved to ST_WORK. Addresses are ascending page addresses of one surface. ++ */ ++static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, void *arg) ++{ ++ uint32_t id = 0; ++ uint64_t index = 0, first = 0, count = 0; ++ Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; ++ bool any_held = false; ++ ++ if (!s) { ++ return; + } +- pthread_mutex_unlock(&census.lock); +- if (now - vc->efault_first > FATAL_AFTER_NS) { ++ for (size_t i = 0; i < n; i++) { ++ uint64_t off = (addrs[i] - s->view_start) >> PG_SHIFT; ++ uint64_t p; ++ uint32_t w; ++ ++ if (off >= s->page_total) { ++ continue; ++ } ++ p = s->pages[off]; ++ w = atomic_load(&census.pg[p]); ++ if (count && first + count != p) { ++ fn(first, count, id, any_held, arg); ++ count = 0; ++ any_held = false; ++ } ++ if (PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { ++ first = count ? first : p; ++ count++; ++ any_held |= PW_ST(w) == ST_HELD; ++ } else if (count) { ++ fn(first, count, id, any_held, arg); ++ count = 0; ++ any_held = false; ++ } ++ } ++ if (count) { ++ fn(first, count, id, any_held, arg); ++ } ++} ++ ++/* Where the pages a vCPU reached as their markers moved go. */ ++typedef struct Came { ++ uint64_t *out; ++ size_t room, found; ++} Came; ++ ++/* ++ * HELD -> ARMED. Poison needs a bare PTE, so the write protection goes first, and a write between ++ * the two would go unseen. So the mapping goes before it, the protection staying behind as a PTE ++ * marker: then a write needs a mapping, and POISON refuses a page mapped. A page it refuses may ++ * hold a write, and it may be what the vCPU's about to read, so there is no retrying it: it goes ++ * back to held, and the library hears, and has what the arm would have left to the page's trap. ++ */ ++static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ Came *refused = arg; ++ uint64_t at = first, end = first + count; ++ int tries = 0; ++ ++ page_drop(first, count); ++ if (page_wp(first, count, false)) { ++ count_error("WP off for arming", first); ++ } ++ while (at < end) { ++ struct uffdio_poison z = { ++ .range = { (uint64_t)census.view + (at << PG_SHIFT), (end - at) << PG_SHIFT }, ++ }; ++ uint64_t done; ++ ++ if (ioctl(census.uffd, UFFDIO_POISON, &z) == 0) { ++ store_pages(at, end - at, ST_ARMED, id); ++ return; ++ } ++ done = z.updated > 0 ? (uint64_t)z.updated >> PG_SHIFT : 0; ++ store_pages(at, done, ST_ARMED, id); ++ at += done; ++ if (at == end) { ++ return; ++ } ++ if (errno == EAGAIN && ++tries < POISON_TRIES) { ++ continue; ++ } ++ if (errno != EEXIST) { ++ count_error("poison", at); ++ } ++ page_drop(at, 1); ++ if (page_wp(at, 1, true)) { ++ count_error("WP on", at); ++ } ++ store_pages(at, 1, ST_HELD, id); ++ if (refused->found < refused->room) { ++ refused->out[refused->found++] = coh_addr(id, at); ++ } ++ at++; ++ tries = 0; ++ } ++} ++ ++/* ++ * Between the poison's going and the write protection a vCPU may have faulted a page in, and ++ * written it: any page the view maps now did so, as the drop left none mapped. ++ */ ++static void came_in(uint64_t first, uint64_t count, uint32_t id, Came *came) ++{ ++ uint64_t bits[512]; ++ ++ for (uint64_t at = first; at < first + count && came->found < came->room;) { ++ uint64_t batch = MIN(first + count - at, sizeof(bits) / sizeof(bits[0])); ++ off_t place = ((uint64_t)census.view >> PG_SHIFT) + at; ++ ssize_t got = pread(census.pagemap, bits, batch * sizeof(bits[0]), place * sizeof(bits[0])); ++ ++ for (ssize_t i = 0; i < got / (ssize_t)sizeof(bits[0]) && came->found < came->room; i++) { ++ if (bits[i] >> 63) { ++ came->out[came->found++] = coh_addr(id, at + i); ++ } ++ } ++ at += batch; ++ } ++} ++ ++static void coh_held_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ /* A page in the run that was open keeps a mapping: it reads and writes as it pleases. */ ++ page_drop(first, count); ++ if (page_wp(first, count, true)) { ++ count_error("WP on", first); ++ } ++ store_pages(first, count, ST_HELD, id); ++ if (arg) { ++ came_in(first, count, id, arg); ++ } ++} ++ ++size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room) ++{ ++ Came found = { refused, room, 0 }; ++ ++ if (!atomic_load(&census.dead)) { ++ coh_runs(addrs, n, MASK(ST_HELD), coh_arm_run, &found); ++ } ++ return found.found; ++} ++ ++size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room) ++{ ++ uint32_t mask = MASK(ST_ARMED) | (open_too ? MASK(ST_OPEN) : 0); ++ Came found = { came, room, 0 }; ++ ++ coh_runs(addrs, n, mask, coh_held_run, came && census.pagemap >= 0 ? &found : NULL); ++ return found.found; ++} ++ ++void census_coh_open(uint64_t addr) ++{ ++ coh_runs(&addr, 1, MASK(ST_HELD), open_held, NULL); ++} ++ ++static void coh_freeze_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ page_drop(first, count); ++ for (uint64_t at = first; at < first + count; at++) { ++ struct uffdio_poison z = { ++ .range = { (uint64_t)census.view + (at << PG_SHIFT), PG }, ++ }; ++ ++ /* A vCPU writing the page wins some of these; a merge needs it out, so it loses in time. */ ++ for (int tries = 0;; tries++) { ++ if (ioctl(census.uffd, UFFDIO_POISON, &z) == 0) { ++ break; ++ } ++ if (tries == FREEZE_TRIES) { ++ count_error("poison to freeze", at); ++ break; ++ } ++ if (errno == EEXIST) { ++ page_drop(at, 1); ++ } ++ } ++ } ++ store_pages(first, count, ST_FROZEN, id); ++} ++ ++static void coh_thaw_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ page_drop(first, count); ++ store_pages(first, count, ST_OPEN, id); ++} ++ ++void census_coh_freeze(const uint64_t *addrs, size_t n) ++{ ++ coh_runs(addrs, n, MASK(ST_OPEN), coh_freeze_run, NULL); ++} ++ ++void census_coh_thaw(const uint64_t *addrs, size_t n) ++{ ++ coh_runs(addrs, n, MASK(ST_FROZEN), coh_thaw_run, NULL); ++} ++ ++bool census_coh_track(uint64_t start, uint64_t len) ++{ ++ uint32_t id; ++ uint64_t index; ++ Surface *s; ++ ++ if (!census.active || !census.coherent || atomic_load(&census.dead)) { + return false; + } +- usleep(20); ++ pthread_mutex_lock(&census.lock); ++ s = coh_surface(start, &id, &index); ++ if (!s || s->overlaps || s->foreign) { ++ pthread_mutex_unlock(&census.lock); ++ return false; ++ } ++ if (!s->tracked) { ++ s->tracked = true; ++ atomic_fetch_add(&census_coherent_pages, s->page_total); ++ take_pages(s, id, MASK(ST_OPEN), true, coh_held_run, NULL); ++ } ++ pthread_mutex_unlock(&census.lock); + return true; + } + ++/* Every marker off the surface, its pages left open. */ ++static void coh_free_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++{ ++ if (any_held) { ++ page_wp(first, count, false); ++ } ++ page_drop(first, count); ++ store_pages(first, count, ST_OPEN, id); ++} ++ ++static void coh_untrack_locked(Surface *s, uint32_t id) ++{ ++ if (s->tracked) { ++ s->tracked = false; ++ atomic_fetch_sub(&census_coherent_pages, s->page_total); ++ take_pages(s, id, MASK(ST_ARMED) | MASK(ST_HELD) | MASK(ST_FROZEN), true, coh_free_run, ++ NULL); ++ } ++} ++ ++void census_coh_untrack(uint64_t start) ++{ ++ uint32_t id; ++ uint64_t index; ++ Surface *s; ++ ++ if (!census.active || !census.coherent) { ++ return; ++ } ++ pthread_mutex_lock(&census.lock); ++ s = coh_surface(start, &id, &index); ++ if (s) { ++ coh_untrack_locked(s, id); ++ } ++ pthread_mutex_unlock(&census.lock); ++} ++ ++bool census_coh_surface(uint32_t mapping_id, uint64_t *start, uint64_t *len) ++{ ++ Surface *s; ++ ++ if (!census.active || !census.coherent || mapping_id == 0 || ++ mapping_id >= CENSUS_MAX_SURFACES || !census.surfaces[mapping_id].live) { ++ return false; ++ } ++ s = &census.surfaces[mapping_id]; ++ *start = s->view_start; ++ *len = s->page_total << PG_SHIFT; ++ return s->tracked; ++} ++ ++void census_coherent_touch(void *main_ptr, uint64_t len, bool write) ++{ ++ uint64_t off = (uint8_t *)main_ptr - census.main_map; ++ ++ if (off >= census.len || !len) { ++ return; ++ } ++ for (uint64_t p = off >> PG_SHIFT; p <= (off + len - 1) >> PG_SHIFT && p < census.pages; p++) { ++ uint32_t w = atomic_load(&census.pg[p]), st = PW_ST(w), id = PW_ID(w); ++ ++ if (w && census.surfaces[id].tracked && ++ (st == ST_ARMED || st == ST_FROZEN || (write && st == ST_HELD))) { ++ census.ops.fault(census.ops.ctx, coh_addr(id, p), write); ++ } ++ } ++} ++ + /* ---- surfaces, from the library's threads ------------------------------------------------- */ + + static void queue_line(const char *format, ...) +@@ -775,12 +1246,15 @@ static void surface_end(Surface *s, uint32_t id, const char *event) + static void surface_close(Surface *s, uint32_t id, const char *event) + { + drain_locked(); ++ coh_untrack_locked(s, id); + take_pages(s, id, MASK(ST_ARMED) | TOUCHED, true, clear_run, NULL); + surface_end(s, id, event); + s->live = false; + free(s->runs); + s->runs = NULL; + s->run_count = 0; ++ free(s->pages); ++ s->pages = NULL; + } + + void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t page_count, +@@ -800,6 +1274,13 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + if (s->live) { + surface_close(s, id, "replaced"); + } ++ s->pages = NULL; ++ s->page_total = 0; ++ s->tracked = false; ++ if (census.coherent && page_count) { ++ s->pages = calloc(page_count, sizeof(*s->pages)); ++ s->view_start = f->base_address - f->offset; ++ } + atomic_store(&s->presents, 0); + atomic_store(&s->window_traps, 0); + atomic_store(&s->saturate, 0); +@@ -830,6 +1311,10 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + census.overlaps++; + continue; + } ++ if (s->pages) { ++ census.coh_index[p] = s->page_total; ++ s->pages[s->page_total++] = p; ++ } + if (s->run_count && s->runs[s->run_count - 1].first + + s->runs[s->run_count - 1].count == p) { + s->runs[s->run_count - 1].count++; +@@ -846,7 +1331,12 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + } + } + for (uint32_t i = 0; i < s->run_count; i++) { +- arm_run(s->runs[i].first, s->runs[i].count, id, false); ++ if (census.coherent) { ++ /* Nothing on it until the library tracks the surface. */ ++ store_pages(s->runs[i].first, s->runs[i].count, ST_OPEN, id); ++ } else { ++ arm_run(s->runs[i].first, s->runs[i].count, id, false); ++ } + } + queue_line("{\"ev\":\"map\",\"t\":%.3f,\"s\":%u,\"id\":%u,\"format\":\"%#x\",\"w\":%" PRIu64 + ",\"h\":%" PRIu64 ",\"row\":%" PRIu64 ",\"alloc\":%" PRIu64 ",\"protection\":%" +@@ -907,6 +1397,9 @@ void census_reset(void) + if (!census.active) { + return; + } ++ if (census.coherent && census.ops.reset) { ++ census.ops.reset(census.ops.ctx); ++ } + pthread_mutex_lock(&census.lock); + for (uint32_t id = 1; id < CENSUS_MAX_SURFACES; id++) { + if (census.surfaces[id].live) { +@@ -916,6 +1409,7 @@ void census_reset(void) + for (int i = 0; i < CENSUS_MAX_VCPUS; i++) { + if (census.vcpus[i]) { + census.vcpus[i]->pending = -1; ++ census.vcpus[i]->probe_rip = 0; + } + } + pthread_mutex_unlock(&census.lock); +@@ -1082,24 +1576,52 @@ static void write_lines(void) + fflush(census.rips); + } + +-/* /mark: "every N" sets the re-arm period; anything else is a label in the timeline. */ ++/* One line of coherent.jsonl per surface, what the library counts, under the label. */ ++static void coherent_report(const char *label) ++{ ++ static char text[1 << 18]; ++ size_t n; ++ ++ if (!census.coherent || !census.ops.report || !census.report) { ++ return; ++ } ++ n = census.ops.report(census.ops.ctx, label, text, sizeof(text)); ++ fwrite(text, 1, MIN(n, sizeof(text)), census.report); ++ fflush(census.report); ++} ++ ++static void read_mark_line(char *text); ++ ++/* ++ * /mark: each line, in order. "every N" sets the re-arm period; anything else is a label in ++ * the timeline. Append with >>: a write over a line the census has not read yet replaces it. ++ */ + static void read_mark(void) + { +- char path[sizeof(census.dir) + 8], text[128] = ""; ++ char path[sizeof(census.dir) + 8], moved[sizeof(path) + 8], text[128]; + FILE *f; +- size_t n; +- unsigned long every; + + snprintf(path, sizeof(path), "%s/mark", census.dir); +- f = fopen(path, "r"); ++ snprintf(moved, sizeof(moved), "%s.read", path); ++ if (rename(path, moved)) { ++ return; ++ } ++ f = fopen(moved, "r"); + if (!f) { + return; + } +- n = fread(text, 1, sizeof(text) - 1, f); ++ while (fgets(text, sizeof(text), f)) { ++ text[strcspn(text, "\n")] = 0; ++ read_mark_line(text); ++ } + fclose(f); +- unlink(path); +- text[n] = 0; +- text[strcspn(text, "\n")] = 0; ++ unlink(moved); ++} ++ ++static void read_mark_line(char *text) ++{ ++ unsigned long every; ++ + pthread_mutex_lock(&census.lock); + if (sscanf(text, "every %lu", &every) == 1 && every > 0) { + atomic_store(&census.every, every); +@@ -1110,6 +1632,9 @@ static void read_mark(void) + } + tick_locked(); + queue_line("{\"mark\":\"%s\",\"t\":%.3f}\n", text, census_time()); ++ pthread_mutex_unlock(&census.lock); ++ coherent_report(text); ++ return; + } + pthread_mutex_unlock(&census.lock); + } +@@ -1122,6 +1647,8 @@ static void *census_thread(void *unused) + pthread_sigmask(SIG_BLOCK, &all, NULL); + pthread_setname_np(pthread_self(), "reims-census"); + census.window_ns = census.tick_ns = now_ns(); ++ bool report_due = false; ++ + for (;;) { + struct timespec until; + uint64_t now, every = atomic_load(&census.every); +@@ -1137,17 +1664,25 @@ static void *census_thread(void *unused) + read_mark(); + pthread_mutex_lock(&census.lock); + drain_locked(); +- saturate_locked(); +- if (atomic_load(&census.presents) - census.presents_at_window >= every || +- now - census.window_ns >= census.window_ms * 1000000ULL) { ++ if (!census.coherent) { ++ saturate_locked(); ++ } ++ if (!census.coherent && ++ (atomic_load(&census.presents) - census.presents_at_window >= every || ++ now - census.window_ns >= census.window_ms * 1000000ULL)) { + close_window_locked(); + } + if (now - census.tick_ns >= census.flush_ms * 1000000ULL) { + tick_locked(); + census.tick_ns = now; ++ report_due = true; + } + pthread_mutex_unlock(&census.lock); + write_lines(); ++ if (report_due) { ++ coherent_report(""); ++ report_due = false; ++ } + } + return unused; + } +@@ -1166,6 +1701,7 @@ void census_flush(void) + static void census_at_exit(void) + { + census_flush(); ++ coherent_report("exit"); + } + + void census_counts(CensusCounts *out) +@@ -1183,6 +1719,11 @@ void census_counts(CensusCounts *out) + }; + } + ++void census_fuse_test(void) ++{ ++ census_fuse("a test"); ++} ++ + int census_page_state(const void *main_addr) + { + uint64_t off = (const uint8_t *)main_addr - census.main_map; +@@ -1226,6 +1767,7 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + census.ticks = open_output("ticks.jsonl"); + census.facts = open_output("surfaces.jsonl"); + census.rips = open_output("rips.jsonl"); ++ census.report = census.coherent ? open_output("coherent.jsonl") : NULL; + if (!census.ticks || !census.facts || !census.rips) { + return census_refuse("the output directory"); + } +@@ -1252,9 +1794,14 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + census.len = len; + census.pages = len >> PG_SHIFT; + census.pg = calloc(census.pages, sizeof(*census.pg)); ++ census.pagemap = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); ++ if (census.coherent) { ++ census.coh_index = calloc(census.pages, sizeof(*census.coh_index)); ++ } + census.surfaces = calloc(CENSUS_MAX_SURFACES, sizeof(*census.surfaces)); + census.rips_seen = calloc(RIP_SEEN_SLOTS, sizeof(*census.rips_seen)); +- if (!census.pg || !census.surfaces || !census.rips_seen) { ++ if (!census.pg || !census.surfaces || !census.rips_seen || ++ (census.coherent && !census.coh_index)) { + return census_refuse("memory"); + } + census.lines_tail = &census.lines_head; +@@ -1277,7 +1824,7 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + queue_line("{\"started\":%.3f,\"pid\":%d,\"bytes\":%" PRIu64 ",\"rw\":%d,\"every\":%u," + "\"min_pixels\":%u,\"saturate\":%u}\n", census_time(), getpid(), len, census.rw, + census.every, census.min_pixels, census.saturate_at); +- fprintf(stderr, "census: on, writing to %s: %" PRIu64 " MiB view at %p\n", census.dir, +- len >> 20, census.view); ++ fprintf(stderr, "census: %s, writing to %s: %" PRIu64 " MiB view at %p\n", ++ census.coherent ? "coherent" : "on", census.dir, len >> 20, census.view); + return true; + } +diff --git a/hw/display/apple-gfx.c b/hw/display/apple-gfx.c +index 935c5850d1..84df348295 100644 +--- a/hw/display/apple-gfx.c ++++ b/hw/display/apple-gfx.c +@@ -1026,6 +1026,8 @@ static void apple_gfx_census_mapped(void *ctx, const RvgCensusSurface *surface, + .compression_type = surface->compression_type, + .read_only = surface->read_only, + .host_only = surface->host_only, ++ .base_address = surface->base_address, ++ .offset = surface->base_offset, + }; + + census_surface_mapped(&facts, pages, page_count, page_size); +@@ -1033,6 +1035,11 @@ static void apple_gfx_census_mapped(void *ctx, const RvgCensusSurface *surface, + + static void apple_gfx_census_unmapped(void *ctx, uint32_t mapping_id) + { ++ uint64_t start, len; ++ ++ if (census_coh_surface(mapping_id, &start, &len)) { ++ rvg_coherent_unmapped(start, len); ++ } + census_surface_unmapped(mapping_id); + } + +@@ -1047,6 +1054,73 @@ static void apple_gfx_census_presented(void *ctx, uint64_t mapping_id) + census_presented(mapping_id); + } + ++static bool apple_gfx_coh_track(void *ctx, uint64_t start, uint64_t len) ++{ ++ return census_coh_track(start, len); ++} ++ ++static void apple_gfx_coh_untrack(void *ctx, uint64_t start) ++{ ++ census_coh_untrack(start); ++} ++ ++static size_t apple_gfx_coh_arm(void *ctx, const uint64_t *addrs, size_t n, uint64_t *refused, ++ size_t room) ++{ ++ return census_coh_arm(addrs, n, refused, room); ++} ++ ++static size_t apple_gfx_coh_hold(void *ctx, const uint64_t *addrs, size_t n, bool open_too, ++ uint64_t *came, size_t room) ++{ ++ return census_coh_hold(addrs, n, open_too, came, room); ++} ++ ++static void apple_gfx_coh_open(void *ctx, uint64_t addr) ++{ ++ census_coh_open(addr); ++} ++ ++static void apple_gfx_coh_freeze(void *ctx, const uint64_t *addrs, size_t n) ++{ ++ census_coh_freeze(addrs, n); ++} ++ ++static void apple_gfx_coh_thaw(void *ctx, const uint64_t *addrs, size_t n) ++{ ++ census_coh_thaw(addrs, n); ++} ++ ++static void apple_gfx_coh_fault(void *ctx, uint64_t addr, bool write) ++{ ++ rvg_coherent_fault(addr, write); ++} ++ ++static void apple_gfx_coh_timed(void *ctx, uint64_t addr, uint64_t ns) ++{ ++ rvg_coherent_timed(addr, ns); ++} ++ ++static void apple_gfx_coh_probe(void *ctx) ++{ ++ rvg_coherent_probe(); ++} ++ ++static void apple_gfx_coh_abandon(void *ctx) ++{ ++ rvg_coherent_abandon(); ++} ++ ++static void apple_gfx_coh_reset(void *ctx) ++{ ++ rvg_coherent_reset(); ++} ++ ++static size_t apple_gfx_coh_report(void *ctx, const char *label, char *out, size_t room) ++{ ++ return rvg_coherent_report(label, out, room); ++} ++ + static void apple_gfx_census_register(void) + { + static const RvgCensusCallbacks callbacks = { +@@ -1057,6 +1131,29 @@ static void apple_gfx_census_register(void) + .presented = apple_gfx_census_presented, + }; + ++ if (census_coherent() && census_active()) { ++ static const CensusCoherentOps ops = { ++ .fault = apple_gfx_coh_fault, ++ .timed = apple_gfx_coh_timed, ++ .probe = apple_gfx_coh_probe, ++ .abandon = apple_gfx_coh_abandon, ++ .reset = apple_gfx_coh_reset, ++ .report = apple_gfx_coh_report, ++ }; ++ static const RvgCoherentHooks hooks = { ++ .struct_size = sizeof(hooks), ++ .track = apple_gfx_coh_track, ++ .untrack = apple_gfx_coh_untrack, ++ .arm = apple_gfx_coh_arm, ++ .hold = apple_gfx_coh_hold, ++ .open = apple_gfx_coh_open, ++ .freeze = apple_gfx_coh_freeze, ++ .thaw = apple_gfx_coh_thaw, ++ }; ++ ++ census_coherent_set_ops(&ops); ++ rvg_coherent_set_hooks(&hooks); ++ } + if (census_active()) { + rvg_census_set_callbacks(&callbacks); + } +diff --git a/include/system/kvm_census.h b/include/system/kvm_census.h +index e242236452..3673e4286e 100644 +--- a/include/system/kvm_census.h ++++ b/include/system/kvm_census.h +@@ -7,6 +7,7 @@ + #ifndef KVM_CENSUS_H + #define KVM_CENSUS_H + ++#include + #include + #include + #include +@@ -28,6 +29,9 @@ typedef struct CensusSurface { + uint32_t compression_type; + bool read_only; + bool host_only; ++ /* Where the library's mapping of the surface's bytes starts, and base's offset in it. */ ++ uint64_t base_address; ++ uint64_t offset; + } CensusSurface; + + typedef struct CensusVcpu CensusVcpu; +@@ -37,6 +41,8 @@ typedef void CensusReadGpa(void *opaque, uint64_t gpa, void *buf, unsigned len); + + /* REIMS_VGPU_CENSUS is set, and nothing has refused it yet. */ + bool census_wanted(void); ++/* With it, REIMS_VGPU_COHERENT: the view serves trap-coherent surfaces, and nothing is sampled. */ ++bool census_coherent(void); + /* The view exists and traps are being counted. */ + bool census_active(void); + +@@ -75,6 +81,52 @@ void census_surface_unmapped(uint32_t mapping_id); + void census_surface_used(uint32_t mapping_id, uint64_t usage, uint64_t storage_mode); + void census_presented(uint64_t mapping_id); + ++/* ++ * Trap-coherent surfaces (census.c, and shadow_coherent.rs in metal2vulkan). Addresses are of the ++ * library's mapping of the surface, whole pages, ascending, one surface to a call. The census_coh_ ++ * calls come only under the surface's lock in the library; fault takes that lock, so nothing here ++ * holds a lock across fault. ++ */ ++typedef struct CensusCoherentOps { ++ void *ctx; ++ /* A page is touched: sync it, and open it for a write. Waits for no GPU work it does not need. */ ++ void (*fault)(void *ctx, uint64_t addr, bool write); ++ /* Trap time, from the access's stop to its rerun. */ ++ void (*timed)(void *ctx, uint64_t addr, uint64_t ns); ++ /* A write hit a held page unnamed: arm every held page, nothing changed in them. */ ++ void (*probe)(void *ctx); ++ /* Every surface catches up and stops being tracked: the markers go now. */ ++ void (*abandon)(void *ctx); ++ /* The guest resets: every surface is forgotten, its pages not written. */ ++ void (*reset)(void *ctx); ++ /* One line a surface, under label: the bytes they took. */ ++ size_t (*report)(void *ctx, const char *label, char *out, size_t room); ++} CensusCoherentOps; ++ ++void census_coherent_set_ops(const CensusCoherentOps *ops); ++bool census_coh_track(uint64_t start, uint64_t len); ++void census_coh_untrack(uint64_t start); ++/* Held pages armed. Into up to `room` of `refused`, those a vCPU reached: held, and how many. */ ++size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room); ++/* Into up to room of came, the pages a vCPU reached as these moved: written, for all it knows. */ ++size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room); ++void census_coh_open(uint64_t addr); ++void census_coh_freeze(const uint64_t *addrs, size_t n); ++void census_coh_thaw(const uint64_t *addrs, size_t n); ++/* The surface of a mapping id, and whether the library tracks it. */ ++bool census_coh_surface(uint32_t mapping_id, uint64_t *start, uint64_t *len); ++ ++/* The pages tracked now: 0 while no surface is. */ ++extern _Atomic uint64_t census_coherent_pages; ++void census_coherent_touch(void *main_ptr, uint64_t len, bool write); ++/* QEMU's own access to guest RAM through the main mapping, on any thread. */ ++static inline void census_coherent_access(void *main_ptr, uint64_t len, bool write) ++{ ++ if (__atomic_load_n((uint64_t *)&census_coherent_pages, __ATOMIC_RELAXED)) { ++ census_coherent_touch(main_ptr, len, write); ++ } ++} ++ + /* Writes everything counted so far. For a test: the census's thread does this on its own. */ + void census_flush(void); + +@@ -85,6 +137,8 @@ typedef struct CensusCounts { + void census_counts(CensusCounts *out); + /* Runs the end of a window now, whatever the presents count. */ + void census_close_window(void); ++/* The fuse, for a test. */ ++void census_fuse_test(void); + /* Whether the page at this main-mapping address carries a marker of the census's, by its state. */ + int census_page_state(const void *main_addr); + +diff --git a/system/physmem.c b/system/physmem.c +index bd5f41cb02..c4f9363775 100644 +--- a/system/physmem.c ++++ b/system/physmem.c +@@ -44,6 +44,12 @@ + #include "system/tcg.h" + #include "system/qtest.h" + #include "system/physmem.h" ++#ifdef CONFIG_LINUX ++#include "system/kvm_census.h" ++#define census_ram_access census_coherent_access ++#else ++#define census_ram_access(ptr, len, write) ++#endif + #include "system/ramblock.h" + #include "qemu/timer.h" + #include "qemu/config-file.h" +@@ -3314,6 +3320,7 @@ static MemTxResult flatview_write_continue_step(MemTxAttrs attrs, + uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l, + false, true); + ++ census_ram_access(ram_ptr, *l, true); + qemu_ram_move(ram_ptr, buf, *l); + invalidate_and_set_dirty(mr, mr_addr, *l); + +@@ -3407,6 +3414,7 @@ static MemTxResult flatview_read_continue_step(MemTxAttrs attrs, uint8_t *buf, + uint8_t *ram_ptr = qemu_ram_ptr_length(mr->ram_block, mr_addr, l, + false, false); + ++ census_ram_access(ram_ptr, *l, false); + qemu_ram_move(buf, ram_ptr, *l); + + return MEMTX_OK; +@@ -3807,7 +3815,11 @@ void *address_space_map(AddressSpace *as, + *plen = flatview_extend_translation(fv, addr, len, mr, xlat, + l, is_write, attrs); + fuzz_dma_read_cb(addr, *plen, mr); +- return qemu_ram_ptr_length(mr->ram_block, xlat, plen, true, is_write); ++ void *mapped = qemu_ram_ptr_length(mr->ram_block, xlat, plen, true, ++ is_write); ++ /* A device reads or fills the pages through this with no further word. */ ++ census_ram_access(mapped, *plen, is_write); ++ return mapped; + } + + /* Unmaps a memory region previously mapped by address_space_map(). diff --git a/patches/qemu/0005-coherent-a-stale-page-is-no-surface-s-an.patch b/patches/qemu/0005-coherent-a-stale-page-is-no-surface-s-an.patch new file mode 100644 index 0000000..ff6a889 --- /dev/null +++ b/patches/qemu/0005-coherent-a-stale-page-is-no-surface-s-an.patch @@ -0,0 +1,214 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 02:23:01 +0000 +Subject: [PATCH 5/9] coherent: a stale page is no surface's, + and a crash stops at one + +census_coherent_touch read the surface of a word before its state, and a stale +page's id is a sentinel: any device access to a page of a surface that had gone +crashed QEMU once anything was tracked. It also now waits for markers in motion, +and address_space_unmap reports a write finished after a page was held again. + +The view is the guest's RAM, so it stays out of a core, and a core is capped at +8 GiB. coherent.running in the census's directory survives a crash, a kill or the +fuse, and until it is removed later starts run with no coherent pages. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 86 +++++++++++++++++++++++++++++++++++++++--- + hw/display/apple-gfx.c | 4 ++ + system/physmem.c | 2 + + 3 files changed, 87 insertions(+), 5 deletions(-) + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +index 3a3740675d..fb34f6f247 100644 +--- a/accel/kvm/census.c ++++ b/accel/kvm/census.c +@@ -80,6 +80,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -99,6 +100,8 @@ + #define RIP_SEEN_SLOTS (1 << 16) + #define POISON_TRIES 4 + #define FREEZE_TRIES 4096 ++#define WORK_WAIT_NS 1000000000 ++#define CORE_LIMIT (8ULL << 30) + #define SPIN_LIMIT 100000 /* about 0.5 s of repeated SIGBUS on one page: real poison */ + /* An -EFAULT unanswered this long is no race: first the markers go, then QEMU's fatal path. */ + #define EFAULT_GAP_NS 5000000 +@@ -199,6 +202,9 @@ static struct { + bool active; + _Atomic bool dead; + char dir[3584]; ++ char guard[3584 + 32]; ++ /* QEMU's own accesses, by the state of the page. */ ++ _Atomic uint64_t touches[8]; + uint8_t *view; + uint8_t *main_map; + uint64_t len; +@@ -1097,10 +1103,23 @@ void census_coherent_touch(void *main_ptr, uint64_t len, bool write) + return; + } + for (uint64_t p = off >> PG_SHIFT; p <= (off + len - 1) >> PG_SHIFT && p < census.pages; p++) { +- uint32_t w = atomic_load(&census.pg[p]), st = PW_ST(w), id = PW_ID(w); ++ uint32_t w = atomic_load(&census.pg[p]), st, id; + +- if (w && census.surfaces[id].tracked && +- (st == ST_ARMED || st == ST_FROZEN || (write && st == ST_HELD))) { ++ /* Markers moving: this access uses the main mapping, which no marker stops, so it waits. */ ++ for (uint64_t start = 0; PW_ST(w) == ST_WORK; w = atomic_load(&census.pg[p])) { ++ start = start ? start : now_ns(); ++ if (now_ns() - start > WORK_WAIT_NS) { ++ count_error("waiting for a page's markers", p); ++ break; ++ } ++ sched_yield(); ++ } ++ st = PW_ST(w); ++ id = PW_ID(w); ++ atomic_fetch_add(&census.touches[st], 1); ++ /* Only these states hold a surface's id: a stale page's is not one. */ ++ if ((st == ST_ARMED || st == ST_FROZEN || (write && st == ST_HELD)) && ++ census.surfaces[id].tracked) { + census.ops.fault(census.ops.ctx, coh_addr(id, p), write); + } + } +@@ -1587,6 +1606,11 @@ static void coherent_report(const char *label) + } + n = census.ops.report(census.ops.ctx, label, text, sizeof(text)); + fwrite(text, 1, MIN(n, sizeof(text)), census.report); ++ fprintf(census.report, "{\"touches\":\"%s\",\"open\":%" PRIu64 ",\"armed\":%" PRIu64 ++ ",\"held\":%" PRIu64 ",\"frozen\":%" PRIu64 ",\"stale\":%" PRIu64 "}\n", label, ++ atomic_load(&census.touches[ST_OPEN]), atomic_load(&census.touches[ST_ARMED]), ++ atomic_load(&census.touches[ST_HELD]), atomic_load(&census.touches[ST_FROZEN]), ++ atomic_load(&census.touches[ST_STALE])); + fflush(census.report); + } + +@@ -1698,10 +1722,53 @@ void census_flush(void) + write_lines(); + } + +-static void census_at_exit(void) ++static void census_at_exit(int status, void *arg) + { + census_flush(); + coherent_report("exit"); ++ fprintf(stderr, "census: touches by QEMU: open %" PRIu64 " armed %" PRIu64 " held %" PRIu64 ++ " frozen %" PRIu64 " stale %" PRIu64 " none %" PRIu64 "\n", ++ atomic_load(&census.touches[ST_OPEN]), atomic_load(&census.touches[ST_ARMED]), ++ atomic_load(&census.touches[ST_HELD]), atomic_load(&census.touches[ST_FROZEN]), ++ atomic_load(&census.touches[ST_STALE]), atomic_load(&census.touches[ST_NONE])); ++ /* Only an exit nothing went wrong in lets the next start use the markers. */ ++ if (census.coherent && status == 0 && !atomic_load(&census.dead)) { ++ unlink(census.guard); ++ } ++} ++ ++/* ++ * A coherent run leaves this file until it exits well: a crash, a kill or the fuse keep it, and ++ * every later start refuses the markers, so that the watchdog's restarts boot the plain VM and a ++ * fault here costs one crash. Removing the file allows the next try. It costs a clean stop that ++ * ended in SIGKILL one plain boot. A crash's core stays bounded too: the view holds no guest RAM ++ * for it, and the limit covers what grows beside it. ++ */ ++static bool coherent_guard(void) ++{ ++ struct rlimit core; ++ char text[64]; ++ int fd, n; ++ ++ snprintf(census.guard, sizeof(census.guard), "%s/coherent.running", census.dir); ++ fd = open(census.guard, O_CREAT | O_EXCL | O_WRONLY | O_CLOEXEC, 0666); ++ if (fd < 0) { ++ fprintf(stderr, "census: %s: %s; the last coherent run did not end well. Starting " ++ "without coherent pages until that file is removed\n", census.guard, ++ errno == EEXIST ? "present" : strerror(errno)); ++ return false; ++ } ++ n = snprintf(text, sizeof(text), "pid %d at %.0f\n", getpid(), census_time()); ++ if (write(fd, text, n) != n || fsync(fd)) { ++ close(fd); ++ return false; ++ } ++ close(fd); ++ if (getrlimit(RLIMIT_CORE, &core) == 0 && core.rlim_cur > CORE_LIMIT) { ++ core.rlim_cur = CORE_LIMIT; ++ setrlimit(RLIMIT_CORE, &core); ++ } ++ return true; + } + + void census_counts(CensusCounts *out) +@@ -1743,6 +1810,9 @@ static bool census_refuse(const char *what) + { + fprintf(stderr, "census: %s: %s; the census is off\n", what, strerror(errno)); + census.wanted = 0; ++ if (census.guard[0]) { ++ unlink(census.guard); ++ } + if (census.view) { + munmap(census.view, census.len); + census.view = NULL; +@@ -1764,6 +1834,10 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + struct uffdio_register reg = { .mode = UFFDIO_REGISTER_MODE_WP }; + + mkdir(census.dir, 0777); ++ if (census.coherent && !coherent_guard()) { ++ census.wanted = 0; ++ return false; ++ } + census.ticks = open_output("ticks.jsonl"); + census.facts = open_output("surfaces.jsonl"); + census.rips = open_output("rips.jsonl"); +@@ -1785,6 +1859,8 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + } + /* A fork's child, which only execs, needs none of this. */ + madvise(census.view, len, MADV_DONTFORK); ++ /* It is the guest's RAM again, which dump-guest-core=off keeps out of a core. */ ++ madvise(census.view, len, MADV_DONTDUMP); + reg.range.start = (uint64_t)census.view; + reg.range.len = len; + if (ioctl(census.uffd, UFFDIO_REGISTER, ®)) { +@@ -1820,7 +1896,7 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + census.active = false; + return census_refuse("the census's thread"); + } +- atexit(census_at_exit); ++ on_exit(census_at_exit, NULL); + queue_line("{\"started\":%.3f,\"pid\":%d,\"bytes\":%" PRIu64 ",\"rw\":%d,\"every\":%u," + "\"min_pixels\":%u,\"saturate\":%u}\n", census_time(), getpid(), len, census.rw, + census.every, census.min_pixels, census.saturate_at); +diff --git a/hw/display/apple-gfx.c b/hw/display/apple-gfx.c +index 84df348295..4d8610c48c 100644 +--- a/hw/display/apple-gfx.c ++++ b/hw/display/apple-gfx.c +@@ -1031,6 +1031,10 @@ static void apple_gfx_census_mapped(void *ctx, const RvgCensusSurface *surface, + }; + + census_surface_mapped(&facts, pages, page_count, page_size); ++ if (census_coherent() && page_count) { ++ rvg_coherent_rehearse(surface->base_address, surface->width, surface->height, ++ surface->bytes_per_row); ++ } + } + + static void apple_gfx_census_unmapped(void *ctx, uint32_t mapping_id) +diff --git a/system/physmem.c b/system/physmem.c +index c4f9363775..146f14163f 100644 +--- a/system/physmem.c ++++ b/system/physmem.c +@@ -3836,6 +3836,8 @@ void address_space_unmap(AddressSpace *as, void *buffer, hwaddr len, + if (mr != NULL) { + if (is_write) { + invalidate_and_set_dirty(mr, addr1, access_len); ++ /* Its page may have been held again since the map: the image takes this write. */ ++ census_ram_access(buffer, access_len, true); + } + if (xen_map_cache_enabled()) { + xen_invalidate_map_cache_entry(buffer); diff --git a/patches/qemu/0006-census-a-surface-s-marks-move-in-runs-of-the-view.patch b/patches/qemu/0006-census-a-surface-s-marks-move-in-runs-of-the-view.patch new file mode 100644 index 0000000..aabe0c0 --- /dev/null +++ b/patches/qemu/0006-census-a-surface-s-marks-move-in-runs-of-the-view.patch @@ -0,0 +1,212 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 03:50:10 +0000 +Subject: [PATCH 6/9] census: a surface's marks move in runs of the view + +A guest's allocator hands a surface's pages out mostly descending, so in the surface's order each +page was a run of its own, three syscalls each. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 103 +++++++++++++++++++++++++++--------- + include/system/kvm_census.h | 4 +- + 2 files changed, 80 insertions(+), 27 deletions(-) + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +index fb34f6f247..28b3d2df95 100644 +--- a/accel/kvm/census.c ++++ b/accel/kvm/census.c +@@ -186,6 +186,8 @@ typedef struct Surface { + uint32_t *pages; + uint64_t page_total; + uint64_t view_start; ++ /* Which of those pages a call means; only the call under the surface's lock uses it. */ ++ uint64_t *wanted; + bool tracked; + _Atomic uint64_t presents; + _Atomic uint32_t window_traps; +@@ -205,6 +207,8 @@ static struct { + char guard[3584 + 32]; + /* QEMU's own accesses, by the state of the page. */ + _Atomic uint64_t touches[8]; ++ /* Runs the marks moved in, each its syscalls: what a sync's hold and an arm cost. */ ++ _Atomic uint64_t coh_runs; + uint8_t *view; + uint8_t *main_map; + uint64_t len; +@@ -817,9 +821,21 @@ static Surface *coh_surface(uint64_t addr, uint32_t *id, uint64_t *index) + return NULL; + } + ++static void run_ends(RunFn *fn, uint64_t first, uint64_t *count, uint32_t id, bool *any_held, void *arg) ++{ ++ if (*count) { ++ fn(first, *count, id, *any_held, arg); ++ atomic_fetch_add(&census.coh_runs, 1); ++ *count = 0; ++ *any_held = false; ++ } ++} ++ + /* + * Calls fn over the runs of contiguous pages of the n addresses' surface, each run of those whose +- * state is in mask, moved to ST_WORK. Addresses are ascending page addresses of one surface. ++ * state is in mask, moved to ST_WORK. Addresses are page addresses of one surface. The runs follow ++ * the view, not the surface: a guest's allocator hands a surface's pages out mostly descending, so ++ * in the surface's order every page would be a run of its own, three syscalls each. + */ + static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, void *arg) + { +@@ -833,33 +849,32 @@ static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, + } + for (size_t i = 0; i < n; i++) { + uint64_t off = (addrs[i] - s->view_start) >> PG_SHIFT; +- uint64_t p; +- uint32_t w; + +- if (off >= s->page_total) { +- continue; +- } +- p = s->pages[off]; +- w = atomic_load(&census.pg[p]); +- if (count && first + count != p) { +- fn(first, count, id, any_held, arg); +- count = 0; +- any_held = false; +- } +- if (PW_ID(w) == id && (mask >> PW_ST(w) & 1) && +- atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { +- first = count ? first : p; +- count++; +- any_held |= PW_ST(w) == ST_HELD; +- } else if (count) { +- fn(first, count, id, any_held, arg); +- count = 0; +- any_held = false; ++ if (off < s->page_total) { ++ s->wanted[off / 64] |= 1ull << (off % 64); + } + } +- if (count) { +- fn(first, count, id, any_held, arg); ++ for (uint32_t r = 0; r < s->run_count; r++) { ++ for (uint64_t p = s->runs[r].first; p < s->runs[r].first + s->runs[r].count; p++) { ++ uint64_t off = census.coh_index[p]; ++ uint64_t bit = 1ull << (off % 64); ++ uint32_t w = atomic_load(&census.pg[p]); ++ ++ if (first + count != p) { ++ run_ends(fn, first, &count, id, &any_held, arg); ++ } ++ if ((s->wanted[off / 64] & bit) && PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { ++ first = count ? first : p; ++ count++; ++ any_held |= PW_ST(w) == ST_HELD; ++ } else { ++ run_ends(fn, first, &count, id, &any_held, arg); ++ } ++ s->wanted[off / 64] &= ~bit; ++ } + } ++ run_ends(fn, first, &count, id, &any_held, arg); + } + + /* Where the pages a vCPU reached as their markers moved go. */ +@@ -974,6 +989,11 @@ size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t + return found.found; + } + ++uint64_t census_coh_run_count(void) ++{ ++ return atomic_load(&census.coh_runs); ++} ++ + void census_coh_open(uint64_t addr) + { + coh_runs(&addr, 1, MASK(ST_HELD), open_held, NULL); +@@ -1274,6 +1294,31 @@ static void surface_close(Surface *s, uint32_t id, const char *event) + s->run_count = 0; + free(s->pages); + s->pages = NULL; ++ free(s->wanted); ++ s->wanted = NULL; ++} ++ ++static int run_order(const void *a, const void *b) ++{ ++ uint64_t x = ((const Run *)a)->first, y = ((const Run *)b)->first; ++ ++ return x < y ? -1 : x > y; ++} ++ ++/* A surface's runs ascending, those that meet joined. */ ++static void runs_by_view(Surface *s) ++{ ++ uint32_t kept = 0; ++ ++ qsort(s->runs, s->run_count, sizeof(*s->runs), run_order); ++ for (uint32_t i = 0; i < s->run_count; i++) { ++ if (kept && s->runs[kept - 1].first + s->runs[kept - 1].count == s->runs[i].first) { ++ s->runs[kept - 1].count += s->runs[i].count; ++ } else { ++ s->runs[kept++] = s->runs[i]; ++ } ++ } ++ s->run_count = kept; + } + + void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t page_count, +@@ -1298,6 +1343,11 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + s->tracked = false; + if (census.coherent && page_count) { + s->pages = calloc(page_count, sizeof(*s->pages)); ++ s->wanted = calloc(page_count / 64 + 1, sizeof(*s->wanted)); ++ if (!s->wanted) { ++ free(s->pages); ++ s->pages = NULL; ++ } + s->view_start = f->base_address - f->offset; + } + atomic_store(&s->presents, 0); +@@ -1349,6 +1399,7 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + s->armed_pages++; + } + } ++ runs_by_view(s); + for (uint32_t i = 0; i < s->run_count; i++) { + if (census.coherent) { + /* Nothing on it until the library tracks the surface. */ +@@ -1606,8 +1657,8 @@ static void coherent_report(const char *label) + } + n = census.ops.report(census.ops.ctx, label, text, sizeof(text)); + fwrite(text, 1, MIN(n, sizeof(text)), census.report); +- fprintf(census.report, "{\"touches\":\"%s\",\"open\":%" PRIu64 ",\"armed\":%" PRIu64 +- ",\"held\":%" PRIu64 ",\"frozen\":%" PRIu64 ",\"stale\":%" PRIu64 "}\n", label, ++ fprintf(census.report, "{\"touches\":\"%s\",\"runs\":%" PRIu64 ",\"open\":%" PRIu64 ",\"armed\":%" PRIu64 ++ ",\"held\":%" PRIu64 ",\"frozen\":%" PRIu64 ",\"stale\":%" PRIu64 "}\n", label, atomic_load(&census.coh_runs), + atomic_load(&census.touches[ST_OPEN]), atomic_load(&census.touches[ST_ARMED]), + atomic_load(&census.touches[ST_HELD]), atomic_load(&census.touches[ST_FROZEN]), + atomic_load(&census.touches[ST_STALE])); +diff --git a/include/system/kvm_census.h b/include/system/kvm_census.h +index 3673e4286e..8fa6568d96 100644 +--- a/include/system/kvm_census.h ++++ b/include/system/kvm_census.h +@@ -83,7 +83,7 @@ void census_presented(uint64_t mapping_id); + + /* + * Trap-coherent surfaces (census.c, and shadow_coherent.rs in metal2vulkan). Addresses are of the +- * library's mapping of the surface, whole pages, ascending, one surface to a call. The census_coh_ ++ * library's mapping of the surface, whole pages, one surface to a call. The census_coh_ + * calls come only under the surface's lock in the library; fault takes that lock, so nothing here + * holds a lock across fault. + */ +@@ -111,6 +111,8 @@ size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t + /* Into up to room of came, the pages a vCPU reached as these moved: written, for all it knows. */ + size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room); + void census_coh_open(uint64_t addr); ++/* Runs of contiguous pages the marks have moved in so far, some syscalls each. */ ++uint64_t census_coh_run_count(void); + void census_coh_freeze(const uint64_t *addrs, size_t n); + void census_coh_thaw(const uint64_t *addrs, size_t n); + /* The surface of a mapping id, and whether the library tracks it. */ diff --git a/patches/qemu/0007-census-held-pages-are-read-only-so-a-CPU-store-reach.patch b/patches/qemu/0007-census-held-pages-are-read-only-so-a-CPU-store-reach.patch new file mode 100644 index 0000000..1be96d4 --- /dev/null +++ b/patches/qemu/0007-census-held-pages-are-read-only-so-a-CPU-store-reach.patch @@ -0,0 +1,465 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 06:33:18 +0000 +Subject: [PATCH 7/9] census: held pages are read-only, so a CPU store reaches + QEMU + +A store to a uffd-wp page in a writable VMA, with guest interrupts on and an +in-kernel LAPIC, never left KVM: GUP failed, hva_to_pfn saw a valid VMA and +answered NEEDS_IO, KVM took the async page fault path, whose worker failed in +turn and was ignored, and the vCPU retried for ever with no exit to userspace. +Every user-mode store to a held page has hung since the first coherent build. +Only MercuryPosterExtension hit it because few CPU codes write GPU-made pixels. + +Held and armed pages now live in a read-only VMA from the moment they are +tracked: vma_is_valid is false, and KVM gives -EFAULT at once. An armed page in +a read-only VMA also fails before its poison, so census_coh_probe arms held +pages and makes armed ones writable, and the rerun's SIGBUS gives the address. +Nothing can land between marks, so hold reports nothing and the pagemap goes. + +The -EFAULT fuse no longer runs for a try that lost its page to a hold: 1290 +such tries in 20 ms tripped it in the race test under load. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 195 +++++++++++++++++++----------------- + hw/display/apple-gfx.c | 11 +- + include/system/kvm_census.h | 9 +- + 3 files changed, 118 insertions(+), 97 deletions(-) + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +index 28b3d2df95..e9c87e06ce 100644 +--- a/accel/kvm/census.c ++++ b/accel/kvm/census.c +@@ -16,23 +16,18 @@ + * = 0 allows and which needs neither /dev/userfaultfd nor a capability: + * + * armed UFFDIO_POISON. Any guest access: KVM's hwpoison path, SIGBUS with the exact address. +- * The vCPU logs the touch, drops the marker and write-protects the page, then runs again. +- * held Write-protected. The same instruction reruns: a read passes, which classifies it. A +- * write comes back at once as a bare -EFAULT, no address: the page is this vCPU's ++ * The vCPU logs the touch, drops the marker, then runs again. ++ * held Read-only, by mprotect. The same instruction reruns: a read passes, which classifies ++ * it. A write comes back at once as a bare -EFAULT, no address: the page is this vCPU's + * pending page while RIP has not moved, and it opens. + * open Nothing on the page. + * +- * A write to a page held since an earlier read gives -EFAULT with nothing to name it, so it +- * opens every held page and counts one upgrade. Every Nth present the census's thread re-arms +- * what was touched: WP off, then DONTNEED, then poison. That order matters: MADV_DONTNEED drops a +- * poison marker but on a file mapping it keeps a uffd-wp one, and POISON refuses a slot that holds +- * either (EEXIST). ++ * Not uffd-wp. Such a write has KVM's non-blocking GUP fail in a writable VMA, which KVM takes for ++ * a page that needs I/O: with the guest's interrupts on it tries an asynchronous page fault, and ++ * that worker's GUP fails too and is ignored, so the store reruns in the kernel for good, and ++ * -EFAULT never comes. A read-only VMA fails KVM's vma_is_valid, which is what makes it -EFAULT. + * +- * UFFDIO_POISON and UFFDIO_WRITEPROTECT work on PTEs, so 98k Reims views and the 12 GiB slot add no +- * VMAs beyond this one mapping: arming splits nothing, and vm.max_map_count sees +1. +- * +- * Registering WP makes the view's read faults map read-only (vma_wants_writenotify), so the first +- * write after a read of each page takes one extra host fault, once per page per mapping. ++ * Each read-only run is a VMA of its own, two with what lies beside it, against vm.max_map_count. + * + * Locks: the fault path takes none, and only touches per-page atomics, madvise and uffd ioctls. It + * queues no GPU work and calls nothing in Reims. census.lock guards the surfaces, never waits on a +@@ -52,14 +47,15 @@ + * open No marker: the guest's CPU may have written, and the image is stale here. + * frozen Poison over an open page, while a sync merges into it. + * +- * A write to a held page gives a bare -EFAULT. Where the vCPU has just trapped on the page, the +- * pending page names it. Where it has not, ops.probe arms every held page, nothing having changed +- * in them, and the rerun's SIGBUS gives the exact address: so a write always has its page. +- * ARMED -> HELD drops the poison and then write-protects, and POISON and WP cannot swap in one +- * step, so a vCPU can reach the page between them: census_coh_hold reads pagemap afterwards, and +- * reports a page that came in meanwhile as written, for a copy in of it. QEMU's own accesses (a +- * device's DMA, KVM's emulator through an MMIO exit) use the main mapping and raise no fault, so +- * address_space_map and the direct RAM paths call census_coherent_access. ++ * A tracked page lies in a read-only VMA from the track on, armed too, so moving its marker takes ++ * no protection change. A write to a held or an armed page gives a bare -EFAULT, as GUP refuses it ++ * before it can see the poison. Where the vCPU has just trapped on the page, the pending page names ++ * it. Where it has not, ops.probe arms the held pages and makes the armed ones writable ++ * (census_coh_probe), and the rerun's SIGBUS gives the exact address: so a write always has its ++ * page. No write can land between marks: only a probe's armed pages are writable, and they are ++ * poisoned first. QEMU's own accesses (a device's DMA, KVM's emulator through an MMIO exit) use the ++ * main mapping and raise no fault, so address_space_map and the direct RAM paths call ++ * census_coherent_access. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +@@ -142,6 +138,7 @@ struct CensusVcpu { + uint32_t spins; + uint64_t efault_first; + uint64_t efault_last; ++ uint64_t probe_holds; + uint64_t seen[256]; + _Atomic uint32_t head; + _Atomic uint32_t tail; +@@ -186,8 +183,10 @@ typedef struct Surface { + uint32_t *pages; + uint64_t page_total; + uint64_t view_start; +- /* Which of those pages a call means; only the call under the surface's lock uses it. */ ++ /* Which of those pages a call of several means: such calls come under the surface's lock. */ + uint64_t *wanted; ++ /* A probe left armed pages writable: holding them must protect them again. */ ++ bool writable; + bool tracked; + _Atomic uint64_t presents; + _Atomic uint32_t window_traps; +@@ -237,7 +236,6 @@ static struct { + CensusCoherentOps ops; + /* A page's index in its surface. */ + uint32_t *coh_index; +- int pagemap; + FILE *report; + + _Atomic uint64_t presents; +@@ -248,6 +246,8 @@ static struct { + uint64_t tick_ns; + + _Atomic uint64_t traps, writes, upgrades, spurious, dropped, errors; ++ /* Runs of pages held: a probe that outlives one may have lost what it opened. */ ++ _Atomic uint64_t holds; + uint64_t overlaps; + } census = { + .wanted = -1, +@@ -331,21 +331,8 @@ static int page_drop(uint64_t page, uint64_t count) + + static int page_wp(uint64_t page, uint64_t count, bool on) + { +- struct uffdio_writeprotect w = { +- .range = { (uint64_t)census.view + (page << PG_SHIFT), count << PG_SHIFT }, +- /* The kernel takes DONTWAKE only for WP off. Nothing ever waits on this uffd. */ +- .mode = on ? UFFDIO_WRITEPROTECT_MODE_WP : UFFDIO_WRITEPROTECT_MODE_DONTWAKE, +- }; +- +- int r; +- +- for (int tries = 0; tries < POISON_TRIES; tries++) { +- r = ioctl(census.uffd, UFFDIO_WRITEPROTECT, &w); +- if (r == 0 || errno != EAGAIN) { +- break; +- } +- } +- return r; ++ return mprotect((uint8_t *)census.view + (page << PG_SHIFT), count << PG_SHIFT, ++ on ? PROT_READ : PROT_READ | PROT_WRITE); + } + + static void store_pages(uint64_t first, uint64_t count, uint32_t state, uint32_t id) +@@ -657,9 +644,7 @@ static void open_held(uint64_t first, uint64_t count, uint32_t id, bool any_held + /* Everything off: the unmap's and the reset's work, done before either returns. */ + static void clear_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +- if (any_held) { +- page_wp(first, count, false); +- } ++ page_wp(first, count, false); + page_drop(first, count); + store_pages(first, count, ST_STALE, 0x1fffffffu); + } +@@ -686,13 +671,18 @@ static void census_fuse(const char *why) + } + } + +-/* An -EFAULT: is it worth another try? Unanswered, the fuses go as in census_on_efault. */ +-static bool efault_again(CensusVcpu *vc, bool at_once) ++/* ++ * An -EFAULT: is it worth another try? Unanswered, the fuses go as in census_on_efault. With lost, ++ * a hold took the page from under this try, which answers it: the fuses' clock stops for it. ++ */ ++static bool efault_again(CensusVcpu *vc, bool at_once, bool lost) + { + uint64_t now = now_ns(); + + if (!vc->efault_first || now - vc->efault_last > EFAULT_GAP_NS) { + vc->efault_first = now; ++ } else if (lost) { ++ vc->efault_first += now - vc->efault_last; + } + vc->efault_last = now; + if (now - vc->efault_first > FUSE_AFTER_NS) { +@@ -710,6 +700,8 @@ static bool efault_again(CensusVcpu *vc, bool at_once) + static bool coherent_on_efault(CensusVcpu *vc) + { + uint64_t rip = vcpu_rip(vc); ++ uint64_t holds = atomic_load(&census.holds); ++ bool lost = vc->probe_holds != holds; + + if (vc->pending >= 0 && vc->pending_rip == rip) { + uint64_t page = vc->pending; +@@ -726,11 +718,12 @@ static bool coherent_on_efault(CensusVcpu *vc) + vc->pending = -1; + if (atomic_load(&census_coherent_pages) && !atomic_load(&census.dead)) { + vc->probe_rip = rip; ++ vc->probe_holds = holds; + atomic_fetch_add(&census.upgrades, 1); + census.ops.probe(census.ops.ctx); +- return efault_again(vc, true); ++ return efault_again(vc, true, lost); + } +- return efault_again(vc, false); ++ return efault_again(vc, false, false); + } + + bool census_on_efault(CensusVcpu *vc) +@@ -792,7 +785,7 @@ bool census_on_efault(CensusVcpu *vc) + * keep returning, something answers it that no write protection explains. + */ + pthread_mutex_unlock(&census.lock); +- return efault_again(vc, false); ++ return efault_again(vc, false, false); + } + + /* ---- coherent pages, each under its surface's lock in the library -------------------------- */ +@@ -847,6 +840,21 @@ static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, + if (!s) { + return; + } ++ /* One page needs no walk, and a vCPU's own call comes with no surface lock to share `wanted`. */ ++ if (n == 1) { ++ uint64_t off = (addrs[0] - s->view_start) >> PG_SHIFT; ++ uint64_t p = off < s->page_total ? s->pages[off] : 0; ++ uint32_t w = off < s->page_total ? atomic_load(&census.pg[p]) : 0; ++ ++ if (off < s->page_total && PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { ++ first = p; ++ count = 1; ++ any_held = PW_ST(w) == ST_HELD; ++ run_ends(fn, first, &count, id, &any_held, arg); ++ } ++ return; ++ } + for (size_t i = 0; i < n; i++) { + uint64_t off = (addrs[i] - s->view_start) >> PG_SHIFT; + +@@ -884,11 +892,10 @@ typedef struct Came { + } Came; + + /* +- * HELD -> ARMED. Poison needs a bare PTE, so the write protection goes first, and a write between +- * the two would go unseen. So the mapping goes before it, the protection staying behind as a PTE +- * marker: then a write needs a mapping, and POISON refuses a page mapped. A page it refuses may +- * hold a write, and it may be what the vCPU's about to read, so there is no retrying it: it goes +- * back to held, and the library hears, and has what the arm would have left to the page's trap. ++ * HELD -> ARMED. The page stays read-only, so no write can land while its mapping goes. POISON ++ * refuses a page mapped: a vCPU read it as the mapping went, and it may be what that vCPU reads ++ * next, so there is no retrying it: it stays held, and the library hears, and has what the arm ++ * would have left to the page's trap. + */ + static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +@@ -897,9 +904,6 @@ static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_he + int tries = 0; + + page_drop(first, count); +- if (page_wp(first, count, false)) { +- count_error("WP off for arming", first); +- } + while (at < end) { + struct uffdio_poison z = { + .range = { (uint64_t)census.view + (at << PG_SHIFT), (end - at) << PG_SHIFT }, +@@ -923,9 +927,6 @@ static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_he + count_error("poison", at); + } + page_drop(at, 1); +- if (page_wp(at, 1, true)) { +- count_error("WP on", at); +- } + store_pages(at, 1, ST_HELD, id); + if (refused->found < refused->room) { + refused->out[refused->found++] = coh_addr(id, at); +@@ -935,39 +936,26 @@ static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_he + } + } + +-/* +- * Between the poison's going and the write protection a vCPU may have faulted a page in, and +- * written it: any page the view maps now did so, as the drop left none mapped. +- */ +-static void came_in(uint64_t first, uint64_t count, uint32_t id, Came *came) ++/* ARMED or OPEN -> HELD: arg says whether open pages are among them, which alone need protecting. */ ++static void coh_held_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +- uint64_t bits[512]; +- +- for (uint64_t at = first; at < first + count && came->found < came->room;) { +- uint64_t batch = MIN(first + count - at, sizeof(bits) / sizeof(bits[0])); +- off_t place = ((uint64_t)census.view >> PG_SHIFT) + at; +- ssize_t got = pread(census.pagemap, bits, batch * sizeof(bits[0]), place * sizeof(bits[0])); +- +- for (ssize_t i = 0; i < got / (ssize_t)sizeof(bits[0]) && came->found < came->room; i++) { +- if (bits[i] >> 63) { +- came->out[came->found++] = coh_addr(id, at + i); +- } ++ if ((arg && *(const bool *)arg) || census.surfaces[id].writable) { ++ if (page_wp(first, count, true)) { ++ count_error("read-only", first); + } +- at += batch; + } ++ page_drop(first, count); ++ store_pages(first, count, ST_HELD, id); ++ atomic_fetch_add(&census.holds, 1); + } + +-static void coh_held_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) ++/* Armed pages a write can reach: see census_coh_probe. */ ++static void coh_writable_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +- /* A page in the run that was open keeps a mapping: it reads and writes as it pleases. */ +- page_drop(first, count); +- if (page_wp(first, count, true)) { +- count_error("WP on", first); +- } +- store_pages(first, count, ST_HELD, id); +- if (arg) { +- came_in(first, count, id, arg); ++ if (page_wp(first, count, false)) { ++ count_error("writable", first); + } ++ store_pages(first, count, ST_ARMED, id); + } + + size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room) +@@ -980,23 +968,42 @@ size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t + return found.found; + } + +-size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room) ++size_t census_coh_probe(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room) + { +- uint32_t mask = MASK(ST_ARMED) | (open_too ? MASK(ST_OPEN) : 0); +- Came found = { came, room, 0 }; ++ Came found = { refused, room, 0 }; ++ uint32_t id = 0; ++ uint64_t index = 0; ++ Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; + +- coh_runs(addrs, n, mask, coh_held_run, came && census.pagemap >= 0 ? &found : NULL); ++ if (s && !atomic_load(&census.dead)) { ++ s->writable = true; ++ coh_runs(addrs, n, MASK(ST_HELD), coh_arm_run, &found); ++ coh_runs(addrs, n, MASK(ST_ARMED), coh_writable_run, NULL); ++ } + return found.found; + } + ++size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room) ++{ ++ uint32_t id = 0; ++ uint64_t index = 0; ++ Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; ++ ++ coh_runs(addrs, n, MASK(ST_ARMED) | (open_too ? MASK(ST_OPEN) : 0), coh_held_run, &open_too); ++ if (s && n >= s->page_total) { ++ s->writable = false; ++ } ++ return 0; ++} ++ + uint64_t census_coh_run_count(void) + { + return atomic_load(&census.coh_runs); + } + +-void census_coh_open(uint64_t addr) ++void census_coh_open(const uint64_t *addrs, size_t n) + { +- coh_runs(&addr, 1, MASK(ST_HELD), open_held, NULL); ++ coh_runs(addrs, n, MASK(ST_HELD), open_held, NULL); + } + + static void coh_freeze_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) +@@ -1056,9 +1063,11 @@ bool census_coh_track(uint64_t start, uint64_t len) + return false; + } + if (!s->tracked) { ++ bool open_too = true; ++ + s->tracked = true; + atomic_fetch_add(&census_coherent_pages, s->page_total); +- take_pages(s, id, MASK(ST_OPEN), true, coh_held_run, NULL); ++ take_pages(s, id, MASK(ST_OPEN), true, coh_held_run, &open_too); + } + pthread_mutex_unlock(&census.lock); + return true; +@@ -1067,9 +1076,7 @@ bool census_coh_track(uint64_t start, uint64_t len) + /* Every marker off the surface, its pages left open. */ + static void coh_free_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +- if (any_held) { +- page_wp(first, count, false); +- } ++ page_wp(first, count, false); + page_drop(first, count); + store_pages(first, count, ST_OPEN, id); + } +@@ -1341,6 +1348,7 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + s->pages = NULL; + s->page_total = 0; + s->tracked = false; ++ s->writable = false; + if (census.coherent && page_count) { + s->pages = calloc(page_count, sizeof(*s->pages)); + s->wanted = calloc(page_count / 64 + 1, sizeof(*s->wanted)); +@@ -1921,7 +1929,6 @@ bool census_start(int fd, uint64_t offset, void *main_map, uint64_t len) + census.len = len; + census.pages = len >> PG_SHIFT; + census.pg = calloc(census.pages, sizeof(*census.pg)); +- census.pagemap = open("/proc/self/pagemap", O_RDONLY | O_CLOEXEC); + if (census.coherent) { + census.coh_index = calloc(census.pages, sizeof(*census.coh_index)); + } +diff --git a/hw/display/apple-gfx.c b/hw/display/apple-gfx.c +index 4d8610c48c..93ff321287 100644 +--- a/hw/display/apple-gfx.c ++++ b/hw/display/apple-gfx.c +@@ -1080,9 +1080,15 @@ static size_t apple_gfx_coh_hold(void *ctx, const uint64_t *addrs, size_t n, boo + return census_coh_hold(addrs, n, open_too, came, room); + } + +-static void apple_gfx_coh_open(void *ctx, uint64_t addr) ++static void apple_gfx_coh_open(void *ctx, const uint64_t *addrs, size_t n) + { +- census_coh_open(addr); ++ census_coh_open(addrs, n); ++} ++ ++static size_t apple_gfx_coh_name(void *ctx, const uint64_t *addrs, size_t n, uint64_t *refused, ++ size_t room) ++{ ++ return census_coh_probe(addrs, n, refused, room); + } + + static void apple_gfx_coh_freeze(void *ctx, const uint64_t *addrs, size_t n) +@@ -1153,6 +1159,7 @@ static void apple_gfx_census_register(void) + .open = apple_gfx_coh_open, + .freeze = apple_gfx_coh_freeze, + .thaw = apple_gfx_coh_thaw, ++ .probe = apple_gfx_coh_name, + }; + + census_coherent_set_ops(&ops); +diff --git a/include/system/kvm_census.h b/include/system/kvm_census.h +index 8fa6568d96..0bac5baa86 100644 +--- a/include/system/kvm_census.h ++++ b/include/system/kvm_census.h +@@ -110,7 +110,14 @@ void census_coh_untrack(uint64_t start); + size_t census_coh_arm(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room); + /* Into up to room of came, the pages a vCPU reached as these moved: written, for all it knows. */ + size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, size_t room); +-void census_coh_open(uint64_t addr); ++/* Held pages open to the guest's writes. */ ++void census_coh_open(const uint64_t *addrs, size_t n); ++/* ++ * Held pages armed, as census_coh_arm, and the armed ones writable, so the next write of any of ++ * them stops with its address: a write to an armed page in a read-only mapping does not. A hold of ++ * the whole surface ends that. ++ */ ++size_t census_coh_probe(const uint64_t *addrs, size_t n, uint64_t *refused, size_t room); + /* Runs of contiguous pages the marks have moved in so far, some syscalls each. */ + uint64_t census_coh_run_count(void); + void census_coh_freeze(const uint64_t *addrs, size_t n); diff --git a/patches/qemu/0008-census-track-a-Coherent-surface-s-rows-as-ranges-not.patch b/patches/qemu/0008-census-track-a-Coherent-surface-s-rows-as-ranges-not.patch new file mode 100644 index 0000000..8e7444a --- /dev/null +++ b/patches/qemu/0008-census-track-a-Coherent-surface-s-rows-as-ranges-not.patch @@ -0,0 +1,365 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 05:49:11 +0000 +Subject: [PATCH 8/9] census: track a Coherent surface's rows as ranges, not + the whole surface + +A surface's pages past its texture rows (an IOSurface's trailing pages, a +second plane) were armed and held with the rows, so a CPU or DMA touch +there met no pending page, came back as a bare -EFAULT and fused the +census (B1). census_coh_track(start, len) now tracks [start, start+len) +as one of up to four ranges of the surface; pages outside every range +stay OPEN, untrack frees one range, and touches outside the ranges pass +straight through. + +Co-Authored-By: Claude Opus 5.5 +--- + accel/kvm/census.c | 188 ++++++++++++++++++++++++++++++++++++--------- + 1 file changed, 153 insertions(+), 35 deletions(-) + +diff --git a/accel/kvm/census.c b/accel/kvm/census.c +index e9c87e06ce..40cb9ea0da 100644 +--- a/accel/kvm/census.c ++++ b/accel/kvm/census.c +@@ -150,6 +150,18 @@ typedef struct Run { + uint32_t count; + } Run; + ++/* ++ * One range a library's surface tracks: pages [lo, hi) of its census surface, in the surface's ++ * order. `writable`: a probe left its armed pages writable, and holding them must protect them ++ * again; a hold of the whole range ends that. Its bounds change only under census.lock, with ++ * every page of it ST_WORK or not yet marked, and a vCPU or a device reads them with no lock. ++ */ ++#define COH_RANGES 4 ++typedef struct CohRange { ++ _Atomic uint64_t lo, hi; ++ bool writable; ++} CohRange; ++ + typedef struct RipCount { + uint64_t rip; + uint64_t count; +@@ -185,8 +197,13 @@ typedef struct Surface { + uint64_t view_start; + /* Which of those pages a call of several means: such calls come under the surface's lock. */ + uint64_t *wanted; +- /* A probe left armed pages writable: holding them must protect them again. */ +- bool writable; ++ /* ++ * The ranges the library tracks, each a texture's rows (a plane of it), never two over one ++ * page: pages outside them stay open and unmarked, so neither a write there nor a device's ++ * touch reaches the library, which names no such page. `tracked` is whether any is. ++ */ ++ CohRange ranges[COH_RANGES]; ++ _Atomic uint32_t range_count; + bool tracked; + _Atomic uint64_t presents; + _Atomic uint32_t window_traps; +@@ -594,9 +611,12 @@ int census_on_sigbus(CensusVcpu *vc, void **addr) + */ + typedef void RunFn(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg); + +-static uint32_t take_pages(Surface *s, uint32_t id, uint32_t mask, bool wait, RunFn *fn, void *arg) ++/* take_pages over the pages [lo, hi) of the surface's order alone: a coherent range's. */ ++static uint32_t take_pages_in(Surface *s, uint32_t id, uint64_t lo, uint64_t hi, uint32_t mask, ++ bool wait, RunFn *fn, void *arg) + { + uint32_t taken = 0; ++ bool all = lo == 0 && hi == UINT64_MAX; + + for (uint32_t i = 0; i < s->run_count; i++) { + uint64_t first = 0, count = 0; +@@ -604,12 +624,13 @@ static uint32_t take_pages(Surface *s, uint32_t id, uint32_t mask, bool wait, Ru + + for (uint64_t p = s->runs[i].first; p < s->runs[i].first + s->runs[i].count; p++) { + uint32_t w = atomic_load(&census.pg[p]); ++ bool in = all || (census.coh_index[p] >= lo && census.coh_index[p] < hi); + +- while (wait && PW_ID(w) == id && PW_ST(w) == ST_WORK) { ++ while (in && wait && PW_ID(w) == id && PW_ST(w) == ST_WORK) { + sched_yield(); + w = atomic_load(&census.pg[p]); + } +- if (PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ if (in && PW_ID(w) == id && (mask >> PW_ST(w) & 1) && + atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { + if (count && first + count != p) { + fn(first, count, id, any_held, arg); +@@ -633,6 +654,11 @@ static uint32_t take_pages(Surface *s, uint32_t id, uint32_t mask, bool wait, Ru + return taken; + } + ++static uint32_t take_pages(Surface *s, uint32_t id, uint32_t mask, bool wait, RunFn *fn, void *arg) ++{ ++ return take_pages_in(s, id, 0, UINT64_MAX, mask, wait, fn, arg); ++} ++ + static void open_held(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { + if (page_wp(first, count, false)) { +@@ -814,6 +840,19 @@ static Surface *coh_surface(uint64_t addr, uint32_t *id, uint64_t *index) + return NULL; + } + ++/* The range of s that holds its page `index`, or NULL. No lock: see CohRange. */ ++static CohRange *coh_range(Surface *s, uint64_t index) ++{ ++ uint32_t n = MIN(atomic_load(&s->range_count), COH_RANGES); ++ ++ for (uint32_t r = 0; r < n; r++) { ++ if (index >= atomic_load(&s->ranges[r].lo) && index < atomic_load(&s->ranges[r].hi)) { ++ return &s->ranges[r]; ++ } ++ } ++ return NULL; ++} ++ + static void run_ends(RunFn *fn, uint64_t first, uint64_t *count, uint32_t id, bool *any_held, void *arg) + { + if (*count) { +@@ -835,18 +874,23 @@ static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, + uint32_t id = 0; + uint64_t index = 0, first = 0, count = 0; + Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; ++ CohRange *range = s ? coh_range(s, index) : NULL; ++ uint64_t lo, hi; + bool any_held = false; + +- if (!s) { ++ /* The addresses are one tracked range's: a page outside every range is no library's. */ ++ if (!range) { + return; + } ++ lo = atomic_load(&range->lo); ++ hi = atomic_load(&range->hi); + /* One page needs no walk, and a vCPU's own call comes with no surface lock to share `wanted`. */ + if (n == 1) { + uint64_t off = (addrs[0] - s->view_start) >> PG_SHIFT; + uint64_t p = off < s->page_total ? s->pages[off] : 0; + uint32_t w = off < s->page_total ? atomic_load(&census.pg[p]) : 0; + +- if (off < s->page_total && PW_ID(w) == id && (mask >> PW_ST(w) & 1) && ++ if (off >= lo && off < hi && PW_ID(w) == id && (mask >> PW_ST(w) & 1) && + atomic_compare_exchange_strong(&census.pg[p], &w, PW(ST_WORK, id))) { + first = p; + count = 1; +@@ -858,7 +902,7 @@ static void coh_runs(const uint64_t *addrs, size_t n, uint32_t mask, RunFn *fn, + for (size_t i = 0; i < n; i++) { + uint64_t off = (addrs[i] - s->view_start) >> PG_SHIFT; + +- if (off < s->page_total) { ++ if (off >= lo && off < hi && off < s->page_total) { + s->wanted[off / 64] |= 1ull << (off % 64); + } + } +@@ -936,10 +980,20 @@ static void coh_arm_run(uint64_t first, uint64_t count, uint32_t id, bool any_he + } + } + +-/* ARMED or OPEN -> HELD: arg says whether open pages are among them, which alone need protecting. */ ++/* What a hold's runs need to know: whether pages it moves may be writable. */ ++typedef struct HeldArg { ++ /* Open pages are among them. */ ++ bool open_too; ++ /* A probe left the range's armed pages writable. */ ++ bool writable; ++} HeldArg; ++ ++/* ARMED or OPEN -> HELD: open pages, or armed ones a probe made writable, need protecting. */ + static void coh_held_run(uint64_t first, uint64_t count, uint32_t id, bool any_held, void *arg) + { +- if ((arg && *(const bool *)arg) || census.surfaces[id].writable) { ++ const HeldArg *held = arg; ++ ++ if (held->open_too || held->writable) { + if (page_wp(first, count, true)) { + count_error("read-only", first); + } +@@ -974,9 +1028,10 @@ size_t census_coh_probe(const uint64_t *addrs, size_t n, uint64_t *refused, size + uint32_t id = 0; + uint64_t index = 0; + Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; ++ CohRange *range = s ? coh_range(s, index) : NULL; + +- if (s && !atomic_load(&census.dead)) { +- s->writable = true; ++ if (range && !atomic_load(&census.dead)) { ++ range->writable = true; + coh_runs(addrs, n, MASK(ST_HELD), coh_arm_run, &found); + coh_runs(addrs, n, MASK(ST_ARMED), coh_writable_run, NULL); + } +@@ -988,10 +1043,12 @@ size_t census_coh_hold(const uint64_t *addrs, size_t n, bool open_too, uint64_t + uint32_t id = 0; + uint64_t index = 0; + Surface *s = n ? coh_surface(addrs[0], &id, &index) : NULL; ++ CohRange *range = s ? coh_range(s, index) : NULL; ++ HeldArg held = { open_too, range && range->writable }; + +- coh_runs(addrs, n, MASK(ST_ARMED) | (open_too ? MASK(ST_OPEN) : 0), coh_held_run, &open_too); +- if (s && n >= s->page_total) { +- s->writable = false; ++ coh_runs(addrs, n, MASK(ST_ARMED) | (open_too ? MASK(ST_OPEN) : 0), coh_held_run, &held); ++ if (range && n >= atomic_load(&range->hi) - atomic_load(&range->lo)) { ++ range->writable = false; + } + return 0; + } +@@ -1047,30 +1104,70 @@ void census_coh_thaw(const uint64_t *addrs, size_t n) + coh_runs(addrs, n, MASK(ST_FROZEN), coh_thaw_run, NULL); + } + ++/* ++ * Holds the pages of [start, start + len), whole pages of one census surface, and tracks them as ++ * a range of their own: pages of the surface outside every range stay open, unmarked. Refused ++ * where the range leaves the surface, meets one tracked already, or the surface has its ranges. ++ * The same range again is tracked already. ++ */ + bool census_coh_track(uint64_t start, uint64_t len) + { +- uint32_t id; +- uint64_t index; ++ uint32_t id, slot = COH_RANGES; ++ uint64_t index, lo, hi; + Surface *s; ++ HeldArg held = { true, false }; ++ bool ok = false; + +- if (!census.active || !census.coherent || atomic_load(&census.dead)) { ++ if (!census.active || !census.coherent || atomic_load(&census.dead) || !len) { + return false; + } + pthread_mutex_lock(&census.lock); + s = coh_surface(start, &id, &index); + if (!s || s->overlaps || s->foreign) { +- pthread_mutex_unlock(&census.lock); +- return false; ++ goto out; + } +- if (!s->tracked) { +- bool open_too = true; ++ lo = (start - s->view_start) >> PG_SHIFT; ++ hi = (start + len - s->view_start + PG - 1) >> PG_SHIFT; ++ if (hi > s->page_total) { ++ goto out; ++ } ++ for (uint32_t r = 0; r < atomic_load(&s->range_count); r++) { ++ uint64_t rlo = atomic_load(&s->ranges[r].lo), rhi = atomic_load(&s->ranges[r].hi); + +- s->tracked = true; +- atomic_fetch_add(&census_coherent_pages, s->page_total); +- take_pages(s, id, MASK(ST_OPEN), true, coh_held_run, &open_too); ++ if (rlo == rhi) { ++ slot = slot == COH_RANGES ? r : slot; ++ continue; ++ } ++ if (rlo == lo && rhi == hi) { ++ ok = true; ++ goto out; ++ } ++ if (rlo < hi && lo < rhi) { ++ goto out; ++ } + } ++ if (slot == COH_RANGES) { ++ if (atomic_load(&s->range_count) == COH_RANGES) { ++ goto out; ++ } ++ slot = atomic_load(&s->range_count); ++ } ++ s->ranges[slot].writable = false; ++ /* The bounds first, never a wrong range on the way, then the pages: a reader of the bounds ++ * finds them before any page is marked. */ ++ atomic_store(&s->ranges[slot].hi, 0); ++ atomic_store(&s->ranges[slot].lo, lo); ++ atomic_store(&s->ranges[slot].hi, hi); ++ if (slot == atomic_load(&s->range_count)) { ++ atomic_store(&s->range_count, slot + 1); ++ } ++ s->tracked = true; ++ atomic_fetch_add(&census_coherent_pages, hi - lo); ++ take_pages_in(s, id, lo, hi, MASK(ST_OPEN), true, coh_held_run, &held); ++ ok = true; ++out: + pthread_mutex_unlock(&census.lock); +- return true; ++ return ok; + } + + /* Every marker off the surface, its pages left open. */ +@@ -1081,29 +1178,50 @@ static void coh_free_run(uint64_t first, uint64_t count, uint32_t id, bool any_h + store_pages(first, count, ST_OPEN, id); + } + ++/* One range's markers off, its pages open, and the range gone. */ ++static void coh_untrack_range_locked(Surface *s, uint32_t id, CohRange *range) ++{ ++ uint64_t lo = atomic_load(&range->lo), hi = atomic_load(&range->hi); ++ ++ if (lo == hi) { ++ return; ++ } ++ take_pages_in(s, id, lo, hi, MASK(ST_ARMED) | MASK(ST_HELD) | MASK(ST_FROZEN), true, ++ coh_free_run, NULL); ++ atomic_fetch_sub(&census_coherent_pages, hi - lo); ++ atomic_store(&range->hi, lo); ++ range->writable = false; ++ s->tracked = false; ++ for (uint32_t r = 0; r < atomic_load(&s->range_count); r++) { ++ s->tracked |= atomic_load(&s->ranges[r].lo) != atomic_load(&s->ranges[r].hi); ++ } ++} ++ + static void coh_untrack_locked(Surface *s, uint32_t id) + { +- if (s->tracked) { +- s->tracked = false; +- atomic_fetch_sub(&census_coherent_pages, s->page_total); +- take_pages(s, id, MASK(ST_ARMED) | MASK(ST_HELD) | MASK(ST_FROZEN), true, coh_free_run, +- NULL); ++ for (uint32_t r = 0; r < atomic_load(&s->range_count); r++) { ++ coh_untrack_range_locked(s, id, &s->ranges[r]); + } ++ atomic_store(&s->range_count, 0); ++ s->tracked = false; + } + ++/* The range that holds start goes: another range of the surface, a plane beside it, stays. */ + void census_coh_untrack(uint64_t start) + { + uint32_t id; + uint64_t index; + Surface *s; ++ CohRange *range; + + if (!census.active || !census.coherent) { + return; + } + pthread_mutex_lock(&census.lock); + s = coh_surface(start, &id, &index); +- if (s) { +- coh_untrack_locked(s, id); ++ range = s ? coh_range(s, index) : NULL; ++ if (range) { ++ coh_untrack_range_locked(s, id, range); + } + pthread_mutex_unlock(&census.lock); + } +@@ -1146,7 +1264,7 @@ void census_coherent_touch(void *main_ptr, uint64_t len, bool write) + atomic_fetch_add(&census.touches[st], 1); + /* Only these states hold a surface's id: a stale page's is not one. */ + if ((st == ST_ARMED || st == ST_FROZEN || (write && st == ST_HELD)) && +- census.surfaces[id].tracked) { ++ coh_range(&census.surfaces[id], census.coh_index[p])) { + census.ops.fault(census.ops.ctx, coh_addr(id, p), write); + } + } +@@ -1348,7 +1466,7 @@ void census_surface_mapped(const CensusSurface *f, const uint64_t *pages, size_t + s->pages = NULL; + s->page_total = 0; + s->tracked = false; +- s->writable = false; ++ atomic_store(&s->range_count, 0); + if (census.coherent && page_count) { + s->pages = calloc(page_count, sizeof(*s->pages)); + s->wanted = calloc(page_count / 64 + 1, sizeof(*s->wanted)); diff --git a/patches/qemu/0009-census-QEMU-s-own-small-loads-and-stores-of-guest-RA.patch b/patches/qemu/0009-census-QEMU-s-own-small-loads-and-stores-of-guest-RA.patch new file mode 100644 index 0000000..0ce46b9 --- /dev/null +++ b/patches/qemu/0009-census-QEMU-s-own-small-loads-and-stores-of-guest-RA.patch @@ -0,0 +1,53 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 07:05:14 +0000 +Subject: [PATCH 9/9] census: QEMU's own small loads and stores of guest RAM + reach the census + +address_space_ld*/st* (ldl_phys, stl_le_phys and the like, and their +_cached_slow forms) read and write RAM through qemu_map_ram_ptr directly, +past flatview_{read,write}_continue_step where census_ram_access is: such an +access of a Coherent surface's armed page read the older bytes, and a write to +a held or armed page never reached the library. Each now calls +census_ram_access first, one relaxed load while nothing is tracked. + +Left alone: address_space_{read,write}_cached's inline memcpy, whose only +user here is virtio's vring caches, which lie in the guest kernel's memory, +never in an IOSurface's pages. + +Co-Authored-By: Claude Opus 5.5 +--- + system/memory_ldst.c.inc | 12 ++++++++++-- + 1 file changed, 10 insertions(+), 2 deletions(-) + +diff --git a/system/memory_ldst.c.inc b/system/memory_ldst.c.inc +index a728b7737c..3239d04ca7 100644 +--- a/system/memory_ldst.c.inc ++++ b/system/memory_ldst.c.inc +@@ -43,8 +43,12 @@ uint64_t glue(address_space_ldm_internal, SUFFIX)(ARG1_DECL, MemOp mop, + r = memory_region_dispatch_read(mr, addr1, &val, mop, attrs); + } else { + /* RAM case */ ++ void *ptr = qemu_map_ram_ptr(mr->ram_block, addr1); ++ + fuzz_dma_read_cb(addr, size, mr); +- val = ldm_p(qemu_map_ram_ptr(mr->ram_block, addr1), mop); ++ /* LOCAL (Atlas): a Coherent surface's armed page is brought up to date first. */ ++ census_ram_access(ptr, size, false); ++ val = ldm_p(ptr, mop); + r = MEMTX_OK; + } + if (result) { +@@ -85,7 +89,11 @@ void glue(address_space_stm_internal, SUFFIX)(ARG1_DECL, MemOp mop, + r = memory_region_dispatch_write(mr, addr1, val, mop, attrs); + } else { + /* RAM case */ +- stm_p(qemu_map_ram_ptr(mr->ram_block, addr1), mop, val); ++ void *ptr = qemu_map_ram_ptr(mr->ram_block, addr1); ++ ++ /* LOCAL (Atlas): a Coherent surface's held or armed page hears of the write first. */ ++ census_ram_access(ptr, size, true); ++ stm_p(ptr, mop, val); + invalidate_and_set_dirty(mr, addr1, size); + r = MEMTX_OK; + } diff --git a/patches/reims-vgpu/0001-REIMS_VGPU_NON_APPLE_GPU-1-answers-as-an.patch b/patches/reims-vgpu/0001-REIMS_VGPU_NON_APPLE_GPU-1-answers-as-an.patch index 801b9d8..6551be2 100644 --- a/patches/reims-vgpu/0001-REIMS_VGPU_NON_APPLE_GPU-1-answers-as-an.patch +++ b/patches/reims-vgpu/0001-REIMS_VGPU_NON_APPLE_GPU-1-answers-as-an.patch @@ -1,7 +1,7 @@ -From 419bdbbc32bc6f041b2bddeaa62ae761f8b3f452 Mon Sep 17 00:00:00 2001 +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: macbox Date: Wed, 28 Oct 2026 14:20:56 +0000 -Subject: [PATCH 1/3] REIMS_VGPU_NON_APPLE_GPU=1 answers as an +Subject: [PATCH 01/25] REIMS_VGPU_NON_APPLE_GPU=1 answers as an Intel/AMD Mac's GPU --- @@ -9,14 +9,14 @@ Subject: [PATCH 1/3] REIMS_VGPU_NON_APPLE_GPU=1 answers as an 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/crates/reims-vgpu-vulkan/src/gpu/caps.rs b/crates/reims-vgpu-vulkan/src/gpu/caps.rs -index 01ec3d5..a785e63 100644 +index 01ec3d57..a785e63e 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/caps.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/caps.rs @@ -181,6 +181,23 @@ const PUBLIC_PRIMITIVE_TYPES: u64 = 5; /// The stand-ins: the Metal platform's answers on an Apple M3 Max, where no documented fact /// gives one (the module's table). The registry ID, which Metal answers at run time, is the /// OS's ([`crate::os::default_registry_id`]). -+/// macbox: `REIMS_VGPU_NON_APPLE_GPU=1` answers the guest as Apple's framework does on an ++/// LOCAL (Atlas): `REIMS_VGPU_NON_APPLE_GPU=1` answers the guest as Apple's framework does on an +/// Intel or AMD Mac's GPU: no Apple GPU family (device info key 28 bit 0 off, key 37 Apple5's +/// fallback 1005, key 44 none), and no tile shaders, imageblocks, raster order groups, +/// framebuffer reads or extended XR10 formats, so the guest's compositor takes the paths it diff --git a/patches/reims-vgpu/0002-REIMS_VGPU_NON_APPLE_GPU-takes-a-list-of.patch b/patches/reims-vgpu/0002-REIMS_VGPU_NON_APPLE_GPU-takes-a-list-of.patch index 67cdf95..4c37702 100644 --- a/patches/reims-vgpu/0002-REIMS_VGPU_NON_APPLE_GPU-takes-a-list-of.patch +++ b/patches/reims-vgpu/0002-REIMS_VGPU_NON_APPLE_GPU-takes-a-list-of.patch @@ -1,7 +1,7 @@ -From 8808e19fccab471f8f967949b68c4db036432c43 Mon Sep 17 00:00:00 2001 +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: macbox Date: Wed, 28 Oct 2026 14:39:45 +0000 -Subject: [PATCH 2/3] REIMS_VGPU_NON_APPLE_GPU takes a list of +Subject: [PATCH 02/25] REIMS_VGPU_NON_APPLE_GPU takes a list of parts --- @@ -9,14 +9,14 @@ Subject: [PATCH 2/3] REIMS_VGPU_NON_APPLE_GPU takes a list of 1 file changed, 26 insertions(+), 21 deletions(-) diff --git a/crates/reims-vgpu-vulkan/src/gpu/caps.rs b/crates/reims-vgpu-vulkan/src/gpu/caps.rs -index a785e63..2def509 100644 +index a785e63e..2def509c 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/caps.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/caps.rs @@ -181,21 +181,26 @@ const PUBLIC_PRIMITIVE_TYPES: u64 = 5; /// The stand-ins: the Metal platform's answers on an Apple M3 Max, where no documented fact /// gives one (the module's table). The registry ID, which Metal answers at run time, is the /// OS's ([`crate::os::default_registry_id`]). --/// macbox: `REIMS_VGPU_NON_APPLE_GPU=1` answers the guest as Apple's framework does on an +-/// LOCAL (Atlas): `REIMS_VGPU_NON_APPLE_GPU=1` answers the guest as Apple's framework does on an -/// Intel or AMD Mac's GPU: no Apple GPU family (device info key 28 bit 0 off, key 37 Apple5's -/// fallback 1005, key 44 none), and no tile shaders, imageblocks, raster order groups, -/// framebuffer reads or extended XR10 formats, so the guest's compositor takes the paths it @@ -28,7 +28,7 @@ index a785e63..2def509 100644 - let on = std::env::var_os("REIMS_VGPU_NON_APPLE_GPU").is_some_and(|v| v == "1"); - if on { - log::info!("REIMS_VGPU_NON_APPLE_GPU: answering as a GPU of no Apple family (no tile shaders, imageblocks, raster order groups, framebuffer reads, XR10)"); -+/// macbox: `REIMS_VGPU_NON_APPLE_GPU` answers the guest as Apple's framework does on an ++/// LOCAL (Atlas): `REIMS_VGPU_NON_APPLE_GPU` answers the guest as Apple's framework does on an +/// Intel or AMD Mac's GPU, part by part, so the guest's compositor takes an immediate-mode GPU's +/// paths instead of a tiler's, which this platform emulates at a cost. Its value is `1` or `all` +/// for every part, or a comma list of: `families` (no Apple GPU family: device info key 28 bit 0 diff --git a/patches/reims-vgpu/0003-port-the-Vulkan-platform-to-metal2vulkan.patch b/patches/reims-vgpu/0003-port-the-Vulkan-platform-to-metal2vulkan.patch index 856c9c4..c38809b 100644 --- a/patches/reims-vgpu/0003-port-the-Vulkan-platform-to-metal2vulkan.patch +++ b/patches/reims-vgpu/0003-port-the-Vulkan-platform-to-metal2vulkan.patch @@ -1,10 +1,10 @@ -From 87d42a94bde8923a373c86ab661176fbd54bb904 Mon Sep 17 00:00:00 2001 +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: macbox Date: Wed, 28 Oct 2026 16:14:23 +0000 -Subject: [PATCH 3/3] port the Vulkan platform to metal2vulkan +Subject: [PATCH 03/25] port the Vulkan platform to metal2vulkan development 0db5825a -Builds on metal2vulkan 0db5825a with patches/metal2vulkan applied: +Builds on /srv/work/tmp/agent/reims-research/m2v-dev (branch atlas/dev-arl): upstream development 0db5825a, the env-gated Arrow Lake-S facts (M2V_ARL_FACTS, c7c4784c) and PixelFormatInfo::of / TextureLevelInfo::of (0d31d29c, the release's format table without a device). Cargo.toml's @@ -104,7 +104,7 @@ Co-Authored-By: Claude Opus 5.5 21 files changed, 239 insertions(+), 942 deletions(-) diff --git a/crates/reims-vgpu-vulkan/src/answer.rs b/crates/reims-vgpu-vulkan/src/answer.rs -index 31d3e13..a42adca 100644 +index 31d3e138..a42adca5 100644 --- a/crates/reims-vgpu-vulkan/src/answer.rs +++ b/crates/reims-vgpu-vulkan/src/answer.rs @@ -401,28 +401,20 @@ pub fn create( @@ -511,7 +511,7 @@ index 31d3e13..a42adca 100644 | Inexact::RateMap | Inexact::FillFormat diff --git a/crates/reims-vgpu-vulkan/src/answer/native.rs b/crates/reims-vgpu-vulkan/src/answer/native.rs -index d62ad80..573b84a 100644 +index d62ad80e..573b84ab 100644 --- a/crates/reims-vgpu-vulkan/src/answer/native.rs +++ b/crates/reims-vgpu-vulkan/src/answer/native.rs @@ -36,11 +36,11 @@ pub struct NativeRelay { @@ -585,7 +585,7 @@ index d62ad80..573b84a 100644 kind: rt::Degradation::HeapTextureLargerThanMetal, ground: "a heap texture's size and alignment are the device's image's, coarser than \ diff --git a/crates/reims-vgpu-vulkan/src/gpu.rs b/crates/reims-vgpu-vulkan/src/gpu.rs -index b31c5a4..7daf3f1 100644 +index b31c5a4b..7daf3f13 100644 --- a/crates/reims-vgpu-vulkan/src/gpu.rs +++ b/crates/reims-vgpu-vulkan/src/gpu.rs @@ -135,22 +135,16 @@ pub(crate) fn runtime_pixel_format(raw: u64) -> Option { @@ -618,7 +618,7 @@ index b31c5a4..7daf3f1 100644 held_by(raw, |raw| crate::gpu::format_info::described(sel, raw)) } diff --git a/crates/reims-vgpu-vulkan/src/gpu/caps.rs b/crates/reims-vgpu-vulkan/src/gpu/caps.rs -index 2def509..d522f41 100644 +index 2def509c..d522f41c 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/caps.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/caps.rs @@ -38,11 +38,11 @@ @@ -637,7 +637,7 @@ index 2def509..d522f41 100644 //! | the address an indirect command's arguments answer for a nil patch or control-point index buffer (`icb::NIL_BUFFER_ADDRESS`) | `0x1eeee1111` | Metal's private getters' answer, which no record holds | //! diff --git a/crates/reims-vgpu-vulkan/src/gpu/caps/tests/metal_tests.rs b/crates/reims-vgpu-vulkan/src/gpu/caps/tests/metal_tests.rs -index 2290c3b..54071ad 100644 +index 2290c3b0..54071ad1 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/caps/tests/metal_tests.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/caps/tests/metal_tests.rs @@ -171,11 +171,8 @@ fn stand_ins_are_the_metal_platforms_here() { @@ -655,7 +655,7 @@ index 2290c3b..54071ad 100644 ( f.supports_memory_order_atomics, diff --git a/crates/reims-vgpu-vulkan/src/gpu/desc.rs b/crates/reims-vgpu-vulkan/src/gpu/desc.rs -index 106de01..7f89814 100644 +index 106de019..7f898143 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/desc.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/desc.rs @@ -387,10 +387,7 @@ impl TextureFields { @@ -705,7 +705,7 @@ index 106de01..7f89814 100644 lacked(sel, raw); } diff --git a/crates/reims-vgpu-vulkan/src/gpu/device.rs b/crates/reims-vgpu-vulkan/src/gpu/device.rs -index a376dae..0b48b45 100644 +index a376daeb..0b48b459 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/device.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/device.rs @@ -233,12 +233,11 @@ static SURFACE_BUFFERS: Mutex>>> = Mutex::n @@ -757,7 +757,7 @@ index a376dae..0b48b45 100644 return Ok(None); } diff --git a/crates/reims-vgpu-vulkan/src/gpu/device/tests/metal_tests.rs b/crates/reims-vgpu-vulkan/src/gpu/device/tests/metal_tests.rs -index 8e9974b..461eb19 100644 +index 8e9974bf..461eb198 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/device/tests/metal_tests.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/device/tests/metal_tests.rs @@ -2448,11 +2448,8 @@ fn the_setter_takes_what_metals_takes() { @@ -775,7 +775,7 @@ index 8e9974b..461eb19 100644 } diff --git a/crates/reims-vgpu-vulkan/src/gpu/encode_blit/tests/metal_tests.rs b/crates/reims-vgpu-vulkan/src/gpu/encode_blit/tests/metal_tests.rs -index cd0f412..c912afd 100644 +index cd0f412e..c912afd5 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/encode_blit/tests/metal_tests.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/encode_blit/tests/metal_tests.rs @@ -2854,13 +2854,6 @@ fn two_bit_alpha_mipmaps_are_the_metal_platforms() { @@ -819,7 +819,7 @@ index cd0f412..c912afd 100644 } diff --git a/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs b/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs -index f5b6cac..82e1a20 100644 +index f5b6caca..82e1a207 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/encode_render.rs @@ -4560,28 +4560,20 @@ pub(crate) mod tests { @@ -870,7 +870,7 @@ index f5b6cac..82e1a20 100644 for format in [RGBA16F, 92] { end(&averaging, format); diff --git a/crates/reims-vgpu-vulkan/src/gpu/fill.rs b/crates/reims-vgpu-vulkan/src/gpu/fill.rs -index 1c30cac..e3fdf35 100644 +index 1c30cac8..e3fdf352 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/fill.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/fill.rs @@ -365,17 +365,15 @@ mod tests { @@ -896,7 +896,7 @@ index 1c30cac..e3fdf35 100644 } diff --git a/crates/reims-vgpu-vulkan/src/gpu/format_info.rs b/crates/reims-vgpu-vulkan/src/gpu/format_info.rs -index c5e1eea..5ec0ade 100644 +index c5e1eea4..5ec0ade8 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/format_info.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/format_info.rs @@ -1,25 +1,27 @@ @@ -1420,7 +1420,7 @@ index c5e1eea..5ec0ade 100644 #[cfg(target_os = "macos")] diff --git a/crates/reims-vgpu-vulkan/src/gpu/format_info/tests/metal_tests.rs b/crates/reims-vgpu-vulkan/src/gpu/format_info/tests/metal_tests.rs -index f57137c..73d4455 100644 +index f57137c8..73d44551 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/format_info/tests/metal_tests.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/format_info/tests/metal_tests.rs @@ -105,12 +105,8 @@ fn descriptions_are_the_metal_platforms_here() { @@ -1439,7 +1439,7 @@ index f57137c..73d4455 100644 (ycbcr8.name.as_deref(), ycbcr8.flags, ycbcr8.cast_class), (Some("MTLPixelFormatYCBCR8_420_2P"), 0x783_001d, cast_class) diff --git a/crates/reims-vgpu-vulkan/src/gpu/metal_space.rs b/crates/reims-vgpu-vulkan/src/gpu/metal_space.rs -index 7c97844..91107b3 100644 +index 7c97844c..91107b32 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/metal_space.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/metal_space.rs @@ -246,7 +246,7 @@ pub fn own_memory( @@ -1465,7 +1465,7 @@ index 7c97844..91107b3 100644 /// line names the texture. pub fn unmeasured_size(sel: &'static str, what: &str, desc: &rt::TextureDescriptor) { diff --git a/crates/reims-vgpu-vulkan/src/gpu/resource.rs b/crates/reims-vgpu-vulkan/src/gpu/resource.rs -index dfb3aaf..549138b 100644 +index dfb3aaf7..549138bc 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/resource.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/resource.rs @@ -1246,12 +1246,8 @@ fn indices(sel: &'static str, r: NsRange) -> std::ops::Range { @@ -1508,7 +1508,7 @@ index dfb3aaf..549138b 100644 /// The runtime's format of a view in `raw`, which `sel` makes, a value [`no_view`] is not nil in: diff --git a/crates/reims-vgpu-vulkan/src/gpu/resource/tests/metal_tests.rs b/crates/reims-vgpu-vulkan/src/gpu/resource/tests/metal_tests.rs -index 8b4d883..d1d242d 100644 +index 8b4d8835..d1d242d8 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/resource/tests/metal_tests.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/resource/tests/metal_tests.rs @@ -6218,7 +6218,6 @@ fn legacy_heap_formats() -> Vec<(u64, FormatBlock)> { @@ -1520,7 +1520,7 @@ index 8b4d883..d1d242d 100644 let block = crate::gpu::format_info::block(sel, raw, "a heap texture of"); let [bw, bh] = block.extent(); diff --git a/crates/reims-vgpu-vulkan/src/gpu/states.rs b/crates/reims-vgpu-vulkan/src/gpu/states.rs -index 3aefccc..535ed80 100644 +index 3aefccc3..535ed80f 100644 --- a/crates/reims-vgpu-vulkan/src/gpu/states.rs +++ b/crates/reims-vgpu-vulkan/src/gpu/states.rs @@ -1147,8 +1147,8 @@ mod tests { @@ -1548,7 +1548,7 @@ index 3aefccc..535ed80 100644 assert_eq!(raised.site().file, "resourceIndex"); let site = catch_death(|| { diff --git a/crates/reims-vgpu-vulkan/src/os.rs b/crates/reims-vgpu-vulkan/src/os.rs -index 4aba15f..dc71a67 100644 +index 4aba15ff..dc71a67c 100644 --- a/crates/reims-vgpu-vulkan/src/os.rs +++ b/crates/reims-vgpu-vulkan/src/os.rs @@ -36,7 +36,7 @@ pub(crate) use host_os::open_loader; @@ -1561,7 +1561,7 @@ index 4aba15f..dc71a67 100644 }; pub use host_os::{ diff --git a/crates/reims-vgpu-vulkan/src/os/linux.rs b/crates/reims-vgpu-vulkan/src/os/linux.rs -index 3a4a7fa..910e1ee 100644 +index 3a4a7fa8..910e1ee2 100644 --- a/crates/reims-vgpu-vulkan/src/os/linux.rs +++ b/crates/reims-vgpu-vulkan/src/os/linux.rs @@ -120,18 +120,6 @@ pub fn install_process_services() { @@ -1584,7 +1584,7 @@ index 3a4a7fa..910e1ee 100644 /// none, as there is no Metal here to ask, so the runtime's own stands. pub fn default_registry_id() -> Option { diff --git a/crates/reims-vgpu-vulkan/src/os/macos.rs b/crates/reims-vgpu-vulkan/src/os/macos.rs -index 24331d6..6975a25 100644 +index 24331d6f..6975a25b 100644 --- a/crates/reims-vgpu-vulkan/src/os/macos.rs +++ b/crates/reims-vgpu-vulkan/src/os/macos.rs @@ -86,11 +86,6 @@ pub(crate) fn open_loader() -> Result { @@ -1600,7 +1600,7 @@ index 24331d6..6975a25 100644 /// of Metal once, on the first call that finds a device: `None` where Metal has none. It is /// the registry ID of the device MoltenVK runs the runtime's default device on where the Mac diff --git a/crates/reims-vgpu-vulkan/src/platform.rs b/crates/reims-vgpu-vulkan/src/platform.rs -index 1438607..8c7771d 100644 +index 1438607d..8c7771db 100644 --- a/crates/reims-vgpu-vulkan/src/platform.rs +++ b/crates/reims-vgpu-vulkan/src/platform.rs @@ -14,7 +14,6 @@ diff --git a/patches/reims-vgpu/0004-REIMS_VGPU_VBLANK-1-re-raises-an-idle-di.patch b/patches/reims-vgpu/0004-REIMS_VGPU_VBLANK-1-re-raises-an-idle-di.patch new file mode 100644 index 0000000..19b9fb5 --- /dev/null +++ b/patches/reims-vgpu/0004-REIMS_VGPU_VBLANK-1-re-raises-an-idle-di.patch @@ -0,0 +1,747 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Wed, 28 Oct 2026 20:24:14 +0000 +Subject: [PATCH 04/25] REIMS_VGPU_VBLANK=1 re-raises an idle + display's present interrupt + +A guest display that has gone a whole refresh period without a present raises +the present interrupt (2) again for the frame it presented last, once a period, +phase-locked to that frame's completion, as a display scanning it out at each +vblank would. Nothing else in the page changes (present count, presented +surface, fieldfa4) and there is no GPU work. It starts with the first present, +yields to every present, stops while the guest sleeps (bit 0), is paused or +restoring, or loses its client, and starts again with the next present. The +period is the host modes' highest refresh rate (24-240 Hz), else 60 Hz. Each +tick is a trace event (vblank ); the first tick logs the +guest's enabled interrupt bits. Off unless the variable is 1. + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/display/guest_display.rs | 28 ++ + .../src/display/guest_display/present.rs | 7 + + .../src/display/guest_display/settings.rs | 9 + + .../src/display/guest_display/tests.rs | 233 ++++++++++++ + .../src/display/guest_display/vblank.rs | 340 ++++++++++++++++++ + 5 files changed, 617 insertions(+) + create mode 100644 crates/reims-vgpu-core/src/display/guest_display/vblank.rs + +diff --git a/crates/reims-vgpu-core/src/display/guest_display.rs b/crates/reims-vgpu-core/src/display/guest_display.rs +index fa75b3f3..c6d0714e 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display.rs +@@ -62,6 +62,7 @@ mod present; + mod settings; + #[cfg(test)] + mod tests; ++mod vblank; + + pub use cursor::CursorGlyphArgs; + +@@ -353,6 +354,8 @@ pub struct GuestDisplay { + pub(crate) guest_display_mutex: Mutex>, + /// What is read without it. + pub(crate) unlocked: GuestDisplayUnlocked

, ++ /// LOCAL (Atlas): the idle vblank ticker, with `REIMS_VGPU_VBLANK=1` ([`vblank`]). ++ vblank: Option>, + } + + /// The shared page, each access one volatile or atomic access of the field's width (D18). +@@ -426,6 +429,23 @@ impl GuestDisplay

{ + device: Weak>, + port: u64, + shared_state_page: u32, ++ ) -> RaiseResult> { ++ Self::new_with_vblank( ++ platform, ++ device, ++ port, ++ shared_state_page, ++ vblank::enabled_by_env(), ++ ) ++ } ++ ++ /// [`GuestDisplay::new`], with the idle vblank ticker or without it ([`vblank`]). ++ pub(crate) fn new_with_vblank( ++ platform: Arc

, ++ device: Weak>, ++ port: u64, ++ shared_state_page: u32, ++ with_vblank: bool, + ) -> RaiseResult> { + // A gone device, or no mapping, gives a NULL address. + let mapping = match device.upgrade() { +@@ -478,6 +498,7 @@ impl GuestDisplay

{ + }), + this: this.clone(), + }, ++ vblank: with_vblank.then(vblank::Vblank::new), + }); + log::info!("guest display on port {port} created"); + { +@@ -615,6 +636,9 @@ impl GuestDisplay

{ + slot.display.is_none() || client.is_none(), + DeathKind::ClientAlreadySet + ); ++ if client.is_none() { ++ self.vblank_idle(); ++ } + *slot = Client { + display: client, + core, +@@ -801,6 +825,7 @@ impl GuestDisplay

{ + crate::fatal_unless!(!l.paused, DeathKind::PauseTwice); + l.paused = true; + self.unlocked.paused.store(true, Ordering::SeqCst); ++ self.vblank_idle(); + } + + /// Unpauses the guest display and runs the state machine. Only the device's unpause, an API +@@ -891,6 +916,9 @@ impl Drop for GuestDisplay

{ + /// The guest display's end: the page's mapping (its unmap), the mode list, then the log. The + /// guest colour space goes with the fields. + fn drop(&mut self) { ++ if let Some(vb) = &self.vblank { ++ vb.stop(); ++ } + let l = self.guest_display_mutex.get_mut(); + drop(l.mapping.take()); + drop(l.mode_list.take()); +diff --git a/crates/reims-vgpu-core/src/display/guest_display/present.rs b/crates/reims-vgpu-core/src/display/guest_display/present.rs +index d04efe7a..0a2582b7 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/present.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/present.rs +@@ -133,6 +133,8 @@ impl GuestDisplay

{ + { + guest_display.poke(); + } ++ // LOCAL (Atlas): the idle vblank's schedule starts from this completion. ++ guest_display.vblank_presented(); + } + }) + } +@@ -615,6 +617,11 @@ impl GuestDisplay

{ + }); + } + l.asleep = asleep; ++ // LOCAL (Atlas): no idle vblank while the guest sleeps (bit 0, as the presents test it), ++ // until its next present. ++ if asleep & 1 != 0 { ++ self.vblank_idle(); ++ } + // The raw byte. + self.page(l).w8(sp::ASLEEP, asleep); + } +diff --git a/crates/reims-vgpu-core/src/display/guest_display/settings.rs b/crates/reims-vgpu-core/src/display/guest_display/settings.rs +index 215225e1..5c93acd4 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/settings.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/settings.rs +@@ -161,6 +161,15 @@ impl GuestDisplay

{ + DeathKind::ModeCountTooLarge + ); + let modes = list.get(..usize::from(sp::MAX_MODES)).unwrap_or(&*list); ++ // LOCAL (Atlas): the idle vblank's period, from the modes' highest refresh rate. ++ if let Some(vb) = &self.vblank { ++ vb.set_refresh_hz( ++ modes ++ .iter() ++ .map(|m| m.refresh_rate_in_hz) ++ .fold(f64::NAN, f64::max), ++ ); ++ } + let count16 = u16::try_from(modes.len()).unwrap_or(sp::MAX_MODES); + let mut changed = false; + if self.page(l).r16(sp::MODE_LIST_COUNT) != count16 { +diff --git a/crates/reims-vgpu-core/src/display/guest_display/tests.rs b/crates/reims-vgpu-core/src/display/guest_display/tests.rs +index e602e806..851c7485 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/tests.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/tests.rs +@@ -210,6 +210,23 @@ impl Fx { + Fx::with_port(0) + } + ++ /// LOCAL (Atlas): a guest display on port 0 with the idle vblank ticker on. ++ fn with_vblank() -> Fx { ++ let td = guest_display_device(); ++ let p = td.platform.clone(); ++ let host = host_of(&td); ++ let guest_display = ++ GuestDisplay::new_with_vblank(p.clone(), Arc::downgrade(&host), 0, PAGE, true) ++ .unwrap(); ++ p.call_log().clear(); ++ Fx { ++ p, ++ _td: td, ++ host, ++ guest_display, ++ } ++ } ++ + fn r8(&self, off: u64) -> u8 { + self.page().read_u8(off, ANY) + } +@@ -2021,3 +2038,219 @@ mod handler_releases_the_display { + assert_eq!(*seen.lock(), vec!["pos=1 props=1"]); + } + } ++ ++/// LOCAL (Atlas): `REIMS_VGPU_VBLANK=1`'s idle vblank (`vblank.rs`), on the virtual clock. ++mod vblank { ++ use super::*; ++ use crate::display::guest_display::vblank::{Vblank, period_of}; ++ ++ /// The real time a tick that is due may take to be raised. ++ const REAL: Duration = Duration::from_secs(10); ++ /// The real time given to a tick that must not come. ++ const SETTLE: Duration = Duration::from_millis(40); ++ const MS: Duration = Duration::from_millis(1); ++ ++ /// The default period (no modes): 60 Hz. ++ fn period() -> Duration { ++ period_of(60.0) ++ } ++ ++ fn vb(fx: &Fx) -> &Arc { ++ fx.guest_display.vblank.as_ref().unwrap() ++ } ++ ++ /// A mapped present with no client: it completes at once (the completion raises 2). ++ fn present(fx: &Fx) { ++ let s = fx.p.fake_surfaces().make_simple(64, 32, FOURCC_BGRA, 4); ++ let task = fx.host.get_kernel_task(OnFault::Return).unwrap(); ++ let mut l = fx.guest_display.guest_display_mutex.lock(); ++ let keep = fx.guest_display.client_core(); ++ present_mapped_with_surface( ++ Hold::NONE, ++ &fx.guest_display, ++ &mut l, ++ keep.as_ref(), ++ task, ++ Some(s), ++ 5, ++ GammaTableArgs::default(), ++ ) ++ .unwrap(); ++ } ++ ++ fn advance(fx: &Fx, d: Duration) { ++ fx.p.virtual_clock().advance(d); ++ } ++ ++ /// Waits (real time) for `n` ticks raised in all. ++ fn raised_reaches(fx: &Fx, n: u64) -> u64 { ++ let mut left = REAL; ++ while vb(fx).raised() < n && !left.is_zero() { ++ std::thread::sleep(MS); ++ left = left.saturating_sub(MS); ++ } ++ vb(fx).raised() ++ } ++ ++ /// The ticks raised once a tick that must not come has had its chance. ++ fn raised_after_settling(fx: &Fx) -> u64 { ++ std::thread::sleep(SETTLE); ++ vb(fx).raised() ++ } ++ ++ /// Off without the switch: no ticker, and a present raises once ++ #[test] ++ fn off_without_the_switch() { ++ let fx = Fx::new(); ++ assert!(fx.guest_display.vblank.is_none()); ++ present(&fx); ++ fx.w32(sp::RAISED_INTERRUPTS, 0); ++ advance(&fx, period() * 5); ++ std::thread::sleep(SETTLE); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), 0); ++ } ++ ++ /// Starts after a period with no present, raises the present bit (poking an enabled port) ++ /// once a period, and leaves the page's present record alone ++ #[test] ++ fn starts_when_idle_and_keeps_the_period() { ++ let fx = Fx::with_vblank(); ++ assert!(!vb(&fx).started()); ++ fx.w32(sp::ENABLED_INTERRUPTS, INTERRUPT_PRESENT); ++ present(&fx); ++ assert!(vb(&fx).started()); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), INTERRUPT_PRESENT); ++ assert_eq!(fx.r64(sp::PRESENT_COUNT), 1); ++ fx.w32(sp::RAISED_INTERRUPTS, 0); ++ let pokes = fx.pokes(); ++ ++ // Not before a whole period. ++ advance(&fx, period() - MS); ++ assert_eq!(raised_after_settling(&fx), 0); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), 0); ++ ++ // At the period: one tick, as the completion raises it. ++ advance(&fx, MS); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), INTERRUPT_PRESENT); ++ assert_eq!(fx.pokes(), pokes + 1); ++ assert_eq!(fx.r64(sp::PRESENT_COUNT), 1); ++ assert_eq!(fx.r64(sp::PRESENTED_MAPPING_ID), 5); ++ ++ // And one each period after. ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 2), 2); ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 3), 3); ++ assert_eq!(raised_after_settling(&fx), 3); ++ assert_eq!(fx.pokes(), pokes + 3); ++ } ++ ++ /// The bit masked: the tick sets it and pokes nothing, as the completion does ++ #[test] ++ fn masked_bit_is_set_without_a_poke() { ++ let fx = Fx::with_vblank(); ++ present(&fx); ++ fx.w32(sp::RAISED_INTERRUPTS, 0); ++ let pokes = fx.pokes(); ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), INTERRUPT_PRESENT); ++ assert_eq!(fx.pokes(), pokes); ++ } ++ ++ /// A late thread raises once, from the newest due point: no burst ++ #[test] ++ fn late_raises_once() { ++ let fx = Fx::with_vblank(); ++ present(&fx); ++ advance(&fx, period() * 5 + MS); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ assert_eq!(raised_after_settling(&fx), 1); ++ // The next on the schedule: 6 periods after the present. ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 2), 2); ++ assert_eq!(raised_after_settling(&fx), 2); ++ } ++ ++ /// Presents that come faster than the period get no tick; the schedule starts from the last ++ #[test] ++ fn stops_while_presents_come() { ++ let fx = Fx::with_vblank(); ++ let half = period() / 2; ++ present(&fx); ++ for _ in 0..6 { ++ advance(&fx, half); ++ present(&fx); ++ } ++ advance(&fx, half); ++ assert_eq!(raised_after_settling(&fx), 0); ++ // A whole period after the last present. ++ advance(&fx, period() - half); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ // A present again: nothing until a period after it. ++ present(&fx); ++ advance(&fx, half); ++ assert_eq!(raised_after_settling(&fx), 1); ++ advance(&fx, period() - half); ++ assert_eq!(raised_reaches(&fx, 2), 2); ++ } ++ ++ /// Sleep stops the ticks; a wake alone does not start them; the next present does ++ #[test] ++ fn respects_sleep() { ++ let fx = Fx::with_vblank(); ++ present(&fx); ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ ++ fx.guest_display.sleep_state(1); ++ advance(&fx, period() * 4); ++ assert_eq!(raised_after_settling(&fx), 1); ++ // A present while asleep: the tick finds the sleep and raises nothing. ++ present(&fx); ++ fx.w32(sp::RAISED_INTERRUPTS, 0); ++ advance(&fx, period() * 4); ++ assert_eq!(raised_after_settling(&fx), 1); ++ assert_eq!(fx.r32(sp::RAISED_INTERRUPTS), 0); ++ ++ fx.guest_display.sleep_state(0); ++ advance(&fx, period() * 4); ++ assert_eq!(raised_after_settling(&fx), 1); ++ ++ present(&fx); ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 2), 2); ++ } ++ ++ /// Paused, the ticker leaves the page alone, and starts again with the next present ++ #[test] ++ fn paused_touches_nothing() { ++ let fx = Fx::with_vblank(); ++ present(&fx); ++ fx.guest_display.pause(); ++ advance(&fx, period() * 4); ++ assert_eq!(raised_after_settling(&fx), 0); ++ fx.guest_display.unpause().unwrap(); ++ advance(&fx, period() * 4); ++ assert_eq!(raised_after_settling(&fx), 0); ++ present(&fx); ++ advance(&fx, period()); ++ assert_eq!(raised_reaches(&fx, 1), 1); ++ } ++ ++ /// The period is the host modes' highest refresh rate ++ #[test] ++ fn period_follows_the_modes() { ++ let fx = Fx::with_vblank(); ++ assert_eq!(vb(&fx).period(), period_of(60.0)); ++ fx.guest_display ++ .set_mode_list( ++ OnFault::Return, ++ Arc::from([mode(3440, 1440, 60.0), mode(1920, 1080, 120.0)]), ++ ) ++ .unwrap(); ++ assert!(sync_mode_list(&fx.guest_display)); ++ assert_eq!(vb(&fx).period(), period_of(120.0)); ++ } ++} +diff --git a/crates/reims-vgpu-core/src/display/guest_display/vblank.rs b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +new file mode 100644 +index 00000000..5c708185 +--- /dev/null ++++ b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +@@ -0,0 +1,340 @@ ++//! LOCAL (Atlas): `REIMS_VGPU_VBLANK=1`, an idle display's vblank. ++//! ++//! Apple's framework gives a display the host's display timing; this library's Linux path has ++//! none, so a guest display raises its present interrupt only when a frame the guest presented is ++//! done. With the variable set to `1`, a guest display that has gone a whole refresh period ++//! without a present raises the present interrupt again, for the frame it presented last, once a ++//! period, on the schedule of that frame's completion (`t0 + k * period`, `k >= 1`), as a display ++//! that scanned the frame out again at each vblank would. It writes nothing else in the page: the ++//! present count, the presented surface and `fieldfa4` stay those of the guest's last present, and ++//! it does no GPU work. ++//! ++//! - **Starts** with the first present's completion, on a thread of its own (one per guest ++//! display that presents). ++//! - **Yields** to the guest's presents: each one starts the schedule again from its completion, ++//! so a guest presenting at the refresh rate or faster gets no tick at all. ++//! - **Stops** while the guest sleeps (bit 0 of its sleep byte, as the presents test it), when the ++//! client goes, and while the guest display is paused or restoring; the ticks start again with ++//! the next present after it. ++//! - **The period** is the host's modes' highest refresh rate, 24 to 240 Hz, else 60 Hz. ++//! - **A late tick** (the thread was not run on time) raises once and goes on from the newest due ++//! point: no burst. ++//! ++//! The variable is read once per process; unset, or any other value than `1`, no guest display ++//! has a ticker and nothing here runs. Each tick is a trace event (`vblank `), and the first tick of the process logs the guest's ++//! enabled interrupt bits, so a run shows whether the guest takes the bit at all. ++ ++use std::sync::atomic::{AtomicBool, Ordering}; ++use std::sync::{Arc, OnceLock, Weak}; ++use std::time::Duration; ++ ++use super::{GuestDisplay, INTERRUPT_PRESENT}; ++use crate::display::page as sp; ++use crate::platform::Platform; ++use crate::platform::os::{Clock as _, Deadline, MonoTime}; ++use crate::sync::{Condvar, Mutex}; ++ ++/// The variable: `1` turns the ticks on. ++pub const ENV_VAR: &str = "REIMS_VGPU_VBLANK"; ++ ++/// The refresh rate without a usable mode. ++const DEFAULT_HZ: f64 = 60.0; ++/// The lowest refresh rate taken from a mode. ++const MIN_HZ: f64 = 24.0; ++/// The highest refresh rate taken from a mode. ++const MAX_HZ: f64 = 240.0; ++/// The ticker thread's stack. ++const STACK: usize = 128 * 1024; ++ ++/// Whether the variable turns the ticks on (read once). ++pub(crate) fn enabled_by_env() -> bool { ++ static ON: OnceLock = OnceLock::new(); ++ *ON.get_or_init(|| std::env::var_os(ENV_VAR).is_some_and(|v| v == "1")) ++} ++ ++/// Whether the first tick of the process was logged. ++static FIRST_TICK_LOGGED: AtomicBool = AtomicBool::new(false); ++ ++/// The period of `hz`, or of 60 Hz outside 24-240 Hz (or not a number). ++pub(crate) fn period_of(hz: f64) -> Duration { ++ let hz = if (MIN_HZ..=MAX_HZ).contains(&hz) { ++ hz ++ } else { ++ DEFAULT_HZ ++ }; ++ Duration::from_secs_f64(1.0 / hz) ++} ++ ++/// What the ticker's thread and its guest display share, under [`Vblank::state`]. ++#[derive(Debug)] ++struct State { ++ /// The thread was started. ++ started: bool, ++ /// The guest display ended: the thread leaves. ++ stopped: bool, ++ /// The monotonic time of the last present's completion, in nanoseconds; `None` before the ++ /// first, and after a sleep, a pause or the client's going until the next. ++ last_present: Option, ++ /// Counts the presents, so a tick decided before one is dropped. ++ epoch: u64, ++ /// The schedule points passed since the last present (the `k` of the last tick). ++ ticks: u64, ++ /// The period, in nanoseconds. ++ period_ns: u64, ++ /// Ticks raised, all told. ++ raised: u64, ++} ++ ++/// A guest display's ticker (module doc). ++#[derive(Debug)] ++pub(crate) struct Vblank { ++ state: Mutex, ++ cv: Condvar, ++} ++ ++/// What a due tick hands the guest display. ++#[derive(Debug, Clone, Copy)] ++pub(crate) struct Tick { ++ epoch: u64, ++ k: u64, ++} ++ ++impl Vblank { ++ /// A ticker at the default period, not started. ++ pub(crate) fn new() -> Arc { ++ Arc::new(Vblank { ++ state: Mutex::new(State { ++ started: false, ++ stopped: false, ++ last_present: None, ++ epoch: 0, ++ ticks: 0, ++ period_ns: nanos(period_of(DEFAULT_HZ)), ++ raised: 0, ++ }), ++ cv: Condvar::new(), ++ }) ++ } ++ ++ /// The modes' highest refresh rate sets the period (module doc). ++ pub(crate) fn set_refresh_hz(&self, hz: f64) { ++ let period = nanos(period_of(hz)); ++ let mut s = self.state.lock(); ++ if s.period_ns != period { ++ s.period_ns = period; ++ self.cv.notify_all(); ++ } ++ } ++ ++ /// The period now. ++ pub(crate) fn period(&self) -> Duration { ++ Duration::from_nanos(self.state.lock().period_ns) ++ } ++ ++ /// A present completed at `now_ns`: the schedule starts again from it. Answers whether the ++ /// thread is to be started (the first present). ++ pub(crate) fn presented(&self, now_ns: u64) -> bool { ++ let mut s = self.state.lock(); ++ s.last_present = Some(now_ns); ++ s.epoch = s.epoch.wrapping_add(1); ++ s.ticks = 0; ++ self.cv.notify_all(); ++ let start = !s.started && !s.stopped; ++ s.started = true; ++ start ++ } ++ ++ /// No ticks until the next present: the guest sleeps, the client went, or a pause. ++ pub(crate) fn idle(&self) { ++ let mut s = self.state.lock(); ++ s.last_present = None; ++ s.epoch = s.epoch.wrapping_add(1); ++ self.cv.notify_all(); ++ } ++ ++ /// The guest display ended: the thread leaves. ++ pub(crate) fn stop(&self) { ++ self.state.lock().stopped = true; ++ self.cv.notify_all(); ++ } ++ ++ /// Whether `tick` is still the newest: no present, sleep or stop since it was decided. ++ fn current(&self, tick: Tick) -> bool { ++ let s = self.state.lock(); ++ !s.stopped && s.epoch == tick.epoch ++ } ++ ++ /// Counts a raised tick. ++ fn count(&self) -> u64 { ++ let mut s = self.state.lock(); ++ s.raised = s.raised.wrapping_add(1); ++ s.raised ++ } ++ ++ /// Ticks raised, all told (tests). ++ #[cfg(test)] ++ pub(crate) fn raised(&self) -> u64 { ++ self.state.lock().raised ++ } ++ ++ /// Whether the thread was started (tests). ++ #[cfg(test)] ++ pub(crate) fn started(&self) -> bool { ++ self.state.lock().started ++ } ++ ++ /// Waits for the next due tick on `clock`: `None` once stopped. ++ fn next_tick(&self, clock: &C) -> Option { ++ let mut s = self.state.lock(); ++ loop { ++ if s.stopped { ++ return None; ++ } ++ let Some(t0) = s.last_present else { ++ clock.wait(&self.cv, &mut s); ++ continue; ++ }; ++ let period = s.period_ns.max(1); ++ let k = s.ticks.saturating_add(1); ++ let due = t0.saturating_add(k.saturating_mul(period)); ++ let now = clock.now_mono().as_nanos(); ++ if now < due { ++ let _ = clock.wait_until( ++ &self.cv, ++ &mut s, ++ Deadline::Mono(MonoTime::from_nanos(due)), ++ ); ++ continue; ++ } ++ // Late: the newest due point, once. ++ let k = now ++ .saturating_sub(t0) ++ .checked_div(period) ++ .unwrap_or(k) ++ .max(k); ++ s.ticks = k; ++ return Some(Tick { epoch: s.epoch, k }); ++ } ++ } ++} ++ ++/// Nanoseconds of `d`, saturating. ++fn nanos(d: Duration) -> u64 { ++ u64::try_from(d.as_nanos()).unwrap_or(u64::MAX) ++} ++ ++impl GuestDisplay

{ ++ /// A present completed: the ticker's schedule starts again, and its thread with the first. ++ pub(super) fn vblank_presented(&self) { ++ let Some(vb) = &self.vblank else { ++ return; ++ }; ++ let now = self.platform.clock().now_mono().as_nanos(); ++ if vb.presented(now) { ++ self.start_vblank_thread(Arc::clone(vb)); ++ } ++ } ++ ++ /// No ticks until the next present. ++ pub(super) fn vblank_idle(&self) { ++ if let Some(vb) = &self.vblank { ++ vb.idle(); ++ } ++ } ++ ++ /// The ticker's thread: holds the guest display weakly, and leaves when it is gone or stopped. ++ fn start_vblank_thread(&self, vb: Arc) { ++ let this: Weak> = self.unlocked.this.clone(); ++ let platform = Arc::clone(&self.platform); ++ log::info!( ++ "guest display on port {}: vblank ticks on ({ENV_VAR}=1), every {:?} while idle", ++ self.port, ++ vb.period() ++ ); ++ crate::rt::thread::spawn_named( ++ self.platform.threads(), ++ "reims-vgpu.vblank", ++ STACK, ++ Box::new(()), ++ move || { ++ while let Some(tick) = vb.next_tick(platform.clock()) { ++ let Some(guest_display) = this.upgrade() else { ++ return; ++ }; ++ guest_display.vblank_tick(&vb, tick); ++ } ++ }, ++ ); ++ } ++ ++ /// One tick: the present interrupt raised again, as the present's completion raises it, ++ /// unless the guest presented, slept or was paused since the tick was decided. ++ fn vblank_tick(&self, vb: &Vblank, tick: Tick) { ++ let l = self.guest_display_mutex.lock(); ++ if l.paused || l.will_restore || l.asleep & 1 != 0 { ++ drop(l); ++ vb.idle(); ++ return; ++ } ++ if !vb.current(tick) { ++ return; ++ } ++ let p = self.page(&l); ++ p.fetch_or(sp::RAISED_INTERRUPTS, INTERRUPT_PRESENT); ++ let enabled = p.load(sp::ENABLED_INTERRUPTS); ++ if enabled & INTERRUPT_PRESENT != 0 { ++ self.poke(); ++ } ++ let raised = vb.count(); ++ drop(l); ++ // vblank: schedule points since the last present, the guest's enabled interrupt bits ++ crate::trace::event("vblank", &[tick.k, u64::from(enabled)]); ++ if !FIRST_TICK_LOGGED.swap(true, Ordering::Relaxed) { ++ log::info!( ++ "guest display on port {}: first vblank tick ({raised} raised); the guest enables \ ++ interrupts {enabled:#x}, so the tick {} the port", ++ self.port, ++ if enabled & INTERRUPT_PRESENT != 0 { ++ "pokes" ++ } else { ++ "only sets the bit and does not poke" ++ } ++ ); ++ } ++ } ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ ++ /// 60 Hz outside 24-240 Hz; the mode's rate inside ++ #[test] ++ fn period_follows_the_mode_within_bounds() { ++ assert_eq!(period_of(60.0), Duration::from_secs_f64(1.0 / 60.0)); ++ assert_eq!(period_of(120.0), Duration::from_secs_f64(1.0 / 120.0)); ++ assert_eq!(period_of(0.0), period_of(60.0)); ++ assert_eq!(period_of(-1.0), period_of(60.0)); ++ assert_eq!(period_of(f64::NAN), period_of(60.0)); ++ assert_eq!(period_of(1000.0), period_of(60.0)); ++ } ++ ++ /// A present starts the thread once, and restarts the schedule ++ #[test] ++ fn presented_starts_once_and_restarts_the_schedule() { ++ let vb = Vblank::new(); ++ assert!(vb.presented(10)); ++ assert!(!vb.presented(20)); ++ let s = vb.state.lock(); ++ assert_eq!((s.last_present, s.epoch, s.ticks), (Some(20), 2, 0)); ++ } ++ ++ /// A stopped ticker starts no thread ++ #[test] ++ fn stopped_starts_no_thread() { ++ let vb = Vblank::new(); ++ vb.stop(); ++ assert!(!vb.presented(10)); ++ } ++} diff --git a/patches/reims-vgpu/0005-REIMS_VGPU_VBLANK_HZ-sets-the-ticks-rate.patch b/patches/reims-vgpu/0005-REIMS_VGPU_VBLANK_HZ-sets-the-ticks-rate.patch new file mode 100644 index 0000000..38c1987 --- /dev/null +++ b/patches/reims-vgpu/0005-REIMS_VGPU_VBLANK_HZ-sets-the-ticks-rate.patch @@ -0,0 +1,67 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 21:27:12 +0000 +Subject: [PATCH 05/25] REIMS_VGPU_VBLANK_HZ sets the ticks' + rate apart from the modes + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/display/guest_display/vblank.rs | 17 ++++++++++++++++- + 1 file changed, 16 insertions(+), 1 deletion(-) + +diff --git a/crates/reims-vgpu-core/src/display/guest_display/vblank.rs b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +index 5c708185..df45dc39 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/vblank.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +@@ -17,6 +17,8 @@ + //! client goes, and while the guest display is paused or restoring; the ticks start again with + //! the next present after it. + //! - **The period** is the host's modes' highest refresh rate, 24 to 240 Hz, else 60 Hz. ++//! `REIMS_VGPU_VBLANK_HZ`, 24 to 1000, replaces it and leaves the modes as they are, so the ++//! ticks can run faster than the rate the guest was told. + //! - **A late tick** (the thread was not run on time) raises once and goes on from the newest due + //! point: no burst. + //! +@@ -37,6 +39,8 @@ use crate::sync::{Condvar, Mutex}; + + /// The variable: `1` turns the ticks on. + pub const ENV_VAR: &str = "REIMS_VGPU_VBLANK"; ++/// The variable: the ticks' rate in Hz, instead of the modes' (module doc). ++pub const HZ_ENV_VAR: &str = "REIMS_VGPU_VBLANK_HZ"; + + /// The refresh rate without a usable mode. + const DEFAULT_HZ: f64 = 60.0; +@@ -44,6 +48,8 @@ const DEFAULT_HZ: f64 = 60.0; + const MIN_HZ: f64 = 24.0; + /// The highest refresh rate taken from a mode. + const MAX_HZ: f64 = 240.0; ++/// The highest rate [`HZ_ENV_VAR`] takes. ++const MAX_ENV_HZ: f64 = 1000.0; + /// The ticker thread's stack. + const STACK: usize = 128 * 1024; + +@@ -53,6 +59,15 @@ pub(crate) fn enabled_by_env() -> bool { + *ON.get_or_init(|| std::env::var_os(ENV_VAR).is_some_and(|v| v == "1")) + } + ++/// The rate [`HZ_ENV_VAR`] sets, if it is a number from 24 to 1000 (read once). ++fn env_hz() -> Option { ++ static HZ: OnceLock> = OnceLock::new(); ++ *HZ.get_or_init(|| { ++ let hz: f64 = std::env::var(HZ_ENV_VAR).ok()?.parse().ok()?; ++ (MIN_HZ..=MAX_ENV_HZ).contains(&hz).then_some(hz) ++ }) ++} ++ + /// Whether the first tick of the process was logged. + static FIRST_TICK_LOGGED: AtomicBool = AtomicBool::new(false); + +@@ -119,7 +134,7 @@ impl Vblank { + + /// The modes' highest refresh rate sets the period (module doc). + pub(crate) fn set_refresh_hz(&self, hz: f64) { +- let period = nanos(period_of(hz)); ++ let period = env_hz().map_or_else(|| nanos(period_of(hz)), |hz| nanos(Duration::from_secs_f64(1.0 / hz))); + let mut s = self.state.lock(); + if s.period_ns != period { + s.period_ns = period; diff --git a/patches/reims-vgpu/0006-the-touch-census-s-news-of-surfaces-and-.patch b/patches/reims-vgpu/0006-the-touch-census-s-news-of-surfaces-and-.patch new file mode 100644 index 0000000..87a7c3d --- /dev/null +++ b/patches/reims-vgpu/0006-the-touch-census-s-news-of-surfaces-and-.patch @@ -0,0 +1,1179 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 21:48:57 +0000 +Subject: [PATCH 06/25] the touch census's news of surfaces and + presents + +rvg_census_set_callbacks gives QEMU's census each surface's pages and facts at map, its unmap +before the guest sees it, textures and presents. Without it, one atomic load per event. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-c/include/reims_vgpu.h | 62 ++ + crates/reims-vgpu-c/src/census.rs | 215 +++++++ + crates/reims-vgpu-c/src/lib.rs | 1 + + crates/reims-vgpu-core/src/census.rs | 64 ++ + .../src/display/guest_display/present.rs | 6 + + crates/reims-vgpu-core/src/iosurface/table.rs | 115 ++++ + crates/reims-vgpu-core/src/lib.rs | 1 + + .../src/resources/manager/texture.rs | 3 + + docs/api.md | 1 + + scripts/internal/census/census_test.c | 548 ++++++++++++++++++ + 10 files changed, 1016 insertions(+) + create mode 100644 crates/reims-vgpu-c/src/census.rs + create mode 100644 crates/reims-vgpu-core/src/census.rs + create mode 100644 scripts/internal/census/census_test.c + +diff --git a/crates/reims-vgpu-c/include/reims_vgpu.h b/crates/reims-vgpu-c/include/reims_vgpu.h +index f0b07f49..705af8c0 100644 +--- a/crates/reims-vgpu-c/include/reims_vgpu.h ++++ b/crates/reims-vgpu-c/include/reims_vgpu.h +@@ -329,6 +329,61 @@ typedef struct { + size_t range_count; + } RvgMemoryMap; + ++// What a surface's descriptor says of it. ++typedef struct { ++ uint32_t mapping_id; ++ uint32_t pixel_format; ++ uint64_t width; ++ uint64_t height; ++ uint64_t bytes_per_row; ++ uint64_t alloc_size; ++ uint64_t protection_options; ++ uint32_t address_format; ++ uint32_t compression_type; ++ bool read_only; ++ bool host_only; ++} RvgCensusSurface; ++ ++// `mapped(ctx, surface, pages, page_count, page_size)`: a surface was mapped, with each guest ++// page's host address. ++// ++// Thread: the IOSurface device's own, under its state's lock. May call: nothing of this header. ++typedef void (*RvgCensusMappedFn)(void *ctx, ++ const RvgCensusSurface *surface, ++ const uint64_t *pages, ++ uintptr_t page_count, ++ uint64_t page_size); ++ ++// `unmapped(ctx, mapping_id)`: the surface goes, and its pages may be reused when this returns. ++// ++// Thread: the IOSurface device's own, under its state's lock. May call: nothing of this header. ++typedef void (*RvgCensusUnmappedFn)(void *ctx, uint32_t mapping_id); ++ ++// `used(ctx, mapping_id, usage, storage_mode)`: a texture was made over the surface. ++// ++// Thread: the task's that made it. May call: nothing of this header. ++typedef void (*RvgCensusUsedFn)(void *ctx, ++ uint32_t mapping_id, ++ uint64_t usage, ++ uint64_t storage_mode); ++ ++// `presented(ctx, mapping_id)`: a frame was presented; 0 for a kernel task's backing. ++// ++// Thread: the present's, under the guest display's mutex: it must not wait. May call: nothing of ++// this header. ++typedef void (*RvgCensusPresentedFn)(void *ctx, uint64_t mapping_id); ++ ++// The touch census's callbacks. ++typedef struct { ++ // sizeof(RvgCensusCallbacks) as the host was compiled. ++ uint32_t struct_size; ++ void *ctx; ++ RvgCensusMappedFn mapped; ++ RvgCensusUnmappedFn unmapped; ++ RvgCensusUsedFn used; ++ RvgCensusPresentedFn presented; ++} RvgCensusCallbacks; ++ + // `destroy(ctx)`: the last call of a callback table (see the preamble). + typedef void (*RvgDestroyFn)(void *ctx); + +@@ -901,6 +956,13 @@ const char *rvg_status_message(RvgStatus status); + // the rest of the process. + void rvg_log_set_handler(RvgLogFn handler, void *ctx, RvgLogLevel max_level); + ++// Registers the touch census's callbacks, the process's one set, before the first device. Without ++// it the library makes none of these calls. RVG_ERR_INVALID_ARGUMENT for NULL or a second set. ++// ++// # Safety ++// `callbacks` is NULL or a live table. Its functions and `ctx` live as long as the process. ++RvgStatus rvg_census_set_callbacks(const RvgCensusCallbacks *callbacks); ++ + // A new device. Refusals, checked in this order: RVG_ERR_INVALID_ARGUMENT for the config + // itself (the files behind its memory map's ranges among it), RVG_ERR_UNSUPPORTED, + // RVG_ERR_DISPLAY_PORT_COUNT, RVG_ERR_NO_METAL_DEVICE, RVG_ERR_MEMORY_CONFLICT, +diff --git a/crates/reims-vgpu-c/src/census.rs b/crates/reims-vgpu-c/src/census.rs +new file mode 100644 +index 00000000..c1fdaafa +--- /dev/null ++++ b/crates/reims-vgpu-c/src/census.rs +@@ -0,0 +1,215 @@ ++//! LOCAL (Atlas): the touch census's news of the guest's surfaces, [`RvgCensusCallbacks`], and ++//! `rvg_census_set_callbacks`. See `reims_vgpu_core::census`. ++ ++use std::ffi::c_void; ++ ++use reims_vgpu_core::census::{self, Sink, SurfaceFacts}; ++ ++use crate::ffi; ++use crate::sized::{self, HostStruct}; ++use crate::status::{RVG_ERR_INVALID_ARGUMENT, RVG_OK, RvgStatus}; ++ ++/// What a surface's descriptor says of it. ++#[repr(C)] ++#[derive(Clone, Copy)] ++#[expect( ++ missing_docs, ++ reason = "the fields' names say what they hold, as the header shows them" ++)] ++pub struct RvgCensusSurface { ++ pub mapping_id: u32, ++ pub pixel_format: u32, ++ pub width: u64, ++ pub height: u64, ++ pub bytes_per_row: u64, ++ pub alloc_size: u64, ++ pub protection_options: u64, ++ pub address_format: u32, ++ pub compression_type: u32, ++ pub read_only: bool, ++ pub host_only: bool, ++} ++ ++/// `mapped(ctx, surface, pages, page_count, page_size)`: a surface was mapped, with each guest ++/// page's host address. ++/// ++/// Thread: the IOSurface device's own, under its state's lock. May call: nothing of this header. ++pub type RvgCensusMappedFn = Option< ++ unsafe extern "C-unwind" fn( ++ ctx: *mut c_void, ++ surface: *const RvgCensusSurface, ++ pages: *const u64, ++ page_count: usize, ++ page_size: u64, ++ ), ++>; ++ ++/// `unmapped(ctx, mapping_id)`: the surface goes, and its pages may be reused when this returns. ++/// ++/// Thread: the IOSurface device's own, under its state's lock. May call: nothing of this header. ++pub type RvgCensusUnmappedFn = ++ Option; ++ ++/// `used(ctx, mapping_id, usage, storage_mode)`: a texture was made over the surface. ++/// ++/// Thread: the task's that made it. May call: nothing of this header. ++pub type RvgCensusUsedFn = Option< ++ unsafe extern "C-unwind" fn( ++ ctx: *mut c_void, ++ mapping_id: u32, ++ usage: u64, ++ storage_mode: u64, ++ ), ++>; ++ ++/// `presented(ctx, mapping_id)`: a frame was presented; 0 for a kernel task's backing. ++/// ++/// Thread: the present's, under the guest display's mutex: it must not wait. May call: nothing of ++/// this header. ++pub type RvgCensusPresentedFn = ++ Option; ++ ++/// The touch census's callbacks. ++#[repr(C)] ++#[derive(Clone, Copy)] ++#[expect( ++ missing_docs, ++ reason = "the fields' names say what they hold, as the header shows them" ++)] ++pub struct RvgCensusCallbacks { ++ /// sizeof(RvgCensusCallbacks) as the host was compiled. ++ pub struct_size: u32, ++ pub ctx: *mut c_void, ++ pub mapped: RvgCensusMappedFn, ++ pub unmapped: RvgCensusUnmappedFn, ++ pub used: RvgCensusUsedFn, ++ pub presented: RvgCensusPresentedFn, ++} ++ ++// SAFETY: repr(C), a u32 first; a pointer and optional function pointers. ++unsafe impl HostStruct for RvgCensusCallbacks {} ++ ++/// The host's callbacks, as the core's sink. ++struct HostSink(RvgCensusCallbacks); ++ ++// SAFETY: the host's contract: its functions run on any of the library's threads, with its `ctx`. ++unsafe impl Send for HostSink {} ++// SAFETY: as above. ++unsafe impl Sync for HostSink {} ++ ++impl Sink for HostSink { ++ fn mapped(&self, facts: &SurfaceFacts, pages: &[u64], page_size: u64) { ++ let Some(mapped) = self.0.mapped else { return }; ++ let surface = RvgCensusSurface { ++ mapping_id: facts.mapping_id, ++ pixel_format: facts.pixel_format, ++ width: facts.width, ++ height: facts.height, ++ bytes_per_row: facts.bytes_per_row, ++ alloc_size: facts.alloc_size, ++ protection_options: facts.protection_options, ++ address_format: facts.address_format, ++ compression_type: facts.compression_type, ++ read_only: facts.read_only, ++ host_only: facts.host_only, ++ }; ++ let (ctx, at, count) = (self.0.ctx, pages.as_ptr(), pages.len()); ++ // SAFETY: the host's function, given its context and what lives across the call. ++ ffi::host_call("census.mapped", || unsafe { ++ mapped(ctx, &raw const surface, at, count, page_size); ++ }); ++ } ++ ++ fn unmapped(&self, mapping_id: u32) { ++ let Some(unmapped) = self.0.unmapped else { ++ return; ++ }; ++ let ctx = self.0.ctx; ++ // SAFETY: the host's function, given its context. ++ ffi::host_call("census.unmapped", || unsafe { unmapped(ctx, mapping_id) }); ++ } ++ ++ fn used(&self, mapping_id: u32, usage: u64, storage_mode: u64) { ++ let Some(used) = self.0.used else { return }; ++ let ctx = self.0.ctx; ++ // SAFETY: the host's function, given its context. ++ ffi::host_call("census.used", || unsafe { ++ used(ctx, mapping_id, usage, storage_mode); ++ }); ++ } ++ ++ fn presented(&self, mapping_id: u64) { ++ let Some(presented) = self.0.presented else { ++ return; ++ }; ++ let ctx = self.0.ctx; ++ // SAFETY: the host's function, given its context. ++ ffi::host_call("census.presented", || unsafe { presented(ctx, mapping_id) }); ++ } ++} ++ ++/// Registers the touch census's callbacks, the process's one set, before the first device. Without ++/// it the library makes none of these calls. RVG_ERR_INVALID_ARGUMENT for NULL or a second set. ++/// ++/// # Safety ++/// `callbacks` is NULL or a live table. Its functions and `ctx` live as long as the process. ++#[unsafe(no_mangle)] ++pub unsafe extern "C" fn rvg_census_set_callbacks( ++ callbacks: *const RvgCensusCallbacks, ++) -> RvgStatus { ++ ffi::export(|| { ++ // SAFETY: the caller's contract. ++ let table = match unsafe { sized::read(callbacks) } { ++ Ok(Some(table)) => table, ++ Ok(None) => return RVG_ERR_INVALID_ARGUMENT, ++ Err(status) => return status, ++ }; ++ if census::register(Box::new(HostSink(table))) { ++ RVG_OK ++ } else { ++ RVG_ERR_INVALID_ARGUMENT ++ } ++ }) ++} ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ use crate::status::RVG_ERR_UNSUPPORTED; ++ ++ /// A table the host wrote is taken once; NULL, a short table and one past the library's are ++ /// refused and take no place. ++ #[test] ++ fn the_callbacks_are_taken_once() { ++ let zeroed = RvgCensusCallbacks { ++ struct_size: 0, ++ ctx: std::ptr::null_mut(), ++ mapped: None, ++ unmapped: None, ++ used: None, ++ presented: None, ++ }; ++ let size = u32::try_from(size_of::()).unwrap(); ++ // SAFETY: NULL, and tables that live across each call. ++ unsafe { ++ assert_eq!( ++ rvg_census_set_callbacks(std::ptr::null()), ++ RVG_ERR_INVALID_ARGUMENT ++ ); ++ assert_eq!(rvg_census_set_callbacks(&zeroed), RVG_ERR_INVALID_ARGUMENT); ++ let mut long = [0u8; 256]; ++ long[..4].copy_from_slice(&255u32.to_ne_bytes()); ++ long[254] = 1; ++ assert_eq!( ++ rvg_census_set_callbacks(long.as_ptr().cast()), ++ RVG_ERR_UNSUPPORTED ++ ); ++ let table = RvgCensusCallbacks { ++ struct_size: size, ++ ..zeroed ++ }; ++ assert_eq!(rvg_census_set_callbacks(&table), RVG_OK); ++ assert_eq!(rvg_census_set_callbacks(&table), RVG_ERR_INVALID_ARGUMENT); ++ } ++ } ++} +diff --git a/crates/reims-vgpu-c/src/lib.rs b/crates/reims-vgpu-c/src/lib.rs +index 4bf0da5a..50b2c78b 100644 +--- a/crates/reims-vgpu-c/src/lib.rs ++++ b/crates/reims-vgpu-c/src/lib.rs +@@ -27,6 +27,7 @@ pub mod status; + + pub mod log; + ++pub mod census; + pub mod device; + pub mod display; + pub mod frame; +diff --git a/crates/reims-vgpu-core/src/census.rs b/crates/reims-vgpu-core/src/census.rs +new file mode 100644 +index 00000000..5d3ac435 +--- /dev/null ++++ b/crates/reims-vgpu-core/src/census.rs +@@ -0,0 +1,64 @@ ++//! LOCAL (Atlas): what the touch census hears of the guest's surfaces: QEMU's ++//! `accel/kvm/census.c` traps the guest CPU's first touch of a page of a large IOSurface, and ++//! needs to know which pages those are, what the guest said of them, and when a frame went by. ++//! ++//! The census is the host's: it registers a [`Sink`] once, before the first device, and only when ++//! it runs. Nothing registered costs one atomic load per map, unmap, texture and present, and ++//! the library's behaviour is the same with one registered: a sink keeps no reference, and answers ++//! nothing. A sink's calls never reach the GPU, and `presented` takes no lock. ++ ++use std::sync::OnceLock; ++ ++/// What a surface's descriptor says of it. ++#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] ++pub struct SurfaceFacts { ++ /// The mapping id. ++ pub mapping_id: u32, ++ /// The IOSurface pixel format. ++ pub pixel_format: u32, ++ /// The width in pixels. ++ pub width: u64, ++ /// The height in pixels. ++ pub height: u64, ++ /// Bytes per row. ++ pub bytes_per_row: u64, ++ /// The allocation's size. ++ pub alloc_size: u64, ++ /// `protection_options`. ++ pub protection_options: u64, ++ /// `address_format`. ++ pub address_format: u32, ++ /// `compression_type`. ++ pub compression_type: u32, ++ /// `readOnly`. ++ pub read_only: bool, ++ /// `hostOnly`: the surface has no guest pages. ++ pub host_only: bool, ++} ++ ++/// Who hears of the surfaces. ++pub trait Sink: Send + Sync { ++ /// A surface was mapped. `pages` is each guest page's host address, none for a host-only one. ++ /// On the IOSurface device's queue, under its state's lock. ++ fn mapped(&self, facts: &SurfaceFacts, pages: &[u64], page_size: u64); ++ /// The surface at `mapping_id` goes. Called before the unmap returns to the guest, and ++ /// returns only when no trap remains on its pages, which the guest may reuse at once. ++ fn unmapped(&self, mapping_id: u32); ++ /// A texture was made over the surface at `mapping_id`. ++ fn used(&self, mapping_id: u32, usage: u64, storage_mode: u64); ++ /// A frame was presented: the surface's mapping id, 0 for a kernel task's backing. ++ fn presented(&self, mapping_id: u64); ++} ++ ++static SINK: OnceLock> = OnceLock::new(); ++ ++/// Registers `sink`, the process's one: false if there is one already. ++pub fn register(sink: Box) -> bool { ++ SINK.set(sink).is_ok() ++} ++ ++/// The registered sink. ++#[inline] ++pub fn sink() -> Option<&'static dyn Sink> { ++ SINK.get().map(|sink| &**sink) ++} +diff --git a/crates/reims-vgpu-core/src/display/guest_display/present.rs b/crates/reims-vgpu-core/src/display/guest_display/present.rs +index 0a2582b7..e5df74d5 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/present.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/present.rs +@@ -227,6 +227,9 @@ impl GuestDisplay

{ + self.present_surface_held(Hold::of(held), held, l, client, surface, gamma) + }) + })?; ++ if let Some(sink) = crate::census::sink() { ++ sink.presented(0); ++ } + let p = self.page(l); + p.bump64(sp::PRESENT_COUNT); + self.page(l).w32(sp::PRESENTED_SURFACE, surface); +@@ -506,6 +509,9 @@ impl GuestDisplay

{ + + /// Counts a mapped present in the page, after its pool and holder have ended. + pub(super) fn presented_mapped_surface(&self, l: &GuestDisplayLocked

, surface: u64) { ++ if let Some(sink) = crate::census::sink() { ++ sink.presented(surface); ++ } + let p = self.page(l); + p.bump64(sp::PRESENT_COUNT); + self.page(l).w64(sp::PRESENTED_MAPPING_ID, surface); +diff --git a/crates/reims-vgpu-core/src/iosurface/table.rs b/crates/reims-vgpu-core/src/iosurface/table.rs +index 49c9aa56..4c1ac55b 100644 +--- a/crates/reims-vgpu-core/src/iosurface/table.rs ++++ b/crates/reims-vgpu-core/src/iosurface/table.rs +@@ -67,6 +67,27 @@ impl SurfaceSlot

{ + } + } + ++/// What the census hears of a surface from its descriptor. ++fn census_facts( ++ mapping_id: u32, ++ descriptor: &IoSurfaceDeviceDescriptor2, ++) -> crate::census::SurfaceFacts { ++ let base = &descriptor.base; ++ crate::census::SurfaceFacts { ++ mapping_id, ++ pixel_format: base.pixel_format, ++ width: base.width, ++ height: base.height, ++ bytes_per_row: base.bytes_per_row, ++ alloc_size: base.alloc_size, ++ protection_options: base.protection_options, ++ address_format: u32::from(base.address_format), ++ compression_type: u32::from(base.compression_type), ++ read_only: base.read_only(), ++ host_only: base.host_only(), ++ } ++} ++ + /// The debug line of a surface `IOSurfaceCreate` refused, describing its properties through + /// `surfaces` only when the line is logged. + fn log_refused_spec(surfaces: &dyn IoSurfaces

, spec: &SurfaceSpec) { +@@ -140,6 +161,13 @@ impl HostDeviceCore

{ + if !made { + slot.pages = Vec::new(); + } ++ if made && let Some(sink) = crate::census::sink() { ++ sink.mapped( ++ &census_facts(mapping_id, &descriptor), ++ &slot.pages, ++ self.page_size(), ++ ); ++ } + if let Some(s) = st.slots.get_mut(index) { + *s = slot; + } +@@ -321,6 +349,9 @@ impl HostDeviceCore

{ + let word = at.read_u32(0, site); + at.write_u32(0, word & !Entry::MAPPED, site); + }); ++ if let Some(sink) = crate::census::sink() { ++ sink.unmapped(mapping_id); ++ } + let page_size = self.page_size(); + if let Some(slot) = st.slots.get_mut(index) { + slot.surface = None; +@@ -381,6 +412,9 @@ impl HostDeviceCore

{ + surface.retain_count() + ); + } ++ if let Some(sink) = crate::census::sink() { ++ sink.unmapped(i.to_u64().low_u32()); ++ } + drop(surface); + if slot.host_only { + host_only = host_only.wrapping_sub(slot.page_count.wrapping_mul(page)); +@@ -407,6 +441,87 @@ mod tests { + use crate::testkit::platform::FakePlatform; + use crate::testkit::script::Step; + ++ /// The census's sink hears of a map with the guest's pages, of an unmap before it returns and ++ /// only when it succeeds, and of each surface a teardown drops. The process has one sink, and ++ /// other tests' rigs run beside this one, so it keeps what this thread does. ++ mod census { ++ use std::thread::ThreadId; ++ ++ use super::*; ++ use crate::census::{Sink, SurfaceFacts}; ++ ++ #[derive(Debug, Clone, PartialEq, Eq)] ++ enum Heard { ++ Mapped(SurfaceFacts, Vec, u64), ++ Unmapped(u32), ++ } ++ ++ static HEARD: parking_lot::Mutex> = ++ parking_lot::Mutex::new(Vec::new()); ++ ++ struct Recorder; ++ ++ impl Sink for Recorder { ++ fn mapped(&self, facts: &SurfaceFacts, pages: &[u64], page_size: u64) { ++ let heard = Heard::Mapped(*facts, pages.to_vec(), page_size); ++ HEARD.lock().push((std::thread::current().id(), heard)); ++ } ++ fn unmapped(&self, mapping_id: u32) { ++ let heard = Heard::Unmapped(mapping_id); ++ HEARD.lock().push((std::thread::current().id(), heard)); ++ } ++ fn used(&self, _: u32, _: u64, _: u64) {} ++ fn presented(&self, _: u64) {} ++ } ++ ++ fn take_mine() -> Vec { ++ let me = std::thread::current().id(); ++ let mut heard = HEARD.lock(); ++ let mine = heard.iter().filter(|h| h.0 == me).map(|h| h.1.clone()); ++ let mine = mine.collect(); ++ heard.retain(|h| h.0 != me); ++ mine ++ } ++ ++ #[test] ++ fn a_sink_hears_the_maps_and_unmaps_the_guest_sees() { ++ crate::census::register(Box::new(Recorder)); ++ let r = rig(); ++ r.mapped_v2(3, 2); ++ let d = v2(2, ROOT); ++ let facts = SurfaceFacts { ++ mapping_id: 3, ++ pixel_format: d.base.pixel_format, ++ width: d.base.width, ++ height: d.base.height, ++ bytes_per_row: d.base.bytes_per_row, ++ alloc_size: d.base.alloc_size, ++ protection_options: d.base.protection_options, ++ address_format: u32::from(d.base.address_format), ++ compression_type: u32::from(d.base.compression_type), ++ read_only: false, ++ host_only: false, ++ }; ++ let pages = vec![r.host(SURF), r.host(SURF + PAGE)]; ++ assert!( ++ take_mine().contains(&Heard::Mapped(facts, pages, PAGE)), ++ "the guest's pages" ++ ); ++ let held = r.held(3); ++ assert_eq!(r.unmap(3), IoReturn::Busy); ++ assert_eq!(take_mine(), [], "a busy unmap keeps the surface"); ++ drop(held); ++ assert_eq!(r.unmap(3), IoReturn::Success); ++ assert_eq!(take_mine(), [Heard::Unmapped(3)]); ++ assert_eq!(r.unmap(3), IoReturn::Success); ++ assert_eq!(take_mine(), [], "an empty slot is nothing to hear"); ++ r.mapped_v2(4, 1); ++ take_mine(); ++ r.teardown(); ++ assert_eq!(take_mine(), [Heard::Unmapped(4)]); ++ } ++ } ++ + mod map { + use super::*; + +diff --git a/crates/reims-vgpu-core/src/lib.rs b/crates/reims-vgpu-core/src/lib.rs +index a81d9ea8..36d4d149 100644 +--- a/crates/reims-vgpu-core/src/lib.rs ++++ b/crates/reims-vgpu-core/src/lib.rs +@@ -32,6 +32,7 @@ + // `deserializer/deserializer.rs`. + #![allow(clippy::module_inception)] + ++pub mod census; + pub mod collections; + pub mod deserializer; + pub mod device; +diff --git a/crates/reims-vgpu-core/src/resources/manager/texture.rs b/crates/reims-vgpu-core/src/resources/manager/texture.rs +index 85ace4c5..5e5052ca 100644 +--- a/crates/reims-vgpu-core/src/resources/manager/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/manager/texture.rs +@@ -1334,6 +1334,9 @@ pub(crate) fn create_mapper_ref_texture>( + return Err(raised); + } + }; ++ if let Some(sink) = crate::census::sink() { ++ sink.used(mapping_id.low_u32(), texture.usage(), texture.storage_mode()); ++ } + let resource = match env.new_mapper_ref_texture_resource(mapping_id, texture) { + Ok(resource) => resource, + Err(raised) => { +diff --git a/docs/api.md b/docs/api.md +index 171b6c64..2ddabe01 100644 +--- a/docs/api.md ++++ b/docs/api.md +@@ -702,6 +702,7 @@ The header's 47 functions, in its order. "May" is the header's mark "may call ba + | 39 | `rvg_video_device_deliver` | may | one message of queue 0: its `subs` surfaces held while the session needs them, the session's status answered; `RVG_VIDEO_INVALID_SESSION` with no session; `RVG_VIDEO_ALLOCATION_FAILED` for a message over `RVG_VIDEO_MAX_MESSAGE_SIZE`, which the session never sees | + | 40 | `rvg_video_device_invalidate` | | a reset's first phase: the session sends the guest nothing more | + | 41 | `rvg_video_device_reset` | | both phases: the session's work drains and it ends | ++| 42a | `rvg_census_set_callbacks` | | LOCAL (Atlas): the touch census's news of surfaces and presents (`reims_vgpu_core::census`); no calls without it | + | 42 | `rvg_scaler_device_create` | | the M2 scaler device (106b:1a0b), from an `RvgScalerDeviceConfig`, made on every host: transforming with the library's own kernel on the GPU it draws with (Metal's on macOS, the Vulkan platform's on Linux, any GPU, a software rasterizer included), and with none where that opens no device or the device refuses the kernel | + | 43 | `rvg_scaler_device_release` | | drops the host's reference; the last one drops its accelerator and ends its guest-memory table | + | 44 | `rvg_scaler_device_scales` | | whether the device has a GPU to transform on; where it has not, the host refuses the guest's features (FEATURES_OK), so the guest's driver does without the device | +diff --git a/scripts/internal/census/census_test.c b/scripts/internal/census/census_test.c +new file mode 100644 +index 00000000..2f9b7d7d +--- /dev/null ++++ b/scripts/internal/census/census_test.c +@@ -0,0 +1,548 @@ ++// census_test.c: the touch census on a real KVM guest, with no QEMU: links QEMU's accel/kvm/census.c ++// as it ships, and plays kvm-all.c's part. The slot is on the census's view, armed by the census's ++// own calls; the guest is a minimal long-mode program, as in the study's probe. ++// gcc -O2 -g -Wall -D_GNU_SOURCE -I$QEMU/include census_test.c $QEMU/accel/kvm/census.c -lpthread ++// ./a.out [slot_mib] ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++#include ++ ++#include "system/kvm_census.h" ++ ++#define PG 4096UL ++#define S0_SIZE (2UL << 20) ++#define CODE 0x10000UL ++#define STACK 0x1f0000UL ++#define DATA_GPA (4UL << 30) ++ ++// clang-format off ++__asm__( ++".pushsection .rodata\n.code64\n" ++"g_start:\n" ++"g_read: mov (%rbx),%rax\n out %al,$0x10\n" ++"g_write: mov %rax,(%rbx)\n out %al,$0x10\n" ++"g_lock: lock incq (%rbx)\n out %al,$0x10\n" ++"g_avx: vmovdqu (%rbx),%ymm0\n out %al,$0x10\n" ++"g_repmovs: rep movsb\n out %al,$0x10\n" ++"g_outs: mov $0x11,%dx\n rep outsb\n out %al,$0x10\n" ++"g_loop: mov (%rbx),%rax\n add %rdx,%rbx\n dec %rcx\n jnz g_loop\n out %al,$0x10\n" ++"g_wloop: mov %rax,(%rbx)\n add %rdx,%rbx\n dec %rcx\n jnz g_wloop\n out %al,$0x10\n" ++"g_end:\n" ++".popsection\n"); ++// clang-format on ++extern const char g_start[], g_end[], g_read[], g_write[], g_lock[], g_avx[], g_repmovs[], g_outs[], ++ g_loop[], g_wloop[]; ++#define STUB(s) (CODE + (uint64_t)((s) - g_start)) ++ ++#define DIE(...) do { fprintf(stderr, __VA_ARGS__); fprintf(stderr, " (%s)\n", strerror(errno)); exit(1); } while (0) ++ ++static int kvmfd, run_sz, vmfd, failures; ++static char *s0, *main_map; ++static uint64_t mem_len; ++static struct kvm_cpuid2 *cpuid; ++ ++struct vcpu { int fd; struct kvm_run *run; CensusVcpu *vc; }; ++ ++#define CHECK(cond, ...) do { \ ++ int ok_ = (cond); printf(" %s ", ok_ ? "ok " : "FAIL"); printf(__VA_ARGS__); printf("\n"); \ ++ failures += !ok_; } while (0) ++ ++static inline uint64_t now_ns(void) { ++ struct timespec t; ++ clock_gettime(CLOCK_MONOTONIC, &t); ++ return t.tv_sec * 1000000000ULL + t.tv_nsec; ++} ++ ++// What kvm-all.c's SIGBUS path holds between the signal and KVM_RUN's return. ++static __thread void *pending_addr; ++static __thread int pending; ++static void on_bus(int s, siginfo_t *i, void *u) { ++ (void)s; (void)u; ++ if (i->si_code != BUS_MCEERR_AR) { signal(SIGBUS, SIG_DFL); raise(SIGBUS); } ++ pending_addr = i->si_addr; pending = 1; ++} ++ ++static void read_gpa(void *opaque, uint64_t gpa, void *buf, unsigned len) { ++ (void)opaque; ++ if (gpa + len <= S0_SIZE) memcpy(buf, s0 + gpa, len); ++} ++ ++static void vcpu_init(struct vcpu *c, int id) { ++ c->fd = ioctl(vmfd, KVM_CREATE_VCPU, id); ++ if (c->fd < 0) DIE("vcpu"); ++ c->run = mmap(0, run_sz, PROT_READ | PROT_WRITE, MAP_SHARED, c->fd, 0); ++ if (ioctl(c->fd, KVM_SET_CPUID2, cpuid)) DIE("cpuid"); ++ struct kvm_sregs s; ++ ioctl(c->fd, KVM_GET_SREGS, &s); ++ struct kvm_segment cs = {.base = 0, .limit = 0xffffffff, .selector = 8, .type = 11, .present = 1, ++ .dpl = 0, .db = 0, .s = 1, .l = 1, .g = 1}; ++ struct kvm_segment ds = {.base = 0, .limit = 0xffffffff, .selector = 16, .type = 3, .present = 1, ++ .dpl = 0, .db = 1, .s = 1, .l = 0, .g = 1}; ++ s.cs = cs; s.ds = s.es = s.fs = s.gs = s.ss = ds; ++ s.cr3 = 0x1000; ++ s.cr4 = (1 << 5) | (1 << 9) | (1 << 10) | (1 << 18); ++ s.cr0 = 0x80000033; ++ s.efer = (1 << 8) | (1 << 10); ++ if (ioctl(c->fd, KVM_SET_SREGS, &s)) DIE("sregs"); ++ struct kvm_xcrs x = {.nr_xcrs = 1}; ++ x.xcrs[0].xcr = 0; x.xcrs[0].value = 7; ++ if (ioctl(c->fd, KVM_SET_XCRS, &x)) DIE("xcrs"); ++ c->vc = census_vcpu(id, c->fd, NULL, read_gpa); ++} ++ ++static void setr(struct vcpu *c, uint64_t rip, uint64_t rax, uint64_t rbx, uint64_t rcx, uint64_t rdx, ++ uint64_t rsi, uint64_t rdi) { ++ struct kvm_regs r = {.rip = rip, .rflags = 2, .rsp = STACK, .rax = rax, .rbx = rbx, .rcx = rcx, ++ .rdx = rdx, .rsi = rsi, .rdi = rdi}; ++ if (ioctl(c->fd, KVM_SET_REGS, &r)) DIE("set regs"); ++} ++static struct kvm_regs getr(struct vcpu *c) { ++ struct kvm_regs r; ++ ioctl(c->fd, KVM_GET_REGS, &r); ++ return r; ++} ++ ++enum { RUN_IO, RUN_MMIO, RUN_FATAL }; ++// kvm_cpu_exec's loop for what the census changes. On an MMIO exit it completes the access through ++// the main mapping, as address_space_rw does. Unblocks SIGBUS only inside KVM_RUN, as QEMU's ++// KVM_SET_SIGNAL_MASK does. ++static int run_guest(struct vcpu *c, int *mmio_exits) { ++ sigset_t bus, old; ++ sigemptyset(&bus); sigaddset(&bus, SIGBUS); ++ for (;;) { ++ alarm(60); ++ pthread_sigmask(SIG_UNBLOCK, &bus, &old); ++ int r = ioctl(c->fd, KVM_RUN, 0); ++ int e = r < 0 ? errno : 0; ++ pthread_sigmask(SIG_BLOCK, &bus, 0); ++ if (pending) { ++ pending = 0; ++ if (census_on_sigbus(c->vc, &pending_addr) == CENSUS_HANDLED) { ++ if (e == EINTR) continue; ++ } else { ++ return RUN_FATAL; ++ } ++ } ++ if (r < 0) { ++ if (e == EINTR) continue; ++ if (e == EFAULT && census_on_efault(c->vc)) continue; ++ return RUN_FATAL; ++ } ++ if (c->run->exit_reason == KVM_EXIT_IO) return RUN_IO; ++ if (c->run->exit_reason == KVM_EXIT_MMIO) { ++ uint64_t off = c->run->mmio.phys_addr - DATA_GPA; ++ if (mmio_exits) (*mmio_exits)++; ++ if (c->run->mmio.is_write) memcpy(main_map + off, c->run->mmio.data, c->run->mmio.len); ++ else memcpy(c->run->mmio.data, main_map + off, c->run->mmio.len); ++ continue; ++ } ++ printf(" exit %u\n", c->run->exit_reason); ++ return RUN_FATAL; ++ } ++} ++ ++static void arm(uint32_t id, uint64_t off, uint64_t pages) { ++ uint64_t *list = malloc(pages * 8); ++ for (uint64_t i = 0; i < pages; i++) list[i] = (uint64_t)main_map + off + i * PG; ++ CensusSurface f = {.mapping_id = id, .width = 2000, .height = 2000, .pixel_format = 0x42475241, ++ .alloc_size = pages * PG, .bytes_per_row = 8000}; ++ census_surface_mapped(&f, list, pages, PG); ++ free(list); ++} ++ ++static void t_classify(struct vcpu *c) { ++ printf("[classify] one page: touch, read or write, resumed, data intact\n"); ++ uint64_t off = 0x100000, gpa = DATA_GPA + off; ++ CensusCounts a, b; ++ arm(1, off, 64); ++ CHECK(census_page_state(main_map + off) == 1, "armed at map"); ++ census_counts(&a); ++ *(uint64_t *)(main_map + off) = 0x1111; ++ setr(c, STUB(g_read), 0, gpa, 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO && getr(c).rax == 0x1111, "read sees what the main mapping holds"); ++ census_counts(&b); ++ CHECK(b.traps == a.traps + 1 && b.writes == a.writes, "one trap, no write"); ++ CHECK(census_page_state(main_map + off) == 2, "held after a read"); ++ setr(c, STUB(g_read), 0, gpa, 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "a second read is free"); ++ census_counts(&a); ++ CHECK(a.traps == b.traps, "no trap on it"); ++ ++ setr(c, STUB(g_write), 0x4444, gpa + PG, 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "write resumes"); ++ CHECK(*(uint64_t *)(main_map + off + PG) == 0x4444, "the write reached the main mapping"); ++ census_counts(&b); ++ CHECK(b.traps == a.traps + 1 && b.writes == a.writes + 1, "one trap, one write"); ++ CHECK(census_page_state(main_map + off + PG) == 3, "open after a write"); ++ ++ setr(c, STUB(g_lock), 0, gpa + 2 * PG, 0, 0, 0, 0); ++ *(uint64_t *)(main_map + off + 2 * PG) = 41; ++ CHECK(run_guest(c, 0) == RUN_IO && *(uint64_t *)(main_map + off + 2 * PG) == 42, "lock inc"); ++ census_counts(&a); ++ CHECK(a.writes == b.writes + 1, "a locked increment is a write"); ++ ++ *(uint64_t *)(main_map + off + 3 * PG + 8) = 0xabcd; ++ setr(c, STUB(g_avx), 0, gpa + 3 * PG + 8, 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "vmovdqu load resumes"); ++ census_counts(&b); ++ CHECK(b.writes == a.writes && b.traps == a.traps + 1, "and is a read"); ++ ++ for (int i = 0; i < 3 * (int)PG; i++) main_map[off + 0x10000 + i] = (char)(i * 7); ++ setr(c, STUB(g_repmovs), 0, 0, 3 * PG, 0, gpa + 0x10000, gpa + 0x20000); ++ CHECK(run_guest(c, 0) == RUN_IO, "rep movsb across armed pages resumes"); ++ CHECK(!memcmp(main_map + off + 0x10000, main_map + off + 0x20000, 3 * PG), "copy byte-correct"); ++ census_counts(&a); ++ CHECK(a.writes == b.writes + 3 && a.traps == b.traps + 6, "3 reads, 3 writes (got %lu traps %lu writes)", ++ a.traps - b.traps, a.writes - b.writes); ++ census_surface_unmapped(1); ++ CHECK(census_page_state(main_map + off) == 6 && census_page_state(main_map + off + PG) == 6, ++ "unmap leaves every page stale"); ++} ++ ++static void t_upgrade(struct vcpu *c) { ++ printf("[upgrade] a write to a page held since an earlier read: bare EFAULT, no pending page\n"); ++ uint64_t off = 0x200000, gpa = DATA_GPA + off; ++ CensusCounts a, b; ++ arm(2, off, 8); ++ setr(c, STUB(g_read), 0, gpa, 0, 0, 0, 0); ++ run_guest(c, 0); ++ setr(c, STUB(g_read), 0, gpa + PG, 0, 0, 0, 0); ++ run_guest(c, 0); ++ census_counts(&a); ++ setr(c, STUB(g_write), 0x77, gpa, 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "resumes"); ++ CHECK(*(uint64_t *)(main_map + off) == 0x77, "the write landed"); ++ census_counts(&b); ++ CHECK(b.upgrades == a.upgrades + 1, "one upgrade"); ++ CHECK(census_page_state(main_map + off + PG) == 3, "every held page opened"); ++ census_surface_unmapped(2); ++} ++ ++static void t_rearm(struct vcpu *c) { ++ printf("[rearm] the window's end arms touched pages again, read and written alike\n"); ++ uint64_t off = 0x300000, gpa = DATA_GPA + off; ++ CensusCounts a, b; ++ arm(3, off, 16); ++ for (int round = 0; round < 3; round++) { ++ census_counts(&a); ++ setr(c, STUB(g_read), 0, gpa, 0, 0, 0, 0); ++ run_guest(c, 0); ++ setr(c, STUB(g_write), round, gpa + PG, 0, 0, 0, 0); ++ run_guest(c, 0); ++ census_counts(&b); ++ CHECK(b.traps == a.traps + 2 && b.writes == a.writes + 1, "round %d: 2 traps, 1 write", round); ++ census_close_window(); ++ CHECK(census_page_state(main_map + off) == 1 && census_page_state(main_map + off + PG) == 1, ++ "round %d: armed again", round); ++ } ++ census_surface_unmapped(3); ++} ++ ++static void t_emulator(struct vcpu *c) { ++ printf("[emulator] KVM's own access to an armed page: an MMIO exit, completed on the main mapping\n"); ++ uint64_t off = 0x400000, gpa = DATA_GPA + off; ++ int mmio = 0; ++ arm(4, off, 4); ++ memset(main_map + off, 0x5a, PG); ++ setr(c, STUB(g_outs), 0, 0, 8, 0, gpa, 0); ++ int r = run_guest(c, &mmio); ++ CHECK(r == RUN_IO && c->run->io.count >= 1, "rep outsb from an armed page exits with data, %d MMIO exits", mmio); ++ if (c->run->exit_reason == KVM_EXIT_IO) ++ printf(" data %#x (port %#x, count %u)\n", *((uint8_t *)c->run + c->run->io.data_offset), ++ c->run->io.port, c->run->io.count); ++ census_surface_unmapped(4); ++} ++ ++static void t_unknown_sigbus(struct vcpu *c) { ++ printf("[fatal] a fault outside the armed set stays fatal, in a child\n"); ++ fflush(stdout); ++ uint64_t off = 0x500000; ++ CensusCounts a; ++ void *in_view = census_hva(main_map + off); ++ census_counts(&a); ++ void *addr = in_view; ++ CHECK(census_on_sigbus(c->vc, &addr) == CENSUS_NOT_OURS, "a page never armed is not ours"); ++ CHECK(addr == main_map + off, "and goes on with the main mapping's address"); ++ int x; ++ addr = &x; ++ CHECK(census_on_sigbus(c->vc, &addr) == CENSUS_NOT_OURS && addr == &x, "nor an address outside the view"); ++ arm(5, off, 2); ++ census_surface_unmapped(5); ++ addr = in_view; ++ CHECK(census_on_sigbus(c->vc, &addr) == CENSUS_HANDLED, "a stale page reruns once"); ++ int fatal = 0; ++ for (int i = 0; i < 200000 && !fatal; i++) { ++ addr = in_view; ++ fatal = census_on_sigbus(c->vc, &addr) == CENSUS_NOT_OURS; ++ } ++ CHECK(fatal && addr == main_map + off, "real poison on a stale page ends fatal, not in a loop"); ++ ++} ++ ++// Last: it ends the census. A genuine bare EFAULT with nothing held, on a slot with no memory behind ++// it: the fuse turns the census off, and the path to QEMU's fatal error must end. ++static void t_fuse(struct vcpu *c) { ++ printf("[fuse] an EFAULT nothing explains: markers dropped, then fatal as before\n"); ++ struct kvm_userspace_memory_region m = {.slot = 9, .guest_phys_addr = DATA_GPA + (64UL << 30), ++ .memory_size = PG, ++ .userspace_addr = (uint64_t)mmap(0, PG, PROT_NONE, ++ MAP_PRIVATE | MAP_ANONYMOUS, -1, 0)}; ++ if (ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &m)) DIE("slot 9"); ++ arm(6, 0x600000, 4); ++ setr(c, STUB(g_read), 0, DATA_GPA + (64UL << 30), 0, 0, 0, 0); ++ int r = run_guest(c, 0); ++ CensusCounts k; ++ census_counts(&k); ++ CHECK(r == RUN_FATAL, "reaches the fatal path"); ++ CHECK(k.dead, "the census is off"); ++ CHECK(census_page_state(main_map + 0x600000) == 6, "with no marker left"); ++} ++ ++static void t_overlap(void) { ++ printf("[overlap] two surfaces over one page: armed once, each unmap clears its own\n"); ++ uint64_t off = 0x700000; ++ CensusCounts a, b; ++ census_counts(&a); ++ arm(7, off, 4); ++ arm(8, off + 2 * PG, 4); ++ census_counts(&b); ++ CHECK(b.overlaps == a.overlaps + 2, "two pages overlap"); ++ census_surface_unmapped(7); ++ CHECK(census_page_state(main_map + off) == 6 && census_page_state(main_map + off + 4 * PG) == 1, ++ "the second surface keeps its own pages"); ++ census_surface_unmapped(8); ++ census_surface_unmapped(8); ++ census_surface_unmapped(9); ++ CHECK(census_page_state(main_map + off + 4 * PG) == 6, "unmap twice, or of nothing, is harmless"); ++} ++ ++static void t_reset(struct vcpu *c) { ++ printf("[reset] a guest reset drops every marker, with surfaces armed and pages held\n"); ++ uint64_t off = 0x800000, gpa = DATA_GPA + off; ++ arm(10, off, 8); ++ setr(c, STUB(g_read), 0, gpa, 0, 0, 0, 0); ++ run_guest(c, 0); ++ census_reset(); ++ setr(c, STUB(g_wloop), 7, gpa, 8, PG, 0, 0); ++ CensusCounts a, b; ++ census_counts(&a); ++ CHECK(run_guest(c, 0) == RUN_IO, "writes run after the reset"); ++ census_counts(&b); ++ CHECK(b.traps == a.traps && b.writes == a.writes && b.upgrades == a.upgrades, "with no trap"); ++} ++ ++// Several vCPUs race for the same pages, while the census's thread re-arms under them. ++struct racer { struct vcpu c; uint64_t off; int rounds, fatal, id; }; ++static void *race_thread(void *p) { ++ struct racer *r = p; ++ for (int i = 0; i < r->rounds && !r->fatal; i++) { ++ if (i & 1) setr(&r->c, STUB(g_wloop), r->id + i, DATA_GPA + r->off, 16, PG, 0, 0); ++ else setr(&r->c, STUB(g_loop), 0, DATA_GPA + r->off, 16, PG, 0, 0); ++ r->fatal = run_guest(&r->c, 0) != RUN_IO; ++ } ++ return 0; ++} ++// A loop of writes, each followed by a look through the main mapping, with arming under it: the ++// census must never lose a write or show a stale byte. ++struct writer { struct vcpu c; uint64_t off; volatile int stop; int fatal, bad; uint64_t rounds; }; ++static void *write_thread(void *p) { ++ struct writer *w = p; ++ for (uint64_t i = 1; !w->stop && !w->fatal; i++, w->rounds++) { ++ setr(&w->c, STUB(g_wloop), i, DATA_GPA + w->off, 16, PG, 0, 0); ++ w->fatal = run_guest(&w->c, 0) != RUN_IO; ++ for (int k = 0; k < 16; k++) ++ w->bad += *(uint64_t *)(main_map + w->off + k * PG) != i; ++ setr(&w->c, STUB(g_read), 0, DATA_GPA + w->off + (i % 16) * PG, 0, 0, 0, 0); ++ w->fatal |= run_guest(&w->c, 0) != RUN_IO; ++ w->bad += getr(&w->c).rax != i; ++ } ++ return 0; ++} ++static void t_truth(void) { ++ printf("[truth] the guest's pages stay the truth, armed again every 100 us\n"); ++ struct writer w; ++ pthread_t th; ++ uint64_t off = 0xa00000; ++ memset(&w, 0, sizeof w); ++ vcpu_init(&w.c, 7); ++ w.off = off; ++ arm(12, off, 16); ++ pthread_create(&th, 0, write_thread, &w); ++ for (int i = 0; i < 5000 && !w.fatal; i++) { ++ census_close_window(); ++ usleep(100); ++ } ++ w.stop = 1; ++ pthread_join(th, 0); ++ census_surface_unmapped(12); ++ CHECK(!w.fatal && !w.bad, "%lu rounds, %d wrong bytes seen", w.rounds, w.bad); ++} ++ ++static void t_race(void) { ++ printf("[race] 4 vCPUs on one surface, reads and writes, armed again under them, then an unmap\n"); ++ enum { N = 4 }; ++ struct racer r[N]; ++ pthread_t th[N]; ++ uint64_t off = 0x900000; ++ CensusCounts a, b; ++ arm(11, off, 16); ++ census_counts(&a); ++ for (int i = 0; i < N; i++) { ++ memset(&r[i], 0, sizeof r[i]); ++ vcpu_init(&r[i].c, i + 1); ++ r[i].off = off; r[i].rounds = 40000; r[i].id = i; ++ pthread_create(&th[i], 0, race_thread, &r[i]); ++ } ++ uint64_t t0 = now_ns(); ++ for (int i = 0; i < 4000; i++) { ++ census_close_window(); ++ usleep(100); ++ if (i % 16 == 0) { census_surface_unmapped(11); arm(11, off, 16); } ++ } ++ int fatal = 0; ++ for (int i = 0; i < N; i++) r[i].rounds = 0; ++ for (int i = 0; i < N; i++) { pthread_join(th[i], 0); fatal |= r[i].fatal; } ++ census_surface_unmapped(11); ++ census_counts(&b); ++ CHECK(!fatal, "no vCPU died in %.1f s: %lu traps, %lu writes, %lu upgrades, %lu spurious", (now_ns() - t0) / 1e9, ++ b.traps - a.traps, b.writes - a.writes, b.upgrades - a.upgrades, b.spurious - a.spurious); ++ CHECK(b.errors == 0 && !b.dead, "no uffd error, fuse not blown (errors %lu)", b.errors); ++ for (uint64_t i = 0; i < 16; i++) ++ if (census_page_state(main_map + off + i * PG) != 6) { CHECK(0, "page %lu state %d", i, census_page_state(main_map + off + i * PG)); } ++} ++ ++// What registering the view costs a page the census never armed. The WP registration makes read ++// faults map read-only, so a read then a write of a fresh page takes one more host fault. ++static double first_touch(struct vcpu *c, uint64_t gpa, uint64_t pages) { ++ uint64_t t0 = now_ns(); ++ setr(c, STUB(g_loop), 0, gpa, pages, PG, 0, 0); ++ run_guest(c, 0); ++ setr(c, STUB(g_wloop), 0, gpa, pages, PG, 0, 0); ++ run_guest(c, 0); ++ return (now_ns() - t0) / 1e3 / pages; ++} ++static void t_plain_fault(struct vcpu *c) { ++ enum { P = 20000 }; ++ uint64_t plain_gpa = DATA_GPA + (32UL << 30), off = 0x4000000; ++ int fd = memfd_create("plain", MFD_CLOEXEC); ++ if (ftruncate(fd, P * PG)) DIE("plain"); ++ char *m = mmap(0, P * PG, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); ++ struct kvm_userspace_memory_region r = {.slot = 8, .guest_phys_addr = plain_gpa, .memory_size = P * PG, ++ .userspace_addr = (uint64_t)m}; ++ if (ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &r)) DIE("slot 8"); ++ memset(m, 1, P * PG); // both page caches filled, EPT and the view's PTEs empty ++ memset(main_map + off, 1, P * PG); ++ printf("[plain] read then write of a fresh page, never armed: %.2f us on a plain mapping, %.2f us on the view\n", ++ first_touch(c, plain_gpa, P), first_touch(c, DATA_GPA + off, P)); ++} ++ ++static void t_cost(struct vcpu *c) { ++ printf("[cost] a guest loop over 2000 pages, trap to resume\n"); ++ uint64_t off = 0x1000000, gpa = DATA_GPA + off; ++ enum { N = 2000 }; ++ uint64_t t0, t1, t2, t3; ++ setr(c, STUB(g_loop), 0, gpa, N, PG, 0, 0); ++ run_guest(c, 0); ++ setr(c, STUB(g_loop), 0, gpa, N, PG, 0, 0); ++ t0 = now_ns(); run_guest(c, 0); t1 = now_ns(); ++ uint64_t base = t1 - t0; ++ for (int kind = 0; kind < 2; kind++) { ++ arm(20, off, N); ++ setr(c, STUB(kind ? g_wloop : g_loop), 0, gpa, N, PG, 0, 0); ++ t2 = now_ns(); run_guest(c, 0); t3 = now_ns(); ++ printf(" %s: %.2f us a page, trap included, against %.3f us untrapped\n", kind ? "write" : "read", ++ (t3 - t2 - base) / 1e3 / N, base / 1e3 / N); ++ t0 = now_ns(); ++ census_surface_unmapped(20); ++ printf(" unmap of %d pages, each touched: %.0f us\n", N, (now_ns() - t0) / 1e3); ++ } ++ t0 = now_ns(); arm(20, off, N); t1 = now_ns(); ++ printf(" arm %d contiguous pages: %.0f us\n", N, (t1 - t0) / 1e3); ++ census_surface_unmapped(20); ++ // Scattered, as a surface's 16 KiB ranges lie: 1400 ranges of 4 pages. ++ uint64_t *list = malloc(1400 * 4 * 8); ++ for (int i = 0; i < 1400; i++) list[i] = (uint64_t)main_map + off + (i * 8UL) * PG; ++ CensusSurface f = {.mapping_id = 21, .width = 3200, .height = 1800}; ++ t0 = now_ns(); ++ census_surface_mapped(&f, list, 1400, 4 * PG); ++ t1 = now_ns(); ++ census_surface_unmapped(21); ++ t2 = now_ns(); ++ printf(" a 3200x1800 surface, 1400 ranges of 16 KiB: arm %.2f ms, disarm %.2f ms\n", (t1 - t0) / 1e6, (t2 - t1) / 1e6); ++ free(list); ++} ++ ++int main(int argc, char **argv) { ++ setvbuf(stdout, 0, _IOLBF, 0); ++ if (argc < 2) return 2; ++ uint64_t mib = argc > 2 ? atol(argv[2]) : 512; ++ setenv(CENSUS_ENV, argv[1], 1); ++ setenv("REIMS_VGPU_CENSUS_MIN_PIXELS", "1", 1); ++ setenv("REIMS_VGPU_CENSUS_WINDOW_MS", "100000", 1); ++ setenv("REIMS_VGPU_CENSUS_EVERY", "1000000", 1); ++ setenv("REIMS_VGPU_CENSUS_FLUSH_MS", "100", 1); ++ setenv("REIMS_VGPU_CENSUS_SATURATE", "100000", 1); ++ kvmfd = open("/dev/kvm", O_RDWR | O_CLOEXEC); ++ if (kvmfd < 0) DIE("/dev/kvm"); ++ run_sz = ioctl(kvmfd, KVM_GET_VCPU_MMAP_SIZE, 0); ++ vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0); ++ if (vmfd < 0) DIE("vm"); ++ ioctl(vmfd, KVM_SET_TSS_ADDR, 0xfffbd000); ++ s0 = mmap(0, S0_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); ++ memcpy(s0 + CODE, g_start, g_end - g_start); ++ uint64_t *pml4 = (uint64_t *)(s0 + 0x1000), *pdpt = (uint64_t *)(s0 + 0x2000); ++ pml4[0] = 0x2000 | 3; ++ for (uint64_t i = 0; i < 512; i++) pdpt[i] = (i << 30) | 0x83; ++ struct kvm_userspace_memory_region m0 = {.slot = 0, .memory_size = S0_SIZE, .userspace_addr = (uint64_t)s0}; ++ if (ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &m0)) DIE("slot 0"); ++ mem_len = mib << 20; ++ int memfd = memfd_create("census-test", MFD_CLOEXEC); ++ if (ftruncate(memfd, mem_len)) DIE("ftruncate"); ++ main_map = mmap(0, mem_len, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, 0); ++ CHECK(census_wanted(), "the variable is read"); ++ if (!census_start(memfd, 0, main_map, mem_len)) DIE("census_start"); ++ CHECK(census_hva(main_map + 8192) != main_map + 8192, "memslots get the view"); ++ CHECK(census_hva(&failures) == &failures, "an address outside RAM stays"); ++ struct kvm_userspace_memory_region m1 = {.slot = 1, .guest_phys_addr = DATA_GPA, .memory_size = mem_len, ++ .userspace_addr = (uint64_t)census_hva(main_map)}; ++ if (ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &m1)) DIE("slot 1"); ++ cpuid = calloc(1, sizeof *cpuid + 256 * sizeof(struct kvm_cpuid_entry2)); ++ cpuid->nent = 256; ++ if (ioctl(kvmfd, KVM_GET_SUPPORTED_CPUID, cpuid)) DIE("cpuid"); ++ struct sigaction sa = {.sa_sigaction = on_bus, .sa_flags = SA_SIGINFO}; ++ sigaction(SIGBUS, &sa, 0); ++ sigset_t bus; ++ sigemptyset(&bus); sigaddset(&bus, SIGBUS); ++ pthread_sigmask(SIG_BLOCK, &bus, 0); ++ ++ struct vcpu c; ++ vcpu_init(&c, 0); ++ t_classify(&c); ++ t_upgrade(&c); ++ t_rearm(&c); ++ t_emulator(&c); ++ t_unknown_sigbus(&c); ++ t_overlap(); ++ t_reset(&c); ++ t_race(); ++ t_truth(); ++ t_plain_fault(&c); ++ t_cost(&c); ++ t_fuse(&c); ++ census_presented(0); ++ census_flush(); ++ usleep(200000); ++ printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); ++ return failures != 0; ++} diff --git a/patches/reims-vgpu/0007-census-the-report-script-and-the-QEMU-fix.patch b/patches/reims-vgpu/0007-census-the-report-script-and-the-QEMU-fix.patch new file mode 100644 index 0000000..f4172c0 --- /dev/null +++ b/patches/reims-vgpu/0007-census-the-report-script-and-the-QEMU-fix.patch @@ -0,0 +1,206 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:01:18 +0000 +Subject: [PATCH 07/25] census: the report script, and the QEMU fix + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/census_report.py | 189 +++++++++++++++++++++++ + 1 file changed, 189 insertions(+) + create mode 100644 scripts/internal/census/census_report.py + +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +new file mode 100644 +index 00000000..9b9fe2a2 +--- /dev/null ++++ b/scripts/internal/census/census_report.py +@@ -0,0 +1,189 @@ ++#!/usr/bin/env python3 ++"""census_report.py: read a touch census's output (REIMS_VGPU_CENSUS=

). ++ ++ census_report.py every surface, from the census's start ++ census_report.py --mark LABEL between the mark LABEL and the next mark ++ census_report.py --marks the marks in the timeline, and each span's totals ++ census_report.py --disasm N the top N RIPs, disassembled with objdump ++ ++A mark is one line in /mark ("motionbench"), which the census reads within 50 ms. "every 1" ++there sets the re-arm period instead. A page counts once a window: touches/frame cannot exceed ++pages/window divided by the window's frames, so read `touched windows` with `every`. ++""" ++ ++import argparse ++import json ++import os ++import subprocess ++import sys ++import tempfile ++ ++ ++def lines(path): ++ if not os.path.exists(path): ++ return ++ with open(path) as f: ++ for line in f: ++ try: ++ yield json.loads(line) ++ except json.JSONDecodeError: ++ pass # a line the VM's end cut short ++ ++ ++def read(directory): ++ surfaces, timeline = {}, [] ++ header = {} ++ for r in lines(os.path.join(directory, "surfaces.jsonl")): ++ s = surfaces.setdefault(r["s"], {"uses": set()}) ++ if r["ev"] == "use": ++ s["uses"].add((r["usage"], r["storage"])) ++ elif r["ev"] == "map": ++ s.update(r) ++ else: ++ s["end"] = r ++ for r in lines(os.path.join(directory, "ticks.jsonl")): ++ if "started" in r: ++ header = r ++ else: ++ timeline.append(r) ++ return header, surfaces, timeline ++ ++ ++def spans(timeline): ++ """The marks in order, each with its start and the next mark's time.""" ++ marks = [r for r in timeline if "mark" in r and not r["mark"].startswith("every ")] ++ out = [] ++ for i, m in enumerate(marks): ++ end = marks[i + 1]["t"] if i + 1 < len(marks) else float("inf") ++ out.append((m["mark"], m["t"], end)) ++ return out ++ ++ ++FIELDS = ("presents", "traps", "writes", "upgrades", "pages_r", "pages_w", "pages_skip", ++ "windows_touched", "windows_saturated", "cpl0", "cpl3") ++ ++ ++def window_totals(timeline, start, end): ++ frames = 0 ++ windows = [r["windows"] for r in timeline if "frames" in r and start <= r["t"] <= end] ++ per = {} ++ for r in timeline: ++ if not start <= r["t"] <= end: ++ continue ++ if "frames" in r: ++ frames += r["frames"] ++ elif "tick" in r: ++ p = per.setdefault(r["tick"], dict.fromkeys(FIELDS, 0) | {"rips": {}}) ++ for k in FIELDS: ++ p[k] += r[k] ++ for rip, n in r["rips"]: ++ p["rips"][rip] = p["rips"].get(rip, 0) + n ++ return frames, (windows[-1] - windows[0] if len(windows) > 1 else 0), per ++ ++ ++def fmt(s): ++ flags = "".join(c for c, on in (("H", s.get("host_only")), ("R", s.get("read_only"))) if on) ++ return (f"{s.get('w', 0)}x{s.get('h', 0)} {s.get('format', '?')} {flags or '-'}" ++ f" prot={s.get('protection', 0)} addr={s.get('address_format', 0)}" ++ f" comp={s.get('compression', 0)}") ++ ++ ++def table(surfaces, per, frames, windows, every, rw): ++ print(f"frames {frames}, windows {windows}" + (f", one window every {every} frames" if every else "")) ++ print(f"{'surface':>7} {'facts':<44} {'present':>7} {'traps':>8} {'/frame':>8} {'win':>6}" ++ f" {'pages R':>8} {'pages W':>8} {'sat':>4} {'upg':>5} {'k/u':>9} top RIPs") ++ rows = [] ++ for serial, p in per.items(): ++ rows.append((p["traps"], serial, p)) ++ for _, serial, p in sorted(rows, key=lambda r: (-r[0], r[1])): ++ s = surfaces.get(serial, {}) ++ top = sorted(p["rips"].items(), key=lambda kv: -kv[1])[:3] ++ rate = p["traps"] / frames if frames else 0 ++ print(f"{serial:>7} {fmt(s):<44.44} {p['presents']:>7} {p['traps']:>8} {rate:>8.2f}" ++ f" {p['windows_touched']:>6} {p['pages_r']:>8}" ++ f" {p['pages_w'] if rw else '-':>8} {p['windows_saturated']:>4} {p['upgrades']:>5}" ++ f" {p['cpl0']:>4}/{p['cpl3']:<4} " + " ".join(f"{r}:{n}" for r, n in top)) ++ armed = [s for s in surfaces.values() if s.get("armed_pages")] ++ untouched = len(armed) - sum(1 for serial in per if surfaces.get(serial, {}).get("armed_pages")) ++ print(f"armed surfaces {len(armed)}, never touched in this span {untouched}") ++ uses = {} ++ for s in surfaces.values(): ++ for u in s["uses"]: ++ uses[u] = uses.get(u, 0) + 1 ++ if uses: ++ print("texture usage/storage over surfaces: " + ++ ", ".join(f"{u}/{m}:{n}" for (u, m), n in sorted(uses.items()))) ++ ++ ++def life(surfaces): ++ """Whole-life totals, from the unmap records: surfaces that lived and went between ticks.""" ++ print(f"{'surface':>7} {'facts':<44} {'frames':>7} {'presents':>8} {'traps':>8} {'writes':>7}" ++ f" {'win':>6} {'touched':>7}") ++ ended = [s for s in surfaces.values() if "end" in s and s["end"]["traps"]] ++ for s in sorted(ended, key=lambda s: -s["end"]["traps"])[:40]: ++ e = s["end"] ++ print(f"{e['s']:>7} {fmt(s):<44.44} {e['frames']:>7} {e['presents']:>8} {e['traps']:>8}" ++ f" {e['writes']:>7} {e['windows']:>6} {e['windows_touched']:>7}") ++ ++ ++def disasm(directory, count, timeline): ++ bytes_at = {r["rip"]: r for r in lines(os.path.join(directory, "rips.jsonl"))} ++ totals = {} ++ for r in timeline: ++ if "tick" in r: ++ for rip, n in r["rips"]: ++ totals[rip] = totals.get(rip, 0) + n ++ for rip, n in sorted(totals.items(), key=lambda kv: -kv[1])[:count]: ++ r = bytes_at.get(rip) ++ text = "(no bytes)" ++ if r and r["bytes"]: ++ with tempfile.NamedTemporaryFile(suffix=".bin") as f: ++ f.write(bytes.fromhex(r["bytes"])) ++ f.flush() ++ out = subprocess.run(["objdump", "-D", "-b", "binary", "-mi386:x86-64", f.name], ++ capture_output=True, text=True).stdout ++ found = [ln.split("\t", 2)[-1] for ln in out.splitlines() if ln.startswith(" 0:")] ++ text = found[0] if found else r["bytes"] ++ print(f"{rip:>20} cpl{r['cpl'] if r else '?'} {n:>9} {text}") ++ ++ ++def main(): ++ a = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawTextHelpFormatter) ++ a.add_argument("dir") ++ a.add_argument("--mark") ++ a.add_argument("--marks", action="store_true") ++ a.add_argument("--disasm", type=int, default=0) ++ args = a.parse_args() ++ header, surfaces, timeline = read(args.dir) ++ if not header: ++ sys.exit(f"{args.dir}: no census output") ++ every, rw = header.get("every"), header.get("rw", 1) ++ health = [r for r in timeline if "frames" in r] ++ if health: ++ h = health[-1] ++ print(f"census pid {header.get('pid')}: traps {h['traps']}, writes {h['writes']}," ++ f" upgrades {h['upgrades']}, spurious {h['spurious']}, dropped {h['dropped']}," ++ f" errors {h['errors']}, off by fuse: {'YES' if h['dead'] else 'no'}") ++ if args.disasm: ++ disasm(args.dir, args.disasm, timeline) ++ elif args.marks: ++ for label, start, end in spans(timeline): ++ frames, windows, per = window_totals(timeline, start, end) ++ traps = sum(p["traps"] for p in per.values()) ++ rate = traps / frames if frames else 0 ++ print(f"{label:<30} frames {frames:>7} windows {windows:>6} surfaces touched" ++ f" {len(per):>4} traps {traps:>9} ({rate:.2f}/frame)") ++ elif args.mark: ++ found = [s for s in spans(timeline) if s[0] == args.mark] ++ if not found: ++ sys.exit(f"no mark {args.mark!r}") ++ frames, windows, per = window_totals(timeline, found[-1][1], found[-1][2]) ++ table(surfaces, per, frames, windows, every, rw) ++ else: ++ frames, windows, per = window_totals(timeline, 0, float("inf")) ++ table(surfaces, per, frames, windows, every, rw) ++ life(surfaces) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/patches/reims-vgpu/0008-census-the-guests-of-the-boot-tests-the-C-harness-an.patch b/patches/reims-vgpu/0008-census-the-guests-of-the-boot-tests-the-C-harness-an.patch new file mode 100644 index 0000000..f9022bb --- /dev/null +++ b/patches/reims-vgpu/0008-census-the-guests-of-the-boot-tests-the-C-harness-an.patch @@ -0,0 +1,420 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:31:52 +0000 +Subject: [PATCH 08/25] census: the guests of the boot tests, the C harness, an + honest write count + +The report keeps pages a write opened without naming one apart from pages +caught writing. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-c/src/census.rs | 7 +- + crates/reims-vgpu-c/tests/c/harness.c | 2 + + crates/reims-vgpu-c/tests/c_harness.rs | 2 +- + .../src/resources/manager/texture.rs | 6 +- + scripts/internal/census/census_report.py | 10 +- + scripts/internal/census/guest/device.c | 107 ++++++++++++++++++ + scripts/internal/census/guest/device_start.S | 15 +++ + scripts/internal/census/guest/guest.S | 101 +++++++++++++++++ + scripts/internal/census/guest/link.ld | 9 ++ + scripts/internal/census/guest/run.sh | 32 ++++++ + 10 files changed, 281 insertions(+), 10 deletions(-) + create mode 100644 scripts/internal/census/guest/device.c + create mode 100644 scripts/internal/census/guest/device_start.S + create mode 100644 scripts/internal/census/guest/guest.S + create mode 100644 scripts/internal/census/guest/link.ld + create mode 100755 scripts/internal/census/guest/run.sh + +diff --git a/crates/reims-vgpu-c/src/census.rs b/crates/reims-vgpu-c/src/census.rs +index c1fdaafa..3a85dfcf 100644 +--- a/crates/reims-vgpu-c/src/census.rs ++++ b/crates/reims-vgpu-c/src/census.rs +@@ -54,12 +54,7 @@ pub type RvgCensusUnmappedFn = + /// + /// Thread: the task's that made it. May call: nothing of this header. + pub type RvgCensusUsedFn = Option< +- unsafe extern "C-unwind" fn( +- ctx: *mut c_void, +- mapping_id: u32, +- usage: u64, +- storage_mode: u64, +- ), ++ unsafe extern "C-unwind" fn(ctx: *mut c_void, mapping_id: u32, usage: u64, storage_mode: u64), + >; + + /// `presented(ctx, mapping_id)`: a frame was presented; 0 for a kernel task's backing. +diff --git a/crates/reims-vgpu-c/tests/c/harness.c b/crates/reims-vgpu-c/tests/c/harness.c +index 3766f310..2f060d5b 100644 +--- a/crates/reims-vgpu-c/tests/c/harness.c ++++ b/crates/reims-vgpu-c/tests/c/harness.c +@@ -176,6 +176,8 @@ static void check_status_and_log(void) { + CHECK(strcmp(rvg_status_message(RVG_ERR_FRAME_STATE + 1), "unknown status") == 0); + rvg_log_set_handler(on_log, NULL, RVG_LOG_TRACE); + rvg_log_set_handler(NULL, NULL, RVG_LOG_OFF); ++ ++ CHECK_STATUS(rvg_census_set_callbacks(NULL), RVG_ERR_INVALID_ARGUMENT); + CHECK(RVG_MAX_DISPLAY_PORTS == 8 && RVG_DEVICE_MMIO_SIZE == 0x4000); + CHECK(RVG_PCI_VENDOR_ID == 0x106b && RVG_PCI_DEVICE_ID == 0xeeee); + CHECK(RVG_PCI_BAR_MMIO == 0 && RVG_PCI_MSI_VECTORS == 1); +diff --git a/crates/reims-vgpu-c/tests/c_harness.rs b/crates/reims-vgpu-c/tests/c_harness.rs +index 2600858e..8e7b6e3e 100644 +--- a/crates/reims-vgpu-c/tests/c_harness.rs ++++ b/crates/reims-vgpu-c/tests/c_harness.rs +@@ -111,7 +111,7 @@ fn c_harness_calls_every_function() { + Some(&rest[..end]) + }) + .collect(); +- assert_eq!(declared.len(), 47, "{declared:?}"); ++ assert_eq!(declared.len(), 48, "{declared:?}"); + let missing: Vec<&str> = declared + .iter() + .copied() +diff --git a/crates/reims-vgpu-core/src/resources/manager/texture.rs b/crates/reims-vgpu-core/src/resources/manager/texture.rs +index 5e5052ca..65d83a0c 100644 +--- a/crates/reims-vgpu-core/src/resources/manager/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/manager/texture.rs +@@ -1335,7 +1335,11 @@ pub(crate) fn create_mapper_ref_texture>( + } + }; + if let Some(sink) = crate::census::sink() { +- sink.used(mapping_id.low_u32(), texture.usage(), texture.storage_mode()); ++ sink.used( ++ mapping_id.low_u32(), ++ texture.usage(), ++ texture.storage_mode(), ++ ); + } + let resource = match env.new_mapper_ref_texture_resource(mapping_id, texture) { + Ok(resource) => resource, +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +index 9b9fe2a2..c3609449 100644 +--- a/scripts/internal/census/census_report.py ++++ b/scripts/internal/census/census_report.py +@@ -9,6 +9,10 @@ + A mark is one line in /mark ("motionbench"), which the census reads within 50 ms. "every 1" + there sets the re-arm period instead. A page counts once a window: touches/frame cannot exceed + pages/window divided by the window's frames, so read `touched windows` with `every`. ++ ++pages R were only read in their window and pages W were caught writing. pages R|W were read, then ++opened when a write hit some page held from an earlier read: at least one of them was written, ++and KVM does not say which. `sat` counts windows the saturation fuse cut short. + """ + + import argparse +@@ -91,7 +95,7 @@ def fmt(s): + def table(surfaces, per, frames, windows, every, rw): + print(f"frames {frames}, windows {windows}" + (f", one window every {every} frames" if every else "")) + print(f"{'surface':>7} {'facts':<44} {'present':>7} {'traps':>8} {'/frame':>8} {'win':>6}" +- f" {'pages R':>8} {'pages W':>8} {'sat':>4} {'upg':>5} {'k/u':>9} top RIPs") ++ f" {'pages R':>8} {'pages W':>8} {'R|W':>8} {'sat':>4} {'k/u':>9} top RIPs") + rows = [] + for serial, p in per.items(): + rows.append((p["traps"], serial, p)) +@@ -99,9 +103,11 @@ def table(surfaces, per, frames, windows, every, rw): + s = surfaces.get(serial, {}) + top = sorted(p["rips"].items(), key=lambda kv: -kv[1])[:3] + rate = p["traps"] / frames if frames else 0 ++ caught = max(p["pages_w"] - p["upgrades"], 0) # pages_w holds every page a write opened + print(f"{serial:>7} {fmt(s):<44.44} {p['presents']:>7} {p['traps']:>8} {rate:>8.2f}" + f" {p['windows_touched']:>6} {p['pages_r']:>8}" +- f" {p['pages_w'] if rw else '-':>8} {p['windows_saturated']:>4} {p['upgrades']:>5}" ++ f" {caught if rw else '-':>8} {p['upgrades'] if rw else '-':>8}" ++ f" {p['windows_saturated']:>4}" + f" {p['cpl0']:>4}/{p['cpl3']:<4} " + " ".join(f"{r}:{n}" for r, n in top)) + armed = [s for s in surfaces.values() if s.get("armed_pages")] + untouched = len(armed) - sum(1 for serial in per if surfaces.get(serial, {}).get("armed_pages")) +diff --git a/scripts/internal/census/guest/device.c b/scripts/internal/census/guest/device.c +new file mode 100644 +index 00000000..1408b8cb +--- /dev/null ++++ b/scripts/internal/census/guest/device.c +@@ -0,0 +1,107 @@ ++/* A multiboot guest that drives the real IOSurface device, as macOS's mapper driver does, over ++ * PCI: real Reims map and unmap commands reach the census, and this guest's own loads and stores ++ * trap. 32-bit protected mode, paging off. Exit code (stage << 1) | 1: 0x11 passes. ++ * CMOS byte 0x34 counts boots: the first boot resets the machine with a surface armed. */ ++#include ++#define RING 0x01000000u ++#define TABLE 0x01001000u ++#define DPAGE 0x01002000u ++#define ROOT 0x01010000u ++#define SURF 0x20000000u ++#define PAGES 1024u ++#define ID 5u ++#define HOST_ONLY_ID 6u ++#define PG 4096u ++ ++static inline void outb(uint16_t p, uint8_t v) { __asm__ volatile("outb %0,%1" : : "a"(v), "Nd"(p)); } ++static inline uint8_t inb(uint16_t p) { uint8_t v; __asm__ volatile("inb %1,%0" : "=a"(v) : "Nd"(p)); return v; } ++static inline void outl(uint16_t p, uint32_t v) { __asm__ volatile("outl %0,%1" : : "a"(v), "Nd"(p)); } ++static inline uint32_t inl(uint16_t p) { uint32_t v; __asm__ volatile("inl %1,%0" : "=a"(v) : "Nd"(p)); return v; } ++static void die(int stage) { outb(0xf4, stage); for (;;) __asm__("hlt"); } ++#define R32(a) (*(volatile uint32_t *)(uintptr_t)(a)) ++#define R64(a) (*(volatile uint64_t *)(uintptr_t)(a)) ++static uint32_t cfg(int d, int r) { outl(0xcf8, 0x80000000u | d << 11 | r); return inl(0xcfc); } ++static void cfgw(int d, int r, uint32_t v) { outl(0xcf8, 0x80000000u | d << 11 | r); outl(0xcfc, v); } ++ ++static uint32_t bar, wr; ++static uint32_t pte(uint32_t pa) { return (pa >> 12) << 2 | 1; } ++ ++static int cmd(uint32_t c, uint32_t id) { ++ uint32_t at = RING + (wr % 256) * 16; ++ R32(at) = c; R32(at + 4) = id; R32(at + 8) = 0x5555; ++ wr++; ++ R32(bar + 0x1018) = wr; ++ for (uint32_t n = 0; R32(bar + 0x1020) != wr; n++) ++ if (n > 400000000u) die(0x30); ++ return R32(at + 8); ++} ++static void descriptor(uint32_t id, int host_only) { ++ uint32_t d = DPAGE + id * 0x400; ++ for (uint32_t i = 0; i < 0x400; i += 4) R32(d + i) = 0; ++ R32(d) = pte(ROOT); ++ R32(d + 4) = 0x42475241; ++ R64(d + 0x18) = PAGES * PG; ++ R64(d + 0x20) = 1024; ++ R64(d + 0x28) = 1024; ++ R64(d + 0x30) = 4096; ++ *(volatile uint16_t *)(d + 0x38) = 4; ++ *(volatile uint8_t *)(d + 0x3c) = 1; ++ *(volatile uint8_t *)(d + 0x3d) = 1; ++ *(volatile uint8_t *)(d + 0x3f) = 0; /* address_format */ ++ *(volatile uint8_t *)(d + 0x40) = 0; /* compression_type */ ++ *(volatile uint8_t *)(d + 0x41) = host_only ? 2 : 0; ++ /* protection_options stays 0: Linux refuses it over guest pages. */ ++} ++static uint32_t page(uint32_t i) { return SURF + 2 * i * PG; } /* scattered, as a surface's are */ ++ ++void main_c(void) { ++ uint8_t boots; ++ outb(0x70, 0x34); boots = inb(0x71); outb(0x70, 0x34); outb(0x71, boots + 1); ++ ++ for (int d = 0; d < 32 && !bar; d++) ++ if (cfg(d, 0) == (0xeeefu << 16 | 0x106b)) { ++ bar = cfg(d, 0x10) & ~0xfu; ++ cfgw(d, 4, cfg(d, 4) | 6); ++ } ++ if (!bar) die(0x31); ++ for (uint32_t i = 0; i < 16; i++) R32(TABLE + 4 * i) = pte(DPAGE + i * PG); ++ for (uint32_t i = 0; i < PAGES; i++) R32(ROOT + 4 * i) = pte(page(i)); ++ R32(bar + 0x1030) = 1; ++ R32(bar + 0x1000) = RING; ++ R32(bar + 0x1008) = 256; ++ R32(bar + 0x1010) = TABLE; ++ wr = R32(bar + 0x1020); ++ if (boots) { ++ /* The second boot: the surface left armed is gone, and a map works again. */ ++ descriptor(ID, 0); ++ if (cmd(1, ID)) die(0x32); ++ for (uint32_t i = 0; i < PAGES; i++) R32(page(i)) = i; ++ if (cmd(2, ID)) die(0x33); ++ die(0x11); ++ } ++ descriptor(ID, 0); ++ if (cmd(1, ID)) die(0x34); ++ for (uint32_t round = 1; round <= 300; round++) { ++ volatile uint32_t sink = 0; ++ for (uint32_t i = 0; i < PAGES; i++) sink += R32(page(i) + 8); ++ for (uint32_t i = 0; i < PAGES / 2; i++) R32(page(i) + 8) = round + i; ++ for (uint32_t i = 0; i < PAGES / 2; i++) ++ if (R32(page(i) + 8) != round + i) die(0x35); ++ for (uint32_t n = 0; n < 300000; n++) __asm__("pause"); ++ } ++ /* The host's own write, through the main mapping and the surface's view: the guest must read ++ * it through a marker. The test command writes each 8-byte word its own offset. */ ++ if (cmd(3, ID)) die(0x36); ++ for (uint32_t i = 0; i < PAGES; i++) ++ if (R32(page(i) + 8) != i * PG + 8) die(0x37); ++ if (cmd(2, ID)) die(0x38); ++ for (uint32_t i = 0; i < PAGES; i++) R32(page(i) + 8) = 7; /* free pages, no trap */ ++ descriptor(HOST_ONLY_ID, 1); ++ if (cmd(1, HOST_ONLY_ID)) die(0x39); ++ if (cmd(2, HOST_ONLY_ID)) die(0x3a); ++ descriptor(ID, 0); ++ if (cmd(1, ID)) die(0x3b); ++ for (uint32_t i = 0; i < PAGES; i++) (void)R32(page(i)); ++ outb(0xcf9, 0x06); /* reset with the surface mapped, armed and held */ ++ for (;;) __asm__("hlt"); ++} +diff --git a/scripts/internal/census/guest/device_start.S b/scripts/internal/census/guest/device_start.S +new file mode 100644 +index 00000000..d1eaf8e9 +--- /dev/null ++++ b/scripts/internal/census/guest/device_start.S +@@ -0,0 +1,15 @@ ++ .set MB_MAGIC, 0x1badb002 ++ .set MB_FLAGS, 0x00010000 ++ .section .multiboot, "a" ++ .align 4 ++header: ++ .long MB_MAGIC, MB_FLAGS, -(MB_MAGIC + MB_FLAGS) ++ .long header, header, _edata, _end, _start ++ .text ++ .globl _start ++_start: ++ cli ++ movl $0x00f00000, %esp ++ call main_c ++1: hlt ++ jmp 1b +diff --git a/scripts/internal/census/guest/guest.S b/scripts/internal/census/guest/guest.S +new file mode 100644 +index 00000000..cdd2ab1a +--- /dev/null ++++ b/scripts/internal/census/guest/guest.S +@@ -0,0 +1,101 @@ ++/* A multiboot guest for the census's boot test: runs in 32-bit protected mode with paging off, ++ * walks the armed range reading, writing and with rep movs, then resets the machine once through ++ * port 0xcf9 and, on the second boot, leaves through isa-debug-exit. CMOS byte 0x34 counts boots. */ ++ .set MB_MAGIC, 0x1badb002 ++ .set MB_FLAGS, 0x00010000 ++ .section .multiboot, "a" ++ .align 4 ++header: ++ .long MB_MAGIC, MB_FLAGS, -(MB_MAGIC + MB_FLAGS) ++ .long header, header, _edata, _end, _start ++ .text ++ .code32 ++ .globl _start ++_start: ++ cli ++ movl $0x00f00000, %esp ++ movb $0x34, %al ++ outb %al, $0x70 ++ inb $0x71, %al ++ movb %al, %bl /* boots so far */ ++ incb %al ++ movb %al, %cl ++ movb $0x34, %al ++ outb %al, $0x70 ++ movb %cl, %al ++ outb %al, $0x71 ++ ++ movl $200, %ebp /* rounds */ ++1: movl $0x20000000, %esi /* armed: 16 MiB at 512 MiB */ ++ movl $4096, %ecx ++2: movl (%esi), %eax /* read every page */ ++ addl $4096, %esi ++ loop 2b ++ movl $0x20000000, %edi ++ movl $2048, %ecx ++3: movl %ebp, (%edi) /* write half of them */ ++ addl $4096, %edi ++ loop 3b ++ movl $0x20400000, %esi /* rep movs across pages, armed to armed */ ++ movl $0x20800000, %edi ++ movl $0x4000, %ecx ++ rep movsl ++ /* the guest's pages are the truth: read back what it wrote */ ++ movl $0x20000000, %edi ++ movl $2048, %ecx ++4: cmpl %ebp, (%edi) ++ jne fail ++ addl $4096, %edi ++ loop 4b ++ movl $20000, %ecx /* some time for the windows to turn */ ++5: pause ++ loop 5b ++ decl %ebp ++ jnz 1b ++ ++ cmpb $0, %bl ++ jne done ++ movb $0x06, %al /* reset with the surface armed */ ++ movw $0xcf9, %dx ++ outb %al, %dx ++6: hlt ++ jmp 6b ++done: ++ movb $0x11, %al /* qemu exits 35: the pass, soon after this */ ++ outb %al, $0xf4 ++10: hlt ++ jmp 10b ++fail: ++ movl (%edi), %esi /* what the guest read, again */ ++ movl %ebp, %eax ++ call hex ++ movl %edi, %eax ++ call hex ++ movl %esi, %eax ++ call hex ++ movl %ecx, %eax ++ call hex ++ movzbl %bl, %eax ++ call hex ++ movl %edi, %esi /* the same word through the next read of a neighbour */ ++ movl -4(%esi), %eax ++ call hex ++ movb $0x22, %al /* qemu exits 69 */ ++ outb %al, $0xf4 ++7: hlt ++ jmp 7b ++hex: /* %eax to the debug console, as 8 digits and a space */ ++ movl %eax, %edx ++ movl $8, %ecx ++8: roll $4, %edx ++ movb %dl, %al ++ andb $15, %al ++ addb $48, %al ++ cmpb $58, %al ++ jb 9f ++ addb $39, %al ++9: outb %al, $0xe9 ++ loop 8b ++ movb $32, %al ++ outb %al, $0xe9 ++ ret +diff --git a/scripts/internal/census/guest/link.ld b/scripts/internal/census/guest/link.ld +new file mode 100644 +index 00000000..bae0ebdc +--- /dev/null ++++ b/scripts/internal/census/guest/link.ld +@@ -0,0 +1,9 @@ ++ENTRY(_start) ++SECTIONS { ++ . = 0x200000; ++ .multiboot : { *(.multiboot) } ++ .text : { *(.text) } ++ _edata = .; ++ .bss : { *(.bss) } ++ _end = .; ++} +diff --git a/scripts/internal/census/guest/run.sh b/scripts/internal/census/guest/run.sh +new file mode 100755 +index 00000000..ec129251 +--- /dev/null ++++ b/scripts/internal/census/guest/run.sh +@@ -0,0 +1,32 @@ ++#!/bin/bash ++# run.sh [ENV=VALUE]...: boots a guest in the run image, own container, ++# as macos-vm does, under taskset 8-19, on lavapipe: no GPU, no disk. Output in ++# /srv/warm/census/out/. Exit 35 is the guest's pass. ++# ++# gcc -m32 -c guest.S -o guest.o && ld -m elf_i386 -T link.ld -o guest.elf guest.o ++# gcc -m32 -O1 -ffreestanding -fno-pic -fno-pie -fno-stack-protector -mno-sse -nostdlib -static \ ++# -c device.c -o device.o && gcc -m32 -c device_start.S -o device_start.o \ ++# && ld -m elf_i386 -T link.ld -o device.elf device_start.o device.o ++# run.sh on $PWD/guest.elf 0 REIMS_VGPU_CENSUS=/srv/warm/census/out/on \ ++# REIMS_VGPU_CENSUS_SELFTEST=20000000:1000000 ++# run.sh dev $PWD/device.elf 1 REIMS_VGPU_CENSUS=/srv/warm/census/out/dev REIMS_VGPU_WINDOW=0 ++name=$1 elf=$2 gfx=$3; shift 3 ++Q=/srv/data/macos-vm/home/.local/share/reims-ctl/qemu/${CENSUS_BUILD:-20261031-220217-2e0d4d1-census} ++O=/srv/warm/census/out/$name ++rm -rf "$O"; mkdir -p "$O" ++args=() ++for e in "$@"; do args+=(-e "$e"); done ++dev=() ++[ "$gfx" = 1 ] && dev=(-device apple-gfx-pci,id=gfx,bus=pcie.0,addr=0x4,displays=1,resolution=1280x720 -device apple-gfx-iosurface-pci,id=iosfc,gpu=gfx,bus=pcie.0,addr=0x6) ++timeout ${LIMIT:-60} docker run -i --rm --name census-boot-$name --init \ ++ --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/kvm \ ++ -e HOME=/srv/warm/census/home -e VK_DRIVER_FILES=/usr/share/vulkan/icd.d/lvp_icd.json \ ++ -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json -e RUST_LOG=${RUST_LOG:-warn} "${args[@]}" \ ++ -v /srv/data/macos-vm/home/.local/share/reims-ctl/qemu:/srv/data/macos-vm/home/.local/share/reims-ctl/qemu:ro \ ++ -v /srv/warm/census:/srv/warm/census macos-vm:run2 \ ++ sh -c 'echo cap $(grep CapEff /proc/self/status) >&2; exec taskset -c 8-19 "$@"' sh $Q/bin/qemu-system-x86_64 -L $Q/share/qemu \ ++ -accel kvm -cpu host -smp 2 -machine q35,memory-backend=ram -display none -vga none \ ++ -object memory-backend-memfd,id=ram,size=1G,share=on \ ++ -device isa-debug-exit,iobase=0xf4,iosize=1 -chardev file,id=dbg,path=$O/console -device isa-debugcon,iobase=0xe9,chardev=dbg -serial none -monitor ${MON:-none} \ ++ -kernel "$elf" "${dev[@]}" ${EXTRA:-} ++echo "exit $?" diff --git a/patches/reims-vgpu/0009-tell-the-platform-which-texture-the-disp.patch b/patches/reims-vgpu/0009-tell-the-platform-which-texture-the-disp.patch new file mode 100644 index 0000000..b07fb49 --- /dev/null +++ b/patches/reims-vgpu/0009-tell-the-platform-which-texture-the-disp.patch @@ -0,0 +1,61 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:11:41 +0000 +Subject: [PATCH 09/25] tell the platform which texture the + display presents + +metal2vulkan's surface_lazy_present leaves a displayed surface's writes out of +its bytes until a present. A default no-op on the Texture trait. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-core/src/display/frame.rs | 4 ++++ + crates/reims-vgpu-core/src/platform/resource.rs | 4 ++++ + crates/reims-vgpu-vulkan/src/gpu/resource.rs | 4 ++++ + 3 files changed, 12 insertions(+) + +diff --git a/crates/reims-vgpu-core/src/display/frame.rs b/crates/reims-vgpu-core/src/display/frame.rs +index c0391480..e28a308e 100644 +--- a/crates/reims-vgpu-core/src/display/frame.rs ++++ b/crates/reims-vgpu-core/src/display/frame.rs +@@ -73,6 +73,10 @@ impl DisplayCore

{ + color_matrix: Option<&dyn ColorMatrix>, + completion: Box, + ) -> RaiseResult<()> { ++ // LOCAL (Atlas): before the display's lock, as this may commit a command buffer. ++ if let Some(frame) = &frame { ++ frame.presented(); ++ } + let mut g = self.display_mutex.lock(); + // present: frames in flight, waited for next + crate::trace::event( +diff --git a/crates/reims-vgpu-core/src/platform/resource.rs b/crates/reims-vgpu-core/src/platform/resource.rs +index 06f98fc4..ca6f92c6 100644 +--- a/crates/reims-vgpu-core/src/platform/resource.rs ++++ b/crates/reims-vgpu-core/src/platform/resource.rs +@@ -81,6 +81,10 @@ pub trait Buffer: GpuResource

{ + /// `didModifyData` (SPI) is not declared: only the tiled EFI path, which nothing reaches, would + /// send it (D33). + pub trait Texture: GpuResource

{ ++ /// LOCAL (Atlas): not a Metal call. The display is about to show the texture's bytes, for a ++ /// platform that leaves a surface's writes out of them until then (metal2vulkan's ++ /// `shadow_lazy.rs`). ++ fn presented(&self) {} + /// sel: arrayLength + fn array_length(&self) -> u64; + /// sel: depth +diff --git a/crates/reims-vgpu-vulkan/src/gpu/resource.rs b/crates/reims-vgpu-vulkan/src/gpu/resource.rs +index 549138bc..546fcd99 100644 +--- a/crates/reims-vgpu-vulkan/src/gpu/resource.rs ++++ b/crates/reims-vgpu-vulkan/src/gpu/resource.rs +@@ -1274,6 +1274,10 @@ fn at(index: u64) -> rt::IdPlacement { + } + + impl Texture for VulkanTexture { ++ fn presented(&self) { ++ plain(|| self.raw.surface_presented()); ++ } ++ + fn array_length(&self) -> u64 { + plain(|| self.raw.array_length()) + } diff --git a/patches/reims-vgpu/0010-tick-bits-and-fixed-grid-experiments-log.patch b/patches/reims-vgpu/0010-tick-bits-and-fixed-grid-experiments-log.patch new file mode 100644 index 0000000..24e0b27 --- /dev/null +++ b/patches/reims-vgpu/0010-tick-bits-and-fixed-grid-experiments-log.patch @@ -0,0 +1,116 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 22:01:09 +0000 +Subject: [PATCH 10/25] tick bits and fixed grid experiments, + log the guest's enabled bits + +Co-Authored-By: Claude Opus 5.5 +--- + .../src/display/guest_display/vblank.rs | 56 ++++++++++++++++--- + 1 file changed, 49 insertions(+), 7 deletions(-) + +diff --git a/crates/reims-vgpu-core/src/display/guest_display/vblank.rs b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +index df45dc39..ce8344a6 100644 +--- a/crates/reims-vgpu-core/src/display/guest_display/vblank.rs ++++ b/crates/reims-vgpu-core/src/display/guest_display/vblank.rs +@@ -19,6 +19,10 @@ + //! - **The period** is the host's modes' highest refresh rate, 24 to 240 Hz, else 60 Hz. + //! `REIMS_VGPU_VBLANK_HZ`, 24 to 1000, replaces it and leaves the modes as they are, so the + //! ticks can run faster than the rate the guest was told. ++//! - **Experiments:** `REIMS_VGPU_VBLANK_BITS` is the mask a tick raises, 2 (present) unless set; ++//! a bit the guest has not enabled is not raised, except 2, as before. `REIMS_VGPU_VBLANK_GRID=1` ++//! keeps the first schedule, so presents no longer move the ticks. Each change of the guest's ++//! enabled bits seen by a tick is logged. + //! - **A late tick** (the thread was not run on time) raises once and goes on from the newest due + //! point: no burst. + //! +@@ -27,7 +31,7 @@ + //! present> `), and the first tick of the process logs the guest's + //! enabled interrupt bits, so a run shows whether the guest takes the bit at all. + +-use std::sync::atomic::{AtomicBool, Ordering}; ++use std::sync::atomic::{AtomicBool, AtomicU32, Ordering}; + use std::sync::{Arc, OnceLock, Weak}; + use std::time::Duration; + +@@ -42,6 +46,11 @@ pub const ENV_VAR: &str = "REIMS_VGPU_VBLANK"; + /// The variable: the ticks' rate in Hz, instead of the modes' (module doc). + pub const HZ_ENV_VAR: &str = "REIMS_VGPU_VBLANK_HZ"; + ++/// The variable: the interrupt bits a tick raises (module doc). ++pub const BITS_ENV_VAR: &str = "REIMS_VGPU_VBLANK_BITS"; ++/// The variable: `1` keeps the ticks on a fixed grid (module doc). ++pub const GRID_ENV_VAR: &str = "REIMS_VGPU_VBLANK_GRID"; ++ + /// The refresh rate without a usable mode. + const DEFAULT_HZ: f64 = 60.0; + /// The lowest refresh rate taken from a mode. +@@ -68,6 +77,26 @@ fn env_hz() -> Option { + }) + } + ++/// The bits a tick raises (read once). ++fn tick_bits() -> u32 { ++ static BITS: OnceLock = OnceLock::new(); ++ *BITS.get_or_init(|| { ++ std::env::var(BITS_ENV_VAR) ++ .ok() ++ .and_then(|v| v.parse().ok()) ++ .unwrap_or(INTERRUPT_PRESENT) ++ }) ++} ++ ++/// Whether presents leave the ticks' schedule alone (read once). ++fn fixed_grid() -> bool { ++ static ON: OnceLock = OnceLock::new(); ++ *ON.get_or_init(|| std::env::var_os(GRID_ENV_VAR).is_some_and(|v| v == "1")) ++} ++ ++/// The guest's enabled bits at the last tick, to log changes. ++static LAST_ENABLED: AtomicU32 = AtomicU32::new(u32::MAX); ++ + /// Whether the first tick of the process was logged. + static FIRST_TICK_LOGGED: AtomicBool = AtomicBool::new(false); + +@@ -151,10 +180,12 @@ impl Vblank { + /// thread is to be started (the first present). + pub(crate) fn presented(&self, now_ns: u64) -> bool { + let mut s = self.state.lock(); +- s.last_present = Some(now_ns); +- s.epoch = s.epoch.wrapping_add(1); +- s.ticks = 0; +- self.cv.notify_all(); ++ if !fixed_grid() || s.last_present.is_none() { ++ s.last_present = Some(now_ns); ++ s.epoch = s.epoch.wrapping_add(1); ++ s.ticks = 0; ++ self.cv.notify_all(); ++ } + let start = !s.started && !s.stopped; + s.started = true; + start +@@ -296,11 +327,22 @@ impl GuestDisplay

{ + return; + } + let p = self.page(&l); +- p.fetch_or(sp::RAISED_INTERRUPTS, INTERRUPT_PRESENT); + let enabled = p.load(sp::ENABLED_INTERRUPTS); +- if enabled & INTERRUPT_PRESENT != 0 { ++ let bits = tick_bits(); ++ let raise = bits & (enabled | INTERRUPT_PRESENT); ++ if raise != 0 { ++ p.fetch_or(sp::RAISED_INTERRUPTS, raise); ++ } ++ if enabled & raise != 0 { + self.poke(); + } ++ if LAST_ENABLED.swap(enabled, Ordering::Relaxed) != enabled { ++ log::info!( ++ "guest display on port {}: the guest enables interrupts {enabled:#x}, tick {}", ++ self.port, ++ tick.k ++ ); ++ } + let raised = vb.count(); + drop(l); + // vblank: schedule points since the last present, the guest's enabled interrupt bits diff --git a/patches/reims-vgpu/0011-trap-coherent-surfaces-hooks-and-tests.patch b/patches/reims-vgpu/0011-trap-coherent-surfaces-hooks-and-tests.patch new file mode 100644 index 0000000..608a4c3 --- /dev/null +++ b/patches/reims-vgpu/0011-trap-coherent-surfaces-hooks-and-tests.patch @@ -0,0 +1,1223 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 01:11:17 +0000 +Subject: [PATCH 11/25] trap-coherent surfaces, hooks and tests + +rvg_coherent_* joins QEMU's page states to metal2vulkan's shadow_coherent, the +unmap of a surface with work running is Busy, and the display samples the +surface's own image. coherent_test.c runs the pages on a real KVM guest; the +census report prints the per-present cost against eager copies. + +Co-Authored-By: Claude Opus 5.5 +--- + Cargo.lock | 1 + + crates/reims-vgpu-c/Cargo.toml | 4 +- + crates/reims-vgpu-c/include/reims_vgpu.h | 45 ++ + crates/reims-vgpu-c/src/census.rs | 8 + + crates/reims-vgpu-c/src/coherent.rs | 258 ++++++++++ + crates/reims-vgpu-c/src/lib.rs | 1 + + crates/reims-vgpu-c/tests/c/harness.c | 8 + + crates/reims-vgpu-c/tests/c_harness.rs | 2 +- + crates/reims-vgpu-core/src/census.rs | 10 + + crates/reims-vgpu-core/src/display/frame.rs | 4 +- + crates/reims-vgpu-core/src/iosurface/table.rs | 24 +- + .../reims-vgpu-core/src/platform/resource.rs | 10 +- + crates/reims-vgpu-vulkan/src/gpu/resource.rs | 14 +- + docs/api.md | 1 + + scripts/internal/census/census_report.py | 41 +- + scripts/internal/census/census_test.c | 13 +- + scripts/internal/census/coherent_test.c | 483 ++++++++++++++++++ + 17 files changed, 906 insertions(+), 21 deletions(-) + create mode 100644 crates/reims-vgpu-c/src/coherent.rs + create mode 100644 scripts/internal/census/coherent_test.c + +diff --git a/Cargo.lock b/Cargo.lock +index c82d6926..a9d888f1 100644 +--- a/Cargo.lock ++++ b/Cargo.lock +@@ -595,6 +595,7 @@ version = "0.1.0" + dependencies = [ + "libc", + "log", ++ "metal2vulkan-runtime", + "objc2", + "objc2-foundation", + "parking_lot", +diff --git a/crates/reims-vgpu-c/Cargo.toml b/crates/reims-vgpu-c/Cargo.toml +index 235b1dd4..9a063ca9 100644 +--- a/crates/reims-vgpu-c/Cargo.toml ++++ b/crates/reims-vgpu-c/Cargo.toml +@@ -21,10 +21,12 @@ macos-sdl = ["reims-vgpu/macos-sdl"] + # The Vulkan platform's proof of concept (architecture 11.1): every handle is made on + # `VulkanPlatform` in place of `MetalPlatform`. Off by default: the Metal platform is the product. + # `cargo xtask dist --platform vulkan` builds the dist tree with it. +-vulkan = ["reims-vgpu/vulkan"] ++vulkan = ["reims-vgpu/vulkan", "dep:metal2vulkan-runtime"] + + [dependencies] + reims-vgpu.workspace = true ++# LOCAL (Atlas): the trap-coherent surfaces' exports (`src/coherent.rs`). ++metal2vulkan-runtime = { workspace = true, optional = true } + # The platform trait's associated types (the unit tests' testkit platform stands where + # MetalPlatform stands in the product), and the log hook. + reims-vgpu-core.workspace = true +diff --git a/crates/reims-vgpu-c/include/reims_vgpu.h b/crates/reims-vgpu-c/include/reims_vgpu.h +index 705af8c0..3adaf431 100644 +--- a/crates/reims-vgpu-c/include/reims_vgpu.h ++++ b/crates/reims-vgpu-c/include/reims_vgpu.h +@@ -342,6 +342,8 @@ typedef struct { + uint32_t compression_type; + bool read_only; + bool host_only; ++ uint64_t base_address; ++ uint64_t base_offset; + } RvgCensusSurface; + + // `mapped(ctx, surface, pages, page_count, page_size)`: a surface was mapped, with each guest +@@ -373,6 +375,29 @@ typedef void (*RvgCensusUsedFn)(void *ctx, + // this header. + typedef void (*RvgCensusPresentedFn)(void *ctx, uint64_t mapping_id); + ++// What marks a trap-coherent surface's pages (metal2vulkan's `shadow_coherent.rs`). Addresses are ++// the library's mapping of the surface, whole pages, ascending, one surface to a call. ++// ++// Thread: any library thread, a vCPU's in `rvg_coherent_fault` among them, under the surface's ++// lock. May call: nothing of this header. ++typedef struct { ++ // sizeof(RvgCoherentHooks) as the host was compiled. ++ uint32_t struct_size; ++ void *ctx; ++ // Every page of the surface held from now on: whether it can be. ++ bool (*track)(void *ctx, uint64_t start, uint64_t len); ++ void (*untrack)(void *ctx, uint64_t start); ++ // Held pages armed. Into up to `room` of `refused`, those a CPU reached as their markers moved, ++ // left held: how many. ++ size_t (*arm)(void *ctx, const uint64_t *addrs, size_t n, uint64_t *refused, size_t room); ++ // Into up to `room` of `came`, the pages a CPU reached as their markers moved: how many. ++ size_t (*hold)(void *ctx, const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, ++ size_t room); ++ void (*open)(void *ctx, uint64_t addr); ++ void (*freeze)(void *ctx, const uint64_t *addrs, size_t n); ++ void (*thaw)(void *ctx, const uint64_t *addrs, size_t n); ++} RvgCoherentHooks; ++ + // The touch census's callbacks. + typedef struct { + // sizeof(RvgCensusCallbacks) as the host was compiled. +@@ -963,6 +988,26 @@ void rvg_log_set_handler(RvgLogFn handler, void *ctx, RvgLogLevel max_level); + // `callbacks` is NULL or a live table. Its functions and `ctx` live as long as the process. + RvgStatus rvg_census_set_callbacks(const RvgCensusCallbacks *callbacks); + ++// Installs what marks the pages of trap-coherent surfaces, once, before the first device. ++// RVG_ERR_INVALID_ARGUMENT for NULL, RVG_ERR_UNSUPPORTED on a platform without them. ++RvgStatus rvg_coherent_set_hooks(const RvgCoherentHooks *hooks); ++// The guest touched the page at `addr` of a surface's mapping: returns with its bytes current, ++// and, for a write, open. Takes the surface's lock: call it holding no lock `hooks` takes. ++void rvg_coherent_fault(uint64_t addr, bool write); ++// From the access's stop to its rerun, for the report. ++void rvg_coherent_timed(uint64_t addr, uint64_t ns); ++// A write hit a held page no one can name: every held page armed, so the rerun names it. ++void rvg_coherent_probe(void); ++// Every surface catches up and stops being tracked: the markers go now. ++void rvg_coherent_abandon(void); ++// The guest resets: every surface is forgotten, with vCPUs stopped. ++void rvg_coherent_reset(void); ++// The surface over `len` bytes from `start` was unmapped, and its pages are the guest's to reuse. ++void rvg_coherent_unmapped(uint64_t start, uint64_t len); ++// One JSON line a surface under `label` into `out`, up to `room` bytes: the bytes they took, ++// which may exceed `room`. ++size_t rvg_coherent_report(const char *label, char *out, size_t room); ++ + // A new device. Refusals, checked in this order: RVG_ERR_INVALID_ARGUMENT for the config + // itself (the files behind its memory map's ranges among it), RVG_ERR_UNSUPPORTED, + // RVG_ERR_DISPLAY_PORT_COUNT, RVG_ERR_NO_METAL_DEVICE, RVG_ERR_MEMORY_CONFLICT, +diff --git a/crates/reims-vgpu-c/src/census.rs b/crates/reims-vgpu-c/src/census.rs +index 3a85dfcf..13179db0 100644 +--- a/crates/reims-vgpu-c/src/census.rs ++++ b/crates/reims-vgpu-c/src/census.rs +@@ -28,6 +28,8 @@ pub struct RvgCensusSurface { + pub compression_type: u32, + pub read_only: bool, + pub host_only: bool, ++ pub base_address: u64, ++ pub base_offset: u64, + } + + /// `mapped(ctx, surface, pages, page_count, page_size)`: a surface was mapped, with each guest +@@ -107,6 +109,8 @@ impl Sink for HostSink { + compression_type: facts.compression_type, + read_only: facts.read_only, + host_only: facts.host_only, ++ base_address: facts.base_address, ++ base_offset: facts.base_offset, + }; + let (ctx, at, count) = (self.0.ctx, pages.as_ptr(), pages.len()); + // SAFETY: the host's function, given its context and what lives across the call. +@@ -115,6 +119,10 @@ impl Sink for HostSink { + }); + } + ++ fn unmapping(&self, start: u64, len: u64) -> bool { ++ crate::coherent::unmapping(start, len) ++ } ++ + fn unmapped(&self, mapping_id: u32) { + let Some(unmapped) = self.0.unmapped else { + return; +diff --git a/crates/reims-vgpu-c/src/coherent.rs b/crates/reims-vgpu-c/src/coherent.rs +new file mode 100644 +index 00000000..d8f6e9ad +--- /dev/null ++++ b/crates/reims-vgpu-c/src/coherent.rs +@@ -0,0 +1,258 @@ ++//! LOCAL (Atlas): trap-coherent surfaces' exports: QEMU's census view marks the pages of the ++//! surfaces metal2vulkan leaves ahead in their images (its `shadow_coherent.rs`), and tells of each ++//! touch here. On a platform without them, nothing here does anything. ++ ++use std::ffi::{CStr, c_char, c_void}; ++ ++use crate::ffi; ++use crate::sized::{self, HostStruct}; ++#[cfg(not(feature = "vulkan"))] ++use crate::status::RVG_ERR_UNSUPPORTED; ++#[cfg(feature = "vulkan")] ++use crate::status::RVG_OK; ++use crate::status::{RVG_ERR_INVALID_ARGUMENT, RvgStatus}; ++ ++/// What marks a trap-coherent surface's pages. See the header. ++#[repr(C)] ++#[derive(Clone, Copy)] ++#[expect( ++ missing_docs, ++ reason = "the fields' names say what they hold, as the header shows them" ++)] ++pub struct RvgCoherentHooks { ++ /// sizeof(RvgCoherentHooks) as the host was compiled. ++ pub struct_size: u32, ++ pub ctx: *mut c_void, ++ pub track: Option bool>, ++ pub untrack: Option, ++ pub arm: Option< ++ unsafe extern "C-unwind" fn( ++ ctx: *mut c_void, ++ addrs: *const u64, ++ n: usize, ++ refused: *mut u64, ++ room: usize, ++ ) -> usize, ++ >, ++ pub hold: Option< ++ unsafe extern "C-unwind" fn( ++ ctx: *mut c_void, ++ addrs: *const u64, ++ n: usize, ++ open_too: bool, ++ came: *mut u64, ++ room: usize, ++ ) -> usize, ++ >, ++ pub open: Option, ++ pub freeze: Option, ++ pub thaw: Option, ++} ++ ++// SAFETY: repr(C), a u32 first; a pointer and optional function pointers. ++unsafe impl HostStruct for RvgCoherentHooks {} ++ ++#[cfg(feature = "vulkan")] ++mod on { ++ use super::RvgCoherentHooks; ++ use metal2vulkan_runtime::shadow_coherent::Pages; ++ ++ pub(super) use metal2vulkan_runtime::shadow_coherent as runtime; ++ ++ /// The host's hooks, as the runtime's marks. ++ pub(super) struct HostPages(pub(super) RvgCoherentHooks); ++ ++ // SAFETY: the host's contract: its functions run on any of the library's threads. ++ unsafe impl Send for HostPages {} ++ // SAFETY: as above. ++ unsafe impl Sync for HostPages {} ++ ++ impl Pages for HostPages { ++ fn track(&self, start: u64, len: u64) -> bool { ++ // SAFETY: the host's function, given its context. ++ self.0.track.is_some_and(|track| unsafe { track(self.0.ctx, start, len) }) ++ } ++ ++ fn untrack(&self, start: u64) { ++ if let Some(untrack) = self.0.untrack { ++ // SAFETY: as above. ++ unsafe { untrack(self.0.ctx, start) }; ++ } ++ } ++ ++ fn arm(&self, pages: &[u64]) -> Vec { ++ let mut refused = vec![0; pages.len()]; ++ let Some(arm) = self.0.arm else { ++ return Vec::new(); ++ }; ++ let (at, count, to) = (pages.as_ptr(), pages.len(), refused.as_mut_ptr()); ++ // SAFETY: as above, with what lives across the call. ++ let found = unsafe { arm(self.0.ctx, at, count, to, count) }; ++ refused.truncate(found); ++ refused ++ } ++ ++ fn hold(&self, pages: &[u64], open_too: bool) -> Vec { ++ let mut came = vec![0; pages.len()]; ++ let Some(hold) = self.0.hold else { ++ return Vec::new(); ++ }; ++ let (at, count, to) = (pages.as_ptr(), pages.len(), came.as_mut_ptr()); ++ // SAFETY: as above. ++ let found = unsafe { hold(self.0.ctx, at, count, open_too, to, count) }; ++ came.truncate(found); ++ came ++ } ++ ++ fn open(&self, page: u64) { ++ if let Some(open) = self.0.open { ++ // SAFETY: as above. ++ unsafe { open(self.0.ctx, page) }; ++ } ++ } ++ ++ fn freeze(&self, pages: &[u64]) { ++ if let Some(freeze) = self.0.freeze { ++ // SAFETY: as above, with what lives across the call. ++ unsafe { freeze(self.0.ctx, pages.as_ptr(), pages.len()) }; ++ } ++ } ++ ++ fn thaw(&self, pages: &[u64]) { ++ if let Some(thaw) = self.0.thaw { ++ // SAFETY: as above, with what lives across the call. ++ unsafe { thaw(self.0.ctx, pages.as_ptr(), pages.len()) }; ++ } ++ } ++ } ++} ++ ++/// Before an unmap of the surface over `len` bytes from `start`: whether it may go on. ++pub(crate) fn unmapping(start: u64, len: u64) -> bool { ++ #[cfg(feature = "vulkan")] ++ return on::runtime::unmapping(start..start.wrapping_add(len)); ++ #[cfg(not(feature = "vulkan"))] ++ { ++ let _ = (start, len); ++ true ++ } ++} ++ ++/// Installs what marks the pages of trap-coherent surfaces. See the header. ++/// ++/// # Safety ++/// `hooks` is NULL or a live table, whose functions and `ctx` live as long as the process. ++#[unsafe(no_mangle)] ++pub unsafe extern "C" fn rvg_coherent_set_hooks(hooks: *const RvgCoherentHooks) -> RvgStatus { ++ ffi::export(|| { ++ // SAFETY: the caller's contract. ++ let table = match unsafe { sized::read(hooks) } { ++ Ok(Some(table)) => table, ++ Ok(None) => return RVG_ERR_INVALID_ARGUMENT, ++ Err(status) => return status, ++ }; ++ #[cfg(feature = "vulkan")] ++ { ++ let pages = std::sync::Arc::new(on::HostPages(table)); ++ on::runtime::set_pages(Some(pages)); ++ RVG_OK ++ } ++ #[cfg(not(feature = "vulkan"))] ++ { ++ let _ = table; ++ RVG_ERR_UNSUPPORTED ++ } ++ }) ++} ++ ++/// A page was touched. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_fault(addr: u64, write: bool) { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::fault(addr, write); ++ #[cfg(not(feature = "vulkan"))] ++ let _ = (addr, write); ++ }); ++} ++ ++/// How long an access stood. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_timed(addr: u64, ns: u64) { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::timed(addr, ns); ++ #[cfg(not(feature = "vulkan"))] ++ let _ = (addr, ns); ++ }); ++} ++ ++/// A write no one can name. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_probe() { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::probe(); ++ }); ++} ++ ++/// Every surface back in its pages. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_abandon() { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::abandon(); ++ }); ++} ++ ++/// The guest resets. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_reset() { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::reset(); ++ }); ++} ++ ++/// A surface went. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_unmapped(start: u64, len: u64) { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::runtime::unmapped(start..start.wrapping_add(len)); ++ #[cfg(not(feature = "vulkan"))] ++ let _ = (start, len); ++ }); ++} ++ ++/// The report. See the header. ++/// ++/// # Safety ++/// `label` is a C string or NULL, and `out` holds `room` bytes. ++#[unsafe(no_mangle)] ++pub unsafe extern "C" fn rvg_coherent_report( ++ label: *const c_char, ++ out: *mut c_char, ++ room: usize, ++) -> usize { ++ ffi::export(|| { ++ if label.is_null() { ++ return 0; ++ } ++ // SAFETY: the caller's contract. ++ let label = unsafe { CStr::from_ptr(label) }.to_string_lossy(); ++ #[cfg(feature = "vulkan")] ++ let text = on::runtime::report(&label); ++ #[cfg(not(feature = "vulkan"))] ++ let text = { ++ let _ = &label; ++ String::new() ++ }; ++ let count = text.len().min(room); ++ if count > 0 && !out.is_null() { ++ // SAFETY: the caller's contract; `count` is within `room`. ++ unsafe { std::ptr::copy_nonoverlapping(text.as_ptr(), out.cast(), count) }; ++ } ++ count ++ }) ++} +diff --git a/crates/reims-vgpu-c/src/lib.rs b/crates/reims-vgpu-c/src/lib.rs +index 50b2c78b..dcbf4cc8 100644 +--- a/crates/reims-vgpu-c/src/lib.rs ++++ b/crates/reims-vgpu-c/src/lib.rs +@@ -28,6 +28,7 @@ pub mod status; + pub mod log; + + pub mod census; ++pub mod coherent; + pub mod device; + pub mod display; + pub mod frame; +diff --git a/crates/reims-vgpu-c/tests/c/harness.c b/crates/reims-vgpu-c/tests/c/harness.c +index 2f060d5b..639e4aff 100644 +--- a/crates/reims-vgpu-c/tests/c/harness.c ++++ b/crates/reims-vgpu-c/tests/c/harness.c +@@ -178,6 +178,14 @@ static void check_status_and_log(void) { + rvg_log_set_handler(NULL, NULL, RVG_LOG_OFF); + + CHECK_STATUS(rvg_census_set_callbacks(NULL), RVG_ERR_INVALID_ARGUMENT); ++ CHECK_STATUS(rvg_coherent_set_hooks(NULL), RVG_ERR_INVALID_ARGUMENT); ++ rvg_coherent_fault(0, false); ++ rvg_coherent_timed(0, 0); ++ rvg_coherent_probe(); ++ rvg_coherent_abandon(); ++ rvg_coherent_reset(); ++ rvg_coherent_unmapped(0, 0); ++ CHECK(rvg_coherent_report("", NULL, 0) == 0); + CHECK(RVG_MAX_DISPLAY_PORTS == 8 && RVG_DEVICE_MMIO_SIZE == 0x4000); + CHECK(RVG_PCI_VENDOR_ID == 0x106b && RVG_PCI_DEVICE_ID == 0xeeee); + CHECK(RVG_PCI_BAR_MMIO == 0 && RVG_PCI_MSI_VECTORS == 1); +diff --git a/crates/reims-vgpu-c/tests/c_harness.rs b/crates/reims-vgpu-c/tests/c_harness.rs +index 8e7b6e3e..7b4c7053 100644 +--- a/crates/reims-vgpu-c/tests/c_harness.rs ++++ b/crates/reims-vgpu-c/tests/c_harness.rs +@@ -111,7 +111,7 @@ fn c_harness_calls_every_function() { + Some(&rest[..end]) + }) + .collect(); +- assert_eq!(declared.len(), 48, "{declared:?}"); ++ assert_eq!(declared.len(), 56, "{declared:?}"); + let missing: Vec<&str> = declared + .iter() + .copied() +diff --git a/crates/reims-vgpu-core/src/census.rs b/crates/reims-vgpu-core/src/census.rs +index 5d3ac435..291bcfcd 100644 +--- a/crates/reims-vgpu-core/src/census.rs ++++ b/crates/reims-vgpu-core/src/census.rs +@@ -34,6 +34,10 @@ pub struct SurfaceFacts { + pub read_only: bool, + /// `hostOnly`: the surface has no guest pages. + pub host_only: bool, ++ /// The host address of the surface's bytes in the library's mapping of them. ++ pub base_address: u64, ++ /// That address's offset in its first page. ++ pub base_offset: u64, + } + + /// Who hears of the surfaces. +@@ -41,6 +45,12 @@ pub trait Sink: Send + Sync { + /// A surface was mapped. `pages` is each guest page's host address, none for a host-only one. + /// On the IOSurface device's queue, under its state's lock. + fn mapped(&self, facts: &SurfaceFacts, pages: &[u64], page_size: u64); ++ /// The guest asks to unmap the surface over `len` bytes from the host address `start`, and ++ /// nothing was checked yet: whether it may go on, else it answers busy. What it held of the ++ /// surface it lets go of. ++ fn unmapping(&self, _start: u64, _len: u64) -> bool { ++ true ++ } + /// The surface at `mapping_id` goes. Called before the unmap returns to the guest, and + /// returns only when no trap remains on its pages, which the guest may reuse at once. + fn unmapped(&self, mapping_id: u32); +diff --git a/crates/reims-vgpu-core/src/display/frame.rs b/crates/reims-vgpu-core/src/display/frame.rs +index e28a308e..551e03d8 100644 +--- a/crates/reims-vgpu-core/src/display/frame.rs ++++ b/crates/reims-vgpu-core/src/display/frame.rs +@@ -74,9 +74,7 @@ impl DisplayCore

{ + completion: Box, + ) -> RaiseResult<()> { + // LOCAL (Atlas): before the display's lock, as this may commit a command buffer. +- if let Some(frame) = &frame { +- frame.presented(); +- } ++ let frame = frame.map(|frame| frame.presented().unwrap_or(frame)); + let mut g = self.display_mutex.lock(); + // present: frames in flight, waited for next + crate::trace::event( +diff --git a/crates/reims-vgpu-core/src/iosurface/table.rs b/crates/reims-vgpu-core/src/iosurface/table.rs +index 4c1ac55b..f0fcf652 100644 +--- a/crates/reims-vgpu-core/src/iosurface/table.rs ++++ b/crates/reims-vgpu-core/src/iosurface/table.rs +@@ -85,6 +85,8 @@ fn census_facts( + compression_type: u32::from(base.compression_type), + read_only: base.read_only(), + host_only: base.host_only(), ++ base_address: 0, ++ base_offset: 0, + } + } + +@@ -162,11 +164,12 @@ impl HostDeviceCore

{ + slot.pages = Vec::new(); + } + if made && let Some(sink) = crate::census::sink() { +- sink.mapped( +- &census_facts(mapping_id, &descriptor), +- &slot.pages, +- self.page_size(), +- ); ++ let mut facts = census_facts(mapping_id, &descriptor); ++ if let Some(surface) = &slot.surface { ++ facts.base_address = surface.base_address().addr() as u64; ++ facts.base_offset = slot.base_offset; ++ } ++ sink.mapped(&facts, &slot.pages, self.page_size()); + } + if let Some(s) = st.slots.get_mut(index) { + *s = slot; +@@ -306,6 +309,17 @@ impl HostDeviceCore

{ + return IoReturn::BadArgument; + } + let index = mapping_id.to_usize(); ++ if let Some(sink) = crate::census::sink() ++ && let Some(slot) = st.slots.get(index) ++ && let Some(surface) = &slot.surface ++ { ++ let start = (surface.base_address().addr() as u64).wrapping_sub(slot.base_offset); ++ let len = slot.page_count.wrapping_mul(self.page_size()); ++ if !sink.unmapping(start, len) { ++ log::debug!("unmap: surface {mapping_id} has GPU work running in its image"); ++ return IoReturn::Busy; ++ } ++ } + let Some(retain_count) = st + .slots + .get(index) +diff --git a/crates/reims-vgpu-core/src/platform/resource.rs b/crates/reims-vgpu-core/src/platform/resource.rs +index ca6f92c6..b8a42032 100644 +--- a/crates/reims-vgpu-core/src/platform/resource.rs ++++ b/crates/reims-vgpu-core/src/platform/resource.rs +@@ -83,8 +83,14 @@ pub trait Buffer: GpuResource

{ + pub trait Texture: GpuResource

{ + /// LOCAL (Atlas): not a Metal call. The display is about to show the texture's bytes, for a + /// platform that leaves a surface's writes out of them until then (metal2vulkan's +- /// `shadow_lazy.rs`). +- fn presented(&self) {} ++ /// `shadow_lazy.rs`). Under `surface_coherent`, the texture to show in its place, which holds ++ /// those bytes ahead of them (`shadow_coherent.rs`). ++ fn presented(&self) -> Option ++ where ++ Self: Sized, ++ { ++ None ++ } + /// sel: arrayLength + fn array_length(&self) -> u64; + /// sel: depth +diff --git a/crates/reims-vgpu-vulkan/src/gpu/resource.rs b/crates/reims-vgpu-vulkan/src/gpu/resource.rs +index 546fcd99..30b1a129 100644 +--- a/crates/reims-vgpu-vulkan/src/gpu/resource.rs ++++ b/crates/reims-vgpu-vulkan/src/gpu/resource.rs +@@ -1274,8 +1274,18 @@ fn at(index: u64) -> rt::IdPlacement { + } + + impl Texture for VulkanTexture { +- fn presented(&self) { +- plain(|| self.raw.surface_presented()); ++ fn presented(&self) -> Option { ++ let raw = plain(|| self.raw.surface_presented())?; ++ Some(Obj::new(TextureObject { ++ raw, ++ device: self.device.clone_ref(), ++ surface: self.surface.clone(), ++ plane: self.plane, ++ under: None, ++ private_usage: self.private_usage, ++ space: self.space.clone(), ++ gpu_read_only: self.gpu_read_only, ++ })) + } + + fn array_length(&self) -> u64 { +diff --git a/docs/api.md b/docs/api.md +index 2ddabe01..900d4255 100644 +--- a/docs/api.md ++++ b/docs/api.md +@@ -703,6 +703,7 @@ The header's 47 functions, in its order. "May" is the header's mark "may call ba + | 40 | `rvg_video_device_invalidate` | | a reset's first phase: the session sends the guest nothing more | + | 41 | `rvg_video_device_reset` | | both phases: the session's work drains and it ends | + | 42a | `rvg_census_set_callbacks` | | LOCAL (Atlas): the touch census's news of surfaces and presents (`reims_vgpu_core::census`); no calls without it | ++| 42b | `rvg_coherent_set_hooks`, `rvg_coherent_fault`, `rvg_coherent_timed`, `rvg_coherent_probe`, `rvg_coherent_abandon`, `rvg_coherent_reset`, `rvg_coherent_unmapped`, `rvg_coherent_report` | | LOCAL (Atlas): trap-coherent surfaces (metal2vulkan's `shadow_coherent.rs`): QEMU's census view marks their pages, and tells of each touch; no effect without the live switch `surface_coherent` | + | 42 | `rvg_scaler_device_create` | | the M2 scaler device (106b:1a0b), from an `RvgScalerDeviceConfig`, made on every host: transforming with the library's own kernel on the GPU it draws with (Metal's on macOS, the Vulkan platform's on Linux, any GPU, a software rasterizer included), and with none where that opens no device or the device refuses the kernel | + | 43 | `rvg_scaler_device_release` | | drops the host's reference; the last one drops its accelerator and ends its guest-memory table | + | 44 | `rvg_scaler_device_scales` | | whether the device has a GPU to transform on; where it has not, the host refuses the guest's features (FEATURES_OK), so the guest's driver does without the device | +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +index c3609449..b13d5aeb 100644 +--- a/scripts/internal/census/census_report.py ++++ b/scripts/internal/census/census_report.py +@@ -5,9 +5,12 @@ + census_report.py

--mark LABEL between the mark LABEL and the next mark + census_report.py --marks the marks in the timeline, and each span's totals + census_report.py --disasm N the top N RIPs, disassembled with objdump ++ census_report.py --coherent [LABEL] the trap-coherent surfaces at a mark ("exit", the last) + +-A mark is one line in /mark ("motionbench"), which the census reads within 50 ms. "every 1" +-there sets the re-arm period instead. A page counts once a window: touches/frame cannot exceed ++A mark is one line of /mark ("motionbench"), which the census reads within 50 ms; append with ++>>, so that one mark never overwrites another. A line "every 1" sets the re-arm period instead. ++Under REIMS_VGPU_COHERENT=1 each mark and the exit write one report per surface to coherent.jsonl. ++A page counts once a window: touches/frame cannot exceed + pages/window divided by the window's frames, so read `touched windows` with `every`. + + pages R were only read in their window and pages W were caught writing. pages R|W were read, then +@@ -132,6 +135,35 @@ def life(surfaces): + f" {e['writes']:>7} {e['windows']:>6} {e['windows_touched']:>7}") + + ++def coherent(directory, wanted): ++ """The last report of each surface at the mark `wanted`, or at the last mark.""" ++ rows = list(lines(os.path.join(directory, "coherent.jsonl"))) ++ label = wanted or (rows[-1].get("label") if rows else None) ++ if label is None: ++ sys.exit(f"{directory}: no coherent.jsonl") ++ found = {} ++ for r in rows: ++ if r.get("label") == label or r.get("label") == "closed": ++ found[r["at"]] = r ++ print(f"mark {label}: per present in us (detect + sync + fetch against what eager copies would" ++ " have cost)") ++ print(f"{'surface':>12} {'size':>10} {'present':>7} {'traps':>7} {'/s':>7} {'pages':>6}" ++ f" {'regions':>7} {'MiB':>7} {'sync ms':>8} {'trap us':>8} {'demote':>6} {'writes':>6}" ++ f" {'detect':>7} {'sync':>7} {'fetch':>7} {'total':>7} {'eager':>8} pays") ++ for r in sorted(found.values(), key=lambda r: -r["traps"]): ++ cost = r["per_present_ns"] ++ trap = r["trap_ns"] / r["resumes"] / 1e3 if r["resumes"] else 0 ++ print(f"{r['at']:>12} {str(r['w']) + 'x' + str(r['h']):>10} {r['presents']:>7} {r['traps']:>7}" ++ f" {r['traps_per_s']:>7.1f} {r['unique_pages']:>6} {r['regions']:>7}" ++ f" {r['bytes_detiled'] / 2 ** 20:>7.1f} {r['sync_ns'] / 1e6:>8.1f} {trap:>8.1f}" ++ f" {r['demotions']:>6} {r['writes']:>6} {cost['detect'] / 1e3:>7.1f}" ++ f" {cost['sync'] / 1e3:>7.1f} {cost['fetch'] / 1e3:>7.1f} {cost['total'] / 1e3:>7.1f}" ++ f" {cost['eager_would'] / 1e3:>8.1f} {'yes' if r['pays'] else 'NO'}") ++ bad = [r for r in found.values() if r["sync_errors"] or r["overwrite_risk"]] ++ for r in bad: ++ print(f"surface {r['at']}: sync_errors {r['sync_errors']}, overwrite_risk {r['overwrite_risk']}") ++ ++ + def disasm(directory, count, timeline): + bytes_at = {r["rip"]: r for r in lines(os.path.join(directory, "rips.jsonl"))} + totals = {} +@@ -159,6 +191,7 @@ def main(): + a.add_argument("--mark") + a.add_argument("--marks", action="store_true") + a.add_argument("--disasm", type=int, default=0) ++ a.add_argument("--coherent", nargs="?", const="", default=None) + args = a.parse_args() + header, surfaces, timeline = read(args.dir) + if not header: +@@ -170,7 +203,9 @@ def main(): + print(f"census pid {header.get('pid')}: traps {h['traps']}, writes {h['writes']}," + f" upgrades {h['upgrades']}, spurious {h['spurious']}, dropped {h['dropped']}," + f" errors {h['errors']}, off by fuse: {'YES' if h['dead'] else 'no'}") +- if args.disasm: ++ if args.coherent is not None: ++ coherent(args.dir, args.coherent) ++ elif args.disasm: + disasm(args.dir, args.disasm, timeline) + elif args.marks: + for label, start, end in spans(timeline): +diff --git a/scripts/internal/census/census_test.c b/scripts/internal/census/census_test.c +index 2f9b7d7d..de2d48ba 100644 +--- a/scripts/internal/census/census_test.c ++++ b/scripts/internal/census/census_test.c +@@ -483,10 +483,8 @@ static void t_cost(struct vcpu *c) { + free(list); + } + +-int main(int argc, char **argv) { +- setvbuf(stdout, 0, _IOLBF, 0); +- if (argc < 2) return 2; +- uint64_t mib = argc > 2 ? atol(argv[2]) : 512; ++// A VM, its slots on the census's view, and vCPU `c`: what every test here starts from. ++static void setup(char **argv, uint64_t mib) { + setenv(CENSUS_ENV, argv[1], 1); + setenv("REIMS_VGPU_CENSUS_MIN_PIXELS", "1", 1); + setenv("REIMS_VGPU_CENSUS_WINDOW_MS", "100000", 1); +@@ -525,7 +523,13 @@ int main(int argc, char **argv) { + sigset_t bus; + sigemptyset(&bus); sigaddset(&bus, SIGBUS); + pthread_sigmask(SIG_BLOCK, &bus, 0); ++} + ++#ifndef CENSUS_TEST_NO_MAIN ++int main(int argc, char **argv) { ++ setvbuf(stdout, 0, _IOLBF, 0); ++ if (argc < 2) return 2; ++ setup(argv, argc > 2 ? atol(argv[2]) : 512); + struct vcpu c; + vcpu_init(&c, 0); + t_classify(&c); +@@ -546,3 +550,4 @@ int main(int argc, char **argv) { + printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); + return failures != 0; + } ++#endif +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +new file mode 100644 +index 00000000..3cb7eca5 +--- /dev/null ++++ b/scripts/internal/census/coherent_test.c +@@ -0,0 +1,483 @@ ++// coherent_test.c: the coherent pages of QEMU's accel/kvm/census.c on a real KVM guest, with no ++// QEMU and no library: this file plays the library's surface (an image, a lock, syncs at faults), ++// census_test.c plays kvm-all.c, and the guest's instructions run in CPL0 on real pages, so ++// `rep movsb` and the emulator's accesses are the ones the kernel copies run. ++// gcc -O2 -g -Wall -D_GNU_SOURCE -I$QEMU/include coherent_test.c $QEMU/accel/kvm/census.c -lpthread ++// ./a.out [slot_mib] ++#define CENSUS_TEST_NO_MAIN ++#include "census_test.c" ++ ++#include ++ ++#define PAGES 16 ++#define WORDS (PG / 8) ++ ++// One surface: the "image" holds each page's current word, and a page is ahead while `ahead`. ++static struct { ++ pthread_mutex_t lock; ++ uint64_t base, off; ++ uint32_t id; ++ uint64_t image[PAGES]; ++ uint8_t ahead[PAGES]; ++ int probing, faults, syncs, writes, probes, resets, abandons; ++ uint64_t came, refused, probed_at; ++ int beside; // A sync leaves the first word of each page: the rectangle stops short of it. ++ uint32_t dirty; // Pages the guest may have written, as the library would know. ++} s = {.lock = PTHREAD_MUTEX_INITIALIZER}; ++ ++static uint64_t page_at(int i) { return s.base + i * PG; } ++ ++static void note(const uint64_t *pages, size_t n) { ++ for (size_t i = 0; i < n; i++) s.dirty |= 1u << ((pages[i] - s.base) / PG); ++} ++ ++static void list_all(uint64_t *out) { ++ for (int i = 0; i < PAGES; i++) out[i] = page_at(i); ++} ++ ++static void sync_all_locked(void) { ++ uint64_t list[PAGES], came[PAGES]; ++ int n = 0; ++ ++ for (int i = 0; i < PAGES; i++) { ++ if (s.ahead[i]) { ++ uint64_t *to = (uint64_t *)page_at(i); ++ ++ for (uint64_t w = s.beside; w < WORDS; w++) to[w] = s.image[i]; ++ s.ahead[i] = 0; ++ list[n++] = page_at(i); ++ s.syncs++; ++ } ++ } ++ if (n) { ++ size_t found = census_coh_hold(list, n, false, came, PAGES); ++ ++ s.came += found; ++ note(came, found); ++ } ++} ++ ++static void on_fault(void *ctx, uint64_t addr, bool write) { ++ uint64_t page = addr & ~(PG - 1); ++ ++ pthread_mutex_lock(&s.lock); ++ s.faults++; ++ sync_all_locked(); ++ if (s.probing && (now_ns() - s.probed_at > 1000000 || (write && s.probing == 1))) { ++ uint64_t list[PAGES], came[PAGES]; ++ ++ s.probing = 0; ++ list_all(list); ++ size_t found = census_coh_hold(list, PAGES, false, came, PAGES); ++ s.came += found; ++ note(came, found); ++ } else if (write && s.probing) { ++ s.probing--; ++ } ++ census_coh_hold(&page, 1, false, NULL, 0); ++ if (write) { ++ census_coh_open(page); ++ note(&page, 1); ++ s.writes++; ++ } ++ pthread_mutex_unlock(&s.lock); ++} ++ ++static void on_timed(void *ctx, uint64_t addr, uint64_t ns) {} ++ ++static void on_probe(void *ctx) { ++ uint64_t list[PAGES]; ++ ++ pthread_mutex_lock(&s.lock); ++ s.probing++; ++ s.probed_at = now_ns(); ++ s.probes++; ++ list_all(list); ++ size_t found = census_coh_arm(list, PAGES, list, PAGES); ++ s.came += found; ++ note(list, found); ++ pthread_mutex_unlock(&s.lock); ++} ++ ++static void on_abandon(void *ctx) { ++ pthread_mutex_lock(&s.lock); ++ sync_all_locked(); ++ s.abandons++; ++ census_coh_untrack(s.base); ++ pthread_mutex_unlock(&s.lock); ++} ++ ++static void on_reset(void *ctx) { s.resets++; } ++ ++static size_t on_report(void *ctx, const char *label, char *out, size_t room) { return 0; } ++ ++static const CensusCoherentOps ops = { ++ .fault = on_fault, .timed = on_timed, .probe = on_probe, .abandon = on_abandon, ++ .reset = on_reset, .report = on_report, ++}; ++ ++static void surface(uint32_t id, uint64_t off) { ++ uint64_t list[PAGES]; ++ ++ for (uint64_t i = 0; i < PAGES; i++) list[i] = (uint64_t)main_map + off + i * PG; ++ CensusSurface f = {.mapping_id = id, .width = 2000, .height = 2000, .pixel_format = 0x42475241, ++ .alloc_size = PAGES * PG, .bytes_per_row = 8000, ++ .base_address = (uint64_t)main_map + off}; ++ census_surface_mapped(&f, list, PAGES, PG); ++ s.id = id; s.off = off; s.base = (uint64_t)main_map + off; ++ memset(s.ahead, 0, sizeof s.ahead); ++ s.probing = 0; ++ if (!census_coh_track(s.base, PAGES * PG)) DIE("track"); ++} ++ ++// The GPU wrote `word` over the pages `mask` holds, and the command buffer completed. ++static void completed(uint32_t mask, uint64_t word) { ++ uint64_t list[PAGES]; ++ int n = 0; ++ ++ pthread_mutex_lock(&s.lock); ++ for (int i = 0; i < PAGES; i++) { ++ if (mask >> i & 1) { ++ s.image[i] = word; ++ s.ahead[i] = 1; ++ list[n++] = page_at(i); ++ } ++ } ++ size_t refused = census_coh_arm(list, n, list, PAGES); ++ if (refused) { ++ // As the library: no trap will come, and no one has heard yet. ++ s.refused += refused; ++ sync_all_locked(); ++ note(list, refused); ++ } ++ pthread_mutex_unlock(&s.lock); ++} ++ ++static int state(int page) { return census_page_state(main_map + s.off + page * PG); } ++static uint64_t gpa(int page) { return DATA_GPA + s.off + page * PG; } ++ ++static void t_read(struct vcpu *c) { ++ printf("[read] the guest reads a page the image is ahead of\n"); ++ surface(1, 0x100000); ++ CHECK(state(0) == 2, "tracked: held"); ++ completed(1, 0x1111); ++ CHECK(state(0) == 1, "completion arms it"); ++ setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO && getr(c).rax == 0x1111, "the read gets the image's word"); ++ CHECK(state(0) == 2, "held after the sync"); ++ int faults = s.faults; ++ setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); ++ run_guest(c, 0); ++ CHECK(s.faults == faults, "a second read is free"); ++ census_surface_unmapped(1); ++} ++ ++static void t_write(struct vcpu *c) { ++ printf("[write] a write to a held page names its page, with no read before it\n"); ++ surface(2, 0x200000); ++ int probes = s.probes; ++ setr(c, STUB(g_write), 0x2222, gpa(3), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "the write ran"); ++ CHECK(s.probes == probes + 1, "by one probe"); ++ CHECK(state(3) == 3 && state(2) == 2 && state(4) == 2, "that page alone is open"); ++ CHECK(*(uint64_t *)(main_map + s.off + 3 * PG) == 0x2222, "the guest's word is in the page"); ++ setr(c, STUB(g_write), 0x3333, gpa(3), 0, 0, 0, 0); ++ run_guest(c, 0); ++ CHECK(s.probes == probes + 1, "an open page takes writes free"); ++ ++ printf("[write] a read that traps, then a write of the page at the same instruction\n"); ++ completed(1 << 5, 0x4444); ++ setr(c, STUB(g_lock), 0, gpa(5), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "lock incq ran"); ++ CHECK(*(uint64_t *)(main_map + s.off + 5 * PG) == 0x4445, "over the synced word"); ++ CHECK(state(5) == 3, "and the page is open"); ++ census_surface_unmapped(2); ++} ++ ++static void t_frozen(struct vcpu *c) { ++ printf("[frozen] a page poisoned while open stops the guest until it thaws\n"); ++ surface(3, 0x300000); ++ setr(c, STUB(g_write), 0x55, gpa(1), 0, 0, 0, 0); ++ run_guest(c, 0); ++ uint64_t page = page_at(1); ++ census_coh_freeze(&page, 1); ++ CHECK(state(1) == 7, "frozen"); ++ setr(c, STUB(g_read), 0, gpa(1), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO && getr(c).rax == 0x55, "the read waited and got the word"); ++ CHECK(state(1) == 3, "open again"); ++ census_surface_unmapped(3); ++} ++ ++static void t_kernel_copy(struct vcpu *c) { ++ printf("[rep movsb] a CPL0 copy over armed pages, as the kernel's at 0xffffff8005901092\n"); ++ surface(4, 0x400000); ++ completed(0xff, 0x6666); ++ int syncs = s.syncs; ++ uint64_t to = DATA_GPA + 0x500000; ++ setr(c, STUB(g_repmovs), 0, 0, 8 * PG, 0, gpa(0), to); ++ CHECK(run_guest(c, 0) == RUN_IO, "the copy ran"); ++ CHECK(*(uint64_t *)(main_map + 0x500000) == 0x6666 && ++ *(uint64_t *)(main_map + 0x500000 + 8 * PG - 8) == 0x6666, "every byte is the image's"); ++ CHECK(s.syncs == syncs + 8, "one sync brought all eight pages"); ++ census_surface_unmapped(4); ++ ++ printf("[rep movsb] a CPL0 copy into held pages\n"); ++ surface(5, 0x400000); ++ memset(main_map + 0x500000, 0x77, 4 * PG); ++ setr(c, STUB(g_repmovs), 0, 0, 4 * PG, 0, DATA_GPA + 0x500000, gpa(2)); ++ CHECK(run_guest(c, 0) == RUN_IO, "the copy ran"); ++ CHECK(state(2) == 3 && state(5) == 3 && state(6) == 2, "the pages it wrote are open"); ++ CHECK(*(uint64_t *)(main_map + s.off + 5 * PG) == 0x7777777777777777ULL, "with its bytes"); ++ census_surface_unmapped(5); ++} ++ ++static void t_emulator_read(struct vcpu *c) { ++ printf("[emulator] QEMU's own access to an armed page faults like the guest's\n"); ++ surface(6, 0x600000); ++ completed(1, 0x8888); ++ census_coherent_access(main_map + s.off, 8, false); ++ CHECK(state(0) == 2 && *(uint64_t *)(main_map + s.off) == 0x8888, "a device's read syncs"); ++ census_coherent_access(main_map + s.off + PG, 8, true); ++ CHECK(state(1) == 3, "a device's write opens the page"); ++ census_surface_unmapped(6); ++} ++ ++static void t_unmap_reset(struct vcpu *c) { ++ printf("[unmap] armed, held and open pages all come off\n"); ++ surface(7, 0x700000); ++ completed(0xf, 1); ++ setr(c, STUB(g_write), 9, gpa(8), 0, 0, 0, 0); ++ run_guest(c, 0); ++ census_surface_unmapped(7); ++ int plain = 1; ++ for (int i = 0; i < PAGES; i++) plain &= state(i) == 6; ++ CHECK(plain, "no marker left"); ++ setr(c, STUB(g_write), 3, gpa(0), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO && *(uint64_t *)(main_map + s.off) == 3, "the guest reuses the page"); ++ printf("[reset] the guest resets\n"); ++ surface(8, 0x700000); ++ completed(0xf, 1); ++ int resets = s.resets; ++ census_reset(); ++ CHECK(s.resets == resets + 1, "the library heard"); ++ setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO, "and a read after it passes"); ++} ++ ++// The guest reads, and must see every word whose completion it could have heard of. ++static atomic_uint_fast64_t done_word; ++static atomic_int race_stop; ++struct guest { struct vcpu c; int fatal, stale, id; uint64_t rounds; }; ++static void *reader(void *p) { ++ struct guest *g = p; ++ ++ while (!atomic_load(&race_stop) && !g->fatal) { ++ uint64_t known = atomic_load(&done_word); ++ int page = g->rounds++ % 8; ++ ++ setr(&g->c, STUB(g_read), 0, gpa(page), 0, 0, 0, 0); ++ g->fatal = run_guest(&g->c, 0) != RUN_IO; ++ uint64_t got = getr(&g->c).rax; ++ if (got < known && g->stale++ < 3) printf(" read page %d: %lu, known %lu state %d\n", page, got, known, state(page)); ++ } ++ return 0; ++} ++// Writes its own page and reads it back through the main mapping: nothing may lose it. ++static void *writer(void *p) { ++ struct guest *g = p; ++ uint64_t *own = (uint64_t *)(main_map + s.off + (8 + g->id) * PG); ++ ++ while (!atomic_load(&race_stop) && !g->fatal) { ++ uint64_t word = ++g->rounds; ++ ++ setr(&g->c, STUB(g_write), word, gpa(8 + g->id), 0, 0, 0, 0); ++ g->fatal = run_guest(&g->c, 0) != RUN_IO; ++ if (*own != word && g->stale++ < 3) printf(" wrote %lu, own %lu state %d rip %llx\n", word, *own, state(8 + g->id), getr(&g->c).rip); ++ } ++ return 0; ++} ++ ++static void t_coherent_race(uint64_t mib) { ++ printf("[race] %lu MiB: 2 readers and 2 writers, armed and held again every 100 us\n", mib); ++ enum { R = 2, W = 2 }; ++ struct guest g[R + W]; ++ pthread_t th[R + W]; ++ CensusCounts a, b; ++ ++ surface(9, 0x800000); ++ census_counts(&a); ++ atomic_store(&done_word, 0); ++ atomic_store(&race_stop, 0); ++ for (int i = 0; i < R + W; i++) { ++ memset(&g[i], 0, sizeof g[i]); ++ vcpu_init(&g[i].c, 20 + i); ++ g[i].id = i - R; ++ pthread_create(&th[i], 0, i < R ? reader : writer, &g[i]); ++ } ++ uint64_t t0 = now_ns(), word = 0; ++ for (int i = 0; i < 30000; i++) { ++ completed(0xff, ++word); ++ // Only now could the guest have heard: its next read must see this word. ++ atomic_store(&done_word, word); ++ if (i % 7 == 0) { ++ uint64_t list[PAGES], came[PAGES]; ++ ++ list_all(list); ++ pthread_mutex_lock(&s.lock); ++ sync_all_locked(); ++ s.came += census_coh_hold(list + 8, 8, true, came, PAGES); ++ pthread_mutex_unlock(&s.lock); ++ } ++ usleep(100); ++ } ++ atomic_store(&race_stop, 1); ++ int fatal = 0, stale = 0; ++ uint64_t reads = 0, writes = 0; ++ for (int i = 0; i < R + W; i++) { ++ pthread_join(th[i], 0); ++ fatal |= g[i].fatal; ++ stale += g[i].stale; ++ *(i < R ? &reads : &writes) += g[i].rounds; ++ } ++ census_counts(&b); ++ CHECK(!fatal, "no vCPU died in %.1f s: %lu reads, %lu writes, %lu traps, %lu probes, %lu spurious", ++ (now_ns() - t0) / 1e9, reads, writes, b.traps - a.traps, b.upgrades - a.upgrades, ++ b.spurious - a.spurious); ++ CHECK(!stale, "no read older than the last completion, no write lost (%d)", stale); ++ CHECK(b.errors == 0 && !b.dead, "no uffd error, fuse not blown (errors %lu)", b.errors); ++ printf(" pages reached as their markers moved: %lu held, %lu armed\n", s.came, s.refused); ++ census_surface_unmapped(9); ++} ++ ++// Writes the first word of a page again and again, beside the rectangles the GPU leaves in it. ++static void *beside_writer(void *p) { ++ struct guest *g = p; ++ ++ while (!atomic_load(&race_stop) && !g->fatal) { ++ setr(&g->c, STUB(g_write), ++g->rounds, gpa(0), 0, 0, 0, 0); ++ g->fatal = run_guest(&g->c, 0) != RUN_IO; ++ } ++ return 0; ++} ++ ++static void t_arm_window(void) { ++ printf("[arming] a write as a page's markers move is in the page and the library knows it\n"); ++ struct guest g; ++ pthread_t th; ++ ++ surface(12, 0xe00000); ++ s.beside = 1; ++ s.dirty = 0; ++ atomic_store(&race_stop, 0); ++ memset(&g, 0, sizeof g); ++ vcpu_init(&g.c, 30); ++ pthread_create(&th, 0, beside_writer, &g); ++ volatile uint64_t *word = (uint64_t *)main_map + s.off / 8; ++ uint64_t known = 0, unseen = 0, seen = 0, list[PAGES], came[PAGES]; ++ list_all(list); ++ for (int i = 0; i < 60000 && !g.fatal; i++) { ++ completed(1, i); ++ for (int spin = 0; spin < 200; spin++) __asm__("pause"); ++ pthread_mutex_lock(&s.lock); ++ sync_all_locked(); ++ size_t found = census_coh_hold(list, 1, true, came, PAGES); ++ note(came, found); ++ // Held: no write gets by from here to the arm. ++ uint64_t now = *word; ++ if (now != known && !(s.dirty & 1)) unseen++; ++ seen += now != known; ++ known = now; ++ s.dirty = 0; ++ pthread_mutex_unlock(&s.lock); ++ } ++ atomic_store(&race_stop, 1); ++ pthread_join(th, 0); ++ CHECK(!g.fatal && !unseen, "%lu writes in the rounds, %lu the library did not hear of", seen, unseen); ++ s.beside = 0; ++ census_surface_unmapped(12); ++} ++ ++static void t_coherent_cost(struct vcpu *c) { ++ printf("[cost] trap to resume of a read whose sync moves one page\n"); ++ surface(10, 0xc00000); ++ enum { N = 2000 }; ++ uint64_t total = 0; ++ for (int i = 0; i < N; i++) { ++ completed(1, i + 1); ++ setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); ++ uint64_t t0 = now_ns(); ++ run_guest(c, 0); ++ total += now_ns() - t0; ++ } ++ printf(" %.2f us a read, the page's arm before it apart\n", total / 1e3 / N); ++ uint64_t t0 = now_ns(); ++ for (int i = 0; i < N; i++) completed(1, 0), sync_all_locked(); ++ printf(" %.2f us to arm and hold a page\n", (now_ns() - t0) / 1e3 / N); ++ census_surface_unmapped(10); ++ ++ printf("[cost] arming and holding a surface's run of pages, nothing touching them\n"); ++ static uint64_t list[3600], refused[3600], came[3600]; ++ uint32_t id = 13; ++ for (size_t n = 937; n <= 3516; n += 2579, id++) { ++ uint64_t off = 0x10000000; ++ ++ for (size_t i = 0; i < n; i++) list[i] = (uint64_t)main_map + off + i * PG; ++ CensusSurface f = {.mapping_id = id, .width = 2000, .height = 2000, .pixel_format = 0x42475241, ++ .alloc_size = n * PG, .bytes_per_row = 8000, ++ .base_address = (uint64_t)main_map + off}; ++ census_surface_mapped(&f, list, n, PG); ++ census_coh_track(list[0], n * PG); ++ uint64_t armed = 0, held = 0; ++ for (int i = 0; i < 300; i++) { ++ uint64_t t0 = now_ns(); ++ census_coh_arm(list, n, refused, 3600); ++ uint64_t t1 = now_ns(); ++ census_coh_hold(list, n, false, came, 3600); ++ armed += t1 - t0; ++ held += now_ns() - t1; ++ } ++ printf(" %zu pages: arm %.1f us, hold %.1f us\n", n, armed / 300e3, held / 300e3); ++ census_surface_unmapped(id); ++ } ++} ++ ++static void t_fuse_coherent(struct vcpu *c) { ++ printf("[fuse] an -EFAULT nothing explains: everything lands, markers off\n"); ++ surface(11, 0xd00000); ++ completed(1, 0x9999); ++ int abandons = s.abandons; ++ census_fuse_test(); ++ CHECK(s.abandons == abandons + 1, "the library caught up first"); ++ CHECK(*(uint64_t *)(main_map + s.off) == 0x9999, "with the image's word"); ++ int plain = 1, faults = s.faults; ++ for (int i = 0; i < PAGES; i++) { ++ setr(c, STUB(g_write), i, gpa(i), 0, 0, 0, 0); ++ plain &= run_guest(c, 0) == RUN_IO && *(uint64_t *)(main_map + s.off + i * PG) == (uint64_t)i; ++ } ++ CHECK(plain && s.faults == faults, "and the guest writes every page with no marker left"); ++} ++ ++int main(int argc, char **argv) { ++ setvbuf(stdout, 0, _IOLBF, 0); ++ if (argc < 2) return 2; ++ uint64_t mib = argc > 2 ? atol(argv[2]) : 512; ++ setenv("REIMS_VGPU_COHERENT", "1", 1); ++ setup(argv, mib); ++ CHECK(census_coherent(), "coherent is on"); ++ census_coherent_set_ops(&ops); ++ struct vcpu c; ++ vcpu_init(&c, 0); ++ t_read(&c); ++ t_write(&c); ++ t_frozen(&c); ++ t_kernel_copy(&c); ++ t_emulator_read(&c); ++ t_unmap_reset(&c); ++ t_coherent_race(mib); ++ t_arm_window(); ++ t_coherent_cost(&c); ++ t_fuse_coherent(&c); ++ census_flush(); ++ printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); ++ return failures != 0; ++} diff --git a/patches/reims-vgpu/0012-REIMS_VGPU_PAGED_BUFFER_CACHE-keeps-page.patch b/patches/reims-vgpu/0012-REIMS_VGPU_PAGED_BUFFER_CACHE-keeps-page.patch new file mode 100644 index 0000000..bf9f47a --- /dev/null +++ b/patches/reims-vgpu/0012-REIMS_VGPU_PAGED_BUFFER_CACHE-keeps-page.patch @@ -0,0 +1,885 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sat, 31 Oct 2026 23:26:32 +0000 +Subject: [PATCH 12/25] REIMS_VGPU_PAGED_BUFFER_CACHE keeps + paged textures' buffers + +Safari's compositor makes a texture per rAF on a dozen recycled page ranges. +Each cost about 148 mmaps, a Vulkan import, a free and munmaps on the exec's +serial path, 53% of the busy task thread. Off by default. A hit needs the +pages the task maps now: every lookup walks the guest's page table. + +Also: mapstat per present, a log line per 600 presents, tdesc and srf events. + +Co-Authored-By: Claude Opus 5.5 +--- + README.md | 1 + + crates/reims-vgpu-core/src/display/frame.rs | 2 + + crates/reims-vgpu-core/src/gpu_task.rs | 1 + + crates/reims-vgpu-core/src/gpu_task/memory.rs | 284 ++++++++++++++++- + .../src/gpu_task/paged_buffers.rs | 300 ++++++++++++++++++ + crates/reims-vgpu-core/src/gpu_task/paging.rs | 3 + + .../reims-vgpu-core/src/gpu_task/service.rs | 18 ++ + .../reims-vgpu-core/src/gpu_task/surfaces.rs | 3 + + crates/reims-vgpu-core/src/resources/host.rs | 11 + + .../reims-vgpu-core/src/resources/texture.rs | 15 +- + 10 files changed, 631 insertions(+), 7 deletions(-) + create mode 100644 crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs + +diff --git a/README.md b/README.md +index 78441900..abfbbda0 100644 +--- a/README.md ++++ b/README.md +@@ -181,6 +181,7 @@ reads them once per process, and an unset or unparsable value is the default. + | `REIMS_VGPU_WINDOW_BORDERLESS=1` | borderless windows that fill their screens | + | `REIMS_VGPU_WINDOW_SCREENS=1,0` | display i opens on the host screen the list's i-th entry names | + | `REIMS_VGPU_WINDOW_REFRESH=120` | the refresh rate of the modes offered to the guest | ++| `REIMS_VGPU_PAGED_BUFFER_CACHE=32` | LOCAL (Atlas): entries per task of the buffers kept over guest pages for paged textures (`gpu_task/paged_buffers.rs`); unset or 0 keeps none | + | `REIMS_VGPU_WINDOW_IMPL=sdl` | macOS: SDL3's window in place of AppKit's | + | `REIMS_VGPU_STRICT=1` | the Vulkan platform: end the process at each difference from Metal's answer it would otherwise log | + | `REIMS_VGPU_EXACT` | the Vulkan platform: the runtime's exact forms in place of its fast ones, by field ([docs/accuracy-ledger.md](docs/accuracy-ledger.md)) | +diff --git a/crates/reims-vgpu-core/src/display/frame.rs b/crates/reims-vgpu-core/src/display/frame.rs +index 551e03d8..8b0aab56 100644 +--- a/crates/reims-vgpu-core/src/display/frame.rs ++++ b/crates/reims-vgpu-core/src/display/frame.rs +@@ -84,6 +84,8 @@ impl DisplayCore

{ + self.wait_for_pending_frames(&mut g); + // presented: frames in flight were waited for + crate::trace::event("presented", &[]); ++ // LOCAL (Atlas): the guest-page mapping counters, per present. ++ crate::gpu_task::paged_buffers::present_counted(); + if self.guest_color_space_dirty() { + // The guest display's colour space (nil without a guest display). The guest display + // pointer is read without its lock held across the call; the cell is a leaf. +diff --git a/crates/reims-vgpu-core/src/gpu_task.rs b/crates/reims-vgpu-core/src/gpu_task.rs +index b4c229e7..d79d51cf 100644 +--- a/crates/reims-vgpu-core/src/gpu_task.rs ++++ b/crates/reims-vgpu-core/src/gpu_task.rs +@@ -59,6 +59,7 @@ mod blit; + mod exec; + mod handshake; + mod memory; ++pub(crate) mod paged_buffers; + mod paging; + mod service; + mod surfaces; +diff --git a/crates/reims-vgpu-core/src/gpu_task/memory.rs b/crates/reims-vgpu-core/src/gpu_task/memory.rs +index 3a4df4dc..26254d2c 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/memory.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/memory.rs +@@ -16,7 +16,9 @@ + //! task (the suite's `service_gap_cmdbuf_outside_ram` and `covgap_mem_reply_in_hole`). + + use std::sync::Arc; ++use std::sync::atomic::Ordering; + ++use super::paged_buffers::{self, Runs, STATS}; + use super::service::GpuTask; + use super::{DeathKind, RaiseKind, exception, raise_carrying, raise_caught, raise_on}; + use crate::device::tasks::MAX_LOCAL_TASK_LENGTH; +@@ -27,6 +29,7 @@ use crate::memory::mapping::Mapping; + use crate::num::{ToU64, ToUsize}; + use crate::platform::Platform; + use crate::platform::device::GpuDevice; ++use crate::platform::os::clock::{Clock, MonoTime}; + use crate::platform::types::ResourceAddressRange; + use crate::raise::{Kept, keep_forever}; + use crate::raise::{OnFault, RaiseResult}; +@@ -91,6 +94,23 @@ impl GpuTask

{ + format_args!("map of 0 bytes at task offset {offset:#x}"), + ); + } ++ let walked = self.mono(); ++ let runs = self.task_page_runs(hold, offset, length)?; ++ paged_buffers::add_since(&STATS.walk_ns, walked, self.mono()); ++ if need_writable && !runs.writable { ++ return Ok(None); ++ } ++ Ok(self.mapping_of_runs(&runs, length)) ++ } ++ ++ /// The monotonic clock, for the mapping counters. ++ pub(super) fn mono(&self) -> MonoTime { ++ self.platform.clock().now_mono() ++ } ++ ++ /// The guest pages behind the task's `offset..offset + length`, one run per contiguous ++ /// stretch, and whether all of them are writable. ++ fn task_page_runs(&self, hold: Hold<'_>, offset: u64, length: u64) -> RaiseResult { + let mut ranges: Vec = Vec::new(); + let mut writable = true; + let mut range = PhysRange::default(); +@@ -124,12 +144,20 @@ impl GpuTask

{ + if range.physical_length != 0 { + ranges.push(range); + } +- if need_writable && !writable { +- return Ok(None); +- } +- Ok(self ++ Ok(Runs { ranges, writable }) ++ } ++ ++ /// A view of `runs`, whose bytes come to `length`. ++ fn mapping_of_runs(&self, runs: &Runs, length: u64) -> Option>> { ++ let made = self.mono(); ++ let mapping = self + .map() +- .and_then(|m| m.new_range_mapping(&ranges, length, u32::from(!writable)))) ++ .and_then(|m| m.new_range_mapping(&runs.ranges, length, u32::from(!runs.writable))); ++ paged_buffers::add_since(&STATS.map_ns, made, self.mono()); ++ if mapping.is_some() { ++ STATS.made.fetch_add(1, Ordering::Relaxed); ++ } ++ mapping + } + + /// The host address of task offset `offset`, through a mapping of the pages +@@ -512,7 +540,9 @@ impl GpuTask

{ + format_args!("Metal made no buffer over task page {page:#x}"), + ); + }; ++ let imported = self.mono(); + let buffer = Self::no_copy_buffer(&device, address, length, mapping)?; ++ paged_buffers::add_since(&STATS.import_ns, imported, self.mono()); + let Some(buffer) = buffer else { + // Metal released the deallocator's reference; the kept one stays + keep_forever(kept, Kept::AddressSpace); +@@ -526,6 +556,40 @@ impl GpuTask

{ + Ok(buffer) + } + ++ /// LOCAL (Atlas): the buffer a paged texture pages through: a kept one over exactly the pages ++ /// the task maps at `page` now, else a new one, kept (`paged_buffers`). Every call walks the ++ /// page table, so a stale entry is never used. ++ pub(super) fn paged_texture_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult { ++ if !self.paged_buffers.enabled() { ++ return self.new_buffer_for_virtual_page(hold, page, length); ++ } ++ let offset = u64::from(page).wrapping_shl(APV_PAGE_SHIFT); ++ Self::check_task_range(offset, length, self.task_length())?; ++ let Some(lease) = self.map().map(|m| Arc::as_ptr(&m).addr()) else { ++ return self.new_buffer_for_virtual_page(hold, page, length); ++ }; ++ let now = self.mono(); ++ drop(self.paged_buffers.sweep(now)); ++ let runs = self.task_page_runs(hold, offset, length)?; ++ paged_buffers::add_since(&STATS.walk_ns, now, self.mono()); ++ let (kept, stale) = self.paged_buffers.lookup(page, length, &runs, &lease, now); ++ drop(stale); ++ if let Some(buffer) = kept { ++ return Ok(buffer); ++ } ++ let buffer = self.new_buffer_for_virtual_page(hold, page, length)?; ++ drop( ++ self.paged_buffers ++ .insert(page, length, runs, lease, &buffer, now), ++ ); ++ Ok(buffer) ++ } ++ + /// A buffer over `address..address + length`, whose deallocator releases `mapping`. + #[allow(unsafe_code)] // architecture 17.3: the call of an `unsafe fn` over host memory + fn no_copy_buffer( +@@ -551,10 +615,13 @@ mod tests { + use super::super::testing::*; + use super::*; + use crate::guest::GuestPtr; ++ use crate::platform::Handle; + use crate::raise::death::DeathKind as Fault; + use crate::resources::host::ResourceHost; ++ use crate::tasks::launcher::ServiceSide; ++ use crate::tasks::wire::ToService; + use crate::testkit::capture::catch_death; +- use crate::testkit::objects::Model; ++ use crate::testkit::objects::{Fk, Model, kind}; + use crate::testkit::script::Step; + + /// The raise a call dies with, with no handler live. +@@ -802,6 +869,211 @@ mod tests { + ); + } + ++ /// LOCAL (Atlas): a task keeping `entries` paged buffers, over four writable pages. ++ fn keeping(entries: usize) -> Tx { ++ let tx = ready(&rw_pages(4)); ++ tx.task.paged_buffers.set_capacity(entries); ++ tx ++ } ++ ++ fn paged(tx: &Tx, page: u32) -> Fk { ++ tx.task ++ .paged_texture_buffer(Hold::NONE, page, PAGE) ++ .unwrap() ++ } ++ ++ fn buffers_made(tx: &Tx) -> usize { ++ tx.p.call_log() ++ .calls_to("newBufferWithBytesNoCopy:length:options:deallocator:") ++ .len() ++ } ++ ++ fn views_freed(tx: &Tx) -> usize { ++ tx.p.call_log().calls_to("mach_vm_deallocate").len() ++ } ++ ++ /// Off, every paged texture gets its own buffer and view, as the plain path does ++ #[test] ++ fn off_keeps_nothing() { ++ let tx = keeping(0); ++ let before = views_freed(&tx); ++ drop(paged(&tx, 1)); ++ drop(paged(&tx, 1)); ++ assert_eq!(buffers_made(&tx), 2); ++ assert_eq!(views_freed(&tx), before + 2); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ } ++ ++ /// The same pages again take the kept buffer: no new view, no new import, nothing freed ++ #[test] ++ fn the_same_pages_take_the_kept_buffer() { ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ let first = paged(&tx, 1); ++ let id = first.id(); ++ drop(first); ++ assert_eq!(paged(&tx, 1).id(), id); ++ assert_eq!(buffers_made(&tx), 1); ++ assert_eq!(views_freed(&tx), before); ++ } ++ ++ /// A guest page moved behind the same task page: the old view is never answered, and goes ++ #[test] ++ fn a_page_the_task_no_longer_maps_is_never_answered() { ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ let old = paged(&tx, 1); ++ let id = old.id(); ++ drop(old); ++ tx.ram.write_u32(page_pa(PT_PAGE) + 4, (DATA_PAGE + 9) | W); ++ let new = paged(&tx, 1); ++ assert_ne!(new.id(), id); ++ assert_eq!(buffers_made(&tx), 2); ++ assert_eq!(views_freed(&tx), before + 1); ++ assert_eq!(tx.task.paged_buffers.len(), 1); ++ } ++ ++ /// A leaf that turned read-only is a different mapping ++ #[test] ++ fn a_leaf_turned_read_only_is_not_the_kept_view() { ++ let tx = keeping(4); ++ let id = paged(&tx, 1).id(); ++ tx.ram.write_u32(page_pa(PT_PAGE) + 4, DATA_PAGE + 1); ++ assert_ne!(paged(&tx, 1).id(), id); ++ } ++ ++ /// A page gone from the page table raises as the plain path does, and keeps nothing new ++ #[test] ++ fn a_hole_is_not_answered_from_the_cache() { ++ let tx = keeping(4); ++ drop(paged(&tx, 1)); ++ tx.ram.write_u32(page_pa(PT_PAGE) + 4, 0); ++ assert!(caught(|hold| tx.task.paged_texture_buffer(hold, 1, PAGE)).is_some()); ++ } ++ ++ /// `UnmapMemory` over the range lets the view go, and one elsewhere does not ++ #[test] ++ fn unmap_lets_the_view_go() { ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ drop(paged(&tx, 1)); ++ let unmap = |offset| { ++ let (_, reply) = sink(); ++ tx.send(ToService::UnmapMemory { ++ offset, ++ length: PAGE, ++ reply, ++ }); ++ }; ++ unmap(2 * PAGE); ++ assert_eq!(views_freed(&tx), before); ++ unmap(PAGE); ++ assert_eq!(views_freed(&tx), before + 1); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ } ++ ++ /// `DiscardResources` and `ReplacePhysical` say nothing of pages: every entry goes ++ #[test] ++ fn discard_and_replace_clear_the_entries() { ++ let tx = keeping(4); ++ drop(paged(&tx, 1)); ++ let (_, reply) = sink(); ++ tx.send(ToService::DiscardResources { ++ resources: Vec::new(), ++ count: 0, ++ reply, ++ }); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ drop(paged(&tx, 1)); ++ let (_, reply) = sink(); ++ tx.send(ToService::ReplacePhysical { physical: 0, reply }); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ } ++ ++ /// An invalidated view a user still holds stays until that user lets it go ++ #[test] ++ fn a_buffer_in_use_outlives_its_entry() { ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ let held = paged(&tx, 1); ++ drop(tx.task.paged_buffers.clear()); ++ assert_eq!(views_freed(&tx), before); ++ drop(held); ++ assert_eq!(views_freed(&tx), before + 1); ++ } ++ ++ /// The task's end lets every view go, before its memory map ++ #[test] ++ fn the_task_end_lets_the_views_go() { ++ // The end frees the task's own mappings too: the control counts those. ++ let control = keeping(4); ++ let before = views_freed(&control); ++ control.task.end(); ++ let own = views_freed(&control) - before; ++ ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ drop(paged(&tx, 1)); ++ drop(paged(&tx, 2)); ++ assert_eq!(views_freed(&tx), before); ++ tx.task.end(); ++ assert_eq!(views_freed(&tx), before + own + 2); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ } ++ ++ /// Another memory map, as a new lease of the guest's RAM would be, is not the kept view ++ #[test] ++ fn another_lease_is_not_the_kept_view() { ++ let tx = keeping(4); ++ let first = paged(&tx, 1); ++ let runs = tx.task.task_page_runs(Hold::NONE, PAGE, PAGE).unwrap(); ++ let lease = tx.task.map().map(|m| Arc::as_ptr(&m).addr()).unwrap(); ++ let now = tx.task.mono(); ++ let (kept, stale) = tx.task.paged_buffers.lookup(1, PAGE, &runs, &lease, now); ++ assert_eq!(kept.map(|b| b.id()), Some(first.id())); ++ drop(stale); ++ let (kept, stale) = tx ++ .task ++ .paged_buffers ++ .lookup(1, PAGE, &runs, &(lease + 1), now); ++ assert!(kept.is_none()); ++ drop(stale); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ } ++ ++ /// Past the capacity the least recently used goes; a hit counts as a use ++ #[test] ++ fn the_capacity_drops_the_least_recently_used() { ++ let tx = keeping(2); ++ let before = views_freed(&tx); ++ drop(paged(&tx, 1)); ++ drop(paged(&tx, 2)); ++ drop(paged(&tx, 1)); ++ drop(paged(&tx, 3)); ++ assert_eq!(views_freed(&tx), before + 1); ++ assert_eq!(buffers_made(&tx), 3); ++ drop(paged(&tx, 1)); ++ assert_eq!(buffers_made(&tx), 3); ++ } ++ ++ /// An entry unused for the idle limit goes at the next sweep, used ones stay ++ #[test] ++ fn idle_entries_go() { ++ let tx = keeping(4); ++ let before = views_freed(&tx); ++ drop(paged(&tx, 1)); ++ let now = tx.task.mono(); ++ drop(tx.task.paged_buffers.sweep(now)); ++ assert_eq!(tx.task.paged_buffers.len(), 1); ++ drop( ++ tx.task ++ .paged_buffers ++ .sweep(now.saturating_add(paged_buffers::MAX_IDLE)), ++ ); ++ assert_eq!(tx.task.paged_buffers.len(), 0); ++ assert_eq!(views_freed(&tx), before + 1); ++ } ++ + #[test] + fn data_from_virtual_offset_copies() { + let tx = ready(&rw_pages(4)); +diff --git a/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +new file mode 100644 +index 00000000..cfd7766b +--- /dev/null ++++ b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +@@ -0,0 +1,300 @@ ++//! LOCAL (Atlas): the buffers over guest pages a paged texture pages through, kept across the ++//! textures that use the same pages. ++//! ++//! macOS's compositor makes and deletes a texture per frame on a dozen recycled ranges of task ++//! pages. Each new texture cost a walk, one `mmap` per run of guest pages (a 2.4 MB texture is ++//! about 148 of them, 2.7 ms on the exec's serial path), a Vulkan import and, at its release, the ++//! free and the `munmap`. `REIMS_VGPU_PAGED_BUFFER_CACHE=` keeps that many buffers per ++//! task; unset or 0 keeps none and the path is the plain one. ++//! ++//! **What a hit proves.** The key is the task page and the length, and every lookup walks the ++//! guest's page table first: a hit needs the same guest-physical runs, the same writability and ++//! the same memory map as the entry. An entry that no longer matches is dropped, never used. So ++//! nothing is ever paged through a view of pages the task does not map at that moment, whatever ++//! the guest did to them meanwhile, told or not. ++//! ++//! **When entries go.** At `UnmapMemory` over their range, at `ReplacePhysical` and ++//! `DiscardResources` (all of them: neither says which pages it means), at a failed match, past ++//! the capacity (least recently used first), [`MAX_IDLE`] after their last use, and at the task's ++//! end and drop, which is also where a device's guest-memory lease ends for its tasks. Dropping ++//! an entry drops one reference: a command buffer or texture still using the buffer keeps it, ++//! and the view goes with the last reference, by the buffer's own deallocator, as ever. ++ ++use std::collections::VecDeque; ++use std::sync::OnceLock; ++use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; ++use std::time::Duration; ++ ++use crate::host::PhysRange; ++use crate::num::ToU64; ++use crate::platform::Handle; ++use crate::platform::os::clock::MonoTime; ++ ++/// The environment variable: entries per task. ++pub(crate) const ENV_VAR: &str = "REIMS_VGPU_PAGED_BUFFER_CACHE"; ++/// The most entries the variable takes. ++const MAX_ENTRIES: usize = 256; ++/// An entry unused for this long goes at the next sweep. ++pub(crate) const MAX_IDLE: Duration = Duration::from_secs(1); ++/// A summary line per this many presents. ++const LOG_EVERY: u64 = 600; ++ ++/// The entries the variable asks for (read once): 0 without it. ++pub(crate) fn entries_by_env() -> usize { ++ static ENTRIES: OnceLock = OnceLock::new(); ++ *ENTRIES.get_or_init(|| { ++ std::env::var(ENV_VAR) ++ .ok() ++ .and_then(|v| v.parse::().ok()) ++ .unwrap_or(0) ++ .min(MAX_ENTRIES) ++ }) ++} ++ ++/// Counters of the guest-page mappings of the whole process, relaxed, cumulative. They count with ++/// the switch off too, for the baseline. ++#[derive(Debug, Default)] ++pub struct MappingStats { ++ /// Views made of task pages for a new buffer. ++ pub made: AtomicU64, ++ /// Buffers a paged texture took from the cache. ++ pub reused: AtomicU64, ++ /// Entries dropped by an invalidation or a failed match (not by capacity or age). ++ pub invalidated: AtomicU64, ++ /// Nanoseconds walking page tables for these buffers. ++ pub walk_ns: AtomicU64, ++ /// Nanoseconds making views: `mmap`, its bookkeeping and locks. ++ pub map_ns: AtomicU64, ++ /// Nanoseconds making no-copy buffers over the views: the Vulkan import. ++ pub import_ns: AtomicU64, ++} ++ ++/// The process's counters. ++pub static STATS: MappingStats = MappingStats { ++ made: AtomicU64::new(0), ++ reused: AtomicU64::new(0), ++ invalidated: AtomicU64::new(0), ++ walk_ns: AtomicU64::new(0), ++ map_ns: AtomicU64::new(0), ++ import_ns: AtomicU64::new(0), ++}; ++ ++impl MappingStats { ++ /// The counters, in the order of the `mapstat` event. ++ fn values(&self) -> [u64; 6] { ++ [ ++ self.made.load(Ordering::Relaxed), ++ self.reused.load(Ordering::Relaxed), ++ self.invalidated.load(Ordering::Relaxed), ++ self.walk_ns.load(Ordering::Relaxed), ++ self.map_ns.load(Ordering::Relaxed), ++ self.import_ns.load(Ordering::Relaxed), ++ ] ++ } ++} ++ ++/// Adds the time since `from` to `counter`. ++pub(crate) fn add_since(counter: &AtomicU64, from: MonoTime, to: MonoTime) { ++ let ns = u64::try_from(to.saturating_duration_since(from).as_nanos()).unwrap_or(u64::MAX); ++ counter.fetch_add(ns, Ordering::Relaxed); ++} ++ ++/// One present: the trace's `mapstat` (made, reused, invalidated, walk, map and import ns, all ++/// cumulative), and with the switch on a log line per [`LOG_EVERY`] presents, of the difference. ++pub fn present_counted() { ++ static PRESENTS: AtomicU64 = AtomicU64::new(0); ++ static LAST: crate::sync::Mutex<[u64; 6]> = crate::sync::Mutex::new([0; 6]); ++ let values = STATS.values(); ++ crate::trace::event("mapstat", &values); ++ let presents = PRESENTS.fetch_add(1, Ordering::Relaxed).wrapping_add(1); ++ if entries_by_env() != 0 && presents.is_multiple_of(LOG_EVERY) { ++ let before = std::mem::replace(&mut *LAST.lock(), values); ++ let [made, reused, gone, walk, map, import] = values; ++ let [made0, reused0, gone0, walk0, map0, import0] = before; ++ let per = |now: u64, then: u64| now.wrapping_sub(then).div_euclid(LOG_EVERY); ++ let us = |now: u64, then: u64| per(now, then).div_euclid(1000); ++ log::info!( ++ "paged buffers, per present: {} views made, {} reused, {} invalidated, walk {} us, \ ++ map {} us, import {} us", ++ per(made, made0), ++ per(reused, reused0), ++ per(gone, gone0), ++ us(walk, walk0), ++ us(map, map0), ++ us(import, import0) ++ ); ++ } ++} ++ ++/// What a walk found behind a task range. ++#[derive(Debug, Clone, Default)] ++pub(crate) struct Runs { ++ /// The guest-physical runs, in task order. ++ pub(crate) ranges: Vec, ++ /// Whether every leaf is writable. ++ pub(crate) writable: bool, ++} ++ ++impl Runs { ++ /// The same pages, in the same order, with the same writability. ++ fn same_as(&self, other: &Runs) -> bool { ++ self.writable == other.writable ++ && self.ranges.len() == other.ranges.len() ++ && self.ranges.iter().zip(&other.ranges).all(|(a, b)| { ++ a.physical_address == b.physical_address && a.physical_length == b.physical_length ++ }) ++ } ++} ++ ++/// One kept buffer. ++struct Entry { ++ page: u32, ++ length: u64, ++ runs: Runs, ++ /// The memory map the view was made through: the guest's RAM under this lease. ++ lease: L, ++ buffer: B, ++ used: MonoTime, ++} ++ ++/// The kept buffers of one task, most recently used last. A leaf lock: every removal hands its ++/// entries back in a [`Gone`], so no buffer is released under it. ++pub(crate) struct PagedBuffers { ++ capacity: AtomicUsize, ++ entries: crate::sync::Mutex>>, ++} ++ ++/// Entries going, to drop outside the lock: a buffer's last release runs its deallocator. ++pub(crate) struct Gone(#[expect(dead_code, reason = "held for its drop")] Vec>); ++ ++impl PagedBuffers { ++ pub(crate) fn new(capacity: usize) -> Self { ++ PagedBuffers { ++ capacity: AtomicUsize::new(capacity), ++ entries: crate::sync::Mutex::new(VecDeque::new()), ++ } ++ } ++ ++ /// Whether the cache is switched on. ++ pub(crate) fn enabled(&self) -> bool { ++ self.capacity() != 0 ++ } ++ ++ fn capacity(&self) -> usize { ++ self.capacity.load(Ordering::Relaxed) ++ } ++ ++ #[cfg(test)] ++ pub(crate) fn set_capacity(&self, capacity: usize) { ++ self.capacity.store(capacity, Ordering::Relaxed); ++ } ++ ++ /// The entry's buffer when it still shows exactly `runs` under `lease`; an entry of the key ++ /// that does not goes. ++ pub(crate) fn lookup( ++ &self, ++ page: u32, ++ length: u64, ++ runs: &Runs, ++ lease: &L, ++ now: MonoTime, ++ ) -> (Option, Gone) { ++ let mut entries = self.entries.lock(); ++ let Some(at) = entries ++ .iter() ++ .position(|e| e.page == page && e.length == length) ++ else { ++ return (None, Gone(Vec::new())); ++ }; ++ let Some(mut entry) = entries.remove(at) else { ++ return (None, Gone(Vec::new())); ++ }; ++ if entry.lease == *lease && entry.runs.same_as(runs) { ++ entry.used = now; ++ let buffer = entry.buffer.clone_ref(); ++ entries.push_back(entry); ++ drop(entries); ++ STATS.reused.fetch_add(1, Ordering::Relaxed); ++ return (Some(buffer), Gone(Vec::new())); ++ } ++ drop(entries); ++ STATS.invalidated.fetch_add(1, Ordering::Relaxed); ++ (None, Gone(vec![entry])) ++ } ++ ++ /// Keeps `buffer` for its key; the oldest goes past the capacity. ++ pub(crate) fn insert( ++ &self, ++ page: u32, ++ length: u64, ++ runs: Runs, ++ lease: L, ++ buffer: &B, ++ now: MonoTime, ++ ) -> Gone { ++ let mut gone = Vec::new(); ++ let mut entries = self.entries.lock(); ++ if let Some(at) = entries ++ .iter() ++ .position(|e| e.page == page && e.length == length) ++ { ++ gone.extend(entries.remove(at)); ++ } ++ while entries.len() >= self.capacity() { ++ gone.extend(entries.pop_front()); ++ } ++ entries.push_back(Entry { ++ page, ++ length, ++ runs, ++ lease, ++ buffer: buffer.clone_ref(), ++ used: now, ++ }); ++ Gone(gone) ++ } ++ ++ /// Entries unused for [`MAX_IDLE`] go. ++ pub(crate) fn sweep(&self, now: MonoTime) -> Gone { ++ let mut entries = self.entries.lock(); ++ let mut gone = Vec::new(); ++ // Oldest first: the order of use. ++ while entries ++ .front() ++ .is_some_and(|e| now.saturating_duration_since(e.used) >= MAX_IDLE) ++ { ++ gone.extend(entries.pop_front()); ++ } ++ Gone(gone) ++ } ++ ++ /// Entries over task bytes `offset..offset + length` go (the ends wrap, as the message's do). ++ pub(crate) fn invalidate_range(&self, offset: u64, length: u64) -> Gone { ++ let end = offset.wrapping_add(length); ++ let mut entries = self.entries.lock(); ++ let (gone, kept): (Vec<_>, Vec<_>) = entries.drain(..).partition(|e| { ++ let first = u64::from(e.page).wrapping_shl(crate::guest::records::APV_PAGE_SHIFT); ++ first < end && offset < first.wrapping_add(e.length) ++ }); ++ entries.extend(kept); ++ drop(entries); ++ STATS ++ .invalidated ++ .fetch_add(gone.len().to_u64(), Ordering::Relaxed); ++ Gone(gone) ++ } ++ ++ /// Every entry goes. ++ pub(crate) fn clear(&self) -> Gone { ++ let gone: Vec<_> = self.entries.lock().drain(..).collect(); ++ STATS ++ .invalidated ++ .fetch_add(gone.len().to_u64(), Ordering::Relaxed); ++ Gone(gone) ++ } ++ ++ #[cfg(test)] ++ pub(crate) fn len(&self) -> usize { ++ self.entries.lock().len() ++ } ++} +diff --git a/crates/reims-vgpu-core/src/gpu_task/paging.rs b/crates/reims-vgpu-core/src/gpu_task/paging.rs +index 65517055..f3ec99b4 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/paging.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/paging.rs +@@ -63,6 +63,9 @@ impl GpuTask

{ + shared_texture_backing_ids: &mut Vec, + ) -> RaiseResult { + let _prepare = self.prepare_lock.lock(); ++ if self.paged_buffers.enabled() { ++ drop(self.paged_buffers.sweep(self.mono())); ++ } + let core = self.core(); + let mut blit_resources: Vec>> = Vec::new(); + let mut compute_resources: Vec>> = Vec::new(); +diff --git a/crates/reims-vgpu-core/src/gpu_task/service.rs b/crates/reims-vgpu-core/src/gpu_task/service.rs +index 4217bc88..b33c112a 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/service.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/service.rs +@@ -11,6 +11,7 @@ use std::sync::{Arc, OnceLock, Weak}; + use parking_lot::{Mutex, MutexGuard}; + + use super::exec::FifoInfo; ++use super::paged_buffers::PagedBuffers; + use super::surfaces::CachedSurface; + use super::{DeathKind, Unsupported, blit, exception, unraised}; + use crate::collections::LookupMap; +@@ -78,6 +79,8 @@ pub struct GpuTask { + pub(super) features: Arc, + /// The IOSurfaces the host lent, by mapping id, with their retain counts (lookup only). + pub(super) surfaces: Mutex>>, ++ /// LOCAL (Atlas): the buffers kept over task pages for paged textures (`paged_buffers`). ++ pub(super) paged_buffers: PagedBuffers, + /// Serializes the execs' and infos' paging-in. + pub(super) prepare_lock: Mutex<()>, + /// Serializes the object lists' mapping. +@@ -150,6 +153,7 @@ impl GpuTask

{ + queue, + features: Arc::new(SharedFeatures::new()), + surfaces: Mutex::new(LookupMap::new()), ++ paged_buffers: PagedBuffers::new(super::paged_buffers::entries_by_env()), + prepare_lock: Mutex::new(()), + object_lists_lock: Mutex::new(()), + fifos: Mutex::new(LookupMap::new()), +@@ -321,6 +325,7 @@ impl GpuTask

{ + /// Replaces the physical page of the task's buffers: 1, or 2 when it raises. + fn replace_physical(&self, physical: u32, reply: Reply) { + self.job(move |this, hold| { ++ drop(this.paged_buffers.clear()); + if let Err(t) = this.core().replace_physical(physical) { + log::warn!( + "task {}: discard of resource {physical} faulted: {}", +@@ -430,6 +435,7 @@ impl GpuTask

{ + /// Discards the resources the records name: 1, or 2 when it raises. + fn discard_resources(&self, resources: Vec, count: u32, reply: Reply) { + self.job(move |this, hold| { ++ drop(this.paged_buffers.clear()); + let discarded = { + let records = records(&resources, count); + this.core().discard_resources(hold, records) +@@ -458,6 +464,7 @@ impl GpuTask

{ + fn unmap_memory(&self, offset: u64, length: u64, reply: Reply) { + self.job(move |this, _| { + let _lock = this.task_lock(Access::Unique); ++ drop(this.paged_buffers.invalidate_range(offset, length)); + let lists = this.lists(); + let end = offset.wrapping_add(length); + let objects = lists.object_list_range(); +@@ -650,6 +657,7 @@ impl ServiceSide

for GpuTask

{ + /// the object lists go with the core's parts; then the surfaces the host lent, the FIFOs' + /// events and the lists the handshake built. + fn end(&self) { ++ drop(self.paged_buffers.clear()); + drop(self.device.lock().take()); + drop(self.command_queue.lock().take()); + if let Some(core) = self.core.get() { +@@ -672,6 +680,7 @@ impl Drop for GpuTask

{ + /// its event and its parts), then the task root's mapping, the page table and the memory + /// map; the rest follows in field order. A task that ended released most of it then. + fn drop(&mut self) { ++ drop(self.paged_buffers.clear()); + drop(self.host.take()); + drop(self.device.get_mut().take()); + drop(self.command_queue.get_mut().take()); +@@ -693,6 +702,15 @@ impl ResourceHost

for GpuTask

{ + GpuTask::new_buffer_for_virtual_page(self, hold, page, length) + } + ++ fn paged_texture_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult { ++ GpuTask::paged_texture_buffer(self, hold, page, length) ++ } ++ + fn prepare_backing( + &self, + _hold: Hold<'_>, +diff --git a/crates/reims-vgpu-core/src/gpu_task/surfaces.rs b/crates/reims-vgpu-core/src/gpu_task/surfaces.rs +index cf57f92a..89c100ca 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/surfaces.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/surfaces.rs +@@ -86,8 +86,11 @@ impl GpuTask

{ + let mut surfaces = self.surfaces.lock(); + if let Some(cached) = surfaces.get_mut(&identifier) { + cached.count = cached.count.wrapping_add(1); ++ // srf: mapping id, 1 for a surface the task already held, 0 for a round trip ++ crate::trace::event("srf", &[identifier, 1]); + return Some(cached.surface.clone_ref()); + } ++ crate::trace::event("srf", &[identifier, 0]); + let slot: Arc>> = ReplySlot::new(); + let fill = Arc::clone(&slot); + // The reply runs on the connection's queue while this one waits holding the cache +diff --git a/crates/reims-vgpu-core/src/resources/host.rs b/crates/reims-vgpu-core/src/resources/host.rs +index 5fa534ae..51801d7c 100644 +--- a/crates/reims-vgpu-core/src/resources/host.rs ++++ b/crates/reims-vgpu-core/src/resources/host.rs +@@ -73,6 +73,17 @@ pub trait ResourceHost: Send + Sync { + length: u64, + ) -> RaiseResult; + ++ /// LOCAL (Atlas): the same for a paged texture's transfer buffer, which a task may answer ++ /// with a buffer it kept over the same pages (`gpu_task::paged_buffers`). ++ fn paged_texture_buffer( ++ &self, ++ hold: Hold<'_>, ++ page: u32, ++ length: u64, ++ ) -> RaiseResult { ++ self.new_buffer_for_virtual_page(hold, page, length) ++ } ++ + /// Prepares an IOSurface backing. `encoder` is `None` when Metal gave no encoder, which + /// nothing checks: the backing's validity writes still happen. + fn prepare_backing( +diff --git a/crates/reims-vgpu-core/src/resources/texture.rs b/crates/reims-vgpu-core/src/resources/texture.rs +index 2a7a5e8c..673ca01e 100644 +--- a/crates/reims-vgpu-core/src/resources/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/texture.rs +@@ -393,8 +393,21 @@ impl PagedTexture

{ + // Trace `tobj`: the guest object whose pages the next `tbuf` imports (4, a texture), its + // page and its length. + crate::trace::event("tobj", &[4, u64::from(self.page), self.length]); ++ if crate::trace::on() { ++ // Trace `tdesc`: that texture's width, height, format, storage mode and usage. ++ crate::trace::event( ++ "tdesc", ++ &[ ++ self.texture.width(), ++ self.texture.height(), ++ self.texture.pixel_format(), ++ self.texture.storage_mode(), ++ self.texture.usage(), ++ ], ++ ); ++ } + // The task raises rather than answer nothing. +- let buffer = task.new_buffer_for_virtual_page(hold, self.page, self.length)?; ++ let buffer = task.paged_texture_buffer(hold, self.page, self.length)?; + // A plain store: a buffer a racing caller stored between the check above and this + // store is overwritten unreleased, and kept for good with its GPU VA and its guest + // pages' mapping. diff --git a/patches/reims-vgpu/0013-Skip-a-new-Private-texture-s-first-page-up-behind-a-.patch b/patches/reims-vgpu/0013-Skip-a-new-Private-texture-s-first-page-up-behind-a-.patch new file mode 100644 index 0000000..05c6e2b --- /dev/null +++ b/patches/reims-vgpu/0013-Skip-a-new-Private-texture-s-first-page-up-behind-a-.patch @@ -0,0 +1,1057 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 00:17:35 +0000 +Subject: [PATCH 13/25] Skip a new Private texture's first page-up, behind a + switch + +Safari's compositor makes about ten 2368x256 Private render targets per +present, and each one builds a view of 148 guest-page runs, imports it +and blits it, on the exec's serial path, only to overwrite a texture +that already reads zero. REIMS_VGPU_PRIVATE_SKIP_PAGE_UP=1|2 skips that +first page-up: no view, no import, no paging buffer. Off by default. + +Mapping reuse found nothing to reuse, because the guest puts other +physical pages behind the recycled ranges. So mapstat gains the stale +runs, the skips, the refusals and the mmaps with their time, and the +trun event gives each walk's run count and hash. + +Co-Authored-By: Claude Opus 5.5 +--- + README.md | 1 + + .../reims-vgpu-core/src/device/lifecycle.rs | 1 + + crates/reims-vgpu-core/src/gpu_task/memory.rs | 9 + + .../src/gpu_task/paged_buffers.rs | 197 ++++++++++++++++-- + crates/reims-vgpu-core/src/gpu_task/paging.rs | 114 +++++++++- + .../reims-vgpu-core/src/gpu_task/service.rs | 14 +- + crates/reims-vgpu-core/src/memory/mapping.rs | 6 + + crates/reims-vgpu-core/src/resources/host.rs | 14 ++ + .../reims-vgpu-core/src/resources/resource.rs | 10 + + .../reims-vgpu-core/src/resources/texture.rs | 51 +++++ + .../src/resources/texture/blit.rs | 186 +++++++++++++++++ + .../reims-vgpu-core/src/resources/validity.rs | 20 +- + .../src/testkit/resource_host.rs | 27 +++ + crates/reims-vgpu-vulkan/src/os/linux/vm.rs | 8 + + 14 files changed, 643 insertions(+), 15 deletions(-) + +diff --git a/README.md b/README.md +index abfbbda0..2f4622de 100644 +--- a/README.md ++++ b/README.md +@@ -182,6 +182,7 @@ reads them once per process, and an unset or unparsable value is the default. + | `REIMS_VGPU_WINDOW_SCREENS=1,0` | display i opens on the host screen the list's i-th entry names | + | `REIMS_VGPU_WINDOW_REFRESH=120` | the refresh rate of the modes offered to the guest | + | `REIMS_VGPU_PAGED_BUFFER_CACHE=32` | LOCAL (Atlas): entries per task of the buffers kept over guest pages for paged textures (`gpu_task/paged_buffers.rs`); unset or 0 keeps none | ++| `REIMS_VGPU_PRIVATE_SKIP_PAGE_UP` | LOCAL (Atlas). A new `Private` texture takes no first page-up: no guest-page view, no import, no paging buffer. `1` unless its host copy was ever marked stale, `2` unless a current copy went stale. Unset or `0` pages as ever. | + | `REIMS_VGPU_WINDOW_IMPL=sdl` | macOS: SDL3's window in place of AppKit's | + | `REIMS_VGPU_STRICT=1` | the Vulkan platform: end the process at each difference from Metal's answer it would otherwise log | + | `REIMS_VGPU_EXACT` | the Vulkan platform: the runtime's exact forms in place of its fast ones, by field ([docs/accuracy-ledger.md](docs/accuracy-ledger.md)) | +diff --git a/crates/reims-vgpu-core/src/device/lifecycle.rs b/crates/reims-vgpu-core/src/device/lifecycle.rs +index 855269b2..bd00794d 100644 +--- a/crates/reims-vgpu-core/src/device/lifecycle.rs ++++ b/crates/reims-vgpu-core/src/device/lifecycle.rs +@@ -333,6 +333,7 @@ impl DeviceCore

{ + mut error: Option<&mut Option>, + ) -> bool { + log::info!("device resuming"); ++ crate::gpu_task::paged_buffers::note_resumed(); + self.run_state.store(RunState::LeavingSuspendEarly); + let Some(bytes) = state else { + set_error(error, RESUME_ERROR_INVALID_CONTENT.cast_unsigned()); +diff --git a/crates/reims-vgpu-core/src/gpu_task/memory.rs b/crates/reims-vgpu-core/src/gpu_task/memory.rs +index 26254d2c..f6fbc6a0 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/memory.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/memory.rs +@@ -144,6 +144,15 @@ impl GpuTask

{ + if range.physical_length != 0 { + ranges.push(range); + } ++ if crate::trace::on() { ++ // Trace `trun`: task offset, length, runs, and a hash of them, equal for two walks ++ // that found the same physical pages. ++ let hash = ranges.iter().fold(0xcbf2_9ce4_8422_2325_u64, |hash, r| { ++ (hash ^ r.physical_address ^ r.physical_length.rotate_left(32)) ++ .wrapping_mul(0x0000_0100_0000_01b3) ++ }); ++ crate::trace::event("trun", &[offset, length, ranges.len().to_u64(), hash]); ++ } + Ok(Runs { ranges, writable }) + } + +diff --git a/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +index cfd7766b..79384ec9 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/paged_buffers.rs +@@ -19,19 +19,33 @@ + //! end and drop, which is also where a device's guest-memory lease ends for its tasks. Dropping + //! an entry drops one reference: a command buffer or texture still using the buffer keeps it, + //! and the view goes with the last reference, by the buffer's own deallocator, as ever. ++//! ++//! **Measured on macOS 26.7.1 and Safari: no reuse.** The guest backs a recycled task range with ++//! other physical pages each time, so every lookup is a stale match. [`SKIP_ENV_VAR`] and the ++//! counters beyond the first six are what that run led to. + +-use std::collections::VecDeque; ++use std::collections::{HashSet, VecDeque}; + use std::sync::OnceLock; +-use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; ++use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, AtomicUsize, Ordering}; + use std::time::Duration; + + use crate::host::PhysRange; ++use crate::memory::mapping::{VIEW_MMAP_NS, VIEW_MMAPS}; + use crate::num::ToU64; + use crate::platform::Handle; + use crate::platform::os::clock::MonoTime; + + /// The environment variable: entries per task. + pub(crate) const ENV_VAR: &str = "REIMS_VGPU_PAGED_BUFFER_CACHE"; ++/// The environment variable of a new `Private` texture's page-up: unset or 0 pages it as ever; 1 ++/// skips it unless the host copy was ever marked stale; 2 skips it unless a current host copy ++/// went stale, so a guest's mark on a texture never used is ignored ++/// (`PagedTexture::skip_first_page_up`). ++pub(crate) const SKIP_ENV_VAR: &str = "REIMS_VGPU_PRIVATE_SKIP_PAGE_UP"; ++/// The pages noted as paged down, past which a task stops skipping. ++pub(crate) const MAX_PAGED_DOWN: usize = 1 << 16; ++/// The counters of the `mapstat` event. ++const COUNTERS: usize = 11; + /// The most entries the variable takes. + const MAX_ENTRIES: usize = 256; + /// An entry unused for this long goes at the next sweep. +@@ -51,6 +65,32 @@ pub(crate) fn entries_by_env() -> usize { + }) + } + ++/// The mode [`SKIP_ENV_VAR`] asks for (read once): 0 without it, or for any other value. ++pub(crate) fn skip_mode_by_env() -> u8 { ++ static MODE: OnceLock = OnceLock::new(); ++ *MODE.get_or_init(|| match std::env::var(SKIP_ENV_VAR).as_deref() { ++ Ok("1") => 1, ++ Ok("2") => 2, ++ _ => 0, ++ }) ++} ++ ++/// Set for good once the process resumes a device: resumed tasks make their `Private` textures ++/// anew over guest pages that hold the contents. ++static RESUMED: AtomicBool = AtomicBool::new(false); ++ ++/// A device of the process is resuming: no first page-up is skipped from now on. ++pub(crate) fn note_resumed() { ++ RESUMED.store(true, Ordering::Relaxed); ++} ++ ++/// Whether a device of the process resumed. Never in tests: their process shares this among ++/// devices that resume on their own; [`FreshPageUps::mode_at`] takes the answer, and its rule is ++/// tested there. ++pub(crate) fn resumed() -> bool { ++ !cfg!(test) && RESUMED.load(Ordering::Relaxed) ++} ++ + /// Counters of the guest-page mappings of the whole process, relaxed, cumulative. They count with + /// the switch off too, for the baseline. + #[derive(Debug, Default)] +@@ -67,6 +107,12 @@ pub struct MappingStats { + pub map_ns: AtomicU64, + /// Nanoseconds making no-copy buffers over the views: the Vulkan import. + pub import_ns: AtomicU64, ++ /// Lookups that found other physical runs behind an entry: part of `invalidated`. ++ pub stale_runs: AtomicU64, ++ /// First page-ups of `Private` textures skipped. ++ pub page_ups_skipped: AtomicU64, ++ /// First page-ups of `Private` textures kept because the host copy was marked stale. ++ pub fresh_told_invalid: AtomicU64, + } + + /// The process's counters. +@@ -77,11 +123,14 @@ pub static STATS: MappingStats = MappingStats { + walk_ns: AtomicU64::new(0), + map_ns: AtomicU64::new(0), + import_ns: AtomicU64::new(0), ++ stale_runs: AtomicU64::new(0), ++ page_ups_skipped: AtomicU64::new(0), ++ fresh_told_invalid: AtomicU64::new(0), + }; + + impl MappingStats { + /// The counters, in the order of the `mapstat` event. +- fn values(&self) -> [u64; 6] { ++ fn values(&self) -> [u64; COUNTERS] { + [ + self.made.load(Ordering::Relaxed), + self.reused.load(Ordering::Relaxed), +@@ -89,6 +138,11 @@ impl MappingStats { + self.walk_ns.load(Ordering::Relaxed), + self.map_ns.load(Ordering::Relaxed), + self.import_ns.load(Ordering::Relaxed), ++ self.stale_runs.load(Ordering::Relaxed), ++ self.page_ups_skipped.load(Ordering::Relaxed), ++ self.fresh_told_invalid.load(Ordering::Relaxed), ++ VIEW_MMAPS.load(Ordering::Relaxed), ++ VIEW_MMAP_NS.load(Ordering::Relaxed), + ] + } + } +@@ -99,33 +153,114 @@ pub(crate) fn add_since(counter: &AtomicU64, from: MonoTime, to: MonoTime) { + counter.fetch_add(ns, Ordering::Relaxed); + } + +-/// One present: the trace's `mapstat` (made, reused, invalidated, walk, map and import ns, all +-/// cumulative), and with the switch on a log line per [`LOG_EVERY`] presents, of the difference. ++/// One present: the trace's `mapstat`, all cumulative: made, reused, invalidated, walk, map and ++/// import ns, stale runs, page-ups skipped, page-ups kept as told, the views' `mmap`s and their ++/// ns. With a switch on, a log line per [`LOG_EVERY`] presents, of the difference. + pub fn present_counted() { + static PRESENTS: AtomicU64 = AtomicU64::new(0); +- static LAST: crate::sync::Mutex<[u64; 6]> = crate::sync::Mutex::new([0; 6]); ++ static LAST: crate::sync::Mutex<[u64; COUNTERS]> = crate::sync::Mutex::new([0; COUNTERS]); + let values = STATS.values(); + crate::trace::event("mapstat", &values); + let presents = PRESENTS.fetch_add(1, Ordering::Relaxed).wrapping_add(1); +- if entries_by_env() != 0 && presents.is_multiple_of(LOG_EVERY) { ++ if (entries_by_env() != 0 || skip_mode_by_env() != 0) && presents.is_multiple_of(LOG_EVERY) { + let before = std::mem::replace(&mut *LAST.lock(), values); +- let [made, reused, gone, walk, map, import] = values; +- let [made0, reused0, gone0, walk0, map0, import0] = before; ++ let [ ++ made, ++ reused, ++ gone, ++ walk, ++ map, ++ import, ++ stale, ++ skipped, ++ told, ++ mmaps, ++ mmap_ns, ++ ] = values; ++ let [ ++ made0, ++ reused0, ++ gone0, ++ walk0, ++ map0, ++ import0, ++ stale0, ++ skipped0, ++ told0, ++ mmaps0, ++ mmap_ns0, ++ ] = before; + let per = |now: u64, then: u64| now.wrapping_sub(then).div_euclid(LOG_EVERY); + let us = |now: u64, then: u64| per(now, then).div_euclid(1000); + log::info!( +- "paged buffers, per present: {} views made, {} reused, {} invalidated, walk {} us, \ +- map {} us, import {} us", ++ "paged buffers, per present: {} views made, {} reused, {} invalidated ({} stale runs), \ ++ walk {} us, map {} us ({} mmaps, {} us in them), import {} us, {} page-ups skipped, \ ++ {} kept as told", + per(made, made0), + per(reused, reused0), + per(gone, gone0), ++ per(stale, stale0), + us(walk, walk0), + us(map, map0), +- us(import, import0) ++ per(mmaps, mmaps0), ++ us(mmap_ns, mmap_ns0), ++ us(import, import0), ++ per(skipped, skipped0), ++ per(told, told0), + ); + } + } + ++/// A task's say over new `Private` textures' page-ups ([`SKIP_ENV_VAR`]). ++pub(crate) struct FreshPageUps { ++ mode: AtomicU8, ++ /// The task pages a `Private` page-down reached, where a later object is paged up as ever; ++ /// `None` past [`MAX_PAGED_DOWN`], where the task skips no more. ++ paged_down: crate::sync::Mutex>>, ++} ++ ++impl FreshPageUps { ++ /// With mode `mode`. ++ pub(crate) fn new(mode: u8) -> Self { ++ FreshPageUps { ++ mode: AtomicU8::new(mode), ++ paged_down: crate::sync::Mutex::new(Some(HashSet::new())), ++ } ++ } ++ ++ /// Turns the mode to `mode`, as the variable would have. ++ #[cfg(test)] ++ pub(crate) fn set_mode(&self, mode: u8) { ++ self.mode.store(mode, Ordering::Relaxed); ++ } ++ ++ /// The mode for a texture at `page`, 0 for none, `resumed` being [`resumed`]. ++ pub(crate) fn mode_at(&self, page: u32, resumed: bool) -> u8 { ++ let mode = self.mode.load(Ordering::Relaxed); ++ if mode == 0 || resumed { ++ return 0; ++ } ++ match self.paged_down.lock().as_ref() { ++ Some(pages) if !pages.contains(&page) => mode, ++ _ => 0, ++ } ++ } ++ ++ /// A `Private` page-down reached `page`. ++ pub(crate) fn note_paged_down(&self, page: u32) { ++ if self.mode.load(Ordering::Relaxed) == 0 { ++ return; ++ } ++ let mut guard = self.paged_down.lock(); ++ if let Some(pages) = guard.as_mut() { ++ pages.insert(page); ++ if pages.len() > MAX_PAGED_DOWN { ++ *guard = None; ++ } ++ } ++ } ++} ++ + /// What a walk found behind a task range. + #[derive(Debug, Clone, Default)] + pub(crate) struct Runs { +@@ -219,6 +354,7 @@ impl PagedBuffers { + } + drop(entries); + STATS.invalidated.fetch_add(1, Ordering::Relaxed); ++ STATS.stale_runs.fetch_add(1, Ordering::Relaxed); + (None, Gone(vec![entry])) + } + +@@ -298,3 +434,40 @@ impl PagedBuffers { + self.entries.lock().len() + } + } ++ ++#[cfg(test)] ++mod tests { ++ use super::*; ++ ++ /// Off, nothing is ever skipped, and nothing is kept ++ #[test] ++ fn mode_zero_says_no() { ++ let fresh = FreshPageUps::new(0); ++ fresh.note_paged_down(3); ++ assert_eq!(fresh.mode_at(4, false), 0); ++ assert!(fresh.paged_down.lock().as_ref().unwrap().is_empty()); ++ } ++ ++ /// On, it says its mode, but not for a page a page-down reached, or after a resume ++ #[test] ++ fn a_paged_down_page_and_a_resume_refuse() { ++ let fresh = FreshPageUps::new(2); ++ assert_eq!(fresh.mode_at(3, false), 2); ++ fresh.note_paged_down(3); ++ assert_eq!(fresh.mode_at(3, false), 0); ++ assert_eq!(fresh.mode_at(4, false), 2); ++ assert_eq!(fresh.mode_at(4, true), 0); ++ } ++ ++ /// Past the bound it forgets the pages, and so skips nothing more for good ++ #[test] ++ fn past_the_bound_it_refuses_for_good() { ++ let fresh = FreshPageUps::new(1); ++ for page in 0..=u32::try_from(MAX_PAGED_DOWN).unwrap() { ++ fresh.note_paged_down(page); ++ } ++ assert_eq!(fresh.mode_at(u32::MAX, false), 0); ++ fresh.note_paged_down(u32::MAX); ++ assert_eq!(fresh.mode_at(u32::MAX, false), 0); ++ } ++} +diff --git a/crates/reims-vgpu-core/src/gpu_task/paging.rs b/crates/reims-vgpu-core/src/gpu_task/paging.rs +index f3ec99b4..a775ede9 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/paging.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/paging.rs +@@ -80,7 +80,9 @@ impl GpuTask

{ + r.set_is_host_valid(false); + } + if let Some(r) = resource.as_ref().filter(|r| r.needs_prepare()) { +- if r.needs_cpu_paging() { ++ if r.skip_first_page_up() { ++ // LOCAL (Atlas): no pages, no blit, no paging buffer. ++ } else if r.needs_cpu_paging() { + cpu_resources.push(Arc::clone(r)); + } else if r.needs_compute_paging() { + compute_resources.push(Arc::clone(r)); +@@ -295,8 +297,16 @@ mod tests { + use super::super::testing::*; + use super::*; + use crate::platform::device::GpuDevice; ++ use crate::resources::host::ResourceHost; ++ use crate::resources::manager::create::Resources; ++ use crate::resources::resource::with_platform; ++ use crate::resources::texture::create_texture_resource; ++ use crate::resources::texture::test_support::{desc, header, mip, paging, rec1}; ++ use crate::testkit::FakePlatform; + use crate::testkit::capture::catch_death; ++ use crate::testkit::impls::new_texture; + use crate::testkit::script::Step; ++ use zerocopy::IntoBytes; + + /// Nothing to page: no buffer, the paging counter unmoved, nothing to wait for + #[test] +@@ -388,4 +398,106 @@ mod tests { + assert!(pos("encodeWaitForEvent:value:") < pos("addCompletedHandler:")); + assert!(pos("addCompletedHandler:") < pos("commit")); + } ++ ++ /// LOCAL (Atlas): a `Private` texture of this task, registered at reference 1. ++ fn fresh_private(tx: &Tx) -> Arc> { ++ let t = new_texture(tx.p.world(), desc(70, 16, 8, 1, 3), None, 0, None, false); ++ t.state().device = Some(tx.p.device().erase()); ++ t.set_storage_mode(2); ++ let task = Arc::clone(&tx.task) as Arc>; ++ let dimension = rec1( ++ header(0x300, 4, 3), ++ vec![ ++ mip(0, 0x200, 64, 16, 8, 1), ++ mip(0x200, 0x80, 32, 8, 4, 1), ++ mip(0x280, 0x20, 16, 4, 2, 1), ++ ], ++ ); ++ let r = with_platform(&tx.p, || { ++ create_texture_resource( ++ Arc::downgrade(&task), ++ 2, ++ &paging(1, 0x1000), ++ &dimension, ++ 0x84, ++ t, ++ ) ++ }) ++ .unwrap() ++ .unwrap(); ++ assert!( ++ tx.task ++ .parts() ++ .1 ++ .register_object::(1, Arc::clone(&r)) ++ ); ++ r ++ } ++ ++ /// The calls of one prepare of reference 1 with `flags`. ++ fn prepared(tx: &Tx, flags: u32) -> Vec<&'static str> { ++ let record = [CmdExecIndirectResource3 { ++ reference: 1, ++ flags, ++ }]; ++ let mark = tx.p.call_log().mark(); ++ let mut ids = Vec::new(); ++ tx.task ++ .prepare_resources( ++ Hold::NONE, ++ Records::new(record.as_bytes(), 1).unwrap(), ++ None, ++ 0, ++ &mut ids, ++ ) ++ .unwrap(); ++ tx.p.call_log().sels_since(mark) ++ } ++ ++ fn pages(sels: &[&str]) -> bool { ++ sels.contains(&"commit") ++ } ++ ++ /// LOCAL (Atlas): on, a new `Private` texture adds no paging buffer to the exec's path ++ #[test] ++ fn a_skipped_page_up_makes_no_paging_buffer() { ++ let tx = ready(&rw_pages(4)); ++ tx.task.fresh_page_ups.set_mode(1); ++ let r = fresh_private(&tx); ++ let sels = prepared(&tx, 0); ++ assert!(!pages(&sels), "{sels:?}"); ++ assert!(sels.is_empty(), "{sels:?}"); ++ assert!(r.state.host_valid()); ++ } ++ ++ /// LOCAL (Atlas): off, the same texture is paged as ever ++ #[test] ++ fn off_pages_a_new_private_texture() { ++ let tx = ready(&rw_pages(4)); ++ let r = fresh_private(&tx); ++ assert!(pages(&prepared(&tx, 0))); ++ assert!(r.state.host_valid()); ++ } ++ ++ /// LOCAL (Atlas): the guest's `invalidateHost` on a new texture pages it in mode 1, and is ++ /// ignored in mode 2 ++ #[test] ++ fn the_guests_mark_decides_by_mode() { ++ for (mode, paged) in [(1, true), (2, false)] { ++ let tx = ready(&rw_pages(4)); ++ tx.task.fresh_page_ups.set_mode(mode); ++ fresh_private(&tx); ++ assert_eq!(pages(&prepared(&tx, 1)), paged, "mode {mode}"); ++ } ++ } ++ ++ /// LOCAL (Atlas): a page a `Private` page-down reached is paged up for the next texture there ++ #[test] ++ fn a_paged_down_page_pages_up() { ++ let tx = ready(&rw_pages(4)); ++ tx.task.fresh_page_ups.set_mode(2); ++ tx.task.note_paged_down(1); ++ fresh_private(&tx); ++ assert!(pages(&prepared(&tx, 0))); ++ } + } +diff --git a/crates/reims-vgpu-core/src/gpu_task/service.rs b/crates/reims-vgpu-core/src/gpu_task/service.rs +index b33c112a..861d9b35 100644 +--- a/crates/reims-vgpu-core/src/gpu_task/service.rs ++++ b/crates/reims-vgpu-core/src/gpu_task/service.rs +@@ -11,7 +11,7 @@ use std::sync::{Arc, OnceLock, Weak}; + use parking_lot::{Mutex, MutexGuard}; + + use super::exec::FifoInfo; +-use super::paged_buffers::PagedBuffers; ++use super::paged_buffers::{FreshPageUps, PagedBuffers}; + use super::surfaces::CachedSurface; + use super::{DeathKind, Unsupported, blit, exception, unraised}; + use crate::collections::LookupMap; +@@ -81,6 +81,8 @@ pub struct GpuTask { + pub(super) surfaces: Mutex>>, + /// LOCAL (Atlas): the buffers kept over task pages for paged textures (`paged_buffers`). + pub(super) paged_buffers: PagedBuffers, ++ /// LOCAL (Atlas): new `Private` textures' page-ups. ++ pub(super) fresh_page_ups: FreshPageUps, + /// Serializes the execs' and infos' paging-in. + pub(super) prepare_lock: Mutex<()>, + /// Serializes the object lists' mapping. +@@ -154,6 +156,7 @@ impl GpuTask

{ + features: Arc::new(SharedFeatures::new()), + surfaces: Mutex::new(LookupMap::new()), + paged_buffers: PagedBuffers::new(super::paged_buffers::entries_by_env()), ++ fresh_page_ups: FreshPageUps::new(super::paged_buffers::skip_mode_by_env()), + prepare_lock: Mutex::new(()), + object_lists_lock: Mutex::new(()), + fifos: Mutex::new(LookupMap::new()), +@@ -711,6 +714,15 @@ impl ResourceHost

for GpuTask

{ + GpuTask::paged_texture_buffer(self, hold, page, length) + } + ++ fn fresh_page_up_mode(&self, page: u32) -> u8 { ++ self.fresh_page_ups ++ .mode_at(page, super::paged_buffers::resumed()) ++ } ++ ++ fn note_paged_down(&self, page: u32) { ++ self.fresh_page_ups.note_paged_down(page); ++ } ++ + fn prepare_backing( + &self, + _hold: Hold<'_>, +diff --git a/crates/reims-vgpu-core/src/memory/mapping.rs b/crates/reims-vgpu-core/src/memory/mapping.rs +index 043d2641..d7cff8c6 100644 +--- a/crates/reims-vgpu-core/src/memory/mapping.rs ++++ b/crates/reims-vgpu-core/src/memory/mapping.rs +@@ -1,12 +1,18 @@ + //! A host range over guest memory, and what its last release does. + + use std::sync::Arc; ++use std::sync::atomic::AtomicU64; + + use crate::host::{HostPtr, HostTask}; + use crate::memory::DeathKind; + use crate::platform::Platform; + use crate::seams::links::MappingDeviceLink; + ++/// LOCAL (Atlas): the `mmap` calls of views of guest pages, for the `mapstat` counters. ++pub static VIEW_MMAPS: AtomicU64 = AtomicU64::new(0); ++/// LOCAL (Atlas): the nanoseconds in those calls. ++pub static VIEW_MMAP_NS: AtomicU64 = AtomicU64::new(0); ++ + /// What a mapping's last release does. + pub enum Teardown { + /// Nothing: the range belongs to someone else. +diff --git a/crates/reims-vgpu-core/src/resources/host.rs b/crates/reims-vgpu-core/src/resources/host.rs +index 51801d7c..d65a4c52 100644 +--- a/crates/reims-vgpu-core/src/resources/host.rs ++++ b/crates/reims-vgpu-core/src/resources/host.rs +@@ -84,6 +84,20 @@ pub trait ResourceHost: Send + Sync { + self.new_buffer_for_virtual_page(hold, page, length) + } + ++ /// LOCAL (Atlas): whether a Private paged texture at task page `page` may skip its first ++ /// page-up (`PagedTexture::skip_first_page_up`): 0 no, else the mode. Only a task of a guest ++ /// process says yes, never the kernel's. ++ fn fresh_page_up_mode(&self, page: u32) -> u8 { ++ let _ = page; ++ 0 ++ } ++ ++ /// LOCAL (Atlas): a Private paged texture at task page `page` was paged down: the guest's ++ /// pages there now hold what a later object over them may need. ++ fn note_paged_down(&self, page: u32) { ++ let _ = page; ++ } ++ + /// Prepares an IOSurface backing. `encoder` is `None` when Metal gave no encoder, which + /// nothing checks: the backing's validity writes still happen. + fn prepare_backing( +diff --git a/crates/reims-vgpu-core/src/resources/resource.rs b/crates/reims-vgpu-core/src/resources/resource.rs +index eb30ee69..7f1f1385 100644 +--- a/crates/reims-vgpu-core/src/resources/resource.rs ++++ b/crates/reims-vgpu-core/src/resources/resource.rs +@@ -171,6 +171,12 @@ pub trait ResourceClass: Send + Sync { + let _ = this; + false + } ++ /// LOCAL (Atlas): whether this first prepare needs no paging at all, the resource then ++ /// host-valid (`PagedTexture::skip_first_page_up`). Only paged textures may. ++ fn skip_first_page_up(&self, this: &Resource

) -> bool { ++ let _ = this; ++ false ++ } + /// Guest to host through a blit encoder. `encoder` is `None` when Metal gave no encoder, + /// which nothing checks: the sends are skipped and the flag writes still happen. + fn prepare_in_encoder( +@@ -355,6 +361,10 @@ impl Resource

{ + pub fn needs_prepare(&self) -> bool { + self.class().needs_prepare(self) + } ++ /// LOCAL (Atlas): whether this first prepare needs no paging at all. ++ pub fn skip_first_page_up(&self) -> bool { ++ self.class().skip_first_page_up(self) ++ } + /// Whether it pages on the CPU. + pub fn needs_cpu_paging(&self) -> bool { + self.class().needs_cpu_paging(self) +diff --git a/crates/reims-vgpu-core/src/resources/texture.rs b/crates/reims-vgpu-core/src/resources/texture.rs +index 673ca01e..b5b88bb0 100644 +--- a/crates/reims-vgpu-core/src/resources/texture.rs ++++ b/crates/reims-vgpu-core/src/resources/texture.rs +@@ -65,14 +65,17 @@ pub mod views; + + use std::fmt; + use std::sync::Arc; ++use std::sync::atomic::Ordering; + + use zerocopy::{FromBytes, IntoBytes}; + ++use crate::gpu_task::paged_buffers::STATS; + use crate::num::ToUsize; + use crate::platform::resource::{GpuResource, Texture}; + use crate::platform::{Handle, Platform}; + use crate::raise::{Kept, RaiseResult, keep_forever}; + use crate::resources::resource::AnyResource; ++use crate::resources::resource::STORAGE_MODE_PRIVATE; + use crate::resources::resource::{ + PURGEABLE_EMPTY, PURGEABLE_NON_VOLATILE, PURGEABLE_VOLATILE, Resource, ResourceClass, + ResourceKind, ResourceState, TaskRef, +@@ -425,6 +428,43 @@ impl PagedTexture

{ + Ok(()) + } + ++ /// LOCAL (Atlas): a new `Private` texture takes no page-up: its first prepare binds no guest ++ /// pages and copies nothing, and the texture is host-valid. It answers whether it did. ++ /// ++ /// No guest CPU writes `Private` bytes, so its pages hold data only where a page-down put it, ++ /// or where the guest says so. A new host texture reads zero (the runtime's `FirstMove`, every ++ /// storage mode but memoryless), so nothing of an earlier texture or another process shows. ++ /// Refused, and paged as ever, for: another storage mode; a protected texture; one with pages ++ /// bound, volatile or already host-valid; the guest kernel's tasks; any process that resumed ++ /// from a suspend; a page a `Private` page-down reached in this task; and, in mode 1, a ++ /// texture whose host copy was ever marked stale, or, in mode 2, only one whose current host ++ /// copy went stale. Pages bind as ever at a later page-down or page-up. ++ pub(crate) fn skip_first_page_up(&self, state: &ResourceState) -> bool { ++ if state.host_valid() ++ || state.is_volatile() ++ || self.texture.storage_mode() != STORAGE_MODE_PRIVATE ++ || self.texture.protection_options() != 0 ++ || self.transfer_buffer.lock().is_some() ++ { ++ return false; ++ } ++ let Some(task) = self.task.upgrade() else { ++ return false; ++ }; ++ let refused = match task.fresh_page_up_mode(self.page) { ++ 1 => state.host_told_invalid(), ++ 2 => state.host_lost_valid(), ++ _ => return false, ++ }; ++ if refused { ++ STATS.fresh_told_invalid.fetch_add(1, Ordering::Relaxed); ++ return false; ++ } ++ state.set_host_valid(true); ++ STATS.page_ups_skipped.fetch_add(1, Ordering::Relaxed); ++ true ++ } ++ + /// Guest to host: devolatilize, then page up unless host-valid. A protected texture is never + /// paged, yet marked host-valid. `ensure` binds the pages, `page` pages them. + pub(crate) fn prepare( +@@ -463,6 +503,11 @@ impl PagedTexture

{ + log::warn!("synchronizing a texture whose host copy is not valid"); + } + if self.texture.protection_options() == 0 { ++ if self.texture.storage_mode() == STORAGE_MODE_PRIVATE ++ && let Some(task) = self.task.upgrade() ++ { ++ task.note_paged_down(self.page); ++ } + ensure()?; + page()?; + } +@@ -486,6 +531,12 @@ impl ResourceClass

for TextureResource

{ + matches!(self.layout, TextureLayout::DualPlaneCompute(_)) + } + ++ fn skip_first_page_up(&self, this: &Resource

) -> bool { ++ // One plane, its levels all paged: generated mipmaps start from a paged level 0. ++ matches!(&self.layout, TextureLayout::SinglePlane(l) if !l.generate_mipmaps) ++ && self.paged.skip_first_page_up(&this.state) ++ } ++ + fn prepare_in_encoder( + &self, + hold: Hold<'_>, +diff --git a/crates/reims-vgpu-core/src/resources/texture/blit.rs b/crates/reims-vgpu-core/src/resources/texture/blit.rs +index dc3c8ad9..e4f5b2d8 100644 +--- a/crates/reims-vgpu-core/src/resources/texture/blit.rs ++++ b/crates/reims-vgpu-core/src/resources/texture/blit.rs +@@ -663,6 +663,10 @@ mod tests { + } + } + ++ fn parts_of(r: &Resource

) -> &TextureResource

{ ++ parts(r).0 ++ } ++ + fn made(e: &Env, ty: u32, levels: u64) -> Arc> { + let t = e.tex(desc(70, 16, 8, 1, levels), false); + e.run(|| create_texture_resource(e.task(), ty, &paging(7, 0x300), &good16x8(), 0x84, t)) +@@ -670,6 +674,188 @@ mod tests { + .unwrap() + } + ++ /// LOCAL (Atlas): a new `Private` texture's first page-up (`skip_first_page_up`). ++ mod fresh_private { ++ use super::*; ++ use crate::gpu_task::paged_buffers::STATS; ++ use std::sync::atomic::Ordering; ++ ++ fn private(e: &Env, ty: u32, levels: u64, mode: u8) -> Arc> { ++ let t = e.tex(desc(70, 16, 8, 1, levels), false); ++ t.set_storage_mode(2); ++ e.rec.set_fresh_page_up_mode(mode); ++ e.run(|| { ++ create_texture_resource(e.task(), ty, &paging(7, 0x300), &good16x8(), 0x84, t) ++ }) ++ .unwrap() ++ .unwrap() ++ } ++ ++ /// Skipped: no guest pages bound, no copy, host-valid, the guest's copy untouched ++ #[test] ++ fn binds_and_copies_nothing() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 1); ++ let m = e.p.call_log().mark(); ++ assert!(r.skip_first_page_up()); ++ assert!(e.sels(m).iter().all(|s| !s.contains("buffer") && *s != UP)); ++ assert!(r.state.host_valid()); ++ assert!(r.state.guest_valid()); ++ assert!(!r.needs_prepare()); ++ assert!(parts_of(&r).paged.transfer_buffer.lock().is_none()); ++ } ++ ++ /// The switch off, the kernel's tasks and a task that refuses: paged as ever ++ #[test] ++ fn mode_zero_pages_as_ever() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 0); ++ assert!(!r.skip_first_page_up()); ++ assert!(!r.state.host_valid()); ++ let enc = e.blit(); ++ let m = e.p.call_log().mark(); ++ r.prepare_in_encoder(Hold::NONE, Some(&enc)).unwrap(); ++ assert_eq!(e.sels(m), vec!["new_buffer_for_virtual_page", UP, UP, UP]); ++ } ++ ++ /// Shared storage holds what the guest's CPU wrote: paged ++ #[test] ++ fn other_storage_pages() { ++ let e = Env::new(); ++ e.rec.set_fresh_page_up_mode(2); ++ let r = made(&e, 2, 3); ++ assert!(!r.skip_first_page_up()); ++ assert!(!r.state.host_valid()); ++ } ++ ++ /// Generated levels start from a paged level 0: paged ++ #[test] ++ fn generated_mipmaps_page() { ++ let e = Env::new(); ++ let r = private(&e, 3, 3, 2); ++ assert!(!r.skip_first_page_up()); ++ assert!(!r.state.host_valid()); ++ } ++ ++ /// A protected texture is never paged anyway; it keeps its own path ++ #[test] ++ fn protected_keeps_its_path() { ++ let e = Env::new(); ++ let t = e.tex(desc(70, 16, 8, 1, 3), false); ++ t.set_storage_mode(2); ++ t.set_protection_options(2); ++ e.rec.set_fresh_page_up_mode(1); ++ let r = e ++ .run(|| { ++ create_texture_resource( ++ e.task(), ++ 2, ++ &paging(7, 0x300), ++ &good16x8(), ++ 0x84, ++ t, ++ ) ++ }) ++ .unwrap() ++ .unwrap(); ++ assert!(!r.skip_first_page_up()); ++ } ++ ++ /// Mode 1 believes any stale mark, even on a texture never used ++ #[test] ++ fn mode_one_refuses_a_marked_texture() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 1); ++ r.set_is_host_valid(false); ++ let before = STATS.fresh_told_invalid.load(Ordering::Relaxed); ++ assert!(!r.skip_first_page_up()); ++ assert!(STATS.fresh_told_invalid.load(Ordering::Relaxed) > before); ++ assert!(!r.state.host_valid()); ++ } ++ ++ /// Mode 2 ignores that mark on a texture that never was current ++ #[test] ++ fn mode_two_ignores_a_mark_on_a_new_texture() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 2); ++ r.set_is_host_valid(false); ++ assert!(r.skip_first_page_up()); ++ assert!(r.state.host_valid()); ++ } ++ ++ /// A skip is the first prepare only: a copy that went stale later is paged, in either ++ /// mode, through pages bound then ++ #[test] ++ fn a_later_stale_mark_pages_up() { ++ for mode in [1, 2] { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, mode); ++ assert!(r.skip_first_page_up()); ++ r.set_is_host_valid(false); ++ assert!(!r.skip_first_page_up()); ++ let enc = e.blit(); ++ let m = e.p.call_log().mark(); ++ r.prepare_in_encoder(Hold::NONE, Some(&enc)).unwrap(); ++ assert_eq!(e.sels(m), vec!["new_buffer_for_virtual_page", UP, UP, UP]); ++ assert!(r.state.host_valid()); ++ } ++ } ++ ++ /// Volatile: `Metal` may have emptied it, and that path devolatilizes first ++ #[test] ++ fn volatile_keeps_its_path() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 2); ++ r.state.set_is_volatile(true); ++ assert!(!r.skip_first_page_up()); ++ assert!(!r.state.host_valid()); ++ } ++ ++ /// Pages bound by an earlier use: they stay in charge ++ #[test] ++ fn bound_pages_refuse() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 2); ++ parts_of(&r).paged.ensure_physical(Hold::NONE).unwrap(); ++ assert!(!r.skip_first_page_up()); ++ } ++ ++ /// After a skip a page-down still binds the pages and copies the real contents, and ++ /// the task hears of the page ++ #[test] ++ fn a_later_page_down_works_and_is_noted() { ++ let e = Env::new(); ++ let r = private(&e, 2, 3, 1); ++ assert!(r.skip_first_page_up()); ++ r.state.set_guest_valid(false); ++ let enc = e.blit(); ++ let m = e.p.call_log().mark(); ++ let (synchronized, lines) = ++ logged(|| r.synchronize_in_encoder(Hold::NONE, Some(&enc))); ++ synchronized.unwrap(); ++ assert_eq!( ++ e.sels(m), ++ vec!["new_buffer_for_virtual_page", DOWN, DOWN, DOWN] ++ ); ++ assert!(lines.is_empty()); ++ assert!(r.state.guest_valid()); ++ assert_eq!(e.rec.paged_down(), vec![7]); ++ } ++ ++ /// A page-down of another storage mode is no one's business ++ #[test] ++ fn shared_page_down_is_not_noted() { ++ let e = Env::new(); ++ e.rec.set_fresh_page_up_mode(1); ++ let r = made(&e, 2, 3); ++ r.state.set_host_valid(true); ++ r.state.set_guest_valid(false); ++ let enc = e.blit(); ++ r.synchronize_in_encoder(Hold::NONE, Some(&enc)).unwrap(); ++ assert!(e.rec.paged_down().is_empty()); ++ } ++ } ++ + mod paging { + use super::*; + +diff --git a/crates/reims-vgpu-core/src/resources/validity.rs b/crates/reims-vgpu-core/src/resources/validity.rs +index 5f2a2312..050343f0 100644 +--- a/crates/reims-vgpu-core/src/resources/validity.rs ++++ b/crates/reims-vgpu-core/src/resources/validity.rs +@@ -21,6 +21,10 @@ pub struct ResourceState { + host_valid: AtomicBool, + guest_valid: AtomicBool, + is_volatile: AtomicBool, ++ /// LOCAL (Atlas): the host copy was marked stale at least once, by anyone. ++ host_told_invalid: AtomicBool, ++ /// LOCAL (Atlas): a host copy that was current went stale at least once. ++ host_lost_valid: AtomicBool, + } + + impl ResourceState { +@@ -30,7 +34,21 @@ impl ResourceState { + } + /// Stores the host-valid flag. + pub fn set_host_valid(&self, v: bool) { +- self.host_valid.store(v, Ordering::Relaxed) ++ let was = self.host_valid.swap(v, Ordering::Relaxed); ++ if !v { ++ self.host_told_invalid.store(true, Ordering::Relaxed); ++ if was { ++ self.host_lost_valid.store(true, Ordering::Relaxed); ++ } ++ } ++ } ++ /// LOCAL (Atlas): whether anything ever marked the host copy stale. ++ pub fn host_told_invalid(&self) -> bool { ++ self.host_told_invalid.load(Ordering::Relaxed) ++ } ++ /// LOCAL (Atlas): whether a current host copy ever went stale. ++ pub fn host_lost_valid(&self) -> bool { ++ self.host_lost_valid.load(Ordering::Relaxed) + } + /// The raw guest-valid flag. + pub fn guest_valid(&self) -> bool { +diff --git a/crates/reims-vgpu-core/src/testkit/resource_host.rs b/crates/reims-vgpu-core/src/testkit/resource_host.rs +index c6723c9b..9274152a 100644 +--- a/crates/reims-vgpu-core/src/testkit/resource_host.rs ++++ b/crates/reims-vgpu-core/src/testkit/resource_host.rs +@@ -13,6 +13,7 @@ + + use crate::testkit::script::Policy; + use std::sync::Arc; ++use std::sync::atomic::{AtomicU8, Ordering}; + + use crate::sync::Mutex; + +@@ -48,6 +49,10 @@ struct Shared { + paging_queue: Mutex>>, + /// What runs inside the next call of a selector, once (a racing caller's whole call). + during: Mutex>, ++ /// The answer of `fresh_page_up_mode`, as the task would give it. ++ fresh_page_up_mode: AtomicU8, ++ /// The pages `note_paged_down` was told of. ++ paged_down: Mutex>, + } + + /// A scripted task. Siblings share one log, script and state. +@@ -77,10 +82,24 @@ impl FakeResourceHost { + provided: Mutex::new(Vec::new()), + paging_queue: Mutex::new(None), + during: Mutex::new(Vec::new()), ++ fresh_page_up_mode: AtomicU8::new(0), ++ paged_down: Mutex::new(Vec::new()), + }), + }) + } + ++ /// Sets the task's answer on skipping a first page-up. ++ pub fn set_fresh_page_up_mode(&self, mode: u8) { ++ self.shared ++ .fresh_page_up_mode ++ .store(mode, Ordering::Relaxed); ++ } ++ ++ /// The pages noted as paged down. ++ pub fn paged_down(&self) -> Vec { ++ self.shared.paged_down.lock().clone() ++ } ++ + /// Another handle on the same state: a second task object whose last release a test + /// controls on its own. + pub fn sibling(&self) -> Arc { +@@ -177,6 +196,14 @@ impl FakeResourceHost { + } + + impl ResourceHost

for FakeResourceHost { ++ fn fresh_page_up_mode(&self, _page: u32) -> u8 { ++ self.shared.fresh_page_up_mode.load(Ordering::Relaxed) ++ } ++ ++ fn note_paged_down(&self, page: u32) { ++ self.shared.paged_down.lock().push(page); ++ } ++ + fn new_buffer_for_virtual_page( + &self, + _hold: Hold<'_>, +diff --git a/crates/reims-vgpu-vulkan/src/os/linux/vm.rs b/crates/reims-vgpu-vulkan/src/os/linux/vm.rs +index 5ef6bc84..c434e8c5 100644 +--- a/crates/reims-vgpu-vulkan/src/os/linux/vm.rs ++++ b/crates/reims-vgpu-vulkan/src/os/linux/vm.rs +@@ -46,7 +46,9 @@ + + use std::os::fd::{AsRawFd as _, FromRawFd as _, OwnedFd}; + use std::sync::Arc; ++use std::sync::atomic::Ordering; + ++use reims_vgpu_core::memory::mapping::{VIEW_MMAP_NS, VIEW_MMAPS}; + use reims_vgpu_core::num::{ToU64, ToUsize}; + use reims_vgpu_core::platform::os::vm::{HostAddr, HostVm, KernReturn, VmProt}; + use reims_vgpu_core::raise::death::{DeathId, DeathKind, fatal_with_detail}; +@@ -530,6 +532,9 @@ impl HostVm for LinuxVm { + return Err(INVALID_ADDRESS); + }; + budget::admit(mappings_added(&pieces), "a view of guest pages"); ++ // LOCAL (Atlas): the calls' share of a view, for the `mapstat` counters. ++ #[allow(clippy::disallowed_methods)] ++ let timed = std::time::Instant::now(); + for (done, piece) in pieces.iter().enumerate() { + if let Err(kr) = file( + Some(target + piece.at), +@@ -543,6 +548,9 @@ impl HostVm for LinuxVm { + return Err(kr); + } + } ++ let ns = u64::try_from(timed.elapsed().as_nanos()).unwrap_or(u64::MAX); ++ VIEW_MMAP_NS.fetch_add(ns, Ordering::Relaxed); ++ VIEW_MMAPS.fetch_add(pieces.len().to_u64(), Ordering::Relaxed); + files::record_pieces(target, pieces); + Ok(HostAddr::from_addr(target.to_usize())) + } diff --git a/patches/reims-vgpu/0014-only-a-sink-that-holds-unmaps-is-asked.patch b/patches/reims-vgpu/0014-only-a-sink-that-holds-unmaps-is-asked.patch new file mode 100644 index 0000000..d4545d5 --- /dev/null +++ b/patches/reims-vgpu/0014-only-a-sink-that-holds-unmaps-is-asked.patch @@ -0,0 +1,82 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 01:25:54 +0000 +Subject: [PATCH 14/25] only a sink that holds unmaps is asked + +The hook read the surface's base for every sink, and a test's call log heard it. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-c/src/census.rs | 4 ++++ + crates/reims-vgpu-core/src/census.rs | 4 ++++ + crates/reims-vgpu-core/src/iosurface/table.rs | 10 +++++++++- + 3 files changed, 17 insertions(+), 1 deletion(-) + +diff --git a/crates/reims-vgpu-c/src/census.rs b/crates/reims-vgpu-c/src/census.rs +index 13179db0..cb9ce227 100644 +--- a/crates/reims-vgpu-c/src/census.rs ++++ b/crates/reims-vgpu-c/src/census.rs +@@ -119,6 +119,10 @@ impl Sink for HostSink { + }); + } + ++ fn holds_unmaps(&self) -> bool { ++ true ++ } ++ + fn unmapping(&self, start: u64, len: u64) -> bool { + crate::coherent::unmapping(start, len) + } +diff --git a/crates/reims-vgpu-core/src/census.rs b/crates/reims-vgpu-core/src/census.rs +index 291bcfcd..87fa5653 100644 +--- a/crates/reims-vgpu-core/src/census.rs ++++ b/crates/reims-vgpu-core/src/census.rs +@@ -45,6 +45,10 @@ pub trait Sink: Send + Sync { + /// A surface was mapped. `pages` is each guest page's host address, none for a host-only one. + /// On the IOSurface device's queue, under its state's lock. + fn mapped(&self, facts: &SurfaceFacts, pages: &[u64], page_size: u64); ++ /// Whether [`unmapping`](Self::unmapping) has anything to say. ++ fn holds_unmaps(&self) -> bool { ++ false ++ } + /// The guest asks to unmap the surface over `len` bytes from the host address `start`, and + /// nothing was checked yet: whether it may go on, else it answers busy. What it held of the + /// surface it lets go of. +diff --git a/crates/reims-vgpu-core/src/iosurface/table.rs b/crates/reims-vgpu-core/src/iosurface/table.rs +index f0fcf652..ade7fcaa 100644 +--- a/crates/reims-vgpu-core/src/iosurface/table.rs ++++ b/crates/reims-vgpu-core/src/iosurface/table.rs +@@ -310,6 +310,7 @@ impl HostDeviceCore

{ + } + let index = mapping_id.to_usize(); + if let Some(sink) = crate::census::sink() ++ && sink.holds_unmaps() + && let Some(slot) = st.slots.get(index) + && let Some(surface) = &slot.surface + { +@@ -503,6 +504,11 @@ mod tests { + let r = rig(); + r.mapped_v2(3, 2); + let d = v2(2, ROOT); ++ let heard = take_mine(); ++ let base_address = heard.iter().find_map(|heard| match heard { ++ Heard::Mapped(facts, ..) => Some(facts.base_address), ++ _ => None, ++ }); + let facts = SurfaceFacts { + mapping_id: 3, + pixel_format: d.base.pixel_format, +@@ -515,10 +521,12 @@ mod tests { + compression_type: u32::from(d.base.compression_type), + read_only: false, + host_only: false, ++ base_address: base_address.unwrap_or_default(), ++ base_offset: 0, + }; + let pages = vec![r.host(SURF), r.host(SURF + PAGE)]; + assert!( +- take_mine().contains(&Heard::Mapped(facts, pages, PAGE)), ++ heard.contains(&Heard::Mapped(facts, pages, PAGE)), + "the guest's pages" + ); + let held = r.held(3); diff --git a/patches/reims-vgpu/0015-a-boot-test-of-coherent-pages-through-NV.patch b/patches/reims-vgpu/0015-a-boot-test-of-coherent-pages-through-NV.patch new file mode 100644 index 0000000..bd21268 --- /dev/null +++ b/patches/reims-vgpu/0015-a-boot-test-of-coherent-pages-through-NV.patch @@ -0,0 +1,480 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 02:23:01 +0000 +Subject: [PATCH 15/25] a boot test of coherent pages through + NVMe DMA + +dma.c swaps through QEMU's NVMe controller over pages armed, held, frozen, stale +and plain, under rvg_coherent_rehearse, and checks every word; run-dma.sh fails +one that reached no such page. coherent_test.c gets the stale page that crashed +the VM. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-c/include/reims_vgpu.h | 3 + + crates/reims-vgpu-c/src/coherent.rs | 58 ++++++ + crates/reims-vgpu-c/tests/c/harness.c | 1 + + crates/reims-vgpu-c/tests/c_harness.rs | 2 +- + docs/api.md | 2 +- + scripts/internal/census/census_report.py | 5 + + scripts/internal/census/coherent_test.c | 15 ++ + scripts/internal/census/guest/dma.c | 213 +++++++++++++++++++++++ + scripts/internal/census/guest/run-dma.sh | 45 +++++ + 9 files changed, 342 insertions(+), 2 deletions(-) + create mode 100644 scripts/internal/census/guest/dma.c + create mode 100755 scripts/internal/census/guest/run-dma.sh + +diff --git a/crates/reims-vgpu-c/include/reims_vgpu.h b/crates/reims-vgpu-c/include/reims_vgpu.h +index 3adaf431..3e4f9ea6 100644 +--- a/crates/reims-vgpu-c/include/reims_vgpu.h ++++ b/crates/reims-vgpu-c/include/reims_vgpu.h +@@ -1004,6 +1004,9 @@ void rvg_coherent_abandon(void); + void rvg_coherent_reset(void); + // The surface over `len` bytes from `start` was unmapped, and its pages are the guest's to reuse. + void rvg_coherent_unmapped(uint64_t start, uint64_t len); ++// With REIMS_VGPU_COHERENT_REHEARSE set, a test's: a thread runs the surface at `base` through every ++// step a texture's command buffers make, with no GPU, until it goes. Otherwise nothing. ++void rvg_coherent_rehearse(uint64_t base, uint32_t width, uint32_t height, uint64_t row); + // One JSON line a surface under `label` into `out`, up to `room` bytes: the bytes they took, + // which may exceed `room`. + size_t rvg_coherent_report(const char *label, char *out, size_t room); +diff --git a/crates/reims-vgpu-c/src/coherent.rs b/crates/reims-vgpu-c/src/coherent.rs +index d8f6e9ad..25278ac4 100644 +--- a/crates/reims-vgpu-c/src/coherent.rs ++++ b/crates/reims-vgpu-c/src/coherent.rs +@@ -59,6 +59,53 @@ mod on { + + pub(super) use metal2vulkan_runtime::shadow_coherent as runtime; + ++ /// While `REIMS_VGPU_COHERENT_REHEARSE=:` says so, ++ /// a thread runs a surface's command buffers over its top half, with no GPU, each "image" a ++ /// new word in every texel, until the surface goes. ++ pub(super) fn rehearse(base: u64, width: u32, height: u32, row: u64) { ++ use metal2vulkan_runtime::shadow_coherent::{Layout, Rehearsal}; ++ use std::time::Duration; ++ ++ let Ok(asked) = std::env::var("REIMS_VGPU_COHERENT_REHEARSE") else { ++ return; ++ }; ++ let (period, every) = asked.split_once(':').unwrap_or((&asked, "0")); ++ let period = Duration::from_micros(period.parse().unwrap_or(500)); ++ let every: u64 = every.parse().unwrap_or(0); ++ let frame = std::sync::Arc::new(std::sync::atomic::AtomicU32::new(0)); ++ let word = frame.clone(); ++ let layout = Layout { base, row, texel: 4, width, height }; ++ let read = Box::new(move |[x0, y0, x1, y1]: [u32; 4], to: *mut u8, stride: u32| { ++ let word = word.load(std::sync::atomic::Ordering::Relaxed) | 0xc0de_0000; ++ for y in 0..u64::from(y1 - y0) { ++ for x in 0..u64::from(x1 - x0) { ++ let at = (y * u64::from(stride) + x) * 4; ++ // SAFETY: the sync gives `to` over the rectangle's rows of `stride` texels. ++ unsafe { to.add(at as usize).cast::().write_unaligned(word) }; ++ } ++ } ++ true ++ }); ++ let area = [0, 0, width, height / 2]; ++ let surface = Rehearsal::new(layout, read); ++ let spawned = std::thread::Builder::new().name("rehearsal".into()).spawn(move || { ++ if !surface.track() { ++ eprintln!("rehearsal: the surface at {base:#x} does not track"); ++ return; ++ } ++ eprintln!("rehearsal: {width}x{height} at {base:#x}"); ++ for id in (1u64 << 48).. { ++ frame.fetch_add(1, std::sync::atomic::Ordering::Relaxed); ++ let take = every != 0 && id % every == 0; ++ if !surface.frame(id, area, period / 4, take) { ++ return; ++ } ++ std::thread::sleep(period); ++ } ++ }); ++ drop(spawned); ++ } ++ + /// The host's hooks, as the runtime's marks. + pub(super) struct HostPages(pub(super) RvgCoherentHooks); + +@@ -225,6 +272,17 @@ pub extern "C" fn rvg_coherent_unmapped(start: u64, len: u64) { + }); + } + ++/// A rehearsal of the surface. See the header. ++#[unsafe(no_mangle)] ++pub extern "C" fn rvg_coherent_rehearse(base: u64, width: u32, height: u32, row: u64) { ++ ffi::export(|| { ++ #[cfg(feature = "vulkan")] ++ on::rehearse(base, width, height, row); ++ #[cfg(not(feature = "vulkan"))] ++ let _ = (base, width, height, row); ++ }); ++} ++ + /// The report. See the header. + /// + /// # Safety +diff --git a/crates/reims-vgpu-c/tests/c/harness.c b/crates/reims-vgpu-c/tests/c/harness.c +index 639e4aff..c47826dd 100644 +--- a/crates/reims-vgpu-c/tests/c/harness.c ++++ b/crates/reims-vgpu-c/tests/c/harness.c +@@ -185,6 +185,7 @@ static void check_status_and_log(void) { + rvg_coherent_abandon(); + rvg_coherent_reset(); + rvg_coherent_unmapped(0, 0); ++ rvg_coherent_rehearse(0, 0, 0, 0); + CHECK(rvg_coherent_report("", NULL, 0) == 0); + CHECK(RVG_MAX_DISPLAY_PORTS == 8 && RVG_DEVICE_MMIO_SIZE == 0x4000); + CHECK(RVG_PCI_VENDOR_ID == 0x106b && RVG_PCI_DEVICE_ID == 0xeeee); +diff --git a/crates/reims-vgpu-c/tests/c_harness.rs b/crates/reims-vgpu-c/tests/c_harness.rs +index 7b4c7053..1c08e961 100644 +--- a/crates/reims-vgpu-c/tests/c_harness.rs ++++ b/crates/reims-vgpu-c/tests/c_harness.rs +@@ -111,7 +111,7 @@ fn c_harness_calls_every_function() { + Some(&rest[..end]) + }) + .collect(); +- assert_eq!(declared.len(), 56, "{declared:?}"); ++ assert_eq!(declared.len(), 57, "{declared:?}"); + let missing: Vec<&str> = declared + .iter() + .copied() +diff --git a/docs/api.md b/docs/api.md +index 900d4255..4459a8b7 100644 +--- a/docs/api.md ++++ b/docs/api.md +@@ -703,7 +703,7 @@ The header's 47 functions, in its order. "May" is the header's mark "may call ba + | 40 | `rvg_video_device_invalidate` | | a reset's first phase: the session sends the guest nothing more | + | 41 | `rvg_video_device_reset` | | both phases: the session's work drains and it ends | + | 42a | `rvg_census_set_callbacks` | | LOCAL (Atlas): the touch census's news of surfaces and presents (`reims_vgpu_core::census`); no calls without it | +-| 42b | `rvg_coherent_set_hooks`, `rvg_coherent_fault`, `rvg_coherent_timed`, `rvg_coherent_probe`, `rvg_coherent_abandon`, `rvg_coherent_reset`, `rvg_coherent_unmapped`, `rvg_coherent_report` | | LOCAL (Atlas): trap-coherent surfaces (metal2vulkan's `shadow_coherent.rs`): QEMU's census view marks their pages, and tells of each touch; no effect without the live switch `surface_coherent` | ++| 42b | `rvg_coherent_set_hooks`, `rvg_coherent_fault`, `rvg_coherent_timed`, `rvg_coherent_probe`, `rvg_coherent_abandon`, `rvg_coherent_reset`, `rvg_coherent_unmapped`, `rvg_coherent_rehearse`, `rvg_coherent_report` | | LOCAL (Atlas): trap-coherent surfaces (metal2vulkan's `shadow_coherent.rs`): QEMU's census view marks their pages, and tells of each touch; no effect without the live switch `surface_coherent` | + | 42 | `rvg_scaler_device_create` | | the M2 scaler device (106b:1a0b), from an `RvgScalerDeviceConfig`, made on every host: transforming with the library's own kernel on the GPU it draws with (Metal's on macOS, the Vulkan platform's on Linux, any GPU, a software rasterizer included), and with none where that opens no device or the device refuses the kernel | + | 43 | `rvg_scaler_device_release` | | drops the host's reference; the last one drops its accelerator and ends its guest-memory table | + | 44 | `rvg_scaler_device_scales` | | whether the device has a GPU to transform on; where it has not, the host refuses the guest's features (FEATURES_OK), so the guest's driver does without the device | +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +index b13d5aeb..b29a1a12 100644 +--- a/scripts/internal/census/census_report.py ++++ b/scripts/internal/census/census_report.py +@@ -138,6 +138,8 @@ def life(surfaces): + def coherent(directory, wanted): + """The last report of each surface at the mark `wanted`, or at the last mark.""" + rows = list(lines(os.path.join(directory, "coherent.jsonl"))) ++ touches = [r for r in rows if "touches" in r] ++ rows = [r for r in rows if "at" in r] + label = wanted or (rows[-1].get("label") if rows else None) + if label is None: + sys.exit(f"{directory}: no coherent.jsonl") +@@ -159,6 +161,9 @@ def coherent(directory, wanted): + f" {r['demotions']:>6} {r['writes']:>6} {cost['detect'] / 1e3:>7.1f}" + f" {cost['sync'] / 1e3:>7.1f} {cost['fetch'] / 1e3:>7.1f} {cost['total'] / 1e3:>7.1f}" + f" {cost['eager_would'] / 1e3:>8.1f} {'yes' if r['pays'] else 'NO'}") ++ for r in touches[-1:]: ++ print(f"QEMU's own accesses: open {r['open']} armed {r['armed']} held {r['held']}" ++ f" frozen {r['frozen']} stale {r['stale']}") + bad = [r for r in found.values() if r["sync_errors"] or r["overwrite_risk"]] + for r in bad: + print(f"surface {r['at']}: sync_errors {r['sync_errors']}, overwrite_risk {r['overwrite_risk']}") +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +index 3cb7eca5..15de6d6d 100644 +--- a/scripts/internal/census/coherent_test.c ++++ b/scripts/internal/census/coherent_test.c +@@ -242,6 +242,20 @@ static void t_emulator_read(struct vcpu *c) { + census_surface_unmapped(6); + } + ++static void t_stale_page(struct vcpu *c) { ++ printf("[stale] QEMU's own access to a surface's old page, another surface tracked\n"); ++ surface(13, 0xf00000); ++ surface(14, 0x1000000); ++ census_surface_unmapped(13); ++ s.off = 0xf00000; ++ CHECK(state(0) == 6, "the page is stale"); ++ census_coherent_access(main_map + 0xf00000, 8 * PG, true); ++ census_coherent_access(main_map + 0xf00000, 8 * PG, false); ++ CHECK(1, "and a device reads and writes it as RAM"); ++ s.off = 0x1000000; ++ census_surface_unmapped(14); ++} ++ + static void t_unmap_reset(struct vcpu *c) { + printf("[unmap] armed, held and open pages all come off\n"); + surface(7, 0x700000); +@@ -472,6 +486,7 @@ int main(int argc, char **argv) { + t_frozen(&c); + t_kernel_copy(&c); + t_emulator_read(&c); ++ t_stale_page(&c); + t_unmap_reset(&c); + t_coherent_race(mib); + t_arm_window(); +diff --git a/scripts/internal/census/guest/dma.c b/scripts/internal/census/guest/dma.c +new file mode 100644 +index 00000000..7c5e6ae1 +--- /dev/null ++++ b/scripts/internal/census/guest/dma.c +@@ -0,0 +1,213 @@ ++/* A multiboot guest that swaps through a real NVMe controller, as a loaded macOS does, while ++ * real surfaces of the IOSurface device are live: QEMU's nvme_read and nvme_write, dma_blk_cb and ++ * address_space_map run on the main thread over pages armed, held, frozen, stale and plain. Each ++ * round fills a page, writes it out, spoils it, reads it back, and checks every word. Exit code ++ * (stage << 1) | 1: 0x11 passes. Run it with REIMS_VGPU_COHERENT_REHEARSE, see run-dma.sh. ++ * ROUNDS the swap-outs, each with its swap-in ++ * CHURN 1 where the rehearsal takes the image, so that a GPU write may rightly replace what a ++ * page of its rectangle held: those pages are then not checked */ ++#define main_c device_main ++#include "device.c" ++#undef main_c ++ ++#ifndef ROUNDS ++#define ROUNDS 1000000u ++#endif ++#ifndef CHURN ++#define CHURN 0 ++#endif ++#define ASQ 0x01100000u ++#define ACQ 0x01101000u ++#define IOSQ 0x01102000u ++#define IOCQ 0x01103000u ++#define PLAIN 0x10000000u /* 64 pages no surface ever held */ ++#define STALE 0x30000000u /* the pages of a surface that went: a word that names no surface */ ++#define STALE_ID 7u ++#define QSIZE 64u ++ ++static uint32_t nv, stride, sqt, cqh_a, sqt_a, cqh, phase, phase_a; /* link.ld keeps no .data */ ++ ++static void nvme_cmd(uint32_t *c, uint32_t sq, uint32_t *tail, uint32_t q) { ++ uint32_t *to = (uint32_t *)(uintptr_t)(sq + *tail * 64); ++ for (int i = 0; i < 16; i++) to[i] = c[i]; ++ *tail = (*tail + 1) % QSIZE; ++ R32(nv + 0x1000 + 2 * q * stride) = *tail; ++} ++static uint32_t nvme_done(uint32_t cq, uint32_t *head, uint32_t *ph, uint32_t q) { ++ volatile uint32_t *e = (volatile uint32_t *)(uintptr_t)(cq + *head * 16); ++ for (uint32_t n = 0; (e[3] >> 16 & 1) != *ph; n++) { ++ if (n > 2000000000u) die(0x40); ++ __asm__("pause"); ++ } ++ uint32_t status = e[3] >> 17; ++ *head = (*head + 1) % QSIZE; ++ if (!*head) *ph ^= 1; ++ R32(nv + 0x1000 + (2 * q + 1) * stride) = *head; ++ return status; ++} ++static void admin(uint32_t op, uint32_t prp, uint32_t cdw10, uint32_t cdw11) { ++ uint32_t c[16] = {0}; ++ c[0] = op; c[1] = op == 6 ? 1 : 0; c[6] = prp; c[10] = cdw10; c[11] = cdw11; ++ nvme_cmd(c, ASQ, &sqt_a, 0); ++ if (nvme_done(ACQ, &cqh_a, &phase_a, 0)) die(0x41); ++} ++/* The one request: `count` pages at the guest's `page`, then the PRP list `list` for more. */ ++static void io(uint32_t op, uint32_t lba, uint32_t prp1, uint32_t prp2, uint32_t count) { ++ uint32_t c[16] = {0}; ++ c[0] = op | (lba & 63) << 16; c[1] = 1; c[6] = prp1; c[8] = prp2; ++ c[10] = lba; c[12] = count * 8 - 1; ++ nvme_cmd(c, IOSQ, &sqt, 1); ++ if (nvme_done(IOCQ, &cqh, &phase, 1)) die(0x42); ++} ++static void nvme_up(void) { ++ phase = phase_a = 1; ++ for (int d = 0; d < 32 && !nv; d++) ++ if (cfg(d, 8) >> 8 == 0x010802) { ++ nv = cfg(d, 0x10) & ~0xfu; ++ cfgw(d, 4, cfg(d, 4) | 6); ++ } ++ if (!nv) die(0x43); ++ stride = 4u << (R32(nv + 4) & 15); ++ R32(nv + 0x14) = 0; ++ for (uint32_t n = 0; R32(nv + 0x1c) & 1; n++) if (n > 400000000u) die(0x44); ++ R32(nv + 0x24) = (QSIZE - 1) << 16 | (QSIZE - 1); ++ R64(nv + 0x28) = ASQ; ++ R64(nv + 0x30) = ACQ; ++ R32(nv + 0x14) = 1 | 6 << 16 | 4 << 20; ++ for (uint32_t n = 0; !(R32(nv + 0x1c) & 1); n++) if (n > 400000000u) die(0x45); ++ admin(5, IOCQ, (QSIZE - 1) << 16 | 1, 1); ++ admin(1, IOSQ, (QSIZE - 1) << 16 | 1, 1 << 16 | 1); ++} ++ ++static void fill(uint32_t p, uint32_t seed) { ++ for (uint32_t w = 0; w < PG / 4; w++) R32(p + 4 * w) = seed ^ (w * 0x9e3779b1u); ++} ++static void hex(uint32_t v) { ++ for (int i = 28; i >= 0; i -= 4) outb(0xe9, "0123456789abcdef"[v >> i & 15]); ++ outb(0xe9, ' '); ++} ++/* On the console: the page, its first wrong word, what it holds, what it should, how many differ. */ ++static int same(uint32_t p, uint32_t seed) { ++ uint32_t wrong = 0, first = 0, held = 0; ++ for (uint32_t w = 0; w < PG / 4; w++) { ++ uint32_t is = R32(p + 4 * w); ++ if (is != (seed ^ (w * 0x9e3779b1u))) { ++ if (!wrong++) { first = w; held = is; } ++ } ++ } ++ if (wrong) { ++ hex(p); hex(first); hex(held); hex(seed ^ (first * 0x9e3779b1u)); hex(wrong); ++ outb(0xe9, '\n'); ++ } ++ return !wrong; ++} ++/* What a pager does to one page, checked or not. */ ++static void swap(uint32_t p, uint32_t seed, uint32_t lba, int check, int stage) { ++ fill(p, seed); ++ io(1, lba, p, 0, 1); ++ fill(p, ~seed); ++ io(2, lba, p, 0, 1); ++ if (check && !same(p, seed)) die(stage); ++} ++ ++/* Long enough for the rehearsal's next frames, at its 500 us. */ ++static void wait_frames(void) { ++ for (uint32_t n = 0; n < 60000; n++) __asm__("pause"); ++} ++ ++static void unmap(uint32_t id) { ++ for (uint32_t n = 0; ; n++) { ++ int r = cmd(2, id); ++ if (!r) return; ++ /* Busy: work runs in its image, and the guest tries again. */ ++ if (n > 2000000) die(0x46); ++ for (int i = 0; i < 2000; i++) __asm__("pause"); ++ } ++} ++ ++void main_c(void) { ++ uint32_t i; ++ for (int d = 0; d < 32 && !bar; d++) ++ if (cfg(d, 0) == (0xeeefu << 16 | 0x106b)) { ++ bar = cfg(d, 0x10) & ~0xfu; ++ cfgw(d, 4, cfg(d, 4) | 6); ++ } ++ if (!bar) die(0x31); ++ for (i = 0; i < 16; i++) R32(TABLE + 4 * i) = pte(DPAGE + i * PG); ++ R32(bar + 0x1030) = 1; ++ R32(bar + 0x1000) = RING; ++ R32(bar + 0x1008) = 256; ++ R32(bar + 0x1010) = TABLE; ++ wr = R32(bar + 0x1020); ++ nvme_up(); ++ /* The driver itself, before any surface. */ ++ swap(PLAIN, 0x0badcafeu, 8 * 4000, 1, 0x48); ++ ++ /* A surface that comes and goes: its pages stay stale for the guest to reuse. */ ++ for (i = 0; i < PAGES; i++) R32(ROOT + 4 * i) = pte(STALE + 2 * i * PG); ++ descriptor(STALE_ID, 0); ++ if (cmd(1, STALE_ID)) die(0x47); ++ for (uint32_t n = 0; n < 40000000; n++) __asm__("pause"); /* its rehearsal gets going */ ++ unmap(STALE_ID); ++ ++ /* The surface that stays: its top half is the rectangle the "GPU" writes (armed, frozen as ++ * pages open), its bottom half stays held. Pages scattered, as a real surface's. */ ++ for (i = 0; i < PAGES; i++) R32(ROOT + 4 * i) = pte(page(i)); ++ descriptor(ID, 0); ++ if (cmd(1, ID)) die(0x34); ++ for (uint32_t n = 0; n < 40000000; n++) __asm__("pause"); ++ ++ /* Direct RAM paths, with no map: a list of pages QEMU reads, an identify it writes. */ ++ for (i = 0; i < 4; i++) { ++ uint32_t list = page(600 + i), armed_list = page(100 + i), target = page(40 + i); ++ R32(list) = target + PG; R32(list + 4) = 0; ++ R32(list + 8) = target + 2 * PG; R32(list + 12) = 0; ++ R32(armed_list) = target + PG; R32(armed_list + 4) = 0; ++ R32(armed_list + 8) = target + 2 * PG; R32(armed_list + 12) = 0; ++ io(2, 8 * 1000, target, list, 3); ++ io(2, 8 * 1000, target, armed_list, 3); ++ admin(6, page(120 + i), 1, 0); ++ admin(6, page(620 + i), 1, 0); ++ admin(6, STALE + 2 * i * PG, 1, 0); ++ } ++ ++ /* The device first, the guest not having touched the page: a swap-in into each state. A ++ * swap-out of an armed page must carry what the GPU left there, not the page's old bytes. */ ++ for (i = 0; i < 300; i++) { ++ uint32_t seed = 0x51ed2700u + i, lba = 8 * (3000 + i); ++ fill(PLAIN + PG, seed); ++ io(1, lba, PLAIN + PG, 0, 1); ++ wait_frames(); ++ io(2, lba, page(i), 0, 1); /* armed, or held since a sync */ ++ if (!CHURN && !same(page(i), seed)) die(0x55); ++ wait_frames(); ++ io(2, lba, page(512 + i), 0, 1); /* held */ ++ if (!same(page(512 + i), seed)) die(0x56); ++ fill(page(i), ~seed); /* open */ ++ wait_frames(); ++ io(2, lba, page(i), 0, 1); /* frozen by the frames since */ ++ if (!CHURN && !same(page(i), seed)) die(0x57); ++ } ++ for (i = 0; i < 100; i++) { ++ wait_frames(); ++ io(1, 8 * (3400 + i), page(400 + i), 0, 1); /* out of an armed page */ ++ io(2, 8 * (3400 + i), PLAIN + 2 * PG, 0, 1); ++ if (R32(PLAIN + 2 * PG) >> 16 != 0xc0de) { hex(R32(PLAIN + 2 * PG)); die(0x58); } ++ } ++ ++ /* Every kind in turn, for ROUNDS swaps: the load of a guest short of memory. */ ++ for (uint32_t round = 0; round < ROUNDS; round++) { ++ uint32_t seed = round * 0x01000193u + 0xa5a5a5a5u, kind = round % 4, at = (round >> 2) % 400; ++ uint32_t lba = 8 * (kind * 512 + at); ++ switch (kind) { ++ case 0: swap(page(at), seed, lba, !CHURN, 0x50); break; /* armed, frozen */ ++ case 1: swap(page(512 + at), seed, lba, 1, 0x51); break; /* held */ ++ case 2: swap(STALE + 2 * at * PG, seed, lba, 1, 0x52); break; /* stale */ ++ default: swap(PLAIN + (at % 64) * PG, seed, lba, 1, 0x53); break; /* plain */ ++ } ++ } ++ unmap(ID); ++ /* A page of the surface, as plain RAM again. */ ++ swap(page(1), 0x12345678u, 8, 1, 0x54); ++ die(0x11); ++} +diff --git a/scripts/internal/census/guest/run-dma.sh b/scripts/internal/census/guest/run-dma.sh +new file mode 100755 +index 00000000..1e9fb760 +--- /dev/null ++++ b/scripts/internal/census/guest/run-dma.sh +@@ -0,0 +1,45 @@ ++#!/bin/bash ++# run-dma.sh [ENV=VALUE]...: dma.c on the installed $CENSUS_BUILD, in the run image ++# as the VM runs: Reims's device, the census, and the NVMe controller of bin/qemu-disk, on lavapipe, ++# CPUs 8-19. Output in /srv/warm/census/out/; exit 35 is the guest's pass. Also prints QEMU's ++# accesses by page state. $RAM (MiB) is 1024; 12288 is the VM's. ++name=$1 elf=$2; shift 2 ++Q=${QEMU_DIR:-/srv/data/macos-vm/home/.local/share/reims-ctl/qemu/${CENSUS_BUILD:?}} ++O=/srv/warm/census/out/$name RAM=${RAM:-1024} ++rm -rf "$O"; mkdir -p "$O" ++printf 'surface_lazy=1\nsurface_lazy_scoped=1\nsurface_coherent=1\n' > "$O/live.conf" ++# NODEMOTE=1: a surface stays armed however often it traps, or pages rarely stay marked for a device. ++[ -n "${NODEMOTE:-}" ] && printf 'surface_coherent_demote_pct=1000\nsurface_coherent_demote_hz=100000000\nsurface_coherent_hold_s=0\n' >> "$O/live.conf" ++truncate -s 64M "$O/disk.raw" ++args=() ++for e in "$@"; do args+=(-e "$e"); done ++timeout ${LIMIT:-280} docker run -i --rm --name census-dma-$name --init \ ++ --ulimit core=${CORE:-0} --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/kvm \ ++ -e HOME=/srv/warm/census/home -e VK_DRIVER_FILES=/usr/share/vulkan/icd.d/lvp_icd.json \ ++ -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json -e RUST_LOG=${RUST_LOG:-error} \ ++ -e M2V_LIVE=$O/live.conf -e REIMS_VGPU_CENSUS=$O -e REIMS_VGPU_COHERENT=1 -e REIMS_VGPU_WINDOW=0 \ ++ "${args[@]}" -w "$O" \ ++ -v /srv/data/macos-vm/home/.local/share/reims-ctl/qemu:/srv/data/macos-vm/home/.local/share/reims-ctl/qemu:ro \ ++ -v /srv/warm/census:/srv/warm/census macos-vm:run2 \ ++ taskset -c 8-19 $Q/bin/qemu-system-x86_64 -L $Q/share/qemu \ ++ -accel kvm -cpu host -smp 2 -machine q35,memory-backend=ram,dump-guest-core=off -display none -vga none \ ++ -object memory-backend-memfd,id=ram,size=${RAM}M,share=on \ ++ -device isa-debug-exit,iobase=0xf4,iosize=1 -chardev file,id=dbg,path=$O/console -device isa-debugcon,iobase=0xe9,chardev=dbg -serial none -monitor none \ ++ -drive id=disk,file=$O/disk.raw,if=none,format=raw,cache=${CACHE:-writeback},discard=unmap \ ++ -device nvme,drive=disk,serial=reims-ctl,addr=0xa \ ++ -device apple-gfx-pci,id=gfx,bus=pcie.0,addr=0x4,displays=1,resolution=1280x720 \ ++ -device apple-gfx-iosurface-pci,id=iosfc,gpu=gfx,bus=pcie.0,addr=0x6 \ ++ -kernel "$elf" ${EXTRA:-} 2>&1 | grep -vE '^metal2vulkan: ' | tail -${TAIL:-12} ++rc=${PIPESTATUS[0]} ++echo "exit $rc" ++# A pass counts only over a page of each state, with no error and the fuse whole. ++[ "${STATES:-1}" = 1 ] && python3 -I - "$O" <<'PY' ++import json, sys ++rows = [json.loads(l) for l in open(sys.argv[1] + "/coherent.jsonl")] ++last = [r for r in rows if "touches" in r][-1] ++missing = [k for k in ("armed", "held", "frozen", "stale") if not last[k]] ++tick = [json.loads(l) for l in open(sys.argv[1] + "/ticks.jsonl")][-1] ++print("states", {k: last[k] for k in ("open", "armed", "held", "frozen", "stale")}, ++ "errors", tick.get("errors"), "dead", tick.get("dead")) ++print("MISSED " + " ".join(missing) if missing else "every state reached") ++PY diff --git a/patches/reims-vgpu/0016-the-census-s-costs-per-sync-and-tests-of.patch b/patches/reims-vgpu/0016-the-census-s-costs-per-sync-and-tests-of.patch new file mode 100644 index 0000000..538c800 --- /dev/null +++ b/patches/reims-vgpu/0016-the-census-s-costs-per-sync-and-tests-of.patch @@ -0,0 +1,355 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 03:50:10 +0000 +Subject: [PATCH 16/25] the census's costs per sync, and tests + of what real surfaces do + +A scattered surface's runs, guest memory on the view against the main mapping beside marker +traffic, the report's copy, hold, arm and spent against saved, and the NVMe boot test reaching the +BARs at the VM's real RAM size. + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/census_report.py | 11 ++ + scripts/internal/census/census_test.c | 9 +- + scripts/internal/census/coherent_test.c | 211 +++++++++++++++++++++++ + scripts/internal/census/guest/dma.c | 23 ++- + scripts/internal/census/guest/run-dma.sh | 2 +- + 5 files changed, 252 insertions(+), 4 deletions(-) + +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +index b29a1a12..635a81c0 100644 +--- a/scripts/internal/census/census_report.py ++++ b/scripts/internal/census/census_report.py +@@ -161,7 +161,18 @@ def coherent(directory, wanted): + f" {r['demotions']:>6} {r['writes']:>6} {cost['detect'] / 1e3:>7.1f}" + f" {cost['sync'] / 1e3:>7.1f} {cost['fetch'] / 1e3:>7.1f} {cost['total'] / 1e3:>7.1f}" + f" {cost['eager_would'] / 1e3:>8.1f} {'yes' if r['pays'] else 'NO'}") ++ print("per sync in ms (sync_ns holds the copy and the hold), and what spent and saved came to") ++ print(f"{'surface':>12} {'defers':>7} {'syncs':>6} {'copy':>6} {'hold':>6} {'arm':>6} {'MB/s':>6}" ++ f" {'spent ms':>9} {'saved ms':>9} synced by") ++ for r in sorted(found.values(), key=lambda r: -r["spent_ns"])[:40]: ++ syncs = r["regions"] or 1 ++ speed = r["bytes_detiled"] / r["copy_ns"] * 1e3 if r["copy_ns"] else 0 ++ by = " ".join(f"{k}:{n}" for k, n in r["sync_by"].items() if n) ++ print(f"{r['at']:>12} {r['defers']:>7} {r['regions']:>6} {r['copy_ns'] / syncs / 1e6:>6.2f}" ++ f" {r['hold_ns'] / syncs / 1e6:>6.2f} {r['arm_ns'] / (r['arms'] or 1) / 1e6:>6.2f}" ++ f" {speed:>6.0f} {r['spent_ns'] / 1e6:>9.1f} {r['saved_ns'] / 1e6:>9.1f} {by}") + for r in touches[-1:]: ++ print(f"marker runs {r.get('runs', 0)}") + print(f"QEMU's own accesses: open {r['open']} armed {r['armed']} held {r['held']}" + f" frozen {r['frozen']} stale {r['stale']}") + bad = [r for r in found.values() if r["sync_errors"] or r["overwrite_risk"]] +diff --git a/scripts/internal/census/census_test.c b/scripts/internal/census/census_test.c +index de2d48ba..9d70f624 100644 +--- a/scripts/internal/census/census_test.c ++++ b/scripts/internal/census/census_test.c +@@ -38,11 +38,18 @@ __asm__( + "g_outs: mov $0x11,%dx\n rep outsb\n out %al,$0x10\n" + "g_loop: mov (%rbx),%rax\n add %rdx,%rbx\n dec %rcx\n jnz g_loop\n out %al,$0x10\n" + "g_wloop: mov %rax,(%rbx)\n add %rdx,%rbx\n dec %rcx\n jnz g_wloop\n out %al,$0x10\n" ++"g_copy: mov %rsi,%r8\n mov %rdi,%r9\n mov %rcx,%r10\n" ++"1: mov %r8,%rsi\n mov %r9,%rdi\n mov %r10,%rcx\n rep movsb\n dec %rax\n jnz 1b\n out %al,$0x10\n" ++"g_vcopy: vpcmpeqd %ymm1,%ymm1,%ymm1\n" ++"1: xor %rdx,%rdx\n" ++"2: vmovdqu (%rsi,%rdx),%ymm0\n vpblendvb %ymm1,%ymm0,%ymm0,%ymm0\n vmovdqu %ymm0,(%rdi,%rdx)\n" ++" add $32,%rdx\n cmp %rcx,%rdx\n jb 2b\n dec %rax\n jnz 1b\n out %al,$0x10\n" ++"g_chase: mov (%rbx),%rbx\n dec %rcx\n jnz g_chase\n out %al,$0x10\n" + "g_end:\n" + ".popsection\n"); + // clang-format on + extern const char g_start[], g_end[], g_read[], g_write[], g_lock[], g_avx[], g_repmovs[], g_outs[], +- g_loop[], g_wloop[]; ++ g_loop[], g_wloop[], g_copy[], g_vcopy[], g_chase[]; + #define STUB(s) (CODE + (uint64_t)((s) - g_start)) + + #define DIE(...) do { fprintf(stderr, __VA_ARGS__); fprintf(stderr, " (%s)\n", strerror(errno)); exit(1); } while (0) +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +index 15de6d6d..28978dc3 100644 +--- a/scripts/internal/census/coherent_test.c ++++ b/scripts/internal/census/coherent_test.c +@@ -455,6 +455,215 @@ static void t_coherent_cost(struct vcpu *c) { + } + } + ++// A surface as macOS lays one out: chunks of `run` pages, descending inside, apart in the view. ++static size_t scattered(uint64_t off, size_t n, size_t run, uint64_t *list) { ++ size_t chunks = 0; ++ ++ for (size_t i = 0; i < n; chunks++) { ++ for (size_t j = 0; j < run && i < n; j++, i++) { ++ list[i] = (uint64_t)main_map + off + (chunks * (run + 1) + run - 1 - j) * PG; ++ } ++ } ++ return chunks; ++} ++ ++enum { SCATTERED = 5650, SCATTERED_RUN = 9 }; ++#define SCATTERED_OFF 0x10000000UL ++static uint64_t scattered_main[SCATTERED], scattered_view[SCATTERED]; ++ ++static size_t scattered_surface(uint32_t id) { ++ size_t chunks = scattered(SCATTERED_OFF, SCATTERED, SCATTERED_RUN, scattered_main); ++ CensusSurface f = {.mapping_id = id, .width = 2000, .height = 2000, .pixel_format = 0x42475241, ++ .alloc_size = SCATTERED * PG, .bytes_per_row = 8000, ++ .base_address = scattered_main[0]}; ++ uint64_t start = 0, len = 0; ++ ++ census_surface_mapped(&f, scattered_main, SCATTERED, PG); ++ census_coh_surface(id, &start, &len); ++ if (!census_coh_track(start, len)) DIE("track"); ++ for (size_t i = 0; i < SCATTERED; i++) scattered_view[i] = start + i * PG; ++ return chunks; ++} ++ ++static void t_scattered(void) { ++ printf("[scattered] pages descending in the surface move in runs of the view\n"); ++ static uint64_t refused[SCATTERED], came[SCATTERED]; ++ uint64_t *view = scattered_view, tries = 200, armed_ns = 0, held_ns = 0, one_ns = 0; ++ size_t chunks = scattered_surface(21); ++ ++ uint64_t runs = census_coh_run_count(); ++ census_coh_arm(view, SCATTERED, refused, SCATTERED); ++ CHECK(census_coh_run_count() - runs == chunks, "an arm is one run a chunk: %llu for %zu chunks", ++ (unsigned long long)(census_coh_run_count() - runs), chunks); ++ int right = 0; ++ for (size_t i = 0; i < SCATTERED; i++) right += census_page_state((void *)scattered_main[i]) == 1; ++ CHECK(right == SCATTERED, "every page armed"); ++ runs = census_coh_run_count(); ++ census_coh_hold(view, SCATTERED, false, came, SCATTERED); ++ CHECK(census_coh_run_count() - runs == chunks, "a hold too"); ++ ++ census_coh_arm(view + 100, 3, refused, SCATTERED); ++ right = 0; ++ for (size_t i = 0; i < SCATTERED; i++) { ++ right += census_page_state((void *)scattered_main[i]) == (i >= 100 && i < 103 ? 1 : 2); ++ } ++ CHECK(right == SCATTERED, "an arm of three pages moved those three, wherever they lie"); ++ census_coh_hold(view, SCATTERED, false, came, SCATTERED); ++ ++ for (uint64_t i = 0; i < tries; i++) { ++ uint64_t t0 = now_ns(); ++ census_coh_arm(view, SCATTERED, refused, SCATTERED); ++ uint64_t t1 = now_ns(); ++ census_coh_hold(view, SCATTERED, false, came, SCATTERED); ++ armed_ns += t1 - t0; ++ held_ns += now_ns() - t1; ++ } ++ for (uint64_t i = 0; i < 20; i++) { ++ uint64_t t0 = now_ns(); ++ for (size_t p = 0; p < SCATTERED; p++) census_coh_arm(view + p, 1, refused, SCATTERED); ++ for (size_t p = 0; p < SCATTERED; p++) census_coh_hold(view + p, 1, false, came, SCATTERED); ++ one_ns += now_ns() - t0; ++ } ++ printf(" %d pages, %zu chunks: arm %.0f us, hold %.0f us; a run a page, both: %.0f us\n", ++ SCATTERED, chunks, armed_ns / 1e3 / tries, held_ns / 1e3 / tries, one_ns / 1e3 / 20); ++ census_surface_unmapped(21); ++} ++ ++// Guest memory speed, the view against the main mapping, beside the markers' traffic. ++#define MAIN_GPA (64UL << 30) ++enum { COPY, VCOPY, CHASE }; ++static struct vcpu speed_vcpus[8]; ++struct speed_run { ++ struct vcpu *c; ++ int kind; ++ uint64_t base, bytes; ++ atomic_int *stop; ++ uint64_t done, ns; ++}; ++ ++static void *speed_vcpu(void *arg) { ++ struct speed_run *r = arg; ++ struct vcpu *c = r->c; ++ uint64_t src = r->base, dst = src + r->bytes, t0, rounds = 0; ++ ++ setr(c, STUB(g_loop), 0, src, r->bytes * 2 / PG, PG, 0, 0); ++ run_guest(c, 0); ++ t0 = now_ns(); ++ while (!atomic_load(r->stop)) { ++ if (r->kind == COPY) setr(c, STUB(g_copy), 4, 0, r->bytes, 0, src, dst); ++ else if (r->kind == VCOPY) setr(c, STUB(g_vcopy), 4, 0, r->bytes, 0, src, dst); ++ else setr(c, STUB(g_chase), 0, src, 1 << 18, 0, 0, 0); ++ if (run_guest(c, 0) != RUN_IO) DIE("speed vcpu"); ++ rounds++; ++ } ++ r->ns = now_ns() - t0; ++ r->done = r->kind == CHASE ? rounds << 18 : rounds * 4 * r->bytes; ++ return 0; ++} ++ ++// The markers beside the vCPUs: the scattered surface armed and held `hz` times a second, 0 flat ++// out, and with `mapped` read by a vCPU of its own while held, so each move has EPT entries to zap ++// and every vCPU a TLB shootdown to take. ++static struct { int hz, mapped; atomic_int *stop; uint64_t frames; int per_page; } traffic; ++ ++static void *traffic_thread(void *unused) { ++ static uint64_t refused[SCATTERED], came[SCATTERED]; ++ uint64_t *view = scattered_view; ++ struct vcpu *reader = &speed_vcpus[7]; ++ uint64_t span = SCATTERED / SCATTERED_RUN * (SCATTERED_RUN + 1); ++ ++ while (!atomic_load(traffic.stop)) { ++ if (traffic.mapped) { ++ setr(reader, STUB(g_loop), 0, DATA_GPA + SCATTERED_OFF, span, PG, 0, 0); ++ if (run_guest(reader, 0) != RUN_IO) DIE("traffic reader"); ++ } ++ if (traffic.per_page) { ++ for (size_t p = 0; p < SCATTERED; p++) census_coh_arm(view + p, 1, refused, SCATTERED); ++ for (size_t p = 0; p < SCATTERED; p++) census_coh_hold(view + p, 1, false, came, SCATTERED); ++ } else { ++ census_coh_arm(view, SCATTERED, refused, SCATTERED); ++ census_coh_hold(view, SCATTERED, false, came, SCATTERED); ++ } ++ traffic.frames++; ++ if (traffic.hz) usleep(1000000 / traffic.hz); ++ } ++ return 0; ++} ++ ++static double speed(int kind, uint64_t base, int n, uint64_t bytes) { ++ static struct speed_run runs[16]; ++ pthread_t t[16], th; ++ atomic_int stop = 0; ++ double total = 0; ++ ++ traffic.stop = &stop; ++ traffic.frames = 0; ++ for (int i = 0; i < n; i++) { ++ uint64_t off = 0x40000000UL + i * 2 * (bytes + (1 << 20)); ++ ++ runs[i] = (struct speed_run){&speed_vcpus[i], kind, base + off, bytes, &stop}; ++ // A ring through every page in a random order, each load a walk of the page tables. ++ for (uint64_t pages = bytes / PG, p = 0, at = 0; kind == CHASE && p < pages; p++) { ++ static uint32_t order[1 << 16]; ++ ++ if (p == 0) { ++ for (uint64_t k = 0; k < pages; k++) order[k] = k; ++ for (uint64_t k = pages - 1; k > 0; k--) { ++ uint64_t j = rand() % (k + 1), t = order[k]; ++ order[k] = order[j]; order[j] = t; ++ } ++ } ++ at = order[p]; ++ *(uint64_t *)(main_map + off + at * PG) = base + off + order[(p + 1) % pages] * PG; ++ } ++ pthread_create(&t[i], 0, speed_vcpu, &runs[i]); ++ } ++ if (traffic.hz >= 0) pthread_create(&th, 0, traffic_thread, 0); ++ usleep(2500000); ++ atomic_store(&stop, 1); ++ if (traffic.hz >= 0) pthread_join(th, 0); ++ for (int i = 0; i < n; i++) { ++ pthread_join(t[i], 0); ++ total += kind == CHASE ? runs[i].ns / (double)runs[i].done : runs[i].done * 1e3 / runs[i].ns; ++ } ++ return kind == CHASE ? total / n : total; ++} ++ ++static void t_guest_speed(void) { ++ printf("[speed] guest memory on the view and on the main mapping (MB/s all vCPUs, ns a load)\n"); ++ static const char *kinds[] = {"rep movsb", "avx2 copy", "chase"}; ++ struct kvm_userspace_memory_region m = {.slot = 2, .guest_phys_addr = MAIN_GPA, .memory_size = mem_len, ++ .userspace_addr = (uint64_t)main_map}; ++ if (ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &m)) DIE("slot 2"); ++ for (int i = 0; i < 8; i++) vcpu_init(&speed_vcpus[i], 40 + i); ++ traffic.hz = -1; ++ for (int n = 1; n <= 7; n += 6) { ++ for (uint64_t bytes = 8 << 20; bytes <= (256 << 20); bytes <<= 5) { ++ for (int kind = COPY; kind <= CHASE; kind++) { ++ double view = speed(kind, DATA_GPA, n, bytes), main = speed(kind, MAIN_GPA, n, bytes); ++ printf(" %d vCPU %3llu MiB %-9s view %9.2f main %9.2f (%+.1f%%)\n", n, ++ (unsigned long long)(bytes >> 20), kinds[kind], view, main, (view / main - 1) * 100); ++ } ++ } ++ } ++ printf("[speed] 7 vCPUs' avx2 copy of 8 MiB on the view, markers beside them\n"); ++ scattered_surface(22); ++ static const struct { const char *what; int hz, mapped, per_page; } cases[] = { ++ {"no markers", -1, 0, 0}, ++ {"26 Hz", 26, 0, 0}, ++ {"26 Hz, read by a vCPU", 26, 1, 0}, ++ {"flat out", 0, 0, 0}, ++ {"flat out, read by a vCPU", 0, 1, 0}, ++ {"26 Hz, read, a run a page", 26, 1, 1}, ++ }; ++ for (size_t i = 0; i < sizeof(cases) / sizeof(cases[0]); i++) { ++ traffic.hz = cases[i].hz; traffic.mapped = cases[i].mapped; traffic.per_page = cases[i].per_page; ++ double rate = speed(VCOPY, DATA_GPA, 7, 8 << 20); ++ printf(" %-28s %9.1f MB/s, %.0f frames/s\n", cases[i].what, rate, traffic.frames / 2.5); ++ } ++ census_surface_unmapped(22); ++} ++ + static void t_fuse_coherent(struct vcpu *c) { + printf("[fuse] an -EFAULT nothing explains: everything lands, markers off\n"); + surface(11, 0xd00000); +@@ -491,6 +700,8 @@ int main(int argc, char **argv) { + t_coherent_race(mib); + t_arm_window(); + t_coherent_cost(&c); ++ t_scattered(); ++ if (getenv("CENSUS_SPEED")) t_guest_speed(); + t_fuse_coherent(&c); + census_flush(); + printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); +diff --git a/scripts/internal/census/guest/dma.c b/scripts/internal/census/guest/dma.c +index 7c5e6ae1..56097df2 100644 +--- a/scripts/internal/census/guest/dma.c ++++ b/scripts/internal/census/guest/dma.c +@@ -59,11 +59,24 @@ static void io(uint32_t op, uint32_t lba, uint32_t prp1, uint32_t prp2, uint32_t + nvme_cmd(c, IOSQ, &sqt, 1); + if (nvme_done(IOCQ, &cqh, &phase, 1)) die(0x42); + } ++/* With the VM's RAM the BARs lie past this 32-bit guest's reach: `at` is below it. */ ++static uint32_t reachable(int d, uint32_t at) { ++ if (cfg(d, 0x10) & 4) { ++ cfgw(d, 4, cfg(d, 4) & ~2u); ++ cfgw(d, 0x14, 0); ++ } ++ if (!(cfg(d, 0x10) & ~0xfu) || cfg(d, 0x10) >= 0xfec00000u || (cfg(d, 0x10) & 4)) { ++ cfgw(d, 4, cfg(d, 4) & ~2u); ++ cfgw(d, 0x10, at); ++ } ++ return cfg(d, 0x10) & ~0xfu; ++} ++ + static void nvme_up(void) { + phase = phase_a = 1; + for (int d = 0; d < 32 && !nv; d++) + if (cfg(d, 8) >> 8 == 0x010802) { +- nv = cfg(d, 0x10) & ~0xfu; ++ nv = reachable(d, 0xb0000000u); + cfgw(d, 4, cfg(d, 4) | 6); + } + if (!nv) die(0x43); +@@ -127,9 +140,15 @@ static void unmap(uint32_t id) { + + void main_c(void) { + uint32_t i; ++#ifdef FILLMEM ++ /* For a core's size: 1 GiB of the guest's RAM in use, then the host may kill it. */ ++ for (uint32_t p = 0x40000000u; p < 0x80000000u; p += PG) R32(p) = p; ++ outb(0xe9, 'F'); ++ for (;;) __asm__("hlt"); ++#endif + for (int d = 0; d < 32 && !bar; d++) + if (cfg(d, 0) == (0xeeefu << 16 | 0x106b)) { +- bar = cfg(d, 0x10) & ~0xfu; ++ bar = reachable(d, 0xa0000000u); + cfgw(d, 4, cfg(d, 4) | 6); + } + if (!bar) die(0x31); +diff --git a/scripts/internal/census/guest/run-dma.sh b/scripts/internal/census/guest/run-dma.sh +index 1e9fb760..660c1442 100755 +--- a/scripts/internal/census/guest/run-dma.sh ++++ b/scripts/internal/census/guest/run-dma.sh +@@ -9,7 +9,7 @@ O=/srv/warm/census/out/$name RAM=${RAM:-1024} + rm -rf "$O"; mkdir -p "$O" + printf 'surface_lazy=1\nsurface_lazy_scoped=1\nsurface_coherent=1\n' > "$O/live.conf" + # NODEMOTE=1: a surface stays armed however often it traps, or pages rarely stay marked for a device. +-[ -n "${NODEMOTE:-}" ] && printf 'surface_coherent_demote_pct=1000\nsurface_coherent_demote_hz=100000000\nsurface_coherent_hold_s=0\n' >> "$O/live.conf" ++[ -n "${NODEMOTE:-}" ] && printf 'surface_coherent_demote=0\nsurface_coherent_hold_s=0\n' >> "$O/live.conf" + truncate -s 64M "$O/disk.raw" + args=() + for e in "$@"; do args+=(-e "$e"); done diff --git a/patches/reims-vgpu/0017-the-NVMe-boot-test-s-BAR-clear-of-the-Q3.patch b/patches/reims-vgpu/0017-the-NVMe-boot-test-s-BAR-clear-of-the-Q3.patch new file mode 100644 index 0000000..75ed5b5 --- /dev/null +++ b/patches/reims-vgpu/0017-the-NVMe-boot-test-s-BAR-clear-of-the-Q3.patch @@ -0,0 +1,24 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 04:11:12 +0000 +Subject: [PATCH 17/25] the NVMe boot test's BAR clear of the + Q35 MMCONFIG window + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/guest/dma.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +diff --git a/scripts/internal/census/guest/dma.c b/scripts/internal/census/guest/dma.c +index 56097df2..74225868 100644 +--- a/scripts/internal/census/guest/dma.c ++++ b/scripts/internal/census/guest/dma.c +@@ -76,7 +76,7 @@ static void nvme_up(void) { + phase = phase_a = 1; + for (int d = 0; d < 32 && !nv; d++) + if (cfg(d, 8) >> 8 == 0x010802) { +- nv = reachable(d, 0xb0000000u); ++ nv = reachable(d, 0xc0000000u); + cfgw(d, 4, cfg(d, 4) | 6); + } + if (!nv) die(0x43); diff --git a/patches/reims-vgpu/0018-the-census-report-reads-a-build-before-t.patch b/patches/reims-vgpu/0018-the-census-report-reads-a-build-before-t.patch new file mode 100644 index 0000000..adc73cf --- /dev/null +++ b/patches/reims-vgpu/0018-the-census-report-reads-a-build-before-t.patch @@ -0,0 +1,28 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 04:59:01 +0000 +Subject: [PATCH 18/25] the census report reads a build before + the ledger + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/census_report.py | 3 ++- + 1 file changed, 2 insertions(+), 1 deletion(-) + +diff --git a/scripts/internal/census/census_report.py b/scripts/internal/census/census_report.py +index 635a81c0..96167622 100644 +--- a/scripts/internal/census/census_report.py ++++ b/scripts/internal/census/census_report.py +@@ -161,10 +161,11 @@ def coherent(directory, wanted): + f" {r['demotions']:>6} {r['writes']:>6} {cost['detect'] / 1e3:>7.1f}" + f" {cost['sync'] / 1e3:>7.1f} {cost['fetch'] / 1e3:>7.1f} {cost['total'] / 1e3:>7.1f}" + f" {cost['eager_would'] / 1e3:>8.1f} {'yes' if r['pays'] else 'NO'}") ++ costed = [r for r in found.values() if "spent_ns" in r] # a build before the ledger has none + print("per sync in ms (sync_ns holds the copy and the hold), and what spent and saved came to") + print(f"{'surface':>12} {'defers':>7} {'syncs':>6} {'copy':>6} {'hold':>6} {'arm':>6} {'MB/s':>6}" + f" {'spent ms':>9} {'saved ms':>9} synced by") +- for r in sorted(found.values(), key=lambda r: -r["spent_ns"])[:40]: ++ for r in sorted(costed, key=lambda r: -r["spent_ns"])[:40]: + syncs = r["regions"] or 1 + speed = r["bytes_detiled"] / r["copy_ns"] * 1e3 if r["copy_ns"] else 0 + by = " ".join(f"{k}:{n}" for k, n in r["sync_by"].items() if n) diff --git a/patches/reims-vgpu/0019-coherent-hooks-for-the-read-only-held-pages-qemu-55c.patch b/patches/reims-vgpu/0019-coherent-hooks-for-the-read-only-held-pages-qemu-55c.patch new file mode 100644 index 0000000..c75ed50 --- /dev/null +++ b/patches/reims-vgpu/0019-coherent-hooks-for-the-read-only-held-pages-qemu-55c.patch @@ -0,0 +1,210 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 06:33:19 +0000 +Subject: [PATCH 19/25] coherent: hooks for the read-only held pages; qemu + 55c70690f1 + +RvgCoherentHooks gains probe, and open takes a list. The harness now runs with +an in-kernel irqchip and can run a guest with interrupts on: t_write_interrupts_on +hangs on the old census, as the guest did, and the race runs both ways. Tests with +IF=0 and no irqchip took the synchronous path that hid this. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-c/include/reims_vgpu.h | 5 +++- + crates/reims-vgpu-c/src/coherent.rs | 29 ++++++++++++++++++++---- + scripts/internal/census/census_test.c | 12 ++++++++-- + scripts/internal/census/coherent_test.c | 29 +++++++++++++++++++++--- + 4 files changed, 65 insertions(+), 10 deletions(-) + +diff --git a/crates/reims-vgpu-c/include/reims_vgpu.h b/crates/reims-vgpu-c/include/reims_vgpu.h +index 3e4f9ea6..4c23a873 100644 +--- a/crates/reims-vgpu-c/include/reims_vgpu.h ++++ b/crates/reims-vgpu-c/include/reims_vgpu.h +@@ -393,9 +393,12 @@ typedef struct { + // Into up to `room` of `came`, the pages a CPU reached as their markers moved: how many. + size_t (*hold)(void *ctx, const uint64_t *addrs, size_t n, bool open_too, uint64_t *came, + size_t room); +- void (*open)(void *ctx, uint64_t addr); ++ void (*open)(void *ctx, const uint64_t *addrs, size_t n); + void (*freeze)(void *ctx, const uint64_t *addrs, size_t n); + void (*thaw)(void *ctx, const uint64_t *addrs, size_t n); ++ // A write in flight made to name its page: held pages armed and armed pages writable. Refused as ++ // `arm`'s, how many. ++ size_t (*probe)(void *ctx, const uint64_t *addrs, size_t n, uint64_t *refused, size_t room); + } RvgCoherentHooks; + + // The touch census's callbacks. +diff --git a/crates/reims-vgpu-c/src/coherent.rs b/crates/reims-vgpu-c/src/coherent.rs +index 25278ac4..5985496e 100644 +--- a/crates/reims-vgpu-c/src/coherent.rs ++++ b/crates/reims-vgpu-c/src/coherent.rs +@@ -44,9 +44,18 @@ pub struct RvgCoherentHooks { + room: usize, + ) -> usize, + >, +- pub open: Option, ++ pub open: Option, + pub freeze: Option, + pub thaw: Option, ++ pub probe: Option< ++ unsafe extern "C-unwind" fn( ++ ctx: *mut c_void, ++ addrs: *const u64, ++ n: usize, ++ refused: *mut u64, ++ room: usize, ++ ) -> usize, ++ >, + } + + // SAFETY: repr(C), a u32 first; a pointer and optional function pointers. +@@ -139,6 +148,18 @@ mod on { + refused + } + ++ fn probe(&self, pages: &[u64]) -> Vec { ++ let mut refused = vec![0; pages.len()]; ++ let Some(probe) = self.0.probe else { ++ return Vec::new(); ++ }; ++ let (at, count, to) = (pages.as_ptr(), pages.len(), refused.as_mut_ptr()); ++ // SAFETY: as above, with what lives across the call. ++ let found = unsafe { probe(self.0.ctx, at, count, to, count) }; ++ refused.truncate(found); ++ refused ++ } ++ + fn hold(&self, pages: &[u64], open_too: bool) -> Vec { + let mut came = vec![0; pages.len()]; + let Some(hold) = self.0.hold else { +@@ -151,10 +172,10 @@ mod on { + came + } + +- fn open(&self, page: u64) { ++ fn open(&self, pages: &[u64]) { + if let Some(open) = self.0.open { +- // SAFETY: as above. +- unsafe { open(self.0.ctx, page) }; ++ // SAFETY: as above, with what lives across the call. ++ unsafe { open(self.0.ctx, pages.as_ptr(), pages.len()) }; + } + } + +diff --git a/scripts/internal/census/census_test.c b/scripts/internal/census/census_test.c +index 9d70f624..eae3d8ae 100644 +--- a/scripts/internal/census/census_test.c ++++ b/scripts/internal/census/census_test.c +@@ -105,12 +105,19 @@ static void vcpu_init(struct vcpu *c, int id) { + struct kvm_xcrs x = {.nr_xcrs = 1}; + x.xcrs[0].xcr = 0; x.xcrs[0].value = 7; + if (ioctl(c->fd, KVM_SET_XCRS, &x)) DIE("xcrs"); ++ struct kvm_mp_state runnable = {.mp_state = KVM_MP_STATE_RUNNABLE}; // No SIPI comes here. ++ if (ioctl(c->fd, KVM_SET_MP_STATE, &runnable)) DIE("mp state"); + c->vc = census_vcpu(id, c->fd, NULL, read_gpa); + } + ++// RFLAGS for the next setr: as a guest's user code runs, with interrupts on, where KVM may answer a ++// fault it cannot resolve at once by an asynchronous page fault instead of an -EFAULT. ++static uint64_t guest_rflags = 2; ++static unsigned guest_alarm = 60; ++ + static void setr(struct vcpu *c, uint64_t rip, uint64_t rax, uint64_t rbx, uint64_t rcx, uint64_t rdx, + uint64_t rsi, uint64_t rdi) { +- struct kvm_regs r = {.rip = rip, .rflags = 2, .rsp = STACK, .rax = rax, .rbx = rbx, .rcx = rcx, ++ struct kvm_regs r = {.rip = rip, .rflags = guest_rflags, .rsp = STACK, .rax = rax, .rbx = rbx, .rcx = rcx, + .rdx = rdx, .rsi = rsi, .rdi = rdi}; + if (ioctl(c->fd, KVM_SET_REGS, &r)) DIE("set regs"); + } +@@ -128,7 +135,7 @@ static int run_guest(struct vcpu *c, int *mmio_exits) { + sigset_t bus, old; + sigemptyset(&bus); sigaddset(&bus, SIGBUS); + for (;;) { +- alarm(60); ++ alarm(guest_alarm); + pthread_sigmask(SIG_UNBLOCK, &bus, &old); + int r = ioctl(c->fd, KVM_RUN, 0); + int e = r < 0 ? errno : 0; +@@ -504,6 +511,7 @@ static void setup(char **argv, uint64_t mib) { + vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0); + if (vmfd < 0) DIE("vm"); + ioctl(vmfd, KVM_SET_TSS_ADDR, 0xfffbd000); ++ if (ioctl(vmfd, KVM_CREATE_IRQCHIP, 0)) DIE("irqchip"); // QEMU's is in the kernel too. + s0 = mmap(0, S0_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + memcpy(s0 + CODE, g_start, g_end - g_start); + uint64_t *pml4 = (uint64_t *)(s0 + 0x1000), *pdpt = (uint64_t *)(s0 + 0x2000); +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +index 28978dc3..16eb14f0 100644 +--- a/scripts/internal/census/coherent_test.c ++++ b/scripts/internal/census/coherent_test.c +@@ -76,7 +76,7 @@ static void on_fault(void *ctx, uint64_t addr, bool write) { + } + census_coh_hold(&page, 1, false, NULL, 0); + if (write) { +- census_coh_open(page); ++ census_coh_open(&page, 1); + note(&page, 1); + s.writes++; + } +@@ -93,7 +93,7 @@ static void on_probe(void *ctx) { + s.probed_at = now_ns(); + s.probes++; + list_all(list); +- size_t found = census_coh_arm(list, PAGES, list, PAGES); ++ size_t found = census_coh_probe(list, PAGES, list, PAGES); + s.came += found; + note(list, found); + pthread_mutex_unlock(&s.lock); +@@ -194,6 +194,23 @@ static void t_write(struct vcpu *c) { + census_surface_unmapped(2); + } + ++// What macOS's user code does: with interrupts on, a write to a held page. The harness's guest ran ++// with them off, where KVM never tries the fault asynchronously. ++static void t_write_interrupts_on(struct vcpu *c) { ++ printf("[write] with interrupts on a write to a held page still reaches the library\n"); ++ surface(30, 0xe00000); ++ guest_rflags = 0x202; ++ guest_alarm = 5; ++ int writes = s.writes; ++ setr(c, STUB(g_write), 0x7777, gpa(3), 0, 0, 0, 0); ++ int r = run_guest(c, 0); ++ guest_alarm = 60; ++ CHECK(r == RUN_IO && s.writes == writes + 1, "the write trapped (%d)", s.writes - writes); ++ CHECK(*(uint64_t *)(main_map + s.off + 3 * PG) == 0x7777, "and landed"); ++ guest_rflags = 2; ++ census_surface_unmapped(30); ++} ++ + static void t_frozen(struct vcpu *c) { + printf("[frozen] a page poisoned while open stops the guest until it thaws\n"); + surface(3, 0x300000); +@@ -324,7 +341,9 @@ static void t_coherent_race(uint64_t mib) { + atomic_store(&race_stop, 0); + for (int i = 0; i < R + W; i++) { + memset(&g[i], 0, sizeof g[i]); +- vcpu_init(&g[i].c, 20 + i); ++ static int next_vcpu = 20; ++ ++ vcpu_init(&g[i].c, next_vcpu++); + g[i].id = i - R; + pthread_create(&th[i], 0, i < R ? reader : writer, &g[i]); + } +@@ -692,12 +711,16 @@ int main(int argc, char **argv) { + vcpu_init(&c, 0); + t_read(&c); + t_write(&c); ++ t_write_interrupts_on(&c); + t_frozen(&c); + t_kernel_copy(&c); + t_emulator_read(&c); + t_stale_page(&c); + t_unmap_reset(&c); + t_coherent_race(mib); ++ guest_rflags = 0x202; ++ t_coherent_race(mib); ++ guest_rflags = 2; + t_arm_window(); + t_coherent_cost(&c); + t_scattered(); diff --git a/patches/reims-vgpu/0020-census-readback-a-content-check-on-a-guest.patch b/patches/reims-vgpu/0020-census-readback-a-content-check-on-a-guest.patch new file mode 100644 index 0000000..16c3b43 --- /dev/null +++ b/patches/reims-vgpu/0020-census-readback-a-content-check-on-a-guest.patch @@ -0,0 +1,250 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Sun, 1 Nov 2026 08:39:47 +0000 +Subject: [PATCH 20/25] census: readback, a content check on a guest + +Renders, reads and writes real pixels through one texture, a second texture, a +second process and an event between queues, and compares them. It found the +second image's staleness that no lane had. + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/readback.swift | 229 +++++++++++++++++++++++++ + 1 file changed, 229 insertions(+) + create mode 100644 scripts/internal/census/readback.swift + +diff --git a/scripts/internal/census/readback.swift b/scripts/internal/census/readback.swift +new file mode 100644 +index 00000000..a118e062 +--- /dev/null ++++ b/scripts/internal/census/readback.swift +@@ -0,0 +1,229 @@ ++// readback.swift: on a guest, swiftc -O readback.swift -o readback && ./readback. RB_PAUSE_MS waits after each CPU write. ++// readback: GPU render, CPU read and write, and GPU read, through each way macOS reaches a surface. ++// Every step compares real pixels with what must be there. ++import Foundation ++import Metal ++import IOSurface ++ ++let W = 1206, H = 2622 ++let source = """ ++#include ++using namespace metal; ++struct V { float4 p [[position]]; float2 uv; }; ++vertex V vs(uint i [[vertex_id]]) { float2 a[3] = {{-1,-1},{3,-1},{-1,3}}; V o; o.p = float4(a[i],0,1); o.uv = (a[i]+1)/2; o.uv.y = 1-o.uv.y; return o; } ++fragment float4 grad(V i [[stage_in]], constant float4 &c [[buffer(0)]]) { return float4(i.uv.x*c.x, i.uv.y*c.y, c.z, c.w); } ++fragment float4 copy(V i [[stage_in]], texture2d t [[texture(0)]]) { constexpr sampler s(filter::nearest); return t.sample(s, i.uv); } ++""" ++let dev = MTLCopyAllDevices().first! ++let lib = try! dev.makeLibrary(source: source, options: nil) ++func pipe(_ f: String) -> MTLRenderPipelineState { ++ let d = MTLRenderPipelineDescriptor() ++ d.vertexFunction = lib.makeFunction(name: "vs"); d.fragmentFunction = lib.makeFunction(name: f) ++ d.colorAttachments[0].pixelFormat = .bgra8Unorm ++ return try! dev.makeRenderPipelineState(descriptor: d) ++} ++let gradP = pipe("grad"), copyP = pipe("copy") ++let queue = dev.makeCommandQueue()! ++ ++func makeSurface() -> IOSurface { ++ IOSurface(properties: [.width: W, .height: H, .bytesPerElement: 4, .pixelFormat: 0x42475241, ++ IOSurfacePropertyKey(rawValue: "IOSurfaceIsGlobal"): true])! ++} ++func texture(_ s: IOSurface) -> MTLTexture { ++ let d = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm, width: W, height: H, mipmapped: false) ++ d.usage = [.renderTarget, .shaderRead]; d.storageMode = .managed ++ return dev.makeTexture(descriptor: d, iosurface: s, plane: 0)! ++} ++func gradient(_ t: MTLTexture, _ c: [Float]) { ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = t ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let b = queue.makeCommandBuffer()!, e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(gradP); var c = c; e.setFragmentBytes(&c, length: 16, index: 0) ++ e.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3); e.endEncoding() ++ b.commit(); b.waitUntilCompleted() ++} ++/// Samples `t` into a new surface and answers with that surface: the GPU's view of `t`. ++func viaGPU(_ t: MTLTexture) -> IOSurface { ++ let out = makeSurface(), o = texture(out) ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = o ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let b = queue.makeCommandBuffer()!, e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(copyP); e.setFragmentTexture(t, index: 0) ++ e.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3); e.endEncoding() ++ b.commit(); b.waitUntilCompleted() ++ return out ++} ++func px(_ s: IOSurface, _ x: Int, _ y: Int) -> [UInt8] { ++ let p = (s.baseAddress + y * s.bytesPerRow + x * 4).assumingMemoryBound(to: UInt8.self) ++ return [p[0], p[1], p[2], p[3]] ++} ++/// BGRA of the gradient `c` at x, y. ++func want(_ c: [Float], _ x: Int, _ y: Int) -> [UInt8] { ++ let u = (Float(x) + 0.5) / Float(W), v = (Float(y) + 0.5) / Float(H) ++ let f = [c[2], v * c[1], u * c[0], c[3]] ++ return f.map { UInt8(max(0, min(255, ($0 * 255).rounded()))) } ++} ++var failed = 0 ++func check(_ name: String, _ s: IOSurface, _ expect: (Int, Int) -> [UInt8]) { ++ s.lock(options: .readOnly, seed: nil) ++ var bad = 0, total = 0, first = "" ++ for y in stride(from: 3, to: H, by: 97) { ++ for x in stride(from: 5, to: W, by: 61) { ++ let g = px(s, x, y), e = expect(x, y) ++ total += 1 ++ if zip(g, e).contains(where: { abs(Int($0) - Int($1)) > 2 }) { ++ bad += 1 ++ if first.isEmpty { first = "(\(x),\(y)) got \(g) want \(e)" } ++ } ++ } ++ } ++ s.unlock(options: .readOnly, seed: nil) ++ print(bad == 0 ? " ok " : " BAD ", name, "\(total - bad)/\(total)", first) ++ if bad != 0 { failed += 1 } ++} ++/// Sets alpha to 0xff on every texel, as vImageOverwriteChannels does. ++let pauseMs = UInt32(ProcessInfo.processInfo.environment["RB_PAUSE_MS"] ?? "0")! ++func settle() { if pauseMs > 0 { usleep(pauseMs * 1000) } } ++func cpuAlpha(_ s: IOSurface) { ++ defer { settle() } ++ s.lock(options: [], seed: nil) ++ for y in 0.. 2, CommandLine.arguments[1] == "reader" { ++ // A second process: the compositor. "read": samples the surface and prints its corners' texels. ++ let s = IOSurfaceLookup(UInt32(CommandLine.arguments[2])!)! as IOSurface ++ let t = texture(s) ++ while let line = readLine() { ++ if line == "quit" { break } ++ let o = viaGPU(t) ++ o.lock(options: .readOnly, seed: nil) ++ let a = px(o, 5, 3), b = px(o, 1200, 2600), c = px(o, 600, 1300) ++ o.unlock(options: .readOnly, seed: nil) ++ print("\(a) \(b) \(c)"); fflush(stdout) ++ } ++ exit(0) ++} ++ ++let A: [Float] = [1, 1, 0.25, 0.5], B: [Float] = [0.5, 1, 0.75, 1] ++print("device:", dev.name, "unified:", dev.hasUnifiedMemory) ++ ++print("[1] GPU render, CPU read") ++do { let s = makeSurface(), t = texture(s); gradient(t, A); check("pixels", s) { want(A, $0, $1) } } ++ ++print("[2] GPU render, CPU alpha fill, GPU read through the same texture") ++do { ++ let s = makeSurface(), t = texture(s) ++ gradient(t, B); gradient(t, A); cpuAlpha(s) ++ let expect: (Int, Int) -> [UInt8] = { var w = want(A, $0, $1); w[3] = 255; return w } ++ check("the CPU's view", s, expect) ++ check("the GPU's view", viaGPU(t), expect) ++} ++ ++print("[3] the same, read through a second texture used before it all") ++do { ++ let s = makeSurface(), t = texture(s), other = texture(s) ++ gradient(t, B); _ = viaGPU(other) ++ gradient(t, A); cpuAlpha(s) ++ let expect: (Int, Int) -> [UInt8] = { var w = want(A, $0, $1); w[3] = 255; return w } ++ check("the second texture's view", viaGPU(other), expect) ++ check("then the first's", viaGPU(t), expect) ++} ++ ++print("[4] a surface the GPU drew, filled by the CPU alone, read by both textures") ++do { ++ let s = makeSurface(), t = texture(s), other = texture(s) ++ gradient(t, B); _ = viaGPU(other); _ = viaGPU(t) ++ cpuFill(s, 0xff336699) ++ check("the CPU's view", s) { _, _ in [0x99, 0x66, 0x33, 0xff] } ++ check("the first texture's view", viaGPU(t)) { _, _ in [0x99, 0x66, 0x33, 0xff] } ++ check("the second's", viaGPU(other)) { _, _ in [0x99, 0x66, 0x33, 0xff] } ++} ++ ++print("[5] another process's texture, used before it all") ++do { ++ let s = makeSurface(), t = texture(s) ++ gradient(t, B) ++ let p = Process(), to = Pipe(), from = Pipe() ++ p.executableURL = URL(fileURLWithPath: CommandLine.arguments[0]) ++ p.arguments = ["reader", String(IOSurfaceGetID(s))] ++ p.standardInput = to; p.standardOutput = from ++ try! p.run() ++ func ask() -> String { ++ to.fileHandleForWriting.write("read\n".data(using: .utf8)!) ++ var line = Data() ++ while true { let b = from.fileHandleForReading.readData(ofLength: 1); if b.isEmpty || b[0] == 10 { break }; line.append(b) } ++ return String(data: line, encoding: .utf8)! ++ } ++ func expectLine(_ c: (Int, Int) -> [UInt8]) -> String { "\(c(5, 3)) \(c(1200, 2600)) \(c(600, 1300))" } ++ func same(_ a: String, _ b: String) -> Bool { ++ let n = { (s: String) in s.split(whereSeparator: { !$0.isNumber }).map { Int($0)! } } ++ return zip(n(a), n(b)).allSatisfy { abs($0 - $1) <= 2 } && n(a).count == n(b).count ++ } ++ func look(_ name: String, _ c: @escaping (Int, Int) -> [UInt8]) { ++ let got = ask(), e = expectLine(c), ok = same(got, e) ++ print(ok ? " ok " : " BAD ", name, ok ? "" : "got \(got) want \(e)") ++ if !ok { failed += 1 } ++ } ++ look("after the first render") { want(B, $0, $1) } ++ gradient(t, A) ++ let aa: (Int, Int) -> [UInt8] = { want(A, $0, $1) } ++ look("after a second render", aa) ++ gradient(t, B); cpuAlpha(s) ++ look("render, CPU alpha fill") { var w = want(B, $0, $1); w[3] = 255; return w } ++ cpuFill(s, 0xff336699) ++ look("CPU fill alone") { _, _ in [0x99, 0x66, 0x33, 0xff] } ++ gradient(t, A); cpuAlpha(s); _ = viaGPU(t) ++ look("render, fill, the writer reads, then this one") { var w = want(A, $0, $1); w[3] = 255; return w } ++ to.fileHandleForWriting.write("quit\n".data(using: .utf8)!) ++ p.waitUntilExit() ++} ++func heavyGradient(_ b: MTLCommandBuffer, _ t: MTLTexture, _ c: [Float], _ draws: Int) { ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = t ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(gradP); var c = c; e.setFragmentBytes(&c, length: 16, index: 0) ++ for _ in 0.. MTLCommandBuffer { ++ let b = q.makeCommandBuffer()! ++ b.encodeWaitForEvent(ev, value: value) ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = out ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(copyP); e.setFragmentTexture(t, index: 0) ++ e.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3); e.endEncoding() ++ return b ++} ++let q2 = dev.makeCommandQueue()! ++for (name, firstReader, draws) in [("read commits before the write", true, 1), ("read commits as the write runs", false, 400)] { ++ print("[6] two queues, an event orders the read after the write:", name) ++ for second in [false, true] { ++ let s = makeSurface(), t = texture(s), other = texture(s), out = makeSurface(), o = texture(out) ++ gradient(t, B); _ = viaGPU(t); _ = viaGPU(other) ++ usleep(5000) ++ let ev = dev.makeSharedEvent()! ++ let reader = eventRead(q2, second ? other : t, o, ev, 1) ++ let writer = queue.makeCommandBuffer()! ++ heavyGradient(writer, t, A, draws) ++ writer.encodeSignalEvent(ev, value: 1) ++ if firstReader { reader.commit(); usleep(2000); writer.commit() } else { writer.commit(); reader.commit() } ++ writer.waitUntilCompleted(); reader.waitUntilCompleted() ++ check(second ? "through the second texture" : "through the same texture", out) { want(A, $0, $1) } ++ } ++} ++print(failed == 0 ? "ALL MATCH" : "\(failed) MISMATCHED") diff --git a/patches/reims-vgpu/0021-census-failing-tests-for-a-surface-page-outside-the-.patch b/patches/reims-vgpu/0021-census-failing-tests-for-a-surface-page-outside-the-.patch new file mode 100644 index 0000000..1f2ce49 --- /dev/null +++ b/patches/reims-vgpu/0021-census-failing-tests-for-a-surface-page-outside-the-.patch @@ -0,0 +1,112 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 02:16:32 +0000 +Subject: [PATCH 21/25] census: failing tests for a surface page outside the + texture's rows + +A write by the guest CPU fuses the whole census; a DMA write untracks the surface while +its armed pages are ahead. + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/coherent_test.c | 52 ++++++++++++++++++++++++- + 1 file changed, 50 insertions(+), 2 deletions(-) + +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +index 16eb14f0..e9744270 100644 +--- a/scripts/internal/census/coherent_test.c ++++ b/scripts/internal/census/coherent_test.c +@@ -27,6 +27,11 @@ static struct { + + static uint64_t page_at(int i) { return s.base + i * PG; } + ++// AUDIT: the pages the library names. Coherent::all_pages() and find() cover Layout.range(), the ++// texture's rows, while census_coh_track holds every page of the census surface: past the rows of ++// a 3200x1800 surface whose allocation has 25 more pages, a plane's neighbour in a biplanar one. ++static int named = PAGES; ++ + static void note(const uint64_t *pages, size_t n) { + for (size_t i = 0; i < n; i++) s.dirty |= 1u << ((pages[i] - s.base) / PG); + } +@@ -60,6 +65,11 @@ static void sync_all_locked(void) { + static void on_fault(void *ctx, uint64_t addr, bool write) { + uint64_t page = addr & ~(PG - 1); + ++ // AUDIT: as shadow_coherent.rs's free fault(): no surface's range() holds it, so it untracks. ++ if ((page - s.base) / PG >= (uint64_t)named) { ++ census_coh_untrack(addr); ++ return; ++ } + pthread_mutex_lock(&s.lock); + s.faults++; + sync_all_locked(); +@@ -68,7 +78,7 @@ static void on_fault(void *ctx, uint64_t addr, bool write) { + + s.probing = 0; + list_all(list); +- size_t found = census_coh_hold(list, PAGES, false, came, PAGES); ++ size_t found = census_coh_hold(list, named, false, came, PAGES); + s.came += found; + note(came, found); + } else if (write && s.probing) { +@@ -93,7 +103,7 @@ static void on_probe(void *ctx) { + s.probed_at = now_ns(); + s.probes++; + list_all(list); +- size_t found = census_coh_probe(list, PAGES, list, PAGES); ++ size_t found = census_coh_probe(list, named, list, PAGES); + s.came += found; + note(list, found); + pthread_mutex_unlock(&s.lock); +@@ -699,6 +709,39 @@ static void t_fuse_coherent(struct vcpu *c) { + CHECK(plain && s.faults == faults, "and the guest writes every page with no marker left"); + } + ++// AUDIT: a guest CPU write to a page of the surface that the library never named. ++static void t_outside_write(struct vcpu *c) { ++ printf("[outside] a guest write to a held page past the texture's rows\n"); ++ named = PAGES - 1; ++ surface(12, 0xe00000); ++ CHECK(state(PAGES - 1) == 2, "census_coh_track held it too"); ++ int abandons = s.abandons, probes = s.probes; ++ uint64_t start = now_ns(); ++ setr(c, STUB(g_write), 0x7777, gpa(PAGES - 1), 0, 0, 0, 0); ++ int run = run_guest(c, 0); ++ printf(" %.1f ms, %d probes\n", (now_ns() - start) / 1e6, s.probes - probes); ++ CHECK(run == RUN_IO, "the write ran"); ++ CHECK(*(uint64_t *)(main_map + s.off + (PAGES - 1) * PG) == 0x7777, "into the page"); ++ CHECK(s.abandons == abandons, "and the census stayed on: every surface is still coherent"); ++ named = PAGES; ++} ++ ++// AUDIT: a device's DMA into that page, then a guest read of a page the image is ahead of. ++static void t_outside_dma(struct vcpu *c) { ++ printf("[outside] DMA into a held page past the texture's rows, with page 0 armed\n"); ++ named = PAGES - 1; ++ surface(13, 0xe80000); ++ *(uint64_t *)(main_map + s.off) = 0x1; ++ completed(1, 0x8888); ++ CHECK(state(0) == 1, "page 0 is armed: the image is ahead"); ++ census_coherent_touch(main_map + s.off + (PAGES - 1) * PG, 8, true); ++ CHECK(state(0) == 1, "and stays armed after the DMA"); ++ setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); ++ CHECK(run_guest(c, 0) == RUN_IO && getr(c).rax == 0x8888, "so the guest reads the image's word"); ++ printf(" the guest read %#llx\n", (unsigned long long)getr(c).rax); ++ named = PAGES; ++} ++ + int main(int argc, char **argv) { + setvbuf(stdout, 0, _IOLBF, 0); + if (argc < 2) return 2; +@@ -709,6 +752,11 @@ int main(int argc, char **argv) { + census_coherent_set_ops(&ops); + struct vcpu c; + vcpu_init(&c, 0); ++ if (getenv("CENSUS_OUTSIDE")) { ++ if (atoi(getenv("CENSUS_OUTSIDE")) == 1) t_outside_write(&c); else t_outside_dma(&c); ++ printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); ++ return failures != 0; ++ } + t_read(&c); + t_write(&c); + t_write_interrupts_on(&c); diff --git a/patches/reims-vgpu/0022-Coherent-tell-trapping-surfaces-of-host-reads-and-wr.patch b/patches/reims-vgpu/0022-Coherent-tell-trapping-surfaces-of-host-reads-and-wr.patch new file mode 100644 index 0000000..b121f45 --- /dev/null +++ b/patches/reims-vgpu/0022-Coherent-tell-trapping-surfaces-of-host-reads-and-wr.patch @@ -0,0 +1,907 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 06:03:53 +0000 +Subject: [PATCH 22/25] Coherent: tell trapping surfaces of host reads and + writes, test B1, add guest probes + +- P1: the video devices read and write guest surfaces outside the runtime + (lends, and the dma-buf a decoder or the scaler binds in place), which + bypassed the traps: a host read missed GPU writes still in the image, a + host write never reached it. lend_surface_bytes{,_for_reading} and the new + GuestSurface::device_reads/device_writes (decoder frame writes, scaler + source and destination in place) now call shadow_coherent::host_reads / + host_writes over the surface's view and each run of its pages. +- B1: vendor/qemu census tracks the rows as ranges (93f37d5d3a); + coherent_test.c tracks only the rows, and t_outside_write, t_outside_dma + and the new t_planes run in the full harness (CENSUS_OUTSIDE=1/2/3 alone). +- Guest probes for the Mac (untested, no swiftc here): probe_encode (P1), + probe_untracked (P2), probe_signal, probe_tail (B1), probe_long. +- Cargo [patch] points at the coherent-b1 m2v worktree. + +Co-Authored-By: Claude Opus 5.5 +--- + .../reims-vgpu-video-core/src/host/decoder.rs | 2 + + crates/reims-vgpu-video-core/src/surfaces.rs | 15 ++ + crates/reims-vgpu-vulkan/src/os.rs | 5 +- + crates/reims-vgpu-vulkan/src/os/surface.rs | 48 ++++++ + crates/reims-vgpu/src/platform.rs | 8 + + crates/reims-vgpu/src/platform/vulkan.rs | 4 + + crates/reims-vgpu/src/scaler/gpu.rs | 3 + + crates/reims-vgpu/src/video.rs | 8 + + scripts/internal/census/coherent_test.c | 59 +++++-- + scripts/internal/census/probe_common.swift | 110 ++++++++++++ + scripts/internal/census/probe_encode.swift | 161 ++++++++++++++++++ + scripts/internal/census/probe_long.swift | 60 +++++++ + scripts/internal/census/probe_signal.swift | 43 +++++ + scripts/internal/census/probe_tail.swift | 52 ++++++ + scripts/internal/census/probe_untracked.swift | 55 ++++++ + 15 files changed, 622 insertions(+), 11 deletions(-) + create mode 100644 scripts/internal/census/probe_common.swift + create mode 100644 scripts/internal/census/probe_encode.swift + create mode 100644 scripts/internal/census/probe_long.swift + create mode 100644 scripts/internal/census/probe_signal.swift + create mode 100644 scripts/internal/census/probe_tail.swift + create mode 100644 scripts/internal/census/probe_untracked.swift + +diff --git a/crates/reims-vgpu-video-core/src/host/decoder.rs b/crates/reims-vgpu-video-core/src/host/decoder.rs +index 31ba3f5e..a122c714 100644 +--- a/crates/reims-vgpu-video-core/src/host/decoder.rs ++++ b/crates/reims-vgpu-video-core/src/host/decoder.rs +@@ -1963,10 +1963,12 @@ impl ZeroCopy { + } + }; + let writing = clock.now(); ++ held.surface.device_writes(); + let written = frame.write_to_guest(&target); + parts.device += clock.elapsed_since(writing); + match written { + Ok(DeviceWrite::Written) => { ++ held.surface.device_writes(); + parts.written += 1; + Some(Ok(())) + } +diff --git a/crates/reims-vgpu-video-core/src/surfaces.rs b/crates/reims-vgpu-video-core/src/surfaces.rs +index be3fd87d..32e75099 100644 +--- a/crates/reims-vgpu-video-core/src/surfaces.rs ++++ b/crates/reims-vgpu-video-core/src/surfaces.rs +@@ -61,6 +61,21 @@ pub trait GuestSurface: Send + Sync { + fn device_memory(&self) -> Option<&dyn DeviceMemory> { + None + } ++ ++ /// LOCAL (Atlas): a device is about to read the surface's ++ /// [`GuestSurface::device_memory`] with no lend (the scaler's source in ++ /// place), so the host makes what its GPU wrote over the same pages ++ /// reach them first. Nothing, the default, for a host without such ++ /// writes. ++ fn device_reads(&self) {} ++ ++ /// LOCAL (Atlas): a device writes, or has just written, the surface's ++ /// [`GuestSurface::device_memory`] with no lend (a decoder's frame, the ++ /// scaler's destination in place), so the host's GPU takes the bytes ++ /// there as new at its next use of them. Called before the write and ++ /// again after it. Nothing, the default, for a host without such ++ /// copies. ++ fn device_writes(&self) {} + } + + /// A surface's layout. +diff --git a/crates/reims-vgpu-vulkan/src/os.rs b/crates/reims-vgpu-vulkan/src/os.rs +index dc71a67c..e73137b4 100644 +--- a/crates/reims-vgpu-vulkan/src/os.rs ++++ b/crates/reims-vgpu-vulkan/src/os.rs +@@ -47,7 +47,10 @@ pub use host_os::{ + pub(crate) use host_os::{PAGE, mapped, writable}; + #[cfg(all(test, target_os = "macos"))] + pub(crate) use macos::MATRIX_KEY; +-pub use surface::{PlaneLayout, lend_surface_bytes, lend_surface_bytes_for_reading, plane_layout}; ++pub use surface::{ ++ PlaneLayout, lend_surface_bytes, lend_surface_bytes_for_reading, plane_layout, ++ surface_device_access, ++}; + pub use wsi::{add_window_extensions, present_fences}; + + #[cfg(test)] +diff --git a/crates/reims-vgpu-vulkan/src/os/surface.rs b/crates/reims-vgpu-vulkan/src/os/surface.rs +index 0e8aafe2..af2d644b 100644 +--- a/crates/reims-vgpu-vulkan/src/os/surface.rs ++++ b/crates/reims-vgpu-vulkan/src/os/surface.rs +@@ -361,6 +361,7 @@ pub fn lend_surface_bytes( + let made = recorded(surface, |made| (made.pages.clone(), made.offset)); + let offset = made.as_ref().map_or(0, |(_, offset)| *offset); + let length = surface.alloc_size().saturating_sub(offset); ++ host_meets(&made, base, offset, length, rt::shadow_coherent::host_writes); + let _lend = LENDING.lock(); + let pages = match made { + Some((pages, _)) if !pages.is_empty() => pages, +@@ -444,6 +445,7 @@ pub fn lend_surface_bytes_for_reading( + let made = recorded(surface, |made| (made.pages.clone(), made.offset)); + let offset = made.as_ref().map_or(0, |(_, offset)| *offset); + let length = surface.alloc_size().saturating_sub(offset); ++ host_meets(&made, base, offset, length, rt::shadow_coherent::host_reads); + let _lend = LENDING.lock(); + let pages = match made { + Some((pages, _)) if !pages.is_empty() => pages, +@@ -519,6 +521,43 @@ impl Lent { + } + } + ++/// LOCAL (Atlas): tells the runtime's trapping surfaces (`shadow_coherent`) that a host outside ++/// it is about to read or write `surface`'s `length` bytes (`notice` is `host_reads` or ++/// `host_writes`), so a read sees the GPU's writes copied back first and a write reaches the ++/// image the next command uses. A trapping surface lies over the buffer a texture on it is over, ++/// which is either the run of the guest's pages the bytes lie in or the surface's own view of ++/// them at `base`, so both are named; a surface that meets neither costs a scan of the tracked ++/// surfaces, none when nothing traps. ++/// LOCAL (Atlas): a device is about to read (`write` false) or write `surface`'s bytes as a lend ++/// would lend them, through memory lent to it outside a lend (a dma-buf of its pages): the ++/// trapping surfaces over them are told, as [`lend_surface_bytes`] tells them. ++pub fn surface_device_access(surface: &OsSurface, write: bool) { ++ let base = surface.base_address().as_ptr::().addr().to_u64(); ++ if surface.protection_options() != 0 || base == 0 { ++ return; ++ } ++ let made = recorded(surface, |made| (made.pages.clone(), made.offset)); ++ let offset = made.as_ref().map_or(0, |(_, offset)| *offset); ++ let length = surface.alloc_size().saturating_sub(offset); ++ let notice = if write { rt::shadow_coherent::host_writes } else { rt::shadow_coherent::host_reads }; ++ host_meets(&made, base, offset, length, notice); ++} ++ ++fn host_meets( ++ made: &Option<(Vec>, u64)>, ++ base: u64, ++ offset: u64, ++ length: u64, ++ mut notice: impl FnMut(std::ops::Range), ++) { ++ notice(base..base.saturating_add(length)); ++ if let Some((pages, _)) = made { ++ for run in in_runs(pages, &(offset..offset.saturating_add(length))) { ++ notice(run); ++ } ++ } ++} ++ + /// Where a lend of `surface`'s `length` bytes, which lie in the runs of the guest's pages + /// `held`, reaches them ([`lend_surface_bytes`]). + fn lent_where(surface: &OsSurface, held: &[std::ops::Range], length: u64) -> Lent { +@@ -781,6 +820,15 @@ pub(crate) mod tests { + ..SurfaceSpec::default() + }; + let s = OsSurfaces::new().create(&spec).unwrap(); ++ let mut noticed = Vec::new(); ++ let made = recorded(&s, |made| (made.pages.clone(), made.offset)); ++ let base = s.base_address().as_ptr::().addr().to_u64(); ++ host_meets(&made, base, 0x40, 2 * page - 0x40, |r| noticed.push(r)); ++ assert_eq!( ++ noticed, ++ [base..base + 2 * page - 0x40, high + 0x40..high + page, low..low + page], ++ "a trapping surface is told of the view and of each run of pages the bytes lie in" ++ ); + let split = (page - 0x40).to_usize(); + let (mut before, mut at, mut landed) = (0, 0, 0); + lend_surface_bytes(&s, &mut |b| { +diff --git a/crates/reims-vgpu/src/platform.rs b/crates/reims-vgpu/src/platform.rs +index 7089be97..133264a7 100644 +--- a/crates/reims-vgpu/src/platform.rs ++++ b/crates/reims-vgpu/src/platform.rs +@@ -108,6 +108,14 @@ pub trait ApiPlatform: Platform { + let _ = surface; + Err("this platform lends no surface's memory to devices") + } ++ ++ /// LOCAL (Atlas): a device reads (`write` false) or writes `surface`'s memory as ++ /// [`ApiPlatform::surface_device_memory`] lent it, outside any lend: the platform's GPU, ++ /// whose textures over the same pages may trap (`shadow_coherent`), is told. Nothing by ++ /// default. ++ fn surface_device_access(&self, surface: &Self::Surface, write: bool) { ++ let _ = (surface, write); ++ } + } + + /// The API's state for one platform instance. +diff --git a/crates/reims-vgpu/src/platform/vulkan.rs b/crates/reims-vgpu/src/platform/vulkan.rs +index b710f9ed..c02aee32 100644 +--- a/crates/reims-vgpu/src/platform/vulkan.rs ++++ b/crates/reims-vgpu/src/platform/vulkan.rs +@@ -94,6 +94,10 @@ impl ApiPlatform for VulkanPlatform { + reims_vgpu_vulkan::os::lend_surface_bytes_for_reading(surface, read) + } + ++ fn surface_device_access(&self, surface: &::Surface, write: bool) { ++ reims_vgpu_vulkan::os::surface_device_access(surface, write); ++ } ++ + /// On Linux, the bytes `lend_surface_bytes` lends, as a dma-buf of their pages made by + /// udmabuf and as the surface's view of them, with an identity attached to the surface + /// (`platform/vulkan/lent_memory.rs`). On macOS none: there the Metal platform lends a +diff --git a/crates/reims-vgpu/src/scaler/gpu.rs b/crates/reims-vgpu/src/scaler/gpu.rs +index 1cca3353..fdd032cb 100644 +--- a/crates/reims-vgpu/src/scaler/gpu.rs ++++ b/crates/reims-vgpu/src/scaler/gpu.rs +@@ -445,6 +445,7 @@ impl GpuAccelerator { + return Err(io_return::BAD_ARGUMENT); + } + let planes = at_base(&src.planes, base).ok_or(io_return::UNSUPPORTED)?; ++ job.source.device_reads(); + return Ok(Bound { + buffer, + planes, +@@ -507,6 +508,7 @@ impl GpuAccelerator { + } + let planes = at_base(&dst.planes, base).ok_or(io_return::UNSUPPORTED)?; + if planes.iter().all(BoundPlane::word_aligned) { ++ job.destination.device_writes(); + return Ok(Bound { + buffer, + planes, +@@ -750,6 +752,7 @@ impl Accelerator for GpuAccelerator { + let staged = destination.staged; + let ran = self.run(job, source, destination); + if !staged { ++ job.destination.device_writes(); + return ran.err().unwrap_or(io_return::SUCCESS); + } + // Staged: the destination's lend says whether it can be written, then what the kernel +diff --git a/crates/reims-vgpu/src/video.rs b/crates/reims-vgpu/src/video.rs +index 910fd69a..dbb06ae6 100644 +--- a/crates/reims-vgpu/src/video.rs ++++ b/crates/reims-vgpu/src/video.rs +@@ -240,6 +240,14 @@ impl GuestSurface for MappedSurface

{ + /// asked for once for this hold: the bytes [`Self::write`] lends, for the decoders' device to + /// write a frame into while the session holds the surface. `None` where the platform lends + /// none, said once a hold at debug. ++ fn device_reads(&self) { ++ self.platform.surface_device_access(&self.surface, false); ++ } ++ ++ fn device_writes(&self) { ++ self.platform.surface_device_access(&self.surface, true); ++ } ++ + fn device_memory(&self) -> Option<&dyn DeviceMemory> { + let lent = self.device.get_or_init(|| { + entry::run(|_| self.platform.surface_device_memory(&self.surface)) +diff --git a/scripts/internal/census/coherent_test.c b/scripts/internal/census/coherent_test.c +index e9744270..9147c35d 100644 +--- a/scripts/internal/census/coherent_test.c ++++ b/scripts/internal/census/coherent_test.c +@@ -27,9 +27,9 @@ static struct { + + static uint64_t page_at(int i) { return s.base + i * PG; } + +-// AUDIT: the pages the library names. Coherent::all_pages() and find() cover Layout.range(), the +-// texture's rows, while census_coh_track holds every page of the census surface: past the rows of +-// a 3200x1800 surface whose allocation has 25 more pages, a plane's neighbour in a biplanar one. ++// The pages the library names: Coherent::range(), the texture's rows, which it tracks ++// (census_coh_track(start, len)); a 3200x1800 surface's allocation has 25 pages past them, and a ++// biplanar one a plane beside them. The census surface is every page; past `named` none is ours. + static int named = PAGES; + + static void note(const uint64_t *pages, size_t n) { +@@ -65,7 +65,7 @@ static void sync_all_locked(void) { + static void on_fault(void *ctx, uint64_t addr, bool write) { + uint64_t page = addr & ~(PG - 1); + +- // AUDIT: as shadow_coherent.rs's free fault(): no surface's range() holds it, so it untracks. ++ // As shadow_coherent.rs's free fault(): no surface's range() holds it, so it untracks. + if ((page - s.base) / PG >= (uint64_t)named) { + census_coh_untrack(addr); + return; +@@ -137,7 +137,7 @@ static void surface(uint32_t id, uint64_t off) { + s.id = id; s.off = off; s.base = (uint64_t)main_map + off; + memset(s.ahead, 0, sizeof s.ahead); + s.probing = 0; +- if (!census_coh_track(s.base, PAGES * PG)) DIE("track"); ++ if (!census_coh_track(s.base, named * PG)) DIE("track"); + } + + // The GPU wrote `word` over the pages `mask` holds, and the command buffer completed. +@@ -709,12 +709,12 @@ static void t_fuse_coherent(struct vcpu *c) { + CHECK(plain && s.faults == faults, "and the guest writes every page with no marker left"); + } + +-// AUDIT: a guest CPU write to a page of the surface that the library never named. ++// A guest CPU write to a page of the surface that the library never named (audit B1). + static void t_outside_write(struct vcpu *c) { +- printf("[outside] a guest write to a held page past the texture's rows\n"); ++ printf("[outside] a guest write to a page past the texture's rows\n"); + named = PAGES - 1; + surface(12, 0xe00000); +- CHECK(state(PAGES - 1) == 2, "census_coh_track held it too"); ++ CHECK(state(PAGES - 1) == 3 && state(PAGES - 2) == 2, "the census holds the rows' pages alone"); + int abandons = s.abandons, probes = s.probes; + uint64_t start = now_ns(); + setr(c, STUB(g_write), 0x7777, gpa(PAGES - 1), 0, 0, 0, 0); +@@ -723,10 +723,11 @@ static void t_outside_write(struct vcpu *c) { + CHECK(run == RUN_IO, "the write ran"); + CHECK(*(uint64_t *)(main_map + s.off + (PAGES - 1) * PG) == 0x7777, "into the page"); + CHECK(s.abandons == abandons, "and the census stayed on: every surface is still coherent"); ++ census_surface_unmapped(12); + named = PAGES; + } + +-// AUDIT: a device's DMA into that page, then a guest read of a page the image is ahead of. ++// A device's DMA into that page, then a guest read of a page the image is ahead of (audit B1). + static void t_outside_dma(struct vcpu *c) { + printf("[outside] DMA into a held page past the texture's rows, with page 0 armed\n"); + named = PAGES - 1; +@@ -739,6 +740,40 @@ static void t_outside_dma(struct vcpu *c) { + setr(c, STUB(g_read), 0, gpa(0), 0, 0, 0, 0); + CHECK(run_guest(c, 0) == RUN_IO && getr(c).rax == 0x8888, "so the guest reads the image's word"); + printf(" the guest read %#llx\n", (unsigned long long)getr(c).rax); ++ census_surface_unmapped(13); ++ named = PAGES; ++} ++ ++// Two planes of one census surface, each a range of its own, as a biplanar surface's two ++// textures are: each tracks, arms and goes alone, and a range over another is refused. ++static void t_planes(struct vcpu *c) { ++ printf("[planes] two ranges of one surface: one's untrack leaves the other armed\n"); ++ named = PAGES / 2; ++ surface(15, 0x1100000); ++ uint64_t second = s.base + (PAGES / 2) * PG, pages0 = census_coherent_pages; ++ CHECK(!census_coh_track(s.base + PG, PG), "a range inside the first is refused"); ++ CHECK(!census_coh_track(second - PG, 2 * PG), "and one that meets it"); ++ CHECK(census_coh_track(s.base, (PAGES / 2) * PG), "the same range again is tracked already"); ++ CHECK(census_coherent_pages == pages0, "and counts once"); ++ CHECK(census_coh_track(second, (PAGES / 2) * PG), "the second plane tracks"); ++ CHECK(census_coherent_pages == pages0 + PAGES / 2, "its pages count"); ++ CHECK(state(PAGES / 2) == 2 && state(PAGES - 1) == 2, "held"); ++ completed(1, 0x9999); ++ uint64_t at = second; ++ CHECK(census_coh_arm(&at, 1, NULL, 0) == 0 && state(PAGES / 2) == 1, "a page of each armed"); ++ CHECK(state(0) == 1, "the first's too"); ++ census_coh_untrack(s.base); ++ CHECK(state(0) == 3 && state(PAGES / 2 - 1) == 3, "the first plane's pages open"); ++ CHECK(state(PAGES / 2) == 1 && state(PAGES - 1) == 2, "the second's stay as they were"); ++ CHECK(census_coherent_pages == pages0, "the first's pages no longer count"); ++ int faults = s.faults; ++ census_coherent_touch(main_map + s.off, 8, true); ++ CHECK(s.faults == faults, "a device's write to the first plane is no one's now"); ++ CHECK(census_coh_track(s.base, (PAGES / 2) * PG), "and it tracks again"); ++ census_coh_untrack(second); ++ CHECK(state(PAGES / 2) == 3 && state(0) == 2, "the second goes alone"); ++ census_surface_unmapped(15); ++ CHECK(state(0) == 6 && census_coherent_pages == pages0 - PAGES / 2, "the unmap takes the rest"); + named = PAGES; + } + +@@ -753,7 +788,8 @@ int main(int argc, char **argv) { + struct vcpu c; + vcpu_init(&c, 0); + if (getenv("CENSUS_OUTSIDE")) { +- if (atoi(getenv("CENSUS_OUTSIDE")) == 1) t_outside_write(&c); else t_outside_dma(&c); ++ int which = atoi(getenv("CENSUS_OUTSIDE")); ++ if (which == 1) t_outside_write(&c); else if (which == 2) t_outside_dma(&c); else t_planes(&c); + printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); + return failures != 0; + } +@@ -773,6 +809,9 @@ int main(int argc, char **argv) { + t_coherent_cost(&c); + t_scattered(); + if (getenv("CENSUS_SPEED")) t_guest_speed(); ++ t_outside_write(&c); ++ t_outside_dma(&c); ++ t_planes(&c); + t_fuse_coherent(&c); + census_flush(); + printf("%s: %d failed\n", failures ? "FAILED" : "PASSED", failures); +diff --git a/scripts/internal/census/probe_common.swift b/scripts/internal/census/probe_common.swift +new file mode 100644 +index 00000000..8c9a0b3a +--- /dev/null ++++ b/scripts/internal/census/probe_common.swift +@@ -0,0 +1,110 @@ ++// probe_common.swift: shared by probe_untracked, probe_signal, probe_tail and probe_long. Build each as ++// swiftc -O probe_common.swift probe_.swift -o probe_ ++// A GPU gradient renderer, a GPU copy (sample one texture into another surface), and pixel checks ++// at a grid of points, as readback.swift has them, for surfaces of any size. ++import Foundation ++import Metal ++import IOSurface ++ ++let probeSource = """ ++#include ++using namespace metal; ++struct V { float4 p [[position]]; float2 uv; }; ++vertex V vs(uint i [[vertex_id]]) { float2 a[3] = {{-1,-1},{3,-1},{-1,3}}; V o; o.p = float4(a[i],0,1); o.uv = (a[i]+1)/2; o.uv.y = 1-o.uv.y; return o; } ++fragment float4 grad(V i [[stage_in]], constant float4 &c [[buffer(0)]]) { return float4(i.uv.x*c.x, i.uv.y*c.y, c.z, c.w); } ++fragment float4 copy(V i [[stage_in]], texture2d t [[texture(0)]]) { constexpr sampler s(filter::nearest); return t.sample(s, i.uv); } ++""" ++let dev = MTLCopyAllDevices().first! ++let lib = try! dev.makeLibrary(source: probeSource, options: nil) ++func pipe(_ f: String) -> MTLRenderPipelineState { ++ let d = MTLRenderPipelineDescriptor() ++ d.vertexFunction = lib.makeFunction(name: "vs"); d.fragmentFunction = lib.makeFunction(name: f) ++ d.colorAttachments[0].pixelFormat = .bgra8Unorm ++ return try! dev.makeRenderPipelineState(descriptor: d) ++} ++let gradP = pipe("grad"), copyP = pipe("copy") ++let queue = dev.makeCommandQueue()! ++ ++func makeSurface(_ w: Int, _ h: Int, allocSize: Int? = nil) -> IOSurface { ++ var p: [IOSurfacePropertyKey: Any] = [.width: w, .height: h, .bytesPerElement: 4, .pixelFormat: 0x42475241, ++ IOSurfacePropertyKey(rawValue: "IOSurfaceIsGlobal"): true] ++ if let a = allocSize { p[.allocSize] = a } ++ return IOSurface(properties: p)! ++} ++func texture(_ s: IOSurface, untracked: Bool = false) -> MTLTexture { ++ let d = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm, width: s.width, height: s.height, mipmapped: false) ++ d.usage = [.renderTarget, .shaderRead]; d.storageMode = .managed ++ if untracked { d.hazardTrackingMode = .untracked } ++ return dev.makeTexture(descriptor: d, iosurface: s, plane: 0)! ++} ++/// Encodes `draws` draws of the gradient `c` into `t` in `b`. ++func encodeGradient(_ b: MTLCommandBuffer, _ t: MTLTexture, _ c: [Float], draws: Int = 1) { ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = t ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(gradP); var c = c; e.setFragmentBytes(&c, length: 16, index: 0) ++ for _ in 0.. IOSurface { ++ let out = makeSurface(t.width, t.height), o = texture(out) ++ let b = queue.makeCommandBuffer()! ++ encodeCopy(b, t, o); b.commit(); b.waitUntilCompleted() ++ return out ++} ++func px(_ s: IOSurface, _ x: Int, _ y: Int) -> [UInt8] { ++ let p = (s.baseAddress + y * s.bytesPerRow + x * 4).assumingMemoryBound(to: UInt8.self) ++ return [p[0], p[1], p[2], p[3]] ++} ++/// BGRA of the gradient `c` at x, y of a w x h surface. ++func want(_ c: [Float], _ w: Int, _ h: Int, _ x: Int, _ y: Int) -> [UInt8] { ++ let u = (Float(x) + 0.5) / Float(w), v = (Float(y) + 0.5) / Float(h) ++ let f = [c[2], v * c[1], u * c[0], c[3]] ++ return f.map { UInt8(max(0, min(255, ($0 * 255).rounded()))) } ++} ++/// Compares `s` with `expect` at a grid of points, the last row and column among them: ++/// (matched, total, the first mismatch). ++func compare(_ s: IOSurface, lock: Bool = true, step: Int = 37, _ expect: (Int, Int) -> [UInt8]) -> (Int, Int, String) { ++ if lock { s.lock(options: .readOnly, seed: nil) } ++ defer { if lock { s.unlock(options: .readOnly, seed: nil) } } ++ var bad = 0, total = 0, first = "" ++ let xs = Array(stride(from: 1, to: s.width, by: step)) + [s.width - 1] ++ let ys = Array(stride(from: 2, to: s.height, by: step)) + [s.height - 1] ++ for y in ys { ++ for x in xs { ++ let g = px(s, x, y), e = expect(x, y) ++ total += 1 ++ if zip(g, e).contains(where: { abs(Int($0) - Int($1)) > 2 }) { ++ bad += 1 ++ if first.isEmpty { first = "(\(x),\(y)) got \(g) want \(e)" } ++ } ++ } ++ } ++ return (total - bad, total, first) ++} ++var failed = 0 ++func check(_ name: String, _ s: IOSurface, lock: Bool = true, _ expect: (Int, Int) -> [UInt8]) { ++ let (good, total, first) = compare(s, lock: lock, expect) ++ print(good == total ? " ok " : " BAD ", name, "\(good)/\(total)", first) ++ if good != total { failed += 1 } ++} ++func finish(_ extra: String = "") -> Never { ++ print(failed == 0 ? "PASS" : "FAIL", "\(failed) checks failed", extra) ++ exit(failed == 0 ? 0 : 1) ++} ++func env(_ name: String, _ fallback: Double) -> Double { ++ Double(ProcessInfo.processInfo.environment[name] ?? "") ?? fallback ++} +diff --git a/scripts/internal/census/probe_encode.swift b/scripts/internal/census/probe_encode.swift +new file mode 100644 +index 00000000..95a3e8c3 +--- /dev/null ++++ b/scripts/internal/census/probe_encode.swift +@@ -0,0 +1,161 @@ ++// probe_encode.swift (P1): on a guest, swiftc -O probe_encode.swift -o probe_encode && ./probe_encode ++// PROBE_SECONDS (default 180) is how long it runs. ++// The GPU renders a moving frame into an IOSurface of 1920x1088 BGRA (2,088,960 texels), and the ++// frame is encoded right away with VTCompressionSession, decoded, and compared. Nothing on the CPU ++// touches the surface between the render and the encode, so the host's video encoder (outside the ++// runtime) is the first to read it. If the host misses the GPU's writes still held in the texture, ++// the decoded frame is a stale one: its frame number, drawn as 32 big blocks, is wrong. ++// Three surfaces are used in turn, so each is reused every third frame. ++// It prints one line per 100 frames and ends with PASS or FAIL plus numbers. ++import Foundation ++import Metal ++import IOSurface ++import CoreVideo ++import CoreMedia ++import VideoToolbox ++ ++let W = 1920, H = 1088 ++let seconds = Double(ProcessInfo.processInfo.environment["PROBE_SECONDS"] ?? "180")! ++let source = """ ++#include ++using namespace metal; ++struct V { float4 p [[position]]; float2 uv; }; ++vertex V vs(uint i [[vertex_id]]) { float2 a[3] = {{-1,-1},{3,-1},{-1,3}}; V o; o.p = float4(a[i],0,1); o.uv = (a[i]+1)/2; o.uv.y = 1-o.uv.y; return o; } ++// The top 80%: 8x4 blocks, block b white where bit b of the frame number is set. ++// The bottom band: a bar that moves with the frame number, over a gradient. ++fragment float4 frame(V i [[stage_in]], constant uint &n [[buffer(0)]]) { ++ if (i.uv.y < 0.8) { ++ uint bx = min(uint(i.uv.x * 8), 7u), by = min(uint(i.uv.y / 0.8 * 4), 3u); ++ float fx = fract(i.uv.x * 8), fy = fract(i.uv.y / 0.8 * 4); ++ if (fx < 0.1 || fx > 0.9 || fy < 0.1 || fy > 0.9) return float4(0.5, 0.5, 0.5, 1); ++ float on = float((n >> (by * 8 + bx)) & 1u); ++ return float4(on, on, on, 1); ++ } ++ float bar = fract(float(n) / 120.0); ++ float d = abs(i.uv.x - bar); ++ return d < 0.02 ? float4(1, 0, 0, 1) : float4(i.uv.x, 0.3, 1 - i.uv.x, 1); ++} ++""" ++let dev = MTLCopyAllDevices().first! ++let lib = try! dev.makeLibrary(source: source, options: nil) ++let pd = MTLRenderPipelineDescriptor() ++pd.vertexFunction = lib.makeFunction(name: "vs"); pd.fragmentFunction = lib.makeFunction(name: "frame") ++pd.colorAttachments[0].pixelFormat = .bgra8Unorm ++let pipe = try! dev.makeRenderPipelineState(descriptor: pd) ++let queue = dev.makeCommandQueue()! ++ ++struct Slot { let surface: IOSurface; let texture: MTLTexture; let buffer: CVPixelBuffer } ++func makeSlot() -> Slot { ++ let s = IOSurface(properties: [.width: W, .height: H, .bytesPerElement: 4, .pixelFormat: 0x42475241, ++ IOSurfacePropertyKey(rawValue: "IOSurfaceIsGlobal"): true])! ++ let d = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .bgra8Unorm, width: W, height: H, mipmapped: false) ++ d.usage = [.renderTarget, .shaderRead]; d.storageMode = .managed ++ let t = dev.makeTexture(descriptor: d, iosurface: s, plane: 0)! ++ var pb: Unmanaged? ++ let r = CVPixelBufferCreateWithIOSurface(nil, unsafeBitCast(s, to: IOSurfaceRef.self), nil, &pb) ++ precondition(r == kCVReturnSuccess, "CVPixelBufferCreateWithIOSurface: \(r)") ++ return Slot(surface: s, texture: t, buffer: pb!.takeRetainedValue()) ++} ++let slots = [makeSlot(), makeSlot(), makeSlot()] ++ ++func render(_ t: MTLTexture, _ n: UInt32) { ++ let p = MTLRenderPassDescriptor(); p.colorAttachments[0].texture = t ++ p.colorAttachments[0].loadAction = .dontCare; p.colorAttachments[0].storeAction = .store ++ let b = queue.makeCommandBuffer()!, e = b.makeRenderCommandEncoder(descriptor: p)! ++ e.setRenderPipelineState(pipe); var n = n; e.setFragmentBytes(&n, length: 4, index: 0) ++ e.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3); e.endEncoding() ++ b.commit(); b.waitUntilCompleted() ++} ++ ++// Encoder: H.264, real time, no reordering, so each frame's sample comes out in order. ++var encoded: [(UInt32, CMSampleBuffer)] = [] ++let encodedLock = NSLock() ++var session: VTCompressionSession? ++let made = VTCompressionSessionCreate(allocator: nil, width: Int32(W), height: Int32(H), ++ codecType: kCMVideoCodecType_H264, encoderSpecification: nil, imageBufferAttributes: nil, ++ compressedDataAllocator: nil, outputCallback: { _, tag, status, _, sample in ++ guard status == noErr, let sample = sample else { print(" encode status \(status)"); return } ++ encodedLock.lock(); encoded.append((UInt32(UInt(bitPattern: tag)), sample)); encodedLock.unlock() ++ }, refcon: nil, compressionSessionOut: &session) ++precondition(made == noErr, "VTCompressionSessionCreate: \(made)") ++let enc = session! ++VTSessionSetProperty(enc, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue) ++VTSessionSetProperty(enc, key: kVTCompressionPropertyKey_AllowFrameReordering, value: kCFBooleanFalse) ++VTSessionSetProperty(enc, key: kVTCompressionPropertyKey_MaxKeyFrameInterval, value: 30 as CFNumber) ++VTSessionSetProperty(enc, key: kVTCompressionPropertyKey_AverageBitRate, value: 20_000_000 as CFNumber) ++ ++var decoder: VTDecompressionSession? ++func makeDecoder(_ format: CMFormatDescription) { ++ let attrs = [kCVPixelBufferPixelFormatTypeKey as String: kCVPixelFormatType_32BGRA] as CFDictionary ++ var d: VTDecompressionSession? ++ let r = VTDecompressionSessionCreate(allocator: nil, formatDescription: format, decoderSpecification: nil, ++ imageBufferAttributes: attrs, outputCallback: nil, decompressionSessionOut: &d) ++ precondition(r == noErr, "VTDecompressionSessionCreate: \(r)") ++ decoder = d ++} ++/// The frame number read back from a decoded BGRA frame: each block's centre, thresholded. ++func readNumber(_ pb: CVPixelBuffer) -> (Int64, Bool) { ++ CVPixelBufferLockBaseAddress(pb, .readOnly); defer { CVPixelBufferUnlockBaseAddress(pb, .readOnly) } ++ let base = CVPixelBufferGetBaseAddress(pb)!, bpr = CVPixelBufferGetBytesPerRow(pb) ++ let w = CVPixelBufferGetWidth(pb), h = CVPixelBufferGetHeight(pb) ++ var n: Int64 = 0, clean = true ++ for b in 0..<32 { ++ let bx = b % 8, by = b / 8 ++ let x = Int((Double(bx) + 0.5) / 8 * Double(w)), y = Int((Double(by) + 0.5) / 4 * 0.8 * Double(h)) ++ let p = (base + y * bpr + x * 4).assumingMemoryBound(to: UInt8.self) ++ let g = Int(p[1]) ++ if g > 160 { n |= 1 << b } else if g >= 96 { clean = false } ++ } ++ return (n, clean) ++} ++func decode(_ sample: CMSampleBuffer) -> (Int64, Bool)? { ++ if decoder == nil { makeDecoder(CMSampleBufferGetFormatDescription(sample)!) } ++ var got: (Int64, Bool)? ++ let r = VTDecompressionSessionDecodeFrame(decoder!, sampleBuffer: sample, flags: [], infoFlagsOut: nil) { ++ status, _, image, _, _ in ++ if status == noErr, let image = image { got = readNumber(image) } ++ } ++ VTDecompressionSessionWaitForAsynchronousFrames(decoder!) ++ return r == noErr ? got : nil ++} ++ ++print("device:", dev.name, "frames of \(W)x\(H) BGRA for \(Int(seconds)) s") ++let start = Date() ++var n: UInt32 = 1, frames = 0, wrong = 0, undecoded = 0, unclean = 0, firstWrong = "" ++var renderNs: UInt64 = 0, encodeNs: UInt64 = 0 ++while Date().timeIntervalSince(start) < seconds { ++ let slot = slots[Int(n) % slots.count] ++ let t0 = DispatchTime.now().uptimeNanoseconds ++ render(slot.texture, n) ++ let t1 = DispatchTime.now().uptimeNanoseconds ++ let pts = CMTime(value: CMTimeValue(n), timescale: 60) ++ let r = VTCompressionSessionEncodeFrame(enc, imageBuffer: slot.buffer, presentationTimeStamp: pts, ++ duration: .invalid, frameProperties: nil, sourceFrameRefcon: UnsafeMutableRawPointer(bitPattern: UInt(n)), ++ infoFlagsOut: nil) ++ precondition(r == noErr, "VTCompressionSessionEncodeFrame: \(r)") ++ // The encoder reads the surface before this returns; the frame must be out of its ++ // queue before the slot is reused three frames on. ++ VTCompressionSessionCompleteFrames(enc, untilPresentationTimeStamp: pts) ++ let t2 = DispatchTime.now().uptimeNanoseconds ++ renderNs += t1 - t0; encodeNs += t2 - t1 ++ encodedLock.lock(); let out = encoded; encoded.removeAll(); encodedLock.unlock() ++ for (tag, sample) in out { ++ frames += 1 ++ guard let (got, clean) = decode(sample) else { undecoded += 1; continue } ++ if !clean { unclean += 1 } ++ if got != Int64(tag & 0xffff_ffff) { ++ wrong += 1 ++ if firstWrong.isEmpty { firstWrong = "frame \(tag) decoded as \(got)" } ++ } ++ } ++ if n % 100 == 0 { ++ print(" \(n) frames, \(wrong) wrong, render \(renderNs / UInt64(n) / 1000) us, encode \(encodeNs / UInt64(n) / 1000) us") ++ fflush(stdout) ++ } ++ n += 1 ++} ++VTCompressionSessionCompleteFrames(enc, untilPresentationTimeStamp: .invalid) ++let ok = wrong == 0 && undecoded == 0 && frames > 0 ++print(ok ? "PASS" : "FAIL", "frames \(frames) wrong \(wrong) undecoded \(undecoded) unclean-blocks-frames \(unclean)", ++ "render-us \(renderNs / UInt64(max(n, 1)) / 1000) encode-us \(encodeNs / UInt64(max(n, 1)) / 1000)", firstWrong) ++exit(ok ? 0 : 1) +diff --git a/scripts/internal/census/probe_long.swift b/scripts/internal/census/probe_long.swift +new file mode 100644 +index 00000000..760a2c4f +--- /dev/null ++++ b/scripts/internal/census/probe_long.swift +@@ -0,0 +1,60 @@ ++// probe_long.swift (long run): swiftc -O probe_common.swift probe_long.swift -o probe_long ++// PROBE_SECONDS (default 300). Thousands of frames over minutes, each one checked: ++// - Four surfaces are used in turn, so each is reused every fourth frame. ++// - Every 37th frame, a surface's texture is dropped and a new one made on it. ++// - Every 101st frame, a surface is dropped and a new one made, which may land on its pages. ++// - Each frame the GPU renders a gradient whose colour changes every frame. Then the CPU reads it, ++// and the GPU reads it through a second texture. ++// - Every 5th frame the CPU writes a band of rows first, and the GPU's read must see it. ++// It prints one line per 500 frames and ends with PASS or FAIL plus numbers. ++import Foundation ++import Metal ++import IOSurface ++ ++let W = 1600, H = 1000 ++let seconds = env("PROBE_SECONDS", 300) ++struct Slot { var surface: IOSurface; var writer: MTLTexture; var reader: MTLTexture } ++func makeSlot() -> Slot { let s = makeSurface(W, H); return Slot(surface: s, writer: texture(s), reader: texture(s)) } ++var slots = (0..<4).map { _ in makeSlot() } ++func colour(_ n: Int) -> [Float] { ++ [Float(n % 7 + 1) / 8, Float(n % 5 + 1) / 6, Float(n % 11) / 11, 1] ++} ++let band = 40, bandValue: UInt32 = 0xff20_4060 ++print("device:", dev.name, "\(W)x\(H) BGRA for \(Int(seconds)) s") ++let start = Date() ++var n = 0, recreatedTextures = 0, recreatedSurfaces = 0, frameFails = 0, firstFail = "" ++while Date().timeIntervalSince(start) < seconds { ++ let i = n % slots.count ++ if n % 101 == 100 { slots[i] = makeSlot(); recreatedSurfaces += 1 } ++ else if n % 37 == 36 { ++ slots[i].writer = texture(slots[i].surface); slots[i].reader = texture(slots[i].surface); recreatedTextures += 1 ++ } ++ let slot = slots[i], c = colour(n), y0 = (n * 13) % (H - band) ++ gradient(slot.writer, c) ++ let cpuBand = n % 5 == 0 ++ if cpuBand { ++ slot.surface.lock(options: [], seed: nil) ++ for y in y0..<(y0 + band) { ++ let row = (slot.surface.baseAddress + y * slot.surface.bytesPerRow).assumingMemoryBound(to: UInt32.self) ++ for x in 0.. [UInt8] = { x, y in ++ cpuBand && y >= y0 && y < y0 + band ? [0x60, 0x40, 0x20, 0xff] : want(c, W, H, x, y) ++ } ++ let (g1, t1, f1) = compare(slot.surface, step: 53, expect) ++ let (g2, t2, f2) = compare(viaGPU(slot.reader), step: 53, expect) ++ if g1 != t1 || g2 != t2 { ++ frameFails += 1 ++ if firstFail.isEmpty { firstFail = "frame \(n) slot \(i): cpu \(g1)/\(t1) \(f1) gpu \(g2)/\(t2) \(f2)" } ++ } ++ n += 1 ++ if n % 500 == 0 { ++ print(" \(n) frames, \(frameFails) failed, \(recreatedTextures) textures and \(recreatedSurfaces) surfaces made again", ++ String(format: "%.1f fps", Double(n) / Date().timeIntervalSince(start))) ++ fflush(stdout) ++ } ++} ++if frameFails != 0 { failed += 1; print(" first:", firstFail) } ++finish("frames \(n) failed-frames \(frameFails) textures-made-again \(recreatedTextures) surfaces-made-again \(recreatedSurfaces)") +diff --git a/scripts/internal/census/probe_signal.swift b/scripts/internal/census/probe_signal.swift +new file mode 100644 +index 00000000..b3d408d9 +--- /dev/null ++++ b/scripts/internal/census/probe_signal.swift +@@ -0,0 +1,43 @@ ++// probe_signal.swift (CPU read after a shared-event signal): swiftc -O probe_common.swift probe_signal.swift -o probe_signal ++// One command buffer renders a gradient into a surface, signals a shared event, then keeps the GPU ++// busy with unrelated work. The CPU waits for the event, not for the command buffer, and reads the ++// surface at once. Metal promises the render's writes are visible at the signal, so the read must ++// see them though the command buffer has not completed. It prints whether the command buffer was ++// still running at the read; a round where it had completed proves nothing and is counted apart. ++import Foundation ++import Metal ++import IOSurface ++ ++let W = 1920, H = 1200 ++let A: [Float] = [1, 1, 0.25, 1], B: [Float] = [0.5, 1, 0.75, 1] ++let rounds = Int(env("PROBE_ROUNDS", 20)) ++let busyDraws = Int(env("PROBE_BUSY_DRAWS", 600)) ++print("device:", dev.name, "rounds \(rounds), busy draws \(busyDraws)") ++let busySurface = makeSurface(W, H), busy = texture(busySurface) ++var running = 0, completed = 0 ++for round in 0..= 1 ++ let stillRunning = b.status != .completed ++ // Read with no lock first, as an app that trusts the event may, then with a read lock. ++ let (good, total, first) = compare(s, lock: false) { want(c, W, H, $0, $1) } ++ let (good2, total2, first2) = compare(s) { want(c, W, H, $0, $1) } ++ b.waitUntilCompleted() ++ if stillRunning { running += 1 } else { completed += 1 } ++ let ok = signalled && good == total && good2 == total2 ++ print(ok ? " ok " : " BAD ", "round \(round)", signalled ? "" : "(the event never came)", ++ stillRunning ? "the buffer was running" : "the buffer had completed", ++ "unlocked \(good)/\(total) locked \(good2)/\(total2)", first, first2) ++ if !ok { failed += 1 } ++} ++finish("rounds-read-while-running \(running) rounds-already-completed \(completed)") +diff --git a/scripts/internal/census/probe_tail.swift b/scripts/internal/census/probe_tail.swift +new file mode 100644 +index 00000000..c0b3641f +--- /dev/null ++++ b/scripts/internal/census/probe_tail.swift +@@ -0,0 +1,52 @@ ++// probe_tail.swift (B1): swiftc -O probe_common.swift probe_tail.swift -o probe_tail ++// A 3200x1800 BGRA surface's rows end at 23,040,000 bytes; it is made with 64 KiB more, so its ++// allocation has whole pages past the rows. The CPU writes those trailing pages (as an app keeping ++// its own data after the pixels may) between GPU renders and CPU reads of the pixels. Every check ++// must pass, and the trailing bytes must keep what the CPU wrote. ++// That Coherent stays on is checked on the host: after a run the census must not have fused ++// ("census: fused" absent from the QEMU log) and the m2v counters closed_by_overlap and ++// closed_by_marks must not grow. The probe prints the time per round too: a surface that fell back ++// to copies costs a full copy each way, so a large jump between rounds is a hint. ++import Foundation ++import Metal ++import IOSurface ++ ++let W = 3200, H = 1800 ++let rowsEnd = W * 4 * H ++let s = makeSurface(W, H, allocSize: rowsEnd + 65536) ++let t = texture(s) ++let A: [Float] = [1, 1, 0.25, 1], B: [Float] = [0.5, 1, 0.75, 1] ++print("device:", dev.name, "bytesPerRow \(s.bytesPerRow) allocationSize \(s.allocationSize) rows end \(s.bytesPerRow * H)") ++let tailStart = s.bytesPerRow * H, tailLength = s.allocationSize - tailStart ++if tailLength < 4096 { print(" note: only \(tailLength) bytes past the rows; the trailing-page case is not reached") } ++func writeTail(_ v: UInt8) { ++ s.lock(options: [], seed: nil) ++ memset(s.baseAddress + tailStart, Int32(v), tailLength) ++ s.unlock(options: [], seed: nil) ++} ++func tailHolds(_ v: UInt8) -> Bool { ++ s.lock(options: .readOnly, seed: nil); defer { s.unlock(options: .readOnly, seed: nil) } ++ let p = (s.baseAddress + tailStart).assumingMemoryBound(to: UInt8.self) ++ return (0.. 2 }) { bad += 1; if first.isEmpty { first = "(\(x),\(y)) got \(g) want \(w)" } } ++ } ++ } ++ print(bad == 0 ? " ok " : " BAD ", "the blit's bytes", "\(total - bad)/\(total)", first) ++ if bad != 0 { failed += 1 } ++} ++finish() diff --git a/patches/reims-vgpu/0023-Coherent-a-CPU-write-through-a-surface-s-own-mapping.patch b/patches/reims-vgpu/0023-Coherent-a-CPU-write-through-a-surface-s-own-mapping.patch new file mode 100644 index 0000000..462f404 --- /dev/null +++ b/patches/reims-vgpu/0023-Coherent-a-CPU-write-through-a-surface-s-own-mapping.patch @@ -0,0 +1,170 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 07:09:42 +0000 +Subject: [PATCH 23/25] Coherent: a CPU write through a surface's own mapping + tells the traps (R1) + +- write_surface_bytes (replaceRegion: into a surface made GPU-read-only, row + by row into the guest's pages) wrote under a trapping surface with no + notice, so the image another texture over the same pages held stayed stale + and its later copy back could overwrite the write. It now tells + shadow_coherent::host_writes of the view of the bytes and of each run of + pages they land in, before they land; the os::surface lend test checks the + ranges and the order. +- The IOSurface ring's test pattern write (IoSurfaceTable::test_pattern) does + the same through a new Surface::host_writes (no-op by default, the Linux + surface's is surface_device_access). +- host_meets' doc comment sat on surface_device_access. +- vendor/qemu 82cef5aac4: address_space_ld*/st* reach the census. + +Not hooked, as they cannot meet a surface's pages: GuestPtr writes and reads +(FIFO rings, device and display registers, task roots and object lists, +placement words, the IOSurface descriptor table, the ICB write-back into its +own guest buffer) are the guest kernel's protocol memory; the display's +property buffers and rate-map samples are the library's own memory; Map::write +and WritableCursor have no product caller over surface pages. Reims tests +single-threaded, before and after: vulkan lib 410 passed 21 failed, core lib +2376 passed 1 failed, the same failures. + +Co-Authored-By: Claude Opus 5.5 +--- + crates/reims-vgpu-core/src/iosurface/table.rs | 2 + + .../src/platform/os/surface.rs | 4 ++ + .../reims-vgpu-vulkan/src/os/linux/surface.rs | 4 ++ + crates/reims-vgpu-vulkan/src/os/surface.rs | 52 ++++++++++++++++--- + 4 files changed, 55 insertions(+), 7 deletions(-) + +diff --git a/crates/reims-vgpu-core/src/iosurface/table.rs b/crates/reims-vgpu-core/src/iosurface/table.rs +index ade7fcaa..045e3b73 100644 +--- a/crates/reims-vgpu-core/src/iosurface/table.rs ++++ b/crates/reims-vgpu-core/src/iosurface/table.rs +@@ -397,6 +397,8 @@ impl HostDeviceCore

{ + return IoReturn::BadArgument; + }; + let _ = surface.lock(0); ++ // LOCAL (Atlas): the host's GPU hears of the write first (`Surface::host_writes`). ++ surface.host_writes(); + let size = surface.alloc_size(); + let base = GuestPtr::new(surface.base_address()); + // The base less the offset, wrapping. +diff --git a/crates/reims-vgpu-core/src/platform/os/surface.rs b/crates/reims-vgpu-core/src/platform/os/surface.rs +index 4f7793c6..481dbff6 100644 +--- a/crates/reims-vgpu-core/src/platform/os/surface.rs ++++ b/crates/reims-vgpu-core/src/platform/os/surface.rs +@@ -44,6 +44,10 @@ pub trait Surface: Handle { + fn lookup_by_port(&self) -> Option + where + Self: Sized; ++ /// LOCAL (Atlas): the library is about to write the surface's bytes through its own mapping, ++ /// not through a texture (`IoSurfaceTable::test_pattern`): the host's GPU, whose textures over ++ /// the same pages may trap (`shadow_coherent` in metal2vulkan), is told. Nothing by default. ++ fn host_writes(&self) {} + } + + /// Creates IOSurfaces. The private keys a spec can carry (client address ranges, direction, +diff --git a/crates/reims-vgpu-vulkan/src/os/linux/surface.rs b/crates/reims-vgpu-vulkan/src/os/linux/surface.rs +index a2540a79..c7bae0a0 100644 +--- a/crates/reims-vgpu-vulkan/src/os/linux/surface.rs ++++ b/crates/reims-vgpu-vulkan/src/os/linux/surface.rs +@@ -334,6 +334,10 @@ impl Surface for LinuxSurface { + self.data().alloc_size + } + ++ fn host_writes(&self) { ++ crate::os::surface_device_access(self, true); ++ } ++ + fn lookup_by_port(&self) -> Option { + Some(LinuxSurface(ManuallyDrop::new(Arc::new(Object( + Arc::clone(&self.0.0), +diff --git a/crates/reims-vgpu-vulkan/src/os/surface.rs b/crates/reims-vgpu-vulkan/src/os/surface.rs +index af2d644b..39ded8fd 100644 +--- a/crates/reims-vgpu-vulkan/src/os/surface.rs ++++ b/crates/reims-vgpu-vulkan/src/os/surface.rs +@@ -292,6 +292,19 @@ pub(crate) fn memory_kind(surface: &OsSurface) -> &'static str { + /// reaches (`replaceRegion:`, measured: `tests/client_direction.rs`). False, writing nothing, + /// for a surface not made over the guest's pages, and for bytes past them. + pub(crate) fn write_surface_bytes(surface: &OsSurface, offset: u64, bytes: &[u8]) -> bool { ++ write_surface_bytes_noticed(surface, offset, bytes, rt::shadow_coherent::host_writes) ++} ++ ++/// [`write_surface_bytes`], `notice` told of each range of host memory it is about to write. ++/// LOCAL (Atlas): the runtime's trapping surfaces (`shadow_coherent::host_writes`) are told ++/// before the bytes land, as a lend's are ([`host_meets`]): the surface's view of the bytes and ++/// each run of the guest's pages they lie in, so the next command over them copies them in. ++fn write_surface_bytes_noticed( ++ surface: &OsSurface, ++ offset: u64, ++ bytes: &[u8], ++ mut notice: impl FnMut(std::ops::Range), ++) -> bool { + let Some(held) = recorded(surface, |made| { + let first = made.offset.saturating_add(offset); + in_runs( +@@ -305,6 +318,13 @@ pub(crate) fn write_surface_bytes(surface: &OsSurface, offset: u64, bytes: &[u8] + if total != bytes.len().to_u64() { + return false; + } ++ let base = surface.base_address().as_ptr::().addr().to_u64(); ++ if base != 0 { ++ notice(base.saturating_add(offset)..base.saturating_add(offset + bytes.len().to_u64())); ++ } ++ for run in &held { ++ notice(run.clone()); ++ } + let mut from = bytes; + for run in held { + let (now, rest) = from.split_at((run.end - run.start).to_usize()); +@@ -521,13 +541,6 @@ impl Lent { + } + } + +-/// LOCAL (Atlas): tells the runtime's trapping surfaces (`shadow_coherent`) that a host outside +-/// it is about to read or write `surface`'s `length` bytes (`notice` is `host_reads` or +-/// `host_writes`), so a read sees the GPU's writes copied back first and a write reaches the +-/// image the next command uses. A trapping surface lies over the buffer a texture on it is over, +-/// which is either the run of the guest's pages the bytes lie in or the surface's own view of +-/// them at `base`, so both are named; a surface that meets neither costs a scan of the tracked +-/// surfaces, none when nothing traps. + /// LOCAL (Atlas): a device is about to read (`write` false) or write `surface`'s bytes as a lend + /// would lend them, through memory lent to it outside a lend (a dma-buf of its pages): the + /// trapping surfaces over them are told, as [`lend_surface_bytes`] tells them. +@@ -543,6 +556,13 @@ pub fn surface_device_access(surface: &OsSurface, write: bool) { + host_meets(&made, base, offset, length, notice); + } + ++/// LOCAL (Atlas): tells the runtime's trapping surfaces (`shadow_coherent`) that a host outside ++/// it is about to read or write `surface`'s `length` bytes (`notice` is `host_reads` or ++/// `host_writes`), so a read sees the GPU's writes copied back first and a write reaches the ++/// image the next command uses. A trapping surface lies over the buffer a texture on it is over, ++/// which is either the run of the guest's pages the bytes lie in or the surface's own view of ++/// them at `base`, so both are named; a surface that meets neither costs a scan of the tracked ++/// surfaces, none when nothing traps. + fn host_meets( + made: &Option<(Vec>, u64)>, + base: u64, +@@ -829,6 +849,24 @@ pub(crate) mod tests { + [base..base + 2 * page - 0x40, high + 0x40..high + page, low..low + page], + "a trapping surface is told of the view and of each run of pages the bytes lie in" + ); ++ // A CPU write through a texture on a GPU-read-only surface (`replaceRegion:`) is told so ++ // too, before its bytes land: the view of the bytes, then each run they lie in. ++ let mut noticed = Vec::new(); ++ let mut landed_first = None; ++ let offset = page - 0x40 - 4; ++ assert!(write_surface_bytes_noticed(&s, offset, &[9; 8], |r| { ++ landed_first.get_or_insert(byte_at(high + page - 4)); ++ noticed.push(r); ++ })); ++ assert_eq!( ++ noticed, ++ [base + offset..base + offset + 8, high + page - 4..high + page, low..low + 4], ++ "a CPU write into the pages tells the trapping surfaces of the view and the runs" ++ ); ++ assert_ne!(landed_first, Some(9), "before the bytes land"); ++ assert_eq!((byte_at(high + page - 4), byte_at(low + 3)), (9, 9)); ++ // SAFETY: the pages are ours, allocated above: the lend below reads them as it found them. ++ unsafe { *std::ptr::with_exposed_provenance_mut::(low.to_usize()) = 7 }; + let split = (page - 0x40).to_usize(); + let (mut before, mut at, mut landed) = (0, 0, 0); + lend_surface_bytes(&s, &mut |b| { diff --git a/patches/reims-vgpu/0024-census-probes-autoreleasepool-per-frame-in-probe_lon.patch b/patches/reims-vgpu/0024-census-probes-autoreleasepool-per-frame-in-probe_lon.patch new file mode 100644 index 0000000..bdd5d74 --- /dev/null +++ b/patches/reims-vgpu/0024-census-probes-autoreleasepool-per-frame-in-probe_lon.patch @@ -0,0 +1,116 @@ +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 +From: macbox +Date: Tue, 3 Nov 2026 07:33:47 +0000 +Subject: [PATCH 24/25] census probes: autoreleasepool per frame in probe_long; + multi-file build through main.swift + +probe_long faulted in the guest driver (SIGBUS) after ~4000 frames on every build, from +per-frame Metal objects never released; with the pool it ran 25339 frames clean on 300fd13. + +Co-Authored-By: Claude Opus 5.5 +--- + scripts/internal/census/probe_long.swift | 53 ++++++++++--------- + scripts/internal/census/probe_signal.swift | 2 +- + scripts/internal/census/probe_tail.swift | 2 +- + scripts/internal/census/probe_untracked.swift | 2 +- + 4 files changed, 32 insertions(+), 27 deletions(-) + +diff --git a/scripts/internal/census/probe_long.swift b/scripts/internal/census/probe_long.swift +index 760a2c4f..d513771f 100644 +--- a/scripts/internal/census/probe_long.swift ++++ b/scripts/internal/census/probe_long.swift +@@ -1,4 +1,5 @@ +-// probe_long.swift (long run): swiftc -O probe_common.swift probe_long.swift -o probe_long ++// probe_long.swift (long run): copy to a folder as main.swift, then ++// swiftc -O probe_common.swift /main.swift -o probe_long + // PROBE_SECONDS (default 300). Thousands of frames over minutes, each one checked: + // - Four surfaces are used in turn, so each is reused every fourth frame. + // - Every 37th frame, a surface's texture is dropped and a new one made on it. +@@ -24,30 +25,34 @@ print("device:", dev.name, "\(W)x\(H) BGRA for \(Int(seconds)) s") + let start = Date() + var n = 0, recreatedTextures = 0, recreatedSurfaces = 0, frameFails = 0, firstFail = "" + while Date().timeIntervalSince(start) < seconds { +- let i = n % slots.count +- if n % 101 == 100 { slots[i] = makeSlot(); recreatedSurfaces += 1 } +- else if n % 37 == 36 { +- slots[i].writer = texture(slots[i].surface); slots[i].reader = texture(slots[i].surface); recreatedTextures += 1 +- } +- let slot = slots[i], c = colour(n), y0 = (n * 13) % (H - band) +- gradient(slot.writer, c) +- let cpuBand = n % 5 == 0 +- if cpuBand { +- slot.surface.lock(options: [], seed: nil) +- for y in y0..<(y0 + band) { +- let row = (slot.surface.baseAddress + y * slot.surface.bytesPerRow).assumingMemoryBound(to: UInt32.self) +- for x in 0.. [UInt8] = { x, y in ++ cpuBand && y >= y0 && y < y0 + band ? [0x60, 0x40, 0x20, 0xff] : want(c, W, H, x, y) ++ } ++ let (g1, t1, f1) = compare(slot.surface, step: 53, expect) ++ let (g2, t2, f2) = compare(viaGPU(slot.reader), step: 53, expect) ++ if g1 != t1 || g2 != t2 { ++ frameFails += 1 ++ if firstFail.isEmpty { firstFail = "frame \(n) slot \(i): cpu \(g1)/\(t1) \(f1) gpu \(g2)/\(t2) \(f2)" } + } +- slot.surface.unlock(options: [], seed: nil) +- } +- let expect: (Int, Int) -> [UInt8] = { x, y in +- cpuBand && y >= y0 && y < y0 + band ? [0x60, 0x40, 0x20, 0xff] : want(c, W, H, x, y) +- } +- let (g1, t1, f1) = compare(slot.surface, step: 53, expect) +- let (g2, t2, f2) = compare(viaGPU(slot.reader), step: 53, expect) +- if g1 != t1 || g2 != t2 { +- frameFails += 1 +- if firstFail.isEmpty { firstFail = "frame \(n) slot \(i): cpu \(g1)/\(t1) \(f1) gpu \(g2)/\(t2) \(f2)" } + } + n += 1 + if n % 500 == 0 { +diff --git a/scripts/internal/census/probe_signal.swift b/scripts/internal/census/probe_signal.swift +index b3d408d9..81f19181 100644 +--- a/scripts/internal/census/probe_signal.swift ++++ b/scripts/internal/census/probe_signal.swift +@@ -1,4 +1,4 @@ +-// probe_signal.swift (CPU read after a shared-event signal): swiftc -O probe_common.swift probe_signal.swift -o probe_signal ++// probe_signal.swift (CPU read after a shared-event signal): copy to a folder as main.swift, then swiftc -O probe_common.swift /main.swift -o probe_signal (top-level code must be in main.swift) + // One command buffer renders a gradient into a surface, signals a shared event, then keeps the GPU + // busy with unrelated work. The CPU waits for the event, not for the command buffer, and reads the + // surface at once. Metal promises the render's writes are visible at the signal, so the read must +diff --git a/scripts/internal/census/probe_tail.swift b/scripts/internal/census/probe_tail.swift +index c0b3641f..10b7f633 100644 +--- a/scripts/internal/census/probe_tail.swift ++++ b/scripts/internal/census/probe_tail.swift +@@ -1,4 +1,4 @@ +-// probe_tail.swift (B1): swiftc -O probe_common.swift probe_tail.swift -o probe_tail ++// probe_tail.swift (B1): copy to a folder as main.swift, then swiftc -O probe_common.swift /main.swift -o probe_tail (top-level code must be in main.swift) + // A 3200x1800 BGRA surface's rows end at 23,040,000 bytes; it is made with 64 KiB more, so its + // allocation has whole pages past the rows. The CPU writes those trailing pages (as an app keeping + // its own data after the pixels may) between GPU renders and CPU reads of the pixels. Every check +diff --git a/scripts/internal/census/probe_untracked.swift b/scripts/internal/census/probe_untracked.swift +index 42622f90..f4499fe2 100644 +--- a/scripts/internal/census/probe_untracked.swift ++++ b/scripts/internal/census/probe_untracked.swift +@@ -1,4 +1,4 @@ +-// probe_untracked.swift (P2): swiftc -O probe_common.swift probe_untracked.swift -o probe_untracked ++// probe_untracked.swift (P2): copy to a folder as main.swift, then swiftc -O probe_common.swift /main.swift -o probe_untracked (top-level code must be in main.swift) + // An untracked-hazard texture is made on a surface first, then a tracked one on the same surface. + // The GPU writes through the tracked one and reads through the untracked one, which the runtime's + // hazard tracking does not see. The read must still see the write: the untracked reader is diff --git a/patches/reims-vgpu/0004-upstream-f6b7e927-video-acceleration-switch.patch b/patches/reims-vgpu/0025-new-Let-a-VM-s-settings-turn-video-acceleration-off.patch similarity index 98% rename from patches/reims-vgpu/0004-upstream-f6b7e927-video-acceleration-switch.patch rename to patches/reims-vgpu/0025-new-Let-a-VM-s-settings-turn-video-acceleration-off.patch index b45e173..e636d11 100644 --- a/patches/reims-vgpu/0004-upstream-f6b7e927-video-acceleration-switch.patch +++ b/patches/reims-vgpu/0025-new-Let-a-VM-s-settings-turn-video-acceleration-off.patch @@ -1,7 +1,7 @@ -From 01eeb65c8a9a24ee722f693bab9eef66b917373f Mon Sep 17 00:00:00 2001 +From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: macbox Date: Wed, 28 Oct 2026 16:46:17 +0200 -Subject: [PATCH] :new: Let a VM's settings turn video acceleration off +Subject: [PATCH 25/25] new: Let a VM's settings turn video acceleration off A user asked to be able to turn off the VideoToolbox and M2 scaler devices that 054ac2080 gave every VM on reims-vgpu, the host's hardware video decode, @@ -27,7 +27,7 @@ is unchanged: with the devices off, its video kexts match nothing. 6 files changed, 166 insertions(+), 20 deletions(-) diff --git a/crates/reims-ctl/src/cli.rs b/crates/reims-ctl/src/cli.rs -index b3e414e..1942034 100644 +index b3e414e7..1942034f 100644 --- a/crates/reims-ctl/src/cli.rs +++ b/crates/reims-ctl/src/cli.rs @@ -337,6 +337,10 @@ pub struct SetArgs { @@ -70,7 +70,7 @@ index b3e414e..1942034 100644 c.logs.enabled = b; } diff --git a/crates/reims-ctl/src/create.rs b/crates/reims-ctl/src/create.rs -index 1326c6d..4c42384 100644 +index 1326c6d5..4c423841 100644 --- a/crates/reims-ctl/src/create.rs +++ b/crates/reims-ctl/src/create.rs @@ -291,6 +291,7 @@ fn base_config(spec: &Spec, profile: Profile, disk: PathBuf, disk_format: ImageF @@ -82,7 +82,7 @@ index 1326c6d..4c42384 100644 gpu: spec.gpu.clone(), audio: Audio { diff --git a/crates/reims-ctl/src/qemu.rs b/crates/reims-ctl/src/qemu.rs -index 08e0d7c..8dc4958 100644 +index 08e0d7c6..8dc49580 100644 --- a/crates/reims-ctl/src/qemu.rs +++ b/crates/reims-ctl/src/qemu.rs @@ -57,6 +57,8 @@ @@ -182,7 +182,7 @@ index 08e0d7c..8dc4958 100644 #[test] fn a_disk_on_nvme_has_a_controller_of_its_own() { diff --git a/crates/reims-ctl/src/tui.rs b/crates/reims-ctl/src/tui.rs -index 0385757..5785e5d 100644 +index 03857573..5785e5dd 100644 --- a/crates/reims-ctl/src/tui.rs +++ b/crates/reims-ctl/src/tui.rs @@ -1442,7 +1442,8 @@ impl App { @@ -291,7 +291,7 @@ index 0385757..5785e5d 100644 fn ctrl_with_enter_or_a_right_click_opens_the_menu_of_everything() { let mut app = test_app(vec![vm_row("alpha", true), vm_row("beta", false)]); diff --git a/crates/reims-ctl/src/vm.rs b/crates/reims-ctl/src/vm.rs -index c6351ea..9922617 100644 +index c6351ea5..99226175 100644 --- a/crates/reims-ctl/src/vm.rs +++ b/crates/reims-ctl/src/vm.rs @@ -32,6 +32,11 @@ pub struct VmConfig { @@ -353,7 +353,7 @@ index c6351ea..9922617 100644 /// gives. #[test] diff --git a/docs/qemu.md b/docs/qemu.md -index 19b62da..c5c3615 100644 +index 19b62da1..c5c36152 100644 --- a/docs/qemu.md +++ b/docs/qemu.md @@ -160,7 +160,8 @@ discrete ones first, where that GPU is a software rasterizer or offers diff --git a/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html b/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html index 370ad8e..e74855f 100644 --- a/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html +++ b/site/pages/guides/gpu-accelerated-macos-vm-on-linux.html @@ -33,20 +33,20 @@

The path macbox uses

Reims vGPU, metal2vulkan, reims-util and their QEMU fork are the work of Anees Iqbal (steelbrain). macbox builds them from pinned commits, applies its patches, and runs them in a container.

What macbox's patches change

-

On every GPU measured so far the GPU is the bottleneck, so what matters is how much extra work translation adds per frame. Before the patches, most of it was copies: macOS's framebuffer and the simulator's display surface live in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% with the desktop. The patch series renders those textures in place, adds Arrow Lake-specific device facts, and a file of live switches for experiments. Details: patches/README.md.

+

On every GPU measured so far the GPU is the bottleneck, so what matters is how much extra work translation adds per frame. Before the patches, most of it was copies: macOS's framebuffer and the simulator's display surface live in guest RAM, and metal2vulkan copied each into a GPU image and back around every render pass. Those shadow copies were 51% of all GPU time with the simulator and 67% with the desktop. The patch series renders those textures in place, counts flat compute grids in 32 bits, adds Arrow Lake-specific device facts and a file of live switches for experiments, and, behind an experimental switch, Coherent: QEMU traps the guest CPU's first touch of a surface's page, so the GPU's image can stay ahead of guest RAM and only the touched pages are synced. Details: patches/README.md.

Results

Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, 8 vCPUs and 12 GB. GlassBench is a table scrolling under Liquid Glass bars and a floating glass toolbar; frames counted at the simulator's screen, two runs each.

- + - - - - + + + +
WorkloadReims 2ce3209Reims 7198fe6macbox build
WorkloadReims 2ce3209Reims 7198fe6macbox, Oct 28macbox nownow, Coherent on
Simulator 1x (402×874)53.557.095.8 / 92.5
Simulator 2x (804×1748)33.537.667.8 / 60.5
Simulator 3x (1206×2622)17.219.238.5 / 39.5
Safari at 3x, blurred floating barn/a39.566.4 / 64.2
Simulator 1x (402×874)53.557.095.8 / 92.5n/mn/m
Simulator 2x (804×1748)33.537.667.8 / 60.5n/mn/m
Simulator 3x (1206×2622)17.219.238.5 / 39.550.670.0
Safari at 3x, blurred floating barn/a39.566.4 / 64.2n/mn/m
-

Frames per second. A Genie minimize on a 3440×1440 desktop went from about 12 to 19.5 frames per animation. At 1x the GPU spends 8.0 ms per frame; at 3x, 25.0 ms. Measure your own GPU with the tools in bench/.

+

Frames per second; n/m: not measured on the new build yet. "Now" and "Coherent on" are one A/B in one window (2026-11-05). With Coherent the GPU is 70% busy at 3x instead of 97%, and QEMU uses about 1.6 more CPU cores. A Genie minimize on a 3440×1440 desktop went from about 12 to 19.5 frames per animation on the Oct 28 build, and 30 with Coherent. Coherent is off by default (MACBOX_COHERENT=on): it hung the GPU once on its first day of daily use, and the cause isn't settled. Measure your own GPU with the tools in bench/.

Which GPUs

@@ -70,7 +70,7 @@

Settings that matter

Known limits

  • Animations start about 0.1 s late on an idle display: macOS's driver for this device sends no vblank while nothing presents (reims-vgpu#141).
  • -
  • 3x is GPU-bound at about 35 to 40 fps in heavy Liquid Glass scenes on this iGPU.
  • +
  • 3x is GPU-bound at about 50 fps in heavy Liquid Glass scenes on this iGPU (70 with Coherent).
  • Tried without gain: answering as an Intel or AMD Mac GPU, compressed heap textures, newer Mesa, hugepages (impossible: Reims maps guest RAM in 16 KiB views). The full list is in docs/performance.md.

To set it up, follow How to run macOS on Linux.

diff --git a/site/pages/guides/ios-simulator-on-linux.html b/site/pages/guides/ios-simulator-on-linux.html index 9a63892..a7cec24 100644 --- a/site/pages/guides/ios-simulator-on-linux.html +++ b/site/pages/guides/ios-simulator-on-linux.html @@ -63,10 +63,10 @@

How fast it is

- +
1x (402×874)53.5 fps95.8 / 92.5 fps
2x (804×1748)33.5 fps67.8 / 60.5 fps
3x (1206×2622)17.2 fps38.5 / 39.5 fps
3x (1206×2622)17.2 fps50.6 fps (70.0 with Coherent, experimental)
-

The GPU sets the ceiling: a 3x simulator keeps that iGPU 96 to 98% busy. Simulators animating at the same time share it. Recording uses the VM's software H.264 encoder, about 50 fps with one simulator animating. Full method: docs/performance.md.

+

1x and 2x are from the October 28 build; 3x is the current one. The GPU sets the ceiling: a 3x simulator keeps that iGPU 96 to 98% busy (70% with Coherent). Simulators animating at the same time share it. Recording uses the VM's software H.264 encoder, about 50 fps with one simulator animating. Full method: docs/performance.md.

Limits

    diff --git a/site/pages/index.html b/site/pages/index.html index bf0025d..044a9fd 100644 --- a/site/pages/index.html +++ b/site/pages/index.html @@ -92,26 +92,27 @@

    Works with the agents you already use.

    Performance

    -

    92–96 fps

    +

    17 → 50 fps

    Smooth enough to judge an animation.

    -

    An iPhone simulator running a heavy Liquid Glass screen on an Intel integrated GPU. That's almost twice as fast as the GPU code macbox builds on.

    +

    An iPhone simulator at 3x, an iPhone's real pixels, running a heavy Liquid Glass screen on an Intel integrated GPU. That's three times what the GPU code macbox builds on manages, and the experimental Coherent switch takes it to 70.

    - iOS Simulator frame rate in a Liquid Glass benchmark. 1x: 53.5 fps upstream, 94.2 with the macbox build. 2x: 33.5 and 64.2. 3x: 17.2 and 39.0. + iOS Simulator at 3x in a Liquid Glass benchmark: 17.2 fps as upstream shipped it, 39.0 with the October 28 macbox build, 50.6 with the current build, 70.0 with Coherent on.
    The exact numbers
    - + - - - - - + + + + + +
    WorkloadReims 2ce3209, upstreammacbox build (two runs)
    WorkloadReims 2ce3209, upstreammacbox, Oct 28macbox nownow, Coherent on
    iOS Simulator 1x (402×874), Liquid Glass53.5 fps95.8 / 92.5 fps
    iOS Simulator 2x (804×1748), Liquid Glass33.5 fps67.8 / 60.5 fps
    iOS Simulator 3x (1206×2622, an iPhone's pixels)17.2 fps38.5 / 39.5 fps
    Safari at 3x, a page under a blurred floating barn/a66.4 / 64.2 fps
    macOS desktop, Genie minimize at 3440×1440~12 frames19.5 frames
    iOS Simulator 3x (1206×2622, an iPhone's pixels)17.2 fps38.5 / 39.5 fps50.6 fps70.0 fps
    iOS Simulator 1x (402×874), Liquid Glass53.5 fps95.8 / 92.5 fpsn/mn/m
    iOS Simulator 2x (804×1748), Liquid Glass33.5 fps67.8 / 60.5 fpsn/mn/m
    macOS desktop, dragging a window (presents/s)105130
    macOS desktop, Safari scrolling (presents/s)75–79110
    macOS desktop, Genie minimize at 3440×1440~12 frames19.5 frames29.5 / 30.5 frames
    -

    Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, a VM with 8 vCPUs and 12 GB. Frames counted at the simulator's own screen, after a warm-up, one simulator animating. The GPU is the limit, so a faster GPU moves every number up; other GPUs haven't been measured yet. Method and patches: docs/performance.md.

    +

    Intel Core Ultra 7 265 (Arrow Lake iGPU), Mesa 26.2.4, macOS 26.7.1, iOS 26.5, a VM with 8 vCPUs and 12 GB. Frames counted at the simulator's own screen, after a warm-up, one simulator animating; n/m: not measured on the new build yet. "Now" and "Coherent on" are one A/B in one window. Coherent is off by default: it hung the GPU once on its first day of daily use, and the cause isn't settled. The GPU is the limit, so a faster GPU moves every number up; other GPUs haven't been measured yet. Method and patches: docs/performance.md.

    diff --git a/skills/simulators/SKILL.md b/skills/simulators/SKILL.md index fa0f86f..f1d8d7d 100644 --- a/skills/simulators/SKILL.md +++ b/skills/simulators/SKILL.md @@ -74,7 +74,7 @@ Report what you verified: the screenshots and recordings you took, and what `sim The iOS Simulator draws on the host's GPU through the VM's virtual GPU, so the host's GPU sets the ceiling: - **Measure motion on a 1x iOS simulator** (`slim-1`, `slim-2`, `full`). Example, measured on an Intel Arrow Lake iGPU (Core Ultra 200S) on 2026-10-28: 1x simulators reached about 92 to 96 frames/s in a heavy Liquid Glass benchmark, well above 60. -- **`sharp` and `small` are 3x.** They have 9 times the pixels, so heavy scenes can fall below 60 because of the host's GPU, not your app (about 35 to 40 frames/s on that same iGPU). Use them for still screenshots, not for frame-rate verdicts. +- **`sharp` and `small` are 3x.** They have 9 times the pixels, so heavy scenes can fall below 60 because of the host's GPU, not your app (about 50 frames/s on that same iGPU). Use them for still screenshots, not for frame-rate verdicts. - **Other simulators animating at the same time share the GPU.** If `sim-fps` shows drops, check `mac-sim status` / `android-sim status` and record again. Record with nobody watching the Device panel too, when you can. - **A stretch `sim-fps` calls "still or frozen"** may be the screen standing still, or the app freezing. Look at the frames on both sides of it. diff --git a/versions.env b/versions.env index aa620d5..e13bd0c 100644 --- a/versions.env +++ b/versions.env @@ -1,8 +1,8 @@ -# Pinned sources for `macbox build`. The defaults are the build that has run day to day on an -# Intel Arrow Lake iGPU (Core Ultra 200S, Mesa 26.2.4 ANV): Reims vGPU 7198fe6 with the patches in -# patches/reims-vgpu, its QEMU at bd95509, and metal2vulkan's development branch at 0db5825a with -# the patches in patches/metal2vulkan. `macbox build --upstream` takes Reims's master instead, -# with no patches. +# Pinned sources for `macbox build`. With the patches in patches/ they make the build that has run day +# to day since 2026-11-05 on an Intel Arrow Lake iGPU (Core Ultra 200S, Mesa 26.2.4 ANV), "cohwl": +# Reims vGPU 7198fe6 + patches/reims-vgpu, its QEMU at bd95509 + patches/qemu, and metal2vulkan's +# development branch at 0db5825a + patches/metal2vulkan. `macbox build --upstream` takes Reims's +# master instead, with no patches. REIMS_REPO=https://github.com/steelbrain/reims-vgpu REIMS_REF=7198fe60c9df14593e86d090cb4640c3e45ac65b # vendor/qemu (github.com/steelbrain/qemu-reims-vgpu)